AI System Operation and Monitoring

ISO/IEC 42001:2023 Annex A.6.2.6 AI system operation and monitoring requires organizations to define and document the necessary elements for ongoing operation, which must at a minimum include system and performance monitoring, repairs, updates, and user support.

To design a compliant monitoring program, organizations must establish continuous monitoring for AI systems compliance by setting defined performance criteria, creating mechanisms to track general errors, and formalizing processes for ongoing support and system repairs.

AI system performance monitoring metrics should encompass technical indicators such as error rates, latency, processing duration, and confidence rates, alongside specific statistical criteria like the F score determined by the AI's intended task.

Organizations define how to monitor model drift in production by continuously comparing live inputs against historical training baselines, and they respond by utilizing documented MLOps procedures to trigger retraining when performance falls below acceptable thresholds.

ISO 42001 AI monitoring controls evidence includes maintaining detailed event logs of system operations, records of performance metric tracking, and audit trails demonstrating when patches, updates, or model retraining events occurred. Tools like WatchDog Security's Compliance Center can help organize these artifacts by control and assign evidence owners, and WatchDog Security's Secure File Sharing can be used to share selected logs and reports with auditors under access controls and audit trails.

By adopting MLOps monitoring and alerting best practices, organizations should route alerts through a formalized AI incident monitoring and response process that engages appropriate technical personnel to investigate, repair, or roll back the system.

Organizations are expected to implement rigorous change management for AI model updates and retraining, ensuring all patches and functional modifications are tested, approved, and clearly communicated to end users before taking effect in production. Tools like WatchDog Security's Policy Management can help maintain documented change procedures and approval evidence, while WatchDog Security's Risk Register can track update-related risks, treatment actions, and residual risk over time.

Monitoring third-party AI services under ISO 42001 involves tracking the external system's uptime and response quality against contractual service level agreements, and reviewing vendor-provided performance data to ensure alignment with internal compliance requirements.

Monitoring outputs should be reviewed continuously or at planned intervals depending on the system's risk profile, with designated managers and technical leads approving any necessary corrective actions, updates, or model retraining.

Auditors evaluating ISO 42001 AI monitoring controls evidence will look for active performance dashboards, logged records of system updates, documented responses to alerts, and methodologies showing how to monitor AI bias and fairness in production.

Auditors typically expect consistent, repeatable evidence that monitoring, updates, and support processes operate as designed (not just that they exist). Tools like WatchDog Security's Compliance Center can map Annex A.6.2.6 requirements to evidence requests, track review cadence, and centralize monitoring reports, while WatchDog Security's Asset Inventory can help maintain an up-to-date list of in-scope AI systems and owners for coverage validation.

Effective operation and monitoring depends on clear procedures for alert handling, updates, retraining, and user support, plus controlled sharing of sensitive logs and reports. Tools like WatchDog Security's Policy Management can keep SOPs version-controlled with acceptance tracking, and WatchDog Security's Secure File Sharing can support encrypted distribution of monitoring reports with access controls and audit logs.