When an AI automation makes a mistake, stop the harmful action, preserve evidence, correct affected records or messages, and investigate before restarting. The response should match the risk, not the excitement around the technology.
First distinguish a bad draft from a real incident
Not every incorrect AI output is an operational incident. A draft that a reviewer catches before sending is a quality failure and a useful test case. An incorrect message sent to a customer, an unauthorized record change or exposed personal information can require immediate incident handling. The difference is whether the error crossed a control boundary and created consequences.
Classify what happened: inaccurate content, missed information, unauthorized action, privacy or security exposure, service interruption, discriminatory outcome or unsafe recommendation. Record when it occurred, which systems were affected, who can stop it and what evidence exists. An incident response plan should be written before automation is allowed to act independently. Our automate-versus-keep-human guide helps identify where approval gates belong.
Fictional example: the wrong customer reply
Illustrative scenario, not a documented incident. Harbor Desk Supplies, an invented office supplier, uses an AI-assisted customer support workflow. The intended process drafts replies for an employee to approve. During a configuration change, an automatic-send rule is enabled. The system sends a customer a confident but incorrect promise of a free replacement and a delivery date the business cannot meet.
The first objective is not to improve the prompt. It is to prevent further unauthorized replies. The owner disables the send rule, switches to manual review and checks the queue for other affected customers. Staff preserve relevant logs, identify which messages were sent and contact affected customers with accurate information. Only after containment should the team diagnose whether the failure arose from permissions, inaccurate source data, prompt behavior, testing gaps or a mistaken workflow setting.
A first-hour response checklist
1. Contain: disable the affected automation or risky action, revoke excessive permissions if necessary and move to a known manual fallback. Avoid deleting logs in the rush to fix the issue.
2. Assess: determine whether customer information, payments, safety, legal obligations or business-critical operations are affected. Escalate immediately to the appropriate owner, security contact or professional adviser for high-impact events.
3. Preserve: capture timestamps, version identifiers, workflow settings, input/output samples and relevant audit records in an access-controlled location. Minimize unnecessary personal data in the incident report.
4. Correct: reverse erroneous actions where safe, correct records, notify affected parties when appropriate and follow applicable contractual or legal reporting requirements.
5. Decide: keep the system paused until an authorized owner approves a controlled restart. A quick prompt tweak is not sufficient proof that the risk is resolved.
Find the failure mode, not a convenient scapegoat
An automation may fail because its instructions are ambiguous, its source data is outdated, its permissions are too broad, a third-party API changes, a human misunderstands the review screen, or a malicious message manipulates the model. The model's output is only one part of the system. Investigate the complete path from trigger to final action.
Ask: What input started the process? What data did the model receive? What output did it produce? Which component decided to act? What permission allowed the action? What monitoring should have caught the problem? Could a simpler non-AI rule have prevented it? Use a cause-and-control table rather than writing “AI hallucinated” and moving on. OWASP documents threats such as prompt injection and excessive agency in its LLM security guidance.
Build a practical severity matrix
Low: a draft contains an obvious formatting error and no one sees it outside the review queue. Log it, fix the test case and continue supervised evaluation.
Moderate: an incorrect customer reply is sent but does not expose private data or create material harm. Pause outbound messages, correct the response and review similar cases.
High: an automation changes orders, billing or access rights incorrectly, or reveals confidential information. Disable relevant integrations, involve security and business leadership, preserve evidence and assess notification obligations.
Critical: the workflow could cause physical harm, significant financial loss or widespread privacy exposure. Follow the organization's emergency and incident procedures and seek qualified specialist support.
These categories are illustrative; actual severity depends on context and law. A business should decide who can declare an incident and who can authorize recovery before deploying the workflow.
Design controls that catch mistakes earlier
The strongest protection is to avoid giving a model authority it does not need. Separate drafting from sending, recommendations from approvals, and data extraction from permanent record changes. Use least-privilege accounts, limits on transactions, restricted destinations and human review for consequential decisions. Where practical, validate outputs against structured rules before any action occurs.
Test routine examples and adversarial cases: missing fields, duplicate requests, conflicting instructions, malicious text, unusual formatting, unavailable systems and stale knowledge. Monitor not only model accuracy but also the rate of human overrides, corrections, complaints, exceptions and unexpected API activity. The AI governance guide and pilot checklist help formalize these decisions.
How to restart safely
Do not restore full autonomy immediately after a fix. Reproduce the failure using a safe test case, verify the new control, test neighboring failure modes and document the change. Restart in a supervised mode with limited volume and an explicit rollback option. Compare outcomes with the original acceptance criteria. If the root cause remains uncertain, continue manual operation.
A restart checklist should include named approval, version number, known limitations, monitoring owner, escalation contact, rollback instructions and the date for a follow-up review. An external consultant should not be the only person who understands how to stop the system. See automation ownership after handover.
Who is accountable?
The business remains responsible for the processes it operates, even when it purchases a tool or hires a consultant. Contract terms may allocate responsibilities among the client, consultant and vendors, but accountability should not be left ambiguous. Identify who approves the use case, controls data, maintains the workflow, monitors outcomes and responds to incidents.
A consultant should explain limitations and transfer documentation. A vendor should provide relevant service and security information. Employees need clear escalation routes and permission to stop unsafe automation. Where legal, privacy or regulated decisions are involved, obtain qualified advice. This is operational guidance, not legal counsel. Our hiring guide includes questions to ask before engaging outside help.
Sample incident report template
Record: incident ID and date; reporter; business process; affected workflow/version; summary of observed behavior; whether an action occurred; affected people or records; severity; immediate containment; evidence location; suspected cause; confirmed root cause; corrective action; testing evidence; restart approval; owner and review date. Separate facts from assumptions.
For the fictional supplier, the report might state: “Automatic-send setting enabled during configuration; three incorrect replies identified; sending disabled at 10:12; customers contacted; manual approval reinstated; regression test added.” Do not invent a root cause before logs support it. A good report helps the next operator understand the incident without relying on memory.
Prevention checklist for the next release
For a small team, ownership can be explicit without being bureaucratic. Assign one person to monitor exceptions, another authorized person to approve restarting, and a backup who can access the manual procedure. Test these responsibilities during normal operations rather than discovering gaps during an outage. Review the incident register monthly, looking for recurring patterns instead of isolated anecdotes. A repeated minor error can signal a larger design flaw. Record whether changes reduce the frequency and impact of failures, and include the cost of human review in your assessment. If an automation is only safe when an expert watches it constantly, its supposed efficiency gains may be overstated. The goal is dependable work, not automation for its own sake.
Before an automation is allowed to send messages, update records or trigger payments, test the controls as carefully as the model output. Confirm that the account has only the permissions required for its job. Verify that a reviewer can see the original source information rather than only the AI summary. Test whether the workflow stops safely when a required field is absent.
Run a deliberately difficult test batch: a duplicate request, an outdated policy, a message with contradictory details, a malicious instruction hidden in source text, a network timeout and a downstream system returning an unexpected error. For each case, record the expected safe behavior. “The model should be smart enough” is not an acceptance criterion.
Decide how you will detect a problem after launch. A weekly review of a small sample may be appropriate for a low-risk drafting tool. Higher-risk processes may need alerts, rate limits, transaction thresholds, detailed audit trails or mandatory approval for every action. The monitoring design should match the potential harm, not simply the size of the company.
Practise a rollback before you need one. The operator should know how to disable the trigger, restore manual processing, identify pending work and check whether any actions already happened. Store instructions somewhere accessible even when the automation platform is unavailable. If the consultant is unreachable, the business must still be able to operate.
Finally, schedule a review after every material workflow change. A new prompt, model, integration, knowledge source or permission can change the system's behavior. Treat these changes as releases that require testing and approval, not invisible tweaks. Record what changed, who approved it and which cases were retested. That discipline is how a small organization turns an experimental automation into a controlled business process.
Authoritative guidance and next steps
The NIST AI Risk Management Framework describes governance, mapping, measurement and management of AI risks. NIST Cybersecurity Framework 2.0 offers a broader structure for governance, detection, response and recovery. CISA incident response resources provide additional operational background.
The central lesson is that an AI mistake should trigger a controlled business response, not improvisation. Define who can stop the workflow, what must be logged and what evidence is required before restart. Then measure whether the automation still creates value after accounting for review and recovery costs; see AI ROI and project success measurement.