Stop Patching the Same Failure Twice
Modern AI systems are often improved through a reactive flywheel of: (i) deploying the model to users; (ii) collecting observed failures from users; (iii) patching these specific errors for the next release of the model. The loop is appealing because it gives concrete directions for AI developers to respond and improve their systems.
We believe that this habit starts to break down when the goal is generalization of an ML model and that reactive test-driven patches are structurally sub-optimal for generalizable AI systems. A better AI flywheel must be goal-oriented and proactive, meaning it must anticipate future potential related failures and not just the ones at hand. Instead of asking only what data resembles the latest failure, it should ask what task condition the failure exposed, then use that condition to choose tests, data, and model updates before the next related failure reaches deployment.
The Failure Unit Is Too Small
A reactive AI flywheel deploys a model, observes a failure, retrieves similar data or trains on that failure, patches the system, and repeats. However, the problem is the unit of improvement. A failure arrives as a single (or a few) data points, while the underlying weaknesses often live under broader conditions.
Consider an autonomous-driving example. A vehicle crashes into a parked fire truck with active lights. A local response improves detection of parked fire trucks. Later, a similar system crashes into a stopped school bus with an active stop sign. A local response improves detection of stopped school buses.
Autonomous vehicle crashes into a parked fire truck with active lights.
Reactive patch: improve detection of parked fire trucks.Autonomous vehicle crashes into a parked school bus with an active stop sign.
Reactive patch: improve detection of stopped school buses.The model struggles with public service vehicles stopped in traffic with pedestrians nearby.
Rather than reactive patching, a proactive response targets the root issue of detecting public service vehicles, reasoning about nearby pedestrians, and responding appropriately. The update is aimed at the task region behind the incidents, including vehicle classes beyond the one that appeared.
Why Proactive Correction Can Scale Better
How many flywheel cycles does a team need before the long tail is covered? Suppose there are \(M\) specific failure scenarios, but those scenarios are governed by \(K\) broader task factors, where \(K\) can be much smaller than \(M\).
A reactive update tries to fix the observed scenario and succeeds with probability \(p_R\). A proactive update tries to fix the broader factor group and succeeds with probability \(p_P\). Note that since fixing a root condition is harder than patching an observed case, we must have \(p_P \lt p_R\).
Then, we can track, in expectation, the number of cycles needed by a reactive strategy \(T^R\) and the number of cycles needed by a proactive one \(T^P\).
Reactive correction scales with the number of scenarios, while proactive correction scales with the number of task factors. This advantage can hold even when proactive fixes are harder per attempt, since broader fixes reduce future flywheel cycles.
The same logic applies to backlog (\(B^R_t\) and \(B^P_t\)) for engineering teams. Unresolved failures must be tracked, triaged, prioritized, retested, and explained. When failures share factors, proactive updates can reduce the backlog.
A Test Space Gives Feedback Context
We propose to organize deployed-system feedback around a test space. A test space is a structured representation of the conditions under which the system needs to succeed, i.e., scenarios, constraints, behaviors, relationships, safety requirements, and edge cases. Coverage over a test space is needed to evolve a reactive flywheel into a proactive, test-driven one. For autonomous driving, this may include driving scenes for low visibility, narrow roads, stopped vehicles, public service vehicles, unusual right-of-way situations, and many other factors.
A test space gives failures a coordinate system. Once feedback is mapped into that space, an incident can reveal an uncovered region of the task, with the failed example as evidence. A proactive flywheel gathers feedback, aligns it to the test space, recognizes weakness patterns, and routes fixes back to data or model updates.
What Has To Be Built
A proactive flywheel needs several technical pieces. To build this, we pose several open research questions.
First, teams need to create a test space to represent factors covering the task at hand. How do we discover the concrete factors implicit in task requirements? Factors may come from safety requirements, evaluation protocols, domain expertise, design documents, or known deployment constraints. Language and multimodal models may help extract and organize these factors from text, images, videos, and logs.
Second, teams need to represent the test space. How do we represent those factors and their relationships? Factors might be represented as embeddings, graphs, symbolic structures, natural-language propositions, scenario programs, or hybrids.
Third, feedback has to be aligned to the test space. How do we map feedback data into the same test space? A failure log, video, image, user report, or trajectory becomes useful when the system can infer which task factors are present and which ones were poorly handled.
Fourth, mapped feedback needs to reveal weakness patterns. How do we identify weakly covered or failure-prone regions? These patterns are where the flywheel becomes proactive: clusters, coverage gaps, and recurring regions of poor performance can all point to future failures before they surface as incidents.
Finally, each weakness has to be routed to the right kind of fix. How do we decide whether a weakness needs data, model changes, evaluation changes, or a combination? Some weaknesses are data problems. Others reflect model limitations, training objectives, evaluation design, or deployment assumptions. Many involve several of these at once.
The test space also has to evolve as deployment conditions change. How do we update the test space as the world changes? This matters most for systems deployed into broad, shifting, high-stakes environments: robotics, autonomous driving, agentic systems, scientific tools, medical AI, and open-ended user-facing applications.