Extended Practice as Robustness

Query The standard framing in AI evaluation is that real-world complexity makes tasks harder for AI — messier inputs, more edge cases, more ambiguity. Search for any counter-arguments, evidence, or theoretical frameworks suggesting that real-world work context can make AI-assisted work more robust rather than less — that the situated context provides compensating mechanisms (feedback loops, peer review, iterative refinement, domain artifacts, social practices) that improve outcomes beyond what isolated benchmarks predict. Specific questions to investigate: • Do AI coding tools perform better in real projects (with test suites, CI/CD, code review, team practices) than on isolated benchmark tasks? Is there evidence that agentic coding systems that engage the full development workflow outperform the same models on stripped-down benchmarks — not just because they have tool access, but because the work context catches errors and refines inputs? • Do AI diagnostic tools perform better when embedded in clinical workflows (with labs, imaging, patient history, clinical team discussion) than on clean benchmark datasets? Or does the messiness of clinical reality degrade performance? • More broadly: is there any literature arguing that the "warp and weft" of actual human practice — the stuff that benchmarks strip out — adds anti-fragility or distributed robustness rather than just adding noise? This connects to concepts like distributed cognition (Hutchins), ecological rationality (Gigerenzer), anti-fragility (Taleb), and communities of practice (Lave & Wenger). • Are there cases where practitioners report that AI works better than expected in their real workflow despite working worse than expected on benchmarks, or vice versa? Those mismatches would be direct evidence. --- Full Report Extended Practice as Robustness in AI-Assisted Work Framing the question A large fraction of AI evaluation—especially for large language models (LLMs) and code/clinical models—implicitly treats "real-world context" as adversarial: messier inputs, more ambiguity, more edge cases, more distribution shift, and therefore lower performance than on clean benchmarks. That framing is often correct at the model-only level (e.g., worse external validation than internal test performance in medical imaging), but it can be misleading about the system-level performance of AI embedded in practice—where the "warp and weft" of work adds constraints, redundancy, and corrective feedback. citeturn4search2turn31view0turn21view0 A useful distinction emerging across multiple literatures is: • Model performance: what the model does in isolation (single-shot answers; curated datasets; minimal context). • Sociotechnical system performance: what happens when a model is embedded in workflows that include artifacts (tests, protocols), tooling (CI, EHR), and social practices (peer review, escalation, audit, accountability). turn17search3turn25search12 This distinction is not just philosophical. In healthcare, Cabitza & Zeitoun explicitly argue that many notions of "validity" are under-specified if they remain purely statistical and do not incorporate how AI is actually used inside collaborative clinical care; they frame "ecological validity" as inseparable from the setting and the human practices around deployment. Similarly, Weidinger et al. argue that a capability can be benign or harmful depending on context, and propose multi-layer sociotechnical evaluation that goes beyond capability benchmarking to human interaction and systemic impacts. The question you posed—whether extended practice can make AI-assisted work more robust than isolated benchmarks predict—is therefore best answered at the workflow/system level rather than at the "model accuracy on a static dataset" level. Mechanisms by which real-world practice can increase robustness Across software engineering and medicine, the most defensible "counter-argument" to real-world = harder is not that reality is cleaner, but that real work often contains compensating structures that benchmarks deliberately strip out. Four mechanisms recur in the empirical and conceptual literature. First, work artifacts turn open-ended generation into constrained search (often making the right move easier to identify). In software, an existing codebase, style conventions, type constraints, and tests radically narrow the space of acceptable patches; in clinical work, guidelines, structured histories, and diagnostic criteria shape what is plausible and what must be ruled out. This aligns with the ecological rationality tradition: performance depends on how strategies exploit the structure of environments, not only on internal "raw intelligence." citeturn2search25turn31view0turn25search12 Second, negative feedback loops catch and correct errors. In software: compilation errors, failing unit tests, linters, CI, staging deploys, and code review comments. In healthcare: confirmatory labs/imaging, differential diagnosis deliberation, follow-up, second readings, and safety monitoring. These loops are a key reason that "single-shot" evaluations can pessimistically estimate end-to-end reliability of AI-assisted work, because single-shot settings remove the very mechanisms practitioners use to detect and repair mistakes. Third, redundancy and distributed cognition can yield robustness even when components are fallible. The distributed cognition perspective (associated with Hutchins' ship navigation work) treats cognition as spread across people, tools, representations, and procedures; reliability is often an emergent property of the whole system, not of any single "smart" component. That framing maps naturally onto modern socio-technical workflows like PR review + CI in software or double reading + arbitration in mammography screening. citeturn2search4turn2search12turn16view0turn10view1 Fourth, communities of practice create "error protocols" and tacit calibration routines. Lave & Wenger–style situated learning emphasizes that competent practice is not just correct answers but participation in the norms, tools, and accountability structures of a community. Wenger's later articulation of communities of practice similarly foregrounds learning and meaning as emerging from engagement in practice over time. These arguments support a hypothesis that real workflows can become anti-fragile in a limited sense: repeated exposure to errors (caught by feedback loops) can drive incremental improvement of prompts, checklists, tests, and usage policies. citeturn2search3turn17search1turn17search4turn4search31turn31view0 A crucial caveat: the same mechanisms can also amplify error (automation bias, social deference to AI, alert fatigue), so "more context" is not automatically better; it depends on workflow design and incentives. citeturn4search5turn4search4turn13view3 Evidence from software development workflows Controlled and field evidence that workflow embedding changes outcomes Two research lines are especially relevant: (i) human productivity and throughput in realistic work settings, and (ii) agentic/interactive coding systems that explicitly incorporate workflow feedback (tests, execution, validation). A controlled experiment by Peng et al. studied an AI pair programmer in a task that resembles real development more than typical algorithmic benchmarks: developers were asked to implement an HTTP server in JavaScript, and the treatment group with a coding assistant finished substantially faster (reported as 55.8% faster) than control. This is not a proof of higher functional correctness on benchmark tasks; it is evidence that in a realistic task framing, the AI+human system changes the speed of producing workable code. A larger-scale set of field experiments (three randomized controlled trials) studied developers in "ordinary course of business" settings and measured outputs in a structured workflow using version control metrics. The study emphasizes that professional software development is organized around tasks tracked in version control and that pull requests lead to code review and merges; it reports an overall increase in completed tasks when access to an AI coding assistant is randomized. The paper's framing matters for your question: it treats workflow structure (PRs, review, builds) as the measurement substrate—i.e., real work context provides both constraints and instrumentation. Industry case studies are less methodologically clean but help triangulate "mismatch" phenomena. A ZoomInfo engineering report (a practitioner publication) describes a multi-phase deployment evaluation across 400+ developers and reports acceptance rates and satisfaction measures; it also explicitly notes that domain-specific logic and inconsistent code quality can require extra scrutiny—an example of real workflow compensating for AI variability via review effort. Direct evidence that workflow feedback loops improve coding outcomes The strongest evidence for "extended practice as robustness" in coding is not "Copilot in production," but rather execution- and test-mediated iterative refinement, which approximates what CI and local test running provide. Early code LLM evaluation work operationalized correctness as "passes unit tests" and demonstrated that generating multiple samples and selecting by tests yields much higher solve rates than single-shot generation. In the Codex evaluation paper, single-sample solve rates are much lower than solve rates under multi-sample generation with test-based selection; the paper explicitly links this to test-driven development norms and to the reality that integration of new code usually depends on writing and passing unit tests. Agentic systems provide even closer analogues to full workflows because they (a) navigate repositories, (b) modify files, and (c) run tests/commands, producing a loop of action → feedback → correction. SWE-agent explicitly argues that interface design and environmental feedback are critical: the system is built to let an LM navigate repos and execute tests, and it reports that (with the same base model) an agent-computer interface can substantially outperform more "bare shell" interactions. Critically for your question, SWE-agent reports ablations showing that interface/feedback design improves solve rates, without changing the underlying LM weights—evidence that workflow scaffolding (not just "better LM") can drive higher task success. A complementary line of work (on repository-level SWE-bench-style tasks) formalizes patch selection as a validation problem where tests are part of the environment. The "Agentless" approach described in Demystifying LLM-Based Software Engineering Agents makes the validation machinery explicit: it samples multiple patches, generates reproduction tests, executes them against the codebase, and uses regression tests for patch ranking and selection. The authors also emphasize that under realistic setups, the original codebase may not provide bug-triggering tests immediately, motivating test generation as part of the workflow. This paper is effectively an existence proof that "practice-like" scaffolding—test generation + execution + selection—can be treated as a robustness layer on top of LLM patch generation. A NeurIPS paper on testing/validating bug fixes similarly characterizes code agents as outperforming unaided LLMs on repository-level tasks and provides a clean micro-result aligned with your feedback-loop hypothesis: instructing a SWE-agent variant to explicitly execute generated test cases increases success rates (reported in their experiments as an increase from 15.9% to 18.5% for a particular variant). Why this can look "better in practice" than on stripped-down coding benchmarks This body of evidence supports a workflow-centric thesis: • Stripped-down benchmarks typically ask for a correct program in a single pass under severe information constraints. • Real development often asks for an incremental change where the environment supplies "oracles" (tests, compiler errors), enabling iterative convergence. However, it's important not to over-claim. Repository-level benchmarks like SWE-bench are themselves attempts to bring workflow realism into evaluation (real repos, patches, tests), and success rates remain far from "solved," suggesting that real-world complexity still dominates in many cases. citeturn0search0turn8view1 And even within such benchmarks, passing tests can be an imperfect proxy for "actually fixed," highlighting that workflow artifacts can be incomplete, biased, or gamed (a caution that mirrors clinical "metric vs outcome" gaps). citeturn0search28turn12view2 Evidence from clinical workflows Clinical AI is often used as the canonical example of benchmarks overestimating reality because of dataset shift, confounding, and operational complexity. That critique is well-supported: for example, a pneumonia detection CNN study reported that internal performance can exceed external performance, and highlighted that models can latch onto site- or department-specific signals that do not generalize. citeturn4search2 But your question asks for the counterpoint: are there settings where embedding AI in workflow improves robustness (or at least produces better outcomes than benchmark-only thinking predicts)? The best evidence here comes from prospective or population-based studies where AI is integrated into existing clinical safety structures, rather than from "AI vs clinician on a dataset" comparisons. Breast screening as a strong real-world example of workflow-mediated robustness Population breast cancer screening already has built-in redundancy (double reading, arbitration, audit of interval cancers). This creates a natural substrate for evaluating whether AI can be integrated as an additional reader/triage mechanism. In the MASAI trial's earlier safety analysis (randomized, population-based screening accuracy framing), PubMed's abstract reports that 80,033 women were randomized; AI-supported screening produced a cancer detection rate above a pre-specified safety threshold and reduced reading workload substantially (reported as 44.3% reduction). In the protocol-defined analysis of the primary endpoint (interval cancer rate) reported in 2026, the PubMed abstract reports that 105,934 women were randomized, interval cancer rates were non-inferior in the AI-supported arm, sensitivity was higher with AI support, and specificity was the same; it also reports fewer interval cancers with unfavorable characteristics in the AI-supported group. These results matter for your thesis because they are not "clean dataset AUC" claims. They are outcomes that arise from embedding AI inside a structured workflow (AI triage to single/double reading + detection support + downstream follow-up for interval cancers). The robustness here is plausibly distributed: even if AI is imperfect, the workflow's redundancy and audit pathways create error-limiting structure. More broadly, prospective cohort evidence in national screening programs (e.g., reporting on AI-CAD use in screening settings) supports the idea that AI can improve diagnostic performance in "real-world" reading conditions, though the details vary by setting and design. citeturn3search28turn4search9 Diabetic retinopathy screening as workflow redesign and constraint management The IDx-DR pivotal trial (NPJ Digital Medicine, 2018) is a central example of a diagnostic model evaluated prospectively in primary care workflows with an explicit comparison standard (specialist reading center + OCT). The paper reports 900 subjects and reports sensitivity, specificity, and imageability exceeding pre-specified endpoints, with FDA authorization as an autonomous diagnostic system. From the perspective of "extended practice as robustness," the key point is not that the model is magically robust to messy reality, but that the workflow is designed to control variability: standardized operator training, defined referral thresholds ("more than mild DR"), and the ability to route uncertain or positive cases for specialist follow-up. This is structurally analogous to software workflows that treat tests and review as gates: an autonomous or semi-autonomous model is embedded in a larger system that limits damage and escalates difficult cases. Sepsis prediction as a counter-example where real-world complexity dominates unless workflow is redesigned Sepsis prediction systems illustrate how reality can degrade performance through institutional variability, calibration differences, and alert fatigue, and how workflow factors become first-order. A 2026 multicenter prospective validation of Epic Sepsis Model v2 reports AUROC ranges across sites but emphasizes high institutional variability, low positive predictive value, and high alert burden; it explicitly recommends local validation and workflow strategies (such as managing false positives and alert silencing) to reduce alert burden. External validations of earlier versions similarly report poor diagnostic characteristics in some settings (e.g., low PPV in emergency department contexts). citeturn3search23 This is an important "qualifier" for the robustness hypothesis: embedding AI into workflow can increase robustness only if the workflow has the capacity and incentives to absorb errors. If the AI produces too many false positives (high alert burden), the human system can become less robust due to alert fatigue—an example of practice structure failing as a compensating mechanism unless redesigned. turn4search5 Human interaction effects show why context can both help and harm Clinical workflows can create robustness through redundancy, but they can also introduce cognitive vulnerabilities. • A systematic review of automation bias in clinical decision support systems characterizes a tendency to over-rely on automation (commission and omission errors), which can undermine safety. citeturn4search5 • "Who goes first?" (Fogliato et al.) experimentally studies sequencing of AI inferences in a radiology-like task and finds that workflow design (e.g., requiring provisional decisions before seeing AI output) changes agreement with AI and second-opinion seeking behaviors—direct evidence that situated workflow policies shape human-AI team performance, not just model accuracy. citeturn4search4 These findings support a nuanced view: the real-world setting is not only a source of noise; it is a control system whose parameters (display sequencing, escalation norms, accountability) determine whether AI integration yields robustness or brittleness. citeturn4search4turn4search5turn21view0 Practice-centered frameworks that explicitly predict robustness from context Your prompt links to several conceptual anchors; the literature contains arguments that directly support why practice structure can be robustness-enhancing. Distributed cognition (Hutchins) is perhaps the most straightforward bridge: cognition is not "in the head" but distributed across people and artifacts, and reliability can arise from representational structures and coordinated practices. The relevance to AI is that LLMs are not merely tools but new representational/transformational artifacts inserted into an already-distributed system (the dev team, the clinical team). In that framing, it is entirely coherent that adding a fallible component can increase system robustness if it increases redundancy, improves error detection, or changes allocation of attention. citeturn2search12turn2search4turn23search3 Ecological rationality (Gigerenzer and related work) predicts that reasoning strategies are "rational" to the extent they fit environmental structure; performance should therefore be studied as a property of the strategy–environment pairing, not as an abstract capability. Applied to AI, this suggests that models may perform better than expected in environments with strong scaffolds (formal tests, checklists, diagnostic criteria) even if they perform worse than expected in unconstrained settings. citeturn2search25turn31view0 Resilience engineering and Safety-II (associated with Hollnagel and others) provide a direct vocabulary for your "anti-fragility / distributed robustness" intuition: in complex sociotechnical systems, successful outcomes and failures can share underlying causes (performance variability), and safety is often achieved through adaptive capacity rather than elimination of variability. This framing aligns with workflows where humans continuously adjust to novel edge cases (e.g., diagnosing atypical presentations; debugging unexpected failures), and suggests that robustness comes from how systems absorb and respond to surprises. citeturn4search39turn4search31turn4search3 Communities of practice and situated learning (Lave & Wenger; Wenger) imply that competent work includes tacit norms, shared artifacts, and apprenticeship-like learning processes. For AI-assisted work, this points to mechanisms such as "prompt literacy," team conventions on validating AI output, code review norms that adapt to new failure modes, and institutional safety practices ("technovigilance" language appears in clinical AI discussions). citeturn2search3turn17search1turn31view0 Finally, Taleb's antifragility provides a formal metaphor that is unusually relevant to workflow design: antifragility is defined as a convex response to variability (benefiting more than harmed by volatility), contrasting with fragility as concave sensitivity. Taleb's Nature letter articulates this convex/concave framing explicitly. citeturn17search4turn17search28 While most clinical and software workflows are better described as striving for robustness/resilience than true antifragility, the idea captures a real phenomenon: organizations can redesign practices in response to failures (new tests after a bug, new checklist steps after an incident), thereby turning error exposure into future robustness—if learning loops function properly. citeturn4search31turn31view0turn12view1 Benchmark–practice mismatches that directly bear on robustness The most diagnostic evidence for your hypothesis is not "AI is good/bad," but mismatches where benchmarks predict one thing and deployed workflows yield another (or vice versa). Cases where benchmarks can underpredict real-world utility Coding is a prime example: single-shot benchmark metrics like pass@1 underestimate what happens in a real loop where developers (or agents) can (a) generate alternatives, (b) run tests, and (c) iteratively repair. The Codex paper's gap between single-sample performance and multi-sample-with-tests performance is a miniature version of CI-driven robustness, and agentic SWE systems make that loop explicit at repository scale. Similarly, field RCT evidence that AI coding assistants increase completed work in version-control–tracked workflows demonstrates benefits that are not captured by typical "algorithmic coding problem" benchmarks. Cases where benchmarks can overpredict real-world safety or performance Medical imaging has repeatedly shown that high internal test performance can fail to generalize, and that models can exploit spurious correlates tied to hospital systems or acquisition processes. Zech et al.'s work on pneumonia-screening CNNs is widely cited precisely because it demonstrates this internal–external performance gap and points to confounding and site effects. citeturn4search2 Sepsis prediction models (including widely deployed proprietary systems) provide a different overprediction mode: even with high AUROC, PPV can be low and alert burden high, which in real contexts can reduce clinical usefulness and create workflow harm unless managed. turn3search23 What these mismatches imply They imply that benchmark design choices that "strip out the mess" can sometimes remove helpful structure (tests, escalation, redundancy) and sometimes remove harmful structure (biases, confounding, automation bias triggers). So ecological validity is not only about adding more noise; it is about modeling the real control system in which AI is embedded. turn4search4turn4search5 Implications for AI evaluation and design If the central hypothesis is that extended practice can create robustness, then a research agenda follows that is neither "benchmarks are useless" nor "benchmarks are enough," but rather: evaluate and design AI-in-workflow systems. A practical synthesis consistent with the sources above is a layered approach: • Capability layer: what the model can do with fixed inputs (still necessary, and often what benchmarks measure). • Interaction/workflow layer: how people use, ignore, correct, or defer to the model; how sequencing, defaults, and UI shape reliance and second-opinion behavior. citeturn4search4turn4search5turn25search12 • Systemic impact layer: how institutional constraints, incentives, monitoring, and distribution shift change outcomes over time (including alert fatigue, work redistribution, and the emergence of "technovigilance" practices). turn17search3turn21view0 For software engineering evaluations, the evidence suggests that it is not enough to measure static generation accuracy; evaluations should explicitly incorporate (and separately measure) the robustness contributions of: (a) execution feedback (compile/test), (b) patch validation/selection procedures, and (c) human review gates. The SWE-agent and Agentless-style results make a concrete methodological demand: when performance improves without changing model weights, the "extra performance" should be attributed to workflow/interface design and treated as a first-class object of study. turn0search0 For clinical AI, the mammography screening results show what "ecologically valid" evaluation can look like: prospective, population-based deployment with safety endpoints like interval cancer rates, not only retrospective AUC. At the same time, sepsis model validations show why local calibration and alert design can dominate real-world usefulness. turn4search2turn31view0 Finally, several healthcare-focused conceptual pieces explicitly advocate moving beyond technology-centric evaluation toward frameworks that complement clinicians' epistemic and cognitive activities and that account for the translational gap between lab success and deployment. citeturn25search4turn25search12turn31view0