Routine, Manageable AI Code Failures

Query This was a follow-up deep dive within the practitioner accounts session, focusing specifically on software engineering accounts where AI failures are described as routine and manageable rather than catastrophic. The prompt asked: what specific workflow habits, team practices, and cultural norms make AI code failures bounded and recoverable? --- Full Report Routine, Manageable AI Code Failures in Software Engineering First-Person Accounts Executive summary Across first‑person accounts by developers, AI‑generated code failures are often described as routine—and manageable—when the surrounding engineering system provides fast, reliable “reality checks”: compilation/runtime errors, tests (especially when written first), linters/typecheckers, CI gates, PR review, and staged rollout patterns. The shared mental model is not “AI is correct,” but “AI is productive under supervision,” with verification practices doing the heavy lifting in bounding risk and reducing surprise. In these accounts, “manageability” is repeatedly tied to workflow habits (test-first loops, small/atomic changes, reviewable diffs, CI feedback ingestion) and socio‑technical norms (explicit accountability, review culture, and treating AI output like any other teammate’s code—never as an authority). Scope and method This report focuses on first‑person accounts by practitioners in software engineering on practitioner‑heavy platforms:,,, and the blog, plus personal blogs and posts (including ). Selection criteria for inclusion: • The author writes in first person (or first‑person plural “we”) about using AI for code or tests. • The quoted line explicitly frames failures as expected/normal and describes a control loop (e.g., tests, CI, review, staging/darkship). • One quote per source, ≤25 words verbatim. Quote catalog with classification Each quote below is verbatim (≤25 words) and followed by: task type, tool used, feedback‑loop details (beyond just “run tests”), and whether manageability is explicitly linked to surrounding practices. | Verbatim quote (≤25 words) | Source (author, date, URL) | Task type | Tool used | Feedback-loop details (verification + correction) | Manageability linked to surrounding practices? | |---|---|---|---|---|---| | “The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you’ll get an error.” |, Mar 2, 2025, https://simonwillison.net/2025/Mar/2/hallucinations-in-code/ | Debugging hallucinated APIs/libraries | General LLM coding assistant (not product-specific) | Immediate runtime/compile feedback; feed the error back to the model; “run it” as fact‑check mechanism. | Yes: manageability is explicitly grounded in executable feedback (“run code → error → fix/iterate”). | | “our CI will run tests and report failures. I can feed those failure logs back to the AI” |, Dec 19, 2025, https://medium.com/@addyosmani/my-llm-coding-workflow-going-into-2026-52fe1681325e | Implementing changes via AI-created PRs | AI coding agents (mentions Jules / Copilot Agent); CI/CD tooling | CI gates on commits/PRs; lint/style (ESLint/Prettier), type checking; staging deployments; “paste linter/test output back in chat”; review (human + bots). | Yes: explicitly attributes “AI-friendly” reliability to robust automation + review culture. | | “We treated code generated by the agent just like code written by our team: it went through automated testing, a human review process” | &, Aug 28, 2025 (upd. Aug 29, 2025), https://github.blog/ai-and-ml/github-copilot/how-we-accelerated-secret-protection-engineering-with-copilot/ | Scaling a repeatable engineering workflow with an agent producing PRs | coding agent in a PR workflow | Automated tests + human PR review; request changes via PR comments; “darkship” stage to observe results before full ship (a staged-release control loop). | Yes: explicitly frames success as grafting the agent into an existing test/review/release framework, incl. staged observation. | | “If it fails, great that's a feedback loop” |, May 5, 2025, https://dev.to/shayy/ai-isnt-my-pair-programmer-its-my-intern-and-it-has-to-pass-my-tests-21ac | Test-first implementation (behavior defined by tests; AI writes code to satisfy) | Cursor + Gemini (as described by author); TypeScript/Node + Vitest | TDD-style loop: write test suite → prompt AI to pass → iterate until green; failure is treated as useful signal; maintainability foregrounded. | Yes: manageability is explicitly the point—guardrails that make “fast” also “safe to build on top of.” | | “If the AI generates a broken REST endpoint, your tests catch it immediately.” |, Mar 2 (year not shown on DEV; likely 2026, approximate), https://dev.to/leena_malhotra/a-simple-framework-for-trusting-ai-without-regret-4boa | Trust calibration: which tasks can be delegated safely vs need verification | General AI coding assistant framing | “Trust gradient”: mistakes are “cheap and obvious” when tests/review/linter catch quickly; emphasizes verification & reversibility as trust criteria. | Yes: manageability is explicitly linked to verifiability and bounded downside. | | “Large changes make code reviews and diffs a nightmare.” |, Jan 20 (year not shown on DEV; likely 2026, approximate), https://dev.to/mcsee/ai-coding-tip-003-force-read-only-planning-1d1m | Reducing AI-induced collateral edits during debugging/refactoring | “AI assistant/copilot” (general); plan/ask mode patterns | “Read-only planning” before edits; stepwise execution; create failing tests; keep diffs reviewable; atomic commits explicitly recommended. | Yes: manageability is framed as controlling blast radius via reviewable diffs and staged, test-anchored edits. | | “Test like a skeptic. Assume AI-generated code is flawed.” |, Feb 18, 2025, https://mikebifulco.com/newsletter/ai-is-your-intern-not-your-boss | Shipping code with AI assistance | LLM tools (author cites “ChatGPT, Copilot, Cursor”) | Iterative debugging + steering like onboarding; tests as pre-ship contract; explicit failure-case testing; “take over” when needed. | Yes: failure manageability is explicitly tied to test discipline + human ownership/oversight. | | “You always have to assume the results are incorrect until you can verify it by hand.” |, Dec 31, 2024, https://stackoverflow.blog/2024/12/31/generative-ai-is-not-going-to-build-your-engineering-team-for-you/ | Integrating generated code into real codebases | GenAI coding tools (mentions ChatGPT/Copilot) | Verification posture: step through output line-by-line before commit; test/instrument/fit code to conventions; emphasizes lifecycle “operating and governing” code. | Yes: manageability is explicitly grounded in review + lifecycle practices (tests/instrumentation/maintainability), not generation. | | “one of the dozen or so tests that Copilot had generated didn't actually test what it purported to.” |, ~Oct 2025 (HN shows “5 months ago,” approximate), https://news.ycombinator.com/item?id=45588283 | Reviewing AI-authored PR (incl. tests) | Copilot Agents + “Sonnet 4.5” (as stated by user) | PR-based verification: human reviews PR after the agent runs; test quality checked by human; spec-guided work (“Markdown spec”) bounds scope. | Yes: ties manageability to “spec → PR → review → fix,” treating imperfect tests as normal, fixable defects. | | “I expect most tests that Copilot writes to fail, but I would hope some would fail in interesting ways.” |, Jul 12, 2021, https://news.ycombinator.com/item?id=27809059 | Using AI to draft tests / probe an API | Copilot (as discussed) | Treats failure as signal: failing tests can reveal edge cases (e.g., off‑by‑one) and drive clearer conventions/docs; implies “run tests → learn → adjust.” | Yes: frames failure as productive feedback (a diagnostic instrument), not a blocker. | | “I usually write the plaintext test descriptions and let the test be generated by AI.” |, ~May 2025 (Reddit shows “10mo ago,” approximate), https://www.reddit.com/r/react/comments/1kqvgli/mycompanyaskedmetouseaitowrite_unit/ | Generating unit tests while controlling intent | AI (unspecified); test scaffolding driven by human-written descriptions | Human supplies requirements as natural-language “test descriptions”; manual tweak of output; “priming” input file to enforce imports/structure conventions. | Yes: manages failure by constraining the model with explicit specs + conventions; treat output as draft requiring edits. | Analytical synthesis These accounts converge on a consistent workflow logic: AI failures are manageable when correctness is forced through deterministic gates. In Willison’s framing, executable code provides built‑in falsification: hallucinated methods fail as soon as you run the program, enabling an immediate “error‑back‑to‑model” loop. In the same spirit, Shayan turns failure into a designed feature of the workflow by writing tests first and treating failed runs as “great”—a signal to iterate, not a reason to abandon the tool. Leena Malhotra makes the trust criterion explicit: tasks deserve more autonomy when “damage is minimal and obvious,” which in practice means failures surface quickly through tests and review. Developers repeatedly report a handful of workflow habits that reduce surprise and cost: (a) behavior-first specification (tests, specs, or plain‑English test descriptions) that constrain the model’s degrees of freedom; (b) tight inner loops (unit tests, local runs, rapid iteration) that convert hallucinations from latent risk into immediate diffs and logs; (c) small-scope changes (atomic commits, avoiding huge diffs, preferring stepwise plans) that bound blast radius and keep review cognitively tractable. The causality is straightforward: hallucinations and logic slips are inevitable, but the cost is dominated by how late they are discovered and how entangled they become with other edits—so developers shrink the time-to-detection and the size of the change surface. A second set of habits pushes the same principle outwards to team and release systems: CI gating, linters/typecheckers, and staging. Osmani describes an “AI-friendly workflow” explicitly as one with robust CI where tests run on every commit/PR, style checks are enforced (ESLint/Prettier), staging deployments exist for branches, and failures become structured artifacts (logs, linter output) that can be pasted back into the model for rapid repair. GitHub’s internal write‑up makes the sociotechnical point vivid: they treated AI-agent output as normal team code—PRs, automated tests, human review—and added a “darkship” stage to observe behavior before fully shipping, creating a safe space to correct errors discovered in realistic operation. Finally, these workflows work because of socio‑technical practices: accountability, review culture, and incentives that reward correctness and maintainability. Majors argues that generated code must be verified “by hand,” and that the hard work is operating/governing systems over time—implicitly placing responsibility on humans and teams rather than the generator. The HN “brushfoot” account shows the same structure even in a solo context: write a spec, let an agent produce a PR, then review and fix (including faulty tests) as a normal part of the loop. In other words: “manageability” is not just a set of technical checks—it is a set of norms about what is acceptable to merge, how work is scoped, and who owns the outcome. Habits, mitigated failure modes, and recommended tool affordances | Developer habit reported in accounts | AI failure modes it mitigates | Recommended AI tool affordances that support the habit | |---|---|---| | Run code early and treat runtime/compile errors as the first truth test | Hallucinated methods, missing imports, wrong API calls that fail fast | One-click “run in sandbox”; capture stack traces and attach them automatically to the next prompt; auto-minimize repro cases | | Tests-first / TDD-style loop (“write tests → ask AI to pass → iterate”) | Subtle behavioral bugs; edge-case misses; plausible but incorrect logic | Test-aware prompting templates (“write code that passes these tests”); automatic failing-test generation from user stories; guardrail that blocks “done” until tests green | | Plain-language test descriptions authored by humans (AI generates test bodies) | Tests that mirror implementation without asserting requirements; incomplete coverage due to missing intent | UI that separates intent layer (descriptions) from implementation layer (generated test code); coverage prompts (“what edge cases are missing?”) integrated with diffs | | CI gating on every commit/PR with unit + integration tests | Late discovery of breakage; “works on my machine”; regressions across modules | Native CI integration: surface failing jobs directly inside the assistant; auto-link to failing lines; “retry with fix” buttons; PR status awareness | | Linters and typecheckers as non-negotiable quality gates | Style drift; type mismatches; inconsistent conventions that increase review/debug burden | Parse linter/type errors into structured tasks; enforce project style via repo-level instruction files; show “diff against conventions” summaries | | Keep changes small and reviewable (atomic commits / avoid huge diffs) | Hidden collateral edits; “AI went off track” rewrites; review fatigue → missed bugs | Assistant modes that default to small patch proposals; “max diff size” setting; automatic commit splitting; stepwise plan execution | | “Plan/read-only” phase before edits; approve steps explicitly | Impulsive multi-file rewrites; architecture misunderstandings; unnecessary changes | Hard “plan mode” that cannot write files; explicit step approval; diff previews with rationale and file-level impact scoring | | Treat AI output like teammate code: PR workflow + human review + comment-driven iteration | Over-trust; unreviewed “looks right” code; low maintainability and incoherent abstractions | First-class PR integration: respond to review comments; generate “what changed & why” notes; highlight generated sections; require a human owner/approver | | Staging / shadow deployment (“darkship”) before full release | Integration mismatches; real-world behavior diverging from tests; doc/API drift | One-click staging deploy from PR; shadow-mode toggles; automatic diff-to-metric dashboards to validate behavior before fully “shipping” | | Feed structured tool output back to the model (CI logs, lints, failing tests) | Slow debugging; model repeating the same wrong fix; “whack-a-mole” without grounded signals | Automatic log extraction + summarization; attach only relevant failure context; “repair” loops that keep a stable plan and track attempted fixes | | Repo-specific priming files / explicit rules for imports/structure/style | Inconsistent patterns; wrong abstractions; friction integrating generated code | Repository “instruction pack” and rule linting; style-guide ingestion; enforcement and drift detection (“this violates rule X”) | End-to-end AI-assisted code change lifecycle with verification and rollback mermaid flowchart TD A[Ticket or task selection] --> B[Define acceptance criteria: spec + tests] B --> C[Plan phase: AI proposes steps] C --> D[Human approves plan + scope] D --> E[AI generates code on a branch] E --> F[Local checks: typecheck + lint + unit tests] F --> G[Small commits / small PR] G --> H[Open PR] H --> I[CI gates: unit + integration + lint + type + build] I --> J[Review: human + bots] J --> K{All gates pass?} K -- No --> L[Send failing logs + review feedback to AI] L --> E K -- Yes --> M[Staging deploy / darkship / canary] M --> N[Observe: logs + metrics + alerts] N --> O{Issue detected?} O -- Yes --> P[Rollback: feature flag off / revert / redeploy] P --> Q[Postmortem: add tests + update prompts/rules] Q --> B O -- No --> R[Merge / release] R --> S[Post-deploy monitoring window] S --> O Implications for AI tool design The strongest design implication from these accounts is that “manageability” emerges when tools tighten the loop between generation and verification: tests and CI results are not external artifacts; they become direct inputs to the assistant’s next action (Osmani’s “feed failure logs back,” Willison’s “feed the error back”). AI coding tools that treat verification as optional (or that bury it behind context switches) will systematically amplify the cost of routine failures; tools that treat verification as the default UI workflow will make those failures predictable and cheap. A second implication is that teams need tooling that forces smallness and reviewability: large diffs and multi-file thrash are repeatedly described as what makes AI failures feel like “nightmares,” because they defeat code review and hide unintended edits. Product affordances like hard plan/read-only modes, automatic atomic commit splitting, and “diff budgets” align directly with the practices developers already adopt manually to keep AI code safe to merge. Finally, these sources suggest that reliable deployment of AI coding in real teams is as much a cultural and governance problem as a model-quality problem: treating AI output “like any other team code” (tests + review + staged observation) and assigning explicit accountability for correctness both reduce surprise and prevent over-trust. Tools that encode these norms—by requiring an owner, surfacing provenance, enforcing gates, and making rollback/staging frictionless—will better match how practitioners already make AI failures routine, bounded, and correctable.