Why LLMs Are Good at Code

Query Investigate explanations for why LLMs perform well at code generation relative to other knowledge tasks. There are at least two possible explanations: (a) the training data for code is uniquely structured — Stack Overflow carries error messages, corrections, version caveats, upvotes/downvotes, "this worked for me" replies; code repositories carry commit messages, test suites, CI/CD results; (b) the use-time context for code is uniquely rich — compilers catch errors, test suites validate, error messages seed the next attempt, IDE integrations provide immediate feedback. Which explanation dominates in the literature? Do people point to both? Is there evidence that LLMs are good at code even without use-time feedback (i.e., raw benchmark performance), or does performance improve dramatically when the feedback loops are present (i.e., agentic coding with tool access)? The distinction matters: if it's mostly training data, then the jaggedness is baked in; if it's mostly use-time extensions, then the jaggedness is designable. --- Full Report Why LLMs Are Good at Code Framing the two hypotheses Two families of explanations recur in research discussions about why large language models (LLMs) generate code unusually well compared to many "knowledge" or "reasoning" tasks. The first explanation is training-data centric: code corpora contain unusually structured learning signals—formal syntax and conventions, rich local repetition, and abundant paired natural-language-to-code mappings (docstrings, README usage snippets, tutorials). The ecosystem also includes "debugging traces" and social curation at scale: Q&A threads with accepted answers and vote-based ranking, repository histories with commits and reviews, issue trackers with bug reports and fixes, and increasingly explicit metadata about what changed and why. The second explanation is use-time (inference-time) centric: code is one of the few high-value domains where the model's output can be mechanically checked. Compilation, typechecking, unit tests, linters, and runtime error messages create fast and often high-signal feedback. That feedback can drive iterative "edit–run–fix" loops (agents), execution-guided decoding, and selection/filtering among multiple candidates. A key theme in the literature is that these explanations are not independent: real-world software workflows generate artifacts (error messages, patches, test outcomes, PR discussions) that can appear both in training data and at use time; and many systems explicitly "distill" use-time feedback loops back into training (instruction data or reinforcement learning). Why code is unusually learnable from training data A foundational observation in software engineering research is that source code is highly "natural" in the language-modeling sense: it is repetitive and predictable, often even more so than natural language, because developers reuse idioms, templates, APIs, style conventions, and project-specific patterns. Early "naturalness of software" work demonstrated that statistical language models capture strong regularities in real code and can already improve code completion, arguing that real programs written by humans are mostly simple and repetitive in ways that models can learn. Follow-up empirical work investigated why code is more predictable than English, arguing that predictability is not just syntactic restriction; it also reflects deliberate human choices to reduce cognitive load and ensure maintainability, with code corpora measurably more repetitive and predictable than natural-language corpora in multiple settings. Modern code-LLM training pipelines amplify these properties by operating over enormous corpora. The BigCode project's releases describe code datasets on the order of terabytes, spanning hundreds of languages, along with governance and processing practices such as near-deduplication. In their dataset paper, they report that near-deduplicating the data significantly boosts performance across model-training experiments, highlighting how distributional repetition and curation choices materially affect downstream code capability. Critically for your hypothesis (a), "code training data" often includes more than raw source files. For example, the StarCoder training dataset description explicitly lists sizeable components of GitHub Issues, GitHub commits, and notebooks (including text–code pairs)—all of which contain natural-language problem statements, partial or broken code, fixes, and explanations. The StarCoder2/The Stack v2 work similarly describes incorporating issue conversations and processing pull requests (code reviews), along with other "adjacent" software artifacts and documentation sources. turn4search1 This is where GitHub matters: it operationalizes large-scale, public, time-stamped trails of "what changed" (diffs, commits) and "why it changed" (issues, PR discussions), which are exactly the sort of paired signals that compress ambiguity compared to open-ended factual questions. Likewise, Stack Overflow formalizes community curation. Its help pages document that answers are (by default) sorted by votes and that accepted answers receive special placement (appearing first under typical conditions). That structure turns many threads into semi-standardized "problem → best-known fix" exemplars, with additional context such as caveats, version notes, and "worked for me" confirmations appearing in surrounding discussion. turn4search33 Taken together, the training-data story in the literature is not simply "there is a lot of code," but "there is a lot of code plus a lot of naturally occurring supervision about code," and the supervision tends to be unusually well-aligned to the tasks developers ask models to perform. How good are code LLMs without use-time feedback There is strong evidence—starting with early code-LLM work—that models can generate correct code without any runtime tool access or interactive feedback, as measured by one-shot functional-correctness benchmarks (where correctness is checked by hidden tests after the fact). The OpenAI Codex paper ("Evaluating Large Language Models Trained on Code") is a canonical reference because it isolates the effect of code-focused training: it reports that fine-tuning GPT-family models on code yields "non-trivial performance" on HumanEval-style synthesis where base GPT models had near-zero performance at comparable sizes, showing that training on code is a first-order driver of raw capability. On HumanEval specifically, the same paper's Table 1 reports substantial pass@1 performance for their largest Codex model (e.g., 28.81% for Codex-12B), as well as steadily improving pass@k with more samples. Importantly, pass@1 here corresponds to "generate once; test later," i.e., it does not require the model to see test outcomes to get credit. This helps answer one of your core questions: LLMs are demonstrably "good at code" even without use-time feedback loops, at least for self-contained tasks that fit into a textual prompt and can be solved by writing a single function. The improvement relative to non-code-trained baselines is large enough that training data (and inductive bias from code-like structure) cannot be dismissed as merely a minor factor. However, the same Codex paper also foreshadows why feedback loops matter by emphasizing evaluation with many samples and by discussing the utility of test suites in competitive-programming-like settings: large gains appear when you can generate many candidates and select those that satisfy tests, even if the selection itself is performed externally. In practice, this "sample-and-select" paradigm is one of the bridges between raw modeling and feedback-driven systems. How much do feedback loops and tools improve code performance While training data explains why pass@1 is non-trivial for small synthesis tasks, the strongest evidence for hypothesis (b) comes from repository-level and long-horizon software engineering benchmarks, where a correct solution is not a single snippet but a set of coordinated edits that must be validated in an environment. SWE-bench is a widely used example because tasks come from real issues in real repositories and are graded by running tests in a controlled environment. In the original SWE-bench paper's baseline results (Table 5), even strong general models under a non-agentic setup solve only a tiny fraction of tasks (e.g., low single-digit "% Resolved," with some configurations near 0%). The OpenAI SWE-bench Verified release notes reinforce the benchmark design: agents receive an issue description and a repo checkout; tests are not shown; correctness is determined by fail-to-pass and pass-to-pass tests. Against this backdrop, tool-using agents and interactive loops produce pronounced jumps. The Google DeepMind-adjacent "SWE-agent" paper (as published) frames its core contribution as interface design that enables an LM to navigate repos, edit files, and run tests; it reports state-of-the-art results on SWE-bench with pass@1 = 12.5%, explicitly emphasizing that the agent-computer interface (ACI) "significantly enhances" the agent's ability to execute tests and programs. This is a direct empirical example of "designable" improvement that does not require changing model weights. A second line of evidence comes from "self-debugging/self-repair" work on self-contained tasks, which isolates the effect of execution feedback. The "Is self-repair a silver bullet for code generation?" study formalizes self-repair as a pipeline that runs code against tests, collects error messages, optionally converts them to richer textual feedback, and then attempts repairs. It reports that self-repair is not always better than simply sampling more solutions at equal compute budget—especially at small budgets—but it can help, and the benefit increases when feedback quality is higher (including using stronger feedback models or human-written explanations). This matters: feedback loops are powerful, but their value depends on signal quality and system design. A third, more "systems" oriented strand integrates execution feedback even more tightly into inference. "Execution Guided Line-by-Line Code Generation" proposes an inference-time method that incorporates real-time execution signals during generation to guide code line by line, reporting significant performance improvements across coding tasks relative to standard decoding. Finally, there is growing evidence that models can be trained to use feedback more effectively rather than relying on ad hoc scaffolds. "RLEF" proposes reinforcement learning to ground code LLMs in execution feedback for multi-turn code generation, explicitly arguing that state-of-the-art LLMs struggle to improve iteratively compared to independent sampling unless trained for that interaction, and reporting large gains and improved sample efficiency. These results align with how developer tooling works in practice. GitHub's documentation describes continuous integration as running tests and providing results in pull requests, and status checks as externally computed pass/fail signals attached to commits—exactly the kind of readily consumable feedback that can be piped back into an LLM-driven workflow. What the literature suggests about which factor "dominates" Across the literature, neither (a) nor (b) fully dominates; instead, the dominant explanation depends on what you mean by "good at code." If you mean one-shot code synthesis for small, well-scoped problems (HumanEval-like), the literature most strongly supports a training-data-dominant story. The magnitude of the "code fine-tuning" effect—turning near-zero general-model performance into substantial pass@1 and pass@k—demonstrates that structured code data and code-adjacent text are sufficient to create meaningful coding skill without any interactive execution loop at inference. The classic "naturalness" results also explain why language modeling is unusually effective on code compared to many other domains: code is repetitive and patterned, which improves sample efficiency and generalization from next-token objectives. If you mean reliably producing correct changes in real codebases, the literature shifts toward a use-time-feedback-dominant story. SWE-bench-style results show that raw prompting with retrieved context yields very low resolution rates, while agentic systems that can explore the repo, run commands, and execute tests perform dramatically better—sometimes without changing model weights, purely by improving the interaction loop and interfaces. A useful way to reconcile the two is to treat training as providing a strong prior over plausible code, while feedback loops provide a strong verifier (or at least a strong error signal) that turns plausibility into correctness. Coding domains are unusually favorable because the verifier is often: fast, automatable, and tightly coupled to the spec (tests). This is precisely the type of external feedback that surveys of automated correction identify as enabling practical, scalable correction, and it is also the type of feedback that work on self-correction in non-code reasoning says is often missing—hence why intrinsic self-correction tends to fail in many reasoning tasks. Two important caveats appear increasingly often in recent papers and are directly relevant to "training vs use-time" attribution. First, benchmark contamination/memorization can blur the story in favor of training data. The "SWE-Bench Illusion" paper presents evidence that high leaderboard scores on SWE-bench Verified may partially reflect memorization: models can identify buggy file paths from issue descriptions alone at suspiciously high rates on SWE-bench Verified, with substantially lower performance on tasks outside the benchmark's repository set, consistent with overlap between training corpora and benchmark data. This does not eliminate the role of tools, but it weakens naive interpretations like "tools alone explain the leap." Second, the community is actively responding with freshness/contamination-resistant evaluations. SWE-bench-Live explicitly reports that the same agent-plus-model pairs perform significantly worse on fresh tasks than on static SWE-bench Verified, attributing some of the gap to overfitting to static benchmarks and emphasizing continuously updated evaluations. This again makes "dominance" contextual: training overlap can inflate apparent raw and agentic ability on static benchmarks, while feedback loops may matter even more on truly novel tasks. Put differently: for capabilities, training data is the primary source; for dependable real-world performance, feedback loops and tool access increasingly determine outcomes; and the boundary is porous because the field repeatedly converts tool-driven traces into new training regimes (instruction datasets, RL from execution feedback). Implications for "jaggedness" and design Your "jaggedness" framing maps well onto what the research implies, but the conclusion is mixed. If jaggedness means "the model is sometimes brilliant, sometimes brittle," the training-data story suggests some jaggedness is baked in: code corpora are uneven across languages, libraries, versions, and domains; public repositories overrepresent certain ecosystems; and duplication plus benchmark overlap can create sharp cliffs between "seen" and "unseen." BigCode's findings that near-dedup materially affects benchmark performance underscores that "what's in the corpus" and "how it's curated" is a first-order determinant of behavior. But if jaggedness means "we can engineer systems to be robust," the use-time story suggests a large portion is designable. The availability of high-quality external feedback (unit tests, compilers, sandboxes) is exactly what many non-code domains lack; and agent/ACI work shows that even with fixed weights, the right interaction loop can unlock large gains on complex tasks. Self-repair and execution-guided decoding further show that "loop quality" is not a binary: the benefit depends on the fidelity of tests, the informativeness of error signals, and cost-aware allocation between generating diverse candidates vs iteratively refining one candidate. A more precise synthesis emerging from recent work is that coding is a "sweet spot" because it supports a virtuous cycle between training-time and use-time feedback: • Use-time verifiers (tests/execution) enable agentic improvement now. • The traces from those improvements can be compiled into datasets like multi-turn code-feedback corpora, which then bake the interaction pattern into weights. • Reinforcement learning approaches explicitly train models to incorporate execution feedback across turns, narrowing the gap between "tool-augmented" and "tool-agnostic" performance. So, the literature most strongly supports a two-factor model: training data largely explains why LLMs are competent at code in a single shot, while use-time feedback loops largely explain why integrated systems can become reliable and scalable on realistic software tasks—subject to important confounds from data overlap and benchmark design.