How AI coding assistants work

A senior engineer at a mid-sized startup told me their team adopted a coding assistant, got impressive demos, and then watched productivity gains evaporate on anything touching their 200-file monorepo. The inline completions worked. The "fix this bug" agent would confidently write code importing modules that didn't exist, referencing functions named differently than what the model invented. After three months, they'd spent more time reviewing hallucinated rewrites than they'd saved on typing.

This is not a model-quality story. It is an engineering story. The difference between a coding assistant that works on a toy repo and one that works on your actual codebase is almost entirely in three places: how the assistant builds its representation of the repository, how it selects and places context, and what edit format it uses to express changes. Let's go through each one.

The repository map problem

The naive approach to "give the model your codebase" is to concatenate every file and feed it in. On a repository with 300 files averaging 200 lines each, that's 60,000 lines. At roughly 4 characters per token and 50 characters per line, that's ~750,000 tokens — far beyond practical context limits even in 2026, and ruinously expensive. You'd also be sending most of it cold: the model doesn't need the full source of your authentication middleware to add a field to a database model.

The answer is a repository map: a compact, structured summary of the codebase that fits in a few thousand tokens and contains enough signal to route the model toward the right files and symbols.

tree-sitter is the standard parser for this job. It's a language-agnostic incremental parser that produces a concrete syntax tree for Python, TypeScript, Rust, Go, and about 40 other languages. Unlike language-server protocol (LSP) tools, tree-sitter doesn't need a language server running — it parses statically and fast, which matters when you want to build a repo map on every invocation without a 3-second LSP boot.

From the syntax tree, you extract the skeleton of every file: class names, function signatures, method names, constants, and import relationships. The result looks roughly like this for a typical Python file:

src/billing/invoice.py:
  class Invoice:
    def create(customer_id: str, items: list[LineItem]) -> Invoice
    def mark_paid(payment_ref: str) -> None
    def to_dict() -> dict
  def calculate_tax(subtotal: float, region: str) -> float

src/billing/stripe_client.py:
  class StripeClient:
    def charge(amount_cents: int, customer_id: str) -> ChargeResult
    def refund(charge_id: str, amount_cents: int) -> RefundResult

For a 300-file repository, this skeleton representation might be 5,000-15,000 tokens — manageable in a 128k window alongside a conversation and the files you're actively editing.

Ranking which symbols to include

But even the skeleton is too large if your repo is 1,000+ files. Aider — an open-source CLI coding assistant — pioneered an approach that applies PageRank to symbol references to rank which symbols are most relevant to the current task. The idea: build a directed graph where an edge from file A to symbol B means A references B. Apply PageRank to find which symbols are most central to the current edit context (the files you've asked about, the error message you've pasted in). High-PageRank symbols get into the map; low-PageRank ones get dropped.

This is a meaningful quality win. A task that touches your payment processing code should surface the symbols from stripe_client.py and invoice.py, not the symbols from your image resizing utilities that happen to have high absolute usage counts. Context relevance beats context volume.

flowchart TD
    FILES["All repo files\n(e.g. 800 files)"] --> TS["tree-sitter\nparse each file"]
    TS --> SYM["Symbol graph\n(class, function, import edges)"]
    SYM --> PR["PageRank\nseeded by current edit files"]
    PR --> TOP["Top-ranked symbols\n~5,000-15,000 tokens"]
    TOP --> MAP["Repo map\ninjected into context"]
    style TS fill:#a855f7,color:#fff
    style PR fill:#00e5ff,color:#0a0a0f
    style MAP fill:#15803d,color:#fff

The repo map tells the model what exists and how things relate. It doesn't replace reading the actual file content — when the model decides it needs to look at stripe_client.py in full, it reads it. The map is the index; the files are the content.

Context selection and placement

Building a good repo map solves the "what exists" problem. The separate problem is what to put in the actual context window alongside the user's request.

The working context for a coding task typically contains:

• The system prompt (editor identity, tool schemas, formatting instructions)
• The repo map skeleton
• Full content of files being edited or directly referenced
• The conversation history (previous turns, prior errors)
• The user's current request
• Tool results from previous steps in the agent loop

With a 128k token window, a typical agentic coding task might budget:

Token budget estimate (128k window):

System prompt + tool schemas:    2,000-8,000 tokens
Repo map skeleton:               5,000-15,000 tokens
Full content of 3-5 active files: 5,000-25,000 tokens
Conversation history:            1,000-10,000 tokens
User request + agent scratchpad: 500-2,000 tokens
Reserve for model output:        2,000-8,000 tokens

Usable budget for files: ~60,000-90,000 tokens
At 50 chars/line → ~4,800-7,200 lines of code in context
A 500-line file is ~6,000 tokens → fits ~10-15 medium-sized files

That's generous. But the placement of content matters as much as the volume, because of a well-documented effect in long-context models.

The lost-in-the-middle effect

Research from 2023-2024 found that when relevant information is placed in the middle of a long context — rather than at the start or end — model recall accuracy drops significantly. The original paper showed 20-30% accuracy degradation for retrieval tasks where the relevant document was placed mid-context compared to beginning or end positions.

For coding assistants, the practical implication is simple: put the most important content first or last. If the user is editing invoice.py, that file's content should be near the start of the context, not buried after 40,000 tokens of repo map and conversation history. The repo map skeleton (important for orientation but not the target of edits) can live in the middle. The specific file being changed belongs at the boundary.

{"type": "lost-middle", "title": "Where you place context affects recall accuracy"}

This also argues against packing context purely by file order or recency. A thoughtful coding assistant ranks content by relevance to the current edit and places the highest-relevance content at position 0.

Edit formats: the underrated quality lever

Once the model has the right context, it needs to express a change. This is where most implementations quietly fail at scale.

Full-file rewrite is the simplest format: ask the model to output the entire file with modifications applied. Fine for files under 100-200 lines. On a 600-line file, the model must reproduce 550+ unchanged lines accurately while modifying 50. In practice, models drift — they silently delete lines, rename variables, reorder methods, or introduce subtle changes not in the prompt. The longer the file, the more confident-looking the drift becomes.

Unified diff is the format Git uses: @@ -42,7 +42,9 @@ lines followed by context and change lines. Models trained on GitHub data recognize this format well. But it's sensitive to exact line numbers, and if the file has changed since the model last read it (or if the model's token window has a slightly stale version), the diff applies to the wrong location or fails entirely.

Search-replace blocks are what Aider settled on after extensive empirical testing:

<<<<<<< SEARCH
def calculate_tax(subtotal: float, region: str) -> float:
    return subtotal * 0.1
=======
def calculate_tax(subtotal: float, region: str) -> float:
    """Calculate tax; rate varies by region."""
    rates = {"CA": 0.0725, "NY": 0.08, "TX": 0.0625}
    return subtotal * rates.get(region, 0.1)
>>>>>>> REPLACE

The application logic is deterministic: find the exact SEARCH text in the file, replace it with REPLACE. If SEARCH doesn't match exactly, the edit fails loudly rather than silently applying to the wrong location. This format avoids full-file reproduction entirely — only the changed section appears in the model's output. A 600-line file edit might produce 30 lines of output instead of 600.

Claude Code uses a variant of this pattern with its own structured diff format, tuned to minimize output tokens and maximize deterministic application. The specific format is less important than the principle: never make the model reproduce content it isn't changing.

Aider's benchmarks (published on their site, though numbers move as models improve) consistently show that the edit format choice affects task success rate by roughly 10-20 percentage points on SWE-Bench-style tasks — holding the model constant. That's a larger lever than most teams expect.

Long-file strategies

Even with search-replace blocks, a 3,000-line file is a problem. Several strategies help:

Selective file loading. Read the full file once to build an AST, then load only the relevant class or function into context — not the whole file. This requires knowing which function to load before you read it, which the repo map helps with. You load the map, identify calculate_tax as the target, then load only the calculate_tax function body and its immediate context.

Windowed context with anchoring. Load a sliding window around the cursor position or the target function, with explicit line-number anchors to orient the model. The risk is that the model lacks visibility into the broader file structure and makes changes inconsistent with the rest of the file.

Top-down editing with skeletons. Show the model the full file skeleton (function signatures only, no bodies) and let it identify which functions to read. Then load those functions' full bodies for editing. This is close to what IDE-integrated assistants do under the hood with LSP hover-type information.

None of these is perfect. Long files remain genuinely harder for coding agents than short ones, independent of context length. A 3,000-line file with deeply intertwined logic is a harder task for a model than splitting it into three 1,000-line files with clean interfaces between them — which is also, not coincidentally, better software architecture.

Inline completion: the pipeline nobody talks about

Everything above describes agentic mode. But the feature that made Copilot a product — the gray text after your cursor — runs a different pipeline, and its constraints are brutal: the suggestion must appear before you type the next character, which means a total budget of roughly 150-200 ms including network round-trip. No repo map, no PageRank, no tool loop. There isn't time.

The model itself is different. Completion models are trained with a fill-in-the-middle (FIM) objective: instead of only predicting what comes next, training examples are restructured with sentinel tokens so the model sees <prefix> ... <suffix> ... <middle> and learns to generate the missing middle. That's why a completion can close a function correctly — it has read the code below your cursor, not just above it. And the models are small: a 1-10B-parameter model served at low latency beats a frontier model that arrives 800 ms after you've already typed the line yourself.

Context assembly is correspondingly cheap and local. Copilot-style tools grab a window around the cursor (prefix and suffix, a few hundred lines), then pad the prompt with snippets from neighboring tabs and recently edited files, selected by fast similarity heuristics like Jaccard overlap on token windows — no embedding call, no parser. The whole prompt is ~1,000-2,000 tokens.

Two pieces of engineering carry the perceived quality. Debouncing: don't fire a request on every keystroke; wait for a ~75-100 ms pause, and cancel in-flight requests when the user keeps typing. Caching: if the prefix matches a recent request, reuse the completion and trim it as the user types through it. A large fraction of "wow, it's fast" moments are cache hits. The failure mode is the mirror image of agentic mode: no grounding beyond open files, so completions cheerfully invent methods on classes defined elsewhere in the repo. That's tolerable here because rejection costs one keystroke.

The agent loop for coding tasks

For agentic tasks (not just inline completion), the pipeline becomes a full tool-using loop. The agent loop is the same pattern covered in that article, but applied to a coding-specific tool set:

sequenceDiagram
    participant U as User
    participant A as Agent
    participant T as Tools

    U->>A: "Add rate limiting to the /checkout endpoint"
    A->>T: read_file(src/api/checkout.py)
    T-->>A: file contents (1,200 tokens)
    A->>T: search_repo("rate limit", "limiter")
    T-->>A: 3 relevant files in results
    A->>T: read_file(src/middleware/rate_limit.py)
    T-->>A: existing middleware (800 tokens)
    A->>A: reason: apply existing middleware pattern to checkout
    A->>T: edit_file(search_replace block)
    T-->>A: edit applied OK
    A->>T: run_tests("src/api/test_checkout.py")
    T-->>A: 2 tests pass, 1 fails: missing import
    A->>T: edit_file(add import)
    T-->>A: edit applied OK
    A->>T: run_tests("src/api/test_checkout.py")
    T-->>A: 3 tests pass
    A-->>U: Done. Added rate limiting using existing RateLimitMiddleware — patched 2 files.

This loop made 6 tool calls. A more complex task — "implement OAuth login" — might make 20-30. Token costs compound. At 5,000 input tokens per call × 25 calls, you're at 125,000 tokens just for input — plus output tokens, plus any large file reads.

The context window management article covers how agents handle context budget across many turns. The coding-specific challenge is that code files are large and the model often needs to re-read them after edits to verify correctness. A naive implementation re-reads entire files on every turn; a better one tracks which file version is in context and only re-reads when the file has changed.

What breaks

Hallucinated imports and symbol names

The most common failure in production. The model writes from src.billing.payments import process_stripe_payment but your actual function is named charge_card in src.integrations.stripe. Without an accurate repo map, the model invents a plausible-sounding name. The error only surfaces at runtime.

Mitigation: Make the repo map explicit. Before any code-writing turn, ensure the model has the actual module path and exact function signature in context. Search-first patterns — where the agent searches for the relevant symbol before writing code that uses it — catch most of these.

The edit that corrupts the file

Search-replace application fails if the SEARCH block doesn't match exactly. The model may have seen a slightly different version of the file (maybe you edited it between when the agent read it and when it produced the edit). Fail loudly, not silently — a failed edit that the agent doesn't notice leads to it reasoning about a state that doesn't exist.

Mitigation: Re-read the file after every edit you're uncertain about. Some implementations add an automatic verify step: after applying an edit, re-read the relevant section and compare against expectations.

Runaway loops and context rot

An agent that can't solve a problem in 5 steps often can't solve it in 25 steps either — it just accumulates more noise in the context. Each failed attempt and error message adds tokens that compete with the actual useful context. After ~10 iterations on a hard problem, context rot is common: the model starts losing track of what it already tried, repeating attempts, or drifting from the original task.

Mitigation: Hard iteration caps (Anthropic's own guidance suggests 3-10 steps for most coding tasks). Budget the token window explicitly and stop if the agent hasn't converged. Surface a partial result rather than letting it spiral. The agent evaluation and failure modes article covers this pattern in full.

Silent wrong answers

The agent finishes, reports success, all tests pass — and the code is subtly wrong in a way the test suite doesn't catch. A business logic change that looked correct to the model but misread the specification. This is the hardest failure mode and the one benchmarks undercount, because benchmarks rely on tests to define correctness.

Mitigation: Keep humans in the review loop for code that touches business logic or security paths. Coding agents are not a substitute for code review — they're a productivity multiplier for the engineer doing review.

Evaluating code generation quality

What benchmarks actually measure

HumanEval (OpenAI, 2021) contains 164 Python programming problems at the function level. You pass a docstring and function signature; the model completes the function. Tests are run to verify correctness. The metric is pass@k — the probability that at least one of k completions passes all tests. It's a clean benchmark for function completion quality but says almost nothing about multi-file, agent-level coding tasks.

MBPP (Google, 2021) is 500 mostly-beginner Python problems. Similar limitations.

SWE-Bench (Princeton, 2023) changed the game: it's drawn from real GitHub issues across 12 popular Python repositories. The task is to produce a patch that resolves the issue, verified against the repository's test suite. This tests repository-level reasoning, multi-file changes, and understanding of real codebases — not toy problems.

SWE-Bench Verified (OpenAI, 2024) is the 500-problem subset of SWE-Bench that a team of professional software engineers manually verified as unambiguously solvable and correctly specified. This is now the standard for evaluating coding agents. As of mid-2026, top agents solve roughly 70-80% of Verified problems in fully automated mode — up from around 50% in late 2024 — but numbers move fast and vendor claims should be checked against independent replication.

The critical lesson: a model that ranks first on HumanEval does not necessarily rank first on SWE-Bench Verified. Function completion and repository-level bug fixing are different skills. Use the benchmark that matches your actual use case.

flowchart LR
    HE["HumanEval\n164 function problems\npass@k"] -->|measures| FC["Function completion\nsingle file, docstring-to-code"]
    MBPP["MBPP\n500 beginner problems"] -->|measures| FC
    SWE["SWE-Bench Verified\n500 real GitHub issues"] -->|measures| RA["Repository-level\nagent task completion"]
    FC -->|"does NOT predict"| RA

    style SWE fill:#00e5ff,color:#0a0a0f
    style RA fill:#a855f7,color:#fff

Running your own evals

For production coding assistant deployments, the benchmarks above measure models, not your specific setup. What you actually want to know: does this system work on your codebase? That requires building a golden dataset.

The pattern:

1. Collect 20-50 real bugs or feature requests from your issue tracker that have known-correct solutions (merged PRs with green test suites).
2. Reproduce the pre-fix state of the repository for each.
3. Run your coding assistant against each task.
4. Score by: tests pass (binary), patch applies cleanly (binary), diff matches human patch (fuzzy).

This is the same principle as offline evaluation for LLM applications — build a labeled dataset from real production cases, not synthetic ones. 20 real issues from your codebase will tell you more about production performance than any public benchmark.

The agent evaluation trajectories article covers how to evaluate multi-step task completions, which is exactly what you need here: not just the final answer, but whether the agent's tool-call trajectory was efficient and didn't hallucinate intermediate steps.

The decision in practice

Inline completion vs. agentic assistant is not primarily a quality question — it's a cost and risk question.

Inline completion (Copilot Tab, Cursor inline) is fast, cheap (~$0.001 per suggestion), and low-risk: a bad suggestion is just ignored. The context is local (surrounding code) and the task is bounded (complete the next few tokens/lines). This is appropriate for the majority of editing time — typo correction, boilerplate, method bodies where the pattern is clear from the signature.

Agentic mode (Claude Code, Cursor Agent, Copilot Workspace) is slower, costs 100-1,000× more per task, and requires that you review the result carefully. It's appropriate for tasks with genuine multi-file reasoning requirements: refactoring a component, implementing a new feature that touches 4 files, tracking down a bug that spans layers. The cost per task (10-50 cents at mid-2026 frontier prices) is easily worth it for a task that would take a human 30 minutes, but not for a task that would take 2 minutes.

The tools that will dominate production codebases are the ones that route correctly between these modes: use completion for typing, use agentic loops for planning tasks, and don't run a 25-step agent loop to rename a variable.

Understanding the repo-map construction, context placement, and edit format mechanics lets you evaluate tools honestly — and debug them when they fail. The model is rarely the bottleneck. The pipeline is.