Why AI Coding Assistants Hallucinate: The Context Problem

AI coding assistants are getting remarkably good.

Tools such as Cursor, Claude Code, Aider, Codex, and other agent based development environments can now write features, refactor code, debug errors, create tests, and navigate surprisingly large software projects.

And yet almost every developer who uses them regularly has experienced the same frustrating moment.

• 01You ask the AI to modify an existing feature. It confidently calls a function that does not exist.
• 02It creates a second implementation of something your application already has.
• 03It ignores a project convention that you explained earlier.
• 04Or it changes one file correctly while completely missing the dependency two folders away that makes the entire feature work.

At first, this looks like a model intelligence problem.

Often, it is actually a context problem.

The quality of an AI coding assistant depends not only on how capable the underlying model is, but also on what information is available to that model when it makes a decision.

That distinction is becoming increasingly important as AI moves from answering coding questions to actively modifying production software.

What does “context” mean for an AI coding assistant?

When a developer works on an existing application, they bring a large amount of background knowledge to the task.

An AI model does not automatically know any of this. It only knows the information that its coding environment makes available during the task. That information is its context.

The model then has to make a decision based on that information. If the correct implementation depends on something missing from the context, the model has a problem. Sometimes it asks for more information. Sometimes it searches the repository. And sometimes it fills the gap with a plausible assumption.

That last behaviour is what developers often describe as a hallucination.

A simple example

Imagine you have an application with an existing payment service:

class PaymentService {
  async createPaymentIntent(
    customerId: string,
    amount: number
  ): Promise<PaymentIntent> {
    // ...
  }
}

Add a checkout endpoint that creates a payment.

But the assistant never sees PaymentService. It might generate something like:

await stripe.payments.create({
  customer: customerId,
  amount
});

Fig. 01 — Missing abstraction

The code looks reasonable. The API looks plausible. The implementation may even resemble code that exists in other Stripe integrations. But it is completely wrong for your application.

The problem is not necessarily that the model cannot write payment code. The problem is that it did not know your application already had an abstraction for creating payments. Give the same model the relevant service definition and project architecture, and the answer can change dramatically.

A good model with bad context can produce bad code.

Bigger context windows do not automatically solve the problem

A natural reaction is: why not just give the AI the entire repository? Modern language models can process very large context windows, so in theory this sounds like the perfect solution.

In practice, more context is not always better. Researchers studying long context language models have observed a phenomenon commonly referred to as “Lost in the Middle.”

The basic idea is that a model’s ability to use information can vary depending on where that information appears inside a long context. Even when a piece of information technically fits within the model’s context window, that does not guarantee the model will use it effectively.

Context capacity is not the same as context quality.

A model may be capable of receiving an enormous amount of code while still struggling to identify which small section is actually important to the current task. A repository might contain hundreds of source files, generated code, migrations, package metadata, tests, documentation, build output, configuration, old implementations, and third party code.

Sending everything may technically give the model more information. It also gives the model far more noise.

Fig. 02 — Signal in the archive

Context bloat creates real problems

Poor context management affects more than code quality. It can also make AI assisted development slower and more expensive.

Maximize relevant context, not total context.

Context engineering vs. prompt engineering

A lot of discussion around AI development focuses on prompt engineering. Prompts certainly matter.

“Refactor this function” is less useful than “Refactor this function to remove the duplicated validation logic while preserving the existing public API. Run the related tests after the change.” The second prompt provides clearer intent and constraints.

But prompt engineering only addresses one part of the problem. In real software projects, the model may also need to discover the relevant interfaces, existing implementations, test expectations, architectural rules, configuration, dependencies, and documentation. You should not have to manually paste all of that information into every prompt.

That is where context engineering becomes more important. Context engineering is about designing the environment around the model so that it can retrieve the information it needs when it needs it.

Instead of asking “How do I write the perfect prompt?”, a more useful question is “How will the agent discover the correct information before changing this code?”

Good context is usually layered

One useful model is to think about context in layers.

Fig. 03 — Layered context

1. Layer 1 — Permanent project rulesArchitectural boundaries, package manager, test commands, files that should not be modified, security requirements, and core conventions. These might live in AGENTS.md, CLAUDE.md, or tool specific configuration. They should usually be concise.
2. Layer 2 — Repository structureDirectory trees, symbol maps, semantic indexes, filename searches, language parsers. The purpose is not to load every implementation. It is to help the agent locate the right implementation.
3. Layer 3 — Task specific codeOnce the agent understands the task, it should inspect the files directly involved: controller, payment service, request schema, database model, related tests. These files deserve much more attention than unrelated parts of the repository.
4. Layer 4 — FeedbackCompiler errors, lint errors, test failures, browser results, runtime logs. This layer reveals incorrect assumptions and guides the next iteration. AI coding works much better when the model does not have to be correct on its first attempt.

Instead, it should be able to make a change, verify it, inspect failures, and correct itself.

The difference between code generation and software engineering

Generating a function in isolation is relatively easy. Changing an established application safely is much harder. A production codebase contains relationships.

A seemingly small feature can depend on database schemas, business rules, authentication, authorization, background jobs, third party integrations, frontend behaviour, caching, infrastructure, and tests.

A developer does not simply ask “Can I write code that solves this problem?” They also ask where the code belongs, what already exists, what the change will affect, and how they will know they have not broken something else.

AI coding agents need access to the same kinds of information. Without that context, they can produce code that is locally convincing but architecturally wrong.

What developers can do today

You do not need a complicated AI infrastructure to improve this. A few habits make a significant difference.

1. Give agents clear project level instructions instead of repeatedly explaining basic architecture in individual prompts.
2. Point the agent toward the relevant part of the codebase when you know where the work belongs.
3. Avoid loading unrelated files simply because you can.
4. Keep documentation close to the code and easy for agents to discover.
5. Most importantly, give the agent a reliable verification loop.

Fig. 04 — Verification loop

That difference becomes increasingly important as agents receive permission to make larger and more autonomous changes.

The key principle: better context beats more context

The future of AI assisted development will not be determined only by which company produces the model with the largest context window. The tooling around those models matters just as much.

It needs to know:

• what information matters
• where to find it
• which constraints must be followed
• how to verify the result

The objective is not to put your entire codebase into the model’s memory. The objective is to make the right information available at the right moment.

That is context engineering. And as AI agents become increasingly involved in real software development, context engineering is becoming part of software engineering itself.