Language Model Basics and the Context Window

Preface

Follow the whole path from text entering a language model to an agent taking action in an environment.

What is a language model?

A language model is a computer program that has learned patterns from a large collection of text. You give it some text—such as a question, an instruction, or a conversation—and it produces more text in response, one small piece at a time. Because it has learned patterns from many kinds of writing, that response might take the form of an answer, a summary, a plan, a translation, a conversation, or computer code.

At its simplest, a language model is text in, text out. It does not by itself open files, search the web, remember every past conversation, or take actions in other software. Those abilities come from the systems built around the model. This module will explain both the language model and those surrounding systems, so you can tell which part is doing what.

When you talk to Claude, ChatGPT, Codex, or another agent, the experience can feel like one continuous intelligence. You type a request. It answers, searches, reads a file, runs a command, or changes something on your behalf.

Underneath that unified experience is a stack of different systems:

environment ↔ tools ↔ harness and scaffold ↔ language model

The distinctions matter. A language model does not directly see your files. It does not execute a web search merely by imagining one. It does not carry a perfect memory of every previous interaction. Surrounding software selects what the model sees, gives it descriptions of possible actions, interprets its requests, acts in the outside world, and returns observations.

This module explains each layer and then puts them back together. By the end, you should be able to trace both information and action through an agent:

outside information
    → retrieval
    → context tokens
    → model output
    → tool request
    → external action or observation
    → new context

That map is the foundation for using agents deliberately. Once you can see the parts, questions such as “Why did it forget?”, “How did it know that?”, “Did it actually inspect the file?”, and “What makes this an agent?” become questions you can reason about rather than mysteries.

Guided tour

A language model is trained, not written rule by rule

Traditional software is usually described through explicit instructions. A programmer writes rules such as:

when this button is pressed, save the record;
if the password is wrong five times, lock the account;
sort these rows by date;
send this request to that server.

The program may be enormous, but its behavior is substantially determined by instructions people deliberately wrote.

A language model is produced differently. Engineers design a neural-network architecture, prepare data, define a training objective, and build the machinery that performs training. The detailed behavior of the resulting model is then learned from examples by adjusting a very large collection of numbers called parameters or weights.

The initial, large-scale stage of this process is called pretraining. Pretraining gives the model its broad ability to work with language, knowledge, and patterns in text. Later post-training shapes that general model toward behaviors such as following instructions and acting as an assistant.

A useful metaphor is that a model is grown rather than assembled. A builder chooses where each beam in a house goes. A gardener shapes the conditions in which a tree develops but does not specify every branch. In the same way, engineers specify the architecture, data pipeline, objective, and training process, but they do not write a separate rule for every explanation, program, translation, or joke the model may later produce.

The metaphor has limits. A language model is not alive, and its training is not mysterious biology. It is a mathematical computation performed by engineered systems. “Grown” is useful only for remembering that much of its internal structure and detailed behavior emerges through training rather than being enumerated by hand.

This is also why the people who build models continue to reverse-engineer what trained models have learned. Anthropic’s accessible Mapping the Mind of a Large Language Model shows researchers finding and manipulating learned internal features after training. These deeper interpretability reports examine real structure inside the network, but they also demonstrate that our understanding remains incomplete.

Pretraining, post-training, and inference are different events

Three activities are easy to blur together:

Pretraining updates a language model to better reflect the patterns in a large collection of text it has been exposed to.
Post-training further shapes the model so it behaves more like a useful assistant—for example, by following instructions and responding helpfully.
Inference is when the trained model is used to respond to a particular request.

Pretraining and post-training shape the model itself. Inference—the activity happening when you use a chatbot or agent—uses the model as it currently exists, together with the information supplied for this interaction.

The distinction matters because placing a fact in a conversation does not normally retrain the model. The model can use that fact in context for the current interaction without permanently adding it to its weights.

Stephen Wolfram’s What Is ChatGPT Doing … and Why Does It Work? builds this account gradually from prediction through neural-network training. Andrej Karpathy’s Deep Dive into LLMs like ChatGPT is a longer general-audience tour of the entire process.

Pretraining: learning by predicting tokens

Modern language models begin with a deceptively simple training objective: predict a missing or next piece of text.

Imagine that the training system presents:

The capital of France is ___

The desired continuation might be Paris. Early in training, the model’s prediction may be poor. The system compares the predicted probabilities with the actual training text and computes a numerical measure of error called loss.

Training then uses backpropagation and an optimization algorithm to adjust the model’s parameters in a direction that should reduce similar errors. This happens again and again across an enormous number of examples.

At a conceptual level, the loop is:

training text
    → predict the next token
    → compare prediction with the actual token
    → calculate loss
    → adjust parameters
    → repeat

Over time, the network becomes better at modeling patterns in language. To predict well across many kinds of text, it must learn a great deal of structure: spelling, syntax, style, factual associations, common reasoning patterns, code, document formats, relationships among concepts, and regularities in how people describe the world.

It is important not to undersell what happens by saying the model is “just autocomplete.” The training objective is next-token prediction, but predicting well across varied text rewards internal representations capable of supporting far more complex behavior than a phone keyboard. Training compresses patterns from the data into the weights of a large neural network.

It is equally important not to leap from that complexity to “therefore it knows that every sentence is true.” The objective rewards predicting text, not checking every claim against reality. A fluent falsehood can be a very plausible continuation.

Tokens: the pieces the model actually processes

A model does not receive a sentence as a row of human-defined words. Before the text reaches the neural network, a tokenizer divides it into pieces and maps each piece to a number called a token ID.

A token might be:

a common word;
part of a longer word;
a space plus a word;
punctuation;
a fragment of code;
one or more characters from another writing system; or
a special marker used by the model or application.

The exact divisions depend on the tokenizer. For example, a common English word may occupy one token while a rare name is divided into several. Capitalizing a word may change its tokens. Equivalent sentences in different languages may use different numbers of tokens.

This has practical consequences:

context windows and API usage are measured in tokens, not words;
some languages or data formats consume more of the window than others;
unusual spelling and character-counting can be awkward for a model;
code, whitespace, and punctuation have token patterns of their own; and
streamed output sometimes visibly arrives in fragments.

Simon Willison’s Understanding GPT Tokenizers is a practical guide to these patterns. You can make the divisions tangible with OpenAI’s interactive Tokenizer: compare an ordinary English sentence, a rare name, a line of code, and the same thought in another language. The colored fragments are a concrete specimen, not a universal map; different model families may tokenize the same text differently.

When generating a response, the model produces one token and then uses the updated sequence to produce the next. That repeated process is why responses appear to stream across the screen rather than arriving as a finished document. Mike Veerman’s interactive TokenSpeed lets you feel what different token-generation speeds are like for prose, code, and agent activity.

From a base model to an assistant

Pretraining produces a base model: a general predictor of text. If prompted with the beginning of a news article, a base model may continue the article. If prompted with dialogue, it may continue the dialogue. It is not automatically a cooperative assistant that reliably interprets every input as an instruction.

Labs therefore perform additional post-training. Exact contemporary recipes vary, but a useful introductory map has three stages:

Pretraining: learn broad patterns by predicting tokens in a large corpus.
Instruction or supervised fine-tuning: train on demonstrations of desirable responses to instructions.
Preference optimization: use human or model feedback to make preferred responses more likely.

The 2022 InstructGPT paper is a primary source for one influential supervised-fine-tuning and reinforcement-learning-from-human- feedback pipeline. Anthropic’s A General Language Assistant as a Laboratory for Alignment records the early “helpful, honest, and harmless” assistant target, while Training a Helpful and Harmless Assistant with RLHF describes an early implementation.

Karpathy’s State of GPT and accompanying slides give the clearest visual overview of the path from base model to assistant.

Post-training changes behavior, but it does not replace the model with an ordinary rule-based program. Nor is post-training the whole story. The product also supplies instructions at inference time.

The “assistant” you encounter is therefore shaped by at least three things:

pretrained model capabilities
    + post-trained behavioral tendencies
    + instructions and context supplied for this run

An optional interpretive frame is to think of a base model as capable of continuing many kinds of text, with “helpful assistant” being one strongly trained and prompted mode. The essays Simulators and the void develop versions of this idea. They are illuminating frames, not settled definitions of what models ultimately are.

The context window: the model’s immediate world

The context window is the finite sequence of tokens available to the model as it predicts the next token.

It is the model’s immediate working material. Depending on the product and the moment in an agent loop, it may contain:

system, developer, project, or user instructions;
the current user message;
earlier user and assistant messages;
examples of desired behavior;
descriptions of available tools;
selected file contents or retrieved passages;
results returned by tools;
environment information;
images or other supported inputs;
summaries of older conversation;
remembered facts retrieved from external storage; and
results produced by other agents.

The context window is not:

every file the application can access;
the entire internet;
the full contents of every connected service;
all of the model’s training data;
every conversation you have ever had;
the complete log of a long-running agent session; or
perfect permanent memory.

This gives us three kinds of information that should not be confused:

Where information lives	What that means
Model weights	Patterns learned during training and encoded in parameters
Current context	Tokens the model can directly use for this generation
External environment or memory	Information that exists elsewhere and must be retrieved before the model can use it

A fact can exist on your computer without being in the context. A harness may have permission to read the file without having read it yet. Even after reading it, the harness might insert the full file, an excerpt, or a summary.

The relationship is:

everything that exists
    └─ what the harness can access
       └─ what the harness retrieves
          └─ what the harness inserts into this request
             └─ what the model can use now

Access, retrieval, selection, and insertion are separate steps.

A context window is a budget, not a memory vault

Context-window size is usually expressed as a maximum number of tokens. Input and generated output must fit within the system’s applicable limits. A larger window allows more material to be considered in one interaction, but capacity is not the same as reliable use.

Research has repeatedly found that models do not use every part of a long context equally well. The Lost in the Middle paper found strong position effects in long-context information retrieval. Context Rot tests degradation as input length grows, even when the underlying task remains simple. Drew Breunig’s How Long Contexts Fail provides a memorable vocabulary:

poisoning: an error enters context and contaminates later work;
distraction: accumulated history pulls attention away from the current problem;
confusion: irrelevant material or excessive tools make the right choice harder; and
clash: different parts of the context contradict one another.

More context can help, but it is not automatically better. A short, relevant context can outperform an enormous context full of stale plans, repeated tool output, and unrelated files.

Anthropic’s Effective Context Engineering for AI Agents describes context as a finite resource and argues for selecting a small set of high-signal tokens. Deciding what to write, retrieve, compress, or keep separate is the practical work of context engineering. The foundational idea is: the model can use only what reaches the window, and everything that reaches the window can influence the result.

The model does not assemble its own first context

Something has to take your message and construct the request sent to the model. That surrounding software is part of the harness.

At the start of a turn, a harness may assemble:

product-level system instructions;
developer or application instructions;
workspace or project instructions;
environment metadata such as the date, platform, or working directory;
tool definitions and input schemas;
relevant memory or retrieved background;
the user-and-assistant message history;
the newest user message;
recent tool calls and their results; and
summaries or compaction of older material.

Not every product uses these names or includes every category. Providers differ in how they serialize, order, cache, label, omit, or dynamically retrieve context. Treat this as a conceptual anatomy, not a universal wire format.

A provider-neutral sketch might look like:

instructions:
  - product behavior and safety rules
  - application or developer instructions
  - project-specific guidance

environment:
  date: ...
  working_directory: ...
  platform: ...

tools:
  - name: search_files
    description: Search filenames and file contents
    input_schema: ...
  - name: read_file
    description: Read a file from the workspace
    input_schema: ...

messages:
  - user: earlier request
  - assistant: earlier response or tool request
  - tool: earlier observation
  - user: newest request

retrieved_context:
  - relevant file excerpt
  - remembered preference

The actual model does not see the indentation as a magical architecture unless the provider sends it that way. The point is that these categories are converted into an ordered token sequence or model-native representation before generation.

Anthropic’s visualization: watching context grow

Anthropic provides a particularly useful concrete picture in Equipping Agents for the Real World with Agent Skills. The diagram shows how Claude’s context changes as it loads a relevant Skill.

Anthropic’s “Skills and the context window” diagram. The source article explains the sequence and the progressive-disclosure design.

The sequence is:

the window begins with a core system prompt, metadata describing installed Skills, and the user’s message;
Claude determines that a PDF Skill is relevant;
Claude uses a tool to read that Skill’s instructions into context;
it reads an additional supporting file only when needed; and
it continues the task with the newly loaded instructions available.

This is one specific implementation, not a diagram of every possible Claude request. Its importance is the pattern it makes visible:

Information can be available to an agent without occupying the context window. The harness can expose a lightweight pointer first, then let the agent retrieve the full material when it becomes relevant.

This pattern is called progressive disclosure. It prevents every possible instruction or reference file from consuming tokens on every turn.

System prompts are real context

The system prompt is not the model’s weights and it is not a mystical hidden personality. It is instruction material supplied by the application at inference time.

Anthropic publishes versions of the system prompts used in its Claude applications. Reading one is clarifying: it contains concrete product facts, behavioral instructions, tone guidance, and rules for particular situations. Simon Willison’s guided reading of a Claude system prompt highlights how specific and sometimes surprising these documents are.

Other instructions may come from the developer who built an application, from project files such as AGENTS.md or CLAUDE.md, or from a retrieved Skill. When instructions conflict, products apply some form of authority and precedence. OpenAI’s Model Spec documents one explicit chain of command. The exact labels and enforcement mechanisms are provider-specific, but the general problem is universal: a harness must decide which instructions enter the request and how conflicts are resolved.

Tool definitions are also context

Before a model can request a tool, it needs a description of that tool. A tool definition commonly includes:

a name;
a description of what the tool does;
an input schema;
sometimes examples, limitations, or usage guidance.

For example:

{
  "name": "check_calendar",
  "description": "Find events on the user's calendar for a given date.",
  "input_schema": {
    "type": "object",
    "properties": {
      "date": {
        "type": "string",
        "description": "The date to check, such as 2026-07-20"
      }
    },
    "required": ["date"]
  }
}

Those tokens affect the model’s behavior. Ambiguous names, overlapping tools, and bloated descriptions can make tool selection worse. Tool design is partly interface design for the model.

The model normally does not execute the tool. It generates structured output that expresses an intention to use it. The harness parses that output, validates it, decides whether approval is needed, executes or routes the operation, and returns a result.

Model, scaffold, harness, tool, environment, and agent

These terms separate the visible conversation into the parts that make it work. Different people sometimes use “scaffold” and “harness” differently. Here, scaffold means what the model sees, while harness means the software that makes the system run.

Model

The trained neural network that maps the current input sequence to probabilities over what comes next.

The model supplies learned capabilities and generates text or structured tool requests. By itself, one model call has no durable session, filesystem access, or ability to execute an external action.

Scaffold

The behavioral material the model sees: system instructions, prompt structure, tool descriptions, examples, selected context, and the conventions used to represent messages and observations. In short:

The scaffold is what the model sees.

Harness

The runtime software that makes the whole process operate. It:

assembles context;
calls the model;
parses responses;
executes or routes tool requests;
returns observations;
manages permissions;
handles errors;
stores session state;
compacts or retrieves history; and
decides whether the loop should continue.

In short:

The harness is what makes it run.

Some people use “harness” as an umbrella for both the runtime and the material shown to the model. Hugging Face’s agent terminology glossary develops the distinction and offers the compact formula Agent = Model + Harness.

Tool

A named operation the model may request. It can inspect or change something outside the model:

read a file;
search the web;
query a database;
run code;
edit a document;
send an API request; or
create an external side effect.

Tools vary enormously in consequence. Reading a public page and sending a payment are both tool calls, but they should not have the same permissions or review requirements.

Environment

The external world in which tools operate:

files and directories;
operating-system processes;
repositories;
browsers and websites;
databases;
APIs and connected applications;
networks;
credentials and permissions;
sandboxes or remote computers; and
people and organizations affected by actions.

The environment contains far more than the current context. Tools are bridges between the context-bound model and that larger world.

Assistant

A model behavior shaped by post-training and runtime instructions to interact helpfully with a user. An assistant can exist without tools or a multi-step action loop.

Agent

A model operating inside a harness that can repeatedly observe, choose an action, use tools, receive the result, and continue toward a goal.

A memorable definition is:

An agent is a model using tools in a loop.

That is a good compression, provided you remember that the harness implements the tool interface, the loop, the permissions, context management, error handling, and stopping behavior.

Anthropic’s Building Effective Agents distinguishes fixed workflows, where code determines the path, from agents, where the model dynamically directs its own process and tool use. The boundary can be gradual: real systems often mix deterministic workflow steps with model-directed decisions.

One complete tool-use loop

Suppose you ask:

Find the largest Markdown file in this project, read it, and explain its purpose.

Here is what may happen.

sequenceDiagram
    participant U as User
    participant H as Harness
    participant M as Model
    participant T as Filesystem tools
    participant E as Project environment

    U->>H: Request
    H->>M: Instructions + messages + tool definitions
    M->>H: Request to search/list files
    H->>T: Validate and execute tool call
    T->>E: Inspect project files
    E-->>T: Filenames and sizes
    T-->>H: Tool result
    H->>M: Prior context + new observation
    M->>H: Request to read largest file
    H->>T: Validate and execute read
    T->>E: Read file
    E-->>T: File contents
    T-->>H: Tool result
    H->>M: Prior context + file contents
    M-->>H: Explanation
    H-->>U: Final response

Step 1: The harness builds the first model request

It includes relevant instructions, the user’s request, environment facts, and definitions for tools such as search_files and read_file.

Step 2: The model requests an action

Rather than answering from guesswork, the model emits a structured search_files or directory-listing request.

This request is still model output: tokens representing an intended action.

Step 3: The harness mediates the request

The harness checks that:

the tool exists;
the arguments match the schema;
the requested path is allowed;
the action complies with permissions; and
user approval is obtained if necessary.

Step 4: The tool touches the environment

The filesystem operation returns filenames and sizes. The model did not directly inspect the disk. The external tool did.

Step 5: The observation enters context

The harness adds the tool result to the message sequence or otherwise makes it available in the next model call.

The model can now use information it did not possess during the first call.

Step 6: The loop repeats

The model requests the largest file. The harness executes the read. The file contents return as another observation. The model then generates its explanation.

Step 7: The harness stops

The loop may stop because the model produced a final response, because a runtime rule was satisfied, because a step limit was reached, or because the harness encountered a failure or approval boundary.

Thorsten Ball’s How to Build an Agent makes this loop concrete by implementing a small coding agent. Sebastian Raschka’s Components of a Coding Agent shows the additional context, memory, tool, and delegation machinery found in a more mature harness.

The context grows during the loop

At the beginning:

[instructions]
[tool definitions]
[user request]

After the first tool call:

[instructions]
[tool definitions]
[user request]
[assistant tool request: search files]
[tool result: filenames and sizes]

After the second:

[instructions]
[tool definitions]
[user request]
[assistant tool request: search files]
[tool result: filenames and sizes]
[assistant tool request: read file]
[tool result: file contents]

This accumulation is powerful: the model can adapt to observations. It is also why agent contexts fill quickly. Search results, command output, files, plans, errors, and attempted fixes all consume space and can influence later decisions.

Harnesses therefore trim, summarize, cache, retrieve, or isolate context. Compaction replaces some older material with a shorter summary. External memory stores information outside the window for later retrieval. Sub-agents can explore within separate windows and return condensed findings. These mechanisms create continuity, but none is identical to the model remembering everything.

The session is not the context window

A product may retain a durable session log containing far more than the model sees on any one call. The harness chooses a view of that session for the next inference.

This distinction is explicit in Anthropic’s discussion of Managed Agents: a durable event log can live outside Claude’s context, while the harness selects, transforms, trims, or compacts material before passing it back to the model.

Therefore:

session history ≠ current context
memory store ≠ current context
accessible file ≠ current context

Each can affect the next response only after the harness retrieves and presents relevant information.

Why the same model can feel like different agents

Suppose two products use the same underlying model. They can still behave very differently because their harnesses and scaffolds differ.

One may provide:

filesystem tools;
a detailed project instruction file;
automatic retrieval of relevant code;
approval before edits;
persistent session notes;
concise tool results;
automatic tests; and
a clear stopping condition.

Another may provide:

only a chat box;
a generic system prompt;
no project access;
a very long unfiltered conversation;
no verification tools; and
no durable memory.

The model weights may be identical, but the working system is not. A large amount of apparent “model quality” is really the result of context selection, tool design, environment access, and loop design.

This is the meaning of the engine-and-car analogy:

The model is the engine. The harness is the car.

The engine matters enormously, but tires, controls, transmission, visibility, navigation, safety systems, and the road determine what the whole vehicle can actually do.

Four boundaries to remember

If you retain only four distinctions from this module, retain these.

1. Weights are not context

Weights contain patterns learned during training. Context contains the information supplied for this generation. Adding a document to a prompt does not ordinarily retrain the model.

2. Access is not retrieval

A harness may be allowed to access a repository, inbox, or database without having loaded its contents. Information affects the model only after it is retrieved and introduced into the current context.

3. A model is not a harness

The model generates tokens. The harness assembles requests, manages state, executes tools, and runs the loop.

4. A tool request is not the external action

The model emits a structured request. The harness validates and routes it. The tool performs the operation in the environment. A trustworthy system preserves evidence of each boundary rather than presenting the entire chain as magic.

Practice

Trace an interaction yourself

Choose a recent request you gave an agent, or use:

Read a document, identify its main argument, and save a summary in a new Markdown file.

Fill in the trace:

Stage	What happened?
User goal
Initial context assembled by the harness
First model output or tool request
Tool action in the environment
Observation returned to context
Next model step
Final result
Stopping condition
Information that remained outside context

Then ask:

Which claims came from model weights?
Which facts came from retrieved context?
What evidence proves the tool actually ran?
What did the harness decide or enforce?
What could have been accessible but never loaded?
If the interaction continued for 100 more tool calls, what would need to be summarized, stored externally, or discarded?

If you can answer those questions, you can see the system rather than merely the conversational surface.

Proof of understanding

Before looking back at the module, answer these questions in your own words:

What is a language model, and what does it produce when you give it text?
What is the difference between training and inference?
What is a token, and why can token count differ from word count?
What is the context window, and why is it different from long-term memory?
Which parts of an agent interaction does the harness place into context?
What is the difference between a model, a harness, a tool, and an environment?
Trace one tool call from the model’s request to the observation returned to the model.
Why can two products using the same language model behave very differently?

If you can answer each question clearly and complete the interaction trace above, you understand the working model this module is meant to establish.

A map to carry forward

The complete picture is:

flowchart LR
    E["Environment<br/>files, apps, web, databases"] <--> T["Tools<br/>named operations"]
    T <--> H["Harness<br/>runtime, permissions, state, loop"]
    H --> C["Scaffold and context<br/>instructions, messages, tool definitions, retrieved material"]
    C --> M["Language model<br/>next-token prediction"]
    M --> H

The model was trained to predict tokens. Post-training shaped it toward assistant behavior. At inference time, the harness assembles a context that includes instructions, messages, tool descriptions, and selected information. The model generates text or a structured tool request. The harness interprets that output, causes a tool to interact with the environment, and places the result back into context. Repeating that process produces an agent loop.

From here, you can explore:

how to choose the smallest complete context;
how to verify an agent’s claims and actions;
how to control permissions, blast radius, and recovery;
how tools connect to external systems; and
how to assemble these parts into a reliable agent.

Further Practice

Paste the same short passage into the OpenAI Tokenizer in two slightly different forms. Change punctuation, spacing, or one unusual word and notice how the tokenization changes.
Give the same bounded task to two different agent products. Compare what each one can access, which tools it uses, what it asks permission to do, and what evidence it gives you. Separate differences in the model from differences in the harness and environment.
Trace a longer agent session with several tool calls. Mark which information is in the current context, which information is stored outside it, and what the harness would need to summarize or retrieve if the session continued.