How LLMs behave: tokens, context, sampling, non-determinism

Learning objectives

• Explain what a token is and why cost and fit are measured in tokens, not words.
• Describe the context window as a fixed per-request budget, and name its two edge behaviors.
• Explain how sampling and temperature make the same prompt return different wording.
• Write tests that assert on properties instead of exact text, and know when an eval with a model-graded judge is needed.

Estimated teaching time: 12 min · Prerequisites: None. This is the first teaching lesson of the course.

Why these four ideas come first

Picture a support assistant for Harbor Goods, a fictional online store we will use throughout the course. In testing it answers every question well. After launch, three things surprise the team: the bill is higher than expected, long conversations start failing, and the same question gets two differently worded answers, which breaks a test that compared exact text.

None of these are bugs in Claude. They are direct consequences of four ideas: tokens, the context window, sampling, and non-determinism. Once you understand them, all three surprises become predictable.

1. Tokens: the unit of input, output, and cost

Token

In plain English: A chunk of text, often a whole common word, part of a longer word, a number fragment, or a punctuation mark. Claude reads and writes tokens, not characters or words.

Why it matters: Models work on a fixed vocabulary of text pieces. Splitting text into tokens is how any input, in any language or format, becomes something the model can process.

Analogy: Like Lego bricks: a familiar shape may be one brick, an unusual one needs several small bricks.

The component that cuts text into tokens is called a tokenizer, and each model generation can use a different one. So the average number of characters per token is not a constant. It depends on the model you call. Treat any characters-per-token rule of thumb as model-dependent, and confirm current tokenizer behavior when you build.

Everything the model processes is counted in tokens: your system prompt and user prompt, the full conversation history, the definitions of any tools, the results those tools return, and the response the model generates. Tokens are the unit of both pricing and budget. When you estimate what a feature costs, or check whether an input fits, you are counting tokens, not words.

Worked example: the Harbor Goods assistant sends a 900-token system prompt, 15 tool definitions, and the whole chat history on every turn. A ten-turn conversation therefore re-sends that system prompt and those tool definitions ten times. That is surprise number one: the bill grows with everything you include on every call, not only with the customer's short question.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

request = {
    "model": "claude-sonnet-5-5",
    "system": "You are the Harbor Goods support assistant. Be concise.",
    "messages": [{"role": "user", "content": "Where is my order #A1042?"}],
}

estimate = client.messages.count_tokens(**request)  # free, but rate limited
print("estimated input tokens:", estimate.input_tokens)

response = client.messages.create(max_tokens=300, **request)
print("input tokens: ", response.usage.input_tokens)
print("output tokens:", response.usage.output_tokens)

estimated input tokens: 27
input tokens:  27
output tokens: 64

• Common mistake: estimating cost or fit from word counts or character counts.
• Common mistake: reusing token counts measured on one model for a different model.
• How to apply: count with the token counting endpoint for the model you will call, and log the usage field of real responses.

2. The context window: a fixed budget

Context window

In plain English: The total number of tokens the model can take in for a single request, including the output it writes.

Why it matters: The model can only work with what is in front of it for this request. The window is its working memory, separate from the data it was trained on.

Analogy: A desk of fixed size. Every document you work with, and the page you are writing the answer on, must fit on the desk at the same time.

The window holds everything at once: the system prompt, the full conversation so far, any documents you inject, every tool result, and the model's output. Because it is a fixed budget, it has two distinct edge behaviors, and they look very different in your code.

Why it only failed after launch: in development, test conversations are short, so the window rarely fills. In production, customers paste long order histories, sessions run for many turns, and tool results pile up. Each turn adds to the total until a request reaches the ceiling. That is surprise number two. Module 2 explores this failure, and how to handle it, in detail.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"


class TruncatedReply(Exception):
    """The model hit the context window ceiling while generating."""


def reply(history: list[dict]) -> str:
    try:
        response = client.messages.create(
            model=MODEL, max_tokens=1024, messages=history
        )
    except anthropic.BadRequestError as err:
        # Edge 1: the input alone exceeds the window ("prompt is too long").
        # Nothing was generated. Other 400s land here too, so check the message.
        if "prompt is too long" in str(err) and len(history) > 1:
            return reply(trim_history(history))
        raise

    if response.stop_reason == "model_context_window_exceeded":
        # Edge 2: the input fit, but generation hit the ceiling.
        # The text is partial: never show it as a complete answer.
        raise TruncatedReply(response)

    return "".join(b.text for b in response.content if b.type == "text")


def trim_history(history: list[dict], budget: int = 150_000) -> list[dict]:
    """Drop the oldest user/assistant pair until the input fits our budget."""
    trimmed = list(history)
    while len(trimmed) > 1 and client.messages.count_tokens(
        model=MODEL, messages=trimmed
    ).input_tokens > budget:
        trimmed = trimmed[2:]
    return trimmed

• Common mistake: assuming old turns fall off automatically.
• Common mistake: treating a model_context_window_exceeded response as a finished answer and showing it to the user.
• How to apply: check stop_reason on every response, and budget the window (count tokens) before each call in long sessions.

3. Sampling: why the same prompt can give different answers

Sampling

In plain English: Choosing each next token at random, weighted by the probabilities the model assigns, rather than always taking one fixed answer.

Why it matters: At every step the model produces a probability distribution over possible next tokens. Sampling from it is how text is generated.

Analogy: Rolling weighted dice. The heavily weighted face wins most of the time, but not every time.

Temperature

In plain English: A setting that shapes the probability distribution the model samples from. A lower temperature concentrates probability on the most likely tokens and makes output more repeatable; a higher temperature spreads it out and makes output more varied.

Why it matters: Source glossary definition (S02).

Because each choice is sampled rather than fixed, the same prompt run twice can return different wording even when both answers are correct. For Harbor Goods, one run says "Your order A1042 shipped yesterday and should arrive tomorrow," another says "Good news! Order A1042 is on its way. Expect it tomorrow." Both are right. This is a property of how the model generates, not a defect.

POST https://api.anthropic.com/v1/messages
x-api-key: $ANTHROPIC_API_KEY
anthropic-version: 2023-06-01
content-type: application/json

{
  "model": "claude-sonnet-5-5",
  "max_tokens": 200,
  "temperature": 0.2,
  "messages": [
    {"role": "user", "content": "Classify this ticket: 'My parcel never arrived.'"}
  ]
}

HTTP 400 invalid_request_error
Fix: remove temperature, top_p and top_k; steer with the prompt.

• Common mistake: assuming temperature 0 makes the model fully deterministic.
• Common mistake: copying sampling settings from an older model's code into a request for a newer model.
• How to apply: for repeatable behavior on current models, be specific in the prompt (exact labels, exact format) and validate the output.

4. Non-determinism: what it means for testing and evals

Non-determinism

In plain English: Identical inputs do not guarantee identical outputs.

Why it matters: It is the primary consequence of sampling.

This changes how you test a Claude feature. A test that asserts the exact text of a response will be inconsistent, because the model can express the same correct answer many ways. That is surprise number three. Instead, assert on the properties that must hold: the structure parses, a required field is present, a value is in an allowed set or within a range.

import json
import re

ALLOWED_INTENTS = {"order_status", "refund", "product_question", "other"}
ORDER_ID = re.compile(r"A\d{4}")


def check_triage(reply: str) -> None:
    data = json.loads(reply)                     # the structure parses
    assert data["intent"] in ALLOWED_INTENTS      # value from an allowed set
    assert ORDER_ID.fullmatch(data["order_id"])   # required field, right shape
    assert 0.0 <= data["confidence"] <= 1.0       # value in range


# Two simulated replies to the same ticket: different text, same meaning.
run_1 = '{"intent": "order_status", "order_id": "A1042", "confidence": 0.93}'
run_2 = '{"order_id": "A1042", "confidence": 0.9, "intent": "order_status"}'

assert run_1 != run_2      # an exact-text test would fail on run 2
for reply in (run_1, run_2):
    check_triage(reply)    # the property checks pass for both runs
print("property checks passed for both runs")

property checks passed for both runs

Eval with a model-graded judge

In plain English: An evaluation that uses a model to judge whether a response's meaning meets a rubric, used when you need to judge meaning rather than structure.

Why it matters: Source glossary definition (S02). Property checks can confirm a reply is valid JSON with an order ID; they cannot tell you whether it was helpful or polite.

This is why the course treats evals as the standard for knowing a feature is correct, and why a later module builds that capability.

References

• Context windows (Claude Platform Docs) · checked 2026-10-20
  What counts toward the window; overflow behavior (400 'prompt is too long' vs model_context_window_exceeded); context rot.
• Token counting (Claude Platform Docs) · checked 2026-10-20
  count_tokens endpoint; counts are estimates; Claude Opus 4.7+ tokenizer produces roughly 30% more tokens for the same text.
• Create a Message (Claude API Reference) · checked 2026-10-20
  Sampling parameter support (temperature, top_p, top_k) varies by model.
• Claude Sonnet 5.5 migration guide · checked 2026-10-20
  Non-default sampling values return a 400 error on Sonnet 5.5.
• Effective context engineering for AI agents (Anthropic Engineering) · checked 2026-10-20
  Supplementary: why more context is not automatically better.
• CertSafari: Claude Developer Foundations study guide · checked 2026-10-20
  Secondary source. Its context-window section (what fills the window; tool results fill it fastest) supports this lesson; its agent material belongs to later modules.

Source notes

• Primary source: S02 of 'MSO Foundations: Developer Module 1' (Developer_M1_vF2.html), four tab panels plus two glossary tooltips.
• Supplementary (not in source, verified in docs): the ~30% tokenizer difference, the exact 400 message 'prompt is too long', the usage field, the count_tokens endpoint, and context rot. These are labelled as clarifications.
• Discrepancy D02: the source says 'Module 3 builds' evals; in the course outline evals are part of Module 04. Narration says 'a later module'.
• Python SDK 1.15.0 (checked 2026-10-20) has no temperature parameter on messages.create, so the sampling example is shown as a raw HTTP request.