LLM Parameters: Model Weights vs Generation Settings

You cannot tune temperature well if you do not know what a token is, or why max_tokens fights the context window. Generation settings sit on top of six ideas every LLM request uses:

This post teaches those six first, then the knobs you actually set on an API call. For APIs, roles, and tool calling, see How Large Language Models Work.

Tokens

An LLM does not read characters or words the way you do. It reads tokens — chunks of text from a fixed vocabulary. A token can be:

• a whole word (Python)
• part of a word (ing, tion)
• punctuation (?, ,)
• whitespace-related text (a leading space is often its own token)
• a special token (<|endoftext|>, a stop marker)

Billing, max_tokens, and the context window are all counted in tokens, not words. English is often ~0.75 words per token; code and other languages can be denser.

Tokenization

Tokenization is the process of converting raw text into those tokens (usually integer IDs). Every prompt you send is tokenized before the model sees it. Different models use different tokenizers, so the same sentence can have a different token count on GPT vs Llama.

You rarely run the tokenizer yourself when using an API — the provider does it. You still need the idea: a short English sentence is a handful of IDs; a long PDF is thousands.

Context window

The context window is the maximum number of tokens the model can consider in one request. Input and generated output usually share that budget. Typical contents:

• system instruction
• conversation history
• retrieved documents
• tool results
• the current user message
• the tokens the model is generating now

If history + documents + max_tokens exceed the window, the request fails, gets truncated, or the model cannot finish. max_tokens is a generation cap inside this window — it does not expand the window.

Parameters

Parameters are the learned numerical values inside the neural network. They determine how the model processes tokens and scores the next token. When people say “70B,” they mean about 70 billion of these weights.

They are not the same as temperature. Weights come from training and stay frozen on a given model snapshot. Temperature is a setting you pass at inference.

Model parameters define capability. Generation parameters define sampling on this call.

Pre-training

Pre-training is the initial large-scale training phase. The model reads massive datasets and learns to predict the next token. That is where grammar, code patterns, and a lot of world knowledge get compressed into the weights.

It is expensive, slow, and done by the lab that trains the model — not by your chat loop. After pre-training you have a base model. Chat models usually get extra fine-tuning and alignment on top, but the bulk of the parameters still come from this phase.

Inference

Inference is using a trained model to generate a response. Every API call is inference. The weights do not update. The model tokenizes your input, fits it in the context window, and samples output tokens.

Generation parameters (temperature, top_p, max_tokens, stop) only exist in inference. They do not change what was learned in pre-training.

How the six ideas connect

With that stack in place, the rest of this post is about the inference knobs.

Where generation parameters go

POST /v1/chat/completions
{
  "model": "gpt-4.1",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Summarize this article." }
  ],
  "temperature": 0.3,
  "top_p": 0.9,
  "max_tokens": 512,
  "stop": ["\n\n---"],
  "stream": true
}

Most providers accept the same core fields on chat completion endpoints. Names may differ slightly (max_completion_tokens vs max_tokens), but the ideas are shared.

How the model picks the next token

LLMs generate text one token at a time. After each token, the model outputs scores (logits) for every token in the vocabulary. Generation parameters reshape those scores before a token is chosen.

That loop runs until the model emits an end token, hits max_tokens, or matches a stop sequence.

Temperature

Temperature divides (or scales) logits before softmax. Low temperature makes the highest-probability token dominate; high temperature spreads probability mass across more tokens.

Practical notes:

• 0 (or omitted on some APIs with “deterministic” modes) — use for code, classification, JSON extraction, and any task where you want the same input to yield the same output as much as the stack allows.
• 0.3–0.7 — good default band for assistants and summarization.
• Above 1.0 — noticeably more varied; useful for brainstorming, risky for facts and code.

Temperature does not make the model “smarter.” It changes which likely continuation gets sampled.

Top-p (nucleus sampling)

Top-p filters the vocabulary to the smallest set of tokens whose cumulative probability is at least p. Rare tail tokens drop out even if temperature would have kept them in play.

Top-k is an alternative cap: only the K highest logits remain. Some APIs expose both; many teams pick either temperature tuning or aggressive top_p/top_k juggling, not all knobs maxed at once.

Rule of thumb: fix temperature first for your task, then adjust top_p if outputs feel too flat or too chaotic.

Length, stopping, and repetition

max_tokens

max_tokens limits how many tokens the model may generate in the completion (not always the same as total context — check your provider’s docs). It controls cost, latency, and whether the model can finish a long answer.

Set it from the expected output size: short labels need dozens of tokens; long reports need thousands. Leaving it unset often defaults to a provider maximum.

stop

stop sequences cut generation early when the model outputs a given string — useful for delimiters, section markers, or preventing the model from rambling into a fake “user” turn.

frequency_penalty and presence_penalty

These reduce repetitive phrasing:

• frequency_penalty — penalizes tokens proportional to how often they already appeared in the output.
• presence_penalty — penalizes any token that has appeared at least once.

Use small positive values when summaries or lists sound looped. They are not a substitute for good prompts or post-processing.

Seed and reproducibility

A seed asks the runtime to make sampling reproducible when supported. In practice:

• The same seed + same model version + same parameters often reproduces output.
• Model updates, different hardware, or parallel batching can still change results.
• For audits and tests, log model id, parameters, and prompt hash — not just seed.

Full reference

Provider docs remain the source of truth for ranges, defaults, and deprecated fields.

Starting points by task

There is no universal preset — but these starting points save time:

Tune with evals: keep a small set of real prompts and measure quality, not vibe.

Model parameters (size and capacity)

When someone says “a 7B model” or “405B parameters,” they mean model parameters — count of weights, roughly correlated with capacity and compute.

More parameters do not replace generation settings. A small model at temperature 0 with good RAG can beat a large model at temperature 1.2 on a narrow task.

Common mistakes

1. High temperature for facts — increases confident-sounding wrong answers; pair factual work with low temperature and retrieval.
2. No max_tokens — runaway completions and surprise bills.
3. Tuning every knob at once — change one parameter, re-run evals, then adjust the next.
4. Expecting seed alone to fix compliance — use schemas, validators, and tests for structured output.

Mental model

• Tokens — the units the model reads and writes.
• Tokenization — text → token IDs.
• Context window — how many tokens fit in one request (input + output).
• Parameters — learned weights (model size) plus generation settings on the call.
• Pre-training — how those weights were learned, once, at huge scale.
• Inference — using the frozen model to generate; this is where temperature and friends apply.

Sampling pipeline at inference: logits → temperature → top_p/top_k → next token → repeat until stop, max_tokens, or context is full.