How Large Language Models Work: A Visual Guide

An LLM (Large Language Model) is a neural network trained on a very large amount of text to learn patterns in language — and predict which tokens should come next. That single objective, repeated at scale, produces a system that can write, explain, translate, and reason through problems.

This post is a practical reference: what an LLM is, how you call one through an API, what every parameter does, and where things break down. Diagrams are included at each step so the concepts stay concrete.

LLM is not the same as an agent

Before anything else, keep this distinction clear:

An LLM primarily generates language. An agent uses an LLM as part of a larger system that can reason about what action to take and interact with tools or an environment. Tool calling, memory, and orchestration live in your application — not inside the model itself.

What is an LLM?

At the core, an LLM is a pattern-matching engine over tokens. Here are the terms you'll see everywhere:

Tokens and tokenization

LLMs don't directly process normal human words. They process tokens — which can be a whole word, part of a word, punctuation, whitespace-related text, or special tokens (like end-of-sequence markers).

Tokenization is the process of converting raw text into those tokens. Every API call starts here.

Context window

The context window is the amount of tokenized information the model can consider during a particular request. This includes everything you send in:

• System instruction
• Conversation history
• User message
• Retrieved documents (RAG)
• Tool results
• The model's own generated output

Everything must fit within the limit. If your history plus documents exceed the window, you truncate, summarize, or retrieve less.

Parameters and pre-training

Parameters are the learned numerical values inside the neural network that determine how the model processes and generates text. More parameters generally mean more capacity, but also more compute.

Pre-training is the initial large-scale training phase where the model learns general language patterns from massive datasets. After pre-training, you have a base model. Chat models go through additional fine-tuning and alignment on top.

Inference

Once an LLM has been trained, using it to generate a response is called inference. Every chat message, every API call, every agent step that asks the model for text — that's inference.

A prompt is the instruction or input you send to the model. In modern APIs, prompts are usually structured as a list of messages with roles (covered below), not a single raw string.

How LLM APIs work

Your application does not run the model locally (unless you self-host). It sends an HTTP request to a provider's API.

Request — the message your application sends to the LLM API. It usually contains the model name, messages (or a prompt), generation settings, and optionally tool definitions.

API endpoint — the URL where your application sends the request, e.g. POST https://api.provider.com/v1/chat/completions.

Authentication — how the API verifies who is making the request. Typically an API key in an Authorization header.

Request payload — the JSON body of the request. Think of it as: endpoint = where, auth = who, payload = what you want.

Response payload — the JSON data returned by the API, containing the model's generated message, token usage, and finish reason.

Streaming vs non-streaming

• Non-streaming — the API generates the complete response first, then sends it back.
• Streaming — the API sends the response piece by piece as it is generated. This uses Server-Sent Events (SSE) in most providers and is essential for chat UIs where users see tokens appear in real time.

Messages and roles

Modern chat APIs structure input as a list of messages, each with a role:

A typical conversation looks like this:

system    → rules and instructions
user      → the user's request
assistant → the model's response
tool      → result from an external function the model requested

The model sees the full message list (within the context window) and generates the next assistant message.

Generation parameters

These control how the LLM generates its response:

Temperature controls randomness. At 0, output is nearly deterministic — the model picks the most likely token every time. At 0.7, there's balanced variation. At 1+, output becomes more creative but less predictable.

Top-p (nucleus sampling) controls how many of the most probable next tokens are considered. Lower values keep the model focused; higher values allow more variety.

Max tokens caps how many tokens the model can generate in the response. Set this to avoid runaway generation and control cost.

Stop sequences tell the model to halt when it produces a specific string — useful for structured outputs or multi-turn templates.

Seed — some APIs and models support a seed value to make generation more reproducible, though perfect determinism is not guaranteed.

Structured outputs

By default, an LLM returns human-readable prose. For applications and agents, you often need machine-readable output instead.

{
  "name": "Prashant",
  "age": 24
}

JSON and JSON Schema

JSON mode constrains the model to return valid JSON. JSON Schema goes further — it defines the exact shape the response must follow:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" }
  },
  "required": ["name", "age"]
}

Pydantic models

In Python, you can define the expected structure using classes:

from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

This is especially useful with LangChain, where structured output parsers validate the model's response against your schema before your application uses it.

Structured output matters for agents because agents need to make decisions and pass information between components. Prose is for humans; JSON is for systems.

Function / tool calling

Tool calling allows an LLM to request that your application execute an external function — search a database, call a weather API, run code, etc.

The flow:

1. You define available tools (name, description, parameter schema).
2. The model receives the user message and decides whether a tool is needed.
3. If yes, it generates a structured tool call (function name + arguments) instead of a final answer.
4. Your application executes the function and sends the result back as a tool message.
5. The model reads the result and produces the final response.

The model does not execute tools itself. It only requests them. Your code does the actual work.

Streaming in practice

Streaming means receiving the LLM's response gradually as it is generated, instead of waiting for the complete response.

• Token streaming — the model generates one token at a time; without streaming, the user waits until the entire response is ready.
• SSE (Server-Sent Events) — the common HTTP mechanism for delivering those incremental updates from server to client.
• Why it matters — lower perceived latency, better UX in chat apps, and visibility into agent execution as it happens.

For production chat interfaces, streaming is the default choice. For batch processing or pipelines where you need the full response before the next step, non-streaming is simpler.

LLM limitations

Every LLM-powered application should account for these:

These are not bugs to fix in a future version — they are properties of how the models work. Production systems use RAG for fresh knowledge, tools for real actions, validation for structured data, and human review where stakes are high.

For a full walkthrough of how an agent wraps an LLM with tools and state, see Architecture of LangChain & LangGraph.

Basic Python LLM application

Putting it all together — a minimal mental model for a chat application:

The pieces:

• API client — a Python library that communicates with the LLM provider (e.g. the OpenAI SDK).
• Prompt / messages — the structured input you send.
• Model call — the actual API request.
• Parse response — the API returns a response object, not plain text. You extract the content from it.
• Handle errors — API calls can fail due to invalid keys, rate limits, network issues, invalid requests, server errors, or timeouts.

A simple chat loop in Python:

while True:
    user_input = input("You: ")

    if user_input == "exit":
        break

    response = call_llm(user_input)
    print("AI:", response)

To maintain conversation history, append each user and assistant message to a list and send the full list on every call. The model has no memory of its own — your application provides the history within the context window.

Complete mental model

When you build with LLMs, this is the stack:

1. Text becomes tokens via tokenization.
2. Messages with roles structure the input (system, user, assistant, tool).
3. Generation parameters control randomness, length, and stopping.
4. The API handles auth, request/response, and optional streaming.
5. Structured output and tool calling turn prose into machine-actionable data.
6. Your application handles history, tools, validation, and error recovery.

The LLM generates language. Everything else — memory, actions, reliability — is engineering you build around it.