Notes by Prashant
← All notes

Architecture of LangChain & LangGraph

12 min read

The 31 phases are a map of what to learn. This is the other view: one user message, and every stop it makes before an answer comes back. Same system, seen from the inside of a single run.

LangChain and LangGraph are doing different jobs on that run. LangChain is the parts: the model client, the prompt, the tool definitions, the retriever. LangGraph is the control flow: state, the order of steps, the branch where a tool gets called, the loop that comes back, and the checkpoint that lets the run pause. A node is usually a LangChain call wrapped in a plain function. The graph decides whether that function runs again.

Here's the path, in the order it actually happens.

1. The API call

Someone sends a message. Your server receives a normal HTTP request: a user id, a thread id, and the text. Nothing about this step is an LLM. It's the same kind of endpoint you'd write for any other backend.

What the handler does is hand that text to a compiled graph:

result = graph.invoke(
    {"messages": [{"role": "user", "content": body.message}]},
    config={"configurable": {"thread_id": body.thread_id}},
)

invoke is the start of the run. The thread_id matters later, when persistence has to find the previous turns of this conversation. Until then, the graph only has one new message and an empty-or-loaded state.

The handler waits. It does not call OpenAI itself. The model call happens inside a node, several steps from here.

2. State is the only thing that moves

A LangGraph run is a loop over one object: the state. Every node reads it and returns a partial update. Nodes don't pass return values to each other directly.

For a chat agent, state is usually the message list, plus whatever else your app needs to remember:

from langgraph.graph import MessagesState

class State(MessagesState):
    user_id: str
    retrieved: list[str]

MessagesState already has a messages channel with the add_messages reducer. That reducer appends. When a node returns {"messages": [response]}, the new message is added to the list. It does not wipe the history. That single detail is why the model can see the tool result on the next pass.

Other keys behave like normal assignment unless you give them a reducer. retrieved gets replaced by whatever the last node wrote.

3. The first node calls the model

The graph starts at START and follows an edge into the agent node. That node is where LangChain shows up.

Inside it, three LangChain pieces usually sit together:

  • A prompt, if you have a system instruction or a template with variables.
  • A chat model — ChatOpenAI, ChatAnthropic, or whatever provider you wrapped. LangChain's model object is just a client with one interface: messages in, AIMessage out.
  • Tools bound to that model, so the provider knows which functions it's allowed to ask for.
model = ChatOpenAI(model="gpt-4.1").bind_tools(tools)

def agent(state: State):
    response = model.invoke(state["messages"])
    return {"messages": [response]}

bind_tools does not run your functions. It sends their names, descriptions, and argument schemas along with the chat request. The provider returns an assistant message. Sometimes that message is the final answer. Sometimes it is a request to call a tool. The node can't tell which one it should be. It just stores the message on state and returns.

That model request is the first real API call to an LLM in the whole process. Your HTTP handler is still blocked on graph.invoke, waiting for the graph to finish.

4. Tool calling

A tool call is structured data on the assistant message, not a side effect the model performs itself. It looks roughly like this:

AIMessage(
    content="",
    tool_calls=[{
        "id": "call_1",
        "name": "get_order",
        "args": {"order_id": "1842"},
    }],
)

The model chose the name and the arguments. Your process still has to execute them. That's a second node, almost always LangGraph's ToolNode or a function you wrote that does the same job:

  1. Read the last assistant message.
  2. Find each tool_call.
  3. Look up the Python function with that name.
  4. Validate the arguments against the schema.
  5. Run the function.
  6. Write a ToolMessage back onto state, with the same tool_call_id, so the model can match result to request.
from langgraph.prebuilt import ToolNode

graph.add_node("tools", ToolNode(tools))

If the model asked for two tools in one message, ToolNode runs both and appends two tool messages. The functions can be anything you could call from ordinary Python: a database query, a search index, a calculator, another internal API. They are not prompts. They are your code, which is why a bad argument schema or a tool with a vague description shows up here as a wrong call, not as a mysterious model failure.

The tool result is now just another message in state. Nothing has been sent to the user yet.

5. Conditional routing

After the agent node, the graph has to decide where to go. That decision is a normal function. It reads state and returns the name of the next node.

from langgraph.graph import END

def route(state: State):
    last = state["messages"][-1]
    if getattr(last, "tool_calls", None):
        return "tools"
    return END

graph.add_conditional_edges("agent", route, {"tools": "tools", END: END})
graph.add_edge("tools", "agent")

Read that slowly, because this is the whole agent loop:

  • START always goes to agent.
  • agent calls the model.
  • route looks at the message that just got appended.
  • Tool calls present: go to tools.
  • No tool calls: the model produced an answer, so go to END.
  • tools always goes back to agent.

The edge from tools back to agent is the loop. The second visit to agent is a new model call. The message list now contains the original user message, the tool request, and the tool result, so the model can answer from the observation instead of guessing.

Routing here is deterministic on purpose. The model decides which tool and which arguments. Your function decides whether the graph is finished. Mixing those up is how people end up with a model that "routes" by writing the word "DONE" and a parser that sometimes misses it. The tool-call field is already a boolean you can trust: empty means answer, non-empty means act.

You can put a model in the router when the branch is a judgment — "is this a billing question or a technical one?" — and send different specialist nodes. That's still a conditional edge. The router returns a node name. The graph follows it.

6. One request, walked end to end

Say the user asks: "Where is order 1842?"

  1. The API handler calls graph.invoke with that message and a thread_id.
  2. If a checkpointer is attached, LangGraph loads whatever state was saved for that thread. A brand-new thread starts with just this message.
  3. agent runs. The chat model sees the system prompt and the user message, and returns a tool call: get_order(order_id="1842").
  4. route sees tool_calls and sends the run to tools.
  5. get_order hits your database and returns "shipped, arriving Thursday."
  6. That string is appended as a ToolMessage.
  7. The edge back to agent fires. The model is called again, now with the tool result in the transcript.
  8. The model returns a normal assistant message and no tool calls: "Order 1842 has shipped and arrives Thursday."
  9. route sees no tool calls and returns END.
  10. invoke returns the final state to your handler.
  11. The handler takes the last assistant message and sends it as the HTTP response.

Two model calls. One database call. Three nodes visited, one of them twice. The user saw one reply. If you only log the final sentence, steps 3 through 7 are invisible, which is why the later section on evaluation cares about the path and not just the answer.

A retrieval system fits the same shape. A fixed RAG pipeline is a node that runs before agent: embed the question, fetch chunks, write them onto retrieved, then call the model with those chunks in the prompt. Agentic RAG makes retrieval a tool instead, so the model can search, read the chunks, and search again with a better query if the first pass was weak. Same pieces. The conditional edge is what makes the second search possible.

7. Persistence, memory, and a human pause

Compile the graph with a checkpointer and every step above is saved:

graph = builder.compile(checkpointer=checkpointer)

After each node, LangGraph writes a checkpoint keyed by thread_id. The next HTTP request with the same thread id resumes from that state, which is how "memory" works in this architecture. You are not stuffing the entire transcript into a global variable. You are reloading the message list that the previous run already appended.

The same checkpoint is how a run pauses. interrupt_before=["tools"] stops the graph after the model has asked for a tool and before the tool runs. Your API can show a person the pending call — "the agent wants to refund order 1842" — and later resume with the same thread_id. Nothing is recomputed from scratch. The state already holds the tool call.

Streaming is the same run, observed while it happens. graph.stream yields tokens from the model node, or state updates after each node, so the HTTP response can start before END. The routing doesn't change. You're just reading the checkpoints as they're written.

8. When a step fails

Failures show up at specific stops, and the graph should say what happens at each one.

  • The model call times out or returns something that isn't valid tool arguments. Retry that node, or append an error message and route back so the model can repair the call.
  • The tool raises — the order id doesn't exist, the database is down. Catch it inside the tool and return the error as the ToolMessage content. The model then sees a failed observation and can apologize or try a different argument, instead of the whole request 500ing.
  • The router gets a state it doesn't recognize. Treat that as END or a dedicated error node. A router that throws kills the run with no answer at all.
  • The output is the wrong shape for your API. A structured-output parser belongs in the node that needed the structure, with one retry edge back to the model. By the time you're building the HTTP response, you want a value you can serialize.

Retries are just edges. A "failed, try again" path is a conditional route with a counter on state, so a broken tool can't loop forever.

9. Evaluation

Evaluation looks at the trace of the run you just walked, not at a vibe check of the last sentence.

If tracing is on, LangSmith records the tree automatically: the graph, each node, each model call, each tool call, the arguments, the tool output, the latency, and the token cost. Open one trace for "where is order 1842?" and you should be able to point at step 3 and step 7 separately. That's the difference between "the answer was wrong" and "the model called get_order with the wrong id" or "the tool returned the right row and the second model call ignored it."

You evaluate three different things:

  • The final answer. Does it match what a good reply would say? Exact match for factual fields, or a grader model for anything worded in sentences.
  • The trajectory. Did the run call the right tools, in an acceptable order, with arguments that match the question? A confident answer that never called get_order is a failure even if the sentence sounds plausible.
  • The route. On examples that should finish in one model call, did route go to END? On examples that need a lookup, did it go to tools? A router bug hides inside "the agent is dumb" until you score the branch on its own.

Build a dataset of inputs you already know the outcome of. Run the compiled graph against that dataset. Score each trace with those three checks. Then change a prompt, a tool description, or a routing condition and run the same dataset again. The scores tell you whether the architecture got better or whether one example just looked better in a demo.

Production traffic is the same measurement on a sample of real traces. A user thumbs-down attaches to the thread_id and the trace, so you can see which node was responsible. Evaluation is not a phase you bolt on after the demo. It is the only way to see steps 3 through 7 once real users are the ones sending the API call.

What leaves the server

invoke gives you the final state. The handler reads the last assistant message, applies any last guardrail you care about (don't return a raw tool error, don't echo a secret the tool happened to include), and writes the HTTP response.

That's the full process. An API call enters a graph. State carries the transcript. A LangChain model either answers or requests a tool. A conditional edge is the difference between those two. The tool runs in your code, the result goes back on state, and the model is called again. A checkpointer makes the next request a continuation. Evaluation scores the path those edges took, not only the sentence at END.

The roadmap is the order to learn each of these pieces. This is how they sit inside one request. If a specific stop is the one you want expanded — tool schemas, the checkpointer, or how a trajectory eval is scored — tell me at @prashant.code.