I Took Apart the Agent I Vibe-Coded

I’ve been obsessed with AI Agents lately. A few rounds of back-and-forth with AI using the laziest possible vibe coding got me a finance Q&A Agent — with RAG (retrieval-augmented generation: look things up first, then answer) and tool calling. Most of the rework went into wiring up retrieval and getting the tool calls right. When it was done, I asked it: “Work out the P&L on one of my stock positions, then discuss the industry outlook based on some research reports.” It actually computed the numbers correctly and gave a real answer — the P&L matched what I had manually verified in my trading app.

The moment it finally ran end to end felt great. Then the awkwardness arrived: someone asked me “how does it actually work?” — from a question going in to an answer coming out, what happens in between? I couldn’t narrate it beyond three layers deep. The project was written by AI; I knew it “ran,” but not “why it ran.”

So I decided to operate on it, and organized the path that worked into three steps — no terrain maps (no file-by-file tour), just a route, as a reference for anyone else starting from vibe coding.

Step one is running the thing, not reading the code.

Why not read the code first? At the time I couldn’t even say how many files the project had; reading it directly would have been groping for furniture in a dark room. Run it once and collect the logs — every name that appears in the logs becomes a signpost for reading the code afterward. Running first gives the reading something to aim at.

I didn’t rush to open any files. First I re-ran that P&L-plus-industry question, and deliberately mixed in a weather question — a capability it didn’t have wired up; I wanted to see how it would react to a job it couldn’t do. To its credit it was honest: it admitted it couldn’t check the weather, then answered the P&L and industry parts as usual. The answer was secondary; what felt reassuring was the logs: the [TRACE] order was node.retrieve → node.agent → node.tools → node.agent → node.generate — agent appears twice. It doubled back mid-flight to call a tool, got the result, and came back to keep thinking.

I stared at those lines for a while, and for the first time had the feeling “this thing isn’t mysterious”: an Agent, seen from outside, is just a few functions standing in line, with something in the middle deciding whether to turn back. What entitles that “something” to decide — the logs alone can’t tell you; you have to enter the code. But the flow was visible to the naked eye, and half the sense of magic evaporated right there.

Step two is grabbing the skeleton.

Time to read code, but only two pieces: whatever manages data flow, and whatever manages the jumps in the flow. In an Agent project most files are glue — config, wrappers, format conversion; if any of it is missing you can slowly patch it back. What truly decides “who it is” comes down to these two. The project uses LangGraph, a graph-style framework: nodes are functions, jumps are edges — every node name in the logs matched up one to one.

The flow isn’t complicated: after retrieve, enter agent. The business of “go to tools when it feels a tool is needed” splits in two inside the code: the LLM is responsible for declaring its intent — when it wants a tool, its reply carries tool_calls; the actual jump is performed by an unremarkable routing function, which reads whether the LLM declared: yes — jump to tools, and when the tool results come back, re-enter agent to keep thinking; no — go to generate to wrap up. Deciding belongs to the LLM; executing belongs to the router. This loop has a name, ReAct — think one step, act one step, look at the result and think again. The name comes from an earlier prompting paradigm; the loop on the graph is just one implementation of it. The termination condition is plain: when the LLM stops issuing tool_calls, the loop ends. I had assumed some deep scheduling algorithm lay behind it all; it turned out to be one sentence.

What stayed with me most from those two files was a very small trade-off. The framework has a concept called a reducer — plainly put, it declares for each field “how new data merges with the old”: messages has one, merging incrementally each turn — that’s where multi-turn memory comes from; retrieval_docs deliberately has none, each turn overwriting the whole field, so last turn’s retrieved documents don’t bleed their flavor into this turn. Honestly, that overwrite scenario has never even triggered in my project — in the logs, retrieve has only run once so far; that judgment is extrapolation, kept on the books as an experiment I owe myself. But the trade-off itself is real: what should accumulate and what should stay fresh — the docs won’t decide for you; every project has to answer it once for itself. This is the kind of place that deserves to be called design.

Step three is verifying with your own hands.

Reading alone isn’t enough: while you read code, your brain auto-fills the runtime behavior. You think you understand, but half of it is imagination. Only by doing it yourself — changing one condition, or auditing the numbers — and watching whether the system reacts as you expected, do you find out what you genuinely understand. The feeling of understanding will lie to you; your hands won’t.

I ran two small experiments.

The first one went straight at the routing function: I changed its return value to a fixed result. The LLM could declare its intent all it liked; the router ignored it across the board and always went to generate to finish. Asked about that P&L again, the answer instantly degraded from “call a tool and give the exact number” to inventing a figure off the top of its head — the invented part was the current price; the cost it actually remembered, because it had been stated in the conversation. The current price had no source, so it had to be made up on the spot. This pinned down exactly where tools sit: retrieval feeds knowledge; live quotes and private state are things it cannot provide — only tools can hand those over. Cut that channel, and the LLM’s only move left is to fabricate — for numbers that don’t exist in its context, tool calling isn’t icing on the cake; it’s the only bridge between the model and the facts. Looking back, it explains why the weather question got an honest answer: checking the weather is plainly beyond its ability, so it confessed outright; the P&L looked like something it could compute, so it dared to invent. “Deciding belongs to the LLM, executing belongs to the router” got hammered home by this experiment too.

The second was computing RRF by hand — an audit, this time. Retrieval in my project does two-way recall: one vector channel, one keyword channel, merged into a single ranking with RRF — a rank converts to 1/(60+rank), scores for the same item from both channels are added together, and everything is re-sorted by total score. I took that stock, picked out the few documents recalled by both channels, added up 1/(60+rank) one by one according to each channel’s rank, and checked the totals against the final ordering the system produced — the order of those few items matched. Before doing the math I thought I understood it; only after computing did I realize I had merely seen it. The real gain was seeing the character of that 60: it flattens the gaps between ranks — first place and third place differ by hardly any score. So RRF is a gentle fusion: it gives both channels a chance to speak, and lets neither one be a winner-take-all. That kind of feel — reading the formula ten times is worth less than adding it up once with your own hands.

As I write this, the debt from step two is still unpaid — ask one more question, let retrieve actually run a second time, and see whether last round’s retrieved documents bleed into this round’s answer; one test settles it. The debt stays on the books, so that “extrapolation” doesn’t quietly turn into “conclusion.”

Three steps done, back to that opening awkwardness: from a question to an answer, three layers — I can now just barely narrate the whole thing. Starting from those [TRACE] log lines, on to why the agent-and-tools loop turns and when it doesn’t, to the reducer on messages, and then to the 60 in the retrieval layer that flattens the rankings; and I can tell which sentences I verified with my own hands and which are still just inference. Details will fade; the backbone is in place.

These three steps draw no terrain, only a route: run it and watch, to build intuition; grab the skeleton, to understand structure; verify by hand, to turn “having seen” into “owning.” I won’t claim this order works everywhere, but the next time I take apart another Agent project, I’ll certainly start by running it — pick up the signposts first, then enter the dark room.

Vibe coding has made code cheap, but understanding doesn’t come at a discount: AI can write the code for me; it can’t think it through for me. That, I suppose, is why I took it apart once.