Pyyan / Agentic engineering
The discipline, not the directory
Building an agent is not choosing a framework. It is nine decisions about control, state, context, tools, safety, measurement and cost, most of which are made badly, early, and only discovered in production. This page is what the people actually shipping these systems have published about each one.
You decide what happens next. Cheaper, faster, testable, and correct for most problems people build agents for.
It decides what happens next. Necessary when you cannot enumerate the steps, and you pay in cost, latency and compounding error.
Anthropic's distinction, and the most consequential line on this page. One wrong step at 95% per-step reliability is a 60% success rate over ten steps. If the path can be written down, write it down.
A → B → CPrompt chainingThe task decomposes cleanly into fixed subtasks.Trades latency for accuracy by making each call simpler.→ ◇ → A | B | CRoutingDistinct input categories are better handled separately.Buys separation of concerns and lets each branch be specialised.→ ⋔ → ⋀ParallelisationSubtasks are independent, or you want several opinions on one thing.Sectioning for speed; voting for confidence.◉ ⇉ ○ ○ ○Orchestrator–workersYou cannot predict the subtasks in advance.The orchestrator decides the decomposition at runtime, not you.A ⇄ ✓Evaluator–optimiserClear evaluation criteria exist and iteration measurably helps.Only works if a critic can articulate what is wrong.Should this be an agent at all?
Most failed agent projects were decided here, before a line of code. The question is not whether an agent could do the job. It is whether anything cheaper could.
Anthropic draws the line at who controls the path. A workflow is LLMs and tools orchestrated through predefined code paths. An agent is a system where the model dynamically directs its own process and tool usage. The distinction is not about sophistication; it is about who decides what happens next.
Agents buy you the ability to handle problems whose steps you cannot enumerate in advance. You pay for that in cost, latency, and compounding error. One wrong step at 95% reliability becomes a 60% success rate over ten steps. If you can enumerate the steps, enumerate them.
Scope is also a safety boundary. An agent that can only read is a different risk class from one that can write, and one that can spend money is different again. Draw that boundary before architecture, because everything downstream inherits it.
You should consider adding complexity only when it demonstrably improves outcomes.
One prompt, fifty tools, and a hope. Tool selection accuracy collapses as the tool count rises, the context fills with descriptions the model never uses, and every failure is unattributable. This is the single most common shape of a broken agent.
A support inbox. You want angry emails escalated to a human and neutral ones auto-acknowledged.
So: The line is not whether a model is involved. It is whether you know the shape of the path. If you know it, encode it. It will be cheaper, faster, testable, and it cannot wander.
What does the system remember, and where does it live?
This is where most teams conflate two different things. State and memory are not synonyms. Memory is one kind of state, and treating them as the same produces systems that cannot be resumed, replayed or debugged.
State is everything the system needs to continue. It splits in two. Execution state is where the agent is in its own process: the step counter, the pending tool call, the retry count, the current plan. Business state is what the agent has actually done to the world: the row written, the payment sent, the ticket opened.
The twelve-factor agents formulation is to unify execution state and business state, keeping them in one place, in your own data model, rather than letting a framework hide the execution half in memory it will lose on restart. If those two diverge, an agent that crashes mid-run will either forget it sent the email or send it twice.
Memory is different. State is what the agent needs to finish this task. Memory is what it should carry into tasks it has not started yet. Persistent state also includes things that are not memory at all: task ledgers, permissions, commitments, provenance and audit trails.
Getting this wrong is subtle. If you store 'the user prefers metric units' in execution state, it vanishes when the run completes. If you store 'currently retrying step 4' in long-term memory, the agent will resume a task that finished last week.
Unify execution state and business state. Make your agent a stateless reducer.
The agent works until the process restarts, then resumes from nothing or from the wrong place. Almost always because execution state lived in a framework's in-memory object rather than in a table you own.
An agent chasing an unpaid invoice. Step 4 of 9 is send_email. The container is preempted mid-step.
1. send_email(#1042) → delivered ✓ 2. container dies before anything is written 3. restart → step counter is gone, run restarts at 1 4. send_email(#1042) → delivered again → the customer gets two emails, and you cannot prove why
1. write { step: 4, pending: send_email, key: inv-1042-r1 } 2. send_email(#1042, idempotency_key: inv-1042-r1) → delivered ✓ 3. write { step: 5, pending: null } 4. container dies → restart reads step 5 and carries on → and if it died between 2 and 3, the key makes the resend a no-op
So: Checkpointing alone is not enough. Replay is the recovery mechanism, so every side effect has to tolerate being attempted twice. The idempotency key is what turns a crash into a resume instead of a duplicate charge.
What should survive this conversation, and in what form?
Memory is not a database you bolt on. It is a decision about what is worth remembering, made repeatedly, under a budget, and the hard part is not storage but consolidation and forgetting.
The useful taxonomy is borrowed from cognitive science and has held up well in practice: working, episodic, semantic and procedural. They differ in what they hold, how long they live, and, critically, how they are written.
Working memory is written by the agent loop and thrown away. Episodic memory is written by logging what happened. Semantic memory is not written directly at all. It is produced by consolidation, where the agent processes its episodes, finds repeated patterns, extracts entities and relationships, and distils them into reusable facts.
The failure everyone hits is temporal. Facts change. 'The customer is on the enterprise plan' was true in March and false in June, and a vector store that returns both with equal confidence has made the agent worse, not better. This is why the strongest systems model when a fact was true, not just that it was.
The benchmark that separates these is LongMemEval, which tests recall of facts that change over time. Zep's Graphiti backend scores 63.8% against Mem0's 49.0%, a fifteen-point gap on exactly the capability most production agents actually need.
The agent maintains precise tallies across thousands of game steps… after context resets, reading its own notes enables continuation of multi-hour sequences.
Every turn written to a vector store, nothing consolidated, nothing expired. Retrieval quality degrades as the corpus grows, so the agent gets measurably worse the longer a customer uses it, the opposite of the intended effect.
In March the customer was on the Pro plan. In June they upgraded to Enterprise. In August the agent is asked what support tier they get.
query → “what plan is this customer on?” 0.91 “Customer is on the Pro plan.” (March) 0.89 “Customer upgraded to Enterprise.” (June) → both are similar, both come back, neither is dated → the agent picks the higher score and answers Pro
plan(customer_88) = Pro valid Mar 02 → Jun 14 plan(customer_88) = Enterprise valid Jun 14 → now → query at t = today returns exactly one fact → and the March answer is still there if you ask about March
So: This is why LongMemEval exists and why the gap on it is fifteen points. Most production memory questions are not “what do you know” but “what is true now”, and similarity search has no opinion about time.
| Kind | Horizon | What it holds | When it should die | Example |
|---|---|---|---|---|
| Working | This turn | The current task, recent turns, the scratchpad. | When the session ends, and that is correct behaviour, not a bug. | The file the agent is currently editing. |
| Episodic | Specific past events | Sequences of what the agent actually did, and what happened. | Never, but it must be consolidated or it becomes a landfill. | “Last Tuesday approach X failed on client Y because of Z.” |
| Semantic | Timeless facts | Facts about the world and the user, distilled from episodes. | When superseded, which is why temporal validity matters. | “This customer is on the enterprise plan.” |
| Procedural | Learned skill | How to do a thing: the agent's repertoire of effective actions. | When the underlying system changes and the skill goes stale. | “The optimal sequence for booking travel in this system.” |
Working memory is written by the loop. Episodic is written by logging. Semantic is not written directly at all. It is produced by consolidation from episodes. Procedural is the one almost nobody implements, and the one that makes an agent get better at a job rather than merely remember it.
What goes into the window on this call, and what comes out?
Context engineering superseded prompt engineering for anything long-horizon. Prompt engineering is discrete: you write instructions once. Context engineering is a loop that owns the entire token lifecycle, from the first system prompt token to the last compacted summary.
The premise is that context is a finite resource with diminishing marginal returns. Anthropic frames it as an attention budget: every token spends from a limited pool. The architecture explains why. A transformer computes n² pairwise relationships for n tokens, and models have seen far fewer long sequences in training than short ones.
The observable consequence is context rot: as the window grows, recall of what is inside it falls. Not a cliff at the limit, a gradient from the beginning, with a well-documented 'lost in the middle' effect. A million-token window is a budget, not a promise.
So the goal is the smallest set of high-signal tokens that produces the outcome. Three techniques carry long-horizon work, and they are chosen by task shape rather than ranked.
Compaction summarises the trajectory and restarts the window with the summary. Tune it by first maximising recall, then trimming for precision. Structured note-taking writes state outside the window entirely. Sub-agents give each subtask a clean window and return 1,000–2,000 token summaries to a lead agent.
Find the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.
The whole repository, the whole conversation, every tool result retained forever, on the theory that a large window makes curation unnecessary. Accuracy falls while the bill rises, and because nothing errors, the degradation is invisible without evals.
A coding agent, ninety turns into a refactor, about to hit the window. It has to be reduced to a summary it can keep working from.
we chose Postgres over Redis for the queue, because ordering matters auth middleware moved to lib/auth.ts, 3 imports still to update the flaky test is a real race in the pool, not a test bug the user asked for British spelling throughout
the full text of 40 files we read and did not change 12 successful test runs that told us nothing new the directory listing, four times our own intermediate reasoning that led nowhere
So: The test for keeping a line is not “was this important?” but “would losing it change what happens next?” Tune the prompt for recall first, missing nothing, then trim for precision. A file can be re-read; a decision cannot be recovered.
What can the agent actually do, and how well is it described?
Tools are the agent's entire causal surface on the world. Anthropic calls this the agent–computer interface and argues it deserves as much design effort as a human-computer interface. Most teams give it almost none.
The first rule is that tools are not an API wrapper. Wrapping every endpoint one-to-one produces a large, flat, ambiguous surface. Consolidate instead: replace list_users, list_events and create_event with a single schedule_event; collapse get_customer_by_id, list_transactions and list_notes into get_customer_context. More tools do not lead to better outcomes.
The second is that tools should return meaning, not data. Prioritise contextual relevance over completeness, and prefer name and file_type to uuid and mime_type, because models handle natural-language identifiers markedly better than cryptic ones. A tool that returns a raw 40-field JSON row has pushed its work into the context window.
The third is token discipline at the boundary. Pagination, filtering, range selection and truncation with sensible defaults. Claude Code caps tool responses at 25,000 tokens by default; when you truncate, say so in the response and tell the agent how to get the rest.
And descriptions are prompt engineering, among the highest-leverage work available. Write them as you would brief a new hire: state the query format, define the terminology, explain how resources relate. Small refinements here produce disproportionate gains.
If a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better.
Two tools whose descriptions do not clearly separate them. The agent picks inconsistently, evals look noisy rather than broken, and the cause is invisible until someone reads the descriptions side by side.
A search tool the agent kept misusing, calling it with natural-language questions and getting nothing back.
search(query: string) """Search for documents.""" → agent calls search("what did Anu say about the Q3 budget") → 0 results. The index only matches keywords.
search(query: string, author?: string, after?: date) """Keyword search over documents. Not semantic. pass 2–4 keywords, not a question. To narrow by person use `author`, not the query string. Example: search("Q3 budget", author="Anu")""" → agent calls it correctly on the first try
So: Nothing about the implementation changed. Writing the description the way you would brief a new hire, saying what it is not, naming the format and showing one example, is among the highest-leverage work available, and almost nobody does it.
What stops it, and who has to say yes?
Guardrails are not a content filter bolted on at the end. They are the runtime expression of the blast radius you drew during scoping, and the most effective one is not a model at all.
The highest-value rule is determinism first, model second. Use ordinary code for anything with a static control flow, meaning routing, validation and permission checks, and spend model calls only on genuinely non-deterministic judgement and natural-language understanding. Every decision moved out of the model is a decision that cannot hallucinate.
The second is the human-in-the-loop gateway: an approval breakpoint before any state-mutating action. Database writes, payments, outbound email. Twelve-factor agents makes the elegant point that this should be a tool call like any other. The agent requests a human the same way it requests a search, which keeps one control flow rather than two.
Then the layered defences. Input classifiers for prompt injection, output rails for what got through, and identity controls treating each agent as a non-human identity with scoped, time-bound, least-privilege credentials rather than standing access.
One caution about the pattern people reach for first. Having a model check a model is weaker than it sounds: LLMs do not reliably self-correct reasoning without external feedback, and self-correction can degrade results. Multi-agent debate barely outperforms plain self-consistency at equal token cost. What does work is grading steps rather than outcomes: process reward models, where an agentic verifier at 4B parameters beat state-of-the-art outcome models by 25%.
LLMs cannot self-correct reasoning yet. Without external feedback, self-correction can degrade performance.
An agent asked to check its own work, reporting high confidence in wrong answers. Self-assessment without external signal mostly measures fluency. Anything that matters needs an independent check: a test suite, a deterministic validator, or a human.
The refund agent has decided a refund is warranted. It must not issue one unsupervised.
if (action.mutates) { pauseAgent(); notifyHuman(); /* … */ } → a second code path for pause, resume, timeout and rejection → which is separately tested, separately broken
tools = [search_orders, issue_refund, ask_human] ask_human({ question: "Refund ₹4,200 on #8812, damaged in transit?", options: ["approve", "reject"], evidence: [photo_url] }) → the agent already knows how to wait for a tool result → pause, resume and timeout are the machinery you already have
So: Contact humans with tool calls. The agent asking a person is not a special case in the runtime. It is a tool that happens to be slow, and modelling it that way removes an entire parallel code path.
How do you know a change made it better?
Without evals you are not engineering, you are redecorating. Agents fail probabilistically and silently. The same prompt passes on Monday and fails on Thursday, so intuition about whether a change helped is worthless at this error rate.
Build the eval set from real traces, not imagined ones. The failures that matter are the ones your users already hit, and production traces are the cheapest source of hard cases you will ever have. This is why observability and evaluation converged into the same products.
Measure more than accuracy. A change that improves task success while tripling tool calls and doubling tokens may be a regression once it meets a bill. Track accuracy, runtime, tool-call count, token consumption and error frequency together.
Then wire it into CI. Braintrust's GitHub Action runs evals on every commit and can block a merge on a quality threshold, treating prompts and tool descriptions as code that can regress, because they are.
Public benchmarks are for calibration, not for your product. SWE-bench Verified for real bug fixes, τ²-bench for tool-agent-user interaction and policy adherence, Terminal-Bench for command-line work, OSWorld for computer use, GAIA for general assistance. As of April 2026, state of the art ran from the high 30s on OSWorld to the mid 70s on SWE-bench Verified, worth knowing before you promise reliability.
A prompt tweaked until a handful of hand-tested examples look good, shipped, then quietly worse in aggregate. Without a held-out set there is no way to distinguish improvement from overfitting to the three cases you kept trying.
A user reports the agent refunded the wrong order. You have the trace. Most teams fix the prompt and move on. This is the step they skip.
So: Twenty cases mined from real failures beat two hundred invented ones. A bug that was never turned into a test case is a bug you have agreed to ship again.
When it fails at 3am, can you tell why?
An agent failure is not a stack trace. It is a decision that looked reasonable at every individual step and was wrong in aggregate, so the unit of debugging is the trace, not the log line.
The trace is the primary object: nested spans across the agent loop, every model call, every tool invocation, every retrieval, with inputs, outputs, latency and token counts attached. Anything less and you are guessing which of forty steps went wrong.
The layer standardised faster than most people expect. OpenTelemetry GenAI semantic conventions mean instrumenting once and switching backends later, a genuinely unusual property in a market this young, and worth exploiting rather than betting on a vendor.
Cost monitoring is not a separate discipline. Token spend per route, per model, per tenant belongs on the same trace as latency and quality, because in agent systems those three trade against each other continuously and a win in one is often a loss in another.
Close the loop deliberately: production traces become eval cases, eval failures become regression tests, regression tests gate deploys. Observability that only produces dashboards is a cost centre; observability that feeds the eval set is how the system improves.
Every model call logged, no parent-child relationship between them. You can see that step 31 returned nonsense but not which planning decision twelve steps earlier caused it, and reconstructing that by hand is where the night goes.
Monday's spend is ₹4,100. Thursday's is ₹12,800. Nothing shipped in between.
So: Agent cost is emergent, not designed. No code changed; a dependency did. Alert on tokens per successful task and you find it while the bill is still small.
Can the system get better without you rewriting it?
The newest phase, and the least settled. The premise is that prompts, tool descriptions and context policies are parameters, and parameters can be optimised rather than hand-tuned.
DSPy started this by treating a prompt as a compiled artefact: declare the signature, supply examples and a metric, and let an optimiser such as MIPROv2 search the prompt space. You stop writing prompts and start specifying objectives.
GEPA sharpened it considerably. Rather than optimising against a scalar score, it reads the full execution trace, diagnoses failures in natural language, assigns credit, and evolves the prompt from that reflection. It reports beating reinforcement-learning methods such as GRPO and prompt optimisers such as MIPROv2 by 10–20% accuracy with up to 35× fewer rollouts, which matters because rollouts are the expensive part.
Agentic Context Engineering (ACE) applies the same reflective idea to the context itself, making localised updates rather than repeated full rewrites, and reports ReAct + ACE outperforming baselines by an average of 10.6%.
Below that sits reinforcement fine-tuning, where the bottleneck has moved. With GRPO and RLOO now standard for reasoning models, wall-clock training time is governed by how fast your environment produces verified rollouts, not by GPU count. Prime Intellect's Environments Hub exists for exactly this reason and carries 2,500+ community environments.
Up to 10–20% higher accuracy than GRPO, with as much as 35× fewer rollouts.
An optimiser will maximise exactly what you measured. If the metric rewards answer length or judge agreeability, that is what improves, and the system gets measurably better at the benchmark while getting worse for users.
An extraction agent scores 71%. You want it higher. The two ways of getting there are not equally efficient.
reward = 0.71 → try a variation → 0.69 → try another → 0.73 → the signal is one number per rollout → it says nothing about which of nine steps was wrong → so it needs thousands of rollouts to find out
read the full execution trace, then reflect in words: "29 of 31 failures had a date in DD/MM. The prompt never states a format, so the model assumed MM/DD." → propose one targeted edit, re-run, keep if better → reported: 10–20% better than GRPO, up to 35× fewer rollouts
So: A scalar tells you that you failed. A trace tells you why. Reflective optimisers are sample-efficient for the same reason a good code review is: they read the work, not just the grade.
This is not settled, and anyone who tells you it is has picked a side. Two of the most credible engineering teams in the field published opposite conclusions within months of each other, and both were reasoning from production experience.
The reconciliation is task shape. Parallel sub-agents work when subtasks are genuinely independent and the output is a synthesis: research, search, review. They fail when subtasks share implicit design decisions, which is most of software engineering. Cognition builds a coding agent; Anthropic described a research system. Both are right about their own problem, and the mistake is generalising either answer to yours.
Ten things that separate an agent demo from an agent system. None of them are about which model you picked.
No mega-prompts with fifty tools. Specialised sub-agents with two to five focused tools each, coordinated by a router.
Deterministic code for static control flow: routing, validation, permissions. Reserve model calls for genuine judgement and language understanding.
Approval breakpoints before any state-mutating action: database writes, payments, outbound email. Implemented as a tool call so pause and resume share one control flow.
Checkpoint to durable storage such as Postgres or Redis after each tool invocation, not at the end of the run. This is what survives a network failure and resumes a paused workflow.
Replay is the recovery mechanism, so every external action must tolerate a second attempt. Idempotency keys on every write, payment and send.
Decide explicitly what enters it and what is dropped. Do not delegate that to a framework default you have not read.
Represent failures concisely. A stack trace pasted whole spends the budget that the recovery attempt needs.
Nested spans over the whole loop. You cannot reconstruct a trace you did not record, and the interesting failures are rare.
A held-out set built from real production failures. Every optimiser and every prompt change is meaningless without one.
Tokens per successful task, tool calls per run. Agent cost is emergent, and a looping retry can multiply a bill with no code change.
This layer moves faster than the rest of the index. Four notable tools in it were acquired or shut down in the eighteen months to August 2026. Where a tool is named here it is because it solves a problem described above, not because it is the largest.