What Is an AI Agent? Anatomy, Types & When to Build One: exercises

The 58% accuracy on M&A documents and the 18% hallucination rate on custom clause references are disqualifying for this expansion, regardless of timeline pressure.

The right recommendation: Do not expand to M&A due diligence at the current quality level. Here's the defense:

Why the accuracy gap matters more than it looks: In contract review, a missed risk clause or a hallucinated clause reference isn't a recoverable error in the normal sense. An M&A deal where your agent falsely certifies a provision as standard (when it isn't) is a legal liability event, not a user experience problem. Blast radius is extremely high.

What the 58% accuracy means: If the head of legal's team reviews 50 M&A documents using this agent and trusts its outputs, roughly 21 will have meaningfully wrong assessments. In an M&A context, each of those is a potential issue that surfaces at closing.

The right path: Two options that aren't "expand the scope":

1. Treat the M&A pilot as a research phase, not a capability expansion. The agent runs in parallel with human review; it surfaces candidates for human attention, not final assessments. Measure agreement rate with human reviewers on 200 documents before any demo.
2. Scope the M&A demo to the specific clause types where the agent's accuracy is above 85% (likely standard boilerplate). Demonstrate quality on what it does well, explicitly exclude what it doesn't.

The board demo with 58% accuracy on M&A documents is a reputational risk to the company, not a showcase. The correct recommendation is to present option 2 to the head of legal: "Here's what the agent can reliably do in M&A today, and here's what we need to reach the accuracy threshold for full due diligence."

Self-assessment checklist:

• Did you identify blast radius as the disqualifying factor, not just the accuracy number in isolation?
• Did you propose a concrete alternative path rather than just "no"?
• Did you recognize the evaluation oracle problem (58% on a 50-document test is statistically weak)?
• Did you resist the timeline pressure with a principled argument, not just caution?

Both the sales rep and the CTO are right about their respective concerns, and the correct design satisfies both.

Step 1: Classify the tools by blast radius:

• query_crm → Read. No blast radius. Never gate.
• send_email → Write. High blast radius. An email sent to 500 customers with an incorrect personalization, or to the wrong segment, is a reputational and relationship risk. Gate.
• update_crm_field → Write. Low-to-medium blast radius. CRM fields can be corrected. The "at-risk: outreach sent" flag is informational, not customer-facing. Gate with lower friction.

Step 2: Design the gates:

For send_email: Use a batch review queue, not per-email approval. The agent runs its full identification and drafting process, then surfaces a queue of N emails with preview. The rep reviews the queue (with one-click approve/edit/reject per row) and clicks "Send approved." This preserves quality control without creating per-email friction. The CTO gets efficiency because reviews happen in batches; the rep gets control because nothing sends without review.

For update_crm_field: Use post-action notification with undo window. The agent updates the field immediately (to avoid blocking the workflow) but sends a digest to the rep: "Marked 12 customers as at-risk today. [View and undo within 24 hours]." Most updates will be correct; the 24-hour undo window handles the rare errors.

Step 3: Define the conditions for non-gated send: Over time, after the rep has reviewed 500+ emails and the override rate falls below 3%, the gate can shift to an anomaly-only model; the agent sends autonomously but flags anything outside learned patterns for review. This is earned autonomy.

The principle: Match gate friction to blast radius and reversibility. Not every write tool gets the same gate. High blast radius + irreversible = hard gate. Low blast radius + reversible = soft gate with undo.

Self-assessment checklist:

• Did you distinguish between gate types (batch review vs. per-action vs. post-action undo)?
• Did you propose a path to earned autonomy rather than permanent gating?
• Did you satisfy both the rep's control requirement and the CTO's efficiency requirement?
• Did you avoid gating the read tool (query_crm)?

Choose Option A for v1, with a clear trigger to switch to Option B.

Here's why:

Option A is the right starting point because:

1. Single agent = single system to debug. When something goes wrong (wrong URL scraped, missed a product announcement, hallucinated a pricing change), you have one trajectory to inspect.
2. A multi-agent pipeline introduces inter-agent communication overhead, state synchronization complexity, and failure modes that are much harder to diagnose solo. If Specialist Agent 3 returns a malformed result, how does the orchestrator handle it? What's the retry logic? Who owns the shared state?
3. 3 weeks is not enough time to build, test, and debug a multi-agent system to production quality as a solo engineer. Option B would likely ship with known bugs that you don't have bandwidth to fix.

What Option B offers that Option A doesn't: Parallelism (5 competitors simultaneously instead of sequentially), which cuts wall-clock time from ~25 minutes to ~5 minutes for a full briefing run. This is a latency argument, not a quality argument.

When to switch: If and only if (a) you've shipped Option A and it works reliably for 4 weeks, AND (b) the 25-minute runtime is actually a problem for your users (it probably isn't for a weekly briefing). Use the qualification scorecard; if Option A's sequential performance is insufficient after evaluation, move to Option B. Not before.

The meta-lesson: "Start simple" isn't just about architecture tier. It applies to complexity within tier too. A simple single agent is simpler than a complex multi-agent system, even when both are "agents."

Self-assessment checklist:

• Did you factor in team size as a first-class constraint?
• Did you identify a concrete trigger condition for switching to Option B?
• Did you recognize that Option B's benefit (parallelism) is a latency gain, not a quality gain?
• Did you resist the impulse to choose the more sophisticated-sounding architecture?

Hypothesis 1: Distribution shift between staging and production inputs.

The most common cause of this pattern. The curated staging set probably over-represented well-formatted, clean documents. Production uploads include scanned PDFs, poorly OCR'd documents, documents in unexpected formats, or documents with layouts the agent's PDF parser doesn't handle well.

How to test: Sample 50 failing production documents. Compare their raw parsed text to 50 passing staging documents. Look for: unusual characters, encoding issues, empty sections, multi-column layouts that parsed incorrectly, or document types the agent wasn't designed for.

Hypothesis 2: The agent is looping on unclear or corrupted inputs.

The 8.4-step average (vs. 3.2 in staging) and 14% max-turns exits strongly suggest the agent is retrying steps on inputs where it can't make progress. Common pattern: the agent reads a page, the content is garbled, it "tries again" with a slightly different tool call, fails again, loops.

How to test: Pull the full trajectories from the 14% of runs that hit max-turns. Look for repeated identical or near-identical tool calls. If you see read_page(7) called 4 times in a row, that's a loop; the agent doesn't know when to give up on a page.

How to fix: Add explicit loop detection in the harness. If the same tool is called with the same arguments twice consecutively, inject an observation: "This step has already been attempted. Try a different approach or acknowledge that the input section is unreadable."

Hypothesis 3: The staging accuracy metric was too easy.

91% in staging may have been inflated if the evaluation oracle was too forgiving, e.g., BLEU score against a reference summary, or a simple length check. In production, if you're measuring user-rated quality or task-relevant accuracy, the 72% might be the honest number, and the 91% was an artifact of a weak eval.

How to test: Re-run 50 staging documents through the production evaluation oracle. If their score also drops to ~72%, the staging eval was wrong.

Self-assessment checklist:

• Did you identify distribution shift as the primary hypothesis?
• Did you connect the trajectory length increase to looping behavior specifically?
• Did you propose a concrete test for each hypothesis (not just "investigate further")?
• Did you include the possibility that staging accuracy was measuring the wrong thing?

Root cause: The agent is stuck in a read → error → re-read → same error → re-read loop because:

1. The verify_identity tool returns different error messages depending on whether the name has special characters or whether it matches the document, and these two errors require different fixes.
2. The agent interprets "name mismatch with document" as "I read the document wrong" and re-reads it, which produces the correct special character name, which then hits the encoding error again.
3. There is no harness-level loop detection. The agent can call read_document as many times as it wants with no consequence.

Immediate fix (now, while paged):

• Terminate all sessions stuck in this loop manually via the admin console.
• Route affected customers (anyone with a non-ASCII name field) to manual review queue as a temporary bypass.
• Deploy a hotfix: add a harness-level check that terminates the session with NEEDS_HUMAN_REVIEW status if the same tool is called with identical arguments twice in the same session.

Permanent architectural change:

1. Harness-level loop detection should have been in v1. Any tool called with identical arguments twice in a session should trigger an explicit harness intervention: "This step has already been attempted with this input. The agent cannot resolve this automatically. Routing to human review."
2. Tool error taxonomy. The verify_identity tool should return structured errors, not free-text strings. {"error_code": "ENCODING_ERROR", "field": "name"} vs. {"error_code": "NAME_MISMATCH", "field": "name"} are meaningfully different errors that require different agent responses. The agent is doing string matching on error messages, which is fragile.
3. Human-review fallback as first-class exit. Any agent session that cannot complete due to a tool error should have a clean fallback path to human review, not just a max-turns exit with no output.

Self-assessment checklist:

• Did you identify the re-read loop as the root cause (not just "the tool is broken")?
• Did you propose both an immediate hotfix and a permanent architectural change?
• Did you identify the lack of loop detection in the harness as the structural failure?
• Did you recommend structured error codes instead of free-text error messages?

The 4× token increase from "Think carefully" is almost certainly because the instruction triggered verbose chain-of-thought reasoning traces in the output; the model is now explaining its reasoning before each summary section, where before it was summarizing directly.

Diagnosis first: Pull 10 sessions from before and after. Compare token usage by section: system prompt, user messages, model output. If the output tokens increased 4× but input tokens didn't change, the model is generating much longer reasoning traces.

Option 1: Bound the reasoning, don't just hide it (highest impact)

Moving reasoning into a hidden scratch pad does not make it free. Reasoning tokens, including a model's extended thinking where available, are billed as output tokens. The savings come from making the reasoning shorter: ask for a brief justification per source (for example, at most two sentences), or set an explicit thinking-token budget, and strip the reasoning from the final output. A two-step pipeline (step 1: reason briefly, step 2: summarize based on that reasoning) also works. Measure how much of the 12,400 tokens is reasoning to estimate the saving.

Option 2: Cascade to smaller model for reasoning, larger model for synthesis

If the 12% quality gain is real, it likely comes from better reasoning, not better summarization. Use a small, cheap model (for example Claude Haiku) for the reasoning step, then pass the reasoning output as context to a more capable model (for example Claude Sonnet) for the final synthesis. Compare cost and quality against the single-model baseline before adopting it.

Option 3: Constrain output format more tightly

Expanded output format instructions may be generating more tokens in the output. Tighten the output schema, if the output is a JSON object with defined fields and field-level length limits ("summary": string max 200 words), the model cannot exceed those bounds regardless of how much it "thinks carefully."

Recommended approach: Do Option 3 first (low risk, low effort). Then instrument with Option 1's scratch pad pattern. Measure quality and cost after each change. Don't revert the prompt; iterate toward the budget target while preserving quality.

Self-assessment checklist:

• Did you diagnose the source of the token increase (output reasoning traces, not just "the prompt is longer")?
• Did you propose a path that preserves most quality gains rather than reverting the change?
• Did you suggest cascading (smaller model for cheap tasks, larger for quality tasks)?
• Did you propose an output format constraint as the lowest-effort first step?

Strong answer structure:

"The distinction is about who controls what happens next. In our current pipeline, code controls every step: extract clause, classify risk, flag it. That's a workflow. It works well for contracts where the relevant clauses are well-defined and the classification is consistent.

An agent is different in one specific way: the model decides what to look at next, based on what it found. For standard NDAs, we don't need that. For M&A due diligence, we do, because a discovery in section 8 about change-of-control provisions might mean the agent needs to re-examine section 3 for related indemnification language. Our current pipeline can't do that; it reads top to bottom and exits.

You're right that agents are more complex. The reasons are real: nondeterminism, harder to debug, more expensive to run, more failure modes. So here's my answer to whether it's worth it: we should only build an agent if we can prove the current workflow fails on our target cases. What I'm proposing is a 2-week evaluation: run 100 real M&A documents through the current pipeline, measure where it fails, and only proceed with the agent architecture if the failure rate is above 20%. If the current pipeline handles 90% of M&A documents well, we shouldn't build an agent."

What makes this a strong answer:

• Defines the distinction concretely without jargon
• Acknowledges the legitimate concern (complexity is real)
• Doesn't advocate for agents unconditionally, proposes an evaluation gate
• Gives the VP a specific decision checkpoint, not a vague promise

Strong answer structure:

Start by identifying the blast radius tiers, not a single threshold:

Tier 1: Fully autonomous (no human approval needed):

• Refund ≤ $X (your base threshold: start at $50, calibrate based on error rate data)
• Customer history: 0 prior disputed refunds
• Reason: standard (order not received, item damaged, wrong item sent; all validatable from order data)
• Product: in-stock, not final sale

These cases represent probably 60–70% of volume. The blast radius of a wrong auto-approval at $50 is meaningful but recoverable.

Tier 2: Auto-approve with notification and undo window:

• Refund $50–$200
• Standard reason, but customer has 1 prior disputed refund
• The agent approves, sends a notification to the ops team, and logs the action with a 4-hour undo window.

Tier 3 (Human review queue):

• Refund > $200 (or any amount where total refunds to this customer in 30 days > $500)
• Flagged reason: fraud indicator, chargeback history, "received damaged" on perishable goods
• Multiple items in same order

Tier 4 (Immediate human escalation):

• Any refund involving a regulatory complaint, social media mention, or legal threat in the conversation
• Account suspected of abuse patterns

How to set the initial dollar threshold: Start conservative ($25–$50). After 2 weeks of data, analyze the cases that fell in Tier 1. What's the false positive rate (auto-approvals a human reviewer would have denied)? If it's below 2%, raise the threshold. Never raise the threshold without empirical evidence.

Self-assessment checklist:

• Did you design multiple autonomy tiers, not a single binary gate?
• Did you specify an undo window for medium-risk actions?
• Did you propose a process for calibrating the threshold based on data?
• Did you include customer behavior signals (not just dollar amount) as a factor?

The question most teams answer badly: "We'll use BLEU score / semantic similarity / LLM-as-judge." These are implementation choices. The harder question is what you're actually trying to measure.

Step 1: Define the task success criteria in plain English

For a research summarization agent, success is: "The summary accurately represents the sources, covers the key claims relevant to the research question, does not introduce claims not supported by the sources, and is usable by a domain expert to make a decision without re-reading the originals."

This gives you four measurable dimensions:

1. Factual accuracy: claims in the summary that can be verified against the source documents
2. Coverage: key claims in the source documents that should be in the summary but aren't
3. Faithfulness: claims in the summary that contradict or are absent from the sources (hallucinations)
4. Usability: domain expert rating: "Would you make a decision based on this summary alone?"

Step 2: Choose how to measure each dimension

1. Factual accuracy: LLM-as-judge (Claude evaluates each claim in the summary against the source) + random human spot-check (20% of cases). Target: ≥ 90% of claims verifiable.
2. Coverage: Build a reference summary for each test document, marking 5–10 "must-have" key claims. Measure what percentage of those appear in the agent's summary. Target: ≥ 80% coverage.
3. Faithfulness: Dedicated hallucination detector: for each claim in the summary, retrieve the most relevant source passage and evaluate whether the claim is supported. Target: hallucination rate < 5%.
4. Usability: Domain expert survey on a weekly sample (10 summaries/week). 5-point scale. Target: ≥ 4.0/5.0 average.

Step 3: Define the minimum acceptable score before shipping

Primary gate: faithfulness < 5% (non-negotiable: hallucinations undermine the entire value proposition). Secondary gate: coverage ≥ 80% (otherwise the summary is systematically incomplete). If either gate fails, do not ship.

Self-assessment checklist:

• Did you define success in plain English before defining measurement?
• Did you include a hallucination-specific metric (not just "accuracy")?
• Did you include at least one human evaluation component?
• Did you define a minimum acceptable score as a shipping gate, not just "as high as possible"?

The competitive framing is a trap. Competitors' demos do not tell you what's in their production systems, what their failure rates are, or how much engineering debt they've taken on. The graveyard of "full agent" launches that quietly regressed to simpler architectures 6 months later is large.

Strong response structure:

"I take the competitive concern seriously; we can't be 6 months behind on capabilities. Here's why I don't think starting with an agent solves it, and what I'd propose instead.

Starting with an agent that isn't ready costs us more time, not less. If we build a full agent now, ship it with 65% task success rate, and watch quality complaints accumulate for 3 months, we lose 3 months fixing what we should have proven first. That's the Klarna scenario.

The Augmented LLM phase isn't a delay; it's a qualification gate. We'll know in 3–4 weeks whether the simpler system is sufficient or whether we genuinely need agent-level capability. If we need the agent, we start the build with real evaluation data that tells us exactly what the agent needs to handle. That's faster than building blind.

What I can commit to: we'll have an evaluation result in 3 weeks that tells us definitively whether we're building a workflow, an agent, or expanding the current RAG system. The competitor who ships an agent without that evidence is the competitor we'll beat in 6 months when they're firefighting quality issues."

What makes this strong:

• Doesn't dismiss the competitive concern
• Reframes "starting simple" as speed, not caution
• Names the specific risk of skipping the ladder (Klarna pattern)
• Commits to a concrete timeline for the evaluation gate

"We'll know it when we see it" means the system cannot be improved, because improvement requires measurement. The first task is establishing measurement, not deciding what to do with the agent.

Week 1: Build the evaluation dataset

Step 1: Pull the last 90 days of agent inputs and outputs. You have raw data.

Step 2: Sample 100 input-output pairs stratified by document type (if there is variation) and date (to check for drift).

Step 3: Have two domain experts rate each output on three dimensions (1–5 scale):

• Completeness: Does the summary cover the important feedback themes?
• Accuracy: Does anything in the summary misrepresent the source feedback?
• Action item quality: Are the action items specific, actionable, and tied to real feedback?

Step 4: Compute inter-rater agreement (Cohen's kappa). If agreement is below 0.6, the rating criteria are ambiguous; refine the rubric before proceeding.

Week 2: Build the automated oracle

Use the human-rated 100 examples as a calibration set. Build an LLM-as-judge that scores new summaries on the same three dimensions. Validate: does the LLM-as-judge agree with your human raters at kappa ≥ 0.7? If yes, you have a scalable oracle.

The decision:

Run the automated oracle over the last 30 days of production output. If average scores are ≥ 3.5/5.0 on all three dimensions with hallucination rate < 5%, keep the agent. If scores are below that threshold on any dimension, you need to fix it or replace it. The $2,400/month is only worth it if the agent produces reliable output.

Self-assessment checklist:

• Did you propose building evaluation data from existing production logs (not starting from zero)?
• Did you include human raters as a calibration step before automating?
• Did you check inter-rater agreement (a signal that your criteria are well-defined)?
• Did you connect the oracle score back to the keep/fix/replace decision?