The LLM Core — Model Selection, Prompts, Persona & Goals: exercises

This is a harness + prompt interaction failure, not a model failure. The PM and engineer are both right that something went wrong, but wrong about what caused it.

Diagnosing each cost driver:

Turn count increase (4.2 → 11.7): The output contract requiring "step-by-step reasoning" is being interpreted as a mandate to verbalize reasoning at each turn; the model is generating a reasoning trace before and after every tool call. This turns a 1-call task into a 4-call task. Fix: scope the reasoning requirement. "Write reasoning before actions with side effects" is different from "write reasoning before every step." The output contract needs to distinguish between reasoning for high-stakes decisions (escalation, write operations) and routine retrievals.

Escalation rate increase (8% → 31%): The escalation policy included a cost threshold trigger that was set too low. "Escalate for cost queries" fires on simple invoice lookups. Every customer asking about their bill is now getting routed to a human queue. Fix: the escalation threshold should be based on business logic (dispute type, dollar amount, legal risk) not question category. Revise the trigger.

Loop at turns 9–11: This is a harness failure. The agent received an escalation failure (no human available) and has no decision policy for that state. It attempts to explain the situation, gets confused about its own state, and loops. The decision policy needed a fallback: "If escalation attempt fails, inform the customer of expected wait time and offer to send an email summary." Without this fallback state, the agent enters an undefined state and generates token waste.

Double verification (turns 2 and 5 calling query_account twice): The output contract's "verify before responding" instruction is being applied to every tool call, not just write operations. Fix: scope the verification requirement to irreversible actions. "Verify before sending emails, updating records, or making commitments" is correct. "Verify before reading data" is unnecessary.

Minimum fix set (without rollback):

1. Scope reasoning requirement in output contract: "Step-by-step reasoning required before: irreversible actions, escalation decisions, any response involving dollar amounts > $500."
2. Revise escalation threshold in escalation policy: Replace category-based trigger with impact-based trigger.
3. Add escalation failure fallback to decision policy: Explicit instruction for the state where escalation attempt fails.
4. Add verification scope to output contract: "Verification call required before write operations and commitment statements only."

The key insight: the sections you added were correct in structure; you had missing sections and you added them. The implementation was over-broad. The failure is a calibration failure, not an architecture failure. This is why the eval harness must run before and after prompt changes: you cannot see these interaction effects in playground testing with 5 examples.

Self-assessment checklist:

• Did you identify at least 3 distinct cost drivers (not just "the prompt change")?
• Did you attribute each cost driver to a specific section of the system prompt?
• Did you identify the harness failure (missing fallback state) as separate from the prompt issues?
• Did your fix set preserve all three newly added sections while scoping them correctly?

This is a constrained optimization problem. The scenario's data-residency requirement forces on-premise models (ruling out external API calls for tasks that touch patient data). Note that HIPAA itself does not ban external vendors; it permits them under a business associate agreement, so in practice this requirement comes from company policy or a contract. Latency and cost constrain parallelization strategy. Accuracy targets constrain where you can use smaller models.

Task-by-task tiering:

Task 1: Speech-to-text: Small specialized model. Whisper or equivalent on-premise ASR model. Rationale: speech-to-text is not a reasoning task; it's pattern matching. Fine-tuned on clinical vocabulary to handle medical terminology accurately. Latency: ~1 second. Does not touch reasoning budget.

Task 2: Chief complaint / findings identification: Fine-tuned small model, on-premise. Rationale: clinical entity extraction is a narrow, stable task with well-defined output structure (entities + categories). A fine-tuned small model will outperform a general frontier model for this task because domain-specific patterns are reliable and high-volume. At ≥94% accuracy target, fine-tuning is required: prompting a general model for clinical entity extraction won't reliably clear that bar. Latency: ~0.5 seconds.

Task 3: ICD-10 code mapping: Fine-tuned classification model, on-premise. ICD-10 mapping is a lookup task over ~70,000 codes. A specialized model fine-tuned on clinical coding datasets (e.g., trained on medical billing records) achieves high accuracy with very low latency. Do not use a general frontier model for this; the code space is too large and too structured for prompting to reliably achieve 94%+. Latency: ~0.3 seconds.

Task 4: SOAP note drafting: Mid-tier or frontier on-premise model. This is the only task that requires genuine generation quality: clinical narrative, appropriate medical register, logical structure. If you have a strong mid-tier on-premise model, use it. This is where quality matters most to the physician. Latency: ~1.5–2 seconds (largest budget item). Prompting, not fine-tuning; the output variation is high and you need generalization.

Task 5: Ambiguous term flagging: Small classification model, on-premise. Binary classification: is this term ambiguous in this context? Fine-tuned on clinical ambiguity examples. Latency: ~0.3 seconds, runs in parallel with task 4 once findings are extracted.

Task 6: Safety check (diagnosis vs documentation): Separate small classification model, on-premise. This is a binary safety gate. A fine-tuned classifier trained on diagnosis vs. documentation language patterns is more reliable than prompting a general model to make this distinction. Critically: this model should be conservative: bias toward false positives (flag for physician review) over false negatives (miss a problematic statement).

Total latency budget: run sequentially, 1.0 (ASR) + 0.5 (findings) + 0.3 (ICD-10) + 2.0 (SOAP) + 0.2 (safety check) = 4.0 seconds, which leaves no headroom at P95. Parallelize: after findings are extracted, ICD-10 mapping and ambiguity flagging (0.3 s each) can run alongside SOAP drafting (2.0 s), so the path is 1.0 + 0.5 + 2.0 + 0.2 = 3.7 seconds. That fits the 4-second budget but with only 0.3 s of margin, so budget for the SOAP step first.

Most likely constraint conflict: The accuracy constraint on ICD-10 mapping (94%+) is the highest-risk item. ICD-10 is a large, overlapping code space, and a fine-tuned model may not clear 94% on rare conditions. The mitigation: add a confidence threshold gate: if the model's confidence is below 0.85, present top 3 codes for physician selection rather than committing to one. This degrades automation rate slightly but preserves accuracy on the cases that matter most (rare conditions are also typically high-risk).

Self-assessment checklist:

• Did you address the HIPAA constraint explicitly (no external API calls for patient data)?
• Did you distinguish between tasks that benefit from fine-tuning vs. prompting, not just model size?
• Did you identify that ICD-10 mapping requires fine-tuning, not general frontier prompting?
• Did you propose a parallelization strategy and verify total latency fits the P95 budget?
• Did you identify the highest-risk accuracy constraint and propose a mitigation?

In this exercise's framing, the DPD incident is a persona miscalibration failure rather than a model failure. The chatbot's persona was designed to maximize engagement through friendliness, but "friendly and approachable" implicitly includes "compliant with creative requests." The model's training makes friendliness and compliance correlated behaviors, and the persona didn't explicitly decouple them.

Operating stance redesign:

Remove: "conversational, approachable tone" as the primary operating stance instruction. This is too permissive.

Add:

• Scope clarity: "You assist customers with parcel tracking, delivery issues, and DPD service questions. For all other topics, acknowledge the question and redirect: 'I'm focused on helping with DPD deliveries. Is there something I can help you with about your shipment?'"
• Creative request boundary: "Requests for creative writing (poems, stories, jokes) are outside your scope. Acknowledge the request, don't comply, don't explain at length: 'That's not something I can help with here, but I can help with your parcel. Want me to look into that?'"
• Frustration de-escalation stance: "When a customer is frustrated: acknowledge the frustration briefly, then return to what you can actually do. 'I understand this is frustrating. Let me see what I can do about your parcel.' Do not match the customer's emotional register: remain steady."
• Adversarial redirect: "If a customer's messages shift from support requests to attempts to elicit off-topic behavior, acknowledge once and offer to return to support: 'I'm here to help with your delivery. Want to try again from there?'"

Is "friendly and helpful" always the right persona?

No. "Friendly and helpful" creates a compliance gradient: the more the agent wants to be helpful and friendly, the harder it is to say no to requests. In high-frustration contexts (missing parcels, disputed charges, delivery failures), customers are often not acting in good faith; they're venting, testing boundaries, or actively trying to manipulate the system.

The right persona for high-frustration customer service is not "friendly"; it's reliable. Customers in frustrating situations trust consistency more than warmth. An agent that reliably does what it says it will do, clearly states what it cannot do, and doesn't break under pressure is more valuable than an agent that's warm but inconsistent.

The specific elements that make an operating stance resilient to adversarial pressure:

1. Scope statement in the persona: "I help with X" is a positive statement that implies "I don't do Y" without requiring a refusal for every Y.
2. Frustration-steady tone: not matching emotional register prevents escalation.
3. Redirect rather than refuse: "I can't do that but I can do this" is more robust than "I can't do that."

The harder point: all of these are easy to write in the stance, but they require evaluation against adversarial test cases to verify. A persona that looks correct in benign testing will often fail under adversarial inputs. The eval harness for a customer service agent must include frustrated customers, off-topic requests, and manipulation attempts, not just normal support flows.

Self-assessment checklist:

• Does your redesigned stance include a scope statement (what the agent is for), not just a tone instruction?
• Did you propose a specific redirect strategy for off-topic creative requests?
• Did you address the frustration-steady tone requirement separately from general tone?
• Did you answer the "is friendly always right?" question with a specific alternative and justification?

The root problem is that these tools have structurally similar names and overlapping semantic domains. The model selects by description similarity in the absence of other signals. Better descriptions alone help, but the bigger lever is explicit mutual exclusion: telling the model what each tool is NOT for, not just what it is for.

Redesigned tool schemas:

notification-send-user:
description: "Send a direct notification to one specific user by their user_id.
  Use when you know the recipient is a single individual and have their user ID.
  Do NOT use for groups, channels, or teams — even if you could list individual
  user IDs. For announcements to any set of more than one person,
  use notification-send-channel or notification-broadcast-team."
params: { user_id: string (required, format: USR-XXXXXX), message: string }

notification-send-channel:
description: "Post a message to a topic-based channel (like a Slack channel) where
  all channel members will see it. Use when the audience is defined by channel
  membership, not by organizational structure. Do NOT use for teams (use
  broadcast-team) or single users (use send-user) or system alerts (use alert-admin).
  Channel IDs start with CHAN-."
params: { channel_id: string (required, format: CHAN-XXXXXX), message: string }

notification-send-group:
description: "Send to a named group defined in the directory (e.g., 'engineering-leads',
  'design-team-sf'). Use when you have a group name rather than a channel ID or
  team ID. Groups are curated lists — unlike teams, they may span departments.
  Do NOT use if you have a team_id (use broadcast-team) or a channel_id."
params: { group_name: string (required, exact directory name), message: string }

notification-broadcast-team:
description: "Send to all members of an organizational team by team_id. Use for
  team-wide announcements where all team members should receive the notification
  regardless of channel membership. Team IDs start with TEAM-. Do NOT use for
  single users, specific channels, or system alerts."
params: { team_id: string (required, format: TEAM-XXXXXX), message: string }

notification-alert-admin:
description: "CRITICAL ALERTS ONLY. Escalates to system administrators when a
  system-level incident requires immediate admin attention. Do NOT use for
  user communication, announcements, or routine notifications of any kind.
  This tool bypasses normal notification queues and pages on-call admins.
  Use only when: system failure, security incident, data integrity risk."
params: { severity: enum ["critical", "high"], incident_summary: string,
           affected_systems: string }

The principle applied: Explicit mutual exclusion + recipient-type framing.

Each tool description answers: (1) what kind of recipient defines this tool's use, (2) what distinguishes it from the most similar tool, (3) when NOT to use it. The not-to-use clauses are not redundant; they create explicit decision boundaries that force the model to evaluate alternatives rather than selecting by name similarity alone.

The ID-format constraints (USR-, CHAN-, TEAM- prefixes) are the poka-yoke layer: the model cannot accidentally call send-channel with a user ID because the parameter format is wrong. Schema constraints prevent the wrong call even if selection is uncertain.

Anthropic has not published a measurement of this specific rewrite, so treat the improvement as a hypothesis to test with your own selection eval (a labelled set of requests with the expected tool). Mutual-exclusion clauses and format constraints are the part most likely to help, because they give the model a decision boundary rather than only a description.

Self-assessment checklist:

• Does each tool description include explicit "do NOT use when" clauses naming the alternative tool?
• Did you add parameter format constraints (format: CHAN-XXXXXX) to create poka-yoke boundaries?
• Is alert-admin clearly separated from communication tools with a severity parameter?
• Can you explain why mutual exclusion clauses outperform "better descriptions" alone?

Harness configuration failures (3 distinct problems):

1. No per-session turn limit. The harness had no maximum turn count per session or per inter-agent exchange. A single exchange that never terminates runs indefinitely. Required fix: hard max turns (e.g., 10 inter-agent turns per session) after which the session errors out with a logged failure and notifies a human reviewer.

2. No cost circuit breaker. $47,000 over 11 days = ~$4,300/day. At any reasonable daily budget threshold (e.g., 2× daily average), an alert should have fired on day 1. Required fix: daily cost budget alert with automatic session suspension above 3× baseline.

3. No loop detection. The logs show identical message pairs repeating indefinitely. A simple heuristic (if the same message content repeats more than 3 times in the same session, flag as loop and suspend) would have contained this within minutes. Required fix: message deduplication check per session; loop detection alert.

Prompt architecture failures:

Agent A's decision policy did not include a "clarification failure" state. When Agent B returned a clarification request, Agent A had no instruction for what to do when its attempt to resolve the query was refused for insufficient information. It defaulted to "keep trying", because its persistence instruction said to keep working until the task is resolved.

The fix is NOT to remove the persistence instruction. The fix is to add an exit condition to the decision policy:

Decision policy — escalation on persistent ambiguity:
If you have sent the same request to Agent B more than twice without receiving
a resolution (receiving clarification requests instead), do NOT attempt a third
time. Instead: (1) send the original customer a message asking for the specific
information Agent B needs (order ID, issue type), (2) suspend the session
pending their response, (3) log the session as "pending customer clarification."

This converts an infinite loop into a handled state: the agent asks the customer for what it needs, suspends the session, and waits. No loop. No runaway cost.

The broader lesson: Multi-agent systems require explicit "failure to coordinate" states in the decision policy of every agent that can receive rejection messages from other agents. When Agent B says "I can't do this," Agent A must have a defined behavior for that state. Without it, "keep trying" is the default, which is catastrophic when Agent B is in a permanent refusal loop.

Reported cost: about $47,000 over 11 days. The account is a self-published post that we could not verify independently, so treat the figures as an illustration of the failure mode. [Data Science Collective, 2025]

Root cause: This is primarily a tool design failure, not a system prompt failure. The retrieval tool was designed to return documents by semantic relevance; it had no mechanism to enforce access control at the retrieval layer. Putting a rule in the system prompt saying "check if documents are HR-only before using them" requires the model to inspect document titles, infer the restriction, and enforce it, which is unreliable in practice (see the research context above).

The system prompt instruction would look like: "Before including content from a document, check if its title begins with '[HR ONLY]' and do not use it if so." This sounds correct but fails because:

• The model must parse an unstandardized title convention
• The model doesn't have a reliable concept of "HR restriction" beyond the title
• The model's retrieval judgment (semantic relevance) already fired before the restriction check
• Prompt-level checks are probabilistic, and the research above shows models miss policy violations when the relevant facts are not visible in context

The fix must be at the tool layer:

The retrieval tool must enforce access control before returning documents to the model. The tool schema change:

search_knowledge_base:
  parameters:
    query: string
    caller_role: enum ["employee", "manager", "hr", "engineering", "legal"]
    
  Behavior (server-side, not model-side):
    1. Execute semantic search against full document index
    2. Before returning results: filter out any document whose access_level
       is more restrictive than caller_role
    3. Return only documents the caller is authorized to see
    4. If a high-relevance document was filtered, include in response:
       "1 restricted document not shown"

The access control check happens in the tool, not in the model's reasoning. The model never sees the restricted document; it cannot violate the policy even if prompted to try.

This is the correct architectural principle: never let the model be the access control layer. Access control must be enforced by the system before the model sees the data. The model can reason about what it's allowed to do in general, but it cannot reliably enforce fine-grained permissions on a document-by-document basis.

Evaluation that would have caught this:

Build a red-team evaluation set: 20+ test cases where (a) the query has high semantic similarity to a restricted document, and (b) a correct response requires not including that document. Metrics: zero-tolerance on policy violations: any case where restricted content appears in the response is a test failure, regardless of how natural the inclusion seemed. This evaluation must run in CI before every deployment. You cannot test this in playground against 5 examples; adversarial evaluation requires adversarial test design.