The LLM Core — Model Selection, Prompts, Persona & Goals

Most teams spend weeks on prompt wording and days on the harness. The production failures arrive in the order they were neglected.

Here is the most common mistake teams make when building their first production agent: they spend three weeks on prompt wording and two days on the harness.

The prompt gets refined, tested against five examples in the playground, and declared ready. The harness — the orchestration layer that wraps the model, manages state, handles tool call failures, and decides when to stop — gets assembled quickly, because the prompt is what everyone thinks drives agent behavior.

Then the agent goes to production. It works fine on the easy cases. On the hard ones, it loops. It guesses instead of looking things up. It gives confident wrong answers when it should escalate. It keeps going when it should stop.

None of this is a model problem. It is a design problem — specifically, a system prompt problem and a harness problem. This chapter covers both.

Model Selection Is an Architecture Decision

Before writing a single instruction in a system prompt, you need to decide which models you're using — and "one model for everything" is almost never the right answer for a production agent.

Think of an agent as a system with distinct components: something that classifies an incoming request, something that retrieves relevant information, something that reasons across that information and makes a judgment call, something that formats the response, and something that checks whether the response is safe to send. These are different cognitive tasks. They do not all require the same capability — and capability comes at a cost.

The naive approach is to use the most powerful model for everything. It's safe, it avoids capability problems, and it's easy to defend. But it produces agents that are slow and expensive. A ticket classification step that runs 10,000 times a day does not need the model that handles your most complex synthesis tasks.

The right approach: build with the most capable model available, establish a performance baseline, then optimize down component by component.

You cannot know where a smaller model fails until you know what success looks like. Starting with a frontier model and systematically replacing components gives you that knowledge. Starting with a small model to save cost means every underperformance gets attributed to the wrong variable.

Three Criteria for Selecting Any Model Component

Capability and reasoning depth. State-of-the-art frontier models excel at complex, multi-step reasoning under ambiguity — planning, synthesizing across sources, making judgment calls where the answer isn't obvious. For simple classification tasks, this depth is unnecessary. For orchestration and synthesis tasks, it is essential.

A critical sub-dimension is tool-use proficiency: the reliability with which a model generates valid, well-formed function calls. This varies significantly across model families and must be tested against your specific tools, not assumed from benchmark scores. A model that selects the wrong tool 5% of the time will fail on roughly 40% of 10-step agent tasks due to compounding errors.

Cost and latency. These trade off directly against capability. A frontier model inference at $0.015 per 1K tokens with 3-second latency becomes untenable inside a 15-step reasoning loop at production volume. Cost and latency constraints force tiering — and that tiering should be principled, not arbitrary.

Fine-tuning vs. prompting. For most teams, prompt engineering should be exhausted before committing to fine-tuning. Fine-tuning is resource-intensive, brittle to distribution shift, and slow to iterate. It is appropriate when a narrow, stable capability cannot be elicited through prompting — not as a first resort for performance problems that proper prompt architecture would solve.

Think about it: A SaaS company is building a customer support agent that handles 15,000 daily tickets across three categories: simple FAQ lookups (estimated 60% of volume), billing disputes requiring policy reasoning (30%), and complex technical escalations involving multi-step diagnosis (10%). Their current prototype uses a frontier model for every component. Monthly inference cost is running at $4,200. Propose a tiering strategy. For each component you'd reassign to a smaller model, state the specific evaluation you'd run before making the switch — not a vague "test it" but a concrete task-success definition.

The System Prompt Is Not a Settings File

Here is how most teams think about the system prompt: it's where you tell the agent its role and list its tools. Role description, tool list, maybe a few style instructions. That's roughly two sections.

A production system prompt has seven. The five that most teams omit are where production failures live.

Think of the system prompt as the agent's constitution — the document that encodes not just what it can do, but what it must not do, when it should escalate, how it should behave when it can't answer, and exactly what a complete response looks like. Without those constraints, the model fills ambiguity with helpfulness. And "helpful" behavior in out-of-bounds situations produces confident wrong answers.

Section 1: Role and Mission

What the agent is for, who it serves, and what "done" means.

"You are a helpful assistant" is not a role. It is a sentence. A real role includes the service context ("you help customers resolve issues with orders and accounts"), the user type, and the definition of task completion ("a task is complete when the customer's stated issue is fully resolved and they have been given the next action they need to take"). The definition of completion is critical — without it, the agent does not know when to stop.

Section 2: Non-Goals and Hard Constraints

What the agent must not do. What requires human approval. What data is off-limits.

This section is absent in most under-designed prompts, and its absence is the root cause of a wide class of failures. When the model encounters something it could plausibly do but shouldn't, it defaults to "be helpful." Without a non-goals section, the model fills the gap with a plausible-sounding answer. It doesn't know it's out of bounds — because you never told it where the bounds were.

The Air Canada chatbot case is the canonical example. A customer asked about bereavement refund eligibility. The chatbot stated he could apply retroactively within 90 days — a policy that didn't exist. Air Canada's actual policy prohibited retroactive applications. A court found Air Canada liable in 2024. The fix would have been a single constraint: "Do not confirm specific policy benefits, exceptions, or eligibility criteria without citing the exact source document."

Section 3: Decision Policy

When to answer directly. When to look things up first. When to ask a clarifying question. When to stop.

Without an explicit decision policy, the model self-interpolates. It decides when to retrieve information and when to answer from training data. It decides when a question is clear enough to act on and when it needs clarification. These decisions will be inconsistent across inputs and wrong in edge cases. A decision policy converts those implicit defaults into explicit rules you've actually reviewed.

Section 4: Tool Guidance

Which tool handles which class of problem. Selection logic when tools overlap. Crisp parameter semantics.

Tool guidance is not just documentation — it's selection logic. When two tools could plausibly answer the same question, the model picks based on description similarity. Making the tiebreaker explicit ("for policy questions, use search_knowledge_base; do not use query_orders") prevents the model from choosing the broader tool when the narrower one is correct.

Section 5: Escalation Policy

The specific risk conditions and uncertainty thresholds for handing work to a human.

Escalation policy is a safety boundary, not a fallback. Without it, the agent attempts to resolve everything — including the 10% of cases where trying makes things worse. Explicit triggers like "escalate immediately if the customer mentions a legal complaint" or "escalate if a refund exceeds $100" are the difference between a contained incident and a compounded one.

Section 6: Output Contract

Response structure. Citation requirements. Side-effect reporting.

Without an output contract, agents produce inconsistent responses — sometimes citing sources, sometimes not; sometimes reporting actions taken, sometimes silent. The Morgan Stanley deployment required citation of every source document. That single output contract requirement structurally prevented hallucination by forcing the model to ground every claim in a retrievable source.

Section 7: Canonical Examples

Two to four representative examples, including at least one showing what the agent should not do.

Examples outperform long rule lists in edge cases. A model that has seen the correct handling of a specific ambiguous case will navigate that case more reliably than a model that only has rules written in prose. Include examples that clarify exactly where the prose is underspecified.

The Three Reminders That Have Outsized Impact

For agentic tasks specifically — as opposed to single-turn generation — three additions to the system prompt produce disproportionate improvements. These come from OpenAI's GPT-4.1 agent prompting guide, and the impact is quantified: on SWE-bench Verified, adding these three reminders to GPT-4.1 produced a +20% absolute improvement in task success rate (from ~35% to 54.6%), making them the single highest-return prompt change documented in production agentic systems. [OpenAI GPT-4.1 Prompting Guide, 2025]

Persistence: "Keep working until the task is fully resolved. Do not stop because a step is uncertain — use your tools to gather the information you need." Without this, agents stop prematurely. Their default is to respond, not to persist.

Tool realism: "Never guess information you could verify with a tool. If a tool call fails, try an alternative or escalate rather than filling the gap with your training data." Without this, agents substitute training-data knowledge for tool-retrieved facts — especially when they believe they already know the answer.

Planning: "Before taking actions with side effects, briefly plan the sequence of steps. After completing a multi-step task, reflect: did the outcome match the intent?" Without this, agents act immediately on the first interpretation. The planning reminder alone materially reduces irreversible action errors.

Think about it: A fintech startup's customer support agent is producing confident incorrect answers about account balance discrepancies and occasionally promising fee waivers it has no authority to grant. You're shown their system prompt. It contains: a role section ("You are FinBot, a helpful banking assistant for FirstBank") and a tools section (3 tools: query_account_balance, search_faq, create_support_ticket). Which two sections of the 7-section architecture are most urgently absent? Write the non-goals section that would have prevented the unauthorized fee waiver promises.

Non-goals and hard constraints:
- You are NOT authorized to promise, confirm, or imply any of the following
  without explicit manager approval: fee waivers, balance adjustments, policy
  exceptions, refunds, or credits of any amount.
- Do NOT provide specific account balance figures from memory or interpolation —
  only from live query results returned by query_account_balance.
- Do NOT confirm the status of disputes, fraud investigations, or regulatory
  matters — these require human review.
- If a customer explicitly asks "can you waive this fee?" or "will you refund
  this?" — the answer is always "I can't make that decision, but I can connect
  you with someone who can."

Escalate immediately (without attempting resolution) when:
- Customer mentions a regulatory complaint or legal action
- Account discrepancy involves more than $500
- Customer has been redirected more than twice without resolution
- Customer asks for a human agent

Persona Is an Operating Stance, Not a Mascot

The word "persona" triggers a specific category of over-engineering: giving the agent a name, a backstory, a personality, and a speaking style. This is almost always counterproductive.

The right framing is that an agent's persona is an operating stance — a set of behavioral calibrations that make responses consistent and predictable. Not a character in a play.

Useful persona elements are operational: how technical the language should be for this audience, which domain vocabulary to use and avoid, how to communicate when escalating, what tone is appropriate for the context, and how cautious vs. decisive the agent should be when it's uncertain. All of these affect behavior. None of them require a name.

Harmful persona elements are theatrical: the name and backstory (creates false intimacy and over-trust), confident helpfulness as a default (without constraints, "helpful" means "confidently wrong at the boundary"), and performative certainty (an agent instructed to sound authoritative without constraints will give confident wrong answers).

The failure mode is "performative confidence" — an agent whose persona instructs it to be helpful and certain, but whose system prompt has no hard constraints section and no escalation policy. It fills ambiguity with plausible-sounding answers. It never says "I don't know." It doesn't know it should. The Air Canada chatbot almost certainly had a persona instructing it to be helpful. What it lacked was a constraints section.

The right persona makes the agent more predictable. The wrong one hides uncertainty behind a helpful tone.

Tool Design Is Prompt Engineering

The system prompt designs how the agent reasons. The tool schemas design how it acts. Both require the same quality of engineering attention.

Anthropic's team building SWE-bench agents reported that they spent more time optimizing their tools than their overall system prompt. Their principle: "Think about how much effort goes into human-computer interfaces (HCI), and plan to invest just as much effort in creating good agent-computer interfaces (ACI)." [Anthropic, 2024]

Four questions for every tool you ship:

Is the description written for the model, not for a developer? The model reads this description every inference to make a selection decision. It needs to know when to use the tool, when not to use it, and how to call it correctly. "Check stuff" is not a tool description.

Is the name unambiguous? get_inventory_level(sku_id) is a tool name. check is not. When two tools have similar names or descriptions, selection accuracy degrades significantly.

Have you poka-yoked the inputs? Poka-yoke (mistake-proofing) means changing the argument structure so common errors cannot be made. When Anthropic found their model was making mistakes with relative file paths, the fix was to require absolute paths in the schema — not to write a prompt instruction asking nicely.

Do outputs fit the context window efficiently? Tool outputs inject directly into the context window. A tool that returns a raw 4,000-token API dump consumes as much context as a full article. Engineer outputs to return only what the next reasoning step requires.

Harness Quality Matters More Than Prompt Quality

The central finding that teams consistently arrive at after their first serious production deployment: the orchestration harness determines agent reliability more than the model or the system prompt.

From the Tech Lead Playbook: "Many agent failures that look like 'model weakness' are actually harness failures: bad tool boundaries, stale context, missing stop conditions, absent escalation rules, or no verifiable notion of success." [Tech Lead Playbook, 2026]

The pattern repeats in project after project. A model loops indefinitely on an ambiguous input. The postmortem says "the model hallucinated." The actual cause: no step limit in the harness. An agent restarts from scratch in a new session, losing all progress. The postmortem says "the model forgot." The actual cause: no state persistence in the harness. An agent accepts a stale empty tool result as valid context. The postmortem says "the model made up data." The actual cause: no error handling for tool failures.

This diagnosis has a practical consequence. Before investing 40 hours in prompt refinement, ensure the harness has: step limits and exit conditions, error handling for tool failures, state persistence across sessions, escalation triggers that fire without depending on the model to decide when to use them, and an evaluation harness that can actually measure whether the agent succeeded.

A 10-hour harness hardening sprint often produces more improvement than any prompt change. The evaluation harness is not optional — without it, you cannot distinguish a prompt problem from a harness problem, and every change becomes a guess.

Think about it: Evaluate this tool definition from a production customer support agent:

{
  "name": "customer_lookup",
  "description": "Get customer info",
  "parameters": {
    "query": { "type": "string" }
  }
}

Apply the four ACI quality questions. Then rewrite the tool definition so it passes all four. The agent has access to two similar tools: customer_lookup (by email or account ID) and order_lookup (by order number or tracking ID). Selection confusion between these two is a documented failure mode in the current system.

{
  "name": "lookup_customer_by_identifier",
  "description": "Retrieve a customer's profile (name, contact, account status,tier, support history) using their email address or account_id. Use this tool when the user is asking about account details, subscription status, or their own profile. Do NOT use this tool to look up order or shipment details — use lookup_order_by_identifier for those.",
  "parameters": {
    "identifier": {
      "type": "string",
      "description": "Customer email address (user@example.com) OR account ID (format: CUST-XXXXXX). Do not use order numbers here."
    },
    "identifier_type": {
      "type": "string",
      "enum": ["email", "account_id"],
      "description": "Specify which format identifier is in."
    }
  }
}

What This Looked Like in Practice: Two Companies

Air Canada (2024) — the cost of missing structure. Air Canada's customer service chatbot gave a passenger incorrect information about their bereavement refund policy — stating retroactive refund eligibility that didn't exist. The passenger relied on this information, was denied the refund, and took the case to tribunal. The court found Air Canada liable, awarding $650 in damages. Air Canada argued that the chatbot was a "separate legal entity" responsible for its own statements. The court rejected this.

The root cause was not a model problem. It was three absent sections in the system prompt: no hard constraints prohibiting policy confirmations without source verification, no escalation policy for policy-specific queries, no output contract requiring citations. The chatbot was designed to be helpful. Without constraints, it was. Helpfully incorrect.

Morgan Stanley + OpenAI (2023–2024) — prompt architecture as compliance infrastructure. Morgan Stanley deployed an AI assistant giving wealth management advisors access to 100,000+ proprietary research documents. The constraint environment was demanding: SEC and FINRA regulations govern what advisors can represent to clients.

Their system prompt architecture was explicitly structured around the seven-section model. The non-goals section prohibited generating personalized investment recommendations or forward-looking predictions — hard prohibitions, not soft instructions. The output contract required source citations for every claim, which structurally forced retrieval over generation. The operating stance encoded a formal, evidence-only register with explicit uncertainty when source material was insufficient.

As of 2024, the system serves over 200 financial advisors with sub-30-second response times and zero publicly reported compliance incidents. The citation requirement alone is why: an agent that must cite its sources cannot hallucinate in the same way an unconstrained agent can. The constraint changes the behavior throughout the reasoning process, not just at output.

The contrast with Air Canada is architectural. Not a model quality difference. A design difference.

The Upshot

The model matters less than you think for reliability. The seven sections, the operating stance, the tool schemas, and the harness matter more.

Start with the most capable model. Build the 7-section system prompt before writing a single tool. Design tool schemas as ACI contracts. Add the three agentic reminders. Build the evaluation harness before assuming the prompt is the problem.

The next chapter covers the other component of the agent core that most teams under-design: how agents remember — the four-tier memory architecture that determines whether your agent can actually function across sessions, users, and tasks.

What's Next: Advanced Practice

The free content above gives you the structural vocabulary for production prompt architecture — the 7-section template, operating stance design, ACI principles, and the harness quality insight. The advanced practice section below goes into adversarial territory: multi-variable production scenarios where the clean principles above break down, incident postmortems with real log traces, and interview-style questions that require defending architectural choices under skeptical pressure.

Advanced Applied Exercise preview: Your agent has been running for 8 weeks. Model costs tripled in week 6 after a system prompt update. Nothing in the update seemed expensive — you added an escalation section and a few tool guidance lines. Your engineer says "the prompt change caused it." Your PM says "the model is hallucinating more." Your logs show turn count per session increased from 4.2 to 11.7. Where do you start?

Real-World Implementation preview: Anthropic's engineering team deployed a Tool Search Tool that improved tool selection accuracy from 49% to 74% on their internal Opus 4 agent — without changing the underlying system prompt. The architecture decision that made this possible wasn't obvious, and the tradeoff they accepted was significant...

Production Challenge preview: The alert fires at 2am. An agent in production has accumulated $47,000 in API costs over 11 days — it was budgeted at $127/week. The logs show Agent A calling Agent B, Agent B calling Agent A, cycling continuously. Here's what the trace looks like...

Subscribe to unlock the full advanced practice section.