Wedge, ICP, and Architecture: exercises

The mandatory review approach (PM's recommendation) is the wrong answer. It eliminates the 22-minute vs. 90-minute time saving that is the product's core value proposition. If attorneys must review everything, the agent becomes a drafting assistant, not a reliable executor, and the pricing model, sales story, and ROI calculation all break.

The confidence-scoring approach (engineering lead's recommendation) is closer, but incomplete. Confidence scores on individual clauses are useful; the harder problem is that attorneys will stop reading the "low confidence" flags after two weeks if they are too frequent, and will miss them if they are too infrequent. This is a known failure mode of warning systems: they get ignored.

The right answer has three components:

1. Clause-level mandatory review for high-risk clause types: Not all clauses carry equal risk. Indemnification, limitation of liability, IP ownership, and governing law are the clauses where a single error has material consequences. Flag these for mandatory review regardless of model confidence. This is a small, explicit set (probably 8–12 clause types) that can be maintained as a product policy.

2. Session-level audit trail: Every contract review produces a record of which clauses were reviewed, which were accepted without modification, and which were changed. This gives the GC accountability visibility without requiring sequential review of every clause.

3. Deal-size threshold for mandatory human review: For deals above a material threshold (e.g., $10M), require human review of the full redline before submission. This is not product limitation; it is defensible product policy that enterprise legal teams expect.

The framing for the GC: "We're adding mandatory review for the 8 clause types that carry the highest legal risk, and requiring full human review on deals above $10M. For everything else, the product continues to deliver the time savings you're using. This is the right split, not 'review everything' and not 'trust everything.'"

Self-assessment checklist:

• Did you reject mandatory full review because of its impact on product value?
• Did you propose a solution scoped to the specific risk class (high-stakes clause types), not a blanket safeguard?
• Did you include a deal-size threshold as a clear, defensible policy?

Do not refactor 6 weeks before launch. This is the right answer, and it needs to be communicated clearly.

The engineer's underlying concern is legitimate: if approval rules are complex and growing, they will be hard to maintain inside the orchestrating agent. That concern deserves a response, just not the response of a pre-launch architecture refactor.

Why to hold: The current architecture is tested and working. "Testability" of an isolated approval module is a valid long-term concern, not a launch blocker. A 6-week refactor 6 weeks before launch introduces new failure modes, new integration surfaces, and resets test coverage. The probability of shipping a regression is high. The benefit (better long-term maintainability) does not pay off before launch.

What to do instead:

1. Document the approval rule complexity as technical debt, with a specific plan to refactor post-launch when you have production data on which rules are actually exercised.
2. Add a test for every approval rule to the existing test suite now; this addresses the "testability" concern within the current architecture.
3. Set a post-launch review date (6 weeks post-launch) to evaluate whether the approval logic has grown complex enough to justify the refactor. If it has (and you now have production data to inform the design) execute then.

The rule: architectural changes this close to launch require a defect with a specific, measurable impact, not a concern about future maintainability. Future maintainability is a valid reason to plan a refactor; it is not a reason to delay a launch.

Self-assessment checklist:

• Did you explicitly reject the pre-launch refactor rather than compromise (e.g., partial refactor)?
• Did you address the underlying concern (approval rule complexity) without the refactor?
• Did you set a specific post-launch review date with clear criteria?

The right prioritization framework: rank by (a) integration reuse vs. net-new integration work, (b) domain corpus transferability, (c) reference customer value.

Company A (Zendesk + Shopify): Zendesk integration already exists. The only new integration is Shopify, one new tool rather than a new workflow environment. The agent's Zendesk-specific context (ticket schemas, routing logic, escalation paths) transfers fully. The billing dispute workflow in e-commerce is structurally similar to SaaS billing disputes. Priority: highest. This is ICP expansion with minimal integration cost and maximum corpus reuse.

Company B (Freshdesk + Stripe): Stripe integration exists. The new integration is Freshdesk, a different ticketing system with different data schemas, different routing logic, and different API contracts. This is a new system environment, not an adjacent expansion. The eval dataset built for Zendesk does not transfer. Priority: second, but with a longer timeline. Assign 6-8 weeks to build and validate the Freshdesk integration properly before selling.

Company C (Salesforce Service Cloud + Stripe): Stripe exists, Salesforce Service Cloud does not. Salesforce Service Cloud is architecturally different from Zendesk, it is a CRM-native support environment, not a purpose-built helpdesk. The integration work is significantly more complex, and the workflow context (CRM-linked cases vs. ticketed support) is meaningfully different. Priority: third. This is a new ICP segment, not just a new integration. Treat it as a separate product expansion, not an ICP adjacent.

What to tell the board: "We're pursuing Company A immediately (6-week integration), Company B in Q3 (new helpdesk integration with full validation), and Company C as a 2026 product line expansion. This is orderly ICP expansion that protects our resolution rate rather than scattering engineering across three parallel integrations simultaneously."

Self-assessment checklist:

• Did you use integration reuse as the primary ranking criterion?
• Did you distinguish between "adjacent expansion" (new tool, same workflow) and "new ICP segment" (different workflow context)?
• Did you propose a timeline that sequences rather than parallelizes the integrations?

Immediate remediation:

1. Update the tool schema to handle both the old string format and the new object format with a compatibility shim, returning just status from the nested object while the fix is in progress.
2. Deploy within 30 minutes. Every minute at 18.7% error rate is customer-visible.
3. Alert the CRM provider's developer relations team; schema changes without versioning or deprecation notice are a partner SLA violation.

Root cause: The Any type on the account_status field meant that schema changes from the CRM provider propagated silently into agent behavior rather than failing loudly at the tool boundary. The agent received a dict where it expected a string, passed the dict to its reasoning logic, and produced incorrect outputs or hard errors.

Architectural fix: Every external API field in every tool contract needs a typed schema. No Any types on fields that influence agent reasoning. The tool layer's job is to be the last line of defense against external API changes, it should fail loudly with a clear error message when it detects a schema mismatch, rather than passing unexpected data upstream.

Process fix: Add provider API changelog monitoring as a standard operational task. Subscribe to changelog feeds for every external API. For any schema change affecting tool contracts, require a tool schema update and regression test run before the agent sees production traffic.

The broader lesson: tool contracts are not documentation. They are runtime assertions. Treating them as such, with strict validation and explicit failure modes, is what separates Layer 1 done well from Layer 1 done poorly.

This is a classic context drift failure. As the session extends to 120,000 tokens, the original instructions, established facts, and critical constraints are being pushed toward the beginning of the context window. The model's attention degrades for information far from the current position, the "Lost in the Middle" phenomenon, causing the agent to produce outputs inconsistent with facts it "knows" but cannot effectively attend to.

Diagnosis confirmation: Check whether the contradictions are concentrated in specific content types, facts from early sources vs. facts from recent sources. If early-source facts are contradicted by late-session outputs (but not vice versa), this confirms context drift rather than a model reasoning failure.

Fix, three components:

1. Context compaction checkpoint at 60,000 tokens: At the halfway point, the agent pauses to produce a structured "facts established so far" summary: key findings, sourced claims, and constraints on the remainder of the report. This summary is pinned to the front of the context window. The full detail of earlier sections is summarized rather than preserved in full.

2. Fact assertion index: Rather than relying on the model to recall established facts from earlier in the session, maintain an explicit key-value store of important facts (company: revenue, source: URL, citation number: N) that is injected into the context window at each new section's start. The agent references the index rather than relying on attending to content 80,000 tokens earlier.

3. Section-level consistency check: After each section is written, run a lightweight check (separate inference call, small model) that compares claims in the new section against claims in the fact assertion index. Flag contradictions before the analyst sees the output.

The broader principle: for any agent session longer than 30 minutes or 50,000 tokens, assume that context drift is an active risk and design explicit management for it, not as a future enhancement, but as a Day 1 architectural requirement.

The engineer's intuition is partially right, experienced product people do develop a sense for good wedges. The rubric's value is not replacing that intuition; it is three other things.

First, it surfaces the specific failing dimension. Intuition produces "this feels wrong" without a diagnosis. The rubric produces "this fails on verifiability because our success criterion is ambiguous to multiple stakeholders." The latter is actionable; the former leads to a stalled decision or a gut-feel override that ignores the real risk.

Second, it creates a shared language for the team. When a PM, an engineer, and a sales lead are all looking at the same 7-dimension score, they are debating the same dimensions. Without the rubric, each person is evaluating the wedge against different implicit criteria. The rubric makes the disagreement legible.

Third, it catches the experienced person's blind spots. Most production failures that a rubric would have caught are in verifiability and reversibility, not in the dimensions experienced product people think about first (pain, frequency). The rubric forces evaluation of the dimensions that are genuinely hardest to predict from demos and early interviews.

The right framing: "The rubric is a forcing function for articulating the intuition we already have, not a replacement for it. If your intuition says 'good wedge,' the rubric should mostly confirm it, and the exceptions are exactly the ones we need to catch before we've built for three months."

The "add evaluation after launch" argument sounds reasonable but is structurally wrong. Here's why, and what minimum viable eval actually looks like.

Why you can't add eval after launch: Without an eval baseline established before launch, you have no measurement of whether your first production prompt and model combination is performing correctly. When the model provider releases an update in month 2, you have no way to detect whether agent behavior changed. When you make your first prompt change, you cannot tell if it improved or degraded performance. Eval is not just a quality tool, it is the measurement infrastructure for all subsequent improvement work.

What minimum viable eval looks like in 6 weeks:

• 50–100 golden examples: real inputs with expected outputs, labeled by a domain expert. This takes 2–3 days for a domain expert and 1–2 days of engineering to automate the comparison.
• A weekly regression run: run the golden examples against the current model and prompt configuration every week. Flag any degradation vs. the launch baseline.
• Error rate tracking in production: log every session that ends in a human escalation or an explicit failure. This is the data source for the next round of golden examples.

That is the minimum viable eval infrastructure. It costs 1 week of engineering time and 2 days of domain expert time. The cost of not having it: you ship without a baseline, make prompt changes you can't measure, miss model drift for months, and discover your quality has degraded when customers start complaining, not when your monitoring detects it.

The right answer to the CTO: "Eval on day one is 1 week of work. Retrofitting eval after 3 months of production changes and model updates is 4–8 weeks of work to reconstruct what baseline looked like. We're not trading speed for quality, we're avoiding a harder problem later."

The integrations themselves are not the moat. The investor is right that integrations can be replicated. The moat is what builds on top of the integrations once they are live.

The actual moat layers, in order of defensibility:

1. Workflow position: The agent sits in the AP workflow before the human reviewer. Every invoice passes through the product. This is the highest-value position in the workflow, and the hardest to displace once a customer has reorganized their AP process around it.

2. Vendor-specific matching intelligence: After 12 months in production at a single customer, the agent has seen thousands of invoices from that customer's specific vendor base. It has learned vendor-specific quirks: invoice numbering conventions, line item naming variations, common discrepancy patterns. This is not general ML; it is customer-specific knowledge that a new competitor cannot have on day one.

3. Evaluation corpus: Every flagged discrepancy and every human override is a labeled training example. After 12 months, the agent has a customer-specific eval corpus that enables fine-tuning for that customer's vendor base. A competitor starting from scratch has the same base model capability but zero of this labeled data.

4. Approval workflow integration depth: The deeper the integration into the ERP's approval workflow (GL coding, multi-level approval routing, exception escalation), the more the product becomes load-bearing infrastructure. Replacing it requires re-integrating the approval workflow, not just swapping the extraction model.

The investor framing: "Yes, a competitor can build the same integrations. They will be 12–18 months behind us in vendor-specific intelligence and approval workflow depth for our existing customers. And our existing customers have reorganized their AP process around our product, switching has a real operational cost, not just a contract cost."

An eval framework for a coding agent needs to measure at three levels: execution correctness, workflow reliability, and production quality.

Level 1, Execution correctness (automated):

• Test pass rate: does the generated code pass the existing test suite? This is the primary automated signal.
• Lint and type check pass rate: does the code conform to the repository's style and type constraints?
• Build success rate: does the code compile/build without errors? These three are fast, objective, and automated. They form the minimum bar, code that fails tests, lint, or build is categorically unacceptable regardless of other quality signals.

Level 2, Workflow reliability (semi-automated):

• PR acceptance rate: what percentage of generated PRs are merged without modification? Requires a 2-week observation window per issue.
• Modification rate: for PRs that are merged, what percentage required significant changes before merge?
• Abandonment rate: what percentage of generated PRs are closed without merge after human review? These signals require production deployment and time to measure, but they are the ground truth for "did the agent do the job?"

Level 3, Production quality (human eval + golden set):

• 50-issue golden set: a set of previously-solved issues with known correct solutions, used to measure consistency against a baseline.
• Weekly regression run: run the golden set against the current model/prompt configuration; flag deviations from the baseline.
• Issue complexity stratification: separate metrics for "simple fix" issues vs. "complex refactor" issues, the agent's performance profile is likely different, and conflating them masks both strengths and weaknesses.

The metric to report to stakeholders: PR acceptance rate (% of generated PRs merged without modification) is the most meaningful single number. It combines execution correctness, workflow reliability, and code quality into one auditable metric that engineers understand and trust.

The CTO's skepticism is probably right. A resolution rate plateau after two rounds of prompt optimization is almost never primarily a model problem. Here is the diagnostic.

Step 1, Segment the 26% non-resolutions by failure category. Do not guess, pull the logs. Typical failure categories: (a) the agent attempted and produced an incorrect answer the user rejected; (b) the agent correctly identified it couldn't help and escalated; (c) the agent produced no response or errored. Each category has a different root cause.

Step 2, For category (a), drill into the failure type. Is the agent failing on factual questions it should know (documentation coverage gap), or on reasoning questions that require multi-step judgment (model capability ceiling), or on context-dependent questions that require data it cannot access (tool/data access gap)? These are three different problems with three different solutions.

Step 3, Apply the appropriate fix:

• Documentation coverage gap → fix the documentation, not the model. Resolution rate will improve without changing anything about the agent.
• Tool/data access gap → add the missing integration. The model cannot compensate for context it cannot see.
• Model capability ceiling → this is the one case where a model switch is warranted. But it should be demonstrated on a specific, isolated failure category, not assumed from aggregate plateau behavior.

The rule: Never change the model before exhausting the documentation and integration explanations. Model switches are expensive (re-validation, potential behavioral drift in previously working cases, integration testing) and they address the narrowest class of resolution rate plateaus. The consultant's recommendation to switch models without a failure category analysis is working backwards from a hypothesis rather than from evidence.