We parse Viridon's corpus with layout-aware parsing rather than naive text extraction, and the distinction matters specifically because of what your documents are: 300+ page proposals with 100+ attachments, ~200-page selection reports, embedded tables and figures, mixed formatting from many authors and source systems. Copying a PDF's text layer and splitting on line breaks scrambles multi-column reading order, flattens tables into meaningless runs of numbers, and silently drops every figure. Instead we render each page, detect its regions with a layout model, recover table structure cell-by-cell, OCR anything without a text layer, and run a vision-language model over figures so that content carried in images becomes searchable text.

The backbone is the Unstructured library, which turns PDF, DOCX, XLSX, and HTML into typed elements — titles, narrative text, tables, images — each carrying its text, metadata, and a bounding box locating it on the page. Three models do the heavy lifting underneath: a layout-detection model (yolox) that finds regions and preserves reading order on complex pages; a table-transformer that recovers row and column structure, including merged cells, and emits each table as addressable HTML; and OCR (optical character recognition — turning page images into machine-readable text) for scanned pages and image regions. We route each document to the right parsing strategy — high-resolution layout-plus-OCR for the messy real-world proposals, a cheaper native-text path for clean digital PDFs, and full-page OCR for scans — rather than forcing a single mode across the corpus.

For complex tables, merged cells and spans are recovered by the table-transformer and preserved all the way through to per-cell records, so a value like "$42M" stays bound to its row ("Phase 2 capex") and column ("2027 estimate"). Nested and irregular tables that defeat structure inference degrade gracefully: we retry without forcing a grid and keep the table as both an image and a VLM-written description, so content is never lost even when the exact grid can't be recovered. Multi-page tables are the one piece that's net-new for Viridon — because we split documents per page for throughput, a table spanning two pages first appears as two elements, and we add a reconciliation pass that detects the continuation (matching headers and column count, no intervening narrative text) and stitches them into one logical table before indexing.

Figures get dedicated handling because so much of the signal in transmission proposals and ISO/RTO studies lives in diagrams, not prose — one-line diagrams, substation layouts, deliverability-study charts. Every figure is extracted and, whether or not it carries any extractable text, passed to a vision-language model — an AI that looks at an image and describes it in words. We index that description alongside a pointer back to the original image. A query about a deliverability constraint near a given substation can then retrieve a one-line diagram that contains zero searchable text. Optionally, multi-modal embeddings can make the image itself retrievable by visual similarity — we'd add that only if evaluation shows figure-heavy queries underperforming, rather than paying for it speculatively.

The commodity stages — raw OCR and the base layout and table models — we take off the shelf rather than rebuild. Our differentiated value is the orchestration around them: strategy routing per document, the dual image-plus-structure table representation, VLM figure indexing, multi-page reconciliation, and the fallbacks that keep your hardest documents parsing into something useful instead of failing silently.

We chunk in two stages. First, structural chunking from the parse (Figure 1, KL·B) breaks a document into typed elements — narrative blocks, table rows, figure descriptions — so each chunk already knows what kind of thing it is and where it sits. Then each element's text is split by a semantic chunker into topic-coherent pieces, and each piece becomes one embedding (a numeric representation of its meaning, so related passages land near each other). We do this because fixed-size windows cut through the middle of an argument or a table; topic-aware splits keep a retrievable chunk on a single idea, which is what drives retrieval quality on long proposals and ~200-page selection reports.

We don't use one chunker for everything — we match the strategy to the content to control both quality and cost. Long-form prose goes through semantic chunking (the primary strategy); short, already-structured fields (table rows, short metadata snippets) use cheaper recursive or fixed-length splitting, because running the full semantic pass on a one-line field spends embedding calls for no retrieval gain. Each Vespa record is capped at five embeddings so no single record becomes a bag of unrelated vectors. The tooling is a custom semantic chunker (Greg Kamradt's percentile-breakpoint method, run on our own embeddings rather than a third-party wrapper), LangChain's recursive character splitter for structured content, and Vespa as the index.

Every chunk lands in our standard_document schema, which carries a deep metadata set on each record: document type and subtype; page number and on-page coordinates (for exact citation); section and hierarchy IDs; table position (row, column, is-table-root); figure pointers; created/updated dates and a version; folder path; and access scope (organization, owner, collaborators). The schema also includes typed key/value maps — string→string, string→int, string→double — so Viridon-specific dimensions like ISO/RTO, sponsor, project, or filing date attach to chunks without a schema change when a new dimension shows up. Maps are how we store arbitrary per-client metadata as dictionaries rather than hard-coding columns.

That metadata drives two kinds of filtering. Hard filtering uses the exact-match maps — "only chunks where ISO_RTO = CAISO and document_type = selection report" — applied before ranking to scope the search precisely. Soft filtering uses metadata that's full-text indexed (titles, filenames, folder path, short structured fields): it's folded into the hybrid score via BM25 — the standard keyword-matching algorithm — so a matching project name boosts a chunk's relevance without excluding anything. (Hybrid search blends this keyword signal with vector search, which matches on meaning rather than exact words.) And because we set this per field, metadata can be made BM25-searchable or kept filter-only by choice — ISO/RTO can be a hard filter, a soft ranking signal, or both, depending on how you want it to behave.

Metadata is created mostly automatically, with light human curation. The parser emits structure, coordinates, page, and hierarchy; file and source systems give dates, folder path, and access scope; a classifier assigns document type/subtype; and an LLM extraction pass pulls Viridon-specific values (ISO/RTO, sponsor, project, key dates) out of the content into the maps. Humans confirm the taxonomies and correct the occasional misclassification — corrections feed back rather than being re-done each time. This whole step is the Ingestion & Structuring component (KL·B) feeding the index (KL·A).

For Viridon we'd run a privacy-first, model-agnostic embedding layer on Amazon Bedrock, so every document is embedded entirely within Viridon's own AWS environment. We treat the embedding model as a swappable component rather than a fixed dependency — the retrieval architecture doesn't change when the model does — but the deployment posture (in-tenant, no data egress) is the part we'd hold fixed, because it's important in ensuring this system becomes an asset in your data room at time of exit.

The reason to anchor on Bedrock is data control. Accessed through an AWS PrivateLink VPC endpoint, embedding traffic stays on the AWS network within Viridon's chosen region and never crosses the public internet. Bedrock does not use inputs or outputs to train any model, and does not share them with model providers; data is encrypted in transit and at rest, optionally under Viridon's own KMS keys; and the service carries the compliance coverage a Blackstone-backed infrastructure company will be asked about (SOC 1/2/3, ISO 27001 and family, HIPAA-eligible, GDPR, FedRAMP). The practical consequence for the exit story: the embedding index is an owned asset with no third-party data exposure that could reprice a deal.

On model choice, the default for text is Amazon Titan Text Embeddings V2 — optimized for RAG (retrieval-augmented generation: answering from retrieved documents), multilingual, with selectable output dimensions (256 / 512 / 1024) and unit-normalized vectors. For the figure-heavy ISO/RTO and transmission material, we'd use a multimodal embedding model — Amazon Nova Multimodal Embeddings (a single model spanning text, documents, images, video and audio, with cross-modal retrieval) or Titan Multimodal Embeddings G1. This is what makes the multimodal retrieval we flagged in Parsing (1.1) real: a one-line diagram or deliverability chart is embedded into the same space as the surrounding text, so a text query can retrieve the figure directly, not only via its written description. If a privacy requirement or an eval result points elsewhere, Cohere embeddings on Bedrock — or self-hosted open-source models (e.g. BGE, E5, GTE) that run entirely inside the VPC — are also available; the choice is eval-gated and privacy-gated, not locked.

Two engineering invariants govern the layer. First, index and query must use the same model and the same dimension — both sides are wired to the chosen Bedrock model, and the vector index's tensor dimension is set to match. Second, we pick the output dimension to balance accuracy against storage and latency; these models use Matryoshka-style dimensions (embeddings trained so they can be safely truncated to a shorter length), so e.g. dropping from 1024 to 512 keeps ~99% of retrieval accuracy at half the storage. This is the detail behind the Indexing & Retrieval component (KL·A) in Figure 1.

Finally, embeddings are one signal of three, not the whole story. Retrieval blends semantic (vector closeness) with BM25 (lexical) and n-gram (fuzzy) matching, and the semantic weight is automatically zeroed for chunks that have no vector (e.g. number-only table cells) so lexical matching still works. That design is what lets the embedding model be swapped without rebuilding retrieval — the full ranker is covered in Section 1.4.

Retrieval is hybrid, not semantic-only, and it's a multi-stage pipeline rather than a single vector lookup. Every search fuses three signals inside one Vespa query — BM25 (lexical/keyword), semantic (vector nearest-neighbour over the chunk embeddings), and n-gram (character-trigram, typo-tolerant) — and several lightweight LLM steps wrap around that core to handle the messiness of real enterprise questions. Vespa (open-source, self-hosted in Viridon's VPC) is the engine because it does true hybrid retrieval and two-phase ranking in a single query; the query is embedded with the same Bedrock model used at index time (Section 1.3).

Before anything is searched, two things happen. Query distillation rewrites a conversational message into a standalone query — "what about their deliverability?" becomes "Sunrise project deliverability study outcome" using the recent conversation history, with explicit instructions not to over-interpret domain terms. Then multi-query expansion generates three additional phrasings (synonyms, full-forms/abbreviations like "CAISO" ↔ "California ISO", and different angles), so we typically search with four parallel queries. This lifts recall on under-specified questions, which is the common failure mode on a corpus this varied.

Each query then runs hybrid search in Vespa, scoped by organization, optional source selection, and hard metadata filters (the maps from 1.2 — e.g. ISO_RTO = CAISO). This is the Scoped Retrieval component (KL·H) in Figure 1. The lexical legs run over text plus high-value fields (name, filename, folder path); the semantic leg runs an approximate-nearest-neighbour search over the multi-vector chunk embeddings, with a candidate set of ~500 before ranking.

Ranking is where the signals combine, and it's fully configurable through Vespa rank profiles. The default profile blends semantic at 0.5, BM25 at 0.4, and n-gram at 0.1 in a first phase, then re-ranks the top 500 in a global phase with each signal linearly normalized (a master profile offers reciprocal-rank fusion — a standard method for merging several ranked lists — as an alternative). On top of that, field-level weights mean a match in a document's name or filename (weight 300) outranks the same match in body text (150), an empty-field discount keeps a chunk from being penalized for missing optional metadata, and the semantic leg auto-zeroes for chunks that have no embedding (number-only table cells) so lexical matching still surfaces them. A per-word significance step scores each query term HIGH / MEDIUM / LOW (1.0 / 0.5 / 0.01) so filler words don't pollute the lexical match while the full-query embedding stays intact.

After the parallel searches return, an application fusion layer merges and de-duplicates across the four queries and boosts documents that matched multiple phrasings (final = top score + 0.2 × second-best) — deliberate recall amplification for results that show up under several framings. Optional precision passes sit on top: Vespa's global-phase rerank is always on; a self-hosted open cross-encoder reranker (e.g. BGE-reranker) — a slower, more accurate model that re-scores the shortlist for precision — can be added; and an optional LLM source-picker can read the shortlist and return an ordered set of the most relevant sources before the answer is generated.

Finally, custom ranking is a first-class lever, not an afterthought — the rank profile is tuned per customer. The signal weights, the field weights, significance on/off, linear-norm vs reciprocal-rank fusion, scope defaults, and image inclusion are all configurable per request or per org without code changes, and we tune them against Viridon's eval set (Section 3). A worked retrieval trace on a representative document is below; we'd deliver the full trace on a Viridon document of your choosing as the requested artifact.

Several stages of the pipeline are genuinely commoditized, and we deliberately use strong off-the-shelf components for them rather than reinventing them. Recognizing what's commodity is a feature — it's what keeps the build lean and lets us concentrate engineering where it actually differentiates. In Figure 1 terms, the commodity sits underneath the boxes; the boxes themselves — how they're composed, tuned, and extended — are where the value is.

The commoditized stages are the foundation models and retrieval primitives. Parsing leans on the Unstructured framework, OCR, and the base layout and table-structure models. Embedding is a commodity capability — Titan, Nova and Cohere on Bedrock and self-hosted open models (BGE, E5) are interchangeable, and the vector math is the same everywhere. Retrieval's core operations — BM25, approximate-nearest-neighbour vector search, n-gram matching — are off-the-shelf Vespa primitives, and basic character-based chunking is a solved problem. The platforms themselves (Unstructured, Vespa, Bedrock) are infrastructure we build on, not things we'd ever rebuild. Reinventing any of these would destroy value, not create it.

Our differentiated value is the orchestration around those commodity pieces, and the Viridon-specific layers on top of them. In parsing: strategy routing per document, the dual image-plus-structure table representation, VLM figure-description indexing, multi-page table reconciliation, and the graceful-degradation fallbacks that keep hard documents from failing silently. In chunking: the two-stage structural-then-semantic design, the tuned semantic chunker, and — the biggest piece — the metadata schema with its typed maps, hard/soft filtering, and BM25-by-choice fields. In embedding: the privacy-first in-VPC deployment posture and the model-agnostic abstraction. In retrieval: the multi-stage LLM-wrapped pipeline (distillation, expansion, term significance, cross-query fusion) and the per-customer rank profiles.

And then there's the layer with no off-the-shelf equivalent at all — the bespoke knowledge-layer modules built for Viridon: the "what changes" map (KL·D), public-doc enrichment (KL·I), the RFI Q&A + SME-delegation memory (KL·J), the standard-terms playbook (KL·K), and the onboarding glossary (KL·L), plus the app-specific tools. No general-purpose tool ships these because they encode Viridon's process, not a generic one. This is exactly the gap an off-the-shelf product leaves: it gives you the commodity retrieval box and stops — which is why, on the diagram, a tool like Glean covers only KL·A.

This division is what makes the platform both efficient and bespoke. We don't pay to rebuild commodity foundations, so the budget goes to the integration, tuning, and Viridon-specific modules that are genuinely differentiated — and those are the parts that sit in Viridon's environment as an owned asset.

We'll ground this in what we demoed: the Proposal Writing Assistant — Setup workflow, end to end. In Figure 1 that's the Setup workflow in layer 4, marked deterministic, and it's the first phase of a larger AI teammate that runs Setup → Strategy → Drafting → Evaluation. The key design choice: Setup is a deterministic chain, not an agent improvising. The steps are known, ordered, and correctness-critical on a 300-page document — so we use the simplest control flow that does the job reliably, which here means a fixed sequence of tool calls rather than a free-roaming agent.

The demoed sequence: past winning proposals are ingested and broken into typed sections (KL·B, KL·C); a working template is auto-derived and the recurring variables are detected — project_name, sponsor, capacity, key dates (T8, KL·F); a "what changes" map flags the parts of the starting proposal that likely need to change this cycle — learned from how proposals have historically changed — versus the parts safe to keep (KL·D); the new bid's brief and documents are then used to fill the template variables and revise the flagged parts, touching only what likely needs to change so the rest is preserved (T4, KL·D); the ~200-page sponsor selection reports are ingested (KL·B); their win/loss themes are extracted and indexed as a retrievable advice module (KL·E); and finally, in AI editing, when the assistant recommends what to edit and how, it pulls that indexed selection-report advice (and prior sections) via retrieval, comments paragraph-by-paragraph, and proposes edits (T1, T5, drawing on KL·A / KL·E / KL·H). The full step-by-step is below.

On framework and dependencies: the orchestrator (Figure 1, foundation) chains these tools, and both the tools and the knowledge layer are exposed over MCP — the Model Context Protocol, an open standard for connecting AI assistants to tools and data. As the diagram says, the orchestrator can be an MCP client, your Claude / GPT desktop app, or a custom router — that's deliberate, because it decouples the control flow from the tools. Setup runs as a deterministic MCP tool chain; the later phases (Strategy, Drafting, Evaluation) move to orchestrated routing where the path depends on content.

The orchestrator also maintains structured memory across the loop, and how we model that memory is itself a design decision. Rather than carrying one ever-growing chat transcript — which conflates different concerns and quickly overflows the context window — we separate session memory into three kinds: conversation (the natural-language turns that capture user intent and constraints, like "don't delete anything"), working state (a structured scratchpad of the IDs and decisions the workflow has accumulated — the source of truth for where we are), and an episodic trace (an ordered log of every tool call, its arguments, and its outcome). Each planner step receives a deliberately bounded package assembled from these — truncated conversation, current working state, the last few trace steps — rather than everything every time. That separation is what keeps the agent within context limits, keeps machine state reliable across steps (the resumability in 2.2), and makes the whole run auditable (the execution trace in 2.3). This is session-scoped memory for the orchestration loop; cross-session institutional memory is a separate concern, covered in Section 4.

Scaling Setup into a full "AI teammate for proposal writing" is, concretely, two things: implementing the knowledge-layer modules, and building a composable tool set — Read paragraph, Create comment, Draft a section, Identify opportunities, Flow updates across the document, Evaluate against criteria, Aggregate attachments, Web research, Grounded Q&A. Those are exactly T1–T8 and T·Q in Figure 1, plus a few document-editor primitives. The teammate itself is a conversational multi-agent loop living in the multiplayer editor: it routes each turn to the right tool or subagent, drafts / comments / researches like a colleague, and proposes changes a human approves.

How we think about orchestration design comes down to a few principles. Use the simplest control flow that works — deterministic where steps are known, agentic only where they aren't. MCP as the interface, so tools are swappable and the orchestrator is replaceable. Human-in-the-loop at the right gates — the AI proposes, the human disposes; no risky or irreversible action (editing the live document, flowing a change across 300 pages) happens without explicit approval. And guardrails, governance and security throughout: RBAC-scoped retrieval (KL·H), the open-source / in-VPC deployment from our deployment principle, per-tool permissions, and a full execution trace of every step (Section 2.3). We build for performance (parallelize what's parallelizable) and extensibility (tools compose and get reused across mini-apps).

We decide between orchestration approaches by the shape of the work, and the patterns nest rather than compete. A deterministic chain for known, ordered, correctness-critical steps (Setup). A router to specialists when requests are heterogeneous and each needs a different capability (the teammate picking T1 vs T2 vs T7 per turn). A manager that decomposes into parallel sub-tasks when the work splits into independent units (evaluating all sections at once, multi-query retrieval, flowing one change across 300 pages). And a multi-agent loop for interactive, open-ended work with a human present (live drafting). For proposal writing this combination is right because Setup demands reliability, drafting is inherently interactive, evaluation is embarrassingly parallel — and, critically, every specialist tool we build composes: the comment, Q&A, research and retrieval tools built here are reused by the RFI drafter, the legal screener, and the onboarding assistant. That reuse is the whole shared-foundation thesis of Figure 1, and it's why we optimize for composition.

Our first reliability technique is to minimize the surface area for failure: the most reliable step is a deterministic one, which is why Setup is a fixed chain (2.1) rather than an agent improvising. For the parts that genuinely need an LLM, we treat it as a fallible component and wrap it in four things — validated structured outputs, checkpointed state, risk-classified human approval, and an in-loop reflection step. Together those cover the three failure questions: what happens when a step fails, how we validate between steps, and how we keep the workflow from drifting.

When a step fails or returns low-quality output, we separate two cases. A hard failure (error, timeout, tool exception) triggers a bounded retry with backoff, then a fallback path where one exists — the same pattern as the parsing fallbacks in 1.1, where a failed table inference degrades to image + description rather than crashing — and if it's still failing, we resume from the last checkpoint and, if exhausted, escalate to a human rather than proceed on a broken step. A soft failure (the step runs but the output is malformed, low-quality, or unsupported) is caught by the validation and reflection gates below, then repaired, retried, or escalated. The principle throughout: never silently pass a bad result downstream.

We validate output between steps by making every step emit a structured, typed output against an explicit schema — so we know exactly what the model produced and can check it programmatically before the next step consumes it. Schema validation deterministically catches malformed or hallucinated structure (missing fields, wrong types, out-of-range values); a grounding check verifies that claims which should be supported by retrieved sources actually are (the anti-hallucination contract that feeds our eval harness in Section 3). Each step has an explicit input/output contract, so a downstream step never has to guess what it received.

State management makes failure recoverable. Each step's inputs and outputs are checkpointed and steps are designed to be idempotent (safe to re-run without applying anything twice), so on any failure we know exactly where we left off and resume from the last good checkpoint — we don't re-parse a 300-page proposal or re-embed the corpus because a later step timed out. This matters most for the proposal workflow specifically, which runs over months, not minutes.

We prevent drift with a reflection step in the loop — a pattern we've already implemented in our agentic orchestration work. After a step (or on a cadence), a critic re-checks the work against the original objective and constraints, catches drift, and either re-anchors, re-plans, or halts. The goal and constraints are carried through every step so the agent never loses the thread, scoped tools limit how far it can wander, and explicit termination criteria plus bounded autonomy (caps on tool calls, recursion, and cost) stop a runaway loop.

Finally, the human gate is itself a reliability control. We classify actions by risk and reversibility: read-only and reversible-in-draft actions (search, comment, propose an edit) run autonomously, while consequential or irreversible actions — applying an edit to the live document, flowing a change across 300 pages, anything external — require explicit human approval. And because applied edits are versioned (the document carries a version field), even an approved change is reversible. All of this sits on a full execution trace (Section 2.3), because you can't make reliable what you can't see.

Yes — fully, top to bottom. The shift that makes this real is treating a workflow run as spans, not log lines: a run is one trace, each step is a span, and nested tool calls and sub-agents are child spans. That tree is exactly what answers "what each step retrieved, decided, and passed downstream," because those relationships are a hierarchy, not a flat stream. We build it on OpenTelemetry (the open industry standard for tracing software) with LLM-specific semantic conventions, and the backend — Langfuse or Phoenix — is open-source and self-hosted, so the trace store lives inside Viridon's VPC alongside everything else. No traces of Viridon's proposals go to a third-party SaaS.

We design for two audiences with two surfaces over the same captured data. Your technical advisor gets the full span tree and audit logs — every tool call, every retrieval, every decision, validation result and hand-off, with token, cost and latency per span and the exact prompt-template and model version that produced each output, plus replay. Erin and end users get explainability instead of raw internals: a "why did it suggest this?" view that traces any AI recommendation back through the advice it used to the source page. Same data underneath; the advisor sees the engine, the user sees the reason.

Mapping directly to your three words: retrieved is the retrieval trace from Section 1.4 captured on each search span — the distilled and expanded queries, the candidate set, the per-signal scores, and what was filtered out, not just what came back. Decided is the planner's tool choice and the alternatives it weighed, the working-state diff for that step, and the validation/reflection verdict. Passed downstream is the typed output and the working-state delta — the bounded planner package handed to the next step.

The substrate already exists. The episodic trace, working state, and bounded planner package from our memory model (2.1) already record what each step did, what changed, and what was passed on. Observability is mostly turning that into spans and a UI — productionizing what the orchestration loop already captures, not bolting on a parallel logging system.

The feature that matters most for proposal writing is provenance lineage: for each AI claim or proposed edit, we record which retrieved chunk supported it and link that chunk back to its source page. So a recommendation traces cleanly as edit → selection-report advice (KL·E) → chunk on p.142 of the 2023 report. End-to-end answer-to-source lineage is what makes the user-facing explainability trustworthy — and it doubles as a clean artifact for a future data room.

Two things worth flagging for a technical reviewer. Logged "reasoning" is the model's stated rationale — an honest record of what it reported, not a proof of the true cause — so "decided" means the recorded decision plus its stated reasoning. And exact reproducibility is bounded by model nondeterminism: we pin prompt-template and model versions and set seeds where the provider allows, so replaying a trace is fully reliable, but re-generating a hosted model's identical output is not guaranteed. Because traces contain document content, redaction (arguments are already redacted), access control on the trace UI, retention limits, and sampling are part of the design, not afterthoughts.

We measure each stage of the pipeline separately, on purpose — because a bad final answer is a symptom, and what makes an eval useful is being able to say why it was bad. The three failure types you name aren't interchangeable: they live in different stages and have different fixes, so we attribute every failure to a stage rather than scoring only the end result. That localization is the whole design of the eval.

The wrong source retrieved is a retrieval-stage failure, scored against a labeled set of which chunks are relevant per query. The headline metric is recall@k / hit-rate — did a relevant chunk make the top-k at all — because if it wasn't retrieved, the generator simply can't use it. Around that we track context recall (did we get all the chunks needed) versus context precision (are the relevant ones ranked above the noise), and MRR / nDCG for how high the first relevant chunk landed. We split a recall miss (relevant chunk absent — usually fatal) from a precision miss (irrelevant chunk ranked high — dilutes context) because they have different fixes. For Viridon we add filter correctness — did a scope like ISO_RTO = CAISO actually apply — because the dangerous failure here is cross-project contamination, which scoped retrieval (KL·H) exists to prevent.

An unsupported claim — a hallucination — is a generation-stage failure, measured as faithfulness / groundedness. The key distinction: we don't score "is it true in the world," we score "is every claim entailed by what we retrieved" — the right contract for RAG, because we control the sources. Mechanically we decompose the output into atomic claims and check each against the retrieved context (supported / unsupported / contradicted), plus citation accuracy — does the cited source actually support the claim, which the provenance lineage from 2.3 makes directly checkable. One subtlety: a hallucination is often a retrieval failure in disguise. If context-recall was low, the model filled the gap — so we only call it a generation bug when recall was high and it still invented something. That's why we measure retrieval separately rather than scoring the final answer alone.

Low output quality is the fuzziest and most domain-specific. The generic dimensions are answer relevance and completeness, instruction-following (did it respect "don't touch the boilerplate", length, tone), coherence, and format / schema validity (already enforced by the structured-output validation in 2.2). But the differentiated quality metric for proposal writing is "winning-ness" — scoring whether an edit makes a section more like the sections that have won, built from the selection-report advice in KL·E. That's a quality rubric grounded in what Viridon actually cares about, which no off-the-shelf eval framework gives you.

Around those three sit a broader taxonomy. Because this is agentic, not only RAG (Section 2), we also measure task success / completion, tool-selection accuracy, and trajectory correctness (did it reach the answer for the right reasons, not by luck), alongside operational signals — drift / escalation rate (2.2), latency, and cost. And the single best real-world quality signal for the assistant is human edit-distance / acceptance: how much Erin changes a proposed edit before accepting it. Low edit distance is high quality, measured on live usage with no labeling. The full taxonomy is in the detail below.

Two limitations to be upfront about. Several of these metrics use an LLM as judge, which is itself fallible — so we calibrate it against human labels, reserve it for scale, and keep humans on the high-stakes and subjective calls. And all of it is only as good as the ground truth it's scored against, which is the next question (3.2). Tooling stays in-VPC per the deployment principle: RAGAS-style metric computation (RAGAS is an open-source toolkit for evaluating RAG systems) plus a self-hosted judge model on Bedrock, fed by the execution traces from 2.3.

Ground truth is the real bottleneck in enterprise RAG evaluation, so we treat SME time as the scarce resource we engineer around, not an afterthought. The first move is to stop thinking of "ground truth" as one thing: it has three layers — retrieval truth (which chunks are relevant to a query), answer truth (the correct answer text), and preference / rubric truth (which of two outputs is better, or how it scores on a rubric). They cost very different amounts to label, so matching the cheapest viable label type to each metric is already a major saver — most retrieval and faithfulness checks need no authored answer at all.

The single biggest unlock for Viridon is that your archive is already a labeled dataset. A library of won proposals and ~200-page selection reports isn't just source material — a winning section is a gold answer for "how should this section read," and a selection report is labeled feedback on what was strong and weak. So a large share of the "right answers" already exist in your corpus; the work is extraction, not authoring. That turns ground truth from a cost you'd carry into an asset you already own.

Around that, we draw labels from the cheapest sources first (the ladder below). Reference-free metrics need zero SME input — faithfulness is checked against the retrieved context, not a gold answer, and schema validity is deterministic. Implicit labels from real usage are free and compounding: every time Erin accepts, edits, or rejects a proposed change, that's a label, and the edit diff tells us how it was wrong. Synthetic generation with human verification handles the rest — an LLM drafts (question, answer, source-chunk) triples from your documents, and the SME's job collapses from authoring to approving or correcting, which is several times faster.

Where SMEs are needed, we minimize their time deliberately: approve, don't author (review LLM-drafted labels rather than writing from scratch — the biggest single lever); active learning (we surface the highest-value cases — where the system is uncertain or where the judge and a human disagree — instead of asking them to label at random); a small, stratified golden set plus a large auto-graded set (a few hundred carefully chosen, human-verified examples anchor a calibrated LLM judge that handles the volume); and capture in the natural workflow (a thumbs-up, an accepted edit, or a "this source is wrong" flag in the product is a label given without extra effort). The result is that SME involvement is bounded and front-loaded, and trends toward near-zero as usage-based labels compound.

On who builds it: we build the harness, generate the synthetic candidates, mine the historical corpus, and run and calibrate the judge; your SMEs spend bounded, high-leverage time approving and correcting the golden set and resolving the contested cases; and the product harvests implicit labels continuously. Two caveats. Ground truth isn't static or singular — SMEs disagree and what "wins" shifts as sponsors change — so we measure inter-annotator agreement, version the golden set, and treat it as living rather than a one-time deliverable. And synthetic labels carry a bias risk — an auto-generated test set can be easy in the same ways the system is good, flattering the scores — which we counter by seeding from your real artifacts and keeping a human-authored slice as the hard anchor.

Evals run as a hybrid at three cadences, not one — so the answer to "automated, your team, or ours" is: all three, at different speeds. A fast automated suite runs in CI on every prompt or model change and blocks the merge if it regresses (the regression gate). A comprehensive batch runs nightly and on-demand against the full golden set for the thorough scorecard. And continuous online monitoring scores sampled production traffic on reference-free metrics. Automation does the volume; humans do the judgment.

On division of labor: BetterBrain builds and maintains the pipeline, writes the metrics, owns the CI gate, triages regressions, and calibrates the judge. The automated system does the bulk of the work — CI on every change, the nightly batch, and live monitoring — with no human in the loop. Your SMEs touch only the irreducibly human part: periodic review of the golden set and adjudicating the handful of borderline cases CI surfaces.

On your time commitment — concretely, because you asked: almost all of it is upfront, establishing and ratifying the golden set and the "winning" rubric — on the order of 15–20 SME-hours, front-loaded over the first few weeks, and mostly approve-not-author (per 3.2). After that there is no standing commitment: ongoing involvement is ad-hoc only — when the golden set needs an update because the corpus or sponsor criteria changed — averaging under 30 minutes a month, and trending down further as implicit usage labels compound.

On regression testing: the locked, versioned golden set is the regression suite. On any prompt, model, embedding/index, or tool change, we re-run it and compare to the previous baseline. Because outputs are non-deterministic we don't assert string equality — we gate on metric thresholds and no-regression deltas ("no metric dropped more than N% vs. baseline"). The most actionable technique is A/B diffing: surface only the examples that flipped pass↔fail, so a reviewer looks at the handful that changed, not all 500. And we slice and gate per segment (doc type, ISO/RTO, question type), because a sub-segment can tank while the average stays flat — the silent-degradation trap that aggregate-only eval misses.

Two operational notes: The judge model is itself non-deterministic, so a "regression" can be judge noise — we pin the judge's model and prompt versions, average over runs on the golden set, and route borderline flips to a human. And we treat eval cases as version-controlled code — the golden set lives in the repo and changes via review, so the suite evolves with the same rigor as the system. The loop closes with 2.3 and 3.2: production failures caught by monitoring are promoted into the golden set, so the regression suite gets harder exactly where the system is weak. Tooling — Promptfoo, Langfuse or Phoenix — is self-hosted in-VPC per the deployment principle. The full operating model is the plan below.

The system learns primarily in the knowledge layer, not in the model weights. When Erin edits or rejects a proposed change, we capture the before→after diff and its context, generate a piece of reusable advice from it ("for CAISO deliverability sections, lead with the quantified outcome"), and index that advice so future retrievals surface it. It's the same machinery as the selection-report advice module (KL·E), and the result is inspectable and correctable — you can read, edit, or delete what the system has "learned" (Section 4.3). And it isn't only advice that improves: the same feedback updates other knowledge-layer components in Figure 1 — the entity & concept map / knowledge graph (KL·G), the "what changes" patterns (KL·D), and the glossary (KL·L) — so a correction can fix an entity or a relationship in the graph, not just a piece of guidance.

On cadence, it's both — split by mechanism. Anything that's just retrieval-time context applies live: a correction becomes an indexed advice entry the very next retrieval can pull, with nothing retrained, and the in-session working memory (2.1) already adapts within a task. Anything that involves synthesis or judgment is deferred to a batch / nightly "reflection": clustering many edits into one durable advice entry, resolving contradictions when SMEs disagree, promoting a pattern only once it's been seen several times, and re-ranking which advice is trusted. Why that cadence: a single edit is noisy, and you don't want one idiosyncratic correction to immediately reshape behavior for everyone — the batch step is deliberate noise control, and it's where the degradation guardrail lives (4.5).

The signal we capture is richer than accept/reject. The most valuable is the edit diff itself — the corrected text tells us not just that a suggestion was wrong but how, and it doubles as a free gold label. Around it: explicit accept / reject / thumbs, behavioral signals (used, ignored, asked a follow-up, re-ran the search), and explicit corrections ("this source is wrong", "this advice doesn't apply here") — the highest-value, lowest-volume signal.

On reinforcement learning, we use the framing deliberately, not the heavy machinery. We treat acceptance as a reward signal and optimize the policy that decides which advice and which retrieval configuration to surface — not the model weights. The genuine reinforcement-style mechanisms on our roadmap are (1) a system that automatically learns which advice and which result-ranking to surface — it tries different options, watches which ones lead Erin to accept the suggestion, and shifts toward the ones that work, essentially self-tuning A/B testing that runs continuously, fully in-VPC, with no model weights touched — and (2) automatic tuning of the prompts and examples against the eval metrics, so they're optimized by measurement rather than by hand. We explicitly do not fine-tune model weights: it would break the inspectability we're selling, complicate the in-VPC deployment, make evals harder, and is the wrong investment at this corpus size. What we do is accumulate the accept/reject preference pairs as an asset — the dataset that would make fine-tuning possible later, to be spent only if the eval gain ever warrants it.

And yes — the same signal feeds evaluation. Every accept / edit / reject is simultaneously a learning signal and an eval label (the human-edit-distance metric in 3.1, the implicit labels in 3.2). That coupling is also the safeguard — a new or updated advice entry is promoted only if it does not regress the golden set, so the feedback loop and the eval loop are the same flywheel: usage → advice → eval-gated promotion → better retrieval → more usage. High-impact advice can require human approval before it goes live, so what the system learns stays governed.

In one line: what changes is data and configuration, not the model. The core mechanism is the concept / advice store — learning adds, updates, and re-ranks indexed advice entries and the entity & concept map (the knowledge graph), along with the other knowledge-layer modules in Figure 1, which together are the institutional memory we cover in 4.3. Everything else follows from that: because new advice is indexed, retrieval surfaces different, better-grounded context, and on the roadmap the ranking itself self-tunes toward what gets accepted. Prompts and examples are auto-tuned against the eval metrics (roadmap), not edited per interaction. Session memory adapts live within a task and resets per run (2.1). And the eval golden set itself grows as production failures are promoted into it — so the system's measurement improves alongside its behavior. Model weights do not change, by design.

What accumulates is structured, human-readable knowledge — not opaque vectors or model weights. The institutional memory is a set of concept and advice entries in the knowledge layer: the advice mined from edits and selection reports (KL·E), the entity and concept map of customers, projects, ISO/RTOs and terms and how they link (KL·G), the "what changes" patterns (KL·D), and the onboarding glossary (KL·L). Each is a record you can read in plain language, not a number in a tensor.

In form, every entry is a structured record: a plain-language statement of the knowledge, the scope it applies to, its provenance (which edits and sources produced it), confidence and usage stats, a status, and a version history. The anatomy is below — it reads like a note with an audit trail.

To the heart of your question — yes, all of it is human-readable, auditable, and correctable. Everything the system learns is exposed to you in plain language and is fully editable: you can inspect any entry and trace it back to the edits and sources that produced it (the provenance from 2.3), correct or rewrite it, and disable, delete, or pin it. Because learning lives in the concept store and not in the weights (4.2), the entire learned state is open to inspection and repair — there is no hidden knowledge baked into a model you can't read. A curation view makes this a first-class surface (4.4).

Auditability comes from the same structure: each entry carries provenance (what created it, when), version history (what changed), and usage stats (how often it's retrieved and applied, and how often accepted) — so you can audit not just what it knows but why it knows it and whether it's actually being used. A correction simply becomes the new entry (eval-gated before it's trusted, per 4.5). And because the whole memory is a readable, ownable artifact rather than a black box, it transfers with the company — the same owned-asset principle that runs through the architecture.

Upkeep is almost entirely automated. The loop in 4.1 does the work with no person in it — generating advice and entity/graph updates from edits, indexing, consolidating and de-duplicating them, scoring confidence, promoting a pattern only once it's been seen enough times, and re-ranking which knowledge is trusted. Humans do not author or maintain the memory by hand.

The human role is oversight by exception. The one thing worth watching for is an incorrect deduction — the system over-generalizing a one-off edit into a rule that shouldn't apply broadly. When that happens, a reviewer corrects, narrows, disables, or deletes the entry (or adds one directly to teach something) through the curation surface (4.3). It's review-and-fix when something looks off, not continuous curation.

By whom, and how little: the curation is done by an SME or power user (e.g. Erin) for the proposal domain, with BetterBrain monitoring the memory's overall health and tuning the loop. The burden stays low because the guardrails catch most bad deductions before they ever reach a human — confidence thresholds, promote-after-N, eval-gating (4.5), and approval on high-impact entries (4.1) — so what reaches manual review is the genuine exceptions. This is consistent with the eval upkeep budget in 3.3: bounded and ad-hoc, well below a standing commitment.

The core risk in any feedback loop is an echo chamber: the system learns from its own outputs and reinforces its own mistakes until errors quietly become "truth." We prevent that with defense-in-depth — multiple independent safeguards, so that if one misses a problem the next one catches it — and one structural choice does most of the work.

That choice: learning is measured against an independent anchor, not against the model's own recent behavior. A new or updated entry is promoted only if it doesn't regress the golden set (3.3), and the golden set is anchored in external truth — selection reports plus a human-authored slice (3.2) — not in whatever the model did lately. So the loop can't drift to merely agree with itself.

On top of that, noise control and negative signal: we consolidate in batch and promote only after a pattern recurs (promote-after-N), so one idiosyncratic edit can't reshape behavior, and contradictions are resolved rather than stacked (4.1). And we learn from rejections and heavy edits, not just acceptances — so the loop isn't one-sidedly reinforcing what it already does.

Then catch and contain: per-segment regression gating and online monitoring (2.3, 3.3) catch degradation even when the aggregate looks fine. And because the memory is data, not model weights (4.1), with full provenance and version history (4.3), a bad deduction is traceable, reversible, and deletable — the blast radius is bounded and nothing compounds silently. High-impact changes can be rolled out gradually — applied to a small slice first (a "canary") and widened only if it holds — before they're fully trusted.

Finally, knowledge also degrades by going stale — a sponsor's criteria shift, an ISO/RTO rule changes. Entries carry recency and versioning, older ones decay or get re-validated, and periodic re-evaluation keeps the memory current. Taken together, errors can't quietly become truth, drift is caught against an external reference, and the whole memory stays inspectable and reversible.