> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firedog.finance/llms.txt
> Use this file to discover all available pages before exploring further.
# Quality & shadow testing
> How Firedog proves a cheaper model holds up — measured cost and quality, without a prompt or response ever leaving your VPC.
Firedog answers two questions a finance or risk owner cannot answer from a vendor invoice: **is the model actually doing the job**, and **would a cheaper model do it just as well**. Both are answered with measured numbers, not opinion — and both are computed inside your own VPC, so no prompt or response content is involved in the answer that reaches the dashboard.
A random sample of live answers is graded by an LLM-as-judge on five dimensions. The verdict travels as `qaOverall` metadata.
Opt-in. The collector re-runs a call against a cheaper sibling and reports the measured saving and the sibling's quality — never its content.
## Quality scoring
A random sample of answers — a default of roughly **25%** of calls — is scored by an **LLM-as-judge**. Each sampled answer is graded on five dimensions:
The five scores collapse into one figure. An answer **passes at `qaOverall ≥ 4.0 / 5`**.
Only the overall number is emitted to the control plane, as the `qaOverall` field on the call's metadata. The judge runs inside the collector, against the local record — the graded prompt and response stay in your VPC. What reaches the dashboard is a single float per sampled call, which the **Answer quality** view aggregates into pass rates per model and per use-case.
The \~25% sample rate is a configurable default, not a fixed guarantee. Confirm the tunable range and the judge model for your deployment with the Firedog team — *\[Placeholder — confirm with the Firedog team]*.
The judge's overall score for a sampled call, from `0.0` to `5.0`. Present only on the sampled subset; absent on unsampled calls.
Sampling means most calls carry no `qaOverall`. Read the field as a **rate across a population** (the share of sampled answers clearing 4.0), not as a verdict on any single call.
## Shadow testing
Quality scoring tells you whether the model you are paying for is doing the job. Shadow testing answers the next question: **could a cheaper model have done the same job**. It is **opt-in per team and per workflow** — nothing runs in the background until you enable it.
When enabled, the collector takes a live call and, in the background, re-runs the same request against a **cheaper sibling** of the model that was used:
| Primary model | Shadow sibling | Input \$/1M | Output \$/1M |
| ------------------ | ------------------------- | ----------- | ------------ |
| opus | sonnet | 15 → 3 | 75 → 15 |
| gcp/gemini-2.5-pro | gcp/gemini-2.0-flash-lite | 2.5 → 0.1 | 10 → 0.4 |
The shadow answer is scored by the same five-dimension judge and priced from the same shared pricing table. The collector then compares the two on **cost** and on **quality**, and emits exactly two numbers to the dashboard:
The measured dollar difference between the primary call and the cheaper shadow — what you would have saved on this call.
The shadow answer's `qaOverall`, judged on the same 4.0 bar — so a saving is never read without the quality it came at.
A saving is only evidence if quality held. Always read `shadowSaving` next to the shadow's quality score: a large saving at a shadow quality below 4.0 means the cheaper model **failed the job**, not that you are overpaying.
### What is emitted, and what is not
```mermaid theme={null}
flowchart LR
A[Live call] --> B[Collector · in your VPC]
B --> C[Shadow re-run
cheaper sibling]
C --> D[Judge scores shadow
Price from pricing table]
D -->|shadowSaving + shadow qaOverall| E[Firedog cloud · control plane]
D -.->|shadow prompt / response| F[(Local Postgres · stays in VPC)]
style F stroke-dasharray: 5 5
```
The shadow call's prompt, its RAG context, and its response are content — so they follow the same rule as every other record: they are written to the **local Postgres in your VPC and never leave it**. Only the saving and the shadow's quality score cross the boundary as metadata. See [/security](/security) for the full split-plane guarantee and [/sensitivity-profiles](/sensitivity-profiles) for how emission is governed.
The measured cost difference (primary minus shadow) for a shadow-tested call, in the collector's currency. Present only on calls where shadow testing ran.
### The cost of shadow testing
A shadow run is a **real second call** to a real provider. It adds the sibling model's provider cost for every call you shadow. Because the sibling is the cheaper model, the added cost is a fraction of the primary — but it is not zero, and it is why shadow testing is opt-in and scoped per workflow rather than on by default.
A common pattern is to enable shadow testing on a high-volume workflow long enough to gather a decision-grade sample, read the result in the **Audit trail** view — where the per-call `shadowSaving` and the shadow's quality sit next to the original call as evidence — and then either migrate the workflow to the cheaper model or turn shadowing back off.
## How the two fit together
`qaOverall` on the live sample tells you the pass rate the current model is holding. This is the bar any cheaper model has to clear.
On the workflows you opt in, `shadowSaving` and the shadow's quality show what a cheaper sibling would have cost and whether it would have passed.
The Answer quality view aggregates pass rates; the Audit trail view holds the per-call saving and quality so a model-change decision is defensible line by line.
Nothing on this page requires content to leave your VPC. The judge and the shadow re-run both execute in the collector against local records; the control plane only ever sees `qaOverall`, `shadowSaving`, and the shadow's quality score.
## Related
The split-plane guarantee — why content stays in your VPC.
strict / standard / open — what the collector is allowed to emit.
Tag every call by team, workflow and user so quality reads per desk and run.