Skip to main content
Firedog answers two questions a finance or risk owner cannot answer from a vendor invoice: is the model actually doing the job, and would a cheaper model do it just as well. Both are answered with measured numbers, not opinion — and both are computed inside your own VPC, so no prompt or response content is involved in the answer that reaches the dashboard.

Quality scoring

A random sample of live answers is graded by an LLM-as-judge on five dimensions. The verdict travels as qaOverall metadata.

Shadow testing

Opt-in. The collector re-runs a call against a cheaper sibling and reports the measured saving and the sibling’s quality — never its content.

​
Quality scoring

A random sample of answers — a default of roughly 25% of calls — is scored by an LLM-as-judge. Each sampled answer is graded on five dimensions:

Accuracy

Completeness

Clarity

Relevance

Tone

The five scores collapse into one figure. An answer passes at qaOverall ≥ 4.0 / 5. Only the overall number is emitted to the control plane, as the qaOverall field on the call’s metadata. The judge runs inside the collector, against the local record — the graded prompt and response stay in your VPC. What reaches the dashboard is a single float per sampled call, which the Answer quality view aggregates into pass rates per model and per use-case.
The ~25% sample rate is a configurable default, not a fixed guarantee. Confirm the tunable range and the judge model for your deployment with the Firedog team — [Placeholder — confirm with the Firedog team].
​
qaOverall
number | undefined
The judge’s overall score for a sampled call, from 0.0 to 5.0. Present only on the sampled subset; absent on unsampled calls.
Sampling means most calls carry no qaOverall. Read the field as a rate across a population (the share of sampled answers clearing 4.0), not as a verdict on any single call.

​
Shadow testing

Quality scoring tells you whether the model you are paying for is doing the job. Shadow testing answers the next question: could a cheaper model have done the same job. It is opt-in per team and per workflow — nothing runs in the background until you enable it. When enabled, the collector takes a live call and, in the background, re-runs the same request against a cheaper sibling of the model that was used:
Primary modelShadow siblingInput $/1MOutput $/1M
opussonnet15 → 375 → 15
gcp/gemini-2.5-progcp/gemini-2.0-flash-lite2.5 → 0.110 → 0.4
The shadow answer is scored by the same five-dimension judge and priced from the same shared pricing table. The collector then compares the two on cost and on quality, and emits exactly two numbers to the dashboard:

shadowSaving

The measured dollar difference between the primary call and the cheaper shadow — what you would have saved on this call.

Shadow quality

The shadow answer’s qaOverall, judged on the same 4.0 bar — so a saving is never read without the quality it came at.
A saving is only evidence if quality held. Always read shadowSaving next to the shadow’s quality score: a large saving at a shadow quality below 4.0 means the cheaper model failed the job, not that you are overpaying.

​
What is emitted, and what is not

The shadow call’s prompt, its RAG context, and its response are content — so they follow the same rule as every other record: they are written to the local Postgres in your VPC and never leave it. Only the saving and the shadow’s quality score cross the boundary as metadata. See /security for the full split-plane guarantee and /sensitivity-profiles for how emission is governed.
​
shadowSaving
number | undefined
The measured cost difference (primary minus shadow) for a shadow-tested call, in the collector’s currency. Present only on calls where shadow testing ran.

​
The cost of shadow testing

A shadow run is a real second call to a real provider. It adds the sibling model’s provider cost for every call you shadow. Because the sibling is the cheaper model, the added cost is a fraction of the primary — but it is not zero, and it is why shadow testing is opt-in and scoped per workflow rather than on by default.
A common pattern is to enable shadow testing on a high-volume workflow long enough to gather a decision-grade sample, read the result in the Audit trail view — where the per-call shadowSaving and the shadow’s quality sit next to the original call as evidence — and then either migrate the workflow to the cheaper model or turn shadowing back off.

​
How the two fit together

1

Scoring establishes the baseline

qaOverall on the live sample tells you the pass rate the current model is holding. This is the bar any cheaper model has to clear.
2

Shadowing tests the alternative

On the workflows you opt in, shadowSaving and the shadow’s quality show what a cheaper sibling would have cost and whether it would have passed.
3

The dashboard turns it into evidence

The Answer quality view aggregates pass rates; the Audit trail view holds the per-call saving and quality so a model-change decision is defensible line by line.
Nothing on this page requires content to leave your VPC. The judge and the shadow re-run both execute in the collector against local records; the control plane only ever sees qaOverall, shadowSaving, and the shadow’s quality score.

Security

The split-plane guarantee — why content stays in your VPC.

Sensitivity profiles

strict / standard / open — what the collector is allowed to emit.

Attribution

Tag every call by team, workflow and user so quality reads per desk and run.