You should be able to see exactly how an audit works before you pay for one.
The CARA methodology in full: how scenarios are designed, how evaluators are calibrated, how scores are computed, and how we keep ourselves honest. No gating. No sales call required to read it.
1 · The scoring model
The CARA score is a 0–100 composite across five dimensions with published weights. Every finding carries a severity assigned against a written rubric, and a frequency across runs — "46% of 24 attempts," never an anecdote.
| Dimension | Weight | Scenario share |
|---|---|---|
| Task Completion | 30% | ~28 |
| Safety & Guardrails | 25% | ~20 |
| Output Consistency | 20% | ~14 |
| Tool Use | 15% | ~12 |
| Eval Infrastructure | 10% | interview + artifact review |
| Severity | Definition | Business-impact test |
|---|---|---|
| CRIT | Wrong action taken, data exposed, or user materially misled | Would trigger an incident review, refund, or legal exposure |
| HIGH | Workflow fails or degrades for a meaningful share of users | Measurable churn, cost, or support-load impact |
| MED | Quality inconsistency or inefficiency without direct harm | Erodes trust or margin over time |
2 · Scenario governance
Every audit runs 80+ scenarios drawn from a versioned library. Each scenario documents its objective, persona, success criteria, and dimension mapping. The library grows from every audit and from every published incident in the wild — Air Canada-class failures become permanent scenarios. Vertical packs exist for fintech, legal, health, HR, and support.
3 · Evaluator calibration
The vetting bar covers domain background, adversarial-testing aptitude, and written-analysis quality — our parent company's sixteen-year core competency is exactly this vetting. Before touching client work, every evaluator completes calibration runs against reference agents with known defects. Evaluators falling below the inter-rater agreement threshold are recalibrated or removed. Every finding is independently reproduced by a second evaluator before it enters a report.
4 · What we deliberately don't do
No LLM-as-judge in scored dimensions — assistive tooling appears in triage only, and is disclosed. No source-code review; that's Ingenize's Technical Integrity Audit, a different discipline and a disclosed relationship. No pass/fail theater: a score with context, never a rubber stamp.
5 · Independence protocol
Findings are delivered before any remediation or operations proposal. The client owns the full report, every finding, and the roadmap outright, with no dependency on buying anything further. Remediation through our parent company Ingenize is optional and disclosed in writing before engagement. Benchmark contributions are anonymized and aggregated. We publish the day the guarantee is ever paid out.
6 · Limitations, stated plainly
Behavioral testing samples behavior; it cannot prove the absence of failure — no method can, and anyone who claims otherwise is selling certainty, not assurance. A CARA score is evidence, dated. Agents drift; that's why retests and continuous coverage exist.