Verified completion rate
The share of admitted work that satisfies its verification contract.
FDLC Benchmarks / 07Proposed
Measure the outcome, not the activity.
FDLC Benchmarks will compare factory effectiveness through verified delivery, quality, cost, human effort, recovery, and risk. No public benchmark results are claimed today.
Verified outcome
Tokens, prompts, lines of code, and concurrent agents can explain operating cost. They do not prove value.
Measurement contract
Twelve metrics worth earning.
A useful benchmark publishes the work type, acceptance contract, risk, environment, evaluator, and uncertainty beside every result.
Cost per verified outcome
Total cohort cost, including failed work and retries, divided by independently verified outcomes. Report cost coverage.
Human review minutes
Judgment time required per accepted change.
Cycle time
Elapsed time from approved intent to accepted outcome.
First-pass verification rate
Candidates that satisfy required checks without corrective work.
Reopen rate
Accepted or closed work that must re-enter governed execution.
Rollback rate
Released outcomes that require reversal.
Escaped defect rate
Defects discovered after the governed release boundary.
Policy violation rate
Attempts blocked or found outside their delegated authority.
Recovery success rate
Interrupted work safely resumed or reconciled without loss of lineage.
Cost by work type
Outcome cost segmented by repeatable factory-line class.
Verifier disagreement rate
Material disagreement among deterministic, model, or human evaluators.
- Accepted outcomes by work type
- Evidence coverage and first-pass quality
- Human attention and recovery
- Complete attributable cost
- Failure, rollback, and escaped defects
- Generated tokens
- Lines of code
- Prompt volume
- Agent count
- Demo throughput without accepted outcomes
Comparable results
Publish the denominator and the missing evidence.
A green service, a completed Attempt, a verified artifact, and an accepted outcome answer different questions. Report each separately.
Declare the cohort
Freeze the work type, risk, repository, Factory Version, evaluator, baseline, and observation window. Count failed work and retries in spend. Verification and acceptance use separate denominators.
Show cost coverage
Label model, tools, retrieval, compute, storage, verification, and human effort as ACTUAL, ESTIMATED, or UNKNOWN. Missing cost is not zero; no successful outcomes means the ratio is undefined.
Keep guardrails visible
Compare quality, recovery, cost, latency, and human intervention together. Trace the critical path and downstream operations per admitted Mission. A failed authority or security gate cannot be averaged away.
Start with your baseline