Choosing threshold values
Start with your observed values and add headroom for normal variation:
| Metric | Observed | Threshold | Headroom |
|---|
| p95 latency | 320 ms | p(95)<500 | ~55% above observed |
| Error rate | 0% | rate<0.01 | Allows up to 1% |
| Check pass rate | 100% | rate>0.99 | Allows up to 1% failure |
Trade-offs in threshold decisions
Threshold-setting involves judgment, not just formulas. Consider these trade-offs for your system:
| Decision | Tighter threshold | Looser threshold |
|---|
| Headroom | Catches small regressions early | Avoids false failures from normal variation |
| p95 vs p99 | p95 reflects most users’ experience | p99 catches tail latency affecting your slowest users |
| Error tolerance | rate<0.001 is strict, good for payment flows | rate<0.01 is practical for non-critical endpoints |
A common starting approach: use p95 with 30-50% headroom for general APIs. Switch to p99 with tighter headroom for latency-sensitive paths like checkout or authentication.
