Best practices for policies

Adaptive Traces gives you the power to control your observability costs and reduce noise by keeping only the traces that provide value. An effective policy configuration is key to achieving this balance.

This guide outlines the best practices for creating and evaluating policies.

The core evaluation logic

Before building, it’s crucial to understand how policies are evaluated.

Drop always wins: If a trace matches any policy configured to drop it, it is immediately discarded. This rule has the highest priority and overrides all others.
Keep if any match: If a trace is not dropped, it only needs to match one keep policy to be saved. Think of your keep policies as a large “OR” condition.
Default is drop: If a trace matches no policies at all (neither keep nor drop), it is discarded.

Best Practice: Build your policies strategically

Don’t create rules randomly. Follow a layered approach for the best results.

Set a probabilistic keep policy

Set a probabilistic (or percentage-based) sampling rule to give you a representative sample of your “normal” day-to-day traffic without incurring the cost of keeping everything.

Start Low: Begin with a low percentage, such as 5% or 10%. You can always increase it later if you find you need more data.
Example: Create a Probabilistic policy to keep 10% of all remaining traces.

Define specific keep policies

Ensure you never lose your most critical signals. These policies guarantee that traces related to incidents, errors, and poor performance are always captured.

Always keep errors: The most important rule. Keep 100% of traces that have a status code of Error.
Capture high latency: Define what slow means for your key services and create policies to keep any trace that exceeds that threshold, for example, latency > 2 seconds.
Example:

Create a Status Code policy to keep all traces where the status is ERROR.

Create drop policies

Use drop policies to eliminate high-volume, low-value traffic that you never need to see, regardless of whether it’s anomalous.

Drop policies have the highest evaluation priority. A trace that matches a drop policy is immediately discarded, and no other policy (including anomaly or diversity) can override that decision. This makes them powerful but potentially dangerous if misused.

Good use cases:

Health checks and readiness probes: Endpoints like /healthz, /ready, and /metrics generate constant, homogeneous traffic with no diagnostic value. Dropping them is safe because there is no interesting anomaly hiding in a synthetic 200 response.
Synthetic test traffic: If your load tests or canary deploys tag traces with a known attribute (for example, app.synthetic: true), a drop policy ensures they never consume your trace budget.
Enforcing required attributes: Drop traces that are missing a mandatory attribute like deployment.environment to keep your data clean.

Example: Create a String Attribute policy to drop all traces where url.path matches /healthz or /metrics.

Antipattern — don’t use drop policies for rate limiting:

It’s tempting to use a drop policy to reduce volume from a noisy but important endpoint (for example, /api/checkout). However, this creates a blind spot: if that endpoint starts returning errors or experiences a latency spike, those traces are silently discarded before anomaly detection can catch them.

Instead, use a probabilistic sampling policy scoped to that traffic. For example, create a composite policy that matches the high-volume attribute and applies a low sampling rate (for example, 5%). This reduces throughput while still giving anomaly and diversity policies a chance to rescue interesting traces that fall outside the sample.

Rule of thumb: If the traffic is never useful, drop it. If the traffic is just noisy, sample it.

Best Practice: Continuously review and refine

As your application and traffic patterns changes, your policies should, too.

Monitor policy impact: Use metrics to see how many traces each policy is keeping or dropping. If a rule is unexpectedly noisy, tighten its conditions.
Use precise attributes: Avoid overly broad rules. Instead of matching a URL with a containsoperator, use equals on a specific attribute like http.target or rpc.method for more predictable results.