This is documentation for the next version of Grafana Alloy Documentation. For the latest stable release, go to the latest version.

Documentation Grafana Alloy Reference Components otelcol otelcol.connector.spanmetrics

otelcol.connector.spanmetrics

otelcol.connector.spanmetrics accepts span data from other otelcol components and aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from the spans:

• Request counts are computed as the number of spans seen per unique set of dimensions, including Errors. Multiple metrics can be aggregated if, for instance, a user wishes to view call counts just on service.name and span.name.

  Requests are tracked using a calls metric with a status.code datapoint attribute set to Ok:

  text Copy

  calls { service.name="shipping", span.name="get_shipping/{shippingId}", span.kind="SERVER", status.code="Ok" }

• Error counts are computed from the number of spans with an Error status code.

  Errors are tracked using a calls metric with a status.code datapoint attribute set to Error:

  text Copy

  calls { service.name="shipping", span.name="get_shipping/{shippingId}, span.kind="SERVER", status.code="Error" }

• Duration is computed from the difference between the span start and end times and inserted into the relevant duration histogram time bucket for each unique set dimensions.

  Span durations are tracked using a duration histogram metric:

  text Copy

  duration { service.name="shipping", span.name="get_shipping/{shippingId}", span.kind="SERVER", status.code="Ok" }

You can specify multiple otelcol.connector.spanmetrics components by giving them different labels.

Usage

Alloy Copy

otelcol.connector.spanmetrics "<LABEL>" {
  histogram {
    ...
  }

  output {
    metrics = [...]
  }
}

Arguments

You can use the following arguments with otelcol.connector.spanmetrics:

The supported values for aggregation_temporality are:

• "CUMULATIVE": The metrics won’t be reset after they’re flushed.
• "DELTA": The metrics will be reset after they’re flushed.

If namespace is set, the generated metric name will be added a namespace. prefix.

Setting metrics_expiration to "0s" means that the metrics will never expire.

resource_metrics_cache_size is mostly relevant for cumulative temporality. It helps avoid issues with increasing memory and with incorrect metric timestamp resets.

metric_timestamp_cache_size is only relevant for delta temporality span metrics. It controls the size of a cache used to keep track of the last time a metric was flushed. When a metric is evicted from the cache, its next data point will indicate a “reset” in the series. Downstream components converting from delta to cumulative may handle these resets by setting cumulative counters back to 0.

resource_metrics_key_attributes can be used to avoid situations where resource attributes may change across service restarts, causing metric counters to break (and duplicate). A resource doesn’t need to have all of the attributes. The list must include enough attributes to properly identify unique resources or risk aggregating data from more than one service and span. For example, ["service.name", "telemetry.sdk.language", "telemetry.sdk.name"].

When the aggregation_cardinality_limit limit is reached, additional unique combinations will be dropped but registered under a new entry with otel.metric.overflow="true". A value of 0 means no limit is applied.

Blocks

You can use the following blocks with otelcol.connector.spanmetrics:

You must specify either an exponential or an explicit block. You can’t specify both blocks in the same configuration.

histogram

The histogram block configures the histogram derived from spans’ durations.

The following attributes are supported:

The supported values for unit are:

• "ms": milliseconds
• "s": seconds

output

debug_metrics

dimension

The dimension block configures dimensions to be added in addition to the default ones.

The default dimensions are:

• service.name
• span.name
• span.kind
• status.code

The default dimensions are always added if not listed in exclude_dimensions. If no additional dimensions are specified, only the default ones will be added.

The following attributes are supported:

otelcol.connector.spanmetrics looks for the name attribute in the span’s collection of attributes. If it’s not found, the resource attributes will be checked.

If the attribute is missing in both the span and resource attributes:

• If default isn’t set, the dimension will be omitted.
• If default is set, the dimension will be added and its value will be set to the value of default.

calls_dimension

The attributes and behavior of the calls_dimension block match the dimension block.

events

The events block configures the events metric, which tracks span events.

The following attributes are supported:

At least one nested dimension block is required if enabled is set to true.

exemplars

The exemplars block configures how to attach exemplars to histograms.

The following attributes are supported:

max_per_data_point can help with reducing memory consumption.

exponential

The exponential block configures a histogram with exponential buckets.

The following attributes are supported:

explicit

The explicit block configures a histogram with explicit buckets.

The following attributes are supported:

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer traces telemetry data. It doesn’t accept metrics and logs.

Handle resource attributes

otelcol.connector.spanmetrics is an OTLP-native component. As such, it aims to preserve the resource attributes of spans.

1. For example, assume that there are two incoming resources spans with the same service.name and k8s.pod.name resource attributes.

   Example JSON of two incoming spans.

2. otelcol.connector.spanmetrics will preserve the incoming service.name and k8s.pod.name resource attributes by attaching them to the output metrics resource. Only one metric resource will be created, because both span resources have identical resource attributes.

   Example JSON of one outgoing metric resource.

3. Now, assume that otelcol.connector.spanmetrics receives two incoming resource spans, each with a different value for the k8s.pod.name recourse attribute.

   Example JSON of two incoming spans.

4. To preserve the values of all resource attributes, otelcol.connector.spanmetrics will produce two resource metrics. Each resource metric will have a different value for the k8s.pod.name recourse attribute. This way none of the resource attributes will be lost during the generation of metrics.

   Example JSON of two outgoing metric resources.

Component health

otelcol.connector.spanmetrics is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.connector.spanmetrics doesn’t expose any component-specific debug information.

Examples

Explicit histogram and extra dimensions

In the example below, http.status_code and http.method are additional dimensions on top of:

• service.name
• span.name
• span.kind
• status.code

Alloy Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    traces  = [otelcol.connector.spanmetrics.default.input]
  }
}

otelcol.connector.spanmetrics "default" {
  // Since a default is not provided, the http.status_code dimension will be omitted
  // if the span does not contain http.status_code.
  dimension {
    name = "http.status_code"
  }

  // If the span is missing http.method, the connector will insert
  // the http.method dimension with value 'GET'.
  dimension {
    name = "http.method"
    default = "GET"
  }

  dimensions_cache_size = 333

  aggregation_temporality = "DELTA"

  histogram {
    unit = "s"
    explicit {
      buckets = ["333ms", "777s", "999h"]
    }
  }

  // The period on which all metrics (whose dimension keys remain in cache) will be emitted.
  metrics_flush_interval = "33s"

  namespace = "test.namespace"

  output {
    metrics = [otelcol.exporter.otlphttp.production.input]
  }
}

otelcol.exporter.otlphttp "production" {
  client {
    endpoint = sys.env("OTLP_SERVER_ENDPOINT")
  }
}

Send metrics via a Prometheus remote write

The generated metrics can be sent to a Prometheus-compatible database such as Grafana Mimir. However, extra steps are required to make sure all metric samples are received. This is because otelcol.connector.spanmetrics aims to preserve resource attributes in the metrics which it outputs.

Unfortunately, the Prometheus data model has no notion of resource attributes. This means that if otelcol.connector.spanmetrics outputs metrics with identical metric attributes, but different resource attributes, otelcol.exporter.prometheus converts the metrics into the same metric series. This problem can be solved by doing either of the following:

• Recommended approach: Prior to otelcol.connector.spanmetrics, remove all resource attributes from the incoming spans which aren’t needed by otelcol.connector.spanmetrics.

  Example configuration to remove unnecessary resource attributes.

• Or, after otelcol.connector.spanmetrics, copy each of the resource attributes as a metric datapoint attribute. This has the advantage that the resource attributes will be visible as metric labels. However, the cardinality of the metrics may be much higher, which could increase the cost of storing and querying them. The example below uses the merge_maps OTTL function.

  Example configuration to add all resource attributes as metric datapoint attributes.

If the resource attributes aren’t treated in either of the ways described above, an error such as this one could be logged by prometheus.remote_write: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp).

Troubleshoot high cardinality span metrics

High cardinality issues in span metrics commonly manifest in APM dashboards as an excessive number of service operations with non-unique names. Examples include URIs with unique identifiers such as GET /product/1YMWWN1N4O or HTTP parameters with random values such as GET /?_ga=GA1.2.569539246.1760114706. These patterns render operation lists difficult to interpret and ineffective for monitoring purposes.

This issue stems from violations of OpenTelemetry semantic conventions, which require span names to have low cardinality. Refer to HTTP span name specs for more information. Beyond degrading APM interfaces with numerous non-meaningful operation names, this problem causes metric time series explosion, resulting in significant performance degradation and increased costs.

otelcol.connector.spanmetrics provides an optional circuit breaker through the aggregation_cardinality_limit attribute, disabled by default, to mitigate cardinality explosion. While this feature addresses performance and cost concerns, it doesn’t resolve the underlying issue of semantically meaningless operation names.

Fix high cardinality span name issues

The ideal long-term solution is to modify the OpenTelemetry instrumentation code to comply with semantic conventions. This prevents the generation of non-compliant high cardinality span names.

However, deploying updated instrumentation libraries can be time-consuming. You often need an immediate interim solution to restore observability backend functionality.

Address high cardinality span names in the ingestion pipeline

An effective short-term solution is to implement a layer that sanitizes span names within the observability ingestion pipeline. You can use otelcol.processor.transform’s set_semconv_span_name() function immediately before otelcol.connector.spanmetrics to enforce semantic conventions on span names.

Simple example configuration to prevent span metrics cardinality explosion

Aggressive span name cleanup may be overly restrictive for instrumentation libraries with incomplete resource attributes. For instance, the cleanup may reduce HTTP service operations to generic names like GET and POST when HTTP spans lack the http.route attribute. This information loss can impact the monitoring of critical business operations.

To preserve operation granularity, you can manually set the http.route attribute when you need detailed operation names. You can typically derive the missing http.route value through pattern matching on other span attributes such as http.target or url.full.

The following example configuration prevents cardinality explosion while preserving meaningful operation names on a service webshop/frontend:

Alloy Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    traces = [otelcol.processor.transform.sanitize_spans.input]
  }
}

// Sanitize spans to prevent span metrics cardinality explosion caused by
// non-compliant high cardinality span names:
// 1. Fix incomplete semconv of critical operation spans to keep meaningful
//    span metrics operation names, adding missing `http.route` and
//    `http.request.method`.
// 2. Sanitize all span names, note that http server spans lacking
//    `http.route` will default to operations `GET`, `POST`, etc.
otelcol.processor.transform "sanitize_spans" {
  error_mode = "ignore"

  // 1. Fix incomplete semconv on the critical http operations of the `frontend` service
  trace_statements {
    context    = "span"
    conditions = [
      `span.kind == SPAN_KIND_SERVER and resource.attributes["service.name"] == "frontend" and resource.attributes["service.namespace"] == "webshop" and span.attributes["http.route"] == nil`,
    ]
    statements = [
      // e.g. /api/checkout
      `set(span.attributes["http.route"], "/api/checkout") where IsMatch(span.attributes["http.target"], "\\/api\\/checkout")`,
      // e.g. /api/products/1YMWWN1N4O
      `set(span.attributes["http.route"], "/api/products/{productId}") where IsMatch(span.attributes["http.target"], "\\/api\\/products\\/.*")`,
    ]
  }

  // 1. Fix incomplete semconv on the critical http operations of other services...

  // 2. Sanitize all span names to prevent span metrics cardinality explosion.
  //    Unsanitized span names, when different, are kept in the `unsanitized_span_name` attribute
  trace_statements {
    context = "span"
    statements = [
      `set_semconv_span_name("1.37.0", "unsanitized_span_name")`,
    ]
  }

  output {
    traces = [otelcol.connector.spanmetrics.default.input]
  }
}

otelcol.connector.spanmetrics "default" {
  histogram {
    explicit {}
  }

  output {
    metrics = [otelcol.exporter.otlphttp.observability_backend.input]
  }
}

otelcol.exporter.otlphttp "observability_backend" {
  client {
    endpoint = sys.env("OTLP_SERVER_ENDPOINT")
  }
}

Address high cardinality span names in the instrumentation code

The preferred long-term solution is to ensure span names and attributes comply with OpenTelemetry Semantic Conventions directly in the instrumentation code.

Custom web frameworks are a common source of high cardinality span names. While default OpenTelemetry instrumentation, for example, Java Servlet, may assign generic span names like GET /my-web-fwk/*, your framework has access to more specific routing information. Overwrite span attributes in your framework code to create compliant, low-cardinality span names that preserve operational granularity.

Example: Custom Web Framework in Java

Consider a custom web framework that intercepts the generic route /my-web-fwk/* and dispatches requests like /my-web-fwk/product/123456ABCD or /my-web-fwk/user/john.doe.

The default Java Servlet instrumentation produces vague span names (GET /my-web-fwk/*), while directly using request URIs creates high cardinality (GET /my-web-fwk/product/123456ABCD).

The solution is to override span attributes with templated route patterns like /my-web-fwk/product/{productId} or /my-web-fwk/user/{userId}:

java Copy

@WebServlet(urlPatterns = "/my-web-fwk/*")
public class MyWebFrameworkServlet extends HttpServlet {

    @Override
    protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        // Default Servlet instrumentation sets vague span names: `GET /my-web-fwk/*` (and `http.route=/my-web-fwk/*`)
        // Using the request URI directly would cause high cardinality and violate semantic conventions
        // Instead, use the framework's low-cardinality routing information below

        // Example routing logic
        String uri = request.getRequestURI();
        MyWebOperation myWebOperation = getWebOperation(uri);

        // Fix span details to add details while complying with semantic conventions and maintaining low cardinality
        String httpRoute = "/my-web-fwk/" + myWebOperation.getSubHttpRoute();
        Span.current().setAttribute(HttpAttributes.HTTP_ROUTE, httpRoute);
        Span.current().updateName(request.getMethod() + " " + httpRoute);

        // execute the web operation
        myWebOperation.execute(request, response);
    }
    ...
}

Compatible components

otelcol.connector.spanmetrics can accept arguments from the following components:

• Components that export OpenTelemetry otelcol.Consumer

otelcol.connector.spanmetrics has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer