Single binary meta-monitoring

Meta monitoring for monolithic mode deployments involves some additional configuration. This approach does not use the Kubernetes Monitoring Helm chart.

Metrics

Configure Alloy or Prometheus to scrape the Loki metrics endpoint, adding the additional labels that are expected by the mixin dashboards, alerts, and recording rules:

Alloy

prometheus.scrape "loki" {
  targets = [{
    __address__ = "localhost:3100",
    cluster     = "prod",
    namespace   = "default",
    job         = "default/loki-single-binary",
    pod         = "loki-single-binary",
    container   = "loki",
  }]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {  
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"
  }  
}

Prometheus

scrape_configs:
  - job_name: loki
    static_configs:
      - targets: ['localhost:3100']
        labels:
          cluster: prod
          namespace: default
          job: default/loki-single-binary
          pod: loki-single-binary
          container: loki

Mixins

To generate the mixins:

Install jb.
Install mixtool.

Clone the Loki repository from Github.

git clone https://github.com/grafana/loki
cd loki

Create the output directory.

mkdir loki/production/loki-mixin-compiled-single-binary

Navigate to the mixin directory.
Bash
```
cd loki/production/loki-mixin
```
Use jb to install mixin dependencies and generate the vender directory.
Bash
```
jb install
```

Create the mixin configuration file single-binary.libsonnet.

cat <<EOF > single-binary.libsonnet
local loki = import 'mixin.libsonnet';

loki + {
  _config+:: {
    meta_monitoring: {
      enabled: true,
    },
    canary+: {
      // Whether or not to include the loki-canary dashboard
      enabled: true
    },
    promtail: {
      // Whether or not to include promtail specific dashboards
      enabled: false,
    },
    // Tunes histogram recording rules to aggregate over this interval.
    // Set to at least twice the scrape interval; otherwise, recording rules will output no data.
    // Set to four times the scrape interval to account for edge cases: https://www.robustperception.io/what-range-should-i-use-with-rate/        
    recording_rules_range_interval: '5m'
  }  
}
EOF

Generate dashboards, alerts, and recording rules.

mixtool generate all \
  --output-alerts ../loki-mixin-compiled-single/alerts.yaml \
  --output-rules ../loki-mixin-compiled-single/rules.yaml \
  --directory ../loki-mixin-compiled-single/dashboards \
  single-binary.libsonnet

See the generated dashboards, alerts, and recording rules in the loki/production/loki-mixin-compiled-single-binary directory.
Follow the instructions in the Install Mixins documentation to set up the dashboards, alerts, and recordings.