Skip to content

Prometheus Metrics

This document lists all Prometheus metrics exposed by koala applications.

Configuration

Enable metrics in your configuration:

default_prometheus_export_metrics_enabled = true
default_prometheus_client_app_port = 9100

The metrics endpoint is served at http://<host>:<port>/metrics (default port 9100).

Loader Metrics

koala_loader_nr_of_sips_processed

Property Value
Type Counter
Description Number of SIPs processed
Labels None
When incremented At the start of each do_work() cycle, after receiving and acknowledging the message from RabbitMQ

Example:

# HELP koala_loader_nr_of_sips_processed Number of SIPs processed
# TYPE koala_loader_nr_of_sips_processed counter
koala_loader_nr_of_sips_processed 15234

koala_loader_nr_of_unhandled_exceptions

Property Value
Type Counter
Description Number of unhandled exceptions
Labels None
When incremented When an unexpected exception causes the loader to shut down via sys.exit(EXIT_FATAL)

Example:

# HELP koala_loader_nr_of_unhandled_exceptions Number of unhandled exceptions
# TYPE koala_loader_nr_of_unhandled_exceptions counter
koala_loader_nr_of_unhandled_exceptions 3

koala_loader_nr_of_error_states_executed

Property Value
Type Counter
Description Number of times the loader error state was executed
Labels None
When incremented In LoaderErrorState.run()

Example:

# HELP koala_loader_nr_of_error_states_executed Number of times the loader error state was executed
# TYPE koala_loader_nr_of_error_states_executed counter
koala_loader_nr_of_error_states_executed 42

koala_loader_ingest_hist_seconds

Property Value
Type Histogram
Description Ingest duration in seconds
Labels None
Buckets 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, +Inf
When observed After each successful ingest, observing (end - start).total_seconds()

Example:

# HELP koala_loader_ingest_hist_seconds Ingest duration in seconds
# TYPE koala_loader_ingest_hist_seconds histogram
koala_loader_ingest_hist_seconds_bucket{le="0.1"} 12
koala_loader_ingest_hist_seconds_bucket{le="0.25"} 45
koala_loader_ingest_hist_seconds_bucket{le="0.5"} 123
koala_loader_ingest_hist_seconds_bucket{le="0.75"} 234
koala_loader_ingest_hist_seconds_bucket{le="1"} 456
koala_loader_ingest_hist_seconds_bucket{le="2.5"} 890
koala_loader_ingest_hist_seconds_bucket{le="5"} 1234
koala_loader_ingest_hist_seconds_bucket{le="10"} 1500
koala_loader_ingest_hist_seconds_bucket{le="30"} 1600
koala_loader_ingest_hist_seconds_bucket{le="60"} 1650
koala_loader_ingest_hist_seconds_bucket{le="120"} 1680
koala_loader_ingest_hist_seconds_bucket{le="300"} 1690
koala_loader_ingest_hist_seconds_bucket{le="600"} 1692
koala_loader_ingest_hist_seconds_bucket{le="+Inf"} 1693
koala_loader_ingest_hist_seconds_sum 45678.5
koala_loader_ingest_hist_seconds_count 1693

koala_loader_ingest_transfer_rate

Property Value
Type Gauge
Description Transfer rate in bytes per second
Labels None
When set After each successful ingest, when cargo.aip_size is available. Set to aip_size / total_seconds.

Example:

# HELP koala_loader_ingest_transfer_rate Transfer rate in bytes per second
# TYPE koala_loader_ingest_transfer_rate gauge
koala_loader_ingest_transfer_rate 5242880

Purger Metrics

koala_purger_nr_of_recoveries_processed

Property Value
Type Counter
Description Number of purger recovery operations processed
Labels None
When incremented Each time the purger completes a recovery operation

Metadata Metrics

koala_metadata_nr_of_files_processed

Property Value
Type Counter
Description Number of files processed during metadata operations
Labels None
When incremented During file validation and checksum calculation

Query Examples

Ingest Rate (per minute)

rate(koala_loader_nr_of_sips_processed[5m]) * 60

Ingest Duration (p95)

histogram_quantile(0.95, rate(koala_loader_ingest_hist_seconds_bucket[5m]))

Ingest Duration (p50)

histogram_quantile(0.50, rate(koala_loader_ingest_hist_seconds_bucket[5m]))

Error Rate (per minute)

rate(koala_loader_nr_of_error_states_executed[5m])

Unhandled Exception Rate

rate(koala_loader_nr_of_unhandled_exceptions[5m])

Average Transfer Rate

avg(koala_loader_ingest_transfer_rate)

Total Ingested Bytes (last hour)

sum(rate(koala_loader_nr_of_sips_processed[1h])) * avg(koala_loader_ingest_hist_seconds_sum) / avg(koala_loader_ingest_hist_seconds_count)

Alerting Rules

High Error Rate

- alert: HighIngestErrorRate
  expr: rate(koala_loader_nr_of_error_states_executed[5m]) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High ingest error rate"
    description: "Loader error rate is {{ $value }} errors/sec"

Loader Crash

- alert: LoaderCrash
  expr: increase(koala_loader_nr_of_unhandled_exceptions[1m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Loader crash detected"
    description: "Unhandled exception in loader"

Slow Ingest

- alert: SlowIngest
  expr: histogram_quantile(0.95, rate(koala_loader_ingest_hist_seconds_bucket[5m])) > 300
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Slow ingest detected"
    description: "95th percentile ingest duration is {{ $value }} seconds"