The Dangers of High-Cardinality Labels in Prometheus

We're all familiar with the warnings: " Don't use user_id as a Prometheus label " or " Don't use transaction codes as labels — they can crash Prometheus ". But do we really understand why these are so dangerous? Before that, we need to know how Prometheus works. How Prometheus Works Prometheus is an open-source systems monitoring and alerting tool that collects and stores its metrics as time-series data. It periodically scrapes metrics from your services based on the configured interval. This is an example of the config: scrape_configs: - job_name: 'golang-app' static_configs: - targets: ['localhost:8080'] scrape_interval: 5s This config will tell Prometheus to: Target : send an HTTP request GET to http://localhost:8080/metrics Periodically : for every 5 seconds Label : with job=golang-app Prometheus has three metric types: Gauges represent current measurements and reflect the current state of a system, such as CPU usage and memory usage. Counters measure discrete events that continuou