Prometheus is a tool for monitoring containerized environment like kubernetes and docker swarm. It could also be used to monitor traditional non-containerized environments like Linux servers
Prometheus helps us monitor performance, detect slowness, errors, hardware issues or low resources for applications running on multiple interconnected servers.
When a server runs out of memory, it stops the application that is consuming all the resources, if that application offer authentication for a database server, users can't login to the database anymore.
Prometheus helps us diagnose the issue and get to its root cause.
Prometheus alerts the admin when something goes wrong through notifications and alerts.
Alerts are triggered according to the parameters we set in the Prometheus configuration file (CPU usage > 80% or RAM usage > 50%, ...)
Data retriever worker: it pulls metric data (CPU usage, number of queries, number of requests,...) from applications, servers,..
Time series database: stores data metrics.
HTTP server : handles PromQL queries (PromDL is a language that can read Prometheus queries)
Prometheus can monitor specific applications or entire servers, these are called targets.
Units are what we monitor for a specific target (application, server,..)
Units could be for example:
Disk space usage.
number of exceptions triggered.
Number of processes.
The units are pulled from the targets and stored in the time series database.
Units that we want to monitor on a target are called metrics.
Type of metrics:
Counter metrics: serve as a counter of events (number of exceptions or number of requests,...).
Gauge metrics: for metrics that change over time (CPU usage, RAM consumption,...)
Histogram metrics: are for metrics that can be measured via their duration and size (time a request takes, size of the request,...).
Gathering of metrics data from targets:
Prometheus through the data retrieval worker retrieves data from the target through HTTP (from the endpoint, "/metric") at the address: "host_name/metrics".
Some services have native Prometheus endpoints, but some services need and exporter to make the endpoints available for Prometheus.
Exporter is a program that gets the metrics from the target and converts them into a format decipherable by Prometheus.
Metrics are made available by the exporter in the its metric endpoint (/metrics), so Prometheus can retrieve it.
Exporters are available as docker images, we can use them as a "sidecar" container along our applications inside a pod.
The exporter container will get data from the database application and make it available from Prometheus through his endpoint /metrics.
There are some programs that don't run long enough for Prometheus to be able to retrieve metrics from them, for example programs that do cleanup tasks and exit.
To be able to monitor this kind of applications, Prometheus uses a push gateway.
The program pushes its metrics when it exits to the push gateway.
Prometheus configuration file:
To know which information Prometheus needs to pull, which targets it should scrape or the scraping intervals,..., Prometheus uses a configuration file called Prometheus.yaml
The rules in the rules_files section, are for combining metrics or for creating alerts when certain things happen, either resources go low or the CPU gets overloaded,.... The scrape_configs section lists the targets Prometheus needs to monitor. In the default file above, it monitors its own metrics at localhost:9000
To trigger alerts, Prometheus has an alert manager, the alerts could come in the form of e-mails, messages, ...
Prometheus data storage:
Prometheus stores its data on disks in a specific format.
We can query data from Prometheus through its API server using the PromQL query language