Prometheus is a tool for monitoring containerized environment like kubernetes and docker swarm. It could also be used to monitor traditional non-containerized environments like Linux servers

Prometheus helps us monitor performance, detect slowness, errors, hardware issues or low resources for  applications running on multiple interconnected servers.

Example:

When a server runs out of memory, it stops the application that is consuming all the resources, if that application offer authentication for a database server, users can't login to the database anymore.

Prometheus helps us diagnose the issue and get to its root cause.

Prometheus alerts the admin when something goes wrong through notifications and alerts.

Alerts are triggered according to the parameters we set in the Prometheus configuration file (CPU usage > 80% or RAM usage > 50%, ...)

Prometheus architecture:

Data retriever worker: it pulls metric data (CPU usage, number of queries, number of requests,...) from applications, servers,.. 

Time series database: stores data metrics.

HTTP server : handles PromQL queries (PromDL is a language that can read Prometheus queries) 

Targets : 

Prometheus can monitor specific applications or entire servers, these are called targets.

Units:

Units are what we monitor for a specific target (application, server,..)

Units could be for example:

• CPU status.
• Memory usage.
• Disk space usage.
• number of exceptions triggered.
• Number of processes.

Prometheus through the data retrieval worker retrieves data from the target through HTTP (from the endpoint, "/metric") at the address: "host_name/metrics".

Some services have native Prometheus endpoints, but some services need and exporter to make the endpoints available for Prometheus.

Exporter:

Exporter is a program that gets the metrics from the target and converts them into a format decipherable by Prometheus.

Metrics are made available by the exporter in the its metric endpoint (/metrics), so Prometheus can retrieve it.

Docker example:

Exporters are available as docker images, we can use them as a "sidecar" container along our applications inside a pod.

The exporter container will get data from the database application and make it available from Prometheus through his endpoint /metrics.

Push gateway:

There are some programs that don't run long enough for Prometheus to be able to retrieve metrics from them, for example programs that do cleanup tasks and exit.

To be able to monitor this kind of applications, Prometheus uses a push gateway. 

The program pushes its metrics when it exits to the push gateway.

.