Setting Up a Prometheus Monitoring Stack

1. **What you'll need**

To set up a Prometheus monitoring stack, you'll need Docker (version 20.10 or later) and docker-compose (version 1.29 or later) installed on your system. You'll also need a basic understanding of Linux command line interfaces.

2. **Step 1: Create a new directory and initialize the Prometheus configuration**

Create a new directory for your Prometheus setup and navigate into it: mkdir prometheus-setup && cd prometheus-setup. Initialize a basic Prometheus configuration file prometheus.yml with the following content:

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

This configuration tells Prometheus to scrape itself every 10 seconds.

3. **Step 2: Create a docker-compose file**

Create a new file named docker-compose.yml with the following content:

version: '3'
services:
  prometheus:
    image: prometheus/prometheus:v2.34.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

This file defines a Prometheus service using the official Prometheus image (version 2.34.0) and maps the prometheus.yml configuration file to the container.

4. **Step 3: Start the Prometheus service**

Start the Prometheus service using the following command: docker-compose up -d. The -d flag runs the service in detached mode.

5. **Step 4: Verify the Prometheus web interface**

Open a web browser and navigate to http://localhost:9090. You should see the Prometheus web interface, which provides a dashboard for exploring metrics and creating alerts.

6. **Step 5: Add a node exporter**

To monitor system metrics, you'll need to add a node exporter. Create a new service in the docker-compose.yml file:

  node-exporter:
    image: prometheus/node-exporter:v1.3.1
    ports:
      - "9100:9100"

Then, update the prometheus.yml file to scrape the node exporter:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Restart the Prometheus service using docker-compose restart.

7. **Verify it works**

Verify that the node exporter is being scraped by navigating to http://localhost:9090/targets. You should see the node exporter listed as an active target.

8. **Common errors**

* Error 1: Unable to connect to Prometheus

If you encounter an error connecting to Prometheus, check that the prometheus.yml file is correctly formatted and that the docker-compose service is running.

* Error 2: Node exporter not being scraped

If the node exporter is not being scraped, check that the prometheus.yml file is correctly configured and that the node exporter service is running.

9. **Next steps**

Now that you have a basic Prometheus monitoring stack set up, you can:

* Explore the Prometheus web interface to learn more about the available metrics and alerts.

* Add additional exporters to monitor other system components, such as databases or web servers.

* Configure alerts to notify you of potential issues before they become critical.