Setup Monitoring for your homelab environment

In this guide, we’ll set up monitoring tools to oversee the operation of your homelab, specifically for Proxmox, VMs, and Docker containers.

Setup InfluxDB

We’ll begin by setting up monitoring for the core of the infrastructure, the hypervisor.

Follow the docker-compose.yml and .env file configuration to set up your InfluxDB container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
version: '3.9'

services:
# Influx Database
  influxdb:
    container_name: influxdb
    image: influxdb:2.7.1-alpine
    restart: unless-stopped
    # Custom DNS for container here:
    #dns:
    #  - 1.1.1.1
    ports:
      - '8086:8086'
    volumes:
      - './influxdb-data:/var/lib/influxdb2'
      - './etc:/etc/influxdb2'
      # # If you're using self-signed certs, uncomment the lines below
      # - /etc/ssl/cert.pem/:/etc/ssl/cert.pem  
      # - /etc/ssl/cert-key.pem/:/etc/ssl/cert-key.pem
    # command: influxd --tls-cert=/etc/ssl/cert.pem --tls-key=/etc/ssl/cert-key.pem 
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUX_USER}
      - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUX_PASS}
      - DOCKER_INFLUXDB_INIT_ORG=${INFLUX_ORG}
      - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUX_BUCKET}
      - DOCKER_INFLUXDB_INIT_RETENTION=1w  #configure data retention by week count
      # The authentication token to associate with the system's initial super-user. If not set, a token will be auto-generated by the system.
    #   - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
    # Traefik Example:
    #labels:
    #  - "traefik.enable=true"
    #  - "traefik.http.routers.influx.entrypoints=http"
    #  - "traefik.http.routers.influx.rule=Host(`influx.cinderblock.tech`)"
    #  - "traefik.http.middlewares.influx-https-redirect.redirectscheme.scheme=https"
    #  - "traefik.http.routers.influx.middlewares=influx-https-redirect"
    #  - "traefik.http.routers.influx-secure.entrypoints=https"
    #  - "traefik.http.routers.influx-secure.rule=Host(`influx.cinderblock.tech`)"
    #  - "traefik.http.routers.influx-secure.tls=true"
    #  - "traefik.http.routers.influx-secure.service=influx"
    #  - "traefik.http.services.influx.loadbalancer.server.port=8086"
    #  - "traefik.docker.network=proxy"    
    
    networks:
      - proxy
      - monitoring

# I'll be using Proxy for my external connection, and monitoring for db to db network communication
networks:
 proxy:
  external: true
 monitoring:
  external: false
  

Next, set your environment variables either in your environment or with a .env file in the same folder.

1
2
3
4
INFLUX_USER=
INFLUX_PASS=
INFLUX_ORG=
INFLUX_BUCKET=

Spin it up!

Setup Proxmox Data API

After setting up and running InfluxDB (secured behind an SSL certificate), go to your Proxmox instance.

Navigate to Datacenter --> Metric Server --> Add --> InfluxDB and provide the necessary information for your InfluxDB server.

You will get the token from the dashboard of your influxdb. Navigate to Load Data --> API Token

Note: If your Proxmox instance does not have a CA certificate, you’ll encounter an SSL error. To resolve this error, you can use Cloudflare to get your CA certificate, and then install it on Proxmox.

If your network uses a proxy that forwards all HTTP requests to HTTPS (like Traefik), you can simply set the protocol to HTTP, and Traefik will handle the rest. This allows you to skip the next part about getting a Cloudflare CA.

Getting your Cloudflare CA

Navigate to Cloudflare, go into your ‘Website’ and drop down into SSL/TLS --> Origin Server. Create a CA using RSA. Use the resulting CA and CA-Key to create respective files:

ca-key.pem
ca.pem

Now, we are going to copy the needed CA to Proxmox to allow it to send metrics to our Influx database.

1
scp `proxmox-user`@`proxmox-url`:/usr/local/share/ca-certificates/ca.crt

Then ssh into your Proxmox server, and update the CA-Certificates

1
update-ca-certificates

Setup Grafana for data visbility

Now that we have InfluxDB setup, and Proxmox is sending its data there, lets setup the visual part of the process.

create your docker-compose.yml file and input the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
version: '3.9'

services:
  grafana:
    image: grafana/grafana:10.0.1
    user: "0" # Set this to prevent permission errors during Grafana deployment
    container_name: grafana-monitor
    #dns:
    #  - 1.1.1.1
    ports:
      - "3020:3000"
    volumes:
      - ./grafana-data:/var/lib/grafana
    #environment:
      #- "GF_SERVER_ROOT_URL=http://grafana.cinderblock.tech/"
      #- GF_INSTALL_PLUGINS=grafana-clock-panel
      #- GF_DEFAULT_INSTANCE_NAME=homelab-monitor
      #- GF_SECURITY_ADMIN_USER=austin
      #- GF_AUTH_GOOGLE_CLIENT_SECRET=${GRAF_SECRET}
      #- GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS=true
      #- GF_FEATURE_TOGGLES_ENABLE=newNavigation
    restart: unless-stopped
    # Example Traefik config
    #labels:
      #- "traefik.enable=true"
      #- "traefik.http.routers.grafana-monitor.entrypoints=http"
      #- "traefik.http.routers.grafana-monitor.rule=Host(`grafana-monitor.cinderblock.tech`)"
      #- "traefik.http.middlewares.grafana-monitor-https-redirect.redirectscheme.scheme=https"
      #- "traefik.http.routers.grafana-monitor.middlewares=grafana-monitor-https-redirect"
      #- "traefik.http.routers.grafana-monitor-secure.entrypoints=https"
      #- "traefik.http.routers.grafana-monitor-secure.rule=Host(`grafana-monitor.cinderblock.tech`)"
      #- "traefik.http.routers.grafana-monitor-secure.tls=true"
      #- "traefik.http.routers.grafana-monitor-secure.service=grafana-monitor"
      #- "traefik.http.services.grafana-monitor.loadbalancer.server.port=3000"
      #- "traefik.docker.network=proxy"    
    networks:
      - proxy
      - monitoring
networks:
 proxy:
  external: true
 monitoring:
  external: false

Spin it up and connect to the URL to sign in. Default username and password are both admin.

Add InfluxDB datasource into Grafana

Now that we have both containers working, navigate to the webpage for Grafana. From here, go into administration --> data sources

Add a source, select InfluxDB.

Change Query language to Flux
Enter your https://FQDN
Basic Auth: User + Pass for Influx
Bucket and organization for Influx
Another API Token from Influx
Save and test, ensure that you can see buckets from Influx!

Visualize the data!

For this, there is a grafana plugin called Proxmox [Flux]. This’ll do the trick here, since we are in this case, importing Proxmox data.

Navigate to the Proxmox [Flux] page, and pull the import ID
At the top of your grafana page, hit the +, import dashboard, and throw that copied import ID in there.

From here navigate to your dashboard for InfluxDB, change the Bucket selected at the top, and you’ve got a visualized dashboard for your Proxmox data!