docs: add documentation for prometheus-operator usage (#171)

This commit is contained in:
Robert Kaussow 2022-03-11 13:37:19 +01:00 committed by GitHub
parent 4dc5bb1bd8
commit e4fef71442
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 179 additions and 90 deletions

View File

@ -19,3 +19,6 @@ VMID
Telegraf Telegraf
QEMU QEMU
cloud-init cloud-init
Prometheus
Kubernetes
namespace

View File

@ -4,9 +4,7 @@ title: CLI options
You can get all available CLI options by running `prometheus-pve-sd --help`: You can get all available CLI options by running `prometheus-pve-sd --help`:
<!-- prettier-ignore-start --> ```Shell
<!-- spellchecker-disable -->
{{< highlight Shell "linenos=table" >}}
$ prometheus-pve-sd --help $ prometheus-pve-sd --help
usage: prometheus-pve-sd [-h] [-c CONFIG_FILE] [-o OUTPUT_FILE] [-d LOOP_DELAY] [--no-service] [-f LOG_FORMAT] [-v] [-q] [--version] usage: prometheus-pve-sd [-h] [-c CONFIG_FILE] [-o OUTPUT_FILE] [-d LOOP_DELAY] [--no-service] [-f LOG_FORMAT] [-v] [-q] [--version]
@ -26,6 +24,4 @@ optional arguments:
-v increase log level -v increase log level
-q decrease log level -q decrease log level
--version show program's version number and exit --version show program's version number and exit
{{< /highlight >}} ```
<!-- spellchecker-enable -->
<!-- prettier-ignore-end -->

View File

@ -2,10 +2,7 @@
title: Default settings title: Default settings
--- ---
<!-- prettier-ignore-start --> ```Shell
<!-- markdownlint-disable -->
<!-- spellchecker-disable -->
{{< highlight YAML "linenos=table" >}}
--- ---
logging: logging:
# supported log levels: debug|info|warning|error|critical # supported log levels: debug|info|warning|error|critical
@ -45,7 +42,4 @@ pve:
# password: secure # password: secure
# auth_timeout: 5 # auth_timeout: 5
# verify_ssl: true # verify_ssl: true
{{< /highlight >}} ```
<!-- spellchecker-enable -->
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

View File

@ -2,10 +2,7 @@
title: Environment Variables title: Environment Variables
--- ---
<!-- prettier-ignore-start --> ```Shell
<!-- markdownlint-disable -->
<!-- spellchecker-disable -->
{{< highlight Shell "linenos=table" >}}
PROMETHEUS_PVE_SD_CONFIG_FILE= PROMETHEUS_PVE_SD_CONFIG_FILE=
# supported log levels: debug|info|warning|error|critical # supported log levels: debug|info|warning|error|critical
@ -32,7 +29,4 @@ PROMETHEUS_PVE_SD_PVE_USER=
PROMETHEUS_PVE_SD_PVE_PASSWORD= PROMETHEUS_PVE_SD_PVE_PASSWORD=
PROMETHEUS_PVE_SD_PVE_AUTH_TIMEOUT=5 PROMETHEUS_PVE_SD_PVE_AUTH_TIMEOUT=5
PROMETHEUS_PVE_SD_PVE_VERIFY_SSL=true PROMETHEUS_PVE_SD_PVE_VERIFY_SSL=true
{{< /highlight >}} ```
<!-- spellchecker-enable -->
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

View File

@ -2,9 +2,7 @@
title: Using docker title: Using docker
--- ---
<!-- prettier-ignore-start --> ```Shell
<!-- spellchecker-disable -->
{{< highlight Shell "linenos=table" >}}
docker run \ docker run \
-e PROMETHEUS_PVE_SD_LOG_LEVEL=info \ -e PROMETHEUS_PVE_SD_LOG_LEVEL=info \
-e PROMETHEUS_PVE_SD_LOG_FORMAT=console \ -e PROMETHEUS_PVE_SD_LOG_FORMAT=console \
@ -16,11 +14,11 @@ docker run \
-e PY_COLORS=1 \ -e PY_COLORS=1 \
-v $(pwd):/out \ -v $(pwd):/out \
thegeeklab/prometheus-pve-sd thegeeklab/prometheus-pve-sd
{{< /highlight >}} ```
<!-- spellchecker-enable -->
<!-- prettier-ignore-end -->
{{< hint info >}} {{< hint info >}}
**Info**\ **Info**\
Keep in mind, that you have to pass SELinux labels (:Z or :z) to your mount option if you are working on SELinux enabled systems. Keep in mind, that you have to pass SELinux labels (:Z or :z) to your mount option if you are working on SELinux enabled systems.
{{< /hint >}} {{< /hint >}}
After configuring and starting the service, Prometheus need to be [configured](/usage/getting-started/#prometheus-configuration) to use the external service discovery.

View File

@ -2,10 +2,7 @@
title: Using pip title: Using pip
--- ---
<!-- prettier-ignore-start --> ```Shell
<!-- markdownlint-disable -->
<!-- spellchecker-disable -->
{{< highlight Shell "linenos=table" >}}
# From PyPI as unprivileged user # From PyPI as unprivileged user
$ pip install prometheus-pve-sd --user $ pip install prometheus-pve-sd --user
@ -14,7 +11,12 @@ $ sudo pip install prometheus-pve-sd
# From Wheel file # From Wheel file
$ pip install https://github.com/thegeeklab/prometheus-pve-sd/releases/download/v0.1.0/prometheus_pve_sd-0.1.0-py3-none-any.whl $ pip install https://github.com/thegeeklab/prometheus-pve-sd/releases/download/v0.1.0/prometheus_pve_sd-0.1.0-py3-none-any.whl
{{< /highlight >}} ```
<!-- spellchecker-enable -->
<!-- markdownlint-restore --> Start the service:
<!-- prettier-ignore-end -->
```Shell
prometheus-pve-sd -vv --loop-delay 900 -o /etc/prometheus/pve.json
```
After configuring and starting the service, Prometheus need to be [configured](/usage/getting-started/#prometheus-configuration) to use the external service discovery.

View File

@ -0,0 +1,80 @@
---
title: Using Prometheus Operator
---
{{< toc >}}
As an alternative to local files service discovery Prometheus also support discoveries form HTTP endpoints. In a Kubernetes setup, with the Prometheus operator, it makes more sense to use this [HTTP SD](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) instead of mounting the output file of the `prometheus-pve-sd` to the container.
The `prometheus-pve-sd` module doesn't have a dedicated HTTP endpoint, therefore you need to use a web server sidecar, that hosts the static file. The following deployment configuration can serve as a starting point for most setups, and just requires some minor adjustments, depending on your Kubernetes setup.
## Kubernetes configuration
Deployment configuration:
```YAML
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: proxmox-service-discovery
name: proxmox-service-discovery
spec:
replicas: 1
selector:
matchLabels:
app: proxmox-service-discovery
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: proxmox-service-discovery
spec:
containers:
- image: nginx
name: webserver
resources: {}
ports:
- containerPort: 80
protocol: TCP
name: http
volumeMounts:
- name: pve-sd-output
mountPath: /usr/share/nginx/html
- image: thegeeklab/prometheus-pve-sd
name: prometheus-pve-sd
resources: {}
env:
- name: PROMETHEUS_PVE_SD_OUTPUT_FILE
value: /tmp/pve/proxmox.json
# Add more configurations here, or use a configMap or secret to inject the remaining configs
volumeMounts:
- name: pve-sd-output
mountPath: /tmp/pve/
volumes:
- name: pve-sd-output
emptyDir: {}
status: {}
```
Additionally you will need a service, that exposes the HTTP endpoint within Kubernetes so Prometheus can scrape it.
Service configuration:
```YAML
apiVersion: v1
kind: Service
metadata:
name: pve-sd-service
spec:
selector:
app: proxmox-service-discovery
ports:
- protocol: TCP
port: 80
targetPort: 80
```
After configuring and starting the service, Prometheus need to be [configured](/usage/getting-started/#prometheus-configuration) to use the external service discovery.

View File

@ -0,0 +1,53 @@
---
title: Getting Started
---
{{< toc >}}
## Prometheus relabeling
### Use IP address labels
Instead of `__meta_pve_name`, it is also possible to configure Prometheus to use the address provided by `__meta_pve_ipv4` or `__meta_pve_ipv4` for connections:
```YAML
relabel_configs:
- replacement: ${1}:9273
source_labels:
- __meta_pve_ipv4
target_label: __address__
```
### Convert tags to custom labels
Tags of a node exposed by the `__meta_pve_tags` might be useful to build more complex configurations in Prometheus. As an example, an `alert` tag can be extracted to dine alerting routes based on the tag value.
**Example:**
Extract `group` and `alert` from a list of tags like this: `__meta_pve_tags="alert:team-1,group:cluster-1,node:node-1"`
```YAML
relabel_configs:
- source_labels:
- __meta_pve_tags
regex: ".*group:([\w\-_]*)"
target_label: "group"
replacement: "${1}"
- source_labels:
- __meta_pve_tags
regex: ".*alert:([\w\-_]*)"
target_label: "alert"
replacement: "${1}"
```
Use the extracted `alert` label to set an Alertmanager route:
```YAML
routes:
- receiver: "empty"
matchers:
- alert = muted
- receiver: "team-1"
matchers:
- alert = team-1
```

View File

@ -4,14 +4,6 @@ title: Getting Started
{{< toc >}} {{< toc >}}
## Start PVE SD
Create a [configuration file](/configuration/defaults/) with the required parameters to connect to your Proxmox VE (PVE) server before you start the service.
```Shell
prometheus-pve-sd -vv --loop-delay 900 -o /etc/prometheus/pve.json
```
## Available Labels ## Available Labels
The following list of meta labels can be used to relabel your scrape results: The following list of meta labels can be used to relabel your scrape results:
@ -51,6 +43,8 @@ The following list of meta labels can be used to relabel your scrape results:
## Prometheus configuration ## Prometheus configuration
### File service discovery
Prometheus needs a basic file service discovery configuration to fetch system metrics from the host's discovered from PVE. Depending on the used metrics exporter the configuration need to be adjusted, using [Telegraf](https://github.com/influxdata/telegraf/#telegraf) a starter configuration might look like this: Prometheus needs a basic file service discovery configuration to fetch system metrics from the host's discovered from PVE. Depending on the used metrics exporter the configuration need to be adjusted, using [Telegraf](https://github.com/influxdata/telegraf/#telegraf) a starter configuration might look like this:
```YAML ```YAML
@ -60,8 +54,6 @@ Prometheus needs a basic file service discovery configuration to fetch system me
job_name: telegraf-pve job_name: telegraf-pve
metrics_path: /metrics metrics_path: /metrics
relabel_configs: relabel_configs:
- replacement: telegraf
target_label: job
- replacement: ${1}:9273 - replacement: ${1}:9273
source_labels: source_labels:
- __meta_pve_name - __meta_pve_name
@ -71,48 +63,21 @@ Prometheus needs a basic file service discovery configuration to fetch system me
target_label: instance target_label: instance
``` ```
### Use IP address labels ### HTTP service discovery
Instead of `__meta_pve_name`, it is also possible to configure Prometheus to use the address provided by `__meta_pve_ipv4` or `__meta_pve_ipv4` for connections: If the static file is served by a web server, e.g. while using the [Prometheus Operator](/setup/prometheus-operator/) setup, a HTTP service discovery configuration is required:
```YAML ```YAML
- http_sd_configs:
url: pve-sd-service:80/proxmox.json
job_name: telegraf-pve
metrics_path: /metrics
relabel_configs: relabel_configs:
- replacement: ${1}:9273 - replacement: ${1}:9273
source_labels: source_labels:
- __meta_pve_ipv4 - __meta_pve_name
target_label: __address__ target_label: __address__
```
### Convert tags to custom labels
Tags of a node exposed by the `__meta_pve_tags` might be useful to build more complex configurations in Prometheus. As an example, an `alert` tag can be extracted to dine alerting routes based on the tag value.
**Example:**
Extract `group` and `alert` from a list of tags like this: `__meta_pve_tags="alert:team-1,group:cluster-1,node:node-1"`
```YAML
relabel_configs:
- source_labels: - source_labels:
- __meta_pve_tags - __meta_pve_name
regex: ".*group:([\w\-_]*)" target_label: instance
target_label: "group"
replacement: "${1}"
- source_labels:
- __meta_pve_tags
regex: ".*alert:([\w\-_]*)"
target_label: "alert"
replacement: "${1}"
```
Use the extracted `alert` label to set an Alertmanager route:
```YAML
routes:
- receiver: "empty"
matchers:
- alert = muted
- receiver: "team-1"
matchers:
- alert = team-1
``` ```

View File

@ -2,10 +2,12 @@
main: main:
- name: Setup - name: Setup
sub: sub:
- name: Using pip - name: Using PIP
ref: "/setup/pip" ref: "/setup/pip"
- name: Using docker - name: Using Docker
ref: "/setup/docker" ref: "/setup/docker"
- name: Using Prometheus Operator
ref: "/setup/prometheus-operator"
- name: Configuration - name: Configuration
ref: "/configuration" ref: "/configuration"
sub: sub:
@ -19,3 +21,5 @@ main:
sub: sub:
- name: Getting Started - name: Getting Started
ref: "/usage/getting-started" ref: "/usage/getting-started"
- name: Advanced
ref: "/usage/advanced"

View File

@ -3,7 +3,7 @@
class PrometheusSDError(Exception): class PrometheusSDError(Exception):
"""Generic exception class for promtheus-pve-sd.""" """Generic exception class for Prometheus-pve-sd."""
def __init__(self, msg, original_exception=""): def __init__(self, msg, original_exception=""):
super(PrometheusSDError, super(PrometheusSDError,