post: Collect JSON metrics with Telegraf
All checks were successful
continuous-integration/drone/pr Build is passing

This commit is contained in:
Robert Kaussow 2022-03-13 12:13:50 +01:00
parent d881c16333
commit b62f5e9d5f
Signed by: xoxys
GPG Key ID: 4E692A2EAECC03C0
3 changed files with 87 additions and 0 deletions

View File

@ -43,3 +43,6 @@ readme
SSL SSL
Telegraf Telegraf
OSCP OSCP
JSON
DockerHub
starlark

Binary file not shown.

After

Width:  |  Height:  |  Size: 192 KiB

View File

@ -0,0 +1,84 @@
---
title: "Collect JSON metrics with Telegraf"
date: 2022-03-13T12:10:00+01:00
authors:
- robert-kaussow
tags:
- Sysadmin
resources:
- name: feature
src: "images/feature.jpg"
params:
anchor: Center
credits: >
[Nicholas Cappello](https://unsplash.com/@bash__profile) on
[Unsplash](https://unsplash.com/photos/Wb63zqJ5gnE)
---
Telegraf is a powerful, plugin based metrics collector that also provides Prometheus compatible outputs. For various purposes, there are a number of input plugins that can collect metrics from various sources. Even more powerful are the processor plugins that allow metrics to be processed and manipulated as they pass through, and immediately output results based on the values they process. In this short blog post I'll explain how to fetch JSON metrics from the Docker registry API to track some data of a DockerHub Repository.
<!--more-->
Even though Prometheus has become very popular, not all applications and API's provide native Prometheus metrics. In this example, I'll focus on the repository metrics provided by `https://hub.docker.com/v2/repositories`. While the API displays the current pull numbers, storing this information to Prometheus has the advantage of displaying time-based information, e.g. the pull frequency/ratio or pulls in a certain period of time.
To fetch metrics from an HTTP endpoint, the generic [HTTP input plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/http) can be used. This plugin allows collecting metrics from one or more HTTP(S) endpoints and also supports a lot of different [data formats](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md) not only [JSON](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json).
By default, only numeric values are extracted from the collected JSON data. In the case of the Docker Registry API endpoint, the status, star_count, pull_count, and collaborator_count fields are used. This behavior can be customized by adding fields for the `json_string_fields` configuration option. Since Prometheus supports only numeric values for metrics, string fields are added as labels to each metric. In some cases, string fields contain useful information that can also be converted to numeric references. For example, status texts such as `OK` and `ERROR` can be converted to `0` and `1` and used as metrics in Prometheus. In the case of the Docker Registry API, I wanted to use the `last_updated` field as a dedicated metric in Prometheus instead of a label. Fortunately, the date time format used can be transformed to a numeric value by converting it to a Unix timestamp.
For simple type conversions, I would recommend to always check the available [converter processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/converter) first. Sadly there is no such converter for date or date time values available yet, so I had to go another way and utilized the starlark processor. This processor calls a starlark function for each matched metric, allowing for custom programmatic metric processing. While starlark is a Python dialect and might look familiar, it only supports a very limited subset of the language, as explained in the [specification](https://github.com/google/starlark-go/blob/master/doc/spec.md). But it's powerful enough for what I wanted to do: Convert the date time value in the format of `2006-01-02T15:04:05.999999Z` to a valid Unix timestamp. That's how the final configuration looks like:
```TOML
#jinja2: lstrip_blocks: True
[[inputs.http]]
name_override = "dockerhub_respository"
urls = [
"https://hub.docker.com/v2/repositories/library/telegraf/"
]
json_string_fields = [
"last_updated",
"name",
"namespace",
"repository_type",
]
data_format = "json"
[[processors.starlark]]
namepass = ["dockerhub_respository"]
source = '''
load("time.star", "time")
def apply(metric):
metric.fields["last_updated"] = time.parse_time(metric.fields["last_updated"], format="2006-01-02T15:04:05.999999Z").unix
return metric
'''
```
To debug and test the Telegraf configuration it's useful to execute the binary with the `--test` flag:
```Plain
./usr/bin/telegraf --debug --test --config etc/telegraf/telegraf.conf --config-directory etc/telegraf/telegraf.d/
2022-03-12T17:01:07Z I! Starting Telegraf 1.21.4
2022-03-12T17:01:07Z I! Loaded inputs: http
2022-03-12T17:01:07Z I! Loaded aggregators:
2022-03-12T17:01:07Z I! Loaded processors: starlark
2022-03-12T17:01:07Z W! Outputs are not used in testing mode!
2022-03-12T17:01:07Z I! Tags enabled: host=localhost project=prometheus
[...]
> dockerhub_respository,host=localhost,name=telegraf,namespace=library,project=prometheus,repository_type=image,url=https://hub.docker.com/v2/repositories/library/telegraf/ collaborator_count=0,last_updated=1646267076i,pull_count=519804664,star_count=560,status=1 1647104469000000000
```
The final metrics generated by the Prometheus output plugin will look like this:
```Plain
dockerhub_respository_collaborator_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 0
dockerhub_respository_last_updated{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1.646267076e+09
dockerhub_respository_pull_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 5.19802966e+08
dockerhub_respository_star_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 560
dockerhub_respository_status{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1
```
That's it. The Telegraf HTTP input plugin is a very flexible but generic way to collect and transform JSON metrics from various sources. If that's still not powerful enough you can pass the raw data fetched by the HTTP input plugin to the starlark converter and write your own functions to parse the input and extract the required information into metrics.