post: Collect JSON metrics with Telegraf

.dictionary

@@ -43,3 +43,6 @@ readme
 SSL
 Telegraf
 OSCP
+JSON
+DockerHub
+starlark

content/posts/2022/collect-json-metrics-with-telegraf/index.md

@@ -0,0 +1,84 @@
+---
+title: "Collect JSON metrics with Telegraf"
+date: 2022-03-13T12:10:00+01:00
+authors:
+  - robert-kaussow
+tags:
+  - Sysadmin
+resources:
+  - name: feature
+    src: "images/feature.jpg"
+    params:
+      anchor: Center
+      credits: >
+        [Nicholas Cappello](https://unsplash.com/@bash__profile) on
+        [Unsplash](https://unsplash.com/photos/Wb63zqJ5gnE)
+---
+
+Telegraf is a powerful, plugin based metrics collector that also provides Prometheus compatible outputs. For various purposes, there are a number of input plugins that can collect metrics from various sources. Even more powerful are the processor plugins that allow metrics to be processed and manipulated as they pass through, and immediately output results based on the values they process. In this short blog post I'll explain how to fetch JSON metrics from the Docker registry API to track some data of a DockerHub Repository.
+
+<!--more-->
+
+Even though Prometheus has become very popular, not all applications and API's provide native Prometheus metrics. In this example, I'll focus on the repository metrics provided by `https://hub.docker.com/v2/repositories`. While the API displays the current pull numbers, storing this information to Prometheus has the advantage of displaying time-based information, e.g. the pull frequency/ratio or pulls in a certain period of time.
+
+To fetch metrics from an HTTP endpoint, the generic [HTTP input plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/http) can be used. This plugin allows collecting metrics from one or more HTTP(S) endpoints and also supports a lot of different [data formats](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md) not only [JSON](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json).
+
+By default, only numeric values are extracted from the collected JSON data. In the case of the Docker Registry API endpoint, the status, star_count, pull_count, and collaborator_count fields are used. This behavior can be customized by adding fields for the `json_string_fields` configuration option. Since Prometheus supports only numeric values for metrics, string fields are added as labels to each metric. In some cases, string fields contain useful information that can also be converted to numeric references. For example, status texts such as `OK` and `ERROR` can be converted to `0` and `1` and used as metrics in Prometheus. In the case of the Docker Registry API, I wanted to use the `last_updated` field as a dedicated metric in Prometheus instead of a label. Fortunately, the date time format used can be transformed to a numeric value by converting it to a Unix timestamp.
+
+For simple type conversions, I would recommend to always check the available [converter processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/converter) first. Sadly there is no such converter for date or date time values available yet, so I had to go another way and utilized the starlark processor. This processor calls a starlark function for each matched metric, allowing for custom programmatic metric processing. While starlark is a Python dialect and might look familiar, it only supports a very limited subset of the language, as explained in the [specification](https://github.com/google/starlark-go/blob/master/doc/spec.md). But it's powerful enough for what I wanted to do: Convert the date time value in the format of `2006-01-02T15:04:05.999999Z` to a valid Unix timestamp. That's how the final configuration looks like:
+
+```TOML
+#jinja2: lstrip_blocks: True
+[[inputs.http]]
+  name_override = "dockerhub_respository"
+  urls = [
+    "https://hub.docker.com/v2/repositories/library/telegraf/"
+  ]
+
+  json_string_fields = [
+    "last_updated",
+    "name",
+    "namespace",
+    "repository_type",
+  ]
+
+  data_format = "json"
+
+[[processors.starlark]]
+  namepass = ["dockerhub_respository"]
+
+  source = '''
+load("time.star", "time")
+def apply(metric):
+
+  metric.fields["last_updated"] = time.parse_time(metric.fields["last_updated"], format="2006-01-02T15:04:05.999999Z").unix
+
+  return metric
+'''
+```
+
+To debug and test the Telegraf configuration it's useful to execute the binary with the `--test` flag:
+
+```Plain
+./usr/bin/telegraf --debug --test --config etc/telegraf/telegraf.conf --config-directory etc/telegraf/telegraf.d/
+2022-03-12T17:01:07Z I! Starting Telegraf 1.21.4
+2022-03-12T17:01:07Z I! Loaded inputs: http
+2022-03-12T17:01:07Z I! Loaded aggregators:
+2022-03-12T17:01:07Z I! Loaded processors: starlark
+2022-03-12T17:01:07Z W! Outputs are not used in testing mode!
+2022-03-12T17:01:07Z I! Tags enabled: host=localhost project=prometheus
+[...]
+> dockerhub_respository,host=localhost,name=telegraf,namespace=library,project=prometheus,repository_type=image,url=https://hub.docker.com/v2/repositories/library/telegraf/ collaborator_count=0,last_updated=1646267076i,pull_count=519804664,star_count=560,status=1 1647104469000000000
+```
+
+The final metrics generated by the Prometheus output plugin will look like this:
+
+```Plain
+dockerhub_respository_collaborator_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 0
+dockerhub_respository_last_updated{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1.646267076e+09
+dockerhub_respository_pull_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 5.19802966e+08
+dockerhub_respository_star_count{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 560
+dockerhub_respository_status{host="localhost",name="telegraf",namespace="library",project="prometheus",repository_type="image",url="https://hub.docker.com/v2/repositories/library/telegraf/"} 1
+```
+
+That's it. The Telegraf HTTP input plugin is a very flexible but generic way to collect and transform JSON metrics from various sources. If that's still not powerful enough you can pass the raw data fetched by the HTTP input plugin to the starlark converter and write your own functions to parse the input and extract the required information into metrics.