System_monitor - BEAM VM telemetry collector (from Klarna)

etnt · December 27, 2023, 6:20pm

I don’t think the klarna/system_monitor has been announced here before which is a pity.
It is a very cool application, as its blurb says:

`system_monitor` is a BEAM VM monitoring and introspection application that helps 
 in troubleshooting live systems. It collects various information about Erlang processes
 and applications.

It is very easy to set it up to store the collected metric in Postgres to be viewed by Grafana.
Read about it here:

Now, I have done some additional work on it.

Split it into a (optional) frontend/backend where the frontend runs in the target node, gathering the metrics which then is sent over TLS to the backend which store it into Postgres.
An easy way to add custom metrics and as a first example a memory collector has been created that store Total/Free/Allocated/Used memory of the BEAM as well as the memory used by the ERTS allocators.

Read about it here:

github.com

etnt/system_monitor/blob/master/CONSUMER_PRODUCER.md

# Consumer - Producer extension
> Output monitor data over a Websocket

This extension to the `system_monitor` application makes
it possible to send the collected info via the `system_monitor_producer`
code over a websocket to a Consumer. If the Consumer is setup
to be the `system_monitor_consumer` it will send along the incoming
data to the `system_monitor_pg` Postgres handler.

So basically, it can be used in two ways, either by inserting
a "bridge" (via the websocket) to the `system_monitor` Postgres
backend, or simply used as a way to output the data over a websocket
to any other backend that is capable of consuming the incoming data.

To make sure no one else can eavesdrop on what is sent to the Backend
we run the Websocket over TLS using certificates both for the
Backend- and the Frontend node.

## Compiling

This file has been truncated. show original

and here:

github.com

etnt/system_monitor/blob/master/CUSTOM_METRICS.md

# Custom Metrics
> Hook in your own metrics collector

## Setup the collector

You can configure your own metrics collector to
be invoked by the `system_monitor`. In this example
we will setup a memory metrics collector.

![Memory](doc/memory.png)

Metrics collection is only done if the `callback_mod` 
application env var is set, indicating that collected
data should be exported. Typically it is set to either
`system_monitor_pg` for local export to Postgres running
on the same Host, or `system_monitor_producer` in case of
external export (automatically taken care of,
see also: [CONSUMER_PRODUCER](CONSUMER_PRODUCER.md) )

The configuration to enable a custom metric is done

This file has been truncated. show original

Cheers, Tobbe