Monitoring & Log Management

Scripting Custom Prometheus Node Exporter Metrics via Bash

prometheus textfile bash ubuntu custom metrics monitoring script

Stop Settling for Default Server Metrics

A neon-lit dive bar scene, a tired sysadmin staring at a glowing laptop screen full of boring default line graphs, cyberpunk style, hyper-detailed --ar 16:9

Node Exporter gives you CPU, memory, and disk space. Great. But what about the stuff you actually care about? Maybe you need to track the number of stuck background jobs. Or how many users are logged in right now. Out of the box, Prometheus doesn't know about your custom weirdness. Here's the thing. You don't need to write a full-blown custom exporter in Go just to count some files or check a backup status. You just need bash and the prometheus textfile bash collector. It's built right in.

Flipping the Switch on Ubuntu

A close-up of a heavy, mechanical industrial switch being flipped on, glowing green fiber optic wires routing data, macro photography, cinematic lighting --ar 16:9

First things first. We need a place for our monitoring script to drop its data. I usually just make a directory under `/var/lib/prometheus/node-exporter`. Create it. Chown it to the prometheus user. Done. Now tell your Node Exporter to actually look there. If you installed it via apt on Ubuntu, crack open `/etc/default/prometheus-node-exporter`. Add `--collector.textfile.directory=/var/lib/prometheus/node-exporter` to your `ARGS`. Restart the service. Boom. It's listening.

Writing the Bash Script That Does the Heavy Lifting

A glowing green bash terminal floating in a dark, messy hacker room, chaotic lines of code resolving into neat, organized data blocks, digital art, high contrast --ar 16:9

Writing the actual script is dead simple. But Prometheus is picky about formatting. It expects the exact Prometheus text-based exposition format. Metric name, optional labels in brackets, and the value. Nothing else. Let's say we want to count files in a temp directory. Write a bash script that runs `ls | wc -l`. Assign that to a variable. Then echo it out. Like this: `echo "temp_files_total $COUNT" > /var/lib/prometheus/node-exporter/temp_files.prom.$$`. Notice the `.prom.$$` part? Always write to a temporary file first, then `mv` it to `.prom`. Atomic writes. Otherwise, Prometheus might scrape half a file while you're still writing it. Messy.

Put That Script on Autopilot with Cron

Your script works. Now it needs to run without you babying it. Grab cron. Edit your crontab and set that bash script to run every minute. Or every five minutes. Whatever makes sense for your ubuntu custom metrics. Just make sure the script is executable and the cron user actually has permissions to write to your textfile directory. Set it and forget it.

Catching the Data in Prometheus

Head over to your Prometheus dashboard. Type your custom metric name into the expression browser. Hit execute. If you followed the steps, your bash-generated metric is sitting right there, ready to be pushed into a Grafana dashboard. If it's not? Check your file permissions. Nine times out of ten, the node exporter user can't read the file your cron job dropped. Fix the permissions. Check again.