System Monitoring and Scheduling with cron and systemd
Read load, memory, disk and port usage correctly, query the journal, and schedule reliable jobs with cron, systemd timers and flock, with the cron gotchas tested.
By The Dark DebuggerUpdated 34 min read
This article covers the two halves of keeping a Linux machine healthy without staring at it: reading its vital signs (load, memory, disk, ports, logs) and scheduling jobs that run on their own, with cron and with systemd timers. It builds on processes, signals, and job control and assumes you can write a small script with arguments and exit codes (your first scripts).
Why this matters
Two situations come up on every team that runs servers.
The first: a disk fills up at 3 a.m., the database stops accepting writes, and the first anyone hears of it is a customer. A five-line check run every five minutes would have warned you two days earlier.
The second: someone writes a backup script, runs it by hand, sees it work, adds it to
cron, and goes home. It never runs successfully once. The command was not on cron's
PATH, the error went to a mailbox nobody reads, and the first person to notice is the
one trying to restore a backup that does not exist.
Both are avoidable with tools that are already installed. The trick is knowing how to read them and knowing exactly how cron's environment differs from your terminal.
Load average: uptime and vmstat
uptime
nproc 21:47:19 up 8 min, 1 user, load average: 10.11, 7.00, 3.35
8The three numbers are the 1-, 5- and 15-minute load averages: exponentially smoothed
counts of tasks that are either running on a CPU, waiting for one, or (on Linux
specifically) stuck in uninterruptible I/O wait, the D state. They are not percentages.
Read them against the number of CPUs:
- Load below
nproc: there is spare CPU capacity. - Load around
nproc: every CPU is busy, nothing is queueing much. - Load well above
nproc: work is waiting. Here, 10.11 on 8 CPUs means that, on average, about two tasks were always waiting for a turn.
The three values together show the trend. 10.11, 7.00, 3.35 is a load that is
rising: the last minute was much busier than the last quarter hour. (This was the
shared lab machine while several builds ran at once; it had been booted 8 minutes
earlier.) A high 15-minute value with a low 1-minute value means a spike that is already
over.
High load does not always mean "CPU-bound". vmstat shows what kind of waiting it is:
vmstat 2 4procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
2 1 0 1779052 209232 3642824 0 0 2423 6004 4315 9 26 15 49 10 0 0
6 1 0 1767320 209512 3642168 0 0 0 1854 3123 4157 27 20 44 9 0 0
6 1 0 1753844 209820 3649864 0 0 0 1170 4985 8680 35 21 36 9 0 0
6 0 0 1791532 210312 3654564 0 0 412 1680 6215 10404 33 18 41 8 0 0The first line is an average since boot; read the later ones. r is the run queue
(tasks wanting CPU), b is tasks blocked on I/O, and the cpu block repeats top's
us/sy/id/wa. Six runnable tasks with 36-44% idle CPU and about 9% wa means
this load is a mix of CPU work and disk writes (bo, blocks out), not one runaway
process. If b and wa are high while us is low, the disk is your bottleneck and
adding CPUs would not help. The si/so columns (swap in/out) should be zero; steady
non-zero values mean the machine is short of RAM.
Memory: what "available" means
free -h total used free shared buff/cache available
Mem: 7.7Gi 2.6Gi 1.6Gi 90Mi 3.7Gi 5.0Gi
Swap: 2.0Gi 0B 2.0GiNew Linux users read the free column, see 1.6 GiB of 7.7, and panic. Don't. Linux
puts otherwise idle RAM to work as page cache (buff/cache), keeping recently read
files in memory, and hands that memory back the moment a program needs it. The number
that answers "how much more can I run before the machine starts swapping" is
available (5.0 GiB here): free memory plus the cache that can be dropped.
free reads /proc/meminfo, which is also the easiest source for scripts:
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree):' /proc/meminfoMemTotal: 8038076 kB
MemFree: 1718064 kB
MemAvailable: 5277132 kB
Buffers: 203696 kB
Cached: 3416500 kB
SwapTotal: 2097152 kB
SwapFree: 2097152 kBAlert on MemAvailable dropping below a percentage of MemTotal, and on swap usage
growing over time. A machine that is genuinely out of memory shows it in the kernel log
(journalctl -k | grep -i 'out of memory') when the OOM killer ends a process with
SIGKILL, the "exit 137" from the previous article.
Disk space: df
df reports space per mounted filesystem:
df -hFilesystem Size Used Avail Use% Mounted on
none 3.9G 0 3.9G 0% /usr/lib/modules/6.18.54.1-microsoft-standard-WSL2
/dev/sdb 97M 97M 0 100% /usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2
none 3.9G 4.0K 3.9G 1% /mnt/wsl
[…]
/dev/sdd 1007G 4.5G 952G 1% /
[…]
tmpfs 785M 24K 785M 1% /run/user/1000On the WSL lab machine df -h lists a dozen helper mounts; a cloud VM shows fewer, but
you still get tmpfs and snap or image mounts. Two habits help. First, ask about the
filesystem you care about, and add -T for its type:
df -hT /Filesystem Type Size Used Avail Use% Mounted on
/dev/sdd ext4 1007G 4.5G 952G 1% /Second, don't panic about read-only images at 100%. The /dev/sdb mount above is a
read-only tools image that is full by design, exactly like snap packages on Ubuntu
servers. A monitoring check should name the mount points it cares about rather than
alert on every line.
For scripts, choose columns with --output and use -P (POSIX format, one line per
filesystem) when parsing:
df -h --output=target,size,used,avail,pcent / /dev/shm
df -P / | awk 'NR == 2 { print $5 }'Mounted on Size Used Avail Use%
/ 1007G 4.5G 952G 1%
/dev/shm 3.9G 66M 3.8G 2%
1%Inodes: the other way a disk fills up
Every file uses one inode, and an ext4 filesystem has a fixed number of them. A
directory with millions of tiny files (session files, a mail queue, a cache nobody
cleans) can exhaust inodes while df -h still shows free gigabytes. The error is the
same No space left on device, which is maddening until you know to check df -i:
df -i /Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdd 67108864 63333 67045531 1% /If IUse% is near 100%, look for the directory with the most files
(find / -xdev -type f | cut -d/ -f2-3 | sort | uniq -c | sort -rn | head) rather than
the biggest one.
Finding what uses the space: du
df says how full a filesystem is; du says which directories are responsible. Here is a
small web shop's data directory:
du -sh .
du -sh -- * | sort -h
du -h --max-depth=2 . | sort -rh | head -620M .
8.0K app
2.4M cache
5.6M uploads
12M logs
20M .
12M ./logs
5.6M ./uploads/2026
5.6M ./uploads
4.4M ./logs/archive
2.4M ./cache/thumbs-s summarizes each argument, -h prints human sizes, and sort -h understands those
suffixes (plain sort -n would put 12M before 8.0K). --max-depth=N (or -d N)
limits how deep the report goes, which is how you walk down a full disk one level at a
time. On a real server, add -x to stay on one filesystem, so du -xh / -d1 does not
wander into /proc or network mounts.
When du and ls disagree
du reports blocks actually allocated, ls -l reports the file's length. Usually they
match. For sparse files (VM images, database files, some logs) they do not:
ls -lh cache/disk.img
du -h cache/disk.img
du -h --apparent-size cache/disk.img-rw-r--r-- 1 sid sid 2.0G Nov 9 21:49 cache/disk.img
0 cache/disk.img
2.0G cache/disk.imgA 2 GB file using zero bytes of disk. The reverse puzzle, df showing a full disk that
du cannot account for, is almost always a deleted file still held open by a process;
lsof +L1 finds it, as shown in the process management
article.
Network ports: ss
ss lists sockets. The combination you will type most is "listening TCP sockets,
numeric, with processes":
ss -tlnpState Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 1000 10.255.255.254:53 0.0.0.0:*
LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
LISTEN 0 5 127.0.0.1:8765 0.0.0.0:* users:(("python3",pid=23411,fd=3))
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 50 *:46109 *:* users:(("java",pid=8195,fd=319))
LISTEN 0 50 [::ffff:127.0.0.1]:8090 *:* users:(("java",pid=1656,fd=10))
LISTEN 0 50 [::ffff:127.0.0.1]:8097 *:* users:(("java",pid=8195,fd=10))
LISTEN 0 50 *:39737 *:* users:(("java",pid=1656,fd=319)) How to read the address column:
127.0.0.1:8765listens on loopback only: reachable from this machine, not from the network. That is what you want for an app behind a reverse proxy.*:46109or0.0.0.0:PORTlistens on every interface, so it is reachable from outside unless a firewall says otherwise. Audit these.- The DNS lines on port 53 have no process shown because they belong to
root; withoutsudo,ss -ponly names your own processes.
ss -s gives a summary, useful for spotting connection leaks (thousands of sockets in
timewait or estab where you expect dozens):
ss -sTotal: 293
TCP: 81 (estab 2, closed 71, orphaned 0, timewait 71)
Transport Total IP IPv6
RAW 0 0 0
UDP 5 4 1
TCP 10 4 6
INET 15 8 7
FRAG 0 0 0 Filters narrow things down: ss -tn state established '( dport = :5432 )' shows open
connections to PostgreSQL, and ss -tlnp 'sport = :8080' shows who owns port 8080.
Logs: journalctl
On systemd machines, service output, kernel messages and syslog all land in the
journal. logger writes into it from scripts, with a tag and a priority, which makes
your own jobs as searchable as any daemon:
logger -t nightly-backup "backup finished: 1.2 GB in 340 s"
logger -t nightly-backup -p user.err "backup FAILED: /srv/db unreadable"
journalctl -t nightly-backup --no-pagerNov 09 21:49:06 lab nightly-backup[24216]: backup finished: 1.2 GB in 340 s
Nov 09 21:49:06 lab nightly-backup[24219]: backup FAILED: /srv/db unreadableEach line is timestamp, host, tag with PID, message. The queries worth memorizing:
| Command | Shows |
|---|---|
journalctl -u cron | One systemd unit (here the cron daemon) |
journalctl --user -u NAME | A unit in your own user instance |
journalctl -t TAG | Messages with a syslog tag, e.g. from logger -t |
journalctl -p err | Priority err and worse (warning, info, debug also work) |
journalctl -b | Since the last boot (-b -1 is the boot before) |
journalctl --since "1 hour ago" | Time window; also --until, or "2026-11-09 02:00" |
journalctl -f | Follow new messages, like tail -f |
journalctl -n 20 | Last 20 lines |
journalctl -k | Kernel messages (OOM kills, disk errors) |
Output formats matter when the journal feeds a script:
journalctl -t nightly-backup -p err --no-pager -o short-iso
journalctl -t nightly-backup -o cat --no-pager
journalctl -t nightly-backup -o json --no-pager | tail -1 | jq '{t: .__REALTIME_TIMESTAMP, prio: .PRIORITY, msg: .MESSAGE}'2026-11-09T21:49:06-05:00 lab nightly-backup[24219]: backup FAILED: /srv/db unreadable
backup finished: 1.2 GB in 340 s
backup FAILED: /srv/db unreadable
{
"t": "1794278946248870",
"prio": "3",
"msg": "backup FAILED: /srv/db unreadable"
}-o cat gives bare messages, -o short-iso gives sortable timestamps, and -o json
gives every field (priority 3 is err). To read the system journal without sudo, your
user needs to be in the adm or systemd-journal group. journalctl --disk-usage tells
you how much space the journal itself takes (490 MB on the lab machine); limits live in
/etc/systemd/journald.conf.
cron: the crontab format
cron runs commands on a schedule. Each user has a crontab, edited with crontab -e,
listed with crontab -l and removed with crontab -r. A line is five time fields and a
command:
# ┌───────────── minute (0-59)
# │ ┌─────────── hour (0-23)
# │ │ ┌───────── day of month (1-31)
# │ │ │ ┌─────── month (1-12 or jan-dec)
# │ │ │ │ ┌───── day of week (0-7 or sun-sat; 0 and 7 are both Sunday)
# │ │ │ │ │
30 2 * * * /usr/local/bin/backup.shEach field accepts:
| Syntax | Meaning | Example | Runs |
|---|---|---|---|
* | every value | * * * * * | every minute |
N | that value | 30 2 * * * | 02:30 daily |
A-B | range | 0 9-17 * * * | on the hour, 09:00 to 17:00 |
A,B,C | list | 0 8,12,18 * * * | 08:00, 12:00, 18:00 |
*/N | every N | */15 * * * * | :00, :15, :30, :45 |
A-B/N | every N within a range | 0 8-18/2 * * 1-5 | every 2 hours 08:00-18:00, weekdays |
And a few shortcuts in place of the five fields:
| Shortcut | Equivalent |
|---|---|
@reboot | once, when cron starts |
@hourly | 0 * * * * |
@daily (or @midnight) | 0 0 * * * |
@weekly | 0 0 * * 0 |
@monthly | 0 0 1 * * |
One rule surprises nearly everyone: if both day-of-month and day-of-week are
restricted, the job runs when either matches. 0 3 1-7 * 1 does not mean "the first
Monday of the month"; it means "days 1 to 7, and also every Monday". The usual workaround
is to restrict one field and test the other in the command: 0 3 * * 1 [ "$(date +\%d)" -le 07 ] && /usr/local/bin/job.sh
(note the \%, explained below).
Besides per-user crontabs, there are system-wide ones: /etc/crontab and files in
/etc/cron.d/ use the same format with an extra user field before the command, and
scripts dropped into /etc/cron.daily/, /etc/cron.weekly/ and so on are run by
run-parts (their file names must not contain a dot, so backup.sh there is silently
skipped; name it backup).
crontab validates what you give it:
printf '61 * * * * true\n' | crontab -"-":0: bad minute
errors in crontab file, can't install.The cron environment is not your shell
This is where most cron jobs die. To see exactly what a job gets, I installed a crontab that dumps its environment, plus a few lines that fail in instructive ways:
# m h dom mon dow command
* * * * * env > /home/sid/cron/cron-env.txt
* * * * * echo "hello from cron"
* * * * * date +%F >> /home/sid/cron/date-broken.txt
* * * * * date +\%F >> /home/sid/cron/date-fixed.txt
* * * * * shellcheck --version >> /home/sid/cron/path.log 2>&1crontab test.crontabAfter the next minute ticked over:
cat cron-env.txtHOME=/home/sid
LOGNAME=sid
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LANG=C.UTF-8
SHELL=/bin/sh
PWD=/home/sidSix variables. Compare that with the dozens in your terminal. Things to notice:
SHELL=/bin/sh. The command line is run bysh(dash on Ubuntu), not Bash. Bash syntax like[[ ]], arrays or{1..5}written directly in a crontab line fails. Put logic in a script with a#!/usr/bin/env bashline, or setSHELL=/bin/bashat the top of the crontab.PATHis the system default. On Ubuntu 24.04 cron loads/etc/environmentthrough PAM, which is where that longPATHcame from; on other distributions it can be as short as/usr/bin:/bin. What it never includes is anything your~/.bashrcor~/.profileadds:~/.local/bin,~/bin, nvm, pyenv, conda, or (on this machine) the lab's tool directory. That is why theshellcheckline failed:
cat path.log/bin/sh: 1: shellcheck: not found- The working directory is
$HOME, so relative paths in a job resolve against your home directory, not the script's directory. - No terminal, no profile. Nothing from
.bashrcruns, and there is no$DISPLAYor SSH agent.
The % sign
cron treats an unescaped % as a newline and feeds everything after the first one to the
command's standard input. The cron log shows what actually ran:
journalctl -u cron --since '2026-11-09 21:49:46' --no-pagerNov 09 21:50:02 lab CRON[27323]: pam_unix(cron:session): session opened for user sid(uid=1000) by sid(uid=0)
[…]
Nov 09 21:50:02 lab CRON[27363]: (sid) CMD (date +%F >> /home/sid/cron/date-fixed.txt)
Nov 09 21:50:02 lab CRON[27365]: (sid) CMD (date +)
Nov 09 21:50:02 lab CRON[27368]: (sid) CMD (shellcheck --version >> /home/sid/cron/path.log 2>&1)
Nov 09 21:50:02 lab CRON[27369]: (sid) CMD (env > /home/sid/cron/cron-env.txt)
Nov 09 21:50:02 lab CRON[27364]: (sid) CMD (echo "hello from cron")
[…]
Nov 09 21:50:02 lab CRON[27324]: (CRON) info (No MTA installed, discarding output)
[…]
Nov 09 21:50:02 lab CRON[27325]: (CRON) info (No MTA installed, discarding output)The broken line ran as date +. Even the >> date-broken.txt redirect was swallowed
into standard input, so that file was never created at all. The escaped date +\%F
version worked and wrote 2026-11-09. Rule: escape every % in a crontab as \%, or
better, keep commands in scripts so the crontab line has no % in it.
Where the output goes
cron collects anything a job prints to stdout or stderr and emails it to the user (or
to MAILTO). On a server without a mail transfer agent, that becomes the line you saw
above: No MTA installed, discarding output. The echo "hello from cron" output and the
error from date + were both thrown away. Your job's errors vanish unless you capture
them yourself. Every cron line you write should therefore end with one of:
>> /var/log/myjob.log 2>&1to append to a log file, or2>&1 | logger -t myjobto send everything to the journal,
and set MAILTO="" at the top of the crontab if you do not want mail even when an MTA
exists. A crontab can set variables (SHELL, PATH, MAILTO, or your own) on lines of
the form NAME=value; they apply to the lines below them.
A monitored cron job
Here is a job written the way cron jobs should be: a script with a clear exit code, a message to the journal when something is wrong, and a heartbeat file that proves it ran.
#!/usr/bin/env bash
# disk-check.sh - warn when a filesystem crosses a usage threshold
# usage: disk-check.sh THRESHOLD_PERCENT MOUNTPOINT...
set -euo pipefail
threshold=${1:?usage: disk-check.sh THRESHOLD_PERCENT MOUNTPOINT...}
shift
state_dir=${STATE_DIR:-$HOME/.local/state/disk-check}
mkdir -p "$state_dir"
status=0
while read -r target pcent; do
used=${pcent%\%}
if (( used >= threshold )); then
msg="$target is ${used}% full (threshold ${threshold}%)"
logger -t disk-check -p user.warning "$msg"
echo "$(date '+%F %T') WARN $msg"
status=1
else
echo "$(date '+%F %T') ok $target ${used}%"
fi
done < <(df --output=target,pcent "$@" | tail -n +2)
touch "$state_dir/last-run" # heartbeat: proves the job ran at all
exit "$status"Test it by hand first, including the failure path (a 1% threshold forces a warning):
./disk-check.sh 90 / /run/shm; echo "exit: $?"
./disk-check.sh 1 /; echo "exit: $?"
journalctl -t disk-check -o short-iso --no-pager2026-11-09 21:50:38 ok / 1%
2026-11-09 21:50:38 ok /run/shm 2%
exit: 0
2026-11-09 21:50:38 WARN / is 1% full (threshold 1%)
exit: 1
2026-11-09T21:50:38-05:00 lab disk-check[29779]: / is 1% full (threshold 1%)Then the crontab. In production this would be */5; for the test I used every minute:
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""
STATE_DIR=/home/sid/cron/state
# m h dom mon dow command
* * * * * flock -n /tmp/disk-check.lock /home/sid/cron/disk-check.sh 90 / >> /home/sid/cron/disk-check.log 2>&1A minute later:
cat disk-check.log
journalctl -u cron --since '2026-11-09 21:50:59' --no-pager | grep -v pam_unix2026-11-09 21:51:01 ok / 1%
Nov 09 21:51:01 lab cron[181]: (sid) RELOAD (crontabs/sid)
Nov 09 21:51:01 lab CRON[31365]: (sid) CMD (flock -n /tmp/disk-check.lock /home/sid/cron/disk-check.sh 90 / >> /home/sid/cron/disk-check.log 2>&1)Every line of that crontab is deliberate: SHELL and PATH are explicit, MAILTO=""
because output goes to a log, absolute paths everywhere, flock so runs never overlap
(next section), and >> log 2>&1 so errors are kept.
The heartbeat solves the problem nothing else does: a job that silently stops running produces no errors at all. The crontab gets wiped, the server is rebuilt without it, the script is renamed. A second, independent check (on another machine, or in your monitoring system) asks "was the heartbeat touched recently?":
find state/last-run -mmin -15state/last-runfind -mmin -15 prints the file only if it was modified in the last 15 minutes; empty
output means the job has stopped. Exercise 3 turns this into a script.
Preventing overlapping runs with flock
A job scheduled every 5 minutes that occasionally takes 7 will eventually run twice at
once: two backups writing the same file, two imports inserting the same rows. flock
from util-linux takes a lock on a file for the duration of a command. Here slow-job.sh
takes five seconds, and a second copy starts half a second after the first:
flock -n /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -n /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait21:50:38 job 29795 started
second run exit: 1
21:50:43 job 29795 finished-n (non-blocking) makes the second copy give up immediately with exit status 1. -E
chooses a different exit code, so your monitoring can tell "skipped because busy" from
"failed":
flock -n -E 75 /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -n -E 75 /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait21:50:43 job 30118 started
second run exit: 75
21:50:48 job 30118 finishedOr wait for the lock with a timeout (-w seconds) when the second run should queue
rather than skip:
flock /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -w 10 /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait21:50:48 job 30465 started
21:50:53 job 30465 finished
21:50:53 job 30910 started
21:50:58 job 30910 finished
second run exit: 0The lock is held by an open file descriptor, so the kernel releases it automatically when
the process exits, even after kill -9 or a crash. That is the big advantage over "PID
file" locks that a crashed job leaves behind. You can also take the lock inside the script
itself, so it is protected no matter who starts it:
#!/usr/bin/env bash
# self-lock.sh - a script that refuses to run twice at the same time
exec 9> "/tmp/${0##*/}.lock"
if ! flock -n 9; then
echo "$(date +%T) another copy is running, exiting" >&2
exit 1
fi
echo "$(date +%T) got the lock, working"
sleep 3
echo "$(date +%T) done"./self-lock.sh & sleep 0.5; ./self-lock.sh; echo "exit: $?"; wait21:51:25 got the lock, working
21:51:26 another copy is running, exiting
exit: 1
21:51:28 doneexec 9> file opens file descriptor 9 for the rest of the script, and flock -n 9 locks
it. The production script template uses exactly
this pattern.
systemd timers: the modern alternative
systemd can run jobs on a schedule too. A timer is two small unit files: a .service
that says what to run, and a .timer that says when. User units live in
~/.config/systemd/user/ and need no root; system units go in /etc/systemd/system/ and
use systemctl without --user.
# ~/.config/systemd/user/disk-check.service
[Unit]
Description=Check disk usage and warn above 90%
[Service]
Type=oneshot
Environment=STATE_DIR=%h/cron/state
ExecStart=%h/cron/disk-check.sh 90 /# ~/.config/systemd/user/disk-check.timer
[Unit]
Description=Run disk-check every minute
[Timer]
OnCalendar=minutely
AccuracySec=1s
Persistent=true
[Install]
WantedBy=timers.target%h expands to your home directory. Type=oneshot means "runs to completion", the right
type for scheduled jobs. In the timer, OnCalendar is the schedule, AccuracySec=1s
asks for precise timing (the default lets systemd batch wake-ups within a minute), and
Persistent=true means a run missed while the machine was off happens at the next boot,
something cron simply cannot do.
Check the files, load them and start the timer:
systemd-analyze --user verify ~/.config/systemd/user/disk-check.service ~/.config/systemd/user/disk-check.timer
systemctl --user daemon-reload
systemctl --user enable --now disk-check.timer
systemctl --user list-timers disk-check.timer --no-pagerCreated symlink /home/sid/.config/systemd/user/timers.target.wants/disk-check.timer → /home/sid/.config/systemd/user/disk-check.timer.
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2026-11-09 21:57:00 EST 54s - - disk-check.timer disk-check.service
1 timers listed.
Pass --all to see loaded but inactive timers, too.verify printed nothing, which means no errors. enable makes the timer start at login
(or boot, for system units) and --now starts it immediately. You can also run the job
on demand, which is something cron has no clean equivalent for:
systemctl --user start disk-check.service
systemctl --user status disk-check.service --no-pager○ disk-check.service - Check disk usage and warn above 90%
Loaded: loaded (/home/sid/.config/systemd/user/disk-check.service; static)
Active: inactive (dead) since Mon 2026-11-09 21:56:05 EST; 37ms ago
TriggeredBy: ● disk-check.timer
Process: 48395 ExecStart=/home/sid/cron/disk-check.sh 90 / (code=exited, status=0/SUCCESS)
Main PID: 48395 (code=exited, status=0/SUCCESS)
CPU: 27ms
Nov 09 21:56:05 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:56:05 lab disk-check.sh[48395]: 2026-11-09 21:56:05 ok / 1%
Nov 09 21:56:05 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.inactive (dead) with status=0/SUCCESS is the healthy state for a oneshot service
between runs. After the next full minute, the timer had fired on its own, and the
script's output was in the journal without any redirection:
journalctl --user -u disk-check.service --since '2026-11-09 21:56:04' --no-pager
systemctl --user list-timers disk-check.timer --no-pagerNov 09 21:56:05 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:56:05 lab disk-check.sh[48395]: 2026-11-09 21:56:05 ok / 1%
Nov 09 21:56:05 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.
Nov 09 21:57:00 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:57:00 lab disk-check.sh[52669]: 2026-11-09 21:57:00 ok / 1%
Nov 09 21:57:00 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2026-11-09 21:58:00 EST 59s Mon 2026-11-09 21:57:00 EST 83ms ago disk-check.timer disk-check.service
1 timers listed.
Pass --all to see loaded but inactive timers, too.To remove a timer: systemctl --user disable --now disk-check.timer, delete the two
files, and run systemctl --user daemon-reload. With Persistent=true, systemd also
keeps a timestamp file under ~/.local/share/systemd/timers/; delete it too if you are
removing the timer for good.
OnCalendar syntax
The format is DayOfWeek Year-Month-Day Hour:Minute:Second, with *, ranges (..),
lists (,) and steps (/). You never have to guess what a schedule means, because
systemd-analyze calendar tells you:
systemd-analyze calendar 'Mon..Fri *-*-* 02:30:00'
systemd-analyze calendar --iterations=3 '*:0/15'Normalized form: Mon..Fri *-*-* 02:30:00
Next elapse: Tue 2026-11-10 02:30:00 EST
(in UTC): Tue 2026-11-10 07:30:00 UTC
From now: 4h 38min left
Original form: *:0/15
Normalized form: *-*-* *:00/15:00
Next elapse: Mon 2026-11-09 22:00:00 EST
(in UTC): Tue 2026-11-10 03:00:00 UTC
From now: 8min left
Iteration #2: Mon 2026-11-09 22:15:00 EST
(in UTC): Tue 2026-11-10 03:15:00 UTC
From now: 23min left
Iteration #3: Mon 2026-11-09 22:30:00 EST
(in UTC): Tue 2026-11-10 03:30:00 UTC
From now: 38min leftAnd the "first Monday of the month" that cron cannot express directly:
systemd-analyze calendar 'Mon *-*-1..7 03:00' Original form: Mon *-*-1..7 03:00
Normalized form: Mon *-*-01..07 03:00:00
Next elapse: Mon 2026-12-07 03:00:00 EST
(in UTC): Mon 2026-12-07 08:00:00 UTC
From now: 3 weeks 6 days leftUnlike cron, systemd combines day-of-week and day-of-month with AND, so this really is the first Monday.
cron or a timer?
| cron | systemd timer | |
|---|---|---|
| Setup | One line | Two unit files |
| Logs | Whatever you redirect | Automatic, in the journal, per unit |
| Missed runs (machine off) | Lost | Persistent=true catches up |
| Overlap protection | Add flock | Built in: a running oneshot is not started again |
| Environment | Minimal, surprising | Explicit (Environment=, EnvironmentFile=) |
| Resource limits, sandboxing | No | MemoryMax=, CPUQuota=, ProtectSystem= … |
| Status and next run | Not really | systemctl list-timers |
| Portability | Every Unix | systemd Linux only |
My recommendation: use a timer for anything that matters on a systemd machine, because
logging, overlap protection and missed-run handling come for free. Use cron for quick,
low-stakes jobs, for systems without systemd, or where a team already manages crontabs.
One caveat for user timers: they run only while your user's systemd instance is
running, which normally means while you are logged in. On a server, an administrator
runs loginctl enable-linger yourname to keep it running, or you install the units as
system units.
Where you'll use this
A common request on a team is a morning report: one screen per server, posted to chat or emailed, so that problems are noticed before they become incidents. Everything in this article fits in about twenty lines:
#!/usr/bin/env bash
# health-snapshot.sh - one-screen summary of a Linux host, for a morning report
set -uo pipefail
disk_threshold=${DISK_THRESHOLD:-80}
read -ra mounts <<< "${DISK_MOUNTS:-/}"
cpus=$(nproc)
echo "== $(hostname -s) at $(date '+%F %T %Z')"
read -r load1 load5 load15 _ < /proc/loadavg
echo "load: $load1 $load5 $load15 on $cpus CPUs ($(uptime -p))"
awk '/^MemTotal:/ {t = $2} /^MemAvailable:/ {a = $2}
END {printf "memory: %.1f GiB available of %.1f GiB (%.0f%%)\n", a/1048576, t/1048576, a*100/t}' /proc/meminfo
echo "disks at or over ${disk_threshold}%:"
df -h --output=target,pcent,avail "${mounts[@]}" |
awk -v limit="$disk_threshold" 'NR > 1 { p = $2; sub("%", "", p); if (p + 0 >= limit) { print " " $0; n++ } }
END { if (!n) print " none" }'
echo "top memory:"
ps -eo pid,rss,comm --sort=-rss | awk 'NR > 1 && NR <= 4 { printf " %-7s %6.0f MiB %s\n", $1, $2/1024, $3 }'
echo "failed units: $(systemctl --failed --no-legend --plain | wc -l)"
echo "journal errors (last 24h): $(journalctl -p err --since '24 hours ago' -q --no-pager | wc -l)"./health-snapshot.sh== lab at 2026-11-09 21:55:49 EST
load: 9.43 7.45 5.30 on 8 CPUs (up 17 minutes)
memory: 4.5 GiB available of 7.7 GiB (59%)
disks at or over 80%:
none
top memory:
1656 853 MiB java
39031 650 MiB java
46722 550 MiB terraform-provi
failed units: 0
journal errors (last 24h): 374To check the disk branch, lower the threshold and list two mounts:
DISK_THRESHOLD=1 DISK_MOUNTS='/ /run/shm' ./health-snapshot.sh | sed -n '4,6p'disks at or over 1%:
/ 1% 952G
/run/shm 2% 3.8GScheduled at 07:30 on weekdays with OnCalendar=Mon..Fri *-*-* 07:30:00 (or
30 7 * * 1-5 in cron) and piped to your chat tool's webhook or mail, it is a useful
daily habit. Note the design choice for disks: the script checks the mounts you name
rather than every line of df, so read-only images at 100% never cause false alarms.
And note the 374 journal errors: on a busy shared machine most are noise from one
chatty program, which is exactly why a real report would group them with
journalctl -p err --since '24 hours ago' -o cat | sort | uniq -c | sort -rn | head.
Common mistakes
crontab - replaces everything
Piping into crontab - is common in setup scripts, and it silently replaces the whole
crontab:
echo '0 3 * * * /usr/local/bin/backup.sh' | crontab -
(crontab -l 2>/dev/null; echo '*/5 * * * * /usr/local/bin/disk-check.sh 90 /') | crontab -
crontab -l
echo '0 4 * * * /usr/local/bin/oops.sh' | crontab -
crontab -l0 3 * * * /usr/local/bin/backup.sh
*/5 * * * * /usr/local/bin/disk-check.sh 90 /
0 4 * * * /usr/local/bin/oops.shThe second command shows the safe way to append (list, add, reinstall). The fourth
command wiped both existing jobs without a word. In automation, prefer a file in
/etc/cron.d/ or a systemd timer, which can be added and removed without touching
anything else.
"It works when I run it"
The job runs by hand but fails in cron. Nine times out of ten it is one of the
environment differences shown above: a command not on cron's PATH
(/bin/sh: 1: shellcheck: not found), Bash syntax run by sh, a relative path resolved
against $HOME, or an unescaped %. To debug, reproduce cron's environment instead of
guessing:
env -i HOME="$HOME" LOGNAME="$USER" PATH=/usr/bin:/bin SHELL=/bin/sh /bin/sh -c 'shellcheck --version'/bin/sh: 1: shellcheck: not foundenv -i starts from an empty environment, so you see the same failure cron saw, in your
terminal, in one second instead of waiting for the next scheduled run. Use the most
restrictive PATH your distribution's cron might use, and make sure the job's output is
redirected somewhere you can read it.
Output disappears
(CRON) info (No MTA installed, discarding output) in journalctl -u cron means your
job printed something (often an error) and cron tried to mail it. Add >> logfile 2>&1
or 2>&1 | logger -t jobname to every line.
Summary
- Load average counts running, runnable and I/O-blocked tasks; compare it with
nprocand read the 1/5/15-minute trend.vmstatseparates CPU pressure (r,us) from I/O pressure (b,wa). - Judge memory by
available, notfree; the page cache is given back on demand. df -hfor space,df -ifor inodes,du -sh -- * | sort -hto find the culprit,--apparent-sizefor sparse files,lsof +L1whendfanddudisagree.ss -tlnplists listening ports;0.0.0.0/*means reachable from the network.logger -t tagwrites to the journal;journalctl -u,-t,-p,--since,-o catand-o jsonfind it again.- Cron fields are minute, hour, day of month, month, day of week; day-of-month and
day-of-week are OR-ed. Escape
%as\%. - Cron runs
/bin/shwith a minimal environment and mails output (usually into the void). Use absolute paths, setSHELL/PATH/MAILTO, redirect output, and add a heartbeat so silence is detectable. flock -n lockfile cmdprevents overlapping runs and is released automatically on exit. systemd timers add journald logs, missed-run catch-up and overlap protection;systemd-analyze calendarchecks a schedule before you trust it.
Exercises
Write crontab lines for: (a) /usr/local/bin/sync-orders.sh every 10 minutes from 08:00
to 17:50 on weekdays, (b) /usr/local/bin/monthly-invoices.sh at 02:30 on the first of
each month, (c) /usr/local/bin/weekly-vacuum.sh at 04:15 every Sunday. Then write the
equivalent OnCalendar= values and check them with systemd-analyze calendar.
Show Solution
*/10 8-17 * * 1-5 /usr/local/bin/sync-orders.sh
30 2 1 * * /usr/local/bin/monthly-invoices.sh
15 4 * * 0 /usr/local/bin/weekly-vacuum.shcrontab ex1.crontab accepted the file and crontab -l printed it back unchanged.
The timer equivalents:
systemd-analyze calendar --iterations=2 'Mon..Fri *-*-* 08..17:00/10:00' '*-*-01 02:30:00' 'Sun *-*-* 04:15:00' | grep -E 'Original|Normalized|Next|Iteration'Normalized form: Mon..Fri *-*-* 08..17:00/10:00
Next elapse: Tue 2026-11-10 08:00:00 EST
Iteration #2: Tue 2026-11-10 08:10:00 EST
Normalized form: *-*-01 02:30:00
Next elapse: Tue 2026-12-01 02:30:00 EST
Iteration #2: Fri 2027-01-01 02:30:00 EST
Normalized form: Sun *-*-* 04:15:00
Next elapse: Sun 2026-11-15 04:15:00 EST
Iteration #2: Sun 2026-11-22 04:15:00 ESTIn (a), the hour range 8-17 includes 17:00 to 17:50; the last run of the day is at
17:50. Sunday is 0 (or 7, or sun) in cron and Sun in OnCalendar.
Print the percentage of memory available, to one decimal place, using only
/proc/meminfo and awk.
Show Solution
awk '/^MemTotal:/ {t = $2} /^MemAvailable:/ {a = $2} END {printf "%.1f%% of memory available\n", a * 100 / t}' /proc/meminfo58.0% of memory availableBoth values are in kB, so the units cancel. %% prints a literal percent sign in
printf.
Write check-heartbeat.sh FILE MAX_AGE_MINUTES that prints ok and exits 0 if FILE was
modified within MAX_AGE_MINUTES, prints an alert and exits 1 if it is older, and exits 2
if the file does not exist at all. Test all three cases (touch -d '40 minutes ago'
helps).
Show Solution
#!/usr/bin/env bash
# check-heartbeat.sh FILE MAX_AGE_MINUTES - alert when a job has stopped running
set -euo pipefail
file=${1:?usage: check-heartbeat.sh FILE MAX_AGE_MINUTES}
max_age=${2:?usage: check-heartbeat.sh FILE MAX_AGE_MINUTES}
if [[ ! -e $file ]]; then
echo "ALERT: $file does not exist; the job has never run"
exit 2
fi
if [[ -n $(find "$file" -mmin +"$max_age") ]]; then
echo "ALERT: last run $(date -r "$file" '+%F %T'), more than $max_age minutes ago"
exit 1
fi
echo "ok: last run $(date -r "$file" '+%F %T')"./check-heartbeat.sh hb/never 15; echo "exit: $?"
touch hb/last-run
./check-heartbeat.sh hb/last-run 15; echo "exit: $?"
touch -d '40 minutes ago' hb/last-run
./check-heartbeat.sh hb/last-run 15; echo "exit: $?"ALERT: hb/never does not exist; the job has never run
exit: 2
ok: last run 2026-11-09 21:55:50
exit: 0
ALERT: last run 2026-11-09 21:15:50, more than 15 minutes ago
exit: 1find FILE -mmin +N prints the file only if it is older than N minutes, so a non-empty
result means "stale". Run this from a different machine or scheduler than the job
itself; a heartbeat checked by the same crontab that lost the job proves nothing.
A report must refresh every 15 minutes from 09:00 to 17:45, Monday to Friday. Write the
OnCalendar= line and prove the first three run times with systemd-analyze.
Show Solution
systemd-analyze calendar --iterations=3 'Mon..Fri *-*-* 09..17:00/15:00' | grep -E 'Normalized|Next|Iteration'Normalized form: Mon..Fri *-*-* 09..17:00/15:00
Next elapse: Tue 2026-11-10 09:00:00 EST
Iteration #2: Tue 2026-11-10 09:15:00 EST
Iteration #3: Tue 2026-11-10 09:30:00 ESTSo the timer gets OnCalendar=Mon..Fri *-*-* 09..17:00/15:00. The hour range 09..17
with minutes 00/15 gives 09:00 through 17:45. The cron equivalent is
*/15 9-17 * * 1-5.
Write load-check.sh [FACTOR] that warns (message to stdout and to the journal with tag
load-check, exit 1) when the 5-minute load average exceeds FACTOR times the number of
CPUs (default factor 1.0). Load averages are decimals, so Bash arithmetic alone will not
do. Make it testable without actually overloading a machine.
Hint
Let an environment variable override the path of /proc/loadavg, and let awk do the
floating-point comparison through its exit status.
Show Solution
#!/usr/bin/env bash
# load-check.sh - warn when the 5-minute load average exceeds the CPU count
# LOADAVG_FILE can point at a fake file for testing.
set -euo pipefail
loadavg_file=${LOADAVG_FILE:-/proc/loadavg}
factor=${1:-1.0} # warn above factor x CPUs
cpus=$(nproc)
read -r _ load5 _ < "$loadavg_file"
limit=$(awk -v c="$cpus" -v f="$factor" 'BEGIN { print c * f }')
if awk -v l="$load5" -v m="$limit" 'BEGIN { exit !(l > m) }'; then
logger -t load-check -p user.warning "5-min load $load5 exceeds $limit ($cpus CPUs x $factor)"
echo "WARN load $load5 > $limit"
exit 1
fi
echo "ok load $load5 <= $limit"./load-check.sh; echo "exit: $?"
echo '12.40 9.75 6.10 9/512 40213' > fake-loadavg
LOADAVG_FILE=fake-loadavg ./load-check.sh; echo "exit: $?"
LOADAVG_FILE=fake-loadavg ./load-check.sh 1.5; echo "exit: $?"
journalctl -t load-check -o cat --no-pager -n 1ok load 7.61 <= 8
exit: 0
WARN load 9.75 > 8
exit: 1
ok load 9.75 <= 12
exit: 0
5-min load 9.75 exceeds 8 (8 CPUs x 1.0)awk 'BEGIN { exit !(l > m) }' exits 0 when the comparison is true, so it plugs straight
into if. The LOADAVG_FILE override is a small example of designing for testability:
the alert path ran in the lab without anyone having to load the CPUs.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), systemd 255, cron
3.0pl1-184ubuntu2, util-linux 2.39.3 (flock), November 2026. Every command was run as shown on a
shared lab machine, which is why the load and process listings show other workloads. The
test crontabs and the user timer were installed, observed for a minute or two, and removed
again. The machine's hostname in journal output is shown as lab. Not tested: loginctl enable-linger and system-wide units in /etc/systemd/system (both need root), and cron
mail delivery, since the lab has no mail server, which is how the "No MTA installed"
message was captured. Paths from the test machine's scratch directories are shown shortened to /home/sid/….
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.