System Monitoring and Scheduling with cron and systemd

Read load, memory, disk and port usage correctly, query the journal, and schedule reliable jobs with cron, systemd timers and flock, with the cron gotchas tested.

By The Dark DebuggerUpdated 34 min read

This article covers the two halves of keeping a Linux machine healthy without staring at it: reading its vital signs (load, memory, disk, ports, logs) and scheduling jobs that run on their own, with cron and with systemd timers. It builds on processes, signals, and job control and assumes you can write a small script with arguments and exit codes (your first scripts).

Why this matters

Two situations come up on every team that runs servers.

The first: a disk fills up at 3 a.m., the database stops accepting writes, and the first anyone hears of it is a customer. A five-line check run every five minutes would have warned you two days earlier.

The second: someone writes a backup script, runs it by hand, sees it work, adds it to cron, and goes home. It never runs successfully once. The command was not on cron's PATH, the error went to a mailbox nobody reads, and the first person to notice is the one trying to restore a backup that does not exist.

Both are avoidable with tools that are already installed. The trick is knowing how to read them and knowing exactly how cron's environment differs from your terminal.

Load average: uptime and vmstat

uptime
nproc
 21:47:19 up 8 min,  1 user,  load average: 10.11, 7.00, 3.35
8

The three numbers are the 1-, 5- and 15-minute load averages: exponentially smoothed counts of tasks that are either running on a CPU, waiting for one, or (on Linux specifically) stuck in uninterruptible I/O wait, the D state. They are not percentages. Read them against the number of CPUs:

  • Load below nproc: there is spare CPU capacity.
  • Load around nproc: every CPU is busy, nothing is queueing much.
  • Load well above nproc: work is waiting. Here, 10.11 on 8 CPUs means that, on average, about two tasks were always waiting for a turn.

The three values together show the trend. 10.11, 7.00, 3.35 is a load that is rising: the last minute was much busier than the last quarter hour. (This was the shared lab machine while several builds ran at once; it had been booted 8 minutes earlier.) A high 15-minute value with a low 1-minute value means a spike that is already over.

High load does not always mean "CPU-bound". vmstat shows what kind of waiting it is:

vmstat 2 4
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 2  1      0 1779052 209232 3642824    0    0  2423  6004 4315    9 26 15 49 10  0  0
 6  1      0 1767320 209512 3642168    0    0     0  1854 3123 4157 27 20 44  9  0  0
 6  1      0 1753844 209820 3649864    0    0     0  1170 4985 8680 35 21 36  9  0  0
 6  0      0 1791532 210312 3654564    0    0   412  1680 6215 10404 33 18 41  8  0  0

The first line is an average since boot; read the later ones. r is the run queue (tasks wanting CPU), b is tasks blocked on I/O, and the cpu block repeats top's us/sy/id/wa. Six runnable tasks with 36-44% idle CPU and about 9% wa means this load is a mix of CPU work and disk writes (bo, blocks out), not one runaway process. If b and wa are high while us is low, the disk is your bottleneck and adding CPUs would not help. The si/so columns (swap in/out) should be zero; steady non-zero values mean the machine is short of RAM.

Memory: what "available" means

free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.6Gi       1.6Gi        90Mi       3.7Gi       5.0Gi
Swap:          2.0Gi          0B       2.0Gi

New Linux users read the free column, see 1.6 GiB of 7.7, and panic. Don't. Linux puts otherwise idle RAM to work as page cache (buff/cache), keeping recently read files in memory, and hands that memory back the moment a program needs it. The number that answers "how much more can I run before the machine starts swapping" is available (5.0 GiB here): free memory plus the cache that can be dropped.

free reads /proc/meminfo, which is also the easiest source for scripts:

grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree):' /proc/meminfo
MemTotal:        8038076 kB
MemFree:         1718064 kB
MemAvailable:    5277132 kB
Buffers:          203696 kB
Cached:          3416500 kB
SwapTotal:       2097152 kB
SwapFree:        2097152 kB

Alert on MemAvailable dropping below a percentage of MemTotal, and on swap usage growing over time. A machine that is genuinely out of memory shows it in the kernel log (journalctl -k | grep -i 'out of memory') when the OOM killer ends a process with SIGKILL, the "exit 137" from the previous article.

Disk space: df

df reports space per mounted filesystem:

df -h
Filesystem      Size  Used Avail Use% Mounted on
none            3.9G     0  3.9G   0% /usr/lib/modules/6.18.54.1-microsoft-standard-WSL2
/dev/sdb         97M   97M     0 100% /usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2
none            3.9G  4.0K  3.9G   1% /mnt/wsl
[…]
/dev/sdd       1007G  4.5G  952G   1% /
[…]
tmpfs           785M   24K  785M   1% /run/user/1000

On the WSL lab machine df -h lists a dozen helper mounts; a cloud VM shows fewer, but you still get tmpfs and snap or image mounts. Two habits help. First, ask about the filesystem you care about, and add -T for its type:

df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdd       ext4 1007G  4.5G  952G   1% /

Second, don't panic about read-only images at 100%. The /dev/sdb mount above is a read-only tools image that is full by design, exactly like snap packages on Ubuntu servers. A monitoring check should name the mount points it cares about rather than alert on every line.

For scripts, choose columns with --output and use -P (POSIX format, one line per filesystem) when parsing:

df -h --output=target,size,used,avail,pcent / /dev/shm
df -P / | awk 'NR == 2 { print $5 }'
Mounted on  Size  Used Avail Use%
/          1007G  4.5G  952G   1%
/dev/shm    3.9G   66M  3.8G   2%
1%

Inodes: the other way a disk fills up

Every file uses one inode, and an ext4 filesystem has a fixed number of them. A directory with millions of tiny files (session files, a mail queue, a cache nobody cleans) can exhaust inodes while df -h still shows free gigabytes. The error is the same No space left on device, which is maddening until you know to check df -i:

df -i /
Filesystem       Inodes IUsed    IFree IUse% Mounted on
/dev/sdd       67108864 63333 67045531    1% /

If IUse% is near 100%, look for the directory with the most files (find / -xdev -type f | cut -d/ -f2-3 | sort | uniq -c | sort -rn | head) rather than the biggest one.

Finding what uses the space: du

df says how full a filesystem is; du says which directories are responsible. Here is a small web shop's data directory:

du -sh .
du -sh -- * | sort -h
du -h --max-depth=2 . | sort -rh | head -6
20M	.
8.0K	app
2.4M	cache
5.6M	uploads
12M	logs
20M	.
12M	./logs
5.6M	./uploads/2026
5.6M	./uploads
4.4M	./logs/archive
2.4M	./cache/thumbs

-s summarizes each argument, -h prints human sizes, and sort -h understands those suffixes (plain sort -n would put 12M before 8.0K). --max-depth=N (or -d N) limits how deep the report goes, which is how you walk down a full disk one level at a time. On a real server, add -x to stay on one filesystem, so du -xh / -d1 does not wander into /proc or network mounts.

When du and ls disagree

du reports blocks actually allocated, ls -l reports the file's length. Usually they match. For sparse files (VM images, database files, some logs) they do not:

ls -lh cache/disk.img
du -h cache/disk.img
du -h --apparent-size cache/disk.img
-rw-r--r-- 1 sid sid 2.0G Nov  9 21:49 cache/disk.img
0	cache/disk.img
2.0G	cache/disk.img

A 2 GB file using zero bytes of disk. The reverse puzzle, df showing a full disk that du cannot account for, is almost always a deleted file still held open by a process; lsof +L1 finds it, as shown in the process management article.

Network ports: ss

ss lists sockets. The combination you will type most is "listening TCP sockets, numeric, with processes":

ss -tlnp
State  Recv-Q Send-Q      Local Address:Port  Peer Address:PortProcess                            
LISTEN 0      1000       10.255.255.254:53         0.0.0.0:*                                      
LISTEN 0      4096           127.0.0.54:53         0.0.0.0:*                                      
LISTEN 0      5               127.0.0.1:8765       0.0.0.0:*    users:(("python3",pid=23411,fd=3))
LISTEN 0      4096        127.0.0.53%lo:53         0.0.0.0:*                                      
LISTEN 0      50                      *:46109            *:*    users:(("java",pid=8195,fd=319))  
LISTEN 0      50     [::ffff:127.0.0.1]:8090             *:*    users:(("java",pid=1656,fd=10))   
LISTEN 0      50     [::ffff:127.0.0.1]:8097             *:*    users:(("java",pid=8195,fd=10))   
LISTEN 0      50                      *:39737            *:*    users:(("java",pid=1656,fd=319))  

How to read the address column:

  • 127.0.0.1:8765 listens on loopback only: reachable from this machine, not from the network. That is what you want for an app behind a reverse proxy.
  • *:46109 or 0.0.0.0:PORT listens on every interface, so it is reachable from outside unless a firewall says otherwise. Audit these.
  • The DNS lines on port 53 have no process shown because they belong to root; without sudo, ss -p only names your own processes.

ss -s gives a summary, useful for spotting connection leaks (thousands of sockets in timewait or estab where you expect dozens):

ss -s
Total: 293
TCP:   81 (estab 2, closed 71, orphaned 0, timewait 71)
 
Transport Total     IP        IPv6
RAW	  0         0         0        
UDP	  5         4         1        
TCP	  10        4         6        
INET	  15        8         7        
FRAG	  0         0         0        

Filters narrow things down: ss -tn state established '( dport = :5432 )' shows open connections to PostgreSQL, and ss -tlnp 'sport = :8080' shows who owns port 8080.

Logs: journalctl

On systemd machines, service output, kernel messages and syslog all land in the journal. logger writes into it from scripts, with a tag and a priority, which makes your own jobs as searchable as any daemon:

logger -t nightly-backup "backup finished: 1.2 GB in 340 s"
logger -t nightly-backup -p user.err "backup FAILED: /srv/db unreadable"
journalctl -t nightly-backup --no-pager
Nov 09 21:49:06 lab nightly-backup[24216]: backup finished: 1.2 GB in 340 s
Nov 09 21:49:06 lab nightly-backup[24219]: backup FAILED: /srv/db unreadable

Each line is timestamp, host, tag with PID, message. The queries worth memorizing:

CommandShows
journalctl -u cronOne systemd unit (here the cron daemon)
journalctl --user -u NAMEA unit in your own user instance
journalctl -t TAGMessages with a syslog tag, e.g. from logger -t
journalctl -p errPriority err and worse (warning, info, debug also work)
journalctl -bSince the last boot (-b -1 is the boot before)
journalctl --since "1 hour ago"Time window; also --until, or "2026-11-09 02:00"
journalctl -fFollow new messages, like tail -f
journalctl -n 20Last 20 lines
journalctl -kKernel messages (OOM kills, disk errors)

Output formats matter when the journal feeds a script:

journalctl -t nightly-backup -p err --no-pager -o short-iso
journalctl -t nightly-backup -o cat --no-pager
journalctl -t nightly-backup -o json --no-pager | tail -1 | jq '{t: .__REALTIME_TIMESTAMP, prio: .PRIORITY, msg: .MESSAGE}'
2026-11-09T21:49:06-05:00 lab nightly-backup[24219]: backup FAILED: /srv/db unreadable
backup finished: 1.2 GB in 340 s
backup FAILED: /srv/db unreadable
{
  "t": "1794278946248870",
  "prio": "3",
  "msg": "backup FAILED: /srv/db unreadable"
}

-o cat gives bare messages, -o short-iso gives sortable timestamps, and -o json gives every field (priority 3 is err). To read the system journal without sudo, your user needs to be in the adm or systemd-journal group. journalctl --disk-usage tells you how much space the journal itself takes (490 MB on the lab machine); limits live in /etc/systemd/journald.conf.

cron: the crontab format

cron runs commands on a schedule. Each user has a crontab, edited with crontab -e, listed with crontab -l and removed with crontab -r. A line is five time fields and a command:

# ┌───────────── minute        (0-59)
# │ ┌─────────── hour          (0-23)
# │ │ ┌───────── day of month  (1-31)
# │ │ │ ┌─────── month         (1-12 or jan-dec)
# │ │ │ │ ┌───── day of week   (0-7 or sun-sat; 0 and 7 are both Sunday)
# │ │ │ │ │
  30 2 * * *  /usr/local/bin/backup.sh

Each field accepts:

SyntaxMeaningExampleRuns
*every value* * * * *every minute
Nthat value30 2 * * *02:30 daily
A-Brange0 9-17 * * *on the hour, 09:00 to 17:00
A,B,Clist0 8,12,18 * * *08:00, 12:00, 18:00
*/Nevery N*/15 * * * *:00, :15, :30, :45
A-B/Nevery N within a range0 8-18/2 * * 1-5every 2 hours 08:00-18:00, weekdays

And a few shortcuts in place of the five fields:

ShortcutEquivalent
@rebootonce, when cron starts
@hourly0 * * * *
@daily (or @midnight)0 0 * * *
@weekly0 0 * * 0
@monthly0 0 1 * *

One rule surprises nearly everyone: if both day-of-month and day-of-week are restricted, the job runs when either matches. 0 3 1-7 * 1 does not mean "the first Monday of the month"; it means "days 1 to 7, and also every Monday". The usual workaround is to restrict one field and test the other in the command: 0 3 * * 1 [ "$(date +\%d)" -le 07 ] && /usr/local/bin/job.sh (note the \%, explained below).

Besides per-user crontabs, there are system-wide ones: /etc/crontab and files in /etc/cron.d/ use the same format with an extra user field before the command, and scripts dropped into /etc/cron.daily/, /etc/cron.weekly/ and so on are run by run-parts (their file names must not contain a dot, so backup.sh there is silently skipped; name it backup).

crontab validates what you give it:

printf '61 * * * * true\n' | crontab -
"-":0: bad minute
errors in crontab file, can't install.

The cron environment is not your shell

This is where most cron jobs die. To see exactly what a job gets, I installed a crontab that dumps its environment, plus a few lines that fail in instructive ways:

# m h dom mon dow  command
* * * * *  env > /home/sid/cron/cron-env.txt
* * * * *  echo "hello from cron"
* * * * *  date +%F >> /home/sid/cron/date-broken.txt
* * * * *  date +\%F >> /home/sid/cron/date-fixed.txt
* * * * *  shellcheck --version >> /home/sid/cron/path.log 2>&1
crontab test.crontab

After the next minute ticked over:

cat cron-env.txt
HOME=/home/sid
LOGNAME=sid
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LANG=C.UTF-8
SHELL=/bin/sh
PWD=/home/sid

Six variables. Compare that with the dozens in your terminal. Things to notice:

  • SHELL=/bin/sh. The command line is run by sh (dash on Ubuntu), not Bash. Bash syntax like [[ ]], arrays or {1..5} written directly in a crontab line fails. Put logic in a script with a #!/usr/bin/env bash line, or set SHELL=/bin/bash at the top of the crontab.
  • PATH is the system default. On Ubuntu 24.04 cron loads /etc/environment through PAM, which is where that long PATH came from; on other distributions it can be as short as /usr/bin:/bin. What it never includes is anything your ~/.bashrc or ~/.profile adds: ~/.local/bin, ~/bin, nvm, pyenv, conda, or (on this machine) the lab's tool directory. That is why the shellcheck line failed:
cat path.log
/bin/sh: 1: shellcheck: not found
  • The working directory is $HOME, so relative paths in a job resolve against your home directory, not the script's directory.
  • No terminal, no profile. Nothing from .bashrc runs, and there is no $DISPLAY or SSH agent.

The % sign

cron treats an unescaped % as a newline and feeds everything after the first one to the command's standard input. The cron log shows what actually ran:

journalctl -u cron --since '2026-11-09 21:49:46' --no-pager
Nov 09 21:50:02 lab CRON[27323]: pam_unix(cron:session): session opened for user sid(uid=1000) by sid(uid=0)
[…]
Nov 09 21:50:02 lab CRON[27363]: (sid) CMD (date +%F >> /home/sid/cron/date-fixed.txt)
Nov 09 21:50:02 lab CRON[27365]: (sid) CMD (date +)
Nov 09 21:50:02 lab CRON[27368]: (sid) CMD (shellcheck --version >> /home/sid/cron/path.log 2>&1)
Nov 09 21:50:02 lab CRON[27369]: (sid) CMD (env > /home/sid/cron/cron-env.txt)
Nov 09 21:50:02 lab CRON[27364]: (sid) CMD (echo "hello from cron")
[…]
Nov 09 21:50:02 lab CRON[27324]: (CRON) info (No MTA installed, discarding output)
[…]
Nov 09 21:50:02 lab CRON[27325]: (CRON) info (No MTA installed, discarding output)

The broken line ran as date +. Even the >> date-broken.txt redirect was swallowed into standard input, so that file was never created at all. The escaped date +\%F version worked and wrote 2026-11-09. Rule: escape every % in a crontab as \%, or better, keep commands in scripts so the crontab line has no % in it.

Where the output goes

cron collects anything a job prints to stdout or stderr and emails it to the user (or to MAILTO). On a server without a mail transfer agent, that becomes the line you saw above: No MTA installed, discarding output. The echo "hello from cron" output and the error from date + were both thrown away. Your job's errors vanish unless you capture them yourself. Every cron line you write should therefore end with one of:

  • >> /var/log/myjob.log 2>&1 to append to a log file, or
  • 2>&1 | logger -t myjob to send everything to the journal,

and set MAILTO="" at the top of the crontab if you do not want mail even when an MTA exists. A crontab can set variables (SHELL, PATH, MAILTO, or your own) on lines of the form NAME=value; they apply to the lines below them.

A monitored cron job

Here is a job written the way cron jobs should be: a script with a clear exit code, a message to the journal when something is wrong, and a heartbeat file that proves it ran.

#!/usr/bin/env bash
# disk-check.sh - warn when a filesystem crosses a usage threshold
# usage: disk-check.sh THRESHOLD_PERCENT MOUNTPOINT...
set -euo pipefail
 
threshold=${1:?usage: disk-check.sh THRESHOLD_PERCENT MOUNTPOINT...}
shift
state_dir=${STATE_DIR:-$HOME/.local/state/disk-check}
mkdir -p "$state_dir"
 
status=0
while read -r target pcent; do
  used=${pcent%\%}
  if (( used >= threshold )); then
    msg="$target is ${used}% full (threshold ${threshold}%)"
    logger -t disk-check -p user.warning "$msg"
    echo "$(date '+%F %T') WARN $msg"
    status=1
  else
    echo "$(date '+%F %T') ok   $target ${used}%"
  fi
done < <(df --output=target,pcent "$@" | tail -n +2)
 
touch "$state_dir/last-run"       # heartbeat: proves the job ran at all
exit "$status"

Test it by hand first, including the failure path (a 1% threshold forces a warning):

./disk-check.sh 90 / /run/shm; echo "exit: $?"
./disk-check.sh 1 /; echo "exit: $?"
journalctl -t disk-check -o short-iso --no-pager
2026-11-09 21:50:38 ok   / 1%
2026-11-09 21:50:38 ok   /run/shm 2%
exit: 0
2026-11-09 21:50:38 WARN / is 1% full (threshold 1%)
exit: 1
2026-11-09T21:50:38-05:00 lab disk-check[29779]: / is 1% full (threshold 1%)

Then the crontab. In production this would be */5; for the test I used every minute:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""
STATE_DIR=/home/sid/cron/state
# m  h  dom mon dow  command
*    *  *   *   *    flock -n /tmp/disk-check.lock /home/sid/cron/disk-check.sh 90 / >> /home/sid/cron/disk-check.log 2>&1

A minute later:

cat disk-check.log
journalctl -u cron --since '2026-11-09 21:50:59' --no-pager | grep -v pam_unix
2026-11-09 21:51:01 ok   / 1%
Nov 09 21:51:01 lab cron[181]: (sid) RELOAD (crontabs/sid)
Nov 09 21:51:01 lab CRON[31365]: (sid) CMD (flock -n /tmp/disk-check.lock /home/sid/cron/disk-check.sh 90 / >> /home/sid/cron/disk-check.log 2>&1)

Every line of that crontab is deliberate: SHELL and PATH are explicit, MAILTO="" because output goes to a log, absolute paths everywhere, flock so runs never overlap (next section), and >> log 2>&1 so errors are kept.

The heartbeat solves the problem nothing else does: a job that silently stops running produces no errors at all. The crontab gets wiped, the server is rebuilt without it, the script is renamed. A second, independent check (on another machine, or in your monitoring system) asks "was the heartbeat touched recently?":

find state/last-run -mmin -15
state/last-run

find -mmin -15 prints the file only if it was modified in the last 15 minutes; empty output means the job has stopped. Exercise 3 turns this into a script.

Preventing overlapping runs with flock

A job scheduled every 5 minutes that occasionally takes 7 will eventually run twice at once: two backups writing the same file, two imports inserting the same rows. flock from util-linux takes a lock on a file for the duration of a command. Here slow-job.sh takes five seconds, and a second copy starts half a second after the first:

flock -n /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -n /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait
21:50:38 job 29795 started
second run exit: 1
21:50:43 job 29795 finished

-n (non-blocking) makes the second copy give up immediately with exit status 1. -E chooses a different exit code, so your monitoring can tell "skipped because busy" from "failed":

flock -n -E 75 /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -n -E 75 /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait
21:50:43 job 30118 started
second run exit: 75
21:50:48 job 30118 finished

Or wait for the lock with a timeout (-w seconds) when the second run should queue rather than skip:

flock /tmp/slow.lock ./slow-job.sh & sleep 0.5; flock -w 10 /tmp/slow.lock ./slow-job.sh; echo "second run exit: $?"; wait
21:50:48 job 30465 started
21:50:53 job 30465 finished
21:50:53 job 30910 started
21:50:58 job 30910 finished
second run exit: 0

The lock is held by an open file descriptor, so the kernel releases it automatically when the process exits, even after kill -9 or a crash. That is the big advantage over "PID file" locks that a crashed job leaves behind. You can also take the lock inside the script itself, so it is protected no matter who starts it:

#!/usr/bin/env bash
# self-lock.sh - a script that refuses to run twice at the same time
exec 9> "/tmp/${0##*/}.lock"
if ! flock -n 9; then
  echo "$(date +%T) another copy is running, exiting" >&2
  exit 1
fi
echo "$(date +%T) got the lock, working"
sleep 3
echo "$(date +%T) done"
./self-lock.sh & sleep 0.5; ./self-lock.sh; echo "exit: $?"; wait
21:51:25 got the lock, working
21:51:26 another copy is running, exiting
exit: 1
21:51:28 done

exec 9> file opens file descriptor 9 for the rest of the script, and flock -n 9 locks it. The production script template uses exactly this pattern.

systemd timers: the modern alternative

systemd can run jobs on a schedule too. A timer is two small unit files: a .service that says what to run, and a .timer that says when. User units live in ~/.config/systemd/user/ and need no root; system units go in /etc/systemd/system/ and use systemctl without --user.

# ~/.config/systemd/user/disk-check.service
[Unit]
Description=Check disk usage and warn above 90%
 
[Service]
Type=oneshot
Environment=STATE_DIR=%h/cron/state
ExecStart=%h/cron/disk-check.sh 90 /
# ~/.config/systemd/user/disk-check.timer
[Unit]
Description=Run disk-check every minute
 
[Timer]
OnCalendar=minutely
AccuracySec=1s
Persistent=true
 
[Install]
WantedBy=timers.target

%h expands to your home directory. Type=oneshot means "runs to completion", the right type for scheduled jobs. In the timer, OnCalendar is the schedule, AccuracySec=1s asks for precise timing (the default lets systemd batch wake-ups within a minute), and Persistent=true means a run missed while the machine was off happens at the next boot, something cron simply cannot do.

Check the files, load them and start the timer:

systemd-analyze --user verify ~/.config/systemd/user/disk-check.service ~/.config/systemd/user/disk-check.timer
systemctl --user daemon-reload
systemctl --user enable --now disk-check.timer
systemctl --user list-timers disk-check.timer --no-pager
Created symlink /home/sid/.config/systemd/user/timers.target.wants/disk-check.timer → /home/sid/.config/systemd/user/disk-check.timer.
NEXT                        LEFT LAST PASSED UNIT             ACTIVATES
Mon 2026-11-09 21:57:00 EST  54s -         - disk-check.timer disk-check.service
 
1 timers listed.
Pass --all to see loaded but inactive timers, too.

verify printed nothing, which means no errors. enable makes the timer start at login (or boot, for system units) and --now starts it immediately. You can also run the job on demand, which is something cron has no clean equivalent for:

systemctl --user start disk-check.service
systemctl --user status disk-check.service --no-pager
○ disk-check.service - Check disk usage and warn above 90%
     Loaded: loaded (/home/sid/.config/systemd/user/disk-check.service; static)
     Active: inactive (dead) since Mon 2026-11-09 21:56:05 EST; 37ms ago
TriggeredBy: ● disk-check.timer
    Process: 48395 ExecStart=/home/sid/cron/disk-check.sh 90 / (code=exited, status=0/SUCCESS)
   Main PID: 48395 (code=exited, status=0/SUCCESS)
        CPU: 27ms
 
Nov 09 21:56:05 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:56:05 lab disk-check.sh[48395]: 2026-11-09 21:56:05 ok   / 1%
Nov 09 21:56:05 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.

inactive (dead) with status=0/SUCCESS is the healthy state for a oneshot service between runs. After the next full minute, the timer had fired on its own, and the script's output was in the journal without any redirection:

journalctl --user -u disk-check.service --since '2026-11-09 21:56:04' --no-pager
systemctl --user list-timers disk-check.timer --no-pager
Nov 09 21:56:05 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:56:05 lab disk-check.sh[48395]: 2026-11-09 21:56:05 ok   / 1%
Nov 09 21:56:05 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.
Nov 09 21:57:00 lab systemd[397]: Starting disk-check.service - Check disk usage and warn above 90%...
Nov 09 21:57:00 lab disk-check.sh[52669]: 2026-11-09 21:57:00 ok   / 1%
Nov 09 21:57:00 lab systemd[397]: Finished disk-check.service - Check disk usage and warn above 90%.
NEXT                        LEFT LAST                          PASSED UNIT             ACTIVATES
Mon 2026-11-09 21:58:00 EST  59s Mon 2026-11-09 21:57:00 EST 83ms ago disk-check.timer disk-check.service
 
1 timers listed.
Pass --all to see loaded but inactive timers, too.

To remove a timer: systemctl --user disable --now disk-check.timer, delete the two files, and run systemctl --user daemon-reload. With Persistent=true, systemd also keeps a timestamp file under ~/.local/share/systemd/timers/; delete it too if you are removing the timer for good.

OnCalendar syntax

The format is DayOfWeek Year-Month-Day Hour:Minute:Second, with *, ranges (..), lists (,) and steps (/). You never have to guess what a schedule means, because systemd-analyze calendar tells you:

systemd-analyze calendar 'Mon..Fri *-*-* 02:30:00'
systemd-analyze calendar --iterations=3 '*:0/15'
Normalized form: Mon..Fri *-*-* 02:30:00
    Next elapse: Tue 2026-11-10 02:30:00 EST
       (in UTC): Tue 2026-11-10 07:30:00 UTC
       From now: 4h 38min left
  Original form: *:0/15
Normalized form: *-*-* *:00/15:00
    Next elapse: Mon 2026-11-09 22:00:00 EST
       (in UTC): Tue 2026-11-10 03:00:00 UTC
       From now: 8min left
   Iteration #2: Mon 2026-11-09 22:15:00 EST
       (in UTC): Tue 2026-11-10 03:15:00 UTC
       From now: 23min left
   Iteration #3: Mon 2026-11-09 22:30:00 EST
       (in UTC): Tue 2026-11-10 03:30:00 UTC
       From now: 38min left

And the "first Monday of the month" that cron cannot express directly:

systemd-analyze calendar 'Mon *-*-1..7 03:00'
  Original form: Mon *-*-1..7 03:00
Normalized form: Mon *-*-01..07 03:00:00
    Next elapse: Mon 2026-12-07 03:00:00 EST
       (in UTC): Mon 2026-12-07 08:00:00 UTC
       From now: 3 weeks 6 days left

Unlike cron, systemd combines day-of-week and day-of-month with AND, so this really is the first Monday.

cron or a timer?

cronsystemd timer
SetupOne lineTwo unit files
LogsWhatever you redirectAutomatic, in the journal, per unit
Missed runs (machine off)LostPersistent=true catches up
Overlap protectionAdd flockBuilt in: a running oneshot is not started again
EnvironmentMinimal, surprisingExplicit (Environment=, EnvironmentFile=)
Resource limits, sandboxingNoMemoryMax=, CPUQuota=, ProtectSystem= …
Status and next runNot reallysystemctl list-timers
PortabilityEvery Unixsystemd Linux only

My recommendation: use a timer for anything that matters on a systemd machine, because logging, overlap protection and missed-run handling come for free. Use cron for quick, low-stakes jobs, for systems without systemd, or where a team already manages crontabs. One caveat for user timers: they run only while your user's systemd instance is running, which normally means while you are logged in. On a server, an administrator runs loginctl enable-linger yourname to keep it running, or you install the units as system units.

Where you'll use this

A common request on a team is a morning report: one screen per server, posted to chat or emailed, so that problems are noticed before they become incidents. Everything in this article fits in about twenty lines:

#!/usr/bin/env bash
# health-snapshot.sh - one-screen summary of a Linux host, for a morning report
set -uo pipefail
 
disk_threshold=${DISK_THRESHOLD:-80}
read -ra mounts <<< "${DISK_MOUNTS:-/}"
cpus=$(nproc)
 
echo "== $(hostname -s) at $(date '+%F %T %Z')"
 
read -r load1 load5 load15 _ < /proc/loadavg
echo "load:    $load1 $load5 $load15 on $cpus CPUs ($(uptime -p))"
 
awk '/^MemTotal:/ {t = $2} /^MemAvailable:/ {a = $2}
     END {printf "memory:  %.1f GiB available of %.1f GiB (%.0f%%)\n", a/1048576, t/1048576, a*100/t}' /proc/meminfo
 
echo "disks at or over ${disk_threshold}%:"
df -h --output=target,pcent,avail "${mounts[@]}" |
  awk -v limit="$disk_threshold" 'NR > 1 { p = $2; sub("%", "", p); if (p + 0 >= limit) { print "  " $0; n++ } }
                                  END { if (!n) print "  none" }'
 
echo "top memory:"
ps -eo pid,rss,comm --sort=-rss | awk 'NR > 1 && NR <= 4 { printf "  %-7s %6.0f MiB  %s\n", $1, $2/1024, $3 }'
 
echo "failed units: $(systemctl --failed --no-legend --plain | wc -l)"
echo "journal errors (last 24h): $(journalctl -p err --since '24 hours ago' -q --no-pager | wc -l)"
./health-snapshot.sh
== lab at 2026-11-09 21:55:49 EST
load:    9.43 7.45 5.30 on 8 CPUs (up 17 minutes)
memory:  4.5 GiB available of 7.7 GiB (59%)
disks at or over 80%:
  none
top memory:
  1656       853 MiB  java
  39031      650 MiB  java
  46722      550 MiB  terraform-provi
failed units: 0
journal errors (last 24h): 374

To check the disk branch, lower the threshold and list two mounts:

DISK_THRESHOLD=1 DISK_MOUNTS='/ /run/shm' ./health-snapshot.sh | sed -n '4,6p'
disks at or over 1%:
  /            1%  952G
  /run/shm     2%  3.8G

Scheduled at 07:30 on weekdays with OnCalendar=Mon..Fri *-*-* 07:30:00 (or 30 7 * * 1-5 in cron) and piped to your chat tool's webhook or mail, it is a useful daily habit. Note the design choice for disks: the script checks the mounts you name rather than every line of df, so read-only images at 100% never cause false alarms. And note the 374 journal errors: on a busy shared machine most are noise from one chatty program, which is exactly why a real report would group them with journalctl -p err --since '24 hours ago' -o cat | sort | uniq -c | sort -rn | head.

Common mistakes

crontab - replaces everything

Piping into crontab - is common in setup scripts, and it silently replaces the whole crontab:

echo '0 3 * * * /usr/local/bin/backup.sh' | crontab -
(crontab -l 2>/dev/null; echo '*/5 * * * * /usr/local/bin/disk-check.sh 90 /') | crontab -
crontab -l
echo '0 4 * * * /usr/local/bin/oops.sh' | crontab -
crontab -l
0 3 * * * /usr/local/bin/backup.sh
*/5 * * * * /usr/local/bin/disk-check.sh 90 /
0 4 * * * /usr/local/bin/oops.sh

The second command shows the safe way to append (list, add, reinstall). The fourth command wiped both existing jobs without a word. In automation, prefer a file in /etc/cron.d/ or a systemd timer, which can be added and removed without touching anything else.

"It works when I run it"

The job runs by hand but fails in cron. Nine times out of ten it is one of the environment differences shown above: a command not on cron's PATH (/bin/sh: 1: shellcheck: not found), Bash syntax run by sh, a relative path resolved against $HOME, or an unescaped %. To debug, reproduce cron's environment instead of guessing:

env -i HOME="$HOME" LOGNAME="$USER" PATH=/usr/bin:/bin SHELL=/bin/sh /bin/sh -c 'shellcheck --version'
/bin/sh: 1: shellcheck: not found

env -i starts from an empty environment, so you see the same failure cron saw, in your terminal, in one second instead of waiting for the next scheduled run. Use the most restrictive PATH your distribution's cron might use, and make sure the job's output is redirected somewhere you can read it.

Output disappears

(CRON) info (No MTA installed, discarding output) in journalctl -u cron means your job printed something (often an error) and cron tried to mail it. Add >> logfile 2>&1 or 2>&1 | logger -t jobname to every line.

Summary

  • Load average counts running, runnable and I/O-blocked tasks; compare it with nproc and read the 1/5/15-minute trend. vmstat separates CPU pressure (r, us) from I/O pressure (b, wa).
  • Judge memory by available, not free; the page cache is given back on demand.
  • df -h for space, df -i for inodes, du -sh -- * | sort -h to find the culprit, --apparent-size for sparse files, lsof +L1 when df and du disagree.
  • ss -tlnp lists listening ports; 0.0.0.0/* means reachable from the network.
  • logger -t tag writes to the journal; journalctl -u, -t, -p, --since, -o cat and -o json find it again.
  • Cron fields are minute, hour, day of month, month, day of week; day-of-month and day-of-week are OR-ed. Escape % as \%.
  • Cron runs /bin/sh with a minimal environment and mails output (usually into the void). Use absolute paths, set SHELL/PATH/MAILTO, redirect output, and add a heartbeat so silence is detectable.
  • flock -n lockfile cmd prevents overlapping runs and is released automatically on exit. systemd timers add journald logs, missed-run catch-up and overlap protection; systemd-analyze calendar checks a schedule before you trust it.

Exercises

Exercise 1: Write the scheduleseasy

Write crontab lines for: (a) /usr/local/bin/sync-orders.sh every 10 minutes from 08:00 to 17:50 on weekdays, (b) /usr/local/bin/monthly-invoices.sh at 02:30 on the first of each month, (c) /usr/local/bin/weekly-vacuum.sh at 04:15 every Sunday. Then write the equivalent OnCalendar= values and check them with systemd-analyze calendar.

Show Solution
*/10 8-17 * * 1-5  /usr/local/bin/sync-orders.sh
30   2    1 * *    /usr/local/bin/monthly-invoices.sh
15   4    * * 0    /usr/local/bin/weekly-vacuum.sh

crontab ex1.crontab accepted the file and crontab -l printed it back unchanged. The timer equivalents:

systemd-analyze calendar --iterations=2 'Mon..Fri *-*-* 08..17:00/10:00' '*-*-01 02:30:00' 'Sun *-*-* 04:15:00' | grep -E 'Original|Normalized|Next|Iteration'
Normalized form: Mon..Fri *-*-* 08..17:00/10:00
    Next elapse: Tue 2026-11-10 08:00:00 EST
   Iteration #2: Tue 2026-11-10 08:10:00 EST
Normalized form: *-*-01 02:30:00
    Next elapse: Tue 2026-12-01 02:30:00 EST
   Iteration #2: Fri 2027-01-01 02:30:00 EST
Normalized form: Sun *-*-* 04:15:00
    Next elapse: Sun 2026-11-15 04:15:00 EST
   Iteration #2: Sun 2026-11-22 04:15:00 EST

In (a), the hour range 8-17 includes 17:00 to 17:50; the last run of the day is at 17:50. Sunday is 0 (or 7, or sun) in cron and Sun in OnCalendar.

Exercise 2: Available memory as a percentageeasy

Print the percentage of memory available, to one decimal place, using only /proc/meminfo and awk.

Show Solution
awk '/^MemTotal:/ {t = $2} /^MemAvailable:/ {a = $2} END {printf "%.1f%% of memory available\n", a * 100 / t}' /proc/meminfo
58.0% of memory available

Both values are in kB, so the units cancel. %% prints a literal percent sign in printf.

Exercise 3: A dead man's switchmedium

Write check-heartbeat.sh FILE MAX_AGE_MINUTES that prints ok and exits 0 if FILE was modified within MAX_AGE_MINUTES, prints an alert and exits 1 if it is older, and exits 2 if the file does not exist at all. Test all three cases (touch -d '40 minutes ago' helps).

Show Solution
#!/usr/bin/env bash
# check-heartbeat.sh FILE MAX_AGE_MINUTES - alert when a job has stopped running
set -euo pipefail
file=${1:?usage: check-heartbeat.sh FILE MAX_AGE_MINUTES}
max_age=${2:?usage: check-heartbeat.sh FILE MAX_AGE_MINUTES}
 
if [[ ! -e $file ]]; then
  echo "ALERT: $file does not exist; the job has never run"
  exit 2
fi
if [[ -n $(find "$file" -mmin +"$max_age") ]]; then
  echo "ALERT: last run $(date -r "$file" '+%F %T'), more than $max_age minutes ago"
  exit 1
fi
echo "ok: last run $(date -r "$file" '+%F %T')"
./check-heartbeat.sh hb/never 15; echo "exit: $?"
touch hb/last-run
./check-heartbeat.sh hb/last-run 15; echo "exit: $?"
touch -d '40 minutes ago' hb/last-run
./check-heartbeat.sh hb/last-run 15; echo "exit: $?"
ALERT: hb/never does not exist; the job has never run
exit: 2
ok: last run 2026-11-09 21:55:50
exit: 0
ALERT: last run 2026-11-09 21:15:50, more than 15 minutes ago
exit: 1

find FILE -mmin +N prints the file only if it is older than N minutes, so a non-empty result means "stale". Run this from a different machine or scheduler than the job itself; a heartbeat checked by the same crontab that lost the job proves nothing.

Exercise 4: Business-hours timermedium

A report must refresh every 15 minutes from 09:00 to 17:45, Monday to Friday. Write the OnCalendar= line and prove the first three run times with systemd-analyze.

Show Solution
systemd-analyze calendar --iterations=3 'Mon..Fri *-*-* 09..17:00/15:00' | grep -E 'Normalized|Next|Iteration'
Normalized form: Mon..Fri *-*-* 09..17:00/15:00
    Next elapse: Tue 2026-11-10 09:00:00 EST
   Iteration #2: Tue 2026-11-10 09:15:00 EST
   Iteration #3: Tue 2026-11-10 09:30:00 EST

So the timer gets OnCalendar=Mon..Fri *-*-* 09..17:00/15:00. The hour range 09..17 with minutes 00/15 gives 09:00 through 17:45. The cron equivalent is */15 9-17 * * 1-5.

Exercise 5: Load alert you can testhard

Write load-check.sh [FACTOR] that warns (message to stdout and to the journal with tag load-check, exit 1) when the 5-minute load average exceeds FACTOR times the number of CPUs (default factor 1.0). Load averages are decimals, so Bash arithmetic alone will not do. Make it testable without actually overloading a machine.

Hint

Let an environment variable override the path of /proc/loadavg, and let awk do the floating-point comparison through its exit status.

Show Solution
#!/usr/bin/env bash
# load-check.sh - warn when the 5-minute load average exceeds the CPU count
# LOADAVG_FILE can point at a fake file for testing.
set -euo pipefail
 
loadavg_file=${LOADAVG_FILE:-/proc/loadavg}
factor=${1:-1.0}            # warn above factor x CPUs
cpus=$(nproc)
 
read -r _ load5 _ < "$loadavg_file"
limit=$(awk -v c="$cpus" -v f="$factor" 'BEGIN { print c * f }')
 
if awk -v l="$load5" -v m="$limit" 'BEGIN { exit !(l > m) }'; then
  logger -t load-check -p user.warning "5-min load $load5 exceeds $limit ($cpus CPUs x $factor)"
  echo "WARN load $load5 > $limit"
  exit 1
fi
echo "ok   load $load5 <= $limit"
./load-check.sh; echo "exit: $?"
echo '12.40 9.75 6.10 9/512 40213' > fake-loadavg
LOADAVG_FILE=fake-loadavg ./load-check.sh; echo "exit: $?"
LOADAVG_FILE=fake-loadavg ./load-check.sh 1.5; echo "exit: $?"
journalctl -t load-check -o cat --no-pager -n 1
ok   load 7.61 <= 8
exit: 0
WARN load 9.75 > 8
exit: 1
ok   load 9.75 <= 12
exit: 0
5-min load 9.75 exceeds 8 (8 CPUs x 1.0)

awk 'BEGIN { exit !(l > m) }' exits 0 when the comparison is true, so it plugs straight into if. The LOADAVG_FILE override is a small example of designing for testability: the alert path ran in the lab without anyone having to load the CPUs.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), systemd 255, cron 3.0pl1-184ubuntu2, util-linux 2.39.3 (flock), November 2026. Every command was run as shown on a shared lab machine, which is why the load and process listings show other workloads. The test crontabs and the user timer were installed, observed for a minute or two, and removed again. The machine's hostname in journal output is shown as lab. Not tested: loginctl enable-linger and system-wide units in /etc/systemd/system (both need root), and cron mail delivery, since the lab has no mail server, which is how the "No MTA installed" message was captured. Paths from the test machine's scratch directories are shown shortened to /home/sid/….

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.