Verifying Your Own Labs Offline
An offline bash harness that grades a RHEL lab against its acceptance criteria, proves a fix survived a reboot by tracking boot IDs, and scores mock exams the way Red Hat does.
Self-study has one structural weakness: nobody marks your work. You finish a lab, it looks right, you move on — and the thing that would have cost you marks goes unnoticed for weeks.
This is a small bash harness that marks it for you. It reads the state of the machine, checks it against the lab's acceptance criteria, and tells you what is unmet — never how to fix it, because that part is the lab.
Bash and coreutils only. No network, nothing to install, nothing sent anywhere. Part of RHCSA to RHCE - A 60-Day Lab Curriculum; source at github.com/Push1697/published_quartz_blog/tree/v4/rhce-labs.
What it looks like
== Lab 2.5 — Partitions, filesystems, swap, fstab
rhel01 · up 0d 00h 04m · 2026-09-09 14:02:11
3. /data is mounted by UUID, persistently
[PASS] /data is mounted right now
[PASS] and it is /dev/vdb1 that is mounted there
[FAIL] the fstab entry uses UUID=, not /dev/vdb1 [P]
/dev/vdb1 /data xfs defaults,noexec,nodev 0 0
the entry does not start with UUID=
[PASS] findmnt --verify is happy with /etc/fstab [P]
--------------------------------------------------------
14 passed 1 failed 0 warn 0 skipped 2 to check by hand
Not yet met:
- the fstab entry uses UUID=, not /dev/vdb1
Three things in that output are deliberate.
It shows what the machine actually said. A failing check prints the captured output, so you diagnose from evidence rather than guessing what the checker wanted.
It never prints the fix. No "run blkid and use the UUID". Working that out
from a stated requirement is the skill being trained.
Some criteria are marked [P]. Those are the ones that vanish on a reboot.
The reboot problem
Red Hat exams grade the machine after a restart. A setsebool without -P,
a firewall rule never made permanent, a service started but not enabled — each
looks perfect until the machine comes back.
So the harness records the boot ID on every clean pass:
sudo ./verify 2.5 # passes
sudo reboot
sudo ./verify 2.5 --after-reboot # the run that counts
OK: passed on a different boot than the last clean run — reboot-proven.
Until you see that line, you have proved nothing. A pass on the same boot as the
last pass says only that the machine is still in the state you left it in.
--after-reboot refuses to run at all if the machine has been up more than half
an hour, so you cannot accidentally claim the credit.
Grading a requirement rather than a command
The interesting checks are the ones that test the outcome rather than the method. Three examples.
Did the filesystem actually grow, or just the volume? Comparing the logical
volume's size against what the filesystem reports catches the half-answer —
lvextend without -r:
lvb=$(lvs --noheadings --units b --nosuffix -o lv_size "$src")
fsb=$(df -B1 --output=size /app | tail -1)
awk -v l="$lvb" -v f="$fsb" 'BEGIN { exit !(f > l * 0.9) }'
Did the SELinux label come from policy, or from chcon? A dry-run relabel
that wants to change something proves the label is not backed by policy and will
not survive restorecon:
out=$(restorecon -Rvn /web 2>&1)
[[ -z ${out//[[:space:]]/} ]]
Does the sticky bit actually work? Not "is the bit set", but "can one group member delete another's file":
runuser -u raj -- touch /srv/project/testfile
runuser -u amit -- rm -f /srv/project/testfile # must fail
That last kind matters most. Checking the mode string tells you what was configured; trying the operation tells you whether it works. Those are different questions, and only the second one is what a grader asks.
A note on the obvious trap: testing permission bits with a regex over the octal mode is wrong in a way that silently passes.
^[0-9]?[0-9]?[2367]matches750through backtracking, so a directory with no group write permission reports as group-writable. Test the bit:(( (8#$mode & 0020) != 0 )).
Scoring mock exams
For the mock papers the harness runs one section per exam task and scores tasks fully correct, scaled to 300 against the real 210 pass mark. A task with one failed check scores nothing for that task, which is how Red Hat grades:
Tasks fully correct: 12/15
Scaled score: 240/300 (80%) — pass mark is 210/300 (70%)
-> Pass, but below the 85% go/no-go bar.
Each mock grader also prints its own paper, with every ambiguous value pinned — usernames, UIDs, sizes, ports — so the grading is deterministic:
./verify ex200-b --spec > ~/paper.txt
Papers: Two RHCSA Mock Exam Papers and Ansible Scenario Labs and the EX294 Mock.
Grading automation with the same checks
The Ansible half of the curriculum re-does every manual lab as a playbook, and
the end state is identical — so the same checkers grade it. The Ansible-lab
checkers bundle the relevant manual check script, ship it to the managed node
with the script module, and run it there, from the control node:
a_remote_verify() {
# library + lab script concatenated into one self-contained file
{ cat lib/verify-lib.sh; grep -v '^\. "\$(dirname' "labs/$lab"; } > "$tmp"
ansible "$pat" -m script -a "$tmp --no-color --no-mutate" --become
}
You never log in, which is the rule for that week (Automating the RHCSA Set with Ansible). The Ansible checkers additionally run your playbooks twice, because idempotence cannot be verified any other way.
What a script will not grade
Some acceptance criteria are judgement, and pretending otherwise would be worse
than admitting it. Those print as [CHECK] and are counted separately from
passes and failures:
[CHECK] You can explain why XFS can never be shrunk
If you cannot say it out loud in one sentence, revisit it now.
[CHECK] You reset the root password from the GRUB prompt
Nobody is checking those but you. In practice they are the ones that decide how the exam goes.
Getting it onto a machine with no network
./verify bundle writes one self-contained file per checker, each with the
library embedded, so a single scp moves a single check:
./verify bundle
scp dist/lab-2.5.sh user@rhel01:
ssh user@rhel01 'sudo ./lab-2.5.sh'
Writing your own checks
The harness is about 470 lines and the API is deliberately small:
section "3. What the requirement asked for"
check "a plain-English statement of the requirement" some_command --flags
check -p "…and it survives a reboot" another_command
check_eq "the value is exactly this" 5000 "$(group_gid developers)"
check_not "this must fail" rm /srv/project/someone-elses-file
manual "something only you can judge" "why the script cannot"
summary
check runs any command or shell function, so the logic for anything
non-trivial lives in a named function in the lab script rather than in a nest of
quoted subshells:
noexec_enforced() {
local f=/data/verify-noexec-$$.sh
printf '#!/bin/bash\necho hi\n' > "$f" && chmod +x "$f"
if "$f" >/dev/null 2>&1; then
echo "a script under /data executed — noexec is not in force"
rm -f "$f"; return 1
fi
rm -f "$f"
}
check "a script under /data genuinely cannot execute" noexec_enforced
Checks that create or delete anything are gated behind mutating, so
--no-mutate gives you a read-only pass, and every artefact is cleaned up on
exit whatever happens.
The daily loop it enforces
Read the requirement (not the hint)
↓
Build it
↓
./verify <lab> ← fix what it reports, not what it suggests
↓
findmnt --verify (if you touched fstab)
↓
sudo reboot
↓
./verify <lab> --after-reboot ← this is the run that counts
↓
"reboot-proven" → the lab is done
The counterpart is a script that breaks the machine instead of grading it: Ten Advanced RHEL Break-Fix Drills.