Ten Advanced RHEL Break-Fix Drills
Ten RHEL faults that hide - silent until a reboot, one symptom with three causes, and two that look like permissions problems and are not. A saboteur stages them, and only reveals the cause once you have repaired it.
A script damages the machine; you repair it with nothing but the machine itself.
Most troubleshooting practice is too easy because the fault announces itself. A service is stopped, so you start it. These ten do not announce themselves. Several stay silent until a reboot. One puts three independent causes behind a single symptom. Two present as permissions problems that are not permissions problems at all.
Part of RHCSA to RHCE - A 60-Day Lab Curriculum, and the harder counterpart to the drills in RHCSA Security Labs - SELinux firewalld and Boot Recovery. Scripts: github.com/Push1697/published_quartz_blog/tree/v4/rhce-labs
Snapshot first, every single time. Some of these fill a filesystem, move the clock, or disable SELinux until you repair them.
virsh snapshot-create-as rhel01 pre-labThe saboteur writes no backups of what it changed. That is deliberate: the snapshot is the escape hatch, and reverting costs you nothing but the drill. There is no undo command, and there should not be. Never run this on a machine you care about.
Running a drill
sudo ./break/break-advanced.sh list # the ten symptoms, no spoilers
sudo ./break/break-advanced.sh random --yes # one unknown fault
sudo ./break/break-advanced.sh f4 --yes # a specific drill
sudo ./break/break-advanced.sh combo --yes # three distinct faults at once
sudo ./break/break-advanced.sh chaos 5 --yes # five, for a hostile mock
sudo ./break/break-advanced.sh status # how many pending, and since when
Then repair, verify, reboot, verify again, and only afterwards:
sudo ./verify advanced --blind # am I done yet? pass/fail only
sudo ./verify advanced # what is still wrong, once you have tried
sudo ./break/break-advanced.sh reveal # the cause, and your elapsed time
Two deliberate design choices, because a drill you can peek at is not a drill:
The saboteur names nothing. list prints only symptoms. reveal gives you
the cause and how long you took, once you are finished.
--blind exists for the verifier. Its normal output names the domain each
check examines, which does some of the hunting for you. Blind mode reports only
which numbered checks pass, so you can tell whether you are finished without
being told where to look.
Do not read
break-advanced.sh. It states the cause of all ten drills in plain English. Reading it once spends the only chance you get to meet that fault cold, andrevealexists so you never need to.
random is the version that matters. A named drill tells you the symptom in
advance, which is useful the first time and worthless the second.
The ten drills
Each gives you a symptom, not a fault. The requirement is always the same three things:
- The symptom is gone.
- The repair is minimal — nothing disabled, no permissions widened, no
setenforce 0, nochmod 777, no blanket policy module. - The repair survives a reboot.
The acceptance criteria say what "done" looks like from the outside. They deliberately do not say where to look.
f1 — Names stop resolving, and /etc/hosts is demonstrably correct
ping rhel02 fails. ping 192.168.124.12 works. You cat /etc/hosts and the
entry is right there, correctly spelled. Nothing is wrong with the network, and
DNS itself answers normally.
-
getent hosts rhel02resolves again - A name that only
/etc/hostsknows about resolves - Whatever manages that configuration on RHEL 9 reports itself consistent
- You changed one thing, not several
- Survives a reboot
Target: 10 minutes.
f2 — A service will not start, on a config error you cannot correct
The unit fails. journalctl -xeu names the file and the line. You open it, fix
it, and cannot save. As root. There is no SELinux denial in the audit log,
the filesystem is read-write, and ls -l shows exactly the permissions you
expect.
- The file is editable by root again
- The configuration parses and the service runs
- Nothing else under
/etcis left in the state that caused this - You can name the one command that would have shown you the cause in a line
- Survives a reboot
Target: 10 minutes.
f3 — The website is unreachable, and fixing one thing is not enough
curl from another host fails. So does curl localhost. This drill has three
independent causes behind that one symptom, and they surface one at a time:
each fix reveals the next failure. Do not stop when the first thing works.
-
curlagainst the configured port returns 200 locally and remotely - SELinux stayed enforcing throughout — no
setenforce 0, not even briefly - A full relabel of the served directory would change nothing
- Every port the server listens on is permitted, permanently
- Runtime and permanent firewall configuration match exactly
- Survives a reboot
Target: 20 minutes. The longest of the ten.
f4 — A unit starts by hand but never at boot, citing something that does not exist
systemctl start httpd works. After a reboot it is dead. systemctl status
mentions a unit name you have never seen, containing an escape sequence, and the
unit file itself looks entirely normal at first glance.
- The unit starts automatically at boot, confirmed by an actual reboot
- Nothing on the system depends on anything that does not exist
- No unit is in a failed state
-
systemd-analyze verifyis quiet about it - You can explain how a filesystem path becomes a systemd unit name, escape sequence and all
Target: 15 minutes.
f5 — The filesystem is full and du cannot account for it
Writes fail. df says 100%. du -sh over the whole filesystem adds up to a
fraction of that. Deleting things does not help, and there is nothing obviously
large anywhere.
- The space is back, without deleting anything that mattered
-
duanddfagree again - You did not reboot to fix it — a reboot would have hidden the cause
- You can explain why the space was invisible to
du - You can explain why a reboot would have "worked", and why that is worse
Target: 10 minutes.
f6 — Everything time-sensitive misbehaves at once
dnf complains about certificates. journalctl output is out of order and its
newest entries are dated next year. Timers show absurd next-run times. sudo
grumbles about a timestamp from the future. SSH still works.
- The clock is correct and consistent with what is installed on the box
- The service that maintains it is configured and running
- No files under
/etcor/rootare left dated in the future -
dnfworks again - Survives a reboot
Target: 15 minutes. Note that an offline lab has no upstream time source, so part of this drill is setting the clock by hand and then making the service that maintains it work against something reachable.
f7 — sudo works for some commands and refuses others
sudo ls is fine. sudo systemctl restart httpd says the command was not found
— though it plainly exists and you can run it by absolute path. Separately, a
group that had sudo rules yesterday now has none, and sudo -l for its members
is thinner than it was. Two things are wrong, not one.
- Administrative commands work through
sudowithout an absolute path - The group's rules are honoured again
-
visudo -cis clean andsudoitself prints no warnings - You fixed the rules rather than granting anyone more than they had before
- Survives a reboot
Target: 15 minutes. Know your way back in before you start: if you remove
your own sudo access entirely, su - with the root password is the only route
left.
f8 — SELinux is off after a reboot, and the config file says enforcing
getenforce reports Disabled. /etc/selinux/config says SELINUX=enforcing
and always has. setenforce 1 refuses. Everything that used to be blocked now
works, which is its own kind of alarming.
-
getenforcereportsEnforcing, and still does after a second reboot -
restorecon -Rvn /etcis silent, and no relabel is left outstanding - You can say why
/etc/selinux/configwas never the authority here - You can say why coming back from this state needs a relabel at all
Target: 15 minutes, plus two reboots.
f9 — DNS works now and breaks on every reboot
You fix /etc/resolv.conf. Resolution works. You reboot to confirm, and it is
broken again, with a nameserver you have never configured. You fix it again. It
survives until the next reboot.
- Resolution survives a reboot and taking the connection down and up
- At least one configured nameserver actually answers
-
/etc/resolv.confis reproducible — nothing regenerates it differently - Hand-editing a generated file was not your final answer
- You can name two other files on a RHEL box that are generated like this one
Target: 10 minutes.
f10 — The NFS mount succeeds and writes fail
mount reports success. findmnt looks right. df shows the share. Writing to
it fails — with one message as your user and a different one as root, which is
the clue.
Stage this one on the server for the server-side variant, or on the client for the client-side variant. They present almost identically and are fixed in different places, which is the lesson.
- A write to the mount succeeds as an ordinary user
- The export permits what it should, and nothing more
- The mount is read-write both now and after a reboot
- You fixed it on the correct node, and can say why the other one was wrong
-
showmount -efrom the client agrees withexportfs -von the server
Target: 15 minutes.
The diagnostic order these drills are teaching
By the end of the ten, this should be reflex rather than recall:
What exactly is the symptom? ← reproduce it, deliberately
↓
What changed, and when? ← journalctl --since, rpm -qa --last, ls -lt /etc
↓
Which LAYER? ← the question these drills exist for
↓
ONE hypothesis, written down
↓
Test it without fixing it
↓
Fix minimally
↓
Verify · REBOOT · verify again
The layer question is what separates these from easier drills. When a service will not start, the cause can be in its own configuration, in something layered on top of its unit, in a file attribute that is not a permission, in SELinux, in the kernel command line, or in the clock. Each of those is invisible to the tool that reveals the others — which is why a checklist of commands is not the same thing as a method.
Work out the list for yourself as you go. That is the exercise. Once you have finished all ten, compare your list against this one:
Spoiler: the layers, and the tool that exposes each Do not open this until every drill is done. It is effectively an index of the ten causes.
Layer The tool that shows it the unit as systemd actually assembles it systemctl cat,systemd-analyze verifyfile attributes beyond permissions lsattrSELinux labels, booleans and ports ls -Z,ausearch -m AVC -ts recent,semanagethe kernel command line /proc/cmdline,grubby --info=DEFAULTname-service resolution order /etc/nsswitch.conf,authselect checkfiles that are generated, not edited nmcli con show,rpm -qf, the managing daemonspace with no directory entry lsof -nP +L1,df -ithe clock timedatectl,rpm -qa --lastIf your own list had all eight before you read this, you are ready for the troubleshooting half of any RHEL exam.
Keep a drill log
The pattern in the "first hypothesis right?" column is the most honest measure of readiness there is. A rising accuracy rate means the layer question is becoming automatic; a flat one means you are still guessing.
| Date | Drill | Time to find | Time to fix | First hypothesis right? | What gave it away |
|---|---|---|---|---|---|
The gate
- All ten drills completed at least once
- Every one re-run cold via
random, under its time target -
comborepaired inside 45 minutes - No drill was ever "fixed" by disabling SELinux, widening permissions, or rebooting blindly
- The drill log is filled in, including the wrong hypotheses
- I wrote my own layer list before reading the spoiler, and it was close
The counterpart to the saboteur is the grader:
Verifying Your Own Labs Offline. ./verify advanced also works as a general
sanity sweep — every section passes on a healthy box, so it is worth running
before any mock exam.