Linux+ Certification Prep

A twenty-page map

On 15 July 2025 CompTIA launched exam XK0-006, the eighth version of Linux+. Its official objectives document, version 5.0, runs to twenty pages, and it is the most useful study aid you will own, because it lists every topic the exam can draw on. This course follows it objective by objective. Each lesson names the objective numbers it covers, so you can check your progress against CompTIA's own list rather than against anyone's summary of it.

Here are the facts from CompTIA's Linux+ page as it read on 27 October 2026. Check them again before you book, because exam details change.

CompTIA's page lists only this version. The previous exam, XK0-005, no longer appears on it, and the page where CompTIA posts draft objectives for exams in development showed no new Linux+ version. So the exam a learner starting now will sit is XK0-006, and that is the exam this course teaches.

The exam has five domains, weighted as follows. The weights tell you roughly how many of your questions each domain supplies.

Notice that Troubleshooting is the second-heaviest domain. You cannot fix what you do not understand, so the troubleshooting lessons come last and draw on everything before them.

Why this matters: the objectives document is the syllabus. When you are unsure whether something is on the exam, look for it there, not on a forum.

Reading an objective

Every objective begins with a verb, and the verb tells you how deep to go. Objective 1.1 says "Explain basic Linux concepts." Objective 1.3 says "Given a scenario, manage storage in a Linux system." The first asks you to understand and describe. The second asks you to choose and apply the right command for a situation, which is the kind of objective that turns into a performance-based question.

Under each objective sits a bulleted list of tools, files and concepts. The document says plainly that these lists are examples and not exhaustive, so other related technologies may appear on the exam. Treat the bullets as a minimum. The document also ends with an acronym list and a sample hardware and software list. That list names the distributions CompTIA recommends for practice: AlmaLinux, Debian, Fedora Linux, openSUSE or SUSE Linux Enterprise Server, Red Hat Enterprise Linux, Rocky Linux and Ubuntu. It even lists access to a large language model, which matters in Lesson 31.

One more line in the document deserves attention. CompTIA does not condone "brain dumps," sites that sell copies of real exam questions, and says candidates who use them will have their certifications revoked and be suspended from future testing. Every question in this course was written for the course. None comes from the exam.

What a performance-based question looks like

CompTIA's page on performance-based questions (PBQs) describes two kinds. Simulations approximate a tool or environment, such as a terminal or an operating system, with limited functionality. Virtual environments are real virtual machines. Linux+ is on CompTIA's list of exams that use simulations. According to that page, a simulation has a reset button, you can skip it and come back later with your work saved, more than one approach can be valid, and partial credit may be given.

A Linux+ simulation might show a terminal and ask you to fix a broken mount, set permissions on a directory, or pick the commands that build a logical volume in the right order. You cannot fake that by memorizing definitions. You prepare for it by typing the commands yourself, which is why this course has you build a lab today.

Why this matters: an objective that begins "Given a scenario" means you should be able to do the task at a prompt, not only recognize its name.

Practical rules before you book

CompTIA delivers its exams at Pearson VUE test centers and online. A few of its published policies are worth knowing months ahead.

• Identification. You need two original, unexpired IDs, one of them a government-issued photo ID, and the name must match your registration exactly.
• Candidates under 18. Since 23 June 2026, CompTIA requires parental consent through its CompTIA Central account before a minor tests, and the consent is needed for each testing event.
• Retakes. There is no waiting period between a first and second attempt. Before a third or later attempt you must wait at least fourteen calendar days.
• Renewal. CompTIA certifications such as Linux+ are valid for three years, and Linux+ is part of CompTIA's continuing education program, so it can be renewed without resitting the exam.
• Price. CompTIA's Linux+ page showed no exam price when this lesson was checked, and prices vary by country, so this course states none.

CompTIA and Linux+ are trademarks of CompTIA. CompTIA does not endorse or sponsor this course, and no course can honestly promise that you will pass. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Two families of distributions

Linux, strictly, is a kernel: the core program that manages memory, processes and hardware. A distribution wraps the kernel with everything else you need, including the shell, libraries, utilities, an installer, and a package manager that installs and updates software. Objective 1.1 sorts distributions by their package format, and the exam expects you to work on both families.

The families differ in more than packages. Configuration files sit in different places, administrators are put in different groups, and the default security module differs. Those differences turn up in exam questions, so your lab has one machine from each family.

How do you tell which family a system belongs to? Read /etc/os-release. The freedesktop.org specification for that file defines ID as a lowercase name for the operating system and ID_LIKE as a list of the systems it derives from or closely resembles. Ubuntu reports ID_LIKE=debian; the specification's own example for a Red Hat rebuild is ID_LIKE="rhel fedora". Scripts use that field to decide which package manager to call.

Which processor is it? Server architectures

Objective 1.1 lists four processor architectures. A program compiled for one will not run natively on another, so you need to know which one a server uses before you download an installer or a package.

The naming trap is in the last two columns. The same 64-bit PC platform is x86_64 in Red Hat file names and amd64 in Debian's, and 64-bit Arm is aarch64 on one side and arm64 on the other. Support also moves over time. Debian 13, "trixie," is the first Debian release to support riscv64 officially, while 32-bit i386 survives there only as a partial userland for running old software on 64-bit PCs. AlmaLinux 10 is offered for x86_64, aarch64, ppc64le and s390x, the last two being IBM's POWER and mainframe platforms.

Why this matters: run uname -m before you download anything for a server. An aarch64 machine needs aarch64 packages and images, whatever the file happens to be called.

Free, open source, copyleft, proprietary

Objective 1.1 also lists four licensing terms. They overlap, and the overlaps are where questions catch people.

• Free software is defined by the Free Software Foundation through four freedoms: to run the program for any purpose (freedom 0), to study and change it, which needs the source code (freedom 1), to share copies (freedom 2), and to distribute your modified versions (freedom 3). The FSF's own gloss is to think of "free" as in free speech, not as in free beer. Free software can be sold; the definition says it must be available for commercial use and distribution.
• Open source is defined by the Open Source Initiative's ten-point Open Source Definition. Among its criteria, a licence may not stop anyone selling or giving the software away, must include or offer the source code, and must allow modifications and derived works. In practice almost every licence that meets one definition meets the other; the two terms reflect different emphases, freedom versus development method.
• Copyleft is a licensing strategy within free software: you may modify and redistribute, but your distributed versions must carry the same freedoms. The GNU General Public License (GPL) is the classic copyleft licence. The Linux kernel is distributed under GPL version 2 only, with an explicit note about system calls, which is why anyone who ships a modified kernel must offer its source. Permissive licences such as MIT and BSD are open source but not copyleft: someone may fold the code into a closed product.
• Proprietary software keeps its source closed or restricts use, copying or modification under a licence agreement. A vendor's binary-only driver or a commercial backup agent are typical examples on a Linux server.

Price is a separate axis. Red Hat Enterprise Linux is built from open source code but sold by subscription with support. AlmaLinux and Rocky Linux rebuild compatible systems from that source and give them away. Both facts are consistent with the GPL, because the licence governs freedoms, not price.

Building the lab, step by step

Every hands-on task in this course happens in a virtual machine on your own computer. A virtual machine is software pretending to be a computer, so you can break it, roll it back and break it again. Never practise on a production server, a school machine or a work laptop you do not administer.

Step 1 is to choose a free hypervisor, the program that runs virtual machines. VirtualBox's platform packages are released under the GPL version 3, and version 7.2 offers them for Windows, for macOS on both Intel and Apple Silicon, and for Linux. You do not need its separate Extension Pack, which has a different licence. On a Mac you can instead use UTM, which is free and open source when downloaded from its website and is built on QEMU; you will meet QEMU properly in Lesson 12. On a computer that already runs Linux, use KVM with virt-manager from your distribution's repositories.

Step 2 is to download two installer images, neither of which needs an account.

• Ubuntu Server 26.04 LTS for the dpkg family. Canonical's download page describes long-term support releases as getting five years of free security and maintenance updates.
• AlmaLinux 10 for the RPM family, a free rebuild compatible with Red Hat Enterprise Linux. Take the minimal or DVD image, which installs without a network.

Step 3 is to verify each image before you trust it. AlmaLinux's installation guide gives the method: import the project's public key with gpg --import, check the signature on the CHECKSUM file with gpg --verify CHECKSUM, then hash the image with sha256sum (on Windows, certutil -hashfile with the SHA256 option) and compare the result with the line for your file. Ubuntu publishes the same pair of files as SHA256SUMS and SHA256SUMS.gpg. The hash proves the file has not changed; the signature proves the list of hashes came from the project. Lesson 24 explains why both steps are needed.

Step 4 is to create the machines. These settings work for both guests and leave room for later labs.

Step 5 is to install with a text-mode server profile, create an ordinary user for yourself, and say yes when Ubuntu offers to install the OpenSSH server. Step 6 is the first thing you do after the first login: take a snapshot named "clean install." A snapshot lets you return to this moment in seconds after a lab goes wrong.

Then run three commands on each machine and compare the answers.

• uname -m prints the architecture.
• cat /etc/os-release prints the distribution, its version and its family in ID_LIKE.
• ip address lists the network interfaces and the address the hypervisor's NAT gave each one.

Change one input: an Apple Silicon Mac

Now run the same procedure on a Mac with an M-series chip, and watch one decision flip. The host processor is AArch64. UTM's documentation explains that ARM64 guests run under Apple's hypervisor framework at near native speed, while x86 guests run through slower emulation. So on this host you download the aarch64 images instead: AlmaLinux publishes them on its download page, and Ubuntu lists its ARM build under alternative architectures. Everything else in the procedure stays the same, and uname -m inside the guest prints aarch64.

That small change is objective 1.1 in miniature. Architecture decides which binaries can run, the distribution family decides which tools manage them, and the licence decides what you may do with them.

How to study with this course

Two habits have unusually strong evidence behind them. In a 2013 review of ten study techniques, Dunlosky and colleagues rated practice testing and distributed practice, spreading study over time, as high utility, and rated highlighting and rereading as low. Every lesson here ends with ten scenario questions whose explanations say why the right answer is right. Take the quiz the day you finish a lesson, then again a week later before you look at the explanations, and record which objective each missed question belongs to. Let those misses, not the lesson order, decide what you review.

For a Linux exam there is a third habit: type every command yourself. Reading lvextend -r is not the same as watching it grow a filesystem you made.

Common misconceptions

• "720 means I need 72 percent." The 720 is a point on a 100-900 scale, not a percentage of questions answered correctly.
• "If a command is not in the bullets, it will not be tested." CompTIA states that the bulleted examples are not exhaustive and that related content may appear.
• "Free software means software you do not pay for." The FSF definition concerns freedom, and free software may be sold. Freeware that is free of charge but closed is proprietary.
• "Open source and copyleft are the same thing." Copyleft is one kind of open source licence. Permissive licences such as MIT allow closed derivatives; the GPL does not.
• "amd64 packages only run on AMD processors." amd64 is Debian's name for the whole 64-bit x86 platform, Intel processors included.

What to carry forward

• XK0-006 allows at most 90 multiple-choice and performance-based questions in 90 minutes, and you pass at 720 on a 100-900 scale.
• The five domains weigh 23, 20, 18, 17 and 22 percent; troubleshooting is the second largest.
• "Given a scenario" objectives mean doing, not recognizing; Linux+ PBQs are simulations that can be skipped, revisited and partly credited.
• RPM-based distributions use rpm with dnf or zypper; dpkg-based ones use dpkg with apt; /etc/os-release tells you which you are on.
• x86_64 and amd64 name the same platform, as do aarch64 and arm64; check uname -m before downloading.
• Free software is about four freedoms, open source about ten criteria, copyleft about keeping those freedoms in derived works, and proprietary licences restrict them.
• Your lab is two verified virtual machines with a clean snapshot. Practise only there.

Sources

1. CompTIA. (2026). CompTIA Linux+ [Certification page: exam details and domains]. Retrieved October 27, 2026. comptia.org
2. CompTIA. (2024). CompTIA Linux+ XK0-006 V8 certification exam objectives (Document version 5.0). CompTIA (PDF)
3. CompTIA. (n.d.). Performance-based questions explained. comptia.org
4. CompTIA. (n.d.). Candidate ID policy; CompTIA certification retake policy; Certification renewal policy. comptia.org (ID policy); comptia.org (renewal)
5. The Linux Foundation. (n.d.). The Linux mark. linuxfoundation.org
6. Free Software Foundation. (n.d.). What is free software? gnu.org
7. Open Source Initiative. (n.d.). The Open Source Definition. opensource.org
8. The Linux kernel documentation. (n.d.). Linux kernel licensing rules. docs.kernel.org
9. freedesktop.org. (n.d.). os-release: Operating system identification (systemd manual). freedesktop.org
10. Debian. (2025). Release notes for Debian 13 (trixie): What's new in Debian 13. debian.org
11. Oracle. (2026). VirtualBox downloads. virtualbox.org; UTM. (n.d.). UTM: Virtual machines for Mac. getutm.app
12. AlmaLinux OS Foundation. (n.d.). Installation guide and Get AlmaLinux. wiki.almalinux.org; almalinux.org
13. Canonical. (2026). Get Ubuntu Server. ubuntu.com
14. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., and Willingham, D. T. (2013). Improving students' learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58. doi:10.1177/1529100612453266

A machine that reports on its own boot

Log in to the AlmaLinux machine you built in Lesson 1 and type systemd-analyze. It prints one line. The example in its manual reads "Startup finished in 2.584s (kernel) + 19.176s (initrd) + 47.847s (userspace) = 1min 9.608s." On a machine with UEFI firmware the line can begin with two more stages, firmware and loader. Those labels are the outline of this lesson. Each names a program that ran, did one job, and handed control to the next.

systemd's own description of the boot, in its bootup manual page, gives the order. Firmware performs minimal hardware initialization and passes control to a boot loader. The boot loader loads the kernel. The kernel mounts an in-memory filesystem that looks for the real root filesystem. Then the system manager takes over and starts everything else. We will follow one boot of your virtual machine through those stages, stopping wherever the exam has a question waiting.

Stage one: firmware

When you press the power button, the processor starts running code stored on the motherboard. On older PCs that code is the BIOS. On almost everything sold in the last decade it is UEFI, the Unified Extensible Firmware Interface. Both test the hardware and then look for something to boot, in the order set in their boot menu.

They look in different places. A BIOS reads the first sector of the boot disk, the master boot record, and runs the tiny program it finds there. UEFI understands partitions and filesystems. It reads a small FAT-formatted partition called the EFI System Partition, which Linux usually mounts at /boot/efi, and runs a boot program stored there as a file. That difference is why a UEFI machine can list several operating systems in its own firmware menu.

UEFI also brings Secure Boot, listed under objective 3.3. Microsoft's documentation describes it as a standard under which the firmware checks the signature of each piece of boot software before running it. The firmware holds databases of trusted keys and of revoked signatures, and refuses to run anything unsigned or revoked. Linux distributions fit into that chain through a small first-stage loader called shim. Ubuntu's security documentation describes its chain like this:

• The firmware validates shim, which Microsoft has signed.
• Shim carries the distribution's own certificate and uses it to validate GRUB and the kernel.
• The kernel loads only kernel modules that are signed. Ubuntu says an unsigned module fails to load with an error from insmod or modprobe.

What about a driver you compile yourself? Shim also keeps a list of Machine Owner Keys (MOKs), local keys you enroll at boot through a screen called MokManager. You sign your module with your MOK, and the kernel will then accept it. The mokutil tool manages that list, and mokutil --sb-state reports whether Secure Boot is on.

Remember: Secure Boot is a chain of signature checks: firmware checks shim, shim checks GRUB and the kernel, and the kernel checks modules. An unsigned third-party driver breaks the chain until you sign it with an enrolled key.

Stage two: the boot loader and its configuration files

On both of your lab machines the boot loader is GRUB 2. It shows a menu of installed kernels, waits a few seconds, then loads the chosen kernel and its initial RAM disk into memory and jumps into the kernel. You will spend more time with GRUB's configuration than with GRUB itself, and the configuration is built in layers.

The GRUB manual defines the variables you will touch most. GRUB_DEFAULT chooses the default menu entry, and its default is 0, the first. GRUB_TIMEOUT is how many seconds the menu waits, 5 by default. GRUB_CMDLINE_LINUX lists arguments added to every Linux entry. GRUB_CMDLINE_LINUX_DEFAULT adds arguments only to the default entry, so they stay out of recovery entries. That is where a distribution puts cosmetic options such as quiet.

Why the layers? Because the menu has to change every time a kernel is installed or removed. A generator can rebuild the menu safely; a person editing grub.cfg by hand would have the edits overwritten at the next kernel update. On AlmaLinux, Red Hat's documentation recommends grubby for persistent kernel arguments, because it writes them straight into each kernel's entry file.

Stage three: the kernel and its parameters

GRUB passes the kernel a command line, a string of parameters. Read the one your machine booted with in /proc/cmdline. A typical line names the kernel image, then root= with the device holding the root filesystem, then ro, then options such as quiet. The bootparam manual page explains ro: the kernel mounts the root filesystem read-only at first so that a filesystem check can run on a quiet filesystem, and it is remounted read-write later in the boot.

The kernel's own parameter documentation adds three rules worth knowing.

• A parameter with a dot in it is meant for a module, in the form module.parameter=value; the kernel's example is usbcore.blinkenlights=1.
• A parameter the kernel does not recognize, and which has no dot, is passed on to the first process, init. That is how systemd.unit=rescue.target reaches systemd.
• Everything after -- is passed to init untouched.

The first process is chosen by init=. If that is unset, bootparam says the kernel tries /sbin/init, then /etc/init, then /bin/init, then /bin/sh, and panics if all of them fail. On your machines /sbin/init is a link to systemd.

Two ways to change parameters exist, and the exam distinguishes them.

systemd's manual lists short forms you can type at that GRUB prompt: rescue, single or 1 mean systemd.unit=rescue.target, and emergency means systemd.unit=emergency.target. Rescue mode mounts the filesystems and gives a root shell with almost no services; emergency mode mounts almost nothing. You will use both in Lesson 33.

Remember: edits at the GRUB menu last one boot; edits made with grubby, or in /etc/default/grub followed by regenerating grub.cfg, last until you remove them.

Stage four: the initial RAM disk

Here is the puzzle the initial RAM disk solves. The kernel needs a driver to read the disk that holds the root filesystem. Drivers usually live as modules on that same root filesystem. If the root sits on an NVMe drive inside an LVM volume inside an encrypted partition, the kernel needs three layers of software before it can read a single file of the real system.

So the boot loader loads a second file alongside the kernel: a small archive of exactly the drivers and tools needed to find and mount the root. The kernel documentation explains that the modern version, initramfs, is a compressed cpio archive the kernel unpacks into an in-memory filesystem, then runs /init from it. The older initrd was a disk image that needed a filesystem driver of its own. The objectives say initrd, and many tools and file names still do, but what your machines load is an initramfs.

On your lab machines the program inside it is systemd itself. The bootup manual describes it reaching initrd.target, mounting the real root at /sysroot, and then switching root to it, the step labelled "transition to host OS."

You rarely build this file by hand, because installing a kernel package builds one automatically. You must rebuild it yourself after changing anything the early boot depends on, such as moving the root filesystem onto RAID or LVM, adding encryption, or installing a storage driver. Objective 1.2 names the tools.

On the Red Hat family the result is /boot/initramfs-<version>.img; on Debian and Ubuntu it is /boot/initrd.img-<version>. Run ls /boot on both machines and you will see one per installed kernel, next to the kernel files named vmlinuz-<version>.

Stage five: systemd and the default target

After the switch, the real system's systemd runs as process 1 and starts the units pulled in by default.target. The systemd.special manual calls default.target simply an alias, usually of multi-user.target, a full text-mode system, or graphical.target, which adds a graphical login on top. Your server installs default to multi-user; a desktop install defaults to graphical. systemctl get-default shows which, and Lesson 18 shows how to change it.

What happens next decides what you see. With multi-user.target you get a text login prompt on the console and an SSH server waiting on the network. With graphical.target, systemd also starts display-manager.service, which systemd's documentation describes as usually an alias for a service such as gdm.service.

The graphical stack: four pieces with confusing names

Objective 1.1 lists display managers, window managers, the X server and Wayland. They are four different jobs.

• A display manager is the graphical login screen. GDM, SDDM and LightDM are examples. It authenticates you and then starts your desktop session.
• The X server is the display server of the X Window System. It owns the screen, keyboard and mouse, and applications connect to it as clients, which is why a remote program can draw on your screen if it is told where your display is through the DISPLAY variable from Lesson 3.
• A window manager, under X, is a separate client that decides where windows go and draws their borders and title bars.
• Wayland is a protocol, not a program. The Wayland documentation explains that in its design the compositor is the display server: one program draws the screen, manages windows and routes input, while applications render their own contents and hand the finished buffers to it.

So "Is this machine running X or Wayland?" is a question about the display server, and "Which window manager?" has a different answer on each. Under X, the window manager is its own program; under Wayland, window management lives inside the compositor. Most servers run neither, which is one reason they boot to multi-user.target.

Booting from the network: PXE

Now change one thing: the machine has no operating system on its disk at all. A data center installing 200 servers does not walk around with USB sticks. It uses the Preboot Execution Environment (PXE), firmware support for booting from the network. Ubuntu's netboot guide lays out the sequence.

1. The firmware asks the network for an address. A DHCP server answers with the address and the name of a boot file, chosen by the client's architecture, for example a BIOS loader or the UEFI loader bootx64.efi.
2. The firmware downloads that boot loader with TFTP, the Trivial File Transfer Protocol, and runs it. On UEFI it is a signed shim followed by a signed GRUB, so Secure Boot still applies.
3. The boot loader fetches its own configuration, then the kernel and the initial RAM disk, over TFTP.
4. The kernel boots, and the RAM disk fetches the installer image and runs the installation.

Ubuntu's guide uses dnsmasq for the DHCP role. Pair PXE with an unattended answer file, such as a Kickstart file or cloud-init data from Lesson 30, and nobody has to touch the keyboard at all.

Remember: PXE is DHCP to find a boot file, TFTP to fetch it, then the same kernel-plus-RAM-disk boot as from disk.

Where a boot can stop

Knowing the order tells you where to look when a boot fails, which is the heart of objective 5.2 and Lesson 33. A message from the firmware, or a GRUB rescue prompt, means the problem is before the kernel. A kernel panic that says it cannot mount the root filesystem usually means the RAM disk lacks a driver, or root= points at the wrong device. An emergency shell after the kernel has started usually means a filesystem listed in /etc/fstab could not be mounted. Each symptom names its stage.

Common misconceptions

• "To change a boot option, edit grub.cfg." grub.cfg is generated. Edit /etc/default/grub and regenerate it, or use grubby on the Red Hat family; hand edits disappear at the next kernel update.
• "The initrd holds the whole operating system." It holds only what is needed to find and mount the real root filesystem, then hands over.
• "Secure Boot stops all third-party drivers." It stops unsigned ones. A driver signed with an enrolled Machine Owner Key loads normally.
• "Wayland is a newer X server." Wayland is a protocol in which the compositor is the display server. It replaces the X model rather than updating it.
• "PXE needs a special operating system on the client." PXE lives in the network card's firmware; the client needs no disk contents at all.

What to remember

• Order of boot: firmware (BIOS or UEFI), boot loader (GRUB 2), kernel with its command line, initial RAM disk, systemd, default.target.
• Secure Boot: firmware verifies a Microsoft-signed shim, shim verifies GRUB and the kernel, the kernel verifies modules; MOKs let you sign your own.
• Edit /etc/default/grub then run grub-mkconfig or update-grub, or use grubby on the Red Hat family; never edit grub.cfg directly.
• Read the current kernel command line in /proc/cmdline; press e at the GRUB menu to change it for one boot.
• Rebuild the initial RAM disk with dracut (mkinitrd is a wrapper for it) or update-initramfs after changing storage drivers, LVM, RAID or encryption under the root filesystem.
• Display manager: login screen. X server: display server for X clients. Window manager: places windows under X. Wayland: protocol where the compositor is the display server.
• PXE: DHCP supplies an address and boot file name, TFTP delivers the boot loader, kernel and RAM disk.

Sources

1. systemd project. (n.d.). bootup: System bootup process; systemd.special: Special systemd units; systemd: Kernel command line; systemd-analyze. freedesktop.org. bootup; systemd.special; systemd; systemd-analyze
2. The Linux kernel documentation. (n.d.). The kernel's command-line parameters. docs.kernel.org
3. The Linux kernel documentation. (n.d.). Ramfs, rootfs and initramfs. docs.kernel.org
4. Linux man-pages project. (n.d.). bootparam(7): Introduction to boot time parameters of the Linux kernel. man7.org
5. GNU Project. (n.d.). GNU GRUB manual: Simple configuration handling. gnu.org; Debian. (n.d.). update-grub(8). manpages.debian.org
6. Red Hat. (n.d.). Configuring kernel command-line parameters. In Managing, monitoring, and updating the kernel (RHEL 9). docs.redhat.com
7. dracut project. (n.d.). dracut(8) and mkinitrd(8). man7.org (dracut); man7.org (mkinitrd); Debian. (n.d.). update-initramfs(8). manpages.debian.org
8. Microsoft. (2026). Secure boot. Microsoft Learn. learn.microsoft.com
9. Canonical. (n.d.). UEFI Secure Boot. Ubuntu security documentation. documentation.ubuntu.com; Debian. (n.d.). mokutil(1). manpages.debian.org
10. Canonical. (n.d.). How to netboot the server installer on amd64. Ubuntu Server documentation. ubuntu.com
11. Wayland project. (n.d.). Wayland: Introduction. freedesktop.org. wayland.freedesktop.org

Command not found, but the program is right there

You copy a program named backup-check into the directory bin inside your home directory, run ls -l ~/bin, and see it listed with execute permission. Then you type backup-check and Bash replies bash: backup-check: command not found. Type ~/bin/backup-check instead and it runs perfectly.

Nothing is wrong with the program. The two commands differ in one way: the second contains a slash and the first does not. That difference, and what the shell does about it, is the first thing to understand about objective 1.5. The rest of this lesson follows from the fix.

How the shell finds a command

When a command name contains a slash, the shell runs that exact file. When it does not, the shell first checks whether the name is an alias, a function or a builtin such as cd, and then searches the directories listed in the PATH variable, in order, and runs the first matching executable it finds. The environ manual page describes PATH as the sequence of directory prefixes that the shell and many other programs use when searching for an executable file.

Print it with echo $PATH. You will see something like /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin: directories separated by colons. Your ~/bin is not in the list, so the shell never looks there. Two commands show what the shell will actually run:

• type -a backup-check lists every alias, builtin or file the name could mean, in the order the shell would choose.
• which ls prints the path of the file that would run.

The fix is to put the directory on the path: export PATH="$HOME/bin:$PATH". Order matters. Putting your directory first lets your version of a program shadow a system one of the same name; putting it last makes it a fallback.

One tempting shortcut is dangerous. Adding ., the current directory, to PATH means that whichever directory you happen to be in can supply programs. If someone leaves an executable called ls in a shared directory and you cd into it, you run their program. The environ manual notes that an empty PATH entry also means the current directory, and calls that legacy feature deprecated. Run programs in the current directory explicitly, as ./script.sh.

In short: a name with a slash runs that file; a bare name is searched for along PATH, left to right, and the first match wins.

Absolute and relative paths

An absolute path starts at the root directory, /, and means the same thing wherever you are: /etc/ssh/sshd_config. A relative path starts from your current directory, which pwd prints. The shell and filesystem give you a few shorthands.

The tilde is expanded by the shell before the command runs, which is why ~ works in your typing but not inside a quoted string in a configuration file that a program reads directly. In scripts and configuration files, write $HOME or the absolute path.

Variables: shell and environment

A shell variable is set with NAME=value, with no spaces around the equals sign, and read with $NAME. It exists only in the current shell. When you export it, it becomes an environment variable, which every program you start from that shell inherits. The environ manual explains the mechanism: a child process created with fork inherits a copy of its parent's environment, and the copy is passed on when a new program starts. A copy, not a link. A child cannot change its parent's variables, which matters later in this lesson.

Objective 1.5 names six variables.

Three of them need a closer look. SHELL is your login shell, not necessarily the one you are typing into now. The X manual gives DISPLAY's format as hostname:displaynumber.screennumber, so :0 means the first display on this machine. A server reached over SSH usually has no DISPLAY at all, which is why a graphical program started there fails with a message about being unable to open the display.

PS1 is built from escape sequences that Bash's manual lists. \u is the user name, \h the host name, \w the current directory with your home shortened to a tilde, and \$ prints # when the effective user is root and $ otherwise. So the familiar prompt alma@lab1:~/projects$ comes from a PS1 such as \u@\h:\w\$ . The trailing # is your warning that you are root.

To inspect variables, env or printenv lists the environment, echo $HOME prints one value, and set with no arguments lists shell variables and functions too. Remove a variable with unset NAME. Lesson 26 returns to variables inside scripts.

Why your setting vanished: startup files

Suppose you fixed PATH at the prompt, logged out, and logged back in. The fix is gone, because a variable lives only as long as its shell. To make it permanent, put the line in a file the shell reads at start-up. Which file depends on what kind of shell starts.

• A login shell is the first shell of a session: a console login or an SSH login. The Bash manual says it reads /etc/profile, then the first readable file among ~/.bash_profile, ~/.bash_login and ~/.profile, in that order.
• An interactive non-login shell is, for example, a new terminal window inside a desktop session, or bash typed at a prompt. It reads ~/.bashrc.

That split explains the classic puzzle: an alias that works in your desktop terminal but not over SSH, or the reverse. The manual's own fix is to have the login file read the other one, with this line in ~/.bash_profile: if [ -f ~/.bashrc ]; then . ~/.bashrc; fi. AlmaLinux's default .bash_profile does this, and Ubuntu's default .profile does the same for .bashrc. So the practical rule is: environment variables such as PATH go in the profile file, aliases and prompt settings go in .bashrc, and the profile pulls .bashrc in. System-wide settings for every user go under /etc, in /etc/profile or a file in /etc/profile.d, which Lesson 14 meets again with new accounts.

After editing a startup file, you do not need to log out. Run source ~/.bashrc, or its short form . ~/.bashrc. The Bash manual describes the dot command as reading and executing commands from the file in the current shell environment. Contrast bash ~/.bashrc, which runs the file in a new child shell. The child sets everything up and exits, and, because a child cannot change its parent, your shell is untouched.

In short: login shells read the profile files, interactive non-login shells read .bashrc, and source applies a file to the shell you are in.

History and the exclamation mark

Bash keeps a numbered list of the commands you have typed. history prints it, history 20 prints the last twenty, and the list is saved to the file named in HISTFILE, by default ~/.bash_history, when the shell exits. HISTSIZE caps how many commands are kept in memory; the manual gives Bash's own default as 500, and many distributions raise it in their startup files.

History expansion, introduced by !, reuses entries without retyping them. The manual's event designators are worth memorizing, because they appear in exam questions.

Expansion happens before the command runs, and Bash prints the expanded command first, so you can see what !sys really ran. Ctrl+R searches history interactively, which is safer when you are not sure what a designator will match.

Aliases, and knowing your system with uname

An alias substitutes text for the first word of a command. alias ll='ls -l' makes ll list in long format, alias alone lists the current aliases, and unalias ll removes one. Distributions often define a few: on AlmaLinux, root's .bashrc aliases rm to rm -i, which asks before every deletion. To bypass an alias once, type \rm or command rm.

There is one trap with aliases. Bash's manual states that aliases are not expanded when the shell is not interactive, unless the expand_aliases option is set. So an alias that works at your prompt does nothing inside a script. Scripts should use full commands or functions, which Lesson 27 covers.

uname prints facts about the running kernel and machine. Its options are short and worth knowing by letter.

• uname -s: the kernel name, Linux; this is the default with no option.
• uname -r: the kernel release, such as a version string ending in .el10 or -generic; you will match it against module directories in Lesson 6.
• uname -m: the machine hardware name, x86_64 or aarch64, from Lesson 1.
• uname -n: the network node host name.
• uname -a: everything at once.

Editing files: vi and nano

Configuration in Linux is text, so you will edit files constantly, often over SSH on a server with no graphical editor. Objective 1.5 names two editors. Nano is the gentle one; vi is the one guaranteed to be there, because POSIX specifies it, and on your machines vi starts Vim, an extended version.

vi is modal, and that is the only hard part. The POSIX specification describes it starting in command mode, entering text input mode through commands such as insert, and returning to command mode with Escape. The keys you need first:

If you remember one thing, remember the escape hatch: press Esc, type :q!, press Enter, and you are out with nothing changed. Vim's manual recommends its built-in thirty-minute tutorial, which you start by typing vimtutor.

nano shows its main shortcuts along the bottom of the screen, where the caret means Ctrl. From nano's own cheat sheet: Ctrl+S saves, Ctrl+O writes the file out under a name you confirm, Ctrl+X exits, Ctrl+K cuts the current line, Ctrl+U pastes, Alt+U undoes, Ctrl+F searches forward and Alt+G goes to a line number. When a configuration error message names a line number, Alt+G takes you straight there.

Which to use? On the exam, either may appear in a scenario. In practice, learn enough vi to edit and exit, because it is on every system, including rescue environments where nano may be missing.

In short: in vi, Esc returns to command mode and :wq or :q! leaves; in nano, the bottom two lines tell you the keys.

Common misconceptions

• "Running bash ~/.bashrc reloads my settings." It runs the file in a child shell that then exits. Use source ~/.bashrc or . ~/.bashrc to apply it to the current shell.
• "The shell searches PATH for every command." Only bare names are searched for. A command containing a slash, such as ./run.sh or /usr/bin/ls, runs that exact file.
• "Adding . to PATH is a harmless convenience." It lets any directory you enter supply programs, including planted ones with familiar names.
• "My aliases work inside my scripts." Aliases are not expanded in non-interactive shells unless expand_aliases is set.
• "Setting a variable makes it visible to the programs I run." Only exported variables are passed to child processes.

The short version

• Bare command names are found by searching PATH left to right; check with type -a and fix with export PATH="$HOME/bin:$PATH".
• Absolute paths start at /; relative paths start at the current directory; ~ is HOME, .. the parent, cd - the previous directory.
• Export a variable to pass it to child programs; children get a copy and cannot change the parent's.
• Login shells read /etc/profile and the first of ~/.bash_profile, ~/.bash_login, ~/.profile; interactive non-login shells read ~/.bashrc; source applies a file now.
• !!, !n, !string and ^old^new^ reuse history; aliases shorten typing but do not work in scripts.
• uname -r gives the kernel release and uname -m the architecture.
• vi: i to insert, Esc to stop, :wq to save and quit, :q! to abandon. nano: Ctrl+S save, Ctrl+X exit.

Sources

1. Free Software Foundation. (n.d.). Bash reference manual: Bash startup files. gnu.org
2. Fox, B., and Ramey, C. (n.d.). bash(1) [Sections on prompting, aliases, history expansion and shell builtins]. Linux man-pages project. man7.org
3. Linux man-pages project. (n.d.). environ(7): User environment. man7.org
4. X.Org Foundation. (n.d.). X(7): A portable, network-transparent window system [Display names]. xorg.freedesktop.org
5. MacKenzie, D. (n.d.). uname(1). GNU coreutils, Linux man-pages project. man7.org
6. The Open Group. (2024). vi: screen-oriented (visual) display editor. The Open Group Base Specifications Issue 8 (POSIX.1-2024). pubs.opengroup.org
7. Moolenaar, B., and the Vim project. (n.d.). Vim user manual, chapter 1: About the manuals [Using the Vim tutor]. vimhelp.org
8. The nano project. (n.d.). Overview of nano's shortcuts. nano-editor.org

A line that arrives thousands of times a day

Every time someone guesses a password on an SSH server, OpenSSH writes a line like this to the authentication log: sshd[2210]: Failed password for invalid user admin from 198.51.100.23 port 41870 ssh2. A server reachable from the internet can collect thousands of them a day. Your manager asks a simple question: which three addresses are trying hardest?

No single Linux command answers that. Six small ones, joined together, answer it in one line. That is the design idea behind the second half of objective 1.5: programs that each do one job on a stream of text, and a shell that connects them. This lesson builds that answer step by step, then changes one input and watches the answer break.

Three streams and a pipe

Every process starts with three open channels, numbered by their file descriptors. Standard input (0) is where it reads, normally the keyboard. Standard output (1) is where its results go, normally the terminal. Standard error (2) is a separate channel for error messages, also the terminal by default. Keeping errors separate is what lets you save results to a file while still seeing what went wrong.

Redirection reconnects those channels before the command starts.

Two details catch people. First, /dev/null is a device that discards everything written to it, so 2> /dev/null silences error messages. Second, order matters. The Bash manual's own example: ls > dirlist 2>&1 sends both streams to the file, but ls 2>&1 > dirlist sends only standard output there. In the second form, standard error was pointed at the terminal, where standard output was at that moment, before standard output moved to the file. The manual also notes that &> word means the same as > word 2>&1.

A pipe connects only standard output. Errors still go to the terminal unless you redirect them too, and that is usually what you want, because an error mixed into the data stream gets counted as data.

The upshot: redirection points a command's channels at files; a pipe points one command's output at the next command's input; and the order of redirections is read left to right.

Step 1: make a test file with a here document

Practise on a copy, never on the live log. A here document builds the copy without an editor. Type this on your Ubuntu machine:

• cat > auth.sample <<'EOF'
• Oct 27 09:14:02 lab1 sshd[2210]: Failed password for root from 203.0.113.7 port 50122 ssh2
• Oct 27 09:14:09 lab1 sshd[2214]: Failed password for root from 203.0.113.7 port 50131 ssh2
• Oct 27 09:15:40 lab1 sshd[2230]: Accepted publickey for alma from 192.0.2.10 port 51002 ssh2
• Oct 27 09:16:12 lab1 sshd[2241]: Failed password for root from 198.51.100.23 port 41866 ssh2
• EOF

The shell feeds every line up to the one containing only EOF into cat, which writes them to the file. The quotes around 'EOF' matter. The Bash manual explains that when any part of the delimiter word is quoted, the lines are not expanded; unquoted, a $ in the text would be treated as a variable. Add more lines of your own so that one address appears five times, another three times and a third twice. The addresses used here come from blocks reserved for documentation, so they belong to nobody.

A here string is the one-line version. The manual describes <<< word as supplying the expanded word, with a newline appended, on standard input. So tr 'a-z' 'A-Z' <<< "$USER" prints your user name in capitals without creating a file.

Step 2: keep only the lines you want with grep

grep 'Failed password' auth.sample prints every line containing that text. grep is the filter you will use most, so learn its common options as a set.

A useful habit: grep -v '^#' /etc/ssh/sshd_config | grep -v '^$' strips comment lines and blank lines, leaving only the settings that are active. The caret anchors a pattern to the start of a line and the dollar sign to the end; Lesson 27 covers regular expressions properly.

Step 3: pull out the field with awk

Each matching line holds the address somewhere in the middle. awk splits every line into whitespace-separated fields named $1, $2 and so on, and the gawk manual notes that it sets NF to the number of fields in the current line. Count the fields in a "Failed password for root" line and the address is field 11, so:

grep 'Failed password' auth.sample | awk '{print $11}'

prints one address per line. awk is a small programming language, not just a field cutter. awk -F: '{print $1}' /etc/passwd sets the field separator to a colon and prints every account name, and awk '{total += $5} END {print total}' sizes.txt adds up a column and prints the sum after the last line.

Why not cut? cut -d: -f1 /etc/passwd also prints account names, and cut is ideal for files with a single, consistent delimiter. But cut treats every single space as a separator, so two spaces in a row create an empty field and shift every count after it. Log files often pad short day numbers with an extra space, so awk, which treats any run of spaces as one separator, is the safer choice here. If you must use cut on such text, squeeze the spaces first with tr -s ' '.

Step 4: count and rank with sort, uniq and head

Now count how often each address appears. The tool for counting is uniq -c, but its manual carries a warning in plain words: uniq does not detect repeated lines unless they are adjacent, so sort the input first. Then sort the counts numerically, largest first, and keep the top three:

grep 'Failed password' auth.sample | awk '{print $11}' | sort | uniq -c | sort -rn | head -3

Read the pipeline aloud and each stage is one plain sentence: find the failures, take the address, group identical addresses, count each group, rank by count, keep three. The second sort needs -n because without it sort compares text character by character, and the text "10" sorts before "9". -r reverses the order. Other sort options you will meet: -k 2 sorts on the second field, -t ',' sets the field separator, and -u removes duplicates as it sorts.

Step 5: keep a copy and still see it, with tee

Append | tee top-attackers.txt to the pipeline. The tee manual describes it exactly: copy standard input to each file, and also to standard output. You see the result and keep a file, and tee -a appends instead of overwriting.

tee also solves a classic permission puzzle. sudo echo 'nameserver 192.0.2.53' > /etc/resolv.conf fails with "Permission denied", because the redirection is performed by your own unprivileged shell before sudo ever runs. echo 'nameserver 192.0.2.53' | sudo tee -a /etc/resolv.conf works, because the program opening the file, tee, is the one running as root.

The upshot: grep selects lines, awk selects fields, sort groups, uniq -c counts adjacent duplicates, sort -rn ranks, head trims, and tee saves a copy along the way.

Change one input: the invalid-user lines

Add this line to your sample twice and run the pipeline again:

Oct 27 09:20:31 lab1 sshd[2290]: Failed password for invalid user admin from 198.51.100.23 port 41870 ssh2

The ranking is now wrong, and a strange "address" appears: the word admin. When the target account does not exist, OpenSSH inserts "invalid user" before the name, which adds two fields, so the address moves from field 11 to field 13. Your pipeline counted user names as addresses.

The fix is to count from the end of the line, where the structure is stable. In both kinds of line the address is three fields before the last, so use awk '{print $(NF-3)}'. Counting from the end also survives a different timestamp format, and timestamp formats do differ between distributions and versions. Another fix is to match the address itself: grep -oE 'from [0-9.]+' auth.sample | awk '{print $2}'.

The general lesson is worth more than the command. Before you trust a field number, check it against every kind of line in the input, and prefer positions that do not move.

The rest of the bench

Objective 1.5 lists more tools than one pipeline needs. Each has a clear job.

Three of these deserve a sentence more. sed -i edits a file in place, and the man page notes that a suffix after it, as in -i.bak, makes a backup first; always use the suffix on a configuration file. bc's manual explains that scale sets the digits after the decimal point and that this version truncates the results of division, which is why 2/3 prints .66 rather than .67. And xargs exists because many commands, such as rm or ls, take file names as arguments rather than reading them from standard input. File names can contain spaces, which xargs would split, so pair find -print0 with xargs -0; the xargs manual says -0 makes items end at a null character and treats every other character literally. With -I {}, xargs runs the command once per item and puts the item where the braces are.

Common misconceptions

• "uniq removes all duplicate lines." It only merges adjacent duplicates. Sort first, or use sort -u.
• "cmd 2>&1 > file captures everything." Redirections are processed left to right, so this sends errors to the terminal. Write cmd > file 2>&1 or cmd &> file.
• "A pipe carries error messages to the next command." A pipe carries standard output only; standard error still reaches the terminal unless redirected.
• "sudo echo text > /etc/file should work." The redirection is opened by your own shell before sudo runs. Use echo text | sudo tee /etc/file.
• "sort puts numbers in order." Without -n it sorts as text, so 10 comes before 9.

Where this leaves us

• Descriptors: 0 standard input, 1 standard output, 2 standard error. > replaces, >> appends, 2> redirects errors, and > file 2>&1 or &> file captures both.
• << WORD feeds the following lines as input, unexpanded if WORD is quoted; <<< string feeds one string.
• The ranking pattern is grep | awk | sort | uniq -c | sort -rn | head; tee saves a copy mid-stream.
• awk handles runs of spaces and counts fields with NF; cut needs one consistent delimiter.
• Check field positions against every kind of input line; count from the end when the start varies.
• tr changes characters, sed edits streams (use -i.bak for in-place edits), bc calculates, and xargs turns input into arguments, with -0 for names containing spaces.

Sources

1. Fox, B., and Ramey, C. (n.d.). bash(1) [Section: Redirection, including here documents and here strings]. Linux man-pages project. man7.org
2. GNU coreutils. (n.d.). uniq(1), tee(1) and tr(1). Linux man-pages project. man7.org (uniq); man7.org (tee); man7.org (tr)
3. GNU findutils. (n.d.). xargs(1). Linux man-pages project. man7.org
4. GNU sed. (n.d.). sed(1): Stream editor for filtering and transforming text. Linux man-pages project. man7.org
5. Robbins, A. (n.d.). gawk(1): Pattern scanning and processing language. Linux man-pages project. man7.org
6. GNU grep. (n.d.). grep(1). Linux man-pages project. man7.org
7. Nelson, P. A. (n.d.). bc(1): An arbitrary precision calculator language. Debian manual pages. manpages.debian.org
8. Cotton, M., and Vegoda, L. (2010). IPv4 address blocks reserved for documentation (RFC 5737). RFC Editor. rfc-editor.org

A standard older than most of its readers

On 19 March 2015 the Linux Foundation published version 3.0 of the Filesystem Hierarchy Standard (FHS), a document whose earliest copyright line dates from 1994. Its abstract says it exists so that applications, administration tools and scripts can predict where files are. That predictability is why you can sit down at a Linux server you have never seen and know where things are before typing anything. Its configuration is under /etc and its logs are under /var/log.

Objective 1.1 asks you to know the main directories, and objective 2.1 asks you to manage files inside them. This lesson is built around comparisons. Most questions in this area turn on the difference between two similar things: two directories, two kinds of link, two ways of finding a file, two kinds of device.

The directories the exam names

Three pairs in that table are easily confused. The first is /tmp and /var/tmp. The FHS describes /var/tmp as temporary files preserved between reboots, while nothing in /tmp may be relied on. The second is /usr and /var. The /usr tree should not change while the system runs, except when software is installed, while /var changes constantly; that is why a full disk is so often a full /var. The third is /bin and /usr/bin. Run ls -l / on either lab machine and you will see that /bin, /sbin and /lib are symbolic links into /usr. Modern distributions have merged them, so /bin/ls and /usr/bin/ls are the same file. The FHS layout still describes where to look.

The /proc directory is different in kind. Nothing in it is stored on disk; the kernel generates each file when you read it. Reading /proc/meminfo with cat reports memory as of that instant. Every running process also has a directory there, named by its process ID, and Lesson 16 uses those directories to inspect processes.

So what?: configuration is in /etc, changing data in /var, installed software in /usr, boot files in /boot, and live kernel information in /proc. When a question asks where something belongs, ask whether it is configuration, software, changing data or kernel state.

Moving, making and removing

The everyday commands are simple, and their options are where the exam looks.

There is no recycle bin at the command line. rm removes the name at once. Before any recursive delete, run the same path through ls first and read what it lists.

Two kinds of link

To understand links you need one idea from Lesson 7 early: an inode. A file's data and its details (owner, permissions, size, timestamps, where the data blocks are) live in an inode, identified by a number. A file name is just a directory entry that points to an inode. ls -i shows the numbers.

A hard link is a second name for the same inode. A symbolic link, or symlink, is a small separate file whose content is a path to another name. The ln manual states the default: ln creates hard links, and ln -s creates symbolic ones.

The symlink manual's own term for a link to a path that does not exist is a dangling link. Symlinks are everywhere in administration: /bin pointing into /usr, the active version of a program, a systemd unit enabled by a link in a target directory. Hard links are rarer and mostly used by backup tools that store unchanged files once.

So what?: a hard link is the same file under a second name on the same filesystem; a symbolic link is a pointer that can go anywhere and can break.

What is this file? stat and file

stat file prints the inode's details: size, blocks, inode number, link count, owner, permissions and three timestamps. Each timestamp means something different. The access time records the last read. The modify time records the last change to the contents. The change time records the last change to the inode itself, such as a permission change. A chmod updates the change time but not the modify time, which is how investigators notice permission tampering.

file name reports what a file actually contains, not what its name claims. Its manual describes three sets of tests in order: filesystem tests, magic tests that recognize fixed formats such as an executable's header, and language tests. A file called report.pdf that file calls "ELF 64-bit LSB executable" is a program, whatever its extension says.

Finding files: find versus locate

The plocate manual describes it as searching an index made by updatedb. So when locate cannot find a file you created a minute ago, run sudo updatedb and try again. The find command needs no index, and its tests combine freely:

• find /etc -name '*.conf' by name; -iname ignores case.
• find /var/log -type f -size +100M: regular files over 100 MiB.
• find /home -mtime -7: modified less than seven days ago; +30 means more than thirty.
• find / -user dana -type f 2>/dev/null: files owned by one user, with permission errors discarded.
• find /tmp -name '*.old' -exec ls -l {} +: run a command on the results. The find manual explains that the + form appends many names to one command line, so the command runs far fewer times than with \;, which runs it once per file.

Comparing files: diff and sdiff

Before you restart a service after editing its configuration, compare the file with its backup. The command diff -u sshd_config.bak sshd_config prints a unified diff. Lines starting with - were removed, lines starting with + were added, and a few unchanged lines around each change give context. This is also the format Git uses in Lesson 29.

The sdiff command shows the two files side by side. It marks differing lines with |, lines only on the left with < and lines only on the right with >. An exit status of 0 from diff means the files match, so scripts can test it.

Who has it open? lsof

lsof lists open files and the processes that hold them. Its manual notes that an open file may be a regular file, a directory, a device, a network socket and more, so the same tool answers several questions:

• lsof /var/log/syslog: which processes have this file open?
• lsof +D /mnt/data: what is open anywhere under this directory? This answers "target is busy" when you try to unmount it.
• lsof -i :22: which process is using port 22?
• lsof +L1: open files that have been unlinked, that is, deleted while still open.

That last one solves a real puzzle. Delete a 20 GB log file while a service still writes to it, and df still shows the disk full. The name is gone, but the inode stays in use until the process closes it. lsof +L1 names the process; restarting it frees the space. Lesson 33 returns to this.

Device files: block, character and special

Linux presents hardware as files in /dev, so programs can read a disk or a terminal with the same calls they use for ordinary files. A long listing of /dev shows the type as the first character of each line. Where a file size would normally be, it shows two numbers, the major and minor device numbers. The major number selects the driver; the minor number selects the device within it.

The kernel's list of device numbers puts the special ones under major number 1:

• minor 3, /dev/null, the null device;
• minor 5, /dev/zero, a source of null bytes;
• minor 7, /dev/full, which returns "no space left" on every write;
• minors 8 and 9, /dev/random and /dev/urandom, the random number generators.

You have already used /dev/null to discard errors, and you will use /dev/zero and /dev/urandom to create test files and to wipe disks in Lessons 10 and 25.

Common misconceptions

• "Deleting a file frees its space immediately." Space is freed when the inode's last link is gone and no process holds it open; lsof +L1 finds deleted files still in use.
• "A hard link is a shortcut to a file." That describes a symbolic link. A hard link is the same file under another name, equal in every way to the original.
• "locate searches the disk." It searches an index built by updatedb, so new files are invisible to it until the index is rebuilt.
• "A file's extension tells you its type." Linux ignores extensions; file reads the contents.
• "/proc is a directory of files on disk." It is a virtual filesystem that the kernel generates as you read it.

Putting it together

• /etc holds configuration, /var changing data and logs, /usr installed software, /boot boot files, /dev devices, /proc live kernel information, /tmp disposable files, /home users' files.
• cp -a keeps metadata, mkdir -p creates parents, rmdir removes only empty directories, and rm has no undo.
• A hard link shares an inode and cannot cross filesystems or point to a directory; a symbolic link holds a path, can go anywhere, and can dangle.
• stat shows inode details and the access, modify and change times; file identifies content by its format.
• find searches live with rich tests and can act on results; locate is fast but only as fresh as its last updatedb.
• diff -u and sdiff compare files; lsof shows who holds a file, a directory or a port open.
• Block devices (b) are storage; character devices (c) are byte streams; /dev/null, /dev/zero and /dev/urandom are kernel-provided special character devices.

Sources

1. LSB Workgroup, The Linux Foundation. (2015). Filesystem Hierarchy Standard (Version 3.0). refspecs.linuxfoundation.org
2. Linux man-pages project. (n.d.). link(2) and symlink(7). man7.org (link); man7.org (symlink)
3. GNU coreutils. (n.d.). ln(1) and stat(1). Linux man-pages project. man7.org (ln); man7.org (stat)
4. GNU findutils. (n.d.). find(1). Linux man-pages project. man7.org
5. Gunderson, S. H. (n.d.). plocate(1). Debian manual pages. manpages.debian.org
6. Darwin, I. F., and Zoulas, C. (n.d.). file(1). Linux man-pages project. man7.org
7. GNU diffutils. (n.d.). diff(1) and sdiff(1). Linux man-pages project. man7.org (diff); man7.org (sdiff)
8. lsof project. (n.d.). lsof(8): List open files. Linux man-pages project. man7.org
9. The Linux kernel documentation. (n.d.). Linux allocated devices. docs.kernel.org

A network adapter that is plugged in and invisible

A technician plugs a USB Ethernet adapter into the AlmaLinux lab server to give it a second network. The adapter's light comes on. But ip link lists the same interfaces as before, and nothing new appears. The technician's conclusion comes quickly: "Linux has no driver for this adapter. I'll find one on the vendor's site and compile it."

That conclusion is plausible, common, and in this case wrong. Downloading and compiling a driver is slow, fragile across kernel updates, and, with Secure Boot on, needs signing as Lesson 2 explained. This lesson traces the problem the methodical way, one layer at a time, and finds where the reasoning failed. The tools you use along the way are most of objective 1.2.

Layer 1: does the bus see the device?

Every device reaches the kernel through a bus, usually PCI Express inside the machine or USB outside it. Two commands list what each bus has found. The lsusb manual describes it as displaying the USB buses and the devices connected to them, using udev's hardware database to turn vendor and product IDs into readable names. lspci does the same for PCI.

The technician runs lsusb and sees a line with the adapter's name and an ID pair such as 0bda:8153, a vendor number then a product number. So the hardware is present and talking to the USB controller. The first part of the conclusion, "Linux cannot see it," is already false.

Next, the kernel's own account. dmesg, whose manual says it examines or controls the kernel ring buffer, prints the messages the kernel has logged since boot. Run sudo dmesg | tail -20 right after plugging something in and you will see the kernel announce a new USB device. What you look for next is a line from a driver claiming it. Here there is none. The device was found, and nothing took charge of it.

Worth holding on to: "detected" and "working" are different states. lsusb, lspci and dmesg tell you whether the hardware was detected; the question of a driver comes next.

Layer 2: does a driver exist?

On Linux most drivers are kernel modules: files ending in .ko, often compressed, stored under /lib/modules/ followed by the kernel release that uname -r prints. The kernel loads them when a matching device appears. Each module carries a list of aliases describing the devices it supports, such as USB vendor and product IDs, and that is how the right module is chosen automatically.

The modinfo manual says it extracts information from a module, searching the same directory modprobe uses when given a bare name. Suppose the adapter belongs to the USB networking family served by the cdc_ether module. modinfo cdc_ether prints its file name, licence, description, its alias lines, and a depends: line naming the modules it needs, here usbnet. The driver exists and is installed. The second half of the technician's conclusion is false too.

Layer 3: is it loaded, and if not, why not?

lsmod, in its manual's words, nicely formats the contents of /proc/modules, showing which modules are loaded, their size, and which other modules use them. lsmod | grep cdc_ether prints nothing. So the driver exists but was not loaded when the device appeared.

Automatic loading can be switched off by configuration in /etc/modprobe.d/. A search finds the cause: grep -r cdc_ether /etc/modprobe.d/ returns blacklist cdc_ether, left in a file by someone months ago. The modprobe.d manual explains the keyword: blacklist means that all of the module's internal aliases are ignored. The kernel saw the device's IDs, looked for a module whose alias matched, and the only match had been told to ignore its aliases.

The same directory holds the other two directives worth knowing. An options line adds parameters every time a module loads, for example options bonding max_bonds=2. An install line, the manual says, tells modprobe to run a command instead of inserting the module; install usb-storage /bin/false is a common way to stop a module being loaded even by hand, which blacklist alone does not do.

Layer 4: loading it, and the second wrong turn

The technician now tries to load the module by its file: sudo insmod /lib/modules/$(uname -r)/kernel/drivers/net/usb/cdc_ether.ko.xz. It fails, and dmesg shows unknown symbol errors. Is the driver broken after all?

No. The insmod manual calls it a trivial program and says most users will want modprobe, which can handle module dependencies. cdc_ether needs usbnet, and insmod does not load dependencies. sudo modprobe cdc_ether reads the dependency list, loads usbnet first, then cdc_ether. A new interface appears in ip link at once.

Where do modprobe's dependency lists come from? The modprobe manual says it expects an up-to-date modules.dep.bin generated by depmod. The depmod manual explains that it reads every module, works out which symbols each one exports and which it needs, and writes the result. Package managers run depmod when they install a kernel. You run it yourself after copying a module file in by hand; otherwise modprobe will not know the new module exists.

The final fixes are configuration. Delete the stale blacklist line so the module loads automatically next time. If you ever need a module loaded at every boot without any device to trigger it, list its name in a file under /etc/modules-load.d/, though systemd's manual notes that relying on automatic loading by device IDs is usually the better idea.

Worth holding on to: check the bus, then the kernel log, then whether a module exists, then whether it is loaded, then what configuration stopped it. Use modprobe, not insmod, unless you have a reason.

Taking inventory: the hardware tools

Not every hardware question is a fault. Asset records, support calls and capacity planning all start with "what exactly is in this machine?" Objective 1.2 lists a tool for each part of the answer.

Two of these work below the operating system. Many servers have a baseboard management controller, a small separate computer on the motherboard that stays reachable over the network even when the main system is hung or powered off. The ipmitool manual describes managing IPMI functions locally through a kernel driver or remotely over IPMI v1.5 and v2.0, including sensor readings and remote chassis power control. A virtual machine has no such controller, so in your lab ipmitool has nothing to talk to. Treat its commands as reading knowledge, and treat access to a real management controller as access to the whole server.

Custom hardware, embedded systems and GPUs

Objective 1.2 ends with hardware that does not fit the server-rack picture.

Embedded systems run Linux inside routers, industrial controllers, cameras and single-board computers. They usually have little memory and storage, a processor such as AArch64 or RISC-V from Lesson 1, and devices that no bus can discover by itself. On many such boards the kernel learns what hardware exists from a device tree, which the kernel documentation describes as a data structure and language for describing hardware, so that the operating system need not hard-code the details of each machine.

Custom hardware, such as a data-acquisition card or a vendor's appliance, may need a driver that is not in the mainline kernel. Such out-of-tree modules must be rebuilt for every new kernel, which is the job of tools such as DKMS, and signed for Secure Boot. Check modinfo before assuming you need one.

GPUs have moved from drawing screens to computation. Typical server use cases include training and running machine-learning models, video transcoding, scientific simulation and remote graphical desktops. A GPU workload needs the right kernel driver loaded, which you check with lspci -k, and monitoring. That is what nvtop is for: its project describes it as a task monitor like htop for GPUs and accelerators, able to show several GPUs at once, with support for AMD, Intel and NVIDIA devices among others. You will meet htop itself in Lesson 16.

When the driver is needed at boot

One more layer ties back to Lesson 2. If the device holds the root filesystem, such as a new RAID controller or NVMe adapter, its module must be inside the initial RAM disk, because the kernel needs it before the real root is mounted. Loading it with modprobe on the running system is not enough. Rebuild the image with dracut --force on AlmaLinux, or update-initramfs -u on Ubuntu, and check that the module is listed with lsinitrd.

Common misconceptions

• "If a device does not work, Linux has no driver for it." Check lsusb or lspci, dmesg, modinfo and lsmod first; the driver is often installed but not loaded.
• "insmod and modprobe do the same job." insmod loads one file and ignores dependencies; modprobe resolves them using depmod's lists and applies /etc/modprobe.d settings.
• "A blacklisted module can never load." blacklist stops loading by alias; an explicit modprobe still works. Use an install line pointing to /bin/false to block it completely.
• "dmidecode tests the hardware." It only decodes the firmware's description table. For live temperatures and voltages, use sensors or ipmitool.
• "A module loaded with modprobe is there after the next reboot." Loading is not persistent. Automatic loading by device ID, or a file in /etc/modules-load.d, makes it so.

Pulling it together

• Trace a dead device by layer: bus (lsusb, lspci), kernel log (dmesg), driver present (modinfo), driver loaded (lsmod), configuration (/etc/modprobe.d).
• Modules live under /lib/modules/ and the kernel release; modprobe loads by name with dependencies, modprobe -r unloads, insmod and rmmod act on one module only.
• depmod builds the dependency lists modprobe reads; run it after adding a module by hand.
• In /etc/modprobe.d, blacklist ignores a module's aliases, options sets parameters, and install substitutes a command; /etc/modules-load.d lists modules to load at boot.
• lscpu, lsmem, lspci -k, lsusb, dmidecode and lshw inventory the machine; sensors and ipmitool read its health, and ipmitool reaches the management controller.
• Embedded boards often describe their hardware with a device tree; GPUs need loaded drivers and can be watched with nvtop; boot-critical drivers belong in the initial RAM disk.

Sources

1. kmod project. (n.d.). modprobe(8), insmod(8), rmmod(8), lsmod(8), modinfo(8), depmod(8) and modprobe.d(5). Linux man-pages project. man7.org (modprobe); man7.org (insmod); man7.org (depmod); man7.org (modprobe.d)
2. systemd project. (n.d.). modules-load.d: Configure kernel modules to load at boot. freedesktop.org. freedesktop.org
3. util-linux project. (n.d.). dmesg(1), lscpu(1) and lsmem(1). Linux man-pages project. man7.org (dmesg); man7.org (lscpu); man7.org (lsmem)
4. Mares, M., and others. (n.d.). lspci(8); usbutils project. (n.d.). lsusb(8). Linux man-pages project. man7.org (lspci); man7.org (lsusb)
5. Debian. (n.d.). dmidecode(8), lshw(1), ipmitool(1) and sensors(1). Debian manual pages. dmidecode; lshw; ipmitool; sensors
6. The Linux kernel documentation. (n.d.). Linux and the Devicetree. docs.kernel.org
7. Syllo, M., and contributors. (n.d.). NVTOP: GPU and accelerator process monitoring. GitHub. github.com

Two gigabytes of nothing

Shut down your AlmaLinux machine, add a new 2 GB virtual disk in the hypervisor's settings, and boot it again. The kernel finds the disk and names it, probably /dev/sdb, or /dev/vdb if the virtual disk uses the VirtIO driver you will meet in Lesson 12. Right now it is two gigabytes of zeros. It has no partition table, no filesystem and nowhere to put a file.

This lesson follows that disk from blank to useful, and then onward through the things that happen to disks in their working lives: they fill up, they run out of inodes while showing free space, they need to grow, and occasionally they are damaged. Each stage is a part of objective 1.3.

First look: lsblk and blkid

Before you touch a disk, find out exactly which one it is. Writing a partition table to the wrong disk is one of the few mistakes in this course that a snapshot cannot help you with on real hardware. lsblk prints every block device as a tree: disks, the partitions inside them, and, later, logical volumes inside those. Its -f option adds the filesystem type, label, UUID and mount point of each.

On your machine, lsblk shows the system disk with its partitions and mount points, and the new disk standing alone, 2G, with no children. That shape, a disk with no partitions and no mount point, is how you confirm you have the right target.

blkid answers a narrower question. Its manual says it determines what kind of content a block device holds, such as a filesystem or swap, and its attributes as NAME=value pairs. Run sudo blkid and each formatted partition shows a UUID and a TYPE. The new disk shows nothing yet, because there is nothing on it to describe.

Choosing a partition table

A partition table divides a disk into regions. The fdisk manual describes the two you need to know.

The protective MBR is a small kindness built into GPT. It makes an old MBR-only tool see the whole disk as one occupied partition, so it will not decide the disk is empty and overwrite it.

Cutting the partition

Three tools in the objectives can do the cutting, and they differ in style more than power.

• fdisk is a dialog-driven program that, its manual says, understands GPT, MBR and several older formats. In sudo fdisk /dev/sdb you press g to create an empty GPT table, n for a new partition, accept the defaults to use the whole disk, p to print the result and w to write it. Until you press w, every change lives only in memory, so q quits with the disk untouched.
• gdisk, "GPT fdisk," works the same way but only with GPT, and its manual notes it will convert an old MBR table to GPT when it opens one.
• parted supports both MS-DOS and GPT tables and is easy to script, because a whole operation fits on one command line: sudo parted /dev/sdb --script mklabel gpt mkpart data 1MiB 100%.

Afterwards, lsblk shows a child under the disk: sdb1. The partition exists. It still cannot hold a file.

Making a filesystem

A filesystem is the structure that turns raw blocks into named files and directories. mkfs is the front end that creates one: sudo mkfs.xfs /dev/sdb1 or sudo mkfs.ext4 -L data /dev/sdb1, where -L sets a label. Run blkid again and the partition now has a UUID and a TYPE. The UUID matters in Lesson 9, because mounting by UUID survives a disk being renamed.

Objective 1.3 names four filesystem types. Run df -T / on both lab machines and you will most likely see the usual default of each family: XFS on AlmaLinux, ext4 on Ubuntu.

tmpfs is no longer a curiosity. Debian 13's release notes announce that /tmp is now stored in a tmpfs by default, with a warning that large files placed there can run the machine out of memory. Run df -h /tmp to see whether yours is.

The core of it: a usable disk needs three layers in order: a partition table, a partition, and a filesystem made with mkfs. Choose XFS for very large filesystems that only grow, ext4 when you may need to shrink, Btrfs for snapshots and checksums, and tmpfs for scratch space in memory.

The other way to run out of space: inodes

Lesson 5 introduced the inode, the record of a file's metadata. Every file and directory needs one. On ext4 the supply is set when the filesystem is created: the mke2fs manual says it makes one inode for every given number of bytes of space, and warns that this ratio cannot be changed after the filesystem is created.

Now picture a web application that stores each user session as a tiny file and never cleans up. After a few months, writes fail with "No space left on device." The administrator runs df -h and sees the filesystem 40 percent full. The puzzle dissolves with one more option: df -i, which, in its manual's words, lists inode information instead of block usage. IUse% reads 100. Millions of tiny files have used every inode while barely touching the blocks.

The fix is to delete or archive the small files, then stop the program creating them, then consider the design. Rebuilding the filesystem with more inodes means backing up, recreating it with a smaller bytes-per-inode value or an explicit inode count, and restoring. XFS and Btrfs allocate inodes dynamically, so this failure is much rarer there, though XFS can still run short if its configured limit on inode space is reached.

Measuring space: df and du

The two measuring tools answer different questions, and confusing them causes real mistakes.

• df asks the filesystem how full it is. df -h shows sizes in readable units, -T adds the type and -i switches to inodes.
• du adds up the files it can see under a directory. du -sh /var/log gives one total; du -h -d 1 /var | sort -h lists each subdirectory of /var, sorted by size, which is how you find what filled a disk.

When they disagree, believe both and find the gap. If df reports far more used than du can find, the usual causes are a deleted file still held open (Lesson 5's lsof +L1) or files hidden underneath a mount point, which Lesson 9 explains.

Growing later: growpart, resize2fs and xfs_growfs

A virtual disk can be enlarged in the hypervisor in seconds, and cloud disks the same way. But the extra space appears at the end of the disk, outside your partition and outside your filesystem. Growing is therefore two steps, in order: first the partition, then the filesystem inside it.

1. Grow the partition. The growpart manual describes it as rewriting the partition table so that a partition takes up all the space it can: sudo growpart /dev/sdb 1. Note the space between the disk and the partition number.
2. Grow the filesystem. For ext4, sudo resize2fs /dev/sdb1; its manual says it can enlarge a mounted ext4 filesystem online, and enlarge or shrink one that is unmounted. For XFS, sudo xfs_growfs /mnt/data; its manual says the filesystem must be mounted to be grown, and the command takes the mount point.

The asymmetry is worth remembering for the exam. resize2fs can shrink ext4, but only while unmounted; XFS cannot be shrunk in practice at all. To shrink an XFS filesystem, you back it up, recreate it smaller and restore.

When something is damaged: fsck and xfs_repair

A power failure, a failing disk or a bug can leave a filesystem inconsistent. Journaling filesystems replay their journal at mount time, which fixes most interrupted writes. For deeper damage you need a checker, and two rules apply to all of them: never run a repair on a mounted filesystem, and take an image of a failing disk before repairing it, as Lesson 10 shows with ddrescue.

• fsck is a front end that calls the checker for the filesystem type. For ext4, that is e2fsck: sudo fsck /dev/sdb1 on the unmounted partition.
• For XFS, fsck does nothing useful. The fsck.xfs manual explains that XFS performs recovery at mount time, so fsck.xfs simply exits with a zero status, and points you to xfs_repair.
• xfs_repair must run on an unmounted filesystem; its manual warns that otherwise the result may be inconsistent or corrupt. Start with xfs_repair -n, no-modify mode, which only reports what it would fix. The -L option zeroes a dirty log, and the manual warns it can lose user files, so it is a last resort.

Common misconceptions

• "df shows free space, so the disk cannot be full." A filesystem can run out of inodes with blocks to spare; check df -i.
• "Enlarging the virtual disk makes the filesystem bigger." The new space sits outside the partition; grow the partition with growpart, then the filesystem with resize2fs or xfs_growfs.
• "Any filesystem can be shrunk the same way it grows." ext4 shrinks only when unmounted, and XFS cannot practically be shrunk at all.
• "fsck repairs every filesystem." fsck.xfs only exits; XFS repairs use xfs_repair, on an unmounted filesystem.
• "Files in /tmp survive a reboot." Where /tmp is a tmpfs, as on Debian 13 by default, it lives in memory and empties at every reboot.

What you now know

• Identify the target disk with lsblk (add -f for filesystems and UUIDs) and blkid before you change anything.
• GPT is the modern partition table: many partitions, 64-bit addresses, a backup copy and a protective MBR. MBR allows four primary partitions and tops out at 2 TiB with 512-byte sectors.
• fdisk changes nothing until w; gdisk is GPT-only; parted is easy to script.
• mkfs creates the filesystem: ext4 grows and shrinks, XFS grows only, Btrfs adds snapshots and checksums, tmpfs lives in memory.
• ext4's inode count is fixed at creation; "no space left" with free blocks means checking df -i.
• df reports what the filesystem says; du adds up what it can see.
• Grow the partition (growpart), then the filesystem (resize2fs device, or xfs_growfs mount point).
• Repair only unmounted filesystems: fsck for ext4, xfs_repair for XFS, starting with -n.

Sources

1. util-linux project. (n.d.). fdisk(8), lsblk(8) and blkid(8). Linux man-pages project. man7.org (fdisk); man7.org (lsblk); man7.org (blkid)
2. Red Hat. (n.d.). Disk partitions [Comparison of partition table types]. In Managing storage devices (RHEL 9). docs.redhat.com
3. GNU parted. (n.d.). parted(8). Linux man-pages project. man7.org; Smith, R. W. (n.d.). gdisk(8). Debian manual pages. manpages.debian.org
4. cloud-utils project. (n.d.). growpart(1). Debian manual pages. manpages.debian.org
5. e2fsprogs project. (n.d.). mke2fs(8) and resize2fs(8). Linux man-pages project. man7.org (mke2fs); man7.org (resize2fs)
6. xfsprogs project. (n.d.). xfs_growfs(8), xfs_repair(8) and fsck.xfs(8). Linux man-pages project. man7.org (xfs_growfs); man7.org (xfs_repair); man7.org (fsck.xfs)
7. Red Hat. (n.d.). Overview of available file systems and Increasing the size of an XFS file system. In Managing file systems (RHEL 9). docs.redhat.com (overview); docs.redhat.com (growing XFS)
8. BTRFS project. (n.d.). Introduction. BTRFS documentation. btrfs.readthedocs.io
9. The Linux kernel documentation. (n.d.). Tmpfs. docs.kernel.org
10. Debian. (2025). Release notes for Debian 13 (trixie): Issues to be aware of [The temporary-files directory /tmp is now stored in a tmpfs]. debian.org
11. GNU coreutils. (n.d.). df(1). Linux man-pages project. man7.org

Storage that is already flexible

Run lsblk on the AlmaLinux machine you installed in Lesson 1 and look under the system disk. You will probably find entries such as almalinux-root and almalinux-swap with the type lvm. The installer did not put your root filesystem directly on a partition. It put it on a logical volume, inside a volume group, built on a physical volume. Ubuntu's server installer offers the same arrangement by default, with names such as ubuntu-vg.

Why would an installer add two layers between the disk and the filesystem? Because partitions are rigid. Growing one means free space must sit right after it on the same disk. The Logical Volume Manager removes that limit. This lesson builds LVM storage step by step from the two spare disks in your lab, then changes one input to show where it pays off. The second half does the same for software RAID, which protects against a failed disk.

Three layers, three command families

The LVM manual defines the layers in three sentences. A Volume Group (VG) is a collection of one or more physical devices, each called a Physical Volume (PV). A Logical Volume (LV) is a virtual block device that can be used by the system or applications. Each block of an LV is stored on one or more PVs, placed by the kernel's Device Mapper. Space is handed out in fixed-size chunks called physical extents. Their size is set when the group is created, and the vgcreate manual warns it is difficult to change afterwards without recreating the group.

The commands follow the layers, and their names are predictable once you see the pattern: a prefix for the layer and a verb.

Worked example: a volume group from two disks

Take a snapshot of the AlmaLinux machine first. The two 2 GB disks from Lesson 7 still carry old filesystems, so clear their signatures, then build upward. Check every device name with lsblk before you type it.

1. sudo wipefs -a /dev/sdb /dev/sdc removes the old partition tables and filesystem signatures.
2. sudo pvcreate /dev/sdb /dev/sdc writes LVM labels, turning both disks into PVs. sudo pvs lists them with their size and free space. A whole disk can be a PV, as here, or a partition can.
3. sudo vgcreate labvg /dev/sdb /dev/sdc pools them. sudo vgs shows one group of nearly 4 GB, all free.
4. sudo lvcreate -n data -L 1G labvg carves a 1 GiB LV named data. -L gives a size; -l 100%FREE would take everything left. sudo lvs lists it.
5. The new LV appears twice in /dev, as /dev/labvg/data and /dev/mapper/labvg-data. Both names point to the same device.
6. sudo mkfs.xfs /dev/labvg/data, then mount it on /mnt/data and check df -h /mnt/data.

Now grow it while it is mounted and in use: sudo lvextend -r -L +1G /dev/labvg/data. The lvextend manual describes -r as resizing the underlying filesystem together with the LV. Without -r, the LV grows but the filesystem inside it stays the same size, and you would have to run xfs_growfs or resize2fs yourself, exactly as in Lesson 7. Check df again and the extra gigabyte is there. No partition was moved and nothing was unmounted.

Bottom line: PVs are disks LVM may use, a VG pools them, and LVs are carved from the pool; lvextend -r grows an LV and its filesystem in one step.

Change one input: the group is full

Run the same extension again with -L +3G. This time lvextend refuses, reporting insufficient free space, because the group's two disks are nearly used up. With plain partitions you would now be planning downtime. With LVM, you add a disk to the pool:

1. Add a third virtual disk in the hypervisor, then sudo pvcreate /dev/sdd.
2. sudo vgextend labvg /dev/sdd adds it to the group; vgs shows the new free space.
3. Repeat the lvextend. It succeeds, and the LV now spans three disks.

The same flexibility lets you retire a disk without unmounting anything. The pvmove manual says it moves the allocated physical extents on a source PV to one or more destination PVs. So sudo pvmove /dev/sdb empties sdb onto the other disks while the filesystem stays mounted, sudo vgreduce labvg /dev/sdb takes it out of the group, and sudo pvremove /dev/sdb wipes its label. This is how administrators replace an ageing disk with no outage.

A few more commands complete the picture. If an underlying disk grows, pvresize tells LVM the PV is bigger; its manual notes the PV can already be in a VG with active LVs on it. vgchange -a n labvg deactivates every LV in a group and -a y activates them; lvchange does the same for one LV. And when you physically move disks to another server, vgexport puts the VG in an exported state that, its manual says, ensures it cannot be used until vgimport brings it in on the new system.

To dismantle everything, work from the top down: unmount, then lvremove, then vgremove, then pvremove. Shrinking is the one direction to treat with care. lvresize can make an LV smaller, but the filesystem must be shrunk first, which ext4 allows only unmounted and XFS does not allow at all.

Bottom line: a full VG is fixed by adding a PV with vgextend; a disk is retired online with pvmove, vgreduce and pvremove.

Software RAID: surviving a dead disk

LVM makes storage flexible, but by default it does not make it safe. If one disk in your three-disk VG fails, the LV that spans it is damaged. RAID addresses that. The mdadm manual describes RAID devices as virtual devices created from two or more real block devices, and notes that some RAID levels include redundancy and so can survive some degree of device failure. Linux implements software RAID through the md (multiple devices) driver, which supports RAID 0, 1, 4, 5, 6 and 10 among other modes.

Read the table as trade-offs. RAID 0 buys speed and capacity with no safety. RAID 1 buys safety with half the capacity. RAID 5 and 6 buy capacity-efficient safety but must rebuild by reading every surviving disk, which takes hours on large drives. RAID 10 buys speed and safety with half the capacity.

Worked example: a RAID 1 mirror with mdadm

Dismantle the LVM lab first, as described above, and wipe the disks with wipefs. Then:

1. sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc builds the mirror. Answer yes if it warns about boot metadata.
2. cat /proc/mdstat shows the array and a progress bar while the two disks synchronize. The md manual describes this file as containing the status of currently running arrays. [UU] means both members are up.
3. sudo mdadm --detail /dev/md0 gives the full report: level, size, state and each member.
4. Make a filesystem on /dev/md0 and mount it like any other block device. You could equally make md0 a PV and put LVM on top, a common production design.

Now break it on purpose. sudo mdadm /dev/md0 --fail /dev/sdc marks a member faulty, and /proc/mdstat changes to [U_]: the array is degraded but your files are all still readable. Remove the failed member with --remove /dev/sdc, then put a replacement in with sudo mdadm /dev/md0 --add /dev/sdc and watch /proc/mdstat show the rebuild.

Two steps make the array survive a reboot reliably. Record it in mdadm's configuration file with sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf on the Red Hat family, or /etc/mdadm/mdadm.conf on Debian and Ubuntu. Then, if the system needs the array during early boot, rebuild the initial RAM disk as Lesson 2 described.

RAID is not a backup

One sentence belongs on every server room wall. A mirror copies your mistakes as faithfully as your data: delete a file and it is deleted on both disks at once, and ransomware encrypts both copies. RAID protects against a failed disk. It does nothing against deletion, corruption, theft or fire. That is the job of backups, in Lesson 10.

Common misconceptions

• "lvextend makes the filesystem bigger." It grows the LV. Add -r, or run xfs_growfs or resize2fs afterwards, to grow the filesystem inside it.
• "LVM protects data if a disk fails." A plain LV spanning several disks is damaged when any of them fails. Redundancy comes from RAID, under or inside LVM.
• "RAID 1 is a backup." It mirrors deletions and corruption instantly; it protects only against disk failure.
• "RAID 0 gives redundancy because it uses two disks." It stripes data with no redundancy, so losing either disk loses everything.
• "To remove a disk from LVM, just pull it out." Move its extents with pvmove, remove it with vgreduce, then wipe it with pvremove.

The takeaway

• LVM layers: PVs (pvcreate) pooled into a VG (vgcreate), carved into LVs (lvcreate), reached as /dev/vg/lv or /dev/mapper/vg-lv.
• pvs, vgs and lvs summarize; the display commands give detail; the scan commands find LVM metadata on disks.
• lvextend -r grows an LV and its filesystem together; vgextend adds a PV when the VG is full.
• pvmove empties a PV while mounted; vgreduce and pvremove finish retiring it; vgexport and vgimport move a VG between systems; vgchange -a activates or deactivates.
• RAID 0 stripes with no safety, RAID 1 mirrors, RAID 5 and 6 add one or two parity blocks, RAID 10 stripes mirrored pairs.
• mdadm --create builds an array, /proc/mdstat and mdadm --detail report it, and --fail, --remove and --add handle a disk replacement; record arrays in mdadm.conf.
• RAID guards against disk failure, never against deletion: keep backups.

Sources

1. LVM2 project. (n.d.). lvm(8), vgcreate(8), lvextend(8), pvmove(8), pvresize(8), vgchange(8) and vgexport(8). Linux man-pages project. man7.org (lvm); man7.org (vgcreate); man7.org (lvextend); man7.org (pvmove); man7.org (vgexport)
2. Red Hat. (n.d.). Configuring and managing logical volumes (RHEL 9). docs.redhat.com
3. Brown, N., and others. (n.d.). mdadm(8) and md(4). Linux man-pages project. man7.org (mdadm); man7.org (md)
4. The Linux kernel documentation. (n.d.). RAID arrays. docs.kernel.org

A reboot that ends in emergency mode

A file server is rebooted after routine updates. Instead of a login prompt, the console shows a request for the root password and a message that the system is in emergency mode. Nothing else starts: no SSH, no file sharing. The cause, found ten minutes later, is one line in /etc/fstab. Last month someone attached a USB disk for a one-off backup, added it to fstab so it would mount automatically, and then took the disk away.

At boot, systemd tried to mount every filesystem listed. One of them did not exist, the mount was required, and so the boot stopped where Lesson 2 said a missing fstab entry stops it. This lesson works out how to fix that server, then how to write fstab so it never happens again, then extends the same ideas to network storage. All of it is objective 1.3.

What mounting is

Linux has one directory tree, starting at /. Each filesystem, whether on a partition, a logical volume, a USB stick or a server across the network, is attached to that tree at a directory called its mount point. sudo mount /dev/sdb1 /mnt/data attaches it and sudo umount /mnt/data detaches it. Note the spelling: umount, with no n.

The mount manual warns about one side effect. The previous contents of the mount point directory, and its owner and mode, become invisible while the filesystem is mounted. Files are not deleted; they are hidden underneath. If files were written to /backup while its disk was not mounted, they fill the root filesystem and du cannot see them once the disk is mounted again. That is one cause of the df and du disagreement from Lesson 7.

Three places tell you what is mounted right now.

• /proc/mounts is the kernel's own list, always current.
• /etc/mtab was once a file that the mount command maintained. Its manual says that on current Linux systems it is better to make it a symbolic link to /proc/mounts, and on your lab machines it is.
• findmnt prints the same information as a readable tree, and mount with no arguments prints a flat list.

Reading /etc/fstab

/etc/fstab lists the filesystems to mount at boot. Each line has six fields separated by spaces.

Prefer UUIDs in field 1. Device names such as /dev/sdb1 are assigned in the order disks are found, and adding a disk can rename the others. A UUID, which you read with blkid, stays with the filesystem.

Never reboot to test an fstab edit. Two commands test it while you can still fix mistakes: sudo findmnt --verify, which its manual says checks that /etc/fstab is parsable and usable, and sudo mount -a, which mounts everything listed that is not already mounted. On systemd systems, run sudo systemctl daemon-reload after editing fstab too, because systemd converts each line into a mount unit, as Lesson 18 explains.

Fixing the server, then preventing a repeat

Back to the server stuck in emergency mode. Enter the root password at the prompt. The root filesystem may be mounted read-only, so make it writable with mount -o remount,rw /; the mount manual describes remount as changing the options of an already-mounted filesystem, and the commonest use is exactly this. Edit /etc/fstab, comment out the line for the missing disk with a #, run systemctl daemon-reload, and reboot. The server comes up normally.

Now prevent it. If a disk may legitimately be absent, such as a removable backup drive, add nofail to its options. systemd's mount documentation explains the effect precisely: with nofail, the mount is only wanted, not required, by local-fs.target, so the boot continues without waiting for it and whether or not it can be mounted.

Key idea: fstab entries are boot requirements. Use UUIDs, test with findmnt --verify and mount -a before rebooting, and mark optional disks nofail.

Mount options and what each one buys

Options go in the fourth fstab field, or after -o on the command line. defaults stands for a standard set that includes rw. The ones the exam lists fall into three groups.

The security trio, nodev, nosuid and noexec, belongs on anything users can write to but should not run programs or devices from: /tmp, /var/tmp, removable media and many network shares. The mount manual notes they are applied to the mount point, not stored in the filesystem, so the same disk can be strict in one place and permissive in another. Be honest about their limits. noexec stops direct execution, so ./tool fails, but a script can still be fed to an interpreter, as in bash /tmp/script.sh. These options are one layer of defense, as Lesson 21 explains.

Modern kernels already avoid most access-time writes by default with a compromise called relatime, so noatime is a modest gain, mostly on filesystems with very many small reads.

Network storage: NFS and SMB

Mounting a filesystem from another machine uses the same mount command with a different type and source. Objective 1.3 names two protocols.

NFS. The nfs manual describes it as an Internet Standard protocol created by Sun Microsystems in 1984 for file sharing between systems on a local network, and says the Linux client may support versions 3, 4.0, 4.1 and 4.2. The source is written server:/path:

• sudo mount -t nfs files01:/exports/projects /mnt/projects
• In fstab: files01:/exports/projects /mnt/projects nfs defaults,nofail 0 0

SMB. Windows file shares, and Linux servers running Samba, use SMB. The mount.cifs manual says it mounts a CIFS or SMB3 filesystem and is usually invoked through mount -t cifs. The source looks like a Windows path with forward slashes:

• sudo mount -t cifs //files02/finance /mnt/finance -o credentials=/root/.smb-finance

The manual prefers a credentials file, holding lines such as username= and password=, over putting passwords on the command line, where other users could see them in the process list. Make that file readable by root only with chmod 600.

How does systemd know to wait for the network before mounting these? It recognizes nfs and cifs as network filesystem types and orders them after the network is up. The _netdev option forces that treatment for mounts whose type does not reveal it, such as an iSCSI disk. Adding nofail to network mounts is also wise, so a file server's outage does not stop your server booting.

Mounting on demand: autofs

A university lab with 2,000 users does not mount every home directory at boot. It mounts each one when its owner logs in and unmounts it after a period of inactivity. That is autofs. The automount manual says it reads the auto.master map and sets up mount points so that file systems are mounted automatically when accessed.

Two files define it. A line in the master map, /etc/auto.master, names a parent directory and a map file: /mnt/nfs /etc/auto.nfs --timeout=300. The map file lists keys and what each one mounts: projects -fstype=nfs files01:/exports/projects. After sudo systemctl enable --now autofs, nothing is mounted until someone types cd /mnt/nfs/projects. Then the share appears, and after 300 idle seconds it is unmounted again. The --timeout option, the auto.master manual says, sets that expiry. A common surprise: ls /mnt/nfs shows an empty directory until a key is accessed by name.

Measuring performance: fio

Objective 1.3 lists fio among the storage utilities. Its documentation describes it as spawning threads or processes that perform a particular type of I/O that you specify, so you can simulate a real workload instead of guessing. A safe first test reads from a scratch file:

fio --name=randread --filename=/mnt/data/fio.test --size=512M --rw=randread --bs=4k --direct=1 --runtime=30 --time_based

The report gives IOPS, bandwidth and latency percentiles. Two rules keep fio safe. Point it at a test file, never at a raw device that holds data, because write tests overwrite whatever is there. And measure before a change as well as after it, so Lesson 36's performance work has a baseline.

Common misconceptions

• "Mounting over a directory deletes what was in it." The old contents are hidden until the filesystem is unmounted, then reappear.
• "The way to test an fstab change is to reboot." Use findmnt --verify and mount -a first; a bad line found at boot can stop the whole system.
• "noexec makes a filesystem safe from scripts." It blocks direct execution, but an interpreter can still read and run a script stored there.
• "/dev/sdb1 will always be the same disk." Device names follow detection order; UUIDs and labels stay with the filesystem.
• "autofs mounts everything at boot." It mounts each share only when it is accessed and unmounts it after the timeout.

Summing up

• mount attaches a filesystem at a mount point and umount detaches it; /proc/mounts is the kernel's list, /etc/mtab links to it, and findmnt displays it.
• fstab fields: device (use UUID), mount point, type, options, dump, fsck pass (1 for root, 2 for others, 0 to skip).
• Test fstab with findmnt --verify and mount -a; reload systemd with daemon-reload.
• From emergency mode: mount -o remount,rw /, fix fstab, reboot. Mark optional disks nofail.
• noatime and nodiratime save writes; nodev, nosuid and noexec harden user-writable mounts; ro and remount control writes.
• NFS sources are server:/path; SMB sources are //server/share with a root-only credentials file.
• autofs mounts on access via auto.master and a map, and unmounts after a timeout; fio measures storage against a scratch file.

Sources

1. util-linux project. (n.d.). mount(8), fstab(5) and findmnt(8). Linux man-pages project. man7.org (mount); man7.org (fstab); man7.org (findmnt)
2. systemd project. (n.d.). systemd.mount: Mount unit configuration [Options nofail and _netdev]. freedesktop.org. freedesktop.org
3. Linux NFS project. (n.d.). nfs(5): fstab format and options for the nfs file systems. Linux man-pages project. man7.org
4. cifs-utils project. (n.d.). mount.cifs(8). Linux man-pages project. man7.org
5. autofs project. (n.d.). automount(8) and auto.master(5). Linux man-pages project. man7.org (automount); man7.org (auto.master)
6. Axboe, J. (n.d.). fio: Flexible I/O tester (rev. 3.42). fio.readthedocs.io

Three numbers from 2012

In a short paper for the US Computer Emergency Readiness Team, Paul Ruggiero and Matthew Heckathorn compressed backup advice into three numbers. Keep 3 copies of any important file: the original and two backups. Keep them on 2 different media types, to protect against different hazards. Store 1 copy offsite. The 3-2-1 rule tells you how many backups to keep. It does not tell you what kind of backup to make, and that is where two competent administrators can disagree for an hour.

Here is the disagreement, set on the small Ubuntu web server in your lab, which holds a website in /var/www, configuration in /etc and a database. Each side has real evidence. Objective 1.6 asks you to know the tools both sides use, and the exam asks you to choose between them in scenarios.

The case for imaging the whole disk

The first administrator wants a nightly image: a bit-for-bit copy of the entire disk, made with dd. The dd manual describes it as copying a file, converting and formatting according to its operands, and a disk is a file in /dev. One command copies everything:

sudo dd if=/dev/vdb of=/backup/vdb.img bs=4M status=progress

Her evidence is strong. An image contains the partition table, the boot loader, every filesystem, permissions, SELinux labels and files nobody remembered to list. Restoring a dead server means writing the image to a new disk with the same command reversed, and the machine boots as it was. There is no list of directories to get wrong.

Her weaknesses are real too. An image of a 500 GB disk is 500 GB, empty space included, unless compressed, and every night copies all of it again. It must be taken from a disk that is not changing, so either the system is shut down or the image comes from a snapshot, such as an LVM or hypervisor snapshot; imaging a mounted, busy filesystem can capture it half-written. Restoring one deleted file means mounting the whole image somewhere first. And dd is unforgiving: if is the input and of the output, and swapping them overwrites the disk you meant to protect. Administrators joke that dd stands for "disk destroyer," and they mean it.

The case for backing up files

The second administrator wants file-level backups: archive the data directories with tar, compress them, and send a copy offsite with rsync. His evidence: the archives hold only real data, so they are small; tar records ownership and permissions; one file can be restored in seconds; and rsync, whose manual says it is famous for a delta-transfer algorithm that sends only the differences between source and destination files, makes the nightly offsite copy fast even over a slow link.

His weaknesses mirror hers. After a total loss he must reinstall the operating system before restoring anything. If a directory was left off the list, it is gone. And copying the files of a running database can capture them mid-write, so databases are backed up with their own dump tools and the dump file is archived instead.

What settles it

Two questions settle the argument for any particular system. How long can the service be down while you restore, the recovery time objective? And how much recent data can you afford to lose, the recovery point objective? The Network+ course covers these terms in its disaster recovery lesson; here they decide the tool. A system disk that must be back in fifteen minutes favors images or snapshots. Data that changes all day, where users ask for last Tuesday's version of one file, favors frequent file-level backups with history.

In practice most organizations do both: an image or VM snapshot of the system, file-level backups of the data, and the 3-2-1 rule across them. One more test settles every dispute about backups. Restore something. A backup that has never been restored is a hope, not a backup.

The point: images restore whole systems quickly but are large and must be taken from a quiet disk; file archives are small and selective but need a working system to restore into. Choose by recovery time and recovery point, and prove the choice with a test restore.

Archiving: tar and cpio

An archive bundles many files and their metadata into one file. Compression is a separate step, which tar can call for you. Learn tar's letters as a set:

Two habits prevent bad restores. List an archive before extracting it, to see whether its paths start with a directory name or with ./. And extract test restores into a scratch directory with -C, never over the live system.

cpio is older and works differently: it takes the list of files on standard input. Its manual describes copy-out mode as reading a list of file names from standard input and writing the archive to standard output. So you pair it with find: find /etc -depth | cpio -ov > etc.cpio creates an archive, and cpio -idv < etc.cpio extracts one, with -d creating directories as needed. You have met cpio's format before: Lesson 2's initial RAM disk is a compressed cpio archive.

Compression: gzip, bzip2, xz, 7-Zip and unzip

Each compressor makes a different trade between speed, memory and size.

Each of gzip, bzip2 and xz has a matching decompressor (gunzip, bunzip2, unxz) and replaces the original file with the compressed one by default, so use -k to keep the original when you need it.

Old logs are usually stored compressed, and you rarely need to decompress them to disk. The gzip package supplies three readers. zcat prints a compressed file. zgrep, its manual says, invokes grep on compressed files and passes all options straight through, so zgrep -c 'Failed password' /var/log/auth.log.2.gz counts matches in an old rotated log. zless pages through compressed or plain text one screen at a time.

Synchronizing: rsync

rsync copies trees of files, locally or over SSH, and on later runs sends only what changed. The options worth memorizing:

• -a, archive mode, which the manual expands as -rlptgoD: recursive, and preserving links, permissions, times, group, owner and devices.
• -v lists files as they go, -z compresses data in transit, and -n (--dry-run) performs a trial run with no changes made.
• --delete deletes files in the destination that no longer exist in the source, making an exact mirror. It also mirrors mistakes: delete a file by accident and the next run deletes the backup copy. Always try it with -n first.

The trailing slash catches everyone once. The rsync manual explains that a trailing slash on the source means "copy the contents of this directory" rather than "copy the directory by name." So rsync -a /var/www/ backup01:/srv/www/ puts the site's files directly into /srv/www on backup01, while rsync -a /var/www backup01:/srv/www/ creates /srv/www/www.

Rescuing a failing disk: ddrescue

When a disk starts returning read errors, dd is the wrong tool: it stops or stalls at the first bad area, and every retry stresses a dying drive. The ddrescue manual describes GNU ddrescue as a data recovery tool that copies from one device or file to another, trying to rescue the good parts first in case of read errors. It also says to always use a mapfile, because without one a rescue cannot be resumed, only restarted.

sudo ddrescue /dev/sdc /mnt/rescue/sdc.img /mnt/rescue/sdc.map

Three rules go with it. Never write anything to the failing disk. Put the image on a different, healthy disk with enough room. And run repairs such as fsck or xfs_repair from Lesson 7 on the image or a copy of it, never on the failing original.

The point: rsync keeps copies current cheaply, dd clones devices exactly, and ddrescue saves what it can from a disk that is failing.

Common misconceptions

• "RAID or a sync to another server is a backup." Both reproduce deletions and corruption at once. A backup keeps older versions you can go back to, ideally with one copy offsite and offline.
• "dd is safe on a running system." Imaging a mounted, changing filesystem can capture it half-written. Image from a snapshot or an unmounted disk, and check if= and of= twice.
• "rsync --delete protects against accidental deletion." It does the opposite: it removes files from the backup when they disappear from the source.
• "A higher compression level is always better." xz at high settings can need gigabytes of memory and much more time for a small gain; gzip is often the right choice for daily jobs.
• "If the backup job reports success, the backup works." Only a test restore proves it.

Looking back

• 3-2-1: three copies, two media types, one offsite. Choose image or file backups by recovery time and recovery point, and test restores.
• tar: c create, x extract, t list, f file, z gzip, j bzip2, J xz, -C target directory, -g incremental snapshots.
• cpio reads file names on standard input: find | cpio -o creates, cpio -id extracts.
• gzip is fastest, bzip2 compresses better, xz usually smallest and hungriest; 7-Zip archives with a high ratio; unzip handles .zip files.
• zcat, zgrep and zless read compressed files without decompressing them to disk.
• rsync -a preserves metadata and sends only changes; a trailing slash on the source copies contents; test --delete with -n.
• dd clones devices bit for bit; ddrescue, with a mapfile, rescues failing disks good parts first.

Sources

1. Ruggiero, P., and Heckathorn, M. A. (2012). Data backup options. United States Computer Emergency Readiness Team. cisa.gov (PDF)
2. GNU tar and GNU coreutils. (n.d.). tar(1) and dd(1). Linux man-pages project. man7.org (tar); man7.org (dd)
3. GNU cpio. (n.d.). cpio(1). Debian manual pages. manpages.debian.org
4. Debian. (n.d.). gzip(1), zgrep(1), zless(1), bzip2(1), xz(1) and unzip(1). Debian manual pages. gzip; zgrep; zless; bzip2; xz; unzip
5. Pavlov, I. (2026). 7-Zip and 7-Zip download. 7-zip.org
6. Tridgell, A., Mackerras, P., and Davison, W. (n.d.). rsync(1). Linux man-pages project. man7.org
7. Diaz Diaz, A. (n.d.). ddrescue(1): Data recovery tool. Debian manual pages. manpages.debian.org

Same output, three different machines underneath

Type ip address on each of your lab machines and the output looks almost identical: a loopback interface, lo, holding 127.0.0.1, and one Ethernet interface holding an address from the hypervisor's NAT. Yet the two machines got there by different routes. AlmaLinux was configured by NetworkManager. Ubuntu Server was configured by Netplan, which handed its settings to another program. And if you changed the address with the ip command itself, the change would vanish at the next reboot on both.

That is the comparison at the heart of objective 1.4. Several layers can configure a Linux network, and you must know which one owns the settings before you change anything. This lesson compares them, then compares the tools that test a network, which the objective lists at length. For the networking theory underneath, addresses, subnets, ports and protocols, the Network+ Certification Prep course (CS 215) is the natural companion; this lesson assumes its basics and focuses on Linux.

How a name becomes an address

When a program asks for files01.example.com, the C library decides where to look by reading /etc/nsswitch.conf, the Name Service Switch configuration. Its hosts: line lists sources in order, typically files then dns, often with others such as myhostname or resolve. The nsswitch.conf manual explains that the first column names a database and the rest list the services to try for it, with optional actions such as returning early.

Two files serve the classic sources.

• /etc/hosts is, in its manual's words, a simple text file that associates IP addresses with hostnames, one line per address: the address, the canonical name, then any aliases. It is checked first when files comes first, so an entry there overrides DNS on this machine. That makes it useful for testing and dangerous when forgotten.
• /etc/resolv.conf tells the resolver which DNS servers to ask, with nameserver lines, up to three according to its manual, and which domains to append to short names, with a search line.

On Ubuntu, resolv.conf usually contains a single line, nameserver 127.0.0.53. That address is a local stub run by systemd-resolved, which forwards queries to the real servers; Lesson 18 shows how to inspect it with resolvectl. Editing that file by hand therefore changes little, which surprises administrators used to older systems.

What matters here: nsswitch.conf sets the order of lookup, /etc/hosts supplies local answers, and resolv.conf, or the stub behind it, names the DNS servers. A wrong answer for one name on one machine often means a stale /etc/hosts line.

Three ways to configure an interface

The objectives also list a tool called nmconnect. NetworkManager ships no command with exactly that name. Its documented interfaces are nmcli, the text menu nmtui, which comes with shortcuts named nmtui-edit and nmtui-connect, and the graphical nm-connection-editor. If an exam item says nmconnect, read it as NetworkManager's connection tooling.

ip: the live view and quick changes

The ip command reads and changes the kernel's network state directly. Its three most used objects are in the objectives.

• ip link shows interfaces and their state; sudo ip link set enp0s8 up brings one up.
• ip address (or ip a) shows addresses; sudo ip address add 192.168.56.20/24 dev enp0s8 adds one. ip -br a gives a one-line-per-interface summary.
• ip route shows the routing table, including the default gateway; sudo ip route add default via 192.168.56.1 sets one.

Everything ip changes is live and temporary, which makes it ideal for testing and for diagnosis, and wrong for permanent configuration. The legacy hostname command similarly prints the host name and can set it only until the next boot; hostnamectl, in Lesson 18, sets it permanently.

NetworkManager and nmcli

NetworkManager thinks in connections, saved profiles of settings, which it applies to devices. nmcli device status lists devices, and nmcli connection show lists profiles, whose names on AlmaLinux often match the interface, such as enp0s3. Setting a static address is a modify followed by an activation:

• sudo nmcli connection modify enp0s8 ipv4.method manual ipv4.addresses 192.168.56.20/24 ipv4.gateway 192.168.56.1 ipv4.dns 192.168.56.1
• sudo nmcli connection up enp0s8

The modify step only edits the saved profile; nothing changes on the wire until the connection is brought up again. The nmcli documentation describes the stored format as a keyfile, and on current Red Hat-family systems these files live in /etc/NetworkManager/system-connections/. Prefer nmcli to editing them by hand, because it validates what you type.

Netplan and its safety net

On Ubuntu Server the configuration is a YAML file under /etc/netplan/, often named 50-cloud-init.yaml. A static address looks like this, and in YAML the indentation is part of the syntax:

network:
  version: 2
  ethernets:
    enp0s8:
      addresses: [192.168.56.10/24]
      routes:
        - to: default
          via: 192.168.56.1
      nameservers:
        addresses: [192.168.56.1]

Netplan's documentation describes netplan apply as generating back-end configuration from the YAML files and invoking systemd-networkd or NetworkManager to bring the interfaces up. Over SSH, one mistyped address and you lose the connection you are typing through. netplan try exists for exactly that: it applies the configuration and automatically rolls it back unless you confirm within a time limit, 120 seconds by default. The documentation notes this is especially useful on remote systems to avoid being locked out. netplan status displays the current configuration in readable form.

What matters here: ip changes the live state and forgets it at reboot; nmcli edits saved NetworkManager profiles that take effect when brought up; Netplan turns YAML into configuration, and netplan try rolls back a change you do not confirm.

The testing tools, compared

Objective 1.4's tool list looks long until you sort it by the question each tool answers, roughly from the bottom of the network stack to the top.

Read the table as a decision aid. If DNS fails but ping by address works, use dig, not traceroute. If a service is "down" but the host answers ping, use ss on the server and nc from the client to find out whether anything is listening and reachable. If users complain that large transfers stall while small ones work, tracepath can reveal a path MTU smaller than expected, a problem Lesson 34 diagnoses in full.

Two of these tools carry legal and ethical weight. nmap and tcpdump are standard administration tools, but scanning networks or capturing traffic you do not own or administer can break the law and your employer's policies. Use them in your lab, on the host-only network between your two virtual machines.

Common misconceptions

• "ip address add makes a permanent change." It changes the running kernel only; use nmcli or Netplan for settings that survive a reboot.
• "nmcli connection modify changes the interface immediately." It edits the saved profile; bring the connection up again to apply it.
• "Editing /etc/resolv.conf on Ubuntu changes the DNS servers." It usually points at the systemd-resolved stub, 127.0.0.53; configure DNS through Netplan or resolved instead.
• "If ping fails, the service is down." Many hosts block ICMP. Test the service's own port with nc or curl before concluding anything.
• "/etc/hosts is only consulted when DNS fails." With files listed first in nsswitch.conf, it is consulted first, and its answers win.

Recap

• Name lookups follow the hosts line of /etc/nsswitch.conf; /etc/hosts gives local answers; /etc/resolv.conf names DNS servers, on Ubuntu usually the 127.0.0.53 stub.
• ip link, ip address and ip route show and change live settings that are lost at reboot.
• NetworkManager stores connection profiles as keyfiles; nmcli connection modify edits them and nmcli connection up applies them; nmtui and nm-connection-editor are menu and graphical alternatives.
• Netplan YAML in /etc/netplan generates back-end configuration; netplan apply applies, netplan try rolls back after 120 seconds unless confirmed, netplan status shows the result.
• Tools by question: ethtool (link), arp or ip neigh (neighbors), ping (reachability), traceroute, tracepath and mtr (path), dig and nslookup (DNS), ss (local ports), nc (remote port), nmap (discovery), curl (web), tcpdump (packets), iperf3 (throughput).

Sources

1. Linux man-pages project. (n.d.). nsswitch.conf(5), hosts(5) and resolv.conf(5). man7.org (nsswitch.conf); man7.org (hosts); man7.org (resolv.conf)
2. iproute2 project. (n.d.). ip(8) and ss(8). Linux man-pages project. man7.org (ip); man7.org (ss)
3. NetworkManager project. (n.d.). nmcli and nmtui reference manuals. networkmanager.dev (nmcli); networkmanager.dev (nmtui)
4. Netplan project. (n.d.). netplan try, netplan apply and netplan status. Netplan documentation. try; apply; status
5. Linux man-pages project. (n.d.). ping(8), traceroute(8), tracepath(8), ethtool(8) and arp(8). ping; traceroute; tracepath; ethtool; arp
6. Debian. (n.d.). mtr(8), dig(1), nc(1) and iperf3(1). Debian manual pages. mtr; dig; nc; iperf3
7. Lyon, G. (n.d.). Nmap reference guide. nmap.org; The Tcpdump Group. (n.d.). tcpdump(1). tcpdump.org; Stenberg, D. (n.d.). curl man page. curl.se

A test server for one week

On a Monday morning a developer asks for a copy of the company's web server, for one week, to test an upgrade. The company runs its servers as virtual machines on a Linux host. By Monday afternoon the copy exists; on Friday it is gone, and in between it has been resized, snapshotted, rolled back and moved to another host without being switched off. Following that one virtual machine through its week covers objective 1.7.

First, what does the host actually run? The KVM project describes KVM, the Kernel-based Virtual Machine, as a full virtualization solution for Linux on hardware with virtualization extensions, Intel VT or AMD-V. It consists of a loadable kernel module, kvm.ko, and a processor-specific module, kvm-intel.ko or kvm-amd.ko. With those loaded, the Linux kernel itself becomes the hypervisor. You can check a processor for the extensions with lscpu, which lists them as vmx (Intel) or svm (AMD), and confirm KVM is ready when /dev/kvm exists.

Who does what: KVM, QEMU and libvirt

KVM alone does not make a usable virtual machine. It lets guest code run directly on the processor, safely isolated. Someone must still provide the rest of a computer: firmware, disks, network cards, a display. That is QEMU's job. QEMU's documentation calls it a generic and open source machine emulator and virtualizer. It can emulate a whole machine in software, which is slow, or work with a hypervisor such as KVM so that the guest runs directly on the host CPU, which is fast.

Remember: KVM accelerates, QEMU emulates the machine, and libvirt manages; virsh and virt-manager are the front ends to libvirt.

Bare metal or virtual?

Before Monday's request is granted, it is fair to ask whether it should be a VM at all.

A one-week test copy is the textbook case for a VM.

Monday: from template to clone

Nobody installs Linux from scratch for each new VM. The company keeps a baseline image template, sometimes called a golden image: a VM installed once, patched, configured to the company standard, then shut down and generalized. Generalizing matters because a straight copy keeps everything that should be unique. The virt-sysprep manual lists what it resets so that clones can be made: SSH host keys, persistent network MAC configuration and user accounts, among many other items. Two clones with the same SSH host keys would let one impersonate the other.

The administrator then clones the template: virt-clone --original web-template --name web-test --auto-clone copies the disk and creates a new definition with a new MAC address. A faster alternative uses qcow2, QEMU's copy-on-write format: qemu-img create -f qcow2 -b web-template.qcow2 -F qcow2 web-test.qcow2 makes a small overlay file that stores only the clone's changes, reading everything else from the template, which must then never change.

Monday afternoon: resources and disk images

Each VM is given four resources: CPU (a number of virtual CPUs), RAM, storage (one or more disk images) and network interfaces. The upgrade test needs more space than the template had, so the administrator turns to qemu-img.

• Image properties. qemu-img info web-test.qcow2 shows the format, the virtual size the guest sees, the space actually used on the host, which the manual notes can differ, and any backing file.
• Resize. qemu-img resize web-test.qcow2 +10G enlarges the virtual disk; inside the guest you then grow the partition and filesystem exactly as in Lesson 7. Shrinking is dangerous: the manual says you must first shrink filesystems and partitions inside the VM, that failure to do so results in data loss, and that the --shrink option is required.
• Convert. qemu-img convert -f vmdk -O qcow2 vendor-appliance.vmdk appliance.qcow2 turns a VMware disk into qcow2. Converting to raw gives the simplest, fastest format; qcow2 adds snapshots, backing files and thin allocation.

The guest's virtual devices matter too. If QEMU emulated a real network card model, the guest would drive it like real hardware, through slow, faithfully copied register accesses. Paravirtualized drivers take a different approach: the guest knows it is virtual and uses devices designed for virtualization. The standard for them is VirtIO, whose specification states the goal that virtual environments should have a straightforward, efficient, standard and extensible mechanism for virtual devices, rather than boutique per-environment mechanisms. A Linux guest using virtio-net and virtio-blk names its disks /dev/vda rather than /dev/sda, which you met in Lesson 7.

Tuesday: plugging it into a network

libvirt can connect a VM to several kinds of virtual network. Its network documentation describes the forward modes, and the objectives list five types.

The test VM goes on an isolated network with a copy of the database server, so its experiments cannot touch production.

Wednesday: states and snapshots

virsh list --all shows every VM and its state. The virsh manual defines the states you will see.

• running: executing on a CPU. idle: waiting, for example on I/O.
• paused: frozen by virsh suspend; it still holds its memory but is not scheduled. virsh resume continues it.
• in shutdown: the guest has been told to stop and is stopping gracefully (virsh shutdown).
• shut off: not running. virsh start boots it; virsh destroy forces a running VM straight to this state, like pulling the plug, without deleting anything.
• crashed, and pmsuspended: suspended by the guest's own power management.

Before running the upgrade, the administrator takes a snapshot: virsh snapshot-create-as web-test before-upgrade. The upgrade breaks the web application, which is what a test is for. virsh snapshot-revert web-test before-upgrade returns the VM to Wednesday morning in seconds. Snapshots are for exactly this. They are not backups: they live with the VM's disk, depend on it, and vanish with it.

Thursday: migration and nested virtualization

On Thursday the host needs a firmware update and a reboot. Instead of shutting the test VM down, the administrator migrates it while it runs: virsh migrate --live web-test qemu+ssh://kvm02/system. Live migration copies the guest's memory to the other host while it keeps running, then switches over in a brief pause. It works most smoothly when both hosts reach the same shared storage; otherwise the disks must be copied too, which the migrate command's storage-copy options handle. An offline migration simply moves a shut-off VM's definition and disks.

One more term from the objectives explains how you will do this lab. Nested virtualization, in the kernel documentation's words, is the ability to run a guest inside another guest. Your lab VMs already run inside VirtualBox, UTM or KVM; to run KVM inside them, the outer hypervisor must pass its virtualization extensions through. VirtualBox documents this as nested virtualization, enabled per VM with a check box in the Processor settings or with VBoxManage modifyvm --nested-hw-virt on, and KVM hosts enable it with the kvm_intel or kvm_amd module's nested parameter. Without it, QEMU can still run guests in pure software emulation, just slowly.

Friday: retirement

The test is done. virsh shutdown web-test stops it, and virsh undefine web-test --remove-all-storage deletes its definition and disks. Nothing about a VM's retirement is irreversible until that last command, so check the name twice.

Remember: the VM's whole week used a handful of verbs: clone from a template, adjust resources, attach a network, snapshot, revert, migrate and undefine.

Common misconceptions

• "KVM and QEMU are competing hypervisors." They cooperate: KVM provides hardware acceleration in the kernel, QEMU provides the emulated machine, and libvirt manages both.
• "virsh destroy deletes a VM." It is a hard power-off. The VM's definition and disks remain until you undefine it.
• "A snapshot is a backup." It depends on the VM's existing disk and is lost with it; keep real backups elsewhere.
• "Shrinking a disk image works like growing one." Shrink the filesystems and partitions inside the guest first, then use qemu-img resize with --shrink, or lose data.
• "A cloned VM is ready to use as is." Without generalizing, clones share SSH host keys and other identifiers; prepare templates with a tool such as virt-sysprep.

What to carry forward

• KVM (kvm.ko plus kvm-intel or kvm-amd) turns Linux into a hypervisor using VT-x or AMD-V; QEMU emulates the machine; libvirt manages it through virsh and virt-manager.
• VMs trade a little performance for fast provisioning, density and easy recovery compared with bare metal.
• Templates are generalized golden images; clones copy them or overlay them with qcow2 backing files.
• qemu-img info shows image properties, convert changes formats, and resize grows images; shrinking needs in-guest shrinking first and --shrink.
• VirtIO paravirtualized devices are faster than emulated hardware; their disks appear as /dev/vda.
• Networks: NAT (default virbr0), bridged (on the LAN), isolated or host-only (no forwarding), routed (no NAT) and open (no libvirt firewall rules).
• States: running, idle, paused, in shutdown, shut off, crashed, pmsuspended. Snapshots roll back; live migration moves running VMs; nested virtualization runs hypervisors inside guests.

Sources

1. KVM project. (n.d.). Kernel Virtual Machine [Main page]. linux-kvm.org
2. QEMU project. (n.d.). About QEMU and QEMU disk image utility (qemu-img). qemu.org (about); qemu.org (qemu-img)
3. libvirt project. (n.d.). Network XML format and virsh. libvirt.org (networks); libvirt.org (virsh)
4. virt-manager project. (n.d.). Virtual Machine Manager. virt-manager.org
5. OASIS. (n.d.). Virtual I/O Device (VIRTIO) Version 1.3. docs.oasis-open.org
6. The Linux kernel documentation. (n.d.). Running nested guests with KVM. docs.kernel.org
7. libguestfs project. (n.d.). virt-sysprep(1). libguestfs.org
8. Oracle. (n.d.). Oracle VirtualBox user guide: Advanced topics [Nested virtualization]. virtualbox.org

A web server in four seconds

On your AlmaLinux machine, install Podman with sudo dnf install podman and then, as your ordinary user, type podman run -d --name web -p 8080:80 docker.io/library/nginx. A few seconds later curl localhost:8080 returns nginx's welcome page. You did not install nginx, edit a configuration file, open a firewall port on the host or become root. A container did all of that inside its own small world.

This lesson walks through objective 2.6 the way you will meet it at work: run something, change it, build your own image, give it storage and a network, then clean up. Partway through, one changed input, SELinux, breaks a step that works on Ubuntu, and you will see why.

Step 0: who actually runs a container?

A container is an ordinary Linux process that the kernel isolates with namespaces, which give it its own view of process IDs, mounts and the network, and limits with control groups. Several programs cooperate to set that up, and the objectives name four.

OCI, the Open Container Initiative, is the standard that makes these interchangeable: an image built by one tool runs under the others. Podman's commands deliberately mirror Docker's, so podman run and docker run take the same options for everything in this lesson. AlmaLinux ships Podman; on Ubuntu you can install either.

In short: Docker and Podman are the tools you type; containerd manages containers for Docker and Kubernetes; runC does the final work of starting the isolated process.

Step 1: images, tags and layers

An image is a read-only template for a container. Its full name has up to four parts, registry/namespace/name:tag, as in docker.io/library/nginx:stable. The tag names a version; leave it off and you get latest, which is a convention, not a promise of newness. In production, pin a specific tag so a rebuild does not silently change versions.

• podman pull docker.io/library/python:3.13-slim downloads an image.
• podman images lists local images; podman tag python:3.13-slim mypython:test adds a second name to the same image.
• podman image inspect prints its metadata, and podman history lists its layers.

Docker's documentation explains that each layer in an image contains a set of filesystem changes: additions, deletions or modifications. Layers are stacked, shared between images that use the same base, and cached, so pulling a second image built on the same base downloads only the new layers. A running container adds one thin writable layer on top. On Linux the stacking is done by the overlay filesystem, which Docker's storage documentation describes as its overlay2 driver and Podman uses as well.

Pulling public images needs no account. Docker Hub does limit how many pulls anonymous users can make, and creating an account there, which you need only to push your own images, comes with its own terms and age rules; read them first, with a parent or guardian if you are under 18.

Step 2: running, watching and entering containers

Port mapping is written host first, then container: -p 8080:80. Why 8080 and not 80? A rootless container runs with your own privileges, and the kernel's documentation says ports below ip_unprivileged_port_start, 1024 by default, require root or the CAP_NET_BIND_SERVICE capability. So a rootless container publishes on a high port, and a reverse proxy or firewall rule can forward 80 to it.

Containers are disposable. Delete the web container and recreate it and anything written inside it is gone, which is the point of the next step.

Step 3: building your own image

Make a directory with a site folder holding an index.html, and next to it a file named Containerfile (Podman's name) or Dockerfile (Docker's; Podman reads both):

FROM docker.io/library/python:3.13-slim
WORKDIR /srv
COPY site/ /srv/
RUN useradd --uid 10001 web
USER web
EXPOSE 8000
ENTRYPOINT ["python3", "-m", "http.server"]
CMD ["8000"]

Build and run it with podman build -t mysite:1.0 . and podman run -d --name site -p 8081:8000 mysite:1.0. Each instruction in the Dockerfile reference has a precise meaning.

• FROM initializes a new build stage and sets the base image; the reference says a valid Dockerfile must start with it.
• USER sets the user, and optionally group, for the rest of the stage, including the ENTRYPOINT and CMD at runtime. Here the server runs as an unprivileged user inside the container instead of root.
• ENTRYPOINT configures the container to run as an executable: here, Python's built-in web server.
• CMD provides defaults for an executing container. When an ENTRYPOINT exists, CMD supplies its default arguments. The reference adds that only the last CMD in a file takes effect.

Watch the difference at run time. podman run mysite:1.0 9000 replaces the CMD, so the server listens on port 9000; the ENTRYPOINT stays. Replacing the ENTRYPOINT needs the explicit --entrypoint option. Each instruction also adds a layer, and podman history shows them; put instructions that change rarely near the top so rebuilds reuse the cache.

Step 4: storage that outlives the container, and a changed input

Data that must survive goes in a volume. A named volume is managed by the container tool: podman volume create webdata, then podman run -d -v webdata:/usr/share/nginx/html -p 8080:80 nginx. Delete and recreate the container with the same volume and the files are still there. A bind mount maps a host directory instead: -v ~/site:/usr/share/nginx/html.

Now change one input. On Ubuntu that bind mount works. On AlmaLinux, with SELinux enforcing, nginx answers 403 Forbidden and its log shows permission denied. Nothing is wrong with the file permissions; the files in your home directory carry an SELinux label that container processes are not allowed to read. Lesson 22 explains labels properly. Podman's manual gives the fix: add :Z to the volume, as in -v ~/site:/usr/share/nginx/html:Z, which tells Podman to relabel the content with a private label that only this container can use. Lowercase :z applies a shared label instead, for content that several containers must read and write.

The objectives also list overlay volume operations. Podman's :O flag mounts a host directory through the overlay filesystem: the container can modify what it sees, but the changes are stored separately in container storage and the host's files stay untouched. That suits giving a build job a source tree it may scribble on.

In short: named volumes persist data across container lifetimes; bind mounts share host directories, and on SELinux systems need :Z or :z; :O gives a container a throwaway writable view of host files.

Step 5: container networks

Containers on the default network reach the outside world through the host. For containers that must find each other, create a network: podman network create labnet, then start both containers with --network labnet. Podman's documentation says its DNS plugin, enabled on bridge networks, performs container-to-container name resolution, so one container can reach another simply by its name. The network types in the objectives come from the drivers.

Podman's documentation notes that when running rootless, the macvlan and ipvlan drivers have no access to the host's interfaces, because rootless networking uses a separate network namespace.

Step 6: privilege, and cleaning up

Podman's manual states that containers are unprivileged by default: they get limited access to devices and cannot modify parts of the operating system. The --privileged option lifts those restrictions and gives the container the same access to devices as the user who launched it. Run as root, that is very nearly the whole host. Treat it as a last resort, and prefer adding the one capability or device a container needs. Two more layers of restraint stack on top: rootless Podman means even a container's root is mapped to your ordinary user on the host, and the USER instruction keeps the application itself from running as root inside the container.

Finally, clean up. Containers, images, volumes and networks accumulate, and each has a prune command that removes the unused ones:

• podman container prune: stopped containers.
• podman image prune: dangling images, the untagged leftovers of rebuilds; add -a for every image no container uses.
• podman volume prune and podman network prune: unused volumes and networks. A pruned volume's data is gone, so check first.
• podman system prune: several of these at once.

Common misconceptions

• "A container is a lightweight virtual machine with its own kernel." Containers share the host's kernel; they are isolated processes, which is why they start in seconds.
• "Files written inside a container are kept." They live in the container's writable layer and disappear with it; use a volume for anything that must persist.
• "The latest tag is always the newest version." latest is only the default tag name; pin explicit tags for repeatable deployments.
• "Permission denied on a bind mount means wrong file permissions." On SELinux systems it is often a label problem, fixed with :Z or :z.
• "--privileged just lets a container bind low ports." It removes most of the isolation between container and host; grant the specific capability needed instead.

What to remember

• runC starts OCI containers; containerd manages them for Docker and Kubernetes; Docker uses a daemon; Podman is daemonless and rootless by default with Docker-compatible commands.
• Images are named registry/namespace/name:tag and built from layers of filesystem changes stacked with overlayfs.
• run -d -p host:container -e VAR=value; ps, logs, exec -it, inspect, stop, start and rm manage containers.
• Dockerfile: FROM sets the base, USER the runtime user, ENTRYPOINT the executable, CMD its default arguments (replaced by arguments to run).
• Volumes persist data; bind mounts need :Z or :z under SELinux; :O gives an overlay that leaves host files untouched.
• Network types: bridge (default), host, none, macvlan, ipvlan and overlay; user-defined bridge networks resolve container names.
• Prefer unprivileged, rootless containers; prune containers, images, volumes and networks you no longer need.

Sources

1. Podman project. (n.d.). What is Podman?; podman-run(1); podman-network-create(1). docs.podman.io; podman-run; podman-network-create
2. Docker Inc. (n.d.). Dockerfile reference. Docker Docs. docs.docker.com
3. Docker Inc. (n.d.). Network drivers and Understanding the image layers. Docker Docs. docs.docker.com (networks); docs.docker.com (layers)
4. Docker Inc. (n.d.). OverlayFS storage driver and Docker Hub usage and limits. Docker Docs. docs.docker.com (overlayfs); docs.docker.com (Hub limits)
5. Open Container Initiative. (n.d.). runc. GitHub. github.com; containerd project. (n.d.). containerd. containerd.io
6. The Linux kernel documentation. (n.d.). IP sysctl: ip_unprivileged_port_start. docs.kernel.org

Five findings on one file server

An internal audit of a small company's Linux file server produced five findings. A contractor who left in June still has a working account. Three administrators share one login and work as root all day. A service account that runs the backup job has an interactive shell. Passwords must change every 30 days and include a symbol, yet two users chose "Summer2026!" and nothing caught it. And nobody uses a second factor to log in.

Each finding maps to part of objective 2.2, local account management, or objective 3.4, account hardening. This lesson works through them in order, starting with the files that every account tool edits.

The three account files

Local accounts live in three text files. Each line of each file is one record, with fields separated by colons.

The x in the second field of passwd means "the password is in shadow." A hash in shadow that begins with ! marks a locked password, which is how the lock commands below work.

Every account has a numeric UID and a primary group with a GID; the kernel tracks the numbers, and names are only labels. Accounts fall into three kinds by role. User accounts belong to people. System accounts are created by the operating system and its packages. Service accounts exist to run a particular daemon or job, such as a web server or a backup agent, so that a compromise of the service does not hand over a person's or root's privileges. The ranges come from /etc/login.defs, whose manual gives the defaults: regular users from UID_MIN 1000 to UID_MAX 60000, and system accounts from 101 to UID_MIN minus one. UID 0 is root, whatever the account is called.

One more pair of terms. A process has a real user ID, who started it, and an effective user ID, whose privileges it is using; the same goes for group IDs. The credentials manual says the effective IDs are what the kernel uses to decide permissions. They differ when you run a program as someone else, through sudo or a setuid program, which Lessons 15 and 21 cover. id prints your UID, GID and groups, and whoami prints the effective user name.

Finding 1: the contractor who never left

Creating an account properly means a home directory, a shell, a comment and, for a contractor, an end date. On AlmaLinux:

• sudo useradd -m -s /bin/bash -c "Dana Ruiz, contractor" -e 2026-12-31 dana creates the account with a home directory (-m), shell, comment and expiry date.
• sudo passwd dana sets an initial password.
• sudo groupadd webteam creates a group and sudo usermod -aG webteam dana adds her to it.

The -a in the last command is easy to forget and costly. The usermod manual says -a adds the user to the supplementary groups listed with -G; without it, -G replaces the user's whole supplementary group list, silently removing her from every other group.

On Debian and Ubuntu you will also meet adduser, which its manual describes as a more Debian-specific front end to useradd, groupadd and usermod, driven by /etc/adduser.conf. It prompts for the password and details and creates the home directory by default. Its partner for removal is deluser. On the Red Hat family, adduser is simply another name for useradd.

Where does a new home directory's content come from? useradd copies the files in /etc/skel, the skeleton directory, which is why every user starts with the same .bashrc and .bash_profile. System-wide login settings for everyone live in /etc/profile and /etc/profile.d/, the files a login shell reads first, as Lesson 3 showed.

The contractor's account should have expired on her last day. Fixing it safely is a sequence, not a single delete:

1. Lock it at once: sudo usermod -L dana or sudo passwd -l dana puts a ! before the hash. A lock blocks password logins but not, for example, an SSH key, so also expire it.
2. Expire it: sudo chage -E 0 dana or sudo usermod -e 1 dana sets an expiry date in the past, which stops every kind of login.
3. Find what she owns: sudo find / -user dana 2>/dev/null, plus any running processes, cron jobs and files in shared directories.
4. Delete when the data is dealt with: sudo userdel -r dana removes the account and its home directory and mail spool; on Debian, sudo deluser --remove-home dana.

Groups have matching commands: groupmod -n renames a group and groupdel deletes one. To change a login shell, chsh -s /bin/bash dana.

The upshot: create accounts with expiry dates when you know the end date, add groups with -aG, and retire accounts by locking, expiring, finding their files and only then deleting.

Who is here, and who was?

Several tools list accounts and sessions.

The last row is changing under your feet. Debian 13's release notes explain that util-linux no longer provides last or lastb and the login package no longer provides lastlog, because the files they read cannot store login times after 2038. The suggested replacements are wtmpdb for last, lastlog2 for lastlog, and lslogins, which shows when accounts were last used; lslogins --failed covers what lastb did. Learn last and lastlog for the exam, and expect to find the replacements on new systems.

Finding 2: everyone is root

Three administrators sharing one root login means no record of who did what, one password shared three ways, and every typing mistake made with unlimited privilege. The fix is personal accounts for each administrator, root logins disabled, and privileges borrowed only for the commands that need them with sudo, which logs each use. Lesson 21 sets that up, including blocking direct root login over SSH. The principle on the exam's list is plain: avoid running as root.

Finding 3: a service account that can log in

The backup account needs to own files and run a job. It does not need a person typing at a shell. Give it a shell that refuses: sudo usermod -s /sbin/nologin backupsvc. The nologin manual says it displays a message that the account is not available and exits, and is intended as a replacement shell to deny login access. Create such accounts as system accounts from the start: sudo useradd -r -s /sbin/nologin backupsvc.

Between full shell and no shell sits the restricted shell, /bin/rbash: Bash started under that name. Bash's manual lists what it disallows, including changing directory with cd, changing PATH or SHELL, running commands whose names contain a slash, and redirecting output. It suits a kiosk-style account that may run a few approved commands. It is not a security boundary on its own, because any allowed program that can start a shell, such as an editor, lets the user escape.

Finding 4: the password policy that let Summer2026! through

Objective 3.4 lists five password properties: complexity, length, expiration, reuse and history. Linux can enforce each, and current guidance disagrees with older habits about two of them.

Defaults for new accounts come from login.defs, and its manual carries a warning worth remembering: PASS_MAX_DAYS, PASS_MIN_DAYS and PASS_WARN_AGE are only used when an account is created; changing them does not affect existing accounts. To change existing accounts, use chage.

Now the dispute. The audit's policy forced changes every 30 days and demanded a symbol, which is exactly what produces Summer2026! followed by Autumn2026!. NIST's digital identity guidelines, SP 800-63B Revision 4, take the opposite line. They require passwords used as a single factor to be at least 15 characters, allow at least 64, and forbid other composition rules such as required character mixes. They forbid periodic forced changes, but require a change when there is evidence of compromise. And they require new passwords to be checked against a blocklist of commonly used, expected or compromised values, including passwords from previous breaches. That last rule is the objective's "checking existing breach lists," and it is what would have rejected Summer2026!.

So what will the exam want? Know all five properties and the tools that enforce them, and know the modern direction: long passwords, screened against breach lists, changed when compromised rather than on a calendar, backed by a second factor.

The upshot: chage manages aging per account, login.defs sets defaults only for new accounts, PAM modules enforce quality, history and lockout, and current guidance favors length and breach checks over forced complexity and rotation.

Finding 5: one factor is not enough

Multifactor authentication (MFA) requires proof from at least two different kinds of factor: something you know, such as a password; something you have, such as a phone generating time-based one-time passwords or a hardware security key; and something you are, such as a fingerprint. A stolen password alone then gets an attacker nowhere. On Linux, MFA is added through PAM modules for logins and sudo, and through combining SSH keys with a second factor for remote access. Lesson 20 shows how PAM stacks modules to make that possible.

Common misconceptions

• "usermod -G webteam dana adds Dana to webteam." Without -a it replaces all her supplementary groups; use -aG.
• "Locking a password locks the account." A lock blocks password authentication only; an SSH key may still work. Expire the account as well.
• "Changing PASS_MAX_DAYS in login.defs updates every user." It applies only to accounts created afterwards; use chage for existing ones.
• "Frequent forced changes and symbol rules make passwords strong." NIST now forbids both and requires length and breach-list checks instead.
• "A restricted shell makes an account safe." Any allowed program that can spawn a shell defeats rbash; for service accounts use /sbin/nologin.

The short version

• /etc/passwd holds account details, /etc/shadow the hashes and aging fields (root only), /etc/group the groups and members.
• UID 0 is root; regular users start at 1000 by default; system and service accounts sit below, often with /sbin/nologin.
• useradd -m creates users from /etc/skel; adduser and deluser are Debian's friendlier front ends; usermod -aG adds groups safely.
• Retire accounts by locking (usermod -L, passwd -l), expiring (chage -E 0), finding their files, then userdel -r.
• getent passwd, id, groups, whoami, who, w, last and lastlog list accounts and sessions; Debian 13 replaces last and lastlog with wtmpdb, lastlog2 and lslogins.
• chage sets aging per account; pam_pwquality, pam_pwhistory and pam_faillock (successor to the removed pam_tally2) enforce quality, history and lockout.
• Avoid working as root, add MFA, and screen passwords against breach lists.

Sources

1. shadow-utils project. (n.d.). passwd(5), shadow(5), login.defs(5), useradd(8), usermod(8) and chage(1). Linux man-pages project. passwd; shadow; login.defs; useradd; usermod; chage
2. Debian. (n.d.). adduser(8). Debian manual pages. manpages.debian.org
3. Linux man-pages project. (n.d.). credentials(7) and nologin(8). credentials; nologin
4. Fox, B., and Ramey, C. (n.d.). bash(1) [Section: Restricted shell]. Linux man-pages project. man7.org
5. Debian. (2025). Release notes for Debian 13 (trixie): Issues to be aware of [The last, lastb and lastlog commands have been replaced]. debian.org
6. Linux-PAM project. (n.d.). NEWS [Release 1.5.0: removal of pam_tally and pam_tally2]. GitHub. github.com; pam_faillock(8). man7.org
7. Temoshok, D., and others. (2025). Digital identity guidelines: Authentication and authenticator management (NIST Special Publication 800-63B-4). National Institute of Standards and Technology. pages.nist.gov

The ticket and the tempting fix

At 09:12 on a Monday, a ticket arrives from the web team: "I get Permission denied when I try to edit Ana's files in /srv/projects. Can someone fix it?" A colleague replies within a minute: "Just run chmod -R 777 /srv/projects. Then everyone can write everything."

That fix would make the error disappear. It would also let every account on the server, including the web server's own service account and any attacker who compromises it, read, change and delete every project file. It hides the cause instead of finding it. This lesson debugs the ticket properly. Along the way it covers the permission part of objective 3.3: modes, ownership, umask, the special bits, ACLs and file attributes.

Reading what ls -l says

Start with evidence. ls -l /srv/projects/site/index.html prints a line beginning -rw-r--r-- 1 ana ana. The first character is the type (a dash for a regular file, d for a directory, l for a link). The next nine characters are three sets of three: permissions for the owning user, the owning group and everyone else. Then come the owner, ana, and the group, also ana.

The directory column surprises people. Deleting a file is an operation on its directory, not on the file, so a user with write and execute permission on a directory can delete a file there even if the file itself is read-only to them. And without x on a directory, nothing below it is reachable, whatever the files' own permissions say; the path resolution manual returns "Permission denied" when a process lacks search permission on any directory along the path.

The rule the kernel actually applies

Here is the reasoning error behind many permission tickets. People assume the kernel checks all three sets and grants the most generous. It does not. The path resolution manual describes the rule: the first set is used when the process's effective user ID equals the file's owner; otherwise the second set is used when the file's group is the process's effective group or one of its supplementary groups; when neither holds, the third set is used. Exactly one set applies, chosen in that order.

Apply it to the ticket. The user, Raj, is not ana, so the owner set does not apply. The file's group is ana, Ana's personal group, and Raj is not in it, so the group set does not apply either. The "other" set, r--, applies: Raj can read but not write. The rule also produces a famous oddity: a file with mode ---rw---- is unreadable by its own owner, even if the owner is in the file's group, because the owner set matched first.

So what?: the kernel picks one class, owner, else group, else other, and uses only that class's bits. To grant a team access, the file's group must be the team's group and the group bits must allow it.

Changing modes and ownership

chmod sets permissions in two notations.

• Octal adds the values r=4, w=2, x=1 for each class: chmod 640 report.txt gives the owner rw (6), the group r (4) and others nothing (0). Common modes: 644 for ordinary files, 755 for programs and directories, 600 for private keys, 750 or 770 for team directories.
• Symbolic changes named bits for named classes, u (user), g (group), o (other) or a (all): chmod g+w file, chmod o-rwx dir, chmod u=rw,go=r file. Symbolic form is safer when you only want to change one bit, because it leaves the others alone.

chown ana:webteam file changes the owner and group together; chgrp webteam file changes only the group; both take -R for a whole tree. Only root can give a file to another owner. An owner may change the group only to a group they belong to.

Cause 1: the umask made every new file group-read-only

Ana's files are 644 because of her umask, a per-process mask of permission bits to remove from newly created files. The umask manual gives the typical default, 022, and works the example: a program asks for mode 666 when creating a file, and 666 with 022 masked out leaves 644, rw-r--r--. Directories start from 777 and become 755. Print yours with umask. For a team that shares files, a umask of 002 leaves group write in place, producing 664 files and 775 directories. Set it in the team members' profiles, or use default ACLs, below, which override it inside one directory.

Cause 2: new files get the wrong group, fixed with setgid

Even with group write, Ana's new files belong to her personal group, ana, not webteam. Normally a new file takes the creator's primary group. The inode manual describes the setgid bit's special use on a directory: files created there inherit their group ID from the directory, and new subdirectories inherit the setgid bit too. So:

• sudo chgrp -R webteam /srv/projects
• sudo chmod 2775 /srv/projects (the leading 2 is setgid), which ls shows as drwxrwsr-x

From now on every file created there belongs to webteam, and with a 002 umask the whole team can edit it.

Cause 3 waiting to happen: anyone can delete anyone's file

Group write on the directory lets every team member delete every file in it, including each other's. That is the job of the sticky bit. The inode manual states that on a directory it means a file can be renamed or deleted only by the file's owner, the directory's owner or a privileged process. /tmp is the classic case: mode 1777, everyone may create files, nobody may delete anyone else's. Adding it to the project directory makes it 3775.

A capital S or T in ls output means the special bit is set but the matching execute bit is not, which is usually a mistake worth investigating.

When modes are not enough: ACLs

Suppose an auditor, Mei, who is not in webteam, needs read access to just one subdirectory. Owner, group and other cannot express that without changing group membership. Access control lists can. The acl manual describes ACLs as a superset of the permission bits, adding entries for named users and named groups.

• setfacl -m u:mei:rx /srv/projects/reports grants Mei read and enter on the directory.
• setfacl -d -m g:webteam:rwx /srv/projects sets a default ACL; the setfacl manual says -d makes the operation apply to the default ACL, which new files and directories inside inherit. This overrides a restrictive umask within that directory.
• getfacl /srv/projects/reports shows every entry; ls -l marks a file that has an ACL with a + after the mode.
• setfacl -x u:mei file removes one entry; setfacl -b file removes them all.

One subtlety causes real outages. When named entries exist, the acl manual says the mask entry denotes the maximum access that named users, named groups and the owning group can receive. On such a file, ls shows the mask in the group position, and running chmod g-w lowers the mask, quietly removing write from every named entry too. getfacl shows the result as "#effective:" comments.

So what?: a team directory needs a team group, setgid so new files inherit it, a umask or default ACL that keeps group write, and the sticky bit so members cannot delete each other's work. ACLs handle the exceptions.

A second ticket: root cannot delete a file

Weeks later another puzzle arrives. The administrator, working as root, runs rm /etc/resolv.conf.custom and gets "Operation not permitted." Root normally ignores permission bits, so modes cannot be the cause. The tool for this layer is lsattr, which lists filesystem attributes, and it shows an i.

The chattr manual explains the immutable attribute: such a file cannot be modified, deleted or renamed, cannot be linked to, cannot be opened for writing, and most of its metadata cannot change; only the superuser or a process with CAP_LINUX_IMMUTABLE can set or clear it. sudo chattr -i file removes it. Its sibling is append only, chattr +a, which the manual says lets the file be opened for writing only in append mode, which suits log files that must never be rewritten. Administrators set +i to protect a critical configuration from accidental change; attackers sometimes set it to protect their own planted files from removal, which is why lsattr belongs in an investigator's toolkit.

Auditing setuid programs

Each setuid-root program is a door to root privileges, so a hardened system keeps as few as possible. List them all with sudo find / -perm -4000 -type f 2>/dev/null (use -2000 for setgid). Compare the list with a known-good baseline or with your package database; anything unexpected, especially in home directories, /tmp or a web root, needs explaining. Remove the bit from programs that do not need it, such as an old tool nobody uses, with sudo chmod u-s /path/to/program, and mount user-writable filesystems with nosuid, as Lesson 9 described. Lesson 25 shows how package verification detects permissions that have changed from what the package installed.

Common misconceptions

• "chmod 777 fixes permission problems." It removes all protection; find the class that applies and fix group ownership, group bits, umask or ACLs instead.
• "The kernel grants the most generous of owner, group and other." It uses exactly one class, owner first, then group, then other.
• "A read-only file cannot be deleted." Deletion needs write and execute on the directory; the sticky bit is what protects other people's files.
• "Root can always delete any file." The immutable attribute stops even root until chattr -i removes it.
• "chmod g-w on a file with an ACL only affects the owning group." It lowers the ACL mask, limiting every named user and group entry too.

Where this leaves us

• r, w and x mean read, change and run on files; list, add or remove entries, and enter on directories.
• The kernel applies one class: owner if the effective UID matches, else group if any of the process's groups matches, else other.
• chmod takes octal (640, 2775) or symbolic (g+w, o-rwx) modes; chown sets owner and group, chgrp the group.
• umask removes bits from new files: 022 gives 644 files, 002 gives 664.
• setuid runs a program as its owner; setgid on a directory makes new files inherit its group; the sticky bit stops users deleting others' files.
• setfacl -m adds ACL entries, -d sets defaults inherited by new files, getfacl shows them, and the mask caps named entries.
• chattr +i makes a file immutable even to root, +a append-only; lsattr reveals them.
• Audit setuid files with find / -perm -4000 and remove the bit where it is not needed.

Sources

1. Linux man-pages project. (n.d.). path_resolution(7) [Section: Permissions] and inode(7) [The file type and mode]. man7.org (path_resolution); man7.org (inode)
2. Linux man-pages project. (n.d.). umask(2). man7.org
3. GNU coreutils. (n.d.). chmod(1), chown(1) and chgrp(1). Linux man-pages project. chmod; chown; chgrp
4. Gruenbacher, A. (n.d.). acl(5), setfacl(1) and getfacl(1). Linux man-pages project. acl; setfacl; getfacl
5. e2fsprogs project. (n.d.). chattr(1) and lsattr(1). Linux man-pages project. chattr; lsattr

Thursday, 14:05: a slow build server

At 14:05 on a Thursday, developers report that the AlmaLinux build server, a machine with two processors, has become sluggish. Typing over SSH lags. The w command, from Lesson 14, shows a load average of 7.8, far more runnable work than two processors can serve. Somewhere a process is misbehaving. This lesson follows the administrator from that first symptom to a permanent fix, and the tools she reaches for along the way make up objective 2.3.

14:06: finding the culprit

Her first stop is top, which redraws a list of processes sorted by CPU use every few seconds. At the head of the list sits a process called report-gen, owned by user dev3, using 99 percent of a CPU, with several copies of itself below it. She could also have used htop, which its project calls an interactive process viewer, with color, scrolling and mouse support, or atop, whose manual describes an interactive monitor of the load on CPUs, memory, disks and network interfaces that can also write raw logs for later analysis, so you can see what happened at 3 a.m.

For a snapshot she can paste into a ticket, she uses ps:

• ps aux lists every process with its user, CPU and memory share, state and command.
• ps -ef lists every process in a format that includes the PPID, the parent process ID.
• ps -eo pid,ppid,stat,ni,pcpu,cmd --sort=-pcpu | head chooses exactly the columns she wants, busiest first.

Every process has a PID, its process ID, and a PPID, the PID of the process that started it. pstree -p draws those relationships as a tree, and it shows something useful: all the report-gen processes descend from one cron job started at 14:00, which itself descends from crond.

Two sysstat tools add numbers over time. mpstat 2 5 reports each processor's activity every two seconds, five times, which confirms both CPUs are saturated in user time. pidstat -p 4211 2 reports one process's activity at the same interval.

14:09: what is it doing? /proc, lsof and strace

Every running process has a directory under /proc named by its PID, a view generated by the kernel, as Lesson 5 explained. cat /proc/4211/status shows its state, parent, user IDs and memory; /proc/4211/cmdline holds its full command line; ls -l /proc/4211/fd lists every file descriptor it has open.

The friendlier view of open files is lsof -p 4211: the report generator has the same 2 GB export file open many times over. Then strace -p 4211, whose manual says it intercepts and records the system calls a process makes and the signals it receives, shows it reading that file in an endless loop. The program has a bug. That diagnosis belongs to the developers; the server's health belongs to her.

Process states, including the undead

The STAT column in ps tells you what each process is doing. The ps manual lists the codes; these five match the objectives.

Two of these confuse people. A process in D state cannot be killed until its I/O returns, so many D processes usually mean a storage or network-filesystem problem, not a CPU problem. A zombie has already exited; only its entry in the process table remains, waiting for its parent to collect the exit status. Sending a zombie a signal does nothing. If zombies pile up, the bug is in the parent: fix it or end it, and the orphaned zombies are adopted and reaped by PID 1.

Worth holding on to: R runs, S waits for events, D waits on I/O and resists signals, T is stopped, and Z is a dead process whose parent has not collected it.

14:12: lowering its priority

Before stopping anything, she buys the other users some breathing room. Linux schedules ordinary processes using a niceness value; the nice manual says niceness ranges from -20, most favorable to the process, to 19, least favorable. Higher means nicer to everyone else.

• nice -n 10 ./long-job starts a command with niceness 10.
• sudo renice -n 15 -p 4211 changes a running process. In top, pressing r does the same.

The renice manual adds a rule worth remembering: users other than the superuser may change only their own processes and may only increase the nice value, lowering the urgency; only root can make a process less nice. The load stays high, but interactive work feels normal again.

14:15: signals

To stop a process you send it a signal, a small numbered message that the kernel delivers. Three are in the objectives by number.

The signal manual makes the key distinction: SIGKILL and SIGSTOP cannot be caught, blocked or ignored. A process can handle SIGTERM by saving its work and removing its lock files; it cannot handle SIGKILL at all. So the order is always TERM first, wait a few seconds, then KILL only if needed.

• kill 4211 sends SIGTERM to one PID; kill -9 4211 or kill -KILL 4211 sends SIGKILL; kill -HUP 812 asks a daemon to reload.
• killall report-gen signals every process running that command name; its manual says SIGTERM is sent if no signal is named.
• pkill -u dev3 report matches by pattern and other criteria, here the user; its companion pgrep lists the matching PIDs first, a sensible dry run.

She runs pgrep -a -u dev3 report, checks the list, then pkill -u dev3 report-gen. The processes exit, and the load average falls over the next few minutes.

Jobs in your own shell

The report generator ran under cron, but developers often start long jobs from a terminal, and the shell gives them job control.

Why does nohup matter? Bash's manual explains that when an interactive shell receives SIGHUP, for example because your SSH connection drops, it resends SIGHUP to all its jobs, running or stopped. A long job started with a plain & dies with your session. The nohup manual says it runs the command ignoring hangup signals and, if output would go to the terminal, appends it to nohup.out instead. The exec builtin is different again: the manual says it replaces the shell without creating a new process, so when the command finishes, the session ends. Scripts use it to hand control to a final program.

Limits: stopping the next runaway

One user should not be able to swamp a shared server. Per-process resource limits cap what a process may use. ulimit -a shows the limits of your shell, such as open files (-n) and user processes (-u). Persistent limits for logins come from /etc/security/limits.conf, applied by the pam_limits module; its manual lists items such as nofile, the maximum number of open file descriptors, and nproc, the maximum number of processes, each with a soft limit a user may raise up to the hard limit. A line such as @developers hard nproc 200 would have stopped the report generator from forking without end. Services started by systemd take their limits from the unit file instead, as Lesson 18 shows.

The permanent fix: scheduling it properly

The report must still run, just not at 14:00 on a busy server. Linux offers three schedulers in the objectives, and choosing between them is a favorite exam question.

A crontab line has five time fields and a command. The crontab manual lists them: minute (0-59), hour (0-23), day of month (1-31), month (1-12) and day of week (0-7, where both 0 and 7 mean Sunday). An asterisk means every value. So the developers' new line reads:

30 2 * * 1-5 nice -n 10 /opt/reports/report-gen --nightly

That runs at 02:30, Monday to Friday, at low priority. System-wide jobs go in /etc/crontab or files in /etc/cron.d/, which take an extra field naming the user to run as.

For the laptop-like machines in the office, cron has a weakness: a job scheduled for 02:30 never runs if the machine is off at 02:30. The anacron manual says it executes commands periodically with a frequency specified in days and, unlike cron, does not assume the machine runs continuously, so a missed daily job runs soon after the next boot. Lesson 18 adds systemd timers, which can do both.

Worth holding on to: cron for repeating schedules on always-on servers, at for one-off jobs, anacron for daily, weekly or monthly jobs on machines that may be off.

Common misconceptions

• "kill -9 is the normal way to stop a process." SIGKILL cannot be caught, so the program cannot clean up; send SIGTERM first.
• "A zombie can be killed with kill -9." It has already exited; its parent must reap it, or the parent must end so PID 1 adopts and reaps it.
• "Any user can raise their own job's priority." Unprivileged users can only make processes nicer, from 0 toward 19; lowering niceness needs root or a nice limit.
• "Ctrl+D stops a program like Ctrl+C." Ctrl+D sends end of input, not a signal; Ctrl+C sends SIGINT.
• "A background job keeps running after I log out." The shell forwards SIGHUP to its jobs; use nohup, or run it as a service or scheduled job.

Putting it together

• top, htop and atop watch live; ps aux and ps -ef snapshot; pstree shows parentage by PID and PPID; mpstat reports per-CPU and pidstat per-process activity.
• /proc/PID, lsof -p and strace -p show a process's details, open files and system calls.
• States: R running, S sleeping, D blocked on I/O, T stopped, Z zombie awaiting its parent.
• Niceness runs from -20 (most favored) to 19; nice starts, renice changes; only root can lower it.
• SIGHUP 1 often reloads, SIGTERM 15 asks politely, SIGKILL 9 cannot be caught; kill targets PIDs, killall names, pkill patterns.
• &, jobs, bg, fg and Ctrl+Z manage jobs; Ctrl+C interrupts; Ctrl+D ends input; nohup survives logout; exec replaces the shell.
• limits.conf and ulimit cap processes and open files per user.
• crontab fields: minute, hour, day of month, month, day of week; at runs once; anacron catches up missed periodic jobs.

Sources

1. procps-ng project. (n.d.). ps(1), pkill(1) and renice(1). Linux man-pages project. ps; pkill; renice
2. Linux man-pages project. (n.d.). signal(7). man7.org
3. GNU coreutils and psmisc. (n.d.). nice(1), nohup(1) and killall(1). Linux man-pages project. nice; nohup; killall
4. Fox, B., and Ramey, C. (n.d.). bash(1) [Sections: Job control, Signals, and the exec builtin]. Linux man-pages project. man7.org
5. strace project. (n.d.). strace(1); Linux-PAM. (n.d.). limits.conf(5). Linux man-pages project. strace; limits.conf
6. cronie project. (n.d.). crontab(5) and anacron(8). Linux man-pages project. crontab; anacron; Debian. (n.d.). at(1). manpages.debian.org
7. Godard, S. (n.d.). mpstat(1) and pidstat(1); Atoptool. (n.d.). atop(1). Debian manual pages. mpstat; pidstat; atop; htop project. (n.d.). htop: an interactive process viewer. htop.dev

One web server, two commands

Install the Apache web server on both lab machines and you type two different commands: sudo dnf install httpd on AlmaLinux and sudo apt install apache2 on Ubuntu. The package names differ, the tools differ, and the files land in different places. Yet the two systems are doing the same job. Each fetches a signed package from a trusted repository and works out what else it needs. Each records everything in a database so the software can be updated or removed cleanly later.

Objective 2.4 is built on that comparison, and so is this lesson. Learn the job once, then learn each family's words for it.

Packages, dependencies and the two layers of tools

A package is an archive of files plus metadata. The metadata holds its name, version, architecture and description, the scripts to run when it is installed or removed, and its dependencies, the other packages it needs. Each family has two layers of tools.

• The low-level tool installs, queries and removes one package file and keeps the package database: rpm on the Red Hat family, dpkg on the Debian family. It checks dependencies but does not fetch them.
• The high-level tool talks to repositories, downloads what is needed and resolves dependencies: dnf on the Red Hat family (zypper on SUSE), apt on the Debian family.

The difference shows when you install a downloaded file. The command sudo rpm -i tool.rpm stops with "Failed dependencies" if a required library is missing. The command sudo dnf install ./tool.rpm installs the same file and fetches the missing dependencies from the enabled repositories. On Ubuntu, sudo apt install ./tool.deb does the same, while dpkg -i would leave the package half-configured until the dependencies arrive.

A conflict is the opposite problem: two packages that cannot be installed together, usually because both provide the same file or the same capability. The package manager refuses rather than overwrite one package's files with another's. The fix is to choose one, not to force the install.

The same jobs, side by side

Red Hat's DNF documentation for RHEL 10 shows dnf history listing each transaction with its actions and the number of packages altered. The same command can undo a transaction, which makes a bad update reversible in one step on AlmaLinux.

The core of it: rpm and dpkg handle single packages and the database; dnf and apt add repositories and dependency resolution. Install local files through dnf or apt so their dependencies come too.

Repositories: where packages come from

A repository is a server, or a directory, holding packages plus signed metadata describing them. Each family configures them differently.

• Red Hat family: one or more .repo files in /etc/yum.repos.d/, each with sections setting a baseurl or mirror list, enabled=, gpgcheck= and gpgkey=. dnf repolist --all shows every repository and whether it is enabled. Red Hat's RHEL 10 documentation adds a repository with dnf config-manager --add-repo and turns one off or on with dnf config-manager --disable or --enable and the repository ID. For a single command, --enablerepo= borrows a disabled repository just once.
• Debian family: files in /etc/apt/sources.list.d/. The sources.list manual describes the newer deb822 format, multi-line stanzas with fields such as Types, URIs, Suites, Components and Signed-By, and notes that Ubuntu uses /etc/apt/sources.list.d/ubuntu.sources. Disable a source by removing or commenting out its file, then run apt update.

Third-party repositories add software the distribution does not ship. The one you will meet most on AlmaLinux is EPEL. AlmaLinux's documentation describes Extra Packages for Enterprise Linux as a Fedora special interest group that maintains a high-quality set of additional packages. They are usually based on their Fedora counterparts and will never conflict with or replace packages in the base distribution. EPEL is enabled with sudo dnf install epel-release. Red Hat's documentation is blunt about other sources: installing from unverified or untrusted repositories is a potential security risk. Every repository you add is someone you trust to run code as root on your server.

GPG signatures: proving where a package came from

Repositories sign their packages and metadata with a GPG key, and the package manager checks the signature before installing. A valid signature proves the package was built by the holder of the key and has not been altered since.

• On the Red Hat family, gpgcheck=1 in a .repo file requires signed packages. The key is imported into the RPM database, either automatically on first use, when dnf asks you to confirm it, or by hand with rpm --import. AlmaLinux's documentation advises comparing the offered key's fingerprint with the published one before accepting. rpm -K tool.rpm checks a downloaded file's signature.
• On the Debian family, each repository's Release file is signed, and the Signed-By option, the sources.list manual explains, requires a repository to verify against a particular set of keys rather than against every key apt trusts. Scoping keys this way means a third-party key can vouch for its own repository but not for Ubuntu's.

If a signature check fails, stop. The cause may be an expired or missing key, which you fix by importing the right key after checking its fingerprint. Or it may be a tampered package, which you must not install. Never answer a signature failure by setting gpgcheck=0.

Holding things back: exclusions and holds

Sometimes one package must not change, for example a kernel needed by a vendor's driver or a database whose upgrade needs planning.

• dnf: the excludepkgs option, in the [main] section of /etc/dnf/dnf.conf or in one repository's section, removes matching packages from every operation, as in excludepkgs=kernel*. Red Hat's documentation notes that --disableexcludes lifts the exclusion for one command.
• apt: sudo apt-mark hold postgresql-17. The apt-mark manual says hold marks a package so it will not be automatically installed, upgraded or removed; unhold cancels it and showhold lists holds.

Holds are a deliberate debt. Every held package misses its security fixes, so record why it is held and when the hold should end.

Choosing a default: update-alternatives

Several programs can do the same job: two Java runtimes, three text editors. The update-alternatives manual explains the problem and the solution. When several programs fulfilling the same function are installed, a system of symbolic links decides which one a generic name such as editor or java runs. The command update-alternatives --display editor shows the choices and the current one, and sudo update-alternatives --config java lets you pick. The Red Hat family uses the same system through the alternatives command.

Software from outside the distribution

Not everything comes from your distribution's repositories, and the objectives name four other routes.

The language managers have one trap each.

• pip. PEP 668 lets a distribution mark its Python as externally managed with a marker file. After that, pip refuses to install packages into it outside a virtual environment, because system-wide pip installs can break tools the distribution relies on. The fix is a virtual environment, covered in Lesson 28.
• cargo install builds and installs binaries into its installation root, by default $HOME/.cargo, so they appear only for that user.
• npm installs into a project's local node_modules folder unless you add -g, global mode, which installs into a shared prefix.

Sandboxed formats trade integration for isolation. Flatpak's documentation describes it as a system for building, distributing and running sandboxed desktop applications. It also lists what the sandbox forbids, such as setuid programs and unfiltered access to the host's processes. AppImage's documentation describes one app per file: a regular file containing an application and its dependencies, which you make executable and run, with nothing installed. Ubuntu's Snap format takes a similar bundled approach. They suit desktop applications and fast-moving tools far more than core server software.

Source builds come last for a reason. Projects document their steps in a README or INSTALL file; projects using the GNU build system typically run ./configure, make and sudo make install. Install such software under /usr/local, which the FHS reserves for software the administrator installs locally, so it never collides with packaged files. Remember that nothing will update it for you.

The core of it: prefer the distribution's signed repositories, then trusted third-party repositories such as EPEL, then everything else, and keep track of what you installed outside the package manager.

Common misconceptions

• "rpm -i and dnf install do the same thing." rpm installs one file and only reports missing dependencies; dnf fetches and installs them.
• "apt upgrade finds new versions on its own." On the Debian family, run apt update first to refresh the package lists.
• "Disabling gpgcheck is a harmless fix for a failed install." It removes the only proof that the package is genuine; import the correct key after checking its fingerprint instead.
• "A held package is safe to forget." It stops receiving updates, including security fixes, until the hold is removed.
• "sudo pip install is the normal way to add Python tools." Distributions that follow PEP 668 block it; use a virtual environment or the distribution's packages.

Pulling it together

• Packages carry files, metadata, scripts and dependencies; rpm and dpkg manage single packages, dnf and apt resolve dependencies from repositories.
• dnf install, upgrade, remove, search, info, provides and history; apt update, install, upgrade, remove, purge, search and show; rpm -qa, -qf, -ql and dpkg -l, -S, -L query the database.
• Repositories live in /etc/yum.repos.d/*.repo or /etc/apt/sources.list.d/; dnf config-manager enables and disables them; EPEL adds packages that never replace base ones.
• GPG signatures prove origin and integrity: gpgcheck=1 and rpm --import on the Red Hat family, signed Release files and Signed-By on the Debian family.
• excludepkgs in dnf.conf and apt-mark hold keep packages from changing; update-alternatives (or alternatives) chooses default commands.
• pip, cargo and npm install outside the system database; Flatpak and AppImage bundle applications; source builds belong in /usr/local and need manual updates.

Sources

1. Red Hat. (n.d.). Managing software with the DNF tool (RHEL 10). docs.redhat.com
2. DNF project. (n.d.). DNF configuration reference. dnf.readthedocs.io; RPM project. (n.d.). rpm(8). rpm.org
3. Debian. (n.d.). apt(8), sources.list(5), apt-mark(8) and update-alternatives(1). Debian manual pages. apt; sources.list; apt-mark; update-alternatives
4. AlmaLinux OS Foundation. (n.d.). Extra repositories: EPEL. AlmaLinux Wiki. wiki.almalinux.org
5. Thomas, G., Klose, M., Lains, F., and others. (2021). PEP 668: Marking Python base environments as "externally managed". Python Software Foundation. peps.python.org
6. The Cargo Book. (n.d.). cargo-install. doc.rust-lang.org; npm Docs. (n.d.). npm install. docs.npmjs.com
7. Flatpak project. (n.d.). Introduction to Flatpak. docs.flatpak.org; AppImage project. (n.d.). Concepts. docs.appimage.org
8. LSB Workgroup, The Linux Foundation. (2015). Filesystem Hierarchy Standard (Version 3.0) [/usr/local]. refspecs.linuxfoundation.org

Two small files instead of a crontab line

In Lesson 16 the developers' report job ended up as one crontab line. This lesson moves it again, into two small text files that systemd reads, report.service and report.timer. On the way it builds a service for a small web server. As Lesson 2 showed, systemd is the first process on both lab machines. It starts, stops, watches and logs almost everything else, and objective 2.5 asks you to manage it.

Units and where they live

systemd manages units, each described by a text file whose suffix names its type. The objectives list four types.

The systemd.unit manual lists the search path. Two directories matter in practice: /usr/lib/systemd/system/, where packages install their unit files, and /etc/systemd/system/, which is searched earlier and so wins. Units you write, and changes you make, belong in /etc. Edit a file under /usr/lib and the next package update silently replaces your work.

Worked example 1: a service for a small web server

On AlmaLinux, create a service account and some content: sudo useradd -r -s /sbin/nologin sitesvc, then a directory /srv/site holding an index.html. Now write /etc/systemd/system/site.service:

[Unit]
Description=Course demo web server
After=network-online.target
Wants=network-online.target

[Service]
User=sitesvc
WorkingDirectory=/srv/site
ExecStart=/usr/bin/python3 -m http.server 8000
Restart=on-failure

[Install]
WantedBy=multi-user.target

Each section has a job. [Unit] describes the unit and orders it after the network is up. [Service] says how to run it: as which user, from which directory, with which command, and what to do if it dies. [Install] says what enabling it means: here, that multi-user.target should want it, so it starts at boot.

Then manage it, step by step.

1. sudo systemctl daemon-reload. The systemctl manual says this reloads the manager configuration, rerunning generators, reloading all unit files and recreating the dependency tree. systemd does not notice a new or changed file until you do this.
2. sudo systemctl start site starts it now; systemctl status site shows whether it is active, its main PID, its recent log lines and its cgroup of processes.
3. sudo systemctl enable site makes it start at boot. The manual explains that enable creates the symlinks encoded in the unit's [Install] section, then reloads the manager configuration. enable --now enables and starts in one step.
4. journalctl -u site shows its full log, which Lesson 32 explores.

Day-to-day verbs follow. The stop verb stops the service, and restart stops and starts it. The reload verb is different. The manual says it asks the service to reload its own configuration, not systemd's unit file, which is what daemon-reload is for. Services that support reload, such as nginx, keep serving while they reread their configuration. Our Python server has no reload action, so systemctl reports that reload is not applicable. Finally, disable removes the boot-time symlinks but still allows manual starts.

Bottom line: daemon-reload makes systemd reread unit files; reload makes a service reread its own configuration; enable affects the next boot, start affects now, and enable --now does both.

Change one input: a typo, an override and a mask

Suppose ExecStart had said /usr/bin/python and that file does not exist on AlmaLinux. systemctl start site fails, and status shows the main process exited with status=203/EXEC. The systemd.exec manual decodes it: 203, EXIT_EXEC, means the execve system call failed, most likely because the executable is missing or not accessible. Fix the path, run daemon-reload, start again. Reading status before guessing saves most of the time spent on failed services.

Now suppose you need to change a unit you did not write, such as raising the open-files limit for a packaged database. Do not edit its file under /usr/lib. Run sudo systemctl edit mariadb instead. The manual says edit creates or replaces a drop-in snippet that extends or overrides the unit. It writes the snippet only if the editor exits successfully, then reloads the configuration, equivalent to daemon-reload. The snippet lands in /etc/systemd/system/mariadb.service.d/override.conf. It could hold just two lines, [Service] and LimitNOFILE=65536, the systemd equivalent of Lesson 16's limits. The command systemctl cat mariadb shows the original file and every drop-in together.

Finally, suppose a service must never run on this machine, even if another unit or a person tries to start it. Disabling it is not enough, because a disabled unit can still be started by hand or pulled in as a dependency. Masking it is: sudo systemctl mask cups. The manual describes mask as linking the unit file to /dev/null, making it impossible to start, a stronger version of disable that prohibits all activation. unmask undoes it.

Worked example 2: a timer instead of cron

Back to the report job. Write two files in /etc/systemd/system. The service does the work once:

[Unit]
Description=Nightly report

[Service]
Type=oneshot
User=reports
ExecStart=/opt/reports/report-gen --nightly
Nice=10

The timer, report.timer, says when:

[Unit]
Description=Run the nightly report on weekdays

[Timer]
OnCalendar=Mon..Fri *-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Then sudo systemctl daemon-reload and sudo systemctl enable --now report.timer. You enable and start the timer, not the service; the timer starts the service of the same name when it fires. The command systemctl list-timers shows every timer with its next and last run.

The calendar syntax is weekday, then date, then time, with * as a wildcard and .. for ranges; systemd.time lists shorthand such as daily and weekly. Test an expression before trusting it: the manual recommends systemd-analyze calendar, which normalizes the expression and prints when it will next occur. The Persistent=true line is the anacron feature from Lesson 16. The systemd.timer manual says it stores the time the service last ran. If the timer would have fired while the machine was off, it triggers the service immediately when the timer starts again.

Why prefer a timer to cron? The job's output goes to the journal automatically. It runs with the unit's limits and user settings, and it can depend on other units. And systemctl status shows whether the last run failed.

Targets: states the system can be in

Targets group units into system states, as Lesson 2 showed. There is multi-user.target for a full text-mode system and graphical.target for one with a graphical login, plus rescue.target and emergency.target for repairs.

• systemctl get-default shows the target used at boot, and sudo systemctl set-default multi-user.target changes it.
• sudo systemctl isolate rescue.target switches the running system to another target now, stopping everything that target does not want. Use it at the console rather than over SSH.

The other tools in the objective

The objectives list a "systemd-blame" utility. No command by that name exists; the feature is systemd-analyze blame, and its manual notes that it skips services of Type=simple, because systemd starts those immediately. A unit at the top of blame took longest to start, which is not the same as delaying the boot; critical-chain shows the chain that actually held boot up.

The hostnamectl manual explains the three host names. The static name is the one you configure. The transient name is a fallback from network configuration, used only when no static name is set. The pretty name may contain spaces and other characters. Setting the hostname changes all three unless you choose one with an option. The timedatectl manual says set-ntp enables and starts the first available network time synchronization service; Lesson 19 configures chrony, the one AlmaLinux uses.

The sysctl command deserves one more line. Settings made with sudo sysctl -w net.ipv4.ip_forward=1 last until reboot. To persist them, put the line in a file under /etc/sysctl.d/ and load all such files with sudo sysctl --system. The sysctl manual says that option reads every system configuration file in a defined order of precedence. Lesson 23 uses exactly this setting to turn a Linux host into a router.

Common misconceptions

• "systemctl reload rereads my edited unit file." reload asks the service to reload its own configuration; daemon-reload makes systemd reread unit files.
• "A disabled service cannot run." It does not start at boot but can still be started by hand or as a dependency; mask prevents all activation.
• "Customize a packaged unit by editing it in /usr/lib/systemd/system." Package updates overwrite it; use systemctl edit to create a drop-in in /etc.
• "To schedule a job, enable its service." Enable and start the timer; it triggers the service.
• "sysctl -w changes survive a reboot." They are runtime only; persist them in /etc/sysctl.d/ and load with sysctl --system.

What you now know

• Unit types: service, timer, mount (fstab entries become mount units) and target. Your units and overrides belong in /etc/systemd/system, which overrides /usr/lib/systemd/system.
• start, stop, restart and reload act now; enable and disable affect boot; mask links a unit to /dev/null so nothing can start it; status shows state, PID and recent logs.
• daemon-reload rereads unit files; systemctl edit writes a drop-in override and reloads for you; systemctl cat shows the merged result.
• status=203/EXEC means the ExecStart program could not be executed.
• Timers pair with services; OnCalendar sets the schedule, Persistent=true catches up missed runs, list-timers shows them, systemd-analyze calendar tests expressions.
• systemd-analyze blame and critical-chain analyze boot; hostnamectl, timedatectl and resolvectl manage the host name, time and DNS; sysctl sets kernel parameters, persistently through /etc/sysctl.d.

Sources

1. systemd project. (n.d.). systemctl and systemd.unit. freedesktop.org. systemctl; systemd.unit
2. systemd project. (n.d.). systemd.exec [Process exit codes]. freedesktop.org. freedesktop.org
3. systemd project. (n.d.). systemd.timer and systemd.time. freedesktop.org. systemd.timer; systemd.time
4. systemd project. (n.d.). systemd-analyze, hostnamectl, timedatectl and resolvectl. freedesktop.org. systemd-analyze; hostnamectl; timedatectl; resolvectl
5. procps-ng project. (n.d.). sysctl(8). Linux man-pages project. man7.org

One server, five requests

A twelve-person architecture office has one spare server and five requests. Staff want to reach the file server as files.office.example instead of by number. The building's door controller keeps drifting minutes off true time. Laptops need addresses when they join the network. The partners want an intranet page. And they want their own email domain. Each request is a network service, and objective 2.4 lists every one of them: DNS, NTP and PTP, DHCP, HTTP with Apache or Nginx, SMTP and IMAP4.

The exam asks for basic configuration, not mastery of each daemon, and the theory behind the protocols is the Network+ course's territory. This lesson works through the five requests in your lab, using the host-only network from Lesson 11 (192.168.56.0/24) and a private domain, office.example, reserved for examples.

A routine that works for every service

Every service in this lesson is configured the same way. Learn the routine once and the differences become details.

1. Install the package with dnf or apt.
2. Edit its configuration under /etc, keeping a backup copy first.
3. Check the syntax with the service's own checker before you restart anything.
4. Open the port in the host firewall, as Lesson 23 explains.
5. Enable and start it with systemctl, or reload it if it is already running.
6. Test it from another machine, not only from the server itself.
7. Read the logs with journalctl -u when anything fails.

Key idea: configure, check syntax, then restart. A syntax checker takes a second; a service that fails to restart because of a typo can take the office offline.

Request 1: names, with BIND

BIND's named daemon can answer for your own zone and also act as a caching resolver for everything else. Its administrator reference manual lists the companion tools you need: named-checkconf for the configuration file and named-checkzone for zone files. A zone file for office.example holds resource records:

$TTL 3600
@     IN SOA  ns1.office.example. admin.office.example. ( 2026102801 3600 900 604800 300 )
      IN NS   ns1.office.example.
      IN MX   10 mail.office.example.
ns1   IN A    192.168.56.10
mail  IN A    192.168.56.10
files IN A    192.168.56.20
www   IN CNAME files

Read it line by line. The SOA record names the primary server and the zone's administrator and carries a serial number, which you must increase every time you edit the zone or secondary servers will ignore the change. NS names the zone's name server, MX the mail server, A records map names to IPv4 addresses, and CNAME makes www an alias. In /etc/named.conf you declare the zone, with a zone "office.example" block pointing at this file, and set which clients may query. Then run named-checkzone office.example /var/named/office.example.zone, start named, and test from a client with dig @192.168.56.10 files.office.example.

Request 2: time, with chrony, and when PTP is needed

On AlmaLinux, time is kept by chronyd, configured in /etc/chrony.conf. Two directives do most of the work. The chrony.conf documentation describes pool as naming a pool of NTP servers whose name resolves to several addresses that may change over time, and allow as designating a subnet from which NTP clients may use this computer as their server. The default, it notes, is that no clients are allowed. So the office server gets:

• pool 2.pool.ntp.org iburst to follow public time servers,
• allow 192.168.56.0/24 so the door controller and laptops can use it.

After sudo systemctl restart chronyd, chronyc sources lists the servers chrony is using and chronyc tracking shows how far the clock is from them, as Red Hat's time synchronization guide demonstrates. Lesson 18's timedatectl confirms the system clock is synchronized.

The objectives pair NTP with the Precision Time Protocol (PTP, IEEE 1588). NTP keeps ordinary servers within milliseconds of true time, plenty for logs, Kerberos and certificates. PTP uses hardware timestamping in network cards and PTP-aware switches to reach sub-microsecond accuracy, which trading systems, telecom equipment and industrial control need. On Linux it is provided by the linuxptp package: ptp4l synchronizes a network card's hardware clock to the network's PTP grandmaster, and phc2sys copies that clock to the system clock. Red Hat's guide even shows a machine acting as a PTP client on one interface and an NTP server for the rest of the network on another. An architecture office does not need PTP; a stock exchange does.

Request 3: addresses, with Kea

For years the standard Linux DHCP server was ISC DHCP. ISC's own page now states that it announced the end of maintenance for ISC DHCP as of the end of 2022 and developed Kea to replace it. Kea's configuration is JSON. The Kea manual's minimal example defines a subnet and a pool of addresses to lease; the office version adds a router and a DNS server to hand out:

{ "Dhcp4": {
    "interfaces-config": { "interfaces": [ "enp0s8" ] },
    "subnet4": [ {
        "id": 1,
        "subnet": "192.168.56.0/24",
        "pools": [ { "pool": "192.168.56.100 - 192.168.56.199" } ],
        "option-data": [
            { "name": "routers", "data": "192.168.56.1" },
            { "name": "domain-name-servers", "data": "192.168.56.10" }
        ]
    } ]
} }

JSON is strict: one missing comma and the server refuses to start, which is why the syntax check matters. Keep static devices such as the server itself outside the pool. And make sure only one DHCP server answers on a network. A second, forgotten one hands out wrong gateways, a classic fault in Lesson 34.

Request 4: the intranet, with Apache or Nginx

Both web servers serve files from a document root and can host several sites on one address, chosen by the host name the browser asks for.

• Apache httpd. Apache's documentation says the main configuration file is usually called httpd.conf and that further files can be added with the Include directive. On AlmaLinux that is /etc/httpd/conf/httpd.conf plus files in /etc/httpd/conf.d/; the default document root is /var/www/html. On Ubuntu the package is apache2, and each site lives in /etc/apache2/sites-available/. The a2ensite manual explains that it enables a site, a file containing a VirtualHost block, by creating a symlink in sites-enabled; a2dissite removes it. A name-based virtual host starts <VirtualHost *:80> and sets ServerName and DocumentRoot. Check with apachectl configtest.
• Nginx. Its beginner's guide builds a site from a server block: a listen directive for the port, server_name for the host name and root for the files, inside the http context of /etc/nginx/nginx.conf or an included file. Test with nginx -t and apply with a reload, which, the guide explains, makes the master process check the new configuration and start new workers with it, without dropping service.

A typical Nginx site for the office:

server {
    listen 80;
    server_name intranet.office.example;
    root /srv/intranet;
}

On AlmaLinux, serving from /srv/intranet also needs the right SELinux label on the files, which Lesson 22 explains. HTTPS, with a certificate, follows in Lesson 24.

Request 5: mail, with Postfix and Dovecot

Email needs two services. SMTP, provided by Postfix, moves mail between servers and accepts mail from users' programs. IMAP4, provided by Dovecot, lets users read mail stored on the server from any device. RFC 9051, the current IMAP4rev2 standard, says a server listens on port 143 for cleartext connections or 993 for implicit TLS.

Postfix's basic configuration guide walks through a handful of parameters in /etc/postfix/main.cf:

myhostname = mail.office.example
mydomain = office.example
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
mynetworks = 127.0.0.0/8, 192.168.56.0/24

myorigin sets the domain on outgoing mail; the guide notes the default is the machine's own name and suggests $mydomain for most sites. mydestination lists the domains delivered locally. mynetworks is the dangerous one. The guide says Postfix forwards mail from clients in authorized networks to any destination, so a value that is too broad turns the server into an open relay that spammers will find and abuse within days. It warns that the subnet style may be too friendly on a wide area network; the current default authorizes only the local machine. Run postfix check and postconf -n before systemctl restart postfix.

Dovecot then needs only a few decisions: the protocols to serve (protocols = imap), where mail is stored, and TLS settings, with plaintext authentication disabled except over TLS. Test with a mail client from a laptop, never only from the server.

Key idea: SMTP carries mail to and between servers; IMAP4 lets people read it. Limit relaying to your own networks or authenticated users, and require TLS for logins.

Common misconceptions

• "Restart first, check later." Every service here has a syntax checker; use it before restarting, or a typo takes the service down.
• "Editing a DNS zone is enough." Increase the SOA serial number too, or secondary servers keep the old data.
• "chrony serves time to the network by default." Its documentation says no clients are allowed until you add an allow directive.
• "ISC DHCP is the current Linux DHCP server." ISC ended its maintenance at the end of 2022; Kea replaces it.
• "A broad mynetworks setting just makes things easier." It can make the server an open relay that sends spam for anyone in those networks.

The takeaway

• Routine: install, back up and edit under /etc, check syntax, open the port, enable and start or reload, test from a client, read journalctl.
• BIND: named.conf declares zones; zone files hold SOA, NS, MX, A and CNAME records; check with named-checkconf and named-checkzone; bump the serial.
• chrony: pool or server lines to follow, allow to serve clients; check with chronyc sources and tracking. PTP (ptp4l, phc2sys) gives sub-microsecond time with hardware support.
• Kea replaces ISC DHCP; its JSON configuration defines subnets, pools and options such as routers and DNS servers.
• Apache: httpd.conf with conf.d, or Debian's sites-available with a2ensite; Nginx: server blocks with listen, server_name and root; test with apachectl configtest or nginx -t.
• Postfix main.cf: myhostname, mydomain, myorigin, mydestination, mynetworks (keep it narrow); Dovecot serves IMAP on 143 or 993.

Sources

1. Internet Systems Consortium. (n.d.). BIND 9 administrator reference manual. bind9.readthedocs.io
2. chrony project. (n.d.). chrony.conf(5) (version 4.6). chrony-project.org
3. Red Hat. (n.d.). Configuring time synchronization. In Configuring basic system settings (RHEL 9). docs.redhat.com
4. Internet Systems Consortium. (n.d.). ISC DHCP and Kea administrator reference manual: The DHCPv4 server. isc.org; kea.readthedocs.io
5. Apache Software Foundation. (n.d.). Configuration files and Name-based virtual host support. Apache HTTP Server 2.4 documentation. httpd.apache.org (configuring); httpd.apache.org (virtual hosts); Debian. (n.d.). a2ensite(8). manpages.debian.org
6. NGINX. (n.d.). Beginner's guide. nginx.org
7. Venema, W. (n.d.). Postfix basic configuration. postfix.org; Dovecot. (n.d.). Dovecot documentation. doc.dovecot.org
8. Melnikov, A., and Leiba, B. (Eds.). (2021). Internet Message Access Protocol (IMAP) - Version 4rev2 (RFC 9051). RFC Editor. rfc-editor.org

A login that should have failed

At 08:57 on a Monday, a developer on her first day types her company password into an SSH prompt on web02, a Linux server nobody has ever created an account on. She is logged in a second later, into a home directory that did not exist a moment before. grep mira /etc/passwd on web02 finds nothing. Yet id mira prints a UID, a primary group and a list of group memberships.

Nothing is broken. web02 belongs to the company's directory, and her account lives there, not in any local file. Following that single login from keystroke to shell prompt covers objective 3.1's authentication and identity methods: PAM, SSSD and realmd, LDAP, Kerberos, Samba with Winbind, and Polkit. The logging and auditing half of the objective comes in Lesson 32.

Two questions every login asks

A login has to answer two separate questions, and Linux uses two separate systems to answer them.

• Who is this? What is mira's UID, primary group, home directory and shell? That is NSS, the Name Service Switch from Lesson 11. Its passwd: and group: lines in /etc/nsswitch.conf list the sources, typically files sss: the local files first, then SSSD. getent passwd mira asks the same question any program would and returns her entry from the directory.
• Prove it, and may she log in? That is PAM, Pluggable Authentication Modules. Each program that authenticates people, such as sshd, login, sudo or su, asks PAM, and PAM runs a configurable stack of modules.

Keeping the two apart explains many puzzles. If getent passwd mira finds nothing, the problem is identity (NSS, SSSD, the directory), and no PAM change will help. If getent finds her but logins fail, look at PAM and authentication.

Inside the PAM stack

Each PAM-aware program has a file in /etc/pam.d/ named after its service, such as sshd. The pam.d manual describes each line as a module type, a control flag, a module and its arguments. There are four module types, which run at different moments.

The control flag decides what one module's result means for the whole stack. The pam.d manual defines them precisely.

• required: failure makes the whole stack fail, but only after the remaining modules have run, so an attacker cannot tell which one failed.
• requisite: like required, but a failure returns control immediately.
• sufficient: if it succeeds and no earlier required module has failed, the stack succeeds at once; its failure is ignored.
• optional: matters only if it is the only module of its type.
• include and substack: pull in the lines of another file, which is how many services share one common stack.

Here is the shape of a real auth stack, from the system-auth file shown in Red Hat's authselect documentation, simplified:

auth  required    pam_env.so
auth  required    pam_faildelay.so delay=2000000
auth  sufficient  pam_unix.so nullok try_first_pass
auth  sufficient  pam_sss.so forward_pass
auth  required    pam_deny.so

Read it top to bottom. A correct local password satisfies pam_unix, which is sufficient, so the stack succeeds. If not, pam_sss tries the directory; success there also ends the stack. If both fail, pam_deny, which always fails, is required, so the login fails. The lesson-14 policies plug into this machinery: pam_faillock in auth for lockout, pam_pwquality and pam_pwhistory in password for strength and reuse, and an MFA module such as one for time-based codes in auth.

Two warnings before you edit. The authselect-generated files on the Red Hat family carry the line "Do not modify this file manually," because authselect manages them through profiles and features; Red Hat's documentation describes choosing and modifying authselect profiles instead. Debian-family systems use pam-auth-update for the shared common-auth files. And whenever you change PAM on a remote server, keep a second root session open until you have tested a fresh login. A broken stack can lock everyone out, including you.

The point: NSS answers "who is this user?"; PAM answers "prove it and may they log in?" through auth, account, password and session stacks whose control flags decide how module results combine.

Where the account really lives: LDAP

Mira's account is an entry in a directory. RFC 4511 defines the Lightweight Directory Access Protocol (LDAP) as giving access to distributed directory services that follow the X.500 data and service models. A directory is a tree of entries, and each entry is named by its distinguished name, a path read from the entry up to the root, such as uid=mira,ou=People,dc=example,dc=com. Entries hold attributes: uid, uidNumber, gidNumber, homeDirectory, loginShell, memberOf. Microsoft's Active Directory, FreeIPA and OpenLDAP all speak LDAP.

The ldapsearch tool, which its manual describes as opening a connection to an LDAP server, binding and searching, lets you see an entry directly: ldapsearch -H ldaps://ldap.example.com -b dc=example,dc=com "(uid=mira)". LDAP traffic should use TLS, as the ldaps:// address does, because a plain bind sends a password across the network.

Proving who she is: Kerberos

Active Directory and FreeIPA authenticate with Kerberos, which proves identity with tickets instead of sending passwords to every server. A trusted key distribution center (KDC) issues a ticket-granting ticket (TGT) when you first authenticate, and later grants service tickets for each server you use, without asking for your password again. The MIT Kerberos user commands make it visible: kinit mira@EXAMPLE.COM obtains a TGT, klist lists the tickets you hold, and kdestroy discards them. The realm name is conventionally the domain in capitals.

Kerberos has one operational requirement that catches people. Tickets carry timestamps, and the MIT krb5.conf documentation sets the default maximum clock skew to 300 seconds, five minutes; beyond that a message is assumed invalid. A server whose clock drifts, perhaps because chrony from Lesson 19 stopped, suddenly rejects every Kerberos login. Time service is part of authentication.

Gluing it together: SSSD and realmd

Something on web02 must speak LDAP and Kerberos on Mira's behalf. That is SSSD, the System Security Services Daemon. Its documentation calls it the client component of centralized identity management solutions such as FreeIPA, 389 Directory Server, Active Directory and OpenLDAP. It serves and caches directory information and provides identity, authentication and authorization services to the host. The cache matters: a laptop off the network can still log a directory user in with cached credentials. SSSD plugs into both questions above, through sss in nsswitch.conf and pam_sss in PAM, and is configured in /etc/sssd/sssd.conf.

Configuring SSSD, Kerberos and PAM by hand for a domain is fiddly. realmd automates it. The realm manual describes a tool for managing enrollment in Kerberos realms such as Active Directory domains: realm discover example.com shows a domain's type and requirements, realm join example.com -U admin joins the machine, realm list shows what it has joined, and realm permit limits which users may log in. That is how web02 came to trust Mira's account. pam_mkhomedir in the session stack, whose manual says it creates a home directory at login if none exists, explains the folder that appeared for her.

Samba and Winbind

Samba is the open source implementation of the SMB protocol used by Windows file sharing, configured in smb.conf, which its manual calls the configuration file for the Samba suite. It serves shares to Windows and Linux clients, and Lesson 9 mounted such shares with mount.cifs. Samba also brings its own way of joining Active Directory: Winbind. The winbindd manual describes a daemon that provides services to the Name Service Switch, to applications through PAM and ntlm_auth, and to Samba itself. In practice a Linux server joined to AD uses either SSSD or Winbind for identity; realm join can set up either, and file servers running Samba commonly use Winbind.

After login: Polkit

Later that morning, Mira, logged in to a Linux workstation, changes its Wi-Fi network from the desktop without typing sudo. Something decided she was allowed to. That something is Polkit. Its manual describes an authorization API for privileged programs, called mechanisms, that offer services to unprivileged programs, called subjects, which the mechanism treats as untrusted. For each request, the mechanism asks Polkit whether the subject is authorized for that particular action: allowed, denied, or allowed after authenticating as an administrator.

Actions are defined by the software that offers them, and administrators write rules, small JavaScript files in /etc/polkit-1/rules.d/, for example allowing members of a support group to restart one service without a password. pkexec is Polkit's command-line face: its manual says it lets an authorized user run a program as another user, root by default. sudo and Polkit overlap but differ: sudo grants whole commands to users through sudoers, as Lesson 21 shows; Polkit authorizes specific actions requested through system services.

The point: SSSD with LDAP and Kerberos supplies identity and proof from a central directory, realmd joins the domain, Winbind is Samba's alternative, and Polkit authorizes particular privileged actions after login.

Common misconceptions

• "If a user is not in /etc/passwd, they cannot log in." NSS may find them in a directory through SSSD or Winbind; check with getent passwd.
• "sufficient means the module is enough on its own, whatever else happens." It succeeds immediately only if no earlier required module has failed.
• "Kerberos sends the password to each server." It sends tickets obtained from the KDC; the password is used to get the TGT, not given to every service.
• "Clock settings are unrelated to logins." Kerberos rejects messages beyond its clock skew, five minutes by default.
• "Polkit and sudo are the same thing." sudo runs whole commands with elevated rights; Polkit decides whether a subject may perform a specific action offered by a privileged service.

Summing up

• NSS (nsswitch.conf, getent) answers identity; PAM (/etc/pam.d) answers authentication and access.
• PAM types: auth, account, password, session. Flags: required (fail at the end), requisite (fail now), sufficient (succeed now if nothing required failed), optional, include and substack.
• On the Red Hat family authselect generates the shared stacks; keep a second root session open when changing PAM.
• LDAP queries a directory of entries named by distinguished names; use TLS.
• Kerberos issues a TGT from the KDC (kinit, klist, kdestroy) and needs clocks within 300 seconds.
• SSSD caches directory identity and plugs into NSS and PAM; realm discover, join, list and permit manage domain membership; pam_mkhomedir creates home directories.
• Samba serves SMB shares (smb.conf), and Winbind offers AD identity through NSS and PAM.
• Polkit authorizes unprivileged subjects' requests to privileged mechanisms, using rules in /etc/polkit-1/rules.d; pkexec runs a program as another user if authorized.

Sources

1. Linux-PAM project. (n.d.). pam.d(5) and pam_mkhomedir(8). Linux man-pages project. man7.org (pam.d); man7.org (pam_mkhomedir)
2. Red Hat. (n.d.). Configuring user authentication using authselect. In Configuring authentication and authorization in RHEL (RHEL 9). docs.redhat.com
3. Sermersheim, J. (Ed.). (2006). Lightweight Directory Access Protocol (LDAP): The protocol (RFC 4511). RFC Editor. rfc-editor.org; OpenLDAP. (n.d.). ldapsearch(1). manpages.debian.org
4. MIT Kerberos. (n.d.). krb5.conf and kinit. MIT Kerberos documentation. web.mit.edu (krb5.conf); web.mit.edu (kinit)
5. SSSD project. (2025). Introduction. sssd.io; Debian. (n.d.). realm(8) and sssd.conf(5). realm; sssd.conf
6. Samba Team. (n.d.). smb.conf(5) and winbindd(8). Debian manual pages. smb.conf; winbindd
7. polkit project. (n.d.). polkit(8) and pkexec(1). Debian manual pages. polkit; pkexec

A new server, minute by minute

A fresh Ubuntu server is reachable from the internet at 10:00. On a typical day it will see its first SSH password guesses within minutes, the kind of log lines Lesson 4 counted. This lesson hardens such a server step by step, in the order an experienced administrator would, and each step is an item in objective 3.3. Do it in your lab VMs, where a mistake costs a snapshot revert, and keep one root session open whenever you change how people log in.

Step 1: personal accounts with sudo

Lesson 14 retired the shared root login. What replaces it is sudo: each administrator logs in as themselves and runs individual commands with elevated rights, and each use is logged under their name. Rules live in /etc/sudoers, which you must edit with visudo. Its manual says visudo locks the file against simultaneous edits, performs basic validity checks and checks for syntax errors before installing the edited file. A syntax error in a hand-edited sudoers file can leave nobody able to use sudo at all.

Both families grant full rights through a group, already present in the default sudoers file: wheel on the Red Hat family and sudo on the Debian family. So making someone an administrator is sudo usermod -aG wheel dana on AlmaLinux or sudo usermod -aG sudo dana on Ubuntu.

Narrower rules belong in files under /etc/sudoers.d/; the sudoers manual describes the includedir directive that reads every file in that directory, which is how packages and administrators add rules without touching the main file. Create one with sudo visudo -f /etc/sudoers.d/webops. A rule reads who, where, as whom, what:

%webops ALL=(root) /usr/bin/systemctl restart nginx, /usr/bin/journalctl -u nginx

Members of the webops group may run exactly those two commands as root, and nothing else. That is least privilege written down.

Two tags in the objectives change how a rule behaves.

• NOPASSWD skips the password prompt. It is convenient for automation, but it means anyone who gains the account, through a stolen SSH key or an unlocked terminal, gets those privileges without knowing any password. Restrict NOPASSWD to narrow commands, never to ALL.
• NOEXEC: the sudoers manual says it prevents a dynamically linked program from running further commands itself. Its example is letting a user run more and vi with shell escapes disabled. Without it, sudo vi hands out a root shell through vi's :!sh.

A few commands complete the picture. sudo -l lists what you may run. sudo -i starts an interactive root login shell, with root's environment, when you genuinely need several root commands in a row. su - also gives a root login shell, but it asks for root's own password and leaves no record of which person used it, which is why sudo replaced it for day-to-day work. The dash matters for both su and its options: without it, you keep your own environment, including your PATH.

What matters here: grant rights through wheel or sudo groups and narrow sudoers.d rules edited with visudo; keep NOPASSWD narrow, and use NOEXEC for programs that can spawn shells.

Step 2: SSH keys instead of passwords

Password guessing works only against password logins. Replace them with key pairs. On your own computer, ssh-keygen -t ed25519 creates a private key, which stays with you, protected by a passphrase, and a public key ending in .pub. ssh-copy-id dana@server appends the public key to ~/.ssh/authorized_keys on the server. Log in again: if the server accepts the key, you are asked only for the key's passphrase, locally.

Typing that passphrase every time gets old, which is the job of the SSH agent: ssh-agent holds decrypted keys in memory for your session, and ssh-add loads one. Be careful with agent forwarding, which lets a remote server use your agent for further hops. The ssh_config manual warns that anyone able to bypass file permissions on the remote host can use your forwarded agent; they cannot copy your keys, but they can authenticate with them while you are connected. Forward the agent only to hosts you trust.

Step 3: tightening sshd

Only now, with key login proven in a second session, change the server's /etc/ssh/sshd_config. On both lab distributions it includes files from /etc/ssh/sshd_config.d/, and that matters, because the sshd_config manual says that for each keyword the first obtained value is used. A setting in an included file read first wins over a later line in the main file.

Then test before you reload. sudo sshd -t is, in its manual's words, test mode: it checks the validity of the configuration file and the keys. Only if it is silent, reload the service and, from a new terminal, prove that you can still log in. Keep the old session open until you have.

Step 4: tunnels and SFTP

SSH carries more than shells. An SSH tunnel forwards a TCP port through the encrypted connection. The ssh manual describes -L as forwarding connections to a port on the local machine to a host and port on the remote side, and -R as the reverse direction. So ssh -L 5433:localhost:5432 dana@db01 lets a laptop reach a database that listens only on db01's loopback, without opening port 5432 to the network. -D makes a dynamic, SOCKS-style forward. The same power can bypass network policy, which is why sshd's AllowTcpForwarding setting exists.

SFTP is file transfer over SSH, and it replaces FTP. For accounts that should only transfer files, sshd can force the built-in SFTP server and lock the user into a directory with ChrootDirectory. The manual sets strict conditions: every component of that path must be owned by root and not writable by group or others, and an SFTP-only session needs nothing else inside it.

That raises chroot in general: changing a process's idea of the root directory, so / means some subdirectory. It can contain a confused or limited program, but the chroot system call's manual is blunt that it is not intended for any kind of security purpose, and that a privileged process can escape it. Treat chroot as one layer, combined with an unprivileged user, never as a sandbox on its own.

Step 5: fail2ban, insecure services and unused filesystems

Even with passwords off, thousands of failed attempts fill logs and waste resources. fail2ban, its project explains, scans log files such as the authentication log and bans addresses with too many failed logins by updating firewall rules for a configurable time. Enable the sshd jail in /etc/fail2ban/jail.local with settings such as maxretry and bantime, then check it with sudo fail2ban-client status sshd.

Next, remove what should not be there. Three old protocols are named in the objectives.

Check with sudo ss -tulpn for anything listening, and remove server packages you do not need. A service that is not installed cannot be exploited.

Finally, disable unused filesystems. The kernel can load drivers for dozens of filesystem types, such as cramfs, hfsplus or udf, whenever a matching device or image appears. Each driver is code an attacker might reach with a crafted USB stick or disk image. Hardening guides block those not in use with Lesson 6's modprobe.d technique: install cramfs /bin/false plus blacklist cramfs in a file under /etc/modprobe.d/. Together with removing needless setuid bits (Lesson 15) and Secure Boot (Lesson 2), these steps complete objective 3.3's hardening list.

What matters here: keys not passwords, no root over SSH, an allow list of users or groups, test with sshd -t, ban repeat offenders with fail2ban, replace Telnet, FTP and TFTP, and block filesystems you never use.

Common misconceptions

• "Editing /etc/sudoers with nano is fine if careful." visudo locks the file and refuses to install it with a syntax error; one typo otherwise can disable sudo for everyone.
• "NOPASSWD is harmless on a trusted team." It gives the privileges to anyone who controls the account, without any password, so keep it to narrow commands.
• "A setting at the end of sshd_config overrides earlier ones." For most keywords the first obtained value wins, including values in included files read first.
• "chroot makes a secure sandbox." Its manual says it is not intended for security; root can escape it.
• "Moving SSH to another port secures it." It reduces log noise but not risk; keys, no root login and fail2ban do the real work.

Looking back

• Use visudo (and visudo -f for /etc/sudoers.d files); wheel (Red Hat) and sudo (Debian) groups grant full rights; group rules can grant single commands.
• NOPASSWD skips the prompt; NOEXEC blocks shell escapes; sudo -l lists rights; sudo -i and su - give root login shells, su with root's password and no per-person record.
• ssh-keygen and ssh-copy-id set up keys; ssh-agent and ssh-add hold them; forward the agent only to trusted hosts.
• sshd_config: PasswordAuthentication no, PermitRootLogin no, AllowUsers or AllowGroups, X11Forwarding no; first value wins; test with sshd -t before reloading.
• ssh -L, -R and -D tunnel ports; SFTP with ChrootDirectory serves file-only accounts; chroot alone is not a security boundary.
• fail2ban bans addresses after repeated failures; replace Telnet, FTP and TFTP; disable unused filesystem modules through modprobe.d.

Sources

1. Miller, T. C., and others. (n.d.). sudoers(5), sudo(8) and visudo(8). Sudo project. sudoers; sudo; visudo
2. OpenBSD project. (n.d.). sshd_config(5), sshd(8), ssh(1), ssh_config(5) and ssh-agent(1). OpenSSH manual pages. sshd_config; sshd; ssh; ssh_config; ssh-agent
3. Linux man-pages project. (n.d.). su(1) and chroot(2). su; chroot
4. Fail2Ban project. (n.d.). Fail2Ban: ban hosts that cause multiple authentication errors. GitHub. github.com
5. Allman, M., and Ostermann, S. (1999). FTP security considerations (RFC 2577). RFC Editor. rfc-editor.org
6. Sollins, K. (1992). The TFTP protocol (revision 2) (RFC 1350). RFC Editor. rfc-editor.org
7. kmod project. (n.d.). modprobe.d(5). Linux man-pages project. man7.org

The 403 that tempts everyone

Lesson 19 built an intranet site served by Nginx from /srv/intranet on AlmaLinux. The files are readable by everyone, the configuration passes nginx -t, and yet every request returns 403 Forbidden. An administrator types sudo setenforce 0, reloads the page, and it works. Problem solved, he says, and he makes the change permanent.

That is the most common wrong answer in Linux security, and this lesson debugs it. The 403 vanished because a whole layer of protection was switched off for every program on the server, while the actual fault, one wrong label on one directory, was left in place. Red Hat's SELinux guide puts the cost plainly: with SELinux disabled, the policy is not loaded, nothing is enforced, no denials are logged, and all of its benefits are lost. The rest of this lesson finds and fixes the real cause, which is objective 3.3's SELinux list and, later, a part of objective 5.4.

What SELinux adds to permissions

Lesson 15's permissions are discretionary: a file's owner decides who may use it, and root may do anything. SELinux adds mandatory access control, rules set by the system's policy that even root cannot override by changing ownership or modes. The kernel checks ordinary permissions first; if they allow the access, SELinux then checks its policy, and both must agree.

The model used on Red Hat-family systems is called type enforcement. Every process runs in a domain, such as httpd_t for the web server, and every file has a type, such as httpd_sys_content_t for web content. The policy lists which domains may do what to which types. Anything not allowed is denied. If an attacker takes over the web server, the process is still httpd_t, and it still cannot read files labeled for SSH keys or users' home directories, whatever their permission bits say.

Labels are called contexts and have four fields: user, role, type and level, as in system_u:object_r:httpd_sys_content_t:s0. For everyday troubleshooting, only the type matters. Add -Z to familiar commands to see them: ls -Z for files, ps -eZ for processes and id -Z for yourself.

Why this matters: SELinux confines each service to the types its policy names, so a compromised service stays boxed in. Turning it off removes that box from every service at once.

Modes and states, used correctly

Red Hat's guide describes three modes. In enforcing mode, the default and recommended mode, the policy is enforced. In permissive mode the system behaves as if it were enforcing, labeling objects and logging denials, but does not actually deny anything. In disabled mode no policy is loaded at all.

Permissive mode has one legitimate use: diagnosis. If a problem disappears in permissive mode, SELinux is involved, and the log now records every access that would have been denied. The guide's advice for debugging, even in production, is to use permissive mode temporarily rather than disabling SELinux. Better still, make just the one domain permissive and leave the rest of the system protected.

Tracing the 403: what was denied, and why?

SELinux denials are recorded by the audit system as AVC messages, in /var/log/audit/audit.log; Lesson 32 covers auditd itself. Two tools turn them into answers.

• sudo ausearch -m AVC -ts recent lists recent denials. Each shows the source context (the process), the target context (the file), the class and the permission that was refused.
• sudo sealert -l "*", from the setroubleshoot package, explains them in sentences. Red Hat's guide shows exactly this case: SELinux is preventing httpd from getattr access on a file, with suggestions for fixing it.

Now look at the labels. ls -Zd /var/www/html /srv/intranet shows httpd_sys_content_t on the standard web root and var_t on the new directory. Red Hat's guide explains why: on Red Hat-family systems /srv is labeled var_t, files created under it inherit that type, and SELinux prevents the web server from accessing it. Nothing was wrong with Nginx or with the file permissions. The directory simply carries a label the web server's domain may not read.

The same guide gives the order to work through. Do not reach for a policy change first; check for a labeling problem, then for a service configured in a way SELinux has not been told about, and only then consider a policy problem.

Fix 1: label the directory properly

Two commands, in this order, fix the label permanently:

1. sudo semanage fcontext -a -t httpd_sys_content_t "/srv/intranet(/.*)?" adds a rule to the policy's file-context configuration: this path, and everything under it, should be web content. The guide stresses that semanage does not change any file by itself.
2. sudo restorecon -Rv /srv/intranet applies the rules. Its manual describes setting the security context of files from that configuration, and it can be rerun at any time to correct inconsistent labels.

Reload the page and it works, in enforcing mode. Run sudo setenforce 1 first if someone turned it off.

Why not chcon? chcon -t httpd_sys_content_t changes the label on the files directly, and the site would work too. But it does not touch the file-context rules, so the next restorecon or full relabel puts the old var_t label back and the site breaks again, perhaps months later. Use chcon for a quick experiment, and semanage fcontext with restorecon for the real fix.

A related trap is moving files. Red Hat's guide shows files moved with mv from a home directory arriving with the user_home_t type, which the web server may not read, because a move keeps the original label. A copy creates new files that take the destination's default label. Either way, restorecon resets moved files to their proper type.

Fix 2: booleans and ports, for changed behavior

A week later the intranet adds a page that Nginx proxies to an application on another server, and those requests fail with a 502 while the access log shows nothing useful. sealert explains: SELinux is preventing nginx from connecting to the network. The web server's domain may serve files, but by default it may not make outbound network connections, which is exactly what a compromised web server would do.

Policy writers anticipate such needs with booleans, switches that turn optional parts of the policy on or off without writing any policy. getsebool -a | grep httpd lists those for web servers, and the guide's own example turns on httpd_can_network_connect_db so Apache may reach a database. Here the general one fits: sudo setsebool -P httpd_can_network_connect on. The -P makes it persist across reboots; without it, the change lasts until the next boot.

Ports are labeled too. If the site must listen on port 8081 instead of 80, SELinux refuses the bind until the port carries the web type. The guide shows the fix: sudo semanage port -a -t http_port_t -p tcp 8081, where -a adds a record, -t gives the type and -p the protocol. semanage port -l | grep http lists the ports already labeled.

Last resort: audit2allow

Sometimes no label or boolean fits, for example with an unusual in-house application. audit2allow reads denial messages and generates policy rules that would allow them: sudo ausearch -m AVC -ts recent | audit2allow -M myapp writes a small module, which semodule -i myapp.pp loads. Red Hat's guide warns against using it as your first option. It would happily allow whatever was denied, including an attacker's actions if the denials came from an attack. Read the generated rules before loading them, and prefer a correct label or boolean whenever one exists.

Why this matters: the fix order is label first (semanage fcontext, restorecon), then booleans and port labels for legitimate new behavior, and only then a reviewed local module from audit2allow. Disabling SELinux is never on the list.

Beyond AlmaLinux

You have met SELinux labels before. The :Z option on Podman volumes in Lesson 13 is exactly a relabel of host files so a container's domain may use them. Ubuntu uses a different mandatory access control system, AppArmor, which confines programs by path-based profiles; the exam's objectives name SELinux, so practise on your AlmaLinux machine.

Common misconceptions

• "If setenforce 0 fixes it, SELinux was the problem, so leave it off." It shows SELinux is involved; the fix is the correct label, boolean or port, then back to enforcing.
• "chcon is the way to fix a label." chcon changes are reverted by the next restorecon or relabel; use semanage fcontext, then restorecon.
• "Root is not affected by SELinux." Policy applies to processes by domain, including those running as root.
• "setsebool changes are permanent." Only with -P; otherwise they last until reboot.
• "audit2allow is the standard fix for denials." Red Hat warns against using it first; it can authorize exactly what an attacker was trying to do.

Recap

• SELinux adds mandatory type enforcement after normal permissions: domains (httpd_t) may access only the types (httpd_sys_content_t) the policy allows, even as root.
• Contexts are user:role:type:level; see them with ls -Z, ps -eZ and id -Z.
• Modes: enforcing, permissive (logs but does not deny) and disabled. getenforce and setenforce act now; /etc/selinux/config sets the boot mode; semanage permissive -a makes one domain permissive.
• Find denials with ausearch -m AVC and explain them with sealert.
• Fix labels with semanage fcontext -a then restorecon -Rv; chcon is temporary; moved files keep their old label until restorecon.
• Booleans (getsebool, setsebool -P) allow optional behaviors; semanage port -a labels non-standard ports.
• audit2allow is a last resort, reviewed before loading.

Sources

1. Red Hat. (n.d.). Using SELinux (RHEL 9) [Chapters on states and modes, troubleshooting, labeling problems and booleans]. docs.redhat.com
2. SELinux project. (n.d.). restorecon(8), semanage-fcontext(8), setsebool(8), getsebool(8), setenforce(8) and audit2allow(1). Linux man-pages project. restorecon; semanage-fcontext; setsebool; getsebool; setenforce; audit2allow
3. GNU coreutils. (n.d.). chcon(1). Linux man-pages project. man7.org

Rules you never wrote

Run sudo nft list ruleset on your AlmaLinux machine and it prints a long ruleset you never typed, beginning with a table named inet firewalld. On Ubuntu, sudo ufw status prints "Status: inactive". Both machines have the same packet-filtering engine in their kernels; they differ in the program that writes its rules and in whether anything has written them yet.

Objective 3.2 names that engine and five tools that program it. They are easiest to learn side by side, because each one is a different answer to the same question: how should an administrator describe which packets may pass?

One engine, several front ends

The engine is netfilter. Its project describes a framework of hooks inside the Linux kernel, places in the network stack where modules can inspect every packet. It lists what that enables: packet filtering, network address and port translation, logging and other packet mangling. The project associates two rule languages with it: iptables and its successor, nftables. The nftables wiki calls nftables the modern Linux kernel packet classification framework and says new code should use it instead of the legacy iptables family.

The practical rule follows from the table: choose one front end per host and use only it. Both firewalld and ufw regenerate their rules. Rules typed directly with iptables or nft beside them are overwritten or, worse, interact in ways nobody expects.

In short: netfilter is the kernel engine, iptables its legacy language and nftables the current one. Sets hold large match lists, and firewalld and ufw are front ends that write the rules for you.

Stateful and stateless

A stateless filter judges each packet alone, by its addresses, protocol and ports. To let replies to outgoing web requests back in, it needs rules for every possible reply. A stateful filter uses netfilter's connection tracking, which remembers every flow and labels each packet's state:

• NEW for the first packet of a connection;
• ESTABLISHED for packets of a known connection;
• RELATED for associated flows, such as an ICMP error about a connection;
• INVALID for packets that fit no known flow.

The iptables-extensions manual describes the conntrack match, --ctstate, that tests these. So nearly every host firewall begins with one stateful rule that accepts established and related traffic. Rules that admit only the NEW connections you intend follow, then a default policy that drops the rest:

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif lo accept
    tcp dport 22 accept
  }
}

That is an nftables ruleset in full. Read it top to bottom: anything part of an existing conversation passes; loopback traffic passes; new SSH connections pass; everything else hits the drop policy. The Network+ course discusses stateful and stateless firewalls as network devices; here they are the same idea inside one Linux host.

firewalld: zones, services and two configurations

The firewalld documentation describes a zone as defining the trust level for a connection, interface or source address. Each interface or source belongs to exactly one zone, while a zone can hold many. A laptop's Wi-Fi in a café belongs in public; the office wired port might be in work or internal. The default zone, usually public, catches anything not assigned elsewhere.

• Services and ports. firewall-cmd --add-service=https opens whatever ports firewalld's https service definition lists; --add-port=8443/tcp opens one port by number. Services are preferred because they name intent and can include several ports. firewall-cmd --get-services lists the definitions.
• Runtime and permanent. The firewall-cmd manual explains that without --permanent a change affects only the runtime configuration, and is lost at the next reload or reboot; with it, the change is written to the permanent configuration but is not effective until a reload. So either run each command twice, with and without --permanent, or make runtime changes, test them, and save them with --runtime-to-permanent, the workflow the manual itself describes.
• Rich rules express what simple options cannot. The rich language manual gives the form rule family="ipv4" source address="192.168.56.0/24" service name="ssh" accept, and notes that a rule using addresses needs the family.
• Inspect everything a zone allows with firewall-cmd --list-all, or every zone with --list-all-zones.

A test question often hinges on the runtime and permanent split: a port opened without --permanent works now and disappears after firewall-cmd --reload.

ufw: deliberately simple

ufw trades flexibility for brevity. The usual first session on an Ubuntu server is:

1. sudo ufw default deny incoming and sudo ufw default allow outgoing
2. sudo ufw allow OpenSSH, an application profile, or sudo ufw allow 22/tcp, a port
3. sudo ufw enable, only after SSH is allowed, or you cut off your own session
4. sudo ufw status verbose, and sudo ufw status numbered to see rule numbers for deletion

The ufw manual shows rules built from allow, deny, reject or limit, with optional interfaces, protocols, addresses, ports or application names. The command ufw limit ssh allows SSH but rate-limits addresses that connect too often, a lightweight cousin of Lesson 21's fail2ban.

Large lists: ipset and nftables sets

Blocking 20,000 known-bad addresses with 20,000 separate rules makes every packet walk a long list. The ipset manual describes IP sets as kernel structures that can store addresses, port numbers, address and port pairs and more. One iptables rule then matches a whole set in a single lookup. The nftables language builds the same idea in, as named sets inside a table, and firewalld exposes it as firewalld ipsets.

Translating addresses: NAT, PAT, SNAT and DNAT

A Linux host with two interfaces can act as a router, and netfilter's nat table can rewrite addresses as packets pass through. First, the host must forward packets at all. That is the kernel parameter from Lesson 18: net.ipv4.ip_forward=1, set at runtime with sysctl and persisted in /etc/sysctl.d/.

NAT is the umbrella term for all of these. The iptables-extensions manual draws one distinction you may be asked about. The MASQUERADE target belongs in the nat table's POSTROUTING chain and is meant for dynamically assigned addresses; with a static address, use SNAT. In firewalld, DNAT is written as --add-forward-port=port=80:proto=tcp:toport=8080:toaddr=192.168.56.20.

In short: enable ip_forward, then SNAT or masquerade (PAT) outbound traffic in POSTROUTING, and DNAT inbound traffic in PREROUTING.

Common misconceptions

• "firewall-cmd --permanent applies the change immediately." It writes the permanent configuration only; reload, or also run the command without --permanent.
• "Mixing ufw or firewalld with hand-written iptables rules is fine." The front ends regenerate their rules; use one tool per host.
• "A stateless filter is just as good with fewer rules." Without connection tracking, every reply needs its own rule, and spoofed packets are harder to reject.
• "NAT works as soon as the rule is added." The host must also forward packets, with net.ipv4.ip_forward set to 1.
• "iptables and nftables are separate firewalls in the kernel." Both are rule languages for netfilter; nftables is the successor.

What to carry forward

• netfilter is the kernel engine; nftables (nft) is the current rule language and iptables the legacy one; ipset and nftables sets hold large match lists.
• Stateful filtering uses connection tracking (NEW, ESTABLISHED, RELATED, INVALID); a sound ruleset accepts established and related traffic, admits chosen new connections, and drops the rest by policy.
• firewalld: zones set trust per interface or source; services are preferred to raw ports; changes are runtime unless --permanent, and --runtime-to-permanent saves tested changes; rich rules add sources, logging and limits.
• ufw: default deny incoming, allow SSH before enable, status numbered, limit for rate-limiting.
• NAT needs ip_forward=1; SNAT and masquerading (PAT) rewrite sources in POSTROUTING; DNAT rewrites destinations in PREROUTING for port forwarding.

Sources

1. The netfilter.org project. (n.d.). The netfilter.org project. netfilter.org; nftables wiki. (n.d.). What is nftables? wiki.nftables.org
2. netfilter project. (n.d.). iptables(8) and iptables-extensions(8). Linux man-pages project. iptables; iptables-extensions
3. Debian. (n.d.). nft(8) and ipset(8). Debian manual pages. nft; ipset
4. firewalld project. (n.d.). firewall-cmd, Zone and firewalld.richlanguage. firewall-cmd; zones; rich language
5. Canonical. (n.d.). ufw(8). Ubuntu manual pages. manpages.ubuntu.com; Firewalls. Ubuntu Server documentation. ubuntu.com
6. The Linux kernel documentation. (n.d.). IP sysctl: ip_forward. docs.kernel.org

One payroll file, three journeys

Every month a finance laptop produces a file called payroll-2026-10.csv: names, bank details and salaries for 140 people. It must reach the payroll server, be stored there for seven years, and never be readable by anyone who intercepts it on the way or steals a disk later. Following that file through one month covers objective 3.5. Cryptography appears at each stage, and each stage uses a different tool for a different threat.

For the mathematics, symmetric and asymmetric encryption, key exchange and signatures, the Security+ course is the natural companion. This lesson stays with the Linux tools and the decisions an administrator makes with them.

Before it leaves: hashes, HMACs and GPG

First, integrity. A hash function turns a file of any size into a short, fixed-length digest; change one byte and the digest changes completely. SHA-256, from NIST's Secure Hash Standard, FIPS 180-4, is the one in the objectives. sha256sum payroll-2026-10.csv prints its digest, and anyone can recompute it after the transfer and compare. A matching digest proves the file was not changed, accidentally or otherwise, since the digest was computed.

A plain hash has a gap: an attacker who can alter the file can also recompute its hash. An HMAC closes it. RFC 2104 describes HMAC as a mechanism for message authentication that combines a cryptographic hash function with a secret shared key. Only someone holding the key can produce a valid tag, so a matching HMAC proves the file came, unchanged, from a key holder. openssl dgst -sha256 -hmac "the-shared-key" payroll-2026-10.csv computes one, though in practice the key would come from a file, not the command line.

Then confidentiality, for the file itself. GnuPG encrypts files with public-key cryptography. The finance team imports the payroll server operator's public key, and encrypts with gpg --encrypt --recipient payroll@example.com payroll-2026-10.csv. Only the matching private key can decrypt the resulting .gpg file. Adding --sign also signs it with the sender's private key, so the recipient's gpg --decrypt verifies who sent it. For quick protection without key pairs, gpg --symmetric encrypts with a passphrase instead. Lesson 1 used the same tool in reverse: gpg --verify checked that a distribution's checksum file was signed by its publisher.

The upshot: a SHA-256 hash detects change, an HMAC also proves a key holder made it, and GPG encryption keeps the file itself private, with signatures proving the sender.

On the wire: TLS, OpenSSL and LibreSSL

The file is uploaded to the payroll server's web interface over HTTPS, which is HTTP inside TLS, Transport Layer Security. TLS encrypts the connection, protects it against tampering and, through the server's certificate, proves the client is talking to the real server. Its versions matter, and the objectives list them for a reason.

On Linux, TLS for most programs comes from a library. OpenSSL is the most common, and its openssl command is a Swiss Army knife for testing: openssl s_client -connect payroll.example.com:443 -tls1_3 shows whether the server accepts TLS 1.3 and prints its certificate chain, and openssl x509 -in cert.pem -noout -text decodes a certificate's subject, issuer and dates. LibreSSL, its project explains, is a version of the TLS and crypto stack forked from OpenSSL in 2014 to modernize the codebase and improve security, developed inside OpenBSD. It provides a compatible openssl command, so the tests look the same.

Between offices: WireGuard

A branch office also sends files to the payroll server, but over a site-to-site VPN rather than the open internet. The objectives name WireGuard, which its project describes as an extremely simple yet fast and modern VPN using state-of-the-art cryptography: the Noise protocol framework, Curve25519, ChaCha20, Poly1305 and BLAKE2, among others. It is built into the Linux kernel, and each side's configuration is short: a private key, the peer's public key, the peer's endpoint, and the address ranges (AllowedIPs) to route through the tunnel. wg-quick up wg0 brings it up and wg show reports peers and their last handshake. There are no cipher choices to configure, and therefore none to get wrong.

At rest: LUKS2 and Argon2

On the payroll server the data lives on an encrypted volume, so a stolen disk, or a decommissioned one sold without wiping, reveals nothing. Linux full-disk encryption is LUKS, managed with cryptsetup. The luksFormat manual says the default format is LUKS2.

1. sudo cryptsetup luksFormat /dev/sdb initializes the encrypted container and asks for a passphrase. It destroys whatever was on the device.
2. sudo cryptsetup open /dev/sdb payroll unlocks it, creating /dev/mapper/payroll.
3. Make a filesystem on /dev/mapper/payroll and mount it as usual. Everything written is encrypted on the disk.
4. An entry in /etc/crypttab unlocks it at boot, together with the fstab line from Lesson 9.

Where does Argon2 fit? A passphrase is not a key. LUKS turns it into one with a password-based key derivation function, and the cryptsetup manual lists the choices: PBKDF2, Argon2i or Argon2id. RFC 9106 describes Argon2 as a memory-hard function for password hashing: deriving the key deliberately needs a lot of memory as well as time, which makes guessing passphrases with graphics cards or custom hardware far more expensive. LUKS2 supports it; the older LUKS1 format accepts only PBKDF2.

Two limits keep encryption at rest honest. It protects powered-off or stolen storage; once the volume is unlocked on a running server, anyone who compromises that server reads the files normally. And a forgotten passphrase, with no backup key slot or header backup, means the data is gone for good.

The upshot: GPG protects individual files wherever they travel; LUKS2 protects whole volumes at rest; TLS and WireGuard protect data on the wire.

Certificates: who vouches for the server?

Encryption to the wrong party is worthless, which is why the browser checks the payroll server's certificate. A certificate binds a public key to a name and is signed by a certificate authority (CA). Clients accept it if a chain of signatures leads to a trusted root certificate in their trust store. On Linux the system trust store is managed centrally.

• Red Hat family: Red Hat's documentation describes a system-wide truststore containing the Mozilla CA list. Add a CA by copying its certificate to /etc/pki/ca-trust/source/anchors/ and running update-ca-trust.
• Debian family: the update-ca-certificates manual says to copy PEM certificates with a .crt extension into /usr/local/share/ca-certificates/ and run update-ca-certificates, which merges them into /etc/ssl/certs.

Certificates come from two kinds of CA. No-cost CAs such as Let's Encrypt, which describes itself as a free, automated and open certificate authority, issue domain-validated certificates through the ACME protocol. Its FAQ says its default certificates are valid for 90 days and recommends renewing every 60, which is why ACME clients run on a timer, a good use for Lesson 18. Commercial CAs charge, and offer what Let's Encrypt says it has no plans to issue, organization-validated and extended-validation certificates, along with support contracts and warranties.

What the objectives warn against is the self-signed certificate on a production service: a certificate signed by its own key, with no chain to any trusted root. Every client shows a warning, users learn to click through warnings, and an attacker's certificate would produce exactly the same warning. For internal services, run an internal CA and add its root to your machines' trust stores; for public services, use a public CA.

Retiring weak algorithms

Cryptography ages. MD5 and SHA-1 have known collision attacks; NIST announced in 2022 that SHA-1 should be phased out by 31 December 2030 in favor of SHA-2 and SHA-3. RC4, DES and 3DES, export ciphers and the old TLS versions above belong on the same list. Removing them is configuration work: TLS settings in each web and mail server, Ciphers and KexAlgorithms in sshd_config, and on the Red Hat family one central switch. Red Hat's documentation describes system-wide cryptographic policies covering TLS, IPsec, SSH, DNSSEC and Kerberos, with predefined levels DEFAULT, LEGACY, FUTURE and FIPS, set with update-crypto-policies --set. It notes LEGACY allows SHA-1 for compatibility with old systems, and that applications follow the policy unless explicitly configured otherwise.

Common misconceptions

• "A matching SHA-256 hash proves who sent the file." It proves the file is unchanged since the hash was made; an HMAC or a signature is needed to prove origin.
• "Full-disk encryption protects a running server from attackers." Once unlocked, data is readable to anyone who compromises the system; LUKS protects storage that is off or stolen.
• "A self-signed certificate is fine because the traffic is still encrypted." Without a trusted chain, clients cannot tell your server from an impostor's.
• "TLS 1.0 is acceptable for old clients." RFC 8996 deprecated TLS 1.0 and 1.1; upgrade the clients.
• "Free certificates are less secure than paid ones." The encryption is identical; commercial CAs sell extra validation types, support and warranties.

What to remember

• SHA-256 (FIPS 180-4) detects change; HMAC (RFC 2104) adds a secret key to prove a key holder produced the data.
• GPG encrypts files to a recipient's public key (--encrypt --recipient), signs (--sign), verifies (--verify) or uses a passphrase (--symmetric).
• TLS 1.3 (RFC 8446) and strong TLS 1.2 are current; TLS 1.0 and 1.1 are deprecated (RFC 8996); test with openssl s_client; LibreSSL is a 2014 fork of OpenSSL.
• WireGuard is a simple, modern in-kernel VPN configured with keys and AllowedIPs; wg-quick and wg show manage it.
• LUKS2 (cryptsetup luksFormat, open, crypttab) encrypts volumes at rest; Argon2 is the memory-hard function that derives keys from passphrases.
• Trust stores: update-ca-trust (Red Hat) and update-ca-certificates (Debian). No-cost CAs such as Let's Encrypt issue 90-day certificates via ACME; commercial CAs add OV and EV; avoid self-signed certificates in production.
• Remove MD5, SHA-1, RC4, DES and old TLS; on the Red Hat family, update-crypto-policies sets DEFAULT, LEGACY, FUTURE or FIPS.

Sources

1. National Institute of Standards and Technology. (2015). Secure Hash Standard (SHS) (FIPS 180-4). csrc.nist.gov; NIST. (2022). NIST retires SHA-1 cryptographic algorithm. nist.gov
2. Krawczyk, H., Bellare, M., and Canetti, R. (1997). HMAC: Keyed-hashing for message authentication (RFC 2104). RFC Editor. rfc-editor.org
3. GnuPG project. (n.d.). Using the GNU Privacy Guard: Invoking GPG. gnupg.org
4. Rescorla, E. (2018). The Transport Layer Security (TLS) protocol version 1.3 (RFC 8446); Moriarty, K., and Farrell, S. (2021). Deprecating TLS 1.0 and TLS 1.1 (RFC 8996). RFC Editor. RFC 8446; RFC 8996
5. OpenSSL Project. (n.d.). openssl(1). docs.openssl.org; LibreSSL. (2026). LibreSSL. libressl.org
6. Donenfeld, J. A. (n.d.). WireGuard: Fast, modern, secure VPN tunnel. wireguard.com
7. cryptsetup project. (n.d.). cryptsetup-luksFormat(8). Linux man-pages project. man7.org; Biryukov, A., Dinu, D., Khovratovich, D., and Josefsson, S. (2021). Argon2 memory-hard function for password hashing and proof-of-work applications (RFC 9106). rfc-editor.org
8. Red Hat. (n.d.). Using shared system certificates (Securing networks) and Using system-wide cryptographic policies (Security hardening), RHEL 9. docs.redhat.com (certificates); docs.redhat.com (crypto policies)
9. Debian. (n.d.). update-ca-certificates(8). manpages.debian.org; Let's Encrypt. (n.d.). About and FAQ. letsencrypt.org (about); letsencrypt.org (FAQ)

An advisory lands on a Monday

On 1 July 2024 a vulnerability was published as CVE-2024-6387, nicknamed regreSSHion. Its record describes a security regression in OpenSSH's server: a race condition in signal handling that an unauthenticated remote attacker may be able to trigger simply by failing to authenticate within a set time, with a CVSS score of 8.1, High. Nearly every Linux server runs sshd. Within hours, managers everywhere asked their administrators the same question: are we affected, and how do we prove it?

Answering that question well is objective 3.6. It needs a shared language for vulnerabilities, a way to check what is actually installed, evidence that nothing has been tampered with, and standards to measure the whole system against. This lesson works through it with the AlmaLinux machine as the server in question.

The language of vulnerabilities: CVE and CVSS

The CVE program, its site explains, identifies, defines and catalogs publicly disclosed cybersecurity vulnerabilities, giving each one an identifier such as CVE-2024-6387 so that vendors, scanners and administrators all mean the same flaw. CVSS, the Common Vulnerability Scoring System maintained by FIRST, scores severity from 0 to 10. Its specification maps scores to ratings: None 0.0, Low 0.1 to 3.9, Medium 4.0 to 6.9, High 7.0 to 8.9 and Critical 9.0 to 10.0.

A score is a starting point, not a priority list by itself. An 8.1 in a service exposed to the internet outranks a 9.8 in a library that nothing on your server uses. Ask three questions of every advisory: is the vulnerable component installed, is the vulnerable code reachable in your configuration, and is it exposed to attackers?

Are we affected? The backporting trap

The upstream fix for regreSSHion arrived in OpenSSH 9.8p1, whose release notes say the flaw affected Portable OpenSSH from 8.5p1 to 9.7p1. The administrator runs rpm -q openssh-server on the AlmaLinux 9 server, sees a version beginning 8.7p1, and concludes the server is vulnerable. A scanner agrees, flagging every RHEL-family host. Both may be wrong.

Red Hat's page on backporting explains why. Enterprise distributions take a security fix from the newest upstream version and apply it to the older version they ship, so updates change as little as possible. Red Hat warns that looking only at a package's version number will not tell you whether it is vulnerable, and that scanners deciding by version number alone produce false positives. The way to check is the vendor's own data: the advisory for the CVE, or the package's changelog, as in rpm -q --changelog openssh-server | grep CVE-2024-6387. On the Debian family, the security tracker and the package changelog serve the same role.

Red Hat's CVE page also lists a temporary mitigation for RHEL 9: setting LoginGraceTime 0 in sshd_config, which blocks the remote code execution path while leaving sshd exposed to denial of service by exhausting connections. That pattern is common: patch when you can, mitigate until you can, and document both.

So what?: on enterprise distributions, a vulnerability is fixed when the vendor says the package release contains the fix, not when the upstream version number goes up.

What is exposed? Port scanners and protocol analyzers

Vulnerability scanning includes the plainest check of all: which services can attackers reach, and are they configured safely? Lesson 11's tools answer it from outside. A port scanner such as nmap lists open ports and, with version detection, what answers on them. A protocol analyzer such as tcpdump or Wireshark shows what those services actually send, for example a login traveling in clear text. Together they find service misconfigurations: a database listening on every interface instead of localhost, an anonymous FTP upload directory, Lesson 19's open mail relay, a debug page left enabled. Scan only systems you are authorized to test.

Measuring against a standard: CIS Benchmarks and OpenSCAP

Auditors rarely ask "is this server secure?" They ask "does it meet the benchmark?" The CIS Benchmarks, in the Center for Internet Security's words, are prescriptive configuration recommendations for more than 25 vendor product families, representing the consensus of security experts. A Linux benchmark lists hundreds of checks, many of which this course has already taught: mount options for /tmp, SSH settings, disabled filesystems, password policy, audit rules.

Checking hundreds of items by hand is error-prone, so they are written in a machine-readable standard, SCAP, and evaluated with OpenSCAP, whose portal describes an ecosystem of tools for assessment, measurement and enforcement of security baselines. On AlmaLinux, the scap-security-guide package supplies profiles, and one command produces a report:

sudo oscap xccdf eval --profile cis_server_l1 --report report.html /usr/share/xml/scap/ssg/content/ssg-almalinux10-ds.xml

The report lists every rule as pass or fail, with the reason and a fix; profile and file names vary by release, and oscap info on the data stream file lists what is available. Compliance then becomes a cycle: scan, fix the failures that apply, document the exceptions that do not, scan again.

Has anything changed? File integrity

An attacker who gets root usually changes something: a replaced binary, a new setuid file, an extra authorized key. File integrity tools notice.

The rpm manual explains the output of verification as a string of codes, one per failed test: S for size, M for mode, 5 for digest, U for user, G for group, T for modification time, among others. So S.5....T. /usr/bin/ssh means a program binary changed size, content and timestamp since installation, which demands an explanation. A changed configuration file, marked c, is usually expected. The dpkg manual describes --verify as comparing installed files with the metadata in its database.

Two caveats keep this honest. An integrity database stored on the same server can be altered by the same attacker, so keep a copy offline or read-only. And these tools detect change; they do not decide whether the change was malicious. That judgment is yours.

Detection and response: anti-malware and IoCs

Linux servers are less often targeted by classic viruses, but they store and relay files for people who use other systems. ClamAV, whose documentation calls it an open source anti-virus toolkit designed especially for e-mail scanning on mail gateways, scans files with clamscan and updates its signatures with freshclam.

An indicator of compromise (IoC) is a piece of evidence that an attack happened or is under way: a file hash known from a malware report, an IP address or domain used by attackers, an unexpected account, an unfamiliar cron job, a setuid file in /tmp, outbound traffic to an unusual port. Threat reports publish IoCs so defenders can search their logs and filesystems for them. Lessons 32 and 35 put them to work.

The software supply chain

On 29 March 2024, CVE-2024-3094 was published: malicious code in the upstream tarballs of xz, starting with version 5.6.0. Its record describes how, through a series of complex obfuscations, the build process extracted a hidden object file from a disguised test file and used it to modify functions in liblzma, a library linked into other software. CVSS rated it 10.0, Critical. Nobody downloaded a suspicious file; the attack arrived inside a trusted project's release.

That is the software supply chain risk: everything you install is code someone else wrote, built and shipped. Reduce it by installing from signed distribution repositories, limiting third-party repositories (Lesson 17), pinning and reviewing dependencies for language packages, keeping an inventory of what runs where, and watching advisories for the components you rely on.

Destroying data properly

Deleting a file only removes its name, as Lesson 5 explained, so retired disks and files need real destruction. The objectives list four methods.

• shred -u file overwrites a file and then deletes it. Its manual carries a caution: it assumes the filesystem and hardware overwrite data in place, which many platforms do not, and backups or mirrors may keep copies. Journaling filesystems and SSDs often defeat it.
• badblocks -w is a disk test in write mode; its manual says it writes patterns (0xaa, 0x55, 0xff, 0x00) to every block and reads them back. That overwrites the entire device four times, so it doubles as a wipe, and destroys everything on it.
• dd if=/dev/urandom of=/dev/sdX bs=4M status=progress overwrites a whole device with random data, using the special file from Lesson 5. Check the device name twice.
• Cryptographic destruction: on a LUKS volume from Lesson 24, destroying every key slot, for example with cryptsetup erase, leaves only ciphertext that nobody can decrypt. It takes seconds and works on SSDs where overwriting is unreliable, which is why media sanitization guidance such as NIST SP 800-88 treats it as a technique in its own right.

Banners: what users see before and after login

Many organizations must warn users that access is authorized only and monitored, both for legal reasons and to deter. Linux has three places for that text.

Keep pre-login banners free of system details such as the distribution or version, which only help attackers. Legal wording comes from your organization's counsel, not from this course.

Common misconceptions

• "The package version is older than the fixed upstream version, so we are vulnerable." Enterprise vendors backport fixes; check the advisory or changelog.
• "A CVSS score tells us what to patch first." Combine it with whether the component is installed, reachable and exposed.
• "shred guarantees a file is gone." Its own manual warns that journaling, copy-on-write and SSD storage may keep copies.
• "File integrity tools detect attacks." They detect change; you decide whether a change was legitimate, and the database must be protected from the attacker.
• "Packages from trusted projects cannot be malicious." The xz backdoor arrived in official upstream releases; supply chain controls reduce that risk without removing it.

The short version

• CVE names vulnerabilities; CVSS scores them 0-10 (Low, Medium, High from 7.0, Critical from 9.0); prioritize by exposure as well as score.
• Enterprise distributions backport fixes, so check vendor advisories and changelogs (rpm -q --changelog) rather than version numbers; apply documented mitigations until patched.
• Port scanners and protocol analyzers find exposed services and misconfigurations; scan only with authorization.
• CIS Benchmarks define hardened configurations; OpenSCAP evaluates them (oscap xccdf eval) and reports pass or fail.
• AIDE and rkhunter detect changed files and rootkit signs; rpm -Va and dpkg --verify compare installed files with package records; signatures prove package origin.
• ClamAV scans for malware; IoCs such as hashes, addresses and odd accounts guide searches.
• The xz backdoor shows supply chain risk: trusted sources, fewer repositories, inventories and advisories.
• Destroy data with shred (with caveats), badblocks -w, dd from /dev/urandom, or by destroying encryption keys.
• /etc/issue before console login, /etc/issue.net via the SSH Banner option, /etc/motd after login.

Sources

1. CVE Program. (n.d.). Overview; CVE-2024-6387; CVE-2024-3094. cve.org; CVE-2024-6387; CVE-2024-3094
2. OpenSSH project. (2024). OpenSSH 9.8 release notes [Security: race condition in sshd]. openssh.com
3. FIRST. (2023). Common Vulnerability Scoring System version 4.0: Specification document. first.org
4. Red Hat. (n.d.). Security backporting practice and CVE-2024-6387. Red Hat Customer Portal. access.redhat.com (backporting); access.redhat.com (CVE)
5. Center for Internet Security. (n.d.). CIS Benchmarks. cisecurity.org; OpenSCAP. (n.d.). OpenSCAP portal. open-scap.org
6. AIDE project. (n.d.). AIDE: Advanced Intrusion Detection Environment. aide.github.io; Debian. (n.d.). rkhunter(8) and dpkg(1). rkhunter; dpkg
7. RPM project. (n.d.). rpm(8) [Verify options and output]. Linux man-pages project. man7.org
8. Cisco Talos. (n.d.). ClamAV documentation: Introduction. docs.clamav.net
9. GNU coreutils and e2fsprogs. (n.d.). shred(1) and badblocks(8). Linux man-pages project. shred; badblocks; Chandramouli, R., and Hibbard, E. (2025). Guidelines for media sanitization (NIST SP 800-88 Rev. 2). csrc.nist.gov
10. util-linux, OpenSSH and Linux man-pages project. (n.d.). agetty(8), sshd_config(5) [Banner] and motd(5). agetty; sshd_config; motd

A disk alert in nineteen lines

Lesson 7 ended with a filesystem filling up unnoticed. This lesson builds the script that would have warned someone: check-disk.sh, which takes a mount point and a percentage, reads the usage, and reports and exits with a code that a monitoring system can act on. It is nineteen lines long, and almost every line is an item from objective 4.2. Build it line by line on either lab machine, run it after each step, and the syntax will stick.

Here is the finished script first, so you can see where each step is heading:

#!/bin/bash
# check-disk.sh: warn when a filesystem passes a usage limit
mount=${1:-/}
limit=${2:-90}

if [[ ! -d "$mount" ]]; then
    echo "No such directory: $mount" >&2
    exit 2
fi

used=$(df --output=pcent "$mount" | tail -n 1 | tr -dc '0-9')

if (( used >= limit )); then
    echo "WARNING: $mount is $used% full (limit $limit%)"
    exit 1
else
    echo "OK: $mount is $used% full"
    exit 0
fi

Step 1: the interpreter directive and permission to run

The first line, #!/bin/bash, is the interpreter directive, often called the shebang. When you run the file, the kernel reads those first two characters and starts the named program with the script as input. Without it, the script runs in whatever shell launched it, which may not be Bash. Lines beginning # after that are comments.

Save the file, make it executable with chmod +x check-disk.sh, and run it as ./check-disk.sh. The ./ is needed because, as Lesson 3 explained, the current directory is not on PATH.

Step 2: arguments and parameter expansion

A script receives its arguments as numbered parameters: $1, $2 and so on, with $0 the script's own name, $# the number of arguments, and "$@" all of them, each kept as a separate word. Line 3, mount=${1:-/}, uses parameter expansion. The Bash manual calls this form "use default values": if the parameter is unset or null, the expansion of the word after :- is substituted. So the script checks / when no argument is given, and line 4 defaults the limit to 90.

Braces also enable a family of useful transformations, worth knowing by shape:

Braces are also how you separate a name from text that follows it: "${used}MB" means the variable used followed by MB, while "$usedMB" asks for a variable named usedMB.

Step 3: variables, quoting and scope

An assignment is name=value with no spaces around the equals sign; limit = 90 runs a command called limit. Read a value with $name, and quote it almost always: "$mount". Quoting stops the shell splitting a value with spaces into several words and expanding wildcards in it. You will see what goes wrong without quotes in Step 8.

The objectives list six ways to manage variables and shell settings.

• export NAME=value makes a variable part of the environment of commands the script runs (Lesson 3).
• local name=value, which the Bash manual says restricts a variable's scope to the function using it; Lesson 27 uses it.
• unset name removes a variable.
• set changes shell options. In scripts, set -u treats unset variables as errors, set -e exits, in the manual's words, if a pipeline, list or compound command exits with a non-zero status, and set -x prints each command before running it, which Lesson 27 uses for debugging.
• alias and unalias create and remove aliases, but Lesson 3 showed that aliases are not expanded in non-interactive scripts, so scripts use functions instead.

Worth holding on to: assign without spaces, quote every expansion, export only what child processes need, and use local inside functions.

Step 4: capturing output with command substitution

Line 11 runs a small pipeline and stores its output: used=$(df --output=pcent "$mount" | tail -n 1 | tr -dc '0-9'). The df manual lists pcent among the fields --output can print; tail keeps the last line, and tr deletes every character that is not a digit, leaving a bare number such as 47.

The Bash manual describes command substitution as letting a command's output replace the command itself, with two forms: $(command) and the older, deprecated backquote form, a command between two ` characters. Prefer $( ): it nests cleanly and is easy to read.

Related but different is the subshell, a list of commands in parentheses. The manual says it runs in a subshell environment, so variable assignments and directory changes inside do not remain afterwards. (cd /var/log && tar -czf ~/logs.tgz .) archives the logs without changing the script's own working directory.

Step 5: tests

Lines 6 and 13 make decisions, and decisions need tests. Bash has three forms: the old test command, its synonym [ ... ], and the Bash keyword [[ ... ]], which is safer because it does not split variables into words. The conditional expressions in the Bash manual cover files, strings and numbers.

The difference between -lt and < catches people: inside [[ ]], < compares text, so "10" sorts before "9". For numbers use -lt and its siblings, or arithmetic in double parentheses, as line 13 does: inside (( )), <, <=, > and >= compare numerically and variables need no dollar sign.

Step 6: return codes

Every command ends with an exit status: 0 for success and 1 to 255 for kinds of failure. $? holds the status of the last command, and the operators && and || run the next command only on success or only on failure. The script sets its own status with exit: 0 for OK, 1 for warning, 2 for a usage error. That convention matters, because a monitoring agent from Lesson 32 or a systemd timer from Lesson 18 decides what to do from the exit code, never from the text. Try ./check-disk.sh /var 1; echo $? and you will see 1.

Step 7: if and case

The if statement runs a list of commands and branches on its exit status: if test; then ... elif other-test; then ... else ... fi. Any command can be the condition; if grep -q error app.log; then is as valid as a [[ ]] test.

When one value must be compared with many alternatives, case is clearer than a chain of elif lines:

case "$1" in
    start|restart) systemctl restart myapp ;;
    stop)          systemctl stop myapp ;;
    status)        systemctl status myapp ;;
    *)             echo "usage: $0 start|stop|status" >&2; exit 2 ;;
esac

Each pattern ends with ), each branch with ;;, | separates alternatives, * catches everything else, and esac closes it.

Step 8: change one input and watch it break

Now feed the script awkward input. Create a directory called /mnt/team data, with a space, and remove the quotes around $mount on line 11. Run ./check-disk.sh "/mnt/team data" 80. The test on line 6 still passes, because [[ ]] does not split words. But df receives two arguments, /mnt/team and data, and fails. Put the quotes back and it works. That single experiment is the best argument for quoting every expansion.

Then pass a limit of ninety. The arithmetic test treats an unknown word as a variable name and the result is nonsense. Guard against it before using the value: [[ $limit =~ ^[0-9]+$ ]] || { echo "limit must be a number" >&2; exit 2; }. Validating input is the difference between a script and a tool. Lesson 27 continues with loops, functions and regular expressions.

Common misconceptions

• "Spaces around = are fine in an assignment." name = value runs a command named name; write name=value.
• "Quotes around variables are optional." Unquoted values with spaces or wildcards split into several arguments or expand into file names.
• "< compares numbers inside [[ ]]." It compares strings lexicographically; use -lt or arithmetic (( )).
• "A cd inside parentheses changes the script's directory." Parentheses start a subshell; its changes vanish when it ends.
• "Exit codes are only for errors." They are how scripts talk to other programs; 0 means success, anything else a specific kind of failure.

Where this leaves us

• Start scripts with #!/bin/bash, make them executable, and run them with a path such as ./script.sh.
• Arguments arrive as $1, $2, $#, "$@"; braces enable defaults, length, prefix and suffix removal and substitution.
• Assign without spaces, quote expansions, export for child processes, local in functions, unset to remove, set -euo pipefail-style options for safety and set -x for tracing.
• $( ) captures output (the backquote form is deprecated); ( ) runs a subshell whose changes do not persist.
• test, [ ] and [[ ]] check files (-d, -f), strings (-z, -n, ==, !=, <, >, =~) and numbers (-eq, -ne, -lt, -le, -gt, -ge, or (( ))).
• $? holds the last exit status; scripts exit 0 for success and other codes for specific failures.
• if branches on exit status; case matches one value against patterns.

Sources

1. Fox, B., and Ramey, C. (n.d.). bash(1) [Sections: Parameter expansion, Command substitution, Compound commands, Conditional expressions, the set and local builtins]. Linux man-pages project. man7.org
2. Free Software Foundation. (n.d.). Bash reference manual. gnu.org
3. The Open Group. (2024). Shell command language and test. The Open Group Base Specifications Issue 8. pubs.opengroup.org (shell); pubs.opengroup.org (test)
4. GNU coreutils. (n.d.). df(1) and tr(1). Linux man-pages project. df; tr

A script that created the wrong accounts

A help-desk analyst wrote add-users.sh to create accounts for a new intake of staff from a file, users.csv, with lines such as dlee,Dana Lee,webteam. On its first run against the lab server it created accounts called Lee,webteam and dlee,Dana, rejected every valid login name, reported the wrong total, and on a second run waited forever. "Bash is unpredictable," the analyst concluded, and asked for the job to be done by hand.

Bash is not unpredictable; it does exactly what each line says. This lesson debugs the script one bug at a time, and each bug teaches a part of objective 4.2 that Lesson 26 did not: loops, field splitting, regular expressions, functions, and the tools that find such bugs before they run.

Bug 1: looping over words instead of lines

The original loop was:

for line in $(cat users.csv); do ... done

A for loop runs its body once for each word in a list. The command substitution produces the file's text, and the shell then splits that text into words wherever the IFS characters appear. The Bash manual defines IFS, the Internal Field Separator, as the characters used for word splitting after expansion and for splitting lines in the read builtin, with a default of space, tab and newline. So "Dana Lee" became two words, and each became a "line". That explains the strange account names.

The fix reads the file line by line and splits each line on commas:

while IFS=, read -r login fullname group; do ... done < users.csv

Three details make this the standard idiom. The while loop repeats as long as its condition, here read, succeeds, and read fails at the end of the file. Setting IFS=, just for the read command makes it split on commas only, so "Dana Lee" stays one field. And -r, the manual says, stops backslash acting as an escape character, so a backslash in the data stays a backslash.

for loops remain the right tool for lists of words and ranges:

• for host in web01 web02 db01; do ssh "$host" uptime; done
• for f in /var/log/*.log; do gzip "$f"; done, where the wildcard produces one word per file, even names with spaces
• for i in {1..5}; do ...; done and the arithmetic form for (( i=0; i<5; i++ )); do ...; done

break leaves a loop early and continue skips to the next pass.

The core of it: for loops iterate over words, so read files with while IFS= read -r, which splits each line exactly as you choose.

Bug 2: a regular expression that matched nothing

The script checked each login name before creating it:

if [[ $login =~ "^[a-z][a-z0-9_-]{2,15}$" ]]; then

The =~ operator, the Bash manual explains, treats the string on its right as a POSIX extended regular expression. But it adds a rule: if any part of the pattern is quoted, that part is matched literally. The quotes turned the whole regular expression into a literal string, so only a login name that was literally ^[a-z]... would pass. Every real name failed.

The reliable fix is to store the pattern in a variable and use it unquoted:

re='^[a-z][a-z0-9_-]{2,15}$' and then [[ $login =~ $re ]]

Read the pattern piece by piece: ^ anchors at the start; [a-z] is one lowercase letter; [a-z0-9_-]{2,15} is two to fifteen more letters, digits, underscores or hyphens; and $ anchors at the end. Without the anchors, the manual notes, the match succeeds if the pattern matches any part of the string, so "Dana Lee" would pass on its letters. When a match succeeds, the BASH_REMATCH array from Lesson 26 holds the matched text and any parenthesized groups.

The same extended syntax works in grep -E from Lesson 4, so a pattern can be tested on sample data before it goes into a script.

Bug 3: a function that changed the wrong variable

The analyst moved account creation into a function, a named block of commands that the script can call like a command. The Bash manual shows the form name () compound-command, usually written make_user() { ...; }. Inside it, $1, $2 and so on are the function's own arguments, not the script's. The function used a variable called count for a retry counter, and the main script used count for its total. Bash variables are global unless declared otherwise, so the function quietly overwrote the total.

The fix is local, which, as Lesson 26 quoted, restricts a variable's scope to the function. Two other function habits matter:

• A function reports success or failure with return and a number from 0 to 255, just as a script uses exit, so callers can write make_user ... && count=$((count + 1)).
• A function hands back data by printing it, and the caller captures the output with command substitution: home=$(get_home dlee).

Bug 4: a loop that waited forever

To avoid clashing with a nightly job, the script waited while a lock file existed:

until [ -e /run/provision.lock ]; do sleep 5; done

The Bash manual describes until as identical to while except that the test is negated: the body runs as long as the condition fails. So this loop waits until the lock file appears, the opposite of the intent, and when there is no lock it waits forever. Either of these does what was meant: while [ -e /run/provision.lock ]; do sleep 5; done, or until [ ! -e /run/provision.lock ]; do sleep 5; done. Use until when the natural English is "keep trying until it works", as in until ping -c1 db01; do sleep 2; done.

The repaired script

#!/bin/bash
# add-users.sh: create accounts listed in a CSV file
set -u
re='^[a-z][a-z0-9_-]{2,15}$'
count=0

make_user() {
    local login=$1 fullname=$2 group=$3
    if id "$login" &>/dev/null; then
        echo "exists: $login"
        return 1
    fi
    useradd -m -c "$fullname" -G "$group" "$login"
}

while [ -e /run/provision.lock ]; do sleep 5; done

while IFS=, read -r login fullname group; do
    if [[ ! $login =~ $re ]]; then
        echo "skipping bad login: $login" >&2
        continue
    fi
    make_user "$login" "$fullname" "$group" && count=$((count + 1))
done < users.csv

echo "created $count accounts"

Formatting output: OFS

The objectives pair IFS with OFS, the output field separator, which belongs to awk rather than Bash. The gawk manual defines it as the output field separator, a space by default, placed between the expressions of a print statement. So a report of logins and groups, colon-separated, for another system to import is one line: awk -F, 'BEGIN { OFS=":" } { print $1, $3 }' users.csv. -F, sets the input separator and OFS the output one. IFS splits input in the shell; OFS joins output in awk.

Finding bugs before they run

Three tools would have caught these bugs in minutes.

Running ShellCheck is called linting: checking code for likely mistakes without running it. Make it a habit before any script touches a real system, and keep scripts in Git, as Lesson 29 shows, so every fix is recorded. Lesson 31 adds one more reason: when an AI assistant drafts a script for you, linting and tracing are how you check its work.

Common misconceptions

• "for line in $(cat file) reads a file line by line." It iterates over words split by IFS; use while IFS= read -r line.
• "Quoting the regex on the right of =~ is safer." Quoted parts match literally; store the regex in a variable and leave it unquoted.
• "Variables inside a function are private." They are global unless declared with local.
• "until is a loop that runs while its condition is true." It runs while the condition is false, stopping when it becomes true.
• "A regex without anchors checks the whole string." It succeeds if any part matches; anchor it with ^ and $.

Putting it together

• for iterates over words, globbed file names and ranges; while repeats while its condition succeeds; until repeats while it fails; break and continue control a loop.
• IFS sets how the shell splits words; while IFS=, read -r a b c splits each line into fields; OFS sets awk's output separator.
• [[ string =~ $re ]] matches an extended regular expression stored in a variable; anchor with ^ and $; quoted parts match literally.
• Functions take their own $1, $2; declare variables local; report status with return; pass data back by printing it.
• Check syntax with bash -n, trace with bash -x or set -x, and lint with ShellCheck before running scripts on real systems.

Sources

1. Fox, B., and Ramey, C. (n.d.). bash(1) [Sections: Shell variables (IFS), Compound commands (for, while, until, [[ ]] and =~), Shell function definitions, and the read and set builtins]. Linux man-pages project. man7.org
2. Robbins, A. (n.d.). gawk(1) [Built-in variables: OFS]. Linux man-pages project. man7.org
3. Holen, V. (n.d.). ShellCheck: A shell script static analysis tool. GitHub. github.com
4. The Open Group. (2024). Regular expressions. The Open Group Base Specifications Issue 8, Base Definitions, chapter 9. pubs.opengroup.org

The same job, twice

Lesson 26's disk check took nineteen lines of Bash, a pipeline of df, tail and tr, and an arithmetic test. Here is the same job in Python:

#!/usr/bin/env python3
"""check_disk.py: warn when a filesystem passes a usage limit."""
import shutil
import sys

mount = sys.argv[1] if len(sys.argv) > 1 else "/"
limit = int(sys.argv[2]) if len(sys.argv) > 2 else 90

usage = shutil.disk_usage(mount)
used_pct = usage.used / usage.total * 100

if used_pct >= limit:
    print(f"WARNING: {mount} is {used_pct:.0f}% full (limit {limit}%)")
    sys.exit(1)
print(f"OK: {mount} is {used_pct:.0f}% full")

No text is parsed at all. The Python documentation describes shutil.disk_usage as returning disk usage statistics for a path as a named tuple with the attributes total, used and free, in bytes. The program works with numbers, not with columns of text that might change format between distributions. Objective 4.3 asks you to summarize Python's basics for system administration, and this lesson does it by comparison: what Python is for, then its data, its modules and its environments.

Bash or Python?

Most administrators use both, often together: a short shell script that calls a Python tool, or a Python program that runs shell commands. The rule of thumb is to switch to Python when you start fighting the shell's quoting and text splitting, as Lesson 27's bugs did.

Which Python? Current versions

Python 2 is finished. The Python Software Foundation's sunset page states that Python 2 was sunset on 1 January 2020 and will not be improved, even for security problems. Every current distribution runs Python 3, started as python3; check yours with python3 --version. The Python developer guide's status table, read in October 2026, lists 3.15, scheduled for first release on 9 October 2026, and 3.14 in bug-fix status, and 3.11 to 3.13 receiving security fixes only, each with a planned end of life about five years after release. Your distribution's packaged Python may be an older 3.x release that the distribution itself maintains. That is the backporting idea from Lesson 25 again.

The interpreter directive #!/usr/bin/env python3 finds python3 on the PATH, which matters inside virtual environments, below.

Indentation is syntax

Python marks blocks by indentation instead of braces or keywords such as fi and done. The tutorial says the statements forming the body of a function start on the next line and must be indented, and the same holds for if, for, while and with. Inconsistent indentation is not a style problem; it is a syntax error, or worse, a different program. PEP 8, the style guide for Python code, settles the details: use 4 spaces per indentation level, never mix tabs and spaces, and limit lines to 79 characters. It also sets naming conventions, such as function and variable names in lowercase with underscores, like used_pct. Following PEP 8 makes code readable to every other Python programmer, which matters when a colleague maintains your tool at 3 a.m.

Bottom line: use Python 3, indent with four spaces, and follow PEP 8 names and line lengths so others can read your code.

The data types the objectives name

Types matter in ways Bash never forces you to think about. "22" + 1 is an error, because a string is not a number; int("22") + 1 is 23. Division of integers gives a float: in the script above, usage.used / usage.total is a fraction such as 0.47 even though both values are integers. And a dictionary is the natural shape of structured data. Many Linux tools can emit JSON; the ip manual, for example, documents -j for JSON output. So:

import json
import subprocess

result = subprocess.run(["ip", "-j", "address"],
                        capture_output=True, text=True, check=True)
for iface in json.loads(result.stdout):
    addrs = [a["local"] for a in iface.get("addr_info", [])]
    print(iface["ifname"], addrs)

Here json.loads turns the text into a list of dictionaries, each interface's addresses become a list, and nothing depends on column positions. The subprocess documentation names run() as the recommended approach to invoking subprocesses. Passing the command as a list, not one string, avoids the shell and its quoting problems entirely.

Batteries included: built-in modules

Python's standard library covers most administration tasks without installing anything. The ones you will use first:

• os and pathlib: files, directories and paths; shutil: copying, moving and disk usage.
• sys: arguments (sys.argv) and exit codes (sys.exit); argparse: proper command-line options with help text.
• subprocess: running other programs; json and csv: structured data; re: regular expressions like Lesson 27's.
• logging: messages with levels and timestamps; datetime: dates and times; socket: low-level network checks.

A module is a file of Python code you import; a package is a directory of modules. This is how Python is extensible: the tutorial's chapter on modules explains import and packages, and thousands more packages are published on the Python Package Index for tasks the standard library does not cover, such as the requests package for web APIs.

Virtual environments and dependencies

Installing third-party packages is where Python on Linux needs care. Lesson 17 showed that distributions marked externally managed under PEP 668 refuse sudo pip install, because system-wide packages can break tools the distribution itself relies on. The answer is a virtual environment. The venv documentation describes lightweight virtual environments, each with its own independent set of packages, created on top of a base Python and isolated by default from the packages in the base environment.

1. python3 -m venv ~/venvs/netops creates one.
2. source ~/venvs/netops/bin/activate activates it; your prompt changes, and python3 and pip now mean the environment's copies.
3. pip install requests installs into the environment only.
4. pip freeze > requirements.txt records the exact versions, and on another machine pip install -r requirements.txt recreates them. The pip guide describes this use of requirements files as achieving repeatable installs.
5. deactivate returns to the system Python.

A script can also run inside an environment without activating it, by naming the environment's interpreter in its first line or in a systemd unit's ExecStart, such as /home/ops/venvs/netops/bin/python3. Keep requirements.txt in Git with the script, which is where Lesson 29 picks up.

Bottom line: one virtual environment per tool, dependencies installed with pip inside it, and versions pinned in requirements.txt.

Common misconceptions

• "python means Python 3 everywhere." On many systems the command is python3; Python 2 was sunset on 1 January 2020 and must not be used.
• "Indentation in Python is a matter of style." It defines the blocks; wrong indentation is a syntax error or a different program.
• "sudo pip install is the way to add a library for a script." Use a virtual environment; PEP 668 distributions refuse system-wide installs.
• "Calling subprocess with one string is the same as a list." A list runs the program directly without a shell, avoiding quoting and injection problems.
• "Python replaces shell scripts." Each fits different jobs; the shell remains the best glue for running commands.

Pulling it together

• Prefer Bash for gluing commands and Python for structured data, APIs and larger logic.
• Python 2 ended on 1 January 2020; the developer guide lists 3.14 and 3.15 in bug-fix status and 3.11 to 3.13 security-only as of October 2026; check python3 --version.
• Indentation defines blocks; PEP 8 asks for 4 spaces, lines of at most 79 characters and lowercase_with_underscores names.
• Core types: bool, int, float, str, list, dict; convert explicitly, as int("22").
• Standard modules (os, pathlib, shutil, sys, argparse, subprocess with run(), json, re, logging) cover most tasks; packages extend Python further.
• python3 -m venv creates an environment; activate it, pip install inside it, pin with pip freeze into requirements.txt and restore with pip install -r.

Sources

1. Python Software Foundation. (n.d.). venv: Creation of virtual environments, shutil, subprocess and Built-in types. Python 3 documentation. venv; shutil; subprocess; built-in types
2. Python Software Foundation. (n.d.). The Python tutorial: More control flow tools and Modules. control flow; modules
3. van Rossum, G., Warsaw, B., and Coghlan, A. (2001). PEP 8: Style guide for Python code. peps.python.org
4. Python Software Foundation. (2026). Status of Python versions. Python Developer's Guide. devguide.python.org; Sunsetting Python 2. python.org
5. Python Packaging Authority. (n.d.). pip user guide [Requirements files]. pip.pypa.io
6. iproute2 project. (n.d.). ip(8) [The -json option]. Linux man-pages project. man7.org

A week in the life of one repository

By the end of Lesson 28 the lab had a small collection of tools: check-disk.sh, add-users.sh, check_disk.py and a requirements.txt. On Monday they live in one administrator's home directory, and one bad edit or one lost laptop away from disappearing. By Friday they live in a Git repository that two administrators share, with every change recorded, explained and reversible, and a tagged release that the servers deploy. Following that week covers objective 4.4.

Git needs no account and no server; everything in this lesson runs on your lab machines. Hosting services such as GitHub add collaboration features, and they do need an account: GitHub's terms require users to be at least 13 years old. If you are under 18, read the terms with a parent or guardian before signing up for any hosting service.

Monday: a repository and the first commit

Git first needs to know who you are, because every commit records its author: git config --global user.name "Dana Lee" and git config --global user.email dana@example.com. Then, in the tools directory:

1. git init creates an empty repository, a hidden .git directory that will hold the entire history.
2. git status shows every file as untracked.
3. git add check-disk.sh add-users.sh check_disk.py requirements.txt stages the files: it copies their current content into the index, the staging area for the next commit.
4. git commit -m "Add disk check and provisioning scripts" records a commit, a snapshot of everything staged, with the author, a timestamp, the message and a unique hash.

Two things should never enter the repository: the Python virtual environment, which can be rebuilt from requirements.txt, and a file of test passwords. A .gitignore file lists patterns to leave out, such as venv/ and *.secret. The gitignore documentation adds a warning worth remembering: it specifies intentionally untracked files, and files already tracked by Git are not affected. Ignoring a file after committing it does not remove it from history.

Later that day, Dana edits the disk check. git diff shows the changes not yet staged, in the unified format from Lesson 5; git diff --staged shows what the next commit will contain. git log --oneline lists commits, newest first.

Key idea: the working tree is what you edit, the index is what you have staged with add, and a commit is a permanent snapshot of the index.

Tuesday: a branch for an experiment

Dana wants to add email alerts to the disk check without breaking the working version. That is what branches are for: a branch is a movable name for a line of commits.

• git branch email-alerts creates a branch; git branch lists them, marking the current one.
• git checkout email-alerts switches to it, changing the files in the working tree to match. Newer Git also offers git switch for this one job, but the objectives name checkout, and git checkout -b name creates and switches in one step.
• She commits twice on the branch. The main branch is untouched.

When the feature works, she merges it: git checkout main, then git merge email-alerts. If main has not moved, Git simply moves main forward; if both have new commits, Git creates a merge commit joining the two histories. Where the same lines changed on both sides, Git stops with a merge conflict, marks the file, and waits for a person to choose and commit the result.

Her branch contained "wip" and "fix typo" commits nobody needs in the main history. A squash merge combines them: the git-merge documentation says --squash produces the working tree and index as if a merge happened, but does not make a commit or record a merge, so the next git commit creates one single commit on top of main.

Wednesday: sharing with clone, fetch, pull and push

A second administrator, Raj, joins. The repository needs a shared copy, a remote. In the lab, a bare repository on the other VM serves perfectly: git init --bare /srv/git/tools.git there, then git remote add origin ops@192.168.56.20:/srv/git/tools.git and git push -u origin main from Dana's machine, over SSH from Lesson 21. Raj runs git clone ops@192.168.56.20:/srv/git/tools.git and has the full history.

The difference between fetch and pull matters on a server. fetch lets you look before anything changes: git fetch, then git log main..origin/main to see what arrived, then merge when ready. pull does both at once.

Thursday: an interruption, and a tidier history

Raj is halfway through editing add-users.sh when an urgent fix is needed on main. He cannot commit half-finished work, and switching branches with uncommitted changes is risky. The git-stash documentation describes exactly this case: git stash records the current state of the working directory and index and returns to a clean working directory. He fixes the bug, commits and pushes it, then runs git stash pop to bring his half-finished edit back.

Meanwhile his own feature branch has fallen behind main. He could merge main into it, but he prefers a straight line of history, so he uses rebase: git rebase main on his branch replays his commits on top of the latest main, as if he had started from there. Rebase rewrites commits, creating new ones with new hashes. Pro Git sums up its one rule: do not rebase commits that exist outside your repository and that people may have based work on. Rebase private branches freely; never rebase something you have already pushed and others have pulled.

Friday: undoing a mistake, and a release tag

Before the release, Dana commits a change, then realizes it was wrong and has not been pushed. git reset moves the branch back, with three modes the documentation distinguishes:

• git reset --soft HEAD~1 leaves the working tree and index unchanged; the undone commit's changes stay staged, ready to recommit.
• git reset HEAD~1, the default mixed mode, unstages them but keeps them in the files.
• git reset --hard HEAD~1 overwrites files to match the earlier commit; the documentation warns it may also overwrite untracked files. Uncommitted work is lost.

For a commit that has already been pushed, rewriting history causes the same problems as rebasing shared commits; the safer tool there is git revert, which adds a new commit undoing the old one.

Finally, the release. A tag names one commit permanently: git tag -a v1.0 -m "First release of the ops tools". Pro Git recommends annotated tags like this one, created with -a, because they are stored as full objects with the tagger, date and message and can be signed with GPG. It also notes that git push does not transfer tags by default, so publish it with git push origin v1.0. The servers can now deploy exactly v1.0, whatever happens on main next week.

Key idea: branches and merges organize work, fetch, pull and push share it, stash and reset handle interruptions and local mistakes, rebase only private work, and tags mark releases.

Common misconceptions

• "Adding a file to .gitignore removes it from the repository." gitignore affects only untracked files; a committed file stays in history until removed with git rm, and leaked secrets must be rotated.
• "git fetch updates my files." It only downloads commits and updates remote-tracking branches; merging or pulling changes your branch.
• "Rebasing is always cleaner than merging." Rebasing shared, pushed commits forces everyone else to repair their history.
• "git reset --hard is a safe undo." It discards uncommitted changes and can overwrite untracked files.
• "Tags are pushed with the branch." git push does not send tags by default; push them by name or with --tags.

What you now know

• git config sets your identity; git init creates a repository and git clone copies one.
• git add stages changes into the index; git commit snapshots them; git status, git diff and git log show state, changes and history.
• .gitignore keeps untracked files such as virtual environments and secrets out, but does not affect tracked files.
• git branch creates branches, git checkout switches, git merge joins them, and merge --squash turns a branch into one commit.
• git fetch downloads, git pull fetches and integrates, git push uploads.
• git stash shelves work in progress; git rebase replays commits on a new base (never on shared history); git reset moves a branch back in soft, mixed or hard mode.
• git tag -a creates an annotated release tag; push it explicitly.

Sources

1. Git project. (n.d.). Git reference documentation: git-pull, git-fetch, git-merge, git-reset, git-stash and gitignore. git-pull; git-fetch; git-merge; git-reset; git-stash; gitignore
2. Chacon, S., and Straub, B. (2014). Pro Git (2nd ed.) [Chapters 2.6 Tagging and 3.6 Rebasing]. Apress. git-scm.com (tagging); git-scm.com (rebasing)
3. GitHub. (n.d.). GitHub terms of service [Account requirements]. docs.github.com

The drift nobody wrote down

A team runs 40 Linux servers. Last month someone ended an outage by editing /etc/ssh/sshd_config by hand on three of them, and nobody recorded the change. Now those three behave differently from the other 37, and nobody remembers why. This slow divergence from the intended setup is called configuration drift, and it is the problem infrastructure as code exists to solve.

Infrastructure as code (IaC) means describing servers, networks and their settings in text files that a tool applies for you. The files live in Git (Lesson 29), so every change is reviewed, recorded and reversible. The files, not anyone's memory, become the record of how systems should look. Good IaC tools are idempotent: you describe the desired state, and the tool changes only what differs. The Ansible documentation puts it plainly: when a system is already in the state your playbook describes, Ansible changes nothing, however many times the playbook runs.

The team agrees on IaC. They disagree about how to deliver it. Dana wants Ansible: "No agents to install, just SSH." Raj wants Puppet: "An agent on every server catches drift by itself." Both are right about something.

The case for agentless push: Ansible

Ansible runs from a control node, such as an administrator's workstation, and connects to managed nodes over SSH with existing OS credentials. Its installation guide says managed nodes do not need Ansible installed. They need Python, to run the code Ansible sends, and a user account that can log in over SSH. Nothing else runs on the servers between changes.

• An inventory lists the managed hosts, sorted into groups. The two built-in formats are INI and YAML, and the classic default file is /etc/ansible/hosts.
• Modules are the units of work: ansible.builtin.dnf installs packages, ansible.builtin.service manages services, ansible.builtin.copy places files. Most modules check the current state first and do nothing if it already matches.
• A playbook is a YAML file holding one or more plays. Each play targets a group of hosts and runs its tasks in order, top to bottom, and each task calls a module.
• Ad hoc commands run a single module without a playbook. The documentation recommends them for tasks you repeat rarely, such as ansible webservers -m ansible.builtin.service -a "name=httpd state=restarted".
• Facts are what Ansible discovers about each host: the OS, IP addresses, filesystems and more, stored in the ansible_facts variable. Playbooks use facts in conditions, for example to install with dnf on AlmaLinux but apt on Ubuntu.
• Collections are the distribution format for Ansible content. One collection can hold playbooks, roles, modules and plugins, and the ansible-galaxy command installs collections from a server such as Ansible Galaxy.

Here is an inventory for the lab, followed by a playbook that keeps time synchronized on the AlmaLinux machine:

[rhel]
alma-lab ansible_host=192.168.56.20

[ubuntu]
ubuntu-lab ansible_host=192.168.56.10

- name: Keep time in sync
  hosts: rhel
  become: true
  tasks:
    - name: chrony is installed
      ansible.builtin.dnf:
        name: chrony
        state: present
    - name: chronyd is running and enabled
      ansible.builtin.service:
        name: chronyd
        state: started
        enabled: true

Dana's case rests on two strengths. First, nothing extra runs on the servers, so there is nothing to patch, monitor or secure apart from SSH. Second, she decides exactly when changes happen. Running ansible-playbook --check reports what would change without changing it, and ansible-lint flags risky patterns before anything runs.

The case for an agent that pulls: Puppet

Raj's tool works the other way round. In Puppet's agent-server architecture, every managed node runs an agent. Periodically the agent sends its facts to the Puppet server and requests a catalog. Facts come from Facter, Puppet's system profiling tool. The server compiles a catalog, the desired state for that one machine, and the agent applies it. Any resource that has drifted is put back, and the agent sends a report to the server. The documentation gives the default as one run every 30 minutes, set by runinterval in puppet.conf.

So the three hand-edited sshd_config files would have been corrected within half an hour, and the reports would show exactly what changed. That is Raj's case: drift gets fixed without anyone remembering to run anything.

• Classes are named blocks of Puppet code, stored in modules and applied only when declared for a node. A class usually configures one chunk of functionality, such as everything for SSH.
• Modules are self-contained bundles of code and data with a fixed directory structure, loaded from the modulepath.
• Certificates secure the conversation. Agent and server talk over HTTPS, and each side checks the other's certificate. Puppet includes its own certificate authority. A new agent requests a certificate, and an administrator signs it, or autosigning rules approve it, before the agent receives any configuration.
• Agentless options exist too. puppet apply compiles and applies code locally with no server, which the documentation calls the stand-alone architecture. Bolt connects over SSH or WinRM and runs tasks with no agent at all.

One piece of recent history matters when you install it. According to the Vox Pupuli community, Perforce, which owns Puppet, stopped publishing public open-source packages in late 2024 and moved development to internal forks. The community now maintains a compatible fork called OpenVox, whose documentation this lesson cites. Its commands are still called puppet, and its fact tool, OpenFact, continues Facter.

What settles it

The argument turns out to mix several separate questions.

Neither default is fixed. The ansible-pull command inverts Ansible's model: each node checks out a Git repository of playbooks and runs them against itself, so a timer can make Ansible pull. Puppet runs without its server through puppet apply or Bolt. Many organizations use both kinds of tool: one for continuous enforcement, and Ansible for orchestrated one-off work such as rolling reboots.

The point: choose by need, not by brand. If drift must be corrected automatically, something has to run on a schedule; if you cannot install agents, use SSH. Either way, the code lives in Git and every change goes through review.

Building the machines: OpenTofu

Ansible and Puppet configure machines that already exist. Something has to create the virtual machines, networks and DNS records in the first place, and on cloud platforms that is done with code too. OpenTofu is an infrastructure as code tool for defining cloud and on-premises resources in configuration files. It began as a fork of Terraform after HashiCorp moved Terraform from an open-source license to the Business Source License, and its backers gave it to the Linux Foundation. Its command is tofu.

• Providers are plugins that talk to a platform's API (application programming interface): a cloud, a software service, or anything else with an accessible API. tofu init installs the providers a configuration declares.
• Resources are the heart of the language. Each resource block describes one or more infrastructure objects, such as a virtual network, a compute instance or a DNS record.
• State is OpenTofu's record of which real objects belong to which resource blocks. By default it is a local file named terraform.tfstate. Before each operation OpenTofu refreshes the state against the real infrastructure, then uses it to decide what to change.

The workflow is write, plan, apply. tofu plan describes what would be created, updated or destroyed; tofu apply carries out the changes once you approve them. The configuration below uses the local provider, which manages files on your own machine, so you can practise without any cloud account:

terraform {
  required_providers {
    local = {
      source = "hashicorp/local"
    }
  }
}

resource "local_file" "motd" {
  filename = "motd.txt"
  content  = "Managed by OpenTofu. Do not edit by hand."
}

State deserves care. It can contain sensitive values such as access keys, and OpenTofu can encrypt state and plan files for that reason. Its documentation recommends keeping state somewhere that versions, encrypts and securely shares it, so add *.tfstate to .gitignore instead of committing it. Losing state is serious too, because OpenTofu would no longer know which real objects it manages. One generated file does belong in Git: the dependency lock file, .terraform.lock.hcl, which records the provider versions in use.

Installing without a person: Kickstart and cloud-init

Before any of these tools can connect, a machine needs an operating system. Unattended deployment answers the installer's questions in advance.

Kickstart is the Red Hat family's method, and AlmaLinux uses it too. A Kickstart file holds every installer answer: language, disk layout, network, packages and users, plus optional %pre and %post script sections. You rarely write one from scratch. Red Hat's documentation notes that each interactive installation saves its choices to /root/anaconda-ks.cfg, which you can copy, edit and reuse. That file includes user and password information, so protect every copy.

• ksvalidator, from the pykickstart package, checks a file's syntax and flags deprecated options. It does not check the contents of the %pre, %post and %packages sections.
• The inst.ks= boot option tells the installer where to fetch the file, for example from a web server or a local volume.

cloud-init solves a different problem. Cloud servers are usually not installed at all; they are copied from a prepared image, and each copy needs its own hostname, users and SSH keys. The cloud-init documentation calls it the industry standard multi-distribution method for cross-platform cloud instance initialization. During boot it identifies the platform it is running on, called its datasource, and works through five stages: Detect, Local, Network, Config and Final. Its main input is user-data, most often YAML that starts with #cloud-config:

#cloud-config
hostname: web01
users:
  - name: ops
    groups: [wheel]
    ssh_authorized_keys:
      - ssh-ed25519 AAAA...example ops@laptop
packages:
  - chrony
runcmd:
  - systemctl enable --now chronyd

For local virtual machines, the NoCloud datasource reads user-data and meta-data from a small disk whose volume label is CIDATA, and meta-data must contain an instance-id. cloud-init status --long reports whether it finished and lists any errors, and cloud-init schema validates a cloud-config file before you rely on it. Ubuntu Server's installer has its own unattended format, autoinstall, which is also YAML and takes the place of debian-installer preseeding.

Secrets and the repository

Every tool in this lesson tempts you to put secrets in files. Kickstart files carry password hashes, playbooks need database passwords, state can hold access keys, and user-data carries keys and accounts. Three habits help:

• Never commit plain secrets. Lesson 29 showed that removing a secret from history is hard, and the secret must be changed anyway.
• Encrypt what must live in the repository. Ansible Vault encrypts variables and files, and OpenTofu can encrypt state and plan files.
• Prefer SSH keys and password hashes to plain passwords, and limit who can read generated files such as anaconda-ks.cfg.

Common misconceptions

• "Agentless means the server needs nothing." Ansible still needs an SSH login and, for most modules, Python on the managed node.
• "IaC tools just run commands, like a shell script." They declare a desired state and change only what differs, which is why running them twice is safe.
• "Puppet only works with an agent and a server." puppet apply and Bolt both work without either.
• "The state file is a disposable cache." It maps real resources to your code; losing it or leaking it causes real trouble.
• "cloud-init installs the operating system." It configures an already-installed system during boot; Kickstart and autoinstall perform installations.

The takeaway

• Infrastructure as code keeps the desired state of systems in version-controlled files that tools apply idempotently, which stops configuration drift.
• Ansible is agentless and pushes over SSH, using inventories, modules, YAML playbooks, ad hoc commands, facts and collections.
• Puppet's agents pull by default: they send facts, receive catalogs built from classes and modules, and trust the server through certificates. puppet apply and Bolt work without agents.
• OpenTofu provisions infrastructure through providers that call platform APIs. Resources describe objects, and state maps them to reality, so state must be protected.
• Kickstart answers the Red Hat family installer; cloud-init configures images at boot from user-data.
• Keep secrets out of repositories, or encrypt them.

Sources

1. Ansible project. (n.d.). Ansible community documentation: Introduction to Ansible; Installation guide; How to build your inventory; Ansible playbooks; Introduction to ad hoc commands; Discovering variables: facts and magic variables; Using Ansible collections. introduction; installation; inventory; playbooks; ad hoc commands; facts; collections
2. OpenVox Project. (n.d.). OpenVox documentation: architecture overview; Language: Classes; Module fundamentals; OpenVox agent on *nix; OpenFact; OpenBolt. architecture; classes; modules; agent; OpenFact; OpenBolt
3. Vox Pupuli. (n.d.). OpenVox automation framework. voxpupuli.org
4. OpenTofu. (n.d.). OpenTofu documentation: What is OpenTofu?; Providers; Resources; State; State and plan encryption; Dependency lock file; Frequently asked questions. introduction; providers; resources; state; encryption; lock file; FAQ
5. Red Hat. (n.d.). Automatically installing RHEL (Red Hat Enterprise Linux 10). docs.redhat.com
6. Canonical and cloud-init contributors. (n.d.). cloud-init documentation: overview; Boot stages; Cloud config; NoCloud; CLI commands. overview; boot stages; cloud config; NoCloud; CLI
7. Canonical. (n.d.). Introduction to autoinstall. Ubuntu installation documentation. canonical-subiquity.readthedocs-hosted.com

The Friday change

At 4:50 on a Friday, an administrator needs a small change to a web server's configuration. An AI assistant drafts it in seconds. It looks right, so she pastes it onto the production host, restarts the only web container, and goes home. The container never comes back. The assistant had used a configuration directive that does not exist in that version of the server, and the one container ran on the one host. Nobody saw the change before it ran, nothing tested it, and nothing could take over.

Each part of that failure has a standard fix, and together they make up objectives 4.1 and 4.5. Pipelines test a change before it lands. Deployment is driven from Git. Orchestration keeps copies running. And a few habits make AI output safe to use.

Catch it before it lands: CI/CD and shift-left testing

NIST describes CI/CD pipelines as workflows that take source code through stages such as building, testing, packaging, deployment and operations, supported by automated tools with feedback. For an administrator the pieces are:

• Version control. Every change, whether a script, a playbook, a Compose file or a Kubernetes manifest, is a commit (Lesson 29), usually made on a branch and reviewed before it merges.
• Continuous integration (CI). Automated checks run on every commit: syntax checks, linters such as ShellCheck and ansible-lint, tests, and builds of container images.
• Continuous delivery and continuous deployment (both CD). Continuous delivery keeps every merged change ready to release. Continuous deployment goes further and releases each change that passes, with no manual step.
• Pipeline. The chain of those stages, defined as code in the repository and run by a CI server or service.

Shift-left testing means moving tests earlier, to the left on a timeline that runs from writing code to production. Microsoft's DevOps guidance describes the aim as performing testing tasks earlier in the pipeline, so that most testing is finished before a change merges into the main branch. The earliest point of all is your own machine. A Git pre-commit hook runs before every commit, and the githooks documentation says a non-zero exit from it aborts the commit:

#!/bin/bash
# .git/hooks/pre-commit: refuse commits that fail basic checks
set -e
for f in *.sh; do
  bash -n "$f"
  shellcheck "$f"
done
ansible-playbook --syntax-check site.yml
ansible-lint site.yml

A test stage running nginx -t (Lesson 19) would have rejected the Friday change in seconds, on a branch, long before production.

DevSecOps builds security into every stage instead of leaving it for a review at the end. NIST's SP 800-204C presents it as the paradigm for cloud-native applications, with CI/CD pipelines among its basic building blocks. Typical additions to a pipeline include scanning dependencies and container images for known CVEs (Lesson 25), detecting secrets committed by mistake, and checking infrastructure code against policy before it is applied.

What matters here: the cheapest place to catch a mistake is the earliest one. Every check that runs automatically before a merge is a production outage that never happens.

Let Git drive deployment: GitOps

GitOps extends the same idea to running systems. The OpenGitOps project defines four principles:

1. Declarative: the desired state of the system is expressed declaratively.
2. Versioned and immutable: that desired state is stored with versioning, immutability and a complete history, which is what Git provides.
3. Pulled automatically: software agents pull the desired state from that source.
4. Continuously reconciled: the agents keep observing the actual state and work to apply the desired state.

Argo CD and Flux are tools that do this for Kubernetes clusters. For administrators, the consequence is that nobody runs deployment commands against production by hand. To change production you merge a reviewed change into the repository, and to roll back you revert the commit. Notice the family resemblance to Puppet's agents in Lesson 30: pulling plus continuous reconciliation is what defeats drift.

Keep it running: orchestration

A container on one host dies with that host. Orchestration tools run containers across a cluster, restart failed ones, spread copies over machines and route traffic to healthy ones. The objectives name three tools: Compose for several containers on one host, Docker Swarm for a cluster with familiar Docker commands, and Kubernetes for full-scale clusters.

Compose: one host, many containers

A Compose file describes a multi-container application in YAML: its services, networks and volumes. Docker's documentation names compose.yaml in the working directory as the preferred default file. Three commands cover most daily use:

• docker compose up -d creates and starts everything in the background. Run again, it picks up changes to the file or the image.
• docker compose logs -f web follows the output of one service.
• docker compose down stops and removes the containers and networks that up created. Named volumes survive unless you add -v.

services:
  web:
    image: docker.io/library/nginx:stable
    ports:
      - "8080:80"
    environment:
      - TZ=UTC
    volumes:
      - site:/usr/share/nginx/html
volumes:
  site:

Podman users run the same files with podman compose. The Podman documentation describes it as a thin wrapper around an external compose provider, such as docker-compose or podman-compose.

Docker Swarm: a cluster with familiar commands

Swarm mode joins several Docker Engines into one cluster. Its key terms come straight from Docker's documentation:

Services are either replicated, with a set number of tasks spread across nodes, or global, with one task on every available node, which suits a monitoring agent. Nodes must reach each other on TCP port 2377 for cluster management, TCP and UDP port 7946 for overlay node discovery, and UDP port 4789 for overlay traffic, so open those in the firewall from Lesson 23.

Kubernetes: everything is a declared object

Kubernetes goes further than Swarm. Everything is an object described in YAML, and controllers keep reality matching the description. These are the objects the objectives name:

• Pod: the smallest deployable unit, a group of one or more containers with shared storage and network resources.
• Deployment: manages a set of pods for an application, usually one that keeps no state. You describe the desired state, such as three replicas of version 1.2. The Deployment controller then changes the actual state to match at a controlled rate, which is how rolling updates work.
• Service: exposes an application running as one or more pods behind one stable name and address, while individual pods come and go.
• ConfigMap: stores non-confidential settings as key-value pairs. Pods consume them as environment variables, command-line arguments or files in a volume. The documentation warns that a ConfigMap provides no secrecy or encryption.
• Secret: holds a small amount of sensitive data, such as a password, token or key. Values in its data field are base64-encoded, which is an encoding, not encryption. By default, Secrets are stored unencrypted in the cluster's etcd data store, so the documentation recommends encryption at rest and least-privilege access rules.
• Volume: lets the containers in a pod access and share data through the filesystem. Ephemeral volumes end with the pod; persistent volumes outlive it.
• Variables: environment variables are set per container with the env or envFrom fields, either directly or from ConfigMaps and Secrets.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:stable
          env:
            - name: TZ
              value: UTC

Running kubectl apply -f web.yaml submits it, kubectl get pods shows three pods, and kubectl logs reads a pod's output. Delete one pod by hand and the Deployment creates a replacement. Desired state wins, the same idea as Puppet and GitOps.

Using AI assistants responsibly

The Friday change began with an AI assistant, but the assistant was not really the problem. Pasting its output without checking was. CompTIA takes the topic seriously enough to list access to a large language model among its suggested practice resources, as Lesson 1 noted. Objective 4.5 covers what assistants are good for and the habits that make them safe. Common uses for an administrator include:

• Generating code: a first draft of a Bash or Python script, a regular expression, or infrastructure as code such as a playbook, Compose file or Kubernetes manifest.
• Explaining and documenting: commenting an inherited script, writing a README, or explaining a cryptic regex or ShellCheck warning.
• Reviewing: suggestions for optimizing code, linting-style feedback, a security review of a configuration, or recommendations for meeting a compliance baseline such as a CIS Benchmark (Lesson 25).

The risks are just as concrete. NIST's Generative AI Profile names confabulation: confidently stated but erroneous or false content, colloquially called hallucination. It also warns about automation bias and over-reliance. In administration work, confabulation looks like commands, options, directives and packages that do not exist. A study by Spracklen and colleagues, presented at USENIX Security 2025, generated 576,000 code samples with 16 models. On average, at least 5.2% of the packages that commercial models suggested did not exist, and 21.7% for open-source models. An attacker who publishes malware under a commonly invented name is then installed by anyone who trusts the suggestion.

A second risk appears when an assistant reads untrusted content, such as a web page, an email or a log file. Text hidden in that content can redirect the assistant. This is prompt injection, first on OWASP's 2025 list of risks for LLM applications. Never give an assistant root access or production credentials on the strength of its good behaviour so far.

The best practices follow directly:

• Never copy and paste without review and quality assurance. Read every line until you can explain it. Treat the output as a draft from a capable but unreliable colleague.
• Verify the output. Check options against man pages and confirm that packages exist in your distribution's repositories. Then run the shift-left checks from this lesson: bash -n, ShellCheck, ansible-lint, --check runs, tofu plan, nginx -t. Test in a lab VM or snapshot, never first on production.
• Keep a human accountable. A named person reviews and approves AI-assisted changes, exactly as for any other change.

Data governance: what may go into a prompt

Whatever you paste into a hosted assistant leaves your control. Data governance asks four questions first:

• Security of data. Never paste secrets, private keys, password hashes, customer data or anything your organization classifies as confidential. Replace real hostnames, addresses and names with placeholders.
• LLM training. Whether a provider may use your input to train future models depends on the provider, the plan and your settings. For example, Anthropic lets users of its consumer plans choose whether their chats are used for training, and says that change does not apply to its commercial or API services. Read the terms for the plan you actually use.
• Private or public. A public consumer service differs from a private deployment run for your organization under contract, which can set rules on training and data retention.
• Local models. Open-weight models can run on hardware you control, with tools such as llama.cpp, so prompts never leave the machine. The costs are hardware, and possibly quality: in the package study above, open-source models invented packages far more often.

Above all, adhere to corporate policy. Organizations commonly approve specific AI tools, forbid others, and set rules for what data may be shared and how AI-assisted work is labelled. Using an unapproved assistant on company code can break policy even when the output is perfect.

Prompt engineering for administrators

Prompt engineering is the craft of writing requests that get useful answers. Anthropic's prompting guide stresses being clear and direct, adding context, and being specific about the output format and constraints. Its test works for any assistant: if a colleague with little context would be confused by your prompt, the model will be too. For administration work that means:

• State the environment: distribution and version, shell, tool versions, and whether SELinux or a particular firewall is in use.
• State the constraints: idempotent, no new packages, must pass ShellCheck, runs as a non-root user.
• Ask for the reasoning: an explanation of each line, the assumptions made, and a way to test the result.
• Give examples of the input, such as three sample log lines for a regex, plus lines it must not match.

A better prompt produces a better draft, but it never removes the need to verify. If you want to try an assistant, check its terms first: each service needs an account and sets a minimum age, and Anthropic's consumer terms, for example, require users to be at least 18. If you are under 18, read any terms with a parent or guardian. This lesson's lab works without an assistant.

Common misconceptions

• "CI/CD means every commit goes straight to production." Continuous delivery keeps changes ready to release; only continuous deployment releases automatically, and both depend on checks passing first.
• "Kubernetes Secrets are encrypted." By default they are base64-encoded and stored unencrypted in etcd; encryption at rest and access rules must be configured.
• "docker compose down deletes my data." It removes containers and networks; named volumes stay unless you pass -v.
• "If the AI's script runs without errors, it is correct." A script can run cleanly and still do the wrong thing, or install a package that should not exist.
• "A local model makes AI use compliant." It settles where the data goes, not whether the output is right or whether policy allows the use.

Summing up

• CI/CD pipelines take changes from version control through build, test and deployment stages; shift-left testing runs checks as early as possible, even in a pre-commit hook.
• DevSecOps builds security checks into every stage. GitOps makes Git the declarative, versioned source that agents pull and continuously reconcile.
• Compose runs multi-container applications on one host with compose.yaml, up, down and logs.
• Docker Swarm runs services as tasks on manager and worker nodes, with overlay networks, replicated or global modes, and scaling.
• Kubernetes runs pods through Deployments, exposes them with Services, configures them with ConfigMaps, Secrets and environment variables, and stores data in volumes.
• AI assistants can draft code, regexes and IaC, write documentation, and review or lint, but their output needs review and verification.
• Protect data, know whether inputs are used for training, prefer private or local options where needed, and follow corporate policy.

Sources

1. Chandramouli, R. (2022). Implementation of DevSecOps for a microservices-based application with service mesh (NIST SP 800-204C). National Institute of Standards and Technology. csrc.nist.gov
2. Microsoft. (n.d.). Shift testing left with unit tests. Microsoft Learn. learn.microsoft.com
3. Git project. (n.d.). githooks: Hooks used by Git. git-scm.com
4. OpenGitOps. (n.d.). GitOps principles v1.0.0. opengitops.dev; see also Argo CD and Flux documentation.
5. Kubernetes. (n.d.). Kubernetes documentation: Pods; Deployments; Service; ConfigMaps; Secrets; Volumes; Define environment variables for a container. pods; deployments; services; ConfigMaps; Secrets; volumes; environment variables
6. Docker. (n.d.). Docker documentation: Swarm mode key concepts; Getting started with Swarm mode; Use Swarm mode routing mesh; Overlay network driver; Scale the service; The Compose application model; docker compose down. key concepts; tutorial and ports; routing mesh; overlay; scaling; Compose model; compose down
7. Podman project. (n.d.). podman-compose(1). docs.podman.io
8. Autio, C., Schwartz, R., Dunietz, J., Jain, S., Stanley, M., Tabassi, E., Hall, P., and Roberts, K. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). National Institute of Standards and Technology. nist.gov
9. Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., and Jadliwala, M. (2025). We have a package for you! A comprehensive analysis of package hallucinations by code generating LLMs. USENIX Security Symposium 2025. arXiv:2406.10279
10. OWASP GenAI Security Project. (2024). OWASP Top 10 for LLM applications 2025. genai.owasp.org
11. Anthropic. (2025, August 28). Updates to consumer terms and privacy policy; and (n.d.) Consumer terms of service. update; terms
12. Anthropic. (n.d.). Prompting best practices. Claude Developer Platform documentation. platform.claude.com
13. ggml-org. (n.d.). llama.cpp: LLM inference in C/C++. GitHub. github.com