Linux Mastery Roadmap — Zero to Advanced

Phase 1 · Foundations — The Zero Mark

Understand the OS, navigate the filesystem, manipulate files confidently from the CLI.

Core Concepts

The mental model: Kernel, Shell, User Space, and where everything lives.

Kernel vs. Shell vs. User Space: understand that the kernel manages hardware/processes, the shell (bash/zsh) interprets your commands, and user space is everything running on top.
FHS — Filesystem Hierarchy Standard: learn the purpose of /etc (config), /var (variable data — logs, mail, spools), /usr (read-only user programs), /home (user home dirs), /root (root's home), /dev (device files), /proc & /sys (kernel virtual FS), /tmp (ephemeral).
Everything is a file: regular files, directories, symlinks, device files, sockets, named pipes. Inspect with ls -l — the first character of the mode field tells you the type.
Absolute vs. relative paths: /home/user/file vs ./file vs ../file. Understand ~ (home) and - (previous dir).

CLI Essentials

Navigation, file manipulation, viewing — the daily 80%.

Navigate: pwd, cd, ls -lah (long, all, human-readable sizes), ls -lt (newest first), tree -L 2.
Manipulate: mkdir -p a/b/c, touch file, cp -r src dst, mv old new, rm -i (interactive — safer than rm -rf).
Read: cat, less (q to quit, /pattern to search), head -n 20, tail -f /var/log/syslog (follow log live).
Find & filter: find . -name "*.log" -mtime -7, grep -rn "pattern" /etc, pipes & redirection cmd1 | cmd2 > out.txt 2>&1.
Help: man cmd, cmd --help, info cmd, and tldr cmd (community examples — third-party, install separately).

🎯 Action & Verification — "Project Scaffold"

Build, copy, archive, verify. The whole loop.

Create a nested project layout under ~/lab/projectA/{src,docs,logs,tests} using a single mkdir -p command with brace expansion.
Generate 5 empty source files inside src/ and a README in docs/.
Copy the entire ~/lab/projectA to ~/lab/backup/ preserving timestamps with cp -rp.
Archive with tar -czvf projectA.tar.gz projectA/ then inspect contents with tar -tzf projectA.tar.gz.

# Build the structure in one shot
mkdir -p ~/lab/projectA/{src,docs,logs,tests}
touch ~/lab/projectA/src/{main.py,utils.py,config.py,db.py,api.py}
echo "# Project A" > ~/lab/projectA/docs/README.md

# Verify file count and tree structure
find ~/lab/projectA -type f | wc -l   # expect: 6
tree -L 2 ~/lab/projectA

# Archive and list contents to confirm
tar -czvf ~/lab/projectA.tar.gz -C ~/lab projectA
tar -tzf ~/lab/projectA.tar.gz | head

Phase 2 · Intermediate System Administration

Permissions, users, processes, storage — the operator's toolkit.

Permissions & Ownership

Read the mode line: -rwxr-xr-- → type | owner | group | other. Each triplet = r(4) w(2) x(1).
Octal (absolute) mode: chmod 755 script.sh, chmod 640 secret.conf.
Symbolic mode: chmod u+x,g-w,o= file, chmod -R go-rwx ~/.ssh.
Ownership: chown user:group file, chown -R deploy:www-data /var/www/app.
umask: default mask (typically 022) — subtracted from 666/777 to get default file/dir perms. Inspect: umask.
Special bits: SUID (chmod u+s, e.g. /usr/bin/passwd), SGID on dirs (new files inherit group), Sticky bit (chmod +t /tmp, only owner can delete own files).

Users, Groups & sudo

Account DB files: /etc/passwd (account info), /etc/shadow (password hashes — root only), /etc/group (groups).
Create accounts: useradd -m -s /bin/bash alice, set password passwd alice.
Modify & groups: usermod -aG sudo alice (the -a is critical — without it you replace groups), groupadd deploy, groups alice.
sudoers — always use visudo: visudo -f /etc/sudoers.d/alice (drop-in files preferred over editing main file). visudo validates syntax before saving.
Granular rules: alice ALL=(root) NOPASSWD: /bin/systemctl restart nginx — limits the user to exactly that one command.

Process Management

List: ps aux, ps -ef, tree view pstree -p.
Live monitors: top (built-in), htop (friendlier, often separate install), btop (modern).
Job control: cmd & (background), Ctrl+Z (suspend), jobs, fg %1, bg %1, nohup cmd & (survive logout), disown.
Signals — graceful vs forceful: kill -15 PID (SIGTERM, asks nicely — default), kill -9 PID (SIGKILL, cannot be trapped — last resort, may leave orphaned resources). Also kill -1 (SIGHUP, reload config) and kill -2 (SIGINT).
By name: pkill nginx, pgrep -fa python, killall firefox.

Storage & Filesystems

Inspect disks: lsblk, blkid, fdisk -l, df -h (free space), du -sh /var/* (size per dir).
Partition: interactive with fdisk /dev/sdb (MBR) or parted /dev/sdb (GPT). ⚠️ Destructive — practice on a VM or spare disk.
Format: mkfs.ext4 /dev/sdb1, mkfs.xfs /dev/sdb1 (XFS is common on RHEL-family for large volumes).
Mount temporarily: mount /dev/sdb1 /mnt/data, unmount umount /mnt/data.
Persistent mount via /etc/fstab: use the UUID (get from blkid), not the device name (which can change across reboots). Test with mount -a before rebooting.

# Example /etc/fstab entry — always test with `mount -a` first!
# UUID                                  mount point   fstype  options              dump  pass
UUID=8f3c1a2b-...-9d4f                   /mnt/data     ext4    defaults,noatime     0     2

🎯 Action & Verification — Restricted Service Operator

Create a user that can only restart one specific service and nothing else.

Create user webops with home dir and bash shell.
Add a dedicated sudoers drop-in granting only systemctl restart nginx.
Verify webops CAN restart nginx and CANNOT run other sudo commands.
Confirm the action was logged via journalctl or /var/log/auth.log.

# 1. Create the user
sudo useradd -m -s /bin/bash webops
sudo passwd webops

# 2. Grant ONLY restart-nginx via a drop-in file
# visudo validates syntax — never edit sudoers files with a plain editor
sudo visudo -f /etc/sudoers.d/webops
# Add the single line:
# webops ALL=(root) NOPASSWD: /bin/systemctl restart nginx

# 3. Verify — should succeed
sudo -u webops sudo systemctl restart nginx

# 4. Verify — should FAIL ("Sorry, user webops is not allowed...")
sudo -u webops sudo apt update

# 5. Confirm in logs (path varies by distro)
sudo journalctl -u nginx --since "5 minutes ago"
sudo grep webops /var/log/auth.log  # Debian/Ubuntu
# RHEL/Fedora: /var/log/secure

Phase 3 · Networking, Security & Shell Scripting

Inspect the network, harden access, automate the boring stuff.

Linux Networking Stack

Interfaces & addresses: ip addr show (or ip a), ip link set eth0 up. Older toolset ifconfig is deprecated on most modern distros — prefer ip.
Routing: ip route show, default gateway inspection, add a static route ip route add 10.0.0.0/24 via 192.168.1.1.
DNS: /etc/resolv.conf (often managed by systemd-resolved or NetworkManager — direct edits may be overwritten), /etc/hosts, query with dig example.com or nslookup, getent hosts example.com.
Connectivity tests: ping -c 4 8.8.8.8, traceroute / mtr, curl -v https://example.com, wget.
Sockets & ports: ss -tlnp (listening TCP w/ process), ss -tunap. netstat is the older equivalent, often not installed by default now.

Security: SSH & Firewall

Generate a key pair: ssh-keygen -t ed25519 -C "you@host" (ed25519 is modern; RSA 4096 is the legacy fallback).
Copy public key: ssh-copy-id user@server (appends to ~/.ssh/authorized_keys).
Harden /etc/ssh/sshd_config: set PermitRootLogin no, PasswordAuthentication no, optionally change Port (security-through-obscurity at best — not a substitute for key auth). Reload: systemctl reload sshd. Keep an open session when changing SSH config — lock yourself out and you're done.
Firewall — UFW (Debian/Ubuntu): ufw default deny incoming, ufw allow 22/tcp, ufw enable, ufw status verbose.
Firewall — firewalld (RHEL/Fedora): firewall-cmd --permanent --add-service=ssh, firewall-cmd --reload, firewall-cmd --list-all. Both UFW and firewalld are front-ends over nftables/iptables.

Bash Scripting & Cron

Anatomy: shebang #!/usr/bin/env bash, variables name="alice", positional params $1 $2 "$@", exit codes $?.
Conditionals: if [[ -f file ]]; then ... fi, [[ -d dir ]], [[ "$a" == "$b" ]], case.
Loops: for f in *.log; do ...; done, while read -r line; do ...; done < file.
Strict mode: set -euo pipefail — fail on error (-e), unset vars (-u), and pipe failures (-o pipefail).
Lint: use shellcheck script.sh — catches most common bash bugs (third-party tool).
Cron schedule: crontab -e. Format: m h dom mon dow command. Example: 0 2 * * * /path/to/script.sh = daily at 02:00. Cron's PATH is minimal — always use absolute paths.

🎯 Action & Verification — Nightly Log Backup

Script + cron + idempotency. The classic ops automation.

Write /usr/local/bin/backup-logs.sh that ensures a backup dir exists, archives /var/log into it with a date-stamped filename, logs every action with timestamps to /var/log/backup-logs.log.
Make it executable, run it manually once, confirm the archive exists.
Schedule it nightly at 02:00 via root's crontab. Verify with crontab -l.

#!/usr/bin/env bash
# /usr/local/bin/backup-logs.sh — archives /var/log nightly
set -euo pipefail

BACKUP_DIR="/var/backups/logs"
LOG_FILE="/var/log/backup-logs.log"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ARCHIVE="$BACKUP_DIR/logs_$TIMESTAMP.tar.gz"

log() { echo "[$(date '+%F %T')] $*" >> "$LOG_FILE"; }

mkdir -p "$BACKUP_DIR"
log "Starting backup -> $ARCHIVE"

if tar -czf "$ARCHIVE" -C /var log 2>> "$LOG_FILE"; then
    log "OK  size=$(du -h "$ARCHIVE" | cut -f1)"
else
    log "FAIL tar exited $?"
    exit 1
fi

# Retain last 7 archives only
ls -1t "$BACKUP_DIR"/logs_*.tar.gz | tail -n +8 | xargs -r rm --
log "Pruned. Done."

# --- Setup ---
sudo chmod +x /usr/local/bin/backup-logs.sh
sudo /usr/local/bin/backup-logs.sh             # test run
ls -lh /var/backups/logs/                     # verify archive

# --- Schedule (run as root) ---
sudo crontab -e
# Add: 0 2 * * * /usr/local/bin/backup-logs.sh
sudo crontab -l

Phase 4 · Advanced Mastery & DevOps Integration

Troubleshoot deeply, tune the kernel, ship services the systemd way.

Logging & Troubleshooting

journalctl: journalctl -xe (recent + explanations), journalctl -u nginx --since today, journalctl -f (follow), journalctl -p err -b (errors this boot).
Plain-text logs: /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/Fedora), kernel ring buffer dmesg -T, auth /var/log/auth.log or /var/log/secure.
Resource bottlenecks: vmstat 1 5 (CPU/IO/memory), iostat -xz 1 (per-device IO — usually from sysstat package), free -h, uptime for load averages.
strace fundamentals: strace -p PID (attach to running), strace -e openat,read,write cmd (filter syscalls). Useful for "why is this hanging?" diagnostics. ⚠️ Adds significant overhead — don't leave running on production.
Open files / sockets per process: lsof -p PID, lsof -i :443.

Performance Tuning

Kernel params: read with sysctl -a, set live with sysctl -w net.core.somaxconn=4096, persist in /etc/sysctl.conf or drop-ins under /etc/sysctl.d/. Reload all: sysctl --system.
Common knobs (verify before applying — workload-dependent): vm.swappiness (lower = avoid swap, default 60), net.ipv4.tcp_fin_timeout, fs.file-max, net.core.rmem_max.
Memory model: in free -h, the available column is what apps can use — buff/cache is reclaimable. "Linux ate my RAM" is usually just caching.
Swap tuning: inspect with swapon --show; create swapfile fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile; persist via /etc/fstab.
Resource limits: ulimit -a, persistent caps via /etc/security/limits.conf or systemd unit LimitNOFILE=.

Modern DevOps Context — systemd, cgroups, namespaces

Container internals: containers aren't VMs — they're processes isolated by namespaces (PID, NET, MNT, UTS, IPC, USER, CGROUP) and resource-limited by cgroups (v2 on modern distros). Docker/Podman/Kubernetes are orchestration around these primitives.
Inspect cgroups: systemctl status shows the cgroup tree per unit; systemd-cgtop = top for cgroups.
systemd basics: systemctl start/stop/restart/reload, systemctl enable --now (start now + boot), systemctl status name, systemctl list-units --failed.
Unit file locations: distribution-shipped units under /lib/systemd/system/, local/custom units under /etc/systemd/system/ (local overrides win). After adding/editing: systemctl daemon-reload.
Timers replace cron in modern setups: a .timer unit + matching .service unit, inspected via systemctl list-timers.

🎯 Action & Verification — Custom systemd Service with Auto-Restart

Wrap a long-running script as a managed, auto-restarting service.

Write a tiny long-running script (/usr/local/bin/heartbeat.sh) that prints a heartbeat every 10s.
Create a unit file /etc/systemd/system/heartbeat.service with Restart=on-failure and a non-root user.
Reload daemon, enable + start, verify state with systemctl status and follow logs via journalctl -fu heartbeat.
Kill the underlying process and confirm systemd restarts it automatically.

# /usr/local/bin/heartbeat.sh
#!/usr/bin/env bash
set -euo pipefail
while true; do
    echo "[$(date '+%F %T')] heartbeat pid=$$"
    sleep 10
done

# /etc/systemd/system/heartbeat.service
[Unit]
Description=Heartbeat demo service
After=network.target

[Service]
Type=simple
User=nobody
ExecStart=/usr/local/bin/heartbeat.sh
Restart=on-failure
RestartSec=3s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

# --- Apply ---
sudo chmod +x /usr/local/bin/heartbeat.sh
sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat.service

# --- Verify ---
systemctl status heartbeat.service
journalctl -fu heartbeat.service

# --- Prove auto-restart ---
PID=$(systemctl show -p MainPID --value heartbeat.service)
sudo kill -9 "$PID"
sleep 4
systemctl status heartbeat.service   # should be active again, new PID

Where to go next

Once the four phases feel routine, these are the natural directions — pick based on the role you want.

SRE / Platform

Prometheus + Grafana, eBPF tooling (bpftrace), distributed tracing, SLO design.

Container / Cloud

Docker → Podman → Kubernetes. IaC with Terraform/Ansible. Cloud-specific networking.

Security

SELinux/AppArmor, auditd, CIS benchmarks, nftables deep-dive, hardening guides.

⚠️ Reminder: certification names, exam costs, and vendor curricula change frequently — verify current details on the official vendor site before committing.

Become a Linux Power User, the Practical Way