Notes on Practical Linux Performance Optimization

Liao Jiayi Liao Jiayi #Linux#Notes

Summaries and reflections from the Geek Time course on Linux performance optimization.

Translated from Chinese with AI · Read the original

Someone recommended this Geek Time course to me. It seemed useful, so I am recording summaries and reflections for each lesson here.

Performance-Tuning Map

linux-tools

Understanding Load Average

Log in to a Linux server and run uptime to see output like this:

 21:17:08 up 81 days, 22:30,  1 user,  load average: 0.15, 0.11, 0.07

The fields show the current time, system uptime, logged-in users, and load averages over 1, 5, and 15 minutes. The definition in man uptime is:

System load averages is the average number of processes that are either in a runnable or uninterruptable state.  A process in a runnable state is either using the CPU or waiting to use the CPU.
A  process  in uninterruptable state is waiting for some I/O access, eg waiting for disk.  The averages are taken over the three time intervals.  Load averages are not normalized for the number
of CPUs in a system, so a load average of 1 means a single CPU system is loaded all the time while on a 4 CPU system it means it was idle 75% of the time.

Load average measures the average number of active processes over time. Active processes include:

  • Runnable processes: Using or waiting for a CPU.
  • Uninterruptible processes: For example, processes waiting for disk I/O.

High load average therefore does not necessarily mean high CPU usage. Possible causes include:

  • CPU-intensive work: Frequent CPU use produces many runnable processes.
  • I/O-intensive work: Frequent disk reads and writes produce many uninterruptible processes.
  • Many processes waiting to be scheduled on a CPU. (What situations create so many waiting processes?)

Hands-on tools:

  • stress: Simulates CPU, I/O, memory, and other loads.
  • mpstat: Monitors processors.
  • pidstat: Monitors resource usage by process.

CPU Context Switching

The CPU stores instructions, data, and addresses temporarily in registers, and tracks instruction addresses with a program counter. These elements describe its current state and execution environment, forming the CPU context.

A context switch is fundamentally a switch between CPU tasks. There are three types:

  • Process context switches
  • Thread context switches
  • Interrupt context switches

Process context switches

Linux separates execution into privilege levels. Ring 0 has the highest privileges and corresponds to kernel space; Ring 3 has restricted access and corresponds to user space. Execution there is called kernel mode or user mode. Entering kernel mode requires saving the user-mode instruction position, but does not switch user-space resources, so this is usually called a privilege-mode switch.

Thread context switches

Process switches are also thread switches. Threads may belong to the same process or different processes; the distinction is whether resources such as virtual memory and global variables must also change.

Interrupt context switches

Hardware interrupts trigger kernel-mode context switches. Even a process in user mode switches immediately; this has the highest priority.

Use vmstat to inspect CPU context-switch information.

Many voluntary switches may indicate threads waiting for I/O. Many involuntary switches suggest threads competing for CPU resources.

Hands-on tools:

  • sysbench: A multithreaded benchmarking tool.
  • /proc/interrupts

Investigating CPU Problems at the System Level

Common tools:

  • perf: A powerful CPU analysis tool; see Perf at Netflix.
  • pstree: Shows parent-child relationships between processes.
  • strace: Traces system calls.
  • lsof: list open files

The commonly used top command:

There are several task categories:

  • running
  • sleeping
  • stopped
  • zombie

There are also several CPU categories:

  • us: user
  • sy: system
  • ni: nice time
  • id: idle
  • wa: IO-wait
  • hi: hardware interrupt
  • si: software interrupt

Process states:

  • R: Running
  • D: Disk sleep, generally indicating hardware interaction.
  • Z: Zombie
  • S: Interruptible sleep.
  • I: Idle

Interrupts

For performance, interrupt handling is split into an upper half (hard interrupt) and a lower half (soft interrupt):

  • Hard interrupt: Interacts with hardware and is usually fast.
  • Soft interrupt: Triggered by the kernel and usually slower.

There are many soft-interrupt types, listed in /proc/softirqs. These tools help investigate network-related soft interrupts:

  • sar: Monitors network traffic, including throughput in bytes per second and packet rates.
  • tcpdump: Captures packets.

CPU Utilization

CPU utilization includes:

  • User CPU: Indicates busy applications.
  • Kernel CPU: Indicates a busy kernel.
  • I/O wait: Indicates time spent waiting for hardware interactions.
  • Soft- and hard-interrupt CPU: Indicates substantial interrupt handling.

cpu

cpu-2

Memory

Linux gives each process its own virtual address space so applications can use contiguous memory. Page tables and the CPU’s memory management unit (MMU) map virtual to physical memory, with CPU-based translation making it fast. Pages are 4 KB. Multilevel page tables prevent the indexing structure from growing too large, while huge pages support large memory allocations.

Two memory-allocation approaches are available:

  • Small allocations use brk(). Memory is retained for reuse rather than reclaimed, but this can cause fragmentation.
  • Large allocations use mmap(). Memory is reclaimed after use, and accesses encounter page faults when virtual mappings are absent, keeping the kernel busy with address-space translation.

When memory runs short, the system can reclaim it by:

  • Evicting cached data with algorithms such as LRU.
  • Moving infrequently used memory to swap.
  • Killing processes through the OOM mechanism.

The free command, whose values mostly come from /proc/meminfo:

  • free: Unused memory.
  • shared: Shared memory.
  • buff/cache: Buffer and cache memory.
  • available: Memory available to new processes, including reclaimable caches and unused memory.

The top command:

  • VIRT: Process virtual-memory size.
  • RES: Resident memory actually in use.
  • SHR: Shared-memory size.

Buffers cache disk data, while caches hold file data. Both serve reads and writes. (So can we understand one as a disk cache and the other as a file cache?)

  • Writes: Applications can return and do other work before data reaches disk; caches and buffers finish the operation.
  • Reads: Frequently accessed disk data can be read faster.

Tools:

  • cachestat
  • cachetop
  • memleak
  • valgrind

I/O Performance

Linux associates files with:

  • Inode: The file’s unique identifier, holding metadata and persisted on disk.
  • Dentry: An internal cache of the directory structure.

A sector is the smallest disk read unit. Contiguous sectors are grouped into logical blocks for more efficient operations. The superblock records overall filesystem state, including inode and logical-block usage.

linux io

Disks

  • HDD: A mechanical disk moves its read/write head to locate tracks. The smallest sector is 512 bytes.
  • SSD: No mechanical track seeking. Random reads and writes generate substantial garbage collection. The smallest sector is 4 KB. Logical blocks enable read-ahead.

Different disk arrangements produce different storage architectures, such as:

  • RAID: Combines several disks into one logical disk, a Redundant Array of Independent Disks.
  • NFS: Provides networked disk storage across a cluster.

Interpreting iostat Metrics