Frequently Used Tools

January 11, 2025

GPU

  1. nvidia-smi -l 1
  2. gpustat -i 1 --show-power
  3. nvtop (install: sudo apt install nvtop)
  4. nvidia-smi dmon -s u -c 100 , NVIDIA 对video/image 的解析做了加速，不走通用 sm 计算
    • sm: streaming multiprocessor utilization
    • mem: gpu memory utilization
    • enc: video encoder utilization(NVENC)
    • dec: video decoder utilization(NVDEC)
    • jpg: jpeg decoder utilization
    • ofa: optical flow accelerator
  5. nsys profile --output=report uv run python run.py --model Qwen3-VL-2B-Instruct --data MM-Math *

Python

  1. py-spy dump --pid 2473
  2. py-spy record -o flamegraph.svg --pid 2473 --duration 60 (原理上用的 ptrace)

System

  1. gdb -p 2473