6. PODsys Metrics

podsys-metrics is a high-performance system monitoring tool for real-time collection of CPU, memory, disk, network (Ethernet/InfiniBand), and NVIDIA GPU performance metrics. It supports both standalone and cluster monitoring modes, with a Web dashboard for cluster-wide visualization.

6.1 Features

• CPU Monitoring: User mode, system mode, IO wait, idle time utilization statistics
• Memory Monitoring: Memory usage, cache, buffer statistics
• Disk Monitoring: Disk read/write I/O statistics (throughput + IOPS)
• Network Monitoring:
  • Ethernet interface send/receive statistics
  • InfiniBand port traffic statistics
• GPU Monitoring: NVIDIA GPU utilization, memory usage, temperature, power consumption, frequency, NVLink bandwidth
• CUDA Information: GPU device details and compute capability detection
• Cluster Web Dashboard: Real-time charts (Chart.js) and tables for multi-node visualization
• Data Persistence: JSONL format per node per module for historical analysis

6.2 Architecture

┌─────────┐      ┌──────────┐
│ server  │─────▶│ client 1 │
│ (Web)   │─────▶│ client 2 │
│         │─────▶│ client N │
└─────────┘      └──────────┘

Component	Binary	Requires	Description
Client	podsys-metrics-client	CUDA + NVML	Runs on compute nodes, collects metrics
Server	podsys-metrics-server	C++17 only	Runs on management node, Web UI + data persistence

6.3 Standalone Usage

After deploying compute nodes using PODsys, the podsys-metrics-client binary is located at /podsys/scripts/podsys-metrics-client on all compute nodes.

Log in to any compute node and run directly:

# Monitor GPU, JSON output, run 100 times
/podsys/scripts/podsys-metrics-client gpu -j -d 100

# Monitor CPU and memory simultaneously
/podsys/scripts/podsys-metrics-client cpu mem

# Monitor all modules
/podsys/scripts/podsys-metrics-client all

Monitoring Modules

Module	Description
cpu	CPU usage
mem	Memory usage
disk	Disk I/O, space, inode
eth	Ethernet traffic
ib	InfiniBand traffic
gpu	GPU utilization, memory, temperature, power, frequency
cuda	CUDA device info (standalone only)
all	All modules

Command Options

Option	Description
-j, --json	Output in JSON format
-d, --duration <num>	Number of iterations (default: 9999)
-h, --help	Display help information

6.4 Cluster Monitoring

Start Daemon on Compute Nodes

Use PDSH to start the daemon on all compute nodes from the management node:

# Start daemon on all nodes (background mode)
pdsh -R ssh -w ^hosts.txt "/podsys/scripts/podsys-metrics-client daemon -p 9001 -f"

# Or start without background (for testing)
pdsh -R ssh -w ^hosts.txt "/podsys/scripts/podsys-metrics-client daemon -p 9001"

Note

• hosts.txt contains the IP addresses of all compute nodes (generated by install_progress.sh)
• Default daemon port is 9001
• Use -f flag to run in background (fork mode)

Start Server on Management Node

The podsys-metrics-server binary is located at podsys-bxx/scripts/podsys-metrics-server on the management node.

cd podsys-bxx/
./scripts/podsys-metrics-server -i hosts.txt <modules...> [options]

Parameters:

Parameter	Description	Default
-i, --iplist <file>	Node IP list file (required)	-
<modules>	Monitoring modules (cpu, mem, gpu, disk, eth, ib, or all)	-
-d, --duration <num>	Number of iterations	1
--server-port <port>	Server listen port	9000
--client-port <port>	Client daemon port	9001
--http-port <port>	Web UI port	8080
--data-dir <dir>	Data output directory	./data
--table	Enable terminal table output (disabled by default)	off
--host <addr>	Bind address	0.0.0.0

Usage Examples

# Monitor all modules for 10 minutes
./scripts/podsys-metrics-server -i hosts.txt all -d 600

# Monitor CPU, memory, GPU for 10 iterations
./scripts/podsys-metrics-server -i hosts.txt cpu mem gpu -d 10

# Monitor GPU with custom Web port
./scripts/podsys-metrics-server -i hosts.txt gpu --http-port 9090

# Enable terminal table output alongside Web UI
./scripts/podsys-metrics-server -i hosts.txt gpu -d 100 --table

Web Dashboard

After starting the server, open http://<manager-ip>:8080 in a browser:

• Node List: Left sidebar shows all connected compute nodes
• Module Tabs: Switch between GPU / CPU / Memory / Disk / Ethernet / InfiniBand
• Metric Selector: (GPU module) Switch between utilization, memory, power, temperature, clock, NVLink bandwidth
• Real-time Charts: Chart.js line charts showing historical trends (last 60 data points, auto-refresh)
• Data Table: All nodes latest metrics overview, auto-refresh every 2 seconds

TIP

The Web UI is disabled by default in terminal output. Use --table to also print tables to the terminal.

Data Persistence

Monitoring data is appended to the data/ directory in JSONL format (one JSON object per line), partitioned by node and module:

data/
├── node01_cpu.jsonl
├── node01_gpu.jsonl
├── node02_cpu.jsonl
└── ...