No description
Find a file
Andy Li 73cbede25a README: add a screenshot of the dashboard under load
A real five-node cluster mid-experiment, which the ASCII excerpt cannot show:
colour bands, per-core cells at 264 cores, the packed disk rows and the
process panel filling the screen. The plain-text excerpt moves into a
<details> block below it.

The capture shows the host names of the cluster this was written for --
labels come from hosts.conf, which is yours. The laptop's own tmux status bar
(which named the machine) is cropped off, and the image is palettised to 610K.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GkwTNg135j6e6dNGkjMx8o
2026-11-18 14:37:02 +11:00
docs README: add a screenshot of the dashboard under load 2026-11-18 14:37:02 +11:00
.gitignore init commit 2026-11-18 14:30:36 +11:00
clusterdash init commit 2026-11-18 14:30:36 +11:00
deploy.sh init commit 2026-11-18 14:30:36 +11:00
hosts.conf.example init commit 2026-11-18 14:30:36 +11:00
README.md README: add a screenshot of the dashboard under load 2026-11-18 14:37:02 +11:00
selftest.py init commit 2026-11-18 14:30:36 +11:00
supervisor.py init commit 2026-11-18 14:30:36 +11:00
zshrc-snippet.zsh init commit 2026-11-18 14:30:36 +11:00

clusterdash

One terminal screen showing every node of a cluster at once: all CPUs together, all memory together, then network, disk, and a scrollable process list per host.

Nothing is installed on the monitored nodes, and the screen survives your laptop changing networks.

clusterdash watching a five-node cluster

A real cluster mid-experiment: 264 cores at one cell each, memory and network for every host, disks packed two to a row, and a process list per node that fills the rest of the screen and scrolls. The labels down the left are whatever you put in hosts.conf.

The same layout as plain text — what --plain prints, at 100 columns
clusterdash  5/5 up  · 264 cores  · cpu/mem 2Hz · net 1Hz · disk 60s  · jk/↑↓ scroll
╭─ cpu ─ one cell per core  ───────────────────────────────────────────────────────────────────────╮
│           cores 0        9        19       29       39       49       59                         │
│ headnode  16.4% ▁▁▂▃··▂·                                                    load  0.5/8   io 1.0%│
│                                                                                                  │
│ node1     66.2% ▅▆▅▆▆█▇▆ ▆██▅██▇▅ ▇█▆▄▂▄█▇ ▆█▄▇▃▄█▅ ▄▃█▃▄██▆ ▄▅▆▄·▃▄█ ▄·▇█  load 46.7/64  io 0.0%│
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ mem  ───────────────────────────────────────────────────────────────────────────────────────────╮
│ node1     27.0G /125.0G ██████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   22%│
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ disk ─ used / total  ───────────────────────────────────────────────────────────────────────────╮
│ headnode /  20.0G/ 29.0G  72%  /mnt/data 307.0G/983.0G  33%                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Most rows are cut from this excerpt. Wider terminals add swap, uptime and disk bars, as in the screenshot above.

Why it is built this way

It runs on a server, not on your laptop. The dashboard holds one ssh connection per monitored node. Run from a laptop, every network change kills all of them and wipes the collected history. Run from a node inside the cluster, those links never roam — only the laptop→server hop does, and tmux makes that one disposable: detach, close the lid, reattach tomorrow, the history is still there.

Nothing is installed on the monitored nodes. The agent is a string of Python inside supervisor.py, piped to python3 -u - over ssh. Each node runs it from stdin and leaves no file behind. Updating the agent is just restarting the dashboard. The only requirement on a node is python3 and an ssh key.

btop is not involved. btop is a TUI, not a data source; parsing its frames would be hopeless. The agent reads /proc directly.

Requirements

Where Needs
Your laptop zsh, ssh, an alias for the running host
Running host python3 ≥ 3.8, tmux ≥ 1.8, ssh keys to every monitored node
Monitored nodes python3 ≥ 3.8, Linux (/proc), an authorised ssh key

Terminology used throughout: the running host is the machine that runs the dashboard and collects from the others — typically a head node or any always-on box in the cluster. The monitored nodes are everything it watches, including itself.

What you have to fill in

Everything in this repo is generic. Exactly two things name your machines:

What Where How to get it
The host list hosts.conf — copy it from hosts.conf.example your own ssh aliases
CLUSTERDASH_HOST zshrc-snippet.zsh, marked CHANGE ME the ssh alias of the running host
CLUSTERDASH_CMD same file, marked CHANGE ME ssh <host> 'echo $HOME/.local/bin/clusterdash'

hosts.conf is listed in .gitignore, so your host names stay out of the repo even after you commit. deploy.sh reads the running host straight out of it, so there is no second place to keep in sync.

Files

File Goes where What it is
supervisor.py ~/.local/libexec/clusterdash/ on the running host Everything: agent, collector, curses UI
clusterdash ~/.local/bin/ on the running host Launcher tmux starts; sets the detach keybinding
hosts.conf ~/.config/clusterdash/ on the running host Which hosts to watch, in display order
zshrc-snippet.zsh appended to your ~/.zshrc dash, dash-start, dash-stop, dash-status
deploy.sh stays here Copies the three files and restarts
selftest.py stays here Renders the layout from fabricated data

Install

1. Name your machines.

cp hosts.conf.example hosts.conf
$EDITOR hosts.conf

Put the running host first with the target local, then one line per node:

headnode    local
node1       node1
node2       node2

The first column is the label shown on screen; the second is the ssh destination as the running host resolves it. Line order is display order across every panel.

2. Server side.

./deploy.sh                 # running host taken from hosts.conf
./deploy.sh otherhost       # or name one explicitly
./deploy.sh --no-restart    # copy files without restarting a live dashboard

deploy.sh compiles supervisor.py and its embedded agent, runs the layout selftest, asks the host where $HOME actually is, creates the directories, copies the files, and restarts the dashboard's own tmux server — and only that one, never the default socket.

3. Laptop side. Append the snippet, fill in the two CHANGE ME values, and reload:

cat zshrc-snippet.zsh >> ~/.zshrc
$EDITOR ~/.zshrc
source ~/.zshrc

CLUSTERDASH_CMD must be an absolute path — see Gotchas. Run any dash command before filling those in and it says so rather than failing obscurely.

4. Use it.

dash          # attach (starts it first if needed)
dash-start    # start detached, collect history without watching
dash-status   # up since when, clients attached
dash-stop     # stop it

Keys

Key Does
q, Esc, Ctrl-C Detach. The dashboard keeps running and collecting
j / k, ↑ / ↓ Scroll the process list
PgUp / PgDn, Space Scroll a page
g / G Top / bottom of the process list

Nothing quits from inside — dash-stop does that. The three detach keys work by different routes: Ctrl-C is caught by tmux and never reaches the program, while q and Esc are read by the UI, which then asks tmux to detach. Outside tmux there is nothing to detach from, so q quits instead.

Adding another server

Two cases, depending on whether the new machine is being watched or is the one running the dashboard.

Watching one more node

1. The running host must be able to ssh to it without a password. Test it from the running host, not from your laptop — that is where the connection is made:

ssh headnode 'ssh -o BatchMode=yes node5 "hostname; python3 -V"'

If that fails, fix it before going further: add the key to the new node's authorized_keys, add a Host node5 block to the running host's ~/.ssh/config, and accept its host key once (StrictHostKeyChecking=accept-new is already passed, so one successful connection is enough).

2. Add a line to hosts.conf:

node5       node5

3. Redeploy:

./deploy.sh

That is all. The layout adapts on its own: process columns re-divide, the core grid re-sizes, and the disk panel re-packs. Nothing is installed on the new node.

Moving the dashboard to a different host

  1. Point hosts.conf at the new arrangement — the new running host becomes local and goes first, the old one becomes an ordinary ssh target.
  2. ./deploy.sh (the new host is now the one marked local, so no argument is needed).
  3. In your ~/.zshrc, set CLUSTERDASH_HOST to the new host and CLUSTERDASH_CMD to its absolute launcher path. Get it with ssh <newhost> 'echo $HOME' — do not assume /home/<user>.
  4. source ~/.zshrc, then dash-stop against the old host if it is still running.

Watching a node that is not on the cluster network

It works, but expect the layout to say so: a node reached over the public internet will drop more often. It shows as STALE after 4 seconds without a sample, then down with the ssh error in its column, and reconnects on its own with exponential backoff up to 60s. One unreachable node degrades to one grey column; nothing else is affected.

Configuration

Refresh rates are per measurement, not global — set near the top of AGENT_SRC in supervisor.py:

Measurement Rate Why
CPU, per-core, memory, load 2 Hz One /proc file each; costs nothing
Network, process list 1 Hz The process scan reads every /proc/<pid> entry
Disk 60 s A statvfs per mount, and usage barely moves

The agent ticks at --interval (0.5s) and each measurement refreshes on its own schedule. Fields that did not refresh are left out of the sample and the supervisor carries the previous value forward, so 2 Hz does not mean twice the bandwidth.

Flags (on the running host, or locally for development):

python3 supervisor.py                      # the TUI
python3 supervisor.py --plain 2            # render 2 frames as text, then exit
python3 supervisor.py --probe 6            # dump raw agent JSON for 6s
python3 supervisor.py --hosts A=hostA,B=local
python3 supervisor.py --interval 1 --top 20
python3 supervisor.py --width 200 --height 50 --plain   # force a size

--hosts overrides hosts.conf for one run, which is the quick way to try a new node before adding it. With neither, the fallback is a single local column — an obvious "not configured yet" rather than a guess at your network.

--plain renders the real layout without a terminal, which is how the layout is tested; --probe shows what the agents are actually sending.

Footprint, measured on a 5-node cluster: one supervisor plus one agent per node, together about 50 MB RSS and well under 1% of a core, plus a few MB for tmux. The agent is a few hundred lines of /proc reads; the cost is dominated by the per-second process scan.

Development

Everything can be exercised on a laptop with no cluster at all: selftest.py fabricates the samples, and --hosts x=local monitors the machine you are on.

python3 selftest.py       # renders fabricated data at four terminal widths

The selftest covers what live data cannot reach on demand: a host at 94% CPU, memory nearly full, a stale host, a host that is down with an ssh error, and terminals too narrow for the columns. Its per-core values have deliberate structure (mixed, ramp, one_hot) — a flat array would render identically whether cell grouping kept the max, the mean, or the first value, hiding real bugs. Run it after touching layout().

Check no row overflows its terminal:

python3 - <<'PY'
import sys, threading; sys.path.insert(0, '.')
import selftest as T, supervisor as S
hosts = T.build(); hosts = hosts[-1:] + hosts[:-1]
for w in (200, 176, 150, 120, 100, 84, 60, 40):
    t = S.render_plain(S.layout(hosts, threading.Lock(), w, 50, 1.0, {})).splitlines()
    over = [(i, len(l)) for i, l in enumerate(t) if len(l) > w]
    print(f"w={w:3d} longest={max(len(l) for l in t):3d} {over or ''}")
PY

layout() is a pure function: hosts in, [(text, style)] rows out. That is why both of the above need no terminal, no ssh and no cluster.

Gotchas

These all cost real debugging time; they are documented in the code too.

$HOME is not always /home/<user>. On the cluster this was written for it is a path on a mounted data volume. Guessing makes the first scp fail — or worse, succeed somewhere useless. deploy.sh asks the host instead, and the launcher expands $HOME on the host rather than through the ssh command line.

Non-interactive ssh has no ~/.local/bin in PATH. That is why CLUSTERDASH_CMD is an absolute path, and why the launcher exists at all instead of a longer tmux command line.

The dashboard gets its own tmux socket (-L clusterdash). The launcher binds Ctrl-C to detach-client, and tmux key tables are per-server, not per-session — on a shared socket that would break Ctrl-C in every other tmux session on the host. The private socket also means dash-stop and deploy.sh can never touch your other work, including long-running experiments on the same machine.

tmux new -A -d is not the idempotent form it looks like. With -A, an existing session flips tmux into attach-session semantics, where -d means "detach other clients" rather than "do not attach" — so it tries to attach with no tty and dies with open terminal failed: not a terminal. dash-start uses has-session || new -d instead.

One dashboard per host, enforced. The supervisor takes an exclusive lock on ~/.cache/clusterdash.lock and refuses to be a second copy, printing how to fix it. Two supervisors means two agents on every node for one screen. This mostly happens when a shell sourced before a change to the snippet still holds the old dash function — source ~/.zshrc in that shell.

Agents must die with their supervisor. An agent whose pipe is gone exits, and exits again if reparented to init; the supervisor also terminates its children on shutdown. An earlier version swallowed BrokenPipeError in a broad handler and left one orphan per node per run, accumulating silently.

Network filesystems are excluded from the disk panel on purpose: statvfs on a hung NFS mount blocks forever and would stall the whole agent.

Process names come from /proc/<pid>/cmdline, not comm. comm is capped at 15 characters, so every Python job in the cluster showed up as python3. Only the rows actually on screen are resolved, to keep the scan cheap.