A real five-node cluster mid-experiment, which the ASCII excerpt cannot show: colour bands, per-core cells at 264 cores, the packed disk rows and the process panel filling the screen. The plain-text excerpt moves into a <details> block below it. The capture shows the host names of the cluster this was written for -- labels come from hosts.conf, which is yours. The laptop's own tmux status bar (which named the machine) is cropped off, and the image is palettised to 610K. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GkwTNg135j6e6dNGkjMx8o |
||
|---|---|---|
| docs | ||
| .gitignore | ||
| clusterdash | ||
| deploy.sh | ||
| hosts.conf.example | ||
| README.md | ||
| selftest.py | ||
| supervisor.py | ||
| zshrc-snippet.zsh | ||
clusterdash
One terminal screen showing every node of a cluster at once: all CPUs together, all memory together, then network, disk, and a scrollable process list per host.
Nothing is installed on the monitored nodes, and the screen survives your laptop changing networks.
A real cluster mid-experiment: 264 cores at one cell each, memory and network
for every host, disks packed two to a row, and a process list per node that
fills the rest of the screen and scrolls. The labels down the left are
whatever you put in hosts.conf.
The same layout as plain text — what --plain prints, at 100 columns
clusterdash 5/5 up · 264 cores · cpu/mem 2Hz · net 1Hz · disk 60s · jk/↑↓ scroll
╭─ cpu ─ one cell per core ───────────────────────────────────────────────────────────────────────╮
│ cores 0 9 19 29 39 49 59 │
│ headnode 16.4% ▁▁▂▃··▂· load 0.5/8 io 1.0%│
│ │
│ node1 66.2% ▅▆▅▆▆█▇▆ ▆██▅██▇▅ ▇█▆▄▂▄█▇ ▆█▄▇▃▄█▅ ▄▃█▃▄██▆ ▄▅▆▄·▃▄█ ▄·▇█ load 46.7/64 io 0.0%│
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ mem ───────────────────────────────────────────────────────────────────────────────────────────╮
│ node1 27.0G /125.0G ██████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 22%│
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ disk ─ used / total ───────────────────────────────────────────────────────────────────────────╮
│ headnode / 20.0G/ 29.0G 72% /mnt/data 307.0G/983.0G 33% │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Most rows are cut from this excerpt. Wider terminals add swap, uptime and disk bars, as in the screenshot above.
Why it is built this way
It runs on a server, not on your laptop. The dashboard holds one ssh connection per monitored node. Run from a laptop, every network change kills all of them and wipes the collected history. Run from a node inside the cluster, those links never roam — only the laptop→server hop does, and tmux makes that one disposable: detach, close the lid, reattach tomorrow, the history is still there.
Nothing is installed on the monitored nodes. The agent is a string of
Python inside supervisor.py, piped to python3 -u - over ssh. Each node
runs it from stdin and leaves no file behind. Updating the agent is just
restarting the dashboard. The only requirement on a node is python3 and an
ssh key.
btop is not involved. btop is a TUI, not a data source; parsing its frames
would be hopeless. The agent reads /proc directly.
Requirements
| Where | Needs |
|---|---|
| Your laptop | zsh, ssh, an alias for the running host |
| Running host | python3 ≥ 3.8, tmux ≥ 1.8, ssh keys to every monitored node |
| Monitored nodes | python3 ≥ 3.8, Linux (/proc), an authorised ssh key |
Terminology used throughout: the running host is the machine that runs the dashboard and collects from the others — typically a head node or any always-on box in the cluster. The monitored nodes are everything it watches, including itself.
What you have to fill in
Everything in this repo is generic. Exactly two things name your machines:
| What | Where | How to get it |
|---|---|---|
| The host list | hosts.conf — copy it from hosts.conf.example |
your own ssh aliases |
CLUSTERDASH_HOST |
zshrc-snippet.zsh, marked CHANGE ME |
the ssh alias of the running host |
CLUSTERDASH_CMD |
same file, marked CHANGE ME |
ssh <host> 'echo $HOME/.local/bin/clusterdash' |
hosts.conf is listed in .gitignore, so your host names stay out of the
repo even after you commit. deploy.sh reads the running host straight out of
it, so there is no second place to keep in sync.
Files
| File | Goes where | What it is |
|---|---|---|
supervisor.py |
~/.local/libexec/clusterdash/ on the running host |
Everything: agent, collector, curses UI |
clusterdash |
~/.local/bin/ on the running host |
Launcher tmux starts; sets the detach keybinding |
hosts.conf |
~/.config/clusterdash/ on the running host |
Which hosts to watch, in display order |
zshrc-snippet.zsh |
appended to your ~/.zshrc |
dash, dash-start, dash-stop, dash-status |
deploy.sh |
stays here | Copies the three files and restarts |
selftest.py |
stays here | Renders the layout from fabricated data |
Install
1. Name your machines.
cp hosts.conf.example hosts.conf
$EDITOR hosts.conf
Put the running host first with the target local, then one line per node:
headnode local
node1 node1
node2 node2
The first column is the label shown on screen; the second is the ssh destination as the running host resolves it. Line order is display order across every panel.
2. Server side.
./deploy.sh # running host taken from hosts.conf
./deploy.sh otherhost # or name one explicitly
./deploy.sh --no-restart # copy files without restarting a live dashboard
deploy.sh compiles supervisor.py and its embedded agent, runs the layout
selftest, asks the host where $HOME actually is, creates the directories,
copies the files, and restarts the dashboard's own tmux server — and only
that one, never the default socket.
3. Laptop side. Append the snippet, fill in the two CHANGE ME values,
and reload:
cat zshrc-snippet.zsh >> ~/.zshrc
$EDITOR ~/.zshrc
source ~/.zshrc
CLUSTERDASH_CMD must be an absolute path — see Gotchas. Run
any dash command before filling those in and it says so rather than failing
obscurely.
4. Use it.
dash # attach (starts it first if needed)
dash-start # start detached, collect history without watching
dash-status # up since when, clients attached
dash-stop # stop it
Keys
| Key | Does |
|---|---|
q, Esc, Ctrl-C |
Detach. The dashboard keeps running and collecting |
j / k, ↑ / ↓ |
Scroll the process list |
PgUp / PgDn, Space |
Scroll a page |
g / G |
Top / bottom of the process list |
Nothing quits from inside — dash-stop does that. The three detach keys work
by different routes: Ctrl-C is caught by tmux and never reaches the program,
while q and Esc are read by the UI, which then asks tmux to detach.
Outside tmux there is nothing to detach from, so q quits instead.
Adding another server
Two cases, depending on whether the new machine is being watched or is the one running the dashboard.
Watching one more node
1. The running host must be able to ssh to it without a password. Test it from the running host, not from your laptop — that is where the connection is made:
ssh headnode 'ssh -o BatchMode=yes node5 "hostname; python3 -V"'
If that fails, fix it before going further: add the key to the new node's
authorized_keys, add a Host node5 block to the running host's
~/.ssh/config, and accept its host key once (StrictHostKeyChecking=accept-new
is already passed, so one successful connection is enough).
2. Add a line to hosts.conf:
node5 node5
3. Redeploy:
./deploy.sh
That is all. The layout adapts on its own: process columns re-divide, the core grid re-sizes, and the disk panel re-packs. Nothing is installed on the new node.
Moving the dashboard to a different host
- Point
hosts.confat the new arrangement — the new running host becomeslocaland goes first, the old one becomes an ordinary ssh target. ./deploy.sh(the new host is now the one markedlocal, so no argument is needed).- In your
~/.zshrc, setCLUSTERDASH_HOSTto the new host andCLUSTERDASH_CMDto its absolute launcher path. Get it withssh <newhost> 'echo $HOME'— do not assume/home/<user>. source ~/.zshrc, thendash-stopagainst the old host if it is still running.
Watching a node that is not on the cluster network
It works, but expect the layout to say so: a node reached over the public
internet will drop more often. It shows as STALE after 4 seconds without a
sample, then down with the ssh error in its column, and reconnects on its
own with exponential backoff up to 60s. One unreachable node degrades to one
grey column; nothing else is affected.
Configuration
Refresh rates are per measurement, not global — set near the top of
AGENT_SRC in supervisor.py:
| Measurement | Rate | Why |
|---|---|---|
| CPU, per-core, memory, load | 2 Hz | One /proc file each; costs nothing |
| Network, process list | 1 Hz | The process scan reads every /proc/<pid> entry |
| Disk | 60 s | A statvfs per mount, and usage barely moves |
The agent ticks at --interval (0.5s) and each measurement refreshes on its
own schedule. Fields that did not refresh are left out of the sample and
the supervisor carries the previous value forward, so 2 Hz does not mean twice
the bandwidth.
Flags (on the running host, or locally for development):
python3 supervisor.py # the TUI
python3 supervisor.py --plain 2 # render 2 frames as text, then exit
python3 supervisor.py --probe 6 # dump raw agent JSON for 6s
python3 supervisor.py --hosts A=hostA,B=local
python3 supervisor.py --interval 1 --top 20
python3 supervisor.py --width 200 --height 50 --plain # force a size
--hosts overrides hosts.conf for one run, which is the quick way to try a
new node before adding it. With neither, the fallback is a single local
column — an obvious "not configured yet" rather than a guess at your network.
--plain renders the real layout without a terminal, which is how the layout
is tested; --probe shows what the agents are actually sending.
Footprint, measured on a 5-node cluster: one supervisor plus one agent per
node, together about 50 MB RSS and well under 1% of a core, plus a few MB for
tmux. The agent is a few hundred lines of /proc reads; the cost is dominated
by the per-second process scan.
Development
Everything can be exercised on a laptop with no cluster at all: selftest.py
fabricates the samples, and --hosts x=local monitors the machine you are on.
python3 selftest.py # renders fabricated data at four terminal widths
The selftest covers what live data cannot reach on demand: a host at 94% CPU,
memory nearly full, a stale host, a host that is down with an ssh error, and
terminals too narrow for the columns. Its per-core values have deliberate
structure (mixed, ramp, one_hot) — a flat array would render identically
whether cell grouping kept the max, the mean, or the first value, hiding real
bugs. Run it after touching layout().
Check no row overflows its terminal:
python3 - <<'PY'
import sys, threading; sys.path.insert(0, '.')
import selftest as T, supervisor as S
hosts = T.build(); hosts = hosts[-1:] + hosts[:-1]
for w in (200, 176, 150, 120, 100, 84, 60, 40):
t = S.render_plain(S.layout(hosts, threading.Lock(), w, 50, 1.0, {})).splitlines()
over = [(i, len(l)) for i, l in enumerate(t) if len(l) > w]
print(f"w={w:3d} longest={max(len(l) for l in t):3d} {over or ''}")
PY
layout() is a pure function: hosts in, [(text, style)] rows out. That is
why both of the above need no terminal, no ssh and no cluster.
Gotchas
These all cost real debugging time; they are documented in the code too.
$HOME is not always /home/<user>. On the cluster this was written for
it is a path on a mounted data volume. Guessing makes the first scp fail —
or worse, succeed somewhere useless. deploy.sh asks the host instead, and
the launcher expands $HOME on the host rather than through the ssh command
line.
Non-interactive ssh has no ~/.local/bin in PATH. That is why
CLUSTERDASH_CMD is an absolute path, and why the launcher exists at all
instead of a longer tmux command line.
The dashboard gets its own tmux socket (-L clusterdash). The launcher
binds Ctrl-C to detach-client, and tmux key tables are per-server, not
per-session — on a shared socket that would break Ctrl-C in every other tmux
session on the host. The private socket also means dash-stop and
deploy.sh can never touch your other work, including long-running
experiments on the same machine.
tmux new -A -d is not the idempotent form it looks like. With -A, an
existing session flips tmux into attach-session semantics, where -d means
"detach other clients" rather than "do not attach" — so it tries to attach
with no tty and dies with open terminal failed: not a terminal.
dash-start uses has-session || new -d instead.
One dashboard per host, enforced. The supervisor takes an exclusive lock
on ~/.cache/clusterdash.lock and refuses to be a second copy, printing how
to fix it. Two supervisors means two agents on every node for one screen. This
mostly happens when a shell sourced before a change to the snippet still
holds the old dash function — source ~/.zshrc in that shell.
Agents must die with their supervisor. An agent whose pipe is gone exits,
and exits again if reparented to init; the supervisor also terminates its
children on shutdown. An earlier version swallowed BrokenPipeError in a
broad handler and left one orphan per node per run, accumulating silently.
Network filesystems are excluded from the disk panel on purpose: statvfs
on a hung NFS mount blocks forever and would stall the whole agent.
Process names come from /proc/<pid>/cmdline, not comm. comm is
capped at 15 characters, so every Python job in the cluster showed up as
python3. Only the rows actually on screen are resolved, to keep the scan
cheap.
