Files
streamline/streamline-hot-deploy-blog.md
T
2026-11-16 17:26:44 -08:00

8.1 KiB

Shipping Code to a Fleet of Phones: Streamline's New Distributed Hot-Deploy

How we went from "scp, ssh, pray" to one-click fleet updates — and why we push a bundle instead of running git pull.

The setup

Streamline is a distributed system for driving real mobile devices at scale. Architecturally it splits into two halves:

  • A control plane — the single source of truth. It holds the task queue, the web UI, and forwards work.
  • A fleet of satellite nodes — each runs on a machine with real Android/iOS devices attached, pulls work from the control plane, and executes it locally (evaluation, fixes, evidence capture).

Today there are a handful of satellites — fei-android-01, fei-ios-01, pns-android-01, and growing. Each one registers with the control plane, heartbeats every 10 seconds, and reports what it's running.

This works well until you change the satellite's code. Then you hit the wall.

The wall

Updating a satellite used to mean, per node: scp the new code (or the whole repo) over, ssh in, pkill the old process, and relaunch it under nohup. Multiply by every node. Forget one, and it silently drifts — running week-old code while you assume the fix is live. There was no way to see which node ran which version, and no way to push an update without a manual SSH tour of the fleet.

For a platform whose entire value is fast iteration on real hardware, that update loop was the bottleneck. So we built distributed hot-deploy.

What it looks like now

There's a new owner-only Deploy panel. It shows every node, its running version versus the latest built version, and whether it's busy. One click — per node or "deploy all stale" — pushes new code and restarts the node onto it. No SSH.

 node          running     latest      status      action
 fei-android   1c8734a     1c8734a     ● 最新       —
 fei-ios-01    0.1.0       1c8734a     ▲ 待更新     [ Deploy ]
 pns-android   0.1.0       1c8734a     ▲ 待更新     [ Deploy ]
                                                    [ Deploy all stale ]

Under the hood, a deploy is four moves:

CONTROL PLANE (your laptop)                REMOTE SATELLITE
───────────────────────────                ────────────────────────────
1. build bundle  ──► satellite.js          run-satellite.sh (keep-alive loop)
   stamp git SHA ──► VERSION                 └─ bun satellite.js
                                                  reports version = <SHA>
2. POST /sat/admin/deploy ─────────────►
       body = satellite.js bytes
       header = sha256, version
                                           3. verify sha256
                                              atomic-swap the bundle (+ .bak)
                                              exit(0)
                                                  │
                                           4. wrapper relaunches on NEW code
                                              re-registers with new SHA
                                                  ▼
       panel: running == latest  ✓  ◄──────  heartbeat

The version each node reports is now its git commit SHA, stamped into the bundle at build time. That turns "is this node up to date?" from a guess into a fact the UI renders directly — and it's how a deploy confirms success: the node comes back reporting the new SHA, and its badge flips from 待更新 to 最新 on its own.

The interesting decision: bundle push, not git pull

The obvious way to update a node that shares the same repo is to git pull on it. We deliberately didn't. Here's the reasoning, because it says something about what kind of system Streamline is.

Why a bundle wins for a prototyping platform

1. You can deploy uncommitted code. This is the big one. Our actual development loop is change a line → see it run on a real device. With git pull, every iteration means commit, push to origin, then pull on the node — a full round-trip through version control just to test a hunch. The bundle push ships exactly what's in your working tree right now, dirty or not. (The version stamp even carries a -dirty suffix so you know it.) For rapid prototyping, being able to fling work-in-progress at a device in one click is the whole point.

2. No git credentials on every node. git pull means each machine needs a deploy key or token to the origin, kept valid over time. The bundle push reuses the shared secret that already authenticates the control→satellite channel. Zero new secrets to distribute or rotate across the fleet.

3. No second network dependency. The control plane already reaches each satellite — that's how it proxies logs and evidence. git pull would add a separate requirement: every node must also reach the git host. For a node on a different network, that may not hold even when the control plane can talk to it fine. Pushing over the channel that already works removes a whole failure mode.

4. Atomic and reversible. The bundle is a single 0.35 MB file. A deploy verifies its sha256, writes to a temp sibling, and atomically renames it into place — keeping the previous file as .bak for instant rollback. A mid-write crash can never leave a truncated bundle the node would try to boot. git pull on a node can hit a dirty index, a merge conflict, or a half-applied state, and needs a clean tree to be safe.

What git pull is genuinely better at

We're not dogmatic. If your discipline is "only ever deploy tagged, pushed commits," git pull gives you a more auditable node state — a real git ref you can inspect, not an opaque blob. That's the right trade for a locked-down production fleet. Streamline is the opposite: an experimentation platform where the cost of a slow iteration loop is higher than the cost of an opaque deploy. So we optimized for speed of iteration, and the bundle won.

(We also considered a fancier bundle protocol — serve a manifest, have the node pull it back and verify. For a 0.35 MB file that was pure ceremony. The whole thing collapses to a single POST with a sha256 header. Simplicity won there too.)

The part everyone forgets: restart

Pushing new code is only half the job. The process has to restart onto it. A satellite launched with bare nohup can't — when it exits, it stays dead, and your "deploy" just bricked the node.

So the deploy exits the process deliberately (exit(0) after the swap), and a tiny keep-alive wrapper brings it back:

# run-satellite.sh — the supervisor that makes hot-deploy possible
while true; do
  bun satellite/dist/satellite.js
  echo "exited; restarting"
  sleep 2
done

Deploy = swap the bundle, exit, let the wrapper relaunch on the new file. It's the simplest possible supervisor — no systemd, no pm2, no per-node service install — and it's the one piece that has to exist on a node before hot-deploy can touch it. (Installing it is the single one-time manual step per node; after that, every update is a click.)

Safety rails

  • sha256 verified before swap — a corrupted or truncated push is rejected with the running bundle untouched.
  • Busy nodes are skipped — a node mid-evaluation or mid-fix isn't interrupted unless you explicitly force it (a deploy would abort its task).
  • Version-confirmed — success isn't "the POST returned 200," it's "the node came back reporting the new SHA." The panel shows you that flip.
  • One-click rollback — the previous bundle is always one .bak away.

The takeaway

The design reflects the system. Streamline exists to shorten the loop between an idea about mobile behavior and seeing it happen on a real phone. A deploy mechanism that forced a commit-push-pull detour through git — with credentials and a second network hop — would have taxed exactly the loop we're trying to make fast. Pushing a self-contained bundle over the channel we already trust, atomically, WIP and all, keeps that loop tight.

Sometimes the "less proper" engineering choice is the right one — when you're honest about what the system is for.