106 lines
8.1 KiB
Markdown
106 lines
8.1 KiB
Markdown
# Shipping Code to a Fleet of Phones: Streamline's New Distributed Hot-Deploy
|
|
|
|
*How we went from "scp, ssh, pray" to one-click fleet updates — and why we push a bundle instead of running `git pull`.*
|
|
|
|
## The setup
|
|
|
|
Streamline is a distributed system for driving real mobile devices at scale. Architecturally it splits into two halves:
|
|
|
|
- A **control plane** — the single source of truth. It holds the task queue, the web UI, and forwards work.
|
|
- A fleet of **satellite** nodes — each runs on a machine with real Android/iOS devices attached, pulls work from the control plane, and executes it locally (evaluation, fixes, evidence capture).
|
|
|
|
Today there are a handful of satellites — `fei-android-01`, `fei-ios-01`, `pns-android-01`, and growing. Each one registers with the control plane, heartbeats every 10 seconds, and reports what it's running.
|
|
|
|
This works well until you change the satellite's code. Then you hit the wall.
|
|
|
|
## The wall
|
|
|
|
Updating a satellite used to mean, per node: `scp` the new code (or the whole repo) over, `ssh` in, `pkill` the old process, and relaunch it under `nohup`. Multiply by every node. Forget one, and it silently drifts — running week-old code while you assume the fix is live. There was no way to *see* which node ran which version, and no way to push an update without a manual SSH tour of the fleet.
|
|
|
|
For a platform whose entire value is *fast iteration on real hardware*, that update loop was the bottleneck. So we built distributed hot-deploy.
|
|
|
|
## What it looks like now
|
|
|
|
There's a new owner-only **Deploy** panel. It shows every node, its **running version** versus the **latest built version**, and whether it's busy. One click — per node or "deploy all stale" — pushes new code and restarts the node onto it. No SSH.
|
|
|
|
```
|
|
node running latest status action
|
|
fei-android 1c8734a 1c8734a ● 最新 —
|
|
fei-ios-01 0.1.0 1c8734a ▲ 待更新 [ Deploy ]
|
|
pns-android 0.1.0 1c8734a ▲ 待更新 [ Deploy ]
|
|
[ Deploy all stale ]
|
|
```
|
|
|
|
Under the hood, a deploy is four moves:
|
|
|
|
```
|
|
CONTROL PLANE (your laptop) REMOTE SATELLITE
|
|
─────────────────────────── ────────────────────────────
|
|
1. build bundle ──► satellite.js run-satellite.sh (keep-alive loop)
|
|
stamp git SHA ──► VERSION └─ bun satellite.js
|
|
reports version = <SHA>
|
|
2. POST /sat/admin/deploy ─────────────►
|
|
body = satellite.js bytes
|
|
header = sha256, version
|
|
3. verify sha256
|
|
atomic-swap the bundle (+ .bak)
|
|
exit(0)
|
|
│
|
|
4. wrapper relaunches on NEW code
|
|
re-registers with new SHA
|
|
▼
|
|
panel: running == latest ✓ ◄────── heartbeat
|
|
```
|
|
|
|
The version each node reports is now its **git commit SHA**, stamped into the bundle at build time. That turns "is this node up to date?" from a guess into a fact the UI renders directly — and it's how a deploy confirms success: the node comes back reporting the new SHA, and its badge flips from *待更新* to *最新* on its own.
|
|
|
|
## The interesting decision: bundle push, not `git pull`
|
|
|
|
The obvious way to update a node that shares the same repo is to `git pull` on it. We deliberately didn't. Here's the reasoning, because it says something about what kind of system Streamline is.
|
|
|
|
### Why a bundle wins for a prototyping platform
|
|
|
|
**1. You can deploy uncommitted code.** This is the big one. Our actual development loop is *change a line → see it run on a real device*. With `git pull`, every iteration means commit, push to origin, then pull on the node — a full round-trip through version control just to test a hunch. The bundle push ships **exactly what's in your working tree right now**, dirty or not. (The version stamp even carries a `-dirty` suffix so you know it.) For rapid prototyping, being able to fling work-in-progress at a device in one click is the whole point.
|
|
|
|
**2. No git credentials on every node.** `git pull` means each machine needs a deploy key or token to the origin, kept valid over time. The bundle push reuses the shared secret that already authenticates the control→satellite channel. Zero new secrets to distribute or rotate across the fleet.
|
|
|
|
**3. No second network dependency.** The control plane already reaches each satellite — that's how it proxies logs and evidence. `git pull` would add a *separate* requirement: every node must also reach the git host. For a node on a different network, that may not hold even when the control plane can talk to it fine. Pushing over the channel that already works removes a whole failure mode.
|
|
|
|
**4. Atomic and reversible.** The bundle is a single 0.35 MB file. A deploy verifies its sha256, writes to a temp sibling, and atomically renames it into place — keeping the previous file as `.bak` for instant rollback. A mid-write crash can never leave a truncated bundle the node would try to boot. `git pull` on a node can hit a dirty index, a merge conflict, or a half-applied state, and needs a clean tree to be safe.
|
|
|
|
### What `git pull` is genuinely better at
|
|
|
|
We're not dogmatic. If your discipline is "only ever deploy tagged, pushed commits," `git pull` gives you a more auditable node state — a real git ref you can inspect, not an opaque blob. That's the right trade for a locked-down production fleet. Streamline is the opposite: an experimentation platform where the cost of a slow iteration loop is higher than the cost of an opaque deploy. So we optimized for *speed of iteration*, and the bundle won.
|
|
|
|
*(We also considered a fancier bundle protocol — serve a manifest, have the node pull it back and verify. For a 0.35 MB file that was pure ceremony. The whole thing collapses to a single POST with a sha256 header. Simplicity won there too.)*
|
|
|
|
## The part everyone forgets: restart
|
|
|
|
Pushing new code is only half the job. The process has to **restart onto it**. A satellite launched with bare `nohup` can't — when it exits, it stays dead, and your "deploy" just bricked the node.
|
|
|
|
So the deploy exits the process *deliberately* (`exit(0)` after the swap), and a tiny keep-alive wrapper brings it back:
|
|
|
|
```bash
|
|
# run-satellite.sh — the supervisor that makes hot-deploy possible
|
|
while true; do
|
|
bun satellite/dist/satellite.js
|
|
echo "exited; restarting"
|
|
sleep 2
|
|
done
|
|
```
|
|
|
|
Deploy = swap the bundle, exit, let the wrapper relaunch on the new file. It's the simplest possible supervisor — no systemd, no pm2, no per-node service install — and it's the one piece that has to exist on a node *before* hot-deploy can touch it. (Installing it is the single one-time manual step per node; after that, every update is a click.)
|
|
|
|
## Safety rails
|
|
|
|
- **sha256 verified before swap** — a corrupted or truncated push is rejected with the running bundle untouched.
|
|
- **Busy nodes are skipped** — a node mid-evaluation or mid-fix isn't interrupted unless you explicitly force it (a deploy would abort its task).
|
|
- **Version-confirmed** — success isn't "the POST returned 200," it's "the node came back reporting the new SHA." The panel shows you that flip.
|
|
- **One-click rollback** — the previous bundle is always one `.bak` away.
|
|
|
|
## The takeaway
|
|
|
|
The design reflects the system. Streamline exists to shorten the loop between *an idea about mobile behavior* and *seeing it happen on a real phone*. A deploy mechanism that forced a commit-push-pull detour through git — with credentials and a second network hop — would have taxed exactly the loop we're trying to make fast. Pushing a self-contained bundle over the channel we already trust, atomically, WIP and all, keeps that loop tight.
|
|
|
|
Sometimes the "less proper" engineering choice is the right one — when you're honest about what the system is *for*.
|