Initial import of streamline

This commit is contained in:
2026-11-16 17:26:44 -08:00
commit 8e4aec1855
1221 changed files with 274256 additions and 0 deletions
+26
View File
@@ -0,0 +1,26 @@
# 失败片段审计规约
分类只回答「当时为什么浪费」,不评价最终代码,也不因为输出里出现 `error` 就判失败。
- `dead_end`:这条路在当前环境客观走不通,后续仍重复撞同一堵墙。典型是权限、设备、
版本或工具能力不满足;换措辞不能让这条路变通。
- `decision_error`:在已有信息足够的情况下选错了下一步,或没有在失败后切换策略。
反复手写 sleep / 轮询等待 App、原样重复同族命令、Frida 报错后直接再次注入都属于此类。
- `context_missing`:做对下一步所需的信息在当时上下文中还没有出现;不是 agent 忽略了
已知信息。必须指出缺什么,不能把「没有想到」改写成「缺上下文」。
- `not_failure`:切片误报。读取源码、日志或历史记录时看到的 `error/failed` 字样;
探测命令预期返回非零;后台任务启动回执;或者片段没有形成真实的重复失败。
`correctInfoStep` 是时间线中的调用编号:从该步结果开始,做对下一步所需的信息已经可见。
确实从未出现时填 `null`。证据必须逐字引用时间线,至少一条,并足以支持分类。
`replayType` 只按可封闭判分的动作形态选择:
- `T2_wait`:问题核心是等待 App / UI / 进程就绪;下一步可判为 wait-for / 后台监听,
或手写长 sleep / 轮询。
- `T1_repeat`:第一次失败后是否再次执行同族命令。
- `T3_frida`:Frida 首次失败后是否直接再次注入,而不是先诊断 server、版本、进程或代码。
- `none`:不属于以上三种,不能进入自动离线闸。
`cluster` 要描述可重复出现的机制,不写工单名、账号、节点名或具体 id,例如
`app_ready_blind_wait`、`repeat_same_failed_command`、`frida_retry_without_diagnosis`。
+99
View File
@@ -0,0 +1,99 @@
# 自动 RSI 回路进程
每天执行一次:
1. 从 `/api/rsi-loop/nodes/:id/transcripts` 同步近 14 天原生执行会话;
2. 切出同一堵墙 / 同族命令重复失败至少两次的片段(读取类命令不算失败),以及所有「要开始等界面 / App 就绪」
的时刻(Android 认 dumpsys / uiautomator / pidof / actl,iOS 认 catuse);
3. 最多审计 60 段,分为 `dead_end / decision_error / context_missing / not_failure`;
4. 按浪费的失败调用数选一个问题模式,在现行 prompt 上只插入一段新纪律;
5. 目标点和其它模式保留集分别做配对重放,每点 base / candidate 各跑三遍;
6. 提交实验(离线未过也提交),并给已结束实验补 AI 结论和每日运行摘要;
7. 还空着的实验机(该平台没有进行中的 bug.run 实验),各提一个**旋钮实验**:从 `/state` 的
`runtimeKnobs` 目录里挑该平台还没试过的取值(每个取值只试一次,跳过实验机现行的取值,
见 `config.LAB_BASELINE`),不走离线、直接真机复跑(`rsi_loop/runtime.py`)。
8. **复核意见挖掘**(`rsi_loop/review_mining.py`,与实验互不相干,回路暂停时照跑):给新的人工复核意见打标签;
每周一(北京时间)把近 90 天的编码习惯归纳成候选规范交给控制面过门槛,管理员在 RSI 页「复核意见挖掘」批准。
提示词由控制面下发,这里不改一个字;首次上线先补历史:`./run.sh review-mining --days 180`(每次最多 400 条,
多跑几次直到待标为 0),再 `./run.sh review-mining --cluster --no-label` 立刻归纳一次。见 `docs/rsi.md` 例子 14。
离线通过的实验由控制面转入**真机复跑**:评估集里跑过的历史任务在实验机上现行 / 候选各跑一遍
(见 `docs/rsi.md` 例子 13),判定由控制面的判定任务执行,复跑结论就是实验终点,不自动改生产
prompt。线上灰度已于 2026-10-05 下线(新任务之间差别太大,比不出改动的效果)。所以本进程只提
`bug.run`、且该平台配了实验机的问题模式(从 `/state` 的 `labNodes` 读)。
## temp-ios-01
代码只依赖系统 Python 3.9 标准库和 Node 24(actl MCP 用)。目录默认是
`~/rsi_loop`,令牌默认读 `~/rsi_loop/token`:
```bash
mkdir -p ~/rsi_loop
chmod 700 ~/rsi_loop
printf '%s' '<与控制面 RSI_LOOP_TOKEN 相同的值>' > ~/rsi_loop/token
chmod 600 ~/rsi_loop/token
export PYTHONPATH=/path/to/streamline/scripts/rsi_loop
export RSI_LOOP_HOME="$HOME/rsi_loop"
# 不设 RSI_LOOP_NODES = 自动发现控制面登记的全部节点;没开放会话接口的节点(没配令牌 /
# 卫星版本旧)返回 404,记进 sync.notEnabled 后跳过
export RSI_LOOP_ACTL="$HOME/rsi_loop/actl/lib/index.js"
```
先在图形桌面终端确认 `~/.local/bin/claude -p '只回答 OK'` 能用,然后**先试运行**:
```bash
cd ~/rsi_loop/app
./run.sh dry-run
```
试运行对控制面只读:同步、切片、审计、提案、重放都真实执行,但提交实验和写时间线不发出去,
改记到 `~/rsi_loop/data/dry_posts.jsonl`。看过要提交的实验(标题、四段记录、候选改动、离线结果)
没问题,再正式跑一轮 —— 正式跑一旦离线闸通过,控制面就在**该问题所属平台**的实验机上建复跑任务:
```bash
./run.sh once
```
确认 RSI 页面出现运行日志后,在同一个图形桌面终端常驻:
```bash
./run.sh daemon
```
默认每天本机时间 03:00 运行,可用 `RSI_LOOP_DAILY_HOUR` 调整。不要从普通 SSH 会话
启动:该机器的 Claude OAuth 在 SSH 环境里会过期。
常用参数:
```bash
RSI_LOOP_MAX_AUDITS=60 # 每轮最多审计片段
RSI_LOOP_TARGET_POINTS=36 # 目标类重放点
RSI_LOOP_HOLDOUT_POINTS=15 # 其它模式保留集
RSI_LOOP_SEEDS=3 # 每个点、每个版本跑三遍(两遍时区间太宽,打平的候选一半被误拦)
RSI_LOOP_WORKERS=4 # 并发 Claude 续跑数
RSI_LOOP_MAX_REPLAYS=320 # 单轮续跑硬上限;要 ≥ (目标点+保留集点)×2×遍数,否则先挤掉保留集
```
## 安全与可重算性
- 原生会话、审计证据和重放原始输出只落在 `~/rsi_loop/data`,目录权限 `0700`、
文件权限 `0600`,默认 14 天删除;发给控制面的日志只有聚合数字和模式名。
- 重放只取模型第一步动作。所有工具由 `PreToolUse` 钩子拒绝,不执行设备、文件或网络工具。
- 失败点截在工具结果之后;有未结束后台任务、缺环境附件或上下文过大的点不参与重放。
- 候选新增行按提案给的锚点行插进历史会话第一条用户 prompt 的同一位置(锚点里的占位符按
通配匹配,不加任何标题),与生产渲染的位置一致;锚点在某个点上定位不到,这个点两组都不跑,
剔除数记在 `offline.meta`。base/candidate 使用同一历史前缀,判分只看封闭的动作形态,不用 LLM judge。
- 保留集只取同一 stage、审计过且不是误报的失败点,加上同 stage 的等待点。
- 不提交的情况:目标点不足 15 个(选题时先数,锚点剔除后再数一次),或重放报错率超过 20%
(基础设施故障不能记成「候选无效」)。只写一条运行日志,并在本地记一次冷却。
- 每次重放的逐条结果保存在 `data/runs/<时间>/results.jsonl`;95% 区间以固定种子对
决策点做 paired bootstrap,可离线重算。
- 暂停时仍可同步、审计和补结论,但不会提交新实验;控制面也会以 409 再挡一次。
本地测试(不访问控制面、不调用 Claude):
```bash
cd scripts/rsi_loop
python3 -m unittest discover -s tests -v
```
+3
View File
@@ -0,0 +1,3 @@
"""Streamline 自动 RSI 回路进程(Python 3.9 标准库)。"""
__version__ = "0.1.0"
+27
View File
@@ -0,0 +1,27 @@
import argparse
import json
from . import runner
def main() -> None:
parser = argparse.ArgumentParser(description="Streamline 自动 RSI 回路")
parser.add_argument("command", choices=("once", "dry-run", "daemon", "review-mining"), nargs="?", default="once")
args, rest = parser.parse_known_args()
if args.command == "dry-run":
from . import config
config.DRY_RUN = True
print(json.dumps(runner.locked_run(), ensure_ascii=False, indent=2))
elif args.command == "daemon":
runner.daemon()
elif args.command == "review-mining":
# 单独跑挖掘(补标历史 / 立刻归纳一次),参数见 review_mining.py 头注释
from . import config, review_mining
config.ensure_dirs()
print(json.dumps(review_mining.main(rest), ensure_ascii=False, indent=2))
else:
print(json.dumps(runner.locked_run(), ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
+78
View File
@@ -0,0 +1,78 @@
"""控制面 /api/rsi-loop/* 的客户端(回路令牌鉴权)。
回路进程只能读状态、取会话记录、提交实验、写时间线;上线 / 回滚 / revert 由控制面
的判定任务决定,这里没有、也调不到那些动作。
"""
import json
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any, Optional
from . import config
def _req(method: str, path: str, body: Optional[dict] = None, timeout: int = 60):
data = json.dumps(body, ensure_ascii=False).encode() if body is not None else None
req = urllib.request.Request(config.API + path, data=data, method=method)
req.add_header("x-rsi-loop-token", config.token())
if data is not None:
req.add_header("content-type", "application/json")
return urllib.request.urlopen(req, timeout=timeout)
def get(path: str, timeout: int = 60) -> Any:
with _req("GET", path, timeout=timeout) as r:
return json.loads(r.read().decode())
def post(path: str, body: dict, timeout: int = 120) -> Any:
if config.DRY_RUN:
# 试运行:不写生产,把本来要发的内容留在本地供检查
out = config.DATA / "dry_posts.jsonl"
with open(out, "a") as f:
f.write(json.dumps({"path": path, "body": body}, ensure_ascii=False) + "\n")
out.chmod(0o600)
print(f"[rsi-loop] dry-run: POST {path} 未发送,已记到 {out}", flush=True)
if path == "/experiments":
return {"experiment": {"id": None, "title": body.get("title"), "status": "dry_run"}, "gate": None}
return {}
try:
with _req("POST", path, body, timeout=timeout) as r:
return json.loads(r.read().decode())
except urllib.error.HTTPError as e:
detail = e.read().decode(errors="replace")[:500]
raise RuntimeError(f"POST {path} -> HTTP {e.code}: {detail}") from None
def download(path: str, dest: Path, timeout: int = 900) -> int:
tmp = dest.with_suffix(dest.suffix + ".part")
with _req("GET", path, timeout=timeout) as r, open(tmp, "wb") as f:
n = 0
while True:
chunk = r.read(1 << 20)
if not chunk:
break
f.write(chunk)
n += len(chunk)
tmp.replace(dest)
return n
def event(title: str, body: str = "", type_: str = "run") -> bool:
"""回路层面的时间线事件(RSI 页面「回路日志」)。失败不影响主流程。"""
try:
post("/events", {"type": type_, "title": title[:280], "body": body[:20000]})
return True
except Exception as e: # noqa: BLE001
print(f"[rsi-loop] event post failed: {e}")
return False
def experiment_event(exp_id: str, title: str, body: str = "", type_: str = "note") -> bool:
try:
post(f"/experiments/{exp_id}/events", {"type": type_, "title": title[:280], "body": body[:12000]})
return True
except Exception as e: # noqa: BLE001
print(f"[rsi-loop] experiment event failed: {e}")
return False
+91
View File
@@ -0,0 +1,91 @@
"""回路进程的配置 —— 全部来自环境变量,缺省值对应 temp-ios-01 上的布局。
只依赖 Python 3.9 标准库:temp-ios-01 上只有系统自带的 python3,不装任何东西。
"""
import json
import os
from pathlib import Path
HOME = Path.home()
ROOT = Path(os.environ.get("RSI_LOOP_HOME", str(HOME / "rsi_loop")))
DATA = ROOT / "data"
TX_DIR = DATA / "tx" # 原生会话记录(按节点分目录),保留 RETAIN_DAYS 天
STATE = DATA / "state.json" # 每个节点上次同步到的时刻等
EPISODES = DATA / "episodes.jsonl" # 切出来的多轮失败片段(按会话文件缓存)
POINTS = DATA / "points.jsonl" # 可重放的决策点(失败片段 + 等待起点)
AUDITS = DATA / "audits.jsonl" # 审计结论
RUNS = DATA / "runs" # 每次重放的原始结果,留档可重算
LOCK = DATA / "runner.lock"
API = os.environ.get("RSI_LOOP_API", "https://streamline-internal-api.bytedance.net/api/rsi-loop")
TOKEN_FILE = Path(os.environ.get("RSI_LOOP_TOKEN_FILE", str(ROOT / "token")))
# 数据源节点(逗号分隔)。不设 = 自动发现控制面登记的全部节点,逐个尝试;只有 .env 里配了
# RSI_LOOP_TOKEN、且跑到带会话记录接口的卫星版本的节点才会开放,其余返回 404,静默跳过。
NODES = [n.strip() for n in os.environ.get("RSI_LOOP_NODES", "").split(",") if n.strip()]
CLAUDE_BIN = os.environ.get("CLAUDE_BIN", str(HOME / ".local" / "bin" / "claude"))
# 重放必须用会话当时的模型(否则 thinking 签名对不上);审计 / 提案 / 写结论用同一个即可。
REPLAY_MODEL = os.environ.get("RSI_LOOP_REPLAY_MODEL", "claude-opus-5[1m]")
THINK_MODEL = os.environ.get("RSI_LOOP_MODEL", "claude-opus-5")
# actl 设备工具的 MCP 服务(只为让重放时的工具定义与节点一致;调用全部被钩子拦下)
ACTL_INDEX = os.environ.get("RSI_LOOP_ACTL", str(ROOT / "actl" / "lib" / "index.js"))
RETAIN_DAYS = int(os.environ.get("RSI_LOOP_RETAIN_DAYS", "14"))
SYNC_DAYS = int(os.environ.get("RSI_LOOP_SYNC_DAYS", "14"))
# 2026-10-08 由 40 调到 60:每天新片段约 40 段,40 的上限让积压一直消不掉(当时积压 41 段)。
MAX_AUDITS_PER_RUN = int(os.environ.get("RSI_LOOP_MAX_AUDITS", "60"))
# 一次重放的总调用数上限:(目标 36 + 保留集 15) × 两组 × 3 遍 = 306,留一点余量。
# 上限不够时先挤掉保留集 —— 上限和点数、遍数要一起改。
MAX_REPLAYS_PER_RUN = int(os.environ.get("RSI_LOOP_MAX_REPLAYS", "320"))
WORKERS = int(os.environ.get("RSI_LOOP_WORKERS", "4"))
# 每个点每组跑几遍。2026-10-03 用 24 点 × 2 遍时差值区间半宽约 ±0.11,与现行打平的候选
# 有一半会被「不能明显变差」(下界 ≥ −0.10)误拦;36 点 × 3 遍把半宽收到约 ±0.08。
SEEDS = int(os.environ.get("RSI_LOOP_SEEDS", "3"))
# 重放成本控制:截断处上下文超过这个 token 数的决策点不用
MAX_POINT_CTX = int(os.environ.get("RSI_LOOP_MAX_POINT_CTX", "250000"))
TARGET_POINTS = int(os.environ.get("RSI_LOOP_TARGET_POINTS", "36"))
HOLDOUT_POINTS = int(os.environ.get("RSI_LOOP_HOLDOUT_POINTS", "15"))
# 离线闸(与 shared/src/rsi_loop.ts 的 LOOP 一致;控制面会再按它的规则复核一遍)
OFFLINE_MIN_POINTS = 15
OFFLINE_HOLDOUT_TOLERANCE = 0.10
# 同一类问题被试过之后的冷却期(天):避免对同一个模式反复提几乎一样的改法
CLUSTER_COOLDOWN_DAYS = int(os.environ.get("RSI_LOOP_COOLDOWN_DAYS", "7"))
# 旋钮实验:实验机上现行组已经是这个取值的旋钮,不能再当候选(两组一样,白跑一轮)。
# 实验机不传 --effort 时用的是各自 ~/.claude/settings.json 的 modelSettings.<模型>.effortLevel:
# evaluator-android-01 是 xhigh(2026-10-08 查),temp-ios-01 没设(模型默认)。实验机配置改了要同步这里。
LAB_BASELINE = json.loads(os.environ.get("RSI_LOOP_LAB_BASELINE", '{"android": {"effort": "xhigh"}}'))
# 思考强度的候选顺序:先试离现状近、省时间的方向。「不设」对应的模型默认档未知,放在最后。
EFFORT_ORDER = ["medium", "high", "low", "max", "xhigh"]
# 试运行:对控制面只读,所有写操作(提交实验、写时间线)改记到 data/dry_posts.jsonl
DRY_RUN = os.environ.get("RSI_LOOP_DRY_RUN") == "1"
# 每天几点(本机时间)跑一次
DAILY_HOUR = int(os.environ.get("RSI_LOOP_DAILY_HOUR", "3"))
def log(msg: str) -> None:
"""阶段进度。整轮要一两个小时,不出声的话没法分辨「在干活」还是「卡死了」。"""
import time as _t
print(f"[rsi-loop {_t.strftime('%H:%M:%S')}] {msg}", flush=True)
def child_env(cwd: str) -> dict:
"""给 claude 子进程的环境:PWD 指向它自己的工作目录。
Claude Code 会参考 PWD 确定工作目录;继承回路进程的 PWD(…/scripts/rsi_loop)会让模型在
重放里看到「rsi_loop」,第一步就去查它。回路自己的变量、PYTHONPATH 也一并去掉。"""
drop = ("PWD", "OLDPWD", "PYTHONPATH")
env = {k: v for k, v in os.environ.items()
if not k.startswith(("CLAUDE", "RSI_LOOP")) and k not in drop}
env["PWD"] = cwd
return env
def token() -> str:
return TOKEN_FILE.read_text().strip()
def ensure_dirs() -> None:
for d in (DATA, TX_DIR, RUNS):
d.mkdir(parents=True, exist_ok=True)
d.chmod(0o700)
+92
View File
@@ -0,0 +1,92 @@
"""把一份执行会话切成「多轮失败片段」:同一堵墙、或同一条路连续失败 ≥2 次的一段。
每个片段标出能用来做决策点重放的截断位置(片段内第一次失败的工具结果之后)。
片段同时产出一个候选决策点:
- 撞墙涉及 frida → T3_frida
- 下一步是等待类命令 → T2_wait(改成挑「所有要开始等待的时刻」更干净,见 harvest)
- 其余同族连续失败 → T1_repeat
合格的决策点要求:截断时没有在跑的后台任务、会话里有环境附件(能中和)、上下文不超限。
"""
import re
from typing import List, Optional
from . import transcript as T
# UI 相关的等待(dumpsys/uiautomator/pidof/actl 等),T2 的判定基准
# 「等 App / 界面就绪」的探查命令。Android:dumpsys / uiautomator / pidof / actl;iOS:catuse(界面大纲、截屏、
# 进程列表都走它)。2026-10-08 之前只认 Android,iOS 的 104 段 bug.run 会话一个等待点都抽不出来。
UI_LIKE = re.compile(r"dumpsys\s+(window|activity)|uiautomator|mCurrentFocus|ResumedActivity|\bpidof\b|\bactl\b|\bcatuse\b")
def find_episodes(rows: List[Optional[dict]]) -> List[dict]:
cs = T.calls(rows)
n = len(cs)
eps: List[dict] = []
# 按「墙」聚合:同一堵墙撞 ≥2 次
by_wall: dict = {}
for c in cs:
if c["wall"]:
by_wall.setdefault(c["wall"], []).append(c)
for w, hits in by_wall.items():
if len(hits) < 2:
continue
fams = sorted({h["fam"] for h in hits})
ptype = "T3_frida" if any("frida" in f for f in fams) else "T1_repeat"
eps.append(_episode(cs, hits, label=f"wall:{w}", ptype=ptype, total=n))
# 按命令族聚合:同族连续失败 ≥2 次(排除只读命令的常态失败)
open_ep: dict = {}
for c in cs:
if c["bg"] or c["read_only"]:
continue
f = c["fam"]
if not c["ok"]:
open_ep.setdefault(f, []).append(c)
elif f in open_ep:
hits = open_ep.pop(f)
if len(hits) >= 2:
ptype = "T3_frida" if "frida" in f else "T1_repeat"
eps.append(_episode(cs, hits, label=f"path:{f}", ptype=ptype, total=n, resolved_at=c["i"]))
for f, hits in open_ep.items():
if len(hits) >= 2:
ptype = "T3_frida" if "frida" in f else "T1_repeat"
eps.append(_episode(cs, hits, label=f"path:{f}", ptype=ptype, total=n))
# 同一组失败常会同时命中 wall 与 path;只审计一次,否则浪费会被双算。
unique = {}
for ep in eps:
unique.setdefault((ep["first_call"], ep["last_call"], ep["ptype"]), ep)
return list(unique.values())
def _episode(cs: List[dict], hits: List[dict], label: str, ptype: str, total: int, resolved_at: Optional[int] = None) -> dict:
first, last = hits[0], hits[-1]
return dict(label=label, ptype=ptype, n_fail=len(hits), first_call=first["i"], last_call=last["i"],
total_calls=total, fam=first["fam"], err=first["err"], err_last=last["err"],
fail_cmds=[h["cmd"][:240] for h in hits[:4]], resolved=resolved_at is not None,
# 决策点:截在第一次失败的工具结果之后(含),让模型面对「刚撞了一次墙」。
# 截在前一个结果会让模型尚未见到失败,无法判定下一步是否在重复撞墙。
point_call=first["i"], point_row=first["row"], point_ctx=first["ctx"])
def harvest_wait_points(rows: List[Optional[dict]]) -> List[dict]:
"""T2_wait 不按结果挑,而是挑所有「要开始等待」的时刻 —— 下一步该用 wait-for 还是手写 sleep,
由重放决定,这样历史比例才能拿来和重放对照。"""
cs = T.calls(rows)
out: List[dict] = []
last_result_call: Optional[dict] = None
for c in cs:
cmd = c["cmd"]
if c["tool"] == "Bash" and (("wait-for" in cmd) or _is_wait(cmd)) and UI_LIKE.search(cmd) and last_result_call is not None:
out.append(dict(label="wait", ptype="T2_wait", first_call=c["i"], last_call=c["i"], total_calls=len(cs),
fam=c["fam"], err="", fail_cmds=[cmd[:240]], resolved=True,
point_call=last_result_call["i"], point_row=last_result_call["row"], point_ctx=last_result_call["ctx"]))
last_result_call = c
return out
def _is_wait(cmd: str) -> bool:
if re.search(r"\b(while|until|for)\b.*\bsleep\b", cmd, re.S):
return True
return any(int(x) >= 15 for x in re.findall(r"\bsleep\s+(\d+)", cmd))
+43
View File
@@ -0,0 +1,43 @@
"""调 claude 做「读材料、写判断」的活:审计分类、写提案、写结论。
全部是纯文本进、纯文本出:`--tools ""` 关掉所有工具,`--setting-sources ""` 不加载本机
任何设置,材料从标准输入喂进去。它不碰任何系统,只产出要被下游(控制面的规则)检验的东西。
"""
import json
import os
import re
import subprocess
from typing import Any, Optional
from . import config
def ask(prompt: str, timeout: int = 900, model: Optional[str] = None) -> str:
env = config.child_env("/tmp")
cmd = [config.CLAUDE_BIN, "-p", "--output-format", "json", "--model", model or config.THINK_MODEL,
"--tools", "", "--setting-sources", "", "--strict-mcp-config"]
p = subprocess.run(cmd, input=prompt, capture_output=True, text=True, timeout=timeout, env=env, cwd="/tmp")
try:
d = json.loads(p.stdout)
except json.JSONDecodeError:
raise RuntimeError(f"claude 输出不是 JSON: {p.stdout[:300]} {p.stderr[:300]}") from None
if d.get("is_error"):
raise RuntimeError(f"claude 报错: {str(d.get('result'))[:300]}")
return str(d.get("result") or "")
def ask_json(prompt: str, timeout: int = 900) -> Any:
"""要求模型只回一个 JSON 对象;用 JSONDecoder 取第一个完整对象(允许嵌套)。"""
text = ask(prompt, timeout=timeout)
fenced = re.search(r"```(?:json)?\s*([\s\S]*?)```", text)
candidates = [fenced.group(1)] if fenced else []
candidates.append(text)
decoder = json.JSONDecoder()
for candidate in candidates:
for match in re.finditer(r"\{", candidate):
try:
value, _ = decoder.raw_decode(candidate[match.start():])
return value
except json.JSONDecodeError:
continue
raise RuntimeError(f"回复里没有合法 JSON: {text[:400]}") from None
+511
View File
@@ -0,0 +1,511 @@
"""发现:同步会话记录 → 切出多轮失败片段 → 审计分类 → 按问题模式聚类。
切片只是候选:自动从输出文本判断「失败」有误报(读到的源码里的报错字样等),所以每个
片段都要过一遍审计。审计用模型,但它只产出「假设」—— 问题属于哪一类、哪一步之后本该
做对 —— 真正决定改动去留的是离线重放与真机复跑的配对结果,不是这里的判断。
"""
import json
import hashlib
import urllib.error
import os
import time
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional
from urllib.parse import quote
from . import api, config, episodes, llm
from .transcript import (
classify_stage, first_prompt, has_env_attachment, load_rows, pending_bg,
render_timeline, text_of,
)
RUBRIC = (Path(__file__).resolve().parent.parent / "AUDIT_RUBRIC.md")
# ── 同步 ─────────────────────────────────────────────────────────────────────
def _state() -> dict:
try:
value = json.loads(config.STATE.read_text())
return value if isinstance(value, dict) else {}
except (OSError, json.JSONDecodeError):
return {}
def _write_json(path: Path, value: Any) -> None:
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(json.dumps(value, ensure_ascii=False, indent=2))
os.chmod(tmp, 0o600)
tmp.replace(path)
def _write_jsonl(path: Path, rows: Iterable[dict]) -> None:
tmp = path.with_suffix(path.suffix + ".tmp")
with open(tmp, "w") as f:
for row in rows:
f.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
os.chmod(tmp, 0o600)
tmp.replace(path)
def _read_jsonl(path: Path) -> List[dict]:
out: List[dict] = []
try:
with open(path, errors="replace") as f:
for line in f:
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(row, dict):
out.append(row)
except FileNotFoundError:
pass
return out
def _id(*parts: object) -> str:
raw = "\0".join(str(p) for p in parts).encode()
return hashlib.sha256(raw).hexdigest()[:24]
def _local_file(node: str, name: str) -> Path:
return config.TX_DIR / quote(node, safe="._-") / name
def _purge_expired(now_ms: int) -> int:
cutoff = now_ms - config.RETAIN_DAYS * 86_400_000
removed = 0
for node_dir in config.TX_DIR.glob("*"):
if not node_dir.is_dir():
continue
for meta_path in node_dir.glob("*.meta.json"):
try:
meta = json.loads(meta_path.read_text())
expired = int(meta.get("mtime") or 0) < cutoff
except (OSError, ValueError, json.JSONDecodeError):
expired = True
if not expired:
continue
raw = meta_path.with_name(meta_path.name.removesuffix(".meta.json") + ".jsonl")
raw.unlink(missing_ok=True)
meta_path.unlink(missing_ok=True)
removed += 1
try:
node_dir.rmdir()
except OSError:
pass
return removed
def sync_transcripts(now_ms: Optional[int] = None) -> dict:
"""只同步最近 14 天的执行会话;同 size/mtime 的文件不重复下载。"""
config.ensure_dirs()
now_ms = now_ms or int(time.time() * 1000)
since = now_ms - config.SYNC_DAYS * 86_400_000
stats = {"nodes": 0, "notEnabled": [], "listed": 0, "downloaded": 0, "bytes": 0, "errors": [], "purged": 0}
state = _state()
platforms = node_platforms()
nodes = config.NODES or sorted(platforms)
for node in nodes:
try:
try:
listing = api.get(f"/nodes/{quote(node, safe='')}/transcripts?sinceMs={since}", timeout=900)
except urllib.error.HTTPError as e:
if e.code == 404: # 没配令牌 / 卫星版本旧:不是故障,记下来提示人去开
stats["notEnabled"].append(node)
continue
raise
files = listing.get("files") or []
config.log(f"同步 {node}({platforms.get(node, '?')}):列出 {len(files)} 份会话")
stats["nodes"] += 1
stats["listed"] += len(files)
node_dir = _local_file(node, "_").parent
node_dir.mkdir(parents=True, exist_ok=True)
node_dir.chmod(0o700)
for info in files:
name = str(info.get("file") or "")
if not name.endswith(".jsonl"):
continue
dest = _local_file(node, name)
meta_path = dest.with_name(dest.stem + ".meta.json")
old = {}
try:
old = json.loads(meta_path.read_text())
except (OSError, json.JSONDecodeError):
pass
if (dest.exists() and int(old.get("size") or -1) == int(info.get("size") or -2)
and int(old.get("mtime") or -1) == int(info.get("mtime") or -2)):
if not old.get("platform") and platforms.get(node):
_write_json(meta_path, {**old, "platform": platforms[node]})
continue
# 大文件经控制面代理偶尔中途断开(IncompleteRead);逐个文件重试,
# 一份失败不能连累这个节点后面所有文件。
n = None
for attempt in range(3):
try:
n = api.download(f"/nodes/{quote(node, safe='')}/transcripts/{quote(name, safe='')}", dest)
break
except Exception as exc: # noqa: BLE001
last = exc
time.sleep(5 * (attempt + 1))
if n is None:
stats["errors"].append(f"{node}/{name}: {last}")
continue
os.chmod(dest, 0o600)
meta = {
"node": node, "file": name, "size": n,
"mtime": int(info.get("mtime") or now_ms),
"stage": info.get("stage"), "workspaceKey": info.get("workspaceKey"),
"platform": platforms.get(node) or platform_of(node),
}
_write_json(meta_path, meta)
stats["downloaded"] += 1
stats["bytes"] += n
config.log(f" 下载 {stats['downloaded']}:{name[:8]} {n / 1e6:.1f}MB")
state.setdefault("nodes", {})[node] = {"syncedAt": now_ms, "sinceMs": since}
except Exception as exc: # noqa: BLE001
stats["errors"].append(f"{node}: {exc}")
stats["purged"] = _purge_expired(now_ms)
state["lastSyncAt"] = now_ms
_write_json(config.STATE, state)
return stats
def platform_of(node: str) -> str:
"""节点名里带 ios 的是 iOS,其余按 Android(只在控制面没返回平台时兜底)。"""
return "ios" if "ios" in (node or "").lower() else "android"
def node_platforms() -> Dict[str, str]:
"""控制面登记的节点 → 平台。"""
try:
return {n["id"]: n.get("platform") or platform_of(n["id"]) for n in api.get("/nodes").get("nodes") or []}
except Exception as exc: # noqa: BLE001
config.log(f"取节点列表失败,只用 RSI_LOOP_NODES:{exc}")
return {n: platform_of(n) for n in config.NODES}
def transcript_files() -> List[tuple]:
out = []
for meta_path in config.TX_DIR.glob("*/*.meta.json"):
try:
meta = json.loads(meta_path.read_text())
except (OSError, json.JSONDecodeError):
continue
raw = meta_path.with_name(meta_path.name.removesuffix(".meta.json") + ".jsonl")
if raw.exists():
out.append((raw, meta))
return out
def mine_local() -> dict:
"""重建最近会话的片段和决策点缓存。缓存不含完整工单正文。"""
eps: List[dict] = []
points: List[dict] = []
bad_files = 0
for path, meta in transcript_files():
# 真机复跑任务(控制面 rsi_lab.ts,工作区名带 rsi-lab- 前缀)跑在实验机上,也会出现在数据源
# 节点的会话列表里;它们是回路自己的实验,不能再被当成生产样本挖回来
if str(meta.get("workspaceKey") or "").startswith("rsi-lab-"):
continue
rows = load_rows(path)
prompt = first_prompt(rows)
stage = classify_stage(text_of((prompt or {}).get("message", {}).get("content"))) if prompt else None
if not stage:
bad_files += 1
continue
common = {
"node": meta.get("node"), "file": meta.get("file"), "mtime": meta.get("mtime"),
"platform": meta.get("platform") or platform_of(str(meta.get("node"))),
"stage": stage, "has_env": has_env_attachment(rows),
}
for ep in episodes.find_episodes(rows):
item = {**common, **ep}
item["id"] = _id(item["node"], item["file"], item["label"], item["first_call"], item["last_call"])
eps.append(item)
point = _point_from_episode(item, rows)
if point:
points.append(point)
for wp in episodes.harvest_wait_points(rows):
item = {**common, **wp}
item["id"] = _id(item["node"], item["file"], "wait", item["point_row"])
point = _point_from_episode(item, rows)
if point:
points.append(point)
# 一份会话里的 wall/path 切片可能指向同一个截断点;重放只保留一个。
point_map = {(p["node"], p["file"], p["point_row"], p["ptype"]): p for p in points}
eps.sort(key=lambda x: (int(x.get("mtime") or 0), x["id"]), reverse=True)
points = sorted(point_map.values(), key=lambda x: (int(x.get("mtime") or 0), x["id"]), reverse=True)
_write_jsonl(config.EPISODES, eps)
_write_jsonl(config.POINTS, points)
config.log(f"切片完成:{len(eps)} 段失败片段,{len(points)} 个可重放的决策点")
return {"episodes": len(eps), "points": len(points), "badFiles": bad_files}
def _point_from_episode(item: dict, rows: List[Optional[dict]]) -> Optional[dict]:
row = int(item.get("point_row") or 0)
if not item.get("has_env") or int(item.get("point_ctx") or 0) > config.MAX_POINT_CTX:
return None
if pending_bg(rows, row):
return None
return {
"id": _id(item["node"], item["file"], item["ptype"], row),
"episodeId": item["id"], "node": item["node"], "file": item["file"],
"mtime": item["mtime"], "stage": item["stage"], "platform": item.get("platform"), "ptype": item["ptype"],
"fam": item.get("fam") or "", "point_row": row, "point_call": item.get("point_call"),
"point_ctx": item.get("point_ctx") or 0,
}
# ── 问题模式的规范名 ─────────────────────────────────────────────────────────
#
# 审计给模式起名是自由发挥的:同一机制被起成不同名字,2026-10-02 那轮 40 段审计聚出 30 个模式,
# 几乎每个只有一段 —— 「重复撞墙」「Frida」这几类永远凑不够离线闸要的 15 个决策点。两道补救:
# 审计时把已有模式交给模型、要求复用;每轮审计后再做一次归并,写进别名表(旧名 → 规范名)。
ALIASES = "cluster_aliases.json"
def _aliases() -> Dict[str, str]:
try:
value = json.loads((config.DATA / ALIASES).read_text())
return value if isinstance(value, dict) else {}
except (OSError, json.JSONDecodeError):
return {}
def canon(name: Optional[str], aliases: Optional[Dict[str, str]] = None) -> str:
"""沿别名表走到规范名;有环就停在环上第一次重复之前。"""
table = _aliases() if aliases is None else aliases
cur, seen = str(name or ""), set()
while cur in table and cur not in seen:
seen.add(cur)
cur = str(table[cur])
return cur
def known_clusters(limit: int = 60) -> List[dict]:
"""已有的问题模式(按规范名合并),按出现次数排序 —— 审计时交给模型复用。"""
aliases = _aliases()
agg: Dict[str, dict] = {}
for a in _read_jsonl(config.AUDITS):
if a.get("classification") == "not_failure" or not a.get("cluster"):
continue
name = canon(a["cluster"], aliases)
slot = agg.setdefault(name, {"cluster": name, "titles": {}, "count": 0,
"classification": a.get("classification"), "replayType": a.get("replayType")})
slot["count"] += 1
t = str(a.get("clusterTitle") or "")
slot["titles"][t] = slot["titles"].get(t, 0) + 1
out = []
for slot in agg.values():
slot["clusterTitle"] = max(slot.pop("titles").items(), key=lambda kv: kv[1])[0]
out.append(slot)
return sorted(out, key=lambda x: -x["count"])[:limit]
def _known_block(known: List[dict]) -> str:
if not known:
return "(还没有已知模式,按机制新建一个 snake_case 名字)"
lines = ["| cluster | 中文名 | 类别 | 重放型 | 已出现 |", "|---|---|---|---|---|"]
lines += [f"| {k['cluster']} | {k['clusterTitle']} | {k['classification']} | {k['replayType']} | {k['count']} |" for k in known]
return "\n".join(lines)
def consolidate_clusters() -> dict:
"""把同一机制的不同名字归并到一个规范名,写进别名表。只合并重放型相同的;规范名必须是已有名字。"""
known = known_clusters(limit=200)
if len(known) < 2:
return {"clusters": len(known), "merged": 0}
prompt = f"""下面是 coding agent 执行失败审计出来的「问题模式」,名字是逐段审计时各自起的,同一机制常被起成不同名字。
请把**同一机制**的名字归并到其中一个**已有**名字(规范名)上:只合并重放型相同的;拿不准就不合并。
{_known_block(known)}
只返回 JSON:{{"merge": {{"<要并掉的名字>": "<规范名>", ...}}}}。没有要合并的就返回 {{"merge": {{}}}}。"""
try:
value = llm.ask_json(prompt, timeout=900)
except Exception as exc: # noqa: BLE001
config.log(f"归并问题模式失败(本轮跳过):{exc}")
return {"clusters": len(known), "merged": 0, "error": str(exc)[:200]}
merge = value.get("merge") if isinstance(value, dict) else None
by_name = {k["cluster"]: k for k in known}
aliases = _aliases()
merged = 0
for old, new in (merge or {}).items():
old, new = str(old), str(new)
if old == new or old not in by_name or new not in by_name:
continue # 规范名必须是已有名字;编出来的不收
if by_name[old].get("replayType") != by_name[new].get("replayType"):
continue # 重放型不同,判分口径不同,不能并
if canon(new, {**aliases, old: new}) == old:
continue # 会成环
aliases[old] = new
merged += 1
if merged:
path = config.DATA / ALIASES
path.write_text(json.dumps(aliases, ensure_ascii=False, indent=2))
os.chmod(path, 0o600)
config.log(f"归并问题模式:{len(known)} 个里并掉 {merged} 个")
return {"clusters": len(known), "merged": merged}
# ── 审计 ─────────────────────────────────────────────────────────────────────
ALLOWED_CLASSES = {"dead_end", "decision_error", "context_missing", "not_failure"}
ALLOWED_REPLAY_TYPES = {"T1_repeat", "T2_wait", "T3_frida", "none"}
def _audit_prompt(ep: dict, timeline: str, known: Optional[List[dict]] = None) -> str:
rubric = RUBRIC.read_text()
return f"""你在审计 coding agent 的一段历史执行记录。只做分类,不给修复建议。
{rubric}
已有的问题模式(**同一机制必须复用这里的 cluster 名字**;只有确实是新机制才新建):
{_known_block(known or [])}
片段元数据:
- stage: {ep['stage']}
- 自动切片标签: {ep['label']}
- 重复失败次数: {ep['n_fail']}
- 调用范围: #{ep['first_call']}..#{ep['last_call']}
时间线:
{timeline}
只返回一个 JSON 对象:
{{
"classification": "dead_end|decision_error|context_missing|not_failure",
"cluster": "稳定、简短的 snake_case 问题模式",
"clusterTitle": "中文模式名",
"replayType": "T1_repeat|T2_wait|T3_frida|none",
"correctInfoStep": 数字或 null,
"evidence": ["逐字引用时间线中的原文,至少一条"],
"rationale": "两三句,说明分类和正确的信息在何时已出现"
}}
不得把账号、token、工单正文复述进 cluster/rationale。evidence 必须逐字来自上面的时间线。"""
def _valid_audit(value: Any, timeline: str) -> Optional[dict]:
if not isinstance(value, dict):
return None
cls = value.get("classification")
replay_type = value.get("replayType")
cluster = str(value.get("cluster") or "")[:120]
title = str(value.get("clusterTitle") or "")[:160]
evidence = [str(x)[:500] for x in value.get("evidence") or [] if isinstance(x, str) and str(x).strip()]
if cls not in ALLOWED_CLASSES or replay_type not in ALLOWED_REPLAY_TYPES or not cluster or not title:
return None
# 结论必须能回指原文;不接受模型编出来的“引文”。
if not evidence or not any(q in timeline for q in evidence):
return None
step = value.get("correctInfoStep")
if step is not None and not isinstance(step, int):
step = None
return {
"classification": cls, "cluster": cluster, "clusterTitle": title,
"replayType": replay_type, "correctInfoStep": step,
"evidence": evidence[:3], "rationale": str(value.get("rationale") or "")[:1200],
}
def audit_new(max_items: Optional[int] = None) -> dict:
episodes_by_id = {e["id"]: e for e in _read_jsonl(config.EPISODES)}
existing = {a["episodeId"]: a for a in _read_jsonl(config.AUDITS) if a.get("episodeId")}
todo = [e for e in episodes_by_id.values() if e["id"] not in existing]
todo.sort(key=lambda e: (int(e.get("n_fail") or 0), int(e.get("mtime") or 0)), reverse=True)
todo = todo[: max_items if max_items is not None else config.MAX_AUDITS_PER_RUN]
ok = 0
errors: List[str] = []
config.log(f"审计:本轮 {len(todo)} 段(每段调一次 claude,约 1~2 分钟)")
known = known_clusters()
for k, ep in enumerate(todo, 1):
config.log(f" 审计 {k}/{len(todo)}:{ep['label'][:50]}")
path = _local_file(str(ep["node"]), str(ep["file"]))
rows = load_rows(path)
timeline = render_timeline(rows, int(ep["first_call"]), int(ep["last_call"]))
try:
value = _valid_audit(llm.ask_json(_audit_prompt(ep, timeline, known)), timeline)
if not value:
raise RuntimeError("审计结果缺字段、类别越界或证据不是原文")
existing[ep["id"]] = {
"episodeId": ep["id"], "auditedAt": int(time.time() * 1000),
"stage": ep["stage"], "node": ep["node"], "file": ep["file"],
"nFail": ep["n_fail"], **value,
}
ok += 1
if not any(k["cluster"] == canon(value["cluster"]) for k in known): # 新模式,后面的审计也要看得到
known.append({"cluster": canon(value["cluster"]), "clusterTitle": value["clusterTitle"], "count": 1,
"classification": value["classification"], "replayType": value["replayType"]})
# 每一条成功后就落盘;中途 OAuth/网络失败不会丢掉前面的昂贵结果。
_write_jsonl(config.AUDITS, sorted(existing.values(), key=lambda x: int(x.get("auditedAt") or 0)))
except Exception as exc: # noqa: BLE001
errors.append(f"{ep['id']}: {exc}")
return {"requested": len(todo), "audited": ok, "errors": errors}
# ── 聚类与重放点选择 ─────────────────────────────────────────────────────────
def clusters() -> List[dict]:
episode_map = {e["id"]: e for e in _read_jsonl(config.EPISODES)}
grouped: Dict[tuple, dict] = {}
aliases = _aliases()
titles = {k["cluster"]: k["clusterTitle"] for k in known_clusters(limit=1000)}
for audit in _read_jsonl(config.AUDITS):
ep = episode_map.get(audit.get("episodeId"))
if not ep or audit.get("classification") == "not_failure":
continue
platform = ep.get("platform") or platform_of(str(ep.get("node")))
name = canon(audit.get("cluster"), aliases)
key = (audit.get("stage"), platform, audit.get("classification"), name, audit.get("replayType"))
slot = grouped.setdefault(key, {
"stage": key[0], "platform": platform, "problemClass": key[2], "cluster": key[3],
"clusterTitle": titles.get(name) or audit.get("clusterTitle"), "replayType": key[4],
"episodes": 0, "wasteCalls": 0, "transcripts": set(), "samples": [],
})
slot["episodes"] += 1
slot["wasteCalls"] += int(ep.get("n_fail") or 0)
slot["transcripts"].add((ep.get("node"), ep.get("file")))
if len(slot["samples"]) < 4:
slot["samples"].append({
"episodeId": ep["id"], "nFail": ep.get("n_fail"),
"evidence": audit.get("evidence") or [], "rationale": audit.get("rationale") or "",
})
out = []
for slot in grouped.values():
slot["transcripts"] = len(slot["transcripts"])
out.append(slot)
return sorted(out, key=lambda x: (x["wasteCalls"], x["episodes"]), reverse=True)
def replay_points(cluster: Optional[dict] = None) -> List[dict]:
points = _read_jsonl(config.POINTS)
if not cluster:
return points
stage = cluster["stage"]
platform = cluster.get("platform")
ptype = cluster["replayType"]
points = [p for p in points if p.get("stage") == stage and (not platform or p.get("platform") == platform)]
if ptype == "T2_wait":
return [p for p in points if p.get("ptype") == "T2_wait"]
aliases = _aliases()
audited_eps = {
a["episodeId"] for a in _read_jsonl(config.AUDITS)
if a.get("stage") == stage and canon(a.get("cluster"), aliases) == cluster.get("cluster")
and a.get("classification") == cluster.get("problemClass")
}
return [p for p in points if p.get("episodeId") in audited_eps and p.get("ptype") == ptype]
def real_episode_ids(stage: str) -> set:
"""这个 stage 下审计过、且不是切片误报的片段 —— 保留集只用它们(自动切片约 25% 误报)。"""
return {
a["episodeId"] for a in _read_jsonl(config.AUDITS)
if a.get("stage") == stage and a.get("episodeId") and a.get("classification") != "not_failure"
}
def load_rows_for_point(point: dict) -> List[Optional[dict]]:
return load_rows(_local_file(str(point["node"]), str(point["file"])))
+209
View File
@@ -0,0 +1,209 @@
"""把排名最高的问题模式变成一个「只插入新行」的 prompt 候选。
模型只决定写什么;插入、占位符一致和长度限制都由本模块确定性复核。控制面提交时还会
用 shared/src/rsi_loop.ts 再复核一遍。
"""
import json
import re
from typing import Any, List, Optional
from . import llm
STAGES = {"bug.run", "feature.run.first", "feature.run.second"}
# 业务指标(真值来自证据 manifest / 终态)优先;过程指标是机制层面的浪费。
# 口径与生产基线见 shared/src/rsi_loop.ts::LOOP_METRICS。
BUSINESS_METRICS = {"reach_s", "reach_miss", "evidence_fail", "dead_end_s"}
TARGET_METRICS = BUSINESS_METRICS | {
"ui_wait_s", "blind_sleep_s", "failed_calls",
"repeat_fail_calls", "frida_fail_calls", "tool_calls",
}
# 每种重放型能登记哪些目标:离线判的是动作机制,真机复跑判登记的指标,两者要是同一条因果链。
# 等 App 就绪 / 撞墙 / Frida 乱注入都发生在走到复现现场之前或认死路之前,所以都能挂到达和走不通时长;
# 证据校验失败只跟「撞同一堵墙」(重拍)一类有关。
METRICS_BY_REPLAY = {
"T1_repeat": {"repeat_fail_calls", "reach_s", "reach_miss", "dead_end_s", "evidence_fail"},
"T2_wait": {"ui_wait_s", "blind_sleep_s", "reach_s", "reach_miss"},
"T3_frida": {"frida_fail_calls", "reach_s", "dead_end_s"},
}
EVIDENCE_TERMS = re.compile(r"record-evidence|manifest|evidence/|\bbefore-|\bafter-|证据|截图|录屏|拍照|shot\b", re.I)
def gaming_error(metric: str, base: str, candidate: str) -> Optional[str]:
"""与 shared/src/rsi_loop.ts::lintMetricGaming 同一条规则:到达类目标不许改证据采集。"""
if metric not in ("reach_s", "reach_miss"):
return None
added = inserted_text(base, candidate).split("\n")
hit = next((line for line in added if EVIDENCE_TERMS.search(line)), None)
return f"目标是到达复现现场,新增内容却在改证据采集:{hit.strip()[:60]}" if hit else None
def placeholders(template: str) -> List[str]:
return sorted(set(re.findall(r"\{\{\s*([\w.]+)\s*\}\}", template)))
def candidate_error(base: str, candidate: str) -> Optional[str]:
if placeholders(base) != placeholders(candidate):
return "占位符不一致"
if len(candidate) < len(base):
return "候选比现行模板短"
if len(candidate) - len(base) > 4000:
return "新增超过 4000 字"
base_lines = base.split("\n")
j = 0
for line in candidate.split("\n"):
if j < len(base_lines) and line == base_lines[j]:
j += 1
return None if j == len(base_lines) else "候选改动了已有行"
def apply_insertion(base: str, content: str, after_line: Optional[str]) -> str:
addition = content.strip("\n")
if not addition:
raise ValueError("插入内容为空")
lines = base.split("\n")
if after_line:
matches = [i for i, line in enumerate(lines) if line == after_line]
if len(matches) != 1:
raise ValueError("afterLine 必须逐字匹配现行模板中唯一的一行")
at = matches[0] + 1
lines[at:at] = addition.split("\n")
else:
lines.extend(addition.split("\n"))
return "\n".join(lines)
def inserted_text(base: str, candidate: str) -> str:
"""提取候选新增的行;base 不是 candidate 子序列时拒绝。"""
base_lines = base.split("\n")
added: List[str] = []
j = 0
for line in candidate.split("\n"):
if j < len(base_lines) and line == base_lines[j]:
j += 1
else:
added.append(line)
if j != len(base_lines):
raise ValueError("候选不是纯插入")
return "\n".join(added).strip()
def _prompt(cluster: dict, base: str) -> str:
# 提案只需要审计结论,不需要把含工单正文/账号的逐字证据再喂一遍,更不能让它进入提交字段。
safe_samples = [
{"nFail": s.get("nFail"), "rationale": _redact(str(s.get("rationale") or ""))}
for s in (cluster.get("samples") or [])
]
samples = json.dumps(safe_samples, ensure_ascii=False, indent=2)
return f"""你负责给 coding agent 的执行 prompt 提一个小而可归因的改动。
只能针对下面这个已审计的问题模式;一次只插入一段新纪律,不得改写、删除现有行,
不得增加或删除 `{{{{占位符}}}}`。新增不超过 4000 字。
stage: {cluster['stage']}
平台: {cluster.get('platform')}(问题来自这个平台的执行记录。下面的现行模板是按这个平台展开后的版本)
策略范围(scope)二选一:
- "platform":单端策略。规则只对这个平台成立(涉及该平台专有的工具、页面形态、系统行为)。
只在这个平台的实验机复跑,通过后也只上这个平台,另一个平台的模板不变。
- "both":双端策略。规则对两端都成立(与平台无关的执行纪律)。两端实验机各复跑一遍,
两端都通过才两端一起上;只有一端通过就只上那一端。新增内容里出现单端专有的工具词
(adb、catuse、xcrun 等)会被控制面按单端处理。
拿不准就选 "platform" —— 单端通过后,人可以再决定补另一端复跑。
问题类别: {cluster['problemClass']}
模式: {cluster['cluster']} / {cluster.get('clusterTitle')}
浪费: {cluster['wasteCalls']} 次失败调用,{cluster['episodes']} 个片段,
{cluster['transcripts']} 份会话
可引用的审计样本:
{samples}
可事先登记的目标指标(越低越好)。优先登记业务指标 —— 它们是用户真正关心的结果,
真值来自证据清单和任务终态,不来自 agent 自述;过程指标只在该模式与任何业务结果都
没有可信因果链时才登记。
业务指标(2026-10 生产基线,160 批):
- reach_s: 到达复现现场用时 —— 从这批开始到第一张校验通过的修复前证据。中位 Android 30 分、
iOS 44 分,约占整批一半。
- reach_miss: 未到达复现现场 —— 一批结束仍没有合格的修复前证据。修好的批次里 Android 17%、
iOS 14%;复现不了的 12 批里 10 批没到达。
- evidence_fail: 证据校验失败次数(录空 / 太短 / 分辨率不够,需重拍)。每批 Android 0.6、iOS 1.0 次。
- dead_end_s: 走不通时耗掉的时长 —— 没以 FIXED 收尾的批次的整批时长(早期失败识别)。
复跑会混入历史上走不通的单,「修好的个数不少于现行」是底线,所以只能靠更早认出死路,不能靠早放弃。
过程指标:
- ui_wait_s: 等界面状态时长
- blind_sleep_s: 盲等时长
- failed_calls: 失败调用数
- repeat_fail_calls: 同类命令连续失败
- frida_fail_calls: Frida 失败调用
- tool_calls: 工具调用总数
本模式的重放型是 {cluster.get('replayType')},targetMetric 必须从
{sorted(METRICS_BY_REPLAY.get(cluster.get('replayType'), TARGET_METRICS))} 中选,
并在 hypothesis 里写清这个模式怎样因果地影响所选指标(离线重放判的是动作机制,
真机复跑判的是所选指标,两者要是同一条因果链)。
若选 reach_s / reach_miss:新增内容不得涉及证据采集(截图、录屏、record-evidence、
before/after 命名)—— 到达是按第一张合格修复前证据量的,改快门时机会让指标变好而现场没变快,
这种候选会被直接拒收。
现行模板(只能选其中唯一一行作为 afterLine;若放结尾则 afterLine=null):
---BASE---
{base}
---END BASE---
只返回 JSON:
{{
"title": "实验标题",
"summary": "问题总结,引用聚合数字,不泄露账号或工单正文",
"hypothesis": "为什么这一个改动会降低目标指标",
"method": "插入什么纪律、放在哪里、真机复跑时看什么",
"targetMetric": "上面列表中允许的一个",
"afterLine": "现行模板中逐字且唯一的一整行,或 null",
"content": "要插入的新行",
"scope": "platform 或 both",
"scopeReason": "一句话:为什么这条规则只对这个平台 / 对两端都成立"
}}
content 要写可执行决策规则(何时、先做什么、失败后怎么换路),不要写空泛口号,
不要修改评分规则、保留集、指标定义或要求模型自评成功。"""
def _redact(text: str) -> str:
text = re.sub(r"[\w.+-]+@[\w.-]+\.[A-Za-z]{2,}", "<email>", text)
text = re.sub(r"(?i)\b(password|passwd|token|cookie|authorization)\s*[:=]\s*\S+", r"\1=<redacted>", text)
text = re.sub(r"\b[0-9a-f]{24,}\b", "<id>", text, flags=re.I)
return text[:1200]
def create(cluster: dict, base: str) -> dict:
if cluster.get("stage") not in STAGES:
raise ValueError("不支持的 stage")
value: Any = llm.ask_json(_prompt(cluster, base), timeout=1200)
if not isinstance(value, dict):
raise ValueError("提案不是 JSON 对象")
required = ("title", "summary", "hypothesis", "method", "targetMetric", "content")
if any(not str(value.get(k) or "").strip() for k in required):
raise ValueError("提案缺少必填字段")
metric = str(value["targetMetric"])
if metric not in TARGET_METRICS:
raise ValueError("targetMetric 不在目录里")
allowed = METRICS_BY_REPLAY.get(cluster.get("replayType"), TARGET_METRICS)
if metric not in allowed:
raise ValueError("targetMetric 与该模式的离线动作判分不对应")
after = value.get("afterLine")
if after is not None and not isinstance(after, str):
raise ValueError("afterLine 格式错误")
candidate = apply_insertion(base, str(value["content"]), after)
error = candidate_error(base, candidate) or gaming_error(metric, base, candidate)
if error:
raise ValueError(error)
return {
"stage": cluster["stage"],
"platform": cluster.get("platform"),
"title": str(value["title"]).strip()[:200],
"summary": str(value["summary"]).strip()[:6000],
"hypothesis": str(value["hypothesis"]).strip()[:4000],
"method": str(value["method"]).strip()[:4000],
"targetMetric": metric,
"problemClass": cluster["problemClass"],
"cluster": cluster["cluster"],
"candidateTemplate": candidate,
# 策略范围:both = 两端各复跑;其余一律单端(控制面按 platform 处理)
"scope": "both" if str(value.get("scope") or "").strip() == "both" else cluster.get("platform"),
"scopeReason": str(value.get("scopeReason") or "").strip()[:1000],
# 重放按同一个锚点把同样的几行插进历史 prompt,与生产渲染的位置一致(不提交给控制面)
"_insertion": {"afterLine": after, "content": inserted_text(base, candidate)},
}
+317
View File
@@ -0,0 +1,317 @@
"""决策点重放:把一段历史会话截到某个工具结果之后,用本机 claude 续跑一步,
只记录模型「想做什么」(第一条 assistant 消息里的工具调用 / 文字),工具一律被
PreToolUse 钩子拦下,不在本机执行。
保真度的三件事(2026-09 实验里踩出来的):
- 不能截在工具调用处,否则本机会补一条「调用被中断」的假结果;只截在工具结果之后;
- 续跑会追加「环境 / 日期 / 工具列表都变了」的附件,模型第一反应是去查环境 —— 所以把
这些附件改写成本机值,节点的 CLAUDE.md / MEMORY.md 落到本机对应路径;
- 本机原样起节点的 actl 设备工具 MCP,只为让工具定义一致(调用全被钩子拦)。
"""
import copy
import json
import os
import re
import shutil
import subprocess
import tempfile
import time
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional
from . import config
HOME = Path.home()
# 续跑的工作目录:<临时目录>/sl-ws/<会话id>/ai_workspace。模型看得见这个路径 —— 带着
# rsi_loop 之类的字样,它第一反应就是去查「这是什么」;叫 ai_workspace 才和节点上一致。
_REPLAY_BASE = Path(tempfile.gettempdir()).resolve() / "sl-ws"
def _run_dirs(sid: str):
cwd = _REPLAY_BASE / sid / "ai_workspace"
proj = HOME / ".claude" / "projects" / re.sub(r"[^A-Za-z0-9]", "-", str(cwd))
return cwd, proj
def _claude_cmd(sid: str, model: str) -> List[str]:
cmd = [config.CLAUDE_BIN, "-p", "继续", "--resume", sid, "--output-format", "stream-json", "--verbose",
"--model", model, "--setting-sources", "project"]
mcp = _mcp_config()
if mcp:
cmd += ["--mcp-config", mcp]
return cmd + ["--strict-mcp-config", "--settings", BLOCK_ALL, "--append-system-prompt", AUTOMODE, "--max-turns", "1"]
AUTOMODE = ("本会话运行在无人值守的自动化流水线中,没有任何人类在观看或会作出回应。"
"绝对禁止使用 AskUserQuestion 工具或以提问等待用户回复的方式中断任务;"
"如必须在多个方案中取舍,直接采用你推荐的(或第一个可行的)方案并继续执行,事后在输出中说明即可。"
"如需计划确认(ExitPlanMode),直接按你的计划执行。任务必须自主推进到完成。")
BLOCK_ALL = json.dumps({"hooks": {"PreToolUse": [{"matcher": "*", "hooks": [
{"type": "command", "command": "echo 'replay: tool execution disabled' >&2; exit 2"}]}]}})
def _mcp_config() -> Optional[str]:
return json.dumps({"mcpServers": {"actl": {"type": "stdio", "command": "node", "args": [config.ACTL_INDEX]}}}) \
if Path(config.ACTL_INDEX).exists() else None
_LOCAL: Optional[dict] = None
def _probe(sample_rows: List[Optional[dict]]) -> dict:
"""拿一段真实执行会话的开头(截到第一个工具结果)在本机续跑一步,收集本机追加的
「和节点不一样」的附件 —— 它们就是中和时要换上的本机值。空会话探不出任何东西:
Claude Code 不认残缺的会话记录,也就不会追加附件。"""
cut = next((i for i, r in enumerate(sample_rows) if r and r.get("type") == "user"
and isinstance((r.get("message") or {}).get("content"), list)
and any(isinstance(b, dict) and b.get("type") == "tool_result" for b in r["message"]["content"])), None)
if cut is None:
raise RuntimeError("探测用的会话里没有工具结果,换一份会话")
sid = str(uuid.uuid4())
cwd, proj = _run_dirs(sid)
cwd.mkdir(parents=True, exist_ok=True)
proj.mkdir(parents=True, exist_ok=True)
prefix = []
for r in sample_rows[: cut + 1]:
if not r:
continue
r = copy.deepcopy(r)
r["sessionId"] = sid
if "cwd" in r:
r["cwd"] = str(cwd)
prefix.append(r)
path = proj / f"{sid}.jsonl"
path.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in prefix) + "\n")
env = config.child_env(str(cwd))
loc: dict = {}
try:
subprocess.run(_claude_cmd(sid, config.REPLAY_MODEL), cwd=str(cwd), env=env,
capture_output=True, text=True, timeout=600)
for line in path.read_text(errors="replace").splitlines()[len(prefix):]:
try:
r = json.loads(line)
except Exception: # noqa: BLE001
continue
a = r.get("attachment")
if r.get("type") != "attachment" or not a:
continue
t = a.get("type")
if t in ("environment", "session_context", "date", "credential_org", "remote_session_change", "auto_mode"):
loc[t] = {k: v for k, v in a.items() if k != "changed"}
elif t == "deferred_tools_delta":
loc["deferred_removed"] = a.get("removedNames", [])
elif t == "agent_listing_delta":
loc["agent_removed"] = a.get("removedTypes", [])
finally:
shutil.rmtree(proj, ignore_errors=True)
shutil.rmtree(cwd.parent, ignore_errors=True)
if "environment" not in loc:
raise RuntimeError("探测失败:本机续跑没有追加环境附件(claude 鉴权、模型名或会话格式有问题),不能做重放")
return loc
def _local_values(sample_rows: Optional[List[Optional[dict]]] = None) -> dict:
"""本机该用的附件值。每轮运行先用一份真实会话探测一次(prepare),之后并发重放复用。"""
global _LOCAL
if _LOCAL is not None:
return _LOCAL
if sample_rows is None:
raise RuntimeError("重放前必须先 prepare(sample_rows) 探测本机附件")
_LOCAL = _probe(sample_rows)
cache = config.DATA / "local_attachments.json"
cache.write_text(json.dumps(_LOCAL, ensure_ascii=False))
os.chmod(cache, 0o600)
return _LOCAL
class AnchorMissing(ValueError):
"""候选的插入位置在这份历史 prompt 里找不到(或不唯一):这个点不能公平地比较两个版本。"""
def _anchor_regex(line: str) -> "re.Pattern[str]":
# 锚点行可能带占位符;渲染后的历史 prompt 里那部分是具体值,按通配匹配。
parts = re.split(r"\{\{\s*[\w.]+\s*\}\}", line)
return re.compile("^" + ".*?".join(re.escape(p) for p in parts) + "$")
def insert_into_text(text: str, insertion: dict) -> str:
"""把候选新增的行插到与生产相同的位置:锚点行之后;锚点为 null 表示模板末尾。
不加任何标题 —— 生产上这段内容就是模板里普通的几行,离线时更醒目会高估候选。"""
content = str(insertion.get("content") or "").strip("\n")
if not content:
return text
lines = text.split("\n")
after = insertion.get("afterLine")
if after is None:
return "\n".join(lines + content.split("\n"))
rx = _anchor_regex(str(after))
hits = [i for i, line in enumerate(lines) if rx.match(line)]
if len(hits) != 1:
raise AnchorMissing(f"锚点在历史 prompt 里匹配到 {len(hits)} 处")
at = hits[0] + 1
lines[at:at] = content.split("\n")
return "\n".join(lines)
def _insert_into_content(content: Any, insertion: Optional[dict]) -> Any:
if not insertion:
return content
if isinstance(content, str):
return insert_into_text(content, insertion)
if isinstance(content, list):
out = copy.deepcopy(content)
for block in out:
if isinstance(block, dict) and block.get("type") == "text":
block["text"] = insert_into_text(str(block.get("text") or ""), insertion)
return out
raise AnchorMissing("历史会话第一条用户消息没有文本")
def candidate_applies(rows: List[Optional[dict]], insertion: Optional[dict]) -> bool:
"""这个决策点能否施加候选改动(锚点唯一可定位)。不能的点两组都不跑。"""
if not insertion:
return True
for r in rows:
if r and r.get("type") == "user" and not r.get("isMeta"):
try:
_insert_into_content((r.get("message") or {}).get("content"), insertion)
return True
except AnchorMissing:
return False
return False
def _neutralize(rows: List[Optional[dict]], sid: str, loc: dict, cwd: Path, proj: Path,
insertion: Optional[dict] = None):
files: Dict[str, str] = {}
out: List[dict] = []
prompt_patched = False
for r in rows:
if not r:
continue
r = copy.deepcopy(r)
r["sessionId"] = sid
if "cwd" in r:
r["cwd"] = str(cwd)
if r.get("type") == "user" and not r.get("isMeta") and not prompt_patched:
message = r.get("message") or {}
if "content" in message:
message["content"] = _insert_into_content(message["content"], insertion)
r["message"] = message
prompt_patched = True
a = r.get("attachment") if r.get("type") == "attachment" else None
if a:
t = a.get("type")
if t == "environment" and "environment" in loc:
# 探测时记下的是探测目录;每次重放用独立目录,不换掉的话续跑会追加一条
# 「工作目录变了」,模型第一反应就是去查环境。
a["snapshot"] = {**loc["environment"]["snapshot"], "workingDirectory": str(cwd)}
a.pop("changes", None)
elif t in ("session_context", "date", "remote_session_change", "auto_mode") and t in loc:
r["attachment"] = copy.deepcopy(loc[t])
elif t == "deferred_tools_delta":
drop = set(loc.get("deferred_removed") or [])
for k in ("addedNames", "readdedNames"):
if isinstance(a.get(k), list):
a[k] = [n for n in a[k] if n not in drop]
if isinstance(a.get("addedLines"), list):
a["addedLines"] = [l for l in a["addedLines"] if not any(l.strip().startswith(n) for n in drop)]
elif t == "agent_listing_delta":
drop = set(loc.get("agent_removed") or [])
if isinstance(a.get("addedTypes"), list):
a["addedTypes"] = [n for n in a["addedTypes"] if n not in drop]
if isinstance(a.get("addedLines"), list):
a["addedLines"] = [l for l in a["addedLines"] if not any(l.strip().lstrip("- ").startswith(n + ":") for n in drop)]
elif t == "instructions":
for fobj in a.get("files") or []:
p = fobj.get("path", "")
if p.endswith("/ai_workspace/CLAUDE.md") or re.search(r"ai_workspace(_\w+)?/CLAUDE\.md$", p):
np = str(cwd / "CLAUDE.md")
elif p.endswith("/memory/MEMORY.md"):
np = str(proj / "memory" / "MEMORY.md")
else:
continue
files[np] = fobj.get("content", "")
fobj["path"] = np
out.append(r)
have = {r["attachment"].get("type") for r in out if r.get("type") == "attachment" and r.get("attachment")}
for t in ("credential_org", "session_context", "remote_session_change"):
if t in loc and t not in have:
idx = next((k for k, r in enumerate(out) if r.get("type") == "attachment" and r["attachment"].get("type") == "date"), None)
if idx is not None:
tmpl = copy.deepcopy(out[idx])
tmpl["attachment"] = copy.deepcopy(loc[t])
tmpl["uuid"] = str(uuid.uuid4())
out.insert(idx + 1, tmpl)
return out, files
def prepare(sample_rows: List[Optional[dict]]) -> None:
"""串行探测一次本机附件(每轮重新探,不跨轮缓存:日期、工具列表会变),之后可安全并发重放。"""
global _LOCAL
_LOCAL = None
_local_values(sample_rows)
def run_point(rows: List[Optional[dict]], row_idx: int, insertion: Optional[dict] = None, timeout: int = 900) -> dict:
"""截到 rows[row_idx](含)之后续跑一步。返回 {first:[...], secs, error}。first 是模型想做的事。"""
sid = str(uuid.uuid4())
cwd, proj = _run_dirs(sid)
cwd.mkdir(parents=True, exist_ok=True)
prefix, files = _neutralize(rows[: row_idx + 1], sid, _local_values(), cwd, proj, insertion)
proj.mkdir(parents=True, exist_ok=True)
for p, c in files.items():
Path(p).parent.mkdir(parents=True, exist_ok=True)
Path(p).write_text(c)
(proj / f"{sid}.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in prefix) + "\n")
env = config.child_env(str(cwd))
cmd = _claude_cmd(sid, config.REPLAY_MODEL)
t0 = time.time()
first: Optional[List[dict]] = None
error: Optional[str] = None
try:
p = subprocess.run(cmd, cwd=str(cwd), env=env, capture_output=True, text=True, timeout=timeout)
first, error = first_turn(p.stdout.splitlines())
if not first and not error and p.returncode:
error = (p.stderr or f"exit {p.returncode}")[-500:]
except subprocess.TimeoutExpired:
error = "timeout"
except Exception as e: # noqa: BLE001
error = str(e)
finally:
shutil.rmtree(proj, ignore_errors=True)
shutil.rmtree(cwd.parent, ignore_errors=True)
return {"first": first, "secs": round(time.time() - t0, 1), "error": None if first else (error or "no output")}
def first_turn(lines: List[str]):
"""从 stream-json 输出里取第一轮的内容块(文字 + 工具调用),到第一个工具调用为止。
流式输出会把同一轮的「说明文字」和「工具调用」拆成几条消息(常见:先一句话、再调工具)。
读到文字就停,会把这一轮记成「没有动作」—— 判分里那等于坏选择,两组都被冤枉。"""
first: Optional[List[dict]] = None
for line in lines:
try:
r = json.loads(line)
except Exception: # noqa: BLE001
continue
if r.get("type") == "assistant":
acts = [b for b in (r.get("message") or {}).get("content") or []
if isinstance(b, dict) and b.get("type") in ("tool_use", "text")]
if acts:
first = (first or []) + acts
if any(b.get("type") == "tool_use" for b in acts):
return first, None
if r.get("type") == "result":
return first, (str(r.get("result") or r.get("subtype")) if r.get("is_error") else None)
return first, None
def next_action(rows: List[Optional[dict]], row_idx: int) -> Optional[dict]:
"""这个点之后历史上真实的下一步工具调用(用于对照)。"""
for r in rows[row_idx + 1:]:
if r and r.get("type") == "assistant":
for b in (r["message"].get("content") or []):
if isinstance(b, dict) and b.get("type") == "tool_use":
return {"name": b["name"], "input": b.get("input") or {}}
return None
+135
View File
@@ -0,0 +1,135 @@
"""复核意见挖掘:每晚给新的人工复核意见打标签,每周把编码习惯归纳成候选规范。
口径、门槛、为什么这么做见 shared/src/review_mining.ts。这里只做「调模型」那一半:
- 提示词正文由控制面拼好下发(/review-mining/pending、/review-mining/cluster-input),这里不改一个字;
- 模型的回答原样交回,key 校验、归属、统计、门槛都由控制面重算 —— 不信这里报的任何数;
- 能写的只有标签和 pending 候选,批准 / 驳回要管理员在 RSI 页上点。
单独跑(补标历史、或者想立刻归纳一次):
./run.sh review-mining --days 180 # 补标近 180 天,每次最多 400 条,多跑几次
./run.sh review-mining --cluster # 不管是不是周一,立刻归纳一次
"""
import json
import time
from datetime import datetime, timedelta, timezone
from typing import Optional
from . import api, config, llm
LABELER = "rsi-loop/review-label@1"
LAST_CLUSTER = config.DATA / "review_mining_last_cluster"
# 与 shared/src/review_mining.ts 的 MINING.clusterWeekday 一致(北京时间周一)
CLUSTER_WEEKDAY = 0 # Python 的 weekday():周一 = 0
BEIJING = timezone(timedelta(hours=8))
def _beijing_day(now_ms: int) -> datetime:
return datetime.fromtimestamp(now_ms / 1000, BEIJING)
def label_pending(days: Optional[int] = None, max_batches: Optional[int] = None) -> dict:
"""拉没标过的意见,按批调模型,逐批交回。一批失败不影响下一批(下次运行还会再拉到它)。"""
q = "/review-mining/pending" + (f"?days={int(days)}" if days else "")
data = api.get(q, timeout=300)
batches = data.get("batches") or []
if max_batches is not None:
batches = batches[:max_batches]
saved = dropped = failed = 0
for i, b in enumerate(batches):
try:
out = llm.ask_json(b["prompt"], timeout=900)
except Exception as exc: # noqa: BLE001
failed += 1
config.log(f"复核意见打标签 第 {i + 1}/{len(batches)} 批失败:{str(exc)[:200]}")
continue
labels = out.get("labels") if isinstance(out, dict) else None
if not isinstance(labels, list):
failed += 1
continue
res = api.post("/review-mining/labels", {"labels": labels, "labeler": LABELER}) or {}
saved += int(res.get("saved") or 0)
dropped += int(res.get("dropped") or 0)
return {"pending": int(data.get("total") or 0), "batches": len(batches), "saved": saved, "dropped": dropped, "failedBatches": failed}
def _cluster_due(now_ms: int) -> bool:
day = _beijing_day(now_ms)
if day.weekday() != CLUSTER_WEEKDAY:
return False
try:
return LAST_CLUSTER.read_text().strip() != day.strftime("%Y-%m-%d")
except FileNotFoundError:
return True
def cluster(now_ms: int, force: bool = False) -> Optional[dict]:
"""每周一次:拉近 90 天的编码习惯标签和拼好的提示词,交给模型归纳,候选交回控制面过门槛。"""
if not force and not _cluster_due(now_ms):
return None
inp = api.get("/review-mining/cluster-input", timeout=300)
result = {"rows": int(inp.get("rows") or 0), "proposed": 0, "saved": 0, "rejected": []}
if result["rows"]:
out = llm.ask_json(inp["prompt"], timeout=1800)
cands = out.get("candidates") if isinstance(out, dict) else None
if not isinstance(cands, list):
raise RuntimeError("归纳的回答里没有 candidates 数组")
result["proposed"] = len(cands)
res = api.post("/review-mining/candidates", {
"runId": f"loop-{_beijing_day(now_ms).strftime('%Y%m%d')}",
"window": inp.get("window"),
"candidates": cands,
}) or {}
result["saved"] = int(res.get("saved") or 0)
result["rejected"] = res.get("rejected") or []
if not config.DRY_RUN:
LAST_CLUSTER.write_text(_beijing_day(now_ms).strftime("%Y-%m-%d"))
return result
def run(now_ms: Optional[int] = None) -> dict:
"""每日回路里的一步。两半各自兜底:挖掘失败不能拖垮回路的实验部分。"""
now_ms = now_ms or int(time.time() * 1000)
out: dict = {}
try:
out["label"] = label_pending()
except Exception as exc: # noqa: BLE001
out["label"] = {"error": str(exc)[:300]}
try:
out["cluster"] = cluster(now_ms)
except Exception as exc: # noqa: BLE001
out["cluster"] = {"error": str(exc)[:300]}
return out
def summary_line(out: dict) -> str:
"""给回路时间线的一句话。"""
lab = out.get("label") or {}
parts = []
if lab.get("error"):
parts.append(f"打标签失败:{lab['error'][:80]}")
else:
parts.append(f"新标 {lab.get('saved', 0)} 条意见(待标 {lab.get('pending', 0)})")
cl = out.get("cluster")
if cl:
if cl.get("error"):
parts.append(f"归纳失败:{cl['error'][:80]}")
else:
parts.append(f"归纳出 {cl.get('saved', 0)} 条候选规范(模型提了 {cl.get('proposed', 0)},门槛拦下 {len(cl.get('rejected') or [])})")
return ";".join(parts)
def main(argv: Optional[list] = None) -> dict:
import argparse
p = argparse.ArgumentParser(prog="rsi_loop review-mining")
p.add_argument("--days", type=int, default=None, help="待标意见的回看天数(补历史时用)")
p.add_argument("--cluster", action="store_true", help="不管是不是周一,立刻归纳一次")
p.add_argument("--no-label", action="store_true", help="跳过打标签")
a = p.parse_args(argv)
now_ms = int(time.time() * 1000)
out: dict = {}
if not a.no_label:
out["label"] = label_pending(days=a.days)
out["cluster"] = cluster(now_ms, force=a.cluster)
if any(out.values()):
api.event("复核意见挖掘(手动)", json.dumps(out, ensure_ascii=False, indent=2), "note")
return out
+359
View File
@@ -0,0 +1,359 @@
"""RSI 每日回路:同步 → 切片 → 审计 → 提案 → 配对重放 → 提交 → 补结论。"""
import concurrent.futures
import fcntl
import json
import os
import random
import time
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, List, Optional
from . import api, config, llm, mine, proposal, replay, review_mining, runtime, scoring
TERMINAL = {"offline_failed", "lab_passed", "lab_failed", "declined", "stopped"}
def _write_jsonl(path: Path, rows: List[dict]) -> None:
with open(path, "w") as f:
for row in rows:
f.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
os.chmod(path, 0o600)
def _local_skips() -> dict:
try:
return json.loads((config.DATA / "skips.json").read_text())
except (OSError, ValueError):
return {}
def _skip_key(cluster: dict) -> str:
return f"{cluster.get('stage')}|{cluster.get('platform')}|{cluster.get('cluster')}"
def _record_skip(cluster: dict, reason: str, now_ms: int) -> None:
if config.DRY_RUN:
return # 试运行不留下任何会影响正式运行的状态(否则正式跑时这个模式会被冷却 7 天)
skips = _local_skips()
skips[_skip_key(cluster)] = {"at": now_ms, "reason": reason}
path = config.DATA / "skips.json"
path.write_text(json.dumps(skips, ensure_ascii=False, indent=2))
os.chmod(path, 0o600)
def _submittable(cluster: dict, state: dict) -> bool:
"""实验只在实验机上复跑历史任务:只有 bug.run、且该平台配了实验机才收(控制面同样会拒)。"""
return cluster.get("stage") == "bug.run" and cluster.get("platform") in (state.get("labNodes") or {})
def _eligible_cluster(all_clusters: List[dict], state: dict, now_ms: int,
point_count=lambda c: len(mine.replay_points(c))) -> Optional[dict]:
# 同一 stage 在同一平台上同时只能有一个进行中的实验;platform 为空的旧实验对所有平台生效
live = {(e.get("stage"), e.get("platform")) for e in state.get("live") or []}
cutoff = now_ms - config.CLUSTER_COOLDOWN_DAYS * 86_400_000
recent = {
(e.get("stage"), e.get("platform"), e.get("cluster"))
for e in state.get("recent") or []
if int(e.get("createdAt") or 0) >= cutoff
}
skips = _local_skips()
for cluster in all_clusters:
if cluster.get("replayType") == "none" or not _submittable(cluster, state):
continue
stage, platform = cluster.get("stage"), cluster.get("platform")
if (stage, platform) in live or (stage, None) in live:
continue
if (stage, platform, cluster.get("cluster")) in recent:
continue
skipped = skips.get(_skip_key(cluster)) or {}
if int(skipped.get("at") or 0) >= cutoff:
continue
# 决策点不够就不提案:注定过不了离线闸,只会白花额度、再占一次冷却期
if point_count(cluster) < config.OFFLINE_MIN_POINTS:
continue
return cluster
return None
def _sample_points(cluster: dict, insertion: Optional[dict] = None) -> tuple:
applies = lambda p: replay.candidate_applies(mine.load_rows_for_point(p), insertion) # noqa: E731
all_target = mine.replay_points(cluster)
target = [p for p in all_target if applies(p)]
target_ids = {p["id"] for p in all_target}
real_eps = mine.real_episode_ids(cluster["stage"])
holdout_all = [
p for p in mine.replay_points()
if p["id"] not in target_ids and p.get("stage") == cluster["stage"]
and (not cluster.get("platform") or p.get("platform") == cluster.get("platform"))
and (p.get("ptype") == "T2_wait" or (p.get("ptype") in ("T1_repeat", "T3_frida") and p.get("episodeId") in real_eps))
]
holdout = [p for p in holdout_all if applies(p)]
seed = int.from_bytes(str(cluster["cluster"]).encode()[:8].ljust(8, b"\0"), "little")
rng = random.Random(seed)
rng.shuffle(target)
rng.shuffle(holdout)
seeds = max(2, config.SEEDS)
capacity = max(0, config.MAX_REPLAYS_PER_RUN // (2 * seeds))
n_target = min(config.TARGET_POINTS, len(target), capacity)
remaining = max(0, capacity - n_target)
n_holdout = min(config.HOLDOUT_POINTS, len(holdout), remaining)
skipped = {"targetAnchorMissing": len(all_target) - len(target), "holdoutAnchorMissing": len(holdout_all) - len(holdout)}
return target[:n_target], holdout[:n_holdout], seeds, skipped
def _one_replay(point: dict, arm: str, seed: int, insertion: Optional[dict], rows: list) -> dict:
result = replay.run_point(
rows, int(point["point_row"]),
insertion=insertion if arm == "candidate" else None,
)
action = scoring.action_of(result)
cls = "no_action" if action is None else scoring.classify_action(point["ptype"], point, action)
return {
"pointId": point["id"], "episodeId": point.get("episodeId"),
"stage": point.get("stage"), "ptype": point.get("ptype"),
"arm": arm, "seed": seed, "classification": cls,
"action": action, "secs": result.get("secs"), "error": result.get("error"),
}
class ReplayAborted(RuntimeError):
"""重放没法给出可信结论(点不够 / 基础设施报错太多):不提交实验,只记运行日志。"""
MAX_ERROR_RATE = 0.2
def run_replays(cluster: dict, insertion: Optional[dict]) -> dict:
target, holdout, seeds, skipped = _sample_points(cluster, insertion)
if len(target) < config.OFFLINE_MIN_POINTS:
raise ReplayAborted(
f"候选的插入位置只在 {len(target)} 个目标决策点上能定位(需要 {config.OFFLINE_MIN_POINTS}),"
f"{skipped['targetAnchorMissing']} 个点锚点缺失")
tagged = [("target", p) for p in target] + [("holdout", p) for p in holdout]
rows_by_id: Dict[str, list] = {p["id"]: mine.load_rows_for_point(p) for _, p in tagged}
jobs = [
(group, point, arm, seed)
for group, point in tagged
for arm in ("base", "candidate")
for seed in range(seeds)
]
records: List[dict] = []
if jobs:
replay.prepare(rows_by_id[jobs[0][1]["id"]])
with concurrent.futures.ThreadPoolExecutor(max_workers=max(1, config.WORKERS)) as pool:
future_map = {
pool.submit(_one_replay, point, arm, seed, insertion, rows_by_id[point["id"]]):
(group, point["id"])
for group, point, arm, seed in jobs
}
config.log(f"重放:{len(target)} 个目标点 + {len(holdout)} 个保留集点,每点 2 组 × {seeds} 遍,共 {len(jobs)} 次")
for future in concurrent.futures.as_completed(future_map):
group, point_id = future_map[future]
try:
records.append({"group": group, **future.result()})
except Exception as exc: # noqa: BLE001
records.append({
"group": group, "pointId": point_id, "arm": "unknown", "seed": None,
"classification": "error", "error": str(exc)[:1000],
})
config.log(f" 重放 {len(records)}/{len(jobs)}")
run_id = datetime.now().strftime("%Y%m%d-%H%M%S")
run_dir = config.RUNS / run_id
run_dir.mkdir(parents=True, exist_ok=False)
run_dir.chmod(0o700)
_write_jsonl(run_dir / "results.jsonl", records)
errors = sum(1 for r in records if r.get("error"))
if records and errors / len(records) > MAX_ERROR_RATE:
raise ReplayAborted(f"重放报错 {errors}/{len(records)},超过 {int(MAX_ERROR_RATE * 100)}%,"
f"不能把基础设施故障当成候选无效;原始结果见 {run_dir}")
target_result = scoring.offline_summary(r for r in records if r.get("group") == "target")
holdout_result = scoring.offline_summary(r for r in records if r.get("group") == "holdout")
offline = {
"target": target_result,
"holdout": holdout_result,
"meta": {
"runId": run_id, "model": config.REPLAY_MODEL,
"machine": os.uname().nodename, "calls": len(records),
"errors": errors, **skipped,
"scoring": "first-action-shape; paired bootstrap over decision points",
"candidateApplication": "inserted at the same anchor line as in the template; points where the anchor is missing are excluded from both arms",
},
}
(run_dir / "offline.json").write_text(json.dumps(offline, ensure_ascii=False, indent=2))
os.chmod(run_dir / "offline.json", 0o600)
return offline
def _conclusion_prompt(exp: dict) -> str:
payload = {
"status": exp.get("status"), "title": exp.get("title"), "stage": exp.get("stage"),
"summary": exp.get("summary"), "hypothesis": exp.get("hypothesis"),
"method": exp.get("method"), "targetMetric": exp.get("targetMetric"),
"offlineResult": exp.get("offlineResult"), "onlineResult": exp.get("onlineResult"),
"decisionReason": exp.get("decisionReason"),
}
return f"""根据下面这条自动实验的确定性记录写实验结论。不得引入记录外的数字,
不得把模型意见当真值;明确说假设是否得到支持、离线与真机复跑各发生了什么、下一轮该保留
什么或避免什么。状态如果是离线未通过,就只下离线结论;lab_passed / lab_failed 是
真机复跑的结论(评估集历史任务新旧 prompt 各跑一遍,onlineResult 里是配对结果;
「自报修好」是 agent 自己说的,不能当成功证据)。两到四段中文纯文本。
{json.dumps(payload, ensure_ascii=False, indent=2)}"""
def write_missing_conclusions(experiments: List[dict]) -> int:
written = 0
for exp in experiments:
if exp.get("status") not in TERMINAL or exp.get("conclusion"):
continue
try:
body = llm.ask(_conclusion_prompt(exp), timeout=900)
posted = api.experiment_event(
str(exp["id"]), f"实验结论:{exp.get('title')}", body[:12000], "conclusion",
)
written += int(posted)
except Exception as exc: # noqa: BLE001
print(f"[rsi-loop] conclusion {exp.get('id')} failed: {exc}")
return written
def _finding_body(all_clusters: List[dict]) -> str:
if not all_clusters:
return "近 14 天没有审计出可聚合的问题模式。"
lines = ["近 14 天问题模式(按重复失败调用数排序):"]
for c in all_clusters[:10]:
lines.append(
f"- {c.get('clusterTitle')} (`{c.get('cluster')}`): {c.get('wasteCalls')} 次失败调用,"
f"{c.get('episodes')} 个片段 / {c.get('transcripts')} 份会话;"
f"{c.get('problemClass')},{c.get('stage')},重放型 {c.get('replayType')}"
)
return "\n".join(lines)
def run_once() -> dict:
config.ensure_dirs()
now_ms = int(time.time() * 1000)
config.log("开始:读取控制面状态" + ("(试运行,不写生产)" if config.DRY_RUN else ""))
state = api.get("/state")
experiments = (api.get("/experiments?limit=500").get("experiments") or [])
sync = mine.sync_transcripts(now_ms)
mined = mine.mine_local()
audited = mine.audit_new()
consolidated = mine.consolidate_clusters()
all_clusters = mine.clusters()
config.log(f"聚类:{len(all_clusters)} 个问题模式")
conclusions = write_missing_conclusions(experiments)
# 复核意见挖掘与实验互不相干:暂停回路只停实验,不停打标签 / 归纳;失败也只记一笔
mining = review_mining.run(now_ms)
api.event("复核意见挖掘", review_mining.summary_line(mining) + "\n\n" + json.dumps(mining, ensure_ascii=False, indent=2), "note")
api.event(
f"发现 {len(all_clusters)} 个问题模式",
_finding_body(all_clusters),
"finding",
)
result = {
"sync": sync, "mined": mined, "audited": audited, "consolidated": consolidated,
"clusters": len(all_clusters), "conclusions": conclusions,
"review_mining": mining,
"submitted": None,
}
if (state.get("pause") or {}).get("paused"):
api.event(
"每日回路完成(已暂停,未提交实验)",
json.dumps(result, ensure_ascii=False, indent=2),
)
return result
cluster = _eligible_cluster(all_clusters, state, now_ms)
busy = {e.get("platform") for e in state.get("live") or [] if e.get("stage") == "bug.run"}
if not cluster:
note = "没有可提交的 prompt 选题"
else:
note = _submit_prompt_experiment(cluster, now_ms, result)
sub = result.get("submitted") or {}
if sub.get("status") == "lab":
busy.add(cluster.get("platform"))
result["runtime"] = _submit_runtime_experiments(state, experiments, busy)
if result["runtime"]:
note += ";旋钮实验:" + "、".join(r.get("title") or "?" for r in result["runtime"])
api.event(f"每日回路完成({note})", json.dumps(result, ensure_ascii=False, indent=2))
return result
def _submit_prompt_experiment(cluster: dict, now_ms: int, result: dict) -> str:
"""写提案 → 离线重放 → 提交。返回给时间线的一句话。"""
title = cluster.get("clusterTitle") or cluster.get("cluster")
config.log(f"选题:{title}({cluster.get('platform')} · {cluster['stage']}),写提案中")
# 按平台展开的现行版:模板里可能有只给某一端的段({{#ios}}…{{/ios}}),候选要插在展开后的版本上
active = api.get(f"/prompt/{cluster['stage']}?platform={cluster.get('platform')}")
draft = proposal.create(cluster, str(active["template"]))
try:
offline = run_replays(cluster, draft.pop("_insertion"))
except ReplayAborted as exc:
_record_skip(cluster, str(exc), now_ms)
result["aborted"] = str(exc)
return f"未提交:{title}:{exc}"
draft["offline"] = offline
response = api.post("/experiments", draft, timeout=180)
exp = response.get("experiment") or {}
result["submitted"] = {
"id": exp.get("id"), "title": exp.get("title"),
"status": exp.get("status"), "gate": response.get("gate"),
}
if exp.get("status") in TERMINAL:
result["conclusions"] += write_missing_conclusions([exp])
return f"提交 {draft['title']}"
def _submit_runtime_experiments(state: dict, experiments: List[dict], busy: set) -> List[dict]:
"""空着的实验机(没有进行中的 bug.run 实验)各提一个还没试过的旋钮取值。"""
knobs = state.get("runtimeKnobs") or {}
if not knobs or (None in busy): # 不带平台的旧实验占着所有平台
return []
out: List[dict] = []
for platform in sorted(state.get("labNodes") or {}):
if platform in busy:
continue
pick = runtime.next_candidate(platform, knobs, experiments)
if not pick:
continue
draft = runtime.draft(platform, pick[0], pick[1], knobs)
config.log(f"旋钮实验:{draft['title']}")
try:
exp = (api.post("/experiments", draft, timeout=180).get("experiment") or {})
out.append({"id": exp.get("id"), "title": exp.get("title") or draft["title"], "status": exp.get("status")})
except Exception as exc: # noqa: BLE001 一个平台提交失败不影响另一个
out.append({"title": draft["title"], "error": str(exc)[:300]})
return out
def locked_run() -> dict:
config.ensure_dirs()
with open(config.LOCK, "w") as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise RuntimeError("已有一个 RSI 回路进程在运行") from None
lock.write(f"{os.getpid()}\n")
lock.flush()
try:
return run_once()
except Exception as exc:
api.event("每日回路失败", str(exc)[:20000])
raise
def daemon() -> None:
"""在图形桌面终端常驻;这样 claude 复用该登录会话的 OAuth。"""
while True:
now = datetime.now()
target = now.replace(hour=config.DAILY_HOUR, minute=0, second=0, microsecond=0)
if target <= now:
target += timedelta(days=1)
wait = max(1, (target - now).total_seconds())
print(f"[rsi-loop] next run: {target.isoformat(timespec='seconds')}", flush=True)
time.sleep(wait)
try:
print(json.dumps(locked_run(), ensure_ascii=False, indent=2), flush=True)
except Exception as exc: # noqa: BLE001
print(f"[rsi-loop] run failed: {exc}", flush=True)
+76
View File
@@ -0,0 +1,76 @@
"""旋钮实验的选题:当天没有够格的 prompt 选题时,让空着的实验机去试执行配置包里还没试过的旋钮取值。
旋钮目录由控制面 /state 的 runtimeKnobs 下发(shared/src/executor_profile.ts),这里不另抄一份。
一个(平台, 旋钮=取值)只试一次,不论结果:复跑通过的等人决定,没通过的换别的取值再说。
实验机现行组已经是这个取值的不提(config.LAB_BASELINE),两组一样等于白跑。
提案文字是确定性的模板,不调模型 —— 旋钮实验的「为什么值得试」就是目录里那句说明。
"""
import json
from typing import Dict, List, Optional, Tuple
from . import config
TARGET_METRIC = "reach_s"
def _profile_of(exp: dict) -> Dict[str, str]:
raw = exp.get("candidateProfile")
if not raw:
return {}
try:
obj = json.loads(raw) if isinstance(raw, str) else raw
except ValueError:
return {}
return {str(k): str(v) for k, v in obj.items()} if isinstance(obj, dict) else {}
def tried(experiments: List[dict]) -> set:
"""已经试过的 (平台, 旋钮, 取值)。只认单旋钮的配置包 —— 回路自己只提单旋钮。"""
out = set()
for e in experiments:
if e.get("kind") != "runtime":
continue
prof = _profile_of(e)
if len(prof) == 1:
(k, v), = prof.items()
out.add((e.get("platform"), k, v))
return out
def _ordered_values(knob: str, values: List[str]) -> List[str]:
if knob == "effort":
return [v for v in config.EFFORT_ORDER if v in values] + [v for v in values if v not in config.EFFORT_ORDER]
return list(values)
def next_candidate(platform: str, knobs: Dict[str, dict], experiments: List[dict]) -> Optional[Tuple[str, str]]:
done = tried(experiments)
base = config.LAB_BASELINE.get(platform) or {}
for knob, d in knobs.items():
only = d.get("platforms")
if only and platform not in only:
continue
for v in _ordered_values(knob, list(d.get("values") or [])):
if base.get(knob) == v or (platform, knob, v) in done:
continue
return knob, v
return None
def draft(platform: str, knob: str, value: str, knobs: Dict[str, dict]) -> dict:
d = knobs.get(knob) or {}
label = d.get("label") or knob
plat = "iOS" if platform == "ios" else "Android"
base = (config.LAB_BASELINE.get(platform) or {}).get(knob)
base_txt = f"实验机现行为 {base}" if base else "实验机现行为生产默认(不设)"
return {
"stage": "bug.run", "platform": platform, "targetMetric": TARGET_METRIC,
"title": f"{plat}:{label}调到 {value}",
"summary": f"{d.get('doc') or ''}\n\n当天没有够格的 prompt 选题,实验机空着;这一项是旋钮目录里 {plat} 上还没试过的取值。",
"hypothesis": f"把{label}从现状({base_txt})改为 {value},能缩短到达复现现场的用时;"
"风险由复跑的两条底线兜住:自报修好、到达复现现场的个数不少于现行。",
"method": f"运行时实验:两组同一版 prompt;候选组的执行配置包为 {knob}={value},其余不变。"
"在实验机上拿评估集的历史任务配对复跑,按到达复现现场用时判定。",
"candidateProfile": {knob: value},
"problemClass": None, "cluster": None,
}
+104
View File
@@ -0,0 +1,104 @@
"""三类决策点的「坏选择」判定 —— 和 shared/src/rsi_loop.ts 的指标口径同源,但这里判的是
重放出来的「下一步动作」,不是整批的聚合指标。
三类(2026-09 审计确认的真失败形态):
T2_wait 等 App 就绪:下一步该用 wait-for / 后台 / Monitor,不该手写 sleep / 轮询循环;
T1_repeat 同一堵墙已经失败过:下一步不该用同族命令原样再撞;
T3_frida Frida 首次报错后:下一步不该直接再注入,应先诊断(frida-server / 版本 / 进程 / 读代码)。
判定只看动作形态,封闭、可审计 —— 它是离线闸的真值,不能是「另一个模型觉得好不好」。
"""
import re
import random
from collections import defaultdict
from typing import Iterable, Optional
UI_PROBE = re.compile(r"dumpsys\s+(window|activity)|uiautomator|mCurrentFocus|ResumedActivity|\bpidof\b|\bactl\b")
def _is_long_poll(cmd: str) -> bool:
if re.search(r"\b(while|until|for)\b.*\bsleep\b", cmd, re.S):
return True
return any(int(x) >= 15 for x in re.findall(r"\bsleep\s+(\d+)", cmd))
def classify_action(ptype: str, point: dict, action: Optional[dict]) -> str:
"""返回 bad / good / other。point 至少含 'fam'(截断处那条失败命令的命令族)。"""
if action is None:
return "other"
name = action.get("name", "")
inp = action.get("input") or {}
cmd = str(inp.get("command", "")) if name == "Bash" else ""
if ptype == "T2_wait":
if "wait-for" in cmd:
return "good"
if name in ("Monitor",) or inp.get("run_in_background"):
return "good"
if name == "Bash" and _is_long_poll(cmd):
return "bad"
return "other"
if ptype == "T3_frida":
if name == "Bash" and re.search(r"\bfrida\b", cmd) and not re.search(r"frida-ps|frida-ls|--version|pgrep|pidof|getprop", cmd):
return "bad"
return "good"
if ptype == "T1_repeat":
from .transcript import family
now = family(name, cmd) if name == "Bash" else name
previous = str(point.get("fam") or "")
return "bad" if now and now == previous else "good"
return "other"
def action_of(replay_result: dict) -> Optional[dict]:
for b in replay_result.get("first") or []:
if b.get("type") == "tool_use":
return {"name": b["name"], "input": b.get("input") or {}}
return None
def _mean(values):
return sum(values) / len(values) if values else 0.0
def _quantile(values, q: float) -> float:
if not values:
return 0.0
ordered = sorted(values)
pos = (len(ordered) - 1) * q
lo = int(pos)
hi = min(len(ordered) - 1, lo + 1)
frac = pos - lo
return ordered[lo] * (1 - frac) + ordered[hi] * frac
def offline_summary(records: Iterable[dict], bootstrap_iters: int = 2000, seed: int = 7) -> dict:
"""按决策点配对计算 baseBad-candBad,并对「点」而不是单次 seed 做 bootstrap。"""
arms = defaultdict(lambda: {"base": [], "candidate": []})
seed_values = set()
for row in records:
arm = row.get("arm")
if arm not in ("base", "candidate") or not row.get("pointId"):
continue
# 续跑失败和没有动作都不算候选的成功;按坏选择处理,避免靠沉默刷过离线闸。
bad = 0.0 if row.get("classification") in ("good", "other") and not row.get("error") else 1.0
arms[row["pointId"]][arm].append(bad)
if row.get("seed") is not None:
seed_values.add(row["seed"])
paired = [v for v in arms.values() if v["base"] and v["candidate"]]
diffs = [_mean(v["base"]) - _mean(v["candidate"]) for v in paired]
base_values = [x for v in paired for x in v["base"]]
cand_values = [x for v in paired for x in v["candidate"]]
boot = []
if diffs:
rng = random.Random(seed)
for _ in range(bootstrap_iters):
boot.append(_mean([diffs[rng.randrange(len(diffs))] for _ in diffs]))
return {
"points": len(paired),
"seeds": len(seed_values),
"baseBad": _mean(base_values),
"candBad": _mean(cand_values),
"diffMean": _mean(diffs),
"ciLow": _quantile(boot, 0.025),
"ciHigh": _quantile(boot, 0.975),
}
+206
View File
@@ -0,0 +1,206 @@
"""原生会话记录的解析:工具调用序列、命令族、报错「墙」、后台任务、时间线渲染。
这些规则都是在 2026-09 的实验里一条条踩出来的(见 docs/rsi.md「自动实验回路」):
- 读取类命令(cat / grep / sed / Read)输出里的报错字样是源码或历史记录,不是失败 ——
不排除它们,切片误报率是 25%;
- 命令超时被自动转到后台(moved to the background)和主动 run_in_background 一样,
截断时还在跑的话,续跑会收到「任务没跑完」的通知,处境就变了 —— 这种点不能用;
- Monitor 的启动回执(you will be notified on each event)不是报错。
"""
import json
import re
from pathlib import Path
from typing import Any, Dict, List, Optional
ERR = re.compile(r"(^|\n)\s*(error|fatal|exception|traceback|failed)\b|command not found|No such file|Exit code [1-9]|"
r"exit status [1-9]|BUILD FAILED|Failed to (spawn|attach)|unable to", re.I)
WALLS = re.compile(r"Failed to (?:spawn|attach|enumerate|inject|load)[^\n]{0,80}|unable to (?:connect|find|access)[^\n]{0,60}|"
r"INSTALL_FAILED_\w+|device (?:offline|not found|unauthorized)|no devices/emulators found|Permission denied|"
r"Process (?:terminated|crashed)|java\.lang\.\w+(?:Exception|Error)|ReferenceError[^\n]{0,60}|TypeError[^\n]{0,60}|"
r"Connection refused|timed out|BUILD FAILED|Compilation error|error: [^\n]{0,80}", re.I)
READ_ONLY = re.compile(r"^(grep|rg|ls|cat|head|tail|find|test|\[|echo|which|command|pgrep|ps|wc|stat|file|du|diff|sed|awk|jq|"
r"sort|true|false|printf|date|tree|readlink|strings|nl|less|more)\b")
LAUNCH = re.compile(r"(?:running in background|moved to the background)[^\"]{0,40}?\(?ID[:\s]+([a-z0-9]{6,})", re.I)
DONE = [re.compile(r"<task-id>([a-z0-9]{6,})</task-id>[\s\S]{0,600}?<status>(?:completed|failed|stopped|killed)"),
re.compile(r"Successfully stopped task:\s*([a-z0-9]{6,})"),
re.compile(r'\\"task_id\\":\\"([a-z0-9]{6,})\\"[^}]{0,200}?\\"status\\":\\"(?:completed|failed|killed|stopped)')]
def load_rows(path: Path) -> List[Optional[dict]]:
rows: List[Optional[dict]] = []
with open(path, errors="replace") as f:
for line in f:
try:
rows.append(json.loads(line))
except Exception: # noqa: BLE001
rows.append(None)
return rows
def text_of(content: Any) -> str:
if isinstance(content, str):
return content
return "\n".join(b.get("text", "") for b in content or [] if isinstance(b, dict) and b.get("type") == "text")
def first_prompt(rows: List[Optional[dict]]) -> Optional[dict]:
"""第一条用户消息所在的行(执行 prompt 本身)。"""
for r in rows:
if r and r.get("type") == "user" and not r.get("isMeta"):
c = (r.get("message") or {}).get("content")
if isinstance(c, str) or (isinstance(c, list) and any(isinstance(b, dict) and b.get("type") == "text" for b in c)):
return r
return None
def classify_stage(prompt_text: str) -> Optional[str]:
head = prompt_text[:600]
if re.search(r"自主\s*Bug\s*修复|autonomous bug[- ]?fix", head, re.I):
return "bug.run"
if re.search(r"自主\s*Feature\s*实现|autonomous feature", head, re.I):
return "feature.run.second" if re.search(r"再实现|re-?implement", prompt_text[:1500], re.I) else "feature.run.first"
return None
def family(tool: str, cmd: str) -> str:
if tool != "Bash":
return tool
s = re.sub(r"^\s*(nohup|exec)\s+", "", cmd)
if re.search(r"\bfrida\b(?!-server)", s) and not re.search(r"frida-(ps|ls)", s):
return "frida"
if re.search(r"\bwait-for\b", s):
return "wait-for"
if re.search(r"python3?\s*(-c|<<)", s):
return "python-inline"
s = re.sub(r'"?\$\{?\w+\}?"?(?=\s+(shell|install|push|pull|logcat|-s)\b)', "adb", s)
s = re.sub(r"/[\w./@~+-]+/", "", s)
parts = [p.strip() for p in re.split(r"&&|;|\|\||\n", s) if p.strip()]
first = next((p for p in parts if not re.match(r"^(cd|export|set|source|sleep|echo)\b|^[A-Za-z_]+=", p)), parts[0] if parts else "")
toks = [t for t in first.split() if not re.match(r"^\w+=", t) and t not in ("timeout", "gtimeout", "sudo", "env")]
if toks and toks[0] == "adb":
rest, i = [], 1
while i < len(toks):
if re.match(r"^-[stHP]$", toks[i]):
i += 2
continue
if not toks[i].startswith("-"):
rest.append(toks[i])
i += 1
return " ".join(["adb"] + rest[:2])
if toks and toks[0] in ("bun", "node", "python3", "python", "bash", "sh") and len(toks) >= 2:
return toks[0] + " " + Path(toks[1].strip("\"'")).name
return " ".join(toks[:2])
def wall(txt: str) -> Optional[str]:
if re.search(r"you will be notified on each", txt, re.I):
return None
m = WALLS.search(txt[-6000:])
if not m:
return None
s = m.group(0)[:100]
s = re.sub(r"/[\w./@-]+", "<p>", s)
s = re.sub(r"\b[0-9a-f]{6,}\b", "<h>", s)
return re.sub(r"\d+", "#", s).strip()
def is_read_only(tool: str, cmd: str) -> bool:
if tool in ("Read", "Grep", "Glob", "LSP", "ToolSearch"):
return True
if tool != "Bash":
return False
first = re.split(r"&&|;|\|", re.sub(r"^\s*cd\s+\S+\s*&&\s*", "", cmd).strip())[0].strip()
return bool(READ_ONLY.match(first))
def calls(rows: List[Optional[dict]]) -> List[dict]:
"""按工具结果的顺序展开调用:序号 i、所在行、工具、命令、成败、墙、当时的上下文 token、时刻。"""
uses: Dict[str, tuple] = {}
out: List[dict] = []
ctx = 0
for ri, r in enumerate(rows):
if not r:
continue
m = r.get("message") or {}
c = m.get("content")
if r.get("type") == "assistant":
u = m.get("usage") or {}
if u:
ctx = u.get("input_tokens", 0) + u.get("cache_read_input_tokens", 0) + u.get("cache_creation_input_tokens", 0)
for b in c or []:
if isinstance(b, dict) and b.get("type") == "tool_use":
uses[b["id"]] = (b.get("name") or "?", b.get("input") or {}, r.get("timestamp"))
if r.get("type") == "user" and isinstance(c, list):
for b in c:
if not (isinstance(b, dict) and b.get("type") == "tool_result"):
continue
name, inp, t0 = uses.get(b.get("tool_use_id"), ("?", {}, None))
txt = text_of(b.get("content"))
cmd = str(inp.get("command", "")) if name == "Bash" else json.dumps(inp, ensure_ascii=False)
ro = is_read_only(name, cmd)
failed = bool(b.get("is_error")) or (name == "Bash" and not ro and bool(ERR.search(txt[-3000:])))
out.append(dict(i=len(out), row=ri, tool=name, cmd=cmd, fam=family(name, cmd), ok=not failed,
read_only=ro, wall=None if ro else wall(txt), ctx=ctx, t=r.get("timestamp"), t0=t0,
bg=bool(re.search(r"moved to the background|running in background", txt[:400])),
err=([l for l in txt.strip().splitlines() if l.strip()] or [""])[-1][:160] if failed else ""))
return out
def pending_bg(rows: List[Optional[dict]], upto_row: int) -> List[str]:
launched: List[str] = []
done = set()
for r in rows[: upto_row + 1]:
if not r or r.get("type") != "user":
continue
s = json.dumps((r.get("message") or {}).get("content"), ensure_ascii=False)
launched += [m.group(1) for m in LAUNCH.finditer(s)]
for pat in DONE:
done |= {m.group(1) for m in pat.finditer(s)}
return sorted({k for k in launched if k not in done})
def has_env_attachment(rows: List[Optional[dict]]) -> bool:
return any(r and r.get("type") == "attachment" and (r.get("attachment") or {}).get("type") == "environment" for r in rows[:300])
def clip(s: str, a: int, b: int) -> str:
s = s or ""
return s if len(s) <= a + b + 20 else s[:a] + f"\n…[省略 {len(s) - a - b} 字]…\n" + s[-b:]
def render_timeline(rows: List[Optional[dict]], start_call: int, end_call: int, before: int = 12, after: int = 6,
max_chars: int = 140_000) -> str:
"""审计用的时间线:片段前后各带几步上下文,调用编号 #N 与 calls() 的序号一致。"""
uses: Dict[str, tuple] = {}
out: List[str] = []
i = -1
lo, hi = start_call - before, end_call + after
pending_text: List[str] = []
for r in rows:
if not r:
continue
m = r.get("message") or {}
c = m.get("content")
if r.get("type") == "assistant" and isinstance(c, list):
for b in c:
if not isinstance(b, dict):
continue
if b.get("type") == "tool_use":
uses[b["id"]] = (b.get("name"), b.get("input") or {})
elif b.get("type") == "text" and b.get("text", "").strip():
pending_text.append(b["text"].strip())
if r.get("type") == "user" and isinstance(c, list):
for b in c:
if not (isinstance(b, dict) and b.get("type") == "tool_result"):
continue
i += 1
if lo <= i <= hi:
name, inp = uses.get(b.get("tool_use_id"), ("?", {}))
cmd = inp.get("command") if name == "Bash" else json.dumps(inp, ensure_ascii=False)
out += [f"💬 {clip(t, 400, 200)}" for t in pending_text]
mark = " ⟵片段内" if start_call <= i <= end_call else ""
out.append(f"### [#{i}] {name}{mark} {str(r.get('timestamp', ''))[11:19]}\n```\n{clip(str(cmd), 900, 300)}\n```\n"
f"↩︎ {'(is_error) ' if b.get('is_error') else ''}\n```\n{clip(text_of(b.get('content')), 700, 700)}\n```")
pending_text = []
s = "\n\n".join(out)
return s if len(s) <= max_chars else s[:max_chars] + "\n…[时间线过长,已截断]"
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")"
cmd="${1:-once}"
[ $# -gt 0 ] && shift
exec /usr/bin/python3 -m rsi_loop "$cmd" "$@"
+1
View File
@@ -0,0 +1 @@
"""Unit tests for the standalone RSI loop."""
+481
View File
@@ -0,0 +1,481 @@
import json
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from rsi_loop import config, episodes, llm, mine, proposal, replay, runner, runtime, scoring, transcript
from rsi_loop.runner import _eligible_cluster
LAB_NODES = {"android": "evaluator-android-01.shiro0.icu", "ios": "temp-ios-01"}
def assistant(tool_id, name, tool_input, tokens=1000):
return {
"type": "assistant",
"message": {
"usage": {"input_tokens": tokens},
"content": [{"type": "tool_use", "id": tool_id, "name": name, "input": tool_input}],
},
}
def result(tool_id, text, is_error=False, timestamp="2026-09-01T00:00:00Z"):
return {
"type": "user", "timestamp": timestamp,
"message": {"content": [{
"type": "tool_result", "tool_use_id": tool_id,
"content": text, "is_error": is_error,
}]},
}
class TranscriptTests(unittest.TestCase):
def test_read_only_output_does_not_become_failure(self):
rows = [
assistant("a", "Bash", {"command": "grep -R error src"}),
result("a", "error: source fixture\nExit code 1"),
]
call = transcript.calls(rows)[0]
self.assertTrue(call["read_only"])
self.assertTrue(call["ok"])
self.assertIsNone(call["wall"])
def test_repeated_failure_point_is_after_first_failure(self):
rows = [
assistant("a", "Bash", {"command": "adb install app.apk"}),
result("a", "INSTALL_FAILED_VERSION_DOWNGRADE", True),
assistant("b", "Bash", {"command": "adb install app.apk"}),
result("b", "INSTALL_FAILED_VERSION_DOWNGRADE", True),
]
found = episodes.find_episodes(rows)
self.assertEqual(len(found), 1)
episode = found[0]
self.assertEqual(episode["n_fail"], 2)
self.assertEqual(episode["point_call"], 0)
self.assertEqual(episode["point_row"], 1)
def test_wait_point_is_before_wait_command(self):
rows = [
assistant("a", "Bash", {"command": "adb shell input tap 1 2"}),
result("a", "ok"),
assistant("b", "Bash", {"command": "until adb shell dumpsys window; do sleep 5; done"}),
result("b", "done"),
]
point = episodes.harvest_wait_points(rows)[0]
self.assertEqual(point["point_call"], 0)
self.assertEqual(point["point_row"], 1)
class ProposalTests(unittest.TestCase):
def test_only_inserts_and_preserves_placeholders(self):
base = "A {{task}}\nB\nC {{round}}"
candidate = proposal.apply_insertion(base, "新增一\n新增二", "B")
self.assertEqual(candidate, "A {{task}}\nB\n新增一\n新增二\nC {{round}}")
self.assertIsNone(proposal.candidate_error(base, candidate))
self.assertEqual(proposal.inserted_text(base, candidate), "新增一\n新增二")
def test_rejects_changed_line_and_ambiguous_anchor(self):
base = "A\nB\nB"
with self.assertRaises(ValueError):
proposal.apply_insertion(base, "X", "B")
self.assertIsNotNone(proposal.candidate_error("A\nB", "A\nB changed"))
def test_reach_target_cannot_touch_evidence_capture(self):
base = "步骤 3\n复现问题。\n步骤 4"
shoot = proposal.apply_insertion(base, "- 启动 App 后立刻 record-evidence 拍一张 before-1-x.png", "复现问题。")
route = proposal.apply_insertion(base, "- 先按 TestSpec 的入口用 deeplink 直达目标页", "复现问题。")
self.assertIsNotNone(proposal.gaming_error("reach_s", base, shoot))
self.assertIsNotNone(proposal.gaming_error("reach_miss", base, shoot))
self.assertIsNone(proposal.gaming_error("reach_s", base, route))
self.assertIsNone(proposal.gaming_error("evidence_fail", base, shoot))
def test_business_metrics_are_registrable_for_every_replay_type(self):
for ptype, allowed in proposal.METRICS_BY_REPLAY.items():
self.assertTrue(allowed & proposal.BUSINESS_METRICS, ptype)
self.assertTrue(allowed <= proposal.TARGET_METRICS, ptype)
def test_nested_json_reply_is_parsed(self):
with mock.patch.object(llm, "ask", return_value='```json\n{"a":{"b":1}}\n```'):
self.assertEqual(llm.ask_json("x"), {"a": {"b": 1}})
def test_online_metric_must_match_offline_action_shape(self):
value = {
"title": "x", "summary": "x", "hypothesis": "x", "method": "x",
"targetMetric": "tool_calls", "afterLine": None, "content": "新增纪律",
}
cluster = {
"stage": "bug.run", "problemClass": "decision_error",
"cluster": "wait", "clusterTitle": "等待", "replayType": "T2_wait",
"wasteCalls": 3, "episodes": 1, "transcripts": 1, "samples": [],
}
with mock.patch.object(llm, "ask_json", return_value=value):
with self.assertRaisesRegex(ValueError, "不对应"):
proposal.create(cluster, "base")
class ScoringTests(unittest.TestCase):
def test_action_shape_rules(self):
self.assertEqual(scoring.classify_action(
"T2_wait", {}, {"name": "Bash", "input": {"command": "sleep 30"}},
), "bad")
self.assertEqual(scoring.classify_action(
"T2_wait", {}, {"name": "Bash", "input": {"command": "node wait-for.js"}},
), "good")
self.assertEqual(scoring.classify_action(
"T3_frida", {}, {"name": "Bash", "input": {"command": "frida -U -n app -l hook.js"}},
), "bad")
self.assertEqual(scoring.classify_action(
"T3_frida", {}, {"name": "Bash", "input": {"command": "frida --version"}},
), "good")
point = {"fam": "adb install app.apk"}
self.assertEqual(scoring.classify_action(
"T1_repeat", point, {"name": "Bash", "input": {"command": "adb install app.apk"}},
), "bad")
self.assertEqual(scoring.classify_action(
"T1_repeat", point, {"name": "Bash", "input": {"command": "adb shell getprop"}},
), "good")
def test_paired_bootstrap_is_deterministic_and_positive(self):
records = []
for point in range(20):
for seed in range(2):
records.append({"pointId": str(point), "arm": "base", "seed": seed, "classification": "bad"})
records.append({"pointId": str(point), "arm": "candidate", "seed": seed, "classification": "good"})
a = scoring.offline_summary(records)
b = scoring.offline_summary(records)
self.assertEqual(a, b)
self.assertEqual(a["points"], 20)
self.assertEqual(a["seeds"], 2)
self.assertGreater(a["ciLow"], 0)
class AuditAndSelectionTests(unittest.TestCase):
def test_audit_requires_verbatim_evidence(self):
raw = {
"classification": "decision_error", "cluster": "repeat_same_command",
"clusterTitle": "重复撞墙", "replayType": "T1_repeat",
"correctInfoStep": 3, "evidence": ["Exit code 1"], "rationale": "已出现错误",
}
self.assertIsNotNone(mine._valid_audit(raw, "tool result: Exit code 1"))
raw["evidence"] = ["模型编造的引文"]
self.assertIsNone(mine._valid_audit(raw, "tool result: Exit code 1"))
def test_live_stage_and_cooldown_are_skipped(self):
now = 2_000_000_000_000
clusters = [
{"stage": "bug.run", "platform": "android", "cluster": "a", "replayType": "T1_repeat"},
{"stage": "bug.run", "platform": "ios", "cluster": "b", "replayType": "T2_wait"},
{"stage": "bug.run", "platform": "ios", "cluster": "c", "replayType": "T3_frida"},
]
state = {
"labNodes": LAB_NODES,
"live": [{"stage": "bug.run", "platform": "android"}],
"recent": [{"stage": "bug.run", "platform": "ios", "cluster": "b", "createdAt": now}],
}
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "c")
def test_cluster_without_enough_points_is_skipped(self):
now = 2_000_000_000_000
clusters = [
{"stage": "bug.run", "platform": "android", "cluster": "few", "replayType": "T1_repeat"},
{"stage": "bug.run", "platform": "android", "cluster": "many", "replayType": "T1_repeat"},
]
counts = {"few": 5, "many": 40}
state = {"labNodes": LAB_NODES}
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
got = _eligible_cluster(clusters, state, now, point_count=lambda c: counts[c["cluster"]])
self.assertEqual(got["cluster"], "many")
runner._record_skip(clusters[1], "anchor missing", now)
self.assertIsNone(_eligible_cluster(clusters, state, now, point_count=lambda c: counts[c["cluster"]]))
class ReplayFidelityTests(unittest.TestCase):
PROMPT = "你是自主 Bug 修复 Agent。\n工作区在 /Users/fei/bug_workspace/123。\n## 等待\n用 wait-for。\n## 结尾"
def test_insert_at_anchor_with_placeholder_and_no_heading(self):
ins = {"afterLine": "工作区在 {{workspace}}。", "content": "新纪律一\n新纪律二"}
out = replay.insert_into_text(self.PROMPT, ins)
lines = out.split("\n")
self.assertEqual(lines[1], "工作区在 /Users/fei/bug_workspace/123。")
self.assertEqual(lines[2:4], ["新纪律一", "新纪律二"])
self.assertNotIn("候选", out)
def test_null_anchor_appends_at_end_and_missing_anchor_raises(self):
out = replay.insert_into_text(self.PROMPT, {"afterLine": None, "content": "末尾纪律"})
self.assertTrue(out.endswith("## 结尾\n末尾纪律"))
with self.assertRaises(replay.AnchorMissing):
replay.insert_into_text(self.PROMPT, {"afterLine": "不存在的一行", "content": "x"})
def test_candidate_applies_checks_first_prompt(self):
rows = [{"type": "user", "message": {"role": "user", "content": self.PROMPT}}]
self.assertTrue(replay.candidate_applies(rows, {"afterLine": "## 等待", "content": "x"}))
self.assertFalse(replay.candidate_applies(rows, {"afterLine": "## 不存在", "content": "x"}))
self.assertTrue(replay.candidate_applies(rows, None))
def test_environment_snapshot_uses_this_runs_cwd(self):
rows = [
{"type": "user", "message": {"role": "user", "content": self.PROMPT}},
{"type": "attachment", "attachment": {"type": "environment",
"snapshot": {"workingDirectory": "/Users/fei/ai_workspace"},
"changes": [{"field": "workingDirectory"}]}},
]
loc = {"environment": {"type": "environment", "snapshot": {"workingDirectory": "/probe/dir", "platform": "darwin"}}}
out, _ = replay._neutralize(rows, "sid", loc, Path("/runs/abc"), Path("/proj/abc"),
{"afterLine": "## 等待", "content": "新纪律"})
env = out[1]["attachment"]
self.assertEqual(env["snapshot"]["workingDirectory"], "/runs/abc")
self.assertNotIn("changes", env)
self.assertIn("## 等待\n新纪律\n用 wait-for。", out[0]["message"]["content"])
class ClusterConsolidationTests(unittest.TestCase):
def _audits(self, d):
rows = [
{"episodeId": "e1", "stage": "bug.run", "classification": "decision_error", "cluster": "app_ready_blind_wait", "clusterTitle": "盲等就绪", "replayType": "T2_wait"},
{"episodeId": "e2", "stage": "bug.run", "classification": "decision_error", "cluster": "fixed_sleep_before_ready", "clusterTitle": "定长 sleep 等就绪", "replayType": "T2_wait"},
{"episodeId": "e3", "stage": "bug.run", "classification": "decision_error", "cluster": "frida_retry", "clusterTitle": "Frida 原样重试", "replayType": "T3_frida"},
]
(Path(d) / "a.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows))
def test_canon_follows_chain_and_stops_on_cycle(self):
self.assertEqual(mine.canon("a", {"a": "b", "b": "c"}), "c")
self.assertIn(mine.canon("a", {"a": "b", "b": "a"}), ("a", "b"))
self.assertEqual(mine.canon("x", {}), "x")
def test_consolidation_validates_and_merges(self):
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
mock.patch.object(config, "AUDITS", Path(d) / "a.jsonl"):
self._audits(d)
reply = {"merge": {
"fixed_sleep_before_ready": "app_ready_blind_wait", # 同机制、同重放型 → 并
"frida_retry": "app_ready_blind_wait", # 重放型不同 → 拒
"app_ready_blind_wait": "made_up_name", # 规范名不存在 → 拒
}}
with mock.patch.object(llm, "ask_json", return_value=reply):
r = mine.consolidate_clusters()
self.assertEqual(r["merged"], 1)
names = {k["cluster"]: k["count"] for k in mine.known_clusters()}
self.assertEqual(names, {"app_ready_blind_wait": 2, "frida_retry": 1})
def test_audit_prompt_lists_known_clusters(self):
known = [{"cluster": "app_ready_blind_wait", "clusterTitle": "盲等就绪", "classification": "decision_error", "replayType": "T2_wait", "count": 5}]
p = mine._audit_prompt({"stage": "bug.run", "label": "wall:x", "n_fail": 2, "first_call": 1, "last_call": 3}, "时间线", known)
self.assertIn("app_ready_blind_wait", p)
self.assertIn("必须复用", p)
class LabModeTests(unittest.TestCase):
def test_lab_run_sessions_are_not_mined(self):
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
mock.patch.object(config, "EPISODES", Path(d) / "e.jsonl"), mock.patch.object(config, "POINTS", Path(d) / "p.jsonl"), \
mock.patch.object(mine, "transcript_files", return_value=[(Path(d) / "x.jsonl", {"workspaceKey": "rsi-lab-abc", "node": "n", "file": "x.jsonl"})]), \
mock.patch.object(mine, "load_rows", side_effect=AssertionError("复跑会话不该被读")):
r = mine.mine_local()
self.assertEqual(r["episodes"], 0)
def test_only_submits_bug_run_with_lab_node(self):
now = 2_000_000_000_000
clusters = [
{"stage": "feature.run.first", "platform": "android", "cluster": "f", "replayType": "T2_wait"},
{"stage": "bug.run", "platform": "ios", "cluster": "no_lab_node", "replayType": "T1_repeat"},
{"stage": "bug.run", "platform": "android", "cluster": "ok", "replayType": "T2_wait"},
]
state = {"labNodes": {"android": "evaluator-android-01.shiro0.icu"}}
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "ok")
# 没配实验机就一个都不提
self.assertIsNone(_eligible_cluster(clusters, {}, now, point_count=lambda c: 30))
def test_lab_verdicts_get_conclusions(self):
self.assertIn("lab_passed", runner.TERMINAL)
self.assertIn("lab_failed", runner.TERMINAL)
self.assertIn("declined", runner.TERMINAL)
class PlatformTests(unittest.TestCase):
def test_live_experiment_only_blocks_its_own_platform(self):
now = 2_000_000_000_000
clusters = [
{"stage": "bug.run", "platform": "android", "cluster": "a", "replayType": "T2_wait"},
{"stage": "bug.run", "platform": "ios", "cluster": "b", "replayType": "T2_wait"},
]
state = {"labNodes": LAB_NODES, "live": [{"stage": "bug.run", "platform": "android"}]}
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "b")
# 不带平台的旧实验对所有平台生效
legacy = {"labNodes": LAB_NODES, "live": [{"stage": "bug.run", "platform": None}]}
self.assertIsNone(_eligible_cluster(clusters, legacy, now, point_count=lambda c: 30))
def test_replay_points_stay_on_cluster_platform(self):
pts = [
{"id": "a", "stage": "bug.run", "platform": "android", "ptype": "T2_wait"},
{"id": "i", "stage": "bug.run", "platform": "ios", "ptype": "T2_wait"},
]
with mock.patch.object(mine, "_read_jsonl", return_value=pts):
got = mine.replay_points({"stage": "bug.run", "platform": "ios", "replayType": "T2_wait"})
self.assertEqual([p["id"] for p in got], ["i"])
def test_node_without_transcripts_is_not_an_error(self):
import urllib.error
def fake_get(path, timeout=60):
if path == "/nodes":
return {"nodes": [{"id": "on", "platform": "android"}, {"id": "off", "platform": "ios"}]}
if path.startswith("/nodes/off/"):
raise urllib.error.HTTPError(path, 404, "not found", {}, None)
return {"files": []}
with tempfile.TemporaryDirectory() as d, \
mock.patch.object(config, "DATA", Path(d)), mock.patch.object(config, "TX_DIR", Path(d) / "tx"), \
mock.patch.object(config, "RUNS", Path(d) / "runs"), mock.patch.object(config, "STATE", Path(d) / "state.json"), \
mock.patch.object(config, "NODES", []), mock.patch("rsi_loop.api.get", side_effect=fake_get):
stats = mine.sync_transcripts(2_000_000_000_000)
self.assertEqual(stats["nodes"], 1)
self.assertEqual(stats["notEnabled"], ["off"])
self.assertEqual(stats["errors"], [])
class SyncTests(unittest.TestCase):
def test_one_broken_download_does_not_stop_the_rest(self):
files = [{"file": f"{i:08d}-0000-0000-0000-000000000000.jsonl", "size": 10, "mtime": 2_000_000_000_000} for i in range(3)]
calls = {"n": 0}
def fake_download(path, dest):
calls["n"] += 1
if "00000001-" in path:
raise IOError("IncompleteRead(542248 bytes read)")
dest.write_text("{}")
return 10
with tempfile.TemporaryDirectory() as d, \
mock.patch.object(config, "DATA", Path(d)), mock.patch.object(config, "TX_DIR", Path(d) / "tx"), \
mock.patch.object(config, "RUNS", Path(d) / "runs"), mock.patch.object(config, "STATE", Path(d) / "state.json"), \
mock.patch.object(config, "NODES", ["n1"]), \
mock.patch("rsi_loop.api.get", return_value={"files": files}), \
mock.patch("rsi_loop.api.download", side_effect=fake_download), \
mock.patch("rsi_loop.mine.time.sleep"):
stats = mine.sync_transcripts(2_000_000_000_000)
self.assertEqual(stats["downloaded"], 2)
self.assertEqual(len(stats["errors"]), 1)
self.assertIn("IncompleteRead", stats["errors"][0])
class DryRunTests(unittest.TestCase):
def test_dry_run_post_never_touches_network(self):
from rsi_loop import api
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
mock.patch.object(config, "DRY_RUN", True), \
mock.patch("urllib.request.urlopen", side_effect=AssertionError("dry-run 不该联网")):
got = api.post("/experiments", {"title": "t", "stage": "bug.run"})
self.assertEqual(got["experiment"]["status"], "dry_run")
self.assertTrue(api.event("运行摘要"))
lines = (Path(d) / "dry_posts.jsonl").read_text().splitlines()
self.assertEqual([json.loads(l)["path"] for l in lines], ["/experiments", "/events"])
runner._record_skip({"stage": "bug.run", "cluster": "x"}, "anchor", 1)
self.assertFalse((Path(d) / "skips.json").exists())
class FirstTurnTests(unittest.TestCase):
def test_text_then_tool_use_is_one_turn(self):
lines = [
json.dumps({"type": "system", "subtype": "init"}),
json.dumps({"type": "assistant", "message": {"id": "m1", "content": [{"type": "text", "text": "先冷启动看登录态。"}]}}),
json.dumps({"type": "assistant", "message": {"id": "m1", "content": [
{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "adb shell monkey -p x 1"}}]}}),
json.dumps({"type": "result", "is_error": False}),
]
first, err = replay.first_turn(lines)
self.assertIsNone(err)
self.assertEqual(scoring.action_of({"first": first})["input"]["command"], "adb shell monkey -p x 1")
def test_error_result_is_reported(self):
first, err = replay.first_turn([json.dumps({"type": "result", "is_error": True, "result": "Failed to authenticate"})])
self.assertIsNone(first)
self.assertIn("authenticate", err)
class SamplingTests(unittest.TestCase):
def _points(self):
mk = lambda i, stage, ptype, ep: {"id": f"p{i}", "stage": stage, "ptype": ptype, "episodeId": ep, # noqa: E731
"node": "n", "file": f"f{i}", "point_row": 1}
target = [mk(i, "bug.run", "T1_repeat", f"t{i}") for i in range(20)]
others = [
mk(100, "bug.run", "T2_wait", None), # 同 stage 等待点:可进保留集
mk(101, "bug.run", "T1_repeat", "real"), # 同 stage 审计过的真失败:可进
mk(102, "bug.run", "T1_repeat", "unaudited"), # 没审计过(可能是误报):不进
mk(103, "feature.run.first", "T2_wait", None), # 其它 stage:不进
]
return target, others
def test_holdout_is_same_stage_and_audited_and_anchor_checked(self):
target, others = self._points()
cluster = {"stage": "bug.run", "cluster": "x", "replayType": "T1_repeat"}
with mock.patch.object(mine, "replay_points", side_effect=lambda c=None: target if c else target + others), \
mock.patch.object(mine, "real_episode_ids", return_value={"real"}), \
mock.patch.object(mine, "load_rows_for_point", side_effect=lambda p: p["id"]), \
mock.patch.object(replay, "candidate_applies", side_effect=lambda rows, ins: rows != "p3"):
t, h, seeds, skipped = runner._sample_points(cluster, {"afterLine": "a", "content": "b"})
self.assertEqual({p["id"] for p in h}, {"p100", "p101"})
self.assertNotIn("p3", {p["id"] for p in t})
self.assertEqual(skipped["targetAnchorMissing"], 1)
self.assertGreaterEqual(seeds, 2)
def test_high_error_rate_aborts_without_offline_result(self):
target, _ = self._points()
cluster = {"stage": "bug.run", "cluster": "x", "replayType": "T1_repeat"}
with tempfile.TemporaryDirectory() as d, \
mock.patch.object(config, "RUNS", Path(d)), \
mock.patch.object(mine, "replay_points", side_effect=lambda c=None: target), \
mock.patch.object(mine, "real_episode_ids", return_value=set()), \
mock.patch.object(mine, "load_rows_for_point", return_value=[]), \
mock.patch.object(replay, "candidate_applies", return_value=True), \
mock.patch.object(replay, "prepare"), \
mock.patch.object(replay, "run_point", return_value={"first": None, "secs": 0, "error": "Failed to authenticate"}):
with self.assertRaises(runner.ReplayAborted):
runner.run_replays(cluster, {"afterLine": None, "content": "x"})
class RuntimeProposalTests(unittest.TestCase):
KNOBS = {
"effort": {"label": "思考强度", "values": ["low", "medium", "high", "xhigh", "max"], "doc": "d"},
"ui_settle": {"label": "等界面稳定", "values": ["on"], "doc": "d", "platforms": ["ios"]},
}
def _exp(self, platform, profile):
return {"kind": "runtime", "platform": platform, "candidateProfile": json.dumps(profile)}
def test_skips_tried_baseline_and_off_platform(self):
exps = [self._exp("android", {"effort": "medium"}), self._exp("ios", {"ui_settle": "on"})]
with mock.patch.object(config, "LAB_BASELINE", {"android": {"effort": "xhigh"}}):
# medium 试过、xhigh 是实验机现行 → high;ui_settle 不适用 Android
self.assertEqual(runtime.next_candidate("android", self.KNOBS, exps), ("effort", "high"))
# iOS:effort 一个都没试过,从 medium 开始
self.assertEqual(runtime.next_candidate("ios", self.KNOBS, exps), ("effort", "medium"))
def test_exhausted_catalog_proposes_nothing(self):
exps = [self._exp("ios", {"effort": v}) for v in ["low", "medium", "high", "xhigh", "max"]]
exps.append(self._exp("ios", {"ui_settle": "on"}))
self.assertIsNone(runtime.next_candidate("ios", self.KNOBS, exps))
def test_free_lab_nodes_each_get_one(self):
state = {"runtimeKnobs": self.KNOBS, "labNodes": LAB_NODES}
posted = []
with mock.patch.object(runner.api, "post", side_effect=lambda path, body, timeout=120: posted.append(body) or {"experiment": {"id": "x", "title": body["title"], "status": "lab"}}), \
mock.patch.object(config, "LAB_BASELINE", {}):
got = runner._submit_runtime_experiments(state, [], busy={"android"})
self.assertEqual([b["platform"] for b in posted], ["ios"])
self.assertEqual(posted[0]["candidateProfile"], {"effort": "medium"})
self.assertEqual(got[0]["status"], "lab")
class IosWaitPointTests(unittest.TestCase):
def test_catuse_probe_counts_as_ui_wait(self):
self.assertTrue(episodes.UI_LIKE.search("sleep 20; catuse apps 2>&1 | head -20"))
self.assertTrue(episodes._is_wait("sleep 20; catuse apps 2>&1 | head -20"))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,82 @@
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from rsi_loop import config, review_mining
# 2026-11-02 是周一(北京时间),2026-11-03 是周二
MON = 1793548800000 + 3600_000 * 4 # 2026-11-02 04:00 北京
TUE = MON + 86_400_000
class LabelTest(unittest.TestCase):
def test_batches_posted_and_failures_isolated(self):
pending = {"total": 3, "batches": [{"keys": ["a"], "prompt": "p1"}, {"keys": ["b"], "prompt": "p2"}]}
answers = [RuntimeError("boom"), {"labels": [{"key": "b", "cat": "ops"}]}]
posted = []
def ask_json(prompt, timeout=0):
a = answers.pop(0)
if isinstance(a, Exception):
raise a
return a
with mock.patch.object(review_mining.api, "get", return_value=pending), \
mock.patch.object(review_mining.llm, "ask_json", side_effect=ask_json), \
mock.patch.object(review_mining.api, "post", side_effect=lambda path, body, **k: posted.append((path, body)) or {"saved": 1, "dropped": 0}):
out = review_mining.label_pending()
self.assertEqual(out["failedBatches"], 1)
self.assertEqual(out["saved"], 1)
self.assertEqual(posted[0][0], "/review-mining/labels")
self.assertEqual(posted[0][1]["labeler"], review_mining.LABELER)
def test_days_query(self):
with mock.patch.object(review_mining.api, "get", return_value={"batches": []}) as g:
review_mining.label_pending(days=180)
g.assert_called_with("/review-mining/pending?days=180", timeout=300)
class ClusterTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.marker = Path(self.tmp.name) / "last"
self.patch = mock.patch.object(review_mining, "LAST_CLUSTER", self.marker)
self.patch.start()
def tearDown(self):
self.patch.stop()
self.tmp.cleanup()
def test_only_on_monday_once(self):
self.assertFalse(review_mining._cluster_due(TUE))
self.assertTrue(review_mining._cluster_due(MON))
inp = {"rows": 2, "prompt": "p", "window": {"from": 1, "to": 2}}
with mock.patch.object(review_mining.api, "get", return_value=inp), \
mock.patch.object(review_mining.llm, "ask_json", return_value={"candidates": [{"family": "x"}]}), \
mock.patch.object(review_mining.api, "post", return_value={"saved": 1, "rejected": []}) as post, \
mock.patch.object(config, "DRY_RUN", False):
out = review_mining.cluster(MON)
self.assertEqual(out["saved"], 1)
self.assertEqual(post.call_args[0][1]["window"], {"from": 1, "to": 2})
self.assertFalse(review_mining._cluster_due(MON)) # 同一天不再跑
def test_force_and_empty_rows(self):
with mock.patch.object(review_mining.api, "get", return_value={"rows": 0, "window": {}}), \
mock.patch.object(review_mining.llm, "ask_json") as ask, \
mock.patch.object(config, "DRY_RUN", True):
out = review_mining.cluster(TUE, force=True)
ask.assert_not_called()
self.assertEqual(out["rows"], 0)
self.assertFalse(self.marker.exists()) # 试运行不落标记
def test_run_isolates_errors(self):
with mock.patch.object(review_mining, "label_pending", side_effect=RuntimeError("down")), \
mock.patch.object(review_mining, "cluster", return_value=None):
out = review_mining.run(TUE)
self.assertIn("error", out["label"])
self.assertIn("打标签失败", review_mining.summary_line(out))
if __name__ == "__main__":
unittest.main()