Initial import of streamline
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
# 失败片段审计规约
|
||||
|
||||
分类只回答「当时为什么浪费」,不评价最终代码,也不因为输出里出现 `error` 就判失败。
|
||||
|
||||
- `dead_end`:这条路在当前环境客观走不通,后续仍重复撞同一堵墙。典型是权限、设备、
|
||||
版本或工具能力不满足;换措辞不能让这条路变通。
|
||||
- `decision_error`:在已有信息足够的情况下选错了下一步,或没有在失败后切换策略。
|
||||
反复手写 sleep / 轮询等待 App、原样重复同族命令、Frida 报错后直接再次注入都属于此类。
|
||||
- `context_missing`:做对下一步所需的信息在当时上下文中还没有出现;不是 agent 忽略了
|
||||
已知信息。必须指出缺什么,不能把「没有想到」改写成「缺上下文」。
|
||||
- `not_failure`:切片误报。读取源码、日志或历史记录时看到的 `error/failed` 字样;
|
||||
探测命令预期返回非零;后台任务启动回执;或者片段没有形成真实的重复失败。
|
||||
|
||||
`correctInfoStep` 是时间线中的调用编号:从该步结果开始,做对下一步所需的信息已经可见。
|
||||
确实从未出现时填 `null`。证据必须逐字引用时间线,至少一条,并足以支持分类。
|
||||
|
||||
`replayType` 只按可封闭判分的动作形态选择:
|
||||
|
||||
- `T2_wait`:问题核心是等待 App / UI / 进程就绪;下一步可判为 wait-for / 后台监听,
|
||||
或手写长 sleep / 轮询。
|
||||
- `T1_repeat`:第一次失败后是否再次执行同族命令。
|
||||
- `T3_frida`:Frida 首次失败后是否直接再次注入,而不是先诊断 server、版本、进程或代码。
|
||||
- `none`:不属于以上三种,不能进入自动离线闸。
|
||||
|
||||
`cluster` 要描述可重复出现的机制,不写工单名、账号、节点名或具体 id,例如
|
||||
`app_ready_blind_wait`、`repeat_same_failed_command`、`frida_retry_without_diagnosis`。
|
||||
@@ -0,0 +1,99 @@
|
||||
# 自动 RSI 回路进程
|
||||
|
||||
每天执行一次:
|
||||
|
||||
1. 从 `/api/rsi-loop/nodes/:id/transcripts` 同步近 14 天原生执行会话;
|
||||
2. 切出同一堵墙 / 同族命令重复失败至少两次的片段(读取类命令不算失败),以及所有「要开始等界面 / App 就绪」
|
||||
的时刻(Android 认 dumpsys / uiautomator / pidof / actl,iOS 认 catuse);
|
||||
3. 最多审计 60 段,分为 `dead_end / decision_error / context_missing / not_failure`;
|
||||
4. 按浪费的失败调用数选一个问题模式,在现行 prompt 上只插入一段新纪律;
|
||||
5. 目标点和其它模式保留集分别做配对重放,每点 base / candidate 各跑三遍;
|
||||
6. 提交实验(离线未过也提交),并给已结束实验补 AI 结论和每日运行摘要;
|
||||
7. 还空着的实验机(该平台没有进行中的 bug.run 实验),各提一个**旋钮实验**:从 `/state` 的
|
||||
`runtimeKnobs` 目录里挑该平台还没试过的取值(每个取值只试一次,跳过实验机现行的取值,
|
||||
见 `config.LAB_BASELINE`),不走离线、直接真机复跑(`rsi_loop/runtime.py`)。
|
||||
8. **复核意见挖掘**(`rsi_loop/review_mining.py`,与实验互不相干,回路暂停时照跑):给新的人工复核意见打标签;
|
||||
每周一(北京时间)把近 90 天的编码习惯归纳成候选规范交给控制面过门槛,管理员在 RSI 页「复核意见挖掘」批准。
|
||||
提示词由控制面下发,这里不改一个字;首次上线先补历史:`./run.sh review-mining --days 180`(每次最多 400 条,
|
||||
多跑几次直到待标为 0),再 `./run.sh review-mining --cluster --no-label` 立刻归纳一次。见 `docs/rsi.md` 例子 14。
|
||||
|
||||
离线通过的实验由控制面转入**真机复跑**:评估集里跑过的历史任务在实验机上现行 / 候选各跑一遍
|
||||
(见 `docs/rsi.md` 例子 13),判定由控制面的判定任务执行,复跑结论就是实验终点,不自动改生产
|
||||
prompt。线上灰度已于 2026-10-05 下线(新任务之间差别太大,比不出改动的效果)。所以本进程只提
|
||||
`bug.run`、且该平台配了实验机的问题模式(从 `/state` 的 `labNodes` 读)。
|
||||
|
||||
## temp-ios-01
|
||||
|
||||
代码只依赖系统 Python 3.9 标准库和 Node 24(actl MCP 用)。目录默认是
|
||||
`~/rsi_loop`,令牌默认读 `~/rsi_loop/token`:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/rsi_loop
|
||||
chmod 700 ~/rsi_loop
|
||||
printf '%s' '<与控制面 RSI_LOOP_TOKEN 相同的值>' > ~/rsi_loop/token
|
||||
chmod 600 ~/rsi_loop/token
|
||||
|
||||
export PYTHONPATH=/path/to/streamline/scripts/rsi_loop
|
||||
export RSI_LOOP_HOME="$HOME/rsi_loop"
|
||||
# 不设 RSI_LOOP_NODES = 自动发现控制面登记的全部节点;没开放会话接口的节点(没配令牌 /
|
||||
# 卫星版本旧)返回 404,记进 sync.notEnabled 后跳过
|
||||
export RSI_LOOP_ACTL="$HOME/rsi_loop/actl/lib/index.js"
|
||||
```
|
||||
|
||||
先在图形桌面终端确认 `~/.local/bin/claude -p '只回答 OK'` 能用,然后**先试运行**:
|
||||
|
||||
```bash
|
||||
cd ~/rsi_loop/app
|
||||
./run.sh dry-run
|
||||
```
|
||||
|
||||
试运行对控制面只读:同步、切片、审计、提案、重放都真实执行,但提交实验和写时间线不发出去,
|
||||
改记到 `~/rsi_loop/data/dry_posts.jsonl`。看过要提交的实验(标题、四段记录、候选改动、离线结果)
|
||||
没问题,再正式跑一轮 —— 正式跑一旦离线闸通过,控制面就在**该问题所属平台**的实验机上建复跑任务:
|
||||
|
||||
```bash
|
||||
./run.sh once
|
||||
```
|
||||
|
||||
确认 RSI 页面出现运行日志后,在同一个图形桌面终端常驻:
|
||||
|
||||
```bash
|
||||
./run.sh daemon
|
||||
```
|
||||
|
||||
默认每天本机时间 03:00 运行,可用 `RSI_LOOP_DAILY_HOUR` 调整。不要从普通 SSH 会话
|
||||
启动:该机器的 Claude OAuth 在 SSH 环境里会过期。
|
||||
|
||||
常用参数:
|
||||
|
||||
```bash
|
||||
RSI_LOOP_MAX_AUDITS=60 # 每轮最多审计片段
|
||||
RSI_LOOP_TARGET_POINTS=36 # 目标类重放点
|
||||
RSI_LOOP_HOLDOUT_POINTS=15 # 其它模式保留集
|
||||
RSI_LOOP_SEEDS=3 # 每个点、每个版本跑三遍(两遍时区间太宽,打平的候选一半被误拦)
|
||||
RSI_LOOP_WORKERS=4 # 并发 Claude 续跑数
|
||||
RSI_LOOP_MAX_REPLAYS=320 # 单轮续跑硬上限;要 ≥ (目标点+保留集点)×2×遍数,否则先挤掉保留集
|
||||
```
|
||||
|
||||
## 安全与可重算性
|
||||
|
||||
- 原生会话、审计证据和重放原始输出只落在 `~/rsi_loop/data`,目录权限 `0700`、
|
||||
文件权限 `0600`,默认 14 天删除;发给控制面的日志只有聚合数字和模式名。
|
||||
- 重放只取模型第一步动作。所有工具由 `PreToolUse` 钩子拒绝,不执行设备、文件或网络工具。
|
||||
- 失败点截在工具结果之后;有未结束后台任务、缺环境附件或上下文过大的点不参与重放。
|
||||
- 候选新增行按提案给的锚点行插进历史会话第一条用户 prompt 的同一位置(锚点里的占位符按
|
||||
通配匹配,不加任何标题),与生产渲染的位置一致;锚点在某个点上定位不到,这个点两组都不跑,
|
||||
剔除数记在 `offline.meta`。base/candidate 使用同一历史前缀,判分只看封闭的动作形态,不用 LLM judge。
|
||||
- 保留集只取同一 stage、审计过且不是误报的失败点,加上同 stage 的等待点。
|
||||
- 不提交的情况:目标点不足 15 个(选题时先数,锚点剔除后再数一次),或重放报错率超过 20%
|
||||
(基础设施故障不能记成「候选无效」)。只写一条运行日志,并在本地记一次冷却。
|
||||
- 每次重放的逐条结果保存在 `data/runs/<时间>/results.jsonl`;95% 区间以固定种子对
|
||||
决策点做 paired bootstrap,可离线重算。
|
||||
- 暂停时仍可同步、审计和补结论,但不会提交新实验;控制面也会以 409 再挡一次。
|
||||
|
||||
本地测试(不访问控制面、不调用 Claude):
|
||||
|
||||
```bash
|
||||
cd scripts/rsi_loop
|
||||
python3 -m unittest discover -s tests -v
|
||||
```
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Streamline 自动 RSI 回路进程(Python 3.9 标准库)。"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,27 @@
|
||||
import argparse
|
||||
import json
|
||||
|
||||
from . import runner
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Streamline 自动 RSI 回路")
|
||||
parser.add_argument("command", choices=("once", "dry-run", "daemon", "review-mining"), nargs="?", default="once")
|
||||
args, rest = parser.parse_known_args()
|
||||
if args.command == "dry-run":
|
||||
from . import config
|
||||
config.DRY_RUN = True
|
||||
print(json.dumps(runner.locked_run(), ensure_ascii=False, indent=2))
|
||||
elif args.command == "daemon":
|
||||
runner.daemon()
|
||||
elif args.command == "review-mining":
|
||||
# 单独跑挖掘(补标历史 / 立刻归纳一次),参数见 review_mining.py 头注释
|
||||
from . import config, review_mining
|
||||
config.ensure_dirs()
|
||||
print(json.dumps(review_mining.main(rest), ensure_ascii=False, indent=2))
|
||||
else:
|
||||
print(json.dumps(runner.locked_run(), ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,78 @@
|
||||
"""控制面 /api/rsi-loop/* 的客户端(回路令牌鉴权)。
|
||||
|
||||
回路进程只能读状态、取会话记录、提交实验、写时间线;上线 / 回滚 / revert 由控制面
|
||||
的判定任务决定,这里没有、也调不到那些动作。
|
||||
"""
|
||||
import json
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from . import config
|
||||
|
||||
|
||||
def _req(method: str, path: str, body: Optional[dict] = None, timeout: int = 60):
|
||||
data = json.dumps(body, ensure_ascii=False).encode() if body is not None else None
|
||||
req = urllib.request.Request(config.API + path, data=data, method=method)
|
||||
req.add_header("x-rsi-loop-token", config.token())
|
||||
if data is not None:
|
||||
req.add_header("content-type", "application/json")
|
||||
return urllib.request.urlopen(req, timeout=timeout)
|
||||
|
||||
|
||||
def get(path: str, timeout: int = 60) -> Any:
|
||||
with _req("GET", path, timeout=timeout) as r:
|
||||
return json.loads(r.read().decode())
|
||||
|
||||
|
||||
def post(path: str, body: dict, timeout: int = 120) -> Any:
|
||||
if config.DRY_RUN:
|
||||
# 试运行:不写生产,把本来要发的内容留在本地供检查
|
||||
out = config.DATA / "dry_posts.jsonl"
|
||||
with open(out, "a") as f:
|
||||
f.write(json.dumps({"path": path, "body": body}, ensure_ascii=False) + "\n")
|
||||
out.chmod(0o600)
|
||||
print(f"[rsi-loop] dry-run: POST {path} 未发送,已记到 {out}", flush=True)
|
||||
if path == "/experiments":
|
||||
return {"experiment": {"id": None, "title": body.get("title"), "status": "dry_run"}, "gate": None}
|
||||
return {}
|
||||
try:
|
||||
with _req("POST", path, body, timeout=timeout) as r:
|
||||
return json.loads(r.read().decode())
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode(errors="replace")[:500]
|
||||
raise RuntimeError(f"POST {path} -> HTTP {e.code}: {detail}") from None
|
||||
|
||||
|
||||
def download(path: str, dest: Path, timeout: int = 900) -> int:
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
with _req("GET", path, timeout=timeout) as r, open(tmp, "wb") as f:
|
||||
n = 0
|
||||
while True:
|
||||
chunk = r.read(1 << 20)
|
||||
if not chunk:
|
||||
break
|
||||
f.write(chunk)
|
||||
n += len(chunk)
|
||||
tmp.replace(dest)
|
||||
return n
|
||||
|
||||
|
||||
def event(title: str, body: str = "", type_: str = "run") -> bool:
|
||||
"""回路层面的时间线事件(RSI 页面「回路日志」)。失败不影响主流程。"""
|
||||
try:
|
||||
post("/events", {"type": type_, "title": title[:280], "body": body[:20000]})
|
||||
return True
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[rsi-loop] event post failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def experiment_event(exp_id: str, title: str, body: str = "", type_: str = "note") -> bool:
|
||||
try:
|
||||
post(f"/experiments/{exp_id}/events", {"type": type_, "title": title[:280], "body": body[:12000]})
|
||||
return True
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[rsi-loop] experiment event failed: {e}")
|
||||
return False
|
||||
@@ -0,0 +1,91 @@
|
||||
"""回路进程的配置 —— 全部来自环境变量,缺省值对应 temp-ios-01 上的布局。
|
||||
|
||||
只依赖 Python 3.9 标准库:temp-ios-01 上只有系统自带的 python3,不装任何东西。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
HOME = Path.home()
|
||||
ROOT = Path(os.environ.get("RSI_LOOP_HOME", str(HOME / "rsi_loop")))
|
||||
DATA = ROOT / "data"
|
||||
TX_DIR = DATA / "tx" # 原生会话记录(按节点分目录),保留 RETAIN_DAYS 天
|
||||
STATE = DATA / "state.json" # 每个节点上次同步到的时刻等
|
||||
EPISODES = DATA / "episodes.jsonl" # 切出来的多轮失败片段(按会话文件缓存)
|
||||
POINTS = DATA / "points.jsonl" # 可重放的决策点(失败片段 + 等待起点)
|
||||
AUDITS = DATA / "audits.jsonl" # 审计结论
|
||||
RUNS = DATA / "runs" # 每次重放的原始结果,留档可重算
|
||||
LOCK = DATA / "runner.lock"
|
||||
|
||||
API = os.environ.get("RSI_LOOP_API", "https://streamline-internal-api.bytedance.net/api/rsi-loop")
|
||||
TOKEN_FILE = Path(os.environ.get("RSI_LOOP_TOKEN_FILE", str(ROOT / "token")))
|
||||
|
||||
# 数据源节点(逗号分隔)。不设 = 自动发现控制面登记的全部节点,逐个尝试;只有 .env 里配了
|
||||
# RSI_LOOP_TOKEN、且跑到带会话记录接口的卫星版本的节点才会开放,其余返回 404,静默跳过。
|
||||
NODES = [n.strip() for n in os.environ.get("RSI_LOOP_NODES", "").split(",") if n.strip()]
|
||||
|
||||
CLAUDE_BIN = os.environ.get("CLAUDE_BIN", str(HOME / ".local" / "bin" / "claude"))
|
||||
# 重放必须用会话当时的模型(否则 thinking 签名对不上);审计 / 提案 / 写结论用同一个即可。
|
||||
REPLAY_MODEL = os.environ.get("RSI_LOOP_REPLAY_MODEL", "claude-opus-5[1m]")
|
||||
THINK_MODEL = os.environ.get("RSI_LOOP_MODEL", "claude-opus-5")
|
||||
# actl 设备工具的 MCP 服务(只为让重放时的工具定义与节点一致;调用全部被钩子拦下)
|
||||
ACTL_INDEX = os.environ.get("RSI_LOOP_ACTL", str(ROOT / "actl" / "lib" / "index.js"))
|
||||
|
||||
RETAIN_DAYS = int(os.environ.get("RSI_LOOP_RETAIN_DAYS", "14"))
|
||||
SYNC_DAYS = int(os.environ.get("RSI_LOOP_SYNC_DAYS", "14"))
|
||||
# 2026-10-08 由 40 调到 60:每天新片段约 40 段,40 的上限让积压一直消不掉(当时积压 41 段)。
|
||||
MAX_AUDITS_PER_RUN = int(os.environ.get("RSI_LOOP_MAX_AUDITS", "60"))
|
||||
# 一次重放的总调用数上限:(目标 36 + 保留集 15) × 两组 × 3 遍 = 306,留一点余量。
|
||||
# 上限不够时先挤掉保留集 —— 上限和点数、遍数要一起改。
|
||||
MAX_REPLAYS_PER_RUN = int(os.environ.get("RSI_LOOP_MAX_REPLAYS", "320"))
|
||||
WORKERS = int(os.environ.get("RSI_LOOP_WORKERS", "4"))
|
||||
# 每个点每组跑几遍。2026-10-03 用 24 点 × 2 遍时差值区间半宽约 ±0.11,与现行打平的候选
|
||||
# 有一半会被「不能明显变差」(下界 ≥ −0.10)误拦;36 点 × 3 遍把半宽收到约 ±0.08。
|
||||
SEEDS = int(os.environ.get("RSI_LOOP_SEEDS", "3"))
|
||||
# 重放成本控制:截断处上下文超过这个 token 数的决策点不用
|
||||
MAX_POINT_CTX = int(os.environ.get("RSI_LOOP_MAX_POINT_CTX", "250000"))
|
||||
TARGET_POINTS = int(os.environ.get("RSI_LOOP_TARGET_POINTS", "36"))
|
||||
HOLDOUT_POINTS = int(os.environ.get("RSI_LOOP_HOLDOUT_POINTS", "15"))
|
||||
# 离线闸(与 shared/src/rsi_loop.ts 的 LOOP 一致;控制面会再按它的规则复核一遍)
|
||||
OFFLINE_MIN_POINTS = 15
|
||||
OFFLINE_HOLDOUT_TOLERANCE = 0.10
|
||||
# 同一类问题被试过之后的冷却期(天):避免对同一个模式反复提几乎一样的改法
|
||||
CLUSTER_COOLDOWN_DAYS = int(os.environ.get("RSI_LOOP_COOLDOWN_DAYS", "7"))
|
||||
# 旋钮实验:实验机上现行组已经是这个取值的旋钮,不能再当候选(两组一样,白跑一轮)。
|
||||
# 实验机不传 --effort 时用的是各自 ~/.claude/settings.json 的 modelSettings.<模型>.effortLevel:
|
||||
# evaluator-android-01 是 xhigh(2026-10-08 查),temp-ios-01 没设(模型默认)。实验机配置改了要同步这里。
|
||||
LAB_BASELINE = json.loads(os.environ.get("RSI_LOOP_LAB_BASELINE", '{"android": {"effort": "xhigh"}}'))
|
||||
# 思考强度的候选顺序:先试离现状近、省时间的方向。「不设」对应的模型默认档未知,放在最后。
|
||||
EFFORT_ORDER = ["medium", "high", "low", "max", "xhigh"]
|
||||
# 试运行:对控制面只读,所有写操作(提交实验、写时间线)改记到 data/dry_posts.jsonl
|
||||
DRY_RUN = os.environ.get("RSI_LOOP_DRY_RUN") == "1"
|
||||
# 每天几点(本机时间)跑一次
|
||||
DAILY_HOUR = int(os.environ.get("RSI_LOOP_DAILY_HOUR", "3"))
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
"""阶段进度。整轮要一两个小时,不出声的话没法分辨「在干活」还是「卡死了」。"""
|
||||
import time as _t
|
||||
print(f"[rsi-loop {_t.strftime('%H:%M:%S')}] {msg}", flush=True)
|
||||
|
||||
|
||||
def child_env(cwd: str) -> dict:
|
||||
"""给 claude 子进程的环境:PWD 指向它自己的工作目录。
|
||||
|
||||
Claude Code 会参考 PWD 确定工作目录;继承回路进程的 PWD(…/scripts/rsi_loop)会让模型在
|
||||
重放里看到「rsi_loop」,第一步就去查它。回路自己的变量、PYTHONPATH 也一并去掉。"""
|
||||
drop = ("PWD", "OLDPWD", "PYTHONPATH")
|
||||
env = {k: v for k, v in os.environ.items()
|
||||
if not k.startswith(("CLAUDE", "RSI_LOOP")) and k not in drop}
|
||||
env["PWD"] = cwd
|
||||
return env
|
||||
|
||||
|
||||
def token() -> str:
|
||||
return TOKEN_FILE.read_text().strip()
|
||||
|
||||
|
||||
def ensure_dirs() -> None:
|
||||
for d in (DATA, TX_DIR, RUNS):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
d.chmod(0o700)
|
||||
@@ -0,0 +1,92 @@
|
||||
"""把一份执行会话切成「多轮失败片段」:同一堵墙、或同一条路连续失败 ≥2 次的一段。
|
||||
每个片段标出能用来做决策点重放的截断位置(片段内第一次失败的工具结果之后)。
|
||||
|
||||
片段同时产出一个候选决策点:
|
||||
- 撞墙涉及 frida → T3_frida
|
||||
- 下一步是等待类命令 → T2_wait(改成挑「所有要开始等待的时刻」更干净,见 harvest)
|
||||
- 其余同族连续失败 → T1_repeat
|
||||
合格的决策点要求:截断时没有在跑的后台任务、会话里有环境附件(能中和)、上下文不超限。
|
||||
"""
|
||||
import re
|
||||
from typing import List, Optional
|
||||
|
||||
from . import transcript as T
|
||||
|
||||
# UI 相关的等待(dumpsys/uiautomator/pidof/actl 等),T2 的判定基准
|
||||
# 「等 App / 界面就绪」的探查命令。Android:dumpsys / uiautomator / pidof / actl;iOS:catuse(界面大纲、截屏、
|
||||
# 进程列表都走它)。2026-10-08 之前只认 Android,iOS 的 104 段 bug.run 会话一个等待点都抽不出来。
|
||||
UI_LIKE = re.compile(r"dumpsys\s+(window|activity)|uiautomator|mCurrentFocus|ResumedActivity|\bpidof\b|\bactl\b|\bcatuse\b")
|
||||
|
||||
|
||||
def find_episodes(rows: List[Optional[dict]]) -> List[dict]:
|
||||
cs = T.calls(rows)
|
||||
n = len(cs)
|
||||
eps: List[dict] = []
|
||||
|
||||
# 按「墙」聚合:同一堵墙撞 ≥2 次
|
||||
by_wall: dict = {}
|
||||
for c in cs:
|
||||
if c["wall"]:
|
||||
by_wall.setdefault(c["wall"], []).append(c)
|
||||
for w, hits in by_wall.items():
|
||||
if len(hits) < 2:
|
||||
continue
|
||||
fams = sorted({h["fam"] for h in hits})
|
||||
ptype = "T3_frida" if any("frida" in f for f in fams) else "T1_repeat"
|
||||
eps.append(_episode(cs, hits, label=f"wall:{w}", ptype=ptype, total=n))
|
||||
|
||||
# 按命令族聚合:同族连续失败 ≥2 次(排除只读命令的常态失败)
|
||||
open_ep: dict = {}
|
||||
for c in cs:
|
||||
if c["bg"] or c["read_only"]:
|
||||
continue
|
||||
f = c["fam"]
|
||||
if not c["ok"]:
|
||||
open_ep.setdefault(f, []).append(c)
|
||||
elif f in open_ep:
|
||||
hits = open_ep.pop(f)
|
||||
if len(hits) >= 2:
|
||||
ptype = "T3_frida" if "frida" in f else "T1_repeat"
|
||||
eps.append(_episode(cs, hits, label=f"path:{f}", ptype=ptype, total=n, resolved_at=c["i"]))
|
||||
for f, hits in open_ep.items():
|
||||
if len(hits) >= 2:
|
||||
ptype = "T3_frida" if "frida" in f else "T1_repeat"
|
||||
eps.append(_episode(cs, hits, label=f"path:{f}", ptype=ptype, total=n))
|
||||
# 同一组失败常会同时命中 wall 与 path;只审计一次,否则浪费会被双算。
|
||||
unique = {}
|
||||
for ep in eps:
|
||||
unique.setdefault((ep["first_call"], ep["last_call"], ep["ptype"]), ep)
|
||||
return list(unique.values())
|
||||
|
||||
|
||||
def _episode(cs: List[dict], hits: List[dict], label: str, ptype: str, total: int, resolved_at: Optional[int] = None) -> dict:
|
||||
first, last = hits[0], hits[-1]
|
||||
return dict(label=label, ptype=ptype, n_fail=len(hits), first_call=first["i"], last_call=last["i"],
|
||||
total_calls=total, fam=first["fam"], err=first["err"], err_last=last["err"],
|
||||
fail_cmds=[h["cmd"][:240] for h in hits[:4]], resolved=resolved_at is not None,
|
||||
# 决策点:截在第一次失败的工具结果之后(含),让模型面对「刚撞了一次墙」。
|
||||
# 截在前一个结果会让模型尚未见到失败,无法判定下一步是否在重复撞墙。
|
||||
point_call=first["i"], point_row=first["row"], point_ctx=first["ctx"])
|
||||
|
||||
|
||||
def harvest_wait_points(rows: List[Optional[dict]]) -> List[dict]:
|
||||
"""T2_wait 不按结果挑,而是挑所有「要开始等待」的时刻 —— 下一步该用 wait-for 还是手写 sleep,
|
||||
由重放决定,这样历史比例才能拿来和重放对照。"""
|
||||
cs = T.calls(rows)
|
||||
out: List[dict] = []
|
||||
last_result_call: Optional[dict] = None
|
||||
for c in cs:
|
||||
cmd = c["cmd"]
|
||||
if c["tool"] == "Bash" and (("wait-for" in cmd) or _is_wait(cmd)) and UI_LIKE.search(cmd) and last_result_call is not None:
|
||||
out.append(dict(label="wait", ptype="T2_wait", first_call=c["i"], last_call=c["i"], total_calls=len(cs),
|
||||
fam=c["fam"], err="", fail_cmds=[cmd[:240]], resolved=True,
|
||||
point_call=last_result_call["i"], point_row=last_result_call["row"], point_ctx=last_result_call["ctx"]))
|
||||
last_result_call = c
|
||||
return out
|
||||
|
||||
|
||||
def _is_wait(cmd: str) -> bool:
|
||||
if re.search(r"\b(while|until|for)\b.*\bsleep\b", cmd, re.S):
|
||||
return True
|
||||
return any(int(x) >= 15 for x in re.findall(r"\bsleep\s+(\d+)", cmd))
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""调 claude 做「读材料、写判断」的活:审计分类、写提案、写结论。
|
||||
|
||||
全部是纯文本进、纯文本出:`--tools ""` 关掉所有工具,`--setting-sources ""` 不加载本机
|
||||
任何设置,材料从标准输入喂进去。它不碰任何系统,只产出要被下游(控制面的规则)检验的东西。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from typing import Any, Optional
|
||||
|
||||
from . import config
|
||||
|
||||
|
||||
def ask(prompt: str, timeout: int = 900, model: Optional[str] = None) -> str:
|
||||
env = config.child_env("/tmp")
|
||||
cmd = [config.CLAUDE_BIN, "-p", "--output-format", "json", "--model", model or config.THINK_MODEL,
|
||||
"--tools", "", "--setting-sources", "", "--strict-mcp-config"]
|
||||
p = subprocess.run(cmd, input=prompt, capture_output=True, text=True, timeout=timeout, env=env, cwd="/tmp")
|
||||
try:
|
||||
d = json.loads(p.stdout)
|
||||
except json.JSONDecodeError:
|
||||
raise RuntimeError(f"claude 输出不是 JSON: {p.stdout[:300]} {p.stderr[:300]}") from None
|
||||
if d.get("is_error"):
|
||||
raise RuntimeError(f"claude 报错: {str(d.get('result'))[:300]}")
|
||||
return str(d.get("result") or "")
|
||||
|
||||
|
||||
def ask_json(prompt: str, timeout: int = 900) -> Any:
|
||||
"""要求模型只回一个 JSON 对象;用 JSONDecoder 取第一个完整对象(允许嵌套)。"""
|
||||
text = ask(prompt, timeout=timeout)
|
||||
fenced = re.search(r"```(?:json)?\s*([\s\S]*?)```", text)
|
||||
candidates = [fenced.group(1)] if fenced else []
|
||||
candidates.append(text)
|
||||
decoder = json.JSONDecoder()
|
||||
for candidate in candidates:
|
||||
for match in re.finditer(r"\{", candidate):
|
||||
try:
|
||||
value, _ = decoder.raw_decode(candidate[match.start():])
|
||||
return value
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
raise RuntimeError(f"回复里没有合法 JSON: {text[:400]}") from None
|
||||
@@ -0,0 +1,511 @@
|
||||
"""发现:同步会话记录 → 切出多轮失败片段 → 审计分类 → 按问题模式聚类。
|
||||
|
||||
切片只是候选:自动从输出文本判断「失败」有误报(读到的源码里的报错字样等),所以每个
|
||||
片段都要过一遍审计。审计用模型,但它只产出「假设」—— 问题属于哪一类、哪一步之后本该
|
||||
做对 —— 真正决定改动去留的是离线重放与真机复跑的配对结果,不是这里的判断。
|
||||
"""
|
||||
import json
|
||||
import hashlib
|
||||
import urllib.error
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional
|
||||
from urllib.parse import quote
|
||||
|
||||
from . import api, config, episodes, llm
|
||||
from .transcript import (
|
||||
classify_stage, first_prompt, has_env_attachment, load_rows, pending_bg,
|
||||
render_timeline, text_of,
|
||||
)
|
||||
|
||||
RUBRIC = (Path(__file__).resolve().parent.parent / "AUDIT_RUBRIC.md")
|
||||
|
||||
|
||||
# ── 同步 ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _state() -> dict:
|
||||
try:
|
||||
value = json.loads(config.STATE.read_text())
|
||||
return value if isinstance(value, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def _write_json(path: Path, value: Any) -> None:
|
||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||
tmp.write_text(json.dumps(value, ensure_ascii=False, indent=2))
|
||||
os.chmod(tmp, 0o600)
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def _write_jsonl(path: Path, rows: Iterable[dict]) -> None:
|
||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||
with open(tmp, "w") as f:
|
||||
for row in rows:
|
||||
f.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
os.chmod(tmp, 0o600)
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def _read_jsonl(path: Path) -> List[dict]:
|
||||
out: List[dict] = []
|
||||
try:
|
||||
with open(path, errors="replace") as f:
|
||||
for line in f:
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(row, dict):
|
||||
out.append(row)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _id(*parts: object) -> str:
|
||||
raw = "\0".join(str(p) for p in parts).encode()
|
||||
return hashlib.sha256(raw).hexdigest()[:24]
|
||||
|
||||
|
||||
def _local_file(node: str, name: str) -> Path:
|
||||
return config.TX_DIR / quote(node, safe="._-") / name
|
||||
|
||||
|
||||
def _purge_expired(now_ms: int) -> int:
|
||||
cutoff = now_ms - config.RETAIN_DAYS * 86_400_000
|
||||
removed = 0
|
||||
for node_dir in config.TX_DIR.glob("*"):
|
||||
if not node_dir.is_dir():
|
||||
continue
|
||||
for meta_path in node_dir.glob("*.meta.json"):
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text())
|
||||
expired = int(meta.get("mtime") or 0) < cutoff
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
expired = True
|
||||
if not expired:
|
||||
continue
|
||||
raw = meta_path.with_name(meta_path.name.removesuffix(".meta.json") + ".jsonl")
|
||||
raw.unlink(missing_ok=True)
|
||||
meta_path.unlink(missing_ok=True)
|
||||
removed += 1
|
||||
try:
|
||||
node_dir.rmdir()
|
||||
except OSError:
|
||||
pass
|
||||
return removed
|
||||
|
||||
|
||||
def sync_transcripts(now_ms: Optional[int] = None) -> dict:
|
||||
"""只同步最近 14 天的执行会话;同 size/mtime 的文件不重复下载。"""
|
||||
config.ensure_dirs()
|
||||
now_ms = now_ms or int(time.time() * 1000)
|
||||
since = now_ms - config.SYNC_DAYS * 86_400_000
|
||||
stats = {"nodes": 0, "notEnabled": [], "listed": 0, "downloaded": 0, "bytes": 0, "errors": [], "purged": 0}
|
||||
state = _state()
|
||||
platforms = node_platforms()
|
||||
nodes = config.NODES or sorted(platforms)
|
||||
for node in nodes:
|
||||
try:
|
||||
try:
|
||||
listing = api.get(f"/nodes/{quote(node, safe='')}/transcripts?sinceMs={since}", timeout=900)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404: # 没配令牌 / 卫星版本旧:不是故障,记下来提示人去开
|
||||
stats["notEnabled"].append(node)
|
||||
continue
|
||||
raise
|
||||
files = listing.get("files") or []
|
||||
config.log(f"同步 {node}({platforms.get(node, '?')}):列出 {len(files)} 份会话")
|
||||
stats["nodes"] += 1
|
||||
stats["listed"] += len(files)
|
||||
node_dir = _local_file(node, "_").parent
|
||||
node_dir.mkdir(parents=True, exist_ok=True)
|
||||
node_dir.chmod(0o700)
|
||||
for info in files:
|
||||
name = str(info.get("file") or "")
|
||||
if not name.endswith(".jsonl"):
|
||||
continue
|
||||
dest = _local_file(node, name)
|
||||
meta_path = dest.with_name(dest.stem + ".meta.json")
|
||||
old = {}
|
||||
try:
|
||||
old = json.loads(meta_path.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
pass
|
||||
if (dest.exists() and int(old.get("size") or -1) == int(info.get("size") or -2)
|
||||
and int(old.get("mtime") or -1) == int(info.get("mtime") or -2)):
|
||||
if not old.get("platform") and platforms.get(node):
|
||||
_write_json(meta_path, {**old, "platform": platforms[node]})
|
||||
continue
|
||||
# 大文件经控制面代理偶尔中途断开(IncompleteRead);逐个文件重试,
|
||||
# 一份失败不能连累这个节点后面所有文件。
|
||||
n = None
|
||||
for attempt in range(3):
|
||||
try:
|
||||
n = api.download(f"/nodes/{quote(node, safe='')}/transcripts/{quote(name, safe='')}", dest)
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last = exc
|
||||
time.sleep(5 * (attempt + 1))
|
||||
if n is None:
|
||||
stats["errors"].append(f"{node}/{name}: {last}")
|
||||
continue
|
||||
os.chmod(dest, 0o600)
|
||||
meta = {
|
||||
"node": node, "file": name, "size": n,
|
||||
"mtime": int(info.get("mtime") or now_ms),
|
||||
"stage": info.get("stage"), "workspaceKey": info.get("workspaceKey"),
|
||||
"platform": platforms.get(node) or platform_of(node),
|
||||
}
|
||||
_write_json(meta_path, meta)
|
||||
stats["downloaded"] += 1
|
||||
stats["bytes"] += n
|
||||
config.log(f" 下载 {stats['downloaded']}:{name[:8]} {n / 1e6:.1f}MB")
|
||||
state.setdefault("nodes", {})[node] = {"syncedAt": now_ms, "sinceMs": since}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
stats["errors"].append(f"{node}: {exc}")
|
||||
stats["purged"] = _purge_expired(now_ms)
|
||||
state["lastSyncAt"] = now_ms
|
||||
_write_json(config.STATE, state)
|
||||
return stats
|
||||
|
||||
|
||||
def platform_of(node: str) -> str:
|
||||
"""节点名里带 ios 的是 iOS,其余按 Android(只在控制面没返回平台时兜底)。"""
|
||||
return "ios" if "ios" in (node or "").lower() else "android"
|
||||
|
||||
|
||||
def node_platforms() -> Dict[str, str]:
|
||||
"""控制面登记的节点 → 平台。"""
|
||||
try:
|
||||
return {n["id"]: n.get("platform") or platform_of(n["id"]) for n in api.get("/nodes").get("nodes") or []}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
config.log(f"取节点列表失败,只用 RSI_LOOP_NODES:{exc}")
|
||||
return {n: platform_of(n) for n in config.NODES}
|
||||
|
||||
|
||||
def transcript_files() -> List[tuple]:
|
||||
out = []
|
||||
for meta_path in config.TX_DIR.glob("*/*.meta.json"):
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
continue
|
||||
raw = meta_path.with_name(meta_path.name.removesuffix(".meta.json") + ".jsonl")
|
||||
if raw.exists():
|
||||
out.append((raw, meta))
|
||||
return out
|
||||
|
||||
|
||||
def mine_local() -> dict:
|
||||
"""重建最近会话的片段和决策点缓存。缓存不含完整工单正文。"""
|
||||
eps: List[dict] = []
|
||||
points: List[dict] = []
|
||||
bad_files = 0
|
||||
for path, meta in transcript_files():
|
||||
# 真机复跑任务(控制面 rsi_lab.ts,工作区名带 rsi-lab- 前缀)跑在实验机上,也会出现在数据源
|
||||
# 节点的会话列表里;它们是回路自己的实验,不能再被当成生产样本挖回来
|
||||
if str(meta.get("workspaceKey") or "").startswith("rsi-lab-"):
|
||||
continue
|
||||
rows = load_rows(path)
|
||||
prompt = first_prompt(rows)
|
||||
stage = classify_stage(text_of((prompt or {}).get("message", {}).get("content"))) if prompt else None
|
||||
if not stage:
|
||||
bad_files += 1
|
||||
continue
|
||||
common = {
|
||||
"node": meta.get("node"), "file": meta.get("file"), "mtime": meta.get("mtime"),
|
||||
"platform": meta.get("platform") or platform_of(str(meta.get("node"))),
|
||||
"stage": stage, "has_env": has_env_attachment(rows),
|
||||
}
|
||||
for ep in episodes.find_episodes(rows):
|
||||
item = {**common, **ep}
|
||||
item["id"] = _id(item["node"], item["file"], item["label"], item["first_call"], item["last_call"])
|
||||
eps.append(item)
|
||||
point = _point_from_episode(item, rows)
|
||||
if point:
|
||||
points.append(point)
|
||||
for wp in episodes.harvest_wait_points(rows):
|
||||
item = {**common, **wp}
|
||||
item["id"] = _id(item["node"], item["file"], "wait", item["point_row"])
|
||||
point = _point_from_episode(item, rows)
|
||||
if point:
|
||||
points.append(point)
|
||||
# 一份会话里的 wall/path 切片可能指向同一个截断点;重放只保留一个。
|
||||
point_map = {(p["node"], p["file"], p["point_row"], p["ptype"]): p for p in points}
|
||||
eps.sort(key=lambda x: (int(x.get("mtime") or 0), x["id"]), reverse=True)
|
||||
points = sorted(point_map.values(), key=lambda x: (int(x.get("mtime") or 0), x["id"]), reverse=True)
|
||||
_write_jsonl(config.EPISODES, eps)
|
||||
_write_jsonl(config.POINTS, points)
|
||||
config.log(f"切片完成:{len(eps)} 段失败片段,{len(points)} 个可重放的决策点")
|
||||
return {"episodes": len(eps), "points": len(points), "badFiles": bad_files}
|
||||
|
||||
|
||||
def _point_from_episode(item: dict, rows: List[Optional[dict]]) -> Optional[dict]:
|
||||
row = int(item.get("point_row") or 0)
|
||||
if not item.get("has_env") or int(item.get("point_ctx") or 0) > config.MAX_POINT_CTX:
|
||||
return None
|
||||
if pending_bg(rows, row):
|
||||
return None
|
||||
return {
|
||||
"id": _id(item["node"], item["file"], item["ptype"], row),
|
||||
"episodeId": item["id"], "node": item["node"], "file": item["file"],
|
||||
"mtime": item["mtime"], "stage": item["stage"], "platform": item.get("platform"), "ptype": item["ptype"],
|
||||
"fam": item.get("fam") or "", "point_row": row, "point_call": item.get("point_call"),
|
||||
"point_ctx": item.get("point_ctx") or 0,
|
||||
}
|
||||
|
||||
|
||||
# ── 问题模式的规范名 ─────────────────────────────────────────────────────────
|
||||
#
|
||||
# 审计给模式起名是自由发挥的:同一机制被起成不同名字,2026-10-02 那轮 40 段审计聚出 30 个模式,
|
||||
# 几乎每个只有一段 —— 「重复撞墙」「Frida」这几类永远凑不够离线闸要的 15 个决策点。两道补救:
|
||||
# 审计时把已有模式交给模型、要求复用;每轮审计后再做一次归并,写进别名表(旧名 → 规范名)。
|
||||
|
||||
ALIASES = "cluster_aliases.json"
|
||||
|
||||
|
||||
def _aliases() -> Dict[str, str]:
|
||||
try:
|
||||
value = json.loads((config.DATA / ALIASES).read_text())
|
||||
return value if isinstance(value, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def canon(name: Optional[str], aliases: Optional[Dict[str, str]] = None) -> str:
|
||||
"""沿别名表走到规范名;有环就停在环上第一次重复之前。"""
|
||||
table = _aliases() if aliases is None else aliases
|
||||
cur, seen = str(name or ""), set()
|
||||
while cur in table and cur not in seen:
|
||||
seen.add(cur)
|
||||
cur = str(table[cur])
|
||||
return cur
|
||||
|
||||
|
||||
def known_clusters(limit: int = 60) -> List[dict]:
|
||||
"""已有的问题模式(按规范名合并),按出现次数排序 —— 审计时交给模型复用。"""
|
||||
aliases = _aliases()
|
||||
agg: Dict[str, dict] = {}
|
||||
for a in _read_jsonl(config.AUDITS):
|
||||
if a.get("classification") == "not_failure" or not a.get("cluster"):
|
||||
continue
|
||||
name = canon(a["cluster"], aliases)
|
||||
slot = agg.setdefault(name, {"cluster": name, "titles": {}, "count": 0,
|
||||
"classification": a.get("classification"), "replayType": a.get("replayType")})
|
||||
slot["count"] += 1
|
||||
t = str(a.get("clusterTitle") or "")
|
||||
slot["titles"][t] = slot["titles"].get(t, 0) + 1
|
||||
out = []
|
||||
for slot in agg.values():
|
||||
slot["clusterTitle"] = max(slot.pop("titles").items(), key=lambda kv: kv[1])[0]
|
||||
out.append(slot)
|
||||
return sorted(out, key=lambda x: -x["count"])[:limit]
|
||||
|
||||
|
||||
def _known_block(known: List[dict]) -> str:
|
||||
if not known:
|
||||
return "(还没有已知模式,按机制新建一个 snake_case 名字)"
|
||||
lines = ["| cluster | 中文名 | 类别 | 重放型 | 已出现 |", "|---|---|---|---|---|"]
|
||||
lines += [f"| {k['cluster']} | {k['clusterTitle']} | {k['classification']} | {k['replayType']} | {k['count']} |" for k in known]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def consolidate_clusters() -> dict:
|
||||
"""把同一机制的不同名字归并到一个规范名,写进别名表。只合并重放型相同的;规范名必须是已有名字。"""
|
||||
known = known_clusters(limit=200)
|
||||
if len(known) < 2:
|
||||
return {"clusters": len(known), "merged": 0}
|
||||
prompt = f"""下面是 coding agent 执行失败审计出来的「问题模式」,名字是逐段审计时各自起的,同一机制常被起成不同名字。
|
||||
请把**同一机制**的名字归并到其中一个**已有**名字(规范名)上:只合并重放型相同的;拿不准就不合并。
|
||||
|
||||
{_known_block(known)}
|
||||
|
||||
只返回 JSON:{{"merge": {{"<要并掉的名字>": "<规范名>", ...}}}}。没有要合并的就返回 {{"merge": {{}}}}。"""
|
||||
try:
|
||||
value = llm.ask_json(prompt, timeout=900)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
config.log(f"归并问题模式失败(本轮跳过):{exc}")
|
||||
return {"clusters": len(known), "merged": 0, "error": str(exc)[:200]}
|
||||
merge = value.get("merge") if isinstance(value, dict) else None
|
||||
by_name = {k["cluster"]: k for k in known}
|
||||
aliases = _aliases()
|
||||
merged = 0
|
||||
for old, new in (merge or {}).items():
|
||||
old, new = str(old), str(new)
|
||||
if old == new or old not in by_name or new not in by_name:
|
||||
continue # 规范名必须是已有名字;编出来的不收
|
||||
if by_name[old].get("replayType") != by_name[new].get("replayType"):
|
||||
continue # 重放型不同,判分口径不同,不能并
|
||||
if canon(new, {**aliases, old: new}) == old:
|
||||
continue # 会成环
|
||||
aliases[old] = new
|
||||
merged += 1
|
||||
if merged:
|
||||
path = config.DATA / ALIASES
|
||||
path.write_text(json.dumps(aliases, ensure_ascii=False, indent=2))
|
||||
os.chmod(path, 0o600)
|
||||
config.log(f"归并问题模式:{len(known)} 个里并掉 {merged} 个")
|
||||
return {"clusters": len(known), "merged": merged}
|
||||
|
||||
|
||||
# ── 审计 ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
ALLOWED_CLASSES = {"dead_end", "decision_error", "context_missing", "not_failure"}
|
||||
ALLOWED_REPLAY_TYPES = {"T1_repeat", "T2_wait", "T3_frida", "none"}
|
||||
|
||||
|
||||
def _audit_prompt(ep: dict, timeline: str, known: Optional[List[dict]] = None) -> str:
|
||||
rubric = RUBRIC.read_text()
|
||||
return f"""你在审计 coding agent 的一段历史执行记录。只做分类,不给修复建议。
|
||||
|
||||
{rubric}
|
||||
|
||||
已有的问题模式(**同一机制必须复用这里的 cluster 名字**;只有确实是新机制才新建):
|
||||
{_known_block(known or [])}
|
||||
|
||||
片段元数据:
|
||||
- stage: {ep['stage']}
|
||||
- 自动切片标签: {ep['label']}
|
||||
- 重复失败次数: {ep['n_fail']}
|
||||
- 调用范围: #{ep['first_call']}..#{ep['last_call']}
|
||||
|
||||
时间线:
|
||||
{timeline}
|
||||
|
||||
只返回一个 JSON 对象:
|
||||
{{
|
||||
"classification": "dead_end|decision_error|context_missing|not_failure",
|
||||
"cluster": "稳定、简短的 snake_case 问题模式",
|
||||
"clusterTitle": "中文模式名",
|
||||
"replayType": "T1_repeat|T2_wait|T3_frida|none",
|
||||
"correctInfoStep": 数字或 null,
|
||||
"evidence": ["逐字引用时间线中的原文,至少一条"],
|
||||
"rationale": "两三句,说明分类和正确的信息在何时已出现"
|
||||
}}
|
||||
不得把账号、token、工单正文复述进 cluster/rationale。evidence 必须逐字来自上面的时间线。"""
|
||||
|
||||
|
||||
def _valid_audit(value: Any, timeline: str) -> Optional[dict]:
|
||||
if not isinstance(value, dict):
|
||||
return None
|
||||
cls = value.get("classification")
|
||||
replay_type = value.get("replayType")
|
||||
cluster = str(value.get("cluster") or "")[:120]
|
||||
title = str(value.get("clusterTitle") or "")[:160]
|
||||
evidence = [str(x)[:500] for x in value.get("evidence") or [] if isinstance(x, str) and str(x).strip()]
|
||||
if cls not in ALLOWED_CLASSES or replay_type not in ALLOWED_REPLAY_TYPES or not cluster or not title:
|
||||
return None
|
||||
# 结论必须能回指原文;不接受模型编出来的“引文”。
|
||||
if not evidence or not any(q in timeline for q in evidence):
|
||||
return None
|
||||
step = value.get("correctInfoStep")
|
||||
if step is not None and not isinstance(step, int):
|
||||
step = None
|
||||
return {
|
||||
"classification": cls, "cluster": cluster, "clusterTitle": title,
|
||||
"replayType": replay_type, "correctInfoStep": step,
|
||||
"evidence": evidence[:3], "rationale": str(value.get("rationale") or "")[:1200],
|
||||
}
|
||||
|
||||
|
||||
def audit_new(max_items: Optional[int] = None) -> dict:
|
||||
episodes_by_id = {e["id"]: e for e in _read_jsonl(config.EPISODES)}
|
||||
existing = {a["episodeId"]: a for a in _read_jsonl(config.AUDITS) if a.get("episodeId")}
|
||||
todo = [e for e in episodes_by_id.values() if e["id"] not in existing]
|
||||
todo.sort(key=lambda e: (int(e.get("n_fail") or 0), int(e.get("mtime") or 0)), reverse=True)
|
||||
todo = todo[: max_items if max_items is not None else config.MAX_AUDITS_PER_RUN]
|
||||
ok = 0
|
||||
errors: List[str] = []
|
||||
config.log(f"审计:本轮 {len(todo)} 段(每段调一次 claude,约 1~2 分钟)")
|
||||
known = known_clusters()
|
||||
for k, ep in enumerate(todo, 1):
|
||||
config.log(f" 审计 {k}/{len(todo)}:{ep['label'][:50]}")
|
||||
path = _local_file(str(ep["node"]), str(ep["file"]))
|
||||
rows = load_rows(path)
|
||||
timeline = render_timeline(rows, int(ep["first_call"]), int(ep["last_call"]))
|
||||
try:
|
||||
value = _valid_audit(llm.ask_json(_audit_prompt(ep, timeline, known)), timeline)
|
||||
if not value:
|
||||
raise RuntimeError("审计结果缺字段、类别越界或证据不是原文")
|
||||
existing[ep["id"]] = {
|
||||
"episodeId": ep["id"], "auditedAt": int(time.time() * 1000),
|
||||
"stage": ep["stage"], "node": ep["node"], "file": ep["file"],
|
||||
"nFail": ep["n_fail"], **value,
|
||||
}
|
||||
ok += 1
|
||||
if not any(k["cluster"] == canon(value["cluster"]) for k in known): # 新模式,后面的审计也要看得到
|
||||
known.append({"cluster": canon(value["cluster"]), "clusterTitle": value["clusterTitle"], "count": 1,
|
||||
"classification": value["classification"], "replayType": value["replayType"]})
|
||||
# 每一条成功后就落盘;中途 OAuth/网络失败不会丢掉前面的昂贵结果。
|
||||
_write_jsonl(config.AUDITS, sorted(existing.values(), key=lambda x: int(x.get("auditedAt") or 0)))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
errors.append(f"{ep['id']}: {exc}")
|
||||
return {"requested": len(todo), "audited": ok, "errors": errors}
|
||||
|
||||
|
||||
# ── 聚类与重放点选择 ─────────────────────────────────────────────────────────
|
||||
|
||||
def clusters() -> List[dict]:
|
||||
episode_map = {e["id"]: e for e in _read_jsonl(config.EPISODES)}
|
||||
grouped: Dict[tuple, dict] = {}
|
||||
aliases = _aliases()
|
||||
titles = {k["cluster"]: k["clusterTitle"] for k in known_clusters(limit=1000)}
|
||||
for audit in _read_jsonl(config.AUDITS):
|
||||
ep = episode_map.get(audit.get("episodeId"))
|
||||
if not ep or audit.get("classification") == "not_failure":
|
||||
continue
|
||||
platform = ep.get("platform") or platform_of(str(ep.get("node")))
|
||||
name = canon(audit.get("cluster"), aliases)
|
||||
key = (audit.get("stage"), platform, audit.get("classification"), name, audit.get("replayType"))
|
||||
slot = grouped.setdefault(key, {
|
||||
"stage": key[0], "platform": platform, "problemClass": key[2], "cluster": key[3],
|
||||
"clusterTitle": titles.get(name) or audit.get("clusterTitle"), "replayType": key[4],
|
||||
"episodes": 0, "wasteCalls": 0, "transcripts": set(), "samples": [],
|
||||
})
|
||||
slot["episodes"] += 1
|
||||
slot["wasteCalls"] += int(ep.get("n_fail") or 0)
|
||||
slot["transcripts"].add((ep.get("node"), ep.get("file")))
|
||||
if len(slot["samples"]) < 4:
|
||||
slot["samples"].append({
|
||||
"episodeId": ep["id"], "nFail": ep.get("n_fail"),
|
||||
"evidence": audit.get("evidence") or [], "rationale": audit.get("rationale") or "",
|
||||
})
|
||||
out = []
|
||||
for slot in grouped.values():
|
||||
slot["transcripts"] = len(slot["transcripts"])
|
||||
out.append(slot)
|
||||
return sorted(out, key=lambda x: (x["wasteCalls"], x["episodes"]), reverse=True)
|
||||
|
||||
|
||||
def replay_points(cluster: Optional[dict] = None) -> List[dict]:
|
||||
points = _read_jsonl(config.POINTS)
|
||||
if not cluster:
|
||||
return points
|
||||
stage = cluster["stage"]
|
||||
platform = cluster.get("platform")
|
||||
ptype = cluster["replayType"]
|
||||
points = [p for p in points if p.get("stage") == stage and (not platform or p.get("platform") == platform)]
|
||||
if ptype == "T2_wait":
|
||||
return [p for p in points if p.get("ptype") == "T2_wait"]
|
||||
aliases = _aliases()
|
||||
audited_eps = {
|
||||
a["episodeId"] for a in _read_jsonl(config.AUDITS)
|
||||
if a.get("stage") == stage and canon(a.get("cluster"), aliases) == cluster.get("cluster")
|
||||
and a.get("classification") == cluster.get("problemClass")
|
||||
}
|
||||
return [p for p in points if p.get("episodeId") in audited_eps and p.get("ptype") == ptype]
|
||||
|
||||
|
||||
def real_episode_ids(stage: str) -> set:
|
||||
"""这个 stage 下审计过、且不是切片误报的片段 —— 保留集只用它们(自动切片约 25% 误报)。"""
|
||||
return {
|
||||
a["episodeId"] for a in _read_jsonl(config.AUDITS)
|
||||
if a.get("stage") == stage and a.get("episodeId") and a.get("classification") != "not_failure"
|
||||
}
|
||||
|
||||
|
||||
def load_rows_for_point(point: dict) -> List[Optional[dict]]:
|
||||
return load_rows(_local_file(str(point["node"]), str(point["file"])))
|
||||
@@ -0,0 +1,209 @@
|
||||
"""把排名最高的问题模式变成一个「只插入新行」的 prompt 候选。
|
||||
|
||||
模型只决定写什么;插入、占位符一致和长度限制都由本模块确定性复核。控制面提交时还会
|
||||
用 shared/src/rsi_loop.ts 再复核一遍。
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from typing import Any, List, Optional
|
||||
|
||||
from . import llm
|
||||
|
||||
STAGES = {"bug.run", "feature.run.first", "feature.run.second"}
|
||||
# 业务指标(真值来自证据 manifest / 终态)优先;过程指标是机制层面的浪费。
|
||||
# 口径与生产基线见 shared/src/rsi_loop.ts::LOOP_METRICS。
|
||||
BUSINESS_METRICS = {"reach_s", "reach_miss", "evidence_fail", "dead_end_s"}
|
||||
TARGET_METRICS = BUSINESS_METRICS | {
|
||||
"ui_wait_s", "blind_sleep_s", "failed_calls",
|
||||
"repeat_fail_calls", "frida_fail_calls", "tool_calls",
|
||||
}
|
||||
# 每种重放型能登记哪些目标:离线判的是动作机制,真机复跑判登记的指标,两者要是同一条因果链。
|
||||
# 等 App 就绪 / 撞墙 / Frida 乱注入都发生在走到复现现场之前或认死路之前,所以都能挂到达和走不通时长;
|
||||
# 证据校验失败只跟「撞同一堵墙」(重拍)一类有关。
|
||||
METRICS_BY_REPLAY = {
|
||||
"T1_repeat": {"repeat_fail_calls", "reach_s", "reach_miss", "dead_end_s", "evidence_fail"},
|
||||
"T2_wait": {"ui_wait_s", "blind_sleep_s", "reach_s", "reach_miss"},
|
||||
"T3_frida": {"frida_fail_calls", "reach_s", "dead_end_s"},
|
||||
}
|
||||
EVIDENCE_TERMS = re.compile(r"record-evidence|manifest|evidence/|\bbefore-|\bafter-|证据|截图|录屏|拍照|shot\b", re.I)
|
||||
|
||||
|
||||
def gaming_error(metric: str, base: str, candidate: str) -> Optional[str]:
|
||||
"""与 shared/src/rsi_loop.ts::lintMetricGaming 同一条规则:到达类目标不许改证据采集。"""
|
||||
if metric not in ("reach_s", "reach_miss"):
|
||||
return None
|
||||
added = inserted_text(base, candidate).split("\n")
|
||||
hit = next((line for line in added if EVIDENCE_TERMS.search(line)), None)
|
||||
return f"目标是到达复现现场,新增内容却在改证据采集:{hit.strip()[:60]}" if hit else None
|
||||
|
||||
|
||||
def placeholders(template: str) -> List[str]:
|
||||
return sorted(set(re.findall(r"\{\{\s*([\w.]+)\s*\}\}", template)))
|
||||
|
||||
|
||||
def candidate_error(base: str, candidate: str) -> Optional[str]:
|
||||
if placeholders(base) != placeholders(candidate):
|
||||
return "占位符不一致"
|
||||
if len(candidate) < len(base):
|
||||
return "候选比现行模板短"
|
||||
if len(candidate) - len(base) > 4000:
|
||||
return "新增超过 4000 字"
|
||||
base_lines = base.split("\n")
|
||||
j = 0
|
||||
for line in candidate.split("\n"):
|
||||
if j < len(base_lines) and line == base_lines[j]:
|
||||
j += 1
|
||||
return None if j == len(base_lines) else "候选改动了已有行"
|
||||
|
||||
|
||||
def apply_insertion(base: str, content: str, after_line: Optional[str]) -> str:
|
||||
addition = content.strip("\n")
|
||||
if not addition:
|
||||
raise ValueError("插入内容为空")
|
||||
lines = base.split("\n")
|
||||
if after_line:
|
||||
matches = [i for i, line in enumerate(lines) if line == after_line]
|
||||
if len(matches) != 1:
|
||||
raise ValueError("afterLine 必须逐字匹配现行模板中唯一的一行")
|
||||
at = matches[0] + 1
|
||||
lines[at:at] = addition.split("\n")
|
||||
else:
|
||||
lines.extend(addition.split("\n"))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def inserted_text(base: str, candidate: str) -> str:
|
||||
"""提取候选新增的行;base 不是 candidate 子序列时拒绝。"""
|
||||
base_lines = base.split("\n")
|
||||
added: List[str] = []
|
||||
j = 0
|
||||
for line in candidate.split("\n"):
|
||||
if j < len(base_lines) and line == base_lines[j]:
|
||||
j += 1
|
||||
else:
|
||||
added.append(line)
|
||||
if j != len(base_lines):
|
||||
raise ValueError("候选不是纯插入")
|
||||
return "\n".join(added).strip()
|
||||
|
||||
|
||||
def _prompt(cluster: dict, base: str) -> str:
|
||||
# 提案只需要审计结论,不需要把含工单正文/账号的逐字证据再喂一遍,更不能让它进入提交字段。
|
||||
safe_samples = [
|
||||
{"nFail": s.get("nFail"), "rationale": _redact(str(s.get("rationale") or ""))}
|
||||
for s in (cluster.get("samples") or [])
|
||||
]
|
||||
samples = json.dumps(safe_samples, ensure_ascii=False, indent=2)
|
||||
return f"""你负责给 coding agent 的执行 prompt 提一个小而可归因的改动。
|
||||
只能针对下面这个已审计的问题模式;一次只插入一段新纪律,不得改写、删除现有行,
|
||||
不得增加或删除 `{{{{占位符}}}}`。新增不超过 4000 字。
|
||||
|
||||
stage: {cluster['stage']}
|
||||
平台: {cluster.get('platform')}(问题来自这个平台的执行记录。下面的现行模板是按这个平台展开后的版本)
|
||||
策略范围(scope)二选一:
|
||||
- "platform":单端策略。规则只对这个平台成立(涉及该平台专有的工具、页面形态、系统行为)。
|
||||
只在这个平台的实验机复跑,通过后也只上这个平台,另一个平台的模板不变。
|
||||
- "both":双端策略。规则对两端都成立(与平台无关的执行纪律)。两端实验机各复跑一遍,
|
||||
两端都通过才两端一起上;只有一端通过就只上那一端。新增内容里出现单端专有的工具词
|
||||
(adb、catuse、xcrun 等)会被控制面按单端处理。
|
||||
拿不准就选 "platform" —— 单端通过后,人可以再决定补另一端复跑。
|
||||
问题类别: {cluster['problemClass']}
|
||||
模式: {cluster['cluster']} / {cluster.get('clusterTitle')}
|
||||
浪费: {cluster['wasteCalls']} 次失败调用,{cluster['episodes']} 个片段,
|
||||
{cluster['transcripts']} 份会话
|
||||
可引用的审计样本:
|
||||
{samples}
|
||||
|
||||
可事先登记的目标指标(越低越好)。优先登记业务指标 —— 它们是用户真正关心的结果,
|
||||
真值来自证据清单和任务终态,不来自 agent 自述;过程指标只在该模式与任何业务结果都
|
||||
没有可信因果链时才登记。
|
||||
业务指标(2026-10 生产基线,160 批):
|
||||
- reach_s: 到达复现现场用时 —— 从这批开始到第一张校验通过的修复前证据。中位 Android 30 分、
|
||||
iOS 44 分,约占整批一半。
|
||||
- reach_miss: 未到达复现现场 —— 一批结束仍没有合格的修复前证据。修好的批次里 Android 17%、
|
||||
iOS 14%;复现不了的 12 批里 10 批没到达。
|
||||
- evidence_fail: 证据校验失败次数(录空 / 太短 / 分辨率不够,需重拍)。每批 Android 0.6、iOS 1.0 次。
|
||||
- dead_end_s: 走不通时耗掉的时长 —— 没以 FIXED 收尾的批次的整批时长(早期失败识别)。
|
||||
复跑会混入历史上走不通的单,「修好的个数不少于现行」是底线,所以只能靠更早认出死路,不能靠早放弃。
|
||||
过程指标:
|
||||
- ui_wait_s: 等界面状态时长
|
||||
- blind_sleep_s: 盲等时长
|
||||
- failed_calls: 失败调用数
|
||||
- repeat_fail_calls: 同类命令连续失败
|
||||
- frida_fail_calls: Frida 失败调用
|
||||
- tool_calls: 工具调用总数
|
||||
本模式的重放型是 {cluster.get('replayType')},targetMetric 必须从
|
||||
{sorted(METRICS_BY_REPLAY.get(cluster.get('replayType'), TARGET_METRICS))} 中选,
|
||||
并在 hypothesis 里写清这个模式怎样因果地影响所选指标(离线重放判的是动作机制,
|
||||
真机复跑判的是所选指标,两者要是同一条因果链)。
|
||||
若选 reach_s / reach_miss:新增内容不得涉及证据采集(截图、录屏、record-evidence、
|
||||
before/after 命名)—— 到达是按第一张合格修复前证据量的,改快门时机会让指标变好而现场没变快,
|
||||
这种候选会被直接拒收。
|
||||
|
||||
现行模板(只能选其中唯一一行作为 afterLine;若放结尾则 afterLine=null):
|
||||
---BASE---
|
||||
{base}
|
||||
---END BASE---
|
||||
|
||||
只返回 JSON:
|
||||
{{
|
||||
"title": "实验标题",
|
||||
"summary": "问题总结,引用聚合数字,不泄露账号或工单正文",
|
||||
"hypothesis": "为什么这一个改动会降低目标指标",
|
||||
"method": "插入什么纪律、放在哪里、真机复跑时看什么",
|
||||
"targetMetric": "上面列表中允许的一个",
|
||||
"afterLine": "现行模板中逐字且唯一的一整行,或 null",
|
||||
"content": "要插入的新行",
|
||||
"scope": "platform 或 both",
|
||||
"scopeReason": "一句话:为什么这条规则只对这个平台 / 对两端都成立"
|
||||
}}
|
||||
content 要写可执行决策规则(何时、先做什么、失败后怎么换路),不要写空泛口号,
|
||||
不要修改评分规则、保留集、指标定义或要求模型自评成功。"""
|
||||
|
||||
|
||||
def _redact(text: str) -> str:
|
||||
text = re.sub(r"[\w.+-]+@[\w.-]+\.[A-Za-z]{2,}", "<email>", text)
|
||||
text = re.sub(r"(?i)\b(password|passwd|token|cookie|authorization)\s*[:=]\s*\S+", r"\1=<redacted>", text)
|
||||
text = re.sub(r"\b[0-9a-f]{24,}\b", "<id>", text, flags=re.I)
|
||||
return text[:1200]
|
||||
|
||||
|
||||
def create(cluster: dict, base: str) -> dict:
|
||||
if cluster.get("stage") not in STAGES:
|
||||
raise ValueError("不支持的 stage")
|
||||
value: Any = llm.ask_json(_prompt(cluster, base), timeout=1200)
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("提案不是 JSON 对象")
|
||||
required = ("title", "summary", "hypothesis", "method", "targetMetric", "content")
|
||||
if any(not str(value.get(k) or "").strip() for k in required):
|
||||
raise ValueError("提案缺少必填字段")
|
||||
metric = str(value["targetMetric"])
|
||||
if metric not in TARGET_METRICS:
|
||||
raise ValueError("targetMetric 不在目录里")
|
||||
allowed = METRICS_BY_REPLAY.get(cluster.get("replayType"), TARGET_METRICS)
|
||||
if metric not in allowed:
|
||||
raise ValueError("targetMetric 与该模式的离线动作判分不对应")
|
||||
after = value.get("afterLine")
|
||||
if after is not None and not isinstance(after, str):
|
||||
raise ValueError("afterLine 格式错误")
|
||||
candidate = apply_insertion(base, str(value["content"]), after)
|
||||
error = candidate_error(base, candidate) or gaming_error(metric, base, candidate)
|
||||
if error:
|
||||
raise ValueError(error)
|
||||
return {
|
||||
"stage": cluster["stage"],
|
||||
"platform": cluster.get("platform"),
|
||||
"title": str(value["title"]).strip()[:200],
|
||||
"summary": str(value["summary"]).strip()[:6000],
|
||||
"hypothesis": str(value["hypothesis"]).strip()[:4000],
|
||||
"method": str(value["method"]).strip()[:4000],
|
||||
"targetMetric": metric,
|
||||
"problemClass": cluster["problemClass"],
|
||||
"cluster": cluster["cluster"],
|
||||
"candidateTemplate": candidate,
|
||||
# 策略范围:both = 两端各复跑;其余一律单端(控制面按 platform 处理)
|
||||
"scope": "both" if str(value.get("scope") or "").strip() == "both" else cluster.get("platform"),
|
||||
"scopeReason": str(value.get("scopeReason") or "").strip()[:1000],
|
||||
# 重放按同一个锚点把同样的几行插进历史 prompt,与生产渲染的位置一致(不提交给控制面)
|
||||
"_insertion": {"afterLine": after, "content": inserted_text(base, candidate)},
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
"""决策点重放:把一段历史会话截到某个工具结果之后,用本机 claude 续跑一步,
|
||||
只记录模型「想做什么」(第一条 assistant 消息里的工具调用 / 文字),工具一律被
|
||||
PreToolUse 钩子拦下,不在本机执行。
|
||||
|
||||
保真度的三件事(2026-09 实验里踩出来的):
|
||||
- 不能截在工具调用处,否则本机会补一条「调用被中断」的假结果;只截在工具结果之后;
|
||||
- 续跑会追加「环境 / 日期 / 工具列表都变了」的附件,模型第一反应是去查环境 —— 所以把
|
||||
这些附件改写成本机值,节点的 CLAUDE.md / MEMORY.md 落到本机对应路径;
|
||||
- 本机原样起节点的 actl 设备工具 MCP,只为让工具定义一致(调用全被钩子拦)。
|
||||
"""
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from . import config
|
||||
|
||||
HOME = Path.home()
|
||||
# 续跑的工作目录:<临时目录>/sl-ws/<会话id>/ai_workspace。模型看得见这个路径 —— 带着
|
||||
# rsi_loop 之类的字样,它第一反应就是去查「这是什么」;叫 ai_workspace 才和节点上一致。
|
||||
_REPLAY_BASE = Path(tempfile.gettempdir()).resolve() / "sl-ws"
|
||||
|
||||
|
||||
def _run_dirs(sid: str):
|
||||
cwd = _REPLAY_BASE / sid / "ai_workspace"
|
||||
proj = HOME / ".claude" / "projects" / re.sub(r"[^A-Za-z0-9]", "-", str(cwd))
|
||||
return cwd, proj
|
||||
|
||||
|
||||
def _claude_cmd(sid: str, model: str) -> List[str]:
|
||||
cmd = [config.CLAUDE_BIN, "-p", "继续", "--resume", sid, "--output-format", "stream-json", "--verbose",
|
||||
"--model", model, "--setting-sources", "project"]
|
||||
mcp = _mcp_config()
|
||||
if mcp:
|
||||
cmd += ["--mcp-config", mcp]
|
||||
return cmd + ["--strict-mcp-config", "--settings", BLOCK_ALL, "--append-system-prompt", AUTOMODE, "--max-turns", "1"]
|
||||
|
||||
AUTOMODE = ("本会话运行在无人值守的自动化流水线中,没有任何人类在观看或会作出回应。"
|
||||
"绝对禁止使用 AskUserQuestion 工具或以提问等待用户回复的方式中断任务;"
|
||||
"如必须在多个方案中取舍,直接采用你推荐的(或第一个可行的)方案并继续执行,事后在输出中说明即可。"
|
||||
"如需计划确认(ExitPlanMode),直接按你的计划执行。任务必须自主推进到完成。")
|
||||
BLOCK_ALL = json.dumps({"hooks": {"PreToolUse": [{"matcher": "*", "hooks": [
|
||||
{"type": "command", "command": "echo 'replay: tool execution disabled' >&2; exit 2"}]}]}})
|
||||
|
||||
|
||||
def _mcp_config() -> Optional[str]:
|
||||
return json.dumps({"mcpServers": {"actl": {"type": "stdio", "command": "node", "args": [config.ACTL_INDEX]}}}) \
|
||||
if Path(config.ACTL_INDEX).exists() else None
|
||||
|
||||
|
||||
_LOCAL: Optional[dict] = None
|
||||
|
||||
|
||||
def _probe(sample_rows: List[Optional[dict]]) -> dict:
|
||||
"""拿一段真实执行会话的开头(截到第一个工具结果)在本机续跑一步,收集本机追加的
|
||||
「和节点不一样」的附件 —— 它们就是中和时要换上的本机值。空会话探不出任何东西:
|
||||
Claude Code 不认残缺的会话记录,也就不会追加附件。"""
|
||||
cut = next((i for i, r in enumerate(sample_rows) if r and r.get("type") == "user"
|
||||
and isinstance((r.get("message") or {}).get("content"), list)
|
||||
and any(isinstance(b, dict) and b.get("type") == "tool_result" for b in r["message"]["content"])), None)
|
||||
if cut is None:
|
||||
raise RuntimeError("探测用的会话里没有工具结果,换一份会话")
|
||||
sid = str(uuid.uuid4())
|
||||
cwd, proj = _run_dirs(sid)
|
||||
cwd.mkdir(parents=True, exist_ok=True)
|
||||
proj.mkdir(parents=True, exist_ok=True)
|
||||
prefix = []
|
||||
for r in sample_rows[: cut + 1]:
|
||||
if not r:
|
||||
continue
|
||||
r = copy.deepcopy(r)
|
||||
r["sessionId"] = sid
|
||||
if "cwd" in r:
|
||||
r["cwd"] = str(cwd)
|
||||
prefix.append(r)
|
||||
path = proj / f"{sid}.jsonl"
|
||||
path.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in prefix) + "\n")
|
||||
env = config.child_env(str(cwd))
|
||||
loc: dict = {}
|
||||
try:
|
||||
subprocess.run(_claude_cmd(sid, config.REPLAY_MODEL), cwd=str(cwd), env=env,
|
||||
capture_output=True, text=True, timeout=600)
|
||||
for line in path.read_text(errors="replace").splitlines()[len(prefix):]:
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
a = r.get("attachment")
|
||||
if r.get("type") != "attachment" or not a:
|
||||
continue
|
||||
t = a.get("type")
|
||||
if t in ("environment", "session_context", "date", "credential_org", "remote_session_change", "auto_mode"):
|
||||
loc[t] = {k: v for k, v in a.items() if k != "changed"}
|
||||
elif t == "deferred_tools_delta":
|
||||
loc["deferred_removed"] = a.get("removedNames", [])
|
||||
elif t == "agent_listing_delta":
|
||||
loc["agent_removed"] = a.get("removedTypes", [])
|
||||
finally:
|
||||
shutil.rmtree(proj, ignore_errors=True)
|
||||
shutil.rmtree(cwd.parent, ignore_errors=True)
|
||||
if "environment" not in loc:
|
||||
raise RuntimeError("探测失败:本机续跑没有追加环境附件(claude 鉴权、模型名或会话格式有问题),不能做重放")
|
||||
return loc
|
||||
|
||||
|
||||
def _local_values(sample_rows: Optional[List[Optional[dict]]] = None) -> dict:
|
||||
"""本机该用的附件值。每轮运行先用一份真实会话探测一次(prepare),之后并发重放复用。"""
|
||||
global _LOCAL
|
||||
if _LOCAL is not None:
|
||||
return _LOCAL
|
||||
if sample_rows is None:
|
||||
raise RuntimeError("重放前必须先 prepare(sample_rows) 探测本机附件")
|
||||
_LOCAL = _probe(sample_rows)
|
||||
cache = config.DATA / "local_attachments.json"
|
||||
cache.write_text(json.dumps(_LOCAL, ensure_ascii=False))
|
||||
os.chmod(cache, 0o600)
|
||||
return _LOCAL
|
||||
|
||||
|
||||
class AnchorMissing(ValueError):
|
||||
"""候选的插入位置在这份历史 prompt 里找不到(或不唯一):这个点不能公平地比较两个版本。"""
|
||||
|
||||
|
||||
def _anchor_regex(line: str) -> "re.Pattern[str]":
|
||||
# 锚点行可能带占位符;渲染后的历史 prompt 里那部分是具体值,按通配匹配。
|
||||
parts = re.split(r"\{\{\s*[\w.]+\s*\}\}", line)
|
||||
return re.compile("^" + ".*?".join(re.escape(p) for p in parts) + "$")
|
||||
|
||||
|
||||
def insert_into_text(text: str, insertion: dict) -> str:
|
||||
"""把候选新增的行插到与生产相同的位置:锚点行之后;锚点为 null 表示模板末尾。
|
||||
不加任何标题 —— 生产上这段内容就是模板里普通的几行,离线时更醒目会高估候选。"""
|
||||
content = str(insertion.get("content") or "").strip("\n")
|
||||
if not content:
|
||||
return text
|
||||
lines = text.split("\n")
|
||||
after = insertion.get("afterLine")
|
||||
if after is None:
|
||||
return "\n".join(lines + content.split("\n"))
|
||||
rx = _anchor_regex(str(after))
|
||||
hits = [i for i, line in enumerate(lines) if rx.match(line)]
|
||||
if len(hits) != 1:
|
||||
raise AnchorMissing(f"锚点在历史 prompt 里匹配到 {len(hits)} 处")
|
||||
at = hits[0] + 1
|
||||
lines[at:at] = content.split("\n")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _insert_into_content(content: Any, insertion: Optional[dict]) -> Any:
|
||||
if not insertion:
|
||||
return content
|
||||
if isinstance(content, str):
|
||||
return insert_into_text(content, insertion)
|
||||
if isinstance(content, list):
|
||||
out = copy.deepcopy(content)
|
||||
for block in out:
|
||||
if isinstance(block, dict) and block.get("type") == "text":
|
||||
block["text"] = insert_into_text(str(block.get("text") or ""), insertion)
|
||||
return out
|
||||
raise AnchorMissing("历史会话第一条用户消息没有文本")
|
||||
|
||||
|
||||
def candidate_applies(rows: List[Optional[dict]], insertion: Optional[dict]) -> bool:
|
||||
"""这个决策点能否施加候选改动(锚点唯一可定位)。不能的点两组都不跑。"""
|
||||
if not insertion:
|
||||
return True
|
||||
for r in rows:
|
||||
if r and r.get("type") == "user" and not r.get("isMeta"):
|
||||
try:
|
||||
_insert_into_content((r.get("message") or {}).get("content"), insertion)
|
||||
return True
|
||||
except AnchorMissing:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _neutralize(rows: List[Optional[dict]], sid: str, loc: dict, cwd: Path, proj: Path,
|
||||
insertion: Optional[dict] = None):
|
||||
files: Dict[str, str] = {}
|
||||
out: List[dict] = []
|
||||
prompt_patched = False
|
||||
for r in rows:
|
||||
if not r:
|
||||
continue
|
||||
r = copy.deepcopy(r)
|
||||
r["sessionId"] = sid
|
||||
if "cwd" in r:
|
||||
r["cwd"] = str(cwd)
|
||||
if r.get("type") == "user" and not r.get("isMeta") and not prompt_patched:
|
||||
message = r.get("message") or {}
|
||||
if "content" in message:
|
||||
message["content"] = _insert_into_content(message["content"], insertion)
|
||||
r["message"] = message
|
||||
prompt_patched = True
|
||||
a = r.get("attachment") if r.get("type") == "attachment" else None
|
||||
if a:
|
||||
t = a.get("type")
|
||||
if t == "environment" and "environment" in loc:
|
||||
# 探测时记下的是探测目录;每次重放用独立目录,不换掉的话续跑会追加一条
|
||||
# 「工作目录变了」,模型第一反应就是去查环境。
|
||||
a["snapshot"] = {**loc["environment"]["snapshot"], "workingDirectory": str(cwd)}
|
||||
a.pop("changes", None)
|
||||
elif t in ("session_context", "date", "remote_session_change", "auto_mode") and t in loc:
|
||||
r["attachment"] = copy.deepcopy(loc[t])
|
||||
elif t == "deferred_tools_delta":
|
||||
drop = set(loc.get("deferred_removed") or [])
|
||||
for k in ("addedNames", "readdedNames"):
|
||||
if isinstance(a.get(k), list):
|
||||
a[k] = [n for n in a[k] if n not in drop]
|
||||
if isinstance(a.get("addedLines"), list):
|
||||
a["addedLines"] = [l for l in a["addedLines"] if not any(l.strip().startswith(n) for n in drop)]
|
||||
elif t == "agent_listing_delta":
|
||||
drop = set(loc.get("agent_removed") or [])
|
||||
if isinstance(a.get("addedTypes"), list):
|
||||
a["addedTypes"] = [n for n in a["addedTypes"] if n not in drop]
|
||||
if isinstance(a.get("addedLines"), list):
|
||||
a["addedLines"] = [l for l in a["addedLines"] if not any(l.strip().lstrip("- ").startswith(n + ":") for n in drop)]
|
||||
elif t == "instructions":
|
||||
for fobj in a.get("files") or []:
|
||||
p = fobj.get("path", "")
|
||||
if p.endswith("/ai_workspace/CLAUDE.md") or re.search(r"ai_workspace(_\w+)?/CLAUDE\.md$", p):
|
||||
np = str(cwd / "CLAUDE.md")
|
||||
elif p.endswith("/memory/MEMORY.md"):
|
||||
np = str(proj / "memory" / "MEMORY.md")
|
||||
else:
|
||||
continue
|
||||
files[np] = fobj.get("content", "")
|
||||
fobj["path"] = np
|
||||
out.append(r)
|
||||
have = {r["attachment"].get("type") for r in out if r.get("type") == "attachment" and r.get("attachment")}
|
||||
for t in ("credential_org", "session_context", "remote_session_change"):
|
||||
if t in loc and t not in have:
|
||||
idx = next((k for k, r in enumerate(out) if r.get("type") == "attachment" and r["attachment"].get("type") == "date"), None)
|
||||
if idx is not None:
|
||||
tmpl = copy.deepcopy(out[idx])
|
||||
tmpl["attachment"] = copy.deepcopy(loc[t])
|
||||
tmpl["uuid"] = str(uuid.uuid4())
|
||||
out.insert(idx + 1, tmpl)
|
||||
return out, files
|
||||
|
||||
|
||||
def prepare(sample_rows: List[Optional[dict]]) -> None:
|
||||
"""串行探测一次本机附件(每轮重新探,不跨轮缓存:日期、工具列表会变),之后可安全并发重放。"""
|
||||
global _LOCAL
|
||||
_LOCAL = None
|
||||
_local_values(sample_rows)
|
||||
|
||||
|
||||
def run_point(rows: List[Optional[dict]], row_idx: int, insertion: Optional[dict] = None, timeout: int = 900) -> dict:
|
||||
"""截到 rows[row_idx](含)之后续跑一步。返回 {first:[...], secs, error}。first 是模型想做的事。"""
|
||||
sid = str(uuid.uuid4())
|
||||
cwd, proj = _run_dirs(sid)
|
||||
cwd.mkdir(parents=True, exist_ok=True)
|
||||
prefix, files = _neutralize(rows[: row_idx + 1], sid, _local_values(), cwd, proj, insertion)
|
||||
proj.mkdir(parents=True, exist_ok=True)
|
||||
for p, c in files.items():
|
||||
Path(p).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(p).write_text(c)
|
||||
(proj / f"{sid}.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in prefix) + "\n")
|
||||
env = config.child_env(str(cwd))
|
||||
cmd = _claude_cmd(sid, config.REPLAY_MODEL)
|
||||
t0 = time.time()
|
||||
first: Optional[List[dict]] = None
|
||||
error: Optional[str] = None
|
||||
try:
|
||||
p = subprocess.run(cmd, cwd=str(cwd), env=env, capture_output=True, text=True, timeout=timeout)
|
||||
first, error = first_turn(p.stdout.splitlines())
|
||||
if not first and not error and p.returncode:
|
||||
error = (p.stderr or f"exit {p.returncode}")[-500:]
|
||||
except subprocess.TimeoutExpired:
|
||||
error = "timeout"
|
||||
except Exception as e: # noqa: BLE001
|
||||
error = str(e)
|
||||
finally:
|
||||
shutil.rmtree(proj, ignore_errors=True)
|
||||
shutil.rmtree(cwd.parent, ignore_errors=True)
|
||||
return {"first": first, "secs": round(time.time() - t0, 1), "error": None if first else (error or "no output")}
|
||||
|
||||
|
||||
def first_turn(lines: List[str]):
|
||||
"""从 stream-json 输出里取第一轮的内容块(文字 + 工具调用),到第一个工具调用为止。
|
||||
|
||||
流式输出会把同一轮的「说明文字」和「工具调用」拆成几条消息(常见:先一句话、再调工具)。
|
||||
读到文字就停,会把这一轮记成「没有动作」—— 判分里那等于坏选择,两组都被冤枉。"""
|
||||
first: Optional[List[dict]] = None
|
||||
for line in lines:
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
if r.get("type") == "assistant":
|
||||
acts = [b for b in (r.get("message") or {}).get("content") or []
|
||||
if isinstance(b, dict) and b.get("type") in ("tool_use", "text")]
|
||||
if acts:
|
||||
first = (first or []) + acts
|
||||
if any(b.get("type") == "tool_use" for b in acts):
|
||||
return first, None
|
||||
if r.get("type") == "result":
|
||||
return first, (str(r.get("result") or r.get("subtype")) if r.get("is_error") else None)
|
||||
return first, None
|
||||
|
||||
|
||||
def next_action(rows: List[Optional[dict]], row_idx: int) -> Optional[dict]:
|
||||
"""这个点之后历史上真实的下一步工具调用(用于对照)。"""
|
||||
for r in rows[row_idx + 1:]:
|
||||
if r and r.get("type") == "assistant":
|
||||
for b in (r["message"].get("content") or []):
|
||||
if isinstance(b, dict) and b.get("type") == "tool_use":
|
||||
return {"name": b["name"], "input": b.get("input") or {}}
|
||||
return None
|
||||
@@ -0,0 +1,135 @@
|
||||
"""复核意见挖掘:每晚给新的人工复核意见打标签,每周把编码习惯归纳成候选规范。
|
||||
|
||||
口径、门槛、为什么这么做见 shared/src/review_mining.ts。这里只做「调模型」那一半:
|
||||
- 提示词正文由控制面拼好下发(/review-mining/pending、/review-mining/cluster-input),这里不改一个字;
|
||||
- 模型的回答原样交回,key 校验、归属、统计、门槛都由控制面重算 —— 不信这里报的任何数;
|
||||
- 能写的只有标签和 pending 候选,批准 / 驳回要管理员在 RSI 页上点。
|
||||
|
||||
单独跑(补标历史、或者想立刻归纳一次):
|
||||
./run.sh review-mining --days 180 # 补标近 180 天,每次最多 400 条,多跑几次
|
||||
./run.sh review-mining --cluster # 不管是不是周一,立刻归纳一次
|
||||
"""
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from . import api, config, llm
|
||||
|
||||
LABELER = "rsi-loop/review-label@1"
|
||||
LAST_CLUSTER = config.DATA / "review_mining_last_cluster"
|
||||
# 与 shared/src/review_mining.ts 的 MINING.clusterWeekday 一致(北京时间周一)
|
||||
CLUSTER_WEEKDAY = 0 # Python 的 weekday():周一 = 0
|
||||
BEIJING = timezone(timedelta(hours=8))
|
||||
|
||||
|
||||
def _beijing_day(now_ms: int) -> datetime:
|
||||
return datetime.fromtimestamp(now_ms / 1000, BEIJING)
|
||||
|
||||
|
||||
def label_pending(days: Optional[int] = None, max_batches: Optional[int] = None) -> dict:
|
||||
"""拉没标过的意见,按批调模型,逐批交回。一批失败不影响下一批(下次运行还会再拉到它)。"""
|
||||
q = "/review-mining/pending" + (f"?days={int(days)}" if days else "")
|
||||
data = api.get(q, timeout=300)
|
||||
batches = data.get("batches") or []
|
||||
if max_batches is not None:
|
||||
batches = batches[:max_batches]
|
||||
saved = dropped = failed = 0
|
||||
for i, b in enumerate(batches):
|
||||
try:
|
||||
out = llm.ask_json(b["prompt"], timeout=900)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
failed += 1
|
||||
config.log(f"复核意见打标签 第 {i + 1}/{len(batches)} 批失败:{str(exc)[:200]}")
|
||||
continue
|
||||
labels = out.get("labels") if isinstance(out, dict) else None
|
||||
if not isinstance(labels, list):
|
||||
failed += 1
|
||||
continue
|
||||
res = api.post("/review-mining/labels", {"labels": labels, "labeler": LABELER}) or {}
|
||||
saved += int(res.get("saved") or 0)
|
||||
dropped += int(res.get("dropped") or 0)
|
||||
return {"pending": int(data.get("total") or 0), "batches": len(batches), "saved": saved, "dropped": dropped, "failedBatches": failed}
|
||||
|
||||
|
||||
def _cluster_due(now_ms: int) -> bool:
|
||||
day = _beijing_day(now_ms)
|
||||
if day.weekday() != CLUSTER_WEEKDAY:
|
||||
return False
|
||||
try:
|
||||
return LAST_CLUSTER.read_text().strip() != day.strftime("%Y-%m-%d")
|
||||
except FileNotFoundError:
|
||||
return True
|
||||
|
||||
|
||||
def cluster(now_ms: int, force: bool = False) -> Optional[dict]:
|
||||
"""每周一次:拉近 90 天的编码习惯标签和拼好的提示词,交给模型归纳,候选交回控制面过门槛。"""
|
||||
if not force and not _cluster_due(now_ms):
|
||||
return None
|
||||
inp = api.get("/review-mining/cluster-input", timeout=300)
|
||||
result = {"rows": int(inp.get("rows") or 0), "proposed": 0, "saved": 0, "rejected": []}
|
||||
if result["rows"]:
|
||||
out = llm.ask_json(inp["prompt"], timeout=1800)
|
||||
cands = out.get("candidates") if isinstance(out, dict) else None
|
||||
if not isinstance(cands, list):
|
||||
raise RuntimeError("归纳的回答里没有 candidates 数组")
|
||||
result["proposed"] = len(cands)
|
||||
res = api.post("/review-mining/candidates", {
|
||||
"runId": f"loop-{_beijing_day(now_ms).strftime('%Y%m%d')}",
|
||||
"window": inp.get("window"),
|
||||
"candidates": cands,
|
||||
}) or {}
|
||||
result["saved"] = int(res.get("saved") or 0)
|
||||
result["rejected"] = res.get("rejected") or []
|
||||
if not config.DRY_RUN:
|
||||
LAST_CLUSTER.write_text(_beijing_day(now_ms).strftime("%Y-%m-%d"))
|
||||
return result
|
||||
|
||||
|
||||
def run(now_ms: Optional[int] = None) -> dict:
|
||||
"""每日回路里的一步。两半各自兜底:挖掘失败不能拖垮回路的实验部分。"""
|
||||
now_ms = now_ms or int(time.time() * 1000)
|
||||
out: dict = {}
|
||||
try:
|
||||
out["label"] = label_pending()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
out["label"] = {"error": str(exc)[:300]}
|
||||
try:
|
||||
out["cluster"] = cluster(now_ms)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
out["cluster"] = {"error": str(exc)[:300]}
|
||||
return out
|
||||
|
||||
|
||||
def summary_line(out: dict) -> str:
|
||||
"""给回路时间线的一句话。"""
|
||||
lab = out.get("label") or {}
|
||||
parts = []
|
||||
if lab.get("error"):
|
||||
parts.append(f"打标签失败:{lab['error'][:80]}")
|
||||
else:
|
||||
parts.append(f"新标 {lab.get('saved', 0)} 条意见(待标 {lab.get('pending', 0)})")
|
||||
cl = out.get("cluster")
|
||||
if cl:
|
||||
if cl.get("error"):
|
||||
parts.append(f"归纳失败:{cl['error'][:80]}")
|
||||
else:
|
||||
parts.append(f"归纳出 {cl.get('saved', 0)} 条候选规范(模型提了 {cl.get('proposed', 0)},门槛拦下 {len(cl.get('rejected') or [])})")
|
||||
return ";".join(parts)
|
||||
|
||||
|
||||
def main(argv: Optional[list] = None) -> dict:
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(prog="rsi_loop review-mining")
|
||||
p.add_argument("--days", type=int, default=None, help="待标意见的回看天数(补历史时用)")
|
||||
p.add_argument("--cluster", action="store_true", help="不管是不是周一,立刻归纳一次")
|
||||
p.add_argument("--no-label", action="store_true", help="跳过打标签")
|
||||
a = p.parse_args(argv)
|
||||
now_ms = int(time.time() * 1000)
|
||||
out: dict = {}
|
||||
if not a.no_label:
|
||||
out["label"] = label_pending(days=a.days)
|
||||
out["cluster"] = cluster(now_ms, force=a.cluster)
|
||||
if any(out.values()):
|
||||
api.event("复核意见挖掘(手动)", json.dumps(out, ensure_ascii=False, indent=2), "note")
|
||||
return out
|
||||
@@ -0,0 +1,359 @@
|
||||
"""RSI 每日回路:同步 → 切片 → 审计 → 提案 → 配对重放 → 提交 → 补结论。"""
|
||||
import concurrent.futures
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from . import api, config, llm, mine, proposal, replay, review_mining, runtime, scoring
|
||||
|
||||
TERMINAL = {"offline_failed", "lab_passed", "lab_failed", "declined", "stopped"}
|
||||
|
||||
|
||||
def _write_jsonl(path: Path, rows: List[dict]) -> None:
|
||||
with open(path, "w") as f:
|
||||
for row in rows:
|
||||
f.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
os.chmod(path, 0o600)
|
||||
|
||||
|
||||
def _local_skips() -> dict:
|
||||
try:
|
||||
return json.loads((config.DATA / "skips.json").read_text())
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
|
||||
def _skip_key(cluster: dict) -> str:
|
||||
return f"{cluster.get('stage')}|{cluster.get('platform')}|{cluster.get('cluster')}"
|
||||
|
||||
|
||||
def _record_skip(cluster: dict, reason: str, now_ms: int) -> None:
|
||||
if config.DRY_RUN:
|
||||
return # 试运行不留下任何会影响正式运行的状态(否则正式跑时这个模式会被冷却 7 天)
|
||||
skips = _local_skips()
|
||||
skips[_skip_key(cluster)] = {"at": now_ms, "reason": reason}
|
||||
path = config.DATA / "skips.json"
|
||||
path.write_text(json.dumps(skips, ensure_ascii=False, indent=2))
|
||||
os.chmod(path, 0o600)
|
||||
|
||||
|
||||
def _submittable(cluster: dict, state: dict) -> bool:
|
||||
"""实验只在实验机上复跑历史任务:只有 bug.run、且该平台配了实验机才收(控制面同样会拒)。"""
|
||||
return cluster.get("stage") == "bug.run" and cluster.get("platform") in (state.get("labNodes") or {})
|
||||
|
||||
|
||||
def _eligible_cluster(all_clusters: List[dict], state: dict, now_ms: int,
|
||||
point_count=lambda c: len(mine.replay_points(c))) -> Optional[dict]:
|
||||
# 同一 stage 在同一平台上同时只能有一个进行中的实验;platform 为空的旧实验对所有平台生效
|
||||
live = {(e.get("stage"), e.get("platform")) for e in state.get("live") or []}
|
||||
cutoff = now_ms - config.CLUSTER_COOLDOWN_DAYS * 86_400_000
|
||||
recent = {
|
||||
(e.get("stage"), e.get("platform"), e.get("cluster"))
|
||||
for e in state.get("recent") or []
|
||||
if int(e.get("createdAt") or 0) >= cutoff
|
||||
}
|
||||
skips = _local_skips()
|
||||
for cluster in all_clusters:
|
||||
if cluster.get("replayType") == "none" or not _submittable(cluster, state):
|
||||
continue
|
||||
stage, platform = cluster.get("stage"), cluster.get("platform")
|
||||
if (stage, platform) in live or (stage, None) in live:
|
||||
continue
|
||||
if (stage, platform, cluster.get("cluster")) in recent:
|
||||
continue
|
||||
skipped = skips.get(_skip_key(cluster)) or {}
|
||||
if int(skipped.get("at") or 0) >= cutoff:
|
||||
continue
|
||||
# 决策点不够就不提案:注定过不了离线闸,只会白花额度、再占一次冷却期
|
||||
if point_count(cluster) < config.OFFLINE_MIN_POINTS:
|
||||
continue
|
||||
return cluster
|
||||
return None
|
||||
|
||||
|
||||
def _sample_points(cluster: dict, insertion: Optional[dict] = None) -> tuple:
|
||||
applies = lambda p: replay.candidate_applies(mine.load_rows_for_point(p), insertion) # noqa: E731
|
||||
all_target = mine.replay_points(cluster)
|
||||
target = [p for p in all_target if applies(p)]
|
||||
target_ids = {p["id"] for p in all_target}
|
||||
real_eps = mine.real_episode_ids(cluster["stage"])
|
||||
holdout_all = [
|
||||
p for p in mine.replay_points()
|
||||
if p["id"] not in target_ids and p.get("stage") == cluster["stage"]
|
||||
and (not cluster.get("platform") or p.get("platform") == cluster.get("platform"))
|
||||
and (p.get("ptype") == "T2_wait" or (p.get("ptype") in ("T1_repeat", "T3_frida") and p.get("episodeId") in real_eps))
|
||||
]
|
||||
holdout = [p for p in holdout_all if applies(p)]
|
||||
seed = int.from_bytes(str(cluster["cluster"]).encode()[:8].ljust(8, b"\0"), "little")
|
||||
rng = random.Random(seed)
|
||||
rng.shuffle(target)
|
||||
rng.shuffle(holdout)
|
||||
seeds = max(2, config.SEEDS)
|
||||
capacity = max(0, config.MAX_REPLAYS_PER_RUN // (2 * seeds))
|
||||
n_target = min(config.TARGET_POINTS, len(target), capacity)
|
||||
remaining = max(0, capacity - n_target)
|
||||
n_holdout = min(config.HOLDOUT_POINTS, len(holdout), remaining)
|
||||
skipped = {"targetAnchorMissing": len(all_target) - len(target), "holdoutAnchorMissing": len(holdout_all) - len(holdout)}
|
||||
return target[:n_target], holdout[:n_holdout], seeds, skipped
|
||||
|
||||
|
||||
def _one_replay(point: dict, arm: str, seed: int, insertion: Optional[dict], rows: list) -> dict:
|
||||
result = replay.run_point(
|
||||
rows, int(point["point_row"]),
|
||||
insertion=insertion if arm == "candidate" else None,
|
||||
)
|
||||
action = scoring.action_of(result)
|
||||
cls = "no_action" if action is None else scoring.classify_action(point["ptype"], point, action)
|
||||
return {
|
||||
"pointId": point["id"], "episodeId": point.get("episodeId"),
|
||||
"stage": point.get("stage"), "ptype": point.get("ptype"),
|
||||
"arm": arm, "seed": seed, "classification": cls,
|
||||
"action": action, "secs": result.get("secs"), "error": result.get("error"),
|
||||
}
|
||||
|
||||
|
||||
class ReplayAborted(RuntimeError):
|
||||
"""重放没法给出可信结论(点不够 / 基础设施报错太多):不提交实验,只记运行日志。"""
|
||||
|
||||
|
||||
MAX_ERROR_RATE = 0.2
|
||||
|
||||
|
||||
def run_replays(cluster: dict, insertion: Optional[dict]) -> dict:
|
||||
target, holdout, seeds, skipped = _sample_points(cluster, insertion)
|
||||
if len(target) < config.OFFLINE_MIN_POINTS:
|
||||
raise ReplayAborted(
|
||||
f"候选的插入位置只在 {len(target)} 个目标决策点上能定位(需要 {config.OFFLINE_MIN_POINTS}),"
|
||||
f"{skipped['targetAnchorMissing']} 个点锚点缺失")
|
||||
tagged = [("target", p) for p in target] + [("holdout", p) for p in holdout]
|
||||
rows_by_id: Dict[str, list] = {p["id"]: mine.load_rows_for_point(p) for _, p in tagged}
|
||||
jobs = [
|
||||
(group, point, arm, seed)
|
||||
for group, point in tagged
|
||||
for arm in ("base", "candidate")
|
||||
for seed in range(seeds)
|
||||
]
|
||||
records: List[dict] = []
|
||||
if jobs:
|
||||
replay.prepare(rows_by_id[jobs[0][1]["id"]])
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=max(1, config.WORKERS)) as pool:
|
||||
future_map = {
|
||||
pool.submit(_one_replay, point, arm, seed, insertion, rows_by_id[point["id"]]):
|
||||
(group, point["id"])
|
||||
for group, point, arm, seed in jobs
|
||||
}
|
||||
config.log(f"重放:{len(target)} 个目标点 + {len(holdout)} 个保留集点,每点 2 组 × {seeds} 遍,共 {len(jobs)} 次")
|
||||
for future in concurrent.futures.as_completed(future_map):
|
||||
group, point_id = future_map[future]
|
||||
try:
|
||||
records.append({"group": group, **future.result()})
|
||||
except Exception as exc: # noqa: BLE001
|
||||
records.append({
|
||||
"group": group, "pointId": point_id, "arm": "unknown", "seed": None,
|
||||
"classification": "error", "error": str(exc)[:1000],
|
||||
})
|
||||
config.log(f" 重放 {len(records)}/{len(jobs)}")
|
||||
run_id = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
run_dir = config.RUNS / run_id
|
||||
run_dir.mkdir(parents=True, exist_ok=False)
|
||||
run_dir.chmod(0o700)
|
||||
_write_jsonl(run_dir / "results.jsonl", records)
|
||||
errors = sum(1 for r in records if r.get("error"))
|
||||
if records and errors / len(records) > MAX_ERROR_RATE:
|
||||
raise ReplayAborted(f"重放报错 {errors}/{len(records)},超过 {int(MAX_ERROR_RATE * 100)}%,"
|
||||
f"不能把基础设施故障当成候选无效;原始结果见 {run_dir}")
|
||||
target_result = scoring.offline_summary(r for r in records if r.get("group") == "target")
|
||||
holdout_result = scoring.offline_summary(r for r in records if r.get("group") == "holdout")
|
||||
offline = {
|
||||
"target": target_result,
|
||||
"holdout": holdout_result,
|
||||
"meta": {
|
||||
"runId": run_id, "model": config.REPLAY_MODEL,
|
||||
"machine": os.uname().nodename, "calls": len(records),
|
||||
"errors": errors, **skipped,
|
||||
"scoring": "first-action-shape; paired bootstrap over decision points",
|
||||
"candidateApplication": "inserted at the same anchor line as in the template; points where the anchor is missing are excluded from both arms",
|
||||
},
|
||||
}
|
||||
(run_dir / "offline.json").write_text(json.dumps(offline, ensure_ascii=False, indent=2))
|
||||
os.chmod(run_dir / "offline.json", 0o600)
|
||||
return offline
|
||||
|
||||
|
||||
def _conclusion_prompt(exp: dict) -> str:
|
||||
payload = {
|
||||
"status": exp.get("status"), "title": exp.get("title"), "stage": exp.get("stage"),
|
||||
"summary": exp.get("summary"), "hypothesis": exp.get("hypothesis"),
|
||||
"method": exp.get("method"), "targetMetric": exp.get("targetMetric"),
|
||||
"offlineResult": exp.get("offlineResult"), "onlineResult": exp.get("onlineResult"),
|
||||
"decisionReason": exp.get("decisionReason"),
|
||||
}
|
||||
return f"""根据下面这条自动实验的确定性记录写实验结论。不得引入记录外的数字,
|
||||
不得把模型意见当真值;明确说假设是否得到支持、离线与真机复跑各发生了什么、下一轮该保留
|
||||
什么或避免什么。状态如果是离线未通过,就只下离线结论;lab_passed / lab_failed 是
|
||||
真机复跑的结论(评估集历史任务新旧 prompt 各跑一遍,onlineResult 里是配对结果;
|
||||
「自报修好」是 agent 自己说的,不能当成功证据)。两到四段中文纯文本。
|
||||
|
||||
{json.dumps(payload, ensure_ascii=False, indent=2)}"""
|
||||
|
||||
|
||||
def write_missing_conclusions(experiments: List[dict]) -> int:
|
||||
written = 0
|
||||
for exp in experiments:
|
||||
if exp.get("status") not in TERMINAL or exp.get("conclusion"):
|
||||
continue
|
||||
try:
|
||||
body = llm.ask(_conclusion_prompt(exp), timeout=900)
|
||||
posted = api.experiment_event(
|
||||
str(exp["id"]), f"实验结论:{exp.get('title')}", body[:12000], "conclusion",
|
||||
)
|
||||
written += int(posted)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"[rsi-loop] conclusion {exp.get('id')} failed: {exc}")
|
||||
return written
|
||||
|
||||
|
||||
def _finding_body(all_clusters: List[dict]) -> str:
|
||||
if not all_clusters:
|
||||
return "近 14 天没有审计出可聚合的问题模式。"
|
||||
lines = ["近 14 天问题模式(按重复失败调用数排序):"]
|
||||
for c in all_clusters[:10]:
|
||||
lines.append(
|
||||
f"- {c.get('clusterTitle')} (`{c.get('cluster')}`): {c.get('wasteCalls')} 次失败调用,"
|
||||
f"{c.get('episodes')} 个片段 / {c.get('transcripts')} 份会话;"
|
||||
f"{c.get('problemClass')},{c.get('stage')},重放型 {c.get('replayType')}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def run_once() -> dict:
|
||||
config.ensure_dirs()
|
||||
now_ms = int(time.time() * 1000)
|
||||
config.log("开始:读取控制面状态" + ("(试运行,不写生产)" if config.DRY_RUN else ""))
|
||||
state = api.get("/state")
|
||||
experiments = (api.get("/experiments?limit=500").get("experiments") or [])
|
||||
sync = mine.sync_transcripts(now_ms)
|
||||
mined = mine.mine_local()
|
||||
audited = mine.audit_new()
|
||||
consolidated = mine.consolidate_clusters()
|
||||
all_clusters = mine.clusters()
|
||||
config.log(f"聚类:{len(all_clusters)} 个问题模式")
|
||||
conclusions = write_missing_conclusions(experiments)
|
||||
# 复核意见挖掘与实验互不相干:暂停回路只停实验,不停打标签 / 归纳;失败也只记一笔
|
||||
mining = review_mining.run(now_ms)
|
||||
api.event("复核意见挖掘", review_mining.summary_line(mining) + "\n\n" + json.dumps(mining, ensure_ascii=False, indent=2), "note")
|
||||
api.event(
|
||||
f"发现 {len(all_clusters)} 个问题模式",
|
||||
_finding_body(all_clusters),
|
||||
"finding",
|
||||
)
|
||||
result = {
|
||||
"sync": sync, "mined": mined, "audited": audited, "consolidated": consolidated,
|
||||
"clusters": len(all_clusters), "conclusions": conclusions,
|
||||
"review_mining": mining,
|
||||
"submitted": None,
|
||||
}
|
||||
if (state.get("pause") or {}).get("paused"):
|
||||
api.event(
|
||||
"每日回路完成(已暂停,未提交实验)",
|
||||
json.dumps(result, ensure_ascii=False, indent=2),
|
||||
)
|
||||
return result
|
||||
cluster = _eligible_cluster(all_clusters, state, now_ms)
|
||||
busy = {e.get("platform") for e in state.get("live") or [] if e.get("stage") == "bug.run"}
|
||||
if not cluster:
|
||||
note = "没有可提交的 prompt 选题"
|
||||
else:
|
||||
note = _submit_prompt_experiment(cluster, now_ms, result)
|
||||
sub = result.get("submitted") or {}
|
||||
if sub.get("status") == "lab":
|
||||
busy.add(cluster.get("platform"))
|
||||
result["runtime"] = _submit_runtime_experiments(state, experiments, busy)
|
||||
if result["runtime"]:
|
||||
note += ";旋钮实验:" + "、".join(r.get("title") or "?" for r in result["runtime"])
|
||||
api.event(f"每日回路完成({note})", json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return result
|
||||
|
||||
|
||||
def _submit_prompt_experiment(cluster: dict, now_ms: int, result: dict) -> str:
|
||||
"""写提案 → 离线重放 → 提交。返回给时间线的一句话。"""
|
||||
title = cluster.get("clusterTitle") or cluster.get("cluster")
|
||||
config.log(f"选题:{title}({cluster.get('platform')} · {cluster['stage']}),写提案中")
|
||||
# 按平台展开的现行版:模板里可能有只给某一端的段({{#ios}}…{{/ios}}),候选要插在展开后的版本上
|
||||
active = api.get(f"/prompt/{cluster['stage']}?platform={cluster.get('platform')}")
|
||||
draft = proposal.create(cluster, str(active["template"]))
|
||||
try:
|
||||
offline = run_replays(cluster, draft.pop("_insertion"))
|
||||
except ReplayAborted as exc:
|
||||
_record_skip(cluster, str(exc), now_ms)
|
||||
result["aborted"] = str(exc)
|
||||
return f"未提交:{title}:{exc}"
|
||||
draft["offline"] = offline
|
||||
response = api.post("/experiments", draft, timeout=180)
|
||||
exp = response.get("experiment") or {}
|
||||
result["submitted"] = {
|
||||
"id": exp.get("id"), "title": exp.get("title"),
|
||||
"status": exp.get("status"), "gate": response.get("gate"),
|
||||
}
|
||||
if exp.get("status") in TERMINAL:
|
||||
result["conclusions"] += write_missing_conclusions([exp])
|
||||
return f"提交 {draft['title']}"
|
||||
|
||||
|
||||
def _submit_runtime_experiments(state: dict, experiments: List[dict], busy: set) -> List[dict]:
|
||||
"""空着的实验机(没有进行中的 bug.run 实验)各提一个还没试过的旋钮取值。"""
|
||||
knobs = state.get("runtimeKnobs") or {}
|
||||
if not knobs or (None in busy): # 不带平台的旧实验占着所有平台
|
||||
return []
|
||||
out: List[dict] = []
|
||||
for platform in sorted(state.get("labNodes") or {}):
|
||||
if platform in busy:
|
||||
continue
|
||||
pick = runtime.next_candidate(platform, knobs, experiments)
|
||||
if not pick:
|
||||
continue
|
||||
draft = runtime.draft(platform, pick[0], pick[1], knobs)
|
||||
config.log(f"旋钮实验:{draft['title']}")
|
||||
try:
|
||||
exp = (api.post("/experiments", draft, timeout=180).get("experiment") or {})
|
||||
out.append({"id": exp.get("id"), "title": exp.get("title") or draft["title"], "status": exp.get("status")})
|
||||
except Exception as exc: # noqa: BLE001 一个平台提交失败不影响另一个
|
||||
out.append({"title": draft["title"], "error": str(exc)[:300]})
|
||||
return out
|
||||
|
||||
|
||||
def locked_run() -> dict:
|
||||
config.ensure_dirs()
|
||||
with open(config.LOCK, "w") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise RuntimeError("已有一个 RSI 回路进程在运行") from None
|
||||
lock.write(f"{os.getpid()}\n")
|
||||
lock.flush()
|
||||
try:
|
||||
return run_once()
|
||||
except Exception as exc:
|
||||
api.event("每日回路失败", str(exc)[:20000])
|
||||
raise
|
||||
|
||||
|
||||
def daemon() -> None:
|
||||
"""在图形桌面终端常驻;这样 claude 复用该登录会话的 OAuth。"""
|
||||
while True:
|
||||
now = datetime.now()
|
||||
target = now.replace(hour=config.DAILY_HOUR, minute=0, second=0, microsecond=0)
|
||||
if target <= now:
|
||||
target += timedelta(days=1)
|
||||
wait = max(1, (target - now).total_seconds())
|
||||
print(f"[rsi-loop] next run: {target.isoformat(timespec='seconds')}", flush=True)
|
||||
time.sleep(wait)
|
||||
try:
|
||||
print(json.dumps(locked_run(), ensure_ascii=False, indent=2), flush=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"[rsi-loop] run failed: {exc}", flush=True)
|
||||
@@ -0,0 +1,76 @@
|
||||
"""旋钮实验的选题:当天没有够格的 prompt 选题时,让空着的实验机去试执行配置包里还没试过的旋钮取值。
|
||||
|
||||
旋钮目录由控制面 /state 的 runtimeKnobs 下发(shared/src/executor_profile.ts),这里不另抄一份。
|
||||
一个(平台, 旋钮=取值)只试一次,不论结果:复跑通过的等人决定,没通过的换别的取值再说。
|
||||
实验机现行组已经是这个取值的不提(config.LAB_BASELINE),两组一样等于白跑。
|
||||
提案文字是确定性的模板,不调模型 —— 旋钮实验的「为什么值得试」就是目录里那句说明。
|
||||
"""
|
||||
import json
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from . import config
|
||||
|
||||
TARGET_METRIC = "reach_s"
|
||||
|
||||
|
||||
def _profile_of(exp: dict) -> Dict[str, str]:
|
||||
raw = exp.get("candidateProfile")
|
||||
if not raw:
|
||||
return {}
|
||||
try:
|
||||
obj = json.loads(raw) if isinstance(raw, str) else raw
|
||||
except ValueError:
|
||||
return {}
|
||||
return {str(k): str(v) for k, v in obj.items()} if isinstance(obj, dict) else {}
|
||||
|
||||
|
||||
def tried(experiments: List[dict]) -> set:
|
||||
"""已经试过的 (平台, 旋钮, 取值)。只认单旋钮的配置包 —— 回路自己只提单旋钮。"""
|
||||
out = set()
|
||||
for e in experiments:
|
||||
if e.get("kind") != "runtime":
|
||||
continue
|
||||
prof = _profile_of(e)
|
||||
if len(prof) == 1:
|
||||
(k, v), = prof.items()
|
||||
out.add((e.get("platform"), k, v))
|
||||
return out
|
||||
|
||||
|
||||
def _ordered_values(knob: str, values: List[str]) -> List[str]:
|
||||
if knob == "effort":
|
||||
return [v for v in config.EFFORT_ORDER if v in values] + [v for v in values if v not in config.EFFORT_ORDER]
|
||||
return list(values)
|
||||
|
||||
|
||||
def next_candidate(platform: str, knobs: Dict[str, dict], experiments: List[dict]) -> Optional[Tuple[str, str]]:
|
||||
done = tried(experiments)
|
||||
base = config.LAB_BASELINE.get(platform) or {}
|
||||
for knob, d in knobs.items():
|
||||
only = d.get("platforms")
|
||||
if only and platform not in only:
|
||||
continue
|
||||
for v in _ordered_values(knob, list(d.get("values") or [])):
|
||||
if base.get(knob) == v or (platform, knob, v) in done:
|
||||
continue
|
||||
return knob, v
|
||||
return None
|
||||
|
||||
|
||||
def draft(platform: str, knob: str, value: str, knobs: Dict[str, dict]) -> dict:
|
||||
d = knobs.get(knob) or {}
|
||||
label = d.get("label") or knob
|
||||
plat = "iOS" if platform == "ios" else "Android"
|
||||
base = (config.LAB_BASELINE.get(platform) or {}).get(knob)
|
||||
base_txt = f"实验机现行为 {base}" if base else "实验机现行为生产默认(不设)"
|
||||
return {
|
||||
"stage": "bug.run", "platform": platform, "targetMetric": TARGET_METRIC,
|
||||
"title": f"{plat}:{label}调到 {value}",
|
||||
"summary": f"{d.get('doc') or ''}\n\n当天没有够格的 prompt 选题,实验机空着;这一项是旋钮目录里 {plat} 上还没试过的取值。",
|
||||
"hypothesis": f"把{label}从现状({base_txt})改为 {value},能缩短到达复现现场的用时;"
|
||||
"风险由复跑的两条底线兜住:自报修好、到达复现现场的个数不少于现行。",
|
||||
"method": f"运行时实验:两组同一版 prompt;候选组的执行配置包为 {knob}={value},其余不变。"
|
||||
"在实验机上拿评估集的历史任务配对复跑,按到达复现现场用时判定。",
|
||||
"candidateProfile": {knob: value},
|
||||
"problemClass": None, "cluster": None,
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
"""三类决策点的「坏选择」判定 —— 和 shared/src/rsi_loop.ts 的指标口径同源,但这里判的是
|
||||
重放出来的「下一步动作」,不是整批的聚合指标。
|
||||
|
||||
三类(2026-09 审计确认的真失败形态):
|
||||
T2_wait 等 App 就绪:下一步该用 wait-for / 后台 / Monitor,不该手写 sleep / 轮询循环;
|
||||
T1_repeat 同一堵墙已经失败过:下一步不该用同族命令原样再撞;
|
||||
T3_frida Frida 首次报错后:下一步不该直接再注入,应先诊断(frida-server / 版本 / 进程 / 读代码)。
|
||||
|
||||
判定只看动作形态,封闭、可审计 —— 它是离线闸的真值,不能是「另一个模型觉得好不好」。
|
||||
"""
|
||||
import re
|
||||
import random
|
||||
from collections import defaultdict
|
||||
from typing import Iterable, Optional
|
||||
|
||||
UI_PROBE = re.compile(r"dumpsys\s+(window|activity)|uiautomator|mCurrentFocus|ResumedActivity|\bpidof\b|\bactl\b")
|
||||
|
||||
|
||||
def _is_long_poll(cmd: str) -> bool:
|
||||
if re.search(r"\b(while|until|for)\b.*\bsleep\b", cmd, re.S):
|
||||
return True
|
||||
return any(int(x) >= 15 for x in re.findall(r"\bsleep\s+(\d+)", cmd))
|
||||
|
||||
|
||||
def classify_action(ptype: str, point: dict, action: Optional[dict]) -> str:
|
||||
"""返回 bad / good / other。point 至少含 'fam'(截断处那条失败命令的命令族)。"""
|
||||
if action is None:
|
||||
return "other"
|
||||
name = action.get("name", "")
|
||||
inp = action.get("input") or {}
|
||||
cmd = str(inp.get("command", "")) if name == "Bash" else ""
|
||||
if ptype == "T2_wait":
|
||||
if "wait-for" in cmd:
|
||||
return "good"
|
||||
if name in ("Monitor",) or inp.get("run_in_background"):
|
||||
return "good"
|
||||
if name == "Bash" and _is_long_poll(cmd):
|
||||
return "bad"
|
||||
return "other"
|
||||
if ptype == "T3_frida":
|
||||
if name == "Bash" and re.search(r"\bfrida\b", cmd) and not re.search(r"frida-ps|frida-ls|--version|pgrep|pidof|getprop", cmd):
|
||||
return "bad"
|
||||
return "good"
|
||||
if ptype == "T1_repeat":
|
||||
from .transcript import family
|
||||
now = family(name, cmd) if name == "Bash" else name
|
||||
previous = str(point.get("fam") or "")
|
||||
return "bad" if now and now == previous else "good"
|
||||
return "other"
|
||||
|
||||
|
||||
def action_of(replay_result: dict) -> Optional[dict]:
|
||||
for b in replay_result.get("first") or []:
|
||||
if b.get("type") == "tool_use":
|
||||
return {"name": b["name"], "input": b.get("input") or {}}
|
||||
return None
|
||||
|
||||
|
||||
def _mean(values):
|
||||
return sum(values) / len(values) if values else 0.0
|
||||
|
||||
|
||||
def _quantile(values, q: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
ordered = sorted(values)
|
||||
pos = (len(ordered) - 1) * q
|
||||
lo = int(pos)
|
||||
hi = min(len(ordered) - 1, lo + 1)
|
||||
frac = pos - lo
|
||||
return ordered[lo] * (1 - frac) + ordered[hi] * frac
|
||||
|
||||
|
||||
def offline_summary(records: Iterable[dict], bootstrap_iters: int = 2000, seed: int = 7) -> dict:
|
||||
"""按决策点配对计算 baseBad-candBad,并对「点」而不是单次 seed 做 bootstrap。"""
|
||||
arms = defaultdict(lambda: {"base": [], "candidate": []})
|
||||
seed_values = set()
|
||||
for row in records:
|
||||
arm = row.get("arm")
|
||||
if arm not in ("base", "candidate") or not row.get("pointId"):
|
||||
continue
|
||||
# 续跑失败和没有动作都不算候选的成功;按坏选择处理,避免靠沉默刷过离线闸。
|
||||
bad = 0.0 if row.get("classification") in ("good", "other") and not row.get("error") else 1.0
|
||||
arms[row["pointId"]][arm].append(bad)
|
||||
if row.get("seed") is not None:
|
||||
seed_values.add(row["seed"])
|
||||
paired = [v for v in arms.values() if v["base"] and v["candidate"]]
|
||||
diffs = [_mean(v["base"]) - _mean(v["candidate"]) for v in paired]
|
||||
base_values = [x for v in paired for x in v["base"]]
|
||||
cand_values = [x for v in paired for x in v["candidate"]]
|
||||
boot = []
|
||||
if diffs:
|
||||
rng = random.Random(seed)
|
||||
for _ in range(bootstrap_iters):
|
||||
boot.append(_mean([diffs[rng.randrange(len(diffs))] for _ in diffs]))
|
||||
return {
|
||||
"points": len(paired),
|
||||
"seeds": len(seed_values),
|
||||
"baseBad": _mean(base_values),
|
||||
"candBad": _mean(cand_values),
|
||||
"diffMean": _mean(diffs),
|
||||
"ciLow": _quantile(boot, 0.025),
|
||||
"ciHigh": _quantile(boot, 0.975),
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
"""原生会话记录的解析:工具调用序列、命令族、报错「墙」、后台任务、时间线渲染。
|
||||
|
||||
这些规则都是在 2026-09 的实验里一条条踩出来的(见 docs/rsi.md「自动实验回路」):
|
||||
- 读取类命令(cat / grep / sed / Read)输出里的报错字样是源码或历史记录,不是失败 ——
|
||||
不排除它们,切片误报率是 25%;
|
||||
- 命令超时被自动转到后台(moved to the background)和主动 run_in_background 一样,
|
||||
截断时还在跑的话,续跑会收到「任务没跑完」的通知,处境就变了 —— 这种点不能用;
|
||||
- Monitor 的启动回执(you will be notified on each event)不是报错。
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
ERR = re.compile(r"(^|\n)\s*(error|fatal|exception|traceback|failed)\b|command not found|No such file|Exit code [1-9]|"
|
||||
r"exit status [1-9]|BUILD FAILED|Failed to (spawn|attach)|unable to", re.I)
|
||||
WALLS = re.compile(r"Failed to (?:spawn|attach|enumerate|inject|load)[^\n]{0,80}|unable to (?:connect|find|access)[^\n]{0,60}|"
|
||||
r"INSTALL_FAILED_\w+|device (?:offline|not found|unauthorized)|no devices/emulators found|Permission denied|"
|
||||
r"Process (?:terminated|crashed)|java\.lang\.\w+(?:Exception|Error)|ReferenceError[^\n]{0,60}|TypeError[^\n]{0,60}|"
|
||||
r"Connection refused|timed out|BUILD FAILED|Compilation error|error: [^\n]{0,80}", re.I)
|
||||
READ_ONLY = re.compile(r"^(grep|rg|ls|cat|head|tail|find|test|\[|echo|which|command|pgrep|ps|wc|stat|file|du|diff|sed|awk|jq|"
|
||||
r"sort|true|false|printf|date|tree|readlink|strings|nl|less|more)\b")
|
||||
LAUNCH = re.compile(r"(?:running in background|moved to the background)[^\"]{0,40}?\(?ID[:\s]+([a-z0-9]{6,})", re.I)
|
||||
DONE = [re.compile(r"<task-id>([a-z0-9]{6,})</task-id>[\s\S]{0,600}?<status>(?:completed|failed|stopped|killed)"),
|
||||
re.compile(r"Successfully stopped task:\s*([a-z0-9]{6,})"),
|
||||
re.compile(r'\\"task_id\\":\\"([a-z0-9]{6,})\\"[^}]{0,200}?\\"status\\":\\"(?:completed|failed|killed|stopped)')]
|
||||
|
||||
|
||||
def load_rows(path: Path) -> List[Optional[dict]]:
|
||||
rows: List[Optional[dict]] = []
|
||||
with open(path, errors="replace") as f:
|
||||
for line in f:
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except Exception: # noqa: BLE001
|
||||
rows.append(None)
|
||||
return rows
|
||||
|
||||
|
||||
def text_of(content: Any) -> str:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
return "\n".join(b.get("text", "") for b in content or [] if isinstance(b, dict) and b.get("type") == "text")
|
||||
|
||||
|
||||
def first_prompt(rows: List[Optional[dict]]) -> Optional[dict]:
|
||||
"""第一条用户消息所在的行(执行 prompt 本身)。"""
|
||||
for r in rows:
|
||||
if r and r.get("type") == "user" and not r.get("isMeta"):
|
||||
c = (r.get("message") or {}).get("content")
|
||||
if isinstance(c, str) or (isinstance(c, list) and any(isinstance(b, dict) and b.get("type") == "text" for b in c)):
|
||||
return r
|
||||
return None
|
||||
|
||||
|
||||
def classify_stage(prompt_text: str) -> Optional[str]:
|
||||
head = prompt_text[:600]
|
||||
if re.search(r"自主\s*Bug\s*修复|autonomous bug[- ]?fix", head, re.I):
|
||||
return "bug.run"
|
||||
if re.search(r"自主\s*Feature\s*实现|autonomous feature", head, re.I):
|
||||
return "feature.run.second" if re.search(r"再实现|re-?implement", prompt_text[:1500], re.I) else "feature.run.first"
|
||||
return None
|
||||
|
||||
|
||||
def family(tool: str, cmd: str) -> str:
|
||||
if tool != "Bash":
|
||||
return tool
|
||||
s = re.sub(r"^\s*(nohup|exec)\s+", "", cmd)
|
||||
if re.search(r"\bfrida\b(?!-server)", s) and not re.search(r"frida-(ps|ls)", s):
|
||||
return "frida"
|
||||
if re.search(r"\bwait-for\b", s):
|
||||
return "wait-for"
|
||||
if re.search(r"python3?\s*(-c|<<)", s):
|
||||
return "python-inline"
|
||||
s = re.sub(r'"?\$\{?\w+\}?"?(?=\s+(shell|install|push|pull|logcat|-s)\b)', "adb", s)
|
||||
s = re.sub(r"/[\w./@~+-]+/", "", s)
|
||||
parts = [p.strip() for p in re.split(r"&&|;|\|\||\n", s) if p.strip()]
|
||||
first = next((p for p in parts if not re.match(r"^(cd|export|set|source|sleep|echo)\b|^[A-Za-z_]+=", p)), parts[0] if parts else "")
|
||||
toks = [t for t in first.split() if not re.match(r"^\w+=", t) and t not in ("timeout", "gtimeout", "sudo", "env")]
|
||||
if toks and toks[0] == "adb":
|
||||
rest, i = [], 1
|
||||
while i < len(toks):
|
||||
if re.match(r"^-[stHP]$", toks[i]):
|
||||
i += 2
|
||||
continue
|
||||
if not toks[i].startswith("-"):
|
||||
rest.append(toks[i])
|
||||
i += 1
|
||||
return " ".join(["adb"] + rest[:2])
|
||||
if toks and toks[0] in ("bun", "node", "python3", "python", "bash", "sh") and len(toks) >= 2:
|
||||
return toks[0] + " " + Path(toks[1].strip("\"'")).name
|
||||
return " ".join(toks[:2])
|
||||
|
||||
|
||||
def wall(txt: str) -> Optional[str]:
|
||||
if re.search(r"you will be notified on each", txt, re.I):
|
||||
return None
|
||||
m = WALLS.search(txt[-6000:])
|
||||
if not m:
|
||||
return None
|
||||
s = m.group(0)[:100]
|
||||
s = re.sub(r"/[\w./@-]+", "<p>", s)
|
||||
s = re.sub(r"\b[0-9a-f]{6,}\b", "<h>", s)
|
||||
return re.sub(r"\d+", "#", s).strip()
|
||||
|
||||
|
||||
def is_read_only(tool: str, cmd: str) -> bool:
|
||||
if tool in ("Read", "Grep", "Glob", "LSP", "ToolSearch"):
|
||||
return True
|
||||
if tool != "Bash":
|
||||
return False
|
||||
first = re.split(r"&&|;|\|", re.sub(r"^\s*cd\s+\S+\s*&&\s*", "", cmd).strip())[0].strip()
|
||||
return bool(READ_ONLY.match(first))
|
||||
|
||||
|
||||
def calls(rows: List[Optional[dict]]) -> List[dict]:
|
||||
"""按工具结果的顺序展开调用:序号 i、所在行、工具、命令、成败、墙、当时的上下文 token、时刻。"""
|
||||
uses: Dict[str, tuple] = {}
|
||||
out: List[dict] = []
|
||||
ctx = 0
|
||||
for ri, r in enumerate(rows):
|
||||
if not r:
|
||||
continue
|
||||
m = r.get("message") or {}
|
||||
c = m.get("content")
|
||||
if r.get("type") == "assistant":
|
||||
u = m.get("usage") or {}
|
||||
if u:
|
||||
ctx = u.get("input_tokens", 0) + u.get("cache_read_input_tokens", 0) + u.get("cache_creation_input_tokens", 0)
|
||||
for b in c or []:
|
||||
if isinstance(b, dict) and b.get("type") == "tool_use":
|
||||
uses[b["id"]] = (b.get("name") or "?", b.get("input") or {}, r.get("timestamp"))
|
||||
if r.get("type") == "user" and isinstance(c, list):
|
||||
for b in c:
|
||||
if not (isinstance(b, dict) and b.get("type") == "tool_result"):
|
||||
continue
|
||||
name, inp, t0 = uses.get(b.get("tool_use_id"), ("?", {}, None))
|
||||
txt = text_of(b.get("content"))
|
||||
cmd = str(inp.get("command", "")) if name == "Bash" else json.dumps(inp, ensure_ascii=False)
|
||||
ro = is_read_only(name, cmd)
|
||||
failed = bool(b.get("is_error")) or (name == "Bash" and not ro and bool(ERR.search(txt[-3000:])))
|
||||
out.append(dict(i=len(out), row=ri, tool=name, cmd=cmd, fam=family(name, cmd), ok=not failed,
|
||||
read_only=ro, wall=None if ro else wall(txt), ctx=ctx, t=r.get("timestamp"), t0=t0,
|
||||
bg=bool(re.search(r"moved to the background|running in background", txt[:400])),
|
||||
err=([l for l in txt.strip().splitlines() if l.strip()] or [""])[-1][:160] if failed else ""))
|
||||
return out
|
||||
|
||||
|
||||
def pending_bg(rows: List[Optional[dict]], upto_row: int) -> List[str]:
|
||||
launched: List[str] = []
|
||||
done = set()
|
||||
for r in rows[: upto_row + 1]:
|
||||
if not r or r.get("type") != "user":
|
||||
continue
|
||||
s = json.dumps((r.get("message") or {}).get("content"), ensure_ascii=False)
|
||||
launched += [m.group(1) for m in LAUNCH.finditer(s)]
|
||||
for pat in DONE:
|
||||
done |= {m.group(1) for m in pat.finditer(s)}
|
||||
return sorted({k for k in launched if k not in done})
|
||||
|
||||
|
||||
def has_env_attachment(rows: List[Optional[dict]]) -> bool:
|
||||
return any(r and r.get("type") == "attachment" and (r.get("attachment") or {}).get("type") == "environment" for r in rows[:300])
|
||||
|
||||
|
||||
def clip(s: str, a: int, b: int) -> str:
|
||||
s = s or ""
|
||||
return s if len(s) <= a + b + 20 else s[:a] + f"\n…[省略 {len(s) - a - b} 字]…\n" + s[-b:]
|
||||
|
||||
|
||||
def render_timeline(rows: List[Optional[dict]], start_call: int, end_call: int, before: int = 12, after: int = 6,
|
||||
max_chars: int = 140_000) -> str:
|
||||
"""审计用的时间线:片段前后各带几步上下文,调用编号 #N 与 calls() 的序号一致。"""
|
||||
uses: Dict[str, tuple] = {}
|
||||
out: List[str] = []
|
||||
i = -1
|
||||
lo, hi = start_call - before, end_call + after
|
||||
pending_text: List[str] = []
|
||||
for r in rows:
|
||||
if not r:
|
||||
continue
|
||||
m = r.get("message") or {}
|
||||
c = m.get("content")
|
||||
if r.get("type") == "assistant" and isinstance(c, list):
|
||||
for b in c:
|
||||
if not isinstance(b, dict):
|
||||
continue
|
||||
if b.get("type") == "tool_use":
|
||||
uses[b["id"]] = (b.get("name"), b.get("input") or {})
|
||||
elif b.get("type") == "text" and b.get("text", "").strip():
|
||||
pending_text.append(b["text"].strip())
|
||||
if r.get("type") == "user" and isinstance(c, list):
|
||||
for b in c:
|
||||
if not (isinstance(b, dict) and b.get("type") == "tool_result"):
|
||||
continue
|
||||
i += 1
|
||||
if lo <= i <= hi:
|
||||
name, inp = uses.get(b.get("tool_use_id"), ("?", {}))
|
||||
cmd = inp.get("command") if name == "Bash" else json.dumps(inp, ensure_ascii=False)
|
||||
out += [f"💬 {clip(t, 400, 200)}" for t in pending_text]
|
||||
mark = " ⟵片段内" if start_call <= i <= end_call else ""
|
||||
out.append(f"### [#{i}] {name}{mark} {str(r.get('timestamp', ''))[11:19]}\n```\n{clip(str(cmd), 900, 300)}\n```\n"
|
||||
f"↩︎ {'(is_error) ' if b.get('is_error') else ''}\n```\n{clip(text_of(b.get('content')), 700, 700)}\n```")
|
||||
pending_text = []
|
||||
s = "\n\n".join(out)
|
||||
return s if len(s) <= max_chars else s[:max_chars] + "\n…[时间线过长,已截断]"
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
cmd="${1:-once}"
|
||||
[ $# -gt 0 ] && shift
|
||||
exec /usr/bin/python3 -m rsi_loop "$cmd" "$@"
|
||||
@@ -0,0 +1 @@
|
||||
"""Unit tests for the standalone RSI loop."""
|
||||
@@ -0,0 +1,481 @@
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from unittest import mock
|
||||
|
||||
from rsi_loop import config, episodes, llm, mine, proposal, replay, runner, runtime, scoring, transcript
|
||||
from rsi_loop.runner import _eligible_cluster
|
||||
|
||||
LAB_NODES = {"android": "evaluator-android-01.shiro0.icu", "ios": "temp-ios-01"}
|
||||
|
||||
|
||||
def assistant(tool_id, name, tool_input, tokens=1000):
|
||||
return {
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"usage": {"input_tokens": tokens},
|
||||
"content": [{"type": "tool_use", "id": tool_id, "name": name, "input": tool_input}],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def result(tool_id, text, is_error=False, timestamp="2026-09-01T00:00:00Z"):
|
||||
return {
|
||||
"type": "user", "timestamp": timestamp,
|
||||
"message": {"content": [{
|
||||
"type": "tool_result", "tool_use_id": tool_id,
|
||||
"content": text, "is_error": is_error,
|
||||
}]},
|
||||
}
|
||||
|
||||
|
||||
class TranscriptTests(unittest.TestCase):
|
||||
def test_read_only_output_does_not_become_failure(self):
|
||||
rows = [
|
||||
assistant("a", "Bash", {"command": "grep -R error src"}),
|
||||
result("a", "error: source fixture\nExit code 1"),
|
||||
]
|
||||
call = transcript.calls(rows)[0]
|
||||
self.assertTrue(call["read_only"])
|
||||
self.assertTrue(call["ok"])
|
||||
self.assertIsNone(call["wall"])
|
||||
|
||||
def test_repeated_failure_point_is_after_first_failure(self):
|
||||
rows = [
|
||||
assistant("a", "Bash", {"command": "adb install app.apk"}),
|
||||
result("a", "INSTALL_FAILED_VERSION_DOWNGRADE", True),
|
||||
assistant("b", "Bash", {"command": "adb install app.apk"}),
|
||||
result("b", "INSTALL_FAILED_VERSION_DOWNGRADE", True),
|
||||
]
|
||||
found = episodes.find_episodes(rows)
|
||||
self.assertEqual(len(found), 1)
|
||||
episode = found[0]
|
||||
self.assertEqual(episode["n_fail"], 2)
|
||||
self.assertEqual(episode["point_call"], 0)
|
||||
self.assertEqual(episode["point_row"], 1)
|
||||
|
||||
def test_wait_point_is_before_wait_command(self):
|
||||
rows = [
|
||||
assistant("a", "Bash", {"command": "adb shell input tap 1 2"}),
|
||||
result("a", "ok"),
|
||||
assistant("b", "Bash", {"command": "until adb shell dumpsys window; do sleep 5; done"}),
|
||||
result("b", "done"),
|
||||
]
|
||||
point = episodes.harvest_wait_points(rows)[0]
|
||||
self.assertEqual(point["point_call"], 0)
|
||||
self.assertEqual(point["point_row"], 1)
|
||||
|
||||
|
||||
class ProposalTests(unittest.TestCase):
|
||||
def test_only_inserts_and_preserves_placeholders(self):
|
||||
base = "A {{task}}\nB\nC {{round}}"
|
||||
candidate = proposal.apply_insertion(base, "新增一\n新增二", "B")
|
||||
self.assertEqual(candidate, "A {{task}}\nB\n新增一\n新增二\nC {{round}}")
|
||||
self.assertIsNone(proposal.candidate_error(base, candidate))
|
||||
self.assertEqual(proposal.inserted_text(base, candidate), "新增一\n新增二")
|
||||
|
||||
def test_rejects_changed_line_and_ambiguous_anchor(self):
|
||||
base = "A\nB\nB"
|
||||
with self.assertRaises(ValueError):
|
||||
proposal.apply_insertion(base, "X", "B")
|
||||
self.assertIsNotNone(proposal.candidate_error("A\nB", "A\nB changed"))
|
||||
|
||||
def test_reach_target_cannot_touch_evidence_capture(self):
|
||||
base = "步骤 3\n复现问题。\n步骤 4"
|
||||
shoot = proposal.apply_insertion(base, "- 启动 App 后立刻 record-evidence 拍一张 before-1-x.png", "复现问题。")
|
||||
route = proposal.apply_insertion(base, "- 先按 TestSpec 的入口用 deeplink 直达目标页", "复现问题。")
|
||||
self.assertIsNotNone(proposal.gaming_error("reach_s", base, shoot))
|
||||
self.assertIsNotNone(proposal.gaming_error("reach_miss", base, shoot))
|
||||
self.assertIsNone(proposal.gaming_error("reach_s", base, route))
|
||||
self.assertIsNone(proposal.gaming_error("evidence_fail", base, shoot))
|
||||
|
||||
def test_business_metrics_are_registrable_for_every_replay_type(self):
|
||||
for ptype, allowed in proposal.METRICS_BY_REPLAY.items():
|
||||
self.assertTrue(allowed & proposal.BUSINESS_METRICS, ptype)
|
||||
self.assertTrue(allowed <= proposal.TARGET_METRICS, ptype)
|
||||
|
||||
def test_nested_json_reply_is_parsed(self):
|
||||
with mock.patch.object(llm, "ask", return_value='```json\n{"a":{"b":1}}\n```'):
|
||||
self.assertEqual(llm.ask_json("x"), {"a": {"b": 1}})
|
||||
|
||||
def test_online_metric_must_match_offline_action_shape(self):
|
||||
value = {
|
||||
"title": "x", "summary": "x", "hypothesis": "x", "method": "x",
|
||||
"targetMetric": "tool_calls", "afterLine": None, "content": "新增纪律",
|
||||
}
|
||||
cluster = {
|
||||
"stage": "bug.run", "problemClass": "decision_error",
|
||||
"cluster": "wait", "clusterTitle": "等待", "replayType": "T2_wait",
|
||||
"wasteCalls": 3, "episodes": 1, "transcripts": 1, "samples": [],
|
||||
}
|
||||
with mock.patch.object(llm, "ask_json", return_value=value):
|
||||
with self.assertRaisesRegex(ValueError, "不对应"):
|
||||
proposal.create(cluster, "base")
|
||||
|
||||
|
||||
class ScoringTests(unittest.TestCase):
|
||||
def test_action_shape_rules(self):
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T2_wait", {}, {"name": "Bash", "input": {"command": "sleep 30"}},
|
||||
), "bad")
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T2_wait", {}, {"name": "Bash", "input": {"command": "node wait-for.js"}},
|
||||
), "good")
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T3_frida", {}, {"name": "Bash", "input": {"command": "frida -U -n app -l hook.js"}},
|
||||
), "bad")
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T3_frida", {}, {"name": "Bash", "input": {"command": "frida --version"}},
|
||||
), "good")
|
||||
point = {"fam": "adb install app.apk"}
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T1_repeat", point, {"name": "Bash", "input": {"command": "adb install app.apk"}},
|
||||
), "bad")
|
||||
self.assertEqual(scoring.classify_action(
|
||||
"T1_repeat", point, {"name": "Bash", "input": {"command": "adb shell getprop"}},
|
||||
), "good")
|
||||
|
||||
def test_paired_bootstrap_is_deterministic_and_positive(self):
|
||||
records = []
|
||||
for point in range(20):
|
||||
for seed in range(2):
|
||||
records.append({"pointId": str(point), "arm": "base", "seed": seed, "classification": "bad"})
|
||||
records.append({"pointId": str(point), "arm": "candidate", "seed": seed, "classification": "good"})
|
||||
a = scoring.offline_summary(records)
|
||||
b = scoring.offline_summary(records)
|
||||
self.assertEqual(a, b)
|
||||
self.assertEqual(a["points"], 20)
|
||||
self.assertEqual(a["seeds"], 2)
|
||||
self.assertGreater(a["ciLow"], 0)
|
||||
|
||||
|
||||
class AuditAndSelectionTests(unittest.TestCase):
|
||||
def test_audit_requires_verbatim_evidence(self):
|
||||
raw = {
|
||||
"classification": "decision_error", "cluster": "repeat_same_command",
|
||||
"clusterTitle": "重复撞墙", "replayType": "T1_repeat",
|
||||
"correctInfoStep": 3, "evidence": ["Exit code 1"], "rationale": "已出现错误",
|
||||
}
|
||||
self.assertIsNotNone(mine._valid_audit(raw, "tool result: Exit code 1"))
|
||||
raw["evidence"] = ["模型编造的引文"]
|
||||
self.assertIsNone(mine._valid_audit(raw, "tool result: Exit code 1"))
|
||||
|
||||
def test_live_stage_and_cooldown_are_skipped(self):
|
||||
now = 2_000_000_000_000
|
||||
clusters = [
|
||||
{"stage": "bug.run", "platform": "android", "cluster": "a", "replayType": "T1_repeat"},
|
||||
{"stage": "bug.run", "platform": "ios", "cluster": "b", "replayType": "T2_wait"},
|
||||
{"stage": "bug.run", "platform": "ios", "cluster": "c", "replayType": "T3_frida"},
|
||||
]
|
||||
state = {
|
||||
"labNodes": LAB_NODES,
|
||||
"live": [{"stage": "bug.run", "platform": "android"}],
|
||||
"recent": [{"stage": "bug.run", "platform": "ios", "cluster": "b", "createdAt": now}],
|
||||
}
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
|
||||
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "c")
|
||||
|
||||
def test_cluster_without_enough_points_is_skipped(self):
|
||||
now = 2_000_000_000_000
|
||||
clusters = [
|
||||
{"stage": "bug.run", "platform": "android", "cluster": "few", "replayType": "T1_repeat"},
|
||||
{"stage": "bug.run", "platform": "android", "cluster": "many", "replayType": "T1_repeat"},
|
||||
]
|
||||
counts = {"few": 5, "many": 40}
|
||||
state = {"labNodes": LAB_NODES}
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
|
||||
got = _eligible_cluster(clusters, state, now, point_count=lambda c: counts[c["cluster"]])
|
||||
self.assertEqual(got["cluster"], "many")
|
||||
runner._record_skip(clusters[1], "anchor missing", now)
|
||||
self.assertIsNone(_eligible_cluster(clusters, state, now, point_count=lambda c: counts[c["cluster"]]))
|
||||
|
||||
|
||||
class ReplayFidelityTests(unittest.TestCase):
|
||||
PROMPT = "你是自主 Bug 修复 Agent。\n工作区在 /Users/fei/bug_workspace/123。\n## 等待\n用 wait-for。\n## 结尾"
|
||||
|
||||
def test_insert_at_anchor_with_placeholder_and_no_heading(self):
|
||||
ins = {"afterLine": "工作区在 {{workspace}}。", "content": "新纪律一\n新纪律二"}
|
||||
out = replay.insert_into_text(self.PROMPT, ins)
|
||||
lines = out.split("\n")
|
||||
self.assertEqual(lines[1], "工作区在 /Users/fei/bug_workspace/123。")
|
||||
self.assertEqual(lines[2:4], ["新纪律一", "新纪律二"])
|
||||
self.assertNotIn("候选", out)
|
||||
|
||||
def test_null_anchor_appends_at_end_and_missing_anchor_raises(self):
|
||||
out = replay.insert_into_text(self.PROMPT, {"afterLine": None, "content": "末尾纪律"})
|
||||
self.assertTrue(out.endswith("## 结尾\n末尾纪律"))
|
||||
with self.assertRaises(replay.AnchorMissing):
|
||||
replay.insert_into_text(self.PROMPT, {"afterLine": "不存在的一行", "content": "x"})
|
||||
|
||||
def test_candidate_applies_checks_first_prompt(self):
|
||||
rows = [{"type": "user", "message": {"role": "user", "content": self.PROMPT}}]
|
||||
self.assertTrue(replay.candidate_applies(rows, {"afterLine": "## 等待", "content": "x"}))
|
||||
self.assertFalse(replay.candidate_applies(rows, {"afterLine": "## 不存在", "content": "x"}))
|
||||
self.assertTrue(replay.candidate_applies(rows, None))
|
||||
|
||||
def test_environment_snapshot_uses_this_runs_cwd(self):
|
||||
rows = [
|
||||
{"type": "user", "message": {"role": "user", "content": self.PROMPT}},
|
||||
{"type": "attachment", "attachment": {"type": "environment",
|
||||
"snapshot": {"workingDirectory": "/Users/fei/ai_workspace"},
|
||||
"changes": [{"field": "workingDirectory"}]}},
|
||||
]
|
||||
loc = {"environment": {"type": "environment", "snapshot": {"workingDirectory": "/probe/dir", "platform": "darwin"}}}
|
||||
out, _ = replay._neutralize(rows, "sid", loc, Path("/runs/abc"), Path("/proj/abc"),
|
||||
{"afterLine": "## 等待", "content": "新纪律"})
|
||||
env = out[1]["attachment"]
|
||||
self.assertEqual(env["snapshot"]["workingDirectory"], "/runs/abc")
|
||||
self.assertNotIn("changes", env)
|
||||
self.assertIn("## 等待\n新纪律\n用 wait-for。", out[0]["message"]["content"])
|
||||
|
||||
|
||||
class ClusterConsolidationTests(unittest.TestCase):
|
||||
def _audits(self, d):
|
||||
rows = [
|
||||
{"episodeId": "e1", "stage": "bug.run", "classification": "decision_error", "cluster": "app_ready_blind_wait", "clusterTitle": "盲等就绪", "replayType": "T2_wait"},
|
||||
{"episodeId": "e2", "stage": "bug.run", "classification": "decision_error", "cluster": "fixed_sleep_before_ready", "clusterTitle": "定长 sleep 等就绪", "replayType": "T2_wait"},
|
||||
{"episodeId": "e3", "stage": "bug.run", "classification": "decision_error", "cluster": "frida_retry", "clusterTitle": "Frida 原样重试", "replayType": "T3_frida"},
|
||||
]
|
||||
(Path(d) / "a.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows))
|
||||
|
||||
def test_canon_follows_chain_and_stops_on_cycle(self):
|
||||
self.assertEqual(mine.canon("a", {"a": "b", "b": "c"}), "c")
|
||||
self.assertIn(mine.canon("a", {"a": "b", "b": "a"}), ("a", "b"))
|
||||
self.assertEqual(mine.canon("x", {}), "x")
|
||||
|
||||
def test_consolidation_validates_and_merges(self):
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
|
||||
mock.patch.object(config, "AUDITS", Path(d) / "a.jsonl"):
|
||||
self._audits(d)
|
||||
reply = {"merge": {
|
||||
"fixed_sleep_before_ready": "app_ready_blind_wait", # 同机制、同重放型 → 并
|
||||
"frida_retry": "app_ready_blind_wait", # 重放型不同 → 拒
|
||||
"app_ready_blind_wait": "made_up_name", # 规范名不存在 → 拒
|
||||
}}
|
||||
with mock.patch.object(llm, "ask_json", return_value=reply):
|
||||
r = mine.consolidate_clusters()
|
||||
self.assertEqual(r["merged"], 1)
|
||||
names = {k["cluster"]: k["count"] for k in mine.known_clusters()}
|
||||
self.assertEqual(names, {"app_ready_blind_wait": 2, "frida_retry": 1})
|
||||
|
||||
def test_audit_prompt_lists_known_clusters(self):
|
||||
known = [{"cluster": "app_ready_blind_wait", "clusterTitle": "盲等就绪", "classification": "decision_error", "replayType": "T2_wait", "count": 5}]
|
||||
p = mine._audit_prompt({"stage": "bug.run", "label": "wall:x", "n_fail": 2, "first_call": 1, "last_call": 3}, "时间线", known)
|
||||
self.assertIn("app_ready_blind_wait", p)
|
||||
self.assertIn("必须复用", p)
|
||||
|
||||
|
||||
class LabModeTests(unittest.TestCase):
|
||||
def test_lab_run_sessions_are_not_mined(self):
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
|
||||
mock.patch.object(config, "EPISODES", Path(d) / "e.jsonl"), mock.patch.object(config, "POINTS", Path(d) / "p.jsonl"), \
|
||||
mock.patch.object(mine, "transcript_files", return_value=[(Path(d) / "x.jsonl", {"workspaceKey": "rsi-lab-abc", "node": "n", "file": "x.jsonl"})]), \
|
||||
mock.patch.object(mine, "load_rows", side_effect=AssertionError("复跑会话不该被读")):
|
||||
r = mine.mine_local()
|
||||
self.assertEqual(r["episodes"], 0)
|
||||
|
||||
def test_only_submits_bug_run_with_lab_node(self):
|
||||
now = 2_000_000_000_000
|
||||
clusters = [
|
||||
{"stage": "feature.run.first", "platform": "android", "cluster": "f", "replayType": "T2_wait"},
|
||||
{"stage": "bug.run", "platform": "ios", "cluster": "no_lab_node", "replayType": "T1_repeat"},
|
||||
{"stage": "bug.run", "platform": "android", "cluster": "ok", "replayType": "T2_wait"},
|
||||
]
|
||||
state = {"labNodes": {"android": "evaluator-android-01.shiro0.icu"}}
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
|
||||
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "ok")
|
||||
# 没配实验机就一个都不提
|
||||
self.assertIsNone(_eligible_cluster(clusters, {}, now, point_count=lambda c: 30))
|
||||
|
||||
def test_lab_verdicts_get_conclusions(self):
|
||||
self.assertIn("lab_passed", runner.TERMINAL)
|
||||
self.assertIn("lab_failed", runner.TERMINAL)
|
||||
self.assertIn("declined", runner.TERMINAL)
|
||||
|
||||
|
||||
class PlatformTests(unittest.TestCase):
|
||||
def test_live_experiment_only_blocks_its_own_platform(self):
|
||||
now = 2_000_000_000_000
|
||||
clusters = [
|
||||
{"stage": "bug.run", "platform": "android", "cluster": "a", "replayType": "T2_wait"},
|
||||
{"stage": "bug.run", "platform": "ios", "cluster": "b", "replayType": "T2_wait"},
|
||||
]
|
||||
state = {"labNodes": LAB_NODES, "live": [{"stage": "bug.run", "platform": "android"}]}
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)):
|
||||
self.assertEqual(_eligible_cluster(clusters, state, now, point_count=lambda c: 30)["cluster"], "b")
|
||||
# 不带平台的旧实验对所有平台生效
|
||||
legacy = {"labNodes": LAB_NODES, "live": [{"stage": "bug.run", "platform": None}]}
|
||||
self.assertIsNone(_eligible_cluster(clusters, legacy, now, point_count=lambda c: 30))
|
||||
|
||||
def test_replay_points_stay_on_cluster_platform(self):
|
||||
pts = [
|
||||
{"id": "a", "stage": "bug.run", "platform": "android", "ptype": "T2_wait"},
|
||||
{"id": "i", "stage": "bug.run", "platform": "ios", "ptype": "T2_wait"},
|
||||
]
|
||||
with mock.patch.object(mine, "_read_jsonl", return_value=pts):
|
||||
got = mine.replay_points({"stage": "bug.run", "platform": "ios", "replayType": "T2_wait"})
|
||||
self.assertEqual([p["id"] for p in got], ["i"])
|
||||
|
||||
def test_node_without_transcripts_is_not_an_error(self):
|
||||
import urllib.error
|
||||
|
||||
def fake_get(path, timeout=60):
|
||||
if path == "/nodes":
|
||||
return {"nodes": [{"id": "on", "platform": "android"}, {"id": "off", "platform": "ios"}]}
|
||||
if path.startswith("/nodes/off/"):
|
||||
raise urllib.error.HTTPError(path, 404, "not found", {}, None)
|
||||
return {"files": []}
|
||||
|
||||
with tempfile.TemporaryDirectory() as d, \
|
||||
mock.patch.object(config, "DATA", Path(d)), mock.patch.object(config, "TX_DIR", Path(d) / "tx"), \
|
||||
mock.patch.object(config, "RUNS", Path(d) / "runs"), mock.patch.object(config, "STATE", Path(d) / "state.json"), \
|
||||
mock.patch.object(config, "NODES", []), mock.patch("rsi_loop.api.get", side_effect=fake_get):
|
||||
stats = mine.sync_transcripts(2_000_000_000_000)
|
||||
self.assertEqual(stats["nodes"], 1)
|
||||
self.assertEqual(stats["notEnabled"], ["off"])
|
||||
self.assertEqual(stats["errors"], [])
|
||||
|
||||
|
||||
class SyncTests(unittest.TestCase):
|
||||
def test_one_broken_download_does_not_stop_the_rest(self):
|
||||
files = [{"file": f"{i:08d}-0000-0000-0000-000000000000.jsonl", "size": 10, "mtime": 2_000_000_000_000} for i in range(3)]
|
||||
calls = {"n": 0}
|
||||
|
||||
def fake_download(path, dest):
|
||||
calls["n"] += 1
|
||||
if "00000001-" in path:
|
||||
raise IOError("IncompleteRead(542248 bytes read)")
|
||||
dest.write_text("{}")
|
||||
return 10
|
||||
|
||||
with tempfile.TemporaryDirectory() as d, \
|
||||
mock.patch.object(config, "DATA", Path(d)), mock.patch.object(config, "TX_DIR", Path(d) / "tx"), \
|
||||
mock.patch.object(config, "RUNS", Path(d) / "runs"), mock.patch.object(config, "STATE", Path(d) / "state.json"), \
|
||||
mock.patch.object(config, "NODES", ["n1"]), \
|
||||
mock.patch("rsi_loop.api.get", return_value={"files": files}), \
|
||||
mock.patch("rsi_loop.api.download", side_effect=fake_download), \
|
||||
mock.patch("rsi_loop.mine.time.sleep"):
|
||||
stats = mine.sync_transcripts(2_000_000_000_000)
|
||||
self.assertEqual(stats["downloaded"], 2)
|
||||
self.assertEqual(len(stats["errors"]), 1)
|
||||
self.assertIn("IncompleteRead", stats["errors"][0])
|
||||
|
||||
|
||||
class DryRunTests(unittest.TestCase):
|
||||
def test_dry_run_post_never_touches_network(self):
|
||||
from rsi_loop import api
|
||||
with tempfile.TemporaryDirectory() as d, mock.patch.object(config, "DATA", Path(d)), \
|
||||
mock.patch.object(config, "DRY_RUN", True), \
|
||||
mock.patch("urllib.request.urlopen", side_effect=AssertionError("dry-run 不该联网")):
|
||||
got = api.post("/experiments", {"title": "t", "stage": "bug.run"})
|
||||
self.assertEqual(got["experiment"]["status"], "dry_run")
|
||||
self.assertTrue(api.event("运行摘要"))
|
||||
lines = (Path(d) / "dry_posts.jsonl").read_text().splitlines()
|
||||
self.assertEqual([json.loads(l)["path"] for l in lines], ["/experiments", "/events"])
|
||||
runner._record_skip({"stage": "bug.run", "cluster": "x"}, "anchor", 1)
|
||||
self.assertFalse((Path(d) / "skips.json").exists())
|
||||
|
||||
|
||||
class FirstTurnTests(unittest.TestCase):
|
||||
def test_text_then_tool_use_is_one_turn(self):
|
||||
lines = [
|
||||
json.dumps({"type": "system", "subtype": "init"}),
|
||||
json.dumps({"type": "assistant", "message": {"id": "m1", "content": [{"type": "text", "text": "先冷启动看登录态。"}]}}),
|
||||
json.dumps({"type": "assistant", "message": {"id": "m1", "content": [
|
||||
{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "adb shell monkey -p x 1"}}]}}),
|
||||
json.dumps({"type": "result", "is_error": False}),
|
||||
]
|
||||
first, err = replay.first_turn(lines)
|
||||
self.assertIsNone(err)
|
||||
self.assertEqual(scoring.action_of({"first": first})["input"]["command"], "adb shell monkey -p x 1")
|
||||
|
||||
def test_error_result_is_reported(self):
|
||||
first, err = replay.first_turn([json.dumps({"type": "result", "is_error": True, "result": "Failed to authenticate"})])
|
||||
self.assertIsNone(first)
|
||||
self.assertIn("authenticate", err)
|
||||
|
||||
|
||||
class SamplingTests(unittest.TestCase):
|
||||
def _points(self):
|
||||
mk = lambda i, stage, ptype, ep: {"id": f"p{i}", "stage": stage, "ptype": ptype, "episodeId": ep, # noqa: E731
|
||||
"node": "n", "file": f"f{i}", "point_row": 1}
|
||||
target = [mk(i, "bug.run", "T1_repeat", f"t{i}") for i in range(20)]
|
||||
others = [
|
||||
mk(100, "bug.run", "T2_wait", None), # 同 stage 等待点:可进保留集
|
||||
mk(101, "bug.run", "T1_repeat", "real"), # 同 stage 审计过的真失败:可进
|
||||
mk(102, "bug.run", "T1_repeat", "unaudited"), # 没审计过(可能是误报):不进
|
||||
mk(103, "feature.run.first", "T2_wait", None), # 其它 stage:不进
|
||||
]
|
||||
return target, others
|
||||
|
||||
def test_holdout_is_same_stage_and_audited_and_anchor_checked(self):
|
||||
target, others = self._points()
|
||||
cluster = {"stage": "bug.run", "cluster": "x", "replayType": "T1_repeat"}
|
||||
with mock.patch.object(mine, "replay_points", side_effect=lambda c=None: target if c else target + others), \
|
||||
mock.patch.object(mine, "real_episode_ids", return_value={"real"}), \
|
||||
mock.patch.object(mine, "load_rows_for_point", side_effect=lambda p: p["id"]), \
|
||||
mock.patch.object(replay, "candidate_applies", side_effect=lambda rows, ins: rows != "p3"):
|
||||
t, h, seeds, skipped = runner._sample_points(cluster, {"afterLine": "a", "content": "b"})
|
||||
self.assertEqual({p["id"] for p in h}, {"p100", "p101"})
|
||||
self.assertNotIn("p3", {p["id"] for p in t})
|
||||
self.assertEqual(skipped["targetAnchorMissing"], 1)
|
||||
self.assertGreaterEqual(seeds, 2)
|
||||
|
||||
def test_high_error_rate_aborts_without_offline_result(self):
|
||||
target, _ = self._points()
|
||||
cluster = {"stage": "bug.run", "cluster": "x", "replayType": "T1_repeat"}
|
||||
with tempfile.TemporaryDirectory() as d, \
|
||||
mock.patch.object(config, "RUNS", Path(d)), \
|
||||
mock.patch.object(mine, "replay_points", side_effect=lambda c=None: target), \
|
||||
mock.patch.object(mine, "real_episode_ids", return_value=set()), \
|
||||
mock.patch.object(mine, "load_rows_for_point", return_value=[]), \
|
||||
mock.patch.object(replay, "candidate_applies", return_value=True), \
|
||||
mock.patch.object(replay, "prepare"), \
|
||||
mock.patch.object(replay, "run_point", return_value={"first": None, "secs": 0, "error": "Failed to authenticate"}):
|
||||
with self.assertRaises(runner.ReplayAborted):
|
||||
runner.run_replays(cluster, {"afterLine": None, "content": "x"})
|
||||
|
||||
|
||||
class RuntimeProposalTests(unittest.TestCase):
|
||||
KNOBS = {
|
||||
"effort": {"label": "思考强度", "values": ["low", "medium", "high", "xhigh", "max"], "doc": "d"},
|
||||
"ui_settle": {"label": "等界面稳定", "values": ["on"], "doc": "d", "platforms": ["ios"]},
|
||||
}
|
||||
|
||||
def _exp(self, platform, profile):
|
||||
return {"kind": "runtime", "platform": platform, "candidateProfile": json.dumps(profile)}
|
||||
|
||||
def test_skips_tried_baseline_and_off_platform(self):
|
||||
exps = [self._exp("android", {"effort": "medium"}), self._exp("ios", {"ui_settle": "on"})]
|
||||
with mock.patch.object(config, "LAB_BASELINE", {"android": {"effort": "xhigh"}}):
|
||||
# medium 试过、xhigh 是实验机现行 → high;ui_settle 不适用 Android
|
||||
self.assertEqual(runtime.next_candidate("android", self.KNOBS, exps), ("effort", "high"))
|
||||
# iOS:effort 一个都没试过,从 medium 开始
|
||||
self.assertEqual(runtime.next_candidate("ios", self.KNOBS, exps), ("effort", "medium"))
|
||||
|
||||
def test_exhausted_catalog_proposes_nothing(self):
|
||||
exps = [self._exp("ios", {"effort": v}) for v in ["low", "medium", "high", "xhigh", "max"]]
|
||||
exps.append(self._exp("ios", {"ui_settle": "on"}))
|
||||
self.assertIsNone(runtime.next_candidate("ios", self.KNOBS, exps))
|
||||
|
||||
def test_free_lab_nodes_each_get_one(self):
|
||||
state = {"runtimeKnobs": self.KNOBS, "labNodes": LAB_NODES}
|
||||
posted = []
|
||||
with mock.patch.object(runner.api, "post", side_effect=lambda path, body, timeout=120: posted.append(body) or {"experiment": {"id": "x", "title": body["title"], "status": "lab"}}), \
|
||||
mock.patch.object(config, "LAB_BASELINE", {}):
|
||||
got = runner._submit_runtime_experiments(state, [], busy={"android"})
|
||||
self.assertEqual([b["platform"] for b in posted], ["ios"])
|
||||
self.assertEqual(posted[0]["candidateProfile"], {"effort": "medium"})
|
||||
self.assertEqual(got[0]["status"], "lab")
|
||||
|
||||
|
||||
class IosWaitPointTests(unittest.TestCase):
|
||||
def test_catuse_probe_counts_as_ui_wait(self):
|
||||
self.assertTrue(episodes.UI_LIKE.search("sleep 20; catuse apps 2>&1 | head -20"))
|
||||
self.assertTrue(episodes._is_wait("sleep 20; catuse apps 2>&1 | head -20"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,82 @@
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from rsi_loop import config, review_mining
|
||||
|
||||
# 2026-11-02 是周一(北京时间),2026-11-03 是周二
|
||||
MON = 1793548800000 + 3600_000 * 4 # 2026-11-02 04:00 北京
|
||||
TUE = MON + 86_400_000
|
||||
|
||||
|
||||
class LabelTest(unittest.TestCase):
|
||||
def test_batches_posted_and_failures_isolated(self):
|
||||
pending = {"total": 3, "batches": [{"keys": ["a"], "prompt": "p1"}, {"keys": ["b"], "prompt": "p2"}]}
|
||||
answers = [RuntimeError("boom"), {"labels": [{"key": "b", "cat": "ops"}]}]
|
||||
posted = []
|
||||
|
||||
def ask_json(prompt, timeout=0):
|
||||
a = answers.pop(0)
|
||||
if isinstance(a, Exception):
|
||||
raise a
|
||||
return a
|
||||
|
||||
with mock.patch.object(review_mining.api, "get", return_value=pending), \
|
||||
mock.patch.object(review_mining.llm, "ask_json", side_effect=ask_json), \
|
||||
mock.patch.object(review_mining.api, "post", side_effect=lambda path, body, **k: posted.append((path, body)) or {"saved": 1, "dropped": 0}):
|
||||
out = review_mining.label_pending()
|
||||
self.assertEqual(out["failedBatches"], 1)
|
||||
self.assertEqual(out["saved"], 1)
|
||||
self.assertEqual(posted[0][0], "/review-mining/labels")
|
||||
self.assertEqual(posted[0][1]["labeler"], review_mining.LABELER)
|
||||
|
||||
def test_days_query(self):
|
||||
with mock.patch.object(review_mining.api, "get", return_value={"batches": []}) as g:
|
||||
review_mining.label_pending(days=180)
|
||||
g.assert_called_with("/review-mining/pending?days=180", timeout=300)
|
||||
|
||||
|
||||
class ClusterTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.marker = Path(self.tmp.name) / "last"
|
||||
self.patch = mock.patch.object(review_mining, "LAST_CLUSTER", self.marker)
|
||||
self.patch.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.patch.stop()
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_only_on_monday_once(self):
|
||||
self.assertFalse(review_mining._cluster_due(TUE))
|
||||
self.assertTrue(review_mining._cluster_due(MON))
|
||||
inp = {"rows": 2, "prompt": "p", "window": {"from": 1, "to": 2}}
|
||||
with mock.patch.object(review_mining.api, "get", return_value=inp), \
|
||||
mock.patch.object(review_mining.llm, "ask_json", return_value={"candidates": [{"family": "x"}]}), \
|
||||
mock.patch.object(review_mining.api, "post", return_value={"saved": 1, "rejected": []}) as post, \
|
||||
mock.patch.object(config, "DRY_RUN", False):
|
||||
out = review_mining.cluster(MON)
|
||||
self.assertEqual(out["saved"], 1)
|
||||
self.assertEqual(post.call_args[0][1]["window"], {"from": 1, "to": 2})
|
||||
self.assertFalse(review_mining._cluster_due(MON)) # 同一天不再跑
|
||||
|
||||
def test_force_and_empty_rows(self):
|
||||
with mock.patch.object(review_mining.api, "get", return_value={"rows": 0, "window": {}}), \
|
||||
mock.patch.object(review_mining.llm, "ask_json") as ask, \
|
||||
mock.patch.object(config, "DRY_RUN", True):
|
||||
out = review_mining.cluster(TUE, force=True)
|
||||
ask.assert_not_called()
|
||||
self.assertEqual(out["rows"], 0)
|
||||
self.assertFalse(self.marker.exists()) # 试运行不落标记
|
||||
|
||||
def test_run_isolates_errors(self):
|
||||
with mock.patch.object(review_mining, "label_pending", side_effect=RuntimeError("down")), \
|
||||
mock.patch.object(review_mining, "cluster", return_value=None):
|
||||
out = review_mining.run(TUE)
|
||||
self.assertIn("error", out["label"])
|
||||
self.assertIn("打标签失败", review_mining.summary_line(out))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user