Coverage for plugin/scripts/check_command_contracts.py: 100%
204 statements
« prev ^ index » next coverage.py v7.16.1, created at 2026-09-15 14:46 +0000
1#!/usr/bin/env python3
2"""Check that the prose commands are executable — the integration test for markdown.
4Every file in ``commands/`` and ``prompts/`` is instructions a model executes at
5runtime. The bundled scripts are covered by unit tests, but the **prose that drives
6them** was previously unverified, and that is where the expensive failures have come
7from: a command referring to a script that no longer exists, passing a flag that was
8renamed, or invoking a slash command that was removed. Those surface only when a user
9runs the command, mid-task, and the model improvises around the breakage.
11This closes that gap by treating each command as a contract and checking it against
12the code it actually calls:
141. **Frontmatter** — a command declares ``description``, ``argument-hint`` and
15 ``allowed-tools``, and the block **parses**. That second half matters: five of seven
16 commands once shipped frontmatter YAML could not read, because a description
17 contained an unquoted ``": "`` — which YAML takes as a nested mapping. The key check
18 was a substring search, so it passed happily on a file no parser would accept.
192. **Bash blocks parse** — every fenced ``bash`` block is valid shell (``bash -n``),
20 with ``<placeholder>`` spans neutralised first since they aren't real syntax.
213. **Scripts exist** — every ``scripts/<name>.py`` a block invokes is shipped.
224. **Flags exist** — every ``--flag`` passed to a bundled script is one that script's
23 ``argparse`` actually accepts. This is the check that catches a renamed CLI, the
24 most likely silent breakage.
255. **Slash commands exist** — every ``/rhiza:<name>`` referenced resolves to a real
26 command, so a removed one can't linger in another command's prose.
276. **allowed-tools covers the binaries used** — a command that runs ``git`` in a block
28 must be permitted to, or the user gets a permission prompt mid-flow.
297. **Prose references resolve** — a ``scripts/<name>.py`` named in ``README.md``,
30 ``CONTRIBUTING.md``, ``CLAUDE.md`` or the docs site exists, and its flags are real.
31 Checked over the whole text, not just fenced blocks: ``CONTRIBUTING.md`` pointed at
32 ``scripts/bump_version.py``, which never existed, and it survived because it sat in
33 inline backticks and because nothing read the top-of-repo prose at all.
348. **Model-invocation policy** — exactly the commands in ``_MODEL_INVOCATION_OPT_OUT``
35 declare ``disable-model-invocation: true``, and no others declare the key at all.
36 The policy is a property of the whole command surface, not of one file, so it is
37 asserted in both directions: a destructive command that quietly loses the key is as
38 much a regression as a harmless one that grows it.
399. **Test references resolve** — a ``tests/test_<name>.py`` named in that same prose
40 exists. Rule 7's other half, and missing for the same reason: ``docs/development.md``
41 went on telling contributors to run ``tests/test_init_e2e.py`` long after that file
42 was folded into ``test_init_scaffold.py``, because only ``scripts/`` was ever scanned.
4310. **One file per command** — no name is claimed by both ``commands/<name>.md`` and
44 ``skills/<name>/SKILL.md``. Claude Code loads both layouts, so the failure mode of a
45 part-migrated plugin is two files answering the same ``/rhiza:<name>`` with whichever
46 one wins undefined. A copy that was never followed by a delete fails here.
48Usage:
49 uv run --python 3.12 --no-project python \
50 scripts/check_command_contracts.py [--root DIR]
52Exits 0 when every contract holds, 1 (listing each violation) otherwise.
53"""
55from __future__ import annotations
57import argparse
58import re
59import shutil
60import subprocess # nosec B404
61import sys
62from pathlib import Path
64from _rhiza_layout import PLUGIN_DIR, PROMPTS_DIR, SCRIPTS_DIR, command_files
66_FRONTMATTER_KEYS = ("description", "argument-hint", "allowed-tools")
67# Commands the model may not invoke off a description match — the user has to name them.
68# The line is drawn at side effects that are not a reviewable proposal: `detach`
69# deletes every managed file, `release` commits and tags, and `completions` writes into
70# the user's home directory, where no diff, PR or `git checkout` can show or undo it.
71# Everything else stays invocable on purpose — `init` and `update` open a PR but never
72# push to the default branch, `docs` only writes files, and `quality` files issues solely
73# from an explicit menu selection. Adding a command here is a deliberate change to the
74# plugin's surface, which is why the set lives in code and is reviewed rather than
75# inferred per file.
76_MODEL_INVOCATION_OPT_OUT = frozenset({"completions", "detach", "release"})
77_OPT_OUT_KEY = "disable-model-invocation"
78_BASH_BLOCK = re.compile(r"```bash\n(.*?)```", re.S)
79# The script path is usually quoted — `"${CLAUDE_PLUGIN_ROOT}/scripts/x.py" --flag` —
80# so the closing quote must be consumed before the arguments, or the argument capture
81# comes back empty and every flag goes unchecked.
82#
83# The `tests/` lookbehind is load-bearing: the test suite mirrors the plugin, so it lives
84# at `tests/scripts/test_x.py`, whose tail is a substring of the shipped-script shape this
85# matches. Without it, prose naming a *test* file is read as naming a missing *script*.
86_SCRIPT_CALL = re.compile(r"(?<!tests/)scripts/([a-z_]+)\.py[\"']?((?:\s+[^\n`]*)?)")
87_SLASH_COMMAND = re.compile(r"/rhiza:([a-z-]+)")
88# The documented way one command delegates to another. Case-insensitive: the phrase
89# is often capitalised at the start of a sentence or a bullet.
90_SKILL_INVOCATION = re.compile(r"invoke the `([a-z-]+)` command via the Skill tool", re.IGNORECASE)
91_ADD_ARGUMENT = re.compile(r"add_argument\(")
92_FLAG = re.compile(r"--[a-zA-Z][a-zA-Z0-9-]*")
93# `Bash(git*)` and friends, from a command's allowed-tools line.
94_ALLOWED_BASH = re.compile(r"Bash\(([a-zA-Z0-9_.-]+)\*?\)")
96# Placeholders that are prose, not shell: `<github|gitlab>`, `<BODY>`, `<TARGET>`.
97_ANGLE_PLACEHOLDER = re.compile(r"<[A-Za-z][A-Za-z0-9|_ -]*>")
99# Binaries a block may call without being declared: shell builtins and control words.
100_SHELL_BUILTINS = frozenset(
101 {"cd", "echo", "test", "if", "then", "else", "fi", "for", "do", "done", "printf", "export"}
102)
105def frontmatter(text: str) -> str | None:
106 """Return the YAML frontmatter block, or None when the file has none."""
107 if not text.startswith("---\n"):
108 return None
109 end = text.find("\n---\n", 4)
110 return None if end == -1 else text[4:end]
113def script_flags(path: Path) -> set[str]:
114 """Return every ``--flag`` the script at *path* declares via argparse.
116 Scans a window after each ``add_argument(`` rather than parsing the call, which
117 handles both the single-line and multi-line styles used across the scripts without
118 importing them (importing would run module-level code).
119 """
120 source = path.read_text(encoding="utf-8")
121 flags: set[str] = set()
122 for match in _ADD_ARGUMENT.finditer(source):
123 window = source[match.end() : match.end() + 400]
124 # Stop at the next add_argument so a long call can't leak into the next one.
125 nxt = window.find("add_argument(")
126 if nxt != -1:
127 window = window[:nxt]
128 flags |= set(_FLAG.findall(window))
129 return flags
132def bash_blocks(text: str) -> list[str]:
133 """Return the fenced ``bash`` blocks in *text*."""
134 return _BASH_BLOCK.findall(text)
137def unquoted_mapping_colon(value: str) -> bool:
138 """Does *value* contain a `: ` that YAML would read as a nested mapping?
140 A plain (unquoted) YAML scalar may not contain ``": "``. Writing
141 ``description: procedures under prompts/: install-uv`` therefore makes the whole
142 frontmatter unparseable — and five of seven commands shipped exactly that, because
143 the key check below was a substring search that never tried to parse anything.
144 """
145 stripped = value.strip()
146 if not stripped or stripped[0] in "'\"|>": # quoted or a block scalar: fine
147 return False
148 return ": " in stripped
151def parse_frontmatter(block: str) -> tuple[dict[str, str], list[str]]:
152 """Parse a simple ``key: value`` frontmatter block; return (mapping, problems).
154 Deliberately a strict subset rather than a YAML library: the scripts are
155 stdlib-only, and the failure being guarded against is precisely a value that a real
156 YAML parser would choke on.
157 """
158 mapping: dict[str, str] = {}
159 problems: list[str] = []
160 for number, line in enumerate(block.splitlines(), start=2):
161 if not line.strip() or line.startswith("#"):
162 continue
163 if line[0].isspace(): # a continuation of the previous value
164 continue
165 if ":" not in line:
166 problems.append(f"line {number} is not `key: value`: {line[:60]!r}")
167 continue
168 key, _, value = line.partition(":")
169 mapping[key.strip()] = value.strip()
170 if unquoted_mapping_colon(value):
171 problems.append(
172 f"`{key.strip()}` contains an unquoted `: `, which YAML reads as a "
173 "nested mapping — quote the value or rewrite the colon"
174 )
175 return mapping, problems
178def check_frontmatter(rel: str, text: str, *, is_command: bool) -> list[str]:
179 """Rule 1: commands declare parseable frontmatter; procedures declare none."""
180 block = frontmatter(text)
181 if not is_command:
182 if block is not None:
183 return [f"{rel}: has frontmatter, but a procedure is not invocable"]
184 return []
185 if block is None:
186 return [f"{rel}: missing frontmatter"]
188 mapping, problems = parse_frontmatter(block)
189 violations = [f"{rel}: {problem}" for problem in problems]
190 violations += [
191 f"{rel}: frontmatter has no `{key}`" for key in _FRONTMATTER_KEYS if key not in mapping
192 ]
193 return violations
196def check_bash_syntax(rel: str, blocks: list[str]) -> list[str]:
197 """Rule 2: every bash block is syntactically valid shell."""
198 bash = shutil.which("bash")
199 if bash is None: # pragma: no cover - bash is present everywhere this runs
200 return []
201 violations = []
202 for i, block in enumerate(blocks, 1):
203 cleaned = _ANGLE_PLACEHOLDER.sub("PLACEHOLDER", block)
204 result = subprocess.run( # nosec B603
205 [bash, "-n"], input=cleaned, capture_output=True, text=True, check=False
206 )
207 if result.returncode != 0:
208 detail = result.stderr.strip().splitlines()[-1] if result.stderr.strip() else "?"
209 violations.append(f"{rel}: bash block {i} is not valid shell — {detail}")
210 return violations
213def check_script_calls(rel: str, blocks: list[str], scripts_dir: Path) -> list[str]:
214 """Rules 3 and 4: invoked scripts exist, and their flags are real.
216 Backslash continuations are joined first. Nearly every invocation in this prose is
217 wrapped across lines, so without joining, only the first line's flags are seen —
218 and a bad flag on a later line would pass unnoticed.
219 """
220 violations = []
221 for raw in blocks:
222 block = raw.replace("\\\n", " ")
223 for name, args in _SCRIPT_CALL.findall(block):
224 path = scripts_dir / f"{name}.py"
225 if not path.is_file():
226 violations.append(f"{rel}: invokes scripts/{name}.py, which does not exist")
227 continue
228 declared = script_flags(path)
229 for flag in _FLAG.findall(args):
230 if flag not in declared:
231 violations.append(
232 f"{rel}: passes {flag} to scripts/{name}.py, which does not accept it"
233 )
234 return violations
237def check_slash_commands(rel: str, text: str, names: set[str]) -> list[str]:
238 """Rule 5: every command we tell the model to *invoke* exists.
240 Only invocations are checked, not mentions. Prose legitimately refers to retired
241 commands to explain history ("the view the retired ``/rhiza:tree`` gave",
242 "absorbs ``/rhiza:validate``"), and flagging those would push authors to delete
243 useful context. What must not dangle is an instruction to run something.
245 *names* is the whole command surface, gathered across both layouts, rather than a
246 directory to stat: a command moved into ``skills/`` is still the same command, and
247 prose that invokes it must not start failing because of where its file sits.
248 """
249 invoked = set(_SKILL_INVOCATION.findall(text))
250 for block in bash_blocks(text):
251 invoked |= set(_SLASH_COMMAND.findall(block))
252 return [
253 f"{rel}: tells the model to invoke `{name}`, which is not a command"
254 for name in sorted(invoked)
255 if name not in names
256 ]
259def _leading_binaries(blocks: list[str]) -> set[str]:
260 """Return the binary invoked at the start of each command line across *blocks*.
262 Backslash continuations are joined first: without that, the second line of a
263 wrapped invocation looks like a command of its own and every flag reads as a
264 binary name.
265 """
266 found: set[str] = set()
267 for block in blocks:
268 joined = block.replace("\\\n", " ")
269 for raw in joined.splitlines():
270 line = raw.strip()
271 if not line or line.startswith("#"):
272 continue
273 word = line.split()[0]
274 # Skip variable assignments (`BRANCH=...`) and anything flag-shaped.
275 if word.startswith("-") or "=" in word:
276 continue
277 found.add(word)
278 return found
281def check_allowed_tools(rel: str, text: str, blocks: list[str]) -> list[str]:
282 """Rule 6: a command may run the binaries its blocks actually call."""
283 block_text = frontmatter(text)
284 if block_text is None:
285 return []
286 line = next((ln for ln in block_text.splitlines() if ln.startswith("allowed-tools:")), None)
287 if line is None:
288 return []
289 permitted = set(_ALLOWED_BASH.findall(line))
290 return [
291 f"{rel}: runs `{binary}` but allowed-tools has no Bash({binary}*)"
292 for binary in sorted(_leading_binaries(blocks))
293 if binary not in permitted and binary not in _SHELL_BUILTINS
294 ]
297def check_model_invocation(rel: str, stem: str, text: str) -> list[str]:
298 """Rule 8: exactly the opt-out commands disable model invocation.
300 Checked in both directions. A missing key on a destructive command is the
301 dangerous failure, but an unexpected key elsewhere matters too: it silently
302 removes a command from what the model can reach, and the user only finds out by
303 the command never firing.
304 """
305 block = frontmatter(text)
306 if block is None: # already reported as missing frontmatter by rule 1
307 return []
308 declared = parse_frontmatter(block)[0].get(_OPT_OUT_KEY)
309 if stem in _MODEL_INVOCATION_OPT_OUT:
310 if declared != "true":
311 return [
312 f"{rel}: has side effects the user must ask for by name, so it must "
313 f"declare `{_OPT_OUT_KEY}: true` (found: {declared or 'nothing'})"
314 ]
315 return []
316 if declared is not None:
317 return [
318 f"{rel}: declares `{_OPT_OUT_KEY}: {declared}` but is not in the opt-out "
319 f"set — add `{stem}` to _MODEL_INVOCATION_OPT_OUT or drop the key"
320 ]
321 return []
324# Prose a contributor reads that is not a command: the top-of-repo files and the docs
325# site. `docs/reports/` is a generated coverage dump, not prose anyone wrote.
326_PROSE_FILES = ("README.md", "CONTRIBUTING.md", "CLAUDE.md", "SECURITY.md")
327_PROSE_GLOB = "docs/**/*.md"
328_PROSE_EXCLUDE = ("docs/reports/",)
329# A test file named in prose. The lookbehind keeps it to *this* repo's tests: the
330# template's synced `.rhiza/tests/test_pyproject.py` is documented here and is not ours
331# to resolve. Globs and `<name>` placeholders never match the `[a-z0-9_]+` body.
332_TEST_REFERENCE = re.compile(r"(?<![\w./-])tests/(test_[a-z0-9_]+)\.py")
335def prose_files(root: Path) -> list[Path]:
336 """Return the non-command markdown whose script references should still resolve."""
337 found = [root / name for name in _PROSE_FILES if (root / name).is_file()]
338 for path in sorted(root.glob(_PROSE_GLOB)):
339 rel = path.relative_to(root).as_posix()
340 if not any(rel.startswith(prefix) for prefix in _PROSE_EXCLUDE):
341 found.append(path)
342 return found
345def check_test_references(rel: str, text: str, tests_dir: Path) -> list[str]:
346 """Rule 9: a `tests/test_<name>.py` named in prose exists.
348 The other half of rule 7, and it was missing for the same reason rule 7 was added.
349 ``docs/development.md`` told contributors to run
350 ``RHIZA_E2E=1 uvx pytest tests/test_init_e2e.py`` — a file folded into
351 ``test_init_scaffold.py`` and deleted — and the instruction survived the deletion
352 because the reference scanner only ever looked at ``scripts/``. A contributor
353 following it gets "file or directory not found" and reasonably concludes the suite
354 is broken.
356 Only ``tests/…`` at a path boundary counts, so the template's own
357 ``.rhiza/tests/test_pyproject.py`` (a synced file, not ours) is left alone.
358 """
359 return [
360 f"{rel}: names tests/{name}.py, which does not exist"
361 for name in _TEST_REFERENCE.findall(text)
362 if not (tests_dir / f"{name}.py").is_file()
363 ]
366def check_script_references(rel: str, text: str, scripts_dir: Path) -> list[str]:
367 """Rule 7: a `scripts/<name>.py` named anywhere in prose exists, with real flags.
369 Scans the **whole text**, not just fenced bash blocks, and that is the point.
370 `CONTRIBUTING.md` told contributors to run ``scripts/bump_version.py`` — a file that
371 has never existed — for however long it had been there. It survived every gate
372 because it sat in inline backticks rather than a ```bash block, and because nothing
373 read the top-of-repo prose at all. A contributor following it hits a traceback; the
374 build stayed green.
375 """
376 violations = []
377 for name, args in _SCRIPT_CALL.findall(text.replace("\\\n", " ")):
378 path = scripts_dir / f"{name}.py"
379 if not path.is_file():
380 violations.append(f"{rel}: names scripts/{name}.py, which does not exist")
381 continue
382 declared = script_flags(path)
383 violations += [
384 f"{rel}: passes {flag} to scripts/{name}.py, which does not accept it"
385 for flag in _FLAG.findall(args)
386 if flag not in declared
387 ]
388 return violations
391def _rel(root: Path, path: Path) -> str:
392 """A command or procedure path as the violations name it — relative to ``plugin/``.
394 ``commands/init.md`` and ``skills/maffay/SKILL.md`` rather than either bare
395 basename: ``SKILL.md`` alone identifies nothing, and the ``plugin/`` prefix is noise
396 repeated on every line.
397 """
398 return path.relative_to(root / PLUGIN_DIR).as_posix()
401def check_one_file_per_command(root: Path, commands: list[tuple[str, Path]]) -> list[str]:
402 """Rule 10: no command name is claimed by both layouts."""
403 seen: dict[str, Path] = {}
404 violations = []
405 for name, path in commands:
406 if seen.setdefault(name, path) is not path:
407 violations.append(
408 f"{_rel(root, path)}: `{name}` is also defined by "
409 f"{_rel(root, seen[name])} — a command is one file, not two"
410 )
411 return violations
414def check_contracts(root: Path) -> list[str]:
415 """Run every rule over the plugin at *root*; return all violations."""
416 scripts_dir = root / SCRIPTS_DIR
417 commands = command_files(root)
418 names = {name for name, _ in commands}
419 procedures = [(path.stem, path) for path in sorted((root / PROMPTS_DIR).glob("*.md"))]
420 violations: list[str] = check_one_file_per_command(root, commands)
422 for group, is_command in ((commands, True), (procedures, False)):
423 for name, path in group:
424 rel = _rel(root, path)
425 text = path.read_text(encoding="utf-8")
426 blocks = bash_blocks(text)
427 violations += check_frontmatter(rel, text, is_command=is_command)
428 violations += check_bash_syntax(rel, blocks)
429 violations += check_script_calls(rel, blocks, scripts_dir)
430 violations += check_slash_commands(rel, text, names)
431 if is_command:
432 violations += check_allowed_tools(rel, text, blocks)
433 violations += check_model_invocation(rel, name, text)
435 tests_dir = root / "tests"
436 for path in prose_files(root):
437 rel = path.relative_to(root).as_posix()
438 text = path.read_text(encoding="utf-8")
439 violations += check_script_references(rel, text, scripts_dir)
440 violations += check_test_references(rel, text, tests_dir)
441 return violations
444def main(argv: list[str] | None = None) -> int:
445 """Entry point: check every command contract and return an exit code."""
446 parser = argparse.ArgumentParser(description="Check the plugin's prose command contracts.")
447 parser.add_argument("--root", default=".", help="Plugin root (default: current directory).")
448 args = parser.parse_args(argv)
450 root = Path(args.root).resolve()
451 violations = check_contracts(root)
452 if violations:
453 print("Command-contract check failed:", file=sys.stderr)
454 for violation in violations:
455 print(f" ✗ {violation}", file=sys.stderr)
456 return 1
458 count = (
459 len(command_files(root))
460 + len(list((root / PROMPTS_DIR).glob("*.md")))
461 + len(prose_files(root))
462 )
463 print(f"command contracts hold ({count} file(s) checked)")
464 return 0
467if __name__ == "__main__":
468 raise SystemExit(main())