v3.22.0: the test suites (scripts grouped, ordered, with setup/teardown)

A suite is the root object: every script belongs to exactly one suite, the
suite holds the ordered member list by reference - the optional setup first,
the members, the optional teardown last - and runs them in one card session,
without resets and without resume.

- test_suites.py: the suite store (test_suites.json, --test-suites), validated
  and atomic like the script store: roles normalised (setup first, teardown
  last, at most one of each, a script appears once per suite), per-member
  on_fail stop/continue and the suite-level require_adm.
- test_scripts.py schema v2: suite_id/require_adm; list_by_suite()/orphans();
  the import reports the id_map a bundle import remaps with.  v3.21.0 scripts
  without a suite are attached once to an auto-created "Imported scripts"
  suite (migrate_scripts_to_suites, called from __main__).
- API: /api/test/suites (+ update|delete|import - delete refused while
  non-empty; the bundle import imports the scripts first, remaps the suite
  references and skips unresolvable ones); the scripts endpoints gain
  suite_id/require_adm plus move|copy (delete detaches from the suite first;
  a suite update refuses a membership change).
- runner: /api/test/run {suite_id} resolves the members and the preset, runs
  the ADM gate, then setup -> members (per-member on_fail) -> teardown.  The
  teardown always runs on a stop or a failure (skipped only when the card is
  gone; it clears the stop flag for its duration); a card reset stops the
  suite, skips the remaining members and is flagged.  Members' live steps land
  in the shared snapshot, their records (status/steps/note/log excerpt/
  counters) in _TEST_RUN['suite']; the summary carries the counts, wall time,
  preset, the SCP80 counters before/after per keyset and the session flags.
  _test_log buffers the semantic lines (bounded) for the report's per-member
  excerpts.
- ADM gate (_ensure_adm): a suite/script require_adm verifies the matched
  preset's key once per card session (the /api/verify-adm badge sets the same
  latch); a failure refuses the start, stamps the session (no automatic
  retry) and needs a manual verify.
- PWA: the Test script pill becomes four views - the suite list (default),
  the suite editor (the script list: role badges, on_fail, up/down, per-script
  Run/Edit/Move/Copy/Delete, Add script with the templates), the script editor
  (the step list; a single run shows its result under the steps and is not
  preserved) and the suite report (per-member blocks with the step table,
  notes and run log; summary; export JSON/Markdown).  Suite bundles export/
  import; the legacy script export imports into "Imported scripts".  EN/RU.
- tests: the suite store/migration, the runner orchestration (order, on_fail,
  setup failure, stop, card reset, teardown, per-member logs) and the ADM gate
  (Python); the suite HTTP paths; the PWA report/Markdown/wiring (Node).
  Tests: 833 frontend + 794 Python; sw cache simple-v397.
This commit is contained in:
2026-10-06 00:26:50 +03:00
parent e6ced47323
commit 5976ef89b4
17 changed files with 3178 additions and 288 deletions
+20 -1
View File
@@ -12,12 +12,31 @@ localStorage export imports as-is), and any API client can create or run them
(`POST /api/test/run` with an inline `script` or a stored `script_id` - see
`docs/api.md`). While a run is active the card belongs to the script.
**Test suites** (v3.22.0) group the scripts: a suite is the root object (every
script belongs to exactly one suite, `suite_id` in the script store) and the
runner executes its scripts in order - the optional **setup** first, the
**member** scripts, the optional **teardown** last - in **one card session,
without resets and without resume**. Per member `on_fail` decides whether a
failure stops the suite (`stop`, default) or the next script runs
(`continue`); the **teardown always runs** on a stop or a failure (its job is
to leave the card ready for a new test) as long as the card is in the reader,
and a card reset (session change) stops the suite and is flagged in the
report. A suite (or a single script) may require the **ADM** verified: the
matched preset's key is tried **once** at the start - a failure refuses the
run and needs a manual verify (the ADM badge) before the next attempt. The
suite run's report (the PWA's report view, exportable as JSON or Markdown)
carries every member's step table, failing notes and run-log excerpt, plus the
summary (pass/warn/fail/skipped, wall time, the SCP80 counters before/after
per keyset and the card-session flag). The suites live in
`~/.pysim-simple-server/test_suites.json` (`--test-suites`).
**Format**: server ≥ 3.15.0 (`kvn`, `por` and `files` checks), scripts stored
server-side since 3.16.0, `menu-select` by text and the spec-order TERMINAL
RESPONSE codings since 3.17.0, the semantic `event` action since 3.18.0, the
`envelope` action's `src` override since 3.18.1, the inline `por` object, the
`alpha`/`sms` checks, `scp80.params.format` and the case-insensitive
`menu-select` text match since 3.19.0.
`menu-select` text match since 3.19.0, the `proactive` cleanup action and the
ok-answering bounded cleanup since 3.21.0, the suites since 3.22.0.
## Writing a test for an applet