From 5976ef89b4abb0101f3b775cb3fa810f41ea9f7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=90=D0=BD=D1=82=D0=BE=D0=BD=20=D0=A2=D1=80=D0=BE=D1=88?= =?UTF-8?q?=D0=B8=D0=BD?= Date: Tue, 6 Oct 2026 00:26:50 +0300 Subject: [PATCH] v3.22.0: the test suites (scripts grouped, ordered, with setup/teardown) A suite is the root object: every script belongs to exactly one suite, the suite holds the ordered member list by reference - the optional setup first, the members, the optional teardown last - and runs them in one card session, without resets and without resume. - test_suites.py: the suite store (test_suites.json, --test-suites), validated and atomic like the script store: roles normalised (setup first, teardown last, at most one of each, a script appears once per suite), per-member on_fail stop/continue and the suite-level require_adm. - test_scripts.py schema v2: suite_id/require_adm; list_by_suite()/orphans(); the import reports the id_map a bundle import remaps with. v3.21.0 scripts without a suite are attached once to an auto-created "Imported scripts" suite (migrate_scripts_to_suites, called from __main__). - API: /api/test/suites (+ update|delete|import - delete refused while non-empty; the bundle import imports the scripts first, remaps the suite references and skips unresolvable ones); the scripts endpoints gain suite_id/require_adm plus move|copy (delete detaches from the suite first; a suite update refuses a membership change). - runner: /api/test/run {suite_id} resolves the members and the preset, runs the ADM gate, then setup -> members (per-member on_fail) -> teardown. The teardown always runs on a stop or a failure (skipped only when the card is gone; it clears the stop flag for its duration); a card reset stops the suite, skips the remaining members and is flagged. Members' live steps land in the shared snapshot, their records (status/steps/note/log excerpt/ counters) in _TEST_RUN['suite']; the summary carries the counts, wall time, preset, the SCP80 counters before/after per keyset and the session flags. _test_log buffers the semantic lines (bounded) for the report's per-member excerpts. - ADM gate (_ensure_adm): a suite/script require_adm verifies the matched preset's key once per card session (the /api/verify-adm badge sets the same latch); a failure refuses the start, stamps the session (no automatic retry) and needs a manual verify. - PWA: the Test script pill becomes four views - the suite list (default), the suite editor (the script list: role badges, on_fail, up/down, per-script Run/Edit/Move/Copy/Delete, Add script with the templates), the script editor (the step list; a single run shows its result under the steps and is not preserved) and the suite report (per-member blocks with the step table, notes and run log; summary; export JSON/Markdown). Suite bundles export/ import; the legacy script export imports into "Imported scripts". EN/RU. - tests: the suite store/migration, the runner orchestration (order, on_fail, setup failure, stop, card reset, teardown, per-member logs) and the ADM gate (Python); the suite HTTP paths; the PWA report/Markdown/wiring (Node). Tests: 833 frontend + 794 Python; sw cache simple-v397. --- docs/api.md | 103 ++- docs/test-scripts.md | 21 +- frontend/help-ru.html | 3 +- frontend/help.html | 3 +- frontend/index.html | 959 ++++++++++++++++++++++------ frontend/sw.js | 2 +- frontend/tests/testscripts.test.js | 60 +- frontend/tests/testsuites.test.js | 151 +++++ pyproject.toml | 2 +- pysim_simple_server/__main__.py | 12 +- pysim_simple_server/server.py | 946 ++++++++++++++++++++++++++- pysim_simple_server/test_scripts.py | 87 ++- pysim_simple_server/test_suites.py | 339 ++++++++++ tests/test_fetch_routes.py | 234 ++++++- tests/test_test_scripts.py | 64 ++ tests/test_test_suites.py | 227 +++++++ tests/test_testscript.py | 253 +++++++- 17 files changed, 3178 insertions(+), 288 deletions(-) create mode 100644 frontend/tests/testsuites.test.js create mode 100644 pysim_simple_server/test_suites.py create mode 100644 tests/test_test_suites.py diff --git a/docs/api.md b/docs/api.md index ae58f96..40eb21d 100644 --- a/docs/api.md +++ b/docs/api.md @@ -48,15 +48,22 @@ a 3.x PWA). | `/api/ram-install-app` | POST | Single INSTALL [for install] / [for make selectable] for an already loaded package (no re-load) | | `/api/cap-info` | POST | Validate a `.cap` archive and estimate its code/NVRAM/RAM requirements (read-only) | | `/api/cap-compat` | POST | CAP compatibility test: LOAD only to the block completing the Import component (nothing committed) | -| `/api/test/run` | POST | Start a test script (actions + proactive expectations; `script` inline or `script_id` from the store) | -| `/api/test/status` | GET | Test script run state and per-step results | -| `/api/test/stop` | POST | Request a running test script to stop | +| `/api/test/run` | POST | Start a test script (actions + proactive expectations; `script` inline or `script_id` from the store) or a test suite (`suite_id`; ordered scripts, one card session) | +| `/api/test/status` | GET | Test script/suite run state, per-step results and the suite report | +| `/api/test/stop` | POST | Request a running test script/suite to stop | | `/api/test/clear` | POST | Clear the finished run report | | `/api/test/scripts` | GET | Test script store (server-side `test_scripts.json`): all scripts + the store path | -| `/api/test/scripts` | POST | Create a test script (`{name, steps}` or `{script}`) | -| `/api/test/scripts/update` | POST | Update a test script (`{id, script}`) | -| `/api/test/scripts/delete` | POST | Delete a test script (`{id}`) | -| `/api/test/scripts/import` | POST | Import scripts (`{scripts: [...], mode: merge\|replace}`; accepts the old localStorage export shape) | +| `/api/test/scripts` | POST | Create a test script (`{name, steps, suite_id}` or `{script}`; the script joins the suite) | +| `/api/test/scripts/update` | POST | Update a test script (`{id, script}`; a changed `suite_id` is a move) | +| `/api/test/scripts/delete` | POST | Delete a test script (`{id}`; detaches it from its suite first) | +| `/api/test/scripts/move` | POST | Move a script to another suite (`{id, suite_id}`) | +| `/api/test/scripts/copy` | POST | Copy a script into another suite (`{id, suite_id, name?}`) | +| `/api/test/scripts/import` | POST | Import scripts (`{scripts: [...], mode: merge\|replace, suite_id?}`; accepts the old localStorage export shape) | +| `/api/test/suites` | GET | Test suite store (server-side `test_suites.json`): all suites + the store path | +| `/api/test/suites` | POST | Create a suite (`{name, require_adm?, scripts?}` or `{suite}`) | +| `/api/test/suites/update` | POST | Update a suite (`{id, suite}`; the member set changes through the script endpoints) | +| `/api/test/suites/delete` | POST | Delete a suite (`{id}`; refused while it holds scripts) | +| `/api/test/suites/import` | POST | Import a suite bundle (`{suites, scripts, mode}`; script references remapped) | | `/api/sp-verify` | POST | Verify secured packet against pySim reference | | `/api/menu` | GET | Current STK menu (title + items + active) | | `/api/menu-select` | POST | ENVELOPE(Menu Selection) with item_id | @@ -786,17 +793,83 @@ same engine the runner uses, so a stored script can never fail at run start for a validation reason. ```json -{"path": "/home/user/.pysim-simple-server/test_scripts.json", "count": 1, "version": 1, - "scripts": [{"id": "…", "name": "applet RFM update", "steps": [...], +{"path": "/home/user/.pysim-simple-server/test_scripts.json", "count": 1, "version": 2, + "scripts": [{"id": "…", "name": "applet RFM update", "suite_id": "…", + "require_adm": false, "steps": [...], "created": 1690000000.0, "updated": 1690000000.0}]} ``` -`POST /api/test/scripts` creates one (`{name, steps}` or `{script: {...}}`), -`POST /api/test/scripts/update` replaces name/steps (`{id, script}`), -`POST /api/test/scripts/delete` removes (`{id}`) and -`POST /api/test/scripts/import` takes `{scripts: [...], mode: "merge"|"replace"}` -(the old localStorage export shape; invalid entries are reported in `errors` -instead of failing the whole import). +**Schema v2 (v3.22.0): every script belongs to exactly one test suite** +(`suite_id`; the suite is the root object) and may declare `require_adm` +(the run verifies the ADM before starting). A v1 entry loads with +`suite_id: null` and the server attaches it to an auto-created +"Imported scripts" suite on startup. + +`POST /api/test/scripts` creates one (`{name, steps, suite_id, require_adm?}` +or `{script: {...}}`; a valid `suite_id` is required - the script joins the +suite's member list) and `POST /api/test/scripts/update` replaces +name/steps/require_adm (`{id, script}`). A `suite_id` that differs from the +current one is a **move** (both suites are updated in the request). +`POST /api/test/scripts/delete` (`{id}`) detaches the script from its suite +first - the suite's member list is the only reference, so no dangling id is +left. `POST /api/test/scripts/move` (`{id, suite_id}`) moves a script to +another suite (the target appends it as a member, keeping the `on_fail` +policy) and `POST /api/test/scripts/copy` (`{id, suite_id, name?}`) clones it +into the target. `POST /api/test/scripts/import` takes +`{scripts: [...], mode: "merge"|"replace", suite_id?}` (the old localStorage +export shape; invalid entries are reported in `errors` instead of failing the +whole import; `suite_id` attaches the imported scripts to that suite). + +### `GET /api/test/suites` and the suite store + +Test suites live next to the scripts (`~/.pysim-simple-server/test_suites.json`, +`--test-suites PATH`): a suite is the ordered list of its scripts - the +optional `setup` first, the `member` scripts, the optional `teardown` last - +plus the per-member `on_fail` policy and the suite-level `require_adm`. + +```json +{"path": "/home/user/.pysim-simple-server/test_suites.json", "count": 1, "version": 1, + "suites": [{"id": "…", "name": "alfa regression", "require_adm": true, + "scripts": [{"script_id": "…", "role": "setup", "on_fail": "stop"}, + {"script_id": "…", "role": "member", "on_fail": "stop"}, + {"script_id": "…", "role": "teardown", "on_fail": "stop"}], + "created": 1690000000.0, "updated": 1690000000.0}]} +``` + +The store normalises the order (setup first, teardown last, at most one of +each; a script appears once per suite) and validates `on_fail` as +`stop` (default) or `continue` - "which failures cascade" is a visible +per-suite decision. + +`POST /api/test/suites` creates one (`{name, require_adm?, scripts?}` or +`{suite: {...}}`), `POST /api/test/suites/update` replaces +name/require_adm/scripts (`{id, suite}`; the **member set** changes through +the script endpoints, not here), `POST /api/test/suites/delete` (`{id}`) +removes it - refused while the suite still holds scripts - and +`POST /api/test/suites/import` takes +`{suites: [...], scripts: [...], mode: "merge"|"replace"}`: the scripts are +imported first (ids preserved; a collision gets a fresh uuid and the suites' +references are remapped), then the suites; an unresolvable reference is +reported in `errors` and the suite is skipped. + +### `POST /api/test/run` (single script or suite) + +`{script, script_id?, preset_id?, require_adm?}` runs one script (the PWA +sends the edited copy; the server validates and runs it). `{suite_id, +preset_id?}` runs a suite: its scripts in order, one card session, no resume +- the ADM prerequisite first (suite- or member-level `require_adm`: the +matched preset's key is verified **once**; a failure refuses the start, is +not retried automatically, and requires a manual verify), then setup, +members (per-member `on_fail`), teardown. **The teardown runs on every stop** +as long as the card is in the reader; a card reset (session change) stops the +suite, skips the remaining members and is flagged in the report. + +The suite run's snapshot adds `suite`: the member records (`status`, `steps`, +`note`, `log` - the semantic run-log excerpt), the `summary` (ok/warning/ +error/skipped counts, `wall_ms`, `stopped`, `session_changed`, the SCP80 +`counters_before`/`counters_after` per keyset) and the run's `adm` state. +The report is exported from the PWA as JSON or Markdown; the run state lives +in the server until `POST /api/test/clear` (no history). ### `GET /api/test/status` diff --git a/docs/test-scripts.md b/docs/test-scripts.md index f3f04b7..68463f0 100644 --- a/docs/test-scripts.md +++ b/docs/test-scripts.md @@ -12,12 +12,31 @@ localStorage export imports as-is), and any API client can create or run them (`POST /api/test/run` with an inline `script` or a stored `script_id` - see `docs/api.md`). While a run is active the card belongs to the script. +**Test suites** (v3.22.0) group the scripts: a suite is the root object (every +script belongs to exactly one suite, `suite_id` in the script store) and the +runner executes its scripts in order - the optional **setup** first, the +**member** scripts, the optional **teardown** last - in **one card session, +without resets and without resume**. Per member `on_fail` decides whether a +failure stops the suite (`stop`, default) or the next script runs +(`continue`); the **teardown always runs** on a stop or a failure (its job is +to leave the card ready for a new test) as long as the card is in the reader, +and a card reset (session change) stops the suite and is flagged in the +report. A suite (or a single script) may require the **ADM** verified: the +matched preset's key is tried **once** at the start - a failure refuses the +run and needs a manual verify (the ADM badge) before the next attempt. The +suite run's report (the PWA's report view, exportable as JSON or Markdown) +carries every member's step table, failing notes and run-log excerpt, plus the +summary (pass/warn/fail/skipped, wall time, the SCP80 counters before/after +per keyset and the card-session flag). The suites live in +`~/.pysim-simple-server/test_suites.json` (`--test-suites`). + **Format**: server ≥ 3.15.0 (`kvn`, `por` and `files` checks), scripts stored server-side since 3.16.0, `menu-select` by text and the spec-order TERMINAL RESPONSE codings since 3.17.0, the semantic `event` action since 3.18.0, the `envelope` action's `src` override since 3.18.1, the inline `por` object, the `alpha`/`sms` checks, `scp80.params.format` and the case-insensitive -`menu-select` text match since 3.19.0. +`menu-select` text match since 3.19.0, the `proactive` cleanup action and the +ok-answering bounded cleanup since 3.21.0, the suites since 3.22.0. ## Writing a test for an applet diff --git a/frontend/help-ru.html b/frontend/help-ru.html index 58fffbf..f8b9e18 100644 --- a/frontend/help-ru.html +++ b/frontend/help-ru.html @@ -562,11 +562,12 @@

8.9 Тест-скрипт

Тест-скрипт — упорядоченный список действий и ожиданий проактивных команд, выполняемый на подключённой карте на стороне сервера. Пока запуск активен, карта принадлежит скрипту: остальные операции с картой отклоняются (HTTP 409), фоновый опрос STATUS приостанавливается; отчёт о запуске хранится в памяти (в том числе после перезагрузки страницы) до нажатия Очистить отчёт. Скрипты хранятся в хранилище на сервере (~/.pysim-simple-server/test_scripts.json, --test-scripts), поэтому все браузеры работают с одним набором; изменения записываются в хранилище по мере правки (кнопки «Сохранить» нет — черновик без валидного шага остаётся локальным); экспорт/импорт JSON остаётся для резервных копий и переноса скриптов между стендами (старый экспорт из localStorage импортируется как есть).

+

У вкладки четыре представления: список наборов (по умолчанию; Добавить набор / Редактировать / Запустить / Экспорт / Удалить), редактор набора (скрипты набора в порядке выполнения — необязательный setup первым и teardown последним, с пометками ролей; у каждого скрипта Запустить/Редактировать/Переместить/Скопировать/Удалить, политика при ошибке остановить (по умолчанию) / продолжить и Требуется ADM), редактор скрипта (список шагов; кнопка Старт выполняет только этот скрипт и показывает результат под шагами — он не сохраняется) и отчёт набора (после запуска набора: статус каждого скрипта, таблица шагов с ожидаемым/фактическим, примечание об ошибке и выдержка из журнала запуска; сводка с успешно/предупреждения/ошибки/пропущено, время выполнения, счётчики SCP80 до → после по каждому keyset, пресет и предупреждение о смене сессии карты; экспорт в JSON или Markdown). Запуск набора выполняет его скрипты по порядку в одной сессии карты, без сбросов и без продолжения с места остановки; при ошибке решает, остановится ли набор на этом скрипте или запустится следующий, teardown выполняется всегда при остановке или ошибке (пока карта в ридере), чтобы следующий тест начался с чистого состояния, а сброс карты останавливает набор и отмечается в отчёте. Требуется ADM (у набора или скрипта) проверяет ключ ADM из подобранного пресета один раз перед запуском — ошибка отклоняет запуск (проверьте вручную через значок ADM; автоматических повторов нет). Скрипты хранятся в хранилище наборов (~/.pysim-simple-server/test_suites.json, --test-suites); каждый скрипт принадлежит ровно одному набору, и скрипт можно переместить или скопировать в другой набор (экспорт набора — это пакет с набором и его скриптами; старый экспорт только скриптов импортируется в автоматически созданный набор «Imported scripts»).

Каждый шаг пишет семантическую строку в журнал сервера под своим номером (TEST-RUN step N: …), запрос перед ответом: отправленный APDU/данные и полученный SW, для SCP80 — открытый C-APDU, собранный защищённый пакет и R-APDU из PoR, для ожидания — тройка FETCH / команда / TERMINAL RESPONSE, очистка proactive — каждый раунд STATUS/FETCH/команда/TR, а очистка в конце запуска — поглощённые команды (отвечаются ok — cancel отправляется только по явному запросу скрипта), а для выбора пункта меню — полный ENVELOPE с определённым пунктом. Запустите сервер с --apdu-trace, чтобы добавить и журнал сырых транспортных APDU.

diff --git a/frontend/help.html b/frontend/help.html index 033cb78..6ed23e3 100644 --- a/frontend/help.html +++ b/frontend/help.html @@ -561,11 +561,12 @@

8.9 Test script

A test script is an ordered list of actions and proactive-command expectations executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP 409) and background STATUS polling is paused; the run report is kept in memory (also after reloading the page) until Clear report. Scripts live in the server-side store (~/.pysim-simple-server/test_scripts.json, --test-scripts), so every browser manages the same set; edits are written to the store as you change them (there is no Save button — a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).

+

The pill has four views: the suite list (default; Add suite / Edit / Run / Export / Delete), the suite editor (the suite's scripts in run order — the optional setup first and the teardown last, marked with role badges; per script Run/Edit/Move/Copy/Delete, the per-member on failure policy stop (default) / continue, and ADM required), the script editor (the step list; its Start runs that single script and shows the result under the steps — not preserved) and the suite report (after a suite run: every script's status, its step table with expected/actual, the failing note and a run-log excerpt; the summary with passed/warnings/failed/skipped, wall time, the SCP80 counters before → after per keyset, the preset and a card-session-changed warning; export as JSON or Markdown). A suite run executes its scripts in order in one card session, without resets and without resume; on failure decides whether the suite stops at that script or runs the next one, the teardown always runs on a stop or a failure (while the card is in the reader) so the next test starts clean, and a card reset stops the suite and is flagged in the report. ADM required (on the suite or a script) verifies the matched preset's ADM key once before the start — a failure refuses the run (verify manually with the ADM badge; nothing is retried automatically). Scripts live in the suite's store (~/.pysim-simple-server/test_suites.json, --test-suites); every script belongs to exactly one suite, and a script can be moved or copied to another suite (a bundle export carries the suite plus its scripts; the old script-only export imports into an auto-created “Imported scripts” suite).

  • Actions — Envelope (Event Download: raw event type + data, with an optional source override — 82 terminal / 83 network) or Event (the semantic forms: Location status, Access technology change, Network rejection, Data connection status change — the fields are edited as a JSON object in the step editor, and the Phone tab's event form shows the built hex for the other events), Menu Selection (item id 1–255, or the item's text with an exact/contains match, case-insensitive by default — the step editor's Match case choice switches to an exact-case match; the text is resolved against the card's menu, which follows the applet's install parameters), UPDATE/READ file (path + mode; reads verify the file data), raw APDU (no auto-handler), SCP80 (C-APDU or a pre-built secured packet, optional keyset number (kvn) / TAR / SPI1 / SPI2 overrides and a command format — compact, or the TS 102 226 §5.2.1 Command Scripting template expanded (AA) / expanded AE 80 around the C-APDU; the keys and the counter come from the card preset matched by the equipped card's ICCID — the run names the stored preset, so it always uses its keysets, TARs and counter — and the counter is written back to it after the run), STATUS (with attempts > 1 it polls until the card announces a command) and Proactive cleanup (poll STATUS and consume every announced command — the TERMINAL RESPONSE result for each, optionally a different result for the first; the attempt/interval bound; an optional require assertion over what was drained. An empty poll passes, the final STATUS must be idle, and a still-pending command is handed to the next step — use it at the start, in the middle or at the end of a script to make sure the card has stopped sending). Every action has a response check: expected SW and data (exact, or a mask with ? nibble wildcards — 91??) plus a PoR check for SCP80 (any / no PoR / por_ok, or a custom assertion of the decoded PoR — status / SW / data; the SEND SHORT MESSAGE transport works the same).
  • Expectations — the card must have announced a proactive command in the previous step (SW 91XX, TS 102 221 §7.4.2.1 / TS 102 223 §6.3); a 9000 means “no command”: the step fails at its fail level — Error stops the script, Warning continues (the tolerant “consume a command if there is one” form) — and a STATUS action with attempts > 1 polls when the applet delivers on poll. The command type, qualifier and contents are verified: text / item / raw checks, an Alpha check (the command's Alpha identifier, tag 05/85 — distinct from the 8D text string), an SMS TPDU check (the submit TPDU of a SEND SHORT MESSAGE: destination / PID / DCS / UDL / user data — for an applet's own SMS), a PoR check (a SEND SHORT MESSAGE carries the PoR when the packet requested PoR-in-submit, SPI2 bit 20: the response status, the R-APDU SW and the response data; make sure an SCP80 step ran first — the R-APDU SW lives inside the response payload: an applet's own TAR returns application data, so check its data and leave the SW empty; a refused ENVELOPE (62XX/63XX) may carry no response packet at all — leave the PoR check at any or expect no PoR for such a step) and a Files check (the File List of a REFRESH, e.g. qualifier 00 = NAA initialization with full file change notification). The scripted TERMINAL RESPONSE (result, item identifier, text, extra TLVs) is part of the instruction.
  • Fail levels — a mismatch flags the step as Error (terminates the script) or Warning (continues); every step failure honours it, including an expectation that finds no pending command; each check is reported with expected vs actual values.
    • -
  • Templates — Empty, STK menu browsing, Applet via SCP80; the step editor is form-based (the event action's fields are a JSON object) and individual scripts export/import as JSON.
    • +
  • Templates — Empty, STK menu browsing, Applet via SCP80; the step editor is form-based (the event action's fields are a JSON object), and a suite exports/imports as a JSON bundle (the suite plus its scripts).

Every step logs a semantic line to the server log under its step number (TEST-RUN step N: …), request before response: the APDU/data sent and the SW received, for SCP80 the plaintext C-APDU, the built secured packet and the PoR's R-APDU, for an expectation the FETCH / command / TERMINAL RESPONSE trio, a proactive cleanup every STATUS/FETCH/command/TR round and the run cleanup the commands it drained (answered ok — a cancel is only sent when a script asks for it), and for a menu selection the complete ENVELOPE with the resolved item. Start the server with --apdu-trace to add the raw transport APDU log as well.

diff --git a/frontend/index.html b/frontend/index.html index d626c4b..c909a80 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -1372,46 +1372,72 @@