diff --git a/docs/api.md b/docs/api.md index ae58f96..40eb21d 100644 --- a/docs/api.md +++ b/docs/api.md @@ -48,15 +48,22 @@ a 3.x PWA). | `/api/ram-install-app` | POST | Single INSTALL [for install] / [for make selectable] for an already loaded package (no re-load) | | `/api/cap-info` | POST | Validate a `.cap` archive and estimate its code/NVRAM/RAM requirements (read-only) | | `/api/cap-compat` | POST | CAP compatibility test: LOAD only to the block completing the Import component (nothing committed) | -| `/api/test/run` | POST | Start a test script (actions + proactive expectations; `script` inline or `script_id` from the store) | -| `/api/test/status` | GET | Test script run state and per-step results | -| `/api/test/stop` | POST | Request a running test script to stop | +| `/api/test/run` | POST | Start a test script (actions + proactive expectations; `script` inline or `script_id` from the store) or a test suite (`suite_id`; ordered scripts, one card session) | +| `/api/test/status` | GET | Test script/suite run state, per-step results and the suite report | +| `/api/test/stop` | POST | Request a running test script/suite to stop | | `/api/test/clear` | POST | Clear the finished run report | | `/api/test/scripts` | GET | Test script store (server-side `test_scripts.json`): all scripts + the store path | -| `/api/test/scripts` | POST | Create a test script (`{name, steps}` or `{script}`) | -| `/api/test/scripts/update` | POST | Update a test script (`{id, script}`) | -| `/api/test/scripts/delete` | POST | Delete a test script (`{id}`) | -| `/api/test/scripts/import` | POST | Import scripts (`{scripts: [...], mode: merge\|replace}`; accepts the old localStorage export shape) | +| `/api/test/scripts` | POST | Create a test script (`{name, steps, suite_id}` or `{script}`; the script joins the suite) | +| `/api/test/scripts/update` | POST | Update a test script (`{id, script}`; a changed `suite_id` is a move) | +| `/api/test/scripts/delete` | POST | Delete a test script (`{id}`; detaches it from its suite first) | +| `/api/test/scripts/move` | POST | Move a script to another suite (`{id, suite_id}`) | +| `/api/test/scripts/copy` | POST | Copy a script into another suite (`{id, suite_id, name?}`) | +| `/api/test/scripts/import` | POST | Import scripts (`{scripts: [...], mode: merge\|replace, suite_id?}`; accepts the old localStorage export shape) | +| `/api/test/suites` | GET | Test suite store (server-side `test_suites.json`): all suites + the store path | +| `/api/test/suites` | POST | Create a suite (`{name, require_adm?, scripts?}` or `{suite}`) | +| `/api/test/suites/update` | POST | Update a suite (`{id, suite}`; the member set changes through the script endpoints) | +| `/api/test/suites/delete` | POST | Delete a suite (`{id}`; refused while it holds scripts) | +| `/api/test/suites/import` | POST | Import a suite bundle (`{suites, scripts, mode}`; script references remapped) | | `/api/sp-verify` | POST | Verify secured packet against pySim reference | | `/api/menu` | GET | Current STK menu (title + items + active) | | `/api/menu-select` | POST | ENVELOPE(Menu Selection) with item_id | @@ -786,17 +793,83 @@ same engine the runner uses, so a stored script can never fail at run start for a validation reason. ```json -{"path": "/home/user/.pysim-simple-server/test_scripts.json", "count": 1, "version": 1, - "scripts": [{"id": "…", "name": "applet RFM update", "steps": [...], +{"path": "/home/user/.pysim-simple-server/test_scripts.json", "count": 1, "version": 2, + "scripts": [{"id": "…", "name": "applet RFM update", "suite_id": "…", + "require_adm": false, "steps": [...], "created": 1690000000.0, "updated": 1690000000.0}]} ``` -`POST /api/test/scripts` creates one (`{name, steps}` or `{script: {...}}`), -`POST /api/test/scripts/update` replaces name/steps (`{id, script}`), -`POST /api/test/scripts/delete` removes (`{id}`) and -`POST /api/test/scripts/import` takes `{scripts: [...], mode: "merge"|"replace"}` -(the old localStorage export shape; invalid entries are reported in `errors` -instead of failing the whole import). +**Schema v2 (v3.22.0): every script belongs to exactly one test suite** +(`suite_id`; the suite is the root object) and may declare `require_adm` +(the run verifies the ADM before starting). A v1 entry loads with +`suite_id: null` and the server attaches it to an auto-created +"Imported scripts" suite on startup. + +`POST /api/test/scripts` creates one (`{name, steps, suite_id, require_adm?}` +or `{script: {...}}`; a valid `suite_id` is required - the script joins the +suite's member list) and `POST /api/test/scripts/update` replaces +name/steps/require_adm (`{id, script}`). A `suite_id` that differs from the +current one is a **move** (both suites are updated in the request). +`POST /api/test/scripts/delete` (`{id}`) detaches the script from its suite +first - the suite's member list is the only reference, so no dangling id is +left. `POST /api/test/scripts/move` (`{id, suite_id}`) moves a script to +another suite (the target appends it as a member, keeping the `on_fail` +policy) and `POST /api/test/scripts/copy` (`{id, suite_id, name?}`) clones it +into the target. `POST /api/test/scripts/import` takes +`{scripts: [...], mode: "merge"|"replace", suite_id?}` (the old localStorage +export shape; invalid entries are reported in `errors` instead of failing the +whole import; `suite_id` attaches the imported scripts to that suite). + +### `GET /api/test/suites` and the suite store + +Test suites live next to the scripts (`~/.pysim-simple-server/test_suites.json`, +`--test-suites PATH`): a suite is the ordered list of its scripts - the +optional `setup` first, the `member` scripts, the optional `teardown` last - +plus the per-member `on_fail` policy and the suite-level `require_adm`. + +```json +{"path": "/home/user/.pysim-simple-server/test_suites.json", "count": 1, "version": 1, + "suites": [{"id": "…", "name": "alfa regression", "require_adm": true, + "scripts": [{"script_id": "…", "role": "setup", "on_fail": "stop"}, + {"script_id": "…", "role": "member", "on_fail": "stop"}, + {"script_id": "…", "role": "teardown", "on_fail": "stop"}], + "created": 1690000000.0, "updated": 1690000000.0}]} +``` + +The store normalises the order (setup first, teardown last, at most one of +each; a script appears once per suite) and validates `on_fail` as +`stop` (default) or `continue` - "which failures cascade" is a visible +per-suite decision. + +`POST /api/test/suites` creates one (`{name, require_adm?, scripts?}` or +`{suite: {...}}`), `POST /api/test/suites/update` replaces +name/require_adm/scripts (`{id, suite}`; the **member set** changes through +the script endpoints, not here), `POST /api/test/suites/delete` (`{id}`) +removes it - refused while the suite still holds scripts - and +`POST /api/test/suites/import` takes +`{suites: [...], scripts: [...], mode: "merge"|"replace"}`: the scripts are +imported first (ids preserved; a collision gets a fresh uuid and the suites' +references are remapped), then the suites; an unresolvable reference is +reported in `errors` and the suite is skipped. + +### `POST /api/test/run` (single script or suite) + +`{script, script_id?, preset_id?, require_adm?}` runs one script (the PWA +sends the edited copy; the server validates and runs it). `{suite_id, +preset_id?}` runs a suite: its scripts in order, one card session, no resume +- the ADM prerequisite first (suite- or member-level `require_adm`: the +matched preset's key is verified **once**; a failure refuses the start, is +not retried automatically, and requires a manual verify), then setup, +members (per-member `on_fail`), teardown. **The teardown runs on every stop** +as long as the card is in the reader; a card reset (session change) stops the +suite, skips the remaining members and is flagged in the report. + +The suite run's snapshot adds `suite`: the member records (`status`, `steps`, +`note`, `log` - the semantic run-log excerpt), the `summary` (ok/warning/ +error/skipped counts, `wall_ms`, `stopped`, `session_changed`, the SCP80 +`counters_before`/`counters_after` per keyset) and the run's `adm` state. +The report is exported from the PWA as JSON or Markdown; the run state lives +in the server until `POST /api/test/clear` (no history). ### `GET /api/test/status` diff --git a/docs/test-scripts.md b/docs/test-scripts.md index f3f04b7..68463f0 100644 --- a/docs/test-scripts.md +++ b/docs/test-scripts.md @@ -12,12 +12,31 @@ localStorage export imports as-is), and any API client can create or run them (`POST /api/test/run` with an inline `script` or a stored `script_id` - see `docs/api.md`). While a run is active the card belongs to the script. +**Test suites** (v3.22.0) group the scripts: a suite is the root object (every +script belongs to exactly one suite, `suite_id` in the script store) and the +runner executes its scripts in order - the optional **setup** first, the +**member** scripts, the optional **teardown** last - in **one card session, +without resets and without resume**. Per member `on_fail` decides whether a +failure stops the suite (`stop`, default) or the next script runs +(`continue`); the **teardown always runs** on a stop or a failure (its job is +to leave the card ready for a new test) as long as the card is in the reader, +and a card reset (session change) stops the suite and is flagged in the +report. A suite (or a single script) may require the **ADM** verified: the +matched preset's key is tried **once** at the start - a failure refuses the +run and needs a manual verify (the ADM badge) before the next attempt. The +suite run's report (the PWA's report view, exportable as JSON or Markdown) +carries every member's step table, failing notes and run-log excerpt, plus the +summary (pass/warn/fail/skipped, wall time, the SCP80 counters before/after +per keyset and the card-session flag). The suites live in +`~/.pysim-simple-server/test_suites.json` (`--test-suites`). + **Format**: server ≥ 3.15.0 (`kvn`, `por` and `files` checks), scripts stored server-side since 3.16.0, `menu-select` by text and the spec-order TERMINAL RESPONSE codings since 3.17.0, the semantic `event` action since 3.18.0, the `envelope` action's `src` override since 3.18.1, the inline `por` object, the `alpha`/`sms` checks, `scp80.params.format` and the case-insensitive -`menu-select` text match since 3.19.0. +`menu-select` text match since 3.19.0, the `proactive` cleanup action and the +ok-answering bounded cleanup since 3.21.0, the suites since 3.22.0. ## Writing a test for an applet diff --git a/frontend/help-ru.html b/frontend/help-ru.html index 58fffbf..f8b9e18 100644 --- a/frontend/help-ru.html +++ b/frontend/help-ru.html @@ -562,11 +562,12 @@

8.9 Тест-скрипт

Тест-скрипт — упорядоченный список действий и ожиданий проактивных команд, выполняемый на подключённой карте на стороне сервера. Пока запуск активен, карта принадлежит скрипту: остальные операции с картой отклоняются (HTTP 409), фоновый опрос STATUS приостанавливается; отчёт о запуске хранится в памяти (в том числе после перезагрузки страницы) до нажатия Очистить отчёт. Скрипты хранятся в хранилище на сервере (~/.pysim-simple-server/test_scripts.json, --test-scripts), поэтому все браузеры работают с одним набором; изменения записываются в хранилище по мере правки (кнопки «Сохранить» нет — черновик без валидного шага остаётся локальным); экспорт/импорт JSON остаётся для резервных копий и переноса скриптов между стендами (старый экспорт из localStorage импортируется как есть).

+

У вкладки четыре представления: список наборов (по умолчанию; Добавить набор / Редактировать / Запустить / Экспорт / Удалить), редактор набора (скрипты набора в порядке выполнения — необязательный setup первым и teardown последним, с пометками ролей; у каждого скрипта Запустить/Редактировать/Переместить/Скопировать/Удалить, политика при ошибке остановить (по умолчанию) / продолжить и Требуется ADM), редактор скрипта (список шагов; кнопка Старт выполняет только этот скрипт и показывает результат под шагами — он не сохраняется) и отчёт набора (после запуска набора: статус каждого скрипта, таблица шагов с ожидаемым/фактическим, примечание об ошибке и выдержка из журнала запуска; сводка с успешно/предупреждения/ошибки/пропущено, время выполнения, счётчики SCP80 до → после по каждому keyset, пресет и предупреждение о смене сессии карты; экспорт в JSON или Markdown). Запуск набора выполняет его скрипты по порядку в одной сессии карты, без сбросов и без продолжения с места остановки; при ошибке решает, остановится ли набор на этом скрипте или запустится следующий, teardown выполняется всегда при остановке или ошибке (пока карта в ридере), чтобы следующий тест начался с чистого состояния, а сброс карты останавливает набор и отмечается в отчёте. Требуется ADM (у набора или скрипта) проверяет ключ ADM из подобранного пресета один раз перед запуском — ошибка отклоняет запуск (проверьте вручную через значок ADM; автоматических повторов нет). Скрипты хранятся в хранилище наборов (~/.pysim-simple-server/test_suites.json, --test-suites); каждый скрипт принадлежит ровно одному набору, и скрипт можно переместить или скопировать в другой набор (экспорт набора — это пакет с набором и его скриптами; старый экспорт только скриптов импортируется в автоматически созданный набор «Imported scripts»).

Каждый шаг пишет семантическую строку в журнал сервера под своим номером (TEST-RUN step N: …), запрос перед ответом: отправленный APDU/данные и полученный SW, для SCP80 — открытый C-APDU, собранный защищённый пакет и R-APDU из PoR, для ожидания — тройка FETCH / команда / TERMINAL RESPONSE, очистка proactive — каждый раунд STATUS/FETCH/команда/TR, а очистка в конце запуска — поглощённые команды (отвечаются ok — cancel отправляется только по явному запросу скрипта), а для выбора пункта меню — полный ENVELOPE с определённым пунктом. Запустите сервер с --apdu-trace, чтобы добавить и журнал сырых транспортных APDU.

diff --git a/frontend/help.html b/frontend/help.html index 033cb78..6ed23e3 100644 --- a/frontend/help.html +++ b/frontend/help.html @@ -561,11 +561,12 @@

8.9 Test script

A test script is an ordered list of actions and proactive-command expectations executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP 409) and background STATUS polling is paused; the run report is kept in memory (also after reloading the page) until Clear report. Scripts live in the server-side store (~/.pysim-simple-server/test_scripts.json, --test-scripts), so every browser manages the same set; edits are written to the store as you change them (there is no Save button — a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).

+

The pill has four views: the suite list (default; Add suite / Edit / Run / Export / Delete), the suite editor (the suite's scripts in run order — the optional setup first and the teardown last, marked with role badges; per script Run/Edit/Move/Copy/Delete, the per-member on failure policy stop (default) / continue, and ADM required), the script editor (the step list; its Start runs that single script and shows the result under the steps — not preserved) and the suite report (after a suite run: every script's status, its step table with expected/actual, the failing note and a run-log excerpt; the summary with passed/warnings/failed/skipped, wall time, the SCP80 counters before → after per keyset, the preset and a card-session-changed warning; export as JSON or Markdown). A suite run executes its scripts in order in one card session, without resets and without resume; on failure decides whether the suite stops at that script or runs the next one, the teardown always runs on a stop or a failure (while the card is in the reader) so the next test starts clean, and a card reset stops the suite and is flagged in the report. ADM required (on the suite or a script) verifies the matched preset's ADM key once before the start — a failure refuses the run (verify manually with the ADM badge; nothing is retried automatically). Scripts live in the suite's store (~/.pysim-simple-server/test_suites.json, --test-suites); every script belongs to exactly one suite, and a script can be moved or copied to another suite (a bundle export carries the suite plus its scripts; the old script-only export imports into an auto-created “Imported scripts” suite).

Every step logs a semantic line to the server log under its step number (TEST-RUN step N: …), request before response: the APDU/data sent and the SW received, for SCP80 the plaintext C-APDU, the built secured packet and the PoR's R-APDU, for an expectation the FETCH / command / TERMINAL RESPONSE trio, a proactive cleanup every STATUS/FETCH/command/TR round and the run cleanup the commands it drained (answered ok — a cancel is only sent when a script asks for it), and for a menu selection the complete ENVELOPE with the resolved item. Start the server with --apdu-trace to add the raw transport APDU log as well.

diff --git a/frontend/index.html b/frontend/index.html index d626c4b..c909a80 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -1372,46 +1372,72 @@