v3.37.0: load testing for test scripts and suites

POST /api/test/run takes an optional iterations count (the PWA's
Iterations field): the run repeats, successful iterations are discarded
(steps/log cleared, only load.done and the timing grow), the first error
stops the loop and keeps that iteration's report, a stop ends it after
the current iteration, and a card session change between iterations is
an error.  The SCP80 counter state is shared across iterations, so
packets continue from what the previous run consumed.

The status gains a load block (iteration/done/failed_iteration/
elapsed_ms/avg_ms; disabled for a normal run).  The suite runner was
split into a per-iteration body (_test_suite_execute_once) + the
single-run finaliser, and two load workers own the loops.

Review fixes: a raised runner error finalises as error (no stuck
running state), a suite-internal session change is an error, a stopped
member ends the suite loop, the final status keeps a warning from any
iteration, avg_ms covers the attempted iterations, and each suite
iteration's counters_before is read fresh.

1041 Python + 1069 frontend; simple-v475.
This commit is contained in:
2026-10-23 23:44:42 +03:00
parent 851ff65af6
commit 2181ef6bbf
13 changed files with 937 additions and 64 deletions
+25
View File
@@ -9,6 +9,31 @@ every change carries a descriptive commit message.
## [Unreleased]
## [3.37.0] — 2026-10-23
### Added
- **Load testing** for test scripts and suites. `POST /api/test/run` takes an optional
`iterations` count (the PWA's **Iterations** field next to *Start* / *Run suite*): the
run repeats N times, successful iterations are discarded (only the iteration counter and
the timing are kept; the per-step log is not rendered or retained), and the first
iteration that ends in an **error** stops the loop and keeps that iteration's full
report. A manual stop ends it after the current iteration, and a card session change
between iterations is an error (a swapped card is never run against another card's
script). The SCP80 counter state is shared across iterations, so packets continue from
what the previous run consumed. The run status gains a `load` block (`iteration`,
`done`, `failed_iteration`, `elapsed_ms`, `avg_ms`); the count is a run parameter, not
part of the stored script or suite.
### Fixed
- The load-test loop closes properly on a raised runner error (it used to leave the run
`running` forever - the PWA polled a dead run and *Clear report* stayed refused), on a
card session change inside a suite iteration, and on a member that was stopped; a
warning in any iteration is reflected in the final status, `avg_ms` covers the attempted
iterations (the failing one included), and each suite iteration's `counters_before` is
read fresh from the preset store.
## [3.36.0] — 2026-10-23
### Added
+12 -1
View File
@@ -1006,11 +1006,22 @@ on every stop** as long as the card is in the reader; a card reset (session
change) stops the suite, skips the remaining members and is flagged in the
report.
`iterations` (optional, 1-1,000,000, default 1) repeats the script or the
whole suite: successful iterations are discarded (only the counter and the
timing are kept; the per-step log is not retained), the first iteration that
ends in an **error** stops the loop and keeps that iteration's report, a
manual stop ends it after the current iteration, and a card session change
between iterations is an error. The status gains `load`:
`{enabled, iterations, iteration, done, failed_iteration, started, finished,
elapsed_ms, avg_ms}`.
The suite run's snapshot adds `suite`: the member records (`status`, `steps`,
`note`, `log` - the semantic run-log excerpt), the `summary` (ok/warning/
error/skipped counts, `wall_ms`, `stopped`, `session_changed`, the SCP80
`counters_before`/`counters_after` per keyset), the `start_reset` flag and
the `reset` result (ICCID + ms) and the run's `adm` state.
the `reset` result (ICCID + ms) and the run's `adm` state. A successful load
test leaves `suite` `null` (no iteration is tracked); a failing one keeps the
failing iteration's suite state.
The report is exported from the PWA as JSON or Markdown; the run state lives
in the server until `POST /api/test/clear` (no history).
+13
View File
@@ -229,6 +229,19 @@ TEST-RUN step 4: MENU-SELECT ENVELOPE=80c2000009d30702020181900101 item=1 ('One'
(`APDU-TRACE(…): <cmd> → SW: … RESP: …`) for every APDU, including the
file-manager reads and pySim's auto-handler traffic.
## Load testing
A run may be repeated: `POST /api/test/run {..., "iterations": N}` (the
PWA's **Iterations** field next to Start / Run suite) executes the script or
the whole suite N times. Successful iterations are discarded - only the
iteration counter and the timing are kept, so memory use stays flat. The
first iteration that ends in an error stops the loop and keeps its full
report (steps, failed checks, log excerpt; for a suite, the member report);
a manual stop ends it after the current iteration, and a card session change
between iterations is an error. The SCP80 counter state is shared across
iterations, so packets continue from what the previous iteration consumed.
The iteration count is a run parameter - the stored script/suite is unchanged.
## Document shape
```json
+1
View File
@@ -573,6 +573,7 @@
<h3 id="test-script" class="text-lg font-medium mb-2">8.9 Тест-скрипт</h3>
<p class="text-sm mb-2"><strong>Тест-скрипт</strong> &mdash; упорядоченный список <strong>действий</strong> и <strong>ожиданий</strong> проактивных команд, выполняемый на подключённой карте на стороне сервера. Пока запуск активен, карта принадлежит скрипту: остальные операции с картой отклоняются (HTTP&nbsp;409), фоновый опрос STATUS и автоматическая обработка проактивных команд приостанавливаются &mdash; объявленная картой команда остаётся ожидающей для скрипта, и оба механизма возобновляются по завершении запуска. Собственные BIP-команды карты (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) — исключение: терминал отвечает на них по-настоящему: каналы продолжают работать во время запуска, а BIP-события приходят для каналов, открытых самой картой (управление BIP остаётся заблокированным) — если только <code class="font-mono text-sm">respond</code> шага не переопределяет ответ через <code class="font-mono text-sm">raw</code> TLV или результат, отличный от ok; отчёт о запуске хранится в памяти (в том числе после перезагрузки страницы) до нажатия <strong>Очистить отчёт</strong>. Скрипты хранятся в <strong>хранилище на сервере</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), поэтому все браузеры работают с одним набором; изменения записываются в хранилище по мере правки (кнопки &laquo;Сохранить&raquo; нет &mdash; черновик без валидного шага остаётся локальным); экспорт/импорт JSON остаётся для резервных копий и переноса скриптов между стендами (старый экспорт из localStorage импортируется как есть).</p>
<p class="text-sm mb-2">У вкладки <strong>четыре представления</strong>: <strong>список наборов</strong> (по умолчанию; <em>Добавить набор</em> / <em>Редактировать</em> / <em>Запустить</em> / <em>Экспорт</em> / <em>Удалить</em>), <strong>редактор набора</strong> (скрипты набора в порядке выполнения &mdash; необязательный <em>setup</em> первым и <em>teardown</em> последним, с пометками ролей; у каждого скрипта <em>Запустить</em>/<em>Редактировать</em>/<em>Переместить</em>/<em>Скопировать</em>/<em>Удалить</em>, политика <em>при ошибке</em> <em>остановить</em> (по умолчанию) / <em>продолжить</em> и <em>Требуется ADM</em>), <strong>редактор скрипта</strong> (список шагов; кнопка <em>Старт</em> выполняет только этот скрипт и показывает результат под шагами &mdash; он не сохраняется) и <strong>отчёт набора</strong> (после запуска набора: статус каждого скрипта, таблица шагов с ожидаемым/фактическим, примечание об ошибке и выдержка из журнала запуска; сводка с успешно/предупреждения/ошибки/пропущено, время выполнения, счётчики SCP80 до&nbsp;&rarr;&nbsp;после по каждому keyset, пресет и предупреждение о смене сессии карты; экспорт в <em>JSON</em> или <em>Markdown</em>). <strong>Запуск набора</strong> выполняет его скрипты по порядку в <strong>одной сессии карты, без сбросов и без продолжения с места остановки</strong>; <em>при ошибке</em> решает, остановится ли набор на этом скрипте или запустится следующий, <strong>teardown выполняется всегда</strong> при остановке или ошибке (пока карта в ридере), чтобы следующий тест начался с чистого состояния, а сброс карты останавливает набор и отмечается в отчёте. <strong>Начать со сброса карты</strong> сбрасывает карту и выполняет полную переинициализацию перед запуском набора (скрипты затем работают со свежеинициализированной картой); пресет должен по-прежнему совпадать с ICCID карты, а требуемый ADM проверяется заново для новой сессии карты. Инициализация поглощает начальные proactive-команды карты, поэтому скрипт не поймает первую команду после сброса (они остаются в журнале proactive). <strong>Требуется ADM</strong> (у набора или скрипта) проверяет ключ ADM из подобранного пресета <strong>один раз</strong> перед запуском &mdash; ошибка отклоняет запуск (проверьте вручную через значок ADM; автоматических повторов нет). Скрипты хранятся в хранилище наборов (<code class="font-mono text-sm">~/.pysim-simple-server/test_suites.json</code>, <code class="font-mono text-sm">--test-suites</code>); каждый скрипт принадлежит ровно одному набору, и скрипт можно переместить или скопировать в другой набор (экспорт набора &mdash; это пакет с набором и его скриптами; старый экспорт только скриптов импортируется в автоматически созданный набор &laquo;Imported scripts&raquo;).</p>
<p class="text-sm mb-2">Прогон можно повторить: поле <strong>«Итерации»</strong> рядом с <em>Start</em> / <em>Run suite</em> выполняет скрипт или весь набор указанное число раз &mdash; это <strong>нагрузочный тест</strong>. Успешные итерации не сохраняются: в статусе видны только счётчик (итерация k / N) и время, журнал шагов остаётся пустым, а память не растёт при любом числе итераций. Первая итерация, завершившаяся ошибкой, останавливает цикл и сохраняет свой полный отчёт (шаги, неудачные проверки, фрагмент журнала, отчёт по скриптам набора); <em>Остановить</em> завершает работу после текущей итерации, а смена карты между итерациями считается ошибкой (следующая карта никогда не запускается со скриптом другой карты). Число итераций &mdash; параметр прогона, сохранённый скрипт/набор не меняется; счётчики SCP80 продолжаются с того значения, которое израсходовала предыдущая итерация.</p>
<ul class="list-disc list-inside text-sm space-y-1 mb-3">
<li><strong>Действия</strong> &mdash; <em>Envelope</em> (Event Download: сырой тип события + данные, с опциональным переопределением источника &mdash; 82 терминал / 83 сеть) или <em>Event</em> (семантические формы: <strong>та же форма, что и в диалоге отправки на вкладке «Телефон»</strong>, для всех смоделированных событий &mdash; четыре события, которые собирает сервер (Location status, Access technology change, Network rejection, Data connection status change), передаются полями, которые он проверяет и собирает во время запуска, остальные несут собранный формой hex данных события (показывается под формой, с проверкой одного ENVELOPE); источник device identities берётся из события (например, MT call &rarr; сеть) или из его поля источника), <em>Menu Selection</em> (item id 1&ndash;255 или текст пункта с точным/частичным совпадением, по умолчанию без учёта регистра &mdash; выбор <em>«Учёт регистра»</em> в редакторе шагов включает точное совпадение; текст подбирается по меню карты, которое следует за параметрами установки апплета), <em>UPDATE/READ файла</em> (путь + режим; чтение проверяет данные файла), <em>сырой APDU</em> (без авто-обработчика), <em>SCP80</em> (C-APDU или готовый защищённый пакет, опциональные переопределения номера keyset (kvn) / TAR / SPI1 / SPI2 и формат команды &mdash; компактный или обёрнутый в шаблон Command Scripting TS 102 226 &sect;5.2.1 <em>expanded</em> (AA) / <em>expanded AE&nbsp;80</em>; ключи и счётчик берутся из пресета карты, подобранного по ICCID подключённой карты &mdash; запуск называет пресет в хранилище, поэтому всегда используются его keysets, TAR и счётчик &mdash; а сервер сохраняет потреблённый счётчик в пресет по ходу запуска), <em>STATUS</em> (при <em>попытках</em> &gt; 1 опрашивает, пока карта не объявит команду) и <em>Очистка очереди проактивных команд</em> (опрос STATUS и поглощение всех объявленных команд &mdash; результат TERMINAL RESPONSE для каждой, опционально другой результат для первой; граница попыток/интервала; опциональная проверка <em>require</em> по поглощённому. Пустой опрос проходит, финальный STATUS должен быть idle, а оставшаяся команда передаётся следующему шагу &mdash; используйте в начале, середине или конце скрипта, чтобы убедиться, что карта перестала слать команды; она же поглощает команду, оставшуюся от предыдущего шага (ответившего <code class="font-mono text-sm">91XX</code>)). У каждого действия есть проверка ответа: ожидаемые SW и данные (точно или маска с подстановкой <code class="font-mono text-sm">?</code> &mdash; <code class="font-mono text-sm">91??</code>) и проверка PoR для SCP80 (<em>любой / без PoR / por_ok</em> или своя проверка декодированного PoR &mdash; статус / SW / данные; транспорт через SEND SHORT MESSAGE работает так же).</li>
<li><strong>Ожидания</strong> &mdash; карта должна объявить проактивную команду на предыдущем шаге (SW <code class="font-mono text-sm">91XX</code>, TS 102 221 &sect;7.4.2.1 / TS 102 223 &sect;6.3); <code class="font-mono text-sm">9000</code> означает &laquo;команды нет&raquo;: шаг падает на своём уровне ошибок &mdash; Ошибка останавливает скрипт, Предупреждение продолжает (терпимая форма &laquo;поглотить команду, если она есть&raquo;) &mdash; а действие STATUS с <em>попытками</em> &gt; 1 опрашивает, когда апплет выдаёт команду при опросе. Проверяются тип команды, квалификатор и содержимое: текст / пункт / raw, проверка <em>Alpha</em> (идентификатор Alpha команды, тег <code class="font-mono text-sm">05</code>/<code class="font-mono text-sm">85</code> &mdash; отличается от текстовой строки <code class="font-mono text-sm">8D</code>), проверка <em>SMS TPDU</em> (submit-TPDU команды SEND SHORT MESSAGE: получатель / PID / DCS / UDL / данные &mdash; для собственного SMS апплета), проверка <em>PoR</em> (SEND SHORT MESSAGE несёт PoR, когда пакет запросил PoR-in-submit, бит SPI2 <code class="font-mono text-sm">20</code>: статус ответа, SW R-APDU и данные ответа; перед этим должен выполниться шаг SCP80 &mdash; SW R-APDU находится <em>внутри</em> полезной нагрузки ответа: собственный TAR апплета возвращает данные приложения, поэтому проверяйте данные, а SW оставляйте пустым; отклонённый ENVELOPE (<code class="font-mono text-sm">62XX</code>/<code class="font-mono text-sm">63XX</code>) может вовсе не нести пакет ответа &mdash; для такого шага оставляйте проверку PoR «любой» или ожидайте «без PoR») и проверка <em>Files</em> (File List команды REFRESH, например квалификатор <code class="font-mono text-sm">00</code> = NAA initialization with full file change notification). <strong>TERMINAL RESPONSE</strong> является частью инструкции: поле <strong>результата</strong> предлагает полный список TS 102 223 &sect;8.12.0 (каждая строка подписана <code class="font-mono text-sm">&lt;hex&gt; - &lt;алиас или имя&gt;</code>, плюс ручной ввод hex; при выбранной команде список сгруппирован по таблице применимости спецификации (сначала возможные для этой команды значения, затем остальные &mdash; ничего не скрывается); алиасы соответствуют спецификации &mdash; <code class="font-mono text-sm">ok</code> 00, <code class="font-mono text-sm">modified</code> 07, <code class="font-mono text-sm">cancel</code> 10, <code class="font-mono text-sm">timeout</code> 12, <code class="font-mono text-sm">refused</code> 22, <code class="font-mono text-sm">not_understood</code> 32, &hellip;), поле <strong>additional information</strong> (дополнительная информация общего результата, по умолчанию <code class="font-mono text-sm">00</code>; предлагаются значения &sect;8.12.2 для проблем терминала), идентификатор пункта, текст (+DCS) и дополнительные TLV.</li>
+1
View File
@@ -572,6 +572,7 @@
<h3 id="test-script" class="text-lg font-medium mb-2">8.9 Test script</h3>
<p class="text-sm mb-2">A <strong>test script</strong> is an ordered list of <strong>actions</strong> and proactive-command <strong>expectations</strong> executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP&nbsp;409), and background STATUS polling and the automatic proactive handling are suspended &mdash; a command the card announces stays pending for the script, and both resume when the run ends. The card&rsquo;s own BIP commands (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) are an exception: the terminal answers them for real &mdash; its channels keep working during the run and the BIP events flow for the channels the card itself opened (BIP <em>control</em> stays blocked) &mdash; unless the step&rsquo;s <code class="font-mono text-sm">respond</code> overrides the answer with <code class="font-mono text-sm">raw</code> TLVs or a non-ok result; the run report is kept in memory (also after reloading the page) until <strong>Clear report</strong>. Scripts live in the <strong>server-side store</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), so every browser manages the same set; edits are written to the store as you change them (there is no Save button &mdash; a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).</p>
<p class="text-sm mb-2">The pill has <strong>four views</strong>: the <strong>suite list</strong> (default; <em>Add suite</em> / <em>Edit</em> / <em>Run</em> / <em>Export</em> / <em>Delete</em>), the <strong>suite editor</strong> (the suite's scripts in run order &mdash; the optional <em>setup</em> first and the <em>teardown</em> last, marked with role badges; per script <em>Run</em>/<em>Edit</em>/<em>Move</em>/<em>Copy</em>/<em>Delete</em>, the per-member <em>on failure</em> policy <em>stop</em> (default) / <em>continue</em>, and <em>ADM required</em>), the <strong>script editor</strong> (the step list; its <em>Start</em> runs that single script and shows the result under the steps &mdash; not preserved) and the <strong>suite report</strong> (after a suite run: every script's status, its step table with expected/actual, the failing note and a run-log excerpt; the summary with passed/warnings/failed/skipped, wall time, the SCP80 counters before&nbsp;&rarr;&nbsp;after per keyset, the preset and a card-session-changed warning; export as <em>JSON</em> or <em>Markdown</em>). A <strong>suite run</strong> executes its scripts in order in <strong>one card session, without resets and without resume</strong>; <em>on failure</em> decides whether the suite stops at that script or runs the next one, the <strong>teardown always runs</strong> on a stop or a failure (while the card is in the reader) so the next test starts clean, and a card reset stops the suite and is flagged in the report. <strong>Start with card reset</strong> resets the card and runs the full re-initialization before the suite starts (the scripts then run against a freshly initialized card); the preset must still match the card's ICCID afterwards and a required ADM is verified again for the new card session. The initialization consumes the card's initial proactive commands, so a script cannot catch the first command after the reset (they stay in the proactive log). <strong>ADM required</strong> (on the suite or a script) verifies the matched preset's ADM key <strong>once</strong> before the start &mdash; a failure refuses the run (verify manually with the ADM badge; nothing is retried automatically). Scripts live in the suite's store (<code class="font-mono text-sm">~/.pysim-simple-server/test_suites.json</code>, <code class="font-mono text-sm">--test-suites</code>); every script belongs to exactly one suite, and a script can be moved or copied to another suite (a bundle export carries the suite plus its scripts; the old script-only export imports into an auto-created &ldquo;Imported scripts&rdquo; suite).</p>
<p class="text-sm mb-2">A run can be repeated: the <strong>Iterations</strong> field next to <em>Start</em> / <em>Run suite</em> runs the script or the whole suite that many times &mdash; a <strong>load test</strong>. Successful iterations are not tracked: the status shows only the counter (iteration k / N) and the timing, the per-step log stays empty, and memory use stays flat however high the count. The first iteration that ends in an error stops the loop and keeps its full report (steps, failed checks, run-log excerpt, suite member report); a manual <em>Stop</em> ends it after the current iteration, and a card change between iterations is reported as an error (the next card is never run against another card&rsquo;s script). The count is a run option &mdash; the saved script/suite is unchanged &mdash; and the SCP80 counters continue from what the previous iteration consumed.</p>
<ul class="list-disc list-inside text-sm space-y-1 mb-3">
<li><strong>Actions</strong> &mdash; <em>Envelope</em> (Event Download: raw event type + data, with an optional source override &mdash; 82 terminal / 83 network) or <em>Event</em> (the semantic forms: the <strong>same form the Phone tab's send dialog uses</strong>, for every modelled event &mdash; the four events the server builds (Location status, Access technology change, Network rejection, Data connection status change) travel as fields it validates and builds at run time, every other event carries the form's built event-data hex (shown live under the form, with the one-ENVELOPE check); the device-identities source comes from the event (e.g. MT call &rarr; network) or its source field), <em>Menu Selection</em> (item id 1&ndash;255, or the item's text with an exact/contains match, case-insensitive by default &mdash; the step editor's <em>Match case</em> choice switches to an exact-case match; the text is resolved against the card's menu, which follows the applet's install parameters), <em>UPDATE/READ file</em> (path + mode; reads verify the file data), <em>raw APDU</em> (no auto-handler), <em>SCP80</em> (C-APDU or a pre-built secured packet, optional keyset number (kvn) / TAR / SPI1 / SPI2 overrides and a command format &mdash; compact, or the TS 102 226 &sect;5.2.1 Command Scripting template <em>expanded</em> (AA) / <em>expanded AE&nbsp;80</em> around the C-APDU; the keys and the counter come from the card preset matched by the equipped card's ICCID &mdash; the run names the stored preset, so it always uses its keysets, TARs and counter &mdash; the server persists the consumed counter into it as the run proceeds), <em>STATUS</em> (with <em>attempts</em> &gt; 1 it polls until the card announces a command) and <em>Proactive drain</em> (poll STATUS and consume every announced command &mdash; the TERMINAL RESPONSE result for each, optionally a different result for the first; the attempt/interval bound; an optional <em>require</em> assertion over what was drained. An empty poll passes, the final STATUS must be idle, and a still-pending command is handed to the next step &mdash; use it at the start, in the middle or at the end of a script to make sure the card has stopped sending; it also consumes a command a previous step left pending (a step that answered <code class="font-mono text-sm">91XX</code>). Every action has a response check: expected SW and data (exact, or a mask with <code class="font-mono text-sm">?</code> nibble wildcards &mdash; <code class="font-mono text-sm">91??</code>) plus a PoR check for SCP80 (<em>any / no PoR / por_ok</em>, or a custom assertion of the decoded PoR &mdash; status / SW / data; the SEND SHORT MESSAGE transport works the same).</li>
<li><strong>Expectations</strong> &mdash; the card must have announced a proactive command in the previous step (SW <code class="font-mono text-sm">91XX</code>, TS 102 221 &sect;7.4.2.1 / TS 102 223 &sect;6.3); a <code class="font-mono text-sm">9000</code> means &ldquo;no command&rdquo;: the step fails at its fail level &mdash; Error stops the script, Warning continues (the tolerant &ldquo;consume a command if there is one&rdquo; form) &mdash; and a STATUS action with <em>attempts</em> &gt; 1 polls when the applet delivers on poll. The command type, qualifier and contents are verified: text / item / raw checks, an <em>Alpha</em> check (the command's Alpha identifier, tag <code class="font-mono text-sm">05</code>/<code class="font-mono text-sm">85</code> &mdash; distinct from the <code class="font-mono text-sm">8D</code> text string), an <em>SMS TPDU</em> check (the submit TPDU of a SEND SHORT MESSAGE: destination / PID / DCS / UDL / user data &mdash; for an applet's own SMS), a <em>PoR</em> check (a SEND SHORT MESSAGE carries the PoR when the packet requested PoR-in-submit, SPI2 bit <code class="font-mono text-sm">20</code>: the response status, the R-APDU SW and the response data; make sure an SCP80 step ran first &mdash; the R-APDU SW lives <em>inside</em> the response payload: an applet's own TAR returns application data, so check its data and leave the SW empty; a refused ENVELOPE (<code class="font-mono text-sm">62XX</code>/<code class="font-mono text-sm">63XX</code>) may carry no response packet at all &mdash; leave the PoR check at <em>any</em> or expect <em>no PoR</em> for such a step) and a <em>Files</em> check (the File List of a REFRESH, e.g. qualifier <code class="font-mono text-sm">00</code> = NAA initialization with full file change notification). The scripted <strong>TERMINAL RESPONSE</strong> is part of the instruction: the <strong>result</strong> offers the full TS 102 223 &sect;8.12.0 list (each entry labelled <code class="font-mono text-sm">&lt;hex&gt; - &lt;alias or name&gt;</code>, plus a manual hex entry; with a command selected the list is grouped by the spec&rsquo;s applicability table (the command-relevant values first, the rest after &mdash; nothing is hidden); the aliases follow the spec &mdash; <code class="font-mono text-sm">ok</code> 00, <code class="font-mono text-sm">modified</code> 07, <code class="font-mono text-sm">cancel</code> 10, <code class="font-mono text-sm">timeout</code> 12, <code class="font-mono text-sm">refused</code> 22, <code class="font-mono text-sm">not_understood</code> 32, &hellip;), the <strong>additional information</strong> byte of the general result (default <code class="font-mono text-sm">00</code>; the &sect;8.12.2 terminal-problem values are suggested), the item identifier, the text (+DCS) and extra TLVs.</li>
+100 -18
View File
@@ -1412,7 +1412,7 @@
</div>
<label class="flex items-center gap-1 text-sm text-gray-700 dark:text-slate-300"><input id="test-suite-adm" type="checkbox" onchange="testSuiteAdmChanged()"> <span data-l10n="ADM required">ADM required</span></label>
<label class="flex items-center gap-1 text-sm text-gray-700 dark:text-slate-300" data-l10n-title="Reset and fully re-initialize the card (ICCID, network state, TERMINAL PROFILE, drain) before the suite starts. The preset must still match the card's ICCID after the reset; the ADM is verified again for the new card session. The card's initial proactive commands are consumed by the initialization, so a script cannot catch the first command after the reset (they stay in the proactive log). A reset later in the run still stops the suite."><input id="test-suite-reset" type="checkbox" onchange="testSuiteResetChanged()"> <span data-l10n="Start with card reset">Start with card reset</span></label>
<button id="test-suite-run-btn" data-needs="card" onclick="testSuiteRunCurrent()" class="px-4 py-1.5 text-sm rounded bg-emerald-600 text-white hover:bg-emerald-700 disabled:opacity-40 disabled:cursor-not-allowed" data-l10n="Run suite">Run suite</button>
<label class="flex items-center gap-1 text-xs text-gray-600 dark:text-slate-400"><span data-l10n="Iterations">Iterations</span><input id="test-suite-iterations" type="number" min="1" max="1000000" value="1" oninput="testIterationsChanged(this)" class="w-20 border border-gray-300 dark:border-slate-600 text-sm rounded px-2 py-1.5 dark:bg-slate-800"></label> <button id="test-suite-run-btn" data-needs="card" onclick="testSuiteRunCurrent()" class="px-4 py-1.5 text-sm rounded bg-emerald-600 text-white hover:bg-emerald-700 disabled:opacity-40 disabled:cursor-not-allowed" data-l10n="Run suite">Run suite</button>
<button onclick="testSuiteExportCurrent()" class="px-3 py-1.5 text-sm rounded bg-gray-200 dark:bg-slate-700 text-gray-700 dark:text-slate-300 hover:bg-gray-300 dark:hover:bg-slate-600" data-l10n="Export">Export</button>
<button onclick="testSuiteDeleteCurrent()" class="px-3 py-1.5 text-sm rounded bg-gray-200 dark:bg-slate-700 text-red-600 dark:text-red-400 hover:bg-gray-300 dark:hover:bg-slate-600" data-l10n="Delete">Delete</button>
</div>
@@ -1434,7 +1434,7 @@
<input id="test-name" oninput="testNameChanged()" class="w-full border border-gray-300 dark:border-slate-600 text-sm rounded px-3 py-1.5 dark:bg-slate-800">
</div>
<label class="flex items-center gap-1 text-sm text-gray-700 dark:text-slate-300"><input id="test-script-adm" type="checkbox" onchange="testScriptAdmChanged()"> <span data-l10n="ADM required">ADM required</span></label>
<button id="test-run-btn" data-needs="card" onclick="testRunStart()" class="px-4 py-1.5 text-sm rounded bg-emerald-600 text-white hover:bg-emerald-700 disabled:opacity-40 disabled:cursor-not-allowed" data-l10n="Start">Start</button>
<label class="flex items-center gap-1 text-xs text-gray-600 dark:text-slate-400"><span data-l10n="Iterations">Iterations</span><input id="test-iterations" type="number" min="1" max="1000000" value="1" oninput="testIterationsChanged(this)" class="w-20 border border-gray-300 dark:border-slate-600 text-sm rounded px-2 py-1.5 dark:bg-slate-800"></label> <button id="test-run-btn" data-needs="card" onclick="testRunStart()" class="px-4 py-1.5 text-sm rounded bg-emerald-600 text-white hover:bg-emerald-700 disabled:opacity-40 disabled:cursor-not-allowed" data-l10n="Start">Start</button>
<button id="test-stop-btn" onclick="testStop()" class="hidden px-4 py-1.5 text-sm rounded bg-red-600 text-white hover:bg-red-700" data-l10n="Stop">Stop</button>
<button onclick="testClearReport()" class="px-3 py-1.5 text-sm rounded bg-gray-200 dark:bg-slate-700 text-gray-700 dark:text-slate-300 hover:bg-gray-300 dark:hover:bg-slate-600" data-l10n="Clear report">Clear report</button>
</div>
@@ -1959,7 +1959,7 @@
// ===== Version =====
// Single source of truth for the PWA version: shown in the header and used
// by the server version check in pysimConnect().
const SIMPLE_VERSION = '3.36.0';
const SIMPLE_VERSION = '3.37.0';
document.getElementById('app-version').textContent = 'v' + SIMPLE_VERSION;
// ===== view history (the browser Back/Forward buttons) =====
@@ -7531,6 +7531,7 @@ function testViewShow(view) {
}
function testRender() {
if (!_testIterationsReady) { _testIterationsReady = true; testIterationsInit(); }
const views = {suites: 'test-view-suites', suite: 'test-view-suite',
script: 'test-view-script', report: 'test-view-report'};
Object.keys(views).forEach(v => {
@@ -7542,8 +7543,12 @@ function testRender() {
if (banner) {
if (running) {
const snap = _testRunState || {};
const what = snap.suite ? t('Test suite running:') : t('Test script running:');
const at = snap.suite ? testSuiteProgress(snap) : ((snap.index + 1) + '/' + snap.total);
const load = snap.load && snap.load.enabled ? snap.load : null;
const isSuite = !!(snap.suite || (load && load.suite));
const what = isSuite ? t('Test suite running:') : t('Test script running:');
const at = load
? (t('Iteration') + ' ' + (load.iteration || 0) + '/' + load.iterations)
: (snap.suite ? testSuiteProgress(snap) : ((snap.index + 1) + '/' + snap.total));
banner.textContent = '\u26a0 ' + what + ' ' + (snap.name || '') + ' (' + at + ')';
banner.classList.remove('hidden');
} else {
@@ -8920,15 +8925,24 @@ function testRenderScriptRun() {
stepsEl.innerHTML = '';
return;
}
const load = snap.load && snap.load.enabled ? snap.load : null;
let head = esc(snap.name || '');
if (snap.status) head += ' · ' + testStatusBadge(snap.status);
else if (snap.running) head += ' · ' + esc(t('running'));
if (load) {
head += ' · ' + esc(t('Iteration')) + ' ' + (load.iteration || 0)
+ ' / ' + load.iterations;
if (load.done) head += ' · ' + esc(t('done')) + ': ' + load.done;
}
if (snap.preset) head += ' · ' + esc(t('Preset')) + ': ' + esc(snap.preset);
if (snap.scp80_counter) head += ' · counter ' + esc(snap.scp80_counter);
if (snap.error) head += ' · ' + esc(snap.error);
statusEl.innerHTML = head;
statusEl.className = 'text-xs mb-2 text-gray-600 dark:text-slate-300';
stepsEl.innerHTML = (snap.steps || []).map(testRunStepHtml).join('');
// a load test shows the counter while it runs; only the failure report is
// rendered (successful iterations are not tracked)
stepsEl.innerHTML = (load && snap.running)
? '' : (snap.steps || []).map(testRunStepHtml).join('');
}
// ── the suite report ──
@@ -8940,16 +8954,28 @@ function testRenderReport() {
if (!titleEl || !sumEl || !memEl) return;
const snap = _testRunState;
const s = snap && snap.suite;
if (!s) {
// a load test keeps no suite state on success: the run summary is enough
const load = snap && snap.load && snap.load.enabled ? snap.load : null;
if (!s && !load) {
titleEl.textContent = t('No suite run yet.');
sumEl.textContent = '';
memEl.innerHTML = '';
return;
}
titleEl.innerHTML = esc(s.name || t('Suite')) + ' · '
titleEl.innerHTML = esc((s && s.name) || (snap && snap.name) || t('Suite')) + ' · '
+ testStatusBadge(snap.status || (snap.running ? 'running' : ''));
const parts = [];
const su = s.summary;
if (load) {
parts.push(t('Iteration') + ' ' + (load.iteration || 0) + ' / ' + load.iterations);
if (load.done) parts.push(t('done') + ': ' + load.done);
if (load.elapsed_ms !== null && load.elapsed_ms !== undefined) {
parts.push((load.elapsed_ms / 1000).toFixed(1) + ' s');
}
if (load.avg_ms !== null && load.avg_ms !== undefined) {
parts.push(t('avg') + ' ' + load.avg_ms + ' ms');
}
}
const su = s && s.summary;
if (su) {
parts.push(t('passed') + ': ' + su.ok, t('warnings') + ': ' + su.warning,
t('failed') + ': ' + su.error, t('skipped') + ': ' + su.skipped);
@@ -8957,19 +8983,19 @@ function testRenderReport() {
if (su.wall_ms !== undefined && su.wall_ms !== null) parts.push((su.wall_ms / 1000).toFixed(1) + ' s');
}
if (snap.preset) parts.push(t('Preset') + ': ' + snap.preset);
if (s.start_reset) {
if (s && s.start_reset) {
const r = s.reset || {};
let line = t('started with a card reset');
if (r.iccid) line += ' (ICCID ' + r.iccid + ')';
if (r.ms !== undefined && r.ms !== null) line += ' \u00b7 ' + (r.ms / 1000).toFixed(1) + ' s';
parts.push(line);
}
if (s.counters_before && Object.keys(s.counters_before).length) {
if (s && s.counters_before && Object.keys(s.counters_before).length) {
const after = s.counters_after || {};
parts.push('SCP80: ' + Object.keys(s.counters_before).map(k =>
k + ' ' + (s.counters_before[k] || '-') + ' \u2192 ' + (after[k] || '-')).join(', '));
}
if (s.session_changed) parts.push('\u26a0 ' + t('card session changed mid-suite'));
if (s && s.session_changed) parts.push('\u26a0 ' + t('card session changed mid-suite'));
if (snap.adm && snap.adm.required) parts.push('ADM: ' + (snap.adm.verified ? t('verified') : t('not verified')));
if (snap.error) parts.push(esc(snap.error));
sumEl.innerHTML = esc(parts.join(' · '));
@@ -8977,15 +9003,21 @@ function testRenderReport() {
// creates fresh <details>, so an expanded Run log would collapse each
// time. Carry the open state across the re-render (the member list is
// fixed at run start, so the index is a stable key).
const showMembers = !!s && (!load || snap.status === 'error' || snap.status === 'stopped');
const openLogs = {};
if (showMembers) {
memEl.querySelectorAll('details[data-member]').forEach(d => {
openLogs[d.dataset.member] = d.open;
});
memEl.innerHTML = (s.members || []).map(testReportMemberHtml).join('');
}
memEl.innerHTML = showMembers
? (s.members || []).map(testReportMemberHtml).join('') : '';
if (showMembers) {
memEl.querySelectorAll('details[data-member]').forEach(d => {
if (openLogs[d.dataset.member]) d.open = true;
});
}
}
function testReportMemberHtml(m, i) {
let html = '<div class="border border-gray-200 dark:border-slate-700 rounded mb-2">'
@@ -9018,7 +9050,8 @@ function testDownloadText(name, text, mime) {
}
function testReportExportJson() {
if (!_testRunState || !_testRunState.suite) {
if (!_testRunState || (!_testRunState.suite
&& !(_testRunState.load && _testRunState.load.enabled))) {
ioStatus('test-io-status', t('No suite run yet.'));
return;
}
@@ -9026,7 +9059,8 @@ function testReportExportJson() {
}
function testReportExportMarkdown() {
if (!_testRunState || !_testRunState.suite) {
if (!_testRunState || (!_testRunState.suite
&& !(_testRunState.load && _testRunState.load.enabled))) {
ioStatus('test-io-status', t('No suite run yet.'));
return;
}
@@ -9132,6 +9166,7 @@ async function testRunStart() {
try {
const body = {script: {name: cur.name || '', steps: cur.steps,
require_adm: !!cur.require_adm}};
body.iterations = testIterationsValue('test-iterations');
if (cur.id) body.script_id = cur.id;
if (presetId) body.preset_id = presetId;
_testRunScriptRef = cur;
@@ -9165,6 +9200,7 @@ async function testSuiteRun(i) {
}
try {
const body = {suite_id: s.id};
body.iterations = testIterationsValue('test-suite-iterations');
if (presetId) body.preset_id = presetId;
const resp = await pysimFetch('/api/test/run', body);
if (resp.error) { ioStatus('test-io-status', resp.error); return; }
@@ -9179,6 +9215,46 @@ async function testSuiteRun(i) {
function testSuiteRunCurrent() { testSuiteRun(_testSuiteIdx); }
// ===== load testing (v3.37.0) =====
// The iteration count is a run parameter, not part of the stored script or
// suite: > 1 repeats the run, the successful iterations are discarded and the
// first failing iteration's report is kept.
const TEST_ITERATIONS_MAX = 1000000;
let _testIterationsReady = false;
function testIterationsInit() {
let n = 1;
try { n = parseInt(localStorage.getItem('simple_test_iterations'), 10) || 1; }
catch (e) { /* private mode */ }
if (!(n >= 1)) n = 1;
if (n > TEST_ITERATIONS_MAX) n = TEST_ITERATIONS_MAX;
['test-iterations', 'test-suite-iterations'].forEach(id => {
const el = document.getElementById(id);
if (el) el.value = String(n);
});
}
function testIterationsChanged(input) {
let n = parseInt((input && input.value) || '1', 10);
if (!(n >= 1)) n = 1;
if (n > TEST_ITERATIONS_MAX) n = TEST_ITERATIONS_MAX;
if (input) input.value = String(n);
try { localStorage.setItem('simple_test_iterations', String(n)); }
catch (e) { /* ignore */ }
['test-iterations', 'test-suite-iterations'].forEach(id => {
const el = document.getElementById(id);
if (el && el !== input) el.value = String(n);
});
}
function testIterationsValue(id) {
const el = document.getElementById(id);
let n = parseInt((el && el.value) || '1', 10);
if (!(n >= 1)) n = 1;
if (n > TEST_ITERATIONS_MAX) n = TEST_ITERATIONS_MAX;
return n;
}
async function testStop() {
try {
@@ -9269,9 +9345,12 @@ async function testInit() {
testRender();
try {
const st = await pysimFetch('/api/test/status');
if (st && st.suite && st.suite.members) {
// a suite run (or its finished report): open the report view
const idx = testSuites().findIndex(s => s.id === st.suite.id);
if (st && ((st.suite && st.suite.members)
|| (st.load && st.load.enabled))) {
// a suite run (or its finished report, a load run included):
// open the report view
const idx = st.suite
? testSuites().findIndex(s => s.id === st.suite.id) : -1;
if (idx >= 0) _testSuiteIdx = idx;
_testRunState = st;
if (_testView !== 'script') _testView = 'report';
@@ -27212,6 +27291,9 @@ const LANG_RU = {
'Move': 'Переместить',
'Copy': 'Копировать',
'No suite run yet.': 'Набор ещё не запускался.',
'Iterations': 'Итерации',
'Iteration': 'Итерация',
'done': 'выполнено',
'Warning': 'Предупреждение',
'Text': 'Текст',
'id': 'id',
+1 -1
View File
@@ -1,4 +1,4 @@
const CACHE = 'simple-v473';
const CACHE = 'simple-v475';
const URLS = [
'index.html',
'help.html',
+166
View File
@@ -0,0 +1,166 @@
const { test } = require('node:test');
const assert = require('node:assert');
const fs = require('node:fs');
const path = require('node:path');
const html = fs.readFileSync(path.join(__dirname, '..', 'index.html'), 'utf8');
function extractFunc(src, name) {
const re = new RegExp('function\\s+' + name + '\\s*\\([^)]*\\)\\s*\\{');
const m = re.exec(src);
if (!m) throw new Error('function ' + name + ' not found');
let i = m.index + m[0].length - 1;
let depth = 0;
for (; i < src.length; i++) {
if (src[i] === '{') depth++;
else if (src[i] === '}') {
depth--;
if (depth === 0) break;
}
}
return src.slice(m.index, i + 1);
}
const FNS = ['esc', 'testStatusBadge', 'testRenderScriptRun', 'testRenderReport',
'testIterationsInit', 'testIterationsChanged', 'testIterationsValue'];
const code = FNS.map(f => extractFunc(html, f)).join('\n');
function fakeEl(id) {
return { id, innerHTML: '', textContent: '', className: '', value: '',
querySelectorAll: () => [] };
}
function build(els, storage, curRef) {
const fn = new Function('els', 'storage', 'curRef', `
var _testRunState = null;
var _testRunScriptRef = curRef.value;
var TEST_ITERATIONS_MAX = 1000000;
var _testIterationsReady = false;
var localStorage = storage;
var document = { getElementById: id => els[id] || null };
var t = k => k;
function testCurrentScript() { return curRef.value; }
function testRunStepHtml(s) { return '<STEP ' + (s.note || '') + '>'; }
function testReportMemberHtml(m, i) { return '<MEMBER ' + (m.name || '') + '>'; }
${code}
return {
setState: v => { _testRunState = v; },
getState: () => _testRunState,
testRenderScriptRun, testRenderReport,
testIterationsValue, testIterationsChanged, testIterationsInit,
};
`);
return fn(els, storage, curRef);
}
test('the iteration count clamps and persists (v3.37.0)', () => {
const els = { 'test-iterations': fakeEl('test-iterations'),
'test-suite-iterations': fakeEl('test-suite-iterations') };
const storage = { data: {}, getItem(k) { return this.data[k] === undefined ? null : this.data[k]; },
setItem(k, v) { this.data[k] = String(v); } };
const app = build(els, storage, { value: null });
els['test-iterations'].value = '5';
assert.strictEqual(app.testIterationsValue('test-iterations'), 5);
assert.strictEqual(app.testIterationsValue('missing'), 1);
els['test-iterations'].value = '99999999';
assert.strictEqual(app.testIterationsValue('test-iterations'), 1000000);
els['test-iterations'].value = 'nope';
assert.strictEqual(app.testIterationsValue('test-iterations'), 1);
app.testIterationsChanged(els['test-iterations']);
els['test-iterations'].value = '7';
app.testIterationsChanged(els['test-iterations']);
assert.strictEqual(storage.data.simple_test_iterations, '7');
assert.strictEqual(els['test-suite-iterations'].value, '7', 'the other input follows');
// init restores the remembered value
storage.data.simple_test_iterations = '12';
app.testIterationsInit();
assert.strictEqual(els['test-iterations'].value, '12');
assert.strictEqual(els['test-suite-iterations'].value, '12');
});
test('the run bodies carry the iterations (v3.37.0)', () => {
const script = extractFunc(html, 'testRunStart');
assert.match(script, /body\.iterations = testIterationsValue\('test-iterations'\)/);
const suite = extractFunc(html, 'testSuiteRun');
assert.match(suite, /body\.iterations = testIterationsValue\('test-suite-iterations'\)/);
// the inputs live next to the two run buttons
assert.match(html, /id="test-iterations" type="number" min="1" max="1000000"/);
assert.match(html, /id="test-suite-iterations" type="number" min="1" max="1000000"/);
});
test('a running load test shows the counter, not the steps (v3.37.0)', () => {
const els = { 'test-run-status': fakeEl('test-run-status'),
'test-run-steps': fakeEl('test-run-steps') };
const curRef = { value: { id: 's1' } };
const app = build(els, { getItem: () => null, setItem: () => {} }, curRef);
app.setState({ running: true, script_id: 's1', name: 'S', steps: [],
load: { enabled: true, iterations: 100, iteration: 3, done: 2 } });
app.testRenderScriptRun();
assert.match(els['test-run-status'].innerHTML, /Iteration 3 \/ 100/);
assert.match(els['test-run-status'].innerHTML, /done: 2/);
assert.strictEqual(els['test-run-steps'].innerHTML, '', 'steps are not tracked');
});
test('a load-test failure shows the failing iteration steps (v3.37.0)', () => {
const els = { 'test-run-status': fakeEl('test-run-status'),
'test-run-steps': fakeEl('test-run-steps') };
const curRef = { value: { id: 's1' } };
const app = build(els, { getItem: () => null, setItem: () => {} }, curRef);
app.setState({ running: false, status: 'error', script_id: 's1', name: 'S',
steps: [{ note: 'boom' }], error: 'iteration 4 failed',
load: { enabled: true, iterations: 100, iteration: 4, done: 3, failed_iteration: 4 } });
app.testRenderScriptRun();
assert.match(els['test-run-status'].innerHTML, /Iteration 4 \/ 100/);
assert.match(els['test-run-status'].innerHTML, /iteration 4 failed/);
assert.match(els['test-run-steps'].innerHTML, /<STEP boom>/);
});
test('the suite report summarises a load run and hides successful iterations (v3.37.0)', () => {
const els = { 'test-report-title': fakeEl('test-report-title'),
'test-report-summary': fakeEl('test-report-summary'),
'test-report-members': fakeEl('test-report-members') };
const app = build(els, { getItem: () => null, setItem: () => {} }, { value: null });
// a completed load run: no suite state, only the summary
app.setState({ running: false, status: 'ok', name: 'Load suite',
load: { enabled: true, iterations: 50, iteration: 50, done: 50,
elapsed_ms: 12000, avg_ms: 240, failed_iteration: null }, suite: null });
app.testRenderReport();
assert.match(els['test-report-title'].innerHTML, /Load suite/);
assert.match(els['test-report-summary'].innerHTML, /Iteration 50 \/ 50/);
assert.match(els['test-report-summary'].innerHTML, /done: 50/);
assert.strictEqual(els['test-report-members'].innerHTML, '');
// a failure keeps that iteration's members
app.setState({ running: false, status: 'error', name: 'Load suite',
error: 'iteration 7 failed',
load: { enabled: true, iterations: 50, iteration: 7, done: 6,
elapsed_ms: 2000, avg_ms: 300, failed_iteration: 7 },
suite: { name: 'Load suite', summary: { ok: 0, warning: 0, error: 1, skipped: 0, stopped: false },
members: [{ name: 'A', status: 'error' }] } });
app.testRenderReport();
assert.match(els['test-report-summary'].innerHTML, /Iteration 7 \/ 50/);
assert.match(els['test-report-members'].innerHTML, /<MEMBER A>/);
// while running, the members are not shown (results are not tracked)
app.setState({ running: true, status: null, name: 'Load suite',
load: { enabled: true, iterations: 50, iteration: 2, done: 1 },
suite: { name: 'Load suite', summary: null, members: [{ name: 'A', status: 'running' }] } });
app.testRenderReport();
assert.strictEqual(els['test-report-members'].innerHTML, '');
});
test('a successful load run can be exported and reopened after a reload (v3.37.0)', () => {
// the report is renderable with suite: null, so the export buttons and the
// reload path must accept a load run too
for (const fn of ['testReportExportJson', 'testReportExportMarkdown']) {
const src = extractFunc(html, fn);
assert.match(src, /_testRunState\.load && _testRunState\.load\.enabled/);
}
const init = extractFunc(html, 'testInit');
assert.match(init, /st\.load && st\.load\.enabled/);
assert.match(init, /const idx = st\.suite\s*\?/, 'a null suite must not break the reopen');
// the banner knows a load suite even when its state is cleared
const render = extractFunc(html, 'testRender');
assert.match(render, /const isSuite = !!\(snap\.suite \|\| \(load && load\.suite\)\);/);
});
+3
View File
@@ -52,6 +52,9 @@ eval(extractFunc(html, 'testScriptProblem'));
eval(extractFunc(html, 'testPorSpecSummary'));
eval(extractFunc(html, 'testCheckSummary'));
eval(extractFunc(html, 'testStepSummary'));
// the load-test run parameter (v3.37.0): the run bodies read the count
eval(html.match(/const TEST_ITERATIONS_MAX = \d+;/)[0].replace('const ', 'var '));
eval(extractFunc(html, 'testIterationsValue'));
eval(extractFunc(html, 'testCommandOptions'));
globalThis.t = s => s;
globalThis.esc = s => String(s);
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "pysim-simple-server"
version = "3.36.0"
version = "3.37.0"
description = "HTTP REST server wrapping pysim for the SIMple PWA"
license = { text = "GPL-2.0-or-later" }
requires-python = ">=3.9"
+295 -32
View File
@@ -37,7 +37,7 @@ from osmocom.tlv import BER_TLV_IE
from osmocom.utils import rpad
VERSION = '3.36.0'
VERSION = '3.37.0'
MAX_ENVELOPE_SEGMENTS = 5 # max SMS segments for outgoing C-APDU in ENVELOPE
@@ -7059,6 +7059,12 @@ _TEST_RUN = {
# suite runs (v3.22.0): None for a single script; the members list and the
# report summary live here, plus the run-log buffer/adm state
'suite': None, 'log': [], 'log_prefix': '', 'adm': None,
# load testing (v3.37.0): a run parameter, not a stored option. `done`
# counts the successful iterations (their results are discarded); the
# failing iteration's steps/log stay as the report.
'load': {'enabled': False, 'iterations': None, 'iteration': 0, 'done': 0,
'failed_iteration': None, 'started': None, 'finished': None,
'elapsed_ms': None, 'avg_ms': None},
}
_TEST_LOCK = threading.Lock()
_TEST_THREAD = None
@@ -7148,6 +7154,37 @@ def _test_command_type(name):
return _TEST_COMMAND_TYPES.get((name or '').upper())
_TEST_LOAD_MAX = 1000000
def _test_load_state(iterations, suite=False):
"""The `load` block of a run: disabled for a normal single run. `suite`
lets the PWA label a suite load test even after its state is cleared."""
enabled = iterations > 1
return {'enabled': enabled, 'suite': bool(suite),
'iterations': iterations if enabled else None,
'iteration': 0, 'done': 0, 'failed_iteration': None,
'started': time.time() if enabled else None, 'finished': None,
'elapsed_ms': None, 'avg_ms': None}
def _test_load_iterations(body):
"""The run's iteration count (1 when omitted). Raises ValueError for a
bad value (the endpoint answers 400)."""
raw = (body or {}).get('iterations')
if raw is None or raw == '':
return 1
try:
n = int(raw)
except (TypeError, ValueError):
raise ValueError('iterations must be an integer')
if isinstance(raw, float) and raw != n:
raise ValueError('iterations must be an integer')
if n < 1 or n > _TEST_LOAD_MAX:
raise ValueError('iterations must be between 1 and %d' % _TEST_LOAD_MAX)
return n
def _test_state_snapshot():
with _TEST_LOCK:
return json.loads(json.dumps(_TEST_RUN))
@@ -8586,8 +8623,9 @@ def _test_run_spawn(server, target, args, thread_name):
raise
def _test_run_start(server, script, preset, script_id=None, adm_info=None):
"""Start a single-script run."""
def _test_run_start(server, script, preset, script_id=None, adm_info=None,
iterations=1):
"""Start a single-script run; `iterations` > 1 makes it a load test."""
with _TEST_LOCK:
_TEST_RUN.update({
'running': True, 'stop': False, 'name': script['name'], 'status': None,
@@ -8598,19 +8636,27 @@ def _test_run_start(server, script, preset, script_id=None, adm_info=None):
'preset': (preset or {}).get('name') or (preset or {}).get('iccid'),
'suite': None, 'log': [], 'log_prefix': '',
'adm': adm_info or {'required': False},
'load': _test_load_state(iterations),
})
_test_run_spawn(server, _test_run_worker, (server, script, preset), 'test-script')
if iterations > 1:
sys.stderr.write('TEST-LOAD: script %r, %d iterations\n'
% (script.get('name') or script_id or 'script',
iterations))
_test_run_spawn(server, _test_load_script_worker,
(server, script, preset, script_id, iterations), 'test-load')
else:
_test_run_spawn(server, _test_run_worker, (server, script, preset),
'test-script')
def _test_suite_execute(server, suite, members, preset, preset_id):
"""The suite worker body: setup -> members in order (per-member on_fail)
-> teardown, one card session, no resume. The teardown runs on every
stop - its purpose is to leave the card ready for a new test - as long as
the card is in the reader; a card reset stops the suite and the remaining
members are skipped (the report flags it)."""
global _TEST_RUNNING
def _test_suite_execute_once(server, suite, members, preset, preset_id, state,
shared=None):
"""One suite iteration: setup -> members in order (per-member on_fail) ->
teardown, one card session, no resume. Fills `state` and the live
`_TEST_RUN` steps/log; returns ``(status, stopped, session_changed)``.
The caller owns the run lifecycle - a load test repeats this per
iteration."""
session_start = getattr(server, 'card_session', 0)
state = _TEST_RUN['suite']
stopped = False
session_changed = False
setup_i = next((i for i, (e, _s) in enumerate(members)
@@ -8622,7 +8668,7 @@ def _test_suite_execute(server, suite, members, preset, preset_id):
# suite, so a second SCP80 script starts from what the first consumed
# (the store persists each packet's counter; the preset dict here is the
# suite-start snapshot).
shared = {}
shared = shared if shared is not None else {}
def run_member(i):
entry, script = members[i]
@@ -8659,8 +8705,11 @@ def _test_suite_execute(server, suite, members, preset, preset_id):
abort = False
if setup_i is not None:
status, _s = run_member(setup_i)
if status not in ('ok', 'warning'):
status, setup_stopped = run_member(setup_i)
if setup_stopped:
stopped = True
abort = True
elif status not in ('ok', 'warning'):
abort = True
skip_pending('the setup script failed')
for i in member_idx:
@@ -8676,13 +8725,17 @@ def _test_suite_execute(server, suite, members, preset, preset_id):
session_changed = True
skip_pending('card session changed (card removed or re-equipped)')
break
status, _s = run_member(i)
status, member_stopped = run_member(i)
if member_stopped:
stopped = True
abort = True
break
if status not in ('ok', 'warning') and members[i][0]['on_fail'] == 'stop':
abort = True
if abort:
skip_pending('a previous script failed (on_fail: stop)')
else:
if stopped:
skip_pending('stopped')
elif abort:
skip_pending('a previous script failed (on_fail: stop)')
if teardown_i is not None:
if not getattr(server, 'card_present', False):
with _TEST_LOCK:
@@ -8715,29 +8768,40 @@ def _test_suite_execute(server, suite, members, preset, preset_id):
'warning': statuses.count('warning'),
'error': statuses.count('error'),
'skipped': statuses.count('skipped'),
'stopped': bool(stopped),
'stopped': bool(stopped or statuses.count('stopped')),
'session_changed': state['session_changed'],
'wall_ms': int((time.time() - (_TEST_RUN['started'] or time.time())) * 1000),
'wall_ms': int((time.time() - (state.get('started') or _TEST_RUN['started'] or time.time())) * 1000),
'counters_before': state.get('counters_before'),
'counters_after': after,
'setup': state['members'][setup_i]['status'] if setup_i is not None else None,
'teardown': state['members'][teardown_i]['status'] if teardown_i is not None else None,
}
if 'error' in statuses:
_TEST_RUN['status'] = 'error'
status = 'error'
elif 'warning' in statuses:
_TEST_RUN['status'] = 'warning'
elif stopped:
_TEST_RUN['status'] = 'stopped'
status = 'warning'
elif stopped or 'stopped' in statuses:
status = 'stopped'
else:
_TEST_RUN['status'] = 'ok'
status = 'ok'
return status, stopped, session_changed
def _test_suite_execute(server, suite, members, preset, preset_id, shared=None):
"""The single suite run: one iteration, then the run state is finalised.
(A load test calls `_test_suite_execute_once` itself.)"""
global _TEST_RUNNING
state = _TEST_RUN['suite']
status, _stopped, _session_changed = _test_suite_execute_once(
server, suite, members, preset, preset_id, state, shared)
with _TEST_LOCK:
_TEST_RUN['status'] = status
_TEST_RUN['running'] = False
_TEST_RUN['finished'] = time.time()
_TEST_RUN['log_prefix'] = '' # the report is done: no stale prefix
_TEST_RUNNING = False
_sim_resume_poll()
def _test_suite_worker(server, suite, members, preset, preset_id):
global _TEST_RUNNING
try:
@@ -8788,6 +8852,7 @@ def _test_suite_state(suite, members, preset, preset_id, session, reset_info=Non
'require_adm': bool(suite.get('require_adm')),
'start_reset': bool(suite.get('start_reset')),
'reset': reset_info or None,
'started': time.time(),
'session_start': session, 'session_end': None,
'session_changed': False, 'preset_id': preset_id,
'preset': (preset or {}).get('name') or (preset or {}).get('iccid'),
@@ -8804,7 +8869,7 @@ def _test_suite_state(suite, members, preset, preset_id, session, reset_info=Non
def _test_suite_start(server, suite, members, preset, preset_id, adm_info=None,
reset_info=None):
reset_info=None, iterations=1):
"""Start a suite run: `members` is the resolved, normalised
``[(entry, script)]`` list in run order. `reset_info` records the
start-of-suite card reset the caller already performed."""
@@ -8821,11 +8886,200 @@ def _test_suite_start(server, suite, members, preset, preset_id, adm_info=None,
'suite': _test_suite_state(suite, members, preset, preset_id, session,
reset_info=reset_info),
'log': [], 'log_prefix': '', 'adm': adm_info or {'required': False},
'load': _test_load_state(iterations, suite=True),
})
if iterations > 1:
sys.stderr.write('TEST-LOAD: suite %r, %d iterations\n'
% (suite.get('name') or 'suite', iterations))
_test_run_spawn(server, _test_load_suite_worker,
(server, suite, members, preset, preset_id, iterations),
'test-load')
else:
_test_run_spawn(server, _test_suite_worker,
(server, suite, members, preset, preset_id), 'test-suite')
def _test_load_finalise(status, done, iterations, failed, attempted=None):
"""Close a load run: the counters, the timing and the final status. Only
the failing iteration's steps/log stay (successful ones were discarded).
`attempted` counts the started iterations (the failing/partial one
included), so `avg_ms` is a real per-iteration time."""
with _TEST_LOCK:
now = time.time()
load = _TEST_RUN['load']
started = load.get('started') or _TEST_RUN.get('started') or now
elapsed = int((now - started) * 1000)
attempted = attempted if attempted is not None else done
load.update({'done': done, 'failed_iteration': failed, 'finished': now,
'elapsed_ms': elapsed,
'avg_ms': int(elapsed / attempted) if attempted else None})
_TEST_RUN['running'] = False
_TEST_RUN['finished'] = now
_TEST_RUN['status'] = status
_TEST_RUN['log_prefix'] = ''
if failed:
how = 'failed at iteration %d/%d' % (failed, iterations)
elif status == 'stopped':
how = 'stopped after %d/%d' % (done, iterations)
else:
how = 'done %d/%d' % (done, iterations)
sys.stderr.write('TEST-LOAD: %s (%s)\n'
% (_TEST_RUN.get('name') or 'run', how))
def _test_load_script_worker(server, script, preset, script_id, iterations):
"""Repeat one script `iterations` times: successful runs are discarded,
an error (or a stop) keeps its steps/log as the report (v3.37.0)."""
global _TEST_RUNNING
shared = {} # the SCP80 counter continues across runs
session = getattr(server, 'card_session', 0)
done = 0
attempted = 0
failed = None
final = 'ok'
worst = 'ok'
try:
for i in range(1, iterations + 1):
with _TEST_LOCK:
if _TEST_RUN['stop']:
final = 'stopped'
break
_TEST_RUN.update({'steps': [], 'index': 0, 'status': None,
'error': None, 'log': [], 'log_prefix': ''})
_TEST_RUN['load'].update({'iteration': i, 'done': done})
attempted = i
if getattr(server, 'card_session', 0) != session:
with _TEST_LOCK:
_TEST_RUN['error'] = ('the card session changed during the '
'load test (card removed or re-equipped)')
failed = i
final = 'error'
break
_ctx, stopped = _test_run_script(server, script, preset, shared=shared)
with _TEST_LOCK:
levels = [st.get('status') for st in _TEST_RUN['steps']]
if stopped:
status = 'stopped'
elif 'error' in levels:
status = 'error'
elif 'warning' in levels:
status = 'warning'
else:
status = 'ok'
if status in ('error', 'stopped'):
if status == 'error':
failed = i
if not _TEST_RUN.get('error'):
_TEST_RUN['error'] = 'iteration %d failed' % i
final = status
else:
if status == 'warning':
worst = 'warning'
done = i
_TEST_RUN['steps'] = []
_TEST_RUN['log'] = []
_TEST_RUN['load']['done'] = done
if final != 'ok':
break
if final == 'ok':
final = worst if done == iterations else 'stopped'
_test_load_finalise(final, done, iterations, failed, attempted)
except Exception as e:
# a raised runner error must not leave the run "running" forever (the
# PWA would poll a dead run and Clear would stay refused)
traceback.print_exc()
with _TEST_LOCK:
_TEST_RUN['error'] = str(e)
if failed is None:
failed = _TEST_RUN['load'].get('iteration') or (done + 1)
_test_load_finalise('error', done, iterations, failed, attempted)
finally:
_TEST_RUNNING = False
_sim_resume_poll()
def _test_load_suite_worker(server, suite, members, preset, preset_id, iterations):
"""Repeat a whole suite `iterations` times (setup/members/teardown per
iteration, one shared counter state); the same success/failure policy as
the script load test (v3.37.0)."""
global _TEST_RUNNING
shared = {}
session = getattr(server, 'card_session', 0)
done = 0
attempted = 0
failed = None
final = 'ok'
worst = 'ok'
try:
for i in range(1, iterations + 1):
with _TEST_LOCK:
if _TEST_RUN['stop']:
final = 'stopped'
break
# the report's counters_before is this iteration's start; the
# protocol state (keys/TARs) stays the run-start snapshot
fresh = preset
store = getattr(server, 'card_presets', None)
if preset_id and store is not None:
fresh = store.get(preset_id) or preset
state = _test_suite_state(suite, members, fresh, preset_id, session)
_TEST_RUN.update({'suite': state, 'steps': [], 'index': 0,
'total': 0, 'status': None, 'error': None,
'log': [], 'log_prefix': ''})
_TEST_RUN['load'].update({'iteration': i, 'done': done})
attempted = i
if getattr(server, 'card_session', 0) != session:
with _TEST_LOCK:
_TEST_RUN['error'] = ('the card session changed during the '
'load test (card removed or re-equipped)')
failed = i
final = 'error'
break
status, stopped, session_changed = _test_suite_execute_once(
server, suite, members, preset, preset_id, state, shared)
if session_changed:
with _TEST_LOCK:
_TEST_RUN['error'] = ('the card session changed during the '
'load test (card removed or re-equipped)')
failed = i
final = 'error'
break
if stopped or status == 'stopped':
final = 'stopped'
break
if status == 'error':
failed = i
final = 'error'
with _TEST_LOCK:
if not _TEST_RUN.get('error'):
_TEST_RUN['error'] = 'iteration %d failed' % i
break
if status == 'warning':
worst = 'warning'
with _TEST_LOCK:
done = i
_TEST_RUN['load']['done'] = done
_TEST_RUN['steps'] = []
_TEST_RUN['log'] = []
# successful iterations are not tracked (a failure keeps its
# own state for the report)
_TEST_RUN['suite'] = None
if final == 'ok':
final = worst if done == iterations else 'stopped'
_test_load_finalise(final, done, iterations, failed, attempted)
except Exception as e:
traceback.print_exc()
with _TEST_LOCK:
_TEST_RUN['error'] = str(e)
if failed is None:
failed = _TEST_RUN['load'].get('iteration') or (done + 1)
_test_load_finalise('error', done, iterations, failed, attempted)
finally:
_TEST_RUNNING = False
_sim_resume_poll()
class PysimHandler(BaseHTTPRequestHandler):
def _send_json(self, data, status=200):
self.send_response(status)
@@ -8895,7 +9149,7 @@ class PysimHandler(BaseHTTPRequestHandler):
self._send_json(resp, 500)
self._log_resp(resp)
def _test_suite_run_request(self, suite_id, body):
def _test_suite_run_request(self, suite_id, body, iterations=1):
"""Start a suite run: resolve the members and the preset, run the ADM
prerequisite, then hand over to the suite runner (one card session,
no resume)."""
@@ -9006,7 +9260,8 @@ class PysimHandler(BaseHTTPRequestHandler):
try:
_test_suite_start(self.server, suite, members, preset,
str(body.get('preset_id') or '') or None,
adm_info=adm_info, reset_info=reset_info)
adm_info=adm_info, reset_info=reset_info,
iterations=iterations)
except Exception as e:
resp = {'error': 'could not start the suite run: %s' % e}
self._send_json(resp, 500)
@@ -11286,9 +11541,16 @@ class PysimHandler(BaseHTTPRequestHandler):
self._send_json(resp, 409)
self._log_resp(resp)
return
try:
iterations = _test_load_iterations(body)
except ValueError as e:
resp = {'error': str(e)}
self._send_json(resp, 400)
self._log_resp(resp)
return
suite_id = str(body.get('suite_id') or '').strip().lower()
if suite_id:
self._test_suite_run_request(suite_id, body)
self._test_suite_run_request(suite_id, body, iterations)
return
script_raw, script_id, script_err = _test_run_script_from_body(self.server, body)
if script_err:
@@ -11329,7 +11591,7 @@ class PysimHandler(BaseHTTPRequestHandler):
return
try:
_test_run_start(self.server, script, preset, script_id or None,
adm_info=adm_info)
adm_info=adm_info, iterations=iterations)
except Exception as e:
resp = {'error': 'could not start the test run: %s' % e}
self._send_json(resp, 500)
@@ -11358,6 +11620,7 @@ class PysimHandler(BaseHTTPRequestHandler):
'scp80_counters': {},
'started': None, 'finished': None, 'error': None,
'suite': None, 'log': [], 'log_prefix': '', 'adm': None,
'load': _test_load_state(1),
})
if busy:
resp = {'error': 'test script is running'}
+3 -3
View File
@@ -264,12 +264,12 @@ class TestProactiveChainOwnership(unittest.TestCase):
take.assert_called_once()
def test_every_run_end_resumes_the_poll(self):
# the two worker bodies, their two exception paths and the spawn
# failure all call the helper
# the two worker bodies, their two exception paths, the spawn failure
# and the two load workers (v3.37.0) all call the helper
import inspect
src = inspect.getsource(S)
self.assertEqual(src.count('\n _sim_resume_poll()'), 2)
self.assertEqual(src.count('\n _sim_resume_poll()'), 3)
self.assertEqual(src.count('\n _sim_resume_poll()'), 5)
if __name__ == '__main__':
+308
View File
@@ -0,0 +1,308 @@
"""The load-testing run mode (v3.37.0): a script or suite repeated N times.
Successful iterations are discarded (only the count grows); the first error
keeps that iteration's steps/log as the report; a manual stop or a card session
change ends the loop. The workers are driven here with the runner mocked, so
the loop itself is what the tests pin.
"""
import copy
import time
import unittest
from unittest import mock
from pysim_simple_server import server as srv
class _FakeServer:
def __init__(self, session=7):
self.card_session = session
def _reset_run():
srv._TEST_RUN.update({
'running': True, 'stop': False, 'name': 'load', 'status': None,
'session': 7, 'index': 0, 'total': 0, 'steps': [], 'preset': 'p',
'script_id': 's1', 'scp80_counter': None, 'scp80_counters': {},
'started': time.time(), 'finished': None, 'error': None,
'suite': None, 'log': [], 'log_prefix': '', 'adm': {'required': False},
'load': srv._test_load_state(1),
})
class LoadIterationsTests(unittest.TestCase):
def test_default_is_one(self):
self.assertEqual(srv._test_load_iterations({}), 1)
self.assertEqual(srv._test_load_iterations({'iterations': None}), 1)
self.assertEqual(srv._test_load_iterations({'iterations': ''}), 1)
def test_the_count_is_validated(self):
self.assertEqual(srv._test_load_iterations({'iterations': '5'}), 5)
self.assertEqual(srv._test_load_iterations({'iterations': 7}), 7)
for bad in (0, -1, 1.5, 'abc', srv._TEST_LOAD_MAX + 1, [2]):
with self.assertRaises(ValueError):
srv._test_load_iterations({'iterations': bad})
def test_the_state_flags_only_repeated_runs(self):
one = srv._test_load_state(1)
self.assertFalse(one['enabled'])
self.assertIsNone(one['iterations'])
many = srv._test_load_state(100)
self.assertTrue(many['enabled'])
self.assertEqual(many['iterations'], 100)
self.assertEqual(many['done'], 0)
class ScriptLoadTests(unittest.TestCase):
def setUp(self):
_reset_run()
self._saved = copy.deepcopy(srv._TEST_RUN)
srv._TEST_RUNNING = True
def tearDown(self):
srv._TEST_RUN.clear()
srv._TEST_RUN.update(self._saved)
srv._TEST_RUNNING = False
def _run(self, statuses, iterations, stop_after=None):
"""Drive the load worker with `statuses` per iteration; returns the
shared dicts passed to the runner and the run counter."""
calls = []
state = {'i': 0}
def fake_run(server, script, preset, shared=None):
i = state['i']
state['i'] += 1
calls.append(shared)
status = statuses[i] if i < len(statuses) else 'ok'
srv._TEST_RUN['steps'].append({'index': 0, 'status': status,
'label': 'step %d' % (i + 1)})
if status == 'error':
srv._TEST_RUN['steps'][-1]['note'] = 'boom'
if stop_after is not None and i + 1 >= stop_after:
srv._TEST_RUN['stop'] = True
return {}, status == 'stopped'
with mock.patch.object(srv, '_test_run_script', fake_run), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(),
{'name': 'S', 'steps': []}, None,
's1', iterations)
return calls
def test_successful_iterations_are_discarded(self):
calls = self._run(['ok'] * 5, 5)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 5)
self.assertEqual(load['iteration'], 5)
self.assertIsNone(load['failed_iteration'])
self.assertEqual(srv._TEST_RUN['status'], 'ok')
self.assertEqual(srv._TEST_RUN['steps'], []) # no tracked results
self.assertEqual(srv._TEST_RUN['log'], [])
self.assertFalse(srv._TEST_RUN['running'])
self.assertIsNotNone(load['elapsed_ms'])
self.assertIsNotNone(load['avg_ms'])
# the counter state is shared by every iteration
self.assertEqual(len(calls), 5)
for shared in calls[1:]:
self.assertIs(shared, calls[0])
def test_an_error_stops_the_loop_and_keeps_that_iteration(self):
self._run(['ok', 'ok', 'error', 'error', 'error'], 5)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 2)
self.assertEqual(load['iteration'], 3)
self.assertEqual(load['failed_iteration'], 3)
self.assertEqual(srv._TEST_RUN['status'], 'error')
self.assertEqual(srv._TEST_RUN['error'], 'iteration 3 failed')
self.assertEqual(len(srv._TEST_RUN['steps']), 1)
self.assertEqual(srv._TEST_RUN['steps'][0]['note'], 'boom')
self.assertFalse(srv._TEST_RUN['running'])
def test_a_warning_does_not_stop_the_loop(self):
self._run(['warning', 'ok', 'warning'], 3)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 3)
self.assertIsNone(load['failed_iteration'])
self.assertEqual(srv._TEST_RUN['status'], 'warning')
def test_a_manual_stop_ends_the_loop(self):
# the fake sets the stop flag after the second iteration
self._run(['ok', 'ok', 'ok'], 50, stop_after=2)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 2)
self.assertEqual(srv._TEST_RUN['status'], 'stopped')
self.assertFalse(srv._TEST_RUN['running'])
def test_a_card_session_change_between_iterations_is_an_error(self):
calls = []
state = {'i': 0}
def fake_run(server, script, preset, shared=None):
state['i'] += 1
calls.append(1)
srv._TEST_RUN['steps'].append({'index': 0, 'status': 'ok'})
server.card_session = 99 # the card was swapped
return {}, False
with mock.patch.object(srv, '_test_run_script', fake_run), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(), {'name': 'S', 'steps': []},
None, 's1', 10)
load = srv._TEST_RUN['load']
self.assertEqual(len(calls), 1)
self.assertEqual(load['failed_iteration'], 2)
self.assertEqual(srv._TEST_RUN['status'], 'error')
self.assertIn('card session changed', srv._TEST_RUN['error'])
class LoadRegressionTests(unittest.TestCase):
"""The review findings: a raised error, warning aggregation, the average
and the per-iteration clearing."""
def setUp(self):
_reset_run()
self._saved = copy.deepcopy(srv._TEST_RUN)
srv._TEST_RUNNING = True
def tearDown(self):
srv._TEST_RUN.clear()
srv._TEST_RUN.update(self._saved)
srv._TEST_RUNNING = False
def test_a_raised_runner_error_finalises_the_run(self):
# the PWA would poll a dead run forever and Clear would stay refused
def boom(*a, **k):
raise OSError('card vanished')
with mock.patch.object(srv, '_test_run_script', boom), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(), {'name': 'S', 'steps': []},
None, 's1', 5)
self.assertFalse(srv._TEST_RUN['running'])
self.assertEqual(srv._TEST_RUN['status'], 'error')
self.assertIn('card vanished', srv._TEST_RUN['error'])
self.assertEqual(srv._TEST_RUN['load']['failed_iteration'], 1)
self.assertIsNotNone(srv._TEST_RUN['load']['finished'])
def test_a_warning_stays_in_the_final_status(self):
statuses = iter(['warning', 'ok', 'ok'])
with mock.patch.object(srv, '_test_run_script',
lambda *a, **k: (srv._TEST_RUN['steps'].append(
{'status': next(statuses)}), ({}, False))[1]), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(), {'name': 'S', 'steps': []},
None, 's1', 3)
self.assertEqual(srv._TEST_RUN['status'], 'warning')
self.assertEqual(srv._TEST_RUN['load']['done'], 3)
def test_the_average_covers_the_failing_iteration(self):
statuses = iter(['ok', 'ok', 'error'])
def fake(*a, **k):
st = next(statuses)
srv._TEST_RUN['steps'].append({'status': st})
return {}, False
with mock.patch.object(srv, '_test_run_script', fake), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(), {'name': 'S', 'steps': []},
None, 's1', 10)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 2)
self.assertIsNotNone(load['avg_ms'])
self.assertLessEqual(load['avg_ms'], load['elapsed_ms'])
def test_the_steps_are_cleared_before_each_iteration(self):
seen = []
def fake(*a, **k):
seen.append(len(srv._TEST_RUN['steps']))
srv._TEST_RUN['steps'].append({'status': 'ok'})
return {}, False
with mock.patch.object(srv, '_test_run_script', fake), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_script_worker(_FakeServer(), {'name': 'S', 'steps': []},
None, 's1', 4)
self.assertEqual(seen, [0, 0, 0, 0], 'every iteration starts empty')
class SuiteLoadTests(unittest.TestCase):
def setUp(self):
_reset_run()
self._saved = copy.deepcopy(srv._TEST_RUN)
srv._TEST_RUNNING = True
self.suite = {'id': 'suite1', 'name': 'S', 'require_adm': False,
'start_reset': False, 'scripts': []}
self.members = [({'script_id': 'a', 'role': 'member', 'on_fail': 'stop'},
{'name': 'A', 'steps': []})]
def tearDown(self):
srv._TEST_RUN.clear()
srv._TEST_RUN.update(self._saved)
srv._TEST_RUNNING = False
def _run(self, statuses, iterations):
calls = []
counter = {'i': 0}
def fake_once(server, suite, members, preset, preset_id, state, shared=None):
i = counter['i']
counter['i'] += 1
calls.append(shared)
state['members'][0]['status'] = 'ok'
state['summary'] = {'ok': 1, 'warning': 0, 'error': 0, 'skipped': 0,
'stopped': False, 'wall_ms': 1}
return statuses[i] if i < len(statuses) else 'ok', False, False
with mock.patch.object(srv, '_test_suite_execute_once', fake_once), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_suite_worker(_FakeServer(), self.suite, self.members,
None, None, iterations)
return calls
def test_successful_iterations_are_discarded(self):
calls = self._run(['ok'] * 4, 4)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 4)
self.assertIsNone(srv._TEST_RUN['suite']) # no tracked reports
self.assertEqual(srv._TEST_RUN['steps'], [])
self.assertEqual(srv._TEST_RUN['status'], 'ok')
self.assertEqual(len(calls), 4)
self.assertIs(calls[0], calls[-1]) # one counter state
def test_an_error_keeps_the_failing_iteration_report(self):
self._run(['ok', 'error', 'error'], 3)
load = srv._TEST_RUN['load']
self.assertEqual(load['done'], 1)
self.assertEqual(load['failed_iteration'], 2)
self.assertEqual(srv._TEST_RUN['status'], 'error')
self.assertIsNotNone(srv._TEST_RUN['suite'])
self.assertEqual(srv._TEST_RUN['suite']['name'], 'S')
def test_a_session_change_between_suite_iterations_is_an_error(self):
def fake_once(server, suite, members, preset, preset_id, state, shared=None):
return 'ok', False, True
with mock.patch.object(srv, '_test_suite_execute_once', fake_once), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_suite_worker(_FakeServer(), self.suite, self.members,
None, None, 5)
self.assertEqual(srv._TEST_RUN['load']['failed_iteration'], 1)
self.assertEqual(srv._TEST_RUN['status'], 'error')
self.assertIn('card session changed', srv._TEST_RUN['error'])
self.assertIsNotNone(srv._TEST_RUN['suite'])
def test_a_stopped_member_ends_the_suite_loop(self):
def fake_once(server, suite, members, preset, preset_id, state, shared=None):
return 'ok', True, False
with mock.patch.object(srv, '_test_suite_execute_once', fake_once), \
mock.patch.object(srv, '_sim_resume_poll'):
srv._test_load_suite_worker(_FakeServer(), self.suite, self.members,
None, None, 5)
self.assertEqual(srv._TEST_RUN['status'], 'stopped')
self.assertEqual(srv._TEST_RUN['load']['done'], 0)
def test_a_stopped_iteration_ends_the_loop(self):
self._run(['stopped'], 5)
self.assertEqual(srv._TEST_RUN['status'], 'stopped')
self.assertEqual(srv._TEST_RUN['load']['done'], 0)
if __name__ == '__main__':
unittest.main()