v3.37.0: load testing for test scripts and suites
POST /api/test/run takes an optional iterations count (the PWA's Iterations field): the run repeats, successful iterations are discarded (steps/log cleared, only load.done and the timing grow), the first error stops the loop and keeps that iteration's report, a stop ends it after the current iteration, and a card session change between iterations is an error. The SCP80 counter state is shared across iterations, so packets continue from what the previous run consumed. The status gains a load block (iteration/done/failed_iteration/ elapsed_ms/avg_ms; disabled for a normal run). The suite runner was split into a per-iteration body (_test_suite_execute_once) + the single-run finaliser, and two load workers own the loops. Review fixes: a raised runner error finalises as error (no stuck running state), a suite-internal session change is an error, a stopped member ends the suite loop, the final status keeps a warning from any iteration, avg_ms covers the attempted iterations, and each suite iteration's counters_before is read fresh. 1041 Python + 1069 frontend; simple-v475.
This commit is contained in:
catarrh committed
1 parent
851ff65af6
commit
2181ef6bbf
13 files changed
+937
-64
No files matched your search
@@ -572,6 +572,7 @@
|
||||
<h3 id="test-script" class="text-lg font-medium mb-2">8.9 Test script</h3>
|
||||
<p class="text-sm mb-2">A <strong>test script</strong> is an ordered list of <strong>actions</strong> and proactive-command <strong>expectations</strong> executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP 409), and background STATUS polling and the automatic proactive handling are suspended — a command the card announces stays pending for the script, and both resume when the run ends. The card’s own BIP commands (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) are an exception: the terminal answers them for real — its channels keep working during the run and the BIP events flow for the channels the card itself opened (BIP <em>control</em> stays blocked) — unless the step’s <code class="font-mono text-sm">respond</code> overrides the answer with <code class="font-mono text-sm">raw</code> TLVs or a non-ok result; the run report is kept in memory (also after reloading the page) until <strong>Clear report</strong>. Scripts live in the <strong>server-side store</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), so every browser manages the same set; edits are written to the store as you change them (there is no Save button — a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).</p>
|
||||
<p class="text-sm mb-2">The pill has <strong>four views</strong>: the <strong>suite list</strong> (default; <em>Add suite</em> / <em>Edit</em> / <em>Run</em> / <em>Export</em> / <em>Delete</em>), the <strong>suite editor</strong> (the suite's scripts in run order — the optional <em>setup</em> first and the <em>teardown</em> last, marked with role badges; per script <em>Run</em>/<em>Edit</em>/<em>Move</em>/<em>Copy</em>/<em>Delete</em>, the per-member <em>on failure</em> policy <em>stop</em> (default) / <em>continue</em>, and <em>ADM required</em>), the <strong>script editor</strong> (the step list; its <em>Start</em> runs that single script and shows the result under the steps — not preserved) and the <strong>suite report</strong> (after a suite run: every script's status, its step table with expected/actual, the failing note and a run-log excerpt; the summary with passed/warnings/failed/skipped, wall time, the SCP80 counters before → after per keyset, the preset and a card-session-changed warning; export as <em>JSON</em> or <em>Markdown</em>). A <strong>suite run</strong> executes its scripts in order in <strong>one card session, without resets and without resume</strong>; <em>on failure</em> decides whether the suite stops at that script or runs the next one, the <strong>teardown always runs</strong> on a stop or a failure (while the card is in the reader) so the next test starts clean, and a card reset stops the suite and is flagged in the report. <strong>Start with card reset</strong> resets the card and runs the full re-initialization before the suite starts (the scripts then run against a freshly initialized card); the preset must still match the card's ICCID afterwards and a required ADM is verified again for the new card session. The initialization consumes the card's initial proactive commands, so a script cannot catch the first command after the reset (they stay in the proactive log). <strong>ADM required</strong> (on the suite or a script) verifies the matched preset's ADM key <strong>once</strong> before the start — a failure refuses the run (verify manually with the ADM badge; nothing is retried automatically). Scripts live in the suite's store (<code class="font-mono text-sm">~/.pysim-simple-server/test_suites.json</code>, <code class="font-mono text-sm">--test-suites</code>); every script belongs to exactly one suite, and a script can be moved or copied to another suite (a bundle export carries the suite plus its scripts; the old script-only export imports into an auto-created “Imported scripts” suite).</p>
|
||||
<p class="text-sm mb-2">A run can be repeated: the <strong>Iterations</strong> field next to <em>Start</em> / <em>Run suite</em> runs the script or the whole suite that many times — a <strong>load test</strong>. Successful iterations are not tracked: the status shows only the counter (iteration k / N) and the timing, the per-step log stays empty, and memory use stays flat however high the count. The first iteration that ends in an error stops the loop and keeps its full report (steps, failed checks, run-log excerpt, suite member report); a manual <em>Stop</em> ends it after the current iteration, and a card change between iterations is reported as an error (the next card is never run against another card’s script). The count is a run option — the saved script/suite is unchanged — and the SCP80 counters continue from what the previous iteration consumed.</p>
|
||||
<ul class="list-disc list-inside text-sm space-y-1 mb-3">
|
||||
<li><strong>Actions</strong> — <em>Envelope</em> (Event Download: raw event type + data, with an optional source override — 82 terminal / 83 network) or <em>Event</em> (the semantic forms: the <strong>same form the Phone tab's send dialog uses</strong>, for every modelled event — the four events the server builds (Location status, Access technology change, Network rejection, Data connection status change) travel as fields it validates and builds at run time, every other event carries the form's built event-data hex (shown live under the form, with the one-ENVELOPE check); the device-identities source comes from the event (e.g. MT call → network) or its source field), <em>Menu Selection</em> (item id 1–255, or the item's text with an exact/contains match, case-insensitive by default — the step editor's <em>Match case</em> choice switches to an exact-case match; the text is resolved against the card's menu, which follows the applet's install parameters), <em>UPDATE/READ file</em> (path + mode; reads verify the file data), <em>raw APDU</em> (no auto-handler), <em>SCP80</em> (C-APDU or a pre-built secured packet, optional keyset number (kvn) / TAR / SPI1 / SPI2 overrides and a command format — compact, or the TS 102 226 §5.2.1 Command Scripting template <em>expanded</em> (AA) / <em>expanded AE 80</em> around the C-APDU; the keys and the counter come from the card preset matched by the equipped card's ICCID — the run names the stored preset, so it always uses its keysets, TARs and counter — the server persists the consumed counter into it as the run proceeds), <em>STATUS</em> (with <em>attempts</em> > 1 it polls until the card announces a command) and <em>Proactive drain</em> (poll STATUS and consume every announced command — the TERMINAL RESPONSE result for each, optionally a different result for the first; the attempt/interval bound; an optional <em>require</em> assertion over what was drained. An empty poll passes, the final STATUS must be idle, and a still-pending command is handed to the next step — use it at the start, in the middle or at the end of a script to make sure the card has stopped sending; it also consumes a command a previous step left pending (a step that answered <code class="font-mono text-sm">91XX</code>). Every action has a response check: expected SW and data (exact, or a mask with <code class="font-mono text-sm">?</code> nibble wildcards — <code class="font-mono text-sm">91??</code>) plus a PoR check for SCP80 (<em>any / no PoR / por_ok</em>, or a custom assertion of the decoded PoR — status / SW / data; the SEND SHORT MESSAGE transport works the same).</li>
|
||||
<li><strong>Expectations</strong> — the card must have announced a proactive command in the previous step (SW <code class="font-mono text-sm">91XX</code>, TS 102 221 §7.4.2.1 / TS 102 223 §6.3); a <code class="font-mono text-sm">9000</code> means “no command”: the step fails at its fail level — Error stops the script, Warning continues (the tolerant “consume a command if there is one” form) — and a STATUS action with <em>attempts</em> > 1 polls when the applet delivers on poll. The command type, qualifier and contents are verified: text / item / raw checks, an <em>Alpha</em> check (the command's Alpha identifier, tag <code class="font-mono text-sm">05</code>/<code class="font-mono text-sm">85</code> — distinct from the <code class="font-mono text-sm">8D</code> text string), an <em>SMS TPDU</em> check (the submit TPDU of a SEND SHORT MESSAGE: destination / PID / DCS / UDL / user data — for an applet's own SMS), a <em>PoR</em> check (a SEND SHORT MESSAGE carries the PoR when the packet requested PoR-in-submit, SPI2 bit <code class="font-mono text-sm">20</code>: the response status, the R-APDU SW and the response data; make sure an SCP80 step ran first — the R-APDU SW lives <em>inside</em> the response payload: an applet's own TAR returns application data, so check its data and leave the SW empty; a refused ENVELOPE (<code class="font-mono text-sm">62XX</code>/<code class="font-mono text-sm">63XX</code>) may carry no response packet at all — leave the PoR check at <em>any</em> or expect <em>no PoR</em> for such a step) and a <em>Files</em> check (the File List of a REFRESH, e.g. qualifier <code class="font-mono text-sm">00</code> = NAA initialization with full file change notification). The scripted <strong>TERMINAL RESPONSE</strong> is part of the instruction: the <strong>result</strong> offers the full TS 102 223 §8.12.0 list (each entry labelled <code class="font-mono text-sm"><hex> - <alias or name></code>, plus a manual hex entry; with a command selected the list is grouped by the spec’s applicability table (the command-relevant values first, the rest after — nothing is hidden); the aliases follow the spec — <code class="font-mono text-sm">ok</code> 00, <code class="font-mono text-sm">modified</code> 07, <code class="font-mono text-sm">cancel</code> 10, <code class="font-mono text-sm">timeout</code> 12, <code class="font-mono text-sm">refused</code> 22, <code class="font-mono text-sm">not_understood</code> 32, …), the <strong>additional information</strong> byte of the general result (default <code class="font-mono text-sm">00</code>; the §8.12.2 terminal-problem values are suggested), the item identifier, the text (+DCS) and extra TLVs.</li>
|
||||
|
||||
Reference in new issue
Block a user