test runner: add the stuck-card watchdog

A card that stops answering (no SW, no driver error) leaves the runner
inside a blocking SCardTransmit while it holds the card lock - Stop
cannot take effect, the presence monitor waits for the lock and a load
test hangs in that iteration without explanation.

The runner now beats a progress heartbeat (each step start/finish and
each attempt of a bounded STATUS/drain loop); after _TEST_STEP_TIMEOUT
(30 s) the watchdog flags the run `stuck`, records the step and the idle
time, sets the stop flag and tries to cancel the blocked transmit by
disconnecting the PC/SC link on a daemon thread.  The disconnect must
run in the background: pcsc-lite serializes it behind the pending
transmit (both take the per-context lock), so a synchronous call would
block the watchdog instead of cancelling anything.  Where a platform
does cancel, the step fails through the ordinary on_fail path; either
way the finalisers map `stuck` to `error` - a stuck card is not a user
stop - and the pending drain is skipped on the dead link.  The
transport is marked stale once the disconnect returns, so the next equip
rebuilds it.

The run status gains `stuck` and `heartbeat`.  Help EN/RU,
docs/test-scripts.md, docs/api.md, the CHANGELOG (folded into the
unreleased 3.37.0) and both AGENTS describe the behaviour and the
background-disconnect caveat; the sw cache is simple-v476 (help is
precached).

Tests: tests/test_test_watchdog.py (17: the check rule, the abort, the
final status of the single/load/suite runs and a blocked-runner
integration case).  1058 Python + 1069 frontend.

CHANGELOG.md

@@ -25,6 +25,22 @@ every change carries a descriptive commit message.
   `done`, `failed_iteration`, `elapsed_ms`, `avg_ms`); the count is a run parameter, not
   part of the stored script or suite.
 
+- The **stuck-card watchdog** for test runs.  A card that stops answering (no status
+  word, no driver error) used to leave the runner inside a blocking `SCardTransmit`
+  holding the card lock — *Stop* could not take effect, the presence monitor waited for
+  the lock and a load test hung in that iteration without explanation.  After 30 s
+  without runner progress (a bounded STATUS/drain loop beats once per attempt) the
+  watchdog marks the run `stuck`, records `the card stopped responding (step N, no
+  answer for N s)`, sets the stop flag and **tries** to cancel the blocked transmit by
+  disconnecting the PC/SC link on a background thread.  Where PC/SC can cancel it, the
+  step fails through the ordinary `on_fail` path and the run finalises as an **error**
+  (never `stopped`); where it cannot (pcsc-lite serializes the disconnect behind the
+  pending transmit), the status already names the step until the driver's own card
+  timeout returns.  The transport is marked stale once the disconnect returns, so the
+  next *Equip* rebuilds it.  The status gains `stuck`; the help EN/RU,
+  `docs/test-scripts.md` and `docs/api.md` describe the behaviour and the
+  background-disconnect caveat.
+
 ### Fixed
 
 - The load-test loop closes properly on a raised runner error (it used to leave the run

docs/api.md

@@ -1039,7 +1039,15 @@ The current (or last) run:
 ```
 
 `status` becomes `ok`/`warning`/`error`/`stopped` when the run finishes;
-expected/actual pairs are reported per check.
+expected/actual pairs are reported per check.  A card that stops answering is
+not waited on silently: after 30 s without runner progress the watchdog sets
+`stuck: true`, `stop` and `error` (`the card stopped responding (step N, no
+answer for N s)`) and tries to cancel the blocked transmit by disconnecting
+the PC/SC link on a background thread (pcsc-lite serializes the disconnect
+behind the pending transmit, so the call returns only when the driver's card
+timeout fires).  Once the step returns, the run finalises as `error` - a stuck
+card is a failure, not a user stop - and the next equip rebuilds the link
+(the transport is marked stale).
 
 ### `POST /api/test/stop`
 

docs/test-scripts.md

@@ -242,6 +242,29 @@ between iterations is an error.  The SCP80 counter state is shared across
 iterations, so packets continue from what the previous iteration consumed.
 The iteration count is a run parameter - the stored script/suite is unchanged.
 
+## A card that stops answering
+
+pyscard's `SCardTransmit` is blocking and has no timeout: a card that stops
+answering (no status word, no driver error) would leave the runner inside the
+step while it holds the card lock - Stop could not take effect, the presence
+monitor would wait for the lock, and a load test would hang in that iteration.
+The runner's **watchdog** watches a progress heartbeat: after 30 s without one
+(a bounded STATUS/drain loop beats once per attempt, so a long poll is not
+mistaken for a stuck card) it marks the run `stuck`, sets the error
+`the card stopped responding (step N, no answer for N s)` and the stop flag,
+and **tries** to cancel the blocked transmit by disconnecting the PC/SC link
+on a background thread.  Where PC/SC can cancel it, the blocking call raises,
+the step fails through the ordinary path (`on_fail` applies) and the run
+finalises as an **error** - never as `stopped`, even though the stop flag
+ended the loop.  The disconnect runs in the background because pcsc-lite
+serializes it behind the pending transmit (both take the per-context lock), so
+on that stack the call returns only when the driver's own card timeout fires
+(libccid derives it from the card's ATR); the run is already flagged and the
+status names the step meanwhile.  The transport is marked stale once the
+disconnect returns, so the next **Equip** rebuilds the link.  A physical card
+removal is unaffected: the driver aborts the transmit and the presence monitor
+clears the card session.
+
 ## Document shape
 
 ```json

frontend/help-ru.html

@@ -571,7 +571,7 @@
 
 <section class="mb-10">
 	<h3 id="test-script" class="text-lg font-medium mb-2">8.9 Тест-скрипт</h3>
-	<p class="text-sm mb-2"><strong>Тест-скрипт</strong> &mdash; упорядоченный список <strong>действий</strong> и <strong>ожиданий</strong> проактивных команд, выполняемый на подключённой карте на стороне сервера. Пока запуск активен, карта принадлежит скрипту: остальные операции с картой отклоняются (HTTP&nbsp;409), фоновый опрос STATUS и автоматическая обработка проактивных команд приостанавливаются &mdash; объявленная картой команда остаётся ожидающей для скрипта, и оба механизма возобновляются по завершении запуска. Собственные BIP-команды карты (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) — исключение: терминал отвечает на них по-настоящему: каналы продолжают работать во время запуска, а BIP-события приходят для каналов, открытых самой картой (управление BIP остаётся заблокированным) — если только <code class="font-mono text-sm">respond</code> шага не переопределяет ответ через <code class="font-mono text-sm">raw</code> TLV или результат, отличный от ok; отчёт о запуске хранится в памяти (в том числе после перезагрузки страницы) до нажатия <strong>Очистить отчёт</strong>. Скрипты хранятся в <strong>хранилище на сервере</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), поэтому все браузеры работают с одним набором; изменения записываются в хранилище по мере правки (кнопки &laquo;Сохранить&raquo; нет &mdash; черновик без валидного шага остаётся локальным); экспорт/импорт JSON остаётся для резервных копий и переноса скриптов между стендами (старый экспорт из localStorage импортируется как есть).</p>
+	<p class="text-sm mb-2"><strong>Тест-скрипт</strong> &mdash; упорядоченный список <strong>действий</strong> и <strong>ожиданий</strong> проактивных команд, выполняемый на подключённой карте на стороне сервера. Пока запуск активен, карта принадлежит скрипту: остальные операции с картой отклоняются (HTTP&nbsp;409), фоновый опрос STATUS и автоматическая обработка проактивных команд приостанавливаются &mdash; объявленная картой команда остаётся ожидающей для скрипта, и оба механизма возобновляются по завершении запуска. Собственные BIP-команды карты (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) — исключение: терминал отвечает на них по-настоящему: каналы продолжают работать во время запуска, а BIP-события приходят для каналов, открытых самой картой (управление BIP остаётся заблокированным) — если только <code class="font-mono text-sm">respond</code> шага не переопределяет ответ через <code class="font-mono text-sm">raw</code> TLV или результат, отличный от ok; отчёт о запуске хранится в памяти (в том числе после перезагрузки страницы) до нажатия <strong>Очистить отчёт</strong>. Скрипты хранятся в <strong>хранилище на сервере</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), поэтому все браузеры работают с одним набором; изменения записываются в хранилище по мере правки (кнопки &laquo;Сохранить&raquo; нет &mdash; черновик без валидного шага остаётся локальным); экспорт/импорт JSON остаётся для резервных копий и переноса скриптов между стендами (старый экспорт из localStorage импортируется как есть).  Карта, переставшая отвечать, не подвешивает запуск молча: через 30&nbsp;с без прогресса сторожевой таймер помечает запуск как ошибку с указанием шага, ставит флаг остановки и пытается прервать зависший обмен, разрывая связь PC/SC в фоновом потоке (связь помечается устаревшей, так что следующий <strong>Equip</strong> пересоздаёт её).  Драйвер с собственным тайм-аутом карты завершает шаг обычным путём.</p>
 	<p class="text-sm mb-2">У вкладки <strong>четыре представления</strong>: <strong>список наборов</strong> (по умолчанию; <em>Добавить набор</em> / <em>Редактировать</em> / <em>Запустить</em> / <em>Экспорт</em> / <em>Удалить</em>), <strong>редактор набора</strong> (скрипты набора в порядке выполнения &mdash; необязательный <em>setup</em> первым и <em>teardown</em> последним, с пометками ролей; у каждого скрипта <em>Запустить</em>/<em>Редактировать</em>/<em>Переместить</em>/<em>Скопировать</em>/<em>Удалить</em>, политика <em>при ошибке</em> <em>остановить</em> (по умолчанию) / <em>продолжить</em> и <em>Требуется ADM</em>), <strong>редактор скрипта</strong> (список шагов; кнопка <em>Старт</em> выполняет только этот скрипт и показывает результат под шагами &mdash; он не сохраняется) и <strong>отчёт набора</strong> (после запуска набора: статус каждого скрипта, таблица шагов с ожидаемым/фактическим, примечание об ошибке и выдержка из журнала запуска; сводка с успешно/предупреждения/ошибки/пропущено, время выполнения, счётчики SCP80 до&nbsp;&rarr;&nbsp;после по каждому keyset, пресет и предупреждение о смене сессии карты; экспорт в <em>JSON</em> или <em>Markdown</em>).  <strong>Запуск набора</strong> выполняет его скрипты по порядку в <strong>одной сессии карты, без сбросов и без продолжения с места остановки</strong>; <em>при ошибке</em> решает, остановится ли набор на этом скрипте или запустится следующий, <strong>teardown выполняется всегда</strong> при остановке или ошибке (пока карта в ридере), чтобы следующий тест начался с чистого состояния, а сброс карты останавливает набор и отмечается в отчёте.  <strong>Начать со сброса карты</strong> сбрасывает карту и выполняет полную переинициализацию перед запуском набора (скрипты затем работают со свежеинициализированной картой); пресет должен по-прежнему совпадать с ICCID карты, а требуемый ADM проверяется заново для новой сессии карты.  Инициализация поглощает начальные proactive-команды карты, поэтому скрипт не поймает первую команду после сброса (они остаются в журнале proactive).  <strong>Требуется ADM</strong> (у набора или скрипта) проверяет ключ ADM из подобранного пресета <strong>один раз</strong> перед запуском &mdash; ошибка отклоняет запуск (проверьте вручную через значок ADM; автоматических повторов нет).  Скрипты хранятся в хранилище наборов (<code class="font-mono text-sm">~/.pysim-simple-server/test_suites.json</code>, <code class="font-mono text-sm">--test-suites</code>); каждый скрипт принадлежит ровно одному набору, и скрипт можно переместить или скопировать в другой набор (экспорт набора &mdash; это пакет с набором и его скриптами; старый экспорт только скриптов импортируется в автоматически созданный набор &laquo;Imported scripts&raquo;).</p>
 	<p class="text-sm mb-2">Прогон можно повторить: поле <strong>«Итерации»</strong> рядом с <em>Start</em> / <em>Run suite</em> выполняет скрипт или весь набор указанное число раз &mdash; это <strong>нагрузочный тест</strong>. Успешные итерации не сохраняются: в статусе видны только счётчик (итерация k / N) и время, журнал шагов остаётся пустым, а память не растёт при любом числе итераций. Первая итерация, завершившаяся ошибкой, останавливает цикл и сохраняет свой полный отчёт (шаги, неудачные проверки, фрагмент журнала, отчёт по скриптам набора); <em>Остановить</em> завершает работу после текущей итерации, а смена карты между итерациями считается ошибкой (следующая карта никогда не запускается со скриптом другой карты). Число итераций &mdash; параметр прогона, сохранённый скрипт/набор не меняется; счётчики SCP80 продолжаются с того значения, которое израсходовала предыдущая итерация.</p>
 	<ul class="list-disc list-inside text-sm space-y-1 mb-3">

frontend/help.html

@@ -570,7 +570,7 @@
 
 <section class="mb-10">
 	<h3 id="test-script" class="text-lg font-medium mb-2">8.9 Test script</h3>
-	<p class="text-sm mb-2">A <strong>test script</strong> is an ordered list of <strong>actions</strong> and proactive-command <strong>expectations</strong> executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP&nbsp;409), and background STATUS polling and the automatic proactive handling are suspended &mdash; a command the card announces stays pending for the script, and both resume when the run ends. The card&rsquo;s own BIP commands (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) are an exception: the terminal answers them for real &mdash; its channels keep working during the run and the BIP events flow for the channels the card itself opened (BIP <em>control</em> stays blocked) &mdash; unless the step&rsquo;s <code class="font-mono text-sm">respond</code> overrides the answer with <code class="font-mono text-sm">raw</code> TLVs or a non-ok result; the run report is kept in memory (also after reloading the page) until <strong>Clear report</strong>. Scripts live in the <strong>server-side store</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), so every browser manages the same set; edits are written to the store as you change them (there is no Save button &mdash; a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).</p>
+	<p class="text-sm mb-2">A <strong>test script</strong> is an ordered list of <strong>actions</strong> and proactive-command <strong>expectations</strong> executed server-side on the equipped card. While a run is active the card belongs to the script: other card actions are refused (HTTP&nbsp;409), and background STATUS polling and the automatic proactive handling are suspended &mdash; a command the card announces stays pending for the script, and both resume when the run ends. The card&rsquo;s own BIP commands (OPEN/SEND/RECEIVE/CLOSE CHANNEL, GET CHANNEL STATUS) are an exception: the terminal answers them for real &mdash; its channels keep working during the run and the BIP events flow for the channels the card itself opened (BIP <em>control</em> stays blocked) &mdash; unless the step&rsquo;s <code class="font-mono text-sm">respond</code> overrides the answer with <code class="font-mono text-sm">raw</code> TLVs or a non-ok result; the run report is kept in memory (also after reloading the page) until <strong>Clear report</strong>. Scripts live in the <strong>server-side store</strong> (<code class="font-mono text-sm">~/.pysim-simple-server/test_scripts.json</code>, <code class="font-mono text-sm">--test-scripts</code>), so every browser manages the same set; edits are written to the store as you change them (there is no Save button &mdash; a draft without a valid step stays local until it is valid); JSON export/import remains for backup and for moving scripts between benches (the old localStorage export imports as-is).  A card that stops answering does not hang the run silently: after 30&nbsp;s without progress the watchdog marks the run as failed, naming the step, sets the stop flag and tries to cancel the blocked exchange by disconnecting the PC/SC link on a background thread (the link is marked stale, so the next <strong>Equip</strong> rebuilds it).  A driver with its own card timeout fails the step through the normal path instead.</p>
 	<p class="text-sm mb-2">The pill has <strong>four views</strong>: the <strong>suite list</strong> (default; <em>Add suite</em> / <em>Edit</em> / <em>Run</em> / <em>Export</em> / <em>Delete</em>), the <strong>suite editor</strong> (the suite's scripts in run order &mdash; the optional <em>setup</em> first and the <em>teardown</em> last, marked with role badges; per script <em>Run</em>/<em>Edit</em>/<em>Move</em>/<em>Copy</em>/<em>Delete</em>, the per-member <em>on failure</em> policy <em>stop</em> (default) / <em>continue</em>, and <em>ADM required</em>), the <strong>script editor</strong> (the step list; its <em>Start</em> runs that single script and shows the result under the steps &mdash; not preserved) and the <strong>suite report</strong> (after a suite run: every script's status, its step table with expected/actual, the failing note and a run-log excerpt; the summary with passed/warnings/failed/skipped, wall time, the SCP80 counters before&nbsp;&rarr;&nbsp;after per keyset, the preset and a card-session-changed warning; export as <em>JSON</em> or <em>Markdown</em>).  A <strong>suite run</strong> executes its scripts in order in <strong>one card session, without resets and without resume</strong>; <em>on failure</em> decides whether the suite stops at that script or runs the next one, the <strong>teardown always runs</strong> on a stop or a failure (while the card is in the reader) so the next test starts clean, and a card reset stops the suite and is flagged in the report.  <strong>Start with card reset</strong> resets the card and runs the full re-initialization before the suite starts (the scripts then run against a freshly initialized card); the preset must still match the card's ICCID afterwards and a required ADM is verified again for the new card session.  The initialization consumes the card's initial proactive commands, so a script cannot catch the first command after the reset (they stay in the proactive log).  <strong>ADM required</strong> (on the suite or a script) verifies the matched preset's ADM key <strong>once</strong> before the start &mdash; a failure refuses the run (verify manually with the ADM badge; nothing is retried automatically).  Scripts live in the suite's store (<code class="font-mono text-sm">~/.pysim-simple-server/test_suites.json</code>, <code class="font-mono text-sm">--test-suites</code>); every script belongs to exactly one suite, and a script can be moved or copied to another suite (a bundle export carries the suite plus its scripts; the old script-only export imports into an auto-created &ldquo;Imported scripts&rdquo; suite).</p>
 	<p class="text-sm mb-2">A run can be repeated: the <strong>Iterations</strong> field next to <em>Start</em> / <em>Run suite</em> runs the script or the whole suite that many times &mdash; a <strong>load test</strong>. Successful iterations are not tracked: the status shows only the counter (iteration k / N) and the timing, the per-step log stays empty, and memory use stays flat however high the count. The first iteration that ends in an error stops the loop and keeps its full report (steps, failed checks, run-log excerpt, suite member report); a manual <em>Stop</em> ends it after the current iteration, and a card change between iterations is reported as an error (the next card is never run against another card&rsquo;s script). The count is a run option &mdash; the saved script/suite is unchanged &mdash; and the SCP80 counters continue from what the previous iteration consumed.</p>
 	<ul class="list-disc list-inside text-sm space-y-1 mb-3">

frontend/sw.js

@@ -1,4 +1,4 @@
-const CACHE = 'simple-v475';
+const CACHE = 'simple-v476';
 const URLS = [
   'index.html',
   'help.html',

pysim_simple_server/server.py

@@ -7056,6 +7056,10 @@ _TEST_RUN = {
     'preset': None, 'script_id': None,
     'scp80_counter': None, 'scp80_counters': {},
     'started': None, 'finished': None, 'error': None,
+    # the stuck-card watchdog (v3.37.0): `heartbeat` is the last runner
+    # progress mark; `stuck` makes the final status an error even when the
+    # stop flag ended the loop
+    'stuck': False, 'heartbeat': None,
     # suite runs (v3.22.0): None for a single script; the members list and the
     # report summary live here, plus the run-log buffer/adm state
     'suite': None, 'log': [], 'log_prefix': '', 'adm': None,
@@ -7190,6 +7194,122 @@ def _test_state_snapshot():
         return json.loads(json.dumps(_TEST_RUN))
 
 
+# ─── the stuck-card watchdog (v3.37.0) ───────────────────────────────────
+# pyscard's SCardTransmit is blocking and has no timeout: a card that stops
+# answering (no SW, no driver error) would leave the runner thread inside the
+# step while it holds _CARD_LOCK - Stop cannot take effect, the card monitor
+# waits for the lock and a load test hangs in that iteration.  The watchdog
+# watches the runner's progress heartbeat; after _TEST_STEP_TIMEOUT seconds
+# without one it flags the run stuck (error + stop) and tries to cancel the
+# pending transmit by disconnecting the PC/SC link.  That attempt runs in the
+# background because PC/SC clients may serialize the disconnect behind the
+# pending transmit (pcsc-lite locks the context for both calls): when the
+# cancel works, the blocking call raises and the normal step-failure path
+# finalises the run; when it does not, the driver's own timeout eventually
+# returns and the run still finalises as an error, and the status already
+# explains the wait.  The transport is marked stale so the next equip rebuilds
+# it.
+
+_TEST_STEP_TIMEOUT = 30        # seconds without runner progress => stuck card
+_TEST_WATCHDOG = None
+_TEST_WATCHDOG_ABORTED = None  # heartbeat value already aborted
+
+
+def _test_heartbeat():
+    """Mark runner progress for the watchdog.  A bounded STATUS/drain loop may
+    legitimately run longer than one exchange, so it beats once per attempt; a
+    single exchange beats when its step starts and when it finishes (so the
+    gap before the next step or the cleanup drain is covered)."""
+    with _TEST_LOCK:
+        _TEST_RUN['heartbeat'] = time.time()
+
+
+def _abort_card_transport():
+    """Try to cancel a blocking SCardTransmit, then mark the transport stale.
+
+    The disconnect runs on its own daemon thread: PC/SC clients may serialize
+    it behind the pending transmit (pcsc-lite takes the per-context lock in
+    both calls), so calling it here would block the watchdog instead of
+    cancelling anything.  Where the platform does cancel, the blocked call
+    raises and the normal step-failure path finalises the run; where it does
+    not, the run stays flagged stuck until the driver returns.  Never takes
+    _CARD_LOCK - the stuck runner holds it."""
+    global _TRANSPORT_STALE
+    scc = getattr(_server_ref, 'scc', None) if _server_ref else None
+    con = getattr(getattr(scc, '_tp', None), '_con', None)
+    if con is None:
+        sys.stderr.write('TEST-WATCHDOG: no PC/SC connection to abort\n')
+        _TRANSPORT_STALE = True
+        return False
+
+    def _run():
+        global _TRANSPORT_STALE
+        try:
+            con.disconnect()
+        except Exception as e:
+            sys.stderr.write('TEST-WATCHDOG: PC/SC disconnect failed: %s\n' % e)
+        finally:
+            _TRANSPORT_STALE = True
+
+    threading.Thread(target=_run, name='test-abort', daemon=True).start()
+    return True
+
+
+def _test_watchdog_check(now=None):
+    """One watchdog pass.  The heartbeat is the authority (it is beaten at
+    every step start/finish and inside the bounded drains), so a stale one
+    means the runner is stuck inside a card exchange.  Returns the stuck error
+    string when it acted (once per heartbeat), else None.  Split from the
+    loop so the rule is testable."""
+    global _TEST_WATCHDOG_ABORTED
+    now = time.time() if now is None else now
+    with _TEST_LOCK:
+        if not _TEST_RUN.get('running'):
+            _TEST_WATCHDOG_ABORTED = None
+            return None
+        hb = _TEST_RUN.get('heartbeat')
+        if not hb:
+            return None
+        idle = now - hb
+        if idle < _TEST_STEP_TIMEOUT:
+            return None
+        if _TEST_WATCHDOG_ABORTED == hb:
+            return None
+        _TEST_WATCHDOG_ABORTED = hb
+        steps = _TEST_RUN.get('steps') or []
+        cur = steps[-1] if steps else None
+        index = cur.get('index') if cur is not None else _TEST_RUN.get('index')
+        msg = ('the card stopped responding (step %d, no answer for %d s)'
+               % ((index or 0) + 1, int(idle)))
+        _TEST_RUN['stuck'] = True
+        _TEST_RUN['error'] = msg
+        _TEST_RUN['stop'] = True
+    sys.stderr.write('TEST-WATCHDOG: %s - trying to cancel the pending '
+                     'transmit\n' % msg)
+    _abort_card_transport()
+    return msg
+
+
+def _test_watchdog_loop():
+    while True:
+        time.sleep(1.0)
+        try:
+            _test_watchdog_check()
+        except Exception as e:
+            sys.stderr.write('TEST-WATCHDOG error: %s\n' % e)
+
+
+def _test_watchdog_start():
+    """Start the singleton watchdog thread (idempotent; it exits with the
+    process)."""
+    global _TEST_WATCHDOG
+    if _TEST_WATCHDOG is not None and _TEST_WATCHDOG.is_alive():
+        return
+    _TEST_WATCHDOG = threading.Thread(target=_test_watchdog_loop,
+                                      name='test-watchdog', daemon=True)
+    _TEST_WATCHDOG.start()
+
+
 def _test_suite_refs_error(script_store, suite):
     """The cross-store rule: every member script must exist in the script
     store (a suite holds references; the scripts are the single source of
@@ -7506,6 +7626,7 @@ def _test_finish_entry(entry, result):
                 'command', 'drained'):
         if result.get(key) is not None:
             fields[key] = result[key]
+    _test_heartbeat()          # the exchange finished: runner progress
     _test_entry_update(entry, fields)
 
 
@@ -7599,6 +7720,7 @@ def _test_run_status(scc, step):
     data, sw = '', ''
     used = 0
     for n in range(p['attempts']):
+        _test_heartbeat()
         data, sw = _send_status(scc)
         used = n + 1
         _test_log('STATUS %d/%d -> %s' % (used, p['attempts'], sw))
@@ -7687,6 +7809,7 @@ def _test_run_drain(server, step):
     sw = ''
     polls = 0
     for n in range(p['attempts']):
+        _test_heartbeat()
         data, sw = _send_status(scc)
         polls = n + 1
         _test_log('STATUS %d/%d -> %s' % (polls, p['attempts'], sw))
@@ -8316,6 +8439,7 @@ def _test_drain_pending(scc, fetch_len, result=0x00, max_commands=3):
     answered = []
     try:
         while len(answered) < max_commands:
+            _test_heartbeat()      # a bounded drain still makes progress
             # fetch_len 0 (a 9100 SW) means "length unknown": fetch the full
             # 255 bytes - `%02x` of 0x100 would emit an odd-length APDU
             fdata, sw = scc._tp.send_apdu('%s120000%02x' % (scc.cat_cla,
@@ -8390,6 +8514,7 @@ def _test_run_script(server, script, preset, shared=None):
         entry = _test_entry(step, index)
         with _TEST_LOCK:
             _TEST_RUN['steps'].append(entry)
+            _TEST_RUN['heartbeat'] = time.time()
         # A proactive-drain step is the tool that consumes a pending command
         # (it polls STATUS - the card re-announces the unfetched command -
         # then fetches and answers it), so the guard does not apply to it
@@ -8437,14 +8562,16 @@ def _test_run_script(server, script, preset, shared=None):
                                        'note': str(e)})
             sys.stderr.write('TEST-RUN step %d failed (%s): %s\n'
                              % (index + 1, level, e))
-            if pending is not None:
+            # A stuck card's transport is already being torn down: draining
+            # would only raise again on the dead link.
+            if pending is not None and not _TEST_RUN.get('stuck'):
                 with _CARD_LOCK:
                     drained = _test_drain_pending(scc, pending)
                 _test_entry_update(entry, {k: v for k, v in drained.items() if k != 'note'})
                 pending = None
             if level == 'error':
                 break
-    if pending is not None:
+    if pending is not None and not _TEST_RUN.get('stuck'):
         with _CARD_LOCK:
             drained = _test_drain_pending(scc, pending)
         with _TEST_LOCK:
@@ -8472,7 +8599,11 @@ def _test_run_execute(server, script, preset):
         _TEST_RUN['finished'] = time.time()
         if _TEST_RUN['status'] is None:
             levels = [s.get('status') for s in _TEST_RUN['steps']]
-            if stopped:
+            if _TEST_RUN.get('stuck'):
+                # the watchdog's stop flag ended the loop; the run still
+                # failed (a stuck card is not a user stop)
+                _TEST_RUN['status'] = 'error'
+            elif stopped:
                 _TEST_RUN['status'] = 'stopped'
             elif 'error' in levels:
                 _TEST_RUN['status'] = 'error'
@@ -8511,9 +8642,12 @@ def _test_member_prefix(index, member):
     return '[%d/%d %s %s] ' % (index + 1, total, role, tag)
 
 
-def _test_member_status(steps, stopped):
+def _test_member_status(steps, stopped, stuck=False):
     """A suite member's verdict from its step levels (an error stops that
-    script); a stop request marks the member stopped."""
+    script); a stop request marks the member stopped.  A stuck card is an
+    error even though the watchdog's stop flag ended the script."""
+    if stuck:
+        return 'error'
     if stopped:
         return 'stopped'
     levels = [s.get('status') for s in steps]
@@ -8596,6 +8730,7 @@ def _test_run_spawn(server, target, args, thread_name):
     """The shared run start: the run owns the card from here - suspend the
     background poll, answer a paused interactive command, spawn the worker."""
     global _TEST_THREAD, _TEST_RUNNING, _POLL_TIMER
+    _test_watchdog_start()
     _TEST_RUNNING = True
     if _POLL_TIMER is not None:
         _POLL_TIMER.cancel()
@@ -8631,7 +8766,8 @@ def _test_run_start(server, script, preset, script_id=None, adm_info=None,
             'running': True, 'stop': False, 'name': script['name'], 'status': None,
             'session': getattr(server, 'card_session', 0), 'index': 0,
             'total': len(script['steps']), 'steps': [], 'started': time.time(),
-            'finished': None, 'error': None, 'scp80_counter': None,
+            'finished': None, 'error': None, 'stuck': False,
+            'heartbeat': time.time(), 'scp80_counter': None,
             'scp80_counters': {}, 'script_id': script_id,
             'preset': (preset or {}).get('name') or (preset or {}).get('iccid'),
             'suite': None, 'log': [], 'log_prefix': '',
@@ -8678,6 +8814,7 @@ def _test_suite_execute_once(server, suite, members, preset, preset_id, state,
             member['started'] = time.time()
             _TEST_RUN['steps'] = []
             _TEST_RUN['index'] = 0
+            _TEST_RUN['heartbeat'] = time.time()
             _TEST_RUN['total'] = len(script['steps'])
             _TEST_RUN['log_prefix'] = _test_member_prefix(i, member)
             log_from = len(_TEST_RUN['log'])
@@ -8686,7 +8823,9 @@ def _test_suite_execute_once(server, suite, members, preset, preset_id, state,
             steps = json.loads(json.dumps(_TEST_RUN['steps']))
             log = list(_TEST_RUN['log'][log_from:])
             member.update({
-                'status': _test_member_status(steps, member_stopped),
+                'status': _test_member_status(
+                    steps, member_stopped,
+                    stuck=bool(_TEST_RUN.get('stuck'))),
                 'steps': steps,
                 'log': log[-_TEST_MEMBER_LOG_MAX:],
                 'log_truncated': len(log) > _TEST_MEMBER_LOG_MAX,
@@ -8880,6 +9019,7 @@ def _test_suite_start(server, suite, members, preset, preset_id, adm_info=None,
             'name': suite.get('name') or 'test suite', 'status': None,
             'session': session, 'index': 0, 'total': 0, 'steps': [],
             'started': time.time(), 'finished': None, 'error': None,
+            'stuck': False, 'heartbeat': time.time(),
             'scp80_counter': None, 'scp80_counters': {},
             'script_id': None,
             'preset': (preset or {}).get('name') or (preset or {}).get('iccid'),
@@ -8946,7 +9086,9 @@ def _test_load_script_worker(server, script, preset, script_id, iterations):
                     final = 'stopped'
                     break
                 _TEST_RUN.update({'steps': [], 'index': 0, 'status': None,
-                                  'error': None, 'log': [], 'log_prefix': ''})
+                                  'error': None, 'stuck': False,
+                                  'heartbeat': time.time(), 'log': [],
+                                  'log_prefix': ''})
                 _TEST_RUN['load'].update({'iteration': i, 'done': done})
             attempted = i
             if getattr(server, 'card_session', 0) != session:
@@ -8959,7 +9101,9 @@ def _test_load_script_worker(server, script, preset, script_id, iterations):
             _ctx, stopped = _test_run_script(server, script, preset, shared=shared)
             with _TEST_LOCK:
                 levels = [st.get('status') for st in _TEST_RUN['steps']]
-                if stopped:
+                if _TEST_RUN.get('stuck'):
+                    status = 'error'
+                elif stopped:
                     status = 'stopped'
                 elif 'error' in levels:
                     status = 'error'
@@ -9026,6 +9170,7 @@ def _test_load_suite_worker(server, suite, members, preset, preset_id, iteration
                 state = _test_suite_state(suite, members, fresh, preset_id, session)
                 _TEST_RUN.update({'suite': state, 'steps': [], 'index': 0,
                                   'total': 0, 'status': None, 'error': None,
+                                  'stuck': False, 'heartbeat': time.time(),
                                   'log': [], 'log_prefix': ''})
                 _TEST_RUN['load'].update({'iteration': i, 'done': done})
             attempted = i
@@ -9045,9 +9190,8 @@ def _test_load_suite_worker(server, suite, members, preset, preset_id, iteration
                 failed = i
                 final = 'error'
                 break
-            if stopped or status == 'stopped':
+            # an error wins over the stop flag: a stuck card sets both, and
-                final = 'stopped'
+            # the report must say it failed, not that the operator stopped it
-                break
             if status == 'error':
                 failed = i
                 final = 'error'
@@ -9055,6 +9199,9 @@ def _test_load_suite_worker(server, suite, members, preset, preset_id, iteration
                     if not _TEST_RUN.get('error'):
                         _TEST_RUN['error'] = 'iteration %d failed' % i
                 break
+            if stopped or status == 'stopped':
+                final = 'stopped'
+                break
             if status == 'warning':
                 worst = 'warning'
             with _TEST_LOCK:

tests/test_test_watchdog.py

@@ -0,0 +1,317 @@
+"""The test-runner stuck-card watchdog (v3.37.0).
+
+pyscard's SCardTransmit is blocking and has no timeout: a card that stops
+answering (no SW, no driver error) leaves the runner inside a step holding
+_CARD_LOCK, so Stop cannot take effect and a load test hangs in that iteration.
+The watchdog notices the missing runner progress, fails the run, and
+disconnects the PC/SC link - the pending transmit then raises and the normal
+step-failure path finalises the run.  The transport is marked stale so the next
+equip/operation rebuilds it.
+"""
+
+import copy
+import threading
+import time
+import unittest
+from unittest import mock
+
+from pysim_simple_server import server as srv
+from pysim_simple_server import testscript as T
+
+
+def _warning_script():
+    return T.normalise_script({'name': 'stuck', 'steps': [
+        {'type': 'action', 'kind': 'status', 'params': {'attempts': 1},
+         'check': {'sw': '9000'}, 'on_fail': 'warning'}]},
+        srv._test_command_type)
+
+
+class _RaisingScc:
+    """send_apdu raises like a disconnected PC/SC handle after the abort."""
+    cat_cla = '80'
+
+    def __init__(self):
+        self._tp = self
+
+    def send_apdu(self, apdu):
+        raise RuntimeError('SCARD_E_INVALID_HANDLE')
+
+
+class _FakeServer:
+    def __init__(self, scc=None, session=7):
+        self.scc = scc
+        self.card_session = session
+        self.card_present = True
+        self.sms_oa = '12345'
+        self.sms_sc = '12345678912'
+        self.stk_pending = None
+        self.menu_active = False
+
+
+class _FakeCon:
+    def __init__(self, fail=False):
+        self.calls = 0
+        self.fail = fail
+        self.done = threading.Event()
+
+    def disconnect(self):
+        self.calls += 1
+        try:
+            if self.fail:
+                raise RuntimeError('dead handle')
+        finally:
+            self.done.set()
+
+
+class _FakeScc:
+    def __init__(self, con):
+        self._tp = type('T', (), {'_con': con})()
+
+
+class _WatchdogTestCase(unittest.TestCase):
+    def setUp(self):
+        self.saved_run = copy.deepcopy(srv._TEST_RUN)
+        self.saved_ref = srv._server_ref
+        self.saved_stale = srv._TRANSPORT_STALE
+        self.saved_aborted = srv._TEST_WATCHDOG_ABORTED
+        srv._TEST_WATCHDOG_ABORTED = None
+        srv._TRANSPORT_STALE = False
+
+    def tearDown(self):
+        srv._TEST_RUN.clear()
+        srv._TEST_RUN.update(self.saved_run)
+        srv._server_ref = self.saved_ref
+        srv._TRANSPORT_STALE = self.saved_stale
+        srv._TEST_WATCHDOG_ABORTED = self.saved_aborted
+
+    def _run(self, **fields):
+        srv._TEST_RUN.update({
+            'running': True, 'stop': False, 'status': None, 'error': None,
+            'stuck': False, 'heartbeat': None, 'steps': [], 'index': 0,
+        })
+        srv._TEST_RUN.update(fields)
+
+    @staticmethod
+    def _step(started, index=0, status='running'):
+        return {'index': index, 'status': status, 'started': started}
+
+
+class WatchdogCheckTests(_WatchdogTestCase):
+    def test_an_idle_runner_is_not_watched(self):
+        now = time.time()
+        self._run(running=False, heartbeat=now - 100,
+                  steps=[self._step(now - 100)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNone(srv._test_watchdog_check(now))
+        abort.assert_not_called()
+
+    def test_a_step_within_the_timeout_is_ignored(self):
+        now = time.time()
+        started = now - srv._TEST_STEP_TIMEOUT + 5
+        self._run(heartbeat=started, steps=[self._step(started)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNone(srv._test_watchdog_check(now))
+        abort.assert_not_called()
+
+    def test_finished_work_with_fresh_progress_is_ignored(self):
+        now = time.time()
+        self._run(heartbeat=now - 1,
+                  steps=[self._step(now - 100, status='ok')])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNone(srv._test_watchdog_check(now))
+        abort.assert_not_called()
+
+    def test_a_stale_heartbeat_between_steps_still_fires(self):
+        # the bounded drains run with no step entry "running": the heartbeat
+        # is the authority, so a hang there is caught too
+        now = time.time()
+        self._run(heartbeat=now - 100,
+                  steps=[self._step(now - 100, index=4, status='ok')])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            msg = srv._test_watchdog_check(now)
+        self.assertIn('step 5', msg)
+        abort.assert_called_once()
+
+    def test_a_stuck_step_fails_the_run_and_aborts_the_link(self):
+        now = time.time()
+        started = now - srv._TEST_STEP_TIMEOUT - 2
+        self._run(heartbeat=started, steps=[self._step(started)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            msg = srv._test_watchdog_check(now)
+        self.assertIn('the card stopped responding', msg)
+        self.assertIn('step 1', msg)
+        self.assertIn('no answer for', msg)
+        self.assertTrue(srv._TEST_RUN['stuck'])
+        self.assertTrue(srv._TEST_RUN['stop'])
+        self.assertEqual(srv._TEST_RUN['error'], msg)
+        abort.assert_called_once()
+
+    def test_the_abort_happens_once_per_heartbeat(self):
+        now = time.time()
+        started = now - 100
+        self._run(heartbeat=started, steps=[self._step(started)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNotNone(srv._test_watchdog_check(now))
+            self.assertIsNone(srv._test_watchdog_check(now + 1))
+            self.assertIsNone(srv._test_watchdog_check(now + 2))
+        abort.assert_called_once()
+
+    def test_new_progress_rearms_the_watchdog(self):
+        now = time.time()
+        started = now - 100
+        self._run(heartbeat=started, steps=[self._step(started)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNotNone(srv._test_watchdog_check(now))
+            # the runner made progress (a new beat) and then hung again
+            srv._TEST_RUN['steps'].append(self._step(started + 1, index=1))
+            srv._TEST_RUN['heartbeat'] = started + 1
+            self.assertIsNotNone(srv._test_watchdog_check(now))
+        self.assertEqual(abort.call_count, 2)
+
+    def test_a_heartbeat_keeps_a_long_poll_loop_alive(self):
+        # a bounded STATUS/drain loop may legitimately run longer than one
+        # exchange: its per-attempt heartbeat must keep the watchdog quiet
+        now = time.time()
+        self._run(heartbeat=now - 1, steps=[self._step(now - 120)])
+        with mock.patch.object(srv, '_abort_card_transport') as abort:
+            self.assertIsNone(srv._test_watchdog_check(now))
+        abort.assert_not_called()
+
+
+class TransportAbortTests(_WatchdogTestCase):
+    def _wait_stale(self):
+        for _ in range(200):
+            if srv._TRANSPORT_STALE:
+                return True
+            time.sleep(0.01)
+        return False
+
+    def test_the_connection_is_disconnected_and_marked_stale(self):
+        # the disconnect is attempted on a background thread (a synchronous
+        # one would block behind the pending transmit on pcsc-lite)
+        con = _FakeCon()
+        srv._server_ref = _FakeServer(_FakeScc(con))
+        self.assertTrue(srv._abort_card_transport())
+        self.assertTrue(con.done.wait(2))
+        self.assertEqual(con.calls, 1)
+        self.assertTrue(self._wait_stale())
+
+    def test_a_missing_connection_is_not_fatal(self):
+        srv._server_ref = _FakeServer(None)
+        self.assertFalse(srv._abort_card_transport())
+        self.assertTrue(srv._TRANSPORT_STALE)
+
+    def test_a_failing_disconnect_is_swallowed(self):
+        con = _FakeCon(fail=True)
+        srv._server_ref = _FakeServer(_FakeScc(con))
+        self.assertTrue(srv._abort_card_transport())
+        self.assertTrue(con.done.wait(2))
+        self.assertEqual(con.calls, 1)
+        self.assertTrue(self._wait_stale())
+
+
+class HeartbeatTests(_WatchdogTestCase):
+    def test_a_finished_step_beats(self):
+        # between steps the heartbeat must stay fresh (the drains run with no
+        # step entry "running")
+        self._run(heartbeat=0)
+        entry = {'index': 0, 'started': time.time() - 5}
+        before = time.time()
+        srv._test_finish_entry(entry, {'status': 'ok'})
+        self.assertGreaterEqual(srv._TEST_RUN['heartbeat'], before)
+
+
+class StuckStatusTests(_WatchdogTestCase):
+    def test_a_stuck_member_is_an_error_not_a_stop(self):
+        self.assertEqual(
+            srv._test_member_status([{'status': 'warning'}], True, stuck=True),
+            'error')
+        self.assertEqual(
+            srv._test_member_status([{'status': 'warning'}], True),
+            'stopped')
+
+    def test_a_stuck_run_finalises_as_error(self):
+        script = _warning_script()
+        self._run(stuck=True, heartbeat=time.time(), stop=True,
+                  total=1, name='stuck')
+        srv._test_run_execute(_FakeServer(_RaisingScc()), script, {})
+        self.assertEqual(srv._TEST_RUN['status'], 'error')
+        self.assertFalse(srv._TEST_RUN['running'])
+
+    def test_the_control_warning_run_without_the_stuck_flag(self):
+        # the same raising warning step is only a warning when the watchdog
+        # did not mark the run: the stuck flag is what changes the verdict
+        script = _warning_script()
+        self._run(stuck=False, heartbeat=time.time(), total=1, name='stuck')
+        srv._test_run_execute(_FakeServer(_RaisingScc()), script, {})
+        self.assertEqual(srv._TEST_RUN['status'], 'warning')
+
+    def test_a_blocked_runner_step_ends_the_run_as_error(self):
+        # end to end: a step blocked inside send_apdu is flagged by the
+        # watchdog; when the call returns (the driver's timeout / the abort)
+        # the ordinary failure path finalises the run as an error even though
+        # the step was a warning one
+        entered = threading.Event()
+        release = threading.Event()
+
+        class _BlockingScc:
+            cat_cla = '80'
+
+            def __init__(self):
+                self._tp = self
+
+            def send_apdu(self, apdu):
+                entered.set()
+                release.wait(5)
+                raise RuntimeError('SCARD_E_NOT_TRANSACTED')
+
+        script = _warning_script()
+        self._run(total=1, name='stuck')
+        saved_timeout = srv._TEST_STEP_TIMEOUT
+        srv._TEST_STEP_TIMEOUT = 0.2
+        try:
+            runner = threading.Thread(
+                target=srv._test_run_execute,
+                args=(_FakeServer(_BlockingScc()), script, {}))
+            runner.start()
+            self.assertTrue(entered.wait(2))
+            time.sleep(0.3)
+            with mock.patch.object(srv, '_abort_card_transport') as abort:
+                msg = srv._test_watchdog_check()
+            self.assertIn('stopped responding', msg)
+            abort.assert_called_once()
+            release.set()
+            runner.join(5)
+        finally:
+            srv._TEST_STEP_TIMEOUT = saved_timeout
+        self.assertFalse(runner.is_alive())
+        self.assertTrue(srv._TEST_RUN['stuck'])
+        self.assertEqual(srv._TEST_RUN['status'], 'error')
+        self.assertFalse(srv._TEST_RUN['running'])
+
+    def test_a_stuck_load_iteration_is_a_failure(self):
+        def fake_run(server, script, preset, shared=None):
+            srv._TEST_RUN['steps'].append(
+                {'index': 0, 'status': 'warning', 'label': 's'})
+            srv._TEST_RUN['stuck'] = True
+            srv._TEST_RUN['error'] = ('the card stopped responding '
+                                      '(step 1, no answer for 31 s)')
+            return {}, True
+
+        self._run(load=srv._test_load_state(3), name='load')
+        srv._TEST_RUN['load'].update({'iteration': 1})
+        with mock.patch.object(srv, '_test_run_script', fake_run), \
+                mock.patch.object(srv, '_sim_resume_poll'):
+            srv._test_load_script_worker(_FakeServer(), {'name': 'S',
+                                                         'steps': []}, None,
+                                         's1', 3)
+        load = srv._TEST_RUN['load']
+        self.assertEqual(srv._TEST_RUN['status'], 'error')
+        self.assertEqual(load['failed_iteration'], 1)
+        self.assertEqual(load['done'], 0)
+        self.assertIn('stopped responding', srv._TEST_RUN['error'])
+        self.assertFalse(srv._TEST_RUN['running'])
+
+
+if __name__ == '__main__':
+    unittest.main()