test runner: add the stuck-card watchdog

A card that stops answering (no SW, no driver error) leaves the runner
inside a blocking SCardTransmit while it holds the card lock - Stop
cannot take effect, the presence monitor waits for the lock and a load
test hangs in that iteration without explanation.

The runner now beats a progress heartbeat (each step start/finish and
each attempt of a bounded STATUS/drain loop); after _TEST_STEP_TIMEOUT
(30 s) the watchdog flags the run `stuck`, records the step and the idle
time, sets the stop flag and tries to cancel the blocked transmit by
disconnecting the PC/SC link on a daemon thread.  The disconnect must
run in the background: pcsc-lite serializes it behind the pending
transmit (both take the per-context lock), so a synchronous call would
block the watchdog instead of cancelling anything.  Where a platform
does cancel, the step fails through the ordinary on_fail path; either
way the finalisers map `stuck` to `error` - a stuck card is not a user
stop - and the pending drain is skipped on the dead link.  The
transport is marked stale once the disconnect returns, so the next equip
rebuilds it.

The run status gains `stuck` and `heartbeat`.  Help EN/RU,
docs/test-scripts.md, docs/api.md, the CHANGELOG (folded into the
unreleased 3.37.0) and both AGENTS describe the behaviour and the
background-disconnect caveat; the sw cache is simple-v476 (help is
precached).

Tests: tests/test_test_watchdog.py (17: the check rule, the abort, the
final status of the single/load/suite runs and a blocked-runner
integration case).  1058 Python + 1069 frontend.
This commit is contained in:
catarrh committed 2026-10-24 00:43:39 +03:00
1 parent 2181ef6bbf
commit 643c6609f3
8 files changed
+527 -16

No files matched your search

+16
View File
@@ -25,6 +25,22 @@ every change carries a descriptive commit message.
`done`, `failed_iteration`, `elapsed_ms`, `avg_ms`); the count is a run parameter, not
part of the stored script or suite.
- The **stuck-card watchdog** for test runs. A card that stops answering (no status
word, no driver error) used to leave the runner inside a blocking `SCardTransmit`
holding the card lock — *Stop* could not take effect, the presence monitor waited for
the lock and a load test hung in that iteration without explanation. After 30 s
without runner progress (a bounded STATUS/drain loop beats once per attempt) the
watchdog marks the run `stuck`, records `the card stopped responding (step N, no
answer for N s)`, sets the stop flag and **tries** to cancel the blocked transmit by
disconnecting the PC/SC link on a background thread. Where PC/SC can cancel it, the
step fails through the ordinary `on_fail` path and the run finalises as an **error**
(never `stopped`); where it cannot (pcsc-lite serializes the disconnect behind the
pending transmit), the status already names the step until the driver's own card
timeout returns. The transport is marked stale once the disconnect returns, so the
next *Equip* rebuilds it. The status gains `stuck`; the help EN/RU,
`docs/test-scripts.md` and `docs/api.md` describe the behaviour and the
background-disconnect caveat.
### Fixed
- The load-test loop closes properly on a raised runner error (it used to leave the run