v3.22.8: the text/dcs load-time validation (review follow-ups)

The review of the v3.22.7 fix asked when an expect can raise after its
FETCH: the realistic case is the scripted TERMINAL RESPONSE encoding -
_encode_text raises UnicodeEncodeError when respond.text has a non-GSM
character while dcs is 00 (the default) or 04, i.e. after the command was
fetched and with the card's proactive session left unanswered.  The other
candidates are guarded (the TPDU/PoR/file-list parsers, the PLI dictionary)
or need the card to be gone (a transport failure between the FETCH and the
TR).

normalise_respond now runs the same coding the TR builder uses and refuses
the script at load with an actionable message ("use dcs 08 (UCS2) for
non-GSM characters"), so the failure happens at save/run start, before any
FETCH; UCS2, ASCII and the empty answer are unaffected.

Also from the review: the guide's run-log line now documents the v3.22.7
format (TEST-RUN step N failed (<level>): ...), and the "unguarded item
text decode" observation proved a non-issue - _decode_stk_text is internally
tolerant (odd UCS2 / unknown prefixes fall back without raising), so the
item parsers are left alone.

Tests: a non-GSM text with the default dcs and with 04 is refused (the
message names dcs 08), UCS2/ASCII/empty pass, and a whole script is refused
with the same message.  843 frontend + 822 Python; sw cache simple-v412.

docs/test-scripts.md

@@ -207,7 +207,8 @@ TEST-RUN step 4: MENU-SELECT ENVELOPE=80c2000009d30702020181900101 item=1 ('One'
 - A `proactive-drain` action logs its rounds (`STATUS i/N -> …`, `FETCH=…`,
   `CMD 0x…`, `TR=… -> SW=…`); the run cleanup logs
   `TEST-RUN drain: <CMD> (0xNN) TR=… -> SW=…`.
-- A failing step adds `TEST-RUN step N failed: …` with the reason.
+- A failing step adds `TEST-RUN step N failed (<level>): …` with the reason
+  (`<level>` is `error` or `warning` - a tolerant step's line names its level).
 - The shared secured-packet sender logs each segment's answer
   (`OTA SEND: ENVELOPE i/N -> <SW>`).
 - `--apdu-trace` adds the raw transport view
@@ -457,7 +458,10 @@ identities, **Result**, Duration, Text string, Item identifier, …):
   uses the spec's recommended codings: `"00"` GSM default alphabet 7 bits
   **packed**, `"04"` GSM default alphabet 8 bits (one octet per character),
   `"08"` **UCS2** (UTF-16-BE).  An empty `text` sends the null text string
-  (Length `00`).
+  (Length `00`).  A text the selected `dcs` cannot code (a non-GSM character
+  with `00`/`04`) is **refused at load** - the error names the coding and
+  points at `dcs: "08"` (the run must not fail after the command was
+  fetched).
 - `item_id` - the Item identifier for a successful SELECT ITEM answer.
 - `raw` - extra COMPREHENSION-TLVs appended after the standard objects.
 

frontend/index.html

@@ -1903,7 +1903,7 @@
 // ===== Version =====
 // Single source of truth for the PWA version: shown in the header and used
 // by the server version check in pysimConnect().
-const SIMPLE_VERSION = '3.22.7';
+const SIMPLE_VERSION = '3.22.8';
 document.getElementById('app-version').textContent = 'v' + SIMPLE_VERSION;
 
 // ===== view history (the browser Back/Forward buttons) =====

frontend/sw.js

@@ -1,4 +1,4 @@
-const CACHE = 'simple-v411';
+const CACHE = 'simple-v412';
 const URLS = [
   'index.html',
   'help.html',

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pysim-simple-server"
-version = "3.22.7"
+version = "3.22.8"
 description = "HTTP REST server wrapping pysim for the SIMple PWA"
 requires-python = ">=3.8"
 # pysim is a git-only dependency installed explicitly by setup.bat/setup.sh.

pysim_simple_server/server.py

@@ -36,7 +36,7 @@ from osmocom.tlv import BER_TLV_IE
 from osmocom.utils import rpad
 
 
-VERSION = '3.22.7'
+VERSION = '3.22.8'
 
 MAX_ENVELOPE_SEGMENTS = 5  # max SMS segments for outgoing C-APDU in ENVELOPE
 

pysim_simple_server/testscript.py

@@ -568,6 +568,16 @@ def normalise_respond(respond, cmd_type=None):
         out['text'] = str(respond['text'])
         dcs = respond.get('dcs')
         out['dcs'] = _int(dcs, 'respond dcs', 0, 255) if dcs is not None else 0x00
+        # The same coding the TERMINAL RESPONSE builder uses: a text the
+        # selected dcs cannot code (a non-GSM character with 00/04) must fail
+        # the script at load, not mid-run after the command was fetched
+        # (v3.22.8 review).
+        try:
+            _encode_text(out['text'], out['dcs'])
+        except Exception:
+            raise ScriptError(
+                'respond: the text cannot be coded with dcs %02X - use dcs 08 '
+                '(UCS2) for non-GSM characters' % out['dcs'])
     extra = respond.get('raw')
     if extra not in (None, ''):
         out['raw'] = _data_hex(extra, 'respond raw')

tests/test_testscript.py

@@ -179,6 +179,27 @@ class TestValidation(unittest.TestCase):
         self.assertEqual(tr.hex().upper(),
                          '8103042300' + '82028182' + '83020000' + '8D00')
 
+    def test_a_text_the_dcs_cannot_code_is_refused_at_load(self):
+        # v3.22.8: the load-time check uses the same coding the TERMINAL
+        # RESPONSE builder does - a non-GSM character with dcs 00/04 must
+        # fail the script at load, not mid-run after the command was fetched.
+        with self.assertRaises(T.ScriptError) as cm:
+            T.normalise_respond({'result': 'ok', 'text': 'Пароль'})
+        self.assertIn('dcs 00', str(cm.exception))
+        self.assertIn('dcs 08', str(cm.exception))
+        with self.assertRaises(T.ScriptError):
+            T.normalise_respond({'result': 'ok', 'text': 'Пароль', 'dcs': '04'})
+        # UCS2 accepts it; ASCII and the empty answer pass as before
+        self.assertEqual(T.normalise_respond({'text': 'Пароль', 'dcs': '08'})['dcs'], 0x08)
+        self.assertEqual(T.normalise_respond({'text': 'hello'})['text'], 'hello')
+        self.assertEqual(T.normalise_respond({'text': ''})['text'], '')
+        # a whole script is refused with the same message at run start
+        with self.assertRaises(T.ScriptError) as cm:
+            T.normalise_script({'steps': [
+                {'type': 'expect', 'command': 'GET INPUT',
+                 'respond': {'result': 'ok', 'text': 'Пароль'}}]}, _resolver)
+        self.assertIn('cannot be coded', str(cm.exception))
+
     def test_menu_select_params_accept_id_or_text(self):
         script = T.normalise_script({'steps': [
             {'type': 'action', 'kind': 'menu-select', 'params': {'item_id': 3}},