v3.22.2: the runner's TERMINAL RESPONSE inherits the terminal's own objects

Two reported gaps:

- a scripted PROVIDE LOCAL INFORMATION expectation answered with an empty
  TR - the PLI dictionary (and the host clock for the date/time qualifier)
  was silently inert during runs while the poll path used it all along, so
  an applet waiting for its data stalled behind a passing expectation.  A
  POLL INTERVAL expectation likewise never echoed the Duration (6.8.4).
  testscript.build_tr() gained `answer_tlvs` (inserted after the Result per
  the 6.8.0 order, before text/item/raw) and the shared
  _test_terminal_response() wires the expect step, the proactive-drain
  action and the run cleanup to it; the script's `raw` overrides, and a
  non-success result gets no data object.
- a GP INSTALL/LOAD/DELETE step always failed the default exact-9000 check
  although the envelope SW 91XX is the successful answer (the re-announced
  SET UP MENU; XX is the pending command's length, never pinnable).
  Documented rather than loosening the default: a new "Driving a RAM install
  from a script" subsection in docs/test-scripts.md (the 9??? mask + por
  any, the per-step pending rule - a proactive-drain/expect after every
  91XX step), a sentence at the sw-default bullet, and an scp80
  default-pinning test.

Docs: api.md's respond clause, help EN/RU's TR Config note.
Tests: 839 frontend + 812 Python; sw cache simple-v406.

docs/api.md

@@ -772,7 +772,10 @@ REFRESH, TS 102 223 8.18; `files` is a list of path hex strings, matched
 order-insensitively); `respond` is the TERMINAL RESPONSE in the TS 102 223
 6.8.0 object order (`result` name or value; `text`+`dcs` for GET INKEY/GET
 INPUT, `dcs` `00` packed / `04` 8-bit / `08` UCS2; `item_id`; extra `raw`
-TLVs appended after the standard objects).
+TLVs appended after the standard objects).  The terminal's own objects ride
+along unless `raw` overrides them: a successful PROVIDE LOCAL INFORMATION
+answer carries the TR Config PLI dictionary entry (or the host clock for the
+date/time qualifier) and a POLL INTERVAL answer the current-interval echo.
 
 The response is the initial state (`running: true`), the final counter
 (`scp80_counter`) and the step list; poll `/api/test/status`.  The **server

docs/test-scripts.md

@@ -352,7 +352,10 @@ next step.
 ```
 
 - `sw` - default `{"mode": "exact", "value": "9000"}` (a `status` action with
-  `attempts > 1` defaults to the `91??` mask).
+  `attempts > 1` defaults to the `91??` mask).  A GP INSTALL/LOAD/DELETE
+  envelope answers `91XX` - the card announces a proactive command - so give
+  those steps the `9???` mask and a `proactive-drain` after them (see
+  *Driving a RAM install from a script*).
 - `data` - the response data (for SCP80: the inline PoR packet).  The
   `proactive-drain` action does not take a `data` check - assert the drained
   commands with `require` and the final status with `sw`.
@@ -445,6 +448,13 @@ identities, **Result**, Duration, Text string, Item identifier, …):
 - `item_id` - the Item identifier for a successful SELECT ITEM answer.
 - `raw` - extra COMPREHENSION-TLVs appended after the standard objects.
 
+The terminal's own command-specific objects ride with the scripted result
+unless `raw` overrides them: a successful PROVIDE LOCAL INFORMATION answer
+takes the TR Config **PLI dictionary** entry for its qualifier (the current
+date and time for the date/time qualifier `03`), and a POLL INTERVAL answer
+echoes the current interval (TS 102 223 §6.8.4) - the same objects the
+terminal's own answers carry.
+
 UCS2 answer to a GET INPUT:
 
 ```json
@@ -544,6 +554,51 @@ For this mode the terminal must **not** reset the card
 (TS 102 223 §6.4.7), so the run continues - the following `file-read`
 verifies the updated content.
 
+### Driving a RAM install from a script
+
+The RAM APDU list (`POST /api/scp81/gen-install`, or the PWA's *Install from
+.cap*) is a sequence of GP commands - INSTALL [for load], the LOAD blocks,
+INSTALL [for install], plus a DELETE first when the package is already
+loaded.  Drive it from a test script as one `scp80` step per APDU, addressed
+to the preset's **ISD TAR** (`000000`; the preset supplies the keys, the
+counter and the SPI1 from that TAR's MSL):
+
+```json
+{"name": "install applet",
+ "steps": [
+  {"type": "action", "kind": "scp80",
+   "params": {"apdu": "<INSTALL [for load] C-APDU>", "tar": "000000"},
+   "check": {"sw": {"mode": "mask", "value": "9???"}, "por": "any"},
+   "label": "INSTALL [for load]"},
+  {"type": "action", "kind": "scp80",
+   "params": {"apdu": "<LOAD 1 C-APDU>", "tar": "000000"},
+   "check": {"sw": {"mode": "mask", "value": "9???"}, "por": "any"},
+   "label": "LOAD 1"},
+  {"type": "action", "kind": "scp80",
+   "params": {"apdu": "<INSTALL [for install] C-APDU>", "tar": "000000"},
+   "check": {"sw": {"mode": "mask", "value": "9???"}, "por": "any"},
+   "label": "INSTALL [for install]"},
+  {"type": "action", "kind": "proactive-drain", "params": {}}
+ ]}
+```
+
+A successful GP step answers **`91XX`**, not `9000`: the card has a pending
+proactive command (the re-announced SET UP MENU).  `XX` is that command's TLV
+length - `915d` for the 93-byte SET UP MENU, `910b` for an 11-byte PROVIDE
+LOCAL INFORMATION, `91b1` for a 177-byte SEND SHORT MESSAGE - so it can never
+be pinned exactly.  The default exact-`9000` check therefore fails a step
+whose side effect succeeded; give every INSTALL/LOAD/DELETE step the `9???`
+mask.  `por` stays `any` by default - assert the remote command's own status
+word with `{"por": {"status": "por_ok", "sw": "…"}}` when you want the
+stronger check.
+
+The runner refuses to start an action while a command is pending: after
+**any** step that answered `91XX` the next step must be an `expect` or a
+`proactive-drain`.  The card re-announces the menu after the delete and after
+the install, so a drain after those two groups is enough; a drain after any
+`91XX` step is always safe.  The drain fetches the SET UP MENU, so the cached
+menu (text-based `menu-select`) is current afterwards.
+
 ### Incoming data → particular PoR, no other actions
 
 Inline PoR:

frontend/help-ru.html

@@ -530,7 +530,7 @@
 		<li><strong>06</strong> Access Technology &middot; <strong>08</strong> IMEISV &middot; <strong>09</strong> Search Mode</li>
 		<li><strong>0A</strong> Battery &middot; <strong>0E</strong> Multiple Access Technologies</li>
 	</ul>
-	<p class="text-sm mb-3">Значения хранятся на сервере до перезапуска. Когда карта выдаёт PLI, сервер вставляет значения словаря в TERMINAL RESPONSE.</p>
+	<p class="text-sm mb-3">Значения хранятся на сервере до перезапуска. Когда карта выдаёт PLI, сервер вставляет значения словаря в TERMINAL RESPONSE — в том числе в ответ тест-скрипта.</p>
 
 	<h3 id="network-simulation" class="text-lg font-medium mb-2">8.7 Симуляция сети</h3>
 	<p class="text-sm mb-2">Воспроизводит шаблоны записи реального телефона при смене сетевых условий (исследование трасс в <code class="font-mono text-sm">projects/UICC_NAA.md</code>): подключение EPS, потеря сервиса / ограниченный сервис, запрет роуминга, откат на 2G, приём SMS, перенастройка cell broadcast и AUTHENTICATE. По кнопке на сценарий; карта должна быть подключена.</p>

frontend/help.html

@@ -529,7 +529,7 @@
 		<li><strong>06</strong> Access Technology &middot; <strong>08</strong> IMEISV &middot; <strong>09</strong> Search Mode</li>
 		<li><strong>0A</strong> Battery &middot; <strong>0E</strong> Multiple Access Technologies</li>
 	</ul>
-	<p class="text-sm mb-3">Values persist server-side until restart. When the card issues PLI, the server injects the dictionary values into the TERMINAL RESPONSE.</p>
+	<p class="text-sm mb-3">Values persist server-side until restart. When the card issues PLI, the server injects the dictionary values into the TERMINAL RESPONSE - also into a test-script expectation's answer.</p>
 
 	<h3 id="network-simulation" class="text-lg font-medium mb-2">8.7 Network simulation</h3>
 	<p class="text-sm mb-2">Replays the write patterns a real phone performs when the network condition changes (trace study in <code class="font-mono text-sm">projects/UICC_NAA.md</code>): EPS attach, service loss / limited service, roaming denial, 2G fallback, SMS delivery, cell-broadcast reconfiguration and AUTHENTICATE. One button per scenario; the card must be equipped.</p>

frontend/index.html

@@ -1903,7 +1903,7 @@
 // ===== Version =====
 // Single source of truth for the PWA version: shown in the header and used
 // by the server version check in pysimConnect().
-const SIMPLE_VERSION = '3.22.1';
+const SIMPLE_VERSION = '3.22.2';
 document.getElementById('app-version').textContent = 'v' + SIMPLE_VERSION;
 
 // ===== Tab switching =====

frontend/sw.js

@@ -1,4 +1,4 @@
-const CACHE = 'simple-v405';
+const CACHE = 'simple-v406';
 const URLS = [
   'index.html',
   'help.html',

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pysim-simple-server"
-version = "3.22.1"
+version = "3.22.2"
 description = "HTTP REST server wrapping pysim for the SIMple PWA"
 requires-python = ">=3.8"
 # pysim is a git-only dependency installed explicitly by setup.bat/setup.sh.

pysim_simple_server/server.py

@@ -36,7 +36,7 @@ from osmocom.tlv import BER_TLV_IE
 from osmocom.utils import rpad
 
 
-VERSION = '3.22.1'
+VERSION = '3.22.2'
 
 MAX_ENVELOPE_SEGMENTS = 5  # max SMS segments for outgoing C-APDU in ENVELOPE
 
@@ -6322,6 +6322,26 @@ def _drain_require_label(req):
     return name + ((' q=%s' % q['value']) if q else '')
 
 
+def _test_terminal_response(cmd_num, cmd_type, dev_dst, dev_src, cmd_qual, respond):
+    """The runner's TERMINAL RESPONSE: the scripted answer plus the terminal's
+    own command-specific object(s) - the same objects the terminal's answers
+    carry (`_build_tr`), unless the script supplied `raw` TLVs, which win.
+    A successful PROVIDE LOCAL INFORMATION answer takes the TR Config PLI
+    dictionary entry for its qualifier (or the host clock for the date/time
+    qualifier, TS 102 223 6.4.15); a POLL INTERVAL answer echoes the current
+    interval (6.8.4).  Without this a scripted expectation silently stalled an
+    applet that waited for its data (v3.22.2 review)."""
+    respond = respond or {}
+    answer = None
+    if not respond.get('raw') and respond.get('result', 0) == 0x00:
+        if cmd_type == 0x26 and cmd_qual is not None:
+            answer = _pli_data_hex(cmd_qual)
+        elif cmd_type == 0x03:
+            answer = '840201%02X' % _POLL_INTERVAL
+    return testscript.build_tr(cmd_num, cmd_type, dev_dst, dev_src, respond,
+                               answer_tlvs=answer or None)
+
+
 def _test_run_drain(server, step):
     """The proactive drain action: poll STATUS and consume every announced
     command - the TERMINAL RESPONSE is `first` for the first one, `respond`
@@ -6358,7 +6378,8 @@ def _test_run_drain(server, step):
             if menu:
                 server.sim_menu = menu
         respond = p['first'] if (not consumed and p.get('first')) else p['respond']
-        tr = testscript.build_tr(cmd_num, cmd_type, dev_dst, dev_src, respond)
+        tr = _test_terminal_response(cmd_num, cmd_type, dev_dst, dev_src,
+                                     cmd_qual, respond)
         tr_rv = scc._tp.send_apdu('%s140000%02x%s' % (scc.cat_cla, len(tr), tr.hex()))
         _test_log('TR=%s -> SW=%s' % (tr.hex().upper(), tr_rv[1]))
         consumed.append({'type_hex': '%02X' % cmd_type, 'type_name': name,
@@ -6909,7 +6930,8 @@ def _test_run_expect(server, step, pending, ctx=None):
             checks.append(_test_check_result(
                 'Files', sorted(paths) == sorted(want), ', '.join(want),
                 ', '.join(paths) or '(none)', c['on_fail']))
-    tr = testscript.build_tr(cmd_num, cmd_type, dev_dst, dev_src, step['respond'])
+    tr = _test_terminal_response(cmd_num, cmd_type, dev_dst, dev_src, cmd_qual,
+                                 step['respond'])
     tr_rv = scc._tp.send_apdu('%s140000%02x%s' % (scc.cat_cla, len(tr), tr.hex()))
     tr_sw = tr_rv[1]
     _test_log('TR=%s -> SW=%s' % (tr.hex().upper(), tr_sw))
@@ -6948,7 +6970,8 @@ def _test_drain_pending(scc, fetch_len, result=0x00, max_commands=3):
                 break
             raw = bytes.fromhex(fdata)
             cmd_num, cmd_type, dev_src, dev_dst, cmd_qual = _parse_proactive_header(raw)
-            tr = testscript.build_tr(cmd_num, cmd_type, dev_dst, dev_src, {'result': result})
+            tr = _test_terminal_response(cmd_num, cmd_type, dev_dst, dev_src,
+                                         cmd_qual, {'result': result})
             rv = scc._tp.send_apdu('%s140000%02x%s' % (scc.cat_cla, len(tr), tr.hex()))
             name = PROACTIVE_TYPE_NAMES.get(cmd_type, '?')
             answered.append({'name': name, 'type': cmd_type,

pysim_simple_server/testscript.py

@@ -606,16 +606,20 @@ def _encode_text(text, dcs):
     return text.encode('gsm03.38')
 
 
-def build_tr(cmd_num, cmd_type, dev_dst, dev_src, respond):
+def build_tr(cmd_num, cmd_type, dev_dst, dev_src, respond, answer_tlvs=None):
     """Flat COMPREHENSION-TLV TERMINAL RESPONSE payload (TS 102 223 6.8) in the
     6.8.0 object order: command details, device identities, Result, then the
-    command-specific objects (Text string for GET INKEY/GET INPUT, Item
+    command-specific objects (the terminal's own `answer_tlvs` - the PLI data
-    identifier for SELECT ITEM) and any extra raw TLVs.  Matches the
+    of PROVIDE LOCAL INFORMATION, the echoed Duration of POLL INTERVAL - the
-    interactive menu TR layout."""
+    Text string for GET INKEY/GET INPUT, the Item identifier for SELECT ITEM)
+    and any extra raw TLVs.  Matches the interactive menu TR layout; the
+    runner passes the terminal objects the script did not override."""
     out = bytearray([0x81, 0x03, cmd_num & 0xFF, cmd_type & 0xFF, 0x00])
     out += bytes([0x82, 0x02, dev_dst & 0xFF, dev_src & 0xFF])
     result = int(respond.get('result', 0))
     out += bytes([0x83, 0x02, result & 0xFF, 0x00])
+    if answer_tlvs:
+        out += bytes.fromhex(answer_tlvs)
     if respond.get('text') is not None:
         text = str(respond['text'])
         if text == '':

tests/test_testscript.py

@@ -56,6 +56,17 @@ class TestValidation(unittest.TestCase):
         ]}, _resolver)
         self.assertEqual(script['steps'][0]['check']['sw'], {'mode': 'exact', 'value': '9000'})
 
+    def test_scp80_defaults_to_an_exact_9000_sw(self):
+        # The default must not drift to a mask: a GP INSTALL/LOAD/DELETE
+        # envelope answers 91XX and the script gives those steps the 9???
+        # mask explicitly (the docs recipe); applet scripts keep asserting
+        # exact 9000.
+        script = T.normalise_script({'steps': [
+            {'type': 'action', 'kind': 'scp80', 'params': {'apdu': '80E2900000'}},
+        ]}, _resolver)
+        self.assertEqual(script['steps'][0]['check']['sw'],
+                         {'mode': 'exact', 'value': '9000'})
+
     def test_rejects_bad_scripts(self):
         bad = [
             'not a dict',
@@ -405,6 +416,26 @@ class TestBuildTr(unittest.TestCase):
         tr = T.build_tr(2, 0x21, 0x82, 0x81, {'result': 0x00})
         self.assertEqual(tr.hex().upper(), '81030221008202828183020000')
 
+    def test_the_terminal_answer_objects_follow_the_result(self):
+        # The runner passes the terminal's own objects (the PLI data, the
+        # POLL INTERVAL echo) - they sit right after the Result, before the
+        # text/item/raw objects (TS 102 223 6.8.0).
+        tr = T.build_tr(1, 0x26, 0x82, 0x81, {'result': 0x00},
+                        answer_tlvs='2607AABBCCDDEEFF')
+        self.assertEqual(tr.hex().upper(),
+                         '810301260082028281830200002607AABBCCDDEEFF')
+        # with a text object the answer object keeps its place
+        tr = T.build_tr(1, 0x23, 0x82, 0x81,
+                        {'result': 0x00, 'text': 'hi', 'dcs': 0x04},
+                        answer_tlvs='8402011E')
+        self.assertEqual(tr.hex().upper(),
+                         '810301230082028281830200008402011E8D03046869')
+        # and the script's raw TLVs stay last
+        tr = T.build_tr(1, 0x26, 0x82, 0x81,
+                        {'result': 0x00, 'raw': '9B0100'}, answer_tlvs='2607AA')
+        self.assertEqual(tr.hex().upper(),
+                         '810301260082028281830200002607AA9B0100')
+
 
 # ─── runner with a fake card ─────────────────────────────────────────────
 
@@ -755,6 +786,98 @@ class TestRunnerProactiveDrain(RunnerTestCase):
         self.assertFalse(run['steps'][0]['checks'][0]['ok'])
 
 
+class TestRunnerTerminalAnswers(RunnerTestCase):
+    """The runner's TERMINAL RESPONSE inherits the terminal's own objects
+    (v3.22.2 review): the PLI dictionary / host clock and the POLL INTERVAL
+    echo - unless the script's `raw` overrides them."""
+
+    PLI_01 = 'D009810301260182028281'      # PROVIDE LOCAL INFORMATION, qual 01
+    PLI_03 = 'D009810301260382028281'      # qual 03: date/time
+    POLL = 'D009810301030082028281'        # POLL INTERVAL
+
+    def _tr(self, scc):
+        return [a for a in scc.sent if a.startswith('8014')][-1]
+
+    def _pending(self, scc, cmd):
+        scc.push('80F2', '', '9109')          # STATUS: the command is pending
+        scc.push('8012', cmd, '9000')
+        scc.push('8014', '', '9000')
+
+    def test_a_pli_expectation_carries_the_dictionary_entry(self):
+        scc = FakeScc()
+        self._pending(scc, self.PLI_01)
+        with mock.patch.dict(S._PLI_DATA, {1: '140853937640650700F4'}):
+            run = self.run_script(FakeServer(scc), [
+                {'type': 'action', 'kind': 'status', 'params': {'attempts': 2}},
+                {'type': 'expect', 'command': 'PROVIDE LOCAL INFORMATION',
+                 'respond': {'result': 'ok'}},
+            ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        self.assertIn('140853937640650700F4', self._tr(scc))
+
+    def test_a_pli_raw_overrides_the_dictionary(self):
+        scc = FakeScc()
+        self._pending(scc, self.PLI_01)
+        with mock.patch.dict(S._PLI_DATA, {1: '140853937640650700F4'}):
+            run = self.run_script(FakeServer(scc), [
+                {'type': 'action', 'kind': 'status', 'params': {'attempts': 2}},
+                {'type': 'expect', 'command': 'PROVIDE LOCAL INFORMATION',
+                 'respond': {'result': 'ok', 'raw': '2607AABBCCDDEEFF'}},
+            ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        tr = self._tr(scc)
+        self.assertIn('2607AABBCCDDEEFF', tr)
+        self.assertNotIn('140853937640650700F4', tr)
+
+    def test_a_pli_date_time_answer_uses_the_host_clock(self):
+        scc = FakeScc()
+        self._pending(scc, self.PLI_03)
+        run = self.run_script(FakeServer(scc), [
+            {'type': 'action', 'kind': 'status', 'params': {'attempts': 2}},
+            {'type': 'expect', 'command': 'PROVIDE LOCAL INFORMATION',
+             'respond': {'result': 'ok'}},
+        ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        self.assertRegex(self._tr(scc), r'2607[0-9A-F]{14}')
+
+    def test_a_refused_pli_gets_no_data_object(self):
+        scc = FakeScc()
+        self._pending(scc, self.PLI_01)
+        with mock.patch.dict(S._PLI_DATA, {1: '140853937640650700F4'}):
+            run = self.run_script(FakeServer(scc), [
+                {'type': 'action', 'kind': 'status', 'params': {'attempts': 2}},
+                {'type': 'expect', 'command': 'PROVIDE LOCAL INFORMATION',
+                 'respond': {'result': 'refused'}},
+            ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        self.assertNotIn('140853937640650700F4', self._tr(scc))
+
+    def test_a_poll_interval_expectation_echoes_the_duration(self):
+        scc = FakeScc()
+        self._pending(scc, self.POLL)
+        run = self.run_script(FakeServer(scc), [
+            {'type': 'action', 'kind': 'status', 'params': {'attempts': 2}},
+            {'type': 'expect', 'command': 'POLL INTERVAL',
+             'respond': {'result': 'ok'}},
+        ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        self.assertRegex(self._tr(scc), r'840201[0-9A-F]{2}')
+
+    def test_a_drained_pli_carries_the_dictionary_entry(self):
+        scc = FakeScc()
+        scc.push('80F2', '', '9109')          # STATUS: PLI pending
+        scc.push('8012', self.PLI_01, '9000')
+        scc.push('8014', '', '9000')
+        scc.push('80F2', '', '9000')          # confirming STATUS: idle
+        with mock.patch.dict(S._PLI_DATA, {1: '140853937640650700F4'}):
+            run = self.run_script(FakeServer(scc), [
+                {'type': 'action', 'kind': 'proactive-drain',
+                 'params': {'attempts': 3, 'interval_ms': 0}},
+            ])
+        self.assertEqual(run['status'], 'ok', run['steps'])
+        self.assertIn('140853937640650700F4', self._tr(scc))
+
+
 class TestRunnerActions(RunnerTestCase):
     def test_script_menu_select_mirrors_menu_active(self):
         # 9000 -> the menu dialogue is over