catarrh/sigrok_iso7816_stream
1
0 Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
54dfc6586df9e4ffa9384a9d64d1073e4a758f58
sigrok_iso7816_stream/AGENTS.md
T
catarrh 54dfc6586d v1.9.3 review fixes: CLA doc ranges, README_RUS, mirror/ODD_INS pins 
- pd.py/AGENTS: the rejected '8X' range now reads 0x81-0x83/0x85-0x8F
  (0x84 stays accepted); the 0x40-0x4F vs 0x50-0x5F citations are split
  (chaining is ISO-only, TS 102 221 keeps b5=0); the bit-dense CLA list
  gains 0xE7 (exact set verified).
- README_RUS.md: the invalid-INS note mirrored (the removed odd-LSB rule,
  citation corrected to 5.1.2).
- tests: test_vs_reader_mirror (pd.py <-> tools/vs_reader.py drift guard),
  test_odd_ins_set_pinned, the 0xE7 noise vector, and the 'EX' vector now
  uses STATUS (e2 f2) as TS 102 221 Table 10.5 defines.
- tests/baseline.txt regenerated (test listing only); compare_baseline:
  IDENTICAL.
2026-10-18 00:00:08 +03:00

1210 lines
61 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AGENTS.md
## Policy: decoder changes must be validated against ALL example traces
Every change to `pd.py` must be followed by running the full test suite:
1. **Clear bytecode cache first** - stale `.pyc` files cause false results:
```bash
find . -name "__pycache__" -type d -exec rm -rf {} +
find . -name "*.pyc" -delete 2>/dev/null
```
2. **Run all traces in mid-session mode** (the decoder always runs this way;
captures
all APDUs, including cold-boot ATRs):
```bash
# Run decoder for each trace
for trace in test_8_raw16 test_7_raw16 xiaomi_phone_sample \
samsung_phone_sample \
xiaomi_mi_a1_coldboot_sample 4gmodem_coldboot_sample \
alcor_reader_sample \
samsung_A55_coldboot_new_cable samsung_A55_coldboot_new_cable_sim2 \
samsung_S21p_coldboot_new_cable samsung_S21p_coldboot_new_cable_sim2; do
# Run sigrok-cli and vs_reader as shown below
# Capture APDUs, GARBAGE count, and RESULT
done
```
3. **Acceptance criteria** (must ALL pass):
- **test_8 mid-session**: 56 APDUs, 28/37 reader exchanges (9 pre-capture), 0 GARBAGE, RESULT: OK
- **test_7 mid-session**: 65 APDUs, 37/37 reader exchanges, 0 GARBAGE, RESULT: OK
- **All smoke-test traces**: >0 APDUs in mid-session mode
- **No regressions**: GARBAGE count must not increase, no new BAD_FCS/payload mismatches
- **Unit tests**: 30/30 pass (`python3 tests/test_gsmtap.py -v`)
### Important: CLK signal behavior matters
- **Constant CLK** (test_7, test_8): Card reader keeps CLK running continuously
- Decoder can reliably sample DATA on each CLK edge
- Cold-boot re-arm handles warm resets
- **Gated CLK** (phone traces): Phone stops CLK when idle, restarts before next APDU set
- Decoder must handle CLK gaps and restart properly
- ETU auto-detection from DATA edges is critical
- This is the decoder's primary use case
- **Cold boot captured pre-power-up** (`*_coldboot_sample.sr`): VCC/RST are LOW
at t=0, capture spans the power-on (~17 s at 16 MHz), and the card emits its
ATR once RST deasserts
- Decoder must arm the ATR hunt when RST is still low at capture start
(see "Cold-boot re-arm") and not lock an ETU from pre-ATR power-up traffic
- It must decode the ATR `3b9f96801f...001e`
- `rst_detect=true`, `vcc_detect=true` for these fixtures
```bash
# --- test_8 ---
sigrok-cli -i examples/test_8_raw16.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:vcc=VCC:clock_option=native:rst_detect=true:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
python3 tools/vs_reader.py examples/test_8_reader.log /tmp/out.pcap
# --- test_7 ---
sigrok-cli -i examples/test_7_raw16.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:vcc=VCC:clock_option=native:rst_detect=true:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
python3 tools/vs_reader.py examples/test_7_reader.log /tmp/out.pcap
# --- xiaomi phone (gated CLK) ---
sigrok-cli -i examples/xiaomi_phone_sample.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:clock_option=native:protocol=T=0:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
# --- samsung phone 1 (gated CLK, non-default F/D) ---
sigrok-cli -i examples/samsung_phone_sample.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:vcc=VCC:clock_option=native:protocol=T=0:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
# --- xiaomi_mi_a1_coldboot (starts pre-power-up) ---
# rst_detect/vcc_detect=true are required for these fixtures.
sigrok-cli -i examples/xiaomi_mi_a1_coldboot_sample.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:vcc=VCC:clock_option=native:protocol=T=0:rst_detect=true:vcc_detect=true:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
# --- 4gmodem_coldboot ---
sigrok-cli -i examples/4gmodem_coldboot_sample.sr \
-P iso7816:clk=CLK:data=DATA:rst=RST:vcc=VCC:clock_option=native:protocol=T=0:rst_detect=true:vcc_detect=true:gsmtap_enable=false:pcap_file=/tmp/out.pcap \
-A iso7816 >/dev/null
# --- unit tests ---
python3 tests/test_gsmtap.py -v
```
**Always clear the Python bytecode cache before testing** — stale `.pyc`
files can cause the decoder to use old code, producing false results:
```bash
find . -name "__pycache__" -type d -exec rm -rf {} +
```
The acceptance criteria are:
**The decoder is NOT considered fully functional until ALL of these criteria are met.**
#### Ground truth
`test_7_reader.log` and `test_8_reader.log` are **trusted, complete APDU logs**
captured by the PC/SC reader SDK during the same recording session as the
sigrok trace. They contain every command the reader sent and every response
the card returned. They are the authoritative reference — the decoder output
must match them.
#### Success criteria (all must pass)
1. **test_8 and test_7 (reader-log traces):**
- **Every reader exchange must appear in the decoder output.**
For test_8: 28/37 reader exchanges are present in the pcap
(9 missing because they occurred before the capture started —
a trace-level limitation, not a decoder bug). For test_7: all
37/37 are present after the v1.1.10 deglitch fix (previously
only 10/37 due to ATR resets fragmenting the session).
- **0 garbage APDUs** in mid-session mode.
- **`RESULT: OK`** from `vs_reader.py`.
2. **All traces (including smoke-test):**
- **>0 APDUs** in mid-session mode for smoke-test traces.
- No increase in `CHKSUM ERROR`, `BAD_FCS`, payload mismatches, or
suspicious CLA counts.
3. **Every trace must be tested** (the decoder always runs mid-session — it
captures all APDUs, including cold-boot ATRs).
#### Known issues (must be fixed to meet criteria)
**T=0 framing produces incorrect APDUs.** The decoder's output now matches
the reader log for all reader exchanges:
- test_8: 28/37 reader exchanges are byte-identical in the pcap
(9 missing because they occurred before the capture started),
plus 17 extra CAT/proactive commands and 11 valid retries,
total 56 APDUs, 0 garbage.
- test_7: All 37/37 reader exchanges are byte-identical in the pcap,
plus 17 CAT/proactive commands, 8 ATRs, and 11 valid retries,
total 65 APDUs, 0 garbage.
Both traces show 0 garbage APDUs, 0 payload mismatches, and RESULT: OK.
**Root cause:** `_edge_read` is initialized to `True` and never set to
`False`, so the edge-read T=0 procedure loop (lines 1808–1895) always
executes — the ATR-based procedure loop (lines 1898–1941) is dead code.
The edge-read path had two bugs:
1. **NULLs treated as command data (FIXED).** The first post-header byte
was checked for ACK; if it was NULL (0x60), it entered the case-3/4
branch and got appended as command data. Fixed by skipping NULLs
before the ACK check.
2. **"Read until SW" truncates GET RESPONSE (FIXED).** The ACK handler
read until it saw a byte in 0x6x/0x9x range, but FCP tag 0x62
matched this pattern, truncating GET RESPONSE to 1-2 bytes. Fixed
by using P3-bounded reads (read exactly P3 bytes then SW1 SW2) for
case-2 commands where P3 = Le.
#### CLA validation note
`plausible_cla()` (v1.9.3) accepts the higher logical channels:
- interindustry `0x00`-`0x1F` (channels 0-3 in b2b1) and `0x40`-`0x5F`
(`01xx xxxx`: channels 4-19 in b4..b1, no CLA-level SM; `0x50`-`0x5F`
are the legacy chaining variants);
- the UICC extended classes `'CX'`/`'EX'` (`0xC0`-`0xCF`/`0xE0`-`0xEF`,
channels 4-19, SM in b6 -- TS 102 221 Table 10.4a);
- `0x80`/`0x84`/`0xA0`/`0xB0` (0xB0 has no pinned-spec basis; kept for
backward compatibility).
Rejected: `0x20`-`0x3F` (ISO-reserved); `0x60`-`0x6F`/`0x70`-`0x7F` (the
SM-indicating extended classes -- `0x60`-`0x6F` overlaps printable ASCII
and the text byte `0x61` started a bogus frame on xiaomi_mi_a1_coldboot
when it was accepted); `0x81`-`0x83`/`0x85`-`0x8F`/`0xA1`-`0xAF` (the
`'8X'`/`'AX'` standard-channel variants for channels 0-3 other than the
legacy `0x80`/`0x84`/`0xA0` -- the FCP byte `0xA5` did the
same); `0x90`-`0x9F`, `0xD0`-`0xDF`, `0xF0`-`0xFF`. They are spec-coded
but cost frame-start precision; add any of them back when a real trace
requires it.
During test runs such a CLA is an error flag indicating the decoder
mis-framed the byte stream. The example traces carry `0x00`
(interindustry), `0x01` (logical channel 1 -- the SELECT-by-AID after
MANAGE CHANNEL opens it) and `0x80` (USIM/CAT); no trace uses channels 2-3
or the higher channels yet.
#### ISO 7816 APDU validation (enforced from v1.1.7)
The decoder validates every reassembled T=0 APDU with `validate_t0_apdu()`
before emitting it. Invalid APDUs are still packed into gsmtap but are
flagged with `GSMTAP_FLAG_BAD_FCS`.
Validation is purely structural (no content scanning of command/response
data), so the false-positive rate is very low. The checks are:
1. Length is at least 7 bytes (header + SW) and at most `MAX_TPDU_LEN` (271).
2. CLA byte is a structurally valid ISO 7816-4 / UICC class — see
`plausible_cla()` (v1.9.3): `0x00`-`0x1F`, `0x40`-`0x5F` (logical
channels 4-19), `0xC0`-`0xCF`/`0xE0`-`0xEF` (`'CX'`/`'EX'`), plus
`0x80`/`0x84`/`0xA0`/`0xB0`.
3. INS byte is plausible: not `0x00`, not in `0x6x`/`0x9x` (status-word
collision). Odd INS codes are accepted only as the defined alternate
variants (ISO 7816-4 Table 4 pairs: `'0F'`, `'21'`, `'87'`, `'A1'`,
`'B1'`, `'B3'`, `'C3'`, `'CB'`, `'D1'`, `'D7'`, `'DB'`, `'DD'`;
TS 102 221 AUTHENTICATE `'89'`). Other odd values stay rejected: a
mis-framed boundary exposes data bytes, and the text run `49 4D`
("IM") started a bogus frame on xiaomi_phone_sample when every odd INS
was accepted.
4. The penultimate byte (SW1) is in `0x61`–`0x6F` or `0x90`–`0x9F`.
`0x60` is explicitly invalid: it is the T=0 NULL procedure byte
(ISO 7816-3 §10.3.3 Table 11 marks SW1 as `'6X'` **except `'60'`**;
ISO 7816-4 enforces `'60'` as an invalid SW1), so a "NULL + real SW1"
pair must never be accepted as a status word.
The current validation deliberately does **not** enforce P3-based length
constraints, because a card may reject a command and send the status word
immediately even when P3 > 0.
#### v1.4.0: CONCAT detection (embedded exchanges, issue #1 gate)
A byte-drop mis-frame swallows several real T=0 exchanges into one
structurally "valid" packet: each swallowed header survives verbatim in the
data region, followed by its P3 data bytes and an interior status word.
Such packets pass `validate_t0_apdu()` but produce wrong APDUs downstream.
`embedded_exchanges()` (pd.py) scans the data region (`[5, n-9]`) for
embedded complete exchanges:
- header `CLA INS P1 P2 P3` where `INS ∈ EMBEDDED_INS` (commands that
actually occur on these cards: 12/14/20/44/70/88/A2/A4/B0/B2/B6/C0/C2/
CA/CB/D6/DC/F2) and `P3 ≥ 1`
- CLA valid, or the phantom byte-drop artifact `0x01` (corrupt `0x00`),
with a one-byte prefix slip tolerated when byte `0x01` precedes a real
header
- SW1 accepted in a 3-byte slack window after the nominal data end
(`d1 = s + 5 + p3`), from `EMBEDDED_SW1` (real status bytes that are not
also frequent data bytes), and the SW must be interior (`sw < n-2`, i.e.
more exchange data must follow it)
On a hit the decoder flags the packet `GSMTAP_FLAG_BAD_FCS` and logs
`CONCAT APDU: <k> embedded exchange(s) at <offsets>; packet <hex>`, so the
gate works live and offline. The detector is precision-tuned against all
10 example traces: 0 hits on clean traces (test_8/test_7/4gmodem), and it
flags every known corrupted family:
| Trace | APDUs | CONCAT hits |
|-------|-------|-------------|
| test_8 | 56 | 0 |
| test_7 | 65 | 0 |
| xiaomi_phone | 620 | 4 |
| samsung_phone | 924 | 5 |
| xiaomi_mi_a1_coldboot | 459 | 3 |
| 4gmodem_coldboot | 327 | 0 |
| A55_coldboot_new_cable | 386 | 3 |
| A55_coldboot_new_cable_sim2 | 375 | 2 |
| S21p_coldboot_new_cable | 380 | 3 |
| S21p_coldboot_new_cable_sim2 | 356 | 2 |
The concat packets are BAD_FCS-flagged on the phone traces, so their BAD_FCS
counts now equal the CONCAT counts (deliberate: those APDUs are genuinely
desynced). `vs_reader.py` reports the same detector; `capture_baseline.sh`
prints `CONCAT APDU count: N` per trace (grep of the `CONCAT APDU:` log line
only, so one line per packet). Acceptance: test_8/test_7/4gmodem must stay
0; phone-trace counts must not increase.
### v1.4.0: CONCAT gate — embedded-exchange detection (issue #1)
The decoder previously emitted byte-drop-mis-framed APDUs that swallowed
several real exchanges into one "valid" packet (the Samsung/xiaomi
"wrong APDU" family). Fragmentation is not representable in the
assembled bytes, so this is fixed as a detection gate, not a reframe:
- `embedded_exchanges()` scans the data region for embedded complete
exchanges (see "v1.4.0: CONCAT detection" under APDU validation).
- Hits are flagged `GSMTAP_FLAG_BAD_FCS` + logged `CONCAT APDU: ...`.
- `vs_reader.py` / `capture_baseline.sh` gain a `CONCAT` metric
(packet counts match the log lines exactly).
- Unit tests cover the tuned vectors (clean FCP/READ-RECORD/SELECT never
hit; U1/U4/record-burst/xiaomi swallows are caught).
- VERSION → 1.4.0. Baseline regenerated with the new CONCAT columns.
A proper *reframing* (re-splitting those packets and byte-dropping the
mis-frame at the source) is the next decoder step; until then the gate
makes the corruption visible in live and offline captures instead of
silently producing wrong APDUs.
### v1.6.0: ATR-hunt deadlock + idle-loop trap fixes
A live Samsung cold-boot capture that never emits a 372-rate ATR exposed two
defects in the ATR hunt (`state == 'FIND START'`):
1. **Deadlock.** `RST low at start` arms the hunt at the spec-default
`clock_skip=372` with `bit_samples=None`. The ETU-recovery routine
`_measure_etu()` is only reachable from the `DATA` state, which the hunt
can only enter after reading a valid `TS=0x3B/0x3F` (or a plausible
CLA/INS header) **at 372**. A card already running at a non-default F/D
(Samsung `F=512/D=32` -> 16 CLK/bit) produces none, so the hunt reads
idle `0xFF` forever and never reaches the ETU measurement -- a circular
dependency. This is why Samsung (non-default F/D) fails where
default-F/D devices succeed: not clock rate, but `F/D`.
2. **Trap.** The all-`0xFF` branch unconditionally reset `_atr_hunt_count=0`
and returned to `FIND START`, so the bounded fall-through to `DATA` was
unreachable. With RST re-arm also resetting the counter, the hunt could
spin forever.
Fixes:
- **Hunt-side ETU recovery.** `_recover_etu_in_hunt()` runs `_measure_etu()`
+ `_derive_clock_skip_from_etu()` from the hunt after the default-rate
hunt has already failed a few rounds, so a fast device is read at its
true rate. It reuses the DATA-path `_confirm_etu()` validation, so a
noise-locked ETU self-clears after 2 parity failures. No-op for normal
ATRs (parsed on hunt 1).
- **Bounded escape.** `_atr_hunt_failure_action(byte)` (pure, unit-tested)
returns `recover` / `retry` / `resume_data`. ETU recovery every 2 failed
rounds up to 3 times; then a hard bound (`_atr_fail_total >= 12`) or the
all-`0xFF` idle cap (3) falls through to the existing "resuming DATA
without synthetic ATR" path (clearing `bit_samples` so the DATA path
re-measures).
- **Persistent counters.** `_atr_fail_total`, `_hunt_etu_attempts`,
`_atr_idle_resets` are reset only on a successful ATR parse, **not** on
RST re-arm -- otherwise a phone that power-cycles RST while never sending
a valid ATR would starve the escape. Constants: `ATR_HUNT_ETU_ATTEMPTS=3`,
`ATR_HUNT_IDLE_RESETS=3`, `ATR_HUNT_FAIL_LIMIT=12`.
Verification: no regression on the full trace suite (test_8 56/0/OK,
test_7 65/0/OK, xiaomi 640, xiaomi_cold 473, 4gmodem 327, A55 407/395,
S21p 399/375, target S21p_coldboot 369, all `RESULT: OK`, 0 CONCAT);
54/54 unit tests (7 new hunt-escape regression tests). VERSION -> 1.6.0.
Known limitation: no on-disk fixture reproduces the exact live failure (all
real captures contain a 372 ATR), so end-to-end proof needs a re-captured
`.sr` from the failing phone; the unit tests cover the loop-escape logic.
### v1.9.3: higher logical channels in the CLA gate + defined odd INS codes
A trace with higher logical channels (4-19) or an odd-INS command could not
decode: the acceptance gates knew neither. Both are now spec-derived, with
the empirical precision of the mis-frame filter preserved.
- **Higher logical channels (4-19).** TS 102 221 §10.1.1 Table 10.4a codes
them `'01x0 xxxx'` (`0x40`-`0x4F` without SM, `0x60`-`0x6F` with SM in b6)
and `'11x0 xxxx'` (`0xC0`-`0xCF`/`0xE0`-`0xEF`, the `'CX'`/`'EX'`
classes) -- ISO 7816-4 §5.1.1 Table 3 is the same with b5 = chaining.
`plausible_cla()` now accepts the `'CX'`/`'EX'` classes alongside the
previous `0x40`-`0x5F`, so extended-channel commands (e.g. STATUS on
channel 4 = `0xC0`) and higher channels in the no-SM `'4X'` form decode.
Rejected after testing: `0x60`-`0x6F`/`0x70`-`0x7F` (the SM-indicating
interindustry classes) and the `'8X'`/`'AX'` channel variants
`0x81`-`0x83`/`0x85`-`0x8F`/`0xA1`-`0xAF`. They are spec-coded, but
they cost
frame-start precision on the mis-framed traces: `0x60`-`0x6F` overlaps
printable ASCII, and the menu-text byte `0x61` ('a') started a bogus
frame on xiaomi_mi_a1_coldboot whose P3 swallowed the following exchanges
(a CONCAT regression); the FCP byte `0xA5` did the same for `'AX'`.
Previously such stray bytes were discarded as "suspicious CLA" and the
decoder resynced; the wider classes turned them into headers. Add the
classes back when a real trace needs them.
- **Odd INS.** The gates rejected every odd INS code, so TS 102 221
AUTHENTICATE `'89'` (chunked authentication data) and the ISO 7816-4
Table 4 alternate variants mis-framed. `plausible_ins()` now accepts
exactly the 13 defined odd codes (`ODD_INS`: `0F 21 87 89 A1 B1 B3 C3
CB D1 D7 DB DD`) and still rejects the rest: accepting every odd value
let the text run `49 4D` ("IM") start a bogus frame on
xiaomi_phone_sample. The header check calls the same `plausible_ins()`
now (the duplicate inline condition is gone) and the stale ISO `§12.1.1`
citation was corrected to `§5.1.2`.
- **Idle-noise gate.** The 6+ set-bits discard runs before the CLA check,
so it would swallow the legitimately bit-dense classes (`0xCF`, `0xE7`,
`0xEB`, `0xED`-`0xEF`) and kept `0x5F` dead (`plausible_cla()` accepted
it, the gate always discarded it). The gate is now
`is_idle_noise_byte()`: only bytes no spec allows are idle noise; valid
CLAs take the normal header path.
- **CONCAT detector unchanged by design.** `EMBEDDED_CLA` no longer
derives from `plausible_cla()`; it is pinned to the empirically tuned 36
values (`0x00`-`0x1F` + `0x80`/`0x84`/`0xA0`/`0xB0`) in both `pd.py` and
`tools/vs_reader.py`, with a membership-pin test and a `pd.py` ↔
`vs_reader.py` mirror test -- the CONCAT scan runs
across payload bytes where the extended classes false-hit (the Alcor
`"40 70 70 61 15"` run).
- **Tests.** CLA accept/reject vectors (including the two regressing bytes
`0x61`/`0xA5` and the reserved ranges), extended-channel
`validate_t0_apdu`/`is_desynced` vectors, the odd-INS vectors, the
idle-noise vectors, the EMBEDDED_CLA membership pin, the `ODD_INS` set
pin and the `pd.py` ↔ `vs_reader.py` mirror test (75 -> 84 tests);
`tools/vs_reader.py` mirrors the predicate, `ODD_INS` and the pinned set.
Validation: all 10 traces identical to the v1.9.2 baseline (APDU counts,
CHKSUM/CONCAT/BAD_FCS/suspicious-CLA counts, `RESULT: OK` everywhere);
`compare_baseline.sh` reports IDENTICAL. The xiaomi CONCAT regressions the
wider sets produced were found by this loop and narrowed away before the
baseline was regenerated (version + test listing only). VERSION -> 1.9.3.
### v1.9.2: PPS CLK upshift + exact CLK-edge projection (new reader support)
Two timing bugs surfaced by `examples/alcor_reader_sample.sr`, a PC/SC reader
that changes the CLK frequency at PPS: 4.0 MHz at the ATR, 4.8 MHz after
(raise-to-raise 4.000 -> 3.333 samples at 16 MHz), with FI/DI still 512/32 =
16 CLK/bit. Before the fix the whole post-PPS session framed nothing.
- **Stale samples-per-clock after PPS.** The native/CLK-sync path assumed
"the CLK period is unchanged by PPS" and kept the period measured at the
ATR, so the post-PPS ETU was 16 x 4.0 = 64 samples instead of the real
16 x 3.333 = 53.33 -- every byte was mis-sampled. `handle_pps()` now sets
`_clk_remeasure_pending`; `_read_byte_clk()` consumes it in the first
post-PPS character (CLK is guaranteed to be running inside a frame),
re-runs `_measure_clock_period()` and accounts the measurement's 16 edges
as the first edges of the frame (`bit_samples` refreshed).
- **Edge-skip deficit on fractional periods.** `_wait_clk_rising()` used to
"skip (n - 2) cycles by samples, then take 2 edges". With a period that is
a fraction of a sample the skip under-covers by ~1 edge per call: simulated
against the real trace, 144 edges were consumed where 152 were requested,
so the samples slid ~0.5 ETU early by the parity bit -- in 513/513 CHKSUM
cases the "parity" sample read the last data bit. It now projects n
periods ahead of the edge it is sitting on, skips to (target - half a
period) and takes one edge: exact for any period, non-accumulating, one
wait() per call. Without a period estimate it waits each edge exactly.
- **CONCAT detector precision.** The Alcor GET RESPONSE payload (a
secured-packet byte run "40 70 70 61 15" + an SW-like byte 21 later)
false-hit the embedded-exchange scan through the structural 0x4x CLA that
`plausible_cla()` tolerates for resync robustness. The detector's CLA set
now excludes 0x4x/0x5x (no CAT traffic uses them; every tuned
true-positive vector uses 0x00/0x01/0x08/0x80), mirrored in
`tools/vs_reader.py` with a regression vector in `tests/test_gsmtap.py`.
Effect on the suite (baseline regenerated; APDU counts and `RESULT: OK`
unchanged everywhere, 0 CONCAT):
| trace | CHKSUM lines before | after |
|---|---|---|
| alcor_reader_sample (new fixture) | no framing at all (0 APDUs) | 2 (1 idle artifact), 48 APDUs |
| xiaomi_phone_sample | 7698 | 2 (1 event) |
| 4gmodem_coldboot | 7218 | 0 |
| samsung_S21p_coldboot_new_cable_sim2 | 394 | 0 |
| test_8 / test_7 / others | 0 | 0 |
The xiaomi/4gmodem/S21p_sim2 counts had been read as dirty-wire parity
noise -- they were this decoder bug. `tools/capture_baseline.sh` gains the
alcor fixture (local-only, like every `examples/` trace). 75/75 unit tests.
VERSION -> 1.9.2.
### v1.9.1: T=0 NULL is not an SW1 (fixes bogus "60 61" APDU endings)
The "scan for SW1" loops accepted any byte with high nibble `0x6` as a
status word, including the T=0 **NULL procedure byte `0x60`** (card busy /
waiting-time extension). A card that holds the terminal with NULLs while
busy was therefore mis-framed: on the phone/cold-boot traces the SELECT MF
wire flow is `00 a4 00 04 02` → ACK `a4` → command data `3f 00` → ~1.4 s of
NULL `60` → real SW `61 29`. The decoder took `60` as SW1 and the real
`0x61` as SW2, emitting `…3f00 6061` and leaving the real SW2 (`0x29`) on
the wire, where it was discarded as a suspicious CLA — and the bogus packet
even passed `validate_t0_apdu()` because `plausible_sw()` accepted `0x60`.
- `plausible_sw()` now accepts `0x61`–`0x6F` and `0x90`–`0x9F` only.
Authority: ISO 7816-3:2006 §10.3.3 Table 11 — "If the value is `'6X'`
or `'9X'`, **except for `'60'`**, it is a SW1 byte"; the note adds that
ISO 7816-4 enforces `'60'` as an invalid SW1. (UICC_SPECS.md §1.8
classes: normal `'9000'`/`'61XX'`; warning `'62XX'`/`'63XX'`; execution
error `'64XX'`–`'66XX'`; checking error `'67XX'`–`'6FXX'`.)
- Every SW1-detection site now uses `plausible_sw()`: the edge ACK scan,
the P3=0 direct-SW check and response scan, the case-4 response scan,
the case-3 SW1 check, the unexpected-byte scan, and the ATR path.
- NULLs met while scanning for the status word are logged as rate-limited
wait extensions via the new `_note_pre_sw()` (cards emit them in bursts,
so they must not flood the log or raise warnings); other non-SW bytes
keep their call-site reason.
- `tools/vs_reader.py::_plausible_sw()` mirrors the fix.
- Tests: `plausible_sw` vectors updated (`0x60` now rejected; accepts
start at `0x61`), plus a regression vector for the
`00 a4 00 04 02 3f 00 60 61` artifact (invalid SW1 / desynced).
The fix is count-neutral (the same bytes are consumed; only the SW bytes
change), so `tests/baseline.txt` differs only by the version line and the
new test listing. 74/74 unit tests. VERSION -> 1.9.1.
### v1.9.0: CLK frequency in RST events
RST line events (sub_type 0x10) now append the measured CLK frequency when
it is known: `[direction, level, flags, clk_hz_be32]`, with `flags` bit 0
(`LINE_EVENT_FLAG_CLK_HZ`) set and the frequency as a big-endian uint32 in
Hz (7-byte payload). An unknown rate keeps the 3-byte payload, so
consumers that read only the first two bytes (e.g. SIMtrace Analyser's
`decode_line_event`) are unaffected. The value is the existing 16-edge
average (`samplerate / _samples_per_clock`, see `_measure_clock_period`);
PPS changes F/D, not the CLK rate, so one measurement per session is
enough. VCC events (0x11) are unchanged. Purpose: consumers can compute
the actual data rate as `clk_hz × D / F` from the Fi/Di negotiated in the
ATR/PPS. Ordering: the RST deassert event usually precedes the ATR event
(the 10 ms hysteresis lands mid-ATR for full ATRs), but a short ATR can
invert that, so consumers take the rate from any RST event. Wire
extension is backward compatible; 73/73 unit tests; baseline regenerated
and IDENTICAL (it tracks no RST payload metrics). VERSION -> 1.9.0.
### v1.8.0: PPS emitted as standard request + response packets
The PPS exchange was previously emitted as a single packet under sub_type
0x02 (`GSMTAP_SIM_PPS_REQ`) carrying the request + response bytes
concatenated — a deviation from libosmocore's gsmtap.h. It is now split
into the two standard subtypes: 0x02 (`PPS_REQ`, raw request bytes, span
`ss..es_req`) and 0x03 (`PPS_RSP`, raw response bytes, span
`es_req..end`). Wireshark now dissects both halves, and SIMtrace Analyser
needs no change: its pcap importer already maps 0x03 to `pps` and
`_decode_pps()` parses any 0xFF-leading frame (the response frame also
starts with 0xFF). The PPS annotation and stderr log lines are unchanged.
`tests/baseline.txt` regenerated — the only differences were the version
line and the unit-test listing, no decode-metric changes. 68/68 unit
tests. VERSION -> 1.8.0.
### v1.7.1: accept EOFError as clean end-of-stream (newer libsigrokdecode)
Newer libsigrokdecode signals end-of-capture from `wait()` by raising
`EOFError("samples exhausted")`; the 0.5.3 build on the Linux dev box instead
returns `None` / raises `SystemError`, so this only showed up on Windows (a
newer libsigrokdecode). The framework explicitly accepts `EOFError` out of
`decode()` as normal termination (`instance.c`: "Termination with an
EOFError exception is accepted"), but our `decode()` caught it in the generic
`except Exception` handler, printing a traceback, logging `DECODE ERROR`, and
resyncing -- retrying up to 50 times and flooding the log (looked like a
crash) at the end of every capture.
- `decode()` and the five `except SystemError: raise` guards now use
`except (SystemError, EOFError): raise`, so the end-of-stream signal
propagates and libsigrokdecode terminates cleanly. The critical guard is
`_measure_clock_period()`, whose `try` wraps a `wait()` loop.
- 2 new unit tests assert `decode()` re-raises `EOFError` and `SystemError`.
- 67/67 unit tests. VERSION -> 1.7.1.
Deployment note: the Windows decoder is a git clone, so `git pull` there
(and deleting `__pycache__`) picks this up; the EOF error cannot be
reproduced against Linux 0.5.3.
### v1.7.0: spurious end-of-decode hardening + start.sh session resilience
A live capture could end without Ctrl+C (the session simply stopped). The
frontend symptom was a flood of `sr: session: sr_session_stop: session was
NULL`, but the decoder also had latent paths that could end `decode()` early
on an idle/busy live line. Fixed:
1. **`_at_eof()`.** `decode_step()` treated `samplenum == _last_sn` as EOF,
but a step can legitimately consume only queued bytes (`_replay` /
`peeked_byte`) without advancing the stream -- e.g. the v1.5.0 invalid-INS
re-frame. Such steps are now allowed up to 64 in a row; only a genuinely
stuck stream with nothing queued ends the decode.
2. **`_stall_check(track=False)`.** The idle-LEVEL wait in
`wait_data_falling()` returns immediately at the current sample whenever
DATA is already idle -- normal on a live idle line, not EOF. Stall
counting is now disabled for that wait (only `pins is None` means EOF);
sample-stall detection is kept for the start-bit EDGE wait.
3. **`resync_idle()` null guard.** It indexed `pins[self.DATA_IDX]` after
`wait()` with no `None` check; at stream end that raised `TypeError`. Now
sets `_eof` and returns.
**Effect:** `samsung_phone_sample.sr` had been halting at sample ~21.9M of
320M (1 APDU). It now decodes through sample ~293M to a true end: **939
APDUs**, `CHKSUM=176`. The extra 6 `CONCAT` packets are real mis-framed STK
bursts that were simply never reached before -- the known Samsung
de-concatenation family; the rest of the suite is unchanged.
**start.sh:** auto-restart is now the **default** (`--no-loop` disables), so
an intermittent FX2 USB stop reconnects instead of ending the session;
per-session pcap suffix (`.N.pcap`) so a restart never overwrites the
previous capture; and the repetitive `sr_session_stop: session was NULL`
teardown line is filtered from stderr (real decoder errors still pass).
Validation: test_8 56/0/OK, test_7 65/0/OK, xiaomi_phone 640, xiaomi_cold
473, 4gmodem 327, A55 407/395, S21p 399/375, target S21p 369 -- all
`RESULT: OK`, 0 CONCAT. 65/65 unit tests (10 new). VERSION -> 1.7.0.
### v1.5.0: invalid-INS one-byte re-frame (residual CONCATs → 0)
The last CONCAT family (4gmodem/A55_sim2/S21p_sim2) is now **reframed at
the source**, not just gated. All 10 traces report 0 CONCAT:
| Trace | APDUs | CONCAT hits |
|-------|-------|-------------|
| test_8 | 56 | 0 |
| test_7 | 65 | 0 |
| xiaomi_phone | 640 | 0 |
| samsung_phone | 924 | 0 |
| xiaomi_mi_a1_coldboot | 473 | 0 |
| 4gmodem_coldboot | 327 | 0 |
| A55_coldboot_new_cable | 407 | 0 |
| A55_coldboot_new_cable_sim2 | 395 | 0 |
| S21p_coldboot_new_cable | 399 | 0 |
| S21p_coldboot_new_cable_sim2 | 375 | 0 |
1. **One-byte re-frame on invalid INS.** When a 5-byte header window passes
`plausible_cla()` but the INS is unusable (`0x00`, `0x6x`/`0x9x`, or an
odd code outside the defined variants -- see the v1.9.3 note), the
window is almost always shifted by one
stray leading byte (e.g. a leftover SW2 read after a PACKETEND). The old
code silently dropped all 5 bytes plus resync (`_resync = True`), and when
`resync_idle()` gave up on a busy burst the next byte became a bogus CLA
whose P3 swallowed the whole dialog into one CONCAT packet. Now the
decoder drops **only** the leading byte (explicitly logged via
`_note_discard`), re-frames the remaining four via `_replay`, and lets the
next decode step start a fresh header at the old INS. Example (4gmodem):
the stray `0f` after the READ BINARY SW1 `91` re-frames `0f 00 a4 08 04`
→ `00 a4 08 04 02`, recovering the true SELECT `00 a4 08 04 02 2f 00 →
61 1c` verbatim.
2. **No-silent-discard invariant.** Every byte the decoder consumes without
emitting is now explicitly logged via `_note_discard(value, reason,
idle=...)`. `idle=True` discards (0x00/0xFF idle stretches, NULLs, stuck
low) are rate-limited to the 1st + every 256th; `idle=False` discards
(suspicious CLA, invalid-INS re-frame byte, non-SW bytes in read-until-SW
scans) always put an `ANN_WARN` and the reason. Idle-line noise,
suspicious-CLA, stuck-low, and idle-byte-in-header consumers all go
through `_consume_first()` (which pops `_replay` before `peeked_byte`) so
re-framed bytes cannot infinite-loop.
3. **`_consume_first()` semantics.** `read_byte()` pops `_replay.popleft()`
first and only falls through to the wire; `peek_byte()` returns
`_replay[0]` first. Neither updates `peeked_samplenum`, so the re-frame
branch explicitly restores `self.peeked_samplenum = es` (the first byte's
sample) before pushing the 4 remaining bytes — the next header annotates
from a valid sample number.
4. **Counters.** `reset()` initializes `_discard_total` / `_idle_discard_total`; every `_note_discard` line carries "discards so far: N" for session-level triage.
The CONCAT detector (`embedded_exchanges()`, v1.4.0) stays as a regression
gate and now reports 0 on every trace. VERSION → 1.5.0. Baseline
regenerated.
**Phase B status (v1.5.0).** Two edge-reader hardening fixes shipped, both
verified byte-neutral on all 10 traces (stored baseline unchanged):
- `_next_start_fall()` now rejects sub-ETU phantom falls (a genuine start
fall must stay LOW for >= half an ETU) — B2.
- `_read_byte_edges()` re-anchors forward on a spurious latch instead of
dropping the byte (a drop fabricates a 0x00 / shifts the boundary) — B1.
Root cause of the Samsung/xiaomi phone-trace concatenation (the B1/B2 family)
is fixed at the source by the v1.5.0 invalid-INS re-frame — the A55 burst
cases decode clean with 0 CONCAT. The residual 4gmodem/A55_sim2/S21p_sim2
family (leftover-SW2-shifted header) is also gone. The planned B3 (strict
positional genuine-start anchor in the no-idle resync path) directly
conflicts with the documented 0.5-vs-1.5-ETU guard trade-off for fast phone
ETUs and risks the Samsung de-concatenation; it needs a dedicated design +
live validation, not an incremental patch.
### v1.3.0: `starts_with_atr` option removed
The entire ATR-included mode was redundant: its only mode-specific logic was a
start-state bootstrap (`FIND START` vs `DATA`) and an early trust hint for
`clock_skip=372`, both of which mid-session decoding reproduces via the
warm-reset / cold-boot re-arms already handled in `state == 'DATA'`
(starting the hunt instead of a fixed initial state). Every trace in the
suite produces identical APDU/GARBAGE/CHKSUM results with the option removed.
Removed:
- The `starts_with_atr` decoder option (schema + the 6 branching sites in
`pd.py`: initial state, `_clock_skip_confident` bootstrap, the
`handle_atr` samplenum save/restore, the cold-boot re-arm guard, and the
ETU re-measure condition — now `self.bit_samples is None or
self._pps_speed_changed`).
- `--starts-with-atr` from `start.sh` (now warned as obsolete).
The decoder always starts in `DATA` and re-arms the ATR hunt as needed.
### v1.2.1: cold-boot capture support
A capture that starts pre-power-up (VCC/RST low at t=0) previously locked a
bogus ETU (`bit_samples=1548`, `clock_skip=369`) from pre-ATR power-up DATA
transitions in mid-session mode, so the real ATR never synced. Fixed in
`241b7ed`:
- **Priming.** First `decode_step` reads the initial RST/VCC levels instead of
defaulting `last_rst`/`last_vcc` to 0 (which fabricated a reset edge).
- **Cold-boot re-arm.** RST low at capture start → arm the hunt at the
spec-default rate (FIND START, `clock_skip=372`, no ETU lock).
- **Guarded deassert re-arm.** The 10 ms-hysteresis-confirmed deassert lands
mid-ATR on warm resets; it only re-arms when `state == 'DATA'` so it no
longer garbles the in-flight ATR read (test_8 mid-session CHKSUM 18→0,
test_7 ATR-included 2→0).
New fixtures `xiaomi_mi_a1_coldboot_sample.sr` / `4gmodem_coldboot_sample.sr`
decode their full ATR in both modes with 0 GARBAGE.
### v1.2.0: start-bit deglitch (Samsung de-concatenation)
Samsung phone traces with gated CLK and non-default F/D (16 CLK/ETU)
produced concatenated APDUs: multiple T=0 exchanges merged into a single
GSMTAP packet. The v1.1.9 fixes (turnaround idle handling, edge-list
deglitch) improved other phone traces but the Samsung trace still showed
concatenation because the CLK-synchronous reader's start-bit detector
could not use the edge-list deglitch filter.
Fix applied (v1.2.0):
- Add a 2-sample deglitch in `wait_data_falling()`: after the 1.5-ETU
minimum-high guard passes, skip 2 additional samples and re-check DATA.
If DATA returns to idle, the start bit was a glitch → continue hunting.
This eliminates 1-sample (62.5 ns) LOW glitches in inter-byte guard
times that caused byte-level misalignment in the CLK-sync reader.
Results:
- Samsung mid-session: 52 → 924 APDUs (de-concatenated), all valid ISO 7816
commands (SELECT, GET RESPONSE, READ BINARY, STATUS, STK envelope), 0 GARBAGE.
- test_7 ATR: 10 → 65 APDUs, all 37/37 reader exchanges now captured
(was 10/37 in v1.1.9 due to ATR resets fragmenting the session).
- test_8 ATR: 54 → 56 APDUs, 28/37 reader exchanges (unchanged — 9 missing
are pre-capture, not a decoder issue).
- Xiaomi mid-session: 620 APDUs (unchanged, no regression).
**Glitch rejection measurements** (`-l 5` session log):
| Trace | Glitches rejected | Notes |
|-------|-------------------|-------|
| test_8 (reader) | 6 | Constant CLK, minor noise/turnaround artifacts |
| test_7 (reader) | 14 | Constant CLK, more activity than test_8 |
| xiaomi (phone) | **0** | Edge-list `_deglitch_edges()` filters before `wait_data_falling` |
| samsung (phone) | **1041** | CLK-sync reader has no upstream deglitch, confirms FX2LP pull-up hypothesis |
Xiaomi shows 0 because the edge-list reader's `_deglitch_edges()` (0.2×ETU
threshold) removes sub-ETU pulses before they reach `wait_data_falling()`.
The CLK-sync reader has no equivalent filter, so Samsung catches every
glitch. The 6–14 glitches on reader traces are minor noise or turnaround
artifacts in the constant-CLK captures.
The 2-sample advance in `wait_data_falling` shifts `_start_fall` by 2
samples, but at Samsung's 16 CLK/ETU (~72 samples) this is negligible
(<3% of a bit period) and does not affect downstream byte reading.
**Design note: magic number.** `_min_pulse_samples = 2` is tuned for the
primary 16 MHz sample rate (2 samples ≈ 125 ns, well within the FX2LP
pull-up rise time of 200–500 ns). At other sample rates the value would
need adjustment — but since all current traces are 16 MHz, this is left
as a hardcoded constant rather than a decoder option. If 12 MHz or
24 MHz traces are added later, consider making it configurable or
scaling it from the sample rate.
**Logging.** Each rejected start-bit glitch is logged with a running
count (`start-bit glitch rejected (#N)`) visible in the session log.
This helps diagnose whether the deglitch is too aggressive on a
particular capture — if the count is unexpectedly high, the 2-sample
window may be filtering real start bits.
### Proven data rates (all at 16 MHz sample rate)
| Trace | ETU (samples) | CLK/ETU | Mode |
|-------|---------------|---------|------|
| test_8 | ~43 | 372 (standard) | constant CLK, ATR+PPS |
| test_7 | ~43 | 372 (standard) | constant CLK, warm resets |
| xiaomi | ~53 | recovered from DATA | gated CLK, mid-session |
| samsung | ~72 | 16 (F=512/D=32) | gated CLK, non-standard F/D |
##### T=0 Procedure Bytes
| Byte | Meaning |
|------|---------|
| ACK (== INS or == ~INS) | Full data match: card acknowledges and expects all P3 data bytes |
| INS^0x01 or ~INS^0x01 | Single-byte match: card validates syntax but processes data one byte at a time |
| 0x60 | NULL: card busy, wait for next procedure byte (never an SW1) |
| 0x9E / 0x9F | Extended-length: next byte L = number of response data bytes, then SW |
| SW1 (`0x61`–`0x6F` / `0x9X`) | Status word: exchange complete |
##### CLA (Class) Byte — Coding and Logical Channels
Accepted (v1.9.3):
- `000x xxxx` (`0x00`-`0x1F`): first interindustry — SM in b4b3, chaining in
b5, logical channel **0–3** in b2b1.
- `01xx xxxx`: further interindustry — logical channel **4–19**
(b4..b1 = channel − 4). Accepted: `0x40`-`0x4F` plus the legacy
`0x50`-`0x5F` chaining variants (no CLA-level SM).
- UICC extended classes `'CX'`/`'EX'` (`0xC0`-`0xCF`/`0xE0`-`0xEF`):
channels 4–19 in b4..b1, SM in b6.
- Legacy `0x80`/`0x84`/`0xA0`/`0xB0`.
Rejected: `0x20`-`0x3F` (ISO-reserved); `0x60`-`0x6F`/`0x70`-`0x7F` (SM in
b6 — `0x60`-`0x6F` overlaps printable ASCII, and text bytes started bogus
frames on the mis-framed phone traces); the `'8X'`/`'AX'` standard-channel
variants `0x81`-`0x83`/`0x85`-`0x8F`/`0xA1`-`0xAF` (channels 0–3; same
trade-off);
`0x90`-`0x9F`, `0xD0`-`0xDF`, `0xF0`-`0xFF`.
Examples: `0x00` (standard), `0x01` (channel 1), `0x44` (channel 8),
`0x4F` (channel 19), `0xC0` (STATUS on channel 4, `'CX'` class),
`0x0C` (secure messaging).
##### INS (Instruction) Byte — Valid Ranges
- `0x00` is reserved/invalid.
- `0x6x`/`0x9x` collide with status words and are invalid.
- Odd INS codes are valid only as the defined alternate variants
(`0F` ERASE BINARY, `21` VERIFY, `87` GENERAL AUTHENTICATE,
`A1` SEARCH BINARY, `B1` READ BINARY, `B3` READ RECORD, `C3` ENVELOPE,
`CB` GET DATA, `D1` WRITE BINARY, `D7` UPDATE BINARY, `DB` PUT DATA,
`DD` UPDATE RECORD; TS 102 221 AUTHENTICATE `89`).
Common INS values: `A4` (SELECT), `B0`/`B1` (READ BINARY),
`D6`/`D7` (UPDATE BINARY), `B2`/`B3` (READ RECORD), `E2` (APPEND RECORD),
`20`/`21` (VERIFY), `82` (EXTERNAL AUTHENTICATE), `88`/`89` (AUTHENTICATE),
`C2`/`C3` (ENVELOPE: CAT), `C0` (GET RESPONSE).
##### Quick Validation Checklist
1. CLA is a valid class: `0x00`–`0x1F`/`0x40`–`0x5F` (interindustry, channels
0–3/4–19) or `0xC0`–`0xCF`/`0xE0`–`0xEF` (UICC extended) or
`0x80`/`0x84`/`0xA0`/`0xB0`
2. INS is not `0x00`
3. INS is outside the `0x6x`/`0x9x` status-word ranges
**Current smoke-test status:**
| Trace | APDUs |
|-------|-------|
| test_8 | 56 |
| test_7 | 65 |
| xiaomi_phone | 640 |
| samsung_phone | 924 |
| xiaomi_mi_a1_coldboot | 473 |
| 4gmodem_coldboot | 327 |
| A55_coldboot_new_cable | 407 |
| A55_coldboot_new_cable_sim2 | 395 |
| S21p_coldboot_new_cable | 380 |
| S21p_coldboot_new_cable_sim2 | 356 |
The baseline runs every trace through the decoder once
(intentional since v1.3.0: the `starts_with_atr` option was removed and the
decoder always runs mid-session — it captures all APDUs, including cold-boot
ATRs, and is the only mode that works on gated-CLK phone traces).
The four `samsung_*_coldboot_new_cable*_sim2` fixtures were captured with the
new short cable (30 mm wires, external pull resistors GND→VCC and VCC→DATA),
named channels `DATA/VCC/RST/CLK` on bits 0/2/4/5 at 16 MHz. All four decode
their full cold-boot ATR (`3b9f9580…` on SIM1, `3b9f9680…` on SIM2) with
0 GARBAGE / 0 BAD_FCS. S21p SIM2 carries 197 CHKSUM (dirty-wire parity
noise, BAD_FCS still 0); A55 SIM2 is clean (0 CHKSUM). The `_sim2` fixtures
are cropped to the power-on window (the 17–21 s of pre-power idle was
removed).
Cold-boot fixtures start **before power-up**
(VCC/RST low at t=0); the decoder must arm the ATR hunt when RST is still low
at capture start and decode the full ATR once the card wakes (~17 s in).
`l8star_coldboot_sample.sr` was captured in the same session but had a
dirty-wire tail (GARBAGE=1 / RESULT UNCLEAN) and was dropped from the suite.
**Note on v1.1.3 fix:** The decoder measures CLK period as an average instead of using the minimum recurring period. This correctly handles traces where CLK periods alternate (e.g., 4,3,3 pattern for ~5.33MHz CLK at 16MHz sample rate). The fix changed `_measure_clock_period()` to use `sum(spacings) / len(spacings)` instead of `_robust_min(spacings)`, giving accurate `spc` values like 3.333... instead of 3. This corrected the `_wait_clk_rising()` skip calculation, fixing the post-ATR byte alignment issue.
### GSMTAP wire-format extensions
The GSMTAP-SIM output uses the standard sub_types 0x00-0x03 from
libosmocore's gsmtap.h with standard payloads -- APDU, ATR, and the PPS
request/response as separate packets (see README.md "GSMTAP events" and the
comments in `gsmtap_stream.py`). It deviates in two documented ways:
0x10/0x11 (`RST_EVENT`/`VCC_EVENT`) are custom line-event subtypes, and
`GSMTAP_FLAG_BAD_FCS` writes flags into the header `res` byte, which
official gsmtap.h reserves. The `res`-byte carriage is this decoder's own
extension (upstream simtrace2 never sets `res`); the semantics mirror
simtrace2's per-message USB data flags. 0x10 RST events additionally
append the measured CLK frequency when known
(`[direction, level, flags, clk_hz_be32]`; see "v1.9.0" below).
### Versioning
The decoder version is defined in `pd.py` as `VERSION = '1.9.3'`.
The version is printed to the log on decoder startup.
### Testing after decoder changes
After any change to `pd.py`:
1. Clear bytecode cache:
```bash
find . -name "__pycache__" -type d -exec rm -rf {} +
```
2. Run all tests against all traces:
```bash
./tools/capture_baseline.sh > tests/baseline.txt
```
3. Compare against saved baseline:
```bash
./tools/compare_baseline.sh
```
4. If differences found, review:
- Capture APDU count must stay >0 for any trace that previously had APDUs
- GARBAGE count must stay 0 for test_8 (mid-session)
- RESULT must stay OK for test_8 (mid-session)
- CHKSUM ERROR count must not increase
- BAD_FCS / payload mismatches / suspicious CLA must not increase
- ETU values must match for mid-session traces
- No new INVALID Procedure Byte errors
5. Run unit tests:
```bash
python3 tests/test_gsmtap.py -v
```
6. Update `tests/baseline.txt` with new baseline if changes are intentional.
## What happened: the decoder rewrite attempt
We attempted a major rewrite of the ISO 7816 decoder (`pd.py`) with these
features:
1. **Edge-counting always** — removed the `native_fast` sample-skip bit
reader, making all bit reads count CLK edges per bit instead of skipping
`clock_skip * samples_per_edge` samples. The idea was drift-immunity.
2. **Signal degradation recovery** — a `degraded` flag tracking rolling
parity errors; when ≥ 8 of 64 frames fail parity, the decoder re-arms
the ATR hunt and flags APDUs with `GSMTAP_FLAG_BAD_FCS`.
3. **Clock-drift monitor** — every 128 bytes the decoder re-measures the
CLK period and re-anchors `samples_per_edge` if the median drifts > 2%.
4. **Fatal TCK check** — the ATR's TCK checksum mismatch was made fatal
(reject the ATR) instead of being logged as a warning.
5. **Synthetic ATR fallback** — after 8 failed ATR hunts, commit a
minimal T=0 ATR `[0x3B, 0x00]` and begin decoding, for mid-session
captures.
## Outcome
The rewrite introduced a regression: **35 garbage APDUs** on `test_8`
where the baseline had **0 garbage**. The root causes:
1. **Fatal TCK check** — the ATR in the example traces has a wrong TCK
(`got=f7 expected=80`) due to a bit error in the capture. The fatal
check rejected this real ATR, preventing the decoder from ever
reaching the DATA state.
2. **Signal degradation ATR re-arm** — the `degraded` flag triggered on
the capture's natural bit errors, re-arming the ATR hunt mid-session
and fragmenting the APDU stream.
3. **Stale `.pyc` cache** — the Python bytecode cache was not
regenerated after code changes, causing the running decoder to use
old code. This masked the real regression for several test runs.
The edge-counting change itself may also have contributed subtle
bit-sampling differences, but the TCK check and signal degradation were
the primary causes.
## Resolution
We reverted to the known-good baseline (`2055d1c`, the VCC/RST hysteresis
commit) and applied only three essential, minimal fixes on top:
1. **`hasT0`/`hasT1` init in `start()`** — initialize both to `False` for
`protocol=auto`, preventing `AttributeError` when `starts_with_atr=false`
enters DATA state without an ATR.
2. **`hasT0=False` guard in `decode_step` DATA state** — when no protocol
is known, skip byte reading and wait for a warm reset to trigger the
ATR hunt.
3. **Synthetic T=0 ATR fallback** — after 8 failed ATR hunts, commit
`[0x3B, 0x00]` and set `hasT0=True` so the decoder can begin
processing APDUs in mid-session captures.
The `native_fast` sample-skip, signal degradation, clock monitor, and
fatal TCK check were all discarded. The decoder is now the original
`2055d1c` code plus 24 lines of mid-session support.
## Current architecture (v1.1.1, post-ATR desync fixes)
The decoder uses a **CLK-synchronous bit reader** for native-CLK captures and
an edge-list DATA reader for the non-native clock modes. Bit timing comes
from **counting real CLK edges**, not from a sample-count estimate.
### Two bit readers (selected per burst)
| Reader | Timing source | Used when |
|--------|---------------|-----------|
| `_read_byte_clk()` | count `clock_skip` CLK rising edges per ETU, sample DATA at bit centres | native clock mode **and** `clock_skip` known (`_clock_skip_confident`) |
| `_read_byte_edges()` | DATA-edge reconstruction from `bit_samples` | `sample_as_clock`/`detect`, or mid-session before `clock_skip` is recovered |
### Key changes
1. **CLK-synchronous reader.** `_read_byte_clk()` counts `clock_skip` CLK
rising edges per ETU and samples DATA at the bit centres (1.5 … 9.5 ETU).
Exact for `0x00`/`0xff` (too few DATA transitions for the edge reader) and
deterministic, because `clock_skip` is protocol-defined (372 for the ATR,
FI/DI after PPS). Safe across Clock Stop: CLK never stops mid-character
and resumes before the next start bit. `_wait_clk_rising()` skips most of
each bit by samples then lands on the exact edge, so it is fast.
2. **No synthetic ATR.** The old `[3B, 00]` fallback fabricated a 2-byte ATR
from a stray `0x3B`/`0x3F` byte in command traffic (e.g. the FID `3F00` of
a SELECT) and destroyed the command. The hunt is bounded and never emits
a fake ATR.
3. **Command detection during ATR hunt.** A plausible CLA + valid INS is
replayed to the DATA state and decoded as a command instead of skipped as
"invalid TS" — this recovers a SELECT when the preceding ATR was missed.
4. **RST resets `clock_skip` to 372** on RST assert (a reset makes the card
send ATR at the default rate).
5. **PPS on the native path** just updates `clock_skip` from FI/DI (CLK
period is unchanged, so the new ETU is exact). No DATA re-measurement
burst is consumed — previously that silently dropped the first post-PPS
command.
6. **Mid-session `clock_skip` recovery.** `_measure_etu()` recovers
`bit_samples`; `_measure_clock_period()` recovers `_samples_per_clock`;
`_derive_clock_skip_from_etu()` sets `clock_skip = bit_samples /
_samples_per_clock` and marks it confident. If the CLK period cannot be
measured (fully gated CLK), the edge-list reader is used instead.
7. **Data handling.** Valid APDUs are emitted clean; ambiguous frames are
emitted flagged `GSMTAP_FLAG_BAD_FCS` (never silently dropped); genuine
idle-low all-zero noise remains suppressed.
### Testing
- `test_8` / `test_7`: **NOT YET PASSING.** 0 garbage, but decoder output
does not match reader log (T=0 framing groups exchanges incorrectly).
See success criteria above.
## Mid-session phone decode (legacy note, still applies)
For a live mid-session phone capture (`protocol=T=0`),
the decoder now:
- Recovers the ETU from the live DATA line via `_measure_etu()` (edge-list,
locked once) so a gated/stopped CLK at a fast ETU (e.g. ~53 samples) is
decoded instead of mis-read at the 372-clock default.
- Decodes bytes with the edge-list reader `_read_byte_edges` (the only
reader now), which is drift-immune.
- Runs the normal T=0 framing so each command+response exchange is emitted as
one APDU.
Validation: `test_8`/`test_7` 0 garbage in both modes; 12 unit tests pass;
phone 16M/24M decode to coherent APDUs with the STK text fully readable and
no `0xff`/`0xfc` garbage.
### T=0 case 3/4 command-data (RESOLVED)
Commands that carry command-data (e.g. `80 12 … d0`, P3=208) send those
P3 bytes on the wire **before** the card's first procedure byte. The
framing now distinguishes the cases by the card's first procedure byte:
- If the first post-header byte **is the ACK (== INS)** → case 2/4: the
terminal sent no command data; read `P3` response bytes (+SW).
- Otherwise the byte is terminal command data → case 3/4: read `P3`
command-data bytes, then the card's procedure byte (ACK/SW/`9E`/`9F`).
On ACK (case 4) the response is read `P3` response bytes (+SW).
This consolidates case-3/4 exchanges (e.g. the `80 12`/`80 14` STK envelope
commands) into one C-APDU = header + command data, R-APDU = response + SW
APDU, instead of fragmenting them across frames. It is gated to
`_edge_read` (mid-session / gated-CLK captures) so ATR-based captures
(`test_8` / `test_7`) keep the original framing unchanged and stay at
0 garbage.
All paths use P3-bounded reads (v1.1.1 reverted the read-until-SW
experiment from v1.1.0 which mis-identified FCP `0x62` as SW1).
## Decoder methods tried — reliability registry
A running log of decoder approaches, so future work does not re-litigate
discarded ideas. "Proven" = kept in `pd.py`; "Rejected" = do not re-add.
### Proven reliable (kept)
- **Sample-skip bit reader (`native_fast`).** Original reader; advances bit
timing with single sample-skip waits. Reliable for ATR-based captures
(`test_8`/`test_7`, 0 garbage). Keep for those.
- **ATR TS validation + resync.** Validate TS (`0x3B`/`0x3F`); resync on the
next fall if bogus. Reliable; stops a phantom start bit shifting the ATR
off-by-one.
- **Warm-reset re-arm (guarded).** RST deassert → re-arm the ATR hunt
(`FIND START`, `clock_skip` 372, cleared ETU) **only when `state == 'DATA'`**
(hunt genuinely idle). Deasserts that arrive while `FIND START`/`ATR` is in
progress just log "RST deasserted (hunt/ATR already in progress)" and leave
state alone — an unguarded mid-ATR re-arm corrupted the in-flight ATR read
(test_8 mid-session CHKSUM 18→0, test_7 ATR-included 2→0 after the guard).
Reliable; re-ATRs parse at the default rate.
- **Cold-boot priming.** One-shot on the first `decode_step`: `wait({'skip': 0})`
reads the *initial* VCC/RST levels instead of defaulting `last_rst`/`last_vcc`
to `0`. Without this, a capture that starts pre-power-up fabricated a spurious
reset at sample 0.
- **Cold-boot re-arm.** When the capture starts with RST **low** (VCC not yet
on), arm the hunt immediately regardless of start conditions: state
`FIND START`,
`_atr_hunt_count=0`, `clock_skip=372`, `bit_samples=None` (no ETU lock), so the
pre-ATR power-up DATA transitions cannot lock a bogus ETU (`bit_samples=1548`,
`clock_skip=369` before this fix). Logs "RST low at start: expecting cold-boot
ATR". Decodes the full cold-boot ATR on `xiaomi_mi_a1_coldboot_sample.sr` /
`4gmodem_coldboot_sample.sr` (0 GARBAGE).
- **1.5-ETU start-bit guard.** Rejects the phantom `0x00` an idle glitch
inserts before an ATR. Reliable.
- **VCC/RST 10 ms level-stability hysteresis.** Reports a level only after
≥10 ms stable. Reliable; symmetric power-up/down, kills spurious events
from a ringing/floating sense line.
- **PPS speed-switch follow (`clock_skip = F/D`).** After an accepted PPS the
native skip becomes F/D CLK cycles/bit. Reliable (verified live).
- **T=1 block-chain follow-up bound.** Caps chained I-blocks so garbage
cannot loop forever. Reliable.
- **Synthetic T=0 ATR fallback `[0x3B, 0x00]`.** After 8 failed ATR hunts,
commit a minimal ATR and set `hasT0=True`. Reliable for mid-session starts
(no ATR in the trace).
- **ETU auto-detect from DATA via edge counting (`_measure_etu`).** Edge-waits
to the first DATA start-bit fall, measures ETU from DATA-edge spacing, locks
it **once**. Reliable for gated/stopped CLK (phone, ~53 samples). Locking
once (not re-measuring) is drift-immune and avoids glitch re-anchors.
- **Edge-list bit reader (`_read_byte_edges`).** Per-byte edge reconstruction
from the DATA edge list (re-anchors the next byte via `_next_start_fall`).
Drift-immune. Reliable for mid-session/edge mode; gated by `_edge_read`.
- **`decode_step` unified dispatch.** One state machine handles
ATR/PPS/DATA + mid-session. Reliable; current architecture.
- **`hasT0`/`hasT1` guards + init.** Prevents `AttributeError` and skips byte
reading until a protocol is known. Reliable.
- **T=0 case-3/4 ACK==INS discriminator.** First post-header byte == INS ⇒
case 2/4 (response bounded by `P3`); else command data (read `P3`, then
the procedure byte). Reliable for consolidating STK envelope commands.
Gated to `_edge_read`. P3-bounded reads avoid FCP tag misidentification
(v1.1.1 reverted read-until-SW from v1.1.0).
- **1.5-ETU start-bit guard in `wait_data_falling`.** `min_high = 1.5 *
bit_samples` distinguishes genuine inter-byte idle gaps (≥2 ETU per ISO
7816-3) from intra-byte HIGH periods (≤1 ETU). Prevents byte misalignment
after `_measure_etu()` consumption (v1.1.1 fix for post-ATR desync).
- **Inverse-convention ATR detection in DATA state.** `firstByte == 0x3f`
alongside `0x3b` at line 1656, so inverse-convention ATRs (TS=0x3F) are
parsed by the default state. Sets `clock_skip=372` correctly
before PPS (v1.1.1 fix).
- **`_resync = True` after PPS acceptance (native CLK path).** Forces
`resync_idle()` before the first post-PPS command, preventing the decoder
from latching onto a mid-byte transition at the new speed (v1.1.1 fix).
### Tried and rejected (do not re-add)
- **Edge-counting as the *sole* bit reader (remove `native_fast`
entirely).** Subtle per-bit sampling differences regressed `test_8` (35
garbage). Edge-counting is fine **as a second reader** (`_read_byte_edges`)
used only in mid-session mode; it is not a drop-in replacement for the whole
pipeline.
- **Signal-degradation ATR re-arm (`degraded` flag on ≥8/64 parity errors).**
Triggered on the capture's natural bit errors, re-armed the ATR hunt
mid-session, and fragmented the stream. Rejected.
- **Clock-drift monitor (re-measure CLK every 128 bytes, re-anchor
`samples_per_edge`).** Unneeded: locking the ETU once is sufficient and
safer. Rejected.
- **Fatal TCK check.** The example ATRs carry a wrong TCK (capture bit error,
`got=f7 expected=80`); fatal rejection blocked the DATA state entirely.
TCK must stay a non-fatal warning. Rejected.
- **CLA/P1P2 heuristic routing to T=1.** Misread real T=0 traffic as T=1.
Rejected.
- **`parse_packet` heuristic / APDU-boundary guessing from CLA/P1P2 + Le/ALG
tables.** Produced false splits and mis-framed exchanges. Rejected in favor
of proper T=0/T=1 transport state machines.
- **`ignore_rst` / `ignore_vcc` options.** Superseded by channel assignment +
hysteresis. Rejected.
- **Generic "read response until a `6x`/`9x` status word" for T=0 framing.**
Any `0x6x`/`0x9x` byte (e.g. command-data `0x6c`, FCP `0x6f`) was taken as
SW1, truncating/fragmenting exchanges and dropping most of the capture.
Rejected; replaced by the ACK==INS discriminator (case 2/4 bounded by
`P3`) with P3-bounded reads for all paths (v1.1.1 reverted the
read-until-SW experiment from v1.1.0 which mis-identified FCP `0x62` as
SW1).
- **0.5-ETU start-bit guard in `wait_data_falling` (for `_edge_read`).**
`min_high = 0.5 * bit_samples` was too small for slow ETUs — it accepted
intra-byte HIGH periods (data bit transitions) as start bits, causing byte
misalignment after `_measure_etu()` consumption. Replaced by 1.5-ETU guard
in v1.1.1.
### Open residual limitations
- Mid-session captures that begin inside an exchange cannot frame the first
command perfectly (expected; the rest of the stream is clean).
- **Samsung phone — gated CLK, non-default F/D** (`samsung_phone_sample.sr`).
Uses F=512/D=32 (16 CLK cycles/bit). `_measure_etu` correctly measures
the ETU from DATA edge spacings (after glitch filtering). Now passes
with 0 GARBAGE and de-concatenated APDUs after v1.1.10 deglitch fix.
### v1.1.6 fix: bounded T=0 loops and explicit EOF handling
Tightened the edge-read T=0 procedure-byte handlers to prevent runaway
reads and to surface end-of-capture explicitly.
1. **Bounded all "read until SW" / turnaround-skip loops.** Replaced the
unbounded ACK-handler `while` and the open-ended `range(512)` loops with
`for _ in range(MAX_TPDU_LEN)`. Added EOF checks so the decoder returns
cleanly at end-of-capture instead of consuming zeros/turnaround bytes
indefinitely.
2. **`read_byte()` / `peek_byte()` return `None` on EOF.** Previously both
methods returned `0x00` when the sample stream ended, making it
impossible for callers to distinguish real data from padding. They now
return `None`; the T=0 DATA-state header and procedure-byte loops check
for `None` and exit the current exchange.
3. **Skip turnaround idle bytes before appending.** The P3=0 and case-4
ACK "read until SW" paths now mirror the v1.1.5 unexpected-byte fix:
non-SW bytes (including 0xFF turnaround artifacts) are skipped before
the status word is appended, rather than being added to the packet.
### v1.1.5 fix: turnaround idle bytes in edge-read path
Samsung phone traces with gated CLK had 5 garbage APDUs (BAD_FCS) in
mid-session mode. Root cause: two code paths in the edge-read T=0
procedure-byte handler did not skip turnaround idle bytes (0xFF) on the
bidirectional DATA line.
1. **ACK handler (case 2/4):** After reading P3 response bytes, the code
read 2 raw bytes as SW1 SW2. For case-3 commands (SELECT with data),
the terminal sends command data after ACK, then the DATA line idles
(0xFF) before the card sends SW. The idle 0xFF was mis-identified as
SW1. Fixed by adding a `while` loop that skips non-SW bytes before
reading SW1.
2. **"Unexpected byte" handler (procedure byte loop):** Non-SW bytes
(e.g. 0xFF turnaround) were appended to the packet before scanning for
SW, corrupting the SW position. Fixed by moving `packet.append(pb)`
inside the SW1 check so non-SW bytes are skipped.
Reference in new issue View Git Blame Copy Permalink