catarrh/simtrace-analyser
1
0 Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Files view: per-file tracking (counters, image, device/card changes)

Browse Source 
- new `simtrace_analyser/filetrack.py`: a single-pass tracker over a
  session's decoded messages, keyed by (DF, FID) and trace-wide (resets only
  wipe the per-channel selection): counters (`select`/`read`/`update`,
  successful operations only), a sparse image of the observed bytes
  (transparent files by offset, record files per record number) and the
  change log
- changes: **device** = an UPDATE seen in the trace (applied to the image,
  `no_change` flags a provable no-op); **card** = a READ whose bytes differ
  from the image on bytes already known (logged at the surfacing read with
  the changed ranges and the before/after hex; never-seen bytes just fill
  the image, they are an observation, not a change)
- SFI-addressed accesses join the same entry as path/FID ones: the SFI ↔ FID
  relations come from the normative tables and from every surfaced FCP `88`
  DO (including the spec's "SFI = 5 LSBs of the FID" rule when the DO is
  absent); an SFI used before its relation is learned stays provisional and
  merges into the resolved entry
- the DF identity follows TS 102 221 §8.4/Table 8.2: `selected_df_fid()` now
  resolves an EF path selection to its parent DF and returns `aid:<hex>` for
  applications selected by an unknown AID (named from EF.DIR when the trace
  surfaced it); `Database._decode_rows` gained an `on_message` hook and
  `track_files()`
- `GET /api/sessions/<id>/files`: the operation timeline, the per-file
  counters/coverage/change log, the SFI relations and the AID labels
- PWA: the side panel's Map/Stats tabs became a view selector (Map / Stats /
  Files); the Files view is a timeline of Read / Update / Changed (card) rows
  (colour-coded, the same file names as the Map view), a row click jumps to
  the APDU (loading the pages it needs) and a card-change row's Δ opens the
  changed ranges with the previous and the new bytes; EN/RU
- bounds: the image freezes past 64 KB of observed bytes per file, the change
  log keeps the last 1000 events per file (counters keep counting)
- tests: tests/test_filetrack.py (counters, resets, channels, image/diff
  semantics, record files, SFI resolution and the provisional merge, EF.DIR
  labels, caps, the endpoint) and Node tests for the Files-view helpers and
  wiring
- v2.3.0; MIN_SERVER_VERSION 2.3.0; SW cache simtrace-analyser-v63
This commit is contained in:
catarrh committed 2026-10-17 20:51:33 +03:00
1 parent b8d9c4cc63
commit c494bf4111
14 files changed
+1301 -35
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+29
View File
@@ -186,6 +186,35 @@ one.
Exchanges larger than 255 bytes (extended-length APDUs) cannot be represented
in the stored wire shape and are skipped with a note in the import report.
### Files view (per-file tracking)
The Trace tab's side panel has a view selector (*Map* / *Stats* / **Files**;
a drop-down, since three tab captions no longer fit). The Files view is a
trace-wide timeline of file operations — every **Read**, **Update** and
**Changed (card)** row shows the same file name as the Map view, and clicking
a row jumps to the APDU in the trace. A *Changed (card)* row carries a Δ
button that opens the changed ranges with the previous and the new bytes.
The data comes from `simtrace_analyser/filetrack.py`
(`GET /api/sessions/<id>/files`), a single-pass tracker over the decoded
session that keeps, keyed by (DF, FID) and **across card resets**: access
counters (`select`/`read`/`update`, successful operations only), a sparse image
of the observed contents and the change log. Device-initiated changes are the
UPDATE operations seen in the trace; card-initiated ones are inferred when a
read returns bytes differing from what the image already knew — bytes never
seen before just fill the image (an observation, not a change). Files reached
by SFI share the entry of their path/FID accesses: the SFI ↔ FID relations come
from the normative tables (TS 102 221 Annex H, TS 31.102 Annex H) and from every
surfaced FCP carrying the `88` DO, including the spec's "SFI = the 5 LSBs of the
FID" rule when the DO is absent; an SFI used before its relation is learned
stays provisional and merges into the resolved entry. A file's DF context
follows TS 102 221 §8.4/Table 8.2 (selecting an EF leaves its parent DF current,
a path selection included); applications selected by an unknown AID get the AID
as their DF identity, named from EF.DIR when the trace surfaced it. Bounds: the
image freezes past 64 KB of observed bytes per file (`image_truncated`) and the
change log keeps the last 1000 events per file (`changes_total` keeps counting).
Counters are returned by the API but not shown in this view yet.
Server options:
```
README_RUS.md
+30
View File
@@ -191,6 +191,36 @@ Application Layer* — два отдельных поля выбора, имен
Обмены длиннее 255 байт (APDU с расширенной длиной) невозможно представить
в хранимом «проводном» виде — они пропускаются с пометкой в отчёте импорта.
### Просмотр файлов (сопровождение состояния файлов)
В боковой панели вкладки «Трассировка» появился выбор вида (*Map* / *Stats* /
**Files** — выпадающий список, три подписи-вкладки уже не помещались). Вид
Files — это таймлайн файловых операций по всей трассе: строки **Read**,
**Update** и **Changed (card)** показывают те же имена файлов, что и вид Map,
а щелчок по строке переходит к APDU в трассе. У строки *Changed (card)* есть
кнопка Δ, открывающая изменённые диапазоны с прежними и новыми байтами.
Данные даёт `simtrace_analyser/filetrack.py` (`GET /api/sessions/<id>/files`) —
однопроходный анализатор сессии, который ведёт по ключу (DF, FID) и **через
сбросы карты**: счётчики доступа (`select`/`read`/`update`, только успешные
операции), разреженный образ наблюдаемого содержимого и журнал изменений.
Изменения со стороны устройства — это операции UPDATE из трассы; изменения со
стороны карты выводятся, когда чтение возвращает байты, отличающиеся от уже
известных образу (никогда не виденные байты просто заполняют образ — это
наблюдение, а не изменение). Файлы, доступные по SFI, попадают в ту же
запись, что и доступ по пути/FID: соответствия SFI ↔ FID берутся из
нормативных таблиц (TS 102 221 Annex H, TS 31.102 Annex H) и из каждого
показанного FCP с DO `88`, включая правило спецификации «SFI = 5 младших бит
FID» при отсутствии DO; SFI, использованный до того, как связь выучена,
остаётся предварительным и сливается с разрешённой записью. Контекст DF
следует TS 102 221 §8.4/Table 8.2 (выбор EF оставляет текущим его родительский
DF, включая выбор по пути); приложения, выбранные по неизвестному AID,
получают в качестве идентичности DF сам AID и имя из EF.DIR, если он
показан в трассе. Ограничения: образ замораживается после 64 КБ
наблюдаемых байт на файл (`image_truncated`), журнал изменений хранит
последние 1000 событий на файл (`changes_total` продолжает считать).
Счётчики API отдаёт, но в этом виде пока не показываются.
Параметры сервера:
```
frontend/index.html
+177 -19
View File
@@ -181,6 +181,20 @@ body { font: 14px/1.5 system-ui, sans-serif; background: var(--bg); color: var(-
.side-tabs { display: flex; margin-bottom: 8px; border-bottom: 1px solid var(--bg-input); }
.side-tab { flex: 1; padding: 6px 0; background: none; border: none; color: var(--text-muted); cursor: pointer; font-size: 12px; text-transform: uppercase; letter-spacing: 1px; border-bottom: 2px solid transparent; }
.side-tab.active { color: var(--accent); border-bottom-color: var(--accent); }
.side-view-select { width: 100%; margin-bottom: 8px; padding: 6px 8px; background: var(--bg-input); color: var(--text);
border: 1px solid var(--border); border-radius: 4px; font-size: 12px; text-transform: uppercase; letter-spacing: 1px; cursor: pointer; }
.file-op { display: flex; align-items: baseline; gap: 6px; padding: 3px 8px; cursor: pointer; border-radius: 4px; font-size: 13px; word-break: break-word; }
.file-op:hover { background: var(--bg-input); }
.file-op-read { color: var(--text-muted); }
.file-op-update { color: var(--accent); }
.file-op-card { color: var(--warn); }
.file-op .file-op-diff { margin-left: auto; background: none; border: none; color: var(--warn); cursor: pointer; font-size: 12px; padding: 0 2px; flex: 0 0 auto; }
.file-op .file-op-diff:hover { color: var(--accent); }
.file-diff-modal { max-width: 640px; }
.file-diff-body { max-height: 50vh; overflow-y: auto; margin-bottom: 16px; }
.file-diff-body table { font-size: 12px; }
.file-diff-body td { padding: 2px 8px 2px 0; vertical-align: top; }
.file-diff-body .file-diff-hex { font-family: ui-monospace, monospace; word-break: break-all; }
.map-item { padding: 3px 8px; cursor: pointer; border-radius: 4px; font-size: 13px; color: var(--text); word-break: break-word; }
.map-item:hover { background: var(--bg-input); }
.map-files-toggle { color: var(--accent); cursor: pointer; }
@@ -333,12 +347,14 @@ body { font: 14px/1.5 system-ui, sans-serif; background: var(--bg); color: var(-
</div>
</div>
<div class="side-panel" id="side-panel">
<div class="side-tabs">
<button class="side-tab active" data-view="map" data-l10n="Map">Map</button>
<button class="side-tab" data-view="stats" data-l10n="Stats">Stats</button>
</div>
<select class="side-view-select" id="side-view-select" aria-label="Side panel view">
<option value="map" data-l10n="Map">Map</option>
<option value="stats" data-l10n="Stats">Stats</option>
<option value="files" data-l10n="Files">Files</option>
</select>
<div id="type-counts">—</div>
<div id="type-map" style="display:none">—</div>
<div id="file-view" style="display:none">—</div>
</div>
</div>
<div class="status-bar">
@@ -419,8 +435,19 @@ body { font: 14px/1.5 system-ui, sans-serif; background: var(--bg); color: var(-
</div>
</div>
<!-- COMPRION trace import dialog -->
<div class="modal-overlay" id="comprion-modal">
<!-- Card-side change detail (Files view) -->
<div class="modal-overlay" id="file-diff-modal">
<div class="modal file-diff-modal">
<h3 data-l10n="Card-side change">Card-side change</h3>
<p id="file-diff-sub"></p>
<div class="file-diff-body" id="file-diff-body"></div>
<div class="modal-buttons">
<button class="cancel" id="file-diff-close" data-l10n="Close">Close</button>
</div>
</div>
</div>
<!-- COMPRION trace import dialog --><div class="modal-overlay" id="comprion-modal">
<div class="modal comprion-modal">
<h3 data-l10n="Import COMPRION trace">Import COMPRION trace</h3>
<p data-l10n="Select the ISO Protocol Layer export and, if available, the matching ISO Application Layer export of the same session.">Select the ISO Protocol Layer export and, if available, the matching ISO Application Layer export of the same session.</p>
@@ -470,6 +497,8 @@ const STATE = {
typeCounts: [],
traceMap: [],
sideView: 'map',
filesData: null,
filesSessionId: null,
pendingDeleteId: null,
serverAvailable: false,
serverName: '',
@@ -1149,6 +1178,7 @@ const API = {
startCapture() { return this._fetch('POST', '/api/capture/start'); },
stopCapture() { return this._fetch('POST', '/api/capture/stop'); },
getLatest(after) { return this._fetch('GET', `/api/capture/latest?after=${after}`); },
getSessionFiles(id) { return this._fetch('GET', `/api/sessions/${id}/files`); },
searchMessages(sessionId, query) { return this._fetch('GET', `/api/apdu/search?session_id=${sessionId}&q=${encodeURIComponent(query)}`); },
filterMessages(sessionId, type) { return this._fetch('GET', `/api/apdu/filter?session_id=${sessionId}&type=${type}`); },
async _importPost(params, body, contentType) {
@@ -2026,11 +2056,111 @@ function renderSidePanel() {
}).join('') || t('No commands');
const showMap = STATE.sideView === 'map';
el('type-counts').style.display = showMap ? 'none' : '';
el('type-counts').style.display = STATE.sideView === 'stats' ? '' : 'none';
mapEl.style.display = showMap ? '' : 'none';
document.querySelectorAll('#side-panel .side-tab').forEach(tab => {
tab.classList.toggle('active', tab.dataset.view === STATE.sideView);
el('file-view').style.display = STATE.sideView === 'files' ? '' : 'none';
el('side-view-select').value = STATE.sideView;
if (STATE.sideView === 'files') renderFileView();
}
// ──────────────────── Files view (per-file operation timeline) ────────────────────
function fileOpClass(kind) {
if (kind === 'update') return 'file-op-update';
if (kind === 'card_change') return 'file-op-card';
return 'file-op-read';
}
function fileOpVerb(kind) {
if (kind === 'update') return t('Update');
if (kind === 'card_change') return t('Changed (card)');
return t('Read');
}
function fileOpLabel(op) {
const name = (op.file && op.file.name) || '?';
return `${fileOpVerb(op.kind)} ${name}`;
}
// Split a change's concatenated before/after hex into per-range rows.
function fileDiffRows(change) {
const rows = [];
let at = 0;
for (const [pos, len] of (change.ranges || [])) {
const take = len * 2;
rows.push({
pos: pos,
len: len,
before: (change.before_hex || '').slice(at, at + take),
after: (change.after_hex || '').slice(at, at + take),
});
at += take;
}
return rows;
}
async function renderFileView() {
const view = el('file-view');
if (STATE.sessionViewId === null) {
view.innerHTML = `<div class="empty-state" style="height:auto;padding:24px;">${t('Open a saved session from the Sessions tab.')}</div>`;
return;
}
if (STATE.filesData === null || STATE.filesSessionId !== STATE.sessionViewId) {
view.innerHTML = `<div class="empty-state" style="height:auto;padding:24px;">${t('Loading…')}</div>`;
try {
STATE.filesData = await API.getSessionFiles(STATE.sessionViewId);
STATE.filesSessionId = STATE.sessionViewId;
} catch (e) {
STATE.filesData = null;
view.innerHTML = `<div class="empty-state" style="height:auto;padding:24px;">${t('Failed to load files')}</div>`;
return;
}
}
const data = STATE.filesData;
const timeline = (data && data.timeline) || [];
if (!timeline.length) {
view.innerHTML = `<div class="empty-state" style="height:auto;padding:24px;">${t('No file operations recorded')}</div>`;
return;
}
view.innerHTML = timeline.map(op => {
const diff = (op.kind === 'card_change' && op.change_ref)
? `<button class="file-op-diff" data-diff="${escapeHtml(op.change_ref.file)}|${op.change_ref.idx}" title="${t('Show the change')}">Δ</button>`
: '';
return `<div class="file-op ${fileOpClass(op.kind)}" data-idx="${op.index}">
<span>${escapeHtml(fileOpLabel(op))}</span>${diff}
</div>`;
}).join('');
}
function showFileDiff(fileKey, idx) {
const data = STATE.filesData;
const entry = data && data.files ? data.files[fileKey] : null;
const change = entry && entry.changes ? entry.changes[idx] : null;
if (!change) return;
const rows = fileDiffRows(change);
el('file-diff-sub').textContent = `${entry.name} — ${change.op}`;
el('file-diff-body').innerHTML = rows.map(r => `<table>
<tr><td>${change.record != null ? t('Record') : t('Offset')}:</td><td>${escapeHtml(change.record != null ? String(r.pos) : '0x' + r.pos.toString(16).toUpperCase())} (${r.len} B)</td></tr>
<tr><td>${t('Before')}:</td><td class="file-diff-hex">${escapeHtml(r.before.toUpperCase())}</td></tr>
<tr><td>${t('After')}:</td><td class="file-diff-hex">${escapeHtml(r.after.toUpperCase())}</td></tr>
<tr><td colspan="2" style="height:6px"></td></tr>
</table>`).join('');
el('file-diff-modal').classList.add('show');
}
// Load the pages up to *idx* (the message list is fetched in pages) and focus
// the message — used by the Files timeline rows.
async function jumpToMessage(idx) {
if (STATE.sessionViewId === null || idx === null || idx === undefined) return;
try {
while (STATE.messages.length <= idx) {
const data = await API.getSession(STATE.sessionViewId, STATE.messages.length);
if (!data.messages || !data.messages.length) break;
STATE.messages.push(...data.messages);
}
renderMessages();
focusMessage(idx);
} catch (e) { /* keep the current view */ }
}
function mapFilesHtml(g) {
@@ -2652,6 +2782,8 @@ async function loadSession(id) {
const data = await API.getSession(id, 0);
STATE.sessionViewId = id;
STATE.filesData = null;
STATE.filesSessionId = null;
STATE.sessionStarted = data.session.started;
STATE.captureActive = false;
STATE.currentSessionName = data.session.name || t('Session #') + id;
@@ -2914,8 +3046,8 @@ function onFilterChange() {
}
// ──────────────────── Versioning ────────────────────
const PWA_VERSION = '2.2.0';
const MIN_SERVER_VERSION = '2.2.0';
const PWA_VERSION = '2.3.0';
const MIN_SERVER_VERSION = '2.3.0';
function parseVersion(v) {
const parts = String(v || '').trim().split('.').map(n => parseInt(n, 10));
@@ -2957,6 +3089,20 @@ const LANG_RU = {
'Command Stats': 'Статистика команд',
'Stats': 'Статистика',
'Map': 'Обзор',
'Files': 'Файлы',
'Read': 'Чтение',
'Update': 'Запись',
'Changed (card)': 'Изменено (картой)',
'Card-side change': 'Изменение на карте',
'Close': 'Закрыть',
'Offset': 'Смещение',
'Record': 'Запись №',
'Before': 'Было',
'After': 'Стало',
'Show the change': 'Показать изменение',
'No file operations recorded': 'Операции с файлами не зафиксированы',
'Failed to load files': 'Не удалось загрузить список файлов',
'Loading…': 'Загрузка…',
'Reads:': 'Чтение:',
'Writes:': 'Запись:',
'Read/write:': 'Чтение/запись:',
@@ -3240,9 +3386,11 @@ function init() {
$on('comprion-cancel', 'click', closeComprionDialog);
$on('comprion-import', 'click', confirmComprionImport);
document.addEventListener('keydown', e => {
if (e.key === 'Escape'
&& el('comprion-modal').classList.contains('show')) {
if (e.key !== 'Escape') return;
if (el('comprion-modal').classList.contains('show')) {
closeComprionDialog();
} else if (el('file-diff-modal').classList.contains('show')) {
el('file-diff-modal').classList.remove('show');
}
});
@@ -3285,6 +3433,22 @@ function init() {
$on('install-btn', 'click', promptInstall);
$on('side-view-select', 'change', () => {
STATE.sideView = el('side-view-select').value;
renderSidePanel();
});
$on('file-view', 'click', e => {
const diff = e.target.closest('.file-op-diff');
if (diff) {
const parts = diff.dataset.diff.split('|');
showFileDiff(parts[0], Number(parts[1]));
return;
}
const row = e.target.closest('.file-op');
if (row) jumpToMessage(Number(row.dataset.idx));
});
$on('file-diff-close', 'click', () => el('file-diff-modal').classList.remove('show'));
$on('delete-cancel', 'click', () => {
el('delete-modal').classList.remove('show');
STATE.pendingDeleteId = null;
@@ -3292,12 +3456,6 @@ function init() {
$on('delete-confirm', 'click', confirmDelete);
el('side-panel').addEventListener('click', e => {
const tab = e.target.closest('.side-tab');
if (tab) {
STATE.sideView = tab.dataset.view;
renderSidePanel();
return;
}
const toggle = e.target.closest('.map-files-toggle');
if (toggle) {
e.stopPropagation();
frontend/sw.js
+1 -1
View File
@@ -1,4 +1,4 @@
const CACHE = 'simtrace-analyser-v62';
const CACHE = 'simtrace-analyser-v63';
const PRECACHE = [
'index.html',
frontend/tests/comprion.test.js
+63 -1
View File
@@ -124,7 +124,7 @@ test('the COMPRION import is a two-slot dialog', () => {
assert.match(src, /time_base === 'application-layer'/);
assert.match(src, /\$on\('import-comprion-btn', 'click', openComprionDialog\)/);
assert.match(src, /\$on\('comprion-cancel', 'click', closeComprionDialog\)/);
assert.match(src, /e\.key === 'Escape'/);
assert.match(src, /e\.key !== 'Escape'/);
});
test('comprionSlotCheck: slot/kind outcomes', () => {
@@ -275,3 +275,65 @@ test('COMPRION dialog: success closes, server errors stay', async () => {
closeComprionDialog();
});
// ── Files view (per-file operation timeline) ───────────────────────────
eval(
'function t(x) { return x; }\n' +
extractFunc(src, 'fileOpClass') + '\n' +
extractFunc(src, 'fileOpVerb') + '\n' +
extractFunc(src, 'fileOpLabel') + '\n' +
extractFunc(src, 'fileDiffRows')
);
test('Files view: operation kinds map to classes and verbs', () => {
assert.equal(fileOpClass('read'), 'file-op-read');
assert.equal(fileOpClass('update'), 'file-op-update');
assert.equal(fileOpClass('card_change'), 'file-op-card');
assert.equal(fileOpVerb('read'), 'Read');
assert.equal(fileOpVerb('update'), 'Update');
assert.equal(fileOpVerb('card_change'), 'Changed (card)');
assert.equal(fileOpLabel({kind: 'read', file: {name: 'EF_ICCID'}}), 'Read EF_ICCID');
assert.equal(fileOpLabel({kind: 'card_change', file: {name: 'EF_LOCI'}}),
'Changed (card) EF_LOCI');
assert.equal(fileOpLabel({kind: 'read', file: null}), 'Read ?');
});
test('Files view: the diff rows slice the concatenated hex', () => {
const change = {
ranges: [[8, 2], [16, 1]],
before_hex: '95f9aa',
after_hex: '000001',
};
assert.deepEqual(fileDiffRows(change), [
{pos: 8, len: 2, before: '95f9', after: '0000'},
{pos: 16, len: 1, before: 'aa', after: '01'},
]);
assert.deepEqual(fileDiffRows({ranges: [], before_hex: '', after_hex: ''}), []);
});
test('Files view: side-panel dropdown and the timeline are wired', () => {
assert.match(src, /<select class="side-view-select" id="side-view-select"[^>]*>/);
assert.match(src, /<option value="files" data-l10n="Files">/);
assert.doesNotMatch(src, /class="side-tab/);
assert.match(src, /id="file-view"/);
assert.match(src, /id="file-diff-modal"/);
assert.match(src, /id="file-diff-body"/);
assert.match(src, /\$on\('side-view-select', 'change'/);
assert.match(src, /\$on\('file-view', 'click'/);
assert.match(src, /API\.getSessionFiles\(STATE\.sessionViewId\)/);
assert.match(src, /if \(row\) jumpToMessage\(Number\(row\.dataset\.idx\)\);/);
assert.match(src, /getSessionFiles\(id\)\s*\{ return this\._fetch\('GET', `\/api\/sessions\/\$\{id\}\/files`\); \}/);
});
test('Files view: strings carry Russian translations', () => {
const lang = extractConst(src, 'LANG_RU');
const keys = new Set();
for (const m of lang.matchAll(/'((?:[^'\\]|\\.)*)'\s*:/g)) {
keys.add(m[1].replace(/\\(['\\])/g, '$1'));
}
for (const s of ['Files', 'Read', 'Update', 'Changed (card)', 'Card-side change',
'Close', 'Offset', 'Record', 'Before', 'After', 'Show the change',
'No file operations recorded', 'Failed to load files', 'Loading…']) {
assert.ok(keys.has(s), 'no LANG_RU entry for: ' + s);
}
});
pyproject.toml
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.backends._legacy:_Backend"
[project]
name = "simtrace-analyser"
version = "2.2.0"
version = "2.3.0"
description = "SIMtrace Analyser — APDU trace analyser for GSMTAP/PCAP captures (bundles the simtrace2-pysniff utility)"
readme = {file = "README.md", content-type = "text/markdown"}
license = {text = "GPL-2.0-or-later"}
simtrace_analyser/database.py
+20 -2
View File
@@ -192,6 +192,20 @@ class Database:
initial_states, clk_hz = self._replay_context(session_id, after_id + 1)
return self._decode_rows(rows, initial_states=initial_states, clk_hz=clk_hz)
def track_files(self, session_id):
"""Track file state over the whole session (see filetrack.py).
Returns the tracker's result dict: the operation timeline, per-file
counters/images/changes, the SFI relations and the AID labels.
"""
from .filetrack import FileTracker
rows = self._conn.execute(
'SELECT id, session_id, elapsed, type, data, flags FROM messages '
'WHERE session_id=? ORDER BY elapsed, id', (session_id,)).fetchall()
tracker = FileTracker()
self._decode_rows(rows, on_message=tracker.observe)
return tracker.result()
def get_messages_raw(self, session_id):
rows = self._conn.execute(
'SELECT elapsed, type, data FROM messages WHERE session_id=? ORDER BY elapsed, id',
@@ -300,7 +314,7 @@ class Database:
def _advance_state(self, d, sel, df, sfi_map):
"""Update (sel, df, sfi_map) for one decoded TPDU."""
from .decode import selected_df_fid, sfi_table, KNOWN_FIDS
new_df = selected_df_fid(d)
new_df = selected_df_fid(d, df)
if new_df is not None and new_df != df:
# Only a successful SELECT changes the current DF.
if (d.get('sw') or {}).get('sw1') in _SUCCESS_SW1:
@@ -363,7 +377,7 @@ class Database:
return new
return sel
def _decode_rows(self, rows, initial_states=None, clk_hz=None):
def _decode_rows(self, rows, initial_states=None, clk_hz=None, on_message=None):
from .decode import atr_rate_fields
msgs = []
states = dict(initial_states or {})
@@ -380,6 +394,10 @@ class Database:
ctx['sfi_map'] = st['sfi_map']
msg = self._row_to_message(row, prev=ctx, clk_hz=clk_hz)
d = msg.get('decoded')
if on_message is not None:
# Observers (the per-file tracker) see the message with the
# channel state as it was *before* this message.
on_message(msg, st, channel)
if msg['type'] == 'tpdu' and d and d.get('ins_hex'):
st['sel'], st['df'], st['sfi_map'] = self._advance_state(d, st['sel'], st['df'], st['sfi_map'])
st['prev'] = self._context_from_decoded(d)
simtrace_analyser/decode.py
+36 -9
View File
@@ -522,26 +522,53 @@ def _is_df_fid(fid):
return fid == '3f00' or fid.startswith('7f') or fid.startswith('5f')
def selected_df_fid(d):
"""Return the current DF after a SELECT, or None if the SELECT does not
(re)enter a DF. Handles SELECT by FID (P1 00/01), by path (P1 08/09),
and by DF name/AID (P1 04)."""
def selected_df_fid(d, current_df=None):
"""Return the current DF after a SELECT, or None if it is unchanged.
TS 102 221 §8.4.1: a DF/MF/ADF selection makes that file the current
directory; selecting an EF leaves the *parent of that EF* as the current
directory. So a path selection that ends in an EF also (re)enters the
DF the EF lives in (Table 8.2: ``'EF1'`` from MF, ``'7FFF||DF3||EF4'``,
…). ``current_df`` is used for "path from the current DF" paths.
Handles SELECT by FID (P1 00/01), by path (P1 08/09), and by DF name/AID
(P1 04; the special FID '7fff' always means the ADF of the current active
application, §8.3).
"""
if d.get('ins_hex') != 'a4':
return None
p1 = (d.get('p1') or {}).get('raw')
h = (d.get('body') or {}).get('hex') or ''
def resolve(fid):
# '7FFF' is the running application's ADF: when that application was
# selected by an unknown AID, keep the AID as the DF identity.
if fid == '7fff' and (current_df or '').startswith('aid:'):
return current_df
return fid
if p1 in ('00', '01'):
return h if len(h) == 4 and _is_df_fid(h) else None
return resolve(h) if len(h) == 4 and _is_df_fid(h) else None
if p1 in ('08', '09'):
comps = [h[i:i + 4] for i in range(0, len(h) - 3, 4)]
for comp in reversed(comps):
if _is_df_fid(comp):
return comp
if not comps:
return None
target = comps[-1]
if _is_df_fid(target):
return resolve(target)
# The selected file is an EF: the current directory is its parent.
for comp in reversed(comps[:-1]):
if _is_df_fid(comp):
return resolve(comp)
if p1 == '08':
return '3f00' # path from MF: directly under MF
return current_df or None # path from the current DF
if p1 == '04': # select by DF name (AID) → map known application AIDs
if h.startswith('a0000000871002'): # USIM AID
return '7fff'
return None
# Unknown application: keep the AID as the DF identity so files under
# it stay distinguishable from the previous DF's (TS 102 221 §11.1.1).
return f'aid:{h.lower()}' if h else None
return None
simtrace_analyser/filetrack.py
+483
View File
@@ -0,0 +1,483 @@
"""Per-file tracking over a recorded session (the Files view's data model).
Unlike the per-channel selection state in ``database.py`` — which only answers
"what is selected right now" and is wiped by every reset — this tracker keeps a
**trace-wide registry keyed by (DF, FID)**: access counters, a sparse image of
the observed contents, and the change log. It is built in a single pass over
the session's decoded messages (``Database.track_files``).
What is tracked
---------------
* **Counters** ``select``/``read``/``update``, successful operations only
(``database._SUCCESS_SW1``), unaffected by card resets.
* **Image** — the bytes we believe the file holds: reads and updates both fill
it (transparent files by offset, record files by record number). Only
*observed* bytes exist; never-read regions stay unknown.
* **Changes**
- *device* — an UPDATE (D6/DC) that modified bytes we already knew, applied
to the image and logged at its own APDU (``no_change`` flags a provable
no-op: every written byte was known and identical);
- *card* — a READ whose bytes differ from the image **on bytes we already
knew**; logged at the read that surfaced it, with the changed ranges and
the before/after bytes for the UI's diff. Bytes we never saw just fill the
image (an observation, not a change).
Registration happens on a successful SELECT **or** on any successfully
attributed read/update — including SFI-addressed operations. SFI → FID
relations come from the normative table (``decode._SFI_BY_DF``) and from every
surfaced FCI (the FCP's ``88`` tag); an SFI access that precedes its relation
gets a provisional ``(DF, 'SFI:n')`` entry which is merged into the resolved
``(DF, FID)`` entry when the relation is learned (counters, image and change
log move over — one file, one image, one change log, whatever the access
path).
Bounds: at most ``IMAGE_CAP`` observed bytes per file (the file is frozen
afterwards — comparing against a partial image would produce fake card-change
signals) and at most ``EVENTS_CAP`` change events per file (``changes_total``
keeps counting).
"""
from .database import _SUCCESS_SW1
from .decode import (_is_df_fid, select_target_fid, selected_df_fid, sfi_table)
IMAGE_CAP = 64 * 1024 # observed bytes kept per file
EVENTS_CAP = 1000 # change events kept per file
_READ_INS = (0xB0, 0xB1, 0xB2) # binary/record reads (SEARCH RECORD is not
_UPDATE_INS = (0xD6, 0xDC) # a content read — it returns record numbers)
_GR_READ_FOR = ('READ BINARY', 'READ RECORD')
def _success(d):
"""True when the message's status word means success (incl. 61xx)."""
sw1 = (d.get('sw') or {}).get('sw1')
return sw1 in _SUCCESS_SW1
def _body_bytes(d):
return bytes.fromhex((d.get('body') or {}).get('hex') or '')
def _file_label(d):
"""The display name of an operation's file (same vocabulary as the Map
view: the EF name, the FID, or the SFI placeholder)."""
f = d.get('file') or {}
if f.get('ef'):
return f['ef']
if f.get('fid'):
return f['fid'].upper()
if f.get('sfi') is not None:
return f'SFI {f["sfi"]}'
return None
def _key_str(key):
df, fid = key
return f'{df}/{fid}'
class _FileEntry:
"""One tracked file: counters, sparse image, change log."""
def __init__(self, key, df, fid, name, image_cap=IMAGE_CAP):
self.key = key
self.df = df
self.fid = fid
self.name = name
self.image_cap = image_cap
self.sfi = None
self.structure = None
self.size = None
self.record_len = None
self.n_records = None
self.counters = {'select': 0, 'read': 0, 'update': 0}
self.changes = []
self.changes_total = 0
self.image_truncated = False
# Observed contents: transparent files in one buffer indexed by the
# byte offset; record files per record number. The masks mark the
# bytes that came from an observation (everything else is unknown).
self.buf = bytearray()
self.mask = bytearray()
self.records = {}
self.rec_mask = {}
# -- image ----------------------------------------------------------
def _tracked_bytes(self):
return len(self.buf) + sum(len(r) for r in self.records.values())
def apply(self, record, offset, data):
"""Apply observed bytes.
Returns ``(diffs, compared_all)``: *diffs* is a list of
``(position, before, after)`` for the positions whose known value
differed (position = byte offset for transparent files, record number
for record files); *compared_all* tells whether every written byte was
already known (so a zero diff proves a no-op).
"""
if self.image_truncated:
return [], False
if record is None:
buf, mask, base = self.buf, self.mask, offset
needed = base + len(data)
else:
buf = self.records.setdefault(record, bytearray())
mask = self.rec_mask.setdefault(record, bytearray())
base = 0
needed = len(data)
if (needed > len(buf)
and self._tracked_bytes() + (needed - len(buf)) > self.image_cap):
self.image_truncated = True
return [], False
if len(buf) < needed:
buf.extend(b'\x00' * (needed - len(buf)))
mask.extend(b'\x00' * (needed - len(mask)))
diffs = []
compared_all = True
i = 0
while i < len(data):
pos = base + i
if not mask[pos]:
compared_all = False
buf[pos] = data[i]
mask[pos] = 1
i += 1
continue
if buf[pos] == data[i]:
i += 1
continue
start = pos
before = bytearray()
after = bytearray()
while i < len(data) and mask[base + i] and buf[base + i] != data[i]:
before.append(buf[base + i])
after.append(data[i])
buf[base + i] = data[i]
i += 1
diffs.append((start if record is None else record,
bytes(before), bytes(after)))
return diffs, compared_all
def coverage(self):
known = sum(self.mask) + sum(sum(m) for m in self.rec_mask.values())
return {'known': known, 'observed': self._tracked_bytes(),
'size': self.size}
class FileTracker:
"""Builds the per-file registry from a session's decoded messages."""
def __init__(self, image_cap=IMAGE_CAP, events_cap=EVENTS_CAP):
self._image_cap = image_cap
self._events_cap = events_cap
self.files = {} # key -> _FileEntry
self.timeline = [] # one item per read/update/card change
self.sfi_relations = {} # df -> {sfi: fid}
self.aid_labels = {} # AID (upper hex) -> application label
self._index = 0
self._pending_read = {} # channel -> {'d', 'df'} read awaiting its data
# -- observation ----------------------------------------------------
def observe(self, msg, state, channel=0):
"""Observe one decoded message; *state* is the channel state before it."""
index = self._index
self._index += 1
d = msg.get('decoded')
if msg.get('type') != 'tpdu' or not d or not d.get('ins_hex'):
return
ins_hex = d['ins_hex']
df = state.get('df') or '3f00'
if ins_hex == 'c0':
self._observe_get_response(d, state, channel, index, msg)
return
self._pending_read.pop(channel, None)
ins = int(ins_hex, 16)
if ins == 0xA4:
self._observe_select(d, df)
elif ins in _UPDATE_INS:
self._observe_write(d, df, index, msg)
elif ins in _READ_INS:
self._observe_read(d, df, channel, index, msg)
# -- file registry ---------------------------------------------------
def _entry(self, df, fid, name=None):
key = (df, fid)
entry = self.files.get(key)
if entry is None:
entry = _FileEntry(key, df, fid,
name or (fid.upper() if fid else '?'),
image_cap=self._image_cap)
self.files[key] = entry
elif name:
placeholders = {entry.fid, entry.fid.upper(), '', '?'}
if entry.sfi is not None:
placeholders.add(f'SFI {entry.sfi}')
placeholders.add(f'SFI:{entry.sfi}')
if entry.name in placeholders:
entry.name = name
return entry
def _resolve_sfi(self, df, sfi):
rel = self.sfi_relations.get(df, {})
if sfi in rel:
return rel[sfi]
return sfi_table(df).get(sfi)
def _learn_sfi(self, df, sfi, fid, entry=None):
self.sfi_relations.setdefault(df, {})[sfi] = fid
if entry is not None:
entry.sfi = sfi
self._merge_provisional(df, sfi, fid)
def _file_ref(self, d, df):
"""(fid-or-provisional-key, sfi) for an operation's file."""
f = d.get('file') or {}
sfi = f.get('sfi')
if sfi is None:
sfi = (d.get('p1p2') or {}).get('sfi')
if sfi is None:
sfi = (d.get('p2') or {}).get('sfi')
fid = f.get('fid')
if fid:
return fid, sfi
if sfi is not None:
resolved = self._resolve_sfi(df, sfi)
if resolved:
self._learn_sfi(df, sfi, resolved)
return resolved, sfi
return f'SFI:{sfi}', sfi
return None, None
def _merge_provisional(self, df, sfi, fid):
"""Fold a provisional (df, 'SFI:n') entry into the resolved one."""
provisional = self.files.get((df, f'SFI:{sfi}'))
if provisional is None:
return
del self.files[(df, f'SFI:{sfi}')]
real = self.files.get((df, fid))
if real is None:
provisional.key = (df, fid)
provisional.fid = fid
provisional.sfi = sfi
self.files[(df, fid)] = provisional
target = provisional
else:
real.sfi = sfi
for kind in ('select', 'read', 'update'):
real.counters[kind] += provisional.counters[kind]
_merge_image(real, provisional)
real.changes = sorted(real.changes + provisional.changes,
key=lambda c: c['index'])
real.changes_total += provisional.changes_total
target = real
for item in self.timeline:
if item.get('entry') is provisional:
item['entry'] = target
item['file'] = self._file_json(target)
def _file_json(self, entry):
return {'key': _key_str(entry.key), 'name': entry.name,
'fid': entry.fid, 'df': entry.df, 'sfi': entry.sfi}
def _push(self, kind, entry, index, msg, **extra):
item = {'kind': kind, 'entry': entry, 'index': index,
'msg_id': msg.get('id'), 'elapsed': msg.get('elapsed'),
'file': self._file_json(entry)}
item.update(extra)
self.timeline.append(item)
return item
def _add_change(self, entry, kind, index, msg, op, diffs, record=None):
entry.changes_total += 1
if len(entry.changes) >= self._events_cap:
return None
event = {
'kind': kind, 'op': op, 'index': index, 'msg_id': msg.get('id'),
'elapsed': msg.get('elapsed'), 'record': record,
'ranges': [[pos, len(before)] for pos, before, _after in diffs],
'before_hex': b''.join(b for _p, b, _a in diffs).hex(),
'after_hex': b''.join(a for _p, _b, a in diffs).hex(),
}
entry.changes.append(event)
return len(entry.changes) - 1
# -- operations ------------------------------------------------------
def _observe_select(self, d, df):
fid = select_target_fid(d)
if not fid or _is_df_fid(fid) or not _success(d):
return
# A path selection of an EF also enters the EF's parent DF
# (TS 102 221 §8.4.1) — key the file by that DF, not the one the
# channel was in before the SELECT.
effective = selected_df_fid(d, df) or df
self._entry(effective, fid).counters['select'] += 1
def _observe_get_response(self, d, state, channel, index, msg):
response = d.get('response') or {}
if d.get('response_for') == 'SELECT' and response:
# The FCP of the selected file: learn the SFI relation and the
# file facts (structure/size/records).
fid = (state.get('sel') or {}).get('fid')
if not fid or _is_df_fid(fid):
return
df = state.get('df') or '3f00'
entry = self._entry(df, fid)
fd = response.get('file_descriptor')
if isinstance(fd, dict):
entry.structure = fd.get('structure', entry.structure)
entry.record_len = fd.get('record_length', entry.record_len)
entry.n_records = fd.get('num_records', entry.n_records)
if response.get('file_size') is not None:
entry.size = response['file_size']
elif response.get('total_file_size') is not None:
entry.size = response['total_file_size']
if response.get('sfi') is not None:
self._learn_sfi(df, response['sfi'], fid, entry)
return
if d.get('response_for') in _GR_READ_FOR:
pending = self._pending_read.pop(channel, None)
# Prefer the data-bearing GET RESPONSE's own attribution; an
# SFI-addressed command keeps its context (the response's
# attribution resolves against the current selection, not the SFI).
src, src_df = d, state.get('df') or '3f00'
if pending:
_fid, sfi = self._file_ref(pending['d'], pending['df'])
if sfi is not None:
src, src_df = pending['d'], pending['df']
self._apply_read(src, src_df, index, msg, _body_bytes(d))
def _observe_read(self, d, df, channel, index, msg):
if not _success(d):
return
if not _body_bytes(d):
# Data pending (T=0): the GET RESPONSE will carry it.
self._pending_read[channel] = {'d': d, 'df': df}
return
self._apply_read(d, df, index, msg, _body_bytes(d))
def _apply_read(self, d, df, index, msg, data):
if not data:
return
fid, sfi = self._file_ref(d, df)
if fid is None:
return
entry = self._entry(df, fid, _file_label(d))
if sfi is not None and entry.sfi is None:
entry.sfi = sfi
entry.counters['read'] += 1
record = self._record_number(d)
diffs, _compared = entry.apply(record, self._offset(d, record), data)
if entry.image_truncated:
return
if diffs:
idx = self._add_change(entry, 'card', index, msg,
d.get('ins_name') or '', diffs, record=record)
self._push('card_change', entry, index, msg, record=record,
change_ref={'file': _key_str(entry.key), 'idx': idx}
if idx is not None else None)
else:
self._push('read', entry, index, msg, record=record)
self._note_dir_content(d)
def _observe_write(self, d, df, index, msg):
data = _body_bytes(d)
if not data or not _success(d):
return
fid, sfi = self._file_ref(d, df)
if fid is None:
return
entry = self._entry(df, fid, _file_label(d))
if sfi is not None and entry.sfi is None:
entry.sfi = sfi
entry.counters['update'] += 1
record = self._record_number(d)
diffs, compared_all = entry.apply(record, self._offset(d, record), data)
if entry.image_truncated:
return
if diffs:
self._add_change(entry, 'device', index, msg,
d.get('ins_name') or '', diffs, record=record)
self._push('update', entry, index, msg, record=record,
no_change=(not diffs and compared_all))
# -- helpers ---------------------------------------------------------
def _offset(self, d, record):
if record is not None:
return 0
return (d.get('p1p2') or {}).get('offset', 0)
def _record_number(self, d):
if d.get('ins_hex') not in ('b2', 'dc'):
return None
raw = (d.get('p1') or {}).get('raw')
return int(raw, 16) if raw is not None else None
def _note_dir_content(self, d):
"""Harvest AID -> label pairs from a surfaced EF.DIR."""
apps = (d.get('file') or {}).get('applications')
if not apps:
return
for app in apps:
aid = (app.get('aid') or '').upper()
if aid and app.get('label'):
self.aid_labels[aid] = app['label']
# -- result ----------------------------------------------------------
def result(self):
files = {}
for key, entry in self.files.items():
files[_key_str(key)] = {
'name': entry.name,
'df': entry.df,
'fid': entry.fid,
'sfi': entry.sfi,
'structure': entry.structure,
'size': entry.size,
'record_len': entry.record_len,
'n_records': entry.n_records,
'counters': dict(entry.counters),
'coverage': entry.coverage(),
'image_truncated': entry.image_truncated,
'changes_total': entry.changes_total,
'changes': entry.changes,
}
timeline = [{k: v for k, v in item.items() if k != 'entry'}
for item in self.timeline]
return {
'timeline': timeline,
'timeline_total': len(timeline),
'files': files,
'sfi_relations': {df: {str(k): v for k, v in m.items()}
for df, m in self.sfi_relations.items()},
'aid_labels': dict(self.aid_labels),
}
def _merge_image(target, source):
"""Merge *source*'s observed image into *target* (used when a provisional
SFI entry joins its resolved counterpart)."""
if len(target.buf) < len(source.buf):
target.buf.extend(b'\x00' * (len(source.buf) - len(target.buf)))
target.mask.extend(b'\x00' * (len(source.buf) - len(target.mask)))
for i, value in enumerate(source.buf):
if source.mask[i] and not target.mask[i]:
target.buf[i] = value
target.mask[i] = 1
for record, buf in source.records.items():
mask = source.rec_mask[record]
dst = target.records.setdefault(record, bytearray())
dst_mask = target.rec_mask.setdefault(record, bytearray())
if len(dst) < len(buf):
dst.extend(b'\x00' * (len(buf) - len(dst)))
dst_mask.extend(b'\x00' * (len(buf) - len(dst_mask)))
for i, value in enumerate(buf):
if mask[i] and not dst_mask[i]:
dst[i] = value
dst_mask[i] = 1
simtrace_analyser/server.py
+14
View File
@@ -141,6 +141,9 @@ class RequestHandler(BaseHTTPRequestHandler):
elif path.startswith('/api/sessions/') and path.endswith('/pcap'):
session_id = int(path.split('/')[-2])
self._handle_get_session_pcap(session_id)
elif path.startswith('/api/sessions/') and path.endswith('/files'):
session_id = int(path.split('/')[-2])
self._handle_session_files(session_id)
elif path.startswith('/api/sessions/'):
session_id = int(path.split('/')[-1])
self._handle_get_session(session_id, params)
@@ -461,6 +464,17 @@ class RequestHandler(BaseHTTPRequestHandler):
'warnings': result['warnings'],
})
def _handle_session_files(self, session_id):
"""Per-file tracking for a session (counters, changes, timeline)."""
session = self.db.get_session(session_id)
if session is None:
self._send_error(404, 'Session not found')
return
data = self.db.track_files(session_id)
_log(f'Session files: session={session_id} '
f'files={len(data["files"])} operations={data["timeline_total"]}')
self._send_json(data)
def _handle_rename_session(self, session_id, name):
self.db.rename_session(session_id, name)
self._send_json({'ok': True})
simtrace_analyser/version.py
+1 -1
View File
@@ -1,3 +1,3 @@
"""Single source of truth for the SIMtrace Analyser version."""
__version__ = '2.2.0'
__version__ = '2.3.0'
tests/test_decode.py
+26
View File
@@ -1690,11 +1690,37 @@ class TestSfiResolution(unittest.TestCase):
self.assertIsNone(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '00'},
'body': {'hex': '6f7e'}}))
self.assertIsNone(selected_df_fid({'ins_hex': 'b0'}))
# SELECT by AID: the USIM maps to its virtual FID, any other
# application keeps its AID as the DF identity.
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '04'},
'body': {'hex': 'a0000000871002ff'}}), '7fff')
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '04'},
'body': {'hex': 'A00000015141434C00'}}),
'aid:a00000015141434c00')
self.assertIsNone(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '04'},
'body': {'hex': ''}}))
# An AID DF has no normative SFI table.
self.assertEqual(sfi_table('aid:a00000015141434c00'), {})
self.assertEqual(sfi_table('7fff')[0x1e], '6fe3')
self.assertEqual(sfi_table('3f00')[0x1e], '2f00')
self.assertEqual(sfi_table('7fff')[0x17], '6f06')
self.assertEqual(sfi_table('7fff')[0x1a], '6fc6')
self.assertEqual(sfi_table('6f7e'), {})
# TS 102 221 §8.4.1: selecting an EF leaves its parent DF current.
# 'path from MF', single component → directly under MF (Table 8.2).
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '08'},
'body': {'hex': '2fe2'}}, '7fff'), '3f00')
# 'path from MF' via the ADF: the EF's parent is 7FFF.
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '08'},
'body': {'hex': '7fff6f07'}}, '3f00'), '7fff')
# 'path from current DF': a single EF stays in the current DF.
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '09'},
'body': {'hex': '6f7e'}}, '7fff'), '7fff')
# …and an unknown application keeps its AID identity for '7fff'.
self.assertEqual(selected_df_fid({'ins_hex': 'a4', 'p1': {'raw': '08'},
'body': {'hex': '7fff4f09'}},
'aid:a00000015141434c00'),
'aid:a00000015141434c00')
def test_record_p2_sfi(self):
# READ RECORD P2 = 0xBC → SFI = 0xBC>>3 = 23, mode = 0x04.
tests/test_filetrack.py
+419
View File
@@ -0,0 +1,419 @@
"""Tests for the per-file tracker (counters, image, changes, SFI relations)."""
import os
import tempfile
import unittest
from simtrace_analyser.database import Database
from simtrace_analyser.filetrack import FileTracker
def sel_fid(fid, le='02'):
"""SELECT by FID (no response data yet)."""
return f'00a4000002{fid}61{le}'
def gr(data_hex, total=None):
"""GET RESPONSE carrying *data_hex* (the FCP or the read data)."""
n = len(data_hex) // 2 if total is None else total
return f'00c00000{n:02x}{data_hex}9000'
def fcp(fid, sfi=None, structure=None, rec_len=None, n_rec=None, size=None):
"""A minimal FCP: file descriptor + FID (+ SFI / size TLVs).
*structure* is the TS 102 221 Table 11.5 nibble (default: transparent,
or linear fixed when record facts are given).
"""
if rec_len is not None and n_rec is not None:
b = 0x40 | 0b010 if structure is None else 0x40 | structure
body = '8205' + f'{b:02x}21{rec_len:04x}{n_rec:02x}'
else:
b = 0x40 | 0b001 if structure is None else 0x40 | structure
body = '8202' + f'{b:02x}21'
body += '8302' + fid
if sfi is not None:
body += '8801' + f'{sfi << 3:02x}'
if size is not None:
body += '8002' + f'{size:04x}'
return '62' + f'{len(body) // 2:02x}' + body
def read_binary(p1, p2, data_hex, sw='9000'):
return f'00b0{p1:02x}{p2:02x}{len(data_hex) // 2:02x}{data_hex}{sw}'
def read_binary_sfi(sfi, offset, data_hex, sw='9000'):
return f'00b0{0x80 | sfi:02x}{offset:02x}{len(data_hex) // 2:02x}{data_hex}{sw}'
def update_binary(p1, p2, data_hex, sw='9000'):
return f'00d6{p1:02x}{p2:02x}{len(data_hex) // 2:02x}{data_hex}{sw}'
def update_binary_sfi(sfi, offset, data_hex, sw='9000'):
return f'00d6{0x80 | sfi:02x}{offset:02x}{len(data_hex) // 2:02x}{data_hex}{sw}'
def read_record(rec, data_hex, sw='9000'):
return f'00b2{rec:02x}0404{data_hex}{sw}'
def update_record(rec, data_hex, sw='9000'):
return f'00dc{rec:02x}0404{data_hex}{sw}'
class TrackerCase(unittest.TestCase):
"""Runs a synthetic TPDU list through the real decode pipeline."""
def track(self, *tpdu_hex, tracker=None):
tmp = tempfile.NamedTemporaryFile(suffix='.db', delete=False)
tmp.close()
db = Database(tmp.name)
self.addCleanup(lambda: (db._conn.close(), os.remove(tmp.name)))
sid = db.create_session('test')
db.insert_messages(sid, [(float(i), 'tpdu', bytes.fromhex(h), 0)
for i, h in enumerate(tpdu_hex)])
return db.track_files(sid) if tracker is None else \
self._track_with(db, sid, tracker)
def _track_with(self, db, sid, tracker):
rows = db._conn.execute(
'SELECT id, session_id, elapsed, type, data, flags FROM messages '
'WHERE session_id=? ORDER BY elapsed, id', (sid,)).fetchall()
db._decode_rows(rows, on_message=tracker.observe)
return tracker.result()
class TestCountersAndRegistration(TrackerCase):
def test_select_read_update_counters(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '9807110900106014000000'[:22]),
update_binary(0, 0, '980711090010601495F9'),
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['counters'], {'select': 2, 'read': 1, 'update': 1})
self.assertEqual(entry['name'], 'EF_ICCID')
self.assertEqual(entry['size'], 10)
def test_failed_select_does_not_register(self):
data = self.track(sel_fid('2fe2', le='00').replace('6100', '6a82'))
self.assertEqual(data['files'], {})
self.assertEqual(data['timeline'], [])
def test_reset_keeps_the_registry(self):
from simtrace2_pysniff.gsmtap import GSMTAP_SIM_ATR
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '98' + '07' * 5),
)
gizmo = {'kind': 'read'}
# An ATR (card reset) does not clear the per-file registry.
tmp = tempfile.NamedTemporaryFile(suffix='.db', delete=False)
tmp.close()
db = Database(tmp.name)
self.addCleanup(lambda: (db._conn.close(), os.remove(tmp.name)))
sid = db.create_session('test')
db.insert_messages(sid, [
(0.0, 'tpdu', bytes.fromhex(sel_fid('2fe2')), 0),
(0.5, 'tpdu', bytes.fromhex(gr(fcp('2fe2', sfi=2, size=10))), 0),
(1.0, 'tpdu', bytes.fromhex(read_binary(0, 0, '98' + '07' * 5)), 0),
(2.0, 'atr', bytes.fromhex('3b9f96801fc78031e073fe2117574a3382033339001e'), 0),
(3.0, 'tpdu', bytes.fromhex(sel_fid('2fe2')), 0),
(3.5, 'tpdu', bytes.fromhex(gr(fcp('2fe2', sfi=2, size=10))), 0),
(4.0, 'tpdu', bytes.fromhex(read_binary(0, 0, '98' + '07' * 5)), 0),
])
data = db.track_files(sid)
self.assertEqual(data['files']['3f00/2fe2']['counters'],
{'select': 2, 'read': 2, 'update': 0})
self.assertEqual(len(data['timeline']), 2)
self.assertTrue(gizmo)
def test_channels_merge_into_one_file(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '9807110900'),
'01' + sel_fid('2fe2')[2:], # same file on channel 1
'01' + gr(fcp('2fe2', sfi=2, size=10))[2:],
'01' + read_binary(0, 0, '9807110900')[2:],
)
self.assertEqual(data['files']['3f00/2fe2']['counters']['read'], 2)
self.assertEqual(len(data['files']), 1)
class TestImageAndChanges(TrackerCase):
def test_read_fills_unknown_bytes_without_a_change(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '980711090010601495F9'),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['coverage']['known'], 10)
self.assertEqual(entry['changes_total'], 0)
self.assertEqual([i['kind'] for i in data['timeline']], ['read'])
def test_card_change_surfaced_by_a_read(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '980711090010601495F9'),
read_binary(0, 0, '98071109001060140000'),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['changes_total'], 1)
change = entry['changes'][0]
self.assertEqual(change['kind'], 'card')
self.assertEqual(change['ranges'], [[8, 2]])
self.assertEqual(change['before_hex'], '95f9')
self.assertEqual(change['after_hex'], '0000')
kinds = [i['kind'] for i in data['timeline']]
self.assertEqual(kinds, ['read', 'card_change'])
ref = data['timeline'][1]['change_ref']
self.assertEqual(ref['file'], '3f00/2fe2')
self.assertEqual(ref['idx'], 0)
# The image follows the card.
self.assertEqual(entry['coverage']['known'], 10)
def test_repeating_a_read_is_not_a_change(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '980711090010601495F9'),
read_binary(0, 0, '980711090010601495F9'),
)
self.assertEqual(data['files']['3f00/2fe2']['changes_total'], 0)
self.assertEqual([i['kind'] for i in data['timeline']], ['read', 'read'])
def test_device_change_from_an_update(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '980711090010601495F9'),
update_binary(0, 0, '98071109001060140000'),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['changes_total'], 1)
self.assertEqual(entry['changes'][0]['kind'], 'device')
self.assertEqual(entry['changes'][0]['before_hex'], '95f9')
self.assertEqual(entry['changes'][0]['after_hex'], '0000')
update_row = [i for i in data['timeline'] if i['kind'] == 'update'][0]
self.assertFalse(update_row['no_change'])
def test_identical_update_is_flagged(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '980711090010601495F9'),
update_binary(0, 0, '980711090010601495F9'),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['changes_total'], 0)
update_row = [i for i in data['timeline'] if i['kind'] == 'update'][0]
self.assertTrue(update_row['no_change'])
def test_update_of_unknown_bytes_learns_without_an_event(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
update_binary(0, 0, '9807110900'),
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['coverage']['known'], 5)
self.assertEqual(entry['changes_total'], 0)
update_row = [i for i in data['timeline'] if i['kind'] == 'update'][0]
self.assertFalse(update_row['no_change']) # cannot prove a no-op
class TestRecordFiles(TrackerCase):
def test_record_images_are_per_record(self):
data = self.track(
sel_fid('7fff'),
sel_fid('6f07'), gr(fcp('6f07', rec_len=8, n_rec=3)),
read_record(1, '0809101112131415'),
read_record(2, '1617181920212223'),
read_record(1, '08091011121314FF'), # record 1 changed
)
entry = data['files']['7fff/6f07']
self.assertEqual(entry['record_len'], 8)
self.assertEqual(entry['n_records'], 3)
self.assertEqual(entry['coverage']['known'], 16)
self.assertEqual(entry['changes_total'], 1)
self.assertEqual(entry['changes'][0]['ranges'], [[1, 1]])
def test_read_data_delivered_by_get_response(self):
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
'00b000000a610a', gr('980711090010601495F9'), # T=0 chaining
)
entry = data['files']['3f00/2fe2']
self.assertEqual(entry['counters']['read'], 1)
self.assertEqual(entry['coverage']['known'], 10)
# The timeline row points at the GET RESPONSE that carried the data.
self.assertEqual(data['timeline'][-1]['kind'], 'read')
def test_record_update_of_another_record(self):
data = self.track(
sel_fid('7fff'),
sel_fid('6f07'), gr(fcp('6f07', rec_len=8, n_rec=3)),
read_record(2, '1617181920212223'),
update_record(2, '1617181920212200'),
)
entry = data['files']['7fff/6f07']
self.assertEqual(entry['counters'], {'select': 1, 'read': 1, 'update': 1})
self.assertEqual(entry['changes'][0]['kind'], 'device')
self.assertEqual(entry['changes'][0]['ranges'], [[2, 1]])
class TestSfiResolution(TrackerCase):
def test_normative_sfi_read(self):
# EF_IMSI (6F07) has SFI 7 in the ADF_USIM table — no FCP needed.
data = self.track(
sel_fid('7fff'), # enter ADF_USIM
read_binary_sfi(7, 0, '0829103254769899'),
)
self.assertIn('7fff/6f07', data['files'])
entry = data['files']['7fff/6f07']
self.assertEqual(entry['counters']['read'], 1)
self.assertEqual(entry['coverage']['known'], 8)
self.assertEqual(entry['sfi'], 7)
def test_sfi_learned_from_the_fcp(self):
data = self.track(
sel_fid('7fff'),
sel_fid('6f07'), gr(fcp('6f07', sfi=7, size=9)),
read_binary_sfi(7, 0, '0829103254769899'),
)
self.assertEqual(list(data['files']), ['7fff/6f07'])
self.assertEqual(data['sfi_relations']['7fff']['7'], '6f07')
self.assertEqual(data['files']['7fff/6f07']['counters'],
{'select': 1, 'read': 1, 'update': 0})
def test_provisional_sfi_entry_is_merged_when_learned_later(self):
# The SFI access comes first (unknown relation), the SELECT with the
# FCP afterwards — one file, one image, one counter set.
data = self.track(
sel_fid('7fff'), # enter ADF_USIM
read_binary_sfi(7, 0, '08291032'), # SFI 7: resolved by the
read_binary_sfi(7, 4, '54769899AA'), # normative USIM table
sel_fid('6f07'), gr(fcp('6f07', sfi=7, size=9)),
read_binary_sfi(7, 9, ''),
)
self.assertEqual(list(data['files']), ['7fff/6f07'])
entry = data['files']['7fff/6f07']
self.assertEqual(entry['counters']['read'], 2)
self.assertEqual(entry['coverage']['known'], 9)
self.assertEqual(entry['changes_total'], 0)
self.assertTrue(all(i['file']['key'] == '7fff/6f07'
for i in data['timeline']))
def test_provisional_merge_for_an_unknown_table(self):
# An AID-DF has no normative table: the SFI read stays provisional
# until the file's FCP teaches the relation.
adf = '00a4040009a00000015141434c009000' # SELECT by AID
data = self.track(
adf,
read_binary_sfi(3, 0, 'AABBCCDD'),
sel_fid('4f09'), gr(fcp('4f09', sfi=3, size=4)),
read_binary_sfi(3, 0, 'AABBCCDD'),
)
keys = list(data['files'])
self.assertEqual(len(keys), 1)
self.assertIn('aid:a00000015141434c00', keys[0])
entry = data['files'][keys[0]]
self.assertEqual(entry['counters'], {'select': 1, 'read': 2, 'update': 0})
self.assertEqual(entry['coverage']['known'], 4)
self.assertEqual(entry['changes_total'], 0)
def test_unresolvable_sfi_stays_provisional(self):
data = self.track(
read_binary_sfi(20, 0, 'AABB'), # SFI 20: not in the MF table
)
self.assertEqual(list(data['files']), ['3f00/SFI:20'])
self.assertEqual(data['files']['3f00/SFI:20']['counters']['read'], 1)
self.assertEqual(data['files']['3f00/SFI:20']['coverage']['known'], 2)
class TestCapsAndLabels(TrackerCase):
def test_image_cap_freezes_the_file(self):
tracker = FileTracker(image_cap=8)
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=10)),
read_binary(0, 0, '9807110900'),
read_binary(5, 0, '10601495F9'),
tracker=tracker,
)
entry = data['files']['3f00/2fe2']
self.assertTrue(entry['image_truncated'])
self.assertEqual(entry['coverage']['known'], 5)
self.assertEqual(entry['counters']['read'], 2) # counters keep going
def test_event_cap_keeps_counting(self):
tracker = FileTracker(events_cap=1)
data = self.track(
sel_fid('2fe2'), gr(fcp('2fe2', sfi=2, size=4)),
read_binary(0, 0, 'AABBCCDD'),
read_binary(0, 0, 'AABBCC00'),
read_binary(0, 0, 'AABB0000'),
tracker=tracker,
)
entry = data['files']['3f00/2fe2']
self.assertEqual(len(entry['changes']), 1)
self.assertEqual(entry['changes_total'], 2)
def test_ef_dir_labels_are_harvested(self):
# EF_DIR record: application template (61) with AID (4F) + label (50).
aid = 'A0000000871002FF49FF0589'
label = '5553494D' # "USIM" in Annex A coding
content = f'4f{len(aid) // 2:02x}{aid}' + f'50{len(label) // 2:02x}{label}'
rec = '61' + f'{len(content) // 2:02x}' + content
data = self.track(
sel_fid('2f00'), gr(fcp('2f00', sfi=30, rec_len=32, n_rec=1)),
read_record(1, rec),
)
self.assertEqual(data['aid_labels'], {aid: 'USIM'})
def test_unattributed_operations_are_skipped(self):
# A read with no selection and no SFI has no file to attribute.
data = self.track(read_binary(0, 0, 'AABBCCDD'))
self.assertEqual(data['files'], {})
self.assertEqual(data['timeline'], [])
class TestEndpoint(unittest.TestCase):
def test_session_files_endpoint(self):
from simtrace_analyser.server import RequestHandler
class Fake(RequestHandler):
def __init__(self, db):
self.db = db
self._status = None
self._json = None
def _send_json(self, data, status=200):
self._status = status
self._json = data
def _send_error(self, status, message, code=None):
self._status = status
self._json = {'error': message}
tmp = tempfile.NamedTemporaryFile(suffix='.db', delete=False)
tmp.close()
db = Database(tmp.name)
self.addCleanup(lambda: (db._conn.close(), os.remove(tmp.name)))
sid = db.create_session('test')
db.insert_messages(sid, [
(0.0, 'tpdu', bytes.fromhex(sel_fid('2fe2')), 0),
(0.5, 'tpdu', bytes.fromhex(gr(fcp('2fe2', sfi=2, size=10))), 0),
(1.0, 'tpdu', bytes.fromhex(read_binary(0, 0, '980711090010601495F9')), 0),
])
h = Fake(db)
h._handle_session_files(sid)
self.assertEqual(h._status, 200)
self.assertEqual(h._json['timeline_total'], 1)
self.assertIn('3f00/2fe2', h._json['files'])
# Unknown session → 404.
h2 = Fake(db)
h2._handle_session_files(99999)
self.assertEqual(h2._status, 404)
if __name__ == '__main__':
unittest.main()
tests/test_pcap_import.py
+1 -1
View File
@@ -224,7 +224,7 @@ class TestImportEndpoint(unittest.TestCase):
class TestVersion(unittest.TestCase):
def test_package_version(self):
from simtrace2_pysniff import __version__
self.assertEqual(__version__, '2.2.0')
self.assertEqual(__version__, '2.3.0')
def test_status_includes_version(self):
import os