The purpose of numeric_base is to indicate for "random" values whether
to use decimal or hexadecimal digits. Indicate that TuakNumberOfKeccak
does not make sense to be randomized by not indicating a numeric_base.
Change-Id: I913878e3f05ad1e39ff45da75c67582a6a1f930f
Jenkins: skip-card-test
Even though the eUICC-Mandatory-services are always set by
saip.PES.rebuild_mandatory_services(), these new params are useful to
audit and validate the values as present in a given DER.
Change-Id: Icddeb2488c4a024c6ee5afcc1b6c8cc0e436c43c
Jenkins: skip-card-test
Because of (1) and (2) below, using enum.IntEnum is not a good choice
for EnumParam. Use a plain dict as value map.
EnumParam's idea is to be a mapping
human-readable-value-for-UI -> PES-value
and with enum.IntEnum we...
(1) ...we impose python namespace rules onto
human-readable-value-for-UI: "My Value" is not allowed, only "my_value".
By using enum.IntEnum, we make it impossible to use human readable
values like:
- "SUCI-on" (dash not allowed in python identifier)
- "True" (python keyword not allowed)
- "2" (numeric constant cannot be a python identifier)
(2) ...we limit PES-values to int. The mapping should support any type
that the given PES attribute needs (int, bool, bytes, ...).
Future: upcoming ConfigurableParameters depend on this patch:
- MncLen (labels "2" and "3")
- EuiccMandatoryServiceParam (values True and False)
- EfUstServiceParam like SuciInUsim (labels "SUCI-in-UE" and
"SUCI-in-USIM")
History: this is near the first version of EnumParam, before it was
modified by fixeria to use enum.IntEnum, as requested by CR. I realize
now that enum.IntEnum will not work for upcoming eSIM parameters.
Change-Id: I690ceccf0ec7ef7067bcaa5cec1303cdaf0f78a4
Jenkins: skip-card-test
Allow calling code to mute specific parameters from an output CSV.
For example, the SUCI CalcInfo data should be identical for all profiles
of a batch, and is also very long data. After this patch, an application
can name specific parameters to omit from an output CSV.
Change-Id: I5e567e59a007cf4b5d75a4dcea5371ff4404bf24
Jenkins: skip-card-test
Leave the decision for the record length to the template: callers can
can modify the value in the template without having to patch this code.
Change-Id: Id24752101ae82c4986209f4103cc9cbdcce8ce1d
Jenkins: skip-card-test
This helped in figuring out some bug -- show all available context info
in raised exceptions.
Change-Id: I38e85ce0902f2ec4db63e158770b6d2cecb2dd30
Jenkins: skip-card-test
Allow returning text/plain Content-Types as 'data' output argument.
So far, all the esim/http_json_api functions require a JSON response.
However, a specific vendor has a list function where the request is JSON
but the response is text/plain CSV data. Allow and return in a dict.
Change-Id: Iba6e4cef1048b376050a435a900c0f395655a790
Jenkins: skip-card-test
SJS1 model lifted off the the legacy
SysmoUSIMSJS1.program(), ADM is ADM1 with CHV 0x0a,
already used by the generic UiccCardBase default.
Change-Id: I32b011c68de5f990e709718b313b6f28a31ca78b
The color scheme takes either
- a raw escape sequence like "\033[33m" (pySim-read,
pySim-prog, contrib/csv-to-pgsql.py)
- a cmd2 color object (pySim-shell)
Color of cmd2 >= 3.0.0 is a StrEnum, which led to
pySim-shell warnings like
yellowWARNING: EF.DIR seems to be empty!
with a stray reset at the end.
Fixed by exclusing Enum from the raw sequence branch.
Change-Id: I8bae9fc3c54f1dab481d743d3b68e32e9d09267a
TS 102 223 6.8.7: TERMINAL RESPONSE shall contain the data object the
command qualifier (6.6.15) asked for.
The card always asks for location information, so at least answer 00,
with the GERAN form of TS 131.111 8.19.1, PLMN 262-01.
Do not answer "terminal currently unable to process - no service"...
Change-Id: I53547da2ea9f0a4ea7a4dd032985a546f2d4277f
pySim-smpp2sim sends "ff" * 32, that byte list tells every card the
terminal has a display, a keypad, a second card slot, a radio it can
query for location and NMR, five BIP bearers and six transport modes and
a toaster and a dog according to TS 102 223 5.2
We only have the twelfth byte and one bit of the seventeenth and no dog.
A card issues annoying weird things like PROVIDE LOCAL INFORMATION or
UDP only because the profile said so, so stop pretending we know what any
of that is.
Annex T table T.1 lists what a Connected Entity (a CAT client that is not
the modem) may announce. Announce that and the SMS-PP download and
SEND SHORT MESSAGE bits the OTA path needs, only the bearer is obviously
made up, it's the host TCP stack and 5.2 closest match is GPRS.
We can still extend and change all of this, but for now something that
works and constrains what the card asks for and is therefore actually
reproducible is important.
Change-Id: I91a60760fc3ad816b7385da8b26ec7330b462abd
handle_OpenChannel() raises where TS 102 223 demands a TERMINAL RESPONSE.
Raising takes the whole proactive session down with it.
All four are a BIP errors and differ only in the cause byte of 8.12.11.
Change-Id: If8f01e6acb9cb3f7af952a0760fc093db81be53a
Currently BcdAdapter(GreedyBytes), i.e. a plain swapped nibble digit
string, which is wrong, the IMEI IE is not a bcd string.
See TS 124.008 10.5.1.4 figure 10.5.4 + table 10.5.4
It currently
- decodes framing nibble as digit
- encodes a 15 digit IMEI as 7 bytes, wrong len, dropped last digit
Fix this, while at it add the 8.74 IMEISV as well + tests for the framing
because the current round trip missed all of this.
Change-Id: Iad7aea77513d413c338edc18ef1bcc37488ae849
TS 102 223 8.105 codes each supported technology in two bytes:
- the technology from 8.61
- state byte, b1 0 disabled/1 enabled
- b2-b8 RFU
Currently declared as FlagsEnum(Int8ub, enabled=0). kw value is used as
bitmask, so currently:
build {'enabled': True} -> b4 02 08 00 disabled
build {'enabled': False} -> b4 02 08 00 disabled
but parsed 08 00 / 08 01 / 08 ff -> all {'enabled': True}
Use the bit the spec names + tests because well this is so easy that we
don't need tests is apparently not the right approach here..
Change-Id: I4dd16182eb06c99265aabab5d889967b890f6bf2
pySim-smpp2sim currently claims to support every feature flag that exists,
but this immediately dies here with a SJA5:
NotImplementedError: No handler method for ProvideLocalInformation(...)
Answer anything unhandled with "performed_successfully" and log. Never
"command_beyond_terminal_capability", tho, answering that for PROVIDE LOCAL
INFORMATION refuses to open the session at all.
Change-Id: I1e55d6f88c872a04b89a3946dd840d30b329621e
Put a helper for wrapping an SMS-DELIVER TPDU in the ENVELOPE
of TS 102 223 section 7.5.1 the assembly next to SMSPPDownload,
where the IEs already live, so it can be reused by other tooling,
for example for triggering scp81 sessions.
Change-Id: Id23227eac53d697f4f13a84e087c32bf1d60f474
to_bytes() appended the BCD filler nibble, so the next call saw one
digit more and derived a wrong length. This affects any code that encodes
an address twice, even for printing/logging.
Man, I really miss my beloved c++ const function decoration...
Fix: keep the filler in a local.
Change-Id: I81691c5a1fc5072d6d20c52d22da1eb2e180d04a
Error output is currently a garbled mess, delimiters are printed with
poutput() or print(), traceback with traceback.print_exc() which goes
to stderr.
Reading stdout alone omits the report but not -->8-- lines.
Reading both streams makes the traceback appear where the two buffers
just happen to interleave, stderr is usually unbuffered but stdout is
not, so I get the error trace, APDU trace, then empty -->8-- lines.
Fix this by aligning the printing funcs.
Change-Id: I2ed37dfe241d014e4a26894cddbc9b21be1fd0c2
Right now APDUs that never return/fail are omitted from the trace output.
Move the print, so it actually tells me what failed..
Change-Id: I828ea36b06da36b6bb0aa073d235d4f49cbb3161
pySim has two APDU tracers: the StdoutApduTracer that --apdu-trace
installs in the transport for every program using pySim.transport, and
pySim-shell Cmd2ApduTracer for "set apdu_trace true", which prints with
cmd2. The command line option exists because the setting
applies too late to see the pySim startup APDUs.
The guard in init_reader() exists but is useless:
if opts.apdu_trace and not 'apdu_tracer' in kwargs:
no caller supplies a tracer that way, every tool calls init_reader()
without one?.
The current help string does not say "trace or maybe not lol" so set
the setting from the transport rather than hardcoding False, so the
option implies "set apdu_trace true", and it finally works as advertised.
Change-Id: I83469e15f4cdd67418cd11df3869f2536105dbc8
get_status() has appended a hardcoded '5c054f9f70c5cc' to the command data
field. Of the data objects in GP CS v2.3.1 Table 11-35 only the AID
search tag 4F is mandatory, the tag list is optional and not supported
in v2.1.1, where Section 9.4.2.3 defines the data field as the search
qualifier. Cards implementing that revision can reject anything else
with 6A80 as per v2.1.1 Table 9-26.
A sysmocom SJA5 does that. Its data field must be one 4F
TLV, the value is free, but nothing may precede or follow it.
So every subset returned nothing at all...
There is no need to guess: v2.1.1/v2.3.1 Section 7.4.1.3 Card Recognition
Data is "shall be present" and contains the GP version on selected SD.
Query it once, and send the tag list to cards that announce v2.2 or later.
SJA5 reports 2.1.1, sysmoEUICC reports 2.2.
Two more problems with the old list:
- A tag list is an inclusion list, old list omits tag 84, so it
suppressed the Executable Module AIDs
- It asks for tag C5 for Executable Load Files, which "may" be answered
with an error status.
Fix this by constricting or omitting the tag list depending on reported
GP version.
Change-Id: I74cd2bd47617d616bede6453397f544cde5abcb7
TS 102 221 section 7.3.1.1.4 clause 4b lets the card answer a case #4
command TPDU with a 62xx/63xx warning, upon which the terminal sends a
dummy GET RESPONSE to obtain the 61xx that announces the response length.
__send_apdu_T0() applies that unconditionally, to the status word that
terminates a GET RESPONSE...
That is not "redundant", as per GPC v2.3.1 section 11.4.3.2 table 11-38
GP GET STATUS (80 F2) answers 6310 "more data available", meaning reissue
the command as get next occurrence(s) with P2 bit 1 set (11.4.2.2 table
11-34), so for example a paginated registry
listing exchange looks like this:
84f22000024f00 => 61e4 first match, e4 bytes waiting
84c00000e4 => <page 1> 6310 more matches pending
84c0000000 => 6982 <- unsolicited, card rejects it
The card rejects the unsolicited GET RESPONSE with 6982, our SCP02
session breaks, and the next command fail with 6985. The 6310 never
reaches ADF_SD.get_status() either, so the pagination loop exits after
page 1 and prints a sliently truncated listing. Observed with a SJA5.
Fix by turning clause 4b würgaround into what it should be: a one shot
reaction to the SW returned for the command TPDU.
While at it, stop ADF_SD.get_status() from silently returning a truncated
registry: it treated every status word other than 6310 as
"nothing more to report". 6A88 is now the explicit empty result and anything
else raises, so a partial listing can no longer silently pass.
Adds unit tests for all of that so we dont break basic T0 things.
Change-Id: I10f8afa8dd5623a49a6a0e7132607b3a1fad2d8c
klein ignores host/port when endpoint_description is set, so -H has
no effect on the TLS listener, it binds every interface.
Append interface= to the string so -H means the same thing with and
without TLS.
Change-Id: I737c2e7cfa1ca7b825bd36a4a489760918031b8b
The ES9+ SM-DP+ address is used for the smdpAddress check and
for serverSigned1.serverAddress, both default to HOSTNAME.
Add --smdp-address (default: HOSTNAME) so the advertised address can carry a
port and still match what the LPA connects to when the TLS endpoint is bound
to other ports than 443. TLS certificate identity is unaffected, this is
only the ES9+ address.
SGP.22 defines both smdpAddress and serverSigned1.serverAddress as an FQDN.
Taken strictly that leaves no way to run an SM-DP+ on a port other than
443, so an address with a port is arguably outside the spec but needed
for testing.
Change-Id: I2fa822c7d5e0d5b68a6704a1779f9923505f9008
pyOpenSSL >= 25.0.0 makes a Context immutable once it has been used and
raises. Downgrading pyOpenSSL is not a fix either: < 25 does not import
against recent cryptography.
This server only speaks HTTP/1.1 anway so ALPN negotiation is unused
and this can be hotpatched for affected versions.
Change-Id: I5d53216f24a20625d12f0757015c19fe341303b9
GP Amendment B "Remote Application Management over HTTP" v1.2 and
ETSI TS 102 226 RFM/RAM server.
APDU command/response bodies use the Expanded Remote Application data
format.
--mode {ram,rfm} selects what we want to do, make sure to
target the right thing in the right mode, SD for RAM, AID for RFM.
This is hand-rolled on purpose:
Klein/Twisted do not help us here, have annoying headers and settings,
and don't work well at all if you want to import the file for tests and
so on. I have wasted quite some time trying to make this prettier by
re-using existing http handler parts because I thought well we import
all kinds of packages already it can't be that hard, unfortunately it is.
Change-Id: I4f243aeca65d0f33ff34e6932e3f4e156446af8b
Adds TS 102 226 tables 5.2a/5.10a indefinite length coding,
recommended by TS 102 226 5.2.1 for RAM/RFM over HTTPS.
Noteworthy notable things to note:
- encode_expanded_cmd()
indefinite version -> inner C-APDU TLVs are still definite
- decode_expanded_resp()s returned container does not care, but
Indef has no 'number of executed commands' TLV -> number_of_commands
is the R-APDU count.
Tests are being fed with some known-good values from my sja5 sessions.
Change-Id: I4e023112e98729489ed443eec3ed5ab45c773b17
The BIP relay ( the "handset" side for SCP81) never worked: the
connect callback in handle_OpenChannel was "never called" as the fixme
says, everything else was missing.
Fixme cause: card APDU I/O is driven synchronously, proactive command loop
lives in a blocking while loop (pySim.transport.LinkBase.send_apdu_checksw)
that runs on the Twisted reactor thread. A Twisted TCP4ClientEndpoint +
connectProtocol only completes when the reactor does reactor things,
but the reactor thread is stuck in that loop for the whole proactive
session...
Fixme fix: don't fight the reactor, just drive the relay channel with a plain
old blocking socket, which fits the synchronous execution model.
Channel numbers now come from the command Device identities (channel_N ->
low nibble) instead of the hard coded chan_nr == 1.
Additionally fix two bugs found on the path to scp81 glory:
- TERMINAL RESPONSE device identities are forced to terminal->UICC per
TS 102 223 6.8.2 (prepare_response() inverts the command identities,
which for a channel-addressed BIP command yields channel_N->UICC).
- Error responses now build a valid AddlInfoBip cause, prepare_response()
hard coded empty "additional information" cannot be encoded for a
BIP error.
And some tests based on real card interactions.
Change-Id: If96c768f2e35c20ea3753e601059410121517b60
A large OTA response (for example GP GET STATUS app registry) is split by the card
into multiple SMS, each carries a TS 23.040 9.2.3.24 'concatenated short messages'
IE in its UDH. Nothing recombines them, so smpp-ota-tool currently only sees the
first incomplete part.
Add ConcatenatedSmsReassembler that accepts TP-User-Data, buffers parts
by reference number, and returns the reassembled TP-User-Data in the canonical
single-part form. Non-concatenated SMS pass through unchanged.
Both the 8-bit+16-bit references are supported.
A reserved value in the concatenation IE is not an error, these messages
are handed back as is rather than rejected, so the caller can deal with that.
Reassembly leads to a result that looks like a fat single part message the
card could have produced given infinite sms sizes, so the existing decode_resp
path is unaffected.
Feeding those parts to the ota tool needs two more fixes because the card returns
the application response as several SMS via proactive SEND SHORT MESSAGE while
the ENVELOPE SMS-PP DOWNLOAD itself contains the POR without a app R-APDU.
smpplib poll() drains everything, so message_received_handler runs on each:
- a later status-only response must not overwrite an application response
already captured, or transceive_apdu finds no last_response_data and raises
- a response that cannot be decoded is be logged and skipped rather raised
out of client.poll() which kills the tool.
Plus tests from a sja5 session.
Incomplete sets are capped (oldest evicted) so they cannot pile up.
Change-Id: I8c81097e607e0d055c4f031bbcc8a74d5c24a0e7
A multi part OTA response (full GP GET STATUS registry or some other fat
response that exceeds one SMS) is delivered as several SMS via proactive
SEND SHORT MESSAGE. The card only gives us another part if it receives a
TERMINAL RESPONSE for the previous one.
Currently smpp2sim Proact.handle_SendShortMessage relays the SMS but
returns None, so the transport falls back to prepare_response(pcmd) with
the ProactiveCommand collection (empty .children) and crashes with
'not enough values to unpack (expected 1, got 0)', dropps the SMPP link,
and never fetches the remaining parts.
Fix: make handle_SendShortMessage return a successful TERMINAL RESPONSE
built from the decoded command so the handshake proceeds.
prepare_response() is extended to handle collection via .decoded
and raises a useful error.
The send_apdu_checksw general_result='FIXME' path is now avoided for
SendShortMessage but remains a problem for other handlers that return
None.
Change-Id: Ib96ce81c4ff093b8a6fc715f79617e95a2dd8433
since the very early days, pySim-shell.py has a commandline option
where the user may supply an ADM pin as commandline parameter.
(-a / --pin-adm and -A --pin-adm-hex) This was introduced to simplify
the usage of pySim-shell.py with shellscripts.
Unfortunately the code that handles those commandline options
duplicates the code of the verify_adm commmand. Fortunately it is
very easy to call pySim-shell commands directly using the CMD2
onecmd_plus_hooks method, so we can just call the verify_adm and
replace the duplicated code with that.
So far we are only able to use ADM1 pins from the commandline, since
we now practically use the verify_adm commnad, we can add another
parameter to allow the verification of ADM as well.
Related: SYS#8239
Change-Id: I7164fad757048774aa7186a84041febde75c351c
Add --format expanded flag to send C-APDUs in the expanded remote
application data format.
Each --apdu becomes its own C-APDU TLV, and the tool
logs the full per-command R-APDU list decoded from the Response Scripting
template and warns if the card reports a truncated response.
Default stays 'compact', unchanged.
Change-Id: Idad90756f85bb7e54ef720f6067bb5d6dcff0c42
TS 102 226 section 5.2 "Command and Response Scripting templates"
The advantage over compact RFM/RAM commands is one C-APDU TLV per command
and one R-APDU TLV per result, so multiple commands return the response
of each one rather than only that of the last.
encode_expanded_cmd() builds the Command Scripting template
decode_expanded_resp() decodes the Response Scripting template into a
Container.
The 'truncated' key must be checked!
OtaDialect.encode_cmd()/decode_resp() now has a remote_format param for
compact and expanded formats, default stays compact,so existing code is
unaffected.
Change-Id: Idec00d16fd1a7d4a7129b2a3b6f0ef37dabcecb7
Move the code from pySim-smpp2sim.py to its own file, so it can be properly
extended.
The current file parses argv, opens a reader and starts the Twisted
reactor at import time, so nothing else can import it.
No functional changes yet, improvements follow in later commits.
Change-Id: Ifd8a15684939977d29ea83a6b669daee14484e88
PySimLogger.setup() installs a process-global print callback.
PySimLogger_Test sets one, a helper that asserts the message equals a global
expected_message, and never removes it, so from the moment test_log runs,
every PySimLogger message emitted anywhere in the process is checked against
whatever string that global happens to hold.
Fortunately unittest discovery runs modules in sorted order, and today
the PySimLogger users that log during tests all sort before test_log, so
this only breaks as soon as I try to add tests, just like anything else
breaks as soon as I try to use it.
Change-Id: I481e2c443fe0f412380b0f1acf6da5971ffca147
SCP.overhead was so far set at construction time (SCP02: 8, SCP03:
s_mode), so the C-MAC length only.
Unfortunately sec lvl >= 3 pads the data field to the cipher block size
before encryption, so the real worst-case overhead is larger,
scc.max_cmd_len (255 - overhead) was too big, and ADF_SD.load()
used a hardcoded chunk_len=240.
Real world issue with a 286 byte CAP + SCP02 + sec lvl 3:
- 240-byte LOAD block is padded to 248,
- encrypted
- gets 8 byte C-MAC appended
-> Lc = 256
That dies with a weird "ValueError: bytes must be in range(0, 256)".
The only "fix" for that was to downgrade the seclevel.
STORE DATA has the same overflow with large max_cmd_len
(247 + padding + MAC = 256 as well).
Therefore the overhead must be properly calculated from the sec level.
While at it adjust the error in case I missed something to get a more
useful ValueError.
Change-Id: Ic208f3959a38896f64fb6ccefb24cc360a3ac3a2
how encrypt_key() pads a kcv:
len(key) % blocksize bytes
what it should do to actually do it right:
blocksize - len(key) % blocksize
so the plaintext handed to the cipher was only block aligned by luck as
long as the key length happened to be a multiple of half the block size.
And of course decrypt_key() did not invert encrypt_key() at all,
the clear text length of GP CardSpec v2.3 Table 11-70 precedes the
ENCRYPTED kcv, but it was parsed out of the DECRYPTED data, and the
length byte itself was fed to the cipher along with the cryptogram.
Fix this up with a helper and tests so it is actually usable.
Change-Id: I02b4f2ed948c31e1741e40f0226fb49757fa2570
while len(foo):
throws an exception when foo == None.
Instead doing
while foo:
fixes a problem when reading in empty SUCI calc info data, e.g. from
TS48v7.0_SAIP2.3_BERTLV_SUCI_NoRAMRFM.der.
Change-Id: Ia4e2356d0241d7a6ca399ba7e8be7f27ec836104
Jenkins: skip-card-test
The tests take long and don't need to be this many.
A patch is coming up that adds another profile to test SUCI in GFM,
this patch makes some room for that.
Change-Id: Ib75b6919a3acfddd99bf9baa9b6847ef731b9e67
Jenkins: skip-card-test
Neels Hofmeyr