Commit Graph
100 Commits
Author SHA1 Message Date
Philipp Maier f9e4291a43 pySim/ara_m, cosmetic: swap --nfc-always and --nfc-never options
The commandline options --nfc-always and --nfc-never appear in the
opposite order when compared to --apdu-never and --apdu-always.

Let's swap the options to make the helpscreen and the code more
consistent.

Change-Id: I7289c3628b1b8dd3eec2f1c8f2132e3015422960
Related: SYS#6959
2026-05-13 15:49:20 +02:00
Philipp Maier 20538775b2 pySim/scp: migrate to pySimLogger
The module scp.py predates the existence of the pySimLogger and still
uses an individually created logger. Let's migrate to pySimLogger to
avoid unexpected effects and to be uniform with the other modules.

Related: SYS#6959
Change-Id: I5db7180f93f116dd2d99c33da264f74ea16a1a37
2026-05-08 17:54:33 +02:00
Philipp Maier ef58c94dfe pySim/filesystem: use pySimLogger instead of print
let's replace the stray print statements with proper logger calls.

Related: SYS#6959
Change-Id: I3a7188ad33706df66b2113e15cc7d06004c9bc39
2026-05-08 17:54:33 +02:00
Philipp Maier 810c51c38f pySim/app: use pySimLogger instead of print
let's replace the stray print statements with proper logger calls.

Related: SYS#6959
Change-Id: I95b4536cc8853e7ba6a5dd573b903dfb85e56b9a
2026-05-08 17:54:33 +02:00
Philipp Maier 66d3b54f92 pySimLogger: fix default log format string
In format string we prepend when we log in verbose mode. We use %(module)s
as format string quaifier. This qualifier is replaced with the name of the
module from where the logger was called. This is mostly equal to the logger
name (__name__) we pass when we create the logger.

However, this is not the behavior we actually want. We want to log the
logger name that we passed when the logger was created. For this, we must
use %(name)s as qualifier.

Related: SYS#6959
Change-Id: I3951a70ad6ce864a7158b093cba46ae9fc1cb5bd
2026-05-08 17:54:33 +02:00
Philipp Maier 7d11f91778 card_key_provider: add a static method to parse --column-keys args
The contents of the --column-keys arguments are currently parsed
in init_card_key_provider. Let's add a static method in
CardKeyFieldCryptor to simplify re-usage of the CardKeyFieldCryptor

Related: SYS#6959
Change-Id: Ic955f271b1de1b1b855b21c82ed10343044e45fa
2026-05-08 17:54:33 +02:00
Philipp Maier 58a324126e card_key_provider: pass CardKeyFieldCryptor to constructor
We currently create the CardKeyFieldCryptor object inside the constructor
of the concrete CardKeyProvider classes. There is currently no problem
with that, but when we create the CardKeyFieldCryptor object first and
then pass it as parameter to the constructor, we gain more flexibility
in case we want to support other CardKeyFieldCryptor variants in the
future.

Related: SYS#6959
Change-Id: If43552740aadacab9126f8a002749a9582eef8f4
2026-05-08 17:54:33 +02:00
Philipp Maier 3cd5c41fb4 card_key_provider: move boiler-plate code into helper functions
in pySim-shell.py we add the commandline options for the card key
provider and do the setup accordingly. Let's put this boilerplate
code into helper functions instead, so that we can re-use it in
other pySim programs as well. Let's use pySim.transport as a
pattern.

Related: SYS#6959
Change-Id: I6d095cbb644e608f4a751a1d0749b1484cdc781d
2026-05-08 17:54:33 +02:00
Philipp Maier 593bfa0911 ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701
2026-04-29 19:39:14 +00:00
Philipp Maier 8fa7727a14 pySim-prog/cards: fix programming of EF.SMSP
The legacy code found in legacy/cards.py does not use the modern
construct based encoder (pySim-read uses it). The card classes either
use their own implementation of update_smsp or use the generic method
provided by the SimCard class. The latter one is true for FairwavesSIM
and WavemobileSim.

Unfortunately the implementation found in the SimCard is wrong. It
adds padding at the end of the file instead of the beginning. This
completely messes up the contents of EF.SMSP for the cards using this
method. To fix this, let's use the leftpad feature provided by
the update_record. This will ensure a correct alignment of the file
contents.

Related: SYS#7765
Change-Id: Ie112418f1f1461762d61365d3863181ca6be7245
2026-04-29 19:39:14 +00:00
Philipp Maier f1609424de pySim/transport: fix GET RESPONSE behaviour
The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc
2026-04-29 19:34:27 +00:00
Philipp Maier c50f4b4a02 requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)
PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows
arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+
patches this, the dependency specification (pyyaml >= 5.1) doesn't
guarantee a safe version. Let's increase the requirement to version
5.4 to ensure a safe version of is used.

This patch is based on suggestions from:
"YanTong C <chyeyantong03@gmail.com>"

Change-Id: I901c76c59e9c1bab030eab81038e04a475b32510
2026-04-16 11:01:19 +00:00
Philipp Maier d5aa963caa pysim/pcsc: do not use getProtocol for protocol selection
The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963
2026-04-02 12:53:30 +02:00
Philipp Maier ca8fada7b6 pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument
pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74
2026-03-31 11:29:46 +00:00
Philipp Maier c995bb1ec2 pySim-shell/cosmetic: remove semicolon
Change-Id: I629bacd432491211b939fcd2bed554b44ef441bc
2026-03-31 11:29:46 +00:00
Philipp Maier ee06ab987f PySimLogger: add parameter to set initial log-level/verbosity
When we initialize a new PySimLogger, we always call the setup method
first and then use the set_verbose and set_level method to configure
the initial log level and the initial log verbosity. However, we
initialize the PySimLogger in all our programs the same way and we
end up with the same boilerplate code every time. Let's add a keyword
parameter to the setup method where we can pass our opts.verbose (bool)
parameter so that the setup method can do the work for the main program.

In case the caller wants a different default configuration he still can
call set_verbose and set_level methods as needed.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8
2026-03-31 11:29:46 +00:00
Philipp Maier a1d3b8f5e8 pySim-read: remove import random
In pySim-read we do not have to compute any random numbers, so
we may remove random from the imports

Change-Id: Iae4ee6aafb339cc682345299b92b4ecd0bbca14e
2026-03-31 11:46:28 +02:00
Philipp Maier add4b991b7 transport: change APDU format paradigm
Unfortunately we have mixed up the concept of TPDUs and APDUs in
earlier versions of pySim-shell. This lead to problems with
detecteding the APDU case properly (see also ISO/IEC 7816-3) and
also prevented us from adding support for T=1.

This problem has been fixed long time ago and all APDUs sent from
the pySim-shell code should be well formed and valid according to
ISO/IEC 7816-3.

To ensure that we continue to format APDUs correctly as APDUs (and
not TPDUs) we have added a mechanism to the LinkBase class that
would either raise an exception or print a warning if someone
mistakenly tries to send an APDU that is really a TPDU. Whether a
warning is printed or an exception is raised is controlled via the
apdu_strict member in the LinkBase class, which is false (print
warning only) by default.

The reason why we have implemneted the mechanism this way was
because we wanted to ensure that existing APDU scripts (pySim-shell
apdu command) keep working, even though when those scripts uses
APDUs which are formally invalid.

Sending a TPDU instead of an APDU via a T=0 link will still work
in almost all cases. This is also the reason why this problem
slipped through unnoticed for long time. However, there may still
be subtile problems araising from this practice. The root of the
problem is that it is impossible to distinguish between APDU case
3 and 4 when a TPDU instead of an APDU is sent. However in order
to handle a case 4 APDU correctly we must be able to distinguish
the APDU case correctly to handle the case correctly.
ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about
the fact that not (only) the status word (e.g. 61xx) but the
APDU case is what matters.

To complete the logic in LinkBaseTpdu and to maintain compatibility
(older APDU scripts), we must still be able to switch between the
'apdu_strict' mode and the non-strict mode. However, since
pySim-shell, pySim-prog and pySim-read internally use proper APDUs,
we may enable the 'apdu_strict' mode by default.

At the same time we will limit the effect of pySim-shell's
apdu_strict setable to the apdu command only. By doing so, the
bahviour of the apdu command is not altered. Users will still
have to enable the 'strict' mode explicitly. At the same time
all the internal functionality of pySim-shell will always use
the 'strict' mode.

Related: OS#6970
Change-Id: I9a531a825def318b28bf58291d811cf119003fab
2026-03-30 10:26:13 +00:00
Philipp Maier 8c81e2cdf9 docs/put_key: add tutorial that explains how to manage global platform keys
With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e
2026-03-25 18:07:32 +00:00
Philipp Maier 969f9c0e4b pySim/EF.SMSP: fix encoding of TP-Destination Address
The TP-Destination Address in EF.SMSP uses the same encoding as the
TS-Service Centre Address field. However, even though the encoding
of both fields looks almost identical, it actually isn't.

The TS-Service Centre Address field encodes the length field as
octets required for the call_number + one octet for ton_npi.
(see also: 3GPP TS 24.011, section 8.2.5.2)

The TP-Destination Address uses the number of digits of the
call_number directly in the length field.
(see also: 3GPP TS 23.040, section 9.1.2.5)

Related: SYS#7765
Change-Id: I55c123c9e244e5a6e71a0348f5d476ef03e618e8
2026-03-25 15:34:56 +01:00
Philipp Maier 2ef9abf23e pySim/EF.SMSP: add an additional de_encode test for EF_SMSP
Let's add another testvector where we test what happens when we populate
none of the fields except for the tp_sc_addr.

Related: SYS#7765
Change-Id: I12b600ab17d1acfdddaffe6006095acf1a4228c9
2026-03-24 15:55:24 +00:00
Philipp Maier 473f31066c pySim/pcsc/cosmetic: reformat comment
Change-Id: Ic04bdfbc6727cc670679c377c1afd1de53504b8f
2026-03-23 19:03:19 +00:00
Philipp Maier b59363b49e pySim/EF.SMSP: remove superflous line break
Change-Id: Ie02e02546e708e2c339810812188bd8e8af2a720
2026-03-23 16:44:38 +01:00
Philipp Maier 0634f77308 esim/http_json_api: allow URL rewriting
The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a rewrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5
2026-03-17 11:17:12 +01:00
Philipp Maier 914abe3309 docs/smpp-ota-tool: Add documentation/tutorial
We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49
2026-03-10 09:23:03 +00:00
Philipp Maier 84754b6ebb contrib/smpp-ota-tool: define commandline arguments in global scope
The commandline arguments are currently defined under __main__ in a
private scope. From there they are not reachable to the sphinx
argparse module. We have to define the arguments globally at the
top. (like in the other applications)

Related: SYS#7881
Change-Id: I2d9782e3f5b1cac78c22d206fdcac4118c7d5e7c
2026-03-10 09:23:03 +00:00
Philipp Maier c47005d408 contrib/smpp-ota-tool: use '-' instead of '_' in command line args
Some commandline arguments have an underscore in their name. Let's
replace those with dashes.

Change-Id: Icbe9d753d59263997e9ca34d46ed0daca36ca16c
Related: SYS#6868
2026-03-10 09:23:03 +00:00
Philipp Maier 2dfaac6e4f contrib/smpp-ota-tool: fix description string (copy+paste error)
Change-Id: I559844bfa1ac372370ef9d148f2f8a6bf4ab4ef5
Related: SYS#6868
2026-03-10 09:23:03 +00:00
Philipp Maier a615ba5138 tests/pySim-smpp2sim_test: add testcases for AES128 and AES256
Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868
2026-03-10 09:23:03 +00:00
Philipp Maier 8ee10ab1a5 tests/pySim-smpp2sim_test/card_sanitizer: update card backup with new test keyset
In our test setup we run the card_sanitizer.py script regualary to ensure that
we have consistent start conditions when running our tests. In case a testcase
crashes for some reason and leaves messed up files on a test card. The
card_sanitizer.py script will ensure that any problem like that is cleaned up
over night.

For the testcases we are about to add in the patch following this one, we need
to provision a new test keyset to one of our test cards. This has been already
done manually. However since the card_sanitizer still has the old keys in its
backup we will have to update that as well.

Change-Id: I5aa8a413b19b3e43a79d03e904daab50b4b1e767
Related: OS#6868
2026-03-10 09:23:03 +00:00
Philipp Maier f10af30aed global_platform/scp: fix dek_encrypt/dek_decrypt for SCP02
The methods dek_encrypt/dek_decrypt use the wrong algorithm and the
wrong key material. The algorithm should be 3DES rather then single
DES and the key must be the DEK session key instead of the static
DEK key from which the DEK session key is derived.

Related: SYS#7902
Change-Id: I3d0cc7378680b346fa39152c8b7074446d2c869d
2026-03-06 15:51:19 +01:00
Philipp Maier b6b4501e37 contrib/smpp-ota-tool: fix boolean commandline parameters
Boolean parameters should be false by default and use store_true when
set.

Change-Id: I0652b48d2ea5efbaaf5bc147aa8cef7ab8b0861d
Related: OS#6868
2026-02-24 09:52:48 +01:00
Philipp Maier 54658fa3a9 contrib/smpp-ota-tool: add missing usage helpstrings
Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868
2026-02-24 09:52:48 +01:00
Philipp Maier eb04bb1082 contrib/smpp-ota-tool: warn about mixed up KIC/KIC indexes
Cards usually have multiple sets of KIC, KID (and KIK). The keys
are selected through an index. However, mixing keys from different
sets is concidered as a security violation and cards should reject
such configurations.

Let's print a warning to make users aware that something is off.

Change-Id: Ieb4e14145baba1c2cb4a237b612b04694940f402
Related: OS#6868
2026-02-24 09:52:48 +01:00
Philipp Maier 453fde5a3a contrib/smpp-ota-tool: use correct kid index
(normally KID index and KIC index should be the same since mixing keys
is a concidered as a security violation. However, in this tool we
want to allow users to specify different indexes for KIC and KIC so that
they can make tests to make sure their cards correctly reject mixed up
key indexes)

Change-Id: I8847ccc39e4779971187e7877b8902fca7f8bfc1
Related: OS#6868
2026-02-17 15:24:25 +01:00
Philipp Maier 57237b650e contrib/smpp-ota-tool/cosmetic: use lazy formatting for logging
Change-Id: I2540472a50b7a49b5a67d088cbdd4a2228eef8f4
Related: OS#6868
2026-02-17 15:24:25 +01:00
Philipp Maier 1f94791240 contrib/smpp-ota-tool/cosmetic: fix sourcecode formatting
Change-Id: Icbce41ffac097d2ef8714bc8963536ba54a77db2
Related: OS#6868
2026-02-17 15:24:25 +01:00
Philipp Maier 1a28575327 pySim-shell_test/euicc: ensure test-profile is enabled
When testing commands like get_profile_info, enable_profile,
disable_profile or the commands to manage notifications, we
should ensure that the correct profile is enabled before
executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74
2026-02-17 09:22:42 +00:00
Philipp Maier e80f3160a9 pySim/euicc: fix encoding/decoding of Iccid
The class Iccid uses a BcdAdapter to encoded/decode the ICCID. This
works fine for ICCIDs that have an even (20) number of digits. In case
the digit count is odd (19), the ICCID the last digit requires padding.

Let's switch to PaddedBcdAdapter for encoding/decoding, to ensure that
odd-length ICCIDs are padded automatically.

Change-Id: I527a44ba454656a0d682ceb590eec6d9d0ac883a
Related: OS#6868
2026-02-10 13:26:45 +00:00
Philipp Maier 8b2a49aa8e esim/http_json_api: add alternative API interface (follow up)
This is a follow up patch to change:
I2a5d4b59b12e08d5eae7a1215814d3a69c8921f6

- do not ignore length of kwargs
- fix role parameter (roles other than 'legacy_client' can be used now)
- use startswith instead of match

Related: SYS#7866
Change-Id: Ifae13e82d671ff09bddf771f063a388d2ab283eb
2026-02-10 13:42:44 +01:00
Philipp Maier f9d7c82b4d esim/http_json_api: add alternative API interface
unfortunately the API changes introduced in change

I277aa90fddb5171c4bf6c3436259aa371d30d092

broke the API interface of http_json_api.py. This was taken into
account and necessary to introduce add the server functionality next
to the already existing client functionality. The changes to the API
were minimal and all code locations that use http_json_api.py
were re-aligned.

Unfortunately it was not clear at this point in time that there are
out-of-tree projects that could be affected by API changes in
http_json_api.py

To mitigate the problem this patch introduces an alternative API
interface to the JsonHttpApiFunction base class. This alternative
API interface works like the old API interface when the class is
instantiated in the original way. To make use of the revised client
the API use has to pass an additional keyword argument that defines
the role.

Related: SYS#7866
Change-Id: I2a5d4b59b12e08d5eae7a1215814d3a69c8921f6
2026-02-09 12:42:28 +00:00
Philipp Maier 39d744010a pySim-shell_test/euicc: fix testcase method name
We have two test_enable_disable_profile method, the second one should
be called test_set_nickname.

Change-Id: I5ff79218fdafc8c42c8b58cc00be3e56e09d808b
2026-02-09 10:10:08 +01:00
Philipp Maier 15691233e1 tests/pySim-smpp2sim_test: add integration test
At the moment pySim.ota codebase is not covered by any of the
integration tests (we have only normal unittests so far). To
increase the test coverage, let's add an integration test that
sends exchanges an RFM OTA-SMS with a real-world card.

However, there is no tool avaliable that can be used as an SMPP
client for pySim-smpp2sim yet. Let's use smpp_ota_apdu2.py on
laforge/ota to develop a tool that we can use to exchange SMS-TPDUs
that contain remote APDU scripts (RFM/RAM).

Finally let's use the tool we have created as a basis to create
an integration test that exchanges an SMS-TPDU with the RFM
application of a sysmoISIM-SJA5 card. The testcase shall pass
when we get the expected response from the card.

Related: OS#6868
Change-Id: If25e38be004cc1c7aeeb130431831377e78fe28d
2026-02-04 14:05:07 +00:00
Philipp Maier 0a1c5a27d7 esim/http_json_api: add missing apidoc
Change-Id: Ibf9cf06197c9e3203c7a3ea5d77004f0ca41cd3f
2026-02-04 12:45:58 +01:00
Philipp Maier e62f160775 contrib/csv-to-pgsql: add missing copyright header
Change-Id: Iad8b2c1abb6a80764d05c823fbd03a9eae0ec0ab
2026-01-31 01:32:27 +00:00
Philipp Maier a4895702d7 transport/init: use PySimLogger to print messages
The module still uses print to output information. Let's replace
those print calls with the more modern PySimLogger method calls.

Change-Id: I2e2ec2b84f3b84dbd8a029ae9bb64b7a96ddbde3
2026-01-28 12:19:54 +01:00
Philipp Maier 2b42877389 pySimLogger: user __name__ of the module when creating a new logger
At the moment we use random identifiers as names when we create a
new logger for pySimLogger. Let's switch to consistently use the
module name here. For the top level modules let's use the program
name so that it will show up in the log instead of __init__.

Change-Id: I49a9beb98845f66247edd42ed548980c97a7151a
2026-01-28 12:19:54 +01:00
Philipp Maier 0a36ba257c pySim/runtime: use log.warning instead of log.warn
The python logger method warn is deprecated since pyton 3.3, let's us
the warning method as suggested.

Change-Id: I3a4c0ca43768198ac6011ebe79050f91c04862e5
2026-01-26 15:21:37 +00:00
Philipp Maier 1f36c9c28a contrib: add utility to receive ES2+handleDownloadProgressInfo calls
We already have a tool to work with the ES2+ API provided by an SMDP+
(es2p_client.py) With this tool we can only make API calls towards
an SMDP+. However, SGP.22 also defines a "reverse direction" ES2+
interface through wich the SMDP+ may make API calls towards the MNO.

At the moment the only possible MNO originated API call is
ES2+handleDownloadProgressInfo. Let's add a simple tool that runs a
HTTP server to receive and log the ES2+handleDownloadProgressInfo
requests.

Related: SYS#7825
Change-Id: I95af30cebae31f7dc682617b1866f4a2dc9b760c
2026-01-22 19:38:01 +01:00
Philipp Maier e00c0becca esim/http_json_api: extend JSON API with server functionality
At the moment http_json_api only supports the client role. Let's also add
support for the server role.

This patch refactors the existing client code. This in particular means
that the following preperations have to be made:

- To use the existing JsonHttpApiFunction definitions in the client and
  server the scheme has to be symetric. It already is for the most part,
  but it treads the header field differently. So let's just treat the
  header field like any other mandatory field and add it input_params.
  (this does not affect the es9p.py code since in ES9+ the requests have
   no header messages, see also SGP.22, section 6.5.1.1)

- The JsonHttpApiFunction class currently also has the code to perform
  the client requests. Let's seperate that code in a JsonHttpApiClient
  class to which we pass an JsonHttpApiFunction object.

- The code that does the encoding and decoding in the client role has
  lots of conditions the treat the header differently. Let's do the
  decisions about the header in the JsonHttpApiClient. The encoder
  and decoder function should do the generic encoding and decoding
  only. (however, some generic header specific conditions will remain).

The code for the server role logically mirrors the code for the client
role. We add a JsonHttpApiServer class that can be used to create
API endpoints. The API user has to pass in a call_handler through which
the application logic is defined. Above that we add an Es2pApiServer
class in es2p. In this class we implement the logic that runs the
HTTP server and receives the requests. The Es2pApiServer supports all
ES2+ functions defined by GSMA SGP.22. The user may use the provided
Es2pApiServerHandler base class to define the application logic for each
ES2+ function.

Related: SYS#7825
Change-Id: I277aa90fddb5171c4bf6c3436259aa371d30d092
2026-01-22 19:38:01 +01:00
Philipp Maier 148d0a6f90 esim/http_json_api: add missing check
The line actual_sec = func_ex_status.get('statusCodeData', None) suggests
that 'statusCodeData' may be None under normal circumstances. So let's guard
sec.update(actual_sec) so that we won't run into an exception in case
'statusCodeData' is not in func_ex_status.

Related: SYS#7825
Change-Id: I8a1a3cd5e029dba4a3aec1a64702e19b0d694ae2
2026-01-22 18:51:16 +01:00
Philipp Maier fddab8639f card_key_provider: add PostgreSQL support
The Card Key Provider currently only has support for CSV files
as input. Unfortunately using CSV files does not scale very well
when the card inventory is very large and continously updated.
In this case a centralized storage in the form of a database
is the more suitable approach.

This patch adds PostgreSQL support next to the existing CSV
file support. It also adds an importer tool to import existing
CSV files into the database.

Change-Id: Icba625c02a60d7e1f519b506a46bda5ded0537d3
Related: SYS#7725
2026-01-12 10:57:27 +01:00
Philipp Maier eb7c5d85d0 runtime/cosmetic: add line break
The other source files have a line break between the character encoding
qualifier line and the python comment. Let's add a line break here
as well to maintain consistency.

Change-Id: Ied6b77eede748f1ddf6fde17c9b434fa4dd1114a
2026-01-06 15:03:53 +01:00
Philipp Maier eda6182edd transport/init/cosmetic: move copryight header to the top
The copyright header seems to be misplaced, let's move it to the top.

Change-Id: I8358cca3bc9adb5a186a8b38a3bd90d7aec60d5c
2026-01-06 15:00:49 +01:00
Philipp Maier 4c8a9478c2 cosmetic: fix company name in copyright header.
The correct abbreviated version of the company name is
"sysmocom - s.f.m.c. GmbH", i.e. lowercase and with dash.

Change-Id: Id768d2f4b78162ff83320a800e4e66f1bd324d6d
2026-01-06 21:41:23 +00:00
Philipp Maier dfe4d9c8ac contrib: add a tool to parse the SIMA response from an eUICC
When an eUICC performs a profile installation it returns a (concatenated)
series of ASN.1 encoded strings as "simaResponse". In case the profile
installation fails for some reason the simaResponse contains diagnostic
information to diagnose why the profile installation failed.

Unfortunately there are currently no practical tools available to decode
and display the information in the simaResponse. Let's add a tool for that.

Related SYS#7617

Change-Id: Ida4c3c5446653b283a3869c0c387f328ae51e55e
2026-01-06 21:41:23 +00:00
Philipp Maier 8e048820d4 pySim-shell: renovate version command
In case pySim-shell is used directly from the git repository (not
installed via a package manager), the version command fails with an
exception because pkg_resources.get_distribution('pySim') fails.

Let's renovate the version command and migrate from pkg_resources to
importlib.resources. There are many users and developers out there who
retrieve pySim-shell directly from the git repository and not via pip3.
To accommodate for that, let's check if pySim-shell.py is located in a
git repository and if so, let's display the HEAD commit hash instead.

Since the version of the currently installed pyosmocom version also
plays a critical role, let's display the pyosmocom version as well.

Related: OS#6830
Change-Id: I2b9038f88cfcaa07894a2f09c7f5ad8a5474083d
2026-01-06 21:11:16 +00:00
Philipp Maier c2ace3d8cf unittests/test_utils: add unittests for enc_imsi and dec_imsi
So far we seem to have no unittests for enc_imsi and dec_imsi.

Change-Id: Iae55485c5ec7763aa4aaa25fd1910b854adaab60
2026-01-06 21:10:40 +00:00
Philipp Maier d764659a30 pySim-shell: do not show user home path in help text
At the moment, the help text for the --csv option shows the path to
the users home. This is due to the default value, which is dynamically
generated. Let's use a static string with "~/" and resolve the full
path later when we need it.

Related: SYS#7725
Change-Id: Ied8b1e553de8f5370369c4485a2360906c874ed2
2026-01-05 14:54:26 +00:00
Philipp Maier 3ca25219bc pySim-shell/cosmetic: remove unnecessary brackets
Change-Id: I6929aa4fa189414217c05a4ef5180d4ed44eb3a4
2026-01-05 14:42:49 +00:00
Philipp Maier 3d42106ad9 pysim/log: also accept ANSI strings to specify the log message colors
the PySimLogger class currently only accepts cmd2 color enum values.
This is what we need for pySim-shell.py. However, in case we want to
use the PySimLogger in stand-alone programs that do not use cmd2, this
is a bit bulky. Let's add some flexibility to PySimLogger, so that we
can specify the colors as raw ANSI strings as well.

Change-Id: I93543e19649064043ae8323f82ecd8c423d1d921
Related: SYS#7725
2025-12-19 16:12:31 +01:00
Philipp Maier 3f3f4e20e2 docs/conf.py: update copyright year
The copyright year of the docs is still at 2023, let's update it
to the current year.

Change-Id: Icf64670847d090a250f732d94d18e780e483239b
2025-11-25 17:14:54 +01:00
Philipp Maier c2fb84251b card_key_provider: add missing type annotation
Related: SYS#7725
Change-Id: I45751d2b31976c04c03006d8baa195eef2576b4f
2025-11-21 17:49:09 +01:00
Philipp Maier 61541e7502 card_key_provider: refactor code and optimize out get_field method
The method get_field in the base class can be optimized out. This
also allows us to remove code dup in the card_key_provider_get_field
function.

Let's also fix the return code behavior. A get method in a
CardKeyProvider implementation should always return None in case
nothing is found. Also it should not crash in that case. This will
allow the card_key_provider_get function to move on to the next
CardKeyProvider. In case no CardKeyProvider yields any results, an
exception is appropriate since it is pointless to continue execution
with "None" as key material.

To make the debugging of problems easier, let's also print some debug
messages that inform the user what key/value pair and which
CardKeyProvider was queried. This will make it easier to investigate
in case an expected result was not found.

Related: SYS#7725
Change-Id: I4d6367b8eb057e7b2c06c8625094d8a1e4c8eef9
2025-11-21 17:49:09 +01:00
Philipp Maier 579214c4d0 card_key_provider: remove method _verify_get_data from base class
The method _verify_get_data was intended to be used to verify the
user input before it further processed but ended up to be a simple
check that only checks the name of the key column very basically.

Unfortunately it is difficult to generalize the check code as the
concrete implementation of those checks is highly format dependent.
With the advent of eUICCs, we now have two data formats with
different lookup keys, so a static list with valid lookup keys is
also no longer up to the task.

After all it makes not much sense to keep this method, so let's
remove it.

(From the technical perspective, the key column is not limitied to
any specif field. In theory it would even be possible to use the KI
as lookup key as well, even though it would not make sense in
practice)

Related: SYS#7725
Change-Id: Ibf5745fb8a4f927397adff33900731524715d6a9
2025-11-21 17:49:09 +01:00
Philipp Maier 4a7651eb65 pySim-shell: re-organize Card Key Provider related options
As we plan to support other formats as data source for the Card Key
Provider soon, the more commandline options may be added and it makes
sense to group the Card Key Provider options in a dedicated group.

Let's also rename the option "--csv-column-key" to just "--column-key".
The column encryption is a generic concept and not CSV format specific.
(let's silently keep the "--csv-column-key" argument so maintain backward
compatibility)

Related: SYS#7725
Change-Id: I5093f8383551f8c9b84342ca6674c1ebdbbfc19c
2025-11-21 17:49:09 +01:00
Philipp Maier 01a6724153 pySim-shell: add command to manually query the Card Key Provider
The Card Key Provider is a built in mechanism of pySim-shell which
allows the user to read key material from a CSV file in order to
avoid having to lookup and enter the key material himself. The
lookup normally done by the pySim-shell commands automatically.

However, in some cases it may also be useful to be able to query the
CSV file manually in order to get certain fields displayed. Such a
command is in particular helpful to check and diagnose the CSV data
source.

Related: SYS#7725
Change-Id: I76e0f883572a029bdca65a5a6b3eef306db1c221
2025-11-21 17:49:09 +01:00
Philipp Maier a6ca5b7cd1 card_key_provider: remove unnecessary class property definitions
The two properties csv_file and csv_filename are defined by the
constructor anyway, let's remove the declaration in the class body
because it is not needed.

Change-Id: Ibbe8e17b03a4ba0041c0e9990a5e9614388d9c03
2025-11-21 17:49:09 +01:00
Philipp Maier bcca2bffc2 card_key_provider: rename parameter filename to csv_filename
let's rename the parameter filename to csv_filename to make it
more clear to what kind of file this parameter refers.

Change-Id: Id5b7c61b5e72fb205e30d2787855b2c276840a7b
2025-11-21 17:49:09 +01:00
Philipp Maier e80f96cc3b card_key_provider: use case-insensitive field names
It is common in CSV files that the columns have uppercase names, so we
have adopted this scheme when we started using the card_key_provider.

This also means that the API of the card_key_provider_get() and
card_key_provider_get_field() function now implicitly requires
uppercase field names like 'ICCID', 'ADM1', etc.

Unfortunately this may be unreliable, so let's convert the field
names to uppercase as soon as we receive them. This makes the API
case-insensitive and gives us the assurance that all field names
we ever work with are in uppercase.

Related: SYS#7725
Change-Id: I9d80752587e2ccff0963c10abd5a2f42f5868d79
2025-11-21 17:49:09 +01:00
Philipp Maier 4550574e03 card_key_provider: separate and refactor CSV column encryption
The CardKeyProviderCsv class implements a column decryption scheme
where columns are protected using a transport key. The CSV files
are enrcypted using contrib/csv-encrypt-columns.py.

The current implementation has two main problems:

- The decryption code in CardKeyProviderCsv is not specific to CSV files.
  It could be re-used in other formats, for example to decrypt columns
  (fields) red from a database. So let's split the decryption code in a
  separate class.

- The encryption code in csv-encrypt-columns.py accesses methods and
  properties in CardKeyProviderCsv. Also having the coresponding
  encryption code somewhere out of tree may be confusing. Let's improve
  the design and put encryption and decryption functions in a single
  class. Let's also make sure the encryption/decryption is covered by
  unittests.

Related: SYS#7725
Change-Id: I180457d4938f526d227c81020e4e03c6b3a57dab
2025-11-21 17:49:09 +01:00
Philipp Maier 08565e8a98 pySim-shell: use log level INFO by default
The default log level of the PySimLogger is DEBUG by default. This is
to ensure that all messages are printed in an unconfigured setup.

However in pySim-Shell we care about configuring the logger, so let's
set the debug log level to INFO in startup. This will allow us to
turn debug messages on and off using the verbose switch.

Change-Id: I89315f830ce1cc2d573887de4f4cf4e19d17543b
Related: SYS#7725
2025-11-21 17:49:09 +01:00
Philipp Maier 784cebded4 card_key_provider: add unit-test
There is no unit-test for the CardKeyProviderCsv class yet. Let's add
one to ensure that the CardKeyProviderCsv class keeps working as expected.

Related: SYS#7725
Change-Id: I52519847a4c4a13a7bca49985133872b01c4aaab
2025-11-18 15:47:40 +01:00
Philipp Maier 4f75aa1c8f card_key_provider: fix sourcecode formatting
Change-Id: I5675f9f087086646937ca077d3545d2729ccd812
2025-11-18 14:08:42 +01:00
Philipp Maier 94811ab585 pySim-shell: allow user friendly selection of the pin type
The CHV commands (verify_chv, enable_chv, disable_chv, unblock_chv)
provide a --pin-nr parameter.

The --pin-nr is a decimal parameter that specifies the pin type to be
used. The exact pin type numbers are specified in ETSI TS 102.221,
Table 9.3.

Unfortunately the --pin-nr parameter is not very intuitive to use, it
it requires the user to manually lookup the numeric value. The specs
list that value as hexadecimal, so the user also has to convert it
to decimal. To make this less complicated, let's also accept
hexadecimal numbers with the --pin-nr parameter.

However, this alone does not improve the user expierience much. Let's
also add a --pin-type parameter (similar to the --adm-type parameter
of the verify_adm command) to specifiy the pin type in a human
readable form.

Change-Id: I0b58c402d95cbc4fe690e6edb214829d463e9f2c
2025-11-01 14:03:23 +00:00
Philipp Maier f3e6e85f99 osmo-smdpp: update documentation
osmo-smdpp has built-in SSL/TLS support for quite some time now. The manual does not
yet mention this feature yet.

Change-Id: I2db5ae32914386a34eab1ed7d2aff8cae82bfa9b
2025-10-31 16:31:45 +01:00
Philipp Maier 7f2cb157c8 osmo-smdpp: update commandline help and default port
osmo-smdpp has built-in TLS support for some time now. Let's update
update the commandline help to be more concise.

Since the built-in SSL/TLS support is enabled by default, let's also
update the default port from 8000 to 443.

Change-Id: Ib5a069a8612beb1a9716a7514b498ec70d141178
2025-10-31 16:31:40 +01:00
Philipp Maier f94f366cf9 runtime: check record/file size before write
When writing data to a transparent or linear fixed (record oriented)
and the data to write exceeds the record/file size, then the UICC will
respond with an error "6700: Checking errors - Wrong length"

In particular when the data is supplied as a JSON object and not as a
hex string, it may not be immediately obvious to the average user what
the problem actually is.

Let's check the record/file size before writing the data and raise an
exception in case the data excieeds the record/file size. Let's also
print an informative string message in case the data length is less
than the record/file size to make the user aware of unwritten bytes
at the end of a record/file.

Related: OS#6864
Change-Id: I7fa717d803ae79398d2c5daf92a7336be660c5ad
2025-10-28 13:39:35 +01:00
Philipp Maier 4429e1cc70 pySim-shell: add a logger class to centralize logging
In many sub modules we still use print() to occassionally print status
messages or warnings. This technically does not hurt, but it is an unclean
solution which we should replace with something more mature.

Let's use python's built in logging framework to create a static logger
class that fits our needs. To maintain compatibility let's also make sure
that the logger class will behave like a normal print() statement when no
configuration parameters are supplied by the API user.

To illustrate how the approach can be used in sub-modules, this patch
replaces the print statements in runtime.py. The other print statements
will the be fixed in follow-up patches.

Related: OS#6864
Change-Id: I187f117e7e1ccdb2a85dfdfb18e84bd7561704eb
2025-10-25 19:46:34 +00:00
Philipp Maier 1ab2f8dd9d commands: do not use b2h with a string
The function h2b expects a bytearray and must not be used on a string.
This is also true for nullstrings ('').

Related: OS#6869
Change-Id: I0e28e6ec476901bf19aa0f8640e41c74aa6e3aa2
2025-10-21 17:17:21 +02:00
Philipp Maier d5da431fd4 saip-tool: add commandline option to edit mandatory services list
Change-Id: I120b98d4b0942c26674bc1365c5711101ec95235
2025-04-16 10:35:15 +02:00
Philipp Maier 59faa02f9a ara_m: add command to lock write access to the ARA-M rules.
Recent versions of the ARA-M applet from Bertrand Martel can lock
the write access to ARA-M rules. Let's add a command for that and
some documentation.

Related: SYS#7245
Change-Id: I71581a0c9f146f9a0921093d9b53b053b4a8946c
2025-04-14 11:14:36 +00:00
Philipp Maier 1dea0f39dc saip-tool: add features to add, remove and inspect application PEs
The PE-Application object is used to provision JAVA-card applications
into an eUICC during profile installation. Let's extend the SAIP-tool
so that we are able to add, remove and inspect applications.

Change-Id: I41db96f2f0ccc29c1725a92215ce6b17d87b76ce
2025-04-14 11:01:24 +00:00
Philipp Maier 40e795a825 saip-tool: add ProfileElement class for application PE
The application profile element has no ProfileElement class yet, so
let's create a ProfileElementApplication class and move the existing
extract-apps code into a method of ProfileElementApplication.

Change-Id: Iaa43036d388fbf1714c53cab1fc21092c4667a21
2025-03-31 12:27:24 +02:00
Philipp Maier dc2b9574c9 saip-tool: allow removing of profile elements by type
At the moment it is only possible to remove profile elements by their identification
number. However, there may be cases where we want to remove all profile elements of
a certain type at once (e.g. when removing all applications).

Change-Id: I92f9f9d5b4382242963f1b3ded814a0d013c4808
2025-03-28 14:35:40 +01:00
Philipp Maier 2b3b2c2a3b saip-tool: add option to extact profile elements to file
In some cases it may be helpful to extract a single profile element
from the sequence to a dedicated file.

Change-Id: I77a80bfaf8970660a84fa61f7e08f404ffc4c2da
2025-03-28 14:34:55 +01:00
Philipp Maier 02a7a2139f saip-tool: add function to write PE sequence
To prevent code duplication and to make the implementation simpler,
let's add a function that takes care of writing the PE sequnece
to an output file.

Change-Id: I38733422270f5b9c18187b7f247b84bf21f9121b
2025-03-28 13:25:30 +00:00
Philipp Maier 8da8b20f58 es8p: fix typo
Change-Id: I241efe0c7ceab190b7729a6d88101501ca37652e
2025-03-10 19:16:20 +00:00
Philipp Maier 74be2e202f filesystem: do not decode short TransRecEF records
A TransRecEF is based on a TransparentEF. This means that a TransRecEF
is basically normal TransparentEF that holds a record oriented data
structure. This also requires that the total length of the TransRecEF
is a multiple of the record length of the data structure that is stored
in it. When this is not the case, the last record will be cut short and
the decoding will fail. We should guard against this case.

Related: OS#6598
Change-Id: Ib1dc4d7ce306f1f0b080bb4b6abc36e72431d3fa
2025-03-10 18:59:08 +00:00
Philipp Maier c4a6b8b3e7 pySim-shell: obey quit command in startup commands+scripts
Startup scripts are executed using the cmd2 provided onecmd_plus_hooks
method. This method can run arbitrary commands, which also includes
the command "run_scrit" that we use to execute startup scripts.

When a script executes a quit command, or when someone issues a quit
command using the --execute-command or the command argument, then
this commands is executed. However a quit command won't actually quit
the process. All it does is to change the return code of
app.onecmd_plus_hooks (see [1]). So we must evaluate the return code
and take care of the quitting ourselves.

[1] https://cmd2.readthedocs.io/en/0.9.15/api/cmd.html#cmd2.cmd2.Cmd.onecmd_plus_hooks

Related: OS#6731
Change-Id: Ic6e9c54cdb6955d65011af3eb5a025eee5da4143
2025-02-25 14:55:49 +01:00
Philipp Maier edcd62435d pySim/transport: add abstract get_atr method to LinkBase
The implementations that inheret from the LinkBase class are expected to
implement a get_atr method. This method is mandatory, since it is one of
the most basic functionalities of pySim to display an ATR. Also the ATR
is sometimes needed to distinguish between different card models.

The modem_atcmd and calypso implementation completely lack the get_atr
method. Apparantly it is not possible to get an ATR in those
environments, so lets add a dummy method there.

Related: OS#6322
Change-Id: I4fc020ca45658af78e495a5c1b985213f83cbb50
2025-01-29 13:35:44 +01:00
Philipp Maier d871e4696f ATR: use lowercase hex strings without spaces as ATR constants
The ATR constants are the only hex string constants where the hex
bytes digits are separated with spaces. Also the hex digits are in
lowercase. Let's use a lowercase string without spaces here like
we do in many other code locations.

Related: OS#6322
Change-Id: I95118115b02523ed262a2fbe4369ace3996cd8f5
2025-01-29 11:26:54 +01:00
Philipp Maier 15140aae44 global_platform: add new command "install_cap"
Installing JAVA-card applets from a CAP file is a multi step process, which is
difficult when done manually. Fortunately it is easy to automate the process,
so let's add a dedicated command for that.

Change-Id: I6cbd37f0fad5579b20e83c27349bd5acc129e6d0
Related: OS#6679
2025-01-22 16:46:32 +01:00
Philipp Maier f688d28107 global_platform: fix usage of the Key Version Number (kvn)
The kvn parameter is used to select a keyset when establishin a secure channel.
At the moment this is a mandatory parameter and it must be within a certain
range.

However GPC_SPE_034 explicitly defines a reserved kvn value 0, that always
refers to the first available key. That effectively makes it an optional
parameter and the commandline interface should have the --key-ver parameter
as an optional parameter.

The ranges also have to be extended to allow 0 as kvn value. We also have to
put a range to support the sysmoUSIM-SJS1, which uses kvn value 1, which is
a non standard value.

Related: OS#6679
Change-Id: I42be2438c7f199b238f2ec7a9434cec5393210a7
2025-01-15 15:02:46 +01:00
Philipp Maier 712946eddb javacard: add parser for JAVA-card CAP file format
To install JAVA-card applets we need to be able to extract the executeable
loadfile and the AIDs of the applet and the loadfile. This patch adds the
parser and related unittests.

Related: OS#6679
Change-Id: I581483ccb9d8a254fcecc995fec3c811c5cf38eb
2025-01-06 11:25:14 +01:00
Philipp Maier 6d2e3853b4 global_platform: add spec reference to help of --install-parameters
Related: OS#6679
Change-Id: I7e8174d469e09ad130d2866663a65bdeb4afc35a
2024-12-20 15:54:17 +01:00
Philipp Maier 2a833b480a global_platform: fix command "delete"
The delete command formats a TPDU, not APDU, which leads to warning messages

Related: OS#6679
Change-Id: Id04c89acbd4f449cb974d3cb05024f11dba4684e
2024-12-19 18:26:58 +01:00
Philipp Maier 6287db4855 global_platform: remove unused code
This commented out part is not needed anymore.

Related: OS#6679
Change-Id: If1de0218f841159789ac86f6a13740c1cbd0a57a
2024-12-19 18:08:54 +01:00
Philipp Maier 9df5e2f171 javacard, cosmetic: fix sourcecode fromatting and improve docstring
The line with TAGS is longer than 120 columns and there is some
comment that should be moved to the python docstring.

Related: OS#6679
Change-Id: I1d02098320cfbe17a0eb1bfdcf3bc85034cc8e20
2024-12-19 18:05:06 +01:00
Philipp Maier 25319c5184 ara_m fix export of AID-REF-DO (empty)
GPD_SPE_013 Table 6-3 defines two types of AID-REF-DO objects (both
are fully independed TLV IEs with the same name). The version with
tag '4F' identifies an SE application. It may contain an AID prefix
or even be of length 0 in case the rule should apply to all SE
applications. Then there is the version with tag 'C0', which must
always have length 0 and serves a flag to apply the rule to the
implicitly selected SE application. Technically both are completely
different things, so we must also treat them separately in the
pySim-shell code.

Related: OS#6681
Change-Id: I771d5e860b12215280e3d0a8c314ce843fe0d6a2
2024-12-11 11:11:44 +01:00