WIP: osmo-smdpp ES9+ support for ASN.1 endpoint

There's actually no additional TLV structure inside the Tag 0x04.

Change-Id: Ic922355308747a888083c5b26765d272b6b20bd0

.checkpatch.conf

@@ -0,0 +1,2 @@
+--exclude ^pySim/esim/asn1/.*\.asn$
+--exclude ^smdpp-data/.*$

.gitignore

@@ -1,2 +1,5 @@
 *.pyc
 .*.swp
+
+/docs/_*
+/docs/generated

README.md

@@ -1,22 +1,61 @@
-pySim - Read, Write and Browse Programmable SIM/USIM Cards
-====================================================
+pySim - Read, Write and Browse Programmable SIM/USIM/ISIM/HPSIM Cards
+=====================================================================
 
-This repository contains Python programs that can be used
-to read, program (write) and browse certain fields/parameters on so-called programmable
-SIM/USIM cards.
+This repository contains a number of Python programs that can be used
+to read, program (write) and browse all fields/parameters/files on
+SIM/USIM/ISIM/HPSIM cards used in 3GPP cellular networks from 2G to 5G.
 
-Such SIM/USIM cards are special cards, which - unlike those issued by
-regular commercial operators - come with the kind of keys that allow you
-to write the files/fields that normally only an operator can program.
+Note that the access control configuration of normal production cards
+issue by operators will restrict significantly which files a normal
+user can read, and particularly write to.
+
+The full functionality of pySim hence can only be used with on so-called
+programmable SIM/USIM/ISIM/HPSIM cards.
+
+Such SIM/USIM/ISIM/HPSIM cards are special cards, which - unlike those
+issued by regular commercial operators - come with the kind of keys that
+allow you to write the files/fields that normally only an operator can
+program.
 
 This is useful particularly if you are running your own cellular
-network, and want to issue your own SIM/USIM cards for that network.
+network, and want to configure your own SIM/USIM/ISIM/HPSIM cards for
+that network.
 
 
-Homepage and Manual
--------------------
+Homepage
+--------
+
+Please visit the [official homepage](https://osmocom.org/projects/pysim/wiki)
+for usage instructions, manual and examples.
+
+
+Documentation
+-------------
+
+The pySim user manual can be built from this very source code by means
+of sphinx (with sphinxcontrib-napoleon and sphinx-argparse).  See the
+Makefile in the 'docs' directory.
+
+A pre-rendered HTML user manual of the current pySim 'git master' is
+available from <https://downloads.osmocom.org/docs/latest/pysim/> and
+a downloadable PDF version is published at
+<https://downloads.osmocom.org/docs/latest/osmopysim-usermanual.pdf>.
+
+A slightly dated video presentation about pySim-shell can be found at
+<https://media.ccc.de/v/osmodevcall-20210409-laforge-pysim-shell>.
+
+
+pySim-shell vs. legacy tools
+----------------------------
+
+While you will find a lot of online resources still describing the use of
+pySim-prog.py and pySim-read.py, those tools are considered legacy by
+now and have by far been superseded by the much more capable
+pySim-shell.  We strongly encourage users to adopt pySim-shell, unless
+they have very specific requirements like batch programming of large
+quantities of cards, which is about the only remaining use case for the
+legacy tools.
 
-Please visit the [official homepage](https://osmocom.org/projects/pysim/wiki) for usage instructions, manual and examples.  The user manual can also be built locally from this source code by ``cd docs && make html latexpdf`` for HTML and PDF format, respectively.
 
 Git Repository
 --------------
@@ -34,17 +73,20 @@ Installation
 
 Please install the following dependencies:
 
+ - bidict
+ - cmd2 >= 1.5.0
+ - colorlog
+ - construct >= 2.9.51
+ - gsm0338
+ - jsonpath-ng
+ - packaging
+ - pycryptodomex
  - pyscard
  - pyserial
  - pytlv
- - cmd2 >= 1.3.0 but < 2.0.0
- - jsonpath-ng
- - construct >= 2.9.51
- - bidict
- - gsm0338
  - pyyaml >= 5.1
+ - smpp.pdu (from `github.com/hologram-io/smpp.pdu`)
  - termcolor
- - colorlog
 
 Example for Debian:
 ```sh
@@ -52,6 +94,7 @@ sudo apt-get install --no-install-recommends \
 	pcscd libpcsclite-dev \
 	python3 \
 	python3-setuptools \
+	python3-pycryptodome \
 	python3-pyscard \
 	python3-pip
 pip3 install --user -r requirements.txt
@@ -59,6 +102,9 @@ pip3 install --user -r requirements.txt
 
 After installing all dependencies, the pySim applications ``pySim-read.py``, ``pySim-prog.py`` and ``pySim-shell.py`` may be started directly from the cloned repository.
 
+In addition to the dependencies above ``pySim-trace.py`` requires ``tshark`` and the python package ``pyshark`` to be installed. It is known that the ``tshark`` package
+in Debian versions before 11 may not work with pyshark.
+
 ### Archlinux Package
 
 Archlinux users may install the package ``python-pysim-git``
@@ -99,32 +145,3 @@ Our coding standards are described at
 
 We are using a gerrit-based patch review process explained at
 <https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit>
-
-
-Documentation
--------------
-
-The pySim user manual can be built from this very source code by means
-of sphinx (with sphinxcontrib-napoleon and sphinx-argparse).  See the
-Makefile in the 'docs' directory.
-
-A pre-rendered HTML user manual of the current pySim 'git master' is
-available from <https://downloads.osmocom.org/docs/latest/pysim/> and
-a downloadable PDF version is published at
-<https://downloads.osmocom.org/docs/latest/osmopysim-usermanual.pdf>.
-
-A slightly dated video presentation about pySim-shell can be found at
-<https://media.ccc.de/v/osmodevcall-20210409-laforge-pysim-shell>.
-
-
-pySim-shell vs. legacy tools
-----------------------------
-
-While you will find a lot of online resources still describing the use of
-pySim-prog.py and pySim-read.py, those tools are considered legacy by
-now and have by far been superseded by the much more capable
-pySim-shell.  We strongly encourage users to adopt pySim-shell, unless
-they have very specific requirements like batch programming of large
-quantities of cards, which is about the only remaining use case for the
-legacy tools.
-

contrib/jenkins.sh

@@ -4,8 +4,11 @@
 # environment variables:
 # * WITH_MANUALS: build manual PDFs if set to "1"
 # * PUBLISH: upload manuals after building if set to "1" (ignored without WITH_MANUALS = "1")
+# * JOB_TYPE: one of 'test', 'pylint', 'docs'
 #
 
+export PYTHONUNBUFFERED=1
+
 if [ ! -d "./pysim-testdata/" ] ; then
 	echo "###############################################"
 	echo "Please call from pySim-prog top directory"
@@ -13,36 +16,47 @@ if [ ! -d "./pysim-testdata/" ] ; then
 	exit 1
 fi
 
-virtualenv -p python3 venv --system-site-packages
-. venv/bin/activate
-pip install -r requirements.txt
+case "$JOB_TYPE" in
+"test")
+	virtualenv -p python3 venv --system-site-packages
+	. venv/bin/activate
 
-# Execute automatically discovered unit tests first
-python -m unittest discover -v -s tests/
+	pip install -r requirements.txt
+	pip install pyshark
 
-# Run pylint to find potential errors
-# Ignore E1102: not-callable
-#   pySim/filesystem.py: E1102: method is not callable (not-callable)
-# Ignore E0401: import-error
-#   pySim/utils.py:276: E0401: Unable to import 'Crypto.Cipher' (import-error)
-#   pySim/utils.py:277: E0401: Unable to import 'Crypto.Util.strxor' (import-error)
-pip install pylint==2.14.5 # FIXME: 2.15 is crashing, see OS#5668
-python -m pylint -j0 --errors-only \
-	--disable E1102 \
-	--disable E0401 \
-	--enable W0301 \
-	pySim *.py
+	# Execute automatically discovered unit tests first
+	python -m unittest discover -v -s tests/
 
-# attempt to build documentation
-pip install sphinx
-pip install sphinxcontrib-napoleon
-pip3 install -e 'git+https://github.com/osmocom/sphinx-argparse@master#egg=sphinx-argparse'
-(cd docs && make html latexpdf)
+	# Run the test with physical cards
+	cd pysim-testdata
+	../tests/pySim-prog_test.sh
+	../tests/pySim-trace_test.sh
+	;;
+"pylint")
+	# Print pylint version
+	pip3 freeze | grep pylint
+	# Run pylint to find potential errors
+	# Ignore E1102: not-callable
+	#   pySim/filesystem.py: E1102: method is not callable (not-callable)
+	# Ignore E0401: import-error
+	#   pySim/utils.py:276: E0401: Unable to import 'Crypto.Cipher' (import-error)
+	#   pySim/utils.py:277: E0401: Unable to import 'Crypto.Util.strxor' (import-error)
+	python3 -m pylint -j0 --errors-only \
+		--disable E1102 \
+		--disable E0401 \
+		--enable W0301 \
+		pySim *.py
+	;;
+"docs")
+	rm -rf docs/_build
+	make -C "docs" html latexpdf
 
-if [ "$WITH_MANUALS" = "1" ] && [ "$PUBLISH" = "1" ]; then
-	make -C "docs" publish publish-html
-fi
-
-# run the test with physical cards
-cd pysim-testdata
-../tests/pysim-test.sh
+	if [ "$WITH_MANUALS" = "1" ] && [ "$PUBLISH" = "1" ]; then
+		make -C "docs" publish publish-html
+	fi
+	;;
+*)
+	set +x
+	echo "ERROR: JOB_TYPE has unexpected value '$JOB_TYPE'."
+	exit 1
+esac

contrib/sim-rest-server.py

@@ -26,7 +26,10 @@ from klein import Klein
 from pySim.transport import ApduTracer
 from pySim.transport.pcsc import PcscSimLink
 from pySim.commands import SimCardCommands
-from pySim.cards import UsimCard
+from pySim.cards import UiccCardBase
+from pySim.utils import dec_iccid, dec_imsi
+from pySim.ts_51_011 import EF_IMSI
+from pySim.ts_102_221 import EF_ICCID
 from pySim.exceptions import *
 
 class ApduPrintTracer(ApduTracer):
@@ -35,18 +38,20 @@ class ApduPrintTracer(ApduTracer):
         pass
 
 def connect_to_card(slot_nr:int):
-    tp = PcscSimLink(slot_nr, apdu_tracer=ApduPrintTracer())
+    tp = PcscSimLink(argparse.Namespace(pcsc_dev=slot_nr), apdu_tracer=ApduPrintTracer())
     tp.connect()
 
     scc = SimCardCommands(tp)
-    card = UsimCard(scc)
+    card = UiccCardBase(scc)
 
     # this should be part of UsimCard, but FairewavesSIM breaks with that :/
     scc.cla_byte = "00"
     scc.sel_ctrl = "0004"
 
     card.read_aids()
-    card.select_adf_by_aid(adf='usim')
+
+    # ensure that MF is selected when we are done.
+    card._scc.select_file('3f00')
 
     return tp, scc, card
 
@@ -96,7 +101,7 @@ class SimRestServer:
         elif sw == '6982':
             return str(ApiError("Security Status not satisfied - Card PIN enabled?", sw))
         else:
-            return str(ApiError("Card Communication Error %s" % failure.value), sw)
+            return str(ApiError("Card Communication Error %s" % failure.value, sw))
 
 
     @app.route('/sim-auth-api/v1/slot/<int:slot>')
@@ -132,10 +137,14 @@ class SimRestServer:
 
         tp, scc, card = connect_to_card(slot)
 
+        ef_iccid = EF_ICCID()
+        (iccid, sw) = card._scc.read_binary(ef_iccid.fid)
+
         card.select_adf_by_aid(adf='usim')
-        iccid, sw = card.read_iccid()
-        imsi, sw = card.read_imsi()
-        res = {"imsi": imsi, "iccid": iccid }
+        ef_imsi = EF_IMSI()
+        (imsi, sw) = card._scc.read_binary(ef_imsi.fid)
+
+        res = {"imsi": dec_imsi(imsi), "iccid": dec_iccid(iccid) }
 
         tp.disconnect()
 

contrib/unber.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+
+# A more useful verion of the 'unber' tool provided with asn1c:
+# Give a hierarchical decode of BER/DER-encoded ASN.1 TLVs
+
+import sys
+import argparse
+
+from pySim.utils import bertlv_parse_one, bertlv_encode_tag, b2h, h2b
+
+def process_one_level(content: bytes, indent: int):
+    remainder = content
+    while len(remainder):
+        tdict, l, v, remainder = bertlv_parse_one(remainder)
+        #print(tdict)
+        rawtag = bertlv_encode_tag(tdict)
+        if tdict['constructed']:
+            print("%s%s l=%d" % (indent*"  ", b2h(rawtag), l))
+            process_one_level(v, indent + 1)
+        else:
+            print("%s%s l=%d %s" % (indent*"  ", b2h(rawtag), l, b2h(v)))
+
+
+option_parser = argparse.ArgumentParser(description='BER/DER data dumper')
+group = option_parser.add_mutually_exclusive_group(required=True)
+group.add_argument('--file', help='Input file')
+group.add_argument('--hex', help='Input hexstring')
+
+
+if __name__ == '__main__':
+    opts = option_parser.parse_args()
+
+    if opts.file:
+        with open(opts.file, 'rb') as f:
+            content = f.read()
+    elif opts.hex:
+        content = h2b(opts.hex)
+
+    process_one_level(content, 0)

docs/Makefile

@@ -9,11 +9,17 @@ SOURCEDIR     = .
 BUILDDIR      = _build
 
 # for osmo-gsm-manuals
-OSMO_GSM_MANUALS_DIR=$(shell pkg-config osmo-gsm-manuals --variable=osmogsmmanualsdir 2>/dev/null)
+OSMO_GSM_MANUALS_DIR ?= $(shell pkg-config osmo-gsm-manuals --variable=osmogsmmanualsdir 2>/dev/null)
 OSMO_REPOSITORY = "pysim"
 UPLOAD_FILES = $(BUILDDIR)/latex/osmopysim-usermanual.pdf
 CLEAN_FILES = $(UPLOAD_FILES)
 
+# Copy variables from Makefile.common.inc that are used in publish-html,
+# as Makefile.common.inc must be included after publish-html
+PUBLISH_REF   ?= master
+PUBLISH_TEMPDIR = _publish_tmpdir
+SSH_COMMAND = ssh -o 'UserKnownHostsFile=$(OSMO_GSM_MANUALS_DIR)/build/known_hosts' -p 48
+
 # Put it first so that "make" without argument is like "make help".
 .PHONY: help
 help:
@@ -23,7 +29,16 @@ $(BUILDDIR)/latex/pysim.pdf: latexpdf
 	@/bin/true
 
 publish-html: html
-	rsync -avz -e "ssh -o 'UserKnownHostsFile=$(OSMO_GSM_MANUALS_DIR)/build/known_hosts' -p 48" $(BUILDDIR)/html/ docs@ftp.osmocom.org:web-files/latest/pysim/
+	rm -rf "$(PUBLISH_TEMPDIR)"
+	mkdir -p "$(PUBLISH_TEMPDIR)/pysim/$(PUBLISH_REF)"
+	cp -r "$(BUILDDIR)"/html "$(PUBLISH_TEMPDIR)/pysim/$(PUBLISH_REF)"
+	cd "$(PUBLISH_TEMPDIR)" && \
+		rsync \
+			-avzR \
+			-e "$(SSH_COMMAND)" \
+			"pysim" \
+			docs@ftp.osmocom.org:web-files/
+	rm -rf "$(PUBLISH_TEMPDIR)"
 
 # put this before the catch-all below
 include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.common.inc
@@ -32,4 +47,6 @@ include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.common.inc
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %:
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@if [ "$@" != "shrink" ]; then \
+		$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O); \
+	fi

docs/conf.py

@@ -18,8 +18,8 @@ sys.path.insert(0, os.path.abspath('..'))
 # -- Project information -----------------------------------------------------
 
 project = 'osmopysim-usermanual'
-copyright = '2009-2022 by Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
-author = 'Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
+copyright = '2009-2023 by Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle, Merlin Chlosta'
+author = 'Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle, Merlin Chlosta'
 
 
 # -- General configuration ---------------------------------------------------

docs/index.rst

@@ -31,6 +31,7 @@ pySim consists of several parts:
 * a python :ref:`library<pySim library>` containing plenty of objects and methods that can be used for
   writing custom programs interfacing with SIM cards.
 * the [new] :ref:`interactive pySim-shell command line program<pySim-shell>`
+* the [new] :ref:`pySim-trace APDU trace decoder<pySim-trace>`
 * the [legacy] :ref:`pySim-prog and pySim-read tools<Legacy tools>`
 
 .. toctree::
@@ -38,8 +39,10 @@ pySim consists of several parts:
    :caption: Contents:
 
    shell
+   trace
    legacy
    library
+   osmo-smdpp
 
 
 Indices and tables

docs/legacy.rst

@@ -98,3 +98,4 @@ pySim-read usage
 .. argparse::
    :module: pySim-read
    :func: option_parser
+   :prog: pySim-read.py

docs/osmo-smdpp.rst

@@ -0,0 +1,93 @@
+osmo-smdpp
+==========
+
+`osmo-smdpp` is a proof-of-concept implementation of a minimal **SM-DP+** as specified for the *GSMA
+Consumer eSIM Remote SIM provisioning*.
+
+At least at this point, it is intended to be used for research and development, and not as a
+production SM-DP+.
+
+Unless you are a GSMA SAS-SM accredited SM-DP+ operator and have related DPtls, DPauth and DPpb
+certificates signed by the GSMA CI, you **can not use osmo-smdpp with regular production eUICC**.
+This is due to how the GSMA eSIM security architecture works.  You can, however, use osmo-smdpp with
+so-called *test-eUICC*, which contain certificates/keys signed by GSMA test certificates as laid out
+in GSMA SGP.26.
+
+At this point, osmo-smdpp does not support anything beyond the bare minimum required to download
+eSIM profiles to an eUICC.  Specifically, there is no ES2+ interface, and there is no built-in
+support for profile personalization yet.
+
+osmo-smdpp currently
+
+* always provides the exact same profile to every request.  The profile always has the same IMSI and
+  ICCID.
+* **is absolutely insecure**, as it
+
+ * does not perform any certificate verification
+ * does not evaluate/consider any *Matching ID* or *Confirmation Code*
+ * stores the sessions in an unencrypted _python shelve_ and is hence leaking one-time key materials
+   used for profile encryption and signing.
+
+
+Running osmo-smdpp
+------------------
+
+osmo-smdpp does not have built-in TLS support as the used *twisted* framework appears to have
+problems when using the example elliptic curve certificates (both NIST and Brainpool) from GSMA.
+
+So in order to use it, you have to put it behind a TLS reverse proxy, which terminates the ES9+
+HTTPS from the LPA, and then forwards it as plain HTTP to osmo-smdpp.
+
+nginx as TLS proxy
+~~~~~~~~~~~~~~~~~~
+
+If you use `nginx` as web server, you can use the following configuration snippet::
+
+  upstream smdpp {
+          server localhost:8000;
+  }
+
+  server {
+          listen 443 ssl;
+          server_name testsmdpplus1.example.com;
+
+          ssl_certificate /my/path/to/pysim/smdpp-data/certs/DPtls/CERT_S_SM_DP_TLS_NIST.pem;
+          ssl_certificate_key /my/path/to/pysim/smdpp-data/certs/DPtls/SK_S_SM_DP_TLS_NIST.pem;
+
+          location / {
+                  proxy_read_timeout 600s;
+
+                  proxy_hide_header X-Powered-By;
+                  proxy_set_header X-Real-IP $remote_addr;
+                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                  proxy_set_header X-Forwarded-Proto https;
+                  proxy_set_header X-Forwarded-Port $proxy_port;
+                  proxy_set_header Host $host;
+
+                  proxy_pass http://smdpp/;
+          }
+  }
+
+You can of course achieve a similar functionality with apache, lighttpd or many other web server
+software.
+
+
+osmo-smdpp
+~~~~~~~~~~
+
+osmo-smdpp currently doesn't have any configuration file or command line options.  You just run it,
+and it will bind its plain-HTTP ES9+ interface to local TCP port 8000.
+
+The `smdpp-data/certs`` directory contains the DPtls, DPauth and DPpb as well as CI certificates
+used; they are copied from GSMA SGP.26 v2.
+
+The `smdpp-data/upp` directory contains the UPP (Unprotected Profile Package) used.
+
+
+DNS setup for your LPA
+~~~~~~~~~~~~~~~~~~~~~~
+
+The LPA must resolve `testsmdpplus1.example.com` to the IP address of your TLS proxy.
+
+It must also accept the TLS certificates used by your TLS proxy.
+

docs/suci-tutorial.rst

@@ -0,0 +1,195 @@
+
+Guide: Enabling 5G SUCI
+========================
+
+SUPI/SUCI Concealment is a feature of 5G-Standalone (SA) to encrypt the
+IMSI/SUPI with a network operator public key.  3GPP Specifies two different
+variants for this:
+
+* SUCI calculation *in the UE*, using data from the SIM
+* SUCI calculation *on the card itself*
+
+pySIM supports writing the 5G-specific files for *SUCI calculation in the UE* on USIM cards, assuming that
+your cards contain the required files, and you have the privileges/credentials to write to them.  This is
+the case using sysmocom sysmoISIM-SJA2 cards (or successor products).
+
+In short, you can enable SUCI with these steps:
+
+* activate USIM **Service 124**
+* make sure USIM **Service 125** is disabled
+* store the public keys in **SUCI_Calc_Info**
+* set the **Routing Indicator** (required)
+
+If you want to disable the feature, you can just disable USIM Service 124 (and 125).
+
+Technical References
+~~~~~~~~~~~~~~~~~~~~
+
+This guide covers the basic workflow of provisioning SIM cards with the 5G SUCI feature. For detailed information on the SUCI feature and file contents, the following documents are helpful:
+
+* USIM files and structure: `TS 31.102 <https://www.etsi.org/deliver/etsi_ts/131100_131199/131102/16.06.00_60/ts_131102v160600p.pdf>`__
+* USIM tests (incl. file content examples) `TS 31.121 <https://www.etsi.org/deliver/etsi_ts/131100_131199/131121/16.01.00_60/ts_131121v160100p.pdf>`__
+
+For specific information on sysmocom SIM cards, refer to Section 9.1 of the `sysmoUSIM User
+Manual <https://www.sysmocom.de/manuals/sysmousim-manual.pdf>`__.
+
+--------------
+
+Admin PIN
+---------
+
+The usual way to authenticate yourself to the card as the cellular
+operator is to validate the so-called ADM1 (admin) PIN.  This may differ
+from card model/vendor to card model/vendor.
+
+Start pySIM-shell and enter the admin PIN for your card. If you bought
+the SIM card from your network operator and don’t have the admin PIN,
+you cannot change SIM contents!
+
+Launch pySIM:
+
+::
+
+    $ ./pySim-shell.py -p 0
+
+    Using PC/SC reader interface
+    Autodetected card type: sysmoISIM-SJA2
+    Welcome to pySim-shell!
+    pySIM-shell (00:MF)>
+
+Enter the ADM PIN:
+
+::
+
+   pySIM-shell (00:MF)> verify_adm XXXXXXXX
+
+Otherwise, write commands will fail with ``SW Mismatch: Expected 9000 and got 6982.``
+
+Key Provisioning
+----------------
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select DF.5GS
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS)> select EF.SUCI_Calc_Info
+
+By default, the file is present but empty:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.SUCI_Calc_Info)> read_binary_decoded
+   missing Protection Scheme Identifier List data object tag
+   9000: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff -> {}
+
+The following JSON config defines the testfile from `TS 31.121 <https://www.etsi.org/deliver/etsi_ts/131100_131199/131121/16.01.00_60/ts_131121v160100p.pdf>`__ Section 4.9.4 with
+test keys from `TS 33.501 <hhttps://www.etsi.org/deliver/etsi_ts/133500_133599/133501/16.05.00_60/ts_133501v160500p.pdf>`__ Annex C.4. Highest priority (``0``) has a
+Profile-B (``identifier: 2``) key in key slot ``1``, which means the key
+with ``hnet_pubkey_identifier: 27``.
+
+.. code:: json
+
+   {
+        "prot_scheme_id_list": [
+           {"priority": 0, "identifier": 2, "key_index": 1},
+           {"priority": 1, "identifier": 1, "key_index": 2},
+           {"priority": 2, "identifier": 0, "key_index": 0}],
+        "hnet_pubkey_list": [
+           {"hnet_pubkey_identifier": 27,
+            "hnet_pubkey": "0272DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD1"},
+           {"hnet_pubkey_identifier": 30,
+            "hnet_pubkey": "5A8D38864820197C3394B92613B20B91633CBD897119273BF8E4A6F4EEC0A650"}]
+   }
+
+Write the config to file (must be single-line input as for now):
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.SUCI_Calc_Info)> update_binary_decoded '{ "prot_scheme_id_list": [ {"priority": 0, "identifier": 2, "key_index": 1}, {"priority": 1, "identifier": 1, "key_index": 2}, {"priority": 2, "identifier": 0, "key_index": 0}], "hnet_pubkey_list": [ {"hnet_pubkey_identifier": 27, "hnet_pubkey": "0272DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD1"}, {"hnet_pubkey_identifier": 30, "hnet_pubkey": "5A8D38864820197C3394B92613B20B91633CBD897119273BF8E4A6F4EEC0A650"}]}'
+
+WARNING: These are TEST KEYS with publicly known/specified private keys, and hence unsafe for live/secure
+deployments! For use in production networks, you need to generate your own set[s] of keys.
+
+Routing Indicator
+-----------------
+
+The Routing Indicator must be present for the SUCI feature. By default,
+the contents of the file is **invalid** (ffffffff):
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select DF.5GS
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS)> select EF.Routing_Indicator
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.Routing_Indicator)> read_binary_decoded
+   9000: ffffffff -> {'raw': 'ffffffff'}
+
+The Routing Indicator is a four-byte file but the actual Routing
+Indicator goes into bytes 0 and 1 (the other bytes are reserved). To set
+the Routing Indicator to 0x71:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.Routing_Indicator)> update_binary 17ffffff
+
+You can also set the routing indicator to **0x0**, which is *valid* and
+means “routing indicator not specified”, leaving it to the modem.
+
+USIM Service Table
+------------------
+
+First, check out the USIM Service Table (UST):
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select EF.UST
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> read_binary_decoded
+   9000: beff9f9de73e0408400170730000002e00000000 -> [2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 25, 27, 28, 29, 33, 34, 35, 38, 39, 42, 43, 44, 45, 46, 51, 60, 71, 73, 85, 86, 87, 89, 90, 93, 94, 95, 122, 123, 124, 126]
+
+.. list-table:: From TS31.102
+   :widths: 15 40
+   :header-rows: 1
+
+   * - Service No.
+     - Description
+   * - 122
+     - 5GS Mobility Management Information
+   * - 123
+     - 5G Security Parameters
+   * - 124
+     - Subscription identifier privacy support
+   * - 125
+     - SUCI calculation by the USIM
+   * - 126
+     - UAC Access Identities support
+   * - 129
+     - 5GS Operator PLMN List
+
+If you’d like to enable/disable any UST service:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_deactivate 124
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_activate 124
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_deactivate 125
+
+In this case, UST Service 124 is already enabled and you’re good to go. The
+sysmoISIM-SJA2 does not support on-SIM calculation, so service 125 must
+be disabled.
+
+USIM Error with 5G and sysmoISIM
+--------------------------------
+
+sysmoISIMs come 5GS-enabled. By default however, the configuration stored
+in the card file-system is **not valid** for 5G networks: Service 124 is enabled,
+but EF.SUCI_Calc_Info and EF.Routing_Indicator are empty files (hence
+do not contain valid data).
+
+At least for Qualcomm’s X55 modem, this results in an USIM error and the
+whole modem shutting 5G down. If you don’t need SUCI concealment but the
+smartphone refuses to connect to any 5G network, try to disable the UST
+service 124.

docs/trace.rst

@@ -0,0 +1,64 @@
+pySim-trace
+===========
+
+pySim-trace is a utility for high-level decode of APDU protocol traces such as those obtained with
+`Osmocom SIMtrace2 <https://osmocom.org/projects/simtrace2/wiki>`_ or `osmo-qcdiag <https://osmocom.org/projects/osmo-qcdiag/wiki>`_.
+
+pySim-trace leverages the existing knowledge of pySim-shell on anything related to SIM cards,
+including the structure/encoding of the various files on SIM/USIM/ISIM/HPSIM cards, and applies this
+to decoding protocol traces.  This means that it shows not only the name of the command (like READ
+BINARY), but actually understands what the currently selected file is, and how to decode the
+contents of that file.
+
+pySim-trace also understands the parameters passed to commands and how to decode them, for example
+of the AUTHENTICATE command within the USIM/ISIM/HPSIM application.
+
+
+Demo
+----
+
+To get an idea how pySim-trace usage looks like, you can watch the relevant part of the 11/2022
+SIMtrace2 tutorial whose `recording is freely accessible <https://media.ccc.de/v/osmodevcall-20221019-laforge-simtrace2-tutorial#t=2134>`_.
+
+
+Running pySim-trace
+-------------------
+
+Running pySim-trace requires you to specify the *source* of the to-be-decoded APDUs.  There are several
+supported options, each with their own respective parameters (like a file name for PCAP decoding).
+
+See the detailed command line reference below for details.
+
+A typical execution of pySim-trace for doing live decodes of *GSMTAP (SIM APDU)* e.g. from SIMtrace2 or
+osmo-qcdiag would look like this:
+
+::
+
+  ./pySim-trace.py gsmtap-udp
+
+This binds to the default UDP port 4729 (GSMTAP) on localhost (127.0.0.1), and decodes any APDUs received
+there.
+
+
+
+pySim-trace command line reference
+----------------------------------
+
+.. argparse::
+   :module: pySim-trace
+   :func: option_parser
+   :prog: pySim-trace.py
+
+
+Constraints
+-----------
+
+* In order to properly track the current location in the filesystem tree and other state, it is
+  important that the trace you're decoding includes all of the communication with the SIM, ideally
+  from the very start (power up).
+
+* pySim-trace currently only supports ETSI UICC (USIM/ISIM/HPSIM) and doesn't yet support legacy GSM
+  SIM.  This is not a fundamental technical constraint, it's just simply that nobody got around
+  developing and testing that part. Contributions are most welcome.
+
+

osmo-smdpp.py

@@ -0,0 +1,714 @@
+#!/usr/bin/env python3
+
+# Early proof-of-concept towards a SM-DP+ HTTP service for GSMA consumer eSIM RSP
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import sys
+import argparse
+import uuid
+import os
+import functools
+from typing import Optional, Dict, List
+from pprint import pprint as pp
+
+import base64
+from base64 import b64decode
+from klein import Klein
+from twisted.web.iweb import IRequest
+import asn1tools
+
+from pySim.utils import h2b, b2h, swap_nibbles
+
+import pySim.esim.rsp as rsp
+from pySim.esim.es8p import *
+
+# HACK: make this configurable
+DATA_DIR = './smdpp-data'
+HOSTNAME = 'testsmdpplus1.example.com' # must match certificates!
+
+
+def b64encode2str(req: bytes) -> str:
+    """Encode given input bytes as base64 and return result as string."""
+    return base64.b64encode(req).decode('ascii')
+
+def set_headers(request: IRequest):
+    """Set the request headers as mandatory by GSMA eSIM RSP."""
+    request.setHeader('Content-Type', 'application/json;charset=UTF-8')
+    request.setHeader('X-Admin-Protocol', 'gsma/rsp/v2.1.0')
+
+def build_status_code(subject_code: str, reason_code: str, subject_id: Optional[str], message: Optional[str]) -> Dict:
+    r = {'subjectCode': subject_code, 'reasonCode': reason_code }
+    if subject_id:
+        r['subjectIdentifier'] = subject_id
+    if message:
+        r['message'] = message
+    return r
+
+def build_resp_header(js: dict, status: str = 'Executed-Success', status_code_data = None) -> None:
+    # SGP.22 v3.0 6.5.1.4
+    js['header'] = {
+        'functionExecutionStatus': {
+            'status': status,
+        }
+    }
+    if status_code_data:
+        js['header']['functionExecutionStatus']['statusCodeData'] = status_code_data
+
+from cryptography.hazmat.primitives.asymmetric.utils import decode_dss_signature, encode_dss_signature
+from cryptography.hazmat.primitives.serialization import load_pem_private_key, Encoding, PublicFormat, PrivateFormat, NoEncryption
+from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.primitives import hashes
+from cryptography.exceptions import InvalidSignature
+from cryptography import x509
+
+
+def ecdsa_dss_to_tr03111(sig: bytes) -> bytes:
+    """convert from DER format to BSI TR-03111; first get long integers; then convert those to bytes."""
+    r, s = decode_dss_signature(sig)
+    return r.to_bytes(32, 'big') + s.to_bytes(32, 'big')
+
+def ecdsa_tr03111_to_dss(sig: bytes) -> bytes:
+    """convert an ECDSA signature from BSI TR-03111 format to DER: first get long integers; then encode those."""
+    assert len(sig) == 64
+    r = int.from_bytes(sig[0:32], 'big')
+    s = int.from_bytes(sig[32:32*2], 'big')
+    return encode_dss_signature(r, s)
+
+
+class CertAndPrivkey:
+    """A pair of certificate and private key, as used for ECDSA signing."""
+    def __init__(self, required_policy_oid: Optional[x509.ObjectIdentifier] = None,
+                 cert: Optional[x509.Certificate] = None, priv_key = None):
+        self.required_policy_oid = required_policy_oid
+        self.cert = cert
+        self.priv_key = priv_key
+
+    def cert_from_der_file(self, path: str):
+        with open(path, 'rb') as f:
+            cert = x509.load_der_x509_certificate(f.read())
+        if self.required_policy_oid:
+            # verify it is the right type of certificate (id-rspRole-dp-auth, id-rspRole-dp-auth-v2, etc.)
+            assert cert_policy_has_oid(cert, self.required_policy_oid)
+        self.cert = cert
+
+    def privkey_from_pem_file(self, path: str, password: Optional[str] = None):
+        with open(path, 'rb') as f:
+            self.priv_key = load_pem_private_key(f.read(), password)
+
+    def ecdsa_sign(self, plaintext: bytes) -> bytes:
+        """Sign some input-data using an ECDSA signature compliant with SGP.22,
+        which internally refers to Global Platform 2.2 Annex E, which in turn points
+        to BSI TS-03111 which states "concatengated raw R + S values". """
+        sig = self.priv_key.sign(plaintext, ec.ECDSA(hashes.SHA256()))
+        # convert from DER format to BSI TR-03111; first get long integers; then convert those to bytes
+        return ecdsa_dss_to_tr03111(sig)
+
+    def get_authority_key_identifier(self) -> x509.AuthorityKeyIdentifier:
+        """Return the AuthorityKeyIdentifier X.509 extension of the certificate."""
+        return list(filter(lambda x: isinstance(x.value, x509.AuthorityKeyIdentifier), self.cert.extensions))[0].value
+
+    def get_subject_alt_name(self) -> x509.SubjectAlternativeName:
+        """Return the SubjectAlternativeName X.509 extension of the certificate."""
+        return list(filter(lambda x: isinstance(x.value, x509.SubjectAlternativeName), self.cert.extensions))[0].value
+
+    def get_cert_as_der(self) -> bytes:
+        """Return certificate encoded as DER."""
+        return self.cert.public_bytes(Encoding.DER)
+
+    def get_curve(self) -> ec.EllipticCurve:
+        return self.cert.public_key().public_numbers().curve
+
+
+
+
+class ApiError(Exception):
+    def __init__(self, subject_code: str, reason_code: str, message: Optional[str] = None,
+                 subject_id: Optional[str] = None):
+        self.status_code = build_status_code(subject_code, reason_code, subject_id, message)
+
+    def encode(self) -> str:
+        """Encode the API Error into a responseHeader string."""
+        js = {}
+        build_resp_header(js, 'Failed', self.status_code)
+        return json.dumps(js)
+
+def cert_policy_has_oid(cert: x509.Certificate, match_oid: x509.ObjectIdentifier) -> bool:
+    """Determine if given certificate has a certificatePolicy extension of matching OID."""
+    for policy_ext in filter(lambda x: isinstance(x.value, x509.CertificatePolicies), cert.extensions):
+        if any(policy.policy_identifier == match_oid for policy in policy_ext.value._policies):
+            return True
+    return False
+
+ID_RSP = "2.23.146.1"
+ID_RSP_CERT_OBJECTS = '.'.join([ID_RSP, '2'])
+ID_RSP_ROLE = '.'.join([ID_RSP_CERT_OBJECTS, '1'])
+
+class oid:
+    id_rspRole_ci = x509.ObjectIdentifier(ID_RSP_ROLE + '.0')
+    id_rspRole_euicc_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.1')
+    id_rspRole_eum_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.2')
+    id_rspRole_dp_tls_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.3')
+    id_rspRole_dp_auth_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.4')
+    id_rspRole_dp_pb_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.5')
+    id_rspRole_ds_tls_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.6')
+    id_rspRole_ds_auth_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.7')
+
+class SmDppHttpServer:
+    app = Klein()
+
+    @staticmethod
+    def load_certs_from_path(path: str) -> List[x509.Certificate]:
+        """Load all DER + PEM files from given directory path and return them as list of x509.Certificate
+        instances."""
+        certs = []
+        for dirpath, dirnames, filenames in os.walk(path):
+            for filename in filenames:
+                cert = None
+                if filename.endswith('.der'):
+                    with open(os.path.join(dirpath, filename), 'rb') as f:
+                        cert = x509.load_der_x509_certificate(f.read())
+                elif filename.endswith('.pem'):
+                    with open(os.path.join(dirpath, filename), 'rb') as f:
+                        cert = x509.load_pem_x509_certificate(f.read())
+                if cert:
+                    # verify it is a CI certificate (keyCertSign + i-rspRole-ci)
+                    if not cert_policy_has_oid(cert, oid.id_rspRole_ci):
+                        raise ValueError("alleged CI certificate %s doesn't have CI policy" % filename)
+                    certs.append(cert)
+        return certs
+
+    def ci_get_cert_for_pkid(self, ci_pkid: bytes) -> Optional[x509.Certificate]:
+        """Find CI certificate for given key identifier."""
+        for cert in self.ci_certs:
+            print("cert: %s" % cert)
+            subject_exts = list(filter(lambda x: isinstance(x.value, x509.SubjectKeyIdentifier), cert.extensions))
+            print(subject_exts)
+            subject_pkid = subject_exts[0].value
+            print(subject_pkid)
+            if subject_pkid and subject_pkid.key_identifier == ci_pkid:
+                return cert
+        return None
+
+    def __init__(self, server_hostname: str, ci_certs_path: str, use_brainpool: bool = False):
+        self.server_hostname = server_hostname
+        self.ci_certs = self.load_certs_from_path(ci_certs_path)
+        # load DPauth cert + key
+        self.dp_auth = CertAndPrivkey(oid.id_rspRole_dp_auth_v2)
+        cert_dir = os.path.join(DATA_DIR, 'certs')
+        if use_brainpool:
+            self.dp_auth.cert_from_der_file(os.path.join(cert_dir, 'DPauth', 'CERT_S_SM_DPauth_ECDSA_BRP.der'))
+            self.dp_auth.privkey_from_pem_file(os.path.join(cert_dir, 'DPauth', 'SK_S_SM_DPauth_ECDSA_BRP.pem'))
+        else:
+            self.dp_auth.cert_from_der_file(os.path.join(cert_dir, 'DPauth', 'CERT_S_SM_DPauth_ECDSA_NIST.der'))
+            self.dp_auth.privkey_from_pem_file(os.path.join(cert_dir, 'DPauth', 'SK_S_SM_DPauth_ECDSA_NIST.pem'))
+        # load DPpb cert + key
+        self.dp_pb = CertAndPrivkey(oid.id_rspRole_dp_pb_v2)
+        if use_brainpool:
+            self.dp_pb.cert_from_der_file(os.path.join(cert_dir, 'DPpb', 'CERT_S_SM_DPpb_ECDSA_BRP.der'))
+            self.dp_pb.privkey_from_pem_file(os.path.join(cert_dir, 'DPpb', 'SK_S_SM_DPpb_ECDSA_BRP.pem'))
+        else:
+            self.dp_pb.cert_from_der_file(os.path.join(cert_dir, 'DPpb', 'CERT_S_SM_DPpb_ECDSA_NIST.der'))
+            self.dp_pb.privkey_from_pem_file(os.path.join(cert_dir, 'DPpb', 'SK_S_SM_DPpb_ECDSA_NIST.pem'))
+        self.rss = rsp.RspSessionStore(os.path.join(DATA_DIR, "sm-dp-sessions"))
+
+    @app.handle_errors(ApiError)
+    def handle_apierror(self, request: IRequest, failure):
+        request.setResponseCode(200)
+        pp(failure)
+        return failure.value.encode()
+
+    @staticmethod
+    def _ecdsa_verify(cert: x509.Certificate, signature: bytes, data: bytes) -> bool:
+        pubkey = cert.public_key()
+        dss_sig = ecdsa_tr03111_to_dss(signature)
+        try:
+            pubkey.verify(dss_sig, data, ec.ECDSA(hashes.SHA256()))
+            return True
+        except InvalidSignature:
+            return False
+
+    @staticmethod
+    def b64decode_members(indict: Dict, keys: List[str]):
+        """base64-decoder all members of 'indict' whose key is in 'keys'."""
+        for key in keys:
+            if key in indict:
+                indict[key] = b64decode(indict[key])
+
+    @staticmethod
+    def b64encode_members(indict: Dict, keys: List[str]):
+        """base64-encoder (to string!) all members of 'indict' whose key is in 'keys'."""
+        for key in keys:
+            if key in indict:
+                indict[key] = b64encode2str(indict[key])
+
+    @staticmethod
+    def asn1decode_member(indict: Dict, key: str, typename: Optional[str]):
+        """decode indict[key] using RSP ASN.1 decoder for type 'typename'."""
+        if not typename:
+            typename = capitalize_first_char(key)
+        if key in indict:
+            indict[key] = rsp.asn1.decode(typename, indict[key])
+
+    @staticmethod
+    def asn1encode_member(indict: Dict, key: str, typename: Optional[str]):
+        """encode indict[key] using RSP ASN.1 decoder for type 'typename'."""
+        if not typename:
+            typename = capitalize_first_char(key)
+        if key in indict:
+            indict[key] = rsp.asn1.encode(typename, indict[key])
+
+    @staticmethod
+    def rsp_api_wrapper(func):
+        """Wrapper that can be used as decorator in order to perform common REST API endpoint entry/exit
+        functionality, such as JSON decoding/encoding and debug-printing."""
+        @functools.wraps(func)
+        def _api_wrapper(self, request: IRequest):
+            # TODO: evaluate User-Agent + X-Admin-Protocol header
+            # TODO: reject any non-JSON Content-type
+
+            content = json.loads(request.content.read())
+            print("Rx JSON: %s" % content)
+            set_headers(request)
+
+            output = func(self, request, content) or {}
+
+            build_resp_header(output)
+            print("Tx JSON: %s" % output)
+            return json.dumps(output)
+        return _api_wrapper
+
+    def initiateAutentication(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ InitiateAuthentication SGP.22 Section 5.6.1"""
+        # Verify that the received address matches its own SM-DP+ address, where the comparison SHALL be
+        # case-insensitive. Otherwise, the SM-DP+ SHALL return a status code "SM-DP+ Address - Refused".
+        if content['smdpAddress'] != self.server_hostname:
+           raise ApiError('8.8.1', '3.8', 'Invalid SM-DP+ Address')
+
+        euiccChallenge = content['euiccChallenge']
+        if len(euiccChallenge) != 16:
+            raise ValueError
+
+        euiccInfo1 = content['euiccInfo1']
+        print("Rx euiccInfo1: %s" % euiccInfo1)
+        #euiccInfo1['svn']
+
+        # TODO: If euiccCiPKIdListForSigningV3 is present ...
+
+        pkid_list = euiccInfo1['euiccCiPKIdListForSigning']
+        if 'euiccCiPKIdListForSigningV3' in euiccInfo1:
+            pkid_list = pkid_list + euiccInfo1['euiccCiPKIdListForSigningV3']
+        # verify it supports one of the keys indicated by euiccCiPKIdListForSigning
+        if not any(self.ci_get_cert_for_pkid(x) for x in pkid_list):
+           raise ApiError('8.8.2', '3.1', 'None of the proposed Public Key Identifiers is supported by the SM-DP+')
+
+        # TODO: Determine the set of CERT.DPauth.SIG that satisfy the following criteria:
+        # * Part of a certificate chain ending at one of the eSIM CA RootCA Certificate, whose Public Keys is
+        #   supported by the eUICC (indicated by euiccCiPKIdListForVerification).
+        # * Using a certificate chain that the eUICC and the LPA both support:
+        #euiccInfo1['euiccCiPKIdListForVerification']
+        #   raise ApiError('8.8.4', '3.7', 'The SM-DP+ has no CERT.DPauth.SIG which chains to one of the eSIM CA Root CA CErtificate with a Public Key supported by the eUICC')
+
+        # Generate a TransactionID which is used to identify the ongoing RSP session. The TransactionID
+        # SHALL be unique within the scope and lifetime of each SM-DP+.
+        transactionId = uuid.uuid4().hex
+        assert not transactionId in self.rss
+
+        # Generate a serverChallenge for eUICC authentication attached to the ongoing RSP session.
+        serverChallenge = os.urandom(16)
+
+        # Generate a serverSigned1 data object as expected by the eUICC and described in section 5.7.13 "ES10b.AuthenticateServer". If and only if both eUICC and LPA indicate crlStaplingV3Support, the SM-DP+ SHALL indicate crlStaplingV3Used in sessionContext.
+        serverSigned1 = {
+            'transactionId': h2b(transactionId),
+            'euiccChallenge': euiccChallenge,
+            'serverAddress': self.server_hostname,
+            'serverChallenge': serverChallenge,
+            }
+        print("Tx serverSigned1: %s" % serverSigned1)
+        serverSigned1_bin = rsp.asn1.encode('ServerSigned1', serverSigned1)
+        print("Tx serverSigned1: %s" % rsp.asn1.decode('ServerSigned1', serverSigned1_bin))
+        output = {}
+        output['serverSigned1'] = serverSigned1
+
+        # Generate a signature (serverSignature1) as described in section 5.7.13 "ES10b.AuthenticateServer" using the SK related to the selected CERT.DPauth.SIG.
+        # serverSignature1 SHALL be created using the private key associated to the RSP Server Certificate for authentication, and verified by the eUICC using the contained public key as described in section 2.6.9. serverSignature1 SHALL apply on serverSigned1 data object.
+        output['serverSignature1'] = b'\x5f\x37\x40' + self.dp_auth.ecdsa_sign(serverSigned1_bin)
+
+        output['transactionId'] = transactionId
+        server_cert_aki = self.dp_auth.get_authority_key_identifier()
+        output['euiccCiPKIdToBeUsed'] = b'\x04\x14' + server_cert_aki.key_identifier
+        output['serverCertificate'] = self.dp_auth.get_cert_as_der() # CERT.DPauth.SIG
+        # FIXME: add those certificate
+        #output['otherCertsInChain'] = ...
+
+        # create SessionState and store it in rss
+        self.rss[transactionId] = rsp.RspSessionState(transactionId, serverChallenge)
+
+        return output
+
+    @app.route('/gsma/rsp2/es9plus/initiateAuthentication', methods=['POST'])
+    @rsp_api_wrapper
+    def json_initiateAuthentication(self, request: IRequest, content: dict):
+        """Transform from JSON binding to generic function and back."""
+        # convert from JSON/BASE64 to decoded ASN.1
+        b64decode_members(content, ['euiccChallenge', 'euiccInfo1', 'lpaRspCapability'])
+        asn1decode_member(content, 'eUICCInfo1')
+
+        # do the actual processing
+        output = self.initiateAuthentication(request, content)
+
+        # convert from decoded ASN.1 to base64 to JSON
+        asn1encode_member(output, 'serverSigned1')
+        b64encode_members(output, ['serverSigned1', 'serverSignature1', 'euiccCiPKIdToBeUsed',
+                                   'serverCertificate', 'otherCertsInChain', 'crlList'])
+        return output
+
+    def authenticateClient(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ AuthenticateClient in SGP.22 Section 5.6.3"""
+        transactionId = content['transactionId']
+
+        authenticateServerResp = content['authenticateServerResponse']
+        print("Rx %s: %s" % authenticateServerResp)
+        if authenticateServerResp[0] == 'authenticateResponseError':
+            r_err = authenticateServerResp[1]
+            #r_err['transactionId']
+            #r_err['authenticateErrorCode']
+            raise ValueError("authenticateResponseError %s" % r_err)
+
+        r_ok = authenticateServerResp[1]
+        euiccSigned1 = r_ok['euiccSigned1']
+        # TODO: use original data, don't re-encode?
+        euiccSigned1_bin = rsp.asn1.encode('EuiccSigned1', euiccSigned1)
+        euiccSignature1_bin = r_ok['euiccSignature1']
+        euiccCertificate_dec = r_ok['euiccCertificate']
+        # TODO: use original data, don't re-encode?
+        euiccCertificate_bin = rsp.asn1.encode('Certificate', euiccCertificate_dec)
+        eumCertificate_dec = r_ok['eumCertificate']
+        eumCertificate_bin = rsp.asn1.encode('Certificate', eumCertificate_dec)
+        # TODO v3: otherCertsInChain
+
+        # load certificate
+        euicc_cert = x509.load_der_x509_certificate(euiccCertificate_bin)
+        eum_cert = x509.load_der_x509_certificate(eumCertificate_bin)
+
+        # TODO: Verify the validity of the eUICC certificate chain
+        #   raise ApiError('8.1.3', '6.1', 'Verification failed')
+        #   raise ApiError('8.1.3', '6.3', 'Expired')
+
+        # TODO: Verify that the Root Certificate of the eUICC certificate chain corresponds to the
+        # euiccCiPKIdToBeUsed or euiccCiPKIdToBeUsedV3
+        #   raise ApiError('8.11.1', '3.9', 'Unknown')
+
+        # Verify euiccSignature1 over euiccSigned1 using pubkey from euiccCertificate.
+        # Otherwise, the SM-DP+ SHALL return a status code "eUICC - Verification failed"
+        if not self._ecdsa_verify(euicc_cert, euiccSignature1_bin, euiccSigned1_bin):
+            raise ApiError('8.1', '6.1', 'Verification failed')
+
+        # Verify that the transactionId is known and relates to an ongoing RSP session.  Otherwise, the SM-DP+
+        # SHALL return a status code "TransactionId - Unknown"
+        ss = self.rss.get(transactionId, None)
+        if ss is None:
+            raise ApiError('8.10.1', '3.9', 'Unknown')
+        ss.euicc_cert = euicc_cert
+        ss.eum_cert = eum_cert # do we need this in the state?
+
+        # TODO: verify eUICC cert is signed by EUM cert
+        # TODO: verify EUM cert is signed by CI cert
+        # TODO: verify EID of eUICC cert is  within permitted range of EUM cert
+
+        ss.eid = ss.euicc_cert.subject.get_attributes_for_oid(x509.oid.NameOID.SERIAL_NUMBER)[0].value
+        print("EID (from eUICC cert): %s" % ss.eid)
+
+        # Verify that the serverChallenge attached to the ongoing RSP session matches the
+        # serverChallenge returned by the eUICC. Otherwise, the SM-DP+ SHALL return a status code "eUICC -
+        # Verification failed".
+        if euiccSigned1['serverChallenge'] != ss.serverChallenge:
+            raise ApiError('8.1', '6.1', 'Verification failed')
+
+        # Put together profileMetadata + _bin
+        ss.profileMetadata = ProfileMetadata(iccid_bin= h2b(swap_nibbles('89000123456789012358')), spn="OsmocomSPN", profile_name="OsmocomProfile")
+        profileMetadata_bin = ss.profileMetadata.gen_store_metadata_request()
+
+        # Put together smdpSigned2 + _bin
+        smdpSigned2 = {
+            'transactionId': h2b(ss.transactionId),
+            'ccRequiredFlag': False,        # whether the Confirmation Code is required
+            #'bppEuiccOtpk': None,           # whether otPK.EUICC.ECKA already used for binding the BPP, tag '5F49'
+            }
+        smdpSigned2_bin = rsp.asn1.encode('SmdpSigned2', smdpSigned2)
+
+        ss.smdpSignature2_do = b'\x5f\x37\x40' + self.dp_pb.ecdsa_sign(smdpSigned2_bin + b'\x5f\x37\x40' + euiccSignature1_bin)
+
+        # update non-volatile state with updated ss object
+        self.rss[transactionId] = ss
+        return {
+            'transactionId': transactionId,
+            'profileMetadata': profileMetadata,
+            'smdpSigned2': smdpSigned2,
+            'smdpSignature2': ss.smdpSignature2_do,
+            'smdpCertificate': self.dp_pb.get_cert_as_der(), # CERT.DPpb.SIG
+        }
+
+    @app.route('/gsma/rsp2/es9plus/authenticateClient', methods=['POST'])
+    @rsp_api_wrapper
+    def json_authenticateClient(self, request: IRequest, content: dict):
+        """Transform from JSON binding to generic function and back."""
+        b64decode_members(content, ['authenticateServerResponse', 'deleteNotifciationForDc'])
+        asn1decode_member(content, 'authenticateServerResponse')
+
+        output = self.authenticateClient(requeset, content)
+
+        asn1encode_member(output, 'smdpSigned2')
+        b64encode_members(output, ['profileMetadata', 'smdpSigned2', 'smdpSignature2', 'smdpCertificate'])
+        return output
+
+
+    def getBoundProfilePackage(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ GetBoundProfilePackage SGP.22 Section 5.6.2"""
+        transactionId = content['transactionId']
+
+        # Verify that the received transactionId is known and relates to an ongoing RSP session
+        ss = self.rss.get(transactionId, None)
+        if not ss:
+            raise ApiError('8.10.1', '3.9', 'The RSP session identified by the TransactionID is unknown')
+
+        prepDownloadResp = content['prepareDownloadResponse']
+        print("Rx %s: %s" % prepDownloadResp)
+
+        if prepDownloadResp[0] == 'downloadResponseError':
+            r_err = prepDownloadResp[1]
+            #r_err['transactionId']
+            #r_err['downloadErrorCode']
+            raise ValueError("downloadResponseError %s" % r_err)
+
+        r_ok = prepDownloadResp[1]
+
+        # Verify the euiccSignature2 computed over euiccSigned2 and smdpSignature2 using the PK.EUICC.SIG attached to the ongoing RSP session
+        euiccSigned2 = r_ok['euiccSigned2']
+        # TODO: use original data, don't re-encode?
+        euiccSigned2_bin = rsp.asn1.encode('EUICCSigned2', euiccSigned2)
+        if not self._ecdsa_verify(ss.euicc_cert, r_ok['euiccSignature2'], euiccSigned2_bin + ss.smdpSignature2_do):
+            raise ApiError('8.1', '6.1', 'eUICC signature is invalid')
+
+        # not in spec: Verify that signed TransactionID is outer transaction ID
+        if h2b(transactionId) != euiccSigned2['transactionId']:
+            raise ApiError('8.10.1', '3.9', 'The signed transactionId != outer transactionId')
+
+        # store otPK.EUICC.ECKA in session state
+        ss.euicc_otpk = euiccSigned2['euiccOtpk']
+        print("euiccOtpk: %s" % (b2h(ss.euicc_otpk)))
+
+        # Generate a one-time ECKA key pair (ot{PK,SK}.DP.ECKA) using the curve indicated by the Key Parameter
+        # Reference value of CERT.DPpb.ECDDSA
+        print("curve = %s" % self.dp_pb.get_curve())
+        ss.smdp_ot = ec.generate_private_key(self.dp_pb.get_curve())
+        # extract the public key in (hopefully) the right format for the ES8+ interface
+        ss.smdp_otpk = ss.smdp_ot.public_key().public_bytes(Encoding.X962, PublicFormat.UncompressedPoint)
+        print("smdpOtpk: %s" % b2h(ss.smdp_otpk))
+        print("smdpOtsk: %s" % b2h(ss.smdp_ot.private_bytes(Encoding.DER, PrivateFormat.PKCS8, NoEncryption())))
+
+        ss.host_id = b'mahlzeit'
+
+        # Generate Session Keys using the CRT, opPK.eUICC.ECKA and otSK.DP.ECKA according to annex G
+        euicc_public_key = ec.EllipticCurvePublicKey.from_encoded_point(ss.smdp_ot.curve, ss.euicc_otpk)
+        ss.shared_secret = ss.smdp_ot.exchange(ec.ECDH(), euicc_public_key)
+        print("shared_secret: %s" % b2h(ss.shared_secret))
+
+        # TODO: Check if this order requires a Confirmation Code verification
+
+        #  Perform actual protection + binding of profile package (or return  pre-bound one)
+        with open(os.path.join(DATA_DIR, 'upp', 'TS48 V2 eSIM_GTP_SAIP2.1_NoBERTLV.rename2der'), 'rb') as f:
+            upp = UnprotectedProfilePackage.from_der(f.read(), metadata=ss.profileMetadata)
+            # HACK: Use empty PPP as we're still debuggin the configureISDP step, and we want to avoid
+            # cluttering the log with stuff happening after the failure
+            #upp = UnprotectedProfilePackage.from_der(b'', metadata=ss.profileMetadata)
+        if False:
+            # Use random keys
+            bpp = BoundProfilePackage.from_upp(upp)
+        else:
+            # Use sesssion keys
+            ppp = ProtectedProfilePackage.from_upp(upp, BspInstance(b'\x00'*16, b'\x11'*16, b'\x22'*16))
+            bpp = BoundProfilePackage.from_ppp(ppp)
+
+        # update non-volatile state with updated ss object
+        self.rss[transactionId] = ss
+        return {
+            'transactionId': transactionId,
+            'boundProfilePackage': bpp.encode(ss, self.dp_pb),
+        }
+
+
+    @app.route('/gsma/rsp2/es9plus/getBoundProfilePackage', methods=['POST'])
+    @rsp_api_wrapper
+    def json_getBoundProfilePackage(self, request: IRequest, content: dict):
+        """Transform from JSON binding to generic function and back."""
+        b64decode_members(content, ['prepareDownloadResponse'])
+        asn1decode_member(content, 'prepareDownlaodResponse')
+
+        output = self.getBoundProfilePackage(request, content)
+
+        asn1encode_member(output, 'boundProfilePackage')
+        b64encode_members(output, ['boundProfilePackage'])
+        return output
+
+    def handleNotification(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ HandleNotification in SGP.22 Section 5.6.4"""
+        pendingNotification = content['pendingNotification']
+        print("Rx %s: %s" % pendingNotification)
+        if pendingNotification[0] == 'profileInstallationResult':
+            profileInstallRes = pendingNotification[1]
+            pird = profileInstallRes['profileInstallationResultData']
+            transactionId = b2h(pird['transactionId'])
+            ss = self.rss.get(transactionId, None)
+            if ss is None:
+                print("Unable to find session for transactionId")
+                return
+            profileInstallRes['euiccSignPIR']
+            # TODO: use original data, don't re-encode?
+            pird_bin = rsp.asn1.encode('ProfileInstallationResultData', pird)
+            # verify eUICC signature
+            if not self._ecdsa_verify(ss.euicc_cert, profileInstallRes['euiccSignPIR'], pird_bin):
+                print("Unable to verify eUICC signature")
+            print("Profile Installation Final Result: ", pird['finalResult'])
+            # remove session state
+            del self.rss[transactionId]
+        elif pendingNotification[0] == 'otherSignedNotification':
+            # TODO
+            pass
+        else:
+            raise ValueError(pendingNotification)
+        return None
+
+    @app.route('/gsma/rsp2/es9plus/handleNotification', methods=['POST'])
+    @rsp_api_wrapper
+    def json_handleNotification(self, request: IRequest, content: dict):
+        """Transform from JSON binding to generic function and back."""
+        b64decode_members(content, ['pendingNotification'])
+        asn1decode_member(content, 'pendingNotification')
+
+        output = self.handleNotification(request, content)
+
+        return output
+
+
+    #@app.route('/gsma/rsp3/es9plus/handleDeviceChangeRequest, methods=['POST']')
+    #@rsp_api_wrapper
+        #"""See ES9+ ConfirmDeviceChange in SGP.22 Section 5.6.6"""
+        # TODO: implement this
+
+    def cancelSession(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ CancelSession in SGP.22 Section 5.6.5"""
+        transactionId = content['transactionId']
+
+        # Verify that the received transactionId is known and relates to an ongoing RSP session
+        ss = self.rss.get(transactionId, None)
+        if ss is None:
+            raise ApiError('8.10.1', '3.9', 'The RSP session identified by the transactionId is unknown')
+
+        cancelSessionResponse = content['cancelSessionResponse']
+        print("Rx %s: %s" % cancelSessionResponse)
+
+        if cancelSessionResponse[0] == 'cancelSessionResponseError':
+            # FIXME: print some error
+            return
+        cancelSessionResponseOk = cancelSessionResponse[1]
+        # TODO: use original data, don't re-encode?
+        ecsr = cancelSessionResponseOk['euiccCancelSessionSigned']
+        ecsr_bin = rsp.asn1.encode('EuiccCancelSessionSigned', ecsr)
+        # Verify the eUICC signature (euiccCancelSessionSignature) using the PK.EUICC.SIG attached to the ongoing RSP session
+        if not self._ecdsa_verify(ss.euicc_cert, cancelSessionResponseOk['euiccCancelSessionSignature'], ecsr_bin):
+            raise ApiError('8.1', '6.1', 'eUICC signature is invalid')
+
+        # Verify that the received smdpOid corresponds to the one in SM-DP+ CERT.DPauth.SIG
+        subj_alt_name = self.dp_auth.get_subject_alt_name()
+        if x509.ObjectIdentifier(ecsr['smdpOid']) != subj_alt_name.oid:
+            raise ApiError('8.8', '3.10', 'The provided SM-DP+ OID is invalid.')
+
+        if ecsr['transactionId'] != h2b(transactionId):
+            raise ApiError('8.10.1', '3.9', 'The signed transactionId != outer transactionId')
+
+        # TODO: 1. Notify the Operator using the function "ES2+.HandleNotification" function
+        # TODO: 2. Terminate the corresponding pending download process.
+        # TODO: 3. If required, execute the SM-DS Event Deletion procedure described in section 3.6.3.
+
+        # delete actual session data
+        del self.rss[transactionId]
+        return { 'transactionId': transactionId }
+
+    @app.route('/gsma/rsp2/es9plus/cancelSession', methods=['POST'])
+    @rsp_api_wrapper
+    def json_cancelSessionn(self, request: IRequest, content: dict) -> dict:
+        """Transform from JSON binding to generic function and back."""
+        b64decode_members(content, ['cancelSessionResponse'])
+        asn1decode_member(content, 'cancelSessionResponse')
+
+        output = self.cancelSession(request, content)
+
+        return output
+
+
+    @app.route('/gsma/rsp2/asn1', methods=['POST'])
+    def asn1(self, request: IRequest) -> dict:
+        # TODO: evaluate User-Agent + X-Admin-Protocol header
+        # TODO: reject any non-ASN.1 Content-type
+
+        content = rsp.asn1.decode('RemoteProfileProvisioningRequest', request.content.read())
+        print("Rx ASN.1: %s" % content)
+        request.setHeader('Content-Type', 'application/x-gsma-rsp-asn1')
+        request.setHeader('X-Admin-Protocol', 'gsma/rsp/v2.1.0')
+
+        operation = content[0]
+        if operation == 'initiateAuthenticationRequest':
+            method = self.initiateAuthentication
+            ochoice = 'initiateAuthenticationResponse'
+        elif operation == 'authenticateClientRequest':
+            method = self.authenticateClient
+            ochoice = 'authenticateClientResponseEs9'
+        elif operation == 'getBoundProfilePackageRequest':
+            method = self.getBoundProfilePackage
+            ochoice = 'getBoundProfilePackageResponse'
+        elif operation == 'cancelSessionRequestEs9':
+            method = self.cancelSession
+            ochoice = 'cancelSessionResponseEs9'
+        elif operation == 'handleNotification':
+            method = self.handleNotification
+            ochoice = None
+
+        output = method(self, request, content[1])
+
+        if ochoice:
+            output = (ochoice, output)
+            print("Tx ASN.1: %s" % output)
+            return rsp.asn1.encode('RemoteProfileProvisioningResponse', output)
+        else:
+            return None
+
+
+def main(argv):
+    parser = argparse.ArgumentParser()
+    #parser.add_argument("-H", "--host", help="Host/IP to bind HTTP to", default="localhost")
+    #parser.add_argument("-p", "--port", help="TCP port to bind HTTP to", default=8000)
+    #parser.add_argument("-v", "--verbose", help="increase output verbosity", action='count', default=0)
+
+    args = parser.parse_args()
+
+    hs = SmDppHttpServer(HOSTNAME, os.path.join(DATA_DIR, 'certs', 'CertificateIssuer'), use_brainpool=True)
+    #hs.app.run(endpoint_description="ssl:port=8000:dhParameters=dh_param_2048.pem")
+    hs.app.run("localhost", 8000)
+
+if __name__ == "__main__":
+    main(sys.argv)

pySim-prog.py

@@ -32,12 +32,14 @@ import re
 import sys
 import traceback
 import json
+import csv
 
 from pySim.commands import SimCardCommands
 from pySim.transport import init_reader
-from pySim.cards import _cards_classes, card_detect
+from pySim.legacy.cards import _cards_classes, card_detect
 from pySim.utils import h2b, swap_nibbles, rpad, derive_milenage_opc, calculate_luhn, dec_iccid
-from pySim.ts_51_011 import EF, EF_AD
+from pySim.ts_51_011 import EF_AD
+from pySim.legacy.ts_51_011 import EF
 from pySim.card_handler import *
 from pySim.utils import *
 
@@ -113,8 +115,8 @@ def parse_options():
                       )
     parser.add_option("--mnclen", dest="mnclen", type="choice",
                       help="Length of Mobile Network Code [default: %default]",
-                      default=2,
-                      choices=[2, 3],
+                      default="auto",
+                      choices=["2", "3", "auto"],
                       )
     parser.add_option("-m", "--smsc", dest="smsc",
                       help="SMSC number (Start with + for international no.) [default: '00 + country code + 5555']",
@@ -149,6 +151,9 @@ def parse_options():
                       default=None,
                       choices=['{:02X}'.format(int(m)) for m in EF_AD.OP_MODE],
                       )
+    parser.add_option("-f", "--fplmn", dest="fplmn", action="append",
+                      help="Set Forbidden PLMN. Add multiple time for multiple FPLMNS",
+                      )
     parser.add_option("--epdgid", dest="epdgid",
                       help="Set Home Evolved Packet Data Gateway (ePDG) Identifier. (Only FQDN format supported)",
                       )
@@ -318,8 +323,15 @@ def gen_parameters(opts):
 
     # MCC always has 3 digits
     mcc = lpad(mcc, 3, "0")
-    # MNC must be at least 2 digits
-    mnc = lpad(mnc, 2, "0")
+
+    # The MNC must be at least 2 digits long. This is also the most common case.
+    # The user may specify an explicit length using the --mnclen option.
+    if opts.mnclen != "auto":
+        if len(mnc) > int(opts.mnclen):
+            raise ValueError('mcc is longer than specified in option --mnclen')
+        mnc = lpad(mnc, int(opts.mnclen), "0")
+    else:
+        mnc = lpad(mnc, 2, "0")
 
     # Digitize country code (2 or 3 digits)
     cc_digits = _cc_digits(opts.country)
@@ -346,8 +358,8 @@ def gen_parameters(opts):
     # ICCID (19 digits, E.118), though some phase1 vendors use 20 :(
     if opts.iccid is not None:
         iccid = opts.iccid
-        if not _isnum(iccid, 19) and not _isnum(iccid, 20):
-            raise ValueError('ICCID must be 19 or 20 digits !')
+        if not _isnum(iccid, 18) and not _isnum(iccid, 19) and not _isnum(iccid, 20):
+            raise ValueError('ICCID must be 18, 19 or 20 digits !')
 
     else:
         if opts.num is None:
@@ -490,6 +502,7 @@ def gen_parameters(opts):
         'impi': opts.impi,
         'impu': opts.impu,
         'opmode': opts.opmode,
+        'fplmn': opts.fplmn,
     }
 
 
@@ -524,40 +537,86 @@ def write_params_csv(opts, params):
         f.close()
 
 
-def _read_params_csv(opts, iccid=None, imsi=None):
-    import csv
-    f = open(opts.read_csv, 'r')
+def find_row_in_csv_file(csv_file_name:str, num=None, iccid=None, imsi=None):
+    """
+    Pick a matching row in a CSV file by row number or ICCID or IMSI. When num
+    is not None, the search parameters iccid and imsi are ignored. When
+    searching for a specific ICCID or IMSI the caller must set num to None. It
+    is possible to search for an ICCID or an IMSI at the same time. The first
+    line that either contains a matching ICCID or IMSI is returned. Unused
+    search parameters must be set to None.
+    """
+
+    f = open(csv_file_name, 'r')
     cr = csv.DictReader(f)
 
+    # Make sure the CSV file contains at least the fields we are searching for
+    if not 'iccid' in cr.fieldnames:
+        raise Exception("wrong CSV file format - no field \"iccid\" or missing header!")
+    if not 'imsi' in cr.fieldnames:
+        raise Exception("wrong CSV file format - no field \"imsi\" or missing header!")
+
+    # Enforce at least one search parameter
+    if not num and not iccid and not imsi:
+        raise Exception("no CSV file search parameters!")
+
     # Lower-case fieldnames
     cr.fieldnames = [field.lower() for field in cr.fieldnames]
 
     i = 0
-    if not 'iccid' in cr.fieldnames:
-        raise Exception("CSV file in wrong format!")
     for row in cr:
-        if opts.num is not None and opts.read_iccid is False and opts.read_imsi is False:
-            if opts.num == i:
+        # Pick a specific row by line number (num)
+        if num is not None and iccid is None and imsi is None:
+            if num == i:
                 f.close()
                 return row
-            i += 1
+
+	# Pick the first row that contains the specified ICCID
         if row['iccid'] == iccid:
             f.close()
             return row
 
+	# Pick the first row that contains the specified IMSI
         if row['imsi'] == imsi:
             f.close()
             return row
 
+        i += 1
+
     f.close()
+    print("Could not read card parameters from CSV file, no matching entry found.")
     return None
 
 
 def read_params_csv(opts, imsi=None, iccid=None):
-    row = _read_params_csv(opts, iccid=iccid, imsi=imsi)
+    """
+    Read the card parameters from a CSV file. This function will generate the
+    same dictionary that gen_parameters would generate from parameters passed as
+    commandline arguments.
+    """
+
+    row = find_row_in_csv_file(opts.read_csv, opts.num, iccid=iccid, imsi=imsi)
     if row is not None:
         row['mcc'] = row.get('mcc', mcc_from_imsi(row.get('imsi')))
-        row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi')))
+
+        # We cannot determine the MNC length (2 or 3 digits?) from the IMSI
+        # alone. In cases where the user has specified an mnclen via the
+        # commandline options we can use that info, otherwise we guess that
+        # the length is 2, which is also the most common case.
+        if opts.mnclen != "auto":
+            if opts.mnclen == "2":
+                row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), False))
+            elif opts.mnclen == "3":
+                row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), True))
+            else:
+                raise ValueError("invalid parameter --mnclen, must be 2 or 3 or auto")
+        else:
+            row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), False))
+
+	# NOTE: We might concider to specify a new CSV field "mnclen" in our
+	# CSV files for a better automatization. However, this only makes sense
+	# when the tools and databases we export our files from will also add
+	# such a field.
 
         pin_adm = None
         # We need to escape the pin_adm we get from the csv
@@ -621,7 +680,7 @@ def write_params_hlr(opts, params):
         conn.close()
 
 
-def write_parameters(opts, params):
+def write_parameters_to_csv_and_hlr(opts, params):
     write_params_csv(opts, params)
     write_params_hlr(opts, params)
 
@@ -668,24 +727,23 @@ def save_batch(opts):
     fh.close()
 
 
-def process_card(opts, first, ch):
+def process_card(scc, opts, first, ch):
 
+    # Connect transport
+    ch.get(first)
+
+    # Get card
+    card = card_detect(opts.type, scc)
+    if card is None:
+        print("No card detected!")
+        return -1
+
+    # Probe only
+    if opts.probe:
+        return 0
+
+    # Erase if requested (not in dry run mode!)
     if opts.dry_run is False:
-        # Connect transport
-        ch.get(first)
-
-    if opts.dry_run is False:
-        # Get card
-        card = card_detect(opts.type, scc)
-        if card is None:
-            print("No card detected!")
-            return -1
-
-        # Probe only
-        if opts.probe:
-            return 0
-
-        # Erase if requested
         if opts.erase:
             print("Formatting ...")
             card.erase()
@@ -698,22 +756,15 @@ def process_card(opts, first, ch):
         imsi = None
         iccid = None
         if opts.read_iccid:
-            if opts.dry_run:
-                # Connect transport
-                ch.get(False)
             (res, _) = scc.read_binary(['3f00', '2fe2'], length=10)
             iccid = dec_iccid(res)
         elif opts.read_imsi:
-            if opts.dry_run:
-                # Connect transport
-                ch.get(False)
             (res, _) = scc.read_binary(EF['IMSI'])
             imsi = swap_nibbles(res)[3:]
         else:
             imsi = opts.imsi
         cp = read_params_csv(opts, imsi=imsi, iccid=iccid)
     if cp is None:
-        print("Error reading parameters from CSV file!\n")
         return 2
     print_parameters(cp)
 
@@ -724,8 +775,8 @@ def process_card(opts, first, ch):
     else:
         print("Dry Run: NOT PROGRAMMING!")
 
-    # Write parameters permanently
-    write_parameters(opts, cp)
+    # Write parameters to a specified CSV file or an HLR database (not the card)
+    write_parameters_to_csv_and_hlr(opts, cp)
 
     # Batch mode state update and save
     if opts.num is not None:
@@ -743,8 +794,6 @@ if __name__ == '__main__':
 
     # Init card reader driver
     sl = init_reader(opts)
-    if sl is None:
-        exit(1)
 
     # Create command layer
     scc = SimCardCommands(transport=sl)
@@ -770,7 +819,7 @@ if __name__ == '__main__':
 
     while 1:
         try:
-            rc = process_card(opts, first, ch)
+            rc = process_card(scc, opts, first, ch)
         except (KeyboardInterrupt):
             print("")
             print("Terminated by user!")

pySim-read.py

@@ -28,20 +28,22 @@ import os
 import random
 import re
 import sys
-from pySim.ts_51_011 import EF, DF, EF_SST_map, EF_AD
-from pySim.ts_31_102 import EF_UST_map, EF_USIM_ADF_map
-from pySim.ts_31_103 import EF_IST_map, EF_ISIM_ADF_map
+
+from pySim.ts_51_011 import EF_SST_map, EF_AD
+from pySim.legacy.ts_51_011 import EF, DF
+from pySim.ts_31_102 import EF_UST_map
+from pySim.legacy.ts_31_102 import EF_USIM_ADF_map
+from pySim.ts_31_103 import EF_IST_map
+from pySim.legacy.ts_31_103 import EF_ISIM_ADF_map
 
 from pySim.commands import SimCardCommands
 from pySim.transport import init_reader, argparse_add_reader_args
 from pySim.exceptions import SwMatchError
-from pySim.cards import card_detect, SimCard, UsimCard, IsimCard
-from pySim.utils import h2b, swap_nibbles, rpad, dec_imsi, dec_iccid, dec_msisdn
-from pySim.utils import format_xplmn_w_act, dec_st
-from pySim.utils import h2s, format_ePDGSelection
+from pySim.legacy.cards import card_detect, SimCard, UsimCard, IsimCard
+from pySim.utils import h2b, h2s, swap_nibbles, rpad, dec_imsi, dec_iccid, dec_msisdn
+from pySim.legacy.utils import format_xplmn_w_act, dec_st
 
-option_parser = argparse.ArgumentParser(prog='pySim-read',
-                                        description='Legacy tool for reading some parts of a SIM card',
+option_parser = argparse.ArgumentParser(description='Legacy tool for reading some parts of a SIM card',
                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 argparse_add_reader_args(option_parser)
 
@@ -72,8 +74,6 @@ if __name__ == '__main__':
 
     # Init card reader driver
     sl = init_reader(opts)
-    if sl is None:
-        exit(1)
 
     # Create command layer
     scc = SimCardCommands(transport=sl)
@@ -253,6 +253,14 @@ if __name__ == '__main__':
             else:
                 print("EHPLMN: Can't read, response code = %s" % (sw,))
 
+        # EF.FPLMN
+        if usim_card.file_exists(EF_USIM_ADF_map['FPLMN']):
+            res, sw = usim_card.read_fplmn()
+            if sw == '9000':
+                print(f'FPLMN:\n{res}')
+            else:
+                print(f'FPLMN: Can\'t read, response code = {sw}')
+
         # EF.UST
         try:
             if usim_card.file_exists(EF_USIM_ADF_map['UST']):

pySim-shell.py

@@ -2,7 +2,7 @@
 
 # Interactive shell for working with SIM / UICC / USIM / ISIM cards
 #
-# (C) 2021-2022 by Harald Welte <laforge@osmocom.org>
+# (C) 2021-2023 by Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,125 +17,87 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from typing import List
+from typing import List, Optional
 
 import json
 import traceback
 
 import cmd2
-from cmd2 import style, fg
+from packaging import version
+from cmd2 import style
+# cmd2 >= 2.3.0 has deprecated the bg/fg in favor of Bg/Fg :(
+if version.parse(cmd2.__version__) < version.parse("2.3.0"):
+    from cmd2 import fg, bg # pylint: disable=no-name-in-module
+    RED = fg.red
+    LIGHT_RED = fg.bright_red
+    LIGHT_GREEN = fg.bright_green
+else:
+    from cmd2 import Fg, Bg # pylint: disable=no-name-in-module
+    RED = Fg.RED
+    LIGHT_RED = Fg.LIGHT_RED
+    LIGHT_GREEN = Fg.LIGHT_GREEN
 from cmd2 import CommandSet, with_default_category, with_argparser
 import argparse
 
 import os
 import sys
+import inspect
 from pathlib import Path
 from io import StringIO
 
 from pprint import pprint as pp
 
 from pySim.exceptions import *
-from pySim.commands import SimCardCommands
 from pySim.transport import init_reader, ApduTracer, argparse_add_reader_args, ProactiveHandler
-from pySim.cards import card_detect, SimCard
-from pySim.utils import h2b, swap_nibbles, rpad, b2h, JsonEncoder, bertlv_parse_one, sw_match
-from pySim.utils import sanitize_pin_adm, tabulate_str_list, boxed_heading_str, Hexstr
+from pySim.utils import h2b, b2h, i2h, swap_nibbles, rpad, JsonEncoder, bertlv_parse_one, sw_match
+from pySim.utils import sanitize_pin_adm, tabulate_str_list, boxed_heading_str, Hexstr, dec_iccid
+from pySim.utils import is_hexstr_or_decimal, is_hexstr, is_decimal
 from pySim.card_handler import CardHandler, CardHandlerAuto
 
-from pySim.filesystem import RuntimeState, CardDF, CardADF, CardModel
-from pySim.profile import CardProfile
-from pySim.ts_102_221 import CardProfileUICC
+from pySim.filesystem import CardMF, CardDF, CardADF
 from pySim.ts_102_222 import Ts102222Commands
-from pySim.ts_31_102 import CardApplicationUSIM
-from pySim.ts_31_103 import CardApplicationISIM
-from pySim.ara_m import CardApplicationARAM
-from pySim.global_platform import CardApplicationISD
 from pySim.gsm_r import DF_EIRENE
 from pySim.cat import ProactiveCommand
 
-# we need to import this module so that the SysmocomSJA2 sub-class of
-# CardModel is created, which will add the ATR-based matching and
-# calling of SysmocomSJA2.add_files.  See  CardModel.apply_matching_models
-import pySim.sysmocom_sja2
-
 from pySim.card_key_provider import CardKeyProviderCsv, card_key_provider_register, card_key_provider_get_field
 
-
-def init_card(sl):
-    """
-    Detect card in reader and setup card profile and runtime state. This
-    function must be called at least once on startup. The card and runtime
-    state object (rs) is required for all pySim-shell commands.
-    """
-
-    # Wait up to three seconds for a card in reader and try to detect
-    # the card type.
-    print("Waiting for card...")
-    try:
-        sl.wait_for_card(3)
-    except NoCardError:
-        print("No card detected!")
-        return None, None
-    except:
-        print("Card not readable!")
-        return None, None
-
-    generic_card = False
-    card = card_detect("auto", scc)
-    if card is None:
-        print("Warning: Could not detect card type - assuming a generic card type...")
-        card = SimCard(scc)
-        generic_card = True
-
-    profile = CardProfile.pick(scc)
-    if profile is None:
-        print("Unsupported card type!")
-        return None, card
-
-    # ETSI TS 102 221, Table 9.3 specifies a default for the PIN key
-    # references, however card manufactures may still decide to pick an
-    # arbitrary key reference. In case we run on a generic card class that is
-    # detected as an UICC, we will pick the key reference that is officially
-    # specified.
-    if generic_card and isinstance(profile, CardProfileUICC):
-        card._adm_chv_num = 0x0A
-
-    print("Info: Card is of type: %s" % str(profile))
-
-    # FIXME: This shouln't be here, the profile should add the applications,
-    # however, we cannot simply put his into ts_102_221.py since we would
-    # have to e.g. import CardApplicationUSIM from ts_31_102.py, which already
-    # imports from ts_102_221.py. This means we will end up with a circular
-    # import, which needs to be resolved first.
-    if isinstance(profile, CardProfileUICC):
-        profile.add_application(CardApplicationUSIM())
-        profile.add_application(CardApplicationISIM())
-        profile.add_application(CardApplicationARAM())
-        profile.add_application(CardApplicationISD())
-
-    # Create runtime state with card profile
-    rs = RuntimeState(card, profile)
-
-    # FIXME: This is an GSM-R related file, it needs to be added throughout,
-    # the profile. At the moment we add it for all cards, this won't hurt,
-    # but regular SIM and UICC will not have it and fail to select it.
-    rs.mf.add_file(DF_EIRENE())
-
-    CardModel.apply_matching_models(scc, rs)
-
-    # inform the transport that we can do context-specific SW interpretation
-    sl.set_sw_interpreter(rs)
-
-    return rs, card
+from pySim.app import init_card
 
 
-class PysimApp(cmd2.Cmd):
+class Cmd2Compat(cmd2.Cmd):
+    """Backwards-compatibility wrapper around cmd2.Cmd to support older and newer
+    releases. See https://github.com/python-cmd2/cmd2/blob/master/CHANGELOG.md"""
+    def run_editor(self, file_path: Optional[str] = None) -> None:
+        if version.parse(cmd2.__version__) < version.parse("2.0.0"):
+            return self._run_editor(file_path) # pylint: disable=no-member
+        else:
+            return super().run_editor(file_path) # pylint: disable=no-member
+
+class Settable2Compat(cmd2.Settable):
+    """Backwards-compatibility wrapper around cmd2.Settable to support older and newer
+    releases. See https://github.com/python-cmd2/cmd2/blob/master/CHANGELOG.md"""
+    def __init__(self, name, val_type, description, settable_object, **kwargs):
+        if version.parse(cmd2.__version__) < version.parse("2.0.0"):
+            super().__init__(name, val_type, description, **kwargs) # pylint: disable=no-value-for-parameter
+        else:
+            super().__init__(name, val_type, description, settable_object, **kwargs) # pylint: disable=too-many-function-args
+
+class PysimApp(Cmd2Compat):
     CUSTOM_CATEGORY = 'pySim Commands'
+    BANNER = """Welcome to pySim-shell!
+(C) 2021-2023 by Harald Welte, sysmocom - s.f.m.c. GmbH and contributors
+Online manual available at https://downloads.osmocom.org/docs/pysim/master/html/shell.html """
 
     def __init__(self, card, rs, sl, ch, script=None):
+        if version.parse(cmd2.__version__) < version.parse("2.0.0"):
+            kwargs = {'use_ipython': True}
+        else:
+            kwargs = {'include_ipy': True}
+
+        # pylint: disable=unexpected-keyword-arg
         super().__init__(persistent_history_file='~/.pysim_shell_history', allow_cli_args=False,
-                         use_ipython=True, auto_load_commands=False, startup_script=script)
-        self.intro = style('Welcome to pySim-shell!', fg=fg.red)
+                         auto_load_commands=False, startup_script=script, **kwargs)
+        self.intro = style(self.BANNER, fg=RED)
         self.default_category = 'pySim-shell built-in commands'
         self.card = None
         self.rs = None
@@ -145,18 +107,17 @@ class PysimApp(cmd2.Cmd):
         self.ch = ch
 
         self.numeric_path = False
-        self.add_settable(cmd2.Settable('numeric_path', bool, 'Print File IDs instead of names',
-                                        onchange_cb=self._onchange_numeric_path))
         self.conserve_write = True
-        self.add_settable(cmd2.Settable('conserve_write', bool, 'Read and compare before write',
-                                        onchange_cb=self._onchange_conserve_write))
         self.json_pretty_print = True
-        self.add_settable(cmd2.Settable('json_pretty_print',
-                          bool, 'Pretty-Print JSON output'))
         self.apdu_trace = False
-        self.add_settable(cmd2.Settable('apdu_trace', bool, 'Trace and display APDUs exchanged with card',
-                                        onchange_cb=self._onchange_apdu_trace))
 
+        self.add_settable(Settable2Compat('numeric_path', bool, 'Print File IDs instead of names', self,
+                                          onchange_cb=self._onchange_numeric_path))
+        self.add_settable(Settable2Compat('conserve_write', bool, 'Read and compare before write', self,
+                                          onchange_cb=self._onchange_conserve_write))
+        self.add_settable(Settable2Compat('json_pretty_print', bool, 'Pretty-Print JSON output', self))
+        self.add_settable(Settable2Compat('apdu_trace', bool, 'Trace and display APDUs exchanged with card', self,
+                                          onchange_cb=self._onchange_apdu_trace))
         self.equip(card, rs)
 
     def equip(self, card, rs):
@@ -171,7 +132,11 @@ class PysimApp(cmd2.Cmd):
         if self.rs:
             lchan = self.rs.lchan[0]
             lchan.unregister_cmds(self)
-        for cmds in [Iso7816Commands, PySimCommands]:
+            if self.rs.profile:
+                for cmd_set in self.rs.profile.shell_cmdsets:
+                    self.unregister_command_set(cmd_set)
+
+        for cmds in [Iso7816Commands, Ts102222Commands, PySimCommands]:
             cmd_set = self.find_commandsets(cmds)
             if cmd_set:
                 self.unregister_command_set(cmd_set[0])
@@ -186,10 +151,19 @@ class PysimApp(cmd2.Cmd):
             self._onchange_conserve_write(
                 'conserve_write', False, self.conserve_write)
             self._onchange_apdu_trace('apdu_trace', False, self.apdu_trace)
+            if self.rs.profile:
+                for cmd_set in self.rs.profile.shell_cmdsets:
+                    self.register_command_set(cmd_set)
             self.register_command_set(Iso7816Commands())
             self.register_command_set(Ts102222Commands())
             self.register_command_set(PySimCommands())
-            self.iccid, sw = self.card.read_iccid()
+
+            try:
+                self.lchan.select('MF/EF.ICCID', self)
+                self.iccid = dec_iccid(self.lchan.read_binary()[0])
+            except:
+                self.iccid = None
+
             self.lchan.select('MF', self)
             rc = True
         else:
@@ -222,7 +196,7 @@ class PysimApp(cmd2.Cmd):
 
     class Cmd2ApduTracer(ApduTracer):
         def __init__(self, cmd2_app):
-            self.cmd2 = app
+            self.cmd2 = cmd2_app
 
         def trace_response(self, cmd, sw, resp):
             self.cmd2.poutput("-> %s %s" % (cmd[:10], cmd[10:]))
@@ -231,7 +205,7 @@ class PysimApp(cmd2.Cmd):
     def update_prompt(self):
         if self.lchan:
             path_str = self.lchan.selected_file.fully_qualified_path_str(not self.numeric_path)
-            self.prompt = 'pySIM-shell (%s)> ' % (path_str)
+            self.prompt = 'pySIM-shell (%02u:%s)> ' % (self.lchan.lchan_nr, path_str)
         else:
             if self.card:
                 self.prompt = 'pySIM-shell (no card profile)> '
@@ -250,19 +224,27 @@ class PysimApp(cmd2.Cmd):
     @cmd2.with_category(CUSTOM_CATEGORY)
     def do_equip(self, opts):
         """Equip pySim-shell with card"""
-        rs, card = init_card(sl)
+        if self.rs and self.rs.profile:
+            for cmd_set in self.rs.profile.shell_cmdsets:
+                self.unregister_command_set(cmd_set)
+        rs, card = init_card(self.sl)
         self.equip(card, rs)
 
     apdu_cmd_parser = argparse.ArgumentParser()
-    apdu_cmd_parser.add_argument('APDU', type=str, help='APDU as hex string')
+    apdu_cmd_parser.add_argument('APDU', type=is_hexstr, help='APDU as hex string')
     apdu_cmd_parser.add_argument('--expect-sw', help='expect a specified status word', type=str, default=None)
 
     @cmd2.with_argparser(apdu_cmd_parser)
     def do_apdu(self, opts):
         """Send a raw APDU to the card, and print SW + Response.
-        DANGEROUS: pySim-shell will not know any card state changes, and
-        not continue to work as expected if you e.g. select a different
-        file."""
+        CAUTION: this command bypasses the logical channel handling of pySim-shell and card state changes are not
+        tracked. Dpending on the raw APDU sent, pySim-shell may not continue to work as expected if you e.g. select
+        a different file."""
+
+        # When sending raw APDUs we access the scc object through _scc member of the card object. It should also be
+        # noted that the apdu command plays an exceptional role since it is the only card accessing command that
+        # can be executed without the presence of a runtime state (self.rs) object. However, this also means that
+        # self.lchan is also not present (see method equip).
         data, sw = self.card._scc._tp.send_apdu(opts.APDU)
         if data:
             self.poutput("SW: %s, RESP: %s" % (sw, data))
@@ -287,23 +269,23 @@ class PysimApp(cmd2.Cmd):
             sys.stderr = self._stderr_backup
 
     def _show_failure_sign(self):
-        self.poutput(style("  +-------------+", fg=fg.bright_red))
-        self.poutput(style("  +   ##   ##   +", fg=fg.bright_red))
-        self.poutput(style("  +    ## ##    +", fg=fg.bright_red))
-        self.poutput(style("  +     ###     +", fg=fg.bright_red))
-        self.poutput(style("  +    ## ##    +", fg=fg.bright_red))
-        self.poutput(style("  +   ##   ##   +", fg=fg.bright_red))
-        self.poutput(style("  +-------------+", fg=fg.bright_red))
+        self.poutput(style("  +-------------+", fg=LIGHT_RED))
+        self.poutput(style("  +   ##   ##   +", fg=LIGHT_RED))
+        self.poutput(style("  +    ## ##    +", fg=LIGHT_RED))
+        self.poutput(style("  +     ###     +", fg=LIGHT_RED))
+        self.poutput(style("  +    ## ##    +", fg=LIGHT_RED))
+        self.poutput(style("  +   ##   ##   +", fg=LIGHT_RED))
+        self.poutput(style("  +-------------+", fg=LIGHT_RED))
         self.poutput("")
 
     def _show_success_sign(self):
-        self.poutput(style("  +-------------+", fg=fg.bright_green))
-        self.poutput(style("  +          ## +", fg=fg.bright_green))
-        self.poutput(style("  +         ##  +", fg=fg.bright_green))
-        self.poutput(style("  +  #    ##    +", fg=fg.bright_green))
-        self.poutput(style("  +   ## #      +", fg=fg.bright_green))
-        self.poutput(style("  +    ##       +", fg=fg.bright_green))
-        self.poutput(style("  +-------------+", fg=fg.bright_green))
+        self.poutput(style("  +-------------+", fg=LIGHT_GREEN))
+        self.poutput(style("  +          ## +", fg=LIGHT_GREEN))
+        self.poutput(style("  +         ##  +", fg=LIGHT_GREEN))
+        self.poutput(style("  +  #    ##    +", fg=LIGHT_GREEN))
+        self.poutput(style("  +   ## #      +", fg=LIGHT_GREEN))
+        self.poutput(style("  +    ##       +", fg=LIGHT_GREEN))
+        self.poutput(style("  +-------------+", fg=LIGHT_GREEN))
         self.poutput("")
 
     def _process_card(self, first, script_path):
@@ -314,7 +296,7 @@ class PysimApp(cmd2.Cmd):
             rc = self.equip(card, rs)
         except:
             self.poutput("")
-            self.poutput("Card initialization failed with an exception:")
+            self.poutput("Card initialization (%s) failed with an exception:" % str(self.sl))
             self.poutput("---------------------8<---------------------")
             traceback.print_exc()
             self.poutput("---------------------8<---------------------")
@@ -383,7 +365,7 @@ class PysimApp(cmd2.Cmd):
                 # In case of failure, try multiple times.
                 for i in range(opts.tries):
                     # fetch card into reader bay
-                    ch.get(first)
+                    self.ch.get(first)
 
                     # if necessary execute an action before we start processing the card
                     if(opts.pre_card_action):
@@ -406,9 +388,9 @@ class PysimApp(cmd2.Cmd):
                 # Depending on success or failure, the card goes either in the "error" bin or in the
                 # "done" bin.
                 if rc < 0:
-                    ch.error()
+                    self.ch.error()
                 else:
-                    ch.done()
+                    self.ch.done()
 
                 # In most cases it is possible to proceed with the next card, but the
                 # user may decide to halt immediately when an error occurs
@@ -429,7 +411,7 @@ class PysimApp(cmd2.Cmd):
                 return
             except:
                 self.poutput("")
-                self.poutput("Card handling failed with an exception:")
+                self.poutput("Card handling (%s) failed with an exception:" % str(self.sl))
                 self.poutput("---------------------8<---------------------")
                 traceback.print_exc()
                 self.poutput("---------------------8<---------------------")
@@ -442,13 +424,13 @@ class PysimApp(cmd2.Cmd):
             first = False
 
     echo_parser = argparse.ArgumentParser()
-    echo_parser.add_argument('string', help="string to echo on the shell")
+    echo_parser.add_argument('string', help="string to echo on the shell", nargs='+')
 
     @cmd2.with_argparser(echo_parser)
     @cmd2.with_category(CUSTOM_CATEGORY)
     def do_echo(self, opts):
         """Echo (print) a string on the console"""
-        self.poutput(opts.string)
+        self.poutput(' '.join(opts.string))
 
     @cmd2.with_category(CUSTOM_CATEGORY)
     def do_version(self, opts):
@@ -502,7 +484,7 @@ class PySimCommands(CommandSet):
 
         if isinstance(self._cmd.lchan.selected_file, CardDF):
             if action_df:
-                action_df(context, opts)
+                action_df(context, **kwargs)
 
         files = self._cmd.lchan.selected_file.get_selectables(
             flags=['FNAMES', 'ANAMES'])
@@ -536,7 +518,32 @@ class PySimCommands(CommandSet):
                 # below, so we must not move up.
                 if skip_df == False:
                     self.walk(indent + 1, action_ef, action_df, context, **kwargs)
-                    fcp_dec = self._cmd.lchan.select("..", self._cmd)
+
+                    parent = self._cmd.lchan.selected_file.parent
+                    df = self._cmd.lchan.selected_file
+                    adf = self._cmd.lchan.selected_adf
+                    if isinstance(parent, CardMF) and (adf and adf.has_fs == False):
+                        # Not every application that may be present on a GlobalPlatform card will support the SELECT
+                        # command as we know it from ETSI TS 102 221, section 11.1.1. In fact the only subset of
+                        # SELECT we may rely on is the OPEN SELECT command as specified in GlobalPlatform Card
+                        # Specification, section 11.9. Unfortunately the OPEN SELECT command only supports the
+                        # "select by name" method, which means we can only select an application and not a file.
+                        # The consequence of this is that we may get trapped in an application that does not have
+                        # ISIM/USIM like file system support and the only way to leave that application is to select
+                        # an ISIM/USIM application in order to get the file system access back.
+                        #
+                        # To automate this escape-route while traversing the file system we will check whether
+                        # the parent file is the MF. When this is the case and the selected ADF has no file system
+                        # support, we will select an arbitrary ADF that has file system support first and from there
+                        # we will then select the MF.
+                        for selectable in parent.get_selectables().items():
+                            if isinstance(selectable[1], CardADF) and selectable[1].has_fs == True:
+                                self._cmd.lchan.select(selectable[1].name, self._cmd)
+                                break
+                        self._cmd.lchan.select(df.get_mf().name, self._cmd)
+                    else:
+                        # Normal DF/ADF selection
+                        fcp_dec = self._cmd.lchan.select("..", self._cmd)
 
             elif action_ef:
                 df_before_action = self._cmd.lchan.selected_file
@@ -697,8 +704,8 @@ class PySimCommands(CommandSet):
 
     def do_reset(self, opts):
         """Reset the Card."""
-        atr = self._cmd.lchan.reset(self._cmd)
-        self._cmd.poutput('Card ATR: %s' % atr)
+        atr = self._cmd.card.reset()
+        self._cmd.poutput('Card ATR: %s' % i2h(atr))
         self._cmd.update_prompt()
 
     def do_desc(self, opts):
@@ -709,11 +716,19 @@ class PySimCommands(CommandSet):
         else:
             self._cmd.poutput("no description available")
 
-    def do_verify_adm(self, arg):
-        """VERIFY the ADM1 PIN"""
-        if arg:
+    verify_adm_parser = argparse.ArgumentParser()
+    verify_adm_parser.add_argument('ADM1', nargs='?', type=is_hexstr_or_decimal,
+                                   help='ADM1 pin value. If none given, CSV file will be queried')
+
+    @cmd2.with_argparser(verify_adm_parser)
+    def do_verify_adm(self, opts):
+        """Verify the ADM (Administrator) PIN specified as argument.  This is typically needed in order
+to get write/update permissions to most of the files on SIM cards.
+
+Currently only ADM1 is supported."""
+        if opts.ADM1:
             # use specified ADM-PIN
-            pin_adm = sanitize_pin_adm(arg)
+            pin_adm = sanitize_pin_adm(opts.ADM1)
         else:
             # try to find an ADM-PIN if none is specified
             result = card_key_provider_get_field(
@@ -727,10 +742,21 @@ class PySimCommands(CommandSet):
                     "cannot find ADM-PIN for ICCID '%s'" % (self._cmd.iccid))
 
         if pin_adm:
-            self._cmd.card.verify_adm(h2b(pin_adm))
+            self._cmd.lchan.scc.verify_chv(self._cmd.card._adm_chv_num, h2b(pin_adm))
         else:
             raise ValueError("error: cannot authenticate, no adm-pin!")
 
+    def do_cardinfo(self, opts):
+        """Display information about the currently inserted card"""
+        self._cmd.poutput("Card info:")
+        self._cmd.poutput(" Name: %s" % self._cmd.card.name)
+        self._cmd.poutput(" ATR: %s" % b2h(self._cmd.lchan.scc.get_atr()))
+        self._cmd.poutput(" ICCID: %s" % self._cmd.iccid)
+        self._cmd.poutput(" Class-Byte: %s" % self._cmd.lchan.scc.cla_byte)
+        self._cmd.poutput(" Select-Ctrl: %s" % self._cmd.lchan.scc.sel_ctrl)
+        self._cmd.poutput(" AIDs:")
+        for a in self._cmd.rs.mf.applications:
+                self._cmd.poutput("  %s" % a)
 
 @with_default_category('ISO7816 Commands')
 class Iso7816Commands(CommandSet):
@@ -777,7 +803,7 @@ class Iso7816Commands(CommandSet):
     verify_chv_parser.add_argument(
         '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
     verify_chv_parser.add_argument(
-        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'pin_code', type=is_decimal, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
 
     @cmd2.with_argparser(verify_chv_parser)
     def do_verify_chv(self, opts):
@@ -785,23 +811,23 @@ class Iso7816Commands(CommandSet):
         call it if you authenticate yourself using the specified PIN.  There usually is at least PIN1 and
         PIN2."""
         pin = self.get_code(opts.pin_code)
-        (data, sw) = self._cmd.card._scc.verify_chv(opts.pin_nr, h2b(pin))
+        (data, sw) = self._cmd.lchan.scc.verify_chv(opts.pin_nr, h2b(pin))
         self._cmd.poutput("CHV verification successful")
 
     unblock_chv_parser = argparse.ArgumentParser()
     unblock_chv_parser.add_argument(
         '--pin-nr', type=int, default=1, help='PUK Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
     unblock_chv_parser.add_argument(
-        'puk_code', type=str, help='PUK code digits \"PUK1\" or \"PUK2\" to get PUK code from external data source')
+        'puk_code', type=is_decimal, help='PUK code digits \"PUK1\" or \"PUK2\" to get PUK code from external data source')
     unblock_chv_parser.add_argument(
-        'new_pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'new_pin_code', type=is_decimal, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
 
     @cmd2.with_argparser(unblock_chv_parser)
     def do_unblock_chv(self, opts):
         """Unblock PIN code using specified PUK code"""
         new_pin = self.get_code(opts.new_pin_code)
         puk = self.get_code(opts.puk_code)
-        (data, sw) = self._cmd.card._scc.unblock_chv(
+        (data, sw) = self._cmd.lchan.scc.unblock_chv(
             opts.pin_nr, h2b(puk), h2b(new_pin))
         self._cmd.poutput("CHV unblock successful")
 
@@ -809,16 +835,16 @@ class Iso7816Commands(CommandSet):
     change_chv_parser.add_argument(
         '--pin-nr', type=int, default=1, help='PUK Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
     change_chv_parser.add_argument(
-        'pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'pin_code', type=is_decimal, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
     change_chv_parser.add_argument(
-        'new_pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'new_pin_code', type=is_decimal, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
 
     @cmd2.with_argparser(change_chv_parser)
     def do_change_chv(self, opts):
         """Change PIN code to a new PIN code"""
         new_pin = self.get_code(opts.new_pin_code)
         pin = self.get_code(opts.pin_code)
-        (data, sw) = self._cmd.card._scc.change_chv(
+        (data, sw) = self._cmd.lchan.scc.change_chv(
             opts.pin_nr, h2b(pin), h2b(new_pin))
         self._cmd.poutput("CHV change successful")
 
@@ -826,31 +852,31 @@ class Iso7816Commands(CommandSet):
     disable_chv_parser.add_argument(
         '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
     disable_chv_parser.add_argument(
-        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'pin_code', type=is_decimal, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
 
     @cmd2.with_argparser(disable_chv_parser)
     def do_disable_chv(self, opts):
         """Disable PIN code using specified PIN code"""
         pin = self.get_code(opts.pin_code)
-        (data, sw) = self._cmd.card._scc.disable_chv(opts.pin_nr, h2b(pin))
+        (data, sw) = self._cmd.lchan.scc.disable_chv(opts.pin_nr, h2b(pin))
         self._cmd.poutput("CHV disable successful")
 
     enable_chv_parser = argparse.ArgumentParser()
     enable_chv_parser.add_argument(
         '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
     enable_chv_parser.add_argument(
-        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+        'pin_code', type=is_decimal, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
 
     @cmd2.with_argparser(enable_chv_parser)
     def do_enable_chv(self, opts):
         """Enable PIN code using specified PIN code"""
         pin = self.get_code(opts.pin_code)
-        (data, sw) = self._cmd.card._scc.enable_chv(opts.pin_nr, h2b(pin))
+        (data, sw) = self._cmd.lchan.scc.enable_chv(opts.pin_nr, h2b(pin))
         self._cmd.poutput("CHV enable successful")
 
     def do_deactivate_file(self, opts):
         """Deactivate the currently selected EF"""
-        (data, sw) = self._cmd.card._scc.deactivate_file()
+        (data, sw) = self._cmd.lchan.scc.deactivate_file()
 
     activate_file_parser = argparse.ArgumentParser()
     activate_file_parser.add_argument('NAME', type=str, help='File name or FID of file to activate')
@@ -872,8 +898,10 @@ class Iso7816Commands(CommandSet):
     @cmd2.with_argparser(open_chan_parser)
     def do_open_channel(self, opts):
         """Open a logical channel."""
-        (data, sw) = self._cmd.card._scc.manage_channel(
+        (data, sw) = self._cmd.lchan.scc.manage_channel(
             mode='open', lchan_nr=opts.chan_nr)
+        # this is executed only in successful case, as unsuccessful raises exception
+        self._cmd.lchan.add_lchan(opts.chan_nr)
 
     close_chan_parser = argparse.ArgumentParser()
     close_chan_parser.add_argument(
@@ -882,28 +910,28 @@ class Iso7816Commands(CommandSet):
     @cmd2.with_argparser(close_chan_parser)
     def do_close_channel(self, opts):
         """Close a logical channel."""
-        (data, sw) = self._cmd.card._scc.manage_channel(
+        (data, sw) = self._cmd.lchan.scc.manage_channel(
             mode='close', lchan_nr=opts.chan_nr)
+        # this is executed only in successful case, as unsuccessful raises exception
+        self._cmd.rs.del_lchan(opts.chan_nr)
+
+    switch_chan_parser = argparse.ArgumentParser()
+    switch_chan_parser.add_argument(
+        'chan_nr', type=int, default=0, help='Channel Number')
+
+    @cmd2.with_argparser(switch_chan_parser)
+    def do_switch_channel(self, opts):
+        """Switch currently active logical channel."""
+        self._cmd.lchan._select_pre(self._cmd)
+        self._cmd.lchan = self._cmd.rs.lchan[opts.chan_nr]
+        self._cmd.lchan._select_post(self._cmd)
+        self._cmd.update_prompt()
 
     def do_status(self, opts):
         """Perform the STATUS command."""
         fcp_dec = self._cmd.lchan.status()
         self._cmd.poutput_json(fcp_dec)
 
-    suspend_uicc_parser = argparse.ArgumentParser()
-    suspend_uicc_parser.add_argument('--min-duration-secs', type=int, default=60,
-                                     help='Proposed minimum duration of suspension')
-    suspend_uicc_parser.add_argument('--max-duration-secs', type=int, default=24*60*60,
-                                     help='Proposed maximum duration of suspension')
-
-    # not ISO7816-4 but TS 102 221
-    @cmd2.with_argparser(suspend_uicc_parser)
-    def do_suspend_uicc(self, opts):
-        """Perform the SUSPEND UICC command. Only supported on some UICC."""
-        (duration, token, sw) = self._cmd.card._scc.suspend_uicc(min_len_secs=opts.min_duration_secs,
-                                                                 max_len_secs=opts.max_duration_secs)
-        self._cmd.poutput(
-            'Negotiated Duration: %u secs, Token: %s, SW: %s' % (duration, token, sw))
 
 class Proact(ProactiveHandler):
     def receive_fetch(self, pcmd: ProactiveCommand):
@@ -913,7 +941,7 @@ class Proact(ProactiveHandler):
 
 
 
-option_parser = argparse.ArgumentParser(prog='pySim-shell', description='interactive SIM card shell',
+option_parser = argparse.ArgumentParser(description='interactive SIM card shell',
                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 argparse_add_reader_args(option_parser)
 
@@ -931,6 +959,11 @@ adm_group.add_argument('-a', '--pin-adm', metavar='PIN_ADM1', dest='pin_adm', de
 adm_group.add_argument('-A', '--pin-adm-hex', metavar='PIN_ADM1_HEX', dest='pin_adm_hex', default=None,
                        help='ADM PIN used for provisioning, as hex string (16 characters long)')
 
+option_parser.add_argument("command", nargs='?',
+                           help="A pySim-shell command that would optionally be executed at startup")
+option_parser.add_argument('command_args', nargs=argparse.REMAINDER,
+                           help="Optional Arguments for command")
+
 
 if __name__ == '__main__':
 
@@ -953,11 +986,6 @@ if __name__ == '__main__':
 
     # Init card reader driver
     sl = init_reader(opts, proactive_handler = Proact())
-    if sl is None:
-        exit(1)
-
-    # Create command layer
-    scc = SimCardCommands(transport=sl)
 
     # Create a card handler (for bulk provisioning)
     if opts.card_handler_config:
@@ -972,7 +1000,7 @@ if __name__ == '__main__':
         rs, card = init_card(sl)
         app = PysimApp(card, rs, sl, ch, opts.script)
     except:
-        print("Card initialization failed with an exception:")
+        print("Card initialization (%s) failed with an exception:" % str(sl))
         print("---------------------8<---------------------")
         traceback.print_exc()
         print("---------------------8<---------------------")
@@ -981,7 +1009,9 @@ if __name__ == '__main__':
             " it should also be noted that some readers may behave strangely when no card")
         print(" is inserted.)")
         print("")
-        app = PysimApp(card, None, sl, ch, opts.script)
+        if opts.script:
+            print("will not execute startup script due to card initialization errors!")
+        app = PysimApp(None, None, sl, ch)
 
     # If the user supplies an ADM PIN at via commandline args authenticate
     # immediately so that the user does not have to use the shell commands
@@ -990,8 +1020,11 @@ if __name__ == '__main__':
         if not card:
             print("Card error, cannot do ADM verification with supplied ADM pin now.")
         try:
-            card.verify_adm(h2b(pin_adm))
+            card._scc.verify_chv(card._adm_chv_num, h2b(pin_adm))
         except Exception as e:
             print(e)
 
-    app.cmdloop()
+    if opts.command:
+        app.onecmd_plus_hooks('{} {}'.format(opts.command, ' '.join(opts.command_args)))
+    else:
+        app.cmdloop()

pySim-trace.py

@@ -6,18 +6,19 @@ import argparse
 from pprint import pprint as pp
 
 from pySim.apdu import *
-from pySim.filesystem import RuntimeState
+from pySim.runtime import RuntimeState
 
-from pySim.cards import UsimCard
+from pySim.cards import UiccCardBase
 from pySim.commands import SimCardCommands
 from pySim.profile import CardProfile
-from pySim.ts_102_221 import CardProfileUICCSIM
+from pySim.ts_102_221 import CardProfileUICC
 from pySim.ts_31_102 import CardApplicationUSIM
 from pySim.ts_31_103 import CardApplicationISIM
 from pySim.transport import LinkBase
 
 from pySim.apdu_source.gsmtap import GsmtapApduSource
 from pySim.apdu_source.pyshark_rspro import PysharkRsproPcap, PysharkRsproLive
+from pySim.apdu_source.pyshark_gsmtap import PysharkGsmtapPcap
 
 from pySim.apdu.ts_102_221 import UiccSelect, UiccStatus
 
@@ -35,18 +36,21 @@ ApduCommands = UiccApduCommands + UsimApduCommands #+ GpApduCommands
 
 class DummySimLink(LinkBase):
     """A dummy implementation of the LinkBase abstract base class.  Currently required
-    as the UsimCard doesn't work without SimCardCommands, which in turn require
+    as the UiccCardBase doesn't work without SimCardCommands, which in turn require
     a LinkBase implementation talking to a card.
 
     In the tracer, we don't actually talk to any card, so we simply drop everything
     and claim it is successful.
 
-    The UsimCard / SimCardCommands should be refactored to make this obsolete later."""
+    The UiccCardBase / SimCardCommands should be refactored to make this obsolete later."""
     def __init__(self, debug: bool = False, **kwargs):
         super().__init__(**kwargs)
         self._debug = debug
         self._atr = h2i('3B9F96801F878031E073FE211B674A4C753034054BA9')
 
+    def __str__(self):
+        return "dummy"
+
     def _send_apdu_raw(self, pdu):
         #print("DummySimLink-apdu: %s" % pdu)
         return [], '9000'
@@ -69,34 +73,49 @@ class DummySimLink(LinkBase):
 
 class Tracer:
     def __init__(self, **kwargs):
-        # we assume a generic SIM + UICC + USIM + ISIM card
-        profile = CardProfileUICCSIM()
+        # we assume a generic UICC profile; as all APDUs return 9000 in DummySimLink above,
+        # all CardProfileAddon (including SIM) will probe successful.
+        profile = CardProfileUICC()
         profile.add_application(CardApplicationUSIM())
         profile.add_application(CardApplicationISIM())
         scc = SimCardCommands(transport=DummySimLink())
-        card = UsimCard(scc)
+        card = UiccCardBase(scc)
         self.rs = RuntimeState(card, profile)
         # APDU Decoder
         self.ad = ApduDecoder(ApduCommands)
         # parameters
         self.suppress_status = kwargs.get('suppress_status', True)
         self.suppress_select = kwargs.get('suppress_select', True)
+        self.show_raw_apdu = kwargs.get('show_raw_apdu', False)
         self.source = kwargs.get('source', None)
 
-    def format_capdu(self, inst: ApduCommand):
+    def format_capdu(self, apdu: Apdu, inst: ApduCommand):
         """Output a single decoded + processed ApduCommand."""
+        if self.show_raw_apdu:
+            print(apdu)
         print("%02u %-16s %-35s %-8s %s %s" % (inst.lchan_nr, inst._name, inst.path_str, inst.col_id, inst.col_sw, inst.processed))
         print("===============================")
 
+    def format_reset(self, apdu: CardReset):
+        """Output a single decoded CardReset."""
+        print(apdu)
+        print("===============================")
+
     def main(self):
         """Main loop of tracer: Iterates over all Apdu received from source."""
+        apdu_counter = 0
         while True:
             # obtain the next APDU from the source (blocking read)
-            apdu = self.source.read()
-            #print(apdu)
+            try:
+                apdu = self.source.read()
+                apdu_counter = apdu_counter + 1
+            except StopIteration:
+                print("%i APDUs parsed, stop iteration." % apdu_counter)
+                return 0
 
             if isinstance(apdu, CardReset):
                 self.rs.reset()
+                self.format_reset(apdu)
                 continue
 
             # ask ApduDecoder to look-up (INS,CLA) + instantiate an ApduCommand derived
@@ -110,34 +129,51 @@ class Tracer:
                 continue
             if self.suppress_status and isinstance(inst, UiccStatus):
                 continue
-            #print(inst)
-            self.format_capdu(inst)
 
-option_parser = argparse.ArgumentParser(prog='pySim-trace', description='Osmocom pySim high-level SIM card trace decoder',
+            self.format_capdu(apdu, inst)
+
+option_parser = argparse.ArgumentParser(description='Osmocom pySim high-level SIM card trace decoder',
                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 
 global_group = option_parser.add_argument_group('General Options')
 global_group.add_argument('--no-suppress-select', action='store_false', dest='suppress_select',
-                          help="Don't suppress displaying SELECT APDUs")
+                          help="""
+    Don't suppress displaying SELECT APDUs. We normally suppress them as they just clutter up
+    the output without giving any useful information.  Any subsequent READ/UPDATE/... operations
+    on the selected file will log the file name most recently SELECTed.""")
 global_group.add_argument('--no-suppress-status', action='store_false', dest='suppress_status',
-                          help="Don't suppress displaying STATUS APDUs")
+                          help="""
+    Don't suppress displaying STATUS APDUs. We normally suppress them as they don't provide any
+    information that was not already received in resposne to the most recent SEELCT.""")
+global_group.add_argument('--show-raw-apdu', action='store_true', dest='show_raw_apdu',
+                          help="""Show the raw APDU in addition to its parsed form.""")
+
 
 subparsers = option_parser.add_subparsers(help='APDU Source', dest='source', required=True)
 
-parser_gsmtap = subparsers.add_parser('gsmtap-udp', help='Live capture of GSMTAP-SIM on UDP port')
+parser_gsmtap = subparsers.add_parser('gsmtap-udp', help="""
+    Read APDUs from live capture by receiving GSMTAP-SIM packets on specified UDP port.
+    Use this for live capture from SIMtrace2 or osmo-qcdiag.""")
 parser_gsmtap.add_argument('-i', '--bind-ip', default='127.0.0.1',
                            help='Local IP address to which to bind the UDP port')
 parser_gsmtap.add_argument('-p', '--bind-port', default=4729,
                            help='Local UDP port')
 
+parser_gsmtap_pyshark_pcap = subparsers.add_parser('gsmtap-pyshark-pcap', help="""
+    Read APDUs from PCAP file containing GSMTAP (SIM APDU) communication; processed via pyshark.
+    Use this if you have recorded a PCAP file containing GSMTAP (SIM APDU) e.g. via tcpdump or
+    wireshark/tshark.""")
+parser_gsmtap_pyshark_pcap.add_argument('-f', '--pcap-file', required=True,
+                                       help='Name of the PCAP[ng] file to be read')
+
 parser_rspro_pyshark_pcap = subparsers.add_parser('rspro-pyshark-pcap', help="""
-    PCAP file containing RSPRO (osmo-remsim) communication; processed via pyshark.
+    Read APDUs from PCAP file containing RSPRO (osmo-remsim) communication; processed via pyshark.
     REQUIRES OSMOCOM PATCHED WIRESHARK!""")
 parser_rspro_pyshark_pcap.add_argument('-f', '--pcap-file', required=True,
                                        help='Name of the PCAP[ng] file to be read')
 
 parser_rspro_pyshark_live = subparsers.add_parser('rspro-pyshark-live', help="""
-    Live capture of RSPRO (osmo-remsim) communication; processed via pyshark.
+    Read APDUs from live capture of RSPRO (osmo-remsim) communication; processed via pyshark.
     REQUIRES OSMOCOM PATCHED WIRESHARK!""")
 parser_rspro_pyshark_live.add_argument('-i', '--interface', required=True,
                                        help='Name of the network interface to capture on')
@@ -146,15 +182,18 @@ if __name__ == '__main__':
 
     opts = option_parser.parse_args()
 
-    logger.info('Opening source %s...' % opts.source)
+    logger.info('Opening source %s...', opts.source)
     if opts.source == 'gsmtap-udp':
         s = GsmtapApduSource(opts.bind_ip, opts.bind_port)
     elif opts.source == 'rspro-pyshark-pcap':
         s = PysharkRsproPcap(opts.pcap_file)
     elif opts.source == 'rspro-pyshark-live':
         s = PysharkRsproLive(opts.interface)
+    elif opts.source == 'gsmtap-pyshark-pcap':
+        s = PysharkGsmtapPcap(opts.pcap_file)
 
-    tracer = Tracer(source=s, suppress_status=opts.suppress_status, suppress_select=opts.suppress_select)
+    tracer = Tracer(source=s, suppress_status=opts.suppress_status, suppress_select=opts.suppress_select,
+                    show_raw_apdu=opts.show_raw_apdu)
     logger.info('Entering main loop...')
     tracer.main()
 

pySim/apdu/__init__.py

@@ -35,7 +35,7 @@ from construct import *
 from construct import Optional as COptional
 from pySim.construct import *
 from pySim.utils import *
-from pySim.filesystem import RuntimeLchan, RuntimeState, lchan_nr_from_cla
+from pySim.runtime import RuntimeLchan, RuntimeState, lchan_nr_from_cla
 from pySim.filesystem import CardADF, CardFile, TransparentEF, LinFixedEF
 
 """There are multiple levels of decode:
@@ -135,7 +135,12 @@ class Apdu(Tpdu):
         if callable(method):
             return method()
         # default case: only 9000 is success
-        return self.sw == b'\x90\x00'
+        if self.sw == b'\x90\x00':
+            return True
+        # This is not really a generic positive APDU SW but specific to UICC/SIM
+        if self.sw[0] == 0x91:
+            return True
+        return False
 
 
 class ApduCommand(Apdu, metaclass=ApduCommandMeta):
@@ -443,4 +448,11 @@ class ApduDecoder(ApduHandler):
 
 
 class CardReset:
-    pass
+    def __init__(self, atr: bytes):
+        self.atr = atr
+
+    def __str__(self):
+        if (self.atr):
+            return '%s(%s)' % (type(self).__name__, b2h(self.atr))
+        else:
+            return '%s' % (type(self).__name__)

pySim/apdu/ts_102_221.py

@@ -20,6 +20,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 import logging
 from pySim.construct import *
 from pySim.filesystem import *
+from pySim.runtime import RuntimeLchan
 from pySim.apdu import ApduCommand, ApduCommandSet
 from typing import Optional, Dict, Tuple
 
@@ -76,7 +77,7 @@ class UiccSelect(ApduCommand, n='SELECT', ins=0xA4, cla=['0X', '4X', '6X']):
                             pass
                         # iterate to next element in path
                         continue
-                logger.warning('SELECT UNKNOWN FID %s (%s)' % (file_hex, '/'.join([b2h(x) for x in path])))
+                logger.warning('SELECT UNKNOWN FID %s (%s)', file_hex, '/'.join([b2h(x) for x in path]))
         elif mode == 'df_ef_or_mf_by_file_id':
             if len(self.cmd_data) != 2:
                 raise ValueError('Expecting a 2-byte FID')
@@ -90,7 +91,7 @@ class UiccSelect(ApduCommand, n='SELECT', ins=0xA4, cla=['0X', '4X', '6X']):
                     #print("\tSELECT %s FAILED" % sels[file_hex])
                     pass
             else:
-                logger.warning('SELECT UNKNOWN FID %s' % (file_hex))
+                logger.warning('SELECT UNKNOWN FID %s', file_hex)
         elif mode == 'df_name':
             # Select by AID (can be sub-string!)
             aid = self.cmd_dict['body']
@@ -101,7 +102,7 @@ class UiccSelect(ApduCommand, n='SELECT', ins=0xA4, cla=['0X', '4X', '6X']):
                 lchan.selected_file = lchan.selected_adf
                 #print("\tSELECT AID %s" % adf)
             else:
-                logger.warning('SELECT UNKNOWN AID %s' % aid)
+                logger.warning('SELECT UNKNOWN AID %s', aid)
                 pass
         else:
             raise ValueError('Select Mode %s not implemented' % mode)
@@ -201,7 +202,7 @@ class ReadRecord(ApduCommand, n='READ RECORD', ins=0xB2, cla=['0X', '4X', '6X'])
             return b2h(self.rsp_data)
         method = getattr(self.file, 'decode_record_bin', None)
         if self.successful and callable(method):
-            return method(self.rsp_data)
+            return method(self.rsp_data, self.cmd_dict['record_number'])
 
 # TS 102 221 Section 11.1.6
 class UpdateRecord(ApduCommand, n='UPDATE RECORD', ins=0xDC, cla=['0X', '4X', '6X']):
@@ -217,7 +218,7 @@ class UpdateRecord(ApduCommand, n='UPDATE RECORD', ins=0xDC, cla=['0X', '4X', '6
             return b2h(self.cmd_data)
         method = getattr(self.file, 'decode_record_bin', None)
         if self.successful and callable(method):
-            return method(self.cmd_data)
+            return method(self.cmd_data, self.cmd_dict['record_number'])
 
 # TS 102 221 Section 11.1.7
 class SearchRecord(ApduCommand, n='SEARCH RECORD', ins=0xA2, cla=['0X', '4X', '6X']):
@@ -393,10 +394,12 @@ class ManageChannel(ApduCommand, n='MANAGE CHANNEL', ins=0x70, cla=['0X', '4X',
             manage_channel = rs.get_lchan_by_cla(self.cla)
             manage_channel.add_lchan(created_channel_nr)
             self.col_id = '%02u' % created_channel_nr
+            return {'mode': mode, 'created_channel': created_channel_nr }
         elif mode == 'close_channel':
-            closed_channel_nr = self.cmd_dict['p2']
+            closed_channel_nr = self.cmd_dict['p2']['logical_channel_number']
             rs.del_lchan(closed_channel_nr)
             self.col_id = '%02u' % closed_channel_nr
+            return {'mode': mode, 'closed_channel': closed_channel_nr }
         else:
             raise ValueError('Unsupported MANAGE CHANNEL P1=%02X' % self.p1)
 
@@ -487,7 +490,7 @@ class RetrieveData(ApduCommand, n='RETRIEVE DATA', ins=0xCB, cla=['8X', 'CX', 'E
         elif self.p2 & 0xdf == 0x40:
             c['mode'] = 'retransmit_previous_block'
         else:
-            logger.warning('%s: invalid P2=%02x' % (self, self.p2))
+            logger.warning('%s: invalid P2=%02x', self, self.p2)
         return c
 
     def _decode_cmd(self):

pySim/apdu/ts_31_102.py

@@ -108,7 +108,7 @@ class UsimAuthenticateOdd(ApduCommand, n='AUTHENTICATE', ins=0x89, cla=['0X', '4
 class UsimGetIdentity(ApduCommand, n='GET IDENTITY', ins=0x78, cla=['8X', 'CX', 'EX']):
     _apdu_case = 4
     _construct_p2 = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1),
-                              'identity_context'/Enum(BitsInteger(7), suci=1))
+                              'identity_context'/Enum(BitsInteger(7), suci=1, suci_5g_nswo=2))
     _tlv_rsp = SUCI_TlvDataObject
 
 ApduCommands = ApduCommandSet('TS 31.102', cmds=[UsimAuthenticateEven, UsimAuthenticateOdd,

pySim/apdu_source/gsmtap.py

@@ -49,7 +49,7 @@ class GsmtapApduSource(ApduSource):
             return ApduCommands.parse_cmd_bytes(gsmtap_msg['body'])
         elif sub_type == 'atr':
             # card has been reset
-            return CardReset()
+            return CardReset(gsmtap_msg['body'])
         elif sub_type in ['pps_req', 'pps_rsp']:
             # simply ignore for now
             pass

pySim/apdu_source/pyshark_gsmtap.py

@@ -0,0 +1,90 @@
+# coding=utf-8
+
+# (C) 2022 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import sys
+import logging
+from pprint import pprint as pp
+from typing import Tuple
+import pyshark
+
+from pySim.utils import h2b, b2h
+from pySim.apdu import Tpdu
+from pySim.gsmtap import GsmtapMessage
+from . import ApduSource, PacketType, CardReset
+
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+ApduCommands = UiccApduCommands + UsimApduCommands + GpApduCommands
+
+logger = logging.getLogger(__name__)
+
+class _PysharkGsmtap(ApduSource):
+    """APDU Source [provider] base class for reading GSMTAP SIM APDU via tshark."""
+
+    def __init__(self, pyshark_inst):
+        self.pyshark = pyshark_inst
+        self.bank_id = None
+        self.bank_slot = None
+        self.cmd_tpdu = None
+        super().__init__()
+
+    def read_packet(self) -> PacketType:
+        p = self.pyshark.next()
+        return self._parse_packet(p)
+
+    def _set_or_verify_bank_slot(self, bsl: Tuple[int, int]):
+        """Keep track of the bank:slot to make sure we don't mix traces of multiple cards"""
+        if not self.bank_id:
+            self.bank_id = bsl[0]
+            self.bank_slot = bsl[1]
+        else:
+            if self.bank_id != bsl[0] or self.bank_slot != bsl[1]:
+                raise ValueError('Received data for unexpected B(%u:%u)' % (bsl[0], bsl[1]))
+
+    def _parse_packet(self, p) -> PacketType:
+        udp_layer = p['udp']
+        udp_payload_hex = udp_layer.get_field('payload').replace(':','')
+        gsmtap = GsmtapMessage(h2b(udp_payload_hex))
+        gsmtap_msg = gsmtap.decode()
+        if gsmtap_msg['type'] != 'sim':
+            raise ValueError('Unsupported GSMTAP type %s' % gsmtap_msg['type'])
+        sub_type = gsmtap_msg['sub_type']
+        if sub_type == 'apdu':
+            return ApduCommands.parse_cmd_bytes(gsmtap_msg['body'])
+        elif sub_type == 'atr':
+            # card has been reset
+            return CardReset(gsmtap_msg['body'])
+        elif sub_type in ['pps_req', 'pps_rsp']:
+            # simply ignore for now
+            pass
+        else:
+            raise ValueError('Unsupported GSMTAP-SIM sub-type %s' % sub_type)
+
+class PysharkGsmtapPcap(_PysharkGsmtap):
+    """APDU Source [provider] class for reading GSMTAP from a PCAP
+    file via pyshark, which in turn uses tshark (part of wireshark).
+    """
+    def __init__(self, pcap_filename):
+        """
+        Args:
+            pcap_filename: File name of the pcap file to be opened
+        """
+        pyshark_inst = pyshark.FileCapture(pcap_filename, display_filter='gsm_sim', use_json=True, keep_packets=False)
+        super().__init__(pyshark_inst)
+

pySim/apdu_source/pyshark_rspro.py

@@ -89,7 +89,7 @@ class _PysharkRspro(ApduSource):
             bsl = self.get_bank_slot(bank_slot)
             self._set_or_verify_bank_slot(bsl)
             data = modem2card.get_field('data').replace(':','')
-            logger.debug("C(%u:%u) -> B(%u:%u): %s" % (csl[0], csl[1], bsl[0], bsl[1], data))
+            logger.debug("C(%u:%u) -> B(%u:%u): %s", csl[0], csl[1], bsl[0], bsl[1], data)
             # store the CMD portion until the RSP portion arrives later
             self.cmd_tpdu = h2b(data)
         elif msg_type == '13': # tpduCardToModem
@@ -101,7 +101,7 @@ class _PysharkRspro(ApduSource):
             bsl = self.get_bank_slot(bank_slot)
             self._set_or_verify_bank_slot(bsl)
             data = card2modem.get_field('data').replace(':','')
-            logger.debug("C(%u:%u) <- B(%u:%u): %s" % (csl[0], csl[1], bsl[0], bsl[1], data))
+            logger.debug("C(%u:%u) <- B(%u:%u): %s", csl[0], csl[1], bsl[0], bsl[1], data)
             rsp_tpdu = h2b(data)
             if self.cmd_tpdu:
                 # combine this R-TPDU with the C-TPDU we saw earlier
@@ -117,7 +117,8 @@ class _PysharkRspro(ApduSource):
             vccPresent, resetActive, clkActive = self.get_pstatus(slot_pstatus)
             if vccPresent and clkActive and not resetActive:
                 logger.debug("RESET")
-                return CardReset()
+                #TODO: extract ATR from RSPRO message and use it here
+                return CardReset(None)
         else:
             print("Unhandled msg type %s: %s" % (msg_type, rspro_msg))
 

pySim/app.py

@@ -0,0 +1,113 @@
+# (C) 2021-2023 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from typing import Tuple
+
+from pySim.transport import LinkBase
+from pySim.commands import SimCardCommands
+from pySim.filesystem import CardModel, CardApplication
+from pySim.cards import card_detect, SimCardBase, UiccCardBase
+from pySim.exceptions import NoCardError
+from pySim.runtime import RuntimeState
+from pySim.profile import CardProfile
+from pySim.cdma_ruim import CardProfileRUIM
+from pySim.ts_102_221 import CardProfileUICC
+from pySim.utils import all_subclasses
+
+# we need to import this module so that the SysmocomSJA2 sub-class of
+# CardModel is created, which will add the ATR-based matching and
+# calling of SysmocomSJA2.add_files.  See  CardModel.apply_matching_models
+import pySim.sysmocom_sja2
+
+# we need to import these modules so that the various sub-classes of
+# CardProfile are created, which will be used in init_card() to iterate
+# over all known CardProfile sub-classes.
+import pySim.ts_31_102
+import pySim.ts_31_103
+import pySim.ts_31_104
+import pySim.ara_m
+import pySim.global_platform
+import pySim.euicc
+
+def init_card(sl: LinkBase) -> Tuple[RuntimeState, SimCardBase]:
+    """
+    Detect card in reader and setup card profile and runtime state. This
+    function must be called at least once on startup. The card and runtime
+    state object (rs) is required for all pySim-shell commands.
+    """
+
+    # Create command layer
+    scc = SimCardCommands(transport=sl)
+
+    # Wait up to three seconds for a card in reader and try to detect
+    # the card type.
+    print("Waiting for card...")
+    sl.wait_for_card(3)
+
+    generic_card = False
+    card = card_detect(scc)
+    if card is None:
+        print("Warning: Could not detect card type - assuming a generic card type...")
+        card = SimCardBase(scc)
+        generic_card = True
+
+    profile = CardProfile.pick(scc)
+    if profile is None:
+        # It is not an unrecoverable error in case profile detection fails. It
+        # just means that pySim was unable to recognize the card profile. This
+        # may happen in particular with unprovisioned cards that do not have
+        # any files on them yet.
+        print("Unsupported card type!")
+        return None, card
+
+    # ETSI TS 102 221, Table 9.3 specifies a default for the PIN key
+    # references, however card manufactures may still decide to pick an
+    # arbitrary key reference. In case we run on a generic card class that is
+    # detected as an UICC, we will pick the key reference that is officially
+    # specified.
+    if generic_card and isinstance(profile, CardProfileUICC):
+        card._adm_chv_num = 0x0A
+
+    print("Info: Card is of type: %s" % str(profile))
+
+    # FIXME: this shouldn't really be here but somewhere else/more generic.
+    # We cannot do it within pySim/profile.py as that would create circular
+    # dependencies between the individual profiles and profile.py.
+    if isinstance(profile, CardProfileUICC):
+        for app_cls in all_subclasses(CardApplication):
+            # skip any intermediary sub-classes such as CardApplicationSD
+            if hasattr(app_cls, '_' + app_cls.__name__ + '__intermediate'):
+                continue
+            profile.add_application(app_cls())
+        # We have chosen SimCard() above, but we now know it actually is an UICC
+        # so it's safe to assume it supports USIM application (which we're adding above).
+        # IF we don't do this, we will have a SimCard but try USIM specific commands like
+        # the update_ust method (see https://osmocom.org/issues/6055)
+        if generic_card:
+            card = UiccCardBase(scc)
+
+    # Create runtime state with card profile
+    rs = RuntimeState(card, profile)
+
+    CardModel.apply_matching_models(scc, rs)
+
+    # inform the transport that we can do context-specific SW interpretation
+    sl.set_sw_interpreter(rs)
+
+    return rs, card
+
+
+

pySim/ara_m.py

@@ -295,7 +295,7 @@ class ADF_ARAM(CardADF):
     @staticmethod
     def get_config(tp, v_major=0, v_minor=0, v_patch=1):
         cmd_do = DeviceConfigDO()
-        cmd_do.from_dict([{'DeviceInterfaceVersionDO': {
+        cmd_do.from_dict([{'device_interface_version_do': {
                          'major': v_major, 'minor': v_minor, 'patch': v_patch}}])
         return ADF_ARAM.xceive_apdu_tlv(tp, '80cadf21', cmd_do, ResponseAramConfigDO)
 
@@ -306,13 +306,13 @@ class ADF_ARAM(CardADF):
 
         def do_aram_get_all(self, opts):
             """GET DATA [All] on the ARA-M Applet"""
-            res_do = ADF_ARAM.get_all(self._cmd.card._scc._tp)
+            res_do = ADF_ARAM.get_all(self._cmd.lchan.scc._tp)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
         def do_aram_get_config(self, opts):
             """Perform GET DATA [Config] on the ARA-M Applet: Tell it our version and retrieve its version."""
-            res_do = ADF_ARAM.get_config(self._cmd.card._scc._tp)
+            res_do = ADF_ARAM.get_config(self._cmd.lchan.scc._tp)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
@@ -349,38 +349,38 @@ class ADF_ARAM(CardADF):
             # REF
             ref_do_content = []
             if opts.aid:
-                ref_do_content += [{'AidRefDO': opts.aid}]
+                ref_do_content += [{'aid_ref_do': opts.aid}]
             elif opts.aid_empty:
-                ref_do_content += [{'AidRefEmptyDO': None}]
-            ref_do_content += [{'DevAppIdRefDO': opts.device_app_id}]
+                ref_do_content += [{'aid_ref_empty_do': None}]
+            ref_do_content += [{'dev_app_id_ref_do': opts.device_app_id}]
             if opts.pkg_ref:
-                ref_do_content += [{'PkgRefDO': opts.pkg_ref}]
+                ref_do_content += [{'pkg_ref_do': {'package_name_string': opts.pkg_ref}}]
             # AR
             ar_do_content = []
             if opts.apdu_never:
-                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'never'}}]
+                ar_do_content += [{'apdu_ar_od': {'generic_access_rule': 'never'}}]
             elif opts.apdu_always:
-                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'always'}}]
+                ar_do_content += [{'apdu_ar_do': {'generic_access_rule': 'always'}}]
             elif opts.apdu_filter:
                 # TODO: multiple filters
-                ar_do_content += [{'ApduArDO': {'apdu_filter': [opts.apdu_filter]}}]
+                ar_do_content += [{'apdu_ar_do': {'apdu_filter': [opts.apdu_filter]}}]
             if opts.nfc_always:
-                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'always'}}]
+                ar_do_content += [{'nfc_ar_do': {'nfc_event_access_rule': 'always'}}]
             elif opts.nfc_never:
-                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'never'}}]
+                ar_do_content += [{'nfc_ar_do': {'nfc_event_access_rule': 'never'}}]
             if opts.android_permissions:
-                ar_do_content += [{'PermArDO': {'permissions': opts.android_permissions}}]
-            d = [{'RefArDO': [{'RefDO': ref_do_content}, {'ArDO': ar_do_content}]}]
+                ar_do_content += [{'perm_ar_do': {'permissions': opts.android_permissions}}]
+            d = [{'ref_ar_do': [{'ref_do': ref_do_content}, {'ar_do': ar_do_content}]}]
             csrado = CommandStoreRefArDO()
             csrado.from_dict(d)
-            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, csrado)
+            res_do = ADF_ARAM.store_data(self._cmd.lchan.scc._tp, csrado)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
         def do_aram_delete_all(self, opts):
             """Perform STORE DATA [Command-Delete[all]] to delete all access rules."""
             deldo = CommandDelete()
-            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, deldo)
+            res_do = ADF_ARAM.store_data(self._cmd.lchan.scc._tp, deldo)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 

pySim/cat.py

@@ -22,7 +22,7 @@ from bidict import bidict
 from typing import List
 from pySim.utils import b2h, h2b, dec_xplmn_w_act
 from pySim.tlv import TLV_IE, COMPR_TLV_IE, BER_TLV_IE, TLV_IE_Collection
-from pySim.construct import BcdAdapter, HexAdapter, GsmStringAdapter, TonNpi
+from pySim.construct import PlmnAdapter, BcdAdapter, HexAdapter, GsmStringAdapter, TonNpi
 from construct import Int8ub, Int16ub, Byte, Bytes, Bit, Flag, BitsInteger
 from construct import Struct, Enum, Tell, BitStruct, this, Padding, RepeatUntil
 from construct import GreedyBytes, Switch, GreedyRange, FlagsEnum
@@ -597,7 +597,7 @@ class ContactlessFunctionalityState(COMPR_TLV_IE, tag=0xD4):
 
 # TS 31.111 Section 8.91
 class RoutingAreaIdentification(COMPR_TLV_IE, tag=0xF3):
-    _construct = Struct('mcc_mnc'/BcdAdapter(Bytes(3)),
+    _construct = Struct('mcc_mnc'/PlmnAdapter(Bytes(3)),
                         'lac'/HexAdapter(Bytes(2)),
                         'rac'/Int8ub)
 
@@ -645,7 +645,7 @@ class GeographicalLocationParameters(COMPR_TLV_IE, tag=0xF6):
 
 # TS 31.111 Section 8.97
 class PlmnList(COMPR_TLV_IE, tag=0xF9):
-    _construct = GreedyRange('mcc_mnc'/HexAdapter(Bytes(3)))
+    _construct = GreedyRange('mcc_mnc'/PlmnAdapter(Bytes(3)))
 
 # TS 102 223 Section 8.98
 class EcatSequenceNumber(COMPR_TLV_IE, tag=0xA1):

pySim/cdma_ruim.py

@@ -0,0 +1,205 @@
+# coding=utf-8
+"""R-UIM (Removable User Identity Module) card profile (see 3GPP2 C.S0023-D)
+
+(C) 2023 by Vadim Yanitskiy <fixeria@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+import enum
+
+from pySim.utils import *
+from pySim.filesystem import *
+from pySim.profile import match_ruim
+from pySim.profile import CardProfile, CardProfileAddon
+from pySim.ts_51_011 import CardProfileSIM
+from pySim.ts_51_011 import DF_TELECOM, DF_GSM
+from pySim.ts_51_011 import EF_ServiceTable
+from pySim.construct import *
+from construct import *
+
+
+# Mapping between CDMA Service Number and its description
+EF_CST_map = {
+    1 : 'CHV disable function',
+    2 : 'Abbreviated Dialing Numbers (ADN)',
+    3 : 'Fixed Dialing Numbers (FDN)',
+    4 : 'Short Message Storage (SMS)',
+    5 : 'HRPD',
+    6 : 'Enhanced Phone Book',
+    7 : 'Multi Media Domain (MMD)',
+    8 : 'SF_EUIMID-based EUIMID',
+    9 : 'MEID Support',
+    10 : 'Extension1',
+    11 : 'Extension2',
+    12 : 'SMS Parameters',
+    13 : 'Last Number Dialled (LND)',
+    14 : 'Service Category Program for BC-SMS',
+    15 : 'Messaging and 3GPD Extensions',
+    16 : 'Root Certificates',
+    17 : 'CDMA Home Service Provider Name',
+    18 : 'Service Dialing Numbers (SDN)',
+    19 : 'Extension3',
+    20 : '3GPD-SIP',
+    21 : 'WAP Browser',
+    22 : 'Java',
+    23 : 'Reserved for CDG',
+    24 : 'Reserved for CDG',
+    25 : 'Data Download via SMS Broadcast',
+    26 : 'Data Download via SMS-PP',
+    27 : 'Menu Selection',
+    28 : 'Call Control',
+    29 : 'Proactive R-UIM',
+    30 : 'AKA',
+    31 : 'IPv6',
+    32 : 'RFU',
+    33 : 'RFU',
+    34 : 'RFU',
+    35 : 'RFU',
+    36 : 'RFU',
+    37 : 'RFU',
+    38 : '3GPD-MIP',
+    39 : 'BCMCS',
+    40 : 'Multimedia Messaging Service (MMS)',
+    41 : 'Extension 8',
+    42 : 'MMS User Connectivity Parameters',
+    43 : 'Application Authentication',
+    44 : 'Group Identifier Level 1',
+    45 : 'Group Identifier Level 2',
+    46 : 'De-Personalization Control Keys',
+    47 : 'Cooperative Network List',
+}
+
+
+######################################################################
+# DF.CDMA
+######################################################################
+
+class EF_SPN(TransparentEF):
+    '''3.4.31 CDMA Home Service Provider Name'''
+
+    _test_de_encode = [
+        ( "010801536b796c696e6b204e57ffffffffffffffffffffffffffffffffffffffffffff",
+          { 'rfu1' : 0, 'show_in_hsa' : True, 'rfu2' : 0,
+            'char_encoding' : 8, 'lang_ind' : 1, 'spn' : 'Skylink NW' } ),
+    ]
+
+    def __init__(self, fid='6f41', sfid=None, name='EF.SPN',
+                 desc='Service Provider Name', size=(35, 35), **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
+        self._construct = BitStruct(
+            # Byte 1: Display Condition
+            'rfu1'/BitsRFU(7),
+            'show_in_hsa'/Flag,
+            # Byte 2: Character Encoding
+            'rfu2'/BitsRFU(3),
+            'char_encoding'/BitsInteger(5), # see C.R1001-G
+            # Byte 3: Language Indicator
+            'lang_ind'/BitsInteger(8), # see C.R1001-G
+            # Bytes 4-35: Service Provider Name
+            'spn'/Bytewise(GsmString(32))
+        )
+
+class EF_AD(TransparentEF):
+    '''3.4.33 Administrative Data'''
+
+    _test_de_encode = [
+        ( "000000", { 'ms_operation_mode' : 'normal', 'additional_info' : '0000', 'rfu' : '' } ),
+    ]
+    _test_no_pad = True
+
+    class OP_MODE(enum.IntEnum):
+        normal = 0x00
+        type_approval = 0x80
+        normal_and_specific_facilities = 0x01
+        type_approval_and_specific_facilities = 0x81
+        maintenance_off_line = 0x02
+        cell_test = 0x04
+
+    def __init__(self, fid='6f43', sfid=None, name='EF.AD',
+                 desc='Service Provider Name', size=(3, None), **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
+        self._construct = Struct(
+            # Byte 1: Display Condition
+            'ms_operation_mode'/Enum(Byte, self.OP_MODE),
+            # Bytes 2-3: Additional information
+            'additional_info'/HexAdapter(Bytes(2)),
+            # Bytes 4..: RFU
+            'rfu'/HexAdapter(GreedyBytesRFU),
+        )
+
+
+class EF_SMS(LinFixedEF):
+    '''3.4.27 Short Messages'''
+    def __init__(self, fid='6f3c', sfid=None, name='EF.SMS', desc='Short messages', **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(2, 255), **kwargs)
+        self._construct = Struct(
+            # Byte 1: Status
+            'status'/BitStruct(
+                'rfu87'/BitsRFU(2),
+                'protection'/Flag,
+                'rfu54'/BitsRFU(2),
+                'status'/FlagsEnum(BitsInteger(2), read=0, to_be_read=1, sent=2, to_be_sent=3),
+                'used'/Flag,
+            ),
+            # Byte 2: Length
+            'length'/Int8ub,
+            # Bytes 3..: SMS Transport Layer Message
+            'tpdu'/Bytes(lambda ctx: ctx.length if ctx.status.used else 0),
+        )
+
+
+class DF_CDMA(CardDF):
+    def __init__(self):
+        super().__init__(fid='7f25', name='DF.CDMA',
+                         desc='CDMA related files (3GPP2 C.S0023-D)')
+        files = [
+            # TODO: lots of other files
+            EF_ServiceTable('6f32', None, 'EF.CST',
+                            'CDMA Service Table', table=EF_CST_map, size=(5, 16)),
+            EF_SPN(),
+            EF_AD(),
+            EF_SMS(),
+        ]
+        self.add_files(files)
+
+
+class CardProfileRUIM(CardProfile):
+    '''R-UIM card profile as per 3GPP2 C.S0023-D'''
+
+    ORDER = 2
+
+    def __init__(self):
+        super().__init__('R-UIM', desc='CDMA R-UIM Card', cla="a0",
+                         sel_ctrl="0000", files_in_mf=[DF_TELECOM(), DF_GSM(), DF_CDMA()])
+
+    @staticmethod
+    def decode_select_response(resp_hex: str) -> object:
+        # TODO: Response parameters/data in case of DF_CDMA (section 2.6)
+        return CardProfileSIM.decode_select_response(resp_hex)
+
+    @staticmethod
+    def match_with_card(scc: SimCardCommands) -> bool:
+        return match_ruim(scc)
+
+class AddonRUIM(CardProfileAddon):
+    """An Addon that can be found on on a combined SIM + RUIM or UICC + RUIM to support CDMA."""
+    def __init__(self):
+        files = [
+            DF_CDMA()
+        ]
+        super().__init__('RUIM', desc='CDMA RUIM', files_in_mf=files)
+
+    def probe(self, card: 'CardBase') -> bool:
+        return card.file_exists(self.files_in_mf[0].fid)

pySim/commands.py

@@ -5,7 +5,7 @@
 
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
-# Copyright (C) 2010-2021  Harald Welte <laforge@gnumonks.org>
+# Copyright (C) 2010-2023  Harald Welte <laforge@gnumonks.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,20 +21,87 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+from typing import List, Optional, Tuple
+import typing # construct also has a Union, so we do typing.Union below
+
 from construct import *
 from pySim.construct import LV
-from pySim.utils import rpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, str_sanitize, expand_hex
+from pySim.utils import rpad, lpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, i2h, str_sanitize, expand_hex
+from pySim.utils import Hexstr, SwHexstr, ResTuple
 from pySim.exceptions import SwMatchError
+from pySim.transport import LinkBase
 
+# A path can be either just a FID or a list of FID
+Path = typing.Union[Hexstr, List[Hexstr]]
+
+def lchan_nr_to_cla(cla: int, lchan_nr: int) -> int:
+    """Embed a logical channel number into the CLA byte."""
+    # TS 102 221 10.1.1 Coding of Class Byte
+    if lchan_nr < 4:
+        # standard logical channel number
+        if cla >> 4 in [0x0, 0xA, 0x8]:
+            return (cla & 0xFC) | (lchan_nr & 3)
+        else:
+            raise ValueError('Undefined how to use CLA %2X with logical channel %u' % (cla, lchan_nr))
+    elif lchan_nr < 16:
+        # extended logical channel number
+        if cla >> 6 in [1, 3]:
+            return (cla & 0xF0) | ((lchan_nr - 4) & 0x0F)
+        else:
+            raise ValueError('Undefined how to use CLA %2X with logical channel %u' % (cla, lchan_nr))
+    else:
+        raise ValueError('logical channel outside of range 0 .. 15')
+
+def cla_with_lchan(cla_byte: Hexstr, lchan_nr: int) -> Hexstr:
+    """Embed a logical channel number into the hex-string encoded CLA value."""
+    cla_int = h2i(cla_byte)[0]
+    return i2h([lchan_nr_to_cla(cla_int, lchan_nr)])
 
 class SimCardCommands:
-    def __init__(self, transport):
+    """Class providing methods for various card-specific commands such as SELECT, READ BINARY, etc.
+    Historically one instance exists below CardBase, but with the introduction of multiple logical
+    channels there can be multiple instances.  The lchan number will then be patched into the CLA
+    byte by the respective instance. """
+    def __init__(self, transport: LinkBase, lchan_nr: int = 0):
         self._tp = transport
-        self.cla_byte = "a0"
+        self._cla_byte = None
         self.sel_ctrl = "0000"
+        self.lchan_nr = lchan_nr
+        # invokes the setter below
+        self.cla_byte = "a0"
+
+    def fork_lchan(self, lchan_nr: int) -> 'SimCardCommands':
+        """Fork a per-lchan specific SimCardCommands instance off the current instance."""
+        ret = SimCardCommands(transport = self._tp, lchan_nr = lchan_nr)
+        ret.cla_byte = self._cla_byte
+        ret.sel_ctrl = self.sel_ctrl
+        return ret
+
+    @property
+    def cla_byte(self) -> Hexstr:
+        """Return the (cached) patched default CLA byte for this card."""
+        return self._cla4lchan
+
+    @cla_byte.setter
+    def cla_byte(self, new_val: Hexstr):
+        """Set the (raw, without lchan) default CLA value for this card."""
+        self._cla_byte = new_val
+        # compute cached result
+        self._cla4lchan = cla_with_lchan(self._cla_byte, self.lchan_nr)
+
+    def cla4lchan(self, cla: Hexstr) -> Hexstr:
+        """Compute the lchan-patched value of the given CLA value. If no CLA
+        value is provided as argument, the lchan-patched version of the SimCardCommands._cla_byte
+        value is used. Most commands will use the latter, while some wish to override it and
+        can pass it as argument here."""
+        if not cla:
+            # return cached result to avoid re-computing this over and over again
+            return self._cla4lchan
+        else:
+            return cla_with_lchan(cla, self.lchan_nr)
 
     # Extract a single FCP item from TLV
-    def __parse_fcp(self, fcp):
+    def __parse_fcp(self, fcp: Hexstr):
         # see also: ETSI TS 102 221, chapter 11.1.1.3.1 Response for MF,
         # DF or ADF
         from pytlv.TLV import TLV
@@ -88,11 +155,11 @@ class SimCardCommands:
         else:
             return int(r[-1][4:8], 16)
 
-    def get_atr(self) -> str:
+    def get_atr(self) -> Hexstr:
         """Return the ATR of the currently inserted card."""
         return self._tp.get_atr()
 
-    def try_select_path(self, dir_list):
+    def try_select_path(self, dir_list: List[Hexstr]) -> List[ResTuple]:
         """ Try to select a specified path
 
         Args:
@@ -110,7 +177,7 @@ class SimCardCommands:
                 return rv
         return rv
 
-    def select_path(self, dir_list):
+    def select_path(self, dir_list: Path) -> List[Hexstr]:
         """Execute SELECT for an entire list/path of FIDs.
 
         Args:
@@ -127,7 +194,7 @@ class SimCardCommands:
             rv.append(data)
         return rv
 
-    def select_file(self, fid: str):
+    def select_file(self, fid: Hexstr) -> ResTuple:
         """Execute SELECT a given file by FID.
 
         Args:
@@ -136,11 +203,11 @@ class SimCardCommands:
 
         return self._tp.send_apdu_checksw(self.cla_byte + "a4" + self.sel_ctrl + "02" + fid)
 
-    def select_parent_df(self):
+    def select_parent_df(self) -> ResTuple:
         """Execute SELECT to switch to the parent DF """
         return self._tp.send_apdu_checksw(self.cla_byte + "a4030400")
 
-    def select_adf(self, aid: str):
+    def select_adf(self, aid: Hexstr) -> ResTuple:
         """Execute SELECT a given Applicaiton ADF.
 
         Args:
@@ -150,7 +217,7 @@ class SimCardCommands:
         aidlen = ("0" + format(len(aid) // 2, 'x'))[-2:]
         return self._tp.send_apdu_checksw(self.cla_byte + "a4" + "0404" + aidlen + aid)
 
-    def read_binary(self, ef, length: int = None, offset: int = 0):
+    def read_binary(self, ef: Path, length: int = None, offset: int = 0) -> ResTuple:
         """Execute READD BINARY.
 
         Args:
@@ -181,7 +248,21 @@ class SimCardCommands:
             chunk_offset += chunk_len
         return total_data, sw
 
-    def update_binary(self, ef, data: str, offset: int = 0, verify: bool = False, conserve: bool = False):
+    def __verify_binary(self, ef, data: str, offset: int = 0):
+        """Verify contents of transparent EF.
+
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+                data : hex string of expected data
+                offset : byte offset in file from which to start verifying
+        """
+        res = self.read_binary(ef, len(data) // 2, offset)
+        if res[0].lower() != data.lower():
+            raise ValueError('Binary verification failed (expected %s, got %s)' % (
+                data.lower(), res[0].lower()))
+
+    def update_binary(self, ef: Path, data: Hexstr, offset: int = 0, verify: bool = False,
+                      conserve: bool = False) -> ResTuple:
         """Execute UPDATE BINARY.
 
         Args:
@@ -198,9 +279,16 @@ class SimCardCommands:
 
         # Save write cycles by reading+comparing before write
         if conserve:
-            data_current, sw = self.read_binary(ef, data_length, offset)
-            if data_current == data:
-                return None, sw
+            try:
+                data_current, sw = self.read_binary(ef, data_length, offset)
+                if data_current == data:
+                    return None, sw
+            except Exception:
+                # cannot read data. This is not a fatal error, as reading is just done to
+                # conserve the amount of smart card writes.  The access conditions of the file
+                # may well permit us to UPDATE but not permit us to READ.  So let's ignore
+                # any such exception during READ.
+                pass
 
         self.select_path(ef)
         total_data = ''
@@ -219,23 +307,10 @@ class SimCardCommands:
             total_data += data
             chunk_offset += chunk_len
         if verify:
-            self.verify_binary(ef, data, offset)
+            self.__verify_binary(ef, data, offset)
         return total_data, chunk_sw
 
-    def verify_binary(self, ef, data: str, offset: int = 0):
-        """Verify contents of transparent EF.
-
-        Args:
-                ef : string or list of strings indicating name or path of transparent EF
-                data : hex string of expected data
-                offset : byte offset in file from which to start verifying
-        """
-        res = self.read_binary(ef, len(data) // 2, offset)
-        if res[0].lower() != data.lower():
-            raise ValueError('Binary verification failed (expected %s, got %s)' % (
-                data.lower(), res[0].lower()))
-
-    def read_record(self, ef, rec_no: int):
+    def read_record(self, ef: Path, rec_no: int) -> ResTuple:
         """Execute READ RECORD.
 
         Args:
@@ -247,8 +322,21 @@ class SimCardCommands:
         pdu = self.cla_byte + 'b2%02x04%02x' % (rec_no, rec_length)
         return self._tp.send_apdu_checksw(pdu)
 
-    def update_record(self, ef, rec_no: int, data: str, force_len: bool = False, verify: bool = False,
-                      conserve: bool = False):
+    def __verify_record(self, ef: Path, rec_no: int, data: str):
+        """Verify record against given data
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+                rec_no : record number to read
+                data : hex string of data to be verified
+        """
+        res = self.read_record(ef, rec_no)
+        if res[0].lower() != data.lower():
+            raise ValueError('Record verification failed (expected %s, got %s)' % (
+                data.lower(), res[0].lower()))
+
+    def update_record(self, ef: Path, rec_no: int, data: Hexstr, force_len: bool = False,
+                      verify: bool = False, conserve: bool = False, leftpad: bool = False) -> ResTuple:
         """Execute UPDATE RECORD.
 
         Args:
@@ -258,6 +346,7 @@ class SimCardCommands:
                 force_len : enforce record length by using the actual data length
                 verify : verify data by re-reading the record
                 conserve : read record and compare it with data, skip write on match
+                leftpad : apply 0xff padding from the left instead from the right side.
         """
 
         res = self.select_path(ef)
@@ -274,35 +363,32 @@ class SimCardCommands:
                 raise ValueError('Data length exceeds record length (expected max %d, got %d)' % (
                     rec_length, len(data) // 2))
             elif (len(data) // 2 < rec_length):
-                data = rpad(data, rec_length * 2)
+                if leftpad:
+                    data = lpad(data, rec_length * 2)
+                else:
+                    data = rpad(data, rec_length * 2)
 
         # Save write cycles by reading+comparing before write
         if conserve:
-            data_current, sw = self.read_record(ef, rec_no)
-            data_current = data_current[0:rec_length*2]
-            if data_current == data:
-                return None, sw
+            try:
+                data_current, sw = self.read_record(ef, rec_no)
+                data_current = data_current[0:rec_length*2]
+                if data_current == data:
+                    return None, sw
+            except Exception:
+                # cannot read data. This is not a fatal error, as reading is just done to
+                # conserve the amount of smart card writes.  The access conditions of the file
+                # may well permit us to UPDATE but not permit us to READ.  So let's ignore
+                # any such exception during READ.
+                pass
 
         pdu = (self.cla_byte + 'dc%02x04%02x' % (rec_no, rec_length)) + data
         res = self._tp.send_apdu_checksw(pdu)
         if verify:
-            self.verify_record(ef, rec_no, data)
+            self.__verify_record(ef, rec_no, data)
         return res
 
-    def verify_record(self, ef, rec_no: int, data: str):
-        """Verify record against given data
-
-        Args:
-                ef : string or list of strings indicating name or path of linear fixed EF
-                rec_no : record number to read
-                data : hex string of data to be verified
-        """
-        res = self.read_record(ef, rec_no)
-        if res[0].lower() != data.lower():
-            raise ValueError('Record verification failed (expected %s, got %s)' % (
-                data.lower(), res[0].lower()))
-
-    def record_size(self, ef):
+    def record_size(self, ef: Path) -> int:
         """Determine the record size of given file.
 
         Args:
@@ -311,7 +397,7 @@ class SimCardCommands:
         r = self.select_path(ef)
         return self.__record_len(r)
 
-    def record_count(self, ef):
+    def record_count(self, ef: Path) -> int:
         """Determine the number of records in given file.
 
         Args:
@@ -320,7 +406,7 @@ class SimCardCommands:
         r = self.select_path(ef)
         return self.__len(r) // self.__record_len(r)
 
-    def binary_size(self, ef):
+    def binary_size(self, ef: Path) -> int:
         """Determine the size of given transparent file.
 
         Args:
@@ -330,14 +416,14 @@ class SimCardCommands:
         return self.__len(r)
 
     # TS 102 221 Section 11.3.1 low-level helper
-    def _retrieve_data(self, tag: int, first: bool = True):
+    def _retrieve_data(self, tag: int, first: bool = True) -> ResTuple:
         if first:
-            pdu = '80cb008001%02x' % (tag)
+            pdu = self.cla4lchan('80') + 'cb008001%02x' % (tag)
         else:
-            pdu = '80cb000000'
+            pdu = self.cla4lchan('80') + 'cb000000'
         return self._tp.send_apdu_checksw(pdu)
 
-    def retrieve_data(self, ef, tag: int):
+    def retrieve_data(self, ef: Path, tag: int) -> ResTuple:
         """Execute RETRIEVE DATA, see also TS 102 221 Section 11.3.1.
 
         Args
@@ -357,17 +443,17 @@ class SimCardCommands:
         return total_data, sw
 
     # TS 102 221 Section 11.3.2 low-level helper
-    def _set_data(self, data: str, first: bool = True):
+    def _set_data(self, data: Hexstr, first: bool = True) -> ResTuple:
         if first:
             p1 = 0x80
         else:
             p1 = 0x00
         if isinstance(data, bytes) or isinstance(data, bytearray):
             data = b2h(data)
-        pdu = '80db00%02x%02x%s' % (p1, len(data)//2, data)
+        pdu = self.cla4lchan('80') + 'db00%02x%02x%s' % (p1, len(data)//2, data)
         return self._tp.send_apdu_checksw(pdu)
 
-    def set_data(self, ef, tag: int, value: str, verify: bool = False, conserve: bool = False):
+    def set_data(self, ef, tag: int, value: str, verify: bool = False, conserve: bool = False) -> ResTuple:
         """Execute SET DATA.
 
         Args
@@ -398,7 +484,7 @@ class SimCardCommands:
             remaining = remaining[255:]
         return rdata, sw
 
-    def run_gsm(self, rand: str):
+    def run_gsm(self, rand: Hexstr) -> ResTuple:
         """Execute RUN GSM ALGORITHM.
 
         Args:
@@ -407,9 +493,9 @@ class SimCardCommands:
         if len(rand) != 32:
             raise ValueError('Invalid rand')
         self.select_path(['3f00', '7f20'])
-        return self._tp.send_apdu(self.cla_byte + '88000010' + rand)
+        return self._tp.send_apdu_checksw(self.cla4lchan('a0') + '88000010' + rand, sw='9000')
 
-    def authenticate(self, rand: str, autn: str, context='3g'):
+    def authenticate(self, rand: Hexstr, autn: Hexstr, context: str = '3g') -> ResTuple:
         """Execute AUTHENTICATE (USIM/ISIM).
 
         Args:
@@ -437,15 +523,15 @@ class SimCardCommands:
             ret = {'successful_3g_authentication': data}
         return (ret, sw)
 
-    def status(self):
+    def status(self) -> ResTuple:
         """Execute a STATUS command as per TS 102 221 Section 11.1.2."""
-        return self._tp.send_apdu_checksw('80F20000ff')
+        return self._tp.send_apdu_checksw(self.cla4lchan('80') + 'F20000ff')
 
-    def deactivate_file(self):
+    def deactivate_file(self) -> ResTuple:
         """Execute DECATIVATE FILE command as per TS 102 221 Section 11.1.14."""
         return self._tp.send_apdu_constr_checksw(self.cla_byte, '04', '00', '00', None, None, None)
 
-    def activate_file(self, fid):
+    def activate_file(self, fid: Hexstr) -> ResTuple:
         """Execute ACTIVATE FILE command as per TS 102 221 Section 11.1.15.
 
         Args:
@@ -453,27 +539,31 @@ class SimCardCommands:
         """
         return self._tp.send_apdu_checksw(self.cla_byte + '44000002' + fid)
 
-    def create_file(self, payload: Hexstr):
+    def create_file(self, payload: Hexstr) -> ResTuple:
         """Execute CREEATE FILE command as per TS 102 222 Section 6.3"""
         return self._tp.send_apdu_checksw(self.cla_byte + 'e00000%02x%s' % (len(payload)//2, payload))
 
-    def delete_file(self, fid):
+    def resize_file(self, payload: Hexstr) -> ResTuple:
+        """Execute RESIZE FILE command as per TS 102 222 Section 6.10"""
+        return self._tp.send_apdu_checksw(self.cla4lchan('80') + 'd40000%02x%s' % (len(payload)//2, payload))
+
+    def delete_file(self, fid: Hexstr) -> ResTuple:
         """Execute DELETE FILE command as per TS 102 222 Section 6.4"""
         return self._tp.send_apdu_checksw(self.cla_byte + 'e4000002' + fid)
 
-    def terminate_df(self, fid):
+    def terminate_df(self, fid: Hexstr) -> ResTuple:
         """Execute TERMINATE DF command as per TS 102 222 Section 6.7"""
         return self._tp.send_apdu_checksw(self.cla_byte + 'e6000002' + fid)
 
-    def terminate_ef(self, fid):
+    def terminate_ef(self, fid: Hexstr) -> ResTuple:
         """Execute TERMINATE EF command as per TS 102 222 Section 6.8"""
         return self._tp.send_apdu_checksw(self.cla_byte + 'e8000002' + fid)
 
-    def terminate_card_usage(self):
+    def terminate_card_usage(self) -> ResTuple:
         """Execute TERMINATE CARD USAGE command as per TS 102 222 Section 6.9"""
         return self._tp.send_apdu_checksw(self.cla_byte + 'fe000000')
 
-    def manage_channel(self, mode='open', lchan_nr=0):
+    def manage_channel(self, mode: str = 'open', lchan_nr: int =0) -> ResTuple:
         """Execute MANAGE CHANNEL command as per TS 102 221 Section 11.1.17.
 
         Args:
@@ -487,18 +577,18 @@ class SimCardCommands:
         pdu = self.cla_byte + '70%02x%02x00' % (p1, lchan_nr)
         return self._tp.send_apdu_checksw(pdu)
 
-    def reset_card(self):
+    def reset_card(self) -> Hexstr:
         """Physically reset the card"""
         return self._tp.reset_card()
 
-    def _chv_process_sw(self, op_name, chv_no, pin_code, sw):
+    def _chv_process_sw(self, op_name: str, chv_no: int, pin_code: Hexstr, sw: SwHexstr):
         if sw_match(sw, '63cx'):
             raise RuntimeError('Failed to %s chv_no 0x%02X with code 0x%s, %i tries left.' %
                                (op_name, chv_no, b2h(pin_code).upper(), int(sw[3])))
         elif (sw != '9000'):
             raise SwMatchError(sw, '9000')
 
-    def verify_chv(self, chv_no: int, code: str):
+    def verify_chv(self, chv_no: int, code: Hexstr) -> ResTuple:
         """Verify a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -525,7 +615,7 @@ class SimCardCommands:
         self._chv_process_sw('unblock', chv_no, pin_code, sw)
         return (data, sw)
 
-    def change_chv(self, chv_no: int, pin_code: str, new_pin_code: str):
+    def change_chv(self, chv_no: int, pin_code: Hexstr, new_pin_code: Hexstr) -> ResTuple:
         """Change a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -539,7 +629,7 @@ class SimCardCommands:
         self._chv_process_sw('change', chv_no, pin_code, sw)
         return (data, sw)
 
-    def disable_chv(self, chv_no: int, pin_code: str):
+    def disable_chv(self, chv_no: int, pin_code: Hexstr) -> ResTuple:
         """Disable a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -553,7 +643,7 @@ class SimCardCommands:
         self._chv_process_sw('disable', chv_no, pin_code, sw)
         return (data, sw)
 
-    def enable_chv(self, chv_no: int, pin_code: str):
+    def enable_chv(self, chv_no: int, pin_code: Hexstr) -> ResTuple:
         """Enable a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -566,7 +656,7 @@ class SimCardCommands:
         self._chv_process_sw('enable', chv_no, pin_code, sw)
         return (data, sw)
 
-    def envelope(self, payload: str):
+    def envelope(self, payload: Hexstr) -> ResTuple:
         """Send one ENVELOPE command to the SIM
 
         Args:
@@ -574,7 +664,7 @@ class SimCardCommands:
         """
         return self._tp.send_apdu_checksw('80c20000%02x%s' % (len(payload)//2, payload))
 
-    def terminal_profile(self, payload: str):
+    def terminal_profile(self, payload: Hexstr) -> ResTuple:
         """Send TERMINAL PROFILE to card
 
         Args:
@@ -585,7 +675,7 @@ class SimCardCommands:
         return (data, sw)
 
     # ETSI TS 102 221 11.1.22
-    def suspend_uicc(self, min_len_secs: int = 60, max_len_secs: int = 43200):
+    def suspend_uicc(self, min_len_secs: int = 60, max_len_secs: int = 43200) -> Tuple[int, Hexstr, SwHexstr]:
         """Send SUSPEND UICC to the card.
 
         Args:
@@ -606,7 +696,7 @@ class SimCardCommands:
 
         def decode_duration(enc: Hexstr) -> int:
             time_unit = enc[:2]
-            length = h2i(enc[2:4])
+            length = h2i(enc[2:4])[0]
             if time_unit == '04':
                 return length * 10*24*60*60
             elif time_unit == '03':
@@ -627,6 +717,19 @@ class SimCardCommands:
         resume_token = data[4:]
         return (negotiated_duration_secs, resume_token, sw)
 
+    # ETSI TS 102 221 11.1.22
+    def resume_uicc(self, token: Hexstr) -> ResTuple:
+        """Send SUSPEND UICC (resume) to the card."""
+        if len(h2b(token)) != 8:
+            raise ValueError("Token must be 8 bytes long")
+        data, sw = self._tp.send_apdu_checksw('8076010008' + token)
+        return (data, sw)
+
     def get_data(self, tag: int, cla: int = 0x00):
         data, sw = self._tp.send_apdu('%02xca%04x00' % (cla, tag))
         return (data, sw)
+
+    # TS 31.102 Section 7.5.2
+    def get_identity(self, context: int) -> Tuple[Hexstr, SwHexstr]:
+        data, sw = self._tp.send_apdu_checksw('807800%02x00' % (context))
+        return (data, sw)

pySim/construct.py

@@ -6,6 +6,8 @@ from construct.core import evaluate, BitwisableString
 from construct.lib import integertypes
 from pySim.utils import b2h, h2b, swap_nibbles
 import gsm0338
+import codecs
+import ipaddress
 
 """Utility code related to the integration of the 'construct' declarative parser."""
 
@@ -34,6 +36,199 @@ class HexAdapter(Adapter):
     def _encode(self, obj, context, path):
         return h2b(obj)
 
+class Utf8Adapter(Adapter):
+    """convert a bytes() type that contains utf8 encoded text to human readable text."""
+
+    def _decode(self, obj, context, path):
+        # In case the string contains only 0xff bytes we interpret it as an empty string
+        if obj == b'\xff' * len(obj):
+                return ""
+        return codecs.decode(obj, "utf-8")
+
+    def _encode(self, obj, context, path):
+        return codecs.encode(obj, "utf-8")
+
+class GsmOrUcs2Adapter(Adapter):
+    """Try to encode into a GSM 03.38 string; if that fails, fall back to UCS-2 as described
+    in TS 102 221 Annex A."""
+    def _decode(self, obj, context, path):
+        # In case the string contains only 0xff bytes we interpret it as an empty string
+        if obj == b'\xff' * len(obj):
+                return ""
+        # one of the magic bytes of TS 102 221 Annex A
+        if obj[0] in [0x80, 0x81, 0x82]:
+            ad = Ucs2Adapter(GreedyBytes)
+        else:
+            ad = GsmString(GreedyBytes)
+        return ad._decode(obj, context, path)
+
+    def _encode(self, obj, context, path):
+        # first try GSM 03.38; then fall back to TS 102 221 Annex A UCS-2
+        try:
+            ad = GsmString(GreedyBytes)
+            return ad._encode(obj, context, path)
+        except:
+            ad = Ucs2Adapter(GreedyBytes)
+            return ad._encode(obj, context, path)
+
+class Ucs2Adapter(Adapter):
+    """convert a bytes() type that contains UCS2 encoded characters encoded as defined in TS 102 221
+    Annex A to normal python string representation (and back)."""
+    def _decode(self, obj, context, path):
+        # In case the string contains only 0xff bytes we interpret it as an empty string
+        if obj == b'\xff' * len(obj):
+                return ""
+        if obj[0] == 0x80:
+            # TS 102 221 Annex A Variant 1
+            return codecs.decode(obj[1:], 'utf_16_be')
+        elif obj[0] == 0x81:
+            # TS 102 221 Annex A Variant 2
+            out = ""
+            # second byte contains a value indicating the number of characters
+            num_of_chars = obj[1]
+            # the third byte contains an 8 bit number which defines bits 15 to 8 of a 16 bit base
+            # pointer, where bit 16 is set to zero, and bits 7 to 1 are also set to zero. These
+            # sixteen bits constitute a base pointer to a "half-page" in the UCS2 code space
+            base_ptr = obj[2] << 7
+            for ch in obj[3:3+num_of_chars]:
+                # if bit 8 of the byte is set to zero, the remaining 7 bits of the byte contain a
+                # GSM Default Alphabet character, whereas if bit 8 of the byte is set to one, then
+                # the remaining seven bits are an offset value added to the 16 bit base pointer
+                # defined earlier, and the resultant 16 bit value is a UCS2 code point
+                if ch & 0x80:
+                    codepoint = (ch & 0x7f) + base_ptr
+                    out += codecs.decode(codepoint.to_bytes(2, byteorder='big'), 'utf_16_be')
+                else:
+                    out += codecs.decode(bytes([ch]), 'gsm03.38')
+            return out
+        elif obj[0] == 0x82:
+            # TS 102 221 Annex A Variant 3
+            out = ""
+            # second byte contains a value indicating the number of characters
+            num_of_chars = obj[1]
+            # third and fourth bytes contain a 16 bit number which defines the complete 16 bit base
+            # pointer to a half-page in the UCS2 code space, for use with some or all of the
+            # remaining bytes in the string
+            base_ptr = obj[2] << 8 | obj[3]
+            for ch in obj[4:4+num_of_chars]:
+                # if bit 8 of the byte is set to zero, the remaining 7 bits of the byte contain a
+                # GSM Default Alphabet character, whereas if bit 8 of the byte is set to one, the
+                # remaining seven bits are an offset value added to the base pointer defined in
+                # bytes three and four, and the resultant 16 bit value is a UCS2 code point, else: #
+                # GSM default alphabet
+                if ch & 0x80:
+                    codepoint = (ch & 0x7f) + base_ptr
+                    out += codecs.decode(codepoint.to_bytes(2, byteorder='big'), 'utf_16_be')
+                else:
+                    out += codecs.decode(bytes([ch]), 'gsm03.38')
+            return out
+        else:
+            raise ValueError('First byte of TS 102 221 UCS-2 must be 0x80, 0x81 or 0x82')
+
+    def _encode(self, obj, context, path):
+        def encodable_in_gsm338(instr: str) -> bool:
+            """Determine if given input string is encode-ale in gsm03.38."""
+            try:
+                # TODO: figure out if/how we can constrain to default alphabet.  The gsm0338
+                # library seems to include the spanish lock/shift table
+                codecs.encode(instr, 'gsm03.38')
+            except ValueError:
+                return False
+            return True
+
+        def codepoints_not_in_gsm338(instr: str) -> typing.List[int]:
+            """Return an integer list of UCS2 codepoints for all characters of 'inster'
+            which are not representable in the GSM 03.38 default alphabet."""
+            codepoint_list = []
+            for c in instr:
+                if encodable_in_gsm338(c):
+                    continue
+                c_codepoint = int.from_bytes(codecs.encode(c, 'utf_16_be'), byteorder='big')
+                codepoint_list.append(c_codepoint)
+            return codepoint_list
+
+        def diff_between_min_and_max_of_list(inlst: typing.List) -> int:
+            return max(inlst) - min(inlst)
+
+        def encodable_in_variant2(instr: str) -> bool:
+            codepoint_prefix = None
+            for c in instr:
+                if encodable_in_gsm338(c):
+                    continue
+                c_codepoint = int.from_bytes(codecs.encode(c, 'utf_16_be'), byteorder='big')
+                if c_codepoint >= 0x8000:
+                    return False
+                c_prefix = c_codepoint >> 7
+                if codepoint_prefix is None:
+                    codepoint_prefix = c_prefix
+                else:
+                    if c_prefix != codepoint_prefix:
+                        return False
+            return True
+
+        def encodable_in_variant3(instr: str) -> bool:
+            codepoint_list = codepoints_not_in_gsm338(instr)
+            # compute delta between max and min; check if it's encodable in 7 bits
+            if diff_between_min_and_max_of_list(codepoint_list) >= 0x80:
+                return False
+            return True
+
+        def _encode_variant1(instr: str) -> bytes:
+            """Encode according to TS 102 221 Annex A Variant 1"""
+            return b'\x80' + codecs.encode(obj, 'utf_16_be')
+
+        def _encode_variant2(instr: str) -> bytes:
+            """Encode according to TS 102 221 Annex A Variant 2"""
+            codepoint_prefix = None
+            # second byte contains a value indicating the number of characters
+            hdr = b'\x81' + len(instr).to_bytes(1, byteorder='big')
+            chars = b''
+            for c in instr:
+                try:
+                    enc = codecs.encode(c, 'gsm03.38')
+                except ValueError:
+                    c_codepoint = int.from_bytes(codecs.encode(c, 'utf_16_be'), byteorder='big')
+                    c_prefix = c_codepoint >> 7
+                    if codepoint_prefix is None:
+                        codepoint_prefix = c_prefix
+                    assert codepoint_prefix == c_prefix
+                    enc = (0x80 + (c_codepoint & 0x7f)).to_bytes(1, byteorder='big')
+                chars += enc
+            if codepoint_prefix == None:
+                codepoint_prefix = 0
+            return hdr + codepoint_prefix.to_bytes(1, byteorder='big') + chars
+
+        def _encode_variant3(instr: str) -> bytes:
+            """Encode according to TS 102 221 Annex A Variant 3"""
+            # second byte contains a value indicating the number of characters
+            hdr = b'\x82' + len(instr).to_bytes(1, byteorder='big')
+            chars = b''
+            codepoint_list = codepoints_not_in_gsm338(instr)
+            codepoint_base = min(codepoint_list)
+            for c in instr:
+                try:
+                    # if bit 8 of the byte is set to zero, the remaining 7 bits of the byte contain a GSM
+                    # Default # Alphabet character
+                    enc = codecs.encode(c, 'gsm03.38')
+                except ValueError:
+                    # if bit 8 of the byte is set to one, the remaining seven bits are an offset
+                    # value added to the base pointer defined in bytes three and four, and the
+                    # resultant 16 bit value is a UCS2 code point
+                    c_codepoint = int.from_bytes(codecs.encode(c, 'utf_16_be'), byteorder='big')
+                    c_codepoint_delta = c_codepoint - codepoint_base
+                    assert c_codepoint_delta < 0x80
+                    enc = (0x80 + c_codepoint_delta).to_bytes(1, byteorder='big')
+                chars += enc
+            # third and fourth bytes contain a 16 bit number which defines the complete 16 bit base
+            # pointer to a half-page in the UCS2 code space
+            return hdr + codepoint_base.to_bytes(2, byteorder='big') + chars
+
+        if encodable_in_variant2(obj):
+            return _encode_variant2(obj)
+        elif encodable_in_variant3(obj):
+            return _encode_variant3(obj)
+        else:
+            return _encode_variant1(obj)
 
 class BcdAdapter(Adapter):
     """convert a bytes() type to a string of BCD nibbles."""
@@ -44,6 +239,23 @@ class BcdAdapter(Adapter):
     def _encode(self, obj, context, path):
         return h2b(swap_nibbles(obj))
 
+class PlmnAdapter(BcdAdapter):
+    """convert a bytes(3) type to BCD string like 262-02 or 262-002."""
+    def _decode(self, obj, context, path):
+        bcd = super()._decode(obj, context, path)
+        if bcd[3] == 'f':
+            return '-'.join([bcd[:3], bcd[4:]])
+        else:
+            return '-'.join([bcd[:3], bcd[3:]])
+
+    def _encode(self, obj, context, path):
+        l = obj.split('-')
+        if len(l[1]) == 2:
+            bcd = l[0] + 'f' + l[1]
+        else:
+            bcd = l[0] + l[1]
+        return super()._encode(bcd, context, path)
+
 class InvertAdapter(Adapter):
     """inverse logic (false->true, true->false)."""
     @staticmethod
@@ -66,26 +278,49 @@ class InvertAdapter(Adapter):
 
 class Rpad(Adapter):
     """
-    Encoder appends padding bytes (b'\\xff') up to target size.
-    Decoder removes trailing padding bytes.
+    Encoder appends padding bytes (b'\\xff') or characters up to target size.
+    Decoder removes trailing padding bytes/characters.
 
     Parameters:
         subcon: Subconstruct as defined by construct library
         pattern: set padding pattern (default: b'\\xff')
+        num_per_byte: number of 'elements' per byte. E.g. for hex nibbles: 2
     """
 
-    def __init__(self, subcon, pattern=b'\xff'):
+    def __init__(self, subcon, pattern=b'\xff', num_per_byte=1):
         super().__init__(subcon)
         self.pattern = pattern
+        self.num_per_byte = num_per_byte
 
     def _decode(self, obj, context, path):
         return obj.rstrip(self.pattern)
 
     def _encode(self, obj, context, path):
-        if len(obj) > self.sizeof():
+        target_size = self.sizeof() * self.num_per_byte
+        if len(obj) > target_size:
             raise SizeofError("Input ({}) exceeds target size ({})".format(
-                len(obj), self.sizeof()))
-        return obj + self.pattern * (self.sizeof() - len(obj))
+                len(obj), target_size))
+        return obj + self.pattern * (target_size - len(obj))
+
+class MultiplyAdapter(Adapter):
+    """
+    Decoder multiplies by multiplicator
+    Encoder divides by multiplicator
+
+    Parameters:
+        subcon: Subconstruct as defined by construct library
+        multiplier: Multiplier to apply to raw encoded value
+    """
+
+    def __init__(self, subcon, multiplicator):
+        super().__init__(subcon)
+        self.multiplicator = multiplicator
+
+    def _decode(self, obj, context, path):
+        return obj * 8
+
+    def _encode(self, obj, context, path):
+        return obj // 8
 
 
 class GsmStringAdapter(Adapter):
@@ -102,6 +337,32 @@ class GsmStringAdapter(Adapter):
     def _encode(self, obj, context, path):
         return obj.encode(self.codec, self.err)
 
+class Ipv4Adapter(Adapter):
+    """
+    Encoder converts from 4 bytes to string representation (A.B.C.D).
+    Decoder converts from string representation (A.B.C.D) to four bytes.
+    """
+    def _decode(self, obj, context, path):
+        ia = ipaddress.IPv4Address(obj)
+        return ia.compressed
+
+    def _encode(self, obj, context, path):
+        ia = ipaddress.IPv4Address(obj)
+        return ia.packed
+
+class Ipv6Adapter(Adapter):
+    """
+    Encoder converts from 16 bytes to string representation.
+    Decoder converts from string representation to 16 bytes.
+    """
+    def _decode(self, obj, context, path):
+        ia = ipaddress.IPv6Address(obj)
+        return ia.compressed
+
+    def _encode(self, obj, context, path):
+        ia = ipaddress.IPv6Address(obj)
+        return ia.packed
+
 
 def filter_dict(d, exclude_prefix='_'):
     """filter the input dict to ensure no keys starting with 'exclude_prefix' remain."""
@@ -137,13 +398,25 @@ def normalize_construct(c):
     return r
 
 
-def parse_construct(c, raw_bin_data: bytes, length: typing.Optional[int] = None, exclude_prefix: str = '_'):
+def parse_construct(c, raw_bin_data: bytes, length: typing.Optional[int] = None, exclude_prefix: str = '_', context: dict = {}):
     """Helper function to wrap around normalize_construct() and filter_dict()."""
     if not length:
         length = len(raw_bin_data)
-    parsed = c.parse(raw_bin_data, total_len=length)
+    try:
+        parsed = c.parse(raw_bin_data, total_len=length, **context)
+    except StreamError as e:
+        # if the input is all-ff, this means the content is undefined.  Let's avoid passing StreamError
+        # exceptions in those situations (which might occur if a length field 0xff is 255 but then there's
+        # actually less bytes in the remainder of the file.
+        if all([v == 0xff for v in raw_bin_data]):
+            return None
+        else:
+            raise e
     return normalize_construct(parsed)
 
+def build_construct(c, decoded_data, context: dict = {}):
+    """Helper function to handle total_len."""
+    return c.build(decoded_data, total_len=None, **context)
 
 # here we collect some shared / common definitions of data types
 LV = Prefixed(Int8ub, HexAdapter(GreedyBytes))
@@ -206,6 +479,20 @@ def GsmString(n):
     '''
     return GsmStringAdapter(Rpad(Bytes(n), pattern=b'\xff'), codec='gsm03.38')
 
+def GsmOrUcs2String(n):
+    '''
+    GSM 03.38 or UCS-2 (TS 102 221 Annex A) encoded byte string of fixed length n.
+    Encoder appends padding bytes (b'\\xff') to maintain
+    length. Decoder removes those trailing bytes.
+
+    Exceptions are raised for invalid characters
+    and length excess.
+
+    Parameters:
+        n (Integer): Fixed length of the encoded byte string
+    '''
+    return GsmOrUcs2Adapter(Rpad(Bytes(n), pattern=b'\xff'))
+
 class GreedyInteger(Construct):
     """A variable-length integer implementation, think of combining GrredyBytes with BytesInteger."""
     def __init__(self, signed=False, swapped=False, minlen=0):

pySim/esim/__init__.py

@@ -0,0 +1,16 @@
+import sys
+from importlib import resources
+
+import asn1tools
+
+def compile_asn1_subdir(subdir_name:str):
+    """Helper function that compiles ASN.1 syntax from all files within given subdir"""
+    asn_txt = ''
+    __ver = sys.version_info
+    if (__ver.major, __ver.minor) >= (3, 9):
+        for i in resources.files('pySim.esim').joinpath('asn1').joinpath(subdir_name).iterdir():
+            asn_txt += i.read_text()
+            asn_txt += "\n"
+    #else:
+        #print(resources.read_text(__name__, 'asn1/rsp.asn'))
+    return asn1tools.compile_string(asn_txt, codec='der')

pySim/esim/asn1/rsp/PKIX1Explicit88.asn

@@ -0,0 +1,657 @@
+PKIX1Explicit88 { iso(1) identified-organization(3) dod(6) internet(1)
+  security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-explicit(18) }
+
+DEFINITIONS EXPLICIT TAGS ::=
+
+BEGIN
+
+-- EXPORTS ALL --
+
+-- IMPORTS NONE --
+
+-- UNIVERSAL Types defined in 1993 and 1998 ASN.1
+-- and required by this specification
+
+-- pycrate: UniversalString, BMPString and UTF8String already in the builtin types
+
+--UniversalString ::= [UNIVERSAL 28] IMPLICIT OCTET STRING
+        -- UniversalString is defined in ASN.1:1993
+
+--BMPString ::= [UNIVERSAL 30] IMPLICIT OCTET STRING
+      -- BMPString is the subtype of UniversalString and models
+      -- the Basic Multilingual Plane of ISO/IEC 10646
+
+--UTF8String ::= [UNIVERSAL 12] IMPLICIT OCTET STRING
+      -- The content of this type conforms to RFC 3629.
+
+-- PKIX specific OIDs
+
+id-pkix  OBJECT IDENTIFIER  ::=
+         { iso(1) identified-organization(3) dod(6) internet(1)
+                    security(5) mechanisms(5) pkix(7) }
+
+-- PKIX arcs
+
+id-pe OBJECT IDENTIFIER ::= { id-pkix 1 }
+        -- arc for private certificate extensions
+id-qt OBJECT IDENTIFIER ::= { id-pkix 2 }
+        -- arc for policy qualifier types
+id-kp OBJECT IDENTIFIER ::= { id-pkix 3 }
+        -- arc for extended key purpose OIDS
+id-ad OBJECT IDENTIFIER ::= { id-pkix 48 }
+        -- arc for access descriptors
+
+-- policyQualifierIds for Internet policy qualifiers
+
+id-qt-cps      OBJECT IDENTIFIER ::=  { id-qt 1 }
+      -- OID for CPS qualifier
+id-qt-unotice  OBJECT IDENTIFIER ::=  { id-qt 2 }
+      -- OID for user notice qualifier
+
+-- access descriptor definitions
+
+id-ad-ocsp         OBJECT IDENTIFIER ::= { id-ad 1 }
+id-ad-caIssuers    OBJECT IDENTIFIER ::= { id-ad 2 }
+id-ad-timeStamping OBJECT IDENTIFIER ::= { id-ad 3 }
+id-ad-caRepository OBJECT IDENTIFIER ::= { id-ad 5 }
+
+-- attribute data types
+
+Attribute               ::= SEQUENCE {
+      type             AttributeType,
+      values    SET OF AttributeValue }
+            -- at least one value is required
+
+AttributeType           ::= OBJECT IDENTIFIER
+
+AttributeValue          ::= ANY -- DEFINED BY AttributeType
+
+AttributeTypeAndValue   ::= SEQUENCE {
+        type    AttributeType,
+        value   AttributeValue }
+
+-- suggested naming attributes: Definition of the following
+--   information object set may be augmented to meet local
+--   requirements.  Note that deleting members of the set may
+--   prevent interoperability with conforming implementations.
+-- presented in pairs: the AttributeType followed by the
+--   type definition for the corresponding AttributeValue
+
+-- Arc for standard naming attributes
+
+id-at OBJECT IDENTIFIER ::= { joint-iso-ccitt(2) ds(5) 4 }
+
+-- Naming attributes of type X520name
+
+id-at-name                AttributeType ::= { id-at 41 }
+id-at-surname             AttributeType ::= { id-at  4 }
+id-at-givenName           AttributeType ::= { id-at 42 }
+id-at-initials            AttributeType ::= { id-at 43 }
+id-at-generationQualifier AttributeType ::= { id-at 44 }
+
+-- Naming attributes of type X520Name:
+--   X520name ::= DirectoryString (SIZE (1..ub-name))
+--
+-- Expanded to avoid parameterized type:
+X520name ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-name)),
+      printableString   PrintableString (SIZE (1..ub-name)),
+      universalString   UniversalString (SIZE (1..ub-name)),
+      utf8String        UTF8String      (SIZE (1..ub-name)),
+      bmpString         BMPString       (SIZE (1..ub-name)) }
+
+-- Naming attributes of type X520CommonName
+
+id-at-commonName        AttributeType ::= { id-at 3 }
+
+-- Naming attributes of type X520CommonName:
+--   X520CommonName ::= DirectoryName (SIZE (1..ub-common-name))
+--
+-- Expanded to avoid parameterized type:
+X520CommonName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-common-name)),
+      printableString   PrintableString (SIZE (1..ub-common-name)),
+      universalString   UniversalString (SIZE (1..ub-common-name)),
+      utf8String        UTF8String      (SIZE (1..ub-common-name)),
+      bmpString         BMPString       (SIZE (1..ub-common-name)) }
+
+-- Naming attributes of type X520LocalityName
+
+id-at-localityName      AttributeType ::= { id-at 7 }
+
+-- Naming attributes of type X520LocalityName:
+--   X520LocalityName ::= DirectoryName (SIZE (1..ub-locality-name))
+--
+-- Expanded to avoid parameterized type:
+X520LocalityName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-locality-name)),
+      printableString   PrintableString (SIZE (1..ub-locality-name)),
+      universalString   UniversalString (SIZE (1..ub-locality-name)),
+      utf8String        UTF8String      (SIZE (1..ub-locality-name)),
+      bmpString         BMPString       (SIZE (1..ub-locality-name)) }
+
+-- Naming attributes of type X520StateOrProvinceName
+
+id-at-stateOrProvinceName AttributeType ::= { id-at 8 }
+
+-- Naming attributes of type X520StateOrProvinceName:
+--   X520StateOrProvinceName ::= DirectoryName (SIZE (1..ub-state-name))
+--
+-- Expanded to avoid parameterized type:
+X520StateOrProvinceName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-state-name)),
+      printableString   PrintableString (SIZE (1..ub-state-name)),
+      universalString   UniversalString (SIZE (1..ub-state-name)),
+      utf8String        UTF8String      (SIZE (1..ub-state-name)),
+      bmpString         BMPString       (SIZE (1..ub-state-name)) }
+
+-- Naming attributes of type X520OrganizationName
+
+id-at-organizationName  AttributeType ::= { id-at 10 }
+
+-- Naming attributes of type X520OrganizationName:
+--   X520OrganizationName ::=
+--          DirectoryName (SIZE (1..ub-organization-name))
+--
+-- Expanded to avoid parameterized type:
+X520OrganizationName ::= CHOICE {
+      teletexString     TeletexString
+                          (SIZE (1..ub-organization-name)),
+      printableString   PrintableString
+                          (SIZE (1..ub-organization-name)),
+      universalString   UniversalString
+                          (SIZE (1..ub-organization-name)),
+      utf8String        UTF8String
+                          (SIZE (1..ub-organization-name)),
+      bmpString         BMPString
+                          (SIZE (1..ub-organization-name))  }
+
+-- Naming attributes of type X520OrganizationalUnitName
+
+id-at-organizationalUnitName AttributeType ::= { id-at 11 }
+
+-- Naming attributes of type X520OrganizationalUnitName:
+--   X520OrganizationalUnitName ::=
+--          DirectoryName (SIZE (1..ub-organizational-unit-name))
+--
+-- Expanded to avoid parameterized type:
+X520OrganizationalUnitName ::= CHOICE {
+      teletexString     TeletexString
+                          (SIZE (1..ub-organizational-unit-name)),
+      printableString   PrintableString
+                          (SIZE (1..ub-organizational-unit-name)),
+      universalString   UniversalString
+                          (SIZE (1..ub-organizational-unit-name)),
+      utf8String        UTF8String
+                          (SIZE (1..ub-organizational-unit-name)),
+      bmpString         BMPString
+                          (SIZE (1..ub-organizational-unit-name)) }
+
+-- Naming attributes of type X520Title
+
+id-at-title             AttributeType ::= { id-at 12 }
+
+-- Naming attributes of type X520Title:
+--   X520Title ::= DirectoryName (SIZE (1..ub-title))
+--
+-- Expanded to avoid parameterized type:
+X520Title ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-title)),
+      printableString   PrintableString (SIZE (1..ub-title)),
+      universalString   UniversalString (SIZE (1..ub-title)),
+      utf8String        UTF8String      (SIZE (1..ub-title)),
+      bmpString         BMPString       (SIZE (1..ub-title)) }
+
+-- Naming attributes of type X520dnQualifier
+
+id-at-dnQualifier       AttributeType ::= { id-at 46 }
+
+X520dnQualifier ::=     PrintableString
+
+-- Naming attributes of type X520countryName (digraph from IS 3166)
+
+id-at-countryName       AttributeType ::= { id-at 6 }
+
+X520countryName ::=     PrintableString (SIZE (2))
+
+-- Naming attributes of type X520SerialNumber
+
+id-at-serialNumber      AttributeType ::= { id-at 5 }
+
+X520SerialNumber ::=    PrintableString (SIZE (1..ub-serial-number))
+
+-- Naming attributes of type X520Pseudonym
+
+id-at-pseudonym         AttributeType ::= { id-at 65 }
+
+-- Naming attributes of type X520Pseudonym:
+--   X520Pseudonym ::= DirectoryName (SIZE (1..ub-pseudonym))
+--
+-- Expanded to avoid parameterized type:
+X520Pseudonym ::= CHOICE {
+   teletexString     TeletexString   (SIZE (1..ub-pseudonym)),
+   printableString   PrintableString (SIZE (1..ub-pseudonym)),
+   universalString   UniversalString (SIZE (1..ub-pseudonym)),
+   utf8String        UTF8String      (SIZE (1..ub-pseudonym)),
+   bmpString         BMPString       (SIZE (1..ub-pseudonym)) }
+
+-- Naming attributes of type DomainComponent (from RFC 4519)
+
+id-domainComponent   AttributeType ::= { 0 9 2342 19200300 100 1 25 }
+
+DomainComponent ::=  IA5String
+
+-- Legacy attributes
+
+pkcs-9 OBJECT IDENTIFIER ::=
+       { iso(1) member-body(2) us(840) rsadsi(113549) pkcs(1) 9 }
+
+id-emailAddress      AttributeType ::= { pkcs-9 1 }
+
+EmailAddress ::=     IA5String (SIZE (1..ub-emailaddress-length))
+
+-- naming data types --
+
+Name ::= CHOICE { -- only one possibility for now --
+      rdnSequence  RDNSequence }
+
+RDNSequence ::= SEQUENCE OF RelativeDistinguishedName
+
+DistinguishedName ::=   RDNSequence
+
+RelativeDistinguishedName ::= SET SIZE (1..MAX) OF AttributeTypeAndValue
+
+-- Directory string type --
+
+DirectoryString ::= CHOICE {
+      teletexString       TeletexString   (SIZE (1..MAX)),
+      printableString     PrintableString (SIZE (1..MAX)),
+      universalString     UniversalString (SIZE (1..MAX)),
+      utf8String          UTF8String      (SIZE (1..MAX)),
+      bmpString           BMPString       (SIZE (1..MAX)) }
+
+-- certificate and CRL specific structures begin here
+
+Certificate  ::=  SEQUENCE  {
+     tbsCertificate       TBSCertificate,
+     signatureAlgorithm   AlgorithmIdentifier,
+     signature            BIT STRING  }
+
+TBSCertificate  ::=  SEQUENCE  {
+     version         [0]  Version DEFAULT v1,
+     serialNumber         CertificateSerialNumber,
+     signature            AlgorithmIdentifier,
+     issuer               Name,
+     validity             Validity,
+     subject              Name,
+     subjectPublicKeyInfo SubjectPublicKeyInfo,
+     issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
+                          -- If present, version MUST be v2 or v3
+     subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
+                          -- If present, version MUST be v2 or v3
+     extensions      [3]  Extensions OPTIONAL
+                          -- If present, version MUST be v3 --  }
+
+Version  ::=  INTEGER  {  v1(0), v2(1), v3(2)  }
+
+CertificateSerialNumber  ::=  INTEGER
+
+Validity ::= SEQUENCE {
+     notBefore      Time,
+     notAfter       Time  }
+
+Time ::= CHOICE {
+     utcTime        UTCTime,
+     generalTime    GeneralizedTime }
+
+UniqueIdentifier  ::=  BIT STRING
+
+SubjectPublicKeyInfo  ::=  SEQUENCE  {
+     algorithm            AlgorithmIdentifier,
+     subjectPublicKey     BIT STRING  }
+
+Extensions  ::=  SEQUENCE SIZE (1..MAX) OF Extension
+
+Extension  ::=  SEQUENCE  {
+     extnID      OBJECT IDENTIFIER,
+     critical    BOOLEAN DEFAULT FALSE,
+     extnValue   OCTET STRING
+                 -- contains the DER encoding of an ASN.1 value
+                 -- corresponding to the extension type identified
+                 -- by extnID
+     }
+
+-- CRL structures
+
+CertificateList  ::=  SEQUENCE  {
+     tbsCertList          TBSCertList,
+     signatureAlgorithm   AlgorithmIdentifier,
+     signature            BIT STRING  }
+
+TBSCertList  ::=  SEQUENCE  {
+     version                 Version OPTIONAL,
+                                   -- if present, MUST be v2
+     signature               AlgorithmIdentifier,
+     issuer                  Name,
+     thisUpdate              Time,
+     nextUpdate              Time OPTIONAL,
+     revokedCertificates     SEQUENCE OF SEQUENCE  {
+          userCertificate         CertificateSerialNumber,
+          revocationDate          Time,
+          crlEntryExtensions      Extensions OPTIONAL
+                                   -- if present, version MUST be v2
+                               }  OPTIONAL,
+     crlExtensions           [0] Extensions OPTIONAL }
+                                   -- if present, version MUST be v2
+
+-- Version, Time, CertificateSerialNumber, and Extensions were
+-- defined earlier for use in the certificate structure
+
+AlgorithmIdentifier  ::=  SEQUENCE  {
+     algorithm               OBJECT IDENTIFIER,
+     parameters              ANY DEFINED BY algorithm OPTIONAL  }
+                                -- contains a value of the type
+                                -- registered for use with the
+                                -- algorithm object identifier value
+
+-- X.400 address syntax starts here
+
+ORAddress ::= SEQUENCE {
+   built-in-standard-attributes BuiltInStandardAttributes,
+   built-in-domain-defined-attributes
+                   BuiltInDomainDefinedAttributes OPTIONAL,
+   -- see also teletex-domain-defined-attributes
+   extension-attributes ExtensionAttributes OPTIONAL }
+
+-- Built-in Standard Attributes
+
+BuiltInStandardAttributes ::= SEQUENCE {
+   country-name                  CountryName OPTIONAL,
+   administration-domain-name    AdministrationDomainName OPTIONAL,
+   network-address           [0] IMPLICIT NetworkAddress OPTIONAL,
+     -- see also extended-network-address
+   terminal-identifier       [1] IMPLICIT TerminalIdentifier OPTIONAL,
+   private-domain-name       [2] PrivateDomainName OPTIONAL,
+   organization-name         [3] IMPLICIT OrganizationName OPTIONAL,
+     -- see also teletex-organization-name
+   numeric-user-identifier   [4] IMPLICIT NumericUserIdentifier
+                                 OPTIONAL,
+   personal-name             [5] IMPLICIT PersonalName OPTIONAL,
+     -- see also teletex-personal-name
+   organizational-unit-names [6] IMPLICIT OrganizationalUnitNames
+                                 OPTIONAL }
+     -- see also teletex-organizational-unit-names
+
+CountryName ::= [APPLICATION 1] CHOICE {
+   x121-dcc-code         NumericString
+                           (SIZE (ub-country-name-numeric-length)),
+   iso-3166-alpha2-code  PrintableString
+                           (SIZE (ub-country-name-alpha-length)) }
+
+AdministrationDomainName ::= [APPLICATION 2] CHOICE {
+   numeric   NumericString   (SIZE (0..ub-domain-name-length)),
+   printable PrintableString (SIZE (0..ub-domain-name-length)) }
+
+NetworkAddress ::= X121Address  -- see also extended-network-address
+
+X121Address ::= NumericString (SIZE (1..ub-x121-address-length))
+
+TerminalIdentifier ::= PrintableString (SIZE (1..ub-terminal-id-length))
+
+PrivateDomainName ::= CHOICE {
+   numeric   NumericString   (SIZE (1..ub-domain-name-length)),
+   printable PrintableString (SIZE (1..ub-domain-name-length)) }
+
+OrganizationName ::= PrintableString
+                            (SIZE (1..ub-organization-name-length))
+  -- see also teletex-organization-name
+
+NumericUserIdentifier ::= NumericString
+                            (SIZE (1..ub-numeric-user-id-length))
+
+PersonalName ::= SET {
+   surname     [0] IMPLICIT PrintableString
+                    (SIZE (1..ub-surname-length)),
+   given-name  [1] IMPLICIT PrintableString
+                    (SIZE (1..ub-given-name-length)) OPTIONAL,
+   initials    [2] IMPLICIT PrintableString
+                    (SIZE (1..ub-initials-length)) OPTIONAL,
+   generation-qualifier [3] IMPLICIT PrintableString
+                    (SIZE (1..ub-generation-qualifier-length))
+                    OPTIONAL }
+  -- see also teletex-personal-name
+
+OrganizationalUnitNames ::= SEQUENCE SIZE (1..ub-organizational-units)
+                             OF OrganizationalUnitName
+  -- see also teletex-organizational-unit-names
+
+OrganizationalUnitName ::= PrintableString (SIZE
+                    (1..ub-organizational-unit-name-length))
+
+-- Built-in Domain-defined Attributes
+
+BuiltInDomainDefinedAttributes ::= SEQUENCE SIZE
+                    (1..ub-domain-defined-attributes) OF
+                    BuiltInDomainDefinedAttribute
+
+BuiltInDomainDefinedAttribute ::= SEQUENCE {
+   type PrintableString (SIZE
+                   (1..ub-domain-defined-attribute-type-length)),
+   value PrintableString (SIZE
+                   (1..ub-domain-defined-attribute-value-length)) }
+
+-- Extension Attributes
+
+ExtensionAttributes ::= SET SIZE (1..ub-extension-attributes) OF
+               ExtensionAttribute
+
+ExtensionAttribute ::=  SEQUENCE {
+   extension-attribute-type [0] IMPLICIT INTEGER
+                   (0..ub-extension-attributes),
+   extension-attribute-value [1]
+                   ANY DEFINED BY extension-attribute-type }
+
+-- Extension types and attribute values
+
+common-name INTEGER ::= 1
+
+CommonName ::= PrintableString (SIZE (1..ub-common-name-length))
+
+teletex-common-name INTEGER ::= 2
+
+TeletexCommonName ::= TeletexString (SIZE (1..ub-common-name-length))
+
+teletex-organization-name INTEGER ::= 3
+
+TeletexOrganizationName ::=
+                TeletexString (SIZE (1..ub-organization-name-length))
+
+teletex-personal-name INTEGER ::= 4
+
+TeletexPersonalName ::= SET {
+   surname     [0] IMPLICIT TeletexString
+                    (SIZE (1..ub-surname-length)),
+   given-name  [1] IMPLICIT TeletexString
+                    (SIZE (1..ub-given-name-length)) OPTIONAL,
+   initials    [2] IMPLICIT TeletexString
+                    (SIZE (1..ub-initials-length)) OPTIONAL,
+   generation-qualifier [3] IMPLICIT TeletexString
+                    (SIZE (1..ub-generation-qualifier-length))
+                    OPTIONAL }
+
+teletex-organizational-unit-names INTEGER ::= 5
+
+TeletexOrganizationalUnitNames ::= SEQUENCE SIZE
+      (1..ub-organizational-units) OF TeletexOrganizationalUnitName
+
+TeletexOrganizationalUnitName ::= TeletexString
+                  (SIZE (1..ub-organizational-unit-name-length))
+
+pds-name INTEGER ::= 7
+
+PDSName ::= PrintableString (SIZE (1..ub-pds-name-length))
+
+physical-delivery-country-name INTEGER ::= 8
+
+PhysicalDeliveryCountryName ::= CHOICE {
+   x121-dcc-code NumericString (SIZE (ub-country-name-numeric-length)),
+   iso-3166-alpha2-code PrintableString
+                               (SIZE (ub-country-name-alpha-length)) }
+
+postal-code INTEGER ::= 9
+
+PostalCode ::= CHOICE {
+   numeric-code   NumericString (SIZE (1..ub-postal-code-length)),
+   printable-code PrintableString (SIZE (1..ub-postal-code-length)) }
+
+physical-delivery-office-name INTEGER ::= 10
+
+PhysicalDeliveryOfficeName ::= PDSParameter
+
+physical-delivery-office-number INTEGER ::= 11
+
+PhysicalDeliveryOfficeNumber ::= PDSParameter
+
+extension-OR-address-components INTEGER ::= 12
+
+ExtensionORAddressComponents ::= PDSParameter
+
+physical-delivery-personal-name INTEGER ::= 13
+
+PhysicalDeliveryPersonalName ::= PDSParameter
+
+physical-delivery-organization-name INTEGER ::= 14
+
+PhysicalDeliveryOrganizationName ::= PDSParameter
+
+extension-physical-delivery-address-components INTEGER ::= 15
+
+ExtensionPhysicalDeliveryAddressComponents ::= PDSParameter
+
+unformatted-postal-address INTEGER ::= 16
+
+UnformattedPostalAddress ::= SET {
+   printable-address SEQUENCE SIZE (1..ub-pds-physical-address-lines)
+        OF PrintableString (SIZE (1..ub-pds-parameter-length)) OPTIONAL,
+   teletex-string TeletexString
+        (SIZE (1..ub-unformatted-address-length)) OPTIONAL }
+
+street-address INTEGER ::= 17
+
+StreetAddress ::= PDSParameter
+
+post-office-box-address INTEGER ::= 18
+
+PostOfficeBoxAddress ::= PDSParameter
+
+poste-restante-address INTEGER ::= 19
+
+PosteRestanteAddress ::= PDSParameter
+
+unique-postal-name INTEGER ::= 20
+
+UniquePostalName ::= PDSParameter
+
+local-postal-attributes INTEGER ::= 21
+
+LocalPostalAttributes ::= PDSParameter
+
+PDSParameter ::= SET {
+   printable-string PrintableString
+                (SIZE(1..ub-pds-parameter-length)) OPTIONAL,
+   teletex-string TeletexString
+                (SIZE(1..ub-pds-parameter-length)) OPTIONAL }
+
+extended-network-address INTEGER ::= 22
+
+ExtendedNetworkAddress ::= CHOICE {
+   e163-4-address SEQUENCE {
+      number      [0] IMPLICIT NumericString
+                       (SIZE (1..ub-e163-4-number-length)),
+      sub-address [1] IMPLICIT NumericString
+                       (SIZE (1..ub-e163-4-sub-address-length))
+                       OPTIONAL },
+   psap-address   [0] IMPLICIT PresentationAddress }
+
+PresentationAddress ::= SEQUENCE {
+    pSelector     [0] EXPLICIT OCTET STRING OPTIONAL,
+    sSelector     [1] EXPLICIT OCTET STRING OPTIONAL,
+    tSelector     [2] EXPLICIT OCTET STRING OPTIONAL,
+    nAddresses    [3] EXPLICIT SET SIZE (1..MAX) OF OCTET STRING }
+
+terminal-type  INTEGER ::= 23
+
+TerminalType ::= INTEGER {
+   telex        (3),
+   teletex      (4),
+   g3-facsimile (5),
+   g4-facsimile (6),
+   ia5-terminal (7),
+   videotex     (8) } (0..ub-integer-options)
+
+-- Extension Domain-defined Attributes
+
+teletex-domain-defined-attributes INTEGER ::= 6
+
+TeletexDomainDefinedAttributes ::= SEQUENCE SIZE
+   (1..ub-domain-defined-attributes) OF TeletexDomainDefinedAttribute
+
+TeletexDomainDefinedAttribute ::= SEQUENCE {
+        type TeletexString
+               (SIZE (1..ub-domain-defined-attribute-type-length)),
+        value TeletexString
+               (SIZE (1..ub-domain-defined-attribute-value-length)) }
+
+--  specifications of Upper Bounds MUST be regarded as mandatory
+--  from Annex B of ITU-T X.411 Reference Definition of MTS Parameter
+--  Upper Bounds
+
+-- Upper Bounds
+ub-name INTEGER ::= 32768
+ub-common-name INTEGER ::= 64
+ub-locality-name INTEGER ::= 128
+ub-state-name INTEGER ::= 128
+ub-organization-name INTEGER ::= 64
+ub-organizational-unit-name INTEGER ::= 64
+ub-title INTEGER ::= 64
+ub-serial-number INTEGER ::= 64
+ub-match INTEGER ::= 128
+ub-emailaddress-length INTEGER ::= 255
+ub-common-name-length INTEGER ::= 64
+ub-country-name-alpha-length INTEGER ::= 2
+ub-country-name-numeric-length INTEGER ::= 3
+ub-domain-defined-attributes INTEGER ::= 4
+ub-domain-defined-attribute-type-length INTEGER ::= 8
+ub-domain-defined-attribute-value-length INTEGER ::= 128
+ub-domain-name-length INTEGER ::= 16
+ub-extension-attributes INTEGER ::= 256
+ub-e163-4-number-length INTEGER ::= 15
+ub-e163-4-sub-address-length INTEGER ::= 40
+ub-generation-qualifier-length INTEGER ::= 3
+ub-given-name-length INTEGER ::= 16
+ub-initials-length INTEGER ::= 5
+ub-integer-options INTEGER ::= 256
+ub-numeric-user-id-length INTEGER ::= 32
+ub-organization-name-length INTEGER ::= 64
+ub-organizational-unit-name-length INTEGER ::= 32
+ub-organizational-units INTEGER ::= 4
+ub-pds-name-length INTEGER ::= 16
+ub-pds-parameter-length INTEGER ::= 30
+ub-pds-physical-address-lines INTEGER ::= 6
+ub-postal-code-length INTEGER ::= 16
+ub-pseudonym INTEGER ::= 128
+ub-surname-length INTEGER ::= 40
+ub-terminal-id-length INTEGER ::= 24
+ub-unformatted-address-length INTEGER ::= 180
+ub-x121-address-length INTEGER ::= 16
+
+-- Note - upper bounds on string types, such as TeletexString, are
+-- measured in characters.  Excepting PrintableString or IA5String, a
+-- significantly greater number of octets will be required to hold
+-- such a value.  As a minimum, 16 octets, or twice the specified
+-- upper bound, whichever is the larger, should be allowed for
+-- TeletexString.  For UTF8String or UniversalString at least four
+-- times the upper bound should be allowed.
+
+END
+

pySim/esim/asn1/rsp/PKIX1Implicit88.asn

@@ -0,0 +1,343 @@
+PKIX1Implicit88 { iso(1) identified-organization(3) dod(6) internet(1)
+  security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-implicit(19) }
+
+DEFINITIONS IMPLICIT TAGS ::=
+
+BEGIN
+
+-- EXPORTS ALL --
+
+IMPORTS
+      id-pe, id-kp, id-qt-unotice, id-qt-cps,
+      ORAddress, Name, RelativeDistinguishedName,
+      CertificateSerialNumber, Attribute, DirectoryString
+      FROM PKIX1Explicit88 { iso(1) identified-organization(3)
+            dod(6) internet(1) security(5) mechanisms(5) pkix(7)
+            id-mod(0) id-pkix1-explicit(18) };
+
+-- ISO arc for standard certificate and CRL extensions
+
+id-ce OBJECT IDENTIFIER  ::=  {joint-iso-ccitt(2) ds(5) 29}
+
+-- authority key identifier OID and syntax
+
+id-ce-authorityKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 35 }
+
+AuthorityKeyIdentifier ::= SEQUENCE {
+    keyIdentifier             [0] KeyIdentifier            OPTIONAL,
+    authorityCertIssuer       [1] GeneralNames             OPTIONAL,
+    authorityCertSerialNumber [2] CertificateSerialNumber  OPTIONAL }
+    -- authorityCertIssuer and authorityCertSerialNumber MUST both
+    -- be present or both be absent
+
+KeyIdentifier ::= OCTET STRING
+
+-- subject key identifier OID and syntax
+
+id-ce-subjectKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 14 }
+
+SubjectKeyIdentifier ::= KeyIdentifier
+
+-- key usage extension OID and syntax
+
+id-ce-keyUsage OBJECT IDENTIFIER ::=  { id-ce 15 }
+
+KeyUsage ::= BIT STRING {
+     digitalSignature        (0),
+     nonRepudiation          (1),  -- recent editions of X.509 have
+                                -- renamed this bit to contentCommitment
+     keyEncipherment         (2),
+     dataEncipherment        (3),
+     keyAgreement            (4),
+     keyCertSign             (5),
+     cRLSign                 (6),
+     encipherOnly            (7),
+     decipherOnly            (8) }
+
+-- private key usage period extension OID and syntax
+
+id-ce-privateKeyUsagePeriod OBJECT IDENTIFIER ::=  { id-ce 16 }
+
+PrivateKeyUsagePeriod ::= SEQUENCE {
+     notBefore       [0]     GeneralizedTime OPTIONAL,
+     notAfter        [1]     GeneralizedTime OPTIONAL }
+     -- either notBefore or notAfter MUST be present
+
+-- certificate policies extension OID and syntax
+
+id-ce-certificatePolicies OBJECT IDENTIFIER ::=  { id-ce 32 }
+
+anyPolicy OBJECT IDENTIFIER ::= { id-ce-certificatePolicies 0 }
+
+CertificatePolicies ::= SEQUENCE SIZE (1..MAX) OF PolicyInformation
+
+PolicyInformation ::= SEQUENCE {
+     policyIdentifier   CertPolicyId,
+     policyQualifiers   SEQUENCE SIZE (1..MAX) OF
+             PolicyQualifierInfo OPTIONAL }
+
+CertPolicyId ::= OBJECT IDENTIFIER
+
+PolicyQualifierInfo ::= SEQUENCE {
+     policyQualifierId  PolicyQualifierId,
+     qualifier          ANY DEFINED BY policyQualifierId }
+
+-- Implementations that recognize additional policy qualifiers MUST
+-- augment the following definition for PolicyQualifierId
+
+PolicyQualifierId ::= OBJECT IDENTIFIER ( id-qt-cps | id-qt-unotice )
+
+-- CPS pointer qualifier
+
+CPSuri ::= IA5String
+
+-- user notice qualifier
+
+UserNotice ::= SEQUENCE {
+     noticeRef        NoticeReference OPTIONAL,
+     explicitText     DisplayText OPTIONAL }
+
+NoticeReference ::= SEQUENCE {
+     organization     DisplayText,
+     noticeNumbers    SEQUENCE OF INTEGER }
+
+DisplayText ::= CHOICE {
+     ia5String        IA5String      (SIZE (1..200)),
+     visibleString    VisibleString  (SIZE (1..200)),
+     bmpString        BMPString      (SIZE (1..200)),
+     utf8String       UTF8String     (SIZE (1..200)) }
+
+-- policy mapping extension OID and syntax
+
+id-ce-policyMappings OBJECT IDENTIFIER ::=  { id-ce 33 }
+
+PolicyMappings ::= SEQUENCE SIZE (1..MAX) OF SEQUENCE {
+     issuerDomainPolicy      CertPolicyId,
+     subjectDomainPolicy     CertPolicyId }
+
+-- subject alternative name extension OID and syntax
+
+id-ce-subjectAltName OBJECT IDENTIFIER ::=  { id-ce 17 }
+
+SubjectAltName ::= GeneralNames
+
+GeneralNames ::= SEQUENCE SIZE (1..MAX) OF GeneralName
+
+GeneralName ::= CHOICE {
+     otherName                 [0]  AnotherName,
+     rfc822Name                [1]  IA5String,
+     dNSName                   [2]  IA5String,
+     x400Address               [3]  ORAddress,
+     directoryName             [4]  Name,
+     ediPartyName              [5]  EDIPartyName,
+     uniformResourceIdentifier [6]  IA5String,
+     iPAddress                 [7]  OCTET STRING,
+     registeredID              [8]  OBJECT IDENTIFIER }
+
+-- AnotherName replaces OTHER-NAME ::= TYPE-IDENTIFIER, as
+-- TYPE-IDENTIFIER is not supported in the '88 ASN.1 syntax
+
+AnotherName ::= SEQUENCE {
+     type-id    OBJECT IDENTIFIER,
+     value      [0] EXPLICIT ANY DEFINED BY type-id }
+
+EDIPartyName ::= SEQUENCE {
+     nameAssigner              [0]  DirectoryString OPTIONAL,
+     partyName                 [1]  DirectoryString }
+
+-- issuer alternative name extension OID and syntax
+
+id-ce-issuerAltName OBJECT IDENTIFIER ::=  { id-ce 18 }
+
+IssuerAltName ::= GeneralNames
+
+id-ce-subjectDirectoryAttributes OBJECT IDENTIFIER ::=  { id-ce 9 }
+
+SubjectDirectoryAttributes ::= SEQUENCE SIZE (1..MAX) OF Attribute
+
+-- basic constraints extension OID and syntax
+
+id-ce-basicConstraints OBJECT IDENTIFIER ::=  { id-ce 19 }
+
+BasicConstraints ::= SEQUENCE {
+     cA                      BOOLEAN DEFAULT FALSE,
+     pathLenConstraint       INTEGER (0..MAX) OPTIONAL }
+
+-- name constraints extension OID and syntax
+
+id-ce-nameConstraints OBJECT IDENTIFIER ::=  { id-ce 30 }
+
+NameConstraints ::= SEQUENCE {
+     permittedSubtrees       [0]     GeneralSubtrees OPTIONAL,
+     excludedSubtrees        [1]     GeneralSubtrees OPTIONAL }
+
+GeneralSubtrees ::= SEQUENCE SIZE (1..MAX) OF GeneralSubtree
+
+GeneralSubtree ::= SEQUENCE {
+     base                    GeneralName,
+     minimum         [0]     BaseDistance DEFAULT 0,
+     maximum         [1]     BaseDistance OPTIONAL }
+
+BaseDistance ::= INTEGER (0..MAX)
+
+-- policy constraints extension OID and syntax
+
+id-ce-policyConstraints OBJECT IDENTIFIER ::=  { id-ce 36 }
+
+PolicyConstraints ::= SEQUENCE {
+     requireExplicitPolicy   [0]     SkipCerts OPTIONAL,
+     inhibitPolicyMapping    [1]     SkipCerts OPTIONAL }
+
+SkipCerts ::= INTEGER (0..MAX)
+
+-- CRL distribution points extension OID and syntax
+
+id-ce-cRLDistributionPoints     OBJECT IDENTIFIER  ::=  {id-ce 31}
+
+CRLDistributionPoints ::= SEQUENCE SIZE (1..MAX) OF DistributionPoint
+
+DistributionPoint ::= SEQUENCE {
+     distributionPoint       [0]     DistributionPointName OPTIONAL,
+     reasons                 [1]     ReasonFlags OPTIONAL,
+     cRLIssuer               [2]     GeneralNames OPTIONAL }
+
+DistributionPointName ::= CHOICE {
+     fullName                [0]     GeneralNames,
+     nameRelativeToCRLIssuer [1]     RelativeDistinguishedName }
+
+ReasonFlags ::= BIT STRING {
+     unused                  (0),
+     keyCompromise           (1),
+     cACompromise            (2),
+     affiliationChanged      (3),
+     superseded              (4),
+     cessationOfOperation    (5),
+     certificateHold         (6),
+     privilegeWithdrawn      (7),
+     aACompromise            (8) }
+
+-- extended key usage extension OID and syntax
+
+id-ce-extKeyUsage OBJECT IDENTIFIER ::= {id-ce 37}
+
+ExtKeyUsageSyntax ::= SEQUENCE SIZE (1..MAX) OF KeyPurposeId
+
+KeyPurposeId ::= OBJECT IDENTIFIER
+
+-- permit unspecified key uses
+
+anyExtendedKeyUsage OBJECT IDENTIFIER ::= { id-ce-extKeyUsage 0 }
+
+-- extended key purpose OIDs
+
+id-kp-serverAuth             OBJECT IDENTIFIER ::= { id-kp 1 }
+id-kp-clientAuth             OBJECT IDENTIFIER ::= { id-kp 2 }
+id-kp-codeSigning            OBJECT IDENTIFIER ::= { id-kp 3 }
+id-kp-emailProtection        OBJECT IDENTIFIER ::= { id-kp 4 }
+id-kp-timeStamping           OBJECT IDENTIFIER ::= { id-kp 8 }
+id-kp-OCSPSigning            OBJECT IDENTIFIER ::= { id-kp 9 }
+
+-- inhibit any policy OID and syntax
+
+id-ce-inhibitAnyPolicy OBJECT IDENTIFIER ::=  { id-ce 54 }
+
+InhibitAnyPolicy ::= SkipCerts
+
+-- freshest (delta)CRL extension OID and syntax
+
+id-ce-freshestCRL OBJECT IDENTIFIER ::=  { id-ce 46 }
+
+FreshestCRL ::= CRLDistributionPoints
+
+-- authority info access
+
+id-pe-authorityInfoAccess OBJECT IDENTIFIER ::= { id-pe 1 }
+
+AuthorityInfoAccessSyntax  ::=
+        SEQUENCE SIZE (1..MAX) OF AccessDescription
+
+AccessDescription  ::=  SEQUENCE {
+        accessMethod          OBJECT IDENTIFIER,
+        accessLocation        GeneralName  }
+
+-- subject info access
+
+id-pe-subjectInfoAccess OBJECT IDENTIFIER ::= { id-pe 11 }
+
+SubjectInfoAccessSyntax  ::=
+        SEQUENCE SIZE (1..MAX) OF AccessDescription
+
+-- CRL number extension OID and syntax
+
+id-ce-cRLNumber OBJECT IDENTIFIER ::= { id-ce 20 }
+
+CRLNumber ::= INTEGER (0..MAX)
+
+-- issuing distribution point extension OID and syntax
+
+id-ce-issuingDistributionPoint OBJECT IDENTIFIER ::= { id-ce 28 }
+
+IssuingDistributionPoint ::= SEQUENCE {
+     distributionPoint          [0] DistributionPointName OPTIONAL,
+     onlyContainsUserCerts      [1] BOOLEAN DEFAULT FALSE,
+     onlyContainsCACerts        [2] BOOLEAN DEFAULT FALSE,
+     onlySomeReasons            [3] ReasonFlags OPTIONAL,
+     indirectCRL                [4] BOOLEAN DEFAULT FALSE,
+     onlyContainsAttributeCerts [5] BOOLEAN DEFAULT FALSE }
+     -- at most one of onlyContainsUserCerts, onlyContainsCACerts,
+     -- and onlyContainsAttributeCerts may be set to TRUE.
+
+id-ce-deltaCRLIndicator OBJECT IDENTIFIER ::= { id-ce 27 }
+
+BaseCRLNumber ::= CRLNumber
+
+-- reason code extension OID and syntax
+
+id-ce-cRLReasons OBJECT IDENTIFIER ::= { id-ce 21 }
+
+CRLReason ::= ENUMERATED {
+     unspecified             (0),
+     keyCompromise           (1),
+     cACompromise            (2),
+     affiliationChanged      (3),
+     superseded              (4),
+     cessationOfOperation    (5),
+     certificateHold         (6),
+     removeFromCRL           (8),
+     privilegeWithdrawn      (9),
+     aACompromise           (10) }
+
+-- certificate issuer CRL entry extension OID and syntax
+
+id-ce-certificateIssuer OBJECT IDENTIFIER ::= { id-ce 29 }
+
+CertificateIssuer ::= GeneralNames
+
+-- hold instruction extension OID and syntax
+
+id-ce-holdInstructionCode OBJECT IDENTIFIER ::= { id-ce 23 }
+
+HoldInstructionCode ::= OBJECT IDENTIFIER
+
+-- ANSI x9 arc holdinstruction arc
+
+holdInstruction OBJECT IDENTIFIER ::=
+          {joint-iso-itu-t(2) member-body(2) us(840) x9cm(10040) 2}
+
+-- ANSI X9 holdinstructions
+
+id-holdinstruction-none OBJECT IDENTIFIER  ::=
+                                      {holdInstruction 1} -- deprecated
+
+id-holdinstruction-callissuer OBJECT IDENTIFIER ::= {holdInstruction 2}
+
+id-holdinstruction-reject OBJECT IDENTIFIER ::= {holdInstruction 3}
+
+-- invalidity date CRL entry extension OID and syntax
+
+id-ce-invalidityDate OBJECT IDENTIFIER ::= { id-ce 24 }
+
+InvalidityDate ::=  GeneralizedTime
+
+END
+

pySim/esim/asn1/rsp/rsp.asn

@@ -0,0 +1,785 @@
+RSPDefinitions {joint-iso-itu-t(2) international-organizations(23) gsma(146) rsp(1) spec-version(1) version-two(2)}
+DEFINITIONS
+AUTOMATIC TAGS
+EXTENSIBILITY IMPLIED ::=
+BEGIN
+ 
+IMPORTS Certificate, CertificateList, Time FROM PKIX1Explicit88 {iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-explicit(18)}
+SubjectKeyIdentifier FROM PKIX1Implicit88 {iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-implicit(19)};
+ 
+id-rsp OBJECT IDENTIFIER ::= {joint-iso-itu-t(2) international-organizations(23) gsma(146) rsp(1)}
+ 
+-- Basic types, for size constraints
+Octet8 ::= OCTET STRING (SIZE(8))
+Octet16 ::= OCTET STRING (SIZE(16))
+OctetTo16 ::= OCTET STRING (SIZE(1..16))
+Octet32 ::= OCTET STRING (SIZE(32))
+Octet1 ::= OCTET STRING(SIZE(1))
+Octet2 ::= OCTET STRING (SIZE(2))
+VersionType ::= OCTET STRING(SIZE(3)) -- major/minor/revision version are coded as binary value on byte 1/2/3, e.g. '02 00 0C' for v2.0.12.
+Iccid ::= [APPLICATION 26] OCTET STRING (SIZE(10)) -- ICCID as coded in EFiccid, corresponding tag is '5A'
+RemoteOpId ::= [2] INTEGER {installBoundProfilePackage(1)}
+TransactionId ::= OCTET STRING (SIZE(1..16))
+ 
+-- Definition of EUICCInfo1 --------------------------
+GetEuiccInfo1Request ::= [32] SEQUENCE { -- Tag 'BF20'
+}
+ 
+EUICCInfo1 ::= [32] SEQUENCE { -- Tag 'BF20'
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    euiccCiPKIdListForVerification [9] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifiers supported on the eUICC for signature verification
+    euiccCiPKIdListForSigning [10] SEQUENCE OF SubjectKeyIdentifier -- List of CI Public Key Identifier supported on the eUICC for signature creation
+}
+ 
+-- Definition of EUICCInfo2 --------------------------
+GetEuiccInfo2Request ::= [34] SEQUENCE { -- Tag 'BF22'
+}
+ 
+EUICCInfo2 ::= [34] SEQUENCE { -- Tag 'BF22'
+    profileVersion [1] VersionType,     -- SIMAlliance Profile package version supported
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    euiccFirmwareVer [3] VersionType,   -- eUICC Firmware version
+    extCardResource [4] OCTET STRING,   -- Extended Card Resource Information according to ETSI TS 102 226
+    uiccCapability [5] UICCCapability,
+    javacardVersion [6] VersionType OPTIONAL,
+    globalplatformVersion [7] VersionType OPTIONAL,
+    rspCapability [8] RspCapability,
+    euiccCiPKIdListForVerification [9] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifiers supported on the eUICC for signature verification
+    euiccCiPKIdListForSigning [10] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifier supported on the eUICC for signature creation
+    euiccCategory [11] INTEGER {
+        other(0),
+        basicEuicc(1),
+        mediumEuicc(2),
+        contactlessEuicc(3)
+    } OPTIONAL,
+    forbiddenProfilePolicyRules [25] PprIds OPTIONAL, -- Tag '99'
+    ppVersion VersionType, -- Protection Profile version
+    sasAcreditationNumber UTF8String (SIZE(0..64)),
+    certificationDataObject [12] CertificationDataObject OPTIONAL
+}
+ 
+-- Definition of RspCapability
+RspCapability ::= BIT STRING {
+    additionalProfile(0), -- at least one more Profile can be installed
+    crlSupport(1), -- CRL
+    rpmSupport(2), -- Remote Profile Management
+    testProfileSupport (3) -- support for test profile
+}
+ 
+-- Definition of CertificationDataObject
+CertificationDataObject ::= SEQUENCE {
+    platformLabel UTF8String,   -- Platform_Label as defined in GlobalPlatform DLOA specification [57]
+    discoveryBaseURL UTF8String -- Discovery Base URL of the SE default DLOA Registrar as defined in GlobalPlatform DLOA specification [57]
+}
+ 
+CertificateInfo ::= BIT STRING {
+ 
+    reserved(0),    -- eUICC has a CERT.EUICC.ECDSA in GlobalPlatform format. The use of this bit is deprecated.
+    certSigningX509(1),     -- eUICC has a CERT.EUICC.ECDSA in X.509 format
+    rfu2(2),
+    rfu3(3),
+    reserved2(4), -- Handling of Certificate in GlobalPlatform format. The use of this bit is deprecated.
+    certVerificationX509(5)-- Handling of Certificate in X.509 format
+}
+ 
+-- Definition of UICCCapability
+UICCCapability ::= BIT STRING {
+/* Sequence is derived from ServicesList[] defined in SIMalliance PEDefinitions*/
+    contactlessSupport(0),  -- Contactless (SWP, HCI and associated APIs)
+    usimSupport(1),         -- USIM as defined by 3GPP
+    isimSupport(2),         -- ISIM as defined by 3GPP
+    csimSupport(3),         -- CSIM as defined by 3GPP2
+ 
+    akaMilenage(4),         -- Milenage as AKA algorithm
+    akaCave(5),             -- CAVE as authentication algorithm
+    akaTuak128(6),          -- TUAK as AKA algorithm with 128 bit key length
+    akaTuak256(7),          -- TUAK as AKA algorithm with 256 bit key length
+    rfu1(8),                    -- reserved for further algorithms
+    rfu2(9),                    -- reserved for further algorithms
+ 
+    gbaAuthenUsim(10),  -- GBA authentication in the context of USIM
+    gbaAuthenISim(11),  -- GBA authentication in the context of ISIM
+    mbmsAuthenUsim(12),     -- MBMS authentication in the context of USIM
+    eapClient(13),          -- EAP client
+ 
+    javacard(14),               -- Javacard support
+    multos(15),             -- Multos support
+ 
+    multipleUsimSupport(16),     -- Multiple USIM applications are supported within the same Profile
+    multipleIsimSupport(17),      -- Multiple ISIM applications are supported within the same Profile
+    multipleCsimSupport(18)   -- Multiple CSIM applications are supported within the same Profile
+}
+ 
+-- Definition of DeviceInfo
+DeviceInfo ::= SEQUENCE {
+    tac Octet8,
+    deviceCapabilities DeviceCapabilities,
+    imei Octet8 OPTIONAL
+}
+ 
+DeviceCapabilities ::= SEQUENCE { -- Highest fully supported release for each definition
+  -- The device SHALL set all the capabilities it supports
+    gsmSupportedRelease VersionType OPTIONAL,
+    utranSupportedRelease VersionType OPTIONAL,
+    cdma2000onexSupportedRelease VersionType OPTIONAL,
+    cdma2000hrpdSupportedRelease VersionType OPTIONAL,
+    cdma2000ehrpdSupportedRelease VersionType OPTIONAL,
+    eutranSupportedRelease VersionType OPTIONAL,
+    contactlessSupportedRelease VersionType OPTIONAL,
+    rspCrlSupportedVersion VersionType OPTIONAL,
+    rspRpmSupportedVersion VersionType OPTIONAL
+}
+ 
+ProfileInfoListRequest ::= [45] SEQUENCE { -- Tag 'BF2D'
+    searchCriteria [0] CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID of the ISD-P, tag '4F'
+        iccid Iccid, -- ICCID, tag '5A'
+        profileClass [21] ProfileClass -- Tag '95'
+    } OPTIONAL,
+    tagList [APPLICATION 28] OCTET STRING OPTIONAL -- tag '5C'
+}
+ 
+-- Definition of ProfileInfoList
+ProfileInfoListResponse ::= [45] CHOICE { -- Tag 'BF2D'
+    profileInfoListOk SEQUENCE OF ProfileInfo,
+    profileInfoListError ProfileInfoListError
+}
+ 
+ProfileInfo ::= [PRIVATE 3] SEQUENCE { -- Tag 'E3'
+    iccid Iccid OPTIONAL,
+    isdpAid [APPLICATION 15] OctetTo16 OPTIONAL, -- AID of the ISD-P containing the Profile, tag '4F'
+    profileState [112] ProfileState OPTIONAL, -- Tag '9F70'
+    profileNickname [16] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '90'
+    serviceProviderName [17] UTF8String (SIZE(0..32)) OPTIONAL, -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '92'
+    iconType [19] IconType OPTIONAL, -- Tag '93'
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94', see condition in ES10c:GetProfilesInfo
+    profileClass [21] ProfileClass DEFAULT operational, -- Tag '95'
+    notificationConfigurationInfo [22] SEQUENCE OF NotificationConfigurationInformation OPTIONAL, -- Tag 'B6'
+    profileOwner [23] OperatorID OPTIONAL, -- Tag 'B7'
+    dpProprietaryData [24] DpProprietaryData OPTIONAL, -- Tag 'B8'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+PprIds ::= BIT STRING {-- Definition of Profile Policy Rules identifiers
+    pprUpdateControl(0), -- defines how to update PPRs via ES6
+    ppr1(1), -- Indicator for PPR1 'Disabling of this Profile is not allowed'
+    ppr2(2), -- Indicator for PPR2 'Deletion of this Profile is not allowed'
+    ppr3(3)  -- Indicator for PPR3 'Deletion of this Profile is required upon its successful disabling'
+}
+ 
+OperatorID ::= SEQUENCE {
+    mccMnc OCTET STRING (SIZE(3)), -- MCC and MNC coded as defined in 3GPP TS 24.008 [32]
+    gid1 OCTET STRING OPTIONAL, -- referring to content of EF GID1 (file identifier '6F3E') as defined in 3GPP TS 31.102 [54]
+    gid2 OCTET STRING OPTIONAL -- referring to content of EF GID2 (file identifier '6F3F') as defined in 3GPP TS 31.102 [54]
+}
+ 
+ProfileInfoListError ::= INTEGER {incorrectInputValues(1), undefinedError(127)}
+ 
+-- Definition of StoreMetadata request
+ 
+StoreMetadataRequest ::= [37] SEQUENCE { -- Tag 'BF25'
+    iccid Iccid,
+    serviceProviderName [17] UTF8String (SIZE(0..32)), -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)), -- Tag '92' (corresponds to 'Short Description' defined in SGP.21 [2])
+    iconType [19] IconType OPTIONAL, -- Tag '93' (JPG or PNG)
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94'(Data of the icon. Size 64 x 64 pixel. This field SHALL only be present if iconType is present)
+    profileClass [21] ProfileClass OPTIONAL, -- Tag '95' (default if absent: 'operational')
+    notificationConfigurationInfo [22] SEQUENCE OF NotificationConfigurationInformation OPTIONAL,
+    profileOwner [23] OperatorID OPTIONAL, -- Tag 'B7'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+NotificationEvent ::= BIT STRING {
+    notificationInstall (0),
+    notificationEnable(1),
+    notificationDisable(2),
+    notificationDelete(3)
+}
+ 
+NotificationConfigurationInformation ::= SEQUENCE {
+    profileManagementOperation NotificationEvent,
+    notificationAddress UTF8String -- FQDN to forward the notification
+}
+ 
+IconType ::= INTEGER {jpg(0), png(1)}
+ProfileState ::= INTEGER {disabled(0), enabled(1)}
+ProfileClass ::= INTEGER {test(0), provisioning(1), operational(2)}
+ 
+-- Definition of UpdateMetadata request
+UpdateMetadataRequest ::= [42] SEQUENCE {  -- Tag 'BF2A'
+    serviceProviderName [17] UTF8String (SIZE(0..32)) OPTIONAL, -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '92'
+    iconType [19] IconType OPTIONAL, -- Tag '93'
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+-- Definition of data objects for command PrepareDownload -------------------------
+PrepareDownloadRequest ::= [33] SEQUENCE {  -- Tag 'BF21'
+    smdpSigned2 SmdpSigned2,            -- Signed information
+    smdpSignature2 [APPLICATION 55] OCTET STRING,   -- DP_Sign1, tag '5F37'
+    hashCc Octet32 OPTIONAL, -- Hash of confirmation code
+    smdpCertificate Certificate -- CERT.DPpb.ECDSA
+}
+ 
+SmdpSigned2 ::= SEQUENCE {
+    transactionId [0] TransactionId,    -- The TransactionID generated by the SM DP+
+    ccRequiredFlag BOOLEAN, --Indicates if the Confirmation Code is required
+    bppEuiccOtpk [APPLICATION 73] OCTET STRING OPTIONAL     -- otPK.EUICC.ECKA already used for binding the BPP, tag '5F49'
+}
+ 
+PrepareDownloadResponse ::= [33] CHOICE {  -- Tag 'BF21'
+    downloadResponseOk PrepareDownloadResponseOk,
+    downloadResponseError PrepareDownloadResponseError
+}
+ 
+PrepareDownloadResponseOk ::= SEQUENCE {
+    euiccSigned2 EUICCSigned2,  -- Signed information
+    euiccSignature2 [APPLICATION 55] OCTET STRING   -- tag '5F37'
+}
+ 
+EUICCSigned2 ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    euiccOtpk [APPLICATION 73] OCTET STRING,        -- otPK.EUICC.ECKA, tag '5F49'
+    hashCc Octet32 OPTIONAL         -- Hash of confirmation code
+}
+ 
+PrepareDownloadResponseError ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    downloadErrorCode DownloadErrorCode
+}
+ 
+DownloadErrorCode ::= INTEGER {invalidCertificate(1), invalidSignature(2), unsupportedCurve(3), noSessionContext(4), invalidTransactionId(5), undefinedError(127)}
+ 
+-- Definition of data objects for command AuthenticateServer--------------------
+AuthenticateServerRequest ::= [56] SEQUENCE { -- Tag 'BF38'
+    serverSigned1 ServerSigned1,                -- Signed information
+    serverSignature1 [APPLICATION 55] OCTET STRING,     -- tag ?5F37?
+    euiccCiPKIdToBeUsed SubjectKeyIdentifier,       -- CI Public Key Identifier to be used
+    serverCertificate Certificate, -- RSP Server Certificate CERT.XXauth.ECDSA
+    ctxParams1 CtxParams1
+}
+ 
+ServerSigned1 ::= SEQUENCE {
+    transactionId [0] TransactionId,        -- The Transaction ID generated by the RSP Server
+    euiccChallenge [1] Octet16,     -- The eUICC Challenge
+    serverAddress [3] UTF8String,   -- The RSP Server address
+    serverChallenge [4] Octet16     -- The RSP Server Challenge
+}
+ 
+CtxParams1 ::= CHOICE {
+    ctxParamsForCommonAuthentication CtxParamsForCommonAuthentication -- New contextual data objects may be defined for extensibility
+}
+ 
+CtxParamsForCommonAuthentication ::= SEQUENCE {
+    matchingId UTF8String OPTIONAL,-- The MatchingId could be the Activation code token or EventID or empty
+    deviceInfo DeviceInfo -- The Device information
+}
+ 
+AuthenticateServerResponse ::= [56] CHOICE { -- Tag 'BF38'
+    authenticateResponseOk AuthenticateResponseOk,
+    authenticateResponseError AuthenticateResponseError
+}
+ 
+AuthenticateResponseOk ::= SEQUENCE {
+    euiccSigned1 EuiccSigned1,      -- Signed information
+    euiccSignature1 [APPLICATION 55] OCTET STRING,  --EUICC_Sign1, tag 5F37
+    euiccCertificate Certificate,   -- eUICC Certificate (CERT.EUICC.ECDSA) signed by the EUM
+    eumCertificate Certificate  -- EUM Certificate (CERT.EUM.ECDSA) signed by the requested CI
+}
+ 
+EuiccSigned1 ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    serverAddress [3] UTF8String,
+    serverChallenge [4] Octet16,    -- The RSP Server Challenge
+    euiccInfo2 [34] EUICCInfo2,
+    ctxParams1 CtxParams1
+}
+ 
+AuthenticateResponseError ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    authenticateErrorCode AuthenticateErrorCode
+}
+ 
+AuthenticateErrorCode ::= INTEGER {invalidCertificate(1), invalidSignature(2), unsupportedCurve(3), noSessionContext(4), invalidOid(5), euiccChallengeMismatch(6), ciPKUnknown(7), undefinedError(127)}
+ 
+-- Definition of Cancel Session------------------------------
+CancelSessionRequest ::= [65] SEQUENCE { -- Tag 'BF41'
+    transactionId TransactionId,     -- The TransactionID generated by the RSP Server
+    reason CancelSessionReason
+}
+ 
+CancelSessionReason ::= INTEGER {endUserRejection(0), postponed(1), timeout(2), pprNotAllowed(3)}
+ 
+CancelSessionResponse ::= [65] CHOICE { -- Tag 'BF41'
+    cancelSessionResponseOk CancelSessionResponseOk,
+    cancelSessionResponseError INTEGER {invalidTransactionId(5), undefinedError(127)}
+}
+ 
+CancelSessionResponseOk ::= SEQUENCE {
+    euiccCancelSessionSigned EuiccCancelSessionSigned,  -- Signed information
+    euiccCancelSessionSignature [APPLICATION 55] OCTET STRING   -- tag '5F37
+}
+ 
+EuiccCancelSessionSigned ::= SEQUENCE {
+    transactionId TransactionId,
+    smdpOid OBJECT IDENTIFIER, -- SM-DP+ OID as contained in CERT.DPauth.ECDSA
+    reason CancelSessionReason
+}
+ 
+-- Definition of Bound Profile Package --------------------------
+BoundProfilePackage ::= [54] SEQUENCE { -- Tag 'BF36'
+    initialiseSecureChannelRequest [35] InitialiseSecureChannelRequest, -- Tag 'BF23'
+    firstSequenceOf87 [0] SEQUENCE OF [7] OCTET STRING, -- sequence of '87' TLVs
+    sequenceOf88 [1] SEQUENCE OF [8] OCTET STRING, -- sequence of '88' TLVs
+    secondSequenceOf87 [2] SEQUENCE OF [7] OCTET STRING OPTIONAL, -- sequence of '87' TLVs
+    sequenceOf86 [3] SEQUENCE OF [6] OCTET STRING -- sequence of '86' TLVs
+}
+ 
+-- Definition of Get eUICC Challenge --------------------------
+GetEuiccChallengeRequest ::= [46] SEQUENCE { -- Tag 'BF2E'
+}
+ 
+GetEuiccChallengeResponse ::= [46] SEQUENCE { -- Tag 'BF2E'
+    euiccChallenge Octet16  -- random eUICC challenge
+}
+ 
+-- Definition of Profile Installation Resulceipt
+ProfileInstallationResult ::= [55] SEQUENCE { -- Tag 'BF37'
+    profileInstallationResultData [39] ProfileInstallationResultData,
+    euiccSignPIR EuiccSignPIR
+}
+ 
+ProfileInstallationResultData ::= [39] SEQUENCE { -- Tag 'BF27'
+    transactionId[0] TransactionId, -- The TransactionID generated by the SM-DP+
+    notificationMetadata[47] NotificationMetadata,
+    smdpOid OBJECT IDENTIFIER OPTIONAL, -- SM-DP+ OID (same value as in CERT.DPpb.ECDSA)
+    finalResult [2] CHOICE {
+        successResult SuccessResult,
+        errorResult ErrorResult
+    }
+}
+ 
+EuiccSignPIR ::= [APPLICATION 55] OCTET STRING -- Tag '5F37', eUICC?s signature
+ 
+SuccessResult ::= SEQUENCE {
+    aid [APPLICATION 15] OCTET STRING (SIZE (5..16)), -- AID of ISD-P
+    simaResponse OCTET STRING -- contains (multiple) 'EUICCResponse' as defined in [5]
+}
+ 
+ErrorResult ::= SEQUENCE {
+    bppCommandId BppCommandId,
+    errorReason ErrorReason,
+    simaResponse OCTET STRING OPTIONAL -- contains (multiple) 'EUICCResponse' as defined in [5]
+}
+ 
+BppCommandId ::= INTEGER {initialiseSecureChannel(0), configureISDP(1), storeMetadata(2), storeMetadata2(3), replaceSessionKeys(4), loadProfileElements(5)}
+ 
+ErrorReason ::= INTEGER {
+    incorrectInputValues(1),
+    invalidSignature(2),
+    invalidTransactionId(3),
+    unsupportedCrtValues(4),
+    unsupportedRemoteOperationType(5),
+    unsupportedProfileClass(6),
+    scp03tStructureError(7),
+    scp03tSecurityError(8),
+    installFailedDueToIccidAlreadyExistsOnEuicc(9), installFailedDueToInsufficientMemoryForProfile(10),
+    installFailedDueToInterruption(11),
+    installFailedDueToPEProcessingError (12),
+    installFailedDueToIccidMismatch(13),
+    testProfileInstallFailedDueToInvalidNaaKey(14),
+    pprNotAllowed(15),
+    installFailedDueToUnknownError(127)
+}
+ 
+ListNotificationRequest ::= [40] SEQUENCE { -- Tag 'BF28'
+    profileManagementOperation [1] NotificationEvent OPTIONAL
+}
+ 
+ListNotificationResponse ::= [40] CHOICE { -- Tag 'BF28'
+    notificationMetadataList SEQUENCE OF NotificationMetadata,
+    listNotificationsResultError INTEGER {undefinedError(127)}
+}
+ 
+NotificationMetadata ::= [47] SEQUENCE { -- Tag 'BF2F'
+    seqNumber [0] INTEGER,
+    profileManagementOperation [1] NotificationEvent, --Only one bit set to 1
+    notificationAddress UTF8String, -- FQDN to forward the notification
+    iccid Iccid OPTIONAL
+}
+ 
+-- Definition of Profile Nickname Information
+SetNicknameRequest ::= [41] SEQUENCE {  -- Tag 'BF29'
+    iccid Iccid,
+    profileNickname [16] UTF8String (SIZE(0..64))
+}
+ 
+SetNicknameResponse ::= [41] SEQUENCE { -- Tag 'BF29'
+    setNicknameResult INTEGER {ok(0), iccidNotFound (1), undefinedError(127)}
+}
+ 
+id-rsp-cert-objects OBJECT IDENTIFIER ::= {  id-rsp cert-objects(2)}
+ 
+id-rspExt OBJECT IDENTIFIER ::= {id-rsp-cert-objects 0}
+ 
+id-rspRole OBJECT IDENTIFIER ::= {id-rsp-cert-objects 1}
+ 
+-- Definition of OIDs for role identification
+id-rspRole-ci OBJECT IDENTIFIER ::= {id-rspRole 0}
+id-rspRole-euicc OBJECT IDENTIFIER ::= {id-rspRole 1}
+id-rspRole-eum OBJECT IDENTIFIER ::= {id-rspRole 2}
+id-rspRole-dp-tls OBJECT IDENTIFIER ::= {id-rspRole 3}
+id-rspRole-dp-auth OBJECT IDENTIFIER ::= {id-rspRole 4}
+id-rspRole-dp-pb OBJECT IDENTIFIER ::= {id-rspRole 5}
+id-rspRole-ds-tls OBJECT IDENTIFIER ::= {id-rspRole 6}
+id-rspRole-ds-auth OBJECT IDENTIFIER ::= {id-rspRole 7}
+ 
+--Definition of data objects for InitialiseSecureChannel Request
+InitialiseSecureChannelRequest ::= [35] SEQUENCE { -- Tag 'BF23'
+    remoteOpId RemoteOpId, -- Remote Operation Type Identifier (value SHALL be set to installBoundProfilePackage)
+    transactionId [0] TransactionId, -- The TransactionID generated by the SM-DP+
+    controlRefTemplate[6] IMPLICIT ControlRefTemplate, -- Control Reference Template (Key Agreement). Current specification considers a subset of CRT specified in GlobalPlatform Card Specification [8], section 6.4.2.3 for the Mutual Authentication Data Field
+    smdpOtpk [APPLICATION 73] OCTET STRING, ---otPK.DP.ECKA as specified in GlobalPlatform Card Specification [8] section 6.4.2.3 for ePK.OCE.ECKA, tag '5F49'
+    smdpSign [APPLICATION 55] OCTET STRING -- SM-DP's signature, tag '5F37'
+}
+ 
+ControlRefTemplate ::= SEQUENCE {
+keyType[0] Octet1, -- Key type according to GlobalPlatform Card Specification [8] Table 11-16, AES= '88', Tag '80'
+keyLen[1] Octet1, --Key length in number of bytes. For current specification key length SHALL by 0x10 bytes, Tag '81'
+hostId[4] OctetTo16 -- Host ID value , Tag '84'
+}
+ 
+--Definition of data objects for ConfigureISDPRequest
+ConfigureISDPRequest ::= [36] SEQUENCE { -- Tag 'BF24'
+    dpProprietaryData [24] DpProprietaryData OPTIONAL -- Tag 'B8'
+}
+ 
+DpProprietaryData ::= SEQUENCE { --  maximum size including tag and length field: 128 bytes
+    dpOid OBJECT IDENTIFIER -- OID in the tree of the SM-DP+ that created the Profile
+    -- additional data objects defined by the SM-DP+ MAY follow
+}
+ 
+-- Definition of request message for command ReplaceSessionKeys
+ReplaceSessionKeysRequest ::= [38] SEQUENCE { -- tag 'BF26'
+/*The new initial MAC chaining value*/
+    initialMacChainingValue OCTET STRING,
+/*New session key value for encryption/decryption (PPK-ENC)*/
+    ppkEnc OCTET STRING,
+/*New session key value of the session key C-MAC computation/verification (PPK-MAC)*/
+    ppkCmac OCTET STRING
+}
+ 
+-- Definition of data objects for RetrieveNotificationsList
+RetrieveNotificationsListRequest ::= [43] SEQUENCE { -- Tag 'BF2B'
+    searchCriteria CHOICE {
+        seqNumber [0] INTEGER,
+        profileManagementOperation [1] NotificationEvent
+    } OPTIONAL
+}
+ 
+RetrieveNotificationsListResponse ::= [43] CHOICE { -- Tag 'BF2B'
+    notificationList SEQUENCE OF PendingNotification,
+    notificationsListResultError INTEGER {noResultAvailable(1), undefinedError(127)}
+}
+ 
+PendingNotification ::= CHOICE {
+    profileInstallationResult [55] ProfileInstallationResult, -- tag 'BF37'
+    otherSignedNotification OtherSignedNotification
+}
+ 
+OtherSignedNotification ::= SEQUENCE {
+    tbsOtherNotification NotificationMetadata,
+    euiccNotificationSignature [APPLICATION 55] OCTET STRING,   -- eUICC signature of tbsOtherNotification, Tag '5F37'
+    euiccCertificate Certificate,   -- eUICC Certificate (CERT.EUICC.ECDSA) signed by the EUM
+    eumCertificate Certificate  -- EUM Certificate (CERT.EUM.ECDSA) signed by the requested CI
+}
+ 
+-- Definition of notificationSent
+NotificationSentRequest ::= [48] SEQUENCE { -- Tag 'BF30'
+    seqNumber [0] INTEGER
+}
+ 
+NotificationSentResponse ::= [48] SEQUENCE { -- Tag 'BF30'
+    deleteNotificationStatus INTEGER {ok(0), nothingToDelete(1), undefinedError(127)}
+}
+ 
+-- Definition of Enable Profile --------------------------
+EnableProfileRequest ::= [49] SEQUENCE { -- Tag 'BF31'
+    profileIdentifier CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+        iccid Iccid -- ICCID, tag '5A'
+    },
+    refreshFlag BOOLEAN -- indicating whether REFRESH is required
+}
+ 
+EnableProfileResponse ::= [49] SEQUENCE { -- Tag 'BF31'
+    enableResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInDisabledState(2), disallowedByPolicy(3), wrongProfileReenabling(4), undefinedError(127)}
+}
+ 
+-- Definition of Disable Profile --------------------------
+DisableProfileRequest ::= [50] SEQUENCE { -- Tag 'BF32'
+    profileIdentifier CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+        iccid Iccid -- ICCID, tag '5A'
+    },
+    refreshFlag BOOLEAN -- indicating whether REFRESH is required
+}
+ 
+DisableProfileResponse ::= [50] SEQUENCE { -- Tag 'BF32'
+    disableResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInEnabledState(2), disallowedByPolicy(3), undefinedError(127)}
+}
+ 
+-- Definition of Delete Profile --------------------------
+DeleteProfileRequest ::= [51] CHOICE { -- Tag 'BF33'
+    isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+    iccid Iccid -- ICCID, tag '5A'
+}
+ 
+DeleteProfileResponse ::= [51] SEQUENCE { -- Tag 'BF33'
+    deleteResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInDisabledState(2), disallowedByPolicy(3), undefinedError(127)}
+}
+ 
+-- Definition of Memory Reset --------------------------
+EuiccMemoryResetRequest ::= [52] SEQUENCE { -- Tag 'BF34'
+    resetOptions [2] BIT STRING {
+        deleteOperationalProfiles(0),
+        deleteFieldLoadedTestProfiles(1),
+        resetDefaultSmdpAddress(2)}
+}
+ 
+EuiccMemoryResetResponse ::= [52] SEQUENCE { -- Tag 'BF34'
+    resetResult INTEGER {ok(0), nothingToDelete(1), undefinedError(127)}
+}
+ 
+-- Definition of Get EID --------------------------
+GetEuiccDataRequest ::= [62] SEQUENCE { -- Tag 'BF3E'
+    tagList [APPLICATION 28] Octet1  -- tag '5C', the value SHALL be set to '5A'
+}
+ 
+GetEuiccDataResponse ::= [62] SEQUENCE { -- Tag 'BF3E'
+    eidValue [APPLICATION 26] Octet16  -- tag '5A'
+}
+ 
+-- Definition of Get Rat
+ 
+GetRatRequest ::= [67] SEQUENCE { -- Tag ' BF43'
+    -- No input data
+}
+ 
+ 
+GetRatResponse ::= [67] SEQUENCE { -- Tag 'BF43'
+    rat RulesAuthorisationTable
+}
+ 
+RulesAuthorisationTable ::= SEQUENCE OF ProfilePolicyAuthorisationRule
+ProfilePolicyAuthorisationRule ::= SEQUENCE {
+    pprIds PprIds,
+    allowedOperators SEQUENCE OF OperatorID,
+    pprFlags BIT STRING {consentRequired(0)}
+}
+ 
+-- Definition of data structure command for loading a CRL
+LoadCRLRequest ::= [53] SEQUENCE { -- Tag 'BF35'
+    -- A CRL-A
+    crl CertificateList
+}
+ 
+-- Definition of data structure response for loading a CRL
+LoadCRLResponse ::= [53] CHOICE {  -- Tag 'BF35'
+loadCRLResponseOk LoadCRLResponseOk,
+loadCRLResponseError LoadCRLResponseError
+}
+ 
+LoadCRLResponseOk ::= SEQUENCE {
+    missingParts SEQUENCE OF SEQUENCE {
+        number INTEGER (0..MAX)
+    } OPTIONAL
+}
+LoadCRLResponseError ::= INTEGER {invalidSignature(1), invalidCRLFormat(2), notEnoughMemorySpace(3), verificationKeyNotFound(4), undefinedError(127)}
+ 
+-- Definition of the extension for Certificate Expiration Date
+id-rsp-expDate OBJECT IDENTIFIER ::= {id-rspExt 1}
+ExpirationDate ::= Time
+ 
+-- Definition of the extension id for total partial-CRL number
+id-rsp-totalPartialCrlNumber OBJECT IDENTIFIER ::= {id-rspExt 2}
+TotalPartialCrlNumber ::= INTEGER
+ 
+ 
+-- Definition of the extension id for the partial-CRL number
+id-rsp-partialCrlNumber OBJECT IDENTIFIER ::= {id-rspExt 3}
+PartialCrlNumber ::= INTEGER
+ 
+-- Definition for ES9+ ASN.1 Binding --------------------------
+RemoteProfileProvisioningRequest ::= [2] CHOICE { -- Tag 'A2'
+    initiateAuthenticationRequest [57] InitiateAuthenticationRequest,  -- Tag 'BF39'
+    authenticateClientRequest [59] AuthenticateClientRequest, -- Tag 'BF3B'
+    getBoundProfilePackageRequest [58] GetBoundProfilePackageRequest,  -- Tag 'BF3A'
+    cancelSessionRequestEs9 [65] CancelSessionRequestEs9, -- Tag 'BF41'
+    handleNotification [61] HandleNotification -- tag 'BF3D'
+}
+ 
+RemoteProfileProvisioningResponse ::= [2] CHOICE { -- Tag 'A2'
+    initiateAuthenticationResponse [57] InitiateAuthenticationResponse, -- Tag 'BF39'
+    authenticateClientResponseEs9 [59] AuthenticateClientResponseEs9, -- Tag 'BF3B'
+    getBoundProfilePackageResponse [58] GetBoundProfilePackageResponse, -- Tag 'BF3A'
+    cancelSessionResponseEs9 [65] CancelSessionResponseEs9, -- Tag 'BF41'
+    authenticateClientResponseEs11 [64] AuthenticateClientResponseEs11 -- Tag 'BF40'
+}
+ 
+InitiateAuthenticationRequest ::= [57] SEQUENCE { -- Tag 'BF39'
+    euiccChallenge [1] Octet16, -- random eUICC challenge
+    smdpAddress [3] UTF8String,
+    euiccInfo1 EUICCInfo1
+}
+ 
+InitiateAuthenticationResponse ::= [57] CHOICE { -- Tag 'BF39'
+    initiateAuthenticationOk InitiateAuthenticationOkEs9,
+    initiateAuthenticationError INTEGER {
+        invalidDpAddress(1),
+        euiccVersionNotSupportedByDp(2),
+        ciPKNotSupported(3)
+    }
+}
+ 
+InitiateAuthenticationOkEs9 ::= SEQUENCE {
+    transactionId [0] TransactionId, -- The TransactionID generated by the SM-DP+
+    serverSigned1 ServerSigned1, -- Signed information
+    serverSignature1 [APPLICATION 55] OCTET STRING, -- Server_Sign1, tag '5F37'
+    euiccCiPKIdToBeUsed SubjectKeyIdentifier, -- The curve CI Public Key to be used as required by ES10b.AuthenticateServer
+    serverCertificate Certificate
+}
+ 
+AuthenticateClientRequest ::= [59] SEQUENCE { -- Tag 'BF3B'
+    transactionId [0] TransactionId,
+    authenticateServerResponse [56] AuthenticateServerResponse -- This is the response from ES10b.AuthenticateServer
+}
+ 
+AuthenticateClientResponseEs9 ::= [59] CHOICE { -- Tag 'BF3B'
+    authenticateClientOk AuthenticateClientOk,
+    authenticateClientError INTEGER {
+        eumCertificateInvalid(1),
+        eumCertificateExpired(2),
+        euiccCertificateInvalid(3),
+        euiccCertificateExpired(4),
+        euiccSignatureInvalid(5),
+        matchingIdRefused(6),
+        eidMismatch(7),
+        noEligibleProfile(8),
+        ciPKUnknown(9),
+        invalidTransactionId(10),
+        undefinedError(127)
+    }
+}
+ 
+AuthenticateClientOk ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    profileMetaData [37] StoreMetadataRequest,
+    prepareDownloadRequest [33] PrepareDownloadRequest
+}
+ 
+GetBoundProfilePackageRequest ::= [58] SEQUENCE { -- Tag 'BF3A'
+    transactionId [0] TransactionId,
+    prepareDownloadResponse [33] PrepareDownloadResponse
+}
+ 
+GetBoundProfilePackageResponse ::= [58] CHOICE { -- Tag 'BF3A'
+    getBoundProfilePackageOk GetBoundProfilePackageOk,
+    getBoundProfilePackageError INTEGER {
+        euiccSignatureInvalid(1),
+        confirmationCodeMissing(2),
+        confirmationCodeRefused(3),
+        confirmationCodeRetriesExceeded(4),
+        invalidTransactionId(95),
+        undefinedError(127)
+    }
+}
+ 
+GetBoundProfilePackageOk ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    boundProfilePackage [54] BoundProfilePackage
+}
+ 
+HandleNotification ::= [61] SEQUENCE {  -- Tag 'BF3D'
+    pendingNotification PendingNotification
+}
+ 
+CancelSessionRequestEs9 ::= [65] SEQUENCE { -- Tag 'BF41'
+    transactionId TransactionId,
+    cancelSessionResponse  CancelSessionResponse -- data structure defined for ES10b.CancelSession function
+}
+ 
+CancelSessionResponseEs9 ::= [65] CHOICE { -- Tag 'BF41'
+    cancelSessionOk CancelSessionOk,
+    cancelSessionError INTEGER {
+        invalidTransactionId(1),
+        euiccSignatureInvalid(2),
+        undefinedError(127)
+    }
+}
+ 
+CancelSessionOk ::= SEQUENCE { -- This function has no output data
+}
+ 
+EuiccConfiguredAddressesRequest ::= [60] SEQUENCE { -- Tag 'BF3C'
+}
+ 
+EuiccConfiguredAddressesResponse ::= [60] SEQUENCE {  -- Tag 'BF3C'
+    defaultDpAddress UTF8String OPTIONAL,  -- Default SM-DP+ address as an FQDN
+    rootDsAddress UTF8String  -- Root SM-DS address as an FQDN
+}
+ 
+ISDRProprietaryApplicationTemplate ::= [PRIVATE 0] SEQUENCE { -- Tag 'E0'  
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    lpaeSupport BIT STRING {
+        lpaeUsingCat(0), -- LPA in the eUICC using Card Application Toolkit
+        lpaeUsingScws(1) -- LPA in the eUICC using Smartcard Web Server
+    } OPTIONAL
+}
+ 
+LpaeActivationRequest ::= [66] SEQUENCE { -- Tag 'BF42'
+    lpaeOption BIT STRING {
+        activateCatBasedLpae(0), -- LPAe with LUIe based on CAT
+        activateScwsBasedLpae(1) -- LPAe with LUIe based on SCWS
+    }
+}
+ 
+LpaeActivationResponse ::= [66] SEQUENCE { -- Tag 'BF42'
+    lpaeActivationResult INTEGER {ok(0), notSupported(1)}
+}
+ 
+SetDefaultDpAddressRequest ::= [63] SEQUENCE { -- Tag 'BF3F'
+    defaultDpAddress UTF8String -- Default SM-DP+ address as an FQDN
+}
+ 
+SetDefaultDpAddressResponse ::= [63] SEQUENCE { -- Tag 'BF3F'
+    setDefaultDpAddressResult INTEGER { ok (0), undefinedError (127)}
+}
+ 
+AuthenticateClientResponseEs11 ::= [64] CHOICE {  -- Tag 'BF40'
+    authenticateClientOk AuthenticateClientOkEs11,
+    authenticateClientError INTEGER {
+        eumCertificateInvalid(1),
+        eumCertificateExpired(2),
+        euiccCertificateInvalid(3),
+        euiccCertificateExpired(4),
+        euiccSignatureInvalid(5),
+        eventIdUnknown(6),
+        invalidTransactionId(7),
+        undefinedError(127)
+    }
+}
+ 
+AuthenticateClientOkEs11 ::= SEQUENCE {
+    transactionId TransactionId,
+    eventEntries SEQUENCE OF EventEntries
+}
+ 
+EventEntries ::= SEQUENCE {
+    eventId UTF8String,
+    rspServerAddress UTF8String
+}
+ 
+END

pySim/esim/bsp.py

@@ -0,0 +1,297 @@
+# Early proof-of-concept implementation of
+# GSMA eSIM RSP (Remote SIM Provisioning BSP (BPP Protection Protocol),
+# where BPP is the Bound  Profile Package.  So the full expansion is the
+# "GSMA eSIM Remote SIM Provisioning Bound Profile Packate Protection Protocol"
+#
+# Originally (SGP.22 v2.x) this was called SCP03t, but it has since been
+# renamed to BSP.
+#
+# (C) 2023 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# SGP.22 v3.0 Section 2.5.3:
+# That block of data is split into segments of a maximum size of 1020 bytes (including the tag, length field and MAC).
+MAX_SEGMENT_SIZE = 1020
+
+import abc
+from typing import List
+import logging
+
+# for BSP key derivation
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.x963kdf import X963KDF
+
+from Cryptodome.Cipher import AES
+from Cryptodome.Hash import CMAC
+
+from pySim.utils import bertlv_encode_len, bertlv_parse_one, b2h
+
+# don't log by default
+logger = logging.getLogger(__name__)
+logger.addHandler(logging.NullHandler())
+
+class BspAlgo(abc.ABC):
+    blocksize: int
+
+    def _get_padding(self, in_len: int, multiple: int, padding: int = 0) -> bytes:
+        """Return padding bytes towards multiple of N."""
+        if in_len % multiple == 0:
+            return b''
+        pad_cnt = multiple - (in_len % multiple)
+        return b'\x00' * pad_cnt
+
+    def _pad_to_multiple(self, indat: bytes, multiple: int, padding: int = 0) -> bytes:
+        """Pad the input data to multiples of 'multiple'."""
+        return indat + self._get_padding(len(indat), self.blocksize, padding)
+
+    def __str__(self):
+        return self.__class__.__name__
+
+class BspAlgoCrypt(BspAlgo, abc.ABC):
+
+    def __init__(self, s_enc: bytes):
+        self.s_enc = s_enc
+        self.block_nr = 1
+
+    def encrypt(self, data:bytes) -> bytes:
+        """Encrypt given input bytes using the key material given in constructor."""
+        padded_data = self._pad_to_multiple(data, self.blocksize)
+        block_nr = self.block_nr
+        ciphertext = self._encrypt(padded_data)
+        logger.debug("encrypt(block_nr=%u, s_enc=%s, plaintext=%s, padded=%s) -> %s",
+                     block_nr, b2h(self.s_enc), b2h(data), b2h(padded_data), b2h(ciphertext))
+        return ciphertext
+
+    def decrypt(self, data:bytes) -> bytes:
+        """Decrypt given input bytes using the key material given in constructor."""
+        return self._unpad(self._decrypt(data))
+
+    @abc.abstractmethod
+    def _unpad(self, padded: bytes) -> bytes:
+        """Remove the padding from padded data."""
+        pass
+
+    @abc.abstractmethod
+    def _encrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+        pass
+
+    @abc.abstractmethod
+    def _decrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+        pass
+
+class BspAlgoCryptAES128(BspAlgoCrypt):
+    name = 'AES-CBC-128'
+    blocksize = 16
+
+    def _get_padding(self, in_len: int, multiple: int, padding: int = 0):
+        # SGP.22 section 2.6.4.4
+        # Append a byte with value '80' to the right of the data block;
+        # Append 0 to 15 bytes with value '00' so that the length of the padded data block
+        # is a multiple of 16 bytes.
+        return b'\x80' + super()._get_padding(in_len + 1, multiple, padding)
+
+    def _unpad(self, padded: bytes) -> bytes:
+        """Remove the customary 80 00 00 ... padding used for AES."""
+        # first remove any trailing zero bytes
+        stripped = padded.rstrip(b'\0')
+        # then remove the final 80
+        assert stripped[-1] == 0x80
+        return stripped[:-1]
+
+    def _get_icv(self):
+        # The binary value of this number SHALL be left padded with zeroes to form a full block.
+        data = self.block_nr.to_bytes(self.blocksize, "big")
+        #iv = bytes([0] * (self.blocksize-1)) + b'\x01'
+        iv = bytes([0] * self.blocksize)
+        # This block SHALL be encrypted with S-ENC to produce the ICV for command encryption.
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, iv)
+        icv = cipher.encrypt(data)
+        logger.debug("_get_icv(block_nr=%u, data=%s) -> icv=%s", self.block_nr, b2h(data), b2h(icv))
+        self.block_nr = self.block_nr + 1
+        return icv
+
+    def _encrypt(self, data: bytes) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv())
+        return cipher.encrypt(data)
+
+    def _decrypt(self, data: bytes) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv())
+        return cipher.decrypt(data)
+
+
+class BspAlgoMac(BspAlgo, abc.ABC):
+    l_mac = 0 # must be overridden by derived class
+
+    def __init__(self, s_mac: bytes, initial_mac_chaining_value: bytes):
+        self.s_mac = s_mac
+        self.mac_chain = initial_mac_chaining_value
+
+    def auth(self, tag: int, data: bytes) -> bytes:
+        assert tag in range (256)
+        # The input data used for C-MAC computation comprises the MAC Chaining value, the tag, the final length and the result of step 2
+        lcc = len(data) + self.l_mac
+        tag_and_length = bytes([tag]) + bertlv_encode_len(lcc)
+        temp_data = self.mac_chain + tag_and_length + data
+        old_mcv = self.mac_chain
+        c_mac = self._auth(temp_data)
+        # The output data is computed by concatenating the following data: the tag, the final length, the result of step 2 and the C-MAC value.
+        ret = tag_and_length + data + c_mac
+        logger.debug("auth(tag=0x%x, mcv=%s, s_mac=%s, plaintext=%s, temp=%s) -> %s",
+                     tag, b2h(old_mcv), b2h(self.s_mac), b2h(data), b2h(temp_data), b2h(ret))
+        return ret
+
+    def verify(self, ciphertext: bytes) -> bool:
+        mac_stripped = ciphertext[0:-self.l_mac]
+        mac_received = ciphertext[-self.l_mac:]
+        temp_data = self.mac_chain + mac_stripped
+        mac_computed = self._auth(temp_data)
+        if mac_received != mac_computed:
+            raise ValueError("MAC value not matching: received: %s, computed: %s" % (mac_received, mac_computed))
+        return mac_stripped
+
+    @abc.abstractmethod
+    def _auth(self, temp_data: bytes) -> bytes:
+        """To be implemented by algorithm specific derived class."""
+        pass
+
+class BspAlgoMacAES128(BspAlgoMac):
+    name = 'AES-CMAC-128'
+    l_mac = 8
+
+    def _auth(self, temp_data: bytes) -> bytes:
+        # The full MAC value is computed using the MACing algorithm as defined in table 4c.
+        cmac = CMAC.new(self.s_mac, ciphermod=AES)
+        cmac.update(temp_data)
+        full_c_mac = cmac.digest()
+        # Subsequent MAC chaining values are the full result of step 4 of the previous data block
+        self.mac_chain = full_c_mac
+        # If the algorithm is AES-CBC-128 or SM4-CBC, the C-MAC value is the 8 most significant bytes of the result of step 4
+        return full_c_mac[0:8]
+
+
+
+def bsp_key_derivation(shared_secret: bytes, key_type: int, key_length: int, host_id: bytes, eid, l : int = 16):
+    """BSP protocol key derivation as per SGP.22 v3.0 Section 2.6.4.2"""
+    assert key_type <= 255
+    assert key_length <= 255
+
+    host_id_lv = bertlv_encode_len(len(host_id)) + host_id
+    eid_lv = bertlv_encode_len(len(eid)) + eid
+    shared_info = bytes([key_type, key_length]) + host_id_lv + eid_lv
+    logger.debug("kdf_shared_info: %s", b2h(shared_info))
+
+    # X9.63 Key Derivation Function with SHA256
+    xkdf = X963KDF(algorithm=hashes.SHA256(), length=l*3, sharedinfo=shared_info)
+    out = xkdf.derive(shared_secret)
+    logger.debug("kdf_out: %s", b2h(out))
+
+    initial_mac_chaining_value = out[0:l]
+    s_enc = out[l:2*l]
+    s_mac = out[l*2:3*l]
+
+    return s_enc, s_mac, initial_mac_chaining_value
+
+
+
+class BspInstance:
+    """An instance of the BSP crypto.  Initialized once with the key material via constructor,
+    then the user can call any number of encrypt_and_mac cycles to protect plaintext and
+    generate the respective ciphertext."""
+    def __init__(self, s_enc: bytes, s_mac: bytes, initial_mcv: bytes):
+        logger.debug("%s(s_enc=%s, s_mac=%s, initial_mcv=%s)", self.__class__.__name__, b2h(s_enc), b2h(s_mac), b2h(initial_mcv))
+        self.c_algo = BspAlgoCryptAES128(s_enc)
+        self.m_algo = BspAlgoMacAES128(s_mac, initial_mcv)
+
+        TAG_LEN = 1
+        length_len = len(bertlv_encode_len(MAX_SEGMENT_SIZE))
+        self.max_payload_size = MAX_SEGMENT_SIZE - TAG_LEN - length_len - self.m_algo.l_mac
+
+    @classmethod
+    def from_kdf(cls, shared_secret: bytes, key_type: int, key_length: int, host_id: bytes, eid: bytes):
+        """Convenience constructor for constructing an instance with keys from KDF."""
+        s_enc, s_mac, initial_mcv = bsp_key_derivation(shared_secret, key_type, key_length, host_id, eid)
+        return cls(s_enc, s_mac, initial_mcv)
+
+    def encrypt_and_mac_one(self, tag: int, plaintext:bytes) -> bytes:
+        """Encrypt + MAC a single plaintext TLV. Returns the protected ciphertex."""
+        assert tag <= 255
+        assert len(plaintext) <= self.max_payload_size
+        logger.debug("encrypt_and_mac_one(tag=0x%x, plaintext=%s)", tag, b2h(plaintext))
+        ciphered = self.c_algo.encrypt(plaintext)
+        maced = self.m_algo.auth(tag, ciphered)
+        return maced
+
+    def encrypt_and_mac(self, tag: int, plaintext:bytes) -> List[bytes]:
+        remainder = plaintext
+        result = []
+        while len(remainder):
+            remaining_len = len(remainder)
+            if remaining_len < self.max_payload_size:
+                segment_len = remaining_len
+                segment = remainder
+                remainder = b''
+            else:
+                segment_len = self.max_payload_size
+                segment = remainder[0:segment_len]
+                remainder = remainder[segment_len:]
+            result.append(self.encrypt_and_mac_one(tag, segment))
+        return result
+
+    def mac_only_one(self, tag: int, plaintext: bytes) -> bytes:
+        """MAC a single plaintext TLV. Returns the protected ciphertex."""
+        assert tag <= 255
+        assert len(plaintext) < self.max_payload_size
+        maced = self.m_algo.auth(tag, plaintext)
+        # The data block counter for ICV caluclation is incremented also for each segment with C-MAC only.
+        self.c_algo.block_nr += 1
+        return maced
+
+    def mac_only(self, tag: int, plaintext:bytes) -> List[bytes]:
+        remainder = plaintext
+        result = []
+        while len(remainder):
+            remaining_len = len(remainder)
+            if remaining_len < self.max_payload_size:
+                segment_len = remaining_len
+                segment = remainder
+                remainder = b''
+            else:
+                segment_len = self.max_payload_size
+                segment = remainder[0:segment_len]
+                remainder = remainder[segment_len:]
+            result.append(self.mac_only_one(tag, segment))
+        return result
+
+    def demac_and_decrypt_one(self, ciphertext: bytes) -> bytes:
+        payload = self.m_algo.verify(ciphertext)
+        tdict, l, val, remain = bertlv_parse_one(payload)
+        logger.debug("tag=%s, l=%u, val=%s, remain=%s", tdict, l, b2h(val), b2h(remain))
+        plaintext = self.c_algo.decrypt(val)
+        return plaintext
+
+    def demac_and_decrypt(self, ciphertext_list: List[bytes]) -> bytes:
+        plaintext_list = [self.demac_and_decrypt_one(x) for x in ciphertext_list]
+        return b''.join(plaintext_list)
+
+    def demac_only_one(self, ciphertext: bytes) -> bytes:
+        payload = self.m_algo.verify(ciphertext)
+        tdict, l, val, remain = bertlv_parse_one(payload)
+        return val
+
+    def demac_only(self, ciphertext_list: List[bytes]) -> bytes:
+        plaintext_list = [self.demac_only_one(x) for x in ciphertext_list]
+        return b''.join(plaintext_list)

pySim/esim/es8p.py

@@ -0,0 +1,185 @@
+# Implementation of GSMA eSIM RSP (Remote SIM Provisioning) ES8+
+# as per SGP22 v3.0 Section 5.5
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import Dict, List, Optional
+from pySim.utils import b2h, h2b, bertlv_encode_tag, bertlv_encode_len
+
+import pySim.esim.rsp as rsp
+from pySim.esim.bsp import BspInstance
+
+# Given that GSMA RSP uses ASN.1 in a very weird way, we actually cannot encode the full data type before
+# signing, but we have to build parts of it separately first, then sign that, so we can put the signature
+# into the same sequence as the signed data.  We use the existing pySim TLV code for this.
+
+def wrap_as_der_tlv(tag: int, val: bytes) -> bytes:
+    """Wrap the 'value' into a DER-encoded TLV."""
+    return bertlv_encode_tag(tag) + bertlv_encode_len(len(val)) + val
+
+def gen_init_sec_chan_signed_part(iscsp: Dict) -> bytes:
+    """Generate the concatenated remoteOpId, transactionId, controlRefTemplate and smdpOtpk data objects
+    without the outer SEQUENCE tag / length or the remainder of initialiseSecureChannel, as is required
+    for signing purpose."""
+    out = b''
+    out += wrap_as_der_tlv(0x82, bytes([iscsp['remoteOpId']]))
+    out += wrap_as_der_tlv(0x80, iscsp['transactionId'])
+
+    crt = iscsp['controlRefTemplate']
+    out_crt = wrap_as_der_tlv(0x80, crt['keyType'])
+    out_crt += wrap_as_der_tlv(0x81, crt['keyLen'])
+    out_crt += wrap_as_der_tlv(0x84, crt['hostId'])
+    out += wrap_as_der_tlv(0xA6, out_crt)
+
+    out += wrap_as_der_tlv(0x5F49, iscsp['smdpOtpk'])
+    return out
+
+
+# SGP.22 Section 5.5.1
+def gen_initialiseSecureChannel(transactionId: str, host_id: bytes, smdp_otpk: bytes, euicc_otpk: bytes, dp_pb):
+    """Generate decoded representation of (signed) initialiseSecureChannel (SGP.22 5.5.2)"""
+    init_scr = { 'remoteOpId': 1, # installBoundProfilePackage
+                 'transactionId': h2b(transactionId),
+                 # GlobalPlatform Card Specification Amendment F [13] section 6.5.2.3 for the Mutual Authentication Data Field
+                 'controlRefTemplate': { 'keyType': bytes([0x88]), 'keyLen': bytes([16]), 'hostId': host_id },
+                 'smdpOtpk': smdp_otpk, # otPK.DP.KA
+                }
+    to_sign = gen_init_sec_chan_signed_part(init_scr) + wrap_as_der_tlv(0x5f49, euicc_otpk)
+    init_scr['smdpSign'] = dp_pb.ecdsa_sign(to_sign)
+    return init_scr
+
+def gen_replace_session_keys(ppk_enc: bytes, ppk_cmac: bytes, initial_mcv: bytes) -> bytes:
+    """Generate encoded (but unsigned) ReplaceSessionKeysReqest DO (SGP.22 5.5.4)"""
+    rsk = { 'ppkEnc': ppk_enc, 'ppkCmac': ppk_cmac, 'initialMacChainingValue': initial_mcv }
+    return rsp.asn1.encode('ReplaceSessionKeysRequest', rsk)
+
+
+class ProfileMetadata:
+    """Representation of Profile metadata. Right now only the mandatory bits are
+    supported, but in general this should follow the StoreMetadataRequest of SGP.22 5.5.3"""
+    def __init__(self, iccid_bin: bytes, spn: str, profile_name: str):
+        self.iccid_bin = iccid_bin
+        self.spn = spn
+        self.profile_name = profile_name
+
+    def gen_store_metadata_request(self) -> bytes:
+        """Generate encoded (but unsigned) StoreMetadataReqest DO (SGP.22 5.5.3)"""
+        smr = {
+            'iccid': self.iccid_bin,
+            'serviceProviderName': self.spn,
+            'profileName': self.profile_name,
+        }
+        return rsp.asn1.encode('StoreMetadataRequest', smr)
+
+
+class ProfilePackage:
+    def __init__(self, metadata: Optional[ProfileMetadata] = None):
+        self.metadata = metadata
+
+class UnprotectedProfilePackage(ProfilePackage):
+    """Representing an unprotected profile package (UPP) as defined in SGP.22 Section 2.5.2"""
+
+    @classmethod
+    def from_der(cls, der: bytes, metadata: Optional[ProfileMetadata] = None) -> 'UnprotectedProfilePackage':
+        """Load an UPP from its DER representation."""
+        inst = cls(metadata=metadata)
+        cls.der = der
+        # TODO: we later certainly want to parse it so we can perform modification (IMSI, key material, ...)
+        # just like in the traditional SIM/USIM dynamic data phase at the end of personalization
+        return inst
+
+    def to_der(self):
+        """Return the DER representation of the UPP."""
+        # TODO: once we work on decoded structures, we may want to re-encode here
+        return self.der
+
+class ProtectedProfilePackage(ProfilePackage):
+    """Representing a protected profile package (PPP) as defined in SGP.22 Section 2.5.3"""
+
+    @classmethod
+    def from_upp(cls, upp: UnprotectedProfilePackage, bsp: BspInstance) -> 'ProtectedProfilePackage':
+        """Generate the PPP as a sequence of encrypted and MACed Command TLVs representing the UPP"""
+        inst = cls(metadata=upp.metadata)
+        inst.upp = upp
+        # store ppk-enc, ppc-mac
+        inst.ppk_enc = bsp.c_algo.s_enc
+        inst.ppk_mac = bsp.m_algo.s_mac
+        inst.initial_mcv = bsp.m_algo.mac_chain
+        inst.encoded = bsp.encrypt_and_mac(0x86, upp.to_der())
+        return inst
+
+    #def __val__(self):
+        #return self.encoded
+
+class BoundProfilePackage(ProfilePackage):
+    """Representing a bound profile package (BPP) as defined in SGP.22 Section 2.5.4"""
+
+    @classmethod
+    def from_ppp(cls, ppp: ProtectedProfilePackage):
+        inst = cls()
+        inst.upp = None
+        inst.ppp = ppp
+        return inst
+
+    @classmethod
+    def from_upp(cls, upp: UnprotectedProfilePackage):
+        inst = cls()
+        inst.upp = upp
+        inst.ppp = None
+        return inst
+
+    def encode(self, ss: 'RspSessionState', dp_pb: 'CertAndPrivkey') -> bytes:
+        """Generate a bound profile package (SGP.22 2.5.4)."""
+
+        def encode_seq(tag: int, sequence: List[bytes]) -> bytes:
+            """Encode a "sequenceOfXX" as specified in SGP.22 specifying the raw SEQUENCE OF tag,
+            and assuming the caller provides the fully-encoded (with TAG + LEN) member TLVs."""
+            payload = b''.join(sequence)
+            return bertlv_encode_tag(tag) + bertlv_encode_len(len(payload)) + payload
+
+        bsp = BspInstance.from_kdf(ss.shared_secret, 0x88, 16, ss.host_id, h2b(ss.eid))
+
+        iscr = gen_initialiseSecureChannel(ss.transactionId, ss.host_id, ss.smdp_otpk, ss.euicc_otpk, dp_pb)
+        # generate unprotected input data
+        conf_idsp_bin = rsp.asn1.encode('ConfigureISDPRequest', {})
+        if self.upp:
+            smr_bin = self.upp.metadata.gen_store_metadata_request()
+        else:
+            smr_bin = self.ppp.metadata.gen_store_metadata_request()
+
+        # we don't use rsp.asn1.encode('boundProfilePackage') here, as the BSP already provides
+        # fully encoded + MACed TLVs including their tag + length values.  We cannot put those as
+        # 'value' input into an ASN.1 encoder, as that would double the TAG + LENGTH :(
+
+        # 'initialiseSecureChannelRequest'
+        bpp_seq = rsp.asn1.encode('InitialiseSecureChannelRequest', iscr)
+        # firstSequenceOf87
+        bpp_seq += encode_seq(0xa0, bsp.encrypt_and_mac(0x87, conf_idsp_bin))
+        # sequenceOF88
+        bpp_seq += encode_seq(0xa1, bsp.mac_only(0x88, smr_bin))
+
+        if self.ppp: # we have to use session keys
+            rsk_bin = gen_replace_session_keys(self.ppp.ppk_enc, self.ppp.ppk_mac, self.ppp.initial_mcv)
+            # secondSequenceOf87
+            bpp_seq += encode_seq(0xa2, bsp.encrypt_and_mac(0x87, rsk_bin))
+        else:
+            self.ppp = ProtectedProfilePackage.from_upp(self.upp, bsp)
+
+        # 'sequenceOf86'
+        bpp_seq += encode_seq(0xa3, self.ppp.encoded)
+
+        # manual DER encode: wrap in outer SEQUENCE
+        return bertlv_encode_tag(0xbf36) + bertlv_encode_len(len(bpp_seq)) + bpp_seq

pySim/esim/rsp.py

@@ -0,0 +1,100 @@
+# Implementation of GSMA eSIM RSP (Remote SIM Provisioning)
+# as per SGP22 v3.0
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from typing import Optional
+import shelve
+import copyreg
+
+from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.primitives.serialization import Encoding
+from cryptography import x509
+from collections.abc import MutableMapping
+
+from pySim.esim import compile_asn1_subdir
+
+asn1 = compile_asn1_subdir('rsp')
+
+class RspSessionState:
+    """Encapsulates the state of a RSP session.  It is created during the initiateAuthentication
+    and subsequently used by further API calls using the same transactionId.  The session state
+    is removed either after cancelSession or after notification.
+    TODO: add some kind of time based expiration / garbage collection."""
+    def __init__(self, transactionId: str, serverChallenge: bytes):
+        self.transactionId = transactionId
+        self.serverChallenge = serverChallenge
+        #  used at a later point between API calsl
+        self.euicc_cert: Optional[x509.Certificate] = None
+        self.eum_cert: Optional[x509.Certificate] = None
+        self.eid: Optional[bytes] = None
+        self.profileMetadata: Optional['ProfileMetadata'] = None
+        self.smdpSignature2_do = None
+        # really only needed while processing getBoundProfilePackage request?
+        self.euicc_otpk: Optional[bytes] = None
+        self.smdp_ot: Optional[ec.EllipticCurvePrivateKey] = None
+        self.smdp_otpk: Optional[bytes] = None
+        self.host_id: Optional[bytes] = None
+        self.shared_secret: Optional[bytes] = None
+
+
+    def __getstate__(self):
+        """helper function called when pickling the object to persistent storage.  We must pickel all
+        members that are not pickle-able."""
+        state = self.__dict__.copy()
+        # serialize eUICC certificate as DER
+        if state.get('euicc_cert', None):
+            state['_euicc_cert'] = self.euicc_cert.public_bytes(Encoding.DER)
+            del state['euicc_cert']
+        # serialize EUM certificate as DER
+        if state.get('eum_cert', None):
+            state['_eum_cert'] = self.eum_cert.public_bytes(Encoding.DER)
+            del state['eum_cert']
+        # serialize one-time SMDP private key to integer + curve
+        if state.get('smdp_ot', None):
+            state['_smdp_otsk'] = self.smdp_ot.private_numbers().private_value
+            state['_smdp_ot_curve'] = self.smdp_ot.curve
+            del state['smdp_ot']
+        return state
+
+    def __setstate__(self, state):
+        """helper function called when unpickling the object from persistent storage. We must recreate all
+        members from the state generated in __getstate__ above."""
+        # restore eUICC certificate from DER
+        if '_euicc_cert' in state:
+            self.euicc_cert = x509.load_der_x509_certificate(state['_euicc_cert'])
+            del state['_euicc_cert']
+        else:
+            self.euicc_cert = None
+        # restore EUM certificate from DER
+        if '_eum_cert' in state:
+            self.eum_cert = x509.load_der_x509_certificate(state['_eum_cert'])
+            del state['_eum_cert']
+        # restore one-time SMDP private key from integer + curve
+        if state.get('_smdp_otsk', None):
+            self.smdp_ot = ec.derive_private_key(state['_smdp_otsk'], state['_smdp_ot_curve'])
+            # FIXME: how to add the public key from smdp_otpk to an instance of EllipticCurvePrivateKey?
+            del state['_smdp_otsk']
+            del state['_smdp_ot_curve']
+        # automatically recover all the remainig state
+        self.__dict__.update(state)
+
+
+class RspSessionStore(shelve.DbfilenameShelf):
+    """A derived class as wrapper around the database-backed non-volatile storage 'shelve', in case we might
+    need to extend it in the future. We use it to store RspSessionState objects indexed by transactionId."""
+    pass

pySim/esim/saip/__init__.py

@@ -0,0 +1,229 @@
+# Implementation of SimAlliance/TCA Interoperable Profile handling
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+from typing import Tuple, List, Optional, Dict
+
+import asn1tools
+
+from pySim.utils import bertlv_parse_tag, bertlv_parse_len
+from pySim.esim import compile_asn1_subdir
+
+asn1 = compile_asn1_subdir('saip')
+
+class oid:
+    class OID:
+        @staticmethod
+        def intlist_from_str(instr: str) -> List[int]:
+            return [int(x) for x in instr.split('.')]
+
+        def __init__(self, initializer):
+            if type(initializer) == str:
+                self.intlist = self.intlist_from_str(initializer)
+            else:
+                self.intlist = initializer
+
+        def __str__(self):
+            return '.'.join([str(x) for x in self.intlist])
+
+        def __repr__(self):
+            return 'OID(%s)' % (str(self))
+
+
+    class eOID(OID):
+        """OID helper for TCA eUICC prefix"""
+        __prefix = [2,23,143,1]
+        def __init__(self, initializer):
+            if type(initializer) == str:
+                initializer = self.intlist_from_str(initializer)
+            super().__init__(self.__prefix + initializer)
+
+    MF = eOID("2.1")
+    DF_CD = eOID("2.2")
+    DF_TELECOM = eOID("2.3")
+    DF_TELECOM_v2 = eOID("2.3.2")
+    ADF_USIM_by_default = eOID("2.4")
+    ADF_USIM_by_default_v2 = eOID("2.4.2")
+    ADF_USIM_not_by_default = eOID("2.5")
+    ADF_USIM_not_by_default_v2 = eOID("2.5.2")
+    ADF_USIM_not_by_default_v3 = eOID("2.5.3")
+    DF_PHONEBOOK_ADF_USIM = eOID("2.6")
+    DF_GSM_ACCESS_ADF_USIM = eOID("2.7")
+    ADF_ISIM_by_default = eOID("2.8")
+    ADF_ISIM_not_by_default = eOID("2.9")
+    ADF_ISIM_not_by_default_v2 = eOID("2.9.2")
+    ADF_CSIM_by_default = eOID("2.10")
+    ADF_CSIM_by_default_v2 = eOID("2.10.2")
+    ADF_CSIM_not_by_default = eOID("2.11")
+    ADF_CSIM_not_by_default_v2 = eOID("2.11.2")
+    DF_EAP = eOID("2.12")
+    DF_5GS = eOID("2.13")
+    DF_5GS_v2 = eOID("2.13.2")
+    DF_5GS_v3 = eOID("2.13.3")
+    DF_5GS_v4 = eOID("2.13.4")
+    DF_SAIP = eOID("2.14")
+    DF_SNPN = eOID("2.15")
+    DF_5GProSe = eOID("2.16")
+    IoT_default = eOID("2.17")
+    IoT_default = eOID("2.18")
+
+
+class ProfileElement:
+    def _fixup_sqnInit_dec(self):
+        """asn1tools has a bug when working with SEQUENCE OF that have DEFAULT values. Let's work around
+        this."""
+        if self.type != 'akaParameter':
+            return
+        sqn_init = self.decoded.get('sqnInit', None)
+        if not sqn_init:
+            return
+        # this weird '0x' value in a string is what we get from our (slightly hacked) ASN.1 syntax
+        if sqn_init == '0x000000000000':
+            # SEQUENCE (SIZE (32)) OF OCTET STRING (SIZE (6))
+            self.decoded['sqnInit'] = [b'\x00'*6] * 32
+
+    def _fixup_sqnInit_enc(self):
+        """asn1tools has a bug when working with SEQUENCE OF that have DEFAULT values. Let's work around
+        this."""
+        if self.type != 'akaParameter':
+            return
+        sqn_init = self.decoded.get('sqnInit', None)
+        if not sqn_init:
+            return
+        for s in sqn_init:
+            if any(s):
+                return
+        # none of the fields were initialized with a non-default (non-zero) value, so we can skip it
+        del self.decoded['sqnInit']
+
+    def parse_der(self, der: bytes):
+        """Parse a sequence of PE and store the result in instance attributes."""
+        self.type, self.decoded = asn1.decode('ProfileElement', der)
+        # work around asn1tools bug regarding DEFAULT for a SEQUENCE OF
+        self._fixup_sqnInit_dec()
+
+    @classmethod
+    def from_der(cls, der: bytes) -> 'ProfileElement':
+        """Construct an instance from given raw, DER encoded bytes."""
+        inst = cls()
+        inst.parse_der(der)
+        return inst
+
+    def to_der(self) -> bytes:
+        """Build an encoded DER representation of the instance."""
+        # work around asn1tools bug regarding DEFAULT for a SEQUENCE OF
+        self._fixup_sqnInit_enc()
+        return asn1.encode('ProfileElement', (self.type, self.decoded))
+
+    def __str__(self):
+        return self.type
+
+
+def bertlv_first_segment(binary: bytes) -> Tuple[bytes, bytes]:
+    """obtain the first segment of a binary concatenation of BER-TLV objects.
+        Returns: tuple of first TLV and remainder."""
+    tagdict, remainder = bertlv_parse_tag(binary)
+    length, remainder = bertlv_parse_len(remainder)
+    tl_length = len(binary) - len(remainder)
+    tlv_length = tl_length + length
+    return binary[:tlv_length], binary[tlv_length:]
+
+class ProfileElementSequence:
+    """A sequence of ProfileElement objects, which is the overall representation of an eSIM profile."""
+    def __init__(self):
+        self.pe_list: List[ProfileElement] = None
+        self.pe_by_type: Dict = {}
+        self.pes_by_naa: Dict = {}
+
+    def get_pes_for_type(self, tname: str) -> List[ProfileElement]:
+        return self.pe_by_type.get(tname, [])
+
+    def get_pe_for_type(self, tname: str) -> Optional[ProfileElement]:
+        l = self.get_pes_for_type(tname)
+        if len(l) == 0:
+            return None
+        assert len(l) == 1
+        return l[0]
+
+    def parse_der(self, der: bytes):
+        """Parse a sequence of PE and store the result in self.pe_list."""
+        self.pe_list = []
+        remainder = der
+        while len(remainder):
+            first_tlv, remainder = bertlv_first_segment(remainder)
+            self.pe_list.append(ProfileElement.from_der(first_tlv))
+        self._process_pelist()
+
+    def _process_pelist(self):
+        self._rebuild_pe_by_type()
+        self._rebuild_pes_by_naa()
+
+    def _rebuild_pe_by_type(self):
+        self.pe_by_type = {}
+        # build a dict {pe_type: [pe, pe, pe]}
+        for pe in self.pe_list:
+            if pe.type in self.pe_by_type:
+                self.pe_by_type[pe.type].append(pe)
+            else:
+                self.pe_by_type[pe.type] = [pe]
+
+    def _rebuild_pes_by_naa(self):
+        """rebuild the self.pes_by_naa dict {naa: [ [pe, pe, pe], [pe, pe] ]} form,
+        which basically means for every NAA there's a lsit of instances, and each consists
+        of a list of a list of PEs."""
+        self.pres_by_naa = {}
+        petype_not_naa_related = ['securityDomain', 'rfm', 'application', 'end']
+        naa = ['mf', 'usim', 'isim', 'csim']
+        cur_naa = None
+        cur_naa_list = []
+        for pe in self.pe_list:
+            # skip all PE that are not related to NAA
+            if pe.type in petype_not_naa_related:
+                continue
+            if pe.type in naa:
+                if cur_naa:
+                    if not cur_naa in self.pes_by_naa:
+                        self.pes_by_naa[cur_naa] = []
+                    self.pes_by_naa[cur_naa].append(cur_naa_list)
+                cur_naa = pe.type
+                cur_naa_list = []
+            cur_naa_list.append(pe)
+        # append the final one
+        if cur_naa and len(cur_naa_list):
+            if not cur_naa in self.pes_by_naa:
+                self.pes_by_naa[cur_naa] = []
+            self.pes_by_naa[cur_naa].append(cur_naa_list)
+
+    @classmethod
+    def from_der(cls, der: bytes) -> 'ProfileElementSequence':
+        """Construct an instance from given raw, DER encoded bytes."""
+        inst = cls()
+        inst.parse_der(der)
+        return inst
+
+    def to_der(self) -> bytes:
+        """Build an encoded DER representation of the instance."""
+        out = b''
+        for pe in self.pe_list:
+            out += pe.to_der()
+        return out
+
+    def __repr__(self):
+        return "PESequence(%s)" % ', '.join([str(x) for x in self.pe_list])
+
+    def __iter__(self):
+        yield from self.pe_list

pySim/esim/saip/personalization.py

@@ -0,0 +1,150 @@
+# Implementation of SimAlliance/TCA Interoperable Profile handling
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+from typing import List, Tuple, Optional
+
+from pySim.esim.saip import ProfileElement, ProfileElementSequence
+
+def remove_unwanted_tuples_from_list(l: List[Tuple], unwanted_key:str) -> List[Tuple]:
+    """In a list of tuples, remove all tuples whose first part equals 'unwanted_key'."""
+    return list(filter(lambda x: x[0] != unwanted_key, l))
+
+def file_replace_content(file: List[Tuple], new_content: bytes):
+    """Completely replace all fillFileContent of a decoded 'File' with the new_content."""
+    file = remove_unwanted_tuples_from_list(file, 'fillFileContent')
+    file.append(('fillFileContent', new_content))
+    return file
+
+class ClassVarMeta(abc.ABCMeta):
+    """Metaclass that puts all additional keyword-args into the class."""
+    def __new__(metacls, name, bases, namespace, **kwargs):
+        #print("Meta_new_(metacls=%s, name=%s, bases=%s, namespace=%s, kwargs=%s)" % (metacls, name, bases, namespace, kwargs))
+        x = super().__new__(metacls, name, bases, namespace)
+        for k, v in kwargs.items():
+            setattr(x, k, v)
+        return x
+
+class ConfigurableParameter(abc.ABC, metaclass=ClassVarMeta):
+    """Base class representing a part of the eSIM profile that is configurable during the
+    personalization process (with dynamic data from elsewhere)."""
+    def __init__(self, value):
+        self.value = value
+
+    @abc.abstractmethod
+    def apply(self, pe_seq: ProfileElementSequence):
+        pass
+
+class Iccid(ConfigurableParameter):
+    """Configurable ICCID.  Expects the value to be in EF.ICCID format."""
+    name = 'iccid'
+    def apply(self, pes: ProfileElementSequence):
+        # patch the header; FIXME: swap nibbles!
+        pes.get_pe_by_type('header').decoded['iccid'] = self.value
+        # patch MF/EF.ICCID
+        file_replace_content(pes.get_pe_by_type('mf').decoded['ef-iccid'], self.value)
+
+class Imsi(ConfigurableParameter):
+    """Configurable IMSI. Expects value to be n EF.IMSI format."""
+    name = 'imsi'
+    def apply(self, pes: ProfileElementSequence):
+        # patch ADF.USIM/EF.IMSI
+        for pe in pes.get_pes_by_type('usim'):
+            file_replace_content(pe.decoded['ef-imsi'], self.value)
+        # TODO: DF.GSM_ACCESS if not linked?
+
+def obtain_singleton_pe_from_pelist(l: List[ProfileElement], wanted_type: str) -> ProfileElement:
+    filtered = list(filter(lambda x: x.type == wanted_type, l))
+    assert len(filtered) == 1
+    return filtered[0]
+
+def obtain_first_pe_from_pelist(l: List[ProfileElement], wanted_type: str) -> ProfileElement:
+    filtered = list(filter(lambda x: x.type == wanted_type, l))
+    return filtered[0]
+
+class Puk(ConfigurableParameter, metaclass=ClassVarMeta):
+    keyReference = None
+    def apply(self, pes: ProfileElementSequence):
+        mf_pes = pes.pes_by_naa['mf'][0]
+        pukCodes = obtain_singleton_pe_from_pelist(mf_pes, 'pukCodes')
+        for pukCode in pukCodes.decoded['pukCodes']:
+            if pukCode['keyReference'] == self.keyReference:
+                pukCode['pukValue'] = self.value
+                return
+        raise ValueError('cannot find pukCode')
+class Puk1(Puk, keyReference=0x01):
+    pass
+class Puk2(Puk, keyReference=0x81):
+    pass
+
+class Pin(ConfigurableParameter, metaclass=ClassVarMeta):
+    keyReference = None
+    def apply(self, pes: ProfileElementSequence):
+        mf_pes = pes.pes_by_naa['mf'][0]
+        pinCodes = obtain_first_pe_from_pelist(mf_pes, 'pinCodes')
+        if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
+            return
+        for pinCode in pinCodes.decoded['pinCodes'][1]:
+            if pinCode['keyReference'] == self.keyReference:
+                 pinCode['pinValue'] = self.value
+                 return
+        raise ValueError('cannot find pinCode')
+class AppPin(ConfigurableParameter, metaclass=ClassVarMeta):
+    keyReference = None
+    def _apply_one(self, pe: ProfileElement):
+        pinCodes = obtain_first_pe_from_pelist(pe, 'pinCodes')
+        if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
+            return
+        for pinCode in pinCodes.decoded['pinCodes'][1]:
+            if pinCode['keyReference'] == self.keyReference:
+                pinCode['pinValue'] = self.value
+                return
+        raise ValueError('cannot find pinCode')
+
+    def apply(self, pes: ProfileElementSequence):
+        for naa in pes.pes_by_naa:
+            if naa not in ['usim','isim','csim','telecom']:
+                continue
+            for instance in pes.pes_by_naa[naa]:
+                self._apply_one(instance)
+class Pin1(Pin, keyReference=0x01):
+    pass
+# PIN2 is special: telecom + usim + isim + csim
+class Pin2(AppPin, keyReference=0x81):
+    pass
+class Adm1(Pin, keyReference=0x0A):
+    pass
+class Adm2(Pin, keyReference=0x0B):
+    pass
+
+
+class AlgoConfig(ConfigurableParameter, metaclass=ClassVarMeta):
+    key = None
+    def apply(self, pes: ProfileElementSequence):
+        for pe in pes.get_pes_for_type('akaParameter'):
+            algoConfiguration = pe.decoded['algoConfiguration']
+            if algoConfiguration[0] != 'algoParameter':
+                continue
+            algoConfiguration[1][self.key] = self.value
+
+class K(AlgoConfig, key='key'):
+    pass
+class Opc(AlgoConfig, key='opc'):
+    pass
+class AlgorithmID(AlgoConfig, key='algorithmID'):
+    pass
+

pySim/esim/saip/validation.py

@@ -0,0 +1,96 @@
+# Implementation of SimAlliance/TCA Interoperable Profile handling
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from pySim.esim.saip import *
+
+class ProfileError(Exception):
+    pass
+
+class ProfileConstraintChecker:
+    def check(self, pes: ProfileElementSequence):
+        for name in dir(self):
+            if name.startswith('check_'):
+                method = getattr(self, name)
+                method(pes)
+
+class CheckBasicStructure(ProfileConstraintChecker):
+    def _is_after_if_exists(self, pes: ProfileElementSequence, opt:str, after:str):
+        opt_pe = pes.get_pe_for_type(opt)
+        if opt_pe:
+            after_pe = pes.get_pe_for_type(after)
+            if not after_pe:
+                raise ProfileError('PE-%s without PE-%s' % (opt.upper(), after.upper()))
+            # FIXME: check order
+
+    def check_start_and_end(self, pes: ProfileElementSequence):
+        if pes.pe_list[0].type != 'header':
+            raise ProfileError('first element is not header')
+        if pes.pe_list[1].type != 'mf':
+            # strictly speaking: permitted, but we don't support MF via GenericFileManagement
+            raise ProfileError('second  element is not mf')
+        if pes.pe_list[-1].type != 'end':
+            raise ProfileError('last element is not end')
+
+    def check_number_of_occurrence(self, pes: ProfileElementSequence):
+        # check for invalid number of occurrences
+        if len(pes.get_pes_for_type('header')) != 1:
+            raise ProfileError('multiple ProfileHeader')
+        if len(pes.get_pes_for_type('mf')) != 1:
+            # strictly speaking: 0 permitted, but we don't support MF via GenericFileManagement
+            raise ProfileError('multiple PE-MF')
+        for tn in ['end', 'cd', 'telecom',
+                   'usim', 'isim', 'csim', 'opt-usim','opt-isim','opt-csim',
+                   'df-saip', 'df-5gs']:
+            if len(pes.get_pes_for_type(tn)) > 1:
+                raise ProfileError('multiple PE-%s' % tn.upper())
+
+    def check_optional_ordering(self, pes: ProfileElementSequence):
+        # ordering and required depenencies
+        self._is_after_if_exists(pes,'opt-usim', 'usim')
+        self._is_after_if_exists(pes,'opt-isim', 'isim')
+        self._is_after_if_exists(pes,'gsm-access', 'usim')
+        self._is_after_if_exists(pes,'phonebook', 'usim')
+        self._is_after_if_exists(pes,'df-5gs', 'usim')
+        self._is_after_if_exists(pes,'df-saip', 'usim')
+        self._is_after_if_exists(pes,'opt-csim', 'csim')
+
+    def check_mandatory_services(self, pes: ProfileElementSequence):
+        """Ensure that the PE for the mandatory services exist."""
+        m_svcs = pes.get_pe_for_type('header').decoded['eUICC-Mandatory-services']
+        if 'usim' in m_svcs and not pes.get_pe_for_type('usim'):
+            raise ProfileError('no PE-USIM for mandatory usim service')
+        if 'isim' in m_svcs and not pes.get_pe_for_type('isim'):
+            raise ProfileError('no PE-ISIM for mandatory isim service')
+        if 'csim' in m_svcs and not pes.get_pe_for_type('csim'):
+            raise ProfileError('no PE-ISIM for mandatory csim service')
+        if 'gba-usim' in m_svcs and not 'usim' in m_svcs:
+            raise ProfileError('gba-usim mandatory, but no usim')
+        if 'gba-isim' in m_svcs and not 'isim' in m_svcs:
+            raise ProfileError('gba-isim mandatory, but no isim')
+        if 'multiple-usim' in m_svcs and not 'usim' in m_svcs:
+            raise ProfileError('multiple-usim mandatory, but no usim')
+        if 'multiple-isim' in m_svcs and not 'isim' in m_svcs:
+            raise ProfileError('multiple-isim mandatory, but no isim')
+        if 'multiple-csim' in m_svcs and not 'csim' in m_svcs:
+            raise ProfileError('multiple-csim mandatory, but no csim')
+        if 'get-identity' in m_svcs and not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('get-identity mandatory, but no usim or isim')
+        if 'profile-a-x25519' in m_svcs and not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('profile-a-x25519 mandatory, but no usim or isim')
+        if 'profile-a-p256' in m_svcs and not not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('profile-a-p256 mandatory, but no usim or isim')

pySim/euicc.py

@@ -0,0 +1,516 @@
+# -*- coding: utf-8 -*-
+
+"""
+Various definitions related to GSMA eSIM / eUICC
+
+Related Specs: GSMA SGP.22, GSMA SGP.02, etc.
+"""
+
+# Copyright (C) 2023 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from pySim.tlv import *
+from pySim.construct import *
+from construct import Optional as COptional
+from construct import *
+import argparse
+from cmd2 import cmd2, CommandSet, with_default_category
+from pySim.commands import SimCardCommands
+from pySim.filesystem import CardADF, CardApplication
+from pySim.utils import Hexstr, SwHexstr
+import pySim.global_platform
+
+class VersionAdapter(Adapter):
+    """convert an EUICC Version (3-int array) to a textual representation."""
+
+    def _decode(self, obj, context, path):
+        return "%u.%u.%u" % (obj[0], obj[1], obj[2])
+
+    def _encode(self, obj, context, path):
+        return [int(x) for x in obj.split('.')]
+
+VersionType = VersionAdapter(Array(3, Int8ub))
+
+# Application Identifiers as defined in GSMA SGP.02 Annex H
+AID_ISD_R           = "A0000005591010FFFFFFFF8900000100"
+AID_ECASD           = "A0000005591010FFFFFFFF8900000200"
+AID_ISD_P_FILE      = "A0000005591010FFFFFFFF8900000D00"
+AID_ISD_P_MODULE    = "A0000005591010FFFFFFFF8900000E00"
+
+sw_isdr = {
+    'ISD-R': {
+        '6a80': 'Incorrect values in command data',
+        '6a82': 'Profile not found',
+        '6a88': 'Reference data not found',
+        '6985': 'Conditions of use not satisfied',
+    }
+}
+
+class SupportedVersionNumber(BER_TLV_IE, tag=0x82):
+    _construct = GreedyBytes
+
+class IsdrProprietaryApplicationTemplate(BER_TLV_IE, tag=0xe0, nested=[SupportedVersionNumber]):
+    # FIXME: lpaeSupport - what kind of tag  would it have?
+    pass
+
+# GlobalPlatform 2.1.1 Section 9.9.3.1 from pySim/global_platform.py extended with E0
+class FciTemplate(BER_TLV_IE, tag=0x6f, nested=pySim.global_platform.FciTemplateNestedList +
+                                               [IsdrProprietaryApplicationTemplate]):
+    pass
+
+
+# SGP.22 Section 5.7.3: GetEuiccConfiguredAddresses
+class DefaultDpAddress(BER_TLV_IE, tag=0x80):
+    _construct = Utf8Adapter(GreedyBytes)
+class RootDsAddress(BER_TLV_IE, tag=0x81):
+    _construct = Utf8Adapter(GreedyBytes)
+class EuiccConfiguredAddresses(BER_TLV_IE, tag=0xbf3c, nested=[DefaultDpAddress, RootDsAddress]):
+    pass
+
+# SGP.22 Section 5.7.4: SetDefaultDpAddress
+class SetDefaultDpAddrRes(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, undefinedError=127)
+class SetDefaultDpAddress(BER_TLV_IE, tag=0xbf3f, nested=[DefaultDpAddress, SetDefaultDpAddrRes]):
+    pass
+
+# SGP.22 Section 5.7.7: GetEUICCChallenge
+class EuiccChallenge(BER_TLV_IE, tag=0x80):
+    _construct = HexAdapter(Bytes(16))
+class GetEuiccChallenge(BER_TLV_IE, tag=0xbf2e, nested=[EuiccChallenge]):
+    pass
+
+# SGP.22 Section 5.7.8: GetEUICCInfo
+class SVN(BER_TLV_IE, tag=0x82):
+    _construct = VersionType
+class SubjectKeyIdentifier(BER_TLV_IE, tag=0x04):
+    _construct = HexAdapter(GreedyBytes)
+class EuiccCiPkiListForVerification(BER_TLV_IE, tag=0xa9, nested=[SubjectKeyIdentifier]):
+    pass
+class EuiccCiPkiListForSigning(BER_TLV_IE, tag=0xaa, nested=[SubjectKeyIdentifier]):
+    pass
+class EuiccInfo1(BER_TLV_IE, tag=0xbf20, nested=[SVN, EuiccCiPkiListForVerification, EuiccCiPkiListForSigning]):
+    pass
+class ProfileVersion(BER_TLV_IE, tag=0x81):
+    _construct = VersionType
+class EuiccFirmwareVer(BER_TLV_IE, tag=0x83):
+    _construct = VersionType
+class ExtCardResource(BER_TLV_IE, tag=0x84):
+    _construct = HexAdapter(GreedyBytes)
+class UiccCapability(BER_TLV_IE, tag=0x85):
+    _construct = HexAdapter(GreedyBytes) # FIXME
+class TS102241Version(BER_TLV_IE, tag=0x86):
+    _construct = VersionType
+class GlobalPlatformVersion(BER_TLV_IE, tag=0x87):
+    _construct = VersionType
+class RspCapability(BER_TLV_IE, tag=0x88):
+    _construct = HexAdapter(GreedyBytes) # FIXME
+class EuiccCategory(BER_TLV_IE, tag=0x8b):
+    _construct = Enum(Int8ub, other=0, basicEuicc=1, mediumEuicc=2, contactlessEuicc=3)
+class PpVersion(BER_TLV_IE, tag=0x04):
+    _construct = VersionType
+class SsAcreditationNumber(BER_TLV_IE, tag=0x0c):
+    _construct = Utf8Adapter(GreedyBytes)
+class IpaMode(BER_TLV_IE, tag=0x90):    # see SGP.32 v1.0
+    _construct = Enum(Int8ub, ipad=0, ipea=1)
+class IotVersion(BER_TLV_IE, tag=0x80): # see SGP.32 v1.0
+    _construct = VersionType
+class IotVersionSeq(BER_TLV_IE, tag=0xa0, nested=[IotVersion]): # see SGP.32 v1.0
+    pass
+class IotSpecificInfo(BER_TLV_IE, tag=0x94, nested=[IotVersionSeq]): # see SGP.32 v1.0
+    pass
+class EuiccInfo2(BER_TLV_IE, tag=0xbf22, nested=[ProfileVersion, SVN, EuiccFirmwareVer, ExtCardResource,
+                                                 UiccCapability, TS102241Version, GlobalPlatformVersion,
+                                                 RspCapability, EuiccCiPkiListForVerification,
+                                                 EuiccCiPkiListForSigning, EuiccCategory, PpVersion,
+                                                 SsAcreditationNumber, IpaMode, IotSpecificInfo]):
+    pass
+
+# SGP.22 Section 5.7.9: ListNotification
+class ProfileMgmtOperation(BER_TLV_IE, tag=0x81):
+    # we have to ignore the first byte which tells us how many padding bits are used in the last octet
+    _construct = Struct(Byte, "pmo"/FlagsEnum(Byte, install=0x80, enable=0x40, disable=0x20, delete=0x10))
+class ListNotificationReq(BER_TLV_IE, tag=0xbf28, nested=[ProfileMgmtOperation]):
+    pass
+class SeqNumber(BER_TLV_IE, tag=0x80):
+    _construct = GreedyInteger()
+class NotificationAddress(BER_TLV_IE, tag=0x0c):
+    _construct = Utf8Adapter(GreedyBytes)
+class Iccid(BER_TLV_IE, tag=0x5a):
+    _construct = BcdAdapter(GreedyBytes)
+class NotificationMetadata(BER_TLV_IE, tag=0xbf2f, nested=[SeqNumber, ProfileMgmtOperation,
+                                                           NotificationAddress, Iccid]):
+    pass
+class NotificationMetadataList(BER_TLV_IE, tag=0xa0, nested=[NotificationMetadata]):
+    pass
+class ListNotificationsResultError(BER_TLV_IE, tag=0x81):
+    _construct = Enum(Int8ub, undefinedError=127)
+class ListNotificationResp(BER_TLV_IE, tag=0xbf28, nested=[NotificationMetadataList,
+                                                           ListNotificationsResultError]):
+    pass
+
+# SGP.22 Section 5.7.11: RemoveNotificationFromList
+class DeleteNotificationStatus(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, nothingToDelete=1, undefinedError=127)
+class NotificationSentReq(BER_TLV_IE, tag=0xbf30, nested=[SeqNumber]):
+    pass
+class NotificationSentResp(BER_TLV_IE, tag=0xbf30, nested=[DeleteNotificationStatus]):
+    pass
+
+# SGP.22 Section 5.7.12: LoadCRL: FIXME
+class LoadCRL(BER_TLV_IE, tag=0xbf35, nested=[]): # FIXME
+    pass
+
+# SGP.22 Section 5.7.15: GetProfilesInfo
+class TagList(BER_TLV_IE, tag=0x5c):
+    _construct = GreedyRange(Int8ub) # FIXME: tags could be multi-byte
+class ProfileInfoListReq(BER_TLV_IE, tag=0xbf2d, nested=[TagList]): # FIXME: SearchCriteria
+    pass
+class IsdpAid(BER_TLV_IE, tag=0x4f):
+    _construct = HexAdapter(GreedyBytes)
+class ProfileState(BER_TLV_IE, tag=0x9f70):
+    _construct = Enum(Int8ub, disabled=0, enabled=1)
+class ProfileNickname(BER_TLV_IE, tag=0x90):
+    _construct = Utf8Adapter(GreedyBytes)
+class ServiceProviderName(BER_TLV_IE, tag=0x91):
+    _construct = Utf8Adapter(GreedyBytes)
+class ProfileName(BER_TLV_IE, tag=0x92):
+    _construct = Utf8Adapter(GreedyBytes)
+class IconType(BER_TLV_IE, tag=0x93):
+    _construct = Enum(Int8ub, jpg=0, png=1)
+class Icon(BER_TLV_IE, tag=0x94):
+    _construct = GreedyBytes
+class ProfileClass(BER_TLV_IE, tag=0x95):
+    _construct = Enum(Int8ub, test=0, provisioning=1, operational=2)
+class ProfileInfo(BER_TLV_IE, tag=0xe3, nested=[Iccid, IsdpAid, ProfileState, ProfileNickname,
+                                                ServiceProviderName, ProfileName, IconType, Icon,
+                                                ProfileClass]): # FIXME: more IEs
+    pass
+class ProfileInfoSeq(BER_TLV_IE, tag=0xa0, nested=[ProfileInfo]):
+    pass
+class ProfileInfoListError(BER_TLV_IE, tag=0x81):
+    _construct = Enum(Int8ub, incorrectInputValues=1, undefinedError=2)
+class ProfileInfoListResp(BER_TLV_IE, tag=0xbf2d, nested=[ProfileInfoSeq, ProfileInfoListError]):
+    pass
+
+# SGP.22 Section 5.7.16:: EnableProfile
+class RefreshFlag(BER_TLV_IE, tag=0x81): # FIXME
+    _construct = Int8ub # FIXME
+class EnableResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInDisabledState=2,
+                      disallowedByPolicy=3, wrongProfileReenabling=4, catBusy=5, undefinedError=127)
+class ProfileIdentifier(BER_TLV_IE, tag=0xa0, nested=[IsdpAid, Iccid]):
+    pass
+class EnableProfileReq(BER_TLV_IE, tag=0xbf31, nested=[ProfileIdentifier, RefreshFlag]):
+    pass
+class EnableProfileResp(BER_TLV_IE, tag=0xbf31, nested=[EnableResult]):
+    pass
+
+# SGP.22 Section 5.7.17 DisableProfile
+class DisableResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInEnabledState=2,
+                      disallowedByPolicy=3, catBusy=5, undefinedError=127)
+class DisableProfileReq(BER_TLV_IE, tag=0xbf32, nested=[ProfileIdentifier, RefreshFlag]):
+    pass
+class DisableProfileResp(BER_TLV_IE, tag=0xbf32, nested=[DisableResult]):
+    pass
+
+# SGP.22 Section 5.7.18: DeleteProfile
+class DeleteResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInDisabledState=2,
+                      disallowedByPolicy=3, undefinedError=127)
+class DeleteProfileReq(BER_TLV_IE, tag=0xbf33, nested=[IsdpAid, Iccid]):
+    pass
+class DeleteProfileResp(BER_TLV_IE, tag=0xbf33, nested=[DeleteResult]):
+    pass
+
+# SGP.22 Section 5.7.20 GetEID
+class EidValue(BER_TLV_IE, tag=0x5a):
+    _construct = HexAdapter(GreedyBytes)
+class GetEuiccData(BER_TLV_IE, tag=0xbf3e, nested=[TagList, EidValue]):
+    pass
+
+# SGP.22 Section 5.7.21: ES10c SetNickname
+class SnrProfileNickname(BER_TLV_IE, tag=0x8f):
+    _construct = Utf8Adapter(GreedyBytes)
+class SetNicknameReq(BER_TLV_IE, tag=0xbf29, nested=[Iccid, SnrProfileNickname]):
+    pass
+class SetNicknameResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidNotFound=1, undefinedError=127)
+class SetNicknameResp(BER_TLV_IE, tag=0xbf29, nested=[SetNicknameResult]):
+    pass
+
+# SGP.32 Section 5.9.10: ES10b: GetCerts
+class GetCertsReq(BER_TLV_IE, tag=0xbf56):
+    pass
+class EumCertificate(BER_TLV_IE, tag=0xa5):
+    _construct = GreedyBytes
+class EuiccCertificate(BER_TLV_IE, tag=0xa6):
+    _construct = GreedyBytes
+class GetCertsError(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, invalidCiPKId=1, undefinedError=127)
+class GetCertsResp(BER_TLV_IE, tag=0xbf56, nested=[EumCertificate, EuiccCertificate, GetCertsError]):
+    pass
+
+# SGP.32 Section 5.9.18: ES10b: GetEimConfigurationData
+class EimId(BER_TLV_IE, tag=0x80):
+    _construct = Utf8Adapter(GreedyBytes)
+class EimFqdn(BER_TLV_IE, tag=0x81):
+    _construct = Utf8Adapter(GreedyBytes)
+class EimIdType(BER_TLV_IE, tag=0x82):
+    _construct = Enum(Int8ub, eimIdTypeOid=1, eimIdTypeFqdn=2, eimIdTypeProprietary=3)
+class CounterValue(BER_TLV_IE, tag=0x83):
+    _construct = GreedyInteger
+class AssociationToken(BER_TLV_IE, tag=0x84):
+    _construct = GreedyInteger
+class EimSupportedProtocol(BER_TLV_IE, tag=0x87):
+    _construct = Enum(Int8ub, eimRetrieveHttps=0, eimRetrieveCoaps=1, eimInjectHttps=2, eimInjectCoaps=3,
+                      eimProprietary=4)
+# FIXME: eimPublicKeyData, trustedPublicKeyDataTls, euiccCiPKId
+class EimConfigurationData(BER_TLV_IE, tag=0x80, nested=[EimId, EimFqdn, EimIdType, CounterValue,
+                                                         AssociationToken, EimSupportedProtocol]):
+    pass
+class EimConfigurationDataSeq(BER_TLV_IE, tag=0xa0, nested=[EimConfigurationData]):
+    pass
+class GetEimConfigurationData(BER_TLV_IE, tag=0xbf55, nested=[EimConfigurationDataSeq]):
+    pass
+
+class ADF_ISDR(CardADF):
+    def __init__(self, aid=AID_ISD_R, name='ADF.ISD-R', fid=None, sfid=None,
+                 desc='ISD-R (Issuer Security Domain Root) Application'):
+        super().__init__(aid=aid, fid=fid, sfid=sfid, name=name, desc=desc)
+        self.shell_commands += [self.AddlShellCommands()]
+
+    @staticmethod
+    def store_data(scc: SimCardCommands, tx_do: Hexstr) -> Tuple[Hexstr, SwHexstr]:
+        """Perform STORE DATA according to Table 47+48 in Section 5.7.2 of SGP.22.
+        Only single-block store supported for now."""
+        capdu = '%sE29100%02x%s' % (scc.cla4lchan('80'), len(tx_do)//2, tx_do)
+        return scc._tp.send_apdu_checksw(capdu)
+
+    @staticmethod
+    def store_data_tlv(scc: SimCardCommands, cmd_do, resp_cls, exp_sw='9000'):
+        """Transceive STORE DATA APDU with the card, transparently encoding the command data from TLV
+        and decoding the response data tlv."""
+        if cmd_do:
+            cmd_do_enc = cmd_do.to_tlv()
+            cmd_do_len = len(cmd_do_enc)
+            if cmd_do_len > 255:
+                return ValueError('DO > 255 bytes not supported yet')
+        else:
+            cmd_do_enc = b''
+        (data, sw) = ADF_ISDR.store_data(scc, b2h(cmd_do_enc))
+        if data:
+            if resp_cls:
+                resp_do = resp_cls()
+                resp_do.from_tlv(h2b(data))
+                return resp_do
+            else:
+                return data
+        else:
+            return None
+
+    def decode_select_response(self, data_hex: Hexstr) -> object:
+        t = FciTemplate()
+        t.from_tlv(h2b(data_hex))
+        d = t.to_dict()
+        return flatten_dict_lists(d['fci_template'])
+
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+
+        es10x_store_data_parser = argparse.ArgumentParser()
+        es10x_store_data_parser.add_argument('TX_DO', help='Hexstring of encoded to-be-transmitted DO')
+
+        @cmd2.with_argparser(es10x_store_data_parser)
+        def do_es10x_store_data(self, opts):
+            """Perform a raw STORE DATA command as defined for the ES10x eUICC interface."""
+            (data, sw) = ADF_ISDR.store_data(self._cmd.lchan.scc, opts.TX_DO)
+
+        def do_get_euicc_configured_addresses(self, opts):
+            """Perform an ES10a GetEuiccConfiguredAddresses function."""
+            eca = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, EuiccConfiguredAddresses(), EuiccConfiguredAddresses)
+            d = eca.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_configured_addresses']))
+
+        set_def_dp_addr_parser = argparse.ArgumentParser()
+        set_def_dp_addr_parser.add_argument('DP_ADDRESS', help='Default SM-DP+ address as UTF-8 string')
+
+        @cmd2.with_argparser(set_def_dp_addr_parser)
+        def do_set_default_dp_address(self, opts):
+            """Perform an ES10a SetDefaultDpAddress function."""
+            sdda_cmd = SetDefaultDpAddress(children=[DefaultDpAddress(decoded=opts.DP_ADDRESS)])
+            sdda = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, sdda_cmd, SetDefaultDpAddress)
+            d = sdda.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['set_default_dp_address']))
+
+        def do_get_euicc_challenge(self, opts):
+            """Perform an ES10b GetEUICCChallenge function."""
+            gec = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, GetEuiccChallenge(), GetEuiccChallenge)
+            d = gec.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_euicc_challenge']))
+
+        def do_get_euicc_info1(self, opts):
+            """Perform an ES10b GetEUICCInfo (1) function."""
+            ei1 = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, EuiccInfo1(), EuiccInfo1)
+            d = ei1.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_info1']))
+
+        def do_get_euicc_info2(self, opts):
+            """Perform an ES10b GetEUICCInfo (2) function."""
+            ei2 = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, EuiccInfo2(), EuiccInfo2)
+            d = ei2.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_info2']))
+
+        def do_list_notification(self, opts):
+            """Perform an ES10b ListNotification function."""
+            ln = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, ListNotificationReq(), ListNotificationResp)
+            d = ln.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['list_notification_resp']))
+
+        rem_notif_parser = argparse.ArgumentParser()
+        rem_notif_parser.add_argument('SEQ_NR', type=int, help='Sequence Number of the to-be-removed notification')
+
+        @cmd2.with_argparser(rem_notif_parser)
+        def do_remove_notification_from_list(self, opts):
+            """Perform an ES10b RemoveNotificationFromList function."""
+            rn_cmd = NotificationSentReq(children=[SeqNumber(decoded=opts.SEQ_NR)])
+            rn = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, rn_cmd, NotificationSentResp)
+            d = rn.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['notification_sent_resp']))
+
+        def do_get_profiles_info(self, opts):
+            """Perform an ES10c GetProfilesInfo function."""
+            pi = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, ProfileInfoListReq(), ProfileInfoListResp)
+            d = pi.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['profile_info_list_resp']))
+
+        en_prof_parser = argparse.ArgumentParser()
+        en_prof_grp = en_prof_parser.add_mutually_exclusive_group()
+        en_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        en_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+        en_prof_parser.add_argument('--refresh-required', action='store_true', help='whether a REFRESH is required')
+
+        @cmd2.with_argparser(en_prof_parser)
+        def do_enable_profile(self, opts):
+            """Perform an ES10c EnableProfile function."""
+            if opts.isdp_aid:
+                p_id = ProfileIdentifier(children=[IsdpAid(decoded=opts.isdp_aid)])
+            if opts.iccid:
+                p_id = ProfileIdentifier(children=[Iccid(decoded=opts.iccid)])
+            ep_cmd_contents = [p_id, RefreshFlag(decoded=opts.refresh_required)]
+            ep_cmd = EnableProfileReq(children=ep_cmd_contents)
+            ep = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, ep_cmd, EnableProfileResp)
+            d = ep.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['enable_profile_resp']))
+
+        dis_prof_parser = argparse.ArgumentParser()
+        dis_prof_grp = dis_prof_parser.add_mutually_exclusive_group()
+        dis_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        dis_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+        dis_prof_parser.add_argument('--refresh-required', action='store_true', help='whether a REFRESH is required')
+
+        @cmd2.with_argparser(dis_prof_parser)
+        def do_disable_profile(self, opts):
+            """Perform an ES10c DisableProfile function."""
+            if opts.isdp_aid:
+                p_id = ProfileIdentifier(children=[IsdpAid(decoded=opts.isdp_aid)])
+            if opts.iccid:
+                p_id = ProfileIdentifier(children=[Iccid(decoded=opts.iccid)])
+            dp_cmd_contents = [p_id, RefreshFlag(decoded=opts.refresh_required)]
+            dp_cmd = DisableProfileReq(children=dp_cmd_contents)
+            dp = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, dp_cmd, DisableProfileResp)
+            d = dp.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['disable_profile_resp']))
+
+        del_prof_parser = argparse.ArgumentParser()
+        del_prof_grp = del_prof_parser.add_mutually_exclusive_group()
+        del_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        del_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+
+        @cmd2.with_argparser(del_prof_parser)
+        def do_delete_profile(self, opts):
+            """Perform an ES10c DeleteProfile function."""
+            if opts.isdp_aid:
+                p_id = IsdpAid(decoded=opts.isdp_aid)
+            if opts.iccid:
+                p_id = Iccid(decoded=opts.iccid)
+            dp_cmd_contents = [p_id]
+            dp_cmd = DeleteProfileReq(children=dp_cmd_contents)
+            dp = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, dp_cmd, DeleteProfileResp)
+            d = dp.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['delete_profile_resp']))
+
+
+        def do_get_eid(self, opts):
+            """Perform an ES10c GetEID function."""
+            (data, sw) = ADF_ISDR.store_data(self._cmd.lchan.scc, 'BF3E035C015A')
+            ged_cmd = GetEuiccData(children=[TagList(decoded=[0x5A])])
+            ged = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, ged_cmd, GetEuiccData)
+            d = ged.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_euicc_data']))
+
+        set_nickname_parser = argparse.ArgumentParser()
+        set_nickname_parser.add_argument('ICCID', help='ICCID of the profile whose nickname to set')
+        set_nickname_parser.add_argument('--profile-nickname', help='Nickname of the profile')
+
+        @cmd2.with_argparser(set_nickname_parser)
+        def do_set_nickname(self, opts):
+            """Perform an ES10c SetNickname function."""
+            nickname = opts.profile_nickname or ''
+            sn_cmd_contents = [Iccid(decoded=opts.ICCID), ProfileNickname(decoded=nickname)]
+            sn_cmd = SetNicknameReq(children=sn_cmd_contents)
+            sn = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, sn_cmd, SetNicknameResp)
+            d = sn.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['set_nickname_resp']))
+
+        def do_get_certs(self, opts):
+            """Perform an ES10c GetCerts() function on an IoT eUICC."""
+            gc = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, GetCertsReq(), GetCertsResp)
+            d = gc.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_certficiates_resp']))
+
+        def do_get_eim_configuration_data(self, opts):
+            """Perform an ES10b GetEimConfigurationData function on an Iot eUICC."""
+            gec = ADF_ISDR.store_data_tlv(self._cmd.lchan.scc, GetEimConfigurationData(),
+                                          GetEimConfigurationData)
+            d = gec.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_eim_configuration_data']))
+
+
+class ADF_ECASD(CardADF):
+    def __init__(self, aid=AID_ECASD, name='ADF.ECASD', fid=None, sfid=None,
+                 desc='ECASD (eUICC Controlling Authority Security Domain) Application'):
+        super().__init__(aid=aid, fid=fid, sfid=sfid, name=name, desc=desc)
+        self.shell_commands += [self.AddlShellCommands()]
+
+    def decode_select_response(self, data_hex: Hexstr) -> object:
+        t = FciTemplate()
+        t.from_tlv(h2b(data_hex))
+        d = t.to_dict()
+        return flatten_dict_lists(d['fci_template'])
+
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        pass
+
+
+
+class CardApplicationISDR(CardApplication):
+    def __init__(self):
+        super().__init__('ISD-R', adf=ADF_ISDR(), sw=sw_isdr)
+
+class CardApplicationECASD(CardApplication):
+    def __init__(self):
+        super().__init__('ECASD', adf=ADF_ECASD(), sw=sw_isdr)

pySim/filesystem.py

@@ -38,8 +38,8 @@ from typing import cast, Optional, Iterable, List, Dict, Tuple, Union
 
 from smartcard.util import toBytes
 
-from pySim.utils import sw_match, h2b, b2h, i2h, is_hex, auto_int, bertlv_parse_one, Hexstr
-from pySim.construct import filter_dict, parse_construct
+from pySim.utils import sw_match, h2b, b2h, i2h, is_hex, auto_int, Hexstr
+from pySim.construct import filter_dict, parse_construct, build_construct
 from pySim.exceptions import *
 from pySim.jsonpath import js_path_find, js_path_modify
 from pySim.commands import SimCardCommands
@@ -51,18 +51,6 @@ CardFileService = Union[int, List[int], Tuple[int, ...]]
 
 Size = Tuple[int, Optional[int]]
 
-def lchan_nr_from_cla(cla: int) -> int:
-    """Resolve the logical channel number from the CLA byte."""
-    # TS 102 221 10.1.1 Coding of Class Byte
-    if cla >> 4 in [0x0, 0xA, 0x8]:
-        # Table 10.3
-        return cla & 0x03
-    elif cla & 0xD0 in [0x40, 0xC0]:
-        # Table 10.4a
-        return 4 + (cla & 0x0F)
-    else:
-        raise ValueError('Could not determine logical channel for CLA=%2X' % cla)
-
 class CardFile:
     """Base class for all objects in the smart card filesystem.
     Serve as a common ancestor to all other file types; rarely used directly.
@@ -148,11 +136,23 @@ class CardFile:
         return ret
 
     def build_select_path_to(self, target: 'CardFile') -> Optional[List['CardFile']]:
+
+        # special-case handling for applications. Applications may be selected
+        # any time from any location. If there is an ADF somewhere in the path,
+        # we may clip everything before that ADF.
+        def clip_path(inter_path):
+            for i in reversed(range(0, len(inter_path))):
+                if isinstance(inter_path[i], CardADF):
+                    return inter_path[i:]
+            return inter_path
+
         """Build the relative sequence of files we need to traverse to get from us to 'target'."""
+        # special-case handling for selecting MF while the MF is selected
+        if target == target.get_mf():
+            return [target]
         cur_fqpath = self.fully_qualified_path_fobj()
         target_fqpath = target.fully_qualified_path_fobj()
         inter_path = []
-        cur_fqpath.pop()  # drop last element (currently selected file, doesn't need re-selection
         cur_fqpath.reverse()
         for ce in cur_fqpath:
             inter_path.append(ce)
@@ -162,7 +162,7 @@ class CardFile:
                     for te2 in target_fqpath[i+1:]:
                         inter_path.append(te2)
                     # we found our common ancestor
-                    return inter_path
+                    return clip_path(inter_path[1:])
         return None
 
     def get_mf(self) -> Optional['CardMF']:
@@ -514,11 +514,12 @@ class CardMF(CardDF):
 class CardADF(CardDF):
     """ADF (Application Dedicated File) in the smart card filesystem"""
 
-    def __init__(self, aid: str, **kwargs):
+    def __init__(self, aid: str, has_fs: bool=False, **kwargs):
         super().__init__(**kwargs)
         # reference to CardApplication may be set from CardApplication constructor
         self.application = None  # type: Optional[CardApplication]
         self.aid = aid           # Application Identifier
+        self.has_fs = has_fs     # Flag to tell whether the ADF supports a filesystem or not
         mf = self.get_mf()
         if mf:
             mf.add_application_df(self)
@@ -649,7 +650,7 @@ class TransparentEF(CardEF):
                 with open(filename, 'w') as text_file:
                     json.dump(orig_json, text_file, indent=4)
                 # run a text editor
-                self._cmd._run_editor(filename)
+                self._cmd.run_editor(filename)
                 with open(filename, 'r') as text_file:
                     edited_json = json.load(text_file)
                 if edited_json == orig_json:
@@ -748,7 +749,7 @@ class TransparentEF(CardEF):
         if callable(method):
             return h2b(method(abstract_data))
         if self._construct:
-            return self._construct.build(abstract_data)
+            return build_construct(self._construct, abstract_data)
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -776,7 +777,7 @@ class TransparentEF(CardEF):
             raw_bin_data = method(abstract_data)
             return b2h(raw_bin_data)
         if self._construct:
-            return b2h(self._construct.build(abstract_data))
+            return b2h(build_construct(self._construct, abstract_data))
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -917,7 +918,7 @@ class LinFixedEF(CardEF):
                 with open(filename, 'w') as text_file:
                     json.dump(orig_json, text_file, indent=4)
                 # run a text editor
-                self._cmd._run_editor(filename)
+                self._cmd.run_editor(filename)
                 with open(filename, 'r') as text_file:
                     edited_json = json.load(text_file)
                 if edited_json == orig_json:
@@ -929,7 +930,7 @@ class LinFixedEF(CardEF):
                         self._cmd.poutput_json(data)
 
     def __init__(self, fid: str, sfid: str = None, name: str = None, desc: str = None,
-                 parent: Optional[CardDF] = None, rec_len: Size = (1, None), **kwargs):
+                 parent: Optional[CardDF] = None, rec_len: Size = (1, None), leftpad: bool = False, **kwargs):
         """
         Args:
             fid : File Identifier (4 hex digits)
@@ -938,14 +939,16 @@ class LinFixedEF(CardEF):
             desc : Description of the file
             parent : Parent CardFile object within filesystem hierarchy
             rec_len : Tuple of (minimum_length, recommended_length)
+            leftpad: On write, data must be padded from the left to fit pysical record length
         """
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, parent=parent, **kwargs)
         self.rec_len = rec_len
+        self.leftpad = leftpad
         self.shell_commands = [self.ShellCommands()]
         self._construct = None
         self._tlv = None
 
-    def decode_record_hex(self, raw_hex_data: str) -> dict:
+    def decode_record_hex(self, raw_hex_data: str, record_nr: int = 1) -> dict:
         """Decode raw (hex string) data into abstract representation.
 
         A derived class would typically provide a _decode_record_bin() or _decode_record_hex()
@@ -954,16 +957,17 @@ class LinFixedEF(CardEF):
 
         Args:
             raw_hex_data : hex-encoded data
+            record_nr : record number (1 for first record, ...)
         Returns:
             abstract_data; dict representing the decoded data
         """
         method = getattr(self, '_decode_record_hex', None)
         if callable(method):
-            return method(raw_hex_data)
+            return method(raw_hex_data, record_nr=record_nr)
         raw_bin_data = h2b(raw_hex_data)
         method = getattr(self, '_decode_record_bin', None)
         if callable(method):
-            return method(raw_bin_data)
+            return method(raw_bin_data, record_nr=record_nr)
         if self._construct:
             return parse_construct(self._construct, raw_bin_data)
         elif self._tlv:
@@ -972,7 +976,7 @@ class LinFixedEF(CardEF):
             return t.to_dict()
         return {'raw': raw_bin_data.hex()}
 
-    def decode_record_bin(self, raw_bin_data: bytearray) -> dict:
+    def decode_record_bin(self, raw_bin_data: bytearray, record_nr: int) -> dict:
         """Decode raw (binary) data into abstract representation.
 
         A derived class would typically provide a _decode_record_bin() or _decode_record_hex()
@@ -981,16 +985,17 @@ class LinFixedEF(CardEF):
 
         Args:
             raw_bin_data : binary encoded data
+            record_nr : record number (1 for first record, ...)
         Returns:
             abstract_data; dict representing the decoded data
         """
         method = getattr(self, '_decode_record_bin', None)
         if callable(method):
-            return method(raw_bin_data)
+            return method(raw_bin_data, record_nr=record_nr)
         raw_hex_data = b2h(raw_bin_data)
         method = getattr(self, '_decode_record_hex', None)
         if callable(method):
-            return method(raw_hex_data)
+            return method(raw_hex_data, record_nr=record_nr)
         if self._construct:
             return parse_construct(self._construct, raw_bin_data)
         elif self._tlv:
@@ -999,7 +1004,7 @@ class LinFixedEF(CardEF):
             return t.to_dict()
         return {'raw': raw_hex_data}
 
-    def encode_record_hex(self, abstract_data: dict) -> str:
+    def encode_record_hex(self, abstract_data: dict, record_nr: int) -> str:
         """Encode abstract representation into raw (hex string) data.
 
         A derived class would typically provide an _encode_record_bin() or _encode_record_hex()
@@ -1008,18 +1013,19 @@ class LinFixedEF(CardEF):
 
         Args:
             abstract_data : dict representing the decoded data
+            record_nr : record number (1 for first record, ...)
         Returns:
             hex string encoded data
         """
         method = getattr(self, '_encode_record_hex', None)
         if callable(method):
-            return method(abstract_data)
+            return method(abstract_data, record_nr=record_nr)
         method = getattr(self, '_encode_record_bin', None)
         if callable(method):
-            raw_bin_data = method(abstract_data)
+            raw_bin_data = method(abstract_data, record_nr=record_nr)
             return b2h(raw_bin_data)
         if self._construct:
-            return b2h(self._construct.build(abstract_data))
+            return b2h(build_construct(self._construct, abstract_data))
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -1027,7 +1033,7 @@ class LinFixedEF(CardEF):
         raise NotImplementedError(
             "%s encoder not yet implemented. Patches welcome." % self)
 
-    def encode_record_bin(self, abstract_data: dict) -> bytearray:
+    def encode_record_bin(self, abstract_data: dict, record_nr : int) -> bytearray:
         """Encode abstract representation into raw (binary) data.
 
         A derived class would typically provide an _encode_record_bin() or _encode_record_hex()
@@ -1036,17 +1042,18 @@ class LinFixedEF(CardEF):
 
         Args:
             abstract_data : dict representing the decoded data
+            record_nr : record number (1 for first record, ...)
         Returns:
             binary encoded data
         """
         method = getattr(self, '_encode_record_bin', None)
         if callable(method):
-            return method(abstract_data)
+            return method(abstract_data, record_nr=record_nr)
         method = getattr(self, '_encode_record_hex', None)
         if callable(method):
-            return h2b(method(abstract_data))
+            return h2b(method(abstract_data, record_nr=record_nr))
         if self._construct:
-            return self._construct.build(abstract_data)
+            return build_construct(self._construct, abstract_data)
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -1162,7 +1169,7 @@ class TransRecEF(TransparentEF):
         if callable(method):
             return b2h(method(abstract_data))
         if self._construct:
-            return b2h(filter_dict(self._construct.build(abstract_data)))
+            return b2h(filter_dict(build_construct(self._construct, abstract_data)))
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -1189,7 +1196,7 @@ class TransRecEF(TransparentEF):
         if callable(method):
             return h2b(method(abstract_data))
         if self._construct:
-            return filter_dict(self._construct.build(abstract_data))
+            return filter_dict(build_construct(self._construct, abstract_data))
         elif self._tlv:
             t = self._tlv() if inspect.isclass(self._tlv) else self._tlv
             t.from_dict(abstract_data)
@@ -1278,484 +1285,6 @@ class BerTlvEF(CardEF):
         self.size = size
         self.shell_commands = [self.ShellCommands()]
 
-
-class RuntimeState:
-    """Represent the runtime state of a session with a card."""
-
-    def __init__(self, card, profile: 'CardProfile'):
-        """
-        Args:
-            card : pysim.cards.Card instance
-            profile : CardProfile instance
-        """
-        self.mf = CardMF(profile=profile)
-        self.card = card
-        self.profile = profile
-        self.lchan = {}
-        # the basic logical channel always exists
-        self.lchan[0] = RuntimeLchan(0, self)
-
-        # make sure the class and selection control bytes, which are specified
-        # by the card profile are used
-        self.card.set_apdu_parameter(
-            cla=self.profile.cla, sel_ctrl=self.profile.sel_ctrl)
-
-        # add application ADFs + MF-files from profile
-        apps = self._match_applications()
-        for a in apps:
-            if a.adf:
-                self.mf.add_application_df(a.adf)
-        for f in self.profile.files_in_mf:
-            self.mf.add_file(f)
-        self.conserve_write = True
-
-        # make sure that when the runtime state is created, the card is also
-        # in a defined state.
-        self.reset()
-
-    def _match_applications(self):
-        """match the applications from the profile with applications on the card"""
-        apps_profile = self.profile.applications
-
-        # When the profile does not feature any applications, then we are done already
-        if not apps_profile:
-            return []
-
-        # Read AIDs from card and match them against the applications defined by the
-        # card profile
-        aids_card = self.card.read_aids()
-        apps_taken = []
-        if aids_card:
-            aids_taken = []
-            print("AIDs on card:")
-            for a in aids_card:
-                for f in apps_profile:
-                    if f.aid in a:
-                        print(" %s: %s (EF.DIR)" % (f.name, a))
-                        aids_taken.append(a)
-                        apps_taken.append(f)
-            aids_unknown = set(aids_card) - set(aids_taken)
-            for a in aids_unknown:
-                print(" unknown: %s (EF.DIR)" % a)
-        else:
-            print("warning: EF.DIR seems to be empty!")
-
-        # Some card applications may not be registered in EF.DIR, we will actively
-        # probe for those applications
-        for f in set(apps_profile) - set(apps_taken):
-            try:
-                data, sw = self.card.select_adf_by_aid(f.aid)
-                if sw == "9000":
-                    print(" %s: %s" % (f.name, f.aid))
-                    apps_taken.append(f)
-            except SwMatchError:
-                pass
-        return apps_taken
-
-    def reset(self, cmd_app=None) -> Hexstr:
-        """Perform physical card reset and obtain ATR.
-        Args:
-            cmd_app : Command Application State (for unregistering old file commands)
-        """
-        # delete all lchan != 0 (basic lchan)
-        for lchan_nr in self.lchan.keys():
-            if lchan_nr == 0:
-                continue
-            del self.lchan[lchan_nr]
-        atr = i2h(self.card.reset())
-        # select MF to reset internal state and to verify card really works
-        self.lchan[0].select('MF', cmd_app)
-        self.lchan[0].selected_adf = None
-        return atr
-
-    def add_lchan(self, lchan_nr: int) -> 'RuntimeLchan':
-        """Add a logical channel to the runtime state.  You shouldn't call this
-        directly but always go through RuntimeLchan.add_lchan()."""
-        if lchan_nr in self.lchan.keys():
-            raise ValueError('Cannot create already-existing lchan %d' % lchan_nr)
-        self.lchan[lchan_nr] = RuntimeLchan(lchan_nr, self)
-        return self.lchan[lchan_nr]
-
-    def del_lchan(self, lchan_nr: int):
-        if lchan_nr in self.lchan.keys():
-            del self.lchan[lchan_nr]
-            return True
-        else:
-            return False
-
-    def get_lchan_by_cla(self, cla) -> Optional['RuntimeLchan']:
-        lchan_nr = lchan_nr_from_cla(cla)
-        if lchan_nr in self.lchan.keys():
-            return self.lchan[lchan_nr]
-        else:
-            return None
-
-
-class RuntimeLchan:
-    """Represent the runtime state of a logical channel with a card."""
-
-    def __init__(self, lchan_nr: int, rs: RuntimeState):
-        self.lchan_nr = lchan_nr
-        self.rs = rs
-        self.selected_file = self.rs.mf
-        self.selected_adf = None
-        self.selected_file_fcp = None
-        self.selected_file_fcp_hex = None
-
-    def add_lchan(self, lchan_nr: int) -> 'RuntimeLchan':
-        """Add a new logical channel from the current logical channel. Just affects
-        internal state, doesn't actually open a channel with the UICC."""
-        new_lchan = self.rs.add_lchan(lchan_nr)
-        # See TS 102 221 Table 8.3
-        if self.lchan_nr != 0:
-            new_lchan.selected_file = self.get_cwd()
-            new_lchan.selected_adf = self.selected_adf
-        return new_lchan
-
-    def selected_file_descriptor_byte(self) -> dict:
-        return self.selected_file_fcp['file_descriptor']['file_descriptor_byte']
-
-    def selected_file_shareable(self) -> bool:
-        return self.selected_file_descriptor_byte()['shareable']
-
-    def selected_file_structure(self) -> str:
-        return self.selected_file_descriptor_byte()['structure']
-
-    def selected_file_type(self) -> str:
-        return self.selected_file_descriptor_byte()['file_type']
-
-    def selected_file_num_of_rec(self) -> Optional[int]:
-        return self.selected_file_fcp['file_descriptor'].get('num_of_rec')
-
-    def get_cwd(self) -> CardDF:
-        """Obtain the current working directory.
-
-        Returns:
-            CardDF instance
-        """
-        if isinstance(self.selected_file, CardDF):
-            return self.selected_file
-        else:
-            return self.selected_file.parent
-
-    def get_application_df(self) -> Optional[CardADF]:
-        """Obtain the currently selected application DF (if any).
-
-        Returns:
-            CardADF() instance or None"""
-        # iterate upwards from selected file; check if any is an ADF
-        node = self.selected_file
-        while node.parent != node:
-            if isinstance(node, CardADF):
-                return node
-            node = node.parent
-        return None
-
-    def interpret_sw(self, sw: str):
-        """Interpret a given status word relative to the currently selected application
-        or the underlying card profile.
-
-        Args:
-            sw : Status word as string of 4 hex digits
-
-        Returns:
-            Tuple of two strings
-        """
-        res = None
-        adf = self.get_application_df()
-        if adf:
-            app = adf.application
-            # The application either comes with its own interpret_sw
-            # method or we will use the interpret_sw method from the
-            # card profile.
-            if app and hasattr(app, "interpret_sw"):
-                res = app.interpret_sw(sw)
-        return res or self.rs.profile.interpret_sw(sw)
-
-    def probe_file(self, fid: str, cmd_app=None):
-        """Blindly try to select a file and automatically add a matching file
-               object if the file actually exists."""
-        if not is_hex(fid, 4, 4):
-            raise ValueError(
-                "Cannot select unknown file by name %s, only hexadecimal 4 digit FID is allowed" % fid)
-
-        try:
-            (data, sw) = self.rs.card._scc.select_file(fid)
-        except SwMatchError as swm:
-            k = self.interpret_sw(swm.sw_actual)
-            if not k:
-                raise(swm)
-            raise RuntimeError("%s: %s - %s" % (swm.sw_actual, k[0], k[1]))
-
-        select_resp = self.selected_file.decode_select_response(data)
-        if (select_resp['file_descriptor']['file_descriptor_byte']['file_type'] == 'df'):
-            f = CardDF(fid=fid, sfid=None, name="DF." + str(fid).upper(),
-                       desc="dedicated file, manually added at runtime")
-        else:
-            if (select_resp['file_descriptor']['file_descriptor_byte']['structure'] == 'transparent'):
-                f = TransparentEF(fid=fid, sfid=None, name="EF." + str(fid).upper(),
-                                  desc="elementary file, manually added at runtime")
-            else:
-                f = LinFixedEF(fid=fid, sfid=None, name="EF." + str(fid).upper(),
-                               desc="elementary file, manually added at runtime")
-
-        self.selected_file.add_files([f])
-        self.selected_file = f
-        return select_resp, data
-
-    def _select_pre(self, cmd_app):
-        # unregister commands of old file
-        if cmd_app and self.selected_file.shell_commands:
-            for c in self.selected_file.shell_commands:
-                cmd_app.unregister_command_set(c)
-
-    def _select_post(self, cmd_app):
-        # register commands of new file
-        if cmd_app and self.selected_file.shell_commands:
-            for c in self.selected_file.shell_commands:
-                cmd_app.register_command_set(c)
-
-    def select_file(self, file: CardFile, cmd_app=None):
-        """Select a file (EF, DF, ADF, MF, ...).
-
-        Args:
-            file : CardFile [or derived class] instance
-            cmd_app : Command Application State (for unregistering old file commands)
-        """
-        # we need to find a path from our self.selected_file to the destination
-        inter_path = self.selected_file.build_select_path_to(file)
-        if not inter_path:
-            raise RuntimeError('Cannot determine path from %s to %s' % (self.selected_file, file))
-
-        self._select_pre(cmd_app)
-
-        for p in inter_path:
-            try:
-                if isinstance(p, CardADF):
-                    (data, sw) = self.rs.card.select_adf_by_aid(p.aid)
-                    self.selected_adf = p
-                else:
-                    (data, sw) = self.rs.card._scc.select_file(p.fid)
-                self.selected_file = p
-            except SwMatchError as swm:
-                self._select_post(cmd_app)
-                raise(swm)
-
-        self._select_post(cmd_app)
-
-    def select(self, name: str, cmd_app=None):
-        """Select a file (EF, DF, ADF, MF, ...).
-
-        Args:
-            name : Name of file to select
-            cmd_app : Command Application State (for unregistering old file commands)
-        """
-        # handling of entire paths with multiple directories/elements
-        if '/' in name:
-            prev_sel_file = self.selected_file
-            pathlist = name.split('/')
-            # treat /DF.GSM/foo like MF/DF.GSM/foo
-            if pathlist[0] == '':
-                pathlist[0] = 'MF'
-            try:
-                for p in pathlist:
-                    self.select(p, cmd_app)
-                return
-            except Exception as e:
-                # if any intermediate step fails, go back to where we were
-                self.select_file(prev_sel_file, cmd_app)
-                raise e
-
-        sels = self.selected_file.get_selectables()
-        if is_hex(name):
-            name = name.lower()
-
-        self._select_pre(cmd_app)
-
-        if name in sels:
-            f = sels[name]
-            try:
-                if isinstance(f, CardADF):
-                    (data, sw) = self.rs.card.select_adf_by_aid(f.aid)
-                else:
-                    (data, sw) = self.rs.card._scc.select_file(f.fid)
-                self.selected_file = f
-            except SwMatchError as swm:
-                k = self.interpret_sw(swm.sw_actual)
-                if not k:
-                    raise(swm)
-                raise RuntimeError("%s: %s - %s" % (swm.sw_actual, k[0], k[1]))
-            select_resp = f.decode_select_response(data)
-        else:
-            (select_resp, data) = self.probe_file(name, cmd_app)
-
-        # store the raw + decoded FCP for later reference
-        self.selected_file_fcp_hex = data
-        self.selected_file_fcp = select_resp
-
-        self._select_post(cmd_app)
-        return select_resp
-
-    def status(self):
-        """Request STATUS (current selected file FCP) from card."""
-        (data, sw) = self.rs.card._scc.status()
-        return self.selected_file.decode_select_response(data)
-
-    def get_file_for_selectable(self, name: str):
-        sels = self.selected_file.get_selectables()
-        return sels[name]
-
-    def activate_file(self, name: str):
-        """Request ACTIVATE FILE of specified file."""
-        sels = self.selected_file.get_selectables()
-        f = sels[name]
-        data, sw = self.rs.card._scc.activate_file(f.fid)
-        return data, sw
-
-    def read_binary(self, length: int = None, offset: int = 0):
-        """Read [part of] a transparent EF binary data.
-
-        Args:
-            length : Amount of data to read (None: as much as possible)
-            offset : Offset into the file from which to read 'length' bytes
-        Returns:
-            binary data read from the file
-        """
-        if not isinstance(self.selected_file, TransparentEF):
-            raise TypeError("Only works with TransparentEF")
-        return self.rs.card._scc.read_binary(self.selected_file.fid, length, offset)
-
-    def read_binary_dec(self) -> Tuple[dict, str]:
-        """Read [part of] a transparent EF binary data and decode it.
-
-        Args:
-            length : Amount of data to read (None: as much as possible)
-            offset : Offset into the file from which to read 'length' bytes
-        Returns:
-            abstract decode data read from the file
-        """
-        (data, sw) = self.read_binary()
-        dec_data = self.selected_file.decode_hex(data)
-        return (dec_data, sw)
-
-    def update_binary(self, data_hex: str, offset: int = 0):
-        """Update transparent EF binary data.
-
-        Args:
-            data_hex : hex string of data to be written
-            offset : Offset into the file from which to write 'data_hex'
-        """
-        if not isinstance(self.selected_file, TransparentEF):
-            raise TypeError("Only works with TransparentEF")
-        return self.rs.card._scc.update_binary(self.selected_file.fid, data_hex, offset, conserve=self.rs.conserve_write)
-
-    def update_binary_dec(self, data: dict):
-        """Update transparent EF from abstract data. Encodes the data to binary and
-        then updates the EF with it.
-
-        Args:
-            data : abstract data which is to be encoded and written
-        """
-        data_hex = self.selected_file.encode_hex(data)
-        return self.update_binary(data_hex)
-
-    def read_record(self, rec_nr: int = 0):
-        """Read a record as binary data.
-
-        Args:
-            rec_nr : Record number to read
-        Returns:
-            hex string of binary data contained in record
-        """
-        if not isinstance(self.selected_file, LinFixedEF):
-            raise TypeError("Only works with Linear Fixed EF")
-        # returns a string of hex nibbles
-        return self.rs.card._scc.read_record(self.selected_file.fid, rec_nr)
-
-    def read_record_dec(self, rec_nr: int = 0) -> Tuple[dict, str]:
-        """Read a record and decode it to abstract data.
-
-        Args:
-            rec_nr : Record number to read
-        Returns:
-            abstract data contained in record
-        """
-        (data, sw) = self.read_record(rec_nr)
-        return (self.selected_file.decode_record_hex(data), sw)
-
-    def update_record(self, rec_nr: int, data_hex: str):
-        """Update a record with given binary data
-
-        Args:
-            rec_nr : Record number to read
-            data_hex : Hex string binary data to be written
-        """
-        if not isinstance(self.selected_file, LinFixedEF):
-            raise TypeError("Only works with Linear Fixed EF")
-        return self.rs.card._scc.update_record(self.selected_file.fid, rec_nr, data_hex, conserve=self.rs.conserve_write)
-
-    def update_record_dec(self, rec_nr: int, data: dict):
-        """Update a record with given abstract data.  Will encode abstract to binary data
-        and then write it to the given record on the card.
-
-        Args:
-            rec_nr : Record number to read
-            data_hex : Abstract data to be written
-        """
-        data_hex = self.selected_file.encode_record_hex(data)
-        return self.update_record(rec_nr, data_hex)
-
-    def retrieve_data(self, tag: int = 0):
-        """Read a DO/TLV as binary data.
-
-        Args:
-            tag : Tag of TLV/DO to read
-        Returns:
-            hex string of full BER-TLV DO including Tag and Length
-        """
-        if not isinstance(self.selected_file, BerTlvEF):
-            raise TypeError("Only works with BER-TLV EF")
-        # returns a string of hex nibbles
-        return self.rs.card._scc.retrieve_data(self.selected_file.fid, tag)
-
-    def retrieve_tags(self):
-        """Retrieve tags available on BER-TLV EF.
-
-        Returns:
-            list of integer tags contained in EF
-        """
-        if not isinstance(self.selected_file, BerTlvEF):
-            raise TypeError("Only works with BER-TLV EF")
-        data, sw = self.rs.card._scc.retrieve_data(self.selected_file.fid, 0x5c)
-        tag, length, value, remainder = bertlv_parse_one(h2b(data))
-        return list(value)
-
-    def set_data(self, tag: int, data_hex: str):
-        """Update a TLV/DO with given binary data
-
-        Args:
-            tag : Tag of TLV/DO to be written
-            data_hex : Hex string binary data to be written (value portion)
-        """
-        if not isinstance(self.selected_file, BerTlvEF):
-            raise TypeError("Only works with BER-TLV EF")
-        return self.rs.card._scc.set_data(self.selected_file.fid, tag, data_hex, conserve=self.rs.conserve_write)
-
-    def unregister_cmds(self, cmd_app=None):
-        """Unregister all file specific commands."""
-        if cmd_app and self.selected_file.shell_commands:
-            for c in self.selected_file.shell_commands:
-                cmd_app.unregister_command_set(c)
-
-
-class FileData:
-    """Represent the runtime, on-card data."""
-
-    def __init__(self, fdesc):
-        self.desc = fdesc
-        self.fcp = None
-
-
 def interpret_sw(sw_data: dict, sw: str):
     """Interpret a given status word.
 
@@ -1819,7 +1348,7 @@ class CardModel(abc.ABC):
 
     @classmethod
     @abc.abstractmethod
-    def add_files(cls, rs: RuntimeState):
+    def add_files(cls, rs: 'RuntimeState'):
         """Add model specific files to given RuntimeState."""
 
     @classmethod
@@ -1834,7 +1363,7 @@ class CardModel(abc.ABC):
         return False
 
     @staticmethod
-    def apply_matching_models(scc: SimCardCommands, rs: RuntimeState):
+    def apply_matching_models(scc: SimCardCommands, rs: 'RuntimeState'):
         """Check if any of the CardModel sub-classes 'match' the currently inserted card
         (by ATR or overriding the 'match' method). If so, call their 'add_files'
         method."""

pySim/global_platform.py

@@ -1,7 +1,7 @@
 # coding=utf-8
 """Partial Support for GlobalPLatform Card Spec (currently 2.1.1)
 
-(C) 2022 by Harald Welte <laforge@osmocom.org>
+(C) 2022-2023 by Harald Welte <laforge@osmocom.org>
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -68,6 +68,10 @@ sw_table = {
 
 # GlobalPlatform 2.1.1 Section 9.1.6
 KeyType = Enum(Byte,    des=0x80,
+                        tls_psk=0x85,                           # v2.3.1 Section 11.1.8
+                        aes=0x88,                               # v2.3.1 Section 11.1.8
+                        hmac_sha1=0x90,                         # v2.3.1 Section 11.1.8
+                        hmac_sha1_160=0x91,                     # v2.3.1 Section 11.1.8
                         rsa_public_exponent_e_cleartex=0xA0,
                         rsa_modulus_n_cleartext=0xA1,
                         rsa_modulus_n=0xA2,
@@ -77,29 +81,93 @@ KeyType = Enum(Byte,    des=0x80,
                         rsa_chines_remainder_pq=0xA6,
                         rsa_chines_remainder_dpi=0xA7,
                         rsa_chines_remainder_dqi=0xA8,
+                        ecc_public_key=0xB0,                    # v2.3.1 Section 11.1.8
+                        ecc_private_key=0xB1,                   # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_p=0xB2,             # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_a=0xB3,             # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_b=0xB4,             # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_g=0xB5,             # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_n=0xB6,             # v2.3.1 Section 11.1.8
+                        ecc_field_parameter_k=0xB7,             # v2.3.1 Section 11.1.8
+                        ecc_key_parameters_reference=0xF0,      # v2.3.1 Section 11.1.8
                         not_available=0xff)
 
 # GlobalPlatform 2.1.1 Section 9.3.3.1
-# example:
-#   e0 48
-#       c0 04 01708010
-#       c0 04 02708010
-#       c0 04 03708010
-#       c0 04 01018010
-#       c0 04 02018010
-#       c0 04 03018010
-#       c0 04 01028010
-#       c0 04 02028010
-#       c0 04 03028010
-#       c0 04 01038010
-#       c0 04 02038010
-#       c0 04 03038010
 class KeyInformationData(BER_TLV_IE, tag=0xc0):
+    _test_de_encode = [
+        ( 'c00401708010', {"key_identifier": 1, "key_version_number": 112, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00402708010', {"key_identifier": 2, "key_version_number": 112, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00403708010', {"key_identifier": 3, "key_version_number": 112, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00401018010', {"key_identifier": 1, "key_version_number": 1, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00402018010', {"key_identifier": 2, "key_version_number": 1, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00403018010', {"key_identifier": 3, "key_version_number": 1, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00401028010', {"key_identifier": 1, "key_version_number": 2, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00402028010', {"key_identifier": 2, "key_version_number": 2, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00403038010', {"key_identifier": 3, "key_version_number": 3, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00401038010', {"key_identifier": 1, "key_version_number": 3, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00402038010', {"key_identifier": 2, "key_version_number": 3, "key_types": [ {"length": 16, "type": "des"} ]} ),
+        ( 'c00402038810', {"key_identifier": 2, "key_version_number": 3, "key_types": [ {"length": 16, "type": "aes"} ]} ),
+    ]
+    KeyTypeLen = Struct('type'/KeyType, 'length'/Int8ub)
     _construct = Struct('key_identifier'/Byte, 'key_version_number'/Byte,
-                        'key_types'/GreedyRange(KeyType))
+                        'key_types'/GreedyRange(KeyTypeLen))
 class KeyInformation(BER_TLV_IE, tag=0xe0, nested=[KeyInformationData]):
     pass
 
+# GlobalPlatform v2.3.1 Section H.4
+class ScpInformation(BER_TLV_IE, tag=0xa0):
+    pass
+class PrivilegesAvailableSSD(BER_TLV_IE, tag=0x81):
+    pass
+class PrivilegesAvailableApplication(BER_TLV_IE, tag=0x82):
+    pass
+class SupportedLFDBHAlgorithms(BER_TLV_IE, tag=0x83):
+    pass
+class CiphersForLFDBEncryption(BER_TLV_IE, tag=0x84):
+    pass
+class CiphersForTokens(BER_TLV_IE, tag=0x85):
+    pass
+class CiphersForReceipts(BER_TLV_IE, tag=0x86):
+    pass
+class CiphersForDAPs(BER_TLV_IE, tag=0x87):
+    pass
+class KeyParameterReferenceList(BER_TLV_IE, tag=0x88):
+    pass
+class CardCapabilityInformation(BER_TLV_IE, tag=0x67, nested=[ScpInformation, PrivilegesAvailableSSD,
+                                                              PrivilegesAvailableApplication,
+                                                              SupportedLFDBHAlgorithms,
+                                                              CiphersForLFDBEncryption, CiphersForTokens,
+                                                              CiphersForReceipts, CiphersForDAPs,
+                                                              KeyParameterReferenceList]):
+    pass
+
+class CurrentSecurityLevel(BER_TLV_IE, tag=0xd3):
+    _construct = Int8ub
+
+# GlobalPlatform v2.3.1 Section 11.3.3.1.3
+class ApplicationAID(BER_TLV_IE, tag=0x4f):
+    _construct = HexAdapter(GreedyBytes)
+class ApplicationTemplate(BER_TLV_IE, tag=0x61, ntested=[ApplicationAID]):
+    pass
+class ListOfApplications(BER_TLV_IE, tag=0x2f00, nested=[ApplicationTemplate]):
+    pass
+
+# GlobalPlatform v2.3.1 Section 11.3.3.1.2 + TS 102 226
+class NumberOFInstalledApp(BER_TLV_IE, tag=0x81):
+    _construct = GreedyInteger()
+class FreeNonVolatileMemory(BER_TLV_IE, tag=0x82):
+    _construct = GreedyInteger()
+class FreeVolatileMemory(BER_TLV_IE, tag=0x83):
+    _construct = GreedyInteger()
+class ExtendedCardResourcesInfo(BER_TLV_IE, tag=0xff21, nested=[NumberOFInstalledApp, FreeNonVolatileMemory,
+                                                                FreeVolatileMemory]):
+    pass
+
+# GlobalPlatform v2.3.1 Section 7.4.2.4 + GP SPDM
+class SecurityDomainManagerURL(BER_TLV_IE, tag=0x5f50):
+    pass
+
+
 # card data sample, returned in response to GET DATA (80ca006600):
 # 66 31
 #    73 2f
@@ -174,11 +242,14 @@ class ProprietaryData(BER_TLV_IE, tag=0xA5, nested=[SecurityDomainManagementData
                                                     MaximumLengthOfDataFieldInCommandMessage]):
     pass
 
+# explicitly define this list and give it a name so pySim.euicc can reference it
+FciTemplateNestedList = [ApplicationID, SecurityDomainManagementData,
+                         ApplicationProductionLifeCycleData,
+                         MaximumLengthOfDataFieldInCommandMessage,
+                         ProprietaryData]
+
 # GlobalPlatform 2.1.1 Section 9.9.3.1
-class FciTemplate(BER_TLV_IE, tag=0x6f, nested=[ApplicationID, SecurityDomainManagementData,
-                                                ApplicationProductionLifeCycleData,
-                                                MaximumLengthOfDataFieldInCommandMessage,
-                                                ProprietaryData]):
+class FciTemplate(BER_TLV_IE, tag=0x6f, nested=FciTemplateNestedList):
     pass
 
 class IssuerIdentificationNumber(BER_TLV_IE, tag=0x42):
@@ -199,7 +270,13 @@ class DataCollection(TLV_IE_Collection, nested=[IssuerIdentificationNumber,
                                                 CardData,
                                                 KeyInformation,
                                                 SequenceCounterOfDefaultKvn,
-                                                ConfirmationCounter]):
+                                                ConfirmationCounter,
+                                                # v2.3.1
+                                                CardCapabilityInformation,
+                                                CurrentSecurityLevel,
+                                                ListOfApplications,
+                                                ExtendedCardResourcesInfo,
+                                                SecurityDomainManagerURL]):
     pass
 
 def decode_select_response(resp_hex: str) -> object:
@@ -223,22 +300,34 @@ class ADF_SD(CardADF):
         def __init__(self):
             super().__init__()
 
+        get_data_parser = argparse.ArgumentParser()
+        get_data_parser.add_argument('data_object_name', type=str,
+            help='Name of the data object to be retrieved from the card')
+
+        @cmd2.with_argparser(get_data_parser)
         def do_get_data(self, opts):
-            tlv_cls_name = opts.arg_list[0]
-            tlv_cls = DataCollection().members_by_name[tlv_cls_name]
-            (data, sw) = self._cmd.card._scc.get_data(cla=0x80, tag=tlv_cls.tag)
+            """Perform the GlobalPlatform GET DATA command in order to obtain some card-specific data."""
+            tlv_cls_name = opts.data_object_name
+            try:
+                tlv_cls = DataCollection().members_by_name[tlv_cls_name]
+            except KeyError:
+                do_names = [camel_to_snake(str(x.__name__)) for x in DataCollection.possible_nested]
+                self._cmd.poutput('Unknown data object "%s", available options: %s' % (tlv_cls_name,
+                                                                                       do_names))
+                return
+            (data, sw) = self._cmd.lchan.scc.get_data(cla=0x80, tag=tlv_cls.tag)
             ie = tlv_cls()
             ie.from_tlv(h2b(data))
             self._cmd.poutput_json(ie.to_dict())
 
         def complete_get_data(self, text, line, begidx, endidx) -> List[str]:
-            #data_dict = {camel_to_snake(str(x.__name__)): x for x in DataCollection.possible_nested}
-            data_dict = {str(x.__name__): x for x in DataCollection.possible_nested}
+            data_dict = {camel_to_snake(str(x.__name__)): x for x in DataCollection.possible_nested}
             index_dict = {1: data_dict}
             return self._cmd.index_based_complete(text, line, begidx, endidx, index_dict=index_dict)
 
 # Card Application of a Security Domain
 class CardApplicationSD(CardApplication):
+    __intermediate = True
     def __init__(self, aid: str, name: str, desc: str):
         super().__init__(name, adf=ADF_SD(aid, name, desc), sw=sw_table)
 

pySim/gsm_r.py

@@ -35,8 +35,8 @@ from construct import Optional as COptional
 from pySim.construct import *
 import enum
 
+from pySim.profile import CardProfileAddon
 from pySim.filesystem import *
-import pySim.ts_102_221
 import pySim.ts_51_011
 
 ######################################################################
@@ -47,10 +47,10 @@ import pySim.ts_51_011
 class FuncNTypeAdapter(Adapter):
     def _decode(self, obj, context, path):
         bcd = swap_nibbles(b2h(obj))
-        last_digit = bcd[-1]
+        last_digit = int(bcd[-1], 16)
         return {'functional_number': bcd[:-1],
-                'presentation_of_only_this_fn': last_digit & 4,
-                'permanent_fn': last_digit & 8}
+                'presentation_of_only_this_fn': bool(last_digit & 4),
+                'permanent_fn': bool(last_digit & 8)}
 
     def _encode(self, obj, context, path):
         return 'FIXME'
@@ -58,9 +58,13 @@ class FuncNTypeAdapter(Adapter):
 
 class EF_FN(LinFixedEF):
     """Section 7.2"""
-
+    _test_decode = [
+        ( "40315801000010ff01",
+          { "functional_number_and_type": { "functional_number": "04138510000001f",
+                "presentation_of_only_this_fn": True, "permanent_fn": True }, "list_number": 1 } ),
+    ]
     def __init__(self):
-        super().__init__(fid='6ff1', sfid=None, name='EF.EN',
+        super().__init__(fid='6ff1', sfid=None, name='EF.FN',
                          desc='Functional numbers', rec_len=(9, 9))
         self._construct = Struct('functional_number_and_type'/FuncNTypeAdapter(Bytes(8)),
                                  'list_number'/Int8ub)
@@ -147,7 +151,12 @@ NextTableType = Enum(Byte, decision=0xf0, predefined=0xf1,
 
 class EF_CallconfC(TransparentEF):
     """Section 7.3"""
-
+    _test_de_encode = [
+        ( "026121ffffffffffff1e000a040a010253600795792426f0",
+          { "pl_conf": 3, "conf_nr": "1612ffffffffffff", "max_rand": 30, "n_ack_max": 10,
+            "pl_ack": 1, "n_nested_max": 10, "train_emergency_gid": 1, "shunting_emergency_gid": 2,
+           "imei": "350670599742620f" } ),
+    ]
     def __init__(self):
         super().__init__(fid='6ff2', sfid=None, name='EF.CallconfC', size=(24, 24),
                          desc='Call Configuration of emergency calls Configuration')
@@ -180,7 +189,9 @@ class EF_CallconfI(LinFixedEF):
 
 class EF_Shunting(TransparentEF):
     """Section 7.6"""
-
+    _test_de_encode = [
+        ( "03f8ffffff000000", { "common_gid": 3, "shunting_gid": "f8ffffff000000" } ),
+    ]
     def __init__(self):
         super().__init__(fid='6ff4', sfid=None,
                          name='EF.Shunting', desc='Shunting', size=(8, 8))
@@ -190,11 +201,22 @@ class EF_Shunting(TransparentEF):
 
 class EF_GsmrPLMN(LinFixedEF):
     """Section 7.7"""
-
+    _test_de_encode = [
+        ( "22f860f86f8d6f8e01", { "plmn": "228-06", "class_of_network": {
+                                    "supported": { "vbs": True, "vgcs": True, "emlpp": True,
+                                    "fn": True, "eirene": True }, "preference": 0 },
+                                  "ic_incoming_ref_tbl": "6f8d", "outgoing_ref_tbl": "6f8e",
+                                  "ic_table_ref": "01" } ),
+        ( "22f810416f8d6f8e02", { "plmn": "228-01", "class_of_network": {
+                                    "supported": { "vbs": False, "vgcs": False, "emlpp": False,
+                                    "fn": True, "eirene": False }, "preference": 1 },
+                                  "ic_incoming_ref_tbl": "6f8d", "outgoing_ref_tbl": "6f8e",
+                                  "ic_table_ref": "02" } ),
+    ]
     def __init__(self):
         super().__init__(fid='6ff5', sfid=None, name='EF.GsmrPLMN',
                          desc='GSM-R network selection', rec_len=(9, 9))
-        self._construct = Struct('plmn'/BcdAdapter(Bytes(3)),
+        self._construct = Struct('plmn'/PlmnAdapter(Bytes(3)),
                                  'class_of_network'/BitStruct('supported'/FlagsEnum(BitsInteger(5), vbs=1, vgcs=2, emlpp=4, fn=8, eirene=16),
                                                               'preference'/BitsInteger(3)),
                                  'ic_incoming_ref_tbl'/HexAdapter(Bytes(2)),
@@ -204,19 +226,29 @@ class EF_GsmrPLMN(LinFixedEF):
 
 class EF_IC(LinFixedEF):
     """Section 7.8"""
-
+    _test_de_encode = [
+        ( "f06f8e40f10001", { "next_table_type": "decision", "id_of_next_table": "6f8e",
+                              "ic_decision_value": "041f", "network_string_table_index": 1 } ),
+        ( "ffffffffffffff", { "next_table_type": "empty", "id_of_next_table": "ffff",
+                              "ic_decision_value": "ffff", "network_string_table_index": 65535 } ),
+    ]
     def __init__(self):
         super().__init__(fid='6f8d', sfid=None, name='EF.IC',
                          desc='International Code', rec_len=(7, 7))
         self._construct = Struct('next_table_type'/NextTableType,
                                  'id_of_next_table'/HexAdapter(Bytes(2)),
                                  'ic_decision_value'/BcdAdapter(Bytes(2)),
-                                 'network_string_table_index'/Int8ub)
+                                 'network_string_table_index'/Int16ub)
 
 
 class EF_NW(LinFixedEF):
     """Section 7.9"""
-
+    _test_de_encode = [
+        ( "47534d2d52204348", "GSM-R CH" ),
+        ( "537769737347534d", "SwissGSM" ),
+        ( "47534d2d52204442", "GSM-R DB" ),
+        ( "47534d2d52524649", "GSM-RRFI" ),
+    ]
     def __init__(self):
         super().__init__(fid='6f80', sfid=None, name='EF.NW',
                          desc='Network Name', rec_len=(8, 8))
@@ -225,8 +257,15 @@ class EF_NW(LinFixedEF):
 
 class EF_Switching(LinFixedEF):
     """Section 8.4"""
-
-    def __init__(self, fid, name, desc):
+    _test_de_encode = [
+        ( "f26f87f0ff00", { "next_table_type": "num_dial_digits", "id_of_next_table": "6f87",
+                            "decision_value": "0fff", "string_table_index": 0 } ),
+        ( "f06f8ff1ff01", { "next_table_type": "decision", "id_of_next_table": "6f8f",
+                            "decision_value": "1fff", "string_table_index": 1 } ),
+        ( "f16f89f5ff05", { "next_table_type": "predefined", "id_of_next_table": "6f89",
+                            "decision_value": "5fff", "string_table_index": 5 } ),
+    ]
+    def __init__(self, fid='1234', name='Switching', desc=None):
         super().__init__(fid=fid, sfid=None,
                          name=name, desc=desc, rec_len=(6, 6))
         self._construct = Struct('next_table_type'/NextTableType,
@@ -237,22 +276,41 @@ class EF_Switching(LinFixedEF):
 
 class EF_Predefined(LinFixedEF):
     """Section 8.5"""
+    _test_de_encode = [
+        ( "f26f85", 1, { "next_table_type": "num_dial_digits", "id_of_next_table": "6f85" } ),
+        ( "f0ffc8", 2, { "predefined_value1": "0fff", "string_table_index1": 200 } ),
+    ]
+    # header and other records have different structure. WTF !?!
+    construct_first = Struct('next_table_type'/NextTableType,
+                             'id_of_next_table'/HexAdapter(Bytes(2)))
+    construct_others = Struct('predefined_value1'/BcdAdapter(Bytes(2)),
+                              'string_table_index1'/Int8ub)
 
-    def __init__(self, fid, name, desc):
+    def __init__(self, fid='1234', name='Predefined', desc=None):
         super().__init__(fid=fid, sfid=None,
                          name=name, desc=desc, rec_len=(3, 3))
-        # header and other records have different structure. WTF !?!
-        self._construct = Struct('next_table_type'/NextTableType,
-                                 'id_of_next_table'/HexAdapter(Bytes(2)),
-                                 'predefined_value1'/HexAdapter(Bytes(2)),
-                                 'string_table_index1'/Int8ub)
-        # TODO: predefined value n, ...
 
+    def _decode_record_bin(self, raw_bin_data : bytes, record_nr : int) -> dict:
+        if record_nr == 1:
+            return parse_construct(self.construct_first, raw_bin_data)
+        else:
+            return parse_construct(self.construct_others, raw_bin_data)
+
+    def _encode_record_bin(self, abstract_data : dict, record_nr : int) -> bytearray:
+        r = None
+        if record_nr == 1:
+            r = self.construct_first.build(abstract_data)
+        else:
+            r = self.construct_others.build(abstract_data)
+        return filter_dict(r)
 
 class EF_DialledVals(TransparentEF):
     """Section 8.6"""
-
-    def __init__(self, fid, name, desc):
+    _test_de_encode = [
+        ( "ffffff22", { "next_table_type": "empty", "id_of_next_table": "ffff", "dialed_digits": "22" } ),
+        ( "f16f8885", { "next_table_type": "predefined", "id_of_next_table": "6f88", "dialed_digits": "58" }),
+    ]
+    def __init__(self, fid='1234', name='DialledVals', desc=None):
         super().__init__(fid=fid, sfid=None, name=name, desc=desc, size=(4, 4))
         self._construct = Struct('next_table_type'/NextTableType,
                                  'id_of_next_table'/HexAdapter(Bytes(2)),
@@ -304,3 +362,15 @@ class DF_EIRENE(CardDF):
                            desc='Free Number Call Type 0 and 8'),
         ]
         self.add_files(files)
+
+
+class AddonGSMR(CardProfileAddon):
+    """An Addon that can be found on either classic GSM SIM or on UICC to support GSM-R."""
+    def __init__(self):
+        files = [
+            DF_EIRENE()
+        ]
+        super().__init__('GSM-R', desc='Railway GSM', files_in_mf=files)
+
+    def probe(self, card: 'CardBase') -> bool:
+        return card.file_exists(self.files_in_mf[0].fid)

pySim/legacy/ts_31_102.py

@@ -0,0 +1,134 @@
+# -*- coding: utf-8 -*-
+
+"""
+Various constants from 3GPP TS 31.102 V17.9.0 usd by *legacy* code
+"""
+
+#
+# Copyright (C) 2020 Supreeth Herle <herlesupreeth@gmail.com>
+# Copyright (C) 2021-2023 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+EF_USIM_ADF_map = {
+    'LI': '6F05',
+    'ARR': '6F06',
+    'IMSI': '6F07',
+    'Keys': '6F08',
+    'KeysPS': '6F09',
+    'DCK': '6F2C',
+    'HPPLMN': '6F31',
+    'CNL': '6F32',
+    'ACMmax': '6F37',
+    'UST': '6F38',
+    'ACM': '6F39',
+    'FDN': '6F3B',
+    'SMS': '6F3C',
+    'GID1': '6F3E',
+    'GID2': '6F3F',
+    'MSISDN': '6F40',
+    'PUCT': '6F41',
+    'SMSP': '6F42',
+    'SMSS': '6F42',
+    'CBMI': '6F45',
+    'SPN': '6F46',
+    'SMSR': '6F47',
+    'CBMID': '6F48',
+    'SDN': '6F49',
+    'EXT2': '6F4B',
+    'EXT3': '6F4C',
+    'BDN': '6F4D',
+    'EXT5': '6F4E',
+    'CCP2': '6F4F',
+    'CBMIR': '6F50',
+    'EXT4': '6F55',
+    'EST': '6F56',
+    'ACL': '6F57',
+    'CMI': '6F58',
+    'START-HFN': '6F5B',
+    'THRESHOLD': '6F5C',
+    'PLMNwAcT': '6F60',
+    'OPLMNwAcT': '6F61',
+    'HPLMNwAcT': '6F62',
+    'PSLOCI': '6F73',
+    'ACC': '6F78',
+    'FPLMN': '6F7B',
+    'LOCI': '6F7E',
+    'ICI': '6F80',
+    'OCI': '6F81',
+    'ICT': '6F82',
+    'OCT': '6F83',
+    'AD': '6FAD',
+    'VGCS': '6FB1',
+    'VGCSS': '6FB2',
+    'VBS': '6FB3',
+    'VBSS': '6FB4',
+    'eMLPP': '6FB5',
+    'AAeM': '6FB6',
+    'ECC': '6FB7',
+    'Hiddenkey': '6FC3',
+    'NETPAR': '6FC4',
+    'PNN': '6FC5',
+    'OPL': '6FC6',
+    'MBDN': '6FC7',
+    'EXT6': '6FC8',
+    'MBI': '6FC9',
+    'MWIS': '6FCA',
+    'CFIS': '6FCB',
+    'EXT7': '6FCC',
+    'SPDI': '6FCD',
+    'MMSN': '6FCE',
+    'EXT8': '6FCF',
+    'MMSICP': '6FD0',
+    'MMSUP': '6FD1',
+    'MMSUCP': '6FD2',
+    'NIA': '6FD3',
+    'VGCSCA': '6FD4',
+    'VBSCA': '6FD5',
+    'GBAP': '6FD6',
+    'MSK': '6FD7',
+    'MUK': '6FD8',
+    'EHPLMN': '6FD9',
+    'GBANL': '6FDA',
+    'EHPLMNPI': '6FDB',
+    'LRPLMNSI': '6FDC',
+    'NAFKCA': '6FDD',
+    'SPNI': '6FDE',
+    'PNNI': '6FDF',
+    'NCP-IP': '6FE2',
+    'EPSLOCI': '6FE3',
+    'EPSNSC': '6FE4',
+    'UFC': '6FE6',
+    'UICCIARI': '6FE7',
+    'NASCONFIG': '6FE8',
+    'PWC': '6FEC',
+    'FDNURI': '6FED',
+    'BDNURI': '6FEE',
+    'SDNURI': '6FEF',
+    'IWL': '6FF0',
+    'IPS': '6FF1',
+    'IPD': '6FF2',
+    'ePDGId': '6FF3',
+    'ePDGSelection': '6FF4',
+    'ePDGIdEm': '6FF5',
+    'ePDGSelectionEm': '6FF6',
+}
+
+LOCI_STATUS_map = {
+    0:	'updated',
+    1:	'not updated',
+    2:	'plmn not allowed',
+    3:	'locatation area not allowed'
+}

pySim/legacy/ts_31_103.py

@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+
+"""
+Various constants from 3GPP TS 31.103 V16.1.0 used by *legacy* code only
+"""
+
+# Copyright (C) 2020 Supreeth Herle <herlesupreeth@gmail.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+EF_ISIM_ADF_map = {
+    'IST': '6F07',
+    'IMPI': '6F02',
+    'DOMAIN': '6F03',
+    'IMPU': '6F04',
+    'AD': '6FAD',
+    'ARR': '6F06',
+    'PCSCF': '6F09',
+    'GBAP': '6FD5',
+    'GBANL': '6FD7',
+    'NAFKCA': '6FDD',
+    'UICCIARI': '6FE7',
+    'SMS': '6F3C',
+    'SMSS': '6F43',
+    'SMSR': '6F47',
+    'SMSP': '6F42',
+    'FromPreferred': '6FF7',
+    'IMSConfigData': '6FF8',
+    'XCAPConfigData': '6FFC',
+    'WebRTCURI': '6FFA'
+}

pySim/legacy/ts_51_011.py

@@ -0,0 +1,257 @@
+# -*- coding: utf-8 -*-
+
+""" Various constants from 3GPP TS 51.011 used by *legacy* code only.
+
+This will likely be removed in future versions of pySim.  Please instead
+use the pySim.filesystem class model with the various profile/application
+specific extension modules.
+"""
+
+# Copyright (C) 2017 Alexander.Chemeris <Alexander.Chemeris@gmail.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+MF_num = '3F00'
+
+DF_num = {
+    'TELECOM': '7F10',
+
+    'GSM': '7F20',
+    'IS-41': '7F22',
+    'FP-CTS': '7F23',
+
+    'GRAPHICS': '5F50',
+
+    'IRIDIUM': '5F30',
+    'GLOBST': '5F31',
+    'ICO': '5F32',
+    'ACeS': '5F33',
+
+    'EIA/TIA-553': '5F40',
+    'CTS': '5F60',
+    'SOLSA': '5F70',
+
+    'MExE': '5F3C',
+}
+
+EF_num = {
+    # MF
+    'ICCID': '2FE2',
+    'ELP': '2F05',
+    'DIR': '2F00',
+
+    # DF_TELECOM
+    'ADN': '6F3A',
+    'FDN': '6F3B',
+    'SMS': '6F3C',
+    'CCP': '6F3D',
+    'MSISDN': '6F40',
+    'SMSP': '6F42',
+    'SMSS': '6F43',
+    'LND': '6F44',
+    'SMSR': '6F47',
+    'SDN': '6F49',
+    'EXT1': '6F4A',
+    'EXT2': '6F4B',
+    'EXT3': '6F4C',
+    'BDN': '6F4D',
+    'EXT4': '6F4E',
+    'CMI': '6F58',
+    'ECCP': '6F4F',
+
+    # DF_GRAPHICS
+    'IMG': '4F20',
+
+    # DF_SoLSA
+    'SAI': '4F30',
+    'SLL': '4F31',
+
+    # DF_MExE
+    'MExE-ST': '4F40',
+    'ORPK': '4F41',
+    'ARPK': '4F42',
+    'TPRPK': '4F43',
+
+    # DF_GSM
+    'LP': '6F05',
+    'IMSI': '6F07',
+    'Kc': '6F20',
+    'DCK': '6F2C',
+    'PLMNsel': '6F30',
+    'HPPLMN': '6F31',
+    'CNL': '6F32',
+    'ACMmax': '6F37',
+    'SST': '6F38',
+    'ACM': '6F39',
+    'GID1': '6F3E',
+    'GID2': '6F3F',
+    'PUCT': '6F41',
+    'CBMI': '6F45',
+    'SPN': '6F46',
+    'CBMID': '6F48',
+    'BCCH': '6F74',
+    'ACC': '6F78',
+    'FPLMN': '6F7B',
+    'LOCI': '6F7E',
+    'AD': '6FAD',
+    'PHASE': '6FAE',
+    'VGCS': '6FB1',
+    'VGCSS': '6FB2',
+    'VBS': '6FB3',
+    'VBSS': '6FB4',
+    'eMLPP': '6FB5',
+    'AAeM': '6FB6',
+    'ECC': '6FB7',
+    'CBMIR': '6F50',
+    'NIA': '6F51',
+    'KcGPRS': '6F52',
+    'LOCIGPRS': '6F53',
+    'SUME': '6F54',
+    'PLMNwAcT': '6F60',
+    'OPLMNwAcT': '6F61',
+    # Figure 8 names it HPLMNAcT, but in the text it's names it HPLMNwAcT
+    'HPLMNAcT': '6F62',
+    'HPLMNwAcT': '6F62',
+    'CPBCCH': '6F63',
+    'INVSCAN': '6F64',
+    'PNN': '6FC5',
+    'OPL': '6FC6',
+    'MBDN': '6FC7',
+    'EXT6': '6FC8',
+    'MBI': '6FC9',
+    'MWIS': '6FCA',
+    'CFIS': '6FCB',
+    'EXT7': '6FCC',
+    'SPDI': '6FCD',
+    'MMSN': '6FCE',
+    'EXT8': '6FCF',
+    'MMSICP': '6FD0',
+    'MMSUP': '6FD1',
+    'MMSUCP': '6FD2',
+}
+
+DF = {
+    'TELECOM':  [MF_num, DF_num['TELECOM']],
+
+    'GSM':      [MF_num, DF_num['GSM']],
+    'IS-41':    [MF_num, DF_num['IS-41']],
+    'FP-CTS':   [MF_num, DF_num['FP-CTS']],
+
+    'GRAPHICS': [MF_num, DF_num['GRAPHICS']],
+
+    'IRIDIUM':  [MF_num, DF_num['IRIDIUM']],
+    'GLOBST':   [MF_num, DF_num['GLOBST']],
+    'ICO':      [MF_num, DF_num['ICO']],
+    'ACeS':     [MF_num, DF_num['ACeS']],
+
+    'EIA/TIA-553': [MF_num, DF_num['EIA/TIA-553']],
+    'CTS':      [MF_num, DF_num['CTS']],
+    'SoLSA':    [MF_num, DF_num['SOLSA']],
+
+    'MExE':     [MF_num, DF_num['MExE']],
+}
+
+
+EF = {
+    'ICCID':  [MF_num, EF_num['ICCID']],
+    'ELP':    [MF_num, EF_num['ELP']],
+    'DIR':    [MF_num, EF_num['DIR']],
+
+    'ADN':    DF['TELECOM']+[EF_num['ADN']],
+    'FDN':    DF['TELECOM']+[EF_num['FDN']],
+    'SMS':    DF['TELECOM']+[EF_num['SMS']],
+    'CCP':    DF['TELECOM']+[EF_num['CCP']],
+    'MSISDN': DF['TELECOM']+[EF_num['MSISDN']],
+    'SMSP':   DF['TELECOM']+[EF_num['SMSP']],
+    'SMSS':   DF['TELECOM']+[EF_num['SMSS']],
+    'LND':    DF['TELECOM']+[EF_num['LND']],
+    'SMSR':   DF['TELECOM']+[EF_num['SMSR']],
+    'SDN':    DF['TELECOM']+[EF_num['SDN']],
+    'EXT1':   DF['TELECOM']+[EF_num['EXT1']],
+    'EXT2':   DF['TELECOM']+[EF_num['EXT2']],
+    'EXT3':   DF['TELECOM']+[EF_num['EXT3']],
+    'BDN':    DF['TELECOM']+[EF_num['BDN']],
+    'EXT4':   DF['TELECOM']+[EF_num['EXT4']],
+    'CMI':    DF['TELECOM']+[EF_num['CMI']],
+    'ECCP':   DF['TELECOM']+[EF_num['ECCP']],
+
+    'IMG':    DF['GRAPHICS']+[EF_num['IMG']],
+
+    'SAI':    DF['SoLSA']+[EF_num['SAI']],
+    'SLL':    DF['SoLSA']+[EF_num['SLL']],
+
+    'MExE-ST': DF['MExE']+[EF_num['MExE-ST']],
+    'ORPK':   DF['MExE']+[EF_num['ORPK']],
+    'ARPK':   DF['MExE']+[EF_num['ARPK']],
+    'TPRPK':  DF['MExE']+[EF_num['TPRPK']],
+
+    'LP':     DF['GSM']+[EF_num['LP']],
+    'IMSI':   DF['GSM']+[EF_num['IMSI']],
+    'Kc':     DF['GSM']+[EF_num['Kc']],
+    'DCK':    DF['GSM']+[EF_num['DCK']],
+    'PLMNsel': DF['GSM']+[EF_num['PLMNsel']],
+    'HPPLMN': DF['GSM']+[EF_num['HPPLMN']],
+    'CNL':    DF['GSM']+[EF_num['CNL']],
+    'ACMmax': DF['GSM']+[EF_num['ACMmax']],
+    'SST':    DF['GSM']+[EF_num['SST']],
+    'ACM':    DF['GSM']+[EF_num['ACM']],
+    'GID1':   DF['GSM']+[EF_num['GID1']],
+    'GID2':   DF['GSM']+[EF_num['GID2']],
+    'PUCT':   DF['GSM']+[EF_num['PUCT']],
+    'CBMI':   DF['GSM']+[EF_num['CBMI']],
+    'SPN':    DF['GSM']+[EF_num['SPN']],
+    'CBMID':  DF['GSM']+[EF_num['CBMID']],
+    'BCCH':   DF['GSM']+[EF_num['BCCH']],
+    'ACC':    DF['GSM']+[EF_num['ACC']],
+    'FPLMN':  DF['GSM']+[EF_num['FPLMN']],
+    'LOCI':   DF['GSM']+[EF_num['LOCI']],
+    'AD':     DF['GSM']+[EF_num['AD']],
+    'PHASE':  DF['GSM']+[EF_num['PHASE']],
+    'VGCS':   DF['GSM']+[EF_num['VGCS']],
+    'VGCSS':  DF['GSM']+[EF_num['VGCSS']],
+    'VBS':    DF['GSM']+[EF_num['VBS']],
+    'VBSS':   DF['GSM']+[EF_num['VBSS']],
+    'eMLPP':  DF['GSM']+[EF_num['eMLPP']],
+    'AAeM':   DF['GSM']+[EF_num['AAeM']],
+    'ECC':    DF['GSM']+[EF_num['ECC']],
+    'CBMIR':  DF['GSM']+[EF_num['CBMIR']],
+    'NIA':    DF['GSM']+[EF_num['NIA']],
+    'KcGPRS': DF['GSM']+[EF_num['KcGPRS']],
+    'LOCIGPRS': DF['GSM']+[EF_num['LOCIGPRS']],
+    'SUME':   DF['GSM']+[EF_num['SUME']],
+    'PLMNwAcT': DF['GSM']+[EF_num['PLMNwAcT']],
+    'OPLMNwAcT': DF['GSM']+[EF_num['OPLMNwAcT']],
+    # Figure 8 names it HPLMNAcT, but in the text it's names it HPLMNwAcT
+    'HPLMNAcT': DF['GSM']+[EF_num['HPLMNAcT']],
+    'HPLMNwAcT': DF['GSM']+[EF_num['HPLMNAcT']],
+    'CPBCCH': DF['GSM']+[EF_num['CPBCCH']],
+    'INVSCAN': DF['GSM']+[EF_num['INVSCAN']],
+    'PNN':    DF['GSM']+[EF_num['PNN']],
+    'OPL':    DF['GSM']+[EF_num['OPL']],
+    'MBDN':   DF['GSM']+[EF_num['MBDN']],
+    'EXT6':   DF['GSM']+[EF_num['EXT6']],
+    'MBI':    DF['GSM']+[EF_num['MBI']],
+    'MWIS':   DF['GSM']+[EF_num['MWIS']],
+    'CFIS':   DF['GSM']+[EF_num['CFIS']],
+    'EXT7':   DF['GSM']+[EF_num['EXT7']],
+    'SPDI':   DF['GSM']+[EF_num['SPDI']],
+    'MMSN':   DF['GSM']+[EF_num['MMSN']],
+    'EXT8':   DF['GSM']+[EF_num['EXT8']],
+    'MMSICP': DF['GSM']+[EF_num['MMSICP']],
+    'MMSUP':  DF['GSM']+[EF_num['MMSUP']],
+    'MMSUCP': DF['GSM']+[EF_num['MMSUCP']],
+}
+

pySim/legacy/utils.py

@@ -0,0 +1,332 @@
+# -*- coding: utf-8 -*-
+
+""" pySim: various utilities only used by legacy tools (pySim-{prog,read})
+"""
+
+# Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from pySim.utils import Hexstr, rpad, enc_plmn, h2i, i2s, s2h
+from pySim.utils import dec_xplmn_w_act, dec_xplmn, dec_mcc_from_plmn, dec_mnc_from_plmn
+
+def hexstr_to_Nbytearr(s, nbytes):
+    return [s[i:i+(nbytes*2)] for i in range(0, len(s), (nbytes*2))]
+
+def format_xplmn_w_act(hexstr):
+    s = ""
+    for rec_data in hexstr_to_Nbytearr(hexstr, 5):
+        rec_info = dec_xplmn_w_act(rec_data)
+        if rec_info['mcc'] == "" and rec_info['mnc'] == "":
+            rec_str = "unused"
+        else:
+            rec_str = "MCC: %s MNC: %s AcT: %s" % (
+                rec_info['mcc'], rec_info['mnc'], ", ".join(rec_info['act']))
+        s += "\t%s # %s\n" % (rec_data, rec_str)
+    return s
+
+
+def format_xplmn(hexstr: Hexstr) -> str:
+    s = ""
+    for rec_data in hexstr_to_Nbytearr(hexstr, 3):
+        rec_info = dec_xplmn(rec_data)
+        if not rec_info['mcc'] and not rec_info['mnc']:
+            rec_str = "unused"
+        else:
+            rec_str = "MCC: %s MNC: %s" % (rec_info['mcc'], rec_info['mnc'])
+        s += "\t%s # %s\n" % (rec_data, rec_str)
+    return s
+
+
+def format_ePDGSelection(hexstr):
+    ePDGSelection_info_tag_chars = 2
+    ePDGSelection_info_tag_str = hexstr[:2]
+    s = ""
+    # Minimum length
+    len_chars = 2
+    # TODO: Need to determine length properly - definite length support only
+    # Inconsistency in spec: 3GPP TS 31.102 version 15.2.0 Release 15, 4.2.104
+    # As per spec, length is 5n, n - number of PLMNs
+    # But, each PLMN entry is made of PLMN (3 Bytes) + ePDG Priority (2 Bytes) + ePDG FQDN format (1 Byte)
+    # Totalling to 6 Bytes, maybe length should be 6n
+    len_str = hexstr[ePDGSelection_info_tag_chars:ePDGSelection_info_tag_chars+len_chars]
+
+    # Not programmed scenario
+    if int(len_str, 16) == 255 or int(ePDGSelection_info_tag_str, 16) == 255:
+        len_chars = 0
+        ePDGSelection_info_tag_chars = 0
+    if len_str[0] == '8':
+        # The bits 7 to 1 denotes the number of length octets if length > 127
+        if int(len_str[1]) > 0:
+            # Update number of length octets
+            len_chars = len_chars * int(len_str[1])
+            len_str = hexstr[ePDGSelection_info_tag_chars:len_chars]
+
+    content_str = hexstr[ePDGSelection_info_tag_chars+len_chars:]
+    # Right pad to prevent index out of range - multiple of 6 bytes
+    content_str = rpad(content_str, len(content_str) +
+                       (12 - (len(content_str) % 12)))
+    for rec_data in hexstr_to_Nbytearr(content_str, 6):
+        rec_info = dec_ePDGSelection(rec_data)
+        if rec_info['mcc'] == 0xFFF and rec_info['mnc'] == 0xFFF:
+            rec_str = "unused"
+        else:
+            rec_str = "MCC: %03d MNC: %03d ePDG Priority: %s ePDG FQDN format: %s" % \
+                (rec_info['mcc'], rec_info['mnc'],
+                 rec_info['epdg_priority'], rec_info['epdg_fqdn_format'])
+        s += "\t%s # %s\n" % (rec_data, rec_str)
+    return s
+
+def enc_st(st, service, state=1):
+    """
+    Encodes the EF S/U/IST/EST and returns the updated Service Table
+
+    Parameters:
+            st - Current value of SIM/USIM/ISIM Service Table
+            service - Service Number to encode as activated/de-activated
+            state - 1 mean activate, 0 means de-activate
+
+    Returns:
+            s - Modified value of SIM/USIM/ISIM Service Table
+
+    Default values:
+            - state: 1 - Sets the particular Service bit to 1
+    """
+    st_bytes = [st[i:i+2] for i in range(0, len(st), 2)]
+
+    s = ""
+    # Check whether the requested service is present in each byte
+    for i in range(0, len(st_bytes)):
+        # Byte i contains info about Services num (8i+1) to num (8i+8)
+        if service in range((8*i) + 1, (8*i) + 9):
+            byte = int(st_bytes[i], 16)
+            # Services in each byte are in order MSB to LSB
+            # MSB - Service (8i+8)
+            # LSB - Service (8i+1)
+            mod_byte = 0x00
+            # Copy bit by bit contents of byte to mod_byte with modified bit
+            # for requested service
+            for j in range(1, 9):
+                mod_byte = mod_byte >> 1
+                if service == (8*i) + j:
+                    mod_byte = state == 1 and mod_byte | 0x80 or mod_byte & 0x7f
+                else:
+                    mod_byte = byte & 0x01 == 0x01 and mod_byte | 0x80 or mod_byte & 0x7f
+                byte = byte >> 1
+
+            s += ('%02x' % (mod_byte))
+        else:
+            s += st_bytes[i]
+
+    return s
+
+
+def dec_st(st, table="sim") -> str:
+    """
+    Parses the EF S/U/IST and prints the list of available services in EF S/U/IST
+    """
+
+    if table == "isim":
+        from pySim.ts_31_103 import EF_IST_map
+        lookup_map = EF_IST_map
+    elif table == "usim":
+        from pySim.ts_31_102 import EF_UST_map
+        lookup_map = EF_UST_map
+    else:
+        from pySim.ts_51_011 import EF_SST_map
+        lookup_map = EF_SST_map
+
+    st_bytes = [st[i:i+2] for i in range(0, len(st), 2)]
+
+    avail_st = ""
+    # Get each byte and check for available services
+    for i in range(0, len(st_bytes)):
+        # Byte i contains info about Services num (8i+1) to num (8i+8)
+        byte = int(st_bytes[i], 16)
+        # Services in each byte are in order MSB to LSB
+        # MSB - Service (8i+8)
+        # LSB - Service (8i+1)
+        for j in range(1, 9):
+            if byte & 0x01 == 0x01 and ((8*i) + j in lookup_map):
+                # Byte X contains info about Services num (8X-7) to num (8X)
+                # bit = 1: service available
+                # bit = 0: service not available
+                avail_st += '\tService %d - %s\n' % (
+                    (8*i) + j, lookup_map[(8*i) + j])
+            byte = byte >> 1
+    return avail_st
+
+
+def enc_ePDGSelection(hexstr, mcc, mnc, epdg_priority='0001', epdg_fqdn_format='00'):
+    """
+    Encode ePDGSelection so it can be stored at EF.ePDGSelection or EF.ePDGSelectionEm.
+    See 3GPP TS 31.102 version 15.2.0 Release 15, section 4.2.104 and 4.2.106.
+
+    Default values:
+            - epdg_priority: '0001' - 1st Priority
+            - epdg_fqdn_format: '00' - Operator Identifier FQDN
+    """
+
+    plmn1 = enc_plmn(mcc, mnc) + epdg_priority + epdg_fqdn_format
+    # TODO: Handle encoding of Length field for length more than 127 Bytes
+    content = '80' + ('%02x' % (len(plmn1)//2)) + plmn1
+    content = rpad(content, len(hexstr))
+    return content
+
+
+def dec_ePDGSelection(sixhexbytes):
+    """
+    Decode ePDGSelection to get EF.ePDGSelection or EF.ePDGSelectionEm.
+    See 3GPP TS 31.102 version 15.2.0 Release 15, section 4.2.104 and 4.2.106.
+    """
+
+    res = {'mcc': 0, 'mnc': 0, 'epdg_priority': 0, 'epdg_fqdn_format': ''}
+    plmn_chars = 6
+    epdg_priority_chars = 4
+    epdg_fqdn_format_chars = 2
+    # first three bytes (six ascii hex chars)
+    plmn_str = sixhexbytes[:plmn_chars]
+    # two bytes after first three bytes
+    epdg_priority_str = sixhexbytes[plmn_chars:plmn_chars +
+                                    epdg_priority_chars]
+    # one byte after first five bytes
+    epdg_fqdn_format_str = sixhexbytes[plmn_chars +
+                                       epdg_priority_chars:plmn_chars + epdg_priority_chars + epdg_fqdn_format_chars]
+    res['mcc'] = dec_mcc_from_plmn(plmn_str)
+    res['mnc'] = dec_mnc_from_plmn(plmn_str)
+    res['epdg_priority'] = epdg_priority_str
+    res['epdg_fqdn_format'] = epdg_fqdn_format_str == '00' and 'Operator Identifier FQDN' or 'Location based FQDN'
+    return res
+
+
+def first_TLV_parser(bytelist):
+    '''
+    first_TLV_parser([0xAA, 0x02, 0xAB, 0xCD, 0xFF, 0x00]) -> (170, 2, [171, 205])
+
+    parses first TLV format record in a list of bytelist
+    returns a 3-Tuple: Tag, Length, Value
+    Value is a list of bytes
+    parsing of length is ETSI'style 101.220
+    '''
+    Tag = bytelist[0]
+    if bytelist[1] == 0xFF:
+        Len = bytelist[2]*256 + bytelist[3]
+        Val = bytelist[4:4+Len]
+    else:
+        Len = bytelist[1]
+        Val = bytelist[2:2+Len]
+    return (Tag, Len, Val)
+
+
+def TLV_parser(bytelist):
+    '''
+    TLV_parser([0xAA, ..., 0xFF]) -> [(T, L, [V]), (T, L, [V]), ...]
+
+    loops on the input list of bytes with the "first_TLV_parser()" function
+    returns a list of 3-Tuples
+    '''
+    ret = []
+    while len(bytelist) > 0:
+        T, L, V = first_TLV_parser(bytelist)
+        if T == 0xFF:
+            # padding bytes
+            break
+        ret.append((T, L, V))
+        # need to manage length of L
+        if L > 0xFE:
+            bytelist = bytelist[L+4:]
+        else:
+            bytelist = bytelist[L+2:]
+    return ret
+
+
+def dec_addr_tlv(hexstr):
+    """
+    Decode hex string to get EF.P-CSCF Address or EF.ePDGId or EF.ePDGIdEm.
+    See 3GPP TS 31.102 version 13.4.0 Release 13, section 4.2.8, 4.2.102 and 4.2.104.
+    """
+
+    # Convert from hex str to int bytes list
+    addr_tlv_bytes = h2i(hexstr)
+
+    # Get list of tuples containing parsed TLVs
+    tlvs = TLV_parser(addr_tlv_bytes)
+
+    for tlv in tlvs:
+        # tlv = (T, L, [V])
+        # T = Tag
+        # L = Length
+        # [V] = List of value
+
+        # Invalid Tag value scenario
+        if tlv[0] != 0x80:
+            continue
+
+        # Empty field - Zero length
+        if tlv[1] == 0:
+            continue
+
+        # Uninitialized field
+        if all([v == 0xff for v in tlv[2]]):
+            continue
+
+        # First byte in the value has the address type
+        addr_type = tlv[2][0]
+        # TODO: Support parsing of IPv6
+        # Address Type: 0x00 (FQDN), 0x01 (IPv4), 0x02 (IPv6), other (Reserved)
+        if addr_type == 0x00:  # FQDN
+            # Skip address tye byte i.e. first byte in value list
+            content = tlv[2][1:]
+            return (i2s(content), '00')
+
+        elif addr_type == 0x01:  # IPv4
+            # Skip address tye byte i.e. first byte in value list
+            # Skip the unused byte in Octect 4 after address type byte as per 3GPP TS 31.102
+            ipv4 = tlv[2][2:]
+            content = '.'.join(str(x) for x in ipv4)
+            return (content, '01')
+        else:
+            raise ValueError("Invalid address type")
+
+    return (None, None)
+
+
+def enc_addr_tlv(addr, addr_type='00'):
+    """
+    Encode address TLV object used in EF.P-CSCF Address, EF.ePDGId and EF.ePDGIdEm.
+    See 3GPP TS 31.102 version 13.4.0 Release 13, section 4.2.8, 4.2.102 and 4.2.104.
+
+    Default values:
+      - addr_type: 00 - FQDN format of Address
+    """
+
+    s = ""
+
+    # TODO: Encoding of IPv6 address
+    if addr_type == '00':  # FQDN
+        hex_str = s2h(addr)
+        s += '80' + ('%02x' % ((len(hex_str)//2)+1)) + '00' + hex_str
+    elif addr_type == '01':  # IPv4
+        ipv4_list = addr.split('.')
+        ipv4_str = ""
+        for i in ipv4_list:
+            ipv4_str += ('%02x' % (int(i)))
+
+        # Unused bytes shall be set to 'ff'. i.e 4th Octet after Address Type is not used
+        # IPv4 Address is in octet 5 to octet 8 of the TLV data object
+        s += '80' + ('%02x' % ((len(ipv4_str)//2)+2)) + '01' + 'ff' + ipv4_str
+
+    return s

pySim/ota.py

@@ -0,0 +1,443 @@
+"""Code related to SIM/UICC OTA according to TS 102 225 + TS 31.115."""
+
+# (C) 2021-2023 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from pySim.construct import *
+from pySim.utils import b2h
+from pySim.sms import UserDataHeader
+from construct import *
+import zlib
+import abc
+import struct
+from typing import Optional
+
+# ETS TS 102 225 gives the general command structure and the dialects for CAT_TP, TCP/IP and HTTPS
+# 3GPP TS 31.115 gives the dialects for SMS-PP, SMS-CB, USSD and HTTP
+
+# CPI CPL CHI CHL SPI KIc KID TAR CNTR PCNTR RC/CC/DS data
+
+#                                           CAT_TP  TCP/IP          SMS
+# CPI                                       0x01    0x01            =IEIa=70,len=0
+# CHI                                       NULL    NULL            NULL
+# CPI, CPL and CHL included in RC/CC/DS     true    true
+# RPI                                       0x02    0x02            =IEIa=71,len=0
+# RHI                                       NULL    NULL
+# RPI, RPL and RHL included in RC/CC/DS     true    true
+# packet-id                             0-bf,ff     0-bf,ff
+# identification packet                     false   102 225 tbl 6
+
+# KVN 1..f; KI1=KIc, KI2=KID, KI3=DEK
+
+# ETSI TS 102 225 Table 5 + 3GPP TS 31.115 Section 7
+ResponseStatus = Enum(Int8ub, por_ok=0, rc_cc_ds_failed=1, cntr_low=2, cntr_high=3,
+                      cntr_blocked=4, ciphering_error=5, undefined_security_error=6,
+                      insufficient_memory=7, more_time_needed=8, tar_unknown=9,
+                      insufficient_security_level=0x0A,
+                      actual_response_sms_submit=0x0B,
+                      actual_response_ussd=0x0C)
+
+# ETSI TS 102 226 Section 5.1.2
+CompactRemoteResp = Struct('number_of_commands'/Int8ub,
+                           'last_status_word'/HexAdapter(Bytes(2)),
+                           'last_response_data'/HexAdapter(GreedyBytes))
+
+RC_CC_DS = Enum(BitsInteger(2), no_rc_cc_ds=0, rc=1, cc=2, ds=3)
+
+# TS 102 225 Section 5.1.1 + TS 31.115 Section 4.2
+SPI = BitStruct(  # first octet
+    Padding(3),
+    'counter'/Enum(BitsInteger(2), no_counter=0, counter_no_replay_or_seq=1,
+                   counter_must_be_higher=2, counter_must_be_lower=3),
+    'ciphering'/Flag,
+    'rc_cc_ds'/RC_CC_DS,
+    # second octet
+    Padding(2),
+    'por_in_submit'/Flag,
+    'por_shall_be_ciphered'/Flag,
+    'por_rc_cc_ds'/RC_CC_DS,
+    'por'/Enum(BitsInteger(2), no_por=0,
+               por_required=1, por_only_when_error=2)
+)
+
+# TS 102 225 Section 5.1.2
+KIC = BitStruct('key'/BitsInteger(4),
+                'algo'/Enum(BitsInteger(4), implicit=0, single_des=1, triple_des_cbc2=5, triple_des_cbc3=9,
+                            aes_cbc=2)
+                )
+
+# TS 102 225 Section 5.1.3.1
+KID_CC = BitStruct('key'/BitsInteger(4),
+                   'algo'/Enum(BitsInteger(4), implicit=0, single_des=1, triple_des_cbc2=5, triple_des_cbc3=9,
+                               aes_cmac=2)
+                   )
+
+# TS 102 225 Section 5.1.3.2
+KID_RC = BitStruct('key'/BitsInteger(4),
+                   'algo'/Enum(BitsInteger(4), implicit=0, crc16=1, crc32=5, proprietary=3)
+                   )
+
+SmsCommandPacket = Struct('cmd_pkt_len'/Int16ub,
+                          'cmd_hdr_len'/Int8ub,
+                          'spi'/SPI,
+                          'kic'/KIC,
+                          'kid'/Switch(this.spi.rc_cc_ds, {'cc': KID_CC, 'rc': KID_RC }),
+                          'tar'/Bytes(3),
+                          'secured_data'/GreedyBytes)
+
+class OtaKeyset:
+    """The OTA related data (key material, counter) to be used in encrypt/decrypt."""
+    def __init__(self, algo_crypt: str, kic_idx: int, kic: bytes,
+                 algo_auth: str, kid_idx: int, kid: bytes, cntr: int = 0):
+        self.algo_crypt = algo_crypt
+        self.kic = bytes(kic)
+        self.kic_idx = kic_idx
+        self.algo_auth = algo_auth
+        self.kid = bytes(kid)
+        self.kid_idx = kid_idx
+        self.cntr = cntr
+
+    @property
+    def auth(self):
+        """Return an instance of the matching OtaAlgoAuth."""
+        return OtaAlgoAuth.fromKeyset(self)
+
+    @property
+    def crypt(self):
+        """Return an instance of the matching OtaAlgoCrypt."""
+        return OtaAlgoCrypt.fromKeyset(self)
+
+class OtaCheckError(Exception):
+    pass
+
+class OtaDialect(abc.ABC):
+    """Base Class for OTA dialects such as SMS, BIP, ..."""
+
+    def _compute_sig_len(self, spi:SPI):
+        if spi['rc_cc_ds'] == 'no_rc_cc_ds':
+            return 0
+        elif spi['rc_cc_ds'] == 'rc': # CRC-32
+            return 4
+        elif spi['rc_cc_ds'] == 'cc': # Cryptographic Checksum (CC)
+            # TODO: this is not entirely correct, as in AES case it could be 4 or 8
+            return 8
+        else:
+            raise ValueError("Invalid rc_cc_ds: %s" % spi['rc_cc_ds'])
+
+    @abc.abstractmethod
+    def encode_cmd(self, otak: OtaKeyset, tar: bytes, apdu: bytes) -> bytes:
+        pass
+
+    @abc.abstractmethod
+    def decode_resp(self, otak: OtaKeyset, apdu: bytes) -> (object, Optional["CompactRemoteResp"]):
+        """Decode a response into a response packet and, if indicted (by a
+        response status of `"por_ok"`) a decoded response.
+
+        The response packet's common characteristics are not fully determined,
+        and (so far) completely proprietary per dialect."""
+        pass
+
+
+from Cryptodome.Cipher import DES, DES3, AES
+from Cryptodome.Hash import CMAC
+
+class OtaAlgo(abc.ABC):
+    iv = property(lambda self: bytes([0] * self.blocksize))
+    blocksize = None
+    enum_name = None
+
+    @staticmethod
+    def _get_padding(in_len: int, multiple: int, padding: int = 0):
+        """Return padding bytes towards multiple of N."""
+        if in_len % multiple == 0:
+            return b''
+        pad_cnt = multiple - (in_len % multiple)
+        return b'\x00' * pad_cnt
+
+    @staticmethod
+    def _pad_to_multiple(indat: bytes, multiple: int, padding: int = 0):
+        """Pad input bytes to multiple of N."""
+        return indat + OtaAlgo._get_padding(len(indat), multiple, padding)
+
+    def pad_to_blocksize(self, indat: bytes, padding: int = 0):
+        """Pad the given input data to multiple of the cipher block size."""
+        return self._pad_to_multiple(indat, self.blocksize, padding)
+
+    def __init__(self, otak: OtaKeyset):
+        self.otak = otak
+
+    def __str__(self):
+        return self.__class__.__name__
+
+class OtaAlgoCrypt(OtaAlgo, abc.ABC):
+    def __init__(self, otak: OtaKeyset):
+        if self.enum_name != otak.algo_crypt:
+            raise ValueError('Cannot use algorithm %s with key for %s' % (self.enum_name, otak.algo_crypt))
+        super().__init__(otak)
+
+    def encrypt(self, data:bytes) -> bytes:
+        """Encrypt given input bytes using the key material given in constructor."""
+        padded_data = self.pad_to_blocksize(data)
+        return self._encrypt(data)
+
+    def decrypt(self, data:bytes) -> bytes:
+        """Decrypt given input bytes using the key material given in constructor."""
+        return self._decrypt(data)
+
+    @abc.abstractmethod
+    def _encrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+        pass
+
+    @abc.abstractmethod
+    def _decrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+        pass
+
+    @classmethod
+    def fromKeyset(cls, otak: OtaKeyset) -> 'OtaAlgoCrypt':
+        """Resolve the class for the encryption algorithm of otak and instantiate it."""
+        for subc in cls.__subclasses__():
+            if subc.enum_name == otak.algo_crypt:
+                return subc(otak)
+        raise ValueError('No implementation for crypt algorithm %s' % otak.algo_auth)
+
+class OtaAlgoAuth(OtaAlgo, abc.ABC):
+    def __init__(self, otak: OtaKeyset):
+        if self.enum_name != otak.algo_auth:
+            raise ValueError('Cannot use algorithm %s with key for %s' % (self.enum_name, otak.algo_crypt))
+        super().__init__(otak)
+
+    def sign(self, data:bytes) -> bytes:
+        """Compute the CC/CR check bytes for the input data using key material
+        given in constructor."""
+        padded_data = self.pad_to_blocksize(data)
+        sig = self._sign(padded_data)
+        return sig
+
+    def check_sig(self, data:bytes, cc_received:bytes):
+        """Compute the CC/CR check bytes for the input data and compare against cc_received."""
+        cc = self.sign(data)
+        if cc_received != cc:
+            raise OtaCheckError('Received CC (%s) != Computed CC (%s)' % (b2h(cc_received), b2h(cc)))
+
+    @abc.abstractmethod
+    def _sign(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+        pass
+
+    @classmethod
+    def fromKeyset(cls, otak: OtaKeyset) -> 'OtaAlgoAuth':
+        """Resolve the class for the authentication algorithm of otak and instantiate it."""
+        for subc in cls.__subclasses__():
+            if subc.enum_name == otak.algo_auth:
+                return subc(otak)
+        raise ValueError('No implementation for auth algorithm %s' % otak.algo_auth)
+
+class OtaAlgoCryptDES(OtaAlgoCrypt):
+    """DES is insecure.  For backwards compatibility with pre-Rel8"""
+    name = 'DES'
+    enum_name = 'single_des'
+    blocksize = 8
+    def _encrypt(self, data:bytes) -> bytes:
+        cipher = DES.new(self.otak.kic, DES.MODE_CBC, self.iv)
+        return cipher.encrypt(data)
+
+    def _decrypt(self, data:bytes) -> bytes:
+        cipher = DES.new(self.otak.kic, DES.MODE_CBC, self.iv)
+        return cipher.decrypt(data)
+
+class OtaAlgoAuthDES(OtaAlgoAuth):
+    """DES is insecure.  For backwards compatibility with pre-Rel8"""
+    name = 'DES'
+    enum_name = 'single_des'
+    blocksize = 8
+    def _sign(self, data:bytes) -> bytes:
+        cipher = DES.new(self.otak.kid, DES.MODE_CBC, self.iv)
+        ciph = cipher.encrypt(data)
+        return ciph[len(ciph) - 8:]
+
+class OtaAlgoCryptDES3(OtaAlgoCrypt):
+    name = '3DES'
+    enum_name = 'triple_des_cbc2'
+    blocksize = 8
+    def _encrypt(self, data:bytes) -> bytes:
+        cipher = DES3.new(self.otak.kic, DES3.MODE_CBC, self.iv)
+        return cipher.encrypt(data)
+
+    def _decrypt(self, data:bytes) -> bytes:
+        cipher = DES3.new(self.otak.kic, DES3.MODE_CBC, self.iv)
+        return cipher.decrypt(data)
+
+class OtaAlgoAuthDES3(OtaAlgoAuth):
+    name = '3DES'
+    enum_name = 'triple_des_cbc2'
+    blocksize = 8
+    def _sign(self, data:bytes) -> bytes:
+        cipher = DES3.new(self.otak.kid, DES3.MODE_CBC, self.iv)
+        ciph = cipher.encrypt(data)
+        return ciph[len(ciph) - 8:]
+
+class OtaAlgoCryptAES(OtaAlgoCrypt):
+    name = 'AES'
+    enum_name = 'aes_cbc'
+    blocksize = 16 # TODO: is this needed?
+    def _encrypt(self, data:bytes) -> bytes:
+        cipher = AES.new(self.otak.kic, AES.MODE_CBC, self.iv)
+        return cipher.encrypt(data)
+
+    def _decrypt(self, data:bytes) -> bytes:
+        cipher = AES.new(self.otak.kic, AES.MODE_CBC, self.iv)
+        return cipher.decrypt(data)
+
+class OtaAlgoAuthAES(OtaAlgoAuth):
+    name = 'AES'
+    enum_name = 'aes_cmac'
+    blocksize = 1 # AES CMAC doesn't need any padding by us
+    def _sign(self, data:bytes) -> bytes:
+        cmac = CMAC.new(self.otak.kid, ciphermod=AES, mac_len=8)
+        cmac.update(data)
+        ciph = cmac.digest()
+        return ciph[len(ciph) - 8:]
+
+
+
+class OtaDialectSms(OtaDialect):
+    """OTA dialect for SMS based transport, as described in 3GPP TS 31.115."""
+    SmsResponsePacket = Struct('rpl'/Int16ub,
+                               'rhl'/Int8ub,
+                               'tar'/Bytes(3),
+                               'cntr'/Bytes(5),
+                               'pcntr'/Int8ub,
+                               'response_status'/ResponseStatus,
+                               'cc_rc'/Bytes(this.rhl-10),
+                               'secured_data'/GreedyBytes)
+
+    def encode_cmd(self, otak: OtaKeyset, tar: bytes, spi: dict, apdu: bytes) -> bytes:
+        # length of signature in octets
+        len_sig = self._compute_sig_len(spi)
+        pad_cnt = 0
+        if spi['ciphering']: # ciphering is requested
+            # append padding bytes to end up with blocksize
+            len_cipher = 6 + len_sig + len(apdu)
+            padding = otak.crypt._get_padding(len_cipher, otak.crypt.blocksize)
+            pad_cnt = len(padding)
+            apdu += padding
+
+        kic = {'key': otak.kic_idx, 'algo': otak.algo_crypt}
+        kid = {'key': otak.kid_idx, 'algo': otak.algo_auth}
+
+        # CHL = number of octets from (and including) SPI to the end of RC/CC/DS
+        # 13 == SPI(2) + KIc(1) + KId(1) + TAR(3) + CNTR(5) + PCNTR(1)
+        chl = 13 + len_sig
+
+        # CHL + SPI (+ KIC + KID)
+        c = Struct('chl'/Int8ub, 'spi'/SPI, 'kic'/KIC, 'kid'/KID_CC, 'tar'/Bytes(3))
+        part_head = c.build({'chl': chl, 'spi':spi, 'kic':kic, 'kid':kid, 'tar':tar})
+        #print("part_head: %s" % b2h(part_head))
+
+        # CNTR + PCNTR (CNTR not used)
+        part_cnt = otak.cntr.to_bytes(5, 'big') + pad_cnt.to_bytes(1, 'big')
+        #print("part_cnt: %s" % b2h(part_cnt))
+
+        envelope_data = part_head + part_cnt + apdu
+        #print("envelope_data: %s" % b2h(envelope_data))
+
+        # 2-byte CPL. CPL is part of RC/CC/CPI to end of secured data, including any padding for ciphering
+        # CPL from and including CPI to end of secured data, including any padding for ciphering
+        cpl = len(envelope_data) + len_sig
+        envelope_data = cpl.to_bytes(2, 'big') + envelope_data
+        #print("envelope_data with cpl: %s" % b2h(envelope_data))
+
+        if spi['rc_cc_ds'] == 'cc':
+            cc = otak.auth.sign(envelope_data)
+            envelope_data = part_cnt + cc + apdu
+        elif spi['rc_cc_ds'] == 'rc':
+            # CRC32
+            crc32 = zlib.crc32(envelope_data) & 0xffffffff
+            envelope_data = part_cnt + crc32.to_bytes(4, 'big') + apdu
+        elif spi['rc_cc_ds'] == 'no_rc_cc_ds':
+            envelope_data = part_cnt + apdu
+        else:
+            raise ValueError("Invalid rc_cc_ds: %s" % spi['rc_cc_ds'])
+
+        #print("envelope_data with sig: %s" % b2h(envelope_data))
+
+        # encrypt as needed
+        if spi['ciphering']: # ciphering is requested
+            ciph = otak.crypt.encrypt(envelope_data)
+            envelope_data = part_head + ciph
+            # prefix with another CPL
+            cpl = len(envelope_data)
+            envelope_data = cpl.to_bytes(2, 'big') + envelope_data
+        else:
+            envelope_data = part_head + envelope_data
+
+        #print("envelope_data: %s" % b2h(envelope_data))
+
+        return envelope_data
+
+    def decode_resp(self, otak: OtaKeyset, spi: dict, data: bytes) -> ("OtaDialectSms.SmsResponsePacket", Optional["CompactRemoteResp"]):
+        if isinstance(data, str):
+            data = h2b(data)
+        # plain-text POR:   027100000e0ab000110000000000000001612f
+        # UDHL RPI IEDLa  RPL  RHL TAR    CNTR       PCNTR STS
+        # 02   71  00     000e 0a  b00011 0000000000 00    00  01 612f
+        # POR with CC:      027100001612b000110000000000000055f47118381175fb01612f
+        # POR with CC+CIPH: 027100001c12b000119660ebdb81be189b5e4389e9e7ab2bc0954f963ad869ed7c
+        if data[0] != 0x02:
+            raise ValueError('Unexpected UDL=0x%02x' % data[0])
+        udhd, remainder = UserDataHeader.fromBytes(data)
+        if not udhd.has_ie(0x71):
+            raise ValueError('RPI 0x71 not found in UDH')
+        rph_rhl_tar = remainder[:6] # RPH+RHL+TAR; not ciphered
+        res = self.SmsResponsePacket.parse(remainder)
+
+        if spi['por_shall_be_ciphered']:
+            # decrypt
+            ciphered_part = remainder[6:]
+            deciph = otak.crypt.decrypt(ciphered_part)
+            temp_data = rph_rhl_tar + deciph
+            res = self.SmsResponsePacket.parse(temp_data)
+            # remove specified number of padding bytes, if any
+            if res['pcntr'] != 0:
+                # this conditional is needed as python [:-0] renders an empty return!
+                res['secured_data'] = res['secured_data'][:-res['pcntr']]
+            remainder = temp_data
+
+        # is there a CC/RC present?
+        len_sig = res['rhl'] - 10
+        if spi['por_rc_cc_ds'] == 'no_rc_cc_ds':
+            if len_sig:
+                raise OtaCheckError('No RC/CC/DS requested, but len_sig=%u' % len_sig)
+        elif spi['por_rc_cc_ds'] == 'cc':
+            # verify signature
+            # UDH is part of CC/RC!
+            udh = data[:3]
+            # RPL, RHL, TAR, CNTR, PCNTR and STSare part of CC/RC
+            rpl_rhl_tar_cntr_pcntr_sts = remainder[:13]
+            # remove the CC/RC bytes
+            temp_data = udh + rpl_rhl_tar_cntr_pcntr_sts + remainder[13+len_sig:]
+            otak.auth.check_sig(temp_data, res['cc_rc'])
+        # TODO: CRC
+        else:
+            raise OtaCheckError('Unknown por_rc_cc_ds: %s' % spi['por_rc_cc_ds'])
+
+        # TODO: ExpandedRemoteResponse according to TS 102 226 5.2.2
+        if res.response_status == 'por_ok':
+            dec = CompactRemoteResp.parse(res['secured_data'])
+        else:
+            dec = None
+        return (res, dec)

pySim/profile.py

@@ -26,9 +26,12 @@ from pySim.filesystem import CardApplication, interpret_sw
 from pySim.utils import all_subclasses
 import abc
 import operator
+from typing import List
 
 
-def _mf_select_test(scc: SimCardCommands, cla_byte: str, sel_ctrl: str) -> bool:
+def _mf_select_test(scc: SimCardCommands,
+                    cla_byte: str, sel_ctrl: str,
+                    fids: List[str]) -> bool:
     cla_byte_bak = scc.cla_byte
     sel_ctrl_bak = scc.sel_ctrl
     scc.reset_card()
@@ -37,7 +40,8 @@ def _mf_select_test(scc: SimCardCommands, cla_byte: str, sel_ctrl: str) -> bool:
     scc.sel_ctrl = sel_ctrl
     rc = True
     try:
-        scc.select_file('3f00')
+        for fid in fids:
+            scc.select_file(fid)
     except:
         rc = False
 
@@ -51,7 +55,7 @@ def match_uicc(scc: SimCardCommands) -> bool:
     """ Try to access MF via UICC APDUs (3GPP TS 102.221), if this works, the
     card is considered a UICC card.
     """
-    return _mf_select_test(scc, "00", "0004")
+    return _mf_select_test(scc, "00", "0004", ["3f00"])
 
 
 def match_sim(scc: SimCardCommands) -> bool:
@@ -59,7 +63,14 @@ def match_sim(scc: SimCardCommands) -> bool:
     is also a simcard. This will be the case for most UICC cards, but there may
     also be plain UICC cards without 2G support as well.
     """
-    return _mf_select_test(scc, "a0", "0000")
+    return _mf_select_test(scc, "a0", "0000", ["3f00"])
+
+
+def match_ruim(scc: SimCardCommands) -> bool:
+    """ Try to access MF/DF.CDMA via 2G APDUs (3GPP TS 11.11), if this works,
+    the card is considered an R-UIM card for CDMA.
+    """
+    return _mf_select_test(scc, "a0", "0000", ["3f00", "7f25"])
 
 
 class CardProfile:
@@ -77,6 +88,7 @@ class CardProfile:
                 shell_cmdsets : List of cmd2 shell command sets of profile-specific commands
                 cla : class byte that should be used with cards of this profile
                 sel_ctrl : selection control bytes class byte that should be used with cards of this profile
+                addons: List of optional CardAddons that a card of this profile might have
         """
         self.name = name
         self.desc = kw.get("desc", None)
@@ -86,6 +98,8 @@ class CardProfile:
         self.shell_cmdsets = kw.get("shell_cmdsets", [])
         self.cla = kw.get("cla", "00")
         self.sel_ctrl = kw.get("sel_ctrl", "0004")
+        # list of optional addons that a card of this profile might have
+        self.addons = kw.get("addons", [])
 
     def __str__(self):
         return self.name
@@ -150,3 +164,34 @@ class CardProfile:
                 return p()
 
         return None
+
+    def add_addon(self, addon: 'CardProfileAddon'):
+        assert(addon not in self.addons)
+        # we don't install any additional files, as that is happening in the RuntimeState.
+        self.addons.append(addon)
+
+class CardProfileAddon(abc.ABC):
+    """A Card Profile Add-on is something that is not a card application or a full stand-alone
+    card profile, but an add-on to an existing profile.  Think of GSM-R specific files existing
+    on what is otherwise a SIM or USIM+SIM card."""
+
+    def __init__(self, name: str, **kw):
+        """
+        Args:
+                desc (str) : Description
+                files_in_mf : List of CardEF instances present in MF
+                shell_cmdsets : List of cmd2 shell command sets of profile-specific commands
+        """
+        self.name = name
+        self.desc = kw.get("desc", None)
+        self.files_in_mf = kw.get("files_in_mf", [])
+        self.shell_cmdsets = kw.get("shell_cmdsets", [])
+        pass
+
+    def __str__(self):
+        return self.name
+
+    @abc.abstractmethod
+    def probe(self, card: 'CardBase') -> bool:
+        """Probe a given card to determine whether or not this add-on is present/supported."""
+        pass

pySim/runtime.py

@@ -0,0 +1,546 @@
+# coding=utf-8
+"""Representation of the runtime state of an application like pySim-shell.
+"""
+
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import Optional, Tuple
+
+from pySim.utils import sw_match, h2b, i2h, is_hex, bertlv_parse_one, Hexstr
+from pySim.exceptions import *
+from pySim.filesystem import *
+
+def lchan_nr_from_cla(cla: int) -> int:
+    """Resolve the logical channel number from the CLA byte."""
+    # TS 102 221 10.1.1 Coding of Class Byte
+    if cla >> 4 in [0x0, 0xA, 0x8]:
+        # Table 10.3
+        return cla & 0x03
+    elif cla & 0xD0 in [0x40, 0xC0]:
+        # Table 10.4a
+        return 4 + (cla & 0x0F)
+    else:
+        raise ValueError('Could not determine logical channel for CLA=%2X' % cla)
+
+class RuntimeState:
+    """Represent the runtime state of a session with a card."""
+
+    def __init__(self, card: 'CardBase', profile: 'CardProfile'):
+        """
+        Args:
+            card : pysim.cards.Card instance
+            profile : CardProfile instance
+        """
+        self.mf = CardMF(profile=profile)
+        self.card = card
+        self.profile = profile
+        self.lchan = {}
+        # the basic logical channel always exists
+        self.lchan[0] = RuntimeLchan(0, self)
+
+        # make sure the class and selection control bytes, which are specified
+        # by the card profile are used
+        self.card.set_apdu_parameter(
+            cla=self.profile.cla, sel_ctrl=self.profile.sel_ctrl)
+
+        for addon_cls in self.profile.addons:
+            addon = addon_cls()
+            if addon.probe(self.card):
+                print("Detected %s Add-on \"%s\"" % (self.profile, addon))
+                for f in addon.files_in_mf:
+                    self.mf.add_file(f)
+
+        # go back to MF before the next steps (addon probing might have changed DF)
+        self.lchan[0].select('MF')
+
+        # add application ADFs + MF-files from profile
+        apps = self._match_applications()
+        for a in apps:
+            if a.adf:
+                self.mf.add_application_df(a.adf)
+        for f in self.profile.files_in_mf:
+            self.mf.add_file(f)
+        self.conserve_write = True
+
+        # make sure that when the runtime state is created, the card is also
+        # in a defined state.
+        self.reset()
+
+    def _match_applications(self):
+        """match the applications from the profile with applications on the card"""
+        apps_profile = self.profile.applications
+
+        # When the profile does not feature any applications, then we are done already
+        if not apps_profile:
+            return []
+
+        # Read AIDs from card and match them against the applications defined by the
+        # card profile
+        aids_card = self.card.read_aids()
+        apps_taken = []
+        if aids_card:
+            aids_taken = []
+            print("AIDs on card:")
+            for a in aids_card:
+                for f in apps_profile:
+                    if f.aid in a:
+                        print(" %s: %s (EF.DIR)" % (f.name, a))
+                        aids_taken.append(a)
+                        apps_taken.append(f)
+            aids_unknown = set(aids_card) - set(aids_taken)
+            for a in aids_unknown:
+                print(" unknown: %s (EF.DIR)" % a)
+        else:
+            print("warning: EF.DIR seems to be empty!")
+
+        # Some card applications may not be registered in EF.DIR, we will actively
+        # probe for those applications
+        for f in sorted(set(apps_profile) - set(apps_taken), key=str):
+            try:
+                # we can not use the lchan provided methods select, or select_file
+                # since those method work on an already finished file model. At
+                # this point we are still in the initialization process, so it is
+                # no problem when we access the card object directly without caring
+                # about updating other states. For normal selects at runtime, the
+                # caller must use the lchan provided methods select or select_file!
+                data, sw = self.card.select_adf_by_aid(f.aid)
+                self.selected_adf = f
+                if sw == "9000":
+                    print(" %s: %s" % (f.name, f.aid))
+                    apps_taken.append(f)
+            except (SwMatchError, ProtocolError):
+                pass
+        return apps_taken
+
+    def reset(self, cmd_app=None) -> Hexstr:
+        """Perform physical card reset and obtain ATR.
+        Args:
+            cmd_app : Command Application State (for unregistering old file commands)
+        """
+        # delete all lchan != 0 (basic lchan)
+        for lchan_nr in list(self.lchan.keys()):
+            if lchan_nr == 0:
+                continue
+            del self.lchan[lchan_nr]
+        atr = i2h(self.card.reset())
+        # select MF to reset internal state and to verify card really works
+        self.lchan[0].select('MF', cmd_app)
+        self.lchan[0].selected_adf = None
+        return atr
+
+    def add_lchan(self, lchan_nr: int) -> 'RuntimeLchan':
+        """Add a logical channel to the runtime state.  You shouldn't call this
+        directly but always go through RuntimeLchan.add_lchan()."""
+        if lchan_nr in self.lchan.keys():
+            raise ValueError('Cannot create already-existing lchan %d' % lchan_nr)
+        self.lchan[lchan_nr] = RuntimeLchan(lchan_nr, self)
+        return self.lchan[lchan_nr]
+
+    def del_lchan(self, lchan_nr: int):
+        if lchan_nr in self.lchan.keys():
+            del self.lchan[lchan_nr]
+            return True
+        else:
+            return False
+
+    def get_lchan_by_cla(self, cla) -> Optional['RuntimeLchan']:
+        lchan_nr = lchan_nr_from_cla(cla)
+        if lchan_nr in self.lchan.keys():
+            return self.lchan[lchan_nr]
+        else:
+            return None
+
+
+class RuntimeLchan:
+    """Represent the runtime state of a logical channel with a card."""
+
+    def __init__(self, lchan_nr: int, rs: RuntimeState):
+        self.lchan_nr = lchan_nr
+        self.rs = rs
+        self.scc = self.rs.card._scc.fork_lchan(lchan_nr)
+
+        # File reference data
+        self.selected_file = self.rs.mf
+        self.selected_adf = None
+        self.selected_file_fcp = None
+        self.selected_file_fcp_hex = None
+
+    def add_lchan(self, lchan_nr: int) -> 'RuntimeLchan':
+        """Add a new logical channel from the current logical channel. Just affects
+        internal state, doesn't actually open a channel with the UICC."""
+        new_lchan = self.rs.add_lchan(lchan_nr)
+        # See TS 102 221 Table 8.3
+        if self.lchan_nr != 0:
+            new_lchan.selected_file = self.get_cwd()
+            new_lchan.selected_adf = self.selected_adf
+        return new_lchan
+
+    def selected_file_descriptor_byte(self) -> dict:
+        return self.selected_file_fcp['file_descriptor']['file_descriptor_byte']
+
+    def selected_file_shareable(self) -> bool:
+        return self.selected_file_descriptor_byte()['shareable']
+
+    def selected_file_structure(self) -> str:
+        return self.selected_file_descriptor_byte()['structure']
+
+    def selected_file_type(self) -> str:
+        return self.selected_file_descriptor_byte()['file_type']
+
+    def selected_file_num_of_rec(self) -> Optional[int]:
+        return self.selected_file_fcp['file_descriptor'].get('num_of_rec')
+
+    def get_cwd(self) -> CardDF:
+        """Obtain the current working directory.
+
+        Returns:
+            CardDF instance
+        """
+        if isinstance(self.selected_file, CardDF):
+            return self.selected_file
+        else:
+            return self.selected_file.parent
+
+    def get_application_df(self) -> Optional[CardADF]:
+        """Obtain the currently selected application DF (if any).
+
+        Returns:
+            CardADF() instance or None"""
+        # iterate upwards from selected file; check if any is an ADF
+        node = self.selected_file
+        while node.parent != node:
+            if isinstance(node, CardADF):
+                return node
+            node = node.parent
+        return None
+
+    def interpret_sw(self, sw: str):
+        """Interpret a given status word relative to the currently selected application
+        or the underlying card profile.
+
+        Args:
+            sw : Status word as string of 4 hex digits
+
+        Returns:
+            Tuple of two strings
+        """
+        res = None
+        adf = self.get_application_df()
+        if adf:
+            app = adf.application
+            # The application either comes with its own interpret_sw
+            # method or we will use the interpret_sw method from the
+            # card profile.
+            if app and hasattr(app, "interpret_sw"):
+                res = app.interpret_sw(sw)
+        return res or self.rs.profile.interpret_sw(sw)
+
+    def probe_file(self, fid: str, cmd_app=None):
+        """Blindly try to select a file and automatically add a matching file
+           object if the file actually exists."""
+        if not is_hex(fid, 4, 4):
+            raise ValueError(
+                "Cannot select unknown file by name %s, only hexadecimal 4 digit FID is allowed" % fid)
+
+        self._select_pre(cmd_app)
+
+        try:
+            # We access the card through the select_file method of the scc object.
+            # If we succeed, we know that the file exists on the card and we may
+            # proceed with creating a new CardEF object in the local file model at
+            # run time. In case the file does not exist on the card, we just abort.
+            # The state on the card (selected file/application) wont't be changed,
+            # so we do not have to update any state in that case.
+            (data, sw) = self.scc.select_file(fid)
+        except SwMatchError as swm:
+            self._select_post(cmd_app)
+            k = self.interpret_sw(swm.sw_actual)
+            if not k:
+                raise(swm)
+            raise RuntimeError("%s: %s - %s" % (swm.sw_actual, k[0], k[1]))
+
+        select_resp = self.selected_file.decode_select_response(data)
+        if (select_resp['file_descriptor']['file_descriptor_byte']['file_type'] == 'df'):
+            f = CardDF(fid=fid, sfid=None, name="DF." + str(fid).upper(),
+                       desc="dedicated file, manually added at runtime")
+        else:
+            if (select_resp['file_descriptor']['file_descriptor_byte']['structure'] == 'transparent'):
+                f = TransparentEF(fid=fid, sfid=None, name="EF." + str(fid).upper(),
+                                  desc="elementary file, manually added at runtime")
+            else:
+                f = LinFixedEF(fid=fid, sfid=None, name="EF." + str(fid).upper(),
+                               desc="elementary file, manually added at runtime")
+
+        self.selected_file.add_files([f])
+
+        self._select_post(cmd_app, f, data)
+
+    def _select_pre(self, cmd_app):
+        # unregister commands of old file
+        if cmd_app and self.selected_file.shell_commands:
+            for c in self.selected_file.shell_commands:
+                cmd_app.unregister_command_set(c)
+
+    def _select_post(self, cmd_app, file:Optional[CardFile] = None, select_resp_data = None):
+        # we store some reference data (see above) about the currently selected file.
+        # This data must be updated after every select.
+        if file:
+            self.selected_file = file
+            if isinstance(file, CardADF):
+                self.selected_adf = file
+            if select_resp_data:
+                self.selected_file_fcp_hex = select_resp_data
+                self.selected_file_fcp = self.selected_file.decode_select_response(select_resp_data)
+
+        # register commands of new file
+        if cmd_app and self.selected_file.shell_commands:
+            for c in self.selected_file.shell_commands:
+                cmd_app.register_command_set(c)
+
+    def select_file(self, file: CardFile, cmd_app=None):
+        """Select a file (EF, DF, ADF, MF, ...).
+
+        Args:
+            file : CardFile [or derived class] instance
+            cmd_app : Command Application State (for unregistering old file commands)
+        """
+        # we need to find a path from our self.selected_file to the destination
+        inter_path = self.selected_file.build_select_path_to(file)
+        if not inter_path:
+            raise RuntimeError('Cannot determine path from %s to %s' % (self.selected_file, file))
+        self._select_pre(cmd_app)
+
+        # be sure the variables that we pass to _select_post contain valid values.
+        selected_file = self.selected_file
+        data = self.selected_file_fcp_hex
+
+        for f in inter_path:
+            try:
+                # We now directly accessing the card to perform the selection. This
+                # will change the state of the card, so we must take care to update
+                # the local state (lchan) as well. This is done in the method
+                # _select_post. It should be noted that the caller must always use
+                # the methods select_file or select. The caller must not access the
+                # card directly since this would lead into an incoherence of the
+                # card state and the state of the lchan.
+                if isinstance(f, CardADF):
+                    (data, sw) = self.rs.card.select_adf_by_aid(f.aid, scc=self.scc)
+                else:
+                    (data, sw) = self.scc.select_file(f.fid)
+                selected_file = f
+            except SwMatchError as swm:
+                self._select_post(cmd_app, selected_file, data)
+                raise(swm)
+
+        self._select_post(cmd_app, f, data)
+
+    def select(self, name: str, cmd_app=None):
+        """Select a file (EF, DF, ADF, MF, ...).
+
+        Args:
+            name : Name of file to select
+            cmd_app : Command Application State (for unregistering old file commands)
+        """
+        # if any intermediate step fails, we must be able to go back where we were
+        prev_sel_file = self.selected_file
+
+        # handling of entire paths with multiple directories/elements
+        if '/' in name:
+            pathlist = name.split('/')
+            # treat /DF.GSM/foo like MF/DF.GSM/foo
+            if pathlist[0] == '':
+                pathlist[0] = 'MF'
+            try:
+                for p in pathlist:
+                    self.select(p, cmd_app)
+                return self.selected_file_fcp
+            except Exception as e:
+                self.select_file(prev_sel_file, cmd_app)
+                raise e
+
+        # we are now in the directory where the target file is located
+        # so we can now refer to the get_selectables() method to get the
+        # file object and select it using select_file()
+        sels = self.selected_file.get_selectables()
+        if is_hex(name):
+            name = name.lower()
+
+        try:
+            if name in sels:
+                self.select_file(sels[name], cmd_app)
+            else:
+                self.probe_file(name, cmd_app)
+        except Exception as e:
+            self.select_file(prev_sel_file, cmd_app)
+            raise e
+
+        return self.selected_file_fcp
+
+    def status(self):
+        """Request STATUS (current selected file FCP) from card."""
+        (data, sw) = self.scc.status()
+        return self.selected_file.decode_select_response(data)
+
+    def get_file_for_selectable(self, name: str):
+        sels = self.selected_file.get_selectables()
+        return sels[name]
+
+    def activate_file(self, name: str):
+        """Request ACTIVATE FILE of specified file."""
+        sels = self.selected_file.get_selectables()
+        f = sels[name]
+        data, sw = self.scc.activate_file(f.fid)
+        return data, sw
+
+    def read_binary(self, length: int = None, offset: int = 0):
+        """Read [part of] a transparent EF binary data.
+
+        Args:
+            length : Amount of data to read (None: as much as possible)
+            offset : Offset into the file from which to read 'length' bytes
+        Returns:
+            binary data read from the file
+        """
+        if not isinstance(self.selected_file, TransparentEF):
+            raise TypeError("Only works with TransparentEF")
+        return self.scc.read_binary(self.selected_file.fid, length, offset)
+
+    def read_binary_dec(self) -> Tuple[dict, str]:
+        """Read [part of] a transparent EF binary data and decode it.
+
+        Args:
+            length : Amount of data to read (None: as much as possible)
+            offset : Offset into the file from which to read 'length' bytes
+        Returns:
+            abstract decode data read from the file
+        """
+        (data, sw) = self.read_binary()
+        dec_data = self.selected_file.decode_hex(data)
+        return (dec_data, sw)
+
+    def update_binary(self, data_hex: str, offset: int = 0):
+        """Update transparent EF binary data.
+
+        Args:
+            data_hex : hex string of data to be written
+            offset : Offset into the file from which to write 'data_hex'
+        """
+        if not isinstance(self.selected_file, TransparentEF):
+            raise TypeError("Only works with TransparentEF")
+        return self.scc.update_binary(self.selected_file.fid, data_hex, offset, conserve=self.rs.conserve_write)
+
+    def update_binary_dec(self, data: dict):
+        """Update transparent EF from abstract data. Encodes the data to binary and
+        then updates the EF with it.
+
+        Args:
+            data : abstract data which is to be encoded and written
+        """
+        data_hex = self.selected_file.encode_hex(data)
+        return self.update_binary(data_hex)
+
+    def read_record(self, rec_nr: int = 0):
+        """Read a record as binary data.
+
+        Args:
+            rec_nr : Record number to read
+        Returns:
+            hex string of binary data contained in record
+        """
+        if not isinstance(self.selected_file, LinFixedEF):
+            raise TypeError("Only works with Linear Fixed EF")
+        # returns a string of hex nibbles
+        return self.scc.read_record(self.selected_file.fid, rec_nr)
+
+    def read_record_dec(self, rec_nr: int = 0) -> Tuple[dict, str]:
+        """Read a record and decode it to abstract data.
+
+        Args:
+            rec_nr : Record number to read
+        Returns:
+            abstract data contained in record
+        """
+        (data, sw) = self.read_record(rec_nr)
+        return (self.selected_file.decode_record_hex(data, rec_nr), sw)
+
+    def update_record(self, rec_nr: int, data_hex: str):
+        """Update a record with given binary data
+
+        Args:
+            rec_nr : Record number to read
+            data_hex : Hex string binary data to be written
+        """
+        if not isinstance(self.selected_file, LinFixedEF):
+            raise TypeError("Only works with Linear Fixed EF")
+        return self.scc.update_record(self.selected_file.fid, rec_nr, data_hex,
+					       conserve=self.rs.conserve_write,
+					       leftpad=self.selected_file.leftpad)
+
+    def update_record_dec(self, rec_nr: int, data: dict):
+        """Update a record with given abstract data.  Will encode abstract to binary data
+        and then write it to the given record on the card.
+
+        Args:
+            rec_nr : Record number to read
+            data_hex : Abstract data to be written
+        """
+        data_hex = self.selected_file.encode_record_hex(data, rec_nr)
+        return self.update_record(rec_nr, data_hex)
+
+    def retrieve_data(self, tag: int = 0):
+        """Read a DO/TLV as binary data.
+
+        Args:
+            tag : Tag of TLV/DO to read
+        Returns:
+            hex string of full BER-TLV DO including Tag and Length
+        """
+        if not isinstance(self.selected_file, BerTlvEF):
+            raise TypeError("Only works with BER-TLV EF")
+        # returns a string of hex nibbles
+        return self.scc.retrieve_data(self.selected_file.fid, tag)
+
+    def retrieve_tags(self):
+        """Retrieve tags available on BER-TLV EF.
+
+        Returns:
+            list of integer tags contained in EF
+        """
+        if not isinstance(self.selected_file, BerTlvEF):
+            raise TypeError("Only works with BER-TLV EF")
+        data, sw = self.scc.retrieve_data(self.selected_file.fid, 0x5c)
+        tag, length, value, remainder = bertlv_parse_one(h2b(data))
+        return list(value)
+
+    def set_data(self, tag: int, data_hex: str):
+        """Update a TLV/DO with given binary data
+
+        Args:
+            tag : Tag of TLV/DO to be written
+            data_hex : Hex string binary data to be written (value portion)
+        """
+        if not isinstance(self.selected_file, BerTlvEF):
+            raise TypeError("Only works with BER-TLV EF")
+        return self.scc.set_data(self.selected_file.fid, tag, data_hex, conserve=self.rs.conserve_write)
+
+    def unregister_cmds(self, cmd_app=None):
+        """Unregister all file specific commands."""
+        if cmd_app and self.selected_file.shell_commands:
+            for c in self.selected_file.shell_commands:
+                cmd_app.unregister_command_set(c)
+
+
+

pySim/sms.py

@@ -0,0 +1,402 @@
+"""Code related to SMS Encoding/Decoding"""
+# simplistic SMS T-PDU code, as unfortunately nobody bothered to port the python smspdu
+# module to python3, and I gave up after >= 3 hours of trying and failing to do so
+
+# (C) 2022 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import typing
+import abc
+from bidict import bidict
+from construct import Int8ub, Byte, Bytes, Bit, Flag, BitsInteger, Flag
+from construct import Struct, Enum, Tell, BitStruct, this, Padding
+from construct import Prefixed, GreedyRange, GreedyBytes
+
+from pySim.construct import HexAdapter, BcdAdapter, TonNpi
+from pySim.utils import Hexstr, h2b, b2h
+
+from smpp.pdu import pdu_types, operations
+
+BytesOrHex = typing.Union[Hexstr, bytes]
+
+class UserDataHeader:
+    # a single IE in the user data header
+    ie_c = Struct('iei'/Int8ub, 'length'/Int8ub, 'value'/Bytes(this.length))
+    # parser for the full UDH: Length octet followed by sequence of IEs
+    _construct = Struct('ies'/Prefixed(Int8ub, GreedyRange(ie_c)),
+                        'data'/GreedyBytes)
+
+    def __init__(self, ies=[]):
+        self.ies = ies
+
+    def __repr__(self) -> str:
+        return 'UDH(%r)' % self.ies
+
+    def has_ie(self, iei:int) -> bool:
+        for ie in self.ies:
+            if ie['iei'] == iei:
+                return True
+        return False
+
+    @classmethod
+    def fromBytes(cls, inb: BytesOrHex) -> typing.Tuple['UserDataHeader', bytes]:
+        if isinstance(inb, str):
+            inb = h2b(inb)
+        res = cls._construct.parse(inb)
+        return cls(res['ies']), res['data']
+
+    def toBytes(self) -> bytes:
+        return self._construct.build({'ies':self.ies, 'data':b''})
+
+
+def smpp_dcs_is_8bit(dcs: pdu_types.DataCoding) -> bool:
+    """Determine if the given SMPP data coding scheme is 8-bit or not."""
+    if dcs == pdu_types.DataCoding(pdu_types.DataCodingScheme.DEFAULT,
+                                   pdu_types.DataCodingDefault.OCTET_UNSPECIFIED):
+        return True
+    if dcs == pdu_types.DataCoding(pdu_types.DataCodingScheme.DEFAULT,
+                                   pdu_types.DataCodingDefault.OCTET_UNSPECIFIED_COMMON):
+        return True
+    # pySim/sms.py:72:21: E1101: Instance of 'DataCodingScheme' has no 'GSM_MESSAGE_CLASS' member (no-member)
+    # pylint: disable=no-member
+    if dcs.scheme == pdu_types.DataCodingScheme.GSM_MESSAGE_CLASS and dcs.schemeData['msgCoding'] == pdu_types.DataCodingGsmMsgCoding.DATA_8BIT:
+        return True
+    else:
+        return False
+
+def ensure_smpp_is_8bit(dcs: pdu_types.DataCoding):
+    """Assert if given SMPP data coding scheme is not 8-bit."""
+    if not smpp_dcs_is_8bit(dcs):
+        raise ValueError('We only support 8bit coded SMS for now')
+
+class AddressField:
+    """Representation of an address field as used in SMS T-PDU."""
+    _construct = Struct('addr_len'/Int8ub,
+                        'type_of_addr'/TonNpi,
+                        'digits'/BcdAdapter(Bytes(this.addr_len//2 + this.addr_len%2)),
+                        'tell'/Tell)
+    smpp_map_npi = bidict({
+        'UNKNOWN': 'unknown',
+        'ISDN': 'isdn_e164',
+        'DATA': 'data_x121',
+        'TELEX': 'telex_f69',
+        'LAND_MOBILE': 'sc_specific6',
+        'NATIONAL': 'national',
+        'PRIVATE': 'private',
+        'ERMES': 'ermes',
+        })
+    smpp_map_ton = bidict({
+        'UNKNOWN': 'unknown',
+        'INTERNATIONAL': 'international',
+        'NATIONAL': 'national',
+        'NETWORK_SPECIFIC': 'network_specific',
+        'SUBSCRIBER_NUMBER': 'short_code',
+        'ALPHANUMERIC': 'alphanumeric',
+        'ABBREVIATED': 'abbreviated',
+        })
+
+
+    def __init__(self, digits, ton='unknown', npi='unknown'):
+        self.ton = ton
+        self.npi = npi
+        self.digits = digits
+
+    def __str__(self):
+        return 'AddressField(TON=%s, NPI=%s, %s)' % (self.ton, self.npi, self.digits)
+
+    @classmethod
+    def fromBytes(cls, inb: BytesOrHex) -> typing.Tuple['AddressField', bytes]:
+        """Construct an AddressField instance from the binary T-PDU address format."""
+        if isinstance(inb, str):
+            inb = h2b(inb)
+        res = cls._construct.parse(inb)
+        #print("size: %s" % cls._construct.sizeof())
+        ton = res['type_of_addr']['type_of_number']
+        npi = res['type_of_addr']['numbering_plan_id']
+        # return resulting instance + remainder bytes
+        return cls(res['digits'][:res['addr_len']], ton, npi), inb[res['tell']:]
+
+    @classmethod
+    def fromSmpp(cls, addr, ton, npi) -> 'AddressField':
+        """Construct an AddressField from {source,dest}_addr_{,ton,npi} attributes of smpp.pdu."""
+        # return the resulting instance
+        return cls(addr.decode('ascii'), AddressField.smpp_map_ton[ton.name], AddressField.smpp_map_npi[npi.name])
+
+    def toSmpp(self):
+        """Return smpp.pdo.*.source,dest}_addr_{,ton,npi} attributes for given AddressField."""
+        return (self.digits, self.smpp_map_ton.inverse[self.ton], self.smpp_map_npi.inverse[self.npi])
+
+    def toBytes(self) -> bytes:
+        """Encode the AddressField into the binary representation as used in T-PDU."""
+        num_digits = len(self.digits)
+        if num_digits % 2:
+            self.digits += 'f'
+        d = {
+            'addr_len': num_digits,
+            'type_of_addr': {
+                'ext': True,
+                'type_of_number': self.ton,
+                'numbering_plan_id': self.npi,
+                },
+            'digits': self.digits,
+            }
+        return self._construct.build(d)
+
+
+class SMS_TPDU(abc.ABC):
+    """Base class for a SMS T-PDU."""
+    def __init__(self, **kwargs):
+        self.tp_mti = kwargs.get('tp_mti', None)
+        self.tp_rp = kwargs.get('tp_rp', False)
+        self.tp_udhi = kwargs.get('tp_udhi', False)
+        self.tp_pid = kwargs.get('tp_pid', None)
+        self.tp_dcs = kwargs.get('tp_dcs', None)
+        self.tp_udl = kwargs.get('tp_udl', None)
+        self.tp_ud = kwargs.get('tp_ud', None)
+
+
+
+class SMS_DELIVER(SMS_TPDU):
+    """Representation of a SMS-DELIVER T-PDU. This is the Network to MS/UE (downlink) direction."""
+    flags_construct = BitStruct('tp_rp'/Flag, 'tp_udhi'/Flag, 'tp_rp'/Flag, 'tp_sri'/Flag,
+                                Padding(1), 'tp_mms'/Flag, 'tp_mti'/BitsInteger(2))
+    def __init__(self, **kwargs):
+        kwargs['tp_mti'] = 0
+        super().__init__(**kwargs)
+        self.tp_lp = kwargs.get('tp_lp', False)
+        self.tp_mms = kwargs.get('tp_mms', False)
+        self.tp_oa = kwargs.get('tp_oa', None)
+        self.tp_scts = kwargs.get('tp_scts', None)
+        self.tp_sri = kwargs.get('tp_sri', False)
+
+    def __repr__(self):
+        return '%s(MTI=%s, MMS=%s, LP=%s, RP=%s, UDHI=%s, SRI=%s, OA=%s, PID=%2x, DCS=%x, SCTS=%s, UDL=%u, UD=%s)' % (self.__class__.__name__, self.tp_mti, self.tp_mms, self.tp_lp, self.tp_rp, self.tp_udhi, self.tp_sri, self.tp_oa, self.tp_pid, self.tp_dcs, self.tp_scts, self.tp_udl, self.tp_ud)
+
+    @classmethod
+    def fromBytes(cls, inb: BytesOrHex) -> 'SMS_DELIVER':
+        """Construct a SMS_DELIVER instance from the binary encoded format as used in T-PDU."""
+        if isinstance(inb, str):
+            inb = h2b(inb)
+        flags = inb[0]
+        d = SMS_DELIVER.flags_construct.parse(inb)
+        oa, remainder = AddressField.fromBytes(inb[1:])
+        d['tp_oa'] = oa
+        offset = 0
+        d['tp_pid'] = remainder[offset]
+        offset += 1
+        d['tp_dcs'] = remainder[offset]
+        offset += 1
+        # TODO: further decode
+        d['tp_scts'] = remainder[offset:offset+7]
+        offset += 7
+        d['tp_udl'] = remainder[offset]
+        offset += 1
+        d['tp_ud'] = remainder[offset:]
+        return cls(**d)
+
+    def toBytes(self) -> bytes:
+        """Encode a SMS_DELIVER instance to the binary encoded format as used in T-PDU."""
+        outb = bytearray()
+        d = {
+            'tp_mti': self.tp_mti, 'tp_mms': self.tp_mms, 'tp_lp': self.tp_lp,
+            'tp_rp': self.tp_rp, 'tp_udhi': self.tp_udhi, 'tp_sri': self.tp_sri,
+            }
+        flags = SMS_DELIVER.flags_construct.build(d)
+        outb.extend(flags)
+        outb.extend(self.tp_oa.toBytes())
+        outb.append(self.tp_pid)
+        outb.append(self.tp_dcs)
+        outb.extend(self.tp_scts)
+        outb.append(self.tp_udl)
+        outb.extend(self.tp_ud)
+
+        return outb
+
+    @classmethod
+    def fromSmpp(cls, smpp_pdu) -> 'SMS_DELIVER':
+        """Construct a SMS_DELIVER instance from the deliver format used by smpp.pdu."""
+        if smpp_pdu.id == pdu_types.CommandId.submit_sm:
+            return cls.fromSmppSubmit(smpp_pdu)
+        else:
+            raise ValueError('Unsupported SMPP commandId %s' % smpp_pdu.id)
+
+    @classmethod
+    def fromSmppSubmit(cls, smpp_pdu) -> 'SMS_DELIVER':
+        """Construct a SMS_DELIVER instance from the submit format used by smpp.pdu."""
+        ensure_smpp_is_8bit(smpp_pdu.params['data_coding'])
+        tp_oa = AddressField.fromSmpp(smpp_pdu.params['source_addr'],
+                                      smpp_pdu.params['source_addr_ton'],
+                                      smpp_pdu.params['source_addr_npi'])
+        tp_ud = smpp_pdu.params['short_message']
+        d = {
+            'tp_lp': False,
+            'tp_mms': False,
+            'tp_oa': tp_oa,
+            'tp_scts': h2b('22705200000000'), # FIXME
+            'tp_sri': False,
+            'tp_rp': False,
+            'tp_udhi': pdu_types.EsmClassGsmFeatures.UDHI_INDICATOR_SET in smpp_pdu.params['esm_class'].gsmFeatures,
+            'tp_pid': smpp_pdu.params['protocol_id'],
+            'tp_dcs': 0xF6, # we only deal with binary SMS here
+            'tp_udl': len(tp_ud),
+            'tp_ud': tp_ud,
+            }
+        return cls(**d)
+
+
+
+class SMS_SUBMIT(SMS_TPDU):
+    """Representation of a SMS-SUBMIT T-PDU. This is the MS/UE -> network (uplink) direction."""
+    flags_construct = BitStruct('tp_srr'/Flag, 'tp_udhi'/Flag, 'tp_rp'/Flag,
+                                'tp_vpf'/Enum(BitsInteger(2), none=0, relative=2, enhanced=1, absolute=3),
+                                'tp_rd'/Flag, 'tp_mti'/BitsInteger(2))
+    def __init__(self, **kwargs):
+        kwargs['tp_mti'] = 1
+        super().__init__(**kwargs)
+        self.tp_rd = kwargs.get('tp_rd', False)
+        self.tp_vpf = kwargs.get('tp_vpf', 'none')
+        self.tp_srr = kwargs.get('tp_srr', False)
+        self.tp_mr = kwargs.get('tp_mr', None)
+        self.tp_da = kwargs.get('tp_da', None)
+        self.tp_vp = kwargs.get('tp_vp', None)
+
+    def __repr__(self):
+        return '%s(MTI=%s, RD=%s, VPF=%u, RP=%s, UDHI=%s, SRR=%s, DA=%s, PID=%2x, DCS=%x, VP=%s, UDL=%u, UD=%s)' % (self.__class__.__name__, self.tp_mti, self.tp_rd, self.tp_vpf, self.tp_rp, self.tp_udhi, self.tp_srr, self.tp_da, self.tp_pid, self.tp_dcs, self.tp_vp, self.tp_udl, self.tp_ud)
+
+    @classmethod
+    def fromBytes(cls, inb:BytesOrHex) -> 'SMS_SUBMIT':
+        """Construct a SMS_SUBMIT instance from the binary encoded format as used in T-PDU."""
+        offset = 0
+        if isinstance(inb, str):
+            inb = h2b(inb)
+        d = SMS_SUBMIT.flags_construct.parse(inb)
+        offset += 1
+        d['tp_mr']= inb[offset]
+        offset += 1
+        da, remainder = AddressField.fromBytes(inb[2:])
+        d['tp_da'] = da
+
+        offset = 0
+        d['tp_pid'] = remainder[offset]
+        offset += 1
+        d['tp_dcs'] = remainder[offset]
+        offset += 1
+        if d['tp_vpf'] == 'none':
+            pass
+        elif d['tp_vpf'] == 'relative':
+            # TODO: further decode
+            d['tp_vp'] = remainder[offset:offset+1]
+            offset += 1
+        elif d['tp_vpf'] == 'enhanced':
+            # TODO: further decode
+            d['tp_vp'] = remainder[offset:offset+7]
+            offset += 7
+            pass
+        elif d['tp_vpf'] == 'absolute':
+            # TODO: further decode
+            d['tp_vp'] = remainder[offset:offset+7]
+            offset += 7
+            pass
+        else:
+            raise ValueError('Invalid VPF: %s' % d['tp_vpf'])
+        d['tp_udl'] = remainder[offset]
+        offset += 1
+        d['tp_ud'] = remainder[offset:]
+        return cls(**d)
+
+    def toBytes(self) -> bytes:
+        """Encode a SMS_SUBMIT instance to the binary encoded format as used in T-PDU."""
+        outb = bytearray()
+        d = {
+            'tp_mti': self.tp_mti, 'tp_rd': self.tp_rd, 'tp_vpf': self.tp_vpf,
+            'tp_rp': self.tp_rp, 'tp_udhi': self.tp_udhi, 'tp_srr': self.tp_srr,
+            }
+        flags = SMS_SUBMIT.flags_construct.build(d)
+        outb.extend(flags)
+        outb.append(self.tp_mr)
+        outb.extend(self.tp_da.toBytes())
+        outb.append(self.tp_pid)
+        outb.append(self.tp_dcs)
+        if self.tp_vpf != 'none':
+            outb.extend(self.tp_vp)
+        outb.append(self.tp_udl)
+        outb.extend(self.tp_ud)
+        return outb
+
+    @classmethod
+    def fromSmpp(cls, smpp_pdu) -> 'SMS_SUBMIT':
+        """Construct a SMS_SUBMIT instance from the format used by smpp.pdu."""
+        if smpp_pdu.id == pdu_types.CommandId.submit_sm:
+            return cls.fromSmppSubmit(smpp_pdu)
+        else:
+            raise ValueError('Unsupported SMPP commandId %s' % smpp_pdu.id)
+
+    @classmethod
+    def fromSmppSubmit(cls, smpp_pdu) -> 'SMS_SUBMIT':
+        """Construct a SMS_SUBMIT instance from the submit format used by smpp.pdu."""
+        ensure_smpp_is_8bit(smpp_pdu.params['data_coding'])
+        tp_da = AddressField.fromSmpp(smpp_pdu.params['destination_addr'],
+                                      smpp_pdu.params['dest_addr_ton'],
+                                      smpp_pdu.params['dest_addr_npi'])
+        tp_ud = smpp_pdu.params['short_message']
+        #vp_smpp = smpp_pdu.params['validity_period']
+        #if not vp_smpp:
+        #    vpf = 'none'
+        d = {
+            'tp_rd': True if smpp_pdu.params['replace_if_present_flag'].name == 'REPLACE' else False,
+            'tp_vpf': None, # vpf,
+            'tp_rp': False, # related to ['registered_delivery'] ?
+            'tp_udhi': pdu_types.EsmClassGsmFeatures.UDHI_INDICATOR_SET in smpp_pdu.params['esm_class'].gsmFeatures,
+            'tp_srr': True if smpp_pdu.params['registered_delivery'] else False,
+            'tp_mr': 0, # FIXME: sm_default_msg_id ?
+            'tp_da': tp_da,
+            'tp_pid': smpp_pdu.params['protocol_id'],
+            'tp_dcs': 0xF6, # FIXME: we only deal with binary SMS here
+            'tp_vp': None, # FIXME: implement VPF conversion
+            'tp_udl': len(tp_ud),
+            'tp_ud': tp_ud,
+            }
+        return cls(**d)
+
+    def toSmpp(self) -> pdu_types.PDU:
+        """Translate a SMS_SUBMIT instance to a smpp.pdu.operations.SubmitSM instance."""
+        esm_class = pdu_types.EsmClass(pdu_types.EsmClassMode.DEFAULT, pdu_types.EsmClassType.DEFAULT)
+        reg_del = pdu_types.RegisteredDelivery(pdu_types.RegisteredDeliveryReceipt.NO_SMSC_DELIVERY_RECEIPT_REQUESTED)
+        if self.tp_rp:
+            repl_if = pdu_types.ReplaceIfPresentFlag.REPLACE
+        else:
+            repl_if = pdu_types.ReplaceIfPresentFlag.DO_NOT_REPLACE
+        # we only deal with binary SMS here:
+        if self.tp_dcs != 0xF6:
+            raise ValueError('Unsupported DCS: We only support DCS=0xF6 for now')
+        dc = pdu_types.DataCoding(pdu_types.DataCodingScheme.DEFAULT, pdu_types.DataCodingDefault.OCTET_UNSPECIFIED)
+        (daddr, ton, npi) = self.tp_da.toSmpp()
+        return operations.SubmitSM(service_type='',
+                                   source_addr_ton=pdu_types.AddrTon.ALPHANUMERIC,
+                                   source_addr_npi=pdu_types.AddrNpi.UNKNOWN,
+                                   source_addr='simcard',
+                                   dest_addr_ton=ton,
+                                   dest_addr_npi=npi,
+                                   destination_addr=daddr,
+                                   esm_class=esm_class,
+                                   protocol_id=self.tp_pid,
+                                   priority_flag=pdu_types.PriorityFlag.LEVEL_0,
+                                   #schedule_delivery_time,
+                                   #validity_period,
+                                   registered_delivery=reg_del,
+                                   replace_if_present_flag=repl_if,
+                                   data_coding=dc,
+                                   #sm_default_msg_id,
+                                   short_message=self.tp_ud)

pySim/sysmocom_sja2.py

@@ -1,7 +1,7 @@
 # coding=utf-8
-"""Utilities / Functions related to sysmocom SJA2 cards
+"""Utilities / Functions related to sysmocom SJA2/SJA5 cards
 
-(C) 2021 by Harald Welte <laforge@osmocom.org>
+(C) 2021-2023 by Harald Welte <laforge@osmocom.org>
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -21,6 +21,7 @@ from pytlv.TLV import *
 from struct import pack, unpack
 from pySim.utils import *
 from pySim.filesystem import *
+from pySim.runtime import RuntimeState
 from pySim.ts_102_221 import CardProfileUICC
 from pySim.construct import *
 from construct import *
@@ -45,69 +46,73 @@ mac_length = {
 
 
 class EF_PIN(TransparentEF):
-    def __init__(self, fid, name):
+    _test_de_encode = [
+        ( 'f1030331323334ffffffff0a0a3132333435363738',
+          { 'state': { 'valid': True, 'change_able': True, 'unblock_able': True, 'disable_able': True,
+                       'not_initialized': False, 'disabled': True },
+           'attempts_remaining': 3, 'maximum_attempts': 3, 'pin': '31323334',
+           'puk': { 'attempts_remaining': 10, 'maximum_attempts': 10, 'puk': '3132333435363738' }
+          } ),
+        ( 'f003039999999999999999',
+          { 'state': { 'valid': True, 'change_able': True, 'unblock_able': True, 'disable_able': True,
+                       'not_initialized': False, 'disabled': False },
+           'attempts_remaining': 3, 'maximum_attempts': 3, 'pin': '9999999999999999',
+           'puk': None } ),
+    ]
+    def __init__(self, fid='6f01', name='EF.CHV1'):
         super().__init__(fid, name=name, desc='%s PIN file' % name)
-
-    def _decode_bin(self, raw_bin_data):
-        u = unpack('!BBB8s', raw_bin_data[:11])
-        res = {'enabled': (True, False)[u[0] & 0x01],
-               'initialized': (True, False)[u[0] & 0x02],
-               'disable_able': (False, True)[u[0] & 0x10],
-               'unblock_able': (False, True)[u[0] & 0x20],
-               'change_able': (False, True)[u[0] & 0x40],
-               'valid': (False, True)[u[0] & 0x80],
-               'attempts_remaining': u[1],
-               'maximum_attempts': u[2],
-               'pin': u[3].hex(),
-               }
-        if len(raw_bin_data) == 21:
-            u2 = unpack('!BB8s', raw_bin_data[11:10])
-            res['attempts_remaining_puk'] = u2[0]
-            res['maximum_attempts_puk'] = u2[1]
-            res['puk'] = u2[2].hex()
-        return res
+        StateByte = FlagsEnum(Byte, disabled=1, not_initialized=2, disable_able=0x10, unblock_able=0x20,
+                                    change_able=0x40, valid=0x80)
+        PukStruct = Struct('attempts_remaining'/Int8ub,
+                           'maximum_attempts'/Int8ub,
+                           'puk'/HexAdapter(Rpad(Bytes(8))))
+        self._construct = Struct('state'/StateByte,
+                                 'attempts_remaining'/Int8ub,
+                                 'maximum_attempts'/Int8ub,
+                                 'pin'/HexAdapter(Rpad(Bytes(8))),
+                                 'puk'/Optional(PukStruct))
 
 
 class EF_MILENAGE_CFG(TransparentEF):
+    _test_de_encode = [
+        ( '40002040600000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000020000000000000000000000000000000400000000000000000000000000000008',
+          {"r1": 64, "r2": 0, "r3": 32, "r4": 64, "r5": 96, "c1": "00000000000000000000000000000000", "c2":
+           "00000000000000000000000000000001", "c3": "00000000000000000000000000000002", "c4":
+           "00000000000000000000000000000004", "c5": "00000000000000000000000000000008"} ),
+      ]
     def __init__(self, fid='6f21', name='EF.MILENAGE_CFG', desc='Milenage connfiguration'):
         super().__init__(fid, name=name, desc=desc)
-
-    def _decode_bin(self, raw_bin_data):
-        u = unpack('!BBBBB16s16s16s16s16s', raw_bin_data)
-        return {'r1': u[0], 'r2': u[1], 'r3': u[2], 'r4': u[3], 'r5': u[4],
-                'c1': u[5].hex(),
-                'c2': u[6].hex(),
-                'c3': u[7].hex(),
-                'c4': u[8].hex(),
-                'c5': u[9].hex(),
-                }
+        self._construct = Struct('r1'/Int8ub, 'r2'/Int8ub, 'r3'/Int8ub, 'r4'/Int8ub, 'r5'/Int8ub,
+                                 'c1'/HexAdapter(Bytes(16)),
+                                 'c2'/HexAdapter(Bytes(16)),
+                                 'c3'/HexAdapter(Bytes(16)),
+                                 'c4'/HexAdapter(Bytes(16)),
+                                 'c5'/HexAdapter(Bytes(16)))
 
 
 class EF_0348_KEY(LinFixedEF):
     def __init__(self, fid='6f22', name='EF.0348_KEY', desc='TS 03.48 OTA Keys'):
         super().__init__(fid, name=name, desc=desc, rec_len=(27, 35))
-
-    def _decode_record_bin(self, raw_bin_data):
-        u = unpack('!BBB', raw_bin_data[0:3])
-        key_algo = (u[2] >> 6) & 1
-        key_length = ((u[2] >> 3) & 3) * 8
-        return {'sec_domain': u[0],
-                'key_set_version': u[1],
-                'key_type': key_type2str[u[2] & 3],
-                'key_length': key_length,
-                'algorithm': key_algo2str[key_algo],
-                'mac_length': mac_length[(u[2] >> 7)],
-                'key': raw_bin_data[3:key_length].hex()
-                }
+        KeyLenAndType = BitStruct('mac_length'/Mapping(Bit, {8:0, 4:1}),
+                                  'algorithm'/Enum(Bit, des=0, aes=1),
+                                  'key_length'/MultiplyAdapter(BitsInteger(3), 8),
+                                  '_rfu'/BitsRFU(1),
+                                  'key_type'/Enum(BitsInteger(2), kic=0, kid=1, kik=2, any=3))
+        self._construct = Struct('security_domain'/Int8ub,
+                                 'key_set_version'/Int8ub,
+                                 'key_len_and_type'/KeyLenAndType,
+                                 'key'/HexAdapter(Bytes(this.key_len_and_type.key_length)))
 
 
 class EF_0348_COUNT(LinFixedEF):
+    _test_de_encode = [
+        ( 'fe010000000000', {"sec_domain": 254, "key_set_version": 1, "counter": "0000000000"} ),
+      ]
     def __init__(self, fid='6f23', name='EF.0348_COUNT', desc='TS 03.48 OTA Counters'):
         super().__init__(fid, name=name, desc=desc, rec_len=(7, 7))
-
-    def _decode_record_bin(self, raw_bin_data):
-        u = unpack('!BB5s', raw_bin_data)
-        return {'sec_domain': u[0], 'key_set_version': u[1], 'counter': u[2]}
+        self._construct = Struct('sec_domain'/Int8ub,
+                                 'key_set_version'/Int8ub,
+                                 'counter'/HexAdapter(Bytes(5)))
 
 
 class EF_SIM_AUTH_COUNTER(TransparentEF):
@@ -117,38 +122,40 @@ class EF_SIM_AUTH_COUNTER(TransparentEF):
 
 
 class EF_GP_COUNT(LinFixedEF):
+    _test_de_encode = [
+        ( '0070000000', {"sec_domain": 0, "key_set_version": 112, "counter": 0, "rfu": 0} ),
+      ]
     def __init__(self, fid='6f26', name='EF.GP_COUNT', desc='GP SCP02 Counters'):
         super().__init__(fid, name=name, desc=desc, rec_len=(5, 5))
-
-    def _decode_record_bin(self, raw_bin_data):
-        u = unpack('!BBHB', raw_bin_data)
-        return {'sec_domain': u[0], 'key_set_version': u[1], 'counter': u[2], 'rfu': u[3]}
-
+        self._construct = Struct('sec_domain'/Int8ub,
+                                 'key_set_version'/Int8ub,
+                                 'counter'/Int16ub,
+                                 'rfu'/Int8ub)
 
 class EF_GP_DIV_DATA(LinFixedEF):
     def __init__(self, fid='6f27', name='EF.GP_DIV_DATA', desc='GP SCP02 key diversification data'):
         super().__init__(fid, name=name, desc=desc, rec_len=(12, 12))
 
-    def _decode_record_bin(self, raw_bin_data):
+    def _decode_record_bin(self, raw_bin_data, **kwargs):
         u = unpack('!BB8s', raw_bin_data)
         return {'sec_domain': u[0], 'key_set_version': u[1], 'key_div_data': u[2].hex()}
 
 
 class EF_SIM_AUTH_KEY(TransparentEF):
+    _test_de_encode = [
+        ( '14000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f',
+          {"cfg": {"sres_deriv_func": 1, "use_opc_instead_of_op": True, "algorithm": "milenage"}, "key":
+           "000102030405060708090a0b0c0d0e0f", "op_opc": "101112131415161718191a1b1c1d1e1f"} ),
+      ]
     def __init__(self, fid='6f20', name='EF.SIM_AUTH_KEY'):
         super().__init__(fid, name=name, desc='USIM authentication key')
         CfgByte = BitStruct(Padding(2),
-                            'use_sres_deriv_func_2'/Bit,
-                            'use_opc_instead_of_op'/Bit,
+                            'sres_deriv_func'/Mapping(Bit, {1:0, 2:1}),
+                            'use_opc_instead_of_op'/Flag,
                             'algorithm'/Enum(Nibble, milenage=4, comp128v1=1, comp128v2=2, comp128v3=3))
         self._construct = Struct('cfg'/CfgByte,
                                  'key'/HexAdapter(Bytes(16)),
-                                 'op'/ If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16))),
-                                 'opc' /
-                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16)))
-                                 )
+                                 'op_opc' /HexAdapter(Bytes(16)))
 
 
 class DF_SYSTEM(CardDF):
@@ -176,13 +183,21 @@ class DF_SYSTEM(CardDF):
 
 
 class EF_USIM_SQN(TransparentEF):
+    _test_de_encode = [
+        ( 'd503000200000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+          {"flag1": {"skip_next_sqn_check": True, "delta_max_check": True, "age_limit_check": False, "sqn_check": True,
+                     "ind_len": 5}, "flag2": {"rfu": 0, "dont_clear_amf_for_macs": False, "aus_concealed": True,
+                                              "autn_concealed": True}, "delta_max": 8589934592, "age_limit":
+           8589934592, "freshness": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+                                     0, 0, 0, 0, 0, 0, 0, 0]} ),
+      ]
     def __init__(self, fid='af30', name='EF.USIM_SQN'):
         super().__init__(fid, name=name, desc='SQN parameters for AKA')
-        Flag1 = BitStruct('skip_next_sqn_check'/Bit, 'delta_max_check'/Bit,
-                          'age_limit_check'/Bit, 'sqn_check'/Bit,
+        Flag1 = BitStruct('skip_next_sqn_check'/Flag, 'delta_max_check'/Flag,
+                          'age_limit_check'/Flag, 'sqn_check'/Flag,
                           'ind_len'/BitsInteger(4))
-        Flag2 = BitStruct('rfu'/BitsRFU(5), 'dont_clear_amf_for_macs'/Bit,
-                          'aus_concealed'/Bit, 'autn_concealed'/Bit)
+        Flag2 = BitStruct('rfu'/BitsRFU(5), 'dont_clear_amf_for_macs'/Flag,
+                          'aus_concealed'/Flag, 'autn_concealed'/Flag)
         self._construct = Struct('flag1'/Flag1, 'flag2'/Flag2,
                                  'delta_max' /
                                  BytesInteger(6), 'age_limit'/BytesInteger(6),
@@ -192,37 +207,61 @@ class EF_USIM_SQN(TransparentEF):
 class EF_USIM_AUTH_KEY(TransparentEF):
     def __init__(self, fid='af20', name='EF.USIM_AUTH_KEY'):
         super().__init__(fid, name=name, desc='USIM authentication key')
-        CfgByte = BitStruct(Padding(1), 'only_4bytes_res_in_3g'/Bit,
-                            'use_sres_deriv_func_2_in_3g'/Bit,
-                            'use_opc_instead_of_op'/Bit,
-                            'algorithm'/Enum(Nibble, milenage=4, sha1_aka=5, xor=15))
+        Algorithm = Enum(Nibble, milenage=4, sha1_aka=5, tuak=6, xor=15)
+        CfgByte = BitStruct(Padding(1), 'only_4bytes_res_in_3g'/Flag,
+                            'sres_deriv_func_in_2g'/Mapping(Bit, {1:0, 2:1}),
+                            'use_opc_instead_of_op'/Mapping(Bit, {False:0, True:1}),
+                            'algorithm'/Algorithm)
         self._construct = Struct('cfg'/CfgByte,
                                  'key'/HexAdapter(Bytes(16)),
-                                 'op' /
-                                 If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16))),
-                                 'opc' /
-                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16)))
-                                 )
+                                 'op_opc' /HexAdapter(Bytes(16)))
+        # TUAK has a rather different layout for the data, so we define a different
+        # construct below and use explicit _{decode,encode}_bin() methods for separating
+        # the TUAK and non-TUAK situation
+        CfgByteTuak = BitStruct(Padding(1),
+                                'key_length'/Mapping(Bit, {128:0, 256:1}),
+                                'sres_deriv_func_in_2g'/Mapping(Bit, {1:0, 2:1}),
+                                'use_opc_instead_of_op'/Mapping(Bit, {False:0, True:1}),
+                                'algorithm'/Algorithm)
+        TuakCfgByte = BitStruct(Padding(1),
+                                'ck_and_ik_size'/Mapping(Bit, {128:0, 256:1}),
+                                'mac_size'/Mapping(BitsInteger(3), {64:0, 128:1, 256:2}),
+                                'res_size'/Mapping(BitsInteger(3), {32:0, 64:1, 128:2, 256:3}))
+        self._constr_tuak = Struct('cfg'/CfgByteTuak,
+                                   'tuak_cfg'/TuakCfgByte,
+                                   'num_of_keccak_iterations'/Int8ub,
+                                   'op_opc'/HexAdapter(Bytes(32)),
+                                   'k'/HexAdapter(Bytes(this.cfg.key_length//8)))
+
+    def _decode_bin(self, raw_bin_data: bytearray) -> dict:
+        if raw_bin_data[0] & 0x0F == 0x06:
+            return parse_construct(self._constr_tuak, raw_bin_data)
+        else:
+            return parse_construct(self._construct, raw_bin_data)
+
+    def _encode_bin(self, abstract_data: dict) -> bytearray:
+        if abstract_data['cfg']['algorithm'] == 'tuak':
+            return build_construct(self._constr_tuak, abstract_data)
+        else:
+            return build_construct(self._construct, abstract_data)
 
 
 class EF_USIM_AUTH_KEY_2G(TransparentEF):
+    _test_de_encode = [
+        ( '14000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f',
+          {"cfg": {"only_4bytes_res_in_3g": False, "sres_deriv_func_in_2g": 1, "use_opc_instead_of_op": True,
+                   "algorithm": "milenage"}, "key": "000102030405060708090a0b0c0d0e0f", "op_opc":
+           "101112131415161718191a1b1c1d1e1f"} ),
+      ]
     def __init__(self, fid='af22', name='EF.USIM_AUTH_KEY_2G'):
         super().__init__(fid, name=name, desc='USIM authentication key in 2G context')
-        CfgByte = BitStruct(Padding(1), 'only_4bytes_res_in_3g'/Bit,
-                            'use_sres_deriv_func_2_in_3g'/Bit,
-                            'use_opc_instead_of_op'/Bit,
-                            'algorithm'/Enum(Nibble, milenage=4, comp128v1=1, comp128v2=2, comp128v3=3))
+        CfgByte = BitStruct(Padding(1), 'only_4bytes_res_in_3g'/Flag,
+                            'sres_deriv_func_in_2g'/Mapping(Bit, {1:0, 2:1}),
+                            'use_opc_instead_of_op'/Flag,
+                            'algorithm'/Enum(Nibble, milenage=4, comp128v1=1, comp128v2=2, comp128v3=3, xor=14))
         self._construct = Struct('cfg'/CfgByte,
                                  'key'/HexAdapter(Bytes(16)),
-                                 'op' /
-                                 If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16))),
-                                 'opc' /
-                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op,
-                                     HexAdapter(Bytes(16)))
-                                 )
+                                 'op_opc' /HexAdapter(Bytes(16)))
 
 
 class EF_GBA_SK(TransparentEF):
@@ -275,3 +314,34 @@ class SysmocomSJA2(CardModel):
                 EF_USIM_SQN(name='EF.ISIM_SQN'),
             ]
             isim_adf.add_files(files_adf_isim)
+
+class SysmocomSJA5(CardModel):
+    _atrs = ["3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 35 75 30 35 02 51 CC",
+             "3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 35 75 30 35 02 65 F8",
+             "3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 35 75 30 35 02 59 C4"]
+
+    @classmethod
+    def add_files(cls, rs: RuntimeState):
+        """Add sysmocom SJA2 specific files to given RuntimeState."""
+        rs.mf.add_file(DF_SYSTEM())
+        # optional USIM application
+        if 'a0000000871002' in rs.mf.applications:
+            usim_adf = rs.mf.applications['a0000000871002']
+            files_adf_usim = [
+                EF_USIM_AUTH_KEY(),
+                EF_USIM_AUTH_KEY_2G(),
+                EF_GBA_SK(),
+                EF_GBA_REC_LIST(),
+                EF_GBA_INT_KEY(),
+                EF_USIM_SQN(),
+            ]
+            usim_adf.add_files(files_adf_usim)
+        # optional ISIM application
+        if 'a0000000871004' in rs.mf.applications:
+            isim_adf = rs.mf.applications['a0000000871004']
+            files_adf_isim = [
+                EF_USIM_AUTH_KEY(name='EF.ISIM_AUTH_KEY'),
+                EF_USIM_AUTH_KEY_2G(name='EF.ISIM_AUTH_KEY_2G'),
+                EF_USIM_SQN(name='EF.ISIM_SQN'),
+            ]
+            isim_adf.add_files(files_adf_isim)

pySim/tlv.py

@@ -23,10 +23,9 @@ from construct import *
 
 from pySim.utils import bertlv_encode_len, bertlv_parse_len, bertlv_encode_tag, bertlv_parse_tag
 from pySim.utils import comprehensiontlv_encode_tag, comprehensiontlv_parse_tag
-from pySim.utils import bertlv_parse_one, comprehensiontlv_parse_one
 from pySim.utils import bertlv_parse_tag_raw, comprehensiontlv_parse_tag_raw
 
-from pySim.construct import parse_construct, LV, HexAdapter, BcdAdapter, BitsRFU, GsmStringAdapter
+from pySim.construct import build_construct, parse_construct, LV, HexAdapter, BcdAdapter, BitsRFU, GsmStringAdapter
 from pySim.exceptions import *
 
 import inspect
@@ -84,15 +83,15 @@ class Transcodable(abc.ABC):
         self.decoded = None
         self._construct = None
 
-    def to_bytes(self) -> bytes:
+    def to_bytes(self, context: dict = {}) -> bytes:
         """Convert from internal representation to binary bytes.  Store the binary result
         in the internal state and return it."""
-        if not self.decoded:
+        if self.decoded == None:
             do = b''
         elif self._construct:
-            do = self._construct.build(self.decoded, total_len=None)
+            do = build_construct(self._construct, self.decoded, context)
         elif self.__class__._construct:
-            do = self.__class__._construct.build(self.decoded, total_len=None)
+            do = build_construct(self.__class__._construct, self.decoded, context)
         else:
             do = self._to_bytes()
         self.encoded = do
@@ -102,16 +101,16 @@ class Transcodable(abc.ABC):
     def _to_bytes(self):
         raise NotImplementedError('%s._to_bytes' % type(self).__name__)
 
-    def from_bytes(self, do: bytes):
+    def from_bytes(self, do: bytes, context: dict = {}):
         """Convert from binary bytes to internal representation. Store the decoded result
         in the internal state and return it."""
         self.encoded = do
         if self.encoded == b'':
             self.decoded = None
         elif self._construct:
-            self.decoded = parse_construct(self._construct, do)
+            self.decoded = parse_construct(self._construct, do, context=context)
         elif self.__class__._construct:
-            self.decoded = parse_construct(self.__class__._construct, do)
+            self.decoded = parse_construct(self.__class__._construct, do, context=context)
         else:
             self.decoded = self._from_bytes(do)
         return self.decoded
@@ -157,11 +156,14 @@ class IE(Transcodable, metaclass=TlvMeta):
     def from_dict(self, decoded: dict):
         """Set the IE internal decoded representation to data from the argument.
         If this is a nested IE, the child IE instance list is re-created."""
+        expected_key_name = camel_to_snake(type(self).__name__)
+        if not expected_key_name in decoded:
+            raise ValueError("Dict %s doesn't contain expected key %s" % (decoded, expected_key_name))
         if self.nested_collection:
-            self.children = self.nested_collection.from_dict(decoded)
+            self.children = self.nested_collection.from_dict(decoded[expected_key_name])
         else:
             self.children = []
-            self.decoded = decoded
+            self.decoded = decoded[expected_key_name]
 
     def is_constructed(self):
         """Is this IE constructed by further nested IEs?"""
@@ -171,27 +173,27 @@ class IE(Transcodable, metaclass=TlvMeta):
             return False
 
     @abc.abstractmethod
-    def to_ie(self) -> bytes:
+    def to_ie(self, context: dict = {}) -> bytes:
         """Convert the internal representation to entire IE including IE header."""
 
-    def to_bytes(self) -> bytes:
-        """Convert the internal representation _of the value part_ to binary bytes."""
+    def to_bytes(self, context: dict = {}) -> bytes:
+        """Convert the internal representation *of the value part* to binary bytes."""
         if self.is_constructed():
             # concatenate the encoded IE of all children to form the value part
             out = b''
             for c in self.children:
-                out += c.to_ie()
+                out += c.to_ie(context=context)
             return out
         else:
-            return super().to_bytes()
+            return super().to_bytes(context=context)
 
-    def from_bytes(self, do: bytes):
-        """Parse _the value part_ from binary bytes to internal representation."""
+    def from_bytes(self, do: bytes, context: dict = {}):
+        """Parse *the value part* from binary bytes to internal representation."""
         if self.nested_collection:
-            self.children = self.nested_collection.from_bytes(do)
+            self.children = self.nested_collection.from_bytes(do, context=context)
         else:
             self.children = []
-            return super().from_bytes(do)
+            return super().from_bytes(do, context=context)
 
 
 class TLV_IE(IE):
@@ -223,15 +225,15 @@ class TLV_IE(IE):
         """Encode the length part assuming a certain binary value. Must be provided by
         derived (TLV format specific) class."""
 
-    def to_ie(self):
-        return self.to_tlv()
+    def to_ie(self, context: dict = {}):
+        return self.to_tlv(context=context)
 
-    def to_tlv(self):
+    def to_tlv(self, context: dict = {}):
         """Convert the internal representation to binary TLV bytes."""
-        val = self.to_bytes()
+        val = self.to_bytes(context=context)
         return self._encode_tag() + self._encode_len(val) + val
 
-    def from_tlv(self, do: bytes):
+    def from_tlv(self, do: bytes, context: dict = {}):
         if len(do) == 0:
             return {}, b''
         (rawtag, remainder) = self.__class__._parse_tag_raw(do)
@@ -245,7 +247,7 @@ class TLV_IE(IE):
         else:
             value = do
             remainder = b''
-        dec = self.from_bytes(value)
+        dec = self.from_bytes(value, context=context)
         return dec, remainder
 
 
@@ -315,7 +317,7 @@ class TLV_IE_Collection(metaclass=TlvCollectionMeta):
         self.members_by_tag = {}
         self.members_by_name = {}
         self.members_by_tag = {m.tag: m for m in self.members}
-        self.members_by_name = {m.__name__: m for m in self.members}
+        self.members_by_name = {camel_to_snake(m.__name__): m for m in self.members}
         # if we are a constructed IE, [ordered] list of actual child-IE instances
         self.children = kwargs.get('children', [])
         self.encoded = None
@@ -340,7 +342,7 @@ class TLV_IE_Collection(metaclass=TlvCollectionMeta):
         else:
             raise TypeError
 
-    def from_bytes(self, binary: bytes) -> List[TLV_IE]:
+    def from_bytes(self, binary: bytes, context: dict = {}) -> List[TLV_IE]:
         """Create a list of TLV_IEs from the collection based on binary input data.
         Args:
             binary : binary bytes of encoded data
@@ -354,15 +356,16 @@ class TLV_IE_Collection(metaclass=TlvCollectionMeta):
         first = next(iter(self.members_by_tag.values()))
         # iterate until no binary trailer is left
         while len(remainder):
+            context['siblings'] = res
             # obtain the tag at the start of the remainder
             tag, r = first._parse_tag_raw(remainder)
             if tag == None:
-                return res
+                break
             if tag in self.members_by_tag:
                 cls = self.members_by_tag[tag]
                 # create an instance and parse accordingly
                 inst = cls()
-                dec, remainder = inst.from_tlv(remainder)
+                dec, remainder = inst.from_tlv(remainder, context=context)
                 res.append(inst)
             else:
                 # unknown tag; create the related class on-the-fly using the same base class
@@ -373,7 +376,7 @@ class TLV_IE_Collection(metaclass=TlvCollectionMeta):
                 cls._to_bytes = lambda s: bytes.fromhex(s.decoded['raw'])
                 # create an instance and parse accordingly
                 inst = cls()
-                dec, remainder = inst.from_tlv(remainder)
+                dec, remainder = inst.from_tlv(remainder, context=context)
                 res.append(inst)
         self.children = res
         return res
@@ -383,33 +386,45 @@ class TLV_IE_Collection(metaclass=TlvCollectionMeta):
         of dicts, where they key indicates the name of the TLV_IE subclass to use."""
         # list of instances of TLV_IE collection member classes appearing in the data
         res = []
+        # iterate over members of the list passed into "decoded"
         for i in decoded:
+            # iterate over all the keys (typically one!) within the current list item dict
             for k in i.keys():
+                # check if we have a member identified by the dict key
                 if k in self.members_by_name:
+                    # resolve the class for that name; create an instance of it
                     cls = self.members_by_name[k]
                     inst = cls()
-                    inst.from_dict(i[k])
+                    if cls.nested_collection_cls:
+                        # in case of collections, we want to pass the raw "value" portion to from_dict,
+                        # as to_dict() below intentionally omits the collection-class-name as key
+                        inst.from_dict(i[k])
+                    else:
+                        inst.from_dict({k: i[k]})
                     res.append(inst)
                 else:
                     raise ValueError('%s: Unknown TLV Class %s in %s; expected %s' %
-                                     (self, i[0], decoded, self.members_by_name.keys()))
+                                     (self, k, decoded, self.members_by_name.keys()))
         self.children = res
         return res
 
     def to_dict(self):
+        # we intentionally return not a dict, but a list of dicts.  We could prefix by
+        # self.__class__.__name__, but that is usually some meaningless auto-generated  collection name.
         return [x.to_dict() for x in self.children]
 
-    def to_bytes(self):
+    def to_bytes(self, context: dict = {}):
         out = b''
+        context['siblings'] = self.children
         for c in self.children:
-            out += c.to_tlv()
+            out += c.to_tlv(context=context)
         return out
 
-    def from_tlv(self, do):
-        return self.from_bytes(do)
+    def from_tlv(self, do, context: dict = {}):
+        return self.from_bytes(do, context=context)
 
-    def to_tlv(self):
-        return self.to_bytes()
+    def to_tlv(self, context: dict = {}):
+        return self.to_bytes(context=context)
 
 
 def flatten_dict_lists(inp):
@@ -421,8 +436,12 @@ def flatten_dict_lists(inp):
                 return False
         return True
 
+    def are_elements_unique(lod):
+        set_of_keys = set([list(x.keys())[0] for x in lod])
+        return len(lod) == len(set_of_keys)
+
     if isinstance(inp, list):
-        if are_all_elements_dict(inp):
+        if are_all_elements_dict(inp) and are_elements_unique(inp):
             # flatten into one shared dict
             newdict = {}
             for e in inp:

pySim/transport/__init__.py

@@ -3,18 +3,20 @@
 """ pySim: PCSC reader transport link base
 """
 
+import os
 import abc
 import argparse
 from typing import Optional, Tuple
+from construct import Construct
 
 from pySim.exceptions import *
 from pySim.construct import filter_dict
-from pySim.utils import sw_match, b2h, h2b, i2h, Hexstr
+from pySim.utils import sw_match, b2h, h2b, i2h, Hexstr, SwHexstr, SwMatchstr, ResTuple
 from pySim.cat import ProactiveCommand, CommandDetails, DeviceIdentities, Result
 
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
-# Copyright (C) 2021-2022 Harald Welte <laforge@osmocom.org>
+# Copyright (C) 2021-2023 Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -60,14 +62,18 @@ class ProactiveHandler(abc.ABC):
 class LinkBase(abc.ABC):
     """Base class for link/transport to card."""
 
-    def __init__(self, sw_interpreter=None, apdu_tracer=None,
+    def __init__(self, sw_interpreter=None, apdu_tracer: Optional[ApduTracer]=None,
                  proactive_handler: Optional[ProactiveHandler]=None):
         self.sw_interpreter = sw_interpreter
         self.apdu_tracer = apdu_tracer
         self.proactive_handler = proactive_handler
 
     @abc.abstractmethod
-    def _send_apdu_raw(self, pdu: str) -> Tuple[str, str]:
+    def __str__(self) -> str:
+        """Implementation specific method for printing an information to identify the device."""
+
+    @abc.abstractmethod
+    def _send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
         """Implementation specific method for sending the PDU."""
 
     def set_sw_interpreter(self, interp):
@@ -75,7 +81,7 @@ class LinkBase(abc.ABC):
         self.sw_interpreter = interp
 
     @abc.abstractmethod
-    def wait_for_card(self, timeout: int = None, newcardonly: bool = False):
+    def wait_for_card(self, timeout: Optional[int] = None, newcardonly: bool = False):
         """Wait for a card and connect to it
 
         Args:
@@ -98,7 +104,7 @@ class LinkBase(abc.ABC):
         """Resets the card (power down/up)
         """
 
-    def send_apdu_raw(self, pdu: str):
+    def send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
         """Sends an APDU with minimal processing
 
         Args:
@@ -115,7 +121,7 @@ class LinkBase(abc.ABC):
             self.apdu_tracer.trace_response(pdu, sw, data)
         return (data, sw)
 
-    def send_apdu(self, pdu):
+    def send_apdu(self, pdu: Hexstr) -> ResTuple:
         """Sends an APDU and auto fetch response data
 
         Args:
@@ -132,11 +138,12 @@ class LinkBase(abc.ABC):
         # xx is the number of response bytes available.
         # See also:
         if (sw is not None):
-            if ((sw[0:2] == '9f') or (sw[0:2] == '61')):
+            while ((sw[0:2] == '9f') or (sw[0:2] == '61')):
                 # SW1=9F: 3GPP TS 51.011 9.4.1, Responses to commands which are correctly executed
                 # SW1=61: ISO/IEC 7816-4, Table 5 — General meaning of the interindustry values of SW1-SW2
                 pdu_gr = pdu[0:2] + 'c00000' + sw[2:4]
-                data, sw = self.send_apdu_raw(pdu_gr)
+                d, sw = self.send_apdu_raw(pdu_gr)
+                data += d
             if sw[0:2] == '6c':
                 # SW1=6C: ETSI TS 102 221 Table 7.1: Procedure byte coding
                 pdu_gr = pdu[0:8] + sw[2:4]
@@ -144,7 +151,7 @@ class LinkBase(abc.ABC):
 
         return data, sw
 
-    def send_apdu_checksw(self, pdu, sw="9000"):
+    def send_apdu_checksw(self, pdu: Hexstr, sw: SwMatchstr = "9000") -> ResTuple:
         """Sends an APDU and check returned SW
 
         Args:
@@ -212,7 +219,8 @@ class LinkBase(abc.ABC):
             raise SwMatchError(rv[1], sw.lower(), self.sw_interpreter)
         return rv
 
-    def send_apdu_constr(self, cla, ins, p1, p2, cmd_constr, cmd_data, resp_constr):
+    def send_apdu_constr(self, cla: Hexstr, ins: Hexstr, p1: Hexstr, p2: Hexstr, cmd_constr: Construct,
+                         cmd_data: Hexstr, resp_constr: Construct) -> Tuple[dict, SwHexstr]:
         """Build and sends an APDU using a 'construct' definition; parses response.
 
         Args:
@@ -237,8 +245,9 @@ class LinkBase(abc.ABC):
             rsp = None
         return (rsp, sw)
 
-    def send_apdu_constr_checksw(self, cla, ins, p1, p2, cmd_constr, cmd_data, resp_constr,
-                                 sw_exp="9000"):
+    def send_apdu_constr_checksw(self, cla: Hexstr, ins: Hexstr, p1: Hexstr, p2: Hexstr,
+                                 cmd_constr: Construct, cmd_data: Hexstr, resp_constr: Construct,
+                                 sw_exp: SwMatchstr="9000") -> Tuple[dict, SwHexstr]:
         """Build and sends an APDU using a 'construct' definition; parses response.
 
         Args:
@@ -260,60 +269,42 @@ class LinkBase(abc.ABC):
         return (rsp, sw)
 
 
-def argparse_add_reader_args(arg_parser):
+def argparse_add_reader_args(arg_parser: argparse.ArgumentParser):
     """Add all reader related arguments to the given argparse.Argumentparser instance."""
-    serial_group = arg_parser.add_argument_group('Serial Reader')
-    serial_group.add_argument('-d', '--device', metavar='DEV', default='/dev/ttyUSB0',
-                              help='Serial Device for SIM access')
-    serial_group.add_argument('-b', '--baud', dest='baudrate', type=int, metavar='BAUD', default=9600,
-                              help='Baud rate used for SIM access')
+    from pySim.transport.serial import SerialSimLink
+    from pySim.transport.pcsc import PcscSimLink
+    from pySim.transport.modem_atcmd import ModemATCommandLink
+    from pySim.transport.calypso import CalypsoSimLink
 
-    pcsc_group = arg_parser.add_argument_group('PC/SC Reader')
-    pcsc_group.add_argument('-p', '--pcsc-device', type=int, dest='pcsc_dev', metavar='PCSC', default=None,
-                            help='PC/SC reader number to use for SIM access')
-
-    modem_group = arg_parser.add_argument_group('AT Command Modem Reader')
-    modem_group.add_argument('--modem-device', dest='modem_dev', metavar='DEV', default=None,
-                             help='Serial port of modem for Generic SIM Access (3GPP TS 27.007)')
-    modem_group.add_argument('--modem-baud', type=int, metavar='BAUD', default=115200,
-                             help='Baud rate used for modem port')
-
-    osmobb_group = arg_parser.add_argument_group('OsmocomBB Reader')
-    osmobb_group.add_argument('--osmocon', dest='osmocon_sock', metavar='PATH', default=None,
-                              help='Socket path for Calypso (e.g. Motorola C1XX) based reader (via OsmocomBB)')
+    SerialSimLink.argparse_add_reader_args(arg_parser)
+    PcscSimLink.argparse_add_reader_args(arg_parser)
+    ModemATCommandLink.argparse_add_reader_args(arg_parser)
+    CalypsoSimLink.argparse_add_reader_args(arg_parser)
 
     return arg_parser
 
 
-def init_reader(opts, **kwargs) -> Optional[LinkBase]:
+def init_reader(opts, **kwargs) -> LinkBase:
     """
     Init card reader driver
     """
-    sl = None  # type : :Optional[LinkBase]
-    try:
-        if opts.pcsc_dev is not None:
-            print("Using PC/SC reader interface")
-            from pySim.transport.pcsc import PcscSimLink
-            sl = PcscSimLink(opts.pcsc_dev, **kwargs)
-        elif opts.osmocon_sock is not None:
-            print("Using Calypso-based (OsmocomBB) reader interface")
-            from pySim.transport.calypso import CalypsoSimLink
-            sl = CalypsoSimLink(sock_path=opts.osmocon_sock, **kwargs)
-        elif opts.modem_dev is not None:
-            print("Using modem for Generic SIM Access (3GPP TS 27.007)")
-            from pySim.transport.modem_atcmd import ModemATCommandLink
-            sl = ModemATCommandLink(
-                device=opts.modem_dev, baudrate=opts.modem_baud, **kwargs)
-        else:  # Serial reader is default
-            print("Using serial reader interface")
-            from pySim.transport.serial import SerialSimLink
-            sl = SerialSimLink(device=opts.device,
-                               baudrate=opts.baudrate, **kwargs)
-        return sl
-    except Exception as e:
-        if str(e):
-            print("Card reader initialization failed with exception:\n" + str(e))
-        else:
-            print(
-                "Card reader initialization failed with an exception of type:\n" + str(type(e)))
-        return None
+    if opts.pcsc_dev is not None or opts.pcsc_regex is not None:
+        from pySim.transport.pcsc import PcscSimLink
+        sl = PcscSimLink(opts, **kwargs)
+    elif opts.osmocon_sock is not None:
+        from pySim.transport.calypso import CalypsoSimLink
+        sl = CalypsoSimLink(opts, **kwargs)
+    elif opts.modem_dev is not None:
+        from pySim.transport.modem_atcmd import ModemATCommandLink
+        sl = ModemATCommandLink(opts, **kwargs)
+    else:  # Serial reader is default
+        print("No reader/driver specified; falling back to default (Serial reader)")
+        from pySim.transport.serial import SerialSimLink
+        sl = SerialSimLink(opts, **kwargs)
+
+    if os.environ.get('PYSIM_INTEGRATION_TEST') == "1":
+        print("Using %s reader interface" % (sl.name))
+    else:
+        print("Using reader %s" % sl)
+
+    return sl

pySim/transport/calypso.py

@@ -20,10 +20,12 @@ import select
 import struct
 import socket
 import os
+import argparse
+from typing import Optional
 
 from pySim.transport import LinkBase
 from pySim.exceptions import *
-from pySim.utils import h2b, b2h
+from pySim.utils import h2b, b2h, Hexstr, ResTuple
 
 
 class L1CTLMessage:
@@ -74,8 +76,10 @@ class L1CTLMessageSIM(L1CTLMessage):
 
 class CalypsoSimLink(LinkBase):
     """Transport Link for Calypso based phones."""
+    name = 'Calypso-based (OsmocomBB) reader'
 
-    def __init__(self, sock_path: str = "/tmp/osmocom_l2", **kwargs):
+    def __init__(self, opts: argparse.Namespace = argparse.Namespace(osmocon_sock="/tmp/osmocom_l2"), **kwargs):
+        sock_path = opts.osmocon_sock
         super().__init__(**kwargs)
         # Make sure that a given socket path exists
         if not os.path.exists(sock_path):
@@ -88,10 +92,13 @@ class CalypsoSimLink(LinkBase):
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         self.sock.connect(sock_path)
 
+        # Remember socket path
+        self._sock_path = sock_path
+
     def __del__(self):
         self.sock.close()
 
-    def wait_for_rsp(self, exp_len=128):
+    def wait_for_rsp(self, exp_len: int = 128):
         # Wait for incoming data (timeout is 3 seconds)
         s, _, _ = select.select([self.sock], [], [], 3.0)
         if not s:
@@ -118,10 +125,10 @@ class CalypsoSimLink(LinkBase):
     def disconnect(self):
         pass  # Nothing to do really ...
 
-    def wait_for_card(self, timeout=None, newcardonly=False):
+    def wait_for_card(self, timeout: Optional[int] = None, newcardonly: bool = False):
         pass  # Nothing to do really ...
 
-    def _send_apdu_raw(self, pdu):
+    def _send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
 
         # Request FULL reset
         req_msg = L1CTLMessageSIM(h2b(pdu))
@@ -154,3 +161,15 @@ class CalypsoSimLink(LinkBase):
         sw = rsp[-2:]
 
         return b2h(data), b2h(sw)
+
+    def __str__(self) -> str:
+        return "osmocon:%s" % (self._sock_path)
+
+    @staticmethod
+    def argparse_add_reader_args(arg_parser: argparse.ArgumentParser):
+        osmobb_group = arg_parser.add_argument_group('OsmocomBB Reader', """Use an OsmocomBB compatible phone
+to access the SIM inserted to the phone SIM slot.  This will require you to run the OsmocomBB firmware inside
+the phone (can be ram-loaded).  It also requires that you run the ``osmocon`` program, which provides a unix
+domain socket to which this reader driver can attach.""")
+        osmobb_group.add_argument('--osmocon', dest='osmocon_sock', metavar='PATH', default=None,
+                                  help='Socket path for Calypso (e.g. Motorola C1XX) based reader (via OsmocomBB)')

pySim/transport/modem_atcmd.py

@@ -20,7 +20,10 @@ import logging as log
 import serial
 import time
 import re
+import argparse
+from typing import Optional
 
+from pySim.utils import Hexstr, ResTuple
 from pySim.transport import LinkBase
 from pySim.exceptions import *
 
@@ -30,8 +33,12 @@ from pySim.exceptions import *
 
 class ModemATCommandLink(LinkBase):
     """Transport Link for 3GPP TS 27.007 compliant modems."""
+    name = "modem for Generic SIM Access (3GPP TS 27.007)"
 
-    def __init__(self, device: str = '/dev/ttyUSB0', baudrate: int = 115200, **kwargs):
+    def __init__(self, opts: argparse.Namespace = argparse.Namespace(modem_dev='/dev/ttyUSB0',
+                                                                     modem_baud=115200), **kwargs):
+        device = opts.modem_dev
+        baudrate = opts.modem_baud
         super().__init__(**kwargs)
         self._sl = serial.Serial(device, baudrate, timeout=5)
         self._echo = False		# this will be auto-detected by _check_echo()
@@ -136,10 +143,10 @@ class ModemATCommandLink(LinkBase):
     def disconnect(self):
         pass  # Nothing to do really ...
 
-    def wait_for_card(self, timeout=None, newcardonly=False):
+    def wait_for_card(self, timeout: Optional[int] = None, newcardonly: bool = False):
         pass  # Nothing to do really ...
 
-    def _send_apdu_raw(self, pdu):
+    def _send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
         # Make sure pdu has upper case hex digits [A-F]
         pdu = pdu.upper()
 
@@ -148,8 +155,9 @@ class ModemATCommandLink(LinkBase):
         log.debug('Sending command: %s',  cmd)
 
         # Send AT+CSIM command to the modem
-        # TODO: also handle +CME ERROR: <err>
         rsp = self.send_at_cmd(cmd)
+        if rsp[-1].startswith(b'+CME ERROR:'):
+            raise ProtocolError('AT+CSIM failed with: %s' % str(rsp))
         if len(rsp) != 2 or rsp[-1] != b'OK':
             raise ReaderError('APDU transfer failed: %s' % str(rsp))
         rsp = rsp[0]  # Get rid of b'OK'
@@ -166,3 +174,16 @@ class ModemATCommandLink(LinkBase):
         sw = rsp_pdu[-4:].decode().lower()
         log.debug('Command response: %s, %s',  data, sw)
         return data, sw
+
+    def __str__(self) -> str:
+        return "modem:%s" % self._device
+
+    @staticmethod
+    def argparse_add_reader_args(arg_parser: argparse.ArgumentParser):
+        modem_group = arg_parser.add_argument_group('AT Command Modem Reader', """Talk to a SIM Card inside a
+mobile phone or cellular modem which is attached to this computer and offers an AT command interface including
+the AT+CSIM interface for Generic SIM access as specified in 3GPP TS 27.007.""")
+        modem_group.add_argument('--modem-device', dest='modem_dev', metavar='DEV', default=None,
+                                 help='Serial port of modem for Generic SIM Access (3GPP TS 27.007)')
+        modem_group.add_argument('--modem-baud', type=int, metavar='BAUD', default=115200,
+                                 help='Baud rate used for modem port')

pySim/transport/pcsc.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
-# Copyright (C) 2010  Harald Welte <laforge@gnumonks.org>
+# Copyright (C) 2010-2023  Harald Welte <laforge@gnumonks.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,6 +17,10 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import argparse
+import re
+from typing import Optional, Union
+
 from smartcard.CardConnection import CardConnection
 from smartcard.CardRequest import CardRequest
 from smartcard.Exceptions import NoCardException, CardRequestTimeoutException, CardConnectionException, CardConnectionException
@@ -24,18 +28,33 @@ from smartcard.System import readers
 
 from pySim.exceptions import NoCardError, ProtocolError, ReaderError
 from pySim.transport import LinkBase
-from pySim.utils import h2i, i2h
+from pySim.utils import h2i, i2h, Hexstr, ResTuple
 
 
 class PcscSimLink(LinkBase):
     """ pySim: PCSC reader transport link."""
+    name = 'PC/SC'
 
-    def __init__(self, reader_number: int = 0, **kwargs):
+    def __init__(self, opts: argparse.Namespace = argparse.Namespace(pcsc_dev=0), **kwargs):
         super().__init__(**kwargs)
+        self._reader = None
         r = readers()
-        if reader_number >= len(r):
-            raise ReaderError('No reader found for number %d' % reader_number)
-        self._reader = r[reader_number]
+        if opts.pcsc_dev is not None:
+            # actual reader index number (integer)
+            reader_number = opts.pcsc_dev
+            if reader_number >= len(r):
+                raise ReaderError('No reader found for number %d' % reader_number)
+            self._reader = r[reader_number]
+        else:
+            # reader regex string
+            cre = re.compile(opts.pcsc_regex)
+            for reader in r:
+                if cre.search(reader.name):
+                    self._reader = reader
+                    break
+            if not self._reader:
+                raise ReaderError('No matching reader found for regex %s' % opts.pcsc_regex)
+
         self._con = self._reader.createConnection()
 
     def __del__(self):
@@ -46,7 +65,7 @@ class PcscSimLink(LinkBase):
             pass
         return
 
-    def wait_for_card(self, timeout: int = None, newcardonly: bool = False):
+    def wait_for_card(self, timeout: Optional[int] = None, newcardonly: bool = False):
         cr = CardRequest(readers=[self._reader],
                          timeout=timeout, newcardonly=newcardonly)
         try:
@@ -68,7 +87,7 @@ class PcscSimLink(LinkBase):
         except NoCardException:
             raise NoCardError()
 
-    def get_atr(self):
+    def get_atr(self) -> Hexstr:
         return self._con.getATR()
 
     def disconnect(self):
@@ -79,7 +98,7 @@ class PcscSimLink(LinkBase):
         self.connect()
         return 1
 
-    def _send_apdu_raw(self, pdu):
+    def _send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
 
         apdu = h2i(pdu)
 
@@ -89,3 +108,20 @@ class PcscSimLink(LinkBase):
 
         # Return value
         return i2h(data), i2h(sw)
+
+    def __str__(self) -> str:
+        return "PCSC[%s]" % (self._reader)
+
+    @staticmethod
+    def argparse_add_reader_args(arg_parser: argparse.ArgumentParser):
+        pcsc_group = arg_parser.add_argument_group('PC/SC Reader',
+        """Use a PC/SC card reader to talk to the SIM card.  PC/SC is a standard API for how applications
+access smart card readers, and is available on a variety of operating systems, such as Microsoft
+Windows, MacOS X and Linux.  Most vendors of smart card readers provide drivers that offer a PC/SC
+interface, if not even a generic USB CCID driver is used.  You can use a tool like ``pcsc_scan -r``
+to obtain a list of readers available on your system. """)
+        dev_group = pcsc_group.add_mutually_exclusive_group()
+        dev_group.add_argument('-p', '--pcsc-device', type=int, dest='pcsc_dev', metavar='PCSC', default=None,
+                               help='Number of PC/SC reader to use for SIM access')
+        dev_group.add_argument('--pcsc-regex', type=str, dest='pcsc_regex', metavar='REGEX', default=None,
+                               help='Regex matching PC/SC reader to use for SIM access')

pySim/transport/serial.py

@@ -18,30 +18,33 @@
 
 import serial
 import time
-import os.path
+import os
+import argparse
+from typing import Optional
 
 from pySim.exceptions import NoCardError, ProtocolError
 from pySim.transport import LinkBase
-from pySim.utils import h2b, b2h
+from pySim.utils import h2b, b2h, Hexstr, ResTuple
 
 
 class SerialSimLink(LinkBase):
     """ pySim: Transport Link for serial (RS232) based readers included with simcard"""
+    name = 'Serial'
 
-    def __init__(self, device: str = '/dev/ttyUSB0', baudrate: int = 9600, rst: str = '-rts',
+    def __init__(self, opts = argparse.Namespace(device='/dev/ttyUSB0', baudrate=9600), rst: str = '-rts',
                  debug: bool = False, **kwargs):
         super().__init__(**kwargs)
-        if not os.path.exists(device):
-            raise ValueError("device file %s does not exist -- abort" % device)
+        if not os.path.exists(opts.device):
+            raise ValueError("device file %s does not exist -- abort" % opts.device)
         self._sl = serial.Serial(
-            port=device,
+            port=opts.device,
             parity=serial.PARITY_EVEN,
             bytesize=serial.EIGHTBITS,
             stopbits=serial.STOPBITS_TWO,
             timeout=1,
             xonxoff=0,
             rtscts=0,
-            baudrate=baudrate,
+            baudrate=opts.baudrate,
         )
         self._rst_pin = rst
         self._debug = debug
@@ -51,7 +54,7 @@ class SerialSimLink(LinkBase):
         if (hasattr(self, "_sl")):
             self._sl.close()
 
-    def wait_for_card(self, timeout=None, newcardonly=False):
+    def wait_for_card(self, timeout: Optional[int] = None, newcardonly: bool = False):
         # Direct try
         existing = False
 
@@ -92,7 +95,7 @@ class SerialSimLink(LinkBase):
     def connect(self):
         self.reset_card()
 
-    def get_atr(self):
+    def get_atr(self) -> Hexstr:
         return self._atr
 
     def disconnect(self):
@@ -104,6 +107,7 @@ class SerialSimLink(LinkBase):
             raise NoCardError()
         elif rv < 0:
             raise ProtocolError()
+        return rv
 
     def _reset_card(self):
         self._atr = None
@@ -183,7 +187,7 @@ class SerialSimLink(LinkBase):
     def _rx_byte(self):
         return self._sl.read()
 
-    def _send_apdu_raw(self, pdu):
+    def _send_apdu_raw(self, pdu: Hexstr) -> ResTuple:
 
         pdu = h2b(pdu)
         data_len = pdu[4]  # P3
@@ -234,3 +238,16 @@ class SerialSimLink(LinkBase):
 
         # Return value
         return b2h(data), b2h(sw)
+
+    def __str__(self) -> str:
+        return "serial:%s" % (self._sl.name)
+
+    @staticmethod
+    def argparse_add_reader_args(arg_parser: argparse.ArgumentParser):
+        serial_group = arg_parser.add_argument_group('Serial Reader', """Use a simple/ultra-low-cost serial reader
+attached to a (physical or USB/virtual) RS232 port.  This doesn't work with all RS232-attached smart card
+readers, only with the very primitive readers following the ancient `Phoenix` or `Smart Mouse` design.""")
+        serial_group.add_argument('-d', '--device', metavar='DEV', default='/dev/ttyUSB0',
+                                  help='Serial Device for SIM access')
+        serial_group.add_argument('-b', '--baud', dest='baudrate', type=int, metavar='BAUD', default=9600,
+                                  help='Baud rate used for SIM access')

pySim/ts_102_221.py

@@ -31,7 +31,9 @@ import pySim.iso7816_4 as iso7816_4
 
 # A UICC will usually also support 2G functionality. If this is the case, we
 # need to add DF_GSM and DF_TELECOM along with the UICC related files
-from pySim.ts_51_011 import DF_GSM, DF_TELECOM
+from pySim.ts_51_011 import DF_GSM, DF_TELECOM, AddonSIM
+from pySim.gsm_r import AddonGSMR
+from pySim.cdma_ruim import AddonRUIM
 
 ts_102_22x_cmdset = CardCommandSet('TS 102 22x', [
     # TS 102 221 Section 10.1.2 Table 10.5 "Coding of Instruction Byte"
@@ -88,14 +90,24 @@ class TotalFileSize(BER_TLV_IE, tag=0x81):
 
 # ETSI TS 102 221 11.1.1.4.3
 class FileDescriptor(BER_TLV_IE, tag=0x82):
+    _test_decode = [
+        # FIXME: this doesn't work as _encode test for some strange reason.
+        ( '82027921', { "file_descriptor_byte": { "shareable": True, "structure": "ber_tlv" }, "record_len": None, "num_of_rec": None } ),
+    ]
+    _test_de_encode = [
+        ( '82027821', { "file_descriptor_byte": { "shareable": True, "file_type": "df", "structure": "no_info_given" }, "record_len": None, "num_of_rec": None }),
+        ( '82024121', { "file_descriptor_byte": { "shareable": True, "file_type": "working_ef", "structure": "transparent" }, "record_len": None, "num_of_rec": None } ),
+        ( '82054221006e05', { "file_descriptor_byte": { "shareable": True, "file_type": "working_ef", "structure": "linear_fixed" }, "record_len": 110, "num_of_rec": 5 } ),
+    ]
     class BerTlvAdapter(Adapter):
         def _parse(self, obj, context, path):
-            if obj == 0x39:
+            data = obj.read()
+            if data == b'\x01\x01\x01\x00\x00\x01':
                 return 'ber_tlv'
             raise ValidationError
         def _build(self, obj, context, path):
             if obj == 'ber_tlv':
-                return 0x39
+                return b'\x01\x01\x01\x00\x00\x01'
             raise ValidationError
 
     FDB = Select(BitStruct(Const(0, Bit), 'shareable'/Flag, 'structure'/BerTlvAdapter(Const(0x39, BitsInteger(6)))),
@@ -198,6 +210,9 @@ class ShortFileIdentifier(BER_TLV_IE, tag=0x88):
 
 # ETSI TS 102 221 11.1.1.4.9
 class LifeCycleStatusInteger(BER_TLV_IE, tag=0x8A):
+    _test_de_encode = [
+        ( '8a0105', 'operational_activated' ),
+    ]
     def _from_bytes(self, do: bytes):
         lcsi = int.from_bytes(do, 'big')
         if lcsi == 0x00:
@@ -574,6 +589,18 @@ SC_DO = DataObjectChoice('security_condition', 'Security Condition',
 
 # TS 102 221 Section 13.1
 class EF_DIR(LinFixedEF):
+    # FIXME: re-encode failure when changing to _test_de_encode
+    _test_decode = [
+        ( '61294f10a0000000871002ffffffff890709000050055553696d31730ea00c80011781025f608203454150',
+          { "application_template": [ { "application_id": h2b("a0000000871002ffffffff8907090000") },
+                                      { "application_label": "USim1" },
+                                      { "discretionary_template": h2b("a00c80011781025f608203454150") } ] }
+        ),
+        ( '61194f10a0000000871004ffffffff890709000050054953696d31',
+          { "application_template": [ { "application_id": h2b("a0000000871004ffffffff8907090000") },
+                                      { "application_label": "ISim1" } ] }
+        ),
+    ]
     class ApplicationLabel(BER_TLV_IE, tag=0x50):
         # TODO: UCS-2 coding option as per Annex A of TS 102 221
         _construct = GreedyString('ascii')
@@ -593,6 +620,9 @@ class EF_DIR(LinFixedEF):
 
 # TS 102 221 Section 13.2
 class EF_ICCID(TransparentEF):
+    _test_de_encode = [
+        ( '988812010000400310f0', { "iccid": "8988211000000430010" } ),
+    ]
     def __init__(self, fid='2fe2', sfid=0x02, name='EF.ICCID', desc='ICC Identification'):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=(10, 10))
 
@@ -604,17 +634,23 @@ class EF_ICCID(TransparentEF):
 
 # TS 102 221 Section 13.3
 class EF_PL(TransRecEF):
+    _test_de_encode = [
+        ( '6465', "de" ),
+        ( '656e', "en" ),
+        ( 'ffff', None ),
+    ]
+
     def __init__(self, fid='2f05', sfid=0x05, name='EF.PL', desc='Preferred Languages'):
         super().__init__(fid, sfid=sfid, name=name,
                          desc=desc, rec_len=2, size=(2, None))
 
-    def _decode_record_bin(self, bin_data):
+    def _decode_record_bin(self, bin_data, **kwargs):
         if bin_data == b'\xff\xff':
             return None
         else:
             return bin_data.decode('ascii')
 
-    def _encode_record_bin(self, in_json):
+    def _encode_record_bin(self, in_json, **kwargs):
         if in_json is None:
             return b'\xff\xff'
         else:
@@ -623,6 +659,28 @@ class EF_PL(TransRecEF):
 
 # TS 102 221 Section 13.4
 class EF_ARR(LinFixedEF):
+    _test_de_encode = [
+        ( '800101a40683010a950108800106900080016097008401d4a40683010a950108',
+         [ [ { "access_mode": [ "read_search_compare" ] },
+             { "control_reference_template": "ADM1" } ],
+           [ { "access_mode": [ "write_append", "update_erase" ] },
+             { "always": None } ],
+           [ { "access_mode": [ "delete_file", "terminate_ef" ] },
+             { "never": None } ],
+           [ { "command_header": { "INS": 212 } },
+             { "control_reference_template": "ADM1" } ]
+         ] ),
+        ( '80010190008001029700800118a40683010a9501088401d4a40683010a950108',
+         [ [ { "access_mode": [ "read_search_compare" ] },
+             { "always": None } ],
+           [ { "access_mode": [ "update_erase" ] },
+             { "never": None } ],
+           [ { "access_mode": [ "activate_file_or_record", "deactivate_file_or_record" ] },
+             { "control_reference_template": "ADM1" } ],
+           [ { "command_header": { "INS": 212 } },
+             { "control_reference_template": "ADM1" } ]
+         ] ),
+    ]
     def __init__(self, fid='2f06', sfid=0x06, name='EF.ARR', desc='Access Rule Reference'):
         super().__init__(fid, sfid=sfid, name=name, desc=desc)
         # add those commands to the general commands of a TransparentEF
@@ -665,7 +723,7 @@ class EF_ARR(LinFixedEF):
                 raise ValueError
         return by_mode
 
-    def _decode_record_bin(self, raw_bin_data):
+    def _decode_record_bin(self, raw_bin_data, **kwargs):
         # we can only guess if we should decode for EF or DF here :(
         arr_seq = DataObjectSequence('arr', sequence=[AM_DO_EF, SC_DO])
         dec = arr_seq.decode_multi(raw_bin_data)
@@ -673,7 +731,7 @@ class EF_ARR(LinFixedEF):
         # 'un-flattening' decoder, and hence would be unable to encode :(
         return dec[0]
 
-    def _encode_record_bin(self, in_json):
+    def _encode_record_bin(self, in_json, **kwargs):
         # we can only guess if we should decode for EF or DF here :(
         arr_seq = DataObjectSequence('arr', sequence=[AM_DO_EF, SC_DO])
         return arr_seq.encode_multi(in_json)
@@ -705,6 +763,12 @@ class EF_ARR(LinFixedEF):
 
 # TS 102 221 Section 13.6
 class EF_UMPC(TransparentEF):
+    _test_de_encode = [
+        ( '3cff02', { "max_current_mA": 60, "t_op_s": 255,
+                      "addl_info": { "req_inc_idle_current": False, "support_uicc_suspend": True } } ),
+        ( '320500', { "max_current_mA": 50, "t_op_s": 5, "addl_info": {"req_inc_idle_current": False,
+                                                                       "support_uicc_suspend": False } } ),
+    ]
     def __init__(self, fid='2f08', sfid=0x08, name='EF.UMPC', desc='UICC Maximum Power Consumption'):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=(5, 5))
         addl_info = FlagsEnum(Byte, req_inc_idle_current=1,
@@ -726,6 +790,11 @@ class CardProfileUICC(CardProfile):
             # FIXME: DF.CD
             EF_UMPC(),
         ]
+        addons = [
+            AddonSIM,
+            AddonGSMR,
+            AddonRUIM,
+        ]
         sw = {
             'Normal': {
                 '9000': 'Normal ending of the command',
@@ -796,7 +865,8 @@ class CardProfileUICC(CardProfile):
         }
 
         super().__init__(name, desc='ETSI TS 102 221', cla="00",
-                         sel_ctrl="0004", files_in_mf=files, sw=sw)
+                         sel_ctrl="0004", files_in_mf=files, sw=sw,
+                         shell_cmdsets = [self.AddlShellCommands()], addons = addons)
 
     @staticmethod
     def decode_select_response(resp_hex: str) -> object:
@@ -810,19 +880,29 @@ class CardProfileUICC(CardProfile):
     def match_with_card(scc: SimCardCommands) -> bool:
         return match_uicc(scc)
 
+    @with_default_category('TS 102 221 Specific Commands')
+    class AddlShellCommands(CommandSet):
+        suspend_uicc_parser = argparse.ArgumentParser()
+        suspend_uicc_parser.add_argument('--min-duration-secs', type=int, default=60,
+                                         help='Proposed minimum duration of suspension')
+        suspend_uicc_parser.add_argument('--max-duration-secs', type=int, default=24*60*60,
+                                         help='Proposed maximum duration of suspension')
 
-class CardProfileUICCSIM(CardProfileUICC):
-    """Same as above, but including 2G SIM support"""
+        # not ISO7816-4 but TS 102 221
+        @cmd2.with_argparser(suspend_uicc_parser)
+        def do_suspend_uicc(self, opts):
+            """Perform the SUSPEND UICC command. Only supported on some UICC (check EF.UMPC)."""
+            (duration, token, sw) = self._cmd.card._scc.suspend_uicc(min_len_secs=opts.min_duration_secs,
+                                                                     max_len_secs=opts.max_duration_secs)
+            self._cmd.poutput(
+                'Negotiated Duration: %u secs, Token: %s, SW: %s' % (duration, token, sw))
 
-    ORDER = 0
+        resume_uicc_parser = argparse.ArgumentParser()
+        resume_uicc_parser.add_argument('token', type=str, help='Token provided during SUSPEND')
 
-    def __init__(self):
-        super().__init__('UICC-SIM')
-
-        # Add GSM specific files
-        self.files_in_mf.append(DF_TELECOM())
-        self.files_in_mf.append(DF_GSM())
-
-    @staticmethod
-    def match_with_card(scc: SimCardCommands) -> bool:
-        return match_uicc(scc) and match_sim(scc)
+        @cmd2.with_argparser(resume_uicc_parser)
+        def do_resume_uicc(self, opts):
+            """Perform the REUSME UICC operation. Only supported on some UICC. Also: A power-cycle
+            of the card is required between SUSPEND and RESUME, and only very few non-RESUME
+            commands are permitted between SUSPEND and RESUME.  See TS 102 221 Section 11.1.22."""
+            self._cmd.card._scc.resume_uicc(opts.token)

pySim/ts_102_222.py

@@ -23,9 +23,6 @@ import cmd2
 from cmd2 import CommandSet, with_default_category, with_argparser
 import argparse
 
-from pySim.ts_31_102 import EF_UST_map, EF_USIM_ADF_map
-from pySim.ts_31_103 import EF_IST_map, EF_ISIM_ADF_map
-
 from pySim.exceptions import *
 from pySim.utils import h2b, swap_nibbles, b2h, JsonEncoder
 
@@ -52,7 +49,7 @@ class Ts102222Commands(CommandSet):
             self._cmd.perror("Refusing to permanently delete the file, please read the help text.")
             return
         f = self._cmd.lchan.get_file_for_selectable(opts.NAME)
-        (data, sw) = self._cmd.card._scc.delete_file(f.fid)
+        (data, sw) = self._cmd.lchan.scc.delete_file(f.fid)
 
     def complete_delete_file(self, text, line, begidx, endidx) -> List[str]:
         """Command Line tab completion for DELETE FILE"""
@@ -73,7 +70,7 @@ class Ts102222Commands(CommandSet):
             self._cmd.perror("Refusing to terminate the file, please read the help text.")
             return
         f = self._cmd.lchan.get_file_for_selectable(opts.NAME)
-        (data, sw) = self._cmd.card._scc.terminate_df(f.fid)
+        (data, sw) = self._cmd.lchan.scc.terminate_df(f.fid)
 
     def complete_terminate_df(self, text, line, begidx, endidx) -> List[str]:
         """Command Line tab completion for TERMINATE DF"""
@@ -89,7 +86,7 @@ class Ts102222Commands(CommandSet):
             self._cmd.perror("Refusing to terminate the file, please read the help text.")
             return
         f = self._cmd.lchan.get_file_for_selectable(opts.NAME)
-        (data, sw) = self._cmd.card._scc.terminate_ef(f.fid)
+        (data, sw) = self._cmd.lchan.scc.terminate_ef(f.fid)
 
     def complete_terminate_ef(self, text, line, begidx, endidx) -> List[str]:
         """Command Line tab completion for TERMINATE EF"""
@@ -107,7 +104,7 @@ class Ts102222Commands(CommandSet):
         if not opts.force_terminate_card:
             self._cmd.perror("Refusing to permanently terminate the card, please read the help text.")
             return
-        (data, sw) = self._cmd.card._scc.terminate_card_usage()
+        (data, sw) = self._cmd.lchan.scc.terminate_card_usage()
 
     create_parser = argparse.ArgumentParser()
     create_parser.add_argument('FILE_ID', type=str, help='File Identifier as 4-character hex string')
@@ -152,7 +149,7 @@ class Ts102222Commands(CommandSet):
                ShortFileIdentifier(decoded=opts.short_file_id),
             ]
         fcp = FcpTemplate(children=ies)
-        (data, sw) = self._cmd.card._scc.create_file(b2h(fcp.to_tlv()))
+        (data, sw) = self._cmd.lchan.scc.create_file(b2h(fcp.to_tlv()))
         # the newly-created file is automatically selected but our runtime state knows nothing of it
         self._cmd.lchan.select_file(self._cmd.lchan.selected_file)
 
@@ -203,6 +200,28 @@ class Ts102222Commands(CommandSet):
                    }
                ies.append(ProprietaryInformation(children=[ToolkitAccessConditions(decoded=toolkit_ac)]))
         fcp = FcpTemplate(children=ies)
-        (data, sw) = self._cmd.card._scc.create_file(b2h(fcp.to_tlv()))
+        (data, sw) = self._cmd.lchan.scc.create_file(b2h(fcp.to_tlv()))
         # the newly-created file is automatically selected but our runtime state knows nothing of it
         self._cmd.lchan.select_file(self._cmd.lchan.selected_file)
+
+    resize_ef_parser = argparse.ArgumentParser()
+    resize_ef_parser.add_argument('NAME', type=str, help='Name or FID of file to be resized')
+    resize_ef_parser._action_groups.pop()
+    resize_ef_required = resize_ef_parser.add_argument_group('required arguments')
+    resize_ef_required.add_argument('--file-size', required=True, type=int, help='Size of file in octets')
+
+    @cmd2.with_argparser(resize_ef_parser)
+    def do_resize_ef(self, opts):
+        """Resize an existing EF below the currently selected DF.  Requires related privileges."""
+        f = self._cmd.lchan.get_file_for_selectable(opts.NAME)
+        ies = [FileIdentifier(decoded=f.fid),
+               FileSize(decoded=opts.file_size)]
+        fcp = FcpTemplate(children=ies)
+        (data, sw) = self._cmd.lchan.scc.resize_file(b2h(fcp.to_tlv()))
+        # the resized file is automatically selected but our runtime state knows nothing of it
+        self._cmd.lchan.select_file(self._cmd.lchan.selected_file)
+
+    def complete_resize_ef(self, text, line, begidx, endidx) -> List[str]:
+        """Command Line tab completion for RESIZE EF"""
+        index_dict = {1: self._cmd.lchan.selected_file.get_selectable_names()}
+        return self._cmd.index_based_complete(text, line, begidx, endidx, index_dict=index_dict)

pySim/ts_102_310.py

@@ -0,0 +1,114 @@
+# coding=utf-8
+"""Utilities / Functions related to ETSI TS 102 310, the EAP UICC spec.
+
+(C) 2024 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from pySim.construct import *
+from construct import *
+from construct import Optional as COptional
+
+#from pySim.utils import *
+from pySim.filesystem import CardDF, TransparentEF
+from pySim.tlv import BER_TLV_IE, TLV_IE_Collection
+
+# TS102 310 Section 7.1
+class EF_EAPKEYS(TransparentEF):
+    class Msk(BER_TLV_IE, tag=0x80):
+        _construct = HexAdapter(GreedyBytes)
+    class Emsk(BER_TLV_IE, tag=0x81):
+        _construct = HexAdapter(GreedyBytes)
+    class MskCollection(TLV_IE_Collection, nested=[EF_EAPKEYS.Msk, EF_EAPKEYS.Emsk]):
+        pass
+
+    def __init__(self, fid='4f01', name='EF.EAPKEYS', desc='EAP derived keys'):
+        super().__init__(fid, sfid=0x01, name=name, desc=desc, size=(1,None))
+        self._tlv = EF_EAPKEYS.MskCollection
+
+# TS 102 310 Section 7.2
+class EF_EAPSTATUS(TransparentEF):
+    def __init__(self, fid='4f02', name='EF.EAPSTATUS', desc='EAP Authentication Status'):
+        super().__init__(fid, sfid=0x02, name=name, desc=desc, size=(1,1))
+        self._construct = Enum(Int8ub, no_auth_started=0, authenticating=1,
+                               authenticated=2, held_auth_failure=3)
+
+# TS 102 310 Section 7.3
+class EF_PUId(TransparentEF):
+    def __init__(self, fid='4f03', name='EF.PUId', desc='Permanent User Identity'):
+        super().__init__(fid, sfid=0x03, name=name, desc=desc, size=(10,None))
+        self._construct = GreedyBytes
+
+# TS 102 310 Section 7.4
+class EF_Ps(TransparentEF):
+    def __init__(self, fid='4f04', name='EF.Ps', desc='Pseudonym'):
+        super().__init__(fid, sfid=0x04, name=name, desc=desc, size=(1,None))
+        self._construct = GreedyBytes
+
+# TS 102 310 Section 7.5
+class EF_CurID(TransparentEF):
+    def __init__(self, fid='4f20', name='EF.CurID', desc='Current Identity'):
+        super().__init__(fid, sfid=0x10, name=name, desc=desc, size=(1,None))
+        self._construct = Struct('type'/Enum(Int8ub, permanent=0, pseudonym=1, re_authentication=2, should_not_be_revealed=255),
+                                 '_len'/Int8ub,
+                                 'value'/Utf8Adapter(this._len))
+
+
+# TS 102 310 Section 7.6
+class EF_ReID(TransparentEF):
+    class Identity(BER_TLV_IE, tag=0x80):
+        _construct = Utf8Adapter(GreedyBytes)
+    class Counter(BER_TLV_IE, tag=0x81):
+        _construct = GreedyInteger
+    class Collection(TLV_IE_Collection, nested=[EF_ReID.Identity, EF_ReID.Counter]):
+        pass
+
+    def __init__(self, fid='4f21', name='EF.ReID', desc='Re-Authentication Identity'):
+        super().__init__(fid, sfid=0x11, name=name, desc=desc, size=(1,None))
+        self._tlv = EF_ReID.Collection
+
+# TS 102 310 Section 7.7
+class EF_Realm(TransparentEF):
+    def __init__(self, fid='4f22', name='EF.Realm', desc='Relm value of the identity'):
+        super().__init__(fid, sfid=0x12, name=name, desc=desc, size=(1,None))
+        self._construct = Struct('_len'/Int8ub,
+                                 'realm'/Utf8Adapter(Bytes(this._len)))
+
+class DF_EAP(CardDF):
+    # DF.EAP has no default FID; it always must be discovered via the EF.DIR entry
+    # and the 0x73 "discretionary template"
+    def __init__(self, fid, name='DF.EAP', desc='EAP client', **kwargs):
+        super().__init__(fid=fid, name=name, desc=desc, **kwargs)
+        files = [
+            EF_EAPKEYS(),
+            EF_EAPSTATUS(),
+            EF_PUId(),
+            EF_CurID(),
+            EF_ReID(),
+        ]
+        self.add_files(files)
+
+
+# TS 102 310 Section 5.2
+class EapSupportedTypesList(BER_TLV_IE, tag=0x80):
+    _construct = GreedyRange(Int8ub)
+class EapDedicatedFilesList(BER_TLV_IE, tag=0x81):
+    _construct = GreedyRange(Int16ub)
+class EapLabel(BER_TLV_IE, tag=0x82):
+    _construct = GreedyBytes
+class EapAppSvcSpecData(BER_TLV_IE, tag=0xa0, nested=[EapSupportedTypesList, EapDedicatedFilesList, EapLabel]):
+    pass
+class DiscretionaryTemplate(BER_TLV_IE, tag=0x73, nested=[EapAppSvcSpecData]):
+    pass

pySim/ts_31_102_telecom.py

@@ -120,10 +120,30 @@ class EF_UServiceTable(TransparentEF):
             cmd.lchan.select_file(self)
         return num_problems
 
+    def ust_update(self, cmd, activate=[], deactivate=[]):
+        service_data, sw = cmd.lchan.read_binary()
+        service_data = h2b(service_data)
 
+        for service in activate:
+            nbyte, nbit = EF_UServiceTable._bit_byte_offset_for_service(service)
+            if nbyte > len(service_data):
+                missing = nbyte - service_data
+                service_data.extend(missing * "00")
+            service_data[nbyte] |= (1 << nbit)
+
+        for service in deactivate:
+            nbyte, nbit = EF_UServiceTable._bit_byte_offset_for_service(service)
+            if nbyte > len(service_data):
+                missing = nbyte - service_data
+                service_data.extend(missing * "00")
+            service_data[nbyte] &= ~(1 << nbit)
+
+        service_data = b2h(service_data)
+        cmd.lchan.update_binary(service_data)
 
 # TS 31.102 Section 4.4.2.1
 class EF_PBR(LinFixedEF):
+    # TODO: a80ac0034f3a02c5034f0904aa0acb034f3d07c2034f4a06
     def __init__(self, fid='4F30', name='EF.PBR', desc='Phone Book Reference', **kwargs):
         super().__init__(fid, name=name, desc=desc, **kwargs)
         #self._tlv = FIXME

pySim/ts_31_103.py

@@ -30,6 +30,7 @@ from pySim.ts_31_102 import ADF_USIM, EF_FromPreferred
 from pySim.ts_31_102_telecom import EF_UServiceTable
 import pySim.ts_102_221
 from pySim.ts_102_221 import EF_ARR
+from pySim.construct import *
 
 # Mapping between ISIM Service Number and its description
 EF_IST_map = {
@@ -56,32 +57,15 @@ EF_IST_map = {
     21: 'MuD and MiD configuration data',
 }
 
-EF_ISIM_ADF_map = {
-    'IST': '6F07',
-    'IMPI': '6F02',
-    'DOMAIN': '6F03',
-    'IMPU': '6F04',
-    'AD': '6FAD',
-    'ARR': '6F06',
-    'PCSCF': '6F09',
-    'GBAP': '6FD5',
-    'GBANL': '6FD7',
-    'NAFKCA': '6FDD',
-    'UICCIARI': '6FE7',
-    'SMS': '6F3C',
-    'SMSS': '6F43',
-    'SMSR': '6F47',
-    'SMSP': '6F42',
-    'FromPreferred': '6FF7',
-    'IMSConfigData': '6FF8',
-    'XCAPConfigData': '6FFC',
-    'WebRTCURI': '6FFA'
-}
-
 # TS 31.103 Section 4.2.2
 class EF_IMPI(TransparentEF):
+    _test_de_encode = [
+        ( '803137333830303630303030303031303140696d732e6d6e633030302e6d63633733382e336770706e6574776f726b2e6f7267',
+          { "nai": "738006000000101@ims.mnc000.mcc738.3gppnetwork.org" } ),
+    ]
+
     class nai(BER_TLV_IE, tag=0x80):
-        _construct = GreedyString("utf8")
+        _construct = Utf8Adapter(GreedyBytes)
 
     def __init__(self, fid='6f02', sfid=0x02, name='EF.IMPI', desc='IMS private user identity', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
@@ -89,8 +73,12 @@ class EF_IMPI(TransparentEF):
 
 # TS 31.103 Section 4.2.3
 class EF_DOMAIN(TransparentEF):
+    _test_de_encode = [
+        ( '8021696d732e6d6e633030302e6d63633733382e336770706e6574776f726b2e6f7267',
+          { "domain": "ims.mnc000.mcc738.3gppnetwork.org" } ),
+    ]
     class domain(BER_TLV_IE, tag=0x80):
-        _construct = GreedyString("utf8")
+        _construct = Utf8Adapter(GreedyBytes)
 
     def __init__(self, fid='6f03', sfid=0x05, name='EF.DOMAIN', desc='Home Network Domain Name', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
@@ -98,8 +86,12 @@ class EF_DOMAIN(TransparentEF):
 
 # TS 31.103 Section 4.2.4
 class EF_IMPU(LinFixedEF):
+    _test_de_encode = [
+        ( '80357369703a37333830303630303030303031303140696d732e6d6e633030302e6d63633733382e336770706e6574776f726b2e6f7267',
+          { "impu": "sip:738006000000101@ims.mnc000.mcc738.3gppnetwork.org" } ),
+    ]
     class impu(BER_TLV_IE, tag=0x80):
-        _construct = GreedyString("utf8")
+        _construct = Utf8Adapter(GreedyBytes)
 
     def __init__(self, fid='6f04', sfid=0x04, name='EF.IMPU', desc='IMS public user identity', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
@@ -119,11 +111,13 @@ class EF_IST(EF_UServiceTable):
 
         def do_ist_service_activate(self, arg):
             """Activate a service within EF.IST"""
-            self._cmd.card.update_ist(int(arg), 1)
+            selected_file = self._cmd.lchan.selected_file
+            selected_file.ust_update(self._cmd, [int(arg)], [])
 
         def do_ist_service_deactivate(self, arg):
             """Deactivate a service within EF.IST"""
-            self._cmd.card.update_ist(int(arg), 0)
+            selected_file = self._cmd.lchan.selected_file
+            selected_file.ust_update(self._cmd, [], [int(arg)])
 
         def do_ist_service_check(self, arg):
             """Check consistency between services of this file and files present/activated.
@@ -139,37 +133,65 @@ class EF_IST(EF_UServiceTable):
 
 # TS 31.103 Section 4.2.8
 class EF_PCSCF(LinFixedEF):
+    _test_de_encode = [
+        ( '802c0070637363662e696d732e6d6e633030302e6d63633733382e7075622e336770706e6574776f726b2e6f7267',
+          {'pcscf_address': { "address": "pcscf.ims.mnc000.mcc738.pub.3gppnetwork.org", "type_of_address": "FQDN" } } ),
+        ( '800501c0a80c22',
+          {'pcscf_address': { "address": "192.168.12.34", "type_of_address": "IPv4" } } ),
+        ( '801102fe800000000000000042d7fffe530335',
+          {'pcscf_address': { "address": "fe80::42:d7ff:fe53:335", "type_of_address": "IPv6" } } ),
+    ]
+    class PcscfAddress(BER_TLV_IE, tag=0x80):
+        _construct = Struct('type_of_address'/Enum(Byte, FQDN=0, IPv4=1, IPv6=2),
+                            'address'/Switch(this.type_of_address,
+                                             {'FQDN': Utf8Adapter(GreedyBytes),
+                                              'IPv4': Ipv4Adapter(GreedyBytes),
+                                              'IPv6': Ipv6Adapter(GreedyBytes)}))
+
     def __init__(self, fid='6f09', sfid=None, name='EF.P-CSCF', desc='P-CSCF Address', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
-
-    def _decode_record_hex(self, raw_hex):
-        addr, addr_type = dec_addr_tlv(raw_hex)
-        return {"addr": addr, "addr_type": addr_type}
-
-    def _encode_record_hex(self, json_in):
-        addr = json_in['addr']
-        addr_type = json_in['addr_type']
-        return enc_addr_tlv(addr, addr_type)
+        self._tlv = EF_PCSCF.PcscfAddress
 
 # TS 31.103 Section 4.2.9
 class EF_GBABP(TransparentEF):
     def __init__(self, fid='6fd5', sfid=None, name='EF.GBABP', desc='GBA Bootstrapping', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
+        self._construct = Struct('rand'/LV,
+                                 'b_tid'/LV,
+                                 'key_lifetime'/LV)
 
 # TS 31.103 Section 4.2.10
 class EF_GBANL(LinFixedEF):
+    class NAF_ID(BER_TLV_IE, tag=0x80):
+        _construct = Struct('fqdn'/Utf8Adapter(Bytes(this._.total_len-5)),
+                            'ua_spi'/HexAdapter(Bytes(5)))
+    class B_TID(BER_TLV_IE, tag=0x81):
+        _construct = Utf8Adapter(GreedyBytes)
+    # pylint: disable=undefined-variable
+    class GbaNlCollection(TLV_IE_Collection, nested=[NAF_ID, B_TID]):
+        pass
     def __init__(self, fid='6fd7', sfid=None, name='EF.GBANL', desc='GBA NAF List', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
+        self._tlv = EF_GBANL.GbaNlCollection
 
 # TS 31.103 Section 4.2.11
 class EF_NAFKCA(LinFixedEF):
+    _test_de_encode = [
+        ( '80296273662e696d732e6d6e633030302e6d63633733382e7075622e336770706e6574776f726b2e6f7267',
+          { 'naf_key_centre_address': 'bsf.ims.mnc000.mcc738.pub.3gppnetwork.org' } ),
+        ( '8030656e65746e61667830312e696d732e6d6e633030302e6d63633733382e7075622e336770706e6574776f726b2e6f7267',
+          { 'naf_key_centre_address': 'enetnafx01.ims.mnc000.mcc738.pub.3gppnetwork.org' }),
+    ]
+    class NafKeyCentreAddress(BER_TLV_IE, tag=0x80):
+        _construct = Utf8Adapter(GreedyBytes)
     def __init__(self, fid='6fdd', sfid=None, name='EF.NAFKCA', desc='NAF Key Centre Address', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
+        self._tlv = EF_NAFKCA.NafKeyCentreAddress
 
 # TS 31.103 Section 4.2.16
 class EF_UICCIARI(LinFixedEF):
     class iari(BER_TLV_IE, tag=0x80):
-        _construct = GreedyString("utf8")
+        _construct = Utf8Adapter(GreedyBytes)
 
     def __init__(self, fid='6fe7', sfid=None, name='EF.UICCIARI', desc='UICC IARI', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
@@ -236,7 +258,7 @@ class EF_XCAPConfigData(BerTlvEF):
 # TS 31.103 Section 4.2.20
 class EF_WebRTCURI(TransparentEF):
     class uri(BER_TLV_IE, tag=0x80):
-        _construct = GreedyString("utf8")
+        _construct = Utf8Adapter(GreedyBytes)
 
     def __init__(self, fid='6ffa', sfid=None, name='EF.WebRTCURI', desc='WebRTC URI', **kwargs):
         super().__init__(fid=fid, sfid=sfid, name=name, desc=desc, **kwargs)
@@ -258,9 +280,9 @@ class EF_MuDMiDConfigData(BerTlvEF):
 
 
 class ADF_ISIM(CardADF):
-    def __init__(self, aid='a0000000871004', name='ADF.ISIM', fid=None, sfid=None,
+    def __init__(self, aid='a0000000871004', has_fs=True, name='ADF.ISIM', fid=None, sfid=None,
                  desc='ISIM Application'):
-        super().__init__(aid=aid, fid=fid, sfid=sfid, name=name, desc=desc)
+        super().__init__(aid=aid, has_fs=has_fs, fid=fid, sfid=sfid, name=name, desc=desc)
 
         files = [
             EF_IMPI(),

pySim/ts_31_104.py

@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+
+"""
+Support for 3GPP TS 31.104 V17.0.0
+"""
+
+# Copyright (C) 2023 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from pySim.filesystem import *
+from pySim.utils import *
+from pySim.tlv import *
+from pySim.ts_31_102 import ADF_USIM
+from pySim.ts_51_011 import EF_IMSI, EF_AD
+import pySim.ts_102_221
+from pySim.ts_102_221 import EF_ARR
+
+
+class ADF_HPSIM(CardADF):
+    def __init__(self, aid='a000000087100A', has_fs=True, name='ADF.HPSIM', fid=None, sfid=None,
+                 desc='HPSIM Application'):
+        super().__init__(aid=aid, has_fs=has_fs, fid=fid, sfid=sfid, name=name, desc=desc)
+
+        files = [
+            EF_ARR(fid='6f06', sfid=0x06),
+            EF_IMSI(fid='6f07', sfid=0x07),
+            EF_AD(fid='6fad', sfid=0x03),
+        ]
+        self.add_files(files)
+        # add those commands to the general commands of a TransparentEF
+        self.shell_commands += [ADF_USIM.AddlShellCommands()]
+
+    def decode_select_response(self, data_hex):
+        return pySim.ts_102_221.CardProfileUICC.decode_select_response(data_hex)
+
+
+# TS 31.104 Section 7.1
+sw_hpsim = {
+    'Security management': {
+        '9862': 'Authentication error, incorrect MAC',
+    }
+}
+
+
+class CardApplicationHPSIM(CardApplication):
+    def __init__(self):
+        super().__init__('HPSIM', adf=ADF_HPSIM(), sw=sw_hpsim)

pySim/ts_51_011.py

@@ -30,9 +30,10 @@ order to describe the files specified in the relevant ETSI + 3GPP specifications
 #
 
 from pySim.profile import match_sim
-from pySim.profile import CardProfile
+from pySim.profile import CardProfile, CardProfileAddon
 from pySim.filesystem import *
 from pySim.ts_31_102_telecom import DF_PHONEBOOK, DF_MULTIMEDIA, DF_MCS, DF_V2X
+from pySim.gsm_r import AddonGSMR
 import enum
 from pySim.construct import *
 from construct import Optional as COptional
@@ -41,236 +42,6 @@ from struct import pack, unpack
 from typing import Tuple
 from pySim.tlv import *
 from pySim.utils import *
-MF_num = '3F00'
-
-DF_num = {
-    'TELECOM': '7F10',
-
-    'GSM': '7F20',
-    'IS-41': '7F22',
-    'FP-CTS': '7F23',
-
-    'GRAPHICS': '5F50',
-
-    'IRIDIUM': '5F30',
-    'GLOBST': '5F31',
-    'ICO': '5F32',
-    'ACeS': '5F33',
-
-    'EIA/TIA-553': '5F40',
-    'CTS': '5F60',
-    'SOLSA': '5F70',
-
-    'MExE': '5F3C',
-}
-
-EF_num = {
-    # MF
-    'ICCID': '2FE2',
-    'ELP': '2F05',
-    'DIR': '2F00',
-
-    # DF_TELECOM
-    'ADN': '6F3A',
-    'FDN': '6F3B',
-    'SMS': '6F3C',
-    'CCP': '6F3D',
-    'MSISDN': '6F40',
-    'SMSP': '6F42',
-    'SMSS': '6F43',
-    'LND': '6F44',
-    'SMSR': '6F47',
-    'SDN': '6F49',
-    'EXT1': '6F4A',
-    'EXT2': '6F4B',
-    'EXT3': '6F4C',
-    'BDN': '6F4D',
-    'EXT4': '6F4E',
-    'CMI': '6F58',
-    'ECCP': '6F4F',
-
-    # DF_GRAPHICS
-    'IMG': '4F20',
-
-    # DF_SoLSA
-    'SAI': '4F30',
-    'SLL': '4F31',
-
-    # DF_MExE
-    'MExE-ST': '4F40',
-    'ORPK': '4F41',
-    'ARPK': '4F42',
-    'TPRPK': '4F43',
-
-    # DF_GSM
-    'LP': '6F05',
-    'IMSI': '6F07',
-    'Kc': '6F20',
-    'DCK': '6F2C',
-    'PLMNsel': '6F30',
-    'HPPLMN': '6F31',
-    'CNL': '6F32',
-    'ACMmax': '6F37',
-    'SST': '6F38',
-    'ACM': '6F39',
-    'GID1': '6F3E',
-    'GID2': '6F3F',
-    'PUCT': '6F41',
-    'CBMI': '6F45',
-    'SPN': '6F46',
-    'CBMID': '6F48',
-    'BCCH': '6F74',
-    'ACC': '6F78',
-    'FPLMN': '6F7B',
-    'LOCI': '6F7E',
-    'AD': '6FAD',
-    'PHASE': '6FAE',
-    'VGCS': '6FB1',
-    'VGCSS': '6FB2',
-    'VBS': '6FB3',
-    'VBSS': '6FB4',
-    'eMLPP': '6FB5',
-    'AAeM': '6FB6',
-    'ECC': '6FB7',
-    'CBMIR': '6F50',
-    'NIA': '6F51',
-    'KcGPRS': '6F52',
-    'LOCIGPRS': '6F53',
-    'SUME': '6F54',
-    'PLMNwAcT': '6F60',
-    'OPLMNwAcT': '6F61',
-    # Figure 8 names it HPLMNAcT, but in the text it's names it HPLMNwAcT
-    'HPLMNAcT': '6F62',
-    'HPLMNwAcT': '6F62',
-    'CPBCCH': '6F63',
-    'INVSCAN': '6F64',
-    'PNN': '6FC5',
-    'OPL': '6FC6',
-    'MBDN': '6FC7',
-    'EXT6': '6FC8',
-    'MBI': '6FC9',
-    'MWIS': '6FCA',
-    'CFIS': '6FCB',
-    'EXT7': '6FCC',
-    'SPDI': '6FCD',
-    'MMSN': '6FCE',
-    'EXT8': '6FCF',
-    'MMSICP': '6FD0',
-    'MMSUP': '6FD1',
-    'MMSUCP': '6FD2',
-}
-
-DF = {
-    'TELECOM':  [MF_num, DF_num['TELECOM']],
-
-    'GSM':      [MF_num, DF_num['GSM']],
-    'IS-41':    [MF_num, DF_num['IS-41']],
-    'FP-CTS':   [MF_num, DF_num['FP-CTS']],
-
-    'GRAPHICS': [MF_num, DF_num['GRAPHICS']],
-
-    'IRIDIUM':  [MF_num, DF_num['IRIDIUM']],
-    'GLOBST':   [MF_num, DF_num['GLOBST']],
-    'ICO':      [MF_num, DF_num['ICO']],
-    'ACeS':     [MF_num, DF_num['ACeS']],
-
-    'EIA/TIA-553': [MF_num, DF_num['EIA/TIA-553']],
-    'CTS':      [MF_num, DF_num['CTS']],
-    'SoLSA':    [MF_num, DF_num['SOLSA']],
-
-    'MExE':     [MF_num, DF_num['MExE']],
-}
-
-
-EF = {
-    'ICCID':  [MF_num, EF_num['ICCID']],
-    'ELP':    [MF_num, EF_num['ELP']],
-    'DIR':    [MF_num, EF_num['DIR']],
-
-    'ADN':    DF['TELECOM']+[EF_num['ADN']],
-    'FDN':    DF['TELECOM']+[EF_num['FDN']],
-    'SMS':    DF['TELECOM']+[EF_num['SMS']],
-    'CCP':    DF['TELECOM']+[EF_num['CCP']],
-    'MSISDN': DF['TELECOM']+[EF_num['MSISDN']],
-    'SMSP':   DF['TELECOM']+[EF_num['SMSP']],
-    'SMSS':   DF['TELECOM']+[EF_num['SMSS']],
-    'LND':    DF['TELECOM']+[EF_num['LND']],
-    'SMSR':   DF['TELECOM']+[EF_num['SMSR']],
-    'SDN':    DF['TELECOM']+[EF_num['SDN']],
-    'EXT1':   DF['TELECOM']+[EF_num['EXT1']],
-    'EXT2':   DF['TELECOM']+[EF_num['EXT2']],
-    'EXT3':   DF['TELECOM']+[EF_num['EXT3']],
-    'BDN':    DF['TELECOM']+[EF_num['BDN']],
-    'EXT4':   DF['TELECOM']+[EF_num['EXT4']],
-    'CMI':    DF['TELECOM']+[EF_num['CMI']],
-    'ECCP':   DF['TELECOM']+[EF_num['ECCP']],
-
-    'IMG':    DF['GRAPHICS']+[EF_num['IMG']],
-
-    'SAI':    DF['SoLSA']+[EF_num['SAI']],
-    'SLL':    DF['SoLSA']+[EF_num['SLL']],
-
-    'MExE-ST': DF['MExE']+[EF_num['MExE-ST']],
-    'ORPK':   DF['MExE']+[EF_num['ORPK']],
-    'ARPK':   DF['MExE']+[EF_num['ARPK']],
-    'TPRPK':  DF['MExE']+[EF_num['TPRPK']],
-
-    'LP':     DF['GSM']+[EF_num['LP']],
-    'IMSI':   DF['GSM']+[EF_num['IMSI']],
-    'Kc':     DF['GSM']+[EF_num['Kc']],
-    'DCK':    DF['GSM']+[EF_num['DCK']],
-    'PLMNsel': DF['GSM']+[EF_num['PLMNsel']],
-    'HPPLMN': DF['GSM']+[EF_num['HPPLMN']],
-    'CNL':    DF['GSM']+[EF_num['CNL']],
-    'ACMmax': DF['GSM']+[EF_num['ACMmax']],
-    'SST':    DF['GSM']+[EF_num['SST']],
-    'ACM':    DF['GSM']+[EF_num['ACM']],
-    'GID1':   DF['GSM']+[EF_num['GID1']],
-    'GID2':   DF['GSM']+[EF_num['GID2']],
-    'PUCT':   DF['GSM']+[EF_num['PUCT']],
-    'CBMI':   DF['GSM']+[EF_num['CBMI']],
-    'SPN':    DF['GSM']+[EF_num['SPN']],
-    'CBMID':  DF['GSM']+[EF_num['CBMID']],
-    'BCCH':   DF['GSM']+[EF_num['BCCH']],
-    'ACC':    DF['GSM']+[EF_num['ACC']],
-    'FPLMN':  DF['GSM']+[EF_num['FPLMN']],
-    'LOCI':   DF['GSM']+[EF_num['LOCI']],
-    'AD':     DF['GSM']+[EF_num['AD']],
-    'PHASE':  DF['GSM']+[EF_num['PHASE']],
-    'VGCS':   DF['GSM']+[EF_num['VGCS']],
-    'VGCSS':  DF['GSM']+[EF_num['VGCSS']],
-    'VBS':    DF['GSM']+[EF_num['VBS']],
-    'VBSS':   DF['GSM']+[EF_num['VBSS']],
-    'eMLPP':  DF['GSM']+[EF_num['eMLPP']],
-    'AAeM':   DF['GSM']+[EF_num['AAeM']],
-    'ECC':    DF['GSM']+[EF_num['ECC']],
-    'CBMIR':  DF['GSM']+[EF_num['CBMIR']],
-    'NIA':    DF['GSM']+[EF_num['NIA']],
-    'KcGPRS': DF['GSM']+[EF_num['KcGPRS']],
-    'LOCIGPRS': DF['GSM']+[EF_num['LOCIGPRS']],
-    'SUME':   DF['GSM']+[EF_num['SUME']],
-    'PLMNwAcT': DF['GSM']+[EF_num['PLMNwAcT']],
-    'OPLMNwAcT': DF['GSM']+[EF_num['OPLMNwAcT']],
-    # Figure 8 names it HPLMNAcT, but in the text it's names it HPLMNwAcT
-    'HPLMNAcT': DF['GSM']+[EF_num['HPLMNAcT']],
-    'HPLMNwAcT': DF['GSM']+[EF_num['HPLMNAcT']],
-    'CPBCCH': DF['GSM']+[EF_num['CPBCCH']],
-    'INVSCAN': DF['GSM']+[EF_num['INVSCAN']],
-    'PNN':    DF['GSM']+[EF_num['PNN']],
-    'OPL':    DF['GSM']+[EF_num['OPL']],
-    'MBDN':   DF['GSM']+[EF_num['MBDN']],
-    'EXT6':   DF['GSM']+[EF_num['EXT6']],
-    'MBI':    DF['GSM']+[EF_num['MBI']],
-    'MWIS':   DF['GSM']+[EF_num['MWIS']],
-    'CFIS':   DF['GSM']+[EF_num['CFIS']],
-    'EXT7':   DF['GSM']+[EF_num['EXT7']],
-    'SPDI':   DF['GSM']+[EF_num['SPDI']],
-    'MMSN':   DF['GSM']+[EF_num['MMSN']],
-    'EXT8':   DF['GSM']+[EF_num['EXT8']],
-    'MMSICP': DF['GSM']+[EF_num['MMSICP']],
-    'MMSUP':  DF['GSM']+[EF_num['MMSUP']],
-    'MMSUCP': DF['GSM']+[EF_num['MMSUCP']],
-}
 
 # Mapping between SIM Service Number and its description
 EF_SST_map = {
@@ -340,23 +111,54 @@ EF_SST_map = {
 # DF.TELECOM
 ######################################################################
 
+
+# TS 51.011 Section 10.5.1 / Table 12
+class ExtendedBcdAdapter(Adapter):
+    """Replace some hex-characters with other ASCII characters"""
+    # we only translate a=* / b=# as they habe a clear representation
+    # in terms of USSD / SS service codes
+    def _decode(self, obj, context, path):
+        if not isinstance(obj, str):
+            return obj
+        return obj.lower().replace("a","*").replace("b","#")
+
+    def _encode(self, obj, context, path):
+        if not isinstance(obj, str):
+            return obj
+        return obj.replace("*","a").replace("#","b")
+
 # TS 51.011 Section 10.5.1
 class EF_ADN(LinFixedEF):
-    def __init__(self, fid='6f3a', sfid=None, name='EF.ADN', desc='Abbreviated Dialing Numbers', **kwargs):
+    _test_decode = [
+            ( '42204841203120536963ffffffff06810628560810ffffffffffffff',
+              { "alpha_id": "B HA 1 Sic", "len_of_bcd": 6, "ton_npi": { "ext": True, "type_of_number":
+                                                                       "unknown", "numbering_plan_id":
+                                                                       "isdn_e164" }, "dialing_nr":
+               "6082658001", "cap_conf_id": 255, "ext1_record_id": 255 }),
+            ( '4B756E64656E626574726575756E67FFFFFF0791947112122721ffffffffffff',
+              {"alpha_id": "Kundenbetreuung", "len_of_bcd": 7, "ton_npi": {"ext": True, "type_of_number":
+                                                                           "international",
+                                                                           "numbering_plan_id": "isdn_e164"},
+               "dialing_nr": "491721217212", "cap_conf_id": 255, "ext1_record_id": 255} )
+        ]
+    _test_no_pad = True
+
+    def __init__(self, fid='6f3a', sfid=None, name='EF.ADN', desc='Abbreviated Dialing Numbers', ext=1, **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(14, 30), **kwargs)
-        self._construct = Struct('alpha_id'/COptional(GsmStringAdapter(Rpad(Bytes(this._.total_len-14)), codec='ascii')),
+        ext_name = 'ext%u_record_id' % ext
+        self._construct = Struct('alpha_id'/COptional(GsmOrUcs2Adapter(Rpad(Bytes(this._.total_len-14)))),
                                  'len_of_bcd'/Int8ub,
                                  'ton_npi'/TonNpi,
-                                 'dialing_nr'/BcdAdapter(Rpad(Bytes(10))),
+                                 'dialing_nr'/ExtendedBcdAdapter(BcdAdapter(Rpad(Bytes(10)))),
                                  'cap_conf_id'/Int8ub,
-                                 'ext1_record_id'/Int8ub)
+                                 ext_name/Int8ub)
 
 # TS 51.011 Section 10.5.5
 class EF_SMS(LinFixedEF):
     def __init__(self, fid='6f3c', sfid=None, name='EF.SMS', desc='Short messages', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(176, 176), **kwargs)
 
-    def _decode_record_bin(self, raw_bin_data):
+    def _decode_record_bin(self, raw_bin_data, **kwargs):
         def decode_status(status):
             if status & 0x01 == 0x00:
                 return (None, 'free_space')
@@ -385,23 +187,36 @@ class EF_SMS(LinFixedEF):
 # TS 51.011 Section 10.5.5
 class EF_MSISDN(LinFixedEF):
     def __init__(self, fid='6f40', sfid=None, name='EF.MSISDN', desc='MSISDN', **kwargs):
-        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(15, 34), **kwargs)
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(15, 34), leftpad=True, **kwargs)
 
-    def _decode_record_hex(self, raw_hex_data):
+    def _decode_record_hex(self, raw_hex_data, **kwargs):
         return {'msisdn': dec_msisdn(raw_hex_data)}
 
-    def _encode_record_hex(self, abstract):
+    def _encode_record_hex(self, abstract, **kwargs):
         msisdn = abstract['msisdn']
         if type(msisdn) == str:
             encoded_msisdn = enc_msisdn(msisdn)
         else:
             encoded_msisdn = enc_msisdn(msisdn[2], msisdn[0], msisdn[1])
-        alpha_identifier = (list(self.rec_len)[
-                            0] - len(encoded_msisdn) // 2) * "ff"
+        alpha_identifier = (list(self.rec_len)[0] - len(encoded_msisdn) // 2) * "ff"
         return alpha_identifier + encoded_msisdn
 
 # TS 51.011 Section 10.5.6
 class EF_SMSP(LinFixedEF):
+    # FIXME: re-encode fails / missing alpha_id at start of output
+    _test_decode = [
+        ( '454e6574776f726b73fffffffffffffff1ffffffffffffffffffffffffffffffffffffffffffffffff0000a7',
+          { "alpha_id": "ENetworks", "parameter_indicators": { "tp_dest_addr": False, "tp_sc_addr": True,
+                                                               "tp_pid": True, "tp_dcs": True, "tp_vp": True },
+            "tp_dest_addr": { "length": 255, "ton_npi": { "ext": True, "type_of_number": "reserved_for_extension",
+                                                          "numbering_plan_id": "reserved_for_extension" },
+                              "call_number": "" },
+            "tp_sc_addr": { "length": 255, "ton_npi": { "ext": True, "type_of_number": "reserved_for_extension",
+                                                        "numbering_plan_id": "reserved_for_extension" },
+                            "call_number": "" },
+            "tp_pid": "00", "tp_dcs": "00", "tp_vp_minutes": 1440 } ),
+    ]
+    _test_no_pad = True
     class ValidityPeriodAdapter(Adapter):
         def _decode(self, obj, context, path):
             if obj <= 143:
@@ -418,11 +233,11 @@ class EF_SMSP(LinFixedEF):
             if obj <= 12*60:
                 return obj/5 - 1
             elif obj <= 24*60:
-                return 143 + ((obj - (12 * 60)) / 30)
+                return 143 + ((obj - (12 * 60)) // 30)
             elif obj <= 30 * 24 * 60:
                 return 166 + (obj / (24 * 60))
             elif obj <= 63 * 7 * 24 * 60:
-                return 192 + (obj / (7 * 24 * 60))
+                return 192 + (obj // (7 * 24 * 60))
             else:
                 raise ValueError
 
@@ -481,7 +296,7 @@ class DF_TELECOM(CardDF):
         super().__init__(fid=fid, name=name, desc=desc, **kwargs)
         files = [
             EF_ADN(),
-            EF_ADN(fid='6f3b', name='EF.FDN', desc='Fixed dialling numbers'),
+            EF_ADN(fid='6f3b', name='EF.FDN', desc='Fixed dialling numbers', ext=2),
             EF_SMS(),
             LinFixedEF(fid='6f3d', name='EF.CCP',
                        desc='Capability Configuration Parameters', rec_len=(14, 14)),
@@ -490,7 +305,8 @@ class DF_TELECOM(CardDF):
             EF_MSISDN(),
             EF_SMSP(),
             EF_SMSS(),
-            # LND, SDN
+            EF_ADN('6f44', None, 'EF.LND', 'Last Number Dialled', ext=1),
+            EF_ADN('6f49', None, 'EF.SDN', 'Service Dialling Numbers', ext=3),
             EF_EXT('6f4a', None, 'EF.EXT1', 'Extension1 (ADN/SSC)'),
             EF_EXT('6f4b', None, 'EF.EXT2', 'Extension2 (FDN/SSC)'),
             EF_EXT('6f4c', None, 'EF.EXT3', 'Extension3 (SDN)'),
@@ -512,18 +328,25 @@ class DF_TELECOM(CardDF):
 
 # TS 51.011 Section 10.3.1
 class EF_LP(TransRecEF):
+    _test_de_encode = [
+            ( "24", "24"),
+        ]
     def __init__(self, fid='6f05', sfid=None, name='EF.LP', size=(1, None), rec_len=1,
                  desc='Language Preference'):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, rec_len=rec_len)
 
-    def _decode_record_bin(self, in_bin):
+    def _decode_record_bin(self, in_bin, **kwargs):
         return b2h(in_bin)
 
-    def _encode_record_bin(self, in_json):
+    def _encode_record_bin(self, in_json, **kwargs):
         return h2b(in_json)
 
 # TS 51.011 Section 10.3.2
 class EF_IMSI(TransparentEF):
+    _test_de_encode = [
+            ( "082982608200002080", { "imsi": "228062800000208" } ),
+            ( "082926101160845740", { "imsi": "262011106487504" } ),
+        ]
     def __init__(self, fid='6f07', sfid=None, name='EF.IMSI', desc='IMSI', size=(9, 9)):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size)
         # add those commands to the general commands of a TransparentEF
@@ -562,17 +385,21 @@ class EF_IMSI(TransparentEF):
 
 # TS 51.011 Section 10.3.4
 class EF_PLMNsel(TransRecEF):
+    _test_de_encode = [
+            ( "22F860",  { "mcc": "228", "mnc": "06" } ),
+            ( "330420",  { "mcc": "334", "mnc": "020" } ),
+        ]
     def __init__(self, fid='6f30', sfid=None, name='EF.PLMNsel', desc='PLMN selector',
                  size=(24, None), rec_len=3, **kwargs):
         super().__init__(fid, name=name, sfid=sfid, desc=desc, size=size, rec_len=rec_len, **kwargs)
 
-    def _decode_record_hex(self, in_hex):
+    def _decode_record_hex(self, in_hex, **kwargs):
         if in_hex[:6] == "ffffff":
             return None
         else:
             return dec_plmn(in_hex)
 
-    def _encode_record_hex(self, in_json):
+    def _encode_record_hex(self, in_json, **kwargs):
         if in_json == None:
             return "ffffff"
         else:
@@ -580,6 +407,9 @@ class EF_PLMNsel(TransRecEF):
 
 # TS 51.011 Section 10.3.6
 class EF_ACMmax(TransparentEF):
+    _test_de_encode = [
+            ( "000000", { "acm_max": 0 } ),
+        ]
     def __init__(self, fid='6f37', sfid=None, name='EF.ACMmax', size=(3, 3),
                  desc='ACM maximum value', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
@@ -590,6 +420,7 @@ class EF_ServiceTable(TransparentEF):
     def __init__(self, fid, sfid, name, desc, size, table):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size)
         self.table = table
+        self.shell_commands += [self.AddlShellCommands()]
 
     @staticmethod
     def _bit_byte_offset_for_service(service: int) -> Tuple[int, int]:
@@ -617,16 +448,14 @@ class EF_ServiceTable(TransparentEF):
         bin_len = 0
         for srv in in_json.keys():
             service_nr = int(srv)
-            (byte_offset, bit_offset) = EF_ServiceTable._bit_byte_offset_for_service(
-                service_nr)
+            (byte_offset, bit_offset) = self._bit_byte_offset_for_service(service_nr)
             if byte_offset >= bin_len:
                 bin_len = byte_offset+1
         # encode the actual data
         out = bytearray(b'\x00' * bin_len)
         for srv in in_json.keys():
             service_nr = int(srv)
-            (byte_offset, bit_offset) = EF_ServiceTable._bit_byte_offset_for_service(
-                service_nr)
+            (byte_offset, bit_offset) = self._bit_byte_offset_for_service(service_nr)
             bits = 0
             if in_json[srv]['allocated'] == True:
                 bits |= 1
@@ -635,8 +464,49 @@ class EF_ServiceTable(TransparentEF):
             out[byte_offset] |= ((bits & 3) << bit_offset)
         return out
 
+    @with_default_category('File-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        def _adjust_service(self, service_nr: int, allocate: Optional[bool] = None, activate : Optional[bool] = None):
+            (byte_offset, bit_offset) = EF_ServiceTable._bit_byte_offset_for_service(service_nr)
+            hex_data, sw = self._cmd.lchan.read_binary(length=1, offset=byte_offset)
+            data = h2b(hex_data)
+            if allocate is not None:
+                if allocate:
+                    data[0] |= (1 << bit_offset)
+                else:
+                    data[0] &= ~(1 << bit_offset)
+            if activate is not None:
+                if activate:
+                    data[0] |= (2 << bit_offset)
+                else:
+                    data[0] &= ~(2 << bit_offset)
+            total_data, sw = self._cmd.lchan.update_binary(b2h(data), offset=byte_offset)
+            return sw
+
+        def do_sst_service_allocate(self, arg):
+            """Allocate a service within EF.SST"""
+            self._adjust_service(int(arg), allocate = True)
+
+        def do_sst_service_deallocate(self, arg):
+            """Deallocate a service within EF.SST"""
+            self._adjust_service(int(arg), allocate = False)
+
+        def do_sst_service_activate(self, arg):
+            """Activate a service within EF.SST"""
+            self._adjust_service(int(arg), activate = True)
+
+        def do_sst_service_deactivate(self, arg):
+            """Deactivate a service within EF.SST"""
+            self._adjust_service(int(arg), activate = False)
+
+
 # TS 51.011 Section 10.3.11
 class EF_SPN(TransparentEF):
+    _test_de_encode = [
+            ( "0147534d2d52204348ffffffffffffffff",
+              { "rfu": 0, "hide_in_oplmn": False, "show_in_hplmn": True, "spn": "GSM-R CH" } ),
+        ]
+
     def __init__(self, fid='6f46', sfid=None, name='EF.SPN',
                  desc='Service Provider Name', size=(17, 17), **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
@@ -646,11 +516,12 @@ class EF_SPN(TransparentEF):
             'hide_in_oplmn'/Flag,
             'show_in_hplmn'/Flag,
             # Bytes 2..17
-            'spn'/Bytewise(GsmString(16))
+            'spn'/Bytewise(GsmOrUcs2String(16))
         )
 
 # TS 51.011 Section 10.3.13
 class EF_CBMI(TransRecEF):
+    # TODO: Test vectors
     def __init__(self, fid='6f45', sfid=None, name='EF.CBMI', size=(2, None), rec_len=2,
                  desc='Cell Broadcast message identifier selection', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, rec_len=rec_len, **kwargs)
@@ -658,13 +529,32 @@ class EF_CBMI(TransRecEF):
 
 # TS 51.011 Section 10.3.15
 class EF_ACC(TransparentEF):
+    # example: acc_list(15) will produce a dict with only ACC15 being True
+    # example: acc_list(2, 4) will produce a dict with ACC2 and ACC4 being True
+    # example: acc_list() will produce a dict with all ACCs being False
+    acc_list = lambda *active: {'ACC{}'.format(c) : c in active for c in range(16)}
+
+    _test_de_encode = [
+            ( "0000", acc_list() ),
+            ( "0001", acc_list(0) ),
+            ( "0002", acc_list(1) ),
+            ( "0100", acc_list(8) ),
+            ( "8000", acc_list(15) ),
+            ( "802b", acc_list(0,1,3,5,15) ),
+        ]
     def __init__(self, fid='6f78', sfid=None, name='EF.ACC',
                  desc='Access Control Class', size=(2, 2), **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
-        self._construct = HexAdapter(Bytes(2))
+        # LSB of octet 2 is ACC=0 ... MSB of octet 1 is ACC=15
+        flags = ['ACC{}'.format(c) / Flag for c in range(16)]
+        self._construct = ByteSwapped(BitsSwapped(BitStruct(*flags)))
 
 # TS 51.011 Section 10.3.16
 class EF_LOCI(TransparentEF):
+    _test_de_encode = [
+            ( "7802570222f81009780000",
+              { "tmsi": "78025702", "lai": "22f8100978", "tmsi_time": 0, "lu_status": "updated" } ),
+        ]
     def __init__(self, fid='6f7e', sfid=None, name='EF.LOCI', desc='Location Information', size=(11, 11)):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size)
         self._construct = Struct('tmsi'/HexAdapter(Bytes(4)), 'lai'/HexAdapter(Bytes(5)), 'tmsi_time'/Int8ub,
@@ -673,6 +563,12 @@ class EF_LOCI(TransparentEF):
 
 # TS 51.011 Section 10.3.18
 class EF_AD(TransparentEF):
+    _test_de_encode = [
+            ( "00ffff",
+              { "ms_operation_mode": "normal", "rfu1": 255, "rfu2": 127, "ofm": True, "extensions": None } ),
+        ]
+    _test_no_pad = True
+
     class OP_MODE(enum.IntEnum):
         normal = 0x00
         type_approval = 0x80
@@ -704,6 +600,9 @@ class EF_AD(TransparentEF):
 
 # TS 51.011 Section 10.3.20 / 10.3.22
 class EF_VGCS(TransRecEF):
+    _test_de_encode = [
+            ( "92f9ffff", "299fffff" ),
+        ]
     def __init__(self, fid='6fb1', sfid=None, name='EF.VGCS', size=(4, 200), rec_len=4,
                  desc='Voice Group Call Service', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, rec_len=rec_len, **kwargs)
@@ -711,14 +610,27 @@ class EF_VGCS(TransRecEF):
 
 # TS 51.011 Section 10.3.21 / 10.3.23
 class EF_VGCSS(TransparentEF):
+    _test_decode = [
+        ( "010000004540fc",
+          { "flags": [ 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+                       0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0 ] }
+        ),
+    ]
     def __init__(self, fid='6fb2', sfid=None, name='EF.VGCSS', size=(7, 7),
                  desc='Voice Group Call Service Status', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
-        self._construct = BitStruct(
-            'flags'/Bit[50], Padding(6, pattern=b'\xff'))
+        self._construct = BitsSwapped(BitStruct(
+            'flags'/Bit[50], Padding(6, pattern=b'\xff')))
 
 # TS 51.011 Section 10.3.24
 class EF_eMLPP(TransparentEF):
+    _test_de_encode = [
+        ( "7c04", { "levels": { "A": False, "B": False, "zero": True, "one": True,
+                                "two": True, "three": True, "four": True },
+                    "fast_call_setup_cond": { "A": False, "B": False, "zero": True, "one": False,
+                                              "two": False, "three": False, "four": False }
+                  }),
+    ]
     def __init__(self, fid='6fb5', sfid=None, name='EF.eMLPP', size=(2, 2),
                  desc='enhanced Multi Level Pre-emption and Priority', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
@@ -729,6 +641,10 @@ class EF_eMLPP(TransparentEF):
 
 # TS 51.011 Section 10.3.25
 class EF_AAeM(TransparentEF):
+    _test_de_encode = [
+        ( "3c", { "auto_answer_prio_levels": { "A": False, "B": False, "zero": True, "one": True,
+                                               "two": True, "three": True, "four": False } } ),
+    ]
     def __init__(self, fid='6fb6', sfid=None, name='EF.AAeM', size=(1, 1),
                  desc='Automatic Answer for eMLPP Service', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
@@ -772,7 +688,7 @@ class EF_CNL(TransRecEF):
                  desc='Co-operative Network List', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, rec_len=rec_len, **kwargs)
 
-    def _decode_record_hex(self, in_hex):
+    def _decode_record_hex(self, in_hex, **kwargs):
         (in_plmn, sub, svp, corp) = unpack('!3sBBB', h2b(in_hex))
         res = dec_plmn(b2h(in_plmn))
         res['network_subset'] = sub
@@ -780,7 +696,7 @@ class EF_CNL(TransRecEF):
         res['corporate_id'] = corp
         return res
 
-    def _encode_record_hex(self, in_json):
+    def _encode_record_hex(self, in_json, **kwargs):
         plmn = enc_plmn(in_json['mcc'], in_json['mnc'])
         return b2h(pack('!3sBBB',
                         h2b(plmn),
@@ -798,30 +714,43 @@ class EF_NIA(LinFixedEF):
 
 # TS 51.011 Section 10.3.32
 class EF_Kc(TransparentEF):
+    _test_de_encode = [
+        ( "837d783609a3858f05", { "kc": "837d783609a3858f", "cksn": 5 } ),
+    ]
     def __init__(self, fid='6f20', sfid=None, name='EF.Kc', desc='Ciphering key Kc', size=(9, 9), **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
         self._construct = Struct('kc'/HexAdapter(Bytes(8)), 'cksn'/Int8ub)
 
 # TS 51.011 Section 10.3.33
 class EF_LOCIGPRS(TransparentEF):
+    _test_de_encode = [
+        ( "ffffffffffffff22f8990000ff01",
+          { "ptmsi": "ffffffff", "ptmsi_sig": "ffffff", "rai": "22f8990000ff", "rau_status": "not_updated" } ),
+    ]
     def __init__(self, fid='6f53', sfid=None, name='EF.LOCIGPRS', desc='GPRS Location Information', size=(14, 14)):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size)
-        self._construct = Struct('ptmsi'/HexAdapter(Bytes(4)), 'ptmsi_sig'/Int8ub, 'rai'/HexAdapter(Bytes(6)),
+        self._construct = Struct('ptmsi'/HexAdapter(Bytes(4)), 'ptmsi_sig'/HexAdapter(Bytes(3)),
+                                 'rai'/HexAdapter(Bytes(6)),
                                  'rau_status'/Enum(Byte, updated=0, not_updated=1, plmn_not_allowed=2,
                                                    routing_area_not_allowed=3))
 
 # TS 51.011 Section 10.3.35..37
 class EF_xPLMNwAcT(TransRecEF):
-    def __init__(self, fid, sfid=None, name=None, desc=None, size=(40, None), rec_len=5, **kwargs):
+    _test_de_encode = [
+        ( '62F2104000', { "mcc": "262", "mnc": "01", "act": [ "E-UTRAN NB-S1", "E-UTRAN WB-S1" ] } ),
+        ( '62F2108000', { "mcc": "262", "mnc": "01", "act": [ "UTRAN" ] } ),
+        ( '62F220488C', { "mcc": "262", "mnc": "02", "act": ['E-UTRAN NB-S1', 'E-UTRAN WB-S1', 'EC-GSM-IoT', 'GSM', 'NG-RAN'] } ),
+    ]
+    def __init__(self, fid='1234', sfid=None, name=None, desc=None, size=(40, None), rec_len=5, **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, rec_len=rec_len, **kwargs)
 
-    def _decode_record_hex(self, in_hex):
+    def _decode_record_hex(self, in_hex, **kwargs):
         if in_hex[:6] == "ffffff":
             return None
         else:
             return dec_xplmn_w_act(in_hex)
 
-    def _encode_record_hex(self, in_json):
+    def _encode_record_hex(self, in_json, **kwargs):
         if in_json == None:
             return "ffffff0000"
         else:
@@ -844,18 +773,18 @@ class EF_xPLMNwAcT(TransRecEF):
         if 'cdma2000 1xRTT' in in_list:
             u16 |= 0x0010
         # E-UTRAN
-        if 'E-UTRAN' in in_list:
+        if 'E-UTRAN WB-S1' in in_list and 'E-UTRAN NB-S1' in in_list:
             u16 |= 0x4000
-        if 'E-UTRAN WB-S1' in in_list:
+        elif 'E-UTRAN WB-S1' in in_list:
             u16 |= 0x6000
-        if 'E-UTRAN NB-S1' in in_list:
+        elif 'E-UTRAN NB-S1' in in_list:
             u16 |= 0x5000
         # GSM mess
         if 'GSM' in in_list and 'EC-GSM-IoT' in in_list:
             u16 |= 0x008C
         elif 'GSM' in in_list:
             u16 |= 0x0084
-        elif 'EC-GSM-IuT' in in_list:
+        elif 'EC-GSM-IoT' in in_list:
             u16 |= 0x0088
         return '%04X' % (u16)
 
@@ -874,8 +803,31 @@ class EF_InvScan(TransparentEF):
         self._construct = FlagsEnum(
             Byte, in_limited_service_mode=1, after_successful_plmn_selection=2)
 
+# TS 51.011 Section 10.3.46
+class EF_CFIS(LinFixedEF):
+    _test_decode = [
+        ( '0100ffffffffffffffffffffffffffff',
+          {"msp_number": 1, "cfu_indicator_status": { "voice": False, "fax": False, "data": False, "rfu": 0 },
+                                                      "len_of_bcd": 255, "ton_npi": {"ext": True,
+                                                                                     "type_of_number": "reserved_for_extension",
+                                                                                     "numbering_plan_id": "reserved_for_extension"},
+                                                      "dialing_nr": "", "cap_conf_id": 255, "ext7_record_id": 255} ),
+    ]
+    def __init__(self, fid='6fcb', sfid=None, name='EF.CFIS', desc='Call Forwarding Indication Status', ext=7, **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(16, 30), **kwargs)
+        ext_name = 'ext%u_record_id' % ext
+        self._construct = Struct('msp_number'/Int8ub,
+                                 'cfu_indicator_status'/BitStruct('voice'/Flag, 'fax'/Flag, 'data'/Flag, 'rfu'/BitsRFU(5)),
+                                 'len_of_bcd'/Int8ub,
+                                 'ton_npi'/TonNpi,
+                                 'dialing_nr'/ExtendedBcdAdapter(BcdAdapter(Rpad(Bytes(10)))),
+                                 'cap_conf_id'/Int8ub,
+                                 ext_name/Int8ub)
+
 # TS 51.011 Section 4.2.58
 class EF_PNN(LinFixedEF):
+    # TODO: 430a82d432bbbc7eb75de432450a82d432bbbc7eb75de432ffffffff
+    # TODO: 430a82c596b34cbfbfe5eb39ffffffffffffffffffffffffffffffffffff
     class FullNameForNetwork(BER_TLV_IE, tag=0x43):
         # TS 24.008 10.5.3.5a
         # TODO: proper decode
@@ -895,13 +847,21 @@ class EF_PNN(LinFixedEF):
 
 # TS 51.011 Section 10.3.42
 class EF_OPL(LinFixedEF):
+    _test_de_encode = [
+        ( '62f2100000fffe01',
+          { "lai": { "mcc_mnc": "262-01", "lac_min": "0000", "lac_max": "fffe" }, "pnn_record_id": 1 } ),
+    ]
     def __init__(self, fid='6fc6', sfid=None, name='EF.OPL', rec_len=(8, 8), desc='Operator PLMN List', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=rec_len, **kwargs)
-        self._construct = Struct('lai'/Struct('mcc_mnc'/BcdAdapter(Bytes(3)),
+        self._construct = Struct('lai'/Struct('mcc_mnc'/PlmnAdapter(Bytes(3)),
                                  'lac_min'/HexAdapter(Bytes(2)), 'lac_max'/HexAdapter(Bytes(2))), 'pnn_record_id'/Int8ub)
 
 # TS 51.011 Section 10.3.44 + TS 31.102 4.2.62
 class EF_MBI(LinFixedEF):
+    _test_de_encode = [
+        ( '0100000000',
+          { "mbi_voicemail": 1, "mbi_fax": 0, "mbi_email": 0, "mbi_other": 0, "mbi_videocall": 0 } ),
+    ]
     def __init__(self, fid='6fc9', sfid=None, name='EF.MBI', rec_len=(4, 5), desc='Mailbox Identifier', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=rec_len, **kwargs)
         self._construct = Struct('mbi_voicemail'/Int8ub, 'mbi_fax'/Int8ub, 'mbi_email'/Int8ub,
@@ -909,6 +869,14 @@ class EF_MBI(LinFixedEF):
 
 # TS 51.011 Section 10.3.45 + TS 31.102 4.2.63
 class EF_MWIS(LinFixedEF):
+    _test_de_encode = [
+        ( '0000000000',
+          {"mwi_status": {"voicemail": False, "fax": False, "email": False, "other": False, "videomail":
+                          False}, "num_waiting_voicemail": 0, "num_waiting_fax": 0, "num_waiting_email": 0,
+           "num_waiting_other": 0, "num_waiting_videomail": None} ),
+    ]
+    _test_no_pad = True
+
     def __init__(self, fid='6fca', sfid=None, name='EF.MWIS', rec_len=(5, 6),
                  desc='Message Waiting Indication Status', **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=rec_len, **kwargs)
@@ -919,9 +887,10 @@ class EF_MWIS(LinFixedEF):
 
 # TS 51.011 Section 10.3.66
 class EF_SPDI(TransparentEF):
+    # TODO: a305800337f800ffffffffffffffffffffffffffffffffffffffffffffff
     class ServiceProviderPLMN(BER_TLV_IE, tag=0x80):
         # flexible numbers of 3-byte PLMN records
-        _construct = GreedyRange(BcdAdapter(Bytes(3)))
+        _construct = GreedyRange(PlmnAdapter(Bytes(3)))
 
     class SPDI(BER_TLV_IE, tag=0xA3, nested=[ServiceProviderPLMN]):
         pass
@@ -966,7 +935,7 @@ class EF_MMSICP(TransparentEF):
 # TS 51.011 Section 10.3.54
 class EF_MMSUP(LinFixedEF):
     class MMS_UserPref_ProfileName(BER_TLV_IE, tag=0x81):
-        pass
+        _construct = GsmOrUcs2Adapter(GreedyBytes)
 
     class MMS_UserPref_Info(BER_TLV_IE, tag=0x82):
         pass
@@ -989,38 +958,40 @@ class EF_MMSUCP(TransparentEF):
 class DF_GSM(CardDF):
     def __init__(self, fid='7f20', name='DF.GSM', desc='GSM Network related files'):
         super().__init__(fid=fid, name=name, desc=desc)
+        self.shell_commands += [self.AddlShellCommands()]
+
         files = [
             EF_LP(),
             EF_IMSI(),
             EF_Kc(),
             EF_PLMNsel(),
             TransparentEF('6f31', None, 'EF.HPPLMN',
-                          'Higher Priority PLMN search period'),
+                          desc='Higher Priority PLMN search period'),
             EF_ACMmax(),
             EF_ServiceTable('6f38', None, 'EF.SST',
-                            'SIM service table', table=EF_SST_map, size=(2, 16)),
+                            desc='SIM service table', table=EF_SST_map, size=(2, 16)),
             CyclicEF('6f39', None, 'EF.ACM',
-                     'Accumulated call meter', rec_len=(3, 3)),
-            TransparentEF('6f3e', None, 'EF.GID1', 'Group Identifier Level 1'),
-            TransparentEF('6f3f', None, 'EF.GID2', 'Group Identifier Level 2'),
+                     desc='Accumulated call meter', rec_len=(3, 3)),
+            TransparentEF('6f3e', None, 'EF.GID1', desc='Group Identifier Level 1'),
+            TransparentEF('6f3f', None, 'EF.GID2', desc='Group Identifier Level 2'),
             EF_SPN(),
             TransparentEF('6f41', None, 'EF.PUCT',
-                          'Price per unit and currency table', size=(5, 5)),
+                          desc='Price per unit and currency table', size=(5, 5)),
             EF_CBMI(),
-            TransparentEF('6f7f', None, 'EF.BCCH',
-                          'Broadcast control channels', size=(16, 16)),
+            TransparentEF('6f74', None, 'EF.BCCH',
+                          desc='Broadcast control channels', size=(16, 16)),
             EF_ACC(),
             EF_PLMNsel('6f7b', None, 'EF.FPLMN',
-                       'Forbidden PLMNs', size=(12, 12)),
+                       desc='Forbidden PLMNs', size=(12, 12)),
             EF_LOCI(),
             EF_AD(),
-            TransparentEF('6fa3', None, 'EF.Phase',
-                          'Phase identification', size=(1, 1)),
+            TransparentEF('6fae', None, 'EF.Phase',
+                          desc='Phase identification', size=(1, 1)),
             EF_VGCS(),
             EF_VGCSS(),
-            EF_VGCS('6fb3', None, 'EF.VBS', 'Voice Broadcast Service'),
+            EF_VGCS('6fb3', None, 'EF.VBS', desc='Voice Broadcast Service'),
             EF_VGCSS('6fb4', None, 'EF.VBSS',
-                     'Voice Broadcast Service Status'),
+                     desc='Voice Broadcast Service Status'),
             EF_eMLPP(),
             EF_AAeM(),
             EF_CBMID(),
@@ -1029,39 +1000,52 @@ class DF_GSM(CardDF):
             EF_DCK(),
             EF_CNL(),
             EF_NIA(),
-            EF_Kc('6f52', None, 'EF.KcGPRS', 'GPRS Ciphering key KcGPRS'),
+            EF_Kc('6f52', None, 'EF.KcGPRS', desc='GPRS Ciphering key KcGPRS'),
             EF_LOCIGPRS(),
-            TransparentEF('6f54', None, 'EF.SUME', 'SetUpMenu Elements'),
+            TransparentEF('6f54', None, 'EF.SUME', desc='SetUpMenu Elements'),
             EF_xPLMNwAcT('6f60', None, 'EF.PLMNwAcT',
-                         'User controlled PLMN Selector with Access Technology'),
+                         desc='User controlled PLMN Selector with Access Technology'),
             EF_xPLMNwAcT('6f61', None, 'EF.OPLMNwAcT',
-                         'Operator controlled PLMN Selector with Access Technology'),
+                         desc='Operator controlled PLMN Selector with Access Technology'),
             EF_xPLMNwAcT('6f62', None, 'EF.HPLMNwAcT',
-                         'HPLMN Selector with Access Technology'),
+                         desc='HPLMN Selector with Access Technology'),
             EF_CPBCCH(),
             EF_InvScan(),
             EF_PNN(),
             EF_OPL(),
-            EF_ADN('6fc7', None, 'EF.MBDN', 'Mailbox Dialling Numbers'),
+            EF_ADN('6fc7', None, 'EF.MBDN', desc='Mailbox Dialling Numbers'),
             EF_MBI(),
             EF_MWIS(),
-            EF_ADN('6fcb', None, 'EF.CFIS',
-                   'Call Forwarding Indication Status'),
-            EF_EXT('6fc8', None, 'EF.EXT6', 'Externsion6 (MBDN)'),
-            EF_EXT('6fcc', None, 'EF.EXT7', 'Externsion7 (CFIS)'),
+            EF_CFIS(),
+            EF_EXT('6fc8', None, 'EF.EXT6', desc='Externsion6 (MBDN)'),
+            EF_EXT('6fcc', None, 'EF.EXT7', desc='Externsion7 (CFIS)'),
             EF_SPDI(),
             EF_MMSN(),
-            EF_EXT('6fcf', None, 'EF.EXT8', 'Extension8 (MMSN)'),
+            EF_EXT('6fcf', None, 'EF.EXT8', desc='Extension8 (MMSN)'),
             EF_MMSICP(),
             EF_MMSUP(),
             EF_MMSUCP(),
         ]
         self.add_files(files)
 
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        def __init__(self):
+            super().__init__()
+
+        authenticate_parser = argparse.ArgumentParser()
+        authenticate_parser.add_argument('rand', help='Random challenge')
+
+        @cmd2.with_argparser(authenticate_parser)
+        def do_authenticate(self, opts):
+            """Perform GSM Authentication."""
+            (data, sw) = self._cmd.lchan.scc.run_gsm(opts.rand)
+            self._cmd.poutput_json(data)
+
 
 class CardProfileSIM(CardProfile):
 
-    ORDER = 2
+    ORDER = 3
 
     def __init__(self):
         sw = {
@@ -1101,11 +1085,18 @@ class CardProfileSIM(CardProfile):
             },
         }
 
+        addons = [
+            AddonGSMR,
+        ]
+
         super().__init__('SIM', desc='GSM SIM Card', cla="a0",
-                         sel_ctrl="0000", files_in_mf=[DF_TELECOM(), DF_GSM()], sw=sw)
+                         sel_ctrl="0000", files_in_mf=[DF_TELECOM(), DF_GSM()], sw=sw, addons = addons)
 
     @staticmethod
     def decode_select_response(resp_hex: str) -> object:
+        # we try to build something that resembles a dict resulting from the TLV decoder
+        # of TS 102.221 (FcpTemplate), so that higher-level code only has to deal with one
+        # format of SELECT response
         resp_bin = h2b(resp_hex)
         struct_of_file_map = {
             0: 'transparent',
@@ -1124,21 +1115,25 @@ class CardProfileSIM(CardProfile):
             'proprietary_info': {},
         }
         ret['file_id'] = b2h(resp_bin[4:6])
-        ret['proprietary_info']['available_memory'] = int.from_bytes(
-            resp_bin[2:4], 'big')
         file_type = type_of_file_map[resp_bin[6]
                                      ] if resp_bin[6] in type_of_file_map else resp_bin[6]
         ret['file_descriptor']['file_descriptor_byte']['file_type'] = file_type
         if file_type in ['mf', 'df']:
+            ret['proprietary_info']['available_memory'] = int.from_bytes(resp_bin[2:4], 'big')
             ret['file_characteristics'] = b2h(resp_bin[13:14])
             ret['num_direct_child_df'] = resp_bin[14]
             ret['num_direct_child_ef'] = resp_bin[15]
             ret['num_chv_unblock_adm_codes'] = int(resp_bin[16])
             # CHV / UNBLOCK CHV stats
         elif file_type in ['working_ef']:
+            ret['file_size'] = int.from_bytes(resp_bin[2:4], 'big')
             file_struct = struct_of_file_map[resp_bin[13]
                                              ] if resp_bin[13] in struct_of_file_map else resp_bin[13]
             ret['file_descriptor']['file_descriptor_byte']['structure'] = file_struct
+            if file_struct != 'transparent':
+                record_len = resp_bin[14]
+                ret['file_descriptor']['record_len'] = record_len
+                ret['file_descriptor']['num_of_rec'] = ret['file_size'] // record_len
             ret['access_conditions'] = b2h(resp_bin[8:10])
             if resp_bin[11] & 0x01 == 0:
                 ret['life_cycle_status_int'] = 'operational_activated'
@@ -1151,3 +1146,17 @@ class CardProfileSIM(CardProfile):
     @staticmethod
     def match_with_card(scc: SimCardCommands) -> bool:
         return match_sim(scc)
+
+
+class AddonSIM(CardProfileAddon):
+    """An add-on that can be found on a UICC in order to support classic GSM SIM."""
+    def __init__(self):
+        files = [
+            DF_GSM(),
+            DF_TELECOM(),
+        ]
+        super().__init__('SIM', desc='GSM SIM', files_in_mf=files)
+
+    def probe(self, card:'CardBase') -> bool:
+        # we assume the add-on to be present in case DF.GSM is found on the card
+        return card.file_exists(self.files_in_mf[0].fid)

pySim/utils.py

@@ -7,7 +7,7 @@ import json
 import abc
 import string
 from io import BytesIO
-from typing import Optional, List, Dict, Any, Tuple
+from typing import Optional, List, Dict, Any, Tuple, NewType
 
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
 # Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
@@ -27,8 +27,10 @@ from typing import Optional, List, Dict, Any, Tuple
 #
 
 # just to differentiate strings of hex nibbles from everything else
-Hexstr = str
-
+Hexstr = NewType('Hexstr', str)
+SwHexstr = NewType('SwHexstr', str)
+SwMatchstr = NewType('SwMatchstr', str)
+ResTuple = Tuple[Hexstr, SwHexstr]
 
 def h2b(s: Hexstr) -> bytearray:
     """convert from a string of hex nibbles to a sequence of bytes"""
@@ -260,15 +262,22 @@ def bertlv_encode_tag(t) -> bytes:
         remainder = inp & ~ (inp << (remain_bits - bitcnt))
         return outp, remainder
 
+    def count_int_bytes(inp: int) -> int:
+        """count the number of bytes require to represent the given integer."""
+        i = 1
+        inp = inp >> 8
+        while inp:
+            i += 1
+            inp = inp >> 8
+        return i
+
     if isinstance(t, int):
-        # FIXME: multiple byte tags
-        tag = t & 0x1f
-        constructed = True if t & 0x20 else False
-        cls = t >> 6
-    else:
-        tag = t['tag']
-        constructed = t['constructed']
-        cls = t['class']
+        # first convert to a dict representation
+        tag_size = count_int_bytes(t)
+        t, remainder = bertlv_parse_tag(t.to_bytes(tag_size, 'big'))
+    tag = t['tag']
+    constructed = t['constructed']
+    cls = t['class']
     if tag <= 30:
         t = tag & 0x1f
         if constructed:
@@ -305,6 +314,8 @@ def bertlv_parse_len(binary: bytes) -> Tuple[int, bytes]:
     else:
         num_len_oct = binary[0] & 0x7f
         length = 0
+        if len(binary) < num_len_oct + 1:
+            return (0, b'')
         for i in range(1, 1+num_len_oct):
             length <<= 8
             length |= binary[i]
@@ -411,7 +422,7 @@ def enc_plmn(mcc: Hexstr, mnc: Hexstr) -> Hexstr:
     if len(mnc) == 0:
         mnc = "FFF"
     elif len(mnc) == 1:
-        mnc = "F0" + mnc
+        mnc = "0" + mnc + "F"
     elif len(mnc) == 2:
         mnc += "F"
 
@@ -433,30 +444,6 @@ def dec_plmn(threehexbytes: Hexstr) -> dict:
     return res
 
 
-def dec_spn(ef):
-    """Obsolete, kept for API compatibility"""
-    from ts_51_011 import EF_SPN
-    abstract_data = EF_SPN().decode_hex(ef)
-    show_in_hplmn = abstract_data['show_in_hplmn']
-    hide_in_oplmn = abstract_data['hide_in_oplmn']
-    name = abstract_data['spn']
-    return (name, show_in_hplmn, hide_in_oplmn)
-
-
-def enc_spn(name: str, show_in_hplmn=False, hide_in_oplmn=False):
-    """Obsolete, kept for API compatibility"""
-    from ts_51_011 import EF_SPN
-    abstract_data = {
-        'hide_in_oplmn': hide_in_oplmn,
-        'show_in_hplmn': show_in_hplmn,
-        'spn': name,
-    }
-    return EF_SPN().encode_hex(abstract_data)
-
-
-def hexstr_to_Nbytearr(s, nbytes):
-    return [s[i:i+(nbytes*2)] for i in range(0, len(s), (nbytes*2))]
-
 # Accepts hex string representing three bytes
 
 
@@ -499,33 +486,37 @@ def dec_mnc_from_plmn_str(plmn: Hexstr) -> str:
 def dec_act(twohexbytes: Hexstr) -> List[str]:
     act_list = [
         {'bit': 15, 'name': "UTRAN"},
-        {'bit': 14, 'name': "E-UTRAN"},
         {'bit': 11, 'name': "NG-RAN"},
-        {'bit':  7, 'name': "GSM"},
         {'bit':  6, 'name': "GSM COMPACT"},
         {'bit':  5, 'name': "cdma2000 HRPD"},
         {'bit':  4, 'name': "cdma2000 1xRTT"},
     ]
     ia = h2i(twohexbytes)
     u16t = (ia[0] << 8) | ia[1]
-    sel = []
+    sel = set()
+    # only the simple single-bit ones
     for a in act_list:
         if u16t & (1 << a['bit']):
-            if a['name'] == "E-UTRAN":
-                # The Access technology identifier of E-UTRAN
-                # allows a more detailed specification:
-                if u16t & (1 << 13) and u16t & (1 << 12):
-                    sel.append("E-UTRAN WB-S1")
-                    sel.append("E-UTRAN NB-S1")
-                elif u16t & (1 << 13):
-                    sel.append("E-UTRAN WB-S1")
-                elif u16t & (1 << 12):
-                    sel.append("E-UTRAN NB-S1")
-                else:
-                    sel.append("E-UTRAN")
-            else:
-                sel.append(a['name'])
-    return sel
+            sel.add(a['name'])
+    # TS 31.102 Section 4.2.5 Table 4.2.5.1
+    eutran_bits = u16t & 0x7000
+    if eutran_bits == 0x4000 or eutran_bits == 0x7000:
+        sel.add("E-UTRAN WB-S1")
+        sel.add("E-UTRAN NB-S1")
+    elif eutran_bits == 0x5000:
+        sel.add("E-UTRAN NB-S1")
+    elif eutran_bits == 0x6000:
+        sel.add("E-UTRAN WB-S1")
+    # TS 31.102 Section 4.2.5 Table 4.2.5.2
+    gsm_bits = u16t & 0x008C
+    if gsm_bits == 0x0080 or gsm_bits == 0x008C:
+        sel.add("GSM")
+        sel.add("EC-GSM-IoT")
+    elif u16t & 0x008C == 0x0084:
+        sel.add("GSM")
+    elif u16t & 0x008C == 0x0086:
+        sel.add("EC-GSM-IoT")
+    return sorted(list(sel))
 
 
 def dec_xplmn_w_act(fivehexbytes: Hexstr) -> Dict[str, Any]:
@@ -542,84 +533,23 @@ def dec_xplmn_w_act(fivehexbytes: Hexstr) -> Dict[str, Any]:
     return res
 
 
-def format_xplmn_w_act(hexstr):
-    s = ""
-    for rec_data in hexstr_to_Nbytearr(hexstr, 5):
-        rec_info = dec_xplmn_w_act(rec_data)
-        if rec_info['mcc'] == "" and rec_info['mnc'] == "":
-            rec_str = "unused"
-        else:
-            rec_str = "MCC: %s MNC: %s AcT: %s" % (
-                rec_info['mcc'], rec_info['mnc'], ", ".join(rec_info['act']))
-        s += "\t%s # %s\n" % (rec_data, rec_str)
-    return s
-
-
-def dec_loci(hexstr):
-    res = {'tmsi': '',  'mcc': 0, 'mnc': 0, 'lac': '', 'status': 0}
-    res['tmsi'] = hexstr[:8]
-    res['mcc'] = dec_mcc_from_plmn(hexstr[8:14])
-    res['mnc'] = dec_mnc_from_plmn(hexstr[8:14])
-    res['lac'] = hexstr[14:18]
-    res['status'] = h2i(hexstr[20:22])
-    return res
-
-
-def dec_psloci(hexstr):
-    res = {'p-tmsi': '', 'p-tmsi-sig': '', 'mcc': 0,
-           'mnc': 0, 'lac': '', 'rac': '', 'status': 0}
-    res['p-tmsi'] = hexstr[:8]
-    res['p-tmsi-sig'] = hexstr[8:14]
-    res['mcc'] = dec_mcc_from_plmn(hexstr[14:20])
-    res['mnc'] = dec_mnc_from_plmn(hexstr[14:20])
-    res['lac'] = hexstr[20:24]
-    res['rac'] = hexstr[24:26]
-    res['status'] = h2i(hexstr[26:28])
-    return res
-
-
-def dec_epsloci(hexstr):
-    res = {'guti': '', 'mcc': 0, 'mnc': 0, 'tac': '', 'status': 0}
-    res['guti'] = hexstr[:24]
-    res['tai'] = hexstr[24:34]
-    res['mcc'] = dec_mcc_from_plmn(hexstr[24:30])
-    res['mnc'] = dec_mnc_from_plmn(hexstr[24:30])
-    res['tac'] = hexstr[30:34]
-    res['status'] = h2i(hexstr[34:36])
-    return res
-
-
 def dec_xplmn(threehexbytes: Hexstr) -> dict:
     res = {'mcc': 0, 'mnc': 0, 'act': []}
     plmn_chars = 6
     # first three bytes (six ascii hex chars)
     plmn_str = threehexbytes[:plmn_chars]
-    res['mcc'] = dec_mcc_from_plmn(plmn_str)
-    res['mnc'] = dec_mnc_from_plmn(plmn_str)
+    res['mcc'] = dec_mcc_from_plmn_str(plmn_str)
+    res['mnc'] = dec_mnc_from_plmn_str(plmn_str)
     return res
 
 
-def format_xplmn(hexstr: Hexstr) -> str:
-    s = ""
-    for rec_data in hexstr_to_Nbytearr(hexstr, 3):
-        rec_info = dec_xplmn(rec_data)
-        if rec_info['mcc'] == 0xFFF and rec_info['mnc'] == 0xFFF:
-            rec_str = "unused"
-        else:
-            rec_str = "MCC: %03d MNC: %03d" % (
-                rec_info['mcc'], rec_info['mnc'])
-        s += "\t%s # %s\n" % (rec_data, rec_str)
-    return s
-
-
 def derive_milenage_opc(ki_hex: Hexstr, op_hex: Hexstr) -> Hexstr:
     """
     Run the milenage algorithm to calculate OPC from Ki and OP
     """
-    from Crypto.Cipher import AES
+    from Cryptodome.Cipher import AES
     # pylint: disable=no-name-in-module
-    from Crypto.Util.strxor import strxor
-    from pySim.utils import b2h
+    from Cryptodome.Util.strxor import strxor
 
     # We pass in hex string and now need to work on bytes
     ki_bytes = bytes(h2b(ki_hex))
@@ -786,202 +716,6 @@ def enc_msisdn(msisdn: str, npi: int = 0x01, ton: int = 0x03) -> Hexstr:
     return ('%02x' % bcd_len) + ('%02x' % npi_ton) + bcd + ("ff" * 2)
 
 
-def dec_st(st, table="sim") -> str:
-    """
-    Parses the EF S/U/IST and prints the list of available services in EF S/U/IST
-    """
-
-    if table == "isim":
-        from pySim.ts_31_103 import EF_IST_map
-        lookup_map = EF_IST_map
-    elif table == "usim":
-        from pySim.ts_31_102 import EF_UST_map
-        lookup_map = EF_UST_map
-    else:
-        from pySim.ts_51_011 import EF_SST_map
-        lookup_map = EF_SST_map
-
-    st_bytes = [st[i:i+2] for i in range(0, len(st), 2)]
-
-    avail_st = ""
-    # Get each byte and check for available services
-    for i in range(0, len(st_bytes)):
-        # Byte i contains info about Services num (8i+1) to num (8i+8)
-        byte = int(st_bytes[i], 16)
-        # Services in each byte are in order MSB to LSB
-        # MSB - Service (8i+8)
-        # LSB - Service (8i+1)
-        for j in range(1, 9):
-            if byte & 0x01 == 0x01 and ((8*i) + j in lookup_map):
-                # Byte X contains info about Services num (8X-7) to num (8X)
-                # bit = 1: service available
-                # bit = 0: service not available
-                avail_st += '\tService %d - %s\n' % (
-                    (8*i) + j, lookup_map[(8*i) + j])
-            byte = byte >> 1
-    return avail_st
-
-
-def first_TLV_parser(bytelist):
-    '''
-    first_TLV_parser([0xAA, 0x02, 0xAB, 0xCD, 0xFF, 0x00]) -> (170, 2, [171, 205])
-
-    parses first TLV format record in a list of bytelist
-    returns a 3-Tuple: Tag, Length, Value
-    Value is a list of bytes
-    parsing of length is ETSI'style 101.220
-    '''
-    Tag = bytelist[0]
-    if bytelist[1] == 0xFF:
-        Len = bytelist[2]*256 + bytelist[3]
-        Val = bytelist[4:4+Len]
-    else:
-        Len = bytelist[1]
-        Val = bytelist[2:2+Len]
-    return (Tag, Len, Val)
-
-
-def TLV_parser(bytelist):
-    '''
-    TLV_parser([0xAA, ..., 0xFF]) -> [(T, L, [V]), (T, L, [V]), ...]
-
-    loops on the input list of bytes with the "first_TLV_parser()" function
-    returns a list of 3-Tuples
-    '''
-    ret = []
-    while len(bytelist) > 0:
-        T, L, V = first_TLV_parser(bytelist)
-        if T == 0xFF:
-            # padding bytes
-            break
-        ret.append((T, L, V))
-        # need to manage length of L
-        if L > 0xFE:
-            bytelist = bytelist[L+4:]
-        else:
-            bytelist = bytelist[L+2:]
-    return ret
-
-
-def enc_st(st, service, state=1):
-    """
-    Encodes the EF S/U/IST/EST and returns the updated Service Table
-
-    Parameters:
-            st - Current value of SIM/USIM/ISIM Service Table
-            service - Service Number to encode as activated/de-activated
-            state - 1 mean activate, 0 means de-activate
-
-    Returns:
-            s - Modified value of SIM/USIM/ISIM Service Table
-
-    Default values:
-            - state: 1 - Sets the particular Service bit to 1
-    """
-    st_bytes = [st[i:i+2] for i in range(0, len(st), 2)]
-
-    s = ""
-    # Check whether the requested service is present in each byte
-    for i in range(0, len(st_bytes)):
-        # Byte i contains info about Services num (8i+1) to num (8i+8)
-        if service in range((8*i) + 1, (8*i) + 9):
-            byte = int(st_bytes[i], 16)
-            # Services in each byte are in order MSB to LSB
-            # MSB - Service (8i+8)
-            # LSB - Service (8i+1)
-            mod_byte = 0x00
-            # Copy bit by bit contents of byte to mod_byte with modified bit
-            # for requested service
-            for j in range(1, 9):
-                mod_byte = mod_byte >> 1
-                if service == (8*i) + j:
-                    mod_byte = state == 1 and mod_byte | 0x80 or mod_byte & 0x7f
-                else:
-                    mod_byte = byte & 0x01 == 0x01 and mod_byte | 0x80 or mod_byte & 0x7f
-                byte = byte >> 1
-
-            s += ('%02x' % (mod_byte))
-        else:
-            s += st_bytes[i]
-
-    return s
-
-
-def dec_addr_tlv(hexstr):
-    """
-    Decode hex string to get EF.P-CSCF Address or EF.ePDGId or EF.ePDGIdEm.
-    See 3GPP TS 31.102 version 13.4.0 Release 13, section 4.2.8, 4.2.102 and 4.2.104.
-    """
-
-    # Convert from hex str to int bytes list
-    addr_tlv_bytes = h2i(hexstr)
-
-    # Get list of tuples containing parsed TLVs
-    tlvs = TLV_parser(addr_tlv_bytes)
-
-    for tlv in tlvs:
-        # tlv = (T, L, [V])
-        # T = Tag
-        # L = Length
-        # [V] = List of value
-
-        # Invalid Tag value scenario
-        if tlv[0] != 0x80:
-            continue
-
-        # Empty field - Zero length
-        if tlv[1] == 0:
-            continue
-
-        # First byte in the value has the address type
-        addr_type = tlv[2][0]
-        # TODO: Support parsing of IPv6
-        # Address Type: 0x00 (FQDN), 0x01 (IPv4), 0x02 (IPv6), other (Reserved)
-        if addr_type == 0x00:  # FQDN
-            # Skip address tye byte i.e. first byte in value list
-            content = tlv[2][1:]
-            return (i2s(content), '00')
-
-        elif addr_type == 0x01:  # IPv4
-            # Skip address tye byte i.e. first byte in value list
-            # Skip the unused byte in Octect 4 after address type byte as per 3GPP TS 31.102
-            ipv4 = tlv[2][2:]
-            content = '.'.join(str(x) for x in ipv4)
-            return (content, '01')
-        else:
-            raise ValueError("Invalid address type")
-
-    return (None, None)
-
-
-def enc_addr_tlv(addr, addr_type='00'):
-    """
-    Encode address TLV object used in EF.P-CSCF Address, EF.ePDGId and EF.ePDGIdEm.
-    See 3GPP TS 31.102 version 13.4.0 Release 13, section 4.2.8, 4.2.102 and 4.2.104.
-
-    Default values:
-      - addr_type: 00 - FQDN format of Address
-    """
-
-    s = ""
-
-    # TODO: Encoding of IPv6 address
-    if addr_type == '00':  # FQDN
-        hex_str = s2h(addr)
-        s += '80' + ('%02x' % ((len(hex_str)//2)+1)) + '00' + hex_str
-    elif addr_type == '01':  # IPv4
-        ipv4_list = addr.split('.')
-        ipv4_str = ""
-        for i in ipv4_list:
-            ipv4_str += ('%02x' % (int(i)))
-
-        # Unused bytes shall be set to 'ff'. i.e 4th Octet after Address Type is not used
-        # IPv4 Address is in octet 5 to octet 8 of the TLV data object
-        s += '80' + ('%02x' % ((len(ipv4_str)//2)+2)) + '01' + 'ff' + ipv4_str
-
-    return s
-
-
 def is_hex(string: str, minlen: int = 2, maxlen: Optional[int] = None) -> bool:
     """
     Check if a string is a valid hexstring
@@ -1037,88 +771,6 @@ def sanitize_pin_adm(pin_adm, pin_adm_hex=None) -> Hexstr:
     return pin_adm
 
 
-def enc_ePDGSelection(hexstr, mcc, mnc, epdg_priority='0001', epdg_fqdn_format='00'):
-    """
-    Encode ePDGSelection so it can be stored at EF.ePDGSelection or EF.ePDGSelectionEm.
-    See 3GPP TS 31.102 version 15.2.0 Release 15, section 4.2.104 and 4.2.106.
-
-    Default values:
-            - epdg_priority: '0001' - 1st Priority
-            - epdg_fqdn_format: '00' - Operator Identifier FQDN
-    """
-
-    plmn1 = enc_plmn(mcc, mnc) + epdg_priority + epdg_fqdn_format
-    # TODO: Handle encoding of Length field for length more than 127 Bytes
-    content = '80' + ('%02x' % (len(plmn1)//2)) + plmn1
-    content = rpad(content, len(hexstr))
-    return content
-
-
-def dec_ePDGSelection(sixhexbytes):
-    """
-    Decode ePDGSelection to get EF.ePDGSelection or EF.ePDGSelectionEm.
-    See 3GPP TS 31.102 version 15.2.0 Release 15, section 4.2.104 and 4.2.106.
-    """
-
-    res = {'mcc': 0, 'mnc': 0, 'epdg_priority': 0, 'epdg_fqdn_format': ''}
-    plmn_chars = 6
-    epdg_priority_chars = 4
-    epdg_fqdn_format_chars = 2
-    # first three bytes (six ascii hex chars)
-    plmn_str = sixhexbytes[:plmn_chars]
-    # two bytes after first three bytes
-    epdg_priority_str = sixhexbytes[plmn_chars:plmn_chars +
-                                    epdg_priority_chars]
-    # one byte after first five bytes
-    epdg_fqdn_format_str = sixhexbytes[plmn_chars +
-                                       epdg_priority_chars:plmn_chars + epdg_priority_chars + epdg_fqdn_format_chars]
-    res['mcc'] = dec_mcc_from_plmn(plmn_str)
-    res['mnc'] = dec_mnc_from_plmn(plmn_str)
-    res['epdg_priority'] = epdg_priority_str
-    res['epdg_fqdn_format'] = epdg_fqdn_format_str == '00' and 'Operator Identifier FQDN' or 'Location based FQDN'
-    return res
-
-
-def format_ePDGSelection(hexstr):
-    ePDGSelection_info_tag_chars = 2
-    ePDGSelection_info_tag_str = hexstr[:2]
-    s = ""
-    # Minimum length
-    len_chars = 2
-    # TODO: Need to determine length properly - definite length support only
-    # Inconsistency in spec: 3GPP TS 31.102 version 15.2.0 Release 15, 4.2.104
-    # As per spec, length is 5n, n - number of PLMNs
-    # But, each PLMN entry is made of PLMN (3 Bytes) + ePDG Priority (2 Bytes) + ePDG FQDN format (1 Byte)
-    # Totalling to 6 Bytes, maybe length should be 6n
-    len_str = hexstr[ePDGSelection_info_tag_chars:ePDGSelection_info_tag_chars+len_chars]
-
-    # Not programmed scenario
-    if int(len_str, 16) == 255 or int(ePDGSelection_info_tag_str, 16) == 255:
-        len_chars = 0
-        ePDGSelection_info_tag_chars = 0
-    if len_str[0] == '8':
-        # The bits 7 to 1 denotes the number of length octets if length > 127
-        if int(len_str[1]) > 0:
-            # Update number of length octets
-            len_chars = len_chars * int(len_str[1])
-            len_str = hexstr[ePDGSelection_info_tag_chars:len_chars]
-
-    content_str = hexstr[ePDGSelection_info_tag_chars+len_chars:]
-    # Right pad to prevent index out of range - multiple of 6 bytes
-    content_str = rpad(content_str, len(content_str) +
-                       (12 - (len(content_str) % 12)))
-    for rec_data in hexstr_to_Nbytearr(content_str, 6):
-        rec_info = dec_ePDGSelection(rec_data)
-        if rec_info['mcc'] == 0xFFF and rec_info['mnc'] == 0xFFF:
-            rec_str = "unused"
-        else:
-            rec_str = "MCC: %03d MNC: %03d ePDG Priority: %s ePDG FQDN format: %s" % \
-                (rec_info['mcc'], rec_info['mnc'],
-                 rec_info['epdg_priority'], rec_info['epdg_fqdn_format'])
-        s += "\t%s # %s\n" % (rec_data, rec_str)
-    return s
-
-
 def get_addr_type(addr):
     """
     Validates the given address and returns it's type (FQDN or IPv4 or IPv6)
@@ -1701,3 +1353,30 @@ class CardCommandSet:
 def all_subclasses(cls) -> set:
     """Recursively get all subclasses of a specified class"""
     return set(cls.__subclasses__()).union([s for c in cls.__subclasses__() for s in all_subclasses(c)])
+
+def is_hexstr_or_decimal(instr: str) -> str:
+    """Method that can be used as 'type' in argparse.add_argument() to validate the value consists of
+    [hexa]decimal digits only."""
+    if instr.isdecimal():
+        return instr
+    if not all(c in string.hexdigits for c in instr):
+        raise ValueError('Input must be [hexa]decimal')
+    if len(instr) & 1:
+        raise ValueError('Input has un-even number of hex digits')
+    return instr
+
+def is_hexstr(instr: str) -> str:
+    """Method that can be used as 'type' in argparse.add_argument() to validate the value consists of
+    an even sequence of hexadecimal digits only."""
+    if not all(c in string.hexdigits for c in instr):
+        raise ValueError('Input must be hexadecimal')
+    if len(instr) & 1:
+        raise ValueError('Input has un-even number of hex digits')
+    return instr
+
+def is_decimal(instr: str) -> str:
+    """Method that can be used as 'type' in argparse.add_argument() to validate the value consists of
+    an even sequence of decimal digits only."""
+    if not instr.isdecimal():
+        raise ValueError('Input must decimal')
+    return instr

pysim-testdata/Fairwaves-SIM.ok

@@ -31,7 +31,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -88,6 +88,18 @@ SIM Service Table: ff3cc3ff030fff0f000fff03f0c0
 	Service 58 - Extension 8
 	Service 59 - MMS User Connectivity Parameters
 
+FPLMN:
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+
 USIM Service Table: 01ea1ffc21360480010000
 	Service 1 - Local Phone Book
 	Service 10 - Short Message Storage (SMS)

pysim-testdata/Wavemobile-SIM.ok

@@ -11,7 +11,7 @@ Show in HPLMN: False
 Hide in OPLMN: False
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -29,7 +29,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -93,6 +93,12 @@ SIM Service Table: ff33ff0f3c00ff0f000cf0c0f0030000
 	Service 58 - Extension 8
 	Service 59 - MMS User Connectivity Parameters
 
+FPLMN:
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+
 USIM Service Table: 9eff1b3c37fe5900000000
 	Service 2 - Fixed Dialling Numbers (FDN)
 	Service 3 - Extension 2

pysim-testdata/fakemagicsim.ok

@@ -1,7 +1,6 @@
 Using PC/SC reader interface
 Reading ...
 Autodetected card type: fakemagicsim
-Can't read AIDs from SIM -- SW match failed! Expected 9000 and got 9404.
 ICCID: 1122334455667788990
 IMSI: 001010000000102
 GID1: Can't read file -- SW match failed! Expected 9000 and got 9404.

pysim-testdata/sysmoISIM-SJA2.ok

@@ -11,7 +11,7 @@ Show in HPLMN: True
 Hide in OPLMN: True
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -25,7 +25,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -39,7 +39,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -103,7 +103,13 @@ SIM Service Table: ff33ffff3f003f0f300cf0c3f00000
 	Service 59 - MMS User Connectivity Parameters
 
 EHPLMN:
-	00f110 # MCC: 001 MNC: 001
+	00f110 # MCC: 001 MNC: 01
+	ffffff # unused
+	ffffff # unused
+	ffffff # unused
+
+FPLMN:
+	ffffff # unused
 	ffffff # unused
 	ffffff # unused
 	ffffff # unused

pysim-testdata/sysmoUSIM-SJS1.ok

@@ -11,7 +11,7 @@ Show in HPLMN: True
 Hide in OPLMN: True
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -25,7 +25,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -39,7 +39,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: E-UTRAN NB-S1, E-UTRAN WB-S1, EC-GSM-IoT, GSM, GSM COMPACT, NG-RAN, UTRAN, cdma2000 1xRTT, cdma2000 HRPD
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -104,6 +104,12 @@ SIM Service Table: ff3fffff3f003f1ff00c00c0f00000
 	Service 58 - Extension 8
 	Service 59 - MMS User Connectivity Parameters
 
+FPLMN:
+	62f201 # MCC: 262 MNC: 10
+	62f202 # MCC: 262 MNC: 20
+	62f203 # MCC: 262 MNC: 30
+	62f207 # MCC: 262 MNC: 70
+
 USIM Service Table: 9e6b1dfc67f6580000
 	Service 2 - Fixed Dialling Numbers (FDN)
 	Service 3 - Extension 2

pysim-testdata/sysmosim-gr1.ok

@@ -1,7 +1,6 @@
 Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmosim-gr1
-Can't read AIDs from SIM -- SW match failed! Expected 9000 and got 9404.
 ICCID: 1122334455667788990
 IMSI: 001010000000102
 GID1: Can't read file -- SW match failed! Expected 9000 and got 9404.

requirements.txt

@@ -1,7 +1,7 @@
 pyscard
 pyserial
 pytlv
-cmd2==1.5
+cmd2>=1.5
 jsonpath-ng
 construct>=2.9.51
 bidict
@@ -9,3 +9,8 @@ gsm0338
 pyyaml>=5.1
 termcolor
 colorlog
+pycryptodomex
+cryptography
+asn1tools
+packaging
+git+https://github.com/hologram-io/smpp.pdu

setup.py

@@ -3,26 +3,31 @@ from setuptools import setup
 setup(
     name='pySim',
     version='1.0',
-    packages=['pySim', 'pySim.transport'],
+    packages=['pySim', 'pySim.legacy', 'pySim.transport', 'pySim.apdu', 'pySim.apdu_source'],
     url='https://osmocom.org/projects/pysim/wiki',
     license='GPLv2',
     author_email='simtrace@lists.osmocom.org',
     description='Tools related to SIM/USIM/ISIM cards',
     install_requires=[
         "pyscard",
-        "serial",
+        "pyserial",
         "pytlv",
-        "cmd2 >= 1.3.0, < 2.0.0",
+        "cmd2 >= 1.5.0",
         "jsonpath-ng",
         "construct >= 2.9.51",
         "bidict",
         "gsm0338",
+        "pyyaml >= 5.1",
         "termcolor",
-        "colorlog"
+        "colorlog",
+        "pycryptodomex",
+        "packaging",
+        "smpp.pdu @ git+https://github.com/hologram-io/smpp.pdu",
     ],
     scripts=[
         'pySim-prog.py',
         'pySim-read.py',
-        'pySim-shell.py'
+        'pySim-shell.py',
+        'pySim-trace.py',
     ]
 )

smdpp-data/certs/CertificateIssuer/CERT_CI_ECDSA_BRP.pem

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICUTCCAfigAwIBAgIJALh086v6bETTMAoGCCqGSM49BAMCMEQxEDAOBgNVBAMM
+B1Rlc3QgQ0kxETAPBgNVBAsMCFRFU1RDRVJUMRAwDgYDVQQKDAdSU1BURVNUMQsw
+CQYDVQQGEwJJVDAgFw0yMDA0MDEwODI3NTFaGA8yMDU1MDQwMTA4Mjc1MVowRDEQ
+MA4GA1UEAwwHVGVzdCBDSTERMA8GA1UECwwIVEVTVENFUlQxEDAOBgNVBAoMB1JT
+UFRFU1QxCzAJBgNVBAYTAklUMFowFAYHKoZIzj0CAQYJKyQDAwIIAQEHA0IABCeH
+tNVu2CSp5r4E4Yh/a5i6/rjHY/UoN/cBE+k2Tt2+E5vAx95+Fo8eXNDBhTT8UGTm
+T2htxTMnyn8dzqhaKZSjgc8wgcwwHQYDVR0OBBYEFMC8cLo2kp1DtGf/V1cFMOV6
+uPzYMA8GA1UdEwEB/wQFMAMBAf8wFwYDVR0gAQH/BA0wCzAJBgdngRIBAgEAMA4G
+A1UdDwEB/wQEAwIBBjAOBgNVHREEBzAFiAOINwEwYQYDVR0fBFowWDAqoCigJoYk
+aHR0cDovL2NpLnRlc3QuZXhhbXBsZS5jb20vQ1JMLUEuY3JsMCqgKKAmhiRodHRw
+Oi8vY2kudGVzdC5leGFtcGxlLmNvbS9DUkwtQi5jcmwwCgYIKoZIzj0EAwIDRwAw
+RAIgPYrf0CKl0FBMUaHx5xS1duTDbQ4wBZN3qKBeNniuux0CIHBek2vLfoANAdtt
+f5u5Ce6DVC2oIfpn5UnS24F3oMqM
+-----END CERTIFICATE-----

smdpp-data/certs/CertificateIssuer/CERT_CI_ECDSA_NIST.pem

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICUDCCAfegAwIBAgIJALh086v6bETTMAoGCCqGSM49BAMCMEQxEDAOBgNVBAMM
+B1Rlc3QgQ0kxETAPBgNVBAsMCFRFU1RDRVJUMRAwDgYDVQQKDAdSU1BURVNUMQsw
+CQYDVQQGEwJJVDAgFw0yMDA0MDEwODI3NTFaGA8yMDU1MDQwMTA4Mjc1MVowRDEQ
+MA4GA1UEAwwHVGVzdCBDSTERMA8GA1UECwwIVEVTVENFUlQxEDAOBgNVBAoMB1JT
+UFRFU1QxCzAJBgNVBAYTAklUMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAElAZX
+pnPcKI+J1S6opHcEmSeR+cNLADbmM+LQy6lFTWXbMusXmBeZ0vJDiO4rlcEJRUbJ
+eQHOrrqWUJGaLiDSKaOBzzCBzDAdBgNVHQ4EFgQU9UFyvfmKldZcvriKOKHBHYAK
+hcMwDwYDVR0TAQH/BAUwAwEB/zAXBgNVHSABAf8EDTALMAkGB2eBEgECAQAwDgYD
+VR0PAQH/BAQDAgEGMA4GA1UdEQQHMAWIA4g3ATBhBgNVHR8EWjBYMCqgKKAmhiRo
+dHRwOi8vY2kudGVzdC5leGFtcGxlLmNvbS9DUkwtQS5jcmwwKqAooCaGJGh0dHA6
+Ly9jaS50ZXN0LmV4YW1wbGUuY29tL0NSTC1CLmNybDAKBggqhkjOPQQDAgNHADBE
+AiBSdWqvwgIKbOy/Ll88IIklEP8pdR0pi9OwFdlgWk/mfQIgV5goNuTSBd3S5sPB
+tFWTf2tuSTtgL9G2bDV0iak192s=
+-----END CERTIFICATE-----

smdpp-data/certs/CertificateIssuer/CI-csr.cnf

@@ -0,0 +1,25 @@
+#openssl x509 extfile params
+extensions = extend
+# This prevent the user to be prompted for values
+prompt = no
+
+distinguished_name = dn-param
+[dn-param] # DN fields
+CN = Test CI
+OU = TESTCERT
+O = RSPTEST
+C = IT
+
+# Extensions for the Test CI 
+[extend] # openssl extensions
+subjectKeyIdentifier = hash
+
+basicConstraints = critical, CA:true
+
+certificatePolicies=critical,2.23.146.1.2.1.0
+
+keyUsage =critical, keyCertSign, cRLSign
+
+subjectAltName = RID:2.999.1
+
+crlDistributionPoints=URI:http://ci.test.example.com/CRL-A.crl, URI:http://ci.test.example.com/CRL-B.crl

smdpp-data/certs/DPauth/PK_S_SM_DP2auth_ECDSA_BRP.pem

@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFowFAYHKoZIzj0CAQYJKyQDAwIIAQEHA0IABC7uB8ltAFvlGV95rR2tzR03jKMJ
+XkT3LupBDwjHVhIskGPJZbf8hSnAyk6bT2WWnCg8ZWNkV4dxGPbRFy1qI2U=
+-----END PUBLIC KEY-----

smdpp-data/certs/DPauth/PK_S_SM_DP2auth_ECDSA_NIST.pem

@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKy6bRdNRwJr0DpFDg6GQOEfyYH3m
+DverJcQOA/jbtlCOFFdQjCvgAXNH7Pob+fd159B+gF5S4ZiLe1hacRGuMw==
+-----END PUBLIC KEY-----

smdpp-data/certs/DPauth/PK_S_SM_DPauth_ECDSA_BRP.pem

@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFowFAYHKoZIzj0CAQYJKyQDAwIIAQEHA0IABCVdfa8A74Qddvp6Y9Y+s/5sn3BJ
+LTgZI/j4vXsk88WtFo6+OwndgPKef/0kpNG+dH+DI+RykoOW3engFRzquhg=
+-----END PUBLIC KEY-----

smdpp-data/certs/DPauth/PK_S_SM_DPauth_ECDSA_NIST.pem

@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAETf7U9GlHkb8Wlc6gMHo1tBgBlpU4
+e7dbfSRHtrUgnwRFrk5eUhzROIjXX+B8hYAiKuINuqwdd812MEmTQhvXOQ==
+-----END PUBLIC KEY-----

smdpp-data/certs/DPauth/SK_S_SM_DP2auth_ECDSA_BRP.pem

@@ -0,0 +1,8 @@
+-----BEGIN EC PARAMETERS-----
+BgkrJAMDAggBAQc=
+-----END EC PARAMETERS-----
+-----BEGIN EC PRIVATE KEY-----
+MHgCAQEEIAwXNVwBHQ/o19rdY/GXhc9sUcvNRmroi+j4G8EFiEb2oAsGCSskAwMC
+CAEBB6FEA0IABC7uB8ltAFvlGV95rR2tzR03jKMJXkT3LupBDwjHVhIskGPJZbf8
+hSnAyk6bT2WWnCg8ZWNkV4dxGPbRFy1qI2U=
+-----END EC PRIVATE KEY-----

smdpp-data/certs/DPauth/SK_S_SM_DP2auth_ECDSA_NIST.pem

@@ -0,0 +1,8 @@
+-----BEGIN EC PARAMETERS-----
+BggqhkjOPQMBBw==
+-----END EC PARAMETERS-----
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIJwyoJXUiELZ/6QE9xJRKqLFQloaJjhqtqFF1YEeA5FBoAoGCCqGSM49
+AwEHoUQDQgAEKy6bRdNRwJr0DpFDg6GQOEfyYH3mDverJcQOA/jbtlCOFFdQjCvg
+AXNH7Pob+fd159B+gF5S4ZiLe1hacRGuMw==
+-----END EC PRIVATE KEY-----

smdpp-data/certs/DPauth/SK_S_SM_DPauth_ECDSA_BRP.pem

@@ -0,0 +1,8 @@
+-----BEGIN EC PARAMETERS-----
+BgkrJAMDAggBAQc=
+-----END EC PARAMETERS-----
+-----BEGIN EC PRIVATE KEY-----
+MHgCAQEEIJP7M9BYTzSbB/i10q+T18PjVLNJo7kTUC5qvAcOTUkpoAsGCSskAwMC
+CAEBB6FEA0IABCVdfa8A74Qddvp6Y9Y+s/5sn3BJLTgZI/j4vXsk88WtFo6+Ownd
+gPKef/0kpNG+dH+DI+RykoOW3engFRzquhg=
+-----END EC PRIVATE KEY-----

smdpp-data/certs/DPauth/SK_S_SM_DPauth_ECDSA_NIST.pem

@@ -0,0 +1,8 @@
+-----BEGIN EC PARAMETERS-----
+BggqhkjOPQMBBw==
+-----END EC PARAMETERS-----
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIAp8wcJE5gxSzVt4B6uMNgwmUkYBUH3KvF3VmLWmFtXVoAoGCCqGSM49
+AwEHoUQDQgAETf7U9GlHkb8Wlc6gMHo1tBgBlpU4e7dbfSRHtrUgnwRFrk5eUhzR
+OIjXX+B8hYAiKuINuqwdd812MEmTQhvXOQ==
+-----END EC PRIVATE KEY-----