pySim-shell_test/euicc: ensure test-profile is enabled

When testing commands like get_profile_info, enable_profile,
disable_profile or the commands to manage notifications, we
should ensure that the correct profile is enabled before
executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

.checkpatch.conf

@@ -0,0 +1,2 @@
+--exclude ^pySim/esim/asn1/.*\.asn$
+--exclude ^smdpp-data/.*$

.github/FUNDING.yml

@@ -0,0 +1 @@
+open_collective: osmocom

.gitignore

@@ -1,2 +1,16 @@
 *.pyc
 .*.swp
+
+/docs/_*
+/docs/generated
+/.cache
+/.local
+/build
+/pySim.egg-info
+/smdpp-data/sm-dp-sessions*
+dist
+tags
+smdpp-data/certs/DPtls/CERT_S_SM_DP_TLS_NIST.pem
+smdpp-data/certs/DPtls/CERT_S_SM_DP_TLS_BRP.pem
+smdpp-data/generated
+smdpp-data/certs/dhparam2048.pem

README.md

@@ -1,96 +1,45 @@
-pySim - Read, Write and Browse Programmable SIM/USIM Cards
-====================================================
+pySim - Tools for reading, decoding, browsing SIM/USIM/ISIM/HPSIM/eUICC Cards
+=============================================================================
 
-This repository contains Python programs that can be used
-to read, program (write) and browse certain fields/parameters on so-called programmable
-SIM/USIM cards.
+This repository contains a number of Python programs related to working with
+subscriber identity modules of cellular networks, including but not limited
+to SIM, UICC, USIM, ISIM, HPSIMs and eUICCs.
 
-Such SIM/USIM cards are special cards, which - unlike those issued by
-regular commercial operators - come with the kind of keys that allow you
-to write the files/fields that normally only an operator can program.
+* `pySim-shell.py` can be used to interactively explore, read and decode contents
+  of any of the supported card models / card applications.  Furthermore, if
+  you have the credentials to your card (ADM PIN), you can also write to the card,
+  i.e. edit its contents.
+* `pySim-read.py` and `pySim-prog.py` are _legacy_ tools for batch programming
+  some very common parameters to an entire batch of programmable cards
+* `pySim-trace.py` is a tool to do an in-depth decode of SIM card protocol traces
+  such as those obtained by [Osmocom SIMtrace2](https://osmocom.org/projects/simtrace2/wiki)
+  or [osmo-qcdiag](https://osmocom.org/projects/osmo-qcdiag/wiki).
+* `osmo-smdpp.py` is a proof-of-concept GSMA SGP.22 Consumer eSIM SM-DP+ for lab/research
+* there are more related tools, particularly in the `contrib` directory.
+
+Note that the access control configuration of normal production cards
+issue by operators will restrict significantly which files a normal
+user can read, and particularly write to.
+
+The full functionality of pySim hence can only be used with on so-called
+programmable SIM/USIM/ISIM/HPSIM cards, such as the various
+[sysmocom programmable card products](https://shop.sysmocom.de/SIM/).
+
+Such SIM/USIM/ISIM/HPSIM cards are special cards, which - unlike those
+issued by regular commercial operators - come with the kind of keys that
+allow you to write the files/fields that normally only an operator can
+program.
 
 This is useful particularly if you are running your own cellular
-network, and want to issue your own SIM/USIM cards for that network.
+network, and want to configure your own SIM/USIM/ISIM/HPSIM cards for
+that network.
 
 
-Homepage and Manual
--------------------
+Homepage
+--------
 
-Please visit the [official homepage](https://osmocom.org/projects/pysim/wiki) for usage instructions, manual and examples.
-
-Git Repository
---------------
-
-You can clone from the official Osmocom  git repository using
-```
-git clone git://git.osmocom.org/pysim.git
-```
-
-There is a cgit interface at <https://git.osmocom.org/pysim>
-
-
-Installation
-------------
-
-Please install the following dependencies:
-
- - pyscard
- - serial
- - pytlv
- - cmd2 >= 1.3.0 but < 2.0.0
- - jsonpath-ng
- - construct
- - bidict
- - gsm0338
-
-Example for Debian:
-```
-apt-get install python3-pyscard python3-serial python3-pip python3-yaml
-pip3 install -r requirements.txt
-```
-
-After installing all dependencies, the pySim applications ``pySim-read.py``, ``pySim-prog.py`` and ``pySim-shell.py`` may be started directly from the cloned repository.
-
-### Archlinux Package
-
-Archlinux users may install the package ``python-pysim-git``
-[![](https://img.shields.io/aur/version/python-pysim-git)](https://aur.archlinux.org/packages/python-pysim-git)
-from the [Arch User Repository (AUR)](https://aur.archlinux.org).
-The most convenient way is the use of an [AUR Helper](https://wiki.archlinux.org/index.php/AUR_helpers),
-e.g. [yay](https://aur.archlinux.org/packages/yay) or [pacaur](https://aur.archlinux.org/packages/pacaur).
-The following example shows the installation with ``yay``.
-
-```sh
-# Install
-yay -Sy python-pysim-git
-
-# Uninstall
-sudo pacman -Rs python-pysim-git
-```
-
-
-Mailing List
-------------
-
-There is no separate mailing list for this project. However,
-discussions related to pysim-prog are happening on the
-<openbsc@lists.osmocom.org> mailing list, please see
-<https://lists.osmocom.org/mailman/listinfo/openbsc> for subscription
-options and the list archive.
-
-Please observe the [Osmocom Mailing List
-Rules](https://osmocom.org/projects/cellular-infrastructure/wiki/Mailing_List_Rules)
-when posting.
-
-
-Contributing
-------------
-
-Our coding standards are described at
-<https://osmocom.org/projects/cellular-infrastructure/wiki/Coding_standards>
-
-We are using a gerrit-based patch review process explained at
-<https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit>
+Please visit the [official homepage](https://osmocom.org/projects/pysim/wiki)
+for usage instructions, manual and examples.
 
 
 Documentation
@@ -113,10 +62,114 @@ pySim-shell vs. legacy tools
 ----------------------------
 
 While you will find a lot of online resources still describing the use of
-pySim-prog.py and pySim-read.py, those tools are considered legacy by
+`pySim-prog.py` and `pySim-read.py`, those tools are considered legacy by
 now and have by far been superseded by the much more capable
-pySim-shell.  We strongly encourage users to adopt pySim-shell, unless
+`pySim-shell.py`.  We strongly encourage users to adopt pySim-shell, unless
 they have very specific requirements like batch programming of large
 quantities of cards, which is about the only remaining use case for the
 legacy tools.
 
+
+Git Repository
+--------------
+
+You can clone from the official Osmocom  git repository using
+```
+git clone https://gitea.osmocom.org/sim-card/pysim.git
+```
+
+There is a web interface at <https://gitea.osmocom.org/sim-card/pysim>.
+
+
+Installation
+------------
+
+Please install the following dependencies:
+
+ - bidict
+ - cmd2 >= 1.5.0
+ - colorlog
+ - construct >= 2.9.51
+ - pyosmocom
+ - jsonpath-ng
+ - packaging
+ - pycryptodomex
+ - pyscard
+ - pyserial
+ - pytlv
+ - pyyaml >= 5.1
+ - smpp.pdu (from `github.com/hologram-io/smpp.pdu`)
+ - termcolor
+
+Example for Debian:
+```sh
+sudo apt-get install --no-install-recommends \
+	pcscd libpcsclite-dev \
+	python3 \
+	python3-setuptools \
+	python3-pycryptodome \
+	python3-pyscard \
+	python3-pip
+pip3 install --user -r requirements.txt
+```
+
+After installing all dependencies, the pySim applications ``pySim-read.py``, ``pySim-prog.py`` and ``pySim-shell.py`` may be started directly from the cloned repository.
+
+In addition to the dependencies above ``pySim-trace.py`` requires ``tshark`` and the python package ``pyshark`` to be installed. It is known that the ``tshark`` package
+in Debian versions before 11 may not work with pyshark.
+
+### Archlinux Package
+
+Archlinux users may install the package ``python-pysim-git``
+[![](https://img.shields.io/aur/version/python-pysim-git)](https://aur.archlinux.org/packages/python-pysim-git)
+from the [Arch User Repository (AUR)](https://aur.archlinux.org).
+The most convenient way is the use of an [AUR Helper](https://wiki.archlinux.org/index.php/AUR_helpers),
+e.g. [yay](https://aur.archlinux.org/packages/yay) or [pacaur](https://aur.archlinux.org/packages/pacaur).
+The following example shows the installation with ``yay``.
+
+```sh
+# Install
+yay -Sy python-pysim-git
+
+# Uninstall
+sudo pacman -Rs python-pysim-git
+```
+
+
+Forum
+-----
+
+We welcome any pySim related discussions in the
+[SIM Card Technology](https://discourse.osmocom.org/c/sim-card-technology/)
+section of the osmocom discourse (web based Forum).
+
+
+Mailing List
+------------
+
+There is no separate mailing list for this project. However,
+discussions related to pySim are happening on the simtrace
+<simtrace@lists.osmocom.org> mailing list, please see
+<https://lists.osmocom.org/mailman/listinfo/simtrace> for subscription
+options and the list archive.
+
+Please observe the [Osmocom Mailing List
+Rules](https://osmocom.org/projects/cellular-infrastructure/wiki/Mailing_List_Rules)
+when posting.
+
+Issue Tracker
+-------------
+
+We use the [issue tracker of the pysim project on osmocom.org](https://osmocom.org/projects/pysim/issues) for
+tracking the state of bug reports and feature requests.  Feel free to submit any issues you may find, or help
+us out by resolving existing issues.
+
+
+Contributing
+------------
+
+Our coding standards are described at
+<https://osmocom.org/projects/cellular-infrastructure/wiki/Coding_standards>
+
+We are using a gerrit-based patch review process explained at
+<https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit>

contrib/analyze_simaResponse.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+
+# A tool to analyze the eUICC simaResponse (series of EUICCResponse)
+#
+# (C) 2025 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+from osmocom.utils import h2b, b2h
+from osmocom.tlv import bertlv_parse_one, bertlv_encode_tag, bertlv_encode_len
+from pySim.esim.saip import *
+
+parser = argparse.ArgumentParser(description="""Utility program to analyze the contents of an eUICC simaResponse.""")
+parser.add_argument('SIMA_RESPONSE', help='Hexstring containing the simaResponse as received from the eUICC')
+
+def split_sima_response(sima_response):
+    """split an eUICC simaResponse field into a list of EUICCResponse fields"""
+
+    remainder = sima_response
+    result = []
+    while len(remainder):
+        tdict, l, v, next_remainder = bertlv_parse_one(remainder)
+        rawtag = bertlv_encode_tag(tdict)
+        rawlen = bertlv_encode_len(l)
+        result = result + [remainder[0:len(rawtag) + len(rawlen) + l]]
+        remainder = next_remainder
+    return result
+
+def analyze_status(status):
+    """
+    Convert a status code (integer) into a human readable string
+    (see eUICC Profile Package: Interoperable Format Technical Specification, section 8.11)
+    """
+
+    # SIMA status codes
+    string_values = {0 : 'ok',
+                     1 : 'pe-not-supported',
+                     2 : 'memory-failure',
+                     3 : 'bad-values',
+                     4 : 'not-enough-memory',
+                     5 : 'invalid-request-format',
+                     6 : 'invalid-parameter',
+                     7 : 'runtime-not-supported',
+                     8 : 'lib-not-supported',
+                     9 : 'template-not-supported ',
+                     10 : 'feature-not-supported',
+                     11 : 'pin-code-missing',
+                     31 : 'unsupported-profile-version'}
+
+    string_value = string_values.get(status, None)
+    if string_value is not None:
+        return "%d = %s (SIMA status code)" % (status, string_value)
+
+    # ISO 7816 status words
+    if status >= 24576 and status <= 28671:
+        return "%d = %04x (ISO7816 status word)" % (status, status)
+    elif status >= 36864 and status <= 40959:
+        return "%d = %04x (ISO7816 status word)" % (status, status)
+
+    # Proprietary status codes
+    elif status >= 40960 and status <= 65535:
+        return "%d = %04x (proprietary)" % (status, status)
+
+    # Unknown status codes
+    return "%d (unknown, proprietary?)" % status
+
+def analyze_euicc_response(euicc_response):
+    """Analyze and display the contents of an EUICCResponse"""
+
+    print(" EUICCResponse: %s" % b2h(euicc_response))
+    euicc_response_decoded = asn1.decode('EUICCResponse', euicc_response)
+
+    pe_status = euicc_response_decoded.get('peStatus')
+    print("  peStatus:")
+    for s in pe_status:
+        print("   status: %s" % analyze_status(s.get('status')))
+        print("   identification: %s" % str(s.get('identification', None)))
+        print("   additional-information: %s" % str(s.get('additional-information', None)))
+        print("   offset: %s" % str(s.get('offset', None)))
+
+    if euicc_response_decoded.get('profileInstallationAborted', False) is None:
+        # This type is defined as profileInstallationAborted NULL OPTIONAL, so when it is present it
+        # will have the value None, otherwise it is simply not present.
+        print("  profileInstallationAborted: True")
+    else:
+        print("  profileInstallationAborted: False")
+
+    status_message = euicc_response_decoded.get('statusMessage', None)
+    print("  statusMessage: %s" % str(status_message))
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+    sima_response = h2b(opts.SIMA_RESPONSE);
+
+    print("simaResponse: %s" % b2h(sima_response))
+    euicc_response_list = split_sima_response(sima_response)
+
+    for euicc_response in euicc_response_list:
+        analyze_euicc_response(euicc_response)

contrib/csv-encrypt-columns.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+
+# Utility program to perform column-based encryption of a CSV file holding SIM/UICC
+# related key materials.
+#
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+import csv
+import argparse
+from Cryptodome.Cipher import AES
+from osmocom.utils import h2b, b2h, Hexstr
+
+from pySim.card_key_provider import CardKeyFieldCryptor
+
+class CsvColumnEncryptor(CardKeyFieldCryptor):
+    def __init__(self, filename: str, transport_keys: dict):
+        self.filename = filename
+        self.crypt = CardKeyFieldCryptor(transport_keys)
+
+    def encrypt(self) -> None:
+        with open(self.filename, 'r') as infile:
+            cr = csv.DictReader(infile)
+            cr.fieldnames = [field.upper() for field in cr.fieldnames]
+
+            with open(self.filename + '.encr', 'w') as outfile:
+                cw = csv.DictWriter(outfile, dialect=csv.unix_dialect, fieldnames=cr.fieldnames)
+                cw.writeheader()
+
+                for row in cr:
+                    for fieldname in cr.fieldnames:
+                            row[fieldname] = self.crypt.encrypt_field(fieldname, row[fieldname])
+                    cw.writerow(row)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('CSVFILE', help="CSV file name")
+    parser.add_argument('--csv-column-key', action='append', required=True,
+                        help='per-CSV-column AES transport key')
+
+    opts = parser.parse_args()
+
+    csv_column_keys = {}
+    for par in opts.csv_column_key:
+        name, key = par.split(':')
+        csv_column_keys[name] = key
+
+    if len(csv_column_keys) == 0:
+        print("You must specify at least one key!")
+        sys.exit(1)
+
+    cce = CsvColumnEncryptor(opts.CSVFILE, csv_column_keys)
+    cce.encrypt()

contrib/csv-to-pgsql.py

@@ -0,0 +1,304 @@
+#!/usr/bin/env python3
+
+# (C) 2025 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# Author: Philipp Maier
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import logging
+import csv
+import sys
+import os
+import yaml
+import psycopg2
+from psycopg2.sql import Identifier, SQL
+from pathlib import Path
+from pySim.log import PySimLogger
+from packaging import version
+
+log = PySimLogger.get(Path(__file__).stem)
+
+class CardKeyDatabase:
+    def __init__(self, config_filename: str, table_name: str, create_table: bool = False, admin: bool = False):
+        """
+        Initialize database connection and set the table which shall be used as storage for the card key data.
+        In case the specified table does not exist yet it can be created using the create_table_type parameter.
+
+        New tables are always minimal tables which follow a pre-defined table scheme. The user may extend the table
+        with additional columns using the add_cols() later.
+
+        Args:
+                tablename : name of the database table to create.
+                create_table_type : type of the table to create ('UICC' or 'EUICC')
+        """
+
+        def user_from_config_file(config, role: str) -> tuple[str, str]:
+            db_users = config.get('db_users')
+            user = db_users.get(role)
+            if user is None:
+                raise ValueError("user for role '%s' not set up in config file." % role)
+            return user.get('name'), user.get('pass')
+
+        self.table = table_name.lower()
+        self.cols = None
+
+        # Depending on the table type, the table name must contain either the substring "uicc_keys" or "euicc_keys".
+        # This convention will allow us to deduct the table type from the table name.
+        if "euicc_keys" not in table_name and "uicc_keys" not in table_name:
+            raise ValueError("Table name (%s) should contain the substring \"uicc_keys\" or \"euicc_keys\"" % table_name)
+
+        # Read config file
+        log.info("Using config file: %s", config_filename)
+        with open(config_filename, "r") as cfg:
+            config = yaml.load(cfg, Loader=yaml.FullLoader)
+        host = config.get('host')
+        log.info("Database host: %s", host)
+        db_name = config.get('db_name')
+        log.info("Database name: %s", db_name)
+        table_names = config.get('table_names')
+        username_admin, password_admin = user_from_config_file(config, 'admin')
+        username_importer, password_importer = user_from_config_file(config, 'importer')
+        username_reader, _ = user_from_config_file(config, 'reader')
+
+        # Switch between admin and importer user
+        if admin:
+            username, password = username_admin, password_admin
+        else:
+            username, password = username_importer, password_importer
+
+        # Create database connection
+        log.info("Database user: %s", username)
+        self.conn = psycopg2.connect(dbname=db_name, user=username, password=password, host=host)
+        self.cur = self.conn.cursor()
+
+        # In the context of this tool it is not relevant if the table name is present in the config file. However,
+        # pySim-shell.py will require the table name to be configured properly to access the database table.
+        if self.table not in table_names:
+            log.warning("Specified table name (%s) is not yet present in config file (required for access from pySim-shell.py)",
+                        self.table)
+
+        # Create a new minimal database table of the specified table type.
+        if create_table:
+            if not admin:
+                raise ValueError("creation of new table refused, use option --admin and try again.")
+            if "euicc_keys" in self.table:
+                self.__create_table(username_reader, username_importer, ['EID'])
+            elif "uicc_keys" in self.table:
+                self.__create_table(username_reader, username_importer, ['ICCID', 'IMSI'])
+
+        # Ensure a table with the specified name exists
+        log.info("Database table: %s", self.table)
+        if self.get_cols() == []:
+            raise ValueError("Table name (%s) does not exist yet" % self.table)
+        log.info("Database table columns: %s", str(self.get_cols()))
+
+    def __create_table(self, user_reader:str, user_importer:str, cols:list[str]):
+        """
+        Initialize a new table. New tables are always minimal tables with one primary key and additional index columns.
+        Non index-columns may be added later using method _update_cols().
+        """
+
+        # Create table columns with primary key
+        query = SQL("CREATE TABLE {} ({} VARCHAR PRIMARY KEY").format(Identifier(self.table),
+                                                                      Identifier(cols[0].lower()))
+        for c in cols[1:]:
+            query += SQL(", {} VARCHAR").format(Identifier(c.lower()))
+        query += SQL(");")
+        self.cur.execute(query)
+
+        # Create indexes for all other columns
+        for c in cols[1:]:
+            self.cur.execute(query = SQL("CREATE INDEX {} ON {}({});").format(Identifier(c.lower()),
+                                                                              Identifier(self.table),
+                                                                              Identifier(c.lower())))
+
+        # Set permissions
+        self.cur.execute(SQL("GRANT INSERT ON {} TO {};").format(Identifier(self.table),
+                                                                 Identifier(user_importer)))
+        self.cur.execute(SQL("GRANT SELECT ON {} TO {};").format(Identifier(self.table),
+                                                                 Identifier(user_reader)))
+
+        log.info("New database table created: %s", self.table)
+
+    def get_cols(self) -> list[str]:
+        """
+        Get a list of all columns available in the current table scheme.
+
+        Returns:
+                list with column names (in uppercase) of the database table
+        """
+
+        # Return cached col list if present
+        if self.cols:
+            return self.cols
+
+        # Request a list of current cols from the database
+        self.cur.execute("SELECT column_name FROM information_schema.columns where table_name = %s;", (self.table,))
+
+        cols_result = self.cur.fetchall()
+        cols = []
+        for c in cols_result:
+            cols.append(c[0].upper())
+        self.cols = cols
+        return cols
+
+    def get_missing_cols(self, cols_expected:list[str]) -> list[str]:
+        """
+        Check if the current table scheme lacks any of the given expected columns.
+
+        Returns:
+                list with the missing columns.
+        """
+
+        cols_present = self.get_cols()
+        return list(set(cols_expected) - set(cols_present))
+
+    def add_cols(self, cols:list[str]):
+        """
+        Update the current table scheme with additional columns. In case the updated columns are already exist, the
+        table schema is not changed.
+
+        Args:
+                table : name of the database table to alter
+                cols : list with updated colum names to add
+        """
+
+        cols_missing = self.get_missing_cols(cols)
+
+        # Depending on the table type (see constructor), we either have a primary key 'ICCID' (for UICC data), or 'EID'
+        # (for eUICC data). Both table formats different types of data and have rather differen columns also. Let's
+        # prevent the excidentally mixing of both types.
+        if 'ICCID' in cols_missing:
+            raise ValueError("Table %s stores eUCCC key material, refusing to add UICC specific column 'ICCID'" % self.table)
+        if 'EID' in cols_missing:
+            raise ValueError("Table %s stores UCCC key material, refusing to add eUICC specific column 'EID'" % self.table)
+
+        # Add the missing columns to the table
+        self.cols = None
+        for c in cols_missing:
+            self.cur.execute(query = SQL("ALTER TABLE {} ADD {} VARCHAR;").format(Identifier(self.table),
+                                                                                  Identifier(c.lower())))
+
+    def insert_row(self, row:dict[str, str]):
+        """
+        Insert a new row into the database table.
+
+        Args:
+                row : dictionary with the colum names and their designated values
+        """
+
+        # Check if the row is compatible with the current table scheme
+        cols_expected = list(row.keys())
+        cols_missing = self.get_missing_cols(cols_expected)
+        if cols_missing != []:
+            raise ValueError("table %s has incompatible format, the row %s contains unknown cols %s" %
+                             (self.table, str(row), str(cols_missing)))
+
+        # Insert row into datbase table
+        row_keys = list(row.keys())
+        row_values = list(row.values())
+        query = SQL("INSERT INTO {} ").format(Identifier(self.table))
+        query += SQL("({} ").format(Identifier(row_keys[0].lower()))
+        for k in row_keys[1:]:
+            query += SQL(", {}").format(Identifier(k.lower()))
+        query += SQL(") VALUES (%s")
+        for v in row_values[1:]:
+            query += SQL(", %s")
+        query += SQL(");")
+        self.cur.execute(query, row_values)
+
+    def commit(self):
+        self.conn.commit()
+        log.info("Changes to table %s committed!", self.table)
+
+def open_csv(opts: argparse.Namespace):
+    log.info("CSV file: %s", opts.csv)
+    csv_file = open(opts.csv, 'r')
+    cr = csv.DictReader(csv_file)
+    if not cr:
+        raise RuntimeError("could not open DictReader for CSV-File '%s'" % opts.csv)
+    cr.fieldnames = [field.upper() for field in cr.fieldnames]
+    log.info("CSV file columns: %s", str(cr.fieldnames))
+    return cr
+
+def open_db(cr: csv.DictReader, opts: argparse.Namespace) -> CardKeyDatabase:
+    try:
+        db = CardKeyDatabase(os.path.expanduser(opts.pgsql), opts.table_name, opts.create_table, opts.admin)
+
+        # Check CSV format against table schema, add missing columns
+        cols_missing = db.get_missing_cols(cr.fieldnames)
+        if cols_missing != [] and (opts.update_columns or opts.create_table):
+            log.info("Adding missing columns: %s", str(cols_missing))
+            db.add_cols(cols_missing)
+            cols_missing = db.get_missing_cols(cr.fieldnames)
+
+        # Make sure the table schema has no missing columns
+        if cols_missing != []:
+            log.error("Database table lacks CSV file columns: %s -- import aborted!", cols_missing)
+            sys.exit(2)
+    except Exception as e:
+        log.error(str(e).strip())
+        log.error("Database initialization aborted due to error!")
+        sys.exit(2)
+
+    return db
+
+def import_from_csv(db: CardKeyDatabase, cr: csv.DictReader):
+    count = 0
+    for row in cr:
+        try:
+            db.insert_row(row)
+            count+=1
+            if count % 100 == 0:
+                log.info("CSV file import in progress, %d rows imported...", count)
+        except Exception as e:
+            log.error(str(e).strip())
+            log.error("CSV file import aborted due to error, no datasets committed!")
+            sys.exit(2)
+    log.info("CSV file import done, %d rows imported", count)
+
+if __name__ == '__main__':
+    option_parser = argparse.ArgumentParser(description='CSV importer for pySim-shell\'s PostgreSQL Card Key Provider',
+                                            formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    option_parser.add_argument("--verbose", help="Enable verbose logging", action='store_true', default=False)
+    option_parser.add_argument('--pgsql', metavar='FILE',
+                               default="~/.osmocom/pysim/card_data_pgsql.cfg",
+                               help='Read card data from PostgreSQL database (config file)')
+    option_parser.add_argument('--csv', metavar='FILE', help='input CSV file with card data', required=True)
+    option_parser.add_argument("--table-name", help="name of the card key table", type=str, required=True)
+    option_parser.add_argument("--update-columns", help="add missing table columns", action='store_true', default=False)
+    option_parser.add_argument("--create-table", action='store_true', help="create new card key table", default=False)
+    option_parser.add_argument("--admin", action='store_true', help="perform action as admin", default=False)
+    opts = option_parser.parse_args()
+
+    PySimLogger.setup(print, {logging.WARN: "\033[33m"})
+    if (opts.verbose):
+        PySimLogger.set_verbose(True)
+        PySimLogger.set_level(logging.DEBUG)
+
+    # Open CSV file
+    cr = open_csv(opts)
+
+    # Open database, create initial table, update column scheme
+    db = open_db(cr, opts)
+
+    # Progress with import
+    if not opts.admin:
+        import_from_csv(db, cr)
+
+    # Commit changes to the database
+    db.commit()

contrib/eidtool.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+
+# Command line tool to compute or verify EID (eUICC ID) values
+#
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+import argparse
+
+from pySim.euicc import compute_eid_checksum, verify_eid_checksum
+
+
+option_parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,
+                                        description="""pySim EID Tool
+This utility program can be used to compute or verify the checksum of an EID
+(eUICC Identifier).  See GSMA SGP.29 for the algorithm details.
+
+Example (verification):
+    $ eidtool.py --verify 89882119900000000000000000001654
+    EID checksum verified successfully
+
+Example (generation, passing first 30 digits):
+    $ eidtool.py --compute 898821199000000000000000000016
+    89882119900000000000000000001654
+
+Example (generation, passing all 32 digits):
+    $ eidtool.py --compute 89882119900000000000000000001600
+    89882119900000000000000000001654
+
+Example (generation, specifying base 30 digits and number to add):
+    $ eidtool.py --compute 898821199000000000000000000000 --add 16
+    89882119900000000000000000001654
+""")
+group = option_parser.add_mutually_exclusive_group(required=True)
+group.add_argument('--verify', help='Verify given EID csum')
+group.add_argument('--compute', help='Generate EID csum')
+option_parser.add_argument('--add', type=int, help='Add value to EID base before computing')
+
+
+if __name__ == '__main__':
+    opts = option_parser.parse_args()
+
+    if opts.verify:
+        res = verify_eid_checksum(opts.verify)
+        if res:
+            print("EID checksum verified successfully")
+            sys.exit(0)
+        else:
+            print("EID checksum invalid")
+            sys.exit(1)
+    elif opts.compute:
+        eid = opts.compute
+        if opts.add:
+            if len(eid) != 30:
+                print("EID base must be 30 digits when using --add")
+                sys.exit(2)
+            eid = str(int(eid) + int(opts.add))
+        res = compute_eid_checksum(eid)
+        print(res)
+

contrib/es2p_client.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import copy
+import argparse
+from pySim.esim import es2p, ActivationCode
+
+EID_HELP='EID of the eUICC for which eSIM shall be made available'
+ICCID_HELP='The ICCID of the eSIM that shall be made available'
+MATCHID_HELP='MatchingID that shall be used by profile download'
+
+parser = argparse.ArgumentParser(description="""
+Utility to manually issue requests against the ES2+ API of an SM-DP+ according to GSMA SGP.22.""")
+parser.add_argument('--url', required=True, help='Base URL of ES2+ API endpoint')
+parser.add_argument('--id', required=True, help='Entity identifier passed to SM-DP+')
+parser.add_argument('--client-cert', help='X.509 client certificate used to authenticate to server')
+parser.add_argument('--server-ca-cert', help="""X.509 CA certificates acceptable for the server side. In
+                    production use cases, this would be the GSMA Root CA (CI) certificate.""")
+subparsers = parser.add_subparsers(dest='command',help="The command (API function) to call")
+
+parser_dlo = subparsers.add_parser('download-order', help="ES2+ DownloadOrder function")
+parser_dlo.add_argument('--eid', help=EID_HELP)
+parser_dlo.add_argument('--iccid', help=ICCID_HELP)
+parser_dlo.add_argument('--profileType', help='The profile type of which one eSIM shall be made available')
+
+parser_cfo = subparsers.add_parser('confirm-order', help="ES2+ ConfirmOrder function")
+parser_cfo.add_argument('--iccid', required=True, help=ICCID_HELP)
+parser_cfo.add_argument('--eid', help=EID_HELP)
+parser_cfo.add_argument('--matchingId', help=MATCHID_HELP)
+parser_cfo.add_argument('--confirmationCode', help='Confirmation code that shall be used by profile download')
+parser_cfo.add_argument('--smdsAddress', help='SM-DS Address')
+parser_cfo.add_argument('--releaseFlag', action='store_true', help='Shall the profile be immediately released?')
+
+parser_co = subparsers.add_parser('cancel-order', help="ES2+ CancelOrder function")
+parser_co.add_argument('--iccid', required=True, help=ICCID_HELP)
+parser_co.add_argument('--eid', help=EID_HELP)
+parser_co.add_argument('--matchingId', help=MATCHID_HELP)
+parser_co.add_argument('--finalProfileStatusIndicator', required=True, choices=['Available','Unavailable'])
+
+parser_rp = subparsers.add_parser('release-profile', help='ES2+ ReleaseProfile function')
+parser_rp.add_argument('--iccid', required=True, help=ICCID_HELP)
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+    #print(opts)
+
+    peer = es2p.Es2pApiClient(opts.url, opts.id, server_cert_verify=opts.server_ca_cert, client_cert=opts.client_cert)
+
+    data = {}
+    for k, v in vars(opts).items():
+        if k in ['url', 'id', 'client_cert', 'server_ca_cert', 'command']:
+            # remove keys from dict that should not end up in JSON...
+            continue
+        if v is not None:
+            data[k] = v
+
+    print(data)
+    if opts.command == 'download-order':
+        res = peer.call_downloadOrder(data)
+    elif opts.command == 'confirm-order':
+        res = peer.call_confirmOrder(data)
+        matchingId = res.get('matchingId', None)
+        smdpAddress = res.get('smdpAddress', None)
+        if matchingId:
+            ac = ActivationCode(smdpAddress, matchingId, cc_required=bool(opts.confirmationCode))
+            print("Activation Code: '%s'" % ac.to_string())
+    elif opts.command == 'cancel-order':
+        res = peer.call_cancelOrder(data)
+    elif opts.command == 'release-profile':
+        res = peer.call_releaseProfile(data)

contrib/es2p_server.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+
+# (C) 2026 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# Author: Philipp Maier
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+import argparse
+import logging
+import json
+import asn1tools
+import asn1tools.codecs.ber
+import asn1tools.codecs.der
+import pySim.esim.rsp as rsp
+import pySim.esim.saip as saip
+from pySim.esim.es2p import param, Es2pApiServerMno, Es2pApiServerHandlerMno
+from osmocom.utils import b2h
+from datetime import datetime
+from analyze_simaResponse import split_sima_response
+from pathlib import Path
+
+logger = logging.getLogger(Path(__file__).stem)
+
+parser = argparse.ArgumentParser(description="""
+Utility to receive and log requests against the ES2+ API of an SM-DP+ according to GSMA SGP.22.""")
+parser.add_argument("--host", help="Host/IP to bind HTTP(S) to", default="localhost")
+parser.add_argument("--port", help="TCP port to bind HTTP(S) to", default=443, type=int)
+parser.add_argument('--server-cert', help='X.509 server certificate used to provide the ES2+ HTTPs service')
+parser.add_argument('--client-ca-cert', help='X.509 CA certificates to authenticate the requesting client(s)')
+parser.add_argument("-v", "--verbose", help="enable debug output", action='store_true', default=False)
+
+def decode_sima_response(sima_response):
+    decoded = []
+    euicc_response_list = split_sima_response(sima_response)
+    for euicc_response in euicc_response_list:
+        decoded.append(saip.asn1.decode('EUICCResponse', euicc_response))
+    return decoded
+
+def decode_result_data(result_data):
+    return rsp.asn1.decode('PendingNotification', result_data)
+
+def decode(data, path="/"):
+    if data is None:
+        return 'none'
+    elif type(data) is datetime:
+        return data.isoformat()
+    elif type(data) is tuple:
+        return {str(data[0]) : decode(data[1], path + str(data[0]) + "/")}
+    elif type(data) is list:
+        new_data = []
+        for item in data:
+            new_data.append(decode(item, path))
+        return new_data
+    elif type(data) is bytes:
+        return b2h(data)
+    elif type(data) is dict:
+        new_data = {}
+        for key, item in data.items():
+            new_key = str(key)
+            if path == '/' and new_key == 'resultData':
+                new_item = decode_result_data(item)
+            elif (path == '/resultData/profileInstallationResult/profileInstallationResultData/finalResult/successResult/' \
+                  or path == '/resultData/profileInstallationResult/profileInstallationResultData/finalResult/errorResult/') \
+                  and new_key == 'simaResponse':
+                new_item = decode_sima_response(item)
+            else:
+                new_item = item
+            new_data[new_key] = decode(new_item, path + new_key + "/")
+        return new_data
+    else:
+        return data
+
+class Es2pApiServerHandlerForLogging(Es2pApiServerHandlerMno):
+    def call_handleDownloadProgressInfo(self, data: dict) -> (dict, str):
+        logging.info("ES2+:handleDownloadProgressInfo: %s" % json.dumps(decode(data)))
+        return {}, None
+
+if __name__ == "__main__":
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.WARNING,
+                        format='%(asctime)s %(levelname)s %(message)s',
+                        datefmt='%Y-%m-%d %H:%M:%S')
+
+    Es2pApiServerMno(args.port, args.host, Es2pApiServerHandlerForLogging(), args.server_cert, args.client_ca_cert)
+

contrib/es9p_client.py

@@ -0,0 +1,318 @@
+#!/usr/bin/env python3
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import sys
+import argparse
+import logging
+import hashlib
+
+from typing import List
+from urllib.parse import urlparse
+
+from cryptography import x509
+from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
+from cryptography.hazmat.primitives.asymmetric import ec
+
+from osmocom.utils import h2b, b2h, swap_nibbles, is_hexstr
+from osmocom.tlv import bertlv_parse_one_rawtag, bertlv_return_one_rawtlv
+
+import pySim.esim.rsp as rsp
+from pySim.esim import es9p, PMO
+from pySim.esim.x509_cert import CertAndPrivkey
+from pySim.esim.es8p import BoundProfilePackage
+
+logging.basicConfig(level=logging.DEBUG)
+
+parser = argparse.ArgumentParser(description="""
+Utility to manually issue requests against the ES9+ API of an SM-DP+ according to GSMA SGP.22.""")
+parser.add_argument('--url', required=True, help='Base URL of ES9+ API endpoint')
+parser.add_argument('--server-ca-cert', help="""X.509 CA certificates acceptable for the server side. In
+                    production use cases, this would be the GSMA Root CA (CI) certificate.""")
+parser.add_argument('--certificate-path', default='.',
+                    help="Path in which to look for certificate and key files.")
+parser.add_argument('--euicc-certificate', default='CERT_EUICC_ECDSA_NIST.der',
+                    help="File name of DER-encoded eUICC certificate file.")
+parser.add_argument('--euicc-private-key', default='SK_EUICC_ECDSA_NIST.pem',
+                    help="File name of PEM-format eUICC secret key file.")
+parser.add_argument('--eum-certificate', default='CERT_EUM_ECDSA_NIST.der',
+                    help="File name of DER-encoded EUM certificate file.")
+parser.add_argument('--ci-certificate', default='CERT_CI_ECDSA_NIST.der',
+                    help="File name of DER-encoded CI certificate file.")
+
+subparsers = parser.add_subparsers(dest='command',help="The command (API function) to call", required=True)
+
+# download
+parser_dl = subparsers.add_parser('download', help="ES9+ download")
+parser_dl.add_argument('--matchingId', required=True,
+                       help='MatchingID that shall be used by profile download')
+parser_dl.add_argument('--output-path', default='.',
+                       help="Path to which the output files will be written.")
+parser_dl.add_argument('--confirmation-code',
+                       help="Confirmation Code for the eSIM download")
+
+# notification
+parser_ntf = subparsers.add_parser('notification', help='ES9+ (other) notification')
+parser_ntf.add_argument('operation', choices=['enable','disable','delete'],
+                        help='Profile Management Operation whoise occurrence shall be notififed')
+parser_ntf.add_argument('--sequence-nr', type=int, required=True,
+                        help='eUICC global notification sequence number')
+parser_ntf.add_argument('--notification-address', help='notificationAddress, if different from URL')
+parser_ntf.add_argument('--iccid', type=is_hexstr, help='ICCID to which the notification relates')
+
+# notification-install
+parser_ntfi = subparsers.add_parser('notification-install', help='ES9+ installation notification')
+parser_ntfi.add_argument('--sequence-nr', type=int, required=True,
+                         help='eUICC global notification sequence number')
+parser_ntfi.add_argument('--transaction-id', required=True,
+                         help='transactionId of previous ES9+ download')
+parser_ntfi.add_argument('--notification-address', help='notificationAddress, if different from URL')
+parser_ntfi.add_argument('--iccid', type=is_hexstr, help='ICCID to which the notification relates')
+parser_ntfi.add_argument('--smdpp-oid', required=True, help='SM-DP+ OID (as in CERT.DPpb.ECDSA)')
+parser_ntfi.add_argument('--isdp-aid', type=is_hexstr, required=True,
+                         help='AID of the ISD-P of the installed profile')
+parser_ntfi.add_argument('--sima-response', type=is_hexstr, required=True,
+                         help='hex digits of BER-encoded SAIP EUICCResponse')
+
+class Es9pClient:
+    def __init__(self, opts):
+        self.opts = opts
+        self.cert_and_key = CertAndPrivkey()
+        self.cert_and_key.cert_from_der_file(os.path.join(opts.certificate_path, opts.euicc_certificate))
+        self.cert_and_key.privkey_from_pem_file(os.path.join(opts.certificate_path, opts.euicc_private_key))
+
+        with open(os.path.join(opts.certificate_path, opts.eum_certificate), 'rb') as f:
+            self.eum_cert = x509.load_der_x509_certificate(f.read())
+
+        with open(os.path.join(opts.certificate_path, opts.ci_certificate), 'rb') as f:
+            self.ci_cert = x509.load_der_x509_certificate(f.read())
+            subject_exts = list(filter(lambda x: isinstance(x.value, x509.SubjectKeyIdentifier), self.ci_cert.extensions))
+            subject_pkid = subject_exts[0].value
+            self.ci_pkid = subject_pkid.key_identifier
+
+        print("EUICC: %s" % self.cert_and_key.cert.subject)
+        print("EUM: %s" % self.eum_cert.subject)
+        print("CI: %s" % self.ci_cert.subject)
+
+        self.eid = self.cert_and_key.cert.subject.get_attributes_for_oid(x509.oid.NameOID.SERIAL_NUMBER)[0].value
+        print("EID: %s" % self.eid)
+        print("CI PKID: %s" % b2h(self.ci_pkid))
+        print()
+
+        self.peer = es9p.Es9pApiClient(opts.url, server_cert_verify=opts.server_ca_cert)
+
+
+    def do_notification(self):
+
+        ntf_metadata = {
+            'seqNumber': self.opts.sequence_nr,
+            'profileManagementOperation': PMO(self.opts.operation).to_bitstring(),
+            'notificationAddress': self.opts.notification_address or urlparse(self.opts.url).netloc,
+        }
+        if self.opts.iccid:
+            ntf_metadata['iccid'] = h2b(swap_nibbles(self.opts.iccid))
+
+        if self.opts.operation == 'install':
+            pird = {
+                'transactionId': h2b(self.opts.transaction_id),
+                'notificationMetadata': ntf_metadata,
+                'smdpOid': self.opts.smdpp_oid,
+                'finalResult': ('successResult', {
+                    'aid': h2b(self.opts.isdp_aid),
+                    'simaResponse': h2b(self.opts.sima_response),
+                    }),
+            }
+            pird_bin = rsp.asn1.encode('ProfileInstallationResultData', pird)
+            signature = self.cert_and_key.ecdsa_sign(pird_bin)
+            pn_dict = ('profileInstallationResult', {
+                'profileInstallationResultData': pird,
+                'euiccSignPIR': signature,
+            })
+        else:
+            ntf_bin = rsp.asn1.encode('NotificationMetadata', ntf_metadata)
+            signature = self.cert_and_key.ecdsa_sign(ntf_bin)
+            pn_dict = ('otherSignedNotification', {
+                'tbsOtherNotification': ntf_metadata,
+                'euiccNotificationSignature': signature,
+                'euiccCertificate': rsp.asn1.decode('Certificate', self.cert_and_key.get_cert_as_der()),
+                'eumCertificate': rsp.asn1.decode('Certificate', self.eum_cert.public_bytes(Encoding.DER)),
+            })
+
+        data = {
+            'pendingNotification': pn_dict,
+            }
+        #print(data)
+        res = self.peer.call_handleNotification(data)
+
+
+    def do_download(self):
+
+        print("Step 1: InitiateAuthentication...")
+
+        euiccInfo1 = {
+                'svn': b'\x02\x04\x00',
+                'euiccCiPKIdListForVerification': [
+                    self.ci_pkid,
+                ],
+                'euiccCiPKIdListForSigning': [
+                    self.ci_pkid,
+                ],
+          }
+
+        data = {
+            'euiccChallenge': os.urandom(16),
+            'euiccInfo1': euiccInfo1,
+            'smdpAddress': urlparse(self.opts.url).netloc,
+          }
+        init_auth_res = self.peer.call_initiateAuthentication(data)
+        print(init_auth_res)
+
+        print("Step 2: AuthenticateClient...")
+
+        #res['serverSigned1']
+        #res['serverSignature1']
+        print("TODO: verify serverSignature1 over serverSigned1")
+        #res['transactionId']
+        print("TODO: verify transactionId matches the signed one in serverSigned1")
+        #res['euiccCiPKIdToBeUsed']
+        # TODO: select eUICC certificate based on CI
+        #res['serverCertificate']
+        # TODO: verify server certificate against CI
+
+        euiccInfo2 = {
+            'profileVersion': b'\x02\x03\x01',
+            'svn': euiccInfo1['svn'],
+            'euiccFirmwareVer': b'\x23\x42\x00',
+            'extCardResource': b'\x81\x01\x00\x82\x04\x00\x04\x9ch\x83\x02"#',
+            'uiccCapability': (b'k6\xd3\xc3', 32),
+            'javacardVersion': b'\x11\x02\x00',
+            'globalplatformVersion': b'\x02\x03\x00',
+            'rspCapability': (b'\x9c', 6),
+            'euiccCiPKIdListForVerification': euiccInfo1['euiccCiPKIdListForVerification'],
+            'euiccCiPKIdListForSigning': euiccInfo1['euiccCiPKIdListForSigning'],
+            #'euiccCategory':
+            #'forbiddenProfilePolicyRules':
+            'ppVersion': b'\x01\x00\x00',
+            'sasAcreditationNumber': 'OSMOCOM-TEST-1', #TODO: make configurable
+            #'certificationDataObject':
+        }
+
+        euiccSigned1 = {
+            'transactionId': h2b(init_auth_res['transactionId']),
+            'serverAddress': init_auth_res['serverSigned1']['serverAddress'],
+            'serverChallenge': init_auth_res['serverSigned1']['serverChallenge'],
+            'euiccInfo2': euiccInfo2,
+            'ctxParams1':
+                ('ctxParamsForCommonAuthentication', {
+                    'matchingId': self.opts.matchingId,
+                    'deviceInfo': {
+                        'tac': b'\x35\x23\x01\x45', # same as lpac
+                        'deviceCapabilities': {},
+                        #imei:
+                    }
+                }),
+        }
+        euiccSigned1_bin = rsp.asn1.encode('EuiccSigned1', euiccSigned1)
+        euiccSignature1 = self.cert_and_key.ecdsa_sign(euiccSigned1_bin)
+        auth_clnt_req = {
+            'transactionId': init_auth_res['transactionId'],
+            'authenticateServerResponse':
+                ('authenticateResponseOk', {
+                    'euiccSigned1': euiccSigned1,
+                    'euiccSignature1': euiccSignature1,
+                    'euiccCertificate': rsp.asn1.decode('Certificate', self.cert_and_key.get_cert_as_der()),
+                    'eumCertificate': rsp.asn1.decode('Certificate', self.eum_cert.public_bytes(Encoding.DER))
+                 })
+        }
+        auth_clnt_res = self.peer.call_authenticateClient(auth_clnt_req)
+        print(auth_clnt_res)
+        #auth_clnt_res['transactionId']
+        print("TODO: verify transactionId matches previous ones")
+        #auth_clnt_res['profileMetadata']
+        # TODO: what's in here?
+        #auth_clnt_res['smdpSigned2']['bppEuiccOtpk']
+        #auth_clnt_res['smdpSignature2']
+        print("TODO: verify serverSignature2 over smdpSigned2")
+
+        smdp_cert = x509.load_der_x509_certificate(auth_clnt_res['smdpCertificate'])
+
+        print("Step 3: GetBoundProfilePackage...")
+        # Generate a one-time ECKA key pair (ot{PK,SK}.DP.ECKA) using the curve indicated by the Key Parameter
+        # Reference value of CERT.DPpb.ECDSA
+        euicc_ot = ec.generate_private_key(smdp_cert.public_key().public_numbers().curve)
+
+        # extract the public key in (hopefully) the right format for the ES8+ interface
+        euicc_otpk = euicc_ot.public_key().public_bytes(Encoding.X962, PublicFormat.UncompressedPoint)
+
+        euiccSigned2 = {
+            'transactionId': h2b(auth_clnt_res['transactionId']),
+            'euiccOtpk': euicc_otpk,
+            #hashCC
+        }
+        # check for smdpSigned2 ccRequiredFlag, and send it in PrepareDownloadRequest hashCc
+        if auth_clnt_res['smdpSigned2']['ccRequiredFlag']:
+            if not self.opts.confirmation_code:
+                raise ValueError('Confirmation Code required but not provided')
+            cc_hash = hashlib.sha256(self.opts.confirmation_code.encode('ascii')).digest()
+            euiccSigned2['hashCc'] = hashlib.sha256(cc_hash + euiccSigned2['transactionId']).digest()
+        euiccSigned2_bin = rsp.asn1.encode('EUICCSigned2', euiccSigned2)
+        euiccSignature2 = self.cert_and_key.ecdsa_sign(euiccSigned2_bin + auth_clnt_res['smdpSignature2'])
+        gbp_req = {
+            'transactionId': auth_clnt_res['transactionId'],
+            'prepareDownloadResponse':
+                ('downloadResponseOk', {
+                    'euiccSigned2': euiccSigned2,
+                    'euiccSignature2': euiccSignature2,
+                 })
+        }
+        gbp_res = self.peer.call_getBoundProfilePackage(gbp_req)
+        print(gbp_res)
+        #gbp_res['transactionId']
+        # TODO: verify transactionId
+        print("TODO: verify transactionId matches previous ones")
+        bpp_bin = gbp_res['boundProfilePackage']
+        print("TODO: verify boundProfilePackage smdpSignature")
+
+        bpp = BoundProfilePackage()
+        upp_bin = bpp.decode(euicc_ot, self.eid, bpp_bin)
+
+        iccid = swap_nibbles(b2h(bpp.storeMetadataRequest['iccid']))
+        base_name = os.path.join(self.opts.output_path, '%s' % iccid)
+
+        print("SUCCESS: Storing files as %s.*.der" % base_name)
+
+        # write various output files
+        with open(base_name+'.upp.der', 'wb') as f:
+            f.write(bpp.upp)
+        with open(base_name+'.isdp.der', 'wb') as f:
+            f.write(bpp.encoded_configureISDPRequest)
+        with open(base_name+'.smr.der', 'wb') as f:
+            f.write(bpp.encoded_storeMetadataRequest)
+
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+
+    c = Es9pClient(opts)
+
+    if opts.command == 'download':
+        c.do_download()
+    elif opts.command == 'notification':
+        c.do_notification()
+    elif opts.command == 'notification-install':
+        opts.operation = 'install'
+        c.do_notification()

contrib/esim-qrcode-gen.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# Small command line utility program to encode eSIM QR-Codes
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+import argparse
+
+from pySim.esim import ActivationCode
+
+
+option_parser = argparse.ArgumentParser(description="""
+eSIM QR code generator. Will encode the given hostname + activation code
+into the eSIM RSP String format as specified in SGP.22 Section 4.1.  If
+a PNG output file is specified, it will also generate a QR code.""")
+option_parser.add_argument('hostname', help='FQDN of SM-DP+')
+option_parser.add_argument('token', help='MatchingID / Token')
+option_parser.add_argument('--oid', help='SM-DP+ OID in CERT.DPauth.ECDSA')
+option_parser.add_argument('--confirmation-code-required', action='store_true',
+                           help='Whether a Confirmation Code is required')
+option_parser.add_argument('--png', help='Output PNG file name (no PNG is written if omitted)')
+
+
+
+if __name__ == '__main__':
+    opts = option_parser.parse_args()
+
+    ac = ActivationCode(opts.hostname, opts.token, opts.oid, opts.confirmation_code_required)
+    print(ac.to_string())
+    if opts.png:
+        with open(opts.png, 'wb') as f:
+            img = ac.to_qrcode()
+            img.save(f)
+            print("# generated QR code stored to '%s'" % (opts.png))

contrib/esim_gen_metadata.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+
+# (C) 2025 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+from osmocom.utils import h2b, swap_nibbles
+from pySim.esim.es8p import ProfileMetadata
+
+parser = argparse.ArgumentParser(description="""Utility program to generate profile metadata in the
+StoreMetadataRequest format based on input values from the command line.""")
+parser.add_argument('--iccid', required=True, help="ICCID of eSIM profile");
+parser.add_argument('--spn', required=True, help="Service Provider Name");
+parser.add_argument('--profile-name', required=True, help="eSIM Profile Name");
+parser.add_argument('--profile-class', choices=['test', 'operational', 'provisioning'],
+                    default='operational', help="Profile Class");
+parser.add_argument('--outfile', required=True, help="Output File Name");
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+
+    iccid_bin = h2b(swap_nibbles(opts.iccid))
+    pmd = ProfileMetadata(iccid_bin, spn=opts.spn, profile_name=opts.profile_name,
+                          profile_class=opts.profile_class)
+
+    with open(opts.outfile, 'wb') as f:
+        f.write(pmd.gen_store_metadata_request())
+    print("Written StoreMetadataRequest to '%s'" % opts.outfile)

contrib/fsdump-diff-apply.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+
+# The purpose of this script is to
+# * load two SIM card 'fsdump' files
+# * determine which file contents in "B" differs from that of "A"
+# * create a pySim-shell script to update the contents of "A" to match that of "B"
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import argparse
+
+# Files that we should not update
+FILES_TO_SKIP = [
+    "MF/EF.ICCID",
+    #"MF/DF.GSM/EF.IMSI",
+    #"MF/ADF.USIM/EF.IMSI",
+    ]
+
+# Files that need zero-padding at the end, not ff-padding
+FILES_PAD_ZERO = [
+    "DF.GSM/EF.SST",
+    "MF/ADF.USIM/EF.UST",
+    "MF/ADF.USIM/EF.EST",
+    "MF/ADF.ISIM/EF.IST",
+    ]
+
+def pad_file(path, instr, byte_len):
+    if path in FILES_PAD_ZERO:
+        pad = '0'
+    else:
+        pad = 'f'
+    return pad_hexstr(instr, byte_len, pad)
+
+def pad_hexstr(instr, byte_len:int, pad='f'):
+    """Pad given hex-string to the number of bytes given in byte_len, using ff as padding."""
+    if len(instr) == byte_len*2:
+        return instr
+    elif len(instr) > byte_len*2:
+        raise ValueError('Cannot pad string of length %u to smaller length %u' % (len(instr)/2, byte_len))
+    else:
+        return instr + pad * (byte_len*2 - len(instr))
+
+def is_all_ff(instr):
+    """Determine if the entire input hex-string consists of f-digits."""
+    if all([x == 'f' for x in instr.lower()]):
+        return True
+    else:
+        return False
+
+parser = argparse.ArgumentParser()
+parser.add_argument('file_a')
+parser.add_argument('file_b')
+
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+
+    with open(opts.file_a, 'r') as file_a:
+        json_a = json.loads(file_a.read())
+    with open(opts.file_b, 'r') as file_b:
+        json_b = json.loads(file_b.read())
+
+    for path in json_b.keys():
+        print()
+        print("# %s" % path)
+
+        if not path in json_a:
+            raise ValueError("%s doesn't exist in file_a!" % path)
+
+        if path in FILES_TO_SKIP:
+            print("# skipped explicitly as it is in FILES_TO_SKIP")
+            continue
+
+        if not 'body' in json_b[path]:
+            print("# file doesn't exist in B so we cannot possibly need to modify A")
+            continue
+
+        if not 'body' in json_a[path]:
+            # file was not readable in original (permissions? deactivated?)
+            print("# ERROR: %s not readable in A; please fix that" % path)
+            continue
+
+        body_a = json_a[path]['body']
+        body_b = json_b[path]['body']
+        if body_a == body_b:
+            print("# file body is identical")
+            continue
+
+        file_size_a = json_a[path]['fcp']['file_size']
+        file_size_b = json_b[path]['fcp']['file_size']
+
+        cmds = []
+        structure = json_b[path]['fcp']['file_descriptor']['file_descriptor_byte']['structure']
+        if structure == 'transparent':
+            val_a = body_a
+            val_b = body_b
+            if file_size_a < file_size_b:
+                if not is_all_ff(val_b[2*file_size_a:]):
+                    print("# ERROR: file_size_a (%u) < file_size_b (%u); please fix!" % (file_size_a, file_size_b))
+                    continue
+                else:
+                    print("# WARN: file_size_a (%u) < file_size_b (%u); please fix!" % (file_size_a, file_size_b))
+                    # truncate val_b to fit in A
+                    val_b = val_b[:file_size_a*2]
+
+            elif file_size_a != file_size_b:
+                print("# NOTE: file_size_a (%u) != file_size_b (%u)" % (file_size_a, file_size_b))
+
+            # Pad to file_size_a
+            val_b = pad_file(path, val_b, file_size_a)
+            if val_b != val_a:
+                cmds.append("update_binary %s" % val_b)
+            else:
+                print("# padded file body is identical")
+        elif structure in ['linear_fixed', 'cyclic']:
+            record_len_a = json_a[path]['fcp']['file_descriptor']['record_len']
+            record_len_b = json_b[path]['fcp']['file_descriptor']['record_len']
+            if record_len_a < record_len_b:
+                print("# ERROR: record_len_a (%u) < record_len_b (%u); please fix!" % (file_size_a, file_size_b))
+                continue
+            elif record_len_a != record_len_b:
+                print("# NOTE: record_len_a (%u) != record_len_b (%u)" % (record_len_a, record_len_b))
+
+            num_rec_a = file_size_a // record_len_a
+            num_rec_b = file_size_b // record_len_b
+            if num_rec_a < num_rec_b:
+                if not all([is_all_ff(x) for x in body_b[num_rec_a:]]):
+                    print("# ERROR: num_rec_a (%u) < num_rec_b (%u); please fix!" % (num_rec_a, num_rec_b))
+                    continue
+                else:
+                    print("# WARN: num_rec_a (%u) < num_rec_b (%u); but they're empty" % (num_rec_a, num_rec_b))
+            elif num_rec_a != num_rec_b:
+                print("# NOTE: num_rec_a (%u) != num_rec_b (%u)" % (num_rec_a, num_rec_b))
+
+            i = 0
+            for r in body_b:
+                if i < len(body_a):
+                    break
+                val_a = body_a[i]
+                # Pad to record_len_a
+                val_b = pad_file(path, body_b[i], record_len_a)
+                if val_a != val_b:
+                    cmds.append("update_record %u %s" % (i+1, val_b))
+                i = i + 1
+            if len(cmds) == 0:
+                print("# padded file body is identical")
+        elif structure == 'ber_tlv':
+            print("# FIXME: Implement BER-TLV")
+        else:
+            raise ValueError('Unsupported structure %s' % structure)
+
+        if len(cmds):
+            print("select %s" % path)
+            for cmd in cmds:
+                print(cmd)

contrib/generate_smdpp_certs.py

@@ -0,0 +1,661 @@
+#!/usr/bin/env python3
+
+"""
+Faithfully reproduces the smdpp certs contained in SGP.26_v1.5_Certificates_18_07_2024.zip
+available at https://www.gsma.com/solutions-and-impact/technologies/esim/gsma_resources/sgp-26-test-certificate-definition-v1-5/
+Only usable for testing, it obviously uses a different CI key.
+"""
+
+import os
+import binascii
+from datetime import datetime
+from cryptography import x509
+from cryptography.x509.oid import NameOID, ExtensionOID
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.backends import default_backend
+
+# Custom OIDs used in certificates
+OID_CERTIFICATE_POLICIES_CI = "2.23.146.1.2.1.0"  # CI cert policy
+OID_CERTIFICATE_POLICIES_TLS = "2.23.146.1.2.1.3"  # DPtls cert policy
+OID_CERTIFICATE_POLICIES_AUTH = "2.23.146.1.2.1.4"  # DPauth cert policy
+OID_CERTIFICATE_POLICIES_PB = "2.23.146.1.2.1.5"  # DPpb cert policy
+
+# Subject Alternative Name OIDs
+OID_CI_RID = "2.999.1"  # CI Registered ID
+OID_DP_RID = "2.999.10"  # DP+ Registered ID
+OID_DP2_RID = "2.999.12"  # DP+2 Registered ID
+OID_DP4_RID = "2.999.14"  # DP+4 Registered ID
+OID_DP8_RID = "2.999.18"  # DP+8 Registered ID
+
+
+class SimplifiedCertificateGenerator:
+    def __init__(self):
+        self.backend = default_backend()
+        # Store generated CI keys to sign other certs
+        self.ci_certs = {}  # {"BRP": cert, "NIST": cert}
+        self.ci_keys = {}   # {"BRP": key, "NIST": key}
+
+    def get_curve(self, curve_type):
+        """Get the appropriate curve object."""
+        if curve_type == "BRP":
+            return ec.BrainpoolP256R1()
+        else:
+            return ec.SECP256R1()
+
+    def generate_key_pair(self, curve):
+        """Generate a new EC key pair."""
+        private_key = ec.generate_private_key(curve, self.backend)
+        return private_key
+
+    def load_private_key_from_hex(self, hex_key, curve):
+        """Load EC private key from hex string."""
+        key_bytes = binascii.unhexlify(hex_key.replace(":", "").replace(" ", "").replace("\n", ""))
+        key_int = int.from_bytes(key_bytes, 'big')
+        return ec.derive_private_key(key_int, curve, self.backend)
+
+    def generate_ci_cert(self, curve_type):
+        """Generate CI certificate for either BRP or NIST curve."""
+        curve = self.get_curve(curve_type)
+        private_key = self.generate_key_pair(curve)
+
+        # Build subject and issuer (self-signed) - same for both
+        subject = issuer = x509.Name([
+            x509.NameAttribute(NameOID.COMMON_NAME, "Test CI"),
+            x509.NameAttribute(NameOID.ORGANIZATIONAL_UNIT_NAME, "TESTCERT"),
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "RSPTEST"),
+            x509.NameAttribute(NameOID.COUNTRY_NAME, "IT"),
+        ])
+
+        # Build certificate - all parameters same for both
+        builder = x509.CertificateBuilder()
+        builder = builder.subject_name(subject)
+        builder = builder.issuer_name(issuer)
+        builder = builder.not_valid_before(datetime(2020, 4, 1, 8, 27, 51))
+        builder = builder.not_valid_after(datetime(2055, 4, 1, 8, 27, 51))
+        builder = builder.serial_number(0xb874f3abfa6c44d3)
+        builder = builder.public_key(private_key.public_key())
+
+        # Add extensions - all same for both
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.BasicConstraints(ca=True, path_length=None),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CertificatePolicies([
+                x509.PolicyInformation(
+                    x509.ObjectIdentifier(OID_CERTIFICATE_POLICIES_CI),
+                    policy_qualifiers=None
+                )
+            ]),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.KeyUsage(
+                digital_signature=False,
+                content_commitment=False,
+                key_encipherment=False,
+                data_encipherment=False,
+                key_agreement=False,
+                key_cert_sign=True,
+                crl_sign=True,
+                encipher_only=False,
+                decipher_only=False
+            ),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectAlternativeName([
+                x509.RegisteredID(x509.ObjectIdentifier(OID_CI_RID))
+            ]),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.CRLDistributionPoints([
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-A.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                ),
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-B.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                )
+            ]),
+            critical=False
+        )
+
+        certificate = builder.sign(private_key, hashes.SHA256(), self.backend)
+
+        self.ci_keys[curve_type] = private_key
+        self.ci_certs[curve_type] = certificate
+
+        return certificate, private_key
+
+    def generate_dp_cert(self, curve_type, subject_cn, serial, key_hex,
+                        cert_policy_oid, rid_oid, validity_start, validity_end):
+        """Generate a DP certificate signed by CI - works for both BRP and NIST."""
+        curve = self.get_curve(curve_type)
+        private_key = self.load_private_key_from_hex(key_hex, curve)
+
+        ci_cert = self.ci_certs[curve_type]
+        ci_key = self.ci_keys[curve_type]
+
+        subject = x509.Name([
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "ACME"),
+            x509.NameAttribute(NameOID.COMMON_NAME, subject_cn),
+        ])
+
+        builder = x509.CertificateBuilder()
+        builder = builder.subject_name(subject)
+        builder = builder.issuer_name(ci_cert.subject)
+        builder = builder.not_valid_before(validity_start)
+        builder = builder.not_valid_after(validity_end)
+        builder = builder.serial_number(serial)
+        builder = builder.public_key(private_key.public_key())
+
+        builder = builder.add_extension(
+            x509.AuthorityKeyIdentifier.from_issuer_public_key(ci_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectAlternativeName([
+                x509.RegisteredID(x509.ObjectIdentifier(rid_oid))
+            ]),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.KeyUsage(
+                digital_signature=True,
+                content_commitment=False,
+                key_encipherment=False,
+                data_encipherment=False,
+                key_agreement=False,
+                key_cert_sign=False,
+                crl_sign=False,
+                encipher_only=False,
+                decipher_only=False
+            ),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CertificatePolicies([
+                x509.PolicyInformation(
+                    x509.ObjectIdentifier(cert_policy_oid),
+                    policy_qualifiers=None
+                )
+            ]),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CRLDistributionPoints([
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-A.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                ),
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-B.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                )
+            ]),
+            critical=False
+        )
+
+        certificate = builder.sign(ci_key, hashes.SHA256(), self.backend)
+
+        return certificate, private_key
+
+    def generate_tls_cert(self, curve_type, subject_cn, dns_name, serial, key_hex,
+                         rid_oid, validity_start, validity_end):
+        """Generate a TLS certificate signed by CI."""
+        curve = self.get_curve(curve_type)
+        private_key = self.load_private_key_from_hex(key_hex, curve)
+
+        ci_cert = self.ci_certs[curve_type]
+        ci_key = self.ci_keys[curve_type]
+
+        subject = x509.Name([
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "ACME"),
+            x509.NameAttribute(NameOID.COMMON_NAME, subject_cn),
+        ])
+
+        builder = x509.CertificateBuilder()
+        builder = builder.subject_name(subject)
+        builder = builder.issuer_name(ci_cert.subject)
+        builder = builder.not_valid_before(validity_start)
+        builder = builder.not_valid_after(validity_end)
+        builder = builder.serial_number(serial)
+        builder = builder.public_key(private_key.public_key())
+
+        builder = builder.add_extension(
+            x509.KeyUsage(
+                digital_signature=True,
+                content_commitment=False,
+                key_encipherment=False,
+                data_encipherment=False,
+                key_agreement=False,
+                key_cert_sign=False,
+                crl_sign=False,
+                encipher_only=False,
+                decipher_only=False
+            ),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.ExtendedKeyUsage([
+                x509.oid.ExtendedKeyUsageOID.SERVER_AUTH,
+                x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH
+            ]),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CertificatePolicies([
+                x509.PolicyInformation(
+                    x509.ObjectIdentifier(OID_CERTIFICATE_POLICIES_TLS),
+                    policy_qualifiers=None
+                )
+            ]),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.AuthorityKeyIdentifier.from_issuer_public_key(ci_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectAlternativeName([
+                x509.DNSName(dns_name),
+                x509.RegisteredID(x509.ObjectIdentifier(rid_oid))
+            ]),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.CRLDistributionPoints([
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-A.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                ),
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-B.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                )
+            ]),
+            critical=False
+        )
+
+        certificate = builder.sign(ci_key, hashes.SHA256(), self.backend)
+
+        return certificate, private_key
+
+    def generate_eum_cert(self, curve_type, key_hex):
+        """Generate EUM certificate signed by CI."""
+        curve = self.get_curve(curve_type)
+        private_key = self.load_private_key_from_hex(key_hex, curve)
+
+        ci_cert = self.ci_certs[curve_type]
+        ci_key = self.ci_keys[curve_type]
+
+        subject = x509.Name([
+            x509.NameAttribute(NameOID.COUNTRY_NAME, "ES"),
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "RSP Test EUM"),
+            x509.NameAttribute(NameOID.COMMON_NAME, "EUM Test"),
+        ])
+
+        builder = x509.CertificateBuilder()
+        builder = builder.subject_name(subject)
+        builder = builder.issuer_name(ci_cert.subject)
+        builder = builder.not_valid_before(datetime(2020, 4, 1, 9, 28, 37))
+        builder = builder.not_valid_after(datetime(2054, 3, 24, 9, 28, 37))
+        builder = builder.serial_number(0x12345678)
+        builder = builder.public_key(private_key.public_key())
+
+        builder = builder.add_extension(
+            x509.AuthorityKeyIdentifier.from_issuer_public_key(ci_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.KeyUsage(
+                digital_signature=False,
+                content_commitment=False,
+                key_encipherment=False,
+                data_encipherment=False,
+                key_agreement=False,
+                key_cert_sign=True,
+                crl_sign=False,
+                encipher_only=False,
+                decipher_only=False
+            ),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CertificatePolicies([
+                x509.PolicyInformation(
+                    x509.ObjectIdentifier("2.23.146.1.2.1.2"),  # EUM policy
+                    policy_qualifiers=None
+                )
+            ]),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectAlternativeName([
+                x509.RegisteredID(x509.ObjectIdentifier("2.999.5"))
+            ]),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.BasicConstraints(ca=True, path_length=0),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CRLDistributionPoints([
+                x509.DistributionPoint(
+                    full_name=[x509.UniformResourceIdentifier("http://ci.test.example.com/CRL-B.crl")],
+                    relative_name=None,
+                    reasons=None,
+                    crl_issuer=None
+                )
+            ]),
+            critical=False
+        )
+
+        # Name Constraints
+        constrained_name = x509.Name([
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "RSP Test EUM"),
+            x509.NameAttribute(NameOID.SERIAL_NUMBER, "89049032"),
+        ])
+
+        name_constraints = x509.NameConstraints(
+            permitted_subtrees=[
+                x509.DirectoryName(constrained_name)
+            ],
+            excluded_subtrees=None
+        )
+
+        builder = builder.add_extension(
+            name_constraints,
+            critical=True
+        )
+
+        certificate = builder.sign(ci_key, hashes.SHA256(), self.backend)
+
+        return certificate, private_key
+
+    def generate_euicc_cert(self, curve_type, eum_cert, eum_key, key_hex):
+        """Generate eUICC certificate signed by EUM."""
+        curve = self.get_curve(curve_type)
+        private_key = self.load_private_key_from_hex(key_hex, curve)
+
+        subject = x509.Name([
+            x509.NameAttribute(NameOID.COUNTRY_NAME, "ES"),
+            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "RSP Test EUM"),
+            x509.NameAttribute(NameOID.SERIAL_NUMBER, "89049032123451234512345678901235"),
+            x509.NameAttribute(NameOID.COMMON_NAME, "Test eUICC"),
+        ])
+
+        builder = x509.CertificateBuilder()
+        builder = builder.subject_name(subject)
+        builder = builder.issuer_name(eum_cert.subject)
+        builder = builder.not_valid_before(datetime(2020, 4, 1, 9, 48, 58))
+        builder = builder.not_valid_after(datetime(7496, 1, 24, 9, 48, 58))
+        builder = builder.serial_number(0x0200000000000001)
+        builder = builder.public_key(private_key.public_key())
+
+        builder = builder.add_extension(
+            x509.AuthorityKeyIdentifier.from_issuer_public_key(eum_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
+
+        builder = builder.add_extension(
+            x509.KeyUsage(
+                digital_signature=True,
+                content_commitment=False,
+                key_encipherment=False,
+                data_encipherment=False,
+                key_agreement=False,
+                key_cert_sign=False,
+                crl_sign=False,
+                encipher_only=False,
+                decipher_only=False
+            ),
+            critical=True
+        )
+
+        builder = builder.add_extension(
+            x509.CertificatePolicies([
+                x509.PolicyInformation(
+                    x509.ObjectIdentifier("2.23.146.1.2.1.1"),  # eUICC policy
+                    policy_qualifiers=None
+                )
+            ]),
+            critical=True
+        )
+
+        certificate = builder.sign(eum_key, hashes.SHA256(), self.backend)
+
+        return certificate, private_key
+
+    def save_cert_and_key(self, cert, key, cert_path_der, cert_path_pem, key_path_sk, key_path_pk):
+        """Save certificate and key in various formats."""
+        # Create directories if needed
+        os.makedirs(os.path.dirname(cert_path_der), exist_ok=True)
+
+        with open(cert_path_der, "wb") as f:
+            f.write(cert.public_bytes(serialization.Encoding.DER))
+
+        if cert_path_pem:
+            with open(cert_path_pem, "wb") as f:
+                f.write(cert.public_bytes(serialization.Encoding.PEM))
+
+        if key and key_path_sk:
+            with open(key_path_sk, "wb") as f:
+                f.write(key.private_bytes(
+                    encoding=serialization.Encoding.PEM,
+                    format=serialization.PrivateFormat.TraditionalOpenSSL,
+                    encryption_algorithm=serialization.NoEncryption()
+                ))
+
+        if key and key_path_pk:
+            with open(key_path_pk, "wb") as f:
+                f.write(key.public_key().public_bytes(
+                    encoding=serialization.Encoding.PEM,
+                    format=serialization.PublicFormat.SubjectPublicKeyInfo
+                ))
+
+
+def main():
+    gen = SimplifiedCertificateGenerator()
+
+    output_dir = "smdpp-data/generated"
+    os.makedirs(output_dir, exist_ok=True)
+
+    print("=== Generating CI Certificates ===")
+
+    for curve_type in ["BRP", "NIST"]:
+        ci_cert, ci_key = gen.generate_ci_cert(curve_type)
+        suffix = "_ECDSA_BRP" if curve_type == "BRP" else "_ECDSA_NIST"
+        gen.save_cert_and_key(
+            ci_cert, ci_key,
+            f"{output_dir}/CertificateIssuer/CERT_CI{suffix}.der",
+            f"{output_dir}/CertificateIssuer/CERT_CI{suffix}.pem",
+            None, None
+        )
+        print(f"Generated CI {curve_type} certificate")
+
+    print("\n=== Generating DPauth Certificates ===")
+
+    dpauth_configs = [
+        ("BRP", "TEST SM-DP+", 256, "93:fb:33:d0:58:4f:34:9b:07:f8:b5:d2:af:93:d7:c3:e3:54:b3:49:a3:b9:13:50:2e:6a:bc:07:0e:4d:49:29", OID_DP_RID, "DPauth"),
+        ("NIST", "TEST SM-DP+", 256, "0a:7c:c1:c2:44:e6:0c:52:cd:5b:78:07:ab:8c:36:0c:26:52:46:01:50:7d:ca:bc:5d:d5:98:b5:a6:16:d5:d5", OID_DP_RID, "DPauth"),
+        ("BRP", "TEST SM-DP+2", 512, "0c:17:35:5c:01:1d:0f:e8:d7:da:dd:63:f1:97:85:cf:6c:51:cb:cd:46:6a:e8:8b:e8:f8:1b:c1:05:88:46:f6", OID_DP2_RID, "DP2auth"),
+        ("NIST", "TEST SM-DP+2", 512, "9c:32:a0:95:d4:88:42:d9:ff:a4:04:f7:12:51:2a:a2:c5:42:5a:1a:26:38:6a:b6:a1:45:d5:81:1e:03:91:41", OID_DP2_RID, "DP2auth"),
+    ]
+
+    for curve_type, cn, serial, key_hex, rid_oid, name_prefix in dpauth_configs:
+        cert, key = gen.generate_dp_cert(
+            curve_type, cn, serial, key_hex,
+            OID_CERTIFICATE_POLICIES_AUTH, rid_oid,
+            datetime(2020, 4, 1, 8, 31, 30),
+            datetime(2030, 3, 30, 8, 31, 30)
+        )
+        suffix = "_ECDSA_BRP" if curve_type == "BRP" else "_ECDSA_NIST"
+        gen.save_cert_and_key(
+            cert, key,
+            f"{output_dir}/DPauth/CERT_S_SM_{name_prefix}{suffix}.der",
+            None,
+            f"{output_dir}/DPauth/SK_S_SM_{name_prefix}{suffix}.pem",
+            f"{output_dir}/DPauth/PK_S_SM_{name_prefix}{suffix}.pem"
+        )
+        print(f"Generated {name_prefix} {curve_type} certificate")
+
+    print("\n=== Generating DPpb Certificates ===")
+
+    dppb_configs = [
+        ("BRP", "TEST SM-DP+", 257, "75:ff:32:2f:41:66:16:da:e1:a4:84:ef:71:d4:87:4f:b0:df:32:95:fd:35:c2:cb:a4:89:fb:b2:bb:9c:7b:f6", OID_DP_RID, "DPpb"),
+        ("NIST", "TEST SM-DP+", 257, "dc:d6:94:b7:78:95:7e:8e:9a:dd:bd:d9:44:33:e9:ef:8f:73:d1:1e:49:1c:48:d4:25:a3:8a:94:91:bd:3b:ed", OID_DP_RID, "DPpb"),
+        ("BRP", "TEST SM-DP+2", 513, "9c:ae:2e:1a:56:07:a9:d5:78:38:2e:ee:93:2e:25:1f:52:30:4f:86:ee:b1:f1:70:8c:db:d3:c0:7b:e2:cd:3d", OID_DP2_RID, "DP2pb"),
+        ("NIST", "TEST SM-DP+2", 513, "66:93:11:49:63:9d:ba:ac:1d:c3:d3:06:c5:8b:d2:df:d2:2f:73:bf:63:ac:86:31:98:32:90:b5:7f:90:93:45", OID_DP2_RID, "DP2pb"),
+    ]
+
+    for curve_type, cn, serial, key_hex, rid_oid, name_prefix in dppb_configs:
+        cert, key = gen.generate_dp_cert(
+            curve_type, cn, serial, key_hex,
+            OID_CERTIFICATE_POLICIES_PB, rid_oid,
+            datetime(2020, 4, 1, 8, 34, 46),
+            datetime(2030, 3, 30, 8, 34, 46)
+        )
+        suffix = "_ECDSA_BRP" if curve_type == "BRP" else "_ECDSA_NIST"
+        gen.save_cert_and_key(
+            cert, key,
+            f"{output_dir}/DPpb/CERT_S_SM_{name_prefix}{suffix}.der",
+            None,
+            f"{output_dir}/DPpb/SK_S_SM_{name_prefix}{suffix}.pem",
+            f"{output_dir}/DPpb/PK_S_SM_{name_prefix}{suffix}.pem"
+        )
+        print(f"Generated {name_prefix} {curve_type} certificate")
+
+    print("\n=== Generating DPtls Certificates ===")
+
+    dptls_configs = [
+        ("BRP", "testsmdpplus1.example.com", "testsmdpplus1.example.com", 9, "3f:67:15:28:02:b3:f4:c7:fa:e6:79:58:55:f6:82:54:1e:45:e3:5e:ff:f4:e8:a0:55:65:a0:f1:91:2a:78:2e", OID_DP_RID, "DP_TLS_BRP"),
+        ("NIST", "testsmdpplus1.example.com", "testsmdpplus1.example.com", 9, "a0:3e:7c:e4:55:04:74:be:a4:b7:a8:73:99:ce:5a:8c:9f:66:1b:68:0f:94:01:39:ff:f8:4e:9d:ec:6a:4d:8c", OID_DP_RID, "DP_TLS_NIST"),
+        ("NIST", "testsmdpplus2.example.com", "testsmdpplus2.example.com", 12, "4e:65:61:c6:40:88:f6:69:90:7a:db:e3:94:b1:1a:84:24:2e:03:3a:82:a8:84:02:31:63:6d:c9:1b:4e:e3:f5", OID_DP2_RID, "DP2_TLS"),
+        ("NIST", "testsmdpplus4.example.com", "testsmdpplus4.example.com", 14, "f2:65:9d:2f:52:8f:4b:11:37:40:d5:8a:0d:2a:f3:eb:2b:48:e1:22:c2:b6:0a:6a:f6:fc:96:ad:86:be:6f:a4", OID_DP4_RID, "DP4_TLS"),
+        ("NIST", "testsmdpplus8.example.com", "testsmdpplus8.example.com", 18, "ff:6e:4a:50:9b:ad:db:38:10:88:31:c2:3c:cc:2d:44:30:7a:f2:81:e9:25:96:7f:8c:df:1d:95:54:a0:28:8d", OID_DP8_RID, "DP8_TLS"),
+    ]
+
+    for curve_type, cn, dns, serial, key_hex, rid_oid, name_prefix in dptls_configs:
+        cert, key = gen.generate_tls_cert(
+            curve_type, cn, dns, serial, key_hex, rid_oid,
+            datetime(2024, 7, 9, 15, 29, 36),
+            datetime(2025, 8, 11, 15, 29, 36)
+        )
+        gen.save_cert_and_key(
+            cert, key,
+            f"{output_dir}/DPtls/CERT_S_SM_{name_prefix}.der",
+            None,
+            f"{output_dir}/DPtls/SK_S_SM_{name_prefix.replace('_BRP', '_BRP').replace('_NIST', '_NIST')}.pem",
+            f"{output_dir}/DPtls/PK_S_SM_{name_prefix.replace('_BRP', '_BRP').replace('_NIST', '_NIST')}.pem"
+        )
+        print(f"Generated {name_prefix} certificate")
+
+    print("\n=== Generating EUM Certificates ===")
+
+    eum_configs = [
+        ("BRP", "12:9b:0a:b1:3f:17:e1:4a:40:b6:fa:4e:d8:23:e0:cf:46:5b:7b:3d:73:24:05:e6:29:5d:3b:23:b0:45:c9:9a"),
+        ("NIST", "25:e6:75:77:28:e1:e9:51:13:51:9c:dc:34:55:5c:29:ba:ed:23:77:3a:c5:af:dd:dc:da:d9:84:89:8a:52:f0"),
+    ]
+
+    eum_certs = {}
+    eum_keys = {}
+
+    for curve_type, key_hex in eum_configs:
+        cert, key = gen.generate_eum_cert(curve_type, key_hex)
+        eum_certs[curve_type] = cert
+        eum_keys[curve_type] = key
+        suffix = "_ECDSA_BRP" if curve_type == "BRP" else "_ECDSA_NIST"
+        gen.save_cert_and_key(
+            cert, key,
+            f"{output_dir}/EUM/CERT_EUM{suffix}.der",
+            None,
+            f"{output_dir}/EUM/SK_EUM{suffix}.pem",
+            f"{output_dir}/EUM/PK_EUM{suffix}.pem"
+        )
+        print(f"Generated EUM {curve_type} certificate")
+
+    print("\n=== Generating eUICC Certificates ===")
+
+    euicc_configs = [
+        ("BRP", "8d:c3:47:a7:6d:b7:bd:d6:22:2d:d7:5e:a1:a1:68:8a:ca:81:1e:4c:bc:6a:7f:6a:ef:a4:b2:64:19:62:0b:90"),
+        ("NIST", "11:e1:54:67:dc:19:4f:33:71:83:e4:60:c9:f6:32:60:09:1e:12:e8:10:26:cd:65:61:e1:7c:6d:85:39:cc:9c"),
+    ]
+
+    for curve_type, key_hex in euicc_configs:
+        cert, key = gen.generate_euicc_cert(curve_type, eum_certs[curve_type], eum_keys[curve_type], key_hex)
+        suffix = "_ECDSA_BRP" if curve_type == "BRP" else "_ECDSA_NIST"
+        gen.save_cert_and_key(
+            cert, key,
+            f"{output_dir}/eUICC/CERT_EUICC{suffix}.der",
+            None,
+            f"{output_dir}/eUICC/SK_EUICC{suffix}.pem",
+            f"{output_dir}/eUICC/PK_EUICC{suffix}.pem"
+        )
+        print(f"Generated eUICC {curve_type} certificate")
+
+    print("\n=== Certificate generation complete! ===")
+    print(f"All certificates saved to: {output_dir}/")
+
+if __name__ == "__main__":
+    main()

contrib/jenkins.sh

@@ -1,56 +1,101 @@
-#!/bin/sh
+#!/bin/sh -xe
 # jenkins build helper script for pysim.  This is how we build on jenkins.osmocom.org
 #
 # environment variables:
 # * WITH_MANUALS: build manual PDFs if set to "1"
 # * PUBLISH: upload manuals after building if set to "1" (ignored without WITH_MANUALS = "1")
+# * JOB_TYPE: one of 'test', 'distcheck', 'pylint', 'docs'
+# * SKIP_CLEAN_WORKSPACE: don't run osmo-clean-workspace.sh (for pyosmocom CI)
 #
 
-set -e
+export PYTHONUNBUFFERED=1
 
-if [ ! -d "./pysim-testdata/" ] ; then
+if [ ! -d "./tests/" ] ; then
 	echo "###############################################"
 	echo "Please call from pySim-prog top directory"
 	echo "###############################################"
 	exit 1
 fi
 
-virtualenv -p python3 venv --system-site-packages
-. venv/bin/activate
-pip install pytlv
-pip install 'pyyaml>=5.1'
-pip install cmd2==1.5
-pip install jsonpath-ng
-pip install construct
-pip install bidict
-pip install gsm0338
-
-# Execute automatically discovered unit tests first
-python -m unittest discover -v -s tests/
-
-# Run pylint to find potential errors
-# Ignore E1102: not-callable
-#   pySim/filesystem.py: E1102: method is not callable (not-callable)
-# Ignore E0401: import-error
-#   pySim/utils.py:276: E0401: Unable to import 'Crypto.Cipher' (import-error)
-#   pySim/utils.py:277: E0401: Unable to import 'Crypto.Util.strxor' (import-error)
-pip install pylint
-python -m pylint --errors-only \
-	--disable E1102 \
-	--disable E0401 \
-	--enable W0301 \
-	pySim *.py
-
-# attempt to build documentation
-pip install sphinx
-pip install sphinxcontrib-napoleon
-pip3 install -e 'git+https://github.com/osmocom/sphinx-argparse@master#egg=sphinx-argparse'
-(cd docs && make html latexpdf)
-
-if [ "$WITH_MANUALS" = "1" ] && [ "$PUBLISH" = "1" ]; then
-	make -C "docs" publish publish-html
+if [ -z "$SKIP_CLEAN_WORKSPACE" ]; then
+	osmo-clean-workspace.sh
 fi
 
-# run the test with physical cards
-cd pysim-testdata
-../tests/pysim-test.sh
+case "$JOB_TYPE" in
+"test")
+	virtualenv -p python3 venv --system-site-packages
+	. venv/bin/activate
+
+	pip install -r requirements.txt
+	pip install pyshark
+
+	# Execute automatically discovered unit tests first
+	python -m unittest discover -v -s tests/unittests
+
+	# Run pySim-prog integration tests (requires physical cards)
+	cd tests/pySim-prog_test/
+        ./pySim-prog_test.sh
+	cd ../../
+
+	# Run pySim-trace test
+	tests/pySim-trace_test/pySim-trace_test.sh
+
+	# Run pySim-shell integration tests (requires physical cards)
+	python3 -m unittest discover -v -s ./tests/pySim-shell_test/
+
+	# Run pySim-smpp2sim test
+	tests/pySim-smpp2sim_test/pySim-smpp2sim_test.sh
+	;;
+"distcheck")
+	virtualenv -p python3 venv --system-site-packages
+	. venv/bin/activate
+
+	pip install .
+	pip install pyshark
+
+	for prog in venv/bin/pySim-*.py; do
+		$prog --help > /dev/null
+	done
+	;;
+"pylint")
+	# Print pylint version
+	pip3 freeze | grep pylint
+
+	virtualenv -p python3 venv --system-site-packages
+	. venv/bin/activate
+
+	pip install .
+
+	# Run pylint to find potential errors
+	# Ignore E1102: not-callable
+	#   pySim/filesystem.py: E1102: method is not callable (not-callable)
+	# Ignore E0401: import-error
+	#   pySim/utils.py:276: E0401: Unable to import 'Crypto.Cipher' (import-error)
+	#   pySim/utils.py:277: E0401: Unable to import 'Crypto.Util.strxor' (import-error)
+	python3 -m pylint -j0 --errors-only \
+		--disable E1102 \
+		--disable E0401 \
+		--enable W0301 \
+		pySim tests/unittests/*.py *.py \
+		contrib/*.py
+	;;
+"docs")
+	virtualenv -p python3 venv --system-site-packages
+	. venv/bin/activate
+
+	pip install -r requirements.txt
+
+	rm -rf docs/_build
+	make -C "docs" html latexpdf
+
+	if [ "$WITH_MANUALS" = "1" ] && [ "$PUBLISH" = "1" ]; then
+		make -C "docs" publish publish-html
+	fi
+	;;
+*)
+	set +x
+	echo "ERROR: JOB_TYPE has unexpected value '$JOB_TYPE'."
+	exit 1
+esac
+
+osmo-clean-workspace.sh

contrib/saip-tool.py

@@ -0,0 +1,489 @@
+#!/usr/bin/env python3
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import sys
+import argparse
+import logging
+from pathlib import Path as PlPath
+from typing import List
+from osmocom.utils import h2b, b2h, swap_nibbles
+from osmocom.construct import GreedyBytes, StripHeaderAdapter
+
+from pySim.esim.saip import *
+from pySim.esim.saip.validation import CheckBasicStructure
+from pySim.pprint import HexBytesPrettyPrinter
+
+pp = HexBytesPrettyPrinter(indent=4,width=500)
+
+parser = argparse.ArgumentParser(description="""
+Utility program to work with eSIM SAIP (SimAlliance Interoperable Profile) files.""")
+parser.add_argument('INPUT_UPP', help='Unprotected Profile Package Input file')
+parser.add_argument("--loglevel", dest="loglevel", choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'],
+                    default='INFO', help="Set the logging level")
+parser.add_argument('--debug', action='store_true', help='Enable DEBUG logging')
+subparsers = parser.add_subparsers(dest='command', help="The command to perform", required=True)
+
+parser_split = subparsers.add_parser('split', help='Split PE-Sequence into individual PEs')
+parser_split.add_argument('--output-prefix', default='.', help='Prefix path/filename for output files')
+
+parser_dump = subparsers.add_parser('dump', help='Dump information on PE-Sequence')
+parser_dump.add_argument('mode', choices=['all_pe', 'all_pe_by_type', 'all_pe_by_naa'])
+parser_dump.add_argument('--dump-decoded', action='store_true', help='Dump decoded PEs')
+
+parser_check = subparsers.add_parser('check', help='Run constraint checkers on PE-Sequence')
+
+parser_rpe = subparsers.add_parser('extract-pe', help='Extract specified PE to (DER encoded) file')
+parser_rpe.add_argument('--pe-file', required=True, help='PE file name')
+parser_rpe.add_argument('--identification', type=int, help='Extract PE matching specified identification')
+
+parser_rpe = subparsers.add_parser('remove-pe', help='Remove specified PEs from PE-Sequence')
+parser_rpe.add_argument('--output-file', required=True, help='Output file name')
+parser_rpe.add_argument('--identification', default=[], type=int, action='append', help='Remove PEs matching specified identification')
+parser_rpe.add_argument('--type', default=[], action='append', help='Remove PEs matching specified type')
+
+parser_rn = subparsers.add_parser('remove-naa', help='Remove specified NAAs from PE-Sequence')
+parser_rn.add_argument('--output-file', required=True, help='Output file name')
+parser_rn.add_argument('--naa-type', required=True, choices=NAAs.keys(), help='Network Access Application type to remove')
+# TODO: add an --naa-index or the like, so only one given instance can be removed
+
+parser_info = subparsers.add_parser('info', help='Display information about the profile')
+parser_info.add_argument('--apps', action='store_true', help='List applications and their related instances')
+
+parser_eapp = subparsers.add_parser('extract-apps', help='Extract applications as loadblock file')
+parser_eapp.add_argument('--output-dir', default='.', help='Output directory (where to store files)')
+parser_eapp.add_argument('--format', default='cap', choices=['ijc', 'cap'], help='Data format of output files')
+
+parser_aapp = subparsers.add_parser('add-app', help='Add application to PE-Sequence')
+parser_aapp.add_argument('--output-file', required=True, help='Output file name')
+parser_aapp.add_argument('--applet-file', required=True, help='Applet file name')
+parser_aapp.add_argument('--aid', required=True, help='Load package AID')
+parser_aapp.add_argument('--sd-aid', default=None, help='Security Domain AID')
+parser_aapp.add_argument('--non-volatile-code-limit', default=None, type=int, help='Non volatile code limit (C6)')
+parser_aapp.add_argument('--volatile-data-limit', default=None, type=int, help='Volatile data limit (C7)')
+parser_aapp.add_argument('--non-volatile-data-limit', default=None, type=int, help='Non volatile data limit (C8)')
+parser_aapp.add_argument('--hash-value', default=None, help='Hash value')
+
+parser_rapp = subparsers.add_parser('remove-app', help='Remove application from PE-Sequence')
+parser_rapp.add_argument('--output-file', required=True, help='Output file name')
+parser_rapp.add_argument('--aid', required=True, help='Load package AID')
+
+parser_aappi = subparsers.add_parser('add-app-inst', help='Add application instance to Application PE')
+parser_aappi.add_argument('--output-file', required=True, help='Output file name')
+parser_aappi.add_argument('--aid', required=True, help='Load package AID')
+parser_aappi.add_argument('--class-aid', required=True, help='Class AID')
+parser_aappi.add_argument('--inst-aid', required=True, help='Instance AID (must match Load package AID)')
+parser_aappi.add_argument('--app-privileges', default='000000', help='Application privileges')
+parser_aappi.add_argument('--volatile-memory-quota', default=None, type=int, help='Volatile memory quota (C7)')
+parser_aappi.add_argument('--non-volatile-memory-quota', default=None, type=int, help='Non volatile memory quota (C8)')
+parser_aappi.add_argument('--app-spec-pars', default='00', help='Application specific parameters (C9)')
+parser_aappi.add_argument('--uicc-toolkit-app-spec-pars', help='UICC toolkit application specific parameters field')
+parser_aappi.add_argument('--uicc-access-app-spec-pars', help='UICC Access application specific parameters field')
+parser_aappi.add_argument('--uicc-adm-access-app-spec-pars', help='UICC Administrative access application specific parameters field')
+parser_aappi.add_argument('--process-data', default=[], action='append', help='Process personalization APDUs')
+
+parser_rappi = subparsers.add_parser('remove-app-inst', help='Remove application instance from Application PE')
+parser_rappi.add_argument('--output-file', required=True, help='Output file name')
+parser_rappi.add_argument('--aid', required=True, help='Load package AID')
+parser_rappi.add_argument('--inst-aid', required=True, help='Instance AID')
+
+esrv_flag_choices = [t.name for t in asn1.types['ServicesList'].type.root_members]
+parser_esrv = subparsers.add_parser('edit-mand-srv-list', help='Add/Remove service flag from/to mandatory services list')
+parser_esrv.add_argument('--output-file', required=True, help='Output file name')
+parser_esrv.add_argument('--add-flag', default=[], choices=esrv_flag_choices, action='append', help='Add flag to mandatory services list')
+parser_esrv.add_argument('--remove-flag', default=[], choices=esrv_flag_choices, action='append', help='Remove flag from mandatory services list')
+
+parser_tree = subparsers.add_parser('tree', help='Display the filesystem tree')
+
+def write_pes(pes: ProfileElementSequence, output_file:str):
+    """write the PE sequence to a file"""
+    print("Writing %u PEs to file '%s'..." % (len(pes.pe_list), output_file))
+    with open(output_file, 'wb') as f:
+        f.write(pes.to_der())
+
+def do_split(pes: ProfileElementSequence, opts):
+    i = 0
+    for pe in pes.pe_list:
+        basename = PlPath(opts.INPUT_UPP).stem
+        if not pe.identification:
+            fname = '%s-%02u-%s.der' % (basename, i, pe.type)
+        else:
+            fname = '%s-%02u-%05u-%s.der' % (basename, i, pe.identification, pe.type)
+        print("writing single PE to file '%s'" % fname)
+        with open(os.path.join(opts.output_prefix, fname), 'wb') as outf:
+            outf.write(pe.to_der())
+        i += 1
+
+def do_dump(pes: ProfileElementSequence, opts):
+    def print_all_pe(pes: ProfileElementSequence, dump_decoded:bool = False):
+        # iterate over each pe in the pes (using its __iter__ method)
+        for pe in pes:
+            print("="*70 + " " + pe.type)
+            if dump_decoded:
+                pp.pprint(pe.decoded)
+
+    def print_all_pe_by_type(pes: ProfileElementSequence, dump_decoded:bool = False):
+        # sort by PE type and show all PE within that type
+        for pe_type in pes.pe_by_type.keys():
+            print("="*70 + " " + pe_type)
+            for pe in pes.pe_by_type[pe_type]:
+                pp.pprint(pe)
+                if dump_decoded:
+                    pp.pprint(pe.decoded)
+
+    def print_all_pe_by_naa(pes: ProfileElementSequence, dump_decoded:bool = False):
+        for naa in pes.pes_by_naa:
+            i = 0
+            for naa_instance in pes.pes_by_naa[naa]:
+                print("="*70 + " " + naa + str(i))
+                i += 1
+                for pe in naa_instance:
+                    pp.pprint(pe.type)
+                    if dump_decoded:
+                        for d in pe.decoded:
+                            print("    %s" % d)
+
+    if opts.mode == 'all_pe':
+        print_all_pe(pes, opts.dump_decoded)
+    elif opts.mode == 'all_pe_by_type':
+        print_all_pe_by_type(pes, opts.dump_decoded)
+    elif opts.mode == 'all_pe_by_naa':
+        print_all_pe_by_naa(pes, opts.dump_decoded)
+
+def do_check(pes: ProfileElementSequence, opts):
+    print("Checking PE-Sequence structure...")
+    checker = CheckBasicStructure()
+    checker.check(pes)
+    print("All good!")
+
+def do_extract_pe(pes: ProfileElementSequence, opts):
+    new_pe_list = []
+    for pe in pes.pe_list:
+        if pe.identification == opts.identification:
+            print("Extracting PE %s (id=%u) to file %s..." % (pe, pe.identification, opts.pe_file))
+            with open(opts.pe_file, 'wb') as f:
+                f.write(pe.to_der())
+
+def do_remove_pe(pes: ProfileElementSequence, opts):
+    new_pe_list = []
+    for pe in pes.pe_list:
+        identification = pe.identification
+        if identification:
+            if identification in opts.identification:
+                print("Removing PE %s (id=%u) from Sequence..." % (pe, identification))
+                continue
+        if pe.type in opts.type:
+            print("Removing PE %s (type=%s) from Sequence..." % (pe, pe.type))
+            continue
+        new_pe_list.append(pe)
+
+    pes.pe_list = new_pe_list
+    pes._process_pelist()
+    write_pes(pes, opts.output_file)
+
+def do_remove_naa(pes: ProfileElementSequence, opts):
+    if not opts.naa_type in NAAs:
+        raise ValueError('unsupported NAA type %s' % opts.naa_type)
+    naa = NAAs[opts.naa_type]
+    print("Removing NAAs of type '%s' from Sequence..." % opts.naa_type)
+    pes.remove_naas_of_type(naa)
+    write_pes(pes, opts.output_file)
+
+def info_apps(pes:ProfileElementSequence):
+    def show_member(dictionary:Optional[dict], member:str, indent:str="\t", mandatory:bool = False, limit:bool = False):
+        if dictionary is None:
+            return
+        value = dictionary.get(member, None)
+        if value is None and mandatory == True:
+            print("%s%s: (missing!)" % (indent, member))
+            return
+        elif value is None:
+            return
+
+        if limit and len(value) > 40:
+            print("%s%s: '%s...%s' (%u bytes)" % (indent, member, b2h(value[:20]), b2h(value[-20:]), len(value)))
+        else:
+            print("%s%s: '%s' (%u bytes)" % (indent, member, b2h(value), len(value)))
+
+    apps = pes.pe_by_type.get('application', [])
+    if len(apps) == 0:
+        print("No Application PE present!")
+        return;
+
+    for app_pe in enumerate(apps):
+        print("Application #%u:" % app_pe[0])
+        print("\tloadBlock:")
+        load_block = app_pe[1].decoded['loadBlock']
+        show_member(load_block, 'loadPackageAID', "\t\t", True)
+        show_member(load_block, 'securityDomainAID', "\t\t")
+        show_member(load_block, 'nonVolatileCodeLimitC6', "\t\t")
+        show_member(load_block, 'volatileDataLimitC7', "\t\t")
+        show_member(load_block, 'nonVolatileDataLimitC8', "\t\t")
+        show_member(load_block, 'hashValue', "\t\t")
+        show_member(load_block, 'loadBlockObject', "\t\t", True, True)
+        for inst in enumerate(app_pe[1].decoded.get('instanceList', [])):
+            print("\tinstanceList[%u]:" % inst[0])
+            show_member(inst[1], 'applicationLoadPackageAID', "\t\t", True)
+            if inst[1].get('applicationLoadPackageAID', None) != load_block.get('loadPackageAID', None):
+                print("\t\t(applicationLoadPackageAID should be the same as loadPackageAID!)")
+            show_member(inst[1], 'classAID', "\t\t", True)
+            show_member(inst[1], 'instanceAID', "\t\t", True)
+            show_member(inst[1], 'extraditeSecurityDomainAID', "\t\t")
+            show_member(inst[1], 'applicationPrivileges', "\t\t", True)
+            show_member(inst[1], 'lifeCycleState', "\t\t", True)
+            show_member(inst[1], 'applicationSpecificParametersC9', "\t\t", True)
+            sys_specific_pars = inst[1].get('systemSpecificParameters', None)
+            if sys_specific_pars:
+                print("\t\tsystemSpecificParameters:")
+                show_member(sys_specific_pars, 'volatileMemoryQuotaC7', "\t\t\t")
+                show_member(sys_specific_pars, 'nonVolatileMemoryQuotaC8', "\t\t\t")
+                show_member(sys_specific_pars, 'globalServiceParameters', "\t\t\t")
+                show_member(sys_specific_pars, 'implicitSelectionParameter', "\t\t\t")
+                show_member(sys_specific_pars, 'volatileReservedMemory', "\t\t\t")
+                show_member(sys_specific_pars, 'nonVolatileReservedMemory', "\t\t\t")
+                show_member(sys_specific_pars, 'ts102226SIMFileAccessToolkitParameter', "\t\t\t")
+                additional_cl_pars = inst.get('ts102226AdditionalContactlessParameters', None)
+                if additional_cl_pars:
+                    print("\t\t\tts102226AdditionalContactlessParameters:")
+                    show_member(additional_cl_pars, 'protocolParameterData', "\t\t\t\t")
+                show_member(sys_specific_pars, 'userInteractionContactlessParameters', "\t\t\t")
+                show_member(sys_specific_pars, 'cumulativeGrantedVolatileMemory', "\t\t\t")
+                show_member(sys_specific_pars, 'cumulativeGrantedNonVolatileMemory', "\t\t\t")
+            app_pars = inst[1].get('applicationParameters', None)
+            if app_pars:
+                print("\t\tapplicationParameters:")
+                show_member(app_pars, 'uiccToolkitApplicationSpecificParametersField', "\t\t\t")
+                show_member(app_pars, 'uiccAccessApplicationSpecificParametersField', "\t\t\t")
+                show_member(app_pars, 'uiccAdministrativeAccessApplicationSpecificParametersField', "\t\t\t")
+            ctrl_ref_tp = inst[1].get('controlReferenceTemplate', None)
+            if ctrl_ref_tp:
+                print("\t\tcontrolReferenceTemplate:")
+                show_member(ctrl_ref_tp, 'applicationProviderIdentifier', "\t\t\t", True)
+            process_data = inst[1].get('processData', None)
+            if process_data:
+                print("\t\tprocessData:")
+                for proc in process_data:
+                    print("\t\t\t" + b2h(proc))
+
+def do_info(pes: ProfileElementSequence, opts):
+    def get_naa_count(pes: ProfileElementSequence) -> dict:
+        """return a dict with naa-type (usim, isim) as key and the count of NAA instances as value."""
+        ret = {}
+        for naa_type in pes.pes_by_naa:
+            ret[naa_type] = len(pes.pes_by_naa[naa_type])
+        return ret
+
+    if opts.apps:
+        info_apps(pes)
+        return;
+
+    pe_hdr_dec = pes.pe_by_type['header'][0].decoded
+    print()
+    print("SAIP Profile Version: %u.%u" % (pe_hdr_dec['major-version'], pe_hdr_dec['minor-version']))
+    print("Profile Type: '%s'" % pe_hdr_dec['profileType'])
+    print("ICCID: %s" % b2h(pe_hdr_dec['iccid']))
+    print("Mandatory Services: %s" % ', '.join(pe_hdr_dec['eUICC-Mandatory-services'].keys()))
+    print()
+    naa_strs = ["%s[%u]" % (k, v) for k, v in get_naa_count(pes).items()]
+    print("NAAs: %s" % ', '.join(naa_strs))
+    for naa_type in pes.pes_by_naa:
+        for naa_inst in pes.pes_by_naa[naa_type]:
+            first_pe = naa_inst[0]
+            adf_name = ''
+            if hasattr(first_pe, 'adf_name'):
+                adf_name = '(' + first_pe.adf_name + ')'
+            print("NAA %s %s" % (first_pe.type, adf_name))
+            if hasattr(first_pe, 'imsi'):
+                print("\tIMSI: %s" % first_pe.imsi)
+
+    # applications
+    print()
+    apps = pes.pe_by_type.get('application', [])
+    print("Number of applications: %u" % len(apps))
+    for app_pe in apps:
+        print("App Load Package AID: %s" % b2h(app_pe.decoded['loadBlock']['loadPackageAID']))
+        print("\tMandated: %s" % ('mandated' in app_pe.decoded['app-Header']))
+        print("\tLoad Block Size: %s" % len(app_pe.decoded['loadBlock']['loadBlockObject']))
+        for inst in app_pe.decoded.get('instanceList', []):
+            print("\tInstance AID: %s" % b2h(inst['instanceAID']))
+
+    # security domains
+    print()
+    sds = pes.pe_by_type.get('securityDomain', [])
+    print("Number of security domains: %u" % len(sds))
+    for sd in sds:
+        print("Security domain Instance AID: %s" % b2h(sd.decoded['instance']['instanceAID']))
+        # FIXME: 'applicationSpecificParametersC9' parsing to figure out enabled SCP
+        for key in sd.keys:
+            print("\t%s" % repr(key))
+
+    # RFM
+    print()
+    rfms = pes.pe_by_type.get('rfm', [])
+    print("Number of RFM instances: %u" % len(rfms))
+    for rfm in rfms:
+        inst_aid = rfm.decoded['instanceAID']
+        print("RFM instanceAID: %s" % b2h(inst_aid))
+        print("\tMSL: 0x%02x" % rfm.decoded['minimumSecurityLevel'][0])
+        adf = rfm.decoded.get('adfRFMAccess', None)
+        if adf:
+            print("\tADF AID: %s" % b2h(adf['adfAID']))
+        tar_list = rfm.decoded.get('tarList', [inst_aid[-3:]])
+        for tar in tar_list:
+            print("\tTAR: %s" % b2h(tar))
+
+def do_extract_apps(pes:ProfileElementSequence, opts):
+    apps = pes.pe_by_type.get('application', [])
+    for app_pe in apps:
+        package_aid = b2h(app_pe.decoded['loadBlock']['loadPackageAID'])
+        fname = os.path.join(opts.output_dir, '%s-%s.%s' % (pes.iccid, package_aid, opts.format))
+        print("Writing Load Package AID: %s to file %s" % (package_aid, fname))
+        app_pe.to_file(fname)
+
+def do_add_app(pes:ProfileElementSequence, opts):
+    print("Applying applet file: '%s'..." % opts.applet_file)
+    app_pe = ProfileElementApplication.from_file(opts.applet_file,
+                                                 opts.aid,
+                                                 opts.sd_aid,
+                                                 opts.non_volatile_code_limit,
+                                                 opts.volatile_data_limit,
+                                                 opts.non_volatile_data_limit,
+                                                 opts.hash_value)
+
+    security_domain = pes.pe_by_type.get('securityDomain', [])
+    if len(security_domain) == 0:
+        print("profile package does not contain a securityDomain, please add a securityDomain PE first!")
+    elif len(security_domain) > 1:
+        print("adding an application PE to profiles with multiple securityDomain is not supported yet!")
+    else:
+        pes.insert_after_pe(security_domain[0], app_pe)
+        print("application PE inserted into PE Sequence after securityDomain PE AID: %s" %
+              b2h(security_domain[0].decoded['instance']['instanceAID']))
+        write_pes(pes, opts.output_file)
+
+def do_remove_app(pes:ProfileElementSequence, opts):
+    apps = pes.pe_by_type.get('application', [])
+    for app_pe in apps:
+        package_aid = b2h(app_pe.decoded['loadBlock']['loadPackageAID'])
+        if opts.aid == package_aid:
+            identification = app_pe.identification
+            opts_remove_pe = argparse.Namespace()
+            opts_remove_pe.identification = [app_pe.identification]
+            opts_remove_pe.type = []
+            opts_remove_pe.output_file = opts.output_file
+            print("Found Load Package AID: %s, removing related PE (id=%u) from Sequence..." %
+                  (package_aid, identification))
+            do_remove_pe(pes, opts_remove_pe)
+            return
+    print("Load Package AID: %s not found in PE Sequence" % opts.aid)
+
+def do_add_app_inst(pes:ProfileElementSequence, opts):
+    apps = pes.pe_by_type.get('application', [])
+    for app_pe in apps:
+        package_aid = b2h(app_pe.decoded['loadBlock']['loadPackageAID'])
+        if opts.aid == package_aid:
+            print("Found Load Package AID: %s, adding new instance AID: %s to Application PE..." %
+                  (opts.aid, opts.inst_aid))
+            app_pe.add_instance(opts.aid,
+                                opts.class_aid,
+                                opts.inst_aid,
+                                opts.app_privileges,
+                                opts.app_spec_pars,
+                                opts.uicc_toolkit_app_spec_pars,
+                                opts.uicc_access_app_spec_pars,
+                                opts.uicc_adm_access_app_spec_pars,
+                                opts.volatile_memory_quota,
+                                opts.non_volatile_memory_quota,
+                                opts.process_data)
+            write_pes(pes, opts.output_file)
+            return
+    print("Load Package AID: %s not found in PE Sequence" % opts.aid)
+
+def do_remove_app_inst(pes:ProfileElementSequence, opts):
+    apps = pes.pe_by_type.get('application', [])
+    for app_pe in apps:
+        if opts.aid == b2h(app_pe.decoded['loadBlock']['loadPackageAID']):
+            print("Found Load Package AID: %s, removing instance AID: %s from Application PE..." %
+                  (opts.aid, opts.inst_aid))
+            app_pe.remove_instance(opts.inst_aid)
+            write_pes(pes, opts.output_file)
+            return
+    print("Load Package AID: %s not found in PE Sequence" % opts.aid)
+
+def do_edit_mand_srv_list(pes: ProfileElementSequence, opts):
+    header = pes.pe_by_type.get('header', [])[0]
+
+    for s in opts.add_flag:
+        print("Adding service '%s' to mandatory services list..." % s)
+        header.mandatory_service_add(s)
+    for s in opts.remove_flag:
+        if s in header.decoded['eUICC-Mandatory-services'].keys():
+            print("Removing service '%s' from mandatory services list..." % s)
+            header.mandatory_service_remove(s)
+        else:
+            print("Service '%s' not present in mandatory services list, cannot remove!" % s)
+
+    print("The following services are now set mandatory:")
+    for s in header.decoded['eUICC-Mandatory-services'].keys():
+        print("\t%s" % s)
+
+    write_pes(pes, opts.output_file)
+
+def do_tree(pes:ProfileElementSequence, opts):
+    pes.mf.print_tree()
+
+if __name__ == '__main__':
+    opts = parser.parse_args()
+
+    if opts.debug:
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.getLevelName(opts.loglevel))
+
+    with open(opts.INPUT_UPP, 'rb') as f:
+        pes = ProfileElementSequence.from_der(f.read())
+
+    print("Read %u PEs from file '%s'" % (len(pes.pe_list), opts.INPUT_UPP))
+
+    if opts.command == 'split':
+        do_split(pes, opts)
+    elif opts.command == 'dump':
+        do_dump(pes, opts)
+    elif opts.command == 'check':
+        do_check(pes, opts)
+    elif opts.command == 'extract-pe':
+        do_extract_pe(pes, opts)
+    elif opts.command == 'remove-pe':
+        do_remove_pe(pes, opts)
+    elif opts.command == 'remove-naa':
+        do_remove_naa(pes, opts)
+    elif opts.command == 'info':
+        do_info(pes, opts)
+    elif opts.command == 'extract-apps':
+        do_extract_apps(pes, opts)
+    elif opts.command == 'add-app':
+        do_add_app(pes, opts)
+    elif opts.command == 'remove-app':
+        do_remove_app(pes, opts)
+    elif opts.command == 'add-app-inst':
+        do_add_app_inst(pes, opts)
+    elif opts.command == 'remove-app-inst':
+        do_remove_app_inst(pes, opts)
+    elif opts.command == 'edit-mand-srv-list':
+        do_edit_mand_srv_list(pes, opts)
+    elif opts.command == 'tree':
+        do_tree(pes, opts)

contrib/saip-tool_example_add-app.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+# This is an example script to illustrate how to add JAVA card applets to an existing eUICC profile package.
+
+PYSIMPATH=../
+INPATH=../smdpp-data/upp/TS48V1-A-UNIQUE.der
+OUTPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-hello.der
+APPPATH=./HelloSTK_09122024.cap
+
+# Download example applet (see also https://gitea.osmocom.org/sim-card/hello-stk):
+if ! [ -f $APPPATH ]; then
+    wget https://osmocom.org/attachments/download/8931/HelloSTK_09122024.cap
+fi
+
+# Step #1: Create the application PE and load the ijc contents from the .cap file:
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $INPATH add-app \
+		      --output-file $OUTPATH --applet-file $APPPATH --aid 'D07002CA44'
+
+# Step #2: Create the application instance inside the application PE created in step #1:
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $OUTPATH add-app-inst --output-file $OUTPATH \
+		      --aid 'D07002CA44' \
+		      --class-aid 'D07002CA44900101' \
+		      --inst-aid 'D07002CA44900101' \
+		      --app-privileges '00' \
+		      --app-spec-pars '00' \
+		      --uicc-toolkit-app-spec-pars '01001505000000000000000000000000'
+
+# Display the contents of the resulting application PE:
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $OUTPATH info --apps
+
+# For an explanation of --uicc-toolkit-app-spec-pars, see:
+# ETSI TS 102 226, section 8.2.1.3.2.2.1

contrib/saip-tool_example_extract-apps.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# This is an example script to illustrate how to extract JAVA card applets from an existing eUICC profile package.
+
+PYSIMPATH=../
+INPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-hello.der
+OUTPATH=./
+
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $INPATH extract-apps --output-dir ./ --format ijc

contrib/saip-tool_example_remove-app-inst.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+# This is an example script to illustrate how to remove a JAVA card applet instance from an application PE inside an
+# existing eUICC profile package.
+
+PYSIMPATH=../
+INPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-hello.der
+OUTPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-hello-no-inst.der
+
+# Remove application PE entirely
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $INPATH remove-app-inst \
+		      --output-file $OUTPATH --aid 'd07002ca44' --inst-aid 'd07002ca44900101'
+
+# Display the contents of the resulting application PE:
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $OUTPATH info --apps

contrib/saip-tool_example_remove-app.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# This is an example script to illustrate how to remove a JAVA card applet from an existing eUICC profile package.
+
+PYSIMPATH=../
+INPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-hello.der
+OUTPATH=../smdpp-data/upp/TS48V1-A-UNIQUE-no-hello.der
+
+# Remove application PE entirely
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $INPATH remove-app \
+		      --output-file $OUTPATH --aid 'D07002CA44'
+
+# Display the contents of the resulting application PE:
+PYTHONPATH=$PYSIMPATH python3 $PYSIMPATH/contrib/saip-tool.py $OUTPATH info --apps

contrib/sim-rest-client.py

@@ -162,6 +162,7 @@ def main(argv):
     parser.add_argument("-v", "--verbose", help="increase output verbosity", action='count', default=0)
     parser.add_argument("-n", "--slot-nr", help="SIM slot number", type=int, default=0)
     subp = parser.add_subparsers()
+    subp.required = True
 
     auth_p = subp.add_parser('auth', help='UMTS AKA Authentication')
     auth_p.add_argument("-c", "--count", help="Auth count", type=int, default=10)

contrib/sim-rest-server.py

@@ -2,7 +2,7 @@
 
 # RESTful HTTP service for performing authentication against USIM cards
 #
-# (C) 2021 by Harald Welte <laforge@osmocom.org>
+# (C) 2021-2022 by Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,12 +21,15 @@ import json
 import sys
 import argparse
 
-from klein import run, route
+from klein import Klein
 
 from pySim.transport import ApduTracer
 from pySim.transport.pcsc import PcscSimLink
 from pySim.commands import SimCardCommands
-from pySim.cards import UsimCard
+from pySim.cards import UiccCardBase
+from pySim.utils import dec_iccid, dec_imsi
+from pySim.ts_51_011 import EF_IMSI
+from pySim.ts_102_221 import EF_ICCID
 from pySim.exceptions import *
 
 class ApduPrintTracer(ApduTracer):
@@ -35,89 +38,118 @@ class ApduPrintTracer(ApduTracer):
         pass
 
 def connect_to_card(slot_nr:int):
-    tp = PcscSimLink(slot_nr, apdu_tracer=ApduPrintTracer())
+    tp = PcscSimLink(argparse.Namespace(pcsc_dev=slot_nr), apdu_tracer=ApduPrintTracer())
     tp.connect()
 
     scc = SimCardCommands(tp)
-    card = UsimCard(scc)
+    card = UiccCardBase(scc)
 
     # this should be part of UsimCard, but FairewavesSIM breaks with that :/
     scc.cla_byte = "00"
     scc.sel_ctrl = "0004"
 
     card.read_aids()
-    card.select_adf_by_aid(adf='usim')
+
+    # ensure that MF is selected when we are done.
+    card._scc.select_file('3f00')
 
     return tp, scc, card
 
+class ApiError:
+    def __init__(self, msg:str, sw=None):
+        self.msg = msg
+        self.sw = sw
 
-@route('/sim-auth-api/v1/slot/<int:slot>')
-def auth(request, slot):
-    """REST API endpoint for performing authentication against a USIM.
-       Expects a JSON body containing RAND and AUTN.
-       Returns a JSON body containing RES, CK, IK and Kc."""
-    try:
-        # there are two hex-string JSON parameters in the body: rand and autn
-        content = json.loads(request.content.read())
-        rand = content['rand']
-        autn = content['autn']
-    except:
-        request.setResponseCode(400)
-        return "Malformed Request"
+    def __str__(self):
+        d = {'error': {'message':self.msg}}
+        if self.sw:
+            d['error']['status_word'] = self.sw
+        return json.dumps(d)
 
-    try:
-        tp, scc, card = connect_to_card(slot)
-    except ReaderError:
-        request.setResponseCode(404)
-        return "Specified SIM Slot doesn't exist"
-    except ProtocolError:
-        request.setResponseCode(500)
-        return "Error"
-    except NoCardError:
+
+def set_headers(request):
+    request.setHeader('Content-Type', 'application/json')
+
+class SimRestServer:
+    app = Klein()
+
+    @app.handle_errors(NoCardError)
+    def no_card_error(self, request, failure):
+        set_headers(request)
         request.setResponseCode(410)
-        return "No SIM card inserted in slot"
+        return str(ApiError("No SIM card inserted in slot"))
+
+    @app.handle_errors(ReaderError)
+    def reader_error(self, request, failure):
+        set_headers(request)
+        request.setResponseCode(404)
+        return str(ApiError("Reader Error: Specified SIM Slot doesn't exist"))
+
+    @app.handle_errors(ProtocolError)
+    def protocol_error(self, request, failure):
+        set_headers(request)
+        request.setResponseCode(500)
+        return str(ApiError("Protocol Error: %s" % failure.value))
+
+    @app.handle_errors(SwMatchError)
+    def sw_match_error(self, request, failure):
+        set_headers(request)
+        request.setResponseCode(500)
+        sw = failure.value.sw_actual
+        if sw == '9862':
+            return str(ApiError("Card Authentication Error - Incorrect MAC", sw))
+        elif sw == '6982':
+            return str(ApiError("Security Status not satisfied - Card PIN enabled?", sw))
+        else:
+            return str(ApiError("Card Communication Error %s" % failure.value, sw))
+
+
+    @app.route('/sim-auth-api/v1/slot/<int:slot>')
+    def auth(self, request, slot):
+        """REST API endpoint for performing authentication against a USIM.
+           Expects a JSON body containing RAND and AUTN.
+           Returns a JSON body containing RES, CK, IK and Kc."""
+        try:
+            # there are two hex-string JSON parameters in the body: rand and autn
+            content = json.loads(request.content.read())
+            rand = content['rand']
+            autn = content['autn']
+        except:
+            set_headers(request)
+            request.setResponseCode(400)
+            return str(ApiError("Malformed Request"))
+
+        tp, scc, card = connect_to_card(slot)
 
-    try:
         card.select_adf_by_aid(adf='usim')
         res, sw = scc.authenticate(rand, autn)
-    except SwMatchError as e:
-        request.setResponseCode(500)
-        return "Communication Error %s" % e
 
-    tp.disconnect()
+        tp.disconnect()
 
-    return json.dumps(res, indent=4)
+        set_headers(request)
+        return json.dumps(res, indent=4)
 
-@route('/sim-info-api/v1/slot/<int:slot>')
-def info(request, slot):
-    """REST API endpoint for obtaining information about an USIM.
-    Expects empty body in request.
-    Returns a JSON body containing ICCID, IMSI."""
+    @app.route('/sim-info-api/v1/slot/<int:slot>')
+    def info(self, request, slot):
+        """REST API endpoint for obtaining information about an USIM.
+        Expects empty body in request.
+        Returns a JSON body containing ICCID, IMSI."""
 
-    try:
         tp, scc, card = connect_to_card(slot)
-    except ReaderError:
-        request.setResponseCode(404)
-        return "Specified SIM Slot doesn't exist"
-    except ProtocolError:
-        request.setResponseCode(500)
-        return "Error"
-    except NoCardError:
-        request.setResponseCode(410)
-        return "No SIM card inserted in slot"
 
-    try:
+        ef_iccid = EF_ICCID()
+        (iccid, sw) = card._scc.read_binary(ef_iccid.fid)
+
         card.select_adf_by_aid(adf='usim')
-        iccid, sw = card.read_iccid()
-        imsi, sw = card.read_imsi()
-        res = {"imsi": imsi, "iccid": iccid }
-    except SwMatchError as e:
-        request.setResponseCode(500)
-        return "Communication Error %s" % e
+        ef_imsi = EF_IMSI()
+        (imsi, sw) = card._scc.read_binary(ef_imsi.fid)
 
-    tp.disconnect()
+        res = {"imsi": dec_imsi(imsi), "iccid": dec_iccid(iccid) }
 
-    return json.dumps(res, indent=4)
+        tp.disconnect()
+
+        set_headers(request)
+        return json.dumps(res, indent=4)
 
 
 def main(argv):
@@ -128,7 +160,8 @@ def main(argv):
 
     args = parser.parse_args()
 
-    run(args.host, args.port)
+    srr = SimRestServer()
+    srr.app.run(args.host, args.port)
 
 if __name__ == "__main__":
     main(sys.argv)

contrib/smpp-ota-tool.py

@@ -0,0 +1,235 @@
+#!/usr/bin/env python3
+
+# (C) 2026 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# Author: Harald Welte, Philipp Maier
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import logging
+import smpplib.gsm
+import smpplib.client
+import smpplib.consts
+import time
+from pySim.ota import OtaKeyset, OtaDialectSms, OtaAlgoCrypt, OtaAlgoAuth, CNTR_REQ, RC_CC_DS, POR_REQ
+from pySim.utils import b2h, h2b, is_hexstr
+from pathlib import Path
+
+logger = logging.getLogger(Path(__file__).stem)
+
+class SmppHandler:
+    client = None
+
+    def __init__(self, host: str, port: int,
+                 system_id: str, password: str,
+                 ota_keyset: OtaKeyset, spi: dict, tar: bytes):
+        """
+        Initialize connection to SMPP server and set static OTA SMS-TPDU ciphering parameters
+        Args:
+                host : Hostname or IPv4/IPv6 address of the SMPP server
+                port : TCP Port of the SMPP server
+                system_id: SMPP System-ID used by ESME (client) to bind
+                password: SMPP Password used by ESME (client) to bind
+                ota_keyset: OTA keyset to be used for SMS-TPDU ciphering
+                spi: Security Parameter Indicator (SPI) to be used for SMS-TPDU ciphering
+                tar: Toolkit Application Reference (TAR) of the targeted card application
+        """
+
+        # Create and connect SMPP client
+        client = smpplib.client.Client(host, port, allow_unknown_opt_params=True)
+        client.set_message_sent_handler(self.message_sent_handler)
+        client.set_message_received_handler(self.message_received_handler)
+        client.connect()
+        client.bind_transceiver(system_id=system_id, password=password)
+        self.client = client
+
+        # Setup static OTA parameters
+        self.ota_dialect = OtaDialectSms()
+        self.ota_keyset = ota_keyset
+        self.tar = tar
+        self.spi = spi
+
+    def __del__(self):
+        if self.client:
+            self.client.unbind()
+            self.client.disconnect()
+
+    def message_received_handler(self, pdu):
+        if pdu.short_message:
+            logger.info("SMS-TPDU received: %s", b2h(pdu.short_message))
+            try:
+                dec = self.ota_dialect.decode_resp(self.ota_keyset, self.spi, pdu.short_message)
+            except ValueError:
+                # Retry to decoding with ciphering disabled (in case the card has problems to decode the SMS-TDPU
+                # we have sent, the response will contain an unencrypted error message)
+                spi = self.spi.copy()
+                spi['por_shall_be_ciphered'] = False
+                spi['por_rc_cc_ds'] = 'no_rc_cc_ds'
+                dec = self.ota_dialect.decode_resp(self.ota_keyset, spi, pdu.short_message)
+            logger.info("SMS-TPDU decoded: %s", dec)
+            self.response = dec
+        return None
+
+    def message_sent_handler(self, pdu):
+            logger.debug("SMS-TPDU sent: pdu_sequence=%s pdu_message_id=%s", pdu.sequence, pdu.message_id)
+
+    def transceive_sms_tpdu(self, tpdu: bytes, src_addr: str, dest_addr: str, timeout: int) -> tuple:
+        """
+        Transceive SMS-TPDU. This method sends the SMS-TPDU to the SMPP server, and waits for a response. The method
+        returns when the response is received.
+
+        Args:
+                tpdu : short message content (plaintext)
+                src_addr : short message source address
+                dest_addr : short message destination address
+                timeout : timeout after which this method should give up waiting for a response
+        Returns:
+                tuple containing the response (plaintext)
+        """
+
+        logger.info("SMS-TPDU sending: %s...", b2h(tpdu))
+
+        self.client.send_message(
+            # TODO: add parameters to switch source_addr_ton and dest_addr_ton between SMPP_TON_INTL and SMPP_NPI_ISDN
+            source_addr_ton=smpplib.consts.SMPP_TON_INTL,
+            source_addr=src_addr,
+            dest_addr_ton=smpplib.consts.SMPP_TON_INTL,
+            destination_addr=dest_addr,
+            short_message=tpdu,
+            # TODO: add parameters to set data_coding and esm_class
+            data_coding=smpplib.consts.SMPP_ENCODING_BINARY,
+            esm_class=smpplib.consts.SMPP_GSMFEAT_UDHI,
+            protocol_id=0x7f,
+            # TODO: add parameter to use registered delivery
+            # registered_delivery=True,
+        )
+
+        logger.info("SMS-TPDU sent, waiting for response...")
+        timestamp_sent=int(time.time())
+        self.response = None
+        while self.response is None:
+            self.client.poll()
+            if int(time.time()) - timestamp_sent > timeout:
+                raise ValueError("Timeout reached, no response SMS-TPDU received!")
+        return self.response
+
+    def transceive_apdu(self, apdu: bytes, src_addr: str, dest_addr: str, timeout: int) -> tuple[bytes, bytes]:
+        """
+        Transceive APDU. This method wraps the given APDU into an SMS-TPDU, sends it to the SMPP server and waits for
+        the response. When the response is received, the last response data and the last status word is extracted from
+        the response and returned to the caller.
+
+        Args:
+                apdu : one or more concatenated APDUs
+                src_addr : short message source address
+                dest_addr : short message destination address
+                timeout : timeout after which this method should give up waiting for a response
+        Returns:
+                tuple containing the last response data and the last status word as byte strings
+        """
+
+        logger.info("C-APDU sending: %s..." % b2h(apdu))
+
+        # translate to Secured OTA RFM
+        secured = self.ota_dialect.encode_cmd(self.ota_keyset, self.tar, self.spi, apdu=apdu)
+        # add user data header
+        tpdu = b'\x02\x70\x00' + secured
+        # send via SMPP
+        response =  self.transceive_sms_tpdu(tpdu, src_addr, dest_addr, timeout)
+
+        # Extract last_response_data and last_status_word from the response
+        sw = None
+        resp = None
+        for container in response:
+            if container:
+                container_dict = dict(container)
+                resp = container_dict.get('last_response_data')
+                sw = container_dict.get('last_status_word')
+        if resp is None:
+            raise ValueError("Response does not contain any last_response_data, no R-APDU received!")
+        if sw is None:
+            raise ValueError("Response does not contain any last_status_word, no R-APDU received!")
+
+        logger.info("R-APDU received: %s %s", resp, sw)
+        return h2b(resp), h2b(sw)
+
+if __name__ == '__main__':
+    option_parser = argparse.ArgumentParser(description='CSV importer for pySim-shell\'s PostgreSQL Card Key Provider',
+                                   formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    option_parser.add_argument("--host", help="Host/IP of the SMPP server", default="localhost")
+    option_parser.add_argument("--port", help="TCP port of the SMPP server", default=2775, type=int)
+    option_parser.add_argument("--system-id", help="System ID to use to bind to the SMPP server", default="test")
+    option_parser.add_argument("--password", help="Password to use to bind to the SMPP server", default="test")
+    option_parser.add_argument("--verbose", help="Enable verbose logging", action='store_true', default=False)
+    algo_crypt_choices = []
+    algo_crypt_classes = OtaAlgoCrypt.__subclasses__()
+    for cls in algo_crypt_classes:
+        algo_crypt_choices.append(cls.enum_name)
+    option_parser.add_argument("--algo-crypt", choices=algo_crypt_choices, default='triple_des_cbc2',
+                               help="OTA crypt algorithm")
+    algo_auth_choices = []
+    algo_auth_classes = OtaAlgoAuth.__subclasses__()
+    for cls in algo_auth_classes:
+        algo_auth_choices.append(cls.enum_name)
+    option_parser.add_argument("--algo-auth", choices=algo_auth_choices, default='triple_des_cbc2',
+                               help="OTA auth algorithm")
+    option_parser.add_argument('--kic', required=True, type=is_hexstr, help='OTA key (KIC)')
+    option_parser.add_argument('--kic_idx', default=1, type=int, help='OTA key index (KIC)')
+    option_parser.add_argument('--kid', required=True, type=is_hexstr, help='OTA key (KID)')
+    option_parser.add_argument('--kid_idx', default=1, type=int, help='OTA key index (KID)')
+    option_parser.add_argument('--cntr', default=0, type=int, help='replay protection counter')
+    option_parser.add_argument('--tar', required=True, type=is_hexstr, help='Toolkit Application Reference')
+    option_parser.add_argument("--cntr_req", choices=CNTR_REQ.decmapping.values(), default='no_counter',
+                               help="Counter requirement")
+    option_parser.add_argument('--ciphering', default=True, type=bool, help='Enable ciphering')
+    option_parser.add_argument("--rc-cc-ds", choices=RC_CC_DS.decmapping.values(), default='cc',
+                               help="message check (rc=redundency check, cc=crypt. checksum, ds=digital signature)")
+    option_parser.add_argument('--por-in-submit', default=False, type=bool,
+                               help='require PoR to be sent via SMS-SUBMIT')
+    option_parser.add_argument('--por-shall-be-ciphered', default=True, type=bool, help='require encrypted PoR')
+    option_parser.add_argument("--por-rc-cc-ds", choices=RC_CC_DS.decmapping.values(), default='cc',
+                               help="PoR check (rc=redundency check, cc=crypt. checksum, ds=digital signature)")
+    option_parser.add_argument("--por_req", choices=POR_REQ.decmapping.values(), default='por_required',
+                               help="Proof of Receipt requirements")
+    option_parser.add_argument('--src-addr', default='12', type=str, help='TODO')
+    option_parser.add_argument('--dest-addr', default='23', type=str, help='TODO')
+    option_parser.add_argument('--timeout', default=10, type=int, help='TODO')
+    option_parser.add_argument('-a', '--apdu', action='append', required=True, type=is_hexstr, help='C-APDU to send')
+    opts = option_parser.parse_args()
+
+    logging.basicConfig(level=logging.DEBUG if opts.verbose else logging.INFO,
+                        format='%(asctime)s %(levelname)s %(message)s',
+                        datefmt='%Y-%m-%d %H:%M:%S')
+
+    ota_keyset = OtaKeyset(algo_crypt=opts.algo_crypt,
+                           kic_idx=opts.kic_idx,
+                           kic=h2b(opts.kic),
+                           algo_auth=opts.algo_auth,
+                           kid_idx=opts.kic_idx,
+                           kid=h2b(opts.kid),
+                           cntr=opts.cntr)
+    spi = {'counter' : opts.cntr_req,
+           'ciphering' : opts.ciphering,
+           'rc_cc_ds': opts.rc_cc_ds,
+           'por_in_submit':opts.por_in_submit,
+           'por_shall_be_ciphered':opts.por_shall_be_ciphered,
+           'por_rc_cc_ds': opts.por_rc_cc_ds,
+           'por': opts.por_req}
+    apdu = h2b("".join(opts.apdu))
+
+    smpp_handler = SmppHandler(opts.host, opts.port, opts.system_id, opts.password, ota_keyset, spi, h2b(opts.tar))
+    resp, sw = smpp_handler.transceive_apdu(apdu, opts.src_addr, opts.dest_addr, opts.timeout)
+    print("%s %s" % (b2h(resp), b2h(sw)))

contrib/suci-keytool.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+
+# small utility program to deal with 5G SUCI key material, at least for the ECIES Protection Scheme
+# Profile A (curve25519) and B (secp256r1)
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+# SPDX-License-Identifier: GPL-2.0+
+
+import argparse
+
+from osmocom.utils import b2h
+from Cryptodome.PublicKey import ECC
+# if used with pycryptodome < v3.21.0 you will get the following error when using curve25519:
+# "Cryptodome.PublicKey.ECC.UnsupportedEccFeature: Unsupported ECC purpose (OID: 1.3.101.110)"
+
+def gen_key(opts):
+    # FIXME: avoid overwriting key files
+    mykey = ECC.generate(curve=opts.curve)
+    data = mykey.export_key(format='PEM')
+    with open(opts.key_file, "wt") as f:
+        f.write(data)
+
+def dump_pkey(opts):
+
+    #with open("curve25519-1.key", "r") as f:
+
+    with open(opts.key_file, "r") as f:
+        data = f.read()
+        mykey = ECC.import_key(data)
+
+        der = mykey.public_key().export_key(format='raw', compress=opts.compressed)
+        print(b2h(der))
+
+arg_parser = argparse.ArgumentParser(description="""Generate or export SUCI keys for 5G SA networks""")
+arg_parser.add_argument('--key-file', help='The key file to use', required=True)
+
+subparsers = arg_parser.add_subparsers(dest='command', help="The command to perform", required=True)
+
+parser_genkey = subparsers.add_parser('generate-key', help='Generate a new key pair')
+parser_genkey.add_argument('--curve', help='The ECC curve to use', choices=['secp256r1','curve25519'], required=True)
+
+parser_dump_pkey = subparsers.add_parser('dump-pub-key', help='Dump the public key')
+parser_dump_pkey.add_argument('--compressed', help='Use point compression', action='store_true')
+
+if __name__ == '__main__':
+
+    opts = arg_parser.parse_args()
+
+    if opts.command == 'generate-key':
+        gen_key(opts)
+    elif opts.command == 'dump-pub-key':
+        dump_pkey(opts)

contrib/unber.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+
+# A more useful version of the 'unber' tool provided with asn1c:
+# Give a hierarchical decode of BER/DER-encoded ASN.1 TLVs
+
+import sys
+import argparse
+
+from osmocom.utils import b2h, h2b
+from osmocom.tlv import bertlv_parse_one, bertlv_encode_tag
+
+def process_one_level(content: bytes, indent: int):
+    remainder = content
+    while len(remainder):
+        tdict, l, v, remainder = bertlv_parse_one(remainder)
+        #print(tdict)
+        rawtag = bertlv_encode_tag(tdict)
+        if tdict['constructed']:
+            print("%s%s l=%d" % (indent*"  ", b2h(rawtag), l))
+            process_one_level(v, indent + 1)
+        else:
+            print("%s%s l=%d %s" % (indent*"  ", b2h(rawtag), l, b2h(v)))
+
+
+option_parser = argparse.ArgumentParser(description='BER/DER data dumper')
+group = option_parser.add_mutually_exclusive_group(required=True)
+group.add_argument('--file', help='Input file')
+group.add_argument('--hex', help='Input hexstring')
+
+
+if __name__ == '__main__':
+    opts = option_parser.parse_args()
+
+    if opts.file:
+        with open(opts.file, 'rb') as f:
+            content = f.read()
+    elif opts.hex:
+        content = h2b(opts.hex)
+    else:
+        # avoid pylint "(possibly-used-before-assignment)" below
+        sys.exit(2)
+
+    process_one_level(content, 0)

docs/Makefile

@@ -4,16 +4,22 @@
 # You can set these variables from the command line, and also
 # from the environment for the first two.
 SPHINXOPTS    ?=
-SPHINXBUILD   ?= sphinx-build
+SPHINXBUILD   ?= python3 -m sphinx.cmd.build
 SOURCEDIR     = .
 BUILDDIR      = _build
 
 # for osmo-gsm-manuals
-OSMO_GSM_MANUALS_DIR=$(shell pkg-config osmo-gsm-manuals --variable=osmogsmmanualsdir 2>/dev/null)
+OSMO_GSM_MANUALS_DIR ?= $(shell pkg-config osmo-gsm-manuals --variable=osmogsmmanualsdir 2>/dev/null)
 OSMO_REPOSITORY = "pysim"
 UPLOAD_FILES = $(BUILDDIR)/latex/osmopysim-usermanual.pdf
 CLEAN_FILES = $(UPLOAD_FILES)
 
+# Copy variables from Makefile.common.inc that are used in publish-html,
+# as Makefile.common.inc must be included after publish-html
+PUBLISH_REF   ?= master
+PUBLISH_TEMPDIR = _publish_tmpdir
+SSH_COMMAND = ssh -o 'UserKnownHostsFile=$(OSMO_GSM_MANUALS_DIR)/build/known_hosts' -p 48
+
 # Put it first so that "make" without argument is like "make help".
 .PHONY: help
 help:
@@ -23,7 +29,16 @@ $(BUILDDIR)/latex/pysim.pdf: latexpdf
 	@/bin/true
 
 publish-html: html
-	rsync -avz -e "ssh -o 'UserKnownHostsFile=$(OSMO_GSM_MANUALS_DIR)/build/known_hosts' -p 48" $(BUILDDIR)/html/ docs@ftp.osmocom.org:web-files/latest/pysim/
+	rm -rf "$(PUBLISH_TEMPDIR)"
+	mkdir -p "$(PUBLISH_TEMPDIR)/pysim/$(PUBLISH_REF)"
+	cp -r "$(BUILDDIR)"/html "$(PUBLISH_TEMPDIR)/pysim/$(PUBLISH_REF)"
+	cd "$(PUBLISH_TEMPDIR)" && \
+		rsync \
+			-avzR \
+			-e "$(SSH_COMMAND)" \
+			"pysim" \
+			docs@ftp.osmocom.org:web-files/
+	rm -rf "$(PUBLISH_TEMPDIR)"
 
 # put this before the catch-all below
 include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.common.inc
@@ -32,4 +47,6 @@ include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.common.inc
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %:
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@if [ "$@" != "shrink" ]; then \
+		$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O); \
+	fi

docs/cap-tutorial.rst

@@ -0,0 +1,103 @@
+
+Guide: Installing JAVA-card applets
+===================================
+
+Almost all modern-day UICC cards have some form of JAVA-card / Sim-Toolkit support, which allows the installation
+of customer specific JAVA-card applets. The installation of JAVA-card applets is usually done via the standardized
+GlobalPlatform (GPC_SPE_034) ISD (Issuer Security Domain) application interface during the card provisioning process.
+(it is also possible to load JAVA-card applets in field via OTA-SMS, but that is beyond the scope of this guide). In
+this guide we will go through the individual steps that are required to load JAVA-card applet onto an UICC card.
+
+
+Preparation
+~~~~~~~~~~~
+
+In this example we will install the CAP file HelloSTK_09122024.cap [1] on an sysmoISIM-SJA2 card. Since the interface
+is standardized, the exact card model does not matter.
+
+The example applet makes use of the STK (Sim-Toolkit), so we must supply STK installation parameters. Those
+parameters are supplied in the form of a hexstring and should be provided by the applet manufacturer. The available
+parameters and their exact encoding is specified in ETSI TS 102 226, section 8.2.1.3.2.1. The installation of
+HelloSTK_09122024.cap [1], will require the following STK installation parameters: "010001001505000000000000000000000000"
+
+During the installation, we also have to set a memory quota for the volatile and for the non volatile card memory.
+Those values also should be provided by the applet manufacturer. In this example, we will allow 255 bytes of volatile
+memory and 255 bytes of non volatile memory to be consumed by the applet.
+
+To install JAVA-card applets, one must be in the possession of the key material belonging to the card. The keys are
+usually provided by the card manufacturer. The following example will use the following keyset:
+
++---------+----------------------------------+
+| Keyname | Keyvalue                         |
++=========+==================================+
+| DEK/KIK | 5524F4BECFE96FB63FC29D6BAAC6058B |
++---------+----------------------------------+
+| ENC/KIC | 542C37A6043679F2F9F71116418B1CD5 |
++---------+----------------------------------+
+| MAC/KID | 34F11BAC8E5390B57F4E601372339E3C |
++---------+----------------------------------+
+
+[1] https://osmocom.org/projects/cellular-infrastructure/wiki/HelloSTK
+
+
+Applet Installation
+~~~~~~~~~~~~~~~~~~~
+
+To prepare the installation, a secure channel to the ISD must be established first:
+
+::
+
+    pySIM-shell (00:MF)> select ADF.ISD
+    {
+        "application_id": "a000000003000000",
+	"proprietary_data": {
+	    "maximum_length_of_data_field_in_command_message": 255
+        }
+    }
+    pySIM-shell (00:MF/ADF.ISD)> establish_scp02 --key-dek 5524F4BECFE96FB63FC29D6BAAC6058B --key-enc 542C37A6043679F2F9F71116418B1CD5 --key-mac 34F11BAC8E5390B57F4E601372339E3C --security-level 1
+    Successfully established a SCP02[01] secure channel
+
+.. warning:: In case you get an "EXCEPTION of type 'ValueError' occurred with message: card cryptogram doesn't match" error message, it is very likely that there is a problem with the key material. The card may lock the ISD access after a certain amount of failed tries. Carefully check the key material any try again.
+
+
+When the secure channel is established, we are ready to install the applet. The installation normally is a multi step
+procedure, where the loading of an executable load file is announced first, then loaded and then installed in a final
+step. The pySim-shell command ``install_cap`` automatically takes care of those three steps.
+
+::
+
+    pySIM-shell (SCP02[01]:00:MF/ADF.ISD)> install_cap /home/user/HelloSTK_09122024.cap --install-parameters-non-volatile-memory-quota 255 --install-parameters-volatile-memory-quota 255 --install-parameters-stk 010001001505000000000000000000000000
+    loading cap file: /home/user/HelloSTK_09122024.cap ...
+    parameters:
+     security-domain-aid: a000000003000000
+     load-file: 569 bytes
+     load-file-aid: d07002ca44
+     module-aid: d07002ca44900101
+     application-aid: d07002ca44900101
+     install-parameters: c900ef1cc80200ffc70200ffca12010001001505000000000000000000000000
+    step #1: install for load...
+    step #2: load...
+    Loaded a total of 573 bytes in 3 blocks. Don't forget install_for_install (and make selectable) now!
+    step #3: install_for_install (and make selectable)...
+    done.
+
+The applet is now installed on the card. We can now quit pySim-shell and remove the card from the reader and test the
+applet in a mobile phone. There should be a new STK application with one menu entry shown, that will greet the user
+when pressed.
+
+
+Applet Removal
+~~~~~~~~~~~~~~
+
+To remove the applet, we must establish a secure channel to the ISD (see above). Then we can delete the applet using the
+``delete_card_content`` command.
+
+::
+
+    pySIM-shell (SCP02[01]:00:MF/ADF.ISD)> delete_card_content D07002CA44 --delete-related-objects
+
+The parameter "D07002CA44" is the load-file-AID of the applet. The load-file-AID is encoded in the .cap file and also
+displayed during the installation process. It is also important to note that when the applet is installed, it cannot
+be installed (under the same AID) again until it is removed.
+
+

docs/card-key-provider.rst

@@ -0,0 +1,342 @@
+Retrieving card-individual keys via CardKeyProvider
+===================================================
+
+When working with a batch of cards, or more than one card in general, it
+is a lot of effort to manually retrieve the card-specific PIN (like
+ADM1) or key material (like SCP02/SCP03 keys).
+
+To increase productivity in that regard, pySim has a concept called the
+`CardKeyProvider`.  This is a generic mechanism by which different parts
+of the pySim[-shell] code can programmatically request card-specific key material
+from some data source (*provider*).
+
+For example, when you want to verify the ADM1 PIN using the `verify_adm`
+command without providing an ADM1 value yourself, pySim-shell will
+request the ADM1 value for the ICCID of the card via the
+CardKeyProvider.
+
+There can in theory be multiple different CardKeyProviders.  You can for
+example develop your own CardKeyProvider that queries some kind of
+database for the key material, or that uses a key derivation function to
+derive card-specific key material from a global master key.
+
+pySim already includes two CardKeyProvider implementations. One to retrieve
+key material from a CSV file (`CardKeyProviderCsv`) and a second one that allows
+to retrieve the key material from a PostgreSQL database (`CardKeyProviderPgsql`).
+Both implementations equally implement a column encryption scheme that allows
+to protect sensitive columns using a *transport key*
+
+
+The CardKeyProviderCsv
+----------------------
+
+The `CardKeyProviderCsv` allows you to retrieve card-individual key
+material from a CSV (comma separated value) file that is accessible to pySim.
+
+The CSV file must have the expected column names, for example `ICCID`
+and `ADM1` in case you would like to use that CSV to obtain the
+card-specific ADM1 PIN when using the `verify_adm` command.
+
+You can specify the CSV file to use via the `--csv` command-line option
+of pySim-shell.  If you do not specify a CSV file, pySim will attempt to
+open a CSV file from the default location at
+`~/.osmocom/pysim/card_data.csv`, and use that, if it exists.
+
+The `CardKeyProviderCsv` is suitable to manage small amounts of key material
+locally. However, if your card inventory is very large and the key material
+must be made available on multiple sites, the `CardKeyProviderPgsql` is the
+better option.
+
+
+The CardKeyProviderPgsql
+------------------------
+
+With the `CardKeyProviderPgsql` you can use a PostgreSQL database as storage
+medium. The implementation comes with a CSV importer tool that consumes the
+same CSV files you would normally use with the `CardKeyProviderCsv`, so you
+can just use your existing CSV files and import them into the database.
+
+
+Requirements
+^^^^^^^^^^^^
+
+The `CardKeyProviderPgsql` uses the `Psycopg` PostgreSQL database adapter
+(https://www.psycopg.org). `Psycopg` is not part of the default requirements
+of pySim-shell and must be installed separately. `Psycopg` is available as
+Python package under the name `psycopg2-binary`.
+
+
+Setting up the database
+^^^^^^^^^^^^^^^^^^^^^^^
+
+From the perspective of the database, the `CardKeyProviderPgsql` has only
+minimal requirements. You do not have to create any tables in advance. An empty
+database and at least one user that may create, alter and insert into tables is
+sufficient. However, for increased reliability and as a protection against
+incorrect operation, the `CardKeyProviderPgsql` supports a hierarchical model
+with three users (or roles):
+
+* **admin**:
+  This should be the owner of the database. It is intended to be used for
+  administrative tasks like adding new tables or adding new columns to existing
+  tables. This user should not be used to insert new data into tables or to access
+  data from within pySim-shell using the `CardKeyProviderPgsql`
+
+* **importer**:
+  This user is used when feeding new data into an existing table. It should only
+  be able to insert new rows into existing tables. It should not be used for
+  administrative tasks or to access data from within pySim-shell using the
+  `CardKeyProviderPgsql`
+
+* **reader**:
+  To access data from within pySim shell using the `CardKeyProviderPgsql` the
+  reader user is the correct one to use. This user should have no write access
+  to the database or any of the tables.
+
+
+Creating a config file
+^^^^^^^^^^^^^^^^^^^^^^
+
+The default location for the config file is `~/.osmocom/pysim/card_data_pgsql.cfg`
+The file uses `yaml` syntax and should look like the example below:
+
+::
+
+   host: "127.0.0.1"
+   db_name: "my_database"
+   table_names:
+   - "uicc_keys"
+   - "euicc_keys"
+   db_users:
+      admin:
+         name: "my_admin_user"
+         pass: "my_admin_password"
+      importer:
+         name: "my_importer_user"
+         pass: "my_importer_password"
+      reader:
+         name: "my_reader_user"
+         pass: "my_reader_password"
+
+This file is used by pySim-shell and by the importer tool. Both expect the file
+in the aforementioned location. In case you want to store the file in a
+different location you may use the `--pgsql` commandline option to provide a
+custom config file path.
+
+The hostname and the database name for the PostgreSQL database is set with the
+`host` and `db_name` fields. The field `db_users` sets the user names and
+passwords for each of the aforementioned users (or roles). In case only a single
+admin user is used, all three entries may be populated with the same user name
+and password (not recommended)
+
+The field `table_names` sets the tables that the `CardKeyProviderPgsql` shall
+use to query to locate card key data. You can set up as many tables as you
+want, `CardKeyProviderPgsql` will query them in order, one by one until a
+matching entry is found.
+
+NOTE: In case you do not want to disclose the admin and the importer credentials
+to pySim-shell you may remove those lines. pySim-shell will only require the
+`reader` entry under `db_users`.
+
+
+Using the Importer
+^^^^^^^^^^^^^^^^^^
+
+Before data can be imported, you must first create a database table. Tables
+are created with the provided importer tool, which can be found under
+`contrib/csv-to-pgsql.py`. This tool is used to create the database table and
+read the data from the provided CSV file into the database.
+
+As mentioned before, all CSV file formats that work with `CardKeyProviderCsv`
+may be used. To demonstrate how the import process works, let's assume you want
+to import a CSV file format that looks like the following example. Let's also
+assume that you didn't get the Global Platform keys from your card vendor for
+this batch of UICC cards, so your CSV file lacks the columns for those fields.
+
+::
+
+   "id","imsi","iccid","acc","pin1","puk1","pin2","puk2","ki","opc","adm1"
+   "card1","999700000000001","8900000000000000001","0001","1111","11111111","0101","01010101","11111111111111111111111111111111","11111111111111111111111111111111","11111111"
+   "card2","999700000000002","8900000000000000002","0002","2222","22222222","0202","02020202","22222222222222222222222222222222","22222222222222222222222222222222","22222222"
+   "card3","999700000000003","8900000000000000003","0003","3333","22222222","0303","03030303","33333333333333333333333333333333","33333333333333333333333333333333","33333333"
+
+Since this is your first import, the database still lacks the table. To
+instruct the importer to create a new table, you may use the `--create-table`
+option. You also have to pick an appropriate name for the table. Any name may
+be chosen as long as it contains the string `uicc_keys` or `euicc_keys`,
+depending on the type of data (`UICC` or `eUICC`) you intend to store in the
+table. The creation of the table is an administrative task and can only be done
+with the `admin` user. The `admin` user is selected using the `--admin` switch.
+
+::
+
+   $ PYTHONPATH=../ ./csv-to-pgsql.py --csv ./csv-to-pgsql_example_01.csv --table-name uicc_keys --create-table --admin
+   INFO: CSV file: ./csv-to-pgsql_example_01.csv
+   INFO: CSV file columns: ['ID', 'IMSI', 'ICCID', 'ACC', 'PIN1', 'PUK1', 'PIN2', 'PUK2', 'KI', 'OPC', 'ADM1']
+   INFO: Using config file: /home/user/.osmocom/pysim/card_data_pgsql.cfg
+   INFO: Database host: 127.0.0.1
+   INFO: Database name: my_database
+   INFO: Database user: my_admin_user
+   INFO: New database table created: uicc_keys
+   INFO: Database table: uicc_keys
+   INFO: Database table columns: ['ICCID', 'IMSI']
+   INFO: Adding missing columns: ['PIN2', 'PUK1', 'PUK2', 'ACC', 'ID', 'PIN1', 'ADM1', 'KI', 'OPC']
+   INFO: Changes to table uicc_keys committed!
+
+The importer has created a new table with the name `uicc_keys`. The table is
+now ready to be filled with data.
+
+::
+
+   $ PYTHONPATH=../ ./csv-to-pgsql.py --csv ./csv-to-pgsql_example_01.csv --table-name uicc_keys
+   INFO: CSV file: ./csv-to-pgsql_example_01.csv
+   INFO: CSV file columns: ['ID', 'IMSI', 'ICCID', 'ACC', 'PIN1', 'PUK1', 'PIN2', 'PUK2', 'KI', 'OPC', 'ADM1']
+   INFO: Using config file: /home/user/.osmocom/pysim/card_data_pgsql.cfg
+   INFO: Database host: 127.0.0.1
+   INFO: Database name: my_database
+   INFO: Database user: my_importer_user
+   INFO: Database table: uicc_keys
+   INFO: Database table columns: ['ICCID', 'IMSI', 'PIN2', 'PUK1', 'PUK2', 'ACC', 'ID', 'PIN1', 'ADM1', 'KI', 'OPC']
+   INFO: CSV file import done, 3 rows imported
+   INFO: Changes to table uicc_keys committed!
+
+A quick `SELECT * FROM uicc_keys;` at the PostgreSQL console should now display
+the contents of the CSV file you have fed into the importer.
+
+Let's now assume that with your next batch of UICC cards your vendor includes
+the Global Platform keys so your CSV format changes. It may now look like this:
+
+::
+
+   "id","imsi","iccid","acc","pin1","puk1","pin2","puk2","ki","opc","adm1","scp02_dek_1","scp02_enc_1","scp02_mac_1"
+   "card4","999700000000004","8900000000000000004","0004","4444","44444444","0404","04040404","44444444444444444444444444444444","44444444444444444444444444444444","44444444","44444444444444444444444444444444","44444444444444444444444444444444","44444444444444444444444444444444"
+   "card5","999700000000005","8900000000000000005","0005","4444","55555555","0505","05050505","55555555555555555555555555555555","55555555555555555555555555555555","55555555","55555555555555555555555555555555","55555555555555555555555555555555","55555555555555555555555555555555"
+   "card6","999700000000006","8900000000000000006","0006","4444","66666666","0606","06060606","66666666666666666666666666666666","66666666666666666666666666666666","66666666","66666666666666666666666666666666","66666666666666666666666666666666","66666666666666666666666666666666"
+
+When importing data from an updated CSV format the database table also has
+to be updated. This is done using the `--update-columns` switch. Like when
+creating new tables, this operation also requires admin privileges, so the
+`--admin` switch is required again.
+
+::
+
+   $ PYTHONPATH=../ ./csv-to-pgsql.py --csv ./csv-to-pgsql_example_02.csv --table-name uicc_keys --update-columns --admin
+   INFO: CSV file: ./csv-to-pgsql_example_02.csv
+   INFO: CSV file columns: ['ID', 'IMSI', 'ICCID', 'ACC', 'PIN1', 'PUK1', 'PIN2', 'PUK2', 'KI', 'OPC', 'ADM1', 'SCP02_DEK_1', 'SCP02_ENC_1', 'SCP02_MAC_1']
+   INFO: Using config file: /home/user/.osmocom/pysim/card_data_pgsql.cfg
+   INFO: Database host: 127.0.0.1
+   INFO: Database name: my_database
+   INFO: Database user: my_admin_user
+   INFO: Database table: uicc_keys
+   INFO: Database table columns: ['ICCID', 'IMSI', 'PIN2', 'PUK1', 'PUK2', 'ACC', 'ID', 'PIN1', 'ADM1', 'KI', 'OPC']
+   INFO: Adding missing columns: ['SCP02_ENC_1', 'SCP02_MAC_1', 'SCP02_DEK_1']
+   INFO: Changes to table uicc_keys committed!
+
+When the new table columns are added, the import may be continued like the
+first one:
+
+::
+
+   $ PYTHONPATH=../ ./csv-to-pgsql.py --csv ./csv-to-pgsql_example_02.csv --table-name uicc_keys
+   INFO: CSV file: ./csv-to-pgsql_example_02.csv
+   INFO: CSV file columns: ['ID', 'IMSI', 'ICCID', 'ACC', 'PIN1', 'PUK1', 'PIN2', 'PUK2', 'KI', 'OPC', 'ADM1', 'SCP02_DEK_1', 'SCP02_ENC_1', 'SCP02_MAC_1']
+   INFO: Using config file: /home/user/.osmocom/pysim/card_data_pgsql.cfg
+   INFO: Database host: 127.0.0.1
+   INFO: Database name: my_database
+   INFO: Database user: my_importer_user
+   INFO: Database table: uicc_keys
+   INFO: Database table columns: ['ICCID', 'IMSI', 'PIN2', 'PUK1', 'PUK2', 'ACC', 'ID', 'PIN1', 'ADM1', 'KI', 'OPC', 'SCP02_ENC_1', 'SCP02_MAC_1', 'SCP02_DEK_1']
+   INFO: CSV file import done, 3 rows imported
+   INFO: Changes to table uicc_keys committed!
+
+On the PostgreSQL console a `SELECT * FROM uicc_keys;` should now show the
+imported data with the added columns. All important data should now also be
+available from within pySim-shell via the `CardKeyProviderPgsql`.
+
+
+Column-Level CSV encryption
+---------------------------
+
+pySim supports column-level CSV encryption.  This feature will make sure
+that your key material is not stored in plaintext in the CSV file (or
+database).
+
+The encryption mechanism uses AES in CBC mode.  You can use any key
+length permitted by AES (128/192/256 bit).
+
+Following GSMA FS.28, the encryption works on column level.  This means
+different columns can be decrypted using different key material.  This
+means that leakage of a column encryption key for one column or set of
+columns (like a specific security domain) does not compromise various
+other keys that might be stored in other columns.
+
+You can specify column-level decryption keys using the
+`--csv-column-key` command line argument.  The syntax is
+`FIELD:AES_KEY_HEX`, for example:
+
+`pySim-shell.py --csv-column-key SCP03_ENC_ISDR:000102030405060708090a0b0c0d0e0f`
+
+In order to avoid having to repeat the column key for each and every
+column of a group of keys within a keyset, there are pre-defined column
+group aliases, which will make sure that the specified key will be used
+by all columns of the set:
+
+* `UICC_SCP02` is a group alias for `UICC_SCP02_KIC1`, `UICC_SCP02_KID1`, `UICC_SCP02_KIK1`
+* `UICC_SCP03` is a group alias for `UICC_SCP03_KIC1`, `UICC_SCP03_KID1`, `UICC_SCP03_KIK1`
+* `SCP03_ECASD` is a group alias for `SCP03_ENC_ECASD`, `SCP03_MAC_ECASD`, `SCP03_DEK_ECASD`
+* `SCP03_ISDA` is a group alias for `SCP03_ENC_ISDA`, `SCP03_MAC_ISDA`, `SCP03_DEK_ISDA`
+* `SCP03_ISDR` is a group alias for `SCP03_ENC_ISDR`, `SCP03_MAC_ISDR`, `SCP03_DEK_ISDR`
+
+NOTE: When using `CardKeyProviderPqsl`, the input CSV files must be encrypted
+before import.
+
+Field naming
+------------
+
+* For look-up of UICC/SIM/USIM/ISIM or eSIM profile specific key
+  material, pySim uses the `ICCID` field as lookup key.
+
+* For look-up of eUICC specific key material (like SCP03 keys for the
+  ISD-R, ECASD), pySim uses the `EID` field as lookup key.
+
+As soon as the CardKeyProvider finds a line (row) in your CSV file
+(or database) where the ICCID or EID match, it looks for the column containing
+the requested data.
+
+
+ADM PIN
+~~~~~~~
+
+The `verify_adm` command will attempt to look up the `ADM1` column
+indexed by the ICCID of the SIM/UICC.
+
+
+SCP02 / SCP03
+~~~~~~~~~~~~~
+
+SCP02 and SCP03 each use key triplets consisting if ENC, MAC and DEK
+keys.  For more details, see the applicable GlobalPlatform
+specifications.
+
+If you do not want to manually enter the key material for each specific
+card as arguments to the `establish_scp02` or `establish_scp03`
+commands, you can make use of the `--key-provider-suffix` option.  pySim
+uses this suffix to compose the column names for the CardKeyProvider as
+follows.
+
+* `SCP02_ENC_` + suffix for the SCP02 ciphering key
+* `SCP02_MAC_` + suffix for the SCP02 MAC key
+* `SCP02_DEK_` + suffix for the SCP02 DEK key
+* `SCP03_ENC_` + suffix for the SCP03 ciphering key
+* `SCP03_MAC_` + suffix for the SCP03 MAC key
+* `SCP03_DEK_` + suffix for the SCP03 DEK key
+
+So for example, if you are using a command like `establish_scp03
+--key-provider-suffix ISDR`, then the column names for the key material
+look-up are `SCP03_ENC_ISDR`, `SCP03_MAC_ISDR` and `SCP03_DEK_ISDR`,
+respectively.
+
+The identifier used for look-up is determined by the definition of the
+Security Domain.  For example, the eUICC ISD-R and ECASD will use the EID
+of the eUICC.  On the other hand, the ISD-P of an eSIM or the ISD of an
+UICC will use the ICCID.

docs/conf.py

@@ -18,9 +18,17 @@ sys.path.insert(0, os.path.abspath('..'))
 # -- Project information -----------------------------------------------------
 
 project = 'osmopysim-usermanual'
-copyright = '2009-2022 by Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
-author = 'Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
+copyright = '2009-2025 by Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle, Merlin Chlosta'
+author = 'Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle, Merlin Chlosta'
 
+# PDF: Avoid that the authors list exceeds the page by inserting '\and'
+# manually as line break (https://github.com/sphinx-doc/sphinx/issues/6875)
+latex_elements = {
+    "maketitle":
+        r"""\author{Sylvain Munaut, Harald Welte, Philipp Maier, \and Supreeth Herle, Merlin Chlosta}
+\sphinxmaketitle
+"""
+}
 
 # -- General configuration ---------------------------------------------------
 

docs/index.rst

@@ -31,6 +31,7 @@ pySim consists of several parts:
 * a python :ref:`library<pySim library>` containing plenty of objects and methods that can be used for
   writing custom programs interfacing with SIM cards.
 * the [new] :ref:`interactive pySim-shell command line program<pySim-shell>`
+* the [new] :ref:`pySim-trace APDU trace decoder<pySim-trace>`
 * the [legacy] :ref:`pySim-prog and pySim-read tools<Legacy tools>`
 
 .. toctree::
@@ -38,8 +39,15 @@ pySim consists of several parts:
    :caption: Contents:
 
    shell
+   trace
    legacy
+   smpp2sim
    library
+   library-esim
+   osmo-smdpp
+   sim-rest
+   suci-keytool
+   saip-tool
 
 
 Indices and tables

docs/legacy.rst

@@ -1,20 +1,20 @@
-Legacy tools
+Legacy tools
 ============
 
 *legacy tools* are the classic ``pySim-prog`` and ``pySim-read`` programs that
 existed long before ``pySim-shell``.
 
-These days, you should primarily use ``pySim-shell`` instead of these
+These days, it is highly recommended to use ``pySim-shell`` instead of these
 legacy tools.
 
 pySim-prog
 ----------
 
-``pySim-prog`` was the first part of the pySim software suite.  It started as
-a tool to write ICCID, IMSI, MSISDN and Ki to very simplistic SIM cards, and
-was later extended to a variety of other cards.  As the number of features supported
-became no longer bearable to express with command-line arguments, `pySim-shell` was
-created.
+``pySim-prog`` was the first part of the pySim software suite. It started as a
+tool to write ICCID, IMSI, MSISDN and Ki to very simplistic SIM cards, and was
+later extended to a variety of other cards. As the number of features supported
+became no longer bearable to express with command-line arguments, `pySim-shell`
+was created.
 
 Basic use cases can still use `pySim-prog`.
 
@@ -22,36 +22,180 @@ Program customizable SIMs
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 Two modes are possible:
 
-  - one where you specify every parameter manually :
+  - one where the user specifies every parameter manually:
 
-``./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -i <IMSI> -s <ICCID>``
+    This is the most common way to use ``pySim-prog``. The user will specify all relevant parameters directly via the
+    commandline. A typical commandline would look like this:
+
+    ``pySim-prog.py -p <pcsc_reader> --ki <ki_value> --opc <opc_value> --mcc <mcc_value> --mnc <mnc_value>
+    --country <country_code> --imsi <imsi_value> --iccid <iccid_value> --pin-adm <adm_pin>``
+
+    Please note, that this already lengthy commandline still only contains the most common card parameters. For a full
+    list of all possible parameters, use the ``--help`` option of ``pySim-prog``. It is also important to mention
+    that not all parameters are supported by all card types. In particular, very simple programmable SIM cards will only
+    support a very basic set of parameters, such as MCC, MNC, IMSI and KI values.
+
+  - one where the parameters are generated from a minimal set:
+
+    It is also possible to leave the generation of certain parameters to ``pySim-prog``. This is in particular helpful
+    when a large number of cards should be initialized with randomly generated key material.
+
+    ``pySim-prog.py -p <pcsc_reader> --mcc <mcc_value> --mnc <mnc_value> --secret <random_secret> --num <card_number> --pin-adm <adm_pin>``
+
+    The parameter ``--secret`` specifies a random seed that is used to generate the card individual parameters. (IMSI).
+    The secret should contain enough randomness to avoid conflicts. It is also recommended to store the secret safely,
+    in case cards have to be re-generated or the current card batch has to be extended later. For security reasons, the
+    key material, which is also card individual, will not be derived from the random seed. Instead a new random set of
+    Ki and OPc will be generated during each programming cycle. This means fresh keys are generated, even when the
+    ``--num`` remains unchanged.
+
+    The parameter ``--num`` specifies a card individual number. This number will be managed into the random seed so that
+    it serves as an identifier for a particular set of randomly generated parameters.
+
+    In the example above the parameters ``--mcc``, and ``--mnc`` are specified as well, since they identify the GSM
+    network where the cards should operate in, it is absolutely required to keep them static. ``pySim-prog`` will use
+    those parameters to generate a valid IMSI that thas the specified MCC/MNC at the beginning and a random tail.
+
+Specifying the card type:
+
+``pySim-prog`` usually autodetects the card type. In case auto detection does not work, it is possible to specify
+the parameter ``--type``. The following card types are supported:
+
+ * Fairwaves-SIM
+ * fakemagicsim
+ * gialersim
+ * grcardsim
+ * magicsim
+ * OpenCells-SIM
+ * supersim
+ * sysmoISIM-SJA2
+ * sysmoISIM-SJA5
+ * sysmosim-gr1
+ * sysmoSIM-GR2
+ * sysmoUSIM-SJS1
+ * Wavemobile-SIM
+
+Specifying the card reader:
+
+It is most common to use ``pySim-prog`` together with a PCSC reader. The PCSC reader number is specified via the
+``--pcsc-device`` or ``-p`` option. However, other reader types (such as serial readers and modems) are supported. Use
+the ``--help`` option of ``pySim-prog`` for more information.
 
 
-  - one where they are generated from some minimal set :
+Card programming using CSV files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-``./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -z <random_string_of_choice> -j <card_num>``
+To simplify the card programming process, ``pySim-prog`` also allows to read
+the card parameters from a CSV file. When a CSV file is used as input, the
+user does not have to craft an individual commandline for each card. Instead
+all card related parameters are automatically drawn from the CSV file.
 
-    With <random_string_of_choice> and <card_num>, the soft will generate
-    'predictable' IMSI and ICCID, so make sure you choose them so as not to
-    conflict with anyone. (for eg. your name as <random_string_of_choice> and
-    0 1 2 ... for <card num>).
+A CSV files may hold rows for multiple (hundreds or even thousands) of
+cards. ``pySim-prog`` is able to identify the rows either by ICCID
+(recommended as ICCIDs are normally not changed) or IMSI.
 
-  You also need to enter some parameters to select the device :
-   -t TYPE : type of card (supersim, magicsim, fakemagicsim or try 'auto')
-   -d DEV  : Serial port device (default /dev/ttyUSB0)
-   -b BAUD : Baudrate (default 9600)
+The CSV file format is a flexible format with mandatory and optional columns,
+here the same rules as for the commandline parameters apply. The column names
+match the command line options. The CSV file may also contain columns that are
+unknown to pySim-prog, such as inventory numbers, nicknames or parameters that
+are unrelated to the card programming process. ``pySim-prog`` will silently
+ignore all unknown columns.
+
+A CSV file may contain the following columns:
+
+* name
+* iccid (typically used as key)
+* mcc
+* mnc
+* imsi (may be used as key, but not recommended)
+* smsp
+* ki
+* opc
+* acc
+* pin_adm, adm1 or pin_adm_hex (must be present)
+* msisdn
+* epdgid
+* epdgSelection
+* pcscf
+* ims_hdomain
+* impi
+* impu
+* opmode
+* fplmn
+
+Due to historical reasons, and to maintain the compatibility between multiple different CSV file formats, the ADM pin
+may be stored in three different columns. Only one of the three columns must be available.
+
+* adm1: This column contains the ADM pin in numeric ASCII digit format. This format is the most common.
+* pin_adm: Same as adm1, only the column name is different
+* pin_adm_hex: If the ADM pin consists of raw HEX digits, rather then of numerical ASCII digits, then the ADM pin
+  can also be provided as HEX string using this column.
+
+The following example shows a typical minimal example
+::
+
+   "imsi","iccid","acc","ki","opc","adm1"
+   "999700000053010","8988211000000530108","0001","51ACE8BD6313C230F0BFE1A458928DF0","E5A00E8DE427E21B206526B5D1B902DF","65942330"
+   "999700000053011","8988211000000530116","0002","746AAFD7F13CFED3AE626B770E53E860","38F7CE8322D2A7417E0BBD1D7B1190EC","13445792"
+   "999700123053012","8988211000000530124","0004","D0DA4B7B150026ADC966DC637B26429C","144FD3AEAC208DFFF4E2140859BAE8EC","53540383"
+   "999700000053013","8988211000000530132","0008","52E59240ABAC6F53FF5778715C5CE70E","D9C988550DC70B95F40342298EB84C5E","26151368"
+   "999700000053014","8988211000000530140","0010","3B4B83CB9C5F3A0B41EBD17E7D96F324","D61DCC160E3B91F284979552CC5B4D9F","64088605"
+   "999700000053015","8988211000000530157","0020","D673DAB320D81039B025263610C2BBB3","4BCE1458936B338067989A06E5327139","94108841"
+   "999700000053016","8988211000000530165","0040","89DE5ACB76E06D14B0F5D5CD3594E2B1","411C4B8273FD7607E1885E59F0831906","55184287"
+   "999700000053017","8988211000000530173","0080","977852F7CEE83233F02E69E211626DE1","2EC35D48DBF2A99C07D4361F19EF338F","70284674"
+
+The following commandline will instruct ``pySim-prog`` to use the provided CSV file as parameter source and the
+ICCID (read from the card before programming) as a key to identify the card. To use the IMSI as a key, the parameter
+``--read-imsi`` can be used instead of ``--read-iccid``. However, this option is only recommended to be used in very
+specific corner cases.
+
+``pySim-prog.py -p <pcsc_reader> --read-csv <path_to_csv_file> --source csv --read-iccid``
+
+It is also possible to pick a row from the CSV file by manually providing an ICCID (option ``--iccid``) or an IMSI
+(option ``--imsi``) that is then used as a key to find the matching row in the CSV file.
+
+``pySim-prog.py -p <pcsc_reader> --read-csv <path_to_csv_file> --source csv --iccid <iccid_value>``
+
+
+Writing CSV files
+~~~~~~~~~~~~~~~~~
+``pySim-prog`` is also able to generate CSV files that contain a subset of the parameters it has generated or received
+from some other source (commandline, CSV-File). The generated file will be header-less and contain the following
+columns:
+
+* name
+* iccid
+* mcc
+* mnc
+* imsi
+* smsp
+* ki
+* opc
+
+A commandline that makes use of the CSV write feature would look like this:
+
+``pySim-prog.py -p <pcsc_reader> --read-csv <path_to_input_csv_file> --read-iccid --source csv --write-csv <path_to_output_csv_file>``
+
+
+Batch programming
+~~~~~~~~~~~~~~~~~
+
+In case larger card batches need to be programmed, it is possible to use the ``--batch`` parameter to run ``pySim-prog`` in batch mode.
+
+The batch mode will prompt the user to insert a card. Once a card is detected in the reader, the programming is carried out. The user may then remove the card again and the process starts over. This allows for a quick and efficient card programming without permanent commandline interaction.
 
 
 pySim-read
 ----------
 
-``pySim-read`` allows you to read some data from a SIM card.  It will only some files
-of the card, and will only read files accessible to a normal user (without any special authentication)
+``pySim-read`` allows to read some of the most important data items from a SIM
+card. This means it will only read some files of the card, and will only read
+files accessible to a normal user (without any special authentication)
 
-These days, you should use the ``export`` command of ``pySim-shell``
-instead. It performs a much more comprehensive export of all of the
-[standard] files that can be found on the card.  To get a human-readable
-decode instead of the raw hex export, you can use ``export --json``.
+These days, it is recommended to use the ``export`` command of ``pySim-shell``
+instead. It performs a much more comprehensive export of all of the [standard]
+files that can be found on the card. To get a human-readable decode instead of
+the raw hex export, you can use ``export --json``.
 
 Specifically, pySim-read will dump the following:
 
@@ -98,3 +242,4 @@ pySim-read usage
 .. argparse::
    :module: pySim-read
    :func: option_parser
+   :prog: pySim-read.py

docs/library-esim.rst

@@ -0,0 +1,95 @@
+pySim eSIM libraries
+====================
+
+The pySim eSIM libraries implement a variety of functionality related to the GSMA eSIM universe,
+including the various interfaces of SGP.21 + SGP.22, as well as Interoperable Profile decioding,
+validation, personalization and encoding.
+
+.. automodule:: pySim.esim
+   :members:
+
+
+GSMA SGP.21/22 Remote SIM Provisioning (RSP) - High Level
+---------------------------------------------------------
+
+pySim.esim.rsp
+~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.rsp
+   :members:
+
+pySim.esim.es2p
+~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.es2p
+   :members:
+
+
+pySim.esim.es8p
+~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.es8p
+   :members:
+
+
+pySim.esim.es9p
+~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.es9p
+   :members:
+
+
+GSMA SGP.21/22 Remote SIM Provisioning (RSP) - Low Level
+--------------------------------------------------------
+
+pySim.esim.bsp
+~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.bsp
+   :members:
+
+pySim.esim.http_json_api
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.http_json_api
+   :members:
+
+pySim.esim.x509_cert
+~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.x509_cert
+   :members:
+
+SIMalliance / TCA Interoperable Profile
+---------------------------------------
+
+
+pySim.esim.saip
+~~~~~~~~~~~~~~~
+.. automodule:: pySim.esim.saip
+   :members:
+
+
+pySim.esim.saip.oid
+~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.saip.oid
+   :members:
+
+pySim.esim.saip.personalization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.saip.personalization
+   :members:
+
+pySim.esim.saip.templates
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.saip.templates
+   :members:
+
+pySim.esim.saip.validation
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.esim.saip.validation
+   :members:

docs/library.rst

@@ -74,18 +74,6 @@ at 9600 bps.  These readers are sometimes called `Phoenix`.
    :members:
 
 
-pySim construct utilities
--------------------------
-
-.. automodule:: pySim.construct
-   :members:
-
-pySim TLV utilities
--------------------
-
-.. automodule:: pySim.tlv
-   :members:
-
 pySim utility functions
 -----------------------
 

docs/osmo-smdpp.rst

@@ -0,0 +1,149 @@
+osmo-smdpp
+==========
+
+`osmo-smdpp` is a proof-of-concept implementation of a minimal **SM-DP+** as specified for the *GSMA
+Consumer eSIM Remote SIM provisioning*.
+
+At least at this point, it is intended to be used for research and development, and not as a
+production SM-DP+.
+
+Unless you are a GSMA SAS-SM accredited SM-DP+ operator and have related DPtls, DPauth and DPpb
+certificates signed by the GSMA CI, you **can not use osmo-smdpp with regular production eUICC**.
+This is due to how the GSMA eSIM security architecture works.  You can, however, use osmo-smdpp with
+so-called *test-eUICC*, which contain certificates/keys signed by GSMA test certificates as laid out
+in GSMA SGP.26.
+
+At this point, osmo-smdpp does not support anything beyond the bare minimum required to download
+eSIM profiles to an eUICC.  Specifically, there is no ES2+ interface, and there is no built-in
+support for profile personalization yet.
+
+osmo-smdpp currently
+
+* [by default] uses test certificates copied from GSMA SGP.26 into `./smdpp-data/certs`, assuming that your
+  osmo-smdpp would be running at the host name `testsmdpplus1.example.com`. You can of course replace those
+  certificates with your own, whether SGP.26 derived or part of a *private root CA* setup with matching eUICCs.
+* doesn't understand profile state. Any profile can always be downloaded any number of times, irrespective
+  of the EID or whether it was downloaded before.  This is actually very useful for R&D and testing, as it
+  doesn't require you to generate new profiles all the time.  This logic of course is unsuitable for
+  production usage.
+* doesn't perform any personalization, so the IMSI/ICCID etc. are always identical (the ones that are stored in
+  the respective UPP `.der` files)
+* **is absolutely insecure**, as it
+
+ * does not perform all of the mandatory certificate verification (it checks the certificate chain, but not
+   the expiration dates nor any CRL)
+ * does not evaluate/consider any *Confirmation Code*
+ * stores the sessions in an unencrypted *python shelve* and is hence leaking one-time key materials
+   used for profile encryption and signing.
+
+
+Running osmo-smdpp
+------------------
+
+osmo-smdpp comes with built-in TLS support which is enabled by default. However, it is always possible to
+disable the built-in TLS support if needed.
+
+In order to use osmo-smdpp without the built-in TLS support, it has to be put behind a TLS reverse proxy,
+which terminates the ES9+ HTTPS traffic from the LPA, and then forwards it as plain HTTP to osmo-smdpp.
+
+NOTE: The built in TLS support in osmo-smdpp makes use of the python *twisted* framework. Older versions
+of this framework appear to have problems when using the example elliptic curve certificates (both NIST and
+Brainpool) from GSMA.
+
+
+nginx as TLS proxy
+~~~~~~~~~~~~~~~~~~
+
+If you chose to use `nginx` as TLS reverse proxy, you can use the following configuration snippet::
+
+  upstream smdpp {
+          server localhost:8000;
+  }
+
+  server {
+          listen 443 ssl;
+          server_name testsmdpplus1.example.com;
+
+          ssl_certificate /my/path/to/pysim/smdpp-data/certs/DPtls/CERT_S_SM_DP_TLS_NIST.pem;
+          ssl_certificate_key /my/path/to/pysim/smdpp-data/certs/DPtls/SK_S_SM_DP_TLS_NIST.pem;
+
+          location / {
+                  proxy_read_timeout 600s;
+
+                  proxy_hide_header X-Powered-By;
+                  proxy_set_header X-Real-IP $remote_addr;
+                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                  proxy_set_header X-Forwarded-Proto https;
+                  proxy_set_header X-Forwarded-Port $proxy_port;
+                  proxy_set_header Host $host;
+
+                  proxy_pass http://smdpp/;
+          }
+  }
+
+You can of course achieve a similar functionality with apache, lighttpd or many other web server
+software.
+
+supplementary files
+~~~~~~~~~~~~~~~~~~~
+
+The `smdpp-data/certs` directory contains the DPtls, DPauth and DPpb as well as CI certificates
+used; they are copied from GSMA SGP.26 v2.  You can of course replace them with custom certificates
+if you're operating eSIM with a *private root CA*.
+
+The `smdpp-data/upp` directory contains the UPP (Unprotected Profile Package) used.  The file names (without
+.der suffix) are looked up by the matchingID parameter from the activation code presented by the LPA.
+
+commandline options
+~~~~~~~~~~~~~~~~~~~
+
+Typically, you just run osmo-smdpp without any arguments, and it will bind its built-in HTTPS ES9+ interface to
+`localhost` TCP port 443. In this case an external TLS reverse proxy is not needed.
+
+osmo-smdpp currently doesn't have any configuration file.
+
+There are command line options for binding:
+
+Bind the HTTPS ES9+ to a port other than 443::
+
+  ./osmo-smdpp.py -p 8443
+
+Disable the built-in TLS support and bind the plain-HTTP ES9+ to a port 8000::
+
+  ./osmo-smdpp.py -p 8000 --nossl
+
+Bind the HTTP ES9+ to a different local interface::
+
+  ./osmo-smdpp.py -H 127.0.0.2
+
+DNS setup for your LPA
+~~~~~~~~~~~~~~~~~~~~~~
+
+The LPA must resolve `testsmdpplus1.example.com` to the IP address of your TLS proxy.
+
+It must also accept the TLS certificates used by your TLS proxy. In case osmo-smdpp is used with built-in TLS support,
+it will use the certificates provided in smdpp-data.
+
+NOTE: The HTTPS ES9+ interface cannot be addressed by the LPA directly via its IP address. The reason for this is that
+the included SGP.26 (DPtls) test certificates explicitly restrict the hostname to `testsmdpplus1.example.com` in the
+`X509v3 Subject Alternative Name` extension. Using a bare IP address as hostname may cause the certificate to be
+rejected by the LPA.
+
+
+Supported eUICC
+~~~~~~~~~~~~~~~
+
+If you run osmo-smdpp with the included SGP.26 (DPauth, DPpb) certificates, you must use an eUICC with matching SGP.26
+certificates, i.e. the EUM certificate must be signed by a SGP.26 test root CA and the eUICC certificate
+in turn must be signed by that SGP.26 EUM certificate.
+
+sysmocom (sponsoring development and maintenance of pySim and osmo-smdpp) is selling SGP.26 test eUICC
+as `sysmoEUICC1-C2T`.  They are publicly sold in the `sysmocom webshop <https://shop.sysmocom.de/eUICC-for-consumer-eSIM-RSP-with-SGP.26-Test-Certificates/sysmoEUICC1-C2T>`_.
+
+In general you can use osmo-smdpp also with certificates signed by any other certificate authority.  You
+just always must ensure that the certificates of the SM-DP+ are signed by the same root CA as those of your
+eUICCs.
+
+Hypothetically, osmo-smdpp could also be operated with GSMA production certificates, but it would require
+that somebody brings the code in-line with all the GSMA security requirements (HSM support, ...) and operate
+it in a GSMA SAS-SM accredited environment and pays for the related audits.

docs/remote-access.rst

@@ -0,0 +1,46 @@
+
+Remote access to an UICC/eUICC
+==============================
+
+To access a card with pySim-shell, it is not strictly necessary to have physical
+access to it. There are solutions that allow remote access to UICC/eUICC cards.
+In this section we will give a brief overview.
+
+
+osmo-remsim
+-----------
+
+osmo-remsim is a suite of software programs enabling physical/geographic
+separation of a cellular phone (or modem) on the one hand side and the
+UICC/eUICC card on the other side.
+
+Using osmo-remsim, you can operate an entire fleet of modems/phones, as well as
+banks of SIM cards and dynamically establish or remove the connections between
+modems/phones and cards.
+
+To access remote cards with pySim-shell via osmo-remseim (RSPRO), the
+provided libifd_remsim_client would be used to provide a virtual PC/SC reader
+on the local machine. pySim-shell can then access this reader like any other
+PC/SC reader.
+
+More information on osmo-remsim can be found under:
+ * https://osmocom.org/projects/osmo-remsim/wiki
+ * https://ftp.osmocom.org/docs/osmo-remsim/master/osmo-remsim-usermanual.pdf
+
+
+Android APDU proxy
+------------------
+
+Android APDU proxy is an Android app that provides a bridge between a host
+computer and the UICC/eUICC slot of an Android smartphone.
+
+The APDU proxy connects to VPCD server that runs on the remote host (in this
+case the local machine where pySim-shell is running). The VPCD server then
+provides a virtual PC/SC reader, that pySim-shell can access like any other
+PC/SC reader.
+
+On the Android side the UICC/eUICC is accessed via OMAPI (Open Mobile API),
+which is available in Android since API level Android 8 (API level 29).
+
+More information Android APDU proxy can be found under:
+ * https://gitea.osmocom.org/sim-card/android-apdu-proxy

docs/saip-tool.rst

@@ -0,0 +1,137 @@
+saip-tool
+=========
+
+eSIM profiles are stored as a sequence of profile element (PE) objects in an ASN.1 DER encoded binary file. To inspect,
+verify or make changes to those files, the `saip-tool.py` utility can be used.
+
+NOTE: The file format, eSIM SAIP (SimAlliance Interoperable Profile) is specified in `TCA eUICC Profile Package:
+Interoperable Format Technical Specification`
+
+
+Profile Package Examples
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+pySim ships with a set of TS48 profile package examples. Those examples can be found in `pysim/smdpp-data/upp`. The
+files can be used as input for `saip-tool.py`. (see also GSMA TS.48 - Generic eUICC Test Profile for Device Testing)
+
+See also: https://github.com/GSMATerminals/Generic-eUICC-Test-Profile-for-Device-Testing-Public
+
+JAVA card applets
+~~~~~~~~~~~~~~~~~
+
+The `saip-tool.py` can also be used to manage JAVA-card applets (Application PE) inside a profile package. The user has
+the option to add, remove and inspect applications and their instances. In the following we will discuss a few JAVA-card
+related use-cases of `saip-tool.py`
+
+NOTE: see also `contrib` folder for script examples (`saip-tool_example_*.sh`)
+
+Inserting applications
+----------------------
+
+An application is usually inserted in two steps. In the first step, the application PE is created and populated with
+the executable code from a provided `.cap` or `.ijc` file. The user also has to pick a suitable load block AID.
+
+The application instance, which exists inside the application PE, is created in a second step. Here the user must
+reference the load block AID and pick, among other application related parameters, a suitable class and instance AID.
+
+Example: Adding a JAVA-card applet to an existing profile package
+::
+
+  # Step #1: Create the application PE and load the ijc contents from the .cap file:
+  $ ./contrib/saip-tool.py upp.der add-app --output-file upp_with_app.der --applet-file app.cap --aid '1122334455'
+  Read 28 PEs from file 'upp.der'
+  Applying applet file: 'app.cap'...
+  application PE inserted into PE Sequence after securityDomain PE AID: a000000151000000
+  Writing 29 PEs to file 'upp_with_app.der'...
+
+  # Step #2: Create the application instance inside the application PE created in step #1:
+  $ ./contrib/saip-tool.py upp_with_app.der add-app-inst --output-file upp_with_app_and_instance.der \
+          --aid '1122334455' \
+          --class-aid '112233445501' \
+          --inst-aid '112233445501' \
+          --app-privileges '00' \
+          --app-spec-pars '00' \
+          --uicc-toolkit-app-spec-pars '01001505000000000000000000000000'
+  Read 29 PEs from file 'upp_with_app.der'
+  Found Load Package AID: 1122334455, adding new instance AID: 112233445501 to Application PE...
+  Writing 29 PEs to file 'upp_with_app_and_instance.der'...
+
+NOTE: The parameters of the sub-commands `add-app` and `add-app-inst` are application specific. It is up to the application
+developer to pick parameters that suit the application correctly. For an exact command reference see section
+`saip-tool syntax`. For parameter details see `TCA eUICC Profile Package: Interoperable Format Technical Specification`,
+section 8.7 and ETSI TS 102 226, section 8.2.1.3.2
+
+
+Inspecting applications
+-----------------------
+
+To inspect the application PE contents of an existing profile package, sub-command `info` with parameter '--apps' can
+be used. This command lists out all application and their parameters in detail. This allows an application developer
+to check if the applet insertaion was carried out as expected.
+
+Example: Listing applications and their parameters
+::
+
+  $ ./contrib/saip-tool.py upp_with_app_and_instance.der info --apps
+  Read 29 PEs from file 'upp_with_app_and_instance.der'
+  Application #0:
+          loadBlock:
+                  loadPackageAID: '1122334455' (5 bytes)
+                  loadBlockObject: '01000fdecaffed010204000105d07002ca440200...681080056810a00633b44104b431066800a10231' (569 bytes)
+          instanceList[0]:
+                  applicationLoadPackageAID: '1122334455' (5 bytes)
+                  classAID: '112233445501' (8 bytes)
+                  instanceAID: '112233445501' (8 bytes)
+                  applicationPrivileges: '00' (1 bytes)
+                  lifeCycleState: '07' (1 bytes)
+                  applicationSpecificParametersC9: '00' (1 bytes)
+                  applicationParameters:
+                          uiccToolkitApplicationSpecificParametersField: '01001505000000000000000000000000' (16 bytes)
+
+In case further analysis with external tools or transfer of applications from one profile package to another is
+necessary, the executable code in the `loadBlockObject` field can be extracted to an `.ijc` or an `.cap` file.
+
+Example: Extracting applications from a profile package
+::
+
+  $ ./contrib/saip-tool.py upp_with_app_and_instance.der extract-apps --output-dir ./apps --format ijc
+  Read 29 PEs from file 'upp_with_app_and_instance.der'
+  Writing Load Package AID: 1122334455 to file ./apps/8949449999999990023f-1122334455.ijc
+
+
+Removing applications
+---------------------
+
+An application PE can be removed using sub-command `remove-app`. The user passes the load package AID as parameter. Then
+`saip-tool.py` will search for the related application PE and delete it from the PE sequence.
+
+Example: Remove an application from a profile package
+::
+
+  $ ./contrib/saip-tool.py upp_with_app_and_instance.der remove-app --output-file upp_without_app.der --aid '1122334455'
+  Read 29 PEs from file 'upp_with_app_and_instance.der'
+  Found Load Package AID: 1122334455, removing related PE (id=23) from Sequence...
+  Removing PE application (id=23) from Sequence...
+  Writing 28 PEs to file 'upp_without_app.der'...
+
+In some cases it is useful to remove only an instance from an existing application PE. This may be the case when the
+an application developer wants to modify parameters of an application by removing and re-adding the instance. The
+operation basically rolls the state back to step 1 explained in section :ref:`Inserting applications`
+
+Example: Remove an application instance from an application PE
+::
+
+  $ ./contrib/saip-tool.py upp_with_app_and_instance.der remove-app-inst --output-file upp_without_app.der --aid '1122334455' --inst-aid '112233445501'
+  Read 29 PEs from file 'upp_with_app_and_instance.der'
+  Found Load Package AID: 1122334455, removing instance AID: 112233445501 from Application PE...
+  Removing instance from Application PE...
+  Writing 29 PEs to file 'upp_with_app.der'...
+
+
+saip-tool syntax
+~~~~~~~~~~~~~~~~
+
+.. argparse::
+   :module: contrib.saip-tool
+   :func: parser
+   :prog: contrib/saip-tool.py

docs/sim-rest.rst

@@ -0,0 +1,118 @@
+sim-rest-server
+===============
+
+Sometimes there are use cases where a [remote] application will need
+access to a USIM for authentication purposes.  This is, for example, in
+case an IMS test client needs to perform USIM based authentication
+against an IMS core.
+
+The pysim repository contains two programs: `sim-rest-server.py` and
+`sim-rest-client.py` that implement a simple approach to achieve the
+above:
+
+`sim-rest-server.py` speaks to a [usually local] USIM via the PC/SC
+API and provides a high-level REST API towards [local or remote]
+applications that wish to perform UMTS AKA using the USIM.
+
+`sim-rest-client.py` implements a small example client program to
+illustrate how the REST API provided by `sim-rest-server.py` can be
+used.
+
+REST API Calls
+--------------
+
+POST /sim-auth-api/v1/slot/SLOT_NR
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+where SLOT_NR is the integer-encoded slot number (corresponds to PC/SC
+reader number).  When using a single sysmoOCTSIM board, this is in the range of 0..7
+
+Example: `/sim-auth-api/v1/slot/0` for the first slot.
+
+Request Body
+############
+
+The request body is a JSON document, comprising of
+    1. the RAND and AUTN parameters as hex-encoded string
+    2. the application against which to authenticate (USIM, ISIM)
+
+Example:
+::
+
+        {
+            "rand": "bb685a4b2fc4d697b9d6a129dd09a091",
+            "autn": "eea7906f8210000004faf4a7df279b56"
+        }
+
+HTTP Status Codes
+#################
+
+HTTP status codes are used to represent errors within the REST server
+and the SIM reader hardware.  They are not used to communicate protocol
+level errors reported by the SIM Card.  An unsuccessful authentication
+will hence have a `200 OK` HTTP Status code and then encode the SIM
+specific error information in the Response Body.
+
+======  =========== ================================
+Status  Code        Description
+------  ----------- --------------------------------
+200     OK          Successful execution
+400     Bad Request Request body is malformed
+404     Not Found   Specified SIM Slot doesn't exist
+410     Gone        No SIM card inserted in slot
+======  =========== ================================
+
+Response Body
+#############
+
+The response body is a JSON document, either
+
+#. a successful outcome; encoding RES, CK, IK as hex-encoded string
+#. a sync failure; encoding AUTS as hex-encoded string
+#. errors
+   #. authentication error (incorrect MAC)
+   #. authentication error (security context not supported)
+   #. key freshness failure
+   #. unspecified card error
+
+Example (success):
+::
+
+        {
+           "successful_3g_authentication": {
+                "res": "b15379540ec93985",
+                "ck": "713fde72c28cbd282a4cd4565f3d6381",
+                "ik": "2e641727c95781f1020d319a0594f31a",
+                "kc": "771a2c995172ac42"
+            }
+        }
+
+Example (re-sync case):
+::
+
+        {
+            "synchronisation_failure": {
+                "auts": "dc2a591fe072c92d7c46ecfe97e5"
+            }
+        }
+
+Concrete example using the included sysmoISIM-SJA2
+--------------------------------------------------
+
+This was tested using SIMs ending in IMSI numbers 45890...45899
+
+The following command were executed successfully:
+
+Slot 0
+::
+
+        $ /usr/local/src/pysim/contrib/sim-rest-client.py -c 1 -n 0 -k 841EAD87BC9D974ECA1C167409357601 -o 3211CACDD64F51C3FD3013ECD9A582A0
+        -> {'rand': 'fb195c7873b20affa278887920b9dd57', 'autn': 'd420895a6aa2000089cd016f8d8ae67c'}
+        <- {'successful_3g_authentication': {'res': '131004db2ff1ce8e', 'ck': 'd42eb5aa085307903271b2422b698bad', 'ik': '485f81e6fd957fe3cad374adf12fe1ca', 'kc': '64d3f2a32f801214'}}
+
+Slot 1
+::
+
+        $ /usr/local/src/pysim/contrib/sim-rest-client.py -c 1 -n 1 -k 5C2CE9633FF9B502B519A4EACD16D9DF -o 9834D619E71A02CD76F00CC7AA34FB32
+        -> {'rand': '433dc5553db95588f1d8b93870930b66', 'autn': '126bafdcbe9e00000026a208da61075d'}
+        <- {'successful_3g_authentication': {'res': '026d7ac42d379207', 'ck': '83a90ba331f47a95c27a550b174c4a1f', 'ik': '31e1d10329ffaf0ca1684a1bf0b0a14a', 'kc': 'd15ac5b0fff73ecc'}}

docs/smpp2sim.rst

@@ -0,0 +1,57 @@
+pySim-smpp2sim
+==============
+
+This is a program to emulate the entire communication path SMSC-CN-RAN-ME
+that is usually between an OTA backend and the SIM card.  This allows
+to play with SIM OTA technology without using a mobile network or even
+a mobile phone.
+
+An external application can act as SMPP ESME and must encode (and
+encrypt/sign) the OTA SMS and submit them via SMPP to this program, just
+like it would submit it normally to a SMSC (SMS Service Centre).  The
+program then re-formats the SMPP-SUBMIT into a SMS DELIVER TPDU and
+passes it via an ENVELOPE APDU to the SIM card that is locally inserted
+into a smart card reader.
+
+The path from SIM to external OTA application works the opposite way.
+
+The default SMPP system_id is `test`.  Likewise, the default SMPP
+password is `test`
+
+Running pySim-smpp2sim
+----------------------
+
+The command accepts the same command line arguments for smart card interface device selection as pySim-shell,
+as well as a few SMPP specific arguments:
+
+.. argparse::
+   :module: pySim-smpp2sim
+   :func: option_parser
+   :prog: pySim-smpp2sim.py
+
+
+Example execution with sample output
+------------------------------------
+
+So for a simple system with a single PC/SC device, you would typically use something like
+`./pySim-smpp2sim.py -p0` to start the program.  You will see output like this at start-up
+::
+
+  Using reader PCSC[HID Global OMNIKEY 3x21 Smart Card Reader [OMNIKEY 3x21 Smart Card Reader] 00 00]
+  INFO     root: Binding Virtual SMSC to TCP Port 2775 at ::
+
+The application has hence bound to local TCP port 2775 and expects your SMS-sending applications to send their
+SMS there.  Once you do, you will see log output like below:
+::
+
+  WARNING  smpp.twisted.protocol: SMPP connection established from ::ffff:127.0.0.1 to port 2775
+  INFO     smpp.twisted.server: Added CommandId.bind_transceiver bind for 'test'. Active binds: CommandId.bind_transceiver: 1, CommandId.bind_transmitter: 0, CommandId.bind_receiver: 0. Max binds: 2
+  INFO     smpp.twisted.protocol: Bind request succeeded for test. 1 active binds
+
+And once your external program is sending SMS to the simulated SMSC, it will log something like
+::
+
+  INFO     root: SMS_DELIVER(MTI=0, MMS=False, LP=False, RP=False, UDHI=True, SRI=False, OA=AddressField(TON=international, NPI=unknown, 12), PID=7f, DCS=f6, SCTS=bytearray(b'"pR\x00\x00\x00\x00'), UDL=45, UD=b"\x02p\x00\x00(\x15\x16\x19\x12\x12\xb0\x00\x01'\xfa(\xa5\xba\xc6\x9d<^\x9d\xf2\xc7\x15]\xfd\xdeD\x9c\x82k#b\x15Ve0x{0\xe8\xbe]")
+  SMSPPDownload(DeviceIdentities({'source_dev_id': 'network', 'dest_dev_id': 'uicc'}),Address({'ton_npi': 0, 'call_number': '0123456'}),SMS_TPDU({'tpdu': '400290217ff6227052000000002d02700000281516191212b0000127fa28a5bac69d3c5e9df2c7155dfdde449c826b236215566530787b30e8be5d'}))
+  INFO     root: ENVELOPE: d147820283818604001032548b3b400290217ff6227052000000002d02700000281516191212b0000127fa28a5bac69d3c5e9df2c7155dfdde449c826b236215566530787b30e8be5d
+  INFO     root: SW 9000: 027100002412b000019a551bb7c28183652de0ace6170d0e563c5e949a3ba56747fe4c1dbbef16642c

docs/suci-keytool.rst

@@ -0,0 +1,58 @@
+suci-keytool
+============
+
+Subscriber concealment is an important feature of the 5G SA architecture: It avoids the many privacy
+issues associated with having a permanent identifier (SUPI, traditionally the IMSI) transmitted in plain text
+over the air interface.  Using SUCI solves this issue not just for the air interface; it even ensures the SUPI/IMSI
+is not known to the visited network (VPLMN) at all.
+
+In principle, the SUCI mechanism works by encrypting the SUPI by asymmetric (public key) cryptography:
+Only the HPLMN is in possession of the private key and hence can decrypt the SUCI to the SUPI, while
+each subscriber has the public key in order to encrypt their SUPI into the SUCI.  In reality, the
+details are more complex, as there are ephemeral keys and cryptographic MAC involved.
+
+In any case, in order to operate a SUCI-enabled 5G SA network, you will have to
+
+#. generate a ECC key pair of public + private key
+#. deploy the public key on your USIMs
+#. deploy the private key on your 5GC, specifically the UDM function
+
+pysim contains (in its `contrib` directory) a small utility program that can make it easy to generate
+such keys: `suci-keytool.py`
+
+Generating keys
+~~~~~~~~~~~~~~~
+
+Example: Generating a *secp256r1* ECC public key pair and storing it to `/tmp/suci.key`:
+::
+
+        $ ./contrib/suci-keytool.py --key-file /tmp/suci.key generate-key --curve secp256r1
+
+Dumping public keys
+~~~~~~~~~~~~~~~~~~~
+
+In order to store the key to SIM cards as part of `ADF.USIM/DF.5GS/EF.SUCI_Calc_Info`, you will need
+a hexadecimal representation of the public key.  You can achieve that using the `dump-pub-key` operation
+of suci-keytool:
+
+Example: Dumping the public key part from a previously generated key file:
+::
+
+        $ ./contrib/suci-keytool.py --key-file /tmp/suci.key dump-pub-key
+        0473152f32523725f5175d255da2bd909de97b1d06449a9277bc629fe42112f8643e6b69aa6dce6c86714ccbe6f2e0f4f4898d102e2b3f0c18ce26626f052539bb
+
+If you want the point-compressed representation, you can use the `--compressed` option:
+::
+
+        $ ./contrib/suci-keytool.py --key-file /tmp/suci.key dump-pub-key --compressed
+        0373152f32523725f5175d255da2bd909de97b1d06449a9277bc629fe42112f864
+
+
+
+suci-keytool syntax
+~~~~~~~~~~~~~~~~~~~
+
+.. argparse::
+   :module: contrib.suci-keytool
+   :func: arg_parser
+   :prog: contrib/suci-keytool.py

docs/suci-tutorial.rst

@@ -0,0 +1,270 @@
+
+Guide: Enabling 5G SUCI
+=======================
+
+SUPI/SUCI Concealment is a feature of 5G-Standalone (SA) to encrypt the
+IMSI/SUPI with a network operator public key.  3GPP Specifies two different
+variants for this:
+
+* SUCI calculation *in the UE*, using key data from the SIM
+* SUCI calculation *on the card itself*
+
+pySim supports writing the 5G-specific files for *SUCI calculation in the UE* on USIM cards, assuming
+that your cards contain the required files, and you have the privileges/credentials to write to them.
+This is the case using sysmocom sysmoISIM-SJA2 or any flavor of sysmoISIM-SJA5.
+
+There is no 3GPP/ETSI standard method for configuring *SUCI calculation on the card*; pySim currently
+supports the vendor-specific method for the sysmoISIM-SJA5-S17).
+
+This document describes both methods.
+
+
+Technical References
+~~~~~~~~~~~~~~~~~~~~
+
+This guide covers the basic workflow of provisioning SIM cards with the 5G SUCI feature. For detailed information on the SUCI feature and file contents, the following documents are helpful:
+
+* USIM files and structure: `3GPP TS 31.102 <https://www.etsi.org/deliver/etsi_ts/131100_131199/131102/16.06.00_60/ts_131102v160600p.pdf>`__
+* USIM tests (incl. file content examples): `3GPP TS 31.121 <https://www.etsi.org/deliver/etsi_ts/131100_131199/131121/16.01.00_60/ts_131121v160100p.pdf>`__
+* Test keys for SUCI calculation: `3GPP TS 33.501 <https://www.etsi.org/deliver/etsi_ts/133500_133599/133501/16.05.00_60/ts_133501v160500p.pdf>`__
+
+For specific information on sysmocom SIM cards, refer to
+
+* the `sysmoISIM-SJA5 User Manual <https://sysmocom.de/manuals/sysmoisim-sja5-manual.pdf>`__ for the current
+  sysmoISIM-SJA5 product
+* the `sysmoISIM-SJA2 User Manual <https://sysmocom.de/manuals/sysmousim-manual.pdf>`__ for the older
+  sysmoISIM-SJA2 product
+
+--------------
+
+
+Enabling 5G SUCI *calculated in the UE*
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In short, you can enable *SUCI calculation in the UE* with these steps:
+
+* activate USIM **Service 124**
+* make sure USIM **Service 125** is disabled
+* store the public keys in **EF.SUCI_Calc_Info**
+* set the **Routing Indicator** (required)
+
+If you want to disable the feature, you can just disable USIM Service 124 (and 125) in `EF.UST`.
+
+
+Admin PIN
+---------
+
+The usual way to authenticate yourself to the card as the cellular
+operator is to validate the so-called ADM1 (admin) PIN.  This may differ
+from card model/vendor to card model/vendor.
+
+Start pySIM-shell and enter the admin PIN for your card. If you bought
+the SIM card from your network operator and don’t have the admin PIN,
+you cannot change SIM contents!
+
+Launch pySIM:
+
+::
+
+    $ ./pySim-shell.py -p 0
+
+    Using PC/SC reader interface
+    Autodetected card type: sysmoISIM-SJA2
+    Welcome to pySim-shell!
+    pySIM-shell (00:MF)>
+
+Enter the ADM PIN:
+
+::
+
+   pySIM-shell (00:MF)> verify_adm XXXXXXXX
+
+Otherwise, write commands will fail with ``SW Mismatch: Expected 9000 and got 6982.``
+
+Key Provisioning
+----------------
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select DF.5GS
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS)> select EF.SUCI_Calc_Info
+
+By default, the file is present but empty:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.SUCI_Calc_Info)> read_binary_decoded
+   missing Protection Scheme Identifier List data object tag
+   9000: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff -> {}
+
+The following JSON config defines the testfile from 3GPP TS 31.121, Section 4.9.4 with
+test keys from 3GPP TS 33.501, Annex C.4. Highest priority (``0``) has a
+Profile-B (``identifier: 2``) key in key slot ``1``, which means the key
+with ``hnet_pubkey_identifier: 27``.
+
+.. code:: json
+
+   {
+        "prot_scheme_id_list": [
+           {"priority": 0, "identifier": 2, "key_index": 1},
+           {"priority": 1, "identifier": 1, "key_index": 2},
+           {"priority": 2, "identifier": 0, "key_index": 0}],
+        "hnet_pubkey_list": [
+           {"hnet_pubkey_identifier": 27,
+            "hnet_pubkey": "0472DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD15A7DED52FCBB097A4ED250E036C7B9C8C7004C4EEDC4F068CD7BF8D3F900E3B4"},
+           {"hnet_pubkey_identifier": 30,
+            "hnet_pubkey": "5A8D38864820197C3394B92613B20B91633CBD897119273BF8E4A6F4EEC0A650"}]
+   }
+
+Write the config to file (must be single-line input as for now):
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.SUCI_Calc_Info)> update_binary_decoded '{ "prot_scheme_id_list": [ {"priority": 0, "identifier": 2, "key_index": 1}, {"priority": 1, "identifier": 1, "key_index": 2}, {"priority": 2, "identifier": 0, "key_index": 0}], "hnet_pubkey_list": [ {"hnet_pubkey_identifier": 27, "hnet_pubkey": "0472DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD15A7DED52FCBB097A4ED250E036C7B9C8C7004C4EEDC4F068CD7BF8D3F900E3B4"}, {"hnet_pubkey_identifier": 30, "hnet_pubkey": "5A8D38864820197C3394B92613B20B91633CBD897119273BF8E4A6F4EEC0A650"}]}'
+
+WARNING: These are TEST KEYS with publicly known/specified private keys, and hence unsafe for live/secure
+deployments! For use in production networks, you need to generate your own set[s] of keys.
+
+Routing Indicator
+-----------------
+
+The Routing Indicator must be present for the SUCI feature. By default,
+the contents of the file is **invalid** (ffffffff):
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select DF.5GS
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS)> select EF.Routing_Indicator
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.Routing_Indicator)> read_binary_decoded
+   9000: ffffffff -> {'raw': 'ffffffff'}
+
+The Routing Indicator is a four-byte file but the actual Routing
+Indicator goes into bytes 0 and 1 (the other bytes are reserved). To set
+the Routing Indicator to 0x71:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/DF.5GS/EF.Routing_Indicator)> update_binary 17ffffff
+
+You can also set the routing indicator to **0x0**, which is *valid* and
+means “routing indicator not specified”, leaving it to the modem.
+
+USIM Service Table
+------------------
+
+First, check out the USIM Service Table (UST):
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select EF.UST
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> read_binary_decoded
+   9000: beff9f9de73e0408400170730000002e00000000 -> [2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 25, 27, 28, 29, 33, 34, 35, 38, 39, 42, 43, 44, 45, 46, 51, 60, 71, 73, 85, 86, 87, 89, 90, 93, 94, 95, 122, 123, 124, 126]
+
+.. list-table:: From 3GPP TS 31.102
+   :widths: 15 40
+   :header-rows: 1
+
+   * - Service No.
+     - Description
+   * - 122
+     - 5GS Mobility Management Information
+   * - 123
+     - 5G Security Parameters
+   * - 124
+     - Subscription identifier privacy support
+   * - 125
+     - SUCI calculation by the USIM
+   * - 126
+     - UAC Access Identities support
+   * - 129
+     - 5GS Operator PLMN List
+
+If you’d like to enable/disable any UST service:
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_deactivate 124
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_activate 124
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_deactivate 125
+
+In this case, UST Service 124 is already enabled and you’re good to go. The
+sysmoISIM-SJA2 does not support on-SIM calculation, so service 125 must
+be disabled.
+
+USIM Error with 5G and sysmoISIM
+--------------------------------
+
+sysmoISIM-SJA2 come 5GS-enabled. By default however, the configuration stored
+in the card file-system is **not valid** for 5G networks: Service 124 is enabled,
+but EF.SUCI_Calc_Info and EF.Routing_Indicator are empty files (hence
+do not contain valid data).
+
+At least for Qualcomm’s X55 modem, this results in an USIM error and the
+whole modem shutting 5G down. If you don’t need SUCI concealment but the
+smartphone refuses to connect to any 5G network, try to disable the UST
+service 124.
+
+sysmoISIM-SJA5 are shipped with a more forgiving default, with valid EF.Routing_Indicator
+contents and disabled Service 124
+
+
+SUCI calculation by the USIM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The SUCI calculation can also be performed by the USIM application on the UICC
+directly. The UE then uses the GET IDENTITY command (see also 3GPP TS 31.102,
+section 7.5) to retrieve a SUCI value.
+
+The sysmoISIM-SJA5-S17 supports *SUCI calculation by the USIM*. The configuration
+is not much different to the above described configuration of *SUCI calculation
+in the UE*.
+
+The main difference is how the key provisioning is done. When the SUCI
+calculation is done by the USIM, then the key material is not accessed by the
+UE. The specification (see also 3GPP TS 31.102, section 7.5.1.1), also does not
+specify any file or file format to store the key material. This means the exact
+way to perform the key provisioning is an implementation detail of the USIM
+card application.
+
+In the case of sysmoISIM-SJA5-S17, the key material for *SUCI calculation by the USIM* is stored in
+`ADF.USIM/DF.SAIP/EF.SUCI_Calc_Info` (**not** in `ADF.USIM/DF.5GS/EF.SUCI_Calc_Info`!).
+
+::
+
+   pySIM-shell (00:MF)> select MF
+   pySIM-shell (00:MF)> select ADF.USIM
+   pySIM-shell (00:MF/ADF.USIM)> select DF.SAIP
+   pySIM-shell (00:MF/ADF.USIM/DF.SAIP)> select EF.SUCI_Calc_Info
+
+The file format is exactly the same as specified in 3GPP TS 31.102, section
+4.4.11.8. This means the above described key provisioning procedure can be
+applied without any changes, except that the file location is different.
+
+To signal to the UE that the USIM is setup up for SUCI calculation, service
+125 must be enabled in addition to service 124 (see also 3GPP TS 31.102,
+section 5.3.48)
+
+::
+
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_activate 124
+   pySIM-shell (00:MF/ADF.USIM/EF.UST)> ust_service_activate 125
+
+To verify that the SUCI calculation works as expected, it is possible to issue
+a GET IDENTITY command using pySim-shell:
+
+::
+
+   select ADF.USIM
+   get_identity
+
+The USIM should then return a SUCI TLV Data object that looks like this:
+
+::
+
+   SUCI TLV Data Object: 0199f90717ff021b027a2c58ce1c6b89df088a9eb4d242596dd75746bb5f3503d2cf58a7461e4fd106e205c86f76544e9d732226a4e1

docs/trace.rst

@@ -0,0 +1,64 @@
+pySim-trace
+===========
+
+pySim-trace is a utility for high-level decode of APDU protocol traces such as those obtained with
+`Osmocom SIMtrace2 <https://osmocom.org/projects/simtrace2/wiki>`_ or `osmo-qcdiag <https://osmocom.org/projects/osmo-qcdiag/wiki>`_.
+
+pySim-trace leverages the existing knowledge of pySim-shell on anything related to SIM cards,
+including the structure/encoding of the various files on SIM/USIM/ISIM/HPSIM cards, and applies this
+to decoding protocol traces.  This means that it shows not only the name of the command (like READ
+BINARY), but actually understands what the currently selected file is, and how to decode the
+contents of that file.
+
+pySim-trace also understands the parameters passed to commands and how to decode them, for example
+of the AUTHENTICATE command within the USIM/ISIM/HPSIM application.
+
+
+Demo
+----
+
+To get an idea how pySim-trace usage looks like, you can watch the relevant part of the 11/2022
+SIMtrace2 tutorial whose `recording is freely accessible <https://media.ccc.de/v/osmodevcall-20221019-laforge-simtrace2-tutorial#t=2134>`_.
+
+
+Running pySim-trace
+-------------------
+
+Running pySim-trace requires you to specify the *source* of the to-be-decoded APDUs.  There are several
+supported options, each with their own respective parameters (like a file name for PCAP decoding).
+
+See the detailed command line reference below for details.
+
+A typical execution of pySim-trace for doing live decodes of *GSMTAP (SIM APDU)* e.g. from SIMtrace2 or
+osmo-qcdiag would look like this:
+
+::
+
+  ./pySim-trace.py gsmtap-udp
+
+This binds to the default UDP port 4729 (GSMTAP) on localhost (127.0.0.1), and decodes any APDUs received
+there.
+
+
+
+pySim-trace command line reference
+----------------------------------
+
+.. argparse::
+   :module: pySim-trace
+   :func: option_parser
+   :prog: pySim-trace.py
+
+
+Constraints
+-----------
+
+* In order to properly track the current location in the filesystem tree and other state, it is
+  important that the trace you're decoding includes all of the communication with the SIM, ideally
+  from the very start (power up).
+
+* pySim-trace currently only supports ETSI UICC (USIM/ISIM/HPSIM) and doesn't yet support legacy GSM
+  SIM.  This is not a fundamental technical constraint, it's just simply that nobody got around
+  developing and testing that part. Contributions are most welcome.
+
+

osmo-smdpp.py

@@ -0,0 +1,913 @@
+#!/usr/bin/env python3
+
+# Early proof-of-concept towards a SM-DP+ HTTP service for GSMA consumer eSIM RSP
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# asn1tools issue https://github.com/eerimoq/asn1tools/issues/194
+# must be first here
+import asn1tools
+import asn1tools.codecs.ber
+import asn1tools.codecs.der
+# do not move the code
+def fix_asn1_oid_decoding():
+    fix_asn1_schema = """
+    TestModule DEFINITIONS ::= BEGIN
+        TestOid ::= SEQUENCE {
+            oid OBJECT IDENTIFIER
+        }
+    END
+    """
+
+    fix_asn1_asn1 = asn1tools.compile_string(fix_asn1_schema, codec='der')
+    fix_asn1_oid_string = '2.999.10'
+    fix_asn1_encoded = fix_asn1_asn1.encode('TestOid', {'oid': fix_asn1_oid_string})
+    fix_asn1_decoded = fix_asn1_asn1.decode('TestOid', fix_asn1_encoded)
+
+    if (fix_asn1_decoded['oid'] != fix_asn1_oid_string):
+        # ASN.1 OBJECT IDENTIFIER Decoding Issue:
+        #
+        # In ASN.1 BER/DER encoding, the first two arcs of an OBJECT IDENTIFIER are
+        # combined into a single value: (40 * arc0) + arc1. This is encoded as a base-128
+        # variable-length quantity (and commonly known as VLQ or base-128 encoding)
+        # as specified in ITU-T X.690 §8.19, it can span multiple bytes if
+        # the value is large.
+        #
+        # For arc0 = 0 or 1, arc1 must be in [0, 39]. For arc0 = 2, arc1 can be any non-negative integer.
+        # All subsequent arcs (arc2, arc3, ...) are each encoded as a separate base-128 VLQ.
+        #
+        # The decoding bug occurs when the decoder does not properly split the first
+        # subidentifier for arc0 = 2 and arc1 >= 40. Instead of decoding:
+        #   - arc0 = 2
+        #   - arc1 = (first_subidentifier - 80)
+        # it may incorrectly interpret the first_subidentifier as arc0 = (first_subidentifier // 40),
+        # arc1 = (first_subidentifier % 40), which is only valid for arc1 < 40.
+        #
+        # This patch handles it properly for all valid OBJECT IDENTIFIERs
+        # with large second arcs, by applying the ASN.1 rules:
+        #   - if first_subidentifier < 40: arc0 = 0, arc1 = first_subidentifier
+        #   - elif first_subidentifier < 80: arc0 = 1, arc1 = first_subidentifier - 40
+        #   - else: arc0 = 2, arc1 = first_subidentifier - 80
+        #
+        # This problem is not uncommon, see for example https://github.com/randombit/botan/issues/4023
+
+        def fixed_decode_object_identifier(data, offset, end_offset):
+            """Decode ASN.1 OBJECT IDENTIFIER from bytes to dotted string, fixing large second arc handling."""
+            def read_subidentifier(data, offset):
+                value = 0
+                while True:
+                    b = data[offset]
+                    value = (value << 7) | (b & 0x7F)
+                    offset += 1
+                    if not (b & 0x80):
+                        break
+                return value, offset
+
+            subid, offset = read_subidentifier(data, offset)
+            if subid < 40:
+                first = 0
+                second = subid
+            elif subid < 80:
+                first = 1
+                second = subid - 40
+            else:
+                first = 2
+                second = subid - 80
+            arcs = [first, second]
+
+            while offset < end_offset:
+                subid, offset = read_subidentifier(data, offset)
+                arcs.append(subid)
+
+            return '.'.join(str(x) for x in arcs)
+
+        asn1tools.codecs.ber.decode_object_identifier = fixed_decode_object_identifier
+        asn1tools.codecs.der.decode_object_identifier = fixed_decode_object_identifier
+
+        # test our patch
+        asn1 = asn1tools.compile_string(fix_asn1_schema, codec='der')
+        decoded = asn1.decode('TestOid', fix_asn1_encoded)['oid']
+        assert fix_asn1_oid_string == str(decoded)
+
+fix_asn1_oid_decoding()
+
+from cryptography.hazmat.primitives.asymmetric.utils import encode_dss_signature # noqa: E402
+from cryptography import x509 # noqa: E402
+from cryptography.exceptions import InvalidSignature # noqa: E402
+from cryptography.hazmat.primitives import hashes # noqa: E402
+from cryptography.hazmat.primitives.asymmetric import ec, dh # noqa: E402
+from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat, PrivateFormat, NoEncryption, ParameterFormat # noqa: E402
+from pathlib import Path # noqa: E402
+import json # noqa: E402
+import sys # noqa: E402
+import argparse # noqa: E402
+import uuid # noqa: E402
+import os # noqa: E402
+import functools # noqa: E402
+from typing import Optional, Dict, List # noqa: E402
+from pprint import pprint as pp # noqa: E402
+
+import base64 # noqa: E402
+from base64 import b64decode # noqa: E402
+from klein import Klein # noqa: E402
+from twisted.web.iweb import IRequest # noqa: E402
+
+from osmocom.utils import h2b, b2h, swap_nibbles # noqa: E402
+
+import pySim.esim.rsp as rsp # noqa: E402
+from pySim.esim import saip, PMO # noqa: E402
+from pySim.esim.es8p import ProfileMetadata,UnprotectedProfilePackage,ProtectedProfilePackage,BoundProfilePackage,BspInstance # noqa: E402
+from pySim.esim.x509_cert import oid, cert_policy_has_oid, cert_get_auth_key_id # noqa: E402
+from pySim.esim.x509_cert import CertAndPrivkey, CertificateSet, cert_get_subject_key_id, VerifyError # noqa: E402
+
+import logging # noqa: E402
+logger = logging.getLogger(__name__)
+
+# HACK: make this configurable
+DATA_DIR = './smdpp-data'
+HOSTNAME = 'testsmdpplus1.example.com' # must match certificates!
+
+
+def b64encode2str(req: bytes) -> str:
+    """Encode given input bytes as base64 and return result as string."""
+    return base64.b64encode(req).decode('ascii')
+
+def set_headers(request: IRequest):
+    """Set the request headers as mandatory by GSMA eSIM RSP."""
+    request.setHeader('Content-Type', 'application/json;charset=UTF-8')
+    request.setHeader('X-Admin-Protocol', 'gsma/rsp/v2.1.0')
+
+def validate_request_headers(request: IRequest):
+    """Validate mandatory HTTP headers according to SGP.22."""
+    content_type = request.getHeader('Content-Type')
+    if not content_type or not content_type.startswith('application/json'):
+        raise ApiError('1.2.1', '2.1', 'Invalid Content-Type header')
+
+    admin_protocol = request.getHeader('X-Admin-Protocol')
+    if admin_protocol and not admin_protocol.startswith('gsma/rsp/v'):
+        raise ApiError('1.2.2', '2.1', 'Unsupported X-Admin-Protocol version')
+
+def get_eum_certificate_variant(eum_cert) -> str:
+    """Determine EUM certificate variant by checking Certificate Policies extension.
+    Returns 'O' for old variant, or 'NEW' for Ov3/A/B/C variants."""
+
+    try:
+        cert_policies_ext = eum_cert.extensions.get_extension_for_oid(
+            x509.oid.ExtensionOID.CERTIFICATE_POLICIES
+        )
+
+        for policy in cert_policies_ext.value:
+            policy_oid = policy.policy_identifier.dotted_string
+            logger.debug(f"Found certificate policy: {policy_oid}")
+
+            if policy_oid == '2.23.146.1.2.1.2':
+                logger.debug("Detected EUM certificate variant: O (old)")
+                return 'O'
+            elif policy_oid == '2.23.146.1.2.1.0.0.0':
+                logger.debug("Detected EUM certificate variant: Ov3/A/B/C (new)")
+                return 'NEW'
+    except x509.ExtensionNotFound:
+        logger.debug("No Certificate Policies extension found")
+    except Exception as e:
+        logger.debug(f"Error checking certificate policies: {e}")
+
+def parse_permitted_eins_from_cert(eum_cert) -> List[str]:
+    """Extract permitted IINs from EUM certificate using the appropriate method
+    based on certificate variant (O vs Ov3/A/B/C).
+    Returns list of permitted IINs (basically prefixes that valid EIDs must start with)."""
+
+    # Determine certificate variant first
+    cert_variant = get_eum_certificate_variant(eum_cert)
+    permitted_iins = []
+
+    if cert_variant == 'O':
+        # Old variant - use nameConstraints extension
+        permitted_iins.extend(_parse_name_constraints_eins(eum_cert))
+
+    else:
+        # New variants (Ov3, A, B, C) - use GSMA permittedEins extension
+        permitted_iins.extend(_parse_gsma_permitted_eins(eum_cert))
+
+    unique_iins = list(set(permitted_iins))
+
+    logger.debug(f"Total unique permitted IINs found: {len(unique_iins)}")
+    return unique_iins
+
+def _parse_gsma_permitted_eins(eum_cert) -> List[str]:
+    """Parse the GSMA permittedEins extension using correct ASN.1 structure.
+    PermittedEins ::= SEQUENCE OF PrintableString
+    Each string contains an IIN (Issuer Identification Number) - a prefix of valid EIDs."""
+    permitted_iins = []
+
+    try:
+        permitted_eins_oid = x509.ObjectIdentifier('2.23.146.1.2.2.0')  # sgp26: 2.23.146.1.2.2.0 = ASN1:SEQUENCE:permittedEins
+
+        for ext in eum_cert.extensions:
+            if ext.oid == permitted_eins_oid:
+                logger.debug(f"Found GSMA permittedEins extension: {ext.oid}")
+
+                # Get the DER-encoded extension value
+                ext_der = ext.value.value if hasattr(ext.value, 'value') else ext.value
+
+                if isinstance(ext_der, bytes):
+                    try:
+                        permitted_eins_schema = """
+                        PermittedEins DEFINITIONS ::= BEGIN
+                            PermittedEins ::= SEQUENCE OF PrintableString
+                        END
+                        """
+                        decoder = asn1tools.compile_string(permitted_eins_schema)
+                        decoded_strings = decoder.decode('PermittedEins', ext_der)
+
+                        for iin_string in decoded_strings:
+                            # Each string contains an IIN -> prefix of euicc EID
+                            iin_clean = iin_string.strip().upper()
+
+                            # IINs is 8 chars per sgp22, var len according to sgp29, fortunately we don't care
+                            if (len(iin_clean) == 8 and
+                                all(c in '0123456789ABCDEF' for c in iin_clean) and
+                                    len(iin_clean) % 2 == 0):
+                                permitted_iins.append(iin_clean)
+                                logger.debug(f"Found permitted IIN (GSMA): {iin_clean}")
+                            else:
+                                logger.debug(f"Invalid IIN format: {iin_string} (cleaned: {iin_clean})")
+                    except Exception as e:
+                        logger.debug(f"Error parsing GSMA permittedEins extension: {e}")
+
+    except Exception as e:
+        logger.debug(f"Error accessing GSMA certificate extensions: {e}")
+
+    return permitted_iins
+
+
+def _parse_name_constraints_eins(eum_cert) -> List[str]:
+    """Parse permitted IINs from nameConstraints extension (variant O)."""
+    permitted_iins = []
+
+    try:
+        # Look for nameConstraints extension
+        name_constraints_ext = eum_cert.extensions.get_extension_for_oid(
+            x509.oid.ExtensionOID.NAME_CONSTRAINTS
+        )
+
+        name_constraints = name_constraints_ext.value
+
+        # Check permittedSubtrees for IIN constraints
+        if name_constraints.permitted_subtrees:
+            for subtree in name_constraints.permitted_subtrees:
+
+                if isinstance(subtree, x509.DirectoryName):
+                    for attribute in subtree.value:
+                        # IINs for O in serialNumber
+                        if attribute.oid == x509.oid.NameOID.SERIAL_NUMBER:
+                            serial_value = attribute.value.upper()
+                            # sgp22 8, sgp29 var len, fortunately we don't care
+                            if (len(serial_value) == 8 and
+                                all(c in '0123456789ABCDEF' for c in serial_value) and
+                                    len(serial_value) % 2 == 0):
+                                permitted_iins.append(serial_value)
+                                logger.debug(f"Found permitted IIN (nameConstraints/DN): {serial_value}")
+
+    except x509.ExtensionNotFound:
+        logger.debug("No nameConstraints extension found")
+    except Exception as e:
+        logger.debug(f"Error parsing nameConstraints: {e}")
+
+    return permitted_iins
+
+
+def validate_eid_range(eid: str, eum_cert) -> bool:
+    """Validate that EID is within the permitted EINs of the EUM certificate."""
+    if not eid or len(eid) != 32:
+        logger.debug(f"Invalid EID format: {eid}")
+        return False
+
+    try:
+        permitted_eins = parse_permitted_eins_from_cert(eum_cert)
+
+        if not permitted_eins:
+            logger.debug("Warning: No permitted EINs found in EUM certificate")
+            return False
+
+        eid_normalized = eid.upper()
+        logger.debug(f"Validating EID {eid_normalized} against {len(permitted_eins)} permitted EINs")
+
+        for permitted_ein in permitted_eins:
+                if eid_normalized.startswith(permitted_ein):
+                    logger.debug(f"EID {eid_normalized} matches permitted EIN {permitted_ein}")
+                    return True
+
+        logger.debug(f"EID {eid_normalized} is not in any permitted EIN list")
+        return False
+
+    except Exception as e:
+        logger.debug(f"Error validating EID: {e}")
+        return False
+
+def build_status_code(subject_code: str, reason_code: str, subject_id: Optional[str], message: Optional[str]) -> Dict:
+    r = {'subjectCode': subject_code, 'reasonCode': reason_code }
+    if subject_id:
+        r['subjectIdentifier'] = subject_id
+    if message:
+        r['message'] = message
+    return r
+
+def build_resp_header(js: dict, status: str = 'Executed-Success', status_code_data = None) -> None:
+    # SGP.22 v3.0 6.5.1.4
+    js['header'] = {
+        'functionExecutionStatus': {
+            'status': status,
+        }
+    }
+    if status_code_data:
+        js['header']['functionExecutionStatus']['statusCodeData'] = status_code_data
+
+
+def ecdsa_tr03111_to_dss(sig: bytes) -> bytes:
+    """convert an ECDSA signature from BSI TR-03111 format to DER: first get long integers; then encode those."""
+    assert len(sig) == 64
+    r = int.from_bytes(sig[0:32], 'big')
+    s = int.from_bytes(sig[32:32*2], 'big')
+    return encode_dss_signature(r, s)
+
+
+class ApiError(Exception):
+    def __init__(self, subject_code: str, reason_code: str, message: Optional[str] = None,
+                 subject_id: Optional[str] = None):
+        self.status_code = build_status_code(subject_code, reason_code, subject_id, message)
+
+    def encode(self) -> str:
+        """Encode the API Error into a responseHeader string."""
+        js = {}
+        build_resp_header(js, 'Failed', self.status_code)
+        return json.dumps(js)
+
+class SmDppHttpServer:
+    app = Klein()
+
+    @staticmethod
+    def load_certs_from_path(path: str) -> List[x509.Certificate]:
+        """Load all DER + PEM files from given directory path and return them as list of x509.Certificate
+        instances."""
+        certs = []
+        for dirpath, dirnames, filenames in os.walk(path):
+            for filename in filenames:
+                cert = None
+                if filename.endswith('.der'):
+                    with open(os.path.join(dirpath, filename), 'rb') as f:
+                        cert = x509.load_der_x509_certificate(f.read())
+                elif filename.endswith('.pem'):
+                    with open(os.path.join(dirpath, filename), 'rb') as f:
+                        cert = x509.load_pem_x509_certificate(f.read())
+                if cert:
+                    # verify it is a CI certificate (keyCertSign + i-rspRole-ci)
+                    if not cert_policy_has_oid(cert, oid.id_rspRole_ci):
+                        raise ValueError("alleged CI certificate %s doesn't have CI policy" % filename)
+                    certs.append(cert)
+        return certs
+
+    def ci_get_cert_for_pkid(self, ci_pkid: bytes) -> Optional[x509.Certificate]:
+        """Find CI certificate for given key identifier."""
+        for cert in self.ci_certs:
+            logger.debug("cert: %s" % cert)
+            subject_exts = list(filter(lambda x: isinstance(x.value, x509.SubjectKeyIdentifier), cert.extensions))
+            logger.debug(subject_exts)
+            subject_pkid = subject_exts[0].value
+            logger.debug(subject_pkid)
+            if subject_pkid and subject_pkid.key_identifier == ci_pkid:
+                return cert
+        return None
+
+    def validate_certificate_chain_for_verification(self, euicc_ci_pkid_list: List[bytes]) -> bool:
+        """Validate that SM-DP+ has valid certificate chains for the given CI PKIDs."""
+        for ci_pkid in euicc_ci_pkid_list:
+            ci_cert = self.ci_get_cert_for_pkid(ci_pkid)
+            if ci_cert:
+                # Check if our DPauth certificate chains to this CI
+                try:
+                    cs = CertificateSet(ci_cert)
+                    cs.verify_cert_chain(self.dp_auth.cert)
+                    return True
+                except VerifyError:
+                    continue
+        return False
+
+    def __init__(self, server_hostname: str, ci_certs_path: str, common_cert_path: str, use_brainpool: bool = False, in_memory: bool = False):
+        self.server_hostname = server_hostname
+        self.upp_dir = os.path.realpath(os.path.join(DATA_DIR, 'upp'))
+        self.ci_certs = self.load_certs_from_path(ci_certs_path)
+        # load DPauth cert + key
+        self.dp_auth = CertAndPrivkey(oid.id_rspRole_dp_auth_v2)
+        cert_dir = common_cert_path
+        if use_brainpool:
+            self.dp_auth.cert_from_der_file(os.path.join(cert_dir, 'DPauth', 'CERT_S_SM_DPauth_ECDSA_BRP.der'))
+            self.dp_auth.privkey_from_pem_file(os.path.join(cert_dir, 'DPauth', 'SK_S_SM_DPauth_ECDSA_BRP.pem'))
+        else:
+            self.dp_auth.cert_from_der_file(os.path.join(cert_dir, 'DPauth', 'CERT_S_SM_DPauth_ECDSA_NIST.der'))
+            self.dp_auth.privkey_from_pem_file(os.path.join(cert_dir, 'DPauth', 'SK_S_SM_DPauth_ECDSA_NIST.pem'))
+        # load DPpb cert + key
+        self.dp_pb = CertAndPrivkey(oid.id_rspRole_dp_pb_v2)
+        if use_brainpool:
+            self.dp_pb.cert_from_der_file(os.path.join(cert_dir, 'DPpb', 'CERT_S_SM_DPpb_ECDSA_BRP.der'))
+            self.dp_pb.privkey_from_pem_file(os.path.join(cert_dir, 'DPpb', 'SK_S_SM_DPpb_ECDSA_BRP.pem'))
+        else:
+            self.dp_pb.cert_from_der_file(os.path.join(cert_dir, 'DPpb', 'CERT_S_SM_DPpb_ECDSA_NIST.der'))
+            self.dp_pb.privkey_from_pem_file(os.path.join(cert_dir, 'DPpb', 'SK_S_SM_DPpb_ECDSA_NIST.pem'))
+        if in_memory:
+            self.rss = rsp.RspSessionStore(in_memory=True)
+            logger.info("Using in-memory session storage")
+        else:
+            # Use different session database files for BRP and NIST to avoid file locking during concurrent runs
+            session_db_suffix = "BRP" if use_brainpool else "NIST"
+            db_path = os.path.join(DATA_DIR, f"sm-dp-sessions-{session_db_suffix}")
+            self.rss = rsp.RspSessionStore(filename=db_path, in_memory=False)
+            logger.info(f"Using file-based session storage: {db_path}")
+
+    @app.handle_errors(ApiError)
+    def handle_apierror(self, request: IRequest, failure):
+        request.setResponseCode(200)
+        pp(failure)
+        return failure.value.encode()
+
+    @staticmethod
+    def _ecdsa_verify(cert: x509.Certificate, signature: bytes, data: bytes) -> bool:
+        pubkey = cert.public_key()
+        dss_sig = ecdsa_tr03111_to_dss(signature)
+        try:
+            pubkey.verify(dss_sig, data, ec.ECDSA(hashes.SHA256()))
+            return True
+        except InvalidSignature:
+            return False
+
+    @staticmethod
+    def rsp_api_wrapper(func):
+        """Wrapper that can be used as decorator in order to perform common REST API endpoint entry/exit
+        functionality, such as JSON decoding/encoding and debug-printing."""
+        @functools.wraps(func)
+        def _api_wrapper(self, request: IRequest):
+            validate_request_headers(request)
+
+            content = json.loads(request.content.read())
+            logger.debug("Rx JSON: %s" % json.dumps(content))
+            set_headers(request)
+
+            output = func(self, request, content)
+            if output == None:
+                return ''
+
+            build_resp_header(output)
+            logger.debug("Tx JSON: %s" % json.dumps(output))
+            return json.dumps(output)
+        return _api_wrapper
+
+    @app.route('/gsma/rsp2/es9plus/initiateAuthentication', methods=['POST'])
+    @rsp_api_wrapper
+    def initiateAutentication(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ InitiateAuthentication SGP.22 Section 5.6.1"""
+        # Verify that the received address matches its own SM-DP+ address, where the comparison SHALL be
+        # case-insensitive. Otherwise, the SM-DP+ SHALL return a status code "SM-DP+ Address - Refused".
+        if content['smdpAddress'] != self.server_hostname:
+           raise ApiError('8.8.1', '3.8', 'Invalid SM-DP+ Address')
+
+        euiccChallenge = b64decode(content['euiccChallenge'])
+        if len(euiccChallenge) != 16:
+            raise ValueError
+
+        euiccInfo1_bin = b64decode(content['euiccInfo1'])
+        euiccInfo1 = rsp.asn1.decode('EUICCInfo1', euiccInfo1_bin)
+        logger.debug("Rx euiccInfo1: %s" % euiccInfo1)
+        #euiccInfo1['svn']
+
+        # TODO: If euiccCiPKIdListForSigningV3 is present ...
+
+        pkid_list = euiccInfo1['euiccCiPKIdListForSigning']
+        if 'euiccCiPKIdListForSigningV3' in euiccInfo1:
+            pkid_list = pkid_list + euiccInfo1['euiccCiPKIdListForSigningV3']
+
+        # Validate that SM-DP+ supports certificate chains for verification
+        verification_pkid_list = euiccInfo1.get('euiccCiPKIdListForVerification', [])
+        if verification_pkid_list and not self.validate_certificate_chain_for_verification(verification_pkid_list):
+            raise ApiError('8.8.4', '3.7', 'The SM-DP+ has no CERT.DPauth.SIG which chains to one of the eSIM CA Root CA Certificate with a Public Key supported by the eUICC')
+
+        # verify it supports one of the keys indicated by euiccCiPKIdListForSigning
+        ci_cert = None
+        for x in pkid_list:
+            ci_cert = self.ci_get_cert_for_pkid(x)
+            # we already support multiple CI certificates but only one set of DPauth + DPpb keys. So we must
+            # make sure we choose a CI key-id which has issued both the eUICC as well as our own SM-DP side
+            # certs.
+            if ci_cert and cert_get_subject_key_id(ci_cert) == self.dp_auth.get_authority_key_identifier().key_identifier:
+                break
+            else:
+                ci_cert = None
+        if not ci_cert:
+           raise ApiError('8.8.2', '3.1', 'None of the proposed Public Key Identifiers is supported by the SM-DP+')
+
+        # Generate a TransactionID which is used to identify the ongoing RSP session. The TransactionID
+        # SHALL be unique within the scope and lifetime of each SM-DP+.
+        transactionId = uuid.uuid4().hex.upper()
+        assert not transactionId in self.rss
+
+        # Generate a serverChallenge for eUICC authentication attached to the ongoing RSP session.
+        serverChallenge = os.urandom(16)
+
+        # Generate a serverSigned1 data object as expected by the eUICC and described in section 5.7.13 "ES10b.AuthenticateServer". If and only if both eUICC and LPA indicate crlStaplingV3Support, the SM-DP+ SHALL indicate crlStaplingV3Used in sessionContext.
+        serverSigned1 = {
+            'transactionId': h2b(transactionId),
+            'euiccChallenge': euiccChallenge,
+            'serverAddress': self.server_hostname,
+            'serverChallenge': serverChallenge,
+            }
+        logger.debug("Tx serverSigned1: %s" % serverSigned1)
+        serverSigned1_bin = rsp.asn1.encode('ServerSigned1', serverSigned1)
+        logger.debug("Tx serverSigned1: %s" % rsp.asn1.decode('ServerSigned1', serverSigned1_bin))
+        output = {}
+        output['serverSigned1'] = b64encode2str(serverSigned1_bin)
+
+        # Generate a signature (serverSignature1) as described in section 5.7.13 "ES10b.AuthenticateServer" using the SK related to the selected CERT.DPauth.SIG.
+        # serverSignature1 SHALL be created using the private key associated to the RSP Server Certificate for authentication, and verified by the eUICC using the contained public key as described in section 2.6.9. serverSignature1 SHALL apply on serverSigned1 data object.
+        output['serverSignature1'] = b64encode2str(b'\x5f\x37\x40' + self.dp_auth.ecdsa_sign(serverSigned1_bin))
+
+        output['transactionId'] = transactionId
+        server_cert_aki = self.dp_auth.get_authority_key_identifier()
+        output['euiccCiPKIdToBeUsed'] = b64encode2str(b'\x04\x14' + server_cert_aki.key_identifier)
+        output['serverCertificate'] = b64encode2str(self.dp_auth.get_cert_as_der()) # CERT.DPauth.SIG
+        # FIXME: add those certificate
+        #output['otherCertsInChain'] = b64encode2str()
+
+        # create SessionState and store it in rss
+        self.rss[transactionId] = rsp.RspSessionState(transactionId, serverChallenge,
+                                                      cert_get_subject_key_id(ci_cert))
+
+        return output
+
+    @app.route('/gsma/rsp2/es9plus/authenticateClient', methods=['POST'])
+    @rsp_api_wrapper
+    def authenticateClient(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ AuthenticateClient in SGP.22 Section 5.6.3"""
+        transactionId = content['transactionId']
+
+        authenticateServerResp_bin = b64decode(content['authenticateServerResponse'])
+        authenticateServerResp = rsp.asn1.decode('AuthenticateServerResponse', authenticateServerResp_bin)
+        logger.debug("Rx %s: %s" % authenticateServerResp)
+        if authenticateServerResp[0] == 'authenticateResponseError':
+            r_err = authenticateServerResp[1]
+            #r_err['transactionId']
+            #r_err['authenticateErrorCode']
+            raise ValueError("authenticateResponseError %s" % r_err)
+
+        r_ok = authenticateServerResp[1]
+        euiccSigned1 = r_ok['euiccSigned1']
+        euiccSigned1_bin = rsp.extract_euiccSigned1(authenticateServerResp_bin)
+        euiccSignature1_bin = r_ok['euiccSignature1']
+        euiccCertificate_dec = r_ok['euiccCertificate']
+        # TODO: use original data, don't re-encode?
+        euiccCertificate_bin = rsp.asn1.encode('Certificate', euiccCertificate_dec)
+        eumCertificate_dec = r_ok['eumCertificate']
+        eumCertificate_bin = rsp.asn1.encode('Certificate', eumCertificate_dec)
+        # TODO v3: otherCertsInChain
+
+        # load certificate
+        euicc_cert = x509.load_der_x509_certificate(euiccCertificate_bin)
+        eum_cert = x509.load_der_x509_certificate(eumCertificate_bin)
+
+        # Verify that the transactionId is known and relates to an ongoing RSP session.  Otherwise, the SM-DP+
+        # SHALL return a status code "TransactionId - Unknown"
+        ss = self.rss.get(transactionId, None)
+        if ss is None:
+            raise ApiError('8.10.1', '3.9', 'Unknown')
+        ss.euicc_cert = euicc_cert
+        ss.eum_cert = eum_cert # TODO: do we need this in the state?
+
+        # Verify that the Root Certificate of the eUICC certificate chain corresponds to the
+        # euiccCiPKIdToBeUsed or TODO: euiccCiPKIdToBeUsedV3
+        if cert_get_auth_key_id(eum_cert) != ss.ci_cert_id:
+            raise ApiError('8.11.1', '3.9', 'Unknown')
+
+        # Verify the validity of the eUICC certificate chain
+        cs = CertificateSet(self.ci_get_cert_for_pkid(ss.ci_cert_id))
+        cs.add_intermediate_cert(eum_cert)
+        # TODO v3: otherCertsInChain
+        try:
+            cs.verify_cert_chain(euicc_cert)
+        except VerifyError:
+            raise ApiError('8.1.3', '6.1', 'Verification failed (certificate chain)')
+        #   raise ApiError('8.1.3', '6.3', 'Expired')
+
+
+        # Verify euiccSignature1 over euiccSigned1 using pubkey from euiccCertificate.
+        # Otherwise, the SM-DP+ SHALL return a status code "eUICC - Verification failed"
+        if not self._ecdsa_verify(euicc_cert, euiccSignature1_bin, euiccSigned1_bin):
+            raise ApiError('8.1', '6.1', 'Verification failed (euiccSignature1 over euiccSigned1)')
+
+        ss.eid = ss.euicc_cert.subject.get_attributes_for_oid(x509.oid.NameOID.SERIAL_NUMBER)[0].value
+        logger.debug("EID (from eUICC cert): %s" % ss.eid)
+
+        # Verify EID is within permitted range of EUM certificate
+        if not validate_eid_range(ss.eid, eum_cert):
+            raise ApiError('8.1.4', '6.1', 'EID is not within the permitted range of the EUM certificate')
+
+        # Verify that the serverChallenge attached to the ongoing RSP session matches the
+        # serverChallenge returned by the eUICC. Otherwise, the SM-DP+ SHALL return a status code "eUICC -
+        # Verification failed".
+        if euiccSigned1['serverChallenge'] != ss.serverChallenge:
+            raise ApiError('8.1', '6.1', 'Verification failed (serverChallenge)')
+
+        # If ctxParams1 contains a ctxParamsForCommonAuthentication data object, the SM-DP+ Shall [...]
+        # TODO: We really do a very simplistic job here, this needs to be properly implemented later,
+        # considering all the various cases, profile state, etc.
+        iccid_str = None
+        if euiccSigned1['ctxParams1'][0] == 'ctxParamsForCommonAuthentication':
+            cpca = euiccSigned1['ctxParams1'][1]
+            matchingId = cpca.get('matchingId', None)
+            if not matchingId:
+                # TODO: check if any pending profile downloads for the EID
+                raise ApiError('8.2.6', '3.8', 'Refused')
+            if matchingId:
+                # look up profile based on matchingID.  We simply check if a given file exists for now..
+                path = os.path.join(self.upp_dir, matchingId) + '.der'
+                # prevent directory traversal attack
+                if os.path.commonprefix((os.path.realpath(path),self.upp_dir)) != self.upp_dir:
+                    raise ApiError('8.2.6', '3.8', 'Refused')
+                if not os.path.isfile(path) or not os.access(path, os.R_OK):
+                    raise ApiError('8.2.6', '3.8', 'Refused')
+                ss.matchingId = matchingId
+                with open(path, 'rb') as f:
+                    pes = saip.ProfileElementSequence.from_der(f.read())
+                    iccid_str = b2h(pes.get_pe_for_type('header').decoded['iccid'])
+        else:
+            # there's currently no other option in the ctxParams1 choice, so this cannot happen
+            raise ApiError('1.3.1', '2.2', 'ctxParams1 missing mandatory ctxParamsForCommonAuthentication')
+
+        # FIXME: we actually want to perform the profile binding herr, and read the profile metadata from the profile
+
+        # Put together profileMetadata + _bin
+        ss.profileMetadata = ProfileMetadata(iccid_bin=h2b(swap_nibbles(iccid_str)), spn="OsmocomSPN", profile_name=matchingId)
+        # enable notifications for all operations
+        for event in ['enable', 'disable', 'delete']:
+            ss.profileMetadata.add_notification(event, self.server_hostname)
+        profileMetadata_bin = ss.profileMetadata.gen_store_metadata_request()
+
+        # Put together smdpSigned2 + _bin
+        smdpSigned2 = {
+            'transactionId': h2b(ss.transactionId),
+            'ccRequiredFlag': False,        # whether the Confirmation Code is required
+            #'bppEuiccOtpk': None,           # whether otPK.EUICC.ECKA already used for binding the BPP, tag '5F49'
+            }
+        smdpSigned2_bin = rsp.asn1.encode('SmdpSigned2', smdpSigned2)
+
+        ss.smdpSignature2_do = b'\x5f\x37\x40' + self.dp_pb.ecdsa_sign(smdpSigned2_bin + b'\x5f\x37\x40' + euiccSignature1_bin)
+
+        # update non-volatile state with updated ss object
+        self.rss[transactionId] = ss
+        return {
+            'transactionId': transactionId,
+            'profileMetadata': b64encode2str(profileMetadata_bin),
+            'smdpSigned2': b64encode2str(smdpSigned2_bin),
+            'smdpSignature2': b64encode2str(ss.smdpSignature2_do),
+            'smdpCertificate': b64encode2str(self.dp_pb.get_cert_as_der()), # CERT.DPpb.SIG
+        }
+
+    @app.route('/gsma/rsp2/es9plus/getBoundProfilePackage', methods=['POST'])
+    @rsp_api_wrapper
+    def getBoundProfilePackage(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ GetBoundProfilePackage SGP.22 Section 5.6.2"""
+        transactionId = content['transactionId']
+
+        # Verify that the received transactionId is known and relates to an ongoing RSP session
+        ss = self.rss.get(transactionId, None)
+        if not ss:
+            raise ApiError('8.10.1', '3.9', 'The RSP session identified by the TransactionID is unknown')
+
+        prepDownloadResp_bin = b64decode(content['prepareDownloadResponse'])
+        prepDownloadResp = rsp.asn1.decode('PrepareDownloadResponse', prepDownloadResp_bin)
+        logger.debug("Rx %s: %s" % prepDownloadResp)
+
+        if prepDownloadResp[0] == 'downloadResponseError':
+            r_err = prepDownloadResp[1]
+            #r_err['transactionId']
+            #r_err['downloadErrorCode']
+            raise ValueError("downloadResponseError %s" % r_err)
+
+        r_ok = prepDownloadResp[1]
+
+        # Verify the euiccSignature2 computed over euiccSigned2 and smdpSignature2 using the PK.EUICC.SIG attached to the ongoing RSP session
+        euiccSigned2 = r_ok['euiccSigned2']
+        euiccSigned2_bin = rsp.extract_euiccSigned2(prepDownloadResp_bin)
+        if not self._ecdsa_verify(ss.euicc_cert, r_ok['euiccSignature2'], euiccSigned2_bin + ss.smdpSignature2_do):
+            raise ApiError('8.1', '6.1', 'eUICC signature is invalid')
+
+        # not in spec: Verify that signed TransactionID is outer transaction ID
+        if h2b(transactionId) != euiccSigned2['transactionId']:
+            raise ApiError('8.10.1', '3.9', 'The signed transactionId != outer transactionId')
+
+        # store otPK.EUICC.ECKA in session state
+        ss.euicc_otpk = euiccSigned2['euiccOtpk']
+        logger.debug("euiccOtpk: %s" % (b2h(ss.euicc_otpk)))
+
+        # Generate a one-time ECKA key pair (ot{PK,SK}.DP.ECKA) using the curve indicated by the Key Parameter
+        # Reference value of CERT.DPpb.ECDDSA
+        logger.debug("curve = %s" % self.dp_pb.get_curve())
+        ss.smdp_ot = ec.generate_private_key(self.dp_pb.get_curve())
+        # extract the public key in (hopefully) the right format for the ES8+ interface
+        ss.smdp_otpk = ss.smdp_ot.public_key().public_bytes(Encoding.X962, PublicFormat.UncompressedPoint)
+        logger.debug("smdpOtpk: %s" % b2h(ss.smdp_otpk))
+        logger.debug("smdpOtsk: %s" % b2h(ss.smdp_ot.private_bytes(Encoding.DER, PrivateFormat.PKCS8, NoEncryption())))
+
+        ss.host_id = b'mahlzeit'
+
+        # Generate Session Keys using the CRT, otPK.eUICC.ECKA and otSK.DP.ECKA according to annex G
+        euicc_public_key = ec.EllipticCurvePublicKey.from_encoded_point(ss.smdp_ot.curve, ss.euicc_otpk)
+        ss.shared_secret = ss.smdp_ot.exchange(ec.ECDH(), euicc_public_key)
+        logger.debug("shared_secret: %s" % b2h(ss.shared_secret))
+
+        # TODO: Check if this order requires a Confirmation Code verification
+
+        #  Perform actual protection + binding of profile package (or return  pre-bound one)
+        with open(os.path.join(self.upp_dir, ss.matchingId)+'.der', 'rb') as f:
+            upp = UnprotectedProfilePackage.from_der(f.read(), metadata=ss.profileMetadata)
+            # HACK: Use empty PPP as we're still debugging the configureISDP step, and we want to avoid
+            # cluttering the log with stuff happening after the failure
+            #upp = UnprotectedProfilePackage.from_der(b'', metadata=ss.profileMetadata)
+        if False:
+            # Use random keys
+            bpp = BoundProfilePackage.from_upp(upp)
+        else:
+            # Use session keys
+            ppp = ProtectedProfilePackage.from_upp(upp, BspInstance(b'\x00'*16, b'\x11'*16, b'\x22'*16))
+            bpp = BoundProfilePackage.from_ppp(ppp)
+
+        # update non-volatile state with updated ss object
+        self.rss[transactionId] = ss
+        return {
+            'transactionId': transactionId,
+            'boundProfilePackage': b64encode2str(bpp.encode(ss, self.dp_pb)),
+        }
+
+    @app.route('/gsma/rsp2/es9plus/handleNotification', methods=['POST'])
+    @rsp_api_wrapper
+    def handleNotification(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ HandleNotification in SGP.22 Section 5.6.4"""
+        # SGP.22 Section 6.3: "A normal notification function execution status (MEP Notification)
+        # SHALL be indicated by the HTTP status code '204' (No Content) with an empty HTTP response body"
+        request.setResponseCode(204)
+        pendingNotification_bin = b64decode(content['pendingNotification'])
+        pendingNotification = rsp.asn1.decode('PendingNotification', pendingNotification_bin)
+        logger.debug("Rx %s: %s" % pendingNotification)
+        if pendingNotification[0] == 'profileInstallationResult':
+            profileInstallRes = pendingNotification[1]
+            pird = profileInstallRes['profileInstallationResultData']
+            transactionId = b2h(pird['transactionId'])
+            ss = self.rss.get(transactionId, None)
+            if ss is None:
+                logger.warning(f"Unable to find session for transactionId: {transactionId}")
+                return None  # Will return HTTP 204 with empty body
+            profileInstallRes['euiccSignPIR']
+            # TODO: use original data, don't re-encode?
+            pird_bin = rsp.asn1.encode('ProfileInstallationResultData', pird)
+            # verify eUICC signature
+            if not self._ecdsa_verify(ss.euicc_cert, profileInstallRes['euiccSignPIR'], pird_bin):
+                raise Exception('ECDSA signature verification failed on notification')
+            logger.debug("Profile Installation Final Result: %s", pird['finalResult'])
+            # remove session state
+            del self.rss[transactionId]
+        elif pendingNotification[0] == 'otherSignedNotification':
+            otherSignedNotif = pendingNotification[1]
+            # TODO: use some kind of partially-parsed original data, don't re-encode?
+            euiccCertificate_bin = rsp.asn1.encode('Certificate', otherSignedNotif['euiccCertificate'])
+            eumCertificate_bin = rsp.asn1.encode('Certificate', otherSignedNotif['eumCertificate'])
+            euicc_cert = x509.load_der_x509_certificate(euiccCertificate_bin)
+            eum_cert = x509.load_der_x509_certificate(eumCertificate_bin)
+            ci_cert_id = cert_get_auth_key_id(eum_cert)
+            # Verify the validity of the eUICC certificate chain
+            cs = CertificateSet(self.ci_get_cert_for_pkid(ci_cert_id))
+            cs.add_intermediate_cert(eum_cert)
+            # TODO v3: otherCertsInChain
+            cs.verify_cert_chain(euicc_cert)
+            tbs_bin = rsp.asn1.encode('NotificationMetadata', otherSignedNotif['tbsOtherNotification'])
+            if not self._ecdsa_verify(euicc_cert, otherSignedNotif['euiccNotificationSignature'], tbs_bin):
+                raise Exception('ECDSA signature verification failed on notification')
+            other_notif = otherSignedNotif['tbsOtherNotification']
+            pmo = PMO.from_bitstring(other_notif['profileManagementOperation'])
+            eid = euicc_cert.subject.get_attributes_for_oid(x509.oid.NameOID.SERIAL_NUMBER)[0].value
+            iccid = other_notif.get('iccid', None)
+            if iccid:
+                iccid = swap_nibbles(b2h(iccid))
+            logger.debug("handleNotification: EID %s: %s of %s" % (eid, pmo, iccid))
+        else:
+            raise ValueError(pendingNotification)
+
+    #@app.route('/gsma/rsp3/es9plus/handleDeviceChangeRequest, methods=['POST']')
+    #@rsp_api_wrapper
+        #"""See ES9+ ConfirmDeviceChange in SGP.22 Section 5.6.6"""
+        # TODO: implement this
+
+    @app.route('/gsma/rsp2/es9plus/cancelSession', methods=['POST'])
+    @rsp_api_wrapper
+    def cancelSession(self, request: IRequest, content: dict) -> dict:
+        """See ES9+ CancelSession in SGP.22 Section 5.6.5"""
+        logger.debug("Rx JSON: %s" % content)
+        transactionId = content['transactionId']
+
+        # Verify that the received transactionId is known and relates to an ongoing RSP session
+        ss = self.rss.get(transactionId, None)
+        if ss is None:
+            raise ApiError('8.10.1', '3.9', 'The RSP session identified by the transactionId is unknown')
+
+        cancelSessionResponse_bin = b64decode(content['cancelSessionResponse'])
+        cancelSessionResponse = rsp.asn1.decode('CancelSessionResponse', cancelSessionResponse_bin)
+        logger.debug("Rx %s: %s" % cancelSessionResponse)
+
+        if cancelSessionResponse[0] == 'cancelSessionResponseError':
+            # FIXME: print some error
+            return
+        cancelSessionResponseOk = cancelSessionResponse[1]
+        # TODO: use original data, don't re-encode?
+        ecsr = cancelSessionResponseOk['euiccCancelSessionSigned']
+        ecsr_bin = rsp.asn1.encode('EuiccCancelSessionSigned', ecsr)
+        # Verify the eUICC signature (euiccCancelSessionSignature) using the PK.EUICC.SIG attached to the ongoing RSP session
+        if not self._ecdsa_verify(ss.euicc_cert, cancelSessionResponseOk['euiccCancelSessionSignature'], ecsr_bin):
+            raise ApiError('8.1', '6.1', 'eUICC signature is invalid')
+
+        # Verify that the received smdpOid corresponds to the one in SM-DP+ CERT.DPauth.SIG
+        subj_alt_name = self.dp_auth.get_subject_alt_name()
+        if x509.ObjectIdentifier(ecsr['smdpOid']) != subj_alt_name.oid:
+            raise ApiError('8.8', '3.10', 'The provided SM-DP+ OID is invalid.')
+
+        if ecsr['transactionId'] != h2b(transactionId):
+            raise ApiError('8.10.1', '3.9', 'The signed transactionId != outer transactionId')
+
+        # TODO: 1. Notify the Operator using the function "ES2+.HandleNotification" function
+        # TODO: 2. Terminate the corresponding pending download process.
+        # TODO: 3. If required, execute the SM-DS Event Deletion procedure described in section 3.6.3.
+
+        # delete actual session data
+        del self.rss[transactionId]
+        return { 'transactionId': transactionId }
+
+
+def main(argv):
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-H", "--host", help="Host/IP to bind HTTP(S) to", default="localhost")
+    parser.add_argument("-p", "--port", help="TCP port to bind HTTP(S) to", default=443)
+    parser.add_argument("-c", "--certdir", help=f"cert subdir relative to {DATA_DIR}", default="certs")
+    parser.add_argument("-s", "--nossl", help="disable built in SSL/TLS support", action='store_true', default=False)
+    parser.add_argument("-v", "--verbose", help="dump more raw info", action='store_true', default=False)
+    parser.add_argument("-b", "--brainpool", help="Use Brainpool curves instead of NIST",
+                        action='store_true', default=False)
+    parser.add_argument("-m", "--in-memory", help="Use ephermal in-memory session storage (for concurrent runs)",
+                        action='store_true', default=False)
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.WARNING)
+
+    common_cert_path = os.path.join(DATA_DIR, args.certdir)
+    hs = SmDppHttpServer(server_hostname=HOSTNAME, ci_certs_path=os.path.join(common_cert_path, 'CertificateIssuer'), common_cert_path=common_cert_path, use_brainpool=args.brainpool)
+    if(args.nossl):
+        hs.app.run(args.host, args.port)
+    else:
+        curve_type = 'BRP' if args.brainpool else 'NIST'
+        cert_derpath = Path(common_cert_path) / 'DPtls' / f'CERT_S_SM_DP_TLS_{curve_type}.der'
+        cert_pempath = Path(common_cert_path) / 'DPtls' / f'CERT_S_SM_DP_TLS_{curve_type}.pem'
+        cert_skpath = Path(common_cert_path) / 'DPtls' / f'SK_S_SM_DP_TLS_{curve_type}.pem'
+        dhparam_path = Path(common_cert_path) / "dhparam2048.pem"
+        if not dhparam_path.exists():
+            print("Generating dh params, this takes a few seconds..")
+            # Generate DH parameters with 2048-bit key size and generator 2
+            parameters = dh.generate_parameters(generator=2, key_size=2048)
+            pem_data = parameters.parameter_bytes(encoding=Encoding.PEM,format=ParameterFormat.PKCS3)
+            with open(dhparam_path, 'wb') as file:
+                file.write(pem_data)
+            print("DH params created successfully")
+
+        if not cert_pempath.exists():
+            print("Translating tls server cert from DER to PEM..")
+            with open(cert_derpath, 'rb') as der_file:
+                der_cert_data = der_file.read()
+
+            cert = x509.load_der_x509_certificate(der_cert_data)
+            pem_cert = cert.public_bytes(Encoding.PEM) #.decode('utf-8')
+
+            with open(cert_pempath, 'wb') as pem_file:
+                pem_file.write(pem_cert)
+
+        SERVER_STRING = f'ssl:{args.port}:privateKey={cert_skpath}:certKey={cert_pempath}:dhParameters={dhparam_path}'
+        print(SERVER_STRING)
+
+        hs.app.run(host=HOSTNAME, port=args.port, endpoint_description=SERVER_STRING)
+
+if __name__ == "__main__":
+    main(sys.argv)

pySim-prog.py

@@ -25,185 +25,168 @@
 #
 
 import hashlib
-from optparse import OptionParser
+import argparse
 import os
 import random
 import re
 import sys
 import traceback
 import json
+import csv
+from osmocom.utils import h2b, swap_nibbles, rpad
 
 from pySim.commands import SimCardCommands
-from pySim.transport import init_reader
-from pySim.cards import _cards_classes, card_detect
-from pySim.utils import h2b, swap_nibbles, rpad, derive_milenage_opc, calculate_luhn, dec_iccid
-from pySim.ts_51_011 import EF, EF_AD
+from pySim.transport import init_reader, argparse_add_reader_args
+from pySim.legacy.cards import _cards_classes, card_detect
+from pySim.utils import derive_milenage_opc, calculate_luhn, dec_iccid
+from pySim.ts_51_011 import EF_AD
+from pySim.legacy.ts_51_011 import EF
 from pySim.card_handler import *
 from pySim.utils import *
 
 
 def parse_options():
 
-    parser = OptionParser(usage="usage: %prog [options]")
+    parser = argparse.ArgumentParser()
+    argparse_add_reader_args(parser)
 
-    parser.add_option("-d", "--device", dest="device", metavar="DEV",
-                      help="Serial Device for SIM access [default: %default]",
-                      default="/dev/ttyUSB0",
-                      )
-    parser.add_option("-b", "--baud", dest="baudrate", type="int", metavar="BAUD",
-                      help="Baudrate used for SIM access [default: %default]",
-                      default=9600,
-                      )
-    parser.add_option("-p", "--pcsc-device", dest="pcsc_dev", type='int', metavar="PCSC",
-                      help="Which PC/SC reader number for SIM access",
-                      default=None,
-                      )
-    parser.add_option("--modem-device", dest="modem_dev", metavar="DEV",
-                      help="Serial port of modem for Generic SIM Access (3GPP TS 27.007)",
-                      default=None,
-                      )
-    parser.add_option("--modem-baud", dest="modem_baud", type="int", metavar="BAUD",
-                      help="Baudrate used for modem's port [default: %default]",
-                      default=115200,
-                      )
-    parser.add_option("--osmocon", dest="osmocon_sock", metavar="PATH",
-                      help="Socket path for Calypso (e.g. Motorola C1XX) based reader (via OsmocomBB)",
-                      default=None,
-                      )
-    parser.add_option("-t", "--type", dest="type",
-                      help="Card type (user -t list to view) [default: %default]",
+    parser.add_argument("-t", "--type", dest="type",
+                      help="Card type (user -t list to view) [default: %(default)s]",
                       default="auto",
                       )
-    parser.add_option("-T", "--probe", dest="probe",
+    parser.add_argument("-T", "--probe", dest="probe",
                       help="Determine card type",
                       default=False, action="store_true"
                       )
-    parser.add_option("-a", "--pin-adm", dest="pin_adm",
+    parser.add_argument("-a", "--pin-adm", dest="pin_adm",
                       help="ADM PIN used for provisioning (overwrites default)",
                       )
-    parser.add_option("-A", "--pin-adm-hex", dest="pin_adm_hex",
+    parser.add_argument("-A", "--pin-adm-hex", dest="pin_adm_hex",
                       help="ADM PIN used for provisioning, as hex string (16 characters long",
                       )
-    parser.add_option("-e", "--erase", dest="erase", action='store_true',
-                      help="Erase beforehand [default: %default]",
+    parser.add_argument("-e", "--erase", dest="erase", action='store_true',
+                      help="Erase beforehand [default: %(default)s]",
                       default=False,
                       )
 
-    parser.add_option("-S", "--source", dest="source",
-                      help="Data Source[default: %default]",
+    parser.add_argument("-S", "--source", dest="source",
+                      help="Data Source[default: %(default)s]",
                       default="cmdline",
                       )
 
     # if mode is "cmdline"
-    parser.add_option("-n", "--name", dest="name",
-                      help="Operator name [default: %default]",
+    parser.add_argument("-n", "--name", dest="name",
+                      help="Operator name [default: %(default)s]",
                       default="Magic",
                       )
-    parser.add_option("-c", "--country", dest="country", type="int", metavar="CC",
-                      help="Country code [default: %default]",
+    parser.add_argument("-c", "--country", dest="country", type=int, metavar="CC",
+                      help="Country code [default: %(default)s]",
                       default=1,
                       )
-    parser.add_option("-x", "--mcc", dest="mcc", type="string",
-                      help="Mobile Country Code [default: %default]",
+    parser.add_argument("-x", "--mcc", dest="mcc",
+                      help="Mobile Country Code [default: %(default)s]",
                       default="901",
                       )
-    parser.add_option("-y", "--mnc", dest="mnc", type="string",
-                      help="Mobile Network Code [default: %default]",
+    parser.add_argument("-y", "--mnc", dest="mnc",
+                      help="Mobile Network Code [default: %(default)s]",
                       default="55",
                       )
-    parser.add_option("--mnclen", dest="mnclen", type="choice",
-                      help="Length of Mobile Network Code [default: %default]",
-                      default=2,
-                      choices=[2, 3],
+    parser.add_argument("--mnclen", dest="mnclen",
+                      help="Length of Mobile Network Code [default: %(default)s]",
+                      default="auto",
+                      choices=["2", "3", "auto"],
                       )
-    parser.add_option("-m", "--smsc", dest="smsc",
+    parser.add_argument("-m", "--smsc", dest="smsc",
                       help="SMSC number (Start with + for international no.) [default: '00 + country code + 5555']",
                       )
-    parser.add_option("-M", "--smsp", dest="smsp",
+    parser.add_argument("-M", "--smsp", dest="smsp",
                       help="Raw SMSP content in hex [default: auto from SMSC]",
                       )
 
-    parser.add_option("-s", "--iccid", dest="iccid", metavar="ID",
+    parser.add_argument("-s", "--iccid", dest="iccid", metavar="ID",
                       help="Integrated Circuit Card ID",
                       )
-    parser.add_option("-i", "--imsi", dest="imsi",
+    parser.add_argument("-i", "--imsi", dest="imsi",
                       help="International Mobile Subscriber Identity",
                       )
-    parser.add_option("--msisdn", dest="msisdn",
+    parser.add_argument("--msisdn", dest="msisdn",
                       help="Mobile Subscriber Integrated Services Digital Number",
                       )
-    parser.add_option("-k", "--ki", dest="ki",
+    parser.add_argument("-k", "--ki", dest="ki",
                       help="Ki (default is to randomize)",
                       )
-    parser.add_option("-o", "--opc", dest="opc",
+    parser.add_argument("-o", "--opc", dest="opc",
                       help="OPC (default is to randomize)",
                       )
-    parser.add_option("--op", dest="op",
+    parser.add_argument("--op", dest="op",
                       help="Set OP to derive OPC from OP and KI",
                       )
-    parser.add_option("--acc", dest="acc",
+    parser.add_argument("--acc", dest="acc",
                       help="Set ACC bits (Access Control Code). not all card types are supported",
                       )
-    parser.add_option("--opmode", dest="opmode", type="choice",
+    parser.add_argument("--opmode", dest="opmode",
                       help="Set UE Operation Mode in EF.AD (Administrative Data)",
                       default=None,
                       choices=['{:02X}'.format(int(m)) for m in EF_AD.OP_MODE],
                       )
-    parser.add_option("--epdgid", dest="epdgid",
+    parser.add_argument("-f", "--fplmn", dest="fplmn", action="append",
+                      help="Set Forbidden PLMN. Add multiple time for multiple FPLMNS",
+                      )
+    parser.add_argument("--epdgid", dest="epdgid",
                       help="Set Home Evolved Packet Data Gateway (ePDG) Identifier. (Only FQDN format supported)",
                       )
-    parser.add_option("--epdgSelection", dest="epdgSelection",
+    parser.add_argument("--epdgSelection", dest="epdgSelection",
                       help="Set PLMN for ePDG Selection Information. (Only Operator Identifier FQDN format supported)",
                       )
-    parser.add_option("--pcscf", dest="pcscf",
+    parser.add_argument("--pcscf", dest="pcscf",
                       help="Set Proxy Call Session Control Function (P-CSCF) Address. (Only FQDN format supported)",
                       )
-    parser.add_option("--ims-hdomain", dest="ims_hdomain",
+    parser.add_argument("--ims-hdomain", dest="ims_hdomain",
                       help="Set IMS Home Network Domain Name in FQDN format",
                       )
-    parser.add_option("--impi", dest="impi",
+    parser.add_argument("--impi", dest="impi",
                       help="Set IMS private user identity",
                       )
-    parser.add_option("--impu", dest="impu",
+    parser.add_argument("--impu", dest="impu",
                       help="Set IMS public user identity",
                       )
-    parser.add_option("--read-imsi", dest="read_imsi", action="store_true",
+    parser.add_argument("--read-imsi", dest="read_imsi", action="store_true",
                       help="Read the IMSI from the CARD", default=False
                       )
-    parser.add_option("--read-iccid", dest="read_iccid", action="store_true",
+    parser.add_argument("--read-iccid", dest="read_iccid", action="store_true",
                       help="Read the ICCID from the CARD", default=False
                       )
-    parser.add_option("-z", "--secret", dest="secret", metavar="STR",
+    parser.add_argument("-z", "--secret", dest="secret", metavar="STR",
                       help="Secret used for ICCID/IMSI autogen",
                       )
-    parser.add_option("-j", "--num", dest="num", type=int,
+    parser.add_argument("-j", "--num", dest="num", type=int,
                       help="Card # used for ICCID/IMSI autogen",
                       )
-    parser.add_option("--batch", dest="batch_mode",
-                      help="Enable batch mode [default: %default]",
+    parser.add_argument("--batch", dest="batch_mode",
+                      help="Enable batch mode [default: %(default)s]",
                       default=False, action='store_true',
                       )
-    parser.add_option("--batch-state", dest="batch_state", metavar="FILE",
+    parser.add_argument("--batch-state", dest="batch_state", metavar="FILE",
                       help="Optional batch state file",
                       )
 
     # if mode is "csv"
-    parser.add_option("--read-csv", dest="read_csv", metavar="FILE",
+    parser.add_argument("--read-csv", dest="read_csv", metavar="FILE",
                       help="Read parameters from CSV file rather than command line")
 
-    parser.add_option("--write-csv", dest="write_csv", metavar="FILE",
+    parser.add_argument("--write-csv", dest="write_csv", metavar="FILE",
                       help="Append generated parameters in CSV file",
                       )
-    parser.add_option("--write-hlr", dest="write_hlr", metavar="FILE",
+    parser.add_argument("--write-hlr", dest="write_hlr", metavar="FILE",
                       help="Append generated parameters to OpenBSC HLR sqlite3",
                       )
-    parser.add_option("--dry-run", dest="dry_run",
+    parser.add_argument("--dry-run", dest="dry_run",
                       help="Perform a 'dry run', don't actually program the card",
                       default=False, action="store_true")
-    parser.add_option("--card_handler", dest="card_handler_config", metavar="FILE",
+    parser.add_argument("--card_handler", dest="card_handler_config", metavar="FILE",
                       help="Use automatic card handling machine")
 
-    (options, args) = parser.parse_args()
+    options = parser.parse_args()
 
     if options.type == 'list':
         for kls in _cards_classes:
@@ -214,15 +197,13 @@ def parse_options():
         return options
 
     if options.source == 'csv':
-        if (options.imsi is None) and (options.batch_mode is False) and (options.read_imsi is False) and (options.read_iccid is False):
-            parser.error(
-                "CSV mode needs either an IMSI, --read-imsi, --read-iccid or batch mode")
+        if (options.imsi is None) and (options.iccid is None)  and (options.read_imsi is False) and (options.read_iccid is False):
+           parser.error("CSV mode requires one additional parameter: --read-iccid, --read-imsi, --iccid or --imsi")
         if options.read_csv is None:
             parser.error("CSV mode requires a CSV input file")
     elif options.source == 'cmdline':
         if ((options.imsi is None) or (options.iccid is None)) and (options.num is None):
-            parser.error(
-                "If either IMSI or ICCID isn't specified, num is required")
+            parser.error("If either IMSI or ICCID isn't specified, num is required")
     else:
         parser.error("Only `cmdline' and `csv' sources supported")
 
@@ -237,9 +218,6 @@ def parse_options():
             parser.error(
                 "Can't give ICCID/IMSI for batch mode, need to use automatic parameters ! see --num and --secret for more information")
 
-    if args:
-        parser.error("Extraneous arguments")
-
     return options
 
 
@@ -318,8 +296,15 @@ def gen_parameters(opts):
 
     # MCC always has 3 digits
     mcc = lpad(mcc, 3, "0")
-    # MNC must be at least 2 digits
-    mnc = lpad(mnc, 2, "0")
+
+    # The MNC must be at least 2 digits long. This is also the most common case.
+    # The user may specify an explicit length using the --mnclen option.
+    if opts.mnclen != "auto":
+        if len(mnc) > int(opts.mnclen):
+            raise ValueError('mcc is longer than specified in option --mnclen')
+        mnc = lpad(mnc, int(opts.mnclen), "0")
+    else:
+        mnc = lpad(mnc, 2, "0")
 
     # Digitize country code (2 or 3 digits)
     cc_digits = _cc_digits(opts.country)
@@ -346,8 +331,8 @@ def gen_parameters(opts):
     # ICCID (19 digits, E.118), though some phase1 vendors use 20 :(
     if opts.iccid is not None:
         iccid = opts.iccid
-        if not _isnum(iccid, 19) and not _isnum(iccid, 20):
-            raise ValueError('ICCID must be 19 or 20 digits !')
+        if not _isnum(iccid, 18) and not _isnum(iccid, 19) and not _isnum(iccid, 20):
+            raise ValueError('ICCID must be 18, 19 or 20 digits !')
 
     else:
         if opts.num is None:
@@ -490,6 +475,7 @@ def gen_parameters(opts):
         'impi': opts.impi,
         'impu': opts.impu,
         'opmode': opts.opmode,
+        'fplmn': opts.fplmn,
     }
 
 
@@ -524,40 +510,86 @@ def write_params_csv(opts, params):
         f.close()
 
 
-def _read_params_csv(opts, iccid=None, imsi=None):
-    import csv
-    f = open(opts.read_csv, 'r')
+def find_row_in_csv_file(csv_file_name:str, num=None, iccid=None, imsi=None):
+    """
+    Pick a matching row in a CSV file by row number or ICCID or IMSI. When num
+    is not None, the search parameters iccid and imsi are ignored. When
+    searching for a specific ICCID or IMSI the caller must set num to None. It
+    is possible to search for an ICCID or an IMSI at the same time. The first
+    line that either contains a matching ICCID or IMSI is returned. Unused
+    search parameters must be set to None.
+    """
+
+    f = open(csv_file_name, 'r')
     cr = csv.DictReader(f)
 
+    # Make sure the CSV file contains at least the fields we are searching for
+    if not 'iccid' in cr.fieldnames:
+        raise Exception("wrong CSV file format - no field \"iccid\" or missing header!")
+    if not 'imsi' in cr.fieldnames:
+        raise Exception("wrong CSV file format - no field \"imsi\" or missing header!")
+
+    # Enforce at least one search parameter
+    if not num and not iccid and not imsi:
+        raise Exception("no CSV file search parameters!")
+
     # Lower-case fieldnames
     cr.fieldnames = [field.lower() for field in cr.fieldnames]
 
     i = 0
-    if not 'iccid' in cr.fieldnames:
-        raise Exception("CSV file in wrong format!")
     for row in cr:
-        if opts.num is not None and opts.read_iccid is False and opts.read_imsi is False:
-            if opts.num == i:
+        # Pick a specific row by line number (num)
+        if num is not None and iccid is None and imsi is None:
+            if num == i:
                 f.close()
                 return row
-            i += 1
+
+	# Pick the first row that contains the specified ICCID
         if row['iccid'] == iccid:
             f.close()
             return row
 
+	# Pick the first row that contains the specified IMSI
         if row['imsi'] == imsi:
             f.close()
             return row
 
+        i += 1
+
     f.close()
+    print("Could not read card parameters from CSV file, no matching entry found.")
     return None
 
 
 def read_params_csv(opts, imsi=None, iccid=None):
-    row = _read_params_csv(opts, iccid=iccid, imsi=imsi)
+    """
+    Read the card parameters from a CSV file. This function will generate the
+    same dictionary that gen_parameters would generate from parameters passed as
+    commandline arguments.
+    """
+
+    row = find_row_in_csv_file(opts.read_csv, opts.num, iccid=iccid, imsi=imsi)
     if row is not None:
         row['mcc'] = row.get('mcc', mcc_from_imsi(row.get('imsi')))
-        row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi')))
+
+        # We cannot determine the MNC length (2 or 3 digits?) from the IMSI
+        # alone. In cases where the user has specified an mnclen via the
+        # commandline options we can use that info, otherwise we guess that
+        # the length is 2, which is also the most common case.
+        if opts.mnclen != "auto":
+            if opts.mnclen == "2":
+                row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), False))
+            elif opts.mnclen == "3":
+                row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), True))
+            else:
+                raise ValueError("invalid parameter --mnclen, must be 2 or 3 or auto")
+        else:
+            row['mnc'] = row.get('mnc', mnc_from_imsi(row.get('imsi'), False))
+
+	# NOTE: We might consider to specify a new CSV field "mnclen" in our
+	# CSV files for a better automatization. However, this only makes sense
+	# when the tools and databases we export our files from will also add
+	# such a field.
 
         pin_adm = None
         # We need to escape the pin_adm we get from the csv
@@ -590,6 +622,9 @@ def read_params_csv(opts, imsi=None, iccid=None):
 
 def write_params_hlr(opts, params):
     # SQLite3 OpenBSC HLR
+    # FIXME: The format of the osmo-hlr database has evolved, so that the code below will no longer work.
+    print("Warning: the database format of recent OsmoHLR versions is not compatible with pySim-prog!")
+
     if opts.write_hlr:
         import sqlite3
         conn = sqlite3.connect(opts.write_hlr)
@@ -621,7 +656,7 @@ def write_params_hlr(opts, params):
         conn.close()
 
 
-def write_parameters(opts, params):
+def write_parameters_to_csv_and_hlr(opts, params):
     write_params_csv(opts, params)
     write_params_hlr(opts, params)
 
@@ -668,52 +703,46 @@ def save_batch(opts):
     fh.close()
 
 
-def process_card(opts, first, ch):
+def process_card(scc, opts, first, ch):
 
+    # Connect transport
+    ch.get(first)
+
+    # Get card
+    card = card_detect(opts.type, scc)
+    if card is None:
+        print("No card detected!")
+        return -1
+
+    # Probe only
+    if opts.probe:
+        return 0
+
+    # Erase if requested (not in dry run mode!)
     if opts.dry_run is False:
-        # Connect transport
-        ch.get(first)
-
-    if opts.dry_run is False:
-        # Get card
-        card = card_detect(opts.type, scc)
-        if card is None:
-            print("No card detected!")
-            return -1
-
-        # Probe only
-        if opts.probe:
-            return 0
-
-        # Erase if requested
         if opts.erase:
             print("Formatting ...")
             card.erase()
             card.reset()
 
+    cp = None
+
     # Generate parameters
     if opts.source == 'cmdline':
         cp = gen_parameters(opts)
     elif opts.source == 'csv':
-        imsi = None
-        iccid = None
         if opts.read_iccid:
-            if opts.dry_run:
-                # Connect transport
-                ch.get(False)
             (res, _) = scc.read_binary(['3f00', '2fe2'], length=10)
             iccid = dec_iccid(res)
-        elif opts.read_imsi:
-            if opts.dry_run:
-                # Connect transport
-                ch.get(False)
+        else:
+            iccid = opts.iccid
+        if opts.read_imsi:
             (res, _) = scc.read_binary(EF['IMSI'])
             imsi = swap_nibbles(res)[3:]
         else:
             imsi = opts.imsi
         cp = read_params_csv(opts, imsi=imsi, iccid=iccid)
     if cp is None:
-        print("Error reading parameters from CSV file!\n")
         return 2
     print_parameters(cp)
 
@@ -724,8 +753,8 @@ def process_card(opts, first, ch):
     else:
         print("Dry Run: NOT PROGRAMMING!")
 
-    # Write parameters permanently
-    write_parameters(opts, cp)
+    # Write parameters to a specified CSV file or an HLR database (not the card)
+    write_parameters_to_csv_and_hlr(opts, cp)
 
     # Batch mode state update and save
     if opts.num is not None:
@@ -743,8 +772,6 @@ if __name__ == '__main__':
 
     # Init card reader driver
     sl = init_reader(opts)
-    if sl is None:
-        exit(1)
 
     # Create command layer
     scc = SimCardCommands(transport=sl)
@@ -770,7 +797,7 @@ if __name__ == '__main__':
 
     while 1:
         try:
-            rc = process_card(opts, first, ch)
+            rc = process_card(scc, opts, first, ch)
         except (KeyboardInterrupt):
             print("")
             print("Terminated by user!")

pySim-read.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 #
-# Utility to display some informations about a SIM card
+# Utility to display some information about a SIM card
 #
 #
 # Copyright (C) 2009  Sylvain Munaut <tnt@246tNt.com>
@@ -28,20 +28,25 @@ import os
 import random
 import re
 import sys
-from pySim.ts_51_011 import EF, DF, EF_SST_map, EF_AD
-from pySim.ts_31_102 import EF_UST_map, EF_USIM_ADF_map
-from pySim.ts_31_103 import EF_IST_map, EF_ISIM_ADF_map
+
+from osmocom.utils import h2b, h2s, swap_nibbles, rpad
+
+from pySim.ts_51_011 import EF_SST_map, EF_AD
+from pySim.legacy.ts_51_011 import EF, DF
+from pySim.ts_31_102 import EF_UST_map
+from pySim.legacy.ts_31_102 import EF_USIM_ADF_map
+from pySim.ts_31_103 import EF_IST_map
+from pySim.legacy.ts_31_103 import EF_ISIM_ADF_map
 
 from pySim.commands import SimCardCommands
 from pySim.transport import init_reader, argparse_add_reader_args
 from pySim.exceptions import SwMatchError
-from pySim.cards import card_detect, SimCard, UsimCard, IsimCard
-from pySim.utils import h2b, swap_nibbles, rpad, dec_imsi, dec_iccid, dec_msisdn
-from pySim.utils import format_xplmn_w_act, dec_st
-from pySim.utils import h2s, format_ePDGSelection
+from pySim.legacy.cards import card_detect, SimCard, UsimCard, IsimCard
+from pySim.utils import dec_imsi, dec_iccid
+from pySim.legacy.utils import format_xplmn_w_act, dec_st, dec_msisdn
+from pySim.ts_51_011 import EF_SMSP
 
-option_parser = argparse.ArgumentParser(prog='pySim-read',
-                                        description='Legacy tool for reading some parts of a SIM card',
+option_parser = argparse.ArgumentParser(description='Legacy tool for reading some parts of a SIM card',
                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 argparse_add_reader_args(option_parser)
 
@@ -72,8 +77,6 @@ if __name__ == '__main__':
 
     # Init card reader driver
     sl = init_reader(opts)
-    if sl is None:
-        exit(1)
 
     # Create command layer
     scc = SimCardCommands(transport=sl)
@@ -86,7 +89,7 @@ if __name__ == '__main__':
     scc.sel_ctrl = "0004"
 
     # Testing for Classic SIM or UICC
-    (res, sw) = sl.send_apdu(scc.cla_byte + "a4" + scc.sel_ctrl + "02" + "3f00")
+    (res, sw) = sl.send_apdu(scc.cla_byte + "a4" + scc.sel_ctrl + "02" + "3f00" + "00")
     if sw == '6e00':
         # Just a Classic SIM
         scc.cla_byte = "a0"
@@ -139,6 +142,15 @@ if __name__ == '__main__':
     (res, sw) = card.read_record('SMSP', 1)
     if sw == '9000':
         print("SMSP: %s" % (res,))
+        ef_smsp = EF_SMSP()
+        smsc_a = ef_smsp.decode_record_bin(h2b(res), 1).get('tp_sc_addr', {})
+        smsc_n = smsc_a.get('call_number', None)
+        if smsc_a.get('ton_npi', {}).get('type_of_number', None) == 'international' and smsc_n is not None:
+            smsc = '+' + smsc_n
+        else:
+            smsc = smsc_n
+        if smsc is not None:
+            print("SMSC: %s" % (smsc,))
     else:
         print("SMSP: Can't read, response code = %s" % (sw,))
 
@@ -253,6 +265,14 @@ if __name__ == '__main__':
             else:
                 print("EHPLMN: Can't read, response code = %s" % (sw,))
 
+        # EF.FPLMN
+        if usim_card.file_exists(EF_USIM_ADF_map['FPLMN']):
+            res, sw = usim_card.read_fplmn()
+            if sw == '9000':
+                print(f'FPLMN:\n{res}')
+            else:
+                print(f'FPLMN: Can\'t read, response code = {sw}')
+
         # EF.UST
         try:
             if usim_card.file_exists(EF_USIM_ADF_map['UST']):

pySim-smpp2sim.py

@@ -0,0 +1,428 @@
+#!/usr/bin/env python3
+#
+# Program to emulate the entire communication path SMSC-MSC-BSC-BTS-ME
+# that is usually between an OTA backend and the SIM card.  This allows
+# to play with SIM OTA technology without using a mobile network or even
+# a mobile phone.
+#
+# An external application must encode (and encrypt/sign) the OTA SMS
+# and submit them via SMPP to this program, just like it would submit
+# it normally to a SMSC (SMS Service Centre).  The program then re-formats
+# the SMPP-SUBMIT into a SMS DELIVER TPDU and passes it via an ENVELOPE
+# APDU to the SIM card that is locally inserted into a smart card reader.
+#
+# The path from SIM to external OTA application works the opposite way.
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import logging
+import colorlog
+
+from twisted.protocols import basic
+from twisted.internet import defer, endpoints, protocol, reactor, task
+from twisted.cred.portal import IRealm
+from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
+from twisted.cred.portal import Portal
+from zope.interface import implementer
+
+from smpp.twisted.config import SMPPServerConfig
+from smpp.twisted.server import SMPPServerFactory, SMPPBindManager
+from smpp.twisted.protocol import SMPPSessionStates, DataHandlerResponse
+
+from smpp.pdu import pdu_types, operations, pdu_encoding
+
+from pySim.sms import SMS_DELIVER, SMS_SUBMIT, AddressField
+
+from pySim.transport import LinkBase, ProactiveHandler, argparse_add_reader_args, init_reader, ApduTracer
+from pySim.commands import SimCardCommands
+from pySim.cards import UiccCardBase
+from pySim.exceptions import *
+from pySim.cat import ProactiveCommand, SendShortMessage, SMS_TPDU, SMSPPDownload, BearerDescription
+from pySim.cat import DeviceIdentities, Address, OtherAddress, UiccTransportLevel, BufferSize
+from pySim.cat import ChannelStatus, ChannelData, ChannelDataLength
+from pySim.utils import b2h, h2b
+
+logger = logging.getLogger(__name__)
+
+# MSISDNs to use when generating proactive SMS messages
+SIM_MSISDN='23'
+ESME_MSISDN='12'
+
+# HACK: we need some kind of mapping table between system_id and card-reader
+# or actually route based on MSISDNs
+hackish_global_smpp = None
+
+class MyApduTracer(ApduTracer):
+    def trace_response(self, cmd, sw, resp):
+        print("-> %s %s" % (cmd[:10], cmd[10:]))
+        print("<- %s: %s" % (sw, resp))
+
+class TcpProtocol(protocol.Protocol):
+    def dataReceived(self, data):
+        pass
+
+    def connectionLost(self, reason):
+        pass
+
+
+def tcp_connected_callback(p: protocol.Protocol):
+    """called by twisted TCP client."""
+    logger.error("%s: connected!" % p)
+
+class ProactChannel:
+    """Representation of a single protective channel."""
+    def __init__(self, channels: 'ProactChannels', chan_nr: int):
+        self.channels = channels
+        self.chan_nr = chan_nr
+        self.ep = None
+
+    def close(self):
+        """Close the channel."""
+        if self.ep:
+            self.ep.disconnect()
+        self.channels.channel_delete(self.chan_nr)
+
+class ProactChannels:
+    """Wrapper class for maintaining state of proactive channels."""
+    def __init__(self):
+        self.channels = {}
+
+    def channel_create(self) -> ProactChannel:
+        """Create a new proactive channel, allocating its integer number."""
+        for i in range(1, 9):
+            if not i in self.channels:
+                self.channels[i] = ProactChannel(self, i)
+                return self.channels[i]
+        raise ValueError('Cannot allocate another channel: All channels active')
+
+    def channel_delete(self, chan_nr: int):
+        del self.channels[chan_nr]
+
+class Proact(ProactiveHandler):
+    #def __init__(self, smpp_factory):
+    #    self.smpp_factory = smpp_factory
+    def __init__(self):
+        self.channels = ProactChannels()
+
+    @staticmethod
+    def _find_first_element_of_type(instlist, cls):
+        for i in instlist:
+            if isinstance(i, cls):
+                return i
+        return None
+
+    """Call-back which the pySim transport core calls whenever it receives a
+    proactive command from the SIM."""
+    def handle_SendShortMessage(self, pcmd: ProactiveCommand):
+        # {'smspp_download': [{'device_identities': {'source_dev_id': 'network',
+        #                                            'dest_dev_id': 'uicc'}},
+        #                     {'address': {'ton_npi': {'ext': True,
+        #                                              'type_of_number': 'international',
+        #                                              'numbering_plan_id': 'isdn_e164'},
+        #                                  'call_number': '79'}},
+        #                     {'sms_tpdu': {'tpdu': '40048111227ff6407070611535004d02700000481516011212000001fe4c0943aea42e45021c078ae06c66afc09303608874b72f58bacadb0dcf665c29349c799fbb522e61709c9baf1890015e8e8e196e36153106c8b92f95153774'}}
+        #                    ]}
+        """Card requests sending a SMS. We need to pass it on to the ESME via SMPP."""
+        logger.info("SendShortMessage")
+        logger.info(pcmd)
+        # Relevant parts in pcmd: Address, SMS_TPDU
+        addr_ie = Proact._find_first_element_of_type(pcmd.children, Address)
+        sms_tpdu_ie = Proact._find_first_element_of_type(pcmd.children, SMS_TPDU)
+        raw_tpdu = sms_tpdu_ie.decoded['tpdu']
+        submit = SMS_SUBMIT.from_bytes(raw_tpdu)
+        submit.tp_da = AddressField(addr_ie.decoded['call_number'], addr_ie.decoded['ton_npi']['type_of_number'],
+                                    addr_ie.decoded['ton_npi']['numbering_plan_id'])
+        logger.info(submit)
+        self.send_sms_via_smpp(submit)
+
+    def handle_OpenChannel(self, pcmd: ProactiveCommand):
+        """Card requests opening a new channel via a UDP/TCP socket."""
+        # {'open_channel': [{'command_details': {'command_number': 1,
+        #                                        'type_of_command': 'open_channel',
+        #                                        'command_qualifier': 3}},
+        #                   {'device_identities': {'source_dev_id': 'uicc',
+        #                                          'dest_dev_id': 'terminal'}},
+        #                   {'bearer_description': {'bearer_type': 'default',
+        #                                           'bearer_parameters': ''}},
+        #                   {'buffer_size': 1024},
+        #                   {'uicc_transport_level': {'protocol_type': 'tcp_uicc_client_remote',
+        #                                             'port_number': 32768}},
+        #                   {'other_address': {'type_of_address': 'ipv4',
+        #                                      'address': '01020304'}}
+        #                  ]}
+        logger.info("OpenChannel")
+        logger.info(pcmd)
+        transp_lvl_ie = Proact._find_first_element_of_type(pcmd.children, UiccTransportLevel)
+        other_addr_ie = Proact._find_first_element_of_type(pcmd.children, OtherAddress)
+        bearer_desc_ie = Proact._find_first_element_of_type(pcmd.children, BearerDescription)
+        buffer_size_ie = Proact._find_first_element_of_type(pcmd.children, BufferSize)
+        if transp_lvl_ie.decoded['protocol_type'] != 'tcp_uicc_client_remote':
+            raise ValueError('Unsupported protocol_type')
+        if other_addr_ie.decoded.get('type_of_address', None) != 'ipv4':
+            raise ValueError('Unsupported type_of_address')
+        ipv4_bytes = h2b(other_addr_ie.decoded['address'])
+        ipv4_str = '%u.%u.%u.%u' % (ipv4_bytes[0], ipv4_bytes[1], ipv4_bytes[2], ipv4_bytes[3])
+        port_nr = transp_lvl_ie.decoded['port_number']
+        print("%s:%u" % (ipv4_str, port_nr))
+        channel = self.channels.channel_create()
+        channel.ep = endpoints.TCP4ClientEndpoint(reactor, ipv4_str, port_nr)
+        channel.prot = TcpProtocol()
+        d = endpoints.connectProtocol(channel.ep, channel.prot)
+        # FIXME: why is this never called despite the client showing the inbound connection?
+        d.addCallback(tcp_connected_callback)
+
+        # Terminal Response example: [
+        #  {'command_details': {'command_number': 1,
+        #                       'type_of_command': 'open_channel',
+        #                       'command_qualifier': 3}},
+        #  {'device_identities': {'source_dev_id': 'terminal', 'dest_dev_id': 'uicc'}},
+        #  {'result': {'general_result': 'performed_successfully', 'additional_information': ''}},
+        #  {'channel_status': '8100'},
+        #  {'bearer_description': {'bearer_type': 'default', 'bearer_parameters': ''}},
+        #  {'buffer_size': 1024}
+        # ]
+        return self.prepare_response(pcmd) + [ChannelStatus(decoded='8100'), bearer_desc_ie, buffer_size_ie]
+
+    def handle_CloseChannel(self, pcmd: ProactiveCommand):
+        """Close a channel."""
+        logger.info("CloseChannel")
+        logger.info(pcmd)
+
+    def handle_ReceiveData(self, pcmd: ProactiveCommand):
+        """Receive/read data from the socket."""
+        # {'receive_data': [{'command_details': {'command_number': 1,
+        #                                        'type_of_command': 'receive_data',
+        #                                        'command_qualifier': 0}},
+        #                   {'device_identities': {'source_dev_id': 'uicc',
+        #                                          'dest_dev_id': 'channel_1'}},
+        #                   {'channel_data_length': 9}
+        #                  ]}
+        logger.info("ReceiveData")
+        logger.info(pcmd)
+        # Terminal Response example: [
+        #  {'command_details': {'command_number': 1,
+        #                       'type_of_command': 'receive_data',
+        #                       'command_qualifier': 0}},
+        #  {'device_identities': {'source_dev_id': 'terminal', 'dest_dev_id': 'uicc'}},
+        #  {'result': {'general_result': 'performed_successfully', 'additional_information': ''}},
+        #  {'channel_data': '16030100040e000000'},
+        #  {'channel_data_length': 0}
+        # ]
+        return self.prepare_response(pcmd) + []
+
+    def handle_SendData(self, pcmd: ProactiveCommand):
+        """Send/write data received from the SIM to the socket."""
+        # {'send_data': [{'command_details': {'command_number': 1,
+        #                                     'type_of_command': 'send_data',
+        #                                     'command_qualifier': 1}},
+        #                {'device_identities': {'source_dev_id': 'uicc',
+        #                                       'dest_dev_id': 'channel_1'}},
+        #                {'channel_data': '160301003c010000380303d0f45e12b52ce5bb522750dd037738195334c87a46a847fe2b6886cada9ea6bf00000a00ae008c008b00b0002c010000050001000101'}
+        #               ]}
+        logger.info("SendData")
+        logger.info(pcmd)
+        dev_id_ie = Proact._find_first_element_of_type(pcmd.children, DeviceIdentities)
+        chan_data_ie = Proact._find_first_element_of_type(pcmd.children, ChannelData)
+        chan_str = dev_id_ie.decoded['dest_dev_id']
+        chan_nr = 1 # FIXME
+        chan = self.channels.channels.get(chan_nr, None)
+        # FIXME chan.prot.transport.write(h2b(chan_data_ie.decoded))
+        # Terminal Response example: [
+        #  {'command_details': {'command_number': 1,
+        #                       'type_of_command': 'send_data',
+        #                       'command_qualifier': 1}},
+        #  {'device_identities': {'source_dev_id': 'terminal', 'dest_dev_id': 'uicc'}},
+        #  {'result': {'general_result': 'performed_successfully', 'additional_information': ''}},
+        #  {'channel_data_length': 255}
+        # ]
+        return self.prepare_response(pcmd) + [ChannelDataLength(decoded=255)]
+
+    def handle_SetUpEventList(self, pcmd: ProactiveCommand):
+        # {'set_up_event_list': [{'command_details': {'command_number': 1,
+        #                                             'type_of_command': 'set_up_event_list',
+        #                                             'command_qualifier': 0}},
+        #                        {'device_identities': {'source_dev_id': 'uicc',
+        #                                               'dest_dev_id': 'terminal'}},
+        #                        {'event_list': ['data_available', 'channel_status']}
+        #                       ]}
+        logger.info("SetUpEventList")
+        logger.info(pcmd)
+        # Terminal Response example: [
+        #  {'command_details': {'command_number': 1,
+        #                       'type_of_command': 'set_up_event_list',
+        #                       'command_qualifier': 0}},
+        #  {'device_identities': {'source_dev_id': 'terminal', 'dest_dev_id': 'uicc'}},
+        #  {'result': {'general_result': 'performed_successfully', 'additional_information': ''}}
+        # ]
+        return self.prepare_response(pcmd)
+
+    def getChannelStatus(self, pcmd: ProactiveCommand):
+        logger.info("GetChannelStatus")
+        logger.info(pcmd)
+        return self.prepare_response(pcmd) + []
+
+    def send_sms_via_smpp(self, submit: SMS_SUBMIT):
+        # while in a normal network the phone/ME would *submit* a message to the SMSC,
+        # we are actually emulating the SMSC itself, so we must *deliver* the message
+        # to the ESME
+        deliver = SMS_DELIVER.from_submit(submit)
+        deliver_smpp = deliver.to_smpp()
+
+        hackish_global_smpp.sendDataRequest(deliver_smpp)
+#       # obtain the connection/binding of system_id to be used for delivering MO-SMS to the ESME
+#       connection = smpp_server.getBoundConnections[system_id].getNextBindingForDelivery()
+#       connection.sendDataRequest(deliver_smpp)
+
+
+
+def dcs_is_8bit(dcs):
+    if dcs == pdu_types.DataCoding(pdu_types.DataCodingScheme.DEFAULT,
+                                   pdu_types.DataCodingDefault.OCTET_UNSPECIFIED):
+        return True
+    if dcs == pdu_types.DataCoding(pdu_types.DataCodingScheme.DEFAULT,
+                                   pdu_types.DataCodingDefault.OCTET_UNSPECIFIED_COMMON):
+        return True
+    # pySim-smpp2sim.py:150:21: E1101: Instance of 'DataCodingScheme' has no 'GSM_MESSAGE_CLASS' member (no-member)
+    # pylint: disable=no-member
+    if dcs.scheme == pdu_types.DataCodingScheme.GSM_MESSAGE_CLASS and dcs.schemeData['msgCoding'] == pdu_types.DataCodingGsmMsgCoding.DATA_8BIT:
+        return True
+    else:
+        return False
+
+
+class MyServer:
+
+    @implementer(IRealm)
+    class SmppRealm:
+        def requestAvatar(self, avatarId, mind, *interfaces):
+            return ('SMPP', avatarId, lambda: None)
+
+    def __init__(self, tcp_port:int = 2775, bind_ip = '::', system_id:str = 'test', password:str = 'test'):
+        smpp_config = SMPPServerConfig(msgHandler=self._msgHandler,
+                                       systems={system_id: {'max_bindings': 2}})
+        portal = Portal(self.SmppRealm())
+        credential_checker = InMemoryUsernamePasswordDatabaseDontUse()
+        credential_checker.addUser(system_id, password)
+        portal.registerChecker(credential_checker)
+        self.factory = SMPPServerFactory(smpp_config, auth_portal=portal)
+        logger.info('Binding Virtual SMSC to TCP Port %u at %s' % (tcp_port, bind_ip))
+        smppEndpoint = endpoints.TCP6ServerEndpoint(reactor, tcp_port, interface=bind_ip)
+        smppEndpoint.listen(self.factory)
+        self.tp = self.scc = self.card = None
+
+    def connect_to_card(self, tp: LinkBase):
+        self.tp = tp
+        self.scc = SimCardCommands(self.tp)
+        self.card = UiccCardBase(self.scc)
+        # this should be part of UiccCardBase, but FairewavesSIM breaks with that :/
+        self.scc.cla_byte = "00"
+        self.scc.sel_ctrl = "0004"
+        self.card.read_aids()
+        self.card.select_adf_by_aid(adf='usim')
+        # FIXME: create a more realistic profile than ffffff
+        self.scc.terminal_profile('ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff')
+
+    def _msgHandler(self, system_id, smpp, pdu):
+        """Handler for incoming messages received via SMPP from ESME."""
+        # HACK: we need some kind of mapping table between system_id and card-reader
+        # or actually route based on MSISDNs
+        global hackish_global_smpp
+        hackish_global_smpp = smpp
+        if pdu.id == pdu_types.CommandId.submit_sm:
+            return self.handle_submit_sm(system_id, smpp, pdu)
+        else:
+            logger.warning('Rejecting non-SUBMIT commandID')
+            return pdu_types.CommandStatus.ESME_RINVCMDID
+
+    def handle_submit_sm(self, system_id, smpp, pdu):
+        """SUBMIT-SM was received via SMPP from ESME. We need to deliver it to the SIM."""
+        # check for valid data coding scheme + PID
+        if not dcs_is_8bit(pdu.params['data_coding']):
+            logger.warning('Rejecting non-8bit DCS')
+            return pdu_types.CommandStatus.ESME_RINVDCS
+        if pdu.params['protocol_id'] != 0x7f:
+            logger.warning('Rejecting non-SIM PID')
+            return pdu_types.CommandStatus.ESME_RINVDCS
+
+        # 1) build a SMS-DELIVER (!) from the SMPP-SUBMIT
+        tpdu = SMS_DELIVER.from_smpp_submit(pdu)
+        logger.info(tpdu)
+        # 2) wrap into the CAT ENVELOPE for SMS-PP-Download
+        tpdu_ie = SMS_TPDU(decoded={'tpdu': b2h(tpdu.to_bytes())})
+        addr_ie = Address(decoded={'ton_npi': {'ext':False, 'type_of_number':'unknown', 'numbering_plan_id':'unknown'}, 'call_number': '0123456'})
+        dev_ids = DeviceIdentities(decoded={'source_dev_id': 'network', 'dest_dev_id': 'uicc'})
+        sms_dl = SMSPPDownload(children=[dev_ids, addr_ie, tpdu_ie])
+        # 3) send to the card
+        envelope_hex = b2h(sms_dl.to_tlv())
+        logger.info("ENVELOPE: %s" % envelope_hex)
+        (data, sw) = self.scc.envelope(envelope_hex)
+        logger.info("SW %s: %s" % (sw, data))
+        if sw in ['9200', '9300']:
+            # TODO send back RP-ERROR message with TP-FCS == 'SIM Application Toolkit Busy'
+            return pdu_types.CommandStatus.ESME_RSUBMITFAIL
+        elif sw == '9000' or sw[0:2] in ['6f', '62', '63'] and len(data):
+            # data something like 027100000e0ab000110000000000000001612f or
+            # 027100001c12b000119660ebdb81be189b5e4389e9e7ab2bc0954f963ad869ed7c
+            # which is the user-data portion of the SMS starting with the UDH (027100)
+            # TODO: return the response back to the sender in an RP-ACK; PID/DCS like in CMD
+            deliver = operations.DeliverSM(service_type=pdu.params['service_type'],
+                                           source_addr_ton=pdu.params['dest_addr_ton'],
+                                           source_addr_npi=pdu.params['dest_addr_npi'],
+                                           source_addr=pdu.params['destination_addr'],
+                                           dest_addr_ton=pdu.params['source_addr_ton'],
+                                           dest_addr_npi=pdu.params['source_addr_npi'],
+                                           destination_addr=pdu.params['source_addr'],
+                                           esm_class=pdu.params['esm_class'],
+                                           protocol_id=pdu.params['protocol_id'],
+                                           priority_flag=pdu.params['priority_flag'],
+                                           data_coding=pdu.params['data_coding'],
+                                           short_message=h2b(data))
+            smpp.sendDataRequest(deliver)
+            return pdu_types.CommandStatus.ESME_ROK
+        else:
+            return pdu_types.CommandStatus.ESME_RSUBMITFAIL
+
+
+option_parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+argparse_add_reader_args(option_parser)
+smpp_group = option_parser.add_argument_group('SMPP Options')
+smpp_group.add_argument('--smpp-bind-port', type=int, default=2775,
+                        help='TCP Port to bind the SMPP socket to')
+smpp_group.add_argument('--smpp-bind-ip', default='::',
+                        help='IPv4/IPv6 address to bind the SMPP socket to')
+smpp_group.add_argument('--smpp-system-id', default='test',
+                        help='SMPP System-ID used by ESME to bind')
+smpp_group.add_argument('--smpp-password', default='test',
+                        help='SMPP Password used by ESME to bind')
+
+if __name__ == '__main__':
+    log_format='%(log_color)s%(levelname)-8s%(reset)s %(name)s: %(message)s'
+    colorlog.basicConfig(level=logging.INFO, format = log_format)
+    logger = colorlog.getLogger()
+
+    opts = option_parser.parse_args()
+
+    tp = init_reader(opts, proactive_handler = Proact())
+    if tp is None:
+        exit(1)
+    tp.connect()
+
+    global g_ms
+    g_ms = MyServer(opts.smpp_bind_port, opts.smpp_bind_ip, opts.smpp_system_id, opts.smpp_password)
+    g_ms.connect_to_card(tp)
+    reactor.run()
+

pySim-trace.py

@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+
+import sys
+import logging, colorlog
+import argparse
+from pprint import pprint as pp
+
+from pySim.apdu import *
+from pySim.runtime import RuntimeState
+
+from osmocom.utils import JsonEncoder
+
+from pySim.cards import UiccCardBase
+from pySim.commands import SimCardCommands
+from pySim.profile import CardProfile
+from pySim.ts_102_221 import CardProfileUICC
+from pySim.ts_31_102 import CardApplicationUSIM
+from pySim.ts_31_103 import CardApplicationISIM
+from pySim.euicc import CardApplicationISDR, CardApplicationECASD
+from pySim.transport import LinkBase
+
+from pySim.apdu_source.gsmtap import GsmtapApduSource
+from pySim.apdu_source.pyshark_rspro import PysharkRsproPcap, PysharkRsproLive
+from pySim.apdu_source.pyshark_gsmtap import PysharkGsmtapPcap
+from pySim.apdu_source.tca_loader_log import TcaLoaderLogApduSource
+from pySim.apdu_source.stdin_hex import StdinHexApduSource
+
+from pySim.apdu.ts_102_221 import UiccSelect, UiccStatus
+
+log_format='%(log_color)s%(levelname)-8s%(reset)s %(name)s: %(message)s'
+colorlog.basicConfig(level=logging.INFO, format = log_format)
+logger = colorlog.getLogger()
+
+# merge all of the command sets into one global set. This will override instructions,
+# the one from the 'last' set in the addition below will prevail.
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+ApduCommands = UiccApduCommands + UsimApduCommands #+ GpApduCommands
+
+
+class DummySimLink(LinkBase):
+    """A dummy implementation of the LinkBase abstract base class.  Currently required
+    as the UiccCardBase doesn't work without SimCardCommands, which in turn require
+    a LinkBase implementation talking to a card.
+
+    In the tracer, we don't actually talk to any card, so we simply drop everything
+    and claim it is successful.
+
+    The UiccCardBase / SimCardCommands should be refactored to make this obsolete later."""
+    def __init__(self, debug: bool = False, **kwargs):
+        super().__init__(**kwargs)
+        self._debug = debug
+        self._atr = h2i('3B9F96801F878031E073FE211B674A4C753034054BA9')
+
+    def __str__(self):
+        return "dummy"
+
+    def _send_apdu(self, pdu):
+        #print("DummySimLink-apdu: %s" % pdu)
+        return [], '9000'
+
+    def connect(self):
+        pass
+
+    def disconnect(self):
+        pass
+
+    def _reset_card(self):
+        return 1
+
+    def get_atr(self):
+        return self._atr
+
+    def wait_for_card(self):
+        pass
+
+
+class Tracer:
+    def __init__(self, **kwargs):
+        # we assume a generic UICC profile; as all APDUs return 9000 in DummySimLink above,
+        # all CardProfileAddon (including SIM) will probe successful.
+        profile = CardProfileUICC()
+        profile.add_application(CardApplicationUSIM())
+        profile.add_application(CardApplicationISIM())
+        profile.add_application(CardApplicationISDR())
+        profile.add_application(CardApplicationECASD())
+        scc = SimCardCommands(transport=DummySimLink())
+        card = UiccCardBase(scc)
+        self.rs = RuntimeState(card, profile)
+        # APDU Decoder
+        self.ad = ApduDecoder(ApduCommands)
+        # parameters
+        self.suppress_status = kwargs.get('suppress_status', True)
+        self.suppress_select = kwargs.get('suppress_select', True)
+        self.show_raw_apdu = kwargs.get('show_raw_apdu', False)
+        self.source = kwargs.get('source', None)
+
+    def format_capdu(self, apdu: Apdu, inst: ApduCommand):
+        """Output a single decoded + processed ApduCommand."""
+        if self.show_raw_apdu:
+            print(apdu)
+        print("%02u %-16s %-35s %-8s %s %s" % (inst.lchan_nr, inst._name, inst.path_str, inst.col_id,
+                                               inst.col_sw, json.dumps(inst.processed, cls=JsonEncoder)))
+        print("===============================")
+
+    def format_reset(self, apdu: CardReset):
+        """Output a single decoded CardReset."""
+        print(apdu)
+        print("===============================")
+
+    def main(self):
+        """Main loop of tracer: Iterates over all Apdu received from source."""
+        apdu_counter = 0
+        while True:
+            # obtain the next APDU from the source (blocking read)
+            try:
+                apdu = self.source.read()
+                apdu_counter = apdu_counter + 1
+            except StopIteration:
+                print("%i APDUs parsed, stop iteration." % apdu_counter)
+                return 0
+
+            if isinstance(apdu, CardReset):
+                self.rs.reset()
+                self.format_reset(apdu)
+                continue
+
+            # ask ApduDecoder to look-up (INS,CLA) + instantiate an ApduCommand derived
+            # class like 'UiccSelect'
+            inst = self.ad.input(apdu)
+            # process the APDU (may modify the RuntimeState)
+            inst.process(self.rs)
+
+            # Avoid cluttering the log with too much verbosity
+            if self.suppress_select and isinstance(inst, UiccSelect):
+                continue
+            if self.suppress_status and isinstance(inst, UiccStatus):
+                continue
+
+            self.format_capdu(apdu, inst)
+
+option_parser = argparse.ArgumentParser(description='Osmocom pySim high-level SIM card trace decoder',
+                                        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+global_group = option_parser.add_argument_group('General Options')
+global_group.add_argument('--no-suppress-select', action='store_false', dest='suppress_select',
+                          help="""
+    Don't suppress displaying SELECT APDUs. We normally suppress them as they just clutter up
+    the output without giving any useful information.  Any subsequent READ/UPDATE/... operations
+    on the selected file will log the file name most recently SELECTed.""")
+global_group.add_argument('--no-suppress-status', action='store_false', dest='suppress_status',
+                          help="""
+    Don't suppress displaying STATUS APDUs. We normally suppress them as they don't provide any
+    information that was not already received in response to the most recent SEELCT.""")
+global_group.add_argument('--show-raw-apdu', action='store_true', dest='show_raw_apdu',
+                          help="""Show the raw APDU in addition to its parsed form.""")
+
+
+subparsers = option_parser.add_subparsers(help='APDU Source', dest='source', required=True)
+
+parser_gsmtap = subparsers.add_parser('gsmtap-udp', help="""
+    Read APDUs from live capture by receiving GSMTAP-SIM packets on specified UDP port.
+    Use this for live capture from SIMtrace2 or osmo-qcdiag.""")
+parser_gsmtap.add_argument('-i', '--bind-ip', default='127.0.0.1',
+                           help='Local IP address to which to bind the UDP port')
+parser_gsmtap.add_argument('-p', '--bind-port', default=4729,
+                           help='Local UDP port')
+
+parser_gsmtap_pyshark_pcap = subparsers.add_parser('gsmtap-pyshark-pcap', help="""
+    Read APDUs from PCAP file containing GSMTAP (SIM APDU) communication; processed via pyshark.
+    Use this if you have recorded a PCAP file containing GSMTAP (SIM APDU) e.g. via tcpdump or
+    wireshark/tshark.""")
+parser_gsmtap_pyshark_pcap.add_argument('-f', '--pcap-file', required=True,
+                                       help='Name of the PCAP[ng] file to be read')
+
+parser_rspro_pyshark_pcap = subparsers.add_parser('rspro-pyshark-pcap', help="""
+    Read APDUs from PCAP file containing RSPRO (osmo-remsim) communication; processed via pyshark.
+    REQUIRES OSMOCOM PATCHED WIRESHARK!""")
+parser_rspro_pyshark_pcap.add_argument('-f', '--pcap-file', required=True,
+                                       help='Name of the PCAP[ng] file to be read')
+
+parser_rspro_pyshark_live = subparsers.add_parser('rspro-pyshark-live', help="""
+    Read APDUs from live capture of RSPRO (osmo-remsim) communication; processed via pyshark.
+    REQUIRES OSMOCOM PATCHED WIRESHARK!""")
+parser_rspro_pyshark_live.add_argument('-i', '--interface', required=True,
+                                       help='Name of the network interface to capture on')
+
+parser_tcaloader_log = subparsers.add_parser('tca-loader-log', help="""
+    Read APDUs from a TCA Loader log file.""")
+parser_tcaloader_log.add_argument('-f', '--log-file', required=True,
+                                  help='Name of the log file to be read')
+
+parser_stdin_hex = subparsers.add_parser('stdin-hex', help="""
+    Read APDUs as hex-string from stdin.""")
+
+
+if __name__ == '__main__':
+
+    opts = option_parser.parse_args()
+
+    logger.info('Opening source %s...', opts.source)
+    if opts.source == 'gsmtap-udp':
+        s = GsmtapApduSource(opts.bind_ip, opts.bind_port)
+    elif opts.source == 'rspro-pyshark-pcap':
+        s = PysharkRsproPcap(opts.pcap_file)
+    elif opts.source == 'rspro-pyshark-live':
+        s = PysharkRsproLive(opts.interface)
+    elif opts.source == 'gsmtap-pyshark-pcap':
+        s = PysharkGsmtapPcap(opts.pcap_file)
+    elif opts.source == 'tca-loader-log':
+        s = TcaLoaderLogApduSource(opts.log_file)
+    elif opts.source == 'stdin-hex':
+        s = StdinHexApduSource()
+    else:
+        raise ValueError("unsupported source %s", opts.source)
+
+    tracer = Tracer(source=s, suppress_status=opts.suppress_status, suppress_select=opts.suppress_select,
+                    show_raw_apdu=opts.show_raw_apdu)
+    logger.info('Entering main loop...')
+    tracer.main()
+

pySim/apdu/__init__.py

@@ -0,0 +1,466 @@
+"""APDU (and TPDU) parser for UICC/USIM/ISIM cards.
+
+The File (and its classes) represent the structure / hierarchy
+of the APDUs as seen in SIM/UICC/SIM/ISIM cards.  The primary use case
+is to perform a meaningful decode of protocol traces taken between card and UE.
+
+The ancient wirshark dissector developed for GSMTAP generated by SIMtrace
+is far too simplistic, while this decoder can utilize all of the information
+we already know in pySim about the filesystem structure, file encoding, etc.
+"""
+
+# (C) 2022-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import abc
+import typing
+from typing import List, Dict, Optional
+from termcolor import colored
+from construct import Byte
+from construct import Optional as COptional
+from osmocom.construct import *
+from osmocom.utils import *
+
+from pySim.runtime import RuntimeLchan, RuntimeState, lchan_nr_from_cla
+from pySim.filesystem import CardADF, CardFile, TransparentEF, LinFixedEF
+
+"""There are multiple levels of decode:
+
+    1) pure TPDU / APDU level (no filesystem state required to decode)
+        1a) the raw C-TPDU + R-TPDU
+        1b) the raw C-APDU + R-APDU
+        1c) the C-APDU + R-APDU split in its portions (p1/p2/lc/le/cmd/rsp)
+        1d) the abstract C-APDU + R-APDU (mostly p1/p2 parsing; SELECT response)
+    2) the decoded DATA of command/response APDU
+        * READ/UPDATE: requires state/context: which file is selected? how to decode it?
+"""
+
+class ApduCommandMeta(abc.ABCMeta):
+    """A meta-class that we can use to set some class variables when declaring
+       a derived class of ApduCommand."""
+    def __new__(mcs, name, bases, namespace, **kwargs):
+        x = super().__new__(mcs, name, bases, namespace)
+        x._name = namespace.get('name', kwargs.get('n', None))
+        x._ins = namespace.get('ins', kwargs.get('ins', None))
+        x._cla = namespace.get('cla', kwargs.get('cla', None))
+        return x
+
+BytesOrHex = typing.Union[bytes, Hexstr]
+
+class Tpdu:
+    def __init__(self, cmd: BytesOrHex, rsp: Optional[BytesOrHex] = None):
+        if isinstance(cmd, str):
+            self.cmd = h2b(cmd)
+        else:
+            self.cmd = cmd
+        if  isinstance(rsp, str):
+            self.rsp = h2b(rsp)
+        else:
+            self.rsp = rsp
+
+    def __str__(self):
+        return '%s(%02X %02X %02X %02X %02X %s %s %s)' % (type(self).__name__, self.cla, self.ins, self.p1,
+                self.p2, self.p3, b2h(self.cmd_data), b2h(self.rsp_data), b2h(self.sw))
+
+    @property
+    def cla(self) -> int:
+        """Return CLA of the C-APDU Header."""
+        return self.cmd[0]
+
+    @property
+    def ins(self) -> int:
+        """Return INS of the C-APDU Header."""
+        return self.cmd[1]
+
+    @property
+    def p1(self) -> int:
+        """Return P1 of the C-APDU Header."""
+        return self.cmd[2]
+
+    @property
+    def p2(self) -> int:
+        """Return P2 of the C-APDU Header."""
+        return self.cmd[3]
+
+    @property
+    def p3(self) -> int:
+        """Return P3 of the C-APDU Header."""
+        return self.cmd[4]
+
+    @property
+    def cmd_data(self) -> int:
+        """Return the DATA portion of the C-APDU"""
+        return self.cmd[5:]
+
+    @property
+    def sw(self) -> Optional[bytes]:
+        """Return Status Word (SW) of the R-APDU"""
+        return self.rsp[-2:] if self.rsp else None
+
+    @property
+    def rsp_data(self) -> Optional[bytes]:
+        """Return the DATA portion of the R-APDU"""
+        return self.rsp[:-2] if self.rsp else None
+
+
+class Apdu(Tpdu):
+    @property
+    def lc(self) -> int:
+        """Return Lc; Length of C-APDU body."""
+        return len(self.cmd_data)
+
+    @property
+    def lr(self) -> int:
+        """Return Lr; Length of R-APDU body."""
+        return len(self.rsp_data)
+
+    @property
+    def successful(self) -> bool:
+        """Was the execution of this APDU successful?"""
+        method = getattr(self, '_is_success', None)
+        if callable(method):
+            return method()
+        # default case: only 9000 is success
+        if self.sw == b'\x90\x00':
+            return True
+        # This is not really a generic positive APDU SW but specific to UICC/SIM
+        if self.sw[0] == 0x91:
+            return True
+        return False
+
+
+class ApduCommand(Apdu, metaclass=ApduCommandMeta):
+    """Base class from which you would derive individual commands/instructions like SELECT.
+       A derived class represents a decoder for a specific instruction.
+       An instance of such a derived class is one concrete APDU."""
+    # fall-back constructs if the derived class provides no override
+    _construct_p1 = Byte
+    _construct_p2 = Byte
+    _construct = GreedyBytes
+    _construct_rsp = GreedyBytes
+    _tlv = None
+    _tlv_rsp = None
+
+    def __init__(self, cmd: BytesOrHex, rsp: Optional[BytesOrHex] = None):
+        """Instantiate a new ApduCommand from give cmd + resp."""
+        # store raw data
+        super().__init__(cmd, rsp)
+        # default to 'empty' ID column. To be set to useful values (like record number)
+        # by derived class {cmd_rsp}_to_dict() or process() methods
+        self.col_id = '-'
+        # fields only set by process_* methods
+        self.file = None
+        self.lchan = None
+        self.processed = None
+        # the methods below could raise exceptions and those handlers might assume cmd_{dict,resp}
+        self.cmd_dict = None
+        self.rsp_dict = None
+        # interpret the data
+        self.cmd_dict = self.cmd_to_dict()
+        self.rsp_dict = self.rsp_to_dict() if self.rsp else {}
+
+
+    @classmethod
+    def from_apdu(cls, apdu:Apdu, **kwargs) -> 'ApduCommand':
+        """Instantiate an ApduCommand from an existing APDU."""
+        return cls(cmd=apdu.cmd, rsp=apdu.rsp, **kwargs)
+
+    @classmethod
+    def from_bytes(cls, buffer:bytes) -> 'ApduCommand':
+        """Instantiate an ApduCommand from a linear byte buffer containing hdr,cmd,rsp,sw.
+        This is for example used when parsing GSMTAP traces that traditionally contain the
+        full command and response portion in one packet: "CLA INS P1 P2 P3 DATA SW" and we
+        now need to figure out whether the DATA part is part of the CMD or the RSP"""
+        apdu_case = cls.get_apdu_case(buffer)
+        if apdu_case in [1, 2]:
+            # data is part of response
+            return cls(buffer[:5], buffer[5:])
+        if apdu_case in [3, 4]:
+            # data is part of command
+            lc = buffer[4]
+            return cls(buffer[:5+lc], buffer[5+lc:])
+        raise ValueError('%s: Invalid APDU Case %u' % (cls.__name__, apdu_case))
+
+    @property
+    def path(self) -> List[str]:
+        """Return (if known) the path as list of files to the file on which this command operates."""
+        if self.file:
+            return self.file.fully_qualified_path()
+        return []
+
+    @property
+    def path_str(self) -> str:
+        """Return (if known) the path as string to the file on which this command operates."""
+        if self.file:
+            return self.file.fully_qualified_path_str()
+        return ''
+
+    @property
+    def col_sw(self) -> str:
+        """Return the ansi-colorized status word. Green==OK, Red==Error"""
+        if self.successful:
+            return colored(b2h(self.sw), 'green')
+        return colored(b2h(self.sw), 'red')
+
+    @property
+    def lchan_nr(self) -> int:
+        """Logical channel number over which this ApduCommand was transmitted."""
+        if self.lchan:
+            return self.lchan.lchan_nr
+        return lchan_nr_from_cla(self.cla)
+
+    def __str__(self) -> str:
+        return '%02u %s(%s): %s' % (self.lchan_nr, type(self).__name__, self.path_str, self.to_dict())
+
+    def __repr__(self) -> str:
+        return '%s(INS=%02x,CLA=%s)' % (self.__class__, self.ins, self.cla)
+
+    def _process_fallback(self, rs: RuntimeState):
+        """Fall-back function to be called if there is no derived-class-specific
+        process_global or process_on_lchan method. Uses information from APDU decode."""
+        self.processed = {}
+        if 'p1' not in self.cmd_dict:
+            self.processed = self.to_dict()
+        else:
+            self.processed['p1'] = self.cmd_dict['p1']
+            self.processed['p2'] = self.cmd_dict['p2']
+            if 'body' in self.cmd_dict and self.cmd_dict['body']:
+                self.processed['cmd'] = self.cmd_dict['body']
+            if 'body' in self.rsp_dict and self.rsp_dict['body']:
+                self.processed['rsp'] = self.rsp_dict['body']
+        return self.processed
+
+    def process(self, rs: RuntimeState):
+        # if there is a global method, use that; else use process_on_lchan
+        method = getattr(self, 'process_global', None)
+        if callable(method):
+            self.processed = method(rs)
+            return self.processed
+        method = getattr(self, 'process_on_lchan', None)
+        if callable(method):
+            self.lchan = rs.get_lchan_by_cla(self.cla)
+            self.processed = method(self.lchan)
+            return self.processed
+        # if none of the two methods exist:
+        return self._process_fallback(rs)
+
+    @classmethod
+    def get_apdu_case(cls, hdr:bytes) -> int:
+        if hasattr(cls, '_apdu_case'):
+            return cls._apdu_case
+        method = getattr(cls, '_get_apdu_case', None)
+        if callable(method):
+            return method(hdr)
+        raise ValueError('%s: Class definition missing _apdu_case attribute or _get_apdu_case method' % cls.__name__)
+
+    @classmethod
+    def match_cla(cls, cla) -> bool:
+        """Does the given CLA match the CLA list of the command?."""
+        if not isinstance(cla, str):
+            cla = '%02X' % cla
+        cla = cla.upper()
+        # see https://github.com/PyCQA/pylint/issues/7219
+        # pylint: disable=no-member
+        for cla_match in cls._cla:
+            cla_masked = ""
+            for i in range(0, 2):
+                if cla_match[i] == 'X':
+                    cla_masked += 'X'
+                else:
+                    cla_masked += cla[i]
+            if cla_masked == cla_match.upper():
+                return True
+        return False
+
+    def cmd_to_dict(self) -> Dict:
+        """Convert the Command part of the APDU to a dict."""
+        method = getattr(self, '_decode_cmd', None)
+        if callable(method):
+            return method()
+        else:
+            return self._cmd_to_dict()
+
+    def _cmd_to_dict(self) -> Dict:
+        """back-end function performing automatic decoding using _construct / _tlv."""
+        r = {}
+        method = getattr(self, '_decode_p1p2', None)
+        if callable(method):
+            r = self._decode_p1p2()
+        else:
+            r['p1'] = parse_construct(self._construct_p1, self.p1.to_bytes(1, 'big'))
+            r['p2'] = parse_construct(self._construct_p2, self.p2.to_bytes(1, 'big'))
+        r['p3'] = self.p3
+        if self.cmd_data:
+            if self._tlv:
+                ie = self._tlv()
+                ie.from_tlv(self.cmd_data)
+                r['body'] = ie.to_dict()
+            else:
+                r['body'] = parse_construct(self._construct, self.cmd_data)
+        return r
+
+    def rsp_to_dict(self) -> Dict:
+        """Convert the Response part of the APDU to a dict."""
+        method = getattr(self, '_decode_rsp', None)
+        if callable(method):
+            return method()
+        else:
+            r = {}
+            if self.rsp_data:
+                if self._tlv_rsp:
+                    ie = self._tlv_rsp()
+                    ie.from_tlv(self.rsp_data)
+                    r['body'] = ie.to_dict()
+                else:
+                    r['body'] = parse_construct(self._construct_rsp, self.rsp_data)
+            r['sw'] = b2h(self.sw)
+            return r
+
+    def to_dict(self) -> Dict:
+        """Convert the entire APDU to a dict."""
+        return {'cmd': self.cmd_dict, 'rsp': self.rsp_dict}
+
+    def to_json(self) -> str:
+        """Convert the entire APDU to JSON."""
+        d = self.to_dict()
+        return json.dumps(d)
+
+    def _determine_file(self, lchan) -> CardFile:
+        """Helper function for read/update commands that might use SFI instead of selected file.
+        Expects that the self.cmd_dict has already been populated with the 'file' member."""
+        if self.cmd_dict['file'] == 'currently_selected_ef':
+            self.file = lchan.selected_file
+        elif self.cmd_dict['file'] == 'sfi':
+            cwd = lchan.get_cwd()
+            self.file = cwd.lookup_file_by_sfid(self.cmd_dict['sfi'])
+
+
+class ApduCommandSet:
+    """A set of card instructions, typically specified within one spec."""
+
+    def __init__(self, name: str, cmds: List[ApduCommand] =[]):
+        self.name = name
+        self.cmds = {c._ins: c for c in cmds}
+
+    def __str__(self) -> str:
+        return self.name
+
+    def __getitem__(self, idx) -> ApduCommand:
+        return self.cmds[idx]
+
+    def __add__(self, other) -> 'ApduCommandSet':
+        if isinstance(other, ApduCommand):
+            if other.ins in self.cmds:
+                raise ValueError('%s: INS 0x%02x already defined: %s' %
+                                 (self, other.ins, self.cmds[other.ins]))
+            self.cmds[other.ins] = other
+        elif isinstance(other, ApduCommandSet):
+            for c in other.cmds.keys():
+                self.cmds[c] = other.cmds[c]
+        else:
+            raise ValueError(
+                '%s: Unsupported type to add operator: %s' % (self, other))
+        return self
+
+    def lookup(self, ins, cla=None) -> Optional[ApduCommand]:
+        """look-up the command within the CommandSet."""
+        ins = int(ins)
+        if not ins in self.cmds:
+            return None
+        cmd = self.cmds[ins]
+        if cla and not cmd.match_cla(cla):
+            return None
+        return cmd
+
+    def parse_cmd_apdu(self, apdu: Apdu) -> ApduCommand:
+        """Parse a Command-APDU. Returns an instance of an ApduCommand derived class."""
+        # first look-up which of our member classes match CLA + INS
+        a_cls = self.lookup(apdu.ins, apdu.cla)
+        if not a_cls:
+            raise ValueError('Unknown CLA=%02X INS=%02X' % (apdu.cla, apdu.ins))
+        # then create an instance of that class and return it
+        return a_cls.from_apdu(apdu)
+
+    def parse_cmd_bytes(self, buf:bytes) -> ApduCommand:
+        """Parse from a buffer (simtrace style). Returns an instance of an ApduCommand derived class."""
+        # first look-up which of our member classes match CLA + INS
+        cla = buf[0]
+        ins = buf[1]
+        a_cls = self.lookup(ins, cla)
+        if not a_cls:
+            raise ValueError('Unknown CLA=%02X INS=%02X' % (cla, ins))
+        # then create an instance of that class and return it
+        return a_cls.from_bytes(buf)
+
+
+
+class ApduHandler(abc.ABC):
+    @abc.abstractmethod
+    def input(self, cmd: bytes, rsp: bytes):
+        pass
+
+
+class TpduFilter(ApduHandler):
+    """The TpduFilter removes the T=0 specific GET_RESPONSE from the TPDU stream and
+       calls the ApduHandler only with the actual APDU command and response parts."""
+    def __init__(self, apdu_handler: ApduHandler):
+        self.apdu_handler = apdu_handler
+        self.state = 'INIT'
+        self.last_cmd = None
+
+    def input_tpdu(self, tpdu:Tpdu):
+        # handle SW=61xx / 6Cxx
+        if tpdu.sw[0] == 0x61 or tpdu.sw[0] == 0x6C:
+            self.state = 'WAIT_GET_RESPONSE'
+            # handle successive 61/6c responses by stupid phone/modem OS
+            if tpdu.ins != 0xC0:
+                self.last_cmd = tpdu.cmd
+            return None
+        else:
+            if self.last_cmd:
+                icmd = self.last_cmd
+                self.last_cmd = None
+            else:
+                icmd = tpdu.cmd
+            apdu = Apdu(icmd, tpdu.rsp)
+            if self.apdu_handler:
+                return self.apdu_handler.input(apdu)
+            return Apdu(icmd, tpdu.rsp)
+
+    def input(self, cmd: bytes, rsp: bytes):
+        if isinstance(cmd, str):
+            cmd = bytes.fromhex(cmd)
+        if isinstance(rsp, str):
+            rsp = bytes.fromhex(rsp)
+        tpdu = Tpdu(cmd, rsp)
+        return self.input_tpdu(tpdu)
+
+class ApduDecoder(ApduHandler):
+    def __init__(self, cmd_set: ApduCommandSet):
+        self.cmd_set = cmd_set
+
+    def input(self, apdu: Apdu):
+        return self.cmd_set.parse_cmd_apdu(apdu)
+
+
+class CardReset:
+    def __init__(self, atr: bytes):
+        self.atr = atr
+
+    def __str__(self):
+        if self.atr:
+            return '%s(%s)' % (type(self).__name__, b2h(self.atr))
+        return '%s' % (type(self).__name__)

pySim/apdu/global_platform.py

@@ -0,0 +1,82 @@
+# coding=utf-8
+"""APDU definition/decoder of GlobalPLatform Card Spec (currently 2.1.1)
+
+(C) 2022-2024 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from construct import FlagsEnum, Struct
+from osmocom.tlv import flatten_dict_lists
+from osmocom.construct import *
+from pySim.apdu import ApduCommand, ApduCommandSet
+from pySim.global_platform import InstallParameters
+
+class GpDelete(ApduCommand, n='DELETE', ins=0xE4, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+class GpStoreData(ApduCommand, n='STORE DATA', ins=0xE2, cla=['8X', 'CX', 'EX']):
+    @classmethod
+    def _get_apdu_case(cls, hdr:bytes) -> int:
+        p1 = hdr[2]
+        if p1 & 0x01:
+            return 4
+        else:
+            return 3
+
+class GpGetDataCA(ApduCommand, n='GET DATA', ins=0xCA, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+class GpGetDataCB(ApduCommand, n='GET DATA', ins=0xCB, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+class GpGetStatus(ApduCommand, n='GET STATUS', ins=0xF2, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+# GPCS Section 11.5.2
+class GpInstall(ApduCommand, n='INSTALL', ins=0xE6, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+    _construct_p1 = FlagsEnum(Byte, more_commands=0x80, for_registry_update=0x40,
+                              for_personalization=0x20, for_extradition=0x10,
+                              for_make_selectable=0x08, for_install=0x04, for_load=0x02)
+    _construct_p2 = Enum(Byte, no_info_provided=0x00, beginning_of_combined=0x01,
+                         end_of_combined=0x03)
+    _construct = Struct('load_file_aid'/Prefixed(Int8ub, GreedyBytes),
+                        'module_aid'/Prefixed(Int8ub, GreedyBytes),
+                        'application_aid'/Prefixed(Int8ub, GreedyBytes),
+                        'privileges'/Prefixed(Int8ub, GreedyBytes),
+                        'install_parameters'/Prefixed(Int8ub, GreedyBytes), # TODO: InstallParameters
+                        'install_token'/Prefixed(Int8ub, GreedyBytes))
+    def _decode_cmd(self):
+        # first use _construct* above
+        res = self._cmd_to_dict()
+        # then do TLV decode of install_parameters
+        ip = InstallParameters()
+        ip.from_tlv(res['body']['install_parameters'])
+        res['body']['install_parameters'] = flatten_dict_lists(ip.to_dict())
+        return res
+
+
+class GpLoad(ApduCommand, n='LOAD', ins=0xE8, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+class GpPutKey(ApduCommand, n='PUT KEY', ins=0xD8, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+class GpSetStatus(ApduCommand, n='SET STATUS', ins=0xF0, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 3
+
+ApduCommands = ApduCommandSet('GlobalPlatform v2.3.1', cmds=[GpDelete, GpStoreData,
+                              GpGetDataCA, GpGetDataCB, GpGetStatus, GpInstall,
+                              GpLoad, GpPutKey, GpSetStatus])

pySim/apdu/ts_102_221.py

@@ -0,0 +1,531 @@
+# coding=utf-8
+"""APDU definitions/decoders of ETSI TS 102 221, the core UICC spec.
+
+(C) 2022-2024 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from typing import Optional, Dict
+import logging
+
+from construct import GreedyRange, Struct
+
+from osmocom.utils import i2h
+from osmocom.construct import *
+
+from pySim.filesystem import *
+from pySim.runtime import RuntimeLchan
+from pySim.apdu import ApduCommand, ApduCommandSet
+from pySim import cat
+
+logger = logging.getLogger(__name__)
+
+# TS 102 221 Section 11.1.1
+class UiccSelect(ApduCommand, n='SELECT', ins=0xA4, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_p1 = Enum(Byte, df_ef_or_mf_by_file_id=0, child_df_of_current_df=1, parent_df_of_current_df=3,
+                         df_name=4, path_from_mf=8, path_from_current_df=9)
+    _construct_p2 = BitStruct(Flag,
+                              'app_session_control'/Enum(BitsInteger(2), activation_reset=0, termination=2),
+                              'return'/Enum(BitsInteger(3), fcp=1, no_data=3),
+                              'aid_control'/Enum(BitsInteger(2), first_or_only=0, last=1, next=2, previous=3))
+
+    @staticmethod
+    def _find_aid_substr(selectables, aid) -> Optional[CardADF]:
+        # full-length match
+        if aid in selectables:
+            return selectables[aid]
+        # sub-string match
+        for s in selectables.keys():
+            if aid[:len(s)] == s:
+                return selectables[s]
+        return None
+
+    def process_on_lchan(self, lchan: RuntimeLchan):
+        mode = self.cmd_dict['p1']
+        if mode in ['path_from_mf', 'path_from_current_df']:
+            # rewind to MF, if needed
+            if mode == 'path_from_mf':
+                lchan.selected_file = lchan.rs.mf
+            path = [self.cmd_data[i:i+2] for i in range(0, len(self.cmd_data), 2)]
+            for file in path:
+                file_hex = b2h(file)
+                if file_hex == '7fff': # current application
+                    if not lchan.selected_adf:
+                        sels = lchan.rs.mf.get_app_selectables(['ANAMES'])
+                        # HACK: Assume USIM
+                        logger.warning('SELECT relative to current ADF, but no ADF selected. Assuming ADF.USIM')
+                        lchan.selected_adf = sels['ADF.USIM']
+                    lchan.selected_file = lchan.selected_adf
+                    #print("\tSELECT CUR_ADF %s" % lchan.selected_file)
+                    # iterate to next element in path
+                    continue
+                else:
+                    sels = lchan.selected_file.get_selectables(['FIDS','MF','PARENT','SELF'])
+                    if file_hex in sels:
+                        if self.successful:
+                            #print("\tSELECT %s" % sels[file_hex])
+                            lchan.selected_file = sels[file_hex]
+                        else:
+                            #print("\tSELECT %s FAILED" % sels[file_hex])
+                            pass
+                        # iterate to next element in path
+                        continue
+                logger.warning('SELECT UNKNOWN FID %s (%s)', file_hex, '/'.join([b2h(x) for x in path]))
+        elif mode == 'df_ef_or_mf_by_file_id':
+            if len(self.cmd_data) != 2:
+                raise ValueError('Expecting a 2-byte FID')
+            sels = lchan.selected_file.get_selectables(['FIDS','MF','PARENT','SELF'])
+            file_hex = b2h(self.cmd_data)
+            if file_hex in sels:
+                if self.successful:
+                    #print("\tSELECT %s" % sels[file_hex])
+                    lchan.selected_file = sels[file_hex]
+                else:
+                    #print("\tSELECT %s FAILED" % sels[file_hex])
+                    pass
+            else:
+                logger.warning('SELECT UNKNOWN FID %s', file_hex)
+        elif mode == 'df_name':
+            # Select by AID (can be sub-string!)
+            aid = b2h(self.cmd_dict['body'])
+            sels = lchan.rs.mf.get_app_selectables(['AIDS'])
+            adf = self._find_aid_substr(sels, aid)
+            if adf:
+                lchan.selected_adf = adf
+                lchan.selected_file = lchan.selected_adf
+                #print("\tSELECT AID %s" % adf)
+            else:
+                logger.warning('SELECT UNKNOWN AID %s', aid)
+        else:
+            raise ValueError('Select Mode %s not implemented' % mode)
+        # decode the SELECT response
+        if self.successful:
+            self.file = lchan.selected_file
+            if 'body' in self.rsp_dict:
+                # not every SELECT is asking for the FCP in response...
+                return lchan.selected_file.decode_select_response(b2h(self.rsp_dict['body']))
+        return None
+
+
+
+# TS 102 221 Section 11.1.2
+class UiccStatus(ApduCommand, n='STATUS', ins=0xF2, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 2
+    _construct_p1 = Enum(Byte, no_indication=0, current_app_is_initialized=1, terminal_will_terminate_current_app=2)
+    _construct_p2 = Enum(Byte, response_like_select=0, response_df_name_tlv=1, response_no_data=0x0c)
+
+    def process_on_lchan(self, lchan):
+        if self.cmd_dict['p2'] == 'response_like_select':
+            return lchan.selected_file.decode_select_response(b2h(self.rsp_dict['body']))
+
+def _decode_binary_p1p2(p1, p2) -> Dict:
+    ret = {}
+    if p1 & 0x80:
+        ret['file'] = 'sfi'
+        ret['sfi'] = p1 & 0x1f
+        ret['offset'] = p2
+    else:
+        ret['file'] = 'currently_selected_ef'
+        ret['offset'] = ((p1 & 0x7f) << 8) & p2
+    return ret
+
+# TS 102 221 Section 11.1.3
+class ReadBinary(ApduCommand, n='READ BINARY', ins=0xB0, cla=['0X', '4X', '6X']):
+    _apdu_case = 2
+    def _decode_p1p2(self):
+        return _decode_binary_p1p2(self.p1, self.p2)
+
+    def process_on_lchan(self, lchan):
+        self._determine_file(lchan)
+        if not isinstance(self.file, TransparentEF):
+            return b2h(self.rsp_data)
+        # our decoders don't work for non-zero offsets / short reads
+        if self.cmd_dict['offset'] != 0 or self.lr < self.file.size[0]:
+            return b2h(self.rsp_data)
+        method = getattr(self.file, 'decode_bin', None)
+        if self.successful and callable(method):
+            return method(self.rsp_data)
+
+# TS 102 221 Section 11.1.4
+class UpdateBinary(ApduCommand, n='UPDATE BINARY', ins=0xD6, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    def _decode_p1p2(self):
+        return _decode_binary_p1p2(self.p1, self.p2)
+
+    def process_on_lchan(self, lchan):
+        self._determine_file(lchan)
+        if not isinstance(self.file, TransparentEF):
+            return b2h(self.rsp_data)
+        # our decoders don't work for non-zero offsets / short writes
+        if self.cmd_dict['offset'] != 0 or self.lc < self.file.size[0]:
+            return b2h(self.cmd_data)
+        method = getattr(self.file, 'decode_bin', None)
+        if self.successful and callable(method):
+            return method(self.cmd_data)
+
+def _decode_record_p1p2(p1, p2):
+    ret = {}
+    ret['record_number'] = p1
+    if p2 >> 3 == 0:
+        ret['file'] = 'currently_selected_ef'
+    else:
+        ret['file'] = 'sfi'
+        ret['sfi'] = p2 >> 3
+    mode = p2 & 0x7
+    if mode == 2:
+        ret['mode'] = 'next_record'
+    elif mode == 3:
+        ret['mode'] = 'previous_record'
+    elif mode == 8:
+        ret['mode'] = 'absolute_current'
+    return ret
+
+# TS 102 221 Section 11.1.5
+class ReadRecord(ApduCommand, n='READ RECORD', ins=0xB2, cla=['0X', '4X', '6X']):
+    _apdu_case = 2
+    def _decode_p1p2(self):
+        r = _decode_record_p1p2(self.p1, self.p2)
+        self.col_id = '%02u' % r['record_number']
+        return r
+
+    def process_on_lchan(self, lchan):
+        self._determine_file(lchan)
+        if not isinstance(self.file, LinFixedEF):
+            return b2h(self.rsp_data)
+        method = getattr(self.file, 'decode_record_bin', None)
+        if self.successful and callable(method):
+            return method(self.rsp_data, self.cmd_dict['record_number'])
+
+# TS 102 221 Section 11.1.6
+class UpdateRecord(ApduCommand, n='UPDATE RECORD', ins=0xDC, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    def _decode_p1p2(self):
+        r = _decode_record_p1p2(self.p1, self.p2)
+        self.col_id = '%02u' % r['record_number']
+        return r
+
+    def process_on_lchan(self, lchan):
+        self._determine_file(lchan)
+        if not isinstance(self.file, LinFixedEF):
+            return b2h(self.cmd_data)
+        method = getattr(self.file, 'decode_record_bin', None)
+        if self.successful and callable(method):
+            return method(self.cmd_data, self.cmd_dict['record_number'])
+
+# TS 102 221 Section 11.1.7
+class SearchRecord(ApduCommand, n='SEARCH RECORD', ins=0xA2, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_rsp = GreedyRange(Int8ub)
+
+    def _decode_p1p2(self):
+        ret = {}
+        sfi = self.p2 >> 3
+        if sfi == 0:
+            ret['file'] = 'currently_selected_ef'
+        else:
+            ret['file'] = 'sfi'
+            ret['sfi'] = sfi
+        mode = self.p2 & 0x7
+        if mode in [0x4, 0x5]:
+            if mode == 0x4:
+                ret['mode'] = 'forward_search'
+            else:
+                ret['mode'] = 'backward_search'
+            ret['record_number'] = self.p1
+            self.col_id = '%02u' % ret['record_number']
+        elif mode == 6:
+            ret['mode'] = 'enhanced_search'
+            # TODO: further decode
+        elif mode == 7:
+            ret['mode'] = 'proprietary_search'
+        return ret
+
+    def _decode_cmd(self):
+        ret = self._decode_p1p2()
+        if self.cmd_data:
+            if ret['mode'] == 'enhanced_search':
+                ret['search_indication'] = b2h(self.cmd_data[:2])
+                ret['search_string'] = b2h(self.cmd_data[2:])
+            else:
+                ret['search_string'] = b2h(self.cmd_data)
+        return ret
+
+    def process_on_lchan(self, lchan):
+        self._determine_file(lchan)
+        return self.to_dict()
+
+# TS 102 221 Section 11.1.8
+class Increase(ApduCommand, n='INCREASE', ins=0x32, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+PinConstructP2 = BitStruct('scope'/Enum(Flag, global_mf=0, specific_df_adf=1),
+                           BitsInteger(2), 'reference_data_nr'/BitsInteger(5))
+# TS 102 221 Section 11.1.9
+class VerifyPin(ApduCommand, n='VERIFY PIN', ins=0x20, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    _construct_p2 = PinConstructP2
+
+    @staticmethod
+    def _pin_process(apdu):
+        processed = {
+            'scope': apdu.cmd_dict['p2']['scope'],
+            'referenced_data_nr': apdu.cmd_dict['p2']['reference_data_nr'],
+            }
+        if apdu.lc == 0:
+            # this is just a question on the counters remaining
+            processed['mode'] = 'check_remaining_attempts'
+        else:
+            processed['pin'] = b2h(apdu.cmd_data)
+        if apdu.sw[0] == 0x63:
+            processed['remaining_attempts'] = apdu.sw[1] & 0xf
+        return processed
+
+    @staticmethod
+    def _pin_is_success(sw):
+        return bool(sw[0] == 0x63)
+
+    def process_on_lchan(self, _lchan: RuntimeLchan):
+        return VerifyPin._pin_process(self)
+
+    def _is_success(self):
+        return VerifyPin._pin_is_success(self.sw)
+
+
+# TS 102 221 Section 11.1.10
+class ChangePin(ApduCommand, n='CHANGE PIN', ins=0x24, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    _construct_p2 = PinConstructP2
+
+    def process_on_lchan(self, _lchan: RuntimeLchan):
+        return VerifyPin._pin_process(self)
+
+    def _is_success(self):
+        return VerifyPin._pin_is_success(self.sw)
+
+
+# TS 102 221 Section 11.1.11
+class DisablePin(ApduCommand, n='DISABLE PIN', ins=0x26, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    _construct_p2 = PinConstructP2
+
+    def process_on_lchan(self, _lchan: RuntimeLchan):
+        return VerifyPin._pin_process(self)
+
+    def _is_success(self):
+        return VerifyPin._pin_is_success(self.sw)
+
+
+# TS 102 221 Section 11.1.12
+class EnablePin(ApduCommand, n='ENABLE PIN', ins=0x28, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    _construct_p2 = PinConstructP2
+    def process_on_lchan(self, _lchan: RuntimeLchan):
+        return VerifyPin._pin_process(self)
+
+    def _is_success(self):
+        return VerifyPin._pin_is_success(self.sw)
+
+
+# TS 102 221 Section 11.1.13
+class UnblockPin(ApduCommand, n='UNBLOCK PIN', ins=0x2C, cla=['0X', '4X', '6X']):
+    _apdu_case = 3
+    _construct_p2 = PinConstructP2
+
+    def process_on_lchan(self, _lchan: RuntimeLchan):
+        return VerifyPin._pin_process(self)
+
+    def _is_success(self):
+        return VerifyPin._pin_is_success(self.sw)
+
+
+# TS 102 221 Section 11.1.14
+class DeactivateFile(ApduCommand, n='DEACTIVATE FILE', ins=0x04, cla=['0X', '4X', '6X']):
+    _apdu_case = 1
+    _construct_p1 = BitStruct(BitsInteger(4),
+                              'select_mode'/Enum(BitsInteger(4), ef_by_file_id=0,
+                                                 path_from_mf=8, path_from_current_df=9))
+
+# TS 102 221 Section 11.1.15
+class ActivateFile(ApduCommand, n='ACTIVATE FILE', ins=0x44, cla=['0X', '4X', '6X']):
+    _apdu_case = 1
+    _construct_p1 = DeactivateFile._construct_p1
+
+# TS 102 221 Section 11.1.16
+auth_p2_construct = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1),
+                              BitsInteger(2),
+                              'reference_data_nr'/BitsInteger(5))
+class Authenticate88(ApduCommand, n='AUTHENTICATE', ins=0x88, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_p2 = auth_p2_construct
+
+# TS 102 221 Section 11.1.16
+class Authenticate89(ApduCommand, n='AUTHENTICATE', ins=0x89, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_p2 = auth_p2_construct
+
+# TS 102 221 Section 11.1.17
+class ManageChannel(ApduCommand, n='MANAGE CHANNEL', ins=0x70, cla=['0X', '4X', '6X']):
+    _apdu_case = 2
+    _construct_p1 = Enum(Flag, open_channel=0, close_channel=1)
+    _construct_p2 = Struct('logical_channel_number'/Int8ub)
+    _construct_rsp = Struct('logical_channel_number'/Int8ub)
+
+    def process_global(self, rs):
+        if not self.successful:
+            return
+        mode = self.cmd_dict['p1']
+        if mode == 'open_channel':
+            created_channel_nr = self.cmd_dict['p2']['logical_channel_number']
+            if created_channel_nr == 0:
+                # auto-assignment by UICC
+                # pylint: disable=unsubscriptable-object
+                created_channel_nr = self.rsp_data[0]
+            manage_channel = rs.get_lchan_by_cla(self.cla)
+            manage_channel.add_lchan(created_channel_nr)
+            self.col_id = '%02u' % created_channel_nr
+            return {'mode': mode, 'created_channel': created_channel_nr }
+        if mode == 'close_channel':
+            closed_channel_nr = self.cmd_dict['p2']['logical_channel_number']
+            rs.del_lchan(closed_channel_nr)
+            self.col_id = '%02u' % closed_channel_nr
+            return {'mode': mode, 'closed_channel': closed_channel_nr }
+        raise ValueError('Unsupported MANAGE CHANNEL P1=%02X' % self.p1)
+
+# TS 102 221 Section 11.1.18
+class GetChallenge(ApduCommand, n='GET CHALLENGE', ins=0x84, cla=['0X', '4X', '6X']):
+    _apdu_case = 2
+
+# TS 102 221 Section 11.1.19
+class TerminalCapability(ApduCommand, n='TERMINAL CAPABILITY', ins=0xAA, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 3
+
+# TS 102 221 Section 11.1.20
+class ManageSecureChannel(ApduCommand, n='MANAGE SECURE CHANNEL', ins=0x73, cla=['0X', '4X', '6X']):
+    @classmethod
+    def _get_apdu_case(cls, hdr:bytes) -> int:
+        p1 = hdr[2]
+        p2 = hdr[3]
+        if p1 & 0x7 == 0:   # retrieve UICC Endpoints
+            return 2
+        if p1 & 0xf in [1,2,3]:   # establish sa, start secure channel SA
+            p2_cmd = p2 >> 5
+            if p2_cmd in [0,2,4]:   # command data
+                return 3
+            if p2_cmd in [1,3,5]: # response data
+                return 2
+        if p1 & 0xf == 4:         # terminate secure channel SA
+            return 3
+        raise ValueError('%s: Unable to detect APDU case for %s' % (cls.__name__, b2h(hdr)))
+
+# TS 102 221 Section 11.1.21
+class TransactData(ApduCommand, n='TRANSACT DATA', ins=0x75, cla=['0X', '4X', '6X']):
+    @classmethod
+    def _get_apdu_case(cls, hdr:bytes) -> int:
+        p1 = hdr[2]
+        if p1 & 0x04:
+            return 3
+        return 2
+
+# TS 102 221 Section 11.1.22
+class SuspendUicc(ApduCommand, n='SUSPEND UICC', ins=0x76, cla=['80']):
+    _apdu_case = 4
+    _construct_p1 = BitStruct('rfu'/BitsInteger(7), 'mode'/Enum(Flag, suspend=0, resume=1))
+
+# TS 102 221 Section 11.1.23
+class GetIdentity(ApduCommand, n='GET IDENTITY', ins=0x78, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+    _construct_p2 = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1), BitsInteger(7))
+
+# TS 102 221 Section 11.1.24
+class ExchangeCapabilities(ApduCommand, n='EXCHANGE CAPABILITIES', ins=0x7A, cla=['80']):
+    _apdu_case = 4
+
+# TS 102 221 Section 11.2.1
+class TerminalProfile(ApduCommand, n='TERMINAL PROFILE', ins=0x10, cla=['80']):
+    _apdu_case = 3
+
+# TS 102 221 Section 11.2.2 / TS 102 223
+class Envelope(ApduCommand, n='ENVELOPE', ins=0xC2, cla=['80']):
+    _apdu_case = 4
+    _tlv = cat.EventCollection
+
+# TS 102 221 Section 11.2.3 / TS 102 223
+class Fetch(ApduCommand, n='FETCH', ins=0x12, cla=['80']):
+    _apdu_case = 2
+    _tlv_rsp = cat.ProactiveCommand
+
+# TS 102 221 Section 11.2.3 / TS 102 223
+class TerminalResponse(ApduCommand, n='TERMINAL RESPONSE', ins=0x14, cla=['80']):
+    _apdu_case = 3
+    _tlv = cat.TerminalResponse
+
+# TS 102 221 Section 11.3.1
+class RetrieveData(ApduCommand, n='RETRIEVE DATA', ins=0xCB, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+
+    @staticmethod
+    def _tlv_decode_cmd(self : ApduCommand) -> Dict:
+        c = {}
+        if self.p2 & 0xc0 == 0x80:
+            c['mode'] = 'first_block'
+            sfi = self.p2 & 0x1f
+            if sfi == 0:
+                c['file'] = 'currently_selected_ef'
+            else:
+                c['file'] = 'sfi'
+                c['sfi'] = sfi
+            c['tag'] = i2h([self.cmd_data[0]])
+        elif self.p2 & 0xdf == 0x00:
+            c['mode'] = 'next_block'
+        elif self.p2 & 0xdf == 0x40:
+            c['mode'] = 'retransmit_previous_block'
+        else:
+            logger.warning('%s: invalid P2=%02x', self, self.p2)
+        return c
+
+    def _decode_cmd(self):
+        return RetrieveData._tlv_decode_cmd(self)
+
+    def _decode_rsp(self):
+        # TODO: parse tag/len/val?
+        return b2h(self.rsp_data)
+
+
+# TS 102 221 Section 11.3.2
+class SetData(ApduCommand, n='SET DATA', ins=0xDB, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 3
+
+    def _decode_cmd(self):
+        c = RetrieveData._tlv_decode_cmd(self)
+        if c['mode'] == 'first_block':
+            if len(self.cmd_data) == 0:
+                c['delete'] = True
+        # TODO: parse tag/len/val?
+        c['data'] = b2h(self.cmd_data)
+        return c
+
+
+# TS 102 221 Section 12.1.1
+class GetResponse(ApduCommand, n='GET RESPONSE', ins=0xC0, cla=['0X', '4X', '6X']):
+    _apdu_case = 2
+
+ApduCommands = ApduCommandSet('TS 102 221', cmds=[UiccSelect, UiccStatus, ReadBinary, UpdateBinary, ReadRecord,
+                              UpdateRecord, SearchRecord, Increase, VerifyPin, ChangePin, DisablePin,
+                              EnablePin, UnblockPin, DeactivateFile, ActivateFile, Authenticate88,
+                              Authenticate89, ManageChannel, GetChallenge, TerminalCapability,
+                              ManageSecureChannel, TransactData, SuspendUicc, GetIdentity,
+                              ExchangeCapabilities, TerminalProfile, Envelope, Fetch, TerminalResponse,
+                              RetrieveData, SetData, GetResponse])

pySim/apdu/ts_102_222.py

@@ -0,0 +1,60 @@
+# coding=utf-8
+"""APDU definitions/decoders of ETSI TS 102 222.
+
+(C) 2022-2024 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+import logging
+
+from construct import Struct
+from osmocom.construct import *
+
+from pySim.apdu import ApduCommand, ApduCommandSet
+from pySim.ts_102_221 import FcpTemplate
+
+logger = logging.getLogger(__name__)
+
+# TS 102 222 Section 6.3
+class CreateFile(ApduCommand, n='CREATE FILE', ins=0xE0, cla=['0X', '4X', 'EX']):
+    _apdu_case = 3
+    _tlv = FcpTemplate
+
+# TS 102 222 Section 6.4
+class DeleteFile(ApduCommand, n='DELETE FILE', ins=0xE4, cla=['0X', '4X']):
+    _apdu_case = 3
+    _construct = Struct('file_id'/Bytes(2))
+
+# TS 102 222 Section 6.7
+class TerminateDF(ApduCommand, n='TERMINATE DF', ins=0xE6, cla=['0X', '4X']):
+    _apdu_case = 1
+
+# TS 102 222 Section 6.8
+class TerminateEF(ApduCommand, n='TERMINATE EF', ins=0xE8, cla=['0X', '4X']):
+    _apdu_case = 1
+
+# TS 102 222 Section 6.9
+class TerminateCardUsage(ApduCommand, n='TERMINATE CARD USAGE', ins=0xFE, cla=['0X', '4X']):
+    _apdu_case = 1
+
+# TS 102 222 Section 6.10
+class ResizeFile(ApduCommand, n='RESIZE FILE', ins=0xD4, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 3
+    _construct_p1 = Enum(Byte, mode_0=0, mode_1=1)
+    _tlv = FcpTemplate
+
+
+ApduCommands = ApduCommandSet('TS 102 222', cmds=[CreateFile, DeleteFile, TerminateDF,
+                                                  TerminateEF, TerminateCardUsage, ResizeFile])

pySim/apdu/ts_31_102.py

@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+
+# without this, pylint will fail when inner classes are used
+# within the 'nested' kwarg of our TlvMeta metaclass on python 3.7 :(
+# pylint: disable=undefined-variable
+
+"""
+APDU commands of 3GPP TS 31.102 V16.6.0
+"""
+from typing import Dict
+
+from construct import BitStruct, Enum, BitsInteger, Int8ub, this, Struct, If, Switch, Const
+from construct import Optional as COptional
+from osmocom.construct import *
+
+from pySim.filesystem import *
+from pySim.ts_31_102 import SUCI_TlvDataObject
+from pySim.apdu import ApduCommand, ApduCommandSet
+
+# Copyright (C) 2022 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# Mapping between USIM Service Number and its description
+
+# TS 31.102 Section 7.1
+class UsimAuthenticateEven(ApduCommand, n='AUTHENTICATE', ins=0x88, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_p2 = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1),
+                              BitsInteger(4),
+                              'authentication_context'/Enum(BitsInteger(3), gsm=0, umts=1,
+                                                            vgcs_vbs=2, gba=4))
+    _cs_cmd_gsm_3g =  Struct('_rand_len'/Int8ub, 'rand'/Bytes(this._rand_len),
+                         '_autn_len'/COptional(Int8ub), 'autn'/If(this._autn_len, Bytes(this._autn_len)))
+    _cs_cmd_vgcs = Struct('_vsid_len'/Int8ub, 'vservice_id'/Bytes(this._vsid_len),
+                          '_vkid_len'/Int8ub, 'vk_id'/Bytes(this._vkid_len),
+                          '_vstk_rand_len'/Int8ub, 'vstk_rand'/Bytes(this._vstk_rand_len))
+    _cmd_gba_bs = Struct('_rand_len'/Int8ub, 'rand'/Bytes(this._rand_len),
+                         '_autn_len'/Int8ub, 'autn'/Bytes(this._autn_len))
+    _cmd_gba_naf = Struct('_naf_id_len'/Int8ub, 'naf_id'/Bytes(this._naf_id_len),
+                          '_impi_len'/Int8ub, 'impi'/Bytes(this._impi_len))
+    _cs_cmd_gba = Struct('tag'/Int8ub, 'body'/Switch(this.tag, { 0xDD: 'bootstrap'/_cmd_gba_bs,
+                                                                 0xDE: 'naf_derivation'/_cmd_gba_naf }))
+    _cs_rsp_gsm = Struct('_len_sres'/Int8ub, 'sres'/Bytes(this._len_sres),
+                         '_len_kc'/Int8ub, 'kc'/Bytes(this._len_kc))
+    _rsp_3g_ok = Struct('_len_res'/Int8ub, 'res'/Bytes(this._len_res),
+                        '_len_ck'/Int8ub, 'ck'/Bytes(this._len_ck),
+                        '_len_ik'/Int8ub, 'ik'/Bytes(this._len_ik),
+                        '_len_kc'/COptional(Int8ub), 'kc'/If(this._len_kc, Bytes(this._len_kc)))
+    _rsp_3g_sync = Struct('_len_auts'/Int8ub, 'auts'/Bytes(this._len_auts))
+    _cs_rsp_3g = Struct('tag'/Int8ub, 'body'/Switch(this.tag, { 0xDB: 'success'/_rsp_3g_ok,
+                                                                0xDC: 'sync_fail'/_rsp_3g_sync}))
+    _cs_rsp_vgcs = Struct(Const(b'\xDB'), '_vstk_len'/Int8ub, 'vstk'/Bytes(this._vstk_len))
+    _cs_rsp_gba_naf = Struct(Const(b'\xDB'), '_ks_ext_naf_len'/Int8ub, 'ks_ext_naf'/Bytes(this._ks_ext_naf_len))
+    def _decode_cmd(self) -> Dict:
+        r = {}
+        r['p1'] = parse_construct(self._construct_p1, self.p1.to_bytes(1, 'big'))
+        r['p2'] = parse_construct(self._construct_p2, self.p2.to_bytes(1, 'big'))
+        auth_ctx = r['p2']['authentication_context']
+        if auth_ctx in ['gsm', 'umts']:
+            r['body'] = parse_construct(self._cs_cmd_gsm_3g, self.cmd_data)
+        elif auth_ctx == 'vgcs_vbs':
+            r['body'] = parse_construct(self._cs_cmd_vgcs, self.cmd_data)
+        elif auth_ctx == 'gba':
+            r['body'] = parse_construct(self._cs_cmd_gba, self.cmd_data)
+        else:
+            raise ValueError('Unsupported authentication_context: %s' % auth_ctx)
+        return r
+
+    def _decode_rsp(self) -> Dict:
+        r = {}
+        auth_ctx = self.cmd_dict['p2']['authentication_context']
+        if auth_ctx == 'gsm':
+            r['body'] = parse_construct(self._cs_rsp_gsm, self.rsp_data)
+        elif auth_ctx == 'umts':
+            r['body'] = parse_construct(self._cs_rsp_3g, self.rsp_data)
+        elif auth_ctx == 'vgcs_vbs':
+            r['body'] = parse_construct(self._cs_rsp_vgcs, self.rsp_data)
+        elif auth_ctx == 'gba':
+            if self.cmd_dict['body']['tag'] == 0xDD:
+                r['body'] = parse_construct(self._cs_rsp_3g, self.rsp_data)
+            else:
+                r['body'] = parse_construct(self._cs_rsp_gba_naf, self.rsp_data)
+        else:
+            raise ValueError('Unsupported authentication_context: %s' % auth_ctx)
+        return r
+
+class UsimAuthenticateOdd(ApduCommand, n='AUTHENTICATE', ins=0x89, cla=['0X', '4X', '6X']):
+    _apdu_case = 4
+    _construct_p2 = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1),
+                              BitsInteger(4),
+                              'authentication_context'/Enum(BitsInteger(3), mbms=5, local_key=6))
+# TS 31.102 Section 7.5
+class UsimGetIdentity(ApduCommand, n='GET IDENTITY', ins=0x78, cla=['8X', 'CX', 'EX']):
+    _apdu_case = 4
+    _construct_p2 = BitStruct('scope'/Enum(Flag, mf=0, df_adf_specific=1),
+                              'identity_context'/Enum(BitsInteger(7), suci=1, suci_5g_nswo=2))
+    _tlv_rsp = SUCI_TlvDataObject
+
+ApduCommands = ApduCommandSet('TS 31.102', cmds=[UsimAuthenticateEven, UsimAuthenticateOdd,
+                                                 UsimGetIdentity])

pySim/apdu_source/__init__.py

@@ -0,0 +1,34 @@
+import abc
+import logging
+from typing import Union
+from pySim.apdu import Apdu, Tpdu, CardReset, TpduFilter
+
+PacketType = Union[Apdu, Tpdu, CardReset]
+
+logger = logging.getLogger(__name__)
+
+class ApduSource(abc.ABC):
+    def __init__(self):
+        self.apdu_filter = TpduFilter(None)
+
+    @abc.abstractmethod
+    def read_packet(self) -> PacketType:
+        """Read one packet from the source."""
+
+    def read(self) -> Union[Apdu, CardReset]:
+        """Main function to call by the user: Blocking read, returns Apdu or CardReset."""
+        apdu = None
+        # loop until we actually have an APDU to return
+        while not apdu:
+            r = self.read_packet()
+            if not r:
+                continue
+            if isinstance(r, Tpdu):
+                apdu = self.apdu_filter.input_tpdu(r)
+            elif isinstance(r, Apdu):
+                apdu = r
+            elif isinstance(r, CardReset):
+                apdu = r
+            else:
+                raise ValueError('Unknown read_packet() return %s' % r)
+        return apdu

pySim/apdu_source/gsmtap.py

@@ -0,0 +1,60 @@
+# coding=utf-8
+
+# (C) 2022 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from osmocom.gsmtap import GsmtapReceiver
+
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_102_222 import ApduCommands as UiccAdmApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+
+from . import ApduSource, PacketType, CardReset
+
+ApduCommands = UiccApduCommands + UiccAdmApduCommands + UsimApduCommands + GpApduCommands
+
+class GsmtapApduSource(ApduSource):
+    """ApduSource for handling GSMTAP-SIM messages received via UDP, such as
+    those generated by simtrace2-sniff.  Note that *if* you use IP loopback
+    and localhost addresses (which is the default), you will need to start
+    this source before starting simtrace2-sniff, as otherwise the latter will
+    claim the GSMTAP UDP port.
+    """
+    def __init__(self, bind_ip:str='127.0.0.1', bind_port:int=4729):
+        """Create a UDP socket for receiving GSMTAP-SIM messages.
+        Args:
+            bind_ip: IP address to which the socket should be bound (default: 127.0.0.1)
+            bind_port: UDP port number to which the socket should be bound (default: 4729)
+        """
+        super().__init__()
+        self.gsmtap = GsmtapReceiver(bind_ip, bind_port)
+
+    def read_packet(self) -> PacketType:
+        gsmtap_msg, _addr = self.gsmtap.read_packet()
+        if gsmtap_msg['type'] != 'sim':
+            raise ValueError('Unsupported GSMTAP type %s' % gsmtap_msg['type'])
+        sub_type = gsmtap_msg['sub_type']
+        if sub_type == 'apdu':
+            return ApduCommands.parse_cmd_bytes(gsmtap_msg['body'])
+        if sub_type == 'atr':
+            # card has been reset
+            return CardReset(gsmtap_msg['body'])
+        if sub_type in ['pps_req', 'pps_rsp']:
+            # simply ignore for now
+            pass
+        else:
+            raise ValueError('Unsupported GSMTAP-SIM sub-type %s' % sub_type)

pySim/apdu_source/pyshark_gsmtap.py

@@ -0,0 +1,88 @@
+# coding=utf-8
+
+# (C) 2022 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import logging
+from typing import Tuple
+import pyshark
+from osmocom.gsmtap import GsmtapMessage
+
+from pySim.utils import h2b
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_102_222 import ApduCommands as UiccAdmApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+
+from . import ApduSource, PacketType, CardReset
+
+ApduCommands = UiccApduCommands + UiccAdmApduCommands + UsimApduCommands + GpApduCommands
+
+logger = logging.getLogger(__name__)
+
+class _PysharkGsmtap(ApduSource):
+    """APDU Source [provider] base class for reading GSMTAP SIM APDU via tshark."""
+
+    def __init__(self, pyshark_inst):
+        self.pyshark = pyshark_inst
+        self.bank_id = None
+        self.bank_slot = None
+        self.cmd_tpdu = None
+        super().__init__()
+
+    def read_packet(self) -> PacketType:
+        p = self.pyshark.next()
+        return self._parse_packet(p)
+
+    def _set_or_verify_bank_slot(self, bsl: Tuple[int, int]):
+        """Keep track of the bank:slot to make sure we don't mix traces of multiple cards"""
+        if not self.bank_id:
+            self.bank_id = bsl[0]
+            self.bank_slot = bsl[1]
+        else:
+            if self.bank_id != bsl[0] or self.bank_slot != bsl[1]:
+                raise ValueError('Received data for unexpected B(%u:%u)' % (bsl[0], bsl[1]))
+
+    def _parse_packet(self, p) -> PacketType:
+        udp_layer = p['udp']
+        udp_payload_hex = udp_layer.get_field('payload').replace(':','')
+        gsmtap = GsmtapMessage(h2b(udp_payload_hex))
+        gsmtap_msg = gsmtap.decode()
+        if gsmtap_msg['type'] != 'sim':
+            raise ValueError('Unsupported GSMTAP type %s' % gsmtap_msg['type'])
+        sub_type = gsmtap_msg['sub_type']
+        if sub_type == 'apdu':
+            return ApduCommands.parse_cmd_bytes(gsmtap_msg['body'])
+        if sub_type == 'atr':
+            # card has been reset
+            return CardReset(gsmtap_msg['body'])
+        if sub_type in ['pps_req', 'pps_rsp']:
+            # simply ignore for now
+            pass
+        else:
+            raise ValueError('Unsupported GSMTAP-SIM sub-type %s' % sub_type)
+
+class PysharkGsmtapPcap(_PysharkGsmtap):
+    """APDU Source [provider] class for reading GSMTAP from a PCAP
+    file via pyshark, which in turn uses tshark (part of wireshark).
+    """
+    def __init__(self, pcap_filename):
+        """
+        Args:
+            pcap_filename: File name of the pcap file to be opened
+        """
+        pyshark_inst = pyshark.FileCapture(pcap_filename, display_filter='gsm_sim || iso7816.atr', use_json=True, keep_packets=False)
+        super().__init__(pyshark_inst)

pySim/apdu_source/pyshark_rspro.py

@@ -0,0 +1,158 @@
+# coding=utf-8
+
+# (C) 2022 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import logging
+from typing import Tuple
+import pyshark
+
+from pySim.utils import h2b
+from pySim.apdu import Tpdu
+from . import ApduSource, PacketType, CardReset
+
+logger = logging.getLogger(__name__)
+
+class _PysharkRspro(ApduSource):
+    """APDU Source [provider] base class for reading RSPRO (osmo-remsim) via tshark."""
+
+    def __init__(self, pyshark_inst):
+        self.pyshark = pyshark_inst
+        self.bank_id = None
+        self.bank_slot = None
+        self.cmd_tpdu = None
+        super().__init__()
+
+    @staticmethod
+    def get_bank_slot(bank_slot) -> Tuple[int, int]:
+        """Convert a 'bankSlot_element' field into a tuple of bank_id, slot_nr"""
+        bank_id = bank_slot.get_field('bankId')
+        slot_nr = bank_slot.get_field('slotNr')
+        return int(bank_id), int(slot_nr)
+
+    @staticmethod
+    def get_client_slot(client_slot) -> Tuple[int, int]:
+        """Convert a 'clientSlot_element' field into a tuple of client_id, slot_nr"""
+        client_id = client_slot.get_field('clientId')
+        slot_nr = client_slot.get_field('slotNr')
+        return int(client_id), int(slot_nr)
+
+    @staticmethod
+    def get_pstatus(pstatus) -> Tuple[int, int, int]:
+        """Convert a 'slotPhysStatus_element' field into a tuple of vcc, reset, clk"""
+        vccPresent = int(pstatus.get_field('vccPresent'))
+        resetActive = int(pstatus.get_field('resetActive'))
+        clkActive = int(pstatus.get_field('clkActive'))
+        return vccPresent, resetActive, clkActive
+
+    def read_packet(self) -> PacketType:
+        p = self.pyshark.next()
+        return self._parse_packet(p)
+
+    def _set_or_verify_bank_slot(self, bsl: Tuple[int, int]):
+        """Keep track of the bank:slot to make sure we don't mix traces of multiple cards"""
+        if not self.bank_id:
+            self.bank_id = bsl[0]
+            self.bank_slot = bsl[1]
+        else:
+            if self.bank_id != bsl[0] or self.bank_slot != bsl[1]:
+                raise ValueError('Received data for unexpected B(%u:%u)' % (bsl[0], bsl[1]))
+
+    def _parse_packet(self, p) -> PacketType:
+        rspro_layer = p['rspro']
+        #print("Layer: %s" %  rspro_layer)
+        rspro_element = rspro_layer.get_field('RsproPDU_element')
+        #print("Element: %s" % rspro_element)
+        msg_type = rspro_element.get_field('msg')
+        rspro_msg = rspro_element.get_field('msg_tree')
+        if msg_type == '12': # tpduModemToCard
+            modem2card = rspro_msg.get_field('tpduModemToCard_element')
+            #print(modem2card)
+            client_slot = modem2card.get_field('fromClientSlot_element')
+            csl = self.get_client_slot(client_slot)
+            bank_slot = modem2card.get_field('toBankSlot_element')
+            bsl = self.get_bank_slot(bank_slot)
+            self._set_or_verify_bank_slot(bsl)
+            data = modem2card.get_field('data').replace(':','')
+            logger.debug("C(%u:%u) -> B(%u:%u): %s", csl[0], csl[1], bsl[0], bsl[1], data)
+            # store the CMD portion until the RSP portion arrives later
+            self.cmd_tpdu = h2b(data)
+        elif msg_type == '13': # tpduCardToModem
+            card2modem = rspro_msg.get_field('tpduCardToModem_element')
+            #print(card2modem)
+            client_slot = card2modem.get_field('toClientSlot_element')
+            csl = self.get_client_slot(client_slot)
+            bank_slot = card2modem.get_field('fromBankSlot_element')
+            bsl = self.get_bank_slot(bank_slot)
+            self._set_or_verify_bank_slot(bsl)
+            data = card2modem.get_field('data').replace(':','')
+            logger.debug("C(%u:%u) <- B(%u:%u): %s", csl[0], csl[1], bsl[0], bsl[1], data)
+            rsp_tpdu = h2b(data)
+            if self.cmd_tpdu:
+                # combine this R-TPDU with the C-TPDU we saw earlier
+                r = Tpdu(self.cmd_tpdu, rsp_tpdu)
+                self.cmd_tpdu = False
+                return r
+        elif msg_type == '14': # clientSlotStatus
+            cl_slotstatus = rspro_msg.get_field('clientSlotStatusInd_element')
+            #print(cl_slotstatus)
+            client_slot = cl_slotstatus.get_field('fromClientSlot_element')
+            bank_slot = cl_slotstatus.get_field('toBankSlot_element')
+            slot_pstatus = cl_slotstatus.get_field('slotPhysStatus_element')
+            vccPresent, resetActive, clkActive = self.get_pstatus(slot_pstatus)
+            if vccPresent and clkActive and not resetActive:
+                logger.debug("RESET")
+                #TODO: extract ATR from RSPRO message and use it here
+                return CardReset(None)
+        else:
+            print("Unhandled msg type %s: %s" % (msg_type, rspro_msg))
+
+
+class PysharkRsproPcap(_PysharkRspro):
+    """APDU Source [provider] class for reading RSPRO (osmo-remsim) from a PCAP
+    file via pyshark, which in turn uses tshark (part of wireshark).
+
+    In order to use this, you need a wireshark patched with RSPRO support,
+    such as can be found at https://gitea.osmocom.org/osmocom/wireshark/src/branch/laforge/rspro
+
+    A STANDARD UPSTREAM WIRESHARK *DOES NOT WORK*.
+    """
+    def __init__(self, pcap_filename):
+        """
+        Args:
+            pcap_filename: File name of the pcap file to be opened
+        """
+        pyshark_inst = pyshark.FileCapture(pcap_filename, display_filter='rspro', use_json=True, keep_packets=False)
+        super().__init__(pyshark_inst)
+
+class PysharkRsproLive(_PysharkRspro):
+    """APDU Source [provider] class for reading RSPRO (osmo-remsim) from a live capture
+    via pyshark, which in turn uses tshark (part of wireshark).
+
+    In order to use this, you need a wireshark patched with RSPRO support,
+    such as can be found at https://gitea.osmocom.org/osmocom/wireshark/src/branch/laforge/rspro
+
+    A STANDARD UPSTREAM WIRESHARK *DOES NOT WORK*.
+    """
+    def __init__(self, interface, bpf_filter='tcp port 9999 or tcp port 9998'):
+        """
+        Args:
+            interface: Network interface name to capture packets on (like "eth0")
+            bfp_filter: libpcap capture filter to use
+        """
+        pyshark_inst = pyshark.LiveCapture(interface=interface, display_filter='rspro', bpf_filter=bpf_filter,
+                                           use_json=True)
+        super().__init__(pyshark_inst)

pySim/apdu_source/stdin_hex.py

@@ -0,0 +1,39 @@
+# coding=utf-8
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from pySim.utils import h2b
+
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_102_222 import ApduCommands as UiccAdmApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+
+from . import ApduSource, PacketType, CardReset
+
+ApduCommands = UiccApduCommands + UiccAdmApduCommands + UsimApduCommands + GpApduCommands
+
+class StdinHexApduSource(ApduSource):
+    """ApduSource for reading apdu hex-strings from stdin."""
+
+    def read_packet(self) -> PacketType:
+        while True:
+            command = input("C-APDU >")
+            if len(command) == 0:
+                continue
+            response = '9000'
+            return ApduCommands.parse_cmd_bytes(h2b(command) + h2b(response))

pySim/apdu_source/tca_loader_log.py

@@ -0,0 +1,48 @@
+# coding=utf-8
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from pySim.utils import h2b
+
+from pySim.apdu.ts_102_221 import ApduCommands as UiccApduCommands
+from pySim.apdu.ts_102_222 import ApduCommands as UiccAdmApduCommands
+from pySim.apdu.ts_31_102 import ApduCommands as UsimApduCommands
+from pySim.apdu.global_platform import ApduCommands as GpApduCommands
+
+from . import ApduSource, PacketType, CardReset
+
+ApduCommands = UiccApduCommands + UiccAdmApduCommands + UsimApduCommands + GpApduCommands
+
+class TcaLoaderLogApduSource(ApduSource):
+    """ApduSource for reading log files created by TCALoader."""
+    def __init__(self, filename:str):
+        super().__init__()
+        self.logfile = open(filename, 'r')
+
+    def read_packet(self) -> PacketType:
+        command = None
+        response = None
+        for line in self.logfile:
+            if line.startswith('Command'):
+                command = line.split()[1]
+                print("Command: '%s'" % command)
+                pass
+            elif command and line.startswith('Response'):
+                response = line.split()[1]
+                print("Response: '%s'" % response)
+                return ApduCommands.parse_cmd_bytes(h2b(command) + h2b(response))
+        raise StopIteration

pySim/app.py

@@ -0,0 +1,128 @@
+# (C) 2021-2023 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from typing import Tuple
+
+from pySim.transport import LinkBase
+from pySim.commands import SimCardCommands
+from pySim.filesystem import CardModel, CardApplication
+from pySim.cards import card_detect, SimCardBase, UiccCardBase, CardBase
+from pySim.runtime import RuntimeState
+from pySim.profile import CardProfile
+from pySim.cdma_ruim import CardProfileRUIM
+from pySim.ts_102_221 import CardProfileUICC
+from pySim.utils import all_subclasses
+from pySim.exceptions import SwMatchError
+
+# we need to import this module so that the SysmocomSJA2 sub-class of
+# CardModel is created, which will add the ATR-based matching and
+# calling of SysmocomSJA2.add_files.  See  CardModel.apply_matching_models
+import pySim.sysmocom_sja2
+
+# we need to import these modules so that the various sub-classes of
+# CardProfile are created, which will be used in init_card() to iterate
+# over all known CardProfile sub-classes.
+import pySim.ts_31_102
+import pySim.ts_31_103
+import pySim.ts_31_104
+import pySim.ara_m
+import pySim.global_platform
+import pySim.euicc
+
+def init_card(sl: LinkBase, skip_card_init: bool = False) -> Tuple[RuntimeState, SimCardBase]:
+    """
+    Detect card in reader and setup card profile and runtime state. This
+    function must be called at least once on startup. The card and runtime
+    state object (rs) is required for all pySim-shell commands.
+    """
+
+    # Create command layer
+    scc = SimCardCommands(transport=sl)
+
+    # Wait up to three seconds for a card in reader and try to detect
+    # the card type.
+    print("Waiting for card...")
+    sl.wait_for_card(3)
+
+    # The user may opt to skip all card initialization. In this case only the
+    # most basic card profile is selected. This mode is suitable for blank
+    # cards that need card O/S initialization using APDU scripts first.
+    if skip_card_init:
+        return None, CardBase(scc)
+
+    generic_card = False
+    card = card_detect(scc)
+    if card is None:
+        print("Warning: Could not detect card type - assuming a generic card type...")
+        card = SimCardBase(scc)
+        generic_card = True
+
+    profile = CardProfile.pick(scc)
+    if profile is None:
+        # It is not an unrecoverable error in case profile detection fails. It
+        # just means that pySim was unable to recognize the card profile. This
+        # may happen in particular with unprovisioned cards that do not have
+        # any files on them yet.
+        print("Unsupported card type!")
+        return None, card
+
+    # ETSI TS 102 221, Table 9.3 specifies a default for the PIN key
+    # references, however card manufactures may still decide to pick an
+    # arbitrary key reference. In case we run on a generic card class that is
+    # detected as an UICC, we will pick the key reference that is officially
+    # specified.
+    if generic_card and isinstance(profile, CardProfileUICC):
+        card._adm_chv_num = 0x0A
+
+    print("Info: Card is of type: %s" % str(profile))
+
+    # FIXME: this shouldn't really be here but somewhere else/more generic.
+    # We cannot do it within pySim/profile.py as that would create circular
+    # dependencies between the individual profiles and profile.py.
+    if isinstance(profile, CardProfileUICC):
+        for app_cls in all_subclasses(CardApplication):
+            # skip any intermediary sub-classes such as CardApplicationSD
+            if hasattr(app_cls, '_' + app_cls.__name__ + '__intermediate'):
+                continue
+            profile.add_application(app_cls())
+        # We have chosen SimCard() above, but we now know it actually is an UICC
+        # so it's safe to assume it supports USIM application (which we're adding above).
+        # IF we don't do this, we will have a SimCard but try USIM specific commands like
+        # the update_ust method (see https://osmocom.org/issues/6055)
+        if generic_card:
+            card = UiccCardBase(scc)
+
+    # Create runtime state with card profile
+    rs = RuntimeState(card, profile)
+
+    CardModel.apply_matching_models(scc, rs)
+
+    # inform the transport that we can do context-specific SW interpretation
+    sl.set_sw_interpreter(rs)
+
+    # try to obtain the EID, if any
+    isd_r = rs.mf.applications.get(pySim.euicc.AID_ISD_R.lower(), None)
+    if isd_r:
+        rs.lchan[0].select_file(isd_r)
+        try:
+            rs.identity['EID'] = pySim.euicc.CardApplicationISDR.get_eid(scc)
+        except SwMatchError:
+            # has ISD-R but not a SGP.22/SGP.32 eUICC - maybe SGP.02?
+            pass
+        finally:
+            rs.reset()
+
+    return rs, card

pySim/ara_m.py

@@ -26,27 +26,29 @@ Support for the Secure Element Access Control, specifically the ARA-M inside an
 #
 
 
-from construct import *
+from construct import GreedyString, Struct, Enum, Int8ub, Int16ub
 from construct import Optional as COptional
-from pySim.construct import *
+from osmocom.construct import *
+from osmocom.tlv import *
+from osmocom.utils import Hexstr
 from pySim.filesystem import *
-from pySim.tlv import *
+import pySim.global_platform
 
 # various BER-TLV encoded Data Objects (DOs)
 
 
 class AidRefDO(BER_TLV_IE, tag=0x4f):
-    # SEID v1.1 Table 6-3
+    # GPD_SPE_013 v1.1 Table 6-3
     _construct = HexAdapter(GreedyBytes)
 
 
 class AidRefEmptyDO(BER_TLV_IE, tag=0xc0):
-    # SEID v1.1 Table 6-3
+    # GPD_SPE_013 v1.1 Table 6-3
     pass
 
 
 class DevAppIdRefDO(BER_TLV_IE, tag=0xc1):
-    # SEID v1.1 Table 6-4
+    # GPD_SPE_013 v1.1 Table 6-4
     _construct = HexAdapter(GreedyBytes)
 
 
@@ -56,41 +58,39 @@ class PkgRefDO(BER_TLV_IE, tag=0xca):
 
 
 class RefDO(BER_TLV_IE, tag=0xe1, nested=[AidRefDO, AidRefEmptyDO, DevAppIdRefDO, PkgRefDO]):
-    # SEID v1.1 Table 6-5
+    # GPD_SPE_013 v1.1 Table 6-5
     pass
 
 
 class ApduArDO(BER_TLV_IE, tag=0xd0):
-    # SEID v1.1 Table 6-8
+    # GPD_SPE_013 v1.1 Table 6-8
     def _from_bytes(self, do: bytes):
         if len(do) == 1:
             if do[0] == 0x00:
                 self.decoded = {'generic_access_rule': 'never'}
                 return self.decoded
-            elif do[0] == 0x01:
+            if do[0] == 0x01:
                 self.decoded = {'generic_access_rule': 'always'}
                 return self.decoded
-            else:
-                return ValueError('Invalid 1-byte generic APDU access rule')
+            return ValueError('Invalid 1-byte generic APDU access rule')
         else:
             if len(do) % 8:
                 return ValueError('Invalid non-modulo-8 length of APDU filter: %d' % len(do))
-            self.decoded['apdu_filter'] = []
+            self.decoded = {'apdu_filter': []}
             offset = 0
             while offset < len(do):
-                self.decoded['apdu_filter'] += {'header': b2h(do[offset:offset+4]),
-                                                'mask': b2h(do[offset+4:offset+8])}
-            self.decoded = res
-            return res
+                self.decoded['apdu_filter'] += [{'header': b2h(do[offset:offset+4]),
+                                                'mask': b2h(do[offset+4:offset+8])}]
+                offset += 8 # Move offset to the beginning of the next apdu_filter object
+            return self.decoded
 
     def _to_bytes(self):
         if 'generic_access_rule' in self.decoded:
             if self.decoded['generic_access_rule'] == 'never':
                 return b'\x00'
-            elif self.decoded['generic_access_rule'] == 'always':
+            if self.decoded['generic_access_rule'] == 'always':
                 return b'\x01'
-            else:
-                return ValueError('Invalid 1-byte generic APDU access rule')
+            return ValueError('Invalid 1-byte generic APDU access rule')
         else:
             if not 'apdu_filter' in self.decoded:
                 return ValueError('Invalid APDU AR DO')
@@ -108,135 +108,134 @@ class ApduArDO(BER_TLV_IE, tag=0xd0):
 
 
 class NfcArDO(BER_TLV_IE, tag=0xd1):
-    # SEID v1.1 Table 6-9
+    # GPD_SPE_013 v1.1 Table 6-9
     _construct = Struct('nfc_event_access_rule' /
                         Enum(Int8ub, never=0, always=1))
 
 
 class PermArDO(BER_TLV_IE, tag=0xdb):
     # Android UICC Carrier Privileges specific extension, see https://source.android.com/devices/tech/config/uicc
+    # based on Table 6-8 of GlobalPlatform Device API Access Control v1.0
     _construct = Struct('permissions'/HexAdapter(Bytes(8)))
 
 
 class ArDO(BER_TLV_IE, tag=0xe3, nested=[ApduArDO, NfcArDO, PermArDO]):
-    # SEID v1.1 Table 6-7
+    # GPD_SPE_013 v1.1 Table 6-7
     pass
 
 
 class RefArDO(BER_TLV_IE, tag=0xe2, nested=[RefDO, ArDO]):
-    # SEID v1.1 Table 6-6
+    # GPD_SPE_013 v1.1 Table 6-6
     pass
 
 
 class ResponseAllRefArDO(BER_TLV_IE, tag=0xff40, nested=[RefArDO]):
-    # SEID v1.1 Table 4-2
+    # GPD_SPE_013 v1.1 Table 4-2
     pass
 
 
 class ResponseArDO(BER_TLV_IE, tag=0xff50, nested=[ArDO]):
-    # SEID v1.1 Table 4-3
+    # GPD_SPE_013 v1.1 Table 4-3
     pass
 
 
 class ResponseRefreshTagDO(BER_TLV_IE, tag=0xdf20):
-    # SEID v1.1 Table 4-4
+    # GPD_SPE_013 v1.1 Table 4-4
     _construct = Struct('refresh_tag'/HexAdapter(Bytes(8)))
 
 
 class DeviceInterfaceVersionDO(BER_TLV_IE, tag=0xe6):
-    # SEID v1.1 Table 6-12
+    # GPD_SPE_013 v1.1 Table 6-12
     _construct = Struct('major'/Int8ub, 'minor'/Int8ub, 'patch'/Int8ub)
 
 
 class DeviceConfigDO(BER_TLV_IE, tag=0xe4, nested=[DeviceInterfaceVersionDO]):
-    # SEID v1.1 Table 6-10
+    # GPD_SPE_013 v1.1 Table 6-10
     pass
 
 
 class ResponseDeviceConfigDO(BER_TLV_IE, tag=0xff7f, nested=[DeviceConfigDO]):
-    # SEID v1.1 Table 5-14
+    # GPD_SPE_013 v1.1 Table 5-14
     pass
 
 
 class AramConfigDO(BER_TLV_IE, tag=0xe5, nested=[DeviceInterfaceVersionDO]):
-    # SEID v1.1 Table 6-11
+    # GPD_SPE_013 v1.1 Table 6-11
     pass
 
 
 class ResponseAramConfigDO(BER_TLV_IE, tag=0xdf21, nested=[AramConfigDO]):
-    # SEID v1.1 Table 4-5
+    # GPD_SPE_013 v1.1 Table 4-5
     pass
 
 
 class CommandStoreRefArDO(BER_TLV_IE, tag=0xf0, nested=[RefArDO]):
-    # SEID v1.1 Table 5-2
+    # GPD_SPE_013 v1.1 Table 5-2
     pass
 
 
 class CommandDelete(BER_TLV_IE, tag=0xf1, nested=[AidRefDO, AidRefEmptyDO, RefDO, RefArDO]):
-    # SEID v1.1 Table 5-4
+    # GPD_SPE_013 v1.1 Table 5-4
     pass
 
 
 class CommandUpdateRefreshTagDO(BER_TLV_IE, tag=0xf2):
-    # SEID V1.1 Table 5-6
+    # GPD_SPE_013 V1.1 Table 5-6
     pass
 
 
 class CommandRegisterClientAidsDO(BER_TLV_IE, tag=0xf7, nested=[AidRefDO, AidRefEmptyDO]):
-    # SEID v1.1 Table 5-7
+    # GPD_SPE_013 v1.1 Table 5-7
     pass
 
 
 class CommandGet(BER_TLV_IE, tag=0xf3, nested=[AidRefDO, AidRefEmptyDO]):
-    # SEID v1.1 Table 5-8
+    # GPD_SPE_013 v1.1 Table 5-8
     pass
 
 
 class CommandGetAll(BER_TLV_IE, tag=0xf4):
-    # SEID v1.1 Table 5-9
+    # GPD_SPE_013 v1.1 Table 5-9
     pass
 
 
 class CommandGetClientAidsDO(BER_TLV_IE, tag=0xf6):
-    # SEID v1.1 Table 5-10
+    # GPD_SPE_013 v1.1 Table 5-10
     pass
 
 
 class CommandGetNext(BER_TLV_IE, tag=0xf5):
-    # SEID v1.1 Table 5-11
+    # GPD_SPE_013 v1.1 Table 5-11
     pass
 
 
 class CommandGetDeviceConfigDO(BER_TLV_IE, tag=0xf8):
-    # SEID v1.1 Table 5-12
+    # GPD_SPE_013 v1.1 Table 5-12
     pass
 
 
 class ResponseAracAidDO(BER_TLV_IE, tag=0xff70, nested=[AidRefDO, AidRefEmptyDO]):
-    # SEID v1.1 Table 5-13
+    # GPD_SPE_013 v1.1 Table 5-13
     pass
 
 
 class BlockDO(BER_TLV_IE, tag=0xe7):
-    # SEID v1.1 Table 6-13
+    # GPD_SPE_013 v1.1 Table 6-13
     _construct = Struct('offset'/Int16ub, 'length'/Int8ub)
 
 
-# SEID v1.1 Table 4-1
+# GPD_SPE_013 v1.1 Table 4-1
 class GetCommandDoCollection(TLV_IE_Collection, nested=[RefDO, DeviceConfigDO]):
     pass
 
-# SEID v1.1 Table 4-2
-
 
+# GPD_SPE_013 v1.1 Table 4-2
 class GetResponseDoCollection(TLV_IE_Collection, nested=[ResponseAllRefArDO, ResponseArDO,
                                                          ResponseRefreshTagDO, ResponseAramConfigDO]):
     pass
 
-# SEID v1.1 Table 5-1
-
 
+# GPD_SPE_013 v1.1 Table 5-1
 class StoreCommandDoCollection(TLV_IE_Collection,
                                nested=[BlockDO, CommandStoreRefArDO, CommandDelete,
                                        CommandUpdateRefreshTagDO, CommandRegisterClientAidsDO,
@@ -245,7 +244,7 @@ class StoreCommandDoCollection(TLV_IE_Collection,
     pass
 
 
-# SEID v1.1 Section 5.1.2
+# GPD_SPE_013 v1.1 Section 5.1.2
 class StoreResponseDoCollection(TLV_IE_Collection,
                                 nested=[ResponseAllRefArDO, ResponseAracAidDO, ResponseDeviceConfigDO]):
     pass
@@ -259,8 +258,11 @@ class ADF_ARAM(CardADF):
         files = []
         self.add_files(files)
 
+    def decode_select_response(self, data_hex):
+        return pySim.global_platform.decode_select_response(data_hex)
+
     @staticmethod
-    def xceive_apdu_tlv(tp, hdr: Hexstr, cmd_do, resp_cls, exp_sw='9000'):
+    def xceive_apdu_tlv(scc, hdr: Hexstr, cmd_do, resp_cls, exp_sw='9000'):
         """Transceive an APDU with the card, transparently encoding the command data from TLV
         and decoding the response data tlv."""
         if cmd_do:
@@ -272,59 +274,55 @@ class ADF_ARAM(CardADF):
             cmd_do_enc = b''
             cmd_do_len = 0
         c_apdu = hdr + ('%02x' % cmd_do_len) + b2h(cmd_do_enc)
-        (data, sw) = tp.send_apdu_checksw(c_apdu, exp_sw)
+        (data, _sw) = scc.send_apdu_checksw(c_apdu, exp_sw)
         if data:
             if resp_cls:
                 resp_do = resp_cls()
                 resp_do.from_tlv(h2b(data))
                 return resp_do
-            else:
-                return data
+            return data
         else:
             return None
 
     @staticmethod
-    def store_data(tp, do) -> bytes:
+    def store_data(scc, do) -> bytes:
         """Build the Command APDU for STORE DATA."""
-        return ADF_ARAM.xceive_apdu_tlv(tp, '80e29000', do, StoreResponseDoCollection)
+        return ADF_ARAM.xceive_apdu_tlv(scc, '80e29000', do, StoreResponseDoCollection)
 
     @staticmethod
-    def get_all(tp):
-        return ADF_ARAM.xceive_apdu_tlv(tp, '80caff40', None, GetResponseDoCollection)
+    def get_all(scc):
+        return ADF_ARAM.xceive_apdu_tlv(scc, '80caff40', None, GetResponseDoCollection)
 
     @staticmethod
-    def get_config(tp, v_major=0, v_minor=0, v_patch=1):
+    def get_config(scc, v_major=0, v_minor=0, v_patch=1):
         cmd_do = DeviceConfigDO()
-        cmd_do.from_dict([{'DeviceInterfaceVersionDO': {
-                         'major': v_major, 'minor': v_minor, 'patch': v_patch}}])
-        return ADF_ARAM.xceive_apdu_tlv(tp, '80cadf21', cmd_do, ResponseAramConfigDO)
+        cmd_do.from_val_dict([{'device_interface_version_do': {
+                               'major': v_major, 'minor': v_minor, 'patch': v_patch}}])
+        return ADF_ARAM.xceive_apdu_tlv(scc, '80cadf21', cmd_do, ResponseAramConfigDO)
 
     @with_default_category('Application-Specific Commands')
     class AddlShellCommands(CommandSet):
-        def __init(self):
-            super().__init__()
-
-        def do_aram_get_all(self, opts):
+        def do_aram_get_all(self, _opts):
             """GET DATA [All] on the ARA-M Applet"""
-            res_do = ADF_ARAM.get_all(self._cmd.card._scc._tp)
+            res_do = ADF_ARAM.get_all(self._cmd.lchan.scc)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
-        def do_aram_get_config(self, opts):
+        def do_aram_get_config(self, _opts):
             """Perform GET DATA [Config] on the ARA-M Applet: Tell it our version and retrieve its version."""
-            res_do = ADF_ARAM.get_config(self._cmd.card._scc._tp)
+            res_do = ADF_ARAM.get_config(self._cmd.lchan.scc)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
         store_ref_ar_do_parse = argparse.ArgumentParser()
         # REF-DO
         store_ref_ar_do_parse.add_argument(
-            '--device-app-id', required=True, help='Identifies the specific device application that the rule appplies to. Hash of Certificate of Application Provider, or UUID. (20/32 hex bytes)')
+            '--device-app-id', required=True, help='Identifies the specific device application that the rule applies to. Hash of Certificate of Application Provider, or UUID. (20/32 hex bytes)')
         aid_grp = store_ref_ar_do_parse.add_mutually_exclusive_group()
         aid_grp.add_argument(
-            '--aid', help='Identifies the specific SE application for which rules are to be stored. Can be a partial AID, containing for example only the RID.  (5-16 hex bytes)')
+            '--aid', help='Identifies the specific SE application for which rules are to be stored. Can be a partial AID, containing for example only the RID. (5-16 or 0 hex bytes)')
         aid_grp.add_argument('--aid-empty', action='store_true',
-                             help='No specific SE application, applies to all applications')
+                             help='No specific SE application, applies to implicitly selected application (all channels)')
         store_ref_ar_do_parse.add_argument(
             '--pkg-ref', help='Full Android Java package name (up to 127 chars ASCII)')
         # AR-DO
@@ -334,7 +332,7 @@ class ADF_ARAM(CardADF):
         apdu_grp.add_argument(
             '--apdu-always', action='store_true', help='APDU access is allowed')
         apdu_grp.add_argument(
-            '--apdu-filter', help='APDU filter: 4 byte CLA/INS/P1/P2 followed by 4 byte mask (8 hex bytes)')
+            '--apdu-filter', help='APDU filter: multiple groups of 8 hex bytes (4 byte CLA/INS/P1/P2 followed by 4 byte mask)')
         nfc_grp = store_ref_ar_do_parse.add_mutually_exclusive_group()
         nfc_grp.add_argument('--nfc-always', action='store_true',
                              help='NFC event access is allowed')
@@ -348,48 +346,60 @@ class ADF_ARAM(CardADF):
             """Perform STORE DATA [Command-Store-REF-AR-DO] to store a (new) access rule."""
             # REF
             ref_do_content = []
-            if opts.aid:
-                ref_do_content += [{'AidRefDO': opts.aid}]
+            if opts.aid is not None:
+                ref_do_content += [{'aid_ref_do': opts.aid}]
             elif opts.aid_empty:
-                ref_do_content += [{'AidRefEmptyDO': None}]
-            ref_do_content += [{'DevAppIdRefDO': opts.device_app_id}]
+                ref_do_content += [{'aid_ref_empty_do': None}]
+            ref_do_content += [{'dev_app_id_ref_do': opts.device_app_id}]
             if opts.pkg_ref:
-                ref_do_content += [{'PkgRefDO': opts.pkg_ref}]
+                ref_do_content += [{'pkg_ref_do': {'package_name_string': opts.pkg_ref}}]
             # AR
             ar_do_content = []
             if opts.apdu_never:
-                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'never'}}]
+                ar_do_content += [{'apdu_ar_do': {'generic_access_rule': 'never'}}]
             elif opts.apdu_always:
-                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'always'}}]
+                ar_do_content += [{'apdu_ar_do': {'generic_access_rule': 'always'}}]
             elif opts.apdu_filter:
-                # TODO: multiple filters
-                ar_do_content += [{'ApduArDO': {'apdu_filter': [opts.apdu_filter]}}]
+                if len(opts.apdu_filter) % 16:
+                    return ValueError('Invalid non-modulo-16 length of APDU filter: %d' % len(do))
+                offset = 0
+                apdu_filter = []
+                while offset < len(opts.apdu_filter):
+                    apdu_filter += [{'header': opts.apdu_filter[offset:offset+8],
+                                     'mask': opts.apdu_filter[offset+8:offset+16]}]
+                    offset += 16 # Move offset to the beginning of the next apdu_filter object
+                ar_do_content += [{'apdu_ar_do': {'apdu_filter': apdu_filter}}]
             if opts.nfc_always:
-                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'always'}}]
+                ar_do_content += [{'nfc_ar_do': {'nfc_event_access_rule': 'always'}}]
             elif opts.nfc_never:
-                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'never'}}]
+                ar_do_content += [{'nfc_ar_do': {'nfc_event_access_rule': 'never'}}]
             if opts.android_permissions:
-                ar_do_content += [{'PermArDO': {'permissions': opts.android_permissions}}]
-            d = [{'RefArDO': [{'RefDO': ref_do_content}, {'ArDO': ar_do_content}]}]
+                ar_do_content += [{'perm_ar_do': {'permissions': opts.android_permissions}}]
+            d = [{'ref_ar_do': [{'ref_do': ref_do_content}, {'ar_do': ar_do_content}]}]
             csrado = CommandStoreRefArDO()
-            csrado.from_dict(d)
-            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, csrado)
+            csrado.from_val_dict(d)
+            res_do = ADF_ARAM.store_data(self._cmd.lchan.scc, csrado)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
-        def do_aram_delete_all(self, opts):
+        def do_aram_delete_all(self, _opts):
             """Perform STORE DATA [Command-Delete[all]] to delete all access rules."""
             deldo = CommandDelete()
-            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, deldo)
+            res_do = ADF_ARAM.store_data(self._cmd.lchan.scc, deldo)
             if res_do:
                 self._cmd.poutput_json(res_do.to_dict())
 
+        def do_aram_lock(self, opts):
+            """Lock STORE DATA command to prevent unauthorized changes
+            (Proprietary feature that is specific to sysmocom's fork of Bertrand Martel’s ARA-M implementation.)"""
+            self._cmd.lchan.scc.send_apdu_checksw('80e2900001A1', '9000')
+
 
 # SEAC v1.1 Section 4.1.2.2 + 5.1.2.2
 sw_aram = {
     'ARA-M': {
         '6381': 'Rule successfully stored but an access rule already exists',
-        '6382': 'Rule successfully stored bu contained at least one unknown (discarded) BER-TLV',
+        '6382': 'Rule successfully stored but contained at least one unknown (discarded) BER-TLV',
         '6581': 'Memory Problem',
         '6700': 'Wrong Length in Lc',
         '6981': 'DO is not supported by the ARA-M/ARA-C',
@@ -410,3 +420,84 @@ sw_aram = {
 class CardApplicationARAM(CardApplication):
     def __init__(self):
         super().__init__('ARA-M', adf=ADF_ARAM(), sw=sw_aram)
+
+    @staticmethod
+    def __export_get_from_dictlist(key, dictlist):
+        # Data objects are organized in lists that contain dictionaries, usually there is only one dictionary per
+        # list item. This function goes through that list and gets the value of the first dictionary that has the
+        # matching key.
+        if dictlist is None:
+            return None
+        for d in dictlist:
+            if key in d:
+                obj = d.get(key)
+                if obj is None:
+                    return ""
+                return obj
+        return None
+
+    @staticmethod
+    def __export_ref_ar_do_list(ref_ar_do_list):
+        export_str = ""
+        ref_do_list = CardApplicationARAM.__export_get_from_dictlist('ref_do', ref_ar_do_list.get('ref_ar_do'))
+        ar_do_list = CardApplicationARAM.__export_get_from_dictlist('ar_do', ref_ar_do_list.get('ref_ar_do'))
+
+        if ref_do_list and ar_do_list:
+            # Get ref_do parameters
+            aid_ref_do = CardApplicationARAM.__export_get_from_dictlist('aid_ref_do', ref_do_list)
+            aid_ref_empty_do = CardApplicationARAM.__export_get_from_dictlist('aid_ref_empty_do', ref_do_list)
+            dev_app_id_ref_do = CardApplicationARAM.__export_get_from_dictlist('dev_app_id_ref_do', ref_do_list)
+            pkg_ref_do = CardApplicationARAM.__export_get_from_dictlist('pkg_ref_do', ref_do_list)
+
+            # Get ar_do parameters
+            apdu_ar_do = CardApplicationARAM.__export_get_from_dictlist('apdu_ar_do', ar_do_list)
+            nfc_ar_do = CardApplicationARAM.__export_get_from_dictlist('nfc_ar_do', ar_do_list)
+            perm_ar_do = CardApplicationARAM.__export_get_from_dictlist('perm_ar_do', ar_do_list)
+
+            # Write command-line
+            export_str += "aram_store_ref_ar_do"
+            if aid_ref_do is not None and len(aid_ref_do) > 0:
+                export_str += (" --aid %s" % aid_ref_do)
+            elif aid_ref_do is not None:
+                export_str += " --aid \"\""
+            if aid_ref_empty_do is not None:
+                export_str += " --aid-empty"
+            if dev_app_id_ref_do:
+                export_str += (" --device-app-id %s" % dev_app_id_ref_do)
+            if apdu_ar_do and 'generic_access_rule' in apdu_ar_do:
+                export_str += (" --apdu-%s" % apdu_ar_do['generic_access_rule'])
+            elif apdu_ar_do and 'apdu_filter' in apdu_ar_do:
+                export_str += (" --apdu-filter ")
+                for apdu_filter in apdu_ar_do['apdu_filter']:
+                    export_str += apdu_filter['header']
+                    export_str += apdu_filter['mask']
+            if nfc_ar_do and 'nfc_event_access_rule' in nfc_ar_do:
+                export_str += (" --nfc-%s" % nfc_ar_do['nfc_event_access_rule'])
+            if perm_ar_do:
+                export_str += (" --android-permissions %s" % perm_ar_do['permissions'])
+            if pkg_ref_do:
+                export_str += (" --pkg-ref %s" % pkg_ref_do['package_name_string'])
+            export_str += "\n"
+        return export_str
+
+    @staticmethod
+    def export(as_json: bool, lchan):
+
+        # TODO: Add JSON output as soon as aram_store_ref_ar_do is able to process input in JSON format.
+        if as_json:
+            raise NotImplementedError("res_do encoder not yet implemented. Patches welcome.")
+
+        export_str = ""
+        export_str += "aram_delete_all\n"
+
+        res_do = ADF_ARAM.get_all(lchan.scc)
+        if not res_do:
+            return export_str.strip()
+
+        for res_do_dict in res_do.to_dict():
+            if not res_do_dict.get('response_all_ref_ar_do', False):
+                continue
+            for ref_ar_do_list in res_do_dict['response_all_ref_ar_do']:
+                export_str += CardApplicationARAM.__export_ref_ar_do_list(ref_ar_do_list)
+
+        return export_str.strip()

pySim/card_handler.py

@@ -7,7 +7,7 @@ there are also automatic card feeders.
 """
 
 #
-# (C) 2019 by Sysmocom s.f.m.c. GmbH
+# (C) 2019 by sysmocom - s.f.m.c. GmbH
 # All Rights Reserved
 #
 # This program is free software: you can redistribute it and/or modify
@@ -24,12 +24,11 @@ there are also automatic card feeders.
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from pySim.transport import LinkBase
-
 import subprocess
 import sys
 import yaml
 
+from pySim.transport import LinkBase
 
 class CardHandlerBase:
     """Abstract base class representing a mechanism for card insertion/removal."""
@@ -97,7 +96,7 @@ class CardHandlerAuto(CardHandlerBase):
         print("Card handler Config-file: " + str(config_file))
         with open(config_file) as cfg:
             self.cmds = yaml.load(cfg, Loader=yaml.FullLoader)
-        self.verbose = (self.cmds.get('verbose') == True)
+        self.verbose = self.cmds.get('verbose') is True
 
     def __print_outout(self, out):
         print("")

pySim/card_key_provider.py

@@ -10,10 +10,10 @@ the need of manually entering the related card-individual data on every
 operation with pySim-shell.
 """
 
-# (C) 2021 by Sysmocom s.f.m.c. GmbH
+# (C) 2021-2025 by sysmocom - s.f.m.c. GmbH
 # All Rights Reserved
 #
-# Author: Philipp Maier
+# Author: Philipp Maier, Harald Welte
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -29,95 +29,228 @@ operation with pySim-shell.
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from typing import List, Dict, Optional
+from Cryptodome.Cipher import AES
+from osmocom.utils import h2b, b2h
+from pySim.log import PySimLogger
 
 import abc
 import csv
+import logging
+import yaml
+
+log = PySimLogger.get(__name__)
 
 card_key_providers = []  # type: List['CardKeyProvider']
 
+class CardKeyFieldCryptor:
+    """
+    A Card key field encryption class that may be used by Card key provider implementations to add support for
+    a column-based encryption to protect sensitive material (cryptographic key material, ADM keys, etc.).
+    The sensitive material is encrypted using a "key-encryption key", occasionally also known as "transport key"
+    before it is stored into a file or database (see also GSMA FS.28). The "transport key" is then used to decrypt
+    the key material on demand.
+    """
+
+    # well-known groups of columns relate to a given functionality.  This avoids having
+    # to specify the same transport key N number of times, if the same key is used for multiple
+    # fields of one group, like KIC+KID+KID of one SD.
+    __CRYPT_GROUPS = {
+            'UICC_SCP02': ['UICC_SCP02_KIC1', 'UICC_SCP02_KID1', 'UICC_SCP02_KIK1'],
+            'UICC_SCP03': ['UICC_SCP03_KIC1', 'UICC_SCP03_KID1', 'UICC_SCP03_KIK1'],
+            'SCP03_ISDR': ['SCP03_ENC_ISDR', 'SCP03_MAC_ISDR', 'SCP03_DEK_ISDR'],
+            'SCP03_ISDA': ['SCP03_ENC_ISDA', 'SCP03_MAC_ISDA', 'SCP03_DEK_ISDA'],
+            'SCP03_ECASD': ['SCP03_ENC_ECASD', 'SCP03_MAC_ECASD', 'SCP03_DEK_ECASD'],
+    }
+
+    __IV = b'\x23' * 16
+
+    @staticmethod
+    def __dict_keys_to_upper(d: dict) -> dict:
+            return {k.upper():v for k,v in d.items()}
+
+    @staticmethod
+    def __process_transport_keys(transport_keys: dict, crypt_groups: dict):
+        """Apply a single transport key to multiple fields/columns, if the name is a group."""
+        new_dict = {}
+        for name, key in transport_keys.items():
+            if name in crypt_groups:
+                for field in crypt_groups[name]:
+                    new_dict[field] = key
+            else:
+                new_dict[name] = key
+        return new_dict
+
+    def __init__(self, transport_keys: dict):
+        """
+        Create new field encryptor/decryptor object and set transport keys, usually one for each column. In some cases
+        it is also possible to use a single key for multiple columns (see also __CRYPT_GROUPS)
+
+        Args:
+                transport_keys : a dict indexed by field name, whose values are hex-encoded AES keys for the
+                                 respective field (column) of the CSV. This is done so that different fields
+                                 (columns) can use different transport keys, which is strongly recommended by
+                                 GSMA FS.28
+        """
+        self.transport_keys = self.__process_transport_keys(self.__dict_keys_to_upper(transport_keys),
+                                                            self.__CRYPT_GROUPS)
+        for name, key in self.transport_keys.items():
+                log.debug("Encrypting/decrypting field %s using AES key %s" % (name, key))
+
+    def decrypt_field(self, field_name: str, encrypted_val: str) -> str:
+        """
+        Decrypt a single field. The decryption is only applied if we have a transport key is known under the provided
+        field name, otherwise the field is treated as plaintext and passed through as it is.
+
+        Args:
+                field_name : name of the field to decrypt (used to identify which key to use)
+                encrypted_val : encrypted field value
+
+        Returns:
+                plaintext field value
+        """
+        if not field_name.upper() in self.transport_keys:
+            return encrypted_val
+        cipher = AES.new(h2b(self.transport_keys[field_name.upper()]), AES.MODE_CBC, self.__IV)
+        return b2h(cipher.decrypt(h2b(encrypted_val)))
+
+    def encrypt_field(self, field_name: str, plaintext_val: str) -> str:
+        """
+        Encrypt a single field. The encryption is only applied if we have a transport key is known under the provided
+        field name, otherwise the field is treated as non sensitive and passed through as it is.
+
+        Args:
+                field_name : name of the field to decrypt (used to identify which key to use)
+                encrypted_val : encrypted field value
+
+        Returns:
+                plaintext field value
+        """
+        if not field_name.upper() in self.transport_keys:
+            return plaintext_val
+        cipher = AES.new(h2b(self.transport_keys[field_name.upper()]), AES.MODE_CBC, self.__IV)
+        return b2h(cipher.encrypt(h2b(plaintext_val)))
 
 class CardKeyProvider(abc.ABC):
     """Base class, not containing any concrete implementation."""
 
-    VALID_FIELD_NAMES = ['ICCID', 'ADM1',
-                         'IMSI', 'PIN1', 'PIN2', 'PUK1', 'PUK2']
-
-    # check input parameters, but do nothing concrete yet
-    def _verify_get_data(self, fields: List[str] = [], key: str = 'ICCID', value: str = "") -> Dict[str, str]:
-        """Verify multiple fields for identified card.
-
-        Args:
-                fields : list of valid field names such as 'ADM1', 'PIN1', ... which are to be obtained
-                key : look-up key to identify card data, such as 'ICCID'
-                value : value for look-up key to identify card data
-        Returns:
-                dictionary of {field, value} strings for each requested field from 'fields'
-        """
-        for f in fields:
-            if (f not in self.VALID_FIELD_NAMES):
-                raise ValueError("Requested field name '%s' is not a valid field name, valid field names are: %s" %
-                                 (f, str(self.VALID_FIELD_NAMES)))
-
-        if (key not in self.VALID_FIELD_NAMES):
-            raise ValueError("Key field name '%s' is not a valid field name, valid field names are: %s" %
-                             (key, str(self.VALID_FIELD_NAMES)))
-
-        return {}
-
-    def get_field(self, field: str, key: str = 'ICCID', value: str = "") -> Optional[str]:
-        """get a single field from CSV file using a specified key/value pair"""
-        fields = [field]
-        result = self.get(fields, key, value)
-        return result.get(field)
-
     @abc.abstractmethod
     def get(self, fields: List[str], key: str, value: str) -> Dict[str, str]:
-        """Get multiple card-individual fields for identified card.
+        """
+        Get multiple card-individual fields for identified card. This method should not fail with an exception in
+        case the entry, columns or even the key column itsself is not found.
 
         Args:
                 fields : list of valid field names such as 'ADM1', 'PIN1', ... which are to be obtained
                 key : look-up key to identify card data, such as 'ICCID'
                 value : value for look-up key to identify card data
         Returns:
-                dictionary of {field, value} strings for each requested field from 'fields'
+                dictionary of {field : value, ...} strings for each requested field from 'fields'. In case nothing is
+                fond None shall be returned.
         """
 
+    def __str__(self):
+        return type(self).__name__
 
 class CardKeyProviderCsv(CardKeyProvider):
-    """Card key provider implementation that allows to query against a specified CSV file"""
-    csv_file = None
-    filename = None
+    """Card key provider implementation that allows to query against a specified CSV file."""
 
-    def __init__(self, filename: str):
+    def __init__(self, csv_filename: str, transport_keys: dict):
         """
         Args:
-                filename : file name (path) of CSV file containing card-individual key/data
+                csv_filename : file name (path) of CSV file containing card-individual key/data
+                transport_keys : (see class CardKeyFieldCryptor)
         """
-        self.csv_file = open(filename, 'r')
+        log.info("Using CSV file as card key data source: %s" % csv_filename)
+        self.csv_file = open(csv_filename, 'r')
         if not self.csv_file:
-            raise RuntimeError("Could not open CSV file '%s'" % filename)
-        self.filename = filename
+            raise RuntimeError("Could not open CSV file '%s'" % csv_filename)
+        self.csv_filename = csv_filename
+        self.crypt = CardKeyFieldCryptor(transport_keys)
 
     def get(self, fields: List[str], key: str, value: str) -> Dict[str, str]:
-        super()._verify_get_data(fields, key, value)
-
         self.csv_file.seek(0)
         cr = csv.DictReader(self.csv_file)
         if not cr:
-            raise RuntimeError(
-                "Could not open DictReader for CSV-File '%s'" % self.filename)
+            raise RuntimeError("Could not open DictReader for CSV-File '%s'" % self.csv_filename)
         cr.fieldnames = [field.upper() for field in cr.fieldnames]
 
-        rc = {}
+        if key not in cr.fieldnames:
+            return None
+        return_dict = {}
         for row in cr:
             if row[key] == value:
                 for f in fields:
                     if f in row:
-                        rc.update({f: row[f]})
+                        return_dict.update({f: self.crypt.decrypt_field(f, row[f])})
                     else:
-                        raise RuntimeError("CSV-File '%s' lacks column '%s'" %
-                                           (self.filename, f))
-        return rc
+                        raise RuntimeError("CSV-File '%s' lacks column '%s'" % (self.csv_filename, f))
+        if return_dict == {}:
+            return None
+        return return_dict
+
+class CardKeyProviderPgsql(CardKeyProvider):
+    """Card key provider implementation that allows to query against a specified PostgreSQL database table."""
+
+    def __init__(self, config_filename: str, transport_keys: dict):
+        """
+        Args:
+                config_filename : file name (path) of CSV file containing card-individual key/data
+                transport_keys : (see class CardKeyFieldCryptor)
+        """
+        import psycopg2
+        log.info("Using SQL database as card key data source: %s" % config_filename)
+        with open(config_filename, "r") as cfg:
+            config = yaml.load(cfg, Loader=yaml.FullLoader)
+            log.info("Card key database name: %s" % config.get('db_name'))
+            db_users = config.get('db_users')
+            user = db_users.get('reader')
+            if user is None:
+                raise ValueError("user for role 'reader' not set up in config file.")
+            self.conn = psycopg2.connect(dbname=config.get('db_name'),
+                                         user=user.get('name'),
+                                         password=user.get('pass'),
+                                         host=config.get('host'))
+            self.tables = config.get('table_names')
+            log.info("Card key database tables: %s" % str(self.tables))
+            self.crypt = CardKeyFieldCryptor(transport_keys)
+
+    def get(self, fields: List[str], key: str, value: str) -> Dict[str, str]:
+        import psycopg2
+        from psycopg2.sql import Identifier, SQL
+        db_result = None
+        for t in self.tables:
+            self.conn.rollback()
+            cur = self.conn.cursor()
+
+            # Make sure that the database table and the key column actually exists. If not, move on to the next table
+            cur.execute("SELECT column_name FROM information_schema.columns where table_name = %s;", (t,))
+            cols_result = cur.fetchall()
+            if cols_result == []:
+                log.warning("Card Key database seems to lack table %s, check config file!" % t)
+                continue
+            if (key.lower(),) not in cols_result:
+                continue
+
+            # Query requested columns from database table
+            query = SQL("SELECT {}").format(Identifier(fields[0].lower()))
+            for f in fields[1:]:
+                query += SQL(", {}").format(Identifier(f.lower()))
+            query += SQL(" FROM {} WHERE {} = %s LIMIT 1;").format(Identifier(t.lower()),
+                                                                  Identifier(key.lower()))
+            cur.execute(query, (value,))
+            db_result = cur.fetchone()
+            cur.close()
+
+            if db_result:
+                break
+
+        if db_result is None:
+            return None
+        result = dict(zip(fields, db_result))
+
+        for k in result.keys():
+            result[k] = self.crypt.decrypt_field(k, result.get(k))
+        return result
 
 
 def card_key_provider_register(provider: CardKeyProvider, provider_list=card_key_providers):
@@ -128,11 +261,11 @@ def card_key_provider_register(provider: CardKeyProvider, provider_list=card_key
             provider_list : override the list of providers from the global default
     """
     if not isinstance(provider, CardKeyProvider):
-        raise ValueError("provider is not a card data provier")
+        raise ValueError("provider is not a card data provider")
     provider_list.append(provider)
 
 
-def card_key_provider_get(fields, key: str, value: str, provider_list=card_key_providers) -> Dict[str, str]:
+def card_key_provider_get(fields: list[str], key: str, value: str, provider_list=card_key_providers) -> Dict[str, str]:
     """Query all registered card data providers for card-individual [key] data.
 
     Args:
@@ -143,17 +276,21 @@ def card_key_provider_get(fields, key: str, value: str, provider_list=card_key_p
     Returns:
             dictionary of {field, value} strings for each requested field from 'fields'
     """
+    key = key.upper()
+    fields = [f.upper() for f in fields]
     for p in provider_list:
         if not isinstance(p, CardKeyProvider):
-            raise ValueError(
-                "provider list contains element which is not a card data provier")
+            raise ValueError("Provider list contains element which is not a card data provider")
+        log.debug("Searching for card key data (key=%s, value=%s, provider=%s)" % (key, value, str(p)))
         result = p.get(fields, key, value)
         if result:
+            log.debug("Found card data: %s" % (str(result)))
             return result
-    return {}
+
+    raise ValueError("Unable to find card key data (key=%s, value=%s, fields=%s)" % (key, value, str(fields)))
 
 
-def card_key_provider_get_field(field: str, key: str, value: str, provider_list=card_key_providers) -> Optional[str]:
+def card_key_provider_get_field(field: str, key: str, value: str, provider_list=card_key_providers) -> str:
     """Query all registered card data providers for a single field.
 
     Args:
@@ -164,11 +301,7 @@ def card_key_provider_get_field(field: str, key: str, value: str, provider_list=
     Returns:
             dictionary of {field, value} strings for the requested field
     """
-    for p in provider_list:
-        if not isinstance(p, CardKeyProvider):
-            raise ValueError(
-                "provider list contains element which is not a card data provier")
-        result = p.get_field(field, key, value)
-        if result:
-            return result
-    return None
+
+    fields = [field]
+    result = card_key_provider_get(fields, key, value, card_key_providers)
+    return result.get(field.upper())

pySim/cdma_ruim.py

@@ -0,0 +1,208 @@
+# coding=utf-8
+"""R-UIM (Removable User Identity Module) card profile (see 3GPP2 C.S0023-D)
+
+(C) 2023 by Vadim Yanitskiy <fixeria@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+import enum
+
+from construct import Bytewise, BitStruct, BitsInteger, Struct, FlagsEnum
+from osmocom.utils import *
+from osmocom.construct import *
+
+from pySim.filesystem import *
+from pySim.profile import CardProfile, CardProfileAddon
+from pySim.ts_51_011 import CardProfileSIM
+from pySim.ts_51_011 import DF_TELECOM, DF_GSM
+from pySim.ts_51_011 import EF_ServiceTable
+
+
+# Mapping between CDMA Service Number and its description
+EF_CST_map = {
+    1 : 'CHV disable function',
+    2 : 'Abbreviated Dialing Numbers (ADN)',
+    3 : 'Fixed Dialing Numbers (FDN)',
+    4 : 'Short Message Storage (SMS)',
+    5 : 'HRPD',
+    6 : 'Enhanced Phone Book',
+    7 : 'Multi Media Domain (MMD)',
+    8 : 'SF_EUIMID-based EUIMID',
+    9 : 'MEID Support',
+    10 : 'Extension1',
+    11 : 'Extension2',
+    12 : 'SMS Parameters',
+    13 : 'Last Number Dialled (LND)',
+    14 : 'Service Category Program for BC-SMS',
+    15 : 'Messaging and 3GPD Extensions',
+    16 : 'Root Certificates',
+    17 : 'CDMA Home Service Provider Name',
+    18 : 'Service Dialing Numbers (SDN)',
+    19 : 'Extension3',
+    20 : '3GPD-SIP',
+    21 : 'WAP Browser',
+    22 : 'Java',
+    23 : 'Reserved for CDG',
+    24 : 'Reserved for CDG',
+    25 : 'Data Download via SMS Broadcast',
+    26 : 'Data Download via SMS-PP',
+    27 : 'Menu Selection',
+    28 : 'Call Control',
+    29 : 'Proactive R-UIM',
+    30 : 'AKA',
+    31 : 'IPv6',
+    32 : 'RFU',
+    33 : 'RFU',
+    34 : 'RFU',
+    35 : 'RFU',
+    36 : 'RFU',
+    37 : 'RFU',
+    38 : '3GPD-MIP',
+    39 : 'BCMCS',
+    40 : 'Multimedia Messaging Service (MMS)',
+    41 : 'Extension 8',
+    42 : 'MMS User Connectivity Parameters',
+    43 : 'Application Authentication',
+    44 : 'Group Identifier Level 1',
+    45 : 'Group Identifier Level 2',
+    46 : 'De-Personalization Control Keys',
+    47 : 'Cooperative Network List',
+}
+
+
+######################################################################
+# DF.CDMA
+######################################################################
+
+class EF_SPN(TransparentEF):
+    '''3.4.31 CDMA Home Service Provider Name'''
+
+    _test_de_encode = [
+        ( "010801536b796c696e6b204e57ffffffffffffffffffffffffffffffffffffffffffff",
+          { 'rfu1' : 0, 'show_in_hsa' : True, 'rfu2' : 0,
+            'char_encoding' : 8, 'lang_ind' : 1, 'spn' : 'Skylink NW' } ),
+    ]
+
+    def __init__(self, fid='6f41', sfid=None, name='EF.SPN',
+                 desc='Service Provider Name', size=(35, 35), **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
+        self._construct = BitStruct(
+            # Byte 1: Display Condition
+            'rfu1'/BitsRFU(7),
+            'show_in_hsa'/Flag,
+            # Byte 2: Character Encoding
+            'rfu2'/BitsRFU(3),
+            'char_encoding'/BitsInteger(5), # see C.R1001-G
+            # Byte 3: Language Indicator
+            'lang_ind'/BitsInteger(8), # see C.R1001-G
+            # Bytes 4-35: Service Provider Name
+            'spn'/Bytewise(GsmString(32))
+        )
+
+class EF_AD(TransparentEF):
+    '''3.4.33 Administrative Data'''
+
+    _test_de_encode = [
+        ( "000000", { 'ms_operation_mode' : 'normal', 'additional_info' : b'\x00\x00', 'rfu' : b'' } ),
+    ]
+    _test_no_pad = True
+
+    class OP_MODE(enum.IntEnum):
+        normal = 0x00
+        type_approval = 0x80
+        normal_and_specific_facilities = 0x01
+        type_approval_and_specific_facilities = 0x81
+        maintenance_off_line = 0x02
+        cell_test = 0x04
+
+    def __init__(self, fid='6f43', sfid=None, name='EF.AD',
+                 desc='Service Provider Name', size=(3, None), **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
+        self._construct = Struct(
+            # Byte 1: Display Condition
+            'ms_operation_mode'/Enum(Byte, self.OP_MODE),
+            # Bytes 2-3: Additional information
+            'additional_info'/Bytes(2),
+            # Bytes 4..: RFU
+            'rfu'/GreedyBytesRFU,
+        )
+
+
+class EF_SMS(LinFixedEF):
+    '''3.4.27 Short Messages'''
+    def __init__(self, fid='6f3c', sfid=None, name='EF.SMS', desc='Short messages', **kwargs):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len=(2, 255), **kwargs)
+        self._construct = Struct(
+            # Byte 1: Status
+            'status'/BitStruct(
+                'rfu87'/BitsRFU(2),
+                'protection'/Flag,
+                'rfu54'/BitsRFU(2),
+                'status'/FlagsEnum(BitsInteger(2), read=0, to_be_read=1, sent=2, to_be_sent=3),
+                'used'/Flag,
+            ),
+            # Byte 2: Length
+            'length'/Int8ub,
+            # Bytes 3..: SMS Transport Layer Message
+            'tpdu'/Bytes(lambda ctx: ctx.length if ctx.status.used else 0),
+        )
+
+
+class DF_CDMA(CardDF):
+    def __init__(self):
+        super().__init__(fid='7f25', name='DF.CDMA',
+                         desc='CDMA related files (3GPP2 C.S0023-D)')
+        files = [
+            # TODO: lots of other files
+            EF_ServiceTable('6f32', None, 'EF.CST',
+                            'CDMA Service Table', table=EF_CST_map, size=(5, 16)),
+            EF_SPN(),
+            EF_AD(),
+            EF_SMS(),
+        ]
+        self.add_files(files)
+
+
+class CardProfileRUIM(CardProfile):
+    '''R-UIM card profile as per 3GPP2 C.S0023-D'''
+
+    ORDER = 20
+
+    def __init__(self):
+        super().__init__('R-UIM', desc='CDMA R-UIM Card', cla="a0",
+                         sel_ctrl="0000", files_in_mf=[DF_TELECOM(), DF_GSM(), DF_CDMA()])
+
+    @staticmethod
+    def decode_select_response(data_hex: str) -> object:
+        # TODO: Response parameters/data in case of DF_CDMA (section 2.6)
+        return CardProfileSIM.decode_select_response(data_hex)
+
+    @classmethod
+    def _try_match_card(cls, scc: SimCardCommands) -> None:
+        """ Try to access MF/DF.CDMA via 2G APDUs (3GPP TS 11.11), if this works,
+        the card is considered an R-UIM card for CDMA."""
+        cls._mf_select_test(scc, "a0", "0000", ["3f00", "7f25"])
+
+
+class AddonRUIM(CardProfileAddon):
+    """An Addon that can be found on on a combined SIM + RUIM or UICC + RUIM to support CDMA."""
+    def __init__(self):
+        files = [
+            DF_CDMA()
+        ]
+        super().__init__('RUIM', desc='CDMA RUIM', files_in_mf=files)
+
+    def probe(self, card: 'CardBase') -> bool:
+        return card.file_exists(self.files_in_mf[0].fid)

pySim/commands.py

@@ -5,7 +5,7 @@
 
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
-# Copyright (C) 2010-2021  Harald Welte <laforge@gnumonks.org>
+# Copyright (C) 2010-2024  Harald Welte <laforge@gnumonks.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,20 +21,163 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from construct import *
-from pySim.construct import LV
-from pySim.utils import rpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, str_sanitize
+from typing import List, Tuple
+import typing # construct also has a Union, so we do typing.Union below
+from construct import Construct, Struct, Const, Select
+from construct import Optional as COptional
+from osmocom.construct import LV, filter_dict
+from osmocom.utils import rpad, lpad, b2h, h2b, h2i, i2h, str_sanitize, Hexstr
+from osmocom.tlv import bertlv_encode_len
+
+from pySim.utils import sw_match, expand_hex, SwHexstr, ResTuple, SwMatchstr
 from pySim.exceptions import SwMatchError
+from pySim.transport import LinkBase
 
+# A path can be either just a FID or a list of FID
+Path = typing.Union[Hexstr, List[Hexstr]]
 
-class SimCardCommands(object):
-    def __init__(self, transport):
+def lchan_nr_to_cla(cla: int, lchan_nr: int) -> int:
+    """Embed a logical channel number into the CLA byte."""
+    # TS 102 221 10.1.1 Coding of Class Byte
+    if lchan_nr < 4:
+        # standard logical channel number
+        if cla >> 4 in [0x0, 0xA, 0x8]:
+            return (cla & 0xFC) | (lchan_nr & 3)
+        else:
+            raise ValueError('Undefined how to use CLA %2X with logical channel %u' % (cla, lchan_nr))
+    elif lchan_nr < 16:
+        # extended logical channel number
+        if cla >> 6 in [1, 3]:
+            return (cla & 0xF0) | ((lchan_nr - 4) & 0x0F)
+        else:
+            raise ValueError('Undefined how to use CLA %2X with logical channel %u' % (cla, lchan_nr))
+    else:
+        raise ValueError('logical channel outside of range 0 .. 15')
+
+def cla_with_lchan(cla_byte: Hexstr, lchan_nr: int) -> Hexstr:
+    """Embed a logical channel number into the hex-string encoded CLA value."""
+    cla_int = h2i(cla_byte)[0]
+    return i2h([lchan_nr_to_cla(cla_int, lchan_nr)])
+
+class SimCardCommands:
+    """Class providing methods for various card-specific commands such as SELECT, READ BINARY, etc.
+    Historically one instance exists below CardBase, but with the introduction of multiple logical
+    channels there can be multiple instances.  The lchan number will then be patched into the CLA
+    byte by the respective instance. """
+    def __init__(self, transport: LinkBase, lchan_nr: int = 0):
         self._tp = transport
-        self.cla_byte = "a0"
         self.sel_ctrl = "0000"
+        self.lchan_nr = lchan_nr
+        # invokes the setter below
+        self.cla_byte = "a0"
+        self.scp = None # Secure Channel Protocol
+
+    def fork_lchan(self, lchan_nr: int) -> 'SimCardCommands':
+        """Fork a per-lchan specific SimCardCommands instance off the current instance."""
+        ret = SimCardCommands(transport = self._tp, lchan_nr = lchan_nr)
+        ret.cla_byte = self.cla_byte
+        ret.sel_ctrl = self.sel_ctrl
+        return ret
+
+    @property
+    def max_cmd_len(self) -> int:
+        """Maximum length of the command apdu data section. Depends on secure channel protocol used."""
+        if self.scp:
+            return 255 - self.scp.overhead
+        else:
+            return 255
+
+    def send_apdu(self, pdu: Hexstr, apply_lchan:bool = True) -> ResTuple:
+        """Sends an APDU and auto fetch response data
+
+        Args:
+           pdu : string of hexadecimal characters (ex. "A0A40000023F00")
+           apply_lchan : apply the currently selected lchan to the CLA byte before sending
+        Returns:
+           tuple(data, sw), where
+                        data : string (in hex) of returned data (ex. "074F4EFFFF")
+                        sw   : string (in hex) of status word (ex. "9000")
+        """
+        if apply_lchan:
+            pdu = cla_with_lchan(pdu[0:2], self.lchan_nr) + pdu[2:]
+        if self.scp:
+            return self.scp.send_apdu_wrapper(self._tp.send_apdu, pdu)
+        else:
+            return self._tp.send_apdu(pdu)
+
+    def send_apdu_checksw(self, pdu: Hexstr, sw: SwMatchstr = "9000", apply_lchan:bool = True) -> ResTuple:
+        """Sends an APDU and check returned SW
+
+        Args:
+           pdu : string of hexadecimal characters (ex. "A0A40000023F00")
+           sw : string of 4 hexadecimal characters (ex. "9000"). The user may mask out certain
+                digits using a '?' to add some ambiguity if needed.
+           apply_lchan : apply the currently selected lchan to the CLA byte before sending
+        Returns:
+                tuple(data, sw), where
+                        data : string (in hex) of returned data (ex. "074F4EFFFF")
+                        sw   : string (in hex) of status word (ex. "9000")
+        """
+        if apply_lchan:
+            pdu = cla_with_lchan(pdu[0:2], self.lchan_nr) + pdu[2:]
+        if self.scp:
+            return self.scp.send_apdu_wrapper(self._tp.send_apdu_checksw, pdu, sw)
+        else:
+            return self._tp.send_apdu_checksw(pdu, sw)
+
+    def send_apdu_constr(self, cla: Hexstr, ins: Hexstr, p1: Hexstr, p2: Hexstr, cmd_constr: Construct,
+                         cmd_data: Hexstr, resp_constr: Construct, apply_lchan:bool = True) -> Tuple[dict, SwHexstr]:
+        """Build and sends an APDU using a 'construct' definition; parses response.
+
+        Args:
+                cla : string (in hex) ISO 7816 class byte
+                ins : string (in hex) ISO 7816 instruction byte
+                p1 : string (in hex) ISO 7116 Parameter 1 byte
+                p2 : string (in hex) ISO 7116 Parameter 2 byte
+                cmd_cosntr : defining how to generate binary APDU command data
+                cmd_data : command data passed to cmd_constr
+                resp_cosntr : defining how to decode binary APDU response data
+                apply_lchan : apply the currently selected lchan to the CLA byte before sending
+        Returns:
+                Tuple of (decoded_data, sw)
+        """
+        cmd = cmd_constr.build(cmd_data) if cmd_data else b''
+        lc = i2h([len(cmd)]) if cmd_data else ''
+        le = '00' if resp_constr else ''
+        pdu = ''.join([cla, ins, p1, p2, lc, b2h(cmd), le])
+        (data, sw) = self.send_apdu(pdu, apply_lchan = apply_lchan)
+        if data:
+            # filter the resulting dict to avoid '_io' members inside
+            rsp = filter_dict(resp_constr.parse(h2b(data)))
+        else:
+            rsp = None
+        return (rsp, sw)
+
+    def send_apdu_constr_checksw(self, cla: Hexstr, ins: Hexstr, p1: Hexstr, p2: Hexstr,
+                                 cmd_constr: Construct, cmd_data: Hexstr, resp_constr: Construct,
+                                 sw_exp: SwMatchstr="9000", apply_lchan:bool = True) -> Tuple[dict, SwHexstr]:
+        """Build and sends an APDU using a 'construct' definition; parses response.
+
+        Args:
+                cla : string (in hex) ISO 7816 class byte
+                ins : string (in hex) ISO 7816 instruction byte
+                p1 : string (in hex) ISO 7116 Parameter 1 byte
+                p2 : string (in hex) ISO 7116 Parameter 2 byte
+                cmd_cosntr : defining how to generate binary APDU command data
+                cmd_data : command data passed to cmd_constr
+                resp_cosntr : defining how to decode  binary APDU response data
+                exp_sw : string (in hex) of status word (ex. "9000")
+        Returns:
+                Tuple of (decoded_data, sw)
+        """
+        (rsp, sw) = self.send_apdu_constr(cla, ins, p1, p2, cmd_constr, cmd_data, resp_constr,
+                                          apply_lchan = apply_lchan)
+        if not sw_match(sw, sw_exp):
+            raise SwMatchError(sw, sw_exp.lower(), self._tp.sw_interpreter)
+        return (rsp, sw)
 
     # Extract a single FCP item from TLV
-    def __parse_fcp(self, fcp):
+    def __parse_fcp(self, fcp: Hexstr):
         # see also: ETSI TS 102 221, chapter 11.1.1.3.1 Response for MF,
         # DF or ADF
         from pytlv.TLV import TLV
@@ -53,6 +196,7 @@ class SimCardCommands(object):
         # checking if the length of the remaining TLV string matches
         # what we get in the length field.
         # See also ETSI TS 102 221, chapter 11.1.1.3.0 Base coding.
+        # TODO: this likely just is normal BER-TLV ("All data objects are BER-TLV except if otherwise # defined.")
         exp_tlv_len = int(fcp[2:4], 16)
         if len(fcp[4:]) // 2 == exp_tlv_len:
             skip = 4
@@ -60,6 +204,7 @@ class SimCardCommands(object):
             exp_tlv_len = int(fcp[2:6], 16)
             if len(fcp[4:]) // 2 == exp_tlv_len:
                 skip = 6
+            raise ValueError('Cannot determine length of TLV-length')
 
         # Skip FCP tag and length
         tlv = fcp[skip:]
@@ -88,11 +233,11 @@ class SimCardCommands(object):
         else:
             return int(r[-1][4:8], 16)
 
-    def get_atr(self) -> str:
+    def get_atr(self) -> Hexstr:
         """Return the ATR of the currently inserted card."""
         return self._tp.get_atr()
 
-    def try_select_path(self, dir_list):
+    def try_select_path(self, dir_list: List[Hexstr]) -> List[ResTuple]:
         """ Try to select a specified path
 
         Args:
@@ -100,17 +245,16 @@ class SimCardCommands(object):
         """
 
         rv = []
-        if type(dir_list) is not list:
+        if not isinstance(dir_list, list):
             dir_list = [dir_list]
         for i in dir_list:
-            data, sw = self._tp.send_apdu(
-                self.cla_byte + "a4" + self.sel_ctrl + "02" + i)
+            data, sw = self.send_apdu(self.cla_byte + "a4" + self.sel_ctrl + "02" + i + "00")
             rv.append((data, sw))
             if sw != '9000':
                 return rv
         return rv
 
-    def select_path(self, dir_list):
+    def select_path(self, dir_list: Path) -> List[Hexstr]:
         """Execute SELECT for an entire list/path of FIDs.
 
         Args:
@@ -120,37 +264,37 @@ class SimCardCommands(object):
                 list of return values (FCP in hex encoding) for each element of the path
         """
         rv = []
-        if type(dir_list) is not list:
+        if not isinstance(dir_list, list):
             dir_list = [dir_list]
         for i in dir_list:
-            data, sw = self.select_file(i)
+            data, _sw = self.select_file(i)
             rv.append(data)
         return rv
 
-    def select_file(self, fid: str):
+    def select_file(self, fid: Hexstr) -> ResTuple:
         """Execute SELECT a given file by FID.
 
         Args:
                 fid : file identifier as hex string
         """
 
-        return self._tp.send_apdu_checksw(self.cla_byte + "a4" + self.sel_ctrl + "02" + fid)
+        return self.send_apdu_checksw(self.cla_byte + "a4" + self.sel_ctrl + "02" + fid + "00")
 
-    def select_parent_df(self):
+    def select_parent_df(self) -> ResTuple:
         """Execute SELECT to switch to the parent DF """
-        return self._tp.send_apdu_checksw(self.cla_byte + "a4030400")
+        return self.send_apdu_checksw(self.cla_byte + "a40304")
 
-    def select_adf(self, aid: str):
-        """Execute SELECT a given Applicaiton ADF.
+    def select_adf(self, aid: Hexstr) -> ResTuple:
+        """Execute SELECT a given Application ADF.
 
         Args:
                 aid : application identifier as hex string
         """
 
         aidlen = ("0" + format(len(aid) // 2, 'x'))[-2:]
-        return self._tp.send_apdu_checksw(self.cla_byte + "a4" + "0404" + aidlen + aid)
+        return self.send_apdu_checksw(self.cla_byte + "a4" + "0404" + aidlen + aid + "00")
 
-    def read_binary(self, ef, length: int = None, offset: int = 0):
+    def read_binary(self, ef: Path, length: int = None, offset: int = 0) -> ResTuple:
         """Execute READD BINARY.
 
         Args:
@@ -169,56 +313,19 @@ class SimCardCommands(object):
         total_data = ''
         chunk_offset = 0
         while chunk_offset < length:
-            chunk_len = min(255, length-chunk_offset)
+            chunk_len = min(self.max_cmd_len, length-chunk_offset)
             pdu = self.cla_byte + \
                 'b0%04x%02x' % (offset + chunk_offset, chunk_len)
             try:
-                data, sw = self._tp.send_apdu_checksw(pdu)
+                data, sw = self.send_apdu_checksw(pdu)
             except Exception as e:
-                raise ValueError('%s, failed to read (offset %d)' %
-                                 (str_sanitize(str(e)), offset))
+                e.add_note('failed to read (offset %d)' % offset)
+                raise e
             total_data += data
             chunk_offset += chunk_len
         return total_data, sw
 
-    def update_binary(self, ef, data: str, offset: int = 0, verify: bool = False, conserve: bool = False):
-        """Execute UPDATE BINARY.
-
-        Args:
-                ef : string or list of strings indicating name or path of transparent EF
-                data : hex string of data to be written
-                offset : byte offset in file from which to start writing
-                verify : Whether or not to verify data after write
-        """
-        data_length = len(data) // 2
-
-        # Save write cycles by reading+comparing before write
-        if conserve:
-            data_current, sw = self.read_binary(ef, data_length, offset)
-            if data_current == data:
-                return None, sw
-
-        self.select_path(ef)
-        total_data = ''
-        chunk_offset = 0
-        while chunk_offset < data_length:
-            chunk_len = min(255, data_length - chunk_offset)
-            # chunk_offset is bytes, but data slicing is hex chars, so we need to multiply by 2
-            pdu = self.cla_byte + \
-                'd6%04x%02x' % (offset + chunk_offset, chunk_len) + \
-                data[chunk_offset*2: (chunk_offset+chunk_len)*2]
-            try:
-                chunk_data, chunk_sw = self._tp.send_apdu_checksw(pdu)
-            except Exception as e:
-                raise ValueError('%s, failed to write chunk (chunk_offset %d, chunk_len %d)' %
-                                 (str_sanitize(str(e)), chunk_offset, chunk_len))
-            total_data += data
-            chunk_offset += chunk_len
-        if verify:
-            self.verify_binary(ef, data, offset)
-        return total_data, chunk_sw
-
-    def verify_binary(self, ef, data: str, offset: int = 0):
+    def __verify_binary(self, ef, data: str, offset: int = 0):
         """Verify contents of transparent EF.
 
         Args:
@@ -231,7 +338,56 @@ class SimCardCommands(object):
             raise ValueError('Binary verification failed (expected %s, got %s)' % (
                 data.lower(), res[0].lower()))
 
-    def read_record(self, ef, rec_no: int):
+    def update_binary(self, ef: Path, data: Hexstr, offset: int = 0, verify: bool = False,
+                      conserve: bool = False) -> ResTuple:
+        """Execute UPDATE BINARY.
+
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+                data : hex string of data to be written
+                offset : byte offset in file from which to start writing
+                verify : Whether or not to verify data after write
+        """
+
+        file_len = self.binary_size(ef)
+        data = expand_hex(data, file_len)
+
+        data_length = len(data) // 2
+
+        # Save write cycles by reading+comparing before write
+        if conserve:
+            try:
+                data_current, sw = self.read_binary(ef, data_length, offset)
+                if data_current == data:
+                    return None, sw
+            except Exception:
+                # cannot read data. This is not a fatal error, as reading is just done to
+                # conserve the amount of smart card writes.  The access conditions of the file
+                # may well permit us to UPDATE but not permit us to READ.  So let's ignore
+                # any such exception during READ.
+                pass
+
+        self.select_path(ef)
+        total_data = ''
+        chunk_offset = 0
+        while chunk_offset < data_length:
+            chunk_len = min(self.max_cmd_len, data_length - chunk_offset)
+            # chunk_offset is bytes, but data slicing is hex chars, so we need to multiply by 2
+            pdu = self.cla_byte + \
+                'd6%04x%02x' % (offset + chunk_offset, chunk_len) + \
+                data[chunk_offset*2: (chunk_offset+chunk_len)*2]
+            try:
+                chunk_data, chunk_sw = self.send_apdu_checksw(pdu)
+            except Exception as e:
+                e.add_note('failed to write chunk (chunk_offset %d, chunk_len %d)' % (chunk_offset, chunk_len))
+                raise e
+            total_data += data
+            chunk_offset += chunk_len
+        if verify:
+            self.__verify_binary(ef, data, offset)
+        return total_data, chunk_sw
+
+    def read_record(self, ef: Path, rec_no: int) -> ResTuple:
         """Execute READ RECORD.
 
         Args:
@@ -241,50 +397,9 @@ class SimCardCommands(object):
         r = self.select_path(ef)
         rec_length = self.__record_len(r)
         pdu = self.cla_byte + 'b2%02x04%02x' % (rec_no, rec_length)
-        return self._tp.send_apdu_checksw(pdu)
+        return self.send_apdu_checksw(pdu)
 
-    def update_record(self, ef, rec_no: int, data: str, force_len: bool = False, verify: bool = False,
-                      conserve: bool = False):
-        """Execute UPDATE RECORD.
-
-        Args:
-                ef : string or list of strings indicating name or path of linear fixed EF
-                rec_no : record number to read
-                data : hex string of data to be written
-                force_len : enforce record length by using the actual data length
-                verify : verify data by re-reading the record
-                conserve : read record and compare it with data, skip write on match
-        """
-        res = self.select_path(ef)
-
-        if force_len:
-            # enforce the record length by the actual length of the given data input
-            rec_length = len(data) // 2
-        else:
-            # determine the record length from the select response of the file and pad
-            # the input data with 0xFF if necessary. In cases where the input data
-            # exceed we throw an exception.
-            rec_length = self.__record_len(res)
-            if (len(data) // 2 > rec_length):
-                raise ValueError('Data length exceeds record length (expected max %d, got %d)' % (
-                    rec_length, len(data) // 2))
-            elif (len(data) // 2 < rec_length):
-                data = rpad(data, rec_length * 2)
-
-        # Save write cycles by reading+comparing before write
-        if conserve:
-            data_current, sw = self.read_record(ef, rec_no)
-            data_current = data_current[0:rec_length*2]
-            if data_current == data:
-                return None, sw
-
-        pdu = (self.cla_byte + 'dc%02x04%02x' % (rec_no, rec_length)) + data
-        res = self._tp.send_apdu_checksw(pdu)
-        if verify:
-            self.verify_record(ef, rec_no, data)
-        return res
-
-    def verify_record(self, ef, rec_no: int, data: str):
+    def __verify_record(self, ef: Path, rec_no: int, data: str):
         """Verify record against given data
 
         Args:
@@ -297,7 +412,60 @@ class SimCardCommands(object):
             raise ValueError('Record verification failed (expected %s, got %s)' % (
                 data.lower(), res[0].lower()))
 
-    def record_size(self, ef):
+    def update_record(self, ef: Path, rec_no: int, data: Hexstr, force_len: bool = False,
+                      verify: bool = False, conserve: bool = False, leftpad: bool = False) -> ResTuple:
+        """Execute UPDATE RECORD.
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+                rec_no : record number to read
+                data : hex string of data to be written
+                force_len : enforce record length by using the actual data length
+                verify : verify data by re-reading the record
+                conserve : read record and compare it with data, skip write on match
+                leftpad : apply 0xff padding from the left instead from the right side.
+        """
+
+        res = self.select_path(ef)
+        rec_length = self.__record_len(res)
+        data = expand_hex(data, rec_length)
+
+        if force_len:
+            # enforce the record length by the actual length of the given data input
+            rec_length = len(data) // 2
+        else:
+            # make sure the input data is padded to the record length using 0xFF.
+            # In cases where the input data exceed we throw an exception.
+            if len(data) // 2 > rec_length:
+                raise ValueError('Data length exceeds record length (expected max %d, got %d)' % (
+                    rec_length, len(data) // 2))
+            elif len(data) // 2 < rec_length:
+                if leftpad:
+                    data = lpad(data, rec_length * 2)
+                else:
+                    data = rpad(data, rec_length * 2)
+
+        # Save write cycles by reading+comparing before write
+        if conserve:
+            try:
+                data_current, sw = self.read_record(ef, rec_no)
+                data_current = data_current[0:rec_length*2]
+                if data_current == data:
+                    return None, sw
+            except Exception:
+                # cannot read data. This is not a fatal error, as reading is just done to
+                # conserve the amount of smart card writes.  The access conditions of the file
+                # may well permit us to UPDATE but not permit us to READ.  So let's ignore
+                # any such exception during READ.
+                pass
+
+        pdu = (self.cla_byte + 'dc%02x04%02x' % (rec_no, rec_length)) + data
+        res = self.send_apdu_checksw(pdu)
+        if verify:
+            self.__verify_record(ef, rec_no, data)
+        return res
+
+    def record_size(self, ef: Path) -> int:
         """Determine the record size of given file.
 
         Args:
@@ -306,7 +474,7 @@ class SimCardCommands(object):
         r = self.select_path(ef)
         return self.__record_len(r)
 
-    def record_count(self, ef):
+    def record_count(self, ef: Path) -> int:
         """Determine the number of records in given file.
 
         Args:
@@ -315,7 +483,7 @@ class SimCardCommands(object):
         r = self.select_path(ef)
         return self.__len(r) // self.__record_len(r)
 
-    def binary_size(self, ef):
+    def binary_size(self, ef: Path) -> int:
         """Determine the size of given transparent file.
 
         Args:
@@ -325,14 +493,14 @@ class SimCardCommands(object):
         return self.__len(r)
 
     # TS 102 221 Section 11.3.1 low-level helper
-    def _retrieve_data(self, tag: int, first: bool = True):
+    def _retrieve_data(self, tag: int, first: bool = True) -> ResTuple:
         if first:
-            pdu = '80cb008001%02x' % (tag)
+            pdu = '80cb008001%02x00' % (tag)
         else:
-            pdu = '80cb000000'
-        return self._tp.send_apdu_checksw(pdu)
+            pdu = '80cb0000'
+        return self.send_apdu_checksw(pdu)
 
-    def retrieve_data(self, ef, tag: int):
+    def retrieve_data(self, ef: Path, tag: int) -> ResTuple:
         """Execute RETRIEVE DATA, see also TS 102 221 Section 11.3.1.
 
         Args
@@ -346,23 +514,23 @@ class SimCardCommands(object):
         # retrieve first block
         data, sw = self._retrieve_data(tag, first=True)
         total_data += data
-        while sw == '62f1' or sw == '62f2':
+        while sw in ['62f1', '62f2']:
             data, sw = self._retrieve_data(tag, first=False)
             total_data += data
         return total_data, sw
 
     # TS 102 221 Section 11.3.2 low-level helper
-    def _set_data(self, data: str, first: bool = True):
+    def _set_data(self, data: Hexstr, first: bool = True) -> ResTuple:
         if first:
             p1 = 0x80
         else:
             p1 = 0x00
-        if isinstance(data, bytes) or isinstance(data, bytearray):
+        if isinstance(data, (bytes, bytearray)):
             data = b2h(data)
         pdu = '80db00%02x%02x%s' % (p1, len(data)//2, data)
-        return self._tp.send_apdu_checksw(pdu)
+        return self.send_apdu_checksw(pdu)
 
-    def set_data(self, ef, tag: int, value: str, verify: bool = False, conserve: bool = False):
+    def set_data(self, ef, tag: int, value: str, verify: bool = False, conserve: bool = False) -> ResTuple:
         """Execute SET DATA.
 
         Args
@@ -387,13 +555,13 @@ class SimCardCommands(object):
         total_len = len(tlv_bin)
         remaining = tlv_bin
         while len(remaining) > 0:
-            fragment = remaining[:255]
+            fragment = remaining[:self.max_cmd_len]
             rdata, sw = self._set_data(fragment, first=first)
             first = False
-            remaining = remaining[255:]
+            remaining = remaining[self.max_cmd_len:]
         return rdata, sw
 
-    def run_gsm(self, rand: str):
+    def run_gsm(self, rand: Hexstr) -> ResTuple:
         """Execute RUN GSM ALGORITHM.
 
         Args:
@@ -402,21 +570,20 @@ class SimCardCommands(object):
         if len(rand) != 32:
             raise ValueError('Invalid rand')
         self.select_path(['3f00', '7f20'])
-        return self._tp.send_apdu(self.cla_byte + '88000010' + rand)
+        return self.send_apdu_checksw('a088000010' + rand + '00', sw='9000')
 
-    def authenticate(self, rand: str, autn: str, context='3g'):
+    def authenticate(self, rand: Hexstr, autn: Hexstr, context: str = '3g') -> ResTuple:
         """Execute AUTHENTICATE (USIM/ISIM).
 
         Args:
                 rand : 16 byte random data as hex string (RAND)
-                autn : 8 byte Autentication Token (AUTN)
+                autn : 8 byte Authentication Token (AUTN)
                 context : 16 byte random data ('3g' or 'gsm')
         """
         # 3GPP TS 31.102 Section 7.1.2.1
-        AuthCmd3G = Struct('rand'/LV, 'autn'/Optional(LV))
+        AuthCmd3G = Struct('rand'/LV, 'autn'/COptional(LV))
         AuthResp3GSyncFail = Struct(Const(b'\xDC'), 'auts'/LV)
-        AuthResp3GSuccess = Struct(
-            Const(b'\xDB'), 'res'/LV, 'ck'/LV, 'ik'/LV, 'kc'/Optional(LV))
+        AuthResp3GSuccess = Struct(Const(b'\xDB'), 'res'/LV, 'ck'/LV, 'ik'/LV, 'kc'/COptional(LV))
         AuthResp3G = Select(AuthResp3GSyncFail, AuthResp3GSuccess)
         # build parameters
         cmd_data = {'rand': rand, 'autn': autn}
@@ -424,7 +591,9 @@ class SimCardCommands(object):
             p2 = '81'
         elif context == 'gsm':
             p2 = '80'
-        (data, sw) = self._tp.send_apdu_constr_checksw(
+        else:
+            raise ValueError("Unsupported context '%s'" % context)
+        (data, sw) = self.send_apdu_constr_checksw(
             self.cla_byte, '88', '00', p2, AuthCmd3G, cmd_data, AuthResp3G)
         if 'auts' in data:
             ret = {'synchronisation_failure': data}
@@ -432,43 +601,47 @@ class SimCardCommands(object):
             ret = {'successful_3g_authentication': data}
         return (ret, sw)
 
-    def status(self):
+    def status(self) -> ResTuple:
         """Execute a STATUS command as per TS 102 221 Section 11.1.2."""
-        return self._tp.send_apdu_checksw('80F20000ff')
+        return self.send_apdu_checksw('80F20000')
 
-    def deactivate_file(self):
+    def deactivate_file(self) -> ResTuple:
         """Execute DECATIVATE FILE command as per TS 102 221 Section 11.1.14."""
-        return self._tp.send_apdu_constr_checksw(self.cla_byte, '04', '00', '00', None, None, None)
+        return self.send_apdu_constr_checksw(self.cla_byte, '04', '00', '00', None, None, None)
 
-    def activate_file(self, fid):
+    def activate_file(self, fid: Hexstr) -> ResTuple:
         """Execute ACTIVATE FILE command as per TS 102 221 Section 11.1.15.
 
         Args:
                 fid : file identifier as hex string
         """
-        return self._tp.send_apdu_checksw(self.cla_byte + '44000002' + fid)
+        return self.send_apdu_checksw(self.cla_byte + '44000002' + fid)
 
-    def create_file(self, payload: Hexstr):
-        """Execute CREEATE FILE command as per TS 102 222 Section 6.3"""
-        return self._tp.send_apdu_checksw(self.cla_byte + 'e00000%02x%s' % (len(payload)//2, payload))
+    def create_file(self, payload: Hexstr) -> ResTuple:
+        """Execute CREATE FILE command as per TS 102 222 Section 6.3"""
+        return self.send_apdu_checksw(self.cla_byte + 'e00000%02x%s' % (len(payload)//2, payload))
 
-    def delete_file(self, fid):
+    def resize_file(self, payload: Hexstr) -> ResTuple:
+        """Execute RESIZE FILE command as per TS 102 222 Section 6.10"""
+        return self.send_apdu_checksw('80d40000%02x%s' % (len(payload)//2, payload))
+
+    def delete_file(self, fid: Hexstr) -> ResTuple:
         """Execute DELETE FILE command as per TS 102 222 Section 6.4"""
-        return self._tp.send_apdu_checksw(self.cla_byte + 'e4000002' + fid)
+        return self.send_apdu_checksw(self.cla_byte + 'e4000002' + fid)
 
-    def terminate_df(self, fid):
+    def terminate_df(self, fid: Hexstr) -> ResTuple:
         """Execute TERMINATE DF command as per TS 102 222 Section 6.7"""
-        return self._tp.send_apdu_checksw(self.cla_byte + 'e6000002' + fid)
+        return self.send_apdu_checksw(self.cla_byte + 'e6000002' + fid)
 
-    def terminate_ef(self, fid):
+    def terminate_ef(self, fid: Hexstr) -> ResTuple:
         """Execute TERMINATE EF command as per TS 102 222 Section 6.8"""
-        return self._tp.send_apdu_checksw(self.cla_byte + 'e8000002' + fid)
+        return self.send_apdu_checksw(self.cla_byte + 'e8000002' + fid)
 
-    def terminate_card_usage(self):
+    def terminate_card_usage(self) -> ResTuple:
         """Execute TERMINATE CARD USAGE command as per TS 102 222 Section 6.9"""
-        return self._tp.send_apdu_checksw(self.cla_byte + 'fe000000')
+        return self.send_apdu_checksw(self.cla_byte + 'fe000000')
 
-    def manage_channel(self, mode='open', lchan_nr=0):
+    def manage_channel(self, mode: str = 'open', lchan_nr: int =0) -> ResTuple:
         """Execute MANAGE CHANNEL command as per TS 102 221 Section 11.1.17.
 
         Args:
@@ -479,21 +652,21 @@ class SimCardCommands(object):
             p1 = 0x80
         else:
             p1 = 0x00
-        pdu = self.cla_byte + '70%02x%02x00' % (p1, lchan_nr)
-        return self._tp.send_apdu_checksw(pdu)
+        pdu = self.cla_byte + '70%02x%02x' % (p1, lchan_nr)
+        return self.send_apdu_checksw(pdu)
 
-    def reset_card(self):
+    def reset_card(self) -> Hexstr:
         """Physically reset the card"""
         return self._tp.reset_card()
 
-    def _chv_process_sw(self, op_name, chv_no, pin_code, sw):
+    def _chv_process_sw(self, op_name: str, chv_no: int, pin_code: Hexstr, sw: SwHexstr):
         if sw_match(sw, '63cx'):
             raise RuntimeError('Failed to %s chv_no 0x%02X with code 0x%s, %i tries left.' %
                                (op_name, chv_no, b2h(pin_code).upper(), int(sw[3])))
-        elif (sw != '9000'):
-            raise SwMatchError(sw, '9000')
+        if sw != '9000':
+            raise SwMatchError(sw, '9000', self._tp.sw_interpreter)
 
-    def verify_chv(self, chv_no: int, code: str):
+    def verify_chv(self, chv_no: int, code: Hexstr) -> ResTuple:
         """Verify a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -501,8 +674,7 @@ class SimCardCommands(object):
                 code : chv code as hex string
         """
         fc = rpad(b2h(code), 16)
-        data, sw = self._tp.send_apdu(
-            self.cla_byte + '2000' + ('%02X' % chv_no) + '08' + fc)
+        data, sw = self.send_apdu(self.cla_byte + '2000' + ('%02X' % chv_no) + '08' + fc)
         self._chv_process_sw('verify', chv_no, code, sw)
         return (data, sw)
 
@@ -515,12 +687,11 @@ class SimCardCommands(object):
                 pin_code : new chv code as hex string
         """
         fc = rpad(b2h(puk_code), 16) + rpad(b2h(pin_code), 16)
-        data, sw = self._tp.send_apdu(
-            self.cla_byte + '2C00' + ('%02X' % chv_no) + '10' + fc)
+        data, sw = self.send_apdu(self.cla_byte + '2C00' + ('%02X' % chv_no) + '10' + fc)
         self._chv_process_sw('unblock', chv_no, pin_code, sw)
         return (data, sw)
 
-    def change_chv(self, chv_no: int, pin_code: str, new_pin_code: str):
+    def change_chv(self, chv_no: int, pin_code: Hexstr, new_pin_code: Hexstr) -> ResTuple:
         """Change a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -529,12 +700,11 @@ class SimCardCommands(object):
                 new_pin_code : new chv code as hex string
         """
         fc = rpad(b2h(pin_code), 16) + rpad(b2h(new_pin_code), 16)
-        data, sw = self._tp.send_apdu(
-            self.cla_byte + '2400' + ('%02X' % chv_no) + '10' + fc)
+        data, sw = self.send_apdu(self.cla_byte + '2400' + ('%02X' % chv_no) + '10' + fc)
         self._chv_process_sw('change', chv_no, pin_code, sw)
         return (data, sw)
 
-    def disable_chv(self, chv_no: int, pin_code: str):
+    def disable_chv(self, chv_no: int, pin_code: Hexstr) -> ResTuple:
         """Disable a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -543,12 +713,11 @@ class SimCardCommands(object):
                 new_pin_code : new chv code as hex string
         """
         fc = rpad(b2h(pin_code), 16)
-        data, sw = self._tp.send_apdu(
-            self.cla_byte + '2600' + ('%02X' % chv_no) + '08' + fc)
+        data, sw = self.send_apdu(self.cla_byte + '2600' + ('%02X' % chv_no) + '08' + fc)
         self._chv_process_sw('disable', chv_no, pin_code, sw)
         return (data, sw)
 
-    def enable_chv(self, chv_no: int, pin_code: str):
+    def enable_chv(self, chv_no: int, pin_code: Hexstr) -> ResTuple:
         """Enable a given CHV (Card Holder Verification == PIN)
 
         Args:
@@ -556,31 +725,30 @@ class SimCardCommands(object):
                 pin_code : chv code as hex string
         """
         fc = rpad(b2h(pin_code), 16)
-        data, sw = self._tp.send_apdu(
-            self.cla_byte + '2800' + ('%02X' % chv_no) + '08' + fc)
+        data, sw = self.send_apdu(self.cla_byte + '2800' + ('%02X' % chv_no) + '08' + fc)
         self._chv_process_sw('enable', chv_no, pin_code, sw)
         return (data, sw)
 
-    def envelope(self, payload: str):
+    def envelope(self, payload: Hexstr) -> ResTuple:
         """Send one ENVELOPE command to the SIM
 
         Args:
                 payload : payload as hex string
         """
-        return self._tp.send_apdu_checksw('80c20000%02x%s' % (len(payload)//2, payload))
+        return self.send_apdu_checksw('80c20000%02x%s' % (len(payload)//2, payload) + "00", apply_lchan = False)
 
-    def terminal_profile(self, payload: str):
+    def terminal_profile(self, payload: Hexstr) -> ResTuple:
         """Send TERMINAL PROFILE to card
 
         Args:
                 payload : payload as hex string
         """
         data_length = len(payload) // 2
-        data, sw = self._tp.send_apdu(('80100000%02x' % data_length) + payload)
+        data, sw = self.send_apdu_checksw(('80100000%02x' % data_length) + payload, apply_lchan = False)
         return (data, sw)
 
     # ETSI TS 102 221 11.1.22
-    def suspend_uicc(self, min_len_secs: int = 60, max_len_secs: int = 43200):
+    def suspend_uicc(self, min_len_secs: int = 60, max_len_secs: int = 43200) -> Tuple[int, Hexstr, SwHexstr]:
         """Send SUSPEND UICC to the card.
 
         Args:
@@ -590,38 +758,49 @@ class SimCardCommands(object):
         def encode_duration(secs: int) -> Hexstr:
             if secs >= 10*24*60*60:
                 return '04%02x' % (secs // (10*24*60*60))
-            elif secs >= 24*60*60:
+            if secs >= 24*60*60:
                 return '03%02x' % (secs // (24*60*60))
-            elif secs >= 60*60:
+            if secs >= 60*60:
                 return '02%02x' % (secs // (60*60))
-            elif secs >= 60:
+            if secs >= 60:
                 return '01%02x' % (secs // 60)
-            else:
-                return '00%02x' % secs
+            return '00%02x' % secs
 
         def decode_duration(enc: Hexstr) -> int:
             time_unit = enc[:2]
-            length = h2i(enc[2:4])
+            length = h2i(enc[2:4])[0]
             if time_unit == '04':
                 return length * 10*24*60*60
-            elif time_unit == '03':
+            if time_unit == '03':
                 return length * 24*60*60
-            elif time_unit == '02':
+            if time_unit == '02':
                 return length * 60*60
-            elif time_unit == '01':
+            if time_unit == '01':
                 return length * 60
-            elif time_unit == '00':
+            if time_unit == '00':
                 return length
-            else:
-                raise ValueError('Time unit must be 0x00..0x04')
+            raise ValueError('Time unit must be 0x00..0x04')
         min_dur_enc = encode_duration(min_len_secs)
         max_dur_enc = encode_duration(max_len_secs)
-        data, sw = self._tp.send_apdu_checksw(
-            '8076000004' + min_dur_enc + max_dur_enc)
+        data, sw = self.send_apdu_checksw('8076000004' + min_dur_enc + max_dur_enc, apply_lchan = False)
         negotiated_duration_secs = decode_duration(data[:4])
         resume_token = data[4:]
         return (negotiated_duration_secs, resume_token, sw)
 
-    def get_data(self, tag: int, cla: int = 0x00):
-        data, sw = self._tp.send_apdu('%02xca%04x00' % (cla, tag))
+    # ETSI TS 102 221 11.1.22
+    def resume_uicc(self, token: Hexstr) -> ResTuple:
+        """Send SUSPEND UICC (resume) to the card."""
+        if len(h2b(token)) != 8:
+            raise ValueError("Token must be 8 bytes long")
+        data, sw = self.send_apdu_checksw('8076010008' + token, apply_lchan = False)
+        return (data, sw)
+
+    # GPC_SPE_034 11.3
+    def get_data(self, tag: int, cla: int = 0x00):
+        data, sw = self.send_apdu_checksw('%02xca%04x00' % (cla, tag))
+        return (data, sw)
+
+    # TS 31.102 Section 7.5.2
+    def get_identity(self, context: int) -> Tuple[Hexstr, SwHexstr]:
+        data, sw = self.send_apdu_checksw('807800%02x00' % (context))
         return (data, sw)

pySim/construct.py

@@ -1,257 +0,0 @@
-from construct.lib.containers import Container, ListContainer
-from construct.core import EnumIntegerString
-import typing
-from construct import *
-from construct.core import evaluate, bytes2integer, integer2bytes, BitwisableString
-from construct.lib import integertypes
-from pySim.utils import b2h, h2b, swap_nibbles
-import gsm0338
-
-"""Utility code related to the integration of the 'construct' declarative parser."""
-
-# (C) 2021-2022 by Harald Welte <laforge@osmocom.org>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-class HexAdapter(Adapter):
-    """convert a bytes() type to a string of hex nibbles."""
-
-    def _decode(self, obj, context, path):
-        return b2h(obj)
-
-    def _encode(self, obj, context, path):
-        return h2b(obj)
-
-
-class BcdAdapter(Adapter):
-    """convert a bytes() type to a string of BCD nibbles."""
-
-    def _decode(self, obj, context, path):
-        return swap_nibbles(b2h(obj))
-
-    def _encode(self, obj, context, path):
-        return h2b(swap_nibbles(obj))
-
-class InvertAdapter(Adapter):
-    """inverse logic (false->true, true->false)."""
-    @staticmethod
-    def _invert_bool_in_obj(obj):
-        for k,v in obj.items():
-            # skip all private entries
-            if k.startswith('_'):
-                continue
-            if v == False:
-                obj[k] = True
-            elif v == True:
-                obj[k] = False
-        return obj
-
-    def _decode(self, obj, context, path):
-        return self._invert_bool_in_obj(obj)
-
-    def _encode(self, obj, context, path):
-        return self._invert_bool_in_obj(obj)
-
-class Rpad(Adapter):
-    """
-    Encoder appends padding bytes (b'\\xff') up to target size.
-    Decoder removes trailing padding bytes.
-
-    Parameters:
-        subcon: Subconstruct as defined by construct library
-        pattern: set padding pattern (default: b'\\xff')
-    """
-
-    def __init__(self, subcon, pattern=b'\xff'):
-        super().__init__(subcon)
-        self.pattern = pattern
-
-    def _decode(self, obj, context, path):
-        return obj.rstrip(self.pattern)
-
-    def _encode(self, obj, context, path):
-        if len(obj) > self.sizeof():
-            raise SizeofError("Input ({}) exceeds target size ({})".format(
-                len(obj), self.sizeof()))
-        return obj + self.pattern * (self.sizeof() - len(obj))
-
-
-class GsmStringAdapter(Adapter):
-    """Convert GSM 03.38 encoded bytes to a string."""
-
-    def __init__(self, subcon, codec='gsm03.38', err='strict'):
-        super().__init__(subcon)
-        self.codec = codec
-        self.err = err
-
-    def _decode(self, obj, context, path):
-        return obj.decode(self.codec)
-
-    def _encode(self, obj, context, path):
-        return obj.encode(self.codec, self.err)
-
-
-def filter_dict(d, exclude_prefix='_'):
-    """filter the input dict to ensure no keys starting with 'exclude_prefix' remain."""
-    if not isinstance(d, dict):
-        return d
-    res = {}
-    for (key, value) in d.items():
-        if key.startswith(exclude_prefix):
-            continue
-        if type(value) is dict:
-            res[key] = filter_dict(value)
-        else:
-            res[key] = value
-    return res
-
-
-def normalize_construct(c):
-    """Convert a construct specific type to a related base type, mostly useful
-    so we can serialize it."""
-    # we need to include the filter_dict as we otherwise get elements like this
-    # in the dict: '_io': <_io.BytesIO object at 0x7fdb64e05860> which we cannot json-serialize
-    c = filter_dict(c)
-    if isinstance(c, Container) or isinstance(c, dict):
-        r = {k: normalize_construct(v) for (k, v) in c.items()}
-    elif isinstance(c, ListContainer):
-        r = [normalize_construct(x) for x in c]
-    elif isinstance(c, list):
-        r = [normalize_construct(x) for x in c]
-    elif isinstance(c, EnumIntegerString):
-        r = str(c)
-    else:
-        r = c
-    return r
-
-
-def parse_construct(c, raw_bin_data: bytes, length: typing.Optional[int] = None, exclude_prefix: str = '_'):
-    """Helper function to wrap around normalize_construct() and filter_dict()."""
-    if not length:
-        length = len(raw_bin_data)
-    parsed = c.parse(raw_bin_data, total_len=length)
-    return normalize_construct(parsed)
-
-
-# here we collect some shared / common definitions of data types
-LV = Prefixed(Int8ub, HexAdapter(GreedyBytes))
-
-# Default value for Reserved for Future Use (RFU) bits/bytes
-# See TS 31.101 Sec. "3.4 Coding Conventions"
-__RFU_VALUE = 0
-
-# Field that packs Reserved for Future Use (RFU) bit
-FlagRFU = Default(Flag, __RFU_VALUE)
-
-# Field that packs Reserved for Future Use (RFU) byte
-ByteRFU = Default(Byte, __RFU_VALUE)
-
-# Field that packs all remaining Reserved for Future Use (RFU) bytes
-GreedyBytesRFU = Default(GreedyBytes, b'')
-
-
-def BitsRFU(n=1):
-    '''
-    Field that packs Reserved for Future Use (RFU) bit(s)
-    as defined in TS 31.101 Sec. "3.4 Coding Conventions"
-
-    Use this for (currently) unused/reserved bits whose contents
-    should be initialized automatically but should not be cleared
-    in the future or when restoring read data (unlike padding).
-
-    Parameters:
-        n (Integer): Number of bits (default: 1)
-    '''
-    return Default(BitsInteger(n), __RFU_VALUE)
-
-
-def BytesRFU(n=1):
-    '''
-    Field that packs Reserved for Future Use (RFU) byte(s)
-    as defined in TS 31.101 Sec. "3.4 Coding Conventions"
-
-    Use this for (currently) unused/reserved bytes whose contents
-    should be initialized automatically but should not be cleared
-    in the future or when restoring read data (unlike padding).
-
-    Parameters:
-        n (Integer): Number of bytes (default: 1)
-    '''
-    return Default(Bytes(n), __RFU_VALUE)
-
-
-def GsmString(n):
-    '''
-    GSM 03.38 encoded byte string of fixed length n.
-    Encoder appends padding bytes (b'\\xff') to maintain
-    length. Decoder removes those trailing bytes.
-
-    Exceptions are raised for invalid characters
-    and length excess.
-
-    Parameters:
-        n (Integer): Fixed length of the encoded byte string
-    '''
-    return GsmStringAdapter(Rpad(Bytes(n), pattern=b'\xff'), codec='gsm03.38')
-
-class GreedyInteger(Construct):
-    """A variable-length integer implementation, think of combining GrredyBytes with BytesInteger."""
-    def __init__(self, signed=False, swapped=False):
-        super().__init__()
-        self.signed = signed
-        self.swapped = swapped
-
-    def _parse(self, stream, context, path):
-        data = stream_read_entire(stream, path)
-        if evaluate(self.swapped, context):
-            data = swapbytes(data)
-        try:
-            return bytes2integer(data, self.signed)
-        except ValueError as e:
-            raise IntegerError(str(e), path=path)
-
-    def __bytes_required(self, i):
-        if self.signed:
-            raise NotImplementedError("FIXME: Implement support for encoding signed integer")
-        nbytes = 1
-        while True:
-            i = i >> 8
-            if i == 0:
-                return nbytes
-            else:
-                nbytes = nbytes + 1
-        # this should never happen, above loop must return eventually...
-        raise IntegerError(f"value {i} is out of range")
-
-    def _build(self, obj, stream, context, path):
-        if not isinstance(obj, integertypes):
-            raise IntegerError(f"value {obj} is not an integer", path=path)
-        length = self.__bytes_required(obj)
-        try:
-            data = integer2bytes(obj, length, self.signed)
-        except ValueError as e:
-            raise IntegerError(str(e), path=path)
-        if evaluate(self.swapped, context):
-            data = swapbytes(data)
-        stream_write(stream, data, length, path)
-        return obj
-
-# merged definitions of 24.008 + 23.040
-TypeOfNumber = Enum(BitsInteger(3), unknown=0, international=1, national=2, network_specific=3,
-                    short_code=4, alphanumeric=5, abbreviated=6, reserved_for_extension=7)
-NumberingPlan = Enum(BitsInteger(4), unknown=0, isdn_e164=1, data_x121=3, telex_f69=4,
-                     sc_specific_5=5, sc_specific_6=6, national=8, private=9,
-                     ermes=10, reserved_cts=11, reserved_for_extension=15)
-TonNpi = BitStruct('ext'/Flag, 'type_of_number'/TypeOfNumber, 'numbering_plan_id'/NumberingPlan)

pySim/esim/__init__.py

@@ -0,0 +1,138 @@
+import sys
+from typing import Optional, Tuple
+from importlib import resources
+
+class PMO:
+    """Convenience conversion class for ProfileManagementOperation as used in ES9+ notifications."""
+    pmo4operation = {
+        'install': 0x80,
+        'enable': 0x40,
+        'disable': 0x20,
+        'delete': 0x10,
+    }
+
+    def __init__(self, op: str):
+        if not op in self.pmo4operation:
+            raise ValueError('Unknown operation "%s"' % op)
+        self.op = op
+
+    def to_int(self):
+        return self.pmo4operation[self.op]
+
+    @staticmethod
+    def _num_bits(data: int)-> int:
+        for i in range(0, 8):
+            if data & (1 << i):
+                return 8-i
+        return 0
+
+    def to_bitstring(self) -> Tuple[bytes, int]:
+        """return value in a format as used by asn1tools for BITSTRING."""
+        val = self.to_int()
+        return (bytes([val]), self._num_bits(val))
+
+    @classmethod
+    def from_int(cls, i: int) -> 'PMO':
+        """Parse an integer representation."""
+        for k, v in cls.pmo4operation.items():
+            if v == i:
+                return cls(k)
+        raise ValueError('Unknown PMO 0x%02x' % i)
+
+    @classmethod
+    def from_bitstring(cls, bstr: Tuple[bytes, int]) -> 'PMO':
+        """Parse a asn1tools BITSTRING representation."""
+        return cls.from_int(bstr[0][0])
+
+    def __str__(self):
+        return self.op
+
+def compile_asn1_subdir(subdir_name:str, codec='der'):
+    """Helper function that compiles ASN.1 syntax from all files within given subdir"""
+    import asn1tools
+    asn_txt = ''
+    __ver = sys.version_info
+    if (__ver.major, __ver.minor) >= (3, 9):
+        for i in resources.files('pySim.esim').joinpath('asn1').joinpath(subdir_name).iterdir():
+            asn_txt += i.read_text()
+            asn_txt += "\n"
+    #else:
+        #print(resources.read_text(__name__, 'asn1/rsp.asn'))
+    return asn1tools.compile_string(asn_txt, codec=codec)
+
+
+class ActivationCode:
+    """SGP.22 section 4.1 Activation Code"""
+    def __init__(self, hostname:str, token:str, oid: Optional[str] = None, cc_required: Optional[bool] = False):
+        if '$' in hostname:
+            raise ValueError('$ sign not permitted in hostname')
+        self.hostname = hostname
+        if '$' in token:
+            raise ValueError('$ sign not permitted in token')
+        self.token = token
+        # TODO: validate OID
+        self.oid = oid
+        self.cc_required = cc_required
+        # only format 1 is specified and supported here
+        self.format = 1
+
+    @staticmethod
+    def decode_str(ac: str) -> dict:
+        """decode an activation code from its string representation."""
+        if ac[0] != '1':
+            raise ValueError("Unsupported AC_Format '%s'!" % ac[0])
+        ac_elements = ac.split('$')
+        d = {
+            'oid': None,
+            'cc_required': False,
+          }
+        d['format'] = ac_elements.pop(0)
+        d['hostname'] = ac_elements.pop(0)
+        d['token'] = ac_elements.pop(0)
+        if len(ac_elements):
+            oid = ac_elements.pop(0)
+            if oid != '':
+                d['oid'] = oid
+        if len(ac_elements):
+            ccr = ac_elements.pop(0)
+            if ccr == '1':
+                d['cc_required'] = True
+        return d
+
+    @classmethod
+    def from_string(cls, ac: str) -> 'ActivationCode':
+        """Create new instance from SGP.22 section 4.1 string representation."""
+        d = cls.decode_str(ac)
+        return cls(d['hostname'], d['token'], d['oid'], d['cc_required'])
+
+    def to_string(self, for_qrcode:bool = False) -> str:
+        """Convert from internal representation to SGP.22 section 4.1 string representation."""
+        if for_qrcode:
+            ret = 'LPA:'
+        else:
+            ret = ''
+        ret += '%d$%s$%s' % (self.format, self.hostname, self.token)
+        if self.oid:
+            ret += '$%s' % (self.oid)
+        elif self.cc_required:
+            ret += '$'
+        if self.cc_required:
+            ret += '$1'
+        return ret
+
+    def __str__(self):
+        return self.to_string()
+
+    def to_qrcode(self):
+        """Encode internal representation to QR code."""
+        import qrcode
+        qr = qrcode.QRCode()
+        qr.add_data(self.to_string(for_qrcode=True))
+        return qr.make_image()
+
+    def __repr__(self):
+        return "ActivationCode(format=%u, hostname='%s', token='%s', oid=%s, cc_required=%s)" % (self.format,
+                                                                                                 self.hostname,
+                                                                                                 self.token,
+                                                                                                 self.oid,
+                                                                                                 self.cc_required)

pySim/esim/asn1/rsp/PKIX1Explicit88.asn

@@ -0,0 +1,657 @@
+PKIX1Explicit88 { iso(1) identified-organization(3) dod(6) internet(1)
+  security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-explicit(18) }
+
+DEFINITIONS EXPLICIT TAGS ::=
+
+BEGIN
+
+-- EXPORTS ALL --
+
+-- IMPORTS NONE --
+
+-- UNIVERSAL Types defined in 1993 and 1998 ASN.1
+-- and required by this specification
+
+-- pycrate: UniversalString, BMPString and UTF8String already in the builtin types
+
+--UniversalString ::= [UNIVERSAL 28] IMPLICIT OCTET STRING
+        -- UniversalString is defined in ASN.1:1993
+
+--BMPString ::= [UNIVERSAL 30] IMPLICIT OCTET STRING
+      -- BMPString is the subtype of UniversalString and models
+      -- the Basic Multilingual Plane of ISO/IEC 10646
+
+--UTF8String ::= [UNIVERSAL 12] IMPLICIT OCTET STRING
+      -- The content of this type conforms to RFC 3629.
+
+-- PKIX specific OIDs
+
+id-pkix  OBJECT IDENTIFIER  ::=
+         { iso(1) identified-organization(3) dod(6) internet(1)
+                    security(5) mechanisms(5) pkix(7) }
+
+-- PKIX arcs
+
+id-pe OBJECT IDENTIFIER ::= { id-pkix 1 }
+        -- arc for private certificate extensions
+id-qt OBJECT IDENTIFIER ::= { id-pkix 2 }
+        -- arc for policy qualifier types
+id-kp OBJECT IDENTIFIER ::= { id-pkix 3 }
+        -- arc for extended key purpose OIDS
+id-ad OBJECT IDENTIFIER ::= { id-pkix 48 }
+        -- arc for access descriptors
+
+-- policyQualifierIds for Internet policy qualifiers
+
+id-qt-cps      OBJECT IDENTIFIER ::=  { id-qt 1 }
+      -- OID for CPS qualifier
+id-qt-unotice  OBJECT IDENTIFIER ::=  { id-qt 2 }
+      -- OID for user notice qualifier
+
+-- access descriptor definitions
+
+id-ad-ocsp         OBJECT IDENTIFIER ::= { id-ad 1 }
+id-ad-caIssuers    OBJECT IDENTIFIER ::= { id-ad 2 }
+id-ad-timeStamping OBJECT IDENTIFIER ::= { id-ad 3 }
+id-ad-caRepository OBJECT IDENTIFIER ::= { id-ad 5 }
+
+-- attribute data types
+
+Attribute               ::= SEQUENCE {
+      type             AttributeType,
+      values    SET OF AttributeValue }
+            -- at least one value is required
+
+AttributeType           ::= OBJECT IDENTIFIER
+
+AttributeValue          ::= ANY -- DEFINED BY AttributeType
+
+AttributeTypeAndValue   ::= SEQUENCE {
+        type    AttributeType,
+        value   AttributeValue }
+
+-- suggested naming attributes: Definition of the following
+--   information object set may be augmented to meet local
+--   requirements.  Note that deleting members of the set may
+--   prevent interoperability with conforming implementations.
+-- presented in pairs: the AttributeType followed by the
+--   type definition for the corresponding AttributeValue
+
+-- Arc for standard naming attributes
+
+id-at OBJECT IDENTIFIER ::= { joint-iso-ccitt(2) ds(5) 4 }
+
+-- Naming attributes of type X520name
+
+id-at-name                AttributeType ::= { id-at 41 }
+id-at-surname             AttributeType ::= { id-at  4 }
+id-at-givenName           AttributeType ::= { id-at 42 }
+id-at-initials            AttributeType ::= { id-at 43 }
+id-at-generationQualifier AttributeType ::= { id-at 44 }
+
+-- Naming attributes of type X520Name:
+--   X520name ::= DirectoryString (SIZE (1..ub-name))
+--
+-- Expanded to avoid parameterized type:
+X520name ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-name)),
+      printableString   PrintableString (SIZE (1..ub-name)),
+      universalString   UniversalString (SIZE (1..ub-name)),
+      utf8String        UTF8String      (SIZE (1..ub-name)),
+      bmpString         BMPString       (SIZE (1..ub-name)) }
+
+-- Naming attributes of type X520CommonName
+
+id-at-commonName        AttributeType ::= { id-at 3 }
+
+-- Naming attributes of type X520CommonName:
+--   X520CommonName ::= DirectoryName (SIZE (1..ub-common-name))
+--
+-- Expanded to avoid parameterized type:
+X520CommonName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-common-name)),
+      printableString   PrintableString (SIZE (1..ub-common-name)),
+      universalString   UniversalString (SIZE (1..ub-common-name)),
+      utf8String        UTF8String      (SIZE (1..ub-common-name)),
+      bmpString         BMPString       (SIZE (1..ub-common-name)) }
+
+-- Naming attributes of type X520LocalityName
+
+id-at-localityName      AttributeType ::= { id-at 7 }
+
+-- Naming attributes of type X520LocalityName:
+--   X520LocalityName ::= DirectoryName (SIZE (1..ub-locality-name))
+--
+-- Expanded to avoid parameterized type:
+X520LocalityName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-locality-name)),
+      printableString   PrintableString (SIZE (1..ub-locality-name)),
+      universalString   UniversalString (SIZE (1..ub-locality-name)),
+      utf8String        UTF8String      (SIZE (1..ub-locality-name)),
+      bmpString         BMPString       (SIZE (1..ub-locality-name)) }
+
+-- Naming attributes of type X520StateOrProvinceName
+
+id-at-stateOrProvinceName AttributeType ::= { id-at 8 }
+
+-- Naming attributes of type X520StateOrProvinceName:
+--   X520StateOrProvinceName ::= DirectoryName (SIZE (1..ub-state-name))
+--
+-- Expanded to avoid parameterized type:
+X520StateOrProvinceName ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-state-name)),
+      printableString   PrintableString (SIZE (1..ub-state-name)),
+      universalString   UniversalString (SIZE (1..ub-state-name)),
+      utf8String        UTF8String      (SIZE (1..ub-state-name)),
+      bmpString         BMPString       (SIZE (1..ub-state-name)) }
+
+-- Naming attributes of type X520OrganizationName
+
+id-at-organizationName  AttributeType ::= { id-at 10 }
+
+-- Naming attributes of type X520OrganizationName:
+--   X520OrganizationName ::=
+--          DirectoryName (SIZE (1..ub-organization-name))
+--
+-- Expanded to avoid parameterized type:
+X520OrganizationName ::= CHOICE {
+      teletexString     TeletexString
+                          (SIZE (1..ub-organization-name)),
+      printableString   PrintableString
+                          (SIZE (1..ub-organization-name)),
+      universalString   UniversalString
+                          (SIZE (1..ub-organization-name)),
+      utf8String        UTF8String
+                          (SIZE (1..ub-organization-name)),
+      bmpString         BMPString
+                          (SIZE (1..ub-organization-name))  }
+
+-- Naming attributes of type X520OrganizationalUnitName
+
+id-at-organizationalUnitName AttributeType ::= { id-at 11 }
+
+-- Naming attributes of type X520OrganizationalUnitName:
+--   X520OrganizationalUnitName ::=
+--          DirectoryName (SIZE (1..ub-organizational-unit-name))
+--
+-- Expanded to avoid parameterized type:
+X520OrganizationalUnitName ::= CHOICE {
+      teletexString     TeletexString
+                          (SIZE (1..ub-organizational-unit-name)),
+      printableString   PrintableString
+                          (SIZE (1..ub-organizational-unit-name)),
+      universalString   UniversalString
+                          (SIZE (1..ub-organizational-unit-name)),
+      utf8String        UTF8String
+                          (SIZE (1..ub-organizational-unit-name)),
+      bmpString         BMPString
+                          (SIZE (1..ub-organizational-unit-name)) }
+
+-- Naming attributes of type X520Title
+
+id-at-title             AttributeType ::= { id-at 12 }
+
+-- Naming attributes of type X520Title:
+--   X520Title ::= DirectoryName (SIZE (1..ub-title))
+--
+-- Expanded to avoid parameterized type:
+X520Title ::= CHOICE {
+      teletexString     TeletexString   (SIZE (1..ub-title)),
+      printableString   PrintableString (SIZE (1..ub-title)),
+      universalString   UniversalString (SIZE (1..ub-title)),
+      utf8String        UTF8String      (SIZE (1..ub-title)),
+      bmpString         BMPString       (SIZE (1..ub-title)) }
+
+-- Naming attributes of type X520dnQualifier
+
+id-at-dnQualifier       AttributeType ::= { id-at 46 }
+
+X520dnQualifier ::=     PrintableString
+
+-- Naming attributes of type X520countryName (digraph from IS 3166)
+
+id-at-countryName       AttributeType ::= { id-at 6 }
+
+X520countryName ::=     PrintableString (SIZE (2))
+
+-- Naming attributes of type X520SerialNumber
+
+id-at-serialNumber      AttributeType ::= { id-at 5 }
+
+X520SerialNumber ::=    PrintableString (SIZE (1..ub-serial-number))
+
+-- Naming attributes of type X520Pseudonym
+
+id-at-pseudonym         AttributeType ::= { id-at 65 }
+
+-- Naming attributes of type X520Pseudonym:
+--   X520Pseudonym ::= DirectoryName (SIZE (1..ub-pseudonym))
+--
+-- Expanded to avoid parameterized type:
+X520Pseudonym ::= CHOICE {
+   teletexString     TeletexString   (SIZE (1..ub-pseudonym)),
+   printableString   PrintableString (SIZE (1..ub-pseudonym)),
+   universalString   UniversalString (SIZE (1..ub-pseudonym)),
+   utf8String        UTF8String      (SIZE (1..ub-pseudonym)),
+   bmpString         BMPString       (SIZE (1..ub-pseudonym)) }
+
+-- Naming attributes of type DomainComponent (from RFC 4519)
+
+id-domainComponent   AttributeType ::= { 0 9 2342 19200300 100 1 25 }
+
+DomainComponent ::=  IA5String
+
+-- Legacy attributes
+
+pkcs-9 OBJECT IDENTIFIER ::=
+       { iso(1) member-body(2) us(840) rsadsi(113549) pkcs(1) 9 }
+
+id-emailAddress      AttributeType ::= { pkcs-9 1 }
+
+EmailAddress ::=     IA5String (SIZE (1..ub-emailaddress-length))
+
+-- naming data types --
+
+Name ::= CHOICE { -- only one possibility for now --
+      rdnSequence  RDNSequence }
+
+RDNSequence ::= SEQUENCE OF RelativeDistinguishedName
+
+DistinguishedName ::=   RDNSequence
+
+RelativeDistinguishedName ::= SET SIZE (1..MAX) OF AttributeTypeAndValue
+
+-- Directory string type --
+
+DirectoryString ::= CHOICE {
+      teletexString       TeletexString   (SIZE (1..MAX)),
+      printableString     PrintableString (SIZE (1..MAX)),
+      universalString     UniversalString (SIZE (1..MAX)),
+      utf8String          UTF8String      (SIZE (1..MAX)),
+      bmpString           BMPString       (SIZE (1..MAX)) }
+
+-- certificate and CRL specific structures begin here
+
+Certificate  ::=  SEQUENCE  {
+     tbsCertificate       TBSCertificate,
+     signatureAlgorithm   AlgorithmIdentifier,
+     signature            BIT STRING  }
+
+TBSCertificate  ::=  SEQUENCE  {
+     version         [0]  Version DEFAULT v1,
+     serialNumber         CertificateSerialNumber,
+     signature            AlgorithmIdentifier,
+     issuer               Name,
+     validity             Validity,
+     subject              Name,
+     subjectPublicKeyInfo SubjectPublicKeyInfo,
+     issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
+                          -- If present, version MUST be v2 or v3
+     subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
+                          -- If present, version MUST be v2 or v3
+     extensions      [3]  Extensions OPTIONAL
+                          -- If present, version MUST be v3 --  }
+
+Version  ::=  INTEGER  {  v1(0), v2(1), v3(2)  }
+
+CertificateSerialNumber  ::=  INTEGER
+
+Validity ::= SEQUENCE {
+     notBefore      Time,
+     notAfter       Time  }
+
+Time ::= CHOICE {
+     utcTime        UTCTime,
+     generalTime    GeneralizedTime }
+
+UniqueIdentifier  ::=  BIT STRING
+
+SubjectPublicKeyInfo  ::=  SEQUENCE  {
+     algorithm            AlgorithmIdentifier,
+     subjectPublicKey     BIT STRING  }
+
+Extensions  ::=  SEQUENCE SIZE (1..MAX) OF Extension
+
+Extension  ::=  SEQUENCE  {
+     extnID      OBJECT IDENTIFIER,
+     critical    BOOLEAN DEFAULT FALSE,
+     extnValue   OCTET STRING
+                 -- contains the DER encoding of an ASN.1 value
+                 -- corresponding to the extension type identified
+                 -- by extnID
+     }
+
+-- CRL structures
+
+CertificateList  ::=  SEQUENCE  {
+     tbsCertList          TBSCertList,
+     signatureAlgorithm   AlgorithmIdentifier,
+     signature            BIT STRING  }
+
+TBSCertList  ::=  SEQUENCE  {
+     version                 Version OPTIONAL,
+                                   -- if present, MUST be v2
+     signature               AlgorithmIdentifier,
+     issuer                  Name,
+     thisUpdate              Time,
+     nextUpdate              Time OPTIONAL,
+     revokedCertificates     SEQUENCE OF SEQUENCE  {
+          userCertificate         CertificateSerialNumber,
+          revocationDate          Time,
+          crlEntryExtensions      Extensions OPTIONAL
+                                   -- if present, version MUST be v2
+                               }  OPTIONAL,
+     crlExtensions           [0] Extensions OPTIONAL }
+                                   -- if present, version MUST be v2
+
+-- Version, Time, CertificateSerialNumber, and Extensions were
+-- defined earlier for use in the certificate structure
+
+AlgorithmIdentifier  ::=  SEQUENCE  {
+     algorithm               OBJECT IDENTIFIER,
+     parameters              ANY DEFINED BY algorithm OPTIONAL  }
+                                -- contains a value of the type
+                                -- registered for use with the
+                                -- algorithm object identifier value
+
+-- X.400 address syntax starts here
+
+ORAddress ::= SEQUENCE {
+   built-in-standard-attributes BuiltInStandardAttributes,
+   built-in-domain-defined-attributes
+                   BuiltInDomainDefinedAttributes OPTIONAL,
+   -- see also teletex-domain-defined-attributes
+   extension-attributes ExtensionAttributes OPTIONAL }
+
+-- Built-in Standard Attributes
+
+BuiltInStandardAttributes ::= SEQUENCE {
+   country-name                  CountryName OPTIONAL,
+   administration-domain-name    AdministrationDomainName OPTIONAL,
+   network-address           [0] IMPLICIT NetworkAddress OPTIONAL,
+     -- see also extended-network-address
+   terminal-identifier       [1] IMPLICIT TerminalIdentifier OPTIONAL,
+   private-domain-name       [2] PrivateDomainName OPTIONAL,
+   organization-name         [3] IMPLICIT OrganizationName OPTIONAL,
+     -- see also teletex-organization-name
+   numeric-user-identifier   [4] IMPLICIT NumericUserIdentifier
+                                 OPTIONAL,
+   personal-name             [5] IMPLICIT PersonalName OPTIONAL,
+     -- see also teletex-personal-name
+   organizational-unit-names [6] IMPLICIT OrganizationalUnitNames
+                                 OPTIONAL }
+     -- see also teletex-organizational-unit-names
+
+CountryName ::= [APPLICATION 1] CHOICE {
+   x121-dcc-code         NumericString
+                           (SIZE (ub-country-name-numeric-length)),
+   iso-3166-alpha2-code  PrintableString
+                           (SIZE (ub-country-name-alpha-length)) }
+
+AdministrationDomainName ::= [APPLICATION 2] CHOICE {
+   numeric   NumericString   (SIZE (0..ub-domain-name-length)),
+   printable PrintableString (SIZE (0..ub-domain-name-length)) }
+
+NetworkAddress ::= X121Address  -- see also extended-network-address
+
+X121Address ::= NumericString (SIZE (1..ub-x121-address-length))
+
+TerminalIdentifier ::= PrintableString (SIZE (1..ub-terminal-id-length))
+
+PrivateDomainName ::= CHOICE {
+   numeric   NumericString   (SIZE (1..ub-domain-name-length)),
+   printable PrintableString (SIZE (1..ub-domain-name-length)) }
+
+OrganizationName ::= PrintableString
+                            (SIZE (1..ub-organization-name-length))
+  -- see also teletex-organization-name
+
+NumericUserIdentifier ::= NumericString
+                            (SIZE (1..ub-numeric-user-id-length))
+
+PersonalName ::= SET {
+   surname     [0] IMPLICIT PrintableString
+                    (SIZE (1..ub-surname-length)),
+   given-name  [1] IMPLICIT PrintableString
+                    (SIZE (1..ub-given-name-length)) OPTIONAL,
+   initials    [2] IMPLICIT PrintableString
+                    (SIZE (1..ub-initials-length)) OPTIONAL,
+   generation-qualifier [3] IMPLICIT PrintableString
+                    (SIZE (1..ub-generation-qualifier-length))
+                    OPTIONAL }
+  -- see also teletex-personal-name
+
+OrganizationalUnitNames ::= SEQUENCE SIZE (1..ub-organizational-units)
+                             OF OrganizationalUnitName
+  -- see also teletex-organizational-unit-names
+
+OrganizationalUnitName ::= PrintableString (SIZE
+                    (1..ub-organizational-unit-name-length))
+
+-- Built-in Domain-defined Attributes
+
+BuiltInDomainDefinedAttributes ::= SEQUENCE SIZE
+                    (1..ub-domain-defined-attributes) OF
+                    BuiltInDomainDefinedAttribute
+
+BuiltInDomainDefinedAttribute ::= SEQUENCE {
+   type PrintableString (SIZE
+                   (1..ub-domain-defined-attribute-type-length)),
+   value PrintableString (SIZE
+                   (1..ub-domain-defined-attribute-value-length)) }
+
+-- Extension Attributes
+
+ExtensionAttributes ::= SET SIZE (1..ub-extension-attributes) OF
+               ExtensionAttribute
+
+ExtensionAttribute ::=  SEQUENCE {
+   extension-attribute-type [0] IMPLICIT INTEGER
+                   (0..ub-extension-attributes),
+   extension-attribute-value [1]
+                   ANY DEFINED BY extension-attribute-type }
+
+-- Extension types and attribute values
+
+common-name INTEGER ::= 1
+
+CommonName ::= PrintableString (SIZE (1..ub-common-name-length))
+
+teletex-common-name INTEGER ::= 2
+
+TeletexCommonName ::= TeletexString (SIZE (1..ub-common-name-length))
+
+teletex-organization-name INTEGER ::= 3
+
+TeletexOrganizationName ::=
+                TeletexString (SIZE (1..ub-organization-name-length))
+
+teletex-personal-name INTEGER ::= 4
+
+TeletexPersonalName ::= SET {
+   surname     [0] IMPLICIT TeletexString
+                    (SIZE (1..ub-surname-length)),
+   given-name  [1] IMPLICIT TeletexString
+                    (SIZE (1..ub-given-name-length)) OPTIONAL,
+   initials    [2] IMPLICIT TeletexString
+                    (SIZE (1..ub-initials-length)) OPTIONAL,
+   generation-qualifier [3] IMPLICIT TeletexString
+                    (SIZE (1..ub-generation-qualifier-length))
+                    OPTIONAL }
+
+teletex-organizational-unit-names INTEGER ::= 5
+
+TeletexOrganizationalUnitNames ::= SEQUENCE SIZE
+      (1..ub-organizational-units) OF TeletexOrganizationalUnitName
+
+TeletexOrganizationalUnitName ::= TeletexString
+                  (SIZE (1..ub-organizational-unit-name-length))
+
+pds-name INTEGER ::= 7
+
+PDSName ::= PrintableString (SIZE (1..ub-pds-name-length))
+
+physical-delivery-country-name INTEGER ::= 8
+
+PhysicalDeliveryCountryName ::= CHOICE {
+   x121-dcc-code NumericString (SIZE (ub-country-name-numeric-length)),
+   iso-3166-alpha2-code PrintableString
+                               (SIZE (ub-country-name-alpha-length)) }
+
+postal-code INTEGER ::= 9
+
+PostalCode ::= CHOICE {
+   numeric-code   NumericString (SIZE (1..ub-postal-code-length)),
+   printable-code PrintableString (SIZE (1..ub-postal-code-length)) }
+
+physical-delivery-office-name INTEGER ::= 10
+
+PhysicalDeliveryOfficeName ::= PDSParameter
+
+physical-delivery-office-number INTEGER ::= 11
+
+PhysicalDeliveryOfficeNumber ::= PDSParameter
+
+extension-OR-address-components INTEGER ::= 12
+
+ExtensionORAddressComponents ::= PDSParameter
+
+physical-delivery-personal-name INTEGER ::= 13
+
+PhysicalDeliveryPersonalName ::= PDSParameter
+
+physical-delivery-organization-name INTEGER ::= 14
+
+PhysicalDeliveryOrganizationName ::= PDSParameter
+
+extension-physical-delivery-address-components INTEGER ::= 15
+
+ExtensionPhysicalDeliveryAddressComponents ::= PDSParameter
+
+unformatted-postal-address INTEGER ::= 16
+
+UnformattedPostalAddress ::= SET {
+   printable-address SEQUENCE SIZE (1..ub-pds-physical-address-lines)
+        OF PrintableString (SIZE (1..ub-pds-parameter-length)) OPTIONAL,
+   teletex-string TeletexString
+        (SIZE (1..ub-unformatted-address-length)) OPTIONAL }
+
+street-address INTEGER ::= 17
+
+StreetAddress ::= PDSParameter
+
+post-office-box-address INTEGER ::= 18
+
+PostOfficeBoxAddress ::= PDSParameter
+
+poste-restante-address INTEGER ::= 19
+
+PosteRestanteAddress ::= PDSParameter
+
+unique-postal-name INTEGER ::= 20
+
+UniquePostalName ::= PDSParameter
+
+local-postal-attributes INTEGER ::= 21
+
+LocalPostalAttributes ::= PDSParameter
+
+PDSParameter ::= SET {
+   printable-string PrintableString
+                (SIZE(1..ub-pds-parameter-length)) OPTIONAL,
+   teletex-string TeletexString
+                (SIZE(1..ub-pds-parameter-length)) OPTIONAL }
+
+extended-network-address INTEGER ::= 22
+
+ExtendedNetworkAddress ::= CHOICE {
+   e163-4-address SEQUENCE {
+      number      [0] IMPLICIT NumericString
+                       (SIZE (1..ub-e163-4-number-length)),
+      sub-address [1] IMPLICIT NumericString
+                       (SIZE (1..ub-e163-4-sub-address-length))
+                       OPTIONAL },
+   psap-address   [0] IMPLICIT PresentationAddress }
+
+PresentationAddress ::= SEQUENCE {
+    pSelector     [0] EXPLICIT OCTET STRING OPTIONAL,
+    sSelector     [1] EXPLICIT OCTET STRING OPTIONAL,
+    tSelector     [2] EXPLICIT OCTET STRING OPTIONAL,
+    nAddresses    [3] EXPLICIT SET SIZE (1..MAX) OF OCTET STRING }
+
+terminal-type  INTEGER ::= 23
+
+TerminalType ::= INTEGER {
+   telex        (3),
+   teletex      (4),
+   g3-facsimile (5),
+   g4-facsimile (6),
+   ia5-terminal (7),
+   videotex     (8) } (0..ub-integer-options)
+
+-- Extension Domain-defined Attributes
+
+teletex-domain-defined-attributes INTEGER ::= 6
+
+TeletexDomainDefinedAttributes ::= SEQUENCE SIZE
+   (1..ub-domain-defined-attributes) OF TeletexDomainDefinedAttribute
+
+TeletexDomainDefinedAttribute ::= SEQUENCE {
+        type TeletexString
+               (SIZE (1..ub-domain-defined-attribute-type-length)),
+        value TeletexString
+               (SIZE (1..ub-domain-defined-attribute-value-length)) }
+
+--  specifications of Upper Bounds MUST be regarded as mandatory
+--  from Annex B of ITU-T X.411 Reference Definition of MTS Parameter
+--  Upper Bounds
+
+-- Upper Bounds
+ub-name INTEGER ::= 32768
+ub-common-name INTEGER ::= 64
+ub-locality-name INTEGER ::= 128
+ub-state-name INTEGER ::= 128
+ub-organization-name INTEGER ::= 64
+ub-organizational-unit-name INTEGER ::= 64
+ub-title INTEGER ::= 64
+ub-serial-number INTEGER ::= 64
+ub-match INTEGER ::= 128
+ub-emailaddress-length INTEGER ::= 255
+ub-common-name-length INTEGER ::= 64
+ub-country-name-alpha-length INTEGER ::= 2
+ub-country-name-numeric-length INTEGER ::= 3
+ub-domain-defined-attributes INTEGER ::= 4
+ub-domain-defined-attribute-type-length INTEGER ::= 8
+ub-domain-defined-attribute-value-length INTEGER ::= 128
+ub-domain-name-length INTEGER ::= 16
+ub-extension-attributes INTEGER ::= 256
+ub-e163-4-number-length INTEGER ::= 15
+ub-e163-4-sub-address-length INTEGER ::= 40
+ub-generation-qualifier-length INTEGER ::= 3
+ub-given-name-length INTEGER ::= 16
+ub-initials-length INTEGER ::= 5
+ub-integer-options INTEGER ::= 256
+ub-numeric-user-id-length INTEGER ::= 32
+ub-organization-name-length INTEGER ::= 64
+ub-organizational-unit-name-length INTEGER ::= 32
+ub-organizational-units INTEGER ::= 4
+ub-pds-name-length INTEGER ::= 16
+ub-pds-parameter-length INTEGER ::= 30
+ub-pds-physical-address-lines INTEGER ::= 6
+ub-postal-code-length INTEGER ::= 16
+ub-pseudonym INTEGER ::= 128
+ub-surname-length INTEGER ::= 40
+ub-terminal-id-length INTEGER ::= 24
+ub-unformatted-address-length INTEGER ::= 180
+ub-x121-address-length INTEGER ::= 16
+
+-- Note - upper bounds on string types, such as TeletexString, are
+-- measured in characters.  Excepting PrintableString or IA5String, a
+-- significantly greater number of octets will be required to hold
+-- such a value.  As a minimum, 16 octets, or twice the specified
+-- upper bound, whichever is the larger, should be allowed for
+-- TeletexString.  For UTF8String or UniversalString at least four
+-- times the upper bound should be allowed.
+
+END
+

pySim/esim/asn1/rsp/PKIX1Implicit88.asn

@@ -0,0 +1,343 @@
+PKIX1Implicit88 { iso(1) identified-organization(3) dod(6) internet(1)
+  security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-implicit(19) }
+
+DEFINITIONS IMPLICIT TAGS ::=
+
+BEGIN
+
+-- EXPORTS ALL --
+
+IMPORTS
+      id-pe, id-kp, id-qt-unotice, id-qt-cps,
+      ORAddress, Name, RelativeDistinguishedName,
+      CertificateSerialNumber, Attribute, DirectoryString
+      FROM PKIX1Explicit88 { iso(1) identified-organization(3)
+            dod(6) internet(1) security(5) mechanisms(5) pkix(7)
+            id-mod(0) id-pkix1-explicit(18) };
+
+-- ISO arc for standard certificate and CRL extensions
+
+id-ce OBJECT IDENTIFIER  ::=  {joint-iso-ccitt(2) ds(5) 29}
+
+-- authority key identifier OID and syntax
+
+id-ce-authorityKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 35 }
+
+AuthorityKeyIdentifier ::= SEQUENCE {
+    keyIdentifier             [0] KeyIdentifier            OPTIONAL,
+    authorityCertIssuer       [1] GeneralNames             OPTIONAL,
+    authorityCertSerialNumber [2] CertificateSerialNumber  OPTIONAL }
+    -- authorityCertIssuer and authorityCertSerialNumber MUST both
+    -- be present or both be absent
+
+KeyIdentifier ::= OCTET STRING
+
+-- subject key identifier OID and syntax
+
+id-ce-subjectKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 14 }
+
+SubjectKeyIdentifier ::= KeyIdentifier
+
+-- key usage extension OID and syntax
+
+id-ce-keyUsage OBJECT IDENTIFIER ::=  { id-ce 15 }
+
+KeyUsage ::= BIT STRING {
+     digitalSignature        (0),
+     nonRepudiation          (1),  -- recent editions of X.509 have
+                                -- renamed this bit to contentCommitment
+     keyEncipherment         (2),
+     dataEncipherment        (3),
+     keyAgreement            (4),
+     keyCertSign             (5),
+     cRLSign                 (6),
+     encipherOnly            (7),
+     decipherOnly            (8) }
+
+-- private key usage period extension OID and syntax
+
+id-ce-privateKeyUsagePeriod OBJECT IDENTIFIER ::=  { id-ce 16 }
+
+PrivateKeyUsagePeriod ::= SEQUENCE {
+     notBefore       [0]     GeneralizedTime OPTIONAL,
+     notAfter        [1]     GeneralizedTime OPTIONAL }
+     -- either notBefore or notAfter MUST be present
+
+-- certificate policies extension OID and syntax
+
+id-ce-certificatePolicies OBJECT IDENTIFIER ::=  { id-ce 32 }
+
+anyPolicy OBJECT IDENTIFIER ::= { id-ce-certificatePolicies 0 }
+
+CertificatePolicies ::= SEQUENCE SIZE (1..MAX) OF PolicyInformation
+
+PolicyInformation ::= SEQUENCE {
+     policyIdentifier   CertPolicyId,
+     policyQualifiers   SEQUENCE SIZE (1..MAX) OF
+             PolicyQualifierInfo OPTIONAL }
+
+CertPolicyId ::= OBJECT IDENTIFIER
+
+PolicyQualifierInfo ::= SEQUENCE {
+     policyQualifierId  PolicyQualifierId,
+     qualifier          ANY DEFINED BY policyQualifierId }
+
+-- Implementations that recognize additional policy qualifiers MUST
+-- augment the following definition for PolicyQualifierId
+
+PolicyQualifierId ::= OBJECT IDENTIFIER ( id-qt-cps | id-qt-unotice )
+
+-- CPS pointer qualifier
+
+CPSuri ::= IA5String
+
+-- user notice qualifier
+
+UserNotice ::= SEQUENCE {
+     noticeRef        NoticeReference OPTIONAL,
+     explicitText     DisplayText OPTIONAL }
+
+NoticeReference ::= SEQUENCE {
+     organization     DisplayText,
+     noticeNumbers    SEQUENCE OF INTEGER }
+
+DisplayText ::= CHOICE {
+     ia5String        IA5String      (SIZE (1..200)),
+     visibleString    VisibleString  (SIZE (1..200)),
+     bmpString        BMPString      (SIZE (1..200)),
+     utf8String       UTF8String     (SIZE (1..200)) }
+
+-- policy mapping extension OID and syntax
+
+id-ce-policyMappings OBJECT IDENTIFIER ::=  { id-ce 33 }
+
+PolicyMappings ::= SEQUENCE SIZE (1..MAX) OF SEQUENCE {
+     issuerDomainPolicy      CertPolicyId,
+     subjectDomainPolicy     CertPolicyId }
+
+-- subject alternative name extension OID and syntax
+
+id-ce-subjectAltName OBJECT IDENTIFIER ::=  { id-ce 17 }
+
+SubjectAltName ::= GeneralNames
+
+GeneralNames ::= SEQUENCE SIZE (1..MAX) OF GeneralName
+
+GeneralName ::= CHOICE {
+     otherName                 [0]  AnotherName,
+     rfc822Name                [1]  IA5String,
+     dNSName                   [2]  IA5String,
+     x400Address               [3]  ORAddress,
+     directoryName             [4]  Name,
+     ediPartyName              [5]  EDIPartyName,
+     uniformResourceIdentifier [6]  IA5String,
+     iPAddress                 [7]  OCTET STRING,
+     registeredID              [8]  OBJECT IDENTIFIER }
+
+-- AnotherName replaces OTHER-NAME ::= TYPE-IDENTIFIER, as
+-- TYPE-IDENTIFIER is not supported in the '88 ASN.1 syntax
+
+AnotherName ::= SEQUENCE {
+     type-id    OBJECT IDENTIFIER,
+     value      [0] EXPLICIT ANY DEFINED BY type-id }
+
+EDIPartyName ::= SEQUENCE {
+     nameAssigner              [0]  DirectoryString OPTIONAL,
+     partyName                 [1]  DirectoryString }
+
+-- issuer alternative name extension OID and syntax
+
+id-ce-issuerAltName OBJECT IDENTIFIER ::=  { id-ce 18 }
+
+IssuerAltName ::= GeneralNames
+
+id-ce-subjectDirectoryAttributes OBJECT IDENTIFIER ::=  { id-ce 9 }
+
+SubjectDirectoryAttributes ::= SEQUENCE SIZE (1..MAX) OF Attribute
+
+-- basic constraints extension OID and syntax
+
+id-ce-basicConstraints OBJECT IDENTIFIER ::=  { id-ce 19 }
+
+BasicConstraints ::= SEQUENCE {
+     cA                      BOOLEAN DEFAULT FALSE,
+     pathLenConstraint       INTEGER (0..MAX) OPTIONAL }
+
+-- name constraints extension OID and syntax
+
+id-ce-nameConstraints OBJECT IDENTIFIER ::=  { id-ce 30 }
+
+NameConstraints ::= SEQUENCE {
+     permittedSubtrees       [0]     GeneralSubtrees OPTIONAL,
+     excludedSubtrees        [1]     GeneralSubtrees OPTIONAL }
+
+GeneralSubtrees ::= SEQUENCE SIZE (1..MAX) OF GeneralSubtree
+
+GeneralSubtree ::= SEQUENCE {
+     base                    GeneralName,
+     minimum         [0]     BaseDistance DEFAULT 0,
+     maximum         [1]     BaseDistance OPTIONAL }
+
+BaseDistance ::= INTEGER (0..MAX)
+
+-- policy constraints extension OID and syntax
+
+id-ce-policyConstraints OBJECT IDENTIFIER ::=  { id-ce 36 }
+
+PolicyConstraints ::= SEQUENCE {
+     requireExplicitPolicy   [0]     SkipCerts OPTIONAL,
+     inhibitPolicyMapping    [1]     SkipCerts OPTIONAL }
+
+SkipCerts ::= INTEGER (0..MAX)
+
+-- CRL distribution points extension OID and syntax
+
+id-ce-cRLDistributionPoints     OBJECT IDENTIFIER  ::=  {id-ce 31}
+
+CRLDistributionPoints ::= SEQUENCE SIZE (1..MAX) OF DistributionPoint
+
+DistributionPoint ::= SEQUENCE {
+     distributionPoint       [0]     DistributionPointName OPTIONAL,
+     reasons                 [1]     ReasonFlags OPTIONAL,
+     cRLIssuer               [2]     GeneralNames OPTIONAL }
+
+DistributionPointName ::= CHOICE {
+     fullName                [0]     GeneralNames,
+     nameRelativeToCRLIssuer [1]     RelativeDistinguishedName }
+
+ReasonFlags ::= BIT STRING {
+     unused                  (0),
+     keyCompromise           (1),
+     cACompromise            (2),
+     affiliationChanged      (3),
+     superseded              (4),
+     cessationOfOperation    (5),
+     certificateHold         (6),
+     privilegeWithdrawn      (7),
+     aACompromise            (8) }
+
+-- extended key usage extension OID and syntax
+
+id-ce-extKeyUsage OBJECT IDENTIFIER ::= {id-ce 37}
+
+ExtKeyUsageSyntax ::= SEQUENCE SIZE (1..MAX) OF KeyPurposeId
+
+KeyPurposeId ::= OBJECT IDENTIFIER
+
+-- permit unspecified key uses
+
+anyExtendedKeyUsage OBJECT IDENTIFIER ::= { id-ce-extKeyUsage 0 }
+
+-- extended key purpose OIDs
+
+id-kp-serverAuth             OBJECT IDENTIFIER ::= { id-kp 1 }
+id-kp-clientAuth             OBJECT IDENTIFIER ::= { id-kp 2 }
+id-kp-codeSigning            OBJECT IDENTIFIER ::= { id-kp 3 }
+id-kp-emailProtection        OBJECT IDENTIFIER ::= { id-kp 4 }
+id-kp-timeStamping           OBJECT IDENTIFIER ::= { id-kp 8 }
+id-kp-OCSPSigning            OBJECT IDENTIFIER ::= { id-kp 9 }
+
+-- inhibit any policy OID and syntax
+
+id-ce-inhibitAnyPolicy OBJECT IDENTIFIER ::=  { id-ce 54 }
+
+InhibitAnyPolicy ::= SkipCerts
+
+-- freshest (delta)CRL extension OID and syntax
+
+id-ce-freshestCRL OBJECT IDENTIFIER ::=  { id-ce 46 }
+
+FreshestCRL ::= CRLDistributionPoints
+
+-- authority info access
+
+id-pe-authorityInfoAccess OBJECT IDENTIFIER ::= { id-pe 1 }
+
+AuthorityInfoAccessSyntax  ::=
+        SEQUENCE SIZE (1..MAX) OF AccessDescription
+
+AccessDescription  ::=  SEQUENCE {
+        accessMethod          OBJECT IDENTIFIER,
+        accessLocation        GeneralName  }
+
+-- subject info access
+
+id-pe-subjectInfoAccess OBJECT IDENTIFIER ::= { id-pe 11 }
+
+SubjectInfoAccessSyntax  ::=
+        SEQUENCE SIZE (1..MAX) OF AccessDescription
+
+-- CRL number extension OID and syntax
+
+id-ce-cRLNumber OBJECT IDENTIFIER ::= { id-ce 20 }
+
+CRLNumber ::= INTEGER (0..MAX)
+
+-- issuing distribution point extension OID and syntax
+
+id-ce-issuingDistributionPoint OBJECT IDENTIFIER ::= { id-ce 28 }
+
+IssuingDistributionPoint ::= SEQUENCE {
+     distributionPoint          [0] DistributionPointName OPTIONAL,
+     onlyContainsUserCerts      [1] BOOLEAN DEFAULT FALSE,
+     onlyContainsCACerts        [2] BOOLEAN DEFAULT FALSE,
+     onlySomeReasons            [3] ReasonFlags OPTIONAL,
+     indirectCRL                [4] BOOLEAN DEFAULT FALSE,
+     onlyContainsAttributeCerts [5] BOOLEAN DEFAULT FALSE }
+     -- at most one of onlyContainsUserCerts, onlyContainsCACerts,
+     -- and onlyContainsAttributeCerts may be set to TRUE.
+
+id-ce-deltaCRLIndicator OBJECT IDENTIFIER ::= { id-ce 27 }
+
+BaseCRLNumber ::= CRLNumber
+
+-- reason code extension OID and syntax
+
+id-ce-cRLReasons OBJECT IDENTIFIER ::= { id-ce 21 }
+
+CRLReason ::= ENUMERATED {
+     unspecified             (0),
+     keyCompromise           (1),
+     cACompromise            (2),
+     affiliationChanged      (3),
+     superseded              (4),
+     cessationOfOperation    (5),
+     certificateHold         (6),
+     removeFromCRL           (8),
+     privilegeWithdrawn      (9),
+     aACompromise           (10) }
+
+-- certificate issuer CRL entry extension OID and syntax
+
+id-ce-certificateIssuer OBJECT IDENTIFIER ::= { id-ce 29 }
+
+CertificateIssuer ::= GeneralNames
+
+-- hold instruction extension OID and syntax
+
+id-ce-holdInstructionCode OBJECT IDENTIFIER ::= { id-ce 23 }
+
+HoldInstructionCode ::= OBJECT IDENTIFIER
+
+-- ANSI x9 arc holdinstruction arc
+
+holdInstruction OBJECT IDENTIFIER ::=
+          {joint-iso-itu-t(2) member-body(2) us(840) x9cm(10040) 2}
+
+-- ANSI X9 holdinstructions
+
+id-holdinstruction-none OBJECT IDENTIFIER  ::=
+                                      {holdInstruction 1} -- deprecated
+
+id-holdinstruction-callissuer OBJECT IDENTIFIER ::= {holdInstruction 2}
+
+id-holdinstruction-reject OBJECT IDENTIFIER ::= {holdInstruction 3}
+
+-- invalidity date CRL entry extension OID and syntax
+
+id-ce-invalidityDate OBJECT IDENTIFIER ::= { id-ce 24 }
+
+InvalidityDate ::=  GeneralizedTime
+
+END
+

pySim/esim/asn1/rsp/rsp.asn

@@ -0,0 +1,785 @@
+RSPDefinitions {joint-iso-itu-t(2) international-organizations(23) gsma(146) rsp(1) spec-version(1) version-two(2)}
+DEFINITIONS
+AUTOMATIC TAGS
+EXTENSIBILITY IMPLIED ::=
+BEGIN
+ 
+IMPORTS Certificate, CertificateList, Time FROM PKIX1Explicit88 {iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-explicit(18)}
+SubjectKeyIdentifier FROM PKIX1Implicit88 {iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-implicit(19)};
+ 
+id-rsp OBJECT IDENTIFIER ::= {joint-iso-itu-t(2) international-organizations(23) gsma(146) rsp(1)}
+ 
+-- Basic types, for size constraints
+Octet8 ::= OCTET STRING (SIZE(8))
+Octet16 ::= OCTET STRING (SIZE(16))
+OctetTo16 ::= OCTET STRING (SIZE(1..16))
+Octet32 ::= OCTET STRING (SIZE(32))
+Octet1 ::= OCTET STRING(SIZE(1))
+Octet2 ::= OCTET STRING (SIZE(2))
+VersionType ::= OCTET STRING(SIZE(3)) -- major/minor/revision version are coded as binary value on byte 1/2/3, e.g. '02 00 0C' for v2.0.12.
+Iccid ::= [APPLICATION 26] OCTET STRING (SIZE(10)) -- ICCID as coded in EFiccid, corresponding tag is '5A'
+RemoteOpId ::= [2] INTEGER {installBoundProfilePackage(1)}
+TransactionId ::= OCTET STRING (SIZE(1..16))
+ 
+-- Definition of EUICCInfo1 --------------------------
+GetEuiccInfo1Request ::= [32] SEQUENCE { -- Tag 'BF20'
+}
+ 
+EUICCInfo1 ::= [32] SEQUENCE { -- Tag 'BF20'
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    euiccCiPKIdListForVerification [9] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifiers supported on the eUICC for signature verification
+    euiccCiPKIdListForSigning [10] SEQUENCE OF SubjectKeyIdentifier -- List of CI Public Key Identifier supported on the eUICC for signature creation
+}
+ 
+-- Definition of EUICCInfo2 --------------------------
+GetEuiccInfo2Request ::= [34] SEQUENCE { -- Tag 'BF22'
+}
+ 
+EUICCInfo2 ::= [34] SEQUENCE { -- Tag 'BF22'
+    profileVersion [1] VersionType,     -- SIMAlliance Profile package version supported
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    euiccFirmwareVer [3] VersionType,   -- eUICC Firmware version
+    extCardResource [4] OCTET STRING,   -- Extended Card Resource Information according to ETSI TS 102 226
+    uiccCapability [5] UICCCapability,
+    javacardVersion [6] VersionType OPTIONAL,
+    globalplatformVersion [7] VersionType OPTIONAL,
+    rspCapability [8] RspCapability,
+    euiccCiPKIdListForVerification [9] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifiers supported on the eUICC for signature verification
+    euiccCiPKIdListForSigning [10] SEQUENCE OF SubjectKeyIdentifier, -- List of CI Public Key Identifier supported on the eUICC for signature creation
+    euiccCategory [11] INTEGER {
+        other(0),
+        basicEuicc(1),
+        mediumEuicc(2),
+        contactlessEuicc(3)
+    } OPTIONAL,
+    forbiddenProfilePolicyRules [25] PprIds OPTIONAL, -- Tag '99'
+    ppVersion VersionType, -- Protection Profile version
+    sasAcreditationNumber UTF8String (SIZE(0..64)),
+    certificationDataObject [12] CertificationDataObject OPTIONAL
+}
+ 
+-- Definition of RspCapability
+RspCapability ::= BIT STRING {
+    additionalProfile(0), -- at least one more Profile can be installed
+    crlSupport(1), -- CRL
+    rpmSupport(2), -- Remote Profile Management
+    testProfileSupport (3) -- support for test profile
+}
+ 
+-- Definition of CertificationDataObject
+CertificationDataObject ::= SEQUENCE {
+    platformLabel UTF8String,   -- Platform_Label as defined in GlobalPlatform DLOA specification [57]
+    discoveryBaseURL UTF8String -- Discovery Base URL of the SE default DLOA Registrar as defined in GlobalPlatform DLOA specification [57]
+}
+ 
+CertificateInfo ::= BIT STRING {
+ 
+    reserved(0),    -- eUICC has a CERT.EUICC.ECDSA in GlobalPlatform format. The use of this bit is deprecated.
+    certSigningX509(1),     -- eUICC has a CERT.EUICC.ECDSA in X.509 format
+    rfu2(2),
+    rfu3(3),
+    reserved2(4), -- Handling of Certificate in GlobalPlatform format. The use of this bit is deprecated.
+    certVerificationX509(5)-- Handling of Certificate in X.509 format
+}
+ 
+-- Definition of UICCCapability
+UICCCapability ::= BIT STRING {
+/* Sequence is derived from ServicesList[] defined in SIMalliance PEDefinitions*/
+    contactlessSupport(0),  -- Contactless (SWP, HCI and associated APIs)
+    usimSupport(1),         -- USIM as defined by 3GPP
+    isimSupport(2),         -- ISIM as defined by 3GPP
+    csimSupport(3),         -- CSIM as defined by 3GPP2
+ 
+    akaMilenage(4),         -- Milenage as AKA algorithm
+    akaCave(5),             -- CAVE as authentication algorithm
+    akaTuak128(6),          -- TUAK as AKA algorithm with 128 bit key length
+    akaTuak256(7),          -- TUAK as AKA algorithm with 256 bit key length
+    rfu1(8),                    -- reserved for further algorithms
+    rfu2(9),                    -- reserved for further algorithms
+ 
+    gbaAuthenUsim(10),  -- GBA authentication in the context of USIM
+    gbaAuthenISim(11),  -- GBA authentication in the context of ISIM
+    mbmsAuthenUsim(12),     -- MBMS authentication in the context of USIM
+    eapClient(13),          -- EAP client
+ 
+    javacard(14),               -- Javacard support
+    multos(15),             -- Multos support
+ 
+    multipleUsimSupport(16),     -- Multiple USIM applications are supported within the same Profile
+    multipleIsimSupport(17),      -- Multiple ISIM applications are supported within the same Profile
+    multipleCsimSupport(18)   -- Multiple CSIM applications are supported within the same Profile
+}
+ 
+-- Definition of DeviceInfo
+DeviceInfo ::= SEQUENCE {
+    tac Octet8,
+    deviceCapabilities DeviceCapabilities,
+    imei Octet8 OPTIONAL
+}
+ 
+DeviceCapabilities ::= SEQUENCE { -- Highest fully supported release for each definition
+  -- The device SHALL set all the capabilities it supports
+    gsmSupportedRelease VersionType OPTIONAL,
+    utranSupportedRelease VersionType OPTIONAL,
+    cdma2000onexSupportedRelease VersionType OPTIONAL,
+    cdma2000hrpdSupportedRelease VersionType OPTIONAL,
+    cdma2000ehrpdSupportedRelease VersionType OPTIONAL,
+    eutranSupportedRelease VersionType OPTIONAL,
+    contactlessSupportedRelease VersionType OPTIONAL,
+    rspCrlSupportedVersion VersionType OPTIONAL,
+    rspRpmSupportedVersion VersionType OPTIONAL
+}
+ 
+ProfileInfoListRequest ::= [45] SEQUENCE { -- Tag 'BF2D'
+    searchCriteria [0] CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID of the ISD-P, tag '4F'
+        iccid Iccid, -- ICCID, tag '5A'
+        profileClass [21] ProfileClass -- Tag '95'
+    } OPTIONAL,
+    tagList [APPLICATION 28] OCTET STRING OPTIONAL -- tag '5C'
+}
+ 
+-- Definition of ProfileInfoList
+ProfileInfoListResponse ::= [45] CHOICE { -- Tag 'BF2D'
+    profileInfoListOk SEQUENCE OF ProfileInfo,
+    profileInfoListError ProfileInfoListError
+}
+ 
+ProfileInfo ::= [PRIVATE 3] SEQUENCE { -- Tag 'E3'
+    iccid Iccid OPTIONAL,
+    isdpAid [APPLICATION 15] OctetTo16 OPTIONAL, -- AID of the ISD-P containing the Profile, tag '4F'
+    profileState [112] ProfileState OPTIONAL, -- Tag '9F70'
+    profileNickname [16] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '90'
+    serviceProviderName [17] UTF8String (SIZE(0..32)) OPTIONAL, -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '92'
+    iconType [19] IconType OPTIONAL, -- Tag '93'
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94', see condition in ES10c:GetProfilesInfo
+    profileClass [21] ProfileClass DEFAULT operational, -- Tag '95'
+    notificationConfigurationInfo [22] SEQUENCE OF NotificationConfigurationInformation OPTIONAL, -- Tag 'B6'
+    profileOwner [23] OperatorID OPTIONAL, -- Tag 'B7'
+    dpProprietaryData [24] DpProprietaryData OPTIONAL, -- Tag 'B8'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+PprIds ::= BIT STRING {-- Definition of Profile Policy Rules identifiers
+    pprUpdateControl(0), -- defines how to update PPRs via ES6
+    ppr1(1), -- Indicator for PPR1 'Disabling of this Profile is not allowed'
+    ppr2(2), -- Indicator for PPR2 'Deletion of this Profile is not allowed'
+    ppr3(3)  -- Indicator for PPR3 'Deletion of this Profile is required upon its successful disabling'
+}
+ 
+OperatorID ::= SEQUENCE {
+    mccMnc OCTET STRING (SIZE(3)), -- MCC and MNC coded as defined in 3GPP TS 24.008 [32]
+    gid1 OCTET STRING OPTIONAL, -- referring to content of EF GID1 (file identifier '6F3E') as defined in 3GPP TS 31.102 [54]
+    gid2 OCTET STRING OPTIONAL -- referring to content of EF GID2 (file identifier '6F3F') as defined in 3GPP TS 31.102 [54]
+}
+ 
+ProfileInfoListError ::= INTEGER {incorrectInputValues(1), undefinedError(127)}
+ 
+-- Definition of StoreMetadata request
+ 
+StoreMetadataRequest ::= [37] SEQUENCE { -- Tag 'BF25'
+    iccid Iccid,
+    serviceProviderName [17] UTF8String (SIZE(0..32)), -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)), -- Tag '92' (corresponds to 'Short Description' defined in SGP.21 [2])
+    iconType [19] IconType OPTIONAL, -- Tag '93' (JPG or PNG)
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94'(Data of the icon. Size 64 x 64 pixel. This field SHALL only be present if iconType is present)
+    profileClass [21] ProfileClass OPTIONAL, -- Tag '95' (default if absent: 'operational')
+    notificationConfigurationInfo [22] SEQUENCE OF NotificationConfigurationInformation OPTIONAL,
+    profileOwner [23] OperatorID OPTIONAL, -- Tag 'B7'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+NotificationEvent ::= BIT STRING {
+    notificationInstall (0),
+    notificationEnable(1),
+    notificationDisable(2),
+    notificationDelete(3)
+}
+ 
+NotificationConfigurationInformation ::= SEQUENCE {
+    profileManagementOperation NotificationEvent,
+    notificationAddress UTF8String -- FQDN to forward the notification
+}
+ 
+IconType ::= INTEGER {jpg(0), png(1)}
+ProfileState ::= INTEGER {disabled(0), enabled(1)}
+ProfileClass ::= INTEGER {test(0), provisioning(1), operational(2)}
+ 
+-- Definition of UpdateMetadata request
+UpdateMetadataRequest ::= [42] SEQUENCE {  -- Tag 'BF2A'
+    serviceProviderName [17] UTF8String (SIZE(0..32)) OPTIONAL, -- Tag '91'
+    profileName [18] UTF8String (SIZE(0..64)) OPTIONAL, -- Tag '92'
+    iconType [19] IconType OPTIONAL, -- Tag '93'
+    icon [20] OCTET STRING (SIZE(0..1024)) OPTIONAL, -- Tag '94'
+    profilePolicyRules [25] PprIds OPTIONAL -- Tag '99'
+}
+ 
+-- Definition of data objects for command PrepareDownload -------------------------
+PrepareDownloadRequest ::= [33] SEQUENCE {  -- Tag 'BF21'
+    smdpSigned2 SmdpSigned2,            -- Signed information
+    smdpSignature2 [APPLICATION 55] OCTET STRING,   -- DP_Sign1, tag '5F37'
+    hashCc Octet32 OPTIONAL, -- Hash of confirmation code
+    smdpCertificate Certificate -- CERT.DPpb.ECDSA
+}
+ 
+SmdpSigned2 ::= SEQUENCE {
+    transactionId [0] TransactionId,    -- The TransactionID generated by the SM DP+
+    ccRequiredFlag BOOLEAN, --Indicates if the Confirmation Code is required
+    bppEuiccOtpk [APPLICATION 73] OCTET STRING OPTIONAL     -- otPK.EUICC.ECKA already used for binding the BPP, tag '5F49'
+}
+ 
+PrepareDownloadResponse ::= [33] CHOICE {  -- Tag 'BF21'
+    downloadResponseOk PrepareDownloadResponseOk,
+    downloadResponseError PrepareDownloadResponseError
+}
+ 
+PrepareDownloadResponseOk ::= SEQUENCE {
+    euiccSigned2 EUICCSigned2,  -- Signed information
+    euiccSignature2 [APPLICATION 55] OCTET STRING   -- tag '5F37'
+}
+ 
+EUICCSigned2 ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    euiccOtpk [APPLICATION 73] OCTET STRING,        -- otPK.EUICC.ECKA, tag '5F49'
+    hashCc Octet32 OPTIONAL         -- Hash of confirmation code
+}
+ 
+PrepareDownloadResponseError ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    downloadErrorCode DownloadErrorCode
+}
+ 
+DownloadErrorCode ::= INTEGER {invalidCertificate(1), invalidSignature(2), unsupportedCurve(3), noSessionContext(4), invalidTransactionId(5), undefinedError(127)}
+ 
+-- Definition of data objects for command AuthenticateServer--------------------
+AuthenticateServerRequest ::= [56] SEQUENCE { -- Tag 'BF38'
+    serverSigned1 ServerSigned1,                -- Signed information
+    serverSignature1 [APPLICATION 55] OCTET STRING,     -- tag ?5F37?
+    euiccCiPKIdToBeUsed SubjectKeyIdentifier,       -- CI Public Key Identifier to be used
+    serverCertificate Certificate, -- RSP Server Certificate CERT.XXauth.ECDSA
+    ctxParams1 CtxParams1
+}
+ 
+ServerSigned1 ::= SEQUENCE {
+    transactionId [0] TransactionId,        -- The Transaction ID generated by the RSP Server
+    euiccChallenge [1] Octet16,     -- The eUICC Challenge
+    serverAddress [3] UTF8String,   -- The RSP Server address
+    serverChallenge [4] Octet16     -- The RSP Server Challenge
+}
+ 
+CtxParams1 ::= CHOICE {
+    ctxParamsForCommonAuthentication CtxParamsForCommonAuthentication -- New contextual data objects may be defined for extensibility
+}
+ 
+CtxParamsForCommonAuthentication ::= SEQUENCE {
+    matchingId UTF8String OPTIONAL,-- The MatchingId could be the Activation code token or EventID or empty
+    deviceInfo DeviceInfo -- The Device information
+}
+ 
+AuthenticateServerResponse ::= [56] CHOICE { -- Tag 'BF38'
+    authenticateResponseOk AuthenticateResponseOk,
+    authenticateResponseError AuthenticateResponseError
+}
+ 
+AuthenticateResponseOk ::= SEQUENCE {
+    euiccSigned1 EuiccSigned1,      -- Signed information
+    euiccSignature1 [APPLICATION 55] OCTET STRING,  --EUICC_Sign1, tag 5F37
+    euiccCertificate Certificate,   -- eUICC Certificate (CERT.EUICC.ECDSA) signed by the EUM
+    eumCertificate Certificate  -- EUM Certificate (CERT.EUM.ECDSA) signed by the requested CI
+}
+ 
+EuiccSigned1 ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    serverAddress [3] UTF8String,
+    serverChallenge [4] Octet16,    -- The RSP Server Challenge
+    euiccInfo2 [34] EUICCInfo2,
+    ctxParams1 CtxParams1
+}
+ 
+AuthenticateResponseError ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    authenticateErrorCode AuthenticateErrorCode
+}
+ 
+AuthenticateErrorCode ::= INTEGER {invalidCertificate(1), invalidSignature(2), unsupportedCurve(3), noSessionContext(4), invalidOid(5), euiccChallengeMismatch(6), ciPKUnknown(7), undefinedError(127)}
+ 
+-- Definition of Cancel Session------------------------------
+CancelSessionRequest ::= [65] SEQUENCE { -- Tag 'BF41'
+    transactionId TransactionId,     -- The TransactionID generated by the RSP Server
+    reason CancelSessionReason
+}
+ 
+CancelSessionReason ::= INTEGER {endUserRejection(0), postponed(1), timeout(2), pprNotAllowed(3)}
+ 
+CancelSessionResponse ::= [65] CHOICE { -- Tag 'BF41'
+    cancelSessionResponseOk CancelSessionResponseOk,
+    cancelSessionResponseError INTEGER {invalidTransactionId(5), undefinedError(127)}
+}
+ 
+CancelSessionResponseOk ::= SEQUENCE {
+    euiccCancelSessionSigned EuiccCancelSessionSigned,  -- Signed information
+    euiccCancelSessionSignature [APPLICATION 55] OCTET STRING   -- tag '5F37
+}
+ 
+EuiccCancelSessionSigned ::= SEQUENCE {
+    transactionId TransactionId,
+    smdpOid OBJECT IDENTIFIER, -- SM-DP+ OID as contained in CERT.DPauth.ECDSA
+    reason CancelSessionReason
+}
+ 
+-- Definition of Bound Profile Package --------------------------
+BoundProfilePackage ::= [54] SEQUENCE { -- Tag 'BF36'
+    initialiseSecureChannelRequest [35] InitialiseSecureChannelRequest, -- Tag 'BF23'
+    firstSequenceOf87 [0] SEQUENCE OF [7] OCTET STRING, -- sequence of '87' TLVs
+    sequenceOf88 [1] SEQUENCE OF [8] OCTET STRING, -- sequence of '88' TLVs
+    secondSequenceOf87 [2] SEQUENCE OF [7] OCTET STRING OPTIONAL, -- sequence of '87' TLVs
+    sequenceOf86 [3] SEQUENCE OF [6] OCTET STRING -- sequence of '86' TLVs
+}
+ 
+-- Definition of Get eUICC Challenge --------------------------
+GetEuiccChallengeRequest ::= [46] SEQUENCE { -- Tag 'BF2E'
+}
+ 
+GetEuiccChallengeResponse ::= [46] SEQUENCE { -- Tag 'BF2E'
+    euiccChallenge Octet16  -- random eUICC challenge
+}
+ 
+-- Definition of Profile Installation Resulceipt
+ProfileInstallationResult ::= [55] SEQUENCE { -- Tag 'BF37'
+    profileInstallationResultData [39] ProfileInstallationResultData,
+    euiccSignPIR EuiccSignPIR
+}
+ 
+ProfileInstallationResultData ::= [39] SEQUENCE { -- Tag 'BF27'
+    transactionId[0] TransactionId, -- The TransactionID generated by the SM-DP+
+    notificationMetadata[47] NotificationMetadata,
+    smdpOid OBJECT IDENTIFIER OPTIONAL, -- SM-DP+ OID (same value as in CERT.DPpb.ECDSA)
+    finalResult [2] CHOICE {
+        successResult SuccessResult,
+        errorResult ErrorResult
+    }
+}
+ 
+EuiccSignPIR ::= [APPLICATION 55] OCTET STRING -- Tag '5F37', eUICC?s signature
+ 
+SuccessResult ::= SEQUENCE {
+    aid [APPLICATION 15] OCTET STRING (SIZE (5..16)), -- AID of ISD-P
+    simaResponse OCTET STRING -- contains (multiple) 'EUICCResponse' as defined in [5]
+}
+ 
+ErrorResult ::= SEQUENCE {
+    bppCommandId BppCommandId,
+    errorReason ErrorReason,
+    simaResponse OCTET STRING OPTIONAL -- contains (multiple) 'EUICCResponse' as defined in [5]
+}
+ 
+BppCommandId ::= INTEGER {initialiseSecureChannel(0), configureISDP(1), storeMetadata(2), storeMetadata2(3), replaceSessionKeys(4), loadProfileElements(5)}
+ 
+ErrorReason ::= INTEGER {
+    incorrectInputValues(1),
+    invalidSignature(2),
+    invalidTransactionId(3),
+    unsupportedCrtValues(4),
+    unsupportedRemoteOperationType(5),
+    unsupportedProfileClass(6),
+    scp03tStructureError(7),
+    scp03tSecurityError(8),
+    installFailedDueToIccidAlreadyExistsOnEuicc(9), installFailedDueToInsufficientMemoryForProfile(10),
+    installFailedDueToInterruption(11),
+    installFailedDueToPEProcessingError (12),
+    installFailedDueToIccidMismatch(13),
+    testProfileInstallFailedDueToInvalidNaaKey(14),
+    pprNotAllowed(15),
+    installFailedDueToUnknownError(127)
+}
+ 
+ListNotificationRequest ::= [40] SEQUENCE { -- Tag 'BF28'
+    profileManagementOperation [1] NotificationEvent OPTIONAL
+}
+ 
+ListNotificationResponse ::= [40] CHOICE { -- Tag 'BF28'
+    notificationMetadataList SEQUENCE OF NotificationMetadata,
+    listNotificationsResultError INTEGER {undefinedError(127)}
+}
+ 
+NotificationMetadata ::= [47] SEQUENCE { -- Tag 'BF2F'
+    seqNumber [0] INTEGER,
+    profileManagementOperation [1] NotificationEvent, --Only one bit set to 1
+    notificationAddress UTF8String, -- FQDN to forward the notification
+    iccid Iccid OPTIONAL
+}
+ 
+-- Definition of Profile Nickname Information
+SetNicknameRequest ::= [41] SEQUENCE {  -- Tag 'BF29'
+    iccid Iccid,
+    profileNickname [16] UTF8String (SIZE(0..64))
+}
+ 
+SetNicknameResponse ::= [41] SEQUENCE { -- Tag 'BF29'
+    setNicknameResult INTEGER {ok(0), iccidNotFound (1), undefinedError(127)}
+}
+ 
+id-rsp-cert-objects OBJECT IDENTIFIER ::= {  id-rsp cert-objects(2)}
+ 
+id-rspExt OBJECT IDENTIFIER ::= {id-rsp-cert-objects 0}
+ 
+id-rspRole OBJECT IDENTIFIER ::= {id-rsp-cert-objects 1}
+ 
+-- Definition of OIDs for role identification
+id-rspRole-ci OBJECT IDENTIFIER ::= {id-rspRole 0}
+id-rspRole-euicc OBJECT IDENTIFIER ::= {id-rspRole 1}
+id-rspRole-eum OBJECT IDENTIFIER ::= {id-rspRole 2}
+id-rspRole-dp-tls OBJECT IDENTIFIER ::= {id-rspRole 3}
+id-rspRole-dp-auth OBJECT IDENTIFIER ::= {id-rspRole 4}
+id-rspRole-dp-pb OBJECT IDENTIFIER ::= {id-rspRole 5}
+id-rspRole-ds-tls OBJECT IDENTIFIER ::= {id-rspRole 6}
+id-rspRole-ds-auth OBJECT IDENTIFIER ::= {id-rspRole 7}
+ 
+--Definition of data objects for InitialiseSecureChannel Request
+InitialiseSecureChannelRequest ::= [35] SEQUENCE { -- Tag 'BF23'
+    remoteOpId RemoteOpId, -- Remote Operation Type Identifier (value SHALL be set to installBoundProfilePackage)
+    transactionId [0] TransactionId, -- The TransactionID generated by the SM-DP+
+    controlRefTemplate[6] IMPLICIT ControlRefTemplate, -- Control Reference Template (Key Agreement). Current specification considers a subset of CRT specified in GlobalPlatform Card Specification [8], section 6.4.2.3 for the Mutual Authentication Data Field
+    smdpOtpk [APPLICATION 73] OCTET STRING, ---otPK.DP.ECKA as specified in GlobalPlatform Card Specification [8] section 6.4.2.3 for ePK.OCE.ECKA, tag '5F49'
+    smdpSign [APPLICATION 55] OCTET STRING -- SM-DP's signature, tag '5F37'
+}
+ 
+ControlRefTemplate ::= SEQUENCE {
+keyType[0] Octet1, -- Key type according to GlobalPlatform Card Specification [8] Table 11-16, AES= '88', Tag '80'
+keyLen[1] Octet1, --Key length in number of bytes. For current specification key length SHALL by 0x10 bytes, Tag '81'
+hostId[4] OctetTo16 -- Host ID value , Tag '84'
+}
+ 
+--Definition of data objects for ConfigureISDPRequest
+ConfigureISDPRequest ::= [36] SEQUENCE { -- Tag 'BF24'
+    dpProprietaryData [24] DpProprietaryData OPTIONAL -- Tag 'B8'
+}
+ 
+DpProprietaryData ::= SEQUENCE { --  maximum size including tag and length field: 128 bytes
+    dpOid OBJECT IDENTIFIER -- OID in the tree of the SM-DP+ that created the Profile
+    -- additional data objects defined by the SM-DP+ MAY follow
+}
+ 
+-- Definition of request message for command ReplaceSessionKeys
+ReplaceSessionKeysRequest ::= [38] SEQUENCE { -- tag 'BF26'
+/*The new initial MAC chaining value*/
+    initialMacChainingValue OCTET STRING,
+/*New session key value for encryption/decryption (PPK-ENC)*/
+    ppkEnc OCTET STRING,
+/*New session key value of the session key C-MAC computation/verification (PPK-MAC)*/
+    ppkCmac OCTET STRING
+}
+ 
+-- Definition of data objects for RetrieveNotificationsList
+RetrieveNotificationsListRequest ::= [43] SEQUENCE { -- Tag 'BF2B'
+    searchCriteria CHOICE {
+        seqNumber [0] INTEGER,
+        profileManagementOperation [1] NotificationEvent
+    } OPTIONAL
+}
+ 
+RetrieveNotificationsListResponse ::= [43] CHOICE { -- Tag 'BF2B'
+    notificationList SEQUENCE OF PendingNotification,
+    notificationsListResultError INTEGER {noResultAvailable(1), undefinedError(127)}
+}
+ 
+PendingNotification ::= CHOICE {
+    profileInstallationResult [55] ProfileInstallationResult, -- tag 'BF37'
+    otherSignedNotification OtherSignedNotification
+}
+ 
+OtherSignedNotification ::= SEQUENCE {
+    tbsOtherNotification NotificationMetadata,
+    euiccNotificationSignature [APPLICATION 55] OCTET STRING,   -- eUICC signature of tbsOtherNotification, Tag '5F37'
+    euiccCertificate Certificate,   -- eUICC Certificate (CERT.EUICC.ECDSA) signed by the EUM
+    eumCertificate Certificate  -- EUM Certificate (CERT.EUM.ECDSA) signed by the requested CI
+}
+ 
+-- Definition of notificationSent
+NotificationSentRequest ::= [48] SEQUENCE { -- Tag 'BF30'
+    seqNumber [0] INTEGER
+}
+ 
+NotificationSentResponse ::= [48] SEQUENCE { -- Tag 'BF30'
+    deleteNotificationStatus INTEGER {ok(0), nothingToDelete(1), undefinedError(127)}
+}
+ 
+-- Definition of Enable Profile --------------------------
+EnableProfileRequest ::= [49] SEQUENCE { -- Tag 'BF31'
+    profileIdentifier CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+        iccid Iccid -- ICCID, tag '5A'
+    },
+    refreshFlag BOOLEAN -- indicating whether REFRESH is required
+}
+ 
+EnableProfileResponse ::= [49] SEQUENCE { -- Tag 'BF31'
+    enableResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInDisabledState(2), disallowedByPolicy(3), wrongProfileReenabling(4), undefinedError(127)}
+}
+ 
+-- Definition of Disable Profile --------------------------
+DisableProfileRequest ::= [50] SEQUENCE { -- Tag 'BF32'
+    profileIdentifier CHOICE {
+        isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+        iccid Iccid -- ICCID, tag '5A'
+    },
+    refreshFlag BOOLEAN -- indicating whether REFRESH is required
+}
+ 
+DisableProfileResponse ::= [50] SEQUENCE { -- Tag 'BF32'
+    disableResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInEnabledState(2), disallowedByPolicy(3), undefinedError(127)}
+}
+ 
+-- Definition of Delete Profile --------------------------
+DeleteProfileRequest ::= [51] CHOICE { -- Tag 'BF33'
+    isdpAid [APPLICATION 15] OctetTo16, -- AID, tag '4F'
+    iccid Iccid -- ICCID, tag '5A'
+}
+ 
+DeleteProfileResponse ::= [51] SEQUENCE { -- Tag 'BF33'
+    deleteResult INTEGER {ok(0), iccidOrAidNotFound (1), profileNotInDisabledState(2), disallowedByPolicy(3), undefinedError(127)}
+}
+ 
+-- Definition of Memory Reset --------------------------
+EuiccMemoryResetRequest ::= [52] SEQUENCE { -- Tag 'BF34'
+    resetOptions [2] BIT STRING {
+        deleteOperationalProfiles(0),
+        deleteFieldLoadedTestProfiles(1),
+        resetDefaultSmdpAddress(2)}
+}
+ 
+EuiccMemoryResetResponse ::= [52] SEQUENCE { -- Tag 'BF34'
+    resetResult INTEGER {ok(0), nothingToDelete(1), undefinedError(127)}
+}
+ 
+-- Definition of Get EID --------------------------
+GetEuiccDataRequest ::= [62] SEQUENCE { -- Tag 'BF3E'
+    tagList [APPLICATION 28] Octet1  -- tag '5C', the value SHALL be set to '5A'
+}
+ 
+GetEuiccDataResponse ::= [62] SEQUENCE { -- Tag 'BF3E'
+    eidValue [APPLICATION 26] Octet16  -- tag '5A'
+}
+ 
+-- Definition of Get Rat
+ 
+GetRatRequest ::= [67] SEQUENCE { -- Tag ' BF43'
+    -- No input data
+}
+ 
+ 
+GetRatResponse ::= [67] SEQUENCE { -- Tag 'BF43'
+    rat RulesAuthorisationTable
+}
+ 
+RulesAuthorisationTable ::= SEQUENCE OF ProfilePolicyAuthorisationRule
+ProfilePolicyAuthorisationRule ::= SEQUENCE {
+    pprIds PprIds,
+    allowedOperators SEQUENCE OF OperatorID,
+    pprFlags BIT STRING {consentRequired(0)}
+}
+ 
+-- Definition of data structure command for loading a CRL
+LoadCRLRequest ::= [53] SEQUENCE { -- Tag 'BF35'
+    -- A CRL-A
+    crl CertificateList
+}
+ 
+-- Definition of data structure response for loading a CRL
+LoadCRLResponse ::= [53] CHOICE {  -- Tag 'BF35'
+loadCRLResponseOk LoadCRLResponseOk,
+loadCRLResponseError LoadCRLResponseError
+}
+ 
+LoadCRLResponseOk ::= SEQUENCE {
+    missingParts SEQUENCE OF SEQUENCE {
+        number INTEGER (0..MAX)
+    } OPTIONAL
+}
+LoadCRLResponseError ::= INTEGER {invalidSignature(1), invalidCRLFormat(2), notEnoughMemorySpace(3), verificationKeyNotFound(4), undefinedError(127)}
+ 
+-- Definition of the extension for Certificate Expiration Date
+id-rsp-expDate OBJECT IDENTIFIER ::= {id-rspExt 1}
+ExpirationDate ::= Time
+ 
+-- Definition of the extension id for total partial-CRL number
+id-rsp-totalPartialCrlNumber OBJECT IDENTIFIER ::= {id-rspExt 2}
+TotalPartialCrlNumber ::= INTEGER
+ 
+ 
+-- Definition of the extension id for the partial-CRL number
+id-rsp-partialCrlNumber OBJECT IDENTIFIER ::= {id-rspExt 3}
+PartialCrlNumber ::= INTEGER
+ 
+-- Definition for ES9+ ASN.1 Binding --------------------------
+RemoteProfileProvisioningRequest ::= [2] CHOICE { -- Tag 'A2'
+    initiateAuthenticationRequest [57] InitiateAuthenticationRequest,  -- Tag 'BF39'
+    authenticateClientRequest [59] AuthenticateClientRequest, -- Tag 'BF3B'
+    getBoundProfilePackageRequest [58] GetBoundProfilePackageRequest,  -- Tag 'BF3A'
+    cancelSessionRequestEs9 [65] CancelSessionRequestEs9, -- Tag 'BF41'
+    handleNotification [61] HandleNotification -- tag 'BF3D'
+}
+ 
+RemoteProfileProvisioningResponse ::= [2] CHOICE { -- Tag 'A2'
+    initiateAuthenticationResponse [57] InitiateAuthenticationResponse, -- Tag 'BF39'
+    authenticateClientResponseEs9 [59] AuthenticateClientResponseEs9, -- Tag 'BF3B'
+    getBoundProfilePackageResponse [58] GetBoundProfilePackageResponse, -- Tag 'BF3A'
+    cancelSessionResponseEs9 [65] CancelSessionResponseEs9, -- Tag 'BF41'
+    authenticateClientResponseEs11 [64] AuthenticateClientResponseEs11 -- Tag 'BF40'
+}
+ 
+InitiateAuthenticationRequest ::= [57] SEQUENCE { -- Tag 'BF39'
+    euiccChallenge [1] Octet16, -- random eUICC challenge
+    smdpAddress [3] UTF8String,
+    euiccInfo1 EUICCInfo1
+}
+ 
+InitiateAuthenticationResponse ::= [57] CHOICE { -- Tag 'BF39'
+    initiateAuthenticationOk InitiateAuthenticationOkEs9,
+    initiateAuthenticationError INTEGER {
+        invalidDpAddress(1),
+        euiccVersionNotSupportedByDp(2),
+        ciPKNotSupported(3)
+    }
+}
+ 
+InitiateAuthenticationOkEs9 ::= SEQUENCE {
+    transactionId [0] TransactionId, -- The TransactionID generated by the SM-DP+
+    serverSigned1 ServerSigned1, -- Signed information
+    serverSignature1 [APPLICATION 55] OCTET STRING, -- Server_Sign1, tag '5F37'
+    euiccCiPKIdToBeUsed SubjectKeyIdentifier, -- The curve CI Public Key to be used as required by ES10b.AuthenticateServer
+    serverCertificate Certificate
+}
+ 
+AuthenticateClientRequest ::= [59] SEQUENCE { -- Tag 'BF3B'
+    transactionId [0] TransactionId,
+    authenticateServerResponse [56] AuthenticateServerResponse -- This is the response from ES10b.AuthenticateServer
+}
+ 
+AuthenticateClientResponseEs9 ::= [59] CHOICE { -- Tag 'BF3B'
+    authenticateClientOk AuthenticateClientOk,
+    authenticateClientError INTEGER {
+        eumCertificateInvalid(1),
+        eumCertificateExpired(2),
+        euiccCertificateInvalid(3),
+        euiccCertificateExpired(4),
+        euiccSignatureInvalid(5),
+        matchingIdRefused(6),
+        eidMismatch(7),
+        noEligibleProfile(8),
+        ciPKUnknown(9),
+        invalidTransactionId(10),
+        undefinedError(127)
+    }
+}
+ 
+AuthenticateClientOk ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    profileMetaData [37] StoreMetadataRequest,
+    prepareDownloadRequest [33] PrepareDownloadRequest
+}
+ 
+GetBoundProfilePackageRequest ::= [58] SEQUENCE { -- Tag 'BF3A'
+    transactionId [0] TransactionId,
+    prepareDownloadResponse [33] PrepareDownloadResponse
+}
+ 
+GetBoundProfilePackageResponse ::= [58] CHOICE { -- Tag 'BF3A'
+    getBoundProfilePackageOk GetBoundProfilePackageOk,
+    getBoundProfilePackageError INTEGER {
+        euiccSignatureInvalid(1),
+        confirmationCodeMissing(2),
+        confirmationCodeRefused(3),
+        confirmationCodeRetriesExceeded(4),
+        invalidTransactionId(95),
+        undefinedError(127)
+    }
+}
+ 
+GetBoundProfilePackageOk ::= SEQUENCE {
+    transactionId [0] TransactionId,
+    boundProfilePackage [54] BoundProfilePackage
+}
+ 
+HandleNotification ::= [61] SEQUENCE {  -- Tag 'BF3D'
+    pendingNotification PendingNotification
+}
+ 
+CancelSessionRequestEs9 ::= [65] SEQUENCE { -- Tag 'BF41'
+    transactionId TransactionId,
+    cancelSessionResponse  CancelSessionResponse -- data structure defined for ES10b.CancelSession function
+}
+ 
+CancelSessionResponseEs9 ::= [65] CHOICE { -- Tag 'BF41'
+    cancelSessionOk CancelSessionOk,
+    cancelSessionError INTEGER {
+        invalidTransactionId(1),
+        euiccSignatureInvalid(2),
+        undefinedError(127)
+    }
+}
+ 
+CancelSessionOk ::= SEQUENCE { -- This function has no output data
+}
+ 
+EuiccConfiguredAddressesRequest ::= [60] SEQUENCE { -- Tag 'BF3C'
+}
+ 
+EuiccConfiguredAddressesResponse ::= [60] SEQUENCE {  -- Tag 'BF3C'
+    defaultDpAddress UTF8String OPTIONAL,  -- Default SM-DP+ address as an FQDN
+    rootDsAddress UTF8String  -- Root SM-DS address as an FQDN
+}
+ 
+ISDRProprietaryApplicationTemplate ::= [PRIVATE 0] SEQUENCE { -- Tag 'E0'  
+    svn [2] VersionType,    -- GSMA SGP.22 version supported (SVN)
+    lpaeSupport BIT STRING {
+        lpaeUsingCat(0), -- LPA in the eUICC using Card Application Toolkit
+        lpaeUsingScws(1) -- LPA in the eUICC using Smartcard Web Server
+    } OPTIONAL
+}
+ 
+LpaeActivationRequest ::= [66] SEQUENCE { -- Tag 'BF42'
+    lpaeOption BIT STRING {
+        activateCatBasedLpae(0), -- LPAe with LUIe based on CAT
+        activateScwsBasedLpae(1) -- LPAe with LUIe based on SCWS
+    }
+}
+ 
+LpaeActivationResponse ::= [66] SEQUENCE { -- Tag 'BF42'
+    lpaeActivationResult INTEGER {ok(0), notSupported(1)}
+}
+ 
+SetDefaultDpAddressRequest ::= [63] SEQUENCE { -- Tag 'BF3F'
+    defaultDpAddress UTF8String -- Default SM-DP+ address as an FQDN
+}
+ 
+SetDefaultDpAddressResponse ::= [63] SEQUENCE { -- Tag 'BF3F'
+    setDefaultDpAddressResult INTEGER { ok (0), undefinedError (127)}
+}
+ 
+AuthenticateClientResponseEs11 ::= [64] CHOICE {  -- Tag 'BF40'
+    authenticateClientOk AuthenticateClientOkEs11,
+    authenticateClientError INTEGER {
+        eumCertificateInvalid(1),
+        eumCertificateExpired(2),
+        euiccCertificateInvalid(3),
+        euiccCertificateExpired(4),
+        euiccSignatureInvalid(5),
+        eventIdUnknown(6),
+        invalidTransactionId(7),
+        undefinedError(127)
+    }
+}
+ 
+AuthenticateClientOkEs11 ::= SEQUENCE {
+    transactionId TransactionId,
+    eventEntries SEQUENCE OF EventEntries
+}
+ 
+EventEntries ::= SEQUENCE {
+    eventId UTF8String,
+    rspServerAddress UTF8String
+}
+ 
+END

pySim/esim/bsp.py

@@ -0,0 +1,327 @@
+"""Implementation of GSMA eSIM RSP (Remote SIM Provisioning BSP (BPP Protection Protocol),
+where BPP is the Bound  Profile Package.  So the full expansion is the
+"GSMA eSIM Remote SIM Provisioning Bound Profile Packate Protection Protocol"
+
+Originally (SGP.22 v2.x) this was called SCP03t, but it has since been renamed to BSP."""
+
+# (C) 2023 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# SGP.22 v3.0 Section 2.5.3:
+# That block of data is split into segments of a maximum size of 1020 bytes (including the tag, length field and MAC).
+
+import abc
+from typing import List
+import logging
+
+# for BSP key derivation
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.x963kdf import X963KDF
+
+from Cryptodome.Cipher import AES
+from Cryptodome.Hash import CMAC
+
+from osmocom.utils import b2h
+from osmocom.tlv import bertlv_encode_len, bertlv_parse_one
+
+# don't log by default
+logger = logging.getLogger(__name__)
+logger.addHandler(logging.NullHandler())
+
+MAX_SEGMENT_SIZE = 1020
+
+class BspAlgo(abc.ABC):
+    """Base class representing a cryptographic algorithm within the BSP (BPP Security Protocol)."""
+    blocksize: int
+
+    def _get_padding(self, in_len: int, multiple: int, padding: int = 0) -> bytes:
+        """Return padding bytes towards multiple of N."""
+        if in_len % multiple == 0:
+            return b''
+        pad_cnt = multiple - (in_len % multiple)
+        return bytes([padding]) * pad_cnt
+
+    def _pad_to_multiple(self, indat: bytes, multiple: int, padding: int = 0) -> bytes:
+        """Pad the input data to multiples of 'multiple'."""
+        return indat + self._get_padding(len(indat), multiple, padding)
+
+    def __str__(self):
+        return self.__class__.__name__
+
+class BspAlgoCrypt(BspAlgo, abc.ABC):
+    """Base class representing an encryption/decryption algorithm within the BSP (BPP Security Protocol)."""
+
+    def __init__(self, s_enc: bytes):
+        self.s_enc = s_enc
+        self.block_nr = 1
+
+    def encrypt(self, data:bytes) -> bytes:
+        """Encrypt given input bytes using the key material given in constructor."""
+        padded_data = self._pad_to_multiple(data, self.blocksize)
+        block_nr = self.block_nr
+        ciphertext = self._encrypt(padded_data)
+        logger.debug("encrypt(block_nr=%u, s_enc=%s, plaintext=%s, padded=%s) -> %s",
+                     block_nr, b2h(self.s_enc)[:20], b2h(data)[:20], b2h(padded_data)[:20], b2h(ciphertext)[:20])
+        return ciphertext
+
+    def decrypt(self, data:bytes) -> bytes:
+        """Decrypt given input bytes using the key material given in constructor."""
+        return self._unpad(self._decrypt(data))
+
+    @abc.abstractmethod
+    def _unpad(self, padded: bytes) -> bytes:
+        """Remove the padding from padded data."""
+
+    @abc.abstractmethod
+    def _encrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+
+    @abc.abstractmethod
+    def _decrypt(self, data:bytes) -> bytes:
+        """Actual implementation, to be implemented by derived class."""
+
+class BspAlgoCryptAES128(BspAlgoCrypt):
+    """AES-CBC-128 implementation of the BPP Security Protocol for GSMA SGP.22 eSIM."""
+    name = 'AES-CBC-128'
+    blocksize = 16
+
+    def _get_padding(self, in_len: int, multiple: int, padding: int = 0):
+        # SGP.22 section 2.6.4.4
+        # Append a byte with value '80' to the right of the data block;
+        # Append 0 to 15 bytes with value '00' so that the length of the padded data block
+        # is a multiple of 16 bytes.
+        return b'\x80' + super()._get_padding(in_len + 1, multiple, padding)
+
+    def _unpad(self, padded: bytes) -> bytes:
+        """Remove the customary 80 00 00 ... padding used for AES."""
+        # first remove any trailing zero bytes
+        stripped = padded.rstrip(b'\0')
+        # then remove the final 80
+        assert stripped[-1] == 0x80
+        return stripped[:-1]
+
+    def _get_icv(self):
+        # The binary value of this number SHALL be left padded with zeroes to form a full block.
+        data = self.block_nr.to_bytes(self.blocksize, "big")
+        #iv = bytes([0] * (self.blocksize-1)) + b'\x01'
+        iv = bytes([0] * self.blocksize)
+        # This block SHALL be encrypted with S-ENC to produce the ICV for command encryption.
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, iv)
+        icv = cipher.encrypt(data)
+        logger.debug("_get_icv(block_nr=%u, data=%s) -> icv=%s", self.block_nr, b2h(data), b2h(icv))
+        self.block_nr = self.block_nr + 1
+        return icv
+
+    def _encrypt(self, data: bytes) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv())
+        return cipher.encrypt(data)
+
+    def _decrypt(self, data: bytes) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv())
+        return cipher.decrypt(data)
+
+
+class BspAlgoMac(BspAlgo, abc.ABC):
+    """Base class representing a message authentication code algorithm within the BSP (BPP Security Protocol)."""
+    l_mac = 0 # must be overridden by derived class
+
+    def __init__(self, s_mac: bytes, initial_mac_chaining_value: bytes):
+        self.s_mac = s_mac
+        self.mac_chain = initial_mac_chaining_value
+
+    def auth(self, tag: int, data: bytes) -> bytes:
+        assert tag in range (256)
+        # The input data used for C-MAC computation comprises the MAC Chaining value, the tag, the final length and the result of step 2
+        lcc = len(data) + self.l_mac
+        tag_and_length = bytes([tag]) + bertlv_encode_len(lcc)
+        temp_data = self.mac_chain + tag_and_length + data
+        old_mcv = self.mac_chain
+        c_mac = self._auth(temp_data)
+
+        # DEBUG: Show MAC computation details
+        logger.debug(f"MAC_DEBUG: tag=0x{tag:02x}, lcc={lcc}")
+        logger.debug(f"MAC_DEBUG: tag_and_length: {tag_and_length.hex()}")
+        logger.debug(f"MAC_DEBUG: mac_chain[:20]: {old_mcv[:20].hex()}")
+        logger.debug(f"MAC_DEBUG: temp_data[:20]: {temp_data[:20].hex()}")
+        logger.debug(f"MAC_DEBUG: c_mac: {c_mac.hex()}")
+
+        # The output data is computed by concatenating the following data: the tag, the final length, the result of step 2 and the C-MAC value.
+        ret = tag_and_length + data + c_mac
+        logger.debug(f"MAC_DEBUG: final_output[:20]: {ret[:20].hex()}")
+
+        logger.debug("auth(tag=0x%x, mcv=%s, s_mac=%s, plaintext=%s, temp=%s) -> %s",
+                     tag, b2h(old_mcv)[:20], b2h(self.s_mac)[:20], b2h(data)[:20], b2h(temp_data)[:20], b2h(ret)[:20])
+        return ret
+
+    def verify(self, ciphertext: bytes) -> bool:
+        mac_stripped = ciphertext[0:-self.l_mac]
+        mac_received = ciphertext[-self.l_mac:]
+        temp_data = self.mac_chain + mac_stripped
+        mac_computed = self._auth(temp_data)
+        if mac_received != mac_computed:
+            raise ValueError("MAC value not matching: received: %s, computed: %s" % (mac_received, mac_computed))
+        return mac_stripped
+
+    @abc.abstractmethod
+    def _auth(self, temp_data: bytes) -> bytes:
+        """To be implemented by algorithm specific derived class."""
+
+class BspAlgoMacAES128(BspAlgoMac):
+    """AES-CMAC-128 implementation of the BPP Security Protocol for GSMA SGP.22 eSIM."""
+    name = 'AES-CMAC-128'
+    l_mac = 8
+
+    def _auth(self, temp_data: bytes) -> bytes:
+        # The full MAC value is computed using the MACing algorithm as defined in table 4c.
+        cmac = CMAC.new(self.s_mac, ciphermod=AES)
+        cmac.update(temp_data)
+        full_c_mac = cmac.digest()
+        # Subsequent MAC chaining values are the full result of step 4 of the previous data block
+        self.mac_chain = full_c_mac
+        # If the algorithm is AES-CBC-128 or SM4-CBC, the C-MAC value is the 8 most significant bytes of the result of step 4
+        return full_c_mac[0:8]
+
+
+
+def bsp_key_derivation(shared_secret: bytes, key_type: int, key_length: int, host_id: bytes, eid, l : int = 16):
+    """BSP protocol key derivation as per SGP.22 v3.0 Section 2.6.4.2"""
+    assert key_type <= 255
+    assert key_length <= 255
+
+    host_id_lv = bertlv_encode_len(len(host_id)) + host_id
+    eid_lv = bertlv_encode_len(len(eid)) + eid
+    shared_info = bytes([key_type, key_length]) + host_id_lv + eid_lv
+    logger.debug("kdf_shared_info: %s", b2h(shared_info))
+
+    # X9.63 Key Derivation Function with SHA256
+    xkdf = X963KDF(algorithm=hashes.SHA256(), length=l*3, sharedinfo=shared_info)
+    out = xkdf.derive(shared_secret)
+    logger.debug("kdf_out: %s", b2h(out))
+
+    initial_mac_chaining_value = out[0:l]
+    s_enc = out[l:2*l]
+    s_mac = out[l*2:3*l]
+
+    logger.debug(f"BSP_KDF_DEBUG: kdf_out = {b2h(out)}")
+    logger.debug(f"BSP_KDF_DEBUG: initial_mcv = {b2h(initial_mac_chaining_value)}")
+    logger.debug(f"BSP_KDF_DEBUG: s_enc = {b2h(s_enc)}")
+    logger.debug(f"BSP_KDF_DEBUG: s_mac = {b2h(s_mac)}")
+
+    return s_enc, s_mac, initial_mac_chaining_value
+
+
+
+class BspInstance:
+    """An instance of the BSP crypto.  Initialized once with the key material via constructor,
+    then the user can call any number of encrypt_and_mac cycles to protect plaintext and
+    generate the respective ciphertext."""
+    def __init__(self, s_enc: bytes, s_mac: bytes, initial_mcv: bytes):
+        logger.debug("%s(s_enc=%s, s_mac=%s, initial_mcv=%s)", self.__class__.__name__, b2h(s_enc), b2h(s_mac), b2h(initial_mcv))
+        self.c_algo = BspAlgoCryptAES128(s_enc)
+        self.m_algo = BspAlgoMacAES128(s_mac, initial_mcv)
+
+        TAG_LEN = 1
+        length_len = len(bertlv_encode_len(MAX_SEGMENT_SIZE))
+        self.max_payload_size = MAX_SEGMENT_SIZE - TAG_LEN - length_len - self.m_algo.l_mac
+
+    @classmethod
+    def from_kdf(cls, shared_secret: bytes, key_type: int, key_length: int, host_id: bytes, eid: bytes):
+        """Convenience constructor for constructing an instance with keys from KDF."""
+        s_enc, s_mac, initial_mcv = bsp_key_derivation(shared_secret, key_type, key_length, host_id, eid)
+        return cls(s_enc, s_mac, initial_mcv)
+
+    def encrypt_and_mac_one(self, tag: int, plaintext:bytes) -> bytes:
+        """Encrypt + MAC a single plaintext TLV. Returns the protected ciphertext."""
+        assert tag <= 255
+        assert len(plaintext) <= self.max_payload_size
+
+        # DEBUG: Show what we're processing
+        logger.debug(f"BSP_DEBUG: encrypt_and_mac_one(tag=0x{tag:02x}, plaintext_len={len(plaintext)})")
+        logger.debug(f"BSP_DEBUG: plaintext[:20]: {plaintext[:20].hex()}")
+        logger.debug(f"BSP_DEBUG: s_enc[:20]: {self.c_algo.s_enc[:20].hex()}")
+        logger.debug(f"BSP_DEBUG: s_mac[:20]: {self.m_algo.s_mac[:20].hex()}")
+
+        logger.debug("encrypt_and_mac_one(tag=0x%x, plaintext=%s)", tag, b2h(plaintext)[:20])
+        ciphered = self.c_algo.encrypt(plaintext)
+        logger.debug(f"BSP_DEBUG: ciphered[:20]: {ciphered[:20].hex()}")
+
+        maced = self.m_algo.auth(tag, ciphered)
+        logger.debug(f"BSP_DEBUG: final_result[:20]: {maced[:20].hex()}")
+        logger.debug(f"BSP_DEBUG: final_result_len: {len(maced)}")
+
+        return maced
+
+    def encrypt_and_mac(self, tag: int, plaintext:bytes) -> List[bytes]:
+        remainder = plaintext
+        result = []
+        while len(remainder):
+            remaining_len = len(remainder)
+            if remaining_len < self.max_payload_size:
+                segment_len = remaining_len
+                segment = remainder
+                remainder = b''
+            else:
+                segment_len = self.max_payload_size
+                segment = remainder[0:segment_len]
+                remainder = remainder[segment_len:]
+            result.append(self.encrypt_and_mac_one(tag, segment))
+        return result
+
+    def mac_only_one(self, tag: int, plaintext: bytes) -> bytes:
+        """MAC a single plaintext TLV. Returns the protected ciphertext."""
+        assert tag <= 255
+        assert len(plaintext) <= self.max_payload_size
+        maced = self.m_algo.auth(tag, plaintext)
+        # The data block counter for ICV calculation is incremented also for each segment with C-MAC only.
+        self.c_algo.block_nr += 1
+        return maced
+
+    def mac_only(self, tag: int, plaintext:bytes) -> List[bytes]:
+        remainder = plaintext
+        result = []
+        while len(remainder):
+            remaining_len = len(remainder)
+            if remaining_len < self.max_payload_size:
+                segment_len = remaining_len
+                segment = remainder
+                remainder = b''
+            else:
+                segment_len = self.max_payload_size
+                segment = remainder[0:segment_len]
+                remainder = remainder[segment_len:]
+            result.append(self.mac_only_one(tag, segment))
+        return result
+
+    def demac_and_decrypt_one(self, ciphertext: bytes) -> bytes:
+        payload = self.m_algo.verify(ciphertext)
+        tdict, l, val, remain = bertlv_parse_one(payload)
+        logger.debug("tag=%s, l=%u, val=%s, remain=%s", tdict, l, b2h(val), b2h(remain))
+        plaintext = self.c_algo.decrypt(val)
+        return plaintext
+
+    def demac_and_decrypt(self, ciphertext_list: List[bytes]) -> bytes:
+        plaintext_list = [self.demac_and_decrypt_one(x) for x in ciphertext_list]
+        return b''.join(plaintext_list)
+
+    def demac_only_one(self, ciphertext: bytes) -> bytes:
+        payload = self.m_algo.verify(ciphertext)
+        _tdict, _l, val, _remain = bertlv_parse_one(payload)
+        # The data block counter for ICV calculation is incremented also for each segment with C-MAC only.
+        self.c_algo.block_nr += 1
+        return val
+
+    def demac_only(self, ciphertext_list: List[bytes]) -> bytes:
+        plaintext_list = [self.demac_only_one(x) for x in ciphertext_list]
+        return b''.join(plaintext_list)

pySim/esim/es2p.py

@@ -0,0 +1,362 @@
+"""GSMA eSIM RSP ES2+ interface according to SGP.22 v2.5"""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import requests
+from klein import Klein
+from twisted.internet import defer, protocol, ssl, task, endpoints, reactor
+from twisted.internet.posixbase import PosixReactorBase
+from pathlib import Path
+from twisted.web.server import Site, Request
+
+import logging
+from datetime import datetime
+import time
+
+from pySim.esim.http_json_api import *
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)
+
+class param:
+    class Iccid(ApiParamString):
+        """String representation of 18 to 20 digits, where the 20th digit MAY optionally be the padding
+        character F."""
+        @classmethod
+        def _encode(cls, data):
+            data = str(data)
+            # SGP.22 version prior to 2.2 do not require support for 19-digit ICCIDs, so let's always
+            # encode it with padding F at the end.
+            if len(data) == 19:
+                data += 'F'
+            return data
+
+        @classmethod
+        def verify_encoded(cls, data):
+            if len(data) not in (18, 19, 20):
+                raise ValueError('ICCID (%s) length (%u) invalid' % (data, len(data)))
+
+        @classmethod
+        def _decode(cls, data):
+            # strip trailing padding (if it's 20 digits)
+            if len(data) == 20 and data[-1] in ['F', 'f']:
+                data = data[:-1]
+            return data
+
+        @classmethod
+        def verify_decoded(cls, data):
+            data = str(data)
+            if len(data) not in (18, 19, 20):
+                raise ValueError('ICCID (%s) length (%u) invalid' % (data, len(data)))
+            if len(data) == 19:
+                decimal_part = data
+            else:
+                decimal_part = data[:-1]
+                final_part = data[-1:]
+                if final_part not in ['F', 'f'] and not final_part.isdecimal():
+                    raise ValueError('ICCID (%s) contains non-decimal characters' % data)
+            if not decimal_part.isdecimal():
+                raise ValueError('ICCID (%s) contains non-decimal characters' % data)
+
+
+    class Eid(ApiParamString):
+        """String of 32 decimal characters"""
+        @classmethod
+        def verify_encoded(cls, data):
+            if len(data) != 32:
+                raise ValueError('EID length invalid: "%s" (%u)' % (data, len(data)))
+
+        @classmethod
+        def verify_decoded(cls, data):
+            if not data.isdecimal():
+                raise ValueError('EID (%s) contains non-decimal characters' % data)
+
+    class ProfileType(ApiParamString):
+        pass
+
+    class MatchingId(ApiParamString):
+        pass
+
+    class ConfirmationCode(ApiParamString):
+        pass
+
+    class SmdsAddress(ApiParamFqdn):
+        pass
+
+    class ReleaseFlag(ApiParamBoolean):
+        pass
+
+    class FinalProfileStatusIndicator(ApiParamString):
+        pass
+
+    class Timestamp(ApiParamString):
+        """String format as specified by W3C: YYYY-MM-DDThh:mm:ssTZD"""
+        @classmethod
+        def _decode(cls, data):
+            return datetime.fromisoformat(data)
+
+        @classmethod
+        def _encode(cls, data):
+            return datetime.isoformat(data)
+
+    class NotificationPointId(ApiParamInteger):
+        pass
+
+    class NotificationPointStatus(ApiParam):
+        pass
+
+    class ResultData(ApiParamBase64):
+        pass
+
+class Es2PlusApiFunction(JsonHttpApiFunction):
+    """Base class for representing an ES2+ API Function."""
+    pass
+
+# ES2+ DownloadOrder function (SGP.22 section 5.3.1)
+class DownloadOrder(Es2PlusApiFunction):
+    path = '/gsma/rsp2/es2plus/downloadOrder'
+    input_params = {
+        'header': JsonRequestHeader,
+        'eid': param.Eid,
+        'iccid': param.Iccid,
+        'profileType': param.ProfileType
+      }
+    input_mandatory = ['header']
+    output_params = {
+        'header': JsonResponseHeader,
+        'iccid': param.Iccid,
+      }
+    output_mandatory = ['header', 'iccid']
+
+# ES2+ ConfirmOrder function (SGP.22 section 5.3.2)
+class ConfirmOrder(Es2PlusApiFunction):
+    path = '/gsma/rsp2/es2plus/confirmOrder'
+    input_params = {
+        'header': JsonRequestHeader,
+        'iccid': param.Iccid,
+        'eid': param.Eid,
+        'matchingId': param.MatchingId,
+        'confirmationCode': param.ConfirmationCode,
+        'smdsAddress': param.SmdsAddress,
+        'releaseFlag': param.ReleaseFlag,
+      }
+    input_mandatory = ['header', 'iccid', 'releaseFlag']
+    output_params = {
+        'header': JsonResponseHeader,
+        'eid': param.Eid,
+        'matchingId': param.MatchingId,
+        'smdpAddress': SmdpAddress,
+      }
+    output_mandatory = ['header', 'matchingId']
+
+# ES2+ CancelOrder function (SGP.22 section 5.3.3)
+class CancelOrder(Es2PlusApiFunction):
+    path = '/gsma/rsp2/es2plus/cancelOrder'
+    input_params = {
+        'header': JsonRequestHeader,
+        'iccid': param.Iccid,
+        'eid': param.Eid,
+        'matchingId': param.MatchingId,
+        'finalProfileStatusIndicator': param.FinalProfileStatusIndicator,
+      }
+    input_mandatory = ['header', 'finalProfileStatusIndicator', 'iccid']
+    output_params = {
+        'header': JsonResponseHeader,
+      }
+    output_mandatory = ['header']
+
+# ES2+ ReleaseProfile function (SGP.22 section 5.3.4)
+class ReleaseProfile(Es2PlusApiFunction):
+    path = '/gsma/rsp2/es2plus/releaseProfile'
+    input_params = {
+        'header': JsonRequestHeader,
+        'iccid': param.Iccid,
+      }
+    input_mandatory = ['header', 'iccid']
+    output_params = {
+        'header': JsonResponseHeader,
+      }
+    output_mandatory = ['header']
+
+# ES2+ HandleDownloadProgress function (SGP.22 section 5.3.5)
+class HandleDownloadProgressInfo(Es2PlusApiFunction):
+    path = '/gsma/rsp2/es2plus/handleDownloadProgressInfo'
+    input_params = {
+        'header': JsonRequestHeader,
+        'eid': param.Eid,
+        'iccid': param.Iccid,
+        'profileType': param.ProfileType,
+        'timestamp': param.Timestamp,
+        'notificationPointId': param.NotificationPointId,
+        'notificationPointStatus': param.NotificationPointStatus,
+        'resultData': param.ResultData,
+    }
+    input_mandatory = ['header', 'iccid', 'profileType', 'timestamp', 'notificationPointId', 'notificationPointStatus']
+    expected_http_status = 204
+
+class Es2pApiClient:
+    """Main class representing a full ES2+ API client. Has one method for each API function."""
+    def __init__(self, url_prefix:str, func_req_id:str, server_cert_verify: str = None, client_cert: str = None):
+        self.func_id = 0
+        self.session = requests.Session()
+        if server_cert_verify:
+            self.session.verify = server_cert_verify
+        if client_cert:
+            self.session.cert = client_cert
+
+        self.downloadOrder = JsonHttpApiClient(DownloadOrder(), url_prefix, func_req_id, self.session)
+        self.confirmOrder = JsonHttpApiClient(ConfirmOrder(), url_prefix, func_req_id, self.session)
+        self.cancelOrder = JsonHttpApiClient(CancelOrder(), url_prefix, func_req_id, self.session)
+        self.releaseProfile = JsonHttpApiClient(ReleaseProfile(), url_prefix, func_req_id, self.session)
+        self.handleDownloadProgressInfo = JsonHttpApiClient(HandleDownloadProgressInfo(), url_prefix, func_req_id, self.session)
+
+    def _gen_func_id(self) -> str:
+        """Generate the next function call id."""
+        self.func_id += 1
+        return 'FCI-%u-%u' % (time.time(), self.func_id)
+
+    def call_downloadOrder(self, data: dict) -> dict:
+        """Perform ES2+ DownloadOrder function (SGP.22 section 5.3.1)."""
+        return self.downloadOrder.call(data, self._gen_func_id())
+
+    def call_confirmOrder(self, data: dict) -> dict:
+        """Perform ES2+ ConfirmOrder function (SGP.22 section 5.3.2)."""
+        return self.confirmOrder.call(data, self._gen_func_id())
+
+    def call_cancelOrder(self, data: dict) -> dict:
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.3)."""
+        return self.cancelOrder.call(data, self._gen_func_id())
+
+    def call_releaseProfile(self, data: dict) -> dict:
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.4)."""
+        return self.releaseProfile.call(data, self._gen_func_id())
+
+    def call_handleDownloadProgressInfo(self, data: dict) -> dict:
+        """Perform ES2+ HandleDownloadProgressInfo function (SGP.22 section 5.3.5)."""
+        return self.handleDownloadProgressInfo.call(data, self._gen_func_id())
+
+class Es2pApiServerHandlerSmdpp(abc.ABC):
+    """ES2+ (SMDP+ side) API Server handler class. The API user is expected to override the contained methods."""
+
+    @abc.abstractmethod
+    def call_downloadOrder(self, data: dict) -> (dict, str):
+        """Perform ES2+ DownloadOrder function (SGP.22 section 5.3.1)."""
+        pass
+
+    @abc.abstractmethod
+    def call_confirmOrder(self, data: dict) -> (dict, str):
+        """Perform ES2+ ConfirmOrder function (SGP.22 section 5.3.2)."""
+        pass
+
+    @abc.abstractmethod
+    def call_cancelOrder(self, data: dict) -> (dict, str):
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.3)."""
+        pass
+
+    @abc.abstractmethod
+    def call_releaseProfile(self, data: dict) -> (dict, str):
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.4)."""
+        pass
+
+class Es2pApiServerHandlerMno(abc.ABC):
+    """ES2+ (MNO side) API Server handler class. The API user is expected to override the contained methods."""
+
+    @abc.abstractmethod
+    def call_handleDownloadProgressInfo(self, data: dict) -> (dict, str):
+        """Perform ES2+ HandleDownloadProgressInfo function (SGP.22 section 5.3.5)."""
+        pass
+
+class Es2pApiServer(abc.ABC):
+    """Main class representing a full ES2+ API server. Has one method for each API function."""
+    app = None
+
+    def __init__(self, port: int, interface: str, server_cert: str = None, client_cert_verify: str = None):
+        logger.debug("HTTP SRV: starting ES2+ API server on %s:%s" % (interface, port))
+        self.port = port
+        self.interface = interface
+        if server_cert:
+            self.server_cert = ssl.PrivateCertificate.loadPEM(Path(server_cert).read_text())
+        else:
+            self.server_cert = None
+        if client_cert_verify:
+            self.client_cert_verify = ssl.Certificate.loadPEM(Path(client_cert_verify).read_text())
+        else:
+            self.client_cert_verify = None
+
+    def reactor(self, reactor: PosixReactorBase):
+        logger.debug("HTTP SRV: listen on %s:%s" % (self.interface, self.port))
+        if self.server_cert:
+            if self.client_cert_verify:
+                reactor.listenSSL(self.port, Site(self.app.resource()), self.server_cert.options(self.client_cert_verify),
+                                  interface=self.interface)
+            else:
+                reactor.listenSSL(self.port, Site(self.app.resource()), self.server_cert.options(),
+                                  interface=self.interface)
+        else:
+            reactor.listenTCP(self.port, Site(self.app.resource()), interface=self.interface)
+        return defer.Deferred()
+
+class Es2pApiServerSmdpp(Es2pApiServer):
+    """ES2+ (SMDP+ side) API Server."""
+    app = Klein()
+
+    def __init__(self, port: int, interface: str, handler: Es2pApiServerHandlerSmdpp,
+                 server_cert: str = None, client_cert_verify: str = None):
+        super().__init__(port, interface, server_cert, client_cert_verify)
+        self.handler = handler
+        self.downloadOrder = JsonHttpApiServer(DownloadOrder(), handler.call_downloadOrder)
+        self.confirmOrder = JsonHttpApiServer(ConfirmOrder(), handler.call_confirmOrder)
+        self.cancelOrder = JsonHttpApiServer(CancelOrder(), handler.call_cancelOrder)
+        self.releaseProfile = JsonHttpApiServer(ReleaseProfile(), handler.call_releaseProfile)
+        task.react(self.reactor)
+
+    @app.route(DownloadOrder.path)
+    def call_downloadOrder(self, request: Request) -> dict:
+        """Perform ES2+ DownloadOrder function (SGP.22 section 5.3.1)."""
+        return self.downloadOrder.call(request)
+
+    @app.route(ConfirmOrder.path)
+    def call_confirmOrder(self, request: Request) -> dict:
+        """Perform ES2+ ConfirmOrder function (SGP.22 section 5.3.2)."""
+        return self.confirmOrder.call(request)
+
+    @app.route(CancelOrder.path)
+    def call_cancelOrder(self, request: Request) -> dict:
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.3)."""
+        return self.cancelOrder.call(request)
+
+    @app.route(ReleaseProfile.path)
+    def call_releaseProfile(self, request: Request) -> dict:
+        """Perform ES2+ CancelOrder function (SGP.22 section 5.3.4)."""
+        return self.releaseProfile.call(request)
+
+class Es2pApiServerMno(Es2pApiServer):
+    """ES2+ (MNO side) API Server."""
+
+    app = Klein()
+
+    def __init__(self, port: int, interface: str, handler: Es2pApiServerHandlerMno,
+                 server_cert: str = None, client_cert_verify: str = None):
+        super().__init__(port, interface, server_cert, client_cert_verify)
+        self.handler = handler
+        self.handleDownloadProgressInfo = JsonHttpApiServer(HandleDownloadProgressInfo(),
+                                                            handler.call_handleDownloadProgressInfo)
+        task.react(self.reactor)
+
+    @app.route(HandleDownloadProgressInfo.path)
+    def call_handleDownloadProgressInfo(self, request: Request) -> dict:
+        """Perform ES2+ HandleDownloadProgressInfo function (SGP.22 section 5.3.5)."""
+        return self.handleDownloadProgressInfo.call(request)

pySim/esim/es8p.py

@@ -0,0 +1,306 @@
+"""Implementation of GSMA eSIM RSP (Remote SIM Provisioning) ES8+ as per SGP22 v3.0 Section 5.5"""
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import Dict, List, Optional
+from cryptography.hazmat.primitives.asymmetric import ec
+from osmocom.utils import b2h, h2b
+from osmocom.tlv import bertlv_encode_tag, bertlv_encode_len, bertlv_parse_one_rawtag
+from osmocom.tlv import bertlv_return_one_rawtlv
+
+import pySim.esim.rsp as rsp
+from pySim.esim.bsp import BspInstance
+from pySim.esim import PMO
+
+import logging
+logger = logging.getLogger(__name__)
+
+# Given that GSMA RSP uses ASN.1 in a very weird way, we actually cannot encode the full data type before
+# signing, but we have to build parts of it separately first, then sign that, so we can put the signature
+# into the same sequence as the signed data.  We use the existing pySim TLV code for this.
+
+def wrap_as_der_tlv(tag: int, val: bytes) -> bytes:
+    """Wrap the 'value' into a DER-encoded TLV."""
+    return bertlv_encode_tag(tag) + bertlv_encode_len(len(val)) + val
+
+def gen_init_sec_chan_signed_part(iscsp: Dict) -> bytes:
+    """Generate the concatenated remoteOpId, transactionId, controlRefTemplate and smdpOtpk data objects
+    without the outer SEQUENCE tag / length or the remainder of initialiseSecureChannel, as is required
+    for signing purpose."""
+    out = b''
+    out += wrap_as_der_tlv(0x82, bytes([iscsp['remoteOpId']]))
+    out += wrap_as_der_tlv(0x80, iscsp['transactionId'])
+
+    crt = iscsp['controlRefTemplate']
+    out_crt = wrap_as_der_tlv(0x80, crt['keyType'])
+    out_crt += wrap_as_der_tlv(0x81, crt['keyLen'])
+    out_crt += wrap_as_der_tlv(0x84, crt['hostId'])
+    out += wrap_as_der_tlv(0xA6, out_crt)
+
+    out += wrap_as_der_tlv(0x5F49, iscsp['smdpOtpk'])
+    return out
+
+
+# SGP.22 Section 5.5.1
+def gen_initialiseSecureChannel(transactionId: str, host_id: bytes, smdp_otpk: bytes, euicc_otpk: bytes, dp_pb):
+    """Generate decoded representation of (signed) initialiseSecureChannel (SGP.22 5.5.2)"""
+    init_scr = { 'remoteOpId': 1, # installBoundProfilePackage
+                 'transactionId': h2b(transactionId),
+                 # GlobalPlatform Card Specification Amendment F [13] section 6.5.2.3 for the Mutual Authentication Data Field
+                 'controlRefTemplate': { 'keyType': bytes([0x88]), 'keyLen': bytes([16]), 'hostId': host_id },
+                 'smdpOtpk': smdp_otpk, # otPK.DP.KA
+                }
+    to_sign = gen_init_sec_chan_signed_part(init_scr) + wrap_as_der_tlv(0x5f49, euicc_otpk)
+    init_scr['smdpSign'] = dp_pb.ecdsa_sign(to_sign)
+    return init_scr
+
+def gen_replace_session_keys(ppk_enc: bytes, ppk_cmac: bytes, initial_mcv: bytes) -> bytes:
+    """Generate encoded (but unsigned) ReplaceSessionKeysReqest DO (SGP.22 5.5.4)"""
+    rsk = { 'ppkEnc': ppk_enc, 'ppkCmac': ppk_cmac, 'initialMacChainingValue': initial_mcv }
+    return rsp.asn1.encode('ReplaceSessionKeysRequest', rsk)
+
+
+class ProfileMetadata:
+    """Representation of Profile metadata. Right now only the mandatory bits are
+    supported, but in general this should follow the StoreMetadataRequest of SGP.22 5.5.3"""
+    def __init__(self, iccid_bin: bytes, spn: str, profile_name: str, profile_class = 'operational'):
+        self.iccid_bin = iccid_bin
+        self.spn = spn
+        self.profile_name = profile_name
+        self.profile_class = profile_class
+        self.icon = None
+        self.icon_type = None
+        self.notifications = []
+
+    def set_icon(self, is_png: bool, icon_data: bytes):
+        """Set the icon that is part of the metadata."""
+        if len(icon_data) > 1024:
+            raise ValueError('Icon data must not exceed 1024 bytes')
+        self.icon = icon_data
+        if is_png:
+            self.icon_type = 1
+        else:
+            self.icon_type = 0
+
+    def add_notification(self, event: str, address: str):
+        """Add an 'other' notification to the notification configuration of the metadata"""
+        self.notifications.append((event, address))
+
+    def gen_store_metadata_request(self) -> bytes:
+        """Generate encoded (but unsigned) StoreMetadataRequest DO (SGP.22 5.5.3)"""
+        smr = {
+            'iccid': self.iccid_bin,
+            'serviceProviderName': self.spn,
+            'profileName': self.profile_name,
+        }
+        if self.profile_class == 'test':
+            smr['profileClass'] = 0
+        elif self.profile_class == 'provisioning':
+            smr['profileClass'] = 1
+        elif self.profile_class == 'operational':
+            smr['profileClass'] = 2
+        else:
+            raise ValueError('Unsupported Profile Class %s' % self.profile_class)
+        if self.icon:
+            smr['icon'] = self.icon
+            smr['iconType'] = self.icon_type
+        nci = []
+        for n in self.notifications:
+            pmo = PMO(n[0])
+            nci.append({'profileManagementOperation': pmo.to_bitstring(), 'notificationAddress': n[1]})
+        if len(nci):
+            smr['notificationConfigurationInfo'] = nci
+        return rsp.asn1.encode('StoreMetadataRequest', smr)
+
+
+class ProfilePackage:
+    def __init__(self, metadata: Optional[ProfileMetadata] = None):
+        self.metadata = metadata
+
+class UnprotectedProfilePackage(ProfilePackage):
+    """Representing an unprotected profile package (UPP) as defined in SGP.22 Section 2.5.2"""
+
+    @classmethod
+    def from_der(cls, der: bytes, metadata: Optional[ProfileMetadata] = None) -> 'UnprotectedProfilePackage':
+        """Load an UPP from its DER representation."""
+        inst = cls(metadata=metadata)
+        cls.der = der
+        # TODO: we later certainly want to parse it so we can perform modification (IMSI, key material, ...)
+        # just like in the traditional SIM/USIM dynamic data phase at the end of personalization
+        return inst
+
+    def to_der(self):
+        """Return the DER representation of the UPP."""
+        # TODO: once we work on decoded structures, we may want to re-encode here
+        return self.der
+
+class ProtectedProfilePackage(ProfilePackage):
+    """Representing a protected profile package (PPP) as defined in SGP.22 Section 2.5.3"""
+
+    @classmethod
+    def from_upp(cls, upp: UnprotectedProfilePackage, bsp: BspInstance) -> 'ProtectedProfilePackage':
+        """Generate the PPP as a sequence of encrypted and MACed Command TLVs representing the UPP"""
+        inst = cls(metadata=upp.metadata)
+        inst.upp = upp
+        # store ppk-enc, ppc-mac
+        inst.ppk_enc = bsp.c_algo.s_enc
+        inst.ppk_mac = bsp.m_algo.s_mac
+        inst.initial_mcv = bsp.m_algo.mac_chain
+        inst.encoded = bsp.encrypt_and_mac(0x86, upp.to_der())
+        return inst
+
+    #def __val__(self):
+        #return self.encoded
+
+class BoundProfilePackage(ProfilePackage):
+    """Representing a bound profile package (BPP) as defined in SGP.22 Section 2.5.4"""
+
+    @classmethod
+    def from_ppp(cls, ppp: ProtectedProfilePackage):
+        inst = cls()
+        inst.upp = None
+        inst.ppp = ppp
+        return inst
+
+    @classmethod
+    def from_upp(cls, upp: UnprotectedProfilePackage):
+        inst = cls()
+        inst.upp = upp
+        inst.ppp = None
+        return inst
+
+    def encode(self, ss: 'RspSessionState', dp_pb: 'CertAndPrivkey') -> bytes:
+        """Generate a bound profile package (SGP.22 2.5.4)."""
+
+        def encode_seq(tag: int, sequence: List[bytes]) -> bytes:
+            """Encode a "sequenceOfXX" as specified in SGP.22 specifying the raw SEQUENCE OF tag,
+            and assuming the caller provides the fully-encoded (with TAG + LEN) member TLVs."""
+            payload = b''.join(sequence)
+            return bertlv_encode_tag(tag) + bertlv_encode_len(len(payload)) + payload
+
+        bsp = BspInstance.from_kdf(ss.shared_secret, 0x88, 16, ss.host_id, h2b(ss.eid))
+
+        iscr = gen_initialiseSecureChannel(ss.transactionId, ss.host_id, ss.smdp_otpk, ss.euicc_otpk, dp_pb)
+        # generate unprotected input data
+        conf_idsp_bin = rsp.asn1.encode('ConfigureISDPRequest', {})
+        if self.upp:
+            smr_bin = self.upp.metadata.gen_store_metadata_request()
+        else:
+            smr_bin = self.ppp.metadata.gen_store_metadata_request()
+
+        # we don't use rsp.asn1.encode('boundProfilePackage') here, as the BSP already provides
+        # fully encoded + MACed TLVs including their tag + length values.  We cannot put those as
+        # 'value' input into an ASN.1 encoder, as that would double the TAG + LENGTH :(
+
+        # 'initialiseSecureChannelRequest'
+        bpp_seq = rsp.asn1.encode('InitialiseSecureChannelRequest', iscr)
+        # firstSequenceOf87
+        logger.debug("BPP_ENCODE_DEBUG: Encrypting ConfigureISDP with BSP keys")
+        logger.debug(f"BPP_ENCODE_DEBUG: BSP S-ENC: {bsp.c_algo.s_enc.hex()}")
+        logger.debug(f"BPP_ENCODE_DEBUG: BSP S-MAC: {bsp.m_algo.s_mac.hex()}")
+        bpp_seq += encode_seq(0xa0, bsp.encrypt_and_mac(0x87, conf_idsp_bin))
+        # sequenceOF88
+        logger.debug("BPP_ENCODE_DEBUG: MAC-only StoreMetadata with BSP keys")
+        bpp_seq += encode_seq(0xa1, bsp.mac_only(0x88, smr_bin))
+
+        if self.ppp: # we have to use session keys
+            rsk_bin = gen_replace_session_keys(self.ppp.ppk_enc, self.ppp.ppk_mac, self.ppp.initial_mcv)
+            # secondSequenceOf87
+            bpp_seq += encode_seq(0xa2, bsp.encrypt_and_mac(0x87, rsk_bin))
+        else:
+            self.ppp = ProtectedProfilePackage.from_upp(self.upp, bsp)
+
+        # 'sequenceOf86'
+        bpp_seq += encode_seq(0xa3, self.ppp.encoded)
+
+        # manual DER encode: wrap in outer SEQUENCE
+        return bertlv_encode_tag(0xbf36) + bertlv_encode_len(len(bpp_seq)) + bpp_seq
+
+    def decode(self, euicc_ot, eid: str, bpp_bin: bytes):
+        """Decode a BPP into the PPP and subsequently UPP. This is what happens inside an eUICC."""
+
+        def split_bertlv_sequence(sequence: bytes) -> List[bytes]:
+            remainder = sequence
+            ret = []
+            while remainder:
+                _tag, _l, tlv, remainder = bertlv_return_one_rawtlv(remainder)
+                ret.append(tlv)
+            return ret
+
+        # we don't use rsp.asn1.decode('boundProfilePackage') here, as the BSP needs
+        # fully encoded + MACed TLVs including their tag + length values.
+        #bpp = rsp.asn1.decode('BoundProfilePackage', bpp_bin)
+
+        tag, _l, v, _remainder = bertlv_parse_one_rawtag(bpp_bin)
+        if len(_remainder):
+            raise ValueError('Excess data at end of TLV')
+        if tag != 0xbf36:
+            raise ValueError('Unexpected outer tag: %s' % tag)
+
+        # InitialiseSecureChannelRequest
+        tag, _l, iscr_bin, remainder = bertlv_return_one_rawtlv(v)
+        iscr = rsp.asn1.decode('InitialiseSecureChannelRequest', iscr_bin)
+
+        # configureIsdpRequest
+        tag, _l, firstSeqOf87, remainder = bertlv_parse_one_rawtag(remainder)
+        if tag != 0xa0:
+            raise ValueError("Unexpected 'firstSequenceOf87' tag: %s" % tag)
+        firstSeqOf87 = split_bertlv_sequence(firstSeqOf87)
+
+        # storeMetadataRequest
+        tag, _l, seqOf88, remainder = bertlv_parse_one_rawtag(remainder)
+        if tag != 0xa1:
+            raise ValueError("Unexpected 'sequenceOf88' tag: %s" % tag)
+        seqOf88 = split_bertlv_sequence(seqOf88)
+
+        tag, _l, tlv, remainder = bertlv_parse_one_rawtag(remainder)
+        if tag == 0xa2:
+            secondSeqOf87 = split_bertlv_sequence(tlv)
+            tag2, _l, seqOf86, remainder = bertlv_parse_one_rawtag(remainder)
+            if tag2 != 0xa3:
+                raise ValueError("Unexpected 'sequenceOf86' tag: %s" % tag)
+            seqOf86 = split_bertlv_sequence(seqOf86)
+        elif tag == 0xa3:
+            secondSeqOf87 = None
+            seqOf86 = split_bertlv_sequence(tlv)
+        else:
+            raise ValueError("Unexpected 'secondSequenceOf87' tag: %s" % tag)
+
+        # extract smdoOtpk from initialiseSecureChannel
+        smdp_otpk = iscr['smdpOtpk']
+
+        # Generate Session Keys using the CRT, opPK.DP.ECKA and otSK.EUICC.ECKA according to annex G
+        smdp_public_key = ec.EllipticCurvePublicKey.from_encoded_point(euicc_ot.curve, smdp_otpk)
+        self.shared_secret = euicc_ot.exchange(ec.ECDH(), smdp_public_key)
+
+        crt = iscr['controlRefTemplate']
+        bsp = BspInstance.from_kdf(self.shared_secret, int.from_bytes(crt['keyType'], 'big'), int.from_bytes(crt['keyLen'], 'big'), crt['hostId'], h2b(eid))
+
+        self.encoded_configureISDPRequest = bsp.demac_and_decrypt(firstSeqOf87)
+        self.configureISDPRequest = rsp.asn1.decode('ConfigureISDPRequest', self.encoded_configureISDPRequest)
+
+        self.encoded_storeMetadataRequest = bsp.demac_only(seqOf88)
+        self.storeMetadataRequest = rsp.asn1.decode('StoreMetadataRequest', self.encoded_storeMetadataRequest)
+
+        if secondSeqOf87 != None:
+            rsk_bin = bsp.demac_and_decrypt(secondSeqOf87)
+            rsk = rsp.asn1.decode('ReplaceSessionKeysRequest', rsk_bin)
+            # process replace_session_keys!
+            bsp = BspInstance(rsk['ppkEnc'], rsk['ppkCmac'], rsk['initialMacChainingValue'])
+            self.replaceSessionKeysRequest = rsk
+
+        self.upp = bsp.demac_and_decrypt(seqOf86)
+        return self.upp

pySim/esim/es9p.py

@@ -0,0 +1,177 @@
+"""GSMA eSIM RSP ES9+ interface according to SGP.22 v2.5"""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import requests
+import logging
+import time
+
+import pySim.esim.rsp as rsp
+from pySim.esim.http_json_api import *
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)
+
+class param:
+    class RspAsn1Par(ApiParamBase64):
+        """Generalized RSP ASN.1 parameter: base64-wrapped ASN.1 DER. Derived classes must provide
+        the asn1_type class variable to indicate the name of the ASN.1 type to use for encode/decode."""
+        asn1_type = None # must be overridden by derived class
+
+        @classmethod
+        def _decode(cls, data):
+            data = ApiParamBase64.decode(data)
+            return rsp.asn1.decode(cls.asn1_type, data)
+
+        @classmethod
+        def _encode(cls, data):
+            data = rsp.asn1.encode(cls.asn1_type, data)
+            return ApiParamBase64.encode(data)
+
+    class EuiccInfo1(RspAsn1Par):
+        asn1_type = 'EUICCInfo1'
+
+    class ServerSigned1(RspAsn1Par):
+        asn1_type = 'ServerSigned1'
+
+    class PrepareDownloadResponse(RspAsn1Par):
+        asn1_type = 'PrepareDownloadResponse'
+
+    class AuthenticateServerResponse(RspAsn1Par):
+        asn1_type = 'AuthenticateServerResponse'
+
+    class SmdpSigned2(RspAsn1Par):
+        asn1_type = 'SmdpSigned2'
+
+    class StoreMetadataRequest(RspAsn1Par):
+        asn1_type = 'StoreMetadataRequest'
+
+    class PendingNotification(RspAsn1Par):
+        asn1_type = 'PendingNotification'
+
+    class CancelSessionResponse(RspAsn1Par):
+        asn1_type = 'CancelSessionResponse'
+
+    class TransactionId(ApiParamString):
+        pass
+
+class Es9PlusApiFunction(JsonHttpApiFunction):
+    pass
+
+# ES9+ InitiateAuthentication function (SGP.22 section 6.5.2.6)
+class InitiateAuthentication(Es9PlusApiFunction):
+    path = '/gsma/rsp2/es9plus/initiateAuthentication'
+    extra_http_req_headers = { 'User-Agent': 'gsma-rsp-lpad' }
+    input_params = {
+        'euiccChallenge': ApiParamBase64,
+        'euiccInfo1': param.EuiccInfo1,
+        'smdpAddress': SmdpAddress,
+      }
+    input_mandatory = ['euiccChallenge', 'euiccInfo1', 'smdpAddress']
+    output_params = {
+        'header': JsonResponseHeader,
+        'transactionId': param.TransactionId,
+        'serverSigned1': param.ServerSigned1,
+        'serverSignature1': ApiParamBase64,
+        'euiccCiPKIdToBeUsed': ApiParamBase64,
+        'serverCertificate': ApiParamBase64,
+      }
+    output_mandatory = ['header', 'transactionId', 'serverSigned1', 'serverSignature1',
+                        'euiccCiPKIdToBeUsed', 'serverCertificate']
+
+# ES9+ GetBoundProfilePackage function (SGP.22 section 6.5.2.7)
+class GetBoundProfilePackage(Es9PlusApiFunction):
+    path = '/gsma/rsp2/es9plus/getBoundProfilePackage'
+    extra_http_req_headers = { 'User-Agent': 'gsma-rsp-lpad' }
+    input_params = {
+        'transactionId': param.TransactionId,
+        'prepareDownloadResponse': param.PrepareDownloadResponse,
+      }
+    input_mandatory = ['transactionId', 'prepareDownloadResponse']
+    output_params = {
+        'header': JsonResponseHeader,
+        'transactionId': param.TransactionId,
+        'boundProfilePackage': ApiParamBase64,
+      }
+    output_mandatory = ['header', 'transactionId', 'boundProfilePackage']
+
+# ES9+ AuthenticateClient function (SGP.22 section 6.5.2.8)
+class AuthenticateClient(Es9PlusApiFunction):
+    path= '/gsma/rsp2/es9plus/authenticateClient'
+    extra_http_req_headers = { 'User-Agent': 'gsma-rsp-lpad' }
+    input_params = {
+        'transactionId': param.TransactionId,
+        'authenticateServerResponse': param.AuthenticateServerResponse,
+      }
+    input_mandatory = ['transactionId', 'authenticateServerResponse']
+    output_params = {
+        'header': JsonResponseHeader,
+        'transactionId': param.TransactionId,
+        'profileMetadata': param.StoreMetadataRequest,
+        'smdpSigned2': param.SmdpSigned2,
+        'smdpSignature2': ApiParamBase64,
+        'smdpCertificate': ApiParamBase64,
+      }
+    output_mandatory = ['header', 'transactionId', 'profileMetadata', 'smdpSigned2',
+                        'smdpSignature2', 'smdpCertificate']
+
+# ES9+ HandleNotification function (SGP.22 section 6.5.2.9)
+class HandleNotification(Es9PlusApiFunction):
+    path = '/gsma/rsp2/es9plus/handleNotification'
+    extra_http_req_headers = { 'User-Agent': 'gsma-rsp-lpad' }
+    input_params = {
+        'pendingNotification': param.PendingNotification,
+      }
+    input_mandatory = ['pendingNotification']
+    expected_http_status = 204
+
+# ES9+ CancelSession function (SGP.22 section 6.5.2.10)
+class CancelSession(Es9PlusApiFunction):
+    path = '/gsma/rsp2/es9plus/cancelSession'
+    extra_http_req_headers = { 'User-Agent': 'gsma-rsp-lpad' }
+    input_params = {
+        'transactionId': param.TransactionId,
+        'cancelSessionResponse': param.CancelSessionResponse,
+      }
+    input_mandatory = ['transactionId', 'cancelSessionResponse']
+
+class Es9pApiClient:
+    def __init__(self, url_prefix:str, server_cert_verify: str = None):
+        self.session = requests.Session()
+        self.session.verify = False # FIXME HACK
+        if server_cert_verify:
+            self.session.verify = server_cert_verify
+
+        self.initiateAuthentication = JsonHttpApiClient(InitiateAuthentication(), url_prefix, '', self.session)
+        self.authenticateClient = JsonHttpApiClient(AuthenticateClient(), url_prefix, '', self.session)
+        self.getBoundProfilePackage = JsonHttpApiClient(GetBoundProfilePackage(), url_prefix, '', self.session)
+        self.handleNotification = JsonHttpApiClient(HandleNotification(), url_prefix, '', self.session)
+        self.cancelSession = JsonHttpApiClient(CancelSession(), url_prefix, '', self.session)
+
+    def call_initiateAuthentication(self, data: dict) -> dict:
+        return self.initiateAuthentication.call(data)
+
+    def call_authenticateClient(self, data: dict) -> dict:
+        return self.authenticateClient.call(data)
+
+    def call_getBoundProfilePackage(self, data: dict) -> dict:
+        return self.getBoundProfilePackage.call(data)
+
+    def call_handleNotification(self, data: dict) -> dict:
+        return self.handleNotification.call(data)
+
+    def call_cancelSession(self, data: dict) -> dict:
+        return self.cancelSession.call(data)

pySim/esim/http_json_api.py

@@ -0,0 +1,464 @@
+"""GSMA eSIM RSP HTTP/REST/JSON interface according to SGP.22 v2.5"""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+import requests
+import logging
+import json
+from re import match
+from typing import Optional
+import base64
+from twisted.web.server import Request
+
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)
+
+class ApiParam(abc.ABC):
+    """A class representing a single parameter in the API."""
+    @classmethod
+    def verify_decoded(cls, data):
+        """Verify the decoded representation of a value. Should raise an exception if something is odd."""
+        pass
+
+    @classmethod
+    def verify_encoded(cls, data):
+        """Verify the encoded representation of a value. Should raise an exception if something is odd."""
+        pass
+
+    @classmethod
+    def encode(cls, data):
+        """[Validate and] Encode the given value."""
+        cls.verify_decoded(data)
+        encoded = cls._encode(data)
+        cls.verify_decoded(encoded)
+        return encoded
+
+    @classmethod
+    def _encode(cls, data):
+        """encoder function, typically [but not always] overridden by derived class."""
+        return data
+
+    @classmethod
+    def decode(cls, data):
+        """[Validate and] Decode the given value."""
+        cls.verify_encoded(data)
+        decoded = cls._decode(data)
+        cls.verify_decoded(decoded)
+        return decoded
+
+    @classmethod
+    def _decode(cls, data):
+        """decoder function, typically [but not always] overridden by derived class."""
+        return data
+
+class ApiParamString(ApiParam):
+    """Base class representing an API parameter of 'string' type."""
+    pass
+
+
+class ApiParamInteger(ApiParam):
+    """Base class representing an API parameter of 'integer' type."""
+    @classmethod
+    def _decode(cls, data):
+        return int(data)
+
+    @classmethod
+    def _encode(cls, data):
+        return str(data)
+
+    @classmethod
+    def verify_decoded(cls, data):
+        if not isinstance(data, int):
+            raise TypeError('Expected an integer input data type')
+
+    @classmethod
+    def verify_encoded(cls, data):
+        if isinstance(data, int):
+            return
+        if not data.isdecimal():
+            raise ValueError('integer (%s) contains non-decimal characters' % data)
+        assert str(int(data)) == data
+
+class ApiParamBoolean(ApiParam):
+    """Base class representing an API parameter of 'boolean' type."""
+    @classmethod
+    def _encode(cls, data):
+        return bool(data)
+
+class ApiParamFqdn(ApiParam):
+    """String, as a list of domain labels concatenated using the full stop (dot, period) character as
+    separator between labels. Labels are restricted to the Alphanumeric mode character set defined in table 5
+    of ISO/IEC 18004"""
+    @classmethod
+    def verify_encoded(cls, data):
+        # FIXME
+        pass
+
+class ApiParamBase64(ApiParam):
+    @classmethod
+    def _decode(cls, data):
+        return base64.b64decode(data)
+
+    @classmethod
+    def _encode(cls, data):
+        return base64.b64encode(data).decode('ascii')
+
+class SmdpAddress(ApiParamFqdn):
+    pass
+
+class JsonResponseHeader(ApiParam):
+    """SGP.22 section 6.5.1.4."""
+    @classmethod
+    def verify_decoded(cls, data):
+        fe_status = data.get('functionExecutionStatus')
+        if not fe_status:
+            raise ValueError('Missing mandatory functionExecutionStatus in header')
+        status = fe_status.get('status')
+        if not status:
+            raise ValueError('Missing mandatory status in header functionExecutionStatus')
+        if status not in ['Executed-Success', 'Executed-WithWarning', 'Failed', 'Expired']:
+            raise ValueError('Unknown/unspecified status "%s"' % status)
+
+class JsonRequestHeader(ApiParam):
+    """SGP.22 section 6.5.1.3."""
+    @classmethod
+    def verify_decoded(cls, data):
+        func_req_id = data.get('functionRequesterIdentifier')
+        if not func_req_id:
+            raise ValueError('Missing mandatory functionRequesterIdentifier in header')
+        func_call_id = data.get('functionCallIdentifier')
+        if not func_call_id:
+            raise ValueError('Missing mandatory functionCallIdentifier in header')
+
+class HttpStatusError(Exception):
+    pass
+
+class HttpHeaderError(Exception):
+    pass
+
+class ApiError(Exception):
+    """Exception representing an error at the API level (status != Executed)."""
+    def __init__(self, func_ex_status: dict):
+        self.status = func_ex_status['status']
+        sec = {
+            'subjectCode': None,
+            'reasonCode': None,
+            'subjectIdentifier': None,
+            'message': None,
+        }
+        actual_sec = func_ex_status.get('statusCodeData', None)
+        if actual_sec:
+            sec.update(actual_sec)
+        self.subject_code = sec['subjectCode']
+        self.reason_code = sec['reasonCode']
+        self.subject_id = sec['subjectIdentifier']
+        self.message = sec['message']
+
+    def __str__(self):
+        return f'{self.status}("{self.subject_code}","{self.reason_code}","{self.subject_id}","{self.message}")'
+
+class JsonHttpApiFunction(abc.ABC):
+    """Base class for representing an HTTP[s] API Function."""
+    # The below class variables are used to describe the properties of the API function. Derived classes are expected
+    # to orverride those class properties with useful values. The prefixes "input_" and "output_" refer to the API
+    # function from an abstract point of view. Seen from the client perspective, "input_" will refer to parameters the
+    # client sends to a HTTP server. Seen from the server perspective, "input_" will refer to parameters the server
+    # receives from the a requesting client. The same applies vice versa to class variables that have an "output_"
+    # prefix.
+
+    # path of the API function (e.g. '/gsma/rsp2/es2plus/confirmOrder')
+    path = None
+
+    # dictionary of input parameters. key is parameter name, value is ApiParam class
+    input_params = {}
+
+    # list of mandatory input parameters
+    input_mandatory = []
+
+    # dictionary of output parameters. key is parameter name, value is ApiParam class
+    output_params = {}
+
+    # list of mandatory output parameters (for successful response)
+    output_mandatory = []
+
+    # list of mandatory output parameters (for failed response)
+    output_mandatory_failed = []
+
+    # expected HTTP status code of the response
+    expected_http_status = 200
+
+    # the HTTP method used (GET, OPTIONS, HEAD, POST, PUT, PATCH or DELETE)
+    http_method = 'POST'
+
+    # additional custom HTTP headers (client requests)
+    extra_http_req_headers = {}
+
+    # additional custom HTTP headers (server responses)
+    extra_http_res_headers = {}
+
+    def __new__(cls, *args, role = None, **kwargs):
+        """
+        Args:
+                args: (see JsonHttpApiClient and JsonHttpApiServer)
+                role: role ('server' or 'client') in which the JsonHttpApiFunction should be created.
+                kwargs: (see JsonHttpApiClient and JsonHttpApiServer)
+        """
+
+        # Create a dictionary with the class attributes of this class (the properties listed above and the encode_
+        # decode_ methods below). The dictionary will not include any dunder/magic methods
+        cls_attr = { attr_name: getattr(cls, attr_name) for attr_name in dir(cls) if not match("__.*__", attr_name) }
+
+        # Normal instantiation as JsonHttpApiFunction:
+        if len(args) == 0:
+            return type(cls.__name__, (abc.ABC,), cls_attr)()
+
+        # Instantiation as as JsonHttpApiFunction with a JsonHttpApiClient or JsonHttpApiServer base
+        role = kwargs.get('role', 'legacy_client')
+        if role == 'legacy_client':
+            # Deprecated: With the advent of the server role (JsonHttpApiServer) the API had to be changed. To maintain
+            # compatibility with existing code (out-of-tree) the original behaviour and API interface and behaviour had
+            # to be preserved. Already existing JsonHttpApiFunction definitions will still work and the related objects
+            # may still be created on the original way: my_api_func = MyApiFunc(url_prefix, func_req_id, self.session)
+            logger.warning('implicit role (falling back to legacy JsonHttpApiClient) is deprecated, please specify role explcitly')
+            result = type(cls.__name__, (JsonHttpApiClient,), cls_attr)(None, *args, **kwargs)
+            result.api_func = result
+            result.legacy = True
+            return result
+        elif role == 'client':
+            # Create a JsonHttpApiFunction in client role
+            # Example: my_api_func = MyApiFunc(url_prefix, func_req_id, self.session, role='client')
+            result = type(cls.__name__, (JsonHttpApiClient,), cls_attr)(None, *args, **kwargs)
+            result.api_func = result
+            return result
+        elif role == 'server':
+            # Create a JsonHttpApiFunction in server role
+            # Example: my_api_func = MyApiFunc(url_prefix, func_req_id, self.session, role='server')
+            result = type(cls.__name__, (JsonHttpApiServer,), cls_attr)(None, *args, **kwargs)
+            result.api_func = result
+            return result
+        else:
+            raise ValueError('Invalid role \'%s\' specified' % role)
+
+    def encode_client(self, data: dict) -> dict:
+        """Validate an encode input dict into JSON-serializable dict for request body."""
+        output = {}
+        for p in self.input_mandatory:
+            if not p in data:
+                raise ValueError('Mandatory input parameter %s missing' % p)
+        for p, v in data.items():
+            p_class = self.input_params.get(p)
+            if not p_class:
+                # pySim/esim/http_json_api.py:269:47: E1101: Instance of 'JsonHttpApiFunction' has no 'legacy' member (no-member)
+                # pylint: disable=no-member
+                if hasattr(self, 'legacy') and self.legacy:
+                    output[p] = JsonRequestHeader.encode(v)
+                else:
+                    logger.warning('Unexpected/unsupported input parameter %s=%s', p, v)
+                    output[p] = v
+            else:
+                output[p] = p_class.encode(v)
+        return output
+
+    def decode_client(self, data: dict) -> dict:
+        """[further] Decode and validate the JSON-Dict of the response body."""
+        output = {}
+        output_mandatory = self.output_mandatory
+
+        # In case a provided header (may be optional) indicates that the API function call was unsuccessful, a
+        # different set of mandatory parameters applies.
+        header = data.get('header')
+        if header:
+            if data['header']['functionExecutionStatus']['status'] not in ['Executed-Success','Executed-WithWarning']:
+                output_mandatory = self.output_mandatory_failed
+
+        for p in output_mandatory:
+            if not p in data:
+                raise ValueError('Mandatory output parameter "%s" missing' % p)
+        for p, v in data.items():
+            p_class = self.output_params.get(p)
+            if not p_class:
+                logger.warning('Unexpected/unsupported output parameter "%s"="%s"', p, v)
+                output[p] = v
+            else:
+                output[p] = p_class.decode(v)
+        return output
+
+    def encode_server(self, data: dict) -> dict:
+        """Validate an encode input dict into JSON-serializable dict for response body."""
+        output = {}
+        output_mandatory = self.output_mandatory
+
+        # In case a provided header (may be optional) indicates that the API function call was unsuccessful, a
+        # different set of mandatory parameters applies.
+        header = data.get('header')
+        if header:
+            if data['header']['functionExecutionStatus']['status'] not in ['Executed-Success','Executed-WithWarning']:
+                output_mandatory = self.output_mandatory_failed
+
+        for p in output_mandatory:
+            if not p in data:
+                raise ValueError('Mandatory output parameter %s missing' % p)
+        for p, v in data.items():
+            p_class = self.output_params.get(p)
+            if not p_class:
+                logger.warning('Unexpected/unsupported output parameter %s=%s', p, v)
+                output[p] = v
+            else:
+                output[p] = p_class.encode(v)
+        return output
+
+    def decode_server(self, data: dict) -> dict:
+        """[further] Decode and validate the JSON-Dict of the request body."""
+        output = {}
+
+        for p in self.input_mandatory:
+            if not p in data:
+                raise ValueError('Mandatory input parameter "%s" missing' % p)
+        for p, v in data.items():
+            p_class = self.input_params.get(p)
+            if not p_class:
+                logger.warning('Unexpected/unsupported input parameter "%s"="%s"', p, v)
+                output[p] = v
+            else:
+                output[p] = p_class.decode(v)
+        return output
+
+class JsonHttpApiClient():
+    def __init__(self, api_func: JsonHttpApiFunction, url_prefix: str, func_req_id: Optional[str],
+                 session: requests.Session):
+        """
+        Args:
+                api_func : API function definition (JsonHttpApiFunction)
+                url_prefix : prefix to be put in front of the API function path (see JsonHttpApiFunction)
+                func_req_id : function requestor id to use for requests
+                session : session object (requests)
+        """
+        self.api_func = api_func
+        self.url_prefix = url_prefix
+        self.func_req_id = func_req_id
+        self.session = session
+
+    def call(self, data: dict, func_call_id: Optional[str] = None, timeout=10) -> Optional[dict]:
+        """Make an API call to the HTTP API endpoint represented by this object. Input data is passed in `data` as
+        json-serializable dict. Output data is returned as json-deserialized dict."""
+
+        # In case a function caller ID is supplied, use it together with the stored function requestor ID to generate
+        # and prepend the header field according to SGP.22, section 6.5.1.1 and 6.5.1.3. (the presence of the header
+        # field is checked by the encode_client method)
+        if func_call_id:
+            data = {'header' : {'functionRequesterIdentifier': self.func_req_id,
+                                'functionCallIdentifier': func_call_id}} | data
+
+        # Encode the message (the presence of mandatory fields is checked during encoding)
+        encoded = json.dumps(self.api_func.encode_client(data))
+
+        # Apply HTTP request headers according to SGP.22, section 6.5.1
+        req_headers = {
+            'Content-Type': 'application/json',
+            'X-Admin-Protocol': 'gsma/rsp/v2.5.0',
+        }
+        req_headers.update(self.api_func.extra_http_req_headers)
+
+        # Perform HTTP request
+        url = self.url_prefix + self.api_func.path
+        logger.debug("HTTP REQ %s - hdr: %s '%s'" % (url, req_headers, encoded))
+        response = self.session.request(self.api_func.http_method, url, data=encoded, headers=req_headers, timeout=timeout)
+        logger.debug("HTTP RSP-STS: [%u] hdr: %s" % (response.status_code, response.headers))
+        logger.debug("HTTP RSP: %s" % (response.content))
+
+        # Check HTTP response status code and make sure that the returned HTTP headers look plausible (according to
+        # SGP.22, section 6.5.1)
+        if response.status_code != self.api_func.expected_http_status:
+            raise HttpStatusError(response)
+        if response.content and not response.headers.get('Content-Type').startswith(req_headers['Content-Type']):
+            raise HttpHeaderError(response)
+        if not response.headers.get('X-Admin-Protocol', 'gsma/rsp/v2.unknown').startswith('gsma/rsp/v2.'):
+            raise HttpHeaderError(response)
+
+        # Decode response and return the result back to the caller
+        if response.content:
+            output = self.api_func.decode_client(response.json())
+            # In case the response contains a header, check it to make sure that the API call was executed successfully
+            # (the presence of the header field is checked by the decode_client method)
+            if 'header' in output:
+                if output['header']['functionExecutionStatus']['status'] not in ['Executed-Success','Executed-WithWarning']:
+                    raise ApiError(output['header']['functionExecutionStatus'])
+            return output
+        return None
+
+class JsonHttpApiServer():
+    def __init__(self, api_func: JsonHttpApiFunction, call_handler = None):
+        """
+        Args:
+                api_func : API function definition (JsonHttpApiFunction)
+                call_handler : handler function to process the request. This function must accept the
+                               decoded request as a dictionary. The handler function must return a tuple consisting
+                               of the response in the form of a dictionary (may be empty), and a function execution
+                               status string ('Executed-Success', 'Executed-WithWarning', 'Failed' or 'Expired')
+        """
+        self.api_func = api_func
+        if call_handler:
+            self.call_handler = call_handler
+        else:
+            self.call_handler = self.default_handler
+
+    def default_handler(self, data: dict) -> (dict, str):
+        """default handler, used in case no call handler is provided."""
+        logger.error("no handler function for request: %s" % str(data))
+        return {}, 'Failed'
+
+    def call(self, request: Request) -> str:
+        """ Process an incoming request.
+        Args:
+                request : request object as received using twisted.web.server
+        Returns:
+                encoded JSON string (HTTP response code and headers are set by calling the appropriate methods on the
+                provided the request object)
+        """
+
+        # Make sure the request is done with the correct HTTP method
+        if (request.method.decode() != self.api_func.http_method):
+            raise ValueError('Wrong HTTP method %s!=%s' % (request.method.decode(), self.api_func.http_method))
+
+        # Decode the request
+        decoded_request = self.api_func.decode_server(json.loads(request.content.read()))
+
+        # Run call handler (see above)
+        data, fe_status = self.call_handler(decoded_request)
+
+        # In case a function execution status is returned, use it to generate and prepend the header field according to
+        # SGP.22, section 6.5.1.2 and 6.5.1.4 (the presence of the header filed is checked by the encode_server method)
+        if fe_status:
+            data = {'header' : {'functionExecutionStatus': {'status' : fe_status}}} | data
+
+        # Encode the message (the presence of mandatory fields is checked during encoding)
+        encoded = json.dumps(self.api_func.encode_server(data))
+
+        # Apply HTTP request headers according to SGP.22, section 6.5.1
+        res_headers = {
+            'Content-Type': 'application/json',
+            'X-Admin-Protocol': 'gsma/rsp/v2.5.0',
+        }
+        res_headers.update(self.api_func.extra_http_res_headers)
+        for header, value in res_headers.items():
+            request.setHeader(header, value)
+        request.setResponseCode(self.api_func.expected_http_status)
+
+        # Return the encoded result back to the caller for sending (using twisted/klein)
+        return encoded
+

pySim/esim/rsp.py

@@ -0,0 +1,179 @@
+"""Implementation of GSMA eSIM RSP (Remote SIM Provisioning) as per SGP22 v3.0"""
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from typing import Optional
+import shelve
+
+from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.primitives.serialization import Encoding
+from cryptography import x509
+from osmocom.utils import b2h
+from osmocom.tlv import bertlv_parse_one_rawtag, bertlv_return_one_rawtlv
+
+from pySim.esim import compile_asn1_subdir
+
+asn1 = compile_asn1_subdir('rsp')
+
+class RspSessionState:
+    """Encapsulates the state of a RSP session.  It is created during the initiateAuthentication
+    and subsequently used by further API calls using the same transactionId.  The session state
+    is removed either after cancelSession or after notification.
+    TODO: add some kind of time based expiration / garbage collection."""
+    def __init__(self, transactionId: str, serverChallenge: bytes, ci_cert_id: bytes):
+        self.transactionId = transactionId
+        self.serverChallenge = serverChallenge
+        #  used at a later point between API calls
+        self.ci_cert_id = ci_cert_id
+        self.euicc_cert: Optional[x509.Certificate] = None
+        self.eum_cert: Optional[x509.Certificate] = None
+        self.eid: Optional[bytes] = None
+        self.profileMetadata: Optional['ProfileMetadata'] = None
+        self.smdpSignature2_do = None
+        # really only needed while processing getBoundProfilePackage request?
+        self.euicc_otpk: Optional[bytes] = None
+        self.smdp_ot: Optional[ec.EllipticCurvePrivateKey] = None
+        self.smdp_otpk: Optional[bytes] = None
+        self.host_id: Optional[bytes] = None
+        self.shared_secret: Optional[bytes] = None
+
+
+    def __getstate__(self):
+        """helper function called when pickling the object to persistent storage.  We must pickel all
+        members that are not pickle-able."""
+        state = self.__dict__.copy()
+        # serialize eUICC certificate as DER
+        if state.get('euicc_cert', None):
+            state['_euicc_cert'] = self.euicc_cert.public_bytes(Encoding.DER)
+            del state['euicc_cert']
+        # serialize EUM certificate as DER
+        if state.get('eum_cert', None):
+            state['_eum_cert'] = self.eum_cert.public_bytes(Encoding.DER)
+            del state['eum_cert']
+        # serialize one-time SMDP private key to integer + curve
+        if state.get('smdp_ot', None):
+            state['_smdp_otsk'] = self.smdp_ot.private_numbers().private_value
+            state['_smdp_ot_curve'] = self.smdp_ot.curve
+            del state['smdp_ot']
+        return state
+
+    def __setstate__(self, state):
+        """helper function called when unpickling the object from persistent storage. We must recreate all
+        members from the state generated in __getstate__ above."""
+        # restore eUICC certificate from DER
+        if '_euicc_cert' in state:
+            self.euicc_cert = x509.load_der_x509_certificate(state['_euicc_cert'])
+            del state['_euicc_cert']
+        else:
+            self.euicc_cert = None
+        # restore EUM certificate from DER
+        if '_eum_cert' in state:
+            self.eum_cert = x509.load_der_x509_certificate(state['_eum_cert'])
+            del state['_eum_cert']
+        # restore one-time SMDP private key from integer + curve
+        if state.get('_smdp_otsk', None):
+            self.smdp_ot = ec.derive_private_key(state['_smdp_otsk'], state['_smdp_ot_curve'])
+            # FIXME: how to add the public key from smdp_otpk to an instance of EllipticCurvePrivateKey?
+            del state['_smdp_otsk']
+            del state['_smdp_ot_curve']
+        # automatically recover all the remaining state
+        self.__dict__.update(state)
+
+
+class RspSessionStore:
+    """A wrapper around the database-backed storage 'shelve' for storing RspSessionState objects.
+    Can be configured to use either file-based storage or in-memory storage.
+    We use it to store RspSessionState objects indexed by transactionId."""
+
+    def __init__(self, filename: Optional[str] = None, in_memory: bool = False):
+        self._in_memory = in_memory
+
+        if in_memory:
+            self._shelf = shelve.Shelf(dict())
+        else:
+            if filename is None:
+                raise ValueError("filename is required for file-based session store")
+            self._shelf = shelve.open(filename)
+
+    # dunder magic
+    def __getitem__(self, key):
+        return self._shelf[key]
+
+    def __setitem__(self, key, value):
+        self._shelf[key] = value
+
+    def __delitem__(self, key):
+        del self._shelf[key]
+
+    def __contains__(self, key):
+        return key in self._shelf
+
+    def __iter__(self):
+        return iter(self._shelf)
+
+    def __len__(self):
+        return len(self._shelf)
+
+    # everything else
+    def __getattr__(self, name):
+        """Delegate attribute access to the underlying shelf object."""
+        return getattr(self._shelf, name)
+
+    def close(self):
+        """Close the session store."""
+        if hasattr(self._shelf, 'close'):
+            self._shelf.close()
+        if self._in_memory:
+            # For in-memory store, clear the reference
+            self._shelf = None
+
+    def sync(self):
+        """Synchronize the cache with the underlying storage."""
+        if hasattr(self._shelf, 'sync'):
+            self._shelf.sync()
+
+def extract_euiccSigned1(authenticateServerResponse: bytes) -> bytes:
+    """Extract the raw, DER-encoded binary euiccSigned1 field from the given AuthenticateServerResponse. This
+    is needed due to the very peculiar SGP.22 notion of signing sections of DER-encoded ASN.1 objects."""
+    rawtag, l, v, remainder = bertlv_parse_one_rawtag(authenticateServerResponse)
+    if len(remainder):
+        raise ValueError('Excess data at end of TLV')
+    if rawtag != 0xbf38:
+        raise ValueError('Unexpected outer tag: %s' % b2h(rawtag))
+    rawtag, l, v1, remainder = bertlv_parse_one_rawtag(v)
+    if rawtag != 0xa0:
+        raise ValueError('Unexpected tag where CHOICE was expected')
+    rawtag, l, tlv2, remainder = bertlv_return_one_rawtlv(v1)
+    if rawtag != 0x30:
+        raise ValueError('Unexpected tag where SEQUENCE was expected')
+    return tlv2
+
+def extract_euiccSigned2(prepareDownloadResponse: bytes) -> bytes:
+    """Extract the raw, DER-encoded binary euiccSigned2 field from the given prepareDownloadrResponse. This is
+    needed due to the very peculiar SGP.22 notion of signing sections of DER-encoded ASN.1 objects."""
+    rawtag, l, v, remainder = bertlv_parse_one_rawtag(prepareDownloadResponse)
+    if len(remainder):
+        raise ValueError('Excess data at end of TLV')
+    if rawtag != 0xbf21:
+        raise ValueError('Unexpected outer tag: %s' % b2h(rawtag))
+    rawtag, l, v1, remainder = bertlv_parse_one_rawtag(v)
+    if rawtag != 0xa0:
+        raise ValueError('Unexpected tag where CHOICE was expected')
+    rawtag, l, tlv2, remainder = bertlv_return_one_rawtlv(v1)
+    if rawtag != 0x30:
+        raise ValueError('Unexpected tag where SEQUENCE was expected')
+    return tlv2

pySim/esim/saip/oid.py

@@ -0,0 +1,120 @@
+"""Implementation of SimAlliance/TCA Interoperable Profile OIDs"""
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from functools import total_ordering
+from typing import List, Union
+
+@total_ordering
+class OID:
+    @staticmethod
+    def intlist_from_str(instr: str) -> List[int]:
+        return [int(x) for x in instr.split('.')]
+
+    @staticmethod
+    def str_from_intlist(intlist: List[int]) -> str:
+        return '.'.join([str(x) for x in intlist])
+
+    @staticmethod
+    def highest_oid(oids: List['OID']) -> 'OID':
+        return sorted(oids)[-1]
+
+    def __init__(self, initializer: Union[List[int], str]):
+        if isinstance(initializer, str):
+            self.intlist = self.intlist_from_str(initializer)
+        else:
+            self.intlist = initializer
+
+    def __str__(self) -> str:
+        return self.str_from_intlist(self.intlist)
+
+    def __repr__(self) -> str:
+        return 'OID(%s)' % (str(self))
+
+    def __eq__(self, other: 'OID'):
+        return (self.intlist == other.intlist)
+
+    def __ne__(self, other: 'OID'):
+        # implement based on __eq__
+        return not (self == other)
+
+    def cmp(self, other: 'OID'):
+        self_len = len(self.intlist)
+        other_len = len(other.intlist)
+        common_len = min(self_len, other_len)
+        max_len = max(self_len, other_len)
+
+        for i in range(0, max_len+1):
+            if i >= self_len:
+                # other list is longer
+                return -1
+            if i >= other_len:
+                # our list is longer
+                return 1
+            if self.intlist[i] > other.intlist[i]:
+                # our version is higher
+                return 1
+            if self.intlist[i] < other.intlist[i]:
+                # other version is higher
+                return -1
+            # continue to next digit
+        return 0
+
+    def __gt__(self, other: 'OID'):
+        if self.cmp(other) > 0:
+            return True
+
+    def prefix_match(self, oid_str: Union[str, 'OID']):
+        """determine if oid_str is equal or below our OID."""
+        return str(oid_str).startswith(str(self))
+
+
+class eOID(OID):
+    """OID helper for TCA eUICC prefix"""
+    __prefix = [2,23,143,1]
+    def __init__(self, initializer):
+        if isinstance(initializer, str):
+            initializer = self.intlist_from_str(initializer)
+        super().__init__(self.__prefix + initializer)
+
+MF = eOID("2.1")
+DF_CD = eOID("2.2")
+DF_TELECOM = eOID("2.3")
+DF_TELECOM_v2 = eOID("2.3.2")
+ADF_USIM_by_default = eOID("2.4")
+ADF_USIM_by_default_v2 = eOID("2.4.2")
+ADF_USIMopt_not_by_default = eOID("2.5")
+ADF_USIMopt_not_by_default_v2 = eOID("2.5.2")
+ADF_USIMopt_not_by_default_v3 = eOID("2.5.3")
+DF_PHONEBOOK_ADF_USIM = eOID("2.6")
+DF_GSM_ACCESS_ADF_USIM = eOID("2.7")
+ADF_ISIM_by_default = eOID("2.8")
+ADF_ISIMopt_not_by_default = eOID("2.9")
+ADF_ISIMopt_not_by_default_v2 = eOID("2.9.2")
+ADF_CSIM_by_default = eOID("2.10")
+ADF_CSIM_by_default_v2 = eOID("2.10.2")
+ADF_CSIMopt_not_by_default = eOID("2.11")
+ADF_CSIMopt_not_by_default_v2 = eOID("2.11.2")
+DF_EAP = eOID("2.12")
+DF_5GS = eOID("2.13")
+DF_5GS_v2 = eOID("2.13.2")
+DF_5GS_v3 = eOID("2.13.3")
+DF_5GS_v4 = eOID("2.13.4")
+DF_SAIP = eOID("2.14")
+DF_SNPN = eOID("2.15")
+DF_5GProSe = eOID("2.16")
+IoT_by_default = eOID("2.17")
+IoTopt_not_by_default = eOID("2.18")

pySim/esim/saip/personalization.py

@@ -0,0 +1,668 @@
+"""Implementation of Personalization of eSIM profiles in SimAlliance/TCA Interoperable Profile."""
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+import io
+from typing import List, Tuple
+
+from osmocom.tlv import camel_to_snake
+from pySim.utils import enc_iccid, enc_imsi, h2b, rpad, sanitize_iccid
+from pySim.esim.saip import ProfileElement, ProfileElementSequence
+from pySim.ts_51_011 import EF_SMSP
+
+def remove_unwanted_tuples_from_list(l: List[Tuple], unwanted_keys: List[str]) -> List[Tuple]:
+    """In a list of tuples, remove all tuples whose first part equals 'unwanted_key'."""
+    return list(filter(lambda x: x[0] not in unwanted_keys, l))
+
+def file_replace_content(file: List[Tuple], new_content: bytes):
+    """Completely replace all fillFileContent of a decoded 'File' with the new_content."""
+    # use [:] to avoid making a copy, as we're doing in-place modification of the list here
+    file[:] = remove_unwanted_tuples_from_list(file, ['fillFileContent', 'fillFileOffset'])
+    file.append(('fillFileContent', new_content))
+    return file
+
+class ClassVarMeta(abc.ABCMeta):
+    """Metaclass that puts all additional keyword-args into the class. We use this to have one
+    class definition for something like a PIN, and then have derived classes for PIN1, PIN2, ..."""
+    def __new__(metacls, name, bases, namespace, **kwargs):
+        #print("Meta_new_(metacls=%s, name=%s, bases=%s, namespace=%s, kwargs=%s)" % (metacls, name, bases, namespace, kwargs))
+        x = super().__new__(metacls, name, bases, namespace)
+        for k, v in kwargs.items():
+            setattr(x, k, v)
+        setattr(x, 'name', camel_to_snake(name))
+        return x
+
+class ConfigurableParameter(abc.ABC, metaclass=ClassVarMeta):
+    r"""Base class representing a part of the eSIM profile that is configurable during the
+    personalization process (with dynamic data from elsewhere).
+
+    This class is abstract, you will only use subclasses in practice.
+
+    Subclasses have to implement the apply_val() classmethods, and may choose to override the default validate_val()
+    implementation.
+    The default validate_val() is a generic validator that uses the following class members (defined in subclasses) to
+    configure the validation; if any of them is None, it means that the particular validation is skipped:
+
+    allow_types: a list of types permitted as argument to validate_val(); allow_types = (bytes, str,)
+    allow_chars: if val is a str, accept only these characters; allow_chars = "0123456789"
+    strip_chars: if val is a str, remove these characters; strip_chars = ' \t\r\n'
+    min_len: minimum length of an input str; min_len = 4
+    max_len: maximum length of an input str; max_len = 8
+    allow_len: permit only specific lengths; allow_len = (8, 16, 32)
+
+    Subclasses may change the meaning of these by overriding validate_val(), for example that the length counts
+    resulting bytes instead of a hexstring length. Most subclasses will be covered by the default validate_val().
+
+    Usage examples, by example of Iccid:
+
+    1) use a ConfigurableParameter instance, with .input_value and .value state::
+
+          iccid = Iccid()
+          try:
+            iccid.input_value = '123456789012345678'
+            iccid.validate()
+          except ValueError:
+            print(f"failed to validate {iccid.name} == {iccid.input_value}")
+
+          pes = ProfileElementSequence.from_der(der_data_from_file)
+          try:
+            iccid.apply(pes)
+          except ValueError:
+            print(f"failed to apply {iccid.name} := {iccid.input_value}")
+
+          changed_der = pes.to_der()
+
+    2) use a ConfigurableParameter class, without state::
+
+          cls = Iccid
+          input_val = '123456789012345678'
+
+          try:
+            clean_val = cls.validate_val(input_val)
+          except ValueError:
+            print(f"failed to validate {cls.get_name()} = {input_val}")
+
+          pes = ProfileElementSequence.from_der(der_data_from_file)
+          try:
+            cls.apply_val(pes, clean_val)
+          except ValueError:
+            print(f"failed to apply {cls.get_name()} = {input_val}")
+
+          changed_der = pes.to_der()
+
+    """
+
+    # A subclass can set an explicit string as name (like name = "PIN1").
+    # If name is left None, then __init__() will set self.name to a name derived from the python class name (like
+    # "pin1"). See also the get_name() classmethod when you have no instance at hand.
+    name = None
+    allow_types = (str, int, )
+    allow_chars = None
+    strip_chars = None
+    min_len = None
+    max_len = None
+    allow_len = None # a list of specific lengths
+    example_input = None
+
+    def __init__(self, input_value=None):
+        self.input_value = input_value # the raw input value as given by caller
+        self.value = None # the processed input value (e.g. with check digit) as produced by validate()
+
+        # if there is no explicit name string set, use the class name
+        self.name = self.get_name()
+
+    @classmethod
+    def get_name(cls):
+        """Return cls.name when it is set, otherwise return the python class name converted from 'CamelCase' to
+        'snake_case'.
+        When using class *instances*, you can just use my_instance.name.
+        When using *classes*, cls.get_name() returns the same name a class instance would have.
+        """
+        if cls.name:
+            return cls.name
+        return camel_to_snake(cls.__name__)
+
+    def validate(self):
+        """Validate self.input_value and place the result in self.value.
+        This is also called implicitly by apply(), if self.value is still None.
+        To override validation in a subclass, rather re-implement the classmethod validate_val()."""
+        try:
+            self.value = self.__class__.validate_val(self.input_value)
+        except (TypeError, ValueError, KeyError) as e:
+            raise ValueError(f'{self.name}: {e}') from e
+
+    def apply(self, pes: ProfileElementSequence):
+        """Place self.value into the ProfileElementSequence at the right place.
+        If self.value is None, this implicitly calls self.validate() first, to generate a sanitized self.value from
+        self.input_value.
+        To override apply() in a subclass, rather override the classmethod apply_val()."""
+        if self.value is None:
+            self.validate()
+            assert self.value is not None
+        try:
+            self.__class__.apply_val(pes, self.value)
+        except (TypeError, ValueError, KeyError) as e:
+            raise ValueError(f'{self.name}: {e}') from e
+
+    @classmethod
+    def validate_val(cls, val):
+        """This is a default implementation, with the behavior configured by subclasses' allow_types...max_len settings.
+        subclasses may override this function:
+        Validate the contents of val, and raise ValueError on validation errors.
+        Return a sanitized version of val, that is ready for cls.apply_val().
+        """
+
+        if cls.allow_types is not None:
+            if not isinstance(val, cls.allow_types):
+                raise ValueError(f'input value must be one of {cls.allow_types}, not {type(val)}')
+        elif val is None:
+            raise ValueError('there is no value (val is None)')
+
+        if isinstance(val, str):
+            if cls.strip_chars is not None:
+                val = ''.join(c for c in val if c not in cls.strip_chars)
+            if cls.allow_chars is not None:
+                if any(c not in cls.allow_chars for c in val):
+                    raise ValueError(f"invalid characters in input value {val!r}, valid chars are {cls.allow_chars}")
+        if cls.allow_len is not None:
+            l = cls.allow_len
+            # cls.allow_len could be one int, or a tuple of ints. Wrap a single int also in a tuple.
+            if not isinstance(l, (tuple, list)):
+                l = (l,)
+            if len(val) not in l:
+                raise ValueError(f'length must be one of {cls.allow_len}, not {len(val)}: {val!r}')
+        if cls.min_len is not None:
+            if len(val) < cls.min_len:
+                raise ValueError(f'length must be at least {cls.min_len}, not {len(val)}: {val!r}')
+        if cls.max_len is not None:
+            if len(val) > cls.max_len:
+                raise ValueError(f'length must be at most {cls.max_len}, not {len(val)}: {val!r}')
+        return val
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        """This is what subclasses implement: store a value in a decoded profile package.
+        Write the given val in the right format in all the right places in pes."""
+        pass
+
+    @classmethod
+    def get_len_range(cls):
+        """considering all of min_len, max_len and allow_len, get a tuple of the resulting (min, max) of permitted
+        value length. For example, if an input value is an int, which needs to be represented with a minimum nr of
+        digits, this function is useful to easily get that minimum permitted length.
+        """
+        vals = []
+        if cls.allow_len is not None:
+            if isinstance(cls.allow_len, (tuple, list)):
+                vals.extend(cls.allow_len)
+            else:
+                vals.append(cls.allow_len)
+        if cls.min_len is not None:
+            vals.append(cls.min_len)
+        if cls.max_len is not None:
+            vals.append(cls.max_len)
+        if not vals:
+            return (None, None)
+        return (min(vals), max(vals))
+
+
+class DecimalParam(ConfigurableParameter):
+    """Decimal digits. The input value may be a string of decimal digits like '012345', or an int. The output of
+    validate_val() is a string with only decimal digits 0-9, in the required length with leading zeros if necessary.
+    """
+    allow_types = (str, int)
+    allow_chars = '0123456789'
+
+    @classmethod
+    def validate_val(cls, val):
+        if isinstance(val, int):
+            min_len, max_len = cls.get_len_range()
+            l = min_len or 1
+            val = '%0*d' % (l, val)
+        return super().validate_val(val)
+
+
+class DecimalHexParam(DecimalParam):
+    """The input value is decimal digits. The decimal value is stored such that each hexadecimal digit represents one
+    decimal digit, useful for various PIN type parameters.
+
+    Optionally, the value is stored with padding, for example: rpad = 8 would store '123' as '123fffff'. This is also
+    common in PIN type parameters.
+    """
+    rpad = None
+    rpad_char = 'f'
+
+    @classmethod
+    def validate_val(cls, val):
+        val = super().validate_val(val)
+        val = ''.join('%02x' % ord(x) for x in val)
+        if cls.rpad is not None:
+            c = cls.rpad_char
+            val = rpad(val, cls.rpad, c)
+        # a DecimalHexParam subclass expects the apply_val() input to be a bytes instance ready for the pes
+        return h2b(val)
+
+class IntegerParam(ConfigurableParameter):
+    allow_types = (str, int)
+    allow_chars = '0123456789'
+
+    # two integers, if the resulting int should be range limited
+    min_val = None
+    max_val = None
+
+    @classmethod
+    def validate_val(cls, val):
+        val = super().validate_val(val)
+        val = int(val)
+        exceeds_limits = False
+        if cls.min_val is not None:
+            if val < cls.min_val:
+                exceeds_limits = True
+        if cls.max_val is not None:
+            if val > cls.max_val:
+                exceeds_limits = True
+        if exceeds_limits:
+            raise ValueError(f'Value {val} is out of range, must be [{cls.min_val}..{cls.max_val}]')
+        return val
+
+class BinaryParam(ConfigurableParameter):
+    allow_types = (str, io.BytesIO, bytes, bytearray)
+    allow_chars = '0123456789abcdefABCDEF'
+    strip_chars = ' \t\r\n'
+
+    @classmethod
+    def validate_val(cls, val):
+        # take care that min_len and max_len are applied to the binary length by converting to bytes first
+        if isinstance(val, str):
+            if cls.strip_chars is not None:
+                val = ''.join(c for c in val if c not in cls.strip_chars)
+            if len(val) & 1:
+                raise ValueError('Invalid hexadecimal string, must have even number of digits:'
+                                 f' {val!r} {len(val)=}')
+            try:
+                val = h2b(val)
+            except ValueError as e:
+                raise ValueError(f'Invalid hexadecimal string: {val!r} {len(val)=}') from e
+
+        val = super().validate_val(val)
+        return bytes(val)
+
+
+class Iccid(DecimalParam):
+    """ICCID Parameter. Input: string of decimal digits.
+    If the string of digits is only 18 digits long, add a Luhn check digit."""
+    name = 'ICCID'
+    min_len = 18
+    max_len = 20
+    example_input = '998877665544332211'
+
+    @classmethod
+    def validate_val(cls, val):
+        iccid_str = super().validate_val(val)
+        return sanitize_iccid(iccid_str)
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        # patch the header
+        pes.get_pe_for_type('header').decoded['iccid'] = h2b(rpad(val, 20))
+        # patch MF/EF.ICCID
+        file_replace_content(pes.get_pe_for_type('mf').decoded['ef-iccid'], h2b(enc_iccid(val)))
+
+class Imsi(DecimalParam):
+    """Configurable IMSI. Expects value to be a string of digits. Automatically sets the ACC to
+    the last digit of the IMSI."""
+
+    name = 'IMSI'
+    min_len = 6
+    max_len = 15
+    example_input = '00101' + ('0' * 10)
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        imsi_str = val
+        # we always use the least significant byte of the IMSI as ACC
+        acc = (1 << int(imsi_str[-1]))
+        # patch ADF.USIM/EF.IMSI
+        for pe in pes.get_pes_for_type('usim'):
+            file_replace_content(pe.decoded['ef-imsi'], h2b(enc_imsi(imsi_str)))
+            file_replace_content(pe.decoded['ef-acc'], acc.to_bytes(2, 'big'))
+        # TODO: DF.GSM_ACCESS if not linked?
+
+class SmspTpScAddr(ConfigurableParameter):
+    """Configurable SMSC (SMS Service Centre) TP-SC-ADDR. Expects to be a phone number in national or
+    international format (designated by a leading +). Automatically sets the NPI to E.164 and the TON based on
+    presence or absence of leading +."""
+
+    name = 'SMSP-TP-SC-ADDR'
+    allow_chars = '+0123456789'
+    strip_chars = ' \t\r\n'
+    max_len = 21 # '+' and 20 digits
+    min_len = 1
+
+    @classmethod
+    def validate_val(cls, val):
+        val = super().validate_val(val)
+        addr_str = str(val)
+        if addr_str[0] == '+':
+            digits = addr_str[1:]
+            international = True
+        else:
+            digits = addr_str
+            international = False
+        if len(digits) > 20:
+            raise ValueError(f'TP-SC-ADDR must not exceed 20 digits: {digits!r}')
+        if not digits.isdecimal():
+            raise ValueError(f'TP-SC-ADDR must only contain decimal digits: {digits!r}')
+        return (international, digits)
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        """val must be a tuple (international[bool], digits[str]).
+        For example, an input of "+1234" corresponds to (True, "1234");
+        An input of "1234" corresponds to (False, "1234")."""
+        international, digits = val
+        for pe in pes.get_pes_for_type('usim'):
+            # obtain the File instance from the ProfileElementUSIM
+            f_smsp = pe.files['ef-smsp']
+            #print("SMSP (orig): %s" % f_smsp.body)
+            # instantiate the pySim.ts_51_011.EF_SMSP class for decode/encode
+            ef_smsp = EF_SMSP()
+            # decode the existing file body
+            ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
+            # patch the actual number
+            ef_smsp_dec['tp_sc_addr']['call_number'] = digits
+            # patch the NPI to isdn_e164
+            ef_smsp_dec['tp_sc_addr']['ton_npi']['numbering_plan_id'] = 'isdn_e164'
+            # patch the TON to international or unknown depending on +
+            ef_smsp_dec['tp_sc_addr']['ton_npi']['type_of_number'] = 'international' if international else 'unknown'
+            # ensure the parameter_indicators.tp_sc_addr is True
+            ef_smsp_dec['parameter_indicators']['tp_sc_addr'] = True
+            # re-encode into the File body
+            f_smsp.body = ef_smsp.encode_record_bin(ef_smsp_dec, 1)
+            #print("SMSP  (new): %s" % f_smsp.body)
+            # re-generate the pe.decoded member from the File instance
+            pe.file2pe(f_smsp)
+
+class SdKey(BinaryParam, metaclass=ClassVarMeta):
+    """Configurable Security Domain (SD) Key.  Value is presented as bytes."""
+    # these will be set by subclasses
+    key_type = None
+    key_id = None
+    kvn = None
+    key_usage_qual = None
+
+    @classmethod
+    def _apply_sd(cls, pe: ProfileElement, value):
+        assert pe.type == 'securityDomain'
+        for key in pe.decoded['keyList']:
+            if key['keyIdentifier'][0] == cls.key_id and key['keyVersionNumber'][0] == cls.kvn:
+                assert len(key['keyComponents']) == 1
+                key['keyComponents'][0]['keyData'] = value
+                return
+        # Could not find matching key to patch, create a new one
+        key = {
+            'keyUsageQualifier': bytes([cls.key_usage_qual]),
+            'keyIdentifier': bytes([cls.key_id]),
+            'keyVersionNumber': bytes([cls.kvn]),
+            'keyComponents': [
+                { 'keyType': bytes([cls.key_type]), 'keyData': value },
+            ]
+        }
+        pe.decoded['keyList'].append(key)
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, value):
+        for pe in pes.get_pes_for_type('securityDomain'):
+            cls._apply_sd(pe, value)
+
+class SdKeyScp80_01(SdKey, kvn=0x01, key_type=0x88, permitted_len=[16,24,32]): # AES key type
+    pass
+class SdKeyScp80_01Kic(SdKeyScp80_01, key_id=0x01, key_usage_qual=0x18): # FIXME: ordering?
+    pass
+class SdKeyScp80_01Kid(SdKeyScp80_01, key_id=0x02, key_usage_qual=0x14):
+    pass
+class SdKeyScp80_01Kik(SdKeyScp80_01, key_id=0x03, key_usage_qual=0x48):
+    pass
+
+class SdKeyScp81_01(SdKey, kvn=0x81): # FIXME
+    pass
+class SdKeyScp81_01Psk(SdKeyScp81_01, key_id=0x01, key_type=0x85, key_usage_qual=0x3C):
+    pass
+class SdKeyScp81_01Dek(SdKeyScp81_01, key_id=0x02, key_type=0x88, key_usage_qual=0x48):
+    pass
+
+class SdKeyScp02_20(SdKey, kvn=0x20, key_type=0x88, permitted_len=[16,24,32]): # AES key type
+    pass
+class SdKeyScp02_20Enc(SdKeyScp02_20, key_id=0x01, key_usage_qual=0x18):
+    pass
+class SdKeyScp02_20Mac(SdKeyScp02_20, key_id=0x02, key_usage_qual=0x14):
+    pass
+class SdKeyScp02_20Dek(SdKeyScp02_20, key_id=0x03, key_usage_qual=0x48):
+    pass
+
+class SdKeyScp03_30(SdKey, kvn=0x30, key_type=0x88, permitted_len=[16,24,32]): # AES key type
+    pass
+class SdKeyScp03_30Enc(SdKeyScp03_30, key_id=0x01, key_usage_qual=0x18):
+    pass
+class SdKeyScp03_30Mac(SdKeyScp03_30, key_id=0x02, key_usage_qual=0x14):
+    pass
+class SdKeyScp03_30Dek(SdKeyScp03_30, key_id=0x03, key_usage_qual=0x48):
+    pass
+
+class SdKeyScp03_31(SdKey, kvn=0x31, key_type=0x88, permitted_len=[16,24,32]): # AES key type
+    pass
+class SdKeyScp03_31Enc(SdKeyScp03_31, key_id=0x01, key_usage_qual=0x18):
+    pass
+class SdKeyScp03_31Mac(SdKeyScp03_31, key_id=0x02, key_usage_qual=0x14):
+    pass
+class SdKeyScp03_31Dek(SdKeyScp03_31, key_id=0x03, key_usage_qual=0x48):
+    pass
+
+class SdKeyScp03_32(SdKey, kvn=0x32, key_type=0x88, permitted_len=[16,24,32]): # AES key type
+    pass
+class SdKeyScp03_32Enc(SdKeyScp03_32, key_id=0x01, key_usage_qual=0x18):
+    pass
+class SdKeyScp03_32Mac(SdKeyScp03_32, key_id=0x02, key_usage_qual=0x14):
+    pass
+class SdKeyScp03_32Dek(SdKeyScp03_32, key_id=0x03, key_usage_qual=0x48):
+    pass
+
+
+
+
+def obtain_all_pe_from_pelist(l: List[ProfileElement], wanted_type: str) -> ProfileElement:
+    return (pe for pe in l if pe.type == wanted_type)
+
+def obtain_singleton_pe_from_pelist(l: List[ProfileElement], wanted_type: str) -> ProfileElement:
+    filtered = list(filter(lambda x: x.type == wanted_type, l))
+    assert len(filtered) == 1
+    return filtered[0]
+
+def obtain_first_pe_from_pelist(l: List[ProfileElement], wanted_type: str) -> ProfileElement:
+    filtered = list(filter(lambda x: x.type == wanted_type, l))
+    return filtered[0]
+
+class Puk(DecimalHexParam):
+    """Configurable PUK (Pin Unblock Code). String ASCII-encoded digits."""
+    allow_len = 8
+    rpad = 16
+    keyReference = None
+    example_input = '0' * allow_len
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        val_bytes = val
+        mf_pes = pes.pes_by_naa['mf'][0]
+        pukCodes = obtain_singleton_pe_from_pelist(mf_pes, 'pukCodes')
+        for pukCode in pukCodes.decoded['pukCodes']:
+            if pukCode['keyReference'] == cls.keyReference:
+                pukCode['pukValue'] = val_bytes
+                return
+        raise ValueError("input template UPP has unexpected structure:"
+                         f" cannot find pukCode with keyReference={cls.keyReference}")
+
+class Puk1(Puk):
+    name = 'PUK1'
+    keyReference = 0x01
+
+class Puk2(Puk):
+    name = 'PUK2'
+    keyReference = 0x81
+
+class Pin(DecimalHexParam):
+    """Configurable PIN (Personal Identification Number).  String of digits."""
+    rpad = 16
+    min_len = 4
+    max_len = 8
+    example_input = '0' * max_len
+    keyReference = None
+
+    @staticmethod
+    def _apply_pinvalue(pe: ProfileElement, keyReference, val_bytes):
+        for pinCodes in obtain_all_pe_from_pelist(pe, 'pinCodes'):
+            if pinCodes.decoded['pinCodes'][0] != 'pinconfig':
+                continue
+
+            for pinCode in pinCodes.decoded['pinCodes'][1]:
+                if pinCode['keyReference'] == keyReference:
+                     pinCode['pinValue'] = val_bytes
+                     return True
+        return False
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        val_bytes = val
+        if not cls._apply_pinvalue(pes.pes_by_naa['mf'][0], cls.keyReference, val_bytes):
+            raise ValueError('input template UPP has unexpected structure:'
+                             + f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference}')
+
+class Pin1(Pin):
+    name = 'PIN1'
+    example_input = '0' * 4  # PIN are usually 4 digits
+    keyReference = 0x01
+
+class Pin2(Pin1):
+    name = 'PIN2'
+    keyReference = 0x81
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        val_bytes = val
+        # PIN2 is special: telecom + usim + isim + csim
+        for naa in pes.pes_by_naa:
+            if naa not in ['usim','isim','csim','telecom']:
+                continue
+            for instance in pes.pes_by_naa[naa]:
+                if not cls._apply_pinvalue(instance, cls.keyReference, val_bytes):
+                    raise ValueError('input template UPP has unexpected structure:'
+                            + f' {cls.get_name()} cannot find pinCode with keyReference={cls.keyReference} in {naa=}')
+
+class Adm1(Pin):
+    name = 'ADM1'
+    keyReference = 0x0A
+
+class Adm2(Adm1):
+    name = 'ADM2'
+    keyReference = 0x0B
+
+class AlgoConfig(ConfigurableParameter):
+    algo_config_key = None
+
+    @classmethod
+    def apply_val(cls, pes: ProfileElementSequence, val):
+        found = 0
+        for pe in pes.get_pes_for_type('akaParameter'):
+            algoConfiguration = pe.decoded['algoConfiguration']
+            if algoConfiguration[0] != 'algoParameter':
+                continue
+            algoConfiguration[1][cls.algo_config_key] = val
+            found += 1
+        if not found:
+            raise ValueError('input template UPP has unexpected structure:'
+                             f' {cls.__name__} cannot find algoParameter with key={cls.algo_config_key}')
+
+class AlgorithmID(DecimalParam, AlgoConfig):
+    algo_config_key = 'algorithmID'
+    allow_len = 1
+    example_input = 1  # Milenage
+
+    @classmethod
+    def validate_val(cls, val):
+        val = super().validate_val(val)
+        val = int(val)
+        valid = (1, 2, 3)
+        if val not in valid:
+            raise ValueError(f'Invalid algorithmID {val!r}, must be one of {valid}')
+        return val
+
+class K(BinaryParam, AlgoConfig):
+    """use validate_val() from BinaryParam, and apply_val() from AlgoConfig"""
+    name = 'K'
+    algo_config_key = 'key'
+    allow_len = (128 // 8, 256 // 8) # length in bytes (from BinaryParam); TUAK also allows 256 bit
+    example_input = '00' * allow_len[0]
+
+class Opc(K):
+    name = 'OPc'
+    algo_config_key = 'opc'
+
+class MilenageRotationConstants(BinaryParam, AlgoConfig):
+    """rotation constants r1,r2,r3,r4,r5 of Milenage, Range 0..127. See 3GPP TS 35.206 Sections 2.3 + 5.3.
+    Provided as octet-string concatenation of all 5 constants.  Expects a bytes-like object of length 5, with
+    each byte in the range of 0..127.  The default value by 3GPP is '4000204060' (hex notation)"""
+    name = 'MilenageRotation'
+    algo_config_key = 'rotationConstants'
+    allow_len = 5 # length in bytes (from BinaryParam)
+    example_input = '0a 0b 0c 0d 0e'
+
+    @classmethod
+    def validate_val(cls, val):
+        "allow_len checks the length, this in addition checks the value range"
+        val = super().validate_val(val)
+        assert isinstance(val, bytes)
+        if any(r > 127 for r in val):
+            raise ValueError('r values must be in the range 0..127')
+        return val
+
+class MilenageXoringConstants(BinaryParam, AlgoConfig):
+    """XOR-ing constants c1,c2,c3,c4,c5 of Milenage, 128bit each. See 3GPP TS 35.206 Sections 2.3 + 5.3.
+    Provided as octet-string concatenation of all 5 constants. The default value by 3GPP is the concetenation
+    of::
+
+     00000000000000000000000000000000
+     00000000000000000000000000000001
+     00000000000000000000000000000002
+     00000000000000000000000000000004
+     00000000000000000000000000000008
+
+    """
+    name = 'MilenageXOR'
+    algo_config_key = 'xoringConstants'
+    allow_len = 80 # length in bytes (from BinaryParam)
+    example_input = ('00000000000000000000000000000000'
+                     ' 00000000000000000000000000000001'
+                     ' 00000000000000000000000000000002'
+                     ' 00000000000000000000000000000004'
+                     ' 00000000000000000000000000000008')
+
+class TuakNumberOfKeccak(IntegerParam, AlgoConfig):
+    """Number of iterations of Keccak-f[1600] permutation as recomended by Section 7.2 of 3GPP TS 35.231"""
+    name = 'KECCAK-N'
+    algo_config_key = 'numberOfKeccak'
+    min_val = 1
+    max_val = 255
+    example_input = '1'

pySim/esim/saip/templates.py

@@ -0,0 +1,975 @@
+"""Implementation of SimAlliance/TCA Interoperable Profile Templates."""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import *
+from copy import deepcopy
+from pySim.utils import all_subclasses, h2b
+from pySim.filesystem import Path
+import pySim.esim.saip.oid as OID
+
+class FileTemplate:
+    """Representation of a single file in a SimAlliance/TCA Profile Template. The argument order
+    is done to match that of the tables in Section 9 of the SAIP specification."""
+    def __init__(self, fid:int, name:str, ftype, nb_rec: Optional[int], size:Optional[int], arr:int,
+                 sfi:Optional[int] = None, default_val:Optional[str] = None, content_rqd:bool = True,
+                 params:Optional[List] = None, ass_serv:Optional[List[int]]=None, high_update:bool = False,
+                 pe_name:Optional[str] = None, repeat:bool = False, ppath: List[int] = []):
+        """
+        Args:
+            fid: The 16bit file-identifier of the file
+            name: The name of the file in human-readable "EF.FOO", "DF.BAR" notation
+            ftype: The type of the file; can be 'MF', 'ADF', 'DF', 'TR', 'LF', 'CY', 'BT'
+            nb_rec: Then number of records (only valid for 'LF' and 'CY')
+            size: The size of the file ('TR', 'BT'); size of each record ('LF, 'CY')
+            arr: The record number of EF.ARR for referenced access rules
+            sfi: The short file identifier, if any
+            default_val: The default value [pattern] of the file
+            content_rqd: Whether an instance of template *must* specify file contents
+            params: A list of parameters that an instance of the template *must* specify
+            ass_serv: The associated service[s] of the service table
+            high_update: Is this file of "high update frequency" type?
+            pe_name: The name of this file in the ASN.1 type of the PE. Auto-generated for most.
+            repeat: Whether the default_val pattern is a repeating pattern.
+            ppath: The intermediate path between the base_df of the ProfileTemplate and this file.  If not
+                   specified, the file will be created immediately underneath the base_df.
+        """
+        # initialize from arguments
+        self.fid = fid
+        self.name = name
+        if pe_name:
+            self.pe_name = pe_name
+        else:
+            self.pe_name = self.name.replace('.','-').replace('_','-').lower()
+        self.file_type = ftype
+        if ftype in ['LF', 'CY']:
+            self.nb_rec = nb_rec
+            self.rec_len = size
+        elif ftype in ['TR', 'BT']:
+            self.file_size = size
+        self.arr = arr
+        self.sfi = sfi
+        self.default_val = default_val
+        self.default_val_repeat = repeat
+        self.content_rqd = content_rqd
+        self.params = params
+        self.ass_serv = ass_serv
+        self.high_update = high_update
+        self.ppath = ppath # parent path, if this FileTemplate is not immediately below the base_df
+        # initialize empty
+        self.parent = None
+        self.children = []
+        if self.default_val:
+            length = self._default_value_len() or 100
+            # run the method once to verify the pattern can be processed
+            self.expand_default_value_pattern(length)
+
+    def __str__(self) -> str:
+        return "FileTemplate(%s)" % (self.name)
+
+    def __repr__(self) -> str:
+        s_fid = "%04x" % self.fid if self.fid is not None else 'None'
+        s_arr = self.arr if self.arr is not None else 'None'
+        s_sfi = "%02x" % self.sfi if self.sfi is not None else 'None'
+        return "FileTemplate(%s/%s, %s, %s, arr=%s, sfi=%s, ppath=%s)" % (self.name, self.pe_name, s_fid, self.file_type, s_arr, s_sfi, self.ppath)
+
+    def print_tree(self, indent:str = ""):
+        """recursive printing of FileTemplate tree structure."""
+        print("%s%s (%s)" % (indent, repr(self), self.path))
+        indent += " "
+        for c in self.children:
+            c.print_tree(indent)
+
+    @property
+    def path(self):
+        """Return the path of the given File within the hierarchy."""
+        if self.parent:
+            return self.parent.path + self.name
+        else:
+            return Path(self.name)
+
+    def get_file_by_path(self, path: List[str]) -> Optional['FileTemplate']:
+        """Return a FileTemplate matching the given path within this ProfileTemplate."""
+        if path[0].lower() != self.name.lower():
+            return None
+        for c in self.children:
+            if path[1].lower() == c.name.lower():
+                return c.get_file_by_path(path[1:])
+
+    def _default_value_len(self):
+        if self.file_type in ['TR']:
+            return self.file_size
+        elif self.file_type in ['LF', 'CY']:
+            return self.rec_len
+
+    def expand_default_value_pattern(self, length: Optional[int] = None) -> Optional[bytes]:
+        """Expand the default value pattern to the specified length."""
+        if length is None:
+            length = self._default_value_len()
+        if length is None:
+            raise ValueError("%s does not have a default length" % self)
+        if not self.default_val:
+            return None
+        if not '...' in self.default_val:
+            return h2b(self.default_val)
+        l = self.default_val.split('...')
+        if len(l) != 2:
+            raise ValueError("Pattern '%s' contains more than one ..." % self.default_val)
+        prefix = h2b(l[0])
+        suffix = h2b(l[1])
+        pad_len = length - len(prefix) - len(suffix)
+        if pad_len <= 0:
+            ret = prefix + suffix
+            return ret[:length]
+        return prefix + prefix[-1:] * pad_len + suffix
+
+
+class ProfileTemplate:
+    """Representation of a SimAlliance/TCA Profile Template.  Each Template is identified by its OID and
+    consists of a number of file definitions.  We implement each profile template as a class derived from this
+    base class.  Each such derived class is a singleton and has no instances."""
+    created_by_default: bool = False
+    optional: bool = False
+    oid: Optional[OID.eOID] = None
+    files: List[FileTemplate] = []
+
+    # indicates that a given template does not have its own 'base DF', but that its contents merely
+    # extends that of the 'base DF' of another template
+    extends: Optional['ProfileTemplate'] = None
+
+    # indicates a parent ProfileTemplate below whose 'base DF' our files should be placed.
+    parent: Optional['ProfileTemplate'] = None
+
+    def __init_subclass__(cls, **kwargs):
+        """This classmethod is called automatically after executing the subclass body. We use it to
+        initialize the cls.files_by_pename from the cls.files"""
+        super().__init_subclass__(**kwargs)
+        cur_df = None
+
+        cls.files_by_pename: dict[str,FileTemplate] = {}
+        cls.tree: List[FileTemplate] = []
+
+        if not cls.optional and not cls.files[0].file_type in ['MF', 'DF', 'ADF']:
+            raise ValueError('First file in non-optional template must be MF, DF or ADF (is: %s)' % cls.files[0])
+        for f in cls.files:
+            if f.file_type in ['MF', 'DF', 'ADF']:
+                if cur_df == None:
+                    cls.tree.append(f)
+                    f.parent = None
+                    cur_df = f
+                else:
+                    # "cd .."
+                    if cur_df.parent:
+                        cur_df = cur_df.parent
+                    f.parent = cur_df
+                    cur_df.children.append(f)
+                    cur_df = f
+            else:
+                if cur_df == None:
+                    cls.tree.append(f)
+                    f.parent = None
+                else:
+                    cur_df.children.append(f)
+                    f.parent = cur_df
+            cls.files_by_pename[f.pe_name] = f
+        ProfileTemplateRegistry.add(cls)
+
+    @classmethod
+    def print_tree(cls):
+        for c in cls.tree:
+            c.print_tree()
+
+    @classmethod
+    def base_df(cls) -> FileTemplate:
+        """Return the FileTemplate for the base DF of the given template.  This may be a DF or ADF
+        within this template, or refer to another template (e.g. mandatory USIM if we are optional USIM."""
+        if cls.extends:
+            return cls.extends.base_df
+        return cls.files[0]
+
+class ProfileTemplateRegistry:
+    """A registry of profile templates.  Exists as a singleton class with no instances and only
+    classmethods."""
+    by_oid = {}
+
+    @classmethod
+    def add(cls, tpl: ProfileTemplate):
+        """Add a ProfileTemplate to the registry.  There can only be one Template per OID."""
+        oid_str = str(tpl.oid)
+        if oid_str in cls.by_oid:
+            raise ValueError("We already have a template for OID %s" % oid_str)
+        cls.by_oid[oid_str] = tpl
+
+    @classmethod
+    def get_by_oid(cls, oid: Union[List[int], str]) -> Optional[ProfileTemplate]:
+        """Look-up the ProfileTemplate based on its OID.  The OID can be given either in dotted-string format,
+        or as a list of integers."""
+        if not isinstance(oid, str):
+            oid = OID.OID.str_from_intlist(oid)
+        return cls.by_oid.get(oid, None)
+
+# below are transcribed template definitions from "ANNEX A (Normative): File Structure Templates Definition"
+# of "Profile interoperability specification V3.3.1 Final" (unless other version explicitly specified).
+
+class FilesAtMF(ProfileTemplate):
+    """Files at MF as per Section 9.2"""
+    created_by_default = True
+    oid = OID.MF
+    files = [
+        FileTemplate(0x3f00, 'MF',           'MF', None, None,  14, None, None, None, params=['pinStatusTemplateDO']),
+        FileTemplate(0x2f05, 'EF.PL',        'TR', None,    2,   1, 0x05, 'FF...FF', None),
+        FileTemplate(0x2f02, 'EF.ICCID',     'TR', None,   10,  11, None, None, True),
+        FileTemplate(0x2f00, 'EF.DIR',       'LF', None, None,  10, 0x1e, None, True, params=['nb_rec', 'size']),
+        FileTemplate(0x2f06, 'EF.ARR',       'LF', None, None,  10, None, None, True, params=['nb_rec', 'size']),
+        FileTemplate(0x2f08, 'EF.UMPC',      'TR', None,    5,  10, 0x08, None, False),
+    ]
+
+
+class FilesCD(ProfileTemplate):
+    """Files at DF.CD as per Section 9.3"""
+    created_by_default = False
+    oid = OID.DF_CD
+    files = [
+        FileTemplate(0x7f11, 'DF.CD',        'DF', None, None,  14, None, None, False, params=['pinStatusTemplateDO']),
+        FileTemplate(0x6f01, 'EF.LAUNCHPAD', 'TR', None, None,   2, None, None, True, params=['size']),
+    ]
+    for i in range(0x40, 0x7f):
+        files.append(FileTemplate(0x6f00+i, 'EF.ICON',      'TR', None, None,   2, None, None, True, params=['size']))
+
+
+# Section 9.4: Do this separately, so we can use them also from 9.5.3
+df_pb_files = [
+    FileTemplate(0x5f3a, 'DF.PHONEBOOK', 'DF', None, None,  14, None, None, True, ['pinStatusTemplateDO']),
+    FileTemplate(0x4f30, 'EF.PBR',       'LF', None, None,   2, None, None, True, ['nb_rec', 'size'], ppath=[0x5f3a]),
+]
+for i in range(0x38, 0x40):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.EXT1', 'LF', None,   13,  5, None, '00FF...FF', False, ['size','sfi'], ppath=[0x5f3a]))
+for i in range(0x40, 0x48):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.AAS', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size'], ppath=[0x5f3a]))
+for i in range(0x48, 0x50):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.GAS', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size'], ppath=[0x5f3a]))
+df_pb_files += [
+    FileTemplate(0x4f22, 'EF.PSC',       'TR', None,    4,   5, None, '00000000', False, ['sfi'], ppath=[0x5f3a]),
+    FileTemplate(0x4f23, 'EF.CC',        'TR', None,    2,   5, None, '0000', False, ['sfi'], high_update=True, ppath=[0x5f3a]),
+    FileTemplate(0x4f24, 'EF.PUID',      'TR', None,    2,   5, None, '0000', False, ['sfi'], high_update=True, ppath=[0x5f3a]),
+]
+for i in range(0x50, 0x58):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.IAP', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x58, 0x60):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.ADN', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x60, 0x68):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.ADN', 'LF', None,    2,  5, None, '00...00', False, ['nb_rec','sfi'], ppath=[0x5f3a]))
+for i in range(0x68, 0x70):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.ANR', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x70, 0x78):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.PURI', 'LF', None, None,  5, None, None, True, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x78, 0x80):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.EMAIL', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x80, 0x88):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.SNE', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x88, 0x90):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.UID', 'LF', None,    2,  5, None, '0000', False, ['nb_rec','sfi'], ppath=[0x5f3a]))
+for i in range(0x90, 0x98):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.GRP', 'LF', None, None,  5, None, '00...00', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+for i in range(0x98, 0xa0):
+    df_pb_files.append(FileTemplate(0x4f00+i, 'EF.CCP1', 'LF', None, None,  5, None, 'FF...FF', False, ['nb_rec','size','sfi'], ppath=[0x5f3a]))
+
+class FilesTelecom(ProfileTemplate):
+    """Files at DF.TELECOM as per Section 9.4 v2.3.1"""
+    created_by_default = False
+    oid = OID.DF_TELECOM
+    base_path = Path('MF')
+    files = [
+        FileTemplate(0x7f10, 'DF.TELECOM',   'DF', None, None,  14, None, None, False, params=['pinStatusTemplateDO']),
+        FileTemplate(0x6f06, 'EF.ARR',       'LF', None, None,  10, None, None, True, ['nb_rec', 'size']),
+        FileTemplate(0x6f53, 'EF.RMA',       'LF', None, None,   3, None, None, True, ['nb_rec', 'size']),
+        FileTemplate(0x6f54, 'EF.SUME',      'TR', None,   22,   3, None, None, True),
+        FileTemplate(0x6fe0, 'EF.ICE_DN',    'LF',   50,   24,   9, None, 'FF...FF', False),
+        FileTemplate(0x6fe1, 'EF.ICE_FF',    'LF', None, None,   9, None, 'FF...FF', False, ['nb_rec', 'size']),
+        FileTemplate(0x6fe5, 'EF.PSISMSC',   'LF', None, None,   5, None, None, True, ['nb_rec', 'size'], ass_serv=[12,91]),
+        FileTemplate(0x5f50, 'DF.GRAPHICS',  'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO']),
+        FileTemplate(0x4f20, 'EF.IMG',       'LF', None, None,   2, None, '00FF...FF', False, ['nb_rec', 'size'], ppath=[0x5f50]),
+        # EF.IIDF below
+        FileTemplate(0x4f21, 'EF.ICE_GRAPHICS','BT',None,None,   9, None, None, False, ['size'], ppath=[0x5f50]),
+        FileTemplate(0x4f01, 'EF.LAUNCH_SCWS','TR',None, None,  10, None, None, True, ['size'], ppath=[0x5f50]),
+        # EF.ICON below
+    ]
+    for i in range(0x40, 0x80):
+        files.append(FileTemplate(0x4f00+i, 'EF.IIDF', 'TR', None, None, 2, None, 'FF...FF', False, ['size'], ppath=[0x5f50]))
+    for i in range(0x80, 0xC0):
+        files.append(FileTemplate(0x4f00+i, 'EF.ICON', 'TR', None, None, 10, None, None, True, ['size'], ppath=[0x5f50]))
+
+    # we copy the objects (instances) here as we also use them below from FilesUsimDfPhonebook
+    df_pb = deepcopy(df_pb_files)
+    files += df_pb
+
+    files += [
+        FileTemplate(0x5f3b, 'DF.MULTIMEDIA','DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[67]),
+        FileTemplate(0x4f47, 'EF.MML',       'BT', None, None,   5, None, None, False, ['size'], ass_serv=[67], ppath=[0x5f3b]),
+        FileTemplate(0x4f48, 'EF.MMDF',      'BT', None, None,   5, None, None, False, ['size'], ass_serv=[67], ppath=[0x5f3b]),
+
+        FileTemplate(0x5f3c, 'DF.MMSS',      'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO']),
+        FileTemplate(0x4f20, 'EF.MLPL',      'TR', None, None,   2, 0x01, None, True, ['size'], ppath=[0x5f3c]),
+        FileTemplate(0x4f21, 'EF.MSPL',      'TR', None, None,   2, 0x02, None, True, ['size'], ppath=[0x5f3c]),
+        FileTemplate(0x4f21, 'EF.MMSSMODE',  'TR', None,    1,   2, 0x03, None, True, ppath=[0x5f3c]),
+    ]
+
+
+class FilesTelecomV2(ProfileTemplate):
+    """Files at DF.TELECOM as per Section 9.4"""
+    created_by_default = False
+    oid = OID.DF_TELECOM_v2
+    base_path = Path('MF')
+    files = [
+        FileTemplate(0x7f10, 'DF.TELECOM',   'DF', None, None,  14, None, None, False, params=['pinStatusTemplateDO']),
+        FileTemplate(0x6f06, 'EF.ARR',       'LF', None, None,  10, None, None, True, ['nb_rec', 'size']),
+        FileTemplate(0x6f53, 'EF.RMA',       'LF', None, None,   3, None, None, True, ['nb_rec', 'size']),
+        FileTemplate(0x6f54, 'EF.SUME',      'TR', None,   22,   3, None, None, True),
+        FileTemplate(0x6fe0, 'EF.ICE_DN',    'LF',   50,   24,   9, None, 'FF...FF', False),
+        FileTemplate(0x6fe1, 'EF.ICE_FF',    'LF', None, None,   9, None, 'FF...FF', False, ['nb_rec', 'size']),
+        FileTemplate(0x6fe5, 'EF.PSISMSC',   'LF', None, None,   5, None, None, True, ['nb_rec', 'size'], ass_serv=[12,91]),
+        FileTemplate(0x5f50, 'DF.GRAPHICS',  'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO']),
+        FileTemplate(0x4f20, 'EF.IMG',       'LF', None, None,   2, None, '00FF...FF', False, ['nb_rec', 'size'], ppath=[0x5f50]),
+        # EF.IIDF below
+        FileTemplate(0x4f21, 'EF.ICE_GRRAPHICS','BT',None,None,   9, None, None, False, ['size'], ppath=[0x5f50]),
+        FileTemplate(0x4f01, 'EF.LAUNCH_SCWS','TR',None, None,  10, None, None, True, ['size'], ppath=[0x5f50]),
+        # EF.ICON below
+    ]
+    for i in range(0x40, 0x80):
+        files.append(FileTemplate(0x4f00+i, 'EF.IIDF', 'TR', None, None, 2, None, 'FF...FF', False, ['size'], ppath=[0x5f50]))
+    for i in range(0x80, 0xC0):
+        files.append(FileTemplate(0x4f00+i, 'EF.ICON', 'TR', None, None, 10, None, None, True, ['size'],ppath=[0x5f50]))
+
+    # we copy the objects (instances) here as we also use them below from FilesUsimDfPhonebook
+    df_pb = deepcopy(df_pb_files)
+    files += df_pb
+
+    files += [
+        FileTemplate(0x5f3b, 'DF.MULTIMEDIA','DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[67]),
+        FileTemplate(0x4f47, 'EF.MML',       'BT', None, None,   5, None, None, False, ['size'], ass_serv=[67], ppath=[0x5f3b]),
+        FileTemplate(0x4f48, 'EF.MMDF',      'BT', None, None,   5, None, None, False, ['size'], ass_serv=[67], ppath=[0x5f3b]),
+
+        FileTemplate(0x5f3c, 'DF.MMSS',      'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO']),
+        FileTemplate(0x4f20, 'EF.MLPL',      'TR', None, None,   2, 0x01, None, True, ['size'], ppath=[0x5f3c]),
+        FileTemplate(0x4f21, 'EF.MSPL',      'TR', None, None,   2, 0x02, None, True, ['size'], ppath=[0x5f3c]),
+        FileTemplate(0x4f21, 'EF.MMSSMODE',  'TR', None,    1,   2, 0x03, None, True, ppath=[0x5f3c]),
+
+        FileTemplate(0x5f3d, 'DF.MCS',       'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv={'usim':109, 'isim': 15}),
+        FileTemplate(0x4f01, 'EF.MST',       'TR', None, None,   2, 0x01, None, True, ['size'], ass_serv={'usim':109, 'isim': 15}, ppath=[0x5f3d]),
+        FileTemplate(0x4f02, 'EF.MCSCONFIG', 'BT', None, None,   2, 0x02, None, True, ['size'], ass_serv={'usim':109, 'isim': 15}, ppath=[0x5f3d]),
+
+        FileTemplate(0x5f3e, 'DF.V2X',       'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[119]),
+        FileTemplate(0x4f01, 'EF.VST',       'TR', None, None,   2, 0x01, None, True, ['size'], ass_serv=[119], ppath=[0x5f3e]),
+        FileTemplate(0x4f02, 'EF.V2X_CONFIG','BT', None, None,   2, 0x02, None, True, ['size'], ass_serv=[119], ppath=[0x5f3e]),
+        FileTemplate(0x4f03, 'EF.V2XP_PC5',  'TR', None, None,   2, None, None, True, ['size'], ass_serv=[119], ppath=[0x5f3e]), # VST: 2
+        FileTemplate(0x4f04, 'EF.V2XP_Uu',   'TR', None, None,   2, None, None, True, ['size'], ass_serv=[119], ppath=[0x5f3e]), # VST: 3
+    ]
+
+
+class FilesUsimMandatory(ProfileTemplate):
+    """Mandatory Files at ADF.USIM as per Section 9.5.1 v2.3.1"""
+    created_by_default = True
+    oid = OID.ADF_USIM_by_default
+    files = [
+        FileTemplate(  None, 'ADF.USIM',    'ADF', None, None,  14, None, None, False, ['aid', 'temp_fid', 'pinStatusTemplateDO']),
+        FileTemplate(0x6f07, 'EF.IMSI',      'TR', None,    9,   2, 0x07, None, True, ['size']),
+        FileTemplate(0x6f06, 'EF.ARR',       'LF', None, None,  10, 0x17, None, True, ['nb_rec','size']),
+        FileTemplate(0x6f08, 'EF.Keys',      'TR', None,   33,   5, 0x08, '07FF...FF', False, high_update=True),
+        FileTemplate(0x6f09, 'EF.KeysPS',    'TR', None,   33,   5, 0x09, '07FF...FF', False, high_update=True, pe_name = 'ef-keysPS'),
+        FileTemplate(0x6f31, 'EF.HPPLMN',    'TR', None,    1,   2, 0x12, '0A', False),
+        FileTemplate(0x6f38, 'EF.UST',       'TR', None,   14,   2, 0x04, None, True),
+        FileTemplate(0x6f3b, 'EF.FDN',       'LF',   20,   26,   8, None, 'FF...FF', False, ass_serv=[2, 89]),
+        FileTemplate(0x6f3c, 'EF.SMS',       'LF',   10,  176,   5, None, '00FF...FF', False, ass_serv=[10]),
+        FileTemplate(0x6f42, 'EF.SMSP',      'LF',    1,   38,   5, None, 'FF...FF', False, ass_serv=[12]),
+        FileTemplate(0x6f43, 'EF.SMSS',      'TR', None,    2,   5, None, 'FFFF', False, ass_serv=[10]),
+        FileTemplate(0x6f46, 'EF.SPN',       'TR', None,   17,  10, None, None, True, ass_serv=[19]),
+        FileTemplate(0x6f56, 'EF.EST',       'TR', None,    1,   8, 0x05, None, True, ass_serv=[2,6,34,35]),
+        FileTemplate(0x6f5b, 'EF.START-HFN', 'TR', None,    6,   5, 0x0f, 'F00000F00000', False, high_update=True),
+        FileTemplate(0x6f5c, 'EF.THRESHOLD', 'TR', None,    3,   2, 0x10, 'FFFFFF', False),
+        FileTemplate(0x6f73, 'EF.PSLOCI',    'TR', None,   14,   5, 0x0c, 'FFFFFFFFFFFFFFFFFFFF0000FF01', False, high_update=True),
+        FileTemplate(0x6f78, 'EF.ACC',       'TR', None,    2,   2, 0x06, None, True),
+        FileTemplate(0x6f7b, 'EF.FPLMN',     'TR', None,   12,   5, 0x0d, 'FF...FF', False),
+        FileTemplate(0x6f7e, 'EF.LOCI',      'TR', None,   11,   5, 0x0b, 'FFFFFFFFFFFFFF0000FF01', False, high_update=True),
+        FileTemplate(0x6fad, 'EF.AD',        'TR', None,    4,  10, 0x03, '00000002', False),
+        FileTemplate(0x6fb7, 'EF.ECC',       'LF',    1,    4,  10, 0x01, None, True),
+        FileTemplate(0x6fc4, 'EF.NETPAR',    'TR', None,  128,   5, None, 'FF...FF', False, high_update=True),
+        FileTemplate(0x6fe3, 'EF.EPSLOCI',   'TR', None,   18,   5, 0x1e, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF000001', False, ass_serv=[85], high_update=True),
+        FileTemplate(0x6fe4, 'EF.EPSNSC',    'LF',    1,   80,   5, 0x18, 'FF...FF', False, ass_serv=[85], high_update=True),
+    ]
+
+class FilesUsimMandatoryV2(ProfileTemplate):
+    """Mandatory Files at ADF.USIM as per Section 9.5.1"""
+    created_by_default = True
+    oid = OID.ADF_USIM_by_default_v2
+    files = [
+        FileTemplate(  None, 'ADF.USIM',    'ADF', None, None,  14, None, None, False, ['aid', 'temp_fid', 'pinStatusTemplateDO']),
+        FileTemplate(0x6f07, 'EF.IMSI',      'TR', None,    9,   2, 0x07, None, True, ['size']),
+        FileTemplate(0x6f06, 'EF.ARR',       'LF', None, None,  10, 0x17, None, True, ['nb_rec','size']),
+        FileTemplate(0x6f08, 'EF.Keys',      'TR', None,   33,   5, 0x08, '07FF...FF', False, high_update=True),
+        FileTemplate(0x6f09, 'EF.KeysPS',    'TR', None,   33,   5, 0x09, '07FF...FF', False, high_update=True, pe_name='ef-keysPS'),
+        FileTemplate(0x6f31, 'EF.HPPLMN',    'TR', None,    1,   2, 0x12, '0A', False),
+        FileTemplate(0x6f38, 'EF.UST',       'TR', None,   17,   2, 0x04, None, True),
+        FileTemplate(0x6f3b, 'EF.FDN',       'LF',   20,   26,   8, None, 'FF...FF', False, ass_serv=[2, 89]),
+        FileTemplate(0x6f3c, 'EF.SMS',       'LF',   10,  176,   5, None, '00FF...FF', False, ass_serv=[10]),
+        FileTemplate(0x6f42, 'EF.SMSP',      'LF',    1,   38,   5, None, 'FF...FF', False, ass_serv=[12]),
+        FileTemplate(0x6f43, 'EF.SMSS',      'TR', None,    2,   5, None, 'FFFF', False, ass_serv=[10]),
+        FileTemplate(0x6f46, 'EF.SPN',       'TR', None,   17,  10, None, None, True, ass_serv=[19]),
+        FileTemplate(0x6f56, 'EF.EST',       'TR', None,    1,   8, 0x05, None, True, ass_serv=[2,6,34,35]),
+        FileTemplate(0x6f5b, 'EF.START-HFN', 'TR', None,    6,   5, 0x0f, 'F00000F00000', False, high_update=True),
+        FileTemplate(0x6f5c, 'EF.THRESHOLD', 'TR', None,    3,   2, 0x10, 'FFFFFF', False),
+        FileTemplate(0x6f73, 'EF.PSLOCI',    'TR', None,   14,   5, 0x0c, 'FFFFFFFFFFFFFFFFFFFF0000FF01', False, high_update=True),
+        FileTemplate(0x6f78, 'EF.ACC',       'TR', None,    2,   2, 0x06, None, True),
+        FileTemplate(0x6f7b, 'EF.FPLMN',     'TR', None,   12,   5, 0x0d, 'FF...FF', False),
+        FileTemplate(0x6f7e, 'EF.LOCI',      'TR', None,   11,   5, 0x0b, 'FFFFFFFFFFFFFF0000FF01', False, high_update=True),
+        FileTemplate(0x6fad, 'EF.AD',        'TR', None,    4,  10, 0x03, '00000002', False),
+        FileTemplate(0x6fb7, 'EF.ECC',       'LF',    1,    4,  10, 0x01, None, True),
+        FileTemplate(0x6fc4, 'EF.NETPAR',    'TR', None,  128,   5, None, 'FF...FF', False, high_update=True),
+        FileTemplate(0x6fe3, 'EF.EPSLOCI',   'TR', None,   18,   5, 0x1e, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF000001', False, ass_serv=[85], high_update=True),
+        FileTemplate(0x6fe4, 'EF.EPSNSC',    'LF',    1,   80,   5, 0x18, 'FF...FF', False, ass_serv=[85], high_update=True),
+    ]
+
+
+class FilesUsimOptional(ProfileTemplate):
+    """Optional Files at ADF.USIM as per Section 9.5.2 v2.3.1"""
+    created_by_default = False
+    optional = True
+    oid = OID.ADF_USIMopt_not_by_default
+    base_path = Path('ADF.USIM')
+    extends = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6f05, 'EF.LI',        'TR', None,    6,   1, 0x02, 'FF...FF', False),
+        FileTemplate(0x6f37, 'EF.ACMmax',    'TR', None,    3,   5, None, '000000', False, ass_serv=[13], pe_name='ef-acmax'),
+        FileTemplate(0x6f39, 'EF.ACM',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[13], high_update=True),
+        FileTemplate(0x6f3e, 'EF.GID1',      'TR', None,    8,   2, None, None, True, ass_serv=[17]),
+        FileTemplate(0x6f3f, 'EF.GID2',      'TR', None,    8,   2, None, None, True, ass_serv=[18]),
+        FileTemplate(0x6f40, 'EF.MSISDN',    'LF',    1,   24,   2, None, 'FF...FF', False, ass_serv=[21]),
+        FileTemplate(0x6f41, 'EF.PUCT',      'TR', None,    5,   5, None, 'FFFFFF0000', False, ass_serv=[13]),
+        FileTemplate(0x6f45, 'EF.CBMI',      'TR', None,   10,   5, None, 'FF...FF', False, ass_serv=[15]),
+        FileTemplate(0x6f48, 'EF.CBMID',     'TR', None,   10,   2, 0x0e, 'FF...FF', False, ass_serv=[19]),
+        FileTemplate(0x6f49, 'EF.SDN',       'LF',   10,   24,   2, None, 'FF...FF', False, ass_serv=[4,89]),
+        FileTemplate(0x6f4b, 'EF.EXT2',      'LF',   10,   13,   8, None, '00FF...FF', False, ass_serv=[3]),
+        FileTemplate(0x6f4c, 'EF.EXT3',      'LF',   10,   13,   2, None, '00FF...FF', False, ass_serv=[5]),
+        FileTemplate(0x6f50, 'EF.CBMIR',     'TR', None,   20,   5, None, 'FF...FF', False, ass_serv=[16]),
+        FileTemplate(0x6f60, 'EF.PLMNwAcT',  'TR', None,   40,   5, 0x0a, 'FFFFFF0000', False, ass_serv=[20], repeat=True),
+        FileTemplate(0x6f61, 'EF.OPLMNwAcT', 'TR', None,   40,   2, 0x11, 'FFFFFF0000', False, ass_serv=[42], repeat=True),
+        FileTemplate(0x6f62, 'EF.HPLMNwAcT', 'TR', None,    5,   2, 0x13, 'FFFFFF0000', False, ass_serv=[43], repeat=True),
+        FileTemplate(0x6f2c, 'EF.DCK',       'TR', None,   16,   5, None, 'FF...FF', False, ass_serv=[36]),
+        FileTemplate(0x6f32, 'EF.CNL',       'TR', None,   30,   2, None, 'FF...FF', False, ass_serv=[37]),
+        FileTemplate(0x6f47, 'EF.SMSR',      'LF',   10,   30,   5, None, '00FF...FF', False, ass_serv=[11]),
+        FileTemplate(0x6f4d, 'EF.BDN',       'LF',   10,   25,   8, None, 'FF...FF', False, ass_serv=[6]),
+        FileTemplate(0x6f4e, 'EF.EXT5',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[44]),
+        FileTemplate(0x6f4f, 'EF.CCP2',      'LF',    5,   15,   5, 0x16, 'FF...FF', False, ass_serv=[14]),
+        FileTemplate(0x6f55, 'EF.EXT4',      'LF',   10,   13,   8, None, '00FF...FF', False, ass_serv=[7]),
+        FileTemplate(0x6f57, 'EF.ACL',       'TR', None,  101,   8, None, '00FF...FF', False, ass_serv=[35]),
+        FileTemplate(0x6f58, 'EF.CMI',       'LF',   10,   11,   2, None, 'FF...FF', False, ass_serv=[6]),
+        FileTemplate(0x6f80, 'EF.ICI',       'CY',   20,   38,   5, 0x14, 'FF...FF0000000001FFFF', False, ass_serv=[9], high_update=True),
+        FileTemplate(0x6f81, 'EF.OCI',       'CY',   20,   37,   5, 0x15, 'FF...FF00000001FFFF', False, ass_serv=[8], high_update=True),
+        FileTemplate(0x6f82, 'EF.ICT',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[9], high_update=True),
+        FileTemplate(0x6f83, 'EF.OCT',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[8], high_update=True),
+        FileTemplate(0x6fb1, 'EF.VGCS',      'TR', None,   20,   2, None, None, True, ass_serv=[57]),
+        FileTemplate(0x6fb2, 'EF.VGCSS',     'TR', None,    7,   5, None, None, True, ass_serv=[57]),
+        FileTemplate(0x6fb3, 'EF.VBS',       'TR', None,   20,   2, None, None, True, ass_serv=[58]),
+        FileTemplate(0x6fb4, 'EF.VBSS',      'TR', None,    7,   5, None, None, True, ass_serv=[58]), # ARR 2!??
+        FileTemplate(0x6fb5, 'EF.eMLPP',     'TR', None,    2,   2, None, None, True, ass_serv=[24]),
+        FileTemplate(0x6fb6, 'EF.AaeM',      'TR', None,    1,   5, None, '00', False, ass_serv=[25]),
+        FileTemplate(0x6fc3, 'EF.HiddenKey', 'TR', None,    4,   5, None, 'FF...FF', False),
+        FileTemplate(0x6fc5, 'EF.PNN',       'LF',   10,   16,  10, 0x19, None, True, ass_serv=[45]),
+        FileTemplate(0x6fc6, 'EF.OPL',       'LF',    5,    8,  10, 0x1a, None, True, ass_serv=[46]),
+        FileTemplate(0x6fc7, 'EF.MBDN',      'LF',    3,   24,   5, None, None, True, ass_serv=[47]),
+        FileTemplate(0x6fc8, 'EF.EXT6',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[47]),
+        FileTemplate(0x6fc9, 'EF.MBI',       'LF',   10,    5,   5, None, None, True, ass_serv=[47]),
+        FileTemplate(0x6fca, 'EF.MWIS',      'LF',   10,    6,   5, None, '00...00', False, ass_serv=[48], high_update=True),
+        FileTemplate(0x6fcb, 'EF.CFIS',      'LF',   10,   16,   5, None, '0100FF...FF', False, ass_serv=[49]),
+        FileTemplate(0x6fcb, 'EF.EXT7',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[49]),
+        FileTemplate(0x6fcd, 'EF.SPDI',      'TR', None,   17,   2, 0x1b, None, True, ass_serv=[51]),
+        FileTemplate(0x6fce, 'EF.MMSN',      'LF',   10,    6,   5, None, '000000FF...FF', False, ass_serv=[52]),
+        FileTemplate(0x6fcf, 'EF.EXT8',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[53]),
+        FileTemplate(0x6fd0, 'EF.MMSICP',    'TR', None,  100,   2, None, 'FF...FF', False, ass_serv=[52]),
+        FileTemplate(0x6fd1, 'EF.MMSUP',     'LF', None, None,   5, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[52]),
+        FileTemplate(0x6fd2, 'EF.MMSUCP',    'TR', None,  100,   5, None, 'FF...FF', False, ass_serv=[52,55]),
+        FileTemplate(0x6fd3, 'EF.NIA',       'LF',    5,   11,   2, None, 'FF...FF', False, ass_serv=[56]),
+        FileTemplate(0x6fd4, 'EF.VGCSCA',    'TR', None, None,   2, None, '00...00', False, ['size'], ass_serv=[64]),
+        FileTemplate(0x6fd5, 'EF.VBSCA',     'TR', None, None,   2, None, '00...00', False, ['size'], ass_serv=[65]),
+        FileTemplate(0x6fd6, 'EF.GBABP',     'TR', None, None,   5, None, 'FF...FF', False, ['size'], ass_serv=[68]),
+        FileTemplate(0x6fd7, 'EF.MSK',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[69], high_update=True),
+        FileTemplate(0x6fd8, 'EF.MUK',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[69]),
+        FileTemplate(0x6fd9, 'EF.EHPLMN',    'TR', None,   15,   2, 0x1d, 'FF...FF', False, ass_serv=[71]),
+        FileTemplate(0x6fda, 'EF.GBANL',     'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[68]),
+        FileTemplate(0x6fdb, 'EF.EHPLMNPI',  'TR', None,    1,   2, None, '00', False, ass_serv=[71,73]),
+        FileTemplate(0x6fdc, 'EF.LRPLMNSI',  'TR', None,    1,   2, None, '00', False, ass_serv=[74]),
+        FileTemplate(0x6fdd, 'EF.NAFKCA',    'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[68,76]),
+        FileTemplate(0x6fde, 'EF.SPNI',      'TR', None, None,  10, None, '00FF...FF', False, ['size'], ass_serv=[78]),
+        FileTemplate(0x6fdf, 'EF.PNNI',      'LF', None, None,  10, None, '00FF...FF', False, ['nb_rec','size'], ass_serv=[79]),
+        FileTemplate(0x6fe2, 'EF.NCP-IP',    'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[80]),
+        FileTemplate(0x6fe6, 'EF.UFC',       'TR', None,   30,  10, None, '801E60C01E900080040000000000000000F0000000004000000000000080', False),
+        FileTemplate(0x6fe8, 'EF.NASCONFIG', 'TR', None,   18,   2, None, None, True, ass_serv=[96]),
+        FileTemplate(0x6fe7, 'EF.UICCIARI',  'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[95]),
+        FileTemplate(0x6fec, 'EF.PWS',       'TR', None, None,  10, None, None, True, ['size'], ass_serv=[97]),
+        FileTemplate(0x6fed, 'EF.FDNURI',    'LF', None, None,   8, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2,99]),
+        FileTemplate(0x6fee, 'EF.BDNURI',    'LF', None, None,   8, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[6,99]),
+        FileTemplate(0x6fef, 'EF.SDNURI',    'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[4,99]),
+        FileTemplate(0x6ff0, 'EF.IWL',       'LF', None, None,   3, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[102]),
+        FileTemplate(0x6ff1, 'EF.IPS',       'CY', None,    4,  10, None, 'FF...FF', False, ['size'], ass_serv=[102], high_update=True),
+        FileTemplate(0x6ff2, 'EF.IPD',       'LF', None, None,   3, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[102], high_update=True),
+    ]
+
+
+# Section 9.5.2
+class FilesUsimOptionalV2(ProfileTemplate):
+    """Optional Files at ADF.USIM as per Section 9.5.2"""
+    created_by_default = False
+    optional = True
+    oid = OID.ADF_USIMopt_not_by_default_v2
+    base_path = Path('ADF.USIM')
+    extends = FilesUsimMandatoryV2
+    files = [
+        FileTemplate(0x6f05, 'EF.LI',        'TR', None,    6,   1, 0x02, 'FF...FF', False),
+        FileTemplate(0x6f37, 'EF.ACMmax',    'TR', None,    3,   5, None, '000000', False, ass_serv=[13]),
+        FileTemplate(0x6f39, 'EF.ACM',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[13], high_update=True),
+        FileTemplate(0x6f3e, 'EF.GID1',      'TR', None,    8,   2, None, None, True, ass_serv=[17]),
+        FileTemplate(0x6f3f, 'EF.GID2',      'TR', None,    8,   2, None, None, True, ass_serv=[18]),
+        FileTemplate(0x6f40, 'EF.MSISDN',    'LF',    1,   24,   2, None, 'FF...FF', False, ass_serv=[21]),
+        FileTemplate(0x6f41, 'EF.PUCT',      'TR', None,    5,   5, None, 'FFFFFF0000', False, ass_serv=[13]),
+        FileTemplate(0x6f45, 'EF.CBMI',      'TR', None,   10,   5, None, 'FF...FF', False, ass_serv=[15]),
+        FileTemplate(0x6f48, 'EF.CBMID',     'TR', None,   10,   2, 0x0e, 'FF...FF', False, ass_serv=[19]),
+        FileTemplate(0x6f49, 'EF.SDN',       'LF',   10,   24,   2, None, 'FF...FF', False, ass_serv=[4,89]),
+        FileTemplate(0x6f4b, 'EF.EXT2',      'LF',   10,   13,   8, None, '00FF...FF', False, ass_serv=[3]),
+        FileTemplate(0x6f4c, 'EF.EXT3',      'LF',   10,   13,   2, None, '00FF...FF', False, ass_serv=[5]),
+        FileTemplate(0x6f50, 'EF.CBMIR',     'TR', None,   20,   5, None, 'FF...FF', False, ass_serv=[16]),
+        FileTemplate(0x6f60, 'EF.PLMNwAcT',  'TR', None,   40,   5, 0x0a, 'FFFFFF0000'*8, False, ass_serv=[20]),
+        FileTemplate(0x6f61, 'EF.OPLMNwAcT', 'TR', None,   40,   2, 0x11, 'FFFFFF0000'*8, False, ass_serv=[42]),
+        FileTemplate(0x6f62, 'EF.HPLMNwAcT', 'TR', None,    5,   2, 0x13, 'FFFFFF0000', False, ass_serv=[43]),
+        FileTemplate(0x6f2c, 'EF.DCK',       'TR', None,   16,   5, None, 'FF...FF', False, ass_serv=[36]),
+        FileTemplate(0x6f32, 'EF.CNL',       'TR', None,   30,   2, None, 'FF...FF', False, ass_serv=[37]),
+        FileTemplate(0x6f47, 'EF.SMSR',      'LF',   10,   30,   5, None, '00FF...FF', False, ass_serv=[11]),
+        FileTemplate(0x6f4d, 'EF.BDN',       'LF',   10,   25,   8, None, 'FF...FF', False, ass_serv=[6]),
+        FileTemplate(0x6f4e, 'EF.EXT5',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[44]),
+        FileTemplate(0x6f4f, 'EF.CCP2',      'LF',    5,   15,   5, 0x16, 'FF...FF', False, ass_serv=[14]),
+        FileTemplate(0x6f55, 'EF.EXT4',      'LF',   10,   13,   8, None, '00FF...FF', False, ass_serv=[7]),
+        FileTemplate(0x6f57, 'EF.ACL',       'TR', None,  101,   8, None, '00FF...FF', False, ass_serv=[35]),
+        FileTemplate(0x6f58, 'EF.CMI',       'LF',   10,   11,   2, None, 'FF...FF', False, ass_serv=[6]),
+        FileTemplate(0x6f80, 'EF.ICI',       'CY',   20,   38,   5, 0x14, 'FF...FF0000000001FFFF', False, ass_serv=[9], high_update=True),
+        FileTemplate(0x6f81, 'EF.OCI',       'CY',   20,   37,   5, 0x15, 'FF...FF00000001FFFF', False, ass_serv=[8], high_update=True),
+        FileTemplate(0x6f82, 'EF.ICT',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[9], high_update=True),
+        FileTemplate(0x6f83, 'EF.OCT',       'CY',    1,    3,   7, None, '000000', False, ass_serv=[8], high_update=True),
+        FileTemplate(0x6fb1, 'EF.VGCS',      'TR', None,   20,   2, None, None, True, ass_serv=[57]),
+        FileTemplate(0x6fb2, 'EF.VGCSS',     'TR', None,    7,   5, None, None, True, ass_serv=[57]),
+        FileTemplate(0x6fb3, 'EF.VBS',       'TR', None,   20,   2, None, None, True, ass_serv=[58]),
+        FileTemplate(0x6fb4, 'EF.VBSS',      'TR', None,    7,   5, None, None, True, ass_serv=[58]), # ARR 2!??
+        FileTemplate(0x6fb5, 'EF.eMLPP',     'TR', None,    2,   2, None, None, True, ass_serv=[24]),
+        FileTemplate(0x6fb6, 'EF.AaeM',      'TR', None,    1,   5, None, '00', False, ass_serv=[25]),
+        FileTemplate(0x6fc3, 'EF.HiddenKey', 'TR', None,    4,   5, None, 'FF...FF', False),
+        FileTemplate(0x6fc5, 'EF.PNN',       'LF',   10,   16,  10, 0x19, None, True, ass_serv=[45]),
+        FileTemplate(0x6fc6, 'EF.OPL',       'LF',    5,    8,  10, 0x1a, None, True, ass_serv=[46]),
+        FileTemplate(0x6fc7, 'EF.MBDN',      'LF',    3,   24,   5, None, None, True, ass_serv=[47]),
+        FileTemplate(0x6fc8, 'EF.EXT6',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[47]),
+        FileTemplate(0x6fc9, 'EF.MBI',       'LF',   10,    5,   5, None, None, True, ass_serv=[47]),
+        FileTemplate(0x6fca, 'EF.MWIS',      'LF',   10,    6,   5, None, '00...00', False, ass_serv=[48], high_update=True),
+        FileTemplate(0x6fcb, 'EF.CFIS',      'LF',   10,   16,   5, None, '0100FF...FF', False, ass_serv=[49]),
+        FileTemplate(0x6fcb, 'EF.EXT7',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[49]),
+        FileTemplate(0x6fcd, 'EF.SPDI',      'TR', None,   17,   2, 0x1b, None, True, ass_serv=[51]),
+        FileTemplate(0x6fce, 'EF.MMSN',      'LF',   10,    6,   5, None, '000000FF...FF', False, ass_serv=[52]),
+        FileTemplate(0x6fcf, 'EF.EXT8',      'LF',   10,   13,   5, None, '00FF...FF', False, ass_serv=[53]),
+        FileTemplate(0x6fd0, 'EF.MMSICP',    'TR', None,  100,   2, None, 'FF...FF', False, ass_serv=[52]),
+        FileTemplate(0x6fd1, 'EF.MMSUP',     'LF', None, None,   5, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[52]),
+        FileTemplate(0x6fd2, 'EF.MMSUCP',    'TR', None,  100,   5, None, 'FF...FF', False, ass_serv=[52,55]),
+        FileTemplate(0x6fd3, 'EF.NIA',       'LF',    5,   11,   2, None, 'FF...FF', False, ass_serv=[56]),
+        FileTemplate(0x6fd4, 'EF.VGCSCA',    'TR', None, None,   2, None, '00...00', False, ['size'], ass_serv=[64]),
+        FileTemplate(0x6fd5, 'EF.VBSCA',     'TR', None, None,   2, None, '00...00', False, ['size'], ass_serv=[65]),
+        FileTemplate(0x6fd6, 'EF.GBABP',     'TR', None, None,   5, None, 'FF...FF', False, ['size'], ass_serv=[68]),
+        FileTemplate(0x6fd7, 'EF.MSK',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[69], high_update=True),
+        FileTemplate(0x6fd8, 'EF.MUK',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[69]),
+        FileTemplate(0x6fd9, 'EF.EHPLMN',    'TR', None,   15,   2, 0x1d, 'FF...FF', False, ass_serv=[71]),
+        FileTemplate(0x6fda, 'EF.GBANL',     'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[68]),
+        FileTemplate(0x6fdb, 'EF.EHPLMNPI',  'TR', None,    1,   2, None, '00', False, ass_serv=[71,73]),
+        FileTemplate(0x6fdc, 'EF.LRPLMNSI',  'TR', None,    1,   2, None, '00', False, ass_serv=[74]),
+        FileTemplate(0x6fdd, 'EF.NAFKCA',    'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[68,76]),
+        FileTemplate(0x6fde, 'EF.SPNI',      'TR', None, None,  10, None, '00FF...FF', False, ['size'], ass_serv=[78]),
+        FileTemplate(0x6fdf, 'EF.PNNI',      'LF', None, None,  10, None, '00FF...FF', False, ['nb_rec','size'], ass_serv=[79]),
+        FileTemplate(0x6fe2, 'EF.NCP-IP',    'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[80]),
+        FileTemplate(0x6fe6, 'EF.UFC',       'TR', None,   30,  10, None, '801E60C01E900080040000000000000000F0000000004000000000000080', False),
+        FileTemplate(0x6fe8, 'EF.NASCONFIG', 'TR', None,   18,   2, None, None, True, ass_serv=[96]),
+        FileTemplate(0x6fe7, 'EF.UICCIARI',  'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[95]),
+        FileTemplate(0x6fec, 'EF.PWS',       'TR', None, None,  10, None, None, True, ['size'], ass_serv=[97]),
+        FileTemplate(0x6fed, 'EF.FDNURI',    'LF', None, None,   8, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2,99]),
+        FileTemplate(0x6fee, 'EF.BDNURI',    'LF', None, None,   8, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[6,99]),
+        FileTemplate(0x6fef, 'EF.SDNURI',    'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[4,99]),
+        FileTemplate(0x6ff0, 'EF.IWL',       'LF', None, None,   3, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[102]),
+        FileTemplate(0x6ff1, 'EF.IPS',       'CY', None,    4,  10, None, 'FF...FF', False, ['size'], ass_serv=[102], high_update=True),
+        FileTemplate(0x6ff2, 'EF.IPD',       'LF', None, None,   3, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[102], high_update=True),
+        FileTemplate(0x6ff3, 'EF.EPDGID',    'TR', None, None,   2, None, None, True, ['size'], ass_serv=[(106, 107)]),
+        FileTemplate(0x6ff4, 'EF.EPDGSELECTION','TR',None,None,  2, None, None, True, ['size'], ass_serv=[(106, 107)]),
+        FileTemplate(0x6ff5, 'EF.EPDGIDEM',  'TR', None, None,   2, None, None, True, ['size'], ass_serv=[(110, 111)]),
+        FileTemplate(0x6ff6, 'EF.EPDGIDEMSEL','TR',None, None,   2, None, None, True, ['size'], ass_serv=[(110, 111)]),
+        FileTemplate(0x6ff7, 'EF.FromPreferred','TR',None,  1,   2, None, '00', False, ass_serv=[114]),
+        FileTemplate(0x6ff8, 'EF.IMSConfigData','BT',None,None,  2, None, None, True, ['size'], ass_serv=[115]),
+        FileTemplate(0x6ff9, 'EF.3GPPPSDataOff','TR',None,  4,   2, None, None, True, ass_serv=[117]),
+        FileTemplate(0x6ffa, 'EF.3GPPPSDOSLIST','LF',None, None, 2, None, None, True, ['nb_rec','size'], ass_serv=[118]),
+        FileTemplate(0x6ffc, 'EF.XCAPConfigData','BT',None,None, 2, None, None, True, ['size'], ass_serv=[120]),
+        FileTemplate(0x6ffd, 'EF.EARFCNLIST','TR', None, None,  10, None, None, True, ['size'], ass_serv=[121]),
+        FileTemplate(0x6ffd, 'EF.MudMidCfgdata','BT', None, None,2, None, None, True, ['size'], ass_serv=[134]),
+    ]
+
+class FilesUsimOptionalV3(ProfileTemplate):
+    """Optional Files at ADF.USIM as per Section 9.5.2.3 v3.3.1"""
+    created_by_default = False
+    optional = True
+    oid = OID.ADF_USIMopt_not_by_default_v3
+    base_path = Path('ADF.USIM')
+    extends = FilesUsimMandatoryV2
+    files = FilesUsimOptionalV2.files + [
+        FileTemplate(0x6f01, 'EF.eAKA', 'TR', None, 1, 3, None, None, True, ['size'], ass_serv=[134]),
+    ]
+
+class FilesUsimDfPhonebook(ProfileTemplate):
+    """DF.PHONEBOOK Files at ADF.USIM as per Section 9.5.3"""
+    created_by_default = False
+    oid = OID.DF_PHONEBOOK_ADF_USIM
+    base_path = Path('ADF.USIM')
+    files = df_pb_files
+
+
+class FilesUsimDfGsmAccess(ProfileTemplate):
+    """DF.GSM-ACCESS Files at ADF.USIM as per Section 9.5.4"""
+    created_by_default = False
+    oid = OID.DF_GSM_ACCESS_ADF_USIM
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x5f3b, 'DF.GSM-ACCESS','DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[27]),
+        FileTemplate(0x4f20, 'EF.Kc',        'TR', None,    9,   5, 0x01, 'FF...FF07', False, ass_serv=[27], high_update=True),
+        FileTemplate(0x4f52, 'EF.KcGPRS',    'TR', None,    9,   5, 0x02, 'FF...FF07', False, ass_serv=[27], high_update=True),
+        FileTemplate(0x4f63, 'EF.CPBCCH',    'TR', None,   10,   5, None, 'FF...FF', False, ass_serv=[39], high_update=True),
+        FileTemplate(0x4f64, 'EF.InvScan',   'TR', None,    1,   2, None, '00', False, ass_serv=[40]),
+    ]
+
+
+class FilesUsimDf5GS(ProfileTemplate):
+    """DF.5GS Files at ADF.USIM as per Section 9.5.11 v2.3.1"""
+    created_by_default = False
+    oid = OID.DF_5GS
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6fc0, 'DF.5GS',               'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[122,126,127,128,129,130], pe_name='df-df-5gs'),
+        FileTemplate(0x4f01, 'EF.5GS3GPPLOCI',       'TR', None,   20,   5, 0x01, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f02, 'EF.5GSN3GPPLOCI',      'TR', None,   20,   5, 0x02, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f03, 'EF.5GS3GPPNSC',        'LF',    1,   57,   5, 0x03, 'FF...FF', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f04, 'EF.5GSN3GPPNSC',       'LF',    1,   57,   5, 0x04, 'FF...FF', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f05, 'EF.5GAUTHKEYS',        'TR', None,  110,   5, 0x05, None, True, ass_serv=[123], high_update=True),
+        FileTemplate(0x4f06, 'EF.UAC_AIC',           'TR', None,    4,   2, 0x06, None, True, ass_serv=[126]),
+        FileTemplate(0x4f07, 'EF.SUCI_Calc_Info',    'TR', None, None,   2, 0x07, 'FF...FF', False, ass_serv=[124]),
+        FileTemplate(0x4f08, 'EF.OPL5G',             'LF', None,   10,  10, 0x08, 'FF...FF', False, ['nb_rec'], ass_serv=[129]),
+        FileTemplate(0x4f09, 'EF.SUPI_NAI',          'TR', None, None,   2, 0x09, None, True, ['size'], ass_serv=[130], pe_name='ef-supinai'),
+        FileTemplate(0x4f0a, 'EF.Routing_Indicator', 'TR', None,    4,   2, 0x0a, 'F0FFFFFF', False, ass_serv=[124]),
+    ]
+
+
+class FilesUsimDf5GSv2(ProfileTemplate):
+    """DF.5GS Files at ADF.USIM as per Section 9.5.11.2"""
+    created_by_default = False
+    oid = OID.DF_5GS_v2
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6fc0, 'DF.5GS',               'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[122,126,127,128,129,130], pe_name='df-df-5gs'),
+        FileTemplate(0x4f01, 'EF.5GS3GPPLOCI',       'TR', None,   20,   5, 0x01, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f02, 'EF.5GSN3GPPLOCI',      'TR', None,   20,   5, 0x02, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f03, 'EF.5GS3GPPNSC',        'LF',    1,   57,   5, 0x03, 'FF...FF', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f04, 'EF.5GSN3GPPNSC',       'LF',    1,   57,   5, 0x04, 'FF...FF', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f05, 'EF.5GAUTHKEYS',        'TR', None,  110,   5, 0x05, None, True, ass_serv=[123], high_update=True),
+        FileTemplate(0x4f06, 'EF.UAC_AIC',           'TR', None,    4,   2, 0x06, None, True, ass_serv=[126]),
+        FileTemplate(0x4f07, 'EF.SUCI_Calc_Info',    'TR', None, None,   2, 0x07, 'FF...FF', False, ass_serv=[124]),
+        FileTemplate(0x4f08, 'EF.OPL5G',             'LF', None,   10,  10, 0x08, 'FF...FF', False, ['nb_rec'], ass_serv=[129]),
+        FileTemplate(0x4f09, 'EF.SUPI_NAI',          'TR', None, None,   2, 0x09, None, True, ['size'], ass_serv=[130]),
+        FileTemplate(0x4f0a, 'EF.Routing_Indicator', 'TR', None,    4,   2, 0x0a, 'F0FFFFFF', False, ass_serv=[124]),
+        FileTemplate(0x4f0b, 'EF.URSP',              'BT', None, None,   2, None, None, False, ass_serv=[132]),
+        FileTemplate(0x4f0c, 'EF.TN3GPPSNN',         'TR', None,    1,   2, 0x0c, '00', False, ass_serv=[135]),
+    ]
+
+
+class FilesUsimDf5GSv3(ProfileTemplate):
+    """DF.5GS Files at ADF.USIM as per Section 9.5.11.3"""
+    created_by_default = False
+    oid = OID.DF_5GS_v3
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6fc0, 'DF.5GS',               'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[122,126,127,128,129,130], pe_name='df-df-5gs'),
+        FileTemplate(0x4f01, 'EF.5GS3GPPLOCI',       'TR', None,   20,   5, 0x01, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f02, 'EF.5GSN3GPPLOCI',      'TR', None,   20,   5, 0x02, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f03, 'EF.5GS3GPPNSC',        'LF',    2,   62,   5, 0x03, 'FF...FF', False, ass_serv=[122,136], high_update=True),
+        # ^ If Service n°136 is not "available" in EF UST, the Profile Creator shall ensure that these files shall contain one record; otherwise, they shall contain 2 records.
+        FileTemplate(0x4f04, 'EF.5GSN3GPPNSC',       'LF',    2,   62,   5, 0x04, 'FF...FF', False, ass_serv=[122,136], high_update=True),
+        # ^ If Service n°136 is not "available" in EF UST, the Profile Creator shall ensure that these files shall contain one record; otherwise, they shall contain 2 records.
+        FileTemplate(0x4f05, 'EF.5GAUTHKEYS',        'TR', None,  110,   5, 0x05, None, True, ass_serv=[123], high_update=True),
+        FileTemplate(0x4f06, 'EF.UAC_AIC',           'TR', None,    4,   2, 0x06, None, True, ass_serv=[126]),
+        FileTemplate(0x4f07, 'EF.SUCI_Calc_Info',    'TR', None, None,   2, 0x07, 'FF...FF', False, ass_serv=[124]),
+        FileTemplate(0x4f08, 'EF.OPL5G',             'LF', None,   10,  10, 0x08, 'FF...FF', False, ['nb_rec'], ass_serv=[129]),
+        FileTemplate(0x4f09, 'EF.SUPI_NAI',          'TR', None, None,   2, 0x09, None, True, ['size'], ass_serv=[130], pe_name='ef-supinai'),
+        FileTemplate(0x4f0a, 'EF.Routing_Indicator', 'TR', None,    4,   2, 0x0a, 'F0FFFFFF', False, ass_serv=[124]),
+        FileTemplate(0x4f0b, 'EF.URSP',              'BT', None, None,   2, None, None, False, ass_serv=[132]),
+        FileTemplate(0x4f0c, 'EF.TN3GPPSNN',         'TR', None,    1,   2, 0x0c, '00', False, ass_serv=[135]),
+    ]
+
+class FilesUsimDf5GSv4(ProfileTemplate):
+    """DF.5GS Files at ADF.USIM as per Section 9.5.11.4"""
+    created_by_default = False
+    oid = OID.DF_5GS_v4
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6fc0, 'DF.5GS',               'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[122,126,127,128,129,130], pe_name='df-df-5gs'),
+        FileTemplate(0x4f01, 'EF.5GS3GPPLOCI',       'TR', None,   20,   5, 0x01, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f02, 'EF.5GSN3GPPLOCI',      'TR', None,   20,   5, 0x02, 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00000001', False, ass_serv=[122], high_update=True),
+        FileTemplate(0x4f03, 'EF.5GS3GPPNSC',        'LF',    2,   62,   5, 0x03, 'FF...FF', False, ass_serv=[122,136], high_update=True),
+        # ^ If Service n°136 is not "available" in EF UST, the Profile Creator shall ensure that these files shall contain one record; otherwise, they shall contain 2 records.
+        FileTemplate(0x4f04, 'EF.5GSN3GPPNSC',       'LF',    2,   62,   5, 0x04, 'FF...FF', False, ass_serv=[122,136], high_update=True),
+        # ^ If Service n°136 is not "available" in EF UST, the Profile Creator shall ensure that these files shall contain one record; otherwise, they shall contain 2 records.
+        FileTemplate(0x4f05, 'EF.5GAUTHKEYS',        'TR', None,  110,   5, 0x05, None, True, ass_serv=[123], high_update=True),
+        FileTemplate(0x4f06, 'EF.UAC_AIC',           'TR', None,    4,   2, 0x06, None, True, ass_serv=[126]),
+        FileTemplate(0x4f07, 'EF.SUCI_Calc_Info',    'TR', None, None,   2, 0x07, 'FF...FF', False, ass_serv=[124]),
+        FileTemplate(0x4f08, 'EF.OPL5G',             'LF', None,   10,  10, 0x08, 'FF...FF', False, ['nb_rec'], ass_serv=[129]),
+        FileTemplate(0x4f09, 'EF.SUPI_NAI',          'TR', None, None,   2, 0x09, None, True, ['size'], ass_serv=[130], pe_name='ef-supinai'),
+        FileTemplate(0x4f0a, 'EF.Routing_Indicator', 'TR', None,    4,   2, 0x0a, 'F0FF0000', False, ass_serv=[124]),
+        FileTemplate(0x4f0b, 'EF.URSP',              'BT', None, None,   2, None, None, False, ass_serv=[132]),
+        FileTemplate(0x4f0c, 'EF.TN3GPPSNN',         'TR', None,    1,   2, 0x0c, '00', False, ass_serv=[135]),
+        FileTemplate(0x4f0d, 'EF.CAG',               'TR', None,    2,   2, 0x0d, None, True, ass_serv=[137]),
+        FileTemplate(0x4f0e, 'EF.SOR_CMCI',          'TR', None, None,   2, 0x0e, None, True, ass_serv=[138]),
+        FileTemplate(0x4f0f, 'EF.DRI',               'TR', None,    7,   2, 0x0f, None, True, ass_serv=[150]),
+        FileTemplate(0x4f10, 'EF.5GSEDRX',           'TR', None,    2,   2, 0x10, None, True, ass_serv=[141]),
+        FileTemplate(0x4f11, 'EF.5GNSWO_CONF',       'TR', None,    1,   2, 0x11, None, True, ass_serv=[142]),
+        FileTemplate(0x4f15, 'EF.MCHPPLMN',          'TR', None,    1,   2, 0x15, None, True, ass_serv=[144]),
+        FileTemplate(0x4f16, 'EF.KAUSF_DERIVATION',  'TR', None,    1,   2, 0x16, None, True, ass_serv=[145]),
+    ]
+
+
+class FilesUsimDfSaip(ProfileTemplate):
+    """DF.SAIP Files at ADF.USIM as per Section 9.5.12"""
+    created_by_default = False
+    oid = OID.DF_SAIP
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x6fd0, 'DF.SAIP',        'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[(124, 125)], pe_name='df-df-saip'),
+        FileTemplate(0x4f01, 'EF.SUCICalcInfo','TR', None, None, 3, None, 'FF...FF', False, ['size'], ass_serv=[125], pe_name='ef-suci-calc-info-usim'),
+    ]
+
+class FilesDfSnpn(ProfileTemplate):
+    """DF.SNPN Files at ADF.USIM as per Section 9.5.13"""
+    created_by_default = False
+    oid = OID.DF_SNPN
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x5fe0, 'DF.SNPN',         'DF', None, None,  14, None, None, False, ['pinStatusTemplateDO'], ass_serv=[143], pe_name='df-df-snpn'),
+        FileTemplate(0x4f01, 'EF.PWS_SNPN',     'TR', None,    1,  10, None, None, True, ass_serv=[143]),
+    ]
+
+class FilesDf5GProSe(ProfileTemplate):
+    """DF.ProSe Files at ADF.USIM as per Section 9.5.14"""
+    created_by_default = False
+    oid = OID.DF_5GProSe
+    base_path = Path('ADF.USIM')
+    parent = FilesUsimMandatory
+    files = [
+        FileTemplate(0x5ff0, 'DF.5G_ProSe',       'DF', None, None,  14, None, None, False, ['pinStatusTeimplateDO'], ass_serv=[139], pe_name='df-df-5g-prose'),
+        FileTemplate(0x4f01, 'EF.5G_PROSE_ST',    'TR', None,    1,   2, 0x01, None,  True, ass_serv=[139]),
+        FileTemplate(0x4f02, 'EF.5G_PROSE_DD',    'TR', None,   26,   2, 0x02, None,  True, ass_serv=[139,1001]),
+        FileTemplate(0x4f03, 'EF.5G_PROSE_DC',    'TR', None,   12,   2, 0x03, None,  True, ass_serv=[139,1002]),
+        FileTemplate(0x4f04, 'EF.5G_PROSE_U2NRU', 'TR', None,   32,   2, 0x04, None,  True, ass_serv=[139,1003]),
+        FileTemplate(0x4f05, 'EF.5G_PROSE_RU',    'TR', None,   29,   2, 0x05, None,  True, ass_serv=[139,1004]),
+        FileTemplate(0x4f06, 'EF.5G_PROSE_UIR',   'TR', None,   32,   2, 0x06, None,  True, ass_serv=[139,1005]),
+    ]
+
+class FilesIsimMandatory(ProfileTemplate):
+    """Mandatory Files at ADF.ISIM as per Section 9.6.1"""
+    created_by_default = True
+    oid = OID.ADF_ISIM_by_default
+    files = [
+        FileTemplate(  None, 'ADF.ISIM',      'ADF', None, None,  14, None, None, False, ['aid','temporary_fid','pinStatusTemplateDO']),
+        FileTemplate(0x6f02, 'EF.IMPI',        'TR', None, None,   2, 0x02, None, True, ['size']),
+        FileTemplate(0x6f04, 'EF.IMPU',        'LF',    1, None,   2, 0x04, None, True, ['size']),
+        FileTemplate(0x6f03, 'EF.Domain',      'TR', None, None,   2, 0x05, None, True, ['size']),
+        FileTemplate(0x6f07, 'EF.IST',         'TR', None,   14,   2, 0x07, None, True),
+        FileTemplate(0x6fad, 'EF.AD',          'TR', None,    3,  10, 0x03, '000000', False),
+        FileTemplate(0x6f06, 'EF.ARR',         'LF', None, None,  10, 0x06, None, True, ['nb_rec','size']),
+    ]
+
+
+class FilesIsimOptional(ProfileTemplate):
+    """Optional Files at ADF.ISIM as per Section 9.6.2 of v2.3.1"""
+    created_by_default = False
+    optional = True
+    oid = OID.ADF_ISIMopt_not_by_default
+    base_path = Path('ADF.ISIM')
+    extends = FilesIsimMandatory
+    files = [
+        FileTemplate(0x6f09, 'EF.P-CSCF',      'LF',    1, None,   2, None, None, True, ['size'], ass_serv=[1,5], pe_name='ef-pcscf'),
+        FileTemplate(0x6f3c, 'EF.SMS',         'LF',   10,  176,   5, None, '00FF...FF', False, ass_serv=[6,8]),
+        FileTemplate(0x6f42, 'EF.SMSP',        'LF',    1,   38,   5, None, 'FF...FF', False, ass_serv=[8]),
+        FileTemplate(0x6f43, 'EF.SMSS',        'TR', None,    2,   5, None, 'FFFF', False, ass_serv=[6,8]),
+        FileTemplate(0x6f47, 'EF.SMSR',        'LF',   10,   30,   5, None, '00FF...FF', False, ass_serv=[7,8]),
+        FileTemplate(0x6fd5, 'EF.GBABP',       'TR', None, None,   5, None, 'FF...FF', False, ['size'], ass_serv=[2]),
+        FileTemplate(0x6fd7, 'EF.GBANL',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2]),
+        FileTemplate(0x6fdd, 'EF.NAFKCA',      'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2,4]),
+        FileTemplate(0x6fe7, 'EF.UICCIARI',    'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[10]),
+    ]
+
+
+class FilesIsimOptionalv2(ProfileTemplate):
+    """Optional Files at ADF.ISIM as per Section 9.6.2"""
+    created_by_default = False
+    optional = True
+    oid = OID.ADF_ISIMopt_not_by_default_v2
+    base_path = Path('ADF.ISIM')
+    extends = FilesIsimMandatory
+    files = [
+        FileTemplate(0x6f09, 'EF.PCSCF',       'LF',    1, None,   2, None, None, True, ['size'], ass_serv=[1,5]),
+        FileTemplate(0x6f3c, 'EF.SMS',         'LF',   10,  176,   5, None, '00FF...FF', False, ass_serv=[6,8]),
+        FileTemplate(0x6f42, 'EF.SMSP',        'LF',    1,   38,   5, None, 'FF...FF', False, ass_serv=[8]),
+        FileTemplate(0x6f43, 'EF.SMSS',        'TR', None,    2,   5, None, 'FFFF', False, ass_serv=[6,8]),
+        FileTemplate(0x6f47, 'EF.SMSR',        'LF',   10,   30,   5, None, '00FF...FF', False, ass_serv=[7,8]),
+        FileTemplate(0x6fd5, 'EF.GBABP',       'TR', None, None,   5, None, 'FF...FF', False, ['size'], ass_serv=[2]),
+        FileTemplate(0x6fd7, 'EF.GBANL',       'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2]),
+        FileTemplate(0x6fdd, 'EF.NAFKCA',      'LF', None, None,   2, None, 'FF...FF', False, ['nb_rec','size'], ass_serv=[2,4]),
+        FileTemplate(0x6fe7, 'EF.UICCIARI',    'LF', None, None,   2, None, None, True, ['nb_rec','size'], ass_serv=[10]),
+        FileTemplate(0x6ff7, 'EF.FromPreferred','TR', None,   1,   2, None, '00', False, ass_serv=[17]),
+        FileTemplate(0x6ff8, 'EF.ImsConfigData','BT', None,None,   2, None, None, True, ['size'], ass_serv=[18]),
+        FileTemplate(0x6ffc, 'EF.XcapconfigData','BT',None,None,   2, None, None, True, ['size'], ass_serv=[19]),
+        FileTemplate(0x6ffa, 'EF.WebRTCURI',   'LF', None, None,   2, None, None, True, ['nb_rec', 'size'], ass_serv=[20]),
+        FileTemplate(0x6ffa, 'EF.MudMidCfgData','BT',None, None,   2, None, None, True, ['size'], ass_serv=[21]),
+    ]
+
+
+# TODO: CSIM
+
+
+class FilesEap(ProfileTemplate):
+    """Files at DF.EAP as per Section 9.8"""
+    created_by_default = False
+    oid = OID.DF_EAP
+    files = [
+        FileTemplate(  None, 'DF.EAP',         'DF', None, None,  14, None, None, False, ['fid','pinStatusTemplateDO'], ass_serv=[(124, 125)]),
+        FileTemplate(0x4f01, 'EF.EAPKEYS',     'TR', None, None,   2, None, None, True, ['size'], high_update=True),
+        FileTemplate(0x4f02, 'EF.EAPSTATUS',   'TR', None,    1,   2, None, '00', False, high_update=True),
+        FileTemplate(0x4f03, 'EF.PUId',        'TR', None, None,   2, None, None, True, ['size']),
+        FileTemplate(0x4f04, 'EF.Ps',          'TR', None, None,   5, None, 'FF...FF', False, ['size'], high_update=True),
+        FileTemplate(0x4f20, 'EF.CurID',       'TR', None, None,   5, None, 'FF...FF', False, ['size'], high_update=True),
+        FileTemplate(0x4f21, 'EF.RelID',       'TR', None, None,   5, None, None, True, ['size']),
+        FileTemplate(0x4f22, 'EF.Realm',       'TR', None, None,   5, None, None, True, ['size']),
+    ]
+
+
+# Section 9.9 Access Rules Definition
+ARR_DEFINITION = {
+     1: ['8001019000', '800102A406830101950108', '800158A40683010A950108'],
+     2: ['800101A406830101950108', '80015AA40683010A950108'],
+     3: ['80015BA40683010A950108'],
+     4: ['8001019000', '80011A9700', '800140A40683010A950108'],
+     5: ['800103A406830101950108', '800158A40683010A950108'],
+     6: ['800111A406830101950108', '80014AA40683010A950108'],
+     7: ['800103A406830101950108', '800158A40683010A950108', '840132A406830101950108'],
+     8: ['800101A406830101950108', '800102A406830181950108', '800158A40683010A950108'],
+     9: ['8001019000', '80011AA406830101950108', '800140A40683010A950108'],
+    10: ['8001019000', '80015AA40683010A950108'],
+    11: ['8001019000', '800118A40683010A950108', '8001429700'],
+    12: ['800101A406830101950108', '80015A9700'],
+    13: ['800113A406830101950108', '800148A40683010A950108'],
+    14: ['80015EA40683010A950108'],
+}
+
+class SaipSpecVersionMeta(type):
+    def __getitem__(self, ver: str):
+        """Syntactic sugar so that SaipSpecVersion['2.3.0'] will work."""
+        return SaipSpecVersion.for_version(ver)
+
+class SaipSpecVersion(object, metaclass=SaipSpecVersionMeta):
+    """Represents a specific version of the SIMalliance / TCA eUICC Profile Package:
+       Interoperable Format Technical Specification."""
+    version = None
+    oids = []
+
+    @classmethod
+    def suports_template_OID(cls, OID: OID.OID) -> bool:
+        """Return if a given spec version supports a template of given OID."""
+        return OID in cls.oids
+
+    @classmethod
+    def version_match(cls, ver: str) -> bool:
+        """Check if the given version-string matches the classes version.  trailing zeroes are ignored,
+        so that for example 2.2.0 will be considered equal to 2.2"""
+        def strip_trailing_zeroes(l: List):
+            while l[-1] == '0':
+                l.pop()
+        cls_ver_l = cls.version.split('.')
+        strip_trailing_zeroes(cls_ver_l)
+        ver_l = ver.split('.')
+        strip_trailing_zeroes(ver_l)
+        return cls_ver_l == ver_l
+
+    @staticmethod
+    def for_version(req_version: str) -> Optional['SaipSpecVersion']:
+        """Return the subclass for the requested version number string."""
+        for cls in all_subclasses(SaipSpecVersion):
+            if cls.version_match(req_version):
+                return cls
+
+
+class SaipSpecVersion101(SaipSpecVersion):
+    version = '1.0.1'
+    oids = [OID.MF, OID.DF_CD, OID.DF_TELECOM, OID.ADF_USIM_by_default, OID.ADF_USIMopt_not_by_default,
+            OID.DF_PHONEBOOK_ADF_USIM, OID.DF_GSM_ACCESS_ADF_USIM, OID.ADF_ISIM_by_default,
+            OID.ADF_ISIMopt_not_by_default, OID.ADF_CSIM_by_default, OID.ADF_CSIMopt_not_by_default]
+
+class SaipSpecVersion20(SaipSpecVersion):
+    version = '2.0'
+    # no changes in filesystem teplates to previous 1.0.1
+    oids = SaipSpecVersion101.oids
+
+class SaipSpecVersion21(SaipSpecVersion):
+    version = '2.1'
+    # no changes in filesystem teplates to previous 2.0
+    oids = SaipSpecVersion20.oids
+
+class SaipSpecVersion22(SaipSpecVersion):
+    version = '2.2'
+    oids = SaipSpecVersion21.oids + [OID.DF_EAP]
+
+class SaipSpecVersion23(SaipSpecVersion):
+    version = '2.3'
+    oids = SaipSpecVersion22.oids + [OID.DF_5GS, OID.DF_SAIP]
+
+class SaipSpecVersion231(SaipSpecVersion):
+    version = '2.3.1'
+    # no changes in filesystem teplates to previous 2.3
+    oids = SaipSpecVersion23.oids
+
+class SaipSpecVersion31(SaipSpecVersion):
+    version = '3.1'
+    oids = [OID.MF, OID.DF_CD, OID.DF_TELECOM_v2, OID.ADF_USIM_by_default_v2, OID.ADF_USIMopt_not_by_default_v2,
+            OID.DF_PHONEBOOK_ADF_USIM, OID.DF_GSM_ACCESS_ADF_USIM, OID.DF_5GS_v2, OID.DF_5GS_v3, OID.DF_SAIP,
+            OID.ADF_ISIM_by_default, OID.ADF_ISIMopt_not_by_default_v2, OID.ADF_CSIM_by_default_v2,
+            OID.ADF_CSIMopt_not_by_default_v2, OID.DF_EAP]
+
+class SaipSpecVersion32(SaipSpecVersion):
+    version = '3.2'
+    # no changes in filesystem teplates to previous 3.1
+    oids = SaipSpecVersion31.oids
+
+class SaipSpecVersion331(SaipSpecVersion):
+    version = '3.3.1'
+    oids = SaipSpecVersion32.oids + [OID.ADF_USIMopt_not_by_default_v3, OID.DF_5GS_v4, OID.DF_SAIP, OID.DF_SNPN, OID.DF_5GProSe, OID.IoT_by_default, OID.IoTopt_not_by_default]

pySim/esim/saip/validation.py

@@ -0,0 +1,166 @@
+"""Implementation of SimAlliance/TCA Interoperable Profile validation."""
+
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from pySim.esim.saip import *
+
+class ProfileError(Exception):
+    """Raised when a ProfileConstraintChecker finds an error in a file [structure]."""
+    pass
+
+class ProfileConstraintChecker:
+    """Base class of a constraint checker for a ProfileElementSequence."""
+    def check(self, pes: ProfileElementSequence):
+        """Execute all the check_* methods of the ProfileConstraintChecker against the given
+        ProfileElementSequence"""
+        for name in dir(self):
+            if name.startswith('check_'):
+                method = getattr(self, name)
+                method(pes)
+
+class CheckBasicStructure(ProfileConstraintChecker):
+    """ProfileConstraintChecker for the basic profile structure constraints."""
+    def _is_after_if_exists(self, pes: ProfileElementSequence, opt:str, after:str):
+        opt_pe = pes.get_pe_for_type(opt)
+        if opt_pe:
+            after_pe = pes.get_pe_for_type(after)
+            if not after_pe:
+                raise ProfileError('PE-%s without PE-%s' % (opt.upper(), after.upper()))
+            # FIXME: check order
+
+    def check_start_and_end(self, pes: ProfileElementSequence):
+        """Check for mandatory header and end ProfileElements at the right position."""
+        if pes.pe_list[0].type != 'header':
+            raise ProfileError('first element is not header')
+        if pes.pe_list[1].type != 'mf':
+            # strictly speaking: permitted, but we don't support MF via GenericFileManagement
+            raise ProfileError('second  element is not mf')
+        if pes.pe_list[-1].type != 'end':
+            raise ProfileError('last element is not end')
+
+    def check_number_of_occurrence(self, pes: ProfileElementSequence):
+        """Check The number of occurrence of various ProfileElements."""
+        # check for invalid number of occurrences
+        if len(pes.get_pes_for_type('header')) != 1:
+            raise ProfileError('multiple ProfileHeader')
+        if len(pes.get_pes_for_type('mf')) != 1:
+            # strictly speaking: 0 permitted, but we don't support MF via GenericFileManagement
+            raise ProfileError('multiple PE-MF')
+        for tn in ['end', 'cd', 'telecom',
+                   'usim', 'isim', 'csim', 'opt-usim','opt-isim','opt-csim',
+                   'df-saip', 'df-5gs']:
+            if len(pes.get_pes_for_type(tn)) > 1:
+                raise ProfileError('multiple PE-%s' % tn.upper())
+
+    def check_optional_ordering(self, pes: ProfileElementSequence):
+        """Check the ordering of optional PEs following the respective mandatory ones."""
+        # ordering and required dependencies
+        self._is_after_if_exists(pes,'opt-usim', 'usim')
+        self._is_after_if_exists(pes,'opt-isim', 'isim')
+        self._is_after_if_exists(pes,'gsm-access', 'usim')
+        self._is_after_if_exists(pes,'phonebook', 'usim')
+        self._is_after_if_exists(pes,'df-5gs', 'usim')
+        self._is_after_if_exists(pes,'df-saip', 'usim')
+        self._is_after_if_exists(pes,'opt-csim', 'csim')
+
+    def check_mandatory_services(self, pes: ProfileElementSequence):
+        """Ensure that the PE for the mandatory services exist."""
+        m_svcs = pes.get_pe_for_type('header').decoded['eUICC-Mandatory-services']
+        if 'usim' in m_svcs and not pes.get_pe_for_type('usim'):
+            raise ProfileError('no PE-USIM for mandatory usim service')
+        if 'isim' in m_svcs and not pes.get_pe_for_type('isim'):
+            raise ProfileError('no PE-ISIM for mandatory isim service')
+        if 'csim' in m_svcs and not pes.get_pe_for_type('csim'):
+            raise ProfileError('no PE-ISIM for mandatory csim service')
+        if 'gba-usim' in m_svcs and not 'usim' in m_svcs:
+            raise ProfileError('gba-usim mandatory, but no usim')
+        if 'gba-isim' in m_svcs and not 'isim' in m_svcs:
+            raise ProfileError('gba-isim mandatory, but no isim')
+        if 'multiple-usim' in m_svcs and not 'usim' in m_svcs:
+            raise ProfileError('multiple-usim mandatory, but no usim')
+        if 'multiple-isim' in m_svcs and not 'isim' in m_svcs:
+            raise ProfileError('multiple-isim mandatory, but no isim')
+        if 'multiple-csim' in m_svcs and not 'csim' in m_svcs:
+            raise ProfileError('multiple-csim mandatory, but no csim')
+        if 'get-identity' in m_svcs and not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('get-identity mandatory, but no usim or isim')
+        if 'profile-a-x25519' in m_svcs and not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('profile-a-x25519 mandatory, but no usim or isim')
+        if 'profile-a-p256' in m_svcs and not ('usim' in m_svcs or 'isim' in m_svcs):
+            raise ProfileError('profile-a-p256 mandatory, but no usim or isim')
+
+    def check_mandatory_services_aka(self, pes: ProfileElementSequence):
+        """Ensure that no unnecessary authentication related services are marked as mandatory but not
+        actually used within the profile"""
+        m_svcs = pes.get_pe_for_type('header').decoded['eUICC-Mandatory-services']
+        # list of tuples (algo_id, key_len_in_octets) for all the akaParameters in the PE Sequence
+        algo_id_klen = [(x.decoded['algoConfiguration'][1]['algorithmID'],
+                         len(x.decoded['algoConfiguration'][1]['key'])) for x in pes.get_pes_for_type('akaParameter')]
+        # just a plain list of algorithm IDs in akaParameters
+        algorithm_ids = [x[0] for x in algo_id_klen]
+        if 'milenage' in m_svcs and not 1 in algorithm_ids:
+            raise ProfileError('milenage mandatory, but no related algorithm_id in akaParameter')
+        if 'tuak128' in m_svcs and not (2, 128/8) in algo_id_klen:
+            raise ProfileError('tuak128 mandatory, but no related algorithm_id in akaParameter')
+        if 'cave' in m_svcs and not pes.get_pe_for_type('cdmaParameter'):
+            raise ProfileError('cave mandatory, but no related cdmaParameter')
+        if 'tuak256' in m_svcs and (2, 256/8) in algo_id_klen:
+            raise ProfileError('tuak256 mandatory, but no related algorithm_id in akaParameter')
+        if 'usim-test-algorithm' in m_svcs and not 3 in algorithm_ids:
+            raise ProfileError('usim-test-algorithm mandatory, but no related algorithm_id in akaParameter')
+
+    def check_identification_unique(self, pes: ProfileElementSequence):
+        """Ensure that each PE has a unique identification value."""
+        id_list = [pe.header['identification'] for pe in pes.pe_list if pe.header]
+        if len(id_list) != len(set(id_list)):
+            raise ProfileError('PE identification values are not unique')
+
+FileChoiceList = List[Tuple]
+
+class FileError(ProfileError):
+    """Raised when a FileConstraintChecker finds an error in a file [structure]."""
+    pass
+
+class FileConstraintChecker:
+    def check(self, l: FileChoiceList):
+        """Execute all the check_* methods of the FileConstraintChecker against the given FileChoiceList"""
+        for name in dir(self):
+            if name.startswith('check_'):
+                method = getattr(self, name)
+                method(l)
+
+class FileCheckBasicStructure(FileConstraintChecker):
+    """Validator for the basic structure of a decoded file."""
+    def check_seqence(self, l: FileChoiceList):
+        """Check the sequence/ordering."""
+        by_type = {}
+        for k, v in l:
+            if k in by_type:
+                by_type[k].append(v)
+            else:
+                by_type[k] = [v]
+        if 'doNotCreate' in by_type:
+            if len(l) != 1:
+                raise FileError("doNotCreate must be the only element")
+        if 'fileDescriptor' in by_type:
+            if len(by_type['fileDescriptor']) != 1:
+                raise FileError("fileDescriptor must be the only element")
+            if l[0][0] != 'fileDescriptor':
+                raise FileError("fileDescriptor must be the first element")
+
+    def check_forbidden(self, l: FileChoiceList):
+        """Perform checks for forbidden parameters as described in Section 8.3.3."""

pySim/esim/x509_cert.py

@@ -0,0 +1,209 @@
+"""Implementation of X.509 certificate handling in GSMA eSIM as per SGP22 v3.0"""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import requests
+from typing import Optional, List
+
+from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.primitives import hashes
+from cryptography import x509
+from cryptography.hazmat.primitives.serialization import load_pem_private_key, Encoding
+from cryptography.hazmat.primitives.asymmetric.utils import decode_dss_signature
+
+from pySim.utils import b2h
+
+def check_signed(signed: x509.Certificate, signer: x509.Certificate) -> bool:
+    """Verify if 'signed' certificate was signed using 'signer'."""
+    # this code only works for ECDSA, but this is all we need for GSMA eSIM
+    pkey = signer.public_key()
+    # this 'signed.signature_algorithm_parameters' below requires cryptography 41.0.0 :(
+    pkey.verify(signed.signature, signed.tbs_certificate_bytes, signed.signature_algorithm_parameters)
+
+def cert_get_subject_key_id(cert: x509.Certificate) -> bytes:
+    """Obtain the subject key identifier of the given cert object (as raw bytes)."""
+    ski_ext = cert.extensions.get_extension_for_class(x509.SubjectKeyIdentifier).value
+    return ski_ext.key_identifier
+
+def cert_get_auth_key_id(cert: x509.Certificate) -> bytes:
+    """Obtain the authority key identifier of the given cert object (as raw bytes)."""
+    aki_ext = cert.extensions.get_extension_for_class(x509.AuthorityKeyIdentifier).value
+    return aki_ext.key_identifier
+
+def cert_policy_has_oid(cert: x509.Certificate, match_oid: x509.ObjectIdentifier) -> bool:
+    """Determine if given certificate has a certificatePolicy extension of matching OID."""
+    for policy_ext in filter(lambda x: isinstance(x.value, x509.CertificatePolicies), cert.extensions):
+        if any(policy.policy_identifier == match_oid for policy in policy_ext.value._policies):
+            return True
+    return False
+
+ID_RSP = "2.23.146.1"
+ID_RSP_CERT_OBJECTS = '.'.join([ID_RSP, '2'])
+ID_RSP_ROLE = '.'.join([ID_RSP_CERT_OBJECTS, '1'])
+
+class oid:
+    id_rspRole_ci = x509.ObjectIdentifier(ID_RSP_ROLE + '.0')
+    id_rspRole_euicc_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.1')
+    id_rspRole_eum_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.2')
+    id_rspRole_dp_tls_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.3')
+    id_rspRole_dp_auth_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.4')
+    id_rspRole_dp_pb_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.5')
+    id_rspRole_ds_tls_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.6')
+    id_rspRole_ds_auth_v2 = x509.ObjectIdentifier(ID_RSP_ROLE + '.7')
+
+class VerifyError(Exception):
+    """An error during certificate verification,"""
+
+class CertificateSet:
+    """A set of certificates consisting of a trusted [self-signed] CA root certificate,
+    and an optional number of intermediate certificates.  Can be used to verify the certificate chain
+    of any given other certificate."""
+    def __init__(self, root_cert: x509.Certificate):
+        check_signed(root_cert, root_cert)
+        # TODO: check other mandatory attributes for CA Cert
+        if not cert_policy_has_oid(root_cert, oid.id_rspRole_ci):
+            raise ValueError("Given root certificate doesn't have rspRole_ci OID")
+        usage_ext = root_cert.extensions.get_extension_for_class(x509.KeyUsage).value
+        if not usage_ext.key_cert_sign:
+            raise ValueError('Given root certificate key usage does not permit signing of certificates')
+        if not usage_ext.crl_sign:
+            raise ValueError('Given root certificate key usage does not permit signing of CRLs')
+        self.root_cert = root_cert
+        self.intermediate_certs = {}
+        self.crl = None
+
+    def load_crl(self, urls: Optional[List[str]] = None):
+        if urls and isinstance(urls, str):
+            urls = [urls]
+        if not urls:
+            # generate list of CRL URLs from root CA certificate
+            crl_ext = self.root_cert.extensions.get_extension_for_class(x509.CRLDistributionPoints).value
+            name_list = [x.full_name for x in crl_ext]
+            merged_list = []
+            for n in name_list:
+                merged_list += n
+            uri_list = filter(lambda x: isinstance(x, x509.UniformResourceIdentifier), merged_list)
+            urls = [x.value for x in uri_list]
+
+        for url in urls:
+            try:
+                crl_bytes = requests.get(url, timeout=10)
+            except requests.exceptions.ConnectionError:
+                continue
+            crl = x509.load_der_x509_crl(crl_bytes)
+            if not crl.is_signature_valid(self.root_cert.public_key()):
+                raise ValueError('Given CRL has incorrect signature and cannot be trusted')
+            # FIXME: various other checks
+            self.crl = crl
+            # FIXME: should we support multiple CRLs? we only support a single CRL right now
+            return
+        # FIXME: report on success/failure
+
+    @property
+    def root_cert_id(self) -> bytes:
+        return cert_get_subject_key_id(self.root_cert)
+
+    def add_intermediate_cert(self, cert: x509.Certificate):
+        """Add a potential intermediate certificate to the CertificateSet."""
+        # TODO: check mandatory attributes for intermediate cert
+        usage_ext = cert.extensions.get_extension_for_class(x509.KeyUsage).value
+        if not usage_ext.key_cert_sign:
+            raise ValueError('Given intermediate certificate key usage does not permit signing of certificates')
+        aki = cert_get_auth_key_id(cert)
+        ski = cert_get_subject_key_id(cert)
+        if aki == ski:
+            raise ValueError('Cannot add self-signed cert as intermediate cert')
+        self.intermediate_certs[ski] = cert
+        # TODO: we could test if this cert verifies against the root, and mark it as pre-verified
+        # so we don't need to verify again and again the chain of intermediate certificates
+
+    def verify_cert_crl(self, cert: x509.Certificate):
+        if not self.crl:
+            # we cannot check if there's no CRL
+            return
+        if self.crl.get_revoked_certificate_by_serial_number(cert.serial_nr):
+            raise VerifyError('Certificate is present in CRL, verification failed')
+
+    def verify_cert_chain(self, cert: x509.Certificate, max_depth: int = 100):
+        """Verify if a given certificate's signature chain can be traced back to the root CA of this
+        CertificateSet."""
+        depth = 1
+        c = cert
+        while True:
+            aki = cert_get_auth_key_id(c)
+            if aki == self.root_cert_id:
+                # last step:
+                check_signed(c, self.root_cert)
+                return
+            parent_cert = self.intermediate_certs.get(aki, None)
+            if not parent_cert:
+                raise VerifyError('Could not find intermediate certificate for AuthKeyId %s' % b2h(aki))
+            check_signed(c, parent_cert)
+            # if we reach here, we passed (no exception raised)
+            c = parent_cert
+            depth += 1
+            if depth > max_depth:
+                raise VerifyError('Maximum depth %u exceeded while verifying certificate chain' % max_depth)
+
+
+def ecdsa_dss_to_tr03111(sig: bytes) -> bytes:
+    """convert from DER format to BSI TR-03111; first get long integers; then convert those to bytes."""
+    r, s = decode_dss_signature(sig)
+    return r.to_bytes(32, 'big') + s.to_bytes(32, 'big')
+
+
+class CertAndPrivkey:
+    """A pair of certificate and private key, as used for ECDSA signing."""
+    def __init__(self, required_policy_oid: Optional[x509.ObjectIdentifier] = None,
+                 cert: Optional[x509.Certificate] = None, priv_key = None):
+        self.required_policy_oid = required_policy_oid
+        self.cert = cert
+        self.priv_key = priv_key
+
+    def cert_from_der_file(self, path: str):
+        with open(path, 'rb') as f:
+            cert = x509.load_der_x509_certificate(f.read())
+        if self.required_policy_oid:
+            # verify it is the right type of certificate (id-rspRole-dp-auth, id-rspRole-dp-auth-v2, etc.)
+            assert cert_policy_has_oid(cert, self.required_policy_oid)
+        self.cert = cert
+
+    def privkey_from_pem_file(self, path: str, password: Optional[str] = None):
+        with open(path, 'rb') as f:
+            self.priv_key = load_pem_private_key(f.read(), password)
+
+    def ecdsa_sign(self, plaintext: bytes) -> bytes:
+        """Sign some input-data using an ECDSA signature compliant with SGP.22,
+        which internally refers to Global Platform 2.2 Annex E, which in turn points
+        to BSI TS-03111 which states "concatenated raw R + S values". """
+        sig = self.priv_key.sign(plaintext, ec.ECDSA(hashes.SHA256()))
+        # convert from DER format to BSI TR-03111; first get long integers; then convert those to bytes
+        return ecdsa_dss_to_tr03111(sig)
+
+    def get_authority_key_identifier(self) -> x509.AuthorityKeyIdentifier:
+        """Return the AuthorityKeyIdentifier X.509 extension of the certificate."""
+        return list(filter(lambda x: isinstance(x.value, x509.AuthorityKeyIdentifier), self.cert.extensions))[0].value
+
+    def get_subject_alt_name(self) -> x509.SubjectAlternativeName:
+        """Return the SubjectAlternativeName X.509 extension of the certificate."""
+        return list(filter(lambda x: isinstance(x.value, x509.SubjectAlternativeName), self.cert.extensions))[0].value
+
+    def get_cert_as_der(self) -> bytes:
+        """Return certificate encoded as DER."""
+        return self.cert.public_bytes(Encoding.DER)
+
+    def get_curve(self) -> ec.EllipticCurve:
+        return self.cert.public_key().public_numbers().curve

pySim/euicc.py

@@ -0,0 +1,634 @@
+# -*- coding: utf-8 -*-
+
+"""
+Various definitions related to GSMA consumer + IoT eSIM / eUICC
+
+Does *not* implement anything related to M2M eUICC
+
+Related Specs: GSMA SGP.21, SGP.22, SGP.31, SGP32
+"""
+
+# Copyright (C) 2023 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+
+from construct import Array, Struct, FlagsEnum, GreedyRange
+from cmd2 import cmd2, CommandSet, with_default_category
+from osmocom.utils import Hexstr
+from osmocom.tlv import *
+from osmocom.construct import *
+
+from pySim.exceptions import SwMatchError
+from pySim.utils import Hexstr, SwHexstr, SwMatchstr
+from pySim.commands import SimCardCommands
+from pySim.ts_102_221 import CardProfileUICC
+import pySim.global_platform
+
+# SGP.02 Section 2.2.2
+class Sgp02Eid(BER_TLV_IE, tag=0x5a):
+    _construct = BcdAdapter(GreedyBytes)
+
+# patch this into global_platform, to allow 'get_data sgp02_eid' in EF.ECASD
+pySim.global_platform.DataCollection.possible_nested.append(Sgp02Eid)
+
+def compute_eid_checksum(eid) -> str:
+    """Compute and add/replace check digits of an EID value according to GSMA SGP.29 Section 10."""
+    if isinstance(eid, str):
+        if len(eid) == 30:
+            # first pad by 2 digits
+            eid += "00"
+        elif len(eid) == 32:
+            # zero the last two digits
+            eid = eid[:-2] + "00"
+        else:
+            raise ValueError("and EID must be 30 or 32 digits")
+        eid_int = int(eid)
+    elif isinstance(eid, int):
+        eid_int = eid
+        if eid_int % 100:
+            # zero the last two digits
+            eid_int -= eid_int % 100
+    # Using the resulting 32 digits as a decimal integer, compute the remainder of that number on division by
+    # 97, Subtract the remainder from 98, and use the decimal result for the two check digits, if the result
+    # is one digit long, its value SHALL be prefixed by one digit of 0.
+    csum = 98 - (eid_int % 97)
+    eid_int += csum
+    return str(eid_int)
+
+def verify_eid_checksum(eid) -> bool:
+    """Verify the check digits of an EID value according to GSMA SGP.29 Section 10."""
+    # Using the 32 digits as a decimal integer, compute the remainder of that number on division by 97. If the
+    # remainder of the division is 1, the verification is successful; otherwise the EID is invalid.
+    return int(eid) % 97 == 1
+
+class VersionAdapter(Adapter):
+    """convert an EUICC Version (3-int array) to a textual representation."""
+
+    def _decode(self, obj, context, path):
+        return "%u.%u.%u" % (obj[0], obj[1], obj[2])
+
+    def _encode(self, obj, context, path):
+        return [int(x) for x in obj.split('.')]
+
+VersionType = VersionAdapter(Array(3, Int8ub))
+
+# Application Identifiers as defined in GSMA SGP.02 Annex H
+AID_ISD_R           = "A0000005591010FFFFFFFF8900000100"
+AID_ECASD           = "A0000005591010FFFFFFFF8900000200"
+AID_ISD_P_FILE      = "A0000005591010FFFFFFFF8900000D00"
+AID_ISD_P_MODULE    = "A0000005591010FFFFFFFF8900000E00"
+
+class SupportedVersionNumber(BER_TLV_IE, tag=0x82):
+    _construct = GreedyBytes
+
+class IsdrProprietaryApplicationTemplate(BER_TLV_IE, tag=0xe0, nested=[SupportedVersionNumber]):
+    # FIXME: lpaeSupport - what kind of tag  would it have?
+    pass
+
+# GlobalPlatform 2.1.1 Section 9.9.3.1 from pySim/global_platform.py extended with E0
+class FciTemplate(BER_TLV_IE, tag=0x6f, nested=pySim.global_platform.FciTemplateNestedList +
+                                               [IsdrProprietaryApplicationTemplate]):
+    pass
+
+
+# SGP.22 Section 5.7.3: GetEuiccConfiguredAddresses
+class DefaultDpAddress(BER_TLV_IE, tag=0x80):
+    _construct = Utf8Adapter(GreedyBytes)
+class RootDsAddress(BER_TLV_IE, tag=0x81):
+    _construct = Utf8Adapter(GreedyBytes)
+class EuiccConfiguredAddresses(BER_TLV_IE, tag=0xbf3c, nested=[DefaultDpAddress, RootDsAddress]):
+    pass
+
+# SGP.22 Section 5.7.4: SetDefaultDpAddress
+class SetDefaultDpAddrRes(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, undefinedError=127)
+class SetDefaultDpAddress(BER_TLV_IE, tag=0xbf3f, nested=[DefaultDpAddress, SetDefaultDpAddrRes]):
+    pass
+
+# SGP.22 Section 5.7.7: GetEUICCChallenge
+class EuiccChallenge(BER_TLV_IE, tag=0x80):
+    _construct = Bytes(16)
+class GetEuiccChallenge(BER_TLV_IE, tag=0xbf2e, nested=[EuiccChallenge]):
+    pass
+
+# SGP.22 Section 5.7.8: GetEUICCInfo
+class SVN(BER_TLV_IE, tag=0x82):
+    _construct = VersionType
+class SubjectKeyIdentifier(BER_TLV_IE, tag=0x04):
+    _construct = GreedyBytes
+class EuiccCiPkiListForVerification(BER_TLV_IE, tag=0xa9, nested=[SubjectKeyIdentifier]):
+    pass
+class EuiccCiPkiListForSigning(BER_TLV_IE, tag=0xaa, nested=[SubjectKeyIdentifier]):
+    pass
+class EuiccInfo1(BER_TLV_IE, tag=0xbf20, nested=[SVN, EuiccCiPkiListForVerification, EuiccCiPkiListForSigning]):
+    pass
+class ProfileVersion(BER_TLV_IE, tag=0x81):
+    _construct = VersionType
+class EuiccFirmwareVer(BER_TLV_IE, tag=0x83):
+    _construct = VersionType
+class ExtCardResource(BER_TLV_IE, tag=0x84):
+    _construct = GreedyBytes
+class UiccCapability(BER_TLV_IE, tag=0x85):
+    _construct = GreedyBytes # FIXME
+class TS102241Version(BER_TLV_IE, tag=0x86):
+    _construct = VersionType
+class GlobalPlatformVersion(BER_TLV_IE, tag=0x87):
+    _construct = VersionType
+class RspCapability(BER_TLV_IE, tag=0x88):
+    _construct = GreedyBytes # FIXME
+class EuiccCategory(BER_TLV_IE, tag=0x8b):
+    _construct = Enum(Int8ub, other=0, basicEuicc=1, mediumEuicc=2, contactlessEuicc=3)
+class PpVersion(BER_TLV_IE, tag=0x04):
+    _construct = VersionType
+class SsAcreditationNumber(BER_TLV_IE, tag=0x0c):
+    _construct = Utf8Adapter(GreedyBytes)
+class IpaMode(BER_TLV_IE, tag=0x90):    # see SGP.32 v1.0
+    _construct = Enum(Int8ub, ipad=0, ipea=1)
+class IotVersion(BER_TLV_IE, tag=0x80): # see SGP.32 v1.0
+    _construct = VersionType
+class IotVersionSeq(BER_TLV_IE, tag=0xa0, nested=[IotVersion]): # see SGP.32 v1.0
+    pass
+class IotSpecificInfo(BER_TLV_IE, tag=0x94, nested=[IotVersionSeq]): # see SGP.32 v1.0
+    pass
+class EuiccInfo2(BER_TLV_IE, tag=0xbf22, nested=[ProfileVersion, SVN, EuiccFirmwareVer, ExtCardResource,
+                                                 UiccCapability, TS102241Version, GlobalPlatformVersion,
+                                                 RspCapability, EuiccCiPkiListForVerification,
+                                                 EuiccCiPkiListForSigning, EuiccCategory, PpVersion,
+                                                 SsAcreditationNumber, IpaMode, IotSpecificInfo]):
+    pass
+
+# SGP.22 Section 5.7.9: ListNotification
+class ProfileMgmtOperation(BER_TLV_IE, tag=0x81):
+    # we have to ignore the first byte which tells us how many padding bits are used in the last octet
+    _construct = Struct(Byte, "pmo"/FlagsEnum(Byte, install=0x80, enable=0x40, disable=0x20, delete=0x10))
+class ListNotificationReq(BER_TLV_IE, tag=0xbf28, nested=[ProfileMgmtOperation]):
+    pass
+class SeqNumber(BER_TLV_IE, tag=0x80):
+    _construct = Asn1DerInteger()
+class NotificationAddress(BER_TLV_IE, tag=0x0c):
+    _construct = Utf8Adapter(GreedyBytes)
+class Iccid(BER_TLV_IE, tag=0x5a):
+    _construct = BcdAdapter(GreedyBytes)
+class NotificationMetadata(BER_TLV_IE, tag=0xbf2f, nested=[SeqNumber, ProfileMgmtOperation,
+                                                           NotificationAddress, Iccid]):
+    pass
+class NotificationMetadataList(BER_TLV_IE, tag=0xa0, nested=[NotificationMetadata]):
+    pass
+class ListNotificationsResultError(BER_TLV_IE, tag=0x81):
+    _construct = Enum(Int8ub, undefinedError=127)
+class ListNotificationResp(BER_TLV_IE, tag=0xbf28, nested=[NotificationMetadataList,
+                                                           ListNotificationsResultError]):
+    pass
+
+# SGP.22 Section 5.7.11: RemoveNotificationFromList
+class DeleteNotificationStatus(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, nothingToDelete=1, undefinedError=127)
+class NotificationSentReq(BER_TLV_IE, tag=0xbf30, nested=[SeqNumber]):
+    pass
+class NotificationSentResp(BER_TLV_IE, tag=0xbf30, nested=[DeleteNotificationStatus]):
+    pass
+
+# SGP.22 Section 5.7.12: LoadCRL: FIXME
+class LoadCRL(BER_TLV_IE, tag=0xbf35, nested=[]): # FIXME
+    pass
+
+# SGP.22 Section 5.7.15: GetProfilesInfo
+class TagList(BER_TLV_IE, tag=0x5c):
+    _construct = GreedyRange(Int8ub) # FIXME: tags could be multi-byte
+class ProfileInfoListReq(BER_TLV_IE, tag=0xbf2d, nested=[TagList]): # FIXME: SearchCriteria
+    pass
+class IsdpAid(BER_TLV_IE, tag=0x4f):
+    _construct = GreedyBytes
+class ProfileState(BER_TLV_IE, tag=0x9f70):
+    _construct = Enum(Int8ub, disabled=0, enabled=1)
+class ProfileNickname(BER_TLV_IE, tag=0x90):
+    _construct = Utf8Adapter(GreedyBytes)
+class ServiceProviderName(BER_TLV_IE, tag=0x91):
+    _construct = Utf8Adapter(GreedyBytes)
+class ProfileName(BER_TLV_IE, tag=0x92):
+    _construct = Utf8Adapter(GreedyBytes)
+class IconType(BER_TLV_IE, tag=0x93):
+    _construct = Enum(Int8ub, jpg=0, png=1)
+class Icon(BER_TLV_IE, tag=0x94):
+    _construct = GreedyBytes
+class ProfileClass(BER_TLV_IE, tag=0x95):
+    _construct = Enum(Int8ub, test=0, provisioning=1, operational=2)
+class ProfileInfo(BER_TLV_IE, tag=0xe3, nested=[Iccid, IsdpAid, ProfileState, ProfileNickname,
+                                                ServiceProviderName, ProfileName, IconType, Icon,
+                                                ProfileClass]): # FIXME: more IEs
+    pass
+class ProfileInfoSeq(BER_TLV_IE, tag=0xa0, nested=[ProfileInfo]):
+    pass
+class ProfileInfoListError(BER_TLV_IE, tag=0x81):
+    _construct = Enum(Int8ub, incorrectInputValues=1, undefinedError=2)
+class ProfileInfoListResp(BER_TLV_IE, tag=0xbf2d, nested=[ProfileInfoSeq, ProfileInfoListError]):
+    pass
+
+# SGP.22 Section 5.7.16:: EnableProfile
+class RefreshFlag(BER_TLV_IE, tag=0x81): # FIXME
+    _construct = Int8ub # FIXME
+class EnableResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInDisabledState=2,
+                      disallowedByPolicy=3, wrongProfileReenabling=4, catBusy=5, undefinedError=127)
+class ProfileIdentifier(BER_TLV_IE, tag=0xa0, nested=[IsdpAid, Iccid]):
+    pass
+class EnableProfileReq(BER_TLV_IE, tag=0xbf31, nested=[ProfileIdentifier, RefreshFlag]):
+    pass
+class EnableProfileResp(BER_TLV_IE, tag=0xbf31, nested=[EnableResult]):
+    pass
+
+# SGP.22 Section 5.7.17 DisableProfile
+class DisableResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInEnabledState=2,
+                      disallowedByPolicy=3, catBusy=5, undefinedError=127)
+class DisableProfileReq(BER_TLV_IE, tag=0xbf32, nested=[ProfileIdentifier, RefreshFlag]):
+    pass
+class DisableProfileResp(BER_TLV_IE, tag=0xbf32, nested=[DisableResult]):
+    pass
+
+# SGP.22 Section 5.7.18: DeleteProfile
+class DeleteResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidOrAidNotFound=1, profileNotInDisabledState=2,
+                      disallowedByPolicy=3, undefinedError=127)
+class DeleteProfileReq(BER_TLV_IE, tag=0xbf33, nested=[IsdpAid, Iccid]):
+    pass
+class DeleteProfileResp(BER_TLV_IE, tag=0xbf33, nested=[DeleteResult]):
+    pass
+
+# SGP.22 Section 5.7.19: EuiccMemoryReset
+class ResetOptions(BER_TLV_IE, tag=0x82):
+    _construct = FlagsEnum(Byte, deleteOperationalProfiles=0x80, deleteFieldLoadedTestProfiles=0x40,
+                           resetDefaultSmdpAddress=0x20)
+class ResetResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, nothingToDelete=1, undefinedError=127)
+class EuiccMemoryResetReq(BER_TLV_IE, tag=0xbf34, nested=[ResetOptions]):
+    pass
+class EuiccMemoryResetResp(BER_TLV_IE, tag=0xbf34, nested=[ResetResult]):
+    pass
+
+# SGP.22 Section 5.7.20 GetEID
+class EidValue(BER_TLV_IE, tag=0x5a):
+    _construct = GreedyBytes
+class GetEuiccData(BER_TLV_IE, tag=0xbf3e, nested=[TagList, EidValue]):
+    pass
+
+# SGP.22 Section 5.7.21: ES10c SetNickname
+class SnrProfileNickname(BER_TLV_IE, tag=0x8f):
+    _construct = Utf8Adapter(GreedyBytes)
+class SetNicknameReq(BER_TLV_IE, tag=0xbf29, nested=[Iccid, SnrProfileNickname]):
+    pass
+class SetNicknameResult(BER_TLV_IE, tag=0x80):
+    _construct = Enum(Int8ub, ok=0, iccidNotFound=1, undefinedError=127)
+class SetNicknameResp(BER_TLV_IE, tag=0xbf29, nested=[SetNicknameResult]):
+    pass
+
+# SGP.32 Section 5.9.10: ES10b: GetCerts
+class GetCertsReq(BER_TLV_IE, tag=0xbf56):
+    pass
+class EumCertificate(BER_TLV_IE, tag=0xa5):
+    _construct = GreedyBytes
+class EuiccCertificate(BER_TLV_IE, tag=0xa6):
+    _construct = GreedyBytes
+class GetCertsError(BER_TLV_IE, tag=0x81):
+    _construct = Enum(Int8ub, invalidCiPKId=1, undefinedError=127)
+class GetCertsResp(BER_TLV_IE, tag=0xbf56, nested=[EumCertificate, EuiccCertificate, GetCertsError]):
+    pass
+
+# SGP.32 Section 5.9.18: ES10b: GetEimConfigurationData
+class EimId(BER_TLV_IE, tag=0x80):
+    _construct = Utf8Adapter(GreedyBytes)
+class EimFqdn(BER_TLV_IE, tag=0x81):
+    _construct = Utf8Adapter(GreedyBytes)
+class EimIdType(BER_TLV_IE, tag=0x82):
+    _construct = Enum(Int8ub, eimIdTypeOid=1, eimIdTypeFqdn=2, eimIdTypeProprietary=3)
+class CounterValue(BER_TLV_IE, tag=0x83):
+    _construct = Asn1DerInteger()
+class AssociationToken(BER_TLV_IE, tag=0x84):
+    _construct = Asn1DerInteger()
+class EimSupportedProtocol(BER_TLV_IE, tag=0x87):
+    _construct = Enum(Int8ub, eimRetrieveHttps=0, eimRetrieveCoaps=1, eimInjectHttps=2, eimInjectCoaps=3,
+                      eimProprietary=4)
+# FIXME: eimPublicKeyData, trustedPublicKeyDataTls, euiccCiPKId
+class EimConfigurationData(BER_TLV_IE, tag=0x80, nested=[EimId, EimFqdn, EimIdType, CounterValue,
+                                                         AssociationToken, EimSupportedProtocol]):
+    pass
+class EimConfigurationDataSeq(BER_TLV_IE, tag=0xa0, nested=[EimConfigurationData]):
+    pass
+class GetEimConfigurationData(BER_TLV_IE, tag=0xbf55, nested=[EimConfigurationDataSeq]):
+    pass
+
+class CardApplicationISDR(pySim.global_platform.CardApplicationSD):
+    def __init__(self):
+        super().__init__(name='ADF.ISD-R', aid=AID_ISD_R,
+                         desc='ISD-R (Issuer Security Domain Root) Application')
+        self.adf.decode_select_response = self.decode_select_response
+        self.adf.shell_commands += [self.AddlShellCommands()]
+        # we attempt to retrieve ISD-R key material from CardKeyProvider identified by EID
+        self.adf.scp_key_identity = 'EID'
+
+    @staticmethod
+    def store_data(scc: SimCardCommands, tx_do: Hexstr, exp_sw: SwMatchstr ="9000") -> Tuple[Hexstr, SwHexstr]:
+        """Perform STORE DATA according to Table 47+48 in Section 5.7.2 of SGP.22.
+        Only single-block store supported for now."""
+        capdu = '80E29100%02x%s00' % (len(tx_do)//2, tx_do)
+        return scc.send_apdu_checksw(capdu, exp_sw)
+
+    @staticmethod
+    def store_data_tlv(scc: SimCardCommands, cmd_do, resp_cls, exp_sw: SwMatchstr = '9000'):
+        """Transceive STORE DATA APDU with the card, transparently encoding the command data from TLV
+        and decoding the response data tlv."""
+        if cmd_do:
+            cmd_do_enc = cmd_do.to_tlv()
+            cmd_do_len = len(cmd_do_enc)
+            if cmd_do_len > 255:
+                return ValueError('DO > 255 bytes not supported yet')
+        else:
+            cmd_do_enc = b''
+        (data, _sw) = CardApplicationISDR.store_data(scc, b2h(cmd_do_enc), exp_sw=exp_sw)
+        if data:
+            if resp_cls:
+                resp_do = resp_cls()
+                resp_do.from_tlv(h2b(data))
+                return resp_do
+            else:
+                return data
+        else:
+            return None
+
+    @staticmethod
+    def get_eid(scc: SimCardCommands) -> str:
+        ged_cmd = GetEuiccData(children=[TagList(decoded=[0x5A])])
+        ged = CardApplicationISDR.store_data_tlv(scc, ged_cmd, GetEuiccData)
+        d = ged.to_dict()
+        return b2h(flatten_dict_lists(d['get_euicc_data'])['eid_value'])
+
+    def decode_select_response(self, data_hex: Hexstr) -> object:
+        t = FciTemplate()
+        t.from_tlv(h2b(data_hex))
+        d = t.to_dict()
+        return flatten_dict_lists(d['fci_template'])
+
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+
+        es10x_store_data_parser = argparse.ArgumentParser()
+        es10x_store_data_parser.add_argument('TX_DO', help='Hexstring of encoded to-be-transmitted DO')
+
+        @cmd2.with_argparser(es10x_store_data_parser)
+        def do_es10x_store_data(self, opts):
+            """Perform a raw STORE DATA command as defined for the ES10x eUICC interface."""
+            (_data, _sw) = CardApplicationISDR.store_data(self._cmd.lchan.scc, opts.TX_DO)
+
+        def do_get_euicc_configured_addresses(self, _opts):
+            """Perform an ES10a GetEuiccConfiguredAddresses function."""
+            eca = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, EuiccConfiguredAddresses(), EuiccConfiguredAddresses)
+            d = eca.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_configured_addresses']))
+
+        set_def_dp_addr_parser = argparse.ArgumentParser()
+        set_def_dp_addr_parser.add_argument('DP_ADDRESS', help='Default SM-DP+ address as UTF-8 string')
+
+        @cmd2.with_argparser(set_def_dp_addr_parser)
+        def do_set_default_dp_address(self, opts):
+            """Perform an ES10a SetDefaultDpAddress function."""
+            sdda_cmd = SetDefaultDpAddress(children=[DefaultDpAddress(decoded=opts.DP_ADDRESS)])
+            sdda = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, sdda_cmd, SetDefaultDpAddress)
+            d = sdda.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['set_default_dp_address']))
+
+        def do_get_euicc_challenge(self, _opts):
+            """Perform an ES10b GetEUICCChallenge function."""
+            gec = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, GetEuiccChallenge(), GetEuiccChallenge)
+            d = gec.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_euicc_challenge']))
+
+        def do_get_euicc_info1(self, _opts):
+            """Perform an ES10b GetEUICCInfo (1) function."""
+            ei1 = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, EuiccInfo1(), EuiccInfo1)
+            d = ei1.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_info1']))
+
+        def do_get_euicc_info2(self, _opts):
+            """Perform an ES10b GetEUICCInfo (2) function."""
+            ei2 = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, EuiccInfo2(), EuiccInfo2)
+            d = ei2.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_info2']))
+
+        def do_list_notification(self, _opts):
+            """Perform an ES10b ListNotification function."""
+            ln = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, ListNotificationReq(), ListNotificationResp)
+            d = ln.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['list_notification_resp']))
+
+        rem_notif_parser = argparse.ArgumentParser()
+        rem_notif_parser.add_argument('SEQ_NR', type=int, help='Sequence Number of the to-be-removed notification')
+
+        @cmd2.with_argparser(rem_notif_parser)
+        def do_remove_notification_from_list(self, opts):
+            """Perform an ES10b RemoveNotificationFromList function."""
+            rn_cmd = NotificationSentReq(children=[SeqNumber(decoded=opts.SEQ_NR)])
+            rn = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, rn_cmd, NotificationSentResp)
+            d = rn.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['notification_sent_resp']))
+
+        def do_get_profiles_info(self, _opts):
+            """Perform an ES10c GetProfilesInfo function."""
+            pi = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, ProfileInfoListReq(), ProfileInfoListResp)
+            d = pi.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['profile_info_list_resp']))
+
+        en_prof_parser = argparse.ArgumentParser()
+        en_prof_grp = en_prof_parser.add_mutually_exclusive_group()
+        en_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        en_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+        en_prof_parser.add_argument('--refresh-required', action='store_true', help='whether a REFRESH is required')
+
+        @cmd2.with_argparser(en_prof_parser)
+        def do_enable_profile(self, opts):
+            """Perform an ES10c EnableProfile function."""
+            if opts.isdp_aid:
+                p_id = ProfileIdentifier(children=[IsdpAid(decoded=opts.isdp_aid)])
+            elif opts.iccid:
+                p_id = ProfileIdentifier(children=[Iccid(decoded=opts.iccid)])
+            else:
+                # this is guaranteed by argparse; but we need this to make pylint happy
+                raise ValueError('Either ISD-P AID or ICCID must be given')
+            ep_cmd_contents = [p_id, RefreshFlag(decoded=opts.refresh_required)]
+            ep_cmd = EnableProfileReq(children=ep_cmd_contents)
+            ep = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, ep_cmd, EnableProfileResp)
+            d = ep.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['enable_profile_resp']))
+
+        dis_prof_parser = argparse.ArgumentParser()
+        dis_prof_grp = dis_prof_parser.add_mutually_exclusive_group()
+        dis_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        dis_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+        dis_prof_parser.add_argument('--refresh-required', action='store_true', help='whether a REFRESH is required')
+
+        @cmd2.with_argparser(dis_prof_parser)
+        def do_disable_profile(self, opts):
+            """Perform an ES10c DisableProfile function."""
+            if opts.isdp_aid:
+                p_id = ProfileIdentifier(children=[IsdpAid(decoded=opts.isdp_aid)])
+            elif opts.iccid:
+                p_id = ProfileIdentifier(children=[Iccid(decoded=opts.iccid)])
+            else:
+                # this is guaranteed by argparse; but we need this to make pylint happy
+                raise ValueError('Either ISD-P AID or ICCID must be given')
+            dp_cmd_contents = [p_id, RefreshFlag(decoded=opts.refresh_required)]
+            dp_cmd = DisableProfileReq(children=dp_cmd_contents)
+            dp = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, dp_cmd, DisableProfileResp)
+            d = dp.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['disable_profile_resp']))
+
+        del_prof_parser = argparse.ArgumentParser()
+        del_prof_grp = del_prof_parser.add_mutually_exclusive_group()
+        del_prof_grp.add_argument('--isdp-aid', help='Profile identified by its ISD-P AID')
+        del_prof_grp.add_argument('--iccid', help='Profile identified by its ICCID')
+
+        @cmd2.with_argparser(del_prof_parser)
+        def do_delete_profile(self, opts):
+            """Perform an ES10c DeleteProfile function."""
+            if opts.isdp_aid:
+                p_id = IsdpAid(decoded=opts.isdp_aid)
+            elif opts.iccid:
+                p_id = Iccid(decoded=opts.iccid)
+            else:
+                # this is guaranteed by argparse; but we need this to make pylint happy
+                raise ValueError('Either ISD-P AID or ICCID must be given')
+            dp_cmd_contents = [p_id]
+            dp_cmd = DeleteProfileReq(children=dp_cmd_contents)
+            dp = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, dp_cmd, DeleteProfileResp)
+            d = dp.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['delete_profile_resp']))
+
+        mem_res_parser = argparse.ArgumentParser()
+        mem_res_parser.add_argument('--delete-operational', action='store_true',
+                                    help='Delete all operational profiles')
+        mem_res_parser.add_argument('--delete-test-field-installed', action='store_true',
+                                    help='Delete all test profiles, except pre-installed ones')
+        mem_res_parser.add_argument('--reset-smdp-address', action='store_true',
+                                    help='Reset the SM-DP+ address')
+
+        @cmd2.with_argparser(mem_res_parser)
+        def do_euicc_memory_reset(self, opts):
+            """Perform an ES10c eUICCMemoryReset function. This will permanently delete the selected subset of
+            profiles from the eUICC."""
+            flags = {}
+            if opts.delete_operational:
+                flags['deleteOperationalProfiles'] = True
+            if opts.delete_test_field_installed:
+                flags['deleteFieldLoadedTestProfiles'] = True
+            if opts.reset_smdp_address:
+                flags['resetDefaultSmdpAddress'] = True
+
+            mr_cmd = EuiccMemoryResetReq(children=[ResetOptions(decoded=flags)])
+            mr = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, mr_cmd, EuiccMemoryResetResp)
+            d = mr.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['euicc_memory_reset_resp']))
+
+        def do_get_eid(self, _opts):
+            """Perform an ES10c GetEID function."""
+            ged_cmd = GetEuiccData(children=[TagList(decoded=[0x5A])])
+            ged = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, ged_cmd, GetEuiccData)
+            d = ged.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_euicc_data']))
+
+        set_nickname_parser = argparse.ArgumentParser()
+        set_nickname_parser.add_argument('--profile-nickname', help='Nickname of the profile')
+        set_nickname_parser.add_argument('ICCID', help='ICCID of the profile whose nickname to set')
+
+        @cmd2.with_argparser(set_nickname_parser)
+        def do_set_nickname(self, opts):
+            """Perform an ES10c SetNickname function."""
+            nickname = opts.profile_nickname or ''
+            sn_cmd_contents = [Iccid(decoded=opts.ICCID), ProfileNickname(decoded=nickname)]
+            sn_cmd = SetNicknameReq(children=sn_cmd_contents)
+            sn = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, sn_cmd, SetNicknameResp)
+            d = sn.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['set_nickname_resp']))
+
+        def do_get_certs(self, _opts):
+            """Perform an ES10c GetCerts() function on an IoT eUICC."""
+            gc = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, GetCertsReq(), GetCertsResp)
+            d = gc.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_certs_resp']))
+
+        def do_get_eim_configuration_data(self, _opts):
+            """Perform an ES10b GetEimConfigurationData function on an Iot eUICC."""
+            gec = CardApplicationISDR.store_data_tlv(self._cmd.lchan.scc, GetEimConfigurationData(),
+                                                     GetEimConfigurationData)
+            d = gec.to_dict()
+            self._cmd.poutput_json(flatten_dict_lists(d['get_eim_configuration_data']))
+
+class CardApplicationECASD(pySim.global_platform.CardApplicationSD):
+    def decode_select_response(self, data_hex: Hexstr) -> object:
+        t = FciTemplate()
+        t.from_tlv(h2b(data_hex))
+        d = t.to_dict()
+        return flatten_dict_lists(d['fci_template'])
+
+    def __init__(self):
+        super().__init__(name='ADF.ECASD', aid=AID_ECASD,
+                         desc='ECASD (eUICC Controlling Authority Security Domain) Application')
+        self.adf.decode_select_response = self.decode_select_response
+        self.adf.shell_commands += [self.AddlShellCommands()]
+        # we attempt to retrieve ECASD key material from CardKeyProvider identified by EID
+        self.adf.scp_key_identity = 'EID'
+
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        pass
+
+class CardProfileEuiccSGP32(CardProfileUICC):
+    ORDER = 5
+
+    def __init__(self):
+        super().__init__(name='IoT eUICC (SGP.32)')
+
+    @classmethod
+    def _try_match_card(cls, scc: SimCardCommands) -> None:
+        # try a command only supported by SGP.32
+        scc.cla_byte = "00"
+        scc.select_adf(AID_ISD_R)
+        CardApplicationISDR.store_data_tlv(scc, GetCertsReq(), GetCertsResp)
+
+class CardProfileEuiccSGP22(CardProfileUICC):
+    ORDER = 6
+
+    def __init__(self):
+        super().__init__(name='Consumer eUICC (SGP.22)')
+
+    @classmethod
+    def _try_match_card(cls, scc: SimCardCommands) -> None:
+        # try to read EID from ISD-R
+        scc.cla_byte = "00"
+        scc.select_adf(AID_ISD_R)
+        eid = CardApplicationISDR.get_eid(scc)
+        # TODO: Store EID identity?
+
+class CardProfileEuiccSGP02(CardProfileUICC):
+    ORDER = 7
+
+    def __init__(self):
+        super().__init__(name='M2M eUICC (SGP.02)')
+
+    @classmethod
+    def _try_match_card(cls, scc: SimCardCommands) -> None:
+        scc.cla_byte = "00"
+        scc.select_adf(AID_ECASD)
+        scc.get_data(0x5a)
+        # TODO: Store EID identity?

pySim/exceptions.py

@@ -24,17 +24,14 @@
 
 class NoCardError(Exception):
     """No card was found in the reader."""
-    pass
 
 
 class ProtocolError(Exception):
     """Some kind of protocol level error interfacing with the card."""
-    pass
 
 
 class ReaderError(Exception):
     """Some kind of general error with the card reader."""
-    pass
 
 
 class SwMatchError(Exception):
@@ -52,9 +49,18 @@ class SwMatchError(Exception):
         self.sw_expected = sw_expected
         self.rs = rs
 
-    def __str__(self):
-        if self.rs:
-            r = self.rs.interpret_sw(self.sw_actual)
+    @property
+    def description(self):
+        if self.rs and self.rs.lchan[0]:
+            r = self.rs.lchan[0].interpret_sw(self.sw_actual)
             if r:
-                return "SW match failed! Expected %s and got %s: %s - %s" % (self.sw_expected, self.sw_actual, r[0], r[1])
-        return "SW match failed! Expected %s and got %s." % (self.sw_expected, self.sw_actual)
+                return "%s - %s" % (r[0], r[1])
+        return ''
+
+    def __str__(self):
+        description = self.description
+        if description:
+            description = ": " + description
+        else:
+            description = "."
+        return "SW match failed! Expected %s and got %s%s" % (self.sw_expected, self.sw_actual, description)

pySim/global_platform.py

@@ -1,256 +0,0 @@
-# coding=utf-8
-"""Partial Support for GlobalPLatform Card Spec (currently 2.1.1)
-
-(C) 2022 by Harald Welte <laforge@osmocom.org>
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-"""
-
-from typing import Optional, List, Dict, Tuple
-from construct import Optional as COptional
-from construct import *
-from bidict import bidict
-from pySim.construct import *
-from pySim.utils import *
-from pySim.filesystem import *
-from pySim.tlv import *
-from pySim.profile import CardProfile
-
-sw_table = {
-    'Warnings': {
-        '6200': 'Logical Channel already closed',
-        '6283': 'Card Life Cycle State is CARD_LOCKED',
-        '6310': 'More data available',
-    },
-    'Execution errors': {
-        '6400': 'No specific diagnosis',
-        '6581': 'Memory failure',
-    },
-    'Checking errors': {
-        '6700': 'Wrong length in Lc',
-    },
-    'Functions in CLA not supported': {
-        '6881': 'Logical channel not supported or active',
-        '6882': 'Secure messaging not supported',
-    },
-    'Command not allowed': {
-        '6982': 'Security Status not satisfied',
-        '6985': 'Conditions of use not satisfied',
-    },
-    'Wrong parameters': {
-        '6a80': 'Incorrect values in command data',
-        '6a81': 'Function not supported e.g. card Life Cycle State is CARD_LOCKED',
-        '6a82': 'Application not found',
-        '6a84': 'Not enough memory space',
-        '6a86': 'Incorrect P1 P2',
-        '6a88': 'Referenced data not found',
-    },
-    'GlobalPlatform': {
-        '6d00': 'Invalid instruction',
-        '6e00': 'Invalid class',
-    },
-    'Application errors': {
-        '9484': 'Algorithm not supported',
-        '9485': 'Invalid key check value',
-    },
-}
-
-# GlobalPlatform 2.1.1 Section 9.1.6
-KeyType = Enum(Byte,    des=0x80,
-                        rsa_public_exponent_e_cleartex=0xA0,
-                        rsa_modulus_n_cleartext=0xA1,
-                        rsa_modulus_n=0xA2,
-                        rsa_private_exponent_d=0xA3,
-                        rsa_chines_remainder_p=0xA4,
-                        rsa_chines_remainder_q=0xA5,
-                        rsa_chines_remainder_pq=0xA6,
-                        rsa_chines_remainder_dpi=0xA7,
-                        rsa_chines_remainder_dqi=0xA8,
-                        not_available=0xff)
-
-# GlobalPlatform 2.1.1 Section 9.3.3.1
-# example:
-#   e0 48
-#       c0 04 01708010
-#       c0 04 02708010
-#       c0 04 03708010
-#       c0 04 01018010
-#       c0 04 02018010
-#       c0 04 03018010
-#       c0 04 01028010
-#       c0 04 02028010
-#       c0 04 03028010
-#       c0 04 01038010
-#       c0 04 02038010
-#       c0 04 03038010
-class KeyInformationData(BER_TLV_IE, tag=0xc0):
-    _construct = Struct('key_identifier'/Byte, 'key_version_number'/Byte,
-                        'key_types'/GreedyRange(KeyType))
-class KeyInformation(BER_TLV_IE, tag=0xe0, nested=[KeyInformationData]):
-    pass
-
-# card data sample, returned in response to GET DATA (80ca006600):
-# 66 31
-#    73 2f
-#        06 07
-#            2a864886fc6b01
-#        60 0c
-#            06 0a
-#                2a864886fc6b02020101
-#        63 09
-#            06 07
-#                2a864886fc6b03
-#        64 0b
-#            06 09
-#                2a864886fc6b040215
-
-# GlobalPlatform 2.1.1 Table F-1
-class ObjectIdentifier(BER_TLV_IE, tag=0x06):
-    _construct = GreedyBytes
-class CardManagementTypeAndVersion(BER_TLV_IE, tag=0x60, nested=[ObjectIdentifier]):
-    pass
-class CardIdentificationScheme(BER_TLV_IE, tag=0x63, nested=[ObjectIdentifier]):
-    pass
-class SecureChannelProtocolOfISD(BER_TLV_IE, tag=0x64, nested=[ObjectIdentifier]):
-    pass
-class CardConfigurationDetails(BER_TLV_IE, tag=0x65):
-    _construct = GreedyBytes
-class CardChipDetails(BER_TLV_IE, tag=0x66):
-    _construct = GreedyBytes
-class CardRecognitionData(BER_TLV_IE, tag=0x73, nested=[ObjectIdentifier,
-                                                        CardManagementTypeAndVersion,
-                                                        CardIdentificationScheme,
-                                                        SecureChannelProtocolOfISD,
-                                                        CardConfigurationDetails,
-                                                        CardChipDetails]):
-    pass
-class CardData(BER_TLV_IE, tag=0x66, nested=[CardRecognitionData]):
-    pass
-
-# GlobalPlatform 2.1.1 Table F-2
-class SecureChannelProtocolOfSelectedSD(BER_TLV_IE, tag=0x64, nested=[ObjectIdentifier]):
-    pass
-class SecurityDomainMgmtData(BER_TLV_IE, tag=0x73, nested=[CardManagementTypeAndVersion,
-                                                           CardIdentificationScheme,
-                                                           SecureChannelProtocolOfSelectedSD,
-                                                           CardConfigurationDetails,
-                                                           CardChipDetails]):
-    pass
-
-# GlobalPlatform 2.1.1 Section 9.1.1
-IsdLifeCycleState = Enum(Byte, op_ready=0x01, initialized=0x07, secured=0x0f,
-                         card_locked = 0x7f, terminated=0xff)
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class ApplicationID(BER_TLV_IE, tag=0x84):
-    _construct = GreedyBytes
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class SecurityDomainManagementData(BER_TLV_IE, tag=0x73):
-    _construct = GreedyBytes
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class ApplicationProductionLifeCycleData(BER_TLV_IE, tag=0x9f6e):
-    _construct = GreedyBytes
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class MaximumLengthOfDataFieldInCommandMessage(BER_TLV_IE, tag=0x9f65):
-    _construct = GreedyInteger()
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class ProprietaryData(BER_TLV_IE, tag=0xA5, nested=[SecurityDomainManagementData,
-                                                    ApplicationProductionLifeCycleData,
-                                                    MaximumLengthOfDataFieldInCommandMessage]):
-    pass
-
-# GlobalPlatform 2.1.1 Section 9.9.3.1
-class FciTemplate(BER_TLV_IE, tag=0x6f, nested=[ApplicationID, SecurityDomainManagementData,
-                                                ApplicationProductionLifeCycleData,
-                                                MaximumLengthOfDataFieldInCommandMessage,
-                                                ProprietaryData]):
-    pass
-
-class IssuerIdentificationNumber(BER_TLV_IE, tag=0x42):
-    _construct = BcdAdapter(GreedyBytes)
-
-class CardImageNumber(BER_TLV_IE, tag=0x45):
-    _construct = BcdAdapter(GreedyBytes)
-
-class SequenceCounterOfDefaultKvn(BER_TLV_IE, tag=0xc1):
-    _construct = GreedyInteger()
-
-class ConfirmationCounter(BER_TLV_IE, tag=0xc2):
-    _construct = GreedyInteger()
-
-# Collection of all the data objects we can get from GET DATA
-class DataCollection(TLV_IE_Collection, nested=[IssuerIdentificationNumber,
-                                                CardImageNumber,
-                                                CardData,
-                                                KeyInformation,
-                                                SequenceCounterOfDefaultKvn,
-                                                ConfirmationCounter]):
-    pass
-
-def decode_select_response(resp_hex: str) -> object:
-    t = FciTemplate()
-    t.from_tlv(h2b(resp_hex))
-    d = t.to_dict()
-    return flatten_dict_lists(d['fci_template'])
-
-# Application Dedicated File of a Security Domain
-class ADF_SD(CardADF):
-    def __init__(self, aid: str, name: str, desc: str):
-        super().__init__(aid=aid, fid=None, sfid=None, name=name, desc=desc)
-        self.shell_commands += [self.AddlShellCommands()]
-
-    @staticmethod
-    def decode_select_response(res_hex: str) -> object:
-        return decode_select_response(res_hex)
-
-    @with_default_category('Application-Specific Commands')
-    class AddlShellCommands(CommandSet):
-        def __init__(self):
-            super().__init__()
-
-        def do_get_data(self, opts):
-            tlv_cls_name = opts.arg_list[0]
-            tlv_cls = DataCollection().members_by_name[tlv_cls_name]
-            (data, sw) = self._cmd.card._scc.get_data(cla=0x80, tag=tlv_cls.tag)
-            ie = tlv_cls()
-            ie.from_tlv(h2b(data))
-            self._cmd.poutput_json(ie.to_dict())
-
-        def complete_get_data(self, text, line, begidx, endidx) -> List[str]:
-            #data_dict = {camel_to_snake(str(x.__name__)): x for x in DataCollection.possible_nested}
-            data_dict = {str(x.__name__): x for x in DataCollection.possible_nested}
-            index_dict = {1: data_dict}
-            return self._cmd.index_based_complete(text, line, begidx, endidx, index_dict=index_dict)
-
-# Card Application of a Security Domain
-class CardApplicationSD(CardApplication):
-    def __init__(self, aid: str, name: str, desc: str):
-        super().__init__(name, adf=ADF_SD(aid, name, desc), sw=sw_table)
-
-# Card Application of Issuer Security Domain
-class CardApplicationISD(CardApplicationSD):
-    # FIXME: ISD AID is not static, but could be different. One can select the empty
-    # application using '00a4040000' and then parse the response FCI to get the ISD AID
-    def __init__(self, aid='a000000003000000'):
-        super().__init__(aid=aid, name='ADF.ISD', desc='Issuer Security Domain')
-
-#class CardProfileGlobalPlatform(CardProfile):
-#    ORDER = 23
-#
-#    def __init__(self, name='GlobalPlatform'):
-#        super().__init__(name, desc='GlobalPlatfomr 2.1.1', cla=['00','80','84'], sw=sw_table)

pySim/global_platform/http.py

@@ -0,0 +1,94 @@
+"""GlobalPlatform Remote Application Management over HTTP Card Specification v2.3 - Amendment B.
+Also known as SCP81 for SIM/USIM/UICC/eUICC/eSIM OTA.
+"""
+
+# (C) 2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from construct import Struct, Int8ub, Int16ub, GreedyString, BytesInteger
+from construct import this, len_, Rebuild, Const
+from construct import Optional as COptional
+from osmocom.construct import Bytes, GreedyBytes
+from osmocom.tlv import BER_TLV_IE
+
+from pySim import cat
+
+
+# Table 3-3 + Section 3.8.1
+class RasConnectionParams(BER_TLV_IE, tag=0x84, nested=cat.OpenChannel.nested_collection_cls.possible_nested):
+    pass
+
+# Table 3-3 + Section 3.8.2
+class SecurityParams(BER_TLV_IE, tag=0x85):
+    _test_de_encode = [
+        ( '850804deadbeef020040', {'kid': 64,'kvn': 0, 'psk_id': b'\xde\xad\xbe\xef', 'sha_type': None} )
+    ]
+    _construct = Struct('_psk_id_len'/Rebuild(Int8ub, len_(this.psk_id)), 'psk_id'/Bytes(this._psk_id_len),
+                        '_kid_kvn_len'/Const(2, Int8ub), 'kvn'/Int8ub, 'kid'/Int8ub,
+                        'sha_type'/COptional(Int8ub))
+
+# Table 3-3 + ?
+class ExtendedSecurityParams(BER_TLV_IE, tag=0xA5):
+    _construct = GreedyBytes
+
+# Table 3-3 + Section 3.8.3
+class SessionRetryPolicyParams(BER_TLV_IE, tag=0x86):
+    _construct = Struct('retry_counter'/Int16ub,
+                        'retry_waiting_delay'/BytesInteger(5),
+                        'retry_report_failure'/COptional(GreedyBytes))
+
+# Table 3-3 + Section 3.8.4
+class AdminHostParam(BER_TLV_IE, tag=0x8A):
+    _test_de_encode = [
+        ( '8a0a61646d696e2e686f7374', 'admin.host' ),
+    ]
+    _construct = GreedyString('utf-8')
+
+# Table 3-3 + Section 3.8.5
+class AgentIdParam(BER_TLV_IE, tag=0x8B):
+    _construct = GreedyString('utf-8')
+
+# Table 3-3 + Section 3.8.6
+class AdminUriParam(BER_TLV_IE, tag=0x8C):
+    _test_de_encode = [
+        ( '8c1668747470733a2f2f61646d696e2e686f73742f757269', 'https://admin.host/uri' ),
+    ]
+    _construct = GreedyString('utf-8')
+
+# Table 3-3
+class HttpPostParams(BER_TLV_IE, tag=0x89, nested=[AdminHostParam, AgentIdParam, AdminUriParam]):
+    pass
+
+# Table 3-3
+class AdmSessionParams(BER_TLV_IE, tag=0x83, nested=[RasConnectionParams, SecurityParams,
+                                                     ExtendedSecurityParams, SessionRetryPolicyParams,
+                                                     HttpPostParams]):
+    pass
+
+# Table 3-3 + Section 3.11.4
+class RasFqdn(BER_TLV_IE, tag=0xD6):
+    _construct = GreedyBytes # FIXME: DNS String
+
+# Table 3-3 + Section 3.11.7
+class DnsConnectionParams(BER_TLV_IE, tag=0xFA, nested=cat.OpenChannel.nested_collection_cls.possible_nested):
+    pass
+
+# Table 3-3
+class DnsResolutionParams(BER_TLV_IE, tag=0xB3, nested=[RasFqdn, DnsConnectionParams]):
+    pass
+
+# Table 3-3
+class AdmSessTriggerParams(BER_TLV_IE, tag=0x81, nested=[AdmSessionParams, DnsResolutionParams]):
+    pass

pySim/global_platform/install_param.py

@@ -0,0 +1,72 @@
+# GlobalPlatform install parameter generator
+#
+# (C) 2024 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from osmocom.construct import *
+from osmocom.utils import *
+from osmocom.tlv import *
+
+class AppSpecificParams(BER_TLV_IE, tag=0xC9):
+    # GPD_SPE_013, table 11-49
+    _construct = GreedyBytes
+
+class VolatileMemoryQuota(BER_TLV_IE, tag=0xC7):
+    # GPD_SPE_013, table 11-49
+    _construct = StripHeaderAdapter(GreedyBytes, 4, steps = [2,4])
+
+class NonVolatileMemoryQuota(BER_TLV_IE, tag=0xC8):
+    # GPD_SPE_013, table 11-49
+    _construct = StripHeaderAdapter(GreedyBytes, 4, steps = [2,4])
+
+class StkParameter(BER_TLV_IE, tag=0xCA):
+    # GPD_SPE_013, table 11-49
+    # ETSI TS 102 226, section 8.2.1.3.2.1
+    _construct = GreedyBytes
+
+class SystemSpecificParams(BER_TLV_IE, tag=0xEF, nested=[VolatileMemoryQuota, NonVolatileMemoryQuota, StkParameter]):
+    # GPD_SPE_013 v1.1 Table 6-5
+    pass
+
+class InstallParams(TLV_IE_Collection, nested=[AppSpecificParams, SystemSpecificParams]):
+    # GPD_SPE_013, table 11-49
+    pass
+
+def gen_install_parameters(non_volatile_memory_quota:int, volatile_memory_quota:int, stk_parameter:str):
+
+    # GPD_SPE_013, table 11-49
+
+    #Mandatory
+    install_params = InstallParams()
+    install_params_dict = [{'app_specific_params': None}]
+
+    #Conditional
+    if non_volatile_memory_quota and volatile_memory_quota and stk_parameter:
+        system_specific_params = []
+        #Optional
+        if non_volatile_memory_quota:
+            system_specific_params += [{'non_volatile_memory_quota': non_volatile_memory_quota}]
+        #Optional
+        if volatile_memory_quota:
+            system_specific_params += [{'volatile_memory_quota': volatile_memory_quota}]
+        #Optional
+        if stk_parameter:
+            system_specific_params += [{'stk_parameter': stk_parameter}]
+        install_params_dict += [{'system_specific_params': system_specific_params}]
+
+    install_params.from_dict(install_params_dict)
+    return b2h(install_params.to_bytes())

pySim/global_platform/scp.py

@@ -0,0 +1,606 @@
+# Global Platform SCP02 + SCP03 (Secure Channel Protocol) implementation
+#
+# (C) 2023-2024 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+import logging
+from typing import Optional
+from Cryptodome.Cipher import DES3, DES
+from Cryptodome.Util.strxor import strxor
+from construct import Struct, Int8ub, Int16ub, Const
+from construct import Optional as COptional
+from osmocom.construct import Bytes
+from osmocom.utils import b2h
+from osmocom.tlv import bertlv_parse_len, bertlv_encode_len
+from pySim.utils import parse_command_apdu
+from pySim.secure_channel import SecureChannel
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)
+
+def scp02_key_derivation(constant: bytes, counter: int, base_key: bytes) -> bytes:
+    assert len(constant) == 2
+    assert(counter >= 0 and counter <= 65535)
+    assert len(base_key) == 16
+
+    derivation_data = constant + counter.to_bytes(2, 'big') + b'\x00' * 12
+    cipher = DES3.new(base_key, DES.MODE_CBC, b'\x00' * 8)
+    return cipher.encrypt(derivation_data)
+
+# TODO: resolve duplication with BspAlgoCryptAES128
+def pad80(s: bytes, BS=8) -> bytes:
+    """ Pad bytestring s: add '\x80' and '\0'* so the result to be multiple of BS."""
+    l = BS-1 - len(s) % BS
+    return s + b'\x80' + b'\0'*l
+
+# TODO: resolve duplication with BspAlgoCryptAES128
+def unpad80(padded: bytes) -> bytes:
+    """Remove the customary 80 00 00 ... padding used for AES."""
+    # first remove any trailing zero bytes
+    stripped = padded.rstrip(b'\0')
+    # then remove the final 80
+    assert stripped[-1] == 0x80
+    return stripped[:-1]
+
+class Scp02SessionKeys:
+    """A single set of GlobalPlatform session keys."""
+    DERIV_CONST_CMAC = b'\x01\x01'
+    DERIV_CONST_RMAC = b'\x01\x02'
+    DERIV_CONST_ENC = b'\x01\x82'
+    DERIV_CONST_DENC = b'\x01\x81'
+    blocksize = 8
+
+    def calc_mac_1des(self, data: bytes, reset_icv: bool = False) -> bytes:
+        """Pad and calculate MAC according to B.1.2.2 - Single DES plus final 3DES"""
+        e = DES.new(self.c_mac[:8], DES.MODE_ECB)
+        d = DES.new(self.c_mac[8:], DES.MODE_ECB)
+        padded_data = pad80(data, 8)
+        q = len(padded_data) // 8
+        icv = b'\x00' * 8 if reset_icv else self.icv
+        h = icv
+        for i in range(q):
+            h = e.encrypt(strxor(h, bytes(padded_data[8*i:8*(i+1)])))
+        h = d.decrypt(h)
+        h = e.encrypt(h)
+        logger.debug("mac_1des(%s,icv=%s) -> %s", b2h(data), b2h(icv), b2h(h))
+        if self.des_icv_enc:
+            self.icv = self.des_icv_enc.encrypt(h)
+        else:
+            self.icv = h
+        return h
+
+    def calc_mac_3des(self, data: bytes) -> bytes:
+        e = DES3.new(self.enc, DES.MODE_ECB)
+        padded_data = pad80(data, 8)
+        q = len(padded_data) // 8
+        h = b'\x00' * 8
+        for i in range(q):
+            h = e.encrypt(strxor(h, bytes(padded_data[8*i:8*(i+1)])))
+        logger.debug("mac_3des(%s) -> %s", b2h(data), b2h(h))
+        return h
+
+    def __init__(self, counter: int, card_keys: 'GpCardKeyset', icv_encrypt=True):
+        self.icv = None
+        self.counter = counter
+        self.card_keys = card_keys
+        self.c_mac = scp02_key_derivation(self.DERIV_CONST_CMAC, self.counter, card_keys.mac)
+        self.r_mac = scp02_key_derivation(self.DERIV_CONST_RMAC, self.counter, card_keys.mac)
+        self.enc = scp02_key_derivation(self.DERIV_CONST_ENC, self.counter, card_keys.enc)
+        self.data_enc = scp02_key_derivation(self.DERIV_CONST_DENC, self.counter, card_keys.dek)
+        self.des_icv_enc = DES.new(self.c_mac[:8], DES.MODE_ECB) if icv_encrypt else None
+
+    def __str__(self) -> str:
+        return "%s(CTR=%u, ICV=%s, ENC=%s, D-ENC=%s, MAC-C=%s, MAC-R=%s)" % (
+                self.__class__.__name__, self.counter, b2h(self.icv) if self.icv else "None",
+                b2h(self.enc), b2h(self.data_enc), b2h(self.c_mac), b2h(self.r_mac))
+
+INS_INIT_UPDATE = 0x50
+INS_EXT_AUTH = 0x82
+CLA_SM = 0x04
+
+class SCP(SecureChannel, abc.ABC):
+    """Abstract base class containing some common interface + functionality for SCP protocols."""
+    def __init__(self, card_keys: 'GpCardKeyset', lchan_nr: int = 0):
+
+        # Spec references that explain KVN ranges:
+        # TS 102 225 Annex A.1 states KVN 0x01..0x0F shall be used for SCP80
+        # GPC_GUI_003 states
+        #   * For the Issuer Security Domain, this is initially Key Version Number 'FF' which has been deliberately
+        #     chosen to be outside of the allowable range ('01' to '7F') for a Key Version Number.
+        #   * It is logical that the initial keys in the Issuer Security Domain be replaced by an initial issuer Key
+        #     Version Number in the range '01' to '6F'.
+        #   * Key Version Numbers '70' to '72' and '74' to '7F' are reserved for future use.
+        #   * On an implementation supporting Supplementary Security Domains, the RSA public key with a Key Version
+        #     Number '73' and a Key Identifier of '01' has the following functionality in a Supplementary Security
+        #     Domain with the DAP Verification privilege [...]
+        # GPC_GUI_010 V1.0.1 Section 6 states
+        #   * Key Version number range ('20' to '2F') is reserved for SCP02
+        #   * Key Version 'FF' is reserved for use by an Issuer Security Domain supporting SCP02, and cannot be used
+        #     for SCP80. This initial key set shall be replaced by a key set with a Key Version Number in the
+        #     ('20' to '2F') range.
+        #   * Key Version number range ('01' to '0F') is reserved for SCP80
+        #   * Key Version number '70' with Key Identifier '01' is reserved for the Token Key, which is either a RSA
+        #     public key or a DES key
+        #   * Key Version number '71' with Key Identifier '01' is reserved for the Receipt Key, which is a DES key
+        #   * Key Version Number '11' is reserved for DAP as specified in ETSI TS 102 226 [2]
+        #   * Key Version Number '73' with Key Identifier '01' is reserved for the DAP verification key as specified
+        #     in sections 3.3.3 and 4 of [4], which is either an RSA public key or DES key
+        #   * Key Version Number '74' is reserved for the CASD Keys (cf. section 9.2)
+        #   * Key Version Number '75' with Key Identifier '01' is reserved for the key used to decipher the Ciphered
+        #     Load File Data Block described in section 4.8 of [5].
+
+        if card_keys.kvn == 0:
+            # Key Version Number 0x00 refers to the first available key, so we won't carry out
+            # a range check in this case. See also: GPC_SPE_034, section E.5.1.3
+            pass
+        elif hasattr(self, 'kvn_range'):
+            if not card_keys.kvn in range(self.kvn_range[0], self.kvn_range[1]+1):
+                raise ValueError('%s cannot be used with KVN outside range 0x%02x..0x%02x' %
+                                 (self.__class__.__name__, self.kvn_range[0], self.kvn_range[1]))
+        elif hasattr(self, 'kvn_ranges'):
+            # pylint: disable=no-member
+            if all([not card_keys.kvn in range(x[0], x[1]+1) for x in self.kvn_ranges]):
+                raise ValueError('%s cannot be used with KVN outside permitted ranges %s' %
+                                 (self.__class__.__name__, self.kvn_ranges))
+
+        self.lchan_nr = lchan_nr
+        self.card_keys = card_keys
+        self.sk = None
+        self.mac_on_unmodified = False
+        self.security_level = 0x00
+
+    @property
+    def do_cmac(self) -> bool:
+        """Should we perform C-MAC?"""
+        return self.security_level & 0x01
+
+    @property
+    def do_rmac(self) -> bool:
+        """Should we perform R-MAC?"""
+        return self.security_level & 0x10
+
+    @property
+    def do_cenc(self) -> bool:
+        """Should we perform C-ENC?"""
+        return self.security_level & 0x02
+
+    @property
+    def do_renc(self) -> bool:
+        """Should we perform R-ENC?"""
+        return self.security_level & 0x20
+
+    def __str__(self) -> str:
+        return "%s[%02x]" % (self.__class__.__name__, self.security_level)
+
+    def _cla(self, sm: bool = False, b8: bool = True) -> int:
+        ret = 0x80 if b8 else 0x00
+        if sm:
+            ret = ret | CLA_SM
+        return ret + self.lchan_nr
+
+    def wrap_cmd_apdu(self, apdu: bytes, *args, **kwargs) -> bytes:
+        # Generic handling of GlobalPlatform SCP, implements SecureChannel.wrap_cmd_apdu
+        # only protect those APDUs that actually are global platform commands
+        if apdu[0] & 0x80:
+            return self._wrap_cmd_apdu(apdu, *args, **kwargs)
+        return apdu
+
+    @abc.abstractmethod
+    def _wrap_cmd_apdu(self, apdu: bytes, *args, **kwargs) -> bytes:
+        """Method implementation to be provided by derived class."""
+        pass
+
+    @abc.abstractmethod
+    def gen_init_update_apdu(self, host_challenge: Optional[bytes]) -> bytes:
+        pass
+
+    @abc.abstractmethod
+    def parse_init_update_resp(self, resp_bin: bytes):
+        pass
+
+    @abc.abstractmethod
+    def gen_ext_auth_apdu(self, security_level: int = 0x01) -> bytes:
+        pass
+
+    def encrypt_key(self, key: bytes) -> bytes:
+        """Encrypt a key with the DEK."""
+        num_pad = len(key) % self.sk.blocksize
+        if num_pad:
+            return bertlv_encode_len(len(key)) + self.dek_encrypt(key + b'\x00'*num_pad)
+        return self.dek_encrypt(key)
+
+    def decrypt_key(self, encrypted_key:bytes) -> bytes:
+        """Decrypt a key with the DEK."""
+        if len(encrypted_key) % self.sk.blocksize:
+            # If the length of the Key Component Block is not a multiple of the block size of the encryption #
+            # algorithm (i.e. 8 bytes for DES, 16 bytes for AES), then it shall be assumed that the key
+            # component value was right-padded prior to encryption and that the Key Component Block was
+            # formatted as described in Table 11-70. In this case, the first byte(s) of the Key Component
+            # Block provides the actual length of the key component value, which allows recovering the
+            # clear-text key component value after decryption of the encrypted key component value and removal
+            # of padding bytes.
+            decrypted = self.dek_decrypt(encrypted_key)
+            key_len, remainder = bertlv_parse_len(decrypted)
+            return remainder[:key_len]
+        else:
+            # If the length of the Key Component Block is a multiple of the block size of the encryption
+            # algorithm (i.e.  8 bytes for DES, 16 bytes for AES), then it shall be assumed that no padding
+            # bytes were added before encrypting the key component value and that the Key Component Block is
+            # only composed of the encrypted key component value (as shown in Table 11-71). In this case, the
+            # clear-text key component value is simply recovered by decrypting the Key Component Block.
+            return self.dek_decrypt(encrypted_key)
+
+    @abc.abstractmethod
+    def dek_encrypt(self, plaintext:bytes) -> bytes:
+        pass
+
+    @abc.abstractmethod
+    def dek_decrypt(self, ciphertext:bytes) -> bytes:
+        pass
+
+
+class SCP02(SCP):
+    """An instance of the GlobalPlatform SCP02 secure channel protocol."""
+
+    constr_iur = Struct('key_div_data'/Bytes(10), 'key_ver'/Int8ub, Const(b'\x02'),
+                        'seq_counter'/Int16ub, 'card_challenge'/Bytes(6), 'card_cryptogram'/Bytes(8))
+    # Key Version Number 0x70 is a non-spec special-case of sysmoISIM-SJA2/SJA5 and possibly more sysmocom products
+    # Key Version Number 0x01 is a non-spec special-case of sysmoUSIM-SJS1
+    kvn_ranges = [[0x01, 0x01], [0x20, 0x2f], [0x70, 0x70]]
+
+    def __init__(self, *args, **kwargs):
+        self.overhead = 8
+        super().__init__(*args, **kwargs)
+
+    def dek_encrypt(self, plaintext:bytes) -> bytes:
+        cipher = DES.new(self.card_keys.dek[:8], DES.MODE_ECB)
+        return cipher.encrypt(plaintext)
+
+    def dek_decrypt(self, ciphertext:bytes) -> bytes:
+        cipher = DES.new(self.card_keys.dek[:8], DES.MODE_ECB)
+        return cipher.decrypt(ciphertext)
+
+    def _compute_cryptograms(self, card_challenge: bytes, host_challenge: bytes):
+        logger.debug("host_challenge(%s), card_challenge(%s)", b2h(host_challenge), b2h(card_challenge))
+        self.host_cryptogram = self.sk.calc_mac_3des(self.sk.counter.to_bytes(2, 'big') + card_challenge + host_challenge)
+        self.card_cryptogram = self.sk.calc_mac_3des(self.host_challenge + self.sk.counter.to_bytes(2, 'big') + card_challenge)
+        logger.debug("host_cryptogram(%s), card_cryptogram(%s)", b2h(self.host_cryptogram), b2h(self.card_cryptogram))
+
+    def gen_init_update_apdu(self, host_challenge: bytes = b'\x00'*8) -> bytes:
+        """Generate INITIALIZE UPDATE APDU."""
+        self.host_challenge = host_challenge
+        return bytes([self._cla(), INS_INIT_UPDATE, self.card_keys.kvn, 0, 8]) + self.host_challenge + b'\x00'
+
+    def parse_init_update_resp(self, resp_bin: bytes):
+        """Parse response to INITIALZIE UPDATE."""
+        resp = self.constr_iur.parse(resp_bin)
+        self.card_challenge = resp['card_challenge']
+        self.sk = Scp02SessionKeys(resp['seq_counter'], self.card_keys)
+        logger.debug(self.sk)
+        self._compute_cryptograms(self.card_challenge, self.host_challenge)
+        if self.card_cryptogram != resp['card_cryptogram']:
+            raise ValueError("card cryptogram doesn't match")
+
+    def gen_ext_auth_apdu(self, security_level: int = 0x01) -> bytes:
+        """Generate EXTERNAL AUTHENTICATE APDU."""
+        if security_level & 0xf0:
+            raise NotImplementedError('R-MAC/R-ENC for SCP02 not implemented yet.')
+        self.security_level = security_level
+        if self.mac_on_unmodified:
+            header = bytes([self._cla(), INS_EXT_AUTH, self.security_level, 0, 8])
+        else:
+            header = bytes([self._cla(True), INS_EXT_AUTH, self.security_level, 0, 16])
+        #return self.wrap_cmd_apdu(header + self.host_cryptogram)
+        mac = self.sk.calc_mac_1des(header + self.host_cryptogram, True)
+        return bytes([self._cla(True), INS_EXT_AUTH, self.security_level, 0, 16]) + self.host_cryptogram + mac
+
+    def _wrap_cmd_apdu(self, apdu: bytes, *args, **kwargs) -> bytes:
+        """Wrap Command APDU for SCP02: calculate MAC and encrypt."""
+        logger.debug("wrap_cmd_apdu(%s)", b2h(apdu))
+
+        if not self.do_cmac:
+            return apdu
+
+        (case, lc, le, data) = parse_command_apdu(apdu)
+
+        # TODO: add support for extended length fields.
+        assert lc <= 256
+        assert le <= 256
+        lc &= 0xFF
+        le &= 0xFF
+
+        # CLA without log. channel can be 80 or 00 only
+        cla = apdu[0]
+        b8 = cla & 0x80
+        if cla & 0x03 or cla & CLA_SM:
+            # nonzero logical channel in APDU, check that are the same
+            assert cla == self._cla(False, b8), "CLA mismatch"
+
+        if self.mac_on_unmodified:
+            mlc = lc
+            clac = cla
+        else:
+            # CMAC on modified APDU
+            mlc = lc + 8
+            clac = cla | CLA_SM
+        mac = self.sk.calc_mac_1des(bytes([clac]) + apdu[1:4] + bytes([mlc]) + data)
+        if self.do_cenc:
+            k = DES3.new(self.sk.enc, DES.MODE_CBC, b'\x00'*8)
+            data = k.encrypt(pad80(data, 8))
+            lc = len(data)
+
+        lc += 8
+        apdu = bytes([self._cla(True, b8)]) + apdu[1:4] + bytes([lc]) + data + mac
+
+        # Since we attach a signature, we will always send some data. This means that if the APDU is of case #4
+        # or case #2, we must attach an additional Le byte to signal that we expect a response. It is technically
+        # legal to use 0x00 (=256) as Le byte, even when the caller has specified a different value in the original
+        # APDU. This is due to the fact that Le always describes the maximum expected length of the response
+        # (see also ISO/IEC 7816-4, section 5.1). In addition to that, it should also important that depending on
+        # the configuration of the SCP, the response may also contain a signature that makes the response larger
+        # than specified in the Le field of the original APDU.
+        if case == 4 or case == 2:
+            apdu += b'\x00'
+
+        return apdu
+
+    def unwrap_rsp_apdu(self, sw: bytes, rsp_apdu: bytes) -> bytes:
+        # TODO: Implement R-MAC / R-ENC
+        return rsp_apdu
+
+
+
+from Cryptodome.Cipher import AES
+from Cryptodome.Hash import CMAC
+
+def scp03_key_derivation(constant: bytes, context: bytes, base_key: bytes, l: Optional[int] = None) -> bytes:
+    """SCP03 Key Derivation Function as specified in Annex D 4.1.5."""
+    # Data derivation shall use KDF in counter mode as specified in NIST SP 800-108 ([NIST 800-108]). The PRF
+    # used in the KDF shall be CMAC as specified in [NIST 800-38B], used with full 16-byte output length.
+    def prf(key: bytes, data:bytes):
+        return CMAC.new(key, data, AES).digest()
+
+    if l is None:
+        l = len(base_key) * 8
+
+    logger.debug("scp03_kdf(constant=%s, context=%s, base_key=%s, l=%u)", b2h(constant), b2h(context), b2h(base_key), l)
+    output_len = l // 8
+    # SCP03 Section 4.1.5 defines a different parameter order than NIST SP 800-108, so we cannot use the
+    # existing Cryptodome.Protocol.KDF.SP800_108_Counter function :(
+    # A 12-byte “label” consisting of 11 bytes with value '00' followed by a 1-byte derivation constant
+    assert len(constant) == 1
+    label = b'\x00' *11 + constant
+    i = 1
+    dk = b''
+    while len(dk) < output_len:
+        # 12B label, 1B separation, 2B L, 1B i, Context
+        info = label + b'\x00' + l.to_bytes(2, 'big') + bytes([i]) + context
+        dk += prf(base_key, info)
+        i += 1
+        if i > 0xffff:
+            raise ValueError("Overflow in SP800 108 counter")
+    return dk[:output_len]
+
+
+class Scp03SessionKeys:
+    # GPC 2.3 Amendment D v1.2 Section 4.1.5 Table 4-1
+    DERIV_CONST_AUTH_CGRAM_CARD = b'\x00'
+    DERIV_CONST_AUTH_CGRAM_HOST = b'\x01'
+    DERIV_CONST_CARD_CHLG_GEN = b'\x02'
+    DERIV_CONST_KDERIV_S_ENC = b'\x04'
+    DERIV_CONST_KDERIV_S_MAC = b'\x06'
+    DERIV_CONST_KDERIV_S_RMAC = b'\x07'
+    blocksize = 16
+
+    def __init__(self, card_keys: 'GpCardKeyset', host_challenge: bytes, card_challenge: bytes):
+        # GPC 2.3 Amendment D v1.2 Section 6.2.1
+        context = host_challenge + card_challenge
+        self.s_enc = scp03_key_derivation(self.DERIV_CONST_KDERIV_S_ENC, context, card_keys.enc)
+        self.s_mac = scp03_key_derivation(self.DERIV_CONST_KDERIV_S_MAC, context, card_keys.mac)
+        self.s_rmac = scp03_key_derivation(self.DERIV_CONST_KDERIV_S_RMAC, context, card_keys.mac)
+
+
+        # The first MAC chaining value is set to 16 bytes '00'
+        self.mac_chaining_value = b'\x00' * 16
+        # The encryption counter’s start value shall be set to 1 (we set it immediately before generating ICV)
+        self.block_nr = 0
+
+    def calc_cmac(self, apdu: bytes):
+        """Compute C-MAC for given to-be-transmitted APDU.
+        Returns the full 16-byte MAC, caller must truncate it if needed for S8 mode."""
+        cmac_input = self.mac_chaining_value + apdu
+        cmac_val = CMAC.new(self.s_mac, cmac_input, ciphermod=AES).digest()
+        self.mac_chaining_value = cmac_val
+        return cmac_val
+
+    def calc_rmac(self, rdata_and_sw: bytes):
+        """Compute R-MAC for given received R-APDU data section.
+        Returns the full 16-byte MAC, caller must truncate it if needed for S8 mode."""
+        rmac_input = self.mac_chaining_value + rdata_and_sw
+        return CMAC.new(self.s_rmac, rmac_input, ciphermod=AES).digest()
+
+    def _get_icv(self, is_response: bool = False):
+        """Obtain the ICV value computed as described in 6.2.6.
+        This method has two modes:
+            * is_response=False for computing the ICV for C-ENC. Will pre-increment the counter.
+            * is_response=False for computing the ICV for R-DEC."""
+        if not is_response:
+            self.block_nr += 1
+        # The binary value of this number SHALL be left padded with zeroes to form a full block.
+        data = self.block_nr.to_bytes(self.blocksize, "big")
+        if is_response:
+            # Section 6.2.7: additional intermediate step: Before encryption, the most significant byte of
+            # this block shall be set to '80'.
+            data = b'\x80' + data[1:]
+        iv = bytes([0] * self.blocksize)
+        # This block SHALL be encrypted with S-ENC to produce the ICV for command encryption.
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, iv)
+        icv = cipher.encrypt(data)
+        logger.debug("_get_icv(data=%s, is_resp=%s) -> icv=%s", b2h(data), is_response, b2h(icv))
+        return icv
+
+    # TODO: Resolve duplication with pySim.esim.bsp.BspAlgoCryptAES128 which provides pad80-wrapping
+    def _encrypt(self, data: bytes, is_response: bool = False) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv(is_response))
+        return cipher.encrypt(data)
+
+    # TODO: Resolve duplication with pySim.esim.bsp.BspAlgoCryptAES128 which provides pad80-unwrapping
+    def _decrypt(self, data: bytes, is_response: bool = True) -> bytes:
+        cipher = AES.new(self.s_enc, AES.MODE_CBC, self._get_icv(is_response))
+        return cipher.decrypt(data)
+
+
+class SCP03(SCP):
+    """Secure Channel Protocol (SCP) 03 as specified in GlobalPlatform v2.3 Amendment D."""
+
+    # Section 7.1.1.6 / Table 7-3
+    constr_iur = Struct('key_div_data'/Bytes(10), 'key_ver'/Int8ub, Const(b'\x03'), 'i_param'/Int8ub,
+                        'card_challenge'/Bytes(lambda ctx: ctx._.s_mode),
+                        'card_cryptogram'/Bytes(lambda ctx: ctx._.s_mode),
+                        'sequence_counter'/COptional(Bytes(3)))
+    kvn_range = [0x30, 0x3f]
+
+    def __init__(self, *args, **kwargs):
+        self.s_mode = kwargs.pop('s_mode', 8)
+        self.overhead = self.s_mode
+        super().__init__(*args, **kwargs)
+
+    def dek_encrypt(self, plaintext:bytes) -> bytes:
+        cipher = AES.new(self.card_keys.dek, AES.MODE_CBC, b'\x00'*16)
+        return cipher.encrypt(plaintext)
+
+    def dek_decrypt(self, ciphertext:bytes) -> bytes:
+        cipher = AES.new(self.card_keys.dek, AES.MODE_CBC, b'\x00'*16)
+        return cipher.decrypt(ciphertext)
+
+    def _compute_cryptograms(self):
+        logger.debug("host_challenge(%s), card_challenge(%s)", b2h(self.host_challenge), b2h(self.card_challenge))
+        # Card + Host Authentication Cryptogram: Section 6.2.2.2 + 6.2.2.3
+        context = self.host_challenge + self.card_challenge
+        self.card_cryptogram = scp03_key_derivation(self.sk.DERIV_CONST_AUTH_CGRAM_CARD, context, self.sk.s_mac, l=self.s_mode*8)
+        self.host_cryptogram = scp03_key_derivation(self.sk.DERIV_CONST_AUTH_CGRAM_HOST, context, self.sk.s_mac, l=self.s_mode*8)
+        logger.debug("host_cryptogram(%s), card_cryptogram(%s)", b2h(self.host_cryptogram), b2h(self.card_cryptogram))
+
+    def gen_init_update_apdu(self, host_challenge: Optional[bytes] = None) -> bytes:
+        """Generate INITIALIZE UPDATE APDU."""
+        if host_challenge is None:
+            host_challenge = b'\x00' * self.s_mode
+        if len(host_challenge) != self.s_mode:
+            raise ValueError('Host Challenge must be %u bytes long' % self.s_mode)
+        self.host_challenge = host_challenge
+        return bytes([self._cla(), INS_INIT_UPDATE, self.card_keys.kvn, 0, len(host_challenge)]) + host_challenge + b'\x00'
+
+    def parse_init_update_resp(self, resp_bin: bytes):
+        """Parse response to INITIALIZE UPDATE."""
+        if len(resp_bin) not in [10+3+8+8, 10+3+16+16, 10+3+8+8+3, 10+3+16+16+3]:
+            raise ValueError('Invalid length of Initialize Update Response')
+        resp = self.constr_iur.parse(resp_bin, s_mode=self.s_mode)
+        self.card_challenge = resp['card_challenge']
+        self.i_param = resp['i_param']
+        # derive session keys and compute cryptograms
+        self.sk = Scp03SessionKeys(self.card_keys, self.host_challenge, self.card_challenge)
+        logger.debug(self.sk)
+        self._compute_cryptograms()
+        # verify computed cryptogram matches received cryptogram
+        if self.card_cryptogram != resp['card_cryptogram']:
+            raise ValueError("card cryptogram doesn't match")
+
+    def gen_ext_auth_apdu(self, security_level: int = 0x01) -> bytes:
+        """Generate EXTERNAL AUTHENTICATE APDU."""
+        self.security_level = security_level
+        header = bytes([self._cla(), INS_EXT_AUTH, self.security_level, 0, self.s_mode])
+        # bypass encryption for EXTERNAL AUTHENTICATE
+        return self.wrap_cmd_apdu(header + self.host_cryptogram, skip_cenc=True)
+
+    def _wrap_cmd_apdu(self, apdu: bytes, skip_cenc: bool = False) -> bytes:
+        """Wrap Command APDU for SCP03: calculate MAC and encrypt."""
+        logger.debug("wrap_cmd_apdu(%s)", b2h(apdu))
+
+        if not self.do_cmac:
+            return apdu
+
+        cla = apdu[0]
+        ins = apdu[1]
+        p1 = apdu[2]
+        p2 = apdu[3]
+        (case, lc, le, cmd_data) = parse_command_apdu(apdu)
+
+        # TODO: add support for extended length fields.
+        assert lc <= 256
+        assert le <= 256
+        lc &= 0xFF
+        le &= 0xFF
+
+        if self.do_cenc and not skip_cenc:
+            if case <= 2:
+                # No encryption shall be applied to a command where there is no command data field. In this
+                # case, the encryption counter shall still be incremented
+                self.sk.block_nr += 1
+            else:
+                # data shall be padded as defined in [GPCS] section B.2.3
+                padded_data = pad80(cmd_data, 16)
+                lc = len(padded_data)
+                if lc >= 256:
+                    raise ValueError('Modified Lc (%u) would exceed maximum when appending padding' % (lc))
+                # perform AES-CBC with ICV + S_ENC
+                cmd_data = self.sk._encrypt(padded_data)
+
+        # The length of the command message (Lc) shall be incremented by 8 (in S8 mode) or 16 (in S16
+        # mode) to indicate the inclusion of the C-MAC in the data field of the command message.
+        mlc = lc + self.s_mode
+        if mlc >= 256:
+            raise ValueError('Modified Lc (%u) would exceed maximum when appending %u bytes of mac' % (mlc, self.s_mode))
+        # The class byte shall be modified for the generation or verification of the C-MAC: The logical
+        # channel number shall be set to zero, bit 4 shall be set to 0 and bit 3 shall be set to 1 to indicate
+        # GlobalPlatform proprietary secure messaging.
+        mcla = (cla & 0xF0) | CLA_SM
+        apdu = bytes([mcla, ins, p1, p2, mlc]) + cmd_data
+        cmac = self.sk.calc_cmac(apdu)
+        apdu += cmac[:self.s_mode]
+
+        # See comment in SCP03._wrap_cmd_apdu()
+        if case == 4 or case == 2:
+            apdu += b'\x00'
+
+        return apdu
+
+    def unwrap_rsp_apdu(self, sw: bytes, rsp_apdu: bytes) -> bytes:
+        # No R-MAC shall be generated and no protection shall be applied to a response that includes an error
+        # status word: in this case only the status word shall be returned in the response. All status words
+        # except '9000' and warning status words (i.e. '62xx' and '63xx') shall be interpreted as error status
+        # words.
+        logger.debug("unwrap_rsp_apdu(sw=%s, rsp_apdu=%s)", sw, rsp_apdu)
+        if not self.do_rmac:
+            assert not self.do_renc
+            return rsp_apdu
+
+        if sw != b'\x90\x00' and sw[0] not in [0x62, 0x63]:
+            return rsp_apdu
+        response_data = rsp_apdu[:-self.s_mode]
+        rmac = rsp_apdu[-self.s_mode:]
+        rmac_exp = self.sk.calc_rmac(response_data + sw)[:self.s_mode]
+        if rmac != rmac_exp:
+            raise ValueError("R-MAC value not matching: received: %s, computed: %s" % (rmac, rmac_exp))
+
+        if self.do_renc:
+            # decrypt response data
+            decrypted = self.sk._decrypt(response_data)
+            logger.debug("decrypted: %s", b2h(decrypted))
+            # remove padding
+            response_data = unpad80(decrypted)
+            logger.debug("response_data: %s", b2h(response_data))
+
+        return response_data

pySim/global_platform/uicc.py

@@ -0,0 +1,107 @@
+# coding=utf-8
+"""GlobalPLatform UICC Configuration 1.0 parameters
+
+(C) 2024 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from construct import Optional as COptional
+from construct import Struct, GreedyRange, FlagsEnum, Int16ub, Int24ub, Padding, Bit, Const
+from osmocom.construct import *
+from osmocom.utils import *
+from osmocom.tlv import *
+
+# Section 11.6.2.3 / Table 11-58
+class SecurityDomainAid(BER_TLV_IE, tag=0x4f):
+    _construct = GreedyBytes
+class LoadFileDataBlockSignature(BER_TLV_IE, tag=0xc3):
+    _construct = GreedyBytes
+class DapBlock(BER_TLV_IE, tag=0xe2, nested=[SecurityDomainAid, LoadFileDataBlockSignature]):
+    pass
+class LoadFileDataBlock(BER_TLV_IE, tag=0xc4):
+    _construct = GreedyBytes
+class Icv(BER_TLV_IE, tag=0xd3):
+    _construct = GreedyBytes
+class CipheredLoadFileDataBlock(BER_TLV_IE, tag=0xd4):
+    _construct = GreedyBytes
+class LoadFile(TLV_IE_Collection, nested=[DapBlock, LoadFileDataBlock, Icv, CipheredLoadFileDataBlock]):
+    pass
+
+# UICC Configuration v1.0.1 / Section 4.3.2
+class UiccScp(BER_TLV_IE, tag=0x81):
+    _construct = Struct('scp'/Int8ub, 'i'/Int8ub)
+
+class AcceptExtradAppsAndElfToSd(BER_TLV_IE, tag=0x82):
+    _construct = GreedyBytes
+
+class AcceptDelOfAssocSd(BER_TLV_IE, tag=0x83):
+    _construct = GreedyBytes
+
+class LifeCycleTransitionToPersonalized(BER_TLV_IE, tag=0x84):
+    _construct = GreedyBytes
+
+class CasdCapabilityInformation(BER_TLV_IE, tag=0x86):
+    _construct = GreedyBytes
+
+class AcceptExtradAssocAppsAndElf(BER_TLV_IE, tag=0x87):
+    _construct = GreedyBytes
+
+# Security Domain Install Parameters (inside C9 during INSTALL [for install])
+class UiccSdInstallParams(TLV_IE_Collection, nested=[UiccScp, AcceptExtradAppsAndElfToSd, AcceptDelOfAssocSd,
+                                                     LifeCycleTransitionToPersonalized,
+                                                     CasdCapabilityInformation, AcceptExtradAssocAppsAndElf]):
+    def has_scp(self, scp: int) -> bool:
+        """Determine if SD Installation parameters already specify given SCP."""
+        for c in self.children:
+            if not isinstance(c, UiccScp):
+                continue
+            if c.decoded['scp'] == scp:
+                return True
+        return False
+
+    def add_scp(self, scp: int, i: int):
+        """Add given SCP (and i parameter) to list of SCP of the Security Domain Install Params.
+        Example: add_scp(0x03, 0x70) for SCP03, or add_scp(0x02, 0x55) for SCP02."""
+        if self.has_scp(scp):
+            raise ValueError('SCP%02x already present' % scp)
+        self.children.append(UiccScp(decoded={'scp': scp, 'i': i}))
+
+    def remove_scp(self, scp: int):
+        """Remove given SCP from list of SCP of the Security Domain Install Params."""
+        for c in self.children:
+            if not isinstance(c, UiccScp):
+                continue
+            if c.decoded['scp'] == scp:
+                self.children.remove(c)
+                return
+        raise ValueError("SCP%02x not present" % scp)
+
+
+# Key Usage:
+# KVN 0x01 .. 0x0F reserved for SCP80
+# KVN 0x11 reserved for DAP specified in ETSI TS 102 226
+# KVN 0x20 .. 0x2F reserved for SCP02
+#   KID 0x01 = ENC; 0x02 = MAC; 0x03 = DEK
+# KVN 0x30 .. 0x3F reserved for SCP03
+#   KID 0x01 = ENC; 0x02 = MAC; 0x03 = DEK
+# KVN 0x70 KID 0x01: Token key (RSA public or DES)
+# KVN 0x71 KID 0x01: Receipt key (DES)
+# KVN 0x73 KID 0x01: DAP verifiation key (RS public or DES)
+# KVN 0x74 reserved for CASD
+#   KID 0x01: PK.CA.AUT
+#   KID 0x02: SK.CASD.AUT (PK) and KS.CASD.AUT (Non-PK)
+#   KID 0x03: SK.CASD.CT (P) and KS.CASD.CT (Non-PK)
+# KVN 0x75 KID 0x01: 16-byte DES key for  Ciphered Load File Data Block
+# KVN 0xFF reserved for ISD with SCP02 without SCP80 s support

pySim/gsm_r.py

@@ -1,13 +1,10 @@
-# -*- coding: utf-8 -*-
-
-# without this, pylint will fail when inner classes are used
-# within the 'nested' kwarg of our TlvMeta metaclass on python 3.7 :(
-# pylint: disable=undefined-variable
-
 """
 The File (and its derived classes) uses the classes of pySim.filesystem in
 order to describe the files specified in UIC Reference P38 T 9001 5.0 "FFFIS for GSM-R SIM Cards"
 """
+# without this, pylint will fail when inner classes are used
+# within the 'nested' kwarg of our TlvMeta metaclass on python 3.7 :(
+# pylint: disable=undefined-variable
 
 #
 # Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
@@ -28,16 +25,13 @@ order to describe the files specified in UIC Reference P38 T 9001 5.0 "FFFIS for
 
 
 from pySim.utils import *
-#from pySim.tlv import *
 from struct import pack, unpack
-from construct import *
+from construct import Struct, Int8ub, Int16ub, Int24ub, Int32ub, FlagsEnum
 from construct import Optional as COptional
-from pySim.construct import *
-import enum
+from osmocom.construct import *
 
+from pySim.profile import CardProfileAddon
 from pySim.filesystem import *
-import pySim.ts_102_221
-import pySim.ts_51_011
 
 ######################################################################
 # DF.EIRENE (FFFIS for GSM-R SIM Cards)
@@ -47,10 +41,10 @@ import pySim.ts_51_011
 class FuncNTypeAdapter(Adapter):
     def _decode(self, obj, context, path):
         bcd = swap_nibbles(b2h(obj))
-        last_digit = bcd[-1]
+        last_digit = int(bcd[-1], 16)
         return {'functional_number': bcd[:-1],
-                'presentation_of_only_this_fn': last_digit & 4,
-                'permanent_fn': last_digit & 8}
+                'presentation_of_only_this_fn': bool(last_digit & 4),
+                'permanent_fn': bool(last_digit & 8)}
 
     def _encode(self, obj, context, path):
         return 'FIXME'
@@ -58,10 +52,14 @@ class FuncNTypeAdapter(Adapter):
 
 class EF_FN(LinFixedEF):
     """Section 7.2"""
-
+    _test_decode = [
+        ( "40315801000010ff01",
+          { "functional_number_and_type": { "functional_number": "04138510000001f",
+                "presentation_of_only_this_fn": True, "permanent_fn": True }, "list_number": 1 } ),
+    ]
     def __init__(self):
-        super().__init__(fid='6ff1', sfid=None, name='EF.EN',
-                         desc='Functional numbers', rec_len={9, 9})
+        super().__init__(fid='6ff1', sfid=None, name='EF.FN',
+                         desc='Functional numbers', rec_len=(9, 9))
         self._construct = Struct('functional_number_and_type'/FuncNTypeAdapter(Bytes(8)),
                                  'list_number'/Int8ub)
 
@@ -73,15 +71,15 @@ class PlConfAdapter(Adapter):
         num = int(obj) & 0x7
         if num == 0:
             return 'None'
-        elif num == 1:
+        if num == 1:
             return 4
-        elif num == 2:
+        if num == 2:
             return 3
-        elif num == 3:
+        if num == 3:
             return 2
-        elif num == 4:
+        if num == 4:
             return 1
-        elif num == 5:
+        if num == 5:
             return 0
 
     def _encode(self, obj, context, path):
@@ -90,13 +88,13 @@ class PlConfAdapter(Adapter):
         obj = int(obj)
         if obj == 4:
             return 1
-        elif obj == 3:
+        if obj == 3:
             return 2
-        elif obj == 2:
+        if obj == 2:
             return 3
-        elif obj == 1:
+        if obj == 1:
             return 4
-        elif obj == 0:
+        if obj == 0:
             return 5
 
 
@@ -107,19 +105,19 @@ class PlCallAdapter(Adapter):
         num = int(obj) & 0x7
         if num == 0:
             return 'None'
-        elif num == 1:
+        if num == 1:
             return 4
-        elif num == 2:
+        if num == 2:
             return 3
-        elif num == 3:
+        if num == 3:
             return 2
-        elif num == 4:
+        if num == 4:
             return 1
-        elif num == 5:
+        if num == 5:
             return 0
-        elif num == 6:
+        if num == 6:
             return 'B'
-        elif num == 7:
+        if num == 7:
             return 'A'
 
     def _encode(self, obj, context, path):
@@ -127,17 +125,17 @@ class PlCallAdapter(Adapter):
             return 0
         if obj == 4:
             return 1
-        elif obj == 3:
+        if obj == 3:
             return 2
-        elif obj == 2:
+        if obj == 2:
             return 3
-        elif obj == 1:
+        if obj == 1:
             return 4
-        elif obj == 0:
+        if obj == 0:
             return 5
-        elif obj == 'B':
+        if obj == 'B':
             return 6
-        elif obj == 'A':
+        if obj == 'A':
             return 7
 
 
@@ -147,9 +145,14 @@ NextTableType = Enum(Byte, decision=0xf0, predefined=0xf1,
 
 class EF_CallconfC(TransparentEF):
     """Section 7.3"""
-
+    _test_de_encode = [
+        ( "026121ffffffffffff1e000a040a010253600795792426f0",
+          { "pl_conf": 3, "conf_nr": "1612ffffffffffff", "max_rand": 30, "n_ack_max": 10,
+            "pl_ack": 1, "n_nested_max": 10, "train_emergency_gid": 1, "shunting_emergency_gid": 2,
+           "imei": "350670599742620f" } ),
+    ]
     def __init__(self):
-        super().__init__(fid='6ff2', sfid=None, name='EF.CallconfC', size={24, 24},
+        super().__init__(fid='6ff2', sfid=None, name='EF.CallconfC', size=(24, 24),
                          desc='Call Configuration of emergency calls Configuration')
         self._construct = Struct('pl_conf'/PlConfAdapter(Int8ub),
                                  'conf_nr'/BcdAdapter(Bytes(8)),
@@ -166,7 +169,7 @@ class EF_CallconfI(LinFixedEF):
     """Section 7.5"""
 
     def __init__(self):
-        super().__init__(fid='6ff3', sfid=None, name='EF.CallconfI', rec_len={21, 21},
+        super().__init__(fid='6ff3', sfid=None, name='EF.CallconfI', rec_len=(21, 21),
                          desc='Call Configuration of emergency calls Information')
         self._construct = Struct('t_dur'/Int24ub,
                                  't_relcalc'/Int32ub,
@@ -180,82 +183,131 @@ class EF_CallconfI(LinFixedEF):
 
 class EF_Shunting(TransparentEF):
     """Section 7.6"""
-
+    _test_de_encode = [
+        ( "03f8ffffff000000", { "common_gid": 3, "shunting_gid": h2b("f8ffffff000000") } ),
+    ]
     def __init__(self):
         super().__init__(fid='6ff4', sfid=None,
-                         name='EF.Shunting', desc='Shunting', size={8, 8})
+                         name='EF.Shunting', desc='Shunting', size=(8, 8))
         self._construct = Struct('common_gid'/Int8ub,
                                  'shunting_gid'/Bytes(7))
 
 
 class EF_GsmrPLMN(LinFixedEF):
     """Section 7.7"""
-
+    _test_de_encode = [
+        ( "22f860f86f8d6f8e01", { "plmn": "228-06", "class_of_network": {
+                                    "supported": { "vbs": True, "vgcs": True, "emlpp": True,
+                                    "fn": True, "eirene": True }, "preference": 0 },
+                                  "ic_incoming_ref_tbl": h2b("6f8d"), "outgoing_ref_tbl": h2b("6f8e"),
+                                  "ic_table_ref": h2b("01") } ),
+        ( "22f810416f8d6f8e02", { "plmn": "228-01", "class_of_network": {
+                                    "supported": { "vbs": False, "vgcs": False, "emlpp": False,
+                                    "fn": True, "eirene": False }, "preference": 1 },
+                                  "ic_incoming_ref_tbl": h2b("6f8d"), "outgoing_ref_tbl": h2b("6f8e"),
+                                  "ic_table_ref": h2b("02") } ),
+    ]
     def __init__(self):
         super().__init__(fid='6ff5', sfid=None, name='EF.GsmrPLMN',
-                         desc='GSM-R network selection', rec_len={9, 9})
-        self._construct = Struct('plmn'/BcdAdapter(Bytes(3)),
+                         desc='GSM-R network selection', rec_len=(9, 9))
+        self._construct = Struct('plmn'/PlmnAdapter(Bytes(3)),
                                  'class_of_network'/BitStruct('supported'/FlagsEnum(BitsInteger(5), vbs=1, vgcs=2, emlpp=4, fn=8, eirene=16),
                                                               'preference'/BitsInteger(3)),
-                                 'ic_incoming_ref_tbl'/HexAdapter(Bytes(2)),
-                                 'outgoing_ref_tbl'/HexAdapter(Bytes(2)),
-                                 'ic_table_ref'/HexAdapter(Bytes(1)))
+                                 'ic_incoming_ref_tbl'/Bytes(2),
+                                 'outgoing_ref_tbl'/Bytes(2),
+                                 'ic_table_ref'/Bytes(1))
 
 
 class EF_IC(LinFixedEF):
     """Section 7.8"""
-
+    _test_de_encode = [
+        ( "f06f8e40f10001", { "next_table_type": "decision", "id_of_next_table": h2b("6f8e"),
+                              "ic_decision_value": "041f", "network_string_table_index": 1 } ),
+        ( "ffffffffffffff", { "next_table_type": "empty", "id_of_next_table": h2b("ffff"),
+                              "ic_decision_value": "ffff", "network_string_table_index": 65535 } ),
+    ]
     def __init__(self):
         super().__init__(fid='6f8d', sfid=None, name='EF.IC',
-                         desc='International Code', rec_len={7, 7})
+                         desc='International Code', rec_len=(7, 7))
         self._construct = Struct('next_table_type'/NextTableType,
-                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'id_of_next_table'/Bytes(2),
                                  'ic_decision_value'/BcdAdapter(Bytes(2)),
-                                 'network_string_table_index'/Int8ub)
+                                 'network_string_table_index'/Int16ub)
 
 
 class EF_NW(LinFixedEF):
     """Section 7.9"""
-
+    _test_de_encode = [
+        ( "47534d2d52204348", "GSM-R CH" ),
+        ( "537769737347534d", "SwissGSM" ),
+        ( "47534d2d52204442", "GSM-R DB" ),
+        ( "47534d2d52524649", "GSM-RRFI" ),
+    ]
     def __init__(self):
         super().__init__(fid='6f80', sfid=None, name='EF.NW',
-                         desc='Network Name', rec_len={8, 8})
+                         desc='Network Name', rec_len=(8, 8))
         self._construct = GsmString(8)
 
 
 class EF_Switching(LinFixedEF):
     """Section 8.4"""
-
-    def __init__(self, fid, name, desc):
+    _test_de_encode = [
+        ( "f26f87f0ff00", { "next_table_type": "num_dial_digits", "id_of_next_table": h2b("6f87"),
+                            "decision_value": "0fff", "string_table_index": 0 } ),
+        ( "f06f8ff1ff01", { "next_table_type": "decision", "id_of_next_table": h2b("6f8f"),
+                            "decision_value": "1fff", "string_table_index": 1 } ),
+        ( "f16f89f5ff05", { "next_table_type": "predefined", "id_of_next_table": h2b("6f89"),
+                            "decision_value": "5fff", "string_table_index": 5 } ),
+    ]
+    def __init__(self, fid='1234', name='Switching', desc=None):
         super().__init__(fid=fid, sfid=None,
-                         name=name, desc=desc, rec_len={6, 6})
+                         name=name, desc=desc, rec_len=(6, 6))
         self._construct = Struct('next_table_type'/NextTableType,
-                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'id_of_next_table'/Bytes(2),
                                  'decision_value'/BcdAdapter(Bytes(2)),
                                  'string_table_index'/Int8ub)
 
 
 class EF_Predefined(LinFixedEF):
     """Section 8.5"""
+    _test_de_encode = [
+        ( "f26f85", 1, { "next_table_type": "num_dial_digits", "id_of_next_table": h2b("6f85") } ),
+        ( "f0ffc8", 2, { "predefined_value1": "0fff", "string_table_index1": 200 } ),
+    ]
+    # header and other records have different structure. WTF !?!
+    construct_first = Struct('next_table_type'/NextTableType,
+                             'id_of_next_table'/Bytes(2))
+    construct_others = Struct('predefined_value1'/BcdAdapter(Bytes(2)),
+                              'string_table_index1'/Int8ub)
 
-    def __init__(self, fid, name, desc):
+    def __init__(self, fid='1234', name='Predefined', desc=None):
         super().__init__(fid=fid, sfid=None,
-                         name=name, desc=desc, rec_len={3, 3})
-        # header and other records have different structure. WTF !?!
-        self._construct = Struct('next_table_type'/NextTableType,
-                                 'id_of_next_table'/HexAdapter(Bytes(2)),
-                                 'predefined_value1'/HexAdapter(Bytes(2)),
-                                 'string_table_index1'/Int8ub)
-        # TODO: predefined value n, ...
+                         name=name, desc=desc, rec_len=(3, 3))
 
+    def _decode_record_bin(self, raw_bin_data : bytes, record_nr : int) -> dict:
+        if record_nr == 1:
+            return parse_construct(self.construct_first, raw_bin_data)
+        else:
+            return parse_construct(self.construct_others, raw_bin_data)
+
+    def _encode_record_bin(self, abstract_data : dict, record_nr : int, **kwargs) -> bytearray:
+        r = None
+        if record_nr == 1:
+            r = self.construct_first.build(abstract_data)
+        else:
+            r = self.construct_others.build(abstract_data)
+        return filter_dict(r)
 
 class EF_DialledVals(TransparentEF):
     """Section 8.6"""
-
-    def __init__(self, fid, name, desc):
-        super().__init__(fid=fid, sfid=None, name=name, desc=desc, size={4, 4})
+    _test_de_encode = [
+        ( "ffffff22", { "next_table_type": "empty", "id_of_next_table": h2b("ffff"), "dialed_digits": "22" } ),
+        ( "f16f8885", { "next_table_type": "predefined", "id_of_next_table": h2b("6f88"), "dialed_digits": "58" }),
+    ]
+    def __init__(self, fid='1234', name='DialledVals', desc=None):
+        super().__init__(fid=fid, sfid=None, name=name, desc=desc, size=(4, 4))
         self._construct = Struct('next_table_type'/NextTableType,
-                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'id_of_next_table'/Bytes(2),
                                  'dialed_digits'/BcdAdapter(Bytes(1)))
 
 
@@ -304,3 +356,15 @@ class DF_EIRENE(CardDF):
                            desc='Free Number Call Type 0 and 8'),
         ]
         self.add_files(files)
+
+
+class AddonGSMR(CardProfileAddon):
+    """An Addon that can be found on either classic GSM SIM or on UICC to support GSM-R."""
+    def __init__(self):
+        files = [
+            DF_EIRENE()
+        ]
+        super().__init__('GSM-R', desc='Railway GSM', files_in_mf=files)
+
+    def probe(self, card: 'CardBase') -> bool:
+        return card.file_exists(self.files_in_mf[0].fid)

pySim/iso7816_4.py

@@ -17,11 +17,9 @@ You should have received a copy of the GNU General Public License
 along with this program.  If not, see <http://www.gnu.org/licenses/>.
 """
 
-from construct import *
-from pySim.construct import *
-from pySim.utils import *
-from pySim.filesystem import *
-from pySim.tlv import *
+from construct import GreedyString
+from osmocom.tlv import *
+from osmocom.construct import *
 
 # Table 91 + Section 8.2.1.2
 class ApplicationId(BER_TLV_IE, tag=0x4f):

pySim/javacard.py

@@ -0,0 +1,142 @@
+# JavaCard related utilities
+#
+# (C) 2024 by sysmocom - s.f.m.c. GmbH
+# All Rights Reserved
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import zipfile
+import struct
+import sys
+import io
+from osmocom.utils import b2h, Hexstr
+from construct import Struct, Array, this, Int32ub, Int16ub, Int8ub
+from osmocom.construct import *
+from osmocom.tlv import *
+from construct import Optional as COptional
+
+def ijc_to_cap(in_file: io.IOBase, out_zip: zipfile.ZipFile, p : str = "foo"):
+    """Convert an ICJ (Interoperable Java Card) file [back] to a CAP file.
+    example usage:
+        with io.open(sys.argv[1],"rb") as f, zipfile.ZipFile(sys.argv[2], "wb") as z:
+        ijc_to_cap(f, z)
+    """
+    TAGS = ["Header", "Directory", "Applet", "Import", "ConstantPool", "Class", "Method", "StaticField", "RefLocation",
+            "Export", "Descriptor", "Debug"]
+    b = in_file.read()
+    while len(b):
+        tag, size = struct.unpack('!BH', b[0:3])
+        out_zip.writestr(p+"/javacard/"+TAGS[tag-1]+".cap", b[0:3+size])
+        b = b[3+size:]
+
+class CapFile():
+
+    # Java Card Platform Virtual Machine Specification, v3.2, section 6.4
+    __header_component_compact = Struct('tag'/Int8ub,
+                                        'size'/Int16ub,
+                                        'magic'/Int32ub,
+                                        'minor_version'/Int8ub,
+                                        'major_version'/Int8ub,
+                                        'flags'/Int8ub,
+                                        'package'/Struct('minor_version'/Int8ub,
+                                                         'major_version'/Int8ub,
+                                                         'AID'/LV),
+                                        'package_name'/COptional(LV)) #since CAP format 2.2
+
+    # Java Card Platform Virtual Machine Specification, v3.2, section 6.6
+    __applet_component_compact = Struct('tag'/Int8ub,
+                                        'size'/Int16ub,
+                                        'count'/Int8ub,
+                                        'applets'/Array(this.count, Struct('AID'/LV,
+                                                                           'install_method_offset'/Int16ub)),
+                                       )
+
+    def __init__(self, filename:str):
+
+        # In this dictionary we will keep all nested .cap file components by their file names (without .cap suffix)
+        # See also: Java Card Platform Virtual Machine Specification, v3.2, section 6.2.1
+        self.__component = {}
+
+        # Extract the nested .cap components from the .cap file
+        # See also: Java Card Platform Virtual Machine Specification, v3.2, section 6.2.1
+        cap = zipfile.ZipFile(filename)
+        cap_namelist = cap.namelist()
+        for i, filename in enumerate(cap_namelist):
+            if filename.lower().endswith('.capx') and not filename.lower().endswith('.capx'):
+                #TODO: At the moment we only support the compact .cap format, add support for the extended .cap format.
+                raise ValueError("incompatible .cap file, extended .cap format not supported!")
+
+            if filename.lower().endswith('.cap'):
+                key = filename.split('/')[-1].removesuffix('.cap')
+                self.__component[key] = cap.read(filename)
+
+        # Make sure that all mandatory components are present
+        # See also: Java Card Platform Virtual Machine Specification, v3.2, section 6.2
+        required_components = {'Header' : 'COMPONENT_Header',
+                               'Directory' : 'COMPONENT_Directory',
+                               'Import' : 'COMPONENT_Import',
+                               'ConstantPool' : 'COMPONENT_ConstantPool',
+                               'Class' : 'COMPONENT_Class',
+                               'Method' : 'COMPONENT_Method',
+                               'StaticField' : 'COMPONENT_StaticField',
+                               'RefLocation' : 'COMPONENT_ReferenceLocation',
+                               'Descriptor' : 'COMPONENT_Descriptor'}
+        for component in required_components:
+            if component not in self.__component.keys():
+                raise ValueError("invalid cap file, %s missing!" % required_components[component])
+
+    def get_loadfile(self) -> bytes:
+        """Get the executable loadfile as hexstring"""
+        # Concatenate all cap file components in the specified order
+        # see also: Java Card Platform Virtual Machine Specification, v3.2, section 6.3
+        loadfile = self.__component['Header']
+        loadfile += self.__component['Directory']
+        loadfile += self.__component['Import']
+        if 'Applet' in self.__component.keys():
+            loadfile += self.__component['Applet']
+        loadfile += self.__component['Class']
+        loadfile += self.__component['Method']
+        loadfile += self.__component['StaticField']
+        if 'Export' in self.__component.keys():
+            loadfile += self.__component['Export']
+        loadfile += self.__component['ConstantPool']
+        loadfile += self.__component['RefLocation']
+        if 'Descriptor' in self.__component.keys():
+            loadfile += self.__component['Descriptor']
+        return loadfile
+
+    def get_loadfile_aid(self) -> Hexstr:
+        """Get the loadfile AID as hexstring"""
+        header = self.__header_component_compact.parse(self.__component['Header'])
+        magic = header['magic'] or 0
+        if magic != 0xDECAFFED:
+            raise ValueError("invalid cap file, COMPONENT_Header lacks magic number (0x%08X!=0xDECAFFED)!" % magic)
+        #TODO: check cap version and make sure we are compatible with it
+        return header['package']['AID']
+
+    def get_applet_aid(self, index:int = 0) -> Hexstr:
+        """Get the applet AID as hexstring"""
+        #To get the module AID, we must look into COMPONENT_Applet. Unfortunately, even though this component should
+        #be present in any .cap file, it is defined as an optional component.
+        if 'Applet' not in self.__component.keys():
+            raise ValueError("can't get the AID, this cap file lacks the optional COMPONENT_Applet component!")
+
+        applet = self.__applet_component_compact.parse(self.__component['Applet'])
+
+        if index > applet['count']:
+            raise ValueError("can't get the AID for applet with index=%u, this .cap file only has %u applets!" %
+                             (index, applet['count']))
+
+        return applet['applets'][index]['AID']
+

pySim/jsonpath.py

@@ -1,8 +1,3 @@
-# coding=utf-8
-import json
-import pprint
-import jsonpath_ng
-
 """JSONpath utility functions as needed within pysim.
 
 As pySim-sell has the ability to represent SIM files as JSON strings,
@@ -10,6 +5,8 @@ adding JSONpath allows us to conveniently modify individual sub-fields
 of a file or record in its JSON representation.
 """
 
+import jsonpath_ng
+
 # (C) 2021 by Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify