transport/bt_rsap.py: fix unknown variable in reset_card()

Change-Id: I50f0f8d9ad30994c4d9693157dfa1a0c52753178

README.md

@@ -1,8 +1,8 @@
-pySim-prog - Utility for programmable SIM/USIM-Cards
+pySim - Read, Write and Browse Programmable SIM/USIM Cards
 ====================================================
 
-This repository contains a Python-language program that can be used
-to program (write) certain fields/parameters on so-called programmable
+This repository contains Python programs that can be used
+to read, program (write) and browse certain fields/parameters on so-called programmable
 SIM/USIM cards.
 
 Such SIM/USIM cards are special cards, which - unlike those issued by
@@ -13,43 +13,68 @@ This is useful particularly if you are running your own cellular
 network, and want to issue your own SIM/USIM cards for that network.
 
 
-Homepage
---------
+Homepage and Manual
+-------------------
 
-The official homepage of the project is
-<http://osmocom.org/projects/pysim/wiki>
+Please visit the [official homepage](https://osmocom.org/projects/pysim/wiki) for usage instructions, manual and examples.
 
-GIT Repository
+Git Repository
 --------------
 
-You can clone from the official libosmocore.git repository using
+You can clone from the official Osmocom  git repository using
+```
+git clone git://git.osmocom.org/pysim.git
+```
 
-	git clone git://git.osmocom.org/pysim.git
-
-There is a cgit interface at <http://git.osmocom.org/pysim/>
+There is a cgit interface at <https://git.osmocom.org/pysim>
 
 
-Dependencies
+Installation
 ------------
 
-pysim requires:
+Please install the following dependencies:
 
-- pyscard
-- serial
-- pytlv (for specific card types)
+ - pyscard
+ - serial
+ - pytlv
+ - cmd2 >= 1.3.0 but < 2.0.0
+ - jsonpath-ng
+ - construct
+ - bidict
+ - gsm0338
 
 Example for Debian:
+```
+apt-get install python3-pyscard python3-serial python3-pip python3-yaml
+pip3 install -r requirements.txt
+```
 
-	apt-get install python3-pyscard python3-serial python3-pip python3-yaml
-	pip3 install pytlv
+After installing all dependencies, the pySim applications ``pySim-read.py``, ``pySim-prog.py`` and ``pySim-shell.py`` may be started directly from the cloned repository.
+
+### Archlinux Package
+
+Archlinux users may install the package ``python-pysim-git``
+[![](https://img.shields.io/aur/version/python-pysim-git)](https://aur.archlinux.org/packages/python-pysim-git)
+from the [Arch User Repository (AUR)](https://aur.archlinux.org).
+The most convenient way is the use of an [AUR Helper](https://wiki.archlinux.org/index.php/AUR_helpers),
+e.g. [yay](https://aur.archlinux.org/packages/yay) or [pacaur](https://aur.archlinux.org/packages/pacaur).
+The following example shows the installation with ``yay``.
+
+```sh
+# Install
+yay -Sy python-pysim-git
+
+# Uninstall
+sudo pacman -Rs python-pysim-git
+```
 
 
 Mailing List
 ------------
 
-There is no separate mailing list for this project.  However,
+There is no separate mailing list for this project. However,
 discussions related to pysim-prog are happening on the
-openbsc@lists.osmocom.org mailing list, please see
+<openbsc@lists.osmocom.org> mailing list, please see
 <https://lists.osmocom.org/mailman/listinfo/openbsc> for subscription
 options and the list archive.
 
@@ -57,41 +82,46 @@ Please observe the [Osmocom Mailing List
 Rules](https://osmocom.org/projects/cellular-infrastructure/wiki/Mailing_List_Rules)
 when posting.
 
+
 Contributing
 ------------
 
 Our coding standards are described at
 <https://osmocom.org/projects/cellular-infrastructure/wiki/Coding_standards>
 
-We are currently accepting patches by e-mail to the above-mentioned
-mailing list.
+We are using a gerrit-based patch review process explained at
+<https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit>
 
-Usage
------
+
+Usage Examples
+--------------
 
  * Program customizable SIMs. Two modes are possible:
 
-  - one where you specify every parameter manually :
-
+  - one where you specify every parameter manually:
+```
 ./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -i <IMSI> -s <ICCID>
+```
 
-
-  - one where they are generated from some minimal set :
-
+  - one where they are generated from some minimal set:
+```
 ./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -z <random_string_of_choice> -j <card_num>
+```
 
-    With <random_string_of_choice> and <card_num>, the soft will generate
-    'predictable' IMSI and ICCID, so make sure you choose them so as not to
-    conflict with anyone. (for eg. your name as <random_string_of_choice> and
-    0 1 2 ... for <card num>).
+With ``<random_string_of_choice>`` and ``<card_num>``, the soft will generate
+'predictable' IMSI and ICCID, so make sure you choose them so as not to
+conflict with anyone. (for e.g. your name as ``<random_string_of_choice>`` and
+0 1 2 ... for ``<card num>``).
 
-  You also need to enter some parameters to select the device :
-   -t TYPE : type of card (supersim, magicsim, fakemagicsim or try 'auto')
-   -d DEV  : Serial port device (default /dev/ttyUSB0)
-   -b BAUD : Baudrate (default 9600)
+You also need to enter some parameters to select the device:
 
- * Interact with SIMs from a python interactive shell (ipython for eg :)
+ -t TYPE : type of card (``supersim``, ``magicsim``, ``fakemagicsim`` or try ``auto``)  
+ -d DEV  : Serial port device (default ``/dev/ttyUSB0``)  
+ -b BAUD : Baudrate (default 9600)  
 
+ * Interact with SIMs from a python interactive shell (e.g. ipython):
+
+```
 from pySim.transport.serial import SerialSimLink
 from pySim.commands import SimCardCommands
 
@@ -105,3 +135,4 @@ print(sc.read_binary(['3f00', '7f20', '6f07']))
 
 	# Run A3/A8
 print(sc.run_gsm('00112233445566778899aabbccddeeff'))
+```

contrib/jenkins.sh

@@ -1,4 +1,10 @@
 #!/bin/sh
+# jenkins build helper script for pysim.  This is how we build on jenkins.osmocom.org
+#
+# environment variables:
+# * WITH_MANUALS: build manual PDFs if set to "1"
+# * PUBLISH: upload manuals after building if set to "1" (ignored without WITH_MANUALS = "1")
+#
 
 set -e
 
@@ -12,8 +18,39 @@ fi
 virtualenv -p python3 venv --system-site-packages
 . venv/bin/activate
 pip install pytlv
-pip install pyyaml
+pip install 'pyyaml>=5.1'
+pip install cmd2==1.5
+pip install jsonpath-ng
+pip install construct
+pip install bidict
+pip install gsm0338
 
+# Execute automatically discovered unit tests first
+python -m unittest discover -v -s tests/
+
+# Run pylint to find potential errors
+# Ignore E1102: not-callable
+#   pySim/filesystem.py: E1102: method is not callable (not-callable)
+# Ignore E0401: import-error
+#   pySim/utils.py:276: E0401: Unable to import 'Crypto.Cipher' (import-error)
+#   pySim/utils.py:277: E0401: Unable to import 'Crypto.Util.strxor' (import-error)
+pip install pylint
+python -m pylint --errors-only \
+	--disable E1102 \
+	--disable E0401 \
+	--enable W0301 \
+	pySim *.py
+
+# attempt to build documentation
+pip install sphinx
+pip install sphinxcontrib-napoleon
+pip3 install -e 'git+https://github.com/osmocom/sphinx-argparse@master#egg=sphinx-argparse'
+(cd docs && make html latexpdf)
+
+if [ "$WITH_MANUALS" = "1" ] && [ "$PUBLISH" = "1" ]; then
+	make -C "docs" publish publish-html
+fi
+
+# run the test with physical cards
 cd pysim-testdata
 ../tests/pysim-test.sh
-

contrib/sim-rest-client.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python3
+#
+# sim-rest-client.py: client program to test the sim-rest-server.py
+#
+# this will generate authentication tuples just like a HLR / HSS
+# and will then send the related challenge to the REST interface
+# of sim-rest-server.py
+#
+# sim-rest-server.py will then contact the SIM card to perform the
+# authentication (just like a 3GPP RAN), and return the results via
+# the REST to sim-rest-client.py.
+#
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import Optional, Dict
+
+import sys
+import argparse
+import secrets
+import requests
+
+from CryptoMobile.Milenage import Milenage
+from CryptoMobile.utils import xor_buf
+
+def unpack48(x:bytes) -> int:
+    """Decode a big-endian 48bit number from binary to integer."""
+    return int.from_bytes(x, byteorder='big')
+
+def pack48(x:int) -> bytes:
+    """Encode a big-endian 48bit number from integer to binary."""
+    return x.to_bytes(48 // 8, byteorder='big')
+
+def milenage_generate(opc:bytes, amf:bytes, k:bytes, sqn:bytes, rand:bytes) -> Dict[str, bytes]:
+    """Generate an MILENAGE Authentication Tuple."""
+    m = Milenage(None)
+    m.set_opc(opc)
+    mac_a = m.f1(k, rand, sqn, amf)
+    res, ck, ik, ak = m.f2345(k, rand)
+
+    # AUTN = (SQN ^ AK) || AMF || MAC
+    sqn_ak = xor_buf(sqn, ak)
+    autn = b''.join([sqn_ak, amf, mac_a])
+
+    return {'res': res, 'ck': ck, 'ik': ik, 'autn': autn}
+
+def milenage_auts(opc:bytes, k:bytes, rand:bytes, auts:bytes) -> Optional[bytes]:
+    """Validate AUTS. If successful, returns SQN_MS"""
+    amf = b'\x00\x00' # TS 33.102 Section 6.3.3
+    m = Milenage(None)
+    m.set_opc(opc)
+    ak = m.f5star(k, rand)
+
+    sqn_ak = auts[:6]
+    sqn = xor_buf(sqn_ak, ak[:6])
+
+    mac_s = m.f1star(k, rand, sqn, amf)
+    if mac_s == auts[6:14]:
+        return sqn
+    else:
+        return False
+
+
+def build_url(suffix:str, base_path="/sim-auth-api/v1") -> str:
+    """Build an URL from global server_host, server_port, BASE_PATH and suffix."""
+    return "http://%s:%u%s%s" % (server_host, server_port, base_path, suffix)
+
+
+def rest_post(suffix:str, js:Optional[dict] = None):
+    """Perform a RESTful POST."""
+    url = build_url(suffix)
+    if verbose:
+        print("POST %s (%s)" % (url, str(js)))
+    resp = requests.post(url, json=js)
+    if verbose:
+        print("-> %s" % (resp))
+    if not resp.ok:
+        print("POST failed")
+    return resp
+
+def rest_get(suffix:str, base_path=None):
+    """Perform a RESTful GET."""
+    url = build_url(suffix, base_path)
+    if verbose:
+        print("GET %s" % url)
+    resp = requests.get(url)
+    if verbose:
+        print("-> %s" % (resp))
+    if not resp.ok:
+        print("GET failed")
+    return resp
+
+
+def main_info(args):
+    resp = rest_get('/slot/%u' % args.slot_nr, base_path="/sim-info-api/v1")
+    if not resp.ok:
+        print("<- ERROR %u: %s" % (resp.status_code, resp.text))
+        sys.exit(1)
+    resp_json = resp.json()
+    print("<- %s" % resp_json)
+
+
+def main_auth(args):
+    #opc = bytes.fromhex('767A662ACF4587EB0C450C6A95540A04')
+    #k = bytes.fromhex('876B2D8D403EE96755BEF3E0A1857EBE')
+    opc = bytes.fromhex(args.opc)
+    k = bytes.fromhex(args.key)
+    amf = bytes.fromhex(args.amf)
+    sqn = bytes.fromhex(args.sqn)
+
+    for i in range(args.count):
+        rand = secrets.token_bytes(16)
+        t = milenage_generate(opc=opc, amf=amf, k=k, sqn=sqn, rand=rand)
+
+        req_json = {'rand': rand.hex(), 'autn': t['autn'].hex()}
+        print("-> %s" % req_json)
+        resp = rest_post('/slot/%u' % args.slot_nr, req_json)
+        if not resp.ok:
+            print("<- ERROR %u: %s" % (resp.status_code, resp.text))
+            break
+        resp_json = resp.json()
+        print("<- %s" % resp_json)
+        if 'synchronisation_failure' in resp_json:
+            auts = bytes.fromhex(resp_json['synchronisation_failure']['auts'])
+            sqn_ms = milenage_auts(opc, k, rand, auts)
+            if sqn_ms is not False:
+                print("SQN_MS = %s" % sqn_ms.hex())
+                sqn_ms_int = unpack48(sqn_ms)
+                # we assume an IND bit-length of 5 here
+                sqn = pack48(sqn_ms_int + (1 << 5))
+            else:
+                raise RuntimeError("AUTS auth failure during re-sync?!?")
+        elif 'successful_3g_authentication' in resp_json:
+            auth_res = resp_json['successful_3g_authentication']
+            assert bytes.fromhex(auth_res['res']) == t['res']
+            assert bytes.fromhex(auth_res['ck']) == t['ck']
+            assert bytes.fromhex(auth_res['ik']) == t['ik']
+            # we assume an IND bit-length of 5 here
+            sqn = pack48(unpack48(sqn) + (1 << 5))
+        else:
+            raise RuntimeError("Auth failure")
+
+
+def main(argv):
+    global server_port, server_host, verbose
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-H", "--host", help="Host to connect to", default="localhost")
+    parser.add_argument("-p", "--port", help="TCP port to connect to", default=8000)
+    parser.add_argument("-v", "--verbose", help="increase output verbosity", action='count', default=0)
+    parser.add_argument("-n", "--slot-nr", help="SIM slot number", type=int, default=0)
+    subp = parser.add_subparsers()
+
+    auth_p = subp.add_parser('auth', help='UMTS AKA Authentication')
+    auth_p.add_argument("-c", "--count", help="Auth count", type=int, default=10)
+    auth_p.add_argument("-k", "--key", help="Secret key K (hex)", type=str, required=True)
+    auth_p.add_argument("-o", "--opc", help="Secret OPc (hex)", type=str, required=True)
+    auth_p.add_argument("-a", "--amf", help="AMF Field (hex)", type=str, default="0000")
+    auth_p.add_argument("-s", "--sqn", help="SQN Field (hex)", type=str, default="000000000000")
+    auth_p.set_defaults(func=main_auth)
+
+    info_p = subp.add_parser('info', help='Information about the Card')
+    info_p.set_defaults(func=main_info)
+
+    args = parser.parse_args()
+    server_host = args.host
+    server_port = args.port
+    verbose = args.verbose
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main(sys.argv)

contrib/sim-rest-server.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+
+# RESTful HTTP service for performing authentication against USIM cards
+#
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import sys
+import argparse
+
+from klein import run, route
+
+from pySim.transport import ApduTracer
+from pySim.transport.pcsc import PcscSimLink
+from pySim.commands import SimCardCommands
+from pySim.cards import UsimCard
+from pySim.exceptions import *
+
+class ApduPrintTracer(ApduTracer):
+    def trace_response(self, cmd, sw, resp):
+        #print("CMD: %s -> RSP: %s %s" % (cmd, sw, resp))
+        pass
+
+def connect_to_card(slot_nr:int):
+    tp = PcscSimLink(slot_nr, apdu_tracer=ApduPrintTracer())
+    tp.connect()
+
+    scc = SimCardCommands(tp)
+    card = UsimCard(scc)
+
+    # this should be part of UsimCard, but FairewavesSIM breaks with that :/
+    scc.cla_byte = "00"
+    scc.sel_ctrl = "0004"
+
+    card.read_aids()
+    card.select_adf_by_aid(adf='usim')
+
+    return tp, scc, card
+
+
+@route('/sim-auth-api/v1/slot/<int:slot>')
+def auth(request, slot):
+    """REST API endpoint for performing authentication against a USIM.
+       Expects a JSON body containing RAND and AUTN.
+       Returns a JSON body containing RES, CK, IK and Kc."""
+    try:
+        # there are two hex-string JSON parameters in the body: rand and autn
+        content = json.loads(request.content.read())
+        rand = content['rand']
+        autn = content['autn']
+    except:
+        request.setResponseCode(400)
+        return "Malformed Request"
+
+    try:
+        tp, scc, card = connect_to_card(slot)
+    except ReaderError:
+        request.setResponseCode(404)
+        return "Specified SIM Slot doesn't exist"
+    except ProtocolError:
+        request.setResponseCode(500)
+        return "Error"
+    except NoCardError:
+        request.setResponseCode(410)
+        return "No SIM card inserted in slot"
+
+    try:
+        card.select_adf_by_aid(adf='usim')
+        res, sw = scc.authenticate(rand, autn)
+    except SwMatchError as e:
+        request.setResponseCode(500)
+        return "Communication Error %s" % e
+
+    tp.disconnect()
+
+    return json.dumps(res, indent=4)
+
+@route('/sim-info-api/v1/slot/<int:slot>')
+def info(request, slot):
+    """REST API endpoint for obtaining information about an USIM.
+    Expects empty body in request.
+    Returns a JSON body containing ICCID, IMSI."""
+
+    try:
+        tp, scc, card = connect_to_card(slot)
+    except ReaderError:
+        request.setResponseCode(404)
+        return "Specified SIM Slot doesn't exist"
+    except ProtocolError:
+        request.setResponseCode(500)
+        return "Error"
+    except NoCardError:
+        request.setResponseCode(410)
+        return "No SIM card inserted in slot"
+
+    try:
+        card.select_adf_by_aid(adf='usim')
+        iccid, sw = card.read_iccid()
+        imsi, sw = card.read_imsi()
+        res = {"imsi": imsi, "iccid": iccid }
+    except SwMatchError as e:
+        request.setResponseCode(500)
+        return "Communication Error %s" % e
+
+    tp.disconnect()
+
+    return json.dumps(res, indent=4)
+
+
+def main(argv):
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-H", "--host", help="Host/IP to bind HTTP to", default="localhost")
+    parser.add_argument("-p", "--port", help="TCP port to bind HTTP to", default=8000)
+    #parser.add_argument("-v", "--verbose", help="increase output verbosity", action='count', default=0)
+
+    args = parser.parse_args()
+
+    run(args.host, args.port)
+
+if __name__ == "__main__":
+    main(sys.argv)

contrib/sim-rest-server.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Osmocom SIM REST server
+
+[Service]
+Type=simple
+# we listen to 0.0.0.0, allowing remote, unauthenticated clients to connect from everywhere!
+ExecStart=/usr/local/src/pysim/contrib/sim-rest-server.py -H 0.0.0.0
+Restart=always
+RestartSec=2
+# this user must be created beforehand; it must have PC/SC access
+User=rest
+
+[Install]
+WantedBy=multi-user.target

docs/Makefile

@@ -0,0 +1,35 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# for osmo-gsm-manuals
+OSMO_GSM_MANUALS_DIR=$(shell pkg-config osmo-gsm-manuals --variable=osmogsmmanualsdir 2>/dev/null)
+OSMO_REPOSITORY = "pysim"
+UPLOAD_FILES = $(BUILDDIR)/latex/osmopysim-usermanual.pdf
+CLEAN_FILES = $(UPLOAD_FILES)
+
+# Put it first so that "make" without argument is like "make help".
+.PHONY: help
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+$(BUILDDIR)/latex/pysim.pdf: latexpdf
+	@/bin/true
+
+publish-html: html
+	rsync -avz -e "ssh -o 'UserKnownHostsFile=$(OSMO_GSM_MANUALS_DIR)/build/known_hosts' -p 48" $(BUILDDIR)/html/ docs@ftp.osmocom.org:web-files/latest/pysim/
+
+# put this before the catch-all below
+include $(OSMO_GSM_MANUALS_DIR)/build/Makefile.common.inc
+
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%:
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,58 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'osmopysim-usermanual'
+copyright = '2009-2021 by Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
+author = 'Sylvain Munaut, Harald Welte, Philipp Maier, Supreeth Herle'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+        "sphinx.ext.autodoc",
+        "sphinxarg.ext",
+        "sphinx.ext.autosectionlabel",
+        "sphinx.ext.napoleon"
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+autoclass_content = 'both'

docs/index.rst

@@ -0,0 +1,50 @@
+.. pysim documentation master file
+
+Welcome to Osmocom pySim
+========================
+
+Introduction
+------------
+
+pySim is a python implementation of various software that helps you with
+managing subscriber identity cards for cellular networks, so-called SIM
+cards.
+
+Many Osmocom (Open Source Mobile Communications) projects relate to operating
+private / custom cellular networks, and provisioning SIM cards for said networks
+is in many cases a requirement to operate such networks.
+
+To make use of most of pySim's features, you will need a `programmable` SIM card,
+i.e. a card where you are the owner/operator and have sufficient credentials (such
+as the `ADM PIN`) in order to write to many if not most of the files on the card.
+
+Such cards are, for example, available from sysmocom, a major contributor to pySim.
+See https://www.sysmocom.de/products/lab/sysmousim/ for more details.
+
+pySim supports classic GSM SIM cards as well as ETSI UICC with 3GPP USIM and ISIM
+applications.  It is easily extensible, so support for additional files, card
+applications, etc. can be added easily by any python developer.  We do encourage you
+to submit your contributions to help this collaborative development project.
+
+pySim consists of several parts:
+
+* a python :ref:`library<pySim library>` containing plenty of objects and methods that can be used for
+  writing custom programs interfacing with SIM cards.
+* the [new] :ref:`interactive pySim-shell command line program<pySim-shell>`
+* the [legacy] :ref:`pySim-prog and pySim-read tools<Legacy tools>`
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   shell
+   legacy
+   library
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/legacy.rst

@@ -0,0 +1,92 @@
+Legacy tools
+============
+
+*legacy tools* are the classic ``pySim-prog`` and ``pySim-read`` programs that
+existed long before ``pySim-shell``.
+
+pySim-prog
+----------
+
+``pySim-prog`` was the first part of the pySim software suite.  It started as
+a tool to write ICCID, IMSI, MSISDN and Ki to very simplistic SIM cards, and
+was later extended to a variety of other cards.  As the number of features supported
+became no longer bearable to express with command-line arguments, `pySim-shell` was
+created.
+
+Basic use cases can still use `pySim-prog`.
+
+Program customizable SIMs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Two modes are possible:
+
+  - one where you specify every parameter manually :
+
+``./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -i <IMSI> -s <ICCID>``
+
+
+  - one where they are generated from some minimal set :
+
+``./pySim-prog.py -n 26C3 -c 49 -x 262 -y 42 -z <random_string_of_choice> -j <card_num>``
+
+    With <random_string_of_choice> and <card_num>, the soft will generate
+    'predictable' IMSI and ICCID, so make sure you choose them so as not to
+    conflict with anyone. (for eg. your name as <random_string_of_choice> and
+    0 1 2 ... for <card num>).
+
+  You also need to enter some parameters to select the device :
+   -t TYPE : type of card (supersim, magicsim, fakemagicsim or try 'auto')
+   -d DEV  : Serial port device (default /dev/ttyUSB0)
+   -b BAUD : Baudrate (default 9600)
+
+
+pySim-read
+----------
+
+``pySim-read`` allows you to read some data from a SIM card.  It will only some files
+of the card, and will only read files accessible to a normal user (without any special authentication)
+
+Specifically, pySim-read will dump the following:
+
+* MF
+
+ * EF.ICCID
+
+* DF.GSM
+
+ * EF,IMSI
+ * EF.GID1
+ * EF.GID2
+ * EF.SMSP
+ * EF.SPN
+ * EF.PLMNsel
+ * EF.PLMNwAcT
+ * EF.OPLMNwAcT
+ * EF.HPLMNAcT
+ * EF.ACC
+ * EF.MSISDN
+ * EF.AD
+ * EF.SST
+
+* ADF.USIM
+
+ * EF.EHPLMN
+ * EF.UST
+ * EF.ePDGId
+ * EF.ePDGSelection
+
+* ADF.ISIM
+
+ * EF.PCSCF
+ * EF.DOMAIN
+ * EF.IMPI
+ * EF.IMPU
+ * EF.UICCIARI
+ * EF.IST
+
+
+pySim-read usage
+~~~~~~~~~~~~~~~~
+
+.. argparse::
+   :module: pySim-read
+   :func: option_parser

docs/library.rst

@@ -0,0 +1,111 @@
+pySim library
+=============
+
+pySim filesystem abstraction
+----------------------------
+
+.. automodule:: pySim.filesystem
+   :members:
+
+pySim commands abstraction
+--------------------------
+
+.. automodule:: pySim.commands
+   :members:
+
+pySim Transport
+---------------
+
+The pySim.transport classes implement specific ways how to
+communicate with a SIM card.  A "transport" provides ways
+to transceive APDUs with the card.
+
+The most commonly used transport uses the PC/SC interface to
+utilize a variety of smart card interfaces ("readers").
+
+Transport base class
+~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: pySim.transport
+   :members:
+
+
+calypso / OsmocomBB transport
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This allows the use of the SIM slot of an OsmocomBB compatible phone with the TI Calypso chipset,
+using the L1CTL interface to talk to the layer1.bin firmware on the phone.
+
+.. automodule:: pySim.transport.calypso
+   :members:
+
+
+AT-command Modem transport
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This transport uses AT commands of a cellular modem in order to get access to the SIM card inserted
+in such a modem.
+
+.. automodule:: pySim.transport.modem_atcmd
+   :members:
+
+
+PC/SC transport
+~~~~~~~~~~~~~~~
+
+PC/SC is the standard API for accessing smart card interfaces
+on all major operating systems, including the MS Windows Family,
+OS X as well as Linux / Unix OSs.
+
+.. automodule:: pySim.transport.pcsc
+   :members:
+
+
+Serial/UART transport
+~~~~~~~~~~~~~~~~~~~~~
+
+This transport implements interfacing smart cards via
+very simplistic UART readers.  These readers basically
+wire together the Rx+Tx pins of a RS232 UART, provide
+a fixed crystal oscillator for clock, and operate the UART
+at 9600 bps.  These readers are sometimes called `Phoenix`.
+
+.. automodule:: pySim.transport.serial
+   :members:
+
+
+pySim construct utilities
+-------------------------
+
+.. automodule:: pySim.construct
+   :members:
+
+pySim TLV utilities
+-------------------
+
+.. automodule:: pySim.tlv
+   :members:
+
+pySim utility functions
+-----------------------
+
+.. automodule:: pySim.utils
+   :members:
+
+pySim exceptions
+----------------
+
+.. automodule:: pySim.exceptions
+   :members:
+
+pySim card_handler
+------------------
+
+.. automodule:: pySim.card_handler
+   :members:
+
+pySim card_key_provider
+-----------------------
+
+.. automodule:: pySim.card_key_provider
+   :members:

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/shell.rst

@@ -0,0 +1,581 @@
+pySim-shell
+===========
+
+pySim-shell is an interactive command line shell for all kind of interactions with SIM cards.
+
+The interactive shell provides command for
+
+* navigating the on-card filesystem hierarchy
+* authenticating with PINs such as ADM1
+* CHV/PIN management (VERIFY, ENABLE, DISABLE, UNBLOCK)
+* decoding of SELECT response (file control parameters)
+* reading and writing of files and records in raw, hex-encoded binary format
+* for some files where related support has been developed:
+
+ * decoded reading (display file data in JSON format)
+ * decoded writing (encode from JSON to binary format, then write)
+
+By means of using the python ``cmd2`` module, various useful features improve usability:
+
+* history of commands (persistent across restarts)
+* output re-direction to files on your computer
+* output piping through external tools like 'grep'
+* tab completion of commands and SELECT-able files/directories
+* interactive help for all commands
+
+Running pySim-shell
+-------------------
+
+pySim-shell has a variety of command line arguments to control
+
+* which transport to use (how to use a reader to talk to the SIM card)
+* whether to automatically verify an ADM pin (and in which format)
+* whether to execute a start-up script
+
+.. argparse::
+   :module: pySim-shell
+   :func: option_parser
+
+
+
+cmd2 basics
+-----------
+
+FIXME
+
+
+
+ISO7816 commands
+----------------
+
+This category of commands relates to commands that originate in the ISO 7861-4 specifications,
+most of them have a 1:1 resemblance in the specification.
+
+select
+~~~~~~
+
+The ``select`` command is used to select a file, either by its FID, AID or by its symbolic name.
+
+Try ``select`` with tab-completion to get a list of all current selectable items:
+
+::
+
+  pySIM-shell (MF)> select
+  ..                2fe2              a0000000871004    EF.ARR            MF
+  2f00              3f00              ADF.ISIM          EF.DIR
+  2f05              7f10              ADF.USIM          EF.ICCID
+  2f06              7f20              DF.GSM            EF.PL
+  2f08              a0000000871002    DF.TELECOM        EF.UMPC
+
+Use ``select`` with a specific FID or name to select the new file.
+
+This will
+
+* output the [JSON decoded, if possible] select response
+* change the prompt to the newly selected file
+* enable any commands specific to the newly-selected file
+
+::
+
+  pySIM-shell (MF)> select ADF.USIM
+  {
+      "file_descriptor": {
+          "shareable": true,
+          "file_type": "df",
+          "structure": "no_info_given"
+      },
+      "df_name": "A0000000871002FFFFFFFF8907090000",
+      "proprietary_info": {
+          "uicc_characteristics": "71",
+          "available_memory": 101640
+      },
+      "life_cycle_status_int": "operational_activated",
+      "security_attrib_compact": "00",
+      "pin_status_template_do": "90017083010183018183010A83010B"
+  }
+  pySIM-shell (MF/ADF.USIM)>
+
+
+
+change_chv
+~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.change_chv_parser
+
+
+disable_chv
+~~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.disable_chv_parser
+
+
+enable_chv
+~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.enable_chv_parser
+
+
+unblock_chv
+~~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.unblock_chv_parser
+
+
+verify_chv
+~~~~~~~~~~
+This command allows you to verify a CHV (PIN), which is how the specifications call
+it if you authenticate yourself with the said CHV/PIN.
+
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.verify_chv_parser
+
+deactivate_file
+~~~~~~~~~~~~~~~
+Deactivate the currently selected file.  This used to be called INVALIDATE in TS 11.11.
+
+
+activate_file
+~~~~~~~~~~~~~
+Activate the currently selected file.  This used to be called REHABILITATE in TS 11.11.
+
+open_channel
+~~~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.open_chan_parser
+
+close_channel
+~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.close_chan_parser
+
+
+suspend_uicc
+~~~~~~~~~~~~
+This command allows you to perform the SUSPEND UICC command on the card.  This is a relatively
+recent power-saving addition to the UICC specifications, allowing for suspend/resume while maintaining
+state, as opposed to a full power-off (deactivate) and power-on (activate) of the card.
+
+The pySim command just sends that SUSPEND UICC command and doesn't perform the full related sequence
+including the electrical power down.
+
+.. argparse::
+   :module: pySim-shell
+   :func: Iso7816Commands.suspend_uicc_parser
+
+
+pySim commands
+--------------
+
+Commands in this category are pySim specific; they do not have a 1:1 correspondence to ISO 7816
+or 3GPP commands. Mostly they will operate either only on local (in-memory) state, or execute
+a complex sequence of card-commands.
+
+desc
+~~~~
+
+Display human readable file description for the currently selected file.
+
+
+dir
+~~~
+.. argparse::
+   :module: pySim-shell
+   :func: PySimCommands.dir_parser
+
+
+export
+~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: PySimCommands.export_parser
+
+Please note that `export` works relative to the current working
+directory, so if you are in `MF`, then the export will contain all known
+files on the card.  However, if you are in `ADF.ISIM`, only files below
+that ADF will be part of the export.
+
+Furthermore, it is strongly advised to first enter the ADM1 pin
+(`verify_adm`) to maximize the chance of having permission to read
+all/most files.
+
+
+tree
+~~~~
+
+Display a tree of the card filesystem.  It is important to note that this displays a tree
+of files that might potentially exist (based on the card profile).  In order to determine if
+a given file really exists on a given card, you have to try to select that file.
+
+
+verify_adm
+~~~~~~~~~~
+
+Verify the ADM (Administrator) PIN specified as argument.  This is typically needed in order
+to get write/update permissions to most of the files on SIM cards.
+
+Currently only ADM1 is supported.
+
+
+reset
+~~~~~
+Perform card reset and display the card ATR.
+
+intro
+~~~~~
+[Re-]Display the introductory banner
+
+
+equip
+~~~~~
+Equip pySim-shell with a card; particularly useful if the program was
+started before a card was present, or after a card has been replaced by
+the user while pySim-shell was kept running.
+
+bulk_script
+~~~~~~~~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: PysimApp.bulk_script_parser
+
+Run a script for bulk-provisioning of multiple cards.
+
+
+echo
+~~~~
+.. argparse::
+   :module: pySim-shell
+   :func: PysimApp.echo_parser
+
+
+
+Linear Fixed EF commands
+------------------------
+
+These commands become enabled only when your currently selected file is of *Linear Fixed EF* type.
+
+read_record
+~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.read_rec_parser
+
+
+read_record_decoded
+~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.read_rec_dec_parser
+
+
+read_records
+~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.read_recs_parser
+
+
+read_records_decoded
+~~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.read_recs_dec_parser
+
+
+update_record
+~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.upd_rec_parser
+
+
+update_record_decoded
+~~~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.upd_rec_dec_parser
+
+
+edit_record_decoded
+~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: LinFixedEF.ShellCommands.edit_rec_dec_parser
+
+This command will read the selected record, decode it to its JSON representation, save
+that JSON to a temporary file on your computer, and launch your configured text editor.
+
+You may then perform whatever modifications to the JSON representation, save + leave your
+text editor.
+
+Afterwards, the modified JSON will be re-encoded to the binary format, and the result written
+back to the record on the SIM card.
+
+This allows for easy interactive modification of records.
+
+
+
+Transparent EF commands
+-----------------------
+
+These commands become enabled only when your currently selected file is of *Transparent EF* type.
+
+
+read_binary
+~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: TransparentEF.ShellCommands.read_bin_parser
+
+
+read_binary_decoded
+~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: TransparentEF.ShellCommands.read_bin_dec_parser
+
+
+update_binary
+~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: TransparentEF.ShellCommands.upd_bin_parser
+
+
+update_binary_decoded
+~~~~~~~~~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: TransparentEF.ShellCommands.upd_bin_dec_parser
+
+In normal operation, update_binary_decoded needs a JSON document representing the entire file contents as
+input.  This can be inconvenient if you want to keep 99% of the content but just toggle one specific
+parameter.   That's where the JSONpath support comes in handy:  You can specify a JSONpath to an element
+inside the document as well as a new value for tat field:
+
+Th below example demonstrates this by modifying the ofm field within EF.AD:
+
+::
+
+  pySIM-shell (MF/ADF.USIM/EF.AD)> read_binary_decoded
+  {
+      "ms_operation_mode": "normal",
+      "specific_facilities": {
+          "ofm": true
+      },
+      "len_of_mnc_in_imsi": 2
+  }
+  pySIM-shell (MF/ADF.USIM/EF.AD)> update_binary_decoded --json-path specific_facilities.ofm false
+  pySIM-shell (MF/ADF.USIM/EF.AD)> read_binary_decoded
+  {
+      "ms_operation_mode": "normal",
+      "specific_facilities": {
+          "ofm": false
+      },
+      "len_of_mnc_in_imsi": 2
+  }
+
+
+edit_binary_decoded
+~~~~~~~~~~~~~~~~~~~
+This command will read the selected binary EF, decode it to its JSON representation, save
+that JSON to a temporary file on your computer, and launch your configured text editor.
+
+You may then perform whatever modifications to the JSON representation, save + leave your
+text editor.
+
+Afterwards, the modified JSON will be re-encoded to the binary format, and the result written
+to the SIM card.
+
+This allows for easy interactive modification of file contents.
+
+
+
+BER-TLV EF commands
+-------------------
+
+BER-TLV EFs are files that contain BER-TLV structured data.  Every file can contain any number
+of variable-length IEs (DOs).  The tag within a BER-TLV EF must be unique within the file.
+
+The commands below become enabled only when your currently selected file is of *BER-TLV EF* type.
+
+retrieve_tags
+~~~~~~~~~~~~~
+
+Retrieve a list of all tags present in the currently selected file.
+
+
+retrieve_data
+~~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: BerTlvEF.ShellCommands.retrieve_data_parser
+
+
+set_data
+~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: BerTlvEF.ShellCommands.set_data_parser
+
+
+del_data
+~~~~~~~~
+.. argparse::
+   :module: pySim.filesystem
+   :func: BerTlvEF.ShellCommands.del_data_parser
+
+
+
+USIM commands
+-------------
+
+authenticate
+~~~~~~~~~~~~
+.. argparse::
+   :module: pySim.ts_31_102
+   :func: ADF_USIM.AddlShellCommands.authenticate_parser
+
+
+ARA-M commands
+--------------
+
+The ARA-M commands exist to manage the access rules stored in an ARA-M applet on the card.
+
+ARA-M in the context of SIM cards is primarily used to enable Android UICC Carrier Privileges,
+please see https://source.android.com/devices/tech/config/uicc for more details on the background.
+
+
+aram_get_all
+~~~~~~~~~~~~
+
+Obtain and decode all access rules from the ARA-M applet on the card.
+
+NOTE: if the total size of the access rules exceeds 255 bytes, this command will fail, as
+it doesn't yet implement fragmentation/reassembly on rule retrieval. YMMV
+
+::
+
+  pySIM-shell (MF/ADF.ARA-M)> aram_get_all
+  [
+      {
+          "ResponseAllRefArDO": [
+              {
+                  "RefArDO": [
+                      {
+                          "RefDO": [
+                              {
+                                  "AidRefDO": "ffffffffffff"
+                              },
+                              {
+                                  "DevAppIdRefDO": "e46872f28b350b7e1f140de535c2a8d5804f0be3"
+                              }
+                          ]
+                      },
+                      {
+                          "ArDO": [
+                              {
+                                  "ApduArDO": {
+                                      "generic_access_rule": "always"
+                                  }
+                              },
+                              {
+                                  "PermArDO": {
+                                      "permissions": "0000000000000001"
+                                  }
+                              }
+                          ]
+                      }
+                  ]
+              }
+          ]
+      }
+  ]
+
+aram_get_config
+~~~~~~~~~~~~~~~
+Perform Config handshake with ARA-M applet: Tell it our version and retrieve its version.
+
+NOTE: Not supported in all ARA-M implementations.
+
+.. argparse::
+   :module: pySim.ara_m
+   :func: ADF_ARAM.AddlShellCommands.get_config_parser
+
+
+aram_store_ref_ar_do
+~~~~~~~~~~~~~~~~~~~~
+Store a [new] access rule on the ARA-M applet.
+
+.. argparse::
+   :module: pySim.ara_m
+   :func: ADF_ARAM.AddlShellCommands.store_ref_ar_do_parse
+
+For example, to store an Android UICC carrier privilege rule for the SHA1 hash of the certificate used to sign the CoIMS android app of Supreeth Herle (https://github.com/herlesupreeth/CoIMS_Wiki) you can use the following command:
+
+::
+
+  pySIM-shell (MF/ADF.ARA-M)> aram_store_ref_ar_do --aid FFFFFFFFFFFF --device-app-id E46872F28B350B7E1F140DE535C2A8D5804F0BE3 --android-permissions 0000000000000001 --apdu-always
+
+
+aram_delete_all
+~~~~~~~~~~~~~~~
+This command will request deletion of all access rules stored within the
+ARA-M applet.  Use it with caution, there is no undo.  Any rules later
+intended must be manually inserted again using `aram_store_ref_ar_do`
+
+
+
+cmd2 settable parameters
+------------------------
+
+``cmd2`` has the concept of *settable parameters* which act a bit like environment variables in an OS-level
+shell: They can be read and set, and they will influence the behavior somehow.
+
+conserve_write
+~~~~~~~~~~~~~~
+
+If enabled, pySim will (when asked to write to a card) always first read the respective file/record and
+verify if the to-be-written value differs from the current on-card value.  If not, the write will be skipped.
+Writes will only be performed if the new value is different from the current on-card value.
+
+If disabled, pySim will always write irrespective of the current/new value.
+
+json_pretty_print
+~~~~~~~~~~~~~~~~~
+
+This parameter determines if generated JSON output should (by default) be pretty-printed (multi-line
+output with indent level of 4 spaces) or not.
+
+The default value of this parameter is 'true'.
+
+debug
+~~~~~
+
+If enabled, full python back-traces will be displayed in case of exceptions
+
+apdu_trace
+~~~~~~~~~~
+
+Boolean variable that determines if a hex-dump of the command + response APDU shall be printed.
+
+numeric_path
+~~~~~~~~~~~~
+
+Boolean variable that determines if path (e.g. in prompt) is displayed with numeric FIDs or string names.
+
+::
+
+  pySIM-shell (MF/EF.ICCID)> set numeric_path True
+  numeric_path - was: False
+  now: True
+  pySIM-shell (3f00/2fe2)> set numeric_path False
+  numeric_path - was: True
+  now: False
+  pySIM-shell (MF/EF.ICCID)> help set

pySim-read.py

@@ -23,344 +23,340 @@
 #
 
 import hashlib
-from optparse import OptionParser
+import argparse
 import os
 import random
 import re
 import sys
-from pySim.ts_51_011 import EF, DF, EF_SST_map, EF_AD_mode_map
+from pySim.ts_51_011 import EF, DF, EF_SST_map, EF_AD
 from pySim.ts_31_102 import EF_UST_map, EF_USIM_ADF_map
 from pySim.ts_31_103 import EF_IST_map, EF_ISIM_ADF_map
 
 from pySim.commands import SimCardCommands
-from pySim.cards import card_detect, Card, UsimCard, IsimCard
+from pySim.transport import init_reader, argparse_add_reader_args
+from pySim.exceptions import SwMatchError
+from pySim.cards import card_detect, SimCard, UsimCard, IsimCard
 from pySim.utils import h2b, swap_nibbles, rpad, dec_imsi, dec_iccid, dec_msisdn
-from pySim.utils import format_xplmn_w_act, dec_spn, dec_st, init_reader, dec_addr_tlv
+from pySim.utils import format_xplmn_w_act, dec_st
 from pySim.utils import h2s, format_ePDGSelection
 
-def parse_options():
+option_parser = argparse.ArgumentParser(prog='pySim-read',
+                                        description='Legacy tool for reading some parts of a SIM card',
+                                        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+argparse_add_reader_args(option_parser)
 
-	parser = OptionParser(usage="usage: %prog [options]")
 
-	parser.add_option("-d", "--device", dest="device", metavar="DEV",
-			help="Serial Device for SIM access [default: %default]",
-			default="/dev/ttyUSB0",
-		)
-	parser.add_option("-b", "--baud", dest="baudrate", type="int", metavar="BAUD",
-			help="Baudrate used for SIM access [default: %default]",
-			default=9600,
-		)
-	parser.add_option("-p", "--pcsc-device", dest="pcsc_dev", type='int', metavar="PCSC",
-			help="Which PC/SC reader number for SIM access",
-			default=None,
-		)
-	parser.add_option("--modem-device", dest="modem_dev", metavar="DEV",
-			help="Serial port of modem for Generic SIM Access (3GPP TS 27.007)",
-			default=None,
-		)
-	parser.add_option("--modem-baud", dest="modem_baud", type="int", metavar="BAUD",
-			help="Baudrate used for modem's port [default: %default]",
-			default=115200,
-		)
-	parser.add_option("--osmocon", dest="osmocon_sock", metavar="PATH",
-			help="Socket path for Calypso (e.g. Motorola C1XX) based reader (via OsmocomBB)",
-			default=None,
-		)
+def select_app(adf: str, card: SimCard):
+    """Select application by its AID"""
+    sw = 0
+    try:
+        if card._scc.cla_byte == "00":
+            data, sw = card.select_adf_by_aid(adf)
+    except SwMatchError as e:
+        if e.sw_actual == "6a82":
+            # If we can't select the file because it does not exist, we just remain silent since it means
+            # that this card just does not have an USIM application installed, which is not an error.
+            pass
+        else:
+            print("ADF." + adf + ": Can't select application -- " + str(e))
+    except Exception as e:
+        print("ADF." + adf + ": Can't select application -- " + str(e))
 
-	(options, args) = parser.parse_args()
-
-	if args:
-		parser.error("Extraneous arguments")
-
-	return options
+    return sw
 
 
 if __name__ == '__main__':
 
-	# Parse options
-	opts = parse_options()
+    # Parse options
+    opts = option_parser.parse_args()
 
-	# Init card reader driver
-	sl = init_reader(opts)
-	if sl is None:
-		exit(1)
+    # Init card reader driver
+    sl = init_reader(opts)
+    if sl is None:
+        exit(1)
 
-	# Create command layer
-	scc = SimCardCommands(transport=sl)
+    # Create command layer
+    scc = SimCardCommands(transport=sl)
 
-	# Wait for SIM card
-	sl.wait_for_card()
+    # Wait for SIM card
+    sl.wait_for_card()
 
-	# Assuming UICC SIM
-	scc.cla_byte = "00"
-	scc.sel_ctrl = "0004"
+    # Assuming UICC SIM
+    scc.cla_byte = "00"
+    scc.sel_ctrl = "0004"
 
-	# Testing for Classic SIM or UICC
-	(res, sw) = sl.send_apdu(scc.cla_byte + "a4" + scc.sel_ctrl + "02" + "3f00")
-	if sw == '6e00':
-		# Just a Classic SIM
-		scc.cla_byte = "a0"
-		scc.sel_ctrl = "0000"
+    # Testing for Classic SIM or UICC
+    (res, sw) = sl.send_apdu(scc.cla_byte + "a4" + scc.sel_ctrl + "02" + "3f00")
+    if sw == '6e00':
+        # Just a Classic SIM
+        scc.cla_byte = "a0"
+        scc.sel_ctrl = "0000"
 
-	# Program the card
-	print("Reading ...")
+    # Read the card
+    print("Reading ...")
 
-	# Initialize Card object by auto detecting the card
-	card = card_detect("auto", scc) or Card(scc)
+    # Initialize Card object by auto detecting the card
+    card = card_detect("auto", scc) or SimCard(scc)
 
-	# Read all AIDs on the UICC
-	card.read_aids()
+    # Read all AIDs on the UICC
+    card.read_aids()
 
-	# EF.ICCID
-	(res, sw) = card.read_iccid()
-	if sw == '9000':
-		print("ICCID: %s" % (res,))
-	else:
-		print("ICCID: Can't read, response code = %s" % (sw,))
+    # EF.ICCID
+    (res, sw) = card.read_iccid()
+    if sw == '9000':
+        print("ICCID: %s" % (res,))
+    else:
+        print("ICCID: Can't read, response code = %s" % (sw,))
 
-	# EF.IMSI
-	(res, sw) = card.read_imsi()
-	if sw == '9000':
-		print("IMSI: %s" % (res,))
-	else:
-		print("IMSI: Can't read, response code = %s" % (sw,))
+    # EF.IMSI
+    (res, sw) = card.read_imsi()
+    if sw == '9000':
+        print("IMSI: %s" % (res,))
+    else:
+        print("IMSI: Can't read, response code = %s" % (sw,))
 
-	# EF.GID1
-	try:
-		(res, sw) = card.read_gid1()
-		if sw == '9000':
-			print("GID1: %s" % (res,))
-		else:
-			print("GID1: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("GID1: Can't read file -- %s" % (str(e),))
+    # EF.GID1
+    try:
+        (res, sw) = card.read_gid1()
+        if sw == '9000':
+            print("GID1: %s" % (res,))
+        else:
+            print("GID1: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("GID1: Can't read file -- %s" % (str(e),))
 
-	# EF.GID2
-	try:
-		(res, sw) = card.read_binary('GID2')
-		if sw == '9000':
-			print("GID2: %s" % (res,))
-		else:
-			print("GID2: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("GID2: Can't read file -- %s" % (str(e),))
+    # EF.GID2
+    try:
+        (res, sw) = card.read_binary('GID2')
+        if sw == '9000':
+            print("GID2: %s" % (res,))
+        else:
+            print("GID2: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("GID2: Can't read file -- %s" % (str(e),))
 
-	# EF.SMSP
-	(res, sw) = card.read_record('SMSP', 1)
-	if sw == '9000':
-		print("SMSP: %s" % (res,))
-	else:
-		print("SMSP: Can't read, response code = %s" % (sw,))
+    # EF.SMSP
+    (res, sw) = card.read_record('SMSP', 1)
+    if sw == '9000':
+        print("SMSP: %s" % (res,))
+    else:
+        print("SMSP: Can't read, response code = %s" % (sw,))
 
-	# EF.SPN
-	try:
-		(res, sw) = card.read_spn()
-		if sw == '9000':
-			print("SPN: %s" % (res[0] or "Not available"))
-			print("Display HPLMN: %s" % (res[1],))
-			print("Display OPLMN: %s" % (res[2],))
-		else:
-			print("SPN: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("SPN: Can't read file -- %s" % (str(e),))
+    # EF.SPN
+    try:
+        (res, sw) = card.read_spn()
+        if sw == '9000':
+            print("SPN: %s" % (res[0] or "Not available"))
+            print("Show in HPLMN: %s" % (res[1],))
+            print("Hide in OPLMN: %s" % (res[2],))
+        else:
+            print("SPN: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("SPN: Can't read file -- %s" % (str(e),))
 
-	# EF.PLMNsel
-	try:
-		(res, sw) = card.read_binary('PLMNsel')
-		if sw == '9000':
-			print("PLMNsel: %s" % (res))
-		else:
-			print("PLMNsel: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("PLMNsel: Can't read file -- " + str(e))
+    # EF.PLMNsel
+    try:
+        (res, sw) = card.read_binary('PLMNsel')
+        if sw == '9000':
+            print("PLMNsel: %s" % (res))
+        else:
+            print("PLMNsel: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("PLMNsel: Can't read file -- " + str(e))
 
-	# EF.PLMNwAcT
-	try:
-		(res, sw) = card.read_plmn_act()
-		if sw == '9000':
-			print("PLMNwAcT:\n%s" % (res))
-		else:
-			print("PLMNwAcT: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("PLMNwAcT: Can't read file -- " + str(e))
+    # EF.PLMNwAcT
+    try:
+        (res, sw) = card.read_plmn_act()
+        if sw == '9000':
+            print("PLMNwAcT:\n%s" % (res))
+        else:
+            print("PLMNwAcT: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("PLMNwAcT: Can't read file -- " + str(e))
 
-	# EF.OPLMNwAcT
-	try:
-		(res, sw) = card.read_oplmn_act()
-		if sw == '9000':
-			print("OPLMNwAcT:\n%s" % (res))
-		else:
-			print("OPLMNwAcT: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("OPLMNwAcT: Can't read file -- " + str(e))
+    # EF.OPLMNwAcT
+    try:
+        (res, sw) = card.read_oplmn_act()
+        if sw == '9000':
+            print("OPLMNwAcT:\n%s" % (res))
+        else:
+            print("OPLMNwAcT: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("OPLMNwAcT: Can't read file -- " + str(e))
 
-	# EF.HPLMNAcT
-	try:
-		(res, sw) = card.read_hplmn_act()
-		if sw == '9000':
-			print("HPLMNAcT:\n%s" % (res))
-		else:
-			print("HPLMNAcT: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("HPLMNAcT: Can't read file -- " + str(e))
+    # EF.HPLMNAcT
+    try:
+        (res, sw) = card.read_hplmn_act()
+        if sw == '9000':
+            print("HPLMNAcT:\n%s" % (res))
+        else:
+            print("HPLMNAcT: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("HPLMNAcT: Can't read file -- " + str(e))
 
-	# EF.ACC
-	(res, sw) = card.read_binary('ACC')
-	if sw == '9000':
-		print("ACC: %s" % (res,))
-	else:
-		print("ACC: Can't read, response code = %s" % (sw,))
+    # EF.ACC
+    (res, sw) = card.read_binary('ACC')
+    if sw == '9000':
+        print("ACC: %s" % (res,))
+    else:
+        print("ACC: Can't read, response code = %s" % (sw,))
 
-	# EF.MSISDN
-	try:
-		(res, sw) = card.read_msisdn()
-		if sw == '9000':
-			# (npi, ton, msisdn) = res
-			if res is not None:
-				print("MSISDN (NPI=%d ToN=%d): %s" % res)
-			else:
-				print("MSISDN: Not available")
-		else:
-			print("MSISDN: Can't read, response code = %s" % (sw,))
-	except Exception as e:
-		print("MSISDN: Can't read file -- " + str(e))
+    # EF.MSISDN
+    try:
+        (res, sw) = card.read_msisdn()
+        if sw == '9000':
+            # (npi, ton, msisdn) = res
+            if res is not None:
+                print("MSISDN (NPI=%d ToN=%d): %s" % res)
+            else:
+                print("MSISDN: Not available")
+        else:
+            print("MSISDN: Can't read, response code = %s" % (sw,))
+    except Exception as e:
+        print("MSISDN: Can't read file -- " + str(e))
 
-	# EF.AD
-	(res, sw) = card.read_binary('AD')
-	if sw == '9000':
-		print("Administrative data: %s" % (res,))
-		if res[:2] in EF_AD_mode_map:
-			print("\tMS operation mode: %s" % (EF_AD_mode_map[res[:2]],))
-		else:
-			print("\tMS operation mode: (unknown 0x%s)" % (res[:2],))
-		if int(res[4:6], 16) & 0x01:
-			print("\tCiphering Indicator: enabled")
-		else:
-			print("\tCiphering Indicator: disabled")
-	else:
-		print("AD: Can't read, response code = %s" % (sw,))
+    # EF.AD
+    (res, sw) = card.read_binary('AD')
+    if sw == '9000':
+        print("Administrative data: %s" % (res,))
+        ad = EF_AD()
+        decoded_data = ad.decode_hex(res)
+        print("\tMS operation mode: %s" % decoded_data['ms_operation_mode'])
+        if decoded_data['ofm']:
+            print("\tCiphering Indicator: enabled")
+        else:
+            print("\tCiphering Indicator: disabled")
+    else:
+        print("AD: Can't read, response code = %s" % (sw,))
 
-	# EF.SST
-	(res, sw) = card.read_binary('SST')
-	if sw == '9000':
-		print("SIM Service Table: %s" % res)
-		# Print those which are available
-		print("%s" % dec_st(res))
-	else:
-		print("SIM Service Table: Can't read, response code = %s" % (sw,))
+    # EF.SST
+    (res, sw) = card.read_binary('SST')
+    if sw == '9000':
+        print("SIM Service Table: %s" % res)
+        # Print those which are available
+        print("%s" % dec_st(res))
+    else:
+        print("SIM Service Table: Can't read, response code = %s" % (sw,))
 
-	# Check whether we have th AID of USIM, if so select it by its AID
-	# EF.UST - File Id in ADF USIM : 6f38
-	if '9000' == card.select_adf_by_aid():
-		# Select USIM profile
-		usim_card = UsimCard(scc)
+    # Check whether we have th AID of USIM, if so select it by its AID
+    # EF.UST - File Id in ADF USIM : 6f38
+    sw = select_app("USIM", card)
+    if sw == '9000':
+        # Select USIM profile
+        usim_card = UsimCard(scc)
 
-		# EF.EHPLMN
-		if usim_card.file_exists(EF_USIM_ADF_map['EHPLMN']):
-			(res, sw) = usim_card.read_ehplmn()
-			if sw == '9000':
-				print("EHPLMN:\n%s" % (res))
-			else:
-				print("EHPLMN: Can't read, response code = %s" % (sw,))
+        # EF.EHPLMN
+        if usim_card.file_exists(EF_USIM_ADF_map['EHPLMN']):
+            (res, sw) = usim_card.read_ehplmn()
+            if sw == '9000':
+                print("EHPLMN:\n%s" % (res))
+            else:
+                print("EHPLMN: Can't read, response code = %s" % (sw,))
 
-		# EF.UST
-		try:
-			if usim_card.file_exists(EF_USIM_ADF_map['UST']):
-				# res[0] - EF content of UST
-				# res[1] - Human readable format of services marked available in UST
-				(res, sw) = usim_card.read_ust()
-				if sw == '9000':
-					print("USIM Service Table: %s" % res[0])
-					print("%s" % res[1])
-				else:
-					print("USIM Service Table: Can't read, response code = %s" % (sw,))
-		except Exception as e:
-			print("USIM Service Table: Can't read file -- " + str(e))
+        # EF.UST
+        try:
+            if usim_card.file_exists(EF_USIM_ADF_map['UST']):
+                # res[0] - EF content of UST
+                # res[1] - Human readable format of services marked available in UST
+                (res, sw) = usim_card.read_ust()
+                if sw == '9000':
+                    print("USIM Service Table: %s" % res[0])
+                    print("%s" % res[1])
+                else:
+                    print("USIM Service Table: Can't read, response code = %s" % (sw,))
+        except Exception as e:
+            print("USIM Service Table: Can't read file -- " + str(e))
 
-		#EF.ePDGId - Home ePDG Identifier
-		try:
-			if usim_card.file_exists(EF_USIM_ADF_map['ePDGId']):
-				(res, sw) = usim_card.read_epdgid()
-				if sw == '9000':
-					print("ePDGId:\n%s" % (len(res) and res or '\tNot available\n',))
-				else:
-					print("ePDGId: Can't read, response code = %s" % (sw,))
-		except Exception as e:
-			print("ePDGId: Can't read file -- " + str(e))
+        # EF.ePDGId - Home ePDG Identifier
+        try:
+            if usim_card.file_exists(EF_USIM_ADF_map['ePDGId']):
+                (res, sw) = usim_card.read_epdgid()
+                if sw == '9000':
+                    print("ePDGId:\n%s" %
+                          (len(res) and res or '\tNot available\n',))
+                else:
+                    print("ePDGId: Can't read, response code = %s" % (sw,))
+        except Exception as e:
+            print("ePDGId: Can't read file -- " + str(e))
 
-		#EF.ePDGSelection - ePDG Selection Information
-		try:
-			if usim_card.file_exists(EF_USIM_ADF_map['ePDGSelection']):
-				(res, sw) = usim_card.read_ePDGSelection()
-				if sw == '9000':
-					print("ePDGSelection:\n%s" % (res,))
-				else:
-					print("ePDGSelection: Can't read, response code = %s" % (sw,))
-		except Exception as e:
-			print("ePDGSelection: Can't read file -- " + str(e))
+        # EF.ePDGSelection - ePDG Selection Information
+        try:
+            if usim_card.file_exists(EF_USIM_ADF_map['ePDGSelection']):
+                (res, sw) = usim_card.read_ePDGSelection()
+                if sw == '9000':
+                    print("ePDGSelection:\n%s" % (res,))
+                else:
+                    print("ePDGSelection: Can't read, response code = %s" % (sw,))
+        except Exception as e:
+            print("ePDGSelection: Can't read file -- " + str(e))
 
-	# Select ISIM application by its AID
-	if '9000' == card.select_adf_by_aid(adf="isim"):
-		# Select USIM profile
-		isim_card = IsimCard(scc)
+    # Select ISIM application by its AID
+    sw = select_app("ISIM", card)
+    if sw == '9000':
+        # Select USIM profile
+        isim_card = IsimCard(scc)
 
-		#EF.P-CSCF - P-CSCF Address
-		try:
-			if isim_card.file_exists(EF_ISIM_ADF_map['PCSCF']):
-				res = isim_card.read_pcscf()
-				print("P-CSCF:\n%s" % (len(res) and res or '\tNot available\n',))
-		except Exception as e:
-			print("P-CSCF: Can't read file -- " + str(e))
+        # EF.P-CSCF - P-CSCF Address
+        try:
+            if isim_card.file_exists(EF_ISIM_ADF_map['PCSCF']):
+                res = isim_card.read_pcscf()
+                print("P-CSCF:\n%s" %
+                      (len(res) and res or '\tNot available\n',))
+        except Exception as e:
+            print("P-CSCF: Can't read file -- " + str(e))
 
-		# EF.DOMAIN - Home Network Domain Name e.g. ims.mncXXX.mccXXX.3gppnetwork.org
-		try:
-			if isim_card.file_exists(EF_ISIM_ADF_map['DOMAIN']):
-				(res, sw) = isim_card.read_domain()
-				if sw == '9000':
-					print("Home Network Domain Name: %s" % (len(res) and res or 'Not available',))
-				else:
-					print("Home Network Domain Name: Can't read, response code = %s" % (sw,))
-		except Exception as e:
-			print("Home Network Domain Name: Can't read file -- " + str(e))
+        # EF.DOMAIN - Home Network Domain Name e.g. ims.mncXXX.mccXXX.3gppnetwork.org
+        try:
+            if isim_card.file_exists(EF_ISIM_ADF_map['DOMAIN']):
+                (res, sw) = isim_card.read_domain()
+                if sw == '9000':
+                    print("Home Network Domain Name: %s" %
+                          (len(res) and res or 'Not available',))
+                else:
+                    print(
+                        "Home Network Domain Name: Can't read, response code = %s" % (sw,))
+        except Exception as e:
+            print("Home Network Domain Name: Can't read file -- " + str(e))
 
-		# EF.IMPI - IMS private user identity
-		try:
-			if isim_card.file_exists(EF_ISIM_ADF_map['IMPI']):
-				(res, sw) = isim_card.read_impi()
-				if sw == '9000':
-					print("IMS private user identity: %s" % (len(res) and res or 'Not available',))
-				else:
-					print("IMS private user identity: Can't read, response code = %s" % (sw,))
-		except Exception as e:
-			print("IMS private user identity: Can't read file -- " + str(e))
+        # EF.IMPI - IMS private user identity
+        try:
+            if isim_card.file_exists(EF_ISIM_ADF_map['IMPI']):
+                (res, sw) = isim_card.read_impi()
+                if sw == '9000':
+                    print("IMS private user identity: %s" %
+                          (len(res) and res or 'Not available',))
+                else:
+                    print(
+                        "IMS private user identity: Can't read, response code = %s" % (sw,))
+        except Exception as e:
+            print("IMS private user identity: Can't read file -- " + str(e))
 
-		# EF.IMPU - IMS public user identity
-		try:
-			if isim_card.file_exists(EF_ISIM_ADF_map['IMPU']):
-				res = isim_card.read_impu()
-				print("IMS public user identity:\n%s" % (len(res) and res or '\tNot available\n',))
-		except Exception as e:
-			print("IMS public user identity: Can't read file -- " + str(e))
+        # EF.IMPU - IMS public user identity
+        try:
+            if isim_card.file_exists(EF_ISIM_ADF_map['IMPU']):
+                res = isim_card.read_impu()
+                print("IMS public user identity:\n%s" %
+                      (len(res) and res or '\tNot available\n',))
+        except Exception as e:
+            print("IMS public user identity: Can't read file -- " + str(e))
 
-		# EF.UICCIARI - UICC IARI
-		try:
-			if isim_card.file_exists(EF_ISIM_ADF_map['UICCIARI']):
-				res = isim_card.read_iari()
-				print("UICC IARI:\n%s" % (len(res) and res or '\tNot available\n',))
-		except Exception as e:
-			print("UICC IARI: Can't read file -- " + str(e))
+        # EF.UICCIARI - UICC IARI
+        try:
+            if isim_card.file_exists(EF_ISIM_ADF_map['UICCIARI']):
+                res = isim_card.read_iari()
+                print("UICC IARI:\n%s" %
+                      (len(res) and res or '\tNot available\n',))
+        except Exception as e:
+            print("UICC IARI: Can't read file -- " + str(e))
 
-	# Check whether we have th AID of ISIM, if so select it by its AID
-	# EF.IST - File Id in ADF ISIM : 6f07
-	if '9000' == card.select_adf_by_aid(adf="isim"):
-		# EF.IST
-		(res, sw) = card.read_binary('6f07')
-		if sw == '9000':
-			print("ISIM Service Table: %s" % res)
-			# Print those which are available
-			print("%s" % dec_st(res, table="isim"))
-		else:
-			print("ISIM Service Table: Can't read, response code = %s" % (sw,))
+        # EF.IST
+        (res, sw) = card.read_binary('6f07')
+        if sw == '9000':
+            print("ISIM Service Table: %s" % res)
+            # Print those which are available
+            print("%s" % dec_st(res, table="isim"))
+        else:
+            print("ISIM Service Table: Can't read, response code = %s" % (sw,))
 
-	# Done for this card and maybe for everything ?
-	print("Done !\n")
+    # Done for this card and maybe for everything ?
+    print("Done !\n")

pySim-shell.py

@@ -0,0 +1,916 @@
+#!/usr/bin/env python3
+
+# Interactive shell for working with SIM / UICC / USIM / ISIM cards
+#
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import List
+
+import json
+import traceback
+
+import cmd2
+from cmd2 import style, fg, bg
+from cmd2 import CommandSet, with_default_category, with_argparser
+import argparse
+
+import os
+import sys
+from pathlib import Path
+from io import StringIO
+
+from pySim.ts_51_011 import EF, DF, EF_SST_map
+from pySim.ts_31_102 import EF_UST_map, EF_USIM_ADF_map
+from pySim.ts_31_103 import EF_IST_map, EF_ISIM_ADF_map
+
+from pySim.exceptions import *
+from pySim.commands import SimCardCommands
+from pySim.transport import init_reader, ApduTracer, argparse_add_reader_args
+from pySim.cards import card_detect, SimCard
+from pySim.utils import h2b, swap_nibbles, rpad, b2h, h2s, JsonEncoder, bertlv_parse_one
+from pySim.utils import dec_st, sanitize_pin_adm, tabulate_str_list, is_hex, boxed_heading_str
+from pySim.card_handler import CardHandler, CardHandlerAuto
+
+from pySim.filesystem import CardMF, RuntimeState, CardDF, CardADF, CardModel
+from pySim.profile import CardProfile
+from pySim.ts_51_011 import CardProfileSIM, DF_TELECOM, DF_GSM
+from pySim.ts_102_221 import CardProfileUICC
+from pySim.ts_102_221 import CardProfileUICCSIM
+from pySim.ts_31_102 import CardApplicationUSIM
+from pySim.ts_31_103 import CardApplicationISIM
+from pySim.ara_m import CardApplicationARAM
+from pySim.gsm_r import DF_EIRENE
+
+# we need to import this module so that the SysmocomSJA2 sub-class of
+# CardModel is created, which will add the ATR-based matching and
+# calling of SysmocomSJA2.add_files.  See  CardModel.apply_matching_models
+import pySim.sysmocom_sja2
+
+from pySim.card_key_provider import CardKeyProviderCsv, card_key_provider_register, card_key_provider_get_field
+
+
+def init_card(sl):
+    """
+    Detect card in reader and setup card profile and runtime state. This
+    function must be called at least once on startup. The card and runtime
+    state object (rs) is required for all pySim-shell commands.
+    """
+
+    # Wait up to three seconds for a card in reader and try to detect
+    # the card type.
+    print("Waiting for card...")
+    try:
+        sl.wait_for_card(3)
+    except NoCardError:
+        print("No card detected!")
+        return None, None
+    except:
+        print("Card not readable!")
+        return None, None
+
+    card = card_detect("auto", scc)
+    if card is None:
+        print("Warning: Could not detect card type - assuming a generic card type...")
+        card = SimCard(scc)
+
+    profile = CardProfile.pick(scc)
+    if profile is None:
+        print("Unsupported card type!")
+        return None, None
+
+    print("Info: Card is of type: %s" % str(profile))
+
+    # FIXME: This shouln't be here, the profile should add the applications,
+    # however, we cannot simply put his into ts_102_221.py since we would
+    # have to e.g. import CardApplicationUSIM from ts_31_102.py, which already
+    # imports from ts_102_221.py. This means we will end up with a circular
+    # import, which needs to be resolved first.
+    if isinstance(profile, CardProfileUICC):
+        profile.add_application(CardApplicationUSIM())
+        profile.add_application(CardApplicationISIM())
+        profile.add_application(CardApplicationARAM())
+
+    # Create runtime state with card profile
+    rs = RuntimeState(card, profile)
+
+    # FIXME: This is an GSM-R related file, it needs to be added throughout,
+    # the profile. At the moment we add it for all cards, this won't hurt,
+    # but regular SIM and UICC will not have it and fail to select it.
+    rs.mf.add_file(DF_EIRENE())
+
+    CardModel.apply_matching_models(scc, rs)
+
+    # inform the transport that we can do context-specific SW interpretation
+    sl.set_sw_interpreter(rs)
+
+    return rs, card
+
+
+class PysimApp(cmd2.Cmd):
+    CUSTOM_CATEGORY = 'pySim Commands'
+
+    def __init__(self, card, rs, sl, ch, script=None):
+        super().__init__(persistent_history_file='~/.pysim_shell_history', allow_cli_args=False,
+                         use_ipython=True, auto_load_commands=False, startup_script=script)
+        self.intro = style('Welcome to pySim-shell!', fg=fg.red)
+        self.default_category = 'pySim-shell built-in commands'
+        self.card = None
+        self.rs = None
+        self.py_locals = {'card': self.card, 'rs': self.rs}
+        self.sl = sl
+        self.ch = ch
+
+        self.numeric_path = False
+        self.add_settable(cmd2.Settable('numeric_path', bool, 'Print File IDs instead of names',
+                                        onchange_cb=self._onchange_numeric_path))
+        self.conserve_write = True
+        self.add_settable(cmd2.Settable('conserve_write', bool, 'Read and compare before write',
+                                        onchange_cb=self._onchange_conserve_write))
+        self.json_pretty_print = True
+        self.add_settable(cmd2.Settable('json_pretty_print',
+                          bool, 'Pretty-Print JSON output'))
+        self.apdu_trace = False
+        self.add_settable(cmd2.Settable('apdu_trace', bool, 'Trace and display APDUs exchanged with card',
+                                        onchange_cb=self._onchange_apdu_trace))
+
+        self.equip(card, rs)
+
+    def equip(self, card, rs):
+        """
+        Equip pySim-shell with the supplied card and runtime state, add (or remove) all required settables and
+        and commands to enable card operations.
+        """
+
+        rc = False
+
+        # Unequip everything from pySim-shell that would not work in unequipped state
+        if self.rs:
+            self.rs.unregister_cmds(self)
+        for cmds in [Iso7816Commands, PySimCommands]:
+            cmd_set = self.find_commandsets(cmds)
+            if cmd_set:
+                self.unregister_command_set(cmd_set[0])
+
+        self.card = card
+        self.rs = rs
+
+        # When a card object and a runtime state is present, (re)equip pySim-shell with everything that is
+        # needed to operate on cards.
+        if self.card and self.rs:
+            self._onchange_conserve_write(
+                'conserve_write', False, self.conserve_write)
+            self._onchange_apdu_trace('apdu_trace', False, self.apdu_trace)
+            self.register_command_set(Iso7816Commands())
+            self.register_command_set(PySimCommands())
+            self.iccid, sw = self.card.read_iccid()
+            rs.select('MF', self)
+            rc = True
+        else:
+            self.poutput("pySim-shell not equipped!")
+
+        self.update_prompt()
+        return rc
+
+    def poutput_json(self, data, force_no_pretty=False):
+        """like cmd2.poutput() but for a JSON serializable dict."""
+        if force_no_pretty or self.json_pretty_print == False:
+            output = json.dumps(data, cls=JsonEncoder)
+        else:
+            output = json.dumps(data, cls=JsonEncoder, indent=4)
+        self.poutput(output)
+
+    def _onchange_numeric_path(self, param_name, old, new):
+        self.update_prompt()
+
+    def _onchange_conserve_write(self, param_name, old, new):
+        if self.rs:
+            self.rs.conserve_write = new
+
+    def _onchange_apdu_trace(self, param_name, old, new):
+        if self.card:
+            if new == True:
+                self.card._scc._tp.apdu_tracer = self.Cmd2ApduTracer(self)
+            else:
+                self.card._scc._tp.apdu_tracer = None
+
+    class Cmd2ApduTracer(ApduTracer):
+        def __init__(self, cmd2_app):
+            self.cmd2 = app
+
+        def trace_response(self, cmd, sw, resp):
+            self.cmd2.poutput("-> %s %s" % (cmd[:10], cmd[10:]))
+            self.cmd2.poutput("<- %s: %s" % (sw, resp))
+
+    def update_prompt(self):
+        if self.rs:
+            path_list = self.rs.selected_file.fully_qualified_path(
+                not self.numeric_path)
+            self.prompt = 'pySIM-shell (%s)> ' % ('/'.join(path_list))
+        else:
+            self.prompt = 'pySIM-shell (no card)> '
+
+    @cmd2.with_category(CUSTOM_CATEGORY)
+    def do_intro(self, _):
+        """Display the intro banner"""
+        self.poutput(self.intro)
+
+    def do_eof(self, _: argparse.Namespace) -> bool:
+        self.poutput("")
+        return self.do_quit('')
+
+    @cmd2.with_category(CUSTOM_CATEGORY)
+    def do_equip(self, opts):
+        """Equip pySim-shell with card"""
+        rs, card = init_card(sl)
+        self.equip(card, rs)
+
+    class InterceptStderr(list):
+        def __init__(self):
+            self._stderr_backup = sys.stderr
+
+        def __enter__(self):
+            self._stringio_stderr = StringIO()
+            sys.stderr = self._stringio_stderr
+            return self
+
+        def __exit__(self, *args):
+            self.stderr = self._stringio_stderr.getvalue().strip()
+            del self._stringio_stderr
+            sys.stderr = self._stderr_backup
+
+    def _show_failure_sign(self):
+        self.poutput(style("  +-------------+", fg=fg.bright_red))
+        self.poutput(style("  +   ##   ##   +", fg=fg.bright_red))
+        self.poutput(style("  +    ## ##    +", fg=fg.bright_red))
+        self.poutput(style("  +     ###     +", fg=fg.bright_red))
+        self.poutput(style("  +    ## ##    +", fg=fg.bright_red))
+        self.poutput(style("  +   ##   ##   +", fg=fg.bright_red))
+        self.poutput(style("  +-------------+", fg=fg.bright_red))
+        self.poutput("")
+
+    def _show_success_sign(self):
+        self.poutput(style("  +-------------+", fg=fg.bright_green))
+        self.poutput(style("  +          ## +", fg=fg.bright_green))
+        self.poutput(style("  +         ##  +", fg=fg.bright_green))
+        self.poutput(style("  +  #    ##    +", fg=fg.bright_green))
+        self.poutput(style("  +   ## #      +", fg=fg.bright_green))
+        self.poutput(style("  +    ##       +", fg=fg.bright_green))
+        self.poutput(style("  +-------------+", fg=fg.bright_green))
+        self.poutput("")
+
+    def _process_card(self, first, script_path):
+
+        # Early phase of card initialzation (this part may fail with an exception)
+        try:
+            rs, card = init_card(self.sl)
+            rc = self.equip(card, rs)
+        except:
+            self.poutput("")
+            self.poutput("Card initialization failed with an exception:")
+            self.poutput("---------------------8<---------------------")
+            traceback.print_exc()
+            self.poutput("---------------------8<---------------------")
+            self.poutput("")
+            return -1
+
+        # Actual card processing step. This part should never fail with an exception since the cmd2
+        # do_run_script method will catch any exception that might occur during script execution.
+        if rc:
+            self.poutput("")
+            self.poutput("Transcript stdout:")
+            self.poutput("---------------------8<---------------------")
+            with self.InterceptStderr() as logged:
+                self.do_run_script(script_path)
+            self.poutput("---------------------8<---------------------")
+
+            self.poutput("")
+            self.poutput("Transcript stderr:")
+            if logged.stderr:
+                self.poutput("---------------------8<---------------------")
+                self.poutput(logged.stderr)
+                self.poutput("---------------------8<---------------------")
+            else:
+                self.poutput("(none)")
+
+            # Check for exceptions
+            self.poutput("")
+            if "EXCEPTION of type" not in logged.stderr:
+                return 0
+
+        return -1
+
+    bulk_script_parser = argparse.ArgumentParser()
+    bulk_script_parser.add_argument(
+        'script_path', help="path to the script file")
+    bulk_script_parser.add_argument('--halt_on_error', help='stop card handling if an exeption occurs',
+                                    action='store_true')
+    bulk_script_parser.add_argument('--tries', type=int, default=2,
+                                    help='how many tries before trying the next card')
+    bulk_script_parser.add_argument('--on_stop_action', type=str, default=None,
+                                    help='commandline to execute when card handling has stopped')
+    bulk_script_parser.add_argument('--pre_card_action', type=str, default=None,
+                                    help='commandline to execute before actually talking to the card')
+
+    @cmd2.with_argparser(bulk_script_parser)
+    @cmd2.with_category(CUSTOM_CATEGORY)
+    def do_bulk_script(self, opts):
+        """Run script on multiple cards (bulk provisioning)"""
+
+        # Make sure that the script file exists and that it is readable.
+        if not os.access(opts.script_path, os.R_OK):
+            self.poutput("Invalid script file!")
+            return
+
+        success_count = 0
+        fail_count = 0
+
+        first = True
+        while 1:
+            # TODO: Count consecutive failures, if more than N consecutive failures occur, then stop.
+            # The ratinale is: There may be a problem with the device, we do want to prevent that
+            # all remaining cards are fired to the error bin. This is only relevant for situations
+            # with large stacks, probably we do not need this feature right now.
+
+            try:
+                # In case of failure, try multiple times.
+                for i in range(opts.tries):
+                    # fetch card into reader bay
+                    ch.get(first)
+
+                    # if necessary execute an action before we start processing the card
+                    if(opts.pre_card_action):
+                        os.system(opts.pre_card_action)
+
+                    # process the card
+                    rc = self._process_card(first, opts.script_path)
+                    if rc == 0:
+                        success_count = success_count + 1
+                        self._show_success_sign()
+                        self.poutput("Statistics: success :%i, failure: %i" % (
+                            success_count, fail_count))
+                        break
+                    else:
+                        fail_count = fail_count + 1
+                        self._show_failure_sign()
+                        self.poutput("Statistics: success :%i, failure: %i" % (
+                            success_count, fail_count))
+
+                # Depending on success or failure, the card goes either in the "error" bin or in the
+                # "done" bin.
+                if rc < 0:
+                    ch.error()
+                else:
+                    ch.done()
+
+                # In most cases it is possible to proceed with the next card, but the
+                # user may decide to halt immediately when an error occurs
+                if opts.halt_on_error and rc < 0:
+                    return
+
+            except (KeyboardInterrupt):
+                self.poutput("")
+                self.poutput("Terminated by user!")
+                return
+            except (SystemExit):
+                # When all cards are processed the card handler device will throw a SystemExit
+                # exception. Also Errors that are not recoverable (cards stuck etc.) will end up here.
+                # The user has the option to execute some action to make aware that the card handler
+                # needs service.
+                if(opts.on_stop_action):
+                    os.system(opts.on_stop_action)
+                return
+            except:
+                self.poutput("")
+                self.poutput("Card handling failed with an exception:")
+                self.poutput("---------------------8<---------------------")
+                traceback.print_exc()
+                self.poutput("---------------------8<---------------------")
+                self.poutput("")
+                fail_count = fail_count + 1
+                self._show_failure_sign()
+                self.poutput("Statistics: success :%i, failure: %i" %
+                             (success_count, fail_count))
+
+            first = False
+
+    echo_parser = argparse.ArgumentParser()
+    echo_parser.add_argument('string', help="string to echo on the shell")
+
+    @cmd2.with_argparser(echo_parser)
+    @cmd2.with_category(CUSTOM_CATEGORY)
+    def do_echo(self, opts):
+        """Echo (print) a string on the console"""
+        self.poutput(opts.string)
+
+
+@with_default_category('pySim Commands')
+class PySimCommands(CommandSet):
+    def __init__(self):
+        super().__init__()
+
+    dir_parser = argparse.ArgumentParser()
+    dir_parser.add_argument(
+        '--fids', help='Show file identifiers', action='store_true')
+    dir_parser.add_argument(
+        '--names', help='Show file names', action='store_true')
+    dir_parser.add_argument(
+        '--apps', help='Show applications', action='store_true')
+    dir_parser.add_argument(
+        '--all', help='Show all selectable identifiers and names', action='store_true')
+
+    @cmd2.with_argparser(dir_parser)
+    def do_dir(self, opts):
+        """Show a listing of files available in currently selected DF or MF"""
+        if opts.all:
+            flags = []
+        elif opts.fids or opts.names or opts.apps:
+            flags = ['PARENT', 'SELF']
+            if opts.fids:
+                flags += ['FIDS', 'AIDS']
+            if opts.names:
+                flags += ['FNAMES', 'ANAMES']
+            if opts.apps:
+                flags += ['ANAMES', 'AIDS']
+        else:
+            flags = ['PARENT', 'SELF', 'FNAMES', 'ANAMES']
+        selectables = list(
+            self._cmd.rs.selected_file.get_selectable_names(flags=flags))
+        directory_str = tabulate_str_list(
+            selectables, width=79, hspace=2, lspace=1, align_left=True)
+        path_list = self._cmd.rs.selected_file.fully_qualified_path(True)
+        self._cmd.poutput('/'.join(path_list))
+        path_list = self._cmd.rs.selected_file.fully_qualified_path(False)
+        self._cmd.poutput('/'.join(path_list))
+        self._cmd.poutput(directory_str)
+        self._cmd.poutput("%d files" % len(selectables))
+
+    def walk(self, indent=0, action=None, context=None):
+        """Recursively walk through the file system, starting at the currently selected DF"""
+        files = self._cmd.rs.selected_file.get_selectables(
+            flags=['FNAMES', 'ANAMES'])
+        for f in files:
+            if not action:
+                output_str = "  " * indent + str(f) + (" " * 250)
+                output_str = output_str[0:25]
+                if isinstance(files[f], CardADF):
+                    output_str += " " + str(files[f].aid)
+                else:
+                    output_str += " " + str(files[f].fid)
+                output_str += " " + str(files[f].desc)
+                self._cmd.poutput(output_str)
+
+            if isinstance(files[f], CardDF):
+                skip_df = False
+                try:
+                    fcp_dec = self._cmd.rs.select(f, self._cmd)
+                except Exception as e:
+                    skip_df = True
+                    df = self._cmd.rs.selected_file
+                    df_path_list = df.fully_qualified_path(True)
+                    df_skip_reason_str = '/'.join(df_path_list) + \
+                        "/" + str(f) + ", " + str(e)
+                    if context:
+                        context['DF_SKIP'] += 1
+                        context['DF_SKIP_REASON'].append(df_skip_reason_str)
+
+                # If the DF was skipped, we never have entered the directory
+                # below, so we must not move up.
+                if skip_df == False:
+                    self.walk(indent + 1, action, context)
+                    fcp_dec = self._cmd.rs.select("..", self._cmd)
+
+            elif action:
+                df_before_action = self._cmd.rs.selected_file
+                action(f, context)
+                # When walking through the file system tree the action must not
+                # always restore the currently selected file to the file that
+                # was selected before executing the action() callback.
+                if df_before_action != self._cmd.rs.selected_file:
+                    raise RuntimeError("inconsistent walk, %s is currently selected but expecting %s to be selected"
+                                       % (str(self._cmd.rs.selected_file), str(df_before_action)))
+
+    def do_tree(self, opts):
+        """Display a filesystem-tree with all selectable files"""
+        self.walk()
+
+    def export(self, filename, context):
+        """ Select and export a single file """
+        context['COUNT'] += 1
+        df = self._cmd.rs.selected_file
+
+        if not isinstance(df, CardDF):
+            raise RuntimeError(
+                "currently selected file %s is not a DF or ADF" % str(df))
+
+        df_path_list = df.fully_qualified_path(True)
+        df_path_list_fid = df.fully_qualified_path(False)
+
+        file_str = '/'.join(df_path_list) + "/" + str(filename)
+        self._cmd.poutput(boxed_heading_str(file_str))
+
+        self._cmd.poutput("# directory: %s (%s)" %
+                          ('/'.join(df_path_list), '/'.join(df_path_list_fid)))
+        try:
+            fcp_dec = self._cmd.rs.select(filename, self._cmd)
+            self._cmd.poutput("# file: %s (%s)" % (
+                self._cmd.rs.selected_file.name, self._cmd.rs.selected_file.fid))
+
+            fd = fcp_dec['file_descriptor']
+            structure = fd['structure']
+            self._cmd.poutput("# structure: %s" % str(structure))
+
+            for f in df_path_list:
+                self._cmd.poutput("select " + str(f))
+            self._cmd.poutput("select " + self._cmd.rs.selected_file.name)
+
+            if structure == 'transparent':
+                result = self._cmd.rs.read_binary()
+                self._cmd.poutput("update_binary " + str(result[0]))
+            elif structure == 'cyclic' or structure == 'linear_fixed':
+                # Use number of records specified in select response
+                if 'num_of_rec' in fd:
+                    num_of_rec = fd['num_of_rec']
+                    for r in range(1, num_of_rec + 1):
+                        result = self._cmd.rs.read_record(r)
+                        self._cmd.poutput("update_record %d %s" %
+                                          (r, str(result[0])))
+                # When the select response does not return the number of records, read until we hit the
+                # first record that cannot be read.
+                else:
+                    r = 1
+                    while True:
+                        try:
+                            result = self._cmd.rs.read_record(r)
+                        except SwMatchError as e:
+                            # We are past the last valid record - stop
+                            if e.sw_actual == "9402":
+                                break
+                            # Some other problem occurred
+                            else:
+                                raise e
+                        self._cmd.poutput("update_record %d %s" %
+                                          (r, str(result[0])))
+                        r = r + 1
+            elif structure == 'ber_tlv':
+                tags = self._cmd.rs.retrieve_tags()
+                for t in tags:
+                    result = self._cmd.rs.retrieve_data(t)
+                    (tag, l, val, remainer) = bertlv_parse_one(h2b(result[0]))
+                    self._cmd.poutput("set_data 0x%02x %s" % (t, b2h(val)))
+            else:
+                raise RuntimeError(
+                    'Unsupported structure "%s" of file "%s"' % (structure, filename))
+        except Exception as e:
+            bad_file_str = '/'.join(df_path_list) + \
+                "/" + str(filename) + ", " + str(e)
+            self._cmd.poutput("# bad file: %s" % bad_file_str)
+            context['ERR'] += 1
+            context['BAD'].append(bad_file_str)
+
+        # When reading the file is done, make sure the parent file is
+        # selected again. This will be the usual case, however we need
+        # to check before since we must not select the same DF twice
+        if df != self._cmd.rs.selected_file:
+            self._cmd.rs.select(df.fid or df.aid, self._cmd)
+
+        self._cmd.poutput("#")
+
+    export_parser = argparse.ArgumentParser()
+    export_parser.add_argument(
+        '--filename', type=str, default=None, help='only export specific file')
+
+    @cmd2.with_argparser(export_parser)
+    def do_export(self, opts):
+        """Export files to script that can be imported back later"""
+        context = {'ERR': 0, 'COUNT': 0, 'BAD': [],
+                   'DF_SKIP': 0, 'DF_SKIP_REASON': []}
+        if opts.filename:
+            self.export(opts.filename, context)
+        else:
+            self.walk(0, self.export, context)
+
+        self._cmd.poutput(boxed_heading_str("Export summary"))
+
+        self._cmd.poutput("# total files visited: %u" % context['COUNT'])
+        self._cmd.poutput("# bad files:           %u" % context['ERR'])
+        for b in context['BAD']:
+            self._cmd.poutput("#  " + b)
+
+        self._cmd.poutput("# skipped dedicated files(s): %u" %
+                          context['DF_SKIP'])
+        for b in context['DF_SKIP_REASON']:
+            self._cmd.poutput("#  " + b)
+
+        if context['ERR'] and context['DF_SKIP']:
+            raise RuntimeError("unable to export %i elementary file(s) and %i dedicated file(s)" % (
+                context['ERR'], context['DF_SKIP']))
+        elif context['ERR']:
+            raise RuntimeError(
+                "unable to export %i elementary file(s)" % context['ERR'])
+        elif context['DF_SKIP']:
+            raise RuntimeError(
+                "unable to export %i dedicated files(s)" % context['ERR'])
+
+    def do_reset(self, opts):
+        """Reset the Card."""
+        atr = self._cmd.rs.reset(self._cmd)
+        self._cmd.poutput('Card ATR: %s' % atr)
+        self._cmd.update_prompt()
+
+    def do_desc(self, opts):
+        """Display human readable file description for the currently selected file"""
+        desc = self._cmd.rs.selected_file.desc
+        if desc:
+            self._cmd.poutput(desc)
+        else:
+            self._cmd.poutput("no description available")
+
+    def do_verify_adm(self, arg):
+        """VERIFY the ADM1 PIN"""
+        if arg:
+            # use specified ADM-PIN
+            pin_adm = sanitize_pin_adm(arg)
+        else:
+            # try to find an ADM-PIN if none is specified
+            result = card_key_provider_get_field(
+                'ADM1', key='ICCID', value=self._cmd.iccid)
+            pin_adm = sanitize_pin_adm(result)
+            if pin_adm:
+                self._cmd.poutput(
+                    "found ADM-PIN '%s' for ICCID '%s'" % (result, self._cmd.iccid))
+            else:
+                raise ValueError(
+                    "cannot find ADM-PIN for ICCID '%s'" % (self._cmd.iccid))
+
+        if pin_adm:
+            self._cmd.card.verify_adm(h2b(pin_adm))
+        else:
+            raise ValueError("error: cannot authenticate, no adm-pin!")
+
+
+@with_default_category('ISO7816 Commands')
+class Iso7816Commands(CommandSet):
+    def __init__(self):
+        super().__init__()
+
+    def do_select(self, opts):
+        """SELECT a File (ADF/DF/EF)"""
+        if len(opts.arg_list) == 0:
+            path_list = self._cmd.rs.selected_file.fully_qualified_path(True)
+            path_list_fid = self._cmd.rs.selected_file.fully_qualified_path(
+                False)
+            self._cmd.poutput("currently selected file: " +
+                              '/'.join(path_list) + " (" + '/'.join(path_list_fid) + ")")
+            return
+
+        path = opts.arg_list[0]
+        fcp_dec = self._cmd.rs.select(path, self._cmd)
+        self._cmd.update_prompt()
+        self._cmd.poutput_json(fcp_dec)
+
+    def complete_select(self, text, line, begidx, endidx) -> List[str]:
+        """Command Line tab completion for SELECT"""
+        index_dict = {1: self._cmd.rs.selected_file.get_selectable_names()}
+        return self._cmd.index_based_complete(text, line, begidx, endidx, index_dict=index_dict)
+
+    def get_code(self, code):
+        """Use code either directly or try to get it from external data source"""
+        auto = ('PIN1', 'PIN2', 'PUK1', 'PUK2')
+
+        if str(code).upper() not in auto:
+            return sanitize_pin_adm(code)
+
+        result = card_key_provider_get_field(
+            str(code), key='ICCID', value=self._cmd.iccid)
+        result = sanitize_pin_adm(result)
+        if result:
+            self._cmd.poutput("found %s '%s' for ICCID '%s'" %
+                              (code.upper(), result, self._cmd.iccid))
+        else:
+            self._cmd.poutput("cannot find %s for ICCID '%s'" %
+                              (code.upper(), self._cmd.iccid))
+        return result
+
+    verify_chv_parser = argparse.ArgumentParser()
+    verify_chv_parser.add_argument(
+        '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
+    verify_chv_parser.add_argument(
+        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+
+    @cmd2.with_argparser(verify_chv_parser)
+    def do_verify_chv(self, opts):
+        """Verify (authenticate) using specified PIN code"""
+        pin = self.get_code(opts.pin_code)
+        (data, sw) = self._cmd.card._scc.verify_chv(opts.pin_nr, h2b(pin))
+        self._cmd.poutput("CHV verification successful")
+
+    unblock_chv_parser = argparse.ArgumentParser()
+    unblock_chv_parser.add_argument(
+        '--pin-nr', type=int, default=1, help='PUK Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
+    unblock_chv_parser.add_argument(
+        'puk_code', type=str, help='PUK code digits \"PUK1\" or \"PUK2\" to get PUK code from external data source')
+    unblock_chv_parser.add_argument(
+        'new_pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+
+    @cmd2.with_argparser(unblock_chv_parser)
+    def do_unblock_chv(self, opts):
+        """Unblock PIN code using specified PUK code"""
+        new_pin = self.get_code(opts.new_pin_code)
+        puk = self.get_code(opts.puk_code)
+        (data, sw) = self._cmd.card._scc.unblock_chv(
+            opts.pin_nr, h2b(puk), h2b(new_pin))
+        self._cmd.poutput("CHV unblock successful")
+
+    change_chv_parser = argparse.ArgumentParser()
+    change_chv_parser.add_argument(
+        '--pin-nr', type=int, default=1, help='PUK Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
+    change_chv_parser.add_argument(
+        'pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+    change_chv_parser.add_argument(
+        'new_pin_code', type=str, help='PIN code digits \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+
+    @cmd2.with_argparser(change_chv_parser)
+    def do_change_chv(self, opts):
+        """Change PIN code to a new PIN code"""
+        new_pin = self.get_code(opts.new_pin_code)
+        pin = self.get_code(opts.pin_code)
+        (data, sw) = self._cmd.card._scc.change_chv(
+            opts.pin_nr, h2b(pin), h2b(new_pin))
+        self._cmd.poutput("CHV change successful")
+
+    disable_chv_parser = argparse.ArgumentParser()
+    disable_chv_parser.add_argument(
+        '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
+    disable_chv_parser.add_argument(
+        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+
+    @cmd2.with_argparser(disable_chv_parser)
+    def do_disable_chv(self, opts):
+        """Disable PIN code using specified PIN code"""
+        pin = self.get_code(opts.pin_code)
+        (data, sw) = self._cmd.card._scc.disable_chv(opts.pin_nr, h2b(pin))
+        self._cmd.poutput("CHV disable successful")
+
+    enable_chv_parser = argparse.ArgumentParser()
+    enable_chv_parser.add_argument(
+        '--pin-nr', type=int, default=1, help='PIN Number, 1=PIN1, 2=PIN2 or custom value (decimal)')
+    enable_chv_parser.add_argument(
+        'pin_code', type=str, help='PIN code digits, \"PIN1\" or \"PIN2\" to get PIN code from external data source')
+
+    @cmd2.with_argparser(enable_chv_parser)
+    def do_enable_chv(self, opts):
+        """Enable PIN code using specified PIN code"""
+        pin = self.get_code(opts.pin_code)
+        (data, sw) = self._cmd.card._scc.enable_chv(opts.pin_nr, h2b(pin))
+        self._cmd.poutput("CHV enable successful")
+
+    def do_deactivate_file(self, opts):
+        """Deactivate the current EF"""
+        (data, sw) = self._cmd.card._scc.deactivate_file()
+
+    def do_activate_file(self, opts):
+        """Activate the specified EF"""
+        path = opts.arg_list[0]
+        (data, sw) = self._cmd.rs.activate_file(path)
+
+    def complete_activate_file(self, text, line, begidx, endidx) -> List[str]:
+        """Command Line tab completion for ACTIVATE FILE"""
+        index_dict = {1: self._cmd.rs.selected_file.get_selectable_names()}
+        return self._cmd.index_based_complete(text, line, begidx, endidx, index_dict=index_dict)
+
+    open_chan_parser = argparse.ArgumentParser()
+    open_chan_parser.add_argument(
+        'chan_nr', type=int, default=0, help='Channel Number')
+
+    @cmd2.with_argparser(open_chan_parser)
+    def do_open_channel(self, opts):
+        """Open a logical channel."""
+        (data, sw) = self._cmd.card._scc.manage_channel(
+            mode='open', lchan_nr=opts.chan_nr)
+
+    close_chan_parser = argparse.ArgumentParser()
+    close_chan_parser.add_argument(
+        'chan_nr', type=int, default=0, help='Channel Number')
+
+    @cmd2.with_argparser(close_chan_parser)
+    def do_close_channel(self, opts):
+        """Close a logical channel."""
+        (data, sw) = self._cmd.card._scc.manage_channel(
+            mode='close', lchan_nr=opts.chan_nr)
+
+    def do_status(self, opts):
+        """Perform the STATUS command."""
+        fcp_dec = self._cmd.rs.status()
+        self._cmd.poutput_json(fcp_dec)
+
+    suspend_uicc_parser = argparse.ArgumentParser()
+    suspend_uicc_parser.add_argument('--min-duration-secs', type=int, default=60,
+                                     help='Proposed minimum duration of suspension')
+    suspend_uicc_parser.add_argument('--max-duration-secs', type=int, default=24*60*60,
+                                     help='Proposed maximum duration of suspension')
+
+    # not ISO7816-4 but TS 102 221
+    @cmd2.with_argparser(suspend_uicc_parser)
+    def do_suspend_uicc(self, opts):
+        """Perform the SUSPEND UICC command. Only supported on some UICC."""
+        (duration, token, sw) = self._cmd.card._scc.suspend_uicc(min_len_secs=opts.min_duration_secs,
+                                                                 max_len_secs=opts.max_duration_secs)
+        self._cmd.poutput(
+            'Negotiated Duration: %u secs, Token: %s, SW: %s' % (duration, token, sw))
+
+
+option_parser = argparse.ArgumentParser(prog='pySim-shell', description='interactive SIM card shell',
+                                        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+argparse_add_reader_args(option_parser)
+
+global_group = option_parser.add_argument_group('General Options')
+global_group.add_argument('--script', metavar='PATH', default=None,
+                          help='script with pySim-shell commands to be executed automatically at start-up')
+global_group.add_argument('--csv', metavar='FILE',
+                          default=None, help='Read card data from CSV file')
+global_group.add_argument("--card_handler", dest="card_handler_config", metavar="FILE",
+                          help="Use automatic card handling machine")
+
+adm_group = global_group.add_mutually_exclusive_group()
+adm_group.add_argument('-a', '--pin-adm', metavar='PIN_ADM1', dest='pin_adm', default=None,
+                       help='ADM PIN used for provisioning (overwrites default)')
+adm_group.add_argument('-A', '--pin-adm-hex', metavar='PIN_ADM1_HEX', dest='pin_adm_hex', default=None,
+                       help='ADM PIN used for provisioning, as hex string (16 characters long)')
+
+
+if __name__ == '__main__':
+
+    # Parse options
+    opts = option_parser.parse_args()
+
+    # If a script file is specified, be sure that it actually exists
+    if opts.script:
+        if not os.access(opts.script, os.R_OK):
+            print("Invalid script file!")
+            sys.exit(2)
+
+    # Register csv-file as card data provider, either from specified CSV
+    # or from CSV file in home directory
+    csv_default = str(Path.home()) + "/.osmocom/pysim/card_data.csv"
+    if opts.csv:
+        card_key_provider_register(CardKeyProviderCsv(opts.csv))
+    if os.path.isfile(csv_default):
+        card_key_provider_register(CardKeyProviderCsv(csv_default))
+
+    # Init card reader driver
+    sl = init_reader(opts)
+    if sl is None:
+        exit(1)
+
+    # Create command layer
+    scc = SimCardCommands(transport=sl)
+
+    # Create a card handler (for bulk provisioning)
+    if opts.card_handler_config:
+        ch = CardHandlerAuto(None, opts.card_handler_config)
+    else:
+        ch = CardHandler(sl)
+
+    # Detect and initialize the card in the reader. This may fail when there
+    # is no card in the reader or the card is unresponsive. PysimApp is
+    # able to tolerate and recover from that.
+    try:
+        rs, card = init_card(sl)
+        app = PysimApp(card, rs, sl, ch, opts.script)
+    except:
+        print("Card initialization failed with an exception:")
+        print("---------------------8<---------------------")
+        traceback.print_exc()
+        print("---------------------8<---------------------")
+        print("(you may still try to recover from this manually by using the 'equip' command.)")
+        print(
+            " it should also be noted that some readers may behave strangely when no card")
+        print(" is inserted.)")
+        print("")
+        app = PysimApp(None, None, sl, ch, opts.script)
+
+    # If the user supplies an ADM PIN at via commandline args authenticate
+    # immediately so that the user does not have to use the shell commands
+    pin_adm = sanitize_pin_adm(opts.pin_adm, opts.pin_adm_hex)
+    if pin_adm:
+        if not card:
+            print("Card error, cannot do ADM verification with supplied ADM pin now.")
+        try:
+            card.verify_adm(h2b(pin_adm))
+        except Exception as e:
+            print(e)
+
+    app.cmdloop()

pySim/__init__.py

@@ -1 +0,0 @@
-

pySim/ara_m.py

@@ -0,0 +1,412 @@
+# -*- coding: utf-8 -*-
+
+# without this, pylint will fail when inner classes are used
+# within the 'nested' kwarg of our TlvMeta metaclass on python 3.7 :(
+# pylint: disable=undefined-variable
+
+"""
+Support for the Secure Element Access Control, specifically the ARA-M inside an UICC.
+"""
+
+#
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+from construct import *
+from construct import Optional as COptional
+from pySim.construct import *
+from pySim.filesystem import *
+from pySim.tlv import *
+
+# various BER-TLV encoded Data Objects (DOs)
+
+
+class AidRefDO(BER_TLV_IE, tag=0x4f):
+    # SEID v1.1 Table 6-3
+    _construct = HexAdapter(GreedyBytes)
+
+
+class AidRefEmptyDO(BER_TLV_IE, tag=0xc0):
+    # SEID v1.1 Table 6-3
+    pass
+
+
+class DevAppIdRefDO(BER_TLV_IE, tag=0xc1):
+    # SEID v1.1 Table 6-4
+    _construct = HexAdapter(GreedyBytes)
+
+
+class PkgRefDO(BER_TLV_IE, tag=0xca):
+    # Android UICC Carrier Privileges specific extension, see https://source.android.com/devices/tech/config/uicc
+    _construct = Struct('package_name_string'/GreedyString("ascii"))
+
+
+class RefDO(BER_TLV_IE, tag=0xe1, nested=[AidRefDO, AidRefEmptyDO, DevAppIdRefDO, PkgRefDO]):
+    # SEID v1.1 Table 6-5
+    pass
+
+
+class ApduArDO(BER_TLV_IE, tag=0xd0):
+    # SEID v1.1 Table 6-8
+    def _from_bytes(self, do: bytes):
+        if len(do) == 1:
+            if do[0] == 0x00:
+                self.decoded = {'generic_access_rule': 'never'}
+                return self.decoded
+            elif do[0] == 0x01:
+                self.decoded = {'generic_access_rule': 'always'}
+                return self.decoded
+            else:
+                return ValueError('Invalid 1-byte generic APDU access rule')
+        else:
+            if len(do) % 8:
+                return ValueError('Invalid non-modulo-8 length of APDU filter: %d' % len(do))
+            self.decoded['apdu_filter'] = []
+            offset = 0
+            while offset < len(do):
+                self.decoded['apdu_filter'] += {'header': b2h(do[offset:offset+4]),
+                                                'mask': b2h(do[offset+4:offset+8])}
+            self.decoded = res
+            return res
+
+    def _to_bytes(self):
+        if 'generic_access_rule' in self.decoded:
+            if self.decoded['generic_access_rule'] == 'never':
+                return b'\x00'
+            elif self.decoded['generic_access_rule'] == 'always':
+                return b'\x01'
+            else:
+                return ValueError('Invalid 1-byte generic APDU access rule')
+        else:
+            if not 'apdu_filter' in self.decoded:
+                return ValueError('Invalid APDU AR DO')
+            filters = self.decoded['apdu_filter']
+            res = b''
+            for f in filters:
+                if not 'header' in f or not 'mask' in f:
+                    return ValueError('APDU filter must contain header and mask')
+                header_b = h2b(f['header'])
+                mask_b = h2b(f['mask'])
+                if len(header_b) != 4 or len(mask_b) != 4:
+                    return ValueError('APDU filter header and mask must each be 4 bytes')
+                res += header_b + mask_b
+            return res
+
+
+class NfcArDO(BER_TLV_IE, tag=0xd1):
+    # SEID v1.1 Table 6-9
+    _construct = Struct('nfc_event_access_rule' /
+                        Enum(Int8ub, never=0, always=1))
+
+
+class PermArDO(BER_TLV_IE, tag=0xdb):
+    # Android UICC Carrier Privileges specific extension, see https://source.android.com/devices/tech/config/uicc
+    _construct = Struct('permissions'/HexAdapter(Bytes(8)))
+
+
+class ArDO(BER_TLV_IE, tag=0xe3, nested=[ApduArDO, NfcArDO, PermArDO]):
+    # SEID v1.1 Table 6-7
+    pass
+
+
+class RefArDO(BER_TLV_IE, tag=0xe2, nested=[RefDO, ArDO]):
+    # SEID v1.1 Table 6-6
+    pass
+
+
+class ResponseAllRefArDO(BER_TLV_IE, tag=0xff40, nested=[RefArDO]):
+    # SEID v1.1 Table 4-2
+    pass
+
+
+class ResponseArDO(BER_TLV_IE, tag=0xff50, nested=[ArDO]):
+    # SEID v1.1 Table 4-3
+    pass
+
+
+class ResponseRefreshTagDO(BER_TLV_IE, tag=0xdf20):
+    # SEID v1.1 Table 4-4
+    _construct = Struct('refresh_tag'/HexAdapter(Bytes(8)))
+
+
+class DeviceInterfaceVersionDO(BER_TLV_IE, tag=0xe6):
+    # SEID v1.1 Table 6-12
+    _construct = Struct('major'/Int8ub, 'minor'/Int8ub, 'patch'/Int8ub)
+
+
+class DeviceConfigDO(BER_TLV_IE, tag=0xe4, nested=[DeviceInterfaceVersionDO]):
+    # SEID v1.1 Table 6-10
+    pass
+
+
+class ResponseDeviceConfigDO(BER_TLV_IE, tag=0xff7f, nested=[DeviceConfigDO]):
+    # SEID v1.1 Table 5-14
+    pass
+
+
+class AramConfigDO(BER_TLV_IE, tag=0xe5, nested=[DeviceInterfaceVersionDO]):
+    # SEID v1.1 Table 6-11
+    pass
+
+
+class ResponseAramConfigDO(BER_TLV_IE, tag=0xdf21, nested=[AramConfigDO]):
+    # SEID v1.1 Table 4-5
+    pass
+
+
+class CommandStoreRefArDO(BER_TLV_IE, tag=0xf0, nested=[RefArDO]):
+    # SEID v1.1 Table 5-2
+    pass
+
+
+class CommandDelete(BER_TLV_IE, tag=0xf1, nested=[AidRefDO, AidRefEmptyDO, RefDO, RefArDO]):
+    # SEID v1.1 Table 5-4
+    pass
+
+
+class CommandUpdateRefreshTagDO(BER_TLV_IE, tag=0xf2):
+    # SEID V1.1 Table 5-6
+    pass
+
+
+class CommandRegisterClientAidsDO(BER_TLV_IE, tag=0xf7, nested=[AidRefDO, AidRefEmptyDO]):
+    # SEID v1.1 Table 5-7
+    pass
+
+
+class CommandGet(BER_TLV_IE, tag=0xf3, nested=[AidRefDO, AidRefEmptyDO]):
+    # SEID v1.1 Table 5-8
+    pass
+
+
+class CommandGetAll(BER_TLV_IE, tag=0xf4):
+    # SEID v1.1 Table 5-9
+    pass
+
+
+class CommandGetClientAidsDO(BER_TLV_IE, tag=0xf6):
+    # SEID v1.1 Table 5-10
+    pass
+
+
+class CommandGetNext(BER_TLV_IE, tag=0xf5):
+    # SEID v1.1 Table 5-11
+    pass
+
+
+class CommandGetDeviceConfigDO(BER_TLV_IE, tag=0xf8):
+    # SEID v1.1 Table 5-12
+    pass
+
+
+class ResponseAracAidDO(BER_TLV_IE, tag=0xff70, nested=[AidRefDO, AidRefEmptyDO]):
+    # SEID v1.1 Table 5-13
+    pass
+
+
+class BlockDO(BER_TLV_IE, tag=0xe7):
+    # SEID v1.1 Table 6-13
+    _construct = Struct('offset'/Int16ub, 'length'/Int8ub)
+
+
+# SEID v1.1 Table 4-1
+class GetCommandDoCollection(TLV_IE_Collection, nested=[RefDO, DeviceConfigDO]):
+    pass
+
+# SEID v1.1 Table 4-2
+
+
+class GetResponseDoCollection(TLV_IE_Collection, nested=[ResponseAllRefArDO, ResponseArDO,
+                                                         ResponseRefreshTagDO, ResponseAramConfigDO]):
+    pass
+
+# SEID v1.1 Table 5-1
+
+
+class StoreCommandDoCollection(TLV_IE_Collection,
+                               nested=[BlockDO, CommandStoreRefArDO, CommandDelete,
+                                       CommandUpdateRefreshTagDO, CommandRegisterClientAidsDO,
+                                       CommandGet, CommandGetAll, CommandGetClientAidsDO,
+                                       CommandGetNext, CommandGetDeviceConfigDO]):
+    pass
+
+
+# SEID v1.1 Section 5.1.2
+class StoreResponseDoCollection(TLV_IE_Collection,
+                                nested=[ResponseAllRefArDO, ResponseAracAidDO, ResponseDeviceConfigDO]):
+    pass
+
+
+class ADF_ARAM(CardADF):
+    def __init__(self, aid='a00000015141434c00', name='ADF.ARA-M', fid=None, sfid=None,
+                 desc='ARA-M Application'):
+        super().__init__(aid=aid, fid=fid, sfid=sfid, name=name, desc=desc)
+        self.shell_commands += [self.AddlShellCommands()]
+        files = []
+        self.add_files(files)
+
+    @staticmethod
+    def xceive_apdu_tlv(tp, hdr: Hexstr, cmd_do, resp_cls, exp_sw='9000'):
+        """Transceive an APDU with the card, transparently encoding the command data from TLV
+        and decoding the response data tlv."""
+        if cmd_do:
+            cmd_do_enc = cmd_do.to_ie()
+            cmd_do_len = len(cmd_do_enc)
+            if cmd_do_len > 255:
+                return ValueError('DO > 255 bytes not supported yet')
+        else:
+            cmd_do_enc = b''
+            cmd_do_len = 0
+        c_apdu = hdr + ('%02x' % cmd_do_len) + b2h(cmd_do_enc)
+        (data, sw) = tp.send_apdu_checksw(c_apdu, exp_sw)
+        if data:
+            if resp_cls:
+                resp_do = resp_cls()
+                resp_do.from_tlv(h2b(data))
+                return resp_do
+            else:
+                return data
+        else:
+            return None
+
+    @staticmethod
+    def store_data(tp, do) -> bytes:
+        """Build the Command APDU for STORE DATA."""
+        return ADF_ARAM.xceive_apdu_tlv(tp, '80e29000', do, StoreResponseDoCollection)
+
+    @staticmethod
+    def get_all(tp):
+        return ADF_ARAM.xceive_apdu_tlv(tp, '80caff40', None, GetResponseDoCollection)
+
+    @staticmethod
+    def get_config(tp, v_major=0, v_minor=0, v_patch=1):
+        cmd_do = DeviceConfigDO()
+        cmd_do.from_dict([{'DeviceInterfaceVersionDO': {
+                         'major': v_major, 'minor': v_minor, 'patch': v_patch}}])
+        return ADF_ARAM.xceive_apdu_tlv(tp, '80cadf21', cmd_do, ResponseAramConfigDO)
+
+    @with_default_category('Application-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        def __init(self):
+            super().__init__()
+
+        def do_aram_get_all(self, opts):
+            """GET DATA [All] on the ARA-M Applet"""
+            res_do = ADF_ARAM.get_all(self._cmd.card._scc._tp)
+            if res_do:
+                self._cmd.poutput_json(res_do.to_dict())
+
+        def do_aram_get_config(self, opts):
+            """GET DATA [Config] on the ARA-M Applet"""
+            res_do = ADF_ARAM.get_config(self._cmd.card._scc._tp)
+            if res_do:
+                self._cmd.poutput_json(res_do.to_dict())
+
+        store_ref_ar_do_parse = argparse.ArgumentParser()
+        # REF-DO
+        store_ref_ar_do_parse.add_argument(
+            '--device-app-id', required=True, help='Identifies the specific device application that the rule appplies to. Hash of Certificate of Application Provider, or UUID. (20/32 hex bytes)')
+        aid_grp = store_ref_ar_do_parse.add_mutually_exclusive_group()
+        aid_grp.add_argument(
+            '--aid', help='Identifies the specific SE application for which rules are to be stored. Can be a partial AID, containing for example only the RID.  (5-16 hex bytes)')
+        aid_grp.add_argument('--aid-empty', action='store_true',
+                             help='No specific SE application, applies to all applications')
+        store_ref_ar_do_parse.add_argument(
+            '--pkg-ref', help='Full Android Java package name (up to 127 chars ASCII)')
+        # AR-DO
+        apdu_grp = store_ref_ar_do_parse.add_mutually_exclusive_group()
+        apdu_grp.add_argument(
+            '--apdu-never', action='store_true', help='APDU access is not allowed')
+        apdu_grp.add_argument(
+            '--apdu-always', action='store_true', help='APDU access is allowed')
+        apdu_grp.add_argument(
+            '--apdu-filter', help='APDU filter: 4 byte CLA/INS/P1/P2 followed by 4 byte mask (8 hex bytes)')
+        nfc_grp = store_ref_ar_do_parse.add_mutually_exclusive_group()
+        nfc_grp.add_argument('--nfc-always', action='store_true',
+                             help='NFC event access is allowed')
+        nfc_grp.add_argument('--nfc-never', action='store_true',
+                             help='NFC event access is not allowed')
+        store_ref_ar_do_parse.add_argument(
+            '--android-permissions', help='Android UICC Carrier Privilege Permissions (8 hex bytes)')
+
+        @cmd2.with_argparser(store_ref_ar_do_parse)
+        def do_aram_store_ref_ar_do(self, opts):
+            """Perform STORE DATA [Command-Store-REF-AR-DO] to store a new access rule."""
+            # REF
+            ref_do_content = []
+            if opts.aid:
+                ref_do_content += [{'AidRefDO': opts.aid}]
+            elif opts.aid_empty:
+                ref_do_content += [{'AidRefEmptyDO': None}]
+            ref_do_content += [{'DevAppIdRefDO': opts.device_app_id}]
+            if opts.pkg_ref:
+                ref_do_content += [{'PkgRefDO': opts.pkg_ref}]
+            # AR
+            ar_do_content = []
+            if opts.apdu_never:
+                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'never'}}]
+            elif opts.apdu_always:
+                ar_do_content += [{'ApduArDO': {'generic_access_rule': 'always'}}]
+            elif opts.apdu_filter:
+                # TODO: multiple filters
+                ar_do_content += [{'ApduArDO': {'apdu_filter': [opts.apdu_filter]}}]
+            if opts.nfc_always:
+                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'always'}}]
+            elif opts.nfc_never:
+                ar_do_content += [{'NfcArDO': {'nfc_event_access_rule': 'never'}}]
+            if opts.android_permissions:
+                ar_do_content += [{'PermArDO': {'permissions': opts.android_permissions}}]
+            d = [{'RefArDO': [{'RefDO': ref_do_content}, {'ArDO': ar_do_content}]}]
+            csrado = CommandStoreRefArDO()
+            csrado.from_dict(d)
+            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, csrado)
+            if res_do:
+                self._cmd.poutput_json(res_do.to_dict())
+
+        def do_aram_delete_all(self, opts):
+            """Perform STORE DATA [Command-Delete[all]] to delete all access rules."""
+            deldo = CommandDelete()
+            res_do = ADF_ARAM.store_data(self._cmd.card._scc._tp, deldo)
+            if res_do:
+                self._cmd.poutput_json(res_do.to_dict())
+
+
+# SEAC v1.1 Section 4.1.2.2 + 5.1.2.2
+sw_aram = {
+    'ARA-M': {
+        '6381': 'Rule successfully stored but an access rule already exists',
+        '6382': 'Rule successfully stored bu contained at least one unknown (discarded) BER-TLV',
+        '6581': 'Memory Problem',
+        '6700': 'Wrong Length in Lc',
+        '6981': 'DO is not supported by the ARA-M/ARA-C',
+        '6982': 'Security status not satisfied',
+        '6984': 'Rules have been updated and must be read again / logical channels in use',
+        '6985': 'Conditions not satisfied',
+        '6a80': 'Incorrect values in the command data',
+        '6a84': 'Rules have been updated and must be read again',
+        '6a86': 'Incorrect P1 P2',
+        '6a88': 'Referenced data not found',
+        '6a89': 'Conflicting access rule already exists in the Secure Element',
+        '6d00': 'Invalid instruction',
+        '6e00': 'Invalid class',
+    }
+}
+
+
+class CardApplicationARAM(CardApplication):
+    def __init__(self):
+        super().__init__('ARA-M', adf=ADF_ARAM(), sw=sw_aram)

pySim/card_handler.py

@@ -1,7 +1,9 @@
-#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 
-""" pySim: card handler utilities
+""" pySim: card handler utilities.  A 'card handler' is some method
+by which cards can be inserted/removed into the card reader.  For
+normal smart card readers, this has to be done manually.  However,
+there are also automatic card feeders.
 """
 
 #
@@ -22,87 +24,124 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+from pySim.transport import LinkBase
 
 import subprocess
 import sys
 import yaml
 
-# Manual card handler: User is prompted to insert/remove card from the reader.
-class card_handler:
 
-	sl = None
+class CardHandlerBase:
+    """Abstract base class representing a mechanism for card insertion/removal."""
 
-	def __init__(self, sl):
-		self.sl = sl
+    def __init__(self, sl: LinkBase):
+        self.sl = sl
 
-	def get(self, first = False):
-		print("Ready for Programming: Insert card now (or CTRL-C to cancel)")
-		self.sl.wait_for_card(newcardonly=not first)
+    def get(self, first: bool = False):
+        """Method called when pySim needs a new card to be inserted.
 
-	def error(self):
-		print("Programming failed: Remove card from reader")
-		print("")
+        Args:
+                first : set to true when the get method is called the
+                        first time. This is required to prevent blocking
+                        when a card is already inserted into the reader.
+                        The reader API would not recognize that card as
+                        "new card" until it would be removed and re-inserted
+                        again.
+        """
+        print("Ready for Programming: ", end='')
+        self._get(first)
 
-	def done(self):
-		print("Programming successful: Remove card from reader")
-		print("")
+    def error(self):
+        """Method called when pySim failed to program a card. Move card to 'bad' batch."""
+        print("Programming failed: ", end='')
+        self._error()
 
-# Automatic card handler: A machine is used to handle the cards.
-class card_handler_auto:
+    def done(self):
+        """Method called when pySim failed to program a card. Move card to 'good' batch."""
+        print("Programming successful: ", end='')
+        self._done()
 
-	sl = None
-	cmds = None
-	verbose = True
+    def _get(self, first: bool = False):
+        pass
 
-	def __init__(self, sl, config_file):
-		print("Card handler Config-file: " + str(config_file))
-		self.sl = sl
-		with open(config_file) as cfg:
-			self.cmds = yaml.load(cfg, Loader=yaml.FullLoader)
+    def _error(self):
+        pass
 
-		self.verbose = (self.cmds.get('verbose') == True)
+    def _done(self):
+        pass
 
-	def __print_outout(self,out):
-		print("")
-		print("Card handler output:")
-		print("---------------------8<---------------------")
-		stdout = out[0].strip()
-		if len(stdout) > 0:
-			print("stdout:")
-			print(stdout)
-		stderr = out[1].strip()
-		if len(stderr) > 0:
-			print("stderr:")
-			print(stderr)
-		print("---------------------8<---------------------")
-		print("")
 
-	def __exec_cmd(self, command):
-		print("Card handler Commandline: " + str(command))
+class CardHandler(CardHandlerBase):
+    """Manual card handler: User is prompted to insert/remove card from the reader."""
 
-		proc = subprocess.Popen([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
-		out = proc.communicate()
-		rc = proc.returncode
+    def _get(self, first: bool = False):
+        print("Insert card now (or CTRL-C to cancel)")
+        self.sl.wait_for_card(newcardonly=not first)
 
-		if rc != 0 or self.verbose:
-			self.__print_outout(out)
+    def _error(self):
+        print("Remove card from reader")
+        print("")
 
-		if rc != 0:
-			print("")
-			print("Error: Card handler failure! (rc=" + str(rc) + ")")
-			sys.exit(rc)
+    def _done(self):
+        print("Remove card from reader")
+        print("")
 
-	def get(self, first = False):
-		print("Ready for Programming: Transporting card into the reader-bay...")
-		self.__exec_cmd(self.cmds['get'])
-		self.sl.connect()
 
-	def error(self):
-		print("Programming failed: Transporting card to the error-bin...")
-		self.__exec_cmd(self.cmds['error'])
-		print("")
+class CardHandlerAuto(CardHandlerBase):
+    """Automatic card handler: A machine is used to handle the cards."""
 
-	def done(self):
-		print("Programming successful: Transporting card into the collector bin...")
-		self.__exec_cmd(self.cmds['done'])
-		print("")
+    verbose = True
+
+    def __init__(self, sl: LinkBase, config_file: str):
+        super().__init__(sl)
+        print("Card handler Config-file: " + str(config_file))
+        with open(config_file) as cfg:
+            self.cmds = yaml.load(cfg, Loader=yaml.FullLoader)
+        self.verbose = (self.cmds.get('verbose') == True)
+
+    def __print_outout(self, out):
+        print("")
+        print("Card handler output:")
+        print("---------------------8<---------------------")
+        stdout = out[0].strip()
+        if len(stdout) > 0:
+            print("stdout:")
+            print(stdout)
+        stderr = out[1].strip()
+        if len(stderr) > 0:
+            print("stderr:")
+            print(stderr)
+        print("---------------------8<---------------------")
+        print("")
+
+    def __exec_cmd(self, command):
+        print("Card handler Commandline: " + str(command))
+
+        proc = subprocess.Popen(
+            [command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
+        out = proc.communicate()
+        rc = proc.returncode
+
+        if rc != 0 or self.verbose:
+            self.__print_outout(out)
+
+        if rc != 0:
+            print("")
+            print("Error: Card handler failure! (rc=" + str(rc) + ")")
+            sys.exit(rc)
+
+    def _get(self, first: bool = False):
+        print("Transporting card into the reader-bay...")
+        self.__exec_cmd(self.cmds['get'])
+        if self.sl:
+            self.sl.connect()
+
+    def _error(self):
+        print("Transporting card to the error-bin...")
+        self.__exec_cmd(self.cmds['error'])
+        print("")
+
+    def _done(self):
+        print("Transporting card into the collector bin...")
+        self.__exec_cmd(self.cmds['done'])
+        print("")

pySim/card_key_provider.py

@@ -0,0 +1,174 @@
+# coding=utf-8
+"""Obtaining card parameters (mostly key data) from external source.
+
+This module contains a base class and a concrete implementation of
+obtaining card key material (or other card-individual parameters) from
+an external data source.
+
+This is used e.g. to keep PIN/PUK data in some file on disk, avoiding
+the need of manually entering the related card-individual data on every
+operation with pySim-shell.
+"""
+
+# (C) 2021 by Sysmocom s.f.m.c. GmbH
+# All Rights Reserved
+#
+# Author: Philipp Maier
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import List, Dict, Optional
+
+import abc
+import csv
+
+card_key_providers = []  # type: List['CardKeyProvider']
+
+
+class CardKeyProvider(abc.ABC):
+    """Base class, not containing any concrete implementation."""
+
+    VALID_FIELD_NAMES = ['ICCID', 'ADM1',
+                         'IMSI', 'PIN1', 'PIN2', 'PUK1', 'PUK2']
+
+    # check input parameters, but do nothing concrete yet
+    def _verify_get_data(self, fields: List[str] = [], key: str = 'ICCID', value: str = "") -> Dict[str, str]:
+        """Verify multiple fields for identified card.
+
+        Args:
+                fields : list of valid field names such as 'ADM1', 'PIN1', ... which are to be obtained
+                key : look-up key to identify card data, such as 'ICCID'
+                value : value for look-up key to identify card data
+        Returns:
+                dictionary of {field, value} strings for each requested field from 'fields'
+        """
+        for f in fields:
+            if (f not in self.VALID_FIELD_NAMES):
+                raise ValueError("Requested field name '%s' is not a valid field name, valid field names are: %s" %
+                                 (f, str(self.VALID_FIELD_NAMES)))
+
+        if (key not in self.VALID_FIELD_NAMES):
+            raise ValueError("Key field name '%s' is not a valid field name, valid field names are: %s" %
+                             (key, str(self.VALID_FIELD_NAMES)))
+
+        return {}
+
+    def get_field(self, field: str, key: str = 'ICCID', value: str = "") -> Optional[str]:
+        """get a single field from CSV file using a specified key/value pair"""
+        fields = [field]
+        result = self.get(fields, key, value)
+        return result.get(field)
+
+    @abc.abstractmethod
+    def get(self, fields: List[str], key: str, value: str) -> Dict[str, str]:
+        """Get multiple card-individual fields for identified card.
+
+        Args:
+                fields : list of valid field names such as 'ADM1', 'PIN1', ... which are to be obtained
+                key : look-up key to identify card data, such as 'ICCID'
+                value : value for look-up key to identify card data
+        Returns:
+                dictionary of {field, value} strings for each requested field from 'fields'
+        """
+
+
+class CardKeyProviderCsv(CardKeyProvider):
+    """Card key provider implementation that allows to query against a specified CSV file"""
+    csv_file = None
+    filename = None
+
+    def __init__(self, filename: str):
+        """
+        Args:
+                filename : file name (path) of CSV file containing card-individual key/data
+        """
+        self.csv_file = open(filename, 'r')
+        if not self.csv_file:
+            raise RuntimeError("Could not open CSV file '%s'" % filename)
+        self.filename = filename
+
+    def get(self, fields: List[str], key: str, value: str) -> Dict[str, str]:
+        super()._verify_get_data(fields, key, value)
+
+        self.csv_file.seek(0)
+        cr = csv.DictReader(self.csv_file)
+        if not cr:
+            raise RuntimeError(
+                "Could not open DictReader for CSV-File '%s'" % self.filename)
+        cr.fieldnames = [field.upper() for field in cr.fieldnames]
+
+        rc = {}
+        for row in cr:
+            if row[key] == value:
+                for f in fields:
+                    if f in row:
+                        rc.update({f: row[f]})
+                    else:
+                        raise RuntimeError("CSV-File '%s' lacks column '%s'" %
+                                           (self.filename, f))
+        return rc
+
+
+def card_key_provider_register(provider: CardKeyProvider, provider_list=card_key_providers):
+    """Register a new card key provider.
+
+    Args:
+            provider : the to-be-registered provider
+            provider_list : override the list of providers from the global default
+    """
+    if not isinstance(provider, CardKeyProvider):
+        raise ValueError("provider is not a card data provier")
+    provider_list.append(provider)
+
+
+def card_key_provider_get(fields, key: str, value: str, provider_list=card_key_providers) -> Dict[str, str]:
+    """Query all registered card data providers for card-individual [key] data.
+
+    Args:
+            fields : list of valid field names such as 'ADM1', 'PIN1', ... which are to be obtained
+            key : look-up key to identify card data, such as 'ICCID'
+            value : value for look-up key to identify card data
+            provider_list : override the list of providers from the global default
+    Returns:
+            dictionary of {field, value} strings for each requested field from 'fields'
+    """
+    for p in provider_list:
+        if not isinstance(p, CardKeyProvider):
+            raise ValueError(
+                "provider list contains element which is not a card data provier")
+        result = p.get(fields, key, value)
+        if result:
+            return result
+    return {}
+
+
+def card_key_provider_get_field(field: str, key: str, value: str, provider_list=card_key_providers) -> Optional[str]:
+    """Query all registered card data providers for a single field.
+
+    Args:
+            field : name valid field such as 'ADM1', 'PIN1', ... which is to be obtained
+            key : look-up key to identify card data, such as 'ICCID'
+            value : value for look-up key to identify card data
+            provider_list : override the list of providers from the global default
+    Returns:
+            dictionary of {field, value} strings for the requested field
+    """
+    for p in provider_list:
+        if not isinstance(p, CardKeyProvider):
+            raise ValueError(
+                "provider list contains element which is not a card data provier")
+        result = p.get_field(field, key, value)
+        if result:
+            return result
+    return None

pySim/cat.py

@@ -0,0 +1,333 @@
+"""Code related to the Card Application Toolkit (CAT) as described in
+mainly) ETSI TS 102 223, ETSI TS 101 220 and 3GPP TS 31.111."""
+
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from pySim.tlv import *
+from pySim.construct import *
+from construct import *
+
+# Tag values as per TS 101 220 Table 7.23
+
+# TS 102 223 Section 8.1
+
+
+class Address(COMPR_TLV_IE, tag=0x06):
+    _construct = Struct('ton_npi'/Int8ub,
+                        'call_number'/BcdAdapter(Bytes(this._.total_len-1)))
+
+# TS 102 223 Section 8.2
+
+
+class AlphaIdentifier(COMPR_TLV_IE, tag=0x05):
+    # FIXME: like EF.ADN
+    pass
+
+# TS 102 223 Section 8.3
+
+
+class Subaddress(COMPR_TLV_IE, tag=0x08):
+    pass
+
+# TS 102 223 Section 8.4
+
+
+class CapabilityConfigParams(COMPR_TLV_IE, tag=0x07):
+    pass
+
+# TS 31.111 Section 8.5
+
+
+class CBSPage(COMPR_TLV_IE, tag=0x0C):
+    pass
+
+# TS 102 223 Section 8.6
+
+
+class CommandDetails(COMPR_TLV_IE, tag=0x01):
+    _construct = Struct('command_number'/Int8ub,
+                        'type_of_command'/Int8ub,
+                        'command_qualifier'/Int8ub)
+
+# TS 102 223 Section 8.7
+
+
+class DeviceIdentities(COMPR_TLV_IE, tag=0x82):
+    DEV_IDS = bidict({
+        0x01: 'keypad',
+        0x02: 'display',
+        0x03: 'earpiece',
+        0x10: 'addl_card_reader_0',
+        0x11: 'addl_card_reader_1',
+        0x12: 'addl_card_reader_2',
+        0x13: 'addl_card_reader_3',
+        0x14: 'addl_card_reader_4',
+        0x15: 'addl_card_reader_5',
+        0x16: 'addl_card_reader_6',
+        0x17: 'addl_card_reader_7',
+        0x21: 'channel_1',
+        0x22: 'channel_2',
+        0x23: 'channel_3',
+        0x24: 'channel_4',
+        0x25: 'channel_5',
+        0x26: 'channel_6',
+        0x27: 'channel_7',
+        0x31: 'ecat_client_1',
+        0x32: 'ecat_client_2',
+        0x33: 'ecat_client_3',
+        0x34: 'ecat_client_4',
+        0x35: 'ecat_client_5',
+        0x36: 'ecat_client_6',
+        0x37: 'ecat_client_7',
+        0x38: 'ecat_client_8',
+        0x39: 'ecat_client_9',
+        0x3a: 'ecat_client_a',
+        0x3b: 'ecat_client_b',
+        0x3c: 'ecat_client_c',
+        0x3d: 'ecat_client_d',
+        0x3e: 'ecat_client_e',
+        0x3f: 'ecat_client_f',
+        0x81: 'uicc',
+        0x82: 'terminal',
+        0x83: 'network',
+    })
+
+    def _from_bytes(self, do: bytes):
+        return {'source_dev_id': self.DEV_IDS[do[0]], 'dest_dev_id': self.DEV_IDS[do[1]]}
+
+    def _to_bytes(self):
+        src = self.DEV_IDS.inverse[self.decoded['source_dev_id']]
+        dst = self.DEV_IDS.inverse[self.decoded['dest_dev_id']]
+        return bytes([src, dst])
+
+# TS 102 223 Section 8.8
+
+
+class Duration(COMPR_TLV_IE, tag=0x04):
+    _construct = Struct('time_unit'/Int8ub,
+                        'time_interval'/Int8ub)
+
+# TS 102 223 Section 8.9
+
+
+class Item(COMPR_TLV_IE, tag=0x0f):
+    _construct = Struct('identifier'/Int8ub,
+                        'text_string'/GsmStringAdapter(GreedyBytes))
+
+# TS 102 223 Section 8.10
+
+
+class ItemIdentifier(COMPR_TLV_IE, tag=0x10):
+    _construct = Struct('identifier'/Int8ub)
+
+# TS 102 223 Section 8.11
+
+
+class ResponseLength(COMPR_TLV_IE, tag=0x11):
+    _construct = Struct('minimum_length'/Int8ub,
+                        'maximum_length'/Int8ub)
+
+# TS 102 223 Section 8.12
+
+
+class Result(COMPR_TLV_IE, tag=0x03):
+    _construct = Struct('general_result'/Int8ub,
+                        'additional_information'/HexAdapter(GreedyBytes))
+
+
+# TS 102 223 Section 8.13  + TS 31.111 Section 8.13
+class SMS_TPDU(COMPR_TLV_IE, tag=0x8B):
+    _construct = Struct('tpdu'/HexAdapter(GreedyBytes))
+
+# TS 102 223 Section 8.15
+
+
+class TextString(COMPR_TLV_IE, tag=0x0d):
+    _construct = Struct('dcs'/Int8ub,
+                        'text_string'/HexAdapter(GreedyBytes))
+
+# TS 102 223 Section 8.16
+
+
+class Tone(COMPR_TLV_IE, tag=0x0e):
+    _construct = Struct('tone'/Int8ub)
+
+# TS 31 111 Section 8.17
+
+
+class USSDString(COMPR_TLV_IE, tag=0x0a):
+    _construct = Struct('dcs'/Int8ub,
+                        'ussd_string'/HexAdapter(GreedyBytes))
+
+
+# TS 101 220 Table 7.17
+class ProactiveCommand(BER_TLV_IE, tag=0xD0):
+    pass
+
+# TS 101 220 Table 7.17 + 31.111 7.1.1.2
+
+
+class SMSPPDownload(BER_TLV_IE, tag=0xD1,
+                    nested=[DeviceIdentities, Address, SMS_TPDU]):
+    pass
+
+# TS 101 220 Table 7.17 + 31.111 7.1.1.3
+
+
+class SMSCBDownload(BER_TLV_IE, tag=0xD2,
+                    nested=[DeviceIdentities, CBSPage]):
+    pass
+
+
+class USSDDownload(BER_TLV_IE, tag=0xD9,
+                   nested=[DeviceIdentities, USSDString]):
+    pass
+
+
+# reasonable default for playing with OTA
+# 010203040506070809101112131415161718192021222324252627282930313233
+# '7fe1e10e000000000000001f43000000ff00000000000000000000000000000000'
+
+# TS 102 223 Section 5.2
+term_prof_bits = {
+    # first byte
+    1: 'Profile download',
+    2: 'SMS-PP data download',
+    3: 'Cell Broadcast data download',
+    4: 'Menu selection',
+    5: 'SMS-PP data download',
+    6: 'Timer expiration',
+    7: 'USSD string DO support in CC by USIM',
+    8: 'Call Control by NAA',
+
+    # first byte
+    9: 'Command result',
+    10: 'Call Control by NAA',
+    11: 'Call Control by NAA',
+    12: 'MO short message control support',
+    13: 'Call Control by NAA',
+    14: 'UCS2 Entry supported',
+    15: 'UCS2 Display supported',
+    16: 'Display Text',
+
+    # third byte
+    17: 'Proactive UICC: DISPLAY TEXT',
+    18: 'Proactive UICC: GET INKEY',
+    19: 'Proactive UICC: GET INPUT',
+    20: 'Proactive UICC: MORE TIME',
+    21: 'Proactive UICC: PLAY TONE',
+    22: 'Proactive UICC: POLL INTERVAL',
+    23: 'Proactive UICC: POLLING OFF',
+    24: 'Proactive UICC: REFRESH',
+
+    # fourth byte
+    25: 'Proactive UICC: SELECT ITEM',
+    26: 'Proactive UICC: SEND SHORT MESSAGE with 3GPP-SMS-TPDU',
+    27: 'Proactive UICC: SEND SS',
+    28: 'Proactive UICC: SEND USSD',
+    29: 'Proactive UICC: SET UP CALL',
+    30: 'Proactive UICC: SET UP MENU',
+    31: 'Proactive UICC: PROVIDE LOCAL INFORMATION (MCC, MNC, LAC, Cell ID & IMEI)',
+    32: 'Proactive UICC: PROVIDE LOCAL INFORMATION (NMR)',
+
+    # fifth byte
+    33: 'Proactive UICC: SET UP EVENT LIST',
+    34: 'Event: MT call',
+    35: 'Event: Call connected',
+    36: 'Event: Call disconnected',
+    37: 'Event: Location status',
+    38: 'Event: User activity',
+    39: 'Event: Idle screen available',
+    40: 'Event: Card reader status',
+
+    # sixth byte
+    41: 'Event: Language selection',
+    42: 'Event: Browser Termination',
+    43: 'Event: Data aailable',
+    44: 'Event: Channel status',
+    45: 'Event: Access Technology Change',
+    46: 'Event: Display parameters changed',
+    47: 'Event: Local Connection',
+    48: 'Event: Network Search Mode Change',
+
+    # seventh byte
+    49: 'Proactive UICC: POWER ON CARD',
+    50: 'Proactive UICC: POWER OFF CARD',
+    51: 'Proactive UICC: PERFORM CARD RESET',
+    52: 'Proactive UICC: GET READER STATUS (Card reader status)',
+    53: 'Proactive UICC: GET READER STATUS (Card reader identifier)',
+    # RFU: 3 bit (54,55,56)
+
+    # eighth byte
+    57: 'Proactive UICC: TIMER MANAGEMENT (start, stop)',
+    58: 'Proactive UICC: TIMER MANAGEMENT (get current value)',
+    59: 'Proactive UICC: PROVIDE LOCAL INFORMATION (date, time and time zone)',
+    60: 'GET INKEY',
+    61: 'SET UP IDLE MODE TEXT',
+    62: 'RUN AT COMMAND',
+    63: 'SETUP CALL',
+    64: 'Call Control by NAA',
+
+    # ninth byte
+    65: 'DISPLAY TEXT',
+    66: 'SEND DTMF command',
+    67: 'Proactive UICC: PROVIDE LOCAL INFORMATION (NMR)',
+    68: 'Proactive UICC: PROVIDE LOCAL INFORMATION (language)',
+    69: 'Proactive UICC: PROVIDE LOCAL INFORMATION (Timing Advance)',
+    70: 'Proactive UICC: LANGUAGE NOTIFICATION',
+    71: 'Proactive UICC: LAUNCH BROWSER',
+    72: 'Proactive UICC: PROVIDE LOCAL INFORMATION (Access Technology)',
+
+    # tenth byte
+    73: 'Soft keys support for SELECT ITEM',
+    74: 'Soft keys support for SET UP MENU ITEM',
+    # RFU: 6 bit (75-80)
+
+    # eleventh byte: max number of soft keys as 8bit value (81..88)
+
+    # twelfth byte
+    89: 'Proactive UICC: OPEN CHANNEL',
+    90: 'Proactive UICC: CLOSE CHANNEL',
+    91: 'Proactive UICC: RECEIVE DATA',
+    92: 'Proactive UICC: SEND DATA',
+    93: 'Proactive UICC: GET CHANNEL STATUS',
+    94: 'Proactive UICC: SERVICE SEARCH',
+    95: 'Proactive UICC: GET SERVICE INFORMATION',
+    96: 'Proactive UICC: DECLARE SERVICE',
+
+    # thirteenth byte
+    97: 'BIP supported Bearer: CSD',
+    98: 'BIP supported Bearer: GPRS',
+    99: 'BIP supported Bearer: Bluetooth',
+    100: 'BIP supported Bearer: IrDA',
+    101: 'BIP supported Bearer: RS232',
+    # 3 bits: number of channels supported (102..104)
+
+    # fourtheenth byte (screen height)
+    # fifteenth byte (screen width)
+    # sixeenth byte (screen effects)
+    # seventeenth byte (BIP supported bearers)
+    129: 'BIP: TCP, UICC in client mode, remote connection',
+    130: 'BIP: UDP, UICC in client mode, remote connection',
+    131: 'BIP: TCP, UICC in server mode',
+    132: 'BIP: TCP, UICC in client mode, local connection',
+    133: 'BIP: UDP, UICC in client mode, local connection',
+    134: 'BIP: direct communication channel',
+    # 2 bits reserved: 135, 136
+
+    # FIXME: remainder
+}

pySim/commands.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 """ pySim: SIM Card commands according to ISO 7816-4 and TS 11.11
@@ -6,7 +5,7 @@
 
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
-# Copyright (C) 2010       Harald Welte <laforge@gnumonks.org>
+# Copyright (C) 2010-2021  Harald Welte <laforge@gnumonks.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -22,185 +21,579 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from pySim.utils import rpad, b2h
+from construct import *
+from pySim.construct import LV
+from pySim.utils import rpad, b2h, h2b, sw_match, bertlv_encode_len, Hexstr, h2i, str_sanitize
+from pySim.exceptions import SwMatchError
+
 
 class SimCardCommands(object):
-	def __init__(self, transport):
-		self._tp = transport
-		self._cla_byte = "a0"
-		self.sel_ctrl = "0000"
+    def __init__(self, transport):
+        self._tp = transport
+        self.cla_byte = "a0"
+        self.sel_ctrl = "0000"
 
-	# Extract a single FCP item from TLV
-	def __parse_fcp(self, fcp):
-		# see also: ETSI TS 102 221, chapter 11.1.1.3.1 Response for MF,
-		# DF or ADF
-		from pytlv.TLV import TLV
-		tlvparser = TLV(['82', '83', '84', 'a5', '8a', '8b', '8c', '80', 'ab', 'c6', '81', '88'])
+    # Extract a single FCP item from TLV
+    def __parse_fcp(self, fcp):
+        # see also: ETSI TS 102 221, chapter 11.1.1.3.1 Response for MF,
+        # DF or ADF
+        from pytlv.TLV import TLV
+        tlvparser = TLV(['82', '83', '84', 'a5', '8a', '8b',
+                        '8c', '80', 'ab', 'c6', '81', '88'])
 
-		# pytlv is case sensitive!
-		fcp = fcp.lower()
+        # pytlv is case sensitive!
+        fcp = fcp.lower()
 
-		if fcp[0:2] != '62':
-			raise ValueError('Tag of the FCP template does not match, expected 62 but got %s'%fcp[0:2])
+        if fcp[0:2] != '62':
+            raise ValueError(
+                'Tag of the FCP template does not match, expected 62 but got %s' % fcp[0:2])
 
-		# Unfortunately the spec is not very clear if the FCP length is
-		# coded as one or two byte vale, so we have to try it out by
-		# checking if the length of the remaining TLV string matches
-		# what we get in the length field.
-		# See also ETSI TS 102 221, chapter 11.1.1.3.0 Base coding.
-		exp_tlv_len = int(fcp[2:4], 16)
-		if len(fcp[4:]) // 2 == exp_tlv_len:
-			skip = 4
-		else:
-			exp_tlv_len = int(fcp[2:6], 16)
-			if len(fcp[4:]) // 2 == exp_tlv_len:
-				skip = 6
+        # Unfortunately the spec is not very clear if the FCP length is
+        # coded as one or two byte vale, so we have to try it out by
+        # checking if the length of the remaining TLV string matches
+        # what we get in the length field.
+        # See also ETSI TS 102 221, chapter 11.1.1.3.0 Base coding.
+        exp_tlv_len = int(fcp[2:4], 16)
+        if len(fcp[4:]) // 2 == exp_tlv_len:
+            skip = 4
+        else:
+            exp_tlv_len = int(fcp[2:6], 16)
+            if len(fcp[4:]) // 2 == exp_tlv_len:
+                skip = 6
 
-		# Skip FCP tag and length
-		tlv = fcp[skip:]
-		return tlvparser.parse(tlv)
+        # Skip FCP tag and length
+        tlv = fcp[skip:]
+        return tlvparser.parse(tlv)
 
-	# Tell the length of a record by the card response
-	# USIMs respond with an FCP template, which is different
-	# from what SIMs responds. See also:
-	# USIM: ETSI TS 102 221, chapter 11.1.1.3 Response Data
-	# SIM: GSM 11.11, chapter 9.2.1 SELECT
-	def __record_len(self, r):
-		if self.sel_ctrl == "0004":
-			tlv_parsed = self.__parse_fcp(r[-1])
-			file_descriptor = tlv_parsed['82']
-			# See also ETSI TS 102 221, chapter 11.1.1.4.3 File Descriptor
-			return int(file_descriptor[4:8], 16)
-		else:
-			return int(r[-1][28:30], 16)
+    # Tell the length of a record by the card response
+    # USIMs respond with an FCP template, which is different
+    # from what SIMs responds. See also:
+    # USIM: ETSI TS 102 221, chapter 11.1.1.3 Response Data
+    # SIM: GSM 11.11, chapter 9.2.1 SELECT
+    def __record_len(self, r) -> int:
+        if self.sel_ctrl == "0004":
+            tlv_parsed = self.__parse_fcp(r[-1])
+            file_descriptor = tlv_parsed['82']
+            # See also ETSI TS 102 221, chapter 11.1.1.4.3 File Descriptor
+            return int(file_descriptor[4:8], 16)
+        else:
+            return int(r[-1][28:30], 16)
 
-	# Tell the length of a binary file. See also comment
-	# above.
-	def __len(self, r):
-		if self.sel_ctrl == "0004":
-			tlv_parsed = self.__parse_fcp(r[-1])
-			return int(tlv_parsed['80'], 16)
-		else:
-			return int(r[-1][4:8], 16)
+    # Tell the length of a binary file. See also comment
+    # above.
+    def __len(self, r) -> int:
+        if self.sel_ctrl == "0004":
+            tlv_parsed = self.__parse_fcp(r[-1])
+            return int(tlv_parsed['80'], 16)
+        else:
+            return int(r[-1][4:8], 16)
 
-	def get_atr(self):
-		return self._tp.get_atr()
+    def get_atr(self) -> str:
+        """Return the ATR of the currently inserted card."""
+        return self._tp.get_atr()
 
-	@property
-	def cla_byte(self):
-		return self._cla_byte
-	@cla_byte.setter
-	def cla_byte(self, value):
-		self._cla_byte = value
+    def try_select_path(self, dir_list):
+        """ Try to select a specified path
 
-	@property
-	def sel_ctrl(self):
-		return self._sel_ctrl
-	@sel_ctrl.setter
-	def sel_ctrl(self, value):
-		self._sel_ctrl = value
+        Args:
+                dir_list : list of hex-string FIDs
+        """
 
-	def try_select_file(self, dir_list):
-		rv = []
-		if type(dir_list) is not list:
-			dir_list = [dir_list]
-		for i in dir_list:
-			data, sw = self._tp.send_apdu(self.cla_byte + "a4" + self.sel_ctrl + "02" + i)
-			rv.append((data, sw))
-			if sw != '9000':
-				return rv
-		return rv
+        rv = []
+        if type(dir_list) is not list:
+            dir_list = [dir_list]
+        for i in dir_list:
+            data, sw = self._tp.send_apdu(
+                self.cla_byte + "a4" + self.sel_ctrl + "02" + i)
+            rv.append((data, sw))
+            if sw != '9000':
+                return rv
+        return rv
 
-	def select_file(self, dir_list):
-		rv = []
-		if type(dir_list) is not list:
-			dir_list = [dir_list]
-		for i in dir_list:
-			data, sw = self._tp.send_apdu_checksw(self.cla_byte + "a4" + self.sel_ctrl + "02" + i)
-			rv.append(data)
-		return rv
+    def select_path(self, dir_list):
+        """Execute SELECT for an entire list/path of FIDs.
 
-	def select_adf(self, aid):
-		aidlen = ("0" + format(len(aid) // 2, 'x'))[-2:]
-		return self._tp.send_apdu_checksw(self.cla_byte + "a4" + "0404" + aidlen + aid)
+        Args:
+                dir_list: list of FIDs representing the path to select
 
-	def read_binary(self, ef, length=None, offset=0):
-		r = self.select_file(ef)
-		if len(r[-1]) == 0:
-			return (None, None)
-		if length is None:
-			length = self.__len(r) - offset
-		total_data = ''
-		while offset < length:
-			chunk_len = min(255, length-offset)
-			pdu = self.cla_byte + 'b0%04x%02x' % (offset, chunk_len)
-			data,sw = self._tp.send_apdu(pdu)
-			if sw == '9000':
-				total_data += data
-				offset += chunk_len
-			else:
-				raise ValueError('Failed to read (offset %d)' % (offset))
-		return total_data, sw
+        Returns:
+                list of return values (FCP in hex encoding) for each element of the path
+        """
+        rv = []
+        if type(dir_list) is not list:
+            dir_list = [dir_list]
+        for i in dir_list:
+            data, sw = self.select_file(i)
+            rv.append(data)
+        return rv
 
-	def update_binary(self, ef, data, offset=0, verify=False):
-		self.select_file(ef)
-		pdu = self.cla_byte + 'd6%04x%02x' % (offset, len(data) // 2) + data
-		res = self._tp.send_apdu_checksw(pdu)
-		if verify:
-			self.verify_binary(ef, data, offset)
-		return res
+    def select_file(self, fid: str):
+        """Execute SELECT a given file by FID.
 
-	def verify_binary(self, ef, data, offset=0):
-		res = self.read_binary(ef, len(data) // 2, offset)
-		if res[0].lower() != data.lower():
-			raise ValueError('Binary verification failed (expected %s, got %s)' % (data.lower(), res[0].lower()))
+        Args:
+                fid : file identifier as hex string
+        """
 
-	def read_record(self, ef, rec_no):
-		r = self.select_file(ef)
-		rec_length = self.__record_len(r)
-		pdu = self.cla_byte + 'b2%02x04%02x' % (rec_no, rec_length)
-		return self._tp.send_apdu(pdu)
+        return self._tp.send_apdu_checksw(self.cla_byte + "a4" + self.sel_ctrl + "02" + fid)
 
-	def update_record(self, ef, rec_no, data, force_len=False, verify=False):
-		r = self.select_file(ef)
-		if not force_len:
-			rec_length = self.__record_len(r)
-			if (len(data) // 2 != rec_length):
-				raise ValueError('Invalid data length (expected %d, got %d)' % (rec_length, len(data) // 2))
-		else:
-			rec_length = len(data) // 2
-		pdu = (self.cla_byte + 'dc%02x04%02x' % (rec_no, rec_length)) + data
-		res = self._tp.send_apdu_checksw(pdu)
-		if verify:
-			self.verify_record(ef, rec_no, data)
-		return res
+    def select_adf(self, aid: str):
+        """Execute SELECT a given Applicaiton ADF.
 
-	def verify_record(self, ef, rec_no, data):
-		res = self.read_record(ef, rec_no)
-		if res[0].lower() != data.lower():
-			raise ValueError('Record verification failed (expected %s, got %s)' % (data.lower(), res[0].lower()))
+        Args:
+                aid : application identifier as hex string
+        """
 
-	def record_size(self, ef):
-		r = self.select_file(ef)
-		return self.__record_len(r)
+        aidlen = ("0" + format(len(aid) // 2, 'x'))[-2:]
+        return self._tp.send_apdu_checksw(self.cla_byte + "a4" + "0404" + aidlen + aid)
 
-	def record_count(self, ef):
-		r = self.select_file(ef)
-		return self.__len(r) // self.__record_len(r)
+    def read_binary(self, ef, length: int = None, offset: int = 0):
+        """Execute READD BINARY.
 
-	def binary_size(self, ef):
-		r = self.select_file(ef)
-		return self.__len(r)
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+                length : number of bytes to read
+                offset : byte offset in file from which to start reading
+        """
+        r = self.select_path(ef)
+        if len(r[-1]) == 0:
+            return (None, None)
+        if length is None:
+            length = self.__len(r) - offset
+        if length < 0:
+            return (None, None)
 
-	def run_gsm(self, rand):
-		if len(rand) != 32:
-			raise ValueError('Invalid rand')
-		self.select_file(['3f00', '7f20'])
-		return self._tp.send_apdu(self.cla_byte + '88000010' + rand)
+        total_data = ''
+        chunk_offset = 0
+        while chunk_offset < length:
+            chunk_len = min(255, length-chunk_offset)
+            pdu = self.cla_byte + \
+                'b0%04x%02x' % (offset + chunk_offset, chunk_len)
+            try:
+                data, sw = self._tp.send_apdu_checksw(pdu)
+            except Exception as e:
+                raise ValueError('%s, failed to read (offset %d)' %
+                                 (str_sanitize(str(e)), offset))
+            total_data += data
+            chunk_offset += chunk_len
+        return total_data, sw
 
-	def reset_card(self):
-		return self._tp.reset_card()
+    def update_binary(self, ef, data: str, offset: int = 0, verify: bool = False, conserve: bool = False):
+        """Execute UPDATE BINARY.
 
-	def verify_chv(self, chv_no, code):
-		fc = rpad(b2h(code), 16)
-		return self._tp.send_apdu_checksw(self.cla_byte + '2000' + ('%02X' % chv_no) + '08' + fc)
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+                data : hex string of data to be written
+                offset : byte offset in file from which to start writing
+                verify : Whether or not to verify data after write
+        """
+        data_length = len(data) // 2
+
+        # Save write cycles by reading+comparing before write
+        if conserve:
+            data_current, sw = self.read_binary(ef, data_length, offset)
+            if data_current == data:
+                return None, sw
+
+        self.select_path(ef)
+        total_data = ''
+        chunk_offset = 0
+        while chunk_offset < data_length:
+            chunk_len = min(255, data_length - chunk_offset)
+            # chunk_offset is bytes, but data slicing is hex chars, so we need to multiply by 2
+            pdu = self.cla_byte + \
+                'd6%04x%02x' % (offset + chunk_offset, chunk_len) + \
+                data[chunk_offset*2: (chunk_offset+chunk_len)*2]
+            try:
+                chunk_data, chunk_sw = self._tp.send_apdu_checksw(pdu)
+            except Exception as e:
+                raise ValueError('%s, failed to write chunk (chunk_offset %d, chunk_len %d)' %
+                                 (str_sanitize(str(e)), chunk_offset, chunk_len))
+            total_data += data
+            chunk_offset += chunk_len
+        if verify:
+            self.verify_binary(ef, data, offset)
+        return total_data, chunk_sw
+
+    def verify_binary(self, ef, data: str, offset: int = 0):
+        """Verify contents of transparent EF.
+
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+                data : hex string of expected data
+                offset : byte offset in file from which to start verifying
+        """
+        res = self.read_binary(ef, len(data) // 2, offset)
+        if res[0].lower() != data.lower():
+            raise ValueError('Binary verification failed (expected %s, got %s)' % (
+                data.lower(), res[0].lower()))
+
+    def read_record(self, ef, rec_no: int):
+        """Execute READ RECORD.
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+                rec_no : record number to read
+        """
+        r = self.select_path(ef)
+        rec_length = self.__record_len(r)
+        pdu = self.cla_byte + 'b2%02x04%02x' % (rec_no, rec_length)
+        return self._tp.send_apdu_checksw(pdu)
+
+    def update_record(self, ef, rec_no: int, data: str, force_len: bool = False, verify: bool = False,
+                      conserve: bool = False):
+        """Execute UPDATE RECORD.
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+                rec_no : record number to read
+                data : hex string of data to be written
+                force_len : enforce record length by using the actual data length
+                verify : verify data by re-reading the record
+                conserve : read record and compare it with data, skip write on match
+        """
+        res = self.select_path(ef)
+
+        if force_len:
+            # enforce the record length by the actual length of the given data input
+            rec_length = len(data) // 2
+        else:
+            # determine the record length from the select response of the file and pad
+            # the input data with 0xFF if necessary. In cases where the input data
+            # exceed we throw an exception.
+            rec_length = self.__record_len(res)
+            if (len(data) // 2 > rec_length):
+                raise ValueError('Data length exceeds record length (expected max %d, got %d)' % (
+                    rec_length, len(data) // 2))
+            elif (len(data) // 2 < rec_length):
+                data = rpad(data, rec_length * 2)
+
+        # Save write cycles by reading+comparing before write
+        if conserve:
+            data_current, sw = self.read_record(ef, rec_no)
+            data_current = data_current[0:rec_length*2]
+            if data_current == data:
+                return None, sw
+
+        pdu = (self.cla_byte + 'dc%02x04%02x' % (rec_no, rec_length)) + data
+        res = self._tp.send_apdu_checksw(pdu)
+        if verify:
+            self.verify_record(ef, rec_no, data)
+        return res
+
+    def verify_record(self, ef, rec_no: int, data: str):
+        """Verify record against given data
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+                rec_no : record number to read
+                data : hex string of data to be verified
+        """
+        res = self.read_record(ef, rec_no)
+        if res[0].lower() != data.lower():
+            raise ValueError('Record verification failed (expected %s, got %s)' % (
+                data.lower(), res[0].lower()))
+
+    def record_size(self, ef):
+        """Determine the record size of given file.
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+        """
+        r = self.select_path(ef)
+        return self.__record_len(r)
+
+    def record_count(self, ef):
+        """Determine the number of records in given file.
+
+        Args:
+                ef : string or list of strings indicating name or path of linear fixed EF
+        """
+        r = self.select_path(ef)
+        return self.__len(r) // self.__record_len(r)
+
+    def binary_size(self, ef):
+        """Determine the size of given transparent file.
+
+        Args:
+                ef : string or list of strings indicating name or path of transparent EF
+        """
+        r = self.select_path(ef)
+        return self.__len(r)
+
+    # TS 102 221 Section 11.3.1 low-level helper
+    def _retrieve_data(self, tag: int, first: bool = True):
+        if first:
+            pdu = '80cb008001%02x' % (tag)
+        else:
+            pdu = '80cb000000'
+        return self._tp.send_apdu_checksw(pdu)
+
+    def retrieve_data(self, ef, tag: int):
+        """Execute RETRIEVE DATA, see also TS 102 221 Section 11.3.1.
+
+        Args
+                ef : string or list of strings indicating name or path of transparent EF
+                tag : BER-TLV Tag of value to be retrieved
+        """
+        r = self.select_path(ef)
+        if len(r[-1]) == 0:
+            return (None, None)
+        total_data = ''
+        # retrieve first block
+        data, sw = self._retrieve_data(tag, first=True)
+        total_data += data
+        while sw == '62f1' or sw == '62f2':
+            data, sw = self._retrieve_data(tag, first=False)
+            total_data += data
+        return total_data, sw
+
+    # TS 102 221 Section 11.3.2 low-level helper
+    def _set_data(self, data: str, first: bool = True):
+        if first:
+            p1 = 0x80
+        else:
+            p1 = 0x00
+        if isinstance(data, bytes) or isinstance(data, bytearray):
+            data = b2h(data)
+        pdu = '80db00%02x%02x%s' % (p1, len(data)//2, data)
+        return self._tp.send_apdu_checksw(pdu)
+
+    def set_data(self, ef, tag: int, value: str, verify: bool = False, conserve: bool = False):
+        """Execute SET DATA.
+
+        Args
+                ef : string or list of strings indicating name or path of transparent EF
+                tag : BER-TLV Tag of value to be stored
+                value : BER-TLV value to be stored
+        """
+        r = self.select_path(ef)
+        if len(r[-1]) == 0:
+            return (None, None)
+
+        # in case of deleting the data, we only have 'tag' but no 'value'
+        if not value:
+            return self._set_data('%02x' % tag, first=True)
+
+        # FIXME: proper BER-TLV encode
+        tl = '%02x%s' % (tag, b2h(bertlv_encode_len(len(value)//2)))
+        tlv = tl + value
+        tlv_bin = h2b(tlv)
+
+        first = True
+        total_len = len(tlv_bin)
+        remaining = tlv_bin
+        while len(remaining) > 0:
+            fragment = remaining[:255]
+            rdata, sw = self._set_data(fragment, first=first)
+            first = False
+            remaining = remaining[255:]
+        return rdata, sw
+
+    def run_gsm(self, rand: str):
+        """Execute RUN GSM ALGORITHM.
+
+        Args:
+                rand : 16 byte random data as hex string (RAND)
+        """
+        if len(rand) != 32:
+            raise ValueError('Invalid rand')
+        self.select_path(['3f00', '7f20'])
+        return self._tp.send_apdu(self.cla_byte + '88000010' + rand)
+
+    def authenticate(self, rand: str, autn: str, context='3g'):
+        """Execute AUTHENTICATE (USIM/ISIM).
+
+        Args:
+                rand : 16 byte random data as hex string (RAND)
+                autn : 8 byte Autentication Token (AUTN)
+                context : 16 byte random data ('3g' or 'gsm')
+        """
+        # 3GPP TS 31.102 Section 7.1.2.1
+        AuthCmd3G = Struct('rand'/LV, 'autn'/Optional(LV))
+        AuthResp3GSyncFail = Struct(Const(b'\xDC'), 'auts'/LV)
+        AuthResp3GSuccess = Struct(
+            Const(b'\xDB'), 'res'/LV, 'ck'/LV, 'ik'/LV, 'kc'/Optional(LV))
+        AuthResp3G = Select(AuthResp3GSyncFail, AuthResp3GSuccess)
+        # build parameters
+        cmd_data = {'rand': rand, 'autn': autn}
+        if context == '3g':
+            p2 = '81'
+        elif context == 'gsm':
+            p2 = '80'
+        (data, sw) = self._tp.send_apdu_constr_checksw(
+            self.cla_byte, '88', '00', p2, AuthCmd3G, cmd_data, AuthResp3G)
+        if 'auts' in data:
+            ret = {'synchronisation_failure': data}
+        else:
+            ret = {'successful_3g_authentication': data}
+        return (ret, sw)
+
+    def status(self):
+        """Execute a STATUS command as per TS 102 221 Section 11.1.2."""
+        return self._tp.send_apdu_checksw('80F20000ff')
+
+    def deactivate_file(self):
+        """Execute DECATIVATE FILE command as per TS 102 221 Section 11.1.14."""
+        return self._tp.send_apdu_constr_checksw(self.cla_byte, '04', '00', '00', None, None, None)
+
+    def activate_file(self, fid):
+        """Execute ACTIVATE FILE command as per TS 102 221 Section 11.1.15.
+
+        Args:
+                fid : file identifier as hex string
+        """
+        return self._tp.send_apdu_checksw(self.cla_byte + '44000002' + fid)
+
+    def manage_channel(self, mode='open', lchan_nr=0):
+        """Execute MANAGE CHANNEL command as per TS 102 221 Section 11.1.17.
+
+        Args:
+                mode : logical channel operation code ('open' or 'close')
+                lchan_nr : logical channel number (1-19, 0=assigned by UICC)
+        """
+        if mode == 'close':
+            p1 = 0x80
+        else:
+            p1 = 0x00
+        pdu = self.cla_byte + '70%02x%02x00' % (p1, lchan_nr)
+        return self._tp.send_apdu_checksw(pdu)
+
+    def reset_card(self):
+        """Physically reset the card"""
+        return self._tp.reset_card()
+
+    def _chv_process_sw(self, op_name, chv_no, pin_code, sw):
+        if sw_match(sw, '63cx'):
+            raise RuntimeError('Failed to %s chv_no 0x%02X with code 0x%s, %i tries left.' %
+                               (op_name, chv_no, b2h(pin_code).upper(), int(sw[3])))
+        elif (sw != '9000'):
+            raise SwMatchError(sw, '9000')
+
+    def verify_chv(self, chv_no: int, code: str):
+        """Verify a given CHV (Card Holder Verification == PIN)
+
+        Args:
+                chv_no : chv number (1=CHV1, 2=CHV2, ...)
+                code : chv code as hex string
+        """
+        fc = rpad(b2h(code), 16)
+        data, sw = self._tp.send_apdu(
+            self.cla_byte + '2000' + ('%02X' % chv_no) + '08' + fc)
+        self._chv_process_sw('verify', chv_no, code, sw)
+        return (data, sw)
+
+    def unblock_chv(self, chv_no: int, puk_code: str, pin_code: str):
+        """Unblock a given CHV (Card Holder Verification == PIN)
+
+        Args:
+                chv_no : chv number (1=CHV1, 2=CHV2, ...)
+                puk_code : puk code as hex string
+                pin_code : new chv code as hex string
+        """
+        fc = rpad(b2h(puk_code), 16) + rpad(b2h(pin_code), 16)
+        data, sw = self._tp.send_apdu(
+            self.cla_byte + '2C00' + ('%02X' % chv_no) + '10' + fc)
+        self._chv_process_sw('unblock', chv_no, pin_code, sw)
+        return (data, sw)
+
+    def change_chv(self, chv_no: int, pin_code: str, new_pin_code: str):
+        """Change a given CHV (Card Holder Verification == PIN)
+
+        Args:
+                chv_no : chv number (1=CHV1, 2=CHV2, ...)
+                pin_code : current chv code as hex string
+                new_pin_code : new chv code as hex string
+        """
+        fc = rpad(b2h(pin_code), 16) + rpad(b2h(new_pin_code), 16)
+        data, sw = self._tp.send_apdu(
+            self.cla_byte + '2400' + ('%02X' % chv_no) + '10' + fc)
+        self._chv_process_sw('change', chv_no, pin_code, sw)
+        return (data, sw)
+
+    def disable_chv(self, chv_no: int, pin_code: str):
+        """Disable a given CHV (Card Holder Verification == PIN)
+
+        Args:
+                chv_no : chv number (1=CHV1, 2=CHV2, ...)
+                pin_code : current chv code as hex string
+                new_pin_code : new chv code as hex string
+        """
+        fc = rpad(b2h(pin_code), 16)
+        data, sw = self._tp.send_apdu(
+            self.cla_byte + '2600' + ('%02X' % chv_no) + '08' + fc)
+        self._chv_process_sw('disable', chv_no, pin_code, sw)
+        return (data, sw)
+
+    def enable_chv(self, chv_no: int, pin_code: str):
+        """Enable a given CHV (Card Holder Verification == PIN)
+
+        Args:
+                chv_no : chv number (1=CHV1, 2=CHV2, ...)
+                pin_code : chv code as hex string
+        """
+        fc = rpad(b2h(pin_code), 16)
+        data, sw = self._tp.send_apdu(
+            self.cla_byte + '2800' + ('%02X' % chv_no) + '08' + fc)
+        self._chv_process_sw('enable', chv_no, pin_code, sw)
+        return (data, sw)
+
+    def envelope(self, payload: str):
+        """Send one ENVELOPE command to the SIM
+
+        Args:
+                payload : payload as hex string
+        """
+        return self._tp.send_apdu_checksw('80c20000%02x%s' % (len(payload)//2, payload))
+
+    def terminal_profile(self, payload: str):
+        """Send TERMINAL PROFILE to card
+
+        Args:
+                payload : payload as hex string
+        """
+        data_length = len(payload) // 2
+        data, sw = self._tp.send_apdu(('80100000%02x' % data_length) + payload)
+        return (data, sw)
+
+    # ETSI TS 102 221 11.1.22
+    def suspend_uicc(self, min_len_secs: int = 60, max_len_secs: int = 43200):
+        """Send SUSPEND UICC to the card.
+
+        Args:
+                 min_len_secs : mimumum suspend time seconds
+                 max_len_secs : maximum suspend time seconds
+        """
+        def encode_duration(secs: int) -> Hexstr:
+            if secs >= 10*24*60*60:
+                return '04%02x' % (secs // (10*24*60*60))
+            elif secs >= 24*60*60:
+                return '03%02x' % (secs // (24*60*60))
+            elif secs >= 60*60:
+                return '02%02x' % (secs // (60*60))
+            elif secs >= 60:
+                return '01%02x' % (secs // 60)
+            else:
+                return '00%02x' % secs
+
+        def decode_duration(enc: Hexstr) -> int:
+            time_unit = enc[:2]
+            length = h2i(enc[2:4])
+            if time_unit == '04':
+                return length * 10*24*60*60
+            elif time_unit == '03':
+                return length * 24*60*60
+            elif time_unit == '02':
+                return length * 60*60
+            elif time_unit == '01':
+                return length * 60
+            elif time_unit == '00':
+                return length
+            else:
+                raise ValueError('Time unit must be 0x00..0x04')
+        min_dur_enc = encode_duration(min_len_secs)
+        max_dur_enc = encode_duration(max_len_secs)
+        data, sw = self._tp.send_apdu_checksw(
+            '8076000004' + min_dur_enc + max_dur_enc)
+        negotiated_duration_secs = decode_duration(data[:4])
+        resume_token = data[4:]
+        return (negotiated_duration_secs, resume_token, sw)

pySim/construct.py

@@ -0,0 +1,186 @@
+from construct.lib.containers import Container, ListContainer
+from construct.core import EnumIntegerString
+import typing
+from construct import *
+from pySim.utils import b2h, h2b, swap_nibbles
+import gsm0338
+
+"""Utility code related to the integration of the 'construct' declarative parser."""
+
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+class HexAdapter(Adapter):
+    """convert a bytes() type to a string of hex nibbles."""
+
+    def _decode(self, obj, context, path):
+        return b2h(obj)
+
+    def _encode(self, obj, context, path):
+        return h2b(obj)
+
+
+class BcdAdapter(Adapter):
+    """convert a bytes() type to a string of BCD nibbles."""
+
+    def _decode(self, obj, context, path):
+        return swap_nibbles(b2h(obj))
+
+    def _encode(self, obj, context, path):
+        return h2b(swap_nibbles(obj))
+
+
+class Rpad(Adapter):
+    """
+    Encoder appends padding bytes (b'\\xff') up to target size.
+    Decoder removes trailing padding bytes.
+
+    Parameters:
+        subcon: Subconstruct as defined by construct library
+        pattern: set padding pattern (default: b'\\xff')
+    """
+
+    def __init__(self, subcon, pattern=b'\xff'):
+        super().__init__(subcon)
+        self.pattern = pattern
+
+    def _decode(self, obj, context, path):
+        return obj.rstrip(self.pattern)
+
+    def _encode(self, obj, context, path):
+        if len(obj) > self.sizeof():
+            raise SizeofError("Input ({}) exceeds target size ({})".format(
+                len(obj), self.sizeof()))
+        return obj + self.pattern * (self.sizeof() - len(obj))
+
+
+class GsmStringAdapter(Adapter):
+    """Convert GSM 03.38 encoded bytes to a string."""
+
+    def __init__(self, subcon, codec='gsm03.38', err='strict'):
+        super().__init__(subcon)
+        self.codec = codec
+        self.err = err
+
+    def _decode(self, obj, context, path):
+        return obj.decode(self.codec)
+
+    def _encode(self, obj, context, path):
+        return obj.encode(self.codec, self.err)
+
+
+def filter_dict(d, exclude_prefix='_'):
+    """filter the input dict to ensure no keys starting with 'exclude_prefix' remain."""
+    if not isinstance(d, dict):
+        return d
+    res = {}
+    for (key, value) in d.items():
+        if key.startswith(exclude_prefix):
+            continue
+        if type(value) is dict:
+            res[key] = filter_dict(value)
+        else:
+            res[key] = value
+    return res
+
+
+def normalize_construct(c):
+    """Convert a construct specific type to a related base type, mostly useful
+    so we can serialize it."""
+    # we need to include the filter_dict as we otherwise get elements like this
+    # in the dict: '_io': <_io.BytesIO object at 0x7fdb64e05860> which we cannot json-serialize
+    c = filter_dict(c)
+    if isinstance(c, Container) or isinstance(c, dict):
+        r = {k: normalize_construct(v) for (k, v) in c.items()}
+    elif isinstance(c, ListContainer):
+        r = [normalize_construct(x) for x in c]
+    elif isinstance(c, list):
+        r = [normalize_construct(x) for x in c]
+    elif isinstance(c, EnumIntegerString):
+        r = str(c)
+    else:
+        r = c
+    return r
+
+
+def parse_construct(c, raw_bin_data: bytes, length: typing.Optional[int] = None, exclude_prefix: str = '_'):
+    """Helper function to wrap around normalize_construct() and filter_dict()."""
+    if not length:
+        length = len(raw_bin_data)
+    parsed = c.parse(raw_bin_data, total_len=length)
+    return normalize_construct(parsed)
+
+
+# here we collect some shared / common definitions of data types
+LV = Prefixed(Int8ub, HexAdapter(GreedyBytes))
+
+# Default value for Reserved for Future Use (RFU) bits/bytes
+# See TS 31.101 Sec. "3.4 Coding Conventions"
+__RFU_VALUE = 0
+
+# Field that packs Reserved for Future Use (RFU) bit
+FlagRFU = Default(Flag, __RFU_VALUE)
+
+# Field that packs Reserved for Future Use (RFU) byte
+ByteRFU = Default(Byte, __RFU_VALUE)
+
+# Field that packs all remaining Reserved for Future Use (RFU) bytes
+GreedyBytesRFU = Default(GreedyBytes, b'')
+
+
+def BitsRFU(n=1):
+    '''
+    Field that packs Reserved for Future Use (RFU) bit(s)
+    as defined in TS 31.101 Sec. "3.4 Coding Conventions"
+
+    Use this for (currently) unused/reserved bits whose contents
+    should be initialized automatically but should not be cleared
+    in the future or when restoring read data (unlike padding).
+
+    Parameters:
+        n (Integer): Number of bits (default: 1)
+    '''
+    return Default(BitsInteger(n), __RFU_VALUE)
+
+
+def BytesRFU(n=1):
+    '''
+    Field that packs Reserved for Future Use (RFU) byte(s)
+    as defined in TS 31.101 Sec. "3.4 Coding Conventions"
+
+    Use this for (currently) unused/reserved bytes whose contents
+    should be initialized automatically but should not be cleared
+    in the future or when restoring read data (unlike padding).
+
+    Parameters:
+        n (Integer): Number of bytes (default: 1)
+    '''
+    return Default(Bytes(n), __RFU_VALUE)
+
+
+def GsmString(n):
+    '''
+    GSM 03.38 encoded byte string of fixed length n.
+    Encoder appends padding bytes (b'\\xff') to maintain
+    length. Decoder removes those trailing bytes.
+
+    Exceptions are raised for invalid characters
+    and length excess.
+
+    Parameters:
+        n (Integer): Fixed length of the encoded byte string
+    '''
+    return GsmStringAdapter(Rpad(Bytes(n), pattern=b'\xff'), codec='gsm03.38')

pySim/exceptions.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 """ pySim: Exceptions
@@ -6,6 +5,7 @@
 
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,19 +21,40 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
-
-try:
-	# This is for compatibility with python 2 and 3
-	from exceptions import Exception
-except:
-	pass
 
 class NoCardError(Exception):
-	pass
+    """No card was found in the reader."""
+    pass
+
 
 class ProtocolError(Exception):
-	pass
+    """Some kind of protocol level error interfacing with the card."""
+    pass
+
 
 class ReaderError(Exception):
-	pass
+    """Some kind of general error with the card reader."""
+    pass
+
+
+class SwMatchError(Exception):
+    """Raised when an operation specifies an expected SW but the actual SW from
+       the card doesn't match."""
+
+    def __init__(self, sw_actual: str, sw_expected: str, rs=None):
+        """
+        Args:
+                sw_actual : the SW we actually received from the card (4 hex digits)
+                sw_expected : the SW we expected to receive from the card (4 hex digits)
+                rs : interpreter class to convert SW to string
+        """
+        self.sw_actual = sw_actual
+        self.sw_expected = sw_expected
+        self.rs = rs
+
+    def __str__(self):
+        if self.rs:
+            r = self.rs.interpret_sw(self.sw_actual)
+            if r:
+                return "SW match failed! Expected %s and got %s: %s - %s" % (self.sw_expected, self.sw_actual, r[0], r[1])
+        return "SW match failed! Expected %s and got %s." % (self.sw_expected, self.sw_actual)

pySim/gsm_r.py

@@ -0,0 +1,306 @@
+# -*- coding: utf-8 -*-
+
+# without this, pylint will fail when inner classes are used
+# within the 'nested' kwarg of our TlvMeta metaclass on python 3.7 :(
+# pylint: disable=undefined-variable
+
+"""
+The File (and its derived classes) uses the classes of pySim.filesystem in
+order to describe the files specified in UIC Reference P38 T 9001 5.0 "FFFIS for GSM-R SIM Cards"
+"""
+
+#
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+from pySim.utils import *
+#from pySim.tlv import *
+from struct import pack, unpack
+from construct import *
+from construct import Optional as COptional
+from pySim.construct import *
+import enum
+
+from pySim.filesystem import *
+import pySim.ts_102_221
+import pySim.ts_51_011
+
+######################################################################
+# DF.EIRENE (FFFIS for GSM-R SIM Cards)
+######################################################################
+
+
+class FuncNTypeAdapter(Adapter):
+    def _decode(self, obj, context, path):
+        bcd = swap_nibbles(b2h(obj))
+        last_digit = bcd[-1]
+        return {'functional_number': bcd[:-1],
+                'presentation_of_only_this_fn': last_digit & 4,
+                'permanent_fn': last_digit & 8}
+
+    def _encode(self, obj, context, path):
+        return 'FIXME'
+
+
+class EF_FN(LinFixedEF):
+    """Section 7.2"""
+
+    def __init__(self):
+        super().__init__(fid='6ff1', sfid=None, name='EF.EN',
+                         desc='Functional numbers', rec_len={9, 9})
+        self._construct = Struct('functional_number_and_type'/FuncNTypeAdapter(Bytes(8)),
+                                 'list_number'/Int8ub)
+
+
+class PlConfAdapter(Adapter):
+    """Section 7.4.3"""
+
+    def _decode(self, obj, context, path):
+        num = int(obj) & 0x7
+        if num == 0:
+            return 'None'
+        elif num == 1:
+            return 4
+        elif num == 2:
+            return 3
+        elif num == 3:
+            return 2
+        elif num == 4:
+            return 1
+        elif num == 5:
+            return 0
+
+    def _encode(self, obj, context, path):
+        if obj == 'None':
+            return 0
+        obj = int(obj)
+        if obj == 4:
+            return 1
+        elif obj == 3:
+            return 2
+        elif obj == 2:
+            return 3
+        elif obj == 1:
+            return 4
+        elif obj == 0:
+            return 5
+
+
+class PlCallAdapter(Adapter):
+    """Section 7.4.12"""
+
+    def _decode(self, obj, context, path):
+        num = int(obj) & 0x7
+        if num == 0:
+            return 'None'
+        elif num == 1:
+            return 4
+        elif num == 2:
+            return 3
+        elif num == 3:
+            return 2
+        elif num == 4:
+            return 1
+        elif num == 5:
+            return 0
+        elif num == 6:
+            return 'B'
+        elif num == 7:
+            return 'A'
+
+    def _encode(self, obj, context, path):
+        if obj == 'None':
+            return 0
+        if obj == 4:
+            return 1
+        elif obj == 3:
+            return 2
+        elif obj == 2:
+            return 3
+        elif obj == 1:
+            return 4
+        elif obj == 0:
+            return 5
+        elif obj == 'B':
+            return 6
+        elif obj == 'A':
+            return 7
+
+
+NextTableType = Enum(Byte, decision=0xf0, predefined=0xf1,
+                     num_dial_digits=0xf2, ic=0xf3, empty=0xff)
+
+
+class EF_CallconfC(TransparentEF):
+    """Section 7.3"""
+
+    def __init__(self):
+        super().__init__(fid='6ff2', sfid=None, name='EF.CallconfC', size={24, 24},
+                         desc='Call Configuration of emergency calls Configuration')
+        self._construct = Struct('pl_conf'/PlConfAdapter(Int8ub),
+                                 'conf_nr'/BcdAdapter(Bytes(8)),
+                                 'max_rand'/Int8ub,
+                                 'n_ack_max'/Int16ub,
+                                 'pl_ack'/PlCallAdapter(Int8ub),
+                                 'n_nested_max'/Int8ub,
+                                 'train_emergency_gid'/Int8ub,
+                                 'shunting_emergency_gid'/Int8ub,
+                                 'imei'/BcdAdapter(Bytes(8)))
+
+
+class EF_CallconfI(LinFixedEF):
+    """Section 7.5"""
+
+    def __init__(self):
+        super().__init__(fid='6ff3', sfid=None, name='EF.CallconfI', rec_len={21, 21},
+                         desc='Call Configuration of emergency calls Information')
+        self._construct = Struct('t_dur'/Int24ub,
+                                 't_relcalc'/Int32ub,
+                                 'pl_call'/PlCallAdapter(Int8ub),
+                                 'cause' /
+                                 FlagsEnum(Int8ub, powered_off=1,
+                                           radio_link_error=2, user_command=5),
+                                 'gcr'/BcdAdapter(Bytes(4)),
+                                 'fnr'/BcdAdapter(Bytes(8)))
+
+
+class EF_Shunting(TransparentEF):
+    """Section 7.6"""
+
+    def __init__(self):
+        super().__init__(fid='6ff4', sfid=None,
+                         name='EF.Shunting', desc='Shunting', size={8, 8})
+        self._construct = Struct('common_gid'/Int8ub,
+                                 'shunting_gid'/Bytes(7))
+
+
+class EF_GsmrPLMN(LinFixedEF):
+    """Section 7.7"""
+
+    def __init__(self):
+        super().__init__(fid='6ff5', sfid=None, name='EF.GsmrPLMN',
+                         desc='GSM-R network selection', rec_len={9, 9})
+        self._construct = Struct('plmn'/BcdAdapter(Bytes(3)),
+                                 'class_of_network'/BitStruct('supported'/FlagsEnum(BitsInteger(5), vbs=1, vgcs=2, emlpp=4, fn=8, eirene=16),
+                                                              'preference'/BitsInteger(3)),
+                                 'ic_incoming_ref_tbl'/HexAdapter(Bytes(2)),
+                                 'outgoing_ref_tbl'/HexAdapter(Bytes(2)),
+                                 'ic_table_ref'/HexAdapter(Bytes(1)))
+
+
+class EF_IC(LinFixedEF):
+    """Section 7.8"""
+
+    def __init__(self):
+        super().__init__(fid='6f8d', sfid=None, name='EF.IC',
+                         desc='International Code', rec_len={7, 7})
+        self._construct = Struct('next_table_type'/NextTableType,
+                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'ic_decision_value'/BcdAdapter(Bytes(2)),
+                                 'network_string_table_index'/Int8ub)
+
+
+class EF_NW(LinFixedEF):
+    """Section 7.9"""
+
+    def __init__(self):
+        super().__init__(fid='6f80', sfid=None, name='EF.NW',
+                         desc='Network Name', rec_len={8, 8})
+        self._construct = GsmString(8)
+
+
+class EF_Switching(LinFixedEF):
+    """Section 8.4"""
+
+    def __init__(self, fid, name, desc):
+        super().__init__(fid=fid, sfid=None,
+                         name=name, desc=desc, rec_len={6, 6})
+        self._construct = Struct('next_table_type'/NextTableType,
+                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'decision_value'/BcdAdapter(Bytes(2)),
+                                 'string_table_index'/Int8ub)
+
+
+class EF_Predefined(LinFixedEF):
+    """Section 8.5"""
+
+    def __init__(self, fid, name, desc):
+        super().__init__(fid=fid, sfid=None,
+                         name=name, desc=desc, rec_len={3, 3})
+        # header and other records have different structure. WTF !?!
+        self._construct = Struct('next_table_type'/NextTableType,
+                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'predefined_value1'/HexAdapter(Bytes(2)),
+                                 'string_table_index1'/Int8ub)
+        # TODO: predefined value n, ...
+
+
+class EF_DialledVals(TransparentEF):
+    """Section 8.6"""
+
+    def __init__(self, fid, name, desc):
+        super().__init__(fid=fid, sfid=None, name=name, desc=desc, size={4, 4})
+        self._construct = Struct('next_table_type'/NextTableType,
+                                 'id_of_next_table'/HexAdapter(Bytes(2)),
+                                 'dialed_digits'/BcdAdapter(Bytes(1)))
+
+
+class DF_EIRENE(CardDF):
+    def __init__(self, fid='7fe0', name='DF.EIRENE', desc='GSM-R EIRENE'):
+        super().__init__(fid=fid, name=name, desc=desc)
+        files = [
+            # Section 7.1.6 / Table 10 EIRENE GSM EFs
+            EF_FN(),
+            EF_CallconfC(),
+            EF_CallconfI(),
+            EF_Shunting(),
+            EF_GsmrPLMN(),
+            EF_IC(),
+            EF_NW(),
+
+            # support of the numbering plan
+            EF_Switching(fid='6f8e', name='EF.CT', desc='Call Type'),
+            EF_Switching(fid='6f8f', name='EF.SC', desc='Short Code'),
+            EF_Predefined(fid='6f88', name='EF.FC', desc='Function Code'),
+            EF_Predefined(fid='6f89', name='EF.Service',
+                          desc='VGCS/VBS Service Code'),
+            EF_Predefined(fid='6f8a', name='EF.Call',
+                          desc='First digit of the group ID'),
+            EF_Predefined(fid='6f8b', name='EF.FctTeam',
+                          desc='Call Type 6 Team Type + Team member function'),
+            EF_Predefined(fid='6f92', name='EF.Controller',
+                          desc='Call Type 7 Controller function code'),
+            EF_Predefined(fid='6f8c', name='EF.Gateway',
+                          desc='Access to external networks'),
+            EF_DialledVals(fid='6f81', name='EF.5to8digits',
+                           desc='Call Type 2 User Identity Number length'),
+            EF_DialledVals(fid='6f82', name='EF.2digits',
+                           desc='2 digits input'),
+            EF_DialledVals(fid='6f83', name='EF.8digits',
+                           desc='8 digits input'),
+            EF_DialledVals(fid='6f84', name='EF.9digits',
+                           desc='9 digits input'),
+            EF_DialledVals(fid='6f85', name='EF.SSSSS',
+                           desc='Group call area input'),
+            EF_DialledVals(fid='6f86', name='EF.LLLLL',
+                           desc='Location number Call Type 6'),
+            EF_DialledVals(fid='6f91', name='EF.Location',
+                           desc='Location number Call Type 7'),
+            EF_DialledVals(fid='6f87', name='EF.FreeNumber',
+                           desc='Free Number Call Type 0 and 8'),
+        ]
+        self.add_files(files)

pySim/iso7816_4.py

@@ -0,0 +1,80 @@
+# coding=utf-8
+"""Utilities / Functions related to ISO 7816-4
+
+(C) 2022 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from construct import *
+from pySim.construct import *
+from pySim.utils import *
+from pySim.filesystem import *
+from pySim.tlv import *
+
+# Table 91 + Section 8.2.1.2
+
+
+class ApplicationId(BER_TLV_IE, tag=0x4f):
+    _construct = GreedyBytes
+
+# Table 91
+
+
+class ApplicationLabel(BER_TLV_IE, tag=0x50):
+    _construct = GreedyBytes
+
+# Table 91 + Section 5.3.1.2
+
+
+class FileReference(BER_TLV_IE, tag=0x51):
+    _construct = GreedyBytes
+
+# Table 91
+
+
+class CommandApdu(BER_TLV_IE, tag=0x52):
+    _construct = GreedyBytes
+
+# Table 91
+
+
+class DiscretionaryData(BER_TLV_IE, tag=0x53):
+    _construct = GreedyBytes
+
+# Table 91
+
+
+class DiscretionaryTemplate(BER_TLV_IE, tag=0x73):
+    _construct = GreedyBytes
+
+# Table 91 + RFC1738 / RFC2396
+
+
+class URL(BER_TLV_IE, tag=0x5f50):
+    _construct = GreedyString('ascii')
+
+# Table 91
+
+
+class ApplicationRelatedDOSet(BER_TLV_IE, tag=0x61):
+    _construct = GreedyBytes
+
+# Section 8.2.1.3 Application Template
+
+
+class ApplicationTemplate(BER_TLV_IE, tag=0x61, nested=[ApplicationId, ApplicationLabel, FileReference,
+                          CommandApdu, DiscretionaryData, DiscretionaryTemplate, URL,
+                          ApplicationRelatedDOSet]):
+    pass

pySim/jsonpath.py

@@ -0,0 +1,49 @@
+# coding=utf-8
+import json
+import pprint
+import jsonpath_ng
+
+"""JSONpath utility functions as needed within pysim.
+
+As pySim-sell has the ability to represent SIM files as JSON strings,
+adding JSONpath allows us to conveniently modify individual sub-fields
+of a file or record in its JSON representation.
+"""
+
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+def js_path_find(js_dict, js_path):
+    """Find/Match a JSON path within a given JSON-serializable dict.
+    Args:
+        js_dict : JSON-serializable dict to operate on
+        js_path : JSONpath string
+    Returns: Result of the JSONpath expression
+    """
+    jsonpath_expr = jsonpath_ng.parse(js_path)
+    return jsonpath_expr.find(js_dict)
+
+
+def js_path_modify(js_dict, js_path, new_val):
+    """Find/Match a JSON path within a given JSON-serializable dict.
+    Args:
+        js_dict : JSON-serializable dict to operate on
+        js_path : JSONpath string
+        new_val : New value for field in js_dict at js_path
+    """
+    jsonpath_expr = jsonpath_ng.parse(js_path)
+    jsonpath_expr.find(js_dict)
+    jsonpath_expr.update(js_dict, new_val)

pySim/profile.py

@@ -0,0 +1,152 @@
+# -*- coding: utf-8 -*-
+
+""" pySim: tell old 2G SIMs apart from UICC
+"""
+
+#
+# (C) 2021 by Sysmocom s.f.m.c. GmbH
+# All Rights Reserved
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from pySim.commands import SimCardCommands
+from pySim.filesystem import CardApplication, interpret_sw
+from pySim.utils import all_subclasses
+import abc
+import operator
+
+
+def _mf_select_test(scc: SimCardCommands, cla_byte: str, sel_ctrl: str) -> bool:
+    cla_byte_bak = scc.cla_byte
+    sel_ctrl_bak = scc.sel_ctrl
+    scc.reset_card()
+
+    scc.cla_byte = cla_byte
+    scc.sel_ctrl = sel_ctrl
+    rc = True
+    try:
+        scc.select_file('3f00')
+    except:
+        rc = False
+
+    scc.reset_card()
+    scc.cla_byte = cla_byte_bak
+    scc.sel_ctrl = sel_ctrl_bak
+    return rc
+
+
+def match_uicc(scc: SimCardCommands) -> bool:
+    """ Try to access MF via UICC APDUs (3GPP TS 102.221), if this works, the
+    card is considered a UICC card.
+    """
+    return _mf_select_test(scc, "00", "0004")
+
+
+def match_sim(scc: SimCardCommands) -> bool:
+    """ Try to access MF via 2G APDUs (3GPP TS 11.11), if this works, the card
+    is also a simcard. This will be the case for most UICC cards, but there may
+    also be plain UICC cards without 2G support as well.
+    """
+    return _mf_select_test(scc, "a0", "0000")
+
+
+class CardProfile(object):
+    """A Card Profile describes a card, it's filesystem hierarchy, an [initial] list of
+    applications as well as profile-specific SW and shell commands.  Every card has
+    one card profile, but there may be multiple applications within that profile."""
+
+    def __init__(self, name, **kw):
+        """
+        Args:
+                desc (str) : Description
+                files_in_mf : List of CardEF instances present in MF
+                applications : List of CardApplications present on card
+                sw : List of status word definitions
+                shell_cmdsets : List of cmd2 shell command sets of profile-specific commands
+                cla : class byte that should be used with cards of this profile
+                sel_ctrl : selection control bytes class byte that should be used with cards of this profile
+        """
+        self.name = name
+        self.desc = kw.get("desc", None)
+        self.files_in_mf = kw.get("files_in_mf", [])
+        self.sw = kw.get("sw", {})
+        self.applications = kw.get("applications", [])
+        self.shell_cmdsets = kw.get("shell_cmdsets", [])
+        self.cla = kw.get("cla", "00")
+        self.sel_ctrl = kw.get("sel_ctrl", "0004")
+
+    def __str__(self):
+        return self.name
+
+    def add_application(self, app: CardApplication):
+        """Add an application to a card profile.
+
+        Args:
+                app : CardApplication instance to be added to profile
+        """
+        self.applications.append(app)
+
+    def interpret_sw(self, sw: str):
+        """Interpret a given status word within the profile.
+
+        Args:
+                sw : Status word as string of 4 hex digits
+
+        Returns:
+                Tuple of two strings
+        """
+        return interpret_sw(self.sw, sw)
+
+    @staticmethod
+    def decode_select_response(data_hex: str) -> object:
+        """Decode the response to a SELECT command.
+
+        This is the fall-back method which doesn't perform any decoding. It mostly
+        exists so specific derived classes can overload it for actual decoding.
+        This method is implemented in the profile and is only used when application
+        specific decoding cannot be performed (no ADF is selected).
+
+        Args:
+                data_hex: Hex string of the select response
+        """
+        return data_hex
+
+    @staticmethod
+    @abc.abstractmethod
+    def match_with_card(scc: SimCardCommands) -> bool:
+        """Check if the specific profile matches the card. This method is a
+        placeholder that is overloaded by specific dirived classes. The method
+        actively probes the card to make sure the profile class matches the
+        physical card. This usually also means that the card is reset during
+        the process, so this method must not be called at random times. It may
+        only be called on startup.
+
+        Args:
+                scc: SimCardCommands class
+        Returns:
+                match = True, no match = False
+        """
+        return False
+
+    @staticmethod
+    def pick(scc: SimCardCommands):
+        profiles = list(all_subclasses(CardProfile))
+        profiles.sort(key=operator.attrgetter('ORDER'))
+
+        for p in profiles:
+            if p.match_with_card(scc):
+                return p()
+
+        return None

pySim/sysmocom_sja2.py

@@ -0,0 +1,278 @@
+# coding=utf-8
+"""Utilities / Functions related to sysmocom SJA2 cards
+
+(C) 2021 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from pytlv.TLV import *
+from struct import pack, unpack
+from pySim.utils import *
+from pySim.filesystem import *
+from pySim.ts_102_221 import CardProfileUICC
+from pySim.construct import *
+from construct import *
+import pySim
+
+key_type2str = {
+    0: 'kic',
+    1: 'kid',
+    2: 'kik',
+    3: 'any',
+}
+
+key_algo2str = {
+    0: 'des',
+    1: 'aes'
+}
+
+mac_length = {
+    0: 8,
+    1: 4
+}
+
+
+class EF_PIN(TransparentEF):
+    def __init__(self, fid, name):
+        super().__init__(fid, name=name, desc='%s PIN file' % name)
+
+    def _decode_bin(self, raw_bin_data):
+        u = unpack('!BBB8s', raw_bin_data[:11])
+        res = {'enabled': (True, False)[u[0] & 0x01],
+               'initialized': (True, False)[u[0] & 0x02],
+               'disable_able': (False, True)[u[0] & 0x10],
+               'unblock_able': (False, True)[u[0] & 0x20],
+               'change_able': (False, True)[u[0] & 0x40],
+               'valid': (False, True)[u[0] & 0x80],
+               'attempts_remaining': u[1],
+               'maximum_attempts': u[2],
+               'pin': u[3].hex(),
+               }
+        if len(raw_bin_data) == 21:
+            u2 = unpack('!BB8s', raw_bin_data[11:10])
+            res['attempts_remaining_puk'] = u2[0]
+            res['maximum_attempts_puk'] = u2[1]
+            res['puk'] = u2[2].hex()
+        return res
+
+
+class EF_MILENAGE_CFG(TransparentEF):
+    def __init__(self, fid='6f21', name='EF.MILENAGE_CFG', desc='Milenage connfiguration'):
+        super().__init__(fid, name=name, desc=desc)
+
+    def _decode_bin(self, raw_bin_data):
+        u = unpack('!BBBBB16s16s16s16s16s', raw_bin_data)
+        return {'r1': u[0], 'r2': u[1], 'r3': u[2], 'r4': u[3], 'r5': u[4],
+                'c1': u[5].hex(),
+                'c2': u[6].hex(),
+                'c3': u[7].hex(),
+                'c4': u[8].hex(),
+                'c5': u[9].hex(),
+                }
+
+
+class EF_0348_KEY(LinFixedEF):
+    def __init__(self, fid='6f22', name='EF.0348_KEY', desc='TS 03.48 OTA Keys'):
+        super().__init__(fid, name=name, desc=desc, rec_len={27, 35})
+
+    def _decode_record_bin(self, raw_bin_data):
+        u = unpack('!BBB', raw_bin_data[0:3])
+        key_algo = (u[2] >> 6) & 1
+        key_length = ((u[2] >> 3) & 3) * 8
+        return {'sec_domain': u[0],
+                'key_set_version': u[1],
+                'key_type': key_type2str[u[2] & 3],
+                'key_length': key_length,
+                'algorithm': key_algo2str[key_algo],
+                'mac_length': mac_length[(u[2] >> 7)],
+                'key': raw_bin_data[3:key_length].hex()
+                }
+
+
+class EF_0348_COUNT(LinFixedEF):
+    def __init__(self, fid='6f23', name='EF.0348_COUNT', desc='TS 03.48 OTA Counters'):
+        super().__init__(fid, name=name, desc=desc, rec_len={7, 7})
+
+    def _decode_record_bin(self, raw_bin_data):
+        u = unpack('!BB5s', raw_bin_data)
+        return {'sec_domain': u[0], 'key_set_version': u[1], 'counter': u[2]}
+
+
+class EF_SIM_AUTH_COUNTER(TransparentEF):
+    def __init__(self, fid='af24', name='EF.SIM_AUTH_COUNTER'):
+        super().__init__(fid, name=name, desc='Number of remaining RUN GSM ALGORITHM executions')
+        self._construct = Struct('num_run_gsm_algo_remain'/Int32ub)
+
+
+class EF_GP_COUNT(LinFixedEF):
+    def __init__(self, fid='6f26', name='EF.GP_COUNT', desc='GP SCP02 Counters'):
+        super().__init__(fid, name=name, desc=desc, rec_len={5, 5})
+
+    def _decode_record_bin(self, raw_bin_data):
+        u = unpack('!BBHB', raw_bin_data)
+        return {'sec_domain': u[0], 'key_set_version': u[1], 'counter': u[2], 'rfu': u[3]}
+
+
+class EF_GP_DIV_DATA(LinFixedEF):
+    def __init__(self, fid='6f27', name='EF.GP_DIV_DATA', desc='GP SCP02 key diversification data'):
+        super().__init__(fid, name=name, desc=desc, rec_len={12, 12})
+
+    def _decode_record_bin(self, raw_bin_data):
+        u = unpack('!BB8s', raw_bin_data)
+        return {'sec_domain': u[0], 'key_set_version': u[1], 'key_div_data': u[2].hex()}
+
+
+class EF_SIM_AUTH_KEY(TransparentEF):
+    def __init__(self, fid='6f20', name='EF.SIM_AUTH_KEY'):
+        super().__init__(fid, name=name, desc='USIM authentication key')
+        CfgByte = BitStruct(Bit[2],
+                            'use_sres_deriv_func_2'/Bit,
+                            'use_opc_instead_of_op'/Bit,
+                            'algorithm'/Enum(Nibble, milenage=4, comp128v1=1, comp128v2=2, comp128v3=3))
+        self._construct = Struct('cfg'/CfgByte,
+                                 'key'/Bytes(16),
+                                 'op' /
+                                 If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op, Bytes(
+                                     16)),
+                                 'opc' /
+                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op, Bytes(
+                                     16))
+                                 )
+
+
+class DF_SYSTEM(CardDF):
+    def __init__(self):
+        super().__init__(fid='a515', name='DF.SYSTEM', desc='CardOS specifics')
+        files = [
+            EF_PIN('6f01', 'EF.CHV1'),
+            EF_PIN('6f81', 'EF.CHV2'),
+            EF_PIN('6f0a', 'EF.ADM1'),
+            EF_PIN('6f0b', 'EF.ADM2'),
+            EF_PIN('6f0c', 'EF.ADM3'),
+            EF_PIN('6f0d', 'EF.ADM4'),
+            EF_MILENAGE_CFG(),
+            EF_0348_KEY(),
+            EF_SIM_AUTH_COUNTER(),
+            EF_SIM_AUTH_KEY(),
+            EF_0348_COUNT(),
+            EF_GP_COUNT(),
+            EF_GP_DIV_DATA(),
+        ]
+        self.add_files(files)
+
+    def decode_select_response(self, resp_hex):
+        return pySim.ts_102_221.CardProfileUICC.decode_select_response(resp_hex)
+
+
+class EF_USIM_SQN(TransparentEF):
+    def __init__(self, fid='af30', name='EF.USIM_SQN'):
+        super().__init__(fid, name=name, desc='SQN parameters for AKA')
+        Flag1 = BitStruct('skip_next_sqn_check'/Bit, 'delta_max_check'/Bit,
+                          'age_limit_check'/Bit, 'sqn_check'/Bit,
+                          'ind_len'/BitsInteger(4))
+        Flag2 = BitStruct('rfu'/BitsRFU(5), 'dont_clear_amf_for_macs'/Bit,
+                          'aus_concealed'/Bit, 'autn_concealed'/Bit)
+        self._construct = Struct('flag1'/Flag1, 'flag2'/Flag2,
+                                 'delta_max' /
+                                 BytesInteger(6), 'age_limit'/BytesInteger(6),
+                                 'freshness'/GreedyRange(BytesInteger(6)))
+
+
+class EF_USIM_AUTH_KEY(TransparentEF):
+    def __init__(self, fid='af20', name='EF.USIM_AUTH_KEY'):
+        super().__init__(fid, name=name, desc='USIM authentication key')
+        CfgByte = BitStruct(Bit, 'only_4bytes_res_in_3g'/Bit,
+                            'use_sres_deriv_func_2_in_3g'/Bit,
+                            'use_opc_instead_of_op'/Bit,
+                            'algorithm'/Enum(Nibble, milenage=4, sha1_aka=5, xor=15))
+        self._construct = Struct('cfg'/CfgByte,
+                                 'key'/Bytes(16),
+                                 'op' /
+                                 If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op, Bytes(
+                                     16)),
+                                 'opc' /
+                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op, Bytes(
+                                     16))
+                                 )
+
+
+class EF_USIM_AUTH_KEY_2G(TransparentEF):
+    def __init__(self, fid='af22', name='EF.USIM_AUTH_KEY_2G'):
+        super().__init__(fid, name=name, desc='USIM authentication key in 2G context')
+        CfgByte = BitStruct(Bit, 'only_4bytes_res_in_3g'/Bit,
+                            'use_sres_deriv_func_2_in_3g'/Bit,
+                            'use_opc_instead_of_op'/Bit,
+                            'algorithm'/Enum(Nibble, milenage=4, comp128v1=1, comp128v2=2, comp128v3=3))
+        self._construct = Struct('cfg'/CfgByte,
+                                 'key'/Bytes(16),
+                                 'op' /
+                                 If(this.cfg.algorithm == 'milenage' and not this.cfg.use_opc_instead_of_op, Bytes(
+                                     16)),
+                                 'opc' /
+                                 If(this.cfg.algorithm == 'milenage' and this.cfg.use_opc_instead_of_op, Bytes(
+                                     16))
+                                 )
+
+
+class EF_GBA_SK(TransparentEF):
+    def __init__(self, fid='af31', name='EF.GBA_SK'):
+        super().__init__(fid, name=name, desc='Secret key for GBA key derivation')
+        self._construct = GreedyBytes
+
+
+class EF_GBA_REC_LIST(TransparentEF):
+    def __init__(self, fid='af32', name='EF.GBA_REC_LIST'):
+        super().__init__(fid, name=name, desc='Secret key for GBA key derivation')
+        # integers representing record numbers in EF-GBANL
+        self._construct = GreedyRange(Int8ub)
+
+
+class EF_GBA_INT_KEY(LinFixedEF):
+    def __init__(self, fid='af33', name='EF.GBA_INT_KEY'):
+        super().__init__(fid, name=name,
+                         desc='Secret key for GBA key derivation', rec_len={32, 32})
+        self._construct = GreedyBytes
+
+
+class SysmocomSJA2(CardModel):
+    _atrs = ["3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 4C 75 30 34 05 4B A9",
+             "3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 4C 75 31 33 02 51 B2",
+             "3B 9F 96 80 1F 87 80 31 E0 73 FE 21 1B 67 4A 4C 52 75 31 04 51 D5"]
+
+    @classmethod
+    def add_files(cls, rs: RuntimeState):
+        """Add sysmocom SJA2 specific files to given RuntimeState."""
+        rs.mf.add_file(DF_SYSTEM())
+        # optional USIM application
+        if 'a0000000871002' in rs.mf.applications:
+            usim_adf = rs.mf.applications['a0000000871002']
+            files_adf_usim = [
+                EF_USIM_AUTH_KEY(),
+                EF_USIM_AUTH_KEY_2G(),
+                EF_GBA_SK(),
+                EF_GBA_REC_LIST(),
+                EF_GBA_INT_KEY(),
+                EF_USIM_SQN(),
+            ]
+            usim_adf.add_files(files_adf_usim)
+        # optional ISIM application
+        if 'a0000000871004' in rs.mf.applications:
+            isim_adf = rs.mf.applications['a0000000871004']
+            files_adf_isim = [
+                EF_USIM_AUTH_KEY(name='EF.ISIM_AUTH_KEY'),
+                EF_USIM_AUTH_KEY_2G(name='EF.ISIM_AUTH_KEY_2G'),
+                EF_USIM_SQN(name='EF.ISIM_SQN'),
+            ]
+            isim_adf.add_files(files_adf_isim)

pySim/tlv.py

@@ -0,0 +1,437 @@
+"""object-oriented TLV parser/encoder library."""
+
+# (C) 2021 by Harald Welte <laforge@osmocom.org>
+# All Rights Reserved
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from typing import Optional, List, Dict, Any, Tuple
+from bidict import bidict
+from construct import *
+
+from pySim.utils import bertlv_encode_len, bertlv_parse_len, bertlv_encode_tag, bertlv_parse_tag
+from pySim.utils import comprehensiontlv_encode_tag, comprehensiontlv_parse_tag
+from pySim.utils import bertlv_parse_one, comprehensiontlv_parse_one
+from pySim.utils import bertlv_parse_tag_raw, comprehensiontlv_parse_tag_raw
+
+from pySim.construct import parse_construct, LV, HexAdapter, BcdAdapter, BitsRFU, GsmStringAdapter
+from pySim.exceptions import *
+
+import inspect
+import abc
+import re
+
+def camel_to_snake(name):
+    name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
+    return re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower()
+
+class TlvMeta(abc.ABCMeta):
+    """Metaclass which we use to set some class variables at the time of defining a subclass.
+    This allows us to create subclasses for each TLV/IE type, where the class represents fixed
+    parameters like the tag/type and instances of it represent the actual TLV data."""
+    def __new__(metacls, name, bases, namespace, **kwargs):
+        #print("TlvMeta_new_(metacls=%s, name=%s, bases=%s, namespace=%s, kwargs=%s)" % (metacls, name, bases, namespace, kwargs))
+        x = super().__new__(metacls, name, bases, namespace)
+        # this becomes a _class_ variable, not an instance variable
+        x.tag = namespace.get('tag', kwargs.get('tag', None))
+        x.desc = namespace.get('desc', kwargs.get('desc', None))
+        nested = namespace.get('nested', kwargs.get('nested', None))
+        if nested is None or inspect.isclass(nested) and issubclass(nested, TLV_IE_Collection):
+            # caller has specified TLV_IE_Collection sub-class, we can directly reference it
+            x.nested_collection_cls = nested
+        else:
+            # caller passed list of other TLV classes that might possibly appear within us,
+            # build a dynamically-created TLV_IE_Collection sub-class and reference it
+            name = 'auto_collection_%s' % (name)
+            cls = type(name, (TLV_IE_Collection,), {'nested': nested})
+            x.nested_collection_cls = cls
+        return x
+
+
+class TlvCollectionMeta(abc.ABCMeta):
+    """Metaclass which we use to set some class variables at the time of defining a subclass.
+    This allows us to create subclasses for each Collection type, where the class represents fixed
+    parameters like the nested IE classes and instances of it represent the actual TLV data."""
+    def __new__(metacls, name, bases, namespace, **kwargs):
+        #print("TlvCollectionMeta_new_(metacls=%s, name=%s, bases=%s, namespace=%s, kwargs=%s)" % (metacls, name, bases, namespace, kwargs))
+        x = super().__new__(metacls, name, bases, namespace)
+        # this becomes a _class_ variable, not an instance variable
+        x.possible_nested = namespace.get('nested', kwargs.get('nested', None))
+        return x
+
+
+class Transcodable(abc.ABC):
+    _construct = None
+    """Base class for something that can be encoded + encoded.  Decoding and Encoding happens either
+     * via a 'construct' object stored in a derived class' _construct variable, or
+     * via a 'construct' object stored in an instance _construct variable, or
+     * via a derived class' _{to,from}_bytes() methods."""
+
+    def __init__(self):
+        self.encoded = None
+        self.decoded = None
+        self._construct = None
+
+    def to_bytes(self) -> bytes:
+        """Convert from internal representation to binary bytes.  Store the binary result
+        in the internal state and return it."""
+        if not self.decoded:
+            do = b''
+        elif self._construct:
+            do = self._construct.build(self.decoded, total_len=None)
+        elif self.__class__._construct:
+            do = self.__class__._construct.build(self.decoded, total_len=None)
+        else:
+            do = self._to_bytes()
+        self.encoded = do
+        return do
+
+    # not an abstractmethod, as it is only required if no _construct exists
+    def _to_bytes(self):
+        raise NotImplementedError
+
+    def from_bytes(self, do: bytes):
+        """Convert from binary bytes to internal representation. Store the decoded result
+        in the internal state and return it."""
+        self.encoded = do
+        if self.encoded == b'':
+            self.decoded = None
+        elif self._construct:
+            self.decoded = parse_construct(self._construct, do)
+        elif self.__class__._construct:
+            self.decoded = parse_construct(self.__class__._construct, do)
+        else:
+            self.decoded = self._from_bytes(do)
+        return self.decoded
+
+    # not an abstractmethod, as it is only required if no _construct exists
+    def _from_bytes(self, do: bytes):
+        raise NotImplementedError
+
+
+class IE(Transcodable, metaclass=TlvMeta):
+    # we specify the metaclass so any downstream subclasses will automatically use it
+    """Base class for various Information Elements. We understand the notion of a hierarchy
+    of IEs on top of the Transcodable class."""
+    # this is overridden by the TlvMeta metaclass, if it is used to create subclasses
+    nested_collection_cls = None
+    tag = None
+
+    def __init__(self, **kwargs):
+        super().__init__()
+        self.nested_collection = None
+        if self.nested_collection_cls:
+            self.nested_collection = self.nested_collection_cls()
+        # if we are a constructed IE, [ordered] list of actual child-IE instances
+        self.children = kwargs.get('children', [])
+        self.decoded = kwargs.get('decoded', None)
+
+    def __repr__(self):
+        """Return a string representing the [nested] IE data (for print)."""
+        if len(self.children):
+            member_strs = [repr(x) for x in self.children]
+            return '%s(%s)' % (type(self).__name__, ','.join(member_strs))
+        else:
+            return '%s(%s)' % (type(self).__name__, self.decoded)
+
+    def to_dict(self):
+        """Return a JSON-serializable dict representing the [nested] IE data."""
+        if len(self.children):
+            v = [x.to_dict() for x in self.children]
+        else:
+            v = self.decoded
+        return {camel_to_snake(type(self).__name__): v}
+
+    def from_dict(self, decoded: dict):
+        """Set the IE internal decoded representation to data from the argument.
+        If this is a nested IE, the child IE instance list is re-created."""
+        if self.nested_collection:
+            self.children = self.nested_collection.from_dict(decoded)
+        else:
+            self.children = []
+            self.decoded = decoded
+
+    def is_constructed(self):
+        """Is this IE constructed by further nested IEs?"""
+        if len(self.children):
+            return True
+        else:
+            return False
+
+    @abc.abstractmethod
+    def to_ie(self) -> bytes:
+        """Convert the internal representation to entire IE including IE header."""
+
+    def to_bytes(self) -> bytes:
+        """Convert the internal representation _of the value part_ to binary bytes."""
+        if self.is_constructed():
+            # concatenate the encoded IE of all children to form the value part
+            out = b''
+            for c in self.children:
+                out += c.to_ie()
+            return out
+        else:
+            return super().to_bytes()
+
+    def from_bytes(self, do: bytes):
+        """Parse _the value part_ from binary bytes to internal representation."""
+        if self.nested_collection:
+            self.children = self.nested_collection.from_bytes(do)
+        else:
+            self.children = []
+            return super().from_bytes(do)
+
+
+class TLV_IE(IE):
+    """Abstract base class for various TLV type Information Elements."""
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+    def _compute_tag(self) -> int:
+        """Compute the tag (sometimes the tag encodes part of the value)."""
+        return self.tag
+
+    @classmethod
+    @abc.abstractmethod
+    def _parse_tag_raw(cls, do: bytes) -> Tuple[int, bytes]:
+        """Obtain the raw TAG at the start of the bytes provided by the user."""
+
+    @classmethod
+    @abc.abstractmethod
+    def _parse_len(cls, do: bytes) -> Tuple[int, bytes]:
+        """Obtain the length encoded at the start of the bytes provided by the user."""
+
+    @abc.abstractmethod
+    def _encode_tag(self) -> bytes:
+        """Encode the tag part. Must be provided by derived (TLV format specific) class."""
+
+    @abc.abstractmethod
+    def _encode_len(self, val: bytes) -> bytes:
+        """Encode the length part assuming a certain binary value. Must be provided by
+        derived (TLV format specific) class."""
+
+    def to_ie(self):
+        return self.to_tlv()
+
+    def to_tlv(self):
+        """Convert the internal representation to binary TLV bytes."""
+        val = self.to_bytes()
+        return self._encode_tag() + self._encode_len(val) + val
+
+    def from_tlv(self, do: bytes):
+        (rawtag, remainder) = self.__class__._parse_tag_raw(do)
+        if rawtag:
+            if rawtag != self.tag:
+                raise ValueError("%s: Encountered tag %s doesn't match our supported tag %s" %
+                                 (self, rawtag, self.tag))
+            (length, remainder) = self.__class__._parse_len(remainder)
+            value = remainder[:length]
+            remainder = remainder[length:]
+        else:
+            value = do
+            remainder = b''
+        dec = self.from_bytes(value)
+        return dec, remainder
+
+
+class BER_TLV_IE(TLV_IE):
+    """TLV_IE formatted as ASN.1 BER described in ITU-T X.690 8.1.2."""
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+    @classmethod
+    def _decode_tag(cls, do: bytes) -> Tuple[dict, bytes]:
+        return bertlv_parse_tag(do)
+
+    @classmethod
+    def _parse_tag_raw(cls, do: bytes) -> Tuple[int, bytes]:
+        return bertlv_parse_tag_raw(do)
+
+    @classmethod
+    def _parse_len(cls, do: bytes) -> Tuple[int, bytes]:
+        return bertlv_parse_len(do)
+
+    def _encode_tag(self) -> bytes:
+        return bertlv_encode_tag(self._compute_tag())
+
+    def _encode_len(self, val: bytes) -> bytes:
+        return bertlv_encode_len(len(val))
+
+
+class COMPR_TLV_IE(TLV_IE):
+    """TLV_IE formated as COMPREHENSION-TLV as described in ETSI TS 101 220."""
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.comprehension = False
+
+    @classmethod
+    def _decode_tag(cls, do: bytes) -> Tuple[dict, bytes]:
+        return comprehensiontlv_parse_tag(do)
+
+    @classmethod
+    def _parse_tag_raw(cls, do: bytes) -> Tuple[int, bytes]:
+        return comprehensiontlv_parse_tag_raw(do)
+
+    @classmethod
+    def _parse_len(cls, do: bytes) -> Tuple[int, bytes]:
+        return bertlv_parse_len(do)
+
+    def _encode_tag(self) -> bytes:
+        return comprehensiontlv_encode_tag(self._compute_tag())
+
+    def _encode_len(self, val: bytes) -> bytes:
+        return bertlv_encode_len(len(val))
+
+
+class TLV_IE_Collection(metaclass=TlvCollectionMeta):
+    # we specify the metaclass so any downstream subclasses will automatically use it
+    """A TLV_IE_Collection consists of multiple TLV_IE classes identified by their tags.
+    A given encoded DO may contain any of them in any order, and may contain multiple instances
+    of each DO."""
+    # this is overridden by the TlvCollectionMeta metaclass, if it is used to create subclasses
+    possible_nested = []
+
+    def __init__(self, desc=None, **kwargs):
+        self.desc = desc
+        #print("possible_nested: ", self.possible_nested)
+        self.members = kwargs.get('nested', self.possible_nested)
+        self.members_by_tag = {}
+        self.members_by_name = {}
+        self.members_by_tag = {m.tag: m for m in self.members}
+        self.members_by_name = {m.__name__: m for m in self.members}
+        # if we are a constructed IE, [ordered] list of actual child-IE instances
+        self.children = kwargs.get('children', [])
+        self.encoded = None
+
+    def __str__(self):
+        member_strs = [str(x) for x in self.members]
+        return '%s(%s)' % (type(self).__name__, ','.join(member_strs))
+
+    def __repr__(self):
+        member_strs = [repr(x) for x in self.members]
+        return '%s(%s)' % (self.__class__, ','.join(member_strs))
+
+    def __add__(self, other):
+        """Extending TLV_IE_Collections with other TLV_IE_Collections or TLV_IEs."""
+        if isinstance(other, TLV_IE_Collection):
+            # adding one collection to another
+            members = self.members + other.members
+            return TLV_IE_Collection(self.desc, nested=members)
+        elif inspect.isclass(other) and issubclass(other, TLV_IE):
+            # adding a member to a collection
+            return TLV_IE_Collection(self.desc, nested=self.members + [other])
+        else:
+            raise TypeError
+
+    def from_bytes(self, binary: bytes) -> List[TLV_IE]:
+        """Create a list of TLV_IEs from the collection based on binary input data.
+        Args:
+            binary : binary bytes of encoded data
+        Returns:
+            list of instances of TLV_IE sub-classes containing parsed data
+        """
+        self.encoded = binary
+        # list of instances of TLV_IE collection member classes appearing in the data
+        res = []
+        remainder = binary
+        first = next(iter(self.members_by_tag.values()))
+        # iterate until no binary trailer is left
+        while len(remainder):
+            # obtain the tag at the start of the remainder
+            tag, r = first._parse_tag_raw(remainder)
+            if tag == None:
+                return res
+            if tag in self.members_by_tag:
+                cls = self.members_by_tag[tag]
+                # create an instance and parse accordingly
+                inst = cls()
+                dec, remainder = inst.from_tlv(remainder)
+                res.append(inst)
+            else:
+                # unknown tag; create the related class on-the-fly using the same base class
+                name = 'unknown_%s_%X' % (first.__base__.__name__, tag)
+                cls = type(name, (first.__base__,), {'tag': tag, 'possible_nested': [],
+                                                     'nested_collection_cls': None})
+                cls._from_bytes = lambda s, a: {'raw': a.hex()}
+                cls._to_bytes = lambda s: bytes.fromhex(s.decoded['raw'])
+                # create an instance and parse accordingly
+                inst = cls()
+                dec, remainder = inst.from_tlv(remainder)
+                res.append(inst)
+        self.children = res
+        return res
+
+    def from_dict(self, decoded: List[dict]) -> List[TLV_IE]:
+        """Create a list of TLV_IE instances from the collection based on an array
+        of dicts, where they key indicates the name of the TLV_IE subclass to use."""
+        # list of instances of TLV_IE collection member classes appearing in the data
+        res = []
+        for i in decoded:
+            for k in i.keys():
+                if k in self.members_by_name:
+                    cls = self.members_by_name[k]
+                    inst = cls()
+                    inst.from_dict(i[k])
+                    res.append(inst)
+                else:
+                    raise ValueError('%s: Unknown TLV Class %s in %s; expected %s' %
+                                     (self, i[0], decoded, self.members_by_name.keys()))
+        self.children = res
+        return res
+
+    def to_dict(self):
+        return [x.to_dict() for x in self.children]
+
+    def to_bytes(self):
+        out = b''
+        for c in self.children:
+            out += c.to_tlv()
+        return out
+
+    def from_tlv(self, do):
+        return self.from_bytes(do)
+
+    def to_tlv(self):
+        return self.to_bytes()
+
+
+def flatten_dict_lists(inp):
+    """hierarchically flatten each list-of-dicts into a single dict. This is useful to
+       make the output of hierarchical TLV decoder structures flatter and more easy to read."""
+    def are_all_elements_dict(l):
+        for e in l:
+            if not isinstance(e, dict):
+                return False
+        return True
+
+    if isinstance(inp, list):
+        if are_all_elements_dict(inp):
+            # flatten into one shared dict
+            newdict = {}
+            for e in inp:
+                key = list(e.keys())[0]
+                newdict[key] = e[key]
+            inp = newdict
+            # process result as any native dict
+            return {k:flatten_dict_lists(inp[k]) for k in inp.keys()}
+        else:
+            return [flatten_dict_lists(x) for x in inp]
+    elif isinstance(inp, dict):
+        return {k:flatten_dict_lists(inp[k]) for k in inp.keys()}
+    else:
+        return inp

pySim/transport/__init__.py

@@ -1,11 +1,19 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 """ pySim: PCSC reader transport link base
 """
 
+import abc
+import argparse
+from typing import Optional, Tuple
+
+from pySim.exceptions import *
+from pySim.construct import filter_dict
+from pySim.utils import sw_match, b2h, h2b, i2h
+
 #
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,85 +29,232 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-class LinkBase(object):
 
-	def wait_for_card(self, timeout=None, newcardonly=False):
-		"""wait_for_card(): Wait for a card and connect to it
+class ApduTracer:
+    def trace_command(self, cmd):
+        pass
 
-		   timeout     : Maximum wait time (None=no timeout)
-		   newcardonly : Should we wait for a new card, or an already
-				 inserted one ?
-		"""
-		pass
+    def trace_response(self, cmd, sw, resp):
+        pass
 
-	def connect(self):
-		"""connect(): Connect to a card immediately
-		"""
-		pass
 
-	def disconnect(self):
-		"""disconnect(): Disconnect from card
-		"""
-		pass
+class LinkBase(abc.ABC):
+    """Base class for link/transport to card."""
 
-	def reset_card(self):
-		"""reset_card(): Resets the card (power down/up)
-		"""
-		pass
+    def __init__(self, sw_interpreter=None, apdu_tracer=None):
+        self.sw_interpreter = sw_interpreter
+        self.apdu_tracer = apdu_tracer
 
-	def send_apdu_raw(self, pdu):
-		"""send_apdu_raw(pdu): Sends an APDU with minimal processing
+    @abc.abstractmethod
+    def _send_apdu_raw(self, pdu: str) -> Tuple[str, str]:
+        """Implementation specific method for sending the PDU."""
 
-		   pdu    : string of hexadecimal characters (ex. "A0A40000023F00")
-		   return : tuple(data, sw), where
-			    data : string (in hex) of returned data (ex. "074F4EFFFF")
-			    sw   : string (in hex) of status word (ex. "9000")
-		"""
-		pass
+    def set_sw_interpreter(self, interp):
+        """Set an (optional) status word interpreter."""
+        self.sw_interpreter = interp
 
-	def send_apdu(self, pdu):
-		"""send_apdu(pdu): Sends an APDU and auto fetch response data
+    @abc.abstractmethod
+    def wait_for_card(self, timeout: int = None, newcardonly: bool = False):
+        """Wait for a card and connect to it
 
-		   pdu    : string of hexadecimal characters (ex. "A0A40000023F00")
-		   return : tuple(data, sw), where
-			    data : string (in hex) of returned data (ex. "074F4EFFFF")
-			    sw   : string (in hex) of status word (ex. "9000")
-		"""
-		data, sw = self.send_apdu_raw(pdu)
+        Args:
+           timeout : Maximum wait time in seconds (None=no timeout)
+           newcardonly : Should we wait for a new card, or an already inserted one ?
+        """
 
-		# When whe have sent the first APDU, the SW may indicate that there are response bytes
-		# available. There are two SWs commonly used for this 9fxx (sim) and 61xx (usim), where
-		# xx is the number of response bytes available.
-		# See also:
-		# SW1=9F: 3GPP TS 51.011 9.4.1, Responses to commands which are correctly executed
-		# SW1=61: ISO/IEC 7816-4, Table 5 — General meaning of the interindustry values of SW1-SW2
-		if (sw is not None) and ((sw[0:2] == '9f') or (sw[0:2] == '61')):
-			pdu_gr = pdu[0:2] + 'c00000' + sw[2:4]
-			data, sw = self.send_apdu_raw(pdu_gr)
+    @abc.abstractmethod
+    def connect(self):
+        """Connect to a card immediately
+        """
 
-		return data, sw
+    @abc.abstractmethod
+    def disconnect(self):
+        """Disconnect from card
+        """
 
-	def send_apdu_checksw(self, pdu, sw="9000"):
-		"""send_apdu_checksw(pdu,sw): Sends an APDU and check returned SW
+    @abc.abstractmethod
+    def reset_card(self):
+        """Resets the card (power down/up)
+        """
 
-		   pdu    : string of hexadecimal characters (ex. "A0A40000023F00")
-		   sw     : string of 4 hexadecimal characters (ex. "9000"). The
-			    user may mask out certain digits using a '?' to add some
-			    ambiguity if needed.
-		   return : tuple(data, sw), where
-			    data : string (in hex) of returned data (ex. "074F4EFFFF")
-			    sw   : string (in hex) of status word (ex. "9000")
-		"""
-		rv = self.send_apdu(pdu)
+    def send_apdu_raw(self, pdu: str):
+        """Sends an APDU with minimal processing
 
-		# Create a masked version of the returned status word
-		sw_masked = ""
-		for i in range(0, 4):
-			if sw.lower()[i] == '?':
-				sw_masked = sw_masked + '?'
-			else:
-				sw_masked = sw_masked + rv[1][i].lower()
+        Args:
+           pdu : string of hexadecimal characters (ex. "A0A40000023F00")
+        Returns:
+           tuple(data, sw), where
+                        data : string (in hex) of returned data (ex. "074F4EFFFF")
+                        sw   : string (in hex) of status word (ex. "9000")
+        """
+        if self.apdu_tracer:
+            self.apdu_tracer.trace_command(pdu)
+        (data, sw) = self._send_apdu_raw(pdu)
+        if self.apdu_tracer:
+            self.apdu_tracer.trace_response(pdu, sw, data)
+        return (data, sw)
 
-		if sw.lower() != sw_masked:
-			raise RuntimeError("SW match failed! Expected %s and got %s." % (sw.lower(), rv[1]))
-		return rv
+    def send_apdu(self, pdu):
+        """Sends an APDU and auto fetch response data
+
+        Args:
+           pdu : string of hexadecimal characters (ex. "A0A40000023F00")
+        Returns:
+           tuple(data, sw), where
+                        data : string (in hex) of returned data (ex. "074F4EFFFF")
+                        sw   : string (in hex) of status word (ex. "9000")
+        """
+        data, sw = self.send_apdu_raw(pdu)
+
+        # When we have sent the first APDU, the SW may indicate that there are response bytes
+        # available. There are two SWs commonly used for this 9fxx (sim) and 61xx (usim), where
+        # xx is the number of response bytes available.
+        # See also:
+        if (sw is not None):
+            if ((sw[0:2] == '9f') or (sw[0:2] == '61')):
+                # SW1=9F: 3GPP TS 51.011 9.4.1, Responses to commands which are correctly executed
+                # SW1=61: ISO/IEC 7816-4, Table 5 — General meaning of the interindustry values of SW1-SW2
+                pdu_gr = pdu[0:2] + 'c00000' + sw[2:4]
+                data, sw = self.send_apdu_raw(pdu_gr)
+            if sw[0:2] == '6c':
+                # SW1=6C: ETSI TS 102 221 Table 7.1: Procedure byte coding
+                pdu_gr = pdu[0:8] + sw[2:4]
+                data, sw = self.send_apdu_raw(pdu_gr)
+
+        return data, sw
+
+    def send_apdu_checksw(self, pdu, sw="9000"):
+        """Sends an APDU and check returned SW
+
+        Args:
+           pdu : string of hexadecimal characters (ex. "A0A40000023F00")
+           sw : string of 4 hexadecimal characters (ex. "9000"). The user may mask out certain
+                        digits using a '?' to add some ambiguity if needed.
+        Returns:
+                tuple(data, sw), where
+                        data : string (in hex) of returned data (ex. "074F4EFFFF")
+                        sw   : string (in hex) of status word (ex. "9000")
+        """
+        rv = self.send_apdu(pdu)
+
+        if sw == '9000' and sw_match(rv[1], '91xx'):
+            # proactive sim as per TS 102 221 Setion 7.4.2
+            rv = self.send_apdu_checksw('80120000' + rv[1][2:], sw)
+            print("FETCH: %s", rv[0])
+        if not sw_match(rv[1], sw):
+            raise SwMatchError(rv[1], sw.lower(), self.sw_interpreter)
+        return rv
+
+    def send_apdu_constr(self, cla, ins, p1, p2, cmd_constr, cmd_data, resp_constr):
+        """Build and sends an APDU using a 'construct' definition; parses response.
+
+        Args:
+                cla : string (in hex) ISO 7816 class byte
+                ins : string (in hex) ISO 7816 instruction byte
+                p1 : string (in hex) ISO 7116 Parameter 1 byte
+                p2 : string (in hex) ISO 7116 Parameter 2 byte
+                cmd_cosntr : defining how to generate binary APDU command data
+                cmd_data : command data passed to cmd_constr
+                resp_cosntr : defining how to decode  binary APDU response data
+        Returns:
+                Tuple of (decoded_data, sw)
+        """
+        cmd = cmd_constr.build(cmd_data) if cmd_data else ''
+        p3 = i2h([len(cmd)])
+        pdu = ''.join([cla, ins, p1, p2, p3, b2h(cmd)])
+        (data, sw) = self.send_apdu(pdu)
+        if data:
+            # filter the resulting dict to avoid '_io' members inside
+            rsp = filter_dict(resp_constr.parse(h2b(data)))
+        else:
+            rsp = None
+        return (rsp, sw)
+
+    def send_apdu_constr_checksw(self, cla, ins, p1, p2, cmd_constr, cmd_data, resp_constr,
+                                 sw_exp="9000"):
+        """Build and sends an APDU using a 'construct' definition; parses response.
+
+        Args:
+                cla : string (in hex) ISO 7816 class byte
+                ins : string (in hex) ISO 7816 instruction byte
+                p1 : string (in hex) ISO 7116 Parameter 1 byte
+                p2 : string (in hex) ISO 7116 Parameter 2 byte
+                cmd_cosntr : defining how to generate binary APDU command data
+                cmd_data : command data passed to cmd_constr
+                resp_cosntr : defining how to decode  binary APDU response data
+                exp_sw : string (in hex) of status word (ex. "9000")
+        Returns:
+                Tuple of (decoded_data, sw)
+        """
+        (rsp, sw) = self.send_apdu_constr(cla, ins,
+                                          p1, p2, cmd_constr, cmd_data, resp_constr)
+        if not sw_match(sw, sw_exp):
+            raise SwMatchError(sw, sw_exp.lower(), self.sw_interpreter)
+        return (rsp, sw)
+
+
+def argparse_add_reader_args(arg_parser):
+    """Add all reader related arguments to the given argparse.Argumentparser instance."""
+    serial_group = arg_parser.add_argument_group('Serial Reader')
+    serial_group.add_argument('-d', '--device', metavar='DEV', default='/dev/ttyUSB0',
+                              help='Serial Device for SIM access')
+    serial_group.add_argument('-b', '--baud', dest='baudrate', type=int, metavar='BAUD', default=9600,
+                              help='Baud rate used for SIM access')
+
+    pcsc_group = arg_parser.add_argument_group('PC/SC Reader')
+    pcsc_group.add_argument('-p', '--pcsc-device', type=int, dest='pcsc_dev', metavar='PCSC', default=None,
+                            help='PC/SC reader number to use for SIM access')
+
+    modem_group = arg_parser.add_argument_group('AT Command Modem Reader')
+    modem_group.add_argument('--modem-device', dest='modem_dev', metavar='DEV', default=None,
+                             help='Serial port of modem for Generic SIM Access (3GPP TS 27.007)')
+    modem_group.add_argument('--modem-baud', type=int, metavar='BAUD', default=115200,
+                             help='Baud rate used for modem port')
+
+    osmobb_group = arg_parser.add_argument_group('OsmocomBB Reader')
+    osmobb_group.add_argument('--osmocon', dest='osmocon_sock', metavar='PATH', default=None,
+                              help='Socket path for Calypso (e.g. Motorola C1XX) based reader (via OsmocomBB)')
+
+    btsap_group = arg_parser.add_argument_group('Bluetooth Device (SIM Access Profile)')
+    btsap_group.add_argument('--bt-addr', dest='bt_addr', metavar='ADDR', default=None,
+                             help='Bluetooth device address')
+
+    return arg_parser
+
+
+def init_reader(opts, **kwargs) -> Optional[LinkBase]:
+    """
+    Init card reader driver
+    """
+    sl = None  # type : :Optional[LinkBase]
+    try:
+        if opts.pcsc_dev is not None:
+            print("Using PC/SC reader interface")
+            from pySim.transport.pcsc import PcscSimLink
+            sl = PcscSimLink(opts.pcsc_dev, **kwargs)
+        elif opts.osmocon_sock is not None:
+            print("Using Calypso-based (OsmocomBB) reader interface")
+            from pySim.transport.calypso import CalypsoSimLink
+            sl = CalypsoSimLink(sock_path=opts.osmocon_sock, **kwargs)
+        elif opts.modem_dev is not None:
+            print("Using modem for Generic SIM Access (3GPP TS 27.007)")
+            from pySim.transport.modem_atcmd import ModemATCommandLink
+            sl = ModemATCommandLink(
+                device=opts.modem_dev, baudrate=opts.modem_baud, **kwargs)
+        elif opts.bt_addr is not None:
+            print("Using Bluetooth device (SIM Access Profile)")
+            from pySim.transport.bt_rsap import BluetoothSapSimLink
+            sl = BluetoothSapSimLink(opts.bt_addr, **kwargs)
+        else:  # Serial reader is default
+            print("Using serial reader interface")
+            from pySim.transport.serial import SerialSimLink
+            sl = SerialSimLink(device=opts.device,
+                               baudrate=opts.baudrate, **kwargs)
+        return sl
+    except Exception as e:
+        if str(e):
+            print("Card reader initialization failed with exception:\n" + str(e))
+        else:
+            print(
+                "Card reader initialization failed with an exception of type:\n" + str(type(e)))
+        return None

pySim/transport/bt_rsap.py

@@ -0,0 +1,554 @@
+# -*- coding: utf-8 -*-
+
+""" pySim: Bluetooth rSAP transport link
+"""
+
+#
+# Copyright (C) 2021  Gabriel K. Gegenhuber <ggegenhuber@sba-research.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import time
+import struct
+import logging
+import bluetooth
+
+from pySim.exceptions import ReaderError, NoCardError, ProtocolError
+from pySim.transport import LinkBase
+from pySim.utils import b2h, h2b, rpad
+
+logger = logging.getLogger(__name__)
+
+
+# thx to osmocom/softsim
+# SAP table 5.16
+SAP_CONNECTION_STATUS = {
+    0x00: "OK, Server can fulfill requirements",
+    0x01: "Error, Server unable to establish connection",
+    0x02: "Error, Server does not support maximum message size",
+    0x03: "Error, maximum message size by Client is too small",
+    0x04: "OK, ongoing call"
+}
+
+# SAP table 5.18
+SAP_RESULT_CODE = {
+    0x00: "OK, request processed correctly",
+    0x01: "Error, no reason defined",
+    0x02: "Error, card not accessible",
+    0x03: "Error, card (already) powered off",
+    0x04: "Error, card removed",
+    0x05: "Error, card already powered on",
+    0x06: "Error, data not available",
+    0x07: "Error, not supported"
+}
+
+# SAP table 5.19
+SAP_STATUS_CHANGE = {
+    0x00: "Unknown Error",
+    0x01: "Card reset",
+    0x02: "Card not accessible",
+    0x03: "Card removed",
+    0x04: "Card inserted",
+    0x05: "Card recovered"
+}
+
+# SAP table 5.15
+SAP_PARAMETERS = [
+    {
+        'name': "MaxMsgSize",
+        'length': 2,
+        'id': 0x00
+    },
+    {
+        'name': "ConnectionStatus",
+        'length': 1,
+        'id': 0x01
+    },
+    {
+        'name': "ResultCode",
+        'length': 1,
+        'id': 0x02
+    },
+    {
+        'name': "DisconnectionType",
+        'length': 1,
+        'id': 0x03
+    },
+    {
+        'name': "CommandAPDU",
+        'length': None,
+        'id': 0x04
+    },
+    {
+        'name': "ResponseAPDU",
+        'length': None,
+        'id': 0x05
+    },
+    {
+        'name': "ATR",
+        'length': None,
+        'id': 0x06
+    },
+    {
+        'name': "CardReaderdStatus",
+        'length': 1,
+        'id': 0x07
+    },
+    {
+        'name': "StatusChange",
+        'length': 1,
+        'id': 0x08
+    },
+    {
+        'name': "TransportProtocol",
+        'length': 1,
+        'id': 0x09
+    },
+    {
+        'name': "CommandAPDU7816",
+        'length': 2,
+        'id': 0x10
+    }
+]
+
+# SAP table 5.1
+SAP_MESSAGES = [
+    {
+        'name': 'CONNECT_REQ',
+        'client_to_server': True,
+        'id': 0x00,
+        'parameters': [(0x00, True)]
+    },
+    {
+        'name': 'CONNECT_RESP',
+        'client_to_server': False,
+        'id': 0x01,
+        'parameters': [(0x01, True), (0x00, False)]
+    },
+    {
+        'name': 'DISCONNECT_REQ',
+        'client_to_server': True,
+        'id': 0x02,
+        'parameters': []
+    },
+    {
+        'name': 'DISCONNECT_RESP',
+        'client_to_server': False,
+        'id': 0x03,
+        'parameters': []
+    },
+    {
+        'name': 'DISCONNECT_IND',
+        'client_to_server': False,
+        'id': 0x04,
+        'parameters': [(0x03, True)]
+    },
+    {
+        'name': 'TRANSFER_APDU_REQ',
+        'client_to_server': True,
+        'id': 0x05,
+        'parameters': [(0x04, False), (0x10, False)]
+    },
+    {
+        'name': 'TRANSFER_APDU_RESP',
+        'client_to_server': False,
+        'id': 0x06,
+        'parameters': [(0x02, True), (0x05, False)]
+    },
+    {
+        'name': 'TRANSFER_ATR_REQ',
+        'client_to_server': True,
+        'id': 0x07,
+        'parameters': []
+    },
+    {
+        'name': 'TRANSFER_ATR_RESP',
+        'client_to_server': False,
+        'id': 0x08,
+        'parameters': [(0x02, True), (0x06, False)]
+    },
+    {
+        'name': 'POWER_SIM_OFF_REQ',
+        'client_to_server': True,
+        'id': 0x09,
+        'parameters': []
+    },
+    {
+        'name': 'POWER_SIM_OFF_RESP',
+        'client_to_server': False,
+        'id': 0x0A,
+        'parameters': [(0x02, True)]
+    },
+    {
+        'name': 'POWER_SIM_ON_REQ',
+        'client_to_server': True,
+        'id': 0x0B,
+        'parameters': []
+    },
+    {
+        'name': 'POWER_SIM_ON_RESP',
+        'client_to_server': False,
+        'id': 0x0C,
+        'parameters': [(0x02, True)]
+    },
+    {
+        'name': 'RESET_SIM_REQ',
+        'client_to_server': True,
+        'id': 0x0D,
+        'parameters': []
+    },
+    {
+        'name': 'RESET_SIM_RESP',
+        'client_to_server': False,
+        'id': 0x0E,
+        'parameters': [(0x02, True)]
+    },
+    {
+        'name': 'TRANSFER_CARD_READER_STATUS_REQ',
+        'client_to_server': True,
+        'id': 0x0F,
+        'parameters': []
+    },
+    {
+        'name': 'TRANSFER_CARD_READER_STATUS_RESP',
+        'client_to_server': False,
+        'id': 0x10,
+        'parameters': [(0x02, True), (0x07, False)]
+    },
+    {
+        'name': 'STATUS_IND',
+        'client_to_server': False,
+        'id': 0x11,
+        'parameters': [(0x08, True)]
+    },
+
+    {
+        'name': 'ERROR_RESP',
+        'client_to_server': False,
+        'id': 0x12,
+        'parameters': []
+    },
+    {
+        'name': 'SET_TRANSPORT_PROTOCOL_REQ',
+        'client_to_server': True,
+        'id': 0x13,
+        'parameters': [(0x09, True)]
+    },
+    {
+        'name': 'SET_TRANSPORT_PROTOCOL_RESP',
+        'client_to_server': False,
+        'id': 0x14,
+        'parameters': [(0x02, True)]
+    },
+
+]
+
+
+class BluetoothSapSimLink(LinkBase):
+    # UUID for SIM Access Service
+    UUID_SIM_ACCESS = '0000112d-0000-1000-8000-00805f9b34fb'
+    SAP_MAX_MSG_SIZE = 0xffff
+
+    def __init__(self, bt_mac_addr, **kwargs):
+        super().__init__(**kwargs)
+        self._bt_mac_addr = bt_mac_addr
+        self._max_msg_size = self.SAP_MAX_MSG_SIZE
+        self._atr = None
+        self.connected = False
+        # at first try to find the bluetooth device
+        if not bluetooth.find_service(address=bt_mac_addr):
+            raise ReaderError(f"Cannot find bluetooth device [{bt_mac_addr}]")
+        # then check for rSAP support
+        self._sim_service = next(iter(bluetooth.find_service(
+            uuid=self.UUID_SIM_ACCESS, address=bt_mac_addr)), None)
+        if not self._sim_service:
+            raise ReaderError(
+                f"Bluetooth device [{bt_mac_addr}] does not support SIM Access service")
+
+    def __del__(self):
+        # TODO: do something here
+        pass
+
+    def wait_for_card(self, timeout=None, newcardonly=False):
+        self.connect()
+
+    def connect(self):
+        try:
+            self._sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
+            self._sock.connect(
+                (self._sim_service['host'], self._sim_service['port']))
+            self.connected = True
+            self.establish_sim_connection()
+            self.retrieve_atr()
+        except:
+            raise ReaderError("Cannot connect to SIM Access service")
+
+    def get_atr(self):
+        return self._atr
+
+    def disconnect(self):
+        if self.connected:
+            self.send_sap_message("DISCONNECT_REQ")
+        self._sock.close()
+        self.connected = False
+
+    def reset_card(self):
+        if self.connected:
+            self.send_sap_message("RESET_SIM_REQ")
+            msg_name, param_list = self._recv_sap_response('RESET_SIM_RESP')
+            connection_status = next(
+                (x[1] for x in param_list if x[0] == 'ConnectionStatus'), 0x01)
+            if connection_status == 0x00:
+                logger.info("SIM Reset successful")
+                return 1
+        else:
+            self.disconnect()
+            self.connect()
+        return 1
+
+    def send_sap_message(self, msg_name, param_list=[]):
+        # maby check for idle state before sending?
+        message = self.craft_sap_message(msg_name, param_list)
+        return self._sock.send(message)
+
+    def _recv_sap_message(self):
+        resp = self._sock.recv(self._max_msg_size)
+        msg_name, param_list = self.parse_sap_message(resp)
+        return msg_name, param_list
+
+    def _recv_sap_response(self, waiting_msg_name):
+        while self.connected:
+            msg_name, param_list = self._recv_sap_message()
+            self.handle_sap_response_generic(msg_name, param_list)
+            if msg_name == waiting_msg_name:
+                return msg_name, param_list
+
+    def establish_sim_connection(self, retries=5):
+        self.send_sap_message(
+            "CONNECT_REQ", [("MaxMsgSize", self._max_msg_size)])
+        msg_name, param_list = self._recv_sap_response('CONNECT_RESP')
+
+        connection_status = next(
+            (x[1] for x in param_list if x[0] == 'ConnectionStatus'), 0x01)
+        if connection_status == 0x00:
+            logger.info("Successfully connected to rSAP server")
+            return
+        elif connection_status == 0x02:  # invalid max size
+            self._max_msg_size = next(
+                (x[1] for x in param_list if x[0] == 'MaxMsgSize'), self._max_msg_size)
+            return self.establish_sim_connection(retries)
+        else:
+            logger.info(
+                "Wait some seconds and make another connection attempt...")
+            time.sleep(5)
+            return self.establish_sim_connection(retries-1)
+
+    def retrieve_atr(self):
+        self.send_sap_message("TRANSFER_ATR_REQ")
+        msg_name, param_list = self._recv_sap_response('TRANSFER_ATR_RESP')
+        result_code = next(
+            (x[1] for x in param_list if x[0] == 'ResultCode'), 0x01)
+        if result_code == 0x00:
+            atr = next((x[1] for x in param_list if x[0] == 'ATR'), None)
+            self._atr = atr
+            logger.debug(f"Recieved ATR from server: {b2h(atr)}")
+
+    def handle_sap_response_generic(self, msg_name, param_list):
+        # print stuff
+        logger.debug(
+            f"Recieved sap message from server: {(msg_name, param_list)}")
+        for param in param_list:
+            param_name, param_value = param
+            if param_name == 'ConnectionStatus':
+                new_status = SAP_CONNECTION_STATUS.get(param_value)
+                logger.debug(f"Connection Status: {new_status}")
+            elif param_name == 'StatusChange':
+                new_status = SAP_STATUS_CHANGE.get(param_value)
+                logger.debug(f"SIM Status: {new_status}")
+            elif param_name == 'ResultCode':
+                response_code = SAP_RESULT_CODE.get(param_value)
+                logger.debug(f"ResultCode: {response_code}")
+
+        # handle some important stuff:
+        if msg_name == 'DISCONNECT_IND':
+            # graceful disconnect --> technically could still send some apdus
+            # however, we just make it short and sweet and directly disconnect
+            self.send_sap_message("DISCONNECT_REQ")
+        elif msg_name == 'DISCONNECT_RESP':
+            self.connected = False
+            logger.info(f"Client disconnected")
+
+        # if msg_name == 'CONNECT_RESP':
+        # elif msg_name == 'DISCONNECT_RESP':
+        # elif msg_name == 'DISCONNECT_IND':
+        # elif msg_name == 'TRANSFER_APDU_RESP':
+        # elif msg_name == 'TRANSFER_ATR_RESP':
+        # elif msg_name == 'POWER_SIM_OFF_RESP':
+        # elif msg_name == 'POWER_SIM_ON_RESP':
+        # elif msg_name == 'RESET_SIM_RESP':
+        # elif msg_name == 'TRANSFER_CARD_READER_STATUS_RESP':
+        # elif msg_name == 'STATUS_IND':
+        # elif msg_name == 'ERROR_RESP':
+        # elif msg_name == 'SET_TRANSPORT_PROTOCOL_RESP':
+        # else:
+        #  logger.error("Unknown message...")
+
+    def craft_sap_message(self, msg_name, param_list=[]):
+        msg_info = next(
+            (x for x in SAP_MESSAGES if x.get('name') == msg_name), None)
+        if not msg_info:
+            raise ProtocolError(f"Unknown SAP message name ({msg_name})")
+
+        msg_id = msg_info.get('id')
+        msg_params = msg_info.get('parameters')
+        # msg_direction = msg_info.get('client_to_server')
+
+        param_cnt = len(param_list)
+
+        msg_bytes = struct.pack(
+            '!BBH',
+            msg_id,
+            param_cnt,
+            0
+        )
+
+        allowed_params = (x[0] for x in msg_params)
+        mandatory_params = (x[0] for x in msg_params if x[1] == True)
+
+        collected_param_ids = []
+
+        for p in param_list:
+            param_name = p[0]
+            param_value = p[1]
+
+            param_id = next(
+                (x.get('id') for x in SAP_PARAMETERS if x.get('name') == param_name), None)
+            if param_id is None:
+                raise ProtocolError(f"Unknown SAP param name ({param_name})")
+            if param_id not in allowed_params:
+                raise ProtocolError(
+                    f"Parameter {param_name} not allowed in message {msg_name}")
+
+            collected_param_ids.append(param_id)
+            msg_bytes += self.craft_sap_parameter(param_name, param_value)
+
+        if not set(mandatory_params).issubset(collected_param_ids):
+            raise ProtocolError(
+                f"Missing mandatory parameter for message {msg_name} (mandatory: {*mandatory_params,}, present: {*collected_param_ids,})")
+
+        return msg_bytes
+
+    def calc_padding_len(self, length, blocksize=4):
+        extra = length % blocksize
+        if extra > 0:
+            return blocksize-extra
+        return 0
+
+    def pad_bytes(self, b, blocksize=4):
+        padding_len = self.calc_padding_len(len(b), blocksize)
+        return b + bytearray(padding_len)
+
+    def craft_sap_parameter(self, param_name, param_value):
+        param_info = next(
+            (x for x in SAP_PARAMETERS if x.get('name') == param_name), None)
+        param_id = param_info.get('id')
+        param_len = param_info.get('length')
+
+        if isinstance(param_value, str):
+            param_value = h2b(param_value)
+
+        if isinstance(param_value, int):
+            # TODO: when param len is not set we have a problem :X
+            param_value = (param_value).to_bytes(param_len, byteorder='big')
+
+        if param_len is None:
+            # just assume param length from bytearray
+            param_len = len(param_value)
+        elif param_len != len(param_value):
+            raise ProtocolError(
+                f"Invalid param length (epected {param_len} but got {len(param_value)} bytes)")
+
+        param_bytes = struct.pack(
+            f'!BBH{param_len}s',
+            param_id,
+            0,  # reserved
+            param_len,
+            param_value
+        )
+        param_bytes = self.pad_bytes(param_bytes)
+        return param_bytes
+
+    def parse_sap_message(self, msg_bytes):
+        header_struct = struct.Struct('!BBH')
+        msg_id, param_cnt, reserved = header_struct.unpack_from(msg_bytes)
+        msg_bytes = msg_bytes[header_struct.size:]
+
+        msg_info = next(
+            (x for x in SAP_MESSAGES if x.get('id') == msg_id), None)
+
+        msg_name = msg_info.get('name')
+        msg_params = msg_info.get('parameters')
+        # msg_direction = msg_info.get('client_to_server')
+
+        # TODO: check if params allowed etc
+        # allowed_params = (x[0] for x in msg_params)
+        # mandatory_params = (x[0] for x in msg_params if x[1] == True)
+
+        param_list = []
+
+        for x in range(param_cnt):
+            param_name, param_value, total_len = self.parse_sap_parameter(
+                msg_bytes)
+            param_list.append((param_name, param_value))
+            msg_bytes = msg_bytes[total_len:]
+
+        return msg_name, param_list
+
+    def parse_sap_parameter(self, param_bytes):
+        header_struct = struct.Struct('!BBH')
+        total_len = header_struct.size
+        param_id, reserved, param_len = header_struct.unpack_from(param_bytes)
+        padding_len = self.calc_padding_len(param_len)
+        paramval_struct = struct.Struct(f'!{param_len}s{padding_len}s')
+        param_value, padding = paramval_struct.unpack_from(
+            param_bytes[total_len:])
+        total_len += paramval_struct.size
+
+        param_info = next(
+            (x for x in SAP_PARAMETERS if x.get('id') == param_id), None)
+        # TODO: check if param found, length plausible, ...
+        param_name = param_info.get('name')
+
+        # if it is set then value was int, otherwise it is byte array
+        if param_info.get('length') is not None:
+            param_value = int.from_bytes(param_value, "big")
+        # param_len = param_info.get('length')
+        return param_name, param_value, total_len
+
+    def _send_apdu_raw(self, pdu):
+        if isinstance(pdu, str):
+            pdu = h2b(pdu)
+        self.send_sap_message("TRANSFER_APDU_REQ", [("CommandAPDU", pdu)])
+
+        msg_name, param_list = self._recv_sap_response('TRANSFER_APDU_RESP')
+        result_code = next(
+            (x[1] for x in param_list if x[0] == 'ResultCode'), 0x01)
+        if result_code == 0x00:
+            response = next(
+                (x[1] for x in param_list if x[0] == 'ResponseAPDU'), None)
+            sw = response[-2:]
+            data = response[0:-2]
+            return b2h(data), b2h(sw)
+        return None, None

pySim/transport/calypso.py

@@ -1,10 +1,5 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-""" pySim: Transport Link for Calypso bases phones
-"""
-
-#
 # Copyright (C) 2018 Vadim Yanitskiy <axilirator@gmail.com>
 #
 # This program is free software: you can redistribute it and/or modify
@@ -21,8 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
-
 import select
 import struct
 import socket
@@ -32,126 +25,132 @@ from pySim.transport import LinkBase
 from pySim.exceptions import *
 from pySim.utils import h2b, b2h
 
+
 class L1CTLMessage(object):
 
-	# Every (encoded) L1CTL message has the following structure:
-	#  - msg_length (2 bytes, net order)
-	#  - l1ctl_hdr (packed structure)
-	#    - msg_type
-	#    - flags
-	#    - padding (2 spare bytes)
-	#  - ... payload ...
+    # Every (encoded) L1CTL message has the following structure:
+    #  - msg_length (2 bytes, net order)
+    #  - l1ctl_hdr (packed structure)
+    #    - msg_type
+    #    - flags
+    #    - padding (2 spare bytes)
+    #  - ... payload ...
 
-	def __init__(self, msg_type, flags = 0x00):
-		# Init L1CTL message header
-		self.data = struct.pack("BBxx", msg_type, flags)
+    def __init__(self, msg_type, flags=0x00):
+        # Init L1CTL message header
+        self.data = struct.pack("BBxx", msg_type, flags)
+
+    def gen_msg(self):
+        return struct.pack("!H", len(self.data)) + self.data
 
-	def gen_msg(self):
-		return struct.pack("!H", len(self.data)) + self.data
 
 class L1CTLMessageReset(L1CTLMessage):
 
-	# L1CTL message types
-	L1CTL_RESET_REQ		= 0x0d
-	L1CTL_RESET_IND		= 0x07
-	L1CTL_RESET_CONF	= 0x0e
+    # L1CTL message types
+    L1CTL_RESET_REQ = 0x0d
+    L1CTL_RESET_IND = 0x07
+    L1CTL_RESET_CONF = 0x0e
 
-	# Reset types
-	L1CTL_RES_T_BOOT	= 0x00
-	L1CTL_RES_T_FULL	= 0x01
-	L1CTL_RES_T_SCHED	= 0x02
+    # Reset types
+    L1CTL_RES_T_BOOT = 0x00
+    L1CTL_RES_T_FULL = 0x01
+    L1CTL_RES_T_SCHED = 0x02
+
+    def __init__(self, type=L1CTL_RES_T_FULL):
+        super(L1CTLMessageReset, self).__init__(self.L1CTL_RESET_REQ)
+        self.data += struct.pack("Bxxx", type)
 
-	def __init__(self, type = L1CTL_RES_T_FULL):
-		super(L1CTLMessageReset, self).__init__(self.L1CTL_RESET_REQ)
-		self.data += struct.pack("Bxxx", type)
 
 class L1CTLMessageSIM(L1CTLMessage):
 
-	# SIM related message types
-	L1CTL_SIM_REQ		= 0x16
-	L1CTL_SIM_CONF		= 0x17
+    # SIM related message types
+    L1CTL_SIM_REQ = 0x16
+    L1CTL_SIM_CONF = 0x17
+
+    def __init__(self, pdu):
+        super(L1CTLMessageSIM, self).__init__(self.L1CTL_SIM_REQ)
+        self.data += pdu
 
-	def __init__(self, pdu):
-		super(L1CTLMessageSIM, self).__init__(self.L1CTL_SIM_REQ)
-		self.data += pdu
 
 class CalypsoSimLink(LinkBase):
+    """Transport Link for Calypso based phones."""
 
-	def __init__(self, sock_path = "/tmp/osmocom_l2"):
-		# Make sure that a given socket path exists
-		if not os.path.exists(sock_path):
-			raise ReaderError("There is no such ('%s') UNIX socket" % sock_path)
+    def __init__(self, sock_path: str = "/tmp/osmocom_l2", **kwargs):
+        super().__init__(**kwargs)
+        # Make sure that a given socket path exists
+        if not os.path.exists(sock_path):
+            raise ReaderError(
+                "There is no such ('%s') UNIX socket" % sock_path)
 
-		print("Connecting to osmocon at '%s'..." % sock_path)
+        print("Connecting to osmocon at '%s'..." % sock_path)
 
-		# Establish a client connection
-		self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-		self.sock.connect(sock_path)
+        # Establish a client connection
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        self.sock.connect(sock_path)
 
-	def __del__(self):
-		self.sock.close()
+    def __del__(self):
+        self.sock.close()
 
-	def wait_for_rsp(self, exp_len = 128):
-		# Wait for incoming data (timeout is 3 seconds)
-		s, _, _ = select.select([self.sock], [], [], 3.0)
-		if not s:
-			raise ReaderError("Timeout waiting for card response")
+    def wait_for_rsp(self, exp_len=128):
+        # Wait for incoming data (timeout is 3 seconds)
+        s, _, _ = select.select([self.sock], [], [], 3.0)
+        if not s:
+            raise ReaderError("Timeout waiting for card response")
 
-		# Receive expected amount of bytes from osmocon
-		rsp = self.sock.recv(exp_len)
-		return rsp
+        # Receive expected amount of bytes from osmocon
+        rsp = self.sock.recv(exp_len)
+        return rsp
 
-	def reset_card(self):
-		# Request FULL reset
-		req_msg = L1CTLMessageReset()
-		self.sock.send(req_msg.gen_msg())
+    def reset_card(self):
+        # Request FULL reset
+        req_msg = L1CTLMessageReset()
+        self.sock.send(req_msg.gen_msg())
 
-		# Wait for confirmation
-		rsp = self.wait_for_rsp()
-		rsp_msg = struct.unpack_from("!HB", rsp)
-		if rsp_msg[1] != L1CTLMessageReset.L1CTL_RESET_CONF:
-			raise ReaderError("Failed to reset Calypso PHY")
+        # Wait for confirmation
+        rsp = self.wait_for_rsp()
+        rsp_msg = struct.unpack_from("!HB", rsp)
+        if rsp_msg[1] != L1CTLMessageReset.L1CTL_RESET_CONF:
+            raise ReaderError("Failed to reset Calypso PHY")
 
-	def connect(self):
-		self.reset_card()
+    def connect(self):
+        self.reset_card()
 
-	def disconnect(self):
-		pass # Nothing to do really ...
+    def disconnect(self):
+        pass  # Nothing to do really ...
 
-	def wait_for_card(self, timeout = None, newcardonly = False):
-		pass # Nothing to do really ...
+    def wait_for_card(self, timeout=None, newcardonly=False):
+        pass  # Nothing to do really ...
 
-	def send_apdu_raw(self, pdu):
-		"""see LinkBase.send_apdu_raw"""
+    def _send_apdu_raw(self, pdu):
 
-		# Request FULL reset
-		req_msg = L1CTLMessageSIM(h2b(pdu))
-		self.sock.send(req_msg.gen_msg())
+        # Request FULL reset
+        req_msg = L1CTLMessageSIM(h2b(pdu))
+        self.sock.send(req_msg.gen_msg())
 
-		# Read message length first
-		rsp = self.wait_for_rsp(struct.calcsize("!H"))
-		msg_len = struct.unpack_from("!H", rsp)[0]
-		if msg_len < struct.calcsize("BBxx"):
-			raise ReaderError("Missing L1CTL header for L1CTL_SIM_CONF")
+        # Read message length first
+        rsp = self.wait_for_rsp(struct.calcsize("!H"))
+        msg_len = struct.unpack_from("!H", rsp)[0]
+        if msg_len < struct.calcsize("BBxx"):
+            raise ReaderError("Missing L1CTL header for L1CTL_SIM_CONF")
 
-		# Read the whole message then
-		rsp = self.sock.recv(msg_len)
+        # Read the whole message then
+        rsp = self.sock.recv(msg_len)
 
-		# Verify L1CTL header
-		hdr = struct.unpack_from("BBxx", rsp)
-		if hdr[0] != L1CTLMessageSIM.L1CTL_SIM_CONF:
-			raise ReaderError("Unexpected L1CTL message received")
+        # Verify L1CTL header
+        hdr = struct.unpack_from("BBxx", rsp)
+        if hdr[0] != L1CTLMessageSIM.L1CTL_SIM_CONF:
+            raise ReaderError("Unexpected L1CTL message received")
 
-		# Verify the payload length
-		offset = struct.calcsize("BBxx")
-		if len(rsp) <= offset:
-			raise ProtocolError("Empty response from SIM?!?")
+        # Verify the payload length
+        offset = struct.calcsize("BBxx")
+        if len(rsp) <= offset:
+            raise ProtocolError("Empty response from SIM?!?")
 
-		# Omit L1CTL header
-		rsp = rsp[offset:]
+        # Omit L1CTL header
+        rsp = rsp[offset:]
 
-		# Unpack data and SW
-		data = rsp[:-2]
-		sw = rsp[-2:]
+        # Unpack data and SW
+        data = rsp[:-2]
+        sw = rsp[-2:]
 
-		return b2h(data), b2h(sw)
+        return b2h(data), b2h(sw)

pySim/transport/modem_atcmd.py

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-""" pySim: Transport Link for 3GPP TS 27.007 compliant modems
-"""
-
 # Copyright (C) 2020 Vadim Yanitskiy <axilirator@gmail.com>
 #
 # This program is free software: you can redistribute it and/or modify
@@ -20,8 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
-
 import logging as log
 import serial
 import time
@@ -33,94 +27,142 @@ from pySim.exceptions import *
 # HACK: if somebody needs to debug this thing
 # log.root.setLevel(log.DEBUG)
 
+
 class ModemATCommandLink(LinkBase):
-	def __init__(self, device='/dev/ttyUSB0', baudrate=115200):
-		self._sl = serial.Serial(device, baudrate, timeout=5)
-		self._device = device
-		self._atr = None
+    """Transport Link for 3GPP TS 27.007 compliant modems."""
 
-		# Trigger initial reset
-		self.reset_card()
+    def __init__(self, device: str = '/dev/ttyUSB0', baudrate: int = 115200, **kwargs):
+        super().__init__(**kwargs)
+        self._sl = serial.Serial(device, baudrate, timeout=5)
+        self._echo = False		# this will be auto-detected by _check_echo()
+        self._device = device
+        self._atr = None
 
-	def __del__(self):
-		self._sl.close()
+        # Check the AT interface
+        self._check_echo()
 
-	def send_at_cmd(self, cmd):
-		# Convert from string to bytes, if needed
-		bcmd = cmd if type(cmd) is bytes else cmd.encode()
-		bcmd += b'\r'
+        # Trigger initial reset
+        self.reset_card()
 
-		# Send command to the modem
-		log.debug('Sending AT command: %s' % cmd)
-		try:
-			wlen = self._sl.write(bcmd)
-			assert(wlen == len(bcmd))
-		except:
-			raise ReaderError('Failed to send AT command: %s' % cmd)
+    def __del__(self):
+        if hasattr(self, '_sl'):
+            self._sl.close()
 
-		# Give the modem some time...
-		time.sleep(0.3)
+    def send_at_cmd(self, cmd, timeout=0.2, patience=0.002):
+        # Convert from string to bytes, if needed
+        bcmd = cmd if type(cmd) is bytes else cmd.encode()
+        bcmd += b'\r'
 
-		# Read the response
-		try:
-			# Skip characters sent back
-			self._sl.read(wlen)
-			# Read the rest
-			rsp = self._sl.read_all()
+        # Clean input buffer from previous/unexpected data
+        self._sl.reset_input_buffer()
 
-			# Strip '\r\n'
-			rsp = rsp.strip()
-			# Split into a list
-			rsp = rsp.split(b'\r\n\r\n')
-		except:
-			raise ReaderError('Failed parse response to AT command: %s' % cmd)
+        # Send command to the modem
+        log.debug('Sending AT command: %s', cmd)
+        try:
+            wlen = self._sl.write(bcmd)
+            assert(wlen == len(bcmd))
+        except:
+            raise ReaderError('Failed to send AT command: %s' % cmd)
 
-		log.debug('Got response from modem: %s' % rsp)
-		return rsp
+        rsp = b''
+        its = 1
+        t_start = time.time()
+        while True:
+            rsp = rsp + self._sl.read(self._sl.in_waiting)
+            lines = rsp.split(b'\r\n')
+            if len(lines) >= 2:
+                res = lines[-2]
+                if res == b'OK':
+                    log.debug('Command finished with result: %s', res)
+                    break
+                if res == b'ERROR' or res.startswith(b'+CME ERROR:'):
+                    log.error('Command failed with result: %s', res)
+                    break
 
-	def reset_card(self):
-		# Make sure that we can talk to the modem
-		if self.send_at_cmd('AT') != [b'OK']:
-			raise ReaderError('Failed to connect to modem')
+            if time.time() - t_start >= timeout:
+                log.info('Command finished with timeout >= %ss', timeout)
+                break
+            time.sleep(patience)
+            its += 1
+        log.debug('Command took %0.6fs (%d cycles a %fs)',
+                  time.time() - t_start, its, patience)
 
-		# Reset the modem, just to be sure
-		if self.send_at_cmd('ATZ') != [b'OK']:
-			raise ReaderError('Failed to reset the modem')
+        if self._echo:
+            # Skip echo chars
+            rsp = rsp[wlen:]
+        rsp = rsp.strip()
+        rsp = rsp.split(b'\r\n\r\n')
 
-		# Make sure that generic SIM access is supported
-		if self.send_at_cmd('AT+CSIM=?') != [b'OK']:
-			raise ReaderError('The modem does not seem to support SIM access')
+        log.debug('Got response from modem: %s', rsp)
+        return rsp
 
-		log.info('Modem at \'%s\' is ready!' % self._device)
+    def _check_echo(self):
+        """Verify the correct response to 'AT' command
+        and detect if inputs are echoed by the device
 
-	def connect(self):
-		pass # Nothing to do really ...
+        Although echo of inputs can be enabled/disabled via
+        ATE1/ATE0, respectively, we rather detect the current
+        configuration of the modem without any change.
+        """
+        # Next command shall not strip the echo from the response
+        self._echo = False
+        result = self.send_at_cmd('AT')
 
-	def disconnect(self):
-		pass # Nothing to do really ...
+        # Verify the response
+        if len(result) > 0:
+            if result[-1] == b'OK':
+                self._echo = False
+                return
+            elif result[-1] == b'AT\r\r\nOK':
+                self._echo = True
+                return
+        raise ReaderError(
+            'Interface \'%s\' does not respond to \'AT\' command' % self._device)
 
-	def wait_for_card(self, timeout=None, newcardonly=False):
-		pass # Nothing to do really ...
+    def reset_card(self):
+        # Reset the modem, just to be sure
+        if self.send_at_cmd('ATZ') != [b'OK']:
+            raise ReaderError('Failed to reset the modem')
 
-	def send_apdu_raw(self, pdu):
-		# Prepare the command as described in 8.17
-		cmd = 'AT+CSIM=%d,\"%s\"' % (len(pdu), pdu)
+        # Make sure that generic SIM access is supported
+        if self.send_at_cmd('AT+CSIM=?') != [b'OK']:
+            raise ReaderError('The modem does not seem to support SIM access')
 
-		# Send AT+CSIM command to the modem
-		# TODO: also handle +CME ERROR: <err>
-		rsp = self.send_at_cmd(cmd)
-		if len(rsp) != 2 or rsp[-1] != b'OK':
-			raise ReaderError('APDU transfer failed: %s' % str(rsp))
-		rsp = rsp[0] # Get rid of b'OK'
+        log.info('Modem at \'%s\' is ready!' % self._device)
 
-		# Make sure that the response has format: b'+CSIM: %d,\"%s\"'
-		try:
-			result = re.match(b'\+CSIM: (\d+),\"([0-9A-F]+)\"', rsp)
-			(rsp_pdu_len, rsp_pdu) = result.groups()
-		except:
-			raise ReaderError('Failed to parse response from modem: %s' % rsp)
+    def connect(self):
+        pass  # Nothing to do really ...
 
-		# TODO: make sure we have at least SW
-		data = rsp_pdu[:-4].decode()
-		sw   = rsp_pdu[-4:].decode()
-		return data, sw
+    def disconnect(self):
+        pass  # Nothing to do really ...
+
+    def wait_for_card(self, timeout=None, newcardonly=False):
+        pass  # Nothing to do really ...
+
+    def _send_apdu_raw(self, pdu):
+        # Make sure pdu has upper case hex digits [A-F]
+        pdu = pdu.upper()
+
+        # Prepare the command as described in 8.17
+        cmd = 'AT+CSIM=%d,\"%s\"' % (len(pdu), pdu)
+        log.debug('Sending command: %s',  cmd)
+
+        # Send AT+CSIM command to the modem
+        # TODO: also handle +CME ERROR: <err>
+        rsp = self.send_at_cmd(cmd)
+        if len(rsp) != 2 or rsp[-1] != b'OK':
+            raise ReaderError('APDU transfer failed: %s' % str(rsp))
+        rsp = rsp[0]  # Get rid of b'OK'
+
+        # Make sure that the response has format: b'+CSIM: %d,\"%s\"'
+        try:
+            result = re.match(b'\+CSIM: (\d+),\"([0-9A-F]+)\"', rsp)
+            (rsp_pdu_len, rsp_pdu) = result.groups()
+        except:
+            raise ReaderError('Failed to parse response from modem: %s' % rsp)
+
+        # TODO: make sure we have at least SW
+        data = rsp_pdu[:-4].decode().lower()
+        sw = rsp_pdu[-4:].decode().lower()
+        log.debug('Command response: %s, %s',  data, sw)
+        return data, sw

pySim/transport/pcsc.py

@@ -1,10 +1,5 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-""" pySim: PCSC reader transport link
-"""
-
-#
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
 # Copyright (C) 2010  Harald Welte <laforge@gnumonks.org>
 #
@@ -24,61 +19,73 @@
 
 from smartcard.CardConnection import CardConnection
 from smartcard.CardRequest import CardRequest
-from smartcard.Exceptions import NoCardException, CardRequestTimeoutException, CardConnectionException
+from smartcard.Exceptions import NoCardException, CardRequestTimeoutException, CardConnectionException, CardConnectionException
 from smartcard.System import readers
 
-from pySim.exceptions import NoCardError, ProtocolError
+from pySim.exceptions import NoCardError, ProtocolError, ReaderError
 from pySim.transport import LinkBase
 from pySim.utils import h2i, i2h
 
 
 class PcscSimLink(LinkBase):
+    """ pySim: PCSC reader transport link."""
 
-	def __init__(self, reader_number=0):
-		r = readers()
-		self._reader = r[reader_number]
-		self._con = self._reader.createConnection()
+    def __init__(self, reader_number: int = 0, **kwargs):
+        super().__init__(**kwargs)
+        r = readers()
+        if reader_number >= len(r):
+            raise ReaderError
+        self._reader = r[reader_number]
+        self._con = self._reader.createConnection()
 
-	def __del__(self):
-		self._con.disconnect()
-		return
+    def __del__(self):
+        try:
+            # FIXME: this causes multiple warnings in Python 3.5.3
+            self._con.disconnect()
+        except:
+            pass
+        return
 
-	def wait_for_card(self, timeout=None, newcardonly=False):
-		cr = CardRequest(readers=[self._reader], timeout=timeout, newcardonly=newcardonly)
-		try:
-			cr.waitforcard()
-		except CardRequestTimeoutException:
-			raise NoCardError()
-		self.connect()
+    def wait_for_card(self, timeout: int = None, newcardonly: bool = False):
+        cr = CardRequest(readers=[self._reader],
+                         timeout=timeout, newcardonly=newcardonly)
+        try:
+            cr.waitforcard()
+        except CardRequestTimeoutException:
+            raise NoCardError()
+        self.connect()
 
-	def connect(self):
-		try:
-			# Explicitly select T=0 communication protocol
-			self._con.connect(CardConnection.T0_protocol)
-		except CardConnectionException:
-			raise ProtocolError()
-		except NoCardException:
-			raise NoCardError()
+    def connect(self):
+        try:
+            # To avoid leakage of resources, make sure the reader
+            # is disconnected
+            self.disconnect()
 
-	def get_atr(self):
-		return self._con.getATR()
+            # Explicitly select T=0 communication protocol
+            self._con.connect(CardConnection.T0_protocol)
+        except CardConnectionException:
+            raise ProtocolError()
+        except NoCardException:
+            raise NoCardError()
 
-	def disconnect(self):
-		self._con.disconnect()
+    def get_atr(self):
+        return self._con.getATR()
 
-	def reset_card(self):
-		self.disconnect()
-		self.connect()
-		return 1
+    def disconnect(self):
+        self._con.disconnect()
 
-	def send_apdu_raw(self, pdu):
-		"""see LinkBase.send_apdu_raw"""
+    def reset_card(self):
+        self.disconnect()
+        self.connect()
+        return 1
 
-		apdu = h2i(pdu)
+    def _send_apdu_raw(self, pdu):
 
-		data, sw1, sw2 = self._con.transmit(apdu)
+        apdu = h2i(pdu)
 
-		sw = [sw1, sw2]
+        data, sw1, sw2 = self._con.transmit(apdu)
 
-		# Return value
-		return i2h(data), i2h(sw)
+        sw = [sw1, sw2]
+
+        # Return value
+        return i2h(data), i2h(sw)

pySim/transport/serial.py

@@ -1,10 +1,5 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-""" pySim: Transport Link for serial (RS232) based readers included with simcard
-"""
-
-#
 # Copyright (C) 2009-2010  Sylvain Munaut <tnt@246tNt.com>
 #
 # This program is free software: you can redistribute it and/or modify
@@ -21,8 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
-
 import serial
 import time
 import os.path
@@ -33,207 +26,211 @@ from pySim.utils import h2b, b2h
 
 
 class SerialSimLink(LinkBase):
+    """ pySim: Transport Link for serial (RS232) based readers included with simcard"""
 
-	def __init__(self, device='/dev/ttyUSB0', baudrate=9600, rst='-rts', debug=False):
-		if not os.path.exists(device):
-			raise ValueError("device file %s does not exist -- abort" % device)
-		self._sl = serial.Serial(
-				port = device,
-				parity = serial.PARITY_EVEN,
-				bytesize = serial.EIGHTBITS,
-				stopbits = serial.STOPBITS_TWO,
-				timeout = 1,
-				xonxoff = 0,
-				rtscts = 0,
-				baudrate = baudrate,
-			)
-		self._rst_pin = rst
-		self._debug = debug
-		self._atr = None
+    def __init__(self, device: str = '/dev/ttyUSB0', baudrate: int = 9600, rst: str = '-rts',
+                 debug: bool = False, **kwargs):
+        super().__init__(**kwargs)
+        if not os.path.exists(device):
+            raise ValueError("device file %s does not exist -- abort" % device)
+        self._sl = serial.Serial(
+            port=device,
+            parity=serial.PARITY_EVEN,
+            bytesize=serial.EIGHTBITS,
+            stopbits=serial.STOPBITS_TWO,
+            timeout=1,
+            xonxoff=0,
+            rtscts=0,
+            baudrate=baudrate,
+        )
+        self._rst_pin = rst
+        self._debug = debug
+        self._atr = None
 
-	def __del__(self):
-		if (hasattr(self, "_sl")):
-			self._sl.close()
+    def __del__(self):
+        if (hasattr(self, "_sl")):
+            self._sl.close()
 
-	def wait_for_card(self, timeout=None, newcardonly=False):
-		# Direct try
-		existing = False
+    def wait_for_card(self, timeout=None, newcardonly=False):
+        # Direct try
+        existing = False
 
-		try:
-			self.reset_card()
-			if not newcardonly:
-				return
-			else:
-				existing = True
-		except NoCardError:
-			pass
+        try:
+            self.reset_card()
+            if not newcardonly:
+                return
+            else:
+                existing = True
+        except NoCardError:
+            pass
 
-		# Poll ...
-		mt = time.time() + timeout if timeout is not None else None
-		pe = 0
+        # Poll ...
+        mt = time.time() + timeout if timeout is not None else None
+        pe = 0
 
-		while (mt is None) or (time.time() < mt):
-			try:
-				time.sleep(0.5)
-				self.reset_card()
-				if not existing:
-					return
-			except NoCardError:
-				existing = False
-			except ProtocolError:
-				if existing:
-					existing = False
-				else:
-					# Tolerate a couple of protocol error ... can happen if
-					# we try when the card is 'half' inserted
-					pe += 1
-					if (pe > 2):
-						raise
+        while (mt is None) or (time.time() < mt):
+            try:
+                time.sleep(0.5)
+                self.reset_card()
+                if not existing:
+                    return
+            except NoCardError:
+                existing = False
+            except ProtocolError:
+                if existing:
+                    existing = False
+                else:
+                    # Tolerate a couple of protocol error ... can happen if
+                    # we try when the card is 'half' inserted
+                    pe += 1
+                    if (pe > 2):
+                        raise
 
-		# Timed out ...
-		raise NoCardError()
+        # Timed out ...
+        raise NoCardError()
 
-	def connect(self):
-		self.reset_card()
+    def connect(self):
+        self.reset_card()
 
-	def get_atr(self):
-		return self._atr
+    def get_atr(self):
+        return self._atr
 
-	def disconnect(self):
-		pass # Nothing to do really ...
+    def disconnect(self):
+        pass  # Nothing to do really ...
 
-	def reset_card(self):
-		rv = self._reset_card()
-		if rv == 0:
-			raise NoCardError()
-		elif rv < 0:
-			raise ProtocolError()
+    def reset_card(self):
+        rv = self._reset_card()
+        if rv == 0:
+            raise NoCardError()
+        elif rv < 0:
+            raise ProtocolError()
 
-	def _reset_card(self):
-		self._atr = None
-		rst_meth_map = {
-			'rts': self._sl.setRTS,
-			'dtr': self._sl.setDTR,
-		}
-		rst_val_map = { '+':0, '-':1 }
+    def _reset_card(self):
+        self._atr = None
+        rst_meth_map = {
+            'rts': self._sl.setRTS,
+            'dtr': self._sl.setDTR,
+        }
+        rst_val_map = {'+': 0, '-': 1}
 
-		try:
-			rst_meth = rst_meth_map[self._rst_pin[1:]]
-			rst_val  = rst_val_map[self._rst_pin[0]]
-		except:
-			raise ValueError('Invalid reset pin %s' % self._rst_pin)
+        try:
+            rst_meth = rst_meth_map[self._rst_pin[1:]]
+            rst_val = rst_val_map[self._rst_pin[0]]
+        except:
+            raise ValueError('Invalid reset pin %s' % self._rst_pin)
 
-		rst_meth(rst_val)
-		time.sleep(0.1)  # 100 ms
-		self._sl.flushInput()
-		rst_meth(rst_val ^ 1)
+        rst_meth(rst_val)
+        time.sleep(0.1)  # 100 ms
+        self._sl.flushInput()
+        rst_meth(rst_val ^ 1)
 
-		b = self._rx_byte()
-		if not b:
-			return 0
-		if ord(b) != 0x3b:
-			return -1
-		self._dbg_print("TS: 0x%x Direct convention" % ord(b))
+        b = self._rx_byte()
+        if not b:
+            return 0
+        if ord(b) != 0x3b:
+            return -1
+        self._dbg_print("TS: 0x%x Direct convention" % ord(b))
 
-		while ord(b) == 0x3b:
-			b = self._rx_byte()
+        while ord(b) == 0x3b:
+            b = self._rx_byte()
 
-		if not b:
-			return -1
-		t0 = ord(b)
-		self._dbg_print("T0: 0x%x" % t0)
-		self._atr = [0x3b, ord(b)]
+        if not b:
+            return -1
+        t0 = ord(b)
+        self._dbg_print("T0: 0x%x" % t0)
+        self._atr = [0x3b, ord(b)]
 
-		for i in range(4):
-			if t0 & (0x10 << i):
-				b = self._rx_byte()
-				self._atr.append(ord(b))
-				self._dbg_print("T%si = %x" % (chr(ord('A')+i), ord(b)))
+        for i in range(4):
+            if t0 & (0x10 << i):
+                b = self._rx_byte()
+                self._atr.append(ord(b))
+                self._dbg_print("T%si = %x" % (chr(ord('A')+i), ord(b)))
 
-		for i in range(0, t0 & 0xf):
-			b = self._rx_byte()
-			self._atr.append(ord(b))
-			self._dbg_print("Historical = %x" % ord(b))
+        for i in range(0, t0 & 0xf):
+            b = self._rx_byte()
+            self._atr.append(ord(b))
+            self._dbg_print("Historical = %x" % ord(b))
 
-		while True:
-			x = self._rx_byte()
-			if not x:
-				break
-			self._atr.append(ord(x))
-			self._dbg_print("Extra: %x" % ord(x))
+        while True:
+            x = self._rx_byte()
+            if not x:
+                break
+            self._atr.append(ord(x))
+            self._dbg_print("Extra: %x" % ord(x))
 
-		return 1
+        return 1
 
-	def _dbg_print(self, s):
-		if self._debug:
-			print(s)
+    def _dbg_print(self, s):
+        if self._debug:
+            print(s)
 
-	def _tx_byte(self, b):
-		self._sl.write(b)
-		r = self._sl.read()
-		if r != b:	# TX and RX are tied, so we must clear the echo
-			raise ProtocolError("Bad echo value. Expected %02x, got %s)" % (ord(b), '%02x'%ord(r) if r else '(nil)'))
+    def _tx_byte(self, b):
+        self._sl.write(b)
+        r = self._sl.read()
+        if r != b:  # TX and RX are tied, so we must clear the echo
+            raise ProtocolError("Bad echo value. Expected %02x, got %s)" % (
+                ord(b), '%02x' % ord(r) if r else '(nil)'))
 
-	def _tx_string(self, s):
-		"""This is only safe if it's guaranteed the card won't send any data
-		during the time of tx of the string !!!"""
-		self._sl.write(s)
-		r = self._sl.read(len(s))
-		if r != s:	# TX and RX are tied, so we must clear the echo
-			raise ProtocolError("Bad echo value (Expected: %s, got %s)" % (b2h(s), b2h(r)))
+    def _tx_string(self, s):
+        """This is only safe if it's guaranteed the card won't send any data
+        during the time of tx of the string !!!"""
+        self._sl.write(s)
+        r = self._sl.read(len(s))
+        if r != s:  # TX and RX are tied, so we must clear the echo
+            raise ProtocolError(
+                "Bad echo value (Expected: %s, got %s)" % (b2h(s), b2h(r)))
 
-	def _rx_byte(self):
-		return self._sl.read()
+    def _rx_byte(self):
+        return self._sl.read()
 
-	def send_apdu_raw(self, pdu):
-		"""see LinkBase.send_apdu_raw"""
+    def _send_apdu_raw(self, pdu):
 
-		pdu = h2b(pdu)
-		data_len = ord(pdu[4])	# P3
+        pdu = h2b(pdu)
+        data_len = pdu[4]  # P3
 
-		# Send first CLASS,INS,P1,P2,P3
-		self._tx_string(pdu[0:5])
+        # Send first CLASS,INS,P1,P2,P3
+        self._tx_string(pdu[0:5])
 
-		# Wait ack which can be
-		#  - INS: Command acked -> go ahead
-		#  - 0x60: NULL, just wait some more
-		#  - SW1: The card can apparently proceed ...
-		while True:
-			b = self._rx_byte()
-			if b == pdu[1]:
-				break
-			elif b != '\x60':
-				# Ok, it 'could' be SW1
-				sw1 = b
-				sw2 = self._rx_byte()
-				nil = self._rx_byte()
-				if (sw2 and not nil):
-					return '', b2h(sw1+sw2)
+        # Wait ack which can be
+        #  - INS: Command acked -> go ahead
+        #  - 0x60: NULL, just wait some more
+        #  - SW1: The card can apparently proceed ...
+        while True:
+            b = self._rx_byte()
+            if ord(b) == pdu[1]:
+                break
+            elif b != '\x60':
+                # Ok, it 'could' be SW1
+                sw1 = b
+                sw2 = self._rx_byte()
+                nil = self._rx_byte()
+                if (sw2 and not nil):
+                    return '', b2h(sw1+sw2)
 
-				raise ProtocolError()
+                raise ProtocolError()
 
-		# Send data (if any)
-		if len(pdu) > 5:
-			self._tx_string(pdu[5:])
+        # Send data (if any)
+        if len(pdu) > 5:
+            self._tx_string(pdu[5:])
 
-		# Receive data (including SW !)
-		#  length = [P3 - tx_data (=len(pdu)-len(hdr)) + 2 (SW1//2) ]
-		to_recv = data_len - len(pdu) + 5 + 2
+        # Receive data (including SW !)
+        #  length = [P3 - tx_data (=len(pdu)-len(hdr)) + 2 (SW1//2) ]
+        to_recv = data_len - len(pdu) + 5 + 2
 
-		data = ''
-		while (len(data) < to_recv):
-			b = self._rx_byte()
-			if (to_recv == 2) and (b == '\x60'): # Ignore NIL if we have no RX data (hack ?)
-				continue
-			if not b:
-				break
-			data += b
+        data = bytes(0)
+        while (len(data) < to_recv):
+            b = self._rx_byte()
+            if (to_recv == 2) and (b == '\x60'):  # Ignore NIL if we have no RX data (hack ?)
+                continue
+            if not b:
+                break
+            data += b
 
-		# Split datafield from SW
-		if len(data) < 2:
-			return None, None
-		sw = data[-2:]
-		data = data[0:-2]
+        # Split datafield from SW
+        if len(data) < 2:
+            return None, None
+        sw = data[-2:]
+        data = data[0:-2]
 
-		# Return value
-		return b2h(data), b2h(sw)
+        # Return value
+        return b2h(data), b2h(sw)

pySim/ts_102_221.py

@@ -0,0 +1,799 @@
+# coding=utf-8
+"""Utilities / Functions related to ETSI TS 102 221, the core UICC spec.
+
+(C) 2021 by Harald Welte <laforge@osmocom.org>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+from pytlv.TLV import *
+from construct import *
+from pySim.construct import *
+from pySim.utils import *
+from pySim.filesystem import *
+from pySim.tlv import *
+from bidict import bidict
+from pySim.profile import CardProfile
+from pySim.profile import match_uicc
+from pySim.profile import match_sim
+import pySim.iso7816_4 as iso7816_4
+
+# A UICC will usually also support 2G functionality. If this is the case, we
+# need to add DF_GSM and DF_TELECOM along with the UICC related files
+from pySim.ts_51_011 import DF_GSM, DF_TELECOM
+
+ts_102_22x_cmdset = CardCommandSet('TS 102 22x', [
+    # TS 102 221 Section 10.1.2 Table 10.5 "Coding of Instruction Byte"
+    CardCommand('SELECT',                   0xA4, ['0X', '4X', '6X']),
+    CardCommand('STATUS',                   0xF2, ['8X', 'CX', 'EX']),
+    CardCommand('READ BINARY',              0xB0, ['0X', '4X', '6X']),
+    CardCommand('UPDATE BINARY',            0xD6, ['0X', '4X', '6X']),
+    CardCommand('READ RECORD',              0xB2, ['0X', '4X', '6X']),
+    CardCommand('UPDATE RECORD',            0xDC, ['0X', '4X', '6X']),
+    CardCommand('SEARCH RECORD',            0xA2, ['0X', '4X', '6X']),
+    CardCommand('INCREASE',                 0x32, ['8X', 'CX', 'EX']),
+    CardCommand('RETRIEVE DATA',            0xCB, ['8X', 'CX', 'EX']),
+    CardCommand('SET DATA',                 0xDB, ['8X', 'CX', 'EX']),
+    CardCommand('VERIFY PIN',               0x20, ['0X', '4X', '6X']),
+    CardCommand('CHANGE PIN',               0x24, ['0X', '4X', '6X']),
+    CardCommand('DISABLE PIN',              0x26, ['0X', '4X', '6X']),
+    CardCommand('ENABLE PIN',               0x28, ['0X', '4X', '6X']),
+    CardCommand('UNBLOCK PIN',              0x2C, ['0X', '4X', '6X']),
+    CardCommand('DEACTIVATE FILE',          0x04, ['0X', '4X', '6X']),
+    CardCommand('ACTIVATE FILE',            0x44, ['0X', '4X', '6X']),
+    CardCommand('AUTHENTICATE',             0x88, ['0X', '4X', '6X']),
+    CardCommand('AUTHENTICATE',             0x89, ['0X', '4X', '6X']),
+    CardCommand('GET CHALLENGE',            0x84, ['0X', '4X', '6X']),
+    CardCommand('TERMINAL CAPABILITY',      0xAA, ['8X', 'CX', 'EX']),
+    CardCommand('TERMINAL PROFILE',         0x10, ['80']),
+    CardCommand('ENVELOPE',                 0xC2, ['80']),
+    CardCommand('FETCH',                    0x12, ['80']),
+    CardCommand('TERMINAL RESPONSE',        0x14, ['80']),
+    CardCommand('MANAGE CHANNEL',           0x70, ['0X', '4X', '6X']),
+    CardCommand('MANAGE SECURE CHANNEL',    0x73, ['0X', '4X', '6X']),
+    CardCommand('TRANSACT DATA',            0x75, ['0X', '4X', '6X']),
+    CardCommand('SUSPEND UICC',             0x76, ['80']),
+    CardCommand('GET IDENTITY',             0x78, ['8X', 'CX', 'EX']),
+    CardCommand('EXCHANGE CAPABILITIES',    0x7A, ['80']),
+    CardCommand('GET RESPONSE',             0xC0, ['0X', '4X', '6X']),
+    # TS 102 222 Section 6.1 Table 1 "Coding of the commands"
+    CardCommand('CREATE FILE',              0xE0, ['0X', '4X']),
+    CardCommand('DELETE FILE',              0xE4, ['0X', '4X']),
+    CardCommand('DEACTIVATE FILE',          0x04, ['0X', '4X']),
+    CardCommand('ACTIVATE FILE',            0x44, ['0X', '4X']),
+    CardCommand('TERMINATE DF',             0xE6, ['0X', '4X']),
+    CardCommand('TERMINATE EF',             0xE8, ['0X', '4X']),
+    CardCommand('TERMINATE CARD USAGE',     0xFE, ['0X', '4X']),
+    CardCommand('RESIZE FILE',              0xD4, ['8X', 'CX']),
+])
+
+
+FCP_TLV_MAP = {
+    '82': 'file_descriptor',
+    '83': 'file_identifier',
+    '84': 'df_name',
+    'A5': 'proprietary_info',
+    '8A': 'life_cycle_status_int',
+    '8B': 'security_attrib_ref_expanded',
+    '8C': 'security_attrib_compact',
+    'AB': 'security_attrib_espanded',
+    'C6': 'pin_status_template_do',
+    '80': 'file_size',
+    '81': 'total_file_size',
+    '88': 'short_file_id',
+}
+
+# ETSI TS 102 221 11.1.1.4.6
+FCP_Proprietary_TLV_MAP = {
+    '80': 'uicc_characteristics',
+    '81': 'application_power_consumption',
+    '82': 'minimum_app_clock_freq',
+    '83': 'available_memory',
+    '84': 'file_details',
+    '85': 'reserved_file_size',
+    '86': 'maximum_file_size',
+    '87': 'suported_system_commands',
+    '88': 'specific_uicc_env_cond',
+    '89': 'p2p_cat_secured_apdu',
+    # Additional private TLV objects (bits b7 and b8 of the first byte of the tag set to '1')
+}
+
+# ETSI TS 102 221 11.1.1.4.3
+
+
+def interpret_file_descriptor(in_hex):
+    in_bin = h2b(in_hex)
+    out = {}
+    ft_dict = {
+        0: 'working_ef',
+        1: 'internal_ef',
+        7: 'df'
+    }
+    fs_dict = {
+        0: 'no_info_given',
+        1: 'transparent',
+        2: 'linear_fixed',
+        6: 'cyclic',
+        0x39: 'ber_tlv',
+    }
+    fdb = in_bin[0]
+    ftype = (fdb >> 3) & 7
+    if fdb & 0xbf == 0x39:
+        fstruct = 0x39
+    else:
+        fstruct = fdb & 7
+    out['shareable'] = True if fdb & 0x40 else False
+    out['file_type'] = ft_dict[ftype] if ftype in ft_dict else ftype
+    out['structure'] = fs_dict[fstruct] if fstruct in fs_dict else fstruct
+    if len(in_bin) >= 5:
+        out['record_len'] = int.from_bytes(in_bin[2:4], 'big')
+        out['num_of_rec'] = int.from_bytes(in_bin[4:5], 'big')
+    return out
+
+# ETSI TS 102 221 11.1.1.4.9
+
+
+def interpret_life_cycle_sts_int(in_hex):
+    lcsi = int(in_hex, 16)
+    if lcsi == 0x00:
+        return 'no_information'
+    elif lcsi == 0x01:
+        return 'creation'
+    elif lcsi == 0x03:
+        return 'initialization'
+    elif lcsi & 0x05 == 0x05:
+        return 'operational_activated'
+    elif lcsi & 0x05 == 0x04:
+        return 'operational_deactivated'
+    elif lcsi & 0xc0 == 0xc0:
+        return 'termination'
+    else:
+        return in_hex
+
+
+# ETSI TS 102 221 11.1.1.4.10
+FCP_Pin_Status_TLV_MAP = {
+    '90': 'ps_do',
+    '95': 'usage_qualifier',
+    '83': 'key_reference',
+}
+
+
+def interpret_ps_templ_do(in_hex):
+    # cannot use the 'TLV' parser due to repeating tags
+    #psdo_tlv = TLV(FCP_Pin_Status_TLV_MAP)
+    # return psdo_tlv.parse(in_hex)
+    return in_hex
+
+
+# 'interpreter' functions for each tag
+FCP_interpreter_map = {
+    '80': lambda x: int(x, 16),
+    '82': interpret_file_descriptor,
+    '8A': interpret_life_cycle_sts_int,
+    'C6': interpret_ps_templ_do,
+}
+
+FCP_prorietary_interpreter_map = {
+    '83': lambda x: int(x, 16),
+}
+
+# pytlv unfortunately doesn't have a setting using which we can make it
+# accept unknown tags.  It also doesn't raise a specific exception type but
+# just the generic ValueError, so we cannot ignore those either.  Instead,
+# we insert a dict entry for every possible proprietary tag permitted
+
+
+def fixup_fcp_proprietary_tlv_map(tlv_map):
+    if 'D0' in tlv_map:
+        return
+    for i in range(0xc0, 0xff):
+        i_hex = i2h([i]).upper()
+        tlv_map[i_hex] = 'proprietary_' + i_hex
+    # Other non-standard TLV objects found on some cards
+    tlv_map['9B'] = 'target_ef'  # for sysmoUSIM-SJS1
+
+
+def tlv_key_replace(inmap, indata):
+    def newkey(inmap, key):
+        if key in inmap:
+            return inmap[key]
+        else:
+            return key
+    return {newkey(inmap, d[0]): d[1] for d in indata.items()}
+
+
+def tlv_val_interpret(inmap, indata):
+    def newval(inmap, key, val):
+        if key in inmap:
+            return inmap[key](val)
+        else:
+            return val
+    return {d[0]: newval(inmap, d[0], d[1]) for d in indata.items()}
+
+# ETSI TS 102 221 Section 9.2.7 + ISO7816-4 9.3.3/9.3.4
+
+
+class _AM_DO_DF(DataObject):
+    def __init__(self):
+        super().__init__('access_mode', 'Access Mode', tag=0x80)
+
+    def from_bytes(self, do: bytes):
+        res = []
+        if len(do) != 1:
+            raise ValueError("We only support single-byte AMF inside AM-DO")
+        amf = do[0]
+        # tables 17..29 and 41..44 of 7816-4
+        if amf & 0x80 == 0:
+            if amf & 0x40:
+                res.append('delete_file')
+            if amf & 0x20:
+                res.append('terminate_df')
+            if amf & 0x10:
+                res.append('activate_file')
+            if amf & 0x08:
+                res.append('deactivate_file')
+        if amf & 0x04:
+            res.append('create_file_df')
+        if amf & 0x02:
+            res.append('create_file_ef')
+        if amf & 0x01:
+            res.append('delete_file_child')
+        self.decoded = res
+
+    def to_bytes(self):
+        val = 0
+        if 'delete_file' in self.decoded:
+            val |= 0x40
+        if 'terminate_df' in self.decoded:
+            val |= 0x20
+        if 'activate_file' in self.decoded:
+            val |= 0x10
+        if 'deactivate_file' in self.decoded:
+            val |= 0x08
+        if 'create_file_df' in self.decoded:
+            val |= 0x04
+        if 'create_file_ef' in self.decoded:
+            val |= 0x02
+        if 'delete_file_child' in self.decoded:
+            val |= 0x01
+        return val.to_bytes(1, 'big')
+
+
+class _AM_DO_EF(DataObject):
+    """ISO7816-4 9.3.2 Table 18 + 9.3.3.1 Table 31"""
+
+    def __init__(self):
+        super().__init__('access_mode', 'Access Mode', tag=0x80)
+
+    def from_bytes(self, do: bytes):
+        res = []
+        if len(do) != 1:
+            raise ValueError("We only support single-byte AMF inside AM-DO")
+        amf = do[0]
+        # tables 17..29 and 41..44 of 7816-4
+        if amf & 0x80 == 0:
+            if amf & 0x40:
+                res.append('delete_file')
+            if amf & 0x20:
+                res.append('terminate_ef')
+            if amf & 0x10:
+                res.append('activate_file_or_record')
+            if amf & 0x08:
+                res.append('deactivate_file_or_record')
+        if amf & 0x04:
+            res.append('write_append')
+        if amf & 0x02:
+            res.append('update_erase')
+        if amf & 0x01:
+            res.append('read_search_compare')
+        self.decoded = res
+
+    def to_bytes(self):
+        val = 0
+        if 'delete_file' in self.decoded:
+            val |= 0x40
+        if 'terminate_ef' in self.decoded:
+            val |= 0x20
+        if 'activate_file_or_record' in self.decoded:
+            val |= 0x10
+        if 'deactivate_file_or_record' in self.decoded:
+            val |= 0x08
+        if 'write_append' in self.decoded:
+            val |= 0x04
+        if 'update_erase' in self.decoded:
+            val |= 0x02
+        if 'read_search_compare' in self.decoded:
+            val |= 0x01
+        return val.to_bytes(1, 'big')
+
+
+class _AM_DO_CHDR(DataObject):
+    """Command Header Access Mode DO according to ISO 7816-4 Table 32."""
+
+    def __init__(self, tag):
+        super().__init__('command_header', 'Command Header Description', tag=tag)
+
+    def from_bytes(self, do: bytes):
+        res = {}
+        i = 0
+        if self.tag & 0x08:
+            res['CLA'] = do[i]
+            i += 1
+        if self.tag & 0x04:
+            res['INS'] = do[i]
+            i += 1
+        if self.tag & 0x02:
+            res['P1'] = do[i]
+            i += 1
+        if self.tag & 0x01:
+            res['P2'] = do[i]
+            i += 1
+        self.decoded = res
+
+    def _compute_tag(self):
+        """Override to encode the tag, as it depends on the value."""
+        tag = 0x80
+        if 'CLA' in self.decoded:
+            tag |= 0x08
+        if 'INS' in self.decoded:
+            tag |= 0x04
+        if 'P1' in self.decoded:
+            tag |= 0x02
+        if 'P2' in self.decoded:
+            tag |= 0x01
+        return tag
+
+    def to_bytes(self):
+        res = bytearray()
+        if 'CLA' in self.decoded:
+            res.append(self.decoded['CLA'])
+        if 'INS' in self.decoded:
+            res.append(self.decoded['INS'])
+        if 'P1' in self.decoded:
+            res.append(self.decoded['P1'])
+        if 'P2' in self.decoded:
+            res.append(self.decoded['P2'])
+        return res
+
+
+AM_DO_CHDR = DataObjectChoice('am_do_chdr', members=[
+    _AM_DO_CHDR(0x81), _AM_DO_CHDR(0x82), _AM_DO_CHDR(0x83), _AM_DO_CHDR(0x84),
+    _AM_DO_CHDR(0x85), _AM_DO_CHDR(0x86), _AM_DO_CHDR(0x87), _AM_DO_CHDR(0x88),
+    _AM_DO_CHDR(0x89), _AM_DO_CHDR(0x8a), _AM_DO_CHDR(0x8b), _AM_DO_CHDR(0x8c),
+    _AM_DO_CHDR(0x8d), _AM_DO_CHDR(0x8e), _AM_DO_CHDR(0x8f)])
+
+AM_DO_DF = AM_DO_CHDR | _AM_DO_DF()
+AM_DO_EF = AM_DO_CHDR | _AM_DO_EF()
+
+
+# TS 102 221 Section 9.5.1 / Table 9.3
+pin_names = bidict({
+    0x01: 'PIN1',
+    0x02: 'PIN2',
+    0x03: 'PIN3',
+    0x04: 'PIN4',
+    0x05: 'PIN5',
+    0x06: 'PIN6',
+    0x07: 'PIN7',
+    0x08: 'PIN8',
+    0x0a: 'ADM1',
+    0x0b: 'ADM2',
+    0x0c: 'ADM3',
+    0x0d: 'ADM4',
+    0x0e: 'ADM5',
+
+    0x11: 'UNIVERSAL_PIN',
+    0x81: '2PIN1',
+    0x82: '2PIN2',
+    0x83: '2PIN3',
+    0x84: '2PIN4',
+    0x85: '2PIN5',
+    0x86: '2PIN6',
+    0x87: '2PIN7',
+    0x88: '2PIN8',
+    0x8a: 'ADM6',
+    0x8b: 'ADM7',
+    0x8c: 'ADM8',
+    0x8d: 'ADM9',
+    0x8e: 'ADM10',
+})
+
+
+class CRT_DO(DataObject):
+    """Control Reference Template as per TS 102 221 9.5.1"""
+
+    def __init__(self):
+        super().__init__('control_reference_template',
+                         'Control Reference Template', tag=0xA4)
+
+    def from_bytes(self, do: bytes):
+        """Decode a Control Reference Template DO."""
+        if len(do) != 6:
+            raise ValueError('Unsupported CRT DO length: %s', do)
+        if do[0] != 0x83 or do[1] != 0x01:
+            raise ValueError('Unsupported Key Ref Tag or Len in CRT DO %s', do)
+        if do[3:] != b'\x95\x01\x08':
+            raise ValueError(
+                'Unsupported Usage Qualifier Tag or Len in CRT DO %s', do)
+        self.encoded = do[0:6]
+        self.decoded = pin_names[do[2]]
+        return do[6:]
+
+    def to_bytes(self):
+        pin = pin_names.inverse[self.decoded]
+        return b'\x83\x01' + pin.to_bytes(1, 'big') + b'\x95\x01\x08'
+
+# ISO7816-4 9.3.3 Table 33
+
+
+class SecCondByte_DO(DataObject):
+    def __init__(self, tag=0x9d):
+        super().__init__('security_condition_byte', tag=tag)
+
+    def from_bytes(self, binary: bytes):
+        if len(binary) != 1:
+            raise ValueError
+        inb = binary[0]
+        if inb == 0:
+            cond = 'always'
+        if inb == 0xff:
+            cond = 'never'
+        res = []
+        if inb & 0x80:
+            cond = 'and'
+        else:
+            cond = 'or'
+        if inb & 0x40:
+            res.append('secure_messaging')
+        if inb & 0x20:
+            res.append('external_auth')
+        if inb & 0x10:
+            res.append('user_auth')
+        rd = {'mode': cond}
+        if len(res):
+            rd['conditions'] = res
+        self.decoded = rd
+
+    def to_bytes(self):
+        mode = self.decoded['mode']
+        if mode == 'always':
+            res = 0
+        elif mode == 'never':
+            res = 0xff
+        else:
+            res = 0
+            if mode == 'and':
+                res |= 0x80
+            elif mode == 'or':
+                pass
+            else:
+                raise ValueError('Unknown mode %s' % mode)
+            for c in self.decoded['conditions']:
+                if c == 'secure_messaging':
+                    res |= 0x40
+                elif c == 'external_auth':
+                    res |= 0x20
+                elif c == 'user_auth':
+                    res |= 0x10
+                else:
+                    raise ValueError('Unknown condition %s' % c)
+        return res.to_bytes(1, 'big')
+
+
+Always_DO = TL0_DataObject('always', 'Always', 0x90)
+Never_DO = TL0_DataObject('never', 'Never', 0x97)
+
+
+class Nested_DO(DataObject):
+    """A DO that nests another DO/Choice/Sequence"""
+
+    def __init__(self, name, tag, choice):
+        super().__init__(name, tag=tag)
+        self.children = choice
+
+    def from_bytes(self, binary: bytes) -> list:
+        remainder = binary
+        self.decoded = []
+        while remainder:
+            rc, remainder = self.children.decode(remainder)
+            self.decoded.append(rc)
+        return self.decoded
+
+    def to_bytes(self) -> bytes:
+        encoded = [self.children.encode(d) for d in self.decoded]
+        return b''.join(encoded)
+
+
+OR_Template = DataObjectChoice('or_template', 'OR-Template',
+                               members=[Always_DO, Never_DO, SecCondByte_DO(), SecCondByte_DO(0x9e), CRT_DO()])
+OR_DO = Nested_DO('or', 0xa0, OR_Template)
+AND_Template = DataObjectChoice('and_template', 'AND-Template',
+                                members=[Always_DO, Never_DO, SecCondByte_DO(), SecCondByte_DO(0x9e), CRT_DO()])
+AND_DO = Nested_DO('and', 0xa7, AND_Template)
+NOT_Template = DataObjectChoice('not_template', 'NOT-Template',
+                                members=[Always_DO, Never_DO, SecCondByte_DO(), SecCondByte_DO(0x9e), CRT_DO()])
+NOT_DO = Nested_DO('not', 0xaf, NOT_Template)
+SC_DO = DataObjectChoice('security_condition', 'Security Condition',
+                         members=[Always_DO, Never_DO, SecCondByte_DO(), SecCondByte_DO(0x9e), CRT_DO(),
+                                  OR_DO, AND_DO, NOT_DO])
+
+# TS 102 221 Section 13.1
+
+
+class EF_DIR(LinFixedEF):
+    class ApplicationLabel(BER_TLV_IE, tag=0x50):
+        # TODO: UCS-2 coding option as per Annex A of TS 102 221
+        _construct = GreedyString('ascii')
+
+    # see https://github.com/PyCQA/pylint/issues/5794
+    #pylint: disable=undefined-variable
+    class ApplicationTemplate(BER_TLV_IE, tag=0x61,
+                              nested=[iso7816_4.ApplicationId, ApplicationLabel, iso7816_4.FileReference,
+                                      iso7816_4.CommandApdu, iso7816_4.DiscretionaryData,
+                                      iso7816_4.DiscretionaryTemplate, iso7816_4.URL,
+                                      iso7816_4.ApplicationRelatedDOSet]):
+        pass
+
+    def __init__(self, fid='2f00', sfid=0x1e, name='EF.DIR', desc='Application Directory'):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, rec_len={5, 54})
+        self._tlv = EF_DIR.ApplicationTemplate
+
+# TS 102 221 Section 13.2
+
+
+class EF_ICCID(TransparentEF):
+    def __init__(self, fid='2fe2', sfid=0x02, name='EF.ICCID', desc='ICC Identification'):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size={10, 10})
+
+    def _decode_hex(self, raw_hex):
+        return {'iccid': dec_iccid(raw_hex)}
+
+    def _encode_hex(self, abstract):
+        return enc_iccid(abstract['iccid'])
+
+# TS 102 221 Section 13.3
+
+
+class EF_PL(TransRecEF):
+    def __init__(self, fid='2f05', sfid=0x05, name='EF.PL', desc='Preferred Languages'):
+        super().__init__(fid, sfid=sfid, name=name,
+                         desc=desc, rec_len=2, size={2, None})
+
+    def _decode_record_bin(self, bin_data):
+        if bin_data == b'\xff\xff':
+            return None
+        else:
+            return bin_data.decode('ascii')
+
+    def _encode_record_bin(self, in_json):
+        if in_json is None:
+            return b'\xff\xff'
+        else:
+            return in_json.encode('ascii')
+
+
+# TS 102 221 Section 13.4
+class EF_ARR(LinFixedEF):
+    def __init__(self, fid='2f06', sfid=0x06, name='EF.ARR', desc='Access Rule Reference'):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc)
+        # add those commands to the general commands of a TransparentEF
+        self.shell_commands += [self.AddlShellCommands()]
+
+    @staticmethod
+    def flatten(inp: list):
+        """Flatten the somewhat deep/complex/nested data returned from decoder."""
+        def sc_abbreviate(sc):
+            if 'always' in sc:
+                return 'always'
+            elif 'never' in sc:
+                return 'never'
+            elif 'control_reference_template' in sc:
+                return sc['control_reference_template']
+            else:
+                return sc
+
+        by_mode = {}
+        for t in inp:
+            am = t[0]
+            sc = t[1]
+            sc_abbr = sc_abbreviate(sc)
+            if 'access_mode' in am:
+                for m in am['access_mode']:
+                    by_mode[m] = sc_abbr
+            elif 'command_header' in am:
+                ins = am['command_header']['INS']
+                if 'CLA' in am['command_header']:
+                    cla = am['command_header']['CLA']
+                else:
+                    cla = None
+                cmd = ts_102_22x_cmdset.lookup(ins, cla)
+                if cmd:
+                    name = cmd.name.lower().replace(' ', '_')
+                    by_mode[name] = sc_abbr
+                else:
+                    raise ValueError
+            else:
+                raise ValueError
+        return by_mode
+
+    def _decode_record_bin(self, raw_bin_data):
+        # we can only guess if we should decode for EF or DF here :(
+        arr_seq = DataObjectSequence('arr', sequence=[AM_DO_EF, SC_DO])
+        dec = arr_seq.decode_multi(raw_bin_data)
+        # we cannot pass the result through flatten() here, as we don't have a related
+        # 'un-flattening' decoder, and hence would be unable to encode :(
+        return dec[0]
+
+    @with_default_category('File-Specific Commands')
+    class AddlShellCommands(CommandSet):
+        def __init__(self):
+            super().__init__()
+
+        @cmd2.with_argparser(LinFixedEF.ShellCommands.read_rec_dec_parser)
+        def do_read_arr_record(self, opts):
+            """Read one EF.ARR record in flattened, human-friendly form."""
+            (data, sw) = self._cmd.rs.read_record_dec(opts.record_nr)
+            data = self._cmd.rs.selected_file.flatten(data)
+            self._cmd.poutput_json(data, opts.oneline)
+
+        @cmd2.with_argparser(LinFixedEF.ShellCommands.read_recs_dec_parser)
+        def do_read_arr_records(self, opts):
+            """Read + decode all EF.ARR records in flattened, human-friendly form."""
+            num_of_rec = self._cmd.rs.selected_file_fcp['file_descriptor']['num_of_rec']
+            # collect all results in list so they are rendered as JSON list when printing
+            data_list = []
+            for recnr in range(1, 1 + num_of_rec):
+                (data, sw) = self._cmd.rs.read_record_dec(recnr)
+                data = self._cmd.rs.selected_file.flatten(data)
+                data_list.append(data)
+            self._cmd.poutput_json(data_list, opts.oneline)
+
+
+# TS 102 221 Section 13.6
+class EF_UMPC(TransparentEF):
+    def __init__(self, fid='2f08', sfid=0x08, name='EF.UMPC', desc='UICC Maximum Power Consumption'):
+        super().__init__(fid, sfid=sfid, name=name, desc=desc, size={5, 5})
+        addl_info = FlagsEnum(Byte, req_inc_idle_current=1,
+                              support_uicc_suspend=2)
+        self._construct = Struct(
+            'max_current_mA'/Int8ub, 't_op_s'/Int8ub, 'addl_info'/addl_info)
+
+
+class CardProfileUICC(CardProfile):
+
+    ORDER = 1
+
+    def __init__(self, name='UICC'):
+        files = [
+            EF_DIR(),
+            EF_ICCID(),
+            EF_PL(),
+            EF_ARR(),
+            # FIXME: DF.CD
+            EF_UMPC(),
+        ]
+        sw = {
+            'Normal': {
+                '9000': 'Normal ending of the command',
+                '91xx': 'Normal ending of the command, with extra information from the proactive UICC containing a command for the terminal',
+                '92xx': 'Normal ending of the command, with extra information concerning an ongoing data transfer session',
+            },
+            'Postponed processing': {
+                '9300': 'SIM Application Toolkit is busy. Command cannot be executed at present, further normal commands are allowed',
+            },
+            'Warnings': {
+                '6200': 'No information given, state of non-volatile memory unchanged',
+                '6281': 'Part of returned data may be corrupted',
+                '6282': 'End of file/record reached before reading Le bytes or unsuccessful search',
+                '6283': 'Selected file invalidated',
+                '6284': 'Selected file in termination state',
+                '62f1': 'More data available',
+                '62f2': 'More data available and proactive command pending',
+                '62f3': 'Response data available',
+                '63f1': 'More data expected',
+                '63f2': 'More data expected and proactive command pending',
+                '63cx': 'Command successful but after using an internal update retry routine X times',
+            },
+            'Execution errors': {
+                '6400': 'No information given, state of non-volatile memory unchanged',
+                '6500': 'No information given, state of non-volatile memory changed',
+                '6581': 'Memory problem',
+            },
+            'Checking errors': {
+                '6700': 'Wrong length',
+                '67xx': 'The interpretation of this status word is command dependent',
+                '6b00': 'Wrong parameter(s) P1-P2',
+                '6d00': 'Instruction code not supported or invalid',
+                '6e00': 'Class not supported',
+                '6f00': 'Technical problem, no precise diagnosis',
+                '6fxx': 'The interpretation of this status word is command dependent',
+            },
+            'Functions in CLA not supported': {
+                '6800': 'No information given',
+                '6881': 'Logical channel not supported',
+                '6882': 'Secure messaging not supported',
+            },
+            'Command not allowed': {
+                '6900': 'No information given',
+                '6981': 'Command incompatible with file structure',
+                '6982': 'Security status not satisfied',
+                '6983': 'Authentication/PIN method blocked',
+                '6984': 'Referenced data invalidated',
+                '6985': 'Conditions of use not satisfied',
+                '6986': 'Command not allowed (no EF selected)',
+                '6989': 'Command not allowed - secure channel - security not satisfied',
+            },
+            'Wrong parameters': {
+                '6a80': 'Incorrect parameters in the data field',
+                '6a81': 'Function not supported',
+                '6a82': 'File not found',
+                '6a83': 'Record not found',
+                '6a84': 'Not enough memory space',
+                '6a86': 'Incorrect parameters P1 to P2',
+                '6a87': 'Lc inconsistent with P1 to P2',
+                '6a88': 'Referenced data not found',
+            },
+            'Application errors': {
+                '9850': 'INCREASE cannot be performed, max value reached',
+                '9862': 'Authentication error, application specific',
+                '9863': 'Security session or association expired',
+                '9864': 'Minimum UICC suspension time is too long',
+            },
+        }
+
+        super().__init__(name, desc='ETSI TS 102 221', cla="00",
+                         sel_ctrl="0004", files_in_mf=files, sw=sw)
+
+    @staticmethod
+    def decode_select_response(resp_hex: str) -> object:
+        """ETSI TS 102 221 Section 11.1.1.3"""
+        fixup_fcp_proprietary_tlv_map(FCP_Proprietary_TLV_MAP)
+        resp_hex = resp_hex.upper()
+        # outer layer
+        fcp_base_tlv = TLV(['62'])
+        fcp_base = fcp_base_tlv.parse(resp_hex)
+        # actual FCP
+        fcp_tlv = TLV(FCP_TLV_MAP)
+        fcp = fcp_tlv.parse(fcp_base['62'])
+        # further decode the proprietary information
+        if 'A5' in fcp:
+            prop_tlv = TLV(FCP_Proprietary_TLV_MAP)
+            prop = prop_tlv.parse(fcp['A5'])
+            fcp['A5'] = tlv_val_interpret(FCP_prorietary_interpreter_map, prop)
+            fcp['A5'] = tlv_key_replace(FCP_Proprietary_TLV_MAP, fcp['A5'])
+        # finally make sure we get human-readable keys in the output dict
+        r = tlv_val_interpret(FCP_interpreter_map, fcp)
+        return tlv_key_replace(FCP_TLV_MAP, r)
+
+    @staticmethod
+    def match_with_card(scc: SimCardCommands) -> bool:
+        return match_uicc(scc)
+
+
+class CardProfileUICCSIM(CardProfileUICC):
+    """Same as above, but including 2G SIM support"""
+
+    ORDER = 0
+
+    def __init__(self):
+        super().__init__('UICC-SIM')
+
+        # Add GSM specific files
+        self.files_in_mf.append(DF_TELECOM())
+        self.files_in_mf.append(DF_GSM())
+
+    @staticmethod
+    def match_with_card(scc: SimCardCommands) -> bool:
+        return match_uicc(scc) and match_sim(scc)

pySim/ts_31_103.py

@@ -1,12 +1,12 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 """
-Various constants from ETSI TS 131 103 V14.2.0
+Various constants from 3GPP TS 31.103 V16.1.0
 """
 
 #
 # Copyright (C) 2020 Supreeth Herle <herlesupreeth@gmail.com>
+# Copyright (C) 2021 Harald Welte <laforge@osmocom.org>
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -22,48 +22,221 @@ Various constants from ETSI TS 131 103 V14.2.0
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+from pySim.filesystem import *
+from pySim.utils import *
+from pySim.tlv import *
+from pySim.ts_51_011 import EF_AD, EF_SMS, EF_SMSS, EF_SMSR, EF_SMSP
+from pySim.ts_31_102 import ADF_USIM, EF_FromPreferred, EF_UServiceTable
+import pySim.ts_102_221
+from pySim.ts_102_221 import EF_ARR
+
 # Mapping between ISIM Service Number and its description
 EF_IST_map = {
-	1: 'P-CSCF address',
-	2: 'Generic Bootstrapping Architecture (GBA)',
-	3: 'HTTP Digest',
-	4: 'GBA-based Local Key Establishment Mechanism',
-	5: 'Support of P-CSCF discovery for IMS Local Break Out',
-	6: 'Short Message Storage (SMS)',
-	7: 'Short Message Status Reports (SMSR)',
-	8: 'Support for SM-over-IP including data download via SMS-PP as defined in TS 31.111 [31]',
-	9: 'Communication Control for IMS by ISIM',
-	10: 'Support of UICC access to IMS',
-	11: 'URI support by UICC',
-	12: 'Media Type support',
-	13: 'IMS call disconnection cause',
-	14: 'URI support for MO SHORT MESSAGE CONTROL',
-	15: 'MCPTT',
-	16: 'URI support for SMS-PP DOWNLOAD as defined in 3GPP TS 31.111 [31]',
-	17: 'From Preferred',
-	18: 'IMS configuration data',
-	19: 'XCAP Configuration Data',
-	20: 'WebRTC URI',
+    1: 'P-CSCF address',
+    2: 'Generic Bootstrapping Architecture (GBA)',
+    3: 'HTTP Digest',
+    4: 'GBA-based Local Key Establishment Mechanism',
+    5: 'Support of P-CSCF discovery for IMS Local Break Out',
+    6: 'Short Message Storage (SMS)',
+    7: 'Short Message Status Reports (SMSR)',
+    8: 'Support for SM-over-IP including data download via SMS-PP as defined in TS 31.111 [31]',
+    9: 'Communication Control for IMS by ISIM',
+    10: 'Support of UICC access to IMS',
+    11: 'URI support by UICC',
+    12: 'Media Type support',
+    13: 'IMS call disconnection cause',
+    14: 'URI support for MO SHORT MESSAGE CONTROL',
+    15: 'MCPTT',
+    16: 'URI support for SMS-PP DOWNLOAD as defined in 3GPP TS 31.111 [31]',
+    17: 'From Preferred',
+    18: 'IMS configuration data',
+    19: 'XCAP Configuration Data',
+    20: 'WebRTC URI',
+    21: 'MuD and MiD configuration data',
 }
 
 EF_ISIM_ADF_map = {
-	'IST': '6F07',
-	'IMPI': '6F02',
-	'DOMAIN': '6F03',
-	'IMPU': '6F04',
-	'AD': '6FAD',
-	'ARR': '6F06',
-	'PCSCF': '6F09',
-	'GBAP': '6FD5',
-	'GBANL': '6FD7',
-	'NAFKCA': '6FDD',
-	'UICCIARI': '6FE7',
-	'SMS': '6F3C',
-	'SMSS': '6F43',
-	'SMSR': '6F47',
-	'SMSP': '6F42',
-	'FromPreferred': '6FF7',
-	'IMSConfigData': '6FF8',
-	'XCAPConfigData': '6FFC',
-	'WebRTCURI': '6FFA'
+    'IST': '6F07',
+    'IMPI': '6F02',
+    'DOMAIN': '6F03',
+    'IMPU': '6F04',
+    'AD': '6FAD',
+    'ARR': '6F06',
+    'PCSCF': '6F09',
+    'GBAP': '6FD5',
+    'GBANL': '6FD7',
+    'NAFKCA': '6FDD',
+    'UICCIARI': '6FE7',
+    'SMS': '6F3C',
+    'SMSS': '6F43',
+    'SMSR': '6F47',
+    'SMSP': '6F42',
+    'FromPreferred': '6FF7',
+    'IMSConfigData': '6FF8',
+    'XCAPConfigData': '6FFC',
+    'WebRTCURI': '6FFA'
 }
+
+# TS 31.103 Section 4.2.2
+
+
+class EF_IMPI(TransparentEF):
+    class nai(BER_TLV_IE, tag=0x80):
+        _construct = GreedyString("utf8")
+
+    def __init__(self, fid='6f02', sfid=0x02, name='EF.IMPI', desc='IMS private user identity'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+        self._tlv = EF_IMPI.nai
+
+# TS 31.103 Section 4.2.3
+
+
+class EF_DOMAIN(TransparentEF):
+    class domain(BER_TLV_IE, tag=0x80):
+        _construct = GreedyString("utf8")
+
+    def __init__(self, fid='6f05', sfid=0x05, name='EF.DOMAIN', desc='Home Network Domain Name'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+        self._tlv = EF_DOMAIN.domain
+
+# TS 31.103 Section 4.2.4
+
+
+class EF_IMPU(LinFixedEF):
+    class impu(BER_TLV_IE, tag=0x80):
+        _construct = GreedyString("utf8")
+
+    def __init__(self, fid='6f04', sfid=0x04, name='EF.IMPU', desc='IMS public user identity'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+        self._tlv = EF_IMPU.impu
+
+# TS 31.103 Section 4.2.8
+
+
+class EF_PCSCF(LinFixedEF):
+    def __init__(self, fid='6f09', sfid=None, name='EF.P-CSCF', desc='P-CSCF Address'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+    def _decode_record_hex(self, raw_hex):
+        addr, addr_type = dec_addr_tlv(raw_hex)
+        return {"addr": addr, "addr_type": addr_type}
+
+    def _encode_record_hex(self, json_in):
+        addr = json_in['addr']
+        addr_type = json_in['addr_type']
+        return enc_addr_tlv(addr, addr_type)
+
+# TS 31.103 Section 4.2.9
+
+
+class EF_GBABP(TransparentEF):
+    def __init__(self, fid='6fd5', sfid=None, name='EF.GBABP', desc='GBA Bootstrapping'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+# TS 31.103 Section 4.2.10
+
+
+class EF_GBANL(LinFixedEF):
+    def __init__(self, fid='6fd7', sfid=None, name='EF.GBANL', desc='GBA NAF List'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+# TS 31.103 Section 4.2.11
+
+
+class EF_NAFKCA(LinFixedEF):
+    def __init__(self, fid='6fdd', sfid=None, name='EF.NAFKCA', desc='NAF Key Centre Address'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+# TS 31.103 Section 4.2.16
+
+
+class EF_UICCIARI(LinFixedEF):
+    class iari(BER_TLV_IE, tag=0x80):
+        _construct = GreedyString("utf8")
+
+    def __init__(self, fid='6fe7', sfid=None, name='EF.UICCIARI', desc='UICC IARI'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+        self._tlv = EF_UICCIARI.iari
+
+# TS 31.103 Section 4.2.18
+
+
+class EF_IMSConfigData(BerTlvEF):
+    def __init__(self, fid='6ff8', sfid=None, name='EF.IMSConfigData', desc='IMS Configuration Data'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+# TS 31.103 Section 4.2.19
+
+
+class EF_XCAPConfigData(BerTlvEF):
+    def __init__(self, fid='6ffc', sfid=None, name='EF.XCAPConfigData', desc='XCAP Configuration Data'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+# TS 31.103 Section 4.2.20
+
+
+class EF_WebRTCURI(TransparentEF):
+    class uri(BER_TLV_IE, tag=0x80):
+        _construct = GreedyString("utf8")
+
+    def __init__(self, fid='6ffa', sfid=None, name='EF.WebRTCURI', desc='WebRTC URI'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+        self._tlv = EF_WebRTCURI.uri
+
+# TS 31.103 Section 4.2.21
+
+
+class EF_MuDMiDConfigData(BerTlvEF):
+    def __init__(self, fid='6ffe', sfid=None, name='EF.MuDMiDConfigData',
+                 desc='MuD and MiD Configuration Data'):
+        super().__init__(fid=fid, sfid=sfid, name=name, desc=desc)
+
+
+class ADF_ISIM(CardADF):
+    def __init__(self, aid='a0000000871004', name='ADF.ISIM', fid=None, sfid=None,
+                 desc='ISIM Application'):
+        super().__init__(aid=aid, fid=fid, sfid=sfid, name=name, desc=desc)
+
+        files = [
+            EF_IMPI(),
+            EF_DOMAIN(),
+            EF_IMPU(),
+            EF_AD(),
+            EF_ARR('6f06', 0x06),
+            EF_UServiceTable('6f07', 0x07, 'EF.IST',
+                             'ISIM Service Table', {1, None}, EF_IST_map),
+            EF_PCSCF(),
+            EF_GBABP(),
+            EF_GBANL(),
+            EF_NAFKCA(),
+            EF_SMS(),
+            EF_SMSS(),
+            EF_SMSR(),
+            EF_SMSP(),
+            EF_UICCIARI(),
+            EF_FromPreferred(),
+            EF_IMSConfigData(),
+            EF_XCAPConfigData(),
+            EF_WebRTCURI(),
+            EF_MuDMiDConfigData(),
+        ]
+        self.add_files(files)
+        # add those commands to the general commands of a TransparentEF
+        self.shell_commands += [ADF_USIM.AddlShellCommands()]
+
+    def decode_select_response(self, data_hex):
+        return pySim.ts_102_221.CardProfileUICC.decode_select_response(data_hex)
+
+
+# TS 31.103 Section 7.1
+sw_isim = {
+    'Security management': {
+        '9862': 'Authentication error, incorrect MAC',
+        '9864': 'Authentication error, security context not supported',
+    }
+}
+
+
+class CardApplicationISIM(CardApplication):
+    def __init__(self):
+        super().__init__('ISIM', adf=ADF_ISIM(), sw=sw_isim)

pySim/utils_test.py

@@ -1,76 +0,0 @@
-#!/usr/bin/pyton
-
-import unittest
-import utils 
-
-class DecTestCase(unittest.TestCase):
-
-	def testSplitHexStringToListOf5ByteEntries(self):
-		input_str = "ffffff0003ffffff0002ffffff0001"
-		expected = [
-			"ffffff0003",
-			"ffffff0002",
-			"ffffff0001",
-		]
-		self.assertEqual(utils.hexstr_to_Nbytearr(input_str, 5), expected)
-
-	def testDecMCCfromPLMN(self):
-		self.assertEqual(utils.dec_mcc_from_plmn("92f501"), 295)
-
-	def testDecMCCfromPLMN_unused(self):
-		self.assertEqual(utils.dec_mcc_from_plmn("ff0f00"), 4095)
-
-	def testDecMNCfromPLMN_twoDigitMNC(self):
-		self.assertEqual(utils.dec_mnc_from_plmn("92f501"), 10)
-
-	def testDecMNCfromPLMN_threeDigitMNC(self):
-		self.assertEqual(utils.dec_mnc_from_plmn("031263"), 361)
-
-	def testDecMNCfromPLMN_unused(self):
-		self.assertEqual(utils.dec_mnc_from_plmn("00f0ff"), 4095)
-
-	def testDecAct_noneSet(self):
-		self.assertEqual(utils.dec_act("0000"), [])
-
-	def testDecAct_onlyUtran(self):
-		self.assertEqual(utils.dec_act("8000"), ["UTRAN"])
-
-	def testDecAct_onlyEUtran(self):
-		self.assertEqual(utils.dec_act("4000"), ["E-UTRAN"])
-
-	def testDecAct_onlyGsm(self):
-		self.assertEqual(utils.dec_act("0080"), ["GSM"])
-
-	def testDecAct_onlyGsmCompact(self):
-		self.assertEqual(utils.dec_act("0040"), ["GSM COMPACT"])
-
-	def testDecAct_onlyCdma2000HRPD(self):
-		self.assertEqual(utils.dec_act("0020"), ["cdma2000 HRPD"])
-
-	def testDecAct_onlyCdma20001xRTT(self):
-		self.assertEqual(utils.dec_act("0010"), ["cdma2000 1xRTT"])
-
-	def testDecAct_allSet(self):
-		self.assertEqual(utils.dec_act("ffff"), ["UTRAN", "E-UTRAN", "GSM", "GSM COMPACT", "cdma2000 HRPD", "cdma2000 1xRTT"])
-
-	def testDecxPlmn_w_act(self):
-		expected = {'mcc': 295, 'mnc': 10, 'act': ["UTRAN"]}
-		self.assertEqual(utils.dec_xplmn_w_act("92f5018000"), expected)
-
-	def testFormatxPlmn_w_act(self):
-		input_str = "92f501800092f5508000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000"
-		expected = '''92f5018000 # MCC: 295 MNC:  10 AcT: UTRAN
-92f5508000 # MCC: 295 MNC:   5 AcT: UTRAN
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-ffffff0000 # unused
-'''
-		self.assertEqual(utils.format_xplmn_w_act(input_str), expected)
-
-if __name__ == "__main__":
-	unittest.main()

pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"

pysim-testdata/Fairwaves-SIM.ok

@@ -7,8 +7,8 @@ GID1: ffffffffffffffff
 GID2: ffffffffffffffff
 SMSP: e1ffffffffffffffffffffffff0581005155f5ffffffffffff000000ffffffffffffffffffffffffffff
 SPN: Fairwaves
-Display HPLMN: False
-Display OPLMN: False
+Show in HPLMN: False
+Hide in OPLMN: False
 PLMNsel: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
 	ffffff0000 # unused
@@ -31,7 +31,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -43,7 +43,7 @@ HPLMNAcT:
 ACC: 0008
 MSISDN: Not available
 Administrative data: 00000002
-	MS operation mode: normal operation
+	MS operation mode: normal
 	Ciphering Indicator: disabled
 SIM Service Table: ff3cc3ff030fff0f000fff03f0c0
 	Service 1 - CHV1 disable function

pysim-testdata/Wavemobile-SIM.ok

@@ -7,11 +7,11 @@ GID1: Can't read file -- SW match failed! Expected 9000 and got 6a82.
 GID2: Can't read file -- SW match failed! Expected 9000 and got 6a82.
 SMSP: e1ffffffffffffffffffffffff0581005155f5ffffffffffff000000ffffffffffffffffffffffffffff
 SPN: wavemobile
-Display HPLMN: False
-Display OPLMN: False
+Show in HPLMN: False
+Hide in OPLMN: False
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -29,7 +29,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -49,8 +49,8 @@ OPLMNwAcT:
 HPLMNAcT: Can't read file -- SW match failed! Expected 9000 and got 6a82.
 ACC: abce
 MSISDN: Not available
-Administrative data: 00ffff02
-	MS operation mode: normal operation
+Administrative data: 00000102
+	MS operation mode: normal
 	Ciphering Indicator: enabled
 SIM Service Table: ff33ff0f3c00ff0f000cf0c0f0030000
 	Service 1 - CHV1 disable function

pysim-testdata/fakemagicsim.ok

@@ -8,8 +8,8 @@ GID1: Can't read file -- SW match failed! Expected 9000 and got 9404.
 GID2: Can't read file -- SW match failed! Expected 9000 and got 9404.
 SMSP: ffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
 SPN: Magic
-Display HPLMN: True
-Display OPLMN: False
+Show in HPLMN: True
+Hide in OPLMN: False
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
 OPLMNwAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
@@ -17,7 +17,7 @@ HPLMNAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
 ACC: ffff
 MSISDN: Not available
 Administrative data: 000000
-	MS operation mode: normal operation
+	MS operation mode: normal
 	Ciphering Indicator: disabled
 SIM Service Table: ff3fff0f0300f003000c
 	Service 1 - CHV1 disable function

pysim-testdata/sysmoISIM-SJA2.data

@@ -4,4 +4,4 @@ ICCID=1122334455667788990
 KI=AABBCCDDEEFFAABBCCDDEEFFAABBCCDD
 OPC=12345678901234567890123456789012
 IMSI=001010000000102
-ADM=11111111
+ADM=67225880

pysim-testdata/sysmoISIM-SJA2.ok

@@ -1,17 +1,17 @@
 Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmoISIM-SJA2
-ICCID: 8988211900000000025
+ICCID: 8988211000000467343
 IMSI: 001010000000102
 GID1: ffffffffffffffffffff
 GID2: ffffffffffffffffffff
 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
-SPN: Not available
-Display HPLMN: False
-Display OPLMN: False
+SPN: Magic
+Show in HPLMN: True
+Hide in OPLMN: True
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -25,7 +25,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -39,7 +39,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -52,10 +52,10 @@ HPLMNAcT:
 	ffffff0000 # unused
 	ffffff0000 # unused
 
-ACC: 0200
+ACC: 0010
 MSISDN (NPI=1 ToN=3): 6766266
 Administrative data: 00000002
-	MS operation mode: normal operation
+	MS operation mode: normal
 	Ciphering Indicator: disabled
 SIM Service Table: ff33ffff3f003f0f300cf0c3f00000
 	Service 1 - CHV1 disable function
@@ -108,7 +108,7 @@ EHPLMN:
 	ffffff # unused
 	ffffff # unused
 
-USIM Service Table: beff9f9de73e0408400170330006002e00000000
+USIM Service Table: beff9f9de73e0408400170330000002e00000000
 	Service 2 - Fixed Dialling Numbers (FDN)
 	Service 3 - Extension 2
 	Service 4 - Service Dialling Numbers (SDN)
@@ -156,8 +156,6 @@ USIM Service Table: beff9f9de73e0408400170330006002e00000000
 	Service 90 - Operator CSG Lists and corresponding indications
 	Service 93 - Communication Control for IMS by USIM
 	Service 94 - Extended Terminal Applications
-	Service 106 - ePDG configuration Information support
-	Service 107 - ePDG configuration Information configured
 	Service 122 - 5GS Mobility Management Information
 	Service 123 - 5G Security Parameters
 	Service 124 - Subscription identifier privacy support

pysim-testdata/sysmoUSIM-SJS1.ok

@@ -7,11 +7,11 @@ GID1: ffffffffffffffffffff
 GID2: ffffffffffffffffffff
 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
 SPN: Magic
-Display HPLMN: True
-Display OPLMN: True
+Show in HPLMN: True
+Hide in OPLMN: True
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -25,7 +25,7 @@ PLMNwAcT:
 	ffffff0000 # unused
 
 OPLMNwAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -39,7 +39,7 @@ OPLMNwAcT:
 	ffffff0000 # unused
 
 HPLMNAcT:
-	00f110ffff # MCC: 001 MNC: 001 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
+	00f110ffff # MCC: 001 MNC: 01 AcT: UTRAN, E-UTRAN WB-S1, E-UTRAN NB-S1, NG-RAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
 	ffffff0000 # unused
 	ffffff0000 # unused
 	ffffff0000 # unused
@@ -55,7 +55,7 @@ HPLMNAcT:
 ACC: 0008
 MSISDN (NPI=1 ToN=1): +77776336143
 Administrative data: 00000002
-	MS operation mode: normal operation
+	MS operation mode: normal
 	Ciphering Indicator: disabled
 SIM Service Table: ff3fffff3f003f1ff00c00c0f00000
 	Service 1 - CHV1 disable function

pysim-testdata/sysmosim-gr1.ok

@@ -8,8 +8,8 @@ GID1: Can't read file -- SW match failed! Expected 9000 and got 9404.
 GID2: Can't read file -- SW match failed! Expected 9000 and got 9404.
 SMSP: ffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
 SPN: Not available
-Display HPLMN: False
-Display OPLMN: False
+Show in HPLMN: False
+Hide in OPLMN: False
 PLMNsel: 00f110ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 PLMNwAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
 OPLMNwAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
@@ -17,7 +17,7 @@ HPLMNAcT: Can't read file -- SW match failed! Expected 9000 and got 9404.
 ACC: 0008
 MSISDN: Not available
 Administrative data: 000000
-	MS operation mode: normal operation
+	MS operation mode: normal
 	Ciphering Indicator: disabled
 SIM Service Table: ff3fff0f0f0000030000
 	Service 1 - CHV1 disable function

requirements.txt

@@ -0,0 +1,9 @@
+pyscard
+pyserial
+pytlv
+cmd2==1.5
+jsonpath-ng
+construct
+bidict
+gsm0338
+pyyaml>=5.1

scripts/sysmoISIM-SJA2/dump-auth-cfg.pysim

@@ -0,0 +1,41 @@
+# script to be used with pySim-shell.py which is part of the Osmocom pysim package,
+# found at https://osmocom.org/projects/pysim/wiki
+set echo true
+
+# TODO: add your card-specific ADM pin at the end of the verify_adm line below
+verify_adm
+
+select DF.SYSTEM
+
+# Milenage configuration (constants)
+select EF.MILENAGE_CFG
+read_binary_decoded
+
+# 2G authentication kay / algorithm
+select EF.SIM_AUTH_KEY
+read_binary_decoded
+
+# OTA keys
+#select EF.0348_KEY
+#read_records_decoded
+
+select ADF.USIM
+# USIM authentication key / algoritmh in 3G security context
+select EF.USIM_AUTH_KEY
+read_binary_decoded
+# USIM authentication key / algorithm in 2G security context
+select EF.USIM_AUTH_KEY_2G
+read_binary_decoded
+# USIM SQN numbers
+select EF.USIM_SQN
+read_binary_decoded
+
+select ADF.ISIM
+# ISIM authentication key / algorithm
+select EF.ISIM_AUTH_KEY
+read_binary_decoded
+# ISIM SQN numbers
+select EF.ISIM_SQN
+read_binary_decoded
+
+quit

setup.cfg

@@ -0,0 +1,3 @@
+[metadata]
+long_description = file: README.md
+long_description_content_type = text/markdown

setup.py

@@ -0,0 +1,26 @@
+from setuptools import setup
+
+setup(
+    name='pySim',
+    version='1.0',
+    packages=['pySim', 'pySim.transport'],
+    url='https://osmocom.org/projects/pysim/wiki',
+    license='GPLv2',
+    author_email='simtrace@lists.osmocom.org',
+    description='Tools related to SIM/USIM/ISIM cards',
+    install_requires=[
+        "pyscard",
+        "serial",
+        "pytlv",
+        "cmd2 >= 1.3.0, < 2.0.0",
+        "jsonpath-ng",
+        "construct >= 2.9",
+        "bidict",
+        "gsm0338",
+    ],
+    scripts=[
+        'pySim-prog.py',
+        'pySim-read.py',
+        'pySim-shell.py'
+    ]
+)

tests/Wavemobile-SIM.data.example

@@ -1,5 +0,0 @@
-MCC=001
-MNC=01
-IMSI=001010000000102
-ADM_HEX=0123456789ABCDEF
-

tests/fakemagicsim.data.example

@@ -1,6 +0,0 @@
-MCC=001
-MNC=01
-ICCID=1122334455667788990
-KI=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-OPC=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-IMSI=001010000000102

tests/pysim-test.sh

@@ -23,6 +23,7 @@
 PYSIM_PROG=../pySim-prog.py
 PYSIM_READ=../pySim-read.py
 TEMPFILE=temp.tmp
+PYTHON=python3
 
 set -e
 
@@ -76,7 +77,7 @@ function check_card {
     CARD_NAME=$2
     echo "Verifying card ..."
     stat ./$CARD_NAME.ok > /dev/null
-    python $PYSIM_READ -p $TERMINAL > $TEMPFILE
+    $PYTHON $PYSIM_READ -p $TERMINAL > $TEMPFILE
     set +e
     CARD_DIFF=$(diff $TEMPFILE ./$CARD_NAME.ok)
     set -e
@@ -106,7 +107,7 @@ function check_card {
 function gen_ok_file {
     TERMINAL=$1
     CARD_NAME=$2
-    python $PYSIM_READ -p $TERMINAL > "$CARD_NAME.ok"
+    $PYTHON $PYSIM_READ -p $TERMINAL > "$CARD_NAME.ok"
     echo "Generated file: $CARD_NAME.ok"
     echo "------------8<------------"
     cat "$CARD_NAME.ok"
@@ -166,7 +167,7 @@ function run_test {
 		ADM_OPT="-A"
 		ADM=$ADM_HEX
 	fi
-	python $PYSIM_PROG -p $I -t $CARD_NAME -o $OPC -k $KI -x $MCC -y $MNC -i $IMSI -s $ICCID --msisdn $MSISDN $ADM_OPT $ADM
+	$PYTHON $PYSIM_PROG -p $I -t $CARD_NAME -o $OPC -k $KI -x $MCC -y $MNC -i $IMSI -s $ICCID --msisdn $MSISDN $ADM_OPT $ADM
 	check_card $I $CARD_NAME
 	echo ""
     done

tests/sysmoUSIM-SJS1.data.example

@@ -1,8 +0,0 @@
-MCC=001
-MNC=01
-ICCID=1122334455667788990
-KI=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-OPC=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-IMSI=001010000000102
-MSISDN=+77776336143
-ADM=12345678

tests/sysmosim-gr1.data.example

@@ -1,7 +0,0 @@
-MCC=001
-MNC=01
-ICCID=1122334455667788990
-KI=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-OPC=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-IMSI=001010000000102
-ADM=DDDDDDDD

tests/test_utils.py

@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+
+import unittest
+from pySim import utils
+from pySim.ts_31_102 import EF_SUCI_Calc_Info
+
+class DecTestCase(unittest.TestCase):
+	# TS33.501 Annex C.4 test keys
+	hnet_pubkey_profile_b = "0272DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD1" # ID 27 in test file
+	hnet_pubkey_profile_a = "5A8D38864820197C3394B92613B20B91633CBD897119273BF8E4A6F4EEC0A650" # ID 30 in test file  
+
+	# TS31.121 4.9.4 EF_SUCI_Calc_Info test file
+	testfile_suci_calc_info = "A006020101020000A14B80011B8121" +hnet_pubkey_profile_b +"80011E8120" +hnet_pubkey_profile_a
+
+	decoded_testfile_suci = {
+		'prot_scheme_id_list': [
+			{'priority': 0, 'identifier': 2, 'key_index': 1},
+			{'priority': 1, 'identifier': 1, 'key_index': 2},
+			{'priority': 2, 'identifier': 0, 'key_index': 0}],
+		'hnet_pubkey_list': [
+			{'hnet_pubkey_identifier': 27, 'hnet_pubkey': hnet_pubkey_profile_b.lower()}, # because h2b/b2h returns all lower-case
+			{'hnet_pubkey_identifier': 30, 'hnet_pubkey': hnet_pubkey_profile_a.lower()}]
+	}
+
+	def testSplitHexStringToListOf5ByteEntries(self):
+		input_str = "ffffff0003ffffff0002ffffff0001"
+		expected = [
+			"ffffff0003",
+			"ffffff0002",
+			"ffffff0001",
+		]
+		self.assertEqual(utils.hexstr_to_Nbytearr(input_str, 5), expected)
+
+	def testDecMCCfromPLMN(self):
+		self.assertEqual(utils.dec_mcc_from_plmn("92f501"), 295)
+
+	def testDecMCCfromPLMN_unused(self):
+		self.assertEqual(utils.dec_mcc_from_plmn("ff0f00"), 4095)
+
+	def testDecMCCfromPLMN_str(self):
+		self.assertEqual(utils.dec_mcc_from_plmn_str("92f501"), "295")
+
+	def testDecMCCfromPLMN_unused_str(self):
+		self.assertEqual(utils.dec_mcc_from_plmn_str("ff0f00"), "")
+
+	def testDecMNCfromPLMN_twoDigitMNC(self):
+		self.assertEqual(utils.dec_mnc_from_plmn("92f501"), 10)
+
+	def testDecMNCfromPLMN_threeDigitMNC(self):
+		self.assertEqual(utils.dec_mnc_from_plmn("031263"), 361)
+
+	def testDecMNCfromPLMN_unused(self):
+		self.assertEqual(utils.dec_mnc_from_plmn("00f0ff"), 4095)
+
+	def testDecMNCfromPLMN_twoDigitMNC_str(self):
+		self.assertEqual(utils.dec_mnc_from_plmn_str("92f501"), "10")
+
+	def testDecMNCfromPLMN_threeDigitMNC_str(self):
+		self.assertEqual(utils.dec_mnc_from_plmn_str("031263"), "361")
+
+	def testDecMNCfromPLMN_unused_str(self):
+		self.assertEqual(utils.dec_mnc_from_plmn_str("00f0ff"), "")
+
+	def test_enc_plmn(self):
+		with self.subTest("2-digit MCC"):
+			self.assertEqual(utils.enc_plmn("001", "01F"), "00F110")
+			self.assertEqual(utils.enc_plmn("001", "01"), "00F110")
+			self.assertEqual(utils.enc_plmn("295", "10"), "92F501")
+
+		with self.subTest("3-digit MCC"):
+			self.assertEqual(utils.enc_plmn("001", "001"), "001100")
+			self.assertEqual(utils.enc_plmn("302", "361"), "031263")
+
+	def testDecAct_noneSet(self):
+		self.assertEqual(utils.dec_act("0000"), [])
+
+	def testDecAct_onlyUtran(self):
+		self.assertEqual(utils.dec_act("8000"), ["UTRAN"])
+
+	def testDecAct_onlyEUtran(self):
+		self.assertEqual(utils.dec_act("4000"), ["E-UTRAN"])
+
+	def testDecAct_onlyNgRan(self):
+		self.assertEqual(utils.dec_act("0800"), ["NG-RAN"])
+
+	def testDecAct_onlyGsm(self):
+		self.assertEqual(utils.dec_act("0080"), ["GSM"])
+
+	def testDecAct_onlyGsmCompact(self):
+		self.assertEqual(utils.dec_act("0040"), ["GSM COMPACT"])
+
+	def testDecAct_onlyCdma2000HRPD(self):
+		self.assertEqual(utils.dec_act("0020"), ["cdma2000 HRPD"])
+
+	def testDecAct_onlyCdma20001xRTT(self):
+		self.assertEqual(utils.dec_act("0010"), ["cdma2000 1xRTT"])
+
+	def testDecAct_allSet(self):
+		self.assertEqual(utils.dec_act("ffff"), ["UTRAN", "E-UTRAN WB-S1", "E-UTRAN NB-S1", "NG-RAN", "GSM", "GSM COMPACT", "cdma2000 HRPD", "cdma2000 1xRTT"])
+
+	def testDecxPlmn_w_act(self):
+		expected = {'mcc': '295', 'mnc': '10', 'act': ["UTRAN"]}
+		self.assertEqual(utils.dec_xplmn_w_act("92f5018000"), expected)
+
+	def testFormatxPlmn_w_act(self):
+		input_str = "92f501800092f5508000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000ffffff0000"
+		expected  = "\t92f5018000 # MCC: 295 MNC: 10 AcT: UTRAN\n"
+		expected += "\t92f5508000 # MCC: 295 MNC: 05 AcT: UTRAN\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		expected += "\tffffff0000 # unused\n"
+		self.assertEqual(utils.format_xplmn_w_act(input_str), expected)
+
+
+	def testDecodeSuciCalcInfo(self):
+		suci_calc_info = EF_SUCI_Calc_Info()
+		decoded = suci_calc_info._decode_hex(self.testfile_suci_calc_info)
+		self.assertDictEqual(self.decoded_testfile_suci, decoded)
+
+	def testEncodeSuciCalcInfo(self):
+		suci_calc_info = EF_SUCI_Calc_Info()
+		encoded = suci_calc_info._encode_hex(self.decoded_testfile_suci)
+		self.assertEqual(encoded.lower(), self.testfile_suci_calc_info.lower())
+
+	def testEnc_msisdn(self):
+		msisdn_encoded = utils.enc_msisdn("+4916012345678", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "0891946110325476f8ffffffffff")
+		msisdn_encoded = utils.enc_msisdn("123456", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "04b1214365ffffffffffffffffff")
+		msisdn_encoded = utils.enc_msisdn("12345678901234567890", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "0bb121436587092143658709ffff")
+		msisdn_encoded = utils.enc_msisdn("+12345678901234567890", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "0b9121436587092143658709ffff")
+		msisdn_encoded = utils.enc_msisdn("", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "ffffffffffffffffffffffffffff")
+		msisdn_encoded = utils.enc_msisdn("+", npi=0x01, ton=0x03)
+		self.assertEqual(msisdn_encoded, "ffffffffffffffffffffffffffff")
+
+	def testDec_msisdn(self):
+		msisdn_decoded = utils.dec_msisdn("0891946110325476f8ffffffffff")
+		self.assertEqual(msisdn_decoded, (1, 1, "+4916012345678"))
+		msisdn_decoded = utils.dec_msisdn("04b1214365ffffffffffffffffff")
+		self.assertEqual(msisdn_decoded, (1, 3, "123456"))
+		msisdn_decoded = utils.dec_msisdn("0bb121436587092143658709ffff")
+		self.assertEqual(msisdn_decoded, (1, 3, "12345678901234567890"))
+		msisdn_decoded = utils.dec_msisdn("ffffffffffffffffffffffffffff")
+		self.assertEqual(msisdn_decoded, None)
+		msisdn_decoded = utils.dec_msisdn("00112233445566778899AABBCCDDEEFF001122330bb121436587092143658709ffff")
+		self.assertEqual(msisdn_decoded, (1, 3, "12345678901234567890"))
+		msisdn_decoded = utils.dec_msisdn("ffffffffffffffffffffffffffffffffffffffff0bb121436587092143658709ffff")
+		self.assertEqual(msisdn_decoded, (1, 3, "12345678901234567890"))
+
+class TestBerTlv(unittest.TestCase):
+    def test_BerTlvTagDec(self):
+        res = utils.bertlv_parse_tag(b'\x01')
+        self.assertEqual(res, ({'tag':1, 'constructed':False, 'class': 0}, b''))
+        res = utils.bertlv_parse_tag(b'\x21')
+        self.assertEqual(res, ({'tag':1, 'constructed':True, 'class': 0}, b''))
+        res = utils.bertlv_parse_tag(b'\x81\x23')
+        self.assertEqual(res, ({'tag':1, 'constructed':False, 'class': 2}, b'\x23'))
+        res = utils.bertlv_parse_tag(b'\x1f\x8f\x00\x23')
+        self.assertEqual(res, ({'tag':0xf<<7, 'constructed':False, 'class': 0}, b'\x23'))
+
+    def test_BerTlvLenDec(self):
+        self.assertEqual(utils.bertlv_encode_len(1), b'\x01')
+        self.assertEqual(utils.bertlv_encode_len(127), b'\x7f')
+        self.assertEqual(utils.bertlv_encode_len(128), b'\x81\x80')
+        self.assertEqual(utils.bertlv_encode_len(0x123456), b'\x83\x12\x34\x56')
+
+    def test_BerTlvLenEnc(self):
+        self.assertEqual(utils.bertlv_parse_len(b'\x01\x23'), (1, b'\x23'))
+        self.assertEqual(utils.bertlv_parse_len(b'\x7f'), (127, b''))
+        self.assertEqual(utils.bertlv_parse_len(b'\x81\x80'), (128, b''))
+        self.assertEqual(utils.bertlv_parse_len(b'\x83\x12\x34\x56\x78'), (0x123456, b'\x78'))
+
+    def test_BerTlvParseOne(self):
+        res = utils.bertlv_parse_one(b'\x81\x01\x01');
+        self.assertEqual(res, ({'tag':1, 'constructed':False, 'class':2}, 1, b'\x01', b''))
+
+class TestComprTlv(unittest.TestCase):
+    def test_ComprTlvTagDec(self):
+        res = utils.comprehensiontlv_parse_tag(b'\x12\x23')
+        self.assertEqual(res, ({'tag': 0x12, 'comprehension': False}, b'\x23'))
+        res = utils.comprehensiontlv_parse_tag(b'\x92')
+        self.assertEqual(res, ({'tag': 0x12, 'comprehension': True}, b''))
+        res = utils.comprehensiontlv_parse_tag(b'\x7f\x12\x34')
+        self.assertEqual(res, ({'tag': 0x1234, 'comprehension': False}, b''))
+        res = utils.comprehensiontlv_parse_tag(b'\x7f\x82\x34\x56')
+        self.assertEqual(res, ({'tag': 0x234, 'comprehension': True}, b'\x56'))
+
+    def test_ComprTlvTagEnc(self):
+        res  = utils.comprehensiontlv_encode_tag(0x12)
+        self.assertEqual(res, b'\x12')
+        res  = utils.comprehensiontlv_encode_tag({'tag': 0x12})
+        self.assertEqual(res, b'\x12')
+        res  = utils.comprehensiontlv_encode_tag({'tag': 0x12, 'comprehension':True})
+        self.assertEqual(res, b'\x92')
+        res  = utils.comprehensiontlv_encode_tag(0x1234)
+        self.assertEqual(res, b'\x7f\x12\x34')
+        res  = utils.comprehensiontlv_encode_tag({'tag': 0x1234, 'comprehension':True})
+        self.assertEqual(res, b'\x7f\x92\x34')
+
+if __name__ == "__main__":
+	unittest.main()