pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

contrib/csv-to-pgsql.py

@@ -285,10 +285,7 @@ if __name__ == '__main__':
     option_parser.add_argument("--admin", action='store_true', help="perform action as admin", default=False)
     opts = option_parser.parse_args()
 
-    PySimLogger.setup(print, {logging.WARN: "\033[33m"})
-    if (opts.verbose):
-        PySimLogger.set_verbose(True)
-        PySimLogger.set_level(logging.DEBUG)
+    PySimLogger.setup(print, {logging.WARN: "\033[33m"}, opts.verbose)
 
     # Open CSV file
     cr = open_csv(opts)

docs/put_key-tutorial.rst

@@ -1,842 +0,0 @@
-Guide: Managing GP Keys
-=======================
-
-Most of todays smartcards follow the GlobalPlatform Card Specification and the included Security Domain model.
-UICCs and eUCCCs are no exception here.
-
-The Security Domain acts as an on-card representative of a card authority or administrator. It is used to perform tasks
-like the installation of applications or the provisioning and rotation of secure channel keys. It also acts as a secure
-key storage and offers all kinds of cryptographic services to applications that are installed under a specific
-Security Domain (see also GlobalPlatform Card Specification, section 7).
-
-In this tutorial, we will show how to work with the key material (keysets) stored inside a Security Domain and how to
-rotate (replace) existing keys. We will also show how to provision new keys.
-
-.. warning:: Making changes to keysets requires extreme caution as misconfigured keysets may lock you out permanently.
-	     It also strongly recommended to maintain at least one backup keyset that you can use as fallback in case
-	     the primary keyset becomes unusable for some reason.
-
-
-Selecting a Security Domain
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A typical smartcard, such as an UICC will have one primary Security Domain, called the Issuer Security Domain (ISD).
-When working with those cards, the ISD will show up in the UICC filesystem tree as `ADF.ISD` and can be selected like
-any other file.
-
-::
-   
-   pySIM-shell (00:MF)> select ADF.ISD
-   {
-      "application_id": "a000000003000000",
-      "proprietary_data": {
-         "maximum_length_of_data_field_in_command_message": 255
-      }
-   }
-
-When working with eUICCs, multiple Security Domains are involved. The model is slightly different from the classic
-model with one primary ISD. In the case of eUICCs, an ISD-R and an ISD-P exists.
-
-The ISD-R (Issuer Security Domain - Root) is indeed the primary ISD. Its purpose is to handle the installation of new
-profiles and to manage the already installed profiles. The ISD-R shows up as a `ADF.ISD-R` and can be selected normally
-(see above) The key material that allows access to the ISD-R is usually only known to the eUICC manufacturer.
-
-The ISD-P (Issuer Security Domain - Profile) is the primary ISD of the currently enabled profile. The ISD-P is
-comparable to the ISD we find on a UICC. The key material for the ISD-P should be known known to the ISP, which
-is the owner of the installed profile.
-
-Since the AID of the ISD-P is allocated during the profile installation and different for each profile, it is not known
-by pySim-shell. This means there will no `ADF.ISD-P` file show up in the file system, but we can simply select the
-ISD-R, request the AID of the ISD-P and switch over to that ISD-P using a raw APDU:
-``00a4040410`` + ``a0000005591010ffffffff8900001000`` + ``00``
-
-::
-
-    pySIM-shell (00:MF)> select ADF.ISD-R
-    {
-        "application_id": "a0000005591010ffffffff8900000100",
-        "proprietary_data": {
-            "maximum_length_of_data_field_in_command_message": 255
-        },
-        "isdr_proprietary_application_template": {
-            "supported_version_number": "020300"
-        }
-    }
-    pySIM-shell (00:MF/ADF.ISD-R)> get_profiles_info 
-    {
-        "profile_info_seq": {
-            "profile_info": {
-                "iccid": "8949449999999990023",
-                "isdp_aid": "a0000005591010ffffffff8900001000",
-                "profile_state": "enabled",
-                "service_provider_name": "OsmocomSPN",
-                "profile_name": "TS48V1-A-UNIQUE",
-                "profile_class": "operational"
-            }
-        }
-    }
-    pySIM-shell (00:MF/ADF.ISD-R)> apdu 00a4040410a0000005591010ffffffff890000100000
-    SW: 9000, RESP: 6f188410a0000005591010ffffffff8900001000a5049f6501ff
-    pySIM-shell (00:MF/ADF.ISD-R)>
-
-After that, the prompt will still show the ADF.ISD-R, but we are actually in ADF.ISD-P and the standard GlobalPlatform
-operations like `establish_scpXX`, `get_data`, and `put_key` should work. The same workaround can also be applied to any
-Supplementary Security Domain as well, provided that the AID is known to the user.
-
-
-Establishing a secure channel
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Before we can make changes to the keysets in the currently selected Security Domain we must first establish a secure channel
-with that Security Domain. The secure channel protocols commonly used for this are `SCP02` (see also GlobalPlatform Card
-Specification, section E.1.1) and `SCP03` (see also GlobalPlatform Card Specification – Amendment D). `SCP02` is slightly
-older and commonly used on UICCs. The more modern `SCP03` is commonly used on eUICCs. The main difference between the
-two is that `SCP02` uses 3DES while `SCP03` is based on AES.
-
-.. warning:: Secure channel protocols like `SCP02` and `SCP03` may manage an error counter to count failed login
-	     attempts. This means attempting to establish a secure channel with a wrong keyset multiple times may lock
-	     you out permanently. Double check the applied keyset before attempting to establish a secure channel.
-
-
-Example: `SCP02`
-----------------
-
-In the following example, we assume that we want to establish a secure channel with the ISD of a `sysmoUSIM-SJA5` UICC.
-Along with the card we have received the following keyset:
-
-+---------+----------------------------------+
-| Keyname | Keyvalue                         |
-+=========+==================================+
-| ENC/KIC | F09C43EE1A0391665CC9F05AF4E0BD10 |
-+---------+----------------------------------+
-| MAC/KID | 01981F4A20999F62AF99988007BAF6CA |
-+---------+----------------------------------+
-| DEK/KIK | 8F8AEE5CDCC5D361368BC45673D99195 |
-+---------+----------------------------------+
-
-This keyset is tied to the key version number KVN 122 and is configured as a DES keyset. We can use this keyset to
-establish a secure channel using the SCP02 Secure Channel Protocol.
-
-::
-
-    pySIM-shell (00:MF/ADF.ISD)> establish_scp02 --key-enc F09C43EE1A0391665CC9F05AF4E0BD10 --key-mac 01981F4A20999F62AF99988007BAF6CA --key-dek 8F8AEE5CDCC5D361368BC45673D99195 --security-level 3
-    Successfully established a SCP02[03] secure channel
-
-
-Example: `SCP03`
-----------------
-
-The establishment of a secure channel via SCP03 works just the same. In the following example we will establish a
-secure channel to the ISD-R of an eUICC. The SCP03 keyset we use is tied to KVN 50 and looks like this:
-
-+---------+------------------------------------------------------------------+
-| Keyname | Keyvalue                                                         |
-+=========+==================================================================+
-| ENC/KIC | 620ff456b0c0328b68dc0d7d5eb24e07dd749aa86c9ff1836a7263e1d8896510 |
-+---------+------------------------------------------------------------------+
-| MAC/KID | b38116a2c85f2c8f46bbdc0081d6e8a04b0a58087d0ce5ee0ccc4c945e4aeda6 |
-+---------+------------------------------------------------------------------+
-| DEK/KIK | d409486cbcb8092a8592ee46d8668dfa97bea5eb7ce9c2b5a3f3bb1db358a153 |
-+---------+------------------------------------------------------------------+
-
-We assume that ADF.ISD-R is already selected. We may now establish the SCP03 secure channel:
-
-::
-   
-   pySIM-shell (00:MF/ADF.ISD-R)> establish_scp03 --key-enc 620ff456b0c0328b68dc0d7d5eb24e07dd749aa86c9ff1836a7263e1d8896510 --key-mac b38116a2c85f2c8f46bbdc0081d6e8a04b0a58087d0ce5ee0ccc4c945e4aeda6 --key-dek d409486cbcb8092a8592ee46d8668dfa97bea5eb7ce9c2b5a3f3bb1db358a153 --key-ver 50 --security-level 3
-   Successfully established a SCP03[03] secure channel
-
-
-Understanding Keysets
-~~~~~~~~~~~~~~~~~~~~~
-
-Before making any changes to keysets, it is recommended to check the status of the currently installed keysets. To do
-so, we use the `get_data` command to retrieve the `key_information`. We cannot read back the key values themselves, but
-we get a summary of the installed keys together with their KVN numbers, IDs, algorithm and key length values.
-
-Example: `key_information` from a `sysmoISIM-SJA5`:
-
-::
-
-    pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> get_data key_information 
-    {
-        "key_information": [
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 112,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 112,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 112,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 1,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 1,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 1,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 2,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 2,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 2,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 47,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 47,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 47,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            }
-        ]
-    }
-
-Example: `key_information` from a `sysmoEUICC1-C2T`:
-
-::
-
-    pySIM-shell (SCP03[03]:00:MF/ADF.ISD-R)> get_data key_information
-    {
-        "key_information": [
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 50,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 50,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 50,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 64,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 64,
-                    "key_types": [
-                        {
-                            "type": "tls_psk",
-                            "length": 16
-                        }
-                    ]
-                }
-            }
-        ]
-    }
-
-The output from those two examples above may seem lengthy, but in order to move on and to provision own keys
-successfully, it is important to understand each aspect of it.
-
-Key Version Number (KVN)
-------------------------
-
-Each key is associated with a Key Version Number (KVN). Multiple keys that share the same KVN belong to the same
-keyset. In the first example above we can see that four keysets with KVN numbers 112, 1, 2 and 47 are provisioned.
-In the second example we see two keysets. One with KVN 50 and one with KVN 64.
-
-The term "Key Version Number" is misleading as this number is not really a version number. It's actually a unique
-identifier for a specific keyset that also defines with which Secure Channel Protocol a key can be used. This means
-that the KVN is not just an arbitrary number. The following (incomplete) table gives a hint which KVN numbers may be
-used with which Secure Channel Protocol.
-
-+-----------+-------------------------------------------------------+
-| KVN range | Secure Channel Protocol                               |
-+===========+=======================================================+
-| 1-15      | reserved for `SCP80` (OTA SMS)                        |
-+-----------+-------------------------------------------------------+
-| 17        | reserved for DAP specified in ETSI TS 102 226         |
-+-----------+-------------------------------------------------------+
-| 32-47     | reserved for `SCP02`                                  |
-+-----------+-------------------------------------------------------+
-| 48-63     | reserved for `SCP03`                                  |
-+-----------+-------------------------------------------------------+
-| 112       | Token key (RSA public or DES, also used with `SCP02`) |
-+-----------+-------------------------------------------------------+
-| 113       | Receipt key (DES)                                     |
-+-----------+-------------------------------------------------------+
-| 115       | DAP verifiation key (RS public or DES)                |
-+-----------+-------------------------------------------------------+
-| 116       | reserved for CASD                                     |
-+-----------+-------------------------------------------------------+
-| 117       | 16-byte DES key for Ciphered Load File Data Block     |
-+-----------+-------------------------------------------------------+
-| 129-143   | reserved for `SCP81`                                  |
-+-----------+-------------------------------------------------------+
-| 255       | reserved for ISD with SCP02 without SCP80 support     |
-+-----------+-------------------------------------------------------+
-
-With that we can now understand that in the first example, the first and the last keyset is intended to be used with
-`SCP02` and that the second and the third keyset is intended to be used with `SCP80` (OTA SMS). In the second example we
-can see that the first keyset is intended to be used with `SCP03`, wheres the second should be usable with `SCP81`.
-
-
-Key Identifier
---------------
-
-Each keyset consists of a number of keys, where each key has a different Key Identifier. The Key Identifier is usually
-an incrementing number that starts counting at 1. The Key Identifier is used to distinguish the keys within the keyset.
-The exact number of keys and their attributes depends on the secure channel protocol for which the keyset is intended
-for. Each secure channel protocol may have its specific requirements on how many keys of which which type, length or
-Key Identifier have to be present.
-
-However, almost all of the classic secure channel protocols (including  `SCP02`, `SCP03` and `SCP81`) make use of the
-following three-key scheme:
-
-+----------------+---------+---------------------------------------+
-| Key Identifier | Keyname | Purpose                               |
-+================+=========+=======================================+
-| 1              | ENC/KIC | encryption/decryption                 |
-+----------------+---------+---------------------------------------+
-| 2              | MAC/KID | cryptographic checksumming/signing    |
-+----------------+---------+---------------------------------------+
-| 3              | DEK/KIK | encryption/decryption of key material |
-+----------------+---------+---------------------------------------+
-
-In this case, all three keys share the same length and are used with the same algorithm. The key length is often used
-to implicitly select sub-types of an algorithm. (e.g. a 16 byte key of type `aes` is associated with `AES128`, where a 32
-byte key would be associated with `AES256`).
-
-That different schemes are possible shows the second example. The `SCP80` keyset from the second example uses a scheme
-that works with two keys:
-
-+----------------+---------+---------------------------------------+
-| Key Identifier | Keyname | Purpose                               |
-+================+=========+=======================================+
-| 1              | TLS-PSK | pre-shared key used for TLS           |
-+----------------+---------+---------------------------------------+
-| 2              | DEK/KIK | encryption/decryption of key material |
-+----------------+---------+---------------------------------------+
-
-It should also be noted that the order in which keysets and keys appear is an implementation detail of the UICC/eUICC
-O/S. The order has no influence on how a keyset is interpreted. Only the Key Version Number (KVN) and the Key Identifier
-matter.
-	
-
-Rotating a keyset
-~~~~~~~~~~~~~~~~~
-
-Rotating keys is one of the most basic tasks one might want to perform on an UICC/eUICC before using it productively. In
-the following example we will illustrate how key rotation can be done. When rotating keys, only the key itself may
-change. For example it is not possible to change the key length or the algorithm used (see also GlobalPlatform Card
-Specification, section 11.8.2.3.3). Any key of the current Security Domain can be rotated, this also includes the key
-that was used to establish the secure channel.
-
-In the following example we assume that the Security Domain is selected and a secure channel is already established. We
-intend to rotate the keyset with KVN 112. Since this keyset uses triple DES keys with a key length of 16, we must
-replace it with a keyset with keys of the same nature.
-
-The new keyset shall look like this:
-
-+----------------+---------+----------------------------------+
-| Key Identifier | Keyname | Keyvalue                         |
-+================+=========+==================================+
-| 1              | ENC/KIC | 542C37A6043679F2F9F71116418B1CD5 |
-+----------------+---------+----------------------------------+
-| 2              | MAC/KID | 34F11BAC8E5390B57F4E601372339E3C |
-+----------------+---------+----------------------------------+
-| 3              | DEK/KIK | 5524F4BECFE96FB63FC29D6BAAC6058B |
-+----------------+---------+----------------------------------+
-
-When passing the keys to the `put_key` commandline, we set the Key Identifier of the first key using the `--key-id`
-parameter. This Key Identifier will be valid for the first key (KIC) we pass. For all consecutive keys, the Key
-Identifier will be incremented automatically (see also GlobalPlatform Card Specification, section 11.8.2.2). To Ensure
-that the new KIC, KID and KIK keys get the correct Key Identifiers, it is crucial to maintain order when passing the
-keys in the `--key-data` arguments. It is also important that each `--key-data` argument is preceded by a `--key-type`
-argument that sets the algorithm correctly (`des` in this case).
-
-Finally we have to target the keyset we want to rotate by its KVN. The `--old-key-version-nr` argument is set to 112
-as this is identifies the keyset we want to rotate. The `--key-version-nr` is also set to 112 as we do not want to the
-KVN to be changed in this example. Changing the KVN while rotating a keyset is possible. In case the KVN has to change
-for some reason, the new KVN must be selected carefully to keep the key usable with the associated Secure Channel
-Protocol.
-
-The commandline that matches the keyset we had laid out above looks like this:
-::
-   
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> put_key --key-id 1 --key-type des --key-data 542C37A6043679F2F9F71116418B1CD5 --key-type des --key-data 34F11BAC8E5390B57F4E601372339E3C --key-type des --key-data 5524F4BECFE96FB63FC29D6BAAC6058B --old-key-version-nr 112 --key-version-nr 112
-
-After executing this put_key commandline, the keyset identified by KVN 122 is equipped with new keys. We can use
-`get_data key_information` to inspect the currently installed keysets. The output should appear unchanged as
-we only swapped out the keys. All other parameters, identifiers etc. should remain constant.
-
-.. warning:: It is technically possible to rotate a keyset in a `non atomic` way using one `put_key` commandline for
-	     each key. However, in case the targeted keyset is the one used to establish the current secure channel,
-	     this method should not be used since, depending on the UICC/eUICC model, half-written key material may
-	     interrupt the current secure channel.
-
-
-Removing a keyset
-~~~~~~~~~~~~~~~~~
-
-In some cases it is necessary to remove a keyset entirely. This can be done with the `delete_key` command. Here it is
-important to understand that `delete_key` only removes one specific key from a specific keyset. This means that you
-need to run a separate `delete_key` command for each key inside a keyset.
-
-In the following example we assume that the Security Domain is selected and a secure channel is already established. We
-intend to remove the keyset with KVN 112. This keyset consists of three keys.
-
-::
-
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> delete_key --key-ver 112 --key-id 1
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> delete_key --key-ver 112 --key-id 2
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> delete_key --key-ver 112 --key-id 3
-
-To verify that the keyset has been deleted properly, we can use the `get_data key_information` command to inspect the
-current status of the installed keysets. We should see that the key with KVN 112 is no longer present.
-
-
-Adding a keyset
-~~~~~~~~~~~~~~~
-
-In the following we will discuss how to add an entirely new keyset. The procedure is almost identical with the key
-rotation procedure we have already discussed and it is assumed that all details about the key rotation are understood.
-In this section we will go into more detail and and illustrate how to provision new 3DES, `AES128` and `AES256` keysets.
-
-It is important to keep in mind that storage space on smartcard is a precious resource. In many cases the amount of
-keysets that a Security Domain can store is limited. In some situations you may be forced to sacrifice one of your
-existing keysets in favor of a new keyset.
-
-The main difference between key rotation and the adding of new keys is that we do not simply replace an existing key.
-Instead an entirely new key is programmed into the Security Domain. Therefore the `put_key` commandline will have no
-`--old-key-version-nr` parameter. From the commandline perspective, this is already the only visible difference from a
-commandline that simply rotates a keyset. Since we are writing an entirely new keyset, we are free to chose the
-algorithm and the key length within the parameter range permitted by the targeted secure channel protocol. Otherwise
-the same rules apply.
-
-For reference, it should be mentioned that it is also possible to add or rotate keyset using multiple `put_key`
-commandlines. In this case one `put_key` commandline for each key is used. Each commandline will specify `--key-id` and
-`--key-version-nr` and one `--key-type` and `--key-data` tuple. However, when rotating or adding a keyset step-by-step,
-the whole process happens in a `non-atomic` way, which is less reliable. Therefore we will favor the `atomic method`
-
-In the following examples we assume that the Security Domain is selected and a secure channel is already established.
-
-
-Example: `3DES` key for `SCP02`
--------------------------------
-
-Let's assume we want to provision a new 3DES keyset that we can use for SCP02. The keyset shall look like this:
-
-+----------------+---------+----------------------------------+
-| Key Identifier | Keyname | Keyvalue                         |
-+================+=========+==================================+
-| 1              | ENC/KIC | 542C37A6043679F2F9F71116418B1CD5 |
-+----------------+---------+----------------------------------+
-| 2              | MAC/KID | 34F11BAC8E5390B57F4E601372339E3C |
-+----------------+---------+----------------------------------+
-| 3              | DEK/KIK | 5524F4BECFE96FB63FC29D6BAAC6058B |
-+----------------+---------+----------------------------------+
-
-The keyset shall be a associated with the KVN 46. We have made sure before that KVN 46 is still unused and that this
-KVN number is actually suitable for SCP02 keys. As we are using 3DES, it is obvious that we have to pass 3 keys with 16
-byte length.
-
-To program the key, we may use the following commandline. As we can see, this commandline is almost the exact same as
-the one from the key rotation example where we were rotating a 3DES key. The only difference is that we didn't specify
-an old KVN number and that we have chosen a different KVN.
-
-::
-   
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> put_key --key-id 1 --key-type des --key-data 542C37A6043679F2F9F71116418B1CD5 --key-type des --key-data 34F11BAC8E5390B57F4E601372339E3C --key-type des --key-data 5524F4BECFE96FB63FC29D6BAAC6058B --key-version-nr 46
-
-In case of success, the keyset should appear in the `key_information` among the other keysets that are already present.
-
-::
-
-    pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> get_data key_information
-    {
-        "key_information": [
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 46,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 46,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 46,
-                    "key_types": [
-                        {
-                            "type": "des",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            ...
-        ]
-    }
-
-
-Example: `AES128` key for `SCP80`
----------------------------------
-
-In this example we intend to provision a new `AES128` keyset that we can use with SCP80 (OTA SMS). The keyset shall look
-like this:
-
-+----------------+---------+----------------------------------+
-| Key Identifier | Keyname | Keyvalue                         |
-+================+=========+==================================+
-| 1              | ENC/KIC | 542C37A6043679F2F9F71116418B1CD5 |
-+----------------+---------+----------------------------------+
-| 2              | MAC/KID | 34F11BAC8E5390B57F4E601372339E3C |
-+----------------+---------+----------------------------------+
-| 3              | DEK/KIK | 5524F4BECFE96FB63FC29D6BAAC6058B |
-+----------------+---------+----------------------------------+
-
-In addition to that, we want to associate this key with KVN 3. We have inspected the currently installed keysets before
-and made sure that KVN 3 is still unused. We are also aware that for SCP80 we may only use KVN values from 1 to 15.
-
-For `AES128`, we specify the algorithm using the `--key-type aes` parameter. The selection between `AES128` and `AES256` is
-done implicitly using the key length. Since we want to use `AES128` in this case, all three keys have a length of 16 byte.
-
-::
-
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> put_key --key-id 1 --key-type aes --key-data 542C37A6043679F2F9F71116418B1CD5 --key-type aes --key-data 34F11BAC8E5390B57F4E601372339E3C --key-type aes --key-data 5524F4BECFE96FB63FC29D6BAAC6058B --key-version-nr 3
-
-In case of success, the keyset should appear in the `key_information` among the other keysets that are already present.
-
-::
-
-    pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> get_data key_information
-    {
-        "key_information": [
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 3,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 3,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 3,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            ...
-        ]
-    }
-
-
-Example: `AES256` key for `SCP03`
----------------------------------
-
-Let's assume we want to provision a new `AES256` keyset that we can use for SCP03. The keyset shall look like this:
-
-+----------------+---------+------------------------------------------------------------------+
-| Key Identifier | Keyname | Keyvalue                                                         |
-+================+=========+==================================================================+
-| 1              | ENC/KIC | 542C37A6043679F2F9F71116418B1CD5542C37A6043679F2F9F71116418B1CD5 |
-+----------------+---------+------------------------------------------------------------------+
-| 2              | MAC/KID | 34F11BAC8E5390B57F4E601372339E3C34F11BAC8E5390B57F4E601372339E3C |
-+----------------+---------+------------------------------------------------------------------+
-| 3              | DEK/KIK | 5524F4BECFE96FB63FC29D6BAAC6058B5524F4BECFE96FB63FC29D6BAAC6058B |
-+----------------+---------+------------------------------------------------------------------+
-
-In addition to that, we assume that we want to associate this key with KVN 51. This KVN number falls in the range of
-48 - 63 and is therefore suitable for a key that shall be usable with SCP03. We also made sure before that KVN 51 is
-still unused.
-
-With that we can go ahead and make up the following commandline:
-::
-   
-   pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> put_key --key-id 1 --key-type aes --key-data 542C37A6043679F2F9F71116418B1CD5542C37A6043679F2F9F71116418B1CD5 --key-type aes --key-data 34F11BAC8E5390B57F4E601372339E3C34F11BAC8E5390B57F4E601372339E3C --key-type aes --key-data 5524F4BECFE96FB63FC29D6BAAC6058B5524F4BECFE96FB63FC29D6BAAC6058B --key-version-nr 51
-
-In case of success, we should see the keyset in the `key_information`
-   
-::
-
-    pySIM-shell (SCP02[03]:00:MF/ADF.ISD)> get_data key_information
-    {
-        "key_information": [
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 51,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 51,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 3,
-                    "key_version_number": 51,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 32
-                        }
-                    ]
-                }
-            },
-            ...
-        ]
-    }
-
-
-Example: `AES128` key for `SCP81`
----------------------------------
-
-In this example we will show how to provision a new `AES128` keyset for `SCP81`. We will provision this keyset under
-KVN 64. The keyset we intend to apply shall look like this:
-
-+----------------+---------+----------------------------------+
-| Key Identifier | Keyname | Keyvalue                         |
-+================+=========+==================================+
-| 1              | TLS-PSK | 000102030405060708090a0b0c0d0e0f |
-+----------------+---------+----------------------------------+
-| 2              | DEK/KIK | 000102030405060708090a0b0c0d0e0f |
-+----------------+---------+----------------------------------+
-
-With that we can put together the following command line:
-
-::
-
-    put_key --key-id 1 --key-type tls_psk --key-data 000102030405060708090a0b0c0d0e0f --key-type aes --key-data 000102030405060708090a0b0c0d0e0f --key-version-nr 64
-
-In case of success, the keyset should appear in the `key_information` as follows:
-
-::
-
-    pySIM-shell (SCP03[03]:00:MF/ADF.ISD-R)> get_data key_information
-    {
-        "key_information": [
-            ...,
-            {
-                "key_information_data": {
-                    "key_identifier": 2,
-                    "key_version_number": 64,
-                    "key_types": [
-                        {
-                            "type": "aes",
-                            "length": 16
-                        }
-                    ]
-                }
-            },
-            {
-                "key_information_data": {
-                    "key_identifier": 1,
-                    "key_version_number": 64,
-                    "key_types": [
-                        {
-                            "type": "tls_psk",
-                            "length": 16
-                        }
-                    ]
-                }
-            }
-        ]
-    }

docs/shell.rst

@@ -68,7 +68,7 @@ Usage Examples
 
    suci-tutorial
    cap-tutorial
-   put_key-tutorial
+
 
 Advanced Topics
 ---------------

pySim-prog.py

@@ -44,6 +44,11 @@ from pySim.legacy.ts_51_011 import EF
 from pySim.card_handler import *
 from pySim.utils import *
 
+from pathlib import Path
+import logging
+from pySim.log import PySimLogger
+
+log = PySimLogger.get(Path(__file__).stem)
 
 def parse_options():
 
@@ -185,6 +190,7 @@ def parse_options():
                       default=False, action="store_true")
     parser.add_argument("--card_handler", dest="card_handler_config", metavar="FILE",
                       help="Use automatic card handling machine")
+    parser.add_argument("--verbose", help="Enable verbose logging", action='store_true', default=False)
 
     options = parser.parse_args()
 
@@ -770,6 +776,9 @@ if __name__ == '__main__':
     # Parse options
     opts = parse_options()
 
+    # Setup logger
+    PySimLogger.setup(print, {logging.WARN: "\033[33m"}, opts.verbose)
+
     # Init card reader driver
     sl = init_reader(opts)
 

pySim-read.py

@@ -46,11 +46,17 @@ from pySim.utils import dec_imsi, dec_iccid
 from pySim.legacy.utils import format_xplmn_w_act, dec_st, dec_msisdn
 from pySim.ts_51_011 import EF_SMSP
 
+from pathlib import Path
+import logging
+from pySim.log import PySimLogger
+
+log = PySimLogger.get(Path(__file__).stem)
+
 option_parser = argparse.ArgumentParser(description='Legacy tool for reading some parts of a SIM card',
                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+option_parser.add_argument("--verbose", help="Enable verbose logging", action='store_true', default=False)
 argparse_add_reader_args(option_parser)
 
-
 def select_app(adf: str, card: SimCard):
     """Select application by its AID"""
     sw = 0
@@ -75,6 +81,9 @@ if __name__ == '__main__':
     # Parse options
     opts = option_parser.parse_args()
 
+    # Setup logger
+    PySimLogger.setup(print, {logging.WARN: "\033[33m"}, opts.verbose)
+
     # Init card reader driver
     sl = init_reader(opts)
 

pySim-shell.py

@@ -107,12 +107,12 @@ Online manual available at https://downloads.osmocom.org/docs/pysim/master/html/
             kwargs = {'include_ipy': True}
 
         self.verbose = verbose
-        self._onchange_verbose('verbose', False, self.verbose);
+        PySimLogger.setup(self.poutput, {logging.WARN: YELLOW})
+        self._onchange_verbose('verbose', False, self.verbose)
 
         # pylint: disable=unexpected-keyword-arg
         super().__init__(persistent_history_file='~/.pysim_shell_history', allow_cli_args=False,
                          auto_load_commands=False, startup_script=script, **kwargs)
-        PySimLogger.setup(self.poutput, {logging.WARN: YELLOW})
         self.intro = style(self.BANNER, fg=RED)
         self.default_category = 'pySim-shell built-in commands'
         self.card = None
@@ -1175,13 +1175,7 @@ if __name__ == '__main__':
     opts = option_parser.parse_args()
 
     # Ensure that we are able to print formatted warnings from the beginning.
-    PySimLogger.setup(print, {logging.WARN: YELLOW})
-    if opts.verbose:
-        PySimLogger.set_verbose(True)
-        PySimLogger.set_level(logging.DEBUG)
-    else:
-        PySimLogger.set_verbose(False)
-        PySimLogger.set_level(logging.INFO)
+    PySimLogger.setup(print, {logging.WARN: YELLOW}, opts.verbose)
 
     # Register csv-file as card data provider, either from specified CSV
     # or from CSV file in home directory

pySim/cdma_ruim.py

@@ -128,7 +128,7 @@ class EF_AD(TransparentEF):
         cell_test = 0x04
 
     def __init__(self, fid='6f43', sfid=None, name='EF.AD',
-                 desc='Administrative Data', size=(3, None), **kwargs):
+                 desc='Service Provider Name', size=(3, None), **kwargs):
         super().__init__(fid, sfid=sfid, name=name, desc=desc, size=size, **kwargs)
         self._construct = Struct(
             # Byte 1: Display Condition

pySim/esim/saip/__init__.py

@@ -151,8 +151,6 @@ class File:
         self.df_name = None
         self.fill_pattern = None
         self.fill_pattern_repeat = False
-        self.pstdo = None # pinStatusTemplateDO, mandatory for DF/ADF
-        self.lcsi = None # optional life cycle status indicator
         # apply some defaults from profile
         if self.template:
             self.from_template(self.template)
@@ -280,8 +278,6 @@ class File:
         elif self.file_type in ['MF', 'DF', 'ADF']:
             fdb_dec['file_type'] = 'df'
             fdb_dec['structure'] = 'no_info_given'
-            # pinStatusTemplateDO is mandatory for DF/ADF
-            fileDescriptor['pinStatusTemplateDO'] = self.pstdo
         # build file descriptor based on above input data
         fd_dict = {}
         if len(fdb_dec):
@@ -308,8 +304,6 @@ class File:
             # desired fill or repeat pattern in the "proprietaryEFInfo" element for the EF in Profiles
             # downloaded to a V2.2 or earlier eUICC.
             fileDescriptor['proprietaryEFInfo'] = pefi
-        if self.lcsi:
-            fileDescriptor['lcsi'] = self.lcsi
         logger.debug("%s: to_fileDescriptor(%s)" % (self, fileDescriptor))
         return fileDescriptor
 
@@ -329,8 +323,6 @@ class File:
         if efFileSize:
             self._file_size = self._decode_file_size(efFileSize)
 
-        self.pstdo = fileDescriptor.get('pinStatusTemplateDO', None)
-        self.lcsi = fileDescriptor.get('lcsi', None)
         pefi = fileDescriptor.get('proprietaryEFInfo', {})
         securityAttributesReferenced = fileDescriptor.get('securityAttributesReferenced', None)
         if securityAttributesReferenced:

pySim/log.py

@@ -63,7 +63,7 @@ class PySimLogger:
         raise RuntimeError('static class, do not instantiate')
 
     @staticmethod
-    def setup(print_callback = None, colors:dict = {}):
+    def setup(print_callback = None, colors:dict = {}, verbose_debug:bool = False):
         """
         Set a print callback function and color scheme. This function call is optional. In case this method is not
         called, default settings apply.
@@ -72,10 +72,20 @@ class PySimLogger:
                              have the following format: print_callback(message:str)
             colors : An optional dict through which certain log levels can be assigned a color.
                      (e.g. {logging.WARN: YELLOW})
+            verbose_debug: Enable verbose logging and set the loglevel DEBUG when set to true. Otherwise the
+                           non-verbose logging is used and the loglevel is set to INFO. This setting can be changed
+                           using the set_verbose and set_level methods at any time.
         """
         PySimLogger.print_callback = print_callback
         PySimLogger.colors = colors
 
+        if (verbose_debug):
+            PySimLogger.set_verbose(True)
+            PySimLogger.set_level(logging.DEBUG)
+        else:
+            PySimLogger.set_verbose(False)
+            PySimLogger.set_level(logging.INFO)
+
     @staticmethod
     def set_verbose(verbose:bool = False):
         """

pySim/transport/__init__.py

@@ -301,24 +301,53 @@ class LinkBaseTpdu(LinkBase):
 
         prev_tpdu = tpdu
         data, sw = self.send_tpdu(tpdu)
+        log.debug("T0: case #%u TPDU: %s => %s %s", case, tpdu, data or "(no data)", sw or "(no status word)")
 
-        # When we have sent the first APDU, the SW may indicate that there are response bytes
-        # available. There are two SWs commonly used for this 9fxx (sim) and 61xx (usim), where
-        # xx is the number of response bytes available.
-        # See also:
+        # After sending the APDU/TPDU the UICC/eUICC or SIM may response with a status word that indicates that further
+        # TPDUs have to be sent in order to complete the task.
         if sw is not None:
-            while (sw[0:2] in ['9f', '61', '62', '63']):
-                # SW1=9F: 3GPP TS 51.011 9.4.1, Responses to commands which are correctly executed
-                # SW1=61: ISO/IEC 7816-4, Table 5 — General meaning of the interindustry values of SW1-SW2
-                # SW1=62: ETSI TS 102 221 7.3.1.1.4 Clause 4b): 62xx, 63xx, 9xxx != 9000
-                tpdu_gr = tpdu[0:2] + 'c00000' + sw[2:4]
-                prev_tpdu = tpdu_gr
-                d, sw = self.send_tpdu(tpdu_gr)
-                data += d
+            if case == 4 or self.apdu_strict == False:
+                # In case the APDU is a case #4 APDU, the UICC/eUICC/SIM may indicate that there is response data
+                # available which has to be retrieved using a GET RESPONSE command TPDU.
+                #
+                # ETSI TS 102 221, section 7.3.1.1.4 is very cleare about the fact that the GET RESPONSE mechanism
+                # shall only apply on case #4 APDUs but unfortunately it is impossible to distinguish between case #3
+                # and case #4 when the APDU format is not strictly followed. In order to be able to detect case #4
+                # correctly the Le byte (usually 0x00) must be present, is often forgotten. To avoid problems with
+                # legacy scripts that use raw APDU strings, we will still loosely apply GET RESPONSE based on what
+                # the status word indicates. Unless the user explicitly enables the strict mode (set apdu_strict true)
+                while True:
+                    if sw in ['9000', '9100']:
+                        # A status word of 9000 (or 9100 in case there is pending data from a proactive SIM command)
+                        # indicates that either no response data was returnd or all response data has been retrieved
+                        # successfully. We may discontinue the processing at this point.
+                        break;
+                    if sw[0:2] in ['61', '9f']:
+                        # A status word of 61xx or 9fxx indicates that there is (still) response data available. We
+                        # send a GET RESPONSE command with the length value indicated in the second byte of the status
+                        # word. (see also ETSI TS 102 221, section 7.3.1.1.4, clause 4a and 3GPP TS 51.011 9.4.1 and
+                        # ISO/IEC 7816-4, Table 5)
+                        le_gr = sw[2:4]
+                    elif sw[0:2] in ['62', '63']:
+                        # There are corner cases (status word is 62xx or 63xx) where the UICC/eUICC/SIM asks us
+                        # to send a dummy GET RESPONSE command. We send a GET RESPONSE command with a length of 0.
+                        # (see also ETSI TS 102 221, section 7.3.1.1.4, clause 4b and ETSI TS 151 011, section 9.4.1)
+                        le_gr = '00'
+                    else:
+                        # A status word other then the ones covered by the above logic may indicate an error. In this
+                        # case we will discontinue the processing as well.
+                        # (see also ETSI TS 102 221, section 7.3.1.1.4, clause 4c)
+                        break
+                    tpdu_gr = tpdu[0:2] + 'c00000' + le_gr
+                    prev_tpdu = tpdu_gr
+                    data_gr, sw = self.send_tpdu(tpdu_gr)
+                    log.debug("T0: GET RESPONSE TPDU: %s => %s %s", tpdu_gr, data_gr or "(no data)", sw or "(no status word)")
+                    data += data_gr
             if sw[0:2] == '6c':
                 # SW1=6C: ETSI TS 102 221 Table 7.1: Procedure byte coding
                 tpdu_gr = prev_tpdu[0:8] + sw[2:4]
                 data, sw = self.send_tpdu(tpdu_gr)
+                log.debug("T0: repated case #%u TPDU: %s => %s %s", case, tpdu_gr, data or "(no data)", sw or "(no status word)")
 
         return data, sw
 

tests/pySim-prog_test/Fairwaves-SIM.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: Fairwaves-SIM
 ICCID: 8988219000000117833

tests/pySim-prog_test/Wavemobile-SIM.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: Wavemobile-SIM
 ICCID: 89445310150011013678

tests/pySim-prog_test/fakemagicsim.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: fakemagicsim
 ICCID: 1122334455667788990

tests/pySim-prog_test/sysmoISIM-SJA2.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmoISIM-SJA2
 ICCID: 8988211000000467343

tests/pySim-prog_test/sysmoISIM-SJA5.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmoISIM-SJA5
 ICCID: 8949440000001155314

tests/pySim-prog_test/sysmoUSIM-SJS1.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmoUSIM-SJS1
 ICCID: 8988211320300000028

tests/pySim-prog_test/sysmosim-gr1.ok

@@ -1,4 +1,4 @@
-Using PC/SC reader interface
+INFO: Using PC/SC reader interface
 Reading ...
 Autodetected card type: sysmosim-gr1
 ICCID: 2222334455667788990