This helped in figuring out some bug -- show all available context info
in raised exceptions.
Change-Id: I38e85ce0902f2ec4db63e158770b6d2cecb2dd30
Jenkins: skip-card-test
Allow returning text/plain Content-Types as 'data' output argument.
So far, all the esim/http_json_api functions require a JSON response.
However, a specific vendor has a list function where the request is JSON
but the response is text/plain CSV data. Allow and return in a dict.
Change-Id: Iba6e4cef1048b376050a435a900c0f395655a790
Jenkins: skip-card-test
while len(foo):
throws an exception when foo == None.
Instead doing
while foo:
fixes a problem when reading in empty SUCI calc info data, e.g. from
TS48v7.0_SAIP2.3_BERTLV_SUCI_NoRAMRFM.der.
Change-Id: Ia4e2356d0241d7a6ca399ba7e8be7f27ec836104
Jenkins: skip-card-test
The tests take long and don't need to be this many.
A patch is coming up that adds another profile to test SUCI in GFM,
this patch makes some room for that.
Change-Id: Ib75b6919a3acfddd99bf9baa9b6847ef731b9e67
Jenkins: skip-card-test
Particular reason: when manipulating the 5G SUCI parameters, the
mandatory services get-identity, profile-a-x25519 and profile-b-p256 may
need to be reconfigured.
In general, it is a good idea to run these checks anyway.
Change-Id: I5e6eef0f1845a25cddb03af8d16c40e305bcdc1f
Jenkins: skip-card-test
Add a new ConfigurableParameter that represents the MNC length
(2 or 3 digits) in EF.AD (Administrative Data).
Change-Id: I6c600faeab00ffb072acbe94c9a8b2d1397c07d3
Co-authored-by: Vadim Yanitskiy <vyanitskiy@sysmocom.de>
Jenkins: skip-card-test
By default, numeric_base = None, to indicate that there are no explicit
limitations on the number space.
For parameters that are definitely decimal, set numeric_base = 10.
For definitely hexadecimal, set numeric_base = 16.
Do the same for ConfigurableParameter as well as ParamSource, so callers
can match them up: if a parameter is numeric_base = 10, then omit
sources that are numeric_base = 16, and vice versa.
Change-Id: Ib0977bbdd9a85167be7eb46dd331fedd529dae01
Jenkins: skip-card-test
apply_val() was re-encoding the SMSP with the minimum total_len of 28,
which produces a 28-byte body with no alpha_id field. After a DER
round-trip, the profile machinery re-pads the body to the original
record length using the template's fill pattern, which may not be 0xFF.
Those non-0xFF fill bytes end up in the alpha_id area, and GSM 7-bit
decoding then fails with a KeyError when the modified profile is read
back.
Fix by:
- setting alpha_id = '' so the field is present but empty
- setting f_smsp.rec_len = 42 (28 fixed bytes + 14 bytes of alpha_id
padding) so the re-encoded body carries 0xFF-padded alpha_id space
and the efFileSize in the fileDescriptor stays consistent
- passing total_len=f_smsp.rec_len to encode_record_bin() so the
alpha_id area is actually padded to that length
Change-Id: Ief6e02517f3e96158a2509d763b88aec4bd5a296
Jenkins: skip-card-test
validate_val() calls len() to check the value against allow_len,
min_len and max_len. len() requires the object to have a __len__()
method, which integers do not — calling len() on an int raises
TypeError.
Fix this by checking for __len__ first: if present, use len(val) as
usual; otherwise fall back to len(str(val)), which gives the number
of decimal digits for integer values.
Change-Id: Ibe91722ed1477b00d20ef5e4e7abd9068ff2f3e4
Jenkins: skip-card-test
secrets.SystemRandom is defined as the most secure random source
available on the given operating system.
Change-Id: I8049cd1292674b3ced82b0926569128535af6efe
Jenkins: skip-card-test
Python's random module uses a PRNG (Mersenne Twister) which is
utterly insecure for key generation - it was so far only used for
testing. Replace it with random.SystemRandom(), which draws from
/dev/urandom and is suitable for generating cryptographic key material.
Change-Id: I6de38c14ac6dd55bc84d53974192509c18d02bfa
Jenkins: skip-card-test
Add ConfigurableParameterTest, which applies each parameter to a real
UPP DER template and reads it back, comparing results against a stored
expected-output snapshot (xo/test_configurable_parameters).
Add TestValidateVal covering validate_val() for Iccid, Imsi, Pin1, Puk1
and K, testing both valid inputs and invalid ones expected to raise
ValueError.
Add TestEnumParam covering the EnumParam methods (validate_val,
map_name_to_val, map_val_to_name, name_normalize, clean_name_str) using
AlgorithmID as the concrete subclass, including fuzzy name matching.
Also add get_value_from_pes() to ConfigurableParameter as a convenience
wrapper around get_values_from_pes() that asserts all returned values
are identical and returns the single result.
Change-Id: Ia55f0d11f8197ca15a948a83a34b3488acf1a0b4
Co-authored-by: Vadim Yanitskiy <vyanitskiy@sysmocom.de>
Jenkins: skip-card-test
The ClassVarMeta metaclass used to derive each ConfigurableParameter's
'name' attribute automatically from the Python class name (via
camel_to_snake()). Stop doing this, for three reasons:
1) Python class names follow constraints that do not fit the naming
commonly used in CSV files. For example, a name like
"5GS-SUCI-CalcInfo" starts with a digit and contains dashes,
neither of which is permissible in a class name.
2) Python class names live in their own namespace, distinct from the
one used to present eSIM parameters to end users. Deriving the UI
name from the class name couples these two namespaces together.
Taken together, (1) and (2) mean that automatic naming both imposes
class-name constraints on the user-visible names and merges the
internal Python namespace with the publicly shown one - a layer
violation from the perspective of UI design.
3) Overriding 'name' from __new__() makes manual naming impossible: a
subclass that sets 'name = "bar"' as a class attribute would still
end up with the value computed by the metaclass, which is
surprising and hard to track down:
class MySuper(metaclass=...): # __new__ sets name = 'foo'
...
class MySub(MySuper):
name = 'bar'
print(MySub().name) # 'foo', not 'bar' as one would expect
Change-Id: I6f631444c6addeb7ccc5f6c55b9be3dc83409169
Jenkins: skip-card-test
By a flag, allow to audit also all Security Domain KVN that we have
*not* created ConfigurableParameter subclasses for.
For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only
Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn
0x04..0x0f in an audit.
This patch includes audits of all SD key kvn there may be in the UPP.
This will help to spot SD keys that may already be present in a UPP
template, with unexpected / unusual kvn.
Change-Id: Icaf6f7b589f117868633c0968a99f2f0252cf612
Jenkins: skip-card-test
'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.
So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.
Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.
Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.
(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)
Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52
Jenkins: skip-card-test
The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.
Change-Id: I2568a032167a10517d4d75d8076a747be6e21890
Jenkins: skip-card-test
The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.
Implement the enum value part abstractly in new EnumParam.
Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn
Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240
Jenkins: skip-card-test
Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.
This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.
For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.
BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:
my_source = MyParam.default_source.from_str( MyParam.default_value )
Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32
Jenkins: skip-card-test
Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.
For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Jenkins: skip-card-test
Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.
Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.
Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a
Jenkins: skip-card-test
When I open the .asn file in vim, pySim should not attempt to read the
vim .swp file as asn.1.
File "/home/moi/osmo-dev/src/pysim/pySim/esim/saip/__init__.py", line 45, in <module>
asn1 = compile_asn1_subdir('saip')
[...]
File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 21: invalid start byte
Related: OS#6937
Change-Id: I37df3fc081e51e2ed2198876c63f6e68ecc8fcd8
For all ConfigurableParameter subclasses, provide an example_input.
This may be useful for downstream projects' user interaction, to suggest
a value or prefill an input field, as appropriate.
Related: SYS#6768
Change-Id: I2672fedcbc32cb7a6cb0c233a4a22112bd9aae03
These names better match what humans expect to read, for example "PIN1"
instead of "Pin1".
(We still fall back to the __class__.__name__ if a subclass omits a
specific name, see the ConfigurableParameter init.)
Change-Id: I31f390d634e58c384589c50a33ca45d6f86d4e10
Refactor SdKey (and subclasses) to the new ConfigurableParameter
implementation style, keeping the same implementation.
But duly note that this implementation does not work!
It correctly patches pe.decoded[], but that gets overridden by
ProfileElementSD._pre_encode().
For a fix, see I07dfc378705eba1318e9e8652796cbde106c6a52.
Change-Id: I427ea851bfa28b2b045e70a19a9e35d361f0d393
Refactor AlgorithmID, K, Opc to the new ConfigurableParameter
implementation style.
K and Opc use a common abstract BinaryParam.
Note from the future: AlgorithmID so far takes "raw" int values, but
will turn to be an "enum" parameter with predefined meaningful strings
in I71c2ec1b753c66cb577436944634f32792353240
Change-Id: I6296fdcfd5d2ed313c4aade57ff43cc362375848
Implement abstract DecimalHexParam, and use it to refactor Puk1 and Puk2
to the new ConfigurableParameter implementation style.
DecimalHexParam will also be used for Pin and Adm soon.
Change-Id: I271e6c030c890778ab7af9ab3bc7997e22018f6a
Main points/rationales of the refactoring, details below:
1) common validation implementation
2) offer classmethods
The new features are optional, and will be heavily used by batch
personalization patches coming soon.
Implement Iccid and Imsi to use the new way, with a common abstract
DecimalParam implementation.
So far leave the other parameter classes working as they always did, to
follow suit in subsequent commits.
Details:
1) common validation implementation:
There are very common validation steps in the various parameter
implementations. It is more convenient and much more readable to
implement those once and set simple validation parameters per subclass.
So there now is a validate_val() classmethod, which subclasses can use
as-is to apply the validation parameters -- or subclasses can override
their cls.validate_val() for specialized validation.
(Those subclasses that this patch doesn't touch still override the
self.validate() instance method. Hence they still work as before this
patch, but don't use the new common features yet.)
2) offer stateless classmethods:
It is useful for...
- batch processing of multiple profiles (in upcoming patches) and
- user input validation
to be able to have classmethods that do what self.validate() and
self.apply() do, but do not modify any self.* members.
So far the paradigm was to create a class instance to keep state about
the value. This remains available, but in addition we make available the
paradigm of a singleton that is stateless (the classmethods).
Using self.validate() and self.apply() still work the same as before
this patch, i.e. via self.input_value and self.value -- but in addition,
there are now classmethods that don't touch self.* members.
Related: SYS#6768
Change-Id: I6522be4c463e34897ca9bff2309b3706a88b3ce8
cmd2 version 3.0 was released, with significant API changes. Limit the
dependency to below 3.0, as already reflected in requirements.txt.
Seeing but not changing the discrepancy in minimum version:
requirements.txt has >2.6.2 while setup.py has >= 1.5.0.
Related: SYS#7775 SYS#7777
Change-Id: I5186f242dbc1b770e3ab8cdca7f27d2a1029fff6
pre-empt this from coming up in patch
I60ea8fd11fb438ec90ddb08b17b658cbb789c051:
E1135: Value 'self.permitted_len' doesn't support membership test (unsupported-membership-test) pickermitted_len
Change-Id: I0343f8dbbffefb4237a1cb4dd40b576f16111073
Add PEM version of smdpp-data/certs/DPtls/CERT_S_SM_DP_TLS_NIST.der
A CERT_S_SM_DP_TLS_NIST.pem file is referenced in docs/osmo-smdpp.rst --
nginx apparently cannot use DER certs, so it is convenient for beginners
if the example from the docs just works without having to know that:
The added file was produced using
openssl x509 -inform DER -in CERT_S_SM_DP_TLS_NIST.der -outform PEM -out CERT_S_SM_DP_TLS_NIST.pem
Change-Id: I41ba6ebacb71df0eb8a248c0c3c9ccd709718d74
These changes are necessary to successfully run
./tests/unittests/test_esim_saip.py with a pySim installed via
'pip install'.
For example:
virtualenv venv
source venv/bin/activate
git clone ssh://gerrit.osmocom.org:29418/pysim
pip install pysim/
cd pysim
./tests/unittests/test_esim_saip.py
Before this patch, that would result first in package pySim.esim.saip
being unknown (not installed at all), and when that is added to
setup.py, in this error:
Traceback (most recent call last):
File "/home/moi/osmo-dev/src/pysim/tests/unittests/./test_esim_saip.py", line 23, in <module>
from pySim.esim.saip import *
File "/home/moi/s/esim/sysmo_esim_mgr/venv/lib/python3.13/site-packages/pySim/esim/saip/__init__.py", line 41, in <module>
asn1 = compile_asn1_subdir('saip')
File "/home/moi/s/esim/sysmo_esim_mgr/venv/lib/python3.13/site-packages/pySim/esim/__init__.py", line 56, in compile_asn1_subdir
for i in resources.files('pySim.esim').joinpath('asn1').joinpath(subdir_name).iterdir():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/pathlib/_local.py", line 577, in iterdir
with os.scandir(root_dir) as scandir_it:
~~~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/moi/s/esim/sysmo_esim_mgr/venv/lib/python3.13/site-packages/pySim/esim/asn1/saip'
After this patch, the test completes successfully.
......
----------------------------------------------------------------------
Ran 6 tests in 0.067s
OK
Related: sysmocom's eSIM manager product that is currently in
development needs to fully use pySim.esim.saip, ideally from a regular
'pip install', and not from using the pySim source tree directly.
Related: SYS#6768
Change-Id: I0d7d6962a308eccca589a42c22546d508ff686f5
Neels Hofmeyr