A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic. See the
related commits in osmo-ci.git.
Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0
Previously, _test_de_encode vectors for TransRecEF subclasses were tested
via decode_record_hex()/encode_record_hex(), i.e. one record at a time,
with the decoded value being a scalar.
Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() /
encode_hex() instead, so that vectors represent whole-file content
(decoded value is a list of records) -- consistent with how LinFixedEF
handles _test_de_encode. Update all existing vectors accordingly.
Change-Id: I4a9610f9ee39833cd0c90f64f89f5fbdd6f0846d
When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'. If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.
Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899
A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it. Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.
Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e
When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.
The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:
* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result
Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900
Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.
The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011). For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.
docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.
Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.
Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316
The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`,
`pySim.esim.http_json_api`) unconditionally import optional server-side
dependencies at module level:
pySim.esim.es2p -- from klein import Klein
pySim.esim.http_json_api -- from twisted.web.server import Request
Both imports fail during a docs build if the packages are absent or
broken, causing three "autodoc: failed to import" warnings and three
missing chapters in the generated manual.
Even when klein and twisted are installed, twisted 23.10.0 (the
version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0`
constraint) is incompatible with Python 3.13+ because twisted.web.http
unconditionally executes `import cgi`, a module that was removed from
the standard library in Python 3.13.
Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py.
Sphinx inserts mock entries into sys.modules before each autodoc import
attempt, so the modules can be imported and documented without requiring
the real packages to be importable at build time.
Change-Id: I71650466f02a6a6d150650deed167c05d2cb6e64
sphinxarg.ext generates generic sub-headings ("Named arguments",
"Positional arguments", "Sub-commands", "General options", ...) for
every argparse command and tool. These repeat across many files and
trigger large numbers of autosectionlabel duplicate-label warnings.
Two-pronged fix:
* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings
(sub-headings inside each individual command block).
* `suppress_warnings` per file silences the residual depth-3 collisions
("serial reader", "decode_hex", "sub-commands", ...) that still
appear across tool documentation files.
Cross-references into these generic argparse-generated sections are not
a supported use-case, so suppressing the warnings is appropriate.
Change-Id: I9cdf2a4f6cbd435b16b90ab668205600ffd7c3b0
The "ADM PIN" and "SCP02 / SCP03" sub-sections of "Field naming" used
the '~' heading character, which Sphinx resolved to level 4 - skipping
level 3 and throwing build ERRORs. As a result, both sub-sections
had no heading at all. Change both to '^' (level 3) to match the
other sub-sections in this file.
While at it, fix a typo: "consisting if" -> "consisting of".
Change-Id: Ia56efc7fadcc0fd62e87e63850b929d2f80851ba
gen_install_parameters() had contradictory logic: the outer guard
required all three arguments to be non-None/non-empty (making them
mutually inclusive), while the inner checks then treated each one
as optional.
Make each parameter independently optional (defaulting to None) and
remove the all-or-nothing check. Simplify the function body to a
straightforward single-pass construction of system_specific_params.
Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f
pySim-shell currently does not work on systems with Python 3.14+:
File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
raise ValueError('argument groups cannot be nested')
ValueError('argument groups cannot be nested')
The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group. argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.
Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness enforcement manually in do_install_cap().
Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579
`members_by_name` is a plain dictionary. Calling it with `()` raises:
TypeError: 'dict' object is not callable
Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c
* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
but the error message prints `otak.algo_auth`. Should be
`otak.algo_crypt` instead.
* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
prints `algo_crypt`. Should be `otak.algo_auth` instead.
Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895
* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3
Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8
Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA. Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.
Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e
The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block. It should return the accumulated response instead.
Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2
The keyword argument should be `nested=`. As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.
Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0
The attribute name is misspelled. The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.
Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed
Recent versions of cmd2 have changed how the 'prog' attribute is
automatically set for ArgumentParser instances. As a result, we
are now seeing an unexpected 'build.py' artifact appearing in
the generated documentation.
Let's use an older release of cmd2, which retains the old expected
behavior. Use it specifically for building documentation.
Change-Id: Ifbad35adc5e9d3141acfd024d7dee2a25f1cb62e
Related: https://github.com/python-cmd2/cmd2/issues/1414
Related: OS#6776
According to [1], the literal block must be indented (and, like all
paragraphs, separated from the surrounding ones by blank lines).
[1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks
While at it, fix tabs-vs-spaces: use 2 spaces like in other places.
Change-Id: If548bf66339433c1f3f9e2a557821e808c6afa26
Use the 'r' (raw) qualifier to avoid rendering '\n' as the actual
line break in the auto-generated documentation.
Change-Id: Ie7f59685a78534eb2c43ec4bc39685d3fd264778
The idea of this new job is to catch package integrity problems,
like the missing entries in setup.py/packages[] or missing deps.
Change-Id: Ic72d58494e8fd0cab8d66ce60f7b70593b770872
Related: osmo-ci.git I9d4d9e9de2b16a4b745791f3c9c93507f43bfa6d
pySim-trace.py is broken if pySim is installed using setup.py:
fixeria@DELL:~$ pySim-trace.py
Traceback (most recent call last):
File "/usr/bin/pySim-trace.py", line 8, in <module>
from pySim.apdu import *
ModuleNotFoundError: No module named 'pySim.apdu'
Change-Id: I371143cb4009db46275ec7a020497b909dcc3b4e
Depending on the card type (SIM or USIM/ISUM), self.cla_byte may
be either 0xa0 or 0x00. Sending RUN GSM ALGORITHM with CLA=0x00
fails with SW=6985 (Command not allowed), so let's make sure
that we always use CLA=0xa0 regardless of the card type.
Change-Id: Ia0abba136dbd4cdea8dbbc3c4d6abe12c2863680
With this version I can get all unittests passing:
python -m unittest discover tests/
We're passing argument 'path' to stream_read_entire(), which was
added in [1] and become available since v2.9.51.
Change-Id: I4223c83570d333ad8d79bc2aa2d8bcc580156cff
Related: [1] bfe71315b027e18e62f00ec4de75043992fd2316 construct.git
Related: OS#5666
Argument 'signed' was added in [1] and become available since v2.10.63.
Therefore using bytes2integer() and integer2bytes() from construct.core
bumps the minimum required version of construct to v2.10.63. For
instance, debian:bullseye currently ships v2.10.58.
There is no strict requirement to use construct's API, so let's use
Python's API instead. This allows using older construct versions
from the v2.9.xx family.
Change-Id: I613dbfebe993f9c19003635371941710fc1b1236
Related: [1] 660ddbe2d9a351731ad7976351adbf413809a715 construct.git
Related: OS#5666
-x Print commands and their arguments as they are executed
-e Exit immediately if a command exits with a non-zero status
Change-Id: I13af70ef770936bec00b050b6c4f988e53ee2833
Use multiple processes to speed up pylint. Specifying -j0 will
auto-detect the number of processors available to use.
On AMD Ryzen 7 3700X this significantly reduces the exec time:
$ time python -m pylint -j1 ... pySim *.py
real 0m12.409s
user 0m12.149s
sys 0m0.136s
$ time python -m pylint -j0 ... pySim *.py
real 0m5.541s
user 0m58.496s
sys 0m1.213s
Change-Id: I76d1696c27ddcab358526f807c4a0a7f0d4c85d4
pylint v2.15 is crashing, let's fall-back to a known to work v2.14.5.
Change-Id: Ie29be6ec6631ff2b3d8cd6b2dd9ac0ed8f505e4f
Related: https://github.com/PyCQA/pylint/issues/7375
Related: OS#5668
Before this patch:
$ ./pySim-shell.py -p 0
Card reader initialization failed with an exception of type:
<class 'pySim.exceptions.ReaderError'>
after:
$ ./pySim-shell.py -p 0
Card reader initialization failed with exception:
No reader found for number 0
Change-Id: Id08c4990857f7083a8d1cefc90ff85fc20ab6fef
Let's just use the scope limited TemporaryDirectory() instead, so
the temporary directory will be removed by Python automatically.
pySim/filesystem.py:679:16: E0602: Undefined variable 'shutil' (undefined-variable)
Change-Id: I4ea833fd79f4342c33899124379be509ba1e35ed
pySim/transport/__init__.py:86:15: E1101:
Instance of 'LinkBase' has no '_send_apdu_raw' member;
maybe 'send_apdu_raw'? (no-member)
Change-Id: I14fcdceca5d1e35491b6ad98f96b4276b69b2fc1
This method has been removed [1] in Python 3.0:
pySim/cards.py:581:14: E1101: Instance of 'dict' has no 'iteritems' member (no-member)
pySim/cards.py:591:24: E1101: Instance of 'dict' has no 'iteritems' member (no-member)
[1] https://wiki.python.org/moin/Python3.0#Built-In_Changes
Change-Id: Iba7ad9ed2a9b197ecedaaed1c6744fe1c721515a
Fixes the following pylint's warnings:
pySim/cards.py:494:18: E1101: Class '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:509:6: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:529:26: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:537:5: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:547:5: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:548:8: E1101: Instance of '_MagicSimBase' has no '_ki_file' member (no-member)
pySim/cards.py:559:26: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:560:11: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
pySim/cards.py:576:14: E1101: Instance of '_MagicSimBase' has no '_files' member (no-member)
Change-Id: I4db9d21258d6e04140962134c540e36631466322
pySim/card_key_provider.py:67:2: E1111:
Assigning result of a function call, where the function
has no return (assignment-from-no-return)
Change-Id: I43bab69f53300fbe837944735cd999fab5405d7a
Having lists and dictionaries as default argument values is a bad
idea, because the same instance of list/dict will be used by all
objects instantiated using such constructor:
def appendItem(itemName, itemList=[]):
itemList.append(itemName)
return itemList
print(appendItem('notebook'))
print(appendItem('pencil'))
print(appendItem('eraser'))
Output:
['notebook']
['notebook', 'pencil']
['notebook', 'pencil', 'eraser']
Change-Id: I83d718ff9c3ff6aef47930f38d7f50424f9b880f
The bug that was attempted to be fixed in [1] actually was in the
encoding API - pySim.utils.enc_plmn(). According to 3GPP TS 31.102,
which points to TS 24.008, the three-digit (E)HPLMN shall be encoded
as shown below (ASCII-art interpretation):
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| MCC Digit 2 | MCC Digit 1 |
+---+---+---+---+---+---+---+---+
| MNC Digit 3 | MCC Digit 3 |
+---+---+---+---+---+---+---+---+
| MNC Digit 2 | MNC Digit 1 |
+---+---+---+---+---+---+---+---+
while pySim.utils.enc_plmn() would produce the following:
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| MCC Digit 2 | MCC Digit 1 |
+---+---+---+---+---+---+---+---+
| MNC Digit 1 | MCC Digit 3 |
+---+---+---+---+---+---+---+---+
| MNC Digit 3 | MNC Digit 2 |
+---+---+---+---+---+---+---+---+
Initially the _decoding_ API was correct, but then got changed in
[1] to follow buggy pySim's encoding API. As a result, a (E)HPLMN
programmed with pySim-prog.py would look correct if verified by
pySim-read.py, but the actual file content would be wrong.
This situation shows that our 'program-read-match' build verification
approach alone is insignificant. The lack of unit test coverage,
at least for the core parts of the project, makes it possible to have
symmetrical bugs in both encoding and decoding API parts unnoticed.
This problem was found while trying to enable dead unit tests in [3].
Change [1] that introduced a symmetrical bug is reverted in [2].
Change-Id: Ic7612502e1bb0d280133dabbcb5cb146fc6997e5
Related: [1] I799469206f87e930d8888367890babcb8ebe23a9
Related: [2] If6bf5383988ad442e275efc7c5a159327d104879
Related: [3] I4d4facfabc75187acd5238ff4d0f26022bd58f82
This reverts commit bdf3d3597b, which
broke pySim.utils.dec_mnc_from_plmn(). According to 3GPP TS 31.102,
which points to TS 24.008, the three-digit EHPLMN shall be encoded
as shown below (ASCII-art interpretation):
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| MCC Digit 2 | MCC Digit 1 |
+---+---+---+---+---+---+---+---+
| MNC Digit 3 | MCC Digit 3 |
+---+---+---+---+---+---+---+---+
| MNC Digit 2 | MNC Digit 1 |
+---+---+---+---+---+---+---+---+
So the original implementation was correct, and we even had a unit
test for it. Most likely, the SIM card itself was programmed
incorrectly?
Makes 'testDecMNCfromPLMN_threeDigitMNC' pass again.
Change-Id: If6bf5383988ad442e275efc7c5a159327d104879
As it turns out, we had this set of unit tests since 2018, but
so far they were not executed during the build verification.
Let's fix this:
* run unittest in discovery mode for all files in 'tests/' (commented out);
* rename this file, so it can be automatically detected and executed;
* properly import the API to be tested.
Currently 2 out of 16 unit tests are failing, so we need to get
them passing first and then uncomment the unittest execution.
Change-Id: I4d4facfabc75187acd5238ff4d0f26022bd58f82
This commit fixes two problems (found by semgrep):
* "'foo' and 'bar' in list" is incorrect, because it's interpreted
as "'foo' and ('bar' in list)". Strings with a non-zero length
evaluate to True, thus it's True if at least 'bar' is present.
* Copy-pasted 'E-UTRAN NB-S1' checked two times.
The first condition is redundant, and the whole block can be
re-implemented using two independent 'if' statements.
Change-Id: Iceb66160cfb571db8879d3810c55d252c763d320
Vadim Yanitskiy