Repository navigation
Conversation
Replace pyFF's lxml + pyxmlsecurity/xmlsec backend with the pure-Rust pyuppsala (XML parse/serialize/XPath/XSD/XSLT) and pybergshamra (XML signing) libraries, and rewrite the entity-processing hot paths that were quadratic under the old per-entity whole-document XPath model. Backend swap - Dependencies: drop lxml and pyxmlsecurity; add pyuppsala>=0.8.0, pybergshamra>=0.6.4, and cryptography>=42. Python floor raised to 3.10 (drops the eval-type-backport 3.9 shim). - utils.py: import etree/Element/ElementTree from pyuppsala.etree. pyuppsala is safe-by-default (entities are never resolved/expanded), so the old set_default_parser(resolve_entities=False) guard is gone. - Schema/validation: pyuppsala's XMLSchema resolves xsd:import / xsd:include relative to the schema file's own directory, so the lxml ResourceResolver + custom parser plumbing is removed; the composite schema is built with XMLSchema(file=..., lenient=True), lenient matching libxml2/lxml built-in datatype validation (real metadata carries anyURI values with spaces). Validation-error filtering is adapted to pyuppsala's ValidationError shape (message with line/column, no embedded source document), fixing a filter that previously dropped every error. - Signing: enveloped signing goes through pybergshamra.sign_enveloped (with PKCS#11/HSM support via Pkcs11Signer), replacing xmlsec.sign. Certificate inspection (CertInfo / cert_info / cert_dict) is reimplemented on the cryptography x509 API, replacing xmlsec.crypto. Performance: quadratic hot paths rewritten to single native passes - Entity-attribute indexing (_eattribute) and certreport walked the whole document with .xpath() once per entity; each XPath evaluation rebuilt the document-wide attribute/index tables, so the mutate->xpath->mutate loop over N entities was O(N x document) -- catastrophic at eduGAIN (~10k entities) scale. Both now iterate entities directly and walk each (small) entity with native .iter(), and certreport dedupes so each distinct certificate URI is validated exactly once. Optional native GIL-free batch fetcher (off by default) - New config flag use_native_fetcher (PYFF_USE_NATIVE_FETCHER). When on, Resource.reload fetches each frontier round's http(s) URLs in a single pyuppsala.fetch_many call (Rust, GIL released), served to the parser through a requests-compatible response shim (_NativeResponse / case-insensitive headers); failures fall back to the existing url_get path, preserving retry/local-backup semantics. requests_cache is bypassed when on. reload also gains a ThreadPoolExecutor frontier so downloads overlap; max_workers defaults to config.worker_pool_size. - pyuppsala may be built without the net feature (fetch_many absent); the import is optional and the flag no-ops in that case. Tests adjusted for the new backend's error/validation surface (test_pipeline, test_utils).
The XML/crypto backend migration left several pyFF-side hot paths on the generic (serialize / iterate-in-Python) routes even though the released pyuppsala 0.10.0 and pybergshamra 0.8.0 wheels expose native equivalents. This ports them, each guarded by hasattr() with the previous behaviour kept as the fallback for older library versions. - sign: sign the working tree in place via pybergshamra.sign_enveloped_document() on the shared native document handle instead of serializing the whole aggregate, signing the string and re-parsing the result (which doubled peak memory on large aggregates). - check_signature: verify via pybergshamra.verify_document() on the native document instead of dumping the tree to a string first. - stats: use Element.fast_count() for the entity / IdP / SP counts instead of materializing a Python proxy per entity. - has_tag: use fast_has() / fast_count() instead of a Python iter(). - with_entity_attributes: collect (Name, [AttributeValue...]) groups in one native pass with fast_collect_grouped_text(). - store.update: store a detached deepcopy of each entity. pyuppsala element proxies keep their owning document alive, so storing entities straight out of a source feed retained every parsed aggregate and its input buffer. The previously stored copy is unindexed before the new one is indexed. - load: release resource.t after reload; the store owns its own copies and the resource graph only needs the metadata. - Raise etree.MAX_XPATH_NODE_VISITS to 50M. The 100k default is an anti-DoS cap for untrusted input; pyFF runs XPath over its own aggregates, which exceed a million nodes at eduGAIN scale.
The document-native sign and verify shortcuts introduced with the pyuppsala/pybergshamra fastpaths operated on etree.native_document(elt), which is the document that OWNS elt. After `first` (an MDQ single-entity request) the working document is an EntityDescriptor still attached to the selected aggregate, so the signature was inserted under the discarded EntitiesDescriptor root and the emitted entity went out unsigned, without any error. Aggregates, whose root is the document root, were unaffected. Gate both shortcuts on the new pyff.utils.is_document_root(), which checks elt.getroottree().getroot() is elt (a deepcopy'd element has no parent but is not a document root either). Subtrees fall back to the serialize/sign/ reparse path, which is cheap for a single entity. Adds SigningTest::test_signing_single_entity (select one entity -> first -> finalize -> sign -> publish) asserting the signature is the entity's first child in both the working document and the published file, and verifies. Aggregate signing performance is unchanged.
- require pybergshamra 0.8.1 and pyuppsala 0.10.0 - support token label, token serial, and slot ID selectors - fix absolute PKCS11 module path parsing - document Luna HA and Chrystoki configuration - add PKCS11 signing configuration tests
- use the owned XML signing path for PKCS11 keys - preserve entities in signed and published aggregates - retain native document signing for software keys - add regression coverage for HSM signing and aggregate contents
Keep PKCS#11 providers alive for the lifetime of each worker process. This prevents the Luna vendor library from being unloaded while its native background threads are still running. Serialize provider initialization and cache providers by module path and token or slot selector. Continue creating a new session for each signing operation. Add tests for provider reuse, selector isolation, and concurrent initialization.
Guard cleanup_namespaces calls because pyuppsala does not provide this optional lxml API. Skip the pruning-specific test when unsupported. This prevents metadata parsing and watcher callbacks from failing.
Remove compatibility fallbacks for APIs guaranteed by pyuppsala 0.11.0 and pybergshamra 0.9.0. Extend the QName namespace regression test through finalize, sign, and publish using the owned-string signing path used by PKCS11.
Error: src/pyff/builtins.py:5:1: I001 Import block is un-sorted or un-formatted
|
Hi, Thanks for this work. I've now managed to switch my (DEV) metadata pipeline to this version. I have some interesting performance observation: on a simple aggregate-publishing batch run ( load local federation + eduGAIN, publish aggregates for local federation, eduGAIN, plus filtred by IdP/SP), I saw a performance drop that more than doubled the run time ( from 1m11s to 2m50s ). However, the performance improvements in per-entity publishing (MDQ) far outweight that and I saw the runtime for the whole pipeline drop to less then half (from about 13 minutes to 5m50s). Thanks a lot for the improvements on the MDQ side - and wondering if there's anything worth addressing on the aggregate side. Cheers, PS: I'll keep the DEV pipeline running on this and monitor for stability - to confirm this also resolves #315. |
|
Hi @kushaldas , I'm running this branch in my DEV environment, so far without a glitches (except for when the incoming eduGAIN feed was invalid in the recent episode, but that tripped up the PROD pipeline on 2.1.7 as well). And this branch is also running the MDQ publishing in multithreaded mode and so far avoided the race-condition that was happening with lxml (#315 ). One extra thing I tried is EC certificates - which are just plain unsupported in current pyFF code. I got Cheers, |
- pybergshamra dependency to 0.9.1 - Require the released pybergshamra 0.9.1 package. - Refresh the uv lockfile with the new wheel and source distribution.
|
Thanks @kushaldas - this was definitely a performance improvement, my workflow now runs in 2m19s - over 30s improvement! |
I will create a separate PR after this one gets merged. Right now software signing path does not specify signature_method, so pybergshamra defaults to RSA-SHA256. So the output is : |
- Require pybergshamra 0.9.2 and refresh the lockfile. - Remove the temporary sibling checkout override. - Serialize PKCS#11 signing per token and release sessions on errors. - Add regression tests for session cleanup and concurrent signing. - Update dependency documentation and record server benchmarks.
|
I have set up the environment with the new pyff 2.1.8.dev0 and HSM as it is used for eduGAIN feed aggregation. It is a simple aggregate/sign/publish batch. A small batch (only one small federation) total size: 28 takes Full eduGAIN: total size: 10737 real 1m0.352s what is a bit alarming is the difference between real and user time. When I'm using key/certificate from local files, processing speed seems normal, as with pyff 2.1.7, it is for small batch and for full eduGAIN so it seems like a HSM issue? My HSM environment is installed in the same way as inside our docker-luna-client used in eduGAIN production. With pyff 2.1.7 and HSM via docker-luna-client I'm getting: and for full: and very similar results when local files are used in place of HSM. Another cumbersome problem for now is that publisher XSL cannot be used because pyuppsala does not handle it. |
Require pyuppsala 0.13.0 and pybergshamra 0.10.0 and re-resolve both in uv.lock. These releases carry the native DOM validation, the inherited-namespace import, the fragment-splice enveloped signing, and the W3C ML-DSA and SLH-DSA identifier support this branch relies on. filter_invalids_from_document validates the whole document once natively and only falls back to the per-entity pass, which pinpoints and removes offenders, when the document as a whole fails. On the eduGAIN aggregate this replaced about 10,700 serialize-and-validate calls with one in-place validation. entitiesdescriptor appends the collected entities with a single extend. pyuppsala's cross-document append deep-copies each subtree natively, including inherited namespace declarations, and never mutates the source, so the former per-entity loop with an explicit deepcopy was a redundant second clone per entity. Output is byte-identical. Add utils.collect_attr and utils.collect_text, which use pyuppsala's fast_collect_attr and fast_collect_text single native passes when present and fall back to iteration otherwise. guess_entity_software gathers SingleSignOnService and AssertionConsumerService Location attributes through collect_attr instead of two ElementPath scans that materialized every endpoint, and the DomainHint collection uses collect_text. Measured on the benchmark host against the previous candidate: the aggregate job 69.8 s to 46.0 s wall with peak RSS 3.7 GB to 2.7 GB, load720 7.0 s to 5.9 s, sign720 8.0 s to 6.7 s; all signed outputs verify with identical entity counts and the unsigned output is byte-identical apart from the pipeline path.
|
Thanks for the latest performance improvements @kushaldas ! Just testing that in my (DEV) pipeline. The aggregate performance has improved to 1m40s. Still somewhat above I get with pyFF 2.1.7 release (still lxml), 1m12s, but significant improvement relative to this branch before the latest commit (my report above from 3 weeks ago says 2m19s). And my whole pipeline (aggregate + MDQ) runs in 3m26s - that's significant improvement over the lxml implementation (order of magnitude on MDQ) and I'm happy with the performance here. |
|
@majagw Thank you for the numbers. The shape of them (about 19 s of extra wall time with With 2.1.7 you did not see this because signing went through pyeleven in the Could you check and post:
If vtl is slow too, removing the unreachable entry should bring the small batch |
|
Since Kushal found the performance cause outside of this project I will probably merge this branch in the beginning of next week and start to work on a 3.0.0 release. Thanks for all your feedback! |
|
First of all - I'm running tests from one server, where I have docker-luna-client running v.10.9.0-65 and new docker image with HSM settings the same as on the docker-luna-client - Chrystoki.conf is the same and in /usr/safenet/lunaclient/cert/server vlt verify takes ~16-17s on both. nc times also look very similar on both docker instances, the same when nc is run from main host e.g. I've just checked that aggregation/signing/publishing of the whole eduGAIN takes < 1min |
|
I found the numbers I needed to debug. for h in dcoa-hsm2 lla-hsm2 tug-hsm2 dcob-hsm2; do
d=/tmp/luna-$h; mkdir -p $d
sed -e '/ServerName0[0-9]/d' -e '/ServerPort0[0-9]/d' -e '/ServerHtl0[0-9]/d' \
-e "/NetClient = 1;/a\ ServerName00 = $h.sunet.se;\n ServerPort00 = 1792;\n ServerHtl00 = 0;" \
/etc/Chrystoki.conf > $d/Chrystoki.conf
echo "== $h"; ( time ChrystokiConfigurationPath=$d vtl verify ) 2>&1 | grep -e Serial -e Label -e real -e '^ *[0-9]'
done |
|
My results: |
entitiesdescriptor appended the store's shared entity elements straight into each request's new EntitiesDescriptor, trusting a comment that pyuppsala's cross-document append never mutates the source. It does: like lxml it has move semantics for Python objects. The subtree is cloned into the target document, but the source's live element proxies are re-pointed onto the clone and the source node is detached from its previous parent. With pyffd serving requests from several threads, request 2 therefore pulled the entities out of the aggregate request 1 was still finalizing and signing. The loud symptom in production (md-signer-2, serving swamid-2.0) was a pyuppsala panic surfacing as "Already borrowed: PyBorrowMutError" from repoint_subtree. The quiet one was worse: with the response cache disabled, 36 concurrent full-aggregate requests on a test host returned 8 empty aggregates and 28 missing entities, all with HTTP 200. Restore the deepcopy of every entity before the append when copy=True, as pyFF did before the backend migration, so each request works on private elements and the store is never re-pointed or detached. Keep copy=False as the explicit in-document move for the caller that wants it. After the change the same 36 concurrent requests all return the complete aggregate, byte-identical apart from the signature. Add TestEntitiesDescriptor: building two aggregates from the same entity list must leave the first intact and the entities unparented (fails on the old code), and copy=False must still move.
pybergshamra 0.11.0 ships kryptering 0.7.0, which keeps the PKCS#11 module loaded for the life of the process and never unloads it, not even when C_Initialize fails. That removes the Thales Luna segmentation fault when a worker dropped its last provider and created another, so the provider cache in builtins is now a lookup optimisation rather than a safety requirement. Refresh uv.lock and state the new minimum in README.rst and docs. Verified against a Luna network HSM: batch signing with direct-slot and HA client configs, provider drop-and-recreate in one process (crashed with 0.10.0), and pyffd serving 36 concurrent uncached aggregate requests plus 36 single entities with every response complete.
|
Just confirming the latest update works for me all fine - with an improvement of about 1-2s on the aggregate signing. |
This branch has not been deployed
4 participants
Summary
This replaces the lxml and pyxmlsecurity/xmlsec backends with the
Rust-backed pyuppsala and pybergshamra libraries.
The migration also addresses several performance, memory, signing, and
PKCS11 issues discovered while testing production-sized aggregates.
Main changes
and XSLT.
traversal and bulk operations.
responses sign the selected entity rather than its owning aggregate.
loading to avoid retaining complete source aggregates.
PYFF_USE_NATIVE_FETCHER.PKCS11 changes
mutating a live pyuppsala document.
a new session for each signing operation.
vendor modules.
These changes prevent the Luna PKCS11 library from being unloaded while
its background threads are still active and avoid the resulting worker
crashes.
Namespace handling
pyuppsala does not provide lxml's optional
cleanup_namespacesAPI, sothat operation is now capability-guarded.
The QName namespace regression coverage from #333 and #335 has been
extended through
finalize,sign, andpublish, including the ownedXML signing path used for PKCS11.
Compatibility cleanup
The required pyuppsala and pybergshamra versions provide the native
counting, attribute collection, signing, and verification APIs used by
pyFF. Compatibility fallbacks for older library versions have therefore
been removed.
Verification
PYFF_SKIP_SLOW_TESTS=1 uv run pytest -q