Repository navigation
NGD slowness #2879
Description
Is anyone familiar with how NGD values are computed from the associated (Sqlite?) database? I’ve noticed that the same query run on a system without load (https://arax.ci.transltr.io/?r=529830) versus with a load (https://arax.ci.transltr.io/?r=529801) the exact same query takes ~50 seconds to decorate the NGD edges, versus ~8 minutes respectively. This makes me think:
- There’s plenty of optimization opportunities to make the population of NGD faster (it shouldn’t take 50 seconds to construct ~3k values from an sqlite DB) and
- There may be some sort of lock/race thing going on that would explain the difference in performance under load/no load
QG used:
{
"edges": {
"e0": {
"attribute_constraints": [],
"knowledge_type": "inferred",
"object": "n1",
"predicates": [
"biolink:treats"
],
"qualifier_constraints": [],
"subject": "n0"
}
},
"nodes": {
"n0": {
"categories": [
"biolink:ChemicalEntity"
],
"constraints": [],
"is_set": false
},
"n1": {
"categories": [
"biolink:Disease"
],
"constraints": [],
"ids": [
"MONDO:0005148"
],
"is_set": false
}
}
}
Metadata
Metadata
Assignees
Labels
No labels
Activity
dkoslicki commentedon Sep 2, 2026
PR #2882 makes some improvements to NGD, namely three things:
_get_pmid_set(): build each curie's PMID set once and reuse it, rather than pul leach time._compute_marginal_and_joint_counts()now takes already-materialized sets and returns the joint set, so the intersection happens once instead of twice._order_node_pairs(): process pairs grouped by whichever endpoint has the longer PMID list, so a bounded cache also works for all-pairs overlays, not just the one-pinned-node shape.Future improvements: detect when we are asking for NGD scores for neighboring nodes and use the
curie_ngd*database for a direct lookup. Care will need to be taken since the NGD overlay is built to be able to add NGDs to any pair of nodes, while thecurie_ngd*is specifically only for pairs of nodes that happen to be neighbors/adjacent. Likely inspecting the query graph should inform this, or the naive approach of trying thecurie_ngd*and falling back to the old PMID-based method might work (though could result in a bunch of "cache" misses, though it's rather rare when we want to decorate pairs of nodes with NGD that aren't neighbors).dkoslicki commentedon Sep 2, 2026
Full story of working with Claude (which suggested some truly terrible "improvements" that would have quite negatively impacted ARAX; like "oh, don't compute NGD for answers that aren't returned" not knowing that NGD is how we primarily choose what is returned; and suggesting to just ignore NGD for any pairs of nodes that aren't neighbors....that kind of thing) contained here
dkoslicki commentedon Sep 2, 2026
Performance after deploying is much less impressive than the isolated tests/benchmarks run against the sqlite database itself (which were ~40x faster):
Looks like some result binding is causing additional issues. Working on fixing them
Merge pull request #2884 from RTXteam/issue-2879-round-2
chunyuma commentedon Sep 16, 2026
Relevant to issue #2889.