Skip to content

Address review: CopraRNA2, IntaRNAsTar, compressed genomes and vector reports - #3

Merged
martin-raden merged 34 commits into
issue-1-python3-rewritefrom
issue-1-python3-bugfixes
Oct 26, 2026
Merged

martin-raden merged 34 commits into
issue-1-python3-rewritefrom
issue-1-python3-bugfixes

Conversation

@Alexander-Mitrofanov

@Alexander-Mitrofanov Alexander-Mitrofanov commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Modernizes the Python follow-up for #1 after Martin Raden’s review, stacked on #2. The runtime now supports CopraRNA2 only and uses IntaRNA >=3 with the IntaRNAsTar personality. File or inline native parameter contents are forwarded to every prediction, including web exports.

Genomes are downloaded and parsed as gzip, shared through a locked database directory, and read directly without per-job copies. IntaRNA CSV intermediates stay compressed. Gap-free helices spanning at least 90% of either effective RNA length are removed before energy fitting, with an audit of removed IDs and coordinates. Prediction cache signatures cover native settings and input contents. The original input FASTA stays byte unchanged at its original path; cleanup also protects supplied native parameter files, their reproducibility snapshot and nested inputs.

The web export applies the same helix filter and selects a retained interaction, preventing discarded best hits from reappearing in diagrams. Scientific diagrams now have vector PDF/SVG exports, all extended-list pages are retained, and the phylogeny has a Python-rendered SVG. Reports handle unavailable comparative scores. The current dependency environment uses BLAST+ and reviewed Python releases; separate optional Perl/R oracle environments avoid conflicting historical XML libraries.

The earlier parsing, statistical, DomClust and plotting corrections remain documented in ChangeLog.md. Function documentation explains scientific conventions and file side effects, and AGENTS.md records the maintenance requirements. Original tracked sources remain byte unchanged. Historical whole-workflow equality is intentionally replaced by explicit contracts for the reviewed scientific defaults; retained helper/statistical oracles still match independently captured output.

Validation: 171 passed, 2 skipped with native workflows, file/inline equivalence, a real perfect-helix prediction, browser vector exports, R/Perl comparisons and all 17 bundled reference genomes (51 region comparisons). The skips cover the intentionally updated archive README oracle and an unavailable optional legacy NCBI module. A fresh production environment installs successfully; both optional interpreter specifications solve. Wheel/sdist builds pass, all 35 retained fixture cases pass from an installed wheel outside the checkout, and that wheel completes a native prediction with web exports. All four CI jobs pass on final commit a3977c8 (Python 3.11–3.13 and the full scientific environment), for both push and pull-request runs.

Follow-up completeness audit (2026-10-22): rechecked the original issue and every review/comment endpoint. Closed input/output basename and symlink collisions, protected cleanup destinations, enforced IntaRNA >=3 at every standalone native entry point, and made native cache signatures use the detected release. Completed the remaining public API documentation and added a requirement-by-requirement implementation/test table to AGENTS.md. All listed requests are covered; original tracked source checksums remain unchanged.

@martin-raden

martin-raden commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

@Alexander-Mitrofanov

thanks, please revise:

some of the following tasks might already be implemented in the current code. double check and ignore if that's the case.

drop CopraRNA1 legacy code

within the reimplementation, we will remove the CopraRNA1 old version legacy code

  • remove all code parts within the new implementation that are needed for the old CopraRNA1 mode
  • remove respective documentation and test cases
  • try to simplify the code base for the CopraRNA2-only implementation

update dependencies and their integration

  • ensure IntaRNA version >= 3 is used
    • update docu and respective conda setups and yaml files if existing
  • enable that an IntaRNA parameter file is provided for the IntaRNA calls and that it is forwarded to the IntaRNA calls
    • also support string-encoded input of the parameter file (instead of a real file) as respective input and create a temporary parameter file in that case for subsequent IntaRNA calls
  • use IntaRNA CSV output mode (instead of its legacy intarna1 output emulator) as the intarna output format and parse and process this
  • remove IntaRNA parameters from the CopraRNA CLI (if present), since they can be set via the intarna parameter argument from above
  • check if there are updated versions of other dependencies and check whether the updated UI is compatible with CopraRNA2
  • call "IntaRNAsTar" personality per default instead of standard IntaRNA
  • make use of gzipped intermediate files and their direct writing/reading/processing
    • e.g. IntaRNA supports gzipped CSV output

"Phylogeny/compatible.treefile"

if the new implementation still produces the file Phylogeny/compatible.treefile, generate an SVG rendering.
within the Freiburg RNA Tools webserver this is done using

nw_display -S -w 500 -v 20 -s Phylogeny/compatible.treefile > Phylogeny/compatible.treefile.svg

check for a python only solution to reduce the dependency list to python packages. if not possible, refer to the given tool

images

  • avoid postscript (ps) output file formats and use PDF or SVG instead
  • when diagrams are created in PDF format (eg. sRNA_conservation_heatmap.pdf) also produce an SVG version
  • if pixelgraphics are produced, use PNG file format, but avoid if vector graphics are possible

input fasta file

the original code moves the input fasta file to a subfolder.

  • dont do, only copy the file (if needed) but keep the original file in place

keep genome files compressed

  • if genome files are available compressed for download use the compressed ones and rewrite the scripts such that they operate directly on the compressed files without explicit full decompression if possible
  • if they are not downloadable compressed, do so directly during download and continue working on the compressed files if possible
  • add support for a local "genome database directory" in which all downloaded genome files are stored in compressed version and searched in before downloading
    • if possible directly process/read the genome files from there without copying

full complementarity check

  • sometimes a queries gene is part of the target list. thus the query and this target will be predicted as a full length (accessibility window or query length) perfect helix (stacking only interaction)
  • such artificial hits can distort subsequent computations like p-value estimates
  • thus, identify and remove interaction pairs from IntaRNA outputs that show predicted interactions that span 90% of either query or target without any bulge or interior loop (stacking only) (or use a similar condition that seems more appropriate)

code documentation

  • extend the new code base with a thorough inline and function documentation to easy a human review
  • add respective directives to the AGENTS.md file

ChangeLog file

add a "ChangeLog.md" file that provides the needed documentation to compile the next release change information:

@Alexander-Mitrofanov Alexander-Mitrofanov changed the title Fix legacy prediction, parsing, statistics and plotting defects with regressions Address review: CopraRNA2, IntaRNAsTar, compressed genomes and vector reports Oct 21, 2026
Alexander-Mitrofanov and others added 6 commits October 23, 2026 15:32
Fix formatting and punctuation in README.md regarding DAVID service requirements and report generation.
Updated email account for DAVID requests and clarified CSV import handling.
…ntext

Fix genomic target accessibility boundaries
Prepare the Python-only CopraRNA 3.0.0 release
@martin-raden
martin-raden merged commit 3894a18 into issue-1-python3-rewrite Oct 26, 2026
8 checks passed
@martin-raden
martin-raden deleted the issue-1-python3-bugfixes branch October 26, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants