Skip to content

Fix DB symlink "File exists" failure and generate-db-symlinks.sh drift - #2901

Open
bazarkua wants to merge 1 commit into
masterfrom
fix-db-symlinks
Open

bazarkua wants to merge 1 commit into
masterfrom
fix-db-symlinks

Conversation

@bazarkua

@bazarkua bazarkua commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

This PR fixes two problems in the ARAX database tooling. They were found in the 2026-10-01 audit of arax.ncats.io.

  1. The database manager crashes when a database symlink already exists.
  2. generate-db-symlinks.sh links files that config_dbs.json does not use anymore.

1. Database manager crashes with "File exists"

On master, symlink_database() runs a plain ln -s. If something is already at the link path, ln fails with File exists and the whole run stops with CalledProcessError.

This happens when the links on disk and db_versions.json do not agree. Three cases I reproduced:

  • A link points to a file that was moved or renamed. This check uses os.path.exists(), which returns False for a broken link. So the manager tries to create the link again and fails.
  • db_versions.json has no entry for a database, but its link is already there.
  • ARAX_database_manager.py -f on an instance that already has links.

The same CLI run is a startup step in three places:

The fix:

def symlink_database(self, symlink_path, target_path):
# -sfn replaces an existing (possibly stale or dangling) symlink in place,
# so re-running the database manager is idempotent. Never clobber a real file.
if os.path.lexists(symlink_path) and not os.path.islink(symlink_path):
raise FileExistsError(f"{symlink_path} exists and is not a symlink; refusing to overwrite")
_run_cmd_in_shell_chk_status(f"ln -sfn {shlex.quote(target_path)} {shlex.quote(symlink_path)}")

ln -sfn replaces a link that is already there, so a second run gives the same result as the first. But -f would also delete a real file at that path, and that file can be a full database copy. So the function now raises FileExistsError for a real file and leaves it alone. The startup code already had the same rule.

The Flask startup on arax.ncats.io (__main__.py L168-L172) uses symlink_from_central_and_write_versions(). That function removes a stale link itself before it calls symlink_database(), so it works the same as before.

2. generate-db-symlinks.sh did not match config_dbs.json

The old script had every file name written by hand, and some of them were out of date:

link "${DB_DIR}/${LATEST_TIER0_VER}/curie_ngd_v1.0_${LATEST_TIER0_VER}.sqlite" "RTX/code/ARAX/KnowledgeSources/NormalizedGoogleDistance/curie_ngd_v1.0_${LATEST_TIER0_VER}.sqlite"
link "${DB_DIR}/${LATEST_TIER0_VER}/curie_to_pmids_v1.0_${LATEST_TIER0_VER}.sqlite" "RTX/code/ARAX/KnowledgeSources/NormalizedGoogleDistance/curie_to_pmids_v1.0_${LATEST_TIER0_VER}.sqlite"
link "${DB_DIR}/${LATEST_TIER0_VER}/fda_approved_drugs_v1.0.pickle" "RTX/code/ARAX/KnowledgeSources/fda_approved_drugs_v1.0.pickle"

  • curie_ngd and curie_to_pmids were v1.0, but config_dbs.json says v1.1. ARAX opens the file name from the config (compute_ngd.py, Path_Finder/utility.py), so in a dev setup made by this script ARAX could not find the NGD databases.
  • fda_approved_drugs_v1.0.pickle was taken from tier0-20260621, but the config points to tier0-20260408.

Now the script reads all paths from config_dbs.json with python3. When someone changes a version in the config, the script does not need a change. The only thing still written in the script is which RTX folder each database goes to. These are the same folders as in ARAX_database_manager.py. If the config gets a new database that the script does not know, the script stops before it creates any link and says what to add.

The overrides for local setups still work:

One new link: tier0_sqlite now goes to KnowledgeSources/Tier0/, because ARAX_decorator.py reads it from there. Nothing in the repo created this link before.

Testing

  • New test_ARAX_database_manager.py with 4 cases: same link twice, stale link, broken link, real file. All 4 fail on master and pass on this branch. In the Test Build run for this PR they passed together with the rest of the suite (181 passed, 144 skipped). The "Create symlinks" step in that run used the new code on Linux and linked all 7 databases without errors.
  • I also ran the manager on a copy of the RTX tree with fake database files and no network. On master it failed with File exists in 5 cases: same link twice, stale link, broken link in update_databases(), missing db_versions.json entry, and -f. With this PR all 5 work. symlink_from_central_and_write_versions() gives the same result as before in all its cases.
  • I ran the script with macOS /bin/bash 3.2 and checked that all 8 links match config_dbs.json. I also tried a second run, a LATEST_TIER0_VER override, an unknown database key, a missing source file, and a real file at the destination.
  • Nothing was run on arax.ncats.io or any other live host.

Not in this PR

update_databases() does not fix a link when only its target changed and the file name stayed the same. For example, an fda link that points into tier0-20260621 stays there, because the version is still 1.0 and the manager skips it. I did not check if any host has such a link. This will be a separate PR.

- ARAX_database_manager.symlink_database() now runs `ln -sfn`, so the CLI
  update_databases() path no longer dies when a link already exists. It
  raises FileExistsError instead of replacing a real file.
- generate-db-symlinks.sh now reads every database path from
  config_dbs.json (it had fda on the wrong tier0 dir and NGD files at
  v1.0). DB_DIR and LATEST_TIER0_VER overrides still work.
- Add test_ARAX_database_manager.py for symlink_database().
Copilot AI balanced review requested due to automatic review settings October 2, 2026 07:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants