Skip to content

Return POST /get_setid results as a JSON list, as NodeNorm Redis does - #48

Draft
gaurav wants to merge 26 commits into
biothings:mainfrom
TranslatorSRI:setid-post-list
Draft

gaurav wants to merge 26 commits into
biothings:mainfrom
TranslatorSRI:setid-post-list

Conversation

@gaurav

@gaurav gaurav commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Builds on #42 — review only the last four commits (from "Let handlers write a top-level JSON list" on). Until #42 merges, this PR's diff against main includes it.

POST /get_setid answers with an object keyed by input index ({"0": {...}, "1": {...}}) because Tornado refuses to write a top-level list. NodeNorm Redis returns a list, and the clients were written against it. This PR lets NodeNormalizationBaseHandler write lists, and then returns the SetIDResponse objects as a list in input order, as the OpenAPI document now says. This is a breaking change for any client of NodeNorm ES that reads body["0"].

Calls worth overruling

What's here

  • POST /get_setid returns a JSON array; the unreachable "empty result → 500" check after the loop is gone. The OpenAPI document's 200 response for it is now an array of SetIDResponse.
  • A unit test for the base handler's list output, and a POST unit test in tests/test_openapi.py that mounts the handler with a fake generate_setid() patched through unittest.mock, so the fake doesn't leak into tests/test_set_identifiers.py when pytest-randomly runs that later. Rewrite the OpenAPI document to describe the Elasticsearch service, and make the endpoints match it #42's tests of the POST errors now read the list, and cover an empty body as well.
  • The integration test in tests/test_set_identifiers.py expects a list.

What it deliberately does not do

The POST body checks (400 for a body that isn't a non-empty JSON array of objects) and the generate_setid() fixes for unknown conflations and empty sets are in #42, not here.

Verification

On 22f1970, openapi-spec-validator passes, and the whole suite passes under several random orderings except the four integration tests that need the su10 host: test_node_normalization_get and test_node_normalization_post in test_normalized_nodes.py, and test_get_endpoint and test_post_endpoint in test_set_identifiers.py. So the list-expecting integration test hasn't been run.

Nothing is blocking beyond #42 merging first.

🤖 Generated with Claude Code

gaurav and others added 7 commits October 12, 2026 00:41
The document was copied from NodeNorm Redis and never pruned, so it
advertised TRAPI 1.5 (info.x-trapi), carried 145 schemas no path
referenced, named RENCI's developer as the contact and sent readers to
/get_curie_prefixes, which this service has never served. It also
failed openapi-spec-validator: the /get_allowed_conflations response
had no description, and FilterKgraphTopNParameters.max_edges declared
maximum 0 (NCATSTranslator/NodeNormalization#409).

Rewritten from scratch against build_handlers(): the six routes this
service registers, nothing else. Responses that were `schema: {}` now
have schemas (NormalizedNodes, SetIDResponse), the FastAPI 422 responses
are replaced by the 400 the Tornado handlers raise, and the examples are
taken from the live CI instance. The servers block lists the ci and
test deployments at ITRB instead of RENCI. Contact is Will Liu and the
team is DOGSURF (biothings#35); infores and
termsOfService are unchanged for now. info.version is bumped to 1.1.0
because /status reports it and the response shapes change in the next
commit.

handlers/curie_prefix.py was an unregistered copy of the semantic-types
handler (its class is even named SemanticTypeHandler), so it goes too.

Refs biothings#35, biothings#36

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GET /get_allowed_conflations finished a bare list, and Tornado's write()
refuses a top-level list ("Lists not accepted for security reasons"),
so it returned HTTP 500 on every deployment. It now returns
{"conflations": [...]}, which is also what NodeNorm Redis returns.

GET /get_semantic_types lower-cased every class ("biolink:namedthing");
NodeNorm Redis returns the Biolink class URI as is ("biolink:NamedThing").

/openapi.json now redirects to /webapp/openapi.json so the FastAPI
default address used by NodeNorm Redis, SmartAPI and Babel Validation
works here too; it is excluded from OpenTelemetry like the other static
paths.

The POST set-ID test posted a single object where the handler expects
a list of sets, and the normalized-nodes docstring showed a "curie" key
where the handler reads "curies"; both corrected. POST /get_setid still
answers with an object keyed "0", "1", ... (the same Tornado limitation);
returning a list as NodeNorm Redis does is a breaking change, so it is
left to its own pull request.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The document is hand-written and served as a static file, so these
tests are what stops it drifting: the spec validates; its paths are
exactly the routes build_handlers() registers; each path documents the
GET/POST methods its handler defines; each GET documents the query
parameters its handler reads (found by scanning the handler source for
get_argument calls); and the SetIDResponse schema lists the dataclass's
fields. Each was checked to fail when the spec is mutated.

Also a regression test for /get_allowed_conflations, which returned
HTTP 500 until the previous commit: it mounts just that handler and
validates the body against the documented ConflationList schema.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
It ran only against the su10 Elasticsearch host before. This mounts the
handler alone with a fake Elasticsearch client and validates the body
against the documented schema.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dot in each redirect pattern was unescaped, so /openapiXjson and
/faviconXico redirected too. The OpenTelemetry exclusions already escape
it. A test now checks the /openapi.json redirect and that a near-miss
path is a 404.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ment

The query-parameter check scanned only get() for get_argument calls, so
a parameter read in a helper method would drift unnoticed; it now scans
the whole handler class. A new check compares the fields each post()
reads with .get("...") against the properties of the documented request
body (CurieList, and the SetIDQuery items of the /get_setid array).
Both were confirmed to fail when a field is removed from the document or
a parameter is renamed in a helper.

The checks still only see literal names; the README now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaurav and others added 19 commits October 12, 2026 01:41
generate_setid() returned the SetIDResponse dataclass itself on the
invalid-conflation and no-normalized-CURIEs paths. Tornado can only
write dicts, so GET /get_setid?conflation=Bogus answered 500 and the
documented `error` field never reached anyone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body has to be a JSON array of {curies, conflations} objects. A
single object (the shape the old document described) passed the length
check and then failed with an AttributeError, and invalid JSON failed in
json.loads; both were 500s. They are now 400s that say what shape the
body must have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The index mapping (for the Babel version) was read before the try
block, so an unreachable cluster raised and /status answered 500
instead of the documented {"status": "error"}. The read now happens
inside the try; babel_version and babel_version_url are null when the
mapping couldn't be read, and the document says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The patterns were written r"/get_setid?", r"/status?" and so on. In a
Tornado route that ? makes the last character optional, so /get_seti and
/statu reached the handlers though the OpenAPI document lists only the
full paths. test_openapi.py hid this by stripping the ? before comparing
routes with the document; it now compares them as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request-body check read only post()'s source, though the README says
fields are found by scanning the handler, and the query-parameter check
already scans the whole class. A body field read in a helper method was
either missed or reported as undocumented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The methods check compared only GET and POST, so the HEAD handler on
/get_allowed_conflations, which the document didn't list, went
unnoticed. It now compares every method Tornado routes except OPTIONS
(answered by the base handler for CORS), and the document lists HEAD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_openapi.py imports it directly; it was only installed because
openapi-spec-validator happens to depend on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parse_boolean() treated every string except "false" as true, so
conflate=0 or conflate=no turned conflation on. The flags now accept the
strings FastAPI/Pydantic accepted in NodeNorm Redis (true/false, 1/0,
yes/no, on/off, t/f, y/n, in any case) and answer 400 for anything else,
including an empty value. POST bodies go through the same check, so
{"conflate": "false"} is no longer truthy. The document lists the
accepted values.

This is a behavior change: a caller sending an unrecognized value now
gets a 400 where it used to get conflation silently switched on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers the second path biothings#47 describes: a set whose CURIEs normalize to
nothing. get_normalized_nodes() is patched out, so it needs no host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://nodenorm-es.transltr.io/ is now live. It goes first, since
Swagger UI and generated clients default to the first server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
json.loads() raises UnicodeDecodeError rather than JSONDecodeError for
bytes that aren't valid UTF-8, so such a body escaped the check and
became a 500, though the document promises a 400 for a malformed body.
The decoding moves to a load_json_body() helper on the base handler so
POST /get_normalized_nodes can use it too.

Raised by Copilot on biothings#42.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The handler called json.loads() with no error handling and then
.get() on the result, so a body that wasn't JSON, wasn't UTF-8, or
was an array instead of an object all became 500s. It now uses
load_json_body() and checks for an object, and the document lists
these under the 400 response.

Same cause as the UTF-8 case Copilot raised for POST /get_setid on biothings#42.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tornado's write() refuses a list ("Lists not accepted for security
reasons"), a guard against JSON hijacking through <script> tags in very
old browsers. That made GET /get_allowed_conflations return HTTP 500,
and made POST /get_setid answer with an object keyed "0", "1", ...
instead of the list NodeNorm Redis returns. NodeNormalizationBaseHandler
now encodes a list the way Tornado encodes a dict, so every handler can
finish() one.

/get_allowed_conflations keeps its {"conflations": [...]} wrapper, which
is the shape NodeNorm Redis returns; only its comment changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /get_setid answered with an object keyed "0", "1", ... because
Tornado would not write a list. Now that the base handler can, it
returns the list of SetIDResponse objects in input order, which is what
NodeNorm Redis returns and what the OpenAPI document now says. This is
a breaking change for any client that reads body["0"].

The "if not set_identifiers: 500" check after the loop could never fire
(the body is already known to be non-empty and each set appends one
result), so it is gone.

The unit test that mounts the handler with a fake generate_setid() used
to assign the module attribute and never put it back, so under
pytest-randomly the fake could leak into tests/test_set_identifiers.py.
It now patches with unittest.mock and undoes the patch after each test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit used to add the POST /get_setid body checks themselves: a
body that wasn't valid JSON, was a single object, or held anything
other than objects reached group.get() and failed with HTTP 500. biothings#42
now makes those checks (through load_json_body()) and tests them, so
after rebasing onto it all that is left is the empty-body case, which
its test didn't cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each set awaited its own Elasticsearch lookups before the next began,
so a POST with many sets took one round trip per set in series. The sets
are independent, so asyncio.gather() now runs them together; it returns
the results in input order.

This raises the number of concurrent Elasticsearch requests one POST
can make, in proportion to the number of sets (see biothings#22 on how much load
the cluster takes). It is a separate commit so it can be dropped on its
own if that is a concern.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant