Annotation service for the Translator Project. Originally apart of the pending.api, we've broken it out into it's own service
git clone https://github.com/biothings/biothings_annotator
python3 -m venv biothings_annotator
cd biothings_annotator
pip install .pip install -e git+https://github.com/biothings/biothings_annotator#egg=biothings_annotatorbiothings_annotator
├── annotator <- annotation backend logic
└── application <- web service logic
├── cli
├── configuration
├── exceptions
├── middleware
└── viewsbiothings_annotator is as a package is separated into the annotator logic and web handler logic.
The annotator logic primarily exists within annotator/annotator.py and annotator/transformer.py.
Whereas the web server application is defined entirely within application directory.
Annotator can be used directly without starting the Sanic service. To select Elasticsearch and
perform a live connection check from an async context:
from biothings_annotator import Annotator
annotator = Annotator(query_backend="es")
verification = await annotator.verify_backend()A successful call returns connection and server identity information:
{
"query_backend": "elasticsearch",
"elasticsearch_connection": "in_cluster",
"host": "http://elasticsearch.es-core-components.svc.cluster.local:9200",
"connected": True,
"server_product": "Elasticsearch",
"cluster_name": "...",
"cluster_uuid": "...",
"node_name": "...",
"server_version": "...",
}Each call makes a fresh request to the configured Elasticsearch root endpoint. Failures raise the
public BackendVerificationError, whose reason attribute identifies configuration, connection,
authentication, HTTP, or response-validation errors. This verifies the instance connection, not a
specific index or the provenance of an annotation result; indexes are selected later from the queried
CURIE, and Elasticsearch queries do not fall back to the BioThings API.
The __main__.py defines the entrypoint to the module for running the sanic web server. After
installation run the following to command to start the annotator service:
python3 -m biothings_annotatorThe __main__.py file points to the application module where it will run the main function within
the launcher. The majority of our implementation exists within the
biothings_annotator/application/cli module. We store the main command line handling logic in
cli/interface.py with argument definitions and other argument handling in cli/arguments.py. We
wanted to maintain the same command line interface as sanic. sanic has a cli
module where it defines the SanicCLIclass for handling their command-line implementation. However
we also have additional arguments we want to support. So we override the SanicCLI to acquire the
original parser handlers. The command-line steps are then divided in three steps shown below.
# Entrypoint
cli = AnnotatorCLI()
cli.attach()
cli.parse()
cli.run()The attach method aggregates all of the ArgumentParser instances stored in what sanic
defines as Group objects. In order to add our custom arguments we define our own implementations
of the Group argument parsers to be called during the attach method.
The parse method then builds the parsers from Group objects. While we want to support the same
interface as sanic, we do limit some of the options. The main limit is the target option which
points to a module or factory for building the AppLoader instance. We hard-set that and a couple
other options so that we cannot accidently change the path we point to for building the web server
implementation. Our factory method for generating the sanic.Sanic application instance is defined
with cli/target.py.
# Original AppLoader instance
app_loader = AppLoader(
self.args.target, self.args.factory, self.args.simple, self.args
)
# biothings-annotator AppLoader instance
application_loader = AppLoader(
module_input=self.args.target, # hard-coded to ""
as_factory=self.args.factory, # hard-coded to False
as_simple=self.args.simple, # hard-coded to False
args=self.args,
factory=functools.partial(build_application, self.server_configuration),
)
# AppLoader definition
class AppLoader:
"""
A helper to load application instances.
Args:
module_input (str): The module to load the application from.
as_factory (bool): Whether the application is a factory.
as_simple (bool): Whether the application is a simple server.
args (Any): Arguments to pass to the application factory.
factory (Callable[[], SanicApp]): A callable that returns a Sanic application instance.
"""
def __init__(
self,
module_input: str = "",
as_factory: bool = False,
as_simple: bool = False,
args: Any = None,
factory: Optional[Callable[[], SanicApp]] = None,
) -> None:The run method builds the AppLoader instance and the runtime arguments builder method. The
default command line arguments are shown below:
default_parameters = {
"access_log": None,
"auto_tls": False,
"coffee": False,
"debug": False,
"fast": False,
"host": None,
"motd": True,
"noisy_exceptions": None,
"port": None,
"single_process": False,
"ssl": None,
"unix": "",
"verbosity": 0,
"workers": 1,
}Some of these arguments are hard-set by the configuration file defaults and cannot be changed at the command-line unless the configuration file is modified.
python3 -m biothings_annotator --host "172.84.29.248"
python3 -m biothings_annotator --host "172.84.29.248" --port 9384
python3 -m biothings_annotator --host "172.84.29.248" --port 9384 --workers 12
python3 -m biothings_annotator --host "172.84.29.248" --port 9384 --workers 12 --debugSERVICE_PROVIDER_API_HOST overrides the BioThings host used for HPO, NCIT, and extra
annotation endpoints and their metadata. When unset, it falls back to
https://biothings.transltr.io in every deployment environment. Set it explicitly in the
container environment to use a matching CI or test BioThings API, such as
https://biothings.ci.transltr.io or https://biothings.test.transltr.io, and restart the service.
The Helm chart requires containers.service_provider_api_host and passes it to the container
as SERVICE_PROVIDER_API_HOST. Set it in the environment's values file (values-ncats.yaml
when using deploy/deploy.sh), for example:
containers:
service_provider_api_host: https://biothings.ci.transltr.ioUse https://biothings.test.transltr.io for test and https://biothings.transltr.io for
production. Helm rendering fails when the value is missing or blank, so each deployment
must explicitly choose its BioThings API host. The deployment environment label does not
select this host. With the biothings query backend, primary gene, chemical, and disease
queries continue to use the public MyGene, MyChem, and MyDisease APIs.
GET /status verifies that the configured query backend can annotate NCBIGene:1017 and
returns the common NodeNorm status fields plus source build versions for both backends.
query_backend identifies the deployment's default backend. available_query_backends lists
metadata for biothings-api and elasticsearch; the biothings-api entry corresponds to the
canonical biothings query backend. This abbreviated example uses illustrative versions; each
backend lists all sources it supports:
{
"status": "running",
"deployment_environment": "ci",
"version": "1.0",
"query_backend": "elasticsearch",
"available_query_backends": {
"biothings-api": {
"checked_at": "2026-10-20T18:00:00+00:00",
"sources": {
"mygene": {
"build_version": "20261019",
"sources": [
{"name": "ensembl", "source_version": "116"},
{"name": "entrez", "source_version": "20261009"},
{"name": "uniprot", "source_version": "20260903"}
]
},
"hpo": {
"build_version": "20250120",
"sources": [
{"name": "hpo", "source_version": "2025-01-19"}
]
}
}
},
"elasticsearch": {
"connection": "ci_cluster",
"checked_at": "2026-10-20T18:00:00+00:00",
"sources": {
"mygene": {
"alias": "gene",
"index": {
"name": "mygene_allspecies_20260705_bgks2qnv",
"build_version": "20260705",
"sources": [
{"name": "ensembl", "source_version": "116"},
{"name": "entrez", "source_version": "20260705"},
{"name": "uniprot", "source_version": "20260610"}
]
}
},
"hpo": {
"alias": "hpo",
"index": {
"name": "hpo_20251027_b3jbx7z4",
"build_version": "20251027",
"sources": [
{"name": "hpo", "source_version": "2025-10-26"}
]
}
},
"pubmed": {
"alias": "annotator-pubmed",
"index": {
"name": "pubmed_20260814_ew8jmctk_202608201240",
"build_version": "20260814",
"sources": [
{"name": "pubmed_metadata", "source_version": "2026aug5"}
]
}
},
"annotator_extra": {
"alias": "annotator-annotations",
"error": {"code": "http_error", "http_status": 404}
}
}
}
}
}The version field is the API version from the served OpenAPI specification. The separate
/version endpoint reports the build's Git commit from version.txt, together with the active
query backend. If runtime backend metadata cannot be constructed, /version still returns
HTTP 200 with version, omitting query_backend and elasticsearch_connection. All
/version responses send Cache-Control: no-store.
For local development, set ANNOTATOR_DEPLOYMENT_ENV=local (or configure the deployment environment
as local). If version.txt is missing, /version uses a Git snapshot of the loaded package's
checkout captured at startup: the full HEAD commit, with -dirty appended for tracked, staged,
or untracked changes. Restart the service to refresh that snapshot. version.txt always takes
precedence. Outside the local environment, a missing file returns Unknown without querying
Git; the local fallback also returns Unknown if Git or the checkout is unavailable or the
query fails.
Source availability follows ANNOTATOR_CLIENTS and its backend support. Both backends share
status display names derived from the client configuration. Typed SDK clients use their
configured <service>.info hostname, yielding mygene, mychem, and mydisease without a
per-source mapping. Endpoint-based clients use their configured endpoint, yielding hpo,
ncit, and annotator_extra. Sources without an API client retain their configured key;
pubmed appears only under Elasticsearch. Custom client URLs outside the <service>.info
convention, unresolved client configurations, and conflicting names fall back to the configured
source key. Names do not depend on a successful metadata request. These are status labels only:
internal source keys, annotation routing, and Elasticsearch aliases remain unchanged.
Missing optional sources supported by that backend remain visible with an
error instead of disappearing. BioThings versions come from each API's /metadata;
Elasticsearch versions come from index mapping _meta, with the configured alias or direct index
in alias. The singular index object contains the resolved concrete name and build_version.
For Elasticsearch, index.sources lists every upstream source recorded in that index's
mapping _meta.src. For BioThings APIs, sources sits alongside build_version and lists every
upstream source in that API's /metadata src. Both lists are sorted by name and use
{"name": "...", "source_version": "..."} entries from each src entry's version field.
There is no whitelist or entry limit; the example above abbreviates the gene sources. API versions
describe the data served by each API, while Elasticsearch versions describe the data used for
that indexed build; these can differ.
Scalar versions are normalized to strings. Missing, blank, boolean, or structured versions are
null, and those sources stay in the list. An absent src produces an empty list. Versions
are never inferred from build versions, index names, or timestamps.
Elasticsearch metadata failures retain alias and report error in place of index. If an alias unexpectedly
resolves to multiple indices, its source reports error.code: multiple_indices; no index is
selected arbitrarily. connection describes the configured
Elasticsearch target:
ELASTICSEARCH_CONNECTION |
Status connection |
|---|---|
ci_forward |
ci_cluster_forwarded |
test_forward |
test_cluster_forwarded |
prod_forward |
prod_cluster_forwarded |
in_cluster |
<deployment_environment>_cluster for ci, test, or prod; otherwise unknown_cluster |
local |
local |
Forwarded environments come from the selected preset's forwarded_environment metadata,
independently of annotator's deployment environment. A loopback address alone cannot identify
which cluster is forwarded. The preset records the intended target; it does not verify the
tunnel's destination or establish a tunnel. Verifying the actual destination would require a
trusted mapping from Elasticsearch cluster UUIDs to environments, or metadata supplied by the
tunnel launcher. A forwarded preset with an unrecognized environment reports
unknown_cluster_forwarded; an unrecognized preset reports unknown. Connection URLs,
credentials, and raw upstream errors are not included in the status response.
The annotator_extra status source uses the BioThings /annotator_extra API and the
Elasticsearch annotator-annotations alias, which remains visible in its alias field.
build_version identifies each source's build. Unknown versions are null; an index name is
never used to infer a version. Metadata failures use a static
error.code, optionally with http_status, at the affected source. These metadata errors are
informational and do not change the health check's HTTP status.
Each worker warms its source metadata at startup and refreshes it every 10 minutes in
the background, independently of incoming requests. GET reads the current snapshot without
waiting for a background refresh. checked_at is the UTC collection time represented by that
snapshot. Upstream calls run concurrently with five-second deadlines; each complete collection
has a 10-second deadline. Concurrent callers share a collection if no snapshot exists yet.
Source-specific failures remain visible alongside healthy sources. Timeouts, connection errors,
HTTP 429, and HTTP 5xx responses cause another collection after 30 seconds. Consecutive
transient failures double that delay (30, 60, 120 seconds, and so on), capped at the normal
10-minute interval. A collection without transient errors resets the retry delay. Expected
missing optional sources, configuration errors, and other source errors use the normal interval.
An unexpected collection failure retains the previous snapshot and its checked_at, adds a
backend-level refresh_error with code metadata_unavailable, and uses the same retry backoff.
If no snapshot exists, it reports a backend-level error with that code instead of sources.
The next completed collection clears refresh_error. Worker shutdown cancels pending collections
and the background timer. Metadata refresh does not change annotation routing.
GET and HEAD share a separate annotation health cache, scoped to the worker and backend
configuration. Successful checks are cached for 5 seconds, failures for 1 second, and
each check has a 10-second total deadline. Concurrent probes share one check; a disconnected
caller cannot cancel it for the others. An expired healthy result is never served after a
failed check. Failures retain {"success": false, "error": "..."}: HTTP 500 when the expected
annotation is missing, or HTTP 400 when the check raises an exception or times out.
Exception details are logged server-side; the public error uses a fixed message.
HEAD /status uses the same health result without reading source metadata or returning a body.
All GET and HEAD status responses, including failures, send Cache-Control: no-store.
Set ANNOTATOR_DEPLOYMENT_ENV to the deployment environment, for example ci, test, or
prod. It takes precedence over application.configuration.DEPLOYMENT_ENVIRONMENT in the
JSON configuration when nonempty. The value is loaded at startup, trimmed, and defaults to
unknown when unset or blank; request headers and hostnames do not determine it. Restart the
service after changing the deployment environment or OpenAPI version.
Kubernetes deployments populate ANNOTATOR_DEPLOYMENT_ENV from the pod's
gov.nih.ncats.appenv label through the Downward API. Set that label under ncats.labels in
the environment's Helm values; the chart defaults to unknown.
The service can export Sanic request spans and downstream HTTPX spans to a Jaeger collector using OTLP over HTTP. Tracing is disabled by default and can be enabled without changing the configuration file:
export OPENTELEMETRY_ENABLED=true
export OPENTELEMETRY_SERVICE_NAME=BioThingsAnnotator
export OPENTELEMETRY_JAEGER_HOST=http://localhost
export OPENTELEMETRY_JAEGER_PORT=4318
python -m biothings_annotatorThe exporter sends spans to
$OPENTELEMETRY_JAEGER_HOST:$OPENTELEMETRY_JAEGER_PORT/v1/traces. Set
OPENTELEMETRY_EXCLUDED_URLS to a comma-separated list of regular expressions
to override the configured route exclusions. The Helm deployment enables
tracing and targets http://jaeger-otel-collector.sri:4318 by default.
The annotator query backend is controlled with ANNOTATOR_QUERY_BACKEND. Supported values are
biothings, biothings-api, elasticsearch, and es. Values are case-insensitive and surrounding
whitespace is trimmed. biothings-api resolves to biothings, and es resolves to
elasticsearch; when unset, the service uses biothings.
The checked-in Helm defaults set ANNOTATOR_QUERY_BACKEND=elasticsearch and
ELASTICSEARCH_CONNECTION=in_cluster; set ANNOTATOR_QUERY_BACKEND to biothings during deployment
to switch back.
Set ELASTICSEARCH_CONNECTION to one of the named presets in
biothings_annotator/annotator/settings.py. The in_cluster preset points at
http://elasticsearch.es-core-components.svc.cluster.local:9200 and is used by both CI and test
deployments. The local preset targets Elasticsearch on localhost:9200. The ci_forward and
test_forward presets also connect to localhost:9200, for an already-established port-forward,
and send the corresponding environment's ingress host header. prod_forward connects to
localhost:9200 without an ingress host override, for a direct forward to the production
Elasticsearch service. Each forwarded preset records its target environment in
forwarded_environment; the status endpoint uses that metadata to format connection.
The /version endpoint reports the active query_backend and, when Elasticsearch is active,
the selected elasticsearch_connection preset name. For a live check of a directly constructed Python instance,
see Direct Python usage.
The GET /curie/{curie}, POST /curie, and POST /trapi endpoints accept an optional
query_backend query parameter. Omit it to use the backend selected for the deployment by
ANNOTATOR_QUERY_BACKEND. Accepted values are biothings, biothings-api, elasticsearch, and
es, case-insensitively with surrounding whitespace trimmed. biothings-api is an alias for
biothings, and es is an alias for elasticsearch; either changes the backend for only that request.
Unsupported values are ignored and the request uses the deployment default.
Successful query responses include the canonical backend in the X-Query-Backend response header.
PubMed supports only Elasticsearch. A single PubMed CURIE requested through BioThings returns
HTTP 422; bulk and TRAPI requests retain their supported results and mark PubMed entries as
skipped, with the source, selected backend, and source_unavailable_for_backend reason.
X-Skipped-Curie-Prefixes summarizes skipped prefixes as comma-separated values. Unknown CURIE
prefixes are not included in this header. See the PubMed section below for examples.
curl 'http://localhost:9000/curie/NCBIGene:1017?query_backend=biothings'
curl -X POST 'http://localhost:9000/curie/?query_backend=elasticsearch' \
-H 'Content-Type: application/json' \
-d '{"ids":["NCBIGene:1017","CHEBI:100024"]}'
curl -X POST 'http://localhost:9000/trapi/?query_backend=es' \
-H 'Content-Type: application/json' \
-d '{"message":{"knowledge_graph":{"nodes":{},"edges":{}}}}'PMID CURIEs are routed unchanged to the standalone annotator-pubmed Elasticsearch alias. When the
annotation-hub pubmed_metadata
source is built and indexed there, the default annotation response includes its pubmed object
(journal, title, volume, issue, publication date, and abstract). Like the other first-class
biomedical sources, PubMed has its own annotator client configuration rather than being merged into
annotator_extra.
PubMed is supported only by the elasticsearch backend. This applies to the PMID, doi,
DOI, PMC, and pmc CURIE prefixes. When the selected backend is biothings, the annotator
does not query a PubMed API and does not switch to Elasticsearch automatically.
URL-encode CURIEs used in the GET path, including the slash in DOI values.
A single request such as GET /curie/PMID:30690000?query_backend=biothings returns HTTP 422
with Cache-Control: no-store, X-Query-Backend: biothings, and this error:
{
"input": "PMID:30690000",
"endpoint": "/curie/",
"error": "unsupported_source_backend",
"message": "Source 'pubmed' is not supported by the 'biothings' backend. Use query_backend=elasticsearch.",
"source": "pubmed",
"query_backend": "biothings",
"supported_backends": ["elasticsearch"]
}Bulk POST /curie requests still return HTTP 200 and annotate supported CURIEs. PubMed entries
receive a one-item skipped result, as shown below. TRAPI requests also return HTTP 200, preserve
the affected nodes, and add the skipped result in a biothings_query_status attribute. Both
responses set X-Skipped-Curie-Prefixes to the affected prefixes and use Cache-Control: no-store.
The notfound field preserves the normal result shape; skipped means no annotation lookup
was attempted, rather than that PubMed contained no matching record:
{
"PMID:31763219": [
{
"query": "PMID:31763219",
"notfound": true,
"skipped": true,
"reason": "source_unavailable_for_backend",
"source": "pubmed",
"query_backend": "biothings"
}
]
}Select query_backend=elasticsearch to annotate PubMed CURIEs. The dedicated publication
metadata endpoints below already use Elasticsearch. /status lists PubMed only under
available_query_backends.elasticsearch.sources.
curl 'http://localhost:9000/curie/PMID:12345678?query_backend=elasticsearch'The dedicated publications API is the PubMed-only fast path requested by the Core Components Working Group. It supports the legacy batch query contract, a path-based single-publication lookup, and a JSON batch contract:
# Legacy-compatible batch lookup
curl 'http://localhost:9000/publications?pubids=PMID:30690000,PMID:82374&request_id=request-123'
# Single-publication lookup
curl 'http://localhost:9000/publications/PMID:30690000?request_id=request-123'
# JSON batch lookup, mixing identifier types
curl -X POST 'http://localhost:9000/publications' \
-H 'Content-Type: application/json' \
-d '{"ids":["PMID:30690000","PMC:PMC1904490","doi:10.1242/jcs.03153"],"request_id":"request-123"}'Both batch forms accept at most 100 identifiers; the path form looks up one identifier. An optional
request_id is round-tripped in the response metadata. For POST, it can be supplied in the JSON
object as shown above.
The response contains legacy-compatible _meta, results, and not_found sections, keyed by the
identifier as submitted. Missing source values are returned as empty strings. The publication date is
projected from PubMed's verbatim rendering when the index carries it, so month ranges survive as
CCWG#15 specifies ("pub_month": "Sep-Dec" for PMID:8000234); otherwise it falls back to splitting
the indexed ISO pub_date into year, month, and day.
The index carries the verbatim value in pubdate_raw — whatever the upstream exporter emits, the
in-house ingest transformation normalizes it to that field, which is also the field the capability probe
checks. pub_date is additionally read as a verbatim value purely defensively, so that a raw value
landing there is not flattened to empty strings if that transformation ever changes. It cannot misread
anything: the verbatim and ISO parsers accept disjoint shapes, so "2019-03-15" is only ever read as an
ISO date and "1994 Sep-Dec" only as a verbatim one.
A bare year range is the one shape that collides with an ISO date, since both open with YYYY-. It is
projected losslessly into pub_year — "1987-1988" gives {"pub_year": "1987-1988", "pub_month": "", "pub_day": ""} — because the legacy fields have no home for a second year and pub_month would misfile
one. Reading it as ISO instead would return "1987" and drop the closing year. A two-digit tail such as
"1987-88" keeps its ISO reading, because it cannot be distinguished from the month in "2026-07".
PMID, PMC, and doi are accepted. Prefix casing is matched case-insensitively across the ASCII
spellings only, and a PMCID keeps PubMed's doubled form — PMC:PMC1904490, not PMC:1904490. The
served patterns spell each prefix out as explicit case pairs to mirror the OpenAPI PublicationId
pattern character-for-character; re.IGNORECASE would case-fold non-ASCII letters such as doİ: into
a match that the published contract rejects and the index cannot resolve. Two further shape constraints
follow from the transport rather than the service:
- A DOI suffix contains slashes, so the path form relies on a
pathroute converter to avoid truncating at the first segment. - A DOI suffix may itself contain a comma, which the legacy comma-separated
pubidsform cannot express. Use the JSON body for those identifiers.
These routes are deliberately separate from the generic annotation pipeline. They always read the
annotator-pubmed Elasticsearch alias and retrieve only the pubmed source object. They do not
perform CURIE grouping, extra annotation lookup, or per-request backend selection. The request has a
two-second total backend deadline by default (configurable with
DOCUMENT_METADATA_REQUEST_TIMEOUT) so a degraded Elasticsearch service fails quickly.
PMIDs are the document _id, so they resolve through one exact-ID Elasticsearch _mget for the whole
batch, including a complete batch of 100. PMCID and DOI resolve against pubmed.identifiers instead,
which costs one _msearch entry per identifier, so a PMID-only request stays on the single-request fast
path and only mixed requests pay for the scoped lookup. The canonical PMID is stored only as _id;
pubmed.identifiers contains its DOI and PMC aliases. The behavioral performance test verifies that a
100-PMID request remains one backend request, and the load benchmark measures what that costs against
the 150 ms p90 service objective — see Load benchmark.
An alias that points to an older index without pubmed.identifiers returns not_found for every PMCID
and DOI lookup. That is indistinguishable from a genuinely absent paper, so the index shape is checked
separately — see Verifying the PubMed index shape.
Index compatibility is treated as a deployment prerequisite rather than something the service polices.
Deliberately, the endpoint does not refuse PMCID and DOI requests while the field is absent: a startup
gate would also take down the PMID fast path, which still works against an older index. Use
check_index_fields() to tell "not in this index" from "not a real paper" during rollout and rollback.
DocumentMetadataService.check_index_fields() probes the live mapping through the alias and reports
which of the fields the API depends on actually exist:
{
"index": "annotator-pubmed",
"fields": {"pubmed.identifiers": false, "pubmed.pubdate_raw": false},
"missing_required_fields": ["pubmed.identifiers"],
"multi_identifier_lookup": false,
"verbatim_publication_date": false
}It reports rather than raises. pubmed.identifiers is required for DOI and PMCID lookup to work at
all, so its absence sets multi_identifier_lookup to false; pubmed.pubdate_raw only changes the
precision of the projected date and has a working fallback, so it is informational.
fields reports mapping presence, while the required-field gate additionally demands that the field be
searchable. A field mapped with index: false is listed in the field-caps response but matches nothing
when queried, so it appears as "pubmed.identifiers": true and in missing_required_fields — that
combination is the signature of a field that exists but was mapped unsearchable. A fatal startup
assertion would be wrong here, because refusing to boot over a missing field would also take down the
PMID fast path that does work.
The probe reads the mapping rather than sampling documents, which is what separates "this field is not in the index" from "this paper is not in the index" — a DOI query against an unmapped field returns zero hits and no error. It stays useful after rollout: it catches a later reindex that drops the field, and an alias left pointing at a stale index.
With the CI Elasticsearch service forwarded to localhost:9200, run the opt-in live checks with:
RUN_PUBMED_ES_INTEGRATION=1 \
PUBMED_INTEGRATION_ELASTICSEARCH_CONNECTION=ci_forward \
python -m pytest -q tests/test_pubmed.py tests/test_document_metadata.py -m integrationPoint PUBMED_INTEGRATION_ELASTICSEARCH_CONNECTION at test_forward to run the same checks
against a forwarded test-instance service.
The document metadata live checks assert the index shape, resolution by every identifier type,
case-insensitive matching, and an upper bound of two alternate identifiers per record. The canonical PMID
remains the document _id, while the alternate list holds the DOI and PMCID. That bound is a bad-export
guard: pubmed2db PR #7 limits a record to its own identifiers, so a record carrying hundreds means the
export regressed and is pulling in cited references' DOIs.
Both were measured against the CI Elasticsearch cluster (es-core-components-cluster, index
pubmed_20260824_ybg350zl_202608250007, 41.0M documents, 21.5 GB per copy, 5 primary shards and 2
replicas), using identifiers sampled from the live index rather than synthesized ones.
Deduplicating pubmed.identifiers so the PMID is not stored twice — adopted, storage win only.
The PMID is already the document _id, so carrying it in identifiers as well was redundant. The
field now holds 43.5M values across 41.0M documents (1.06 per document; 80% of documents carry any
identifier at all), and re-adding the PMID would take that to 84.5M, near enough a doubling.
Measured by building two indices from the same 29,983 real documents with the live mapping and
analysis settings, differing only in whether the PMID is present in identifiers, then force-merging
both to one segment:
| deduped | PMID re-added | delta | |
|---|---|---|---|
| index store size | 31.0 MB | 31.7 MB | +2.3% |
pubmed.identifiers total |
0.9 MB | 1.4 MB | +50.1% |
| — inverted index | 0.5 MB | 0.7 MB | +48.4% |
| — doc_values | 0.5 MB | 0.7 MB | +51.8% |
_source |
26.5 MB | 26.7 MB | +1.0% |
_msearch took, 100 identifiers |
4 ms | 4 ms | 0% |
_mget response, 100 PMIDs |
152.7 kB | 154.2 kB | +1.0% |
Extrapolated to the live index that is 1.0 GB saved per copy, 3.0 GB across the three copies — the
field itself roughly halved, which is 2.3% of the index. The latency gain is nil, and measurably so:
identical _msearch timings and a _mget response 1% smaller. Worth having as a storage and index-size
change; it is not a latency optimization and should not be quoted as one.
Resolving to _id first, then one combined _mget — rejected. The idea was to ask the _msearch
only for _id, then fetch everything through a single exact-ID _mget. Direct Elasticsearch measurements
showed higher backend bytes at every identifier mix and added a serial round trip:
| identifier mix | current total | two-phase total | delta | current _msearch took |
two-phase _msearch took |
ES round trips |
|---|---|---|---|---|---|---|
| 75 PMID / 25 other | 154.0 kB | 158.9 kB | +3.2% | 9 ms | 8 ms | 1 parallel → 2 serial |
| 50 / 50 | 160.5 kB | 167.1 kB | +4.1% | 12 ms | 12 ms | 1 parallel → 2 serial |
| 25 / 75 | 166.0 kB | 173.7 kB | +4.7% | 18 ms | 16 ms | 1 parallel → 2 serial |
| 0 / 100 other | 168.7 kB | 181.9 kB | +7.8% | 22 ms | 21 ms | 1 → 2 serial |
These measurements did not establish full-path latency. The timing columns are only the top-level
_msearch.took; the two-phase value omits the _mget that follows it. They do show that source
retrieval is not an expensive part of _msearch: 100 identifiers that all resolve take 22 ms of
Elasticsearch wall-clock, and "_source": false removes about 1 ms. It cuts that leg's response by 84%
(169 kB to 26 kB), but those bytes reappear in the combined _mget with its per-document envelope, so
total backend response bytes rise 3–8%. The endpoint A/B later confirmed that the sequential strategy
was not worth adopting, so its implementation and selector were removed before the next experiment.
Two corrections to earlier reasoning come out of this. The shard fan-out is not the practical problem it looked like: per-identifier cost falls with batch size, 0.40 ms at 10 identifiers to 0.22 ms at 100, so Elasticsearch is parallelising the sub-searches well across the 5 shards. And the identifier path is not where the latency budget goes at all — 22 ms for a fully-resolving 100-identifier batch, against the ~114 kB of response body that the load benchmark shows to be the actual constraint. Optimisation effort belongs on payload size, not on identifier resolution.
One bulk alternative-ID search — temporary endpoint A/B. The current arm still runs the PMID _mget
and the alternative-ID _msearch concurrently. Its PubMed-specific _msearch now queries only the
confirmed root pubmed.identifiers keyword field; the old generic pubmed.identifiers.keyword clause was
a guaranteed miss for this mapping. Generic annotator searches keep their root-plus-.keyword compatibility
behavior because their mappings are not uniform.
The bulk-search arm also starts the PMID _mget immediately, but replaces the per-identifier _msearch
entries with one terms _search over every DOI and PMCID. It does not wait for or prune against _mget
results. Returned documents are mapped back to submitted identifiers from pubmed.identifiers in linear
time. The arm falls back to the cleaned exact _msearch for the whole alternative-ID batch if Elasticsearch
reports a truncated total, an identifier belongs to multiple documents, or a hit cannot be mapped safely.
The temporary request selector and paired benchmark below compare these two concurrent strategies on one
deployment.
A measurement note for anyone repeating this. Run through an SSH tunnel, every Elasticsearch round trip
costs 90-190 ms and drifts during a session, which is two orders of magnitude above in-cluster and more
than the entire effect being measured. Tunnel wall-clock does establish how the forwarded path behaves,
but it exaggerates the production cost of a second round trip. Per-sub-request took values must not be
summed either — Elasticsearch runs _msearch entries concurrently, so their sum is total work, not
latency. Compare the complete endpoint paths through CI's own processing_time_ms instead.
CCWG#15 sets a service objective of a 150 ms p90 for requests carrying up to 100 publication
identifiers. benchmarks/publications measures a deployment against it. The endpoint is meant as a
drop-in replacement for DocumentMetadataAPI, so the benchmark deliberately covers only
/publications — none of the generic annotation routes share its code path.
# The headline CCWG#15 case: 100 identifiers per request, against CI.
python -m benchmarks.publications --requests 200
# Load-bearing sweep: where does the objective stop holding?
python -m benchmarks.publications --ramp 1,4,8,16 --requests 200
# The JSON body form, and the cached case the specification recommends.
python -m benchmarks.publications --method POST
python -m benchmarks.publications --unique-ratio 0.0 --hot-pool 200
# Mixed identifiers, which take the per-identifier _msearch path instead of the batched _mget.
python -m benchmarks.publications --pmid-ratio 0.5
# Temporary one-deployment A/B selector. Every identifier batch goes to both implementations.
python -m benchmarks.publications --method POST --compare-lookup-strategies \
--identifier-file real-publication-ids.txt --concurrency 1 \
--requests 200 --warmup 20 --seed 2026
# Explicit ordered comparison: deltas are experiment minus control.
python -m benchmarks.publications --method POST \
--compare-strategies bulk-search combined-search \
--identifier-file real-publication-ids.txt --concurrency 1 \
--requests 200 --warmup 20 --seed 2026
# A single arm remains available for an absolute load-bearing sweep.
python -m benchmarks.publications --lookup-strategy combined-search --ramp 1,4,8,16 --requests 200
# Machine-readable output; exit status is 0 only if every stage met the objective.
python -m benchmarks.publications --json --slo-basis server--lookup-strategy sends the undocumented X-Publications-Lookup-Strategy benchmark header. The
endpoint defaults to current, holds independent current, bulk-search, and combined-search service
instances, and echoes the selection as _meta.lookup_strategy. The harness rejects a missing or mismatched
attribution rather than admitting that latency into the result. The endpoint also reports
_meta.lookup_fallback; comparison mode requires it to be the boolean false for both arms. A true marker
means the requested strategy fell back after rejecting its speculative Elasticsearch response, while a
missing marker usually means the deployment is stale or mixed. Either case invalidates the pair, so the
reported deltas measure only the requested normal paths. --compare-strategies CONTROL EXPERIMENT selects
any two distinct implementations in an explicit order. Every reported delta is experiment minus control,
so a negative value favors the experiment. The existing bare --compare-lookup-strategies flag remains
shorthand for --compare-strategies current bulk-search.
One comparison invocation precomputes each batch, sends that exact batch to both implementations, and
balances which strategy runs first after the batches are known. In comparison mode, --requests and
--warmup count pairs, so 200 measured pairs produce 400 HTTP requests. The measured pair count must be
even and at least two, ensuring both request orders are represented. --concurrency remains the maximum
number of HTTP requests in flight; the two members of a pair are sequential rather than doubling the
offered load.
The paired report labels both arms dynamically and keeps control-first and experiment-first deltas separate
as well as combined. That is necessary because the second request can benefit from Elasticsearch state
warmed by the first. Ordering is also balanced within the batches that exercise code differing between the
selected pair: DOI/PMCID batches for current versus bulk-search, PMID-containing batches for
bulk-search versus combined-search, and every nonempty batch for current versus combined-search.
Any batch containing a non-ASCII alternative identifier is a no-change control because all strategies use
the exact-search fallback. Alternative-identifier counts remain visible as workload information, while the
pair-specific changed-path count says whether the selected implementations actually differed. A stage with
zero changed-path pairs may retain an integrity pass, but its report explicitly says that it cannot select a
winner. This controls first-order bias, but one shared deployment cannot give both treatments a cold first
lookup. If the order strata disagree in direction, treat the run as cache/order-sensitive rather than
declaring a winner. The mixed A/B wall time is not a standalone capacity measurement; use the existing
single-strategy mode with --ramp for that. Comparison mode is intentionally closed-loop and cannot be
combined with --users. The selector is temporary measurement scaffolding and should be removed with the
losing implementation after the A/B.
Comparison mode requires --identifier-file with one real, resolving identifier per line and at least one
batch's worth of identifiers; blank lines and lines beginning with # are ignored. Prefer one identifier
per distinct document and a mix representing the traffic being decided. A 100%-PMID corpus is valid and is
the direct changed-path case for bulk-search versus combined-search; for current versus bulk-search
it is a useful no-change control. Synthesized DOI values mostly miss, so use real alternatives when the
alternative path matters. Any identifier reported as not_found by either arm invalidates its pair and
makes the comparison exit nonzero.
Every run reports two latencies per request, because they answer different questions and can disagree:
- server is the endpoint's own
_meta.processing_time_ms. It is what the service controls and is comparable from any vantage point, so it is what--slo-basisgates on by default. - client is wall-clock time to a fully-read response, which is what a UI actually experiences. It
includes the network path from wherever the benchmark runs. A full batch of 100 returns roughly
115 kB of abstracts, so from outside the cluster transit and transfer alone add 150–250 ms at p90 and
the client-side verdict fails on network cost rather than on service time. Run the benchmark from
inside the cluster, or from the UI's own vantage point, before reading a client-side number as an SLO
result. The report prints the measured
net p90difference so this is never invisible.
Four methodology choices are worth knowing about, because each one would otherwise flatter the result:
- Warmup is discarded. Each stage first issues throwaway requests to fill the keep-alive pool, so no measured sample is charged for a TLS handshake. The warmup uses a separate seeded draw; synthetic fresh identifiers therefore stay separate, while a fixed identifier pool can overlap and is reported as a cache-warming workload.
- Identifiers are drawn fresh by default.
--unique-ratio 1.0makes every lookup a first-time lookup, which is the cold-cache bound. Elasticsearch caches aggressively — the same batch replayed drops from roughly 70 ms to under 30 ms — so a benchmark that reused identifiers would report the best case as if it were typical.--verify-corpusis available for a known-present corpus, and any run using it is labelledcache_primed, because verification is itself a query that warms the cache. - The hit ratio is reported. PMIDs are drawn at random from the range PubMed has issued, so around
96% resolve. That matters because an all-
not_foundresponse is cheap to serve; a run whose identifiers mostly missed would show a fast p90 for work the service never did. - Percentiles are nearest-rank. An interpolated p90 reports a latency that was never observed. The report also prints the fraction of requests strictly under the threshold, which is the more literal reading of "90% of requests should take <150ms" and needs no interpolation convention to reproduce.
Full batches of 100 identifiers against https://annotator.ci.transltr.io, cold cache, server-side
latency in milliseconds:
| workload | p50 | p90 | p95 | p99 | verdict |
|---|---|---|---|---|---|
| 1 identifier | 5 | 7 | 7 | 8 | pass |
| 10 identifiers | 12 | 16 | 18 | 21 | pass |
100 identifiers, GET |
62 | 73 | 80 | 84 | pass |
100 identifiers, POST |
68 | 77 | 82 | 86 | pass |
| 100 identifiers, replayed (warm cache) | 27 | 49 | 55 | 62 | pass |
| 100 identifiers, 50% PMCID and DOI | 34 | 54 | 181 | 199 | pass at p90 |
The PMID fast path meets the objective with roughly half the budget to spare, and scales sublinearly
with batch size: a hundredfold larger batch costs about ten times a single lookup, which is the
batched _mget behaving as intended.
Sweeping concurrency, across three independent runs plus a single-identifier control:
| concurrency | run A | run B | run C | 1 id (control) |
|---|---|---|---|---|
| 8 | 46 rps / 52 ms | 37 rps / 45 ms | 35 rps / 44 ms | 85 rps / 6 ms |
| 16 | 81 rps / 83 ms | 85 rps / 74 ms | 63 rps / 78 ms | 156 rps / 6 ms |
| 24 | 109 rps / 106 ms | 100 rps / 109 ms | 86 rps / 77 ms | 184 rps / 6 ms |
| 32 | — | 106 rps / 152 ms | 99 rps / 129 ms | 221 rps / 6 ms |
Under smooth closed-loop load the objective holds to about 100 rps at 24 concurrent full batches, and the p90 reaches the 150 ms line around 32. The single-identifier control never leaves 6 ms, so nothing about the concurrency itself is hard: what costs is the response body, and a full batch is ~114 kB of which abstracts are 75% (titles 8%). Caddy compresses it 2.9x to ~42 kB on the wire.
A fourth run measured a knee at 12 concurrent, with the p90 reaching 720 ms at 16 — roughly an order of magnitude worse than the three runs above at the same concurrency, and with throughput plateauing rather than climbing. It did not reproduce. CI is a shared single-replica deployment and the benchmark ran from outside the cluster, so a run like that should be treated as an environment sample rather than a property of the service, and repeated before anything is concluded from it. Run the ramp at least twice before quoting a ceiling.
The concurrency figures above are a capacity instrument, not a headcount. A closed-loop worker issues
its next request the instant the previous one returns, so c=8 describes eight requests permanently in
flight — eight infinitely impatient robots, not eight people. A reader who spends half a minute looking
at a result page before triggering another lookup occupies the service for about 90 ms of those 30
seconds, and therefore offers roughly 0.03 requests per second rather than the ~11 a saturated worker
offers. The two differ by nearly three orders of magnitude, which is why one cannot be read as the
other.
Little's Law converts between them:
users = throughput x (service latency + think time)
So the 46 rps that CI sustains inside the objective corresponds to:
| mean think time | requests per user | supported readers |
|---|---|---|
| 5 s | 0.197 rps | ~230 |
| 10 s | 0.099 rps | ~460 |
| 30 s | 0.033 rps | ~1,380 |
| 60 s | 0.017 rps | ~2,750 |
The think time is an assumption about reader behaviour, not something the benchmark can measure, so the range is the answer rather than any single row. Even the pessimistic end is far above expected Translator UI traffic.
To measure a population directly instead of inferring one, use user mode:
# 300 readers, each pausing ~30 s between lookups, for 90 seconds.
python -m benchmarks.publications --users 300 --think-time 30 --duration 90User mode differs from the concurrency ramp in three ways that all matter:
- Think times are exponentially distributed, not fixed. A fixed gap keeps simulated users locked in whatever phase they started in, producing periodic bursts a real independent population does not have. Arrivals are also staggered across the first think-time window, so the run does not begin with a synchronized thundering herd.
- Users share one popularity-ranked catalogue (
--catalog-size,--zipf). Real readers' interests overlap heavily, and that overlap is what a backend cache actually gets to absorb. Independent random draws per user would hide it. Classic Zipf is the default;--zipf 0gives a uniform catalogue for comparison. - The report distinguishes offered from achieved rate. A population that was served without saturating the service bounds nothing from above, and the report says so rather than letting the achieved rate be mistaken for a ceiling.
Measured against CI at a 30 s think time, over 300 s so the joining transient has washed out:
| readers | offered | achieved | p50 | p90 | p99 | success | verdict |
|---|---|---|---|---|---|---|---|
| 300 | 10 rps | 12 rps | 31 | 42 | 54 | 100% | pass |
| 600 | 20 rps | 25 rps | 25 | 33 | 56 | 100% | pass |
| 1,200 | 40 rps | 42 rps | 22 | 28 | 38 | 100% | pass |
| 2,400 | 80 rps | 53 rps | 31 | 808 | 1554 | 100% | fail |
So roughly 1,200 concurrent readers fit inside the objective, with the p90 at 28 ms — a fifth of the budget. At 2,400 the population offers 80 rps, the service delivers 53, and the p90 goes to 808 ms.
Worth noting that 1,200 readers pass at a better p90 (28 ms) than 100 readers do (87 ms, p99 333 ms). That is not noise: at a low request rate the shared hot set falls out of cache between requests, so a lightly loaded service does more cold lookups than a moderately loaded one.
The population fails at 53 rps while the closed-loop ramp sustained ~100 rps, and the difference is burstiness, not a contradiction. Closed-loop load is perfectly smooth — exactly N requests in flight, never more. Exponential think times produce Poisson arrivals, whose instantaneous rate clumps well above its mean, and those clumps queue behind the 8 Sanic workers. Real traffic is bursty, so the lower number is the more realistic one and the smooth ramp should be read as an optimistic bound.
Around that rate the service is at its edge. One 100 rps run was served cleanly; another at the same
offered rate shed about 1.5% of requests as HTTP 500s, which is the two-second backend deadline
(DOCUMENT_METADATA_REQUEST_TIMEOUT) doing what it was built to do. That matters for how capacity is
read: failures leave the latency distribution, so a stage that sheds load posts a better p90 than one
that serves everything slowly. The benchmark gates its capacity figure on a 99% success rate for that
reason, and flags any stage that passes on latency while shedding.
One limit on all of the above. At these rates a single load generator becomes a plausible bottleneck of its own — 90 rps of 92 kB responses is around 8 MB/s to decompress and parse on one host. Server-side latency is measured in the handler and stays trustworthy, so the report uses it as the discriminator: a shortfall in achieved rate alongside a healthy server-side p90 and no errors points at the generator, and the report says so instead of crediting the service with a ceiling it was never pushed to. Settling it needs a run from inside the cluster, or a generator split across hosts.
The benchmark is also wrapped as an opt-in pytest module. The offline tests cover the harness itself — percentile arithmetic, verdict boundaries, and the accounting that keeps a failed or malformed response from scoring as a fast one — and run with the normal suite. The live tests generate real load and are gated:
RUN_PUBLICATIONS_LOAD_TEST=1 python -m pytest -q tests/test_publications_load.py -m performancePUBLICATIONS_LOAD_TEST_BASE_URL, PUBLICATIONS_LOAD_TEST_REQUESTS,
PUBLICATIONS_LOAD_TEST_CONCURRENCY, and PUBLICATIONS_LOAD_TEST_SLO_BASIS override the target and
the load. The ramp test reports the concurrency levels that held the objective rather than asserting a
specific one, because that number describes the deployment's current sizing rather than the service.
docker
├── configuration
│ ├── Caddyfile
│ └── supervisord.conf
└── DockerfileWe have a Dockerfile and service through docker-compose for the biothings-annotator service. The
Docker file lives in ~/docker/Dockerfile and defines two build stages. The first pulls down the
repository and creates a wheel for the python package. There are two optional arguments for
controlling the cloning process.
ANNOTATOR_REPOANNOTATOR_BRANCH
The second build stage sets up the docker environment. It installs packages and then creates the
annotator user and home environment. It then creates a virtual environment for the annotator user
and installs the wheels generated from the previous builder stage
The entrypoint is set as:
ENTRYPOINT ["supervisord"]
CMD ["-c", "/etc/supervisor/supervisord.conf"]
This leverages supervisord to launch to different services. The first is the annotator web server
[program:python_app]
command=/home/annotator/venv/bin/python -m biothings_annotator --conf=/home/annotator/configuration/default.jsonThis command will call the __main__.py entrypoint of the package itself. This should start the sanic web
service for hosting the annotation service. For configuration of the service itself, modify the
configuration found under biothings_annotator/application/configuration/sanic.json.
The second service launched is caddy. We use caddy in this case as a reverse proxy.
[program:caddy]
command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfileThe default port for the web server is 9001, and the caddy proxy forwards to that upstream web server via port
9000 for deployment purposes. The main purposes at the moment for caddy are for compression and
potential future usage as a load balancer
-
default
~/biothings_annotator$ docker build ./docker~/biothings_annotator$ docker build . -f docker/Dockerfile
-
build arguments
~/biothings_annotator$ docker build --build-arg ANNOTATOR_BRANCH=main ./docker/~/biothings_annotator$ docker build --build-arg ANNOTATOR_REPO=https://github.com/biothings/biothings_annotator.git ./docker/~/biothings_annotator$ docker build --build-arg ANNOTATOR_REPO=https://github.com/biothings/biothings_annotator.git --build-arg ANNOTATOR_BRANCH=main ./docker/
-
tag
~/biothings_annotator$ docker build ./docker/ --tag=annotator
-
without caching
~/biothings_annotator$ docker build ./docker/ --no-cache
-
default
~/biothings_annotator$ docker compose build
-
build arguments
~/biothings_annotator$ docker compose build --build-arg ANNOTATOR_REPO=https://github.com/biothings/biothings_annotator.git --build-arg ANNOTATOR_BRANCH=main ./docker/
- `~/biothings_annotator$ docker run <annotator-image-name>`
- `~/biothings_annotator$ docker run <annotator-image-name>`
The tests are implemented with pytest in mind.
To install the test dependencies pip install .[tests].
- Test Overview
pytest --setup-plan
(biothings_annotator) ~/biothings_annotator$ python3 -m pytest tests/ --setup-plan
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.10.12, pytest-8.2.2, pluggy-1.5.0 -- ~/biothings_annotator/bin/python3
cachedir: .pytest_cache
rootdir: ~/biothings_annotator
configfile: pyproject.toml
collected 51 items
tests/test_curie.py::test_curie_parsing[NCBIGene]
SETUP F curie_prefix['NCBIGene']
tests/test_curie.py::test_curie_parsing[NCBIGene] (fixtures used: curie_prefix)
TEARDOWN F curie_prefix['NCBIGene']
tests/test_curie.py::test_curie_parsing[ENSEMBL]
SETUP F curie_prefix['ENSEMBL']
tests/test_curie.py::test_curie_parsing[ENSEMBL] (fixtures used: curie_prefix)
TEARDOWN F curie_prefix['ENSEMBL']
tests/test_curie.py::test_curie_parsing[UniProtKB]
SETUP F curie_prefix['UniProtKB']
tests/test_curie.py::test_curie_parsing[UniProtKB] (fixtures used: curie_prefix)
TEARDOWN F curie_prefix['UniProtKB']
tests/test_curie.py::test_curie_parsing[INCHIKEY]
SETUP F curie_prefix['INCHIKEY']
tests/test_curie.py::test_curie_parsing[INCHIKEY] (fixtures used: curie_prefix)
TEARDOWN F curie_prefix['INCHIKEY']
...