Replace in-memory stores with persistent/distributed storage for multi-instance deployment compatibility #261
Description
Summary
Several components use isolated in-memory maps as runtime stores. These stores are not suitable for production multi-instance deployments because:
- All state is lost on service restart
- State is not shared across instances (horizontal scaling breaks job polling)
- Stores grow unboundedly over time (memory exhaustion risk)
This issue was identified during code review of PR #256 (comment: #256 (comment)), requested by @anjuchamantha.
Affected In-Memory Stores
| File | Variable | Type | Risk |
|---|---|---|---|
fhir-service/bulk_member_match.bal (line 31) |
bulkMatchJobStore |
map<BulkMemberMatchJob> |
Async job state lost on restart; not visible across instances |
fhir-service/davinci_data_export.bal (line 31) |
davinciExportJobStore |
map<DaVinciExportJob> |
Async export job state lost on restart; not visible across instances |
fhir-service/service.bal (line 1504) |
consentStore |
map<Consent> |
Consent decisions lost on restart; inconsistent across instances |
file-service/inMemoryStorage.bal (line 17) |
exportResults |
map<ExportTask> |
Export task results lost on restart |
bulk-export-client/inMemoryStorage.bal (line 17) |
exportTasks |
map<ExportTask> |
Export task state lost on restart |
Recommended Actions
- Short-term: Ensure TTL-based eviction is consistently applied to all job stores (currently partially implemented in
bulk_member_match.balanddavinci_data_export.bal, but not inconsentStoreor thefile-service/bulk-export-clientstores). - Medium-term: Add a maximum capacity guard with eviction of oldest/failed entries before inserting new ones.
- Long-term: Replace in-memory maps with a persistent store (e.g., a database) or a distributed cache (e.g., Redis) to support multi-instance deployments and survive restarts.
Activity