Runbook: Rebuild the Solr data cores

How to rebuild the Solr search cores from the database. Companion to advaita-service-solr. Source: ADV-273 (Radu + Graeme investigation).

What this does

The search index (ars-dih) and autocomplete index (ars-suggester) are built from the ars PostgreSQL database by Solr's DataImportHandler (DIH). Rebuilding the cores runs the DIH full-import, which executes the configured SQL against ars and writes a fresh Lucene index to the core's data directory on EFS.

Cores and where they live

• Cores: ars-dih (search) and ars-suggester (autocomplete).
• On EFS: /mnt/efs[-rw]/cache/v<VERSION>/solr/ — e.g. legacy v1906 (2019). Each version dir holds ars-dih/, ars-suggester/, configsets/, solr.xml, zoo.cfg.
• Per core: ars-dih/solr/ars-dih/conf/solrconfig.xml + solr-data-config.xml, and conf/dataimport.properties (records <entity>.last_index_time).

How the import is wired

• solrconfig.xml registers <requestHandler name="/dataimport" class="…DataImportHandler"> with config = solr-data-config.xml.
• solr-data-config.xml holds the JDBC <dataSource> (driver org.postgresql.Driver, url jdbc:postgresql://<ars-rds>:5432/ars?sslmode=require, user/password — hardcoded in the file) and <document name="advaita_search"> with one <entity> per type (gene, pathway, microrna, drug, go, disease). Each entity is an SQL query mapped to Solr fields via TemplateTransformer.

DIH entity model (verified from the live v1906 cores, 2026-06-17)

Read from the legacy cores at /mnt/cores-src through the bastion. Engine is Solr 5 (luceneMatchVersion 5.0.0); each core registers <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> with config = solr-data-config.xml. solr.xml is SolrCloud-flavored (<solrcloud>, hostPort 8983), and the cores run by core discovery (a core.properties per core).

Data source (both cores). jdbc:postgresql://clouddev.c5ikqcjohlrx.us-west-2.rds.amazonaws.com:5432/ars?sslmode=require (prod: prod-cloud…), user ars, password hardcoded in solr-data-config.xml (redacted here — externalize on the git move). Queries read the ars.* schema and filter joined database rows to status = 'PRODUCTION'.

ars-dih (search). Five top-level entities, each emitting one Solr doc with id = '<type>:'||<dbid>, identifier, name, description, organism_info / organism_name, type, num_relationships, plus relationship scores from nested subqueries:

• gene (num_relationships 5) — nested pathway, go_term, mirna, disease, drug info + *_count scores.
• pathway (4) — nested gene, drug, mirna, disease.
• disease (3) — nested gene, pathway, drug.
• mirna (2) — nested gene, pathway.
• go (1) — nested gene.

Each *_count subquery is count(DISTINCT …) / ${dataimporter.request.total} * 100.0 written to a <x>_relationship_score field (pathway/go_term/mirna/disease/drug/gene). These are the fields the edismax bf boost sums in solr-client / biob.

ars-suggester (autocomplete). Term entities producing term + the same relationship scores for type-ahead: gene_name, gene_symbol, go_name, go_symbol, pathway, disease, mirna (per dataimport.properties).

Last full import. ars-dih 2020-10-27, ars-suggester 2020-10-27 (dev v1906); a few suggester entities still carry 2015 stamps. Prod last rebuilt Feb 2022 (ADV-273).

Where the data lives. Read-only source: /mnt/cores-src (legacy s3cache cache/v1906/solr). The clone that solr-bddev:6 serves read-write is /mnt/efs/cores-test/solr on the shared EFS, mounted at /data/solr. A v19.4 build lands in a new cores tree on the shared EFS.

Prerequisites

• The Solr task runs with read-write EFS on the cores tree (Solr 5 needs a write.lock).
• Single writer only — stop the legacy solr before importing on the Fargate copy. One RW cores tree, never two writers.
• The Solr container can reach the ars PostgreSQL RDS on 5432.
• Admin / API reachable at solr.bddev / solr-dev.bddev:8443 (per ADV-269).

Verified on the migrated tier (bddev, 2026-06-17). solr-bddev:6 runs core/solr:5.0.0-fargate. The cores mount solr-cores → /data/solr is read-write (IAM access point fsap-07a8a2b756381f5ba on the shared EFS fs-072246d867e8a8c71 — the ADV-269 cores clone), so DIH can take the write.lock. The container carries no env; Solr config is baked into the image. DIH data source ars: dev RDS clouddev.c5ikqcjohlrx.us-west-2.rds.amazonaws.com:5432/ars (Postgres 15.17), prod prod-cloud.c5ikqcjohlrx.us-west-2.rds.amazonaws.com — matching the hardcoded JDBC url in solr-data-config.xml.

Rebuild an existing core

• Confirm solr-data-config.xml points at the correct ars database and the SQL matches the current schema.
• Trigger a full import per core:- GET /solr/ars-dih/dataimport?command=full-import&clean=true&commit=true
• GET /solr/ars-suggester/dataimport?command=full-import&clean=true&commit=true
• Monitor: GET /solr/<core>/dataimport?command=status until indexing completes; check dataimport.properties last_index_time.
• Verify: doc count GET /solr/<core>/select?q=*:*&rows=0 and a sample query.

Create v19.4 cores

Current cores are v1906 (2019); prod was last rebuilt Feb 2022. To produce v19.4:

• Copy a core's conf from the configset / an existing version into a v19.4 cores directory.
• Update solr-data-config.xml to target the v19.4 ars data.
• Register the cores (solr.xml / core discovery) and run the full-import above.

Back up the configs

The core configs exist only on EFS. Back them up and move them to a GitHub repository (planned in ADV-273) so they are version-controlled: solrconfig.xml, solr-data-config.xml, schema, configsets.

Cautions

• DB credentials are hardcoded in solr-data-config.xml — externalize when the configs move to git.
• clean=true deletes the existing index before import. For a no-downtime rebuild, import into a copy and swap.
• One writer only; the RW cores tree is shared with the legacy file system.

Open / TBD

• v19.4 data source: the same ars DB refreshed, or a distinct v19.4 schema/DB. TBD.
• Exact admin auth for the :8443 route. TBD.
• Solr 9 (target engine) removed the built-in DataImportHandler from core (it became a community package). The rebuild method for the Solr 5 → 9 step needs revisiting — DIH package vs an alternative reindex path. TBD.