Q: As your Backstage catalog grew past 8,000 components and 25,000 TechDocs pages, developer searches began timing out. The Backstage backend CPU spikes to 100% due to the default in-memory Lunr search engine. How do you migrate and tune Backstage search using ElasticSearch?
Scaling Backstage search across 10,000 components, API specs, and TechDocs pages using ElasticSearch to resolve Lunr in-memory search timeouts.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Deploy ElasticSearch / OpenSearch Cluster on Kubernetes
Deploy a managed or operator-backed OpenSearch cluster with dedicated master and data nodes, enabling TLS and role-based access.
# app-config.yaml
search:
engine:
type: elasticsearch
elasticsearch:
node: https://opensearch.platform.acme.internal:9200
auth:
username: ${OPENSEARCH_USER}
password: ${OPENSEARCH_PASSWORD}Configure Dedicated Document Collators and Incremental Indexing
In `packages/backend/src/plugins/search.ts`, register the `DefaultCatalogCollatorFactory` and `DefaultTechDocsCollatorFactory` with batch sizes (e.g. 500 documents) and staggered schedules.
indexBuilder.addSearchEngine({
engine: ElasticSearchSearchEngine.fromConfig({ config }),
});
indexBuilder.addCollator({
schedule: env.scheduler.createScheduledTaskRunner({ frequency: { minutes: 15 }, timeout: { minutes: 5 } }),
factory: DefaultCatalogCollatorFactory.fromConfig(config, { discovery: env.discovery }),
});Tune Search Relevance and Tokenizers
Configure ngram tokenizers in OpenSearch for partial matching so developers finding `auth-svc` matches `authentication-service` accurately.
- Migrate from default in-memory Lunr engine to OpenSearch/ElasticSearch for enterprise catalog scale.
- Configure staggered collator schedules to index TechDocs and Catalog entities asynchronously.
- Implement partial-match ngram tokenizers to improve service discovery and developer search UX.