Q: As your Backstage catalog grew past 8,000 components and 25,000 TechDocs pages, developer searches began timing out. The Backstage backend CPU spikes to 100% due to the default in-memory Lunr search engine. How do you migrate and tune Backstage search using ElasticSearch?
Scaling Backstage search across 10,000 components, API specs, and TechDocs pages using ElasticSearch to resolve Lunr in-memory search timeouts.
@backstage/plugin-search-backend-module-elasticsearch engine with dedicated indices for Catalog, TechDocs, and APIs."
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: The default Lunr search engine in Backstage builds indexes in Node.js heap memory, crashing under large enterprise catalogs. Production enterprise search requires deploying the @backstage/plugin-search-backend-module-elasticsearch engine with dedicated indices for Catalog, TechDocs, and APIs.
Deploy ElasticSearch / OpenSearch Cluster on Kubernetes
Deploy a managed or operator-backed OpenSearch cluster with dedicated master and data nodes, enabling TLS and role-based access.
# app-config.yaml
search:
engine:
type: elasticsearch
elasticsearch:
node: https://opensearch.platform.acme.internal:9200
auth:
username: ${OPENSEARCH_USER}
password: ${OPENSEARCH_PASSWORD}
Configure Dedicated Document Collators and Incremental Indexing
In packages/backend/src/plugins/search.ts, register the DefaultCatalogCollatorFactory and DefaultTechDocsCollatorFactory with batch sizes (e.g. 500 documents) and staggered schedules.
indexBuilder.addSearchEngine({
engine: ElasticSearchSearchEngine.fromConfig({ config }),
});
indexBuilder.addCollator({
schedule: env.scheduler.createScheduledTaskRunner({ frequency: { minutes: 15 }, timeout: { minutes: 5 } }),
factory: DefaultCatalogCollatorFactory.fromConfig(config, { discovery: env.discovery }),
});
Tune Search Relevance and Tokenizers
Configure ngram tokenizers in OpenSearch for partial matching so developers finding auth-svc matches authentication-service accurately.
- Migrate from default in-memory Lunr engine to OpenSearch/ElasticSearch for enterprise catalog scale.
- Configure staggered collator schedules to index TechDocs and Catalog entities asynchronously.
- Implement partial-match ngram tokenizers to improve service discovery and developer search UX.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Internal Developer Portal Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
- Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Internal Developer Portal Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
- Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Internal Developer Portal Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
- Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Internal Developer Portal Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.