SREcon25 Americas

Leonardo Antônio dos Santos, Workday, Inc.


Managing Elasticsearch at tens of petabyte scale requires innovative approaches to overcome the limits of traditional single-cluster designs. In this talk, we introduce a scalable, cost-effective multi-cluster architecture that handles trillions of indexed logs monthly while reducing operational complexity. By shifting to a "Cluster of Clusters" design, we optimize ingestion, search, and cross-cluster search traffic using a centralized management cluster and standardized data clusters.

Key highlights include leveraging a custom cluster health service based on the USE Method for intelligent query routing, implementing real-time auditing for problematic query detection, and automating rate-limiting for high-demand users. Attendees will learn how these strategies cut compute costs by 57%, achieved significant storage savings, and enhanced scalability and migration efficiency.

This session provides practical insights, benchmarks, and real-world examples to help organizations sustainably optimize Elasticsearch while maintaining performance and reducing costs — which is ideal for those overseeing large-scale log data or anticipating Elasticsearch growth.


https://www.usenix.org/conference/srecon25americas/presentation/santos