We Handled a 200GB/day Log Volume Without Breaking the Bank: A Practical Framework for Cost-Effective Observability at Scale - Rahul Bhatia (International Journal of Intelligent Systems and Applications in Engineering, 2026)
Scaling observability in a cost-effective way is one of the most pressing challenges facing modern engineering teams. As distributed systems grow in complexity and traffic, daily log volumes can reach hundreds of gigabytes, creating a compounding burden on storage infrastructure, indexing engines, and operational budgets. This article presents a practitioner-driven case study documenting the architectural decisions, tooling evaluations, and pipeline optimizations used to manage a sustained high-volume logging workload without exhausting financial resources or degrading system visibility. The strategies explored include log filtering and structured enrichment at the collection edge, dynamic data tiering, field-selective indexing, retention policy governance, and license-aware tooling decisions. In this article, they are contextualized against established literature in distributed systems observability, telemetry pipeline design, and cloud cost optimization. The leads are in a repeatable, layered framework applicable to platform engineers, site reliability engineers, and infrastructure architects responsible for managing large-scale telemetry in production environments. DOI: https://doi.org/10.17762/ijisae.v14i1s.8234