Multi-Agent Orchestration for Autonomous Data Pipeline Governance: Schema Evolution, Anomaly Detection, and Incident Remediation in Cloud-Native Data Platforms - Praveen Kumar Dora Mallareddi (International Journal of Intelligent Systems and Applications in Engineering, 2026)
The rapid adoption of cloud-native data platforms has enabled organizations to scale data processing pipelines to unprecedented levels. However, governance mechanisms—particularly around schema evolution, anomaly detection, and incident remediation—remain largely manual, leading to increased operational risk and degraded data reliability. This paper proposes a novel multi-agent orchestration framework for autonomous data pipeline governance. The system leverages specialized agents for schema monitoring, anomaly detection, service-level agreement (SLA) tracking, and incident remediation, coordinated through a shared state and communication protocol. Evaluated against production-like workloads, the framework demonstrates significant improvements in detection latency, mean time to resolution (MTTR), and system reliability. The results suggest that agentic AI can address critical governance gaps in modern data infrastructures while maintaining safety through controlled autonomy. DOI: https://doi.org/10.17762/ijisae.v14i1.8344