Integrating Event Streams into Data Mesh Architectures
Keywords:
Data Mesh, Event Streaming, Kafka, Real-Time Data, Distributed Systems, Data ArchitectureAbstract
The integration of real-time event streaming with domain-oriented data mesh architectures marks a critical evolution in distributed data systems. Traditional analytical platforms rely heavily on batch processing, resulting in latency and reduced responsiveness to business needs. Event streaming platforms, by contrast, deliver continuous and low-latency data flows. This paper examines the theoretical underpinnings and practical methodologies for embedding event streams as first class entities within a federated data mesh. We propose a comprehensive integration model that addresses schema evolution, metadata governance, and cross-domain interoperability while maintaining the decentralized ethos of the data mesh. The proposed architecture is validated through an experimental deployment involving Apache Kafka, Apache Flink, and Trino, with a federated metadata and governance layer implemented via DataHub. Empirical evaluation in a simulated financial transaction environment demonstrates significant improvements in data latency, consumer onboarding efficiency, and schema evolution stability, highlighting both the potential and complexity of this convergence.
Downloads
References
A. Bellemare, “Data mesh architectures with event streams,” Online article, 2021. [Online]. Available: https://www.confluent.io/resources/ ebook/data-mesh-architectures-with-event-streams/
S. Blog, “Rethink your data architecture with data mesh and event streams,” Blog post, 2021. [Online]. Available: https://www.striim.com/
blog/data-mesh-event-stream-architecture/
A. A. Munshi and Y. A.-R. I. Mohamed, “Data lake lambda architecture for smart grids big data analytics,” IEEE Access, vol. 6, pp. 40463– 40471, 2018.
Zhou, Q., Simmhan, Y. and Prasanna, V., 2017. Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams. Future Generation Computer Systems, 76, pp.391-406.
Moradi, S., Qiao, N., Stefanini, F. and Indiveri, G., 2017. A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs). IEEE transactions on biomedical circuits and systems, 12(1), pp.106-122.
Saxena, S. and Gupta, S., 2017. Practical real-time data processing and analytics: distributed computing and event processing using Apache Spark, Flink, Storm, and Kafka. Packt Publishing Ltd.
Bellavista, P., Giannelli, C., Lagkas, T. and Sarigiannidis, P., 2018. Quality management of surveillance multimedia streams via federated sdn controllers in fiwi-iot integrated deployment environments. IEEE Access, 6, pp.21324-21341.
Mandala, V., 2017. Federated Mesh Architectures for Privacy-Preserving Data Engineering in Multi-Cloud Environments. Global Research Development (GRD) ISSN: 2455-5703, 2(12).
M. J. Divan´ and M. L. Sanchez´ Reynoso, An Architecture for the Real-Time Data Stream Monitoring in IoT. Singapore: Springer Singapore, 2020, pp. 59–100. [Online]. Available: https: //doi.org/10.1007/978-981-13-8759-3 3
S. Intorruk and T. Numnonda, “A comparative study on performance and resource utilization of real-time distributed messaging systems for big data,” in 2019 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 2019, pp. 102–107.
G. van Dongen and D. Van den Poel, “Evaluation of stream processing frameworks,” IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 8, pp. 1845–1858, 2020.
K. Wahner,¨ “The heart of the data mesh beats real-time with apache kafka,” Blog post, 2022.
[Online]. Available: https://www.kai-waehner.de/blog/2022/07/28/ the-heart-of-the-data-mesh-beats-real-time-with-apache-kafka/
N. T. Blog, “Data mesh: a data movement and processing platform @ netflix,” Blog post,
[Online]. Available: https://netflixtechblog.com/
data-mesh-a-data-movement-and-processing-platform-netflix-1288bcab2873
S. T and S. N. K, “A study on modern messaging systemskafka, rabbitmq and nats streaming,” 2019. [Online]. Available:
https://arxiv.org/abs/1912.03715
K. Wahner, “The heart of the data mesh beats real-time with apache¨ kafka,” DZone article, 2022. [Online]. Available: https://dzone.com/ articles/the-heart-of-the-data-mesh-beats-real-time-with-ap
B. R. Hiraman, C. Viresh M., and K. Abhijeet C., “A study of apache kafka in big data stream processing,” in 2018 International Conference on Information , Communication, Engineering and Technology (ICICET), 2018, pp. 1–3.
L. K. Boyanov, “Financial data processing in big data platforms,” Economic Alternatives, vol. 27, no. 4, pp. 534–546, 2021. [Online].
Available: https://www.unwe.bg/doi/eajournal/2021.4/EA.2021.4.03.pdf
F. Wang, M. Zhang, X. Wang, X. Ma, and J. Liu, “Deep learning for edge computing applications: A state-of-the-art survey,” IEEE Access, vol. 8, pp. 58322–58336, 2020.
A. Theorin, K. Bengtsson, J. Provost, M. Lieder, C. Johnsson, T. Lundholm, and B. Lennartson, “An event-driven manufacturing information system architecture for industry 4.0,” International Journal of Production Research, vol. 55, no. 5, pp. 1297–1311, 2017. [Online]. Available: https://doi.org/10.1080/00207543.2016.1201604
B. Blamey, A. Hellander, and S. Toor, “Apache spark streaming, kafka and harmonicio: A performance benchmark and architecture comparison for enterprise and scientific computing,” 2019. [Online]. Available: https://arxiv.org/abs/1807.07724
Devireddy, R.R. 2020. Real-Time Data Processing in Data Warehousing: Integrating SQL Warehouses with In-Memory Analytics. International Journal of Enhanced Research in Science, Technology & Engineering (IJERSTE), 9(11), pp.11–17. ISSN 2319-7463
Das, S.S. 2020. Optimizing Employee Performance through Data-Driven Management Practices. European Journal of Advances in Engineering and Technology (EJAET), 7(1), pp.76–81.
Paruchuri, V.B. 2021. Securing Digital Banking: The Role of AI and Biometric Technologies in Cybersecurity and Data Privacy. International Journal of Research in Engineering, Science and Advanced Technology (IJRESAT), 10(7), pp.128–133. ISSN 2456–5083.
Z. Dehghani, “The evolution of streaming architectures: The case of data mesh,” in Streaming Audio Signal Processing Workshop, 2021. [Online]. Available: https://martinfowler.com/articles/data-mesh-principles.html
Roffia, L., Azzoni, P., Aguzzi, C., Viola, F., Antoniazzi, F. and Salmon Cinotti, T., 2018. Dynamic linked data: A SPARQL event processing architecture. Future Internet, 10(4), p.36.
Paruchuri, V.B. 2020. Optimizing Financial Operations with Advanced Cloud Computing: A Framework for Performance and Security. International Journal of Enhanced Research in Science, Technology & Engineering (IJERSTE), 9(9), pp.45–49. ISSN 2319–7463.
H. Isah, T. Abughofa, S. Mahfuz, D. Ajerla, F. Zulkernine, and S. Khan, “A survey of distributed data stream processing frameworks,” IEEE Access, vol. 7, pp. 154300–154316, 2019.
R. Sousa, R. Miranda, A. Moreira, C. Alves, N. Lori, and J. Machado, “Software tools for conducting real-time information processing and visualization in industry: An up-to-date review,” Applied Sciences, vol. 11, no. 11, 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/11/4800
Indrasiri, K. and Siriwardena, P., 2018. APIs, Events, and Streams. In Microservices for the Enterprise: Designing, Developing, and Deploying (pp. 293-312). Berkeley, CA: Apress.
Aguzzi, C., Antoniazzi, F., Azzoni, P., Bononi, L., Brasini, F., Canegallo, R., D'Elia, A., De Lisa, A., Di Felice, M., Franchi, E. and Perilli, L., 2018, November. From Heterogeneous Sensor Networks to Integrated Software Services: Design and Implementation of a Semantic Architecture for the Internet of Things at ARCES@ UNIBO. In 2018 23rd Conference of Open Innovations Association (FRUCT) (pp. 10-18). IEEE.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
