A Resilient Framework for Detecting and Managing Schema Drift in Evolving Microservice Contracts
Keywords:
Schema Drift, Microservices, Contract Evolution, Machine Learning, API Compatibility, Predictive Analytics, Service Reliability.Abstract
Microservice architectures have allowed organizations to build scalable and independently deployable software systems as a result of their rapid adoption. The ongoing development of services brings about schema drift, whereby modifications in data formatting, API contracts, and service interfaces cause compatibility issues between interdependent elements. The current methods primarily revolve around the reactive validation and failure detection, and have little ability to predict the possibility of contract evolution risks. The study presents a robust model of schema drift detection and control in changing contracts of microservice predictive control based on machine learning. The secondary data composed of 10,000 records of microservice contract evolutions is processed by using data analytics and machine learning with Python. Preprocessing data, feature engineering, and exploratory analysis are conducted to determine key factors of schema drift. The algorithm of Random Forest and Gradient Boosting are used in order to categorize the risks of schema drift as low-risk, medium-risk, and high-risk. The experimental assessment shows that the accuracy and F1-score of the Random Forest are better, with 91.40% and 91.31%, respectively, which is better than that of the Gradient Boosting. The analysis of feature importance demonstrates that the failure count, compatibility status, and version difference are significant factors causing the occurrence of schema drift. The framework offered is a proactive solution to enhance the success of contracts, minimize compatibility breakdowns, and enable ongoing transformation of distributed microservice systems.
Downloads
References
Taylor, K. and Reza, A., 2018. Automated Data Quality Monitoring in Streaming Data Transformations. International Journal of Data Engineering and Intelligent Computing, 1(2), pp.01-12. https://worldcometresearchgroup.com/index.php/ijdeic/article/download/557/537
Nkosi, T., 2019. Continuous Data Transformation in Event-Driven Microservices. International Journal of Artificial Intelligence & Digital Transformation, 2(2), pp.01-14. https://worldcometresearchgroup.com/index.php/ijaidt/article/download/202/192
Lan, N.T., 2021. Resilient Data Ingestion Patterns in Cloud-Native Data Mesh Architectures. International Journal of Artificial Intelligence & Digital Transformation, 4(2), pp.01-12. https://worldcometresearchgroup.com/index.php/ijaidt/article/download/224/214
Apolinário, D.R. and de França, B.B., 2021. A method for monitoring the coupling evolution of microservice-based architectures. Journal of the Brazilian Computer Society, 27(1), p.17. https://link.springer.com/content/pdf/10.1186/s13173-021-00120-y.pdf
Singh, D., 2022. Managing API Evolution in Large-scale Microservices: Versioning and Backward Compatibility. International Journal of Science, Technology and Convergence, 4(4). https://ijcdra.us/index.php/IJSTC/article/download/61/8
Chopra, A.K., 2022. Interaction-oriented software engineering: Programming abstractions for autonomy and decentralization. AI Communications, 35(4), pp.381-391. https://www.lancaster.ac.uk/staff/chopraak/pdfs/chopra-vision.pdf
Tonelli, R., Lunesu, M.I., Pinna, A., Taibi, D. and Marchesi, M., 2019, February. Implementing a microservices system with blockchain smart contracts. In 2019 IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE) (pp. 22-31). IEEE. https://iris.unica.it/bitstream/11584/262865/1/Microservices_blockchain-preprint.pdf
Wang, S., Zhang, X., Yu, W., Hu, K. and Zhu, J., 2020, July. Smart contract microservitization. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC) (pp. 1569-1574). IEEE. https://iris.unica.it/bitstream/11584/262865/1/Microservices_blockchain-preprint.pdf
Nagothu, D., Xu, R., Nikouei, S.Y. and Chen, Y., 2018, September. A microservice-enabled architecture for smart surveillance using blockchain technology. In 2018 IEEE international smart cities conference (ISC2) (pp. 1-4). IEEE. https://arxiv.org/pdf/1807.07487
Sirimalla, A., 2022. End-to-end automation for cross-database DevOps deployments: CI/CD pipelines, schema drift detection, and performance regression testing in the cloud. World Journal of Advanced Research and Reviews, 14(3), pp.871-889. https://doi.org/10.30574/wjarr.2022.14.3.0555
Keshireddy, S.R. and Kavuluri, H.V.R., 2019. Design of Fault Tolerant ETL Workflows for Heterogeneous Data Sources in Enterprise Ecosystems. International Journal of Communication and Computer Technologies, 7(1), pp.42-46. https://www.eccsubmit.com/index.php/ijccts/article/download/134/335
Kotla, M.R.T., 2023. Autonomous enterprise integration: The future of self-healing data and API ecosystems. International Journal of Research and Applied Innovations, 6(3), pp.5968-5971. https://www.ijrai.org/index.php/ijrai/article/download/599/550
Bhaskaran, S.V., 2020. Integrating data quality services (dqs) in big data ecosystems: Challenges, best practices, and opportunities for decision-making. Journal of Applied Big Data Analytics, Decision-Making, and Predictive Modelling Systems, 4(11), pp.1-12. https://polarpublications.com/index.php/JABADP/article/download/4/4
Liu, Z., Fan, G., Yu, H. and Chen, L., 2021. An Approach to Modeling and Analyzing Reliability for Microservice‐Oriented Cloud Applications. Wireless Communications and Mobile Computing, 2021(1), p.5750646. https://onlinelibrary.wiley.com/doi/pdf/10.1155/2021/5750646
Chirumamilla, K.R., 2023. Predicting data contract failures using machine learning. The Eastasouth Journal of Information System and Computer Science, 1(01), pp.144-155. https://esj.eastasouth-institute.com/index.php/esiscs/article/download/843/657
Singh, D., 2022. Managing API Evolution in Large-scale Microservices: Versioning and Backward Compatibility. International Journal of Science, Technology and Convergence, 4(4). https://ijcdra.us/index.php/IJSTC/article/download/61/8
Egelund-Müller, B., Elsman, M., Henglein, F. and Ross, O., 2017. Automated execution of financial contracts on blockchains. Business & Information Systems Engineering, 59(6), pp.457-467. https://papers.ssrn.com/sol3/Delivery.cfm?abstractid=2898670
Sánchez-Gómez, N., Torres-Valderrama, J., García-García, J.A., Gutiérrez, J.J. and Escalona, M.J., 2020. Model-based software design and testing in blockchain smart contracts: A systematic literature review. IEEE Access, 8, pp.164556-164569. https://ieeexplore.ieee.org/iel7/6287639/6514899/09186040.pdf
Xu, D., Xu, W., Tu, M., Shen, N., Chu, W. and Chang, C.H., 2015. Automated integration testing using logical contracts. IEEE Transactions on Reliability, 65(3), pp.1205-1222. https://www.researchgate.net/profile/Manghui-Tu-2/publication/283822941_Automated_Integration_Testing_Using_Logical_Contracts/links/5e4472b0458515072d96d050/Automated-Integration-Testing-Using-Logical-Contracts.pdf
Lee, Y., Lee, Y., Lee, E. and Lee, T., 2023. Explainable artificial intelligence-based model drift detection applicable to unsupervised environments. Computers, Materials and Continua, 76(2), pp.1701-1719. https://www.sciencedirect.com/org/science/article/pii/S1546221823001558
Bukhari, T.T., Oladimeji, O., Etim, E.D. and Ajayi, J.O., 2022. Systematic review of metadata-driven data orchestration in modern analytics engineering. Gyanshauryam, International Scientific Refereed Research Journal, 5(4), pp.536-564. https://gisrrj.com/paper/GISRRJ225429.pdf
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J. and Zhang, G., 2018. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12), pp.2346-2363. https://arxiv.org/pdf/2004.05785
Yu, H., Zhang, Q., Liu, T., Lu, J., Wen, Y. and Zhang, G., 2022. Meta-ADD: A meta-learning based pre-trained model for concept drift active detection. Information Sciences, 608, pp.996-1009. https://www.sciencedirect.com/science/article/pii/S0020025522007125
Alonso, J., Orue-Echevarria, L., Osaba, E., López Lobo, J., Martinez, I., Diaz de Arcaya, J. and Etxaniz, I., 2021. Optimization and prediction techniques for self-healing and self-learning applications in a trustworthy cloud continuum. Information, 12(8), p.308. https://www.mdpi.com/2078-2489/12/8/308
Nallaperuma, D., Nawaratne, R., Bandaragoda, T., Adikari, A., Nguyen, S., Kempitiya, T., De Silva, D., Alahakoon, D. and Pothuhera, D., 2019. Online incremental machine learning platform for big data-driven smart traffic management. IEEE Transactions on Intelligent Transportation Systems, 20(12), pp.4679-4690. https://ieeexplore.ieee.org/iel7/6979/4358928/08759919.pdf
Khan, M.N.I., 2022. A Systematic Review of Legal Technology Adoption In Contract Management, Data Governance, And Compliance Monitoring. American Journal of Interdisciplinary Studies, 3(01), pp.01-30. https://ajisresearch.com/index.php/ajis/article/download/27/26
Chu, X., Ilyas, I.F., Krishnan, S. and Wang, J., 2016, June. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data (pp. 2201-2206).https://dl.acm.org/doi/pdf/10.1145/2882903.2912574
Wang, T.C., Guo, R.S. and Chen, C., 2023. An integrated data-driven procedure for product specification recommendation optimization with LDA-LightGBM and QFD. Sustainability, 15(18), p.13642. https://www.mdpi.com/2071-1050/15/18/13642
Leonelli, S., 2016. Locating ethics in data science: responsibility and accountability in global and distributed knowledge production systems. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2083), p.20160122. https://pmc.ncbi.nlm.nih.gov/articles/PMC5124067/pdf/rsta20160122.pdf
Munikoti, S., Agarwal, D., Das, L., Halappanavar, M. and Natarajan, B., 2023. Challenges and opportunities in deep reinforcement learning with graph neural networks: A comprehensive review of algorithms and applications. IEEE transactions on neural networks and learning systems, 35(11), pp.15051-15071. https://arxiv.org/pdf/2206.07922
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
