Optimizing Big Data Processing Workflows using PySpark and Google Cloud Platform: A Performance Evaluation of Data Locality and Caching Strategies - Thulasiram Yachamaneni (International Journal of Intelligent Systems and Applications in Engineering, 2024)
The increasing volume and complexity of Big Data have led to the development of distributed processing frameworks such as Apache Spark, particularly its Python interface, PySpark, which allows for large-scale data processing in cloud environments. This paper investigates the optimization of data locality and caching strategies to improve the performance and scalability of Big Data workflows running on Google Cloud Platform (GCP). Through a series of experiments, various configurations of data placement, replication, and caching techniques—both in-memory and disk-based—were evaluated for their impact on key performance metrics, including execution time, latency, and throughput. The study also assesses the scalability of workflows as data sizes increase, identifying the configurations that allow PySpark workflows to handle growing datasets efficiently. The results reveal that optimized data locality, combined with well-tuned caching strategies, can significantly improve performance and scalability, offering a pathway for businesses to enhance their cloud-based Big Data systems. Furthermore, the findings provide valuable insights for organizations seeking to reduce costs, accelerate decision-making, and improve the efficiency of their data processing workflows. This paper contributes to the ongoing efforts to optimize distributed Big Data processing frameworks in cloud environments and offers practical guidelines for configuring PySpark workflows for maximum performance.