With the increased usage of the in-memory distributed computation framework Apache Spark, tools are needed to study, predict, and better understand the performance of a given algorithm in a specific cluster of computers. The execution time, required memory, and input/output (I/O) cost are the most important measurements. The Spark cluster is made up of a master node and worker nodes laid out as multiple sequential stages. The data is split for distributed and parallel processing in this layout. The nodes can in turn vary in their respective computing and memory characteristics as well as the possible random uncertainty in their availability. The amount of data processed is a critical factor as well for performance tuning.

The authors explore a simulation-based performance prediction tool. The two options attempted were to either reduce the amount of data processed such that each step in the cluster has at least one block of data to process or to experiment with a smaller computing cluster. The results of the experiments are extrapolated to determine how the tasks will perform in the real configuration. To reduce the bias in the selection of the data sample for the experiments, the available data is divided into multiple subsets and each subset has equal probability of inclusion in the simulation execution. This approach helps us better understand the contribution of each stage of the computing layout and the abnormalities and interdependencies with the other connected stages.

The applications used to evaluate the effectiveness of the suggested simulation approach were WordCount, logistic regression, k-means clustering, and PageRank. Execution time and memory are predicted better via simulation than the other potential methods. The I/O cost is not effectively predicted for certain applications due to reduced network I/O when working with a smaller setup than the original. Hadoop-based performance predictions cannot be reused for Spark due to the technology difference, primarily the in-memory capabilities unique to Spark. Machine learning-based approaches could not be tried due to scarcity of training data.