Improving Apache Spark performance on k8s

We used the following Github repositories and packaged both artifacts in a single docker image:

With both in place, you can easily create jobs generating data and running queries. In Datafy we do this by defining airflow tasks and using our custom SparkSubmitOperator, but most settings can be ported to another Spark operator.

The most important things to highlight are:

We run two similar jobs, one job without disks attached and a second job with 4 100Gb disks attached to the executors.
Other important configuration options:

The results for 3 queries with and without additional storage are as follows:

Two things stand out when looking at the results:

Spark is a distributed compute engine and requires exchanging data between nodes when performing joins, aggregations, and sorts across multiple executors. It uses a shuffle process that writes data on local disks before sending it over the network when other executors require that data.

In this blogpost we explained why it is useful to add external disks when running Spark jobs on kubernetes. The reason is that it can significantly improve the performance of Spark jobs that require a lot of shuffles. This was illustrated with query 64 of the TPC-DS benchmark, which had a 60% performance boost on a 1Tb dataset. For users that do not use Datafy but are running Spark on Kubernetes, it might be interesting to look into adding external disks as well, if you notice that Spark is doing a lot of shuffling.

Related posts:

Life is a journey of twists and turns, peaks and valleys, mountains to climb and oceans to explore. Good times and bad times. Happy times and sad times. But always, life is a movement forward. Image…

I think oftentimes we only look at what is happening around us, without asking why. In my experience, knowing why things are the way they are has helped me understand the world, myself, and my kids…