Spark Executor Calculator

How the calculator works

1. Tasks: your data size ÷ partition size gives the number of partitions, and Spark runs one task per input partition.
2. Slots: executors × cores each = how many tasks run at the same time.
3. Waves: tasks ÷ slots, rounded up = how many rounds the stage needs.
4. Memory per executor: heap + overhead. It then checks whether the cores' working memory fits in the heap, using a factor for your job type (from 1.0 for filters to 2.0 for joins and sorts) and a 1.5 safety margin.

Worked example

With the starting values: 10 TB of data, 128 MB partitions, 100 executors with 5 cores and 8 GB each, 20% overhead and a mixed ETL job.

• Tasks = 10 TB ÷ 128 MB = 81,920
• Slots = 100 × 5 = 500 tasks at once
• Waves = 81,920 ÷ 500 ≈ 164
• Container per executor = 8 GB + 1.6 GB overhead = 9.6 GB, so the cluster needs 960 GB
• Working memory = 128 MB × 1.5 × 5 cores × 1.5 safety ≈ 1.4 GB, well within 8 GB, so the result shows OK

Common sizing tips

• About 5 cores per executor is a widely used starting point. Many more cores per executor can hurt input/output throughput; very few waste memory on overhead.
• Partitions of 128–256 MB keep tasks a sensible size. Very small partitions add scheduling overhead.
• Many small files slow jobs down. Fill in the "Number of input files" field to get a warning.

See the Spark tuning guide for more.

Frequently asked questions