How to build Apache Spark 2.1.0

Venkatesh-Prasad Ranganath

Mar 28, 2017·1 min read

I have encountered some sort of failure every time I have built a new version of Apache Spark. So, I decided to document the steps that lead to a successful build.

1. Unpack spark-2.1.0.tgz and changes into spark folder.
2. Ensure you are running Java 8. If you are on Mac OSX, then run /usr/libexec/java_home -V and export the location of JDK 8 as JAVA_HOME, e.g., export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home.
3. Issue the build command ./build/mvn clean package. Since this command runs tests and VersionsSuite test suite contacts an external server, make sure you are on a fast network at the time of testing.
4. If the build fails (or you interrupt the build), then try shutting down Zinc server by executing ./build/zinc-<version>/bin/zinc -shutdown and reissuing the build command ./build/mvn clean package.
5. Test out the build with ./bin/run-example SparkPi 10 .

Note: I will update this post (or create new posts) as I try out newer versions of Spark.