I have encountered some sort of failure every time I have built a new version of Apache Spark. So, I decided to document the steps that lead to a successful build.

  1. Unpack spark-2.1.0.tgz and changes into spark folder.
  2. Ensure you are running Java 8. If you are on Mac OSX, then run /usr/libexec/java_home -V and export the location of JDK 8 as JAVA_HOME, e.g., export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home.
  3. Issue the build command ./build/mvn clean package. Since this command runs tests and VersionsSuite test suite contacts an external server, make sure you are on a fast network at the time of testing.
  4. If the build fails (or you interrupt the build), then try shutting down Zinc server by executing ./build/zinc-<version>/bin/zinc -shutdown and reissuing the build command ./build/mvn clean package.
  5. Test out the build with ./bin/run-example SparkPi 10 .

Note: I will update this post (or create new posts) as I try out newer versions of Spark.

Written by

Programming, experimenting, writing | Past: SWE, Researcher, Professor | Present: SWE

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store