I have encountered some sort of failure every time I have built a new version of Apache Spark. So, I decided to document the steps that lead to a successful build.
- Unpack spark-2.1.0.tgz and changes into spark folder.
- Ensure you are running Java 8. If you are on Mac OSX, then run
/usr/libexec/java_home -Vand export the location of JDK 8 as JAVA_HOME, e.g.,
- Issue the build command
./build/mvn clean package. Since this command runs tests and VersionsSuite test suite contacts an external server, make sure you are on a fast network at the time of testing.
- If the build fails (or you interrupt the build), then try shutting down Zinc server by executing
./build/zinc-<version>/bin/zinc -shutdownand reissuing the build command
./build/mvn clean package.
- Test out the build with
./bin/run-example SparkPi 10.
Note: I will update this post (or create new posts) as I try out newer versions of Spark.