#INSTALL SPARK ON WINDOWS WITH JUPYTER INSTALL#
#INSTALL SPARK ON WINDOWS WITH JUPYTER CODE#

As of this writing, a pip-installable package wasn’t available for Toree v. Next, we’ll clone the Toree repository from GitHub. export SPARK_HOME=/opt/sparkĮxport PYTHONPATH=$PYTHONPATH:$SPARK_HOME/python:$SPARK_HOME/python/lib

#INSTALL SPARK ON WINDOWS WITH JUPYTER INSTALL#

The install location doesn’t matter too much, but you should definitely make sure that you set a couple important environment variables. In this example we’ll assume that Spark was installed in the folder /opt/spark/. You can add the following lines to your shell’s startup script if you want them to be applied automatically in the future. In this example we’ll use Spark 2.0.1 with support for Hadoop 2.4 but you can use whatever Hadoop version matches your existing stack, or just not worry about it if you’re not using Hadoop at all.

The first step is to download and install Spark itself.

git (the easiest way to get the latest Toree code).

We will need to have access to certain things in the environment before we start: If you’re on Mac or Windows, I suggest looking into the Anaconda platform.

#INSTALL SPARK ON WINDOWS WITH JUPYTER CODE#

In this post I’ll describe how we go from a clean Ubuntu installation to being able to run Spark 2.0 code on Jupyter. Enter Apache Toree, a project meant to solve this problem by acting as a middleman between a running Spark cluster and other applications. The platform supports a number of kernels (the things that actually run the code) besides the out-of-the-box Python, but connecting Jupyter to Spark is a little trickier. The advantage to the user is being able to write code in small chunks which can be run independently but share the same namespace, greatly facilitating testing or trying multiple approaches in a modular fashion. It essentially connects a browser-based frontend, the Jupyter Server, to an interactive REPL underneath that can process snippets of code. Jupyter notebooks are an interactive way to code that can enable rapid prototyping and exploration. While Spark’s latest release is at 2.1.0 at the time of publishing, we’ll use the example of 2.0.1 throughout this post.

One of the great things about Spark is that it’s relatively autonomous and doesn’t require a lot of extra infrastructure to work. The recent 2.0 release milestone brought a number of significant improvements including DataSets, an improved version of DataFrames, more support for SparkR, and a lot more. Spark is one of the most popular open-source distributed computation engines and offers a scalable, flexible framework for processing huge amounts of data efficiently.

Install spark on windows with jupyter