How to install and configure Apache Toree for Jupyter notebook in windows 10?

Question

Can some one help me with the installation and configuration of apache toree for jupyter notebook in Windows 10? I tried it but was unsuccessful. Error encountered is as follows.

Failed to start Kernel

Unknown server error.

     Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\notebook\base\handlers.py", line 516, in wrapper
    result = yield gen.maybe_future(method(self, *args, **kwargs))
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "C:\Anaconda3\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Anaconda3\lib\site-packages\notebook\services\sessions\handlers.py", line 75, in post
    type=mtype))
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "C:\Anaconda3\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 79, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "C:\Anaconda3\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 92, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "C:\Anaconda3\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Anaconda3\lib\site-packages\tornado\gen.py", line 307, in wrapper
    yielded = next(result)
  File "C:\Anaconda3\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 94, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "C:\Anaconda3\lib\site-packages\jupyter_client\multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "C:\Anaconda3\lib\site-packages\jupyter_client\manager.py", line 243, in start_kernel
    **kw)
  File "C:\Anaconda3\lib\site-packages\jupyter_client\manager.py", line 189, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "C:\Anaconda3\lib\site-packages\jupyter_client\launcher.py", line 123, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "C:\Anaconda3\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Anaconda3\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

Answer 1

Apache Toree uses %PROG_HOME%\bin\run.sh to start the kernel.

Usually PROG_HOME is C:\Users\{Account_Name}\AppData\Roaming\jupyter\kernels\apache_toree_scala on Windows.

Since Windows cannot run shell script, it throws OS error:

[WinError 193] %1 is not a valid Win32 application.

You need to follow below steps:

1. Download Spark version compatible with Scala 2.11 and set SPARK_HOME environment variable. Note that Apache Toree kernel version 0.3.0-incubating uses Scala version 2.11.

2. Create a Windows batch file (run.bat) or Windows Command script file (run.cmd) at %PROG_HOME%/bin. Similar to run.sh, use below command to start the kernel through SparkSubmit.scala class.

%JAVA_HOME%\bin\java -cp "%SPARK_HOME%\jars\*;%PROG_HOME%\lib\toree-assembly-0.3.0-incubating.jar;." -Dscala.usejavacp=true org.apache.spark.deploy.SparkSubmit %SPARK_OPTS% --class org.apache.toree.Main %PROG_HOME%\lib\toree-assembly-0.3.0-incubating.jar %TOREE_OPTS% %*

3. Update the argv parameter value in kernel.json file in PROG_HOME folder from run.sh to run.cmd

4. Start Anaconda Prompt. Run jupyter notebook command. Select "Apache Toree - Scala" kernel from browser. You can see kernel connection status on Anaconda prompt.

Answer 2

would like to add to @UmeshD's answer above If you are using toree-assembly-0.3.0-incubating, create a file run.cmd (in C:\Users\{Account_Name}\AppData\Roaming\jupyter\kernels\apache_toree_scala/bin/) and paste the following code below

@echo off
pushd "%~dp0\..\"
set PROG_HOME=%cd%
popd

if not defined SPARK_HOME ( 
    echo "SPARK_HOME must be set to the location of a Spark distribution!"
    Exit /b
)

echo "Starting Spark Kernel with SPARK_HOME=%SPARK_HOME%"


rem for /f %%i in ('dir /B toree-assembly-*.jar') do set KERNEL_ASSEMBLY=%%i popd

rem disable randomized hash for string in Python 3.3+
set PYTHONHASHSEED=0
rem set TOREE_ASSEMBLY=%PROG_HOME%/lib/%KERNEL_ASSEMBLY%
rem The SPARK_OPTS values during installation are stored in __TOREE_SPARK_OPTS__. This allows values to be specified during
rem install, but also during runtime. The runtime options take precedence over the install options.
if not defined SPARK_OPTS (
    if defined __TOREE_SPARK_OPTS__ (
        set SPARK_OPTS=%__TOREE_SPARK_OPTS__%
    )
)

if not defined TOREE_OPTS (
    if defined __TOREE_SPARK_OPTS__ (
        set TOREE_OPTS=%__TOREE_OPTS__%
    )
)

%JAVA_HOME%\bin\java -cp "%SPARK_HOME%\jars\*;%PROG_HOME%\lib\toree-assembly-0.3.0-incubating.jar;." -Dscala.usejavacp=true org.apache.spark.deploy.SparkSubmit %SPARK_OPTS% --class org.apache.toree.Main %PROG_HOME%\lib\toree-assembly-0.3.0-incubating.jar %TOREE_OPTS% %*