Jupyter Notebooks
Jupyter Notebooks
Publishing Jupyter Notebooks
The Jupyter tool is a useful place to develop code and analyses in a notebook format. Hub users can easily share their notebooks with other users by publishing their notebooks as tools. A published Jupyter notebook enables other users to interact with the notebook, stepping through its cells and even changing them. But, when users run your published notebook, any changes they have made to it will not persist.
This set of instructions takes you through publishing a hub tool based on your existing Jupyter notebook. Here, we'll assume that the short name for your tool is toolname and that you are a registered, logged-in user. To develop the notebook tool, all you need is access to Jupyter. You'll navigate between your Jupyter home directory, the Jupyter terminal, and, optionally, your tool's status page from the Tool Pipeline (The Tool Pipeline is typically found at yourhub.org/resources/add-contribution/tool-pipeline but this may vary by Hub).
Jupyter version
When developing Jupyter notebooks or Jupyter based tools, you should use the most recent version of Jupyter deployed on your Hub.
To deploy a Jupyter notebook:
1. CREATE THE TOOL
To create the tool for your Jupyter notebook, navigate to Tools and click "Create a New Tool" on the upper left. Fill in the Create Tool page that the system displays:
- Give your tool a brief name (no spaces or hyphens), a full title, and the at-a-glance description.
- Select "Deploy as Jupyter notebook", and add your username in the development team.
- The Access section enables you to restrict tool access to a specific hub Group, if you wish.
- For other fields, you may accept the defaults, and submit.
- Finally, flip the tool status to Registered, and click Apply Change.
1B. Register as a Debian10 tool
The final step of tool registration: for some hubs, you will need to submit a Hub ticket, indicating the short name of your tool, and asking that it be registered as a Debian10 tool. This will ensure that the new tool uses current packages and kernels.
2. ADD THE TOOL REPO
Next, we may need to add the repository containing your Jupyter notebook tool code. If the Add Repo button is now available to you in your tool status page:
- Click the Add Repo button.
- Then, if the page shows a success message, flip the status to Created and click Apply Change.
Otherwise, an administrator must add the repo for you; you should wait for a status email indicating the repo has been created.
3. CHECK OUT THE TOOL REPO
Now the new notebook tool's code repo is Created and ready to use. To do so, we must first check out the repo.
Using Subversion (svn)
Open the Jupyter tool, navigate to your home notebooks directory, and open a terminal by selecting New, and then Terminal. Using the terminal, check out the newly created tool repo locally using this command (toolname is the brief name you gave your tool on the Create Tool page):
svn checkout https://yourhub.org/tools/toolname/svn/trunk toolname
TODO: Using Git
4. ADD NOTEBOOK CODE
It's now time to add the code that will run for your notebook. Back in the Jupyter tool file listing, you should see the toolname directory under your home notebooks directory. Into that directory, copy a working notebook (or develop one in place).
You can configure your notebook to access additional Python packages by loading an alternate kernel in the Jupyter notebook UI. To do so, consult the Kernel dropdown in the Jupyter interface. Different kernels may be available now on your Hub with additional packages. File a ticket or get in touch to let us know what packages you need.
You may need additional data files or code to run the notebook. The Hubzero team recommends putting the main notebook in the top level tool directory. Other files your notebook needs (say, pythonfile.py) can be organized in subdirectories such as data/. Then, you can load any Python files in your notebook as if they were modules. Your notebook will load the Python source data/pythonfile.py this way:
import data.pythonfile
5. EDIT INVOKE SCRIPT
Finally, to tell the hub how to launch the notebook, you need to edit the invoke script that was automatically created at tool creation time. The invoke shell script is found in the toolname/middleware/ directory. To edit it, double-click on the invoke script in the Jupyter file listing, and the editor will launch.
In the invoke script you specify the filename of your Jupyter notebook, the version of Anaconda to use, and other parameters. Here we suppose that your notebook is called your-jupyter-notebook-name.ipynb.
A basic script should look like the following for a Jupyter notebook with Python 3. Use your own notebook's name in place of 'your-jupyter-notebook-name.ipynb' :
#!/bin/sh /usr/bin/invoke_app "$@" -C "start_jupyter -T @tool your-jupyter-notebook-name.ipynb" -r none -u anaconda-6
If your notebook needs additional modules, list them at the end after the -u.
For details on invoke script command line options, refer to the Hubzero invoke documentation.
6. TESTING
Next, you'll test that your working notebook starts properly as a Hub tool. When the notebook passes testing, you are ready to proceed.
7. COMMIT CHANGES
Once you have saved your invoke script and your notebook, check them in to the repository management software. You'll use subversion or git.
For subversion: From a Jupyter terminal, navigate to your tool's directory (get there as we did in step 3. above). First, add the notebook to svn (similarly, add any other needed files, using "svn add"):
svn add your-jupyter-notebook-name.ipynb
then, once all files have been added in this way, commit the changes:
svn commit
Subversion will now prompt you for a commit message. Type in a suitable initial commit message ("initial commit of toolname" will do) and press ctrl-X to exit.
To alert the administrator that your tool is ready for installation, you can now visit your tool's status page, either from the Tool Pipeline, or specifying a URL like this:
https://yourhub.org/tools/toolname/status
Here, click the link that reads, "My code is committed, working, and ready to be installed." If you have special instructions, caveats, compile steps, or other dependencies for your installation, enter them in the available text box now. The tool administrators will be alerted about your tool status and perform the installation along with any required steps you describe.
TODO: Describe how to check code into git
8. INSTALL
It's time to install the tool source. This action will depend on your access privileges; you may need the help of an administrator. On the hub, visit your tool's status page, either from the Tool Pipeline, or specifying a URL like this:
https://yourhub.org/tools/toolname/status
Here you can click the Install button and then on success message, flip the status to Installed and apply the change.
If the Install button is not available to you, this task will be executed by an administrator. You will receive a status email when it is complete.
TODO Edit: Final step for jupyter6deb10 tools: To publish the tool you'll need to let us know the tool name so we can update the docker image mapping so the tool loads in the deb10 env. To do this, enter a hub ticket letting us know tool name and asking for an update of the docker image mapping.
9. TEST AND PUBLISH
To test your tool, go to the hub's Tool Pipeline and select your tool's link, or specify the tool URL directly:
https://yourhub.org/tools/toolname
In the status page, click the button to test run the tool. If the tool does not display or otherwise fails your test, there is still work to do. Revisit your development steps, starting with the TEST section above.
If the notebook test is successful, and it displays and functions as expected, you are almost done! Return to the tool status page. There you can indicate to administrators that you Approve the tool for publication. If special instructions or compilation steps need to be performed for your tool, indicate that here.
Depending on your access privileges on the Hub, you may be able to set the tool as Published. If the link is available to you, click Publish, and then on display of the green success message, flip the status to Published and apply the change.
You will receive a status change email when the tool has transitioned to Published. When you receive word that your tool is Published, you should verify again that it works as expected.
That should do it--your Jupyter notebook is now a published tool available to other Hub users. If you have questions, concerns, or run into a snag, please email the hub administrator. Include any error messages you see, and we'll give you a hand.
10. MAKING CHANGES
To make changes to a published notebook, you must only revisit some of the steps outlined above.
If you are planning to edit an existing Hub tool, please convert it into a Debian10 tool first (See 1B. above). To do so, submit a ticket on your hub, indicating the short name of your tool, and asking that it be registered as a Debian10 tool. This will ensure that the tool has access to current packages and kernels. Hub staff will let you know when the change has been made.
Next, to make edits to the tool:
- Change your notebook code as necessary, revisiting the TEST and COMMIT CHANGES steps above when complete.
- INSTALL your changed code as above
- TEST AND PUBLISH the notebook tool as above
Each time you make changes, be sure to test the notebook and confirm that it works properly.