JupTunnel

• Key Features
• Architecture Overview
• Installation
• Usage
• Resilience and Connection Management
• Advanced Features
• Micromamba Environment Setup
• Configuration Reference
• Troubleshooting
• Examples
• Software notes

Automated SSH Tunneling and Jupyter Notebook Management

JupTunnel is a powerful tool that automates the creation of SSH tunnels to remote servers, starts Jupyter Notebook instances, and manages the entire lifecycle of your remote computing sessions. It's designed to simplify the process of working with remote Jupyter environments while providing robust session management and resilience features.

Key Features

• Automated SSH Tunneling: Establishes secure SSH connections with optional multiplexing
• Remote Jupyter Management: Starts, monitors, and manages Jupyter Notebook servers
• Session Persistence: Maintains session state across disconnections
• Configuration Flexibility: Supports both CLI arguments and YAML configuration files
• Graceful Shutdown: Handles SIGTERM and Ctrl+C with proper cleanup
• Multi-platform Support: Works on Unix-like systems with SSH access

Architecture Overview

JupTunnel consists of two main components:

1. Main Binary (jup-tunnel): Runs on your local machine and manages the overall session
2. Remote Agent (jup-tunnel-agent): Runs on the remote server and handles Jupyter process management

The communication between these components uses a secure RPC protocol over the established SSH tunnel.

graph TD
    A[Local: jup-tunnel] -->|SSH Tunnel| B[Remote Server] --> C
    B -->|Local Port| A
    A -->|RPC Commands| C[Remote: jup-tunnel-agent]
    C -->|Jupyter Management| D[Jupyter Notebook]
    C -->|Process Monitoring| D

Installation

Prerequisites

• Rust toolchain (stable version)
• SSH client with access to remote servers
• Python environment on remote servers for Jupyter

Build from Source

# Clone the repository
git clone https://github.com/your-repo/jup-tunnel.git
cd jup-tunnel

# Build the release version
cargo build --release

# The binaries will be available at:
# - jup-tunnel: target/release/jup-tunnel
# - jup-tunnel-agent: target/release/jup-tunnel-agent

Quick Installation Check

# Verify the binary works
./target/release/jup-tunnel --version

# Check available commands
./target/release/jup-tunnel --help

Usage

Basic usage

Basic Command Structure

jup-tunnel [OPTIONS] [COMMAND] [MORE OPTIONS]

Command Line Options

JupTunnel provides comprehensive command-line options for configuration notably related to session management:

Common options

• -d, --debug: To activate debug traces of all operations

Connection Options

• -u, --remote-user <USERNAME>: Remote server username
• -t, --remote-host <HOST>: Remote server hostname or IP
• -l, --local-port <PORT>: Local port for tunneling (default: 8888)
• -r, --remote-port <PORT>: Remote port for Jupyter (default: 8888)

Jupyter Configuration

Jupyter needs be installed on the remote compute. These parameters allows for finding the correct installation:

• -k, --kernel-name <NAME>: Jupyter kernel name (default: my_kernel). This is not really used at the moment.
• -e, --env-path <PATH>: Path to remote Python environment. Not really used, will be updated later.
• --conda-env <NAME>: Conda environment name to activate, in which to find the correct python/jupyter.
• --conda-exec-name <EXEC PATH>: Path of the conda/mamba executable. This is used to execute the jupyter command in the correct environment.
• -d, --work-dir <PATH>: Working directory for Jupyter on remote machine. This will be the root for the interactive file browser over http.
• -v, --verbose: Show verbose information on the session startup.
• -j, --verbose-jupyter: Show detailed Jupyter process output on the remote.

SSH Configuration

• -a, --ssh-control-auto: Auto-generate temporary SSH ControlPath (default)
• --ssh-control-path <PATH>: SSH ControlPath for connection multiplexing
• --ssh-control-command <CMD>: SSH control command on shutdown (e.g., "exit")

Advanced Options

• --jupyter-auto-port: Let Jupyter choose its own port
• --local-port-random: Allocate local port randomly to match to jupyter
• -b, --background: Run in background after Jupyter starts
• -k, --keep-running: Keep running regardless of tunnel/Jupyter status

Configuration Management

• -c, --config <FILE>: Use YAML configuration file
• --version: Show version information
• --help: Show help message

Configuration File Support

All command-line options can be specified in a YAML configuration file for easier management:

# Example configuration file
remote_user: username
remote_host: your.server.com
local_port: 8888
remote_port: 8888
kernel_name: my_kernel
env_path: "/home/user/venv/bin/python"
conda_env: "myenv"
conda_exec_name: "/fullpath/to/bin/micromamba"
work_dir: "/home/user/projects"
verbose: true
ssh_control_auto: true

Priority Order: CLI arguments override configuration file values, which override built-in defaults.

Starting a Session

# Basic usage with CLI arguments
jup-tunnel session new -u myuser -t remote.server.com -v

# Using a configuration file
jup-tunnel session new --config myconfig.yaml

# Mixed approach (CLI overrides config)
jup-tunnel session new --config base-config.yaml --local-port 9999 -v

Session Management

JupTunnel provides robust session management capabilities:

Session Commands

# List all active sessions
jup-tunnel session list

# Close a specific session
jup-tunnel session close <SESSION_ID>

# Get magic URL for a session to access remote jupyter
jup-tunnel session magic-url <SESSION_ID>

# Clear all sessions (doesn't terminate running processes)
jup-tunnel clear-sessions

Session Persistence

JupTunnel maintains session state in a local database, allowing you to:

• Resume interrupted sessions: Reconnect to existing Jupyter instances
• Retrieve connection details: Get URLs and tokens for active sessions
• Monitor remote processes: Check status of running Jupyter servers
• Graceful recovery: Sessions persist across application restarts

RPC Commands

The tool supports Remote Procedure Calls to interact with running sessions:

# Ping a session to check if it's alive
jup-tunnel session rpc <SESSION_ID> ping

# Check tunnel status
jup-tunnel session rpc <SESSION_ID> tunnel-alive

# Resume a broken tunnel connection
jup-tunnel session rpc <SESSION_ID> resume-tunnel

# Get session information
jup-tunnel session rpc <SESSION_ID> info

# Gracefully shutdown a session
jup-tunnel session rpc <SESSION_ID> shutdown

# List Jupyter sessions
jup-tunnel session rpc <SESSION_ID> list-jupyter-sessions

# Poll Jupyter output
jup-tunnel session rpc <SESSION_ID> poll-jupyter-output --jupyter-id 0 --kind stdout

Resilience and Connection Management

Session Persistence

JupTunnel is designed to handle connection interruptions gracefully:

• Session Database: All session information is stored locally and persists across application restarts
• Process Tracking: Remote Jupyter processes continue running even if the local client disconnects
• Reconnection Support: You can reconnect to existing sessions using session IDs
• State Preservation: Configuration, tokens, and connection details are maintained

Handling Disconnections

When a connection is lost:

1. Remote processes continue running: Jupyter servers remain active on the remote machine
2. Session data is preserved: All connection details are stored in the local database
3. Manual reconnection: Use jup-tunnel session list to find your session and reconnect
4. RPC interface: Use RPC commands to check status and resume operations

Resuming a Tunnel

To resume a disconnected session:

# List existing sessions
jup-tunnel session list

# Find your session ID and use RPC to resume the tunnel
jup-tunnel session rpc <SESSION_ID> resume-tunnel

# Check tunnel status
jup-tunnel session rpc <SESSION_ID> tunnel-alive

# If the tunnel cannot be resumed, create a new session with the same parameters
jup-tunnel session new --config original-config.yaml

Advanced Features

SSH Connection Multiplexing

# Enable automatic SSH control path
jup-tunnel -u myuser -t remote.server.com -a

# Use a specific control path
jup-tunnel -u myuser -t remote.server.com --ssh-control-path ~/.ssh/control:socket

Background Operation

# Run in background after Jupyter starts
jup-tunnel -u myuser -t remote.server.com -b

# Keep running regardless of tunnel status
jup-tunnel -u myuser -t remote.server.com -k

Shell Completions

Generate shell completions for your preferred shell:

# Bash
jup-tunnel completion bash > /etc/bash_completion.d/jup-tunnel

# Zsh
jup-tunnel completion zsh > ~/.zsh/completions/_jup-tunnel

# Fish
jup-tunnel completion fish > ~/.config/fish/completions/jup-tunnel.fish

Micromamba Environment Setup

JupTunnel works seamlessly with micromamba environments. Here's how to set up a micromamba environment with Jupyter on your remote server:

Installing Micromamba on Remote Server

# Connect to your remote server
ssh your-username@remote.server.com

# Download and install micromamba
curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba

# Move micromamba to your bin directory
mkdir -p ~/.local/bin
mv bin/micromamba ~/.local/bin/

# Add micromamba to your PATH (add this to your ~/.bashrc or ~/.zshrc)
export MAMBA_ROOT_PREFIX=~/micromamba
export PATH="$HOME/.local/bin:$PATH"

# Source the changes
source ~/.bashrc

# Initialize micromamba
micromamba shell init -s bash -p ~/micromamba
source ~/.bashrc

Creating a Jupyter Environment

# Create a new environment with Jupyter
micromamba create -n jupyter-env -c conda-forge jupyterlab notebook python=3.10

# Activate the environment
micromamba activate jupyter-env

# Install additional packages as needed
micromamba install -c conda-forge numpy pandas matplotlib scikit-learn

# Test Jupyter installation
jupyter notebook --version

Micromamba Configuration for JupTunnel

To use your micromamba environment with JupTunnel, you need to configure the conda_exec_name and conda_env options:

# Configuration for micromamba environment
remote_user: your-username
remote_host: remote.server.com
local_port: 8888
remote_port: 8888
kernel_name: python3
conda_exec_name: micromamba  # Specify micromamba instead of conda
conda_env: jupyter-env       # Your micromamba environment name
work_dir: /home/your-username/projects
env_path: ""                  # Leave empty when using conda/micromamba
ssh_control_auto: true
verbose: true

Complete Micromamba Example Configuration

Here's a complete configuration file for using JupTunnel with micromamba:

# micromamba-jupyter-config.yaml
# Complete configuration for JupTunnel with micromamba

# Required connection settings
remote_user: researcher
remote_host: compute-server.example.com

# Port configuration
local_port: 8888
remote_port: 8888

# Jupyter kernel configuration
kernel_name: python3

# Micromamba environment settings
conda_exec_name: micromamba  # Use micromamba instead of conda
conda_env: jupyter-env       # Your micromamba environment name

# Working directory on remote server
work_dir: /home/researcher/data-analysis

# SSH configuration for better performance
ssh_control_auto: true
ssh_control_command: "exit"

# Verbose output for debugging
verbose: true
verbose_jupyter: true

# Let Jupyter choose its own port if 8888 is busy
jupyter_auto_port: true

Starting JupTunnel with Micromamba

# Start a session using the micromamba configuration
jup-tunnel session new --config micromamba-jupyter-config.yaml

# Or use CLI arguments directly
jup-tunnel session new \
  -u researcher \
  -t compute-server.example.com \
  --conda-exec-name micromamba \
  --conda-env jupyter-env \
  --work-dir /home/researcher/data-analysis \
  -v

Troubleshooting Micromamba Issues

Micromamba not found: Ensure micromamba is in your PATH on the remote server and the conda_exec_name is set correctly.

Environment not found: Verify the environment name matches exactly what you created with micromamba create.

Package conflicts: Use micromamba env export to check your environment configuration and recreate if needed.

Configuration Reference

For complete configuration details, see:

• doc/CONFIGURATION.md - Configuration file format and options
• example/example-config.yaml - Complete example configuration

Troubleshooting

Common Issues

Connection refused: Verify SSH access to the remote server and that the specified ports are available.

Permission denied: Ensure you have proper SSH credentials and permissions on the remote machine.

Port already in use: Use --local-port-random to automatically find an available local port.

Jupyter startup failure: Check that Python and Jupyter are properly installed on the remote server.

Debugging

# Enable debug output
jup-tunnel -u myuser -t remote.server.com --debug

# Check session status
jup-tunnel session list

# Review logs (when debug is enabled)
cat jup-tunnel-debug.log

Examples

Basic Remote Jupyter Session

# Start a simple session
jup-tunnel -u researcher -t compute-server.example.com -v

# Access Jupyter at http://localhost:8888 with the displayed token

Advanced Configuration with Conda

# Use a specific conda environment
jup-tunnel -u myuser -t remote.server.com \
  --conda-env myenv \
  --work-dir /home/myuser/projects \
  --local-port 9999 \
  --remote-port 9999

Session Management Workflow

# Start a session in the background
jup-tunnel session new -u myuser -t remote.server.com -b

# List all sessions
jup-tunnel session list

# Get the magic URL for session 1
jup-tunnel session magic-url 1

# Later, close the session
jup-tunnel session close 1

Software notes

License

MIT License - See the LICENSE file for details.

Contributing

Contributions are welcome! Please open issues for bugs and feature requests, and submit pull requests for improvements.

Support

For help and discussion, please open a GitHub issue or check the project documentation.

Disclaimer

Some part of the software and documentation were generated with LLM assistance. The software and documentation has been manually reviewed and amended by the human authors.