[ZEPPELIN-5745] Add headless CLI to run a note without starting the server - #5461
[ZEPPELIN-5745] Add headless CLI to run a note without starting the server#5461HwangRock wants to merge 2 commits into
Conversation
…erver
Add a bin/run-note.sh CLI that executes a single note headlessly -- no
Jetty/REST/WebSocket -- by assembling the interpreter runtime directly,
in the spirit of papermill for Jupyter. Supports parameter injection via
-p (${var} placeholders) and saving results to a separate note via -o.
A failed paragraph exits non-zero so CI/batch callers detect the failure.
On exit the process tears down its interpreter processes, RemoteScheduler
pools, the event server, and ExecutorFactory pools, then System.exit()s so
the JVM does not hang on the runtime's non-daemon threads.
a2a6180 to
1806fe2
Compare
… tree NotebookRunnerIntegrationTest counted RemoteInterpreterServer JVMs machine-wide via `jps -l`, so any concurrent zeppelin-server test that spawned its own interpreter made the orphan assertion see non-zero survivors (expected:0 but was:8 in the core-modules CI job). Poll the descendant pids of the run-note.sh process while it is alive and assert none of them survive the parent exit, scoping the check to this test's own process tree.
|
@HwangRock Could you please explain why this feature is needed? E.g. specific usecases? No server exeuction looks good, but Zeppelin focuses on manaing lifecycle for interpreters. Without Zeppelin server, we cannot do it. Am I understanding correctly? |
|
Thanks for good questions @jongyoul .
Right now a note can only be run on a live server. You either drive it from the browser or call the REST API of a running instance, so every run is tied to a long-lived server. That's awkward for batch/CI. Think of a cron job, a CI check, or an Airflow task that starts one container, Some earlier context:
So the goal is a lightweight way to run a single note without a server, for batch/CI, using the exit code to tell success from failure.
The interpreter lifecycle isn't handled by Jetty (the web server) itself, but by the InterpreterSettingManager object. |
2 participants
What is this PR for?
Today a note can only be run through a live server — you start the web server
(Jetty/REST/WebSocket) and drive it from the browser, or call the REST API of an
already-running instance. That ties every execution to a long-lived server
process, which is awkward for cron jobs, CI checks, and batch pipelines.
This PR adds a small CLI,
bin/run-note.sh, that runs a single note headlessly— no web server at all — in the spirit of papermill
for Jupyter. It assembles just the interpreter runtime, executes the note, saves
the result, and exits.
Usage
-i— logical path of the note to run (e.g./Demo, as shown in the note list). Required.-o— save the executed note to a new path instead of overwriting the input.-p— inject a parameter, substituted into${key}/${key=default}placeholders in the note. Repeatable.Example:
How it works
NotebookRunnerbuilds the notebook + interpreter runtime with manual dependencyinjection (the same recipe the interpreter tests already use), so no Jetty/HK2 is
involved. The interpreter event server is still started — interpreters run as
separate processes and need it to call results back — but the web layer is not.
Paragraph output streams to stdout as the note runs, and the executed note is
saved through the normal
NotebookRepo.Behavior worth calling out
FINISHEDstate (error, abort, orskipped after an earlier failure) fails the run with a non-zero exit, so CI/batch
callers can detect failures; a clean run exits 0.
assumption. On exit the process stops the interpreter processes, each
RemoteScheduler's own thread pool, the event server, andExecutorFactory'spools, then calls
System.exit()as a backstop so a stray non-daemon threadcannot hang the JVM.
> filecaptures onlythe results); the "saved / finished" summary goes to stderr.
Known limitation
Notes that drive other paragraphs programmatically via
z.run()/z.runNote()are not supported yet — that path executes paragraphs asynchronously and can
race the run's completion check. Straight top-to-bottom notes (the common batch
case) are unaffected. This is noted in
HeadlessProcessListenerand can be afollow-up.
What type of PR is it?
Feature
What is the Jira issue?
How should this be tested?
Automated — 19 unit/integration tests under
notebook/cli/, including two thatlaunch
bin/run-note.shin a real separate JVM and assert the process exitswithin a timeout (guarding against the JVM-hang class of bug) with the correct exit
code (0 on success, 1 on a failed paragraph):
Manual:
runs a note end to end, prints its output, saves the result, and returns to the
shell prompt on its own.
Screenshots (if appropriate)
2026-09-05.10.10.53.mov
Questions: