Skip to content

Fix kf-notebook config hidden by workspace volume - #3456

Merged
lresende merged 1 commit into
elyra-ai:mainfrom
lresende:fix-kf-notebook-config
Nov 13, 2026
Merged

lresende merged 1 commit into
elyra-ai:mainfrom
lresende:fix-kf-notebook-config

Conversation

@lresende

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

The elyra/kf-notebook image loses its Elyra customizations on Kubeflow. With an empty volume on the home directory, which is what a new Kubeflow workspace volume looks like, the server reports the Airflow runtime and no component catalogs:

Runtime types Component catalogs
4.2.0, as built KUBEFLOW_PIPELINES, LOCAL kubeflow_pipelines_examples
4.2.0, empty home APACHE_AIRFLOW, KUBEFLOW_PIPELINES, LOCAL none

The build wrote both customizations to $HOME: ~/.jupyter/jupyter_elyra_config.py, which enables only the KFP runtime, and the examples catalog in ~/.local/share/jupyter/metadata. Kubeflow mounts the workspace volume on /home/jovyan, and its 01-copy-tmp-home init script seeds a new volume only from /tmp_home, so neither file reached the notebook server. Elyra's packages aren't affected: /opt/conda belongs to jovyan, so pip installs into the conda environment. This is related to #3453, which was about packages in the elyra/elyra image.

Image (etc/docker/kubeflow/Dockerfile):

  • The Elyra config goes to jupyter_core's sys.prefix config dir (/opt/conda/etc/jupyter). Jupyter loads it before ~/.jupyter, so users can still override it there.
  • The config is written directly (c.PipelineProcessorRegistry.runtimes = ['kfp']) instead of generating the template and editing it with sed, which silently did nothing if the template's commented-out default changed.
  • The examples catalog is created with ELYRA_METADATA_PATH pointing at the sys.prefix data dir (/opt/conda/share/jupyter). Elyra lists metadata from there as factory instances, next to its runtime images, and a user's own instance with the same name still takes precedence. Elyra writes new metadata for the owner only, so the catalog is made world-readable to match runtime-images.
  • The pip upgrade uses --no-cache-dir, so it no longer leaves a cache in $HOME.

Validation (etc/docker/kubeflow/validate-image.py, make validate-kf-notebook-image):

  • Starts the image with its own entrypoint (/init), as built and with an empty tmpfs on /home/jovyan, and pipes the script into the running container.
  • Asks the server through Elyra's REST API whether the Kubeflow Pipelines runtime is enabled, the Airflow runtime is disabled, and the examples catalog is available. Only the standard library is used.
  • create-release.py runs it after validate-elyra-image, so a broken image fails the release before anything is published. make validate-kf-notebook-image image=<name> validates any other image.

The kubeflow README documents the target and explains why the image doesn't write to $HOME.

How was this pull request tested?

The image was built on the real ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter:v1.8.0 base. Docker Hub wasn't reachable from the build environment, so the build used a copy of the Dockerfile without its # syntax=docker/dockerfile:experimental line (no other changes; nothing in this Dockerfile needs that frontend) and the legacy builder (DOCKER_BUILDKIT=0), because BuildKit couldn't get a token for ghcr.io.

  • Fails on 4.2.0: with an empty home, the Airflow and catalog checks fail on quay.io/elyra/kf-notebook:4.2.0. As built, all three pass.
  • Passes with the fix: all three checks pass, both as built and with an empty home. The config is in /opt/conda/etc/jupyter, the catalog is in /opt/conda/share/jupyter/metadata/component-catalogs with the same permissions as runtime-images (directory 755, file 644), and nothing Elyra-related is in $HOME.
  • User override: with c.PipelineProcessorRegistry.runtimes = ["kfp", "airflow"] in ~/.jupyter/jupyter_elyra_config.py on a persistent volume, the server enables Airflow after a restart.
  • Make target: the recipe runs both modes, adds the tmpfs mount only in empty-home mode, removes each container even when a check fails, and fails when a check fails. This was tested with a fake container runtime. The checks above ran the same docker run/docker exec commands directly, because the build environment's sandbox doesn't allow make to reach the Docker socket.
  • black passes, and flake8 passes with the repo's 120-character limit.
  • To build and validate locally: make release kf-notebook-image validate-kf-notebook-image.
  • Not covered: a Kubeflow deployment.

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

The kf-notebook image wrote its Elyra customizations to $HOME during
the build: a jupyter_elyra_config.py that enables only the Kubeflow
Pipelines runtime, and the examples component catalog. Kubeflow mounts
the workspace volume on the home directory and seeds a new volume only
from /tmp_home, so neither reached the notebook server. Users got the
Airflow runtime and no examples catalog.

Install both in the conda environment instead. The config goes to
jupyter_core's sys.prefix config dir, which Jupyter loads before
~/.jupyter, so users can still override it. The catalog is created
with ELYRA_METADATA_PATH pointing at the sys.prefix data dir, where
Elyra lists it as a factory instance next to its runtime images, and
is made world-readable like them.

Write the config directly instead of generating the template and
editing it with sed, which silently did nothing if the template's
commented-out default changed. The pip upgrade no longer leaves a
cache in $HOME either.

Add `make validate-kf-notebook-image`, which starts the image as built
and with an empty tmpfs on the home directory, and checks the runtime
types and catalogs through Elyra's REST API. Run it from
create-release.py before images are published.

Signed-off-by: Luciano Resende <lresende@apple.com>
@lresende lresende added this to the 4.3.0 milestone Nov 13, 2026
@lresende lresende added the kind:bug Something isn't working label Nov 13, 2026
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.82%. Comparing base (26674d3) to head (4c14233).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3456      +/-   ##
==========================================
- Coverage   84.83%   84.82%   -0.01%     
==========================================
  Files         159      159              
  Lines       19297    19297              
  Branches      502      500       -2     
==========================================
- Hits        16370    16369       -1     
- Misses       2887     2889       +2     
+ Partials       40       39       -1     
Flag Coverage Δ
integration-editors 48.14% <ø> (+0.03%) ⬆️
integration-misc 25.64% <ø> (ø)
integration-pipeline 47.68% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lresende
lresende merged commit 6b6fb9d into elyra-ai:main Nov 13, 2026
19 checks passed
@lresende
lresende deleted the fix-kf-notebook-config branch November 13, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant