Skip to content

ui: Show camera viewing ray when hovering a pixel in the image viewer - #4584

Open
behnamasadi wants to merge 1 commit into
colmap:mainfrom
behnamasadi:feat/hover-pixel-ray
Open

ui: Show camera viewing ray when hovering a pixel in the image viewer#4584
behnamasadi wants to merge 1 commit into
colmap:mainfrom
behnamasadi:feat/hover-pixel-ray

Conversation

@behnamasadi

Copy link
Copy Markdown
Contributor

Summary

Adds a rerun-style interaction to the model viewer: hovering a pixel in a
camera's image draws that pixel's back-projected viewing ray in the 3D view.

  • Double-click a camera frustum → its photo opens in the image window (existing).
  • Hover a pixel in that photo → a ray shoots from the camera center through the
    pixel into the 3D scene
    , updating live as the cursor moves and clearing when
    it leaves the image.
  • If the hovered pixel is near an observed keypoint, the ray snaps to and
    highlights the corresponding 3D point
    (yellow); otherwise it extends across
    the scene along the pixel's direction.

This makes the 2D image and the 3D reconstruction directly cross-referenceable:
point at anything in a photo and see where it lies in 3D.

Demo

hover-pixel-ray demo

How it works

  1. Capture the hovered pixel — the image window installs an event filter on
    its QGraphicsView viewport (mouse tracking on) and maps the cursor to image
    pixel coordinates.
  2. Unproject to a world rayCamera::CamFromImg(pixel) gives the normalized
    camera-frame direction (x, y, 1), transformed by the image pose
    (Inverse(image.CamFromWorld())) into a world-space ray from
    image.ProjectionCenter().
  3. Snap to a 3D point — the nearest observed Point2D with a point3D_id
    within ~1% of the image size is found; if present, the ray ends exactly at
    that 3D point and highlights it.
  4. Draw in 3D — a dedicated LinePainter (ray) and PointPainter (highlight)
    render in ModelViewerWidget::paintGL, using the same
    model_scale_ * (world + model_origin_) transform as the rest of the scene.

Notes

  • Files: model_viewer_widget.{h,cc} (SetHoverRay/ClearHoverRay + painters),
    image_viewer_widget.{h,cc} (event filter + unprojection). image_viewer_widget.h
    moves graphics_scene_/graphics_view_ from private to protected so the
    database image viewer can attach the hover filter.
  • GUI feature — built with -DGUI_ENABLED=ON and verified interactively; not
    covered by the headless unit tests. The ray geometry was checked numerically
    (computed pixel direction matches the camera→3D-point direction).

Note on the second commit (cherry-picked)

For convenience this branch also carries a second commit,
"ui: Add reprojection-error image colormap for camera frustums", cherry-picked
from #4579, so both GUI viewer features can be built and demoed from a single
branch. That colormap change is reviewed separately in #4579 — happy to drop
the commit here and keep this PR scoped to the hover-ray change alone if you
prefer.

@behnamasadi
behnamasadi force-pushed the feat/hover-pixel-ray branch from 844f616 to f43845f Compare July 23, 2026 16:27
@behnamasadi

Copy link
Copy Markdown
Contributor Author

@ahojnnes what do you say?

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it's a cool visualization to have but I am admittedly not very much convinced this adds a lot of practical value. Could you share a bit how you are using this in your workflow? Happy to be convinced if others find it useful.

Apart from these general concerns, it looks like you accidentally entangled a few other changes into this PR.

Comment thread src/colmap/ui/colormaps.h Outdated
Comment thread src/colmap/ui/image_viewer_widget.cc Outdated
Comment thread src/colmap/ui/image_viewer_widget.cc Outdated
Comment thread src/colmap/ui/image_viewer_widget.cc Outdated
Comment thread src/colmap/ui/image_viewer_widget.cc Outdated
Comment thread src/colmap/ui/model_viewer_widget.cc Outdated
@behnamasadi

Copy link
Copy Markdown
Contributor Author

@ahojnnes Thanks for your reviews. I have rebased this on top of my other feature: #4579
That feature is about setting the camera frustums based on reprojection error so the user gets an intuitive feeling about the reliability of each camera pose. Regarding your question about how I use this feature, it is basically a visual debugging tool to see if 2D/3D correspondence is correct and they land exactly on the 3D scene, Rerun heavily uses it and I get the idea from there. I mean, if you feel like adding this PR, I can make the changes you mentioned. Regards.

Hovering the mouse over a pixel in the model viewer's image window now draws the
back-projected viewing ray from that camera's center through the pixel into the
3D scene, as a visual check that a 2D observation lines up with the 3D geometry
it should correspond to.

The pixel is unprojected with Camera::CamFromImg and rotated into world
coordinates; the ray is drawn with a dedicated LinePainter in the model viewer.
The image viewer installs an event filter on its graphics view to track hover,
and clears the ray when the cursor leaves the image.

The per-image state needed for the back-projection (camera, camera center,
rotation and ray length) is extracted once in ShowImageWithId() rather than on
every mouse move. The ray length is the distance to the farthest 3D point
observed by the image, so the ray stays visible at any model scale and does not
degenerate when the camera sits near the world origin.

Claude-Session: https://claude.ai/code/session_01JTjt8PHYuiZSJMFyPX16zB
@behnamasadi

Copy link
Copy Markdown
Contributor Author

@ahojnnes A rebase moved the line anchors, so the review threads above are marked outdated — all six points are addressed in the current head (44d8463), and I've replied inline to each. Summary:

  • The colormaps.h change from ui: Color camera frustums by reprojection error in the model viewer #4579 is gone; the diff is now only the four hover-ray files.
  • Per-image state is extracted once in ShowImageWithId() instead of on every mouse move.
  • Both suggestion diffs applied.
  • The extra readout is removed — only the ray is drawn.
  • Ray length is the max distance to any 3D point observed by the image, so it no longer degenerates when the camera sits near the world origin.

On the value question: I use it as a correspondence sanity check. Hovering a pixel and watching where its ray lands in the 3D view immediately shows whether the 2D observations and the triangulated geometry agree — a ray that misses the surface it visually points at means a bad pose, a wrong camera model, or a mis-registered image, and that is much faster to spot than reading reprojection-error numbers. It's the same idea as Rerun's ray picking. It's a debugging aid, not a workflow feature, and it costs nothing when unused: no state is built unless an image is open and the ray is only computed on hover.

If you'd still rather not carry it in the GUI, I'm fine with closing — but the review comments are all resolved either way, so it's ready if you want it.

@behnamasadi

Copy link
Copy Markdown
Contributor Author

@ahojnnes Gentle ping on this one. The six review threads are marked outdated after a rebase but all are addressed in 44d8463; the diff is four files under src/colmap/ui/. Happy to close it if GUI debugging aids aren't something you want to carry — just don't want it sitting in your queue undecided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants