Repository navigation
CatalogProvider errors are badly mangled #1226
Description
I'm working on a setup where we use a python CatalogProvider with register_catalog_provider:
class MyCatalog:
...
ctx.register_catalog_provider('datafusion', MyCatalog())
ctx.sql(...)This results in a call stack that goes python -> rust -> python and back. As a result, if an error is raised by MyCatalog, it gets badly mangled before being reraised (for example by ctx.sql):
DataFusion error: Execution("PyErr { type: <class 'internal.CatalogClientError'>, value: CatalogClientError('Table \".nonexistant_table\" not found...')"
There's no way to recover anything useful from this exception without string-parsing.
To fix this, we'd probably need to add DataFusionError::Ffi(Box<dyn Error>) upstream, then construct it here:
datafusion-python/src/errors.rs
Line 96 in f0bbad7
Then, we could check for it here, and, if it matches, potentially return the original PyErr unchanged:
I haven't tested this approach, but if it sounds reasonable I could give it a shot.
Activity
mesejo commentedon Sep 9, 2025
Perhaps we can use DataFusionError::External for this? I have a POC in here.
colinmarc commentedon Sep 9, 2025
Your POC looks exactly right to me!
colinmarc commentedon Sep 19, 2025
@mesejo want to open a PR with your branch? I can also do it if you don't have cycles.
mesejo commentedon Sep 23, 2025
@colinmarc Sorry for the late reply. After thinking about it, I've concluded that the best place for this change to happen is the upstream repo. I have opened an issue for that: apache/datafusion#17745
AdMub commentedon Jan 22, 2026
I tested this on the latest main branch. The behavior seems to have changed.
Instead of a mangled error, the Python exception is now logged to stdout (CatalogProvider schema returned error: ...), but the exception raised to Python is a generic Diagnostic error (table ... not found).
This confirms that the underlying error is not being propagated programmatically, but fixing it likely requires the upstream changes discussed above.
fix: mangled errors
fix: mangled errors (#1377)