Already Borrowed Error When using SessionContext from Multiple Threads #366

Activity

1. kdbrooks

   added

   bugSomething isn't workingSomething isn't working

   on May 3, 2023
2. Michael-J-Ward commented on Oct 29, 2024

   

   Michael-J-Ward

   on Oct 29, 2024

   Contributor

   More actions

   @kylebrooks-8451 - Are you still experiencing this error with recent releases of datafusion-python?

   If so, would you mid providing a minimal reproducible example?
3. ntjohnson1 commented on Sep 19, 2025

   

   ntjohnson1

   on Sep 19, 2025

   Last edited by ntjohnson1

   Contributor

   More actions

   from __future__ import annotations
   
   from multiprocessing import cpu_count
   from multiprocessing.pool import ThreadPool
   
   import datafusion as dfn
   
   
   def run_threads() -> None:
       ctx = dfn.SessionContext()
       with ThreadPool(cpu_count()) as pool:
           results = []
           for i in range(100):
               results.append(pool.apply_async(make_dataframe, args=(ctx, i)))
           for result in results:
               result.get()
   
   
   def make_dataframe(ctx: dfn.SessionContext, i: int) -> None:
       df = ctx.from_pydict({"a": [i]})
       df.write_parquet(f"output_{i}.parquet")
   
   
   if __name__ == "__main__":
       run_threads()

   RuntimeError: Already borrowed

   (on `'48.0.0')
4. ntjohnson1 commented on Sep 19, 2025

   

   ntjohnson1

   on Sep 19, 2025

   Contributor

   More actions

   Basically changing all the mut self to self in src/context.rs resolves the error, compiles and all the tests pass. I'm not sure if there is some subtlety to that. I assume the contexts are thread safe in rust.

   🎉React with 🎉1Reacted by Kyle Brooks
5. timsaucer

   closed this as completedin #1248on Sep 24, 2025