You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bigquery: read large unordered ARROW results with more than one stream #18804
Before ARROW, to_arrow() and to_dataframe() read query results with as many streams as BigQuery allows unless _contains_order_by() matched the query (_download_table_bqstorage() requests 0 streams, which means unlimited). ARROW query results always read with one stream, so very large unordered results may be slower than on the old path. Raised by @tswast in #18654 (comment).
Job _default stream (jobs.query, or jobs.insert with an anonymous destination): read directly with no read session, so there's no stream count. Check whether SplitReadStream works on job default streams, or whether there's another way to parallelize.
Compare throughput against the non-ARROW path for large unordered results.
Still confirming whether very large results are in scope for the ARROW work. The Node.js client (createQueryArrowStream()) has the same design.
Activity