Repository navigation
Display of Lag With with_column uses autogenerated name and provided name #1234
Description
Describe the bug
with_column typically add a new column with the provided name. However, when using lag we get two new columns, with the provided name and automatically generate one.
To Reproduce
import datafusion as dfn
from datafusion import col, lit, functions as F
import pyarrow as pa
def datafusion_example() -> None:
table = pa.table({"a": [1.0, 2.0, 3.0]})
ctx = dfn.SessionContext()
df = ctx.from_arrow(table)
print(
df.with_column(
"previous_a",
F.lag(
col("a"),
default_value=None,
),
)
)
print(df.with_column("something_else", col("a") + lit(1.0)))
if __name__ == "__main__":
datafusion_example()Output
DataFrame()
+-----+-----------------------------------------------------------------------------------------------------------------+------------+
| a | lag(ced8c2b3710c14382bd5eb58d49ffbd53.a,Int64(1),NULL) ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING | previous_a |
+-----+-----------------------------------------------------------------------------------------------------------------+------------+
| 1.0 | | |
| 2.0 | 1.0 | 1.0 |
| 3.0 | 2.0 | 2.0 |
+-----+-----------------------------------------------------------------------------------------------------------------+------------+
DataFrame()
+-----+----------------+
| a | something_else |
+-----+----------------+
| 1.0 | 2.0 |
| 2.0 | 3.0 |
| 3.0 | 4.0 |
+-----+----------------+Expected behavior
DataFrame()
+-----+------------+
| a | previous_a |
+-----+------------+
| 1.0 | |
| 2.0 | 1.0 |
| 3.0 | 2.0 |
+-----+------------+
DataFrame()
+-----+----------------+
| a | something_else |
+-----+----------------+
| 1.0 | 2.0 |
| 2.0 | 3.0 |
| 3.0 | 4.0 |
+-----+----------------+
Activity
renato2099 commentedon Sep 14, 2025
took a stab at this problem, happy to improve it further if required.
timsaucer commentedon Sep 17, 2025
This feels like an upstream bug and related to this prior issue: apache/datafusion#11982
timsaucer commentedon Sep 17, 2025
I reproduced this in upstream datafusion and opened this issue: apache/datafusion#17630
Okay to close in this repo or do you want to keep it open until upstream fixes and it filters down to this project?
ntjohnson1 commentedon Sep 17, 2025
Slight preference to keep it open until it lands here but don't feel strongly about it if this project prefers to have people just watch upstream.
renato2099 commentedon Dec 6, 2025
@ntjohnson1 the fix for this was merged into datafusion apache/datafusion#17639
ntjohnson1 commentedon Dec 6, 2025
@renato2099 thank you!