跳到主内容
@wquguru
精选88Polars（GitHub Releases）语言与库

Polars Python v2.0.0-rc.2：破坏性变更与弃用

Python Polars 2.0.0-rc.2

原文
发到 X
推荐理由

v2.0 是重大版本迭代，涉及 ENUM 类型处理、API 弃用及核心行为变更。请检查代码中是否依赖旧版枚举解析或已弃用的 cut/qcut 接口，及时适配新签名。

💥 Breaking changes

💥 破坏性变更

  • Read Parquet ENUM type as pl.String (#29331)
  • More map operations (#29296)
  • Deprecate cut/qcut (#29329)
  • Remove legacy POLARS_STREAMING_CHUNK_SIZE (#29046)
  • Introduce the Map dtype (#28984)
  • Update behavior of DataFrame/LazyFrame functions on 0-width input (#29099)
  • 将 Parquet ENUM 类型读取为 pl.String (#29331)
  • 更多映射操作 (#29296)
  • 弃用 cut/qcut (#29329)
  • 移除遗留的 POLARS_STREAMING_CHUNK_SIZE (#29046)
  • 引入 Map 数据类型 (#28984)
  • 更新 DataFrame/LazyFrame 函数在零宽度输入上的行为 (#29099)

🚀 Performance improvements

🚀 性能改进

  • Order pushed parquet predicate columns by measured selectivity (#29397)
  • Coerce float literals to decimal instead of casting the column (#29395)
  • Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
  • Restrict correlated SQL aggregates to requested keys (#29383)
  • Prepare Parquet scans for row-group splitting in Polars Cloud (#29295)
  • Skip row groups by join runtime ranges without a statistics frame (#29370)
  • Reduce copy in scan_lines (#29310)
  • Increase HTTP read rate-limit default (#29363)
  • Evaluate a pushed parquet predicate sequentially per conjuct (#29352)
  • Disable system certificates for CloudScheme::Http sources (#29284)
  • Reduce rechunk in sort_in_place (#29343)
  • Dynamic predicates for hash joins (#29312)
  • Use HTTP suffix range for Parquet size and footer (#29308)
  • Derive predicates from join conditions (#29304)
  • Lower uncorrelated subqueries to semi joins and push semi/anti joins below inner joins (#29289)
  • Make leaf name iterator unique (#29291)
  • Don't clone the full frame per arm in when/then/otherwise (#29258)
  • Push inner joins before outer joins and rewrite left-join-is-null to anti join (#29277)
  • Use stats to decide cross join buffering side (#29270)
  • Improve cache-removal and join-order cost estimates (#29263)
  • Improve CSPE cost evaluation (#29250)
  • Fuse group-by pre-select into node after partition (#29251)
  • Inline hot small functions (#29244)
  • Fix plan-time regressions in projection pushdown for wide frames (#28724)
  • Rechunk before selecting in group-by pre-select (#29219)
  • Reuse Iceberg data file sizes (#29063)
  • Avoid gather for >= 75% cold keys in streaming group-by (#29217)
  • Elide memset on pread for FileByteSource (#29162)
  • Add prefetching to OOC (#29081)
  • Push more predicates past caches (#29182)
  • Prevent cache trashing in row encoding for fixed width types (#29175)
  • Improve group-by cardinality estimates (#29174)
  • Split join clusters across filters and bound key domains by value range (#29166)
  • Make file read concurrency semaphore global (#29163)
  • Skip redundant output validity construction in all-valid arr.dot (#29115)
  • Default posix_fadv to Normal (#29157)
  • Avoid a full fsync per file when writing Parquet (#29108)
  • Replace mmap with file I/O for parquet scan (#29091)
  • Use adaptive concurrency for head requests (#29131)
  • Improve join-reordering (#29092)
  • 按测量的选择性对推入的 Parquet 谓词列进行排序 (#29397)
  • 将浮点字面量强制转换为 decimal 而不是转换列 (#29395)
  • 修复 TPCH SQL 上的 OOM 并修复模糊测试错误 (#29389)
  • 将相关 SQL 聚合限制为请求的键 (#29383)
  • 为 Polars Cloud 中的行组拆分准备 Parquet 扫描 (#29295)
  • 在没有统计信息帧的情况下，通过连接运行时范围跳过行组 (#29370)
  • 减少 scan_lines 中的拷贝 (#29310)
  • 增加 HTTP 读取速率限制的默认值 (#29363)
  • 逐个合取项顺序评估推入的 Parquet 谓词 (#29352)
  • 禁用 CloudScheme::Http 源的系统证书 (#29284)
  • 减少 sort_in_place 中的重分块 (#29343)
  • 哈希连接中的动态谓词 (#29312)
  • 使用 HTTP 后缀范围获取 Parquet 大小和页脚 (#29308)
  • 从连接条件推导谓词 (#29304)
  • 将非相关子查询降低为半连接，并将半连接/反连接推入到内部连接之下 (#29289)
  • 使叶名称迭代器唯一化 (#29291)
  • 不要在 when/then/otherwise 中为每个 arm 克隆完整帧 (#29258)
  • 在外部连接之前推送内部连接，并将左连接为空重写为反连接 (#29277)
  • 使用统计信息决定交叉连接缓冲侧 (#29270)
  • 改进缓存移除和连接顺序成本估算 (#29263)
  • 改进 CSPE 成本评估 (#29250)
  • 将分组预选择融合到分区后的节点中 (#29251)
  • 内联热点小函数 (#29244)
  • 修复宽帧投影下推中的计划时回归问题 (#28724)
  • 在分组预选择选择之前重新分块 (#29219)
  • 复用 Iceberg 数据文件大小 (#29063)
  • 在流式分组中避免对 >= 75% 冷键进行收集 (#29217)
  • 对 FileByteSource 的 pread 省略 memset (#29162)
  • 向 OOC 添加预取 (#29081)
  • 将更多谓词推送到缓存之后 (#29182)
  • 防止固定宽度类型行编码中的缓存抖动 (#29175)
  • 改进分组基数估算 (#29174)
  • 跨过滤器拆分连接集群，并通过值范围限制键域 (#29166)
  • 使文件读取并发信号量全局化 (#29163)
  • 在所有有效 arr.dot 中跳过冗余的输出有效性构造 (#29115)
  • 将 posix_fadv 默认设置为 Normal (#29157)
  • 在写入 Parquet 时避免每个文件执行完整的 fsync (#29108)
  • 用文件 I/O 替换 parquet 扫描的 mmap (#29091)
  • 对 head 请求使用自适应并发 (#29131)
  • 改进连接重排序 (#29092)

✨ Enhancements

✨ 增强功能

  • Expose the registered source as scan_fn.io_source (#28897)
  • Support selectors in join keys (#29233)
  • Honor Iceberg sort orders in native sinks (#29318)
  • Add APPROX_QUANTILE to the SQL frontend (#29288)
  • Add support for approx_quantile in the streaming engine (#29237)
  • Read Parquet ENUM type as pl.String (#29331)
  • Expose more scan_iceberg/delta-related attributes in the visitor for cudf_polars (#29297)
  • More map operations (#29296)
  • Binning functions (#28888)
  • Support collect and collect_batches using RemoteEngine (#28914)
  • Support GROUP BY GROUPING SETS, ROLLUP, CUBE and GROUPING() (#29278)
  • Fix tpch SQL issues (#29269)
  • Fuse filters in (inner) join operation (#29218)
  • Add in-memory support for approximate quantile (#29206)
  • Disable casts from String to Time (#29215)
  • Add organization to Config.enable_monitoring (#29221)
  • Add a workspace argument to Config.enable_monitoring (#29196)
  • Remove legacy POLARS_STREAMING_CHUNK_SIZE (#29046)
  • Add LazyFrame resolver API (#29003)
  • Support incremental append Iceberg scans (#28820)
  • Support date/integer arithmetic and decimal non-equi join comparisons (#29156)
  • Update documentation analytics implementation (#29127)
  • Introduce the Map dtype (#28984)
  • Update behavior of DataFrame/LazyFrame functions on 0-width input (#29099)
  • 将已注册的源暴露为 scan_fn.io_source (#28897)
  • 在连接键中支持选择器 (#29233)
  • 在本机 Sink 中遵循 Iceberg 排序顺序 (#29318)
  • 在 SQL 前端添加 APPROX_QUANTILE (#29288)
  • 在流式引擎中添加对 approx_quantile 的支持 (#29237)
  • 将 Parquet ENUM 类型读取为 pl.String (#29331)
  • 在 cudf_polars 的访问器中暴露更多与 scan_iceberg/delta 相关的属性 (#29297)
  • 更多映射操作 (#29296)
  • 分箱函数 (#28888)
  • 使用 RemoteEngine 支持 collect 和 collect_batches (#28914)
  • 支持 GROUP BY GROUPING SETS、ROLLUP、CUBE 和 GROUPING() (#29278)
  • 修复 tpch SQL 问题 (#29269)
  • 在（内）连接操作中融合过滤器 (#29218)
  • 为近似分位数添加内存支持 (#29206)
  • 禁用从 String 到 Time 的转换 (#29215)
  • 向 Config.enable_monitoring 添加组织参数 (#29221)
  • 向 Config.enable_monitoring 添加工作区参数 (#29196)
  • 移除旧的 POLARS_STREAMING_CHUNK_SIZE (#29046)
  • 添加 LazyFrame 解析器 API (#29003)
  • 支持增量追加 Iceberg 扫描 (#28820)
  • 支持日期/整数算术以及非等值连接的十进制比较 (#29156)
  • 更新文档分析实现 (#29127)
  • 引入 Map 数据类型 (#28984)
  • 更新 DataFrame/LazyFrame 函数在 0 宽度输入上的行为 (#29099)

🐞 Bug fixes

🐞 错误修复

  • Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
  • Size row-index table statistics from the statistics frame (#29381)
  • Preserve categories object in pyo3-polars (#29385)
  • Strip the leading slash from Windows Delta table roots (#29386)
  • Isolate expanded Python dataset scans (#29378)
  • Make Iceberg bucket sort keys serializable (#29376)
  • Apply "schema_overrides" in read_database for Arrow-based drivers (#29273)
  • Remove duplicated word in rate-limit comment (#29372)
  • Check whether Datetime is monotonically increasing dynamically (#29293)
  • Keep input order of unmatched build rows in ordered streaming equi join (#29371)
  • Unaliased constants in SQL SELECT with GROUP BY (#29367)
  • Consistent Date and Decimal means between the streaming and in-memory engines (#29359)
  • Respect string statistics for enum columns during parquet scanning (#29366)
  • Support selectors properly in DataFrame n_unique (#29360)
  • Decimal Parquet statistics for decimal/i128/f16 (#29350)
  • Early check for converting integer map keys (#29348)
  • Lowering for input-independent filter (#29340)
  • Correlation of constant column returning non-NaN for larger inputs (#29319)
  • Incorrect height in multi-input GroupBy (#29332)
  • Fix panic in scan_iceberg for snapshot_id before a schema change (#28895)
  • Resolve arithmetic Struct supertypes per-field (#29261)
  • Fix comparison expression method comment (#29257)
  • Don't panic on an empty or null quantile expression input (#29240)
  • Detect list-valued quantile literals in approx_quantile auto (#29236)
  • Various map issues (#29147)
  • Rolling quantile should respect window trimming (#29200)
  • Ensure ignored columns are excluded from dtype Wildcard selector (#29220)
  • Include the library path in the plugin dlopen error message (#29213)
  • Respect Parquet schema when reading Int96 timestamps (#29208)
  • Wrap IoError in PathIoError such that callers can extract untruncated path (#29128)
  • Slice on Series with negative index beyond length (#29191)
  • Fix OOB access due to empty chunks (#29177)
  • Compare struct AnyValues through their field dtypes (#29185)
  • Improve group-by cardinality estimates (#29174)
  • Preserve scalar arrays in arr.dot dispatch (#29000)
  • Empty array collect missing dtype in join (#29095)
  • Apply the OVER clause to multi-argument aggregates (#29160)
  • Respect NULLS FIRST/LAST in a window's ORDER BY (#29159)
  • Resolve relation aliases declared inside a parenthesized JOIN (#29158)
  • Fix a DataFrame construction race (and generator data loss) (#29132)
  • Properly compute null count in parquet statistics for list of categorical/enum (#29140)
  • Address read_database Oracle regression, following introduction of Arrow fast-path (#29142)
  • Synchronize PR label matchers (#29144)
  • Unblock Python thread during collect_all_async (#29097)
  • Match engine output dtype in planner for Decimal true division with integers (#29122)
  • Error instead of segfault for exp/log1p on non-numeric dtypes (#29112)
  • Prevent panic in offset_by when target date is out-of-range (#29059)
  • Incorrect broadcast height min/max_horizontal for scalar inputs (#29083)
  • Proper seq id relabeling before distributing (#29060)
  • 修复 TPCH SQL 的 OOM 问题并修复模糊测试错误 (#29389)
  • 从统计帧中获取行索引表的大小统计信息 (#29381)
  • 在 pyo3-polars 中保留 categories 对象 (#29385)
  • 去除 Windows Delta 表根路径的前导斜杠 (#29386)
  • 隔离扩展后的 Python 数据集扫描 (#29378)
  • 使 Iceberg 桶排序键可序列化 (#29376)
  • 在基于 Arrow 的驱动程序中为 read_database 应用 "schema_overrides" (#29273)
  • 删除速率限制注释中的重复单词 (#29372)
  • 动态检查 Datetime 是否单调递增 (#29293)
  • 在有序流式等值连接中保持未匹配构建行的输入顺序 (#29371)
  • 带有 GROUP BY 的 SQL SELECT 中的非别名常量 (#29367)
  • 流式引擎与内存引擎之间 Date 和 Decimal 含义的一致性 (#29359)
  • 在 parquet 扫描期间尊重枚举列的字符串统计信息 (#29366)
  • 在 DataFrame n_unique 中正确支持选择器 (#29360)
  • decimal/i128/f16 的 Decimal Parquet 统计信息 (#29350)
  • 转换整数映射键时的早期检查 (#29348)
  • 降低与输入无关的过滤器的开销 (#29340)
  • 对于较大输入，常量列的相关性返回非 NaN 值 (#29319)
  • 多输入 GroupBy 中的高度不正确 (#29332)
  • 修复 scan_iceberg 在架构更改前 snapshot_id 导致的 panic (#28895)
  • 按字段解析算术 Struct 超类型 (#29261)
  • 修复比较表达式方法的注释 (#29257)
  • 对空或为 null 的分位数表达式输入不要恐慌 (#29240)
  • 在 approx_quantile 自动处理中检测列表值的分位数字面量 (#29236)
  • 各种 map 问题 (#29147)
  • 滚动分位数应尊重窗口修剪 (#29200)
  • 确保被忽略的列从 dtype 通配符选择器中排除 (#29220)
  • 在插件 dlopen 错误消息中包含库路径 (#29213)
  • 读取 Int96 时间戳时尊重 Parquet 模式 (#29208)
  • 将 IoError 包装为 PathIoError，以便调用者可以提取未截断的路径 (#29128)
  • 对超出长度的 Series 使用负索引进行切片 (#29191)
  • 修复因空块导致的越界访问 (#29177)
  • 通过字段数据类型比较 struct AnyValues (#29185)
  • 改进 group-by 基数估计 (#29174)
  • 在 arr.dot 分发中保留标量数组 (#29000)
  • 连接中空数组收集缺失的数据类型 (#29095)
  • 将 OVER 子句应用于多参数聚合函数 (#29160)
  • 在窗口的 ORDER BY 中尊重 NULLS FIRST/LAST (#29159)
  • 解析括号内 JOIN 中声明的关系别名 (#29158)
  • 修复 DataFrame 构造的竞争条件（以及生成器数据丢失）(#29132)
  • 正确计算分类/枚举列表的 parquet 统计信息中的空值计数 (#29140)
  • 解决引入 Arrow 快速路径后 read_database Oracle 回归问题 (#29142)
  • 同步 PR 标签匹配器 (#29144)
  • 在 collect_all_async 期间解除 Python 线程阻塞 (#29097)
  • 在规划器中匹配 Decimal 与整数真除法引擎输出的数据类型 (#29122)
  • 对非数值数据类型的 exp/log1p 报错而非段错误 (#29112)
  • 防止在 offset_by 中因目标日期超出范围而引发恐慌 (#29059)
  • 标量输入的 broadcast height min/max_horizontal 不正确 (#29083)
  • 在分发前正确重新标记 seq id (#29060)

📖 Documentation

📖 文档

  • Clarify that ambiguous parameter refers to DST transitions (#28873)
  • Document gzip/zstd compression for read_csv and scan_csv (#29387)
  • Add LazyFrameResolver to reference guide (#29379)
  • Add user-guide for new enable_monitoring feature (#29358)
  • Improve join_where engine tag (#29234)
  • Add 2 to the version switcher dropdown (#29194)
  • Show more information in engine tags (#29154)
  • Engine tags for is_empty/ has_nulls (#29152)
  • More engine tags (#29148)
  • Fix misleading reinterpret() docstring (#29075)
  • 澄清模糊参数指的是 DST 转换 (#28873)
  • 为 read_csv 和 scan_csv 记录 gzip/zstd 压缩 (#29387)
  • 在参考指南中添加 LazyFrameResolver (#29379)
  • 为新 enable_monitoring 功能添加用户指南 (#29358)
  • 改进 join_where 引擎标签 (#29234)
  • 在版本切换器下拉菜单中添加 2 (#29194)
  • 在引擎标签中显示更多信息 (#29154)
  • is_empty/ has_nulls 的引擎标签 (#29152)
  • 更多引擎标签 (#29148)
  • 修复误导性 reinterpret() 文档字符串 (#29075)

📦 Build system

📦 构建系统

  • Add a fast-release profile without LTO (#29168)
  • 添加不带 LTO 的快速发布配置 (#29168)

🛠️ Other improvements

🛠️ 其他改进

更进一步：量化金融体系

看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力

进入量化体系 →
话题#破坏性#Polars#Python

相似阅读

关联信息，但可能不是同一事件