跳到主内容
@wquguru
精选75Polars(GitHub Releases)语言与库

Polars Rust 0.55.1:性能提升与多项新功能

Rust Polars 0.55.1

原文
发到 X

This Releases' DSL matches Python 1.43.2.

🚀 Performance improvements

  • Pushdown len() to concat/union inputs (#28570)
  • Optimize not(bool_f) to not_bool_f (#28474)
  • Split multiplexers that directly scan from in-memory DataFrame (#28376)
  • Make DNS cache global (#28352)
  • Do not remove cache if predicates not pushed to all inputs (#28341)
  • Environment variable for logging slow DNS lookup (#28211)
  • Pre-partition on left, right and semi joins on hive partitioned data (#28374)
  • Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
  • Tighten redundant comparisons and propagate filter constraints (#27999)
  • Canonicalize predicates (#28212)
  • Remove excess memory copy in streaming IPC sink (#28141)
  • Parallelize scan_iceberg resolve (#28168)
  • Chunk-aware morsel splitting on single-file IO sink (#27950)
  • Don't allocate for full_null() with small lengths (#28181)
  • Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)
  • Add a sampled resolve mode for multi-file parquet metadata (#28111)
  • Speed up small dtype series sums with a upcast sum kernel (#27958)
  • Don't block on path expansion (#28073)
  • Add bytes-based concurrency control for cloud IO (#27924)
  • Stop flushing multiplexer on phase change (#28013)
  • Prevent empty chunks from triggering copy in rechunk (#27953)
  • Improve performance of struct unnest inside list eval on streaming (#27905)

✨ Enhancements

  • Add struct.drop() (#28666)
  • Improve error message when CSV name de-duplication fails (#28658)
  • Always keep first metadata per source for Parquet (#28661)
  • Improve plan-time row estimates for multi-file parquet scans (#28380)
  • Serialize and deserialize SinkTypeIR::Partitioned (#28112)
  • Add infer_schema_files parameter to scan_csv (#28440)
  • Support reading IEEE 754 total order Parquet column order (#27896)
  • Expose more Expr nodes for cudf_polars (pt 2) (#28404)
  • Allow callback sinks on cloud (#28458)
  • Pre-partition group-by on hive keys (#28444)
  • Add ewm_sum and ewm_sum_by #28151 (#28215)
  • Serve stale records from object_store DNS cache (#28256)
  • Expose more expr nodes for cudf-polars (#28117)
  • Make partition_hive an opt flag (#28345)
  • When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
  • Show only differing dtypes in assert_frame_equal error (#27816)
  • Add Clone to SpillFrame/Token (#28302)
  • Add Series.degrees/radians and clarify trig functions use radians (#22961)
  • Merge sorted with multikey (#28067)
  • Add list expression which consistently packs elements together into new List type (#27990)
  • Construct pyarrow predicates via PyO3 instead of eval str (#27271)
  • Optimize joins with redundant equi-join keys (#28017)
  • Implement missing datetime options for CastColumnsPolicy (#28084)
  • Add Expr.cat.to and Expr.cat.physical (#28096)
  • Add POLARS_OOC_DISK_BUDGET_MB (#28045)
  • Close ResourceWarning leaks in database tests (#28066)
  • Expose IRFunctionExpr::Fused in the python node visitor (#27889)
  • Deprecate casts from string to temporal dtypes (#28056)
  • Add naive out-of-core spilling to Polars (#27998)
  • Add experimental strict mode (#28023)
  • Optimize non-negative len and null_count comparisons (#28002)
  • Optimize len and null_count comparisons (#27727)
  • Support SQL's implicit JOIN syntax (#27890)
  • Sample from bivariate distribution for spill context Thompson sampling (#27973)
  • Add Expr.is_sorted (#26708)
  • Add suggestions in wrong api usage or mismatched column names (#27954)
  • Add external object_store support for non-native schemes (#27871)
  • Skip Delta files and parquet row groups using per-field struct statistics (#27887)
  • Add re-registration of OOC SpillTokens (#27909)
  • Improve IR sortedness detection for with_row_index and explode (#27875)
  • Add DataFrame.is_sorted() (#27870)

🐞 Bug fixes

  • Don't hand IO plugins a predicate they cannot evaluate (#28643)
  • Don't hang or crash on sqrt/cbrt/pct_change/ewm_* with a Struct column (#28696)
  • Raise on overflow in Decimal sum instead of silently wrapping (#28688)
  • Preserve nulls when importing Arrow maps (#28680)
  • Propagate null by column values in rolling_*_by (#27367)
  • Fix self-referencing field in struct.with_fields with over (#28678)
  • Fix Arrow buffer offset for Utf8 and Binary (#28662)
  • Clamp group-by slice offset (#28579)
  • Ensure categorical fill_null(strategy="min"/"max") uses lexical min/max (#28636)
  • Release the GIL while resolving the schema in __arrow_c_stream__ (#28642)
  • Fix data corruption on Series from nested Arrow LargeList (#28632)
  • Properly propagate enum metadata for iceberg column mapping (#28580)
  • Correctly write parquet field IDs for enums and categoricals (#28581)
  • Fix arrow C interop binview offset (#28623)
  • Fix duplicate hive values in hive pre-partition rewrite (#28621)
  • Double-counted offset for Arrow export of sliced Series(Array) (#28602)
  • Struct/List/Array equality handling in sort and join (#28601)
  • Use try_new in StructArray construction in polars-json (#27489)
  • Raise ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX (#27892)
  • Run type coercion pass on pivot's internally generated group_by (#27897)
  • More careful slice pushdown into joins (#28578)
  • Preserve ordering in sliced unions (#28576)
  • Drop input sortedness when casting to a string (#28574)
  • Clear sortedness flags in StringChunked substring kernels (#28573)
  • Bad mask handling when reading optional parquet column (#28547)
  • Any operation on Unknown(Int) and Unknown(Float) should result in Unknown(Float) (#28545)
  • Flip nulls_last after Expr.reverse() (#28572)
  • Fix high blocking thread use in sink_parquet with async local path (#28543)
  • Serialize LazyFrames backed by bytes (#28568)
  • Release GIL in SQLContext.execute() (#28549)
  • Propagate nulls_last in function_expr_sortedness (#28544)
  • Incorrect slicing when a join requires sorting (#28541)
  • Fix panic on self join of scan_delta / scan_iceberg (#28468)
  • Decrement ParkGroup worker count when worker exits (#28492)
  • Add missing Send bounds in Connector (#28497)
  • Fix UB on first/last_non_null with empty chunk (#28495)
  • Do not CSE non-column height expr on streaming engine (#28480)
  • Propagate adjusted sortedness through Gather (#28493)
  • Deduplicate the expression equality checks (#28483)
  • Address 3VL issue with SQL NOT IN interaction with NULL values and joins (#28484)
  • Use total comparison in Expr literal values (#28482)
  • Ensure SQL SUM and CORR aggregates return NULL for all-null inputs, add TOTAL (#28475)
  • Share null_count_dtype helper between Delta and Iceberg, fixing SchemaError (#28479)
  • Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
  • Fix dropped slice on multiple unions (#28477)
  • Fix in-memory engine incorrect slice on maintain order join (#28478)
  • Check join schema by position (#28455)
  • Block predicate pushdown past overwritten window keys (#28429)
  • Avoid panic when union slice skips all rows (#28420)
  • Solve panic in dt.replace when there were multiple chunks (#28437)
  • Propagate is_scalar from the input to the output of .sort() and .sort_by() (#28438)
  • Insert missing coercions from Unknown(_) in list/array arithmetic (#28411)
  • Resolve CSV column names overwrite in DSL->IR conversion (#28383)
  • Avoid IEJoin rewrite for Categorical comparisons (#28427)
  • Do not rewrite sort().reverse() to sort(descending=True) when maintain_order=True (#28403)
  • Propagate bigidx to polars-plan from polars (#28396)
  • Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
  • Invalid offset in strptime (#28388)
  • Incorrect schema type for decimal <-> primitive division (#28373)
  • Panic in in-memory CSEE handling (#28371)
  • Apply same type coercion to IsBetween as binary comparisons (#28300)
  • Fix offset in arrow ffi export of sliced struct arrays (#28369)

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

另一事件,读法相近