9.0.0

9.0.0 (2024-04-30)

⚠ BREAKING CHANGES

• udf: The schema parameter for UDF definition has been removed. A new catalog parameter has been added. Ibis uses the word database to refer to a collection of tables, and the word catalog to refer to a collection of databases. You can use a combination of catalog and database to specify a hierarchical location for the UDF.
• pyspark: Arguments to create_database, drop_database, and get_schema are now keyword-only except for the name args. Calls to these functions that have relied on positional argument ordering need to be updated.
• dask: the dask backend no longer supports cov/corr with how="pop".
• duckdb: Calling the get or contains method on NULL map
  values now returns NULL. Use coalesce(map.get(...), default) or
  coalesce(map.contains(), False) to get the previous behavior.
• api: Integer inputs to select and mutate are now always interpreted as literals. Columns can still be accessed by their integer index using square-bracket syntax.
• api: strings passed to table.mutate() are now interpreted as
  column references instead of literals, use ibis.literal(string) to
  pass the string as a literal
• ir: Schema.apply_to() is removed, use ibis.formats.pandas.PandasConverter.convert_frame() instead
• ddl: We are removing the word schema in its hierarchical
  sense. We use database to mean a collection of tables. The behavior of
  all *_database methods now applies only to collections of tables and
  never to collections of database (formerly schema)
• CanListDatabases abstract methods now all refer to
  collections of tables.
• CanCreateDatabases abstract methods now all refer to
  collections of tables.
• list_databases now takes a kwarg catalog.
• create_database now takes a kwarg catalog.
• drop_database now takes a kwarg catalog.
• current_database now refers to the current collection of tables.
• CanCreateSchema is deprecated and create_schema, drop_schema,
  list_schemas, and current_schema are deprecated and redirect to the
  corresponding method/property ending in database.
• We add a CanListCatalog and CanCreateCatalog that can list and
  create collections of database, respectively.
  The new methods are list_catalogs, create_catalog, drop_catalog,
• There is a new current_catalog property.
• api: timecontext feature is removed
• api: The by argument from asof_join is removed. Calls to asof_join that previously used by should pass those arguments to predicates instead.
• cleanup: Deprecated methods and properties op, output_dtype, and output_shape are removed. op is no longer needed, and use .dtype and .shape respectively for the other two.
• api: expr.topk(...) now includes null counts. The row count of the topk call will not differ, but the number of nulls counted will no longer be zero. To drop the null row use the dropna method.
• api: ibis.rows_with_max_lookback() function and ibis.window(max_lookback) argument are removed
• strings: Backends that previously used initcap (analogous to str.title) to implement StringValue.capitalize() will produce different results when the input string contains multiple words (a word's definition being backend-specific).
• impala: Impala UDFs no longer require explicit registration. Remove any calls to Function.register. If you were passing database to Function.register, pass that to scalar_function or aggregate_function as appropriate.
• pandas: the timecontext feature is not supported anymore
• api: on paremater of table.asof_join() is now only
  accept a single predicate, use predicates to supply additional
  join predicates.

Features

• add to_date function to StringValue (#9030) (0701978), closes #8908
• api: add .as_scalar() method for turning expressions into scalar subqueries (#8350) (8130169)
• api: add catalog and database kwargs to ibis.table (#8801) (7d593c4)
• api: add describe method to compute summary stats of table expressions (#8739) (c8d98a1)
• api: add ibis.today() for retrieving the current date (#8664) (5e10d17)
• api: add a to_polars() method for returning query results as polars objects (53454c1)
• api: add a uuid function for returning a new uuid (#8438) (965b6d9)
• api: add API for unwrapping JSON values into backend-native values (#8958) (aebb5cf)
• api: add disconnect method (#8341) (32665af), closes #5940
• api: allow *arg syntax with GroupedTable methods (#8923) (489bb89)
• api: count nulls with topk (#8531) (54c2c70)
• api: expose common types in the top-level ibis namespace (#9008) (3f3ed27), closes #8717
• api: include bad type in NotImplementedError (#8291) (36da06b)
• api: natively support polars dataframes in ibis.memtable (464bebc)
• api: support Table.order_by(*keys) (6ade4e9)
• api: support all dtypes in MapGet and MapContains (#8648) (401e0a4)
• api: support converting ibis types & schemas to/from polars types & schemas (73add93)
• api: support Deferreds in Array.map and .filter (#8267) (8289d2c)
• api: support the inner join convenience to not repeat fields known to be equal (#8127) (798088d)
• api: support variadic arguments on Table.group_by() (#8546) (665bc4f)
• backends: introducing ibish the infinite scale backend you always wanted (#8785) (1d51243)
• bigquery: support polars memtables (26d103d)
• common: add Dispatched base class for convenient visitor pattern implementation (f80c5b3)
• common: add Node.find_below() methods to exclude the root node from filtering (#8861) (80d12a2)
• common: add a memory efficient Node.map() implementation (e3f2217)
• common: also traverse nodes used as dictionary keys (#9041) (02c6607)
• common: introduce FrozenOrderedDict (#9081) (f926995), closes #9063
• datafusion, flink, mssql: add uuid operation (#8545) (2f85a42)
• datafusion: add array and strings functions (#8895) (2f23223)
• datafusion: implement arbitrary (43a8f50)
• datafusion: port to new sqlglot backend (3aa109a)
• datatypes: convert money and small money datatype to decimal datatype (#8556) (ecc5d70)
• duckdb: add support for read_mysql (#8656) (4ea4a1d)
• duckdb: allow all-null columns in memtables (#8367) (b2ae64a)
• duckdb: support asof joins including tolerance parameter (104cb9b)
• exasol: add support for bit operations (#8741) (4aa721e)
• exasol: add support for DateDelta (dd639ca)
• exasol: add support for DayOfWeekName (#8589) (de4e988)
• exasol: add support for extract epoch seconds (#8726) (db79aae)
• exasol: add support for extract seconds (#8723) (fb7e533)
• exasol: add support for ExtractDayOfYear (#8578) (df2b69e)
• exasol: add support for extracting milliseconds from timestamps (#8722) (1778de5)
• exasol: add support for ExtractQuarter (#8587) (0d9b676)
• exasol: add support for ExtractWeekOfYear (#8588) (68925f6)
• exasol: add support for hexdigest (#8740) (76d8ef0)
• exasol: add support for TimestampNow (#8563) (94e79e4)
• flink: add map support (#8425) (68739a2)
• flink: implement support for array expansion (#8511) (a6e6564)
• flink: implement UDF support for the backend (#8142) (a3b1cc6)
• geo-duckdb: support casting binary to geometry (#9062) (1926eb4)
• geospatial: add support for duckdb operations on literals (#8570) (b4c4369)
• graphviz: node- and edge-specific custom attributes (#8527) (98c52aa)
• graphviz: support custom node and edge attributes in ibis.visualize (#8510) (ee821b1)
• ir: add StringSlice (#8832) (e4e3531)
• ir: add default implementation of pretty formatting nodes (#8880) (a696c70)
• ir: more flexible dereferencing support for join right hand side (#8992) (d7a31aa), closes #9043 #9041
• ir: support pretty printing arbitrary traversable objects (#9043) (68dfe39)
• ir: support showing variable names used to create an expression in repr() (#8630) (220085e)
• mssql: add datatype mapping for hierarchyid (#8397) (2fd2c30)
• mssql: use integrated auth when no user or password supplied (#8668) (0a78414)
• new backend issue template (#8449) (e4edc78)
• pandas, polars, dask, datafusion: enable create local backends with empty url (#8860) (9dabae0), closes #8450
• polars: add limited support for table dot sql (#8528) (b2a4fbb)
• polars: implement arbitrary (973c3d3)
• postgres: add mappings for more esoteric dtypes (#9055) (5cb83fc), closes #8845
• postgres: support loading tables with pgvector column types (#9037) (8846514)
• postgres: use port in connection string (d561c01)
• pyspark: add catalog support to pyspark (#9042) (2c1a58e), closes #9038
• pyspark: add support for PySpark 3.5 (65717f4)
• pyspark: support ibis.pyspark.connect() (#8515) (0f663e6)
• python: support python 3.12 (7056dea)
• risingwave: add streaming DDLs (#8239) (356e459)
• snowflake: allow empty url when using ibis.connect (#8428) (0275c9b), closes #8422
• snowflake: create an ibis backend from a snowpark session (#8962) (f15d033)
• snowflake: support connecting with no arguments (#8422) (543a2ec)
• sql: add option to enable/disable select merging (#9065) (4bc9314), closes #9064 #9058
• sql: extract common table expressions (0324372)
• sql: lower expressions to SQL-like relational operations (6f7f190)
• sql: use SELECT * for complete reprojections (#9075) (a9aa8a7)
• trino: implement existing json functionality (#8963) (964ac3e)
• trino: port to sqlglot (9c5a907)
• ux: add Table and Column.preview() (#7915) (1c03ad0), closes #7408 #7172

Bug Fixes

• api: forbid using asc/desc in selections (62992c3)
• api: improve error message raised on improper calls to array map or filter (#8602) (0236370)
• api: restore and deprecate ir.Table.to_array() (#8227) (22de674)
• api: return NULL when NULL is passed to Array.zip (#8652) (fac85f0)
• api: selection using a selector yielding multiple columns (#8215) (869889b)
• api: support passing literal booleans to filter (2aa31f4)
• backends: make string concat-with-null behavior consistent across backends (#8305) (2d97b8e), closes #8302
• bigquery: do not overwrite the entire default query job config (b42fb1c)
• bigquery: ensure session creation before creating temp tables (#8976) (314abe4), closes #8975
• bigquery: get literals working again (#8577) (6369734)
• bigquery: restore option to specify table path in table name (a9beadb)
• clickhouse: adjust for new timestamp behavior and regen sql (4bdc040)
• clickhouse: avoid forcing UTC to allow connection to servers that do not allow it (#8762) (52eeea9)
• clickhouse: make arrays non nullable (#8501) (1caf6de)
• clickhouse: use backwards compatible string search function (bb736fe)
• common: Node.map_clear should have return type annotation Any (#8564) (8d7baae)
• common: don't match an Object pattern with more positional arguments defined than __match_args__ has (2e63bba)
• common: intermediate result removal fails if there are duplicated dependencies (e3e17db)
• comparison: wrap isnull equality check in parens (#8366) (247e2f7)
• conversion: convert decimals to the exact precision and scale requested by the input type (8c1e6f4)
• dask: don't call compute when executing argmin/argmax (1204c56)
• dask: don't call compute when executing cov/corr (a876c47)
• dask: fix argmin/argmax implementation for dask (93834f1)
• dask: pin dask version to avoid automatically picking up dask-expr (#8629) (f1d0f65)
• datafusion: ensure that to_pyarrow_batches does do compute (d1a62d0)
• datatypes: always quote sqlglot struct fields (#8777) (18bb91b), closes #8771
• datatypes: convert UUIDs to strings (#8262) (6f32374)
• decompile: ensure that SelfReference is decompiled with a call to .view() (4a44c57)
• deps: bump dependencies' lower bounds to reflect tested minimum version (#8977) (9c29f28), closes #8795
• deps: bump polars lower bound (#8841) (125e4ad)
• deps: bump sqlglot to pick up duckdb array fixes (#8682) (a3bd853)
• deps: support pandas 2.2 (#8758) (4b476ba)
• deps: update dependency datafusion to v36 (#8612) (5a67102)
• deps: update dependency pyarrow to v16 (#9033) (a687ec1)
• deps: update dependency sqlglot to >=22,<22.5 (#8635) (267f4bc)
• deps: update dependency sqlglot to >=23.4,<23.10 (#8787) (0f00101)
• deps: update dependency sqlglot to >=23.4,<23.11 (#8957) (2b7f7b1)
• deps: update dependency sqlglot to >=23.4,<23.12 (#9029) (1cace01)
• deps: update dependency sqlglot to >=23.4,<23.13 (#9056) (5dac34d)
• deps: update dependency sqlglot to v21 (#8272) (efaa365)
• deps: update dependency sqlglot to v22 (8aa4222)
• deps: update dependency sqlglot to v23 (#8688) (5041894)
• druid: array_string_join and to_polars extra column (2311d4f)
• druid: import pydruid.db module explicitly (#8782) (550ada0)
• duckdb-sql: ignore importlib package errors when importing ibis.snowflake for transpilation (#8389) (b968301)
• duckdb: add flip_coordinates translation to sqlglot duckdb backend (f7df510)
• duckdb: allow connection to motherduck via ibis.connect (#8357) (42f45fe), closes #8355
• duckdb: allow passing both overwrite and temp to create_table (b9b19e0)
• duckdb: ensure that create_schema and create_database are actually tested (ba31f82)
• duckdb: ensure that structs can be used with sqlglot 20.1.0 (17be43a)
• duckdb: generate struct fields with propertyeq instead of slice (d2c1316)
• duckdb: load extension when executing geospatial expressions (#9080) (1960d54)
• duckdb: parenthesize argument to StructField operation to support field access on CASE statements (#8486) (1371016)
• duckdb: pass global replace flag to ops.RegexReplace translation rule (e46260d)
• duckdb: udfs builtins taking zero args (ab39344)
• duckdb: workaround for duckdb Map NULL bugs (#8649) (75d32e5)
• duckdb: workaround remaining null map issues (#8985) (b6c71d7), closes #8632
• fix SQL backend has_operation to include operations supported through rewrite rules (133a1f1)
• flink: avoid non-existent sge.NULL (9f190eb)
• flink: cast map key lookups because flink requires exact type matches (#8724) (6893a5f)
• flink: fix compilation of memtable with nested data (#8751) (364a6ee), closes #8516
• flink: fix compilation of over aggregation query in flink backend (#8359) (de174a2)
• graphviz: show proper field attributes of accessed relations and do not display join link property accesses (#8521) (69d6c73)
• impala: remove no-longer-used temporary database and paths that may prevent connection success (#8489) (32fcce6), closes #8466
• ir: asof join tolerance parameter should post-filter and post-join instead of adding a predicate (e380e79)
• ir: accidentally remapping fields during bind() (#8988) (f4cee67), closes #8884
• ir: compute InSubquery.shape property from needle input (#8364) (13d675e), closes #8361
• ir: fix window boundaries being forcefully casted (#8400) (09b6ada)
• ir: make impure ibis.random() and ibis.uuid() functions return unique node instances (#8967) (741063a)
• ir: only dereference comparisons not generic binary operations (05ac73a)
• ir: self reference fields were incorrectly dereferenced to the parent relation (7bfebe2)
• make devcontainer work correctly (#9019) (a696c58), closes #9011
• mssql: don't use the removed sge.TRUE and sge.FALSE literals (7e0b735)
• mssql: restore any, all and cumulative versions (#8409) (99a4022), closes #8073
• mssql: restore unbounded window functions (#8411) (0211d4f)
• mysql: remove not-allowed frame clause from rank window function (ee96cef)
• oracle: allow passing both overwrite and temp to create_table (3ce4766)
• oracle: clean up memtables at exit (dc34f61)
• oracle: enable dropping temporary tables (1dffd5e)
• oracle: map bare NUMBER to int64 and consolidate data type mapping code for shared inference (#8626) (b5f9bbe)
• pandas: make case work for non-RangeIndex dataframes (#9083) (73dd685)
• pandas: map date type to datetime64[s] (#8667) (6bd965e)
• pandas: use mergesort for deterministic sorting (6042a71)
• polars: columns are picked from the correct side in case of conflicting names (#8134) (4273cef)
• polars: ensure t.select(col=scalar) results in len(t) rows (#8665) (6c00579)
• polars: ensure that reading from a compressed csv triggers in-memory read (b3bbde1)
• polars: force null sorting to match the rest of ibis (b475c36)
• polars: reference the correct field in the ops.SelfReference rule (a371274)
• polars: support order by computed column (ddf56cb)
• polars: use value type of array type for int_ranges construction (c24c54e)
• polars: user newer drop API in asof join implementation (c65d9f8)
• polars: user newer drop API to avoid deprecation warning (43424ea)
• postgres: fix compilation of array string join and map/struct field extraction (306d0fc)
• postgres: fix json type conversion in to_pyarrow output (#8439) (b338517), closes #8318
• postgres: pass through additional kwargs in pguri (7ab4fda)
• pyarrow: map date type to arrow date32 not date64 (05575b7)
• pyarrow: support accepting pyarrow dictionary types as inputs (#8276) (14c4226), closes #8207
• pyspark: don't use the removed sge.NULL, sge.TRUE and sge.FALSE literals (dffb44a)
• pyspark: ensure that to_delta works and is tested (64af56a)
• pyspark: ensure that the output of zip matches the expected ibis schema (#9052) (be9d5da), closes #9049
• pyspark: force sqlglot to generate first/last (fbfc3c1)
• pyspark: remove use of attribute that prevents using spark connect (#9061) (b48f451), closes #9060
• pyspark: unwind catalog/database settings in same order they were set (#9067) (962ee00)
• rewrites: add missing filter arguments for node.replace() calls (196e716)
• rewrites: change TableColumn -> Field in rewrites (#8448) (3730eb6)
• risingwave: gen correct jsonb extract path function (7b0d6a9)
• risingwave: set implicit flush to true (#8929) (fe16877)
• snowflake-snowpark: disable the reconnect method (#8969) (e31eded)
• snowflake: bring back default nth behavior from before the-epic-split (e943667)
• snowflake: handle udf function naming (bec36ca)
• snowflake: import the connector at the correct scope (6bbb9c8)
• snowflake: initialize _from_snowpark variable in constructor to ensure it is defined (#8970) (5722a10)
• snowflake: initialize the parent class on construction (#8972) (de3a169)
• snowflake: manually construct quantile calls with WITHIN GROUP (#8846) (261a544)
• snowflake: set con outside of _setup_session call (#8979) (3b1a6ef)
• snowflake: use _safe_raw_sql for insert implementation (2ceb5a6)
• sql: avoid calling .subquery on subqueries (7ad32bd)
• sql: avoid excessive inlining during Select merge (#8825) (ba931da)
• sql: don't generate table aliases for ops.JoinLink (3da1abf)
• sqlite: don't use the removed sge.NULL literal (e8ed08a)
• sqlite: ensure ibis.uuid() generates a unique uuid per row (#8535) (c097a2d), closes #8532
• sql: look for CTEs under value expressions as well (#8633) (14358fe)
• sql: outer order by should take precedence over inner order by (4376c35)
• sql: overwrite the original sort key on successive order_by calls ordering by the same key (103dc68)
• sql: replace CTEs within CTEs (#8572) (182b6a5)
• sql: support set operations wrapping subqueries (#8414) (aab0c13)
• strings: make StringValue.capitalize() consistent across backends (#8270) (c4055d6), closes #8271
• structs: ensure that isin works with struct membership (#8978) (c0c508e)
• timestamps: use timezone aware objects instead of utcfromtimestamp (0b6ac2d)
• trino: compile property literal values directly instead of going throughh the pipeline (b2761c9)
• trino: generate first_value/lasts_value instead of arbitrary (710f8ac)
• trino: re-enable native TABLESAMPLE support (#8284) (75d154a)
• update error message when executing against unbound tables (#8695) (384b10f), closes #8677

Documentation

• 2024H1 roadmap and why VoDa supports Ibis (#8184) (7fa4334)
• add blog link on the homepage (ad7fa68)
• add blog on using duckdb for RAG (#8387) (fedf21b)
• add concepts guide on Datatypes and Datashapes (#8557) (cbeb6cf)
• add constants to width and height (#8640) (01b3f28)
• add explanation in rag blog and note on fixed length arrays (#8413) (ea58d22)
• add link to style and formatting in env setup section (#8936) (5095349)
• add lms for data post (#8222) (8f35010)
• add missing dev docker instructions for backends (#8352) (a20f44a)
• add missing import to RowNumber example (#8523) (ba21ec0)
• add Python + SQL section to why ibis (#8526) (211f336)
• add testimonial on combining Ibis with Kedro (5d23818)
• add v8 blog (#8200) (4adb5e3)
• added section on duckdb reading gcs files (#8651) (c2b06f6)
• altair: remove to_pandas from altair example (#8951) (eecbcea)
• api: expose relevant mixin members in temporal docs (cb97806)
• blog: add post introducing the Flink backend (#7912) (7bf764f), closes #1000 #2000 #3000 #7739
• blog: add post on stream-batch unification (#8293) (96b251d)
• blog: blog on why DuckDB is the default backend (#8378) (c916f6a), closes #8230
• blog: fix a formatting issue (#8473) (92bb6fd)
• blog: make point on stream-batch unification (#8316) (8ed4d4b)
• blog: needle haystack post (#8824) (ca97867)
• blog: portable dataflows with Ibis and Hamilton (#8798) (e97aa46)
• blog: update date on hamilton blog (#8851) (9cb4c8d)
• blog: wow analysis blog post (#8441) (6bd6c26)
• build: cache notebooks during render (#8950) (fd8858d)
• change date and remove links from why voda blog (#8287) (4d89300)
• clean up cursor closing and link to Table.sql API docs (#8417) (753a268), closes #8345
• colima: move section to its own header (865779b)
• concepts: remove outdated refs to SQLAlchemy (e932f6f)
• contribute: add a section to explain the different pytest markers that we use (3c74d84)
• create examples tab and populate with ibis-examples repo content (40db711)
• datafusion: update website link (#9072) (8df106f), closes /github.com/apache/datafusion/issues/9691#issuecomment-2063690461
• deps: pin itables to preserve grid styling (#8420) (fb91766)
• development: move pip to end of env creation tabset (d475660)
• dev: give the callout box a title (#8487) (c2a9ba9)
• duckdb: add short section on enabling duckdb geospatial (#8744) (c964bc5)
• env: give more actionable advice for using nix on apple m1 (#8716) (6639163)
• escape necessary characters (a4eb2d2)
• exasol: remove deprecated parameters from example (#8720) (becdca9)
• fix a couple of formatting issues (#8802) (7a8f65c)
• fix displaying the PyPI package name warning (#8237) (2d3297f)
• fix typo in order_by docstring (#8456) (ae3db0e)
• flink: remove conda warning from flink setup page (#8952) (dd9e928)
• flink: update conda-based installation guide (#8238) (f2ef90a)
• freeze clickhouse tutorial (#8312) (4bc7dc3)
• geospatial: update blog to use as_scalar and fix dependencies (#8543) (a845030)
• hierarchy: add brief doc listing backend hierarchy types (190b2c7)
• how-to: add a how-to guide for executing unbound expressions on backends (#8522) (66b4dc0), closes #8297
• how-to: add graphviz section (#8513) (f1bb060)
• improve nested type docstrings (#8358) (bc9d757)
• include return type in Interval component properties (#8193) (9b00657)
• links: fix release notes link in 8.0 release (#8315) (9387ac0)
• minor fixes to LM blog (#8236) (36f8904)
• presentations: fix quarto markup for Ibis origins slide in linkedin-meetup presentation (#9050) (e04c3e5)
• presentations: presentation for linkedin meetup 2024-04-24 (#8983) (01b521c)
• release-notes: fix up version header (#8941) (e583d2b)
• remove all extra installation from contributing docs (#9004) (03825b6)
• remove experimental indicator from all backends (586f979)
• rework why Ibis article to explain what Ibis is and other updates (#8490) (c44f357), closes #8251 #8488
• rewrite readme (#8524) (9a741f7), closes #8436
• risingwave: add backend docs page for RisingWave (a6e7920)
• roadmap 2024 H1 blog (#8362) (2750fe5)
• show how Array.unique() keeps NULLs (#8766) (7308249)
• snowflake: add blog post showing insertion into snowflake from postgres (#8426) (3a8c7cc)
• tutorial: add a tutorial for the Flink backend (#8085) (e2a3fb6)
• tutorial: add examples for complex topk (#8831) (2d9afe0)
• tutorial: deploy a jupyterlite repl on the website (#9009) (9355281)
• tutorial: update documentation regarding topk (#8583) (4115860)
• typo fix to why voda blog (882ade2)
• udfs: improve udf docstring examples (#9079) (fead40b)
• uniform connect via url across backends in why ibis (#8865) (e7f8ea8), closes #8860
• update contributing docs and env for m1/m2 devs (#8495) (1b84cd1)
• update environment setup instructions (#8476) (a18ab29)
• update links to duckdb csv parameters (#8733) (7cc67d3), closes #8732
• update outdated comments in Chaining Expressions docs (#8601) (306cdad)
• update sankey diagram (#9077) (b691959)
• update the geospatial blogpost to use explicit scalar subqueries (#8330) (aa74515)
• update the haystack blog publish date (#8949) (4ebeb19)
• update to_delta kwargs docs (#8623) (b77aeec), closes /github.com/ibis-project/ibis/blob/e94366700ae460052f675188674e24002b755ec5/ibis/backends/init.py#L536
• update TODOs in 8.0 blog (#8314) (4d4fdab)
• use algolia for searchability (#8410) (f8d8ca3)

Refactors

• add polars data mapper implementation (17f5e97)
• add polars format (40ada17)
• api: make input value coercion of mutate() identical to select() (#8878) (38e7e14)
• api: remove by of asof_join() in favor of predicates (#8700) (1a8eec8), closes #7869
• api: remove now unsupported max_lookback window attribute (99dda5b)
• api: remove the remnants of now unsupported timecontext feature (#8721) (0a00a05)
• api: remove unnecessary methods used to implement negate() (#8812) (f59f423)
• api: restrict arbitrary input nesting (#8917) (fd35b66)
• api: revamp asof join predicates (9fb3627)
• api: treat integer inputs as literals instead of column references (#8884) (feeb8ae), closes #8878
• backend-api: remove the database() API and implementation (#8406) (ff5d078), closes #8405
• backends: remove _metadata method in favor of _get_schema_using_query (#8627) (4ec7daf)
• backends: remove singledispatchmethod from the sql backends (#8338) (78dc393), closes #8283
• benchmarks: remove pandas benchmarking and replace with more-representative duckdb version (#8322) (e540575)
• bigquery: port to sqlglot (bcfd7e7)
• cleanup: remove deprecated methods and properties (#8701) (90c5a86)
• common: consolidate egraph implementation by using FrozenSlotted (#8702) (d244b75)
• common: make FrozenDict a subclass of dict (#8693) (32b7514), closes #8687 #8687
• common: support union types as well as forward references in the dispatch utilities (e4769de)
• compilers: conslidate StringJoin impl (28fb6ec)
• consolidate rewrite rule implementations (c9b8a08)
• dask: port the dask backend to the new execution model (#8005) (c925640)
• ddl: deprecate schema keyword in truncate_table (edde6a3)
• ddl: deprecate all *_schema methods (c43c0f1)
• delete some unused visit methods in sql compiler (f2e1465)
• deps: remove multipledispatch as a dependency (#8332) (d587166)
• druid: port to sqlglot (85e2b16)
• dtype: move all the castable logic to a single function (#8335) (580536c)
• duckdb/clickhouse: implement sqlglot backends and re-enable ci (b7821bf)
• duckdb: initial cut of sqlglot DuckDB compiler (ca95204)
• duckdb: remove the need for a specialized _to_geodataframe method (a8add4b)
• duckdb: use .sql instead of .execute in performance-senitive locations (#8669) (aa6aa0c)
• exasol: add custom TimestampTruncate (#8590) (66f12c3)
• exasol: add temp kwarg to create_table for api consistency (4267c72)
• exasol: port to sqlglot (#8032) (b8bcbf2)
• flink: port to sqlglot (8fdc75d)
• formats: remove unnecessary schema argument from schema inference (#8814) (91ea332)
• get_schema: remove schema kwarg, add catalog, kw-only (6273e7e)
• impala: port to sqlglot (4a0be76)
• ir: accept any relation in ops.ExistsSubquery (#8264) (68287db)
• ir: add JoinTable operation unique to JoinChain instead of using the globally unique SelfReference (2a7ae3f)
• ir: dereference literal expressions (cd9219b)
• ir: give unbound tables namespaces (8ac2d2a)
• ir: impure values are never dereferencible (#9023) (33286f2)
• ir: loosen the join integrity checks (#8817) (2bc903d)
• ir: merge ops.WindowFrame node into ops.WindowFunction (#8779) (3cd5a1a)
• ir: remove find_first_base_table() analysis function (#8451) (5cbd472)
• ir: remove deprecated Schema.apply_to() method (06962b2)
• ir: remove now obsolete __window_op__ property (8d27d14)
• ir: remove now obsolete ops.Named mixin (#8244) (3cff9f6)
• ir: remove the decimal precision promotion logic (0db3ec7)
• ir: split the relational operations (2a2f8c6)
• ir: stricter scalar subquery integrity checks (d269776)
• ir: support join of joins while avoiding nesting (40c1af7)
• ir: wrap JoinChain.first in ops.SelfReference similar to the rest of the join tables (15bd926)
• list_tables: deprecate schema keyword (ed69960)
• modules: remove unnecessary base submodule (#8405) (6946f98)
• mssql: port to sqlglot (e12faa5)
• mysql: port to sqlglot (#7926) (cba2f98)
• ops: use catalog and database kwargs for Namespace op (21f57d4)
• oracle: port to sqlglot (#8020) (fbdc909)
• oracle: remove dependency on private function for temp tables (#8480) (809912c)
• oracle: simplify oracle timestamp overrides (7d44d77)
• pandas: port the pandas backend with an improved execution model (#7797) (3b35d77), closes #7752
• pandas: simplify pandas helpers (41ddd41)
• polars: allow passing temp=False to polars create_table (89abf8e)
• polars: update the polars backend to use the new relational abstractions (#7868) (29b5b53)
• postgres: port to sqlglot (#7877) (6916c1d)
• pyspark: reimplement the backend using the new relational operations an spark SQL (32efbe7)
• pyspark: remove sqlalchemy dependency from pyspark (e9656fb)
• remove duplicated rewrite_sample rule (dd019e7)
• rename SupportSchema -> SchemaLike, fix type definition (#8427) (ad1f53a)
• risingwave: port to sqlglot (#8171) (1023e56)
• snowflake: adjust window frames to use _minimize_spec due to upstream snowflake changes (3a6c9f3)
• snowflake: remove no longer necessary pyarrow warnings filter (6a16899)
• snowflake: use sqlglot for the snowflake backend (83acf48)
• sql: automatically add simple ops implementations (#8349) (2c64b3f), closes #8338
• sql: deprecate schema kwarg in insert (92fcbdf)
• sql: deprecate schema in _view ddl methods (2af37e8)
• sqlglot: clean up explode usage (4d99314)
• sqlglot: make anonymous functions easier to use and remove array_func hack (5891546)
• sqlglot: remove duplicate StringAscii definitions (0bdeb8b)
• sqlglot: remove duplicated simple compilation rules and sort (d44001f)
• sqlglot: use a more backend-agnostic expression for non-finite constants (776079c)
• sqlglot: various sqlglot compiler and backend clean ups (#7904) (db45e41), closes #7871
• sqlite: port to SQLGlot (#8154) (4d24502)
• sql: move dialects to always-importable location (#8279) (c8f4afe)
• sql: remove old compiler (#8307) (88e8384)
• sql: remove sqlalchemy from the codebase (#8074) (22004ed)
• sql: remove temporary table creation when using inline sql (#8149) (ea428ba)
• sql: reorganize sqlglot rewrites (9f4851c)
• sql: simplify FirstValue/LastValue usage (#8568) (6ed2e39)
• table ddl: remove hierarchical schema from *_table methods (14b0944)
• table: deprecate schema (dfb8734)
• udf: remove hierarchical usage of schema (#9078) (f5d9084)
• udfs: consolidate builtin udf compilation and failure modes for unimplemented udfs (#8398) (0320d01)
• util: remove unused get_logger utility function (#8760) (0aceefc)

Performance

• common: improve equality caching by explicitly invalidating the entry on __del__ (#8708) (ac86f91)
• common: improve the performance of replacing nodes using mappings (#8638) (a2e733a)
• common: reduce the average recusrion depth in _flatten_collections (#8709) (3d52904)
• dask: avoid triggering compute for dynamic limit/offset (#8747) (b3e27eb)
• ir: avoid exponential growth on name attribute access (#8445) (7667328), closes #8432
• sql: don't compile pretty sql by default (#8616) (af402f9)
• sql: prevent sqlglot from extensive deepcopying every time we create a sqlglot object (#8592) (461293b), closes #8484

Deprecations

• api: deprecate how parameter to arbitrary (b514c23)
• deprecate and warn on legacy udf usage (#8617) (e561889)