Spice v0.13.1-alpha (May 27, 2024)

By Spice AI (@spice_ai) | Monday, May 27, 2024

Tags:

The v0.13.1-alpha release of Spice is a minor update focused on stability, quality, and operability. Query result caching provides protection against bursts of queries and schema support for datasets has been added logical grouping. An issue where Refresh SQL predicates were not pushed down underlying data sources has been resolved along with improved Acceleration Refresh logging.

Highlights in v0.13.1-alpha

Results Caching: Introduced query results caching to handle bursts of requests and support caching of non-accelerated results, such as refresh data returned on zero results. Results caching is enabled by default with a 1s item time-to-live (TTL). Learn more.
Query History Logging: Recent queries are now logged in the new spice.runtime.query_history dataset with a default retention of 24-hours. Query history is initially enabled for HTTP queries only (not Arrow Flight queries).
Dataset Schemas: Added support for dataset schemas, allowing logical grouping of datasets by separating the schema name from the table name with a .. E.g.
```
datasets:
  - from: mysql:app1.identities
    name: app.users

  - from: postgres:app2.purchases
    name: app.purchases
```
In this example, queries against app.users will be federated to my_schema.my_table, and app.purchases will be federated to app2.purchases.

Contributors

@y-f-u @Jeadie @sgrebnov @ewgenius @phillipleblanc @lukekim @gloomweaver @Sevenannn

New in this release

Add more type support on mysql connector by @y-f-u in https://github.com/spiceai/spiceai/pull/1449
Add in-memory caching support for Arrow Flight queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1450
Fix the table reference to use the full table reference, not just the table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1460
Make file_format parameter required for S3/FTP/SFTP connector by @ewgenius in https://github.com/spiceai/spiceai/pull/1455
Add more verbose logging when acceleration refresh update is finished by @y-f-u in https://github.com/spiceai/spiceai/pull/1453
Fix snowflake dataset path when using federation query by @y-f-u in https://github.com/spiceai/spiceai/pull/1474
Update cargo to use spiceai datafusion fork by @y-f-u in https://github.com/spiceai/spiceai/pull/1475
Enable in-memory results caching by default by @sgrebnov in https://github.com/spiceai/spiceai/pull/1473
Add basic integration test for MySQL federation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1477
Update results_cache config names per final spec by @sgrebnov in https://github.com/spiceai/spiceai/pull/1487
Add DuckDB quickstart to E2E tests by @lukekim in https://github.com/spiceai/spiceai/pull/1461
Add X-Cache header for http queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1472
Add telemetry for in-memory caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1456
Pin Git dependencies to a specific commit hash by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1490
Detect file_format from dataset path by @ewgenius in https://github.com/spiceai/spiceai/pull/1489
Add file_format to helm chart sample dataset by @ewgenius in https://github.com/spiceai/spiceai/pull/1493
Improve duckdb data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1486
Add file_format prompt for s3 and ftp datasets in Dataset Configure CLI if no extension detected by @ewgenius in https://github.com/spiceai/spiceai/pull/1494
Add llms to the spicepod definition and use throughout by @Jeadie in https://github.com/spiceai/spiceai/pull/1447
Fix duckdb acceleration converting null into default values. by @y-f-u in https://github.com/spiceai/spiceai/pull/1500
Separate runtime Dataset from spicepod Dataset by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1503
Duckdb e2e test OSX support by @y-f-u in https://github.com/spiceai/spiceai/pull/1505
Use TableReference for dataset name by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1506
Tweak Results Cache naming and output by @lukekim in https://github.com/spiceai/spiceai/pull/1509
Fix refresh_sql not properly passing down filters by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1510
Allow datasets to specify a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1507
Cache invalidation for accelerated tables by @sgrebnov in https://github.com/spiceai/spiceai/pull/1498
Improve spark data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1497
Parse postgres table schema from prepare statement to support empty tables by @ewgenius in https://github.com/spiceai/spiceai/pull/1445
Improve clarity of README and add FAQ by @lukekim in https://github.com/spiceai/spiceai/pull/1512
Use binary data transfer for ftp by @gloomweaver in https://github.com/spiceai/spiceai/pull/1517
Add support for time64 for SQL insertion statement by @y-f-u in https://github.com/spiceai/spiceai/pull/1519
Add Spice Extensions PoC by @ewgenius in https://github.com/spiceai/spiceai/pull/1476
Add results cache metrics, pod and quantile filters to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1513
Add unit tests for results caching utils by @sgrebnov in https://github.com/spiceai/spiceai/pull/1514
Add E2E tests for results caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1515
Pass table_reference full string into spark_session table so it can query across schemas or catalogs by @y-f-u in https://github.com/spiceai/spiceai/pull/1521
Trace on debug level for tables in runtime schema by @ewgenius in https://github.com/spiceai/spiceai/pull/1524
Update SparkSessionBuilder::remote and update spark fork hash by @Sevenannn in https://github.com/spiceai/spiceai/pull/1495
Fix federation push-down for datasets with schemas by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1526
Store history of queries in ‘spice.runtime.query_history’ by @Jeadie in https://github.com/spiceai/spiceai/pull/1501
Disable cache for system queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1528
Register runtime tables with runtime schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1532
Fix acknowledgments workflow to include all cargo features by @Jeadie in https://github.com/spiceai/spiceai/pull/1531

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.0-alpha...v0.13.1-alpha

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Twitter: @spice_ai
Discord: https://discord.gg/kZnTfneP5u
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: [email protected]