fsspec-data

Arrow interchange planning for the fsspec data ecosystem.

fsspec-data reconciles provided and requested tabular schemas and formats without taking ownership of database discovery or dataframe query planning. A pure-Rust core and Python bindings apply the same decisions to Arrow record batches and PyArrow objects.

The first release supports format-aware conversion plans, exact/projection/compatible/ coerce schema policies, registered Arrow IPC/Parquet/CSV/JSONL codecs, bounded lazy batch iteration with cancellation, and read-only conversion through chained fsspec URLs.

Documentation

• Convert tabular data in a guided first example.

• Connect a batch producer or consumer to the interchange layer.

• Convert a file through an fsspec chain while retaining its source filesystem.

• Understand the interchange design and package boundaries.

• Look up the interchange API, including policies, codecs, and errors.