# Interchange design

`fsspec-data` is the interchange boundary between packages that produce tabular data and
packages that consume it. It reconciles formats and schemas without absorbing database,
dataframe, or user-interface responsibilities.

## Data, schemas, projection, and conversion

Three independently located inputs define one interchange request: data source A, provided
schema B, and requested schema C. B describes how to interpret A. C describes the columns and
types the consumer wants. A projection is the logical mapping from B to C; conversion is the
physical decoding and encoding around that mapping.

```mermaid
flowchart LR
    A["A · Data source<br/>database · S3 · filesystem"]
    B["B · Provided schema<br/>inline · file · registry"]
    C["C · Requested schema<br/>inline · file · registry"]

    B --> RB["Resolve with B credentials"]
    C --> RC["Resolve with C credentials"]
    RB --> BS["Provided Arrow schema"]
    RC --> CS["Requested Arrow schema"]
    BS --> P["Projection plan<br/>select · reorder · rename · add · cast"]
    CS --> P

    A --> D["Decode source format"]
    D --> AB["Arrow record batches"]
    AB --> X["Apply projection"]
    P --> X
    X --> E["Encode requested format"]
    E --> O["Requested representation<br/>Parquet · Arrow · CSV · JSONL"]

    A -.->|native model/store route| R["Converter registry<br/>built-in · entry point"]
    R --> N["Native conversion<br/>Xarray ↔ Zarr · plugin X ↔ Y"]
    N --> NO["Native target<br/>object · store · resource"]

    classDef external fill:#eef5ff,stroke:#4c78a8,color:#152238
    classDef schema fill:#f4eeff,stroke:#7a5195,color:#24152e
    classDef operation fill:#fff4df,stroke:#d28e2c,color:#38250a
    classDef arrow fill:#e8f6ef,stroke:#3a8f67,color:#102d21
    class A,B,C,O,NO external
    class RB,RC,BS,CS schema
    class D,P,X,E,R,N operation
    class AB arrow
```

Schema resolution may perform I/O, but it completes before source data is opened. Each
reference carries only its own storage or registry credentials. Once B and C are Arrow
schemas, planning is pure: it validates the requested projection without reading A. Execution
then decodes, applies that stable plan batch by batch, and encodes the requested representation.

## Why resource converters sit beside Arrow codecs

Arrow batches are the right boundary for tabular, single-object formats. They are not a
lossless universal representation for every data model. An Xarray dataset can contain
N-dimensional variables, named dimensions, coordinates, and attributes, while a Zarr group
is a hierarchy of metadata and chunk objects rather than one seekable file.

Forcing those resources through the file-codec path would either flatten model semantics or
pretend a multi-object store is one file. The converter registry therefore provides a sibling
route. A converter owns the semantics of its source and target types and may return an object,
write a store, or use Arrow internally when that is appropriate. Entry-point discovery lets
integration packages add routes without adding their dependencies or release cadence to
`fsspec-data`.

`DataFileSystem` remains the read-only façade for single-file tabular conversion. Native
resource conversion uses `ConverterRegistry` directly because its output may not satisfy a
file-like `open()` contract.

## Arrow is the internal boundary

Arrow provides one typed, columnar representation for schema comparison, casting, and
batch transport. Codecs translate external encodings into Arrow batches and back again.
Adding a format therefore requires one Arrow decoder and encoder instead of converters for
every pair of formats.

The Rust core and Python bindings share this boundary. Schema decisions, cast
classifications, batch limits, and cancellation semantics remain consistent whether an
integration enters through Rust or PyArrow.

## Planning precedes execution

External schema references are resolved to Arrow schemas before an `InterchangeRequest` is
created. Resolution may read another filesystem or registry and records schema provenance;
it is separate from planning.

JSON Schema resolution translates only structural type information. Arrow schemas do not
encode the complete JSON Schema validation language, so composition, references, ambiguous
unions, and format-dependent conversions are rejected. Validation-only constraints are not
enforced during interchange.

An `InterchangeRequest` separates validation from source-data access. Planning checks that codecs,
field mappings, nullability changes, and casts can satisfy the requested contract without
reading input. Execution then applies that stable plan to each batch.

This split lets an integrating package reject an unsupported request before starting a
database scan or opening an output sink. Runtime checks remain necessary only for facts the
schema cannot prove, such as whether a string contains an integer or a nullable column
actually contains nulls.

## Streaming is the default execution model

Record batches keep memory bounded and allow a consumer to stop early. Row, batch, and byte
limits protect previews and interactive clients from unexpectedly large inputs.
Cancellation propagates through the planned stream to its decoder.

Encoded conversion is a buffering adapter on top of that stream. It exists for consumers
that require a complete byte buffer, while the batch iterator remains the primary boundary
for scans and previews.

## Package responsibilities remain narrow

- Database libraries own connections, discovery, SQL, predicate pushdown, and
  database-to-Arrow mapping.
- Dataframe integrations own expression translation and local fallback execution.
- Browsers own pagination, rendering, and request lifecycles.
- `fsspec-data` owns schema resolution, projections, format conversion, casts, and
  interchange limits.

These boundaries keep backend-specific semantics close to each backend and make the
interchange layer reusable by all of them.

## Current transport boundary

Python codec methods accept encoded input as bytes-like objects or binary file-like readers.
Reader-backed input lets row limits and cancellation stop upstream reads before EOF instead
of requiring the complete encoded source to cross the Python/Rust boundary first.

Parquet readers must be seekable because Parquet metadata is stored in the footer and column
chunks can reside at different offsets. Arrow IPC, CSV, and JSONL consume readers
sequentially, but use the same seekable reader contract so registry consumers have one input
boundary.

`DataFileSystem` opens its inner source through fsspec, applies its schema plan batch by
batch, and writes encoded output directly into a seekable spooled file. This bounds
intermediate transport memory without changing the filesystem contract: `_open()` still
completes conversion before returning the spool, and `info()` may perform a complete
conversion to determine output size. Individual format writers may also buffer internally;
in particular, Parquet can defer encoded output until `finish`.