Interchange API reference

DataFileSystem

Read-only chained filesystem registered as the fsspec-data protocol. The outer path names the requested representation; fo names the source object on the target filesystem.

Constructor parameters:

• fo: str: source object path or URL.

• target_protocol: str | None: protocol used to construct the target filesystem.

• target_options: dict | None: options passed to the target filesystem.

• fs: AbstractFileSystem | None: existing target filesystem. Mutually exclusive with target_protocol.

• provided_format: DataFormat | str | None: source format. Inferred from fo when omitted.

• requested_format: DataFormat | str | None: output format. Inferred from the opened path when omitted.

• provided_schema: source pyarrow.Schema, nested schema options, or schema reference. Required for CSV and JSONL.

• requested_schema: output pyarrow.Schema, nested schema options, or schema reference. Defaults to the source schema.

• schema_policy: SchemaPolicy | str: reconciliation policy. Defaults to exact.

• batch_size: int: maximum rows per decoded batch. Defaults to 1024.

• row_limit: int | None: maximum decoded rows.

• byte_limit: int | None: maximum cumulative Arrow array memory in decoded batches.

• spool_max_size: int: converted bytes retained in memory before the seekable output rolls over to a temporary file. Defaults to 8 MiB.

Nested schema options have a fields list. Each field has name, type, and optional nullable keys. String types use PyArrow aliases such as int64, string, and timestamp[ms].

A schema reference is a SchemaRef, URL string, or mapping with url, optional format, and optional storage_options. Reference storage options apply only to that schema source. Built-in formats are arrow-json for nested field descriptors, arrow-ipc-schema for a serialized Arrow schema or IPC stream, and json-schema for the supported structural JSON Schema subset. .arrow, .ipc, and .arrowschema infer arrow-ipc-schema; ambiguous extensions require an explicit format.

The confluent://SUBJECT/versions/VERSION provider accepts latest, -1, or a positive 32-bit integer as VERSION, and these storage_options:

• registry_url: str: required HTTP or HTTPS Schema Registry base URL.

• username: str and password: str: optional Basic authentication pair.

• headers: Mapping[str, str]: optional additional request headers.

• timeout: int | float: optional request timeout in seconds.

The provider supports registry documents with schemaType JSON and no external schema references. Avro, Protobuf, and referenced documents raise ValueError.

After conversion starts, provided_schema_provenance and requested_schema_provenance describe resolved schema sources without exposing storage options.

Recognized suffixes are .arrow and .ipc for Arrow IPC streams, .parquet and .pq for Parquet, .csv for CSV, and .jsonl and .ndjson for line-delimited JSON.

open accepts read mode and returns a seekable spooled file. Conversion currently reads the complete encoded source and produces a complete encoded output before returning the file.

See How to convert a file through an fsspec chain for usage.

SchemaRef

Identifies an independently resolvable schema.

Attributes:

• url: str: schema location.

• format: SchemaFormat | str | None: schema representation. Inferred for recognized Arrow schema suffixes.

• storage_options: Mapping[str, Any]: options passed only to the schema filesystem. Omitted from repr.

ResolvedSchema

Contains the resolved pyarrow.Schema and SchemaProvenance. Resolution performs I/O before interchange planning; the planner continues to receive only Arrow schemas.

SchemaProvenance

Describes a resolved schema without credentials.

Attributes:

• source: str: original schema reference URL or inline.

• format: str: decoded schema representation.

• provider: str: inline, fsspec, confluent, or a registered provider name.

• identifier: str | None: provider-specific schema identifier.

• version: str | None: provider-specific schema version.

SchemaDocument

Contains provider-returned schema bytes, their SchemaFormat, and SchemaProvenance.

SchemaResolverRegistry

Separates schema transport providers from schema-language decoders.

register(format, decoder)

Registers a decoder accepting schema bytes and returning pyarrow.Schema.

register_provider(protocol, provider)

Registers a provider accepting SchemaRef and returning SchemaDocument.

DEFAULT_SCHEMA_RESOLVERS includes decoders for arrow-json, arrow-ipc-schema, and json-schema; a confluent provider; and an fsspec provider fallback for other protocols.

Use resolve_schema to resolve an inline schema or normalized SchemaRef directly.

json_schema_to_arrow

Converts the supported JSON Schema subset directly to pyarrow.Schema. The root must have type object. Supported value types are object, array with one items schema, string, boolean, integer as Arrow int64, number as Arrow float64, and null. A type array may contain null and one non-null type.

Object required entries and null unions determine Arrow field nullability. References, composition, multiple non-null union members, tuple arrays, schema-valued additional properties, and format annotations raise ValueError with a schema path. JSON Schema validation constraints are not represented or enforced by Arrow.

Converter

Describes one native conversion route.

Attributes:

• source_type: str: normalized, case-insensitive source type.

• target_type: str: normalized, case-insensitive target type.

• handler: Callable: conversion function, omitted from repr.

The handler signature is:

handler(
    source,
    target,
    *,
    source_options,
    target_options,
    conversion_options,
)

All option arguments are dictionaries. The handler return value is the converter result.

convert(source, target=None, *, source_options=None, target_options=None, conversion_options=None)

Copies the three option mappings and calls the handler. Raises TypeError when an option value is not a mapping.

ConverterRegistry

Maps normalized (source_type, target_type) pairs to Converter objects. Installed converter entry points are loaded at most once, on first lookup.

Constructor parameters:

• entry_point_group: str: discovery group. Defaults to fsspec_data.converters.

• discover_entry_points: bool: enables installed entry-point discovery. Defaults to True.

register(converter, *, replace=False)

Registers a Converter. Raises ValueError for a duplicate route unless replace=True.

load_entry_points()

Loads entry points from the configured group. Each entry point must load one Converter. Duplicate routes and invalid objects raise errors before discovered converters are added.

get(source_type, target_type)

Returns the converter for a route. Raises ValueError when no converter is registered.

convert(source_type, target_type, source, target=None, **options)

Gets the route and calls Converter.convert.

DEFAULT_CONVERTERS

Default ConverterRegistry. Built-in routes are zarr to xarray and xarray to zarr. They import Xarray only when invoked and require the fsspec-data[xarray] optional dependency.

The Zarr-to-Xarray route calls xarray.open_zarr. source_options become its storage_options; conversion_options supply options such as group and chunks. It returns an Xarray Dataset and rejects a target or target options.

The Xarray-to-Zarr route accepts an Xarray Dataset or DataArray and calls to_zarr. target_options become its storage_options; conversion_options supply options such as mode, group, and zarr_format.

DataFormat

Identifies an interchange encoding. Values are arrow, parquet, csv, and jsonl. arrow denotes an Arrow IPC stream. jsonl denotes line-delimited JSON records.

SchemaPolicy

Controls schema reconciliation.

Value	Behavior
exact	Requires equal names, order, Arrow types, and nullability.
projection	Selects or reorders fields without casts or nullable narrowing.
compatible	Preserves field order and permits lossless casts and nullable widening.
coerce	Preserves field order and applies casts registered by the coercion core.

Compatible casts are signed and unsigned integer widening, unsigned-to-signed integer casts where the target has more bits, float widening, UTF-8 to large UTF-8, date32 to date64, and null to a requested type. Coercion additionally supports primitive numeric conversions, large UTF-8 to UTF-8, and conversions between primitive numeric or boolean types and strings.

Coercions are classified as safe, lossy, or runtime_checked. Nullable-to-required coercion performs a runtime null check. Nested-type coercions are not registered.

FieldMapping

Describes one output field.

Attributes:

• source_index: int: index in the provided schema.

• target_index: int: index in the requested schema.

• cast: str | None: safe, lossy, runtime_checked, or None.

• check_nulls: bool: whether execution must reject null input values.

CodecCapabilities

Describes operations supported by a codec.

Attributes:

• encode: bool: accepts Arrow record batches and produces encoded bytes.

• decode: bool: accepts encoded bytes and produces Arrow record batches.

• streaming: bool: consumes and produces record batches without requiring an Arrow table.

DecodedBatches

Contains buffered decoded output.

Attributes:

• schema: pyarrow.Schema: decoded schema.

• batches: tuple[pyarrow.RecordBatch, ...]: decoded batches in order.

DecodedBatchStream

Iterates lazily over decoded pyarrow.RecordBatch values.

Attributes:

• schema: pyarrow.Schema: decoded schema.

Methods:

cancel()

Cancels the stream. Its next iteration raises RuntimeError; later iterations stop.

collect()

Consumes the stream and returns DecodedBatches.

Codec

Represents a registered format codec.

Attributes:

• format: DataFormat: encoding handled by the codec.

• capabilities: CodecCapabilities: supported operations.

encode_batches(batches, *, schema=None)

Returns encoded bytes for Arrow record batches. schema is required when batches is empty. All batches must match the encoding schema.

Raises TypeError for non-Arrow inputs and ValueError for missing or mismatched schemas.

encode_batches_to(batches, output, *, schema=None)

Consumes an iterable of Arrow record batches and writes encoded data to a binary file-like output. Returns None and leaves output open. schema is required when batches is empty. All batches must match the encoding schema.

Raises TypeError for non-Arrow batches or a non-schema schema, ValueError for missing or mismatched schemas, and OSError when the output writer fails.

iter_batches(data, *, schema=None, batch_size=1024, row_limit=None, byte_limit=None)

Returns DecodedBatchStream.

Parameters:

• data: bytes | bytearray | memoryview | BinaryIO: encoded input. File-like inputs must return bytes from read; Parquet inputs must also support seek.

• schema: pyarrow.Schema | None: required for CSV and JSONL; optional for Arrow IPC and Parquet.

• batch_size: int: maximum rows yielded in one batch. Must be greater than zero.

• row_limit: int | None: maximum total rows yielded. The last batch is sliced as needed.

• byte_limit: int | None: maximum cumulative Arrow array memory referenced by yielded batches. Exceeding it raises ValueError.

decode_batches(...)

Calls iter_batches(...), consumes the stream, and returns DecodedBatches. Parameters and errors match iter_batches.

Rust CodecReader

A marker trait for reader-backed encoded input. Implementations provide Read + Seek + Send.

Codec::decode_reader accepts a boxed CodecReader. The built-in Arrow IPC, Parquet, CSV, and JSONL codecs decode directly from the reader. The default implementation for other codecs buffers the reader and calls decode_stream, preserving compatibility with existing Codec implementations.

Parquet decoding uses seekable range reads because file metadata and column chunks may be located at different offsets.

Rust CodecWriter

A resumable encoded-output session returned by Codec::start_writer.

Methods:

• write_batch(&mut self, batch: &RecordBatch): validates the batch against the session schema and submits it to the format writer.

• finish(self: Box<Self>): writes the format footer and consumes the session. The borrowed sink remains owned by the caller.

Arrow IPC, Parquet, CSV, and JSONL codecs support resumable writers.

Arrow IPC makes encoded batch bytes available to the sink during write_batch. The Parquet writer accepts batches incrementally but may buffer its output until finish.

Codec::encode_stream uses the same writer session for all four codecs.

Python Codec.encode_batches_to uses an owned writer session backed by the supplied binary file object.

Codec::start_writer(schema, output)

Borrows a Write + Send sink and returns a writer session with the same lifetime. This form supports scoped encoding while preserving direct access to the sink after finish.

Codec::start_owned_writer(schema, output)

Takes a boxed Write + Send sink and returns a writer session without a borrowed lifetime. This form supports storing the writer in a long-lived file or stream object. Shared or cloneable sinks allow the caller to observe output while the session owns its sink handle.

CodecRegistry

get(format)

Returns the registered Codec for a DataFormat or its string value. Raises ValueError for unknown values and NotImplementedError when a known format has no registered codec.

DEFAULT_REGISTRY contains Arrow IPC stream, Parquet, CSV, and JSONL codecs.

Arrow IPC preserves input batch boundaries up to batch_size. Parquet preserves schema and row order but may select different output batch boundaries. CSV writes a header row. CSV and JSONL require an explicit decode schema because neither encoding stores a complete Arrow schema.

InterchangeRequest

Describes one planned conversion.

Attributes:

• provided_format: DataFormat: source encoding.

• requested_format: DataFormat: target encoding.

• provided_schema: pyarrow.Schema: source schema.

• requested_schema: pyarrow.Schema: target schema.

• policy: SchemaPolicy: reconciliation policy.

plan()

Validates registered source and target codecs and returns InterchangePlan. Planning reads no data.

InterchangePlan

Represents a validated format and schema conversion.

Attributes:

• provided_schema: pyarrow.Schema: schema accepted as input.

• requested_schema: pyarrow.Schema: schema produced as output.

• policy: SchemaPolicy: applied schema policy.

• mappings: tuple[FieldMapping, ...]: ordered field mappings.

• provided_format: DataFormat: source encoding.

• requested_format: DataFormat: target encoding.

apply_table(table)

Returns a pyarrow.Table with mappings, casts, and null checks applied.

apply_batch(batch)

Returns a pyarrow.RecordBatch with mappings, casts, and null checks applied.

Both methods raise ValueError when the input schema differs from provided_schema, a runtime null check fails, or a runtime-checked cast fails.

iter_batches(data, *, batch_size=1024, row_limit=None, byte_limit=None)

Returns PlannedBatchStream. It decodes provided_format lazily and applies the schema plan to each batch. Limits apply to decoded input batches.

convert(data, *, batch_size=1024, row_limit=None, byte_limit=None)

Consumes iter_batches, encodes requested_format, and returns bytes. This method is an explicit buffering adapter.

PlannedBatchStream

Iterates lazily over planned pyarrow.RecordBatch values. schema is the requested schema. cancel() cancels the underlying decoded stream.

plan_schema(provided_schema, requested_schema, policy)

Returns an Arrow-to-Arrow InterchangePlan without reading data.

Raises TypeError for non-PyArrow schemas and ValueError when schemas violate the selected policy or require an unregistered cast.

import pyarrow as pa

from fsspec_data import SchemaPolicy, plan_schema

provided = pa.schema([pa.field("id", pa.int32(), nullable=False)])
requested = pa.schema([pa.field("id", pa.int64(), nullable=True)])
plan = plan_schema(provided, requested, SchemaPolicy.COMPATIBLE)