How to convert a file through an fsspec chain

This guide shows you how to expose a source file in another tabular format and Arrow schema without replacing its filesystem.

Define schemas for text input

CSV and JSONL sources require an Arrow schema. Use nested dictionaries when the same options must cross the fsspec-rs Python bridge:

schema = {
    "fields": [
        {"name": "id", "type": "int64", "nullable": False},
        {"name": "name", "type": "string", "nullable": True},
    ]
}

Pass a pyarrow.Schema instead when the chain remains in Python. Arrow IPC and Parquet sources provide their own schemas, so provided_schema is optional for those formats.

Resolve schemas from independent filesystems

Use a schema reference when a schema is stored separately from the data. Each reference owns its storage options, so provided and requested schemas can use different backends or credentials:

provided_schema = {
    "url": "s3://input-schemas/orders.json",
    "format": "arrow-json",
    "storage_options": {"profile": "input-schema-reader"},
}
requested_schema = {
    "url": "s3://output-schemas/orders.schema.json",
    "format": "json-schema",
    "storage_options": {"profile": "output-schema-reader"},
}

arrow-json uses the fields descriptor shown above. arrow-ipc-schema accepts bytes from pyarrow.Schema.serialize() or an Arrow IPC stream. A string URL is shorthand for a reference when its suffix identifies an Arrow IPC schema:

requested_schema = "s3://output-schemas/orders.arrow"

json-schema accepts a structural JSON Schema whose root is an object. It maps JSON Schema objects, arrays with one items schema, strings, booleans, integers, numbers, and nulls to Arrow. A union may contain null and one other type. Object required entries determine Arrow field nullability; optional properties and explicit null unions are nullable.

References, composition keywords, multiple non-null union members, schema-valued additionalProperties, tuple arrays, and format annotations are rejected with a path to the unsupported field. Validation-only keywords such as numeric bounds and string lengths are not represented or enforced by the Arrow schema.

Resolve JSON Schema from Confluent Schema Registry

Use a confluent://SUBJECT/versions/VERSION reference for a JSON Schema registered with Confluent. Put registry connection options on that reference:

provided_schema = {
    "url": "confluent://orders-value/versions/latest",
    "format": "json-schema",
    "storage_options": {
        "registry_url": "https://schema-registry.example.com",
        "username": "schema-api-key",
        "password": "schema-api-secret",
        "timeout": 10,
    },
}

Pass headers in storage_options for additional string-valued HTTP headers. The resolver supports JSON Schema records without external schema references. It rejects Avro, Protobuf, and registry records that contain references.

Open the converted representation

Place the requested representation on the left and the source object on the right:

import fsspec
import pyarrow.parquet as pq

url = "fsspec-data://orders.parquet::memory://orders.csv"

with fsspec.open(url, provided_schema=schema) as file:
    orders = pq.read_table(file)

The outer filename selects Parquet output. The inner filename selects CSV input. Use provided_format or requested_format when a filename has no recognized suffix.

To project, reorder, or cast fields, pass requested_schema and the corresponding schema_policy:

with fsspec.open(
    url,
    provided_schema=provided_schema,
    requested_schema=requested_schema,
    schema_policy="projection",
) as file:
    names = pq.read_table(file)

The data filesystem and each schema reference keep separate options. For example, the inner S3 URL can use fsspec’s s3 options while the provided and requested schema references use different registry credentials.

For a database source, pass connection settings to the inner protocol:

url = "fsspec-data://orders.parquet::db+duckdb:///main/orders.arrow"

with fsspec.open(url, **{"db+duckdb": {"database": "warehouse.duckdb"}}) as file:
    orders = pq.read_table(file)

See the API reference for constructor options and supported suffixes.