fsspec_data

Submodules

• fsspec_data._version
• fsspec_data.backends
• fsspec_data.confluent
• fsspec_data.converters
• fsspec_data.filesystem
• fsspec_data.interchange
• fsspec_data.json_schema
• fsspec_data.schema
• fsspec_data.tests
• fsspec_data.xarray_zarr

Attributes

CONVERTER_ENTRY_POINT_GROUP
DEFAULT_CONVERTERS
DEFAULT_REGISTRY
DEFAULT_SCHEMA_RESOLVERS
__version__

Classes

Converter
ConverterRegistry
DataFileSystem	Read-only format and schema conversion layered over another filesystem.
Codec
CodecCapabilities
CodecRegistry
DataFormat	str(object='') -> str
DecodedBatches
DecodedBatchStream
FieldMapping
InterchangePlan
InterchangeRequest
PlannedBatchStream
SchemaPolicy	str(object='') -> str
ResolvedSchema
SchemaDocument
SchemaFormat	str(object='') -> str
SchemaProvenance
SchemaRef
SchemaResolverRegistry

Functions

plan_schema(→ InterchangePlan)
json_schema_to_arrow(→ pyarrow.Schema)
resolve_schema(→ ResolvedSchema | None)

Package Contents

fsspec_data.CONVERTER_ENTRY_POINT_GROUP = 'fsspec_data.converters'

fsspec_data.DEFAULT_CONVERTERS

class fsspec_data.Converter

source_type: str

target_type: str

handler: ConverterHandler

__post_init__() → None

convert(source: Any, target: Any | None = None, *, source_options: collections.abc.Mapping[str, Any] | None = None, target_options: collections.abc.Mapping[str, Any] | None = None, conversion_options: collections.abc.Mapping[str, Any] | None = None) → Any

class fsspec_data.ConverterRegistry(*, entry_point_group: str = CONVERTER_ENTRY_POINT_GROUP, discover_entry_points: bool = True)

entry_point_group = 'fsspec_data.converters'

discover_entry_points = True

_converters: dict[tuple[str, str], Converter]

_entry_points_loaded = False

register(converter: Converter, *, replace: bool = False) → None

load_entry_points() → None

get(source_type: str, target_type: str) → Converter

convert(source_type: str, target_type: str, source: Any, target: Any | None = None, *, source_options: collections.abc.Mapping[str, Any] | None = None, target_options: collections.abc.Mapping[str, Any] | None = None, conversion_options: collections.abc.Mapping[str, Any] | None = None) → Any

class fsspec_data.DataFileSystem(fo: str, target_protocol: str | None = None, target_options: dict[str, Any] | None = None, fs: fsspec.AbstractFileSystem | None = None, provided_format: fsspec_data.interchange.DataFormat | str | None = None, requested_format: fsspec_data.interchange.DataFormat | str | None = None, provided_schema: fsspec_data.schema.SchemaInput = None, requested_schema: fsspec_data.schema.SchemaInput = None, schema_policy: fsspec_data.interchange.SchemaPolicy | str = SchemaPolicy.EXACT, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None, spool_max_size: int = 8 * 1024 * 1024, **kwargs: Any)

Bases: fsspec.implementations.chained.ChainedFileSystem

Read-only format and schema conversion layered over another filesystem.

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters:

• use_listings_cache – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• listings_expiry_time – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• max_paths – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• skip_instance_cache (bool) – If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

• asynchronous (bool)

• loop (asyncio-compatible IOLoop or None)

protocol = 'fsspec-data'

fs = None

fo

provided_format

requested_format

provided_schema = None

requested_schema = None

provided_schema_provenance: fsspec_data.schema.SchemaProvenance | None = None

requested_schema_provenance: fsspec_data.schema.SchemaProvenance | None = None

schema_policy

batch_size = 1024

row_limit = None

byte_limit = None

spool_max_size = 8388608

_sizes: dict[str, int]

_open(path: str, mode: str = 'rb', block_size: int | None = None, autocommit: bool = True, cache_options: dict[str, Any] | None = None, **kwargs: Any)

Return raw bytes-mode file-like from the file-system

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

_convert(path: str, output) → None

fsspec_data.DEFAULT_REGISTRY

class fsspec_data.Codec

format: DataFormat

capabilities: CodecCapabilities

encode_batches(batches: list[pyarrow.RecordBatch] | tuple[pyarrow.RecordBatch, Ellipsis], *, schema: pyarrow.Schema | None = None) → bytes

encode_batches_to(batches: collections.abc.Iterable[pyarrow.RecordBatch], output: BinaryIO, *, schema: pyarrow.Schema | None = None) → None

decode_batches(data: bytes | bytearray | memoryview | BinaryIO, *, schema: pyarrow.Schema | None = None, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None) → DecodedBatches

iter_batches(data: bytes | bytearray | memoryview | BinaryIO, *, schema: pyarrow.Schema | None = None, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None) → DecodedBatchStream

class fsspec_data.CodecCapabilities

encode: bool

decode: bool

streaming: bool

class fsspec_data.CodecRegistry

get(format: DataFormat | str) → Codec

class fsspec_data.DataFormat

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

Initialize self. See help(type(self)) for accurate signature.

ARROW = 'arrow'

PARQUET = 'parquet'

CSV = 'csv'

JSONL = 'jsonl'

class fsspec_data.DecodedBatches

schema: pyarrow.Schema

batches: tuple[pyarrow.RecordBatch, Ellipsis]

class fsspec_data.DecodedBatchStream(schema: pyarrow.Schema, native: Any)

schema

_native

__iter__() → DecodedBatchStream

__next__() → pyarrow.RecordBatch

cancel() → None

collect() → DecodedBatches

class fsspec_data.FieldMapping

source_index: int

target_index: int

cast: str | None = None

check_nulls: bool = False

class fsspec_data.InterchangePlan

provided_schema: pyarrow.Schema

requested_schema: pyarrow.Schema

policy: SchemaPolicy

mappings: tuple[FieldMapping, Ellipsis]

provided_format: DataFormat

requested_format: DataFormat

apply_table(table: pyarrow.Table) → pyarrow.Table

apply_batch(batch: pyarrow.RecordBatch) → pyarrow.RecordBatch

_apply_mapping(array: pyarrow.Array | pyarrow.ChunkedArray, mapping: FieldMapping)

_validate_input_schema(schema: pyarrow.Schema) → None

iter_batches(data: bytes | bytearray | memoryview | BinaryIO, *, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None) → PlannedBatchStream

convert(data: bytes | bytearray | memoryview | BinaryIO, *, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None) → bytes

class fsspec_data.InterchangeRequest

provided_format: DataFormat

requested_format: DataFormat

provided_schema: pyarrow.Schema

requested_schema: pyarrow.Schema

policy: SchemaPolicy

plan() → InterchangePlan

class fsspec_data.PlannedBatchStream(plan: InterchangePlan, source: DecodedBatchStream)

schema

_plan

_source

__iter__() → PlannedBatchStream

__next__() → pyarrow.RecordBatch

cancel() → None

class fsspec_data.SchemaPolicy

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

Initialize self. See help(type(self)) for accurate signature.

EXACT = 'exact'

PROJECTION = 'projection'

COMPATIBLE = 'compatible'

COERCE = 'coerce'

fsspec_data.plan_schema(provided_schema: pyarrow.Schema, requested_schema: pyarrow.Schema, policy: SchemaPolicy | str) → InterchangePlan

fsspec_data.json_schema_to_arrow(value: collections.abc.Mapping[str, Any]) → pyarrow.Schema

fsspec_data.DEFAULT_SCHEMA_RESOLVERS

class fsspec_data.ResolvedSchema

schema: pyarrow.Schema

provenance: SchemaProvenance

class fsspec_data.SchemaDocument

data: bytes

format: SchemaFormat

provenance: SchemaProvenance

class fsspec_data.SchemaFormat

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

Initialize self. See help(type(self)) for accurate signature.

ARROW_IPC = 'arrow-ipc-schema'

ARROW_JSON = 'arrow-json'

JSON_SCHEMA = 'json-schema'

class fsspec_data.SchemaProvenance

source: str

format: str

provider: str

identifier: str | None = None

version: str | None = None

class fsspec_data.SchemaRef

url: str

format: SchemaFormat | str | None = None

storage_options: collections.abc.Mapping[str, Any]

__post_init__() → None

class fsspec_data.SchemaResolverRegistry

_decoders: dict[SchemaFormat, SchemaDecoder]

_providers: dict[str, SchemaProvider]

register(format: SchemaFormat | str, decoder: SchemaDecoder) → None

register_provider(protocol: str, provider: SchemaProvider) → None

resolve(reference: SchemaRef) → ResolvedSchema

fsspec_data.resolve_schema(value: pyarrow.Schema | SchemaRef | None, registry: SchemaResolverRegistry | None = None) → ResolvedSchema | None

fsspec_data.__version__ = '0.2.3'