fsspec_data.filesystem

Attributes

_SUFFIX_FORMATS

Classes

DataFileSystem

Read-only format and schema conversion layered over another filesystem.

Functions

_format_from_path(→ fsspec_data.interchange.DataFormat)

Module Contents

class fsspec_data.filesystem.DataFileSystem(fo: str, target_protocol: str | None = None, target_options: dict[str, Any] | None = None, fs: fsspec.AbstractFileSystem | None = None, provided_format: fsspec_data.interchange.DataFormat | str | None = None, requested_format: fsspec_data.interchange.DataFormat | str | None = None, provided_schema: fsspec_data.schema.SchemaInput = None, requested_schema: fsspec_data.schema.SchemaInput = None, schema_policy: fsspec_data.interchange.SchemaPolicy | str = SchemaPolicy.EXACT, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None, spool_max_size: int = 8 * 1024 * 1024, **kwargs: Any)[source]

Bases: fsspec.implementations.chained.ChainedFileSystem

Read-only format and schema conversion layered over another filesystem.

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters:

  • use_listings_cache – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

  • listings_expiry_time – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

  • max_paths – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

  • skip_instance_cache (bool) – If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

  • asynchronous (bool)

  • loop (asyncio-compatible IOLoop or None)

protocol = 'fsspec-data'
fs = None
fo
provided_format
requested_format
provided_schema = None
requested_schema = None
provided_schema_provenance: fsspec_data.schema.SchemaProvenance | None = None
requested_schema_provenance: fsspec_data.schema.SchemaProvenance | None = None
schema_policy
batch_size = 1024
row_limit = None
byte_limit = None
spool_max_size = 8388608
_sizes: dict[str, int]
_open(path: str, mode: str = 'rb', block_size: int | None = None, autocommit: bool = True, cache_options: dict[str, Any] | None = None, **kwargs: Any)[source]

Return raw bytes-mode file-like from the file-system

info(path: str, **kwargs: Any) → dict[str, Any][source]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

  • dict with keys (name (full path in the FS), size (in bytes), type (file,)

  • directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any)[source]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)

  • size of the entry, in bytes. If the value cannot be determined, will be None.

  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

  • path (str)

  • detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

  • kwargs (may have additional backend-specific options, such as version) – information

Returns:

  • List of strings if detail is False, or list of directory information

  • dicts if detail is True.

_convert(path: str, output) → None[source]
fsspec_data.filesystem._SUFFIX_FORMATS
fsspec_data.filesystem._format_from_path(path: str) → fsspec_data.interchange.DataFormat[source]