fsspec_data.backends.base

Classes

BaseFileSystem	Docs: https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend
BaseBufferedFile	Convenient class to derive from to provide buffering

Module Contents

class fsspec_data.backends.base.BaseFileSystem(*args, **storage_options)

Bases: fsspec.AbstractFileSystem

Docs: https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend Minimal: https://github.com/fsspec/filesystem_spec/blob/45dcfa99e2dc320bf072c28bef767b71f8299a4a/fsspec/implementations/github.py#L10

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

Parameters:

• use_listings_cache – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• listings_expiry_time – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• max_paths – passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.

• skip_instance_cache (bool) – If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.

• asynchronous (bool)

• loop (asyncio-compatible IOLoop or None)

mkdir(path, create_parents=True, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters:

• path (str) – location

• create_parents (bool) – if True, this is equivalent to makedirs

• kwargs – may be permissions, etc.

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters:

• path (str) – leaf directory name

• exist_ok (bool (False)) – If False, will error if the target already exists

rmdir(path)

Remove a directory, if empty

abstractmethod ls(path, detail=True, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

abstractmethod cp_file(path1, path2, **kwargs)

abstractmethod _rm(path)

Delete one file

abstractmethod created(path)

Return the created timestamp of a file as a datetime.datetime

abstractmethod modified(path)

Return the modified timestamp of a file as a datetime.datetime

abstractmethod sign(path, expiration=100, **kwargs)

Create a signed URL representing the given path

Some implementations allow temporary URLs to be generated, as a way of delegating credentials.

Parameters:

• path (str) – The path on the filesystem

• expiration (int) – Number of seconds to enable the URL for (if supported)

Returns:

URL – The signed URL

Return type:

str

:raises NotImplementedError : if method is not implemented for a filesystem:

class fsspec_data.backends.base.BaseBufferedFile(fs, path, mode='rb', block_size='default', autocommit=True, cache_type='readahead', cache_options=None, size=None, **kwargs)

Bases: fsspec.spec.AbstractBufferedFile

Convenient class to derive from to provide buffering

In the case that the backend does not provide a pythonic file-like object already, this class contains much of the logic to build one. The only methods that need to be overridden are _upload_chunk, _initiate_upload and _fetch_range.

Template for files with buffered reading and writing

Parameters:

• fs (instance of FileSystem)

• path (str) – location in file-system

• mode (str) – Normal file modes. Currently only ‘wb’, ‘ab’ or ‘rb’. Some file systems may be read-only, and some may not support append.

• block_size (int) – Buffer size for reading or writing, ‘default’ for class default

• autocommit (bool) – Whether to write to final destination; may only impact what happens when file is being closed.

• cache_type ({"readahead", "none", "mmap", "bytes"}, default "readahead") – Caching policy in read mode. See the definitions in core.

• cache_options (dict) – Additional options passed to the constructor for the cache specified by cache_type.

• size (int) – If given and in read mode, suppressed having to look up the file size

• kwargs – Gets stored as self.kwargs

commit()

Move from temp to final destination

discard()

Throw away temporary file

_initiate_upload()

Create remote file/upload

abstractmethod _fetch_range(start, end)

Get the specified set of bytes from remote