API

The public Python API is intentionally small. Most users construct filesystems through fsspec.filesystem("db+sqlite", ...), fsspec.filesystem("db+postgresql", ...), or fsspec.filesystem("db+mysql", ...); backend authors use AbstractDatabase and AbstractDatabaseFileSystem.

Package Exports

class fsspec_db.AbstractDatabase

Bases: ABC

Minimal database contract used by AbstractDatabaseFileSystem.

class fsspec_db.AbstractDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

fsspec filesystem adapter for an AbstractDatabase implementation.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.DBFile(fs, path, mode='rb', block_size='default', autocommit=True, cache_type='readahead', cache_options=None, size=None, **kwargs)

Bases: AbstractBufferedFile

Buffered fsspec file used for database relation writes.

class fsspec_db.MySQLDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

MySQL-backed fsspec filesystem registered as db+mysql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.PostgresDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

PostgreSQL-backed fsspec filesystem registered as db+postgresql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.SQLiteDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

SQLite-backed fsspec filesystem registered as db+sqlite.

Overwrite writes replace table contents; use append mode to preserve rows.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

Base Classes

class fsspec_db.spec.AbstractDatabase

Bases: ABC

Minimal database contract used by AbstractDatabaseFileSystem.

class fsspec_db.spec.AbstractDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

fsspec filesystem adapter for an AbstractDatabase implementation.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.spec.DBFile(fs, path, mode='rb', block_size='default', autocommit=True, cache_type='readahead', cache_options=None, size=None, **kwargs)

Bases: AbstractBufferedFile

Buffered fsspec file used for database relation writes.

SQLite Backend

class fsspec_db.sqlite.SQLiteDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

SQLite-backed fsspec filesystem registered as db+sqlite.

Overwrite writes replace table contents; use append mode to preserve rows.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

PostgreSQL Backend

class fsspec_db.postgres.PostgresDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

PostgreSQL-backed fsspec filesystem registered as db+postgresql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

MySQL Backend

class fsspec_db.mysql.MySQLDatabaseFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

MySQL-backed fsspec filesystem registered as db+mysql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote