Python API reference

Registered backend classes inherit from fsspec.AbstractFileSystem. AbstractDatabase defines the custom Python backend contract.

Package Exports

class fsspec_db.AbstractDatabase

Bases: ABC

Database primitive contract for Python-defined backends.

The method set mirrors the Rust Database trait, except for Rust-only default methods. Metadata methods return the exported *Info classes. query returns a pyarrow.Table. insert receives a pyarrow.Table and returns the number of rows inserted.

abstractmethod dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

abstractmethod insert(schema: str, relation: str, table: Any, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

abstractmethod list_columns(schema: str, relation: str) → list[ColumnInfo]

Return columns for schema.relation.

abstractmethod list_constraints(schema: str, relation: str) → list[ConstraintInfo]

Return constraints for schema.relation.

abstractmethod list_indexes(schema: str, relation: str) → list[IndexInfo]

Return indexes for schema.relation.

abstractmethod list_relations(schema: str) → list[RelationInfo]

Return tables and views in schema.

abstractmethod list_schemas() → list[SchemaInfo]

Return all schemas visible to the backend.

abstractmethod query(sql: str, params: list[Any] | None = None) → Any

Return a pyarrow.Table for sql and optional scalar params.

abstractmethod relation_info(schema: str, relation: str) → RelationInfo

Return metadata for one table or view.

abstractmethod view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.AbstractDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

fsspec filesystem adapter for an AbstractDatabase implementation.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.DuckDBDatabase(database: str = ':memory:', *, connection: Any = None, read_only: bool = False)

Bases: AbstractDatabase

Arrow-native Python backend for an embedded DuckDB database.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str)

Return columns for schema.relation.

list_constraints(schema: str, relation: str)

Return constraints for schema.relation.

list_indexes(schema: str, relation: str)

Return indexes for schema.relation.

list_relations(schema: str)

Return tables and views in schema.

list_schemas()

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str)

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.DuckDBDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem

class fsspec_db.MySQLDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

MySQL-backed fsspec filesystem registered as db+mysql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.OdbcDatabase(connection_string: str, *, user: str = '', password: str = '', batch_size: int = 65535)

Bases: AbstractDatabase

Arrow-native ODBC backend using the arrow-odbc connection API.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str)

Return columns for schema.relation.

list_constraints(schema: str, relation: str)

Return constraints for schema.relation.

list_indexes(schema: str, relation: str)

Return indexes for schema.relation.

list_relations(schema: str)

Return tables and views in schema.

list_schemas()

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str)

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.OdbcDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem

class fsspec_db.PostgresDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

PostgreSQL-backed fsspec filesystem registered as db+postgresql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.PyDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

Direct Python filesystem adapter for an AbstractDatabase.

Unlike AbstractDatabaseFileSystem, this class does not call the Rust bridge. It is primarily useful for Python-only backend implementations and conformance testing.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.SQLAlchemyDatabase(url: str | None = None, *, engine: Any = None, **engine_options: Any)

Bases: AbstractDatabase

SQLAlchemy engine adapter implementing the Python database contract.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str) → list[ColumnInfo]

Return columns for schema.relation.

list_constraints(schema: str, relation: str) → list[ConstraintInfo]

Return constraints for schema.relation.

list_indexes(schema: str, relation: str) → list[IndexInfo]

Return indexes for schema.relation.

list_relations(schema: str) → list[RelationInfo]

Return tables and views in schema.

list_schemas() → list[SchemaInfo]

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str) → RelationInfo

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.SQLAlchemyDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem

class fsspec_db.SQLiteDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

SQLite-backed fsspec filesystem registered as db+sqlite.

Overwrite writes replace table contents; use append mode to preserve rows.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

Base Classes

class fsspec_db.spec.AbstractDatabase

Bases: ABC

Database primitive contract for Python-defined backends.

The method set mirrors the Rust Database trait, except for Rust-only default methods. Metadata methods return the exported *Info classes. query returns a pyarrow.Table. insert receives a pyarrow.Table and returns the number of rows inserted.

abstractmethod dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

abstractmethod insert(schema: str, relation: str, table: Any, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

abstractmethod list_columns(schema: str, relation: str) → list[ColumnInfo]

Return columns for schema.relation.

abstractmethod list_constraints(schema: str, relation: str) → list[ConstraintInfo]

Return constraints for schema.relation.

abstractmethod list_indexes(schema: str, relation: str) → list[IndexInfo]

Return indexes for schema.relation.

abstractmethod list_relations(schema: str) → list[RelationInfo]

Return tables and views in schema.

abstractmethod list_schemas() → list[SchemaInfo]

Return all schemas visible to the backend.

abstractmethod query(sql: str, params: list[Any] | None = None) → Any

Return a pyarrow.Table for sql and optional scalar params.

abstractmethod relation_info(schema: str, relation: str) → RelationInfo

Return metadata for one table or view.

abstractmethod view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.spec.AbstractDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

fsspec filesystem adapter for an AbstractDatabase implementation.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

class fsspec_db.spec.DatabaseDdlMixin

Bases: object

Guarded database DDL mapped onto fsspec mutation methods.

class fsspec_db.spec.DeferredDatabaseFile(commit: Any)

Bases: BytesIO

Write buffer committed or discarded by fsspec.Transaction.

close() → None

Disable all I/O operations.

class fsspec_db.spec.IntrospectionCacheMixin

Bases: object

Cache database listings and metadata using fsspec’s bounded TTL cache.

AbstractDatabaseFileSystem and the native SQLite, PostgreSQL, and MySQL filesystem classes expose query(sql, params=None) for pyarrow.Table results and open_query(sql, params=None) for Arrow IPC stream bytes.

Python Filesystem Adapter

class fsspec_db.python.PyDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

Direct Python filesystem adapter for an AbstractDatabase.

Unlike AbstractDatabaseFileSystem, this class does not call the Rust bridge. It is primarily useful for Python-only backend implementations and conformance testing.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

SQLite Backend

class fsspec_db.sqlite.SQLiteDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

SQLite-backed fsspec filesystem registered as db+sqlite.

Overwrite writes replace table contents; use append mode to preserve rows.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

PostgreSQL Backend

class fsspec_db.postgres.PostgresDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

PostgreSQL-backed fsspec filesystem registered as db+postgresql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

MySQL Backend

class fsspec_db.mysql.MySQLDatabaseFileSystem(*args, **kwargs)

Bases: DatabaseDdlMixin, IntrospectionCacheMixin, AbstractFileSystem

MySQL-backed fsspec filesystem registered as db+mysql.

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs: Any) → bytes

Get the content of a file

Parameters:

• path (URL of file on this filesystems)

• start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

• kwargs (passed to open().)

info(path: str, **kwargs: Any) → dict[str, Any]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns:

• dict with keys (name (full path in the FS), size (in bytes), type (file,)

• directory, or something else) and other FS-specific keys.

ls(path: str, detail: bool = True, **kwargs: Any) → list[dict[str, Any]] | list[str]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will be None.

• type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

• path (str)

• detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

• kwargs (may have additional backend-specific options, such as version) – information

Returns:

• List of strings if detail is False, or list of directory information

• dicts if detail is True.

pipe_file(path: str, value: bytes, mode: str = 'overwrite', **kwargs: Any) → None

Set the bytes of given file

put_file(lpath: str, rpath: str, callback: Any = None, mode: str = 'overwrite', **kwargs: Any) → None

Copy single file to remote

DuckDB Backend

class fsspec_db.duckdb.DuckDBDatabase(database: str = ':memory:', *, connection: Any = None, read_only: bool = False)

Bases: AbstractDatabase

Arrow-native Python backend for an embedded DuckDB database.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str)

Return columns for schema.relation.

list_constraints(schema: str, relation: str)

Return constraints for schema.relation.

list_indexes(schema: str, relation: str)

Return indexes for schema.relation.

list_relations(schema: str)

Return tables and views in schema.

list_schemas()

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str)

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.duckdb.DuckDBDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem

SQLAlchemy Backend

class fsspec_db.sqlalchemy.SQLAlchemyDatabase(url: str | None = None, *, engine: Any = None, **engine_options: Any)

Bases: AbstractDatabase

SQLAlchemy engine adapter implementing the Python database contract.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str) → list[ColumnInfo]

Return columns for schema.relation.

list_constraints(schema: str, relation: str) → list[ConstraintInfo]

Return constraints for schema.relation.

list_indexes(schema: str, relation: str) → list[IndexInfo]

Return indexes for schema.relation.

list_relations(schema: str) → list[RelationInfo]

Return tables and views in schema.

list_schemas() → list[SchemaInfo]

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str) → RelationInfo

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.sqlalchemy.SQLAlchemyDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem

ODBC Backend

class fsspec_db.odbc.OdbcDatabase(connection_string: str, *, user: str = '', password: str = '', batch_size: int = 65535)

Bases: AbstractDatabase

Arrow-native ODBC backend using the arrow-odbc connection API.

dialect() → str

Return generic, sqlite, postgres, postgresql, or mysql.

insert(schema: str, relation: str, table: Table, mode: str = 'append') → int

Insert table into schema.relation with append or truncate mode.

list_columns(schema: str, relation: str)

Return columns for schema.relation.

list_constraints(schema: str, relation: str)

Return constraints for schema.relation.

list_indexes(schema: str, relation: str)

Return indexes for schema.relation.

list_relations(schema: str)

Return tables and views in schema.

list_schemas()

Return all schemas visible to the backend.

query(sql: str, params: list[Any] | None = None) → Table

Return a pyarrow.Table for sql and optional scalar params.

relation_info(schema: str, relation: str)

Return metadata for one table or view.

view_definition(schema: str, view: str) → str

Return SQL text for one view.

class fsspec_db.odbc.OdbcDatabaseFileSystem(*args, **kwargs)

Bases: PyDatabaseFileSystem