fsspec-rs filesystems¶

fsspec-rs exposes Rust-backed filesystems through normal fsspec classes. Importing fsspec_rs registers the protocols with fsspec:

Filesystem	Protocols	Class
Local filesystem	`file-rs`, `local-rs`	`fsspec_rs.LocalFileSystem`
Amazon S3	`s3-rs`	`fsspec_rs.S3FileSystem`

Use these protocols when you want the Rust-backed implementation without changing the standard file, local, or s3 protocol names owned by fsspec and s3fs.

import fsspec
import fsspec_rs

with fsspec.open("file-rs:///tmp/example.txt", "rb") as f:
    data = f.read()

Local filesystem¶

LocalFileSystem wraps the Rust LocalFs backend, which uses std::fs for local paths. It is marked as a local filesystem (local_file = True) and supports the common fsspec primitives.

from fsspec_rs import LocalFileSystem

fs = LocalFileSystem(auto_mkdir=True)
fs.pipe_file("/tmp/fsspec-rs/example.txt", b"hello")

assert fs.exists("/tmp/fsspec-rs/example.txt")
assert fs.isfile("/tmp/fsspec-rs/example.txt")
print(fs.info("/tmp/fsspec-rs/example.txt"))

Paths¶

Direct method calls use normal local paths:

fs.cat_file("/tmp/fsspec-rs/example.txt")

fsspec URLs use file-rs:// or local-rs://:

with fsspec.open("local-rs:///tmp/fsspec-rs/example.txt", "rb") as f:
    print(f.read())

LocalFileSystem._strip_protocol() removes only the Rust-specific local protocols. The regular file:// and local:// protocols remain owned by fsspec’s Python implementation.

Opening files¶

Supported modes are rb, wb, ab, and xb, plus their textless aliases r, w, a, and x at the Rust layer. Python wrappers expose binary file objects that implement read, write, seek, tell, flush, close, readable, writable, and seekable.

with fs.open("/tmp/fsspec-rs/out.bin", "wb") as f:
    f.write(b"payload")

with fs.open("/tmp/fsspec-rs/out.bin", "rb") as f:
    f.seek(2)
    print(f.read())

auto_mkdir=True creates missing parent directories for write, append, exclusive-create, and copy operations. With the default auto_mkdir=False, missing parents are reported by the operating system.

Supported operations¶

Local filesystem methods delegated to Rust include:

Listing and metadata: ls, info, exists, isdir, isfile, size
Reads and writes: open, cat_file, pipe_file, head, tail, read_text, write_text, touch
Directory operations: mkdir, makedirs, rmdir, walk, find, du
Mutation and transfer: rm_file, rm, cp_file, copy, mv, get_file, put_file

The cache_type, block_size, and max_blocks options are accepted by open() for API compatibility. Local reads are already direct local file operations, so those cache options do not change local read behavior.

S3¶

S3FileSystem wraps the Rust S3Fs backend, which uses Apache Arrow’s object_store crate and runs its async S3 calls through an embedded Tokio runtime. The Python protocol is s3-rs, so it can coexist with s3fs.

from fsspec_rs import S3FileSystem

fs = S3FileSystem(bucket="my-bucket", region="us-east-1")
fs.pipe_file("reports/hello.txt", b"hello")
print(fs.cat_file("reports/hello.txt"))

Paths¶

S3FileSystem is constructed for one bucket. Method calls can use keys within that bucket:

fs.ls("reports")
fs.info("reports/hello.txt")

fsspec URLs include the bucket and can infer constructor options through _get_kwargs_from_urls():

with fsspec.open(
    "s3-rs://my-bucket/reports/hello.txt",
    "rb",
    region="us-east-1",
) as f:
    print(f.read())

Internally, object names are normalized without a leading slash. Directory entries are S3 prefixes, not real directories; mkdir() and rmdir() are no-ops.

Credentials and endpoints¶

Credential resolution follows this order:

Explicit constructor arguments: key, secret, token, region, endpoint_url, and anon
fsspec.config.conf["s3"], including values populated from fsspec config environment variables
object_store’s own AWS credential chain, such as environment variables, profiles, and instance or task roles

Use client_kwargs={"endpoint_url": ...} when porting code from s3fs. If the endpoint starts with http://, the Rust backend allows plain HTTP automatically for local S3-compatible services.

fs = S3FileSystem(
    bucket="benchmark",
    key="minioadmin",
    secret="minioadmin",
    endpoint_url="http://localhost:9000",
    region="us-east-1",
)

Anonymous access is available with anon=True.

Caching reads¶

S3 reads support optional Rust-side cache strategies through cache_type:

`cache_type`	Behavior
`none`	Fetch each requested byte range directly.
`readahead`	Keep one lookahead window for sequential reads.
`block`	Cache fixed-size blocks with LRU eviction.
`all`	Fetch the whole object on first read.

with fs.open(
    "large/table.parquet",
    "rb",
    cache_type="block",
    block_size=8 * 1024 * 1024,
) as f:
    header = f.read(4096)

block_size defaults to 4 MiB. max_blocks defaults to 32 and only affects cache_type="block".

Without an explicit cache_type, S3 read mode eagerly loads the whole object into memory. Use readahead, block, or none when reading selected ranges from large objects.

Supported operations¶

S3 methods delegated to Rust include:

Listing and metadata: ls, info, exists, isdir, isfile, size
Reads and writes: open, cat_file, pipe_file, head, tail, read_text
Prefix traversal: walk, find
Object mutation: rm_file, rm, cp_file

Current S3 writes buffer object contents in memory before upload. Multipart concurrent uploads and signed URLs are not yet part of the public API.