# Rust traits The Rust crate is named `fsspec_rs`. It can be used independently from the Python package and exposes the same template-method shape as fsspec: implement a small set of backend primitives, then get higher-level filesystem methods from default trait implementations. ```toml [dependencies] fsspec_rs = "0.1.1" ``` For local development in this repository, the Python extension depends on the crate by path: ```toml fsspec_rs = { path = "./rust", version = "*" } ``` ## Core model Most public Rust types are re-exported from the crate root: ```rust use fsspec_rs::{ CacheType, FileInfo, FileSystem, FsFile, FsResult, LocalFs, OpenMode, OpenOptions, S3Config, S3Fs, }; ``` The main pieces are: | Type | Purpose | | ---------------------------- | -------------------------------------------------------------------------------- | | `FileSystem` | Synchronous filesystem trait with primitive methods and default helpers. | | `AsyncFileSystem` | Experimental async mirror of `FileSystem`; no shipped backend implements it yet. | | `FsFile` | File-like trait returned by `FileSystem::open()`. | | `FileInfo` | Metadata for files, directories, and other entries. | | `OpenMode` and `OpenOptions` | Open mode and buffered I/O options. | | `FsError` and `FsResult` | Shared error and result types. | | `Cache` and `CacheType` | Read cache strategy interface and selector. | ## FileSystem `FileSystem` is the sync trait implemented by `LocalFs` and `S3Fs`. Backends must provide seven core primitives plus protocol metadata: ```rust use fsspec_rs::{FileInfo, FileSystem, FsFile, FsResult, OpenMode, OpenOptions}; struct MyFs; impl FileSystem for MyFs { fn protocol(&self) -> &[&str] { &["myfs"] } fn ls(&self, path: &str, detail: bool) -> FsResult> { todo!() } fn rm_file(&self, path: &str) -> FsResult<()> { todo!() } fn cp_file(&self, src: &str, dst: &str) -> FsResult<()> { todo!() } fn open( &self, path: &str, mode: OpenMode, opts: Option, ) -> FsResult> { todo!() } fn info(&self, path: &str) -> FsResult { todo!() } fn mkdir(&self, path: &str, create_parents: bool) -> FsResult<()> { todo!() } fn rmdir(&self, path: &str) -> FsResult<()> { todo!() } } ``` `root_marker()` defaults to `""` and `sep()` defaults to `"/"`. Override them for local or platform-specific behavior. `strip_protocol()` and `unstrip_protocol()` also have defaults and can be overridden when a backend needs bucket-aware or authority-aware paths. The default methods built on those primitives include: - Path helpers: `strip_protocol`, `unstrip_protocol`, `parent` - Metadata checks: `exists`, `isdir`, `isfile`, `size`, `sizes` - File I/O helpers: `cat_file`, `pipe_file`, `head`, `tail`, `touch`, `read_text`, `write_text` - Tree operations: `walk`, `find`, `copy`, `mv`, `rm`, `du`, `makedirs` - Local transfers: `get_file`, `put_file` ## AsyncFileSystem `AsyncFileSystem` mirrors `FileSystem`, but primitive methods return futures and default helpers are `async fn`s. It is experimental: the shipped `LocalFs` and `S3Fs` backends currently implement the synchronous `FileSystem` trait. The default async implementation avoids recursive async futures in `walk()` by using an explicit stack. Method semantics otherwise match `FileSystem`. ## FsFile Opened files implement `FsFile`: ```rust use std::io::{Read, Seek, Write}; use fsspec_rs::{FileInfo, FsFile, FsResult}; struct MyFile; impl Read for MyFile { /* ... */ } impl Write for MyFile { /* ... */ } impl Seek for MyFile { /* ... */ } impl FsFile for MyFile { fn info(&self) -> FsResult { todo!() } fn size(&self) -> FsResult> { todo!() } } ``` `FsFile` requires `Read + Write + Seek + Send`. `commit()` and `discard()` default to no-ops and are available for buffered writes or future transaction support. ## Metadata and open options `FileInfo` is the common metadata value. It stores: - `name`: full path or backend path - `size`: bytes, usually `0` for directories - `file_type`: `FileType::File`, `FileType::Directory`, or `FileType::Other` - `created` and `modified`: optional `SystemTime` values - `extra`: backend-specific string metadata such as S3 ETags Convenience constructors are available: ```rust let file = FileInfo::file("/tmp/data.bin", 1024); let dir = FileInfo::directory("/tmp"); ``` `OpenMode::from_str_mode()` accepts `rb`, `wb`, `ab`, `xb`, and the equivalent single-letter forms. `OpenOptions::default()` uses a 4 MiB block size, `autocommit = true`, no explicit cache strategy, and `max_blocks = 32`. ## LocalFs `LocalFs` uses `std::fs` and implements `FileSystem`. ```rust use fsspec_rs::{FileSystem, LocalFs}; fn main() -> fsspec_rs::FsResult<()> { let fs = LocalFs::with_auto_mkdir(true); fs.pipe_file("/tmp/fsspec-rs.txt", b"hello")?; let _data = fs.cat_file("/tmp/fsspec-rs.txt", None, None)?; Ok(()) } ``` `LocalFs::new()` starts with `auto_mkdir = false`. Use `LocalFs::with_auto_mkdir(true)` to create missing parent directories during write, append, exclusive-create, and copy operations. ## S3Fs `S3Fs` wraps `object_store::aws::AmazonS3` and presents a sync `FileSystem` API through an embedded Tokio runtime. ```rust use fsspec_rs::{FileSystem, S3Config, S3Fs}; fn main() -> fsspec_rs::FsResult<()> { let mut cfg = S3Config::new("my-bucket"); cfg.region = Some("us-east-1".to_string()); let fs = S3Fs::new(cfg)?; let _objects = fs.ls("", true)?; Ok(()) } ``` `S3Config` supports: - `bucket` - `region` - `endpoint_url` - `access_key_id` - `secret_access_key` - `session_token` - `anon` - `virtual_hosted_style_request` Plain HTTP is allowed automatically when `endpoint_url` starts with `http://`, which is useful for MinIO and LocalStack. ## Python fsspec bridge The optional bridge crate is named `fsspec_rs_bridge`. It adapts Python fsspec filesystem objects to the Rust `FileSystem` trait: ```toml [dependencies] fsspec_rs = "0.1.1" fsspec_rs_bridge = "0.1.2" ``` For local development in this repository: ```toml fsspec_rs = { path = "./rust", version = "*" } fsspec_rs_bridge = { path = "./rust/bridge", version = "*" } ``` The bridge is intentionally separate from the pure Rust core crate. It depends on PyO3 and Python fsspec at runtime, but it is an `rlib` helper crate rather than a Python extension module. It defines no `#[pyclass]` types, which keeps downstream PyO3 extension modules from sharing Python class statistics across crate boundaries. Use the bridge when Rust code is running inside a Python process and needs to consume an installed fsspec implementation that does not yet have a native Rust backend. ```rust use fsspec_rs::FileSystem; use fsspec_rs_bridge::{url_to_fs, StorageOptions}; fn main() -> fsspec_rs::FsResult<()> { let storage_options = StorageOptions::new(); let (fs, start_path) = url_to_fs("memory://", &storage_options)?; fs.pipe_file("/example.txt", b"hello from Python fsspec")?; let data = fs.cat_file("/example.txt", None, None)?; assert_eq!(start_path, "/"); assert_eq!(data, b"hello from Python fsspec"); Ok(()) } ``` You can also construct a bridge from a protocol name: ```rust use fsspec_rs::FileSystem; use fsspec_rs_bridge::{PyFsspecFs, StorageOptions}; fn main() -> fsspec_rs::FsResult<()> { let fs = PyFsspecFs::from_protocol("memory", &StorageOptions::new())?; assert_eq!(fs.protocol(), &["python"]); assert_eq!(fs.source_protocol(), "memory"); assert_eq!(fs.root_marker(), "/"); Ok(()) } ``` `PyFsspecFs` implements the same `FileSystem` trait as `LocalFs` and `S3Fs`. It delegates primitives such as `ls`, `info`, `open`, `cat_file`, `pipe_file`, `cp_file`, `rm_file`, `mkdir`, `rmdir`, `get_file`, and `put_file` to the underlying Python filesystem object. Python exceptions are mapped back into `FsError` variants where possible. `StorageOptions` values preserve Python-compatible null, boolean, signed and unsigned integer, floating-point, string, list, and nested-map types. Nested maps support per-protocol options for chained URLs: ```rust use fsspec_rs_bridge::{url_to_fs, StorageOptions}; let mut memory = StorageOptions::new(); memory.insert("skip_instance_cache".to_string(), true.into()); let mut options = StorageOptions::new(); options.insert("memory".to_string(), memory.into()); let (fs, path) = url_to_fs("simplecache::memory://bucket/data.csv", &options)?; # Ok::<(), fsspec_rs::FsError>(()) ``` Because the bridge calls Python, it initializes and attaches to the Python interpreter internally. Consumers should still treat it as Python-backed I/O: it is best suited for Python extension crates or embedded-Python contexts, not standalone Rust binaries that need a Python-free dependency graph. ## Caching The caching layer is used by buffered reads. `CacheType::from_str()` accepts: | Value | Strategy | | --------------------------- | -------------------------------------------------- | | `none` | `NoCache`: every fetch goes to the backend. | | `readahead` or `read_ahead` | `ReadAheadCache`: one lookahead window. | | `block` or `blockcache` | `BlockCache`: fixed-size blocks with LRU eviction. | | `all` or `bytes` | `AllBytesCache`: whole-file cache on first read. | Each cache wraps a `Fetcher`, a `FnMut(u64, u64) -> FsResult>` that retrieves the half-open byte range `[start, end)` from the backend. ## Error handling All trait methods return `FsResult = Result`. `FsError` maps common filesystem semantics into stable variants: - `NotFound` - `PermissionDenied` - `AlreadyExists` - `NotADirectory` - `IsADirectory` - `IoError` - `InvalidArgument` - `NotSupported` - `Other` The PyO3 layer maps those variants to Python exceptions such as `FileNotFoundError`, `PermissionError`, `FileExistsError`, `NotADirectoryError`, `IsADirectoryError`, `ValueError`, `NotImplementedError`, and `OSError`.