fsspec_data.backends.base ========================= .. py:module:: fsspec_data.backends.base Classes ------- .. autoapisummary:: fsspec_data.backends.base.BaseFileSystem fsspec_data.backends.base.BaseBufferedFile Module Contents --------------- .. py:class:: BaseFileSystem(*args, **storage_options) Bases: :py:obj:`fsspec.AbstractFileSystem` Docs: https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend Minimal: https://github.com/fsspec/filesystem_spec/blob/45dcfa99e2dc320bf072c28bef767b71f8299a4a/fsspec/implementations/github.py#L10 Create and configure file-system instance Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish. A reasonable default should be provided if there are no arguments. Subclasses should call this method. :param use_listings_cache: passed to ``DirCache``, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching. :param listings_expiry_time: passed to ``DirCache``, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching. :param max_paths: passed to ``DirCache``, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching. :param skip_instance_cache: If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance. :type skip_instance_cache: bool :param asynchronous: :type asynchronous: bool :param loop: :type loop: asyncio-compatible IOLoop or None .. py:method:: mkdir(path, create_parents=True, **kwargs) Create directory entry at path For systems that don't have true directories, may create an for this instance only and not touch the real filesystem :param path: location :type path: str :param create_parents: if True, this is equivalent to ``makedirs`` :type create_parents: bool :param kwargs: may be permissions, etc. .. py:method:: makedirs(path, exist_ok=False) Recursively make directories Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file. :param path: leaf directory name :type path: str :param exist_ok: If False, will error if the target already exists :type exist_ok: bool (False) .. py:method:: rmdir(path) Remove a directory, if empty .. py:method:: ls(path, detail=True, **kwargs) :abstractmethod: List objects at path. This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested. The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include: - full path to the entry (without protocol) - size of the entry, in bytes. If the value cannot be determined, will be ``None``. - type of entry, "file", "directory" or other Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc. May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive. :param path: :type path: str :param detail: if True, gives a list of dictionaries, where each is the same as the result of ``info(path)``. If False, gives a list of paths (str). :type detail: bool :param kwargs: information :type kwargs: may have additional backend-specific options, such as version :returns: * *List of strings if detail is False, or list of directory information* * *dicts if detail is True.* .. py:method:: cp_file(path1, path2, **kwargs) :abstractmethod: .. py:method:: _rm(path) :abstractmethod: Delete one file .. py:method:: created(path) :abstractmethod: Return the created timestamp of a file as a datetime.datetime .. py:method:: modified(path) :abstractmethod: Return the modified timestamp of a file as a datetime.datetime .. py:method:: sign(path, expiration=100, **kwargs) :abstractmethod: Create a signed URL representing the given path Some implementations allow temporary URLs to be generated, as a way of delegating credentials. :param path: The path on the filesystem :type path: str :param expiration: Number of seconds to enable the URL for (if supported) :type expiration: int :returns: **URL** -- The signed URL :rtype: str :raises NotImplementedError : if method is not implemented for a filesystem: .. py:class:: BaseBufferedFile(fs, path, mode='rb', block_size='default', autocommit=True, cache_type='readahead', cache_options=None, size=None, **kwargs) Bases: :py:obj:`fsspec.spec.AbstractBufferedFile` Convenient class to derive from to provide buffering In the case that the backend does not provide a pythonic file-like object already, this class contains much of the logic to build one. The only methods that need to be overridden are ``_upload_chunk``, ``_initiate_upload`` and ``_fetch_range``. Template for files with buffered reading and writing :param fs: :type fs: instance of FileSystem :param path: location in file-system :type path: str :param mode: Normal file modes. Currently only 'wb', 'ab' or 'rb'. Some file systems may be read-only, and some may not support append. :type mode: str :param block_size: Buffer size for reading or writing, 'default' for class default :type block_size: int :param autocommit: Whether to write to final destination; may only impact what happens when file is being closed. :type autocommit: bool :param cache_type: Caching policy in read mode. See the definitions in ``core``. :type cache_type: {"readahead", "none", "mmap", "bytes"}, default "readahead" :param cache_options: Additional options passed to the constructor for the cache specified by `cache_type`. :type cache_options: dict :param size: If given and in read mode, suppressed having to look up the file size :type size: int :param kwargs: Gets stored as self.kwargs .. py:method:: commit() Move from temp to final destination .. py:method:: discard() Throw away temporary file .. py:method:: _initiate_upload() Create remote file/upload .. py:method:: _fetch_range(start, end) :abstractmethod: Get the specified set of bytes from remote