fsspec_data.filesystem
======================

.. py:module:: fsspec_data.filesystem


Attributes
----------

.. autoapisummary::

   fsspec_data.filesystem._SUFFIX_FORMATS


Classes
-------

.. autoapisummary::

   fsspec_data.filesystem.DataFileSystem


Functions
---------

.. autoapisummary::

   fsspec_data.filesystem._format_from_path


Module Contents
---------------

.. py:class:: DataFileSystem(fo: str, target_protocol: str | None = None, target_options: dict[str, Any] | None = None, fs: fsspec.AbstractFileSystem | None = None, provided_format: fsspec_data.interchange.DataFormat | str | None = None, requested_format: fsspec_data.interchange.DataFormat | str | None = None, provided_schema: fsspec_data.schema.SchemaInput = None, requested_schema: fsspec_data.schema.SchemaInput = None, schema_policy: fsspec_data.interchange.SchemaPolicy | str = SchemaPolicy.EXACT, batch_size: int = 1024, row_limit: int | None = None, byte_limit: int | None = None, spool_max_size: int = 8 * 1024 * 1024, **kwargs: Any)

   Bases: :py:obj:`fsspec.implementations.chained.ChainedFileSystem`


   Read-only format and schema conversion layered over another filesystem.

   Create and configure file-system instance

   Instances may be cachable, so if similar enough arguments are seen
   a new instance is not required. The token attribute exists to allow
   implementations to cache instances if they wish.

   A reasonable default should be provided if there are no arguments.

   Subclasses should call this method.

   :param use_listings_cache: passed to ``DirCache``, if the implementation supports
                              directory listing caching. Pass use_listings_cache=False
                              to disable such caching.
   :param listings_expiry_time: passed to ``DirCache``, if the implementation supports
                                directory listing caching. Pass use_listings_cache=False
                                to disable such caching.
   :param max_paths: passed to ``DirCache``, if the implementation supports
                     directory listing caching. Pass use_listings_cache=False
                     to disable such caching.
   :param skip_instance_cache: If this is a cachable implementation, pass True here to force
                               creating a new instance even if a matching instance exists, and prevent
                               storing this instance.
   :type skip_instance_cache: bool
   :param asynchronous:
   :type asynchronous: bool
   :param loop:
   :type loop: asyncio-compatible IOLoop or None


   .. py:attribute:: protocol
      :value: 'fsspec-data'



   .. py:attribute:: fs
      :value: None



   .. py:attribute:: fo


   .. py:attribute:: provided_format


   .. py:attribute:: requested_format


   .. py:attribute:: provided_schema
      :value: None



   .. py:attribute:: requested_schema
      :value: None



   .. py:attribute:: provided_schema_provenance
      :type:  fsspec_data.schema.SchemaProvenance | None
      :value: None



   .. py:attribute:: requested_schema_provenance
      :type:  fsspec_data.schema.SchemaProvenance | None
      :value: None



   .. py:attribute:: schema_policy


   .. py:attribute:: batch_size
      :value: 1024



   .. py:attribute:: row_limit
      :value: None



   .. py:attribute:: byte_limit
      :value: None



   .. py:attribute:: spool_max_size
      :value: 8388608



   .. py:attribute:: _sizes
      :type:  dict[str, int]


   .. py:method:: _open(path: str, mode: str = 'rb', block_size: int | None = None, autocommit: bool = True, cache_options: dict[str, Any] | None = None, **kwargs: Any)

      Return raw bytes-mode file-like from the file-system



   .. py:method:: info(path: str, **kwargs: Any) -> dict[str, Any]

      Give details of entry at path

      Returns a single dictionary, with exactly the same information as ``ls``
      would with ``detail=True``.

      The default implementation calls ls and could be overridden by a
      shortcut. kwargs are passed on to ```ls()``.

      Some file systems might not be able to measure the file's size, in
      which case, the returned dict will include ``'size': None``.

      :returns: * **dict with keys** (*name (full path in the FS), size (in bytes), type (file,*)
                * *directory, or something else) and other FS-specific keys.*



   .. py:method:: ls(path: str, detail: bool = True, **kwargs: Any)

      List objects at path.

      This should include subdirectories and files at that location. The
      difference between a file and a directory must be clear when details
      are requested.

      The specific keys, or perhaps a FileInfo class, or similar, is TBD,
      but must be consistent across implementations.
      Must include:

      - full path to the entry (without protocol)
      - size of the entry, in bytes. If the value cannot be determined, will
        be ``None``.
      - type of entry, "file", "directory" or other

      Additional information
      may be present, appropriate to the file-system, e.g., generation,
      checksum, etc.

      May use refresh=True|False to allow use of self._ls_from_cache to
      check for a saved listing and avoid calling the backend. This would be
      common where listing may be expensive.

      :param path:
      :type path: str
      :param detail: if True, gives a list of dictionaries, where each is the same as
                     the result of ``info(path)``. If False, gives a list of paths
                     (str).
      :type detail: bool
      :param kwargs: information
      :type kwargs: may have additional backend-specific options, such as version

      :returns: * *List of strings if detail is False, or list of directory information*
                * *dicts if detail is True.*



   .. py:method:: _convert(path: str, output) -> None


.. py:data:: _SUFFIX_FORMATS

.. py:function:: _format_from_path(path: str) -> fsspec_data.interchange.DataFormat