Storage

Data storage configuration that applies across any provider.

Attributes

• retries: int = 3

  • The number of retries to attempt.

• backoff: datetime.timedelta = datetime.timedelta(seconds=5)

  • The backoff time between retries.

• enable_debug: bool = False

  • Whether to enable debug logging.

• attach_execution_metadata: bool = True

  • Whether to attach execution metadata.

Constructors

• Constructs a Storage object. Initializes attributes with default values or values provided during instantiation.

Methods

def get_fsspec_kwargs(anonymous: bool = false, kwargs: dict) - > dict

• Returns the configuration as kwargs for constructing an fsspec filesystem.

• Parameters

  • anonymous: bool
    • Whether to use anonymous access.
  • kwargs: dict
    • Additional keyword arguments.

• Return Value: dict

  • The configuration as kwargs for constructing an fsspec filesystem.

def auto()

• Construct the config object automatically from environment variables.

• Return Value: Storage

  • A Storage object configured from environment variables.