argopy.stores.httpstore.download_url

argopy.stores.httpstore.download_url#

httpstore.download_url(url, max_attempt: int = 5, cat_opts: dict = {}, errors: str = 'raise') → Any[source]#

Resilient URL data downloader

This is basically a fsspec.implementations.http.HTTPFileSystem.cat_file() that is able to handle a 429 “Too many requests” error from a server, by waiting and sending requests several time.

Parameters:

  • url (str) – URL to download

  • max_attempt (int, default = 5) – Maximum number of attempts to perform before failing

  • cat_opts (dict, default = {}) – Options to be passed to the HTTPFileSystem cat_file method

  • errors (str, default: raise) –

    Define how to handle errors:

    • raise (default): Raise any error encountered

    • ignore: Do not stop processing, simply issue a debug message in logging console

    • silent: Do not stop processing and do not issue log message

Contents