argopy.stores.s3store.open_mfjson

argopy.stores.s3store.open_mfjson#

s3store.open_mfjson(urls, max_workers: int = 6, method: str = 'thread', progress: bool | str = False, preprocess=None, url_follow=False, errors: str = 'ignore', *args, **kwargs)#

Open multiple json urls

This is a parallelized version of open_json. Use a Threads Pool by default for parallelization.

Parameters:

  • urls (list(str))

  • max_workers (int) – Maximum number of threads or processes.

  • method –

    The parallelization method to execute calls asynchronously:

    • ’thread’ (Default): use a pool of at most max_workers threads

    • ’process’: use a pool of at most max_workers processes

    • (XFAIL) Dask client object: use a Dask distributed client object

    Use ‘seq’ to simply open data sequentially

  • progress (bool) – Display a progress bar (True by default, not for dask client method)

  • preprocess ((callable, optional)) – If provided, call this function on each json set

  • url_follow (bool, False) – Follow the URL to the preprocess method as url argument.

Return type:

list()

Contents