Argo meta-data#

Index of profiles#

Since the Argo measurements dataset is quite complex, it comes with a collection of index files, or lookup tables with meta data. These index help you determine what you can expect before retrieving the full set of measurements. argopy has a specific fetcher for index files:

In [1]: from argopy import IndexFetcher as ArgoIndexFetcher

You can use the Index fetcher with the region or float access points, similarly to data fetching:

In [2]: idx = ArgoIndexFetcher(src='gdac').float(2901623).load()

In [3]: idx.index
Out[3]: 
                                       file  ...                             profiler
0    nmdis/2901623/profiles/R2901623_000.nc  ...  Provor, Seabird conductivity sensor
1   nmdis/2901623/profiles/R2901623_000D.nc  ...  Provor, Seabird conductivity sensor
2    nmdis/2901623/profiles/R2901623_001.nc  ...  Provor, Seabird conductivity sensor
3    nmdis/2901623/profiles/R2901623_002.nc  ...  Provor, Seabird conductivity sensor
4    nmdis/2901623/profiles/R2901623_003.nc  ...  Provor, Seabird conductivity sensor
..                                      ...  ...                                  ...
93   nmdis/2901623/profiles/R2901623_092.nc  ...  Provor, Seabird conductivity sensor
94   nmdis/2901623/profiles/R2901623_093.nc  ...  Provor, Seabird conductivity sensor
95   nmdis/2901623/profiles/R2901623_094.nc  ...  Provor, Seabird conductivity sensor
96   nmdis/2901623/profiles/R2901623_095.nc  ...  Provor, Seabird conductivity sensor
97   nmdis/2901623/profiles/R2901623_096.nc  ...  Provor, Seabird conductivity sensor

[98 rows x 11 columns]

Alternatively, you can use argopy.IndexFetcher.to_dataframe():

In [4]: idx = ArgoIndexFetcher(src='gdac').float(2901623)

In [5]: df = idx.to_dataframe()

The difference is that with the load method, data are stored in memory and not fetched on every call to the index attribute.

The index fetcher has pretty much the same methods than the data fetchers. you can check them all here: argopy.fetchers.ArgoIndexFetcher.

Reference tables#

The Argo netcdf format is strict and based on a collection of variables fully documented and conventioned. All reference tables can be found in the Argo user manual.

However, a machine-to-machine access to these tables is often required. This is possible thanks to the work of the Argo Vocabulary Task Team (AVTT) that is a team of people responsible for the NVS collections under the Argo Data Management Team governance.

Note

The GitHub organization hosting the AVTT is the ‘NERC Vocabulary Server (NVS)’, aka ‘nvs-vocabs’. This holds a list of NVS collection-specific GitHub repositories. Each Argo GitHub repository is called after its corresponding collection ID (e.g. R01, RR2, R03 etc.). The current list is given here.

The management of issues related to vocabularies managed by the Argo Data Management Team is done on this repository.

argopy provides the utility class ArgoNVSReferenceTables to easily fetch and get access to all Argo reference tables. If you already know the name of the reference table you want to retrieve, you can simply get it like this:

In [6]: from argopy import ArgoNVSReferenceTables

In [7]: NVS = ArgoNVSReferenceTables()

In [8]: NVS.tbl('R01')
Out[8]: 
  altLabel  ...                                                 id
0    BPROF  ...  http://vocab.nerc.ac.uk/collection/R01/current...
1    BTRAJ  ...  http://vocab.nerc.ac.uk/collection/R01/current...
2     META  ...  http://vocab.nerc.ac.uk/collection/R01/current...
3    MPROF  ...  http://vocab.nerc.ac.uk/collection/R01/current...
4    MTRAJ  ...  http://vocab.nerc.ac.uk/collection/R01/current...
5     PROF  ...  http://vocab.nerc.ac.uk/collection/R01/current...
6    SPROF  ...  http://vocab.nerc.ac.uk/collection/R01/current...
7     TECH  ...  http://vocab.nerc.ac.uk/collection/R01/current...
8     TRAJ  ...  http://vocab.nerc.ac.uk/collection/R01/current...

[9 rows x 5 columns]

The reference table is returned as a pandas.DataFrame. If you want the exact name of this table:

In [9]: NVS.tbl_name('R01')
Out[9]: 
('DATA_TYPE',
 'Terms describing the type of data contained in an Argo netCDF file.',
 'http://vocab.nerc.ac.uk/collection/R01/current/')

On the other hand, if you want to retrieve all reference tables, you can do it with the ArgoNVSReferenceTables.all_tbl() method. It will return a dictionary with table short names as key and pandas.DataFrame as values.

In [10]: all = NVS.all_tbl()

In [11]: all.keys()
Out[11]: odict_keys(['ARGO_WMO_INST_TYPE', 'CYCLE_TIMING_VARIABLE', 'DATA_CENTRE_CODES', 'DATA_STATE_INDICATOR', 'DATA_TYPE', 'DM_QC_FLAG', 'GROUNDED', 'HISTORY_ACTION', 'HISTORY_STEP', 'MEASUREMENT_CODE_CATEGORY', 'MEASUREMENT_CODE_ID', 'OCEAN_CODE', 'PARAMETER', 'PLATFORM_FAMILY', 'PLATFORM_MAKER', 'PLATFORM_TYPE', 'POSITIONING_SYSTEM', 'POSITION_ACCURACY', 'PROF_QC_FLAG', 'REPRESENTATIVE_PARK_PRESSURE_STATUS', 'RTQC_TESTID', 'RT_QC_FLAG', 'SENSOR', 'SENSOR_MAKER', 'SENSOR_MODEL', 'STATUS', 'TRANS_SYSTEM', 'VERTICAL_SAMPLING_SCHEME'])

Deployment Plan#

It may be useful to be able to retrieve meta-data from Argo deployments. argopy can use the OceanOPS API for metadata access to retrieve these information. The returned deployment plan is a list of all Argo floats ever deployed, together with their deployment location, date, WMO, program, country, float model and current status.

To fetch the Argo deployment plan, argopy provides a dedicated utility class: OceanOPSDeployments that can be used like this:

In [12]: from argopy import OceanOPSDeployments

In [13]: deployment = OceanOPSDeployments()

In [14]: df = deployment.to_dataframe()
---------------------------------------------------------------------------
ClientOSError                             Traceback (most recent call last)
Cell In[14], line 1
----> 1 df = deployment.to_dataframe()

File ~/checkouts/readthedocs.org/user_builds/argopy/checkouts/latest/argopy/utilities.py:3349, in OceanOPSDeployments.to_dataframe(self)
   3342 def to_dataframe(self):
   3343     """Return the deployment plan as :class:`pandas.DataFrame`
   3344 
   3345     Returns
   3346     -------
   3347     :class:`pandas.DataFrame`
   3348     """
-> 3349     data = self.to_json()
   3350     if data['total'] == 0:
   3351         raise DataNotFound('Your search matches no results')

File ~/checkouts/readthedocs.org/user_builds/argopy/checkouts/latest/argopy/utilities.py:3339, in OceanOPSDeployments.to_json(self)
   3337 """Return OceanOPS API request response as a json object"""
   3338 if self.data is None:
-> 3339     self.data = self.fs.open_json(self.uri)
   3340 return self.data

File ~/checkouts/readthedocs.org/user_builds/argopy/checkouts/latest/argopy/stores/filesystems.py:706, in httpstore.open_json(self, url, **kwargs)
    697 log.debug("Opening json from: %s" % url)
    698 # try:
    699 #     with self.open(url) as of:
    700 #         js = json.load(of, **kwargs)
   (...)
    704 # except json.JSONDecodeError:
    705 #     raise
--> 706 data = self.fs.cat_file(url)
    707 js = json.loads(data, **kwargs)
    708 if len(js) == 0:

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/fsspec/asyn.py:115, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    112 @functools.wraps(func)
    113 def wrapper(*args, **kwargs):
    114     self = obj or args[0]
--> 115     return sync(self.loop, func, *args, **kwargs)

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/fsspec/asyn.py:100, in sync(loop, func, timeout, *args, **kwargs)
     98     raise FSTimeoutError from return_result
     99 elif isinstance(return_result, BaseException):
--> 100     raise return_result
    101 else:
    102     return return_result

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/fsspec/asyn.py:55, in _runner(event, coro, result, timeout)
     53     coro = asyncio.wait_for(coro, timeout=timeout)
     54 try:
---> 55     result[0] = await coro
     56 except Exception as ex:
     57     result[0] = ex

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/fsspec/implementations/http.py:229, in HTTPFileSystem._cat_file(self, url, start, end, **kwargs)
    227     kw["headers"] = headers
    228 session = await self.set_session()
--> 229 async with session.get(self.encode_url(url), **kw) as r:
    230     out = await r.read()
    231     self._raise_not_found_for_status(r, url)

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/aiohttp/client.py:1141, in _BaseRequestContextManager.__aenter__(self)
   1140 async def __aenter__(self) -> _RetType:
-> 1141     self._resp = await self._coro
   1142     return self._resp

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/aiohttp/client.py:560, in ClientSession._request(self, method, str_or_url, params, data, json, cookies, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, raise_for_status, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx, read_bufsize)
    558 resp = await req.send(conn)
    559 try:
--> 560     await resp.start(conn)
    561 except BaseException:
    562     resp.close()

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/aiohttp/client_reqrep.py:899, in ClientResponse.start(self, connection)
    897 try:
    898     protocol = self._protocol
--> 899     message, payload = await protocol.read()  # type: ignore[union-attr]
    900 except http.HttpProcessingError as exc:
    901     raise ClientResponseError(
    902         self.request_info,
    903         self.history,
   (...)
    906         headers=exc.headers,
    907     ) from exc

File ~/checkouts/readthedocs.org/user_builds/argopy/envs/latest/lib/python3.8/site-packages/aiohttp/streams.py:616, in DataQueue.read(self)
    614 self._waiter = self._loop.create_future()
    615 try:
--> 616     await self._waiter
    617 except (asyncio.CancelledError, asyncio.TimeoutError):
    618     self._waiter = None

ClientOSError: [Errno 104] Connection reset by peer

In [15]: df
Out[15]: 
                                       file  ...                             profiler
0    nmdis/2901623/profiles/R2901623_000.nc  ...  Provor, Seabird conductivity sensor
1   nmdis/2901623/profiles/R2901623_000D.nc  ...  Provor, Seabird conductivity sensor
2    nmdis/2901623/profiles/R2901623_001.nc  ...  Provor, Seabird conductivity sensor
3    nmdis/2901623/profiles/R2901623_002.nc  ...  Provor, Seabird conductivity sensor
4    nmdis/2901623/profiles/R2901623_003.nc  ...  Provor, Seabird conductivity sensor
..                                      ...  ...                                  ...
93   nmdis/2901623/profiles/R2901623_092.nc  ...  Provor, Seabird conductivity sensor
94   nmdis/2901623/profiles/R2901623_093.nc  ...  Provor, Seabird conductivity sensor
95   nmdis/2901623/profiles/R2901623_094.nc  ...  Provor, Seabird conductivity sensor
96   nmdis/2901623/profiles/R2901623_095.nc  ...  Provor, Seabird conductivity sensor
97   nmdis/2901623/profiles/R2901623_096.nc  ...  Provor, Seabird conductivity sensor

[98 rows x 11 columns]

OceanOPSDeployments can also take an index box definition as argument in order to restrict the deployment plan selection to a specific region or period:

deployment = OceanOPSDeployments([-90, 0, 0, 90])
# deployment = OceanOPSDeployments([-20, 0, 42, 51, '2020-01', '2021-01'])
# deployment = OceanOPSDeployments([-180, 180, -90, 90, '2020-01', None])

Note that if the starting date is not provided, it will be set automatically to the current date.

Last, OceanOPSDeployments comes with a plotting method:

fig, ax = deployment.plot_status()
_images/scatter_map_deployment_status.png

Note

The list of possible deployment status name/code is given by:

OceanOPSDeployments().status_code

Status

Id

Description

PROBABLE

0

Starting status for some platforms, when there is only a few metadata available, like rough deployment location and date. The platform may be deployed

CONFIRMED

1

Automatically set when a ship is attached to the deployment information. The platform is ready to be deployed, deployment is planned

REGISTERED

2

Starting status for most of the networks, when deployment planning is not done. The deployment is certain, and a notification has been sent via the OceanOPS system

OPERATIONAL

6

Automatically set when the platform is emitting a pulse and observations are distributed within a certain time interval

INACTIVE

4

The platform is not emitting a pulse since a certain time

CLOSED

5

The platform is not emitting a pulse since a long time, it is considered as dead