ascat package

Subpackages

Submodules

ascat.accessors module

class ascat.accessors.CFDiscreteGeometryAccessor(xarray_obj: Dataset)[source]

Bases: object

property array_type: str
resample_to_orthomulti(**kwargs)[source]
sel_instances(instance_vals: Sequence[int, str] | np.ndarray | None = None, instance_lookup_vector: np.ndarray[Any, np.dtype[np.bool]] | None = None, **kwargs)[source]
set_coord_vars(coord_vars: Sequence[str])[source]
set_instance_vars(instance_vars: Sequence[str])[source]
set_sample_dimension(sample_dim: str)[source]
property timeseries_id: str
to_contiguous_ragged(**kwargs)[source]
to_indexed_ragged(**kwargs)[source]
to_orthomulti(**kwargs)[source]
to_point_array()[source]
to_raster(*args, **kwargs)[source]
class ascat.accessors.PyGeoGriddedArrayAccessor(xarray_obj: Dataset)[source]

Bases: object

property grid
lonlat_vars_from_gpi_var(gpi_var, lon_var='lon', lat_var='lat') → tuple[DataArray, DataArray][source]
sel_bbox(bbox: Sequence[float]) → Dataset[source]

Select data for a bounding box.

bboxtuple, optional

Tuple of (latmin, latmax, lonmin, lonmax) coordinates.

sel_cells(cells: Sequence[float]) → Dataset[source]
sel_coords(coords: Sequence[Sequence[float]], max_coord_dist: float = inf) → Dataset[source]
sel_geom(geom: BaseGeometry) → Dataset[source]
sel_gpis(gpis: Sequence[int] | None = None, lookup_vector: ndarray | None = None) → Dataset[source]
set_grid_name(grid_name: str, grid_class: type | None = None)[source]

ascat.array_utils module

Lightweight, dependency-free array primitives.

These helpers depend only on numpy (no xarray, netCDF4 or shapely), so they can be used by the CF layer (ascat.cf_conversions, ascat.ragged_array) without pulling in heavier dependencies. The project sentinel fill values live here and are recognised by ascat.utils.mask_dtype_nans().

ascat.array_utils.fill_value(dtype)[source]

Return the fill value used to pad missing elements of a given dtype.

Uses the project-wide sentinel fills from dtype_to_nan (so they are recognised by ascat.utils.mask_dtype_nans()), and NaT for datetimes.

Parameters:

dtype (numpy.dtype) – Data type.

Returns:

fill – Fill value for the given dtype.

Return type:

scalar

ascat.array_utils.pad_to_2d(var, x, y, shape)[source]

Scatter a 1d array into a padded 2d array.

Parameters:
Returns:

padded – 2d array, with missing elements set to the dtype fill value.

Return type:

numpy.ndarray

ascat.array_utils.vrange(starts, stops)[source]

Create concatenated ranges of integers for multiple start/stop values.

Parameters:
Returns:

ranges – Concatenated ranges.

Return type:

numpy.ndarray

Example

>>> vrange(np.array([1, 3, 4, 6]), np.array([1, 5, 7, 6]))
array([3, 4, 4, 5, 6])

ascat.cell module

class ascat.cell.CellGridFiles(root_path, file_class, grid, fn_format='{cell:04d}.nc', sf_format=None, preprocessor=None)[source]

Bases: object

convert_to_contiguous(out_dir, print_progress=True, **kwargs)[source]

Convert all files in the collection to contiguous format and write to disk.

Parameters:

  • out_dir (str) – Output directory.

  • print_progress (bool, optional) – Whether to print progress messages to console. Default is True.

  • kwargs (dict) – Keyword arguments passed to the reprocess method.

classmethod from_product_class(root_path, product_class, **kwargs)[source]
classmethod from_product_id(root_path, product_id, **kwargs)[source]
read(cell=None, location_id=None, coords=None, bbox=None, geom=None, max_coord_dist=inf, date_range=None, **kwargs)[source]

Read data matching a spatial and temporal criterion.

Parameters:

  • cell (int or list of int) – Grid cell number to read.

  • location_id (int or list of int) – Location id.

  • coords (tuple of numeric or tuple of iterable of numeric) –

    Tuple of (lon, lat) coordinates. lon and lat could each be numpy arrays in order to read multiple coordinates. For each coordinate the nearest grid point within max_coord_dist (in spherical cartesian coordinates) will be selected.

    Note that if any passed coordinates share the same nearest grid point, that grid point will only be represented once in the output dataset.

  • bbox (tuple) – Tuple of (latmin, latmax, lonmin, lonmax) coordinates.

  • geom (shapely.geometry) – Geometry object.

  • max_coord_dist (float) – The maximum distance a coordinate’s nearest grid point can be from it to be selected (in spherical cartesian coordinates). Default is np.inf.

  • date_range (tuple of np.datetime64) – Tuple of (start, end) dates.

Returns:

Filtered and merged data for the specified spatiotemporal region.

Return type:

xarray.Dataset

reprocess(out_dir, func, parallel=True, **kwargs)[source]

Use Filenames.reprocess to apply a function to all files in the collection and save the results to out_dir.

Parameters:

  • out_dir (str) – Output directory.

  • func (callable) – Function to apply to each file.

  • parallel (bool, optional) – Whether to process files in parallel. Default is True.

  • kwargs (dict) – Keyword arguments passed to func.

Search files for cells matching a spatial criterion. All args are declared as optional; but one and only one should be passed.

Parameters:

  • cell (int or list of int) – Grid cell number to read.

  • location_id (int or list of int) – Location id.

  • coords (tuple of numeric or tuple of iterable of numeric) – Tuple of (lon, lat) coordinates.

  • bbox (tuple) – Tuple of (latmin, latmax, lonmin, lonmax) coordinates.

  • geom (shapely.geometry) – Geometry object.

Returns:

filenames – Filenames.

Return type:

list of str

class ascat.cell.OrthoMultiTimeseriesCell(filenames)[source]

Bases: Filenames

Class to read and merge orthomulti cell files.

read(date_range=None, location_id=None, lookup_vector=None, preprocessor=None, parallel=False, **kwargs)[source]

Read data from OrthoMulti Cell files.

Parameters:

  • date_range (tuple of np.datetime64) – Tuple of (start, end) dates.

  • location_id (list of int) – List of timeseries IDs to read.

  • lookup_vector (np.ndarray) – Lookup vector.

  • preprocessor (callable, optional) – Function to preprocess the dataset.

  • parallel (bool, optional) – Whether or not to read/preprocess in parallel. Default is False.

class ascat.cell.RaggedArrayTs(filenames)[source]

Bases: Filenames

Class to read and merge ragged array cell files.

read(date_range=None, location_id=None, lookup_vector=None, preprocessor=None, return_format=None, parallel=False, **kwargs)[source]

Read data from Ragged Array Cell files.

Parameters:

  • date_range (tuple of np.datetime64) – Tuple of (start, end) dates.

  • location_id (list of int) – List of timeseries IDs to read.

  • lookup_vector (np.ndarray) – Lookup vector.

  • preprocessor (callable, optional) – Function to preprocess the dataset.

  • return_format (str, optional) – CF discrete geometry format to return data as. Can be “point”, “indexed”, or “contiguous”.

  • parallel (bool, optional) – Whether or not to read/preprocess in parallel. Default is False.

  • **kwargs (dict)

ascat.cf_array module

class ascat.cf_array.CFDiscreteGeom(xarray_obj: Dataset, coord_vars: Sequence[str] | None = None, instance_vars: Sequence[str] | None = None, contiguous_sort_vars: Sequence[str] | None = None)[source]

Bases: object

property array_type
classmethod from_dataset(ds, **kwargs)[source]

Detect the array type of ds and wrap it in the right class.

property timeseries_id

Name of the variable carrying cf_role='timeseries_id'.

class ascat.cf_array.OrthoMultiTimeseriesArray(xarray_obj: Dataset, coord_vars: Sequence[str] | None = None, instance_vars: Sequence[str] | None = None, contiguous_sort_vars: Sequence[str] | None = None)[source]

Bases: CFDiscreteGeom

sel_instances(instance_vals: Sequence[int | str] | ndarray | None = None, instance_lookup_vector: ndarray | None = None)[source]

Select requested timeseries instances from an orthomulti timeseries array dataset.

Parameters:

  • instance_vals (Union[Sequence[Union[int, str]], np.ndarray], optional) – List of instance values to select, by default None

  • instance_lookup_vector (Union[np.ndarray], optional) – Lookup vector for instance values, by default None

set_sample_dimension(sample_dim: str)[source]
to_contiguous_ragged(sample_dim: str = 'obs', **kwargs) → Dataset[source]

Convert to a contiguous ragged array (via a point array).

to_indexed_ragged(sample_dim: str = 'obs', **kwargs) → Dataset[source]

Convert to an indexed ragged array (via a point array).

to_point_array(sample_dim: str = 'obs')[source]

Convert the orthomulti timeseries array to a point array.

The instance and sample dimensions are stacked into a single sample dimension, so every instance/time combination becomes one observation (an orthomulti array is dense by construction).

to_raster(x_var, y_var)[source]
class ascat.cf_array.PointArray(xarray_obj: Dataset, coord_vars: Sequence[str] | None = None, instance_vars: Sequence[str] | None = None, contiguous_sort_vars: Sequence[str] | None = None)[source]

Bases: CFDiscreteGeom

class ascat.cf_array.RaggedArray(xarray_obj: Dataset, coord_vars: Sequence[str] | None = None, instance_vars: Sequence[str] | None = None, contiguous_sort_vars: Sequence[str] | None = None)[source]

Bases: CFDiscreteGeom

sel_instances(instance_vals: Sequence[int | str] | ndarray | None = None, instance_lookup_vector: ndarray | None = None) → Dataset[source]
set_sample_dimension(sample_dim: str)[source]
to_contiguous_ragged(count_var: str = 'row_size', sort_vars: Sequence[str] | None = None) → Dataset[source]
to_indexed_ragged(index_var: str = 'locationIndex') → Dataset[source]
to_point_array()[source]
class ascat.cf_array.TimeseriesPointArray(xarray_obj: Dataset, coord_vars: Sequence[str] | None = None, instance_vars: Sequence[str] | None = None, contiguous_sort_vars: Sequence[str] | None = None)[source]

Bases: PointArray

Assumptions made beyond basic CF conventions:

  • cf_role=”timeseries_id” is used to identify the timeseries ID variable for purposes

    of selecting instances and converting to ragged arrays. If you only have a single timeseries there’s not much point in using this class.

resample_to_orthomulti(instance_dim: str = 'locations', timeseries_id: str = 'location_id', count_var: str = 'row_size', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None, sort_vars: Sequence[str] | None = None, vars_to_resample: Sequence[str] | None = None, resample_method: Callable = <function mean>, resample_period: str = '1ME')[source]
sel_instances(instance_vals: Sequence[int | str] | ndarray | None = None, instance_lookup_vector: ndarray | None = None, timeseries_id: str = 'location_id')[source]
set_sample_dimension(sample_dim: str)[source]
to_contiguous_ragged(instance_dim: str = 'locations', timeseries_id: str = 'location_id', count_var: str = 'row_size', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None, sort_vars: Sequence[str] | None = None) → Dataset[source]
to_indexed_ragged(instance_dim: str = 'locations', timeseries_id: str = 'location_id', index_var: str = 'locationIndex', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None) → Dataset[source]
to_orthomulti(instance_dim: str = 'locations', timeseries_id: str = 'location_id', count_var: str = 'row_size', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None, sort_vars: Sequence[str] | None = None)[source]
to_point_array()[source]
ascat.cf_array.cf_array_class(ds, array_type, **kwargs)[source]

Wrap ds in the array class matching array_type.

ascat.cf_array.cf_array_type(ds)[source]

Detect the CF discrete sampling geometry array type of ds.

ascat.cf_array.check_orthomulti_ts(ds)[source]

ascat.cf_conversions module

Conversions between CF discrete sampling geometry representations.

These are plain functions operating on xarray.Dataset objects with the relevant dimension/variable names passed explicitly. They do not depend on the wrapper classes in ascat.cf_array or ascat.ragged_array, so they can be used directly on any dataset that follows the CF conventions.

Representations

  • point : one sample dimension, no grouping.

  • indexed ragged : an index variable maps each sample to an instance (CF 9.3.4).

  • contiguous ragged: a count variable gives each instance’s number of samples (CF 9.3.3).

  • orthogonal multidim : a dense (instance x element) array sharing one element axis, complete/unpadded (CF 9.3.1).

  • incomplete multidim : a dense (instance x element) array with a positional element axis, padded with fill values (CF 9.3.2).

ascat.cf_conversions.contiguous_to_incomplete(ds: Dataset, sample_dim: str, instance_dim: str, count_var: str, element_dim: str | None = None, instance_id_var: str | None = None) → Dataset[source]

Convert a contiguous ragged array to an incomplete multidimensional array (CF 9.3.2).

Each instance’s samples are placed in the leading columns of a dense (instance x element) array; the trailing columns are padded with the dtype fill value. Variables with extra (non-sample) dimensions keep them, and instance-level coordinates are preserved.

ascat.cf_conversions.contiguous_to_indexed(ds: Dataset, sample_dim: str, instance_dim: str, count_var: str, index_var: str) → Dataset[source]

Convert a contiguous ragged array dataset to an indexed ragged one.

ascat.cf_conversions.contiguous_to_orthogonal(ds: Dataset, sample_dim: str, instance_dim: str, count_var: str, element_coord: str, element_dim: str | None = None, instance_id_var: str | None = None, strict: bool = True) → Dataset[source]

Convert a contiguous ragged array to an orthogonal multidimensional array (CF 9.3.1).

All instances must share the same set of element_coord values (e.g. the same time axis); the samples are pivoted onto that shared 1-D coordinate.

Parameters:

  • element_coord (str) – Name of the per-sample coordinate variable defining the shared element axis (e.g. "time").

  • strict (bool, optional) – If True (default), raise if the instances do not form a complete rectangular grid on element_coord (i.e. it is not truly orthogonal).

Raises:

ValueError – If strict and the instances do not share a common, complete element axis.

ascat.cf_conversions.contiguous_to_point(ds: Dataset, sample_dim: str, instance_dim: str, count_var: str) → Dataset[source]

Convert a contiguous ragged array dataset to a point dataset.

ascat.cf_conversions.detect_cf_representation(ds: Dataset) → str[source]

Detect the CF discrete sampling geometry representation of a dataset.

Returns one of “contiguous”, “indexed”, “orthogonal”, “incomplete” or “point”. Ragged arrays are identified by their marker attributes (sample_dimension / instance_dimension); a multidimensional array is recognised by a data variable with two or more dimensions and classed as orthogonal when one of those dimensions is a shared coordinate axis, else incomplete; anything else is treated as point data.

ascat.cf_conversions.finalize_cf(ds: Dataset, feature_type: str | None = 'timeSeries', instance_id_var: str | None = None, coordinates: Sequence[str] | None = None) → Dataset[source]

Fill in CF discrete-sampling-geometry metadata on a dataset.

Sets the global featureType attribute, marks the instance identifier with cf_role="timeseries_id" (for non-point feature types), and gives each measurement variable a coordinates attribute listing the spatiotemporal coordinate variables. Existing cf_role / coordinates attributes are kept. The input dataset is not modified.

Parameters:

  • ds (xarray.Dataset) – Dataset in any CF discrete sampling geometry representation.

  • feature_type (str or None, optional) – Value for the global featureType attribute (default “timeSeries”; None to leave it unchanged).

  • instance_id_var (str, optional) – Name of the instance identifier variable to mark with cf_role="timeseries_id".

  • coordinates (sequence of str, optional) – Coordinate variable names for the coordinates attribute. If None, they are auto-detected (lon/lat/time/alt by name or standard_name).

Returns:

ds – A copy with the CF metadata filled in.

Return type:

xarray.Dataset

ascat.cf_conversions.incomplete_to_contiguous(ds: Dataset, instance_dim: str, element_dim: str, count_var: str = 'row_size', sample_dim: str = 'obs') → Dataset[source]

Convert an incomplete multidimensional array (CF 9.3.2) to a contiguous ragged array.

Padded (fill-valued) elements are dropped, so each instance’s valid samples are packed contiguously. A cell is considered padding if it equals the dtype fill value (or is NaN for floats).

Notes

Because the incomplete representation marks padding with fill values, this conversion cannot distinguish a genuine observation whose value equals the fill sentinel (or NaN) from padding — such observations are dropped. It also assumes the padding pattern is uniform across variables and takes the valid mask from the first (instance, element) variable. Use the contiguous or indexed representations to preserve fill-valued observations.

ascat.cf_conversions.indexed_to_contiguous(ds: Dataset, sample_dim: str, instance_dim: str, count_var: str, index_var: str, sort_vars: Sequence[str] | None = None) → Dataset[source]

Convert an indexed ragged array dataset to a contiguous ragged one.

ascat.cf_conversions.indexed_to_point(ds: Dataset, sample_dim: str, instance_dim: str, index_var: str) → Dataset[source]

Convert an indexed ragged array dataset to a point dataset.

ascat.cf_conversions.orthogonal_to_contiguous(ds: Dataset, instance_dim: str, element_dim: str, element_coord: str | None = None, count_var: str = 'row_size', sample_dim: str = 'obs') → Dataset[source]

Convert an orthogonal multidimensional array (CF 9.3.1) to a contiguous ragged array.

Every (instance, element) cell is a valid observation, so each instance has element samples and the shared element coordinate is broadcast to every observation.

ascat.cf_conversions.point_to_contiguous(ds: Dataset, sample_dim: str, instance_dim: str, timeseries_id: str, count_var: str = 'row_size', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None, sort_vars: Sequence[str] | None = None) → Dataset[source]

Convert a point dataset to a contiguous ragged array dataset.

ascat.cf_conversions.point_to_indexed(ds: Dataset, sample_dim: str, instance_dim: str, timeseries_id: str, index_var: str = 'locationIndex', instance_vars: Sequence[str] | None = None, coord_vars: Sequence[str] | None = None) → Dataset[source]

Convert a point dataset to an indexed ragged array dataset.

ascat.cgls module

CGLS SWI interface.

class ascat.cgls.SWI_TS(data_path, parameters=['SWI_001', 'SWI_005', 'SWI_010', 'SWI_015', 'SWI_020', 'SWI_040', 'SWI_060', 'SWI_100', 'SSF'], dt=None, version=None, grid_fname=None, read_bulk=True, fname_template='c_gls_SWI-TS_{dt}_C{cell}_ASCAT_V{version}', cell_fn='{:04d}')[source]

Bases: GriddedNcOrthoMultiTs

SWI TS reader for timeseries data from CGLOPS

Parameters:

  • data_path (string) – path to the netCDF files

  • parameters (list) – list of parameters to read from netCDF file

  • dt (string, optional) – datetime in the filenames of the cells. If not given it is detected from the files in the data_path. Automatic detection only works if the files follow the CGLS naming convention.

  • version (string, optional) – version number of the files If not given it is detected from the files in the data_path. Automatic detection only works if the files follow the CGLS naming convention.

  • grid_fname (string, optional) – filename + path of the grid netCDF file, default is the standard grid file (c_gls_SWI-STATIC-DGG_201501010000_GLOBE_ASCAT_V3.0.1.nc) in the same folder as the data

  • read_bulk (boolean, optional) – if set to true then a complete 5x5 degree cell will be read at once providing speedup if the complete data is needed.

  • fname_template (string, optional) – Filename template. Has to have three slots for {dt}, {version} and a slot for the {cell} number that is available for further formatting. The has to be without the .nc ending since this is added during reading.

  • cell_fn (string, optional) – cell number in the fname_template.

ascat.file_handling module

File search methods.

class ascat.file_handling.ChronFiles(root_path, cls, fn_templ, sf_templ, cls_kwargs=None, err=True, fn_read_fmt=None, sf_read_fmt=None, fn_write_fmt=None, sf_write_fmt=None, cache_size=0)[source]

Bases: MultiFileHandler

Managing chronological files with a date field in the filename.

read_period(dt_start, dt_end, dt_delta=datetime.timedelta(days=1), dt_buffer=datetime.timedelta(days=1), search_date_fmt='%Y%m%d*', date_field='date', date_field_fmt='%Y%m%d', end_inclusive=True, fmt_kwargs={}, **kwargs)[source]

Read data for given interval.

Parameters:

  • dt_start (datetime) – Start datetime.

  • dt_end (datetime) – End datetime.

  • dt_delta (timedelta, optional) – Time delta used to jump through search date.

  • dt_buffer (timedelta, optional) – Search buffer used to find files which could possibly contain data but would be left out because of dt_start.

  • search_date_fmt (str, optional) – Search date string format used during file search (default: %Y%m%d*).

  • date_field (str, optional) – Date field name (default: “date”).

  • date_field_fmt (str, optional) – Date field string format (default: %Y%m%d).

Returns:

data – Data stored in file.

Return type:

dict, numpy.ndarray

search_date(timestamp, search_date_fmt='%Y%m%d*', date_field='date', date_field_fmt='%Y%m%d', return_date=False, **fmt_kwargs)[source]

Search files for given date.

Parameters:

  • timestamp (datetime) – Search date.

  • search_date_fmt (str, optional) – Search date string format used during file search (default: %Y%m%d*).

  • date_field (str, optional) – Date field name (default: “date”)

  • date_field_format (str, optional) – Date field string format (default: %Y%m%d).

  • return_date (bool, optional) – Return date parsed from filename (default: False).

Returns:

  • filenames (list of str) – Filenames.

  • dates (list of datetime) – Parsed date of filename (only returned if return_date=True).

search_period(dt_start, dt_end, dt_delta=datetime.timedelta(days=1), search_date_fmt='%Y%m%d*', date_field='date', date_field_fmt='%Y%m%d', end_inclusive=True, **fmt_kwargs)[source]

Search files for time period.

Parameters:

  • dt_start (datetime) – Start datetime.

  • dt_end (datetime) – End datetime.

  • dt_delta (timedelta, optional) – Time delta used to jump through search date.

  • search_fmt (str, optional) – Search date string format used during file search (default: %Y%m%d*).

  • date_field (str, optional) – Date field name (default: “date”).

  • date_field_fmt (str, optional) – Date field string format (default: %Y%m%d).

  • end_inclusive (bool, optional) – Include files from a dt_delta length period beyond dt_end if True (default: False).

Returns:

filenames – Filenames.

Return type:

list of str

class ascat.file_handling.CsvFile(filename, mode='r')[source]

Bases: Filenames

Read and write single CSV file.

header2dtype(header)[source]

Convert header string to dtype info.

Parameters:

header (str) – Header string with dtype info.

Returns:

dtype – Data type.

Return type:

numpy.dtype

read_period(dt_start, dt_end)[source]

Read subset data from CSV file for given interval.

Parameters:

interval ((datetime, datetime)) – Time interval to extract data.

Returns:

data – Data.

Return type:

numpy.ndarray

class ascat.file_handling.CsvFiles(root_path)[source]

Bases: ChronFiles

Write CSV files.

class ascat.file_handling.FileSearch(root_path, fn_pattern, sf_pattern=None)[source]

Bases: object

FileSearch class.

create_isearch_func(func, recursive=False)[source]

Create custom search function returning it.

Parameters:

  • func (function) – Search function with its own args/kwargs returning a filename format dictionary and subfolder format dictionary depending on the passed arguments.

  • recursive (bool, optional) – If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories (default: False).

Returns:

custom_search – Custom search function returning an iterator of path/file names that match.

Return type:

function

create_search_func(func, recursive=False)[source]

Create custom search function returning it.

Parameters:

  • func (function) – Search function with its own args/kwargs returning a filename format dictionary and subfolder format dictionary depending on the passed arguments.

  • recursive (bool, optional) – If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories (default: False).

Returns:

custom_search – Custom search function returning a possibly-empty list of path/file names that match.

Return type:

function

isearch(fn_fmt, sf_fmt=None, recursive=False)[source]

Search filesystem for given pattern returning iterator.

Parameters:

  • fn_fmt (dict) – Filename format dictionary.

  • sf_fmt (dict of dicts, optional) – Format dictionary for subfolders (default: None).

  • recursive (bool, optional) – If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories (default: False).

Returns:

filenames – Iterator which yields the same values as search() without actually storing them all simultaneously.

Return type:

iterator

search(fn_fmt, sf_fmt=None, recursive=False)[source]

Search filesystem for given pattern returning list.

Parameters:

  • fn_fmt (dict) – Filename format dictionary.

  • sf_fmt (dict of dicts, optional) – Format dictionary for subfolders (default: None).

  • recursive (bool, optional) – If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories (default: False).

Returns:

filenames – Return a possibly-empty list of path/file names that match.

Return type:

list of str

class ascat.file_handling.FilenameTemplate(root_path, fn_templ, sf_templ=None)[source]

Bases: object

FilenameTemplate class.

build_basename(fmt)[source]

Create file basename from format dictionary.

Parameters:

fmt (dict) – Filename format applied on filename pattern (fn_pattern). e.g. fn_pattern = “{date}*.{suffix}” with fmt = {“date”: “20000101”, “suffix”: “nc”} returns “20000101*.nc”

Returns:

filename – Filename with format_dict applied.

Return type:

str

build_filename(fn_fmt, sf_fmt=None)[source]

Create filename from format dictionary.

Parameters:

  • fn_fmt (dict) –

    Filename format applied on filename pattern (fn_pattern), e.g.:

    fn_pattern = "{date}*.{suffix}"
fn_format_dict = {"date": "20000101", "suffix": "nc"}
# returns "20000101*.nc"

  • sf_fmt (dict of dicts) –

    Format dictionary for subfolders. Each subfolder contains a dictionary defining the format of the folder name, e.g.:

    sf_templ = {"years": {year}, "months": {month}}
sf_format = {"years": {"year": "2000"},
             "months": {"month": "02"}}
# returns ["2000", "02"]

Returns:

filename – Filename with format_dict applied.

Return type:

str

build_subfolder(fmt)[source]

Create subfolder path from format dictionary.

Parameters:

fmt (dict of dicts) –

Format dictionary for subfolders. Each subfolder contains a dictionary defining the format of the folder name, e.g.:

sf_pattern = {"years": {year}, "months": {month}}
format_dict = {"years": {"year": "2000"},
               "months": {"month": "02"}}
# returns ["2000", "02"]

Returns:

subfolder – Subfolder with format_dict applied.

Return type:

list of str

property template

Name property.

class ascat.file_handling.Filenames(filenames)[source]

Bases: object

A class to handle operations on multiple filenames.

This class provides methods for reading from, writing to, and merging data from multiple files.

close()[source]

Close file(s).

This method can be overridden in subclasses if necessary.

iter_read(print_progress=False, **kwargs)[source]

Iterate over all files and yield data.

Yields:

object – Data read from each file.

iter_read_nbytes(max_nbytes, print_progress=False, **kwargs)[source]

Iterate over all files and yield data until the specified number of bytes is reached. If _read returns dask objects, they are computed (in parallel) before merging the data.

merge(data)[source]

Merge data from multiple data objects.

Parameters:

data (list) – List of data objects.

Returns:

Merged data, or None if the input list is empty.

Return type:

object

read(parallel=False, closer_attr=None, **kwargs)[source]

Read all data from files.

Returns:

Merged data from all files.

Return type:

object

reprocess(out_dir, func, parallel=False, print_progress=False, read_kwargs=None, **write_kwargs)[source]

Reprocess data from all files through func, writing the results to out_dir. Assumes that if any files have the same name, they should be merged.

Parameters:

  • out_dir (Path) – Directory to write the output files. This will be prepended to the filenames.

  • func (function) – The function to apply to the data before writing out.

  • parallel (bool, optional) – Whether to process the data in parallel (default: False).

  • **kwargs (dict) – Additional keyword arguments for writing.

write(data, parallel=False, print_progress=False, **kwargs)[source]

Write data to file.

If there’s only one filename in self.filenames, write provided data to that file. If there is more than one filename, write each element of the provided data list to the corresponding filename.

Parameters:

data (list of objects) – The data to write. Should be a list with the same length as self.filenames, where each element is the data to be written to the corresponding filename.

class ascat.file_handling.MultiFileHandler(root_path, cls, fn_templ, sf_templ=None, cls_kwargs=None, err=False, cache_size=0)[source]

Bases: object

MultiFileHandler class.

read(*fmt_args, fmt_kwargs=None, cls_kwargs=None)[source]

Read data.

Parameters:

  • fmt_args (tuple) – Format arguments.

  • fmt_kwargs (dict, optional) – Format keywords (Default: None).

  • cls_kwargs (dict, optional) – Class keywords (Default: None).

Returns:

data – Data stored in file.

Return type:

dict, numpy.ndarray

read_file(filename, cls_kwargs=None)[source]

Read data for given filename.

Parameters:

filename (str) – Filename.

search(fn_search_pattern, sf_search_pattern=None, custom_fn_templ=None, custom_sf_templ=None)[source]

Search files for given root path and filename/folder pattern.

Returns:

filenames – Filenames.

Return type:

list of str

write(data, *fmt_args, fmt_kwargs=None, cls_kwargs=None)[source]

Write data.

Parameters:

  • data (dict, numpy.ndarray) – Data to write.

  • fmt_args (tuple) – Format arguments.

  • fmt_kwargs (dict, optional) – Format keywords (Default: None).

  • cls_kwargs (dict, optional) – Class keywords (Default: None).

write_file(data, filename, cls_kwargs=None)[source]

Write data for given filename.

Parameters:

filename (str) – Filename.

ascat.gridded_ragged_array module

Grid-aware reading of ragged-array cell files.

A pygeogrids CellGrid links grid point indices (gpi) and coordinates to cell numbers, and each cell is stored in its own ragged-array file. A gpi (or the nearest gpi to a lon/lat) is translated to a cell, the cell file is located and read with the matching ascat.ragged_array class, and the single grid point’s time series is returned.

Use GriddedContiguousRaggedArray for contiguous ragged cell files and GriddedIndexedRaggedArray for indexed ragged cell files.

class ascat.gridded_ragged_array.GridRegistry[source]

Bases: object

Resolve grid names to pygeogrids grids.

"fibgrid_<spacing>" (e.g. “fibgrid_12.5”) builds a FibGrid on demand; any other name must be registered with register(), either as a grid object or a zero-argument factory (e.g. lambda: load_grid(path)). Resolved grids are cached, so a grid is built or loaded only once per name.

Each instance holds its own registry and cache. A shared module-level instance, grid_registry, is used by the readers by default; create a separate GridRegistry() when you want isolated state (e.g. in tests).

get(name: str) → CellGrid[source]

Return the grid for name, building and caching it on first use.

register(name: str, grid) → None[source]

Register a grid object or zero-argument factory under name.

class ascat.gridded_ragged_array.GriddedContiguousRaggedArray(root_path: str | Path, grid: CellGrid | str, fn_format: str = '{:04d}.nc', count_var: str = 'row_size', instance_dim: str = 'locations', instance_id_var: str = 'location_id', trim: bool = True, cache: bool = False)[source]

Bases: GriddedRaggedArray

Read gridded contiguous ragged-array cell files.

See GriddedRaggedArray for the gridded reading and caching behavior. Each cell is read with ContiguousRaggedArray.

Parameters:

  • root_path (str or pathlib.Path) – Directory containing the cell files.

  • grid (pygeogrids.grids.CellGrid or str) – Grid object or registry name.

  • fn_format (str, optional) – Cell file name format (default: “{:04d}.nc”).

  • count_var (str, optional) – Count variable name (default: “row_size”).

  • instance_dim (str, optional) – Instance dimension name (default: “locations”).

  • instance_id_var (str, optional) – Variable holding the instance identifiers, matching the grid gpi (default: “location_id”).

  • trim (bool, optional) – Drop fill/padding locations when reading a cell (default: True).

  • cache (bool, optional) – Keep every read cell in memory (default: False).

class ascat.gridded_ragged_array.GriddedIndexedRaggedArray(root_path: str | Path, grid: CellGrid | str, fn_format: str = '{:04d}.nc', index_var: str = 'locationIndex', sample_dim: str = 'obs', instance_id_var: str = 'location_id', cache: bool = False)[source]

Bases: GriddedRaggedArray

Read gridded indexed ragged-array cell files.

See GriddedRaggedArray for the gridded reading and caching behavior. Each cell is read with IndexedRaggedArray.

Parameters:

  • root_path (str or pathlib.Path) – Directory containing the cell files.

  • grid (pygeogrids.grids.CellGrid or str) – Grid object or registry name.

  • fn_format (str, optional) – Cell file name format (default: “{:04d}.nc”).

  • index_var (str, optional) – Index variable name (default: “locationIndex”).

  • sample_dim (str, optional) – Sample dimension name (default: “obs”).

  • instance_id_var (str, optional) – Variable holding the instance identifiers, matching the grid gpi (default: “location_id”).

  • cache (bool, optional) – Keep every read cell in memory (default: False).

class ascat.gridded_ragged_array.GriddedOrthoMultiArray(root_path: str | Path, grid: CellGrid | str, fn_format: str = '{:04d}.nc', instance_dim: str = 'locations', element_dim: str = 'time', element_coord: str = None, instance_id_var: str = 'location_id', cache: bool = False)[source]

Bases: GriddedRaggedArray

Read gridded orthogonal multidimensional array cell files (CF 9.3.1).

See GriddedRaggedArray for the gridded reading and caching behavior. Each cell is read with OrthogonalMultidimArray; all locations in a cell share the same element (e.g. time) axis.

Parameters:

  • root_path (str or pathlib.Path) – Directory containing the cell files.

  • grid (pygeogrids.grids.CellGrid or str) – Grid object or registry name.

  • fn_format (str, optional) – Cell file name format (default: “{:04d}.nc”).

  • instance_dim (str, optional) – Instance dimension name (default: “locations”).

  • element_dim (str, optional) – Element dimension name (default: “time”).

  • element_coord (str, optional) – Shared element coordinate variable name (default: element_dim).

  • instance_id_var (str, optional) – Variable holding the instance identifiers, matching the grid gpi (default: “location_id”).

  • cache (bool, optional) – Keep every read cell in memory (default: False).

class ascat.gridded_ragged_array.GriddedRaggedArray(root_path: str | Path, grid: CellGrid | str, fn_format: str = '{:04d}.nc', cache: bool = False)[source]

Bases: object

Base class: read gridded ragged-array cell files by grid point or location.

A CellGrid maps grid point indices (gpi) and coordinates to cell numbers; each cell is stored in its own ragged-array file. A gpi (or the nearest gpi to a lon/lat) is translated to a cell via grid.gpi2cell, the cell file is located with fn_format and read, and the grid point’s time series is selected by its identifier.

Two caching strategies control how cell files are kept in memory:

  • cache=False (default): only the most recently read cell is kept, like pynetcf’s GriddedNcTs. Efficient for reads that stay within one cell; re-reads when hopping between cells.

  • cache=True: every cell that is read is retained in memory. Efficient when reading many grid points spread across (and revisiting) cells, at the cost of memory.

This base class is not used directly; use GriddedContiguousRaggedArray or GriddedIndexedRaggedArray.

Parameters:

  • root_path (str or pathlib.Path) – Directory containing the cell files (searched recursively).

  • grid (pygeogrids.grids.CellGrid or str) – Grid object, or a name resolvable by the module-level grid_registry (e.g. “fibgrid_12.5”).

  • fn_format (str, optional) – Format string for cell file names, formatted with the cell number, e.g. “{:04d}.nc” (default) or “H120_{:04d}.nc”.

  • cache (bool, optional) – Keep every read cell in memory instead of only the last one (default: False).

clear_cache()[source]

Close all open cell files and drop cached cells (alias of close).

close()[source]

Close all open cell files and drop cached cells.

gpi_from_coords(lon: float, lat: float, max_dist: float = inf) → int[source]

Nearest grid point index to a lon/lat.

Parameters:

  • lon (float) – Coordinates.

  • lat (float) – Coordinates.

  • max_dist (float, optional) – Maximum allowed distance in meters (default: no limit).

Returns:

gpi – Grid point index of the nearest grid point.

Return type:

int

Raises:

ValueError – If the nearest grid point is farther than max_dist.

read(gpi=None, lon=None, lat=None, max_dist: float = inf)[source]

Read the time series of one or more grid points.

Either gpi or both lon and lat must be given. Each may be a scalar (single grid point) or array-like (many grid points at once).

Parameters:

  • gpi (int or array-like of int, optional) – Grid point index/indices.

  • lon (float or array-like of float, optional) – Coordinates; the nearest grid point to each is used.

  • lat (float or array-like of float, optional) – Coordinates; the nearest grid point to each is used.

  • max_dist (float, optional) – Maximum allowed distance in meters for a lon/lat lookup. For a single lon/lat this raises if exceeded; for arrays, coordinates beyond it are dropped.

Returns:

ds – For a scalar request, the time series (or None if the grid point has no data in its cell file). For an array request, a dict mapping each grid point index to its time series (or None). Cell files are read once per cell, so grouping many grid points is efficient.

Return type:

xarray.Dataset or None, or dict of {gpi: (xarray.Dataset or None)}

read_cell(cell: int)[source]

Read a whole cell.

Parameters:

cell (int) – Cell number.

Returns:

data – Ragged array for the cell (cached per the caching strategy).

Return type:

ContiguousRaggedArray or IndexedRaggedArray

ascat.gridded_ragged_array.cells_to_incomplete_zarr(root_path, store, fn_pattern: str = '*.nc', count_var: str = 'row_size', instance_dim: str = 'locations', instance_id_var: str = 'location_id', element_dim: str = 'element', trim: bool = True, chunks: dict = None, shards: dict = None, n_workers: int = 1)[source]

Convert contiguous ragged cell files to a monolithic incomplete Zarr store.

Every cell file under root_path is read, converted to an incomplete multidimensional array (CF 9.3.2) — a dense (instance, element) block padded with fill values — and written into its slice of one store spanning every location across all cells. The measurements become (instance, element) arrays and the per-observation coordinates (e.g. time) are carried as matching 2-D arrays.

An empty store is created first with the requested chunk/shard layout, then each cell is written into a disjoint region of the instance dimension, so the full array is never held in memory and cells can be processed in parallel. Zarr compresses the fill padding on disk.

Parameters:

  • root_path (str or pathlib.Path) – Directory containing the contiguous ragged cell files (searched recursively).

  • store (str or pathlib.Path or MutableMapping) – Target Zarr store.

  • fn_pattern (str, optional) – Glob pattern for the cell files (default: “*.nc”).

  • count_var (str, optional) – Count variable name (default: “row_size”).

  • instance_dim (str, optional) – Instance dimension name (default: “locations”).

  • instance_id_var (str, optional) – Instance identifier variable (default: “location_id”).

  • element_dim (str, optional) – Name of the (positional) element dimension in the output (default: “element”).

  • trim (bool, optional) – Drop fill/padding locations when reading each cell (default: True).

  • chunks (dict, optional) – Chunk size per dimension, e.g. {"locations": 5000, "element": 500}. Dimensions not given are a single chunk.

  • shards (dict, optional) – Shard size per dimension (Zarr v3). Each shard size must be a multiple of the corresponding chunk size.

  • n_workers (int, optional) – Number of worker threads for reading/converting cells (default: 1). Writes to the store are serialised; the parallelism speeds up the per-cell conversion.

Returns:

store – The store that was written.

Return type:

str or pathlib.Path or MutableMapping

ascat.gridded_ragged_array.grid_registry = <ascat.gridded_ragged_array.GridRegistry object>

Shared grid registry used by the readers when a grid name (str) is passed.

ascat.h_saf module

Readers for H SAF soil moisture products.

class ascat.h_saf.AscatNrtBufrFileList(root_path, product_id='*', filename_template=None, subfolder_template=None)[source]

Bases: ChronFiles

Class reading ASCAT NRT BUFR files.

class ascat.h_saf.AscatSsmDataRecord(cdr_path, grid_path, fn_format=None, grid_filename='TUW_WARP5_grid_info_2_2.nc', static_layer_path=None, **kwargs)[source]

Bases: AscatGriddedNcTs

Class reading Metop ASCAT soil moisture data record.

class ascat.h_saf.H121Zarr[source]

Bases: object

Class reading ASCAT SSM CDR v8 12.5 km (H121) in zarr data format stored as incomplete multidimensional array representation.

This class is for testing purpose only.

read(*args)[source]

Read time series either by GPI (1 argument) or lon/lat (2 arguments).

Parameters:

  • gpi (int) – Grid point index.

  • or

  • lon (float) – Longitude in degrees.

  • lat (float) – Latitude in degrees.

Returns:

Time series data.

Return type:

pandas.DataFrame

read_gpi(gpi)[source]

Read time series for given grid point (Fibonacci 12.5 km).

Parameters:

gpi (int32) – Grid point index.

Returns:

df – Time series data.

Return type:

pandas.DataFrame

read_lonlat(lon, lat, max_dist=15000.0)[source]

Read the time series data for the grid point closest to the given lon/lat coordinates.

Parameters:

  • lon (float32) – Longitude coordinate.

  • lat (float32) – Latitude coordinate.

  • max_dist (float32) – Maximum searching distance.

Returns:

df – Time series data.

Return type:

pandas.DataFrame

class ascat.h_saf.H14Grib(filename, expand_grid=True, metadata_fields=['units', 'name'])[source]

Bases: Filenames

Class reading H14 soil moisture in GRIB format.

class ascat.h_saf.H14GribFileList(path)[source]

Bases: ChronFiles

Reads H SAF H08 data.

read_period(dt_end, delta)[source]

Read period not implemented.

ascat.ragged_array module

class ascat.ragged_array.ContiguousRaggedArray(ds: Dataset, count_var: str, instance_dim: str, instance_id_var: str = None)[source]

Bases: object

Contiguous ragged array representation (CF convention).

In an contiguous ragged array representation, the dataset for all time series are stored in a single 1D array. Additional variables or dimensions provide the metadata needed to map these values back to their respective time series.

The contiguous ragged array representation can be used only if the size of each instance is known at the time that it is created. In this representation the data for each instance will be contiguous on disk.

If the instance dimension exists as a variable, it is assumed that the values represent the identifiers for each instance otherwise they are count upwards from 0.

instance_dim

Name of the instance dimension.

Type:

str

sample_dim

Name of the sample dimension. The variable bearing the sample_dimension attribute (i.e. count_var) must have the instance dimension as its single dimension, and must have an integer type.

Type:

str

count_var

Name of the count variable. The count variable must be an integer type and must have the instance dimension as its sole dimension. The count variable are identifiable by the presence of an attribute, sample_dimension, found on the count variable, which names the sample dimension being counted.

Type:

str

ds

Contiguous ragged array dataset.

Type:

xarray.Dataset

instance_variables

Per-instance metadata variables (over the instance dimension).

Type:

list

observation_variables

Measurement variables (over the sample dimension).

Type:

list

instance_ids

List of instance ids.

Type:

list

sel_instance(i)[source]

Read time series for given instance.

iter()[source]

Yield time series for each instance.

append(other)[source]

Append another contiguous ragged array (in place).

The two collections are merged by instance: observations of shared instances are combined, and instances present in only one are added.

Parameters:

other (ContiguousRaggedArray or xarray.Dataset) – Contiguous ragged array to append (same structure).

apply(func)[source]

Apply a function to each instance’s time series.

property ds

Dataset.

Returns:

ds – Contiguous ragged array dataset.

Return type:

xr.Dataset

classmethod from_file(filename: str, count_var: str, instance_dim: str, instance_id_var: str = None, trim: bool = False, **kwargs)[source]

Load time series from file.

Parameters:

  • filename (str) – Filename.

  • count_var (str) – Count variable name.

  • instance_dim (str) – Instance dimension name.

  • instance_id_var (str, optional) – Variable used as instance identifier (default: None).

  • trim (bool, optional) – If True, drop fill/padding locations after loading (default: False).

Returns:

data – ContiguousRaggedArray object loaded from a file.

Return type:

ContiguousRaggedArray

get_observation_variables(include_dtype: bool = False) → list[source]

Observation variables, optionally with their dtypes.

Parameters:

include_dtype (bool, optional) – If True, return (name, dtype) tuples (default: False).

Returns:

observation_variables – Observation variable names (dimensioned by the sample dimension).

Return type:

list of str or list of (str, dtype)

property instance_ids: list

Instance ids

Returns:

instance_ids – Instance ids.

Return type:

list of int

property instance_variables: list

Instance variables (dimensioned by the instance dimension only, i.e. the per-instance metadata such as lon/lat/location_id).

Returns:

instance_variables – Instance variable names.

Return type:

list of str

iter()[source]

Explicit iterator method.

Returns:

ds – Time series for instance.

Return type:

xr.Dataset

property observation_variables: list

Observation variables (dimensioned by the sample dimension, i.e. the measurements that vary per observation).

Returns:

observation_variables – Observation variable names.

Return type:

list of str

save(filename: str)[source]

Write to file (“.nc” or “.zarr”) with CF metadata finalized.

sel_instance(i: int)[source]

Read time series

sel_instances(i: ndarray) → ContiguousRaggedArray[source]

Select several instances, preserving request order.

Parameters:

i (np.ndarray) – Array of instance IDs.

Returns:

data – A contiguous ragged array with the selected instances (in the order of i), or None if none of the ids are present.

Return type:

ContiguousRaggedArray or None

property size: int

Number of instances.

Returns:

instance_ids – Number of instance.

Return type:

int

to_incomplete()[source]

Convert to an incomplete multidimensional array (CF 9.3.2).

Each instance’s samples are packed into the leading columns of a dense (instance x element) array and the rest padded with fill values.

Returns:

data – Incomplete multidimensional array time series.

Return type:

IncompleteMultidimArray

to_indexed(index_var: str = 'locationIndex')[source]

Convert to indexed ragged array.

Parameters:

index_var (str, optional) – Name of the index variable to create (default: “locationIndex”).

Returns:

data – Indexed ragged array time series.

Return type:

IndexedRaggedArray

to_orthogonal(element_coord: str, element_dim: str = None, strict: bool = True)[source]

Convert to an orthogonal multidimensional array (CF 9.3.1).

All instances must share the same set of element_coord values (e.g. the same time axis); the samples are pivoted onto that shared coordinate.

Parameters:

  • element_coord (str) – Name of the per-sample coordinate defining the shared element axis.

  • element_dim (str, optional) – Name of the resulting element dimension (default: element_coord).

  • strict (bool, optional) – If True (default), raise if the instances do not form a complete grid on element_coord.

Returns:

data – Orthogonal multidimensional array time series.

Return type:

OrthogonalMultidimArray

to_point_data()[source]

Convert to point data.

Instance-level variables are broadcast to the sample dimension so that every observation carries its instance’s coordinates.

Returns:

data – Point data.

Return type:

PointData

trim()[source]

Drop fill/padding locations.

Cell files often over-allocate the instance dimension and pad the unused locations with fill values (a negative fill for the integer count variable). This removes those locations and keeps only the observations belonging to real ones. Locations with a valid count of zero are kept.

Returns:

data – Trimmed contiguous ragged array (self if nothing to trim).

Return type:

ContiguousRaggedArray

validate()[source]

Validate format.

class ascat.ragged_array.IncompleteMultidimArray(ds: Dataset, instance_dim: str = 'locations', element_dim: str = 'time', instance_id_var: str = None)[source]

Bases: MultidimArray

Incomplete multidimensional array representation (CF 9.3.2).

Instances may have different numbers of elements (and different element coordinates); the dense (instance, element) array is padded with the dtype fill value. The element dimension is a positional index.

classmethod from_file(filename: str, instance_dim: str = 'locations', element_dim: str = 'time', instance_id_var: str = None, **kwargs)[source]

Load an incomplete multidimensional array from a file.

Parameters:

  • filename (str) – Filename.

  • instance_dim (str, optional) – Instance dimension name (default: “locations”).

  • element_dim (str, optional) – Element dimension name (default: “time”).

  • instance_id_var (str, optional) – Variable used as instance identifier (default: None).

Returns:

data – Incomplete multidimensional array loaded from a file.

Return type:

IncompleteMultidimArray

class ascat.ragged_array.IndexedRaggedArray(ds: Dataset, index_var: str, sample_dim: str, instance_id_var: str = None)[source]

Bases: object

Indexed ragged array representation (CF convention).

In an indexed ragged array representation, the dataset is structured to store variable-length data (e.g., time series with varying lengths) compactly. To achieve this, auxiliary indexing variables that map the flat array storage to meaningful groups (e.g. locations).

If the instance dimension exists as a variable, it is assumed that the values represent the identfiers for each instance otherwise they counting upwards from 0.

index_var

The indexed ragged array representation must contain an index variable, which must be an integer type, and must have the sample dimension as its single dimension. The index variable can be identified by having an attribute ‘instance_dimension’ whose value is the instance dimension.

Type:

str

sample_dim

Name of the sample dimension. The sample dimension indicates the number of instances (e.g. stations, locations).

Type:

str

instance_dim

The name of the instance dimension. The value is defined by the ‘instance_dimension’ attribute, which must be present on the index variable. All variables having the instance dimension are instance variables, i.e. variables holding time series data.

Type:

str

ds

Indexed ragged array dataset.

Type:

xarray.Dataset

instance_variables

Per-instance metadata variables (over the instance dimension).

Type:

list

observation_variables

Measurement variables (over the sample dimension).

Type:

list

instance_ids

List of instance ids.

Type:

list

sel_instance(i)[source]

Read time series for given instance.

iter()[source]

Yield time series for each instance.

append(other)[source]

Append another indexed ragged array (in place).

The two collections are merged by instance (via a contiguous ragged array): observations of shared instances are combined, and instances present in only one are added.

Parameters:

other (IndexedRaggedArray or xarray.Dataset) – Indexed ragged array to append (same structure).

apply(func)[source]

Apply a function to each instance’s time series.

property ds: Dataset

Dataset.

Returns:

ds – Indexed ragged array dataset.

Return type:

xr.Dataset

classmethod from_file(filename: str, index_var: str, sample_dim: str, instance_id_var: str = None)[source]

Read data from file.

Parameters:

  • filename (str) – Filename.

  • index_var (str) – Index variable name.

  • sample_dim (str) – Sample dimension name.

  • instance_id_var (str, optional) – Variable holding the instance identifiers (default: None).

Returns:

data – IndexRaggedArray object loaded from a file.

Return type:

IndexRaggedArray

property instance_ids: list

Instance ids.

Returns:

instance_ids – Instance ids.

Return type:

list of int

property instance_variables: list

Instance variables (dimensioned by the instance dimension only, i.e. the per-instance metadata such as lon/lat/location_id).

Returns:

instance_variables – Instance variable names.

Return type:

list of str

iter() → Dataset[source]

Explicit iterator method.

Returns:

ds – Time series for instance.

Return type:

xr.Dataset

property observation_variables: list

Observation variables (dimensioned by the sample dimension, i.e. the measurements that vary per observation).

Returns:

observation_variables – Observation variable names.

Return type:

list of str

save(filename: str)[source]

Write data to file with CF metadata finalized.

Parameters:

filename (str) – Filename (“.nc” or “.zarr”).

sel_instance(i: int) → Dataset[source]

Read time series.

Parameters:

i (int) – Instance identifier.

Returns:

ds – Time series for instance, or None if the instance is not present.

Return type:

xr.Dataset or None

sel_instances(i: array, ignore_missing: bool = True) → IndexedRaggedArray[source]

Select several instances, preserving request order.

Parameters:

i (numpy.array) – Instance identifier(s).

Returns:

data – An indexed ragged array with the selected instances.

Return type:

IndexedRaggedArray

property size: int

Number of instances.

Returns:

instance_ids – Number of instance.

Return type:

int

to_contiguous(count_var: str = 'row_size') → ContiguousRaggedArray[source]

Convert to contiguous ragged array.

Parameters:

count_var (str, optional) – Count variable (default: “row_size”).

Returns:

data – Contiguous ragged array time series.

Return type:

ContiguousRaggedArray

to_incomplete() → IncompleteMultidimArray[source]

Convert to an incomplete multidimensional array (via a contiguous array).

Returns:

data – Incomplete multidimensional array time series.

Return type:

IncompleteMultidimArray

to_orthogonal(element_coord: str, element_dim: str = None, strict: bool = True) → OrthogonalMultidimArray[source]

Convert to an orthogonal multidimensional array (via a contiguous array).

Returns:

data – Orthogonal multidimensional array time series.

Return type:

OrthogonalMultidimArray

to_point_data()[source]

Convert to point data.

Instance-level variables are broadcast to the sample dimension so that every observation carries its instance’s coordinates.

Returns:

data – Point data.

Return type:

PointData

validate()[source]

Validate format.

class ascat.ragged_array.MultidimArray(ds: Dataset, instance_dim: str = 'locations', element_dim: str = 'time', instance_id_var: str = None)[source]

Bases: object

Base class for CF multidimensional array representations.

Holds a dense (instance, element) dataset and the behaviour shared by the orthogonal (CF 9.3.1) and incomplete (CF 9.3.2) representations. The conversions back to ragged/point form are defined by the subclasses, which differ in whether the element dimension is a shared coordinate axis (orthogonal) or a padded positional index (incomplete).

instance_dim

Name of the instance dimension.

Type:

str

element_dim

Name of the element dimension.

Type:

str

ds

Multidimensional array dataset.

Type:

xarray.Dataset

append(other)[source]

Append another multidimensional array (in place).

The two collections are merged by instance via a contiguous ragged array (observations of shared instances are combined, new instances added), then converted back to this representation.

Parameters:

other (MultidimArray or xarray.Dataset) – Multidimensional array to append (same structure).

apply(func)[source]

Apply a function to each instance time series.

property ds
property instance_ids

Instance ids.

property instance_variables: list

Instance variables (dimensioned by the instance dimension only, i.e. the per-instance metadata such as lon/lat/location_id).

Returns:

instance_variables – Instance variable names.

Return type:

list of str

iter()[source]

Explicit iterator method

save(filename: str)[source]

Write to file (“.nc” or “.zarr”) with CF metadata finalized.

sel_instance(i: int)[source]

Read time series for a given instance id.

sel_instances(i: ndarray)[source]

Select several instances, preserving request order.

Returns:

data – A multidimensional array with the selected instances, or None if none of the ids are present.

Return type:

MultidimArray or None

property size: int

Number of instances.

to_contiguous(count_var: str = 'row_size', sample_dim: str = 'obs')[source]

Convert to a contiguous ragged array.

Returns:

data – Contiguous ragged array time series.

Return type:

ContiguousRaggedArray

to_indexed(count_var: str = 'row_size', sample_dim: str = 'obs', index_var: str = 'locationIndex')[source]

Convert to an indexed ragged array (via a contiguous ragged array).

Returns:

data – Indexed ragged array time series.

Return type:

IndexedRaggedArray

to_point_data(count_var: str = 'row_size', sample_dim: str = 'obs')[source]

Convert to point data (via a contiguous ragged array).

Returns:

data – Point data.

Return type:

PointData

validate()[source]

Validate format.

class ascat.ragged_array.OrthogonalMultidimArray(ds: Dataset, instance_dim: str = 'locations', element_dim: str = 'time', element_coord: str = None, instance_id_var: str = None)[source]

Bases: MultidimArray

Orthogonal multidimensional array representation (CF 9.3.1).

All instances share the same element coordinate axis (e.g. one time axis), stored as a single shared 1-D coordinate; the array is complete (no padding).

Parameters:

element_coord (str, optional) – Name of the shared element coordinate variable (default: the element dimension name).

classmethod from_file(filename: str, instance_dim: str = 'locations', element_dim: str = 'time', element_coord: str = None, instance_id_var: str = None, **kwargs)[source]

Load an orthogonal multidimensional array from a file.

Parameters:

  • filename (str) – Filename.

  • instance_dim (str, optional) – Instance dimension name (default: “locations”).

  • element_dim (str, optional) – Element dimension name (default: “time”).

  • element_coord (str, optional) – Shared element coordinate variable name (default: element_dim).

  • instance_id_var (str, optional) – Variable used as instance identifier (default: None).

Returns:

data – Orthogonal multidimensional array loaded from a file.

Return type:

OrthogonalMultidimArray

class ascat.ragged_array.PointData(ds: Dataset, sample_dim: str)[source]

Bases: object

Point data represent scattered locations and times with no implied relationship among of coordinate positions, both data and coordinates must share the same (sample) instance dimension.

append(other)[source]

Append point data along the sample dimension (in place).

Parameters:

other (PointData or xarray.Dataset) – Point data to append. It must follow the same point structure (all variables over the sample dimension).

property ds
classmethod from_file(filename: str, sample_dim: str = None, **kwargs)[source]

Load point data from a file.

Parameters:

  • filename (str) – Filename.

  • sample_dim (str, optional) – Sample dimension name (inferred from the file if not given).

Returns:

data – Point data loaded from a file.

Return type:

PointData

save(filename: str)[source]

Write point data to file (“.nc” or “.zarr”) with CF metadata.

to_contiguous(count_var: str = 'row_size', instance_dim: str = 'locations')[source]

Convert point data to contiguous ragged array.

Parameters:

  • count_var (str) – Name of the new count variable to be added (default: ‘row_size’).

  • instance_dim (str) – Name of the instance dimension (default: ‘locations’).

Returns:

contiguous – Contiguous ragged array object.

Return type:

ContiguousRaggedArray

to_incomplete(count_var: str = 'row_size', instance_dim: str = 'locations')[source]

Convert point data to an incomplete multidimensional array (CF 9.3.2).

Parameters:

  • count_var (str, optional) – Name of the intermediate count variable (default: “row_size”).

  • instance_dim (str, optional) – Name of the instance dimension (default: “locations”).

Returns:

data – Incomplete multidimensional array time series.

Return type:

IncompleteMultidimArray

to_indexed(index_var: str = 'locationIndex', instance_dim: str = 'locations')[source]

Convert point data to indexed ragged array.

Parameters:

  • index_var (str) – Name of the new index variable to be added.

  • instance_dim (str) – Name of the instance dimension.

Returns:

indexed – Indexed ragged array object.

Return type:

IndexedRaggedArray

to_orthogonal(element_coord: str, count_var: str = 'row_size', instance_dim: str = 'locations', element_dim: str = None, strict: bool = True)[source]

Convert point data to an orthogonal multidimensional array (CF 9.3.1).

Parameters:

  • element_coord (str) – Name of the per-sample coordinate defining the shared element axis.

  • count_var (str, optional) – Name of the intermediate count variable (default: “row_size”).

  • instance_dim (str, optional) – Name of the instance dimension (default: “locations”).

  • element_dim (str, optional) – Name of the resulting element dimension (default: element_coord).

  • strict (bool, optional) – If True (default), raise if the instances do not form a complete grid on element_coord.

Returns:

data – Orthogonal multidimensional array time series.

Return type:

OrthogonalMultidimArray

validate()[source]

Validate format.

ascat.ragged_array.open_cf(source, instance_id_var: str = None, sample_dim: str = None, instance_dim: str = None, element_dim: str = None, element_coord: str = None, **kwargs)[source]

Open a CF discrete sampling geometry dataset or file and return the matching wrapper.

The representation is auto-detected with ascat.cf_conversions.detect_cf_representation(). Contiguous ragged, indexed ragged and point datasets are configured automatically from their CF marker attributes; multidimensional arrays require instance_dim and element_dim.

Parameters:

  • source (str, pathlib.Path or xarray.Dataset) – A file path, or an already-open dataset.

  • instance_id_var (str, optional) – Variable holding the instance identifiers (e.g. “location_id”).

  • sample_dim (str, optional) – Sample dimension name (point data; inferred if not given).

  • instance_dim (str, optional) – Instance / element dimension names (required for multidimensional arrays).

  • element_dim (str, optional) – Instance / element dimension names (required for multidimensional arrays).

  • element_coord (str, optional) – Shared element coordinate name (orthogonal arrays; default element_dim).

  • **kwargs – Passed to xarray.open_dataset() when source is a path.

Returns:

data – The wrapper matching the detected representation.

Return type:

PointData, ContiguousRaggedArray, IndexedRaggedArray, OrthogonalMultidimArray or IncompleteMultidimArray

ascat.ragged_array.verify_contiguous_ragged(ds: Dataset, count_var: str, instance_dim: str) → None[source]

Verify dataset follows contiguous ragged array CF definition.

Parameters:

  • ds (xarray.Dataset) – Dataset to be verified.

  • count_var (str) – Name of the count variable. Count variable contains the length of each time series feature. It is identified by having an attribute with name ‘sample_dimension’ whose value is name of the sample dimension. The count variable implicitly partitions into individual instances all variables that have the sample dimension.

Raises:

RuntimeError if verification fails. –

ascat.ragged_array.verify_indexed_ragged(ds: Dataset, index_var: str, sample_dim: str) → None[source]

Verify dataset follows indexed ragged array CF definition.

Parameters:

  • ds (xarray.Dataset) – Dataset.

  • index_var (str) – The index variable can be identified by having an attribute with name of instance_dimension whose value is the instance dimension.

  • sample_dim (str) – Name of the sample dimension.

Raises:

RuntimeError if verification fails. –

ascat.ragged_array.verify_multidim(ds: Dataset, instance_dim: str, element_dim: str) → None[source]

Verify a dataset follows the CF multidimensional array definition (orthogonal or incomplete).

Parameters:

  • ds (xarray.Dataset) – Dataset to be verified.

  • instance_dim (str) – Name of the instance dimension.

  • element_dim (str) – Name of the element dimension.

Raises:

RuntimeError if verification fails. –

ascat.ragged_array.verify_point_array(ds: Dataset, sample_dim: str) → None[source]

Verify dataset follows the CF point data array convention.

Parameters:

  • ds (xarray.Dataset) – Dataset to be verified.

  • sample_dim (str) – Name of the sample dimension.

Raises:

RuntimeError if verification fails. –

ascat.swath module

class ascat.swath.Swath(filenames)[source]

Bases: Filenames

Class to read and merge swath files given one or more file paths.

static combine_attributes(attrs_list, context)[source]

Decides which attributes to keep when merging swath files.

Parameters:
read(parallel=False, mask_and_scale=True, **kwargs)[source]

Read the file or a subset of it.

Parameters:

  • parallel (bool, optional) – If True, read files in parallel.

  • mask_and_scale (bool, optional) – If True, mask and scale the data.

  • kwargs (dict) – Additional keyword arguments passed to Filenames.read.

Returns:

ds – Dataset.

Return type:

xarray.Dataset

class ascat.swath.SwathGridFiles(root_path, fn_templ, sf_templ, grid_name, date_field_fmt, cell_fn_format=None, cls_kwargs=None, err=True, fn_read_fmt=None, sf_read_fmt=None, fn_write_fmt=None, sf_write_fmt=None, preprocessor=None, postprocessor=None, cache_size=0)[source]

Bases: ChronFiles

Class to manage chronological swath files with a date field in the filename.

classmethod from_product_class(path, product_class)[source]

Create a SwathGridFiles from a given io_class.

Returns a SwathGridFiles object initialized with the given io_class.

Parameters:

  • path (str or Path) – Path to the swath file collection.

  • io_class (class) – Class to use for reading and writing the swath files.

Examples

>>> my_swath_collection = SwathFileCollection.from_io_class(
...     "/path/to/swath/files",
...     AscatH129Swath,
... )
classmethod from_product_id(path, product_id)[source]

Create a SwathGridFiles object based on a product_id.

Returns a SwathGridFiles object initialized with an io_class specified by product_id (case-insensitive).

Parameters:

  • path (str or Path) – Path to the swath file collection.

  • product_id (str) – Identifier for the specific ASCAT product the swath files are part of.

Raises:

ValueError – If product_id is not recognized.

Examples

>>> my_swath_collection = SwathFileCollection.from_product_id(
...     "/path/to/swath/files",
...     "H129",
... )
read(date_range, dt_delta=None, search_date_fmt='%Y%m%d*', date_field='date', end_inclusive=True, cell=None, location_id=None, coords=None, max_coord_dist=None, bbox=None, geom=None, read_kwargs=None, **fmt_kwargs)[source]

Extract data from swath files within a time range and spatial criterion.

Parameters:

  • date_range (tuple of datetime.datetime) – Start and end date.

  • dt_delta (timedelta) – Time delta.

  • search_date_fmt (str) – Search date format.

  • date_field (str) – Date field.

  • end_inclusive (bool) – If True (default), include data from the end date in the result. Otherwise, exclude it.

  • cell (int or list of int) – Grid cell number to read.

  • location_id (int or list of int) – Location id to read.

  • coords (tuple of numeric or tuple of iterable of numeric) – Tuple of (lon, lat) coordinates to read.

  • max_coord_dist (float) – Maximum distance in meters to search for grid points near the given coordinates. If None, the default is np.inf.

  • bbox (tuple) – Tuple of (latmin, latmax, lonmin, lonmax) coordinates to bound the data.

  • geom (shapely.geometry) – Geometry to bound the data.

Returns:

Dataset.

Return type:

xarray.Dataset

stack_to_cell_files(out_dir, max_nbytes, date_range=None, fmt_kwargs=None, cells=None, print_progress=True, parallel=True)[source]

Stack all swath files to cell files, writing them in parallel.

Parameters:

  • out_dir (str) – Output directory.

  • max_nbytes (int) – Maximum number of bytes to open as xarray datasets before dumping to disk.

  • date_range (tuple of datetime.datetime, optional) – Start and end date for the search.

  • fmt_kwargs (dict, optional) – Additional keyword arguments passed to ascat.file_handling.ChronFiles.search_period.

  • cells (list of int, optional) – List of grid cell numbers to read. If None (default), all cells are read.

  • print_progress (bool, optional) – If True (default), print progress bars.

  • parallel (bool, optional) – If True, write data to files in parallel (use all available resources).

Search for swath files within a time range and spatial criterion.

Parameters:

  • dt_start (datetime) – Start date.

  • dt_end (datetime) – End date.

  • dt_delta (timedelta) – Time delta.

  • search_date_fmt (str) – Search date format.

  • date_field (str) – Date field.

  • end_inclusive (bool) – End date inclusive.

  • cell (int or list of int) – Grid cell number to read.

  • location_id (int or list of int) – Location id.

  • coords (tuple of numeric or tuple of iterable of numeric) – Tuple of (lon, lat) coordinates.

  • bbox (tuple) – Tuple of (latmin, latmax, lonmin, lonmax) coordinates.

  • geom (shapely.geometry) – Geometry.

  • fmt_kwargs (dict) – Additional keyword arguments passed to ascat.file_handling.ChronFiles.search_period.

Returns:

Filenames.

Return type:

list of str

ascat.utils module

class ascat.utils.Spacecraft(name)[source]

Bases: object

Spacecraft class.

valid_spacecraft_names = ['METOPA', 'METOPB', 'METOPC', 'METOP-A', 'METOP-B', 'METOP-C', 'METOP-SG B1', 'METOP-SG B2', 'METOP-SG B3']
ascat.utils.append_to_netcdf(filename, ds_to_append, unlimited_dim)[source]

Appends an xarray dataset to an existing netCDF file along a given unlimited dim.

Parameters:

  • filename (str or Path) – Filename of netCDF file to append to.

  • ds_to_append (xarray.Dataset) – Dataset to append.

  • unlimited_dim (str or list of str) – Name of the unlimited dimension to append along.

Raises:

ValueError – If more than one unlimited dim is given.

ascat.utils.boxcar(radius, distance)[source]

Boxcar filter

Parameters:

n (int) – Length.

Returns:

  • weights (numpy.ndarray) – Distance weights.

  • tw (float32) – Sum of weigths.

ascat.utils.create_variable_encodings(ds, custom_variable_encodings=None, custom_dtypes=None)[source]

Create an encoding dictionary for a dataset, optionally overriding the default encoding or adding additional encoding parameters. New parameters cannot be added to default encoding for a variable, only overridden.

E.g. if you want to add a “units” encoding to “lon”, you should also pass “dtype”, “zlib”, “complevel”, and “_FillValue” if you don’t want to lose those.

Parameters:

  • ds (xarray.Dataset) – Dataset.

  • custom_variable_encodings (dict, optional) – Custom encodings.

Returns:

ds – Dataset with encodings.

Return type:

xarray.Dataset

ascat.utils.daterange(start_date, end_date)[source]

Generator for daily datetimes.

Parameters:

  • start_date (datetime) – Start date.

  • end_date (datetime) – End date.

ascat.utils.db2lin(val)[source]

Converting from linear to dB domain.

Parameters:

val (numpy.ndarray) – Values in dB domain.

Returns:

val – Values in linear domain.

Return type:

numpy.ndarray

ascat.utils.get_bit(a, bit_pos)[source]

Returns 1 or 0 if bit is set or not.

Parameters:

  • a (int or numpy.ndarray) – Input array.

  • bit_pos (int) – Bit position. First bit position is right.

Returns:

b – 1 if bit is set and 0 if not.

Return type:

numpy.ndarray

ascat.utils.get_file_format(filename)[source]

Try to guess the file format from the extension.

Parameters:

filename (str) – File name.

Returns:

file_format – File format indicator.

Return type:

str

ascat.utils.get_grid_gpis(grid, cell=None, location_id=None, coords=None, bbox=None, geom=None, max_coord_dist=inf, return_lookup: bool = False)[source]

Get grid point indices.

Parameters:

  • grid (pygeogrids.CellGrid) – Grid object.

  • cell (int or iterable of int, optional) – Cell number(s).

  • location_id (int or iterable of int, optional) – Location ID.

  • coords (tuple, optional) – Tuple of (lon, lat) coordinates.

  • bbox (tuple, optional) – Tuple of (latmin, latmax, lonmin, lonmax) coordinates.

  • geom (shapely.geometry.BaseGeometry, optional) – Geometry object.

  • max_coord_dist (float, optional) – Maximum distance from coordinates to return a gpi.

Returns:

  • gpi (int) – Grid point index.

  • lookup_vector (numpy.ndarray) – Lookup vector. (only if return_lookup is True)

ascat.utils.get_roi_subset(ds, roi)[source]

Filter dataset for given region of interest.

Parameters:

  • ds (xarray.Dataset) – Dataset to be filtered for region of interest.

  • roi (tuple of 4 float) – Region of interest: latmin, lonmin, latmax, lonmax

Returns:

ds – Filtered dataset.

Return type:

xarray.Dataset

ascat.utils.get_toi_subset(ds, toi)[source]

Filter dataset for given time of interest.

Parameters:

  • ds (xarray.Dataset) – Dataset to be filtered for time of interest.

  • toi (tuple of datetime) – Time of interest.

Returns:

ds – Filtered dataset.

Return type:

xarray.Dataset

ascat.utils.get_window_radius(window, hp_radius)[source]

Calculates the required radius of a window function in order to achieve the provided half power radius.

Parameters:

  • window (string) –

    Window function name. Current supported windows:

    • Hamming

    • Boxcar

  • hp_radius (float32) – Half power radius. Radius of window function for weight equal to 0.5 (-3 dB). In the spatial domain this corresponds to half of the spatial resolution one would like to achieve with the given window.

Returns:

r – Window radius needed to achieve the given half power radius

Return type:

float32

ascat.utils.get_window_weights(window, radius, distance, norm=False)[source]

Function returning weights for the provided window function

Parameters:

  • window (str) – Window function name

  • radius (float) – Radius of the window.

  • distance (numpy.ndarray) – Distance array

  • norm (boolean) – If true, normalised weights will be returned.

Returns:

weights – Weights according to distances and given window function

Return type:

numpy.ndarray

ascat.utils.gpis_to_lookup(grid, gpis)[source]

Create lookup vector from grid point indices.

Parameters:

  • grid (pygeogrids.BasicGrid) – Grid object.

  • gpis (numpy.ndarray) – Grid point indices.

Returns:

lookup_vector – Lookup vector.

Return type:

numpy.ndarray

ascat.utils.hamming_window(radius, distances)[source]

Hamming window filter.

Parameters:

  • radius (float32) – Radius of the window.

  • distances (numpy.ndarray) – Array with distances.

Returns:

  • weights (numpy.ndarray) – Distance weights.

  • tw (float32) – Sum of weigths.

ascat.utils.lin2db(val)[source]

Converting from linear to dB domain.

Parameters:

val (numpy.ndarray) – Values in linear domain.

Returns:

val – Values in dB domain.

Return type:

numpy.ndarray

ascat.utils.mask_dtype_nans(ds)[source]

Mask NaNs in a dataset based on the dtypes of its variables.

ascat.utils.set_bit(a, bit_pos, value=1)[source]

Set bit at given position.

Parameters:

  • a (int or numpy.ndarray) – Input array.

  • bit_pos (int) – Bit position. First bit starts right.

  • value (1 or 0, optional) – Set bit either to 1 or 0 (default: 1).

Returns:

a – Modified input array with bit=value.

Return type:

numpy.ndarray

ascat.utils.tmp_unzip(filename)[source]

Unzip file to temporary directory.

Parameters:

filename (str) – Filename.

Returns:

unzipped_filename – Unzipped filename

Return type:

str

Module contents