Data recipes

OceanVal provides built-in recipes for popular observational datasets. These work for both gridded and point datasets, and cover variables including temperature, salinity, oxygen, nitrate, and chlorophyll.

Gridded data recipes

Downloading and wrangling metadata is tedious — recipes take care of it for you. Use the recipe argument with oceanval.add_gridded_comparison, click any row for the exact call to use. There is also a point data recipe.

oceanval.add_gridded_comparison(
    name="temperature",          # report name
    model_variable="thetao",      # model NetCDF variable
    recipe={"temperature": "cobe2"}, # observation recipe
    climatology=True,              # climatological comparison
)

OceanVal downloads the observational data automatically when oceanval.matchup is called — in most cases via THREDDS servers, so only what's needed is downloaded. Model and observations are then regridded to a common grid and averaged per month and year, where appropriate.

Every example below, in one file — a starting point you can copy the blocks you need from, rather than clicking through the table row by row.

oceanval.add_gridded_comparison(
    name="alkalinity",
    model_variable="talk",
    recipe={"alkalinity": "glodap"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="chlorophyll",
    model_variable="chl",
    recipe={"chlorophyll": "occci"},
    climatology=False,
)

oceanval.add_gridded_comparison(
    name="kd490",
    model_variable="kd490",
    recipe={"kd490": "occci"},
    climatology=False,
)

oceanval.add_gridded_comparison(
    name="nitrate",
    model_variable="no3",
    recipe={"nitrate": "woa23"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="oxygen",
    model_variable="o2",
    recipe={"oxygen": "woa23"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="ph",
    model_variable="ph",
    recipe={"ph": "glodap"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="phosphate",
    model_variable="po4",
    recipe={"phosphate": "woa23"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="salinity",
    model_variable="so",
    recipe={"salinity": "woa23"},
    start=2005, end=2014,
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="silicate",
    model_variable="si",
    recipe={"silicate": "woa23"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="temperature",
    model_variable="thetao",
    recipe={"temperature": "cobe2"},
    climatology=False,
)

oceanval.add_gridded_comparison(
    name="temperature",
    model_variable="thetao",
    recipe={"temperature": "woa23"},
    start=2005, end=2014,
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="ammonium",
    model_variable="ammonium",
    recipe={"ammonium": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="chlorophyll",
    model_variable="chlorophyll",
    recipe={"chlorophyll": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="nitrate",
    model_variable="nitrate",
    recipe={"nitrate": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="oxygen",
    model_variable="oxygen",
    recipe={"oxygen": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="phosphate",
    model_variable="phosphate",
    recipe={"phosphate": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="salinity",
    model_variable="salinity",
    recipe={"salinity": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="silicate",
    model_variable="silicate",
    recipe={"silicate": "nsbc"},
    climatology=True,
)

oceanval.add_gridded_comparison(
    name="temperature",
    model_variable="temperature",
    recipe={"temperature": "nsbc"},
    climatology=True,
)

Point data recipes

Point recipes are used with oceanval.add_point_comparison in place of obs_path. Instead of reading csv files from disk, OceanVal downloads the observations when oceanval.matchup runs, for the years being matched and the lon_lim/lat_lim area given to matchup. If no lon_lim and lat_lim are given, the model's own longitude/latitude extent is used. See how ICES data is treated for what is kept.

oceanval.add_point_comparison(
    model_variable="votemper",         # model NetCDF variable
    recipe={"temperature": "ices"},    # observation recipe
    vertical=True,                     # also validate below the surface
)

How ICES data is treated

What is downloaded. For each comparison, OceanVal asks ICES for an export of every profile in the lon_lim/lat_lim area (or the model’s own extent) in the years where the simulation and the comparison’s start and end overlap. Everything but temperature and salinity comes from the bottle and low resolution CTD data. Temperature and salinity take two exports, one of the high resolution CTD profiles and one of the bottle and low resolution CTD data, which are joined (see Profiles in both datasets below). An export holds every variable measured at its profiles, so large areas or long periods make for a large download.

Which values are used. Each variable is read from one ICES parameter code. ICES splits some analytes over several codes, by method, and only the code below is used: nitrate is nitrate alone, for example, not nitrate+nitrite.

Profiles in both datasets. Bottles are fired on CTD casts, so a cast can be in both the CTD and the bottle data, with the same values. For temperature and salinity, bottle values at a station the CTD data already has (the same date, and the same position to 0.01°) are left out, so each cast is counted once, and matchup says how many were. The bottle data adds the stations without a CTD profile, which are many in older years.

Quality flags. A value is kept only if its own quality flag and the flag of its depth are both 0 (good). Values flagged 1 (unknown quality), 4 (questionable) or 8 (bad) are left out. Recent years have many values flagged 1, so they can have few observations, or none.

Depth. The depth is ICES’s adopted depth (ADEPZZ01), in metres. Observations at 5 m or less make up the surface layer. Through the water column, an observation above the centre of the model’s top cell, as many bottle values at 0 m are, is compared with the top cell.

Time. Only the date of an observation is used, not the time of day. Times without a time zone are taken as UTC. A time written as hour 24 (1981-04-27T24) is 00:00 of the next day, as ISO 8601 has it. An observation dated only to the month or the year, or whose time cannot be read, is left out rather than given a day, and matchup says how many were.

Values. Values are used as ICES gives them, in the units in the table above. OceanVal does not screen them for range or remove duplicates, beyond the quality flags and the repeated stations above. ICES revises its data, so a later download can differ.

oceanval.add_gridded_comparison(
    name="temperature",
    model_variable="thetao",
    recipe={"temperature": "woa23"},
    start=1995, end=2004,
    climatology=True,
)

oceanval.matchup(sim_dir="/path/to/model/output", start=1995, end=2004)
oceanval.validate()

Your own recipes

A recipe describes a set of observations once, so that you can use it by name in any matchup, like OceanVal’s own. Your recipes are kept in .oceanvalrc files, which are JSON: .oceanvalrc in the directory you run OceanVal from holds the recipes for that directory only, and .oceanvalrc in your home directory holds the recipes for everywhere (the OCEANVALRC environment variable names another file to use in its place). Where the same recipe is in both, the one in the directory is used.

A recipe is used by the name of its source, in lower case: a source called MySat, for chlorophyll, is recipe={"chlorophyll": "mysat"}, with add_gridded_comparison or add_point_comparison as for any other recipe. The variable can be one of OceanVal’s, or a new one.

Registering a recipe

Choose Add your own validation data for future use on the first page of the oceanval window, then point or gridded data, and describe it.

• Gridded data is netCDF data, on this machine (a file, a directory of files, or a pattern ending in .nc such as obs/chl_*.nc), on a THREDDS server (OPeNDAP addresses, with dodsC in them) or at a web address, which is downloaded when it is matched up.
• Point data is a directory of csv files on this machine, with lon, lat and observation columns and any of year, month, day, depth and source.

OceanVal opens the data as you describe it, as a matchup will, and checks it: that the variable is in the netCDF files, that the years fit what you say about a climatology, that the csv columns can be used, and that a server answers. What it works out, such as the units, it fills in in red and bold. A recipe is only saved once its data has been checked, so one on a server that cannot be reached is not saved. You choose whether to save it for this directory only or for everywhere. OceanVal refuses a source name that is one of its own recipes, or is already in the file, and warns you if the other file has a recipe of the same name.

Data you add in the Own data step of a matchup can be saved as a recipe there too: Add this data for use now and in future adds it to the matchup, as Add this data for use now only does, and saves it as a recipe. A page first asks for what a recipe needs that the form did not: the units, where to save it and, if you left them out, the source information and the variable’s names in the report. What OceanVal will use unless you change it is filled in in amber: the units in the netCDF file, “Source for” and the source’s name, the variable’s name, and everywhere (your home directory). The units of csv files, which do not say, and of a netCDF variable without them, are marked in red, and have to be given. Data on a server is checked first, as above. In the matchup it is saved from, it is registered as your own data, so that matchup’s recipes window shows it in its variable’s row, ticked and tagged Your data, in place of its recipe.

Using them

A saved recipe is offered in the recipes window beside OceanVal’s own, with a Yours tag, ticked wherever a model variable is found for it. For a variable OceanVal has no recipes for, it can only identify the model variable if exactly one is named for it, or has its description in its long_name; otherwise the model variable is left blank for you to fill in. The matchup script registers it with a recipe= argument, and the units step lists its units like those of any other recipe.

The file

{
  "version": 1,
  "recipes": {
    "chlorophyll": {
      "mysat": {
        "kind": "gridded",
        "source": "MySat",
        "source_info": "My satellite chlorophyll, v2",
        "location": "thredds",
        "obs_path": ["https://example.org/thredds/dodsC/chl/2010.nc"],
        "obs_variable": "chl",
        "climatology": false,
        "depth_resolved": false,
        "units": "mg m-3"
      }
    }
  }
}

A point recipe has "kind": "point", with obs_path the directory of csv files and no location, obs_variable or climatology. A recipe for a variable of your own also has long_name, short_name and short_title, which name it in the report. You can edit the files by hand: OceanVal ignores, and says so, any file or recipe in them that it cannot use. The file in your home directory also keeps the oceanval window’s settings, under "settings" beside "recipes"; saving either keeps the other.

The Units column of the gridded and point recipes tables gives the units of the observations as the dataset's files state them. If your model's units differ, pass obs_multiplier and obs_adder to add_gridded_comparison or add_point_comparison (the observations become observations × obs_multiplier + obs_adder); the oceanval window's Units step lists both sets of units, fills these in where it can tell how to convert one into the other, and writes them for you. Always check dataset units and the climatology period before comparing with model output — see How to use OceanVal in Python for matching and time-resolution guidance.