Page tree

GRIB-API support is being discontinued at the end of 2018. Please consider upgrading to ecCodes


Skip to end of metadata
Go to start of metadata

There is a separate GRIB API Installation FAQ for questions related to the installation of grib_api.

What are namespaces?

A namespace in GRIB API is just a fancy word for a grouping of related keys. The following are available:

  • ls: This is the namespace used by the grib_ls tool and has the most-commonly used keys e.g. centre, shortName, level etc
  • parameter: Contains keys like paramId, shortName, units which relate to the meteorological parameter
  • statistics: Contains keys that relate to the statistics of the data values e.g. maximum, minimum, average, standard deviation etc
  • time: Contains keys that describe the forecast runs e.g. forecast date, validity date, steps etc
  • geography: Contains keys that describe the grid geometry e.g. bounding box of the grid, number of points along a parallel etc
  • vertical: Contains keys that describe the levels and layers e.g. type of the level, list of coefficients of the vertical coordinate etc
  • mars: Contains the list of MARS (ECMWF's Meteorological Archive and Retrieval System) keywords like class, stream, type etc

Why am I not able to set the missing value in the GRIB message?

Missing values cannot be encoded in a GRIB message. The GRIB format does keep track of missing values but through the use of a bitmap  It does not allow the specification of a missing value. Setting the missing value is a feature that can be used only when reading the data values stored in a GRIB message. This of course means that is the responsibility of the user to know what missing value is meaningful to the data being read. A default value of 9999 is set for the missing value in the library (not the GRIB message!). That means that when retrieving the values from a message without having set the missing value key, all missing values in the data will be replaced with the default value of 9999.
A small example on the use of the missing value during encoding can be found here: set_bitmap.f90.
During decoding it is advisable to query the bitmap directly to discover missing data values. See example here: get_data.f90

How do I append messages to an existing GRIB file?

Try using grib_open_file (Fortran interface) with mode "a" instead of "w". The "a" mode means: Append to a file. Writing operations append data at the end of the file. The file is created if it does not exist.

How do I copy a selected number of messages from a GRIB file?

Say you want to copy the first 3 messages. This can be done with:

% grib_copy -w count=1/2/3 in.grib out.grib

But it is better to write a rules file and use grib_filter:

my.filter
if (count < 4) {
  print "Copying message number [count]";
  write;
}

Then run this as follows:

% grib_filter -o out.grib my.filter in.grib

Of course you can make the IF condition more complicated e.g. select a range of messages to be copied etc

How can I change from second_order to simple packing?

You may be tempted to try it like this:

% grib_set -s packingType=grid_simple second_order.grib simple.grib

But this will not work and issue errors re codedValues and bitsPerValue! You need to use the repacking option as shown:

% grib_set -r -s packingType=grid_simple second_order.grib simple.grib

Also see the code example: precision.f90

What happens when I set the packingType to "grid_jpeg" on a GRIB1 message?

Actually nothing! The JPEG packing is not supported for GRIB edition 1 but rather than fail, the library leaves the message as is. This was designed for cases where a file (or set of files) containing a mix of grib1 and grib2 messages is being processed to use JPEG encoding. We convert grib2 messages and leave the grib1 messages alone

How do I know what the native type of a key value is?

For the moment there is no simple way of knowing what the type of a key value is. However, the library provides a function that can be used to find out what the native type of a key value is. The function, available only in the C API, is called 'grib_get_native_type'. Possible key value types are the following: undefined, long, double, string, bytes, section, label or missing. It is not possible at the moment to know if the value of a key is of type scalar or vector.

How do I use an OR condition (logical disjunction) in a "where" clause?

For example in grib_get you may want to show all messages which have level of 100, 150 or 200. The way to do this is to use the "/" (forward slash) character in the "where" clause (for the -w switch)

% grib_get -w level=100/150/200 ...

This can be combined with an "AND" condition (logical conjunction). So if I want to show all messages whose level is either 0 or 1 and whose type is "an":

% grib_ls -w level=0/1,dataType=an ...

This would be equivalent to the pseudo-code:

if (dataType == "an" AND (level == 0 OR level == 1)) ...

grib_filter: How can I check if a key exists (is defined) in a message?

Easy peasy. Use the "defined" function which takes a single key name as argument. e.g.

grib_filter example
if (defined(Latin1)) {
    print "Key Latin1 is there for this message";
}

How do I know if I am dealing with a multi-field grib file?

At the moment, the only possible way of detecting if a grib file is multi-field is by using the grib tools or the API to turn the support for multiple fields in a single grib message on or off and observe the results. For example you can do a "grib_count" and compare the result with "grib_ls", in a multi-field grib file the latter will report more messages than the former.

The use of the multi-field feature is not recommended.


Why can I read certain parameters using the grib tools, but not from my program?

Your file may be encoded as a multi-field grib file. By default, grib tools have multi-field support enabled and the API disabled. Check the examples to know how to enable this feature on your program:

Confused about stepUnits?

The key stepUnits is a transient key and it is not written in the grib message. So, you cannot expect to get the same stepUnits as when you have encoded the data. By convention, when encoding, GRIB API will choose the best unit to encode the step in so that it fits the space available in the grib message. When decoding, GRIB API will return the step in hours by default.

The stepUnits can only be set in the following situations:

  • before encoding, in order to tell GRIB API what units are we dealing with
  • when decoding, in order to tell GRIB API what units to get the step in

Examples:

   grib_ls and grib_get will always return the step in hours as one cannot set the units before printing

   % grib_set -s step=3600 file.grib out.grib 
   % grib_get -p startStep,endStep,stepRange,stepUnits:s,stepType out.grib
   3600 3600 3600 h instant 
   % grib_set -s stepUnits=m,step=3600 file.grib out.grib 
   % grib_get -p startStep,endStep,stepRange,stepUnits:s,stepType out.grib
   60 60 60 h instant

   To get the step in the units we want, one could use grib_filter like this:

   % cat step.filter
   set stepUnits="m";
   print "[startStep] [endStep] [stepRange] [stepUnits:s] [stepType]";
   
   % grib_filter step.filter out.grib
   3600 3600 3600 m instant

Starting with GRIB API version 1.9.0 one can set values in a grib message before printing its contents with tools like grib_ls and grib_get.

  % grib_ls -s stepUnits="m" -p startStep,endStep,stepRange,stepUnits:s,stepType out.grib

Confused about key types?

Some keys like shortName are strings (native type string) and it doesn't make any sense to set them as an integer or float as they are not possible values for the key.

Some keys are native type integer like number and you can set/get them as integer, float or string as you like.

Some keys are codetables and for them you have a code (integer) and an abbreviation (string) like for example "centre". You can set them as a string or as an integer, not as a float because there is no mapping for those keys into a float. An example is:

% grib_set -s centre:s=ecmwf in.grib out.grib
% grib_set -s centre:i=98 in.grib out.grib

 

Some keys are concepts and they are usually strings like stepType, gridType, packingType. Again setting them as integer doesn't make any sense as they don't have any mapping as integers.

A concept is a key with a special association with other keys. Let's take shortName as an example. Its definition in the .def files is made with a long list of entries like:

'cape' = {
     discipline = 0 ;
     parameterCategory = 7 ;
     parameterNumber = 6 ;
     typeOfFirstFixedSurface = 1 ;
     typeOfSecondFixedSurface = 8 ;
}

this means that if you set shortName="cape" ( as a string ) GRIB API will automatically set the keys listed in curly brackets to the corresponding values.

Conversely when you get shortName grib_api is looking for the best match of the keys listed in the full definition of shortName (not only cape) and it is returning "cape" only if

discipline == 0
parameterCategory == 7
parameterNumber == 6
typeOfFirstFixedSurface == 1
typeOfSecondFixedSurface == 8

in the grib and there isn't a better match of keys in the list.

Again saying shortName=123 doesn't make any sense as in the context of shortName we don't need numbers.

The idea is that the user should know what to set before choosing the type to use. In the sense that the user should know the meaning of the key before setting it to a value. At this purpose we have built the documentation and I agree that we need to do a big review (with your help) of it to fix all the wrong things and to add what is missing.

There was a bit of confusion regarding the type of step and paramId for the following reason:

step

For me step was a string because you can have values like 24-36 which you cannot represent as numbers. In mars the step is only the endStep and therefore I had to modify grib_api to be mars compliant. Now the step is an integer and is step=endStep. As we need a step indicating a range I have introduced stepRange which is a string and cannot be set as an integer and will never be set as an integer because it allows values like stepRange=24-36. It is true that most of the time you have stepRange=36 or stepRange=24 or another single number, but this is because in those cases you have also stepType=instant and endStep=startStep and we don't want to write stepRange=24-24 which is only redundant. We also use to write stepRange=24 when stepRange=0-24 (mars compatibility which I don't like). If you prefer to set the step as an integer instead of using a string you have starStep, endStep which are integers and they can also be set as string because it is possible to convert for example the number 24 into the string "24". Therefore you can do:

% grib_set -s endStep=24 in out
% grib_set -s endStep:s=24 in out
 

paramId

It is a concept like shortName and unfortunately in the previous version the native type for a concept was string even if the meaning of it is number. In the new version paramId is still a concept, but it has native type number. This means that it can be set as a number, but cannot be set as a string because only the numbers are valid values for paramId. It can still be get as a string because it is possible to convert a number into a string. You cannot set as a string because it isn't always possible to convert a string into a number.

Why I cannot set negative values for the longitude in GRIB 2

GRIB 1 regulates that the longitude can be in either [-180, 180] or [0,360], GRIB 2 regulates that the longitude can only be in the interval [0,360]. GRIB API does comply with these regulations and in the case of GRIB 2, it will scale the longitude to fall in the interval [0,360]. GRIB API does not offer an edition independent view of the longitude because of the uncertainty brought by GRIB 1, where you do not know what interval a longitude is in exactly.

For GRIB edition 1, why does the centre change when I set stepType?

This happens for stepType of "max" and "min".  Unfortunately edition 1 does not support maximum and minimum in its "Time Range Indicator" table (Table 5). So we invented our own centre-specific combination to support this. There is no such issue in GRIB edition 2.

Why is the "bottomLevel" of Soil Temperature level 4 reported as MISSING?

The parameter 236 (Soil temperature level 4) specifies that its top level is 100cm and its bottom level is 289cm so to encode this information in GRIB edition 1 we would need to set the value of the key "bottomLevel" to 289 (octet 12 in section 1. See GRIB edition 1 section 1 spec).
However since this key is only one octet it cannot accommodate any value larger than 255! Therefore it is not possible to encode levels larger than that value in GRIB1 so we simply set all the bits to 1 (which means MISSING).

GRIB edition 2 addresses this shortcoming.

Possible GOTCHA with wave spectra fields

[ Courtesy of Paul Dando ] I've just been investigating an issue with the decoding of the wave spectra fields with the new default version of GRIB API (version 1.9.16).  I want to make you all aware of this in case any of your users hit the same problem.
In the wave spectra, all frequency/direction components with values less than some (field-dependent) threshold are set to 'missing' when encoded in GRIB;  values larger than this threshold are encoded as log10 of the actual value to an accuracy of 9 bits (bitsPerValue=9).

With the previous GRIB API version 1.9.9, grib_get_data (and the grib_get_data and grib_get of the 'values' key API functions) returned a zero for the 'missing' values:

% grib_get_data -m MISSING  spec.grb

Latitude, Longitude, Value
   51.500    2.500 0.0000000000e+00
Latitude, Longitude, Value
   51.500    2.500 -1.4684238434e+00

With the current GRIB API 1.9.16, grib_get_data correctly returns missing for these values:

% grib_get_data -m MISSING  spec.grb

Latitude, Longitude, Value
   51.500    2.500 MISSING
Latitude, Longitude, Value
   51.500    2.500 -1.4684238434e+00

To interpret these values correctly, the user should do:

   if (value == missingValue) then
     value = 0.0
   else
     value = 10.0**value  // 10 to the power of the value
   endif

Special undocumented environment variables?

They are not undocumented any more because you are reading this FAQ (smile). Here they are:

If the library is built with only one of the jpeg libraries (jasper or openjpeg) it will work without any environment variable. If grib_api is built with both, then you have always to link both and you can switch from one to the other with the env variable:

GRIB_JPEG=jasper
GRIB_JPEG=openjpeg

This env variable was only for debugging/development purposes, but it can be used by users. There is also another variable GRIB_DUMP_JPG_FILE=filename which provides a dump of the jpeg image on a separate file, again for debugging purposes.

The env variable regarding the output stream is

GRIB_API_LOG_STREAM=stdout (errors and warnings are sent to standard out)
GRIB_API_LOG_STREAM=stderr (errors and warnings are sent to standard err)

default is GRIB_API_LOG_STREAM=stderr

 

Also see GRIB API Environment Variables

How can I remove the PV array (list of vertical coordinates)?

This can be done by setting the key "deletePV" as shown:

% grib_set -s deletePV=1 in.grib out.grib

Which is in fact equivalent to setting the number of coordinate values to 0 and clearing the "pv" array:

% grib_set -s PVPresent=0,NV=0 in.grib out.grib

Assertion failure setting key changeDecimalPrecision

If you issue the command "grib_set -s changeDecimalPrecision=1 spectral.grib output", you may get an assertion!

grib_scaling.c at line 48: assertion failure Assert(bpval >= 1)Abort

This is because that key does not work for spectral_complex! So first check the packingType key.  (GRIB API needs a better error message here)

Why do I get different results from grib_compare when blacklisting certain keys?

For example if you are interested in comparing the differences between the data values of two GRIB files, you may be tempted to blacklist (exclude) certain keys which show up in the comparison e.g.

% grib_compare file1.grib1 file2.grib1
long [binaryScaleFactor]: [-4] != [-11]
double [referenceValue]: [-1.21940258789062500000e+03] != [-1.26698560714721679688e+01]
        absolute diff. = 1206.73, relative diff. = 0.98961
        tolerance=0.000244141
long [N]: [2694] != [1958]
long [P]: [1385] != [712]
double [values]: 4139 out of 4160 different
 max absolute diff. = 9.1467330932617188e+01, relative diff. = 0.319192
        max diff. element 0: 1.95091751098632812500e+02 2.86559082031250000000e+02
        tolerance=0.0000000000000000e+00 packingError: [0.0313721] [0.000244617]

% grib_compare -bbinaryScaleFactor,referenceValue,N,P file1.grib1 file2.grib1
double [values]: 4139 out of 4160 different
 max absolute diff. = 3.8698000000000000e+04, relative diff. = 0.711452
        max diff. element 78: 1.56950000000000000000e+04 5.43930000000000000000e+04
        tolerance=0.0000000000000000e+00 packingError: [0.5] [0.5]

You can see when we used the blacklist, the results reported are different! This is because these keys are used to decode the values and blacklisting them causes the differences.

In these situations it is best to include what you want to compare rather than exclude with a blacklist. E.g.

% grib_compare -c values file1.grib1 file2.grib1

Now only the values array is compared.  Now you can use the -P, -R and -A flags to control the tolerances.

What are the keys numberOfDataPoints, numberOfValues and numberOfMissing?

numberOfDataPoints: This is the total number of points on the grid and includes missing as well as 'real' values

numberOfValues (=numberOfCodedValues): These two keys are the same (one is an alias for the other). This is the number of 'real' values in the field and excludes the number of missing ones

numberOfMissing: You guessed it. The number of missing values in the field

So you can write the equation:

    numberOfDataPoints = numberOfCodedValues + numberOfMissing

How can I multiply the field values by a constant?

One way is to use grib_set with the key "scaleValuesBy".  Let's say you have a field in orig.grib whose values you want to multiply by 2.1. First check the original data value statistics:

% grib_ls -p min,max,avg orig.grib
min         max         avg
234.554     312.085     278.977

You could have also checked the statistics by "grib_ls -n statistics orig.grib".
Now let's scale all values up by a factor of 2.1:

% grib_set -s scaleValuesBy=2.1  orig.grib  out.grib

And check the new values after the multiplication:

% grib_ls -p min,max,avg out.grib
min         max         avg
492.563     655.376     585.853

For adding or subtracting a constant, you can use the key "offsetValuesBy".

Configure script: What features are enabled/disabled by default?

First do a ./configure --help which will list the features available to be turned off/on.  If a feature is turned off by default, then its prefix is "--enable-" and if it is turned on by default, then its prefix is "--disable-".

So for example if the help text shows "--enable-python" you know it is disabled by default (i.e. if you do not pass this option to configure you will not get python support). Similarly if the help text shows "--disable-fortran" you know Fortran support is enabled by default.

Is GRIB-API thread-safe?

GRIB API is not thread-safe by default, unless it is built with pthread support. To do so, you need to make sure that the option "--enable-pthread" is present in your configure command (Note: there are known thread safety issues when multi-field support is enabled).

For CMake, use the option:

  -DENABLE_GRIB_THREADS=ON

Building on High Performance Computer systems

Some HPC batch systems have a different hardware architecture for their login (or frontend node) to the batch node, but the frontend compilation system is targeted at the batch nodes. This is known as cross-compilation. If this is the case you may see failures in the 'make check' stage because the checks, although compiled for the backend batch nodes, are being run on the frontend nodes and therefore may not work correctly. If this is the case on your system, we recommend using a batch job to do the 'configure; make; make check; make install' steps.

In some cases, the batch system cannot be used for compilation at all. In this case, you have to compile on the frontend but without extra flags 'configure' will assume the build is for the frontend. You can make use of the --host option to ensure the build is correct for the architecture of the batch system. Again though, the tests will fail, a small serial batch job is recommended to make sure grib_api is installed correctly.

Note if you plan on using the grib_api software in your own software that runs on the frontend nodes you will need to install grib_api twice; one for the batch system and again for the frontend system.

If you have any questions installing grib_api in this type of environment, please contact: openifs-support@ecmwf.int for assistance.

How do I create my own local definitions?

See the slides in this Training course presentation

Where can I find the version history?

Here

Feature Requests

  • How can I get the number of grib messages from my index selection? (usw)

maybe something like grib_index_get_selected_messages_size(index, &num_selected_messages);

 

  • Can a C-version of the grib_get_data() function be implemented? (usw)

Iterating over geo-referenced values in a grib file can be done in the C interface by using the 'grib_iterator'. See Iterating on latitude,longitude,values.

There is also a 'grib_get_data' present, but not documented. What it basically does is to use an iterator to return ALL geo-referenced values in an input array. Prototype for that is:

int grib_get_data(grib_handle* h,double* lats, double* lons,double* values,size_t* size);
  • Can we have function to de-select a selection from an index? (usw)
  • Can we implement the "-i index" option in grib_get (as in grib_ls) ? (usd) DONE
  • Can we make "grib_copy -p key" actually print something without needing the -v option ? (usd)
  • Can grib_get_data print the correct latitude and longitude values for rotated grids ? (usd) DONE
  • Can we have more documentation about the namespaces (e.g., statistics, mars, ls) ? (usd)

Done... look further up in this FAQ

  • Can we have more documentation about the computed keys (distinctLatitudes, distinctLongitudes, etc) ? (usd)