19 August 2015

Setting up to work with GRIB2

We are still receiving requests for translation of data files from GRIB2 to GRIB1.  Archiving the same data in two different formats takes up disk space that could otherwise be used to provide other useful data.  That is why our policy is to offer data in its native format.


In a very few cases, where a broad community could benefit, we will archive translations.  E.g. NCEP ADP Upper Air data in ds351.0 is offered in both its native BUFR and translated to little_R format for ingest into WRF (the Weather Research Forecast model).

As explained in GRIB1, GRIB2, NetCDF: What do I use?, GRIB1 files are much larger (2x) than GRIB2 files. They use up more disk space AND network bandwidth.  Smaller files means we can serve more data to more users.  That's a good thing.

In the distant past, setting up to work with GRIB2 was complicated.  That is no longer the case as long as your machine is set up for common scientific computing tasks.

GRIB2 files are smaller than GRIB1 files due to file compression with JPEG2000.  This commonly requires installation of the JasPer library, an open-source implementation of JPEG2000.  The JasPer website is very straightforward and assumes users know precisely what they are doing.  It's understandable to get stuck and decide to put off JasPer installation to 'later'.

There is an easier way.

Install NCEP's wgrib2 utility--the Swiss army knife of tools for GRIB2!  The wgrib2 tar package includes all the libraries it needs, including JasPer.  'Make' wgrib2, and you also compile and install JasPer.

Follow the instructions found in Source Code and Compiling Hints section of the main wgrib2 website.  I am honored that my wgrib2 installation guide for OS X and Linux are included in that section.

Try installing JasPer the easy way and you'll be able to use it for GRADS g2clib and WRF ungrib.  If you do run WRF, you may want to read Guidance for WRF users for new NCEP GFS and FNL GRIB2 files.

Tradeoffs:

File compression and decompression take time and computational resources.  Some national centers continued to disseminate data in GRIB1 format for this reason.  However, data can be packed (compressed) in different ways.  A slightly less aggressive packing will result in a greatly reduced packing/unpacking time with only a slight increase in file size.  (See this discussion at gradsusr.org.)

Beginning January 14, 2015, NCEP switched to a less aggressive file compression, which results in a 10x reduction (from 30 seconds to 3 seconds) for unpacking a FNL file with wgrib2.  Sometimes, this is called GRIB2 version 2.  In the near future, more national centers, notably ECMWF, will be switching from GRIB1 to GRIB2 version 2.

It's time to set up your workflow to handle GRIB2 files.

6 comments:

  1. Hello Grace,

    Thank you for providing us with a wgrib2 installation tutorial.

    I am trying to set it up on my Mac. I followed your steps carefully and it turns out that I get a fortran error when I run the make command:

    makefile:362: *** ERROR, fortran compiler (enironment vararible FC) is not recognized. Stop.

    Would you have any suggestions for my problem?

    Best regards,

    Sebastien Boulnois.

    ReplyDelete
  2. Did you install gfortran per the instructions here?
    https://rda.ucar.edu/datasets/ds083.2/software/wgrib2_install+.txt

    ReplyDelete
  3. Hi,

    I'm trying to help someone install wgrib2 on OSX, and we're running into an error I haven't been able to fix during compilation.

    We followed all your instructions, but when we type "make" we get:

    Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
    cd /usr/local/wgrib2/grib2/jasper-1.900.1 && export CC=gcc && ./configure --without-x --disable-libjpeg --disable-opengl --prefix=/usr/local/wgrib2/grib2 && make -j 1 check install
    checking build system type... i386-apple-darwin15.6.0
    checking host system type... i386-apple-darwin15.6.0
    checking target system type... i386-apple-darwin15.6.0
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for gawk... no
    checking for mawk... no
    checking for nawk... no
    checking for awk... awk
    checking whether make sets $(MAKE)... yes
    checking for gcc... gcc
    checking for C compiler default output file name... configure: error: C compiler cannot create executables
    See `config.log' for more details.
    make: * [/usr/local/wgrib2/grib2/lib/libjasper.a] Error 77

    Any ideas? Let me know if you want any additional information.

    ReplyDelete
    Replies
    1. I am getting the exact same error using gcc and gfortran for compiling the wgrib2 v2.0.6+ on OS X. Any input will be greatly appreciated.

      Delete
    2. I am getting the same issue as well. Was anyone able to resolve it?

      Thanks greatly!

      Delete
    3. Make sure that your version of gcc is not too old:
      https://stackoverflow.com/questions/10357804/configure-error-c-compiler-cannot-create-executables

      Run "gcc --version" to see what version you have, and if it is lower than 5, you need to upgrade.

      Delete

This section is for people who want to discuss using our data holdings effectively. Moderators will delete irrelevant comments.