2016-09-28

WRF Vtable Carpentry

When we announced the introduction of WRF-able data sets (Vtables auto-generated from RDA-collected metadata) in January 2016, WRF Preprocessing System (WPS) component, ungrib, was able to use the RDA Vtables without any hiccups.

Around the time of the release of WRF 3.8.0 in April 2016, rdahelp@ucar.edu began receiving reports of error messages when running ungrib with RDA-provided Vtables.  If the Vtable includes lines describing fields that are not contained in that particular input GRIB file, ungrib stops instead of ignoring that field (as it had done previously.)

WPS V3.8 Updates

The simple solution is for the user to edit the RDA-provided Vtable to remove the extraneous line(s) referring to fields not contained in the input GRIB file.

After investigation and consideration, wrfhelp and rdahelp decided not to change either the WPS code or the auto-generated Vtables to return to plug and play.  Here's why.

The RDA provides archives of analysis, forecast and reanalysis data sets.  Reanalysis data sets contain the same variables/parameters throughout the entire time series.  Operational analysis and forecast systems vary over time.  Parameters or vertical levels may (and should!) change over the time series.

With advances in computational power and modeling of physical processes, newer fields and levels can be added.  Bandwidth is limited.  Dropping vestigial parameters when they are replaced with better ones is necessary.

RDA-provided Vtables are complete.  They include lines describing every WRF-usable parameter found in the data set time series.  To see how parameters and vertical levels changed over time, click on the "Variables by dataset product" or "detailed metadata for level information" links from the data set home pages.

Consider the long-running NCEP analysis series, FNL, ds083.2. When it began in 1999, it was in GRIB (aka GRIB1) format. Beginning in 2007, it is offered in GRIB2 format. Several times, the model was changed and fields were added (and sometimes subtracted.)

View the vertical level information of long-running NCEP analysis series, FNL, ds083.2.  Select "detailed metadata" to view GRIB1 details; select "GRIB2 level table" to view GRIB2 details.

Defaults to "Parameter View".  Also explore "Vertical Level View."
Notice that the soil layers expanded from a single layer 10-200 cm below the surface to three layers in 2005
Vtable.RDA_ds083.2 contains lines describing the entire time series from 1999 to the present. GRIB1 codes are on the left; GRIB2 codes are on the right. Notice how there are lines with GRIB1 or GRIB2 codes only. (One set of columns is blank.) If you are using a GRIB2 file (2007 and later), then edit out the lines that pertain only to GRIB1 files (GRIB2 columns blank.) If you are using a GRIB1 file, delete the lines referring to parameters found only in the GRIB2 files.

Delete the lines that refer to variables not found in the file you are using.
Similarly, if you are using a 2005 and older GRIB1 file, delete the six lines referring to three layers of soil temperature or moisture between 10 and 200 cm. If you are using a 2005-05-31 and later GRIB1 or GRIB2 file, delete the two lines referring to Soil temperature/moisture 10-200 cm below ground.

These soil layers are found only in FNL files 2005-05-31 and later.  Delete for earlier dates.
In closing, "plug and play" is both a blessing and a curse.  We all want things that "just work" right out of the box.  But, your data should not be a black box.  You don't need to become an expert in the arcana of data syntax, but you should peek "under the hood" and understand what is in your data file.

To learn more about GRIB data sets, read our GRIB blog series, particularly Setting up to work with GRIB2. Use wgrib and wgrib2 to explore the contents of your GRIB or GRIB2 files. You might find a parameter you aren't already using but might be helpful. Or, you may see a lot of parameters and vertical layers that you don't need. In that case, may I suggest you Subset to save time and bandwidth?

2016-09-16

NCAR RDA partial outage September 19, 2016

The NCAR CISL High Performance Storage System (HPSS) will be down for Monday, September 19, 2016,  to prepare for the installation of Cheyenne, NCAR's new high-performance supercomputer.


Although the RDA's primary and backup copies of data are stored on HPSS, the most popular data, approximately a third of our holdings, are replicated on our web server for faster data access. Thus, most users will not be impacted.

Users requesting HPSS data will experience a delay, but need not do anything special.  HPSS data services will automatically resume upon completion of this work.
SCHEDULED OUTAGE NOTIFICATION

HPSS will be down next Monday Sept 19 from 8:30 AM to 6:30 PM for hardware and systems integration work to help prepare the way for HPSS support of Cheyenne.

START: Mon Sep 19 2016 8:30 AM MDT
END: Mon Sep 19 2016 6:30 PM MDT

2016-09-08

NCAR RDA outage September 14, 2016

UPDATE Wed Sep 14 14:13:35 MDT:
All RDA data services are back online. Thank-you for your patience.

UPDATE:
NCAR computers will be down for 10 hours, beginning Wed Sep 14 2016 6:00 AM MDT.  All RDA services will be down during that time.

START: Wed Sep 14 2016 6:00 AM MDT
END: Wed Sep 14 2016 16:00 AM MDT

NCAR CISL central file disk systems will be down for system maintenance from 7:00 on Wednesday September 14, 2016 until 10:00 MDT. All data downloads, including access through THREDDS servers, will be unavailable during this outage. Custom data orders will be held and automatically restarted after the work is complete.
START: Wed Sep 14 2016 7:00 AM MDT
END: Wed Sep 14 2016 10:00 AM MDT

SERVICES AFFECTED
GLADE Administration