NCAR's Research Data Archive featured on UCARConnect 2

The November 2016 issue of UCARConnect, Data: The Currency of Science is now live.  The NCAR RDA is proud to have contributed to this issue.

RDA Data Specialist, Grace Peng, wrote Big Data, Big Planet

We’re experiencing a big data explosion both in cultural awareness and in penetration into many aspects of everyday life. How did we get here? What role do weather and climate play in our data moment?

Big data is characterized by its "Vs": Volume, Variety and Velocity. Data volume is “big” if it is too large to reside on or be processed with a personal computer.

Did you know that even in the pre-computer era, weather data was the original big data use case? It’s most useful when we have a lot of it and we need the data in real-time. Furthermore, we use a large variety of physical measurements to characterize the state of the atmosphere and ocean.

RDA manager, Steve Worley, and former intern, Sophie Hou, speak about the history of the RDA and scientific data in general in this video.

Sophie Hou has joined NCAR permanently as our new Data Curation & Stewardship Coordinator. Read her career profile about what led her to this exciting career.

UCARConnect is a (roughly) bimonthly publication of UCAR aimed at science enthusiasts of all ages, including K-12 students and teachers.

GRIB2 file carpentry

We've learned from several users that the second WRF method in our post about How to work around operational changes to GFS/GDAS/FNL does not work.

Use the first method described.

First, you can work around the problem by removing the new vertical levels in the files after the GFS/GDAS change. Then all of your WPS input will be consistent. You can do this yourself with wgrib2, or our "Get a Subset" tool.

Here's the detailed recipe for those of you not familiar with wgrib2 tool for manipulating grib2 files.

We're going to use tip #4 and #4a from Wesley's Tricks for wgrib2 page to extract a range of records from a grib2 file.

Start by making a short inventory to identify the records that you want to keep and to remove.

wgrib2 gdas1.fnl0p25.2016051106.f00.grib2 > gdas1.fnl0p25.2016051106.f00.inv

wgrib2 gdas1.fnl0p25.2016051112.f00.grib2 > gdas1.fnl0p25.2016051112.f00.inv

Note that gdas1.fnl0p25.2016051106.f00.inv has 322 lines and gdas1.fnl0p25.2016051112.f00.inv has 352 lines. The extra 30 lines/records are in records 5-34.

We'll extract fields 1-4 and 35-352 separately and then recombine them with the UNIX command, cat.  It would be tedious to type in each and every file I want to convert. To save time, I put all the files I want to convert into one directory and ran this bash script.

#!/bin/bash

for file in gdas1*f00.grib2; do
  wgrib2 $file -for 1:4 -grib temp1
  wgrib2 $file -for 35:352 -grib temp2
  cat temp1 temp2 > $file.27
done


NCAR's Research Data Archive featured on UCARConnect

Learn more about UCARConnect