2017-01-12

Where to get WRF help

A large subset or RDA users also run WRF.  In order to clear up confusion about the appropriate place to get help, I'm compiling this information in one place.
For fastest answers to common RDA questions, read the RDA FAQ page or search this blog.

2017-01-10

RDA Data Access Services back online

All RDA data access services are back online.  The compute systems that process subset, format conversion, and re-staging of data from tape to disk requests are still offline, but should be back online within a few hours.  Once these systems come back online request processing will resume.  Thanks for your patience during today's maintenance window.

2017-01-09

RDA Data Access Services Offline on Tuesday, Jan 10.

All RDA data transfer services will be going offline tomorrow, Tuesday, Jan 10 while system maintenance if performed.  We hope to have these services restored in the afternoon, MST.

2017-01-06

Remembering Roy Jenne (1931-2016)

Roy L. Jenne, the founding leader of NCAR's data support services and a researcher of global climatology, died November 18 at Foothills Hospital in Boulder. He was 85.

Shortly after arriving at NCAR in 1965, Jenne started the Data Support Section (DSS), which is now part of NCAR's Computational and Information Systems Laboratory (CISL). DSS has helped researchers at NCAR and in the atmospheric and Earth system science community ever since.

"With a small team, he recognized the importance of collecting, preserving, and providing access to scientific data," said Steven Worley, CISL's current DSS manager. "In addition to data, he also loved the science that relied upon it."

Under Jenne's direction, NCAR became one of the world's primary archives of observational and model data to support weather and climate research, including the efforts of the Intergovernmental Panel on Climate Change and of global atmospheric reanalyses in the United States and abroad.
Read more from Remembering Roy Jenne (1931-2016).
A memorial gathering for family, friends, and colleagues has been scheduled for the Damon Room at the Mesa Lab on February 16 from 3 to 5 p.m.
If you can't attend in person, please share your favorite Roy Jenne stories as a comment.

2017-01-05

RDA Data Access Services Restored

All RDA data access services have been restored.   We apologize for the any inconvenience during yesterday's downtime.

2017-01-04

RDA Data Access Capabilities Unavailable

Storage systems that support the RDA are currently down for unscheduled maintenance, and all RDA data transfer services are unavailable.  An update will be posted when these services are restored.  Thank you for your patience.

2016-11-28

NCAR's Research Data Archive featured on UCARConnect 2

The November 2016 issue of UCARConnectData: The Currency of Science is now live.  The NCAR RDA is proud to have contributed to this issue.

RDA Data Specialist, Grace Peng, wrote Big Data, Big Planet
We’re experiencing a big data explosion both in cultural awareness and in penetration into many aspects of everyday life. How did we get here? What role do weather and climate play in our data moment?

Big data is characterized by its "Vs": Volume, Variety and Velocity. Data volume is “big” if it is too large to reside on or be processed with a personal computer.

Did you know that even in the pre-computer era, weather data was the original big data use case? It’s most useful when we have a lot of it and we need the data in real-time. Furthermore, we use a large variety of physical measurements to characterize the state of the atmosphere and ocean.
RDA manager, Steve Worley, and former intern, Sophie Hou, speak about the history of the RDA and scientific data in general in this video.

Sophie Hou has joined NCAR permanently as our new Data Curation & Stewardship Coordinator. Read her career profile about what led her to this exciting career.

UCARConnect is a (roughly) bimonthly publication of UCAR aimed at science enthusiasts of all ages, including K-12 students and teachers.

2016-11-18

GRIB2 file carpentry

We've learned from several users that the second WRF method in our post about How to work around operational changes to GFS/GDAS/FNL does not work.

Use the first method described.
First, you can work around the problem by removing the new vertical levels in the files after the GFS/GDAS change. Then all of your WPS input will be consistent. You can do this yourself with wgrib2, or our "Get a Subset" tool.
Here's the detailed recipe for those of you not familiar with wgrib2 tool for manipulating grib2 files.

We're going to use tip #4 and #4a from Wesley's Tricks for wgrib2 page to extract a range of records from a grib2 file.

Start by making a short inventory to identify the records that you want to keep and to remove.
wgrib2 gdas1.fnl0p25.2016051106.f00.grib2 > gdas1.fnl0p25.2016051106.f00.inv

wgrib2 gdas1.fnl0p25.2016051112.f00.grib2 > gdas1.fnl0p25.2016051112.f00.inv

Note that gdas1.fnl0p25.2016051106.f00.inv has 322 lines and gdas1.fnl0p25.2016051112.f00.inv has 352 lines. The extra 30 lines/records are in records 5-34.
We'll extract fields 1-4 and 35-352 separately and then recombine them with the UNIX command, cat.  It would be tedious to type in each and every file I want to convert. To save time, I put all the files I want to convert into one directory and ran this bash script.
#!/bin/bash

for file in gdas1*f00.grib2; do
  wgrib2 $file -for 1:4 -grib temp1
  wgrib2 $file -for 35:352 -grib temp2
  cat temp1 temp2 > $file.27
done

2016-10-31

NCAR RDA outage Nov 1, 2016

UPDATE Nov 1 15:11:39 MDT 2016: All RDA services are back in service. Thank you for your patience.

NCAR CISL central file disk systems will be down for system maintenance from 8:00 MDT on Tuesday, November 1, 2016 for approximately 4 hours.

All data downloads, including access through THREDDS servers, will be unavailable during this outage.

NCAR CISL supercomputers will be down for system maintenance from 8:00 MDT on Tuesday, November 1, 2016 for approximately 8 hours.

Custom data orders will be held and automatically restarted after the work is complete.

Change notice from the Daily B:
The Yellowstone, Geyser, and Caldera clusters will be unavailable from 8 a.m. to 6 p.m. MDT Tuesday, November 1, to allow CISL staff to apply a critical security patch. The GLADE file systems will be unavailable from 8 a.m. to noon.

A system reservation will be put in place 12 hours before the scheduled downtime. Users’ jobs with specified job times that overlap the reservation period will remain on hold until the system is restored to service. Running jobs that have not finished by 8 a.m. Tuesday will be killed and will need to be resubmitted after the maintenance period.

We apologize for any inconvenience this might cause. Users will be informed via the CISL Notifier service when the systems are returned to production.