Best practices when asking for help

Get answers to your questions faster! Follow these best practices.

1. Check that you are logged in.  Log out, log back in to check that you set your current email and password correctly in your scripts.
2. Send the question to rdahelp@ucar.edu.  If you have a question about a specific dataset, cc the data specialist for that dataset as well for faster service.
3. Give us your name and how you would like to be addressed.
4. Give us the email address used for your RDA registration.  We look you up in our user database with this email address.  It would be even better if you also use this email address to write to us.
5. State the dataset number, dsnnn.n, if applicable.
6. Describe the nature of your problem in detail.  Include the error message(s) or a screen shot of what you see.  Tell us your system configuration, time you had the problem, the sequence of steps you took...
7. If you are having problems with a download script, substitute 'wget -d' for 'wget' (debug mode) for more descriptive error messages.  Then include the script that you used and the error messages in your email.

Click on the RDA About/Contact tab for contact information for the entire staff of the RDA http://rda.ucar.edu/#!about.

RDA at AGU 2015 Fall Meeting

Several members of the RDA staff will be attending the American Geophysical Union's 2015 Fall Meeting in San Francisco December 14-18.  Come to our sessions to introduce yourself and/or ask questions.

A video tour of an RDA data set home page

In this RDA tutorial video, we tour the elements of an individual data set's main information page.

The 6:28 minute video may seem long, but that is much quicker than sending a question via email and waiting for a response. ;-)

GRIB Practical Exercise 2: Test Spin

NASA's Panoply Data Viewer provides an easy-to-use way to open up and explore many earth science datasets .  It recognizes different flavors of GRIB, HDF and NetCDF and allows you to plot or view data without knowing specifics of a file.  If you haven't already done so, install it now.

What to do if you experience a "403: Forbidden" access error

Recently a number of RDA users have reported experiencing a "403:  Forbidden" error when attempting to download files with scripts.

If you are experiencing this error, please remove the associated cookie file that you are using to support the script, typically named "auth.rda_ucar_edu", and rerun the script with your RDA username and password to authenticate and download a new cookie.

What is GRIB2?

I want to expand upon a couple of points that I made in What is GRIB? because we continue to receive questions at rdahelp related to these two ideas.  I  hope, by explaining the why of GRIB, you can better use data in this format.

RDA chosen for new WCRP reanalysis intercomparison

Excerpt from CISL News:

Masatomo Fujiwara of Hokkaido University in Sapporo, Japan leads a program to understand differences between the world’s leading global atmospheric reanalysis datasets, and his team determined that almost all of the datasets for this intercomparison project are available in CISL’s Research Data Archive (RDA). The SPARC Reanalysis Intercomparison Project (S-RIP) is coordinated by one of the core projects of the World Climate Research Programme (WCRP), named Stratosphere-troposphere Processes And their Role in Climate (SPARC).

“Reanalysis datasets with varied data formats and different access protocols are archived at various reanalysis centers around the world, and some older reanalysis datasets are no longer archived at some centers,” explains project lead Masatomo Fujiwara. “The NCAR RDA houses key climate datasets and is extremely useful for worldwide data users to obtain both older and newer reanalysis datasets.”

The S-RIP project is comparing reanalysis data sets, investigating the causes of differences among reanalyses, helping researchers use reanalysis products in scientific studies, and working to improve future reanalysis products by collaborating with reanalysis centers. S-RIP collaborators have selected the RDA as the primary data source for this project: of the 12 reanalysis datasets being used, researchers are directed to the RDA for 11 of them.

Read the rest of the article.

While there are many places that archive weather and climate data, the variety of data and the availability of older data makes RDA unique.

A video tour of the RDA home page

Learn where to find and how to use the features found on the RDA home page--first in a series of RDA tutorial videos.

Windsday

The NCAR Mesa Lab has a reputation as a particularly windy site. This video taken from my office window shows why.


The ML real-time measurements, courtesy of NCAR's Earth Observing Lab, quantify just how gusty it is outside.

Setting up to work with GRIB2

We are still receiving requests for translation of data files from GRIB2 to GRIB1.  Archiving the same data in two different formats takes up disk space that could otherwise be used to provide other useful data.  That is why our policy is to offer data in its native format.

Interoperable Access to NCAR Research Data Archive Collections

To enhance user experience and utility, the RDA now offers THREDDS Data Server (TDS) access for many highly valued gridded dataset collections (http://rda.ucar.edu/thredds).  TDS offered datasets with native formats in GRIB1 and GRIB2 are presented as aggregations,  enabling users to access an entire dataset collection that can be comprised of 1000’s of files, through a single virtual file.  Individual files can also be accessed from all GRIB1, GRIB2, and NetCDF format dataset collections found on the TDS. 

Cite your data

Science should be reproducible.  Citing your data as clearly and unambiguously as possible is as important a part of science communication as explaining your methodology.

It's not enough to say that you used FNL or GFS or CALIPSO data.  The software that produced FNL and GFS analyses and forecasts evolve over time.  New parameters or methods are introduced.  Bugs are fixed.   Satellite processing algorithms similarly evolve.

These changes are often not well documented.   For someone else to understand (and possibly reproduce) your work, they need to know exactly which version of the data you used for your work.

New Terms and Conditions of Use for Japanese 55-year Reanalysis (JRA-55) Products

As of July 31, 2015, the JMA has relaxed its Terms and Conditions of Use for Japanese 55-year Reanalysis (JRA-55) Products. The main change is that "JMA provides the datasets with no imposition of restrictions regarding the purpose of use and/or redistribution", especially for commercial purposes. This impacts RDA datasets ds628.0, ds628.1, ds628.2, ds628.3 and ds628.4 (JRA-55, JRA-55C, JRA-55AMIP). The new conditions are as follows:

Accessing RDA collections from Yellowstone

The CISL User Services Section will hold a 60- to 75-minute seminar and discussion for Yellowstone users at 10 a.m. Friday, August 7, in the Small Seminar Room at the Foothills Lab in Boulder (FL2-1001). The session will also be webcast via UCARConnect.

Three brief presentations are planned:

Accessing RDA collections from Yellowstone: Grace Peng (CISL/DSS)—A large subset of NCAR/CISL Research Data Archive (RDA) dataset collections can be accessed directly from the GLADE file spaces on Yellowstone. This presentation will provide an overview of the RDA resources and tools, including metadata access and subsetting services, that are available to Yellowstone users.

Simple “command file” syntax for task parallelism: Dick Valent (CISL/USS/CSG)—By using command file syntax to exploit characteristics of your workflow, you may be able to achieve task parallelism to make more efficient use of your Yellowstone allocation and reduce time to solution. Command file syntax can be used in forms from simple to complex. Dick will present the simplest form of command file.

Real-time, ensemble forecasting using CISL resources: Ryan Sobash (MMM)—A daily, real-time, ensemble forecasting system has been running at NCAR since late March using Yellowstone and other CISL resources. Ryan will talk about some unique computing challenges this type of system poses, solutions developed to maintain reliability and timeliness, and the development of an ensemble visualization system and website (http://ensemble.ucar.edu) to view the forecasts.

All Yellowstone users are welcome. If you plan to attend in person, please sign up here to give us an idea of how many participants to expect.

#HPC #bigdata

NCEP Climate Forecast System Data: Some New Ways to Get a Subset

The Climate Forecast System Reanalysis (CFSR) was created by the National Centers for Environmental Prediction in 2010 by running their updated Climate Forecast System (CFSv2) retrospectively from January 1, 1979. CFSv2 continues to run operationally to produce NCEP's monthly and seasonal forecasts. You can read about the differences between reanalyses and operational model output here, and you can see a list of all of the reanalysis data hosted by the RDA from our home page, under the "Atmospheric Reanalysis Data" heading.

The International Surface Pressure Databank version 2: A new publication in the Geoscience Data Journal

The RDA is pleased to announce the publication of the data paper titled "The International Surface Pressure Databank version 2", which was recently published in the Geoscience Data Journal and is available at http://dx.doi.org/10.1002/gdj3.25.  The article provides a full description of version 2 of the International Surface Pressure Databank (ISPDv2), a dataset which is archived in the RDA and can be accessed from the ds132.0 dataset page.

JRA-55C in the RDA: The Japanese 55-year Reanalysis Using Conventional Data Only

As a subset of the Japanese 55-year Reanalysis (JRA-55) project, the Meteorological Research Institute of the Japan Meteorological Agency has conducted a global atmospheric reanalysis that assimilates only conventional surface and upper air observations, with no use of satellite observations, using the same data assimilation system as the JRA-55. The project, named the JRA-55 Conventional (JRA-55C), aims to produce a more homogeneous dataset over a long period, unaffected by changes in historical satellite observing systems. The dataset is intended to be suitable for studies of climate change or multi-decadal variability. The reanalysis period of JRA-55C is from November 1972 to December 2012. The JMA recommends the use of JRA-55 to extend JRA-55C back to January 1958.

The Data Support Section at NCAR has downloaded all JRA-55C data. The entire archive has been reorganized into single parameter time series, and model resolution data has been transformed to a regular Gaussian grid. The JRA-55C products are currently being made accessible to RDA registered users of JRA-55, and will appear incrementally via the Data Access tab on the dataset homepage(s) JRA-55C: The Japanese 55-year Reanalysis Using Conventional Data Only, and JRA-55C: Monthly Means and Variances.

GRIB Practical Exercise 1: Data Discovery

If you want to learn more about a specific data format, scroll down to the blue box at the bottom of the NCAR RDA home page and follow the links.  The Format Descriptions link to the encyclopedic WMO documentation for people who need to write interfaces to GRIB data.

Scroll to the bottom of rda.ucar.edu and follow the links to GRIB documentation.

Where's my data?

The RDA considers the ecosystem that data resides in when deciding what to archive.  For instance, we archive both NCEP GFS and FNL analyses*; they are part of the NCEP Global Data Assimilation System (GDAS) system.

• List of RDA datasets related to GDAS
• ds351.0 NCEP ADP Global Upper Air Observational Weather Data, October 1999 - continuing
• ds461.0 NCEP ADP Global Surface Observational Weather Data, October 1999 - continuing
• ds735.0 NCEP GDAS Satellite Radiance Data

Researchers may want to examine the data assimilated into the analyses.  They could be useful for future reanalyses, Observation System Simulation Experiments (OSSE) or regional monitoring.  Thus, we archive several of the major data sources in the GDAS data flow.  You can see the data shortfall in the AIRS 2015 total web server inventory listing.

What is GRIB?

"What is GRIB?" is a nontrivial question. There isn't even agreement on whether it stands for GRIdded Binary or General Regularly-distributed Information in Binary form.   This is an idiosyncratic and non-official take on GRIB by an autodidact of met data.

GRIB and BUFR are World Meteorological Organization (WMO) standards for data exchange.  WMO calls them data exchange formats, but many refer to them simply as data formats.  The loss of that one middle word is significant.

Transferring RDA Data with Globus

The RDA supports many different data access pathways for our users, and it should come as no surprise that the most popular method is web downloads via HTTP.  With HTTP, our users can download data files directly from our website, or via cURL and wget commands.

The popularity of web downloads from the RDA has been growing exponentially in recent years.  In 2014 alone, we served more than 1.1 petabytes of data to over 11,000 users. We fulfilled more than 4,000 customized data requests for our users, the majority (if not all) of which were downloaded via HTTP.

In order to keep up with this fast growth in data usage, the RDA has developed the capability for its users to transfer data via the Globus data transfer service.

What is Globus?

Globus (globus.org) can be described in the simplest terms as a third-party data transfer service, and provides a fast, secure, and reliable method for transferring large data volumes.  Data transfers are carried out using the GridFTP protocol and are facilitated by a Globus Connect server running on an IBM cluster housed at NCAR.

Step-by-step instructions on how to create a Globus account and transfer files are available on the Globus website.  To initiate a transfer, a user defines two endpoints—one origination endpoint and one receiving endpoint—and then submits the transfer task to Globus.  Once a data transfer job has been started, Globus takes care of the rest, including processing the transfer and monitoring its performance and progress.  After the transfer completes, Globus sends you an e-mail notification with a report detailing the transfer statistics and performance.

Users can initiate and manage Globus data transfers from their web browser; more savvy users who wish to integrate their data transfers into their automated workflows can use the Globus command line interface (CLI) or Python/Java API client.

How to transfer RDA data using Globus

Starting from any RDA dataset description page (see, for example, the 20th Century Reanalysis dataset page), select the Data Access tab, then select the link labeled 'Request Globus Invitation'.  Globus transfers can also be requested for customized data requests by selecting the 'Globus Download' button on the data download page.

Example Data Access matrix from a dataset page on the RDA website.  Selecting the link labeled 'Request Globus Invitation' initiates a Globus data share for the dataset.

At this point, a pop-up window will appear and you will be asked to confirm that you are requesting a Globus invitation to transfer the data from the dataset.  Once you have submitted the request, Globus will send you an e-mail invitation containing a unique URL to accept the data share invitation.  After accepting the invitation, you may then begin transferring data to your receiving endpoint via the Globus website.

The data transfer interface on the Globus website.  The RDA shared endpoint is shown on the left and the user's receiving endpoint is displayed on the right.

Alternate Identity login

To use the Globus service, users must register for a (free) Globus account.  If you do not have one, Globus will prompt you to register for an account prior to accepting the data share and transferring data. After you have a Globus account, you may then link your RDA and Globus accounts under your Globus user settings (after logging in, select Account –> Identities –> Link another identity).  Doing so will allow you to log into Globus using your RDA user e-mail and password, thus requiring you to remember only one username/password combination.

The Globus alternate identity provider login interface.  RDA users may log into their Globus account using their RDA user e-mail and password by selecting the 'NCAR RDA' identity provider.

Data endpoint management

All Globus data shares created for our users are listed under each user's 'dashboard' (select 'Dashboard' at the top of the RDA home page).  Here, users can view and manage all active Globus shares assigned to their account.  Data shares can be deleted; or, if a user has misplaced the Globus e-mail invitation for a data share, a new one can be sent.

A listing of Globus data shares, displayed in the RDA user's dashboard.

ds608.0 NARR 20090401-20150131 rerun4 updates

Updates of NCEP North American Regional Reanalysis (NARR) data from April 1, 2009 to January 31, 2015 have been archived as "rerun4" version of ds608.0 dataset in rda.ucar.edu in May 2015.  This update fixes the codes that misread Mexican precipitation data and a bug introduced when NCEP switched the computer systems. The direct effects of these changes are in the precipitation and in the soil moisture fields.

Review rr4.pdf for details.

GRIB1, GRIB2, NetCDF: What do I use?

[Note: An older version of this information is available under the Documentation Tab of both ds083.2 FNL and ds084.1 GFS as README_Formats.pdf.  This page is maintained and should be considered the definitive version.]

We get this question often from users setting up their data workflow, particularly for datasets for which the data translation option, "Get Converted Files" is available.

Data Access Matrix for ds083.2, which is available in its native GRIB1 and GRIB2 formats or translated to NetCDF.  If you want data translation to NetCDF, click on the options circled in red.

The answer (or non answer) is it depends upon which dataset you want to use (and dates within the series) and which tool(s) you will use to analyze the data.

The data starts here

At the Water Conservation Gardens Fair yesterday in Berthoud, Colorado (near NCAR's Boulder home), several vendors showed soil moisture monitoring systems.  I took a picture to show where data comes from.

Soil moisture monitor/data logger and one sensor.

2015 Unidata Users Workshop

The Unidata Users Committee invites you to join Unidata staff, community members, and distinguished speakers at the 2015 Unidata Users Workshop, scheduled to take place 22-25 June 2015 in Boulder, Colorado.

Accessing Atmospheric and Oceanographic Data Formats with ArcGIS

Recently we've received a number of inquiries about accessing RDA datasets with ArcGIS.  ArcGIS provides the capability to ingest most data formats that are used in the Atmospheric Science and Oceanography communities.  These include GRIB-1, GRIB-2, NetCDF, HDF4, and HDF5.  OPeNDAP access is also supported in ArcGIS.  A list of related links describing this support is included below.

CFDDA: a global, hourly, 40 km resolution reanalysis

NCAR RDA announces the debut of The NCAR Global Climate Four-Dimensional Data Assimilation (CFDDA) Hourly 40 km Reanalysis, a 21-year climatography.

High-resolution temporal and spatial resolution data--usually created with regional-scale models—have traditionally been proprietary and costly to obtain. Freely available global model data suffers from either lower spatial or temporal resolution, or both. Low spatial resolution fails to realistically represent wind speeds in complex terrain. Low temporal resolution fails to capture the full diurnal cycle of wind behavior.

The NCAR Global Climate Four-Dimensional Data Assimilation (CFDDA) Hourly 40 km Reanalysis was developed in 2009-2010 by the Research Applications Laboratory (RAL) to provide the most accurate boundary layer wind estimates available at that time. CFDDA used 28 sigma levels, with 19 between the surface and 700 hPa, a four-fold improvement over the contemporary NWP models. The dataset spans 21 years, 1985-2005, providing hourly atmospheric parameters, including winds, on 28 vertical levels on a global 40 km grid.

Unlike most global reanalyses, CFDDA output is available hourly to fully capture the diurnal range of weather phenomena.  For instance, if the land-sea breeze or mountain-valley circulation in a region peaks at around 14:00 local time, and that does not coincide with the 00, 06, 12, or 18 UTC times of the saved reanalysis, then the strength of the phenomena will be underestimated in that region.

Analysis, forecast, reanalysis--what's the difference?

[Note: An older version of this information is available under the Documentation Tab of both ds083.2 FNL and ds084.1 GFS as Analysis.pdf.  This page is maintained and should be considered the definitive version.]

This extends the discussion in What's the difference between GFS and FNL? Read that, and then come back. I'll wait.

From Dee et al:

Reanalysis data provide a multivariate, spatially complete, and coherent record of the global atmospheric circulation. Unlike archived weather analyses from operational forecasting systems, a reanalysis is produced with a single version of a data assimilation system—including the forecast model used—and is therefore not affected by changes in method.

What's the difference between GFS and FNL?

[Note: An older version of this information is available under the Documentation Tab of both ds083.2 FNL and ds084.1 GFS as FNLvGFS.pdf.  This page is maintained and should be considered the definitive version.]

GFS stands for Global Forecast System, the current NOAA NCEP global numerical weather prediction system.

FNL refers to the "Final" analysis, though it is now mainly referred to as the GDAS analysis, a part of the Global Data Assimilation System.

FNL and GFS are related, yet different products from the same data assimilation and forecast system. They share the same underlying model and data assimilation techniques. They contain the same data sources--but there is a subtle difference in the amount of "real" data assimilated into the initial conditions for GFS and FNL.

Exploring the NOAA/CIRES Twentieth Century Global Reanalysis Version 2c

Each reanalyses has its strengths and weaknesses. To better select the optimal dataset for your research, it's helpful to take several datasets out for a test spin.

I put our newest reanalysis dataset, the NOAA/CIRES Twentieth Century Global Reanalysis Version 2c, and a timely topic, Atmospheric Rivers (ARs), together into a data exploration exercise by visualizing a strong AR event from Dec 2004 to Jan 2005.

The atmospheric river that caused the Los Angeles flood of 1938

At least one person asked why I used a reanalysis that does not assimilate satellite water vapor data to study an atmospheric river (AR) event.

That's a good question because the NOAA/CIRES Twentieth Century Global Reanalysis Version 2c (20thCR V2c) only ingests three things: surface pressure, sea ice coverage and sea surface temperature. The rest of the analysis is generated by the physical models of NOAA's Global Forecast System (GFS).