- Check that you are logged in. Log out, log back in to check that you set your current email and password correctly in your scripts.
- Send the question to firstname.lastname@example.org. If you have a question about a specific dataset, cc the data specialist for that dataset as well for faster service.
- Give us your name and how you would like to be addressed.
- Give us the email address used for your RDA registration. We look you up in our user database with this email address. It would be even better if you also use this email address to write to us.
- State the dataset number, dsnnn.n, if applicable.
- Describe the nature of your problem in detail. Include the error message(s) or a screen shot of what you see. Tell us your system configuration, time you had the problem, the sequence of steps you took...
- If you are having problems with a download script, substitute 'wget -d' for 'wget' (debug mode) for more descriptive error messages. Then include the script that you used and the error messages in your email.
The 6:28 minute video may seem long, but that is much quicker than sending a question via email and waiting for a response. ;-)
If you are experiencing this error, please remove the associated cookie file that you are using to support the script, typically named "auth.rda_ucar_edu", and rerun the script with your RDA username and password to authenticate and download a new cookie.
All RDA services, including data download and data request processing, will be unavailable during system maintenance starting 13:30 MDT on Wednesday October 28, 2015.
This notice will be updated when work is complete.
All RDA data services, including data download and data request processing, are back online.
Thank you for your patience during the outage.
All RDA related data access and processing services are experiencing an unexpected outage at this time. Systems engineers are working to diagnose and resolve the issues. We will send out another notice once they come back online.
We apologize for the inconvenience.
Masatomo Fujiwara of Hokkaido University in Sapporo, Japan leads a program to understand differences between the world’s leading global atmospheric reanalysis datasets, and his team determined that almost all of the datasets for this intercomparison project are available in CISL’s Research Data Archive (RDA). The SPARC Reanalysis Intercomparison Project (S-RIP) is coordinated by one of the core projects of the World Climate Research Programme (WCRP), named Stratosphere-troposphere Processes And their Role in Climate (SPARC).Read the rest of the article.
“Reanalysis datasets with varied data formats and different access protocols are archived at various reanalysis centers around the world, and some older reanalysis datasets are no longer archived at some centers,” explains project lead Masatomo Fujiwara. “The NCAR RDA houses key climate datasets and is extremely useful for worldwide data users to obtain both older and newer reanalysis datasets.”
The S-RIP project is comparing reanalysis data sets, investigating the causes of differences among reanalyses, helping researchers use reanalysis products in scientific studies, and working to improve future reanalysis products by collaborating with reanalysis centers. S-RIP collaborators have selected the RDA as the primary data source for this project: of the 12 reanalysis datasets being used, researchers are directed to the RDA for 11 of them.
While there are many places that archive weather and climate data, the variety of data and the availability of older data makes RDA unique.
Plan your data flow in advance to minimize the impact on your work. We apologize for the inconvenience.
UPDATE 2015-09-22 12:52:00 MDT
System maintenance is complete and data processing queues are back in operation.
We apologize for the inconvenience.
We apologize for the inconvenience.
CISL Daily Bulletin about the outage.
UPDATE: Fri Aug 21 16:26:34 MDT 2015
The system is back up. All jobs waiting in the queue should start automatically. Thank-you for your patience.
It's not enough to say that you used FNL or GFS or CALIPSO data. The software that produced FNL and GFS analyses and forecasts evolve over time. New parameters or methods are introduced. Bugs are fixed. Satellite processing algorithms similarly evolve.
These changes are often not well documented. For someone else to understand (and possibly reproduce) your work, they need to know exactly which version of the data you used for your work.
As of July 31, 2015, the JMA has relaxed its Terms and Conditions of Use for Japanese 55-year Reanalysis (JRA-55) Products. The main change is that "JMA provides the datasets with no imposition of restrictions regarding the purpose of use and/or redistribution", especially for commercial purposes. This impacts RDA datasets ds628.0, ds628.1, ds628.2, ds628.3 and
ds628.4 (JRA-55, JRA-55C, JRA-55AMIP). The new conditions are as follows:
Three brief presentations are planned:
Accessing RDA collections from Yellowstone: Grace Peng (CISL/DSS)—A large subset of NCAR/CISL Research Data Archive (RDA) dataset collections can be accessed directly from the GLADE file spaces on Yellowstone. This presentation will provide an overview of the RDA resources and tools, including metadata access and subsetting services, that are available to Yellowstone users.
Simple “command file” syntax for task parallelism: Dick Valent (CISL/USS/CSG)—By using command file syntax to exploit characteristics of your workflow, you may be able to achieve task parallelism to make more efficient use of your Yellowstone allocation and reduce time to solution. Command file syntax can be used in forms from simple to complex. Dick will present the simplest form of command file.
Real-time, ensemble forecasting using CISL resources: Ryan Sobash (MMM)—A daily, real-time, ensemble forecasting system has been running at NCAR since late March using Yellowstone and other CISL resources. Ryan will talk about some unique computing challenges this type of system poses, solutions developed to maintain reliability and timeliness, and the development of an ensemble visualization system and website (http://ensemble.ucar.edu) to view the forecasts.
All Yellowstone users are welcome. If you plan to attend in person, please sign up here to give us an idea of how many participants to expect.
The International Surface Pressure Databank version 2: A new publication in the Geoscience Data Journal
UPDATE Tue Jul 28 15:43:32 MDT 2015:
Maintenance has been completed and all systems are back to normal.
The Data Support Section at NCAR has downloaded all JRA-55C data. The entire archive has been reorganized into single parameter time series, and model resolution data has been transformed to a regular Gaussian grid. The JRA-55C products are currently being made accessible to RDA registered users of JRA-55, and will appear incrementally via the Data Access tab on the dataset homepage(s) JRA-55C: The Japanese 55-year Reanalysis Using Conventional Data Only, and JRA-55C: Monthly Means and Variances.
|Scroll to the bottom of rda.ucar.edu and follow the links to GRIB documentation.|
- List of RDA datasets related to GDAS
- ds351.0 NCEP ADP Global Upper Air Observational Weather Data, October 1999 - continuing
- ds461.0 NCEP ADP Global Surface Observational Weather Data, October 1999 - continuing
- ds735.0 NCEP GDAS Satellite Radiance Data
For up-to-date information, visit the NCAR CISL Resource Status page.
UPDATE Tue Jul 7 18:28:02 MDT 2015:
At this time Yellowstone, Geyser and Caldera has been released for production workload.Data request processing has resumed. If your job did not automatically restart and complete satisfactorily, resubmit a new one.
GRIB and BUFR are World Meteorological Organization (WMO) standards for data exchange. WMO calls them data exchange formats, but many refer to them simply as data formats. The loss of that one middle word is significant.
The popularity of web downloads from the RDA has been growing exponentially in recent years. In 2014 alone, we served more than 1.1 petabytes of data to over 11,000 users. We fulfilled more than 4,000 customized data requests for our users, the majority (if not all) of which were downloaded via HTTP.
In order to keep up with this fast growth in data usage, the RDA has developed the capability for its users to transfer data via the Globus data transfer service.
What is Globus?Globus (globus.org) can be described in the simplest terms as a third-party data transfer service, and provides a fast, secure, and reliable method for transferring large data volumes. Data transfers are carried out using the GridFTP protocol and are facilitated by a Globus Connect server running on an IBM cluster housed at NCAR.
Step-by-step instructions on how to create a Globus account and transfer files are available on the Globus website. To initiate a transfer, a user defines two endpoints—one origination endpoint and one receiving endpoint—and then submits the transfer task to Globus. Once a data transfer job has been started, Globus takes care of the rest, including processing the transfer and monitoring its performance and progress. After the transfer completes, Globus sends you an e-mail notification with a report detailing the transfer statistics and performance.
Users can initiate and manage Globus data transfers from their web browser; more savvy users who wish to integrate their data transfers into their automated workflows can use the Globus command line interface (CLI) or Python/Java API client.
How to transfer RDA data using GlobusStarting from any RDA dataset description page (see, for example, the 20th Century Reanalysis dataset page), select the Data Access tab, then select the link labeled 'Request Globus Invitation'. Globus transfers can also be requested for customized data requests by selecting the 'Globus Download' button on the data download page.
|Example Data Access matrix from a dataset page on the RDA website. Selecting the link labeled 'Request Globus Invitation' initiates a Globus data share for the dataset.|
At this point, a pop-up window will appear and you will be asked to confirm that you are requesting a Globus invitation to transfer the data from the dataset. Once you have submitted the request, Globus will send you an e-mail invitation containing a unique URL to accept the data share invitation. After accepting the invitation, you may then begin transferring data to your receiving endpoint via the Globus website.
|The data transfer interface on the Globus website. The RDA shared endpoint is shown on the left and the user's receiving endpoint is displayed on the right.|
Alternate Identity loginTo use the Globus service, users must register for a (free) Globus account. If you do not have one, Globus will prompt you to register for an account prior to accepting the data share and transferring data. After you have a Globus account, you may then link your RDA and Globus accounts under your Globus user settings (after logging in, select Manage Identities –> Add linked identity –> Add single sign-on identity). Doing so will allow you to log into Globus using your RDA user e-mail and password, thus requiring you to remember only one username/password combination.
|The Globus alternate identity provider login interface. RDA users may log into their Globus account using their RDA user e-mail and password by selecting the 'NCAR RDA' identity provider.|
Data endpoint management
|A listing of Globus data shares, displayed in the RDA user's dashboard.|
Review rr4.pdf for details.
We get this question often from users setting up their data workflow, particularly for datasets for which the data translation option, "Get Converted Files" is available.
|Data Access Matrix for ds083.2, which is available in its native GRIB1 and GRIB2 formats or translated to NetCDF. If you want data translation to NetCDF, click on the options circled in red.|
|Soil moisture monitor/data logger and one sensor.|
High-resolution temporal and spatial resolution data--usually created with regional-scale models—have traditionally been proprietary and costly to obtain. Freely available global model data suffers from either lower spatial or temporal resolution, or both. Low spatial resolution fails to realistically represent wind speeds in complex terrain. Low temporal resolution fails to capture the full diurnal cycle of wind behavior.
The NCAR Global Climate Four-Dimensional Data Assimilation (CFDDA) Hourly 40 km Reanalysis was developed in 2009-2010 by the Research Applications Laboratory (RAL) to provide the most accurate boundary layer wind estimates available at that time. CFDDA used 28 sigma levels, with 19 between the surface and 700 hPa, a four-fold improvement over the contemporary NWP models. The dataset spans 21 years, 1985-2005, providing hourly atmospheric parameters, including winds, on 28 vertical levels on a global 40 km grid.
Unlike most global reanalyses, CFDDA output is available hourly to fully capture the diurnal range of weather phenomena. For instance, if the land-sea breeze or mountain-valley circulation in a region peaks at around 14:00 local time, and that does not coincide with the 00, 06, 12, or 18 UTC times of the saved reanalysis, then the strength of the phenomena will be underestimated in that region.
This extends the discussion in What's the difference between GFS and FNL? Read that, and then come back. I'll wait.
From Dee et al:
Reanalysis data provide a multivariate, spatially complete, and coherent record of the global atmospheric circulation. Unlike archived weather analyses from operational forecasting systems, a reanalysis is produced with a single version of a data assimilation system—including the forecast model used—and is therefore not affected by changes in method.
GFS stands for Global Forecast System, the current NOAA NCEP global numerical weather prediction system.
FNL refers to the "Final" analysis, though it is now mainly referred to as the GDAS analysis, a part of the Global Data Assimilation System.
FNL and GFS are related, yet different products from the same data assimilation and forecast system. They share the same underlying model and data assimilation techniques. They contain the same data sources--but there is a subtle difference in the amount of "real" data assimilated into the initial conditions for GFS and FNL.
I put our newest reanalysis dataset, the NOAA/CIRES Twentieth Century Global Reanalysis Version 2c, and a timely topic, Atmospheric Rivers (ARs), together into a data exploration exercise by visualizing a strong AR event from Dec 2004 to Jan 2005.
That's a good question because the NOAA/CIRES Twentieth Century Global Reanalysis Version 2c (20thCR V2c) only ingests three things: surface pressure, sea ice coverage and sea surface temperature. The rest of the analysis is generated by the physical models of NOAA's Global Forecast System (GFS).