23 January 2023
Scheduled downtime: Globus login with NCAR RDA identity unavailable January 24
19 January 2023
New RDA website launching next Tuesday, Jan 24
The NCAR Research Data Archive is launching a new, redesigned website next Tuesday, Jan 24 that will provide a more modern, simplified, and improved user experience. The new site is more accessible and mobile-friendly, thus users will be able to browse and use most features of the site from a mobile phone or tablet. All of our data access, download, search, and discoverability features will remain as they are on the current website. We invite you to explore the new site once it has launched, and send your feedback and/or questions via email to rdahelp@ucar.edu or through the RDA Support Portal.
 |
| New RDA homepage |
 |
| Redesigned RDA dataset page |
 |
| Data search and discovery user interface |
02 December 2022
ERA5 Reanalysis Model Level Data in the RDA
NCAR/CISL/DECS personnel are in the process of downloading ECMWF Meteorological Archival and Retrieval System (MARS) ERA5 GRIB 2 and GRIB 1 model level T639 spherical harmonic and N320 reduced Gaussian data, and synthesizing or interpolating this to a 1280 longitude x 640 Gaussian latitude 0.28125° gridded dataset with 137 levels. In addition, the final gridded product is being converted to CF-compliant compressed netCDF4/HDF5 format. The period of coverage is 1979 to the present time. Please see ds633.6,
ERA5 Reanalysis Model Level Data
for processing and archiving status. Data is made available in netCDF4/HDF5 format only, via the RDA website and NCAR/CISL GLADE storage. A snapshot of the metadata content of a typical file is presented below.
netcdf e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105 {
dimensions:
time = UNLIMITED ; // (6 currently)
level = 137 ;
latitude = 640 ;
longitude = 1280 ;
half_level = 138 ;
variables:
float T(time, level, latitude, longitude) ;
T:long_name = "Temperature" ;
T:standard_name = "air_temperature" ;
T:short_name = "t" ;
T:units = "K" ;
T:original_format = "WMO GRIB 2" ;
T:wmo_discipline = 0 ;
T:wmo_master_table = 5 ;
T:wmo_local_table = 0 ;
T:wmo_center = 98 ;
T:wmo_parameter_category = 0 ;
T:wmo_parameter_number = 0 ;
T:_FillValue = 9.999e+20f ;
T:missing_value = 9.999e+20f ;
T:minimum_value = 175.1045f ;
T:maximum_value = 314.6406f ;
T:grid_specification = "0.28125° x ~0.28125° from 0E to 359.71875E and 89.78488N to 89.78488S (1280 x 640 Longitude/Gaussian Latitude), ~31km at Equator" ;
T:original_data_representation = "T639 spherical harmonics, ECMWF Meteorological Archival and Retrieval System (MARS)" ;
T:rda_dataset = "ds633.6" ;
T:rda_dataset_url = "https:/rda.ucar.edu/datasets/ds633.6/" ;
T:rda_dataset_doi = "DOI: 10.5065/XV5R-5344" ;
T:rda_dataset_group = "ERA5 atmospheric model level analysis [netCDF4]" ;
T:QuantizeGranularBitGroomNumberOfSignificantDigits = 7 ;
double a_half(half_level) ;
a_half:long_name = "half level a coefficient" ;
a_half:alternate_name = "interface a coefficient" ;
a_half:short_name = "a_half" ;
a_half:units = "Pa" ;
a_half:source = "https://confluence.ecmwf.int/display/UDOC/L137+model+level+definitions, column 2" ;
double a_model(level) ;
a_model:long_name = "model level a coefficient" ;
a_model:short_name = "a_model" ;
a_model:units = "Pa" ;
a_model:source = "https://www.ecmwf.int/en/elibrary/9203-ifs-documentation-cy40r1-part-iii-dynamics-and-numerical-procedures, pdf download, page 7, in Section 2.2.1 Vertical Discretization, model level a coefficients are found by average of adjacent half levels (interfaces)" ;
a_model:for_an_alternative_derivation_see = "Simmons, A. J., and D. M. Burridge, 1981: An energy and angular-momentum conserving finite-difference scheme and hybrid vertical coordinates. Mon. Wea. Rev., 109, 758-766.; Trenberth, K. E., J. C. Berry, and L. E. Buja, 1993: Vertical interpolation and truncation of model-coordinate data. NCAR Technical Note NCAR/TN-396+STR, 54 pp." ;
double b_half(half_level) ;
b_half:long_name = "half level b coefficient" ;
b_half:alternate_name = "interface b coefficient" ;
b_half:short_name = "b_half" ;
b_half:units = "1" ;
b_half:source = "https://confluence.ecmwf.int/display/UDOC/L137+model+level+definitions, column 3" ;
double b_model(level) ;
b_model:long_name = "model level b coefficient" ;
b_model:short_name = "b_model" ;
b_model:units = "1" ;
b_model:source = "https://www.ecmwf.int/en/elibrary/9203-ifs-documentation-cy40r1-part-iii-dynamics-and-numerical-procedures, pdf download, page 7, in Section 2.2.1 Vertical Discretization, model level b coefficients are found by average of adjacent half levels (interfaces)" ;
b_model:for_an_alternative_derivation_see = "Simmons, A. J., and D. M. Burridge, 1981: An energy and angular-momentum conserving finite-difference scheme and hybrid vertical coordinates. Mon. Wea. Rev., 109, 758-766.; Trenberth, K. E., J. C. Berry, and L. E. Buja, 1993: Vertical interpolation and truncation of model-coordinate data. NCAR Technical Note NCAR/TN-396+STR, 54 pp." ;
int half_level(half_level) ;
half_level:long_name = "half level" ;
half_level:alternate_name = "interface" ;
half_level:short_name = "hlev" ;
half_level:formula = "p = a + b*ps" ;
half_level:order = "top of atmosphere to surface" ;
half_level:units = "1" ;
double latitude(latitude) ;
latitude:long_name = "gaussian latitude" ;
latitude:standard_name = "latitude" ;
latitude:short_name = "lat" ;
latitude:units = "degrees_north" ;
int level(level) ;
level:long_name = "model level" ;
level:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
level:short_name = "lev" ;
level:formula = "p = a + b*ps" ;
level:order = "near top of atmosphere to near surface" ;
level:units = "1" ;
double longitude(longitude) ;
longitude:long_name = "longitude" ;
longitude:standard_name = "longitude" ;
longitude:short_name = "lon" ;
longitude:units = "degrees_east" ;
int time(time) ;
time:long_name = "time" ;
time:standard_name = "time" ;
time:units = "hours since 1900-01-01 00:00:00" ;
time:calendar = "gregorian" ;
int utc_date(time) ;
utc_date:long_name = "UTC date yyyy-mm-dd hh:00:00 as yyyymmddhh" ;
utc_date:units = "Gregorian_year month day hour" ;
double weight(latitude) ;
weight:long_name = "gaussian weight" ;
weight:short_name = "weight" ;
weight:units = "1" ;
double zero(latitude) ;
zero:long_name = "gaussian zero" ;
zero:short_name = "zero" ;
zero:units = "1" ;
// global attributes:
:RDA_DATASET = "ds633.6" ;
:RDA_DATASET_URL = "https:/rda.ucar.edu/datasets/ds633.6/" ;
:RDA_DATASET_DOI = "DOI: 10.5065/XV5R-5344" ;
:RDA_DATASET_GROUP = "ERA5 atmospheric model level analysis [netCDF4]" ;
:DATA_SOURCE = "Downloaded by NCAR/CISL/DECS from ECMWF\"s Meteorological Archival and Retrieval System (MARS): ERA5 GRIB 2 data (T639 Spherical harmonics and N320 reduced Gaussian data)" ;
:DOWNLOAD_SPECIALIST = "Chi-Fan Shih, NCAR/CISL/DECS, chifan@ucar.edu" ;
:DATA_INTERPOLATION = "Computed by NCAR/CISL/DECS: ERA5 GRIB 2 T639/N320 data synthesized or interpolated to 1280 longitude x 640 Gaussian latitude GRIB 2 gridded data using emoslib-4.3.9 (compiled with cmake with grib_api-1.14.7 and fftw-3.3.8 support for gnu/gfortran-9.1.0). NOTE: u-component and v-component of horizontal wind derived from spectral vorticity and divergence." ;
:COMPUTATIONAL_PLATFORM = "cheyenne supercomputer, NCAR-Wyoming Supercomputing Center (NWSC), https://www2.cisl.ucar.edu/" ;
:COMPUTATIONAL_SPECIALIST = "Dave Stepaniak, NCAR/CISL/DECS, davestep@ucar.edu" ;
:NETCDF_CONVERSION = "Conversion by NCAR/CISL/DECS: Conversion from synthesized or interpolated 1280 longitude x 640 Gaussian latitude GRIB 2 gridded data to netCDF4/HDF5." ;
:NETCDF_VERSION = "4.8.1" ;
:Conventions = "CF-1.6" ;
:NETCDF_COMPRESSION = "Precision-preserving compression to netCDF4/HDF5 using NCO (see \"history\" and \"NCO\" global attributes below for specifics)." ;
:CONVERSION_AND_COMPRESSION_PLATFORM = "cheyenne node employed: Linux r14i0n16 4.12.14-95.51-default #1 SMP Fri Apr 17 08:14:12 UTC 2020 (c6bab98) x86_64 x86_64 x86_64 GNU/Linux" ;
:CONVERSION_AND_COMPRSSION_DATE = "Fri Dec 2 12:05:36 MST 2022" ;
:CONVERSION_AND_COMPRESSION_SPECIALIST = "Dave Stepaniak, NCAR/CISL/DECS, davestep@ucar.edu" ;
:history = "Fri Dec 2 12:05:50 2022: ncks -4 --ppc T=7 e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105.unc.nc e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105.nc" ;
:NCO = "netCDF Operators version 5.0.3 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
17 October 2022
RDA Data Services Unavailable Oct 17-20, 2022
Due to scheduled maintenance on the HPC systems at NCAR, all RDA data access services, including downloads, data subsetting, and THREDDS data server access, will be unavailable OCT 17–20, 2022. Please follow our Twitter profile (@NCAR_RDA) for updates.
10 August 2022
Avoid the need to download RDA datasets to your local system
From the NCAR CISL Daily Bulletin:
New data analysis allocation opportunity for universities
17 June 2022
Google Problem for RDA Registration and Authentication
As of June 21, 2022, this problem has been resolved and users may resume using gmail.com and G Suite email addresses.
If you are a gmail.com or G Suite user, this information applies to you:
Beginning in late May, Google began blocking our automated messages as "unsolicited spam". We currently don't know when this problem will be fixed.
If you are a new user:- You won't be able to register a gmail.com address - we are blocking the ability to do this on our end.
- If your organization (other than UCAR) uses G Suite, you won't be able to receive our automated messages and therefore won't be able to successfully register your organization email address. UCAR email addresses are not affected by this problem.
- We have no way of knowing which organizations use G Suite, so we cannot block this on our end. If you don't receive an activation code from us after 48 hours, it means that your organization blocked our message.
- You won't be able to change your email address to a gmail.com address - we are block the ability to do this on our end.
- If you change your email address to a G Suite email address, the change will not be successful because you won't receive our automated verification message.
- We have no way of knowing which organizations use G Suite, so we cannot block this on our end. If you don't receive a verification message from us within an hour, it means that your organization blocked our message.
- If you are already registered with a gmail.com or G Suite address, you can continue to use that email address as long as you know your password. If you forget your password, you will not be able to reset it, and in that case, you will need to register a new, non-gmail.com, non-G-Suite email address to regain access.
03 May 2022
RDA data services unavailable May 9-13, 2022
Due to scheduled maintenance on the HPC systems at NCAR, all RDA data access services, including downloads, data subsetting, and THREDDS data server access, will be unavailable May 9–13, 2022. Please follow our Twitter profile (@NCAR_RDA) for updates.
14 April 2022
ECMWF and JMA Reanalyses in the RDA and on CISL HPC GLADE
Over the past two decades, independent state-of-the-art international atmospheric reanalysis programs have been conducted by ECMWF (European Centre for Medium-Range Weather Forecasts) and JMA (Japan Meteorological Agency). The most recent reanalyses released by these programs are ERA5 (from ECMWF, spanning 1950 to the current time), and JRA-55 (from JMA, spanning 1958 to the current time). In addition, ECMWF produces near real-time ultra high resolution operational atmospheric forecasts that are an important complement to ERA5.
In a collaborative effort within CISL ISD, DECS (the Data Engineering and Curation Section), downloads, processes, archives, and provides web-based access and services for ERA5, JRA-55, and ECMWF operational forecast data to a wide cross-section of users and organizations located worldwide. In an ongoing effort to improve the generality and usability of these datasets, processing by DECS personnel using CISL HPC resources include quality control, regridding to standardized lat-lon grids, restructuring incoming data into single-parameter time series, and in the case of ECMWF data, conversion of GRIB format to CF-conforming compressed netCDF4-HDF5 files. In addition, extensive logistical support and scientific consulting is provided via several platforms, including the NCAR Research Data Archive (RDA) Helpdesk
These efforts support a broad range of climate diagnostics, model development and validation, and intercomparison projects to name a few. The ERA5, JRA-55, and ECMWF operational forecast data is organized into 17 datasets within the Research Data Archive which is managed by DECS. The total volume of these datasets is 2 PB, all of which is stored on the CISL GLADE shared file system. Metrics for web-access alone over the last 15 months indicate that 3600 users have downloaded 2.7 PB of RDA ERA5, JRA-55, and ECMWF operational forecast data. We encourage CISL HPC users to explore this archive via the RDA website, and especially to take advantage of the availability on GLADE. Three RDA datasets in particular are good entry points to begin your investigations:
- ERA5 Reanalysis (0.25 Degree Latitude-Longitude Grid
- GLADE root: /gpfs/fs1/collections/rda/data/ds633.0
- Japanese 55-year Reanalysis, Daily 3-Hourly and 6-Hourly Data
- GLADE root: /gpfs/fs1/collections/rda/data/ds628.0
- ECMWF IFS CY41r2 High-Resolution Operational Forecasts
- GLADE root: /gpfs/fs1/collections/rda/data/ds113.1
HPC users will find that the above links provide more detailed listings of GRIB and netCDF4-HDF5 files located on GLADE, supporting metadata, documentation, and cross references to related RDA ERA5 and JRA-55 datasets including monthly means and variances, back extensions of ERA5 data to 1950, JRA-55 near real-time data, and supplementary and derived products. We endeavor that this serves the HPC community well and in many capacities.
19 July 2021
RDA services unavailable July 25 - 29, 2021
Due to scheduled maintenance on the HPC systems at NCAR, all RDA data access services, including downloads, data subsetting, and THREDDS data server access, will be unavailable July 25–29, 2021. Please follow our Twitter profile (@NCAR_RDA) for updates.
17 June 2021
403 Forbidden for Slow Connections
If you find yourself receiving 403 forbidden HTTP responses when downloading data from the RDA, it may be the result of a slow connection.
The RDA currently limits connections that are below 200 KB/s averaged over a two minutes. If a user stays below this threshold, they will be temporarily blocked from accessing our servers for 30 minutes.
Once blocked, downloaded files may appear to be smaller than expected and contain HTML. This occurs, because the server is sending the RDA's 401 forbidden page instead of the file.
At this point, there are two options to overcome this problem. The first would be to improve your network speed. Often, using a VPN or proxy server can overcome this issue
Alternatively, you can consider using our Globus download option. On each data access page, you can find an option to download the files using Globus as seen in the screenshot below. Globus is a service that transfers files using gridFTP and has built in error tolerance and unsupervised downloads.
