ERA5 Reanalysis Model Level Data in the RDA

NCAR/CISL/DECS personnel are in the process of downloading ECMWF Meteorological Archival and Retrieval System (MARS) ERA5 GRIB 2 and GRIB 1 model level T639 spherical harmonic and N320 reduced Gaussian data, and synthesizing or interpolating this to a 1280 longitude x 640 Gaussian latitude 0.28125° gridded dataset with 137 levels. In addition, the final gridded product is being converted to CF-compliant compressed netCDF4/HDF5 format. The period of coverage is 1979 to the present time. Please see ds633.6,

ERA5 Reanalysis Model Level Data

for processing and archiving status. Data is made available in netCDF4/HDF5 format only, via the RDA website and NCAR/CISL GLADE storage. A snapshot of the metadata content of a typical file is presented below.

netcdf e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105 {
dimensions:
        time = UNLIMITED ; // (6 currently)
        level = 137 ;
        latitude = 640 ;
        longitude = 1280 ;
        half_level = 138 ;
variables:
        float T(time, level, latitude, longitude) ;
                T:long_name = "Temperature" ;
                T:standard_name = "air_temperature" ;
                T:short_name = "t" ;
                T:units = "K" ;
                T:original_format = "WMO GRIB 2" ;
                T:wmo_discipline = 0 ;
                T:wmo_master_table = 5 ;
                T:wmo_local_table = 0 ;
                T:wmo_center = 98 ;
                T:wmo_parameter_category = 0 ;
                T:wmo_parameter_number = 0 ;
                T:_FillValue = 9.999e+20f ;
                T:missing_value = 9.999e+20f ;
                T:minimum_value = 175.1045f ;
                T:maximum_value = 314.6406f ;
                T:grid_specification = "0.28125° x ~0.28125° from 0E to 359.71875E and 89.78488N to 89.78488S (1280 x 640 Longitude/Gaussian Latitude), ~31km at Equator" ;
                T:original_data_representation = "T639 spherical harmonics, ECMWF Meteorological Archival and Retrieval System (MARS)" ;
                T:rda_dataset = "ds633.6" ;
                T:rda_dataset_url = "https:/rda.ucar.edu/datasets/ds633.6/" ;
                T:rda_dataset_doi = "DOI: 10.5065/XV5R-5344" ;
                T:rda_dataset_group = "ERA5 atmospheric model level analysis [netCDF4]" ;
                T:QuantizeGranularBitGroomNumberOfSignificantDigits = 7 ;
        double a_half(half_level) ;
                a_half:long_name = "half level a coefficient" ;
                a_half:alternate_name = "interface a coefficient" ;
                a_half:short_name = "a_half" ;
                a_half:units = "Pa" ;
                a_half:source = "https://confluence.ecmwf.int/display/UDOC/L137+model+level+definitions, column 2" ;
        double a_model(level) ;
                a_model:long_name = "model level a coefficient" ;
                a_model:short_name = "a_model" ;
                a_model:units = "Pa" ;
                a_model:source = "https://www.ecmwf.int/en/elibrary/9203-ifs-documentation-cy40r1-part-iii-dynamics-and-numerical-procedures, pdf download, page 7, in Section 2.2.1 Vertical Discretization, model level a coefficients are found by average of adjacent half levels (interfaces)" ;
                a_model:for_an_alternative_derivation_see = "Simmons, A. J., and D. M. Burridge, 1981: An energy and angular-momentum conserving finite-difference scheme and hybrid vertical coordinates. Mon. Wea. Rev., 109, 758-766.;  Trenberth, K. E., J. C. Berry, and L. E. Buja, 1993: Vertical interpolation and truncation of model-coordinate data. NCAR Technical Note NCAR/TN-396+STR, 54 pp." ;
        double b_half(half_level) ;
                b_half:long_name = "half level b coefficient" ;
                b_half:alternate_name = "interface b coefficient" ;
                b_half:short_name = "b_half" ;
                b_half:units = "1" ;
                b_half:source = "https://confluence.ecmwf.int/display/UDOC/L137+model+level+definitions, column 3" ;
        double b_model(level) ;
                b_model:long_name = "model level b coefficient" ;
                b_model:short_name = "b_model" ;
                b_model:units = "1" ;
                b_model:source = "https://www.ecmwf.int/en/elibrary/9203-ifs-documentation-cy40r1-part-iii-dynamics-and-numerical-procedures, pdf download, page 7, in Section 2.2.1 Vertical Discretization, model level b coefficients are found by average of adjacent half levels (interfaces)" ;
                b_model:for_an_alternative_derivation_see = "Simmons, A. J., and D. M. Burridge, 1981: An energy and angular-momentum conserving finite-difference scheme and hybrid vertical coordinates. Mon. Wea. Rev., 109, 758-766.;  Trenberth, K. E., J. C. Berry, and L. E. Buja, 1993: Vertical interpolation and truncation of model-coordinate data. NCAR Technical Note NCAR/TN-396+STR, 54 pp." ;
        int half_level(half_level) ;
                half_level:long_name = "half level" ;
                half_level:alternate_name = "interface" ;
                half_level:short_name = "hlev" ;
                half_level:formula = "p = a + b*ps" ;
                half_level:order = "top of atmosphere to surface" ;
                half_level:units = "1" ;
        double latitude(latitude) ;
                latitude:long_name = "gaussian latitude" ;
                latitude:standard_name = "latitude" ;
                latitude:short_name = "lat" ;
                latitude:units = "degrees_north" ;
        int level(level) ;
                level:long_name = "model level" ;
                level:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
                level:short_name = "lev" ;
                level:formula = "p = a + b*ps" ;
                level:order = "near top of atmosphere to near surface" ;
                level:units = "1" ;
        double longitude(longitude) ;
                longitude:long_name = "longitude" ;
                longitude:standard_name = "longitude" ;
                longitude:short_name = "lon" ;
                longitude:units = "degrees_east" ;
        int time(time) ;
                time:long_name = "time" ;
                time:standard_name = "time" ;
                time:units = "hours since 1900-01-01 00:00:00" ;
                time:calendar = "gregorian" ;
        int utc_date(time) ;
                utc_date:long_name = "UTC date yyyy-mm-dd hh:00:00 as yyyymmddhh" ;
                utc_date:units = "Gregorian_year month day hour" ;
        double weight(latitude) ;
                weight:long_name = "gaussian weight" ;
                weight:short_name = "weight" ;
                weight:units = "1" ;
        double zero(latitude) ;
                zero:long_name = "gaussian zero" ;
                zero:short_name = "zero" ;
                zero:units = "1" ;

// global attributes:
                :RDA_DATASET = "ds633.6" ;
                :RDA_DATASET_URL = "https:/rda.ucar.edu/datasets/ds633.6/" ;
                :RDA_DATASET_DOI = "DOI: 10.5065/XV5R-5344" ;
                :RDA_DATASET_GROUP = "ERA5 atmospheric model level analysis [netCDF4]" ;
                :DATA_SOURCE = "Downloaded by NCAR/CISL/DECS from ECMWF\"s Meteorological Archival and Retrieval System (MARS): ERA5 GRIB 2 data (T639 Spherical harmonics and N320 reduced Gaussian data)" ;
                :DOWNLOAD_SPECIALIST = "Chi-Fan Shih, NCAR/CISL/DECS, chifan@ucar.edu" ;
                :DATA_INTERPOLATION = "Computed by NCAR/CISL/DECS: ERA5 GRIB 2 T639/N320 data synthesized or interpolated to 1280 longitude x 640 Gaussian latitude GRIB 2 gridded data using emoslib-4.3.9 (compiled with cmake with grib_api-1.14.7 and fftw-3.3.8 support for gnu/gfortran-9.1.0). NOTE: u-component and v-component of horizontal wind derived from spectral vorticity and divergence." ;
                :COMPUTATIONAL_PLATFORM = "cheyenne supercomputer, NCAR-Wyoming Supercomputing Center (NWSC), https://www2.cisl.ucar.edu/" ;
                :COMPUTATIONAL_SPECIALIST = "Dave Stepaniak, NCAR/CISL/DECS, davestep@ucar.edu" ;
                :NETCDF_CONVERSION = "Conversion by NCAR/CISL/DECS: Conversion from synthesized or interpolated 1280 longitude x 640 Gaussian latitude GRIB 2 gridded data to netCDF4/HDF5." ;
                :NETCDF_VERSION = "4.8.1" ;
                :Conventions = "CF-1.6" ;
                :NETCDF_COMPRESSION = "Precision-preserving compression to netCDF4/HDF5 using NCO (see \"history\" and \"NCO\" global attributes below for specifics)." ;
                :CONVERSION_AND_COMPRESSION_PLATFORM = "cheyenne node employed: Linux r14i0n16 4.12.14-95.51-default #1 SMP Fri Apr 17 08:14:12 UTC 2020 (c6bab98) x86_64 x86_64 x86_64 GNU/Linux" ;
                :CONVERSION_AND_COMPRSSION_DATE = "Fri Dec  2 12:05:36 MST 2022" ;
                :CONVERSION_AND_COMPRESSION_SPECIALIST = "Dave Stepaniak, NCAR/CISL/DECS, davestep@ucar.edu" ;
                :history = "Fri Dec  2 12:05:50 2022: ncks -4 --ppc T=7 e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105.unc.nc e5.oper.an.ml.0_5_0_0_0_t.regn320sc.1999050100_1999050105.nc" ;
                :NCO = "netCDF Operators version 5.0.3 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
                

RDA Data Services Unavailable Oct 17-20, 2022

 Due to scheduled maintenance on the HPC systems at NCAR, all RDA data access services, including downloads, data subsetting, and THREDDS data server access, will be unavailable OCT 17–20, 2022.  Please follow our Twitter profile (@NCAR_RDA) for updates.

Avoid the need to download RDA datasets to your local system

 

From the NCAR CISL Daily Bulletin: 

New data analysis allocation opportunity for universities

University researchers have a new opportunity to use NCAR’s data analysis and storage resources for their work in Earth system science with this week’s introduction of Data Analysis allocations. These new allocations are available to researchers from any eligible institution regardless of the source of funding for the planned analysis, as long as they identify one or more NCAR-curated data sets that are essential to their analysis activities.

Researchers who receive Data Analysis allocations get access to NCAR’s Casper cluster and the Campaign Storage file system to allow them to conduct their analyses. See our university allocations documentation on the Advanced Research Computing portal for details on this new opportunity and other recent allocation policy updates. 

Google Problem for RDA Registration and Authentication

As of June 21, 2022, this problem has been resolved and users may resume using gmail.com and G Suite email addresses.

If you are a gmail.com or G Suite user, this information applies to you:

Beginning in late May, Google began blocking our automated messages as "unsolicited spam". We currently don't know when this problem will be fixed.

If you are a new user:

• You won't be able to register a gmail.com address - we are blocking the ability to do this on our end.
• If your organization (other than UCAR) uses G Suite, you won't be able to receive our automated messages and therefore won't be able to successfully register your organization email address. UCAR email addresses are not affected by this problem.
• We have no way of knowing which organizations use G Suite, so we cannot block this on our end. If you don't receive an activation code from us after 48 hours, it means that your organization blocked our message.

If you are an existing user:

• You won't be able to change your email address to a gmail.com address - we are block the ability to do this on our end.
• If you change your email address to a G Suite email address, the change will not be successful because you won't receive our automated verification message.
• We have no way of knowing which organizations use G Suite, so we cannot block this on our end. If you don't receive a verification message from us within an hour, it means that your organization blocked our message.
• If you are already registered with a gmail.com or G Suite address, you can continue to use that email address as long as you know your password. If you forget your password, you will not be able to reset it, and in that case, you will need to register a new, non-gmail.com, non-G-Suite email address to regain access.

RDA data services unavailable May 9-13, 2022

Due to scheduled maintenance on the HPC systems at NCAR, all RDA data access services, including downloads, data subsetting, and THREDDS data server access, will be unavailable May 9–13, 2022.  Please follow our Twitter profile (@NCAR_RDA) for updates.

ECMWF and JMA Reanalyses in the RDA and on CISL HPC GLADE

Over the past two decades, independent state-of-the-art international atmospheric reanalysis programs have been conducted by ECMWF (European Centre for Medium-Range Weather Forecasts) and JMA (Japan Meteorological Agency). The most recent reanalyses released by these programs are ERA5 (from ECMWF, spanning 1950 to the current time), and JRA-55 (from JMA, spanning 1958 to the current time). In addition, ECMWF produces near real-time ultra high resolution operational atmospheric forecasts that are an important complement to ERA5.

In a collaborative effort within CISL ISD, DECS (the Data Engineering and Curation Section), downloads, processes, archives, and provides web-based access and services for ERA5, JRA-55, and ECMWF operational forecast data to a wide cross-section of users and organizations located worldwide. In an ongoing effort to improve the generality and usability of these datasets, processing by DECS personnel using CISL HPC resources include quality control, regridding to standardized lat-lon grids, restructuring incoming data into single-parameter time series, and in the case of ECMWF data, conversion of GRIB format to CF-conforming compressed netCDF4-HDF5 files. In addition, extensive logistical support and scientific consulting is provided via several platforms, including the NCAR Research Data Archive (RDA) Helpdesk

These efforts support a broad range of climate diagnostics, model development and validation, and intercomparison projects to name a few. The ERA5, JRA-55, and ECMWF operational forecast data is organized into 17 datasets within the Research Data Archive which is managed by DECS. The total volume of these datasets is 2 PB, all of which is stored on the CISL GLADE shared file system. Metrics for web-access alone over the last 15 months indicate that 3600 users have downloaded 2.7 PB of RDA ERA5, JRA-55, and ECMWF operational forecast data. We encourage CISL HPC users to explore this archive via the RDA website, and especially to take advantage of the availability on GLADE. Three RDA datasets in particular are good entry points to begin your investigations:

• ERA5 Reanalysis (0.25 Degree Latitude-Longitude Grid
• GLADE root: /gpfs/fs1/collections/rda/data/ds633.0

• Japanese 55-year Reanalysis, Daily 3-Hourly and 6-Hourly Data
• GLADE root: /gpfs/fs1/collections/rda/data/ds628.0

• ECMWF IFS CY41r2 High-Resolution Operational Forecasts
• GLADE root: /gpfs/fs1/collections/rda/data/ds113.1

HPC users will find that the above links provide more detailed listings of GRIB and netCDF4-HDF5 files located on GLADE, supporting metadata, documentation, and cross references to related RDA ERA5 and JRA-55 datasets including monthly means and variances, back extensions of ERA5 data to 1950, JRA-55 near real-time data, and supplementary and derived products. We endeavor that this serves the HPC community well and in many capacities.