02 August 2019

RDA ERA5 now spans 1979 through early 2019.

Currently RDA ERA5 is represented by two datasets.

0.25° ERA5 1979-2001 is available online as netCDF in ds633.0:

ERA5 Reanalysis (0.25 Degree Latitude-Longitude Grid)

which is being updated continuously and is expected to extend through at least mid 2019 by late September of this year. The original RDA ERA5 dataset is ds630.0:

ERA5 Reanalysis

which utilizes a 0.281° 1280x640 regular Gaussian grid and is available online as netCDF spanning 2002-2019.

We encourage users to employ the 0.25° version of ERA5 (ds633.0) as it is uncertain whether ds630.0 will be extended. In addition, note that the online data format is netCDF — please refer to the previous blog post for further information.

22 July 2019

NCAR RDA supported ERA5, ECMWF operational, and future reanalysis products to be provided in NetCDF format

The Research Data Archive at NCAR plans to restructure and translate future reanalysis products, selected operational NWP products, and selected in-situ observational data products into CF-compliant NetCDF from the native NWP formats moving forward.  This action is being taken for a variety of reasons as described below.

NetCDF provides:

Additionally, restructuring and translating the data from native GRIB

formats to CF-compliant NetCDF solves interoperability issues
created by non-standard GRIB1 and GRIB2 encoding practices used by NWP centers to serve their internal use case needs.

For WRF users: Community efforts are underway to develop tools to translate NetCDF data into WRF-intermediate format.  Currently ungrib is the only tool provided to do this in the WRF preprocessing system (WPS) software package.  Some WRF users have developed their own tools and workflows to translate data from NetCDF into the WRF-intermediate format.

26 June 2019

Research Data Archive at NCAR certified as a CoreTrustSeal Repository by the World Data System

The World Data System (WDS), an international consortium promoting scientific data integrity, trust and stewardship, has recognized the Research Data Archive (RDA) at NCAR as a data center meeting the requirements of the CoreTrustSeal (https://www.coretrustseal.org/) .  This universal catalogue of criteria reflects the core characteristics of trustworthy data repositories and indicates that the RDA meets the highest international requirements for quality assured scientific data, services, and data archiving and stewardship.
A list of certified data centers can be found here: https://www.coretrustseal.org/why-certification/certified-repositories/

20 May 2019

ERA5 1979 onward becoming available in the RDA

CISL DECS is processing ERA5, 1979 onward, into netCDF4/HDF5 files containing single parameter 0.25° gridded time series. Please refer to ds633.0,

ERA5 Reanalysis (0.25 Degree Latitude-Longitude Grid)

for further information and availability.

Please note: DECS is producing a CF 1.6 compliant netCDF-4/HDF5 version of ERA5 for the CISL RDA at NCAR. The netCDF-4/HDF5 version is the de facto RDA ERA5 online data format. The GRIB1 data format is only available via NCAR's High Performance Storage System (HPSS). There is a one-to-one correspondence between the netCDF-4/HDF5 and GRIB1 files, with as much GRIB1 metadata as possible incorporated into the attributes of the netCDF-4/HDF5 counterpart.

06 December 2018

NCAR RDA at AGU 2018 Fall Meeting

Sophie Hou, Tom Cram, and Doug Schuster will be representing the RDA at the AGU 2018 fall meeting.  A schedule of their events is provided below:

02 April 2018

Notice: new Globus endpoints

The RDA data collections were recently moved to a new filesystem (and new directory paths), and as a result of the move, the two shared Globus endpoints providing access to RDA data products were rendered obsolete.  To remedy this, we have created two new shared endpoints and have transferred all active Globus shares to the new endpoints.

The caveat to this change is the new endpoints have new endpoint IDs, and this will have an impact on users who automate Globus transfers from RDA shared endpoints in their workflow scripts.  For those users who have automated transfers set up, please update your scripts to the new endpoint IDs as follows:

Endpoint name: NCAR RDA Dataset Archive (rda#datashare)
Old endpoint ID: db57de42-6d04-11e5-ba46-22000b92c6ec
New endpoint ID: 2869611a-36aa-11e8-b95e-0ac6873fc732

Endpoint name: NCAR RDA Data Requests (rda#data_request)
Old endpoint ID: d20e610e-6d04-11e5-ba46-22000b92c6ec
New endpoint ID: 68823254-36aa-11e8-b95e-0ac6873fc732

Please e-mail us at rdahelp@ucar.edu if you have questions.


19 March 2018

Alternate access to NCEP data products through NCEP Data Portals

The RDA republishes many NCEP data products.  Users with urgent data needs during extended RDA downtimes may obtain the data directly from NCEP servers.

The 10-day archive of GFS products (including GDAS/FNL analysis grids) can be found at the real-time site http://www.nco.ncep.noaa.gov/pmb/products/gfs/

GDAS data up to ~ 9 months old may be found at this online archive  https://nomads.ncdc.noaa.gov/data/gdas/ (which may be decommissioned soon.)

THREDDS users can use https://www.ncei.noaa.gov/thredds/model/model.html

Older data can be requested from this long-term archive https://www.ncdc.noaa.gov/data-access/model-data

Some of the data is available for immediate download.  For example, you can obtain GFS analysis files from ftp://nomads.ncdc.noaa.gov/GFS/analysis_only/201803/20180315/

Note that data read from tape is organized and delivered in daily (01Jan2001​–21Feb2012) or hourly (13Feb2012-present) tar files that can be very large and include things that you may not need.  For instance, one 6-hourly GDAS tar file is 47 GB and includes the initial hour and all forecast hours for global 0.25, 0.5, and 1.0 degree resolution grids.

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-data-assimilation-system-gdas

Subscribe to this blog via RSS or follow us by email (see right column for widget) to get the latest updates.

08 December 2017

NCAR data experts seek input at AGU fall meeting for summer 2018 workshop on digital data repository service

NCAR data management experts will engage the community in a poster session on December 11 at the AGU 2017 Fall Meeting to help shape a 2018 National Science Foundation workshop focused on developing requirements and expectations for a prospective new digital repository service.
The envisioned Geoscience Digital Data Resource and Repository Service (GeoDaRRS) would complement existing NSF-funded data facilities by providing data management planning support resources for the general community and repository services for researchers who have data that do not fit in any existing repository. GeoDaRRS would support NSF-funded researchers in meeting data archiving requirements set by the NSF and publishers for geosciences, thereby ensuring the availability of digital data for use and reuse in scientific research going forward.
See this abstract on the AGU Fall Meeting site for more information.
For additional information on the workshop, please see the workshop website.

19 July 2017

Accessing RDA OPeNDAP endpoints with authentication

Update:
As an alternative to the steps below, it may be simpler specify the username and password within the URL to satisfy the authentication.

For example,
https://USERNAME%40DOMAIN:PASSWORD@rda.ucar.edu/thredds/dodsC/files/g/ds083.3/2016/201602/gdas1.fnl0p25.2016020218.f00.grib2'

For me, it might look like,

https://rpconroy%40ucar.edu:MySuperSecretPassword@rda.ucar.edu/thredds/dodsC/files/g/ds083.3/2016/201602/gdas1.fnl0p25.2016020218.f00.grib2'

Note the %40 in place of the '@' in the email.


All RDA OPeNDAP supported datasets found under https://rda.ucar.edu/thredds now require user authentication for access.  Here are details on how to configure your system, such that most applications including NCL, will work properly with the authentication step (these instructions are based on what is provided by the hdfeos group):

1. RDA Registration and Setting Up Cookies

NCAR RDA OPeNDAP server access requires user registration and cookies. If you do not have RDA Login, please register first and have RDA username and password ready.

Once you have RDA Login username and password, and verify that they work by logging into https://rda.ucar.edu, you need to set up cookies to access data.  We highlight the key steps here.

To set up cookies properly, you will need the following 3 files in your home directory (e.g., /home/rdahelp on Linux or /Users/rdahelp on Mac assuming that rdahelp is your UNIX system's login name).
  1. /home/rdahelp/.netrc
  2. /home/rdahelp/.rda_cookies
  3. /home/rdahelp/.dodsrc
1.1 CREATE .NETRC FILE FOR LOGIN / PASSWORD
The first file .netrc should have RDA login and password. For example, if your username is rdahelp and password is 1234abcd, the file should have the following line.
machine rda.ucar.edu login rdahelp password 1234abcd
Please edit the above file to match your username and password. Since the file contains the password, make sure that others cannot see it by changing permission.


$chmod go-rwx /home/rdahelp/.netrc
1.2 CREATE .RDA_COOKIES FILE USING CURL OR WGET
The second file, .rda_cookies, should be created automatically by either wget or curl command.
If you like to use wget, issue the following command to create the cookie file. 
$wget --load-cookies ~/.rda_cookies --save-cookies ~/.rda_cookies --auth-no-challenge=on --keep-session-cookies https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -O test.dds
If you like to use curl, issue the following command to create the cookie file. 
$curl -n -c ~/.rda_cookies -b ~/.rda_cookies -L -g --url https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -o test.dods
If the above command succeeds, you will get the following output when you check the content of test.dods file.
$more test.dods
Dataset {                                                        Int32 LatLon_Projection;                                    Float32 lat[lat = 181];                                          Float32 lon[lon = 360];                                                         ... }files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1
If you don't see the above output, your login/password and system is not working properly with NCAR RDA OPeNDAP server. You cannot access data until this step works. If you get the same output, you can delete the temporary output test.dods file.
1.3 CREATE .DODSRC FILE FOR .NETRC AND .RDA_COOKIES FILES
The final step is to create .dodsrc file if you don't already have one and add the following lines.
HTTP.COOKIEJAR=/home/rdahelp/.rda_cookies
HTTP.NETRC=/home/rdahelp/.netrc

2. Test NCL open and read of OPeNDAP data (NCL version 6.4.0 is required)

The following example shows an example of how to test if NCL can open and read an OPeNDAP file after the above steps have been completed.
$ ncl                                                             ncl 0> f = addfile("https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1","r")                     ncl 1> print(f)                                               Variable: f                                                       Type: file                                                         filename:    fnl_20000201_06_00                                   path: https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1                                         file global attributes:                                              Originating_or_generating_Center : US National Weather Service, National Centres for Environmental Prediction (NCEP)          Originating_or_generating_Subcenter : 0                            GRIB_table_version : 0,2                                          ... etc                         
In the above code, the url points to the remote FNL file being served by OPeNDAP. Use print command to check if the url file is opened successfully and to see what variables are available inside the file.

26 April 2017

Transferring custom file lists with Globus

The RDA has supported Globus file transfers for over two years, which enables users to transfer data files from RDA datasets to their destination endpoint using the Globus data transfer service (see http://ncarrda.blogspot.com/2015/06/transferring-rda-data-with-globus.html for details).  Transferring files using Globus is an efficient, reliable, and secure method to transfer large amounts of data from one endpoint to another.

But what if you're interested in a smaller or specific collection of files within a dataset?  In this scenario, the user would navigate to the Globus web app transfer interface and then browse through the files on the RDA shared endpoint to locate the files of interest.  This workflow, however, is not ideal from a search and discovery perspective, since users don't have the convenience of filtering the file sets via faceted browsing or viewing the supporting metadata alongside each file listed on the Globus web interface.

The standard Globus web app transfer interface.  In this workflow, users must browse through the full set of files within a RDA dataset to locate specific files to be transferred.

The good news: RDA users now have the ability to select and initiate Globus transfers of custom file sets directly from the RDA web interface without having to search for the files on the Globus transfer interface.  This method of transferring data uses the Globus Browse Endpoint helper API, and the only step required to be done by the user on the Globus web app is to select the destination endpoint that will receive the data files in the transfer.  The advantage to this approach is users are able to make use of the faceted search and discovery tools on the RDA website and avoid the bottleneck of making "blind" searches of the files on the Globus web interface.

How to make a Globus transfer of custom file sets


Starting from any RDA dataset page, go to the 'Data Access' tab, then select the 'Web file listing' link associated with any of the data products listed.  At this point, you may choose either the 'Faceted browse' or 'Complete file list' option, and then follow the instructions to locate your files of interest.

Procured file lists using the 'Complete file list' (left) and 'Faceted browse' search function (right) on the RDA website. Once a collection of files has been selected, the user initiates a Globus transfer by selecting the 'Globus' button.

To initiate a Globus transfer, select the files you wish to download, then select the 'Globus download' button.  This will redirect you to the Globus web app where you will authenticate with Globus and select the destination endpoint that will receive the data files.  After a destination endpoint has been chosen, select the 'Submit' button.  You will then be redirected back to the RDA website, at which point the data transfer will be submitted on your behalf.  A status message displaying the status of your data transfer will appear, and a notification e-mail will be sent to your RDA user e-mail address when the transfer has successfully completed.

The browse endpoint interface on the Globus web app, where users select the destination endpoint.

As a reminder, RDA users can log into the Globus web app with their RDA user e-mail login and password credentials; this is the recommended method for logging into Globus.  To do this, simply choose the 'NCAR RDA' organizational login on the Globus login page.