Integrating RDA data processing capabilities into a workflow

Do you have a workflow that leverages repeated subset requests from an RDA dataset or would you like to avoid continually filling out web forms? Would you like to download data once a request is ready without checking your email?  The RDA provides the capability for users to submit subset requests and download data programmatically through the external applications API.  The API provides users with the following capabilities:

• Get a summary of datasets that have subsetting available
• Get a list of parameters available for subsetting by dataset
• Submit a temporal, spatial, parameter subset request from a dataset
• Check on data request processing status
• Get a list of output files found in a completed data request (supports data transfer)
• Set up a Globus endpoint share for a completed data request
• Purge data request

A complete description of API capabilities is described here.

A couple of example applications have been developed in python that utilize the capabilities provided through the API.

• rdams-client.py:  The rdams-client python utility can be run by registered RDA users to get parameters available for subsetting by dataset, to submit subset requests on select gridded data sets, to check on the processing status of any subset request, and to download completed request output files to a local system.   This utility may be useful to users who want integrate data subset request submission and download as part of a broader workflow. Download and execute './rdams-client.py -help' for more details (make sure to set 'rdams-client.py' as executable).  For additional information, see the CISL rdams documentation page.
• rda-request-manager.py:  The rda-request-manager python utility can be run by registered RDA users to check RDA data request processing status, download completed request output files to a local system, and purge request files from the RDA data server.  This utility may be useful to users who submit any type of request through the web interface (subset, file format conversion, or restage data from tape for download), and would like to create a cron job to check on request processing status, and download requests as they become available, instead of waiting for email notifications and downloading data through the web interface.  Download and execute './rda-request-manager.py -help' for additional information (make sure to set 'rda-request-manager.py' as executable).

Disk Cleanup & Faster Data Transfers

Our transfer disks have been averaging 90% full. Occasionally, it has been so full that processing of new data orders has to be halted until there is more space.

In the interest of keeping your data requests flowing, we ask that you purge your data to free up disk space after you have downloaded it. Just click on the 'Purge request' button to let us know you are done.

Click on 'Purge request' after you download your data.

Please request data in chunks that you can reasonably download in the default 5 day expiration limit.

Users experience faster data transfer speeds using globus.

A few RDA users work in regions where internet speeds are especially slow. Just let us know and we can extend the 5 day expiration limit for you.  Also, consider having data shipped out to you on a hard drive.

The RDA User Dashboard

One of the "perks" that you get when you register with the RDA (besides the primary one - access to data) is that everything related to your data access activities can be presented to you in a convenient dashboard. To access the dashboard, sign in to the RDA and then click the "dashboard" link at the top of the page next to your username:

In your dashboard, you will see several sections, as shown below:

I'll go through these sections in detail, one-by-one.

User Profile
Click the "Edit/Change Profile" button to open your user profile and make changes to it. You can change your email address, password, and affiliation, and you can gain access to restricted data (if you meet the requirements for access).

Bookmarked Datasets
In this section, you will see a list of all of the datasets that you have bookmarked. You can click a dataset title in the list to go directly to that dataset.

You can bookmark a dataset by clicking the hollow star next to the dataset ID (under the title) on a dataset description page:

Once bookmarked, the star will change to solid gold and the bookmark will appear in your list of bookmarked datasets. You can remove a bookmarked dataset from the dashboard, or by clicking the gold star on the dataset description page.

Customized Data Requests
Customized data requests are special requests for data that you make. These may be subsets of the data, staging of files that only reside in our tape archive, or requests to convert from the native data format to some other format. You can manage various aspects of your data requests from the dashboard:

• Click the calendar to change the "Available Until" date. By default, data from special requests are purged (removed) from our server 5 days after they are prepared. You can extend the date by as much as two weeks, and you may request multiple extensions.
• Click the "X" if you have finished downloading your data and you want to remove the request from your list (this will also remove the data from our server).
• Click the envelope if you have any questions about your request. This will allow you to send a message to the dataset specialist.

You can navigate to your data request by clicking either the request number or the status (if the status is "Completed").

Globus Endpoints Shared With You
If you have requested any Globus transfers, for whole datasets and/or your customized data requests, then the shares will appear in this list. To start a data transfer, you can navigate to Globus by clicking the URL for a particular share. You can delete shares that you no longer need by clicking the "X".

Customized OPeNDAP Aggregations
Any customized OPeNDAP aggregations that you have created will appear here. You can copy and paste the URL of your aggregation into your OPeNDAP aware tools and scripts. You also can manage your aggregations:

• Click the magnifying glass to see more detailed information about your aggregation
• Click the "X" to delete the aggregation from your list (this will also remove the aggregation from our server)
• Click the blue circle arrow to extend the expiration of your aggregation. Note that this option will only be available if the current date is within 7 days of the aggregation expiration. Aggregations, by default, expire 28 days after they are created, unless you request an extension. You may request multiple extensions.

Data Citation
We strongly encourage users to cite their data source(s) in published literature. Two key components of a data citation are the dataset title and the access date. You can use our data citation tool to get a citation for your data that you can use in your publication:

To begin, you need to either know the dataset that you want to cite, or the approximate date that you downloaded data from us.

If you know the dataset, click the "By Dataset" tab.

• You will see a list of all datasets that you have downloaded from the RDA. Click the specific dataset that you want to cite. You will then see a year/month calendar for all of your accesses. For many accesses over time, the suggestion is to choose the most recent access. Click the month to get a calendar for that month. When you click a specific day, a data citation will be generated.

If you know the approximate date, click the "By Date" tab.

• You will see a year/month calendar showing you when and how many times you downloaded data from the RDA. Click the month that you downloaded the data that you want to cite. You will then see a list of datasets that you downloaded in that month. Hopefully you will recognize that dataset that you want to cite from this list. Click the specific dataset to get a calendar for the month, and then once you click a specific day, a data citation will be generated.

Over time as we develop new services, you will see new sections appear in the dashboard. So when you need more time to download a dataset subset that you have requested, or you want to jump quickly to a dataset that you have previously used (and bookmarked), remember the RDA user dashboard.