27 February 2017

Securing the RDA Website

On Wednesday, March 1, the RDA website will change to serving all content and services over secure HTTP. The are multiple benefits to doing this and if you want to learn more, do a web search for "HTTP Everywhere" or "Securing the Web", for example.

For most users, this change will be barely noticeable. When you request an insecure URL (e.g. - http://rda.ucar.edu), your browser will receive a redirect message, and then the browser will initiate a new connection over HTTPS. You will see a padlock in the URL bar and the URL will begin with https, but otherwise all content will be the same and all services will perform as they currently do.

The RDA web server will also enable HTTP Strict Transport Security (HSTS) by sending an HSTS header message that tells your browser to only connect over secure HTTP in the future. If you have an HSTS-aware browser, then even if you type http://rda.ucar.edu into the URL bar, the browser will convert it to https://rda.ucar.edu for you before initiating the connection.

The main impact will be to users who are downloading data files via scripts that use curl. We have found that curl does not automatically follow a redirect - that will only happen if the "-L" flag is included with the curl command. Without the "-L" flag, users requesting a data file over http will receive a file that doesn't actually contain data - it will contain HTML that explains that the file has moved permanently. For curl scripts generated by the RDA server, we will begin including the "-L" flag as well as writing URLs beginning with https. If you have an older script generated by our server, or you have written your own script using curl, you will need to make the change yourself. We have found that wget (at least newer versions of it) will automatically follow the redirect.

If you run into any trouble after the change-over, please let us know right away at rdahelp@ucar.edu.

UPDATE (Mar 15): We discovered another issue - POST requests, even if the client follows the redirect, will not work because they get resubmitted as GET requests. This issue will affect both curl and wget. You will need to change URLs to https to get around this problem.

22 February 2017

RDA Data Rejuvenation Spa

The theme for day 5 of International Love Your Data Week 2017 is Data Rescue.  At the RDA, we don't just do Data Rescue, we offer total Data Rejuvenation at our Data Spa™.

At the RDA, saving data from death by bit rot is just one of our many services.

Boxes of tapes from various universities and the National Archives in Denver awaiting the RDA Data Rejuvenation treatment.
You want a scenic locale?  Can Canyon Ranch or Ojai Valley Inn top this?

This is not photoshopped.  It actually looks like this when a late Spring storm snows on the Continental Divide while it rains at the lower elevations.

The plush treatment for these lucky tapes began during their journey to the NCAR, but most boxes of tapes arrive without any cushioning.

These tapes were lovingly packed with cushy materials.

First, the tapes are bathed to bring out their natural beauty.

Tape cleaner, the first data rejuvenation spa treatment.
Then they are read.

Is the old data packed in 6-bit words instead of the now-standard 8-bit word?  Does it have an extra parity bit for error checking?  No problem.  They will be matched by our data whisperer, Bob Dattore, to the right tape reader that can understand their language.
We have two 9-track tape drives and one 7-track one.  6-bit words?  Parity bit?  No problem!
Atmospheric and oceanic data may be sanity checked by RDA data specialists cross-trained in meteorology or oceanography.  If needed they can be translated to modern data formats and outfitted with modern metadata, readable by both humans and machines.

Then the data is put on transfer disks, ready for their owner to pull over the internet using NCAR's blazing fast network.