2016-08-26

Yes, Ireland does have droughts: Notes from the 9th ACRE Workshop

The RDA hosts many climate reanalysis collections, yet only two in our archive are unique in that they extend beyond the modern (post-1948) radiosonde era: the NOAA-CIRES 20th Century Reanalysis (1851–2014; 20CR) and the ERA-20C (1900–2010) produced under ECMWF's ERA-CLIM project.  In order to produce a climate reconstruction dating back over a century, both of these reanalyses rely on assimilating very old surface and marine weather observations, which have been assembled into long-term observational datasets such as the International Surface Pressure Databank (ISPD) and the International Comprehensive Ocean-Atmosphere Dataset (ICOADS).  How do these historic observations find their way into a modern climate reconstruction such as 20CR and ERA-20C?

It all starts with the Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative, which is an international collaboration between meteorological organizations, atmospheric scientists, librarians, and volunteers who are dedicated to the recovery and digitization of historic global weather observations.  By facilitating the process of rescuing, digitizing, and performing quality control of data from old weather log books, ACRE ensures that these important observations provide a better understanding of past climate, which in turn aids in modern climate applications and studies of climate impacts.

The 9th ACRE Workshop


ACRE holds annual workshops to discuss all things related to data rescue, and I attended the 9th ACRE Workshop and Historical Weather and Climate Data Forum, held June 20-24 at Maynooth University in Ireland.  The meeting agenda was populated with talks illustrating the full cycle of how historic data observations make their way from being discovered and digitized all the way to being assimilated into climate reconstructions and applications.

Most notably, the annual ACRE workshop is held in different locations around the world, and the local ACRE partners which host the workshop are featured prominently in the meeting agenda.  This not only brings attention to data rescue efforts in the region which hosts the workshop and encourages local groups to become more involved, but it also highlights the importance of how data rescue facilitated by ACRE leads to a better understanding of the past local climate and subsequent future impacts.  At the most recent meeting in June, climate scientists from Maynooth University and Met √Čireann, the Irish national weather service, gave illuminating talks on how they're using ACRE supported observations and reanalyses to understand long-term Irish precipitation trends, the Irish droughts of the 19th and 20th centuries, and how weather played a role in the Easter Rising of April 1916.


St. Patrick's College at Maynooth University provided a beautiful setting for the 9th ACRE Workshop

Other highlights included presentations on the following topics:
  • Dona Cuppett and Rick Crouthamel from the International Environmental Data Rescue Organization (IEDRO) volunteer their time to travel to developing countries around the world to assist in setting up a data rescue and digitization operation.  Their presentation described the logistical challenges involved in undertaking this effort, and underscored the importance of rescuing historic weather observations in developing countries, which are most vulnerable to climate impacts.
  • Reports on efforts to scan and digitize pages from historic weather logs.  These included talks on citizen science projects such as OldWeather.orgWeatherDetective, and the Sir Charles Todd folios project led by Mac Benoy.
  • Stefan Br√∂nnimann presented on the Mount Tambora eruption of 1815 and the subsequent period of global cooling, commonly known as the "Year Without a Summer".
  • Dispatches from ACRE regional chapters leading data rescue efforts around the world.
  • Updates from various groups leading climate reanalysis initiatives, including the 20CR, UERRA, and ERA-CLIM2 projects.


Wanted: Southern Hemisphere data


The benchmark of a good climate reanalysis is performing better than climatology, yet the uncertainty in a reanalysis is controlled by the amount of underlying observations assimilated into the analysis.  More observations correspond to a higher confidence in the analysis, whereas a data sparse region returns an analysis with low confidence.  For long-term reanalyses such as the 20CR and ERA-20C, the analysis in the 19th and early 20th centuries poses a challenge due to the overall scarcity of input data.  In particular, there is a great need for data in the Southern Hemisphere, not only for earlier periods, but for more recent years as well.  Philip Brohan's "Fog of Ignorance" visualizations have become a well-known highlight at ACRE meetings, and illustrate how ground-truth observations improve the certainty in historical climate reconstructions.

Antarctic weather 1909-10 from Philip Brohan illustrates the "Fog of Ignorance" in the 20th Century Reanalysis due to a lack of observations. Note the grey fog disappears in the vicinity of observations (marked by yellow dots) made available by ACRE facilitated data rescue.

ACRE is therefore making a strong push to recover historical data from south of the Equator, with efforts being led in regional chapters such as Chile, Southeast Asia, Southwest Pacific, and Antarctica, the latter of which is recovering observations from Antarctic whaling expeditions in the late 19th and early 20th centuries.

Learn more about ACRE at http://met-acre.net/.

2016-08-19

NCAR RDA outage August 22-24, 2016

UPDATE Thu Aug 25, 2016:
All systems are back up and operating normally.  Thank-you for your patience.

NCAR CISL supercomputers and central file disk systems will be down for system maintenance from 0:01 MDT on Monday, August 22, 2016 until late Wednesday, August 24  The HPSS systems will begin to shut down at 21:00 MDT on Sunday, August 21.

When we come back, we'll have more storage and be able to provide more data for you on disks.

From the Daily B:
The Yellowstone, Geyser, Caldera, and GLADE systems will be unavailable beginning at 12:01 a.m. Monday, and the HPSS system will be unavailable starting at 9 p.m. Sunday, earlier than previously announced. All of these systems are expected to be back in production by late Wednesday afternoon, August 24.

Facilities upgrades during the downtime are related to expansion of the GLADE file systems and preparation for the new Cheyenne HPC system at the NCAR-Wyoming Supercomputing Center. Power will be unavailable for much of that period.

When the systems are returned to service following the downtime, users will have larger GLADE home space quotas. The current 10-GB quota will be increased to 25 GB. The larger GLADE file systems will serve both the Yellowstone and Cheyenne supercomputers. Users will be informed via the CISL Notifier service when each system is returned to production.

2016-08-01

NCAR RDA Partial Outage Aug 2, 2016

NCAR CISL supercomputers and central file disk system will be down for system maintenance 7:00 to 17:00 MDT on Tuesday Aug 2, 2016.

The RDA web server will remain in operation, but users will not be able to transfer data while the central file disk system is unavailable.  Additionally, data subset processing and translation services will be unavailable during this outage.

2016-07-13

What spatial resolution can I expect when using gridded dataset products?

Occasionally, rdahelp@ucar.edu fields requests for higher grid resolutions than what we can currently offer. Some users wonder why older data is not available at the higher resolutions in newer data.

The requests have often come from new users of RDA resources--beginning researchers or experienced researchers from communities (e.g. engineering) outside of weather and climate.

In general, data resolution is limited by computational power and the underlying observations.

Globally-gridded analyses and forecasts are limited to the computational power at the time they were created.  This table shows typical resolutions in different eras.

Time RangeRes (Deg)Res (km)
1970s-1980s2-3200-500
1990s-20101100
2010s-current0.2-0.525-60

Regional models can offer higher resolution than global models, especially if they have higher data density.  For instance, ds609.0 NCEP North American Mesoscale (NAM) could offer 12 km resolution in 2012, a time when NCEP's global models only provided 0.5 to 1.0 degree (about 50-100 km) resolution.

Models also may not run on rectilinear grids, which span the globe in even degree increments.  Rectilinear grid points are far apart at the equator and close together at the poles.  This makes for vastly differently sized tiles, which can slow down models (or create numerical artifacts).

Some models use Gaussian grids that maintain approximately the same horizontal spacing over the globe by reducing the number of grid points approaching the pole.

ECMWF N80 Reduced Gaussian Grid
Other global models, like GFS, are spectral models that represent the atmospheric state as a superposition of wave functions.  Model output is in the form of spectral coefficients.  The higher the number of basis functions, the higher the resolution of the model.

To learn more about the coupling between the spectral functions and the Gaussian grid points, read this explanation from ECMWF.

The RDA sometimes interpolates spectral and Gaussian models to rectilinear grids to simplify data analysis tasks for our users.  Interested users may access the full spectral coefficients of GFS in ds084.6 and reconstruct the data to their own grid.

While one can interpolate data to higher density (smaller spatial separation) grids, it won't increase the resolution of the underlying data, which is limited by the model resolution of the time.

Reanalyses, retrospective analyses performed with higher resolution models grids, can provide higher resolutions than the operational analyses of the era represented in the early part of the reanalysis span.

If your research involves reconstructing what planners could have been able to foresee, then use analysis data for that time.  For instance, one RDA user constructed a Palmer drought index for Brazil using historical analysis data.  By using only data that was available for that time, decision makers in Brazil can see what information would have been available in past droughts.

If your research involves a long time span and you want the data processed consistently, use reanalysis data.

I alluded earlier to the resolution of the underlying data.  Ground-based sensor networks are limited in spatial coverage and change over time.  While continental US (CONUS) and Europe may enjoy high spatial density coverage of ground stations, many other areas, such as the oceans and sub-Saharan Africa, do not.

Short-duration field campaigns may offer high spatial and/or temporal resolution, but only for a limited time.  We offer a few of those data sets, but their usefulness is limited by their short duration.

For truly global coverage, we turn to satellites.  Satellite data is also not spatially uniform due to differences in satellite orbits and sensors.

Different parameters (T, water vapor, cloud tops, albedo) are measured by different sensors at different wavelengths.  The horizontal resolution of satellite measurements is related to the wavelength and the size of the telescope lens or mirror.

Courtesy of NASA GSFC

Characteristics measured in the visible range (albedo) may have higher resolution than characteristics measured in the longer-wavelength microwave range (T and water vapor).

Remote Sensing Systems (REMSS) offers some excellent materials to get-started in understanding satellite data products.

For instance, the resolution of different wavebands on The Special Sensor Microwave Imager (SSM/I) vary so surface temperature and water vapor, even from the same satellite and instrument, will have different resolutions.


Ocean winds, like those obtained from Windsat, are derived from several microwave bands, whose resolutions also vary by wavelength.


Satellite orbits also determine resolution.  Geostationary weather satellites fly above the equator and in a much higher orbit (35,800 km above MSL) than polar satellites such as NOAA-NN and DMSP-NN (800 km above MSL).  The difference in height also changes the possible resolutions.  Look at the difference in total columnar water vapor resolution between TOVS (geostationary) and SSMI (polar) in Stephens et al.

Notice the blockiness of the contours with geostationary TOVS.
The contours are noticeably smoother with polar SSMI.
Even with the same orbits and same wavelengths, satellite resolutions have improved (and become closer to the theoretical limits) with improvements in optics and vibration isolation on spacecraft.  Satellite data improves over time, but is subject to theoretical limits.

No amount of money or technology can change the theoretical limits.  Running models at higher resolutions than the data supports also has limited utility.

One solution for obtaining higher resolutions is to dynamically downscale the data with a physical model such as The Weather Research and Forecasting Model (WRF).  Physics-based models use high-resolution terrain and landuse information, couple them with lower-resolution gridded data, and then model the state of the atmosphere with equations describing atmospheric dynamics.

Online and in-person tutorials can help you learn how to use this free community-supported model.

2016-07-11

NCAR RDA Partial Outage July 12, 2016

UPDATE Tue Jul 12 18:42:57 MDT 2016
Full RDA data services are back in operation.  Thank-you for your patience.

UPDATE Tue Jul 12 11:45:28 MDT 2016
The Glade (disk) outage is over. Full data file downloads are available again. I'll update again when full data services are restored.

UPDATE Tue Jul 12 09:40:05 MDT 2016
We are currently experiencing a disk failure and no data are accessible from this webserver, from /glade (for NCAR users), or from directories where data have been prepared for you at your request. Also, many of our dataset file lists are unavailable at this time. We are working to resolve the problem and apologize for any inconvenience.

NCAR CISL supercomputers, Yellowstone and Geyser, will be down for system maintenance 7:00 to 17:00 MDT on Tuesday July 12, 2016.

The RDA web server will remain in operation and users may continue to browse for and download whole data files.

However, data subsetting and translation services will be unavailable during this outage.

2016-06-24

NCAR RDA System Outage June 28, 2016

UPDATE Jun 28 15:40:05 MDT 2016:
All systems and services are up and running. Thank-you for your patience.

All RDA services will be unavailable starting at 6:00 AM MDT on June 28 due to system maintenance. It is estimated that RDA services will be back online by early evening.

We apologize for the inconvenience.

2016-06-03

NCAR RDA System Outage June 7, 2016

UPDATE Jun 7 21:36:04 MDT 2016:
All systems and services are up and running. Thank-you for your patience.

SCHEDULED OUTAGE NOTIFICATION

The Yellowstone, Geyser, and Caldera clusters will be unavailable during a scheduled maintenance period from 8 a.m. to 5 p.m. MDT on Tuesday, June 7. CISL staff will use this downtime to upgrade the system's GPFS client software to version 4.2.0.3.

A system reservation will be put in place 12 hours before the scheduled downtime. Users' jobs with specified job times that overlap the reservation period will remain on hold until the system is restored to service. Longer-running jobs that have not finished by 7 a.m. Tuesday will be killed and need to be resubmitted after the maintenance period. We apologize for the short notice and for any inconvenience this might cause.

START: Tue Jun 7 2016 6:00 AM MDT
DURATION: 11 hours

SERVICES AFFECTED
Geyser and Caldera
Yellowstone
Direct access of RDA data from /GLADE will not be available during this time.

The RDA web server will remain up and continue to accept data requests. But, the requests will not be filled until after the clusters are brought back up. Jobs submitted up to 12 hours before the system outage may also be held until after the outage. Data request jobs should run automatically after system restart.

To avoid uncertainty, submit your data requests early, so that they may complete before the system maintenance outage.

We apologize for the inconvenience.

You can check the CISL Resource Status Page for updates.

2016-06-02

Open Access Data

The RDA is an open access data archive. Everyone, regardless of affiliation status--including no affiliation--can obtain an RDA account.

The majority of RDA data holdings are freely available to everyone without restriction other than the standard UCAR/NCAR Terms of Use.

Registering for an RDA account

Affiliation with an academic or research institution is NOT required for registration, but we recommend that you register with one, if you qualify.  Selecting an Organization Type and giving us the Organization Name helps us learn about our users, and may enable you to access restricted datasets.

Your Organization Type choices:
  1. UCAR: Select only if your registration email address ends with @ucar.edu
  2. University/College
  3. Other Education: K-12 students and teachers, other education not covered by the first two categories.  
  4. Commercial
  5. Government: Federal, State, Local
  6. Military
  7. Public/Non-profit: National Labs, FFRDCs, NGOs, .org
  8. Unaffiliated: Those who do not fit into one of the above categories.  We encourage and support citizen-scientists.
Fill in ALL required fields in the registration screen (denoted with a *.)
After you submit the registration form, you will receive a confirmation message to the email you used for your registration.  You must respond within 48 hours or else your registration will be deleted.  If you don't receive an email from us, check your spam folder.

If you forgot your password, select "Forgot Password?" and we will send a change password link to your email account on record.

If you already have an account, but have changed affiliations, log in to your RDA account using your old email address and the password you set up.  Then edit your profile to reflect your new affiliation.

If you no longer have access to that email account and you forgot your password, email rdahelp@ucar.edu with your name, your former email account and your new one.  We'll help you update your information and you will be accessing data again in no time.

Please do not create more than one account per person.  Do not share your account with others.  Doing so may cause you to be banned from our site.

In summary

  • Register.
  • Keep your registration up-to-date and accurate.
  • We'll support you, no matter your affiliation status.

2016-04-25

NCAR RDA System Outage April 27, 2016

UPDATE Apr 27 22:04:20 MDT 2016: All systems and services are up and running. Thank-you for your patience.

UPDATE Apr 27 17:00:00 MDT 2016: RDA systems are still down due to extended maintenance. It is
estimated that they should be back online by 5PM MDT.
We have no estimate of when they will be back on line. We appreciate your patience during this extended outage.

RDA services, including data download and data request processing, will be unavailable during system maintenance on Wednesday, April 27, 2016.

Work will start at 4:30 MDT and complete around 9:00 MDT. Data request jobs should run automatically after system restart.

We apologize for the inconvenience.