17 June 2021

403 Forbidden for Slow Connections

 If you find yourself receiving 403 forbidden HTTP responses when downloading data from the RDA, it may be the result of a slow connection. 

The RDA currently limits connections that are below 200 KB/s averaged over a two minutes.  If a user stays below this threshold, they will be temporarily blocked from accessing our servers for 30 minutes.

Once blocked, downloaded files may appear to be smaller than expected and contain HTML. This occurs, because the server is sending the RDA's 401 forbidden page instead of the file.


At this point, there are two options to overcome this problem. The first would be to improve your network speed. Often, using a VPN or proxy server can overcome this issue


Alternatively, you can consider using our Globus download option. On each data access page, you can find an option to download the files using Globus as seen in the screenshot below. Globus is a service that transfers files using gridFTP and has built in error tolerance and unsupervised downloads. 




10 June 2021

Concerning ERA5 Single Level Forecast Data, December 1978

The ERA5 single level forecast fields, 1978-12 as obtained from the ECMWF Climate Data Store (CDS) are incomplete -— they do not form a continuous time series with 1979 onward because 6 forecast hours have been omitted. This is particularly true for fluxes defined by mean rates which includes for example precipitation products. The four affected groups in the RDA dataset ds633.4, ERA5 back extension 1950-1978 (Preliminary version) are:

  • ERA5 atmospheric surface forecast (accumulated), 1978-12
  • ERA5 atmospheric surface forecast (instantaneous), 1978-12
  • ERA5 atmospheric surface forecast (mean rates or fluxes), 1978-12
  • ERA5 atmospheric surface forecast (minimum-maximum), 1978-12

NCAR's CISL/DECS has brought this to the attention of ECMWF, and are waiting for ECMWF to fill in the gap on the CDS. Apologies for the inconvenience, and thank you for your patience.

A more detailed discussion follows:

The ERA5 forecast data 'time' conventions are somewhat awkward (but entirely the prerogative of ECMWF and especially the Copernicus Climate Data Store -- CDS). For any given day, CDS provides 2 forecast initial times, 06 and 18Z. For each forecast there are 12 forecast hours, 1,2,3,4,5,6,7,8,9,10,11, and 12 hours. In the netCDF files as produced by CISL/DECS for the RDA, this is represented with two coordinate dimensions,

forecast_initial_time

forecast_hour

The valid time of the forecast data is then

forecast_initial_time + forecast_hour

A discrete 24 hour time series of, for example, "mean rates or fluxes" forecast data (including "Mean total precipitation rate", kg m**-2 s**-1) in this case is given by


06Z forecast initial time current day + 1hr(a mean flux/rate between 6 and 7 hr of current day)
06Z forecast initial time current day + 2hr(a mean flux/rate between 7 and 8 hr of current day)
06Z forecast initial time current day + 3hr(a mean flux/rate between 8 and 9 hr of current day)
06Z forecast initial time current day + 4hr(a mean flux/rate between 9 and 10 hr of current day)
06Z forecast initial time current day + 5hr(a mean flux/rate between 10 and 11 hr of current day)
06Z forecast initial time current day + 6hr(a mean flux/rate between 11 and 12 hr of current day)
06Z forecast initial time current day + 7hr(a mean flux/rate between 12 and 13 hr of current day)
06Z forecast initial time current day + 8hr(a mean flux/rate between 13 and 14 hr of current day)
06Z forecast initial time current day + 9hr(a mean flux/rate between 14 and 15 hr of current day)
06Z forecast initial time current day + 10hr(a mean flux/rate between 15 and 16 hr of current day)
06Z forecast initial time current day + 11hr(a mean flux/rate between 16 and 17 hr of current day)
06Z forecast initial time current day + 12hr(a mean flux/rate between 17 and 18 hr of current day)
18Z forecast initial time current day + 1hr(a mean flux/rate between 18 and 19 hr of current day)
18Z forecast initial time current day + 2hr(a mean flux/rate between 19 and 20 hr of current day)
18Z forecast initial time current day + 3hr(a mean flux/rate between 20 and 21 hr of current day)
18Z forecast initial time current day + 4hr(a mean flux/rate between 21 and 22 hr of current day)
18Z forecast initial time current day + 5hr(a mean flux/rate between 22 and 23 hr of current day)
18Z forecast initial time current day + 6hr(a mean flux/rate between 23 and 24 hr of current day)
18Z forecast initial time current day + 7hr(a mean flux/rate between 0 and 1 hr of *next day*)
18Z forecast initial time current day + 8hr(a mean flux/rate between 1 and 2 hr of *next day*)
18Z forecast initial time current day + 9hr(a mean flux/rate between 2 and 3 hr of *next day*)
18Z forecast initial time current day + 10hr(a mean flux/rate between 3 and 4 hr of *next day*)
18Z forecast initial time current day + 11hr(a mean flux/rate between 4 and 5 hr of *next day*)
18Z forecast initial time current day + 12hr(a mean flux/rate between 5 and 6 hr of *next day*)

The crux of the matter is, for 31 December 1978, 1978-12-31, the CDS has omitted the last 6 entries,


18Z forecast initial time 1978-12-31 + 7hr(a mean flux/rate between 0 and 1 hr of 1979-01-01)
18Z forecast initial time 1978-12-31 + 8hr(a mean flux/rate between 1 and 2 hr of 1979-01-01)
18Z forecast initial time 1978-12-31 + 9hr(a mean flux/rate between 2 and 3 hr of 1979-01-01)
18Z forecast initial time 1978-12-31 + 10hr(a mean flux/rate between 3 and 4 hr of 1979-01-01)
18Z forecast initial time 1978-12-31 + 11hr(a mean flux/rate between 4 and 5 hr of 1979-01-01)
18Z forecast initial time 1978-12-31 + 12hr(a mean flux/rate between 5 and 6 hr of 1979-01-01)

which prevents CISL/DECS from forming an uninterrupted time series with ERA5 data for 1 January 1979, 1979-01-01, because the first entry for 1979-01-01, *as provided by CDS*, is


06Z forecast initial 1979-01-01 + 1hr(a mean flux/rate between 6 and 7 hr of 1979-01-01)