2015 Pan Am and Parapan Am Legacy Observational Data

59 Pages·2016·2.09 MB·English

Cadre de
Gestion des
Données
Data
Management
Framework

2015 Pan Am and Parapan Am Legacy
Observational Data

- Product User Guide-

June 7, 2016
Version 2.0
DMF Team Pan Am Dataset User Guide
June 7, 2016 Pan Am Dataset User Guide Page 2 of 59
DMF Team Pan Am Dataset User Guide
Table of Contents

1. DOCUMENT INFORMATION ................................................................................................................................. 4
1.1 HISTORY .................................................................................................................................................................... 4
2. INTRODUCTION ........................................................................................................................................................ 5
2.1 OVERVIEW ................................................................................................................................................................. 5
2.2 PURPOSE AND SCOPE ................................................................................................................................................. 6
2.3 INTENDED AUDIENCE ................................................................................................................................................. 6
3. DATA STANDARDIZATION .................................................................................................................................... 7
3.1 UNITS ......................................................................................................................................................................... 7
3.2 CODES ........................................................................................................................................................................ 7
4. FILE FORMAT AND STRUCTURE ........................................................................................................................ 9
4.1 STRUCTURAL ORGANIZATION OF THE FILE ................................................................................................................ 9
4.2 DATE-TIME .............................................................................................................................................................. 10
4.3 QUALITY ASSESSMENT FLAG ................................................................................................................................... 11
5. DATASETS ................................................................................................................................................................ 13
5.1 OVERVIEW ............................................................................................................................................................... 13
5.2 DATA ELEMENT TERMINOLOGY ............................................................................................................................... 13
5.2.1 Wind ............................................................................................................................................................... 13
5.2.2 Precipitation and Rainfall ............................................................................................................................. 14
5.2.3 Station elevation ............................................................................................................................................ 15
5.3 PAN AM – MINUTELY MSC, S&T & PARTNER SURFACE WEATHER NETWORK ........................................................ 15
5.4 PAN AM – MINUTELY MSC ULTRAVIOLET RADIATION NETWORK - UV ................................................................... 21
5.5 PANAM – MSC MOORED BUOY (WATCHKEEPER™ MESSAGES) ............................................................................. 22
5.6 PAN AM – MINUTELY NAV CANADA AWOS ...................................................................................................... 24
6. APPENDICIES........................................................................................................................................................... 27
6.1 GLOSSARY ............................................................................................................................................................... 27
6.2 UNITS OF MEASURE ................................................................................................................................................. 29
6.3 UNIT CONVERSIONS ................................................................................................................................................. 32
6.4 STANDARD CODE TABLES ........................................................................................................................................ 34
6.4.1 buoy_type ....................................................................................................................................................... 34
6.4.2 present_weather ............................................................................................................................................. 35
6.4.3 total_cloud_amount ....................................................................................................................................... 53
6.4.4 station_type .................................................................................................................................................... 54
6.4.5 report_type ..................................................................................................................................................... 55
6.4.6 rapid_pressure_change ................................................................................................................................. 55
6.4.7 tendency_characteristic ................................................................................................................................. 56
6.5 PAN AM MESONET STATION METADATA ................................................................................................................. 57
6.6 PANAM MESONET MAP ........................................................................................................................................... 58
June 7, 2016 Pan Am Dataset User Guide Page 3 of 59
DMF Team Pan Am Dataset User Guide
1. Document Information
1.1 History

Author Date Ver. Remarks
Tahreem Ali February 10, 2016 1.0d Initial draft
Tahreem Ali, Dale June 7, 2016 2.0 Final version
Boudreau, Joan
Klaassen

June 7, 2016 Pan Am Dataset User Guide Page 4 of 59
DMF Team Pan Am Dataset User Guide
2. Introduction
2.1 Overview
The creation of a legacy product of surface weather observations from the 2015 Pan Am and Parapan
Am Games’ (hereafter referred to as Pan Am) Mesonet has been requested as part of the Pan Am
Games Project deliverables. In response, the Data Management Framework (DMF) team at the
Meteorological Service of Canada (MSC) is providing external clients with a concise, easy to decode
product containing Pan Am surface and buoy weather data (and other observation, forecast and
warning data products not included in this specific user’s guide). This product is generated by
applications within the Data Management System (DMS). The DMS is a collection of real-time data
acquisition, decoding, standardization, quality assessment (Qa) and product generation components for
observation, forecast, and warning data.
Within the DMS observational data has been stored elementally in the DMS Data Warehouse (DW),
which follows a flat table format. The DW stored all Pan Am Mesonet data that was collected over
the period of May 12 – August 31, 2015. These data were then extracted and output to comma-
delimited text files (CSV). Given the size of the files, this component of the Pan Am legacy data
product consists of a total of 20 files, with each file containing all station observations for a period of
one week. Each of the 20 files starts on a Sunday and ends before the following Sunday.
All data in these datasets are essentially ‘raw’, having undergone only minimal automated Quality
Control (QC) (manual QC of the ATMOS Mesonet data will be posted to the Science Showcase
section of the Legacy Archive, when available). The user is referred to an additional Instrumentation
Metadata XLS file posted with the Datasets that describes specific data issues that the users must be
aware of before using the datasets.
During the production of these files, the following tasks were carried out:
 Incoming DMS element packages were assigned an abbreviated label
 Unit conversion from incoming units to standard units may have been performed
 If the incoming element was a numeric code or a text value from a list of controlled vocabulary
(so in effect a code), then code substitution was performed to map to a DMS standard code
 A quality assessment (Qa) summary column (using incoming ‘native’ and DMS real-time
quality assessments whenever available) was attached to each of the data elements to form a
column pair
The CSV files are intended for clients to obtain Pan Am data in an easy to read, user friendly format.
Although this format easily loads into a Microsoft Excel spreadsheet, for clients needing to interact
with data over long intervals or to compare observations from multiple stations geospatially, the use of
tools such as programmatic parsers, graphical displays and GIS is advisable.
There are several different streams of the observational component of the Pan Am legacy files, each
containing data from the following networks (a short reference name is to the right of the arrow):
Environment Canada’s (EC)1 new Mesonet surface weather stations reported each minute (i.e.
‘minutely’) and marine buoy reported every 10 minutes. This higher temporal resolution is in contrast

1 As of November 2015, Environment Canada has become Environment and Climate Change Canada (ECCC).
June 7, 2016 Pan Am Dataset User Guide Page 5 of 59
DMF Team Pan Am Dataset User Guide
to the normal hourly reporting from the existing surface and marine weather networks. The new
Mesonet stations included:
 MSC Reference Climate surface weather stations  RCS-minutely (3 stations; CA-RCS network)
 MSC Compact weather stations  COMPACT-minutely (40 stations; CA-COMPACT network)
 EC’s Science and Technology Branch (S&T) ATMOS stations  ATMOS-minutely (10 stations;
CA-ATMOS network)
 MSC Ultraviolet stations  UV-minutely (4 stations; UV network)
 MSC WatchKeeper™ moored buoy  WATCHKEEPER (1 buoy; WATCHKEEPER network)
 NAV CANADA (NC) AWOS minutely weather stations  NC-AWOS minutely (6 stations; NC
AWOS network)
To associate the station identifier with a particular network or station type, please refer to the station
list in Appendix 6.5.

2.2 Purpose and Scope
The main purpose of the CSV files is to provide easy access to the Legacy Pan Am dataset for internal
and external clients who prefer to not interact with archived XML files stored within the DMS. If
clients want to view the full decoded XML, each observation will be accessible via its own URI within
the DMS, or possibly a file system containing the XML files referenced using a filename (e.g. the
Canadian Meteorological Centre’s (CMC) DataDepot available through dd.weather.gc.ca).
The observational elements included in the majority of the existing networks are from these basic
observation groups:
 pressure
 wind
 temperature
 humidity
 precipitation
 radiation
In addition to the elemental data reported from the Mesonet, the CSV file may also contain optional
quality assessment (Qa) information in the form of a summary quality flag beside each element,
whenever available. This flag's value is computed by considering any incoming Qa performed on the
element at the source (so-called “NativeQC”), any real-time automatic Qa conducted in-house by the
DMS, or a combination of the two. The absence of a Qa summary flag on an element in the file
indicates the quality may not have been assessed, so is unknown.

2.3 Intended Audience
Any clients interested in performing weather or climate analyses during the period surrounding the
2015 Pan Am and Parapan Am Games in Southern Ontario will find the CSV file product easy to
ingest and interact with.
June 7, 2016 Pan Am Dataset User Guide Page 6 of 59
DMF Team Pan Am Dataset User Guide
3. Data Standardization
3.1 Units
All DMS elements have a standard element assigned by the DMS based on the data class (e.g. wind
velocity, temperature, pressure, etc.), although some variations exist for certain elements within a
class. For example, most pressure-related elements will have a DMS standard unit of hPa, but in the
case of altimeter, the standard unit is ‘in Hg’ (i.e. inches of mercury) since it is used by a specific
client community and in practically all cases measured and used in that unit. Examples of typical
standard units for some classes are:
 precipitation amount = mm
 wind speed = km/h
 pressure = hPa
 visibility = km
 height = m
 temperature = °C
The conversion to DMS standard units is only done at the last possible moment, typically when data
leaves the DMS via product generators or during the population of custom datamarts and display tools
where client requirements need to be satisfied. The DMS standard units will meet the majority of
client needs, but inevitably some clients will have different preferences and will need to do some
conversions. To assist in this, Appendix 6.3 has a list of unit conversions so clients can see what was
used to convert incoming units to DMS standard units for a given element, or to apply client-side
conversions. The dataset tables in Section 5 show the incoming uom (unit of measure) and the
standard units they were converted to.
3.2 Codes
A “standard” code table is also associated with any incoming code tables for a given element. The
master list of code tables that the DMS maintains has cross referenced similar code tables for a given
entity to a DMS standard table, which is in effect a superset of all the similar code tables for that
entity. This allows for products or clients to use one standard code value for an element to map to
their preferred codes, expressions or interpretation rather than having to map from many different
tables for a given element across multiple networks. For example, present weather is reported by
many aviation networks, but most use different code tables or even text strings (note, in the DMS text
strings that are controlled vocabulary are also treated as if they were codes).

June 7, 2016 Pan Am Dataset User Guide Page 7 of 59
DMF Team Pan Am Dataset User Guide
Example 1. Present weather arriving to the DMS is mapped to a code value in a DMS standard code
table. Below is an example:
Network Observation Incoming Incoming Incoming DMS code- DMS code-type DMS code
code-src code-type value src value
Minutely Snow detected ca precipitation_type 4 std_code_src present_weather 374
MSC and
S&T surface
weather
networks
(CA
Minutely)

To see the meaning of the standard code table values for coded elements, please refer to Appendix 6.4.
With this information clients can map the standard code values to their preferred expression or code
using a single mapping table, rather than one for each network.
June 7, 2016 Pan Am Dataset User Guide Page 8 of 59
DMF Team Pan Am Dataset User Guide
4. File Format and Structure
4.1 Structural Organization of the File

The format of the product is a CSV file which can be viewed in either Microsoft Excel or any form of
text editor, e.g. Notepad. Below is a sample of the raw file in a text editor:

Data Description:
The first row of the file contains the headers that sequentially identify the data elements in the
rows below. When imported into a spreadsheet these would be the column headers. The first
few fields are metadata elements about the observation. For instance, one could find the time
of observation, the reporting station identifier (e.g. MSC ID, TC ID, etc.), the station's latitude,
longitude and elevation, etc. Note that some datasets may not have data for all these metadata
elements. Below is an example of the metadata in columnar form:

June 7, 2016 Pan Am Dataset User Guide Page 9 of 59
DMF Team Pan Am Dataset User Guide
NULL & MSNG:

A null in the spreadsheet indicates that an element was never received or derived. A value of
missing (MSNG) means that the element was received or a derived element was created, but
the value was not available. For observed elements this could be because the value could not
be measured or was illegal (e.g. over-ranged). For derived elements it means that the
derivation could not proceed due to the unavailability of usable input elements (e.g. too many
missing observations or the ones that were available failed quality assessment).

Observational Data Elements:

From row 2 onwards in the file are the actual data associated with each header field in row 1.
The first set of elements in these rows contain data associated with station information, while
the remainder to the right of this set contain actual observational data. The observational data
are to be treated as pairs; the element’s value followed by its Qa flag (if available) E.g. air
temperature followed by the air temperature Qa flag, wind speed followed by the wind speed
Qa flag, etc. If Qa information is available for the element, then the corresponding Qa column
is populated, otherwise it is left empty. Below is an example where the first two elements do
not have a Qa flag, whereas the third element does:

See Section 4.3 for details on the creation of the Qa flag and the meaning of the code value.

4.2 Date-Time
The date-time of the observation is in UTC (otherwise known as Greenwich Mean Time, or GMT) and
is present in the column “OBS_DATETIME_UTC”. The format of the date and time might differ
depending on the program being used to view it.

When viewing the file in a text editor such as Notepad, the format is as follows:
2015/08/01 18:01
Year/ month/ day time (hour:minute, 24 hour clock)
However, the same data in Excel will be formatted to:
01/08/2015 18:01
Day/ month/ year time (hour:minute, 24 hour clock)

Clicking on the cell in Excel will result in the format:
01/08/2015 6:01:00 PM
June 7, 2016 Pan Am Dataset User Guide Page 10 of 59