NASA Technical Reports Server (NTRS) 20100036464: An Objective Verification of the North American Mesoscale Model for Kennedy Space Center and Cape Canaveral Air Force Station

2.8 MB·English

by  NASA Technical Reports Server (NTRS)

4.2 AN OBJECTIVE VERIFICATION OF THE NORTH AMERICAN MESOSCALE MODEL FOR
KENNEDY SPACE CENTER AND CAPE CANAVERAL AIR FORCE STATION
William H. Bauman III *
NASA Applied Meteorology Unit / ENSCO, Inc. / Cape Canaveral Air Force Station, Florida
1. INTRODUCTION
Table 1. Towers, launch adivities and sensor heights at
The 45th Weather Squadron (45 WS) Launch KSC and CCAFS used in the objective analysis to verify
Weather Officers (LWO's) use the 12-km the MesoNAM forecasts.
resolution North American Mesoscale (NAM)
model (MesoNAM) text and graphical product
forecasts extensively to support launch weather
operations. However, the actual performance of
the model at Kennedy Space Center (KSC) and
Cape Canaveral Air Force Station (CCAFS) has
not been measured objectively. In order to have
tangible evidence of model performance, the 45
WS tasked the Applied Meteorology Unit (AMU;
Bauman et ai, 2004) to conduct a detailed
statistical analysis of model output compared to
observed values. The model products are
provided to the 45 WS by ACTA, Inc. and include
hourly forecasts from 0 to 84 hours based on
model initialization times of 00, 06, 12 and
18 UTC. The objective analysis compared the
MesoNAM forecast winds, temperature (T) and
dew pOint (Td ), as well as the changes in these
Therefore, the period of record (POR) for this data
parameters over time, to the observed values from
set starts with the first cool season month of
the sensors in the KSC/CCAFS wind tower
October 2006. The KSC/CCAFS wind tower data
network shown in Table 1. These objective
were acquired for the period October 2006 to April
statistics give the forecasters knowledge of the
2009 from the AMU archive, and the AMU wind
model's strengths and weaknesses, which will
tower quality control (QC) software was used to
result in improved forecasts for operations.
remove erroneous observations from the dataset.
2. BACKGROUND The statistical analysis software S-PLUS®
(Insightful Corporation 2007) was used to process
The 45 WS requested the data sets be
the wind tower data. Scripts were written in S
stratified by year, warm season (May-September),
PLUS to import and modify the QC'd wind tower
cool season (October-April), month and model
observation files to remove unneeded time periods
initialization time. They also requested verification
and sensor heights from the dataset for each
the model forecasts for the current operational
tower. The locations of the towers used for the
version of the MesoNAM. This paper will address
verification are shown on the map of KSC/CCAFS
the following statistics requested by the 45 WS:
in Figure 1.
• Bias (mean difference),
Since the tower data were reported every
• Standard deviation of Bias, and 5 minutes and the MesoNAM forecasts were
hourly, the 45 WS requested the AMU calculate
• Root Mean Square Error (RMSE).
the mean value for each observed parameter from
3. WIND TOWER DATA the tower data at the top of every hour using the
observations from 30 minutes prior and 30
The current version of the operational
minutes after the hour. The S-PLUS scripts were
MesoNAM became available in August 2006.
written to reformat the tower data and calculate
the mean values in this manner.
·Corresponding author address: William H. Bauman III,
ENSCO, Inc., 1980 North Atlantic Ave, Suite 830, Cocoa
Beach, FL 32931; e-mail: [email protected]
1
•
•
•
nnn~~I_.
• LaunchlLancllng Tower.
• OtMrT. .....
• ....a NAII Grid Po. ...
Figure 1. Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM forecasts
(red pentagons labeled with tower number and the supported launch activity), the locations of the MesoNAM
model grid points (green circles with black dot) and CCAFS weather station (magenta square labeled KXMR).
4. MESONAM FORECAST PRODUCTS An inventory of the MesoNAM files in the POR
revealed 128 missing files, or model runs, out of a
The AMU requested and obtained the
possible 3772 files for the 943 days. Some days
archived MesoNAM forecasts from ACTA, Inc. The
were missing less than four model runs while
current operational version of the MesoNAM is the
others were missing all four model runs. This
12-km Weather Research and Forecasting (WRF)
resulted in a total of 910 days containing at least
model. Based on the seasonal stratifications
one model run.
requested by the 45 WS and model availability,
the MesoNAM forecasts were evaluated beginning 5. FILE FORMATTING
with the October 2006 data, the first cool season
Microsoft Visual Basic scripts were written to
month in the data set. The POR included three
import the MesoNAM files into Microsoft Excel
cool seasons: 2006-2007, 2007-2008 and 2008-
spreadsheets and reformatted to match the wind
2009; and two warm seasons: 2007 and 2008.
tower observation spreadsheets. This included
The MesoNAM forecast files were provided to converting the temperature and dew point from
the AMU as space-delimited text files. Each file Celsius to Fahrenheit and moving rows and
was based on a single model initialization time and columns in the MesoNAM spreadsheets to match
was valid at a single point extracted from the the wind tower spreadsheets. Visual Basic scripts
model forecasts. This point was identified as were written to create an Excel workbook for each
"KXMR" in each file and represents the location of of the 910 days with at least one model run. Each
the CCAFS weather station, which is located near workbook included up to four worksheets, one for
the center of CCAFS and is identified in Figure 1 each available model run, containing combined
by the magenta square. The closest model grid wind tower observations and MesoNAM data for
point, which represents the point data used by the each sensor on every tower. This resulted in a
45 WS, is located 5.8 km southwest of KXMR over total of 24,570 workbooks.
the Banana River. It is shown in Figure 1 as a
green circle with a black dot and labele:d "NAM".
2
6. VERIFICATION STATISTICS 6.1. Temperature and Dew Point Example
Verification statistics were calculated once the Figure 2 shows a graph of the model bias of T
files were properly formatted and stratified. First and Td from a 1200 UTe model initialization at
the model bias was calculated for each model Tower 0020 at a sensor height of 6 ft for the month
forecast against every observation. The means of January in the POR. Preliminary results indicate
and standard deviations of the model bias for all a periodic fluctuation was present in the model
stratifications (e.g., one month) as well as the root bias of T as can be seen in Figure 2. This result is
mean square error (RMSE) were calculated using consistent among the first three towers and all
the following equations: sensor levels analyzed as of the writing of this
paper. The periodic fluctuation was observed in all
In four model runs per day. In January, the model
BiaSMean = ~ C/;. - °i) had a positive T bias and a negative T d bias. The
i=l biases also decreased with forecast hour, with T
Where: coming closer to 0, but T d becoming more
= negative.
n number of available model forecasts in
any given stratification,
Model T Mel Ttl BI.
12 UTe Inltlellzlltlon, Tower 0020,6 ft,.JMu.-y
f= MesoNAM forecast of T, Td, wind speed 3.0,-------------
or wind direction, and
=
o observed T, T d, wind speed or wind _ 1.0 4--~~-_\_-~_\_-~f_+_-
!I-
direction from each tower/sensor height. .. 0.0 -t-----.,--....,--,--.--.--,.--:;.--.--.\--,.-..p-,-+~
1~i -1.0 +;-,,;.6~ ~. .1.:2:. ::.1..8.. c::.2:.A.. ....3=.0:. ...3.:6.. :c~-4-=2-. ..:4..:8:. ...5..4.- '---'-':...:.::..:.--'-=-~ - T("F)
~ \.J\ 1\, ---Td("F)
~ -~o ~-~~V~~'--~-'\-~-\------
JICX
_X)2 ~ -3.0 4----------1~\ ~\_"/\~ -r\ -,
STDEVBias = n "'.0
4----------.#----..3ri~..;_
-s,o
Where: ..1....-._ __________
Foreceat v.lld nme from Modellnltlellzlltlon
=
n number of available model forecasts in
Figure 2. Graph of T and Td model bias from a
any given stratification, 1200 UTe model initialization at Tower 0020 and
= sensor height of 6 ft for January. The blue line is T
x model bias of each forecast,
and the red dashed line is T d.
=
i mean bias of each forecast period in any Figure 3 shows the model bias of T and T d from a
given stratification. 1200 UTe model initialization at Tower 0020 at a
sensor height of 6 ft for the month of May in the
POR. The T bias was more negative than January
RMSE = -vMSE and displayed a similar periodic fluctuation. The
1nI model Td bias was also negative in May.
= -
MSE n (VI't. - 0t' ) 2 Model Tn T.. BI.
i=l 12 UTt Inltlellzlltlon, Tower 0020,6 ft, M-V
1.0 ,..-------------
Where:
=
n number of available model forecasts in
tL
any given stratification, t...
~ -1.0 -lP------+--=----a--h-----:+-~_4___I--
f= MesoNAM forecast of T, Td, wind speed ~ - T("F)
or wind direction, and El -~o ~~-~~~~~~~~-+ ---Td("F)
~
=
o observed T, T d, wind speed or wind -3.0 -t-----------------'''"--
direction from each tower/sensor height.
-4.0.L.-------------
For.utt v.lld nme from ModellnltleUzlltlon
Figure 3. As in Figure 2 but for May.
3
The model standard deviation and RMSE of T
Model Wind Speed Blu
and Td are shown in Figure 4 and 5, respectively, 12 UTe Initlellzlltlon, Tower 0020,54 ft,.-.u.y
for a 1200 UTC model initialization at Tower 0020 6.0,------------------
at a sensor height of 6 ft for the month of January.
Both graphs indicate the model performance
~4.0 tt---~'----+--I----\--JL----­
degraded with time through the 84-hr forecast.
g
~O ~----------------
This preliminary result of model degradation with
1
1
time was also consistent among the three towers
~O +-----------------
evaluated against the model thus far.
1.0 +-----------------
Model StMdard Deviation ofT lind Tel BI.
12 UTC Inltlellzlltlon, Tower 0020,6 ft,"'"'*'1 o 6 U a 2A 30 36 G 48 54 60 66 72 n 84
8.0 ,-------------- Forec.t valid Time from Modellnltlellzldlon
7.0 +-------------J.~
E
Figure 6. Graph of model wind speed bias from a
6.0 +-----------..~--=­
f 5.0 +----..,.-------:-~,......-,pA~~~-­ 1200 UTe model initialization at Tower 0020 and
1;:,i 4.0 ~~~'t-r~~_I___'V_~\-I---- sensor height of 54 ft for January.
i
3.0 -f1i--+-l~---------------­ - T("I')
E ---Td('F) Model Suncwd DevIation of Wind Speed Blu
~ ~o +------------- 12 UTe Inltlellzldlon, Tower 0020,54 ft, ~
1.0 +--------------
5.0 ...
0.0 +o- .,-6- ,--1,2- --1-8- -r2-A- ---3.0- --3.6- --4.-2- r-4r8- -154,. .-6-0- -,.6-6- .-7-2-- -7.-8- --8r4 44..50 1'\ r\ 1"\ 7 Y'J\j'l"'""
i L \ fr'./ ""'" V
Forec.t valid Time from Modellnltlellzldlon 3.5 7'
"""""
~ 3.0 II
Figure 4. As in Figure 2 but for standard deviation of - ~5
"a
bias. c: t M
1.5
1.0
Model Tn Tel RMSE 0.5
12 UTC Inltlellzatlon, Tower 0020, 6 ft, Jenu.-y 0.0
9.0,-------------- o 6 U a 2A 30 36 G 48 54 60 66 72 n 84
&0 +-------------7'~' Forecelt valid Time from Modellnltlllllzidlon
ii:' 7.0 +----------::-----:..-->N'---
~I 6.0 +--------=--#r-~~:..w--..I---=­ Figure 7. Same as in Figure 6 but for standard
~o ~r+~-+~~~~~~~-­ deviation.
~ 4.0 +-1~l-".;hJi&f_--'\,.,I\--J---JIf__---.::I!~--- - T("I')
~ ~o ~-~~-~------­ ---Td("I') Model Wind Speed RMSE
~ ~o +-------------- 12 UTe Initlallzlltlon, Tower 0020, 54 ft, Jenurt
1.0 +------------- 8.0,------------------
0.0 +-.,--,--,------r-----.---.---.--r-.,--,,..----,.--.-----.----r
o 6 12 11 2A 30 36 42 48 54 60 66 72 78 84 7.0 +-----:----------:::----~~--
_I6 .0 ~~~r_r_~~~_r~~~_1~---
Forec.t "-lId Time from Modellnltlellzldlon 1/1
~O tt-------~---------
Figure 5. As in Figure 2 but for RMSE. - 4.01t-----------------
1
" 3.0 t_----------------
6.2. Wind Speed and Direction Example ct ~O t_----------------
t-----------------
Figures 6, 7 and 8 show the graphs of wind 1.0
speed bias, standard deviation of bias and RMSE, 0.0 +---,---,---.-----.------r---,----r---,,--,---,----,---.--.----,-
o 6 U a 2A 30 36 G 48 54 60 66 72 n 84
respectively from a 1200 UTC model initialization
Forecelt "-lId Time from Modellnltlellzldlon
at Tower 0020 at a sensor height of 54 ft for the
month of January in the POR. These results Figure 8. Same as in Figure 6 but for RMSE.
indicate the MesoNAM forecasted wind speed 4-5
Figures 9, 10 and 11 show the wind direction
kt too high throughout the entire 84 hour model
graphs of bias, standard deviation of bias and
forecast period for January. Preliminary results
RMSE, respectively from a 1200 UTe model
indicate the MesoNAM also forecasted wind speed
initialization at Tower 0020 at a sensor height of
too high at the other three towers evaluated thus
54 ft for the month of January in the POR. The
far. Unlike T and Td, the wind speed forecast error
MesoNAM bias of wind direction was more
did not increase Significantly throughout the 84-hr
negative, or to the right, of the observed winds.
forecast period but remained fairly constant.
The standard deviation shows the bias was highly
variable between 30 and 60°.
4
45°, 90° and 180° with the sector directions
Model Wind Direction BI.
12 UTe InItI.!lutlon, Tower 0020, 54 ft, *"'-Y oriented to maximize discrimination between
lO ~----------------------------- onshore and offshore flow.
5 +---------------------~------- Model Wind Direction RMSE
I O +-~+_~~~~~~~~~~~~ 12 UTe Inltlellzalon, Tower 0020,54 ft, J..w.ry
!
70 ~-------------------
S I
-5 ~ ~----------------------------
1$ e. so +--+-t--------A-~-----::+c+\o:_-+'Wor-1~__\_1_
• -lO ~~_ff--~_Hr+~_+_+~--~~~
c5 40 +-I------\---1t--'--------".....\~--~~------­
-15 +-f.-------Hl~--!..---I.+_I_------*_--- S 30 4L-----...L.Jf-----------------------
1$
-~ ~----Fo-r-ec-u-t v-.l-ld- nm-e- f-ro-m- M-o-de-lln-lt-lll-li-za-tio-n- --- a• ~ +-------------------------------
10 r-----------------------------
Figure 9. Graph of model wind direction bias from a
1200 UTC model initialization at Tower 0020 and
Forecut Vllild nme from Modellnltllllizatlon
sensor height of 54 ft for January.
There was also a slight degradation in the wind Figure 11. As in Figure 9 but for RMSE.
direction forecast through the 84-hrforecast period
In order to present the data to the LWO's in a
for this tower as shown in the standard deviation
manageable and user-friendly manner, a graphical
of the bias and RMSE in Figures 10 and 11.
user interface may be developed. A total of 9,240
graphs would be generated based on the statistics
Model St.nct.rd Oevlalon of Wind Direction BI.
12 UTe Inltlllllzlltlon, Tower 0020, 54 ft,~ being calculated for the monthly and seasonal
stratifications for all sensors on all towers. The
~ ~----------------------------
number of graphs could exceed 36,000 if the
I ~ ~----------------------------
so ~-+-t--------A-~-----:-+~-+'bor-1~-+1_ sample size is significant enough to stratify the
!4O +-I------+----IIf-----,f---C--~f1L-----K.-------­ data by sector.
.
S 30 ~--..L..lf_------------------ 8. REFERENCES
1$
a ~ ~------------------- Bauman, W. H., W. P. Roeder, R. A. Lafosse, D.
10 ~---------------------------- W. Sharp, and F. J. Merceret, 2004: The
Applied Meteorology Unit - Operational
Contributions to Spaceport Canaveral.
Forecut Vllild nme from Modellnltlellzatlon Preprints, 11th Conference on Aviation,
Range, and Aerospace Meteorology, Amer.
Figure 10. As in Figure 9 but for standard deviation
Meteor. Soc., Hyannis, MA, 4-8 October 2004,
of bias.
24 pp.
7. FUTURE WORK
Insightful Corporation, 2007: S-PLUS® 8 for
In addition to the statistics presented in this Windows® User's Guide, Insightful Corp.,
paper, the 45 WS has requested conducting Seattle, WA, 584 pp.
= =
hypothesis tests for bias 0, RMSE 0 and if the
=
composited bias and RMSE O. Also, if justified
by the sample size, the data will be stratified by
NOTICE
Mention of a copyrighted, trademarked, or proprietary product, service, or document does not constitute endorsement
thereof by the author, ENSCO, Inc., the AMU, the National Aeronautics and Space Administration, or the United
States Government. Any such mention is solely for the purpose of fully informing the reader of the resources used to
conduct the work reported herein.
5