NASA Technical Reports Server (NTRS) 20120001617: Directed Design of Experiments for Validating Probability of Detection Capability of a Testing System

0.99 MB·English

by  NASA Technical Reports Server (NTRS)

mu uuuu ui iiui imi uui uiu imi uui uiu mii uuii uu uii mi
United States Patent
(12) (1o) Patent No.: US 8,108,178 B2
Generazio (45) Date of Patent: Jan. 31, 2012
(54) DIRECTED DESIGN OF EXPERIMENTS FOR 6,799,145 B2 * 9/2004 Kennedy et al . .............. 702/182
VALIDATING PROBABILITY OF DETECTION 7,155,369 B2 12/2006 Papadimitriou et al.
7,333,648 B2 2/2008 Edic et al.
CAPABILITY OF A TESTING SYSTEM
7,403,871 B2 7/2008 Papadimitriou et al.
7,490,319 B2 * 2/2009 Blackwell et al . ............ 717/124
(75) Inventor: Edward R. Generazio, Yorktown, VA 7,917,895 B2 * 3/2011 Givoni et al . ................. 717/124
(US) 2006/0010426 Al* 1/2006 Lewis et al .................... 717/124
2008/0183402 Al 7/2008 Malkin et al.
(73) Assignee: The United States of America as
represented by the Administrator of OTHER PUBLICATIONS
the National Aeronautics and Space
Yee et al., Assessment of NDE Reliability Data, Final Report, Jul.
Administration, Washington, DC (US)
1974-Sep. 1975 (General Dynamics/Fort Worth and Vanderbilt Uni-
(*) Notice: Subject to any disclaimer, the term of this versity, Nashville, Tennessee) NASA-CR-134991.
patent is extended or adjusted under 35 Ward D. Rummel, Recommended Practice for Demonstration of
U.S.C. 154(b) by 352 days. Nondestructive Evaluation (NDE) Reliability on Aircraft Production
Parts, Materials Evaluation, vol. 40, Aug. 1982.
(21) Appl. No.: 12/467,475
(Continued)
(22) Filed: May 18, 2009
Primary Examiner John H Le
(65) Prior Publication Data
(74) Attorney, Agent, or Firm Thomas K. McBride, Jr.;
US 2010/0122117 Al May 13, 2010 Robin W. Edwards
Related U.S. Application Data
(57) ABSTRACT
(60) Provisional application No. 61/053,694, filed on May
16, 2008, provisional application No. 61/109,531, A method of validating a probability of detection (POD)
filed on Oct. 30, 2008, provisional application No. testing system using directed design of experiments (DOE)
61/158,868, filed on Mar. 10, 2009. includes recording an input data set of observed hit and miss
or analog data for sample components as a function of size of
(51) Int. Cl. a flaw in the components. The method also includes process-
G06F 19100 (2006.01) ing the input data set to generate an output data set having an
G06F 11130 (2006.01) optimal class width, assigning a case number to the output
(52) U.S. Cl. ........... 702/179; 702/81; 702/182; 717/124 data set, and generating validation instructions based on the
(58) Field of Classification Search .................... 702/81, assigned case number. An apparatus includes a host machine
702/179, 182; 700/79, 245; 714/26, 35, for receiving the input data set from the testing system and an
714/39; 717/124 algorithm for executing DOE to validate the test system. The
See application file for complete search history. algorithm applies DOE to the input data set to determine a
data set having an optimal class width, assigns a case number
(56) References Cited to that data set, and generates validation instructions based on
the case number.
U.S. PATENT DOCUMENTS
6,157,699 A 12/2000 Dunn
6,195,624 B1 2/2001 Woodman et al. 18 Claims, 6 Drawing Sheets
100
Receive and Record 102
Hit/Miss Input Data [221
Determine Data Set Having 104
Optimal Class Width
Assign Case Number to106
Optimal Data Set
Generate Output 108
Instructions (241
Finish
US 8,108,178 B2
Page 2
OTHER PUBLICATIONS tems and Qualification of Inspectors (DOEPOD)" Quality Leader-
ship Forum, KSC, Mar. 18-19, 2009.
Standard NDE Guidelines and Requirements for Fracture Control
Edward R. Generazio, "Nondestructive Testing for Mission Assur-
Programs, MSFC-STD-1249, National Aeronautics and Space ance", NDE/NDT for Highways and Bridges: Structural Materials
Administration, Sep. 1985. Technology (SMT) 2008, Oakland, CA Sep. 8-12, 2008.
NASA-STD-5009, Nondestructive Evaluation Requirements for Edward R. Generazio, "Directed Design of Experiments for Validat-
Fracture C cal Metallic Components, Apr. 7, 2008. ing Probability of Detection Capability of NDE Systems
Mil-Handbook-1823, Nondestructive Evaluation System Reliability (DOEPOD)", 34th Annual Review in Quantitative Nondestructive
Assessment, Apr. 30, 1999. Evaluation, Jul. 2007.
Edward R. Generzaio, "Design of Experiments for Validating Prob-
ability of Detection Capability of Nondestructive Evaluation Sys- * cited by examiner
U.S. Patent
Jan. 31, 2012 Sheet 1 of 6 US 8,108,178 B2
T22
Test Validation
12 System Host 10
1 100
24
q
^ q

q
1120
El ^ FIG.
1s
(Inspector
14
100
Start/—
Receive and Record102
Hit/Miss Input Data [22] K
Determine Data Set Having 104
Optimal Class Width
Assign Case Number to106
Optimal Data Set
r
Generate Output 108
Instructions [24]
Finish
FIG. 3
U.S. Patent Jan. 31, 2012 Sheet 2 of 6 US 8,108,178 B2
22

/-
Typical Hit / Miss Data
1
0.9
30
0.8
X
0 ©.7
0.6
0
0.5
a
x Hit /(Miss
0.4
0 0.3
L.
a. 0.2
0.1
0
0.000 0.500 1.000 1.500 2.000 2.500 3.000 3.500
Size (inch), X1
FIG. 2
U.S. Patent
Jan. 31, 2012 Sheet 3 of 6 US 8,108,178 B2
Observed Probability of Hit (POH)
1.00
0.90
0.80
_ 0.70
O
0.s0
0.50
0
0.40
sE
co U
.n
v 0.30
0.20
0.10
0.00
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50
Size (inch): X1 OPOH 0.001" Classwidth
0 PL 0.001" Classwidth
FIG. 4 X
Hit/Miss
U.S. Patent
Jan. 31, 2012 Sheet 4 of 6 US 8,108,178 B2
Observed Probability of Hit (POH)
1.00
0.90
0.80
0.70
o ^
0.60
0.50
=C:
0
U 0.40
o a)
0.30
0.20
0.10
0.00 0.50 1.00 1.50 2.00 2.50 3.00 150
Size (inch), X1
XHit/Miss
POH 0.100" Classwidth
FIG. 5
o PL 0.100" Classwidth
U.S. Patent
Jan. 31, 2012 Sheet 5 of 6 US 8,108,178 B2
--
60
Probability of Success in Determining if the POD of Large Flaws is Less than 90/95 POD
62
1.0
U ;
AL
^ 0.9
U) 0 0.8 77
HIGH
O U
CONFIDENCE
U' 'n o i ZONE
U 0.6 N90195 sac = 25 -
0
U ^U1 0.5
0.4 1
POS A6003H
IG H RI K A)N E
0 3 A LC of POS A6003H
m
o m 0.2 _ 0 POS D1002BD
IL 1♦T LCL] of POS ID1002IBD
J 0.1
00.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
Number of Unique Random Flaws Larger Than 90/95 POD, Xpod
FIG. 6
U.S. Patent
Jan. 31, 2012 Sheet 6 of 6 US 8,108,178 B2
0
e`
N XO .W ' lG GOOo DI LN(D 4Nl ¢.-l W^ Sv1 G N?1G C^ ^O, Opf-^^m'i L G^O LCj ^^°U -TC X
3 ..... .o ^ n e' x G
m a. ^ m ^^p X ^ ^ , ^ y m 4 ^ a^i ^ ^ F^-y ^ ^a ,i x o y >'
cQE^l^c4) .rB.a0 -r:3^0O^°i .^L.^r^^n rnmAawm .3v3^0`C, "ULLmmo°Oi' ^V-0 ydtrai at^i c,dwa^^ i u.-mN-mgC0, I .^SyNm,y^t'i l] m, .mG 'C9pAcaNJ: .LLL^Oidc^1 m o EGNw?4 . mm0aCj?'S_paS u^LwcmC°; X4^Nfa^3 ^LmNC U-1Q^ c^ccc LrL~itm^$tnO1 .Nt.,i0^a-€ "oie4w_ .> ^r ca»a^. L-L^O^eN° ' 'N^^N3^W_' cr'd'nTtM^^lo CcNN, M,s-tmc3`o-- m zha0Sa s.'Fc3MO 3mC^^N3..°
6`O>c14t,, !OXCQrm^.L] e^tri. ^OaA7-7qC^6 . Nm%Gqw`-n L- mnmoOOf}pmaCGoi ' €'?`^v- . TQaids' .'rxDO3^'oa'C. ^,-m`vp °wi-Q `n^ ^.n^m4> cRv.' CO^^Xm'o 7 ^fEa f "d y2 a_cmd .Q b^-^^Q XO°mvo_- . CccyN^Q?_Gm1Go 7°6nm ^ i Cicnm`S p^rOoc . X^nmaa ^ .'N e-oaH+ ^ .r,o' XaOnioo- .Eca^N o^"v .i . v^eO-aoi,,. ^rEp3m-H^ La_^r —`GwvgnyOm, - '`c4-c^'r
D rn^ `f a rn4.^ m< M 6 Bi E`t °m^.W d rn n N' °
co
3 ^mr^ 000 O z< zQ Qz Q zQ
a
m ^ cUc
N ^ nl
-fir <^^g < -y 0
z
z
w^ o
II
000 0 000 0 z
1 z
fl
0 C) 0
x
!mil ``, '` tF =°' mss.
O LiX.!
h^
U W
@ Q
0 0 0 0 0000 0 0
rn o _€a
t- cv ter' cn co W
W W 311 W W W W L J Lid s f1LIjJ
-) U U 4^J Cam] S¢J U co U
N
I^-
CL u-
US 8,108,178 B2
1 2
DIRECTED DESIGN OF EXPERIMENTS FOR The determination of estimated POH at a selected flaw size
VALIDATING PROBABILITY OF DETECTION may be a directly measured or observed value between 0 and
CAPABILITY OF A TESTING SYSTEM 1. For a single trial, a "miss" is equal to 0 and a "hit" is equal
to 1. Knowledge of an estimated POH yields a measure of the
CROSS-REFERENCE TO RELATED 5 lower confidence bound, or Pi. This process is statistically
APPLICATIONS referred to as "observation of occurrences" and is distinct
from use of functional forms that predict POD.
This application claims priority to and the benefit of U.S. Traditionally, binomial distributions have been used for
Provisional Application 61/053,694 filed on May 16, 2008, determining POD by direct observation of occurrences. Con-
U.S. Provisional Application 61/109,531 filed on Oct. 30, lo ventional binomial methodologies use a selection of arrange-
2008, and U.S. Provisional Application 61/158,868 filed on ments for grouping flaws of similar characteristics. These
Mar. 10, 2009, each of which is hereby incorporated by ref- approaches have led to the general acceptance of using the 29
erence in its entirety. out of 29 (29/29) binomial point estimate method, in combi-
15 nation with validation that the POD is increasing with flaw
ORIGIN OF THE INVENTION size, in order to meet certain governmental requirements or
standards, e.g., MSFC-STD-1249, NASA-STD-5009, or
The invention described herein was made by employees of similar standards.
the United States Government and may be manufactured and
used by or for the Government of the United States of SUMMARY OF THE INVENTION
20
America for governmental purposes without the payment of
any royalties thereon or therefor. Accordingly, a method and an apparatus as set forth herein
provide a cost-effective way to validate the detection capa-
TECHNICAL FIELD bility of various inspection or testing systems, with the term
25 "validating" as used herein referring to an approval decision
The present invention relates generally to the validation of reflecting that the testing system meets a particular inspection
a statistics-based testing system, and in particular to a com- requirement or accuracy threshold. The present invention
puter-executed process or method that uses directed design of works in binomial applications for POD by adding the con-
experiments (DOE) to validate the probability of detection cept of a computer-executable lower confidence bound opti-
(POD) capability of such a testing system. 30 mization process as the driver for establishing a POD thresh-
old, e.g., a 90/95 POD according to one embodiment, or any
BACKGROUND OF THE INVENTION other desired POD threshold such as but not limited to 90/99
POD, 80/95 POD, etc., depending on the particular applica-
Certain applications may require nondestructive inspec- tion.
tion evaluation (NDE) of new or used fracture-critical and/or 35 The method and apparatus satisfy the requirement for criti-
failure-critical components. For example, in space-based and cal applications where validation of inspection or testing
certain aeronautical applications, there may be elevated con- systems, individual procedures, and ultimate qualification of
cern relating to the use of certain components due to aging human or robotic operators is required. Additionally, the
and/or impact damage of the components. The presence of method and apparatus yield an observed estimate of POD
one-of-a-kind or few-of-a-kind critical components having a 4o rather than a predicted estimate of POD, with functionality
limited inspection history and use, and/or that are constructed based on the application of the binomial distribution to a set of
of materials having limited availability, has only enhanced the flaws that are automatically grouped into classes having pre-
overall inspection concern. determined widths, i.e., class widths.
The determination of the capability of conventional inspec- The classes are automatically and systematically varied in
tion systems and methodologies using curve fitting or other 45 class width using an automatic iteration process and DOE,
techniques may be insufficient for use with updated and rap- with a host machine processing the input data set as described
idly changing inspection requirements for such systems. For below to determine a data set having an optimal class width.
example, the National Aeronautics and Space Administration In one embodiment the iteration may start at a minimally
(NASA) currently requires on-orbit inspections of the Space sized class width, e.g., approximately 0.001", and change by
Shuttle Orbiter's external thermal protection system. On- 50 constant values, e.g., increments of 0.001" up to a maximum
orbit testing is typically performed by trained astronauts as an expected flaw size. Class width groupings may also start at the
extravehicular activity (EVA). Inspection of fracture-critical largest expected flaws and move toward the smallest expected
and failure-critical components requires inspection to be at flaw size. Flaw size may be any flaw dimension such as width,
90% probability of detection (POD) with a 95% level of height, depth, volume, shape, etc., when used to describe
confidence, commonly referred to in the art and herein as a 55 physical flaws, or another value such as delivery time, flavor
90/95 POD. level, engineering-quality, etc., for other testing systems not
Design of experiments or DOE describes a statistics-based concerned with physical flaws, without departing from the
process in which changes are made to various input variables intended scope of the invention.
of a system, and the effects on response variables are mea- The largest class length in the first class width group may
sured and recorded. DOE may utilize the concept of "point 6o be assigned as the identifier in the group. The next moving
estimate probability of a hit" or POH at a given "flaw" size, class width group may be identified by decrementing the
with the term "flaw" referring to a physical flaw such as a upper and lower class lengths by the constant value, e.g.,
crack in a component when used with physical inspection 0.001" in one embodiment. The present invention may also
systems. When used with other systems, the term "flaw" may require for the purposes of validation that the POD increases
refer to any other variable one wishes to inspect for, e.g., 65 with flaw size within the range of flaw sizes for which the
delivery times, flavor levels in a food product, engineering results are valid, and may require inclusion of larger flaw
properties, etc. sizes in the optimization process as set forth hereinbelow.
US 8,108,178 B2
3 4
The present invention evaluates the lower confidence mined threshold. e.g., 90/95 POD or another threshold, and if
bound (Pi) obtained from any class width group. If the lower desired, for qualifying an operator or inspector 14 for opera-
confidence bound equals or exceeds 0.90 at any given class tion of the testing system 12. In FIG. 1, the testing system 12
width group, there exists a grouping of flaws detected at the is represented as a computer device for simplicity; however,
desired 90/95 POD or greater level. Otherwise, such a group- 5 the testing system 12 may include, or may itself be configured
ing does not exist. As an output or deliverable product, the as, an inspection procedure, e.g., where an inspector uses
present invention may provide a detailed set of instructions to liquid penetrants and spray developers with ultraviolet (UV)
a user of the testing system for obtaining the desired POD at light and a I Ox magnifier, etc.
a given or an alternate flaw size. Testing system 12 may be, according to one embodiment,
In particular, the present invention provides a method and io a non-destructive inspection and evaluation (NDE) inspec-
apparatus for optimizing the lower confidence bound by tion system configured for use in the inspection of samples
adjusting the class widths used in the binomial analysis. Once 16. In such an embodiment, the samples 16 may be physical
the optimized lower confidence bound is determined, the components, and inspection may be performed to identify
input data set is identified as a particular case. After deter- cracks, pits, chips, or other flaws. Those of ordinary skill in
mining the case, the test system is either validated to be at the 15 the art will recognize other potential variations of the testing
threshold inspection capability or the test system is not vali- system 12 unrelated to inspection of physical components,
dated to be at threshold inspection capability. If the inspection e.g., physical delivery or logistical systems, food flavor or
system is not validated to be at the threshold inspection capa- other quality sampling, engineering property sampling, etc.,
bility then instructions are given, that, when executed suc- that are nevertheless statistical in nature, may also be used
cessfully, yield an inspection system that is at a threshold 20 within the scope of the invention. However, for simplicity, the
inspection capability or an alternate threshold inspection inspection of samples 16 in the form of physical components
capability, or the inspection system is not capable of demon- will be described hereinbelow.
strating the threshold inspection capability. Additional vali- The testing system 12 and host 10 may be configured as
dation at the threshold inspection level is performed to assure microprocessor-based devices having such common ele-
that the inspection capability is increasing with flaw size, by 25 ments as a microprocessor or CPU, memory including but not
including a number of large flaws in the sample set. The limited to: read only memory (ROM), random access
capability to include other POD data sets to extend the range memory (RAM), electrically-programmable read-only
of validation and to limit the sample requirements to meet memory (EPROM), etc., and circuitry including but not lim-
geometric needs is included. The false call analysis requiring ited to: a high-speed clock (not shown), analog-to-digital
a minimum specified number of false call opportunities is 30 (A/D) circuitry, digital-to-analog (D/A) circuitry, a digital
required to complete all validation and qualifications. signal processor or DSP, and the necessary input/output (I/O)
The above features and advantages and other features and devices and other signal conditioning and/or buffer circuitry.
advantages of the present invention are readily apparent from An inspector 14, e.g., a human inspector or an automated
the following detailed description of the best modes for car- inspection device or robot, physically inspects each of the
rying out the invention when taken in connection with the 35 samples 16 and records the inspection results 20. Samples 16
accompanying drawings. may be physical components as noted above such as parts of
a space vehicle, platform, aircraft, etc., or anything else to be
BRIEF DESCRIPTION OF THE DRAWINGS inspected. The inspection results 20 may describe the
observed size of each of the flaws detected by the inspector
FIG.1 is a schematic illustration of a host machine config- 40 14, or the detected amplitude or analog values as noted above.
ured for validating a test system in accordance with the inven- When referring to something other than a physical compo-
tion; nent, the term "flaws" may describe a predetermined varia-
FIG. 2 is a chart describing an input data set of hit/miss data tion from the expected norm.
for the test system shown in FIG. 1; The testing system 12 includes a calibrated data set 18 of
FIG. 3 is a flow chart describing a method that may be 45 the actual or known flaws contained in components 16. That
executed using the host machine shown in FIG. 1; is, the collective set of samples 16 has known flaws and size
FIG. 4 is a chart describing an initial set of observed prob- distributions. For example, calibrated data set 18 may be
ability of hit (POH) data; determined via direct measurement and/or testing, whether
FIG. 5 is a chart describing a partially optimized set of nondestructive or destructive, and recorded in memory within
observed POH data; 50 or accessible by the testing system 12. After the data set 18 is
FIG. 6 is a chart describing a probability of success in recorded, the inspector 14 is provided with the samples 16
determining if the POD of large flaws is less than 90/95 POD and is required to identify each of the known flaws in the
in an exemplary embodiment; and components 16.
FIG. 7 is a table describing a set of cases useable by the host Referring to FIG. 2, once the inspection results 20 are
machine of FIG. 1. 55 recorded by the testing system 12, the test system may auto-
matically compare the inspection results 20 to the values in
DESCRIPTION OF THE PREFERRED the calibrated data set 18 to determine whether a hit or miss is
EMBODIMENTS observed for each test. When used to detect physical flaws,
flaw length may be the detected value according to one
Referring to the drawings, wherein like reference numbers 60 embodiment, and length is therefore used hereinbelow for
represent like components throughout the several figures, and simplicity even though descriptive values other than length
beginning with FIG. 1, a validation host machine 10, herein- may also be used. Alternately, analog values may be entered
after referred to as the host 10, includes an algorithm 100 by the inspector 14 with the testing system 12 including a
suitable for executing a test system validation method as set threshold, and the testing system 12 or host 10 may compare
forthbelow withreference to FIG. 3. The host 10 may be used 65 the results 20 with the threshold to determine the hit/miss
in conjunction with a testing system 12 for validating the results test. The analog threshold may be optimized to pro-
detection capability of the testing system 12 to a predeter- vide a tradeoff between optimum POD and false call rates.