Metrics for Offline Evaluation of Prognostic Performance

20 Pages·2010·0.65 MB·English

by  Tolga Kurtoglu

International Journal of Prognostics and Health Management
Metrics for Offline Evaluation of Prognostic Performance
Abhinav Saxena1, Jose Celaya1, Bhaskar Saha2,
Sankalita Saha2, and Kai Goebel3
1SGT Inc., NASA Ames Research Center, Intelligent Systems Division, Moffett Field, CA 94035, USA
[email protected]
[email protected]

2MCT Inc., NASA Ames Research Center, Intelligent Systems Division, MS 269-4, Moffett Field, CA 94035, USA
[email protected]
[email protected]

3NASA Ames Research Center, Intelligent Systems Division, MS 269-4, Moffett Field, CA 94035, USA
[email protected]

Several methods are suggested to customize
ABSTRACT these metrics for different applications.
Guidelines are provided to help choose one
Prognostic performance evaluation has gained method over another based on distribution
significant attention in the past few characteristics. Various issues faced by
years.*Currently, prognostics concepts lack prognostics and its performance evaluation are
standard definitions and suffer from discussed followed by a formal notational
ambiguous and inconsistent interpretations. framework to help standardize subsequent
This lack of standards is in part due to the developments.
varied end-user requirements for different
applications, time scales, available
1 INTRODUCTION
information, domain dynamics, etc. to name a
few. The research community has used a In the systems health management context, prognostics
variety of metrics largely based on can be defined as predicting the Remaining Useful Life
convenience and their respective requirements. (RUL) of a system from the inception of a fault based
Very little attention has been focused on on a continuous health assessment made from direct or
establishing a standardized approach to indirect observations from the ailing system. By
compare different efforts. This paper presents definition prognostics aims to avoid catastrophic
several new evaluation metrics tailored for eventualities in critical systems through advance
prognostics that were recently introduced and warnings. However, it suffers from inherent
were shown to effectively evaluate various uncertainties involved with future operating loads and
algorithms as compared to other conventional environment in addition to common sources of errors
metrics. Specifically, this paper presents a like model inaccuracies, data noise, and observer faults
detailed discussion on how these metrics among others. This imposes a strict requirement on
should be interpreted and used. These metrics prognostics methods to be proven and established
have the capability of incorporating though a rigorous performance evaluation before they
probabilistic uncertainty estimates from can be certified for critical applications.
prognostic algorithms. In addition to Prognostics can be considered an emerging research
quantitative assessment they also offer a field and Prognostic Health Management (PHM) has in
comprehensive visual perspective that can be most respects been accepted by the engineered systems
used in designing the prognostic system. community in general, and the aerospace industry in
particular, as the technology of the future. However, for
* This is an open-access article distributed under the terms of this field to mature, it must make a convincing business
the Creative Commons Attribution 3.0 United States License, case to the decision makers in research and
which permits unrestricted use, distribution, and reproduction development as well as fielded applications. So far, in
in any medium, provided the original author and source are
the early stages, focus has been on developing
credited.
prognostic methods themselves and very little has been
1
International Journal of Prognostics and Health Management
done to identify a common ground when it comes to - key prognostic concepts and a formal definition of
testing and comparing different algorithms. In two prognostic framework
surveys on methods for prognostics, one of data-driven - new performance evaluation metrics, their
methods (Schwabacher, 2005) and one of artificial- application and interpretation of results
intelligence-based methods (Schwabacher and Goebel, - research issues and other practical aspects to
2007) , it can be seen that there is a lack of standardized address for successful deployment of prognostics
methodology for performance evaluation and in many Relevant sources have been cited for more detailed
cases performance evaluation is not even formally studies and several findings have been summarized
addressed. Even the ISO standard (ISO, 2004) for with suitable examples and illustrations.
prognostics in condition monitoring and diagnostics of
machines lacks a firm definition of such metrics. This 1.2 Paper Organization
called for a dedicated effort to develop methods and
Section 2 motivates the development of prognostic
metrics to evaluate prognostic algorithms.
metrics. A comprehensive literature review of
Metrics can create a standardized language with
performance assessment for predicting/forecasting
which technology developers and users can
applications is then presented in section 3. This section
communicate their findings with each other and
also categorizes prognostic applications in several
compare results. This aids in the dissemination of
classes and identifies the differences from other
scientific information as well as decision making.
forecasting disciplines. Key aspects for prognostic
Metrics may also be viewed as a feedback tool to close
performance evaluation are discussed in Section 4.
the loop on research and development by using them as
Technical development of new performance metrics
objective functions to be minimized or maximized, as
and a mathematical framework for the prognostics
appropriate, by the research effort. Recently there has
problem are then presented in detail in Section 5.
been a significant push towards crafting suitable
Section 6 follows with a brief case study as an example
metrics to evaluate prognostic performance.
for application of these metrics. The paper ends with
Researchers from academia and industry are working
future work proposals and concluding discussions
closely to arrive at useful performance measures.
respectively in Sections 7 and 8.
Keeping these objectives in mind a set of metrics were
proposed and developed in the past couple years. Based
2 MOTIVATION
on experience gained from a variety of prognostic
applications these metrics were further refined. In the This research is motivated by two-fold benefits of
first phase these metrics aim to tackle offline establishing standard methods for performance
performance evaluation methods for applications where assessment (see Figure 1). One, it will help create a
run-to-failure data are available and true End-of-Life foundation for assessing and comparing performance of
(EoL) is known a priori. These metrics are particularly various prognostics methods and approaches as far as
useful for the algorithm development phase where low level algorithm development is concerned. Two,
feedback from these metrics can be used to fine-tune from a top-down perspective, it will help generate
prognostic methods. This will help associate a specifications for requirements that are imposed by
sufficient degree of confidence to the algorithms and cost-benefit-risk constraints at different system
allow their application in real in-situ environments. lifecycle stages to ensure safety, availability, and
reliability.
1.1 Main Goals of the Paper
2.1 Prognostic Performance Evaluation
This paper presents a concise discussion on prognostics
metrics that were developed as part of prognostics Most of the published work in the field of prognostics
theme within NASA‟s Integrated Vehicle Health has been exploratory in nature, such as proof-of-
Management (IVHM) project under the Aviation Safety concepts or one-off applications. A lack of standardized
program (NASA, 2009). The reader is expected to guidelines has led researchers to use common accuracy
enhance his/her understanding of the following: and precision based metrics, mostly borrowed from the
- appreciation of why a separate class of diagnostics domain. In some cases these are modified
performance metrics is needed for prognostics on an ad hoc basis to suit specific applications. This
- difference in user objectives and corresponding makes it rather difficult to compare various efforts and
needs from performance evaluation view point choose a winning candidate from several algorithms,
- what can or cannot be borrowed from other especially for safety critical applications. Research
forecasting related disciplines efforts are focusing on developing algorithms that can
- issues and challenges in prognostics and prognostic provide a RUL estimate, generate a confidence bound
performance evaluation around the predictions, and be easily integrated with
2
International Journal of Prognostics and Health Management
existing diagnostic systems. A key step in successful to capture these salient features and are expected to
deployment of a PHM system is prognosis certification. lead the way towards pragmatic and successful
Since prognostics is still considered relatively deployment of Prognostics and Health Management
immature (as compared to diagnostics), more focus so (PHM) systems.
far has been on developing prognostic methods rather
than evaluating and comparing their performances. 3 LITERATURE REVIEW
Consequently, there is a need for dedicated attention
As research activities gain momentum in the area of
towards developing standard methods to evaluate
PHM, efforts are underway to standardize prognostics
prognostic performance from a viewpoint of how post
research (Uckun et al., 2008). (Engel et al, 2000)
prognostic reasoning will be integrated into the
provides a detailed overview of prognostics along with
decision making process.
its distinction from detection and diagnosis. The
Desired
Cost of lost minimum importance of uncertainty management and the various
or performance
incomplete Time criteria other challenges in determining remaining useful life
mission needed to Algorithm
unsCcohsetd oufl ed inCcousrtr eodf prmeadkiceti oan Complexity are well presented. Understanding the challenges in
repairs damages Algorithm prognostics research is an important first step in
eLfofigciisetniccsy cFritaiciluarliety Selection standardizing the evaluation and performance
Time Best Prognostics
froerq rueipreadir aaclhgioervitahbmle Metrics assessment. Thus, we draw on the existing literature
Fault actions fidelity and provide an overview of the important concepts in
evolution
rate
Requirement Algorithm prognostic performance evaluation before defining the
Variables Fine-tuning
new metrics.
Prognostics Metrics Prognostics Metrics
Performance Performance 3.1 Prediction Performance Evaluation Methods
Specifications Evaluation
Prediction or Forecasting applications are common in

medicine, weather, nuclear, finance and economics,
Figure 1: Prognostics metrics facilitate performance
automotive, aerospace, and electronics. Metrics based
evaluation and also help in requirements specification
on accuracy and precision with slight variations are
most commonly used in all these fields in addition to a
2.2 Prognostic Requirements Specification few metrics customized to the domain. In medicine and
finance, statistical measures are heavily used exploiting
Technology Readiness Level (TRL) for the current
the availability of large datasets. Predictions in
prognostics technology is considered low. This can be
medicine are evaluated based on hypothesis testing
attributed to several factors lacking today such as
methodologies while in finance errors calculated based
 assessment of prognosability of a system,
on reference prediction models are used for
 concrete Uncertainty Representation and
performance evaluation. Both of them use some form
Management (URM) approaches,
of precision and accuracy metrics such as MSE (mean
 stringent Validation and Verification (V&V)
squared error), SD (Standard Deviation), MAD (mean
methods for prognostics
absolute deviation), MdAD (median absolute
 understanding of how to incorporate risk and
deviation), MAPE (mean absolute percentage error)
reliability concepts for prognostics in decision
and similar variants. Other domains like aerospace,
making
electronics, and nuclear are relatively immature as far
Managers of critical systems/applications have
as fielded prognostics applications are concerned. In
consequently struggled while defining concrete
addition to conventional accuracy and precision
prognostic performance specifications. In most cases
measures, a significant focus has been on metrics that
performance requirements are either derived from prior
assess business merits such as ROI (return on
experiences like diagnostics in Condition Based
investment), TV (technical value), and life cycle cost,
Maintenance (CBM) or are very loosely specified. This
rather than reliability based metrics like MTBF (mean
calls for a set of performance metrics that not only
time between failure) or the ratio MTBF/MTBUR
encompass key aspects of predicting into the future but
(mean time between unit replacements). Notions of
also accommodate notions from practical aspects such
false positives, false negatives and ROC curves have
as logistics, safety, reliability, mission criticality,
also been adapted for prognostics (Goebel and
economic viability, etc. The key concept that ties all
Bonnisone, 2005).
these notions in a prognostic framework is performance
tracking as time evolves and various trade-offs
continuously arise in a dynamic situation. The
prognostics metrics presented in this paper are intended
3
International Journal of Prognostics and Health Management
3.2 Summary of the Review requirement specification process. Therefore, further
attention in this effort was focused on algorithmic
Active research and the quest to find out what
performance metrics.
constitutes performance evaluation in forecasting
related tasks in other domains painted a wider Table 1: Classification of prognostic metrics based on
landscape of requirements and domain specific end user requirements
characteristics than initially anticipated. This naturally
translated into identifying the similarities and Category End User Goals Metrics
differences in various prediction applications to Avisasbeilsitsy tohfe p erocgonnoosmisic Cost-benefit type metrics that
Program translate prognostics performance
determine what can or cannot be borrowed from those Manager technology for specific in terms of tangible and intangible
applications before it can
domains. A classification tree was generated that listed be approved and funded. cost savings.
Accuracy and precision based
key characteristics of various forecasting applications
Resource allocation and metrics that compute RUL
and examples of domains that exhibited those. While a Plant mission planning based estimates for specific UUTs. Such
Manager on available prognostic predictions are based on
more detailed discussion can be found in (Saxena et al., information. degradation or damage
Operations accumulation models.
2008) the classification tree is shown in Figure 2. Take appropriate action Accuracy and precision based
and carry out re-planning metrics that compute RUL
Forecasting Applications Operator in the event of estimates for specific UUTs. These
contingency during predictions are based on fault
mission. growth models for critical failures.
Plan maintenance in Accuracy and precision based
End-of-Life predictions Future behavior predictions Maintainer advance to reduce UUT metrics that compute RUL
downtime and maximize estimates based on damage
A failure threshold exists Non-monotonic models availability. accumulation models.
Use monotonic decay models No thresholds
Implement the
prognostic system within Reliability based metrics to evaluate
RUL Continuous predictions the constraints of user a design and identify performance
Event predictions Prediction Weather, Finance Designer sppeerfcoirfmicaatniocnes b. yIm prove bpoetrtfloernmeacnkcse. C mometpriuctsa ttoio mnael et
Decay predictions TPrraejdeiccttoioryn Discrete predictions Engineering modifying design. resource constraints.
Develop and Implement Accuracy and Precision based
Economics, Supply Chain robust performance metrics that employ uncertainty
No/Little history data Model-based Researcher assessment algorithms management and output
Aerospace, Nuclear Data-driven Qualitative wleivthe ldse.sired confidence porf oubnacbeirlitsatiinc cporenddiictitoionnss. in presence
History data Sbeta atipspticlies dcan Idneccrreeaassiningg o trrends Thoa zaasrsdess (ss apfoetteyn, tial CAcocsut-rbaecnye afint-dr iPskre mceisaiosnu rbeass, ed RUL
Policy measures to establish guidelines &
Nominal & failure data Quantitative Regulatory Makers eacnodn eosmtaibc,l iashn dp osloicciieasl) t o tfliemeetl ainneds/ ofor rr epshoausrincge oaullot coaf taiognin fgo r
Medicine, Structures, Predict minimize their effects. future projects.
Mechanical systems numerical values

Nominal data only
Electronics, Aerospace Performance Metrics
Figure 2: Categories of the forecasting applications
Algorithm Performance
e
n
In (Coble and Hines, 2008) prognostic algorithms ilnO gn Accuracy
are categorized into three categories based on type of ik
c
models/information used for predictions. As defined arT Robustness
e
loapteerr atiino ntahle apnadp eern, vtihroensem etynptaels looaf disn faorrem aatni oinn haebroeuntt nilffO citatS Precision
Convergence
part of prognostic problems and must be used wherever
available. From the survey it was identified that not Computational Performance
only did the applications differ in nature, the metrics
Time & Complexity
within domains also varied based on functionality and
nature of the performance data end use. This led to Memory & I/O
classifying the metrics based on end usage (see Table
Cost-Benefit-Risk
1) and their functional characteristics (Figure 3). In a
ROI based
similar effort (Wheeler et al., 2009) classified the end
users from health management stakeholder point of Costs & Savings

view. Their top-level categories include Operations,
Regulatory, and Engineering, which provides another Figure 3: Functional classification of prognostics
abstraction level in classifying different user groups. metrics
While the final packets of information may be shaped
There are different types of outputs from various
differently, the core of performance assessment lies in
prognostic algorithms. Some algorithms assess Health
estimates of accuracy and precision of the prognostic
Index (HI) or Probability of Failure (PoF) at any given
algorithm. For instance, it can be argued that low level
point and others carry out an assessment of RUL based
algorithmic performance metrics are connected to
on a predetermined Failure Threshold (FT) (Saxena et
operational and regulatory branches through
4
International Journal of Prognostics and Health Management
al., 2008), (Orsagh et al, 2001), (Coble and Hines, approaches are beyond the scope of the current
2008) . Ability to generate probability distributions for discussion.
predictions further distinguishes some algorithms from
others that generate only point estimates of the 4 CHALLENGES IN PROGNOSTICS
predictions. This led to the conclusion that a formal
There are several unsolved issues in prognostics that
prognostic framework must be devised and custom
complicate the performance evaluation task. These
performance metrics should be developed that
complications have contributed in part to the lack of
accommodate most of these scenarios in an intuitive
standardized procedures. A good set of metrics should
way.
accommodate all or most of these issues but not
For further details on these classifications and
necessarily require all of them to have been addressed
examples of different applications the reader is referred
together in any single application. Enumerating these
to (Saxena et al., 2008).
issues briefly here should help understanding the
discussions on metrics development later.
3.3 Recent Developments in the PHM Domain
Acausality: Prognostics is an acausal problem that
To update the survey conducted in (Saxena et al., 2008) requires an input from future events, for instance the
relevant developments were tracked during the last two knowledge about operational conditions and load
years. A significant push has been directed towards profiles in order to make more accurate predictions.
developing metrics that measure economic viability of Similarly, to accurately assess the performance
prognostics. In (Leao et al., 2008) authors suggested a (accuracy or precision) one must know the actual end-
variety of metrics for prognostics based on commonly of-life to compare with the predicted estimates.
used diagnostic metrics. Metrics like false positives and Whereas, the knowledge about these quantities is never
negatives, prognostics effectiveness, coverage, ROC available, some estimates can be derived based on past
curve, etc. were suggested with slight modifications to usage history, plan for the mission profile, and
their original definitions. Attention was more focused predictions for future operating conditions that are not
on integrating these metrics into user requirements and controllable (e.g., weather conditions). This however,
cost-benefit analysis. A simple tool is introduced in adds uncertainty to the over all process and makes it
(Drummon and Yang, 2008) to evaluate a prognostic difficult to judiciously evaluate prognostic
algorithm by estimating the cost savings expected from performance.
its deployment. By accounting for variable repair costs Run-to-Failure Data from Real Applications:
and changing failure probabilities this tool is useful for Another aspect that makes this evaluation further
demonstrating the cost savings that prognostics can complicated is the paradox of prognostics. In real
yield at the operational levels. A commercial tool to applications if a failure is predicted an immediate
calculate the Return on Investment (ROI) for maintenance action is taken to prevent any downtime.
prognostics for electronics systems was developed Ironically this alters the original system and leaves no
(Feldman et al, 2008). The „returns‟ that are considered way to confirm whether the prediction about failure
could be the cost savings, profit, or cost avoidance by would have been accurate or not. Therefore, it has been
the use of prognostics in a system. (Wheeler et al., a challenging task to assess long term prognostic
2009) compiled a comprehensive set of user results. For instance consider the following situation. In
requirements and mapped them to performance metrics the airline industry aircraft engines undergo regular
separately for diagnostics and prognostics. maintenance and continuous monitoring for any
On algorithm performance assessment side, (Wang deterioration. A decision about when to perform the
and Lee, 2009) proposed simple metrics adapted from maintenance, if not scheduled, is a rather complex one
the classification discipline and also suggested a new that should be based on current health condition, next
metric called “Algorithm Performance Profile” that flight duration, expected operational (weather)
tracks the performance of an algorithm using a conditions, availability of spares and a maintenance
accuracy score each time a prediction is generated. In opportunity, options available for alternate planning,
(Yang and Letourneau, 2007), authors presented two costs, risk absorbing capacity, etc. In this situation one
new metrics for prognostics. They defined a reward could arguably evaluate a prognostic result against
function for predicting the correct time-to-failure that statistical (reliability) data about the RULs from similar
also took into account prediction and fault detection systems. However, once the maintenance operation is
coverage. They also proposed a cost-benefit analysis carried out two problems arise from the performance
based metric for prognostics. In some other approaches evaluation perspective. One, there is no way to verify
model based techniques are adopted where discrete whether the prognostic estimate was correct, and two,
event simulations are run and results evaluated based the useful life of the system has now changed and must
on different degrees of prediction error rates. Those have moved the end-of-life point in time from its
5
International Journal of Prognostics and Health Management
previous estimate. Alternatively, allowing the system to loss of information due to data preprocessing,
fail to evaluate the prognosis would be cost-prohibitive. approximations and simplifications),
Offline Performance Evaluation: The  operating environment uncertainties,
aforementioned considerations lead to an argument in  future load profile uncertainties (arising from
favor of controlled run-to-failure (RtF) experiments for unforeseen future and variability in usage
the algorithm development phase. While this makes it history data),
simpler for the offline performance evaluation some  input data uncertainties (estimate of initial
issues still remain. First, it is difficult to extend the state of the system, variability in material
results of offline setup to a real-time scenario. Second, properties, manufacturing variability), etc.
often in an RtF experiment the setup needs frequent It is often very difficult to assess the levels and
disassemblies to gather ground truth data. This process characteristics of uncertainties arising from each of
alters the original system and any predictions made these sources. Further, it is even more difficult to assess
prior to the disassembly may not be valid. The new how these uncertainties that are introduced at different
end-of-life point may change from what it may have stages of the prognostic process combine and propagate
been in the beginning of the experiment. This will through the system, which most likely has complex,
indicate a poor performance early on because the non-linear dynamics. This problem is made worse if the
evaluation will be based on actual EoL observed only at statistical properties do not follow any known
the end of the experiment. Therefore, one must be parametric distributions, thereby eliminating any scope
careful while interpreting the performance assessment of analytical solutions.
results. Owing to all of these challenges Uncertainty
There is no simple answer to tackle these issues. Representation and Management (URM) has become
However, using reasonable assumptions they can be an active area of research in the field of PHM. A
tackled one step at a time. For instance, most conscious effort in this direction is clearly evident from
prognostics algorithms make implicit assumptions of recent developments in prognostics (Orchard et al,
perfect knowledge about the future in a variety of ways 2009b), (DeNeufville, 2004), (Ng and Abramson,
such as following: 1990), (Sankararaman et al, 2009), (Tang et al, 2009).
 operating conditions remain more or less the These developments must be adequately supported by
same throughout systems life suitable methods for performance evaluation that can
 any change in these conditions does not affect incorporate various expressions of uncertainties in the
the life of the system significantly, or prognostic outputs.
 any controllable change (e.g., operating mode Although several approaches for uncertainty
profile) is known (deterministically or representation have been explored by researchers in this
probabilistically) and is used as an input to the area, the most popular approach has been probabilistic
algorithm representation. A well founded Bayesian framework
Although these assumptions do not hold true in most has led to many analytical approaches that have shown
real-world situations, the science of prognostics can be promise (Guan et al., 2009), (Orchard et al, 2009c),
advanced and later improved by making adjustments (Saha and Goebel, 2009). In these cases a prediction is
for them as new methodologies develop. represented by a corresponding Probability Density
Uncertainty in Prognostics: Accounting for various Function (PDF). When it comes to performance
uncertainties is of key importance in prognostics. A assessment, in many cases a simplifying assumption is
good prognostics system not only provides accurate and made about the form of distribution being Normal or
precise estimates for the RUL predictions but also any other known kernel. The experience from several
specifies the level of confidence associated with such applications, however, shows that this is hardly ever the
predictions. Without such information any prognostic case. Mostly these distributions are non-parametric and
estimate is of limited use and cannot be incorporated in are represented by sampled outputs.
mission critical applications (Uckun et al., 2008). This paper presents prognostic performance metrics
Uncertainties rise from various sources in a PHM that incorporate these cases irrespective of their
system (Coppe et al, 2009), (Hastings and McManus, distribution characteristics.
2004), (Orchard et al, 2009b). Some of these sources
include: 5 PROGNOSTIC FRAMEWORK
 modeling uncertainties (modeling errors in
First, a notational framework is developed to establish
both system model and fault propagation
relevant context and terminology for further
model),
discussions. This section provides a list of terms and
 measurement uncertainties (arise from sensor
definitions that will be used to describe the prognostics
noise, ability of sensor to detect and
problem and related concepts to develop the
disambiguate between various fault modes,
6
International Journal of Prognostics and Health Management
performance evaluation framework. Similar concepts predictions are updated until EoL is reached.
have been described in the literature, sometimes using EoUP End-of-Useful Predictions – time index
different terms; furthermore, the literature may use the beyond which it is futile to update a RUL
same terms to describe different concepts. This section prediction because no corrective action is
is intended to resolve ambiguities in interpreting these possible in the time available before EoL.
terms in the context of discussions in this paper.
EoL End-of-Life – time instant when a prediction
crosses a FT. This is determined through RtF
Prognostic Terms experiments for a specific UUT.
UUT Unit Under Test – an individual system for
which prognostics is being developed. rl(k)
Although the same methodology may be Logistics
Lead Time
applicable for multiple systems in a fleet, life
L
predictions are generated specific to each UR
UUT.
rl(k)
*
PA Prognostic Algorithm – An algorithm that
t
tracks and predicts the growth of a fault mode EoL*
t t t t
with time. PA may be data driven, model- P k EoUP EoP
based or a hybrid. Time Index (i)
RUL Remaining Useful Life – amount of time left Figure 4: An illustration depicting some important
for which a UUT is usable before some prognostic time indices (definitions and concepts).
corrective action is required. It can be
Symbols and Notations
specified in relative or absolute time units,
i time index representing time instant t
e.g., load cycles, flight hours, minutes, etc. i
l is the index for lth unit under test (UUT)
FT Failure Threshold – a limit on damage index
p set of all indexes when a prediction is made
beyond which a UUT is not usable. FT does
the first element of p is P and the last is EoP
not necessarily indicate complete failure of the
t time instant at End-of-Life (EoL)
EoL
system but a conservative estimate beyond
t time for End-of-Useful-Prediction (EoUP)
EoUP
which risk of complete failure exceeds
t time taken by a reparative action for a system
repair
tolerance limits.
t time instant when the first prediction is made
P
RtF Run-to-Failure – refers to scenario where a t time instant when a fault is detected
D
system has been allowed to fail and fli nth feature value for the lth UUT at time ti
corresponding observation data collected for n
later analysis. cml i mth operational condition for the lth UUT at ti
rl(i) predicted RUL for the lth UUT at time t
i
reference to l may be omitted for a single UUT
Important Time Indices (Figure 4)
rl(i) ground truth for RUL at time t
* i
t0 Initial time index when health monitoring for l(i| j) Prediction at time ti given data up to time tj for
a UUT begins. It can be considered the time of
the lth UUT. Prediction may be made in any
birth of a system.
domain, e.g., feature, health, RUL, etc.
F Time index when a fault of interest initiates in
l(i) Trajectory of predictions at time index i for
the UUT. This is an unobservable event until
the lth UUT
fault grows to a detectable limit.
hl(i) Health of system for the lth UUT at time ti
D Time index when a fault is detected by a
α accuracy modifier
diagnostic system. It is when a prognostic
α+ maximum allowable positive error
routine is triggered the first time.
α- minimum allowable negative error
P Time index when a prognostics routine makes λ time window modifier s.t. t t EoLt 
its first prediction. Generally speaking, there is  P P
β minimum desired probability threshold
a finite delay before predictions are available
ω weight factor for each Gaussian component
once a fault is detected.
θ parameters of RUL distribution
EoP End-of-Prediction – time index for the last π total probability mass within α-bounds [α-,α+]
prediction before EoL is reached. this is a
φ non-parameterized probability distribution
conceptual time index that depends on
φ parameterized probability distribution
θ
frequency of prediction and assumes
M(i) a performance metric of interest at time t
i
7
International Journal of Prognostics and Health Management
C center of mass as a measure of convergence  EoUP is reached and any further predictions may
M
for a metric M not be useful for failure avoidance operations
xc,yc x and y coordinates for center of mass (CM)  the system has failed (unexpectedly)
 the case where problem symptoms have
Assumptions for the Framework
disappeared (can occur if there were false alarms,
 Prognostics is condition based health assessment intermittent fault, etc.)
that includes detection of failure precursors from
sensor data, prediction of RUL by generating a Definitions
current state estimate and using expected future
Time Index: Time defines prognostics. In a
operational conditions for a specific system.
prognostics application time can be discrete or
 A suitable diagnostic algorithm correctly detects,
continuous. We will use a time index i instead of the
identifies and isolates the system fault before it
actual time, e.g., i=10 means t . This takes care of
10
triggers a PA to predict evolution for that specific
cases where sampling time is not uniform. Furthermore,
fault mode.
time indexes are invariant to time-scales.
 If the information about future operational
Time of Detection of Fault: Let D be the time index
conditions is available it may be explicitly used in
(t ) at which the diagnostic or fault detection algorithm
D
the predictions. Any prediction, otherwise,
detected the fault. This process will trigger the
implicitly assumes current conditions would exist
prognostics algorithm which should start making RUL
in future and/or variations from current operating
predictions as soon as enough data has been collected,
conditions do not affect the life of a system.
usually shortly after the fault was detected. For some
 RUL estimation is a prediction/ forecasting/
applications, there may not be an explicit declaration of
extrapolation process.
fault detection, e.g., applications like battery health
 Algorithms incorporate uncertainty representation
management, where prognosis is carried out on a decay
and management methods to produce RUL
process. For such applications t can be considered
D
distributions. Point estimates for RUL may be
equal to t (time of birth) i.e., prognostics is expected to
0
generated from these distributions through
trigger as soon as enough data has been collected
suitable methods when needed.
instead of waiting for an explicit diagnostic flag (see
 RtF data are available that include continuous
Figure 5).
sensor measurements, operating condition
Time to Start Prediction: Time indices for times
information, EoL ground truth.
when a fault is detected (t ) and when the system starts
D
 A definition of failure threshold is available that
predicting (t ) are differentiated. For certain algorithms
P
determines the EoL for a system beyond which
t = t but in general t ≥ t as PAs need some time to
D P P D
the system is not recommended for further use.
tune with additional fault progression data before they
 In the absence of true EoL (determined
can start making predictions (Figure 5). Cases where a
experimentally) statistical (reliability) data such
continuous data collection system is employed even
as MTTF (Mean Time to Failure) or MTBF
before a fault is detected, sufficient data may already be
(Mean Time Between Failures) may be used to
available to start making predictions and hence t = t .
P D
define expected EoL with appropriate caution.
Prognostics Features: Let fl(i) be a feature at time
In a generic scenario a PA is triggered by an n
index i, where n = 1, 2, …, N is the feature index, and l
independent diagnostic algorithm whenever it detects
= 1, 2, … , L is the UUT index (an index identifying the
a fault in the system with high certainty. PA may take
different units under test). In prognostics, irrespective
some time to gather more data and tune itself before it
of the analysis domain, i.e., time, frequency, wavelet,
starts predicting the growth of that fault. Based on a
etc., features take the form of time series and can be
user defined FT the PA determines where the fault is
physical variables, system parameters or any other
expected to cross the FT and system’s EoL is reached.
quantity that can be computed from observable
An estimate of RUL is generated by computing the
variables of the system that provide or aid prognosis.
difference between estimated EoL and the current time
Features can be also referred to as a 1xN feature vector
index (t ). As time progresses more measurement data
P Fl(i) of the lth UUT at time index i.
become available that are used to make another
prediction and the estimates of EoL and RUL are Operational Conditions: Let cl (i) be an operational
m
correspondingly updated. This process continues until condition at time index i, where m = 1, 2, … , M is the
one of the following happens. condition index, and l = 1, 2, … , L is the UUT index.
 the system is taken down for maintenance Operational conditions describe how the system is
8
International Journal of Prognostics and Health Management
being operated and also include the load on the system. extrapolate features and then aggregate them to
The conditions can also be referred to as a 1xM vector compute health and in other cases features are
Cl(i) of the lth UUT at time index i. The matrix Cl for all aggregated to a health and then extrapolated to estimate
times < t is referred to as load history and for times ≥ RUL.
P
tP as operational (load) profile for the system. Trajectory Prediction: Let l(i)be a trajectory of
predictions formed by point predictions for a variable
no1 40 of interest from time index i onwards such that
itidnoc d noitidnoC 20 nolt(eid) t=h at oln(liy| i)t,hel (ilas1t |ip)o,..i.n,t lo(Ef otPhi|si) tr.a jIetc tomruys, t i.eb.e,
a
o 0
L 0 50 100 150 200 250 l(EoP|i) is used to estimate RUL.
556
xednI htlaeH1 erutaeF 551550240 50 100 150 200 250 elbairaV l(i|i)l(ir1l|(ii)) l(r(i)|i)
48 de
erutaeF2 erutaeF474.57 r*l(P) tciderP Failure ThrteFsholdtD tP ti Time tr(i)
0 50 100 150 200 250
t0 tF tD tP Time Index (i) tEoL Figure 6: Illustration showing a trajectory prediction.
Predictions, and hence the corresponding RUL
Figure 5: Features and conditions for lth UUT. estimates, get updated every time instant.
Health Index: Let hl(i) be a health index at time RUL Estimate: Let rl(i) be the remaining useful
index i for UUT l = 1, 2, …, L. h can be considered a
life estimate at time index i given that the information
normalized aggregate of health indicators (relevant
(features and conditions) up to time index i and an
features) and operational conditions.
expected operational profile for the future are available.
Ground Truth: Ground truth, denoted by the
As shown in Figure 4, prediction is made at time t and
i
subscript , represents our best belief of the true value
* it predicts the RUL given information up to time i for
of a system variable. In the feature domain fl (i) may the lth UUT. RUL is estimated as
*n
be directly or indirectly calculated from measurements. rl(i)arghl(z)0i. Corresponding ground truth is
In the health domain, hl(i) is the computed health at z
time index i for UUT l* = 1, 2, …, L after a run-to- computed as rl(i)arghl(z)0iEoL i.
* * *
z
failure test. This health index represents an aggregate of
RUL* Plotted with Time
information provided by features and operational 200
conditions up to time index i. For an offline study EoL
* 150 rl(125)
is the known end-of-life point for the system.
rl(150)
History Data: History data, denoted by the subscript )i(LU 100 rl(100)
R
#, encapsulates all the a priori information we have rl(200)
about a system. Such information may be of the form of 50
archived measurements or EoL distributions, and can
0
refer to variables in both the feature and health domains 0 50 100 150 200 250
t t t t Time Index (i) t
represented by fl (i) and hl(i) respectively. By this 0 F D P EoL
#n #
definition for a fleet of systems all reliability estimates Figure 7: Comparing RUL predictions from ground
such as MTTF or MTBF would be considered history truth (t [70,240], t = 240, t > 240).
P EoL EoP
data.
Point Prediction: Let l(i| j) be a point prediction 5.1 Incorporating Uncertainty Estimates
of a variable of interest at time index i given As discussed in section 4, prognostics is meaningless
information up to time tj, where tj ≤ ti. l(i| j) for i = unless the uncertainties in the predictions are accounted
EoL represents the critical threshold for a given health for. PAs can handle these uncertainties in various ways
indicator. Predictions can be made in any domain, such as propagating through time the prior probabilities
features or health. In some cases it is useful to of uncertain inputs and estimating posteriori
9
International Journal of Prognostics and Health Management
distributions of EoL and RUL quantities. Therefore, the
+ should be designed such that they can make use of Asymmetric
α-bounds
these distributions while assessing the performance.
The first step in doing so is to define a reasonable point
estimate from these distributions such that no
interesting features get ignored in decision making. Predicted
EoLDistribution
Computationally the simplest, and hence most widely Total Probability
used, practice has been to compute mean and variance of EoL within
α-bounds
of these distributions. In reality these distributions are [EoL]
neither smooth nor symmetric resulting in large errors 
due to such simplifying assumptions especially while
carrying out performance assessment. It is, therefore,  EoL EoL*
suggested that other measures of location and variance EoLPoint EoLGround
be used instead of mean and standard deviation, which Estimate Truth
is correct only for Normal cases. For situations were
Figure 9: Concepts for incorporating uncertainties.
normality of the distribution cannot be established, it is
preferable to rely on median as a measure of location Computing the total probability mass requires
and the quartiles or Inter Quartile Range (IQR) as a integrating the probability distribution between the α-
measure of spread (Hoaglin et al., 1983). Various types bounds (Figure 10).
of distributions are categorized into four categories and
corresponding methods to compute more appropriate Obtain RUL
distribution
location and spread measures are suggested in Table 2.
For the purpose of plotting and visualizing the data use
of error bars and box-plots is suggested (Figure 8); Analyze RUL
more explanation is given in the following sections. distribution
+95% bound
Y Known
Normal
Distribution ?
25% quartile
50% quartile N
1 2 3
1 2 3 Identified with a Are there N
known parametric outliers?
-95% bound distribution Classify as a
non-parametric
Y
distribution
Optional
outliers
Estimate Remove kernel
Figure 8: Visual representation for distributions. parameters (θ) outliers smoothing
Trim data to ignore
While point estimates are good for easy unexplained outliers
Analytical form Probability
understanding they can be less robust when deviations of PDF Histogram
from assumed distribution category are random and
frequent. Furthermore, given the fact that there will be Integrate the pdf Sum samples in the
uncertainty in any prediction one must make provisions between α-bounds to pdf histogram that
to account for those. One common way to do so is to obtain total probability fall within α-bounds
specify an allowable error bound around the point of
interest and instead of basing a decision on a point
estimates one could estimate the total probability of
failure within that error bound. This error bound may   Total probability  
bspierne cfaees ryriemtd m isoe vtoreifrct eane s lpaaertcegi uapellrdye dtiihncat itto hnae.n cTaehsaeersl yeo fi pdpreeradosig cntcioaosnnt i cbises, [EoL(x])dx;x of RαU-bLo buentdwse en [EoL(]x);xΙ
analytically incorporated into the numerical aspect of
Figure 10: Procedure to compute total probability of
the metrics by computing total probability mass of a
RULs being within specified α-bounds.
prediction falling within the specified α-bounds. These
concepts have been illustrated in Figure 9.
10