Tennis Ball Tracking for Automatic Annotation of Broadcast Tennis Video

142 Pages·2008·16.61 MB·English

Tennis Ball Tracking for
Automatic Annotation of Broadcast Tennis Video
Fei Yan
SubmittedfortheDegreeof
DoctorofPhilosophy
fromthe
UniversityofSurrey
CentreforVision,SpeechandSignalProcessing
SchoolofElectronicsandPhysicalSciences
UniversityofSurrey
Guildford,SurreyGU27XH,U.K.
June 2007
(cid:13)c FeiYan2007

Summary
Thisthesisdescribesseveralalgorithmsfortrackingtennisballsinbroadcasttennisvideos. The
algorithms are used to provide a ball tracking module for an automatic tennis video annotation
system.
As an overall strategy, we have decided to adopt the Track-After-Detection (TAD) approach to
thetrackingproblem. ATADapproachdecomposesatrackingproblemintotwosub-problems:
object candidate detection and object candidate tracking. In the second sub-problem, resolving
data association ambiguity is the key. In this thesis, we propose a novel tennis ball candidate
detectionalgorithm,andthreenoveldataassociationalgorithms.
Theballcandidatedetectionalgorithmextractsforegroundmovingblobsusingtemporaldiffer-
encing, and classify the blobs into ball candidates and non-candidates. A novel gradient-based
featureformeasuringthedepartureoftheshapeofablobfromanellipseisproposed.
Theproposeddataassociationalgorithmstakeasinputthepositionsoftheballcandidates,and
try to resolve the object-candidate association ambiguity. The first data association algorithm
is based on a particle filter with an improved sampling scheme. Smoothing and an explicit
data association process are also used to increase the accuracy of the tracking results. The
second data association algorithm is based on the Viterbi algorithm. It seeks the sequence of
object-candidate associations with maximum a posteriori probability, using a modified version
oftheViterbialgorithm. Thethirddataassociationalgorithmisamulti-layeredonewithgraph-
theoretic formulation. At the bottom layer, “tracklets” are “grown” from ball candidates with
anefficientvariantoftheRANdomSAmpleConsensus(RANSAC)algorithm. Theassociation
problemisthenformulatedasanall-pairsshortestpath(APSP)probleminagraphwithtracklets
asitsnodes,andissolvedwithanovelfastAPSPalgorithm.
The proposed data association algorithms have been tested using tennis sequences from the
Australian Open tournaments. Their performances are compared both mutually and to existing
objecttrackingalgorithms. Experimentsshowthattheparticlefilterbasedalgorithmgivessim-
ilar performance to the state-of-the-art tennis ball tracking method in the literature: the Robust
DataAssociation(RDA),whiletheothertwoproposedalgorithmsbothoutperformRDA.
Key words: Sports Video Annotation, Tennis Ball Tracking, Data Association, Sequential
Monte-CarloMethod,ViterbiAlgorithm,GraphTheory,ShortestPath
Email: [email protected]
WWW: http://www.ee.surrey.ac.uk/cvssp
Acknowledgements
I would like to thank my supervisors, Dr. William Christmas and Prof. Josef Kittler, for their
guidance and support throughout the course of this work. I would also like to thank everyone
who has offeredhis or her help, encouragementand friendship. Inparticular, thanks are due to
YonggangJinandmycolleaguesintheVAMPIREteam, IliasKoloniasandAlexeyKostin, for
thestimulatingdiscussions.
Thisworkwassupportedunderthefollowingprojects:
EUIST-2001-34401ProjectVAMPIRE
EUIST-507752MUSCLENetworkofExcellence
Contents
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 DifficultiesofBallTrackinginBroadcastTennisVideo . . . . . . . . . . . . . 2
1.3 TheOverallStrategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 OverviewofthisThesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 TheTennisVideoAnnotationSystem 9
3 LiteratureReview 15
3.1 DataAssociationandEstimation . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 KalmanFilterandKalmanSmoother . . . . . . . . . . . . . . . . . . . . . . . 16
3.3 NearestNeighbourStandardFilter(NNSF) . . . . . . . . . . . . . . . . . . . 18
3.4 OptimalBayesianFilter(OBF) . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 ProbabilisticDataAssociation(PDA) . . . . . . . . . . . . . . . . . . . . . . 20
3.6 ProbabilisticMulti-HypothesisTracker(PMHT) . . . . . . . . . . . . . . . . . 21
3.7 SequentialMonte-CarloMethod . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.8 TheRANdomSAmpleConsensus(RANSAC)Algorithm . . . . . . . . . . . . 24
3.9 ExistingTennisBallTrackingAlgorithms . . . . . . . . . . . . . . . . . . . . 26
3.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 BallCandidateDetection 29
4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 MotionSegmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.3 ForegroundBlobClassification . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
v
vi Contents
5 DataAssociationwithSequentialMonte-CarloMethod 39
5.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2 ProblemFormulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 ParticleFilteringwithanImprovedSamplingEfficiency . . . . . . . . . . . . 41
5.4 UsingaSecondDynamicModel . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.5 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.6 DataAssociation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6 DataAssociationwiththeViterbiAlgorithm 59
6.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.2 ProblemFormulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.3 TheViterbiAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6.4 DefiningtheEdgeWeight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.5 ModifyingtheViterbiAlgorithmtoAllowSuccessiveMisdetections . . . . . . 64
6.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
7 LayeredDataAssociationwithGraph-TheoreticFormulation 73
7.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.2 OverviewoftheStrategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.3 CandidateLevelAssociation . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.3.1 LookingForaSeedTriplet . . . . . . . . . . . . . . . . . . . . . . . . 77
7.3.2 FittingaModeltotheSeedTriplet . . . . . . . . . . . . . . . . . . . . 78
7.3.3 OptimisingtheModelRecursively . . . . . . . . . . . . . . . . . . . . 79
7.3.4 StoppingCriteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.3.5 DefiningTracklet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
7.4 TrackletLevelAssociation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
7.4.1 ConstructingaWeightedandDirectedGraph . . . . . . . . . . . . . . 85
7.4.2 FindingtheSingle-PairShortestPath . . . . . . . . . . . . . . . . . . 87
7.5 PathLevelAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Contents vii
7.5.1 ThePathsReduction(PR)Algorithm . . . . . . . . . . . . . . . . . . 90
7.5.2 DefiningQualityandCompatibilityofPaths. . . . . . . . . . . . . . . 91
7.5.3 ApplyingthePRAlgorithmtoQ. . . . . . . . . . . . . . . . . . . . . 93
7.6 ProcessingSpeedConsiderations . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.6.1 CandidateLevelAssociation . . . . . . . . . . . . . . . . . . . . . . . 96
7.6.2 TrackletLevelAssociation . . . . . . . . . . . . . . . . . . . . . . . . 96
7.6.3 PathLevelAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.7 ExperimentsSessionI:TrackingaSingleBallSemiAutomatically . . . . . . . 100
7.8 ExperimentsSessionII:TrackingMultipleBallsAutomatically . . . . . . . . . 103
7.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8 ComparisonoftheAlgorithmsandDiscussion 111
8.1 AlgorithmsforComparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.2 PerformanceofRDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.3 ComparisonoftheAlgorithmsandDiscussion . . . . . . . . . . . . . . . . . . 116
8.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
9 SummaryandFutureWork 123
9.1 SummaryoftheThesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
9.2 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
viii Contents
Chapter 1
Introduction
1.1 Motivation
The rapid growth of sports video database demands efficient tools for automatic sports video
annotation. Owing to recent advances in both computer hardware and computer vision algo-
rithms,buildingsuchtoolshasbecomepossible. Thesportsbeingannotatedincludesoccer[78],
Formula-1racing[48],snooker[18],tennis[16,33]. Suchsportsvideoannotationsystemshave
manypotentialapplications,e.g.videoretrieval,enhancedbroadcast,object-basedvideoencod-
ing,videostreaming,analysisofplayertactics,computeraidedcoaching.
Inthepastthreeyears,mycolleaguesandIhavebeenbuildingatennisvideoannotationsystem.
Thatis,acomputervisionsystemthatcan“understand”tennis,inthesensethatittakesasinput
abroadcasttennisvideo, andgenerateshighlevelannotation, suchasthescoreofthegame, as
thegameprogresses[6,32,34].
Inautomaticvideoannotation,highleveldescriptionsrelyonlowlevelfeatures. Inthecontext
of a tennis game, the ball trajectory contains a rich set of information for the annotation. The
motivationoftheworkpresentedinthisthesiswastoprovideatennisballtrackingmodulefor
ourtennisvideoannotationsystem. Thereconstructedballtrajectoriesgivenbytheballtracking
moduleareusedtogetherwithotherlowlevelfeatures,suchasplayertrajectoriesandcourtlines,
togeneratehighlevelannotation.
1
2 Chapter1. Introduction
1.2 Difficulties of Ball Tracking in Broadcast Tennis Video
Tennisballtrackinginbroadcasttennisvideoisachallengingtask. Atennisballhasaverysmall
size,andcantravelatveryhighvelocity. Forexample,inaframewithresolution268×670,a
tennis ball can be as small as only 5 pixels when it is at the far side of the court. A tennis ball
can travel at up to 70 meters per second, which means in the image plan, the distance between
itspositionsintwosuccessiveframescanbemorethan20pixels.
Atennisballisalsosubjecttomotiondeformation,motionblur,successiveocclusion,andsud-
den change of motion direction. All these problems pose great challenges for the tracking,
especiallywhentheyarecombined. Forexample,whentheballishitbyaplayer,itchangesits
motiondrastically. Sincetheballtravelsatveryhighvelocityafterbeinghit, itisoftenblurred
into background, and cannot be detected in the first few frames. As a result, the next detected
ball position can be very far from the predicted position that is obtained using an obsolete mo-
tion model. This is much more challenging than tracking an object with a smooth manoeuvre.
Manyexistingtrackerswouldlosetrackinsuchasituation.
Theuncontrollabilityofthemakingoftennisvideoaddsfurthercomplexity. Forexample,there
maybemultipleplaysinonesequence;theremaybeballsthrowninbyballboyswhentheball
usedforplayisstillinthescene;thecameramaynotpanortiltfastenoughtokeepupwiththe
ball, as a result, the ball may exit the scene and then re-enter. In brief, tennis ball tracking in
monocular sequences is a multiple object tracking problem where the objects can change their
motionsabruptly;newobjectscanappear;existingobjectscandisappearandreappear. Itistoo
challengingforclassicalmethodstoworkdirectlywithoutmuchcarefulandpertinentredesign.
SomeexamplesofthedifficultiesofballtrackinginmonoculartennisvideoareshowninFig.1.1
andFig.1.2.
1.3 The Overall Strategy
Objecttrackingisoneofthekeytopicsofcomputervision. Brieflyspeaking,objecttrackingis
concerned with finding and following the object of interest in a video sequence. Two pieces of
informationareessentialforanyobjecttrackingproblem: objectappearanceandobjectdynam-
ics. Allobjecttrackingtechniquessomehowmakeuseofbothpiecesofinformation. Depending