Profiling Finger-Hand Function of Rheumatoid Arthritis Patients Using a Telerehabilitation Gaming

228 Pages·2014·10.86 MB·English

Profiling Finger-Hand Function of Rheumatoid
Arthritis Patients Using a Telerehabilitation
Gaming System
by
Daniel Lockery
AThesissubmittedtotheFacultyofGraduateStudiesof
TheUniversityofManitoba
inpartialfulfilmentoftherequirementsofthedegreeof
DOCTOROFPHILOSOPHY
DepartmentofElectricalandComputerEngineering
UniversityofManitoba
Winnipeg
Copyright(cid:13)c 2014byDanielLockery
Abstract
Theproblemconsideredinthisthesisisdevelopingasetofdigitalfeaturesrelevant
in describing finger-hand function of early-onset rheumatoid arthritis (RA) patients.
The premise is based on a novel telerehabilitation gaming system that operates on a
store-and-forward design. The solution to this problem was to develop a full-scale
gamingplatformtoexamineclientmovementperformanceforprecisionaimingtasks
based on a set of digital features. To complement the movement performance, still
imageryinthreeuniqueposesarecapturedduringasessiontodetectvisualsymptoms
duringdiseaseactivityandearlywarningsignsofdeformitiesthatcanarisefromjoint
damage. Resulting data is gathered in a clinic or housed in a content management
system where features are extracted and analyzed, providing reports/queries for care
providers and allowing remote monitoring. The goal is to help automate monitoring
patientfinger-handfunctionbetweenofficevisitsfromaremotelocation,onasmaller
scaleandwithminimalsupervision. Thecontributionspresentedinthisworkinclude
development of a detailed set of digital features derived from a custom built gaming
platform to highlight client movement performance and algorithms to extract hand
structuretoapproximategoniometrymeasurementsofjointanglesmonitoringforpo-
tentialchangesduringprogressionofthedisease. Thesignificanceofthiscontribution
isthatitprovidesareadilyaccessible,experimentalplatformfortheprovisionofphys-
icaltherapytailoredtotheindividualRApatientthroughtheuseofatelerehabilitation
gamingplatform.
Keywords: Telerehabilitation,rheumatoidarthritis,finger-handfunction,seriousgames,
featurevalues,movementperformanceanalysis,imageanalysis,goniometry.
ii
Acknowledgements
Therearemanypeoplethathavebeenagreathelptomeonthisjourneyandithas
beenmyprivilegetoknowandworkwithallofthem. Aspecialthankstomyadvisor,
Dr. J.F. Peters has been a constant source of encouragement with great vision and
without which, none of this would have been possible. Many thanks to my advisory
committee, Dr. McLeod, Dr. Hossein and Dr. Szturm, I appreciate your suggestions
andalsoyourtimeandeffortinsupportofthisendeavour. Similarly,I’dliketothank
myexternalexaminerDr. Jain, Iappreciateyourtimeandconsiderationofmywork.
Also,IneedtothankCILabfriends,bothpastandpresentforputtingupwithmeand
helping suggest ideas when I back myself into a corner. In addition, I would like to
thank the Canadian Institute of Health Research (CIHR) and the Canadian Arthritis
Network (CAN) for their generous support, without which this also would not have
beenpossible. Finally,ahugethankstomyfamilyandfriendsforputtingupwithme
and also for making do without me during the busier times and to those that did not
makeittotheendwithme,youwillbemissed.
iii
Contents
Abstract ii
Acknowledgements iii
ListofTables vii
ListofFigures viii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Goalsandobjectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Noveltyandcontributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Organizationofdocument . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Background 6
2.1 Telerehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Rheumatoidarthritis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Inflammation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 Goniometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Measuringmovementperformance . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Self-reportedquestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5 Visualanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.1 Imagerepresentation . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.2 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.6 Developingpatterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 SystemArchitecture 39
3.1 Thephysicalsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
iv
3.2 Therehabilitationgamingplatform . . . . . . . . . . . . . . . . . . . . . . 46
3.3 Imageacquisitionforstillhandposes . . . . . . . . . . . . . . . . . . . . 52
3.4 Movementperformancefeatures . . . . . . . . . . . . . . . . . . . . . . . 59
3.5 Computervisionappliedtohandimages . . . . . . . . . . . . . . . . . . . 66
3.5.1 Imagepreparation . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.5.2 Handstructure: Pose1,topview,bothhandsincluded . . . . . . . 72
3.5.3 Handstructure: Pose2,lateralview,dominanthandonly . . . . . . 83
3.5.4 Handstructure: Pose3,topview,dominanthandonly . . . . . . . 88
3.5.5 Jointfeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.6 Contentmanagementsystem . . . . . . . . . . . . . . . . . . . . . . . . . 96
4 SystemVerificationandExperimentalWork 98
4.1 Systemoverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.2 Therehabilitationgamingplatformandoutcomemeasures . . . . . . . . . 100
4.3 Imageacquisitionsetupverification . . . . . . . . . . . . . . . . . . . . . 107
4.4 Jointfeatureverification . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.4.1 Jointswelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.4.2 Jointredness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.4.3 Jointtexture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.5 Handstructure-topview . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.6 Handstructure-lateralview . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.7 Rangeofmotion-topview . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5 ResultsandDiscussion 139
5.1 Visualizingoutcomemeasures . . . . . . . . . . . . . . . . . . . . . . . . 139
5.2 Telerehabilitationgamingresults . . . . . . . . . . . . . . . . . . . . . . . 143
5.2.1 Experiment1-RApatientsn=30 . . . . . . . . . . . . . . . . . . 143
5.2.2 Experiment2-RApatientsn=4 . . . . . . . . . . . . . . . . . . 150
v
5.2.3 Experiment3-normativesubjectsn=10 . . . . . . . . . . . . . . 156
5.3 Furtherdiscussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6 ConclusionsandRecommendations 165
6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.2 Recommendationsandfuturework . . . . . . . . . . . . . . . . . . . . . . 169
A AppendixA-Terminology 172
A AppendixB-TelerehabilitationGamingDataAnalysisManuals 174
A AppendixC-ExampleOutputData 191
References 199
Index 216
vi
List of Tables
1 Adjustablegameparameters . . . . . . . . . . . . . . . . . . . . . . . . . 50
2 Adjustablegameparametersforshapes . . . . . . . . . . . . . . . . . . . 51
3 Newgameparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Movementperformancefeatures . . . . . . . . . . . . . . . . . . . . . . . 66
5 Storageofdigitlengthpercentages . . . . . . . . . . . . . . . . . . . . . . 69
6 Exampledigitlengthpercentages . . . . . . . . . . . . . . . . . . . . . . . 70
7 Angleofseparationbetweenrelaxedmetacarpalbones . . . . . . . . . . . 80
8 Pose3exampleresults . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9 Jointswellingarea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
10 Jointrednesshistogramaveragemeasure . . . . . . . . . . . . . . . . . . . 122
11 Jointrednessnearnessmeasure . . . . . . . . . . . . . . . . . . . . . . . . 122
12 Jointtexture,contrastmeasure . . . . . . . . . . . . . . . . . . . . . . . . 124
13 Jointtexture,contrast,correlation,energyandhomogeneity . . . . . . . . . 126
14 Jointtextureverification,contrast,correlation,energyandhomogeneity . . 126
15 Manuallocationoftipandinflectionpoints . . . . . . . . . . . . . . . . . 128
16 Software-basedlocationoftipandinflectionpoints . . . . . . . . . . . . . 128
17 Fingertip/inflectionpointlocationerror(pixels) . . . . . . . . . . . . . . . 129
18 Jointplacementerror(pixels) . . . . . . . . . . . . . . . . . . . . . . . . . 130
19 Lateralpose,DIPandPIPjointanglemeasurements . . . . . . . . . . . . . 136
20 Rangeofmotionanglemeasurementsviasoftwareandgoniometer . . . . . 138
21 Fivecategoriesofmovementperformancefeatures . . . . . . . . . . . . . 142
22 Rheumatoidarthritissymptoms . . . . . . . . . . . . . . . . . . . . . . . . 143
vii
List of Figures
1 Jointsofinterest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Boutonnieredeformityofindexfinger . . . . . . . . . . . . . . . . . . . . 14
3 Goniometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Examplemovementmeasures . . . . . . . . . . . . . . . . . . . . . . . . 22
5 Phaseplaneplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6 Matrixrepresentationofagreyscaleimage . . . . . . . . . . . . . . . . . . 27
7 Handsinfrontofhighcontrastbackground . . . . . . . . . . . . . . . . . 27
8 Boundary/interiorvisualization . . . . . . . . . . . . . . . . . . . . . . . . 35
9 Systemdiagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
10 Inputcontrolobjectsandexample . . . . . . . . . . . . . . . . . . . . . . 43
11 Viewofthegamescreen . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
12 Painandstiffnesspop-upwindow . . . . . . . . . . . . . . . . . . . . . . 49
13 Handpose1: topview,relaxedposture . . . . . . . . . . . . . . . . . . . . 53
14 Handpose2: sideview,indexfinger,dominanthand . . . . . . . . . . . . 54
15 Handpose3: topview,spreadfingers,dominanthand . . . . . . . . . . . . 55
16 Webcamusedforcapturingstillimageryofhandposes . . . . . . . . . . . 56
17 Sketchofimageacquisitionsetup . . . . . . . . . . . . . . . . . . . . . . 58
18 Crosssectionofexampledatafromagamesession . . . . . . . . . . . . . 60
19 Averagemovementtrajectoryexample . . . . . . . . . . . . . . . . . . . . 63
20 Landmarksdisplayedontherighthand . . . . . . . . . . . . . . . . . . . . 69
21 Exampleresultsfromk-meanssegmentation . . . . . . . . . . . . . . . . . 70
22 Removalofartifactsfromsegmentationviamorphologicalclosing . . . . . 71
23 Morphologicalskeletonizationofapairofhands . . . . . . . . . . . . . . 73
24 Firstskintonepixelinrow1 . . . . . . . . . . . . . . . . . . . . . . . . . 76
25 Re-positioningfingertip . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
26 Customskeletonizationofapairofhands . . . . . . . . . . . . . . . . . . 79
viii
27 Metacarpalbonesandwristhighlightedonhandimage . . . . . . . . . . . 80
28 Stem plot of error measure from linear model and wrist localization data
pointshownonplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
29 Handimagewithcompletestructuralinformationoverlaidontop . . . . . . 83
30 Sideviewhandposedemonstratingincreasedshading . . . . . . . . . . . . 84
31 Sideviewhandposesdemonstratingk-meansclusteringsegmentation . . . 85
32 Handimagewithindexfingerjointsandbonesegmentsoverlaid . . . . . . 87
33 Thirdpose,topview,dominanthand,fingersextendedandabducted . . . . 88
34 Segmentationofthirdhandpose . . . . . . . . . . . . . . . . . . . . . . . 89
35 Landmarksplacedonthirdhandpose . . . . . . . . . . . . . . . . . . . . 90
36 HandgeometryfromthirdposeformeasuringROM . . . . . . . . . . . . . 91
37 Handimagewithjointsub-imagesextracted . . . . . . . . . . . . . . . . . 93
38 Red,greenandbluechannelhistograms . . . . . . . . . . . . . . . . . . . 93
39 Representationofdistance-1neighbourhoodtoestablishtheGLCM . . . . 95
40 Contentmanagementsystemhomepageandqueryform . . . . . . . . . . 98
41 Blockdiagramofverificationformovementperformancefeatures . . . . . 99
42 Blockdiagramofverificationforimagefeatures . . . . . . . . . . . . . . . 100
43 Falsemovementtriggers . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
44 Findingtheendofamovementtrajectory . . . . . . . . . . . . . . . . . . 104
45 Averagemediummovementsfromhealthyindividuals . . . . . . . . . . . 106
46 Examplebackgroundmediums . . . . . . . . . . . . . . . . . . . . . . . . 108
47 Examplebackgroundmediums-2 . . . . . . . . . . . . . . . . . . . . . . 109
48 Shadowsfromfluorescentlighting . . . . . . . . . . . . . . . . . . . . . . 109
49 Threepointlightingsetup . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
50 Demonstratingperspective . . . . . . . . . . . . . . . . . . . . . . . . . . 112
51 DIPandPIPjoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
52 1%increase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
ix
53 5%increase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
54 10%increase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
55 WrinklesinextendedPIPjoint . . . . . . . . . . . . . . . . . . . . . . . . 116
56 Gradientofajointsub-image . . . . . . . . . . . . . . . . . . . . . . . . . 117
57 Centroidofconnectedregions . . . . . . . . . . . . . . . . . . . . . . . . 119
58 Radiuslimitedneighbourhood . . . . . . . . . . . . . . . . . . . . . . . . 119
59 Nearnessofpoints,(cid:15)=0.01,r=16 . . . . . . . . . . . . . . . . . . . . . . 121
60 Nearnessofpoints,(cid:15)=0.05,r=16 . . . . . . . . . . . . . . . . . . . . . . 121
61 Artificiallyaddedredness,35,10and5perpixel . . . . . . . . . . . . . . 123
62 Selectionofregionofinterestvianearnessmeasure . . . . . . . . . . . . . 125
63 Automaticplacementoflandmarksfortopviewhandimage . . . . . . . . 129
64 Completestructurefortopviewhandimage . . . . . . . . . . . . . . . . . 131
65 Errormeasurebetweenalgorithmjointanglesandmanualgoniometry,MCP132
66 Errormeasurebetweenalgorithmjointanglesandmanualgoniometry,PIP . 133
67 Errormeasurebetweenalgorithmjointanglesandmanualgoniometry,DIP 134
68 DIPandPIPjointangle,lateralview . . . . . . . . . . . . . . . . . . . . . 135
69 ThirdlateralposeexaminingflexedDIPandPIPjoints . . . . . . . . . . . 135
70 Errorbetweensoftwareandgoniometermeasurements . . . . . . . . . . . 136
71 TopviewformeasuringROM . . . . . . . . . . . . . . . . . . . . . . . . 138
72 ResultsofsoftwaremeasuringROMangles,overlay . . . . . . . . . . . . . 139
73 Discrepancybetweensoftwareandgoniometermeasurements . . . . . . . 140
74 Movementparsingprocess . . . . . . . . . . . . . . . . . . . . . . . . . . 141
75 Movementdataexhibitinglowermovementefficiency . . . . . . . . . . . . 142
76 PreliminaryresultsfromRApatientsshowspoor/goodperformance . . . . 144
77 Instrumentedinputcontrolobjectsfortherehabilitationgamingplatform . . 146
78 Accuracy measure from RA patients, subject 1 - coffee cup, subject 2 -
wineglass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
x