Using the ARCS Framework to explore motivational factors that may influence rates of offering

222 Pages·2016·3.52 MB·English

by  Anne Carter

Using the ARCS Framework
to explore motivational factors
that may influence
rates of offering adults vaccinations
in primary care

A DISSERTATION
SUBMITTED TO THE FACULTY OF
UNIVERSITY OF MINNESOTA
BY

Wendy Michelle Scheckel

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

Laël Cranmer Gatewood, MS, PhD, FACMI and
Barbara Yawn, MD, MSc, FAAFP

May, 2016
© Wendy Michelle Scheckel, 2016
Acknowledgements

I would like to acknowledge my committee members: Don Connelly, MD, PhD, FACMI,
Lael Gatewood, MS, PhD, FACMI, Sripriya Rajamani, MBBS, PhD, MPH,
Stuart Speedie, PhD, FACMI, and Barbara Yawn, MD, MSc, FAAFP. In particular, I
want to acknowledge Dr. Speedie for being the chair of my committee. I want to
especially thank my co-advisers: Dr. Gatewood and Dr. Yawn. Their mentoring and
assistance throughout the entire process was greatly appreciated.

I would like to sincerely thank Dr. Peter Wollan who acted as my biostatistician
consultant. Dr. Wollan has an amazing ability to explain complex statistical methods in
an easy-to-understand format that helped me tremendously. His knowledge, expertise,
and experience were invaluable and appreciated beyond what words can express. His
technical support was only surpassed by his emotional support and encouragement. I
would not have made it without his help and friendship.

I would also like to acknowledge the leadership, staff, and patients at the healthcare
organization where the study took place. Without the support of the leadership and
participation from the staff and patients, this research would not have been possible.

I would like to thank the many colleagues and staff of mine for their continued support
and encouragement throughout this process. Their concern and kind words were greatly
appreciated.

Lastly, I would like to acknowledge and thank all of my family and friends for their
prayers and support.
iii
Dedication

First of all, this dissertation is dedicated to my Lord and Savior, Jesus Christ.
“I can do all things through him who strengthens me” (Philippians 4:13 ). He is my rock
and strong tower who is merciful, faithful, and true. “Your steadfast love, O LORD,
extends to the heavens, your faithfulness to the clouds” (Psalm 36:5).

Secondly, I dedicate this dissertation to my husband, Matt and son, Christian. Their
continued support and love carried me throughout the entire process. They were patient,
understanding, and encouraging every step of the way. They made many sacrifices
willingly and without complaint. They deserve my utmost respect and gratitude for all of
their help throughout this entire journey. I love them more than words could ever express
and I am eternally blessed to have them in my lives.

Finally, I would like to dedicate this dissertation to my Mom, Nancy. She was my
biggest fan. Cheering me along the entire path, she always had words of encouragement
and reassurance. Her support was invaluable and appreciated more than I could ever let
her know.
iv
Abstract
This research study explored motivational factors of clinical staff in 12 primary care sites
at one integrated, regional healthcare organization that may help explain variation in rates
of offering pneumococcal, influenza, and pertussis vaccinations to adults age 18 and
older using Keller’s Attention, Relevance, Confidence, and Satisfaction (ARCS)
framework. Analysis of the organization’s comprehensive EHR and its use of a statewide
registry was used to calculate eligible encounters and rates of vaccine administration
success to adult patients. The Keller Course Interest Survey originally designed to assess
motivation in students was modified to a clinical care construct and administered to all
primary care staff with patient contact. Separate analyses for influenza, pertussis and
pneumococcal vaccines approximated state and US levels, with large variability between
sites. Motivational scores differed significantly between sites and between clinicians,
rooming staff and receptionists at each site. Rooming staff interviews revealed
variability related to adherence to organizational policy and protocol including a lack of
documentation of patient refusals to receive vaccinations and clinicians functioning
autonomously and independently outside the documented guidelines. Several patient
demographics, site, and the motivational domains of attention and satisfaction were all
found to be statistically significant in a stepwise multiple logistic regression model of
factors contributing to the success of offering influenza, pertussis, and pneumococcal
vaccinations to adults. Rates of offering vaccines to adults can be improved using
motivational design methods to improve and support the motivational domains of
attention and satisfaction by implementing targeted educational strategies and key
electronic health record design changes. The goal of these interventions would be to
more easily identify eligible patients, better understand vaccination guidelines, offer
clinical decision support, and provide valuable feedback in the success of offering
vaccinations to adults.

v
Table of Contents

List of Tables ..................................................................................................................... ix
List of Figures ................................................................................................................... xii
CHAPTER 1 – INTRODUCTION ..................................................................................... 1
1.1 – STATEMENT OF THE PROBLEM ..................................................................... 2
1.2 – BACKGROUND ................................................................................................... 3
1.3 – RATIONALE OF THE STUDY AND ITS METHODS ...................................... 5
1.4 – STATEMENT OF SPECIFIC AIMS..................................................................... 5
1.5 – LAYOUT OF THESIS .......................................................................................... 6
CHAPTER 2 – LITERATURE REVIEW .......................................................................... 8
2.1 – HEALTH INFORMATION TECHNOLOGY ...................................................... 8
2.2 – PREVENTIVE SERVICES ................................................................................. 10
2.3 – TRANSLATIONAL RESEARCH ...................................................................... 12
2.4 – GUIDELINES IN EVIDENCE-BASED PRACTICES....................................... 14
2.5 – CLINICIAN ADHERENCE TO GUIDELINES ................................................. 16
2.6 – THEORETICAL FRAMEWORKS OF BEHAVIOR ......................................... 18
2.7 – THE ARCS MODEL OVERVIEW ..................................................................... 21
2.7.1 – ARCS – Attention ......................................................................................... 22
2.7.2 – ARCS – Relevance ....................................................................................... 23
2.7.3 – ARCS – Confidence ...................................................................................... 23
2.7.4 – ARCS – Satisfaction ..................................................................................... 24
2.7.5 – ARCS Instrumentation – Course Interest Survey (CIS) ............................... 24
2.8 – SURVEY METHODS ......................................................................................... 26
2.9 – INTERVIEW METHODS ................................................................................... 28
CHAPTER 3 – RESEARCH METHODS ........................................................................ 32
3.1 – RESEARCH DESIGN ......................................................................................... 32
3.1.1 – Setting ........................................................................................................... 33
3.1.2 – Participants .................................................................................................... 34
3.1.3 – Institutional Review Board and State Law ................................................... 35
3.2 –COMPARING RATES USING THE EHR METHODS...................................... 35
3.2.1 – Data Collection – Query Development ......................................................... 35
3.2.2 – Rate Calculation ............................................................................................ 37
3.2.3 – Data Analysis and Validation Methods ........................................................ 42
3.3 – MOTIVATION SURVEY METHODS ............................................................... 44
3.3.1 – Motivational Model ...................................................................................... 45
3.3.2 – Motivational Survey Tool ............................................................................. 45
3.3.3 – Motivational Survey Distribution ................................................................. 48
3.3.4 – Motivational Survey Pilot ............................................................................. 51
3.3.5 – Motivational Survey Tracking ...................................................................... 52
3.3.6 – Motivational Survey Scoring Methods ......................................................... 53
3.4 – STAFF INTERVIEW METHODS ...................................................................... 54
3.4.1 – Interview Participants ................................................................................... 54
3.4.2 – Conducting the Interview .............................................................................. 55
vi
3.4.3 – Documenting the Interview........................................................................... 57
3.4.4 – Systematic Interview Analysis ...................................................................... 58
3.4.5 – Interview Pilot ............................................................................................... 59
3.5 – MODEL DEVELOPMENT METHODS ............................................................ 60
CHAPTER 4 – RESULTS ................................................................................................ 62
4.1 – SUBJECT/SITE CHARACTERISTICS .............................................................. 62
4.1.1 – Patient Encounters ........................................................................................ 62
4.1.2 – Clinic Sites – Size and Geography................................................................ 68
4.1.3 – Clinic Sites – Patient Demographic Characteristics ..................................... 70
4.1.4 – Clinic Sites – Staff ........................................................................................ 74
4.2 – FINDINGS FOR AIM 1: COMPARING RATES USING THE EHR................ 76
4.2.1 – Comparing Rates of Offering Influenza Vaccination ................................... 76
4.2.2 – Comparing Rates of Offering Pertussis Vaccination .................................... 84
4.2.3 – Comparing Rates of Offering Pneumococcal Vaccination ........................... 91
4.3 –MOTIVATION SURVEY FINDINGS ................................................................ 95
4.3.1 – Survey Adaptation ........................................................................................ 95
4.3.2 – Pilot Study to Test Clinical Care Construct Survey...................................... 99
4.3.3 – Response Rates ........................................................................................... 102
4.3.4 – Missing Data ............................................................................................... 103
4.3.5 – ARCS Scores .............................................................................................. 105
4.4 – STAFF INTERVIEW FINDINGS ..................................................................... 110
4.4.1 – Organizational Policy.................................................................................. 110
4.4.2 – Response ..................................................................................................... 111
4.4.3 – Summary Analysis ...................................................................................... 114
4.5 – FINDINGS FOR AIM 4: MOTIVATION AS A PREDICTOR FOR
VACCINATION SUCCESS ...................................................................................... 116
4.5.1 – Influenza ..................................................................................................... 116
4.5.2 – Pertussis ...................................................................................................... 123
4.5.3 – Pneumococcal ............................................................................................. 128
4.6 – SUMMARY ....................................................................................................... 133
CHAPTER 5 – DISCUSSION ........................................................................................ 134
5.1 – SUBJECT/SITE CHARACTERISTICS ............................................................ 134
5.1.1 – Patient Encounters ...................................................................................... 134
5.1.2 – Clinic Sites .................................................................................................. 135
5.2 – AIM 1: COMPARING RATES USING THE EHR .......................................... 137
5.2.1 – Comparing Rates of Offering Influenza Vaccination ................................. 138
5.2.2 – Comparing Rates of Offering Pertussis Vaccination .................................. 140
5.2.3 – Comparing Rates of Offering Pneumococcal Vaccination ......................... 141
5.3 – AIM 2: MOTIVATION SURVEY .................................................................... 144
5.3.1 – Survey Adaptation ...................................................................................... 144
5.3.2 – Response Rates ........................................................................................... 146
5.3.3 – Missing Data ............................................................................................... 146
5.3.4 – ARCS Scores .............................................................................................. 147
5.4 – AIM 3: INTERVIEWS ...................................................................................... 149
vii
5.5 – AIM 4: MOTIVATION AS A PREDICTOR FOR VACCINATION SUCCESS
..................................................................................................................................... 152
5.5.1 – Influenza ..................................................................................................... 152
5.5.2 – Pertussis ...................................................................................................... 155
5.5.3 – Pneumococcal ............................................................................................. 156
5.4.4 – Aims 1-4: Integration of Results across Vaccinations ................................ 158
5.6 – RECOMMENDATIONS ................................................................................... 161
5.7 – COMPARISON WITH WORK OF OTHERS .................................................. 163
5.8 – STRENGTHS .................................................................................................... 166
5.9 – LIMITATIONS AND ASSUMPTIONS ........................................................... 168
5.10 – CONTRIBUTIONS ......................................................................................... 171
5.11 – SUGGESTIONS FOR FUTURE RESEARCH ............................................... 174
CHAPTER 6 – CONCLUSIONS ................................................................................... 177
Bibliography ....................................................................................................................181
Appendix ..........................................................................................................................186
viii
List of Tables

Table 1. Course Interest Survey Internal Consistency Estimates (4)................................ 26
Table 2. At-Risk Conditions for the Pneumococcal Vaccination and the Corresponding
ICD-9 Diagnosis Codes to Determine Eligibility for Vaccination (5) ............................. 40
Table 3. Statements from the Original Course Interest Survey Developed by Keller (4) 47
Table 4. Number of Encounters with the Number of Eligible Vaccinations .................... 63
Table 5. Number of Encounters and Individual Patients by Sex ...................................... 64
Table 6. Original Race Categories and Number of Corresponding Encounters; Grouping
of Original Race into Broader Race Categories and Number of Corresponding Encounters
........................................................................................................................................... 65
Table 7. Number of Encounters and Individual Patients by Ethnicity ............................. 66
Table 8. Number of Encounters and Individual Patients by Language ............................ 66
Table 9. Number of Encounters and Individual Patients by Insurance Status .................. 67
Table 10. Site Groupings by Geographical Location and Size and Total Number of
Encounters......................................................................................................................... 69
Table 11. Number of Staff by Job Category by Site ......................................................... 75
Table 12. Percent of Offering Influenza Vaccination Success by Site ............................. 77
Table 13. Univariate Analysis of Demographic Factors to Determine Contribution to the
Success of Offering Influenza Vaccination ...................................................................... 78
Table 14. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Influenza Vaccination Using Multiple Analysis ................................................ 78
Table 15. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Influenza Vaccination Using Multiple Analysis Removing Insignificant Factors
(Ethnicity) ......................................................................................................................... 79
Table 16. Final Multiple Logistic Regression Model of Offering Influenza Vaccination
Success .............................................................................................................................. 83
Table 17. Percent of Offering Pertussis Vaccination Success by Site .............................. 84
Table 18. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Pertussis Vaccination Using Univariate Analysis .............................................. 85
Table 19. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Pertussis Vaccination Using Multiple Analysis ................................................. 85
Table 20. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Pertussis Vaccination Using Multiple Analysis Removing Insignificant Factors
(Ethnicity and Language) .................................................................................................. 86
Table 21. Final Multiple Logistic Regression Model of Offering Pertussis Vaccination
Success .............................................................................................................................. 90
Table 22. Percent of Offering Pneumococcal Vaccination Success by Site ..................... 91
Table 23. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Pneumococcal Vaccination Using Univariate Analysis ..................................... 92
Table 24. Patient Demographics Analyzed to Determine Contribution to the Success of
Offering Pneumococcal Vaccination Using Multiple Analysis ........................................ 92
ix
Table 25. Final Multiple Logistic Regression Model of Offering Pneumococcal
Vaccination Success.......................................................................................................... 94
Table 26. Clinical Care Construct Survey (Questions 1-16) ............................................ 97
Table 27. Adjustments Made to the CCCS Based on Comments During the Pilot Phase
......................................................................................................................................... 101
Table 28. Survey Rate of Return by Site and Job Category ........................................... 102
Table 29. Unanswered Questions by ARCS Domain and Job Category per Returned
Survey ............................................................................................................................. 104
Table 30. Univariate Analysis of ARCS Domain Scores and Successful Offering of
Influenza Vaccination ..................................................................................................... 117
Table 31. Domain Score and Job Category as Contributing Factor to the Success of
Offering Influenza Vaccination using Univariate Analysis ............................................ 118
Table 32. Multiple Logistic Regression of ARCS Domain Scores and Successful Offering
of Influenza Vaccination for All Staff ............................................................................ 119
Table 33. Logistic Regression of ARCS Domain Scores and Successful Offering of
Influenza Vaccination for Each Job Category ................................................................ 120
Table 34. Logistic Regression of ARCS Domain Scores and Successful Offering of
Influenza Vaccination Adjusting for Significant Patient Demographic Characteristics 121
Table 35. Logistic Regression of ARCS Domain Scores and Successful Offering of
Influenza Vaccination Adjusting for Significant Patient Demographic Characteristics 122
Table 36. Univariate Analysis of ARCS Domain Scores and Successful Offering of
Pertussis Vaccination ...................................................................................................... 123
Table 37. Univariate Analysis of ARCS Domain Scores and Successful Offering of
Pertussis Vaccination ...................................................................................................... 124
Table 38. Multiple Logistic Regression of ARCS Domain Scores and Successful Offering
of Pertussis Vaccination for All Staff ............................................................................. 125
Table 39. Logistic Regression of ARCS Domain Scores and Successful Offering of
Pertussis Vaccination for Each Job Category ................................................................. 126
Table 40. Logistic Regression of ARCS Domain Scores and Successful Offering of
Pertussis Vaccination Adjusting for Significant Patient Demographic Characteristics . 127
Table 41. Univariate Analysis of ARCS Domain Scores and Successful Offering of
Pneumococcal Vaccination ............................................................................................. 128
Table 42. Univariate Analysis of ARCS Domain Scores and Successful Offering of
Pertussis Vaccination ...................................................................................................... 129
Table 43. Multiple Logistic Regression of ARCS Domain Scores and Successful Offering
of Pneumococcal Vaccination for All Staff .................................................................... 130
Table 44. Logistic Regression of ARCS Domain Scores and Successful Offering of
Pneumococcal Vaccination for Each Job Category ........................................................ 131
Table 45. Logistic Regression of ARCS Domain Scores and Successful Offering of
Pneumococcal Vaccination Adjusting for Significant Patient Demographic
Characteristics ................................................................................................................. 132
Table 46. Sites by the Number of Encounters with the Encounter to Clinician Ratio ... 136
Table 47. Ratio of Rooming Staff and Receptionists to Clinician by Site ..................... 137
Table 48. Average Domain Score (All Job Categories) Ranking by Site ....................... 148
x