ERIC ED381481: Demistifying Research in Education and the Social Sciences. A Primer for LRE Folks.

10 Pages·1993·0.38 MB·English

by  ERIC

DOCUMENT RESUME
SO 024 945
ED 381 481
McBee, Robin Haskell
AUTHOR
Demistifying Research in Education and the Social
TITLE
Sciences. A Primer for LRE Folks.
PUB DATE
[93]
NOTE
10p.
Evaluative/Feasibility (142)
Reports
PUB TYPE
MFO1 /PCO1 Plus Postage.
EDRS PRICE
Citizenship Education; *Educational Research; Higher
DESCRIPTORS
Education; *Law Related Education; *Qualitative
Research; Social Studies; *Statistical Analysis
ABSTRACT
The article argues that the law-related educational
better
(LRE) community should understand education research methods
the value of LRE
to demonstrate to policy makers and funding sources
To facilitate the
and to evaluate alternative educational strategies.
LRE
reading and conducting of research, the article explains for
practitioners research procedures and the format in which research
described as a
results are presented. The scientific method is
systematically and objectively
process of forming a hypothesis and
experimentation. The
testing the hypothesis through observation and
qualitative
article explains the differences between quantitative and
basic and
research, experimental and non-experimental research,
sampling,
applied research, and the problems of subject selection,
(JD)
reliability, and validity. Contains seven references.
***********************************************************************
Reproductions supplied by EDRS are the best that can be made
from the original document.
***************************************************************4*******
Demystifying Research in Education and the Social Sciences
A Primer for LRE Folks
by
Robin Haskell McBee
U.S. DEPARTMENT Of EDUCATION
Office ol Educatonal Remarth and Improvement
EDUCATIONAL RESOURCES INFORMATION
CENTER (ERIC)
yiThis document has been reproduced as
, Pectived from the person or Organization
originating rt
'
O Minor changes have been made lo improve
reproduction quality
Points of view or opinions slated in this dOCu
ment do not necessarily represent official
OERI position or policy
PERMISSION TO REPRODUCE THIS
MATERIAL HAS BEEN GRANTED BY
C
TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
DEMYSTIFYING RESEARCH IN EDUCATION AND THE SOCIAL SCIENCES
A Primer for LRE Folks
by Robin Haskell McBee
Virginia Institute for Law and Citizenship Studies
Virginia Commonwealth University
if we don't understand the
informed consumers
Why Bother?
language of the
becomes
reports?
It
research
Most of us are involved in the field of law-related
imperative, therefore, that we at least learn how to
education because we intuitively know that it works.
understand the talk, even if we never talk it ourselves.
Our own personal experiences with the content and
Research in education, psychology, sociology, and
strategies, our informal observations of its "light bulb"
follows the examples and
social sciences
other
impact on teachers and kids, and our anecdotal
standards set in the more rigorous natural sciences.
inventory of success stories all tell us that it works.
However, the language of this research need not leave
Unfortunately, not everyone gets our front-row seat in
us fit to be tied. There are some basics which we can
the LRE theater; so not everyone benefits from the full
acquire rather quickly, and that is the intent of this
picture we get. Further, the world is full of doubters -
paper, though this will by no means make us a group
people who simply won't take our word for
it.
If you want to learn more, you might try
of experts.
Actually, they really shouldn't have to take our word
taking a course at a local university. All of the social
if it works, we should be able to prove it.
for it;
sciences have graduate level courses designed to teach
Whenever people in other fields make a claim that
the novice how to understand and interpret research in
something works, the rest of us ask for proof - be it
that field, although sometimes the textbooks are just as
medicine, business, sociology, education, whatever.
dense and difficult to read and understand as the
Even in law we have our own system of gathering
research, itself (another reason why I decided to try my
factual data in order to prove that something is so
hand at this paper).
And it
is painfully
"beyond a reasonable doubt".
The Reason for
evident to many of us that the educational decision
Scientific Research
makers and power mongers require some substantiation
Scientific research as we know it today has its
before they adopt (AND FUND) a new program or
rational
roots in the growth of rationality (rational law,
It is, therefore, important for us to begin
approach.
religion, rational economics, etc.), intellectualism, and
acquiring some of the rudimentary skills for generating
As
cultural stability in the medieval western world.
such proof.
toleration for scientific inquiry grew and scientists
One of the "skills" for proving your assertions is
began to seek answers to questions about human
being able to understand and use others' research to
existence, they also sought cold, hard, observable
bolster your own claims or to serve as a starting point
In order to prove something, one had to be
proof.
is a
for your own investigations. In education, there
carefully eliminating one
very logical and methodical -
whole "club" of individuals, known as educational
Slowly, the process of
explanation after another.
and
researchers, who make their livings generating
scientific inquiry (scientific method) evolved into a
educational
reporting on data which substantiate one
hypothetical
with
a
of
beginning
steps,
series
They have procedures and a
claim or another.
by
explanation for a question of importance, followed
field and
language that are somewhat unique to their
systematically and objectively testing that hypothesis,
difficult - if not impossible - for the non-researcher or
and characterized by the use of observation
and
Such researchers exist in
lay person to under Land.
experimentation which could be verified.
further and
other fields, as well, and as we move
describes it,
As sociologist Daniel Chirot (1986)
we
arena,
prevention
the
viole_ice
into
further
the "crowning achievement" of western
science
is
their
with
face-to-face
ourselves
find
repeatedly
rationality.
Even if we never
research reports and claims, too.
It is based on calculability, on proof,
'mow which
generate our own research, how will we
and on empirical- observation in a way
which
of those claims are legitimate and which are not,
that no economy, legal or political
discard if
to recommend to others and which ones to
structure, or religious ethic can be. It
idea what those reports say
we haven't got the foggiest
the domain of
par excellence,
is,
least be
and how to interpret them? How can we at
3
The body of the actual report usually begins with
t
It has also
highly trained specialists.
an introduction, which offers a context for the research
taken on a life of its own... (p. 49)
and problem to be studied and focuses on the reason
This verifiable proof, based on objective observation,
.
for and significance of the study to be taken. Usually,
is the essence of rational thinking and lies at the
research question
in
the
posed
is
the
specific
foundation of research in both the natural and social
it has
or
paragraphs
well.
introductory paragraph
Over the decades and centuries,
as
sciences.
Following the introduction is a review of literature,
become accepted procedure to go through certain steps
which can be anywhere from a few lines to several
It is these
for proving the validity of your research.
paragraphs. In it, the researchers outline what kind of
steps and their explanation which seem so difficult to
related research has already been done by them and
understand to someone unfamiliar with such scientif.c
The review focuses specifically on related
others.
procedures and reports.
questions that have been "answered" and on holes in
The scientific method and experimenter objecti. ty
that research which lead to the question for this study.
the
are key to generating proof that is acceptable to
Following the literature review, the report offers a
larger world. A successful experiment or study is very
specific hypothesis, followed by the description of the
specific and limited enough in scope so that the
method or design of the study. Often set aside in its
researcher can be confident that the results obtained are
own section, the research design specifically describes
logically connected to or caused by the hypothetical
who or what was studied and how they or it was
explanation that has been offered. The method used to
studied. This includes describing who the subjects of
test the hypothesis must have a design which eliminates
the study were and how they were obtained, what was
rival
explanations (known as
possible other
all
done to the subjects (procedures), and what instruments
There are, therefore, a
explanations) for the results.
etc.) were used to
observations,
surveys,
(tests,
variety of procedures used in selecting who or what
This section
measure the impact of the procedures.
will be studied, how they or it will be studied, and
also describes the statistical test(s) that was(were) used
how the results or data will be analyzed (usually
to analyze the results.
These
tests).
statistical
sophisticated
involves
The method section is followed by a report of the
procedures and their explanations have become the
results. Commonly, tables and graphs are included in
language of social science research, and the remainder
In the final two sections of the research
this section.
of this paper will seek to define and explain some of
report, the author offers a discussion or analysis of the
the basics of that language.
The
results and conclusions based on the analysis.
The Scientific Process
discussion section explains the results and interprets
inquiry process begins with a
The scientific
them in light of other research and any weaknesses in
question, problem, or idea which is clearly stated. The
the design (where there are possible rival explanations
question is related to already established conclusions or
The
for the results other than the procedure, itself).
knowledge in the field. In other words, somebody else
conclusion summarizes the answer to the research
has already proven this, but no one has proven that.
question based on inferences made from the results,
A hypothesis is then offered as a possible explanation.
weaknesses in the design, and the relationship of this
That hypothesis must be credible to others at the
study to others. The researcher also often recommends
Following the generation of a hypothesis, an
outset.
results of the
areas for further research based on the
experiment or study is designed and conducted. The
study.
design must be careful not to leave holes in the process
Types of Research
which allow for rival explanations. After the study is
If
research:
types
general
two
are
There
conducted, the results are collected and analyzed and
Quantitative research
quantitative and qualitative.
conclusions are drawn. The analysis and conclusions
It is more numbers-
comes from the hard sciences.
world as
must also be logical and credible to the larger
oriented, requires complete objectivity on the part of
well as to the researcher.
the researcher, and calls for analyzing data using
The Parts of a Resecrch Report
deductive and statistical methods. Qualitative research
The typical research report in the social sciences
is not tied to numbers at all but rather to observations
follows a similar pattern. After the title and author(s)
and descriptions of one or more subjects in natural
is(are) given, often an abstract of the study is offered.
settings. The researcher often becomes more involved
the
In most reports this is a paragraph in small print at
subjects and analyzes
inductively,
data
with the
beginning of the report which summarizes the research
In LRE, a typical
stressing themes and trends.
similar
question, design, results, and conclusions. It is
quantitative study might be based on the institute
for larger grant
to the abstract you are asked to write
evaluations teachers fill out; whereas, a qualitative
proposals.
4
3
the attitudes and behaviors of a group that works with
study might focus on extended observations of an
ORPs and one which does not. Another LRE example
exemplary mock trial coach in order to determine what
of applied research might be testing the impact of
it is that mock trial coaches do.
student courts as a behavior management technique.
Research can also be divided into experimental and
Action research is a type of applied research which
non-experimental categories. In experimental research,
is used for a specific classroom setting or educational
the experimenters are able to control or manipulate
decision and is usually limited in scope. You might
They are
certain factors which affect the subjects.
want to know, for instance, whether or not doing a full
studying cause and effect or causal relationships
group debriefing session on a daily basis at your
between different factors. In non-experimental studies,
summer institute is more effective than small group
researchers are describing or analyzing information and
debriefing, and you might conduct action research
cannot manipulate or control the factors being studied.
specific to your own situation in order to investigate
They are seeking either to define or describe simple
In another example of action research,
this question.
information or relationships (known as correlations)
you might question the ability of a high tech interactive
which tie two factors together or to analyze them (as in
computer program to promote greater understanding of
historical and legal analysis). An LRE experimental
the law by having one of your classes use Tom
study might analyze the impact of the case study
Snyder's Decisions, Decisions: Prejudice to prepare an
method on students' ability to understand the evolution
essay on conflicting rights and another class use your
of civil rights in the United States by using the case
lawyer partner's presentation and a case study to do the
study method, lecture, and text reading with one class
same.
and only lecture and text reading with another class.
On the other hand, if you were trying to decide
In such an example, the instructional strategies are the
whether or not to do a full-scale, system-wide or state-
factors or treatments that are being controlled and
for
Civic
of
Center
the
implementation
wide
manipulated. Surveys of LRE prevalence in classroom
Education's Violence in Our Schools program, then
teaching would be an example of non-experimental
you might test it out on a selected group of classes to
Here we are finding out what
research in our field.
start and make your decision based on the results of
people are doing, but we are not changing any factors
that study. This is a form of evaluation, which seeks
that influence what it is that they are doing.
to make larger, more wide scale decisions about the
from time to time, run into
You may also,
effectiveness of a past or potential program.
In
references to action or evaluation research. These are
another example, you might analyze the effect of your
refer
two of four general descriptive categories which
sufficient
providing
summer LRE
in
institutes
research.
to the reason or purpose for conducting the
information and demonstrations .o make your teacher
are basic and applied
The other two categories
the
teaching
about
comfortable
participants
feel
research.
content, or you might, several months later, survey the
Basic research is intended to develop theories or to
that
same teachers to determine what content from
Since this type of
understand and explain phenomena.
Finally, you might
institute has actually been taught.
inquiry usually lies at the foundation of the learning,
what
test the teacher participants' students to determine
prevention
violence
and
prevention,
delinquency
has been learned as a result of teaching that same
theories with which we are more familiar, it normally
content.
takes place in contrived settings such as laboratories,
Variables
and it is not commonly found in the educational arena.
Within educational and social science research
Examining how one builds a cognitive framework for
there are frequent references to several kinds of
problem solving, or conditions that promote or detract
variables which further describe the study being
from memory building, or the age at which children
As the common usage of this word
undertaken.
point of view are all
are capable of taking another's
implies, variables are aspects of the study whose
examples of basic research.
values, degrees, or categories can or do change.
Applied research is geared toward the practical
Gender, age, socioeconomic status, attitude, behavior,
application of theories and ideas, and it normally takes
teaching method, and achievement are all examples of
This is much more
place in the natural setting.
Commonly in research studies there are
variables.
In law-related
common in the education arena.
references to independent and dependent variables.
education, one might examine the application of
or
controlled
usually
-
variables
Independent
bonding theory in delinquency prevention and the use
manipulated by the researcher - influence, predict, or
of outside legal resource people (ORPs) by comparing
4
influence the results and can vary from one group to
cause dependent variables to change in some way.
another, the way in which they are selected is a critical
Most quantitative research is trying to determine how
issue in applying the study results to other conditions
one (or more) thing(s) affects another one (or more)
and situations.
thing(s). The "things" are the variables, and those that
Randomly selected samples are selected in such a
cause the effect are independent while those that are
way that every member of the larger population has the
affected are dependent.
There are several means
same chance to be selected.
If we were studying the effect of participating in
of accomplishing this. With systematic sampling every
mock trials on knowledge of the criminal justice
nth member is selected. Stratified sampling divides the
system, we might deliver similar units on the criminal
larger population into predetermined groups (e.g. new
justice system to twelve classes, with six of the classes
African-American
and
and experienced
or treatment groups including a mock trial as a
teachers,
European-American, or low, middle, and high SES)
culminating experience in the unit and six classes
and randomly selects equal numbers of subjects from
serving as the control groups. We would then look at
each group. In cluster sampling, subjects are randomly
the posttest results to see if there was a difference
selected in equal numbers from naturally occurring
In this
between the treatment and control groups.
groups, such as school systems in the northern section
example, the independent variable is an instructional
of the state or groups in certain neighborhoods of the
mock trials, and the dependent variable
strategy
city.
would be knowledge of the criminal justice system.
If subjects are selected from a population for a
or
intervening,
extraneous,
other
Sometimes
particular reason (they are conveniently available; they
confounding variables enter into the picture and muddy
volunteered; they are representative of a particular
These variables are not controlled by the
it up.
problem to be studied), then they are not randomly
researcher or accounted for in the research design, and
selected and, consequently, do not accurately represent
whether or not the
to
questions
raise
as
they
the population. Therefore, you cannot generalize the
the
in
change
the
caused
variable
independent
study's results to the population from which that
dependent variable or the change was caused by one of
Further, if subjects volunteer to take
sample came.
these other variables. Examples of such intrusions on
part in a study, they may bring particular biases to that
include academic
our study of mock trials might
study which could further cloud the legitimacy of the
ability, socioeconomic status (SES), and age or grade
For example, if you conducted a survey on
results.
If levels of these variables are not consistent
level.
disciplinary practices of all the teachers in the school
it might be one of
across treatment and control groups,
system, and the responses were completely voluntary,
which
these variables - rather than the mock trials
those teachers who are either very pleased with the
measured on the
causes the difference in knowledge
school system's and their own practices or very
of
posttest. A teacher's prolonged absence, the arrest
displeased with them would be more likely
to
the uneven use of
a student in one of the classes, or
Therefore, the responses would
voluntarily respond.
other interactive strategies (some use them, some do
likely be heavily weighted in one of those two
not) might also confound the results.
directions. Consequently, your picture of discipline in
Common Issues in Research
the system's schools would leave out significant
It is not possible in the limited scope of this paper
portions of individuals somewhere in the middle.
possible
(or of my own knowledge) to cover all of the
Reliability
threats or compromises to the validity of a particular
Within a particular study or evaluation, specific
Indeed, some of these threats
piece of research.
instruments (tests, surveys, observations, etc.) are used
involve a sophisticated understanding of educational or
the
to measure information or collect data related to
However, there are some
other types of research.
research question. It is assumed that no instrument can
fundamental issues or concerns with respect to research
perfectly indicate that which is being measured. There
that might help you better understand, analyze, and use
is always some degree of error associated with the
research to your benefit.
Responses may change over time or
measurement.
Subject Selection and Sampling
show a lack of stability (that is, subjects who are not
As earlier indicated, subjects are the people or
the same
part of any study perform differently on
larger population to be
groups who are selected from a
different times). They
measure when it is given at two
studied or to participate in the study. They form the
depending on the
may vary by or lack equivalence
called
sample, and the process of selecting them is
of 4th graders
group being questioned (e.g. one group
Since backgrounds and characteristics can
sampling.
C
5
avoid possible
inferences and
threats
gives a different set of responses than another similar
to
the
to
conclusions that are drawn?
Has the researcher
Responses may also lack
group of 4th graders).
internal consistency or differ within the test
accounted for other legitimate explanations for the
itself
results if they exist?
depending on how the question is asked (e.g. subjects
One type of internal validity is instrument validity,
strongly agree that police are helpful in one part of the
which is not the same as reliability.
Instead,
instrument and disagree with a similar statement in
it
instrument
degree which
to
the the
examines
another part).
legitimately measures what we say
measures.
it
Since these changes or errors in measurement can
Instrument validity can be established in one of three
have an impact on inferences and conclusions that are
We can compare our instruments to other
drawn, researchers must strive to keep such errors to
ways.
similar, but already established measures (criterion-
a minimum. Reliability, therefore, refers to the degree
If, for example, there
It is tested by
related evidence for validity).
is error free.
to which the mean
already exists an academic test of knowledge on the
comparing scores a) on the same measure taken by the
criminal justice system whose reliability is proven, then
same group at two different times (stability); b) on the
we might administer that test and our own test to a
same measure taken by two different but similar groups
group of students similar to those who'll participate in
(equivalence); or c) scores on responses to similar
the study and compare the results. A second way of
questions in different parts of the same instrument
establishing instrument validity is by having outside
Where the instrument is an
(internal consistency).
content of the
experts judge the
instrument
to
observation scale, the reliability is tested by comparing
determine whether its content truly represents the
observation or rating scores for the same subject(s) but
larger domain which is being considered. Known as
given by different observers (interrater reliability).
this type of
content-related evidence for validity,
Scores on tests of reliability are compared by
evidence might be exemplified in the LRE field by
computing a correlation coefficient which can range
having several criminal justice experts review our test
between .00 to .99. With correlations, we are trying
for the mock trial study to see if it is comprehensive
to determine how closely one score is correlated to
enough. The third means of demonstrating instrument
another. The closer to 1.00 the coefficient is, the more
validity is called construct-related evidence. This type
strongly correlated the scores and, therefore, the more
of evidence is used with instruments that measure
correlation
general,
In
instrument.
the
reliable
difficult to observe variables, such as intelligence,
coefficients above .60 are considered acceptable and
In this validation
or self-confidence.
creativity,
When reading a formal research
above .75 good.
process, the researcher seeks to tie together certain
report, you should look for information indicating the
attitude
and
such
as
constructs,
psychological
If it is not offered, this
reliability of the instrument.
motivation or self-concept and hostile interpretation of
should raise questions for you as to whether the
others' actions. Whichever form of evidence is used
instrument used to measure change is able to do so
to demonstrate the validity of our measures - whether
accurately.
it's proven instruments, judges, or constructs, it is
Validity
important to do so in a context similar to that of the
Outside of research, when we speak of something
study (e.g. similar age group, gender mix, academic
being valid, we normally mean that it is solid, well-
ability, etc.).
founded, or persuasive. We might even be referring to
Internal validity relates to the experimental design,
However, in research arenas,
it having been tested.
How strong is the design in overcoming a
itself.
validity has much more specific meanings. In general,
variety of challenges to the inferences and conclusions
and
appropriateness
the
refers
to
validity
Does the design eliminate the
that are drawn?
meaningfulness of inferences that are made both within
possibility of alternative explanations for our results?
the study and in applying the study's results to larger
We've already seen that the instrument and extraneous
populations. We must ask ourselves, does the study
variables can be threats to the study's validity. Other
have internal validity, or are the inferences we make
typical threats to internal validity include significant
Does the test we're
from the test data appropriate?
differences in the subjects assigned to different groups
using in this study really measure understanding of the
(known as subject selection); bias on the part of the
that
court process, for example, or how do we know
subjects; maturation of the subjects during the study
this new personality inventory really does measure
(that is, the fact that they have aged or that significant
what we have defined as attitude toward the law?
Is
time has passed may strongly influence the results);
the experimental design constructed in such a way as
4
6
experimental design can legitimately lead others to
loss of subjects during the study (known as subject
conclude that alternative explanations have not been
attrition or mortality); subjects changing just because
not critical for you to
they are a part of the study (they feel
While it
special,
eliminated.
is
understand all of these types of proced ures (and it is
important, left out, don't want to let the experimenter
down, etc.) or changing because of what they have
far beyond my capability to explain them), it is helpful
learned or heard from other subjects (known as
to at least be able to recognize these terms and the fact
that they are statistical tests when you read research
diffusion of treatment); and subjects changing because
Typical statistical procedures you will see
reports.
they have taken a pretest (from which they may have
in research reports include chi-square,
described
learned something new or been alerted to certain types
analysis of variance or
regression analysis,
t-test,
of information). Also, what the experimenter says or
of
variance
(ANOVA,
analysis
multivariate
does may affect subject responses; the experimenter
MANOVA), and analysis of covariance or multivariate
may have a bias which influences observations or
analysis of covariance (ANCOVA, MANCOVA).
interpretations; or the number of times the treatment is
It is also helpful to recognize that all statistical tests
tested on different subject or subject groups may be
are built on certain assumptions which, if not met in
insufficient to draw conclusions other than that the
the experiment's design, also threaten the validity of
result,: happened by chance (known as a treatment
Statistical procedures
conclusions.
statistical
the
replications threat).
assume that the sample which is studied is drawn from
When it comes time to analyze the results of a
In other
a population which is normally distributed.
quantitative study, the research could suffer from yet
words, the degree to which a particular characteristic
another type of validity issue - statistical conclusion
appears in a particular population will be consistently
In quantitative research, standard statistical
validity.
distributed along a normal or bell curve. (This means
tests are used to analyze the statistical significance of
that most of the population displays the characteristic
(It is important to differentiate between
the results.
to an average degree and is clustered around the
statistical significance and real-world significance; a
middle "hill" portion of the curve, while much smaller
study's results may be statistically significant but
portions of the population display the characteristic to
useless in the real world.) A lay person might ask why
a much lesser degree or to a much higher degree as
just look
we bother with the statistics at all? Why not
represented by the two "valleys" or tails of the curve.)
at the results and, if one is more than the other, then
The larger the population from which the subjects are
referred to as descriptive
just say so?
(This
is
drawn, the more likely it is that this is a normally
statistics.) However, there is an important justification
distributed population.
for using statistical analysis. Such an analysis enables
Another assumption behind statistical tests is that
us to infer that our results have meaning beyond our
subjects have been randomly selected and randomly
Otherwise, we would not know
specific study.
The randomness is critical
assigned to test groups.
whether or not our results are just a fluke or a matter
because it assures that we have variability or that any
of chance. Further, no one is likely to care about our
variations that typically appear in the population also
results unless we can somehow demonstrate that these
appear in the sample. Because of the randomness, it is
results are likely to happen again and again with
also assumed that the level that any characteristic
Here, then,
groups similar to those who were studied
varies in a group will be similar across all groups in
is where the field of inferential statistics - which is
the experiment.
Using
based on the laws of probability - comes in.
Finally, it is important for the number of subjects
various statistical tests we can determine how likely it
levels in
or subject groups not to go below critical
is that our study's results will be true for the larger
order for the conclusions to be valid. If a whole group
population from which our sample came. We can also
of people, like a class or a reading group, receives the
determine the degree to which we can be certain that
treatment at one time, this is considered one subject
this conclusion is true. The degree of certainty about
conventional
group or one treatment replicatkn. The
the conclusion is referred to as level of significance.
wisdom for sample size is 15 treatment replications or
You will see this reported as p 5.10 or .05 or .001.
subjects for experiments and 30 for non-experimental
is generally considered to be
A p of .05 or less
studies. Now that you know all of this, you should be
statistically significant.
studies are
wary of the validity of results when
There are many types of statistical procedures used
conducted on small numbers of subjects or on subjects
Failure to use the
for various types of experiments.
not randomly selected from the larger population.
particular
the
for
procedures
appropriate
most
7
Even if the study is internally valid - that is, the
researcher, ask if he or she has adequately accounted
results are statistically significant and there are no
for any personal or professional information which
may have affected
analysis, and
significant alternative explanations for the results, it
collection,
data
This refers to our
interpretation? Are there any biases on the part of the
may not have external validity.
researcher of evaluator? Has that individual changed?
ability to generalize about the results or to apply them
Has that individual been properly prepared and trained?
to more generalized populations. For example, results
Have the participants had or were they likely to have
from studies conducted on populations that only
had strong reactions to the researcher which may have
represent one socioeconomic status (SES), such as the
biased the way in which they acted or responded?
very poor, are not necessarily generalizable to middle
Finally, ask if there is ample data to support any
Similarly, studies of in-school
and high SES groups.
conclusions that are drawn and if the researcher has
12th graders may not be generalizable to dropouts, or
adequately considered negative cases or exceptions to
by juvenile
delinquents may
not
be
responses
the trends that appear in the data.
generalized to the larger teenaged population.
Case Study: Analysis of A Well Known LRE
Credibilkyo0ualitative Research
Experimental Study
There are issues of credibility on the qualitative
Many of us are familiar with the results reported
side of research, as well. However, they have a very
from a
study
3-year,
national
funded,
federally
different orientation. Here, the major questions are the
conducted in the early 1980's. In fact, that study's
integrity and validity of the study, the accuracy of the
results have served as our flagship in the relatively
findings, and the credibility of the researcher.
foggy void of studies generated by our field over the
The purpose of qualitative research is to describe
last twenty years. Unfortunately, some of the methods
a phenomenon, not to explain what caused it or to
used in the study raise serious questions about the
other
groups.
to
about
make generalizations
it
internal and external validity of its reported results.
Therefore the approach to
conducting qualitative
The purpose of this brief case study is not to denigrate
research is very different from that for its counterpart.
is to offer a familiar
the research so much as
it
Sample sizes are small and purposeful. That is, a few
example with which we might apply some of the
highly representative or informative cases are selected
principles described earlier.
The
for in depth study rather than large numbers.
this study, with
There were several stages to
study is conducted in the natural setting, and the report
In the first
various approaches used at each stage.
is very detailed in describing the context specifics of
year's small study of LRE and control classes at ten
Also, the study uses
the phenomenon being studied.
sites, researchers observed seven of the LRE classes
multiple means (methods and resources) of generating
and developed predictions as to which ones would
information on the question being investigated.
It
result in greater student knowledge of the law and
characteristically involves the researcher conducting
greater improvements in self-reported attitudes and
detailed observations or interviews. The researcher is
behavior toward the law. In all ten sites, students were
expected to get much more involved with those who
pre- and posttested using a 41-item measure developed
subjects)
are being studied (called participants not
study and scored
by the group conducting the
since this involvement is considered important to
independently of the classroom observations. Sites for
knowing what to ask and how to ask it.
Scientific
the study were not randomly selected (instead, they
objectivity is, therefore, out of the question. Also, the
were selected based on previous involvement and
often becomes the instrument of data
researcher
interest in LRE); nor was student assignment to LRE
collection, so credibility of the results often hinges on
Overall test scores for
and control classes random.
the background, qualifications, and style or approach
LRE classes improved in four classes, declined in four
of the researcher.
and remained the same
The
in
two.
ask how
classes,
research,
When reading
qualitative
however, were found to
predictions of success,
rigorous the researcher has been in gathering the data
favorably compare with those groups who showed the
and triangulating it or using multiple methods for such
most gains on the test, and it was out of this study that
data gathering and analysis (i.e. repeated observations;
the "six prescriptions for successful LRE instruction"
interviews coupled with surveys and document or
were born.
record reviews; interviews of people from varying
The second year's larger study of 30 sites across
perspectives; using different people to analyze the
the nation again used observations and the 41-item
Triangulation ensures an information richness,
data).
attitude
measure of student knowledge and self-reported
regard to the
accuracy, reliability, and validity. With
8
the third year, there were only eleven treatment
Again, sites were chosen because of
and behavior.
- not enough for any valid statistical
replications
their involvement in LRE, and most LRE and control
classes had students who had not been randomly
conclusions. Further, no evidence of independent tests
assigned, thus resulting in a lack of equivalence
for reliability was given for either the achievement
At one junior high school in
portion of the student test or the attitude portion of that
between groups.
Nor was any corroborating data (such as
measure.
Colorado, there were six randomly assigned classes (3
teacher or principal reports or police reports) given to
Researchers report that there were
LRE, 3 control).
support the students' self-report of delinquent activity.
14 out of 32 (44%) possible main effects (statistical
Finally, generalizability or external validity is highly
analysis procedure) for the LRE classes, and all were
questionable given the limited number of treatment
in a favorable direction.
study focused only on the
replications offered in this particular portion of the
The third year's
study and the likelihood that the school's population
Colorado junior high school where they could get a
only represents one socio-economic status rather than
random
subject
design
with
experimental
true
all (no data provided).
assignment. Teachers received summer LRE training
While these are not all the questions that have been
(enhanced by the study's earlier findings of what leads
raised about this study, it is easy to see from what little
to successful LRE), and eleven classes of students at
you've learned from this article that there are some
the school received either LRE or civics during the fall
very serious questions about the study's internal and
Out of 41 possible measured
semester's instruction.
The study's results are, at best,
external validity.
effects of LRE (related to achievement, delinquency
encouraging - not conclusive. We cannot rely on these
theory, and delinquent behavior), 24 (59%) showed
particular results as being truth, and we should be
statistically significant favorable effects for LRE. The
careful to look with a critical eye at the results of all
conclusion was that LRE in that school "had dramatic
studies prior to adopting their fiats or promoting the
favorable impact on the students who participated"
Further, we can never
11) and that the program had
products which they evaluate.
(Johnson, 1984, p.
expect to be taken seriously as a field by the larger
reduced the students' delinquent behavior.
educational community and its decision makers, until
The first question about the 3rd year study that
we generate strong and valid data to support what we
jumps to mind regards the conclusions themselves.
Only 24 out of 41 or three fifths of the overall
believe and assert to be true about LRE.
measures and less than one third of the specific
Sources
Chirot, D. (1986). Social change in the modem era.
delinquent behavior measures showed LRE to have a
New York: Harcourt Brace Jovanovich.
statistically significant impact. Another way of looking
Cook and Campbell. (1979). Quasi-experimentation:
at the results is that there was no significant impact on
Design and analysis issues for field settings.
Chicago:
(Results from the 2nd year
40% of the responses!
Rand McNally; pp. 37-84.
This is hardly a "dramatic
were even weaker.)
Johnson, G. (1984). When LRE is a deterrent to
favorable impact" and should cause you to wonder
Paper presented at Rocky Mountain
delinquency
about the bias of the researchers.
Regional Conference of the National Council for the
Beyond the questionable conclusions, the study,
Social Studies, Phoenix. (ERIC Document Reproduction
itself, had several challenges to internal and external
Service No. ED 252 459)
sites were not selected
For example,
validity.
McMillan, J.H. (1992). Educational research
randomly for participation in any year of the study.
New York:
consumer.
the
for
fundamentals
Sites were selected because they were already involved
HarperCollins.
Muraskin, L.D. (1993). Understanding evaluation
in LRE, and so the results could only be compared to
the way to better evaluation programs. [A publication of
(In other
sites with similar previous involvement.
Inc., developed with funds from the U.S.
Westat,
words, we have no idea how the LRE would have
Department of Education, contract number LC89089001].
influenced students and teachers in a school where
Patton, M.Q. (1990). Qualitative Evaluation and
LRE was a brand new phenomenon.) During the first
Research Methods. (2nd ed.). Newbury Park, CA: Sage.
two years, subjects were not randomly assigned, so
Shaver, J.P. (1984). LRE project: A methodological
accurate reliable comparisons between the experimental
Paper presented at Rocky Mountain Regional
critique.
and control classes could not be made. We just can't
Conference of the National Council for the Social
know if the results were caused by the characteristics
Studies, Phoenix. (ERIC Document Reproduction Service
of who took part in the LRE and control classes or by
No. ED 252 460)
the approach, itself. While this problem was solved in
1 0