Tuesday, February 23, 2010

Survey Development: Continuous Judgments and Direct Estimation Methods

Many of us develop surveys for use in classroom or clinical settings. When we do, we have to consider the variables that we are interested in, and often we find that these are continuous rather than categorical. One method of collecting continuous data is via a direct estimation method, where subjects are required to indicate their response by a mark on a line or by checking a box. One example of such a method would be a visual analogue scale for pain: a line of 10cm anchored with descriptors at one end of “no pain” and at the other with “worst pain imaginable.” But if you think about it, you could ask this question using different means of collecting the data. For example, you could offer the subject a series of choices, such as boxes with these descriptors: no pain, minor pain, moderate pain, severe pain, worst pain. This adjectival scale gives 5 choices from which to select. This is also called a unipolar scale, in that the descriptors range from no or little of the attribute at one end to the maximal amount at the other. This is in contrast to the Likert scale, which is bipolar, which means that it is often used to tap agreement (i.e., strongly disagree, disagree, no opinion, agree, strongly agree). But the choice of a proper scale for a question is important, both for continuous variables or categorical/dichotomous ones.

These are some of the issues you should consider when constructing a survey which uses continuous scaling:

1. How many steps should there be? This is an important consideration; if the number of levels is less than the participant’s ability to discriminate, the results will be a loss of information. Research on this question is unclear, but suggests that 5-7 steps seems to provide best information with little loss of reliability.

2. Is there a maximum number of categories? There seems to be some evidence that people cannot discriminate beyond 7 levels. But keep in mind that it is also known that people have “end aversion,” meaning that they typically avoid the end points of any scale.

3. Should there be an even or odd number of categories? For a unipolar scale, this would not matter, but for a bipolar scale the use of an odd number allows you to build in a “no opinion” option.

4. Should all the points on a scale be labeled, or only the ends? While research indicates that there is relatively little difference between scale with adjectives under each box and end-anchored scales, that same research indicates that respondents are more satisfied when many or all the points on a scale are labeled. Also, if you do not label all boxes, research has shown that people will often select a box that has been labeled, not one that has not.

5. Should the neutral point always be in the middle? Where positive or negative responses to an item are equally possible, it makes sense to have the neutral point in the middle. There are more technical reasons to unbalance a scale but I will not describe them here.

6. Do the adjectives always convey the same meaning? What does it mean to agree or to strongly agree? How does “often” differ from “not too often?” The use of these quantifiers requires care and consideration.

7. Do numbers placed under the boxes influence response? Yes, to be direct. When a set of VAS scales were used in one study, with the only difference being that one scale ran from 0-10 and the other from -5 to +5, a much higher percentage of respondents did not use the lower half of the scale in the latter group (87%), while 34% of the first group did use it.

8. Do questions influence the response to other questions? Yes. People wish to seem consistent and will often refer back to questions when answering new ones. Also, participants often try to interpret what the question is asking them, so they can respond appropriately.

The take home message from all this is that it is very hard to develop good questions in surveys and a great deal of thought and testing needs to go into testing the questions you write before you use them in survey research. This is but a small part of information on this topic, but my reference for this is from the following book: Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use, 3rd edition. New York City, NY; Oxford Press, 2003.

Monday, February 15, 2010

Promoting Active Member Participation in Group Process Settings

For those of us who use small-group learning processes in the classroom, helping those small groups become effective may involve a number of important considerations. Literature on small-group effectiveness has identified several characteristics that differ between new groups and groups that have existed for some time (1). Among these are:

Group Trust and Attraction: People’s willingness to be attracted to a group (and thereby participate) relates to the level of trust members have in each other. This occurs when members see each other reliably complete tasks over time; thus, newer groups may lack trust that older groups have established. A tactic here would be to provide small tasks at the outset that can be completed, thus beginning to build that trust.

Motivation to Achieve Group Goals: Group goals are seen as a key in the development of group trust and cohesiveness. In groups with high levels of diversity, finding common goals is helpful in developing team identity. Goals also provide a basis for team interaction. Highly cohesive groups are generally more effective at achieving group goals.

Willingness to Help Each Other: Effective groups are generally comprised of people willing to help one another. They fell responsibility toward one another and are more willing to provide interpersonal support.

Awareness of Each Other’s Skills and Abilities: New groups do not yet understand what each member brings to the table, and their perceptions of each other may initially be based on stereotyping and observable physical characteristics. Over time, as they work together, each person’s skills are brought to the fore.

Effective Sharing of Task-Related Information: Information sharing in new groups is not likely to support high task performance on tasks; thus, exchange of task-related information is likely to be low and focused on lower-level learning. Interpersonal issues seem to take precedence. Over time, as groups develop, group members begin to get more comfortable with each other and shift to task-related issues. Long-lived groups have a lessened reliance on their best member.

Willingness to Disagree: Completion of tasks requires some constructive conflict, but new groups generally withhold information that would make such constructive conflict possible. That is, they suppress information that might create conflict in order to maintain harmony at all costs.

Methods of Resolving Conflict: Conflict resolution differs between new small groups and established ones (groups with more than, say, 20 hours of action). For example, voting is often used in new groups to resolve conflict, while consensus emerges as a main conflict resolution process in more established groups.

Overall Ability to Complete Difficult Intellectual Tasks: In new groups, members need to work on tasks while at the same time learn to work with each other. This can lead to some dysfunction, but this dysfunction is also actually useful for the group to later find ways to work out its issues and begin to trust one another. Give-and-take discussions need to occur at some point.

The point of all this is simply to note that you can anticipate certain problems when you first constitute small groups for classroom learning, but that you can also develop tactics and strategies to help smooth the process from new group to established group. This then enhances learning.

1. Michaelson LK, Bauman Knight A, Fink LD, eds. Team-based learning: a transformative use of small groups in college teaching. Sterling, VA; Stylus Publishing, 2004

Monday, February 8, 2010

A Short Description of Multiple Regression

The context for evidence-based practice revolves around the diagnosis and management of a patient. When a physician is confronted with a patient for whom he or she does not know the best method to proceed, that physician can search the literature to find guidance and direction. One of the challenges of living in a evidence-based world is to be able to then interpret the literature that is uncovered. And because most of us trained as clinical practitioners (for those that are involved in patient care), we do not generally have statistical expertise. We come across terms that confuse us: confidence intervals, p-values, t tests, Pearson’s r, Cronbach’s alpha, linear and multiple regression, analysis of variance (ANOVA), etc. And we simply glitch over those terms, looking to see if our answer is somehow embedded in that paper, but not understanding that answer when it appears. To help, I wish to discuss one form of analysis we often come across: multiple regression.

Multiple regression is nothing more than a statistical method for studying the relationship between several independent or predictor variables and a single dependent or criterion variable. This is a technique widely used in social sciences and increasingly common in biological clinical research. It uses linear equations with more than 2 variables, in the forms y= a + b1x1 + b2x2, where y is the dependent variable, a and b are constant numbers and x1 and x2 are independent variables.

There are two main uses for multiple regression. One is for prediction, and the other is for causal analysis. In prediction studies, what the research is attempting to do is to develop a formula for making predictions about the dependent variable, based on the observed values of the independent variables. (1) (Note: A dependent variable is what you measure in the experiment and what is affected during the experiment. The dependent variable responds to the independent variable. It is called dependent because it "depends" on the independent variable. In a scientific experiment, you cannot have a dependent variable without an independent variable. Further, dependent variables are also called response variables or outcome variables; independent variables may be called predictor variables or explanatory variables). We might, for example, want to predict future episodes of low back pain based on such variables as past episodes of pain, pain intensity, length of episode, and age. In a causal study, the independent variables are seen as the cause of the dependent variable and therefore the aim of the study becomes determining whether a given independent variable affects the dependent variable in a meaningful way, and to estimate how large that effect is. We might have data showing that people who participate in a back school have less severe episodes of later back pain. A multiple regression can determine if this relationship is real or if it could be explained away by the fact that the people who took the back school were younger, fitter and did more exercise than those who did not.

Multiple regression has some truly notable attributes. In prediction studies, it makes it possible to combine variables to optimize predictions about the dependent variable. But, in causal studies it actually separates the effects of independent variables on the dependent variable so you can look at the contribution of each variable on its own.

Some caveats: First, one of the main conceptual problems with regression techniques are that they can ascertain relationships, but cannot be sure about underlying causal mechanisms. Second, the more predictor variables you add to the model, the more likely some will appear to be significant due to chance alone.

1. Allison PD. Multiple regression: a primer. Thousand Oaks, CA: Pine Oaks Press, 1999

Monday, February 1, 2010

How to Find the Best Evidence

Keeping up with the current best evidence is a difficult process for all of us. New knowledge is expanding at a frightening rate, and it is easy for us to become overwhelmed by it. The authors of the small text Evidence-Based Medicine: How to Practice and Teach EBM (1) suggest that one method by which we can remain up-to-date is to use a form of “learning by inquiry.” This is based on the idea that when we are confronted with a clinical question where we are unsure of the answer, we develop methods for finding the current best answer as efficiently as possible. The authors remind us that there is a hierarchy of evidence that exists, which they base on the concept of the 4S (see below). We need to be able to develop a clinical question which will also allow us to do an efficient and effective search, no matter that the search is done on a web-based database or simply by hand searching a textbook or pocket manual. With regard to orienting ourselves to evidence-based information sources, the authors suggest the following:

1. Burn your traditional textbooks. This hurts, of course, because I have developed and published textbooks, but I also recognize the truth here. I know that the first textbook I wrote took 4 years to complete; by the time I was done, it was out of date. And this is endemic to most textbook publishing. But part of the problem is that not everything in the text is out of date; just some of it is. But we don’t know which parts, making it a less trustworthy source of clinical information. They note that textbooks are good at answering what are called “background” questions, questions such as, say, information about pathophysiology, but they are not good at “foreground” questions concerning causal factors, risks, and management.

2. Take a “4S” approach to evidence-based information. They note the rapid evolution of information services occurring at present. Thus, they suggest noting the presence of a 4S hierarchical structure, where studies lie at the base of a pyramid, with syntheses on top of that, synopses higher yet, and systems at the pinnacle. You should always try to locate information that will answer your question that is located at the highest level of this pyramid. A system would “integrate and concisely summarize all relevant and important research evidence about a clinical problem” and would help link you to the relevant information. Systems do not tell you what to do, but need to be integrated with the specifics of the patient’s situation. Some systems do exist; one noted in the text is Clinical Evidence (http://www.clinicalevidence.com, also available in Ovid). PIER (Physician’s Information and Evidence Resource)is another (http://pier.acponline.,org/index.html). These are, unlike textbooks, regularly updated. Synopses are reviews of individual studies and reviews. A perfect synopsis would provide just the information you need to support your clinical action. A synthesis is useful when no systems or synopses are available. The Cochrane database is one example of a repository of syntheses. These look for evidence and then summarize the findings.Finally, studies are, of course, individual pieces of research on a topic. If no systems, synopses or syntheses exist, one can rely on studies to help guide action.

3. Organize access to evidence-based information services. Most of the references we use can be found on the internet. And because we work at an academic institution, we are provided access to a rich database of resources, but we need to keep mindful that not all of these resources are useful or appropriate for our needs. We have to determine what we want. And for those not in academia, they may be limited to open access resources such as Biomed Central (http://www.biomedcentral.com/), which offers access to all its publications at no cost to the consumer.

4. Is it time to change how you seek best evidence? How many of us use the 4S approach? How often is that instead we use the basic idea that systematic reviews are the pinnacle of information we need? Can we change our behavior to use better resources?

This is simply a guide to doing just that, to looking at the use of systems as the pinnacle of information seeking for clinical guidance.

1. Struas S, Richardson WS, Glasziou P, Haynes RB. Evidence-Based Medicine: How to Practice and Teach EBM, 3rd edition. New York, NY: Elsevier, 2005:31