Monday, December 15, 2008

Portfolios in Chiropractic Education

Part of our strategy as teachers in chiropractic educational settings is to encourage our students to take greater responsibility for their own learning, and where possible to personalize their learning experiences. For example, in the clinical bioethics course I teach to graduate students in clinical research, students use portfolios in part to collect articles and other ephemera that they themselves are specifically interested in. There is no caveat on what those articles might be; they simply have to reflect the student’s interests. Of course, as a means of assessment, any tool we use should enhance and support learning as well as be able to measure performance (1). One method that is gaining greater use is the student portfolio.

A portfolio is “a collection of papers and other forms of evidence that learning has taken place” (2) or “a collection of student work that exhibits the student’s efforts, progress and achievements in one or more areas” (3). Portfolios have been introduced into public school education as early as elementary school, and have found homes within professional nursing and medical education.

Friedman Ben David et al (4) note that portfolios contain student work over time demonstrating evidence for learning and progress toward educational outcomes or learning objectives. A portfolio may contain any of the following: best essays, written reports or research projects, samples of performance evaluations, videotapes of interactions with patients, records of practical procedures, letters of recommendation, annotated patient records, written reflections, journal articles, etc.

They further note that portfolios can contribute quite well to assessment. They do allow for assessment of learning outcomes, even to the point of being an effective method for examining outcomes not easily addressed by other means, such as personal growth and reflective ability. A portfolio can provide evidence of performance, which can come from numerous sources. That evidence is collected over a period of time, thus demonstrating development over time. It can track progress toward the learning outcomes as well. Finally, it has use for both summative and formative assessment.

Other benefits include the fact that portfolios may enhance interactions between students and teachers, by allowing dialogue between students and educators, by reminding students that learning is a two-way process, by stimulating teachers to reassess their teaching strategies, and by raising teacher expectation in relation to thinking ability and problem solving. Further, portfolios can enhance the use of reflective strategies on the part of the student.

As I noted above, I require my clinical bioiethics students to keep a portfolio. In it, they place their responses to a series of clinical ethics scenarios which they must address and analyze. They keep journal articles that they use for preparing a short classroom presentation, and they also keep other journal articles that they collect regarding their own personal interests. They can place newspaper articles which they find over the course of the term. They will add their final paper to the portfolio. They are encouraged to ask me to review it at any time, but they are not required to have me do so until the very end of the term. No assignment has a specific due date, except that all my students know the date at which the portfolio is to be turned in with all work completed. This allows them to allocate their time as they see fit. It has been a great success for me to see the quality of the work they do. I recommend that those interested review the references below and consider the use of the portfolio in class.

This will be my final post of the year. We are all getting ready to go on winter break, and I am doing so as well. I want to wish you all a great holiday break, good time with family and friends, and I will be back when 2009 begins.

References

1. Koretz D, Broadfoot P, Wolf A. Editorial. Assessment in Education, 1998;5(3)
2. Davis MH, Friedman Ben David M, Harden RM, et al. Portfolio assessment in medical students’ final examinations. Med Teacher 2001;23:357-366
3. Martin-Kneipp GO. Becoming a better teacher. Alexandria, VA; ASCP, 2000
4. Friedman Ben David M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. AMEE Medical Education Guide No. 24: portfolios as a method of student assessment. Med Teacher 2001;23:535-552

Monday, December 8, 2008

Relative Risk vs. Risk Difference

It is necessary to distinguish between relative risk (RR) and the risk difference (RD), because the relative risk is typically much larger than then the risk difference so that when you read results in the form of a RR it can be rather misleading. Think about reading a statement that says something has reduced the patient’s risk by 50%. This sounds quite good. However, if you think about it, it might mean nothing more than you reduced the risk from, say, 2% to 1%.

More clinically, imagine that there are three subpopulations of patients, each getting the same treatment, and for each group we see a decrease in risk of 1/3 (that is, RRR= 0.33; RR= 0.67). If this was administered to a subpopulation that has a 30% risk of dying, getting the treatment reduces that risk to 20%. When it is given to a subpopulation that has a 10% risk of dying, the treatment reduces the risk to 6.7%. And when given to a subpopulation that was just a 1% risk of dying, the treatment reduces the risk to 0.67%.

In each group, the treatment reduces the risk of dying by a third, but it really fails to adequately express the full impact of treatment. What if this treatment had a high rate of serious adverse events? Would it make much sense to offer it to the patients in the lowest risk group (where the RD is only 0.3%)? For the middle group, we’d certainly want to let them know about the risks and benefits of treatment, inasmuch as the as the absolute reduction in risk is about 3%. In the high risk group, it makes complete sense to offer this treatment, because there is an absolute benefit of 10%.

The authors of the Users’ Guide suggest that the relative risk reduction be examined in light of your assessment of your patient’s baseline risk. Obviously, you need to have information on the risk reduction of a therapy, which is not always available for chiropractic interventions but could be calculated as per our last blog post. If your know your therapy offers a relative risk reduction of 30% in a general group of patients, you might view it differently if one patient is 40 years old and in generally good health compared to one who is 70 years old and has significant co-morbidity.

This is not the easiest concept to grasp. However, the Users’ Guide offers a great resource in understanding this (1). Further information can be found at http://www.annalsofian.org/temp/AnnIndianAcadNeurol104225-401063_110826.pdf

NOTE: As I was home over the weekend, I came across this article in the Washington Post: Howell D. Making sense of science reporting (2). In it, the Washington Post Ombudsman, Deborah Howell, defends the nature of science reporting in her newspaper, but offers the following comment regarding one such article: “FIAR Director George M. Carter's chief complaint was that stories emphasized a change in "relative risk" -- a 44 percent fall in the number of heart attacks, strokes and surgical procedures among people taking the statin, compared with those in the placebo group. He said the fact that everyone in the study had an extremely low "absolute risk" for heart problems should have been emphasized more. About 1.36 percent of people taking the placebo suffered a heart attack or stroke; that fell to 0.8 percent among those taking the statin. That means that nearly 97 percent of the people using the drug would not see any benefit, he said.”

The article goes on to note the author of the news report “quoted a skeptic in the ninth paragraph and noted near the story's end that ‘the actual risk reduction for an individual would be very small, given the relatively low risk for most middle-age people, so that the benefits easily could be outweighed by the costs of thousands more people taking tests, drugs and being monitored by doctors.’”

Finally, the author said "While I would have liked to have explored many of the nuances of this study more fully, I feel confident we struck a responsible balance. I think it's crucial to provide readers with both the evidence supporting new claims and enough context and interpretation to help them gauge its significance." Independent experts, he said, concluded the study was "a very well done, very convincing piece of research."

Which, of course, was not the point. It is only that the average reader cannot draw this distinction between the absolute and relative risk reductions. That much remains clear, and points to why we need to better understand such concepts.


References

1. Guyatt G, Rennie D, Meade MO, Cook DJ. Users’ guides to the medical literature: a manual for evidence-based practice. New York, NY; McGraw Hill;208:90-91
2. Howell. D. Making sense of science reporting. http://www.washingtonpost.com/wp-dyn/content/article/2008/12/05/AR2008120502959.html?hpid=opinionsbox1

Monday, December 1, 2008

Risk, Risk Reducation and Relative Risk

One of the more common reasons we read clinical trial literature is look at the association between a given treatment, such as spinal adjustment, and the outcome of that treatment (for example, reduced pain or disability). But often we do not delve with any depth into the results, and the concept of relative risk is one that we should be looking at as we assess the outcomes for a research paper.

Let us say we have a paper in which in the course of a randomized trial we are comparing a dichotomous outcome (for example, mortality rates) in patients undergoing either Treatment A or Treatment B. And further, let us say that 36 of 128 patients in Treatment group A died, while 58 of 130 in Treatment group B died. From this, we can construct what is called a 2x2 contingency table (here referring to the columns “Death” and Survival” and the rows “Treatment A” and “Treatment B,” in which we put exposure on the left and the outcome across the top. When we do, we will see that in Treatment Group A, 36 people died and 92 survived, and we will call these "a" and "b"; in Treatment group B, 58 died and 72 survived, and we will label these "c" and "d."

This now allows us to be able to calculate risk. First, we can look at the absolute risk for mortality in both groups. This is defined as the risk of the adverse event in each group, sometimes called baseline risk. So, the risk of mortality in Treatment group A is the number of people who died divided by the total number of people in the group: a/(a+b) = 36/(36+92) = 36/128 = .28 or 28%, and in Treatment group B = 58/130 = .45, or 45%.

The difference between these rates is known as the risk difference, or absolute risk reduction. That is, if we compare these risks, we can see that a patient is less likely to die if they have treatment A. The absolute risk reduction is simply the difference between groups (difference meaning subtraction is involved) = c/(c+d) – a/(a+b). Here, that is .45-.28= .17, or 17%.

Finally, rather than look at the difference between groups, we can also look at the ratio; ratio implies that we will be dividing the rates, rather than subtracting them. When we do this, we calculate the risk ratio, or relative risk. Mathematically, this would be the risk in the Treatment group A divided by the risk in Treatment group B (which is typically a control group), or [a/(a+b) / c/(c+d)]. Here, we can see that the math is .28/.45 = .63, meaning that the risk of death is Treatment group A is about 63% (or about two-thirds) that of Treatment group B.

There are complexities to this assessment, which our next entry will take up. For those who wish to read a bit further on this, let me recommend the “Users’ Guide” as a good resource (1).

References

1. Guyatt G, Rennie D, Meade MO, Cook DJ. Users’ guides to the medical literature: a manual for evidence-based practice. New York, NY; McGraw Hill;208:87-90