Sensitivity and Specificity: Examples from the Literature

Here are three examples of how sensitivity and specificity are used in the literature.

Kongsted A, Johannesen E, Leboeuf-Yde C. Feasibility of the STarT back screening tool in chiropractic clinics: a cross-sectional study of patients with low back pain. Chiropr Man Therap.2011 Apr 28;19:10.

ABSTRACT

The STarT back screening tool (SBT) allocates low back pain (LBP) patients into three risk groups and is intended to assist clinicians in their decisions about choice of treatment. The tool consists of domains from larger questionnaires that previously have been shown to be predictive of non-recovery from LBP. This study was performed to describe the distribution of depression, fear avoidance and catastrophising in relation to the SBT risk groups. A total of 475 primary care patients were included from 19 chiropractic clinics. They completed the SBT, the Major Depression Inventory (MDI), the Fear Avoidance Beliefs Questionnaire (FABQ), and the Coping Strategies Questionnaire. Associations between the continuous scores of the psychological questionnaires and the SBT were tested by means of linear regression, and the diagnostic performance of the SBT in relation to the other questionnaires was described in terms of sensitivity, specificity and likelihood ratios.In this cohort 59% were in the SBT low risk, 29% in the medium risk and 11% in high risk group. The SBT risk groups were positively associated with all of the psychological questionnaires. The SBT high risk group had positive likelihood ratios for having a risk profile on the psychological scales ranging from 3.8 (95% CI 2.3 - 6.3) for the MDI to 7.6 (95% CI 4.9 - 11.7) for the FABQ. The SBT questionnaire was feasible to use in chiropractic practice and risk groups were related to the presence of well-established psychological prognostic factors. If the tool proves to predict prognosis in future studies, it would be a relevant alternative in clinical practice to other more comprehensive questionnaires.

Leboeuf-Yde C, Rosenbaum A, Axén I, Lövgren PW, Jørgensen K, Halasz L, Eklund A, Wedderkopp N. The Nordic Subpopulation Research Programme: prediction of treatment outcome in patients with low back pain treated by chiropractors--does the psychological profile matter? Chiropr Osteopat. 2009 Dec 30;17:14

ABSTRACT

Background:  It is clinically important to be able to select patients suitable for treatment and to be able to predict with some certainty the outcome for patients treated for low back pain (LBP). It is not known to what degree outcome among chiropractic patients is affected by psychological factors.

Objectives:  To investigate if some demographic, psychological, and clinical variables can predict outcome with chiropractic care in patients with LBP.

Methods:  A prospective multi-center practice-based study was carried out, in which demographic, clinical and psychological information was collected at base-line. Outcome was established at the 4(th )visit and after three months. The predictive value was studied for all base-line variables, individually and in a multivariable analysis.

Results:  In all, 55 of 99 invited chiropractors collected information on 731 patients. At the 4(th )visit data were available on 626 patients and on 464 patients after 3 months. Fee subsidization (OR 3.2; 95% CI 1.9-5.5), total duration of pain in the past year (OR 1.5; 95% CI 1.0-2.2), and general health (OR 1.2; 95% CI 1.1-1.4) remained in the final model as predictors of treatment outcome at the 4(th )visit. The sensitivity was low (12%), whereas the specificity was high (97%). At the three months follow-up, duration of pain in the past year (OR 2.1; 95% CI 1.4-3.1), and pain in other parts of the spine in the past year (OR1.6; 1.1-2.5) were independently associated with outcome. However, both the sensitivity and specificity were relatively low (60% and 50%). The addition of the psychological variables did not improve the models and none of the psychological variables remained significant in the final analyses. There was a positive gradient in relation to the number of positive predictor variables and outcome, both at the 4(th )visit and after 3 months.

Conclusion:  Psychological factors were not found to be relevant in the prediction of treatment outcome in Swedish chiropractic patients with LBP.

Vanti C, Bonfiglioli R, Calabrese M, Marinelli F, Violante FS, Pillastrini P. Relationship between interpretation and accuracy of the upper limb neurodynamic test 1 in carpal tunnel syndrome. J Manipulative Physiol Ther. 2012 Jan;35(1):54-63. Epub 2011 Oct 27

ABSTRACT

Objective:  This prospective diagnostic test study evaluated the relationship between interpretation criteria and accuracy of the Upper Limb Neurodynamic Test 1 (ULNT1) in the diagnosis of carpal tunnel syndrome.

Methods: A blind comparison with a reference criterion of typical clinical presentation and abnormal median nerve conduction was used. All subjects were first tested with nerve conduction studies and then with ULNT1. Each examiner was blinded to the results collected by the other examiners.

Results: We analyzed 47 subjects and 84 limbs. Considering ULNT1 as positive in the presence of reproduction of symptoms only in the thumb or lateral 2 fingers, we estimated sensitivity to be equal to 40% (95% confidence interval [CI], 0.256-0.564), specificity 79.59% (95% CI, 0.664-0.885), positive predictive value 58.33% (95% CI, 0.388-0.755), negative predictive value 65% (95% CI, 0.524-0.758), positive likelihood ratio 1.96 (95% CI, 1.275-3.012), and negative likelihood ratio 0.7538 (95% CI, 0.490-1.159). Considering the increase of symptoms with contralateral or decrease of symptoms with ipsilateral cervical side bending as mandatory positivity criterion, specificity improved, but sensitivity decreased.

Conclusion: Our investigation suggests that the reproduction of symptoms in the thumb or lateral 2 fingers of the affected arm during ULNT1 has weak diagnostic accuracy for carpal tunnel syndrome.

Sensitivity and Specificity

Sensitivity and specificity are measures used to assess the value of a diagnostic test. For any given test in any given patient, please remember that the disease or condition itself may be present or absent, and the patient may test positive or negative. Given this, we need to know how useful our test really is. We can construct a 2X2 contingency table, where across the top are two columns: disease present, disease absent. We can add along the side test positive, test negative. This creates 4 cells:

• 1.       Disease present, test positive (true positives)
• 2.       Disease absent, test positive (false positives)
• 3.       Disease present, test negative (false negatives)
• 4.       Disease absent, test negative (true negatives)

Keeping this in mind, we can now define sensitivity and specificity and calculate them from our table.

Sensitivity is defined as the ability of a test to correctly identify a patient who truly has the disease in question. It is equal to the number of people with the disease who test positive divided by all the people in which the disease is present: true positives/(true positives + false negatives).

Specificity is the ability of test to properly identify those people who do not have the disease in question. It is equal to the number of people without the disease divided by all the people in which the disease is absent: true negatives/(true negatives + false positives).

Imagine a orthopedic diagnostic test for a herniated intervertebral  disc, tried out on 100 people with acute low back pain. The actual presence absence of herniated disc was diagnosed from MRI and we are comparing the results of our own orthopedic test against that “gold standard.”  The results of our 2x2 table show that our diagnostic test was positive in 20 people in which MRI also demonstrated the presence of a herniated disc; it was negative in 5 people even though there was MRI proof of herniation (false negatives); it was positive in 30 people for which the MRI did not show evidence of herniation (false positives); and it was absent in 45 people when MRI also showed no evidence of herniation (true negatives).  Sensitivity would therefore be 20/(20 + 5) = 20/25 + 0.8, or 80%. If a herniation is present, there is an 80% chance of the test properly identifying it being present.  Specificity would therefore be 45/(30 + 45) =45/75 = 0.6, or 60%. If no herniation is present, there is a 60% chance of the test being negative. However, keep in mind that there is a 40% chance of having a false positive result.

Next week, we will look at some of the implications of sensitivity and specificity.

New Research from Biomed Central

1. Varghese J, Faith M, Jacob M. Impact of e-resources on learning in Biochemistry: first-year medical students' perceptions. BMC Medical Education 2012, 12:21 doi:10.1186/1472-6920-12-21

ABSTRACT 

Background: E-learning resources (e-resources) have been widely used to facilitate self-directed learning among medical students. The Department of Biochemistry at Christian Medical College (CMC), Vellore, India, has made available e-resources to first-year medical students to supplement conventional lecture-based teaching in the subject. This study was designed to assess students' perceptions of the impact of these e-resources on various aspects of their learning in biochemistry.

Methods: Sixty first-year medical students were the subjects of this study. At the end of the one-year course in biochemistry, the students were administered a questionnaire that asked them to assess the impact of the e-resources on various aspects of their learning in biochemistry. 

Results: Ninety-eight percent of students had used the e-resources provided to varying extents. Most of them found the e-resources provided useful and of a high quality. The majority of them used these resources to prepare for periodic formative and final summative assessments in the course. The use of these resources increased steadily as the academic year progressed. Students said that the extent to which they understood the subject (83%) and their ability to answer questions in assessments (86%) had improved as a result of using these resources. They also said that they found biochemistry interesting (73%) and felt motivated to study the subject (59%).

Conclusions: We found that first-year medical students extensively used the e-resources in biochemistry that were provided. They perceived that these resources had made a positive impact on various aspects of their learning in biochemistry. We conclude that e-resources are a useful supplement to conventional lecture-based teaching in the medical curriculum. 

2.       2. Morris H, Ryan C, Lanchlan D, Field M. Do medical student attitudes towards patients with chronic low back pain improve during training? A cross-sectional study. BMC Medical Education 2012, 12:10 doi:10.1186/1472-6920-12-10

ABSTRACT 

Background: Health care professionals with positive attitudes towards the functional abilities of patients with low back pain are more likely to encourage activity and avoidance of rest as per recommended guidelines. This study investigated whether medical student training fosters positive attitudes towards patients with back pain and their ability to function.

Methods: First (n = 202) and final (n = 146) year medical students at the University of Glasgow completed the Health Care Professionals' Pain and Impairment Relationship Scale (HC-PAIRS) questionnaire. This measures attitudes of clinicians towards the functional ability of patients with back pain. A group of first (n = 62) and final year (n = 61) business students acted as non-health care controls. Attitudes were compared using two-way ANOVA with year of study and discipline of degree as independent variables.

Results: Both year of study [F(1,465) = 39.5, p < 0.01] and discipline of degree [F(1,465) = 43.6, p < 0.01] had significant effects on total HC-PAIRS scores and there was a significant interaction effect [F(1,465) = 9.5, p < 0.01]. Medical students commenced their course with more positive attitudes than non-health care students (65.7 vs. 69.2 respectively; p < 0.01) - lower scores translating into more positive attitudes. In their final year, the difference between the two student groups had widened (56.4 vs. 65.3; p < 0.01).

Conclusions: Undergraduate medical training promotes positive attitudes towards the functional ability of patients with back pain, suggesting that students may be more likely to develop an evidence-based approach to this patient group after qualification. Some adjustments to training may be warranted to encourage a more positive shift in attitudes. 

3.      3. Emmerton L, Fejzec J, Tett SE. Consumers' experiences and values in conventional and alternative medicine paradigms: a problem detection study (PDS). BMC Complementary and Alternative Medicine 2012, 12:39 doi:10.1186/1472-6882-12-39

ABSTRACT 

Background: This study explored consumer perceptions of complementary and alternative medicine (CAM) and relationships with CAM and conventional medicine practitioners. A problem detection study (PDS) was used. The qualitative component to develop the questionnaire used a CAM consumer focus group to explore conventional and CAM paradigms in healthcare. 32 key issues, seven main themes, informed the questionnaire (the quantitative PDS component - 36 statements explored using five-point Likert scales.)

Results: Of 300 questionnaires distributed (Brisbane, Australia), 83 consumers responded. Results indicated that consumers felt empowered by using CAM and they reported positive relationships with CAM practitioners. The perception was that CAM were used most effectively as long-term therapy (63% agreement), but that conventional medicines would be the best choice for emergency treatment (81% agreement). A majority (65%) reported that doctors appeared uncomfortable about consumers' visits to CAM practitioners. Most consumers (72%) believed that relationships with and between health practitioners could be enhanced by improved communication. It was agreed that information sharing between consumers and healthcare practitioners is important, and reported that "enough" information is shared between CAM practitioners and consumers. Consumers felt comfortable discussing their medicines with pharmacists, general practitioners and CAM practitioners, but felt most comfortable with their CAM practitioners.

Conclusions: This PDS has emphasized the perceived importance of open communication between consumers, CAM and conventional providers, and has exposed areas where CAM consumers perceive that issues exist across the CAM and conventional medicine paradigms. There is a lot of information which is perceived as not being shared at present and there are issues of discomfort and distrust which require resolution to develop concordant relationships in healthcare. Further research should be based on optimisation of information sharing, spanning both conventional and CAM fields of healthcare, due to both the relevance of concordance principles within CAM modalities and the widespread use of CAM by consumers.

Incidence and Prevalence

Incidence and prevalence are terms used in epidemiological research that have been used as interchangeable words in our general conversations. But they are not the same thing.

Incidence is defined as the number of new cases of a condition over a given period of time (typically one year) in a percentage of the population. It may be expressed as either a number per 100,000 people (as is often the case in news reports) or as a percentage. For example, if every year in a practice of 1,000 people, you diagnose 15 of them with Meniere’s disease, the yearly incidence would be 1.5% (or, 1500 per 100,000).

Prevalence is the number of existing cases of a condition at either a single point in time (point prevalence) or over a given period of time (period prevalence). For example, at the time of study 90 people in a practice of 1000 patients were suffering from Meniere’s disease (15 who were diagnosed this year and 75 who were diagnosed in past years). The prevalence here would be 9% (or 9,000 per 100,000).

It should be clear that with chronic diseases, such as diabetes mellitus, the incidence will be lower than the prevalence. Here, each new case adds to the large number of already existing cases. With a short-term illness, such as the common cold, the opposite will be true. Over the course of a year, maybe 50% of a population will have a cold, but at any given time (point prevalence), the number with a cold will be low, perhaps at 2%.

One final thought. It is important to note the relationship of incidence to the concept of risk. Risk, such as is calculated in epidemiological cohort studies, looks to see how many new cases develop as new follow a group of people forward in time to see if exposure to some risk factor leads to the condition of interest.  We start with a group of people, none of whom have the condition of interest. As we follow the patients, some will then develop that condition. Since they are all new cases, we can calculate true incidence rates in a population, and from that determine risk (the probability that an event will occur, which is calculated by dividing the number of events- new cases- by the number of people at risk- the population).

Confidence Intervals

Confidence intervals (CI) are an important concept for clinicians and academics in healthcare to grasp. You will now see them reported in up to 75% of all papers. CIs are typically used when instead of wanting the mean value of a sample, we want a range that is likely to contain the true population variable. Now, the idea of a “true value” is actually a bit hard to get our arms around, but it refers to the mean value we would find if we could gather data for the entire population from which our sample is drawn.

As it turns out, statisticians can calculate a range (or an interval, if you will) in which we can be pretty sure (that is, confident) that the “true value” lies within it. For example, we might be interested in pain reduction with spinal adjusting. From a sample of research participants, we can work out the mean reduction in pain as measured on, say, a Visual Analogue Scale. However, this will only be the mean for our specific sample. If we gathered a second group of research participants, we would not get, nor expect to get, the exact same value. This could be due to chance, biological variation, etc. The CI gives the range in which the true value is likely to lie, if we could do this an infinite number of times.

Example: the average pain score prior to adjusting in study A was 7/10 in a group of 80 low back pain patients. After adjusting, the mean pain score dropped by 3 points. If the 95% CI is 1-5, we can be 95% confident that the true effect of treatment is to lower pain by 1-5 points. If in study B, using a different chiropractic technique, and also reducing their mean pain score by 3 points, there is a wider 95% CI of -1-5. This CI includes 0 (it runs from -1 to +5) and the inclusion of 0 (no change) means there is more than a 5% chance that there was no true change in pain, and the treatment was actually ineffective. (This is, you may note, similar to saying that p>.05, but the CI is much easier to intuitively understand).

A point to remember is that the size of the CI is related to the sample size of the study. Larger studies will almost always have narrower CIs.

Made to Stick: Why Some Ideas Survive and Others Die

As I noted last week in my first post about this wonderful little book (1), I was using principles noted in the book without knowing that I was doing so. In my case, what caught my students’ attention and turned what should have been a boring and uninteresting lecture (to them, that is!) was an unexpected outcome; that when one searched for “Brain CT” in Pubmed, one found only 5 links rather than the hundreds students expected. In this book, the authors found that there were 6 themes that seemed common to “sticky” ideas (ideas that stay with people), and I would like to list them here.

Simplicity: The authors argue that you need to find the core of your idea; that if you give people 10 good points, they will not remember any of them. You need to be a master of exclusion, and create something that is both simple and profound. As an example, the authors cite the Golden Rule- a single sentence that is profound, with lifelong ramifications.

Unexpectedness: what we want is for people to pay attention to our ideas, which becomes a challenge when you need time to get ideas across to them. They suggest that we violate people’s expectations, be counter-intuitive, use surprise. But they caution that just using surprise is not enough; we must generate interest and curiosity. This is what my “Brain CT” example does- it demonstrates an unexpected gap in their knowledge, which is combined with my simple comment that “Google is not Pubmed.”

Concreteness: In order to make our ideas clear, we need to explain them in terms of human actions and sensory information. The authors state that this is where so many mission statements and strategies go wrong, because they are often ambiguous to the point of being meaningless. By being concrete, we ensure that our idea means the same thing to everyone in the audience: Palmer College is, for example, “The trusted leader in chiropractic education.”

Credibility: This relates to how we make people believe our ideas. Sticky ideas need to carry their own credentials. People need to test our ideas for themselves. The authors recommend not relying on hard numbers (where they would work), and use the example of political discussion to make this point. Do we relate to statistics that are presented to us by our political leaders? No, we do not; they do not carry meaning to us. But what of Ronald Reagan’s question back when he was running against Jimmy Carter” Before you vote, ask yourself if you are better off today than you were four years ago.”

Emotions: In order to get people to care about our ideas, we should make them feel something. This is well known in the charitable society world; research has shown we are more likely to donate to a single needy individual than to an entire impoverished region. We feel for people, not for abstractions. The authors note that this is not always easy. They use an example that it can be easier to get a teenager to quit smoking not by citing hard facts about cancer but by appealing to their anti-corporate beliefs about Big Tobacco.

Stories: One of the best ways to get people to act on our ideas is by telling stories, and we teachers are natural storytellers. When we do, we multiply our experiences and make it easier for others to use that experience when there is a need to do so. Stories act to help us respond more quickly and effectively.

These are the six principles of successful ideas, and this book then goes on to amplify each one of these, applying them to both business and to teaching. This is a wonderful book, well worth having.

References
1. Heath C, Heath D. Made to stick: why some ideas survive and others die. New York City, NY; Random House, 2008

An Introduction to “Made to Stick”

My friend and colleague Bill Meeker has been singing the praises of the book “Made to Stick,” by Chip and Dan Heath (1). Having just finished reading the book, I understand why. Before I dive into the book, in next week’s post (or my next post, given next week we are off on Monday for Spring break), I want to tell a story to position how this book could inform education and, more specifically, what we do in the classroom or clinic.

I teach a course in evidence-based chiropractic practice. I frame the entire class around this idea: there is a patient in your office who presents you with a new challenge, one that you are unsure what to do about. And over the course of the term, we look at what evidence-based practice is, what evidence itself is, how to locate evidence, and then how to extract information from various kinds of papers (research reports, literature reviews, case reports, epidemiological papers, and diagnostic studies). In the third week of class, the topic is how to conduct a literature search (i.e., locating evidence). Since this is a late trimester course, students look forward to this lecture with about as much excitement as they would look forward to a visit to the dentist; they think they already know how to do a literature search.

Before I launch into the lecture, I ask the class how they might look for information on some new band they just heard on the radio. As expected, they usually will say that they go to either google or, for a few, to Wikipedia. And I follow up by noting that yes, most people these days typically use google for their initial information needs, chiropractic students not excepted. So, I ask, how many links would you find on google, to, say, chiropractic? They will say over a million. And how many can you trust? They laugh quietly, knowing that not all the information is good. Hold that thought, I tell them.

I then ask them how many use pubmed? Many do, but oddly, not very often. This leads me to the question I have been waiting to ask: if we did a search on brain ct, how many links on pubmed would we find? Students will guess anywhere from a few hundred to several thousand. Now, go do that search; I’ll wait…

When they see the results, I make my point: pubmed is not google. It does not work the same way. It uses different algorithms for how it searches information, and clearly, you are not aware of them. But look at the query box… So, I then ask, what if we did a search for “spinal stenosis CT?” And again, the results are not what students expect- here because to pubmed, CT does not mean “computed tomography” but “contraindications.”

What I have done in this exercise is to use one of the key ideas in “Made to Stick.” Be unexpected. This engages learning in unique ways. Next week, let’s look at this a bit more.

References
1. Heath C, Heath D. Made to stick: why some ideas survive and others die. New York; Random House, 2008