Testing

EPM Testing

Table of Contents

  1. Contact
  2. Summary
  3. Introduction
  4. Reasons for False Test Results
  5. Evaluating the Reliability of Diagnostic Tests
  6. The Effect of Disease Prevalence on The Reliability of Test Results
  7. Using Disease Prevalence Estimates to Aid in Interpreting Diagnostic Tests for EPM
  8. Conclusions
  9. Recommended Reading on the Clinical Epidemiology of Diagnostic Tests
  10. References

Contact

Paul S. Morley, D.V.M., Ph.D.
William J.A. Saville, D.V.M., Ph.D.
Department of Veterinary Preventive Medicine
The Ohio State University
1900 Coffey Rd
Columbus, Ohio 43210-1092

Table of contents

Summary

We consider Western blot analysis of CSF a reliable method of diagnosing Equine Protozoal Myeloencephalitis in horses that have clinical neurologic disease. However, results of diagnostic tests must always be interpreted in light of the prevalence of disease in horses with similar clinical signs and exposures. False positive test results are common when the prevalence of disease is low while false negative results are common when prevalence is high. There is evidence to suggest that only a low percentage of clinically normal horses have detectable concentrations of antibody to S. neurona in their CSF. This low prevalence is expected to lead to a large proportion of false positive test results in this type of horse. Therefore, Western blot analysis of CSF obtained from clinically normal horses is not a reliable method of determining exposure status in individual clinically normal horses.

Table of contents

Introduction

Diagnostic tests are used widely in medicine to aid clinicians in effective diagnosis and treatment of disease. In recent years, advanced microbiologic techniques have given rise to an increasing number of tests which can be used to reliably aid in the diagnosis of disease. However, no diagnostic test is perfect and it is critical to understand test sensitivity, specificity, and predictive values in order to better understand when a test result may be true and alternatively when a result may be false. Nowhere in veterinary medicine is this more true than when practitioners use antemortem tests to aid in diagnosis of Equine Protozoal Myeloencephalitis (EPM).

Equine Protozoal Myeloencephalitis is a significant cause of neurologic disease of horses.1-5 Sarcocystis neurona is a protozoan parasite which appears to have a predilection for infecting the central nervous system in horses and is thought to be the agent which causes Equine Protozoal Myeloencephalitis.1-5 Recent investigations suggest that the genome of S. neurona recovered from affected horses is nearly identical to that of Sarcocystis falcatula, a parasite whose definitive host is the opossum (Didelphis virginiana).6, 7 Currently, the most widely applied antemortem diagnostic test uses Western blot analysis to identify antibodies to S. neurona. Horses are commonly exposed to this parasite in geographic regions which the opossum inhabits. Surveys of horses from these regions have found that seroprevalence to S. neurona is often 50% or greater.8-10 Use of the Western blot assay to test for EPM relies on the assumption that antibodies to S. neurona are only produced intrathecally when the organism has infected tissues of the central nervous system. Comparison of results of Western blot analysis of CSF to results of rigorous postmortem examination of horses with neurologic disease suggests that the Western blot is approximately 89% sensitive and 89% specific for diagnosis of EPM (Dr. David Granstrom, personal communication, 1997).

The Western blot assay has been available on a commercial basis since 1992, and is used with confidence by many practitioners to aid in diagnosis of EPM. This is due in part to extensive validation and application of the Western blot. However, anecdotal reports of seemingly spurious results continue to cause controversy and frustrate clinicians. For example, it has become a common practice to collect CSF samples from yearling horses immediately after sales and submit them for testing using the Western blot assay. Some anecdotal reports suggest that as many as 16 out of a group of 20 clinically normal yearling horses may test positive for antibody to S. neurona in their CSF (Dr. David Granstrom; Personal Communication, 1996). It is very difficult to make appropriate recommendations regarding these results without understanding how the rate of false test results is affected by the sensitivity, specificity, and predictive values of the Western blot assay. Confusion about proper interpretation of this test is common as is illustrated by discussion among clinicians who subscribe to the Equine Clinicians Network (ECN), a popular E-mail based discussion group. The validity of the Western blot is a regular topic of discussion on this mailing list and a high level of frustration is evident among equine clinicians. These concerns could be partially alleviated if clinicians had a better understanding of the principles of clinical epidemiology related to diagnostic testing.

Table of contents

Reasons for False Test Results

Many veterinarians believe that false test results are most commonly caused by problems with the testing procedure. However, the reliability of diagnostic tests can be affected by several factors which can be broken down into four broad categories: problems with the biological marker that is measured by the test, differences among individual patients, problems with sample collection or storage, and problems with the testing procedure.

Problems with the Marker of Disease: Diagnostic tests are meant to identify some characteristic or marker of disease that is not found in healthy animals. An ideal marker of disease would be found in every affected animal early in the disease process as well as throughout its course, and would never be found in any unaffected animals. However, no marker of disease is perfect, and the usefulness in differentiating between healthy and diseased animals varies among different markers of disease. In addition, some tests assay for general markers of inflammation or disease, while others are designed to detect markers of very specific diseases. There are disadvantages to using both types of markers. Markers which identify generalized disease states may not be helpful when symptoms can be caused by many different conditions, particularly if the diseases require very different treatments. For example, CSF protein concentration, as well as CK and AST activities in CSF can be used as indicators of CNS inflammation and disease, but they cannot be used to specifically differentiate EPM from other diseases which cause similar clinical signs. Polymerase chain reaction (PCR) and monoclonal antibodies can be used to identify very specific markers of infection, but variability in the genomic and antigenic makeup of pathogens can potentially cause these diagnostic tests to yield results which do not reflect the true disease status of the patient.

Differences Among Individual Patients: Because of biologic variation, some healthy animals may express a marker which is usually only expressed by diseased animals, and some diseased animals may not express this marker when most others do. The degree that markers of disease are expressed may also vary among horses in different stages of disease (acute versus chronic). For example, clinicians have reported that horses in the very early stages of EPM may not have detectable concentrations of antibody in the CSF, but will be positive if retested a week or two later.2, 11, 12 The occurrence of other diseases can also affect the results of tests. For example, conditions affecting the integrity of the blood-brain barrier can allow serum antibody to leak into the CSF and thus affect results of diagnostic tests for EPM.

Problems with Sample Collection or Storage: Veterinarians frequently obtain cerebrospinal fluid from the lumbosacral or atlanto-occipital spaces for the purposes of diagnostic testing. However, it is easy to inadvertently contaminate samples with blood during the sampling procedure which can cause CSF samples to have falsely elevated S. neurona antibody concentrations. This is particularly problematic because of the high seroprevalence to S. neurona.8-10 This is why the CSF indices (albumin quotient and IgG index) are used to aid in determining if antibody in CSF was produced intrathecally.13-15 Inappropriate handling during sampling or storage can damage samples and increase the likelihood that false test results will be obtained. For example, immunoglobulin and other proteins can be denatured by exposure to extreme heat or UV radiation.

Problems with the Testing Procedure: Inherently, some diagnostic tests have more variability than others and it may require considerable skill and finesse to produce consistent results. The variability in skill and diligence used when performing diagnostic tests can easily affect results, as is true of any technical task. It is worthwhile to mention that variation in diagnostic test results can be manifest as inaccuracy as well as imprecision. Differences between these two types of variability are often illustrated by correlating multiple test results with the pattern that a shotgun might make on a target. Accuracy describes how well the pattern is aligned with the center of the target. Precision describes how small the pattern is. Known standard samples should be tested with every batch in order to ensure a high degree of accuracy between batches. Procedures should be strictly standardized whenever possible to increase the accuracy and precision between batches.

Table of contents

Evaluating the Reliability of Diagnostic Tests

In order to estimate the usefulness or reliability of a new diagnostic test, results are compared to the true disease status of a population of animals. Parallel analysis of samples using two different assays allows calculation of the new test's sensitivity and specificity. Unfortunately, it is not possible to be 100% confident of the disease status of any animal, and so results of the new test are usually compared to those of an accepted method of diagnosis, a "gold standard." It is important to emphasize that results of the gold standard only provide a different estimate of the true disease status, and results will in fact not be correct in some cases. However, for purposes of comparison, this standard is used to provide the best or most practical estimate of true disease status that is available.

In epidemiologic terms, sensitivity and specificity describe the rate of false positive and false negative results that will be obtained from a diagnostic test. SENSITIVITY is the proportion of truly diseased animals that test positive, while SPECIFICITY is the proportion of non-diseased animals that test negative (Figure 1). In other words, a test that has a high probability of correctly identifying diseased animals has a high sensitivity, and a test that is highly likely to confirm that healthy animals are not diseased has a high specificity. Using the estimate cited previously for the sensitivity and specificity of Western blot analysis of CSF when used to diagnosis EPM, 89% of horses with post-mortem findings suggestive of EPM would be expected to have detectable concentrations of antibodies in their CSF, and 89% of horses with other neurologic diseases would be expected to test negative using Western blot analysis of CSF. Unfortunately, tests that are extremely sensitive will often have a lower specificity, and tests that are extremely specific often have a lower sensitivity. Tests that are both highly sensitive and highly specific are often expensive or technically difficult to perform. It should be noted that microbiologists sometimes use the terms sensitivity and specificity in a different way: microbiologic sensitivity can be described as the lower limit of detection of a diagnostic test (i.e., the lowest antibody concentration can be detected), and specificity is used to describe whether a diagnostic test will detect exposure to several different microorganisms.

Figure 1.

Estimates of reliability for diagnostic tests.
Test ResultsTrue Status (+)* True Status (-)* Totals
(+) True Positive
(a)
False Positive
(b)
Test Positive Animals
(a+b)
(-) False Negative
(c)
True Negative
(d)
(c+d)
Test Negative Animals

Disease Positive
(a+c)
Disease Negative
(b+d)
All Animals
(a+b+c+d)

*True Disease Status ("Gold Standard" Test Results).

Test Sensitivity

Proportion of True Positive results among Disease Positive animals = a / (a+c)

Test Specificity

Proportion of True Negative results among Disease Negative animals = d / (b+d)

Positive Predictive Value

Proportion of Disease Positive animals among with those with Positive Test results = a / (a+b)

Negative Predictive Value

Proportion of Disease Negative animals among those with Negative Test results = d / (c+d)

True Prevalence

Proportion of all animals that are truly diseased = (a+c) / (a+b+c+d)

Apparent Prevalence

Proportion of all animals that are Test Positive = (a+b) / (a+b+c+d)

Table of contents

The Effect of Disease Prevalence on The Reliability of Test Results

Most practitioners understand that a test with high sensitivity and specificity is more reliable than a test with low sensitivity and specificity, but few are able to extend this understanding to aid in interpretation of results of diagnostic tests for individual patients. Sensitivity and specificity describe the proportion of truly diseased animals that will test positive, and the proportion of truly healthy animals that will test negative. However, this is not exactly the information that a practitioner needs when interpreting test results for a patient. The clinician actually needs to know what proportion of animals that test positive are truly diseased (POSITIVE PREDICTIVE VALUE), and what proportion of animals that test negative are not diseased (NEGATIVE PREDICTIVE VALUE) (Figure 1). The difference between sensitivity and positive predictive value (PPV), as well as the difference between specificity and negative predictive value (NPV) may seem subtle, but are extremely important. While sensitivity and specificity are fixed characteristics of a test that will remain the same when different populations of animals are tested, positive and negative predictive values are greatly affected by the true prevalence of disease in the population. Figure 2 shows how the positive predictive value for a test with 89% sensitivity and 89% specificity changes as the prevalence of disease changes. The positive predictive value is very low when disease prevalence is low and is maximized when prevalence is high. This means that when using the Western blot to diagnose EPM, the rate of false positive and false negative test results will change depending upon the symptoms of disease that are shown by patients.

Figure 2.

Changes in positive predictive value.

Changes in positive predictive value relative to changes in the prevalence of disease for a test that is 89% sensitive and 89% specific.

It may seem odd that the ability of a test to predict the presence or absence of disease (predictive value) changes even though the sensitivity and specificity of a test are fixed characteristics. The reason that this happens is illustrated using three hypothetical populations with disease prevalences of 95%, 50%, and 1% in Figure 3, Figure 4, and Figure 5, respectively. The sensitivity and specificity is fixed in these examples (89% and 89%), but the positive and negative predictive values are markedly different (respectively, PPV=99%, 89%, 8%; NPV=31%, 89%, 100%). Remember that the proportion of true positive and false positive test results determines the positive predictive value, and animals with false positive test results do not actually have disease. In situations where the disease prevalence is high (and most of the animals are truly diseased), only a minority of animals are disease negative so that the number of true positive test results far outweighs the number of false positive results and the positive predictive value will be high (Figure 3). If the true disease prevalence is low (and most animals are not diseased), the actual number of false positive results far outweighs the number of true positive results even though the proportion of each is determined by fixed sensitivity and specificity, and the positive predictive value will be low (Figure 5). Notice when the true prevalence of disease is 1% (Figure 5), the apparent prevalence of disease (i.e. the proportion of all animals that test positive) is very different from the true prevalence of disease (12% versus 1%). Negative predictive value is similarly affected by disease prevalence, except that it is maximized when disease prevalence is low (Figure 3 and Figure 5).

Figure 3.

Example with 95% prevalence of disease (test sensitivity=89%, test specificity=89%).
Test Results True Status (+)*True Status (-)*Totals
(+) 846 4 850
(-) 104 46 150

950 50 1000

*True Disease Status ("Gold Standard" Test Results).

  • Sensitivity: 89%
  • Specificity: 89%
  • True Prevalence: 95%
  • Apparent Prevalence: 85%
  • Positive Predictive Value: 99%
  • Negative Predictive Value: 31%

Figure 4.

Example with 50% prevalence of disease (test sensitivity=89%, test specificity=89%).
Test Results True Status (+)*True Status (-)* Totals
(+) 445 55 500
(-) 55 445 500

500 500 Total
1000

*True Disease Status ("Gold Standard" Test Results)

  • Sensitivity:89%
  • Specificity: 89%
  • True Prevalence: 50%
  • Apparent Prevalence: 50%
  • Positive Predictive Value: 89%
  • Negative Predictive Value: 89%

Figure 5.

Example with 1% prevalence of disease (test sensitivity=89%, test specificity=89%).
Test Results True Status (+)*True Status (-)*Totals
(+) 9 109 118
(-) 1 881 882

10 990 1000

*True Disease Status ("Gold Standard" Test Results)

  • Sensitivity: 89%
  • Specificity: 89%
  • True Prevalence: 1%
  • Apparent Prevalence: 12%
  • Positive Predictive Value: 8%
  • Negative Predictive Value: 100%

Table of contents

Using Disease Prevalence Estimates to Aid in Interpreting Diagnostic Tests for EPM

It may seem somewhat circular that prevalence must be accounted for in order to correctly interpret diagnostic test results: disease status must be known in order to estimate disease prevalence and yet, if the true disease status of the animals were known there would be no need to run the tests. However, it is not necessary to have exact knowledge of disease prevalence. Rough estimates of disease prevalence in similar animals can be used in order to make practical generalizations about the predictive value of diagnostic test results.

For example, it is possible to categorize horses into different risk groups for EPM (Figure 6). Approximately 50% of horses admitted to The OSU Veterinary Medical Center with neurologic problems are found to have detectable concentrations of antibodies to S. neurona in their CSF and are therefore diagnosed with EPM. Among horses in this population with detectable neurologic problems, the prevalence of EPM among horses that are seropositive is 67% (i.e., seropositive horses with neurologic disease are much more likely to have antibodies to S. neurona in their CSF than are seronegative animals with neurologic disease). Using information from post-mortem evaluation of horses with neurologic disease and available estimates of horse numbers, the overall population prevalence of clinical EPM has been estimated to be approximately 0.5-1% (Dr. David Granstrom, personal communication, 1997). It has been suggested that, in general, horses are unlikely to have antibody to S. neurona in their CSF without showing clinical signs.11 If this is true, then overall less than 1% of clinically normal horses would be expected to have detectable concentrations of S. neurona in their CSF.

When interpreting results of the Western blot analysis of CSF obtained in clinical situations, it is therefore possible to classify a patient using these three distinct categories and make generalized estimates of the predictive value of this diagnostic test for that individual animal (Figure 6). In other words, the credibility of information obtained from diagnostic tests can be weighted if the prevalence of disease in similar animals can be estimated.

If the horse is clinically normal the predictive value of a positive result from Western blot analysis of CSF is probably very low; if assumptions from this example are correct, the positive predictive value in clinically normal horses is 8% or lower (Figure 6). This means that only 8% of positive test results would be expected to correctly predict the presence of disease! However, if the clinician uses clinical skills to screen which horses will be tested using the Western blot, it is possible to use this test in situations that it is more likely to yield useful information. When the test is applied in horses presented to The Ohio State University Veterinary Medical Center with clinical neurologic disease, the prevalence of disease is expected to be approximately 50% and the predictive value of a positive test is approximately 89% (Figure 6). If the horse has detectable neurologic disease and is also seropositive the positive predictive value is even higher, approximately 94% (Figure 6).

Figure 6.

Positive Predictive Value for Western Blot Analysis of CSF among Horses with Different Clinical Presentations (Assuming 89% Sensitivity and 89% Specifivity).
Type of HorseEstimated True Prevalence of Horses with Antibodies to S. Neurona in CSF*Apparent Prevalence (Proportion of horses with positive test results)Positive Predictive Value
No Neurologic Abnormalities 1% 12% 8%
Neurologic Abnormalities Present 50% 50% 89%
Neurologic Abnormalities and Seropositive 67% 63% 94%

* Based upon the prevalence in horses presented to The Ohio State University Veterinary Medical Center. These values will be different in other populations depending upon the likelihood of exposure to S. neurona and other risk factors.

Table of contents

Conclusions

Further work is needed to corroborate the high sensitivity and specificity that has been reported for the Western blot assay when used to diagnose EPM. However, this test has been used extensively in the past five years for this purpose and available information suggests that it is highly sensitive and highly specific. It will not, however, be useful when used in horses where the disease prevalence is very low (Figure 6). Considering available information about the prevalence of disease, it is not recommended that this test be used in clinically normal horses to determine if individual animals have been exposed to S. neurona. This diagnostic test appears to be much more reliable when used in horses that have clinical signs which could be attributed to EPM (Figure 6).

Table of contents

Recommended Reading on the Clinical Epidemiology of Diagnostic Tests

Sackett DL, RB Haynes, Tugwell. Clinical Epidemiology: A Basic Science for Clinical Medicine, 2nd ed. 1991. Little, Brown, and Company, Boston/Toronto. 441 pp.

Table of contents

References

  1. Granstrom DE, Dubey JP, Davis SW, et al. Equine protozoal myeloencephalitis: antigen analysis of cultured Sarcocystis neurona merozoites. J Vet Diagn Invest, 1993;5:88-90.
  2. Granstrom DE, Dubey JP, Giles RC, et al. Equine protozoal myeloencephalitis: Biology and epidemiology. In: Nakajima H, Plowright W, eds. Proc VII Intern Conf Equine Infect Dis, Tokyo, Japan: R & W Publications Ltd., Newmarket, UK; 1994:109-111.
  3. Granstrom DE, Saville WJ. Equine Protozoal Myeloencephalitis. In: Reed SM, Bailey WM, eds. Equine Internal Medicine. Philadelphia, Pa.: WB Saunders Company; 1995:In Press.
  4. Reed SM, Granstrom DE. Equine protozoal encephalomyelitis. Proc Am Coll Vet Intern Med Forum. Washington, DC; 1993:591-592.
  5. Reed SM, Granstrom D, Rivas LJ, Saville WA, Moore BR, Mitten LA. Results of cerebrospinal fluid analysis in 119 horses testing positive to the western blot test on both serum and CSF to equine protozoal encephalomyelitis. Proc Am Assoc Equine Pract. Vancouver, BC; 1994:199.
  6. Fenger CK, Granstrom DE, Langemeier JL, et al. Identification of Opossums (Didelphis virginiana) as the putative definitive host of Sarcocystis neurona. J. Parasitol 1995;81:916-919.
  7. Dame JB, MacKay RJ, Yowell CA, Cutler TJ, Marsh A, Greiner EC. Sarcocystis falcatula from passerine and psittacine birds: Synonymy with Sarcocystis neurona, agent of Equine Protozoal Myeloencephalitis. J. Parasitol 1995;81:930-935.
  8. Saville WJ, Reed SM, Granstrom DE, et al. Prevalence of serum antibodies to Sarcocystis neurona in horses residing in Ohio. J Am Vet Med Assoc 1997;210:519-524.
  9. Blythe LL, Granstrom DE, Hansen DE, Walker LL, Bartlett J, Stamper S. Seroprevalence of antibodies to Sarcocystis neurona in horses residing in Oregon. J Am Vet Med Assoc 1997;210:525-527.
  10. Bentz BG, Granstrom D, Stamper S. Seroprevalence of antibodies to Sarcocystis neurona in horses residing in a county of southeastern Pennsylvania. J Am Vet Med Assoc 1996;210:517-518.
  11. Reed SM, Saville WJA. Equine Protozoal Encephalomyelitis. In: Zinninger SE, ed. Proc Am Assoc Equine Pract, Denver, CO; 1996:75-79.
  12. Miller MM, Bernard WV. Usefulness of cerebrospinal fluid indices and the polymerase chain reaction for Sarcocystis neurona in diagnosing equine protozoal myeloencephalitis. In: Zinninger SE, ed. Proc Am Assoc Equine Pract, Denver, CO; 1996:82-84.
  13. Andrews FM, Maddux JM, Faulk D. Total protein, albumin quotient, IgG and IgG index determinations for horse cerebrospinal fluid. P Vet Neuro 1991;1:197-204.
  14. Andrews FM, Provenza M. Differentiating neurologic diseases in the horse using albumin quotient and IgG index determination. In: DeNovo R, ed. Proc Am Coll Vet Intern Med Forum, 13 ed. Lake Buena Vista, Fl; 1995:600-603.
  15. Andrews FM, Granstrom DE, Provenza M. Differentiation of neurologic diseases in the horse by the use of albumin quotient and IgG index determinations. Proc Am Assoc Equine Pract, Lexington, KY; 1995:215-217.