Appropriate Outcome Measures for Infantile Nystagmus Therapies:

“Science-based” not “Evidence-based” Studies

 

Louis F. Dell’Osso, PhD

 

 

OMLAB Editorial #010410

Written: 1/4/10; Placed on Web Page: 1/4/10; Last Modified: 9/24/10

Downloaded from: OMLAB.ORG-- Click here for the PDF.

 

Send questions, comments, and suggestions to: lfd@case.edu

 

 

The past forty years of research into infantile nystagmus syndrome (INS, aka “congenital nystagmus”) (1) have resulted in basic changes in our understanding of its underlying mechanism, provided new and more powerful methods of eye-movement-based analysis, laid to rest some simplistic ophthalmological misconceptions, and resulted in a paradigm shift in our approach to extraocular muscle (EOM) surgical and non-surgical nystagmus therapies. Unfortunately, incorporating these new approaches and analysis measures into studies of therapeutic efficacy in INS has met with resistance and suggestions by funding agencies to replace the best scientific outcome measures with less effective, inadequate measures (see below, interchange at a scientific meeting).

 

This editorial will: 1) briefly summarize what we have learned from predominantly classical studies of INS; 2) discuss the implications for studies of potential therapies; 3) identify current problems facing scientists; and 4) suggest a solution. Although each of these topics could easily be the subject of an individual editorial, because they are interrelated, they have been combined herein.

 

Review of the Science

Early in the ocular motor study of INS, it became obvious that INS was the same ocular motor disorder despite differences in the associated visual sensory conditions of the patient population. All patients shared the same pathognomonic waveforms (2) and characteristic changes with gaze angle, vergence angle, fixating eye, etc. that were uncorrelated to the presence or absence of any sensory deficits. The commonality of INS waveforms suggested the same underlying ocular motor mechanism(s) and the same therapies to improve motor function in all INS patients regardless of the presence or absence of sensory deficits. This important point is often overlooked in studies that artificially separate patient populations based on either their associated sensory deficits or, more recently, some genetic marker whose relationship to INS is as yet unknown.

 

In the first ocular motor study of the effects of EOM surgery, we noted that the surgery produced beneficial, unanticipated, and unexplained effects (3). In addition to shifting an eccentric INS null region to primary position, the four-muscle Kestenbaum procedure (i.e., bilateral horizontal rectus recessions and resections) also improved the waveforms and, more importantly, did so over a broader range of gaze angles. This is best expressed in terms of the eXpanded Nystagmus Acuity Function (NAFX), which is linearly proportional to the best-corrected visual acuity possible with that waveform (i.e., presuming no afferent deficits) (4). Four-muscle EOM surgery improves the peak NAFX and broadens the range of gaze angles with high NAFX values (i.e., the longest foveation domain, or LFD) (5). The 1979 Kestenbaum study (3) produced the hypothesis that these therapeutic improvements would result from simply tenotomizing, and reattaching at their original insertions, the four EOM in the plane of the INS—the tenotomy and reattachment (T&R) procedure (6).

 

Because the NAFX was specifically designed to account for only the motor factors that reduce the quality of target foveation, it uniquely separates out the motor component of measured visual acuity from any sensory component that may be present. This important quality enables, for the first time, pre-therapy estimation of post-therapy measured peak visual acuity and the range of gaze angles with higher acuity. It is important, and not universally appreciated, that the measured improvements in INS are produced equally in patients with and without associated sensory deficits. This new methodology (7,8) should have an important role in establishing more scientifically based, and ethically sound, inclusion criteria for future studies of therapeutic efficacy (see below, The Solution).

 

The T&R surgical procedure is a peripheral therapy that, in addition to INS, also improves the waveforms of acquired pendular nystagmus (in multiple sclerosis) and downbeat nystagmus, thereby reducing oscillopsia (9,10). The procedure reduces small-signal gain without altering saccadic gain or inducing undesirable plasticity changes (11). Similar improvements result from either the use of base-out prisms (to induce convergence) or soft contact lenses (12). In addition to the two static therapeutic improvements in visual function (higher peak NAFX and broader LFD), four-muscle EOM surgery shortens the longer-than-normal target acquisition time (Lt) in INS (13,14).

 

Finally, we must consider outcome measures. INS is a motor disorder and optical (prisms, soft contact lenses), surgical, and pharmaceutical therapies are all aimed at altering motor function (i.e., improving foveation quality of INS waveforms). The most clinically relevant measure of improved motor function is the direct measure of foveation improvement via the NAFX applied to eye-movement data (15). The primary outcome measure of strabismus surgery (aimed at eye alignment) is the direct measurement of post-operative alignment, not the indirect, hoped for medical outcome of improved stereopsis. It follows therefore, that the primary outcome measure of nystagmus surgery (aimed at improved foveation quality) is the direct measurement of foveation quality via the NAFX, not the indirect, hoped for medical outcome of improved peak visual acuity (which is not even the most important improvement in visual function achieved by INS therapy). The greatest improvement in visual function is broadening of the high-acuity range of gaze angles. In the past, that has been the reason for the many reports of patient satisfaction (they can “see more”) in cases where no improvement in peak visual acuity was measured.

 

The Problem

Today’s scientists find themselves in a quandary created by the attempt to apply the methodology of “evidence-based” medicine to research into the mechanisms of ocular motility and the disorders that result in nystagmus and other ocular motor dysfunction. Whether writing proposals for scientific study or assessing the mechanisms or efficacy of a proposed therapeutic approach, they are being pressured to adopt the “drug-therapy” approach to their investigations with all the supposed protections against bias, intentional or accidental, of such protocols. That, in effect, both introduces the possibilities of scientific error and (by the intentional “masking” of the data) deprives the world of the most important factor in the scientific research—the unfettered and inquisitive mind of the scientist. It also introduces some serious conflicts of interest. Both government committees and pharmaceutical representatives (for industry-funded, drug-therapy studies) have assumed, indeed insisted upon, a greater role in the design of the study, the methods of analysis, the types of analysis, and even the discussion and conclusions to be drawn from the study. This is not only a dangerous but also a scientifically problematic intrusion into the domain of the principal investigators who, although remaining open to scientifically sound suggestions, must retain freedom from intrusive pressure when making the final decisions in all phases of their proposed research.

 

An example using a proposed study of INS should clarify the issue. At a recent international scientific meeting about nystagmus research, one speaker requested a discussion of what should be the primary outcome measure of a study of therapeutic efficacy in INS. A scientist in attendance suggested that both the direct ocular motor and the indirect clinical measures should be included. The speaker then responded that he was required by the funding agency to specify visual acuity as the primary outcome measure. The (astonished) scientist responded that the primary outcome measure, or measures, must be the direct ocular motor outcome measures since both surgical and pharmaceutical INS therapies affected the INS waveforms, not the afferent visual sensory system. A second scientist opined that this was too “rigid” and that visual acuity was a more “medically acceptable” measure for funding agencies because it was “functional.” The first scientist responded that the funding agencies must then be educated about the science involved. A third scientist supported visual acuity and called for a vote on whether it should be the primary outcome measure of INS therapy; she and several others in attendance raised their hands, which prompted her to declare “victory” for visual acuity. The first scientist pointed out to that scientist and to those who raised their hands that, “Science is not a democracy. We don’t vote on scientific truth; we test for it.” He added that not too long ago there was unanimous agreement among ophthalmologists (including pediatric- and neuro-) that the T&R procedure could not improve INS and nearly unanimous agreement by ocular motor basic scientists that the hypothesized proprioceptive mechanism had no effect on eye movements. Clearly, if the proposer of both the procedure and the hypothesized mechanism had been influenced by such opinions/votes, it would have prevented the dramatic progress of the past 10 years in INS research and therapy—fortunately, he was not. There was no response to those relevant reminders, prematurely truncating what began as a scientific discussion of the benefits of specific ocular motor outcome measures. When faced with the universal doubts of one’s peers, it is comforting to recall the words of Hermann von Helmholtz upon returning from a lecture he had just given, “Not a single scientist in the meeting believed a word of what I said. Now I know I am right.”

 

That meeting witnessed representatives of governmental funding agencies and principal investigators who: suspended their otherwise good scientific judgment; failed, or were unable, to provide scientific arguments in support of their position; and tried to justify with a show of hands, a decision about primary outcome measures that was contraindicated by published scientific findings (see above, Review of the Science).

 

The Solution

The “evidence-based” approach was developed to assess the efficacy of drug therapies and is a reasonable approach to do so. My single experience with a “clinical trial” where the data were taken in one lab by one group of scientists and then, after masking it, sent to me for analysis was one of boredom, exasperation, and the absence of the joy of any new discovery. Based on my previous work, the outcomes were predictable and, because of the data masking, I was prevented from making any new or insightful observations. Because of that intellectually barren experience and my discovery of errors made in data acquisition, masking accuracy, and patient inclusion, I decided to never again agree take part in that type of science. In my opinion, the “blind-monkey” approach of “evidence-based” medical studies is both inefficient and problematic as it is currently being implemented. To the insightful scientist, it is anathema. Prevented by overzealous protocols designed to separate scientists from their data, the most important factor in new discovery—the key, insightful, often serendipitous observation by the prepared mind—is lost. In its place are statistical manipulations of often-massive amounts of possibly poor-quality data collected and analyzed by individuals, usually in different locations, who may not possess expertise in the specific disorder under study. They cannot function as insightful scientists because they are blinded to the source and type of data, and the details of how it was actually (protocols not withstanding) taken in diverse settings by different collaborators with their individual interpretations of exactly how to implement the study’s protocol. That is not the way science should be conducted and brings to mind the old computer adage, “garbage in, garbage out.”

 

Furthermore, the study of basic physiological mechanisms, their dysfunctions, and therapeutic approaches to the amelioration of the latter, is best investigated by scientists with both the proper physiological/engineering training and the ability to use their scientific insight to analyze data gathered by them, in their labs, and under their control. Those whose aims or expertise is in statistical manipulation of data made noisy by the myriad of unknown and unwanted confounding factors introduced by the very design of multi-person and multi-lab data collection or, even worse, multi-site analyses, are much less likely to make the meaningful, often-unexpected discoveries that drive scientific advances. The history of science favors both the insightful scientist and his often-serendipitous observations.

 

What will be the effects on an “evidence-based,” therapeutic-efficacy study in INS where the primary outcome measure was limited to visual acuity?

1. There will be false negatives. Patients with high peak NAFX values but low LFD values will not show improvements in peak visual acuity. However, they will show important (to visual function) improvements in LFD and in Lt that may go unmeasured if the principal investigator limits outcome measures to those dictated by those with interests that may conflict with scientific discovery.
2. There will be excessive negatives. Patients, who should have been excluded based on simultaneously high NAFX and LFD values, will show no improvements in any of the measures of visual function.

 

What will be the effects on the governmental or pharmaceutical funding agency that insisted on the primary outcome measure being limited to visual acuity?

Funds and resources will be wasted by a study that, by design, was incapable of revealing the true value of the therapy but instead, will underestimate it.

 

Can the true therapeutic outcomes of such a study be resurrected?

Fortunately, yes, as long as pre- and post-therapy ocular motor data were also collected despite the misguided insistence of the funding body on using only visual acuity (see below).

 

How should a “science-based” study of INS therapy be conducted?

1.     The NAFX should be used pre-therapy to exclude patients with simultaneously high NAFX and LFD values from the study, because they could not benefit from the therapy.

2.     The NAFX should be used pre-therapy to determine which of the primary ocular motor outcome measures could improve in each patient and by how much.

3.     The NAFX, LFD, and Lt should be used to better define “success” for each patient, as “success” will differ idiosyncratically depending on the pre-therapy characteristics of their INS. These expectations for success should be carefully explained to each patient so that any patient surveys are not unduly confounded by unrealistic patient expectations.

4.     Pre- and post-therapy clinical measures, not only peak visual acuity but also visual acuities across a range of gaze angles, should be included and correlated with the NAFX and LFD primary outcome values and estimates.

Unlike the evidence-based protocols, where all INS patients who fit a clinical profile based only on unrelated associated visual deficits are entered, a science-based protocol would determine and exclude from the study those patients who, by virtue of their ocular motor profile, could not benefit from an ocular motor therapy; that is scientifically more valid and ethically more palatable.

 

Pharmaceutical companies, and governmental funding agencies, have excessively inserted themselves into areas that should be the prerogative of the principal investigators and their colleagues. Because of the non-scientific and possibly conflicting interests of these groups, not only is the science suspect but also, as in this example, the resulting study could not fully reveal the effectiveness of the therapy—surely not what the funding agencies intended when they imposed an inferior primary outcome measure upon the scientists responsible for the study. Institutions that oversee scientific research must insulate their scientists from any pressure from funding agencies and ensure that: 1) the science is done in accordance with in-house scientific committees; 2) the analyses are conducted by the investigators with neither interference from, nor consultation with, the funding agencies; and 3) the results published with peer-review and no prior notification to, or consultation with, the funding agencies. These minimal safeguards are necessary to avoid either the appearance of, or actual, conflicts of interest.

 

We have a responsibility as individual scientists to design all aspects of our research to ensure the scientific validity of our research. Toward that end:

1. We must insist on defining both primary and secondary outcome measures as well as the exclusion criteria based on current scientific knowledge, not the desires of funding agencies;
2. We must refuse to be parties to the second-rate science that is the likely outcome of research directed or defined by any one other than the responsible scientist(s).

The integrity of both the science and the scientist demands no less.

                                                                                                                                                                      

Acknowledgement

This work was supported in part by the Office of Research and Development, Medical Research Service, Department of Veterans Affairs.

 

References

1.              CEMAS_Working_Group. A National Eye Institute Sponsored Workshop and Publication on The Classification of Eye Movement Abnormalities and Strabismus (CEMAS). In The National Eye Institute Publications (www.nei.nih.gov/news/statements/cemas.pdf). 2001, National Institutes of Health, National Eye Institute: Bethesda, MD.

 

2.              Dell'Osso LF, Daroff RB. Congenital nystagmus waveforms and foveation strategy. Doc Ophthalmol 1975; 39:155-82.

 

3.              Dell'Osso LF, Flynn JT. Congenital nystagmus surgery: a quantitative evaluation of the effects. Arch Ophthalmol 1979; 97:462-9.

 

4.              Dell'Osso LF, Jacobs JB. An expanded nystagmus acuity function: intra- and intersubject prediction of best-corrected visual acuity. Doc Ophthalmol 2002; 104:249-76.

 

5.              Wang Z, Dell'Osso LF, Jacobs JB, Burnstine RA, Tomsak RL. Effects of tenotomy on patients with infantile nystagmus syndrome: foveation improvement over a broadened visual field. JAAPOS 2006; 10:552-60.

 

6.              Dell'Osso LF. Extraocular muscle tenotomy, dissection, and suture: A hypothetical therapy for congenital nystagmus. J Pediatr Ophthalmol Strab 1998; 35:232-3.

 

7.              Wang ZI, Dell'Osso LF. A review of the tenotomy nystagmus surgery: origin, mechanism, and general efficacy. Neuro-Ophthalmol 2007; 31:157-65.

 

8.              Dell'Osso LF. New treatments for infantile and other forms of nystagmus. In: Leigh RJ, Devereaux MW, eds. Advances in Understanding Mechanisms and Treatment of Congenital Forms of Nystagmus. New York: Oxford University Press, 2008; 87-98.

 

9.              Tomsak RL, Dell'Osso LF, Rucker JC, Leigh RJ, Bienfang DC, Jacobs JB. Treatment of acquired pendular nystagmus from multiple sclerosis with eye muscle surgery followed by oral memantine. DJO 2005; 11(4):1-11.

 

10.           Wang ZI, Dell'Osso LF, Tomsak RL, Jacobs JB. Combining recessions (nystagmus and strabismus) with tenotomy improved visual function and decreased oscillopsia and diplopia in acquired downbeat nystagmus and in horizontal infantile nystagmus syndrome. J AAPOS 2007; 11:135-41.

 

11.           Wang Z, Dell'Osso LF, Zhang Z, Leigh RJ, Jacobs JB. Tenotomy does not affect saccadic velocities: Support for the "small-signal" gain hypothesis. Vision Res 2006; 46:2259-67.

 

12.           Taibbi G, Wang ZI, Dell'Osso LF. Infantile nystagmus syndrome: broadening the high-foveation-quality field with contact lenses. Clin Ophthalmol 2008; 2:585-90.

 

13.           Wang ZI, Dell'Osso LF. Being "slow to see" is a dynamic visual function consequence of infantile nystagmus syndrome: Model predictions and patient data identify stimulus timing as its cause. Vision Res 2007; 47(11):1550-60.

 

14.           Wang ZI, Dell'Osso LF. Tenotomy procedure alleviates the "slow to see" phenomenon in infantile nystagmus syndrome: model prediction and patient data. Vision Res 2008; 48:1409-19.

 

15.           Dell'Osso LF. Nystagmus therapies: types, sites, and measures. OMLAB Report 2005; #1119005:1-4. http://www.omlab.org//Teaching/teaching.html

 

 

 

Citation

Although the information contained in this paper and its downloading are free, please acknowledge its source by citing the paper as follows:

 

Dell’Osso, L.F.: Appropriate Outcome Measures for Infantile Nystagmus Therapies: “Science-based” not “Evidence-based” Studies. OMLAB Editorial #010410, 1-6, 2010. http://www.omlab.org//Editorial/editorial.html

Note added

In reviewing the requirements for manuscript submission to the journal, Ophthalmology, I was delighted to find the following:
The title page must include the statement, “The sponsor or funding organization had no role in the design or conduct of this research.” This requirement by the Editors of Ophthalmology reflects exactly my opinions (expressed above) regarding the exclusion of any influence or coercion by the sponsors or funding agencies on the scientific choices of the principal investigator.

L.F. Dell’Osso (9/24/10)