Why do we put so much import on the MCAT?

Every year thousands of potential medical students spend a lot of time and money to study for the MCAT. Students spend a lot of money, for example,  $1,749 to take the Kaplan Complete MCAT Preparation and $1,000 for the Princeton Review. Why?? They spend this money preparing themselves to sit for this life changing examination. The test itself is actually comparatively inexpensive-- $235 dollars. 

The worst part is that the MCAT is probably not measuring any of the most important variables for our matriculating medical students. I have been on the admissions committee at two different medical schools.  Both were State supported medical schools with a strong commitment to graduating students interested in primary care, rural practice, and practice in underserved areas. But at the Admissions Committee level there is a serious lack of understanding of the importance of pre-matriculant noncognitive data and the variables used to select students to enter the school. Many of my colleagues (physicians and basic scientists) over the years have held the belief to some extent or another that higher pre-admission scores lead to better medical students which leads to better doctors.

Nothing could be further from the truth. In fact, for most of the variables there is very little correlation. And even more worrisome for many of the important characteristics of being a physician, there is an inverse relationship.  At this point, many of you are thinking, "Delzell is completely off his rocker!". In fact, one of my basic science colleagues said as much last year.  Well, that may be true, but I do have evidence to back up these statements.

Let's go back. Way back, to 1978. Harrison Gough, PhD, a psychologist at the university of California-Berkley, published one of the most fascinating studies (1) that I have seen in the medical educational literature. It is also one of the best written papers that I have ever read, and as an editor for a major medical journal I get to read a lot of manuscripts. Dr. Gough collected data on medical students from the University of California-San Francisco from 1951 to 1977. Wow! That is like the Framingham study of medical students. A longitudinal study of a medical school and its medical students.  This study reports on 1,195 UCSF students from 1972 to 1977. Data collected included MCAT scores, undergraduate GPA, and a measure to assess scientific preferences.  He created a composite index score by adding the "measures of scientific talent" (MCAT Science subtest and Undergraduate Science GPA) to Science Preference.

Science Preference is a fascinating concept that was developed by Goldstein (2) and modified by Dr. Gough. Students rated the three subjects from college that they liked best and the three that they liked least. The average score for the three least liked subjects was subtracted from the average score for the three subjects liked best. This gives an overall score, which was termed the students' preferences for science.  

The next step was to measure correlations between these measures and performance during medical school. He looked at grades in all four years of school. As you would expect, there was a significant correlation between the composite index and GPA in year 1 (Pearson Product- Moment correlation .34; p < 0.01) and year 2 (Pearson Product- Moment correlation .21; p < 0.01). But, there was no significant correlation with GPA in years 3 and 4. More importantly medical school faculty ratings of clinical competence and general competence were not significantly correlated (Pearson Product- Moment correlation .01; p = ns).

Ok, so maybe MCAT and GPA are not great at predicting things after the first two years but those years are important. Right???  Someone very smart once told me, "they would be really important if we were trying to make Step 1 passers, but we are not trying to make Step 1 passers, we are trying to make doctors."

Fortunately, Dr. Gough didn't stop there, there was another part to his study. He selected 70 students for an intensive study of personality at the UC- Berkley Institute of Personality Assessment and Research. The students were evaluated by 10-15 trained assessment staff members. Students were observed closely for an entire day and the staff members described them using a 300 item Adjective check list. These descriptors were then correlated with the four previously evaluated science predictors for each student.  

There was no statistical correlation between MCAT scores and personality descriptors. But, Science GPA was correlated with "painstaking"(r =.26) and "silent"( .26). There were negative correlations with adjectives such as "poised" (r = - .32), "self-controlled" (- .30), and "interests wide" (- .27).  When compared to the composite index there were also several significant correlations, such as "awkward" (r = .27), "cautious" ( .23), and "conservative" ( .27).  The composite index was negatively related to several descriptors, such as "stable" (r = - .28) and "relaxed" (- .28).  

Each of the students was also judged by all of the observers. The reliability of their judgment is striking. The inter-rater reliability was greater than .80.  The students with higher composite index were rated lower in their ability to communicate (r = - .28), breadth of interests (r= - .35), self-acceptance (r = - .26), and verbal fluency (r= -.29). All of these correlations were significant. By the way, the MCAT by itself also had a negative correlation with every measure, with r values between - .11 and - .20.  Dr Gough's conclusion was that scientifically oriented students were "less adept in interpersonal skills, less articulate, narrower in interests, and less adaptable than their fellows.". 

Wow! That is amazing. Why didn't anybody tell about this when I joined the admissions committee? For my next blog, I am going to write about some important characteristics of future physicians and how we can better select students that will have those characteristics.

References

(1) Gough HG. Some Predictive Implications of Premedical Scientific Competence and Preferences. J Med Educ  1978; 53: 291-300.

(2) Goldschmid ML.  Prediction of College Major by Personality Tests. J Counseling Psychol  1967; 14: 302-308.