Do not rank!

Do not rank!
http://www.gter.net 2002-7-2 11:27:31 stanford.edu 作者:Gerhard Casper

stanford官员对usnews大学排名的看法，有借鉴意义. --- 柯南转贴

http://www.stanford.edu/dept/pres-
provost/president/speeches/961206gcfallow.html

I know that this is from 1996 but the ideas mentioned are relevant and in
addition, the rankings are still just as screwed up

The following was a private letter from Gerhard Casper, president of
Stanford University, to James Fallows, editor of U.S. News & World Report.
With the permission of both, it since has entered the public domain.

STANFORD UNIVERSITY
OFFICE OF THE PRESIDENT

GERHARD CASPER

                                              September 23, 1996

Mr. James Fallows
Editor
U.S. News & World Report
2400 N Street NW
Washington, DC 20037

Dear Mr. Fallows:

      I appreciate that, as the new editor of U.S. News & World Report,
you  have much to do at this moment. However, it is precisely because you
are the  new editor that I write to you, personally.

      I emphasize you, because of your demonstrated willingness to examine
journalism in the same way that journalism examines all other facets of
society. And I say personally because my letter is for your consideration,
and  not a letter to the editor for publication.

      My timing also is related to the recent appearance of the annual
U.S.  News "America's Best Colleges" rankings. As the president of a
university that  is among the top-ranked universities, I hope I have the
standing to persuade  you that much about these rankings - particularly
their specious formulas and  spurious precision - is utterly misleading. I
wish I could forego this letter  since, after all, the rankings are only
another newspaper story. Alas, alumni,  foreign newspapers, and many others
do not bring a sense of perspective to the  matter.

      I am extremely skeptical that the quality of a university - any more
than  the quality of a magazine - can be measured statistically. However,
even if it  can, the producers of the U.S. News rankings remain far from
discovering the  method. Let me offer as prima facie evidence two great
public universities: the  University of Michigan-Ann Arbor and the
University of California-Berkeley.
      These clearly are among the very best universities in America - one
could make  a strong argument for either in the top half-dozen. Yet, in the
last three  years, the U.S. News formula has assigned them ranks that lead
many readers to  infer that they are second rate: Michigan 21-24-24, and
Berkeley 23-26-27.

      Such movement itself - while perhaps good for generating attention
and  sales - corrodes the credibility of these rankings and your magazine
itself.  Universities change very slowly - in many ways more slowly than
even I would  like. Yet, the people behind the U.S. News rankings lead
readers to believe  either that university quality pops up and down like
politicians in polls, or  that last year's rankings were wrong but this
year's are right (until, of  course, next year's prove them wrong). What
else is one to make of Harvard's  being #1 one year and #3 the next, or
Northwestern's leaping in a single bound  from #13 to #9? And it is not
just this year. Could Johns Hopkins be the 22nd  best national university
two years ago, the 10th best last year, and the 15th  best this year? Which
is correct, that Columbia is #9 (two years ago), #15  (last year) or #11
(this year)?

      Knowing that universities - and, in most cases, the statistics they
submit - change little from one year to the next, I can only conclude that
what  are changing are the formulas the magazine's number massagers employ.
And,  indeed, there is marked evidence of that this year.

      In the category "Faculty resources," even though few of us had
significant changes in our faculty or student numbers, our class sizes, or
our  finances, the rankings' producers created a mad scramble in rank
order, for  example:

Down Last year This year     Up Last year This year
Harvard #1 #11   MIT #6 #2
Stanford  3  15   Duke 13  4
Brown 12  22   Yale 10  6
Johns Hopkins 15  19
Dartmouth 18  24

      One component of this category, "Student/faculty ratio," changed
equally  sharply, and not just in rank order but in what the magazine has
presented as  absolute numbers. Again, this is with very little change in
our student or  faculty counts:

Worse Last year This year    Better Last year This year
Johns Hopkins  7/1 14/1   Chicago 13/1 7/1
Harvard 11/1 12/1   Penn 11/1 6/1
Stanford 12/1 13/1   Yale 11/1 9/1
Duke 12/1 14/1

      Then there is "Financial resources," where Stanford dropped from #6
to  #9, Harvard from #5 to #7. Our resources did not fall; did other
institutions'  rise so sharply?

      I infer that, in each case, the formulas were simply changed, with
notification to no one, not even your readers, who are left to assume that
some  schools have suddenly soared, others precipitously plummeted.

      One place where a change was made openly was, perhaps, the most
openly  absurd. This is the new category "Value added." I quote the
magazine:

Researchers have long sought ways to measure the educational value added
by  individual colleges. We believe that we have created such an
indicator.
Developed in consultation with academic experts, it focuses on the
difference  between a school's predicted graduation rate - based upon the
median or average  SAT or ACT scores of its students and its educational
expenditures per student -  and its actual graduation rate.
      This passage is correct that such a measure has long been sought.
However, like the Holy Grail, no one has found it, certainly not the "we"
of  this passage. The method employed here is, indeed, the apotheosis of
the errors  of the creators of these ratings: valid questions are answered
with invalid  formulas and numbers.

      Let me examine an example in "Value added": The California Institute
of  Technology offers a rigorous and demanding curriculum that undeniably
adds  great value to its students. Yet, Caltech is crucified for having
a "predicted"  graduation rate of 99% and an actual graduation rate of 85%.
Did it ever occur  to the people who created this "measure" that many
students do not graduate  from Caltech precisely because they find Caltech
too rigorous and demanding -  that is, adding too much value - for them?
Caltech could easily meet  the "predicted" graduation rate of 99% by
offering a cream-puff curriculum and  automatic A's. Would that be adding
value? How can the people who came up with  this formula defend graduation
rate as a measure of value added? And even if  they could, precisely how do
they manage to combine test scores and "education  expenditures" - itself a
suspect statistic - to predict a graduation rate?

      Were U.S. News, under your leadership, to walk away from these
misleading  rankings, it would be a powerful display of common sense. I
fear, however, that  these rankings and their byproducts have become too
attention-catching for that  to happen.

      Could there not, though, at least be a move toward greater honesty
with,  and service to, your readers by moving away from the false
precision? Could you  not do away with rank ordering and overall scores,
thus admitting that the  method is not nearly that precise and that the
difference between #1 and #2 -  indeed, between #1 and #10 - may be
statistically insignificant? Could you not,  instead of tinkering
to "perfect" the weightings and formulas, question the  basic premise?
Could you not admit that quality may not be truly quantifiable,  and that
some of the data you use are not even truly available (e.g., many high
schools do not report whether their graduates are in the top 10% of their
class)?

      Parents are confused and looking for guidance on the best choice for
their particular child and the best investment of their hard-earned money.
Your  demonstrated record gives me hope that you can begin to lead the way
away from  football-ranking mentality and toward helping to inform, rather
than mislead,  your readers.

                                              Sincerely,

                                              Gerhard Casper
　

>>打印此页<<

>>关闭窗口<<