Loading Library Hours...

Caution and Controversy

Rankings Links & Information

College and University Rankings: Caution and Controversy

Many people question the use and usefulness of rankings services such as those found on this site. Without a doubt, knowledge of how rankings are constructed, leavened with a dash of skepticism, can go a long way to ensuring rankings are used appropriately. In addition to the articles listed in our rankings bibliography, many additional interesting resources may be found on the web about using rankings effectively. Among these are “Ignore College Rankings — Become an Educated Consumer” by Shirley Levin [ 1].

Two organizations which do not rank schools, but rather give comparative information about schools, also provide discussion of how to assess the quality of schools and ranking services. The College Board offers “Rankings & Ratings” [ 2], an interesting discussion of the impact of rankings, while Peterson’s, via the Association of American Colleges and Universities, presents “Considering College Quality” [ 3], a discussion of assessing institutional quality.

Just as the general public is concerned about rankings, academicians increasingly debate the importance and validity of college rankings. Two articles from the Chronicle of Higher Education, the premier news publication for academe, offer excellent perspectives on the subject. “Changes in Annual College Guides Fail to Quell Criticisms on Their Validity” [ 4] and “College Rankings Will Never Die” [ 5] provide insight into the ongoing controversy. In 2006, an international group of educators, higher-education experts, and publishers developed 16 principles of good practice, the Berlin Principles on Ranking of Higher Education Institutions, which are meant to serve as guidelines for groups that produce rankings [ 6].

Many universities, including highly ranked ones, question both the data and the processes used by some of the ranking services. Of special concern are the aspects of the rankings which deal with the difficult-to-measure concept of institutional reputation. For interesting insights into the topic of university reputations, read “Building Reputations: How the Game is Played” by William Sharp of Columbia University [ 7]. The Top American Research Universities Report [ 8] uses a variety of factors including funding, awards, and research and development as the basis for rankings to evaluate the best universities.

The news magazine U.S. News & World Report has long been the target of criticism for its annual ranking of colleges and universities. In 1996, Stanford University President Gerhard Casper kicked off a wave of controversy over the U.S. News rankings with his letter to James Fallows, the magazine’s editor [ 9]. Casper followed this up with “An Alternative to the U.S. News and World Report College Survey,” a press release from Stanford University [ 10]. Since then numerous articles critiquing the U.S. News and World Report rankings, as well as articles recommending changes to enhance those rankings have appeared.

The online publication Slate ran a story in August of 1999 questioning U.S. News‘ motives and methods. In an article titled “Cooking the School Books: How U.S. News cheats in picking its best American colleges,” author Bruce Gottlieb put forth many criticisms of the rankings and accused U.S. News of changing its methodology from year to year in order to shake up the top ranked schools and keep the public’s interest [ 11]. Gottlieb stated, “A successful feature like this [i.e., the U.S. News rankings] requires surprise, which means volatility.  Nobody’s going to pay much attention if it’s Harvard, Yale, and Princeton again and again, year after year.” Two U.S. News editors, Peter Cary and Brian Duffy, took to the pages of Slate to respond with “Dissension in the Rankings: U.S. News responds to Slate’s best colleges story” [ 12].

In recent years, rankings have also expanded to assess the value of higher education to income as reported in articles such as “The Best Colleges for Making Money.” [13]  Interests of specific population groups can be seen in rankings offered for the mobility impaired, ethnic, and racial groups (see the list of general and undergraduate rankings sources). Additionally, an increased interest in comparative worldwide rankings of universities has led to rankings services such as the Academic Ranking of World Universities and others that can be found on our international rankings page.

In February 1998, a call for an end to rankings was released by the Association of American Law Schools (AALS) [ 14]. The AALS called the U.S. News & World Report law school rankings “misleading and dangerous,” and released a study questioning the validity of the rankings [ 15]. In 2005, an article “Outranked and Underrated,” discusses flaws in rankings systems in general in regard to law programs, and suggests alternatives and ways to effectively consider existing rankings [ 16].

Business school rankings have also come under scrutiny. “What’s Wrong With MBA Ranking Surveys?” is the title of an article by Martin Schatz, Ph.D [ 17]. Dr. Schatz points out some of the flaws of the major business school rankings and notes that no one program can possibly be predetermined as the best for any individual.

Publishers have been quick to answer with information about their ranking procedures. With the release of each of its undergraduate rankings, U.S. News & World Report provides further explanation of its system. The site’s “About the Rankings/Methodology” offers information on why and how they rank as well as how one should use the rankings [ 18]. They also offer “About the Best Graduate Schools Rankings” [ 19] for their graduate school site along with “Frequently Asked Questions: Graduate Schools Rankings” [ 20].

Furthermore, U.S. News presents the article “Universities Use Rankings Too,” which points out that colleges often rank students but seem to prefer not to be ranked themselves [ 21].

Efforts are underway by various concerned parties to introduce some uniformity into the debate over evaluating colleges and universities. A few major publishers are working with the educational community on the Common Data Set Initiative, an attempt to standardize institutional data gathering and reporting.

Overall, there is a bit of a “can’t live with ’em, can’t live without ’em” aspect to college rankings. Colleges routinely disparage rankings but are quick to trumpet their high standing and paste the U.S. News best college graphic on their web site. Schools are even known to tinker with their admissions policies, alumni files, and other “ranking factors” in order to maintain or boost their U.S. News rating. (For more on this, see the Machung and Monks entries on our rankings bibliography.) Like them or leave them, colleges will certainly devote no less attention to rankings in the future. Likewise, students and parents will continue to seek out the best school and continue to consult the wide variety of available college rankings. Being aware of the controversy and “game playing” involved in college rankings can only help interested parties make better use of rankings in their decision making processes.

 

Bibliography 

  1. Levine, Shirley. (December, 1997). “Ignore College Rankings: Become an Educated Consumer”.
  2. College Board (Summer 1997). Rankings and ratings. Enrollment Management Review 13(4).
  3. Boyer, Ernest L. (2001). Considering College Quality. Greater Expectations National Panel, Briefing Paper #17. Association of American Colleges and Universities.
  4. Crissey, Michael (September 5, 1997). “Changes in annual college guides fail to quell criticisms on their validity.”   Chronicle of Higher Education.
  5. Carey, Kevin. (March 19, 2009).  “College Rankings Will Never Die.”  The Chronicle Review. Chronicle of Higher Education.
  6. Bollag, Burton. (June 1, 2006). “Group Endorses Principles for Ranking of Higher-Education Institutions.” The Chronicle of Higher Education.
  7. Sharp, William. (1995). Building reputations: how the game is played.
  8. Lombardi, John V., Craig, Diane D., and Capaldi, Elizabeth D. The Top American Research Universities Report.
  9. Casper, Gerhard (Sept. 23, 1996). Letter to James Fallows.
  10. Casper, Gerhard (April 18, 1997). An alternative to the US News and World Report College survey.
  11. Gottlieb, Bruce ( August 31, 1999). Cooking the school books: How US News cheats in picking its best American colleges. Slate.
  12. Carey, Peter and Duffy, Brian (September 7, 1999). Dissension in the rankings: US News responds to Slate’s “Best Colleges” story. Slate.
  13. Parmar, Neil. The Best Colleges for Making Money. SmartMoney. December 16, 2008.
  14. Monk, Carl C. (February 18, 1998). Association of American Law Schools calls on US News & World Report to stop ranking Law Schools; Study challenges validity of magazine’s system. AALS News release.
  15. Klein, Stephen P. and Hamilton, Laura. (February 18, 1998). The validity of the US News & World Report rankings of ABA Law Schools.
  16. Bradburn, Norman. “Outranked and Underrated”. Legal Affairs. Nov/Dec 2005.
  17. Schatz, Martin (1993). “What’s Wrong With MBA Ranking Surveys?” Management Research News 16 (7), p.15-18. (Only available to UIUC affiliates.)
  18. Anonymous (2013). About the Rankings/Methodology. US News & World Report . (updated annually)
  19. Anonymous (2014). About the Rankings. US News & World Report . (updated annually)
  20. Anonymous (2014). Frequently Asked Questions: Grad Rankings. US News & World Report . (updated annually)
  21. Lemann, Nicholas. (Aug 31, 1998): “Universities Use Rankings Too.” US News & World Report, v125 n8: p81. (Only available to UIUC affiliates.)