The Problem with Ranking Universities (League Tables) - Sixty Symbols
TLDRThe video discusses the flawed nature of university league tables, criticizing their use of pseudo-statistics and the impact they have on higher education decisions. It highlights the vested interests of media organizations in creating volatile rankings to sell newspapers and the fundamental issue of trying to rank institutions that are fundamentally unratable. The speaker urges students to approach league tables with skepticism and not to base their university choices solely on these rankings.
Takeaways
- 📊 League tables, whether UK or international, are considered 'nonsense' by the speaker due to their pseudo-statistical nature.
- 🧐 The speaker emphasizes the frustration with how much fate is tied to these tables, which are seen as unsatisfying for scientists.
- 🤔 The problem with league tables is the reduction of complex and varied data into a single score, losing the nuances and context of the data.
- 🏫 Organizations that create these tables, often newspapers, have a vested interest in volatility to sell papers, which affects the reliability of the rankings.
- 🔄 The churn and constant change in league tables are driven by the need for new headlines and sales, rather than stability and accuracy.
- 📉 The fundamental issue is the attempt to rank something inherently unrankable, such as the 'best' university, which is subjective and depends on what is being measured.
- 🏅 The Research Excellence Framework (REF) is highlighted as an example of a flawed system that ranks universities based on questionable criteria.
- 💡 The speaker suggests that league tables should be approached with skepticism, like a scientist, considering the reliability and context of the numbers presented.
- 📈 There is a call for students to make decisions based on more than just league table positions, considering the vested interests and flawed methodologies behind them.
- 🌟 The importance of considering individual university experiences, such as a welcoming atmosphere, over just numerical rankings is emphasized.
- 🚫 The speaker advocates for universities to stop using league tables in their publicity and to recognize their flaws, potentially by signing a declaration against their use.
Q & A
What is the main issue with university league tables according to the speaker?
-The main issue with university league tables is that they are considered 'nonsense' and 'pseudo statistical faux quantitative nonsense', as they oversimplify and misrepresent the complex nature of universities by reducing them to a single ranking number.
Why do newspapers produce league tables every year?
-Newspapers produce league tables every year to sell newspapers. They have a vested interest in volatility, as a changing league table generates news and attracts readers, such as worried parents or prospective students.
What is the fundamental problem with trying to rank universities?
-The fundamental problem is that it's an attempt to order something that is fundamentally unorderable. The concept of the 'best' university is subjective and depends on what criteria one is trying to measure.
What is the Research Excellence Framework and how does it affect universities?
-The Research Excellence Framework (REF) is a system used in the UK to assess the quality of research in higher education institutions. It affects universities by determining how government funding is allocated among different departments, making it a significant factor in their financial support.
How does the speaker describe the error in the ranking of university departments based on the REF scores?
-The speaker describes the error as significant, pointing out that a difference of 0.01 in scores can result in a significant change in ranking, which does not reflect a meaningful difference in quality or performance.
What is the problem with using spend per student as an indicator of a university's quality?
-Using spend per student as an indicator can be misleading because it may suggest that a university with higher costs is better, even if the outcomes are the same as a less expensive university. This could actually indicate inefficiency rather than quality.
What does the speaker suggest students should do instead of relying on league tables for university decisions?
-The speaker suggests that students should approach the decision-making process like a scientist, critically evaluating the numbers and the methodology behind the league tables, rather than taking the rankings at face value.
How does the speaker compare league tables to sports league tables?
-The speaker notes that sports league tables are based on clear, measurable outcomes like points earned, whereas university league tables combine various factors into a single number, which is not as straightforward or objective.
What is the San Francisco Declaration on Research Assessment and how does it relate to league tables?
-The San Francisco Declaration on Research Assessment is an initiative where universities pledge not to use flawed metrics such as the journal impact factor for research assessment. The speaker suggests a similar commitment could be made regarding the use of league tables.
What alternative approach to league tables does the speaker mention and what is its advantage?
-The speaker mentions an alternative approach where individual factors are presented, allowing the user to weigh the importance of each factor themselves. This approach is more personalized and does not imply a single 'right' answer.
What is the speaker's final recommendation regarding the use of league tables by universities?
-The speaker recommends that universities should stop using league tables in their publicity and advertising, and instead focus on providing meaningful information to prospective students, thus undermining the credibility and influence of league tables.
Outlines
📊 Critique of University League Tables
The speaker expresses frustration with the prevalence and influence of university league tables worldwide, arguing that they are misleading and oversimplified. They emphasize the importance of quantitative analysis in their field as a scientist, and criticize the reduction of complex data into a single ranking system. The speaker points out the practical issues with league tables, such as the vested interests of newspapers in creating volatility to sell copies, and the fundamental issue of attempting to rank institutions on an arbitrary scale. They also discuss the Research Excellence Framework, highlighting the absurdity of small numerical differences in rankings and the lack of transparency in how these numbers are derived.
🔍 Questioning the Validity of League Table Rankings
This paragraph delves deeper into the skepticism around league tables, questioning the calculation of error bars and the reliability of the rankings. The speaker compares the situation to the Olympic Games, where outcomes are clear-cut, unlike the ambiguous criteria used in educational league tables. They argue that the combination of various metrics to produce a single score for universities is flawed, as it doesn't account for the nuanced differences between institutions. The speaker also criticizes the emphasis on spending per student as a measure of quality, suggesting it rewards inefficiency rather than educational excellence.
🏆 The Flaws in University Ranking Systems
The speaker continues to critique university ranking systems, pointing out the inconsistencies and the lack of a standard, reliable method for evaluation. They discuss the Teaching Excellence Framework (TEF) and its星级 rating system, which creates artificial 'cliff edges' between institutions. The speaker also reflects on personal experiences as an admissions tutor, noting the lack of substantial changes between years despite significant shifts in rankings, underscoring the volatility and unreliability of league tables.
🤔 The Impact of League Tables on University Perception
In this paragraph, the speaker discusses the impact of league tables on the perception of universities and the decision-making of prospective students. They argue that league tables oversimplify the complex factors that contribute to a university's quality and experience. The speaker also touches on the idea of critical thinking in education and how league tables can mislead students away from making informed choices based on more meaningful criteria.
🌟 The Search for Alternatives to League Tables
The speaker contemplates the possibility of creating a more accurate and fair system for comparing universities, suggesting a more personalized approach that allows individuals to weigh different factors according to their own priorities. They acknowledge the difficulty of this task, recognizing that there may not be a perfect solution. The speaker also discusses the San Francisco Declaration on Research Assessment, which discourages the use of flawed metrics like journal impact factors, and suggests a similar approach could be beneficial for addressing the issues with league tables.
🤝 The Importance of Welcoming University Environments
The final paragraph shifts focus to the personal experiences and anecdotes of students, emphasizing the importance of a welcoming and friendly environment in a university. The speaker reflects on the significance of these qualitative aspects, which cannot be captured by league tables, and the need for universities to focus on providing a positive and supportive atmosphere for students. They also mention the potential for universities to lead by example by not engaging with flawed league table systems.
Mindmap
Keywords
League tables
Pseudo statistical
Error bars
Significant figures
Volatility
Research Excellence Framework (REF)
Value added score
Spend per student
Vested interest
Critical thinking
Teaching Excellence Framework (TEF)
Highlights
League tables are pervasive globally and are often misleading in higher education evaluation.
The creation of league tables is criticized as a pseudo-statistical approach lacking in quantitative rigor.
Various organizations compile data to rank universities, but the methods are often questionable and change annually.
Media organizations have a vested interest in league table volatility to drive newspaper sales.
The concept of 'best' university is fundamentally unorderable due to the diverse criteria for evaluation.
Subdividing league tables by subject, such as physics, does not eliminate the problem of undefined 'best' metrics.
The Research Excellence Framework (REF) ranks universities and allocates government funding, but its rankings can vary significantly.
Uncertainty and error bars are crucial in evaluating league table rankings but are often overlooked.
Some league tables consider metrics like student outcomes and staff-student ratios, but also problematic ones like expenditure per student.
The argument that higher spending per student equates to a better university is flawed and can mislead.
Different league tables can rank the same university vastly different, indicating inconsistency and unreliability.
The analogy of sports league tables is flawed when applied to education due to the different purposes and outcomes.
Universities should focus on their mission of education and research rather than gaming league table systems.
There is a call for universities to sign a declaration against using flawed league table metrics in their decision-making.
An alternative approach to league tables could be providing raw data and letting individuals weigh the importance of different factors.
The discussion highlights the need for critical thinking and skepticism when considering university rankings.
The video concludes with a plea to students to not solely rely on league table positions when making educational decisions.