Metrics govern so much of how we interpret the success of what is important in higher education. Yet what do we miss if we only focus on the things that can be counted?
Drawn from evidence from a QAA (Scotland) collaborative cluster project ‘Beyond the metrics: the intangibles’, Fiona Smart, Alastair Robertson and Liz Cleaver look beyond number-crunching to evolve their understanding of things of value which matter and are the heartbeat of our work, but can’t readily be counted.
Is our predilection for quantifiable measurement limiting our vision?”
The pursuit of metrics to portray the picture of quality learning and desirable outcomes for students is now a well-established and dominant weave in the fabric of UK higher education. We collect quantitative data and use it to compare across years and between institutions. We then use data to speak to the fulfilment of our missions and to direct activity intended to enhance practices.
Whilst being fully aware of the current drive and increasing conviction amongst decision-makers that quantitative data tells the story of what matters, we argue here that we need to be careful that the sector doesn’t fall into the trap of the so-called “McNamara fallacy”.
The McNamara Fallacy
McNamara was blind-sided to the fullness of information available to him.”
The McNamara fallacy takes its name from Robert McNamara, the first president of the Ford Motor Company outside the Ford family. In this role he generated further success by choosing particular data points and then optimising processes to improve efficiency, costs and quality. It’s fair to say he was a numbers man.
In January 1961, McNamara became the U.S. Secretary of Defence for the Kennedy administration; a position he still held in 1965 as the Vietnam war started. He applied his commitment to quantitative data in seeking to determine the success of this conflict, using body count as the main metric. What this approach missed was that such numbers were a poor measure of the impact of the war in numerous ways including the public mood and international relations (and notwithstanding the lives of the people the metrics represent). In effect, McNamara was blind-sided to the fullness of information available to him.
So our challenge to higher education is this: is our predilection for quantifiable measurement limiting our vision?
What are we losing by placing less value on the things that we can’t count, but nevertheless enhance the quality of our students’ learning experiences?
Further, has this constriction narrowed still further since Graham Gibbs acknowledged almost 10 years ago that teaching quality is a messy problem that cannot be judged by numbers or proxy output measures alone?
We suggest yes, and not just because of the advent of the Teaching Excellence Framework (TEF).
In 2015, the Higher Education Funding Council for England (HEFCE) commissioned significant research into trying to measure “learning gain”, funding 13 projects at a cost of £4 million. The recently published outputs of the initiative reinforce the complexity of the issues, urge caution on the use of readily available metrics for such purposes and question the robustness of such approaches. There are clearly important aspects of higher education which are not easily measurable or quantifiable: what we term our intangible assets.
Five intangible assets to contemplate
We have had the opportunity to evolve our thinking about intangible assets through leading a QAA Scotland funded collaborative cluster project as part of the current Enhancement Theme “Evidence for Enhancement”. The aims of the project were to:
- Work with the sector to identify which ‘intangibles’ are considered key to the success of teaching quality, student success and higher education more broadly
- Develop tools and a process by which the value and impact of these ‘intangibles’ might be evidenced
- Consider the potential implications for higher education policy and practice at both national and sectoral levels of such intangible assets.
We ran a series of nine workshops, with 147 participants across Scotland and England. Participants included academic and professional services staff, senior managers and student representatives, who worked to collectively identify the intangible assets that risk being missed, or moved to the periphery of our vision, if we persist in focusing only on the things which can be counted.
Assets that participants felt were of key importance to their higher education practice often eluded quantifiable measurement”
The overwhelming feedback from participants was that commonly-used metrics used in the UK higher education sector – student satisfaction, graduate earnings, student attendance etc – are insufficient proxies for the things that we and our students value the most. Although we were unsurprised by this finding, it certainly provided reassurance that this area is one of significant interest and concern for the wider higher education community.
Our data showed that the assets that participants felt were of key importance to their higher education practice often eluded quantifiable measurement. Here are their top five:
- Sense of belonging to a (learning) community.
- Building effective relationships (between students and staff and between staff).
- The transformational impact of (a university) education.
- Wellbeing (of students and staff).
- Student engagement in their own learning and the wider student learning experience.
‘So what?’ you might ask. It’s true, these five intangible assets are pretty unsurprising. They form the heartbeat of the higher education that we are proud to offer in our own institutions and are writ-large in our institutional strategies. We are sure they are in yours too. But are their associated KPIs?
It’s this very fact that spurs us on to speak back to the drive to focus solely on the counting, measuring and number-crunching and to imagine instead a common quest that seeks out the full picture of what matters across the higher education constituency, irrespective of how hard it is to capture. We believe in the need to navigate the NcNamara trap. Do you?
Further reading and details of the project:
‘Beyond the metrics: the intangibles‘ project overview
Utility of the McNamara Fallacy BMJ 2009;339:b3141
By Dr Alastair Robertson, Professor Liz Cleaver and Dr Fiona Smart
Dr Alastair Robertson
Professor Liz Cleaver
Dr Fiona Smart
Dr Alastair Robertson is Director of Teaching and Learning Enhancement at Abertay University with 25 years’ experience in the HE sector; his current interests include academic leadership, strategic change, learning spaces, student engagement and national HE policy development, for example the TEF.
Professor Elizabeth Cleaver is University Director of Learning and Teaching at the University of the West of England where she supports the DVC and Provost on developing strategy and embedding and delivering student-focused and inclusive teaching and practice-led learning.
Dr Fiona Smart is the Head of the Department of Learning and Teaching Enhancement at Edinburgh Napier University; she is a Principal Fellow of the Higher Education Academy (now AdvanceHE) and the Convenor of the Scottish Higher Educational Developers group.