Quality in an Age of Measurement: The Limitations of Performance Indicators
QUALITY IN AN AGE OF MEASUREMENT: THE LIMITATIONS OF PERFORMANCE INDICATORS
THE SYDNEY LEADERSHIP ALUMNI LECTURE
THE BENEVOLENT SOCIETY
THE HONOURABLE J J SPIGELMAN AC,
CHIEF JUSTICE OF NEW SOUTH WALES
SYDNEY, 28 NOVEMBER 2001
The balance between quantitative and qualitative assessment of the performance of governmental functions arises in many different fields. Should art galleries or museums be judged by attendances? Should university researchers be judged by numbers of publications? Should educational institutions be judged by the earnings of their graduates? Should either the ABC or SBS be judged by television ratings? Should courts be judged by the time it takes to dispose of proceedings?
The answer to each of these questions is “Yes, in part”. I do not doubt that matters capable of a measurement are relevant to a process of assessment both for internal purposes and in terms of public accountability. . The issue is the extent to which crucial decisions are made only or primarily on the basis of what can be measured.
My central proposition is a simple one. Not everything that counts can be counted. Some matters can only be judged, that is to say they can only be assessed in a qualitative way. It is of primary significance to recognise that there are major differences between one area of government activity and another as to the centrality of those matters that are capable of being reduced to quantitative terms. In some spheres of governmental decision-making the things that can be measured are the important things. In other spheres the things that are important are not measurable.
I became interested in these matters by reason of my responsibilities for the administration of the Court. I recognise that in the administration of justice certain matters, such as delay, are capable of quantification. The compilation and publication of statistics relating to the measurement of delay is a perfectly appropriate activity. Nevertheless, the most important functions performed by a court are not capable of measurement. In particular the fundamental issue of whether or not the system produces fair outcomes arrived at by fair procedures is not capable of quantification at all.
It quickly became clear to me that advocates of quantification in the context of the administration of justice do not necessarily share my opinion as to the limited significance of quantification for the determination of important decisions affecting the courts. In particular there seemed to be an expectation that the use of quantification would inevitably expand to influence and, even determine, more and more decisions. These include the proposals for national benchmarks and the publication of comparative data which would in some way lead to competition between courts, perhaps directly by indicating role models, or indirectly, through an incentive to improve performance occasioned by the publication of invidious comparisons. Furthermore, some expect that, in the future, matters of quantification would become of central significance for the determination of the allocation of resources, e.g. expanding one court in a judicial hierarchy at the expense of another or transferring jurisdiction to non-judicial tribunals. Finally, at least two of the statutory tribunals responsible for the determination of judicial salaries, those of the Commonwealth and Queensland, proposed to take into account some unspecified quantified measure of performance in the course of deciding changes in judicial salaries.
Over the course of the last two decades the organisation of governmental activity has undergone a radical transformation. It has been referred to as a “global public management revolution”. In Australia, New Zealand, England and some European countries, the change has been called “the new public management”. The courts are an arm of government. They have not been and cannot be insulated from changes in attitude about the proper role of government and the appropriate means to conduct governmental activity. The courts have responded and will continue to do so.
My reading of the literature of the new public management indicated that in some areas of public decision-making in which quality was of great significance, such as education and health, the imperatives of measurement had originally been applied in much the same kind of low key way as is presently applicable or proposed in the context of the administration of justice. There are perfectly acceptable and legitimate purposes served by quantitative indicators. However, in the areas of health and education they did not stop there. Performance indicators and funding formulae have come to be applied in a mechanical and rigid way to determine budgets, remuneration and tenure with the most dramatic effects on the delivery of services and the determination of curricula and research. I was, and remain, concerned that nothing of this character should occur in the context of the administration of justice.
Perhaps the most definitive characteristic of the “new public management” is the greater salience that is given to what is called the “three E’s” – economy, efficiency and effectiveness – in competition with other values of government activity, such as accessibility, openness, fairness, impartiality, legitimacy, participation, honesty and rationality. We have not reached the stage where the “three E’s” are accepted to override completely these other values. Very real issues arise about the extent to which they ought be permitted to do so. In particular, we must recognise that the significance of these other values varies from one area of government activity to another.
The root of the difficulty is one of perception. Quantitative measurement appears to be objective and value free. Qualitative assessment appears to be subjective and value laden. In fact, quantitative measures – whether in the form of a funding formula or of a performance indicator – contain and conceal important value judgments.
The sense of precision that quantitative measurement sometimes gives can be altogether false. Nevertheless, it can also be comforting. There is an institutionalised bias against qualitative assessment, sometimes reflected in a personal fear of making qualitative judgments. There are significant benefits from decisions becoming virtually automatic, instead of requiring a formal process of assessment. Where persons, by reason of their institutional position, have ongoing relations with each other, it is easier to say that a decision is determined by a formula. It is much harder to make a decision which is based on an express assessment that one person or institution is not as good as another.
The apparent objectivity of the application of a pre-agreed formula to a particular situation is no doubt preferable to many. Judges who are used to making qualitative assessments, virtually on a daily basis, do not share the bureaucratic squeamishness about such assessments.
Of particular significance in this area of discourse is what appears to me to be a power struggle between the proponents of the “new public management” like treasury officials, departmental finance officers and auditors (to whom I find it convenient to refer as the “managers”) and persons like teachers, doctors or lawyers involved in public decision-making processes (to whom I will refer as the “professionals”). The professionals in the various areas of public sector decision-making in which they are involved tend to emphasise the significance of qualitative considerations. Managers tend to emphasise measurable indicators and objective formulae.
It is perfectly understandable why this should be so. To the extent to which qualitative considerations are given weight, the professionals will have the greater say. Unless matters can be reduced to measurable standards and indicators, the managers will not be able to exert significant influence. The managers simply do not have the capacity to make qualitative judgments. They have an inbuilt institutional bias to downgrade the significance of quality or to treat it in such a way that measurable factors actually determine the decision-making processes. As a regrettably anonymous pundit put it: “Where you stand depends on where you sit”.
The managers have a self-image of themselves as the true custodians of the objectives of an organisation and, often, as the representatives of the taxpayer in the interests of minimising expenditure and maximising efficiency. They sometimes resent the high degree of autonomy of professionals and categorise their pre-occupation with matters of quality as rent seeking activity. They tend – sometimes with reason - to regard professionals as liable to engage in self-serving conduct and to have no capacity to prioritise or to regard their professional standards as anything but absolutes. In a world where real choices have to be made about the allocation of resources, there are no such absolutes.
This conflict between managers and professionals has been reflected in numerous battles over budgets and programmes in virtually every area of public sector activity in advanced economies over the last two decades. Advocates of the new public management accept, of course, that considerations of “quality” are very important. The problem is that as a matter of practical application, experience suggests that these incantations often have a ritualistic quality. Quality considerations receive only lip service and the matters capable of a quantification more often than not determine the actual outcome.
Quantitative measurement has, by reason of its concreteness, acquired a disproportionate and inappropriate influence over considerations of quality, which appear to be amorphous. Decisions that plainly call for judgment are now often made on the basis of partial, purportedly “objective” considerations, with dramatic consequences which, probably, no-one would have chosen in a more comprehensively based decision-making process. In many areas of public decision-making, including the administration of justice, there is simply no escaping qualitative judgment. Decisions can never be made automatic. Quantitative measurement is necessarily, by its very nature, partial and incomplete. It is implemented in the name of rationality. However, it is a very partial irrationality, which by reason of its incompleteness may in substance prove to be fundamentally irrational.
Advocates of the new public management go through a process of assessing matters of quality by means, if possible, of some kind of measurement. Reference to quality considerations is customarily included in reporting on performance. This is inspired by private sector management which has adopted quality measures such as defect rates in manufactured goods or customer complaint statistics. In the areas of public decision-making with which I am concerned there is simply no measurable indicator of quality, even at the level of defect rates or complaint levels. There is just no escaping qualitative assessment. What that must mean is that decision-making processes based substantially on quantitative measurement are defective and, in my opinion, in substance, irrational.
The difficulties of measuring quality can be illustrated by the proposal that is sometimes made by the managers as to how to assess that quality of decision-making in the courts. It is proposed that there be some form of survey of opinions about various matters including the quality of the court’s decision-making processes. I have no doubt that in certain areas of court administration surveys of persons attending at court are useful, e.g. adequacy of signage, convenience of facilities and the like. I do not believe that there is any proper basis for surveys, even of lawyers let alone of litigants, for purposes of assessing the judgments of a court.
Surveys of opinion, even by legal practitioners, about courts generally or about individual judges, contain an inherent contradiction at the heart of the process. The value of the opinions depends on the quality of the person expressing it.
Some opinions are expressed by individuals of considerable capacity and with substantial experience about the court or the individual judge about whom the opinion is expressed. Other persons surveyed, including legal practitioners let alone litigants, are of much lesser capacity and may have a bitter experience about a particular proceeding or may have no relevant experience of a particular judge (as would be the case with almost all legal practitioners about any individual judge). Accordingly their “opinions” reflect a second or third-hand view with no proper foundation.
Opinion surveys about quality are, necessarily, in large measure surveys of reputation. Reputation is not necessarily related in any direct, or even rational way, to the matter sought to be assessed in this qualitative manner. They are a grotesquely unreliable form of assessing quality.
Some opinions are entitled to greater weight than others. In order to assess the value of the opinion being expressed, it is necessary to ask the very same questions about the quality of the person expressing the opinion as is being asked in the survey. Any attempt to assess the quality of the persons surveyed suffers, of course, from exactly the same difficulty as the first level of opinions being expressed. This is an infinite regression from which there is no escape.
No doubt such surveys give the appearance of being democratic, in the sense of involving a wide range of relevant opinion. It may be unfashionable to say so, but quality is by its nature not susceptible to democratic assessment. The assessment of quality is often inherently elitist, in the sense that it can only be made by persons with education and experience pertinent to the assessment being undertaken. In the case of judges and courts, surveys will not be of any utility. Quality is hard to assess. But if we ignore it, we do so at the peril of perverting the decision-making processes which we are seeking to improve.
The perennial discourse over the proper role of government can be characterised as a debate between those who emphasise government failure and those who emphasise market failure. Over the last two decades those who emphasise government failure have been in the ascendancy. This ascendancy has been reflected in a variety of changes such as privatization, contracting out, deregulation, reinventing government, value for money, re-engineering, market based solutions, downsizing, purchaser/provider arrangements, cost benefit analyses, performance indicators, program budgeting, output and outcome frameworks, benchmarking, performance audits, mission statements – even vision statements - strategic plans, citizen charters, business plans, targets, etc. etc.
A central theme of the “new public management” is the application to the public sector of the institutional structure and decision-making techniques said to be characteristic of the private sector. One of the difficulties with this approach is that it tends to reduce citizens to consumers.
There is nothing wrong with being a consumer or in ensuring that organisations take into account how well the functions they perform meet the requirements of people as consumers or clients. Nevertheless, it is important to recognise that a person’s interest as a “consumer” is only one part of a person’s status as a citizen. The consumer analogy has become, in many respects, a feral metaphor that has acquired a disproportionate degree of prominence.
Consumers have desires or needs. Citizens have rights and duties. The perspective of citizenship is of greater significance for many areas of public activity than the perspective of consumerism. This is the case for the administration of justice.
In the context of court administration, the consumer perspective treats courts merely as a provider of dispute resolution services. Indeed, the most recent court created in Australia, the Federal Magistrates Court is, by specific statutory provision, permitted to be called the “Federal Magistrates Service”. It is the policy of the government to prefer that denotation.
The identification of the courts as merely a publicly funded dispute resolution service is too narrow. Indeed, in my opinion, it is potentially subversive of the rule of law.
Human life cannot be characterised simply as a series of consumer choices. Litigants are not consumers. Litigants have rights. They come to court to assert their rights, not to exercise some form of consumer choice. This is clearest in the criminal justice system where, in substance, the community asserts rights by way of protecting itself. In all cases, litigants are, and should be, treated in the courts as, citizens not consumers.
The courts do not deliver a “service”. The courts administer justice in accordance with law. They no more deliver a “service” in the form of judgments and decisions, than a Parliament delivers a “service” in the form of debates and statutes. I do not doubt that courts serve the people. But they do not provide services to the people. This distinction is not merely semantic, it is fundamental.
Courts perform functions that go well beyond resolving disputes. The enforcement of legal rights and obligations, the articulation and development of the law, the resolution of private disputes by a public affirmation of who is right and who is wrong, the denunciation of conduct in both criminal and civil trials, the deterrence of conduct by a public process with public outcomes - these are all public purposes served by the courts, even in the resolution of private disputes.
The judgments of courts are part of a broader public discourse by which a society and polity affirms its core values, applies them and adapts them to changing circumstances. This is a governmental function of a broadly similar character to one of the functions performed by legislatures. This has no relevant parallel in most other spheres of public activity, let alone private activity.
I have no doubt that there are important areas of government in which the emphasis on a consumer perspective and the analogy with the free market have been adopted with substantial benefits. However not all areas of government are capable of being moulded by analogy with the operation of a free market. The administration of justice is an area in which this analogy has little useful to contribute.
No-one advocates that commercial corporations should conduct their affairs in public, nor that they should publish reasons for their decisions, or observe any of the other principles of open justice. Nor should the operations of commercial corporations be seen as having a determinative relevance to the administration of justice.
One characteristic of the administration of justice is its inefficiency when compared with some other systems of decision making. This is not an accident.
There is no doubt that a much greater volume of cases could be handled by a specific number of judges if they could sit in camera, dispense with the presumption of innocence, not be constrained by obligations of procedural fairness or the need to provide a manifestly fair trial, could act on the basis that no-one has any rights and not to have to publish reasons for their decisions. Even greater “efficiency” would be apparent if judges had made up their minds before a case began. There are places where such a mode of decision making has been, and indeed is being, followed. We do not regard them as role models.
Our system of justice is not the most efficient mode of dispute resolution. Nor is democracy the most efficient mode of government. We have deliberately chosen inefficient ways of decision making in the law in order to protect rights and freedoms. We have deliberately chosen inefficient ways of governmental decision making in order to ensure that governments act with the consent of the government.
There is a tendency amongst managers to regard measurement as benign and that no harm can be done from quantification even if it does not prove useful. That is wrong. The process of deciding what and how to measure so called “performance”, is capable of having very real effects on behaviour and to distort actual conduct in a manner that no-one would actually chose. This is a critical manifestation of the irrationality of partial rationality.
Advocates of the “new public management” like to think of themselves as business-like. They approach government decision-making with a single market driven private sector model which will, by the application of some natural law, lead to the most desirable outcome. A different perspective is available. An emphasis on strategic plans, business plans and performance indicators characterises the long tradition of socialist planning. The one thing of which there was no shortage in the former Soviet Union was performance indicators. They called it a five year plan. The Soviet experience indicates dramatically the distortions that can arise by reason of an emphasis on quantification.
Nikita Khrushchev, in one of his speeches attacking what he called the “steel eaters” of heavy industry revealed that when the five year plan for nail manufacturers identified output in terms of tonnes, every manufacturing plant in the country made large nails and there was a shortage of small nails. Accordingly, in the next five year plan output was stated in terms of numbers of nails. The inevitable happened. Everyone made small nails and there was a shortage of big nails.
It is of critical importance to recognise that measurement is not neutral in its effects. Measurement has consequences. Indeed it is expected to have consequences. However, what occurs may not be as intended. The more significant the application of quantitative measurement, the more likely it is that it will have unintended consequences. There is a hierarchy of increasing significance for the use of quantitative factors which can be set out, broadly as follows:
(i) Internal management.
(ii) External accountability.
(iii) Allocation of resources.
(iv) Job security and remuneration.
In the literature of the new public management a distinction is drawn between outputs and outcomes. To give an example of the difference in the case of occupational health and safety inspectorate, an “output” would be the number of inspections. The “outcome” is the ultimate objective such as a decline in rates of injury in the workplace. It has long been accepted that the attempt to develop measurable indicators for “outcomes” has been a failure Shead “Outcomes and Outputs – Who’s Accountable for What”,1998 4 Accountability and Performance 89 at 90-92.
. Some managers have recently expressed concern about an “output fixation”. The problem is that “outputs” can be measured, whereas “outcomes” usually cannot be. The greater the role and significance of measurement, the more likely it is that “outputs” will be targeted, if necessary at the expense of “outcomes”.
This may not be of concern. In some areas of government activity what can be measured, at least on a proxy basis, may include quality aspects of “outputs” and important “outcomes”. However, to the extent that what matters cannot be measured, as I contend is the case for the administration of justice, then it is necessary to be concerned about the pathology of measurement.
The example of the Soviet Five Year Plan can be matched with numerous examples in the West, including in private corporations. Performance indicators often have perverse effects, especially when they determine resource allocation and job security or remuneration.
One Harvard Business School professor calls the process “Paying People to Lie” Michael C Jensen “Paying People to Lie: The Truth About the Budgeting Process” Harvard Business School Working Paper No 01-072 April 2001 accessible at http://papers.ssrn.com.
. He reports an example of bonuses being paid on the basis of sales revenues within a certain period. At great cost and inconvenience, unfinished industrial products were shipped from the plant in England to the Netherlands, where they could be assembled close to the customer and the sales revenue could be booked before the end of the quarter. The managers made their bonuses but the company’s profit was substantially reduced. In another case, management announced that prices would increase by 10% in four months time, which had the effect of bringing forward sales and meeting targets to which bonuses were tied.
Sometimes the response is fraudulent. He reports a case where the United States Securities and Exchange Commission brought charges against a company for falsely increasing earnings by several hundred millions of dollars over a certain period. This occurred because of the internal incentives and bonuses paid to managers. They brought forward revenues by backdating sales agreements, entered into secret side agreements granting rights to refunds after the relevant period, recognised revenue on disputed claims, etc. In another case fraudulent reporting of completely fictitious sales ensured that managers acquired bonuses tied to sales targets.
As the author concludes:
“Tell a manager that he or she will get a bonus when targets are realised then two things are sure to happen. First, managers will attempt to set targets that are easily reachable, and once the targets are set, they will do their best to see that the targets are met even if it damages the company to do so.”
On the issue of target setting he agrees with the observation that:
“The more people lie about how much they cannot do, the more they are rewarded.”
All of this behaviour, he says, has become an accepted part of business life, although it is “undiscussable”. He concludes:
“To stop the gaming of budgets and targets and restore integrity to the planning and management process we must begin not by telling managers to stop lying, nor by eliminating the use of budgets, but by eliminating the use of budgets and targets in compensation formulas and promotion systems.”
The distortions created by the application of rigid funding formulae which allocate resources, and by indicators that determine employment, appear to exist in the university context.
The difficulties are well stated in a recent assessment of the impact on American universities of the emphasis given to publication in the grant of tenure to American academics. It is worth quoting at some length.
“As long as the candidate proves an inoffensive teacher and a reasonable department member, only one question sits on the meeting room table: Is the research project finished? If the junior colleague has a book in hand or an acceptance letter from the director of the university press, tenure is a fait accompli. If the work remains in manuscript, promising but incomplete, no promotion. That is the employment equation. Tenure has boiled down to a six-year composition scheme. Junior faculty now face a demystified production schedule, and senior faculty enter the tenure meeting with a one-checkbox form in their heads. No more messy discussions about quality. No more anxiety about whether the department has enough discernment, or too much. Administrators have an objective criterion to point to should any outsiders challenge the proceedings. Judgment has been externalised, handed over to the university editorial board. The assistant professor has inherited a job task that takes priority over teaching students, that is, marketing his revised dissertation to academic press editors.
While the book criterion has clarified the tenure process, it has fundamentally altered the nature of scholarship in the humanities. The system discourages research that is time-consuming, that involves tracking down information secreted in libraries and archives, that may yield numerous dead ends before a discovery occurs. Junior faculty must envision book-length projects that can be executed well in advance of the crucial tenure meeting, which takes place in the middle of the candidate’s sixth year of employment. … Books that require lengthy inquiries do not get written. … Clear-sighted professors will avoid empirical methods, aware that it takes too much time to verify propositions about culture, to corroborate facts with multiple sources, to consult primary documents, and to compile evidence adequate to inductive conclusions. They will seek out research models whose premises are already in place, not in need of proof, and whose exercise proceeds without too much deliberation over inquiry guidelines. Speculation will prevail over fact-finding, theory and politics over erudition. Inquirers will limit their sources to a handful of primary texts and broach them with a popular academic theory or through a socio-political theme. In sum, facing a process that issues in either lifetime security or joblessness, junior faculty will relax their scruples and select a critical practice that fosters their own professional survival, a practice that offers timely shortcuts to publication and still enjoys institutional sanction.” Mark Bauerlein “Social Constructionism: Philosophy for the Academic Workplace” (2001) Vol LXVIII (2) Partisan Review accessible at www.bu.edu/partisanreview.
The central theme of this article is that these pressures have distorted American intellectual life by creating an environment in which post modernism in its various simplistic forms can flourish.
As can be seen from this experience, the attempt to abolish judgment and replace it by an allegedly “objective” criterion can have significant distorting effects. There is every reason to believe that in Australia an egalitarian instinct may exacerbate these kinds of distortions. The exercise of judgment may appear to be elitist. The substitution of an allegedly objective, measurable indicator can be appealing.
That such considerations may be having an impact on the Australian university is suggested in a recent address by Professor Stuart Macintyre, Dean of the Faculty of Arts at the University of Melbourne. He identified the effects of funding formulae developed over the decades for the allocation of resources to, and accordingly within, universities. He noted the significance of external research funding support in determining public funding for research. Professor McIntyre concluded:
“The new regimen has brought considerable change on the ways that universities manage their affairs and on the way that academics conduct their teaching and research. There is both an institutional and individual pre-occupation with measurement of performance. The aggregate of research funding has become a measure of the research performance of the university; its newsletter and glossy promotional literature will feature the research project that attracts the largest grant as the paradigm of excellence. Academics, who are usually so resistant to external direction of their activity, show a surprising responsiveness to these market signals. A lawyer will write a journal article rather than a case note because the former is included in the publication index and the latter is not. It becomes more difficult to find a book reviewer or a journal editor, because these activities, so necessary to the scholarly infrastructure, are not recognised as research activities for funding purposes.
The core disciplines of the sciences, social sciences and humanities are especially disadvantaged by the emphasis on research income as a determinant of funding. They find it harder to attract industry funding, which is concentrated in the biological and technological sciences. Some of the disciplines are especially disadvantaged by the new conditions on post graduate research: in linguistics or anthropology, where substantial fieldwork is required, a candidate will have great difficulty in completing his or her thesis in the time that is required for funding purposes. More generally, the application of simple aggregate measures across the range of research fields plays little heed to issues of quality. The British system where discipline panels evaluate research performance on a qualitative basis is far more conducive to breadth and excellence.” Stuart Macintyre ” ‘Funny You Should Ask for That’: Higher Education as a Market” paper to
“The Idea of a University: Enterprise or Academy?“ Conference organised by The Australian Institute and Manning Clark House, ANU, 26 July 2001.
In addition to external funding, the formula gives weight to number of publications. What one would expect to occur as a result of the application of a funding formula of this character is an increase in the amount of published research but a decline in the quality of that research. A recent study of Australia’s research performance suggests that this has in fact occurred Linda Butler Monitoring Australia’s Scientific Research: Partial Indicators of Australia’s Research Performance Australian Academy of Science, October 2001.
. The ANU’s Research Evaluation and Policy Project published in October of this year an assessment of all Australian publications indexed from 1981 to 1999 by the Institute for Scientific Information, concentrating on, although not limited to, pure science. Driven by the university sector, the number of publications and Australia’s international share of publications, has increased significantly.
The report focuses not only on number of publications but on how often those publications are quoted by others: called “relative citation impact”. The study concludes that over the 19 year period Australia’s relative citation impact has declined, unlike other countries. The author concludes that this occurred because Australian publications tended to appear in journals of lower impact Ibid at 12-13.
. The author notes that the introduction of performance indicators for universities appeared to be responsible for the increased research output to a significant degree. However, the author concludes:
“There are indications that Australia’s relative citation impact may have been adversely affected by the push to evaluate research on the basis of publication output, with little reference to the quality of that output.” Ibid at 14, see also at 24-25.
This study shows the difficulty of measuring quality. The proxy measurement for “quality:” is frequency of citation. This is by no means obviously a reliable indicator of quality. Scientific research is in part a fashion industry, influenced to a degree by sources of available funds, which are differently directed in different places. Of course, this very difficulty magnifies the problem. The consequences suggested by this research – of a decline in research quality related to a system that rewards quantity – are so obviously contestable, that the congratulations which came from the increase in quantity, may not be offset by qualms about a decline in quality.
A new funding formula is to be introduced which gives greater emphasis to completion of postgraduate degrees within strict time limits. Similar distorting effects are now predicted under the new formula Ruth Neumann and James Guthrie The Corporatisation of Research in Australian Higher Education Macquarie Graduate School of Management May 2001.
Let us not assume that the kinds of distortions that appear in other organisations will not affect the courts. I am aware of one jurisdiction, not in Australia or the United States, in which the court administrator had a performance package related to the quantum of cases processed. Counsel contacting the court to say that a case was settled were discouraged from having the matter removed from the list by consent. Rather, settlement was done by an appearance in court. When the matter was so listed it counted for purposes of the administrator’s remuneration.
Measurement, I repeat, is not neutral. It has consequences. The greater the incentives given, whether by means of performance bonuses to individuals or enhanced funds and budget allocations, the greater the likely consequences.
The internal use of information for purposes of managing the organisation does not create any significant prospect of distortion. It is the external use of this information which creates the possibility of such distortion. Distortion arises because the things that can be measured are not the only things that matter. Insofar as external judgments are made on an information base which is too narrow, then the incentives created by performance indicators will operate perversely.
Performance indicators are always partial and are always manipulable. The persons who administer the measurement system always have superior private information about how their own actions influence the measured results, than do the persons to whom the results are reported. Strategies of targeting the indicator, rather than doing the job properly, are always capable of being adopted. These rarely have adverse consequences for those responsible because of the difficulty of auditing the distortions which occur. The objectives distorted are often either not measurable or not measured.
An organisation can always improve performance as measured, by reducing quality in ways which are not necessarily detectable. In the case of a slow degradation of the quality of justice, nothing particularly dramatic would occur. More corners are cut, as more pressures for expedition are exerted. The wishes of litigants are overridden more often. The quality of justice may be progressively compromised in small, incremental and barely perceptible steps but with an ultimate consequence that is unacceptable. By then it is too late.
|
|
