One of the most devastating critiques of modern government intervention was written nearly forty years ago by the American sociologist Peter H. Rossi. His 1987 paper, The Iron Law of Evaluation and Other Metallic Rules, should be compulsory reading for every politician, bureaucrat and policy adviser who believes that society can be improved simply by establishing another government program, appointing another committee and allocating another billion dollars of taxpayers' money. Rossi's conclusions were expressed with deliberate humour, but the underlying message was deadly serious. When large-scale social programs are subjected to rigorous evaluation, their measurable benefits are frequently disappointingly small, sometimes nonexistent, and occasionally contrary to what their designers intended.
Rossi's famous Iron Law of Evaluation states that the expected net impact of a large-scale social program is zero. This was not intended as a mathematical theorem proving that government intervention can never work. It was a provocative generalisation drawn from decades of American program evaluation. Rossi observed that ambitious initiatives designed to improve social conditions often failed to produce the substantial benefits anticipated by their sponsors. Governments announced their objectives, experts designed their interventions, bureaucracies administered the programs, and public money flowed through the system. Yet when researchers attempted to establish what had actually changed because of the intervention, the results were frequently unimpressive.
The political implications are enormous. Modern government rests increasingly upon the assumption that social problems can be identified, analysed and corrected through administrative intervention. Poverty, unemployment, educational disadvantage, crime, homelessness, family breakdown and countless other difficulties are treated as problems requiring publicly funded programs. Each new initiative is accompanied by assurances that expert knowledge, targeted expenditure and sophisticated management will deliver measurable improvements. Yet Rossi's work raises the uncomfortable possibility that much of this activity produces considerably less benefit than its advocates imagine.
His second principle, the Stainless Steel Law, is even more damaging to political optimism. Rossi argued that the better designed an evaluation becomes, the more likely it is to discover little or no net impact. This observation exposes a fundamental weakness in the political presentation of government achievements. Ministers frequently point to the number of people assisted, the amount of money distributed, the number of training places created or the quantity of services delivered. Such figures establish that activity occurred. They do not establish that the program caused a worthwhile improvement in the lives of its intended beneficiaries.
Consider a government employment program that provides training to thousands of unemployed people. Ministers may celebrate the number of participants who subsequently find jobs. But the relevant question is how many would have found employment anyway. If the same proportion of comparable unemployed people obtained work without participating in the program, the apparent success may largely disappear. The government has purchased training, employed administrators and generated favourable publicity, but it has not necessarily improved employment outcomes.
The distinction between activity and effectiveness is one of the most important in public policy. A hospital may conduct more consultations without producing better health outcomes. A school may receive additional funding without improving literacy or numeracy. A crime-prevention initiative may organise hundreds of workshops without reducing offending. A homelessness program may record thousands of contacts with clients while leaving the underlying problem substantially unchanged. None of these possibilities proves that the particular programs are ineffective. They illustrate why counting activities is not the same as demonstrating results.
Rossi's Brass Law concerns interventions designed to change individuals. He argued that programs attempting to rehabilitate people or transform their behaviour were particularly likely to produce negligible net effects. This conclusion challenges one of the most persistent assumptions of modern social policy: that undesirable behaviour can be corrected through sufficiently sophisticated professional intervention. Governments have repeatedly funded programs intended to change attitudes, improve motivation, prevent offending, encourage employment and transform personal conduct. Some interventions undoubtedly help some participants, but lasting behavioural change is difficult, especially when the conditions sustaining the original behaviour remain largely untouched.
The political attraction of such programs is obvious. They allow governments to demonstrate compassion and activity without necessarily confronting more difficult structural questions. A program can be launched, funded and publicised within a parliamentary term. Its long-term effectiveness may take years to establish, by which time the responsible minister has moved to another portfolio or retired from politics. The political rewards arrive immediately, while the evidence of failure may emerge much later.
Rossi's fourth principle, the Zinc Law, adds another layer of irony. He suggested that only programs likely to fail are evaluated. The proposition was deliberately exaggerated, but it points to the politics of evaluation itself. Programs that appear successful may attract little scrutiny, while controversial or troubled initiatives become subjects of intensive investigation. Conversely, governments may have incentives to commission evaluations that emphasise favourable outcomes, select convenient measures or avoid questions capable of producing politically embarrassing answers.
This does not mean that all evaluations are manipulated. It means that evaluation takes place within political institutions whose interests are not always identical with the pursuit of objective knowledge. A minister seeking re-election, a department defending its budget and an organisation dependent upon government grants may all have reasons to prefer favourable findings. Independent evaluation is therefore not merely a technical requirement. It is a safeguard against institutional self-interest.
The Australian relevance of Rossi's analysis is considerable. Commonwealth and state governments spend enormous sums on education, employment services, welfare administration, health initiatives, Indigenous affairs, housing assistance and countless other programs intended to improve social outcomes. Many of these activities perform essential functions, and some have demonstrably beneficial effects. But expenditure alone is no measure of success. The crucial question is whether particular interventions produce improvements that would not otherwise have occurred, and whether those improvements justify their costs.
Indigenous policy illustrates the importance of this distinction. Governments have spent decades introducing programs intended to address disadvantage in health, education, employment, housing and community safety. Significant gaps remain across numerous indicators. That persistence does not establish that every intervention has failed, nor does it mean that spending has been without benefit. It does, however, demonstrate why governments must distinguish between good intentions, resources committed and actual improvements in outcomes. A policy should not be protected from critical examination merely because its declared objectives are morally admirable.
The same applies to education. Political debates often assume that increasing expenditure will automatically improve educational performance. Yet the relationship between funding and outcomes is more complicated. How money is used, the quality of instruction, school leadership, student attendance, family circumstances and curriculum design can all influence results. A government that announces billions in additional education funding has not thereby demonstrated that students will learn more. That proposition requires evidence.
Employment policy provides another example. Training programs, wage subsidies, job-placement services and employment incentives may be effective under particular circumstances. But their success must be measured against credible comparisons, including the possibility that participants would have obtained employment without assistance. Governments should also consider displacement effects, where subsidised employment for one group reduces opportunities for another. A program can appear successful for its participants while producing a much smaller benefit for the labour market as a whole.
There is a deeper philosophical problem here, closely related to Friedrich Hayek's critique of central planning. Society is an immensely complicated network of individuals, institutions, incentives, customs and local circumstances. Political authorities can identify desirable objectives, but they cannot assume that interventions designed in government departments will produce the intended results once they encounter the complexity of real human behaviour. People respond to incentives, adapt to regulations and sometimes behave in ways policymakers never anticipated.
A government program is therefore not like a machine whose output can be predicted simply by specifying its inputs. Social interventions operate within systems that change in response to the intervention itself. A subsidy may encourage the activity it supports, but it may also raise prices, displace private expenditure or create dependence upon continued public funding. A regulation may eliminate one undesirable practice while encouraging another. A welfare measure may relieve immediate hardship while unintentionally altering incentives affecting employment or household decisions. These possibilities do not make intervention illegitimate, but they make confident predictions of success hazardous.
Rossi himself did not draw the conclusion that government should abandon social programs. He believed that evaluation research could improve policy by identifying the reasons interventions failed and developing a stronger knowledge base for designing successful programs. This is an important qualification. His metallic laws were warnings against excessive confidence, not arguments for the abolition of social policy. Nevertheless, a more sceptical political conclusion follows naturally from his findings: the burden of proof should rest upon those proposing large-scale interventions, rather than upon citizens who question their effectiveness.
This reverses the usual political presumption. Governments frequently announce programs on the basis of desirable objectives, with critics then expected to explain why the proposals should not proceed. Rossi's work suggests that a better starting point would be to demand credible evidence that the proposed intervention is likely to work. What is the causal mechanism? What comparable programs have succeeded? What unintended consequences are foreseeable? How will effectiveness be measured? What happens if the results are disappointing?
These questions should be asked before substantial public money is committed, not merely after a program has become politically entrenched. Pilot schemes, independent evaluations, transparent data and credible comparison groups should be normal requirements where practical. Programs that fail should be modified or discontinued rather than perpetuated because their objectives remain attractive or their beneficiaries politically influential.
Unfortunately, modern politics rewards announcements more reliably than successful outcomes. A minister gains publicity by launching a new initiative, promising ambitious targets and allocating impressive sums. There is much less political glory in admitting that an established program does not work. Closing an ineffective program may provoke criticism from employees, contractors, advocacy organisations and recipients who have developed an interest in its continuation. The result is a powerful institutional tendency towards accumulation rather than correction.
This is one reason government programs can survive long after the evidence supporting them has weakened. Bureaucracies develop around them, budgets become established, professional careers depend upon them, and political reputations become associated with their continued existence. A failed program can acquire a constituency whose interest lies in preserving the program rather than solving the problem it was originally established to address.
The consequences extend beyond wasted expenditure. When governments devote resources to ineffective interventions, those resources are unavailable for alternatives that might produce greater benefits. Taxpayers bear the financial burden, while intended beneficiaries may lose opportunities for genuinely useful assistance. The moral cost of failure is therefore not confined to accounting figures. A program that consumes money without improving lives can represent a serious failure of public responsibility.
Rossi's metallic laws remain relevant because the political temptation they expose has not disappeared. In 2024, Douglas Besharov and colleagues revisited Rossi's argument, emphasising the continuing challenge of making evaluation genuinely useful for policy decisions. The problem is not simply determining whether a particular intervention succeeded in one setting. Policymakers also need to understand whether similar results can be expected elsewhere, under different institutional and social conditions. Evidence from a small experimental program does not automatically establish that the same intervention will work when expanded nationally.
That issue of scale is particularly important. A carefully managed pilot may benefit from unusually committed staff, intensive supervision and participants who volunteer because they are motivated to succeed. A nationwide program must operate through ordinary institutions, with variable administrative quality and a much more diverse population. Success under experimental conditions may therefore be difficult to reproduce. The political desire to scale up promising interventions can run ahead of the evidence that they will remain effective.
Rossi's argument ultimately challenges the self-image of the modern administrative state. Governments like to present themselves as rational problem-solving institutions guided by expertise and evidence. Yet expertise does not abolish uncertainty, and the existence of a government department does not guarantee that it possesses the knowledge required to transform society. Sometimes the most valuable contribution of research is to establish that an apparently attractive intervention has little effect.
The proper political response should be intellectual humility. Governments should acknowledge the limits of their knowledge, resist the temptation to promise social transformation through administrative machinery, and allow greater room for local experimentation, voluntary institutions and individual initiative.
https://gwern.net/doc/sociology/1987-rossi.pdf