Limitations

“First, the WHO research team should have been sure that their estimates are robust. Can they, in good conscience, make that claim? An artificially high ranking, for example, could take the wind out of the sails of desirable health-reform efforts. Similarly, an artificially low ranking could assign a bad grade to past reform efforts that were actually commendable. Rumour in the health services research community has it that France’s no.1 rank was driven in part by a flawed measure of national educational attainment. Under the methodology used by WHO, the more the level of educational attainment or of health spending is underestimated for a country, the higher will be the ratio of actual to ideal performance for that country and the higher will be the nation’s ranking.

“Second, if the report is addressed to policy-makers, one must judge it poorly written. To be sure, it has a number of fascinating, if chatty, chapters; but these are only loosely connected to the actual work underlying this study. To see what was actually done, one must plough through the cryptic commentary that accompanies the tables in the Annex or dig up and read sundry sources cited in the references. Few policymakers and even fewer journalists will go to that trouble.

“To be useful as a policy analysis, the report ought to have started with the crisp executive summary that is now de rigueur among policy analysts, certainly in the United States. That summary would have presented the main conclusions emerging from the study and described, in layman’s terms, the methodology that was used to reach these conclusions. Most important of all, the executive summary should have contained the many caveats that must, in good conscience, accompany ambitious analyses of this sort.”

Source: Reinhardt, U., and T. Cheng. “The world health report 2000 – Health systems: improving performance.” Bulletin of the World Health Organization vol. 78,8 (2000): 1064.

“The main criticisms were, however, of technical aspects of the methods used to assess performance. By common consent those undertaking this exercise faced a major problem if they were to include all 192 WHO Member States, some of which barely functioned as states and certainly lacked control over all of their territory. Many lacked even the most basic of information. Only a minority had any functioning system of vital registration, so if measures such as health outcomes were to be included they had to be modelled. This would be complicated enough if only the simplest of measures, such as life expectancy, was used. In many countries, there were established methods to calculate this by applying data from surveys of child mortality to standard life tables, but both inputs incorporated many assumptions. Yet a more complex measure, Disability Adjusted Life Years (Murray et al. 2000; Mathers et al. 2001) was used, even though even fewer data on disability were available and their application required highly contentious assumptions about valuations of different health states (Anand and Hanson 1997). The estimates of distribution of health outcomes also attracted criticism; derived from data from only a few countries and involving complex modelling, the measure was essentially the sum of intra-individual differences in outcome. This takes no account of the social and ethnic patterning of health whereby a summary measure may designate a system as reasonably fair even though there was systematic discrimination against a minority (Braveman et al. 2000).

“The problems in assessing responsiveness were even greater, with data being modelled on the basis of results from a survey of 1791 respondents in 35 countries, many of whom were WHO staff (Williams 2001). The modelling process did exclude those results for which ‘no rational explanation’ could be found and adjusted them for the tendency of those living in dictatorships to score their systems especially highly, but, even after this process, others showed poor correlation between the opinions of key informants and those surveys that existed (Blendon et al. 2001).

“Other criticisms were levelled at the measures of fairness of financing, for not taking account of the greater needs of the poor or the scale of redistribution, the use of a composite index of quite different parameters whose meaning was difficult to visualise, and the lack of external peer review (Almeida et al. 2001). The WHO has responded to these criticisms, for example by asking whether there was an alternative to modelling data where none exist (Murray et al. 2001) and highlighting the uncertainty levels around the estimates (Murray et al. 2000b), although even now, a decade later, these different views have not been resolved.

“Finally, to assess performance, it was necessary to estimate how much a health system could reasonably expect to achieve, given the myriad other influences on health. This estimate was derived from years of education, which, as Williams (2001) has noted, is an extreme over-simplification.”

Source: Martin McKee, The World Health Report 2000: 10 years on, Health Policy and Planning, Volume 25, Issue 5, September 2010, Pages 346–348, doi.org/10.1093/heapol/czq032

“Undoubtedly, many of the concepts and measures used in the report require further refinement and development. To date, our knowledge about health systems has been hampered by the weakness of routine information systems and insufficient attention to research. This report has thus required a major effort to assemble data, collect new information, and carry out the required analysis and synthesis. It has also drawn on the views of a large number of respondents, within and outside WHO, concerning the interpretation of data and the relative importance of different goals. The material in this report cannot provide definitive answers to every question about health systems performance. It does though bring together the best available evidence to date. It demonstrates that, despite the complexity of the topic and the limitations of the data, it is possible to get a reasonable approximation of the current situation, in a way that provides an exciting agenda for future work.”

Source: Message from the Director-General of the World Health Organization, Dr Gro Harlem Brundtland, in the introduction to The World Health Report 2000: Health Systems : Improving Performance. Geneva: World Health Organization, 2000.

Breaking News

• McKee on the WHO World Health Report 2000
  “The main criticisms were, however, of technical aspects of the methods used to assess performance. By common consent those undertaking this exercise faced a major problem if they were to include all 192 WHO Member States, some of which barely functioned as states and certainly lacked control over all of their territory. Many lacked even ...
• Reinhardt and Cheng on the WHO World Health Report 2000
  “First, the WHO research team should have been sure that their estimates are robust. Can they, in good conscience, make that claim? An artificially high ranking, for example, could take the wind out of the sails of desirable health-reform efforts. Similarly, an artificially low ranking could assign a bad grade to past reform efforts that ...
• WHO World Health Report 2000: Methodology
  “Undoubtedly, many of the concepts and measures used in the report require further refinement and development. To date, our knowledge about health systems has been hampered by the weakness of routine information systems and insufficient attention to research. This report has thus required a major effort to assemble data, collect new information, and carry out ...
• WHO World Health Report 2000: Rankings
  WHO World Health Report Annex Table 10: Health System Performance in all Member States, WHO Indexes. Click on thumbnails to view pages as individual image files (jpg format) or click on the link below to view a PDF of the data. whr2000-annex-table-10-1Download 1 2 3 4
• WHO World Health Report: Results
  “The U.S. health system spends a higher portion of its gross domestic product than any other country but ranks 37 out of 191 countries according to its performance, the report finds. The United Kingdom, which spends just six percent of GDP on health services, ranks 18 th . Several small countries – San Marino, Andorra, ...
• WHO World Health Report 2000: Overview and Defense
  “Differentiation between attainment and efficiency in health systems is crucial. Every society should be concerned about attainment of standards of health, responsiveness, inequalities in both of these, and fairness in financial contribution. What explains variation in these five key outcomes is an important scientific issue. Table 9 in the World Health Report 2000,9 contained the ...
• WHO World Health Report: Overview
  “To assess a health system, one must measure five things: the overall level of health; the distribution of health in the population; the overall level of responsiveness; the distribution of responsiveness; and the distribution of financial contribution. For each one, WHO has used existing sources or newly generated data to calculate measures of attainment for ...

World Health Systems Facts is a project of the Real Reporting Foundation. We provide reliable statistics and other data from authoritative sources regarding health systems in the US and sixteen other nations.

Page last updated March 29, 2025 by Doug McVay, Editor.