“The main criticisms were, however, of technical aspects of the methods used to assess performance. By common consent those undertaking this exercise faced a major problem if they were to include all 192 WHO Member States, some of which barely functioned as states and certainly lacked control over all of their territory. Many lacked even the most basic of information. Only a minority had any functioning system of vital registration, so if measures such as health outcomes were to be included they had to be modelled. This would be complicated enough if only the simplest of measures, such as life expectancy, was used. In many countries, there were established methods to calculate this by applying data from surveys of child mortality to standard life tables, but both inputs incorporated many assumptions. Yet a more complex measure, Disability Adjusted Life Years (Murray et al. 2000; Mathers et al. 2001) was used, even though even fewer data on disability were available and their application required highly contentious assumptions about valuations of different health states (Anand and Hanson 1997). The estimates of distribution of health outcomes also attracted criticism; derived from data from only a few countries and involving complex modelling, the measure was essentially the sum of intra-individual differences in outcome. This takes no account of the social and ethnic patterning of health whereby a summary measure may designate a system as reasonably fair even though there was systematic discrimination against a minority (Braveman et al. 2000).
“The problems in assessing responsiveness were even greater, with data being modelled on the basis of results from a survey of 1791 respondents in 35 countries, many of whom were WHO staff (Williams 2001). The modelling process did exclude those results for which ‘no rational explanation’ could be found and adjusted them for the tendency of those living in dictatorships to score their systems especially highly, but, even after this process, others showed poor correlation between the opinions of key informants and those surveys that existed (Blendon et al. 2001).
“Other criticisms were levelled at the measures of fairness of financing, for not taking account of the greater needs of the poor or the scale of redistribution, the use of a composite index of quite different parameters whose meaning was difficult to visualise, and the lack of external peer review (Almeida et al. 2001). The WHO has responded to these criticisms, for example by asking whether there was an alternative to modelling data where none exist (Murray et al. 2001) and highlighting the uncertainty levels around the estimates (Murray et al. 2000b), although even now, a decade later, these different views have not been resolved.
“Finally, to assess performance, it was necessary to estimate how much a health system could reasonably expect to achieve, given the myriad other influences on health. This estimate was derived from years of education, which, as Williams (2001) has noted, is an extreme over-simplification.”
Source: Martin McKee, The World Health Report 2000: 10 years on, Health Policy and Planning, Volume 25, Issue 5, September 2010, Pages 346–348, https://doi.org/10.1093/heapol/czq032 https://www.ncbi.nlm.nih.gov/pubmed/20798126
