New tools to improve statistical data for policy

Those who produce statistical data for policy making typically adhere to very high quality standards, yet there is always room for improvement. One area of concern involves the representativity of statistics derived from survey samples and registers. Simply looking at response rates may not provide a clear picture of how representative the statistics are of what they are measuring.

In order to improve the quality of statistical data from surveys and registers, the RISQ research project, a consortium of European researchers, is advocating the use of special methodological tools known as ‘R’ (representativity) indicators. These indicators can be used to assess levels of data quality more effectively. Importantly – especially for international policy coordination – the research suggests that the levels of data quality produced using this method can be used as benchmarks for comparing different data sources, including those from different EU Member States.

Involving researchers from the national statistical institutes of The Netherlands, Norway and Slovenia, with input from the universities of Leuven (Belgium) and Southampton (UK), the RISQ project specifically aimed at improving the scientific evidence base for statistics based on surveys and registers. By developing open-source software integrating R-indicators, the project goes beyond the theoretical level, offering companies and institutions the means to produce high quality statistics more efficiently.

Though somewhat technical in nature, the methodology advocated by the RISQ researchers focuses on properly assessing the phenomenon of ‘non-response’ (i.e. the incidence of not providing any answer to a survey request or query). Non-response, as the consortium observes, has two main consequences. First of all, it logically reduces the sample size, thereby decreasing the precision of the estimates. Second, it affects the sampling design. Sampling design is driven by theories of probability which are extremely important for generating meaningful survey data. As the researchers explain, “the probability to obtain an observation depends on both the selection probabilities specified in the sampling design and the unknown probabilities of responding.” It is on these ‘probabilities of responding’ that RISQ has focused its attention, proposing R-indicators as a way of reducing non-response bias in survey data.

Unfortunately, non-response bias is not eliminated by a high response rate. Indeed, even if a survey does have a high response rate, there can still be a high non-response bias if there are big differences between those who respond and those who do not respond. So while precision may be directly related to response rate, bias is a function of both response rate and the extent to which respondents and non-respondents differ.

By helping to reduce this non-response bias, R-indicators are a tool to help produce more accurate statistical data. Helpfully, RISQ has provided the tools for applying these indicators to many different kinds of data that are used by policy makers on a routine basis. Software for this purpose is now available on the project website (http://www.risq-project.eu/index.html) and the researchers have plans to continue developing the programs in the future.

After successfully testing its tools on multiple surveys in different countries, the RISQ consortium concluded that:

  • “potentially the method using R indicators can be extended to many other surveys and register based statistics and thus improve the underlying evidence base for many different policies”.

The researchers note that their findings could be:

  • “especially useful for those managing surveys and registers as well as for policy makers that use survey or registry data to inform their decisions”.

RISQ – Representativity Indicators for Survey Quality (duration: 1/3/2008 – 30/6/2010) was a Specific Targeted Research Project funded under the 7th Framework Programme for Research of the European Union, Thematic Priority 6 – Socio-economic and scientific indicators.

See: http://www.risq-project.eu/publications.html

Contact: Barry Schouten, jg.schouten@cbs.nl