Meta-regression analysis: Producing credible estimates from diverse evidence

Meta-regression methods can be used to develop evidence-based policies when the evidence base lacks credibility

Deakin University, Australia

one-pager full article

Elevator pitch

Good policy requires reliable scientific knowledge, but there are many obstacles. Most econometric estimates lack adequate statistical power; some estimates cannot be replicated; publication selection bias (the selective reporting of results) is common; and there is wide variation in the evidence base on most policy issues. Meta-regression analysis offers a way to increase statistical power, correct the evidence base for a range of biases, and make sense of the unceasing flow of contradictory econometric estimates. It enables policymakers to develop evidence-based policies even when the initial evidence base lacks credibility.

A research record that has missing
                        estimates can be improved by meta-regression

Key findings

Pros

Meta-regression offers a systematic scientific approach to research synthesis and quantitative review of the evidence base.

Meta-regression methods can be used to formally test hypotheses and rival theories.

Meta-regression is particularly useful for explaining sources of the wide variation in reported econometric estimates.

Meta-regression can correct the evidence base for specification bias and reduce the effects of publication selection bias.

Involving simple regression analysis and standard software, with well-established guidelines and protocols, meta-regression is easy to implement.

Cons

Identifying relevant studies and extracting and coding the data needed to conduct meta-regression analysis is very labor intensive.

Inexperienced coders can make coding errors, thereby biasing estimates.

Researcher bias in making modeling choices can negatively affect meta-regression analysis just as it can affect primary data analysis.

Meta-regression methods are less effective for research areas in which there are only a handful of econometric studies.

Author's main message

To formulate effective labor market interventions, policymakers need reliable estimates of key relationships. Yet the reported evidence cannot be taken at face value. Most empirical studies report an excessively wide range of estimates, and some of the evidence base is missing from the public record because of publication bias. Traditional evidence reviews are vulnerable to subjective interpretation, potentially distorting policy conclusions. Policymakers should encourage meta-regression to provide systematic reviews of the evidence and enable testing of theory and correction of biases that afflict reported econometric estimates.

Full citation

Full citation

Data source(s)

Data type(s)

Method(s)

Countries

How can we improve IZA World of Labor? Click here to take our 5-minute survey and enter a prize draw.