Much promise, little practice: counterfactual impact analysis in Austrian RTI evaluation

Kofler, Jakob and Streicher, Jürgen and Régent, Verena and Wagner, Isabella (2026) Much promise, little practice: counterfactual impact analysis in Austrian RTI evaluation. Working Paper. fteval Preprints. (Submitted)

Preview	Text (PDF file) 5902_KWA-Studie_submitted-261006.pdf Download (138kB) | Preview
	Text (Word document file) 5902_KWA-Studie_submitted-261006.docx Download (49kB)

Abstract (english)

Counterfactual impact analyses are widely regarded as the reference standard for demonstrating whether policy measures actually work. How widespread they are in the evaluation of Austrian research, technology and innovation (RTI) policy, which conditions facilitate or impede their use, and what follows for practice is the subject of a cooperative study conducted by the Austrian Platform for Research and Technology Policy Evaluation (fteval) with member institutions between July 2025 and July 2026. This article presents its findings. A systematic review of Austrian evaluation reports identified seventeen counterfactual impact analyses in fifteen publications; six of these meet the methodological minimum standard for robust causal evidence. By international comparison the absolute count is high, though in domestic practice counterfactual designs remain the exception rather than a standard instrument. Three structural bottlenecks explain the limited uptake: small sample sizes, funding accumulation, and temporal lags in secondary data. Three developments are loosening them: regulatory and disciplinary pressure, an expanded methodological toolkit, and improved data access through the Austrian Micro Data Center (AMDC). Fifteen stakeholder interviews show that limited uptake reflects institutional routines and procurement conditions rather than methodological rejection. We conclude that method choice should remain a deliberate, context-specific decision, and that triangulation with qualitative and theory-based approaches is necessary to obtain a holistic view of the impacts of RTI programmes, to understand how these impacts come about, and to identify the levers for increasing a programme's effectiveness and efficiency.

Subjects:	fteval Preprints Assessments
Divisions:	Ministry for Women, Science and Research Joanneum Research KMU Forschung Austria WPZ Research GmbH fteval - Austrian Platform for Research and Technology Policy Evaluation
Number of Pages:	9
Identification Number:	10.22163/fteval.2026.769
Date Deposited:	09 Oct 2026 07:55
URI:	https://repository.fteval.at/id/eprint/850

Actions (login required)

View Item