Evidence Generation for Digital Care Interventions

The Problem with RCTs

The Randomised Controlled Trial is appropriate for pharmaceutical interventions where a single variable can be isolated. It is structurally inappropriate for complex, platform-based digital care interventions for four reasons: networks cannot be decomposed; personalised packages evolve and cannot be frozen for evaluation; social learning is the intervention itself; and freezing development to conduct an RCT slows adoption in the comparator conditions.

"Realistic Evaluation changes the question from 'did it work?' to 'what works for whom, in what ways and under what conditions?'" — Pawson and Tilley (1997)

A Sociotechnical Approach to Evidence

The alternative proposed across this knowledgebase is a practice-based, action-research model grounded in Developmental Evaluation (Patton, 2010) and Realistic Evaluation. Key principles: evidence is integral to the intervention, not a separate activity conducted after deployment; formative and summative evidence coexist; context is not controlled for — it is engaged with; and evidence must meet the needs of all Gateholders.

Two-Level Evidence Architecture

First level (in-silo): Evidence specific to a particular disease pathway, care setting or population. Meta-level (cross-silo): Pooled evidence from multiple deployments, analysed for aggregate information about the platform's value across integrated care. The meta-level is where the economic case for platform investment is made: individual silo-level evidence is insufficient because the costs and benefits of a platform are distributed across the care system.

Five Classes of Evidence

Class A — Activity Data: Platform usage patterns, call volumes, data generated, engagement rates. Class B — Service Reviews: Planned reviews of service change. Class C — Sociotechnical Evaluations: Gateholder experience of the technology. Class D — Social Impact: Dignity, confidence, independence, family engagement (linked to Social Value Act 2015). Class E — System Impact: Statutory data including A&E attendances, GP visits, length of stay, unplanned admissions, care package costs.

Instrumented Platforms

The practical mechanism for embedding evidence generation is the instrumented platform — a digital platform with automated, granular data capture built into its core. The v-connect call records, the Lincus RAVEN analytics engine, and the digitised Health Equalities Framework (eHEF) together created, in the BOLD-TC project, a multi-angle assessment of care delivery with no known precedent.

Barriers to Evidence Generation in Practice

The Local Investment Programme evaluation documents a systematic pattern of evidence failure across 19 LIP councils: most projects failed to put adequate data collection mechanisms in place from the outset; many changed their outcome targets during the project; the 12-month timescale was consistently too short; GDPR implementation disrupted existing arrangements; and DARS approval took up to 15 months. The prescription: a discovery phase is essential; baseline data must be collected before the intervention begins; outcome measures must be agreed with all Gateholders at design stage.