Through the ASSIST study sponsored by the European Space Agency (ESA), the development of a Telemedicine assessment methodology and a self-assessment tool have been carried out. ASSIST allows project managers to analyse the positive and negative impacts of their Telemedicine services way before they go live.
There have been numerous projects in the area of Telemedicine that developed fantastic technical solutions which allow moving medical information instead of people. What seems to be an engineering exercise often turns out to have much more facets than initially expected. The result is that many projects struggle to transition to viable services, as they lack some of the elements needed to complete the picture. To better understand this phenomenon and to reduce the effect of this spiral, ESA launched the AsseSSment and evaluatIon toolS for Telemedicine (ASSIST) study, sponsored by the ESA General Studies Programme (GSP).
ASSIST conducted a literature review and did not find enough information to assess a Telemedicine service in all its dimensions. A consistent approach did not exist. Many assessment methods restricted their view to analysing expenditure, but there are numerous benefits that don’t come with a price tag. For example, how do patients value the comfort of not needing to travel?
Many methods base their results on clinical trials that require time and manpower to coordinate as well as a certain degree of technical maturity. When developing a service, maturity increases with every new version, but flexibility diminishes. One would need a method that could be applied at the beginning and mature along with the service to guide the development. Nevertheless, the results should always remain accurate. Data that can only be understood internally, and is therefore difficult to translate and compare, is not useful. If the method of extrapolation instead estimates its value in a broader sense, the results can then be tested more accurately.
Another aspect is the consideration of the different stakeholders affected by the adoption of Telemedicine. Often projects overlook that there are many of them: hospitals, doctors, nurses, general practitioners (GPs), informal carers, those paying for the service, technology providers, just to name a few. And, of course, the patients. What happens, for instance, when a service can be performed by the GP instead of the hospital because the practice has a Telemedicine system supporting them in difficult cases? This may be beneficial for the practice, but is it good or bad for the hospital? Overall, bearing in mind the many different factors involved with introducing such a service, is it good or bad? What is the long-term effect? Is it expected to cause difficulties at first, and improve with time? The financial officer of the hospital might have a different answer to this than the hospital doctor freed from workload. No hospital would object to a new service that improves the bottom line, if that is all that is considered.
That is why the ASSIST project had to develop their own assessment framework. It builds on Cost-Benefit Analysis (CBA). In several countries, CBA is the recommended method for evaluation of public investments. The UK Green Book is an example. CBA looks at all impacts generated by the service whether they are investments, increased effort or improved wellbeing. It turns these impacts into monetary units which makes them comparable. The result of a CBA is the difference between positive impacts (benefits) and negative impacts (costs). Positive and negative impacts are not only of financial nature (e.g. money spent or saved), but include other aspects such as time saved for a specific stakeholder, or time needed for training. The key assessment figure calculated by ASSIST is the Socio Economic Return, which depends on the specific stakeholder, the point in time considered and what has happened until this point in time.
Every ASSIST assessment is a comparison between a given status, which the evaluator will use for comparison, and an intervention via Telemedicine. A comprehensive understanding of usual care is the best status quo measure. In the case of ASSIST, the intervention is not a single point in time but a process of changing from one status to another. It includes development, piloting and scale-up. ASSIST’s asset is that it can be used for formative evaluations projecting potential future gains and losses and direct further development and investment decisions.
ASSIST has defined 30 stakeholder stereotypes that are meaningful to Telemedicine services. An assessment does not necessarily need to make use of all 30 different types of stakeholders. Most assessments only involve four or five of them. Each stakeholder has its own set of pre-defined indicators covering the most important impacts occurring in Telemedicine. In total, ASSIST has defined 339 indicators for its 30 stakeholders. For example, an indicator such as ‘avoided admissions’ connects the output of the technical system (control of patient’s conditions) with outcomes (better health through close control) and their impact on a stakeholder (less or completely avoided time in hospital). This has an impact on:
· the patient who doesn’t need to stay in the hospital,
· the hospital that does not need to provide the service and may lose income,
· the health insurance organisation that saves money as care didn’t need to take place.
Indicators might or might not apply in the specific case. Admissions will not be avoided in every case. It could be that only the time spent in hospital is shorter, but with the proposed indicators ASSIST wants to make the users think whether this impact occurs or not. Developing and refining the indicators’ set was the biggest challenge of ASSIST. It needed to be generic enough to cover all conceivable configurations of services and to be specific enough that the user is not overwhelmed with random indicators that do not fit his situation.
The ASSIST method was implemented in a spreadsheet self-assessment tool. Giving non-health economist access to expert knowledge via an easy to use tool was a major aim of the project. The tool and method were then validated with several ESA Telemedicine pilot projects. Arnaud Runge, technical officer at ESA, said: “The validation of the ASSIST toolkit has been performed involving four pilot Telemedicine projects in different stage of maturity. It was very interesting to see that through the instantiation of this toolkit, each project team discovered important aspects about the sustainability of their service. ASSIST proved to be a tool that helps project teams question and better understand the consequences of their ideas, even before predicting the trend of the net impact brought by Telemedicine in the months to come. As a consequence, we are now systematically using this tool for our new Telemedicine projects.”
Empirica (D), the company leading the project, has made the ASSIST tool and method available under GPL Licence for everyone to use free of cost. It can be accessed under http://www.assist-telemedicine.net. Project partners were Telbios (I) and IRER (I).
For more information, please contact ESA at:
Arnaud Runge (TEC-MMG) or
Francesco Feliciani (TIA-APC)