Digitally Enabled Therapies assessment criteria

Overview

Digitally Enabled Therapy (DET) products deliver a substantial portion of therapy online but are designed to be used with therapist assistance. The DET Assessment Criteria have been developed to provide assurance that DETs are suitable for use in NHS Talking Therapies for anxiety and depression services.  The assessment criteria are now being piloted, with products being assessed against the criteria. The results of the pilot will be published in early 2024.

This process will support services to have more confidence in the technologies they consider, select, commission and use locally with their patients. It will also support developers to design and deliver DET products which are well suited for use in NHS Talking Therapies services.

This builds on previous work the national team completed with NICE, to assess and evaluate digitally enabled therapy products.

In order to ensure that therapists are well equipped to work with digital, their training programmes have been updated to cover this new way of working.

Six products have now been assessed against the Digitally Enabled Therapies (DET) Assessment Criteria. The outcomes of these assessments are summarised below. The full assessment reports are available via the NHS Talking Therapies Futures platform or by emailing england.digital.iapt@nhs.net

Many of the products assessed were also included in NICE’s Early Value assessments on products for depression and anxiety.

Product: Deprexis
Condition: Depression
Assessment outcome: Not compliant with the DET Assessment Criteria

Deprexis is a Digitally Enabled Therapy (DET) for the treatment of depression. It creates a personalised pathway through the content based on input from the user. Deprexis is primarily designed as a self-help, rather than a guided self-help programme. Though it contains a lot of content which is compliant with NICE treatment protocols NHS Talking Therapies clinicians are trained to deliver, it also contains a substantial proportion of content which is not compliant with these. It does not therefore comply with the required clinical content DET assessment criteria. Evidence was presented demonstrating that Deprexis complies with the evidence of effectiveness criteria. However, due to the issues with the clinical content, overall, Deprexis is non-compliant with the criteria.

  • Core assessment criteria: 7 criteria met, 1 criteria not-met
  • Clinical content assessment criteria: 0 criteria met, 2 criteria not-met
  • Evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

Product: Wysa
Condition: Generalised Anxiety Disorder
Assessment outcome: Partially compliant with the DET Assessment Criteria (emerging evidence – promising new product which is yet to be formally evaluated)

Wysa offers a range of products relevant to NHS Talking Therapies for anxiety and depression. This assessment is of their Digitally Enabled Therapy (DET) for Generalised Anxiety Disorder (GAD). The product is designed to be an interactive DET which aims to engage the user in a dialogue that simulates a therapeutic conversation and responds according to user input. It combines a series of seven ‘conversations’, each of which contains a brief introductory video, with a toolpack outlining key strategies and tools. The programme is structured, and requires people to complete four daily check-ins between ‘conversations’ with a chatbot. They can access the toolpack at anytime. Therapists are able to view their patients’ progress and data. The majority of content closely aligns with CBT protocols which NHS Talking Therapies clinicians are trained to deliver, with some additional content for example in mindfulness, which the suppliers have agreed to present only to those who opt for it.

Overall Wysa is compliant with the core and clinical content criteria. Wysa’s DET for GAD is a new product, and there is not yet sufficient research demonstrating impact.

  • Core assessment criteria: 8 criteria met, 0 criteria not-met
  • Clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • Evidence of effectiveness criteria: 0 criteria met, 3 criteria not-met

Product: Silvercloud Space From Depression
Condition: Depression
Assessment outcome: Fully compliant with the DET Assessment Criteria

Silvercloud offers a range of products relevant to NHS Talking Therapies for anxiety and depression. This assessment is of Space from Depression, a Digitally Enabled Therapy (DET) for depression. Space from Depression comprises 7 modules which deliver CBT content consistent with the protocols NHS Talking Therapies clinicians are trained to deliver. Users are able to personalise their use of the app, and a therapist-supported version enables therapists to review patient’s progress and input. Evidence was presented demonstrating that Space from Depression can be clinically effective, achieves outcomes equivalent to NHS Talking Therapies treatments for depression, and does not cause harm.

Overall Wysa is compliant with the core and clinical content criteria. Wysa’s DET for GAD is a new product, and there is not yet sufficient research demonstrating impact.

  • Core assessment criteria: 8 criteria met, 0 criteria not-met
  • Clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • Evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

Product: Mahana
Condition: Irritable Bowel Syndrome
Assessment outcome: Partially compliant with the DET Assessment Criteria (Emerging evidence: promising product with evidence demonstrating positive impact but which does not fully meet the evidence of effectiveness criteria yet)

Mahana is a Digitally Enabled Therapy (DET) for the treatment of Irritable Bowel Syndrome (IBS). It comprises 9 modules of content which closely maps to the strategies employed in clinical trials of Cognitive Behavioural Therapy (CBT). Though primarily designed as a self-help tool, its content and structure mean that it could be supported by an NHS Talking Therapies clinician, provided appointments were arranged to enable this. Evaluation evidence presented demonstrates partial compliance with the criteria, but as the evidence relates to previous versions of the tools and/or use of Mahana without therapist support, further evidence is needed to fully meet this criteria. The suppliers have also been asked to remove links to websites which contain advertisements.

Overall Wysa is compliant with the core and clinical content criteria. Wysa’s DET for GAD is a new product, and there is not yet sufficient research demonstrating impact.

  • Core assessment criteria: 8 criteria met, 0 criteria not-met
  • Clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • Evidence of effectiveness criteria: 3 criteria partially-met

Product: Minddistrict
Condition: Generalised Anxiety Disorder (GAD)
Assessment outcome: Not compliant with the DET Assessment Criteria

Minddistrict is a platform for a number of potential treatment programmes for a variety of problems and presentations. The programme reviewed here appears to be intended as a step-3 intervention for Generalised Anxiety Disorder. The programme comprises 7 modules. Clinicians are able to review patients’ progress through the programme. The content is consistent with NICE recommended protocols NHS Talking Therapies services are trained to deliver, however, it does not follow one structured protocol. Elements of different models are included, and certain techniques in recommended protocols are missing. The evidence presented does not relate specifically to the step 3 GAD module under review, and so evidence of effectiveness criteria are not currently met.

  • Core assessment criteria: 7 criteria met, 1 criteria not-met
  • Clinical content assessment criteria: 2 criteria partially-met
  • Evidence of effectiveness criteria: 0 criteria met, 3 criteria not-met

Product: Beating the blues
Condition: Depression
Assessment outcome: Fully compliant with the DET Assessment Criteria

Beating the blues is a Digitally Enabled Therapy for the treatment of depression. It comprises 8 section, each of which contain 3-4 modules. It employs a range of Cognitive Behavioural techniques for managing depression, which closely align with the NICE compliant treatment protocols NHS Talking Therapies clinicians are trained to deliver. There are a number of interactivity and interoperability features supporting the clinician to review patient’s input and progress. Evidence was presented demonstrating compliance with the evidence of effectiveness criteria.

  • Core assessment criteria: 8 criteria met, 0 criteria not-met
  • Clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • Evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

What are the DET assessment criteria?

The criteria are divided into 5 sections.

Section 1: Company and product information

This section captures relevant information about the product.

1.1 – please provide the name of your company

1.2 – please provide the name of the product to be assessed and what NHS Talking Therapies condition it has been developed to treat

1.3 – please provide the name and job title of the individual who will be the key contact at your organisation

1.4 – please provide key contact’s email and phone number

1.5 – briefly describe the product and how it fits into the NHS Talking Therapies care pathway (in no more than 1,000 words)

Section 2: Core assessment criteria

This sections is developed to deliver a yes/no response and must be met before the product can be considered for the clinical content assessment.

2.1 – has the product passed the national baseline digital health and care technology assessment criteria (DTAC)?  For further information see DTAC guidance

2.2 – which of the 13 clinical conditions covered by NHS Talking Therapies is the product is designed to treat?  List of conditions in Annex A

2.3 – is the product designed to treat adults (18+)?

2.4 – is the product designed to deliver a NICE recommended therapeutic intervention?

2.5 – is the product designed to be used as a Low Intensity (Step 2) or High Intensity (Step 3) intervention?

2.6, Is the product designed to be used in NHS Talking Therapies services with therapist assistance? Definition of assistance: Product use is based on therapist review of a patient’s progress and the therapist regularly (weekly or bi weekly) interacting with a patient about their progress

2.7 – the NHS Talking Therapies training/qualification requirements for clinicians supporting delivery of the product should be clear. This should include which step (2 or 3) and, if step 3, which modality (e.g. CBT).

2.8 – will the product be made available to be commissioned by NHS organisations?

Section 3: Self-reported assessment criteria

This section captures information to indicate the desirable capabilities of products. These criteria are not pass/fail, but developers are obliged to disclose their answers to support commissioner decision making.

3.1 – does the product allow the collection of sessional outcome data from the patient? For further information please see Annex B

3.2 – does the product present a graphical view of the sessional outcome data to the patient?

3.3 – does the product present a graphical view of each patient’s sessional outcomes data under the therapist’s care to the therapist?

3.4 – does the product present a graphical view of each patient’s level of risk under the therapist’s care to the therapist?

3.5, Does the product indicate if a patient has increased risk over the course of treatment to the therapist?

3.6 – does the product use NHS Talking Therapies approved outcome measures for each condition as detailed in the NHS Talking Therapies manual? For further information please see Annex B

3.7 – does the product have the capability to flow outcomes data to the NHS Talking Therapies system suppliers in a format that they can be incorporated into the NHS Talking Therapies minimum dataset?

3.8 – have you (the supplier) committed to the maintenance and update of the technology and content of the product?

3.9 – does the supplier have the capacity to deliver telephone and online helpdesk support

3.10 – does the supplier have training materials for the implementation and ongoing use of the product within NHS Talking Therapies settings

3.11 – are there costs for de-commissioning your product? If so, please share the details

3.12 – does the product have open APIs to enable interoperability between other technology products used within NHS Talking Therapies?

3.13 – have service users been involved in the development of the product? Please outline how you ensure the product meets user needs and how you engage service users

3.14 – does the product collect user feedback to inform product improvement?

3.15 – is the product designed to allow two-way communication between the therapist and patient?

3.16 – does the product allow the therapist to view a patient’s written inputs on the programme?

Section 4: Clinical content assessment

This sections is developed to ensure that products used within NHS Talking Therapies services meet the NHS Talking Therapies principles of providing evidence-based treatment. Therefore, the content and structure of the treatment within the DET must mirror a NICE recommended psychological therapy intervention for the relevant condition. The therapy protocols used by NHS Talking Therapies practitioners and services are based on these research study therapy protocols. Any product used in NHS Talking Therapies needs to also be based on these approved therapy protocols.

4.0 – Please provide log in access to both the patient and therapist portal. Please outline the details required to log in here, including any links.

4.1 – How does the product deliver an intervention recommended by NICE for the relevant clinical condition? Please detail the treatment protocol, with reference to the relevant NICE guidance’Refer to NICE guidance for relevant clinical condition

4.2 – Does the product deliver these interventions using approved NHS Talking Therapies treatment protocols (as detailed in the relevant NHS Talking Therapies Training curriculum and Competence Frameworks)  Refer to Annex C for NHS Talking Therapies protocol content assessments

Section 5: Evidence of effectiveness assessment

This section is designed to ensure that products have an adequate level of evidence for use within NHS Talking Therapies services.

5.1 – Please present evidence of the product’s clinical efficacy/efficiency: Options for trial design: DET vs Waitlist with continuing GP care (DET needs to be superior to Wait); DET versus well -conducted NICE recommended alternative treatment (DET needs to be non-inferior or superior to the other treatment)DET versus an alternative intervention that controls for non-specific therapy effects (such as therapist attention etc). Here the DET should be superior to the “placebo” control. Other – if another methodology was used, please specify

5.2 – Please present results on equivalent outcomes

5.3 – Please present results on absence of harm and clinical safety:

How are DETs assessed?

The overall outcome of the assessment is dependant on the products’ performance against each of the criteria. The process is sequential so a product needs to be compliant with one section of the criteria before moving on to the next.

The national team coordinate and support completion of a Digital Technology Assessment Criteria (DTAC) assessment. The NHS Talking Therapies Review and the Clinical Evidence review are carried out by appointed clinical assessors with expertise in the relevant NHS Talking Therapies competency frameworks and treatment protocols. A ratification panel then reviews and ratifies assessments to ensure products have been assessed fairly and consistently.

NICE’s Early Value Assessment

NICE are piloting the Early Value Assessment (EVA) which aims to conduct a rapid assessment of evidence for emerging technologies. The topics include:

Many products will be in scope for both processes, and offer different assessments to help strengthen the robustness of assessment.