Digitally enabled therapies assessment criteria

Overview

Digitally Enabled Therapy (DET) products deliver a substantial portion of therapy online and are designed to be used with therapist assistance. The DET Assessment Criteria have been developed to provide assurance that DETs are suitable for use in NHS Talking Therapies for anxiety and depression services.

This process gives services more confidence in the technologies they consider, select, commission and use locally with their patients. It also supports developers to design and deliver DET products which are well suited for use in NHS Talking Therapies services.

In order to ensure that therapists are well equipped to work with digital, training programmes for clinicians now include training in this way of working.

Six products have now been assessed against the Digitally Enabled Therapies (DET) Assessment Criteria. The outcomes of these assessments are summarised below. The full assessment reports are available via the NHS Talking Therapies Futures platform or by emailing england.digital.iapt@nhs.net.

Product: Deprexis
Condition: Depression
Assessment outcome: Not compliant with the DET Assessment Criteria

Deprexis is a Digitally Enabled Therapy (DET) for the treatment of depression. It creates a personalised pathway through the content based on input from the user. Deprexis is primarily designed as a self-help, rather than a guided self-help programme. Though it contains a lot of content which is compliant with NICE treatment protocols NHS Talking Therapies clinicians are trained to deliver, it also contains a substantial proportion of content which is not compliant with these. It does not therefore comply with the required clinical content DET assessment criteria. Evidence was presented demonstrating that Deprexis complies with the evidence of effectiveness criteria. However, due to the issues with the clinical content, overall, Deprexis is non-compliant with the criteria.

  • core assessment criteria: 7 criteria met, 1 criteria not-met
  • clinical content assessment criteria: 0 criteria met, 2 criteria not-met
  • evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

Product: Wysa
Condition: Generalised Anxiety Disorder
Assessment outcome: Partially compliant with the DET Assessment Criteria (emerging evidence – promising new product which is yet to be formally evaluated)

Wysa offers a range of products relevant to NHS Talking Therapies for anxiety and depression. This assessment is of their Digitally Enabled Therapy (DET) for Generalised Anxiety Disorder (GAD). The product is designed to be an interactive DET which aims to engage the user in a dialogue that simulates a therapeutic conversation and responds according to user input. It combines a series of seven ‘conversations’, each of which contains a brief introductory video, with a tool pack outlining key strategies and tools. The programme is structured, and requires people to complete four daily check-ins between ‘conversations’ with a chatbot. They can access the tool pack at anytime. Therapists are able to view their patients’ progress and data. The majority of content closely aligns with CBT protocols which NHS Talking Therapies clinicians are trained to deliver, with some additional content for example in mindfulness, which the suppliers have agreed to present only to those who opt for it.

Overall Wysa is compliant with the core and clinical content criteria. Wysa’s DET for GAD is a new product, and there is not yet sufficient research demonstrating impact.

  • core assessment criteria: 8 criteria met, 0 criteria not-met
  • clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • evidence of effectiveness criteria: 0 criteria met, 3 criteria not-met

Product: Silvercloud Space From Depression
Condition: Depression
Assessment outcome: Fully compliant with the DET Assessment Criteria

Silvercloud offers a range of products relevant to NHS Talking Therapies for anxiety and depression. This assessment is of Space from Depression, a Digitally Enabled Therapy (DET) for depression. Space from Depression comprises 7 modules which deliver CBT content consistent with the protocols NHS Talking Therapies clinicians are trained to deliver. Users are able to personalise their use of the app, and a therapist-supported version enables therapists to review patient’s progress and input. Evidence was presented demonstrating that Space from Depression can be clinically effective, achieves outcomes equivalent to NHS Talking Therapies treatments for depression, and does not cause harm.

  • core assessment criteria: 8 criteria met, 0 criteria not-met
  • clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

Product: Mahana
Condition: Irritable Bowel Syndrome
Assessment outcome: Partially compliant with the DET Assessment Criteria (Emerging evidence: promising product with evidence demonstrating positive impact but which does not fully meet the evidence of effectiveness criteria yet)

Mahana is a Digitally Enabled Therapy (DET) for the treatment of Irritable Bowel Syndrome (IBS). It comprises 9 modules of content which closely maps to the strategies employed in clinical trials of Cognitive Behavioural Therapy (CBT). Though primarily designed as a self-help tool, its content and structure mean that it could be supported by an NHS Talking Therapies clinician, provided appointments were arranged to enable this. Evaluation evidence presented demonstrates partial compliance with the criteria, but as the evidence relates to previous versions of the tools and/or use of Mahana without therapist support, further evidence is needed to fully meet this criteria. The suppliers have also been asked to remove links to websites which contain advertisements.

  • core assessment criteria: 8 criteria met, 0 criteria not-met
  • clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • evidence of effectiveness criteria: 3 criteria partially-met

Product: Minddistrict
Condition: Generalised Anxiety Disorder (GAD)
Assessment outcome: Not compliant with the DET Assessment Criteria

Minddistrict is a platform for a number of potential treatment programmes for a variety of problems and presentations. The programme reviewed here appears to be intended as a step-3 intervention for Generalised Anxiety Disorder. The programme comprises 7 modules. Clinicians are able to review patients’ progress through the programme. The content is consistent with NICE recommended protocols NHS Talking Therapies services are trained to deliver, however, it does not follow one structured protocol. Elements of different models are included, and certain techniques in recommended protocols are missing. The evidence presented does not relate specifically to the step 3 GAD module under review, and so evidence of effectiveness criteria are not currently met.

  • core assessment criteria: 7 criteria met, 1 criteria not-met
  • clinical content assessment criteria: 2 criteria partially-met
  • evidence of effectiveness criteria: 0 criteria met, 3 criteria not-met

Product: Beating the blues
Condition: Depression
Assessment outcome: Fully compliant with the DET Assessment Criteria

Beating the blues is a Digitally Enabled Therapy for the treatment of depression. It comprises 8 section, each of which contain 3-4 modules. It employs a range of Cognitive Behavioural techniques for managing depression, which closely align with the NICE compliant treatment protocols NHS Talking Therapies clinicians are trained to deliver. There are a number of interactivity and interoperability features supporting the clinician to review patient’s input and progress. Evidence was presented demonstrating compliance with the evidence of effectiveness criteria.

  • core assessment criteria: 8 criteria met, 0 criteria not-met
  • clinical content assessment criteria: 2 criteria met, 0 criteria not-met
  • evidence of effectiveness criteria: 3 criteria met, 0 criteria not-met

NICE Early Value Assessment

NICE is piloting the Early Value Assessment (EVA) which aims to conduct a rapid assessment of evidence for emerging technologies. The topics include:

What are the DET assessment criteria?

The Assessment Criteria was reviewed and updated in July 2024.

The assessment process

Two clinical assessors, individuals with expertise in the relevant treatment protocols, test the DET programme and produce a report detailing the products’ performance against each of the criteria. The report will include information from the DET supplier and the clinical assessors on the products performance against the criteria. It will also provide an overall outcome of the assessment. Assessment reports will be reviewed by a ratification panel, comprising individuals with a range of expertise including policy, clinical, research & patient and public experience. This panel will ensure that the assessment process has been consistent and fair, support the reaching of consensus if necessary and ratify the assessment decisions.  Throughout the process, the DET supplier will be notified of any concerns or issues so that they can be discussed and, if possible, rectified.

Figure 1: Digitally enabled therapies assessment processThe overall outcome of the assessment is dependant on the products’ performance against each of the criteria. The process is sequential, so a product needs to be compliant with one section of the criteria before moving on to the next. Section 1 is the evidence of effectiveness review, which has 3 possible outcomes: not compliant, partially compliant, or compliant. If a product is partially compliant, there are 2 further outcomes: - promising new product which is yet to be formally evaluated - promising product with evidence demonstrating positive impact but which does not fully meet the evidence of effectiveness criteria Note: products included in NICE Early Value Assessments will not be assessed against these criteria. Section 2 is the Clinical Content Review. This has 3 possible outcomes: not compliant, partially compliant, or compliant. If a product is not compliant or partially compliant, advice is given with an invitation to make changes. If a product is complaint, its outcome is ‘Fully compliant, subject to local completion of DTAC.’

Image description: a flow chart illustrating the two sequential sections of the digitally enabled therapies assessment process.

Products will be prioritised for assessment to ensure that:

  1. products recommended for use in the NHS in NICE Early Value Assessments are assessed
  2. products for the range of conditions NHS Talking Therapies services treat are assessed
  3. products from a range of suppliers are assessed

DET Assessment Criteria

Eligibility criteria

The following criteria must be met in order for a product to be eligible to continue through the assessment process. The assessment team will assess the supplier’s response to ascertain whether eligibility criteria are met. Products which do not meet eligibility criteria will not progress through the assessment process.

2.1 – Which of the 13 clinical conditions covered by NHS Talking Therapies is the product is designed to treat? List of conditions in Annex A.

2.2 – Is the product designed to treat adults (18+)?

2.3 – Is the product designed to deliver a NICE recommended therapeutic intervention for the condition identified in 2.2?

2.4 – Is the content of the product aligned with a NICE recommended Low Intensity (Step 2) intervention or a NICE recommended High Intensity (Step 3) intervention?

2.5 – Can the product be used in NHS Talking Therapies services with therapist assistance?

Information about functionality in the product which supports and optimises this is collected in section 3.

2.6 – The NHS Talking Therapies training/qualification requirements for clinicians supporting delivery of the product should be clear. This should include whether the supporting therapist should be a Psychological Wellbeing Practitioner (PWP) or High intensity therapist. If High Intensity, please specify which modality (e.g. CBT, Mindfulness based cognitive therapy, Brief dynamic interpersonal therapy etc).

2.7 – Will the product be made available to be commissioned by NHS organisations?

Information to support commissioning decisions

There are a number of features that commissioners will want to know about, but which are either not assessed through this process, or are not always required in order for a product to be safely and effectively used in services. Suppliers must provide information on each of these criteria so that it can be shared with the system to support commissioning decisions. The responses will not be assessed, but the assessment team may choose to share relevant comments based on their experience trialling the product.

3.1 – Has the product successfully demonstrated compliance with the national baseline digital health and care technology assessment criteria (DTAC)?

DTAC assessments must be conducted as part of the procurement process at the point of commissioning. However, suppliers are asked to confirm whether they have successfully demonstrated compliance to provide commissioners with an indicated as to their level of DTAC readiness.

For further information see DTAC guidance.

3.2 – Does the product allow the collection of NHS Talking Therapies minimum dataset outcome measures from the patient?.

For further information please see Annex B.

3.3 – Does the product present a graphical view of the sessional outcome data to the patient?

3.4 – Does the product present a graphical view of each patient’s sessional outcomes data under the therapist’s care to the therapist?

3.5 – Does the product use NHS Talking Therapies’ approved outcome measures for each condition as detailed in the NHS Talking Therapies manual?

For further information please see Annex B.

3.6 – Does the product have the capability to flow outcomes data to the NHS Talking Therapies patient record systems in a format that enables them to be automatically incorporated into routine data submissions?

3.7 – Have you (the supplier) committed to the maintenance and update of the technology and content of the product?

3.8 – Does the supplier have the capacity to deliver telephone and online helpdesk support to services and clinicians using the product?

3.9 – Does the supplier have training materials for clinicians to support them with implementation and ongoing use of the product within NHS Talking Therapies service settings?

3.10 – Are there costs for de-commissioning your product? If so, please share the details.

3.11 – Does the product have open APIs to enable interoperability between other technology products used within NHS Talking Therapies?

3.12 – Does the product collect user feedback to inform product improvement?

3.13 – Is the product designed to allow two-way communication between the therapist and patient?

3.14 – Does the product allow the therapist to view a patient’s written inputs on the programme?

3.15 – Have people with patient and public experience been involved in the development of the product?

3.16 – Have you designed the product in such a way as to reduce and/or address inequalities?

Clinical content assessment

The clinical content assessment has been developed to ensure that products used within NHS Talking Therapies services deliver evidence-based treatment protocols that NHS Talking Therapies clinicians are trained to deliver. Therefore, the content and structure of the treatment within the DET must mirror a NICE recommended psychological therapy intervention for the relevant condition.

NICE guidance provides recommendations regarding the type of therapy that is effective for conditions based on evidence derived from RCTs. The therapy protocols NHS Talking Therapies clinicians are trained to deliver are based on the therapy protocols used in those research studies. Products therefore also need to be based on these approved therapy protocols.

Clinical content will be assessed by clinical experts with substantial experience of delivering, supervising and delivering training in the relevant NHS Talking Therapies treatment. To complete the assessment, clinical assessors will require access to the product from a patient and therapist viewpoint.

4.1 – How does the product deliver an intervention recommended by NICE for the relevant clinical condition? Please detail the treatment protocol, with reference to the relevant NICE guidance’

Refer to NICE guidance for relevant clinical condition.

4.2 – Does the product deliver these interventions using approved NHS Talking Therapies treatment protocols. For reference, Step 3 protocols are detailed in the relevant NHS Talking Therapies for anxiety and depression training curriculum and Competence Frameworks, and a wealth of materials on step 2 protocols can be found on the clinical education development and research (CEDAR) website.

Refer to Annex C for NHS Talking Therapies protocol content assessments

4.3 – Does the product’s interface support the clinician to conduct a person centred risk assessment?

Evidence of effectiveness assessment

Evidence of effectiveness criteria must be met in order for products to be suitable for routine use in NHS Talking Therapies services. NHS England expects that DETs in routine use in NHS Talking Therapies services should be able to demonstrate their compliance with the following criteria.

Products which were included in NICE’s Early Value Assessments for Depression or Anxiety will not be assessed against these criteria as part of the DET Assessment Criteria process. 

  • clinical effectiveness – This can be demonstrated by one or more peer reviewed publication(s) focused  on a relevant clinical group which is representative of an NHS Talking Therapies population’.. At least one peer reviewed publication should be a randomized controlled trial (RCT) (See further information on RCT design below) but a combination of RCT and non-RCT evidence is desirable with the latter including user feedback, in-service evaluations, etc)
  • equivalent outcomes – This needs to demonstrate outcomes (reliable recovery, reliable improvement, pre-post effect sizes) achieved with use of the DET are at least as good as those obtained with the relevant non-digital equivalent in NHS Talking Therapies services. Evidence should include NHS Talking Therapies metrics calculated in the same way and using the same measures. If evidence of at least equivalent outcomes comes from benchmarking rather than RCTs, take must be taken to ensure that data for the DET and the non-digital intervention are based on similar populations
  • absence of harm – This can be demonstrated via studies which report adverse events such as reliable deterioration, drop-out rates and serious adverse events, including suicide

Ideally, suppliers of DETs will be able to present pre-registered, published, and peer-reviewed randomised control trials.

However, if these are not available, suppliers should present evidence gathered from pilots and case studies that benchmark against nationally published NHS Talking Therapies datasets available through NHS Digital, and support answering the below items. If another type of study is submitted, the evidence will still be evaluated, including the strengths and weakness of the study.

For all pieces of evidence submitted suppliers must confirm:

  • that they have submitted the full article including any supplementary information (e.g., tables, charts). Please note that abstracts will not be accepted
  • whether they have conducted any trials that yielded a null result? If so, please summarise

Whether the studies submitted include other products that are not the main product being assessed? If yes, please specify.

5.1 – Please present evidence of the product’s clinical efficacy/efficiency:

Options for trial design:

  • DET vs Waitlist with continuing GP care (DET needs to be superior to Wait);
  • DET versus well -conducted NICE recommended alternative treatment (DET needs to be non-inferior or superior to the other treatment)
  • DET versus an alternative intervention that controls for non-specific therapy effects (such as therapist attention etc). Here the DET should be superior to the “placebo” control.
  • Other – if another methodology was used, please specify

5.2 – Please present results on equivalent outcomes.

5.3 – Results on absence of harm and clinical safety.