Guidance

Abdominal aortic aneurysm screening equipment evaluation report 2019

Published 23 August 2019

1. Summary

This report describes the evaluation exercise undertaken to select the next generation of ultrasound scanners to be used in the NHS Abdominal Aortic Aneurysm (AAA) Screening Programme and the Wales Abdominal Aortic Aneurysm Screening Programme.

We invited equipment suppliers to price against a comprehensive set of specification criteria developed by a multi-disciplinary team of experts in ultrasound, national screening and equipment procurement. The team included staff working for local screening providers.

At the end of the evaluation period, 2 machines emerged as being best matched to the requirements of the programmes. These were the MIS Samsung HM70A with Plus and the Mindray M8.

2. Background

The previous equipment evaluation for the NHS AAA Screening Programme was completed in 2014 and for the Wales AAA Screening Programme in 2012.

The joint 2019 equipment evaluation exercise built on that work. It is anticipated that the working life of the new ultrasound systems will be approximately 5 to 6 years.

3. Equipment

The selection of imaging equipment to be used in a population screening programme must satisfy a number of criteria. These include:

• performance and sensitivity
• portability of equipment from site to site
• ease of use for sonographers
• safety
• reliability
• one touch image storage
• IT connectivity
• cost
• easy access to servicing and repair

4. Project stages

4.1 Agree the composition of the working group

The working group incorporated a wide range of experts including ultrasound users, educators, trainers, physicists, NHS supply chain staff and core members of the national screening programmes.

They were all given equal status regardless of employment grade. In particular, it was essential to have the input of screening technicians and clinical skills trainers as they use the equipment on a daily basis.

4.2 Development of the 2019 specification

The equipment specification document developed by the NHS AAA Screening Programme in 2014 was used as a starting point to develop the new specification. This was amended to:

• include advances in portable ultrasound technology
• reflect the working practices of local programmes

4.3 Issue specification to suppliers

The working group developed a system for scoring the written supplier responses against the specification criteria.

A score of 2 represents ‘fully meets the criterion’, 1 represents ‘partially meets’ and 0 represents ‘does not meet’.

The NHS Supply Chain issued the specification to ultrasound suppliers on the NHS Supply Chain framework with a 2-week deadline for response.

4.4 Evaluation scoring method

A 3-stage evaluation process was agreed.

1. The first stage was paper-based, scoring the manufacturers’ written responses against ‘must have’ and ‘should have’ specification criteria. Any machines that did not meet all of the ‘must have’ criteria were not evaluated further.

2. Qualifying manufacturers were invited to bring their machines along to an evaluation day where they were again scored by the working group using both ‘must have’ and ‘should have’ criteria and using the 2,1,0 scores as described above.

3. The 3 top scoring suppliers conducted field testing with local services to test the equipment in a clinical environment. The field tests were centred around 5 main categories: image quality, ease of use, physical features, workflow management, portability and ease of cleaning. To allow for greater discrimination, we used a 0 to 9 scoring range and a weighting was attached to the priority elements of the testing. The field tests were carried out by experienced technicians using the equipment in clinics with invited members of the public, some of whom were already on surveillance.

5. NHS Supply Chain and initial evaluation of written submissions

We evaluated a total of 14 systems to identify equipment that met the specification best while also considering the overall cost of procurement and maintenance.

We agreed with NHS Supply Chain that suppliers should only present their best option at the evaluation day.

Two systems did not meet the criteria at the initial written evaluation stage due to:

• excessive weight and size
• not providing an ethernet connection

We told the suppliers of both these systems they would not be invited to the evaluation day for these reasons.

NHS Supply Chain invited the remaining 8 manufactures to bring their equipment, including any trolleys or transport cases, to a practical evaluation day to allow for thorough testing and evaluation of the equipment, including its portability between scanning venues.

6. Practical evaluation day

We divided the evaluation process into 5 main workstreams.

1. Technical evaluation by a team of experienced sonographers and screening technicians looked at scanner performance, portability and ergonomics. This included using a volunteer from the local screening service.

2. Medical physics assessed the image quality and features of the machines using a phantom.

3. IT evaluation looked at connectivity, image storage and transfer via networks. This included one PHE AAA IT expert and a Northgate AAA expert resource. The connectivity and IT compatibility of the machines were tested using a pass or fail result.

4. Clinical evaluation of the machines for their clinical performance, system display, usability and safety using a volunteer.

5. Management evaluation considered maintenance agreements, software enhancements, usability and infection control.

The local AAA service invited men under surveillance to act as volunteer test subjects for the technical and clinical teams.

We developed a timetable for the evaluation day with clearly defined time slots for each system to be evaluated. Thirteen experts from different disciplines assessed the 8 machines over the course of the day using an agreed scoring evaluation sheet. These were collated to produce an overall score.

Following collation of the scoring evaluation sheet results the working group reached agreement on reducing the 8 machines down to 3. The suppliers that were not selected were provided with a feedback sheet showing where the equipment had not met the criteria as successfully as the selected 3.

7. Field testing

A further evaluation phase allowed technicians, clinical skills trainers and others to use the equipment in screening and surveillance clinics. Following consultation between local AAA services and national programme managers, we agreed to hold field testing at 3 sites in the north-east of England, central England and south-east Wales.

We asked each supplier to attend each local service in turn during June 2019. The supplier application specialists set the machines up and trained the technicians so field testing could be carried out.

The equipment was then used for a minimum of 2 full days of scanning with screening and surveillance patients.

The working group developed a field test scoring sheet and included an element of weighting to make sure the highest priority elements of the requirements were taken into account.

8. Results

We collated and distributed scoring results and feedback from the field test services to all working group members and field test programme managers.

The results were discussed in detail and a consensus reached based on considerations of the combined results of the supplier evaluation day and the field tests.

The MIS Samsung HM70A with Plus and the Mindray M8 equipment were chosen as the approved ultrasound machines for AAA screening providers in England and Wales for the next 5 years.

Providers can now progress with plans to procure ultrasound equipment from one of these 2 suppliers.

Trusts can contact NHS Supply Chain to purchase either system by emailing womenshealth@supplychain.nhs.uk.

Providers should be aware that the standard maintenance agreements provided by the suppliers for this exercise should be considered and discussed on an individual basis with the suppliers to make sure they meet local programme requirements.

A further evaluation exercise will be required to report in August 2024.

9. Composition of the working group

Lisa Summers (NHS AAA Screening Programme Manager)

Llywela Wilson (Wales AAA Screening Programme Manager

Andrea Thomas (Wales AAA Screening Programme Quality, Education and Training Lead)

Colin Nice (consultant interventional radiologist)

Tim Hartshorne (chief clinical vascular scientist)

Lynda Pike (Torbay and South Devon AAA Screening Programme Manager)

Clarrissa Sanders (medical physicist)

Owen White (medical physicist)

Alex Thompson (AAA screening technician team leader)

Libby Lockwood (clinical skills trainer)

Phil Gardner (screening IT systems manager)

Kimberley Virr, Michelle Walker and Jon Lambert (NHS Supply Chain)

Sam Mullins (PHE project support)

Phil Walker (PHE project manager)