How to set your evaluation criteria when buying Digital Outcomes and Specialists services
How to set what you’ll evaluate digital outcomes, digital specialists, user research participants and studio suppliers on.
You must publish your Digital Outcomes and Specialists’ evaluation criteria with your requirements on the Digital Marketplace. When suppliers apply for the work, this will help them understand how you’re going to evaluate them later. You can’t change your evaluation criteria after you’ve published them.
What you’ll evaluate on
You need to set evaluation criteria for:
- technical competence, for example how well the supplier’s skills or proposal meet your needs
- cultural fit, for example how the supplier will work in your organisation (for outcomes and specialists only)
- availability (for user research participant recruitment)
- price of the proposal
Technical competence
Technical competence is how well the supplier can do the work you need them to do. It’s also known as ‘technical merit and functional fit’. If a supplier has the right skills and experience, a good understanding of your requirements, and works in an agile way, they should have the technical competence to do the work.
You must assess a supplier’s technical competence by reviewing their:
- essential skills and experience
- nice-to-have skills and experience
- proposal
Skills and experience criteria
List your essential and nice-to-have skills and experience criteria, for example:
- the supplier must have experience designing services for users with low digital literacy
- it would be good if the supplier has evidence of delivering at scale
The more specific you are, the easier it will be to shortlist suppliers after the closing date. Add more essential and nice-to-have skills and experience for the things that are important to you. This will give suppliers clarity on what you’re looking for and exclude more suppliers that don’t meet your needs.
Proposal criteria
List the criteria you’ll use to evaluate the proposal, for example:
- the proposed technical solution
- the proposed approach and methodology
- how the approach or solution meets your business or policy goal
- how the approach or solution meets user needs
- estimated timeframes for the work
- how they’ve identified risks and dependencies and offered approaches to manage them
- team structure
- value for money
Cultural fit
Only specialist and outcomes suppliers are scored on cultural fit. Cultural fit is about how well you and the supplier work together.
Cultural fit is:
- how the supplier works with other people
- how the supplier solves problems
- the supplier’s approach to making decisions
- how the supplier shares knowledge and experience
- the supplier’s attitude to making mistakes
Cultural fit isn’t:
- whether the supplier looks, talks or behaves in the same way as you
- about having agile skills – that’s technical competence
List the criteria you’ll use to evaluate cultural fit, for example:
- work as a team with our organisation and other suppliers
- be transparent and collaborative when making decisions
- have a no-blame culture and encourage people to learn from their mistakes
- take responsibility for their work
- share knowledge and experience with other team members
- challenge the status quo
- be comfortable standing up for their discipline
- can work with clients with low technical expertise
Availability
Only user research participant suppliers are scored on availability. Availability is whether the supplier can recruit user research participants when you need them.
Price
Price is scored based on how close each supplier’s quote is to the cheapest supplier’s quote.
You need to tell outcomes suppliers how you want them to provide a quote for their proposal. They can quote for ‘fixed price’, ‘time and materials’ or ‘capped time and materials’.
Fixed price
A fixed price approach can be more expensive because the supplier owns most of the risk. It works best when your requirements are well defined so the supplier can provide an accurate quote without incorporating extra costs.
The supplier must do all the work you specify in each particular statement of work within the time you agreed it would take.
Time and materials
A time and materials approach means you own more of the risk. The supplier can send a pricing matrix showing how long they need to provide the work. If it takes longer to do the work defined in a statement of work, you pay the day rate and expenses for any extra days worked.
Capped time and materials
A capped time and materials approach is the same as a time and materials approach but there’s a limit on how much you have to pay for the work. If you reach the limit before the work is finished, the supplier has to complete the work at their own cost. If the supplier finishes the work before they said they would, you only pay them for the time they took to do the work.
Digital specialist prices
You must pay specialist suppliers for the time they work using the day rate agreed in the contract. The day rate excludes VAT, travel and expenses. Travel and expenses should be agreed before invoicing and capped according to your organisation’s expense policy.
Participant recruitment prices
The price participant recruitment suppliers provide includes all incentives, recruitment, and travel and subsistence costs paid to participants. You don’t pay for participants who don’t attend on the day. Suppliers are responsible for paying participants directly.
Research studio prices
Studio suppliers provide a total cost for the studios based on the facilities you need and how long you need them for.
Read more about paying for Digital Outcomes and Specialists services.
Choosing your evaluation criteria weighting
Depending on the service you’re buying, you’ll need to give a ‘weighting’ to technical competence, cultural fit, availability or price. You should set your weightings by deciding their relative importance to you, for example if price is most important to you, give it a higher weighting than technical competence and cultural fit.
For projects with a large budget or a long contract, the proposals may be very different from each other. You should place more importance on technical competence for projects like this.
For simple requirements when you know that many suppliers could meet your needs, price may be more important to you than technical competence.
If you need user research participants and your user research dates are fixed, availability may be more important to you than technical competence.
Weightings you can use
Your technical competence, cultural fit, availability and price weightings must be within a fixed range. When you add your weightings together, they must add up to 100%.
Weightings for technical competence must be between:
- 10% and 75% for outcomes and specialists
- 15% and 75% for user research studios
- 10% and 70% for user research participants
Weightings for cultural fit must be between 5% and 20%.
Weightings for availability must be between 10% and 70%.
Weightings for price can be between:
- 20% and 85% for outcomes, specialists and user research participants
- 25% and 85% for user research studios
Giving points to individual criteria
If you want to give points to the individual criteria in technical competence and cultural fit, you must tell suppliers the points you’ll give to each one when you publish your criteria, for example:
- the supplier must have experience designing services for users with low digital literacy - 15 points
- it would be good if the supplier has evidence of delivering at scale - 5 points
- the proposed technical solution - 20 points
The points you give for nice-to-have skills and experience must be lower than the points you give to the supplier’s essential skills and experience and proposal.
Read about:
Updates to this page
Last updated 1 October 2019 + show all updates
-
Weightings for technical competency must be between 10% and 70% for user research participants. It was previously between 10% and 75%.
-
First published.