Research and analysis

Appendix 2: Research method

Published 29 October 2021

1. Rapid evidence assessment

The rapid review was conducted to review evidence published from January 2000 to June 2020. It was based on rapid evidence assessment (REA) methods, and the rapid evidence assessment guidance produced by the EPPI-Centre for Civil Service on rapid evidence assessment (Civil Service, 2014) was used as a framework for the review.

1.1 Inclusion criteria for review

In order to complete the REA within the time available only empirically based evidence identified within the available time was included to address the first 3 research questions (RQ1, RQ2, RQ3)

The following inclusion criteria were used for academic literature:

• empirical studies (quantitative or qualitative), case studies, and systematic reviews of literature relating to conversion therapy that address one or more of the research questions

• studies published in English since 1 January 2000 that were obtainable within the time available

• studies published in academic journals or on the websites of professional organisations

• studies from any country

• studies on any form of conversion therapy

The following exclusion criteria were applied:

• narrative and systematic literature reviews
• books and extracts from books
• book reviews
• editorials and commentaries
• studies not published in English
• studies published prior to 2000
• studies that did not provide data that would help answer the research questions

An additional search of the grey literature was included to address the final research question (RQ4) on international practice. For RQ4 the inclusion criteria were that the evidence related to legislative measures to end conversion therapy. The mere expression of opposition to the practice of conversion therapy – for example, government ministers announcing their opposition, or psychological professional bodies expressing opposition – were excluded.

Search strategy, screening and data extraction

3 approaches were taken for identifying evidence: academic database searches, a call for evidence and a grey literature search.

A systematic search was conducted of the following academic databases:

• PsychARTICLES
• PsychINFO
• MEDLINE
• CINAHL Complete
• Academic Search Complete
• Proquest Central
• Scopus

Search terms were selected based on those used in previous reviews of conversion therapy in consultation with an psychology subject librarian and agreed in conjunction with GEO. Search terms used included ‘conversion therapy’ and its synonyms (‘reparative therapy’, ‘ex-gay’, ‘reorientation therapy’, ‘change efforts’, ‘cure therapy’, ‘change therapy’, ‘reintegrative therapy’, ‘corrective rape’) in combination with ‘sexual orientation’ or ‘gender identity’ and synonyms (‘sexual minority’, ‘lgbt’, ‘lgb’, ‘gay’, ‘lesbian’, ‘bisexual’, ‘transex’, ‘transgender’, ‘homosex*’). All articles which included the search terms in the title, abstract or keywords were retrieved. A full reference list search was not conducted due to the time limitations for completion of the rapid evidence assessment.

A call for evidence was sent using 2 email lists for psychologists specialising in sexual orientation, gender identity or LGBT issues (one UK based and one international), an email list for UK LGBT health researchers and to representatives of the MoU on conversion therapy working group.

Once the academic literature search was complete the retrieved items were screened. First, the titles and abstracts were reviewed against the inclusion and exclusion criteria. Full texts of articles were then obtained. The full texts were read and compared against the inclusion and exclusion criteria. If papers were judged not to meet the inclusion criteria they were excluded. The original search was run at the beginning of the project in January 2019 for the period 1 January 2000 to 31 December 2018. The search was then re-run for the period 1 January 2019 – 30 June 2020 to capture any studies published in the intervening period so the rapid evidence assessment was as up-to-date as possible. This resulted in 46 articles being identified in total.

The strengths and weaknesses of each study were critically appraised in relation to methodological issues. The Mixed Methods Appraisal Tool (MMAT) was used to critically appraise the articles (Pluye and others, 2011). The MMAT can be used to appraise most types of methodology and design. Due to the lack of evidence for answering some research questions, no studies were excluded on the basis of methodological weaknesses. Instead, methodological weaknesses within the literature are addressed in the reporting of findings. All articles were reviewed by one reviewer and a sample of 10% were independently assessed by a second reviewer to ensure consistency.

Data was extracted according to the author and year, study design, number of participants and findings relevant to each research question. In order to produce the final report, the data collected for each of the research questions were synthesised. The data were explored for patterns and the findings of studies were interpreted in light of the findings of other studies.

2. Quality of studies

The quality of studies was generally found to be average using the MMAT tool with the quality of studies assessing effectiveness being lower due to design limitations (see Table 8 for common methodological limitations).

2.1 Methodological limitations of published studies

A lack of prospective, controlled studies

There is a lack of randomised controlled trials that involve random allocation of participants to treatment groups and a control condition or studies that take measurements pre and post treatment. As such, it is not possible to make robust claims of effectiveness or make robust causal inferences.

Studies rely on retrospective self-reports

Any ‘pre-test’ measures are usually based on remembering how they felt prior to the therapy. This makes the findings vulnerable to biases deriving from the individual justifying to themselves the time, effort and money they’ve invested in the therapy, and the individual perceiving change because they have been led to expect change. Studies typically don’t use objective measures. Findings from qualitative studies may not be accurate due to the unreliability of recall.

A lack of studies that follow individuals over time

Most studies take measures at a single time point either during or post intervention. It is therefore not possible to assess the long-term effects of conversion therapies or whether any reported changes are sustained in the long term.

Studies use different measures of ‘success’

Measures used typically relate to sexual attraction, sexual behaviour or sexual identity. However, changes in sexual behaviour or identification are not a valid indicator that a change in sexual orientation has occurred. Many studies address one or several of these outcomes.

Studies include a wide variety of conversion therapy methods

Studies adopt broad definitions of conversion therapy with varied approaches and techniques included within a single study. As such, it is difficult to attribute any efficacy, benefit or harm to particular intervention approaches, components or providers.

Studies rely on self-selecting samples

Studies generally do not use recruitment strategies designed to obtain representative samples. Studies often rely on self-selected sampling and recruit either via conversion therapy networks and or LGBT networks. Such sampling strategies may introduce sampling biases that make it difficult to generalise the findings to the wider population.

To answer RQ4, which relates to measures taken around the world, a search of the grey literature was required as none of the studies from the academic literature could adequately answer this question or provide up-to-date information. This involved searching the websites of organisations including the International Lesbian, Gay, Bisexual, Transgender and Intersex Association (ILGA), entries for conversion therapy in Wikipedia and Equaldex (a wiki-style collaborative LGBT rights knowledge base) and media sources via Openly (a global digital platform by Thompson Reuters for LGBT+ news).

Efforts were made to verify information on these websites via accessing original sources, triangulation (for example, by comparing information from one source with information from another source) and through contact with international experts compiling similar information.

As RQ4 relates to the legal status of conversion therapy in different countries, many original sources were not in English. Where possible sources were translated to verify information from the original source. Every effort has been made to ensure the accuracy of information presented however in some cases we had to rely on secondary sources that may not always be accurate.

3. Qualitative interviews

For the qualitative study, semi-structured interviews were carried out with 30 individuals with experience of efforts to change sexual orientation or gender identity. Prior to data collection ethical approval was granted through Coventry University’s institutional ethics procedures. Interviewees were provided with an information sheet and consent form that explained the purpose of the study, what would be involved and how their data would be handled.

Most interviews were conducted either face-to-face, through video calls or by phone. Interviews were conducted between April and July 2019. The average length of interviews was 62 minutes (ranging from 30 to 102 minutes). Interviews were audio recorded with the interviewees’ consent and fully transcribed.

2 participants with autism chose to provide written responses to the questions instead of a verbal interview. Interviewees were asked about their personal background, the circumstances leading up to their experiences of conversion therapy, what it involved and what the outcome was. Following the interview, interviewees were given a £10 shopping voucher to thank them for their time and were provided with information signposting them to sources of support. Interview data were anonymised to protect their identities.

3.1 Sampling and sample characteristics

A wide range of recruitment strategies were used:

• a call for participants was distributed through social media and via professional email lists for psychologists and those working in the a rapid evidence assessment of LGBT health

• a variety of stakeholder organisations and individuals were contacted about the study including LGBT organisations, religious organisations, networks for LGBT+ people of various religious faiths as well as mental health professional bodies

• The Ozanne Foundation^{[footnote 1]} contacted respondents of their faith and sexuality survey who had reported experiencing conversion therapy and had consented to be contacted regarding future research

• fliers about the research were distributed at 2 pride festivals (Birmingham and Coventry) that coincided with the data collection period – as the word ‘therapy’ may not accurately reflect the nature of all conversion efforts, we specifically invited anyone who had experienced ‘efforts to change their sexual orientation and gender identity’

Potential participants were directed to a screening survey to register their interest in taking part in the project, where they were provided with full information about the research. If they were willing to take part after reading the information they were asked to provide some information about themselves (including age, gender, sexual orientation, gender identity, ethnicity, religion), some basic information about the conversion therapy (for example, how long ago they experienced conversion therapy) and their contact details.

Screening was conducted to screen for eligibility and to help recruit a sample of participants with a diverse range of experiences. Participants who indicated that they had experienced conversion therapy within the last 20 years were contacted to invite them to take part in an interview. The criteria for taking part was that the participant must have had first-hand experience of efforts to change from a minority sexual orientation or gender identity to heterosexual or cisgender. For the purposes of this research, efforts to change sexual orientation through gender reassignment was not included (see Ashley, 2019 for a discussion of gender affirmative care in relation to conversion therapy).

The final sample consisted of 30 interviewees, of which 28 lived in England, one in Scotland and one in Northern Ireland. The majority of interviewees (87%) lived in urban areas. The sample was predominantly White (28) and Christian (22) despite attempts to reach out to Black and ethnic minority and non-Christian faith groups. A summary of sample characteristics is provided in Table 1 (see Section 2.2). To be eligible all indicated that they had experienced conversion therapy in the last 20 years and two-thirds (20) of the sample had experienced conversion therapy within the last 10 years.^{[footnote 2]} Some interviewees’ experience of conversion therapy extended over a prolonged period of time, in some cases over a number of years.

3.2 Data analysis

The data was analysed using framework analysis (Ritchie and Spencer, 1994), an approach specifically developed for applied policy research which has also been widely used in healthcare, public policy and psychological research. It is a matrix-based method involving the construction of a framework into which data can be coded.

Data was analysed using a 5-step approach:

Familiarisation – immersion in the raw data Identifying a framework – identifying the key issues by which the data can be indexed Indexing – systematically applying the framework to each interview transcript Charting – summarising data according to category and participant within a table Interpretation – finding patterns and making sense of the data, in relation to the research questions

1. The Ozanne Foundation works with religious organisations to eliminate discrimination based on sexuality or gender. In 2018 the Foundation conducted a UK-based Faith and Sexuality survey to examine the role of religious belief on people’s understanding and acceptance of their sexuality. ↩

2. No major differences were noted in the experiences of those who had experienced conversion therapy in the last 10 years compared to those who had experienced it within the last 20 years. ↩