- Open Access
Acceptability and face validity of two mental health screening tools for use in the routine surgical setting
BMC Psychology volume 9, Article number: 171 (2021)
Preoperative assessment of mental health rarely occurs within routine surgery. Any screening tool selected to form part of this process must be deemed practical, acceptable and valid by clinicians and consumers alike. This study aims to assess the acceptability and face validity of two existing mental health screening tools to select one for further development and use in the routine surgical setting.
A survey of clinicians and consumers was conducted from October 2020 to March 2021 at a tertiary hospital in Sydney, Australia. Using a Likert scale (1–5, lowest to highest rating), the clinicians evaluated four domains for acceptability and two for validity (six overall) and the consumers four domains for acceptability and one for validity (five overall) on the preoperative use of the amended Kessler Psychological Distress Scale (K10) and the Somatic and Psychological Health Report-12 (SPHERE-12). Consensus was achieved through a rating of 4 or 5 being given by 70% or more of participants with domains able to remain unchanged. Free text responses were analysed into themes.
A total of 73 participants (51 clinicians; 22 consumers) were included. The K10 received consensus scores (≥ 70%) in four out of six domains for clinicians (4/4 acceptability; 0/2 validity), and all five domains for consumers (4/4 acceptability; 1/1 validity). The SPHERE-12 received consensus scores (≥ 70%) in three domains for clinicians (3/4 acceptability; 0/2 validity), and three domains for consumers (3/4 acceptability; 0/1 validity). Six qualitative themes were described including (1) amendments to tool structure and language; (2) scale response options; (3) difficulty with somatic questions; (4) practicality and familiarity with K10; (5) challenges for specific patient cohorts and (6) timing considerations for patients.
Adequate acceptability was established for the K10. However further development is required to strengthen its validity for this specific surgical cohort and purpose. Future research to determine the feasibility and acceptability of implementing and using the K10 in the routine surgical setting is now needed.
In order to provide high-value care within surgery, identifying patient-specific preoperative risk factors is essential for optimising surgical treatment. Whilst screening for medical comorbidities such as cardiovascular or respiratory conditions is well embedded and occurs routinely in preparing patients for surgery, consideration of mental health is rarely, if ever, undertaken . This is concerning, given the significantly poorer surgical outcomes experienced by patients with serious mental illness (SMI), compared to the general population. This includes increased rate of postoperative complications, length of stay in hospital and hospital readmissions across a range of procedures and health systems internationally [2,3,4,5]. SMI, which is defined as a mental, behavioural and/or emotional disorder that has episodic, recurrent or persistent features resulting in severe impairment , impacts on three percent of the population and includes a highly vulnerable group of patients due to the range of health, social and occupational challenges they face .
Given this known gap in surgical practise and the critical needs of these patients, it is apparent that simple mental health screening may be a beneficial mechanism for proactively identifying those patients who need further support with their mental health. Although limited, there is evidence to show that even modest preoperative psychological interventions can have favourable effects on surgical patient outcomes by matching a range of interventions to the needs of the patients . In addition, and perhaps most critically, patients with SMI have described their support and desire for having the status of their mental health acknowledged prior to surgery and identified screening as a potential solution . Currently there are no known mental health screening tools designed specifically for use in the routine surgical setting that cover a broad range of psychological conditions and symptoms. It is also imperative that any existing tool selected and amended for this purpose, is able to be seamlessly included into the existing preoperative planning processes and deemed practical, acceptable and valid to both clinicians and consumers. As such, the aim of this study is to assess the acceptability and face validity of two existing mental health screening tools using clinician and consumer feedback, to enable selection of one for further development and use in the routine surgical setting.
Study design and ethics
A survey of clinicians and consumers was conducted on Royal Prince Alfred (RPA) Hospital campus in Sydney, Australia between October 2020 and March 2021 to measure the acceptability and face validity of two amended mental health screening tools with the intention to select one for further development and pilot testing. A focus on colorectal and cardiothoracic surgical specialties for both the clinicians and consumers was undertaken based on the higher rates of surgical patients with mental health comorbidities identified in these groups . Ethics approval was granted by the Sydney Local Health District Ethics Review Committee—Royal Prince Alfred (RPA) Hospital Zone (X20-0386).
Feedback was sought from a minimum of 50 clinicians and 20 consumers to voluntarily participate from the following groups:
Clinicians: Consultant level clinicians potentially involved in the surgical care of people with SMI including cardiothoracic and colorectal surgeons, anaesthetists, psychiatrists, psychologists and mental health or surgical clinical nurse consultants (CNC) were invited to participate in the study via email, which included a link to the online survey and a softcopy, by their own head of department (at arm’s length from the study investigators);
Consumers: Both surgical and mental health consumers were invited to participate in the study. Postoperative adult inpatients from either the cardiothoracic or colorectal wards at RPA Hospital were provided with a hardcopy survey and advised to return it to the Nursing Unit Manager. People with self-reported mental illness who were members of the consumer group; SLHD Lived Experiences Advisory Panel (LEAP) meeting and/or attending the Collaborative Centre for Cardiometabolic Health in Psychosis (ccCHIP) outpatient clinic were either emailed the survey with a link to the online survey or provided with a hardcopy survey by their nursing team.
Recruitment into the study closed once the minimum number of participants had responded. Due to the ethical requirement for the study investigators to stay at arms length from the participants, the overall response rate was unable to be determined and minimum response numbers were sought instead.
Amended mental health screening tools
Following a comprehensive literature review of 470 studies completed by these authors (paper in submission under review), two tools were identified from a detailed review of 32 tools as being potentially suitable for use in the routine surgical setting; the Kessler Psychological Distress Scale (K10) , which includes 10-items measuring anxiety, depression and distress symptoms over the last 30 days using a 5-point Likert scale (1–5) and the Somatic and Psychological Health Report-12 (SPHERE-12), which includes 12-items measuring anxiety, depression, fatigue and somatization captured within two sub-scales (PSYCH and SOMA) over the last few weeks using a 3-point Likert scale (1–2) . Their features included being self-report, applicable to adults with a broad range of psychological disorders and symptoms, brief to complete, easy to score, free to use and with adequate psychometric properties across internal consistency, test re-test reliability, validity and responsiveness. To ensure the tools were fit-for-purpose for use in the preoperative environment and focusing on people with SMI, the investigators, with expertise in both psychiatry and surgery, developed four additional questions to add at the end of the tools, which were initially derived from several sources [12, 13], and then refined following consultation with colleagues and a consensus meeting.The four additional questions were added under the heading ‘Overview of your mental health and wellbeing’ and included the following: (1) Do you currently see or have you ever seen anyone for assistance with your mental wellbeing?, (2) Have you ever been told by a healthcare worker that you have one of the following conditions?, (3) Do you, or your carer/support person, have any serious concerns about your upcoming surgical procedure affecting your mental health and wellbeing? e.g. about your medication, your treatment, how you feel etc., (4) Would you, or your carer/support person, like to talk with someone about your mental health before your surgical procedure or receive information about mental health services available for you to use? (Additional file 1).
Following review of the K10 and SPHERE-12, both the clinicians and consumers anonymously completed four survey questions exploring the following domains on each tool as a whole; (1) clarity of wording, (2) suitability of the tool length, (3) ease of use, and (4) their perception on the willingness of patients to complete the tool. Both participants groups used a five-point Likert scale to rate each domain (score range 1–5 with 1 = very unclear/unsuitable/not easy to use/unwilling to complete and 5 = very clear/suitable/easy to use/willing to complete).
In addition, to further measure acceptability, the readability and comprehension difficulty of the K10, SPHERE-12 and the additional four questions added were assessed using the commonly used ‘Flesch Reading Ease’ and ‘Flesch-Kincaid grade level’ methods [14,15,16]. These measures use word and sentence length to provide the readability and education level of a text. Flesch readability scores range from 0 to 100 with scores ≥ 60 indicating a document is well written and easy to follow. A reading grade aimed at 6th–7th grade (11–12 years old) has been shown to be suitable for patient level information, which is indicated by corresponding scores between 6 and 7 [14,15,16]. Microsoft Word was used to provide these reading statistics, which has been shown to be valid and reliable .
Face validity assessment
After reviewing the K10 and SPHERE-12, the clinicians anonymously completed two survey questions exploring the (5) usefulness and (6) thoroughness of each tool as a whole. The surgical and mental health consumers rated one domain exploring the (5) design of the tool. Again both participants groups used a five-point Likert scale to score each question (score range 1–5 with 1 = not useful/unthorough/not well designed and 5 = very useful/thorough/well designed), which was developed using other measures of face validity as a guide for this assessment [18, 19]. Participants were advised ‘Face Validity’ is assessing whether the tools are appropriate for the aim of preoperative mental health screening for all routine surgical patients.
Overall the clinicians answered six and the consumers’ five survey questions in total, for each tool (Table 1). There was also a free text option for all participants to make comments or suggested changes to each of the tools.
Survey response data was stored within a Research Electronic Data Capture (REDCap) database. A basic descriptive analysis was performed according to participant grouping of clinicians and consumers using SPSS Software version 25 (IBM Corporation, Armonk, NY, USA).Consensus was achieved through a rating of 4 or 5 being given by 70% or more of participants, with domains able to remain unchanged.This was based on similar approaches previously described whereby between 60 and 80% of support is required [18, 20, 21]. Questions that received lower consensus percentages were established as areas requiring future revision to improve the tool for use in the routine surgical setting.. The Shapiro–Wilk test was used in order to determine normality. The Chi-square test or the Fisher exact test was used to determine the difference between clinicians (i.e. Anaethetists, Clinical Nurse Consultants, Psychiatrists, Psychologists, and Surgeons) and consumer groups (i.e. mental health and surgical consumers) who rated the mental health screening tools a low (< 4) or high score (≥ 4) across the acceptability and face validity domains assessed P < 0.05 was considered statistically significant for all the analyses.
The free text was quantitatively described and qualitatively analysed into themes through thematic analysis as described by Braun and Clarke . This flexible method was selected given the views of the participants were not known and it is an under-researched area. Responses were manually coded into themes and independently assigned by two investigators (KM and DS), who took a realist or straightforward interpretation of the data. Once the codes were determined, the investigators worked together to sort the codes into main themes for both the clinician and consumer groups. Both the quantitative and qualitative data was then synthesised by the investigators to select one tool for further development and future pilot testing.
A total of 73 participants provided feedback on the proposed use of the amended K10 and SPHERE-12 screening tools in the routine surgical setting. This included 51 clinicians, comprising seven colorectal and three cardiothoracic surgeons, 10 anaesthetists, 10 psychiatrists, 10 psychologists, and five colorectal, four mental health, one cardiothoracic and one preadmission CNC. Of the 22 consumer participants, 13 were surgical consumers and nine were mental health consumers.
For the clinicians, consensus was achieved for all four acceptability domains of the K10 to remain unchanged. The SPHERE-12 received consesus in three out of the four acceptability domains. Neither the K10 or the SPHERE-12 achieved consensus in either of the two face validity domains (Table 2).There was a significant difference found between the consensus percentages reported across the disciplines for both validity domains of usefulness and thoroughness of the K10 and for the domains of length, usefulness and the willingness to complete for the SPHERE-12.
For the consumers, consensus was achieved for all five domains of the K10 and three out of five for the SPHERE-12(Table 3). There was a significant difference between the consensus percentages reported for the domain of clarity for the SPHERE-12, with the mental health consumers rating it higher than the surgical consumers.
The free text option was utilised by 33 (64.7%) of clinicians and 15 (68.2%) of the consumers. For the K10, supportive comments to use the tool were provided by 9 (27.3%) clinicians and 10 (66.7%) consumers. Unsupportive comments were provided by 5 (15.2%) clinicians and 0 (0%) consumers. For the SPHERE-12, supportive comments to use the tool were provided by 5 (15.2%) clinicians and 7 (46.7%) consumers. Unsupportive comments were provided by 14 (42.4%) clinicians and 3 (20.0%) consumers.
The feedback was qualitatively analysed into the following six themes, which are summarised in Table 4; (1) Amendments to tool structure and language, (2) Scale response options (3 vs. 5), (3) Difficulty with inclusion of somatic symptoms in SPHERE-12, (4) Greater practicality and familiarity with using the K10, (5) Challenges for specific patient cohorts to complete the tool, and (6) Pre-operative timing consideration for surgical patients to complete the tool.
The ‘Flesch Reading Ease’ measure was 78.4 for the K10 indicating it was the simplest and easiest to read and well above the minimum scores required. The SPHERE-12 and the four additional questions added to the tools scored lower at 59.7 and 59.2 respectively, but these were still within the range considered well written and easy to follow (≥ 60). The ‘Flesch-Kincaid grade level’ measure was 5.4 for the K10 indicating the lowest reading grade level at the 5th–6th grade, 6.9 for the SPHERE-12 equivalent to the 6th–7th grade and 8.9 or equivalent to the 8th–9th grade for the four additional questions added to the tools.
Feedback from both clinicians and consumers was used to measure the acceptability and face validity of two amended mental health screening tools, the K10 and SPHERE-12, for potential use in the routine surgical setting. Overall the consensus achieved on the domains of the K10 was higher than the SPHERE-12 for both clinicians and consumers. The qualitative feedback was also more supportive of using the K10. This included greater practicality and familiarity with the K10 due to it being widely used in mental health services across Australia, the larger number of response options perceived as being more comprehensive for both patients and clinicians, and the preferred style of questioning. Furthermore, the readability and comprehension difficulty of the K10 was also found to be most suitable using the ‘Flesch Reading Ease’ and ‘Flesch-Kincaid grade level’ measures. Based on this collective quantitative and qualitative feedback, the K10 tool is recommended for further development, implementation and evaluation in the routine surgical setting.
It is important to note the findings presented in this study form the first phase of a larger project, which will involve pilot testing the feasibility, acceptability and accuracy of implementing the selected K10 tool into the preoperative screening process for all routine surgical patients booked for cardiothoracic and colorectal surgery. These specialties were chosen based on the results of a previous study demonstrating the higher prevalence of SMI in these groups . This pilot study will also determine the feasibility and acceptability of providing additional mental health support to those surgical patients identified by the K10 tool. It will also provide critical data on the capacity of the tool to identify patients with SMI, which is known to be a considerable challenge [23, 24]. The information gathered from the pilot will then assist in the future planning of a much-needed randomised control trial to determine whether proactive management of patient’s mental health through preoperative screening can improve the surgical outcomes and experience for patients with SMI.
Prior to the commencement of the pilot and based on the results of this study, it is evident the K10 may benefit from further development to strengthen its face validity to be used in the routine surgical setting. This includes improving the ‘usefulness’ and ‘thoroughness’ of the tool. Interestingly three main themes from the qualitative feedback emerged that related specifically to these two domains. Firstly, the number of response options on the tool was highly polarizing amongst all participant groups. On the one hand, the five response options of the K10 was framed as being more thorough in providing patients with additional options to describe their mental health and more useful in giving clinicians greater detail about the patient. Conversely, the three response options of the SPHERE-12 was felt by both clinicians and patients to be simpler, less overwhelming and more useful for patients to articulate their symptoms. Secondly, the inclusion of somatic questions on the tools was highlighted to be problematic for surgical patients, whereby it might result in false positives, and reduce the usefulness of the screening tools for this particular cohort. Clinicians reported this as a particular concern with the use of the SPHERE-12, which includes six questions focused on somatic symptoms. Finally, the thoroughness of the tools in identifying patients with SMI was raised with conflicting feedback. Overall clinicians viewed the four additional questions as being a positive inclusion that added value and strength to the tools. In contrast, several psychiatrists indicated that due to the known mistrust of the health system and fear of stigmatization, patients with SMI will not respond truthfully to the tool no matter what is included, resulting in false negatives. Whilst these challenges are well known  and important to highlight, none of the mental health consumers in this study reported being unwilling to use the tool, which is in line with the feedback previously highlighted regarding surgical patients with SMI wanting to have their mental health acknowledged . The brevity and accuracy of the K10 has also been demonstrated as being a suitable tool for screening SMI in the general population . Certainly, the planned research pilot will be critical for determining how or if the usefulness and thoroughness of the tool can be enhanced. Furthermore, helpful amendments were also suggested to improve the structure and language of the four additional questions, as well as the inclusion of a number of extra questions or response options, which will be incorporated into the amended tool for the pilot study. The results of the ‘Flesch Reading Ease’ and ‘Flesch-Kincaid grade level’ assessments also indicated simplification of the language to a lower grade level would be beneficial for patients.
The results of this study highlighted interesting variation in the scores provided by the different clinician groups that warrants further consideration. In assessing both the ‘usefulness’ and ‘thoroughness’ of the tools, the surgeons reported the highest and the psychiatrists the lowest consensus percentages for both tools with a statistically significant difference. Likewise, the ‘willingness of surgical patients to complete the tools preoperatively’ was reported with similar differences. Whilst limited inferences can be made from this data, these results possibly reflect the contrasting levels of expertise relating to mental health patients and experiences in screening them. Certainly, surgeons have previously reported a lack of confidence in managing surgical patients with mental illness  and may therefore have considered any tool that would assist in facilitating this process in a more positive light. Their experience with routinely screening for other medical comorbidities preoperatively may have also informed their impression that screening is a viable intervention that all patients would undertake. With a more storied experience, psychiatrists are likely to have been more knowledgeable about the range of challenges associated with screening patients about their mental health, and in particular for patients with SMI , and this may have been reflected in the lower scores they provided overall. It is evident that consideration of both perspectives is critical in being able to implement feasible interventions that help to bridge the gap for patients between the fields of surgery and psychiatry. Further examination of the details driving these points of view is needed.
Whilst this study makes an important contribution to the little researched area of mental health screening in the routine surgical setting, it does have limitations. The findings should be considered within the context of the participants involved and the single campus in which it was conducted. As both the clinician and consumer participants volunteered for the study, and we were unable to determine overall response rates, it may be possible they were biased or had particular experiences that compelled them to participate. The lack of demographic data collected also made this difficult to determine. Challenges in completing the online questionnaire were encountered by the mental health consumers with many opening the online link to the survey and selecting their participant grouping, but then not answering any questions, and thus not being included. Due to the anonymous nature of the study recruitment, it is not known why this was the case however it may reflect the online survey was not user friendly or straight forward to complete. Certainly it was a missed opportunity to seek their valuable feedback. It is evident that pilot testing the REDCap survey with a patient sample beforehand may have improved the completion rates and is an approach to be incorporated in future work. Furthermore, although important input was provided via the free-text item on the survey, there would be benefit for subsequent qualitative studies to examine in greater detail the areas that participants felt required attention including major benefits and drawbacks, along with the proposed administration process. Finally, due to the tools having previously been validated, this study was a high-level assessment of the tools as a whole in regard to their use within a routine surgical setting and did not involve participants considering individual items within the tools. There is an opportunity for future studies to undertake an assessment at a more granular level to further improve the tools for use in this unique setting. Similarly, whilst the methodological approach utilised to assess the tools was in line with other studies , there are limitations in using mean scores. It may be beneficial for further detailed assessments of the tools to apply different methodologies.
In conclusion, this study recommends the K10 mental health screening tool for preoperative use in the routine surgical setting. Important changes were highlighted as being required within the additional questions added to make the tool fit-for-purpose and focused on patients with SMI. Further development and testing is also needed to establish and strengthen the face validity of the K10 for this specific surgical cohort and purpose.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
McBride KE, Solomon MJ, Steffens D, et al. Mental illness and surgery: Do we care? ANZ J Surg. 2019;89(6):630–1.
McBride KE, Solomon MJ, Young JM, et al. Impact of serious mental illness on surgical patient outcomes. ANZ J Surg. 2018;88(7–8):673–7.
Cooke BK, Magas LT, Virgo KS, et al. Appendectomy for appendicitis in patients with schizophrenia. Am J Surg. 2007;193(1):41–8.
Maeda T, Babazono A, Nishi T, et al. Influence of psychiatric disorders on surgical outcomes and care resource use in Japan. Gen Hosp Psychiatry. 2014;36(5):523–7.
Menendez ME, Neuhaus V, Bot AG, et al. Psychiatric disorders and major spine surgery: epidemiology and perioperative outcomes. Spine. 2014;39(2):E111–22.
National Institute of Mental Health 2021. Mental health information. Mental illness statstics. https://www.nimh.nih.gov/health/statistics/mental-illness.shtml. Accessed 3 March 2021.
Australian Institute of Health and Welfare 2019. Mental health services in Australia: in brief 2019. Cat. no. HSE 228. Canberra: AIHW. https://doi.org/10.25816/5ec5bac5ed175.
Mumford E, Schlesinger HJ, Glass GV. The effect of psychological intervention on recovery from surgery and heart attacks: an analysis of the literature. Am J Public Health. 1982;72(2):141–51.
McBride KE, Solomon MJ, Lambert T, et al. Surgical experience for patients with serious mental illness: a qualitative study. BMC Psychiatry. 2021;21(1):1–9.
Kessler RC, Andrews G, Colpe LJ, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–76.
Hickie IB, Davenport TA, Hadzi-Paviovic D, et al. Development of a simple screening tool for common mental disorders in general practice. Med J Aust. 2001;175:S10–7.
Slade T, Johnston A, Oakley Browne MA, et al. 2007 National survey of mental health and wellbeing: methods and key findings. Aust N Z J Psychiatry. 2009;43(7):594–605.
Patel D, Sharpe L, Thewes B, et al. Feasibility of using risk factors to screen for psychological disorder during routine breast care nurse consultations. Cancer Nurs. 2010;33(1):19–27.
Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(221):1948.
Farr JN, Jenkins JJ, Paterson DG. Simplification of flesch reading ease formula. J Appl Psychol. 1951;35(5):333.
Kincaid JP, Fishburne Jr RP, Rogers RL, et al. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command Millington TN Research Branch; 1975.
Paasche-Orlow MK, Taylor HA, Brancati FL. Readability standards for informed-consent forms as compared with actual readability. N Engl J Med. 2003;348(8):721–6.
Irvine-Meek J, Gould ON, Wheaton H, et al. Acceptability and face validity of a geriatric self-medication assessment tool. Can J Hosp Pharmacy. 2010;63(3):225.
Moores K, Jones G, Radley S. Development of an instrument to measure face validity, feasibility and utility of patient questionnaire use during health care: the QQ-10. Int J Qual Health Care. 2012;24(5):517–24.
Cuerda MC, Apezetxea A, Carrillo L, et al. Development and validation of a specific questionnaire to assess health-related quality of life in patients with home enteral nutrition: NutriQoL® development. Patient Prefer Adherence. 2016;10:2289.
Hardesty DM, Bearden WO. The use of expert judges in scale development: implications for improving face validity of measures of unobservable constructs. J Bus Res. 2004;57(2):98–107.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
Lester H, Tritter JQ, Sorohan H. Patients’ and health professionals’ views on primary care for people with serious mental illness: focus group study. Br Med J. 2005;330(7500):1122.
Cranwell K, Polacsek M, McCann TV. Mental health consumers’ with medical co-morbidity experience of the transition through tertiary medical services to primary care. Int J Ment Health Nurs. 2016;25(2):127–35.
McCabe MP, Leas L. A qualitative study of primary health care access, barriers and satisfaction among people with mental illness. Psychol Health Med. 2008;13(3):303–12.
Kessler RC, Barker PR, Colpe LJ, et al. Screening for serious mental illness in the general population. Arch Gen Psychiatry. 2003;60(2):184–9.
We would like to thank the participants for their generous contribution towards this study. The authors had full access to all data in the study, were involved in the development, the drafting and completion of the final version of this article.
This study was internally funded.
Ethics approval and consent to participate
Ethics approval was obtained for this study from the Royal Prince Alfred Ethics Review Committee (X20-0386). All study methods were carried out in accordance with relevant guidelines and regulations along with the Ethics approval obtained. All participants provided informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
McBride, K.E., Steffens, D., Lambert, T. et al. Acceptability and face validity of two mental health screening tools for use in the routine surgical setting. BMC Psychol 9, 171 (2021). https://doi.org/10.1186/s40359-021-00672-w
- Mental illness
- Preoperative screening
- Face validity