Psychometric evaluation of the depression, anxiety, and stress scale-21 (DASS-21) among Chinese primary and middle school teachers

Abstract

Background

Teachers in high-stress roles face increasing psychological distress such as anxiety and depression, underscoring the need for validated assessment instruments. Given the current absence of a comprehensive, designated, and time-efficient scale capable of evaluating depression, anxiety, and stress among the teacher population, the Depression, Anxiety, and Stress Scale-21 (DASS-21) presents itself as a promising alternative. Despite the widespread application of the DASS-21 for assessing psychological distress across various populations, its validity among teachers, along with questions about its factor structure and its potential property of time equivalence, remain unverified. This study endeavors to address these considerations by investigating the psychometric properties of the DASS-21 specifically within the population of Chinese primary and middle school teachers.

Methods

Cross-sectional (n = 9,030) and longitudinal surveys (n = 1,642) were conducted using a non-probability sampling method. In addition to the DASS-21, the Chinese version of Chinese Teachers’ Job Burnout Questionnaire (CTJBO) was utilized to evaluate the criterion validity of this scale. Three different approaches, namely confirmatory factor analysis, Rasch analysis, and network analysis, were employed to evaluate internal reliability, construct validity, as well as time invariance of the DASS-21.

Results

The DASS-21 demonstrated a high degree of internal consistency (Cronbach’s α > 0.85) as well as excellent convergent validity, despite poor discriminant validity as determined by average variance extracted. Confirmatory factor analysis and network analysis further supported convergent validity. The three-factor structure outperformed one- and two-factor alternatives, establishing time invariance. Rasch analysis at the item level identified six inappropriate items within the anxiety and stress subscales, which were subsequently removed. Network analysis presented a better revised network. Regression analysis with emotional exhaustion as the criterion provided logical and accurate results.

Conclusion

The DASS-21 was found to be a reliable and valid tool for measuring the mental health of teachers over time. To assess the instrument’s psychometric properties, a combination of confirmatory factor analysis, Rasch analysis, and network analysis was utilized, which proved effective and is recommended for evaluating contentious instruments. Based on the results of the study, researchers and healthcare professionals are recommended to use the DASS-21 for assessing teachers’ psychological distress. However, certain items identified in the study may need to be removed to enhance the instrument’s appropriateness for this specific population.

Background

Teaching has been reported as a comparatively high-stress occupation when compared to other professions in numerous studies [1,2,3,4]. Teacher stress, as a particular type of occupational stress, refers to the psychological discomfort and disorder that teachers experience both mentally and physically in the school environment [4]. It can be attributed to multiple factors such as difficulty dealing with interpersonal relationships, and administrative pressure [5, 6]. Besides, the increasing use of information technology and globalization has made teaching more demanding in recent years. The mandatory requirements for technology integration without sufficient technical resources and training has become another potential source of stress [7, 8]. In addition, excessive workload, disruptive student behavior, and unrealistic expectations were identified as the three main sources of stress commonly experienced by teachers in Collie and Mansfield’s study [9]. Furthermore, García-Carmona et al.’s systematic review and meta-analysis underscores several risk factors contributing to teacher stress. These encompass issues such as unsatisfactory income, time pressure, overcrowded classrooms, inadequate facilities, and a dearth of training or promotional opportunities [10]. Teachers are consequently burdened by these myriad sources of stress.

Prior research indicates that stress not only undermines teachers’ mental and physical health but also engenders negative emotional disorders such as anxiety and depression, thereby impeding their work productivity and capacity to handle daily tasks [11]. Skaalvik and Skaalvik [12] demonstrated that teacher-experienced stress could instigate anxiety, foster feelings of incompetence, and necessitate self-protection. Moreover, stress has been implicated in the development of depression [13]. In a meta-analysis, Montgomery and Rupp scrutinized the causes and effects of stress among teachers, concluding that stress was correlated with emotional responses in this population, such that experiences of stress evoked distress, anxiety, and depression [14].

For teachers, such psychological distress can engender deleterious consequences. They may lead to lower job satisfaction, life satisfaction and increased their attrition rates [2, 6]. Those who choose to stay in the profession may experience decreased motivation and teaching efficacy [15]. As teachers are critical roles in the educational system, taking care of their mental health is important because their poor mental health can significantly influence educational outcomes [16]. Mental disorders among teachers pose a major challenge as they are associated with various variables that can impact a student’s academic success. Prior research has shown that teachers’ mental health is strongly linked to student behavior and can bring about poor student academic achievement and damaged teacher-student relationships in the long run [17]. As a result, to measure teachers’ mental health, a reliable and valid scale is urgently needed.

The majority of quantitative studies examining the mental health of Chinese teachers employ the Symptom Checklist 90-Revised (SCL-90-R), a self-report measure, as the instrument of choice [18]. The SCL-90-R encompasses 90 items, subdivided into nine dimensions, including depression and anxiety. Yang et al.’s meta-analysis concluded that while the SCL-90-R is commonly used among teachers, the findings of empirical studies using it are inconsistent: some indicate a good state of teachers’ health, while others suggest poorer health conditions compared to the general population [18]. Moreover, questions have been raised regarding the clinical validity of many of the SCL-90-R’s symptoms [19]. Furthermore, the 90-item scale is time-consuming and lacks efficiency. In light of these shortcomings, this study introduces a different scale for evaluation within the teacher population.

The Depression, Anxiety, and Stress Scale which is shortened as DASS is a widely used mental health assessment tool. The original 42-item full version is divided into three subscales: Depression, Anxiety, and Stress [20]. The instrument, developed by Lovibond & Lovibond in 1990s, was designed to recognize and distinguish common mental disorders using standardized measurement system [21]. It aimed to supply clinical diagnosis with psychometric indicators and serve as a rapid, efficient measurement instrument for related studies [20]. Later on, the full version of DASS-42 was shortened to be a 21-item version (i.e., the DASS-21), a version retains all dimensions of the original scale while enhancing the identification and evaluation of emotional disorders based on corresponding symptoms.

Due to its simplicity, novelty, uniqueness, and convenient operation, the DASS-21 has gained popularity. Different versions have been translated and validated for research and application worldwide, including Arabic [22], Bangla [23], Chinese [21], Hindi [24], Malaysian [25], Persian [26], etc. Moreover, it has also been utilized to assess the psychological distress symptomatology of diverse populations, including university students [27], athletes and non-athletes [28], adolescents [29], parents of young children [30], as well as teaching staff [31].

Previous studies have used the DASS-21 among teachers, including primary and secondary schoolteachers in the USA and mainland China [32, 33]. However, to our knowledge, the current literature does not have psychometric testing studies reporting the DASS-21’s reliability and validity for assessing psychological distress among teachers. Moreover, there is no evidence to indicate its factor structure is time invariant. In addition, different factor structures with regard to this scale have been reported: Some studies have supported a one-factor structure [34], or bifactor structure [28, 35], while others are in favor of the traditional oblique three-factor structure [36, 37].

Moreover, in terms of psychometric approaches, previous studies mainly adopted confirmatory factor analysis (CFA) (e.g., using CFA to assess the factorial validity in [26, 27]). Prior psychometric evidence of the DASS-21 using Rasch measurement model [38, 39] or network analysis [40] was established on populations other than teachers. Therefore, it is unclear if Rasch analysis and network analysis results of the DASS-21 can be replicated on teacher population. Given that psychometric evidence of an instrument needs to be cumulated using different statistical approaches, simultaneously using CFA, Rasch model, and network analysis could provide strong evidence for the psychometric properties of the DASS-21. The Rasch measurement model, also known as the Rasch analysis, was originally proposed by the Danish mathematician Georg Rasch [41]. It is an item response theory model that characterizes each item based on a single parameter, the item difficulty [42]. The analysis is based on the idea that a respondent’s ability, as well as the item’s difficulty, determines the chances of correctly answering an item [43]. Additionally, the Rasch analysis can be used to refine item checks through item addition or removal [44]. By analyzing the data rigorously and extensively, the Rasch analysis can provide additional psychometric information not available through traditional approaches [45]. To the best of our knowledge, the Rasch analysis has been applied to the DASS in two studies [38, 39]. In Shea et al.’s study, Rasch analysis was conducted to the DASS-21 and it was suggested removing one item from each subscale in order to achieve an adequate model fit for each subscale [38]. In contrast, Medvedev et al. found that non-fitting item 5 should be removed after conducting Rasch analysis to achieve the best model fit [39]. However, neither study included teachers as participants nor tested the time invariance of the DASS-21.

The network analysis is to present information about a particular system in the form of a network, which is composed of nodes and edges [46]. As a method of visualizing the complex relationships between symptoms on the basis of the dynamic systems modeling [47], the network analysis can be used to represent, analyze, and study systems in their full complexity [48]. Van den Bergh et al. conducted a network analysis of the DASS-21 symptoms in a large number of international respondents, revealing that certain symptoms can connect three subscale clusters, leading to a better understanding of comorbidity and reciprocal influences [49].

Given that CFA, Rasch analysis, and network analysis each offer distinct insights in the field of psychometrics due to their unique focal points, simultaneously reporting findings from the three different statistical analyses could provide a comprehensive picture for researchers to better understand the properties of a psychometric instrument. Other benefits of using three methods simultaneously include (i) having the comparable findings from the three methods because the same sample was used for the psychometric testing and (ii) some psychometric properties of the instrument (e.g., unidimensionality for each DASS-21 subscale) could be verified using different methods. However, most studies have typically employed a combination of two out of the three methodologies to thoroughly evaluate the reliability and validity of scales, as exemplified by research such as [25, 40]. In the context of the DASS-21—a scale that has been subject to considerable debate regarding its factor structure [34,35,36,37]—the provision of three types of psychometric testing in one study (like the present one) is informative. Specifically, CFA can serve to validate the theoretical structure, addressing the controversy surrounding the scale’s factorial validity [26, 27]. Rasch Analysis provides important insights into the functionality of individual items, evaluating item validity in line with previous studies of the DASS-21 [38, 39]. This approach extends these insights to include a teacher population sample, and it also helps to identify any potentially misfitting items. Following this, network analysis uncovers complex relationships among items that extend beyond the underlying constructs [49]. After item diagnosis via Rasch analysis, network analysis, through its visual representation of nodes and edges, offers an intuitive understanding, allowing for a visual comparison between the original and any potentially revised structures of the DASS-21. This combined application of the three methodologies ensures a robust and versatile psychometric examination, enhancing our understanding of the DASS-21 and its potential applications.

To summarize, the literature review has identified two research gaps. Firstly, the SCL-90-R, which is currently used to measure teachers’ mental health, exhibits several deficiencies. In addition, the DASS-21, although validated among different populations, has not had its psychometric properties specifically tested with respect to teachers as a demographic. Therefore, it is of significant importance to evaluate the DASS-21 among teachers. Another identified gap is that previous studies evaluating the DASS-21’s psychometric properties among different populations or in different languages have not combined three approaches - CFA, the Rasch analysis, and network analysis - in a single study. To address these gaps, the main purpose of the present study was to assess the DASS-21’s psychometric properties (i.e., internal reliability, construct validity, as well as time invariance) among primary and middle school teachers using the aforementioned approaches successively. Additionally, we aimed to test the criterion validity of this scale by examining its association with emotional exhaustion. According to the Stressor–Strain–Outcome (SSO) model [50], emotional exhaustion indicates strain, while psychological distress is an outcome. These two are closely related, and many empirical studies on teachers have confirmed their significant correlation [51, 52].

Methods and procedures

Participants

The present study comprised a cross-sectional survey and a longitudinal survey, both of which adhered to the same inclusion criteria: (1) being a primary or middle school teacher; (2) being able to read and write in Chinese; (3) providing electronic informed consent. The shared exclusion criterion was: non-participation in school work due to illness or other reasons. A non-probability sampling strategy was adopted for both surveys.

The cross-sectional survey was conducted online among primary and middle school teachers using an online questionnaire collection platform (i.e., Sojump) from May to June in 2020. Three provinces—Sichuan, Jiangxi, and Shandong, representing three parts of mainland China - the western, central, and eastern parts- respectively, were selected. These three provinces together account for half of the nation’s population. We contacted the principals who are in charge of primary and middle schools, informed them of the study’s purpose and duration, and assured them that participation was voluntary and anonymous, and all data collected would be used for research purposes only. Only after obtaining electronical informed consent did we proceed with the study. In the end, we obtained a total of 9,030 valid questionnaires.

The longitudinal survey was also conducted online in a city located in a central province of China. With the strong support and assistance from local education authorities, we carried out two waves of online surveys among research subjects. The first wave was conducted in November 2021 (Wave 1), two weeks after emergency remote teaching implementation during the COVID-19 pandemic, and the follow-up survey was carried out two months later in January 2022 (Wave 2) when campuses had reopened, and in-person classes had been in place for two weeks. The interval was set in this way to better detect the mental health state of teachers during and after the pandemic emergency remote teaching. The education administration staff transmitted the online questionnaire hyperlink to primary and middle school teachers within their jurisdiction. During the first wave, participants who were interested in taking part in the follow-up survey two months later were invited to leave their email addresses. On the first page of the questionnaire, we informed participants of our purpose, the researcher’s affiliation, and guaranteed that the collected data would be stored and curated appropriately to ensure privacy. After obtaining electronical informed consent, we continued with the online survey. Ultimately, 1,642 teachers completed both waves of the online survey.

The present study got approval from the Institutional Review Board (IRB) of the Jiangxi Psychological Consultant Association (IRB ref: JXSXL-2020-J013). Please refer to the Results section below for information on the participants’ characteristics.

Measures

The depression, anxiety, and stress scale-21 (DASS-21)

We adopted the Chinese version of the Depression Anxiety and Stress Scale-21 (DASS-21) to measure psychological distress among primary and middle school teachers in the present study. The DASS-21 consists of three subscales, namely depression, anxiety, and stress, each consisting of 7 items [20]. Example items for each subscale are: “I was unable to become enthusiastic about anything” (depression); “I felt scared without any good reason” (anxiety); and “I tended to over-react to situation” (stress). Each item is rated on a four-point Likert scale, indicating how the statement applied over the past week to the respondents [53]. The higher total score means that the psychological distress is more severe. The DASS-21 has been extensively applied in Chinese samples [21, 54], and previous studies have shown good internal consistency in teaching staff [32, 33]. In this study, the DASS-21 of the Chinese version was demonstrated with high internal consistency in both the cross-sectional and longitudinal surveys (Cronbach’s α for all the three subscales ranging from 0.86 to 0.92). The cut-off score indicating the presence of clinical depression, anxiety, and stress is 5 or more, 4 or more, and 8 or more respectively [20].

Chinese teachers’ job burnout questionnaire (CTJBO)

With the purpose of evaluating the criterion validity of the DASS-21, we chose to utilize the subscale called Emotional exhaustion in the Chinese version of CTJBO to measure emotional exhaustion among primary and middle school teachers during Wave 2 of the longitudinal study. This scale was revised by Wu et al. [55] from the Maslach Burnout Inventory (MBI) [56] to meet Chinese culture. The revised 22-item questionnaire consists of 3 dimensions. Besides Emotional exhaustion (8 items) that we adopted in the present study, the other two are Depersonalization (8 items) and Personal accomplishment (6 items) [55]. All items are scored on a seven-point Likert scale to test whether the respondent feels the same way about one’s job from 1 (never) to 7 (every day) [56]. A sample item is “I feel emotionally drained from my work”. Wu et al. found that the CTJBO demonstrated good construct validity and reliability, and that job burnout was positively correlated with depression [55]. Being a significant factor of burnout, emotional exhaustion was selected as the criterion to test the validity of the DASS-21. The internal reliability of the adopted Emotional exhaustion subscale was good (Cronbach’s α = 0.95).

Analysis strategy

Descriptive statistics, test-retest reliability and CFA

Descriptive statistics were adopted by the present study to present the participants’ characteristics and the level emotional exhaustion reported. Additionally, the prevalence of clinical depression, anxiety, and stress was determined using cutoff points provided by Lovibond and Lovibond [20]. Meanwhile, intraclass correlation coefficients (ICC) were used to assess the test-retest reliability. Subsequently, the classical test approach – CFA using diagonally weighted least squares (DWLS) estimation, given the ordinal response options – was conducted to test the factorial validity. This included one-factor, two-factor (depression and a combined factor of anxiety with stress) [57], and three-factor models [20]. The criteria adopted to evaluate model fit include: comparative fit index (CFI) and the non-normed fit index (NNFI) greater than 0.90; root mean square error of approximation (RMSEA) less than 0.06 and standardized root mean square residual (SRMR) less than 0.08 [58]. In addition, Akaike information criterion (AIC) and ΔAIC (AIC higher - AIC lower) were used to determine the best-fitting model, with a smaller AIC indicating a better fit [59]. As a further step, convergent and discriminant validity were assessed using composite reliability (CR) and average variance extracted (AVE). For each construct, evidence of convergent validity was considered present if CR was greater than 0.70 and AVE was greater than 0.50 [60]. To ensure discriminant validity, the square root of the AVE of the construct must exceed the correlation between it and other latent variables [60]. Lastly, time invariance across different assessment periods was evaluated using various models. We used the method introduced by Wu and Estabrook [61], which asserts that the traditional approach for testing measurement invariance, which involves setting a baseline model and imposing increasing restrictions on loadings and thresholds, is not the optimal for ordered categorical variables. Instead, they propose an alternative process. After creating the configural model, thresholds should be constrained first, followed by restrictions on factor loadings. Following this approach, our test procedures included: (1) Creating a configural invariance model, (2) Forming a threshold invariant model, (3) Constructing a model with both threshold and factor loading invariance, (4) Building a model with threshold, factor loading, and intercept invariance, and (5) Establishing a model with invariance in threshold, factor loading, intercept, and residual variance. We compared the different models using changes in CFI (ΔCFI), RMSEA (ΔRMSEA), and SRMR (ΔSRMR). Invariance was supported by the following criteria: ΔCFI greater than − 0.01, ΔRMSEA less than 0.015, ΔSRMR less than 0.03 (for factor loading constrained) or less than 0.01 (for item threshold constrained) [62].

Rasch analysis

Rasch analysis was conducted using the following strategies. Firstly, to determine whether to use the Andrich Rating Scale Model (RSM) or the Partial Credit Rasch Model (PCM), a likelihood ratio test was conducted. It was found that the PCM should be used as it was statistically significant. Next, the threshold ordering (category structure) of each item was examined to check whether the category calibration increases in an orderly manner. We selected the visual method using Person-Item map to scrutinize the functionality of the response categories. Additionally, the requirement of uni-dimensionality for Rasch analysis and the person fit to the model was assessed using the following criterion: in terms of the uni-dimensionality, eigenvalue ratios [i.e., the ratio of the first to second eigenvalues by the principal component analysis (PCA)] higher than 4.00 [63] and the eigenvalue of the first component in the PCA of the residuals less than 2.00 [64]; for a good model fit, the standard deviation of the mean persons’ residuals should be less than 1.50 [38, 65]. Moreover, person reliability was also evaluated with the acceptable criterion being higher than 0.70 [41].

For the diagnostic items, in addition to point-biserial correlation of each item, information-weighted fit statistic (infit) mean square (MnSq), and outlier-sensitive fit statistic (outfit) MnSq were also used. For the point biserial correlation of each item, a positive direction with the value between 0.30 and 0.70 is expected and this reflects the item potentially contribute to the latent variable [66, 67]. Acceptable good fit values for infit/outfit ranged from 0.7 to 1.3 [68]. If the infit/outfit value was below 0.7, it meant that the item provided no additional information beyond what was already given by the other items on the scale, which could happen due to the existence of similar items or a high degree of correlation among items, or when one item was contingent on another. Conversely, if the infit/outfit value was above 1.3, it indicated that this item did not capture same construct of the other items, and was either poorly constructed, poorly understood, or ambiguously defined. Consequently, items with subpar fit statistics should be considered for elimination from the scale [68].

Network analysis

To examine the impact of removing inappropriate items by Rasch analysis (if necessary) on the findings of network analysis, the next step involved visualizing the results. The network was constructed using the Extended Bayesian Information Criterion Graphical Least Absolute Shrinkage and Selection Operator (EBICglasso). A tuning parameter of 0.5 was set to generate a more parsimonious and straightforward network with fewer connections, greater specificity, and higher sensitivity. Nodes (i.e., items) were connected using edges, with thickness indicating strength, and blue and red colors denoting correlations are positive and negative, respectively. To evaluate the centrality of nodes, we computed measures of betweenness (connectivity), closeness (distance centrality), strength (degree centrality), and expected influence (a node’s cumulative impact on a network, representing the activation, persistence, and remission of the node in the network), as recommended by previous studies [69, 70]. Before presenting the centrality results, we checked the stability and accuracy of the network. Centrality stability was determined using case-dropping subset bootstrapping, which analyzes whether centrality indices remain constant across different subsets [71]. The correlation stability coefficient (CS-coefficient) was used to determine the stability of centrality, with a value above 0.5 preferred and a value below 0.25 considered unstable and uninterpretable [71]. Additionally, non-parametric bootstrapped samples were used to calculate 95% confidence intervals (CIs) to determine accuracy, with narrower CIs indicating more accurate edge estimates [71].

Criterion validity

A regression analysis was conducted to examine DASS-21’s criterion validity with emotional exhaustion. The model included teachers’ emotional exhaustion as the dependent variable and factor scores of the three subscales as the explanatory variables, along with control variables such as sex and age. The analysis also involved a comparison between the original DASS-21 and a modified version obtained by removing any inappropriate items identified by Rasch analysis. These steps helped ensure the accuracy and reliability of our findings.

Results

Table 1 displays participants’ characteristics in both surveys. The cross-sectional survey was conducted between May and June 2020, and the second longitudinal survey was conducted in November 2021 (Wave 1) and January 2022 (Wave 2), with a two-month interval. For the cross-sectional survey, 9030 participants were recruited, of which 5838 (64.65%) were from primary school, and female occupied the vast majority (6563, 72.7%). They taught different subjects, with Chinese literature being the most followed by English. Based on the DASS-21, the number of teachers with probable depression was 1843 (20.41%), probable anxiety was 2388 (26.44%), and probable stress was 917 (10.15%). For the longitudinal survey, 1642 participants were recruited, with the majority from primary school (1159, 70.6%), and a high proportion of female teachers (1305, 79.5%). As for teaching subject, Chinese literature was the most followed by Mathematics. The DASS-21 score for Wave 1 showed that the number of teachers with probable depression was 368 (22.41%), probable anxiety was 589 (35.87%), and probable stress was 265 (16.14%). In Wave 2, the number of teachers with probable depression was 386 (23.51%), while the other two decreased slightly to 570 (34.71%) for anxiety and 258 (15.71%) for stress. The score of teacher emotional exhaustion was 3.46. Moreover, ICC (2,1) of depression, anxiety, and stress subscales was 0.73, 0.71, and 0.71, demonstrating acceptable test-retest reliability.

Table 1 The characteristics of the participants

Table 2 presents the comparison of the model fit between different factor structures. The results showed that all three factor structure types were acceptable for participants in both cross-sectional and longitudinal surveys, including two waves of data (CFI = 0.992 to 0.996; NNFI = 0.991 to 0.996; RMSEA = 0.049 to 0.063; and SRMR = 0.029 to 0.045). Although all three factor structures had acceptable model fits, the three-factor model outperformed the other two factor structures across all studies. Specifically, the ΔAICs for the three-factor model compared to the other two factor models were all greater than 10, indicating that the three-factor model was strongly supported by the empirical data (see Table 2).

Table 2 Results of CFA Model Fit

Following the confirmation that the three-factor structure had the best model fit, the factor loading in this model was used to calculate CR and AVE for evaluations of convergent validity (see Table 3). The results showed that CR was higher than 0.90 and AVE was greater than 0.60, indicating excellent convergent validity. However, there was a lack of discriminant validity across all subscales of the DASS-21 due to the overly high correlations between each pair of factors (see Table S1). In order to test the time-invariant property of the DASS-21, the configural model and the respective nested models were compared after confirming that the three-factor structure was acceptable for both waves in the longitudinal study. Based on the results presented in Table 4, we can conclude that the DASS-21 exhibits full time-invariant properties, specifically the equivalence of thresholds, factor loadings, intercept, and residual variances, across two-month intervals. This conclusion is supported by the fact that the ΔCFI, ΔRMSEA, and ΔSRMR all meet the specified criterion.

Table 3 Item properties in the DASS-21 across two surveys of the participants

Rasch analysis was performed with the purpose of evaluating the item fit of the DASS-21. A Person-Item map was generated (refer to Figure S1 to S3) and it revealed that disordered category thresholds were not present in any of the subscales of the DASS-21 for all samples. Table 5 presents fit statistics showing that all three factors in the DASS-21 were unidimensional in all datasets, including the cross-sectional survey and two waves in the longitudinal survey. This is supported by the following evidence: (i) the eigenvalue ratios (i.e., first eigenvalue / second eigenvalue ratio by PCA) were all higher than 4, and (ii) the eigenvalues of the first component of the residuals were all less than 2. Additionally, there were no substantial misfits among the participants as the standard deviations of the mean person residual were all less than 0.15 (i.e., ranging from − 0.11 to -0.16). Moreover, the person reliability of three subscales was all above 0.70 across all surveys, indicating acceptable internal consistency among school teachers.

Table 4 Fit indexes in measurement invariance across different times

Table 5 Fit statistics of Rasch analysis for the subscales of the DASS-21.

Regarding item properties in the Rasch results (Table 6), all items had a point biserial correlation greater than 0.50. This value also indicated good convergent validity. The most difficult items were Item 17 (0.75 logits, “not worth much as a person”) in the depression subscale, Item 7 (0.45 logits, “experienced breathing difficulty”) in the anxiety subscale, and Item 6 (0.43 logits, “tended to over-react”) in the stress subscale. The easiest items were Item 13 (-0.52 logits, “felt down-hearted and blue”) in the depression subscale, Item 9 (-0.68 logits, “worried about situations of panic and making a fool of myself”) in the anxiety subscale, and Item 8 (-0.37 logits, “using a lot of nervous energy”) in the stress subscale. Only a few items were found with unsatisfactory infit and outfit MnSq, including Item 2 (“dryness of my mouth”), Item 15 (“feeling of close to panic”), and Item 20 (“felt scared without any good reason”) in the anxiety subscale; Item 1 (“hard to wind down”), Item 11 (“getting agitated”), as well as Item 12 (“difficult to relax”) in the stress subscale.

Table 6 Fit statistics of Rasch model for Items

To visually compare the original and the revised structure of the DASS-21 [(after removing items with unacceptable infit and outfit MnSq (i.e., Item 2, 15, 20, 1, 11, and 12)], network analysis results showed that centrality indices were stable as coefficients were generally above 0.50 (refer to Figure S4-S9). Additionally, the 95% non-parametric CIs for edge weights using the bootstrap method were narrow for the cross-sectional survey dataset, with very few CIs including zero, indicating a high level of accuracy (refer to Figures S10 and S11). However, caution is needed when interpreting the order of these edges in the network for the longitudinal study datasets as the findings were not replicated (refer to Figure S12-S15). Therefore, only the network results obtained from the dataset of the cross-sectional survey are presented.

The comparison of the DASS-21 networks between the original and revised versions showed that the latter was better aligned with the expected structure (see Figs. 1 and 2). This pattern was consistently found in both waves of the longitudinal study; however, interpretation of edge-weights in these networks should be approached with caution. The centrality indices (see Table S2) revealed higher values in betweenness, closeness, strength, and expected influence for Items 17 (“not worth much as a person”) and 13 (“felt downhearted and blue”), 7 (“experienced trembling”), and 6 (“tended to overreact”), indicating that these items had the strongest connections to other nodes (i.e., items).

Fig. 1

Network analysis of Cross-sectional survey. DEP = Depression; ANX = Anxiety.

Fig. 2

Network analysis of Cross-sectional survey (revised). DEP = Depression; ANX = Anxiety.

Table 7 showed that when the inappropriate items retained in the DASS-21, only the factor score of stress at Wave 1 was positively related to emotional exhaustion at Wave 2 (b = 4.73, t = 7.62, p < 0.01), while the other two factors were not. Furthermore, problematic results emerged, that is, depression at Wave 2 and emotional exhaustion at Wave 2 were significantly and negatively correlated (b = -1.66, t = -2.52, p = 0.01). However, stress at Wave 2 was positively related to emotional exhaustion at Wave 2 (b = 8.62, t = 13.60, p < 0.01). Variance inflation factor (VIF) of the original DASS-21 showed that there was a non-negligible multicollinearity problem between explanatory variables and the VIFs of three factors were all higher than 5.50. Alternatively, regression analysis yielded accurate and logical results when inappropriate items were removed. In addition to the unchanged results of the significant positive relationship between stress (including Wave 1 and Wave 2) and emotional exhaustion (b at Wave 1 was 3.54, t = 6.71, p < 0.01; b at Wave 2 was 7.48, t = 13.95, p < 0.01), the relation between depression at Wave 1 and emotional exhaustion was changed to be significantly positive (b = 1.24, t = 2.13, p = 0.03); the original negative association of depression at Wave 2 with emotional exhaustion was also changed to be non-significant. Furthermore, the VIFs of all explanatory variables were under 4.5 after the inappropriate items were removed. Although the values still approximated 5, multicollinearity seemed to be reduced compared to the previously tested version of the instrument.

Table 7 Regression model on the emotional exhaustion between original and revised factor score of the DASS-21.

Discussion

Given the escalating levels of stress, depression, and anxiety experienced by teachers—and their subsequent negative impact on both mental and physical health, which may in turn affect student behavior and achievement [8]—the present study evaluated the suitability of the DASS-21 for assessing psychological distress among schoolteachers. The psychometric properties of the DASS-21 were evaluated using three approaches – CFA, Rasch analysis, and network analysis – with the simultaneous aim of providing a comprehensive view. Two surveys, a cross-sectional and a longitudinal survey consisting of two waves with a two-month interval, were employed. The results of the CFA indicated that the DASS-21 demonstrated good factorial validity; the three-factor structure outperformed both the one- and two-factor structures. Additionally, the DASS-21 was found to possess time-invariant properties and excellent convergent validity. However, the scale exhibited relatively poor discriminant validity. The Rasch analysis identified and removed three items with unsatisfactory infit and outfit MnSq from the subscales of anxiety and stress, respectively. Following the removal of these six inappropriate items, the network analysis demonstrated a superior network compared to the original 21-item version. Finally, a regression analysis using emotional exhaustion as the criterion revealed that multicollinearity problems were not found in the revised DASS-21.

Reliability, factorial, convergent and discriminant validity of the DASS-21

The present study, in line with previous research with teaching staff as samples [32, 33], demonstrated that the DASS-21 had excellent internal consistency in both cross-sectional and longitudinal surveys. Using the CFA approach, we found that all three types of factor structures were acceptable, but compared with one- and two-factor structures, the three-factor structure was much better, which is consistent with previous studies [36, 37, 54]. Moreover, in line with a prior study [72], the test results of the DASS-21’s convergent validity and discriminant validity demonstrated quite different outcomes; the former was found to be excellent, but the latter was poor, indicating that items were highly correlated. Based on these findings, we recommend utilizing the DASS-21 as a composite construct rather than breaking it down into three separate components.

Time invariance of the DASS-21

The DASS-21 has been previously investigated for measurement invariance across various factors such as cultures [36] and gender [28]. However, to the best of our knowledge, no study has examined the time invariance of the DASS-21 in a longitudinal study of teachers’ population. In the current study, the three-factor structure of this instrument was found to remain invariant across a two-month interval, suggesting that the instrument had consistent construct and item descriptions over time. This finding provides solid evidence for mental health interventions aimed at teachers experiencing psychological distress, as only a time-invariant instrument allows for the evaluation of the same concepts over time [73]. This property enables healthcare providers and researchers to compare the levels of teachers’ psychological distress before and after interventions, and assess their effectiveness.

Three approaches of psychometric evaluation of the DASS-21

To our knowledge, the current study is the first to combine CFA, Rasch analysis as well as network analysis to evaluate the properties of the DASS-21. Using Rasch analysis to scrutinize the properties of the individual items of the DASS-21, the present study found 6 unsatisfactory items, with three items (Items 2, 15, and 20) from the anxiety subscale, three (Items 1, 11, and 12) from the stress subscale, and none from the depression subscale. This finding differs from Shea et al.’s [38] and Medvedev et al.’s [39] studies applying Rasch analysis to the DASS-21. In order to achieve the best model fit, the former study removed Item 5 (depression subscale), Item 2 (anxiety subscale), and Item 11 (stress subscale), while the latter only removed Item 5 from the depression subscale. Subsequently, by visually comparing the original 21-item version with the revised 15-item version using network analysis, we demonstrated that revisions to the DASS structure were consistent with the original version. Furthermore, using emotional exhaustion as a criterion, we found that the revised version had good criterion validity through regression analysis. On account of this, we recommend using CFA, Rasch analysis, and network analysis in conjunction, as they complement each other. For example, network analysis can help to visually illustrate the changes that occur after the removal of inappropriate items. The combination of these three analyses is proposed to be applied to the evaluation of widely used scales with disputes such as SCL-90 and SCL-90-R.

Strengths and limitation

The strengths of this study include the large sample size and the multiple methods to assess the psychometric properties for the DASS-21 among the teacher population. This study provides several key advancements in the understanding and application of the scale. In line with findings from previous studies conducted on different populations [27,28,29,30,31], our study reaffirms the three-factor structure of the scale. By confirming its validity in our population, we are providing further robustness to the DASS-21’s application for both research and clinical use on teachers. A particularly significant contribution is the identification of DASS-21’s time invariance, an unexplored attribute that facilitates longitudinal assessments of psychological distress, critical for measuring the effectiveness of mental health interventions. The study introduces an innovative methodological combination of CFA, Rasch analysis, and network analysis for a more comprehensive evaluation. In contrast to studies that are limited to integrating a maximum of two methods [74, 75], our approach unveils potential areas of refinement, leading to the proposition of an optimized 15-item version. Additionally, this methodology establishes a foundation for evaluating other contested psychological scales. Essentially, this study enhances DASS-21’s theoretical comprehension and practical use.

Despite the contributions listed above, it is important to note there are some limitations to the present study. First, the DASS-21’s criterion validity was evaluated by assessing participants’ emotional exhaustion only at Wave 2 of the longitudinal study. Therefore, the relevance of the data for participants in the cross-sectional survey and Wave 1 of the longitudinal survey remains unclear. Second, non-probability sampling was used in the study, so results may be limited in their generalizability, despite the large cross-sectional sample.

Conclusions

In summary, this study affirms the DASS-21 as a reliable, valid, and time-invariant tool for assessing mental health among teachers in primary and middle schools. The combined use of CFA, Rasch analysis, and network analysis offers an innovative method for evaluating psychometric properties, emphasizing excellent convergent validity but poor discriminant validity. Despite confirming the superiority of the three-factor structure, the study suggests that it is more beneficial to use the DASS-21 as a measure of overall psychological distress rather than treating depression, anxiety, and stress as distinct constructs. This research paves the way for future studies to employ similar methodological combinations in evaluating widely disputed psychological instruments.