Table 6 Overview of emotion recognition based on multimodality
From: Development and application of emotion recognition technology — a systematic literature review
No | Author & year | Feature extraction | N | Emotions | Test sample | Pattern recognition methods | Accuracy |
|---|---|---|---|---|---|---|---|
1 | M. Shamim Hossain (2016) | Video, Audio | 100 | Pain, Tension, Normal | College students | GMM | 99.4% |
2 | Amico F (2016) | 1ERP, GSR, 2HRV, facial emotion and degree of pupil dilation | 48 | Fear, Sadness, Joy, Anger, Disgust, Surprise | Depression patients, BD patients, BPD patients | NA | NA |
3 | Gillian M. Sandstrom (2016) | Self-report, psychological, daily behavior | NA | Mania, Depression | BD patients | Mean, Standard deviation, Friedman test, SVM | NA |
4 | Xinfang Ding (2019) | EEG, eye tracking information, GSR | 348 | Depression | 3MDD patients | Chi-square test, T test, Random forest, LR, SVM | Accuracy is 79.63% Precision is 76.67% |
5 | Bai Ran (2021) | Clinician rating scale, Sself-rating scale, telephone usage data, sleep data, step data | 334 | Steady-remission, Steady-depressed, Swing-drastic, Swing-moderate | MDD patients | SVM, KNN, DT, Naïve bayes, RF, LR | Steady-depressed: 84.27% Swing-drastic: 85.33% |
6 | Geerling B (2021) | Graphical representation of mood swings, online monitoring of sleep | 17 | NA | BD patients | 4LCM | NA |
7 | Haiyun Huang (2021) | Behavior scale, EEG, pupillary response, gaze distance | 18 | Happiness, Anger, Sadness | Disturbance of consciousnes | Spectral turbulence measurement, SVM-RFE | 91.5 ± 6.34% |
8 | Mano, Leandro Y (2016) | Images, physiological signals | NA | Neutrality, Happiness, Sadness, Fear, Anger, Surprise | Extended Cohn-Kanade (CK+) dataset | T test, Wilcoxon rank sum test, 5KNN, 6DT, Fuzzy logic, Bayesian network, SVM | 99.75% |
9 | Yuying Tong (2020) | HRSD, HAMA, EEG, facial expression | 50 | Depression | Depression patients | Three-factor repeated measurement variance analysis | Happy: 87.68 ± 7.50% Neutral:82.87 ± 10.14% Sad: 75.06 ± 13.32% |
10 | Yulong Li (2021) | SAS, SDS, HAMD, EEG | 54 | Depression | Androgen alopecia patients | FAW-FS algorithm, ANOVA, Mutual information, χ2 test, LR, DT, KNN, SVM, 7RF | LR: 80.87% DT: 79.24% KNN: 80.42% SVM: 83.07% RF: 81.45% |