arXiv:2607.08573 – SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits
Note: This paper describes a current, yet to be peer-reviewed arXiv preprint (as of July 9, 2026). The reported results come from the authors’ own publication and have not yet been externally replicated.
What is it about?
Systems that are supposed to recognize which emotion or sentiment is present from text, speech, and video simultaneously must somehow merge the individual signal sources. A recent arXiv preprint by Adis Alihodzic and Selma Skopljakovic Hubljar examines an approach where this merging is not rigid but is controlled in a comprehensible way: A set of specialized models – each for individual modalities as well as for combinations of two or three modalities – provides individual assessments, and an explainability-based method called TreeSHAP determines for each individual example how strongly each model is weighted. At its core, it is about the question of whether methods from explainable AI (XAI) can make it more transparent and yet accurate to decide which “expert” should have the final say for which data point.
What is it good for / what does it bring?
Classical fusion methods combine modalities either at the very beginning of processing (“early fusion”) or only at the very end during the final decision (“late fusion”) – usually with fixed, unexplained weightings. The approach examined here aims to show that an attribution-based, example-adapted weighting can deliver similarly good results while additionally revealing which signal channel was decisive for which specific decision. For applications where traceability is important – such as in the analysis of customer conversations or in clinical behavioral observation – a system that not only recognizes an emotion but also justifies the basis for that assessment would be practically valuable.
How does it work (simplified)?
The system consists of several specialized models: individual models per modality (such as only text or only audio) as well as additional models that evaluate two or all three modalities together. For each individual example, the method calculates a kind of significance value for each of these experts using TreeSHAP – an established method for explaining model decisions – and uses this value as a weight when combining the individual assessments. Because the experts process different numbers of features (a triple expert has more input dimensions than a single model), the study compares three variants of how the many individual SHAP values are summarized into a weight: as a mean, as a median, or as the sum of the absolute values. According to the study, the sum variant has the advantage of retaining the entire attribution mass of an expert instead of artificially reducing it through averaging.
The approach was tested on two established public datasets from multimodal emotion research: MELD, a dataset with annotated dialogue scenes for emotion recognition, and CMU-MOSEI, a widely used dataset for multimodal sentiment and emotion analysis. In emotion recognition on MELD, the sum variant achieved a weighted F1 score of 0.5983 compared to 0.6018 with classical early fusion – a statistically insignificant difference according to the authors – while simultaneously outperforming late fusion. In sentiment analysis on CMU-MOSEI, the sum variant even slightly surpassed the early fusion variant with 0.6519 compared to 0.6485. According to the authors, the significant performance gain comes less from the complex, example-specific weighting itself and more from incorporating additional experts for modality combinations – especially the triple expert – into the system.
Classification & Limitations
This is a current, not independently peer-reviewed arXiv preprint by two authors; the reported figures come from their own evaluation and have not yet been externally replicated. The investigation relies on two public benchmark datasets from the English-speaking world, leaving it unclear how well the results can be transferred to other languages, cultures, or real-world application scenarios. The authors themselves point out that the gain from the elaborate, explainability-based weighting is small compared to simpler early fusion and that it is primarily the additional combination experts that make the difference – the method is thus more a contribution to the traceability of fusion than a significant leap in pure recognition accuracy. The complete text of the paper (introduction, related work, details of the model architecture) was not technically accessible at the time of research; this contribution therefore relies on the summary published by the authors.
