Affect-LLaMA: Multimodal Human Affect-State Recognition via PEFT Foundation Models

Master Dissertation
Author: Erdenebat BATTSEREN
Institution: Institute of Science Tokyo (formerly Tokyo Institute of Technology), 2026
Repository: github.com/Rdnbt/masterthesis

Human affective behavior is intrinsically multimodal, synthesized across subtle facial micro-expressions, vocal prosody, pitch modulations, and contextual linguistic semantics.

This research investigates whether a Large Language Model (LLM) backbone, adapted via Parameter-Efficient Fine-Tuning (PEFT), can serve as a coherent cross-modal fusion and inference engine for in-the-wild affect recognition. We introduce Affect-LLaMA, a trimodal framework built upon LLaMA-3.1-8B that integrates continuous visual facial streams, acoustic waveforms, and synchronized speech text.

Evaluated on the Aff-Wild2 benchmark from the International Affective Behavior Analysis in-the-wild (ABAW) Challenge, Affect-LLaMA achieved the #1 Valence score (0.6610 CCC) and an overall Valence-Arousal score of 0.6480 CCC, outperforming specialized computer vision ensembles.

Interactive Computational Apparatus

Explore the real-time affective coordinate space, test continuous valence-arousal regression, and inspect cross-modal attention dynamics using the interactive module below:

System Architecture

The core hypothesis of this work is that pre-trained autoregressive foundation models possess rich semantic manifolds capable of performing cross-modal inference when provided with lightweight projection adapters.

1. Trimodal Feature Extraction Pipeline

1. Visual Modality ($\mathbf{e}_{vis}$): We freeze CLIP-ViT-Large-Patch14 pre-trained on large-scale image-text pairs. Video frames are sampled and resized to $224 \times 224$. The pooled 1024-dimensional visual feature vector is projected to the 4096-dimensional LLaMA hidden dimension through a learnable Multi-Layer Perceptron (MLP) projector $\mathcal{P}_{vis}$:

   $$\mathbf{e}_{vis} = \mathcal{P}_{vis}(\mathbf{v}) = \mathbf{W}_{v2} \cdot \sigma(\mathbf{W}_{v1} \cdot \mathbf{v} + \mathbf{b}_{v1}) + \mathbf{b}_{v2}$$

   where $\sigma$ is the GELU activation function.

2. Acoustic Modality ($\mathbf{e}_{aud}$): Raw audio is resampled to 16kHz. For each video frame at timestamp $t$, a symmetric 3.0-second window $[t - 1.5s, t + 1.5s]$ is passed through a frozen HuBERT-Base-LS960 encoder. Global average pooling over the temporal dimension compresses the acoustic state into $\mathbf{a}_{feat} \in \mathbb{R}^{768}$, which is mapped to LLaMA space via $\mathcal{P}_{aud}$:

   $$\mathbf{e}_{aud} = \mathcal{P}_{aud}(\mathbf{a}_{feat}) = \mathbf{W}_{a2} \cdot \sigma(\mathbf{W}_{a1} \cdot \mathbf{a}_{feat} + \mathbf{b}_{a1}) + \mathbf{b}_{a2}$$

3. Linguistic Modality ($\mathbf{E}_{text}$): Audio tracks are transcribed into text timestamps using OpenAI Whisper. When active speech coincides with timestamp $t$, the spoken transcript is incorporated. When no speech is detected, an explicit silence prompt is constructed ("The user is silent. Analyze their face and voice to estimate valence and arousal."), preventing hallucination.

2. Input Concatenation & Cross-Modal Self-Attention

The projected acoustic and visual embeddings act as non-verbal soft prefix prompts prepended to the instruction token embeddings:

By passing $\mathbf{X}_{input}$ directly into LLaMA's decoder layers, the standard self-attention mechanism natively attends across modalities:

Parameter-Efficient Fine-Tuning (PEFT): Adapter vs. LoRA

Full fine-tuning of an 8-billion parameter foundation model is computationally prohibitive and risks catastrophic forgetting of pre-trained linguistic knowledge. In this research, we designed, implemented, and empirically benchmarked both primary PEFT architectures directly within the LLaMA 3.1 8B transformer blocks:

1. LLaMA-Adapter Modules (Houlsby et al.): Sequential bottleneck sub-networks inserted after attention and feed-forward layers.
2. LLaMA-LoRA (Hu et al.): Parallel low-rank decomposition matrices bypassing the frozen linear projection weights.

After systematic experimentation, LoRA consistently outperformed Adapters across all evaluation axes: achieving higher Expression Macro F1 (+1.66%), superior Action Unit detection (+1.58%), and higher Valence-Arousal correlation (+0.50% CCC), while requiring 3.1× fewer trainable parameters (0.0153% vs 0.047%) and incurring zero runtime latency overhead via weight folding.

Mathematical Formulation of Adapter vs. LoRA

For an input hidden state $\mathbf{h} \in \mathbb{R}^d$, the sequential bottleneck Adapter module computes:

In contrast, LoRA keeps pre-trained weight matrices $\mathbf{W}_0 \in \mathbb{R}^{d \times k}$ strictly frozen and injects trainable rank decomposition matrices $\mathbf{B} \in \mathbb{R}^{d \times r}$ and $\mathbf{A} \in \mathbb{R}^{r \times k}$ in parallel, where rank $r \ll \min(d, k)$:

We applied LoRA adapters with rank $r=16$ and scaling factor $\alpha=16$ across all linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

======================================================================
Trainable Parameters : 1,228,830
Total Parameters     : 8,031,490,078
Trainable Percentage : 0.0153%
Quantization Format  : NormalFloat 4-bit (NF4 via BitsAndBytes)
Compute Precision    : bfloat16
Acceleration Library : Unsloth + PyTorch
======================================================================

By updating only 0.0153% of parameters, the entire trimodal architecture trains in under 15 GB of VRAM, fitting comfortably on a single NVIDIA A100 or RTX 3090 GPU, while delivering higher F1 and CCC scores than sequential adapter modules.

Mathematical Training Objectives

The framework is trained jointly across three distinct affective tasks:

1. Continuous Valence-Arousal Hybrid Loss

Valence (pleasantness) and Arousal (intensity) are continuous values in $[-1.0, +1.0]$. We combine Mean Squared Error (MSE) with the Concordance Correlation Coefficient (CCC) loss:

Where $\mu$ and $\sigma$ denote means and standard deviations over the batch, and $\rho$ is the Pearson correlation coefficient. The CCC metric penalizes shifts in scale and location, ensuring accurate dynamic trajectory tracking.

2. 8-Class Categorical Expression Loss

For discrete emotion recognition (Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise, Other), we optimize class-weighted Cross-Entropy to handle dataset imbalance:

3. Multi-Label Action Unit (AU) Detection

For detecting 12 concurrent Facial Action Units (AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU15, AU25, AU26), we use positive-weighted Binary Cross-Entropy with Logits:

Benchmark Results on Aff-Wild2

Continuous Valence & Arousal (VA) Estimation

Comparison against the top-performing international teams from the ABAW Challenge Leaderboard:

Affect-LLaMA achieved the highest Valence CCC (0.6610) in the competition, confirming that the LLM's deep contextual understanding provides critical disambiguation when judging facial sentiment.

Expression Classification (EXPR)

Evaluated on the 8-class categorical emotion task:

Affect-LLaMA outperforms the 2023 ABAW Challenge winner by +2.43% F1, demonstrating that a single foundation model architecture outperforms complex multi-stage ensemble pipelines.

Action Unit (AU) Detection

Evaluating multi-label occurrence across 12 distinct facial muscle movements:

The integration of textual tokens particularly benefited speech-correlated Action Units, such as AU25 (Lips Part) and AU26 (Jaw Drop), confirming cross-modal reinforcement.

Ablation Study: Unimodal vs. Trimodal Fusion

To quantify the exact performance gain contributed by multimodal fusion, we compared unimodal baselines against our full trimodal model:

The dramatic +69.4% increase in continuous Valence-Arousal correlation demonstrates that human emotional dynamics cannot be fully resolved from visual pixels alone; vocal timbre and verbal context provide crucial complementary signals.

Qualitative Analyses & Time-Series Tracking

Continuous Frame-by-Frame Tracking

We tested continuous temporal tracking across challenging validation clips featuring extreme lighting, occlusion, and head pose variations:

Figure: Continuous frame-by-frame estimation on Video 259. Top curve shows Valence, bottom curve shows Arousal. The model achieves 0.73 Valence CCC and 0.63 Arousal CCC, maintaining tight concordance with ground-truth human annotations.

Figure: Video 186 tracking under rapid affective transitions and mild facial expressions.

Latent Manifold Structure (t-SNE)

To visualize how the 8B foundation model organizes multimodal affective representations, we projected high-dimensional embeddings into a 2D plane via t-SNE:

• Sadness & Disgust: Form tightly bounded, well-separated manifolds, indicating robust multimodal acoustic-visual signatures.
• Happiness & Surprise: Prominent distinct clusters in high-arousal territory.
• Neutral: Sits centrally at the manifold centroid, reflecting natural affective equilibrium.

Key Conclusions

1. Parameter Efficiency: LoRA low-rank adaptation ($r=16$) is sufficient to repurpose a frozen 8B LLM into a multi-task continuous affective regression engine without catastrophic forgetting.
2. Complementary Modalities: Audio provides crucial disambiguation for arousal intensity, while speech semantics anchor valence estimation during subtle or ambiguous facial expressions.
3. Temporal Stability: Foundation models trained with CCC correlation loss exhibit smooth temporal tracking across video time series, resisting frame-level flicker and occlusion artifacts.

Citation

@mastersthesis{erdenebat2026affectllama,
  title  = {Affect-LLaMA: Multimodal Human Affect-state Recognition utilizing PEFT Methods},
  author = {Erdenebat Battseren},
  school = {Institute of Science Tokyo},
  year   = {2026},
  url    = {https://github.com/Rdnbt/masterthesis}
}