Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues can be addressed by exploiting the shared information between modalities to compensate for the impaired one. To this end, we analyze multimodal interactions – redundant (shared), unique (exclusive), and synergistic (emergent) task-relevant information provided by the modalities – to determine their impacts on model reliability. Specifically, amplifying redundant interactions would increase this exploitable shared information to resolve these issues; yet, modern instruction datasets often eliminate redundancies to prioritize visual grounding. We bridge this gap through a self-captioning workflow featuring a Multimodal Interaction Gate: a mechanism to convert unique interactions into redundant interactions. Our findings suggest that increasing redundancy can reduce visual induced errors by 38.3% and improve consistency by 16.8%.

Current computational approaches to meme analysis primarily focus on individual memes, with an emphasis on tasks such as hateful content detection and sentiment analysis. In contrast, corpus-level analyses, which are necessary to reveal in-depth thematic narratives embedded in meme corpora, have remained within the purview of social science research. While qualitative methods such as inductive content analysis provide deeper insights, they are labor-intensive and lack scalability. To address this gap, we introduce MemeTopicTrees (MemeTT), a zero-shot pipeline that clusters multimodal memes based on their targets, aspects, sentiments, and opinions. In addition to clustering, MemeTT generates a descriptive narrative for each cluster to provide a nuanced understanding of meme corpora. By integrating multimodal aspect-based sentiment analysis with hierarchical clustering, MemeTT automates both semantic analysis and narrative generation at scale. Evaluated on a combined pool of three datasets spanning political, public health, and defense domains, MemeTT successfully produces fine-grained clusters and coherent narratives, with narrative relevance most pronounced at the target and aspect levels. Evaluations show that the best-performing models are highly accurate at identifying meme targets and aspects, although maintaining high accuracy for opinions and sentiments remains challenging. Furthermore, while cluster distinctiveness is robust at the target level, room for improvement remains at lower clustering levels. Despite these challenges, this approach offers a scalable solution for analyzing public sentiment and discourse in meme corpora. We lay the foundation for future research on the understudied task of automated meme corpus analysis.

Understanding humor is a core aspect of social intelligence, yet it remains a significant challenge for Large Multimodal Models (LMMs). We introduce PixelHumor, a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences. Experiments with state-of-the-art LMMs reveal substantial gaps: for instance, top models achieve only 61% accuracy in panel sequencing, far below human performance. This underscores critical limitations in current models’ integration of visual and textual cues for coherent narrative and humor understanding. By providing a rigorous framework for evaluating multimodal contextual and narrative reasoning, PixelHumor aims to drive the development of LMMs that better engage in natural, socially aware interactions.

AI increasingly mediates and augments daily life, yet its technical properties risk reshaping the way users choose (what they consume), own (what they create), learn, and work. These risks reflect widely recognized AI principles, which unfortunately remain largely high-level and unoperationalizable. As the risks arising from AI assistance continue to subtly erode human agency, this work takes a different approach to operationalizing AI principles by translating the risks into concrete research questions that can guide the design of AI systems. This enables the research community to balance human agency and AI assistance in AI development and align it with the goal of benefiting humans.

Humans naturally anticipate tasks from context. A geometry figure in a quiz primes a student to expect trigonometry before the question is read. These heuristics reflect how much (partial) information each input modality potentially carries before even fixing a task. Generalizing this intuition, interpreting how a LLM anticipates task could help inform tasks with sequential targets. We extend partial information decomposition (PID) to accommodate these structured targets and evaluate the empirical interpretations. We test this by quantifying redundant (shared), unique (exclusive), and synergistic (emergent) task-relevant information across successive targets: a domain or topic, followed by the actual task or question. Our method, PID-Chain, shows that specifying the task resolves an apparently redundancy-dominated dataset into distinct per-task structures. The first task can be purely redundant while the subsequent task becomes synergistic. Decomposing contributions in sequential targets isolates key parts of a reasoning chain that lead to emergent properties like sarcasm. Further experiments show that PID-Chain recovers the enumerated ground-truth decomposition on synthetic benchmarks and yields results consistent with previous estimators.