Memotion Dataset

Multimodal Social Media

In recent years, the widespread presence of Internet memes on social media has garnered significant interest. Memes, among the most frequently typed English words (Sonnad, 2018), often stem from shared social and cultural experiences, like TV shows or popular characters (e.g., the “One Does Not Simply” meme from “The Lord of the Rings”). Memes are deeply embedded in Internet culture, reflecting community opinions and social discourse (Gal et al., 2016). However, the rise of online hate speech has compounded the challenge of moderating offensive content. The addition of memes, which combine visual cues and language, complicates detection efforts. Current moderation strategies rely heavily on human intervention, but the volume of multimodal content is rapidly becoming unmanageable. Thus, there’s a pressing need for automated multimodal social media analysis.

The Memotion Analysis Task

• Task A: Sentiment Classification: Classify a given meme as positive, negative, or neutral.
• Task B: Humor Classification: Identify the type of humor in a meme, categorizing it as sarcastic, humorous, offensive, or other. A meme can belong to more than one category.
• Task C: Scales of Semantic Classes: Quantify the extent to which a particular effect (e.g., sarcasm, humor, offensiveness) is expressed in the meme. Detailed quantifications are provided in the dataset.

Evaluation Criteria

• Task A: Evaluated using macro F1 score.
• Tasks B and C: Evaluated using macro F1 score for each subtask, with an overall average score reported.