Vision-language model adapts Qwen3-VL-8B-Instruct for Nepali meme classification in CHiPSAL 2026 shared task

A system description paper introduces a two-stage vision-language adaptation approach for the CHiPSAL 2026 shared task on multimodal hate speech and sentiment detection in Nepali memes. The authors adapt the Robust Adaptation of Hateful Meme Detection (RA-HMD) framework using Qwen3-VL-8B-Instruct, a vision-language model with native Devanagari support. The pipeline consists of two stages: LoRA fine-tuning with an MLP projection head for generative classification, followed by contrastive backbone fine-tuning with a supervised InfoNCE loss. To address class imbalance, the authors apply minority oversampling, image augmentation, and focal loss. At inference, token probabilities from Stage 1 are ensembled with Stage 2 classifier scores using validation-tuned weights.

The approach eliminates error propagation from separate OCR and translation pipelines by relying on the model's native Devanagari understanding. The system achieved second place in hate speech detection with an F1 score of 0.797 and fourth place in sentiment analysis with an F1 score of 0.518. The paper includes detailed ablations, error analysis, and insights into adapting large vision-language models for low-resource South Asian languages.