Researchers propose Summarize-Judge-Refine architecture to decouple multimodal content moderation from policy-specific training

Content moderation systems have historically coupled multimodal understanding with policy-specific classification, which forces full pipeline retraining whenever policies change and exacerbates label scarcity because multimedia data cannot be meaningfully augmented. The proposed Summarize-Judge-Refine (SJR) architecture addresses these limitations by introducing a two-model design that separates these concerns through a natural language interface.

In the SJR framework, a multimodal Content Model generates structured text summaries of multimedia inputs, while a text-only Policy Model classifies those summaries against explicit policy definitions. An iterative co-training loop refines the Content Model using GRPO, and text-space augmentation generates adversarial summary variants—an augmentation pathway that is infeasible with raw multimedia—thereby enabling few-shot policy bootstrapping.

The authors evaluate SJR on misleading advertisement detection and report a 23.6% relative improvement in non-misleading F1 over a zero-shot chain-of-thought baseline. The system also outperforms end-to-end supervised fine-tuning (SFT), STaR/RFT, and RLFT baselines. Notably, a variant trained without any real violating examples—using only synthetically generated positive-class data—matches the performance of a model trained on the full dataset within 0.2% relative on violating F1, suggesting that new policies can be launched without access to real violation data.

Every moderation decision in SJR is grounded in a human-readable summary, which provides interpretability as a structural byproduct of the architecture. This design choice directly addresses the opacity often associated with black-box content moderation systems and offers a pathway to more transparent and auditable decision-making.