Adaptive AI system improves at solving CAPTCHAs by learning from its own outputs

A new system pairs a fine-tuned YOLOv8 object detector with an open-weight vision-language model (VLM) behind a confidence-based router, operating from screenshots and OS-level input events without browser automation or DOM access. The design explicitly avoids reliance on brittle automation pathways that CAPTCHA systems might block.

Across 16 CAPTCHA classes, the combined system achieves 85.4% overall accuracy and 84.2% macro accuracy, outperforming either component in isolation. The authors attribute this to complementary strengths: the VLM generalizes across unseen categories, while the detector refines precision on familiar ones.

Critically, every answer the VLM produces is used as a training label for the detector, enabling it to absorb new CAPTCHA categories after one or two exposures without human annotation. The same feedback loop also repairs the detector when it degrades, turning transient errors into long-term improvements.

The authors simulate a year-long arms race in which a CAPTCHA operator perturbs images monthly to defeat the detector. Even when perturbations reduce detector accuracy to 0%, the VLM remains unaffected and its labels allow the detector to recover within a small number of additional attempts. A cheap ~70%-accurate open-weight teacher model hardens the detector as effectively as a perfect oracle, indicating that robust defenses may not require expensive, high-accuracy models.

The findings suggest that CAPTCHA defenses assuming a failing bot stays failing understate how quickly adaptive solvers can return to effectiveness. The authors argue that future CAPTCHA designs should account for systems that learn and adapt over time rather than rely solely on static, non-adaptive bots.