Apple researchers develop pseudo-annotation pipeline to expand sign language datasets

Apple's machine learning research team developed a pseudo-annotation system designed to reduce the annotation burden that has constrained sign language AI development. The approach combines sparse model predictions from fingerspelling and isolated sign recognition systems with a few-shot language model strategy to automatically label video sequences with glosses, fingerspelled words, and sign classifiers, then ranks candidates by confidence.

The researchers first established baseline models for two foundational tasks. Their fingerspelling recognizer achieved 6.7% character error rate on the FSBoard benchmark, while their isolated sign recognizer reached 74% top-1 accuracy on the ASL Citizen dataset, both representing state-of-the-art results on those benchmarks at time of publication.

To validate the pipeline's output, a professional interpreter manually annotated 500 videos from the ASL STEM Wiki dataset at the sequence level, creating a gold-standard benchmark. Apple is releasing this professional annotation set alongside over 300 hours of pseudo-annotations in supplemental material accompanying their CVPR paper, making both resources available to the broader research community.