AnyGroundBench
A Multi-Domain Adaptation Benchmark for Video Grounding in VLMs
* Equal contribution
Vision-Language Models (VLMs) have shown strong performance in Spatio-Temporal Video Grounding (STVG), yet they are still evaluated mostly in a zero-shot manner on general-purpose benchmarks of everyday scenes. This creates a critical disconnect from real-world applications in specialized domains, where models inevitably encounter rare visual or textual concepts. Since exhaustive pre-training across infinite data distributions is infeasible, the ability to adapt to novel domains with limited data is essential.
To bridge this gap, we introduce AnyGroundBench, a domain-adaptation benchmark designed to shift the STVG evaluation paradigm from static zero-shot testing to rigorous domain adaptation. Targeting five specialized domains (animal, industry, sports, surgery, and public security), AnyGroundBench pairs newly captured, expert-annotated videos with established datasets, unifying them through dense, high-fidelity spatio-temporal annotations. Crucially, the benchmark provides dedicated limited training subsets, enabling systematic evaluation of domain adaptability under limited training data.
We benchmark 23 state-of-the-art VLMs in the zero-shot setting and further evaluate five adaptation strategies, spanning training-free and fine-tuning-based approaches, on representative models, assessing their zero-shot generalization and adaptation capacity. Our results show that current VLMs remain far from practical performance in the zero-shot setting, that training-free adaptation produces highly variable effects depending on the model and domain, and that fine-tuning-based adaptation, though more effective, still falls short of real-world requirements, with gains varying markedly across domains. These findings expose fundamental limitations in current VLMs' spatio-temporal reasoning, pointing to concrete directions for future research.
Each cell reports STVG / TVG / SVG. STVG uses vIoU@0.3, TVG uses tIoU@0.3, and SVG uses sIoU@0.3. Bold and underlined values denote the best and second-best results, respectively.
| Models | Animal | Industry | Sports | Surgery | Public Security |
|---|---|---|---|---|---|
| Proprietary VLMs | |||||
| GPT-4o | 0.00 / 16.6 / 7.64 | 2.96 / 8.88 / 14.8 | 0.61 / 17.2 / 0.00 | 0.00 / 13.4 / 0.00 | 0.80 / 56.0 / 4.40 |
| GPT-5.1 | 3.18 / 19.7 / 55.4 | 5.33 / 9.47 / 28.4 | 0.00 / 25.8 / 1.84 | 1.39 / 22.7 / 0.67 | 4.80 / 61.2 / 35.6 |
| GPT-5.4 | 6.37 / 21.7 / 65.6 | 4.73 / 12.4 / 33.1 | 0.61 / 32.5 / 5.52 | 1.39 / 38.9 / 2.68 | 2.00 / 68.0 / 53.6 |
| GPT-5.6-Sol | 15.3 / 25.5 / 92.4 | 10.7 / 24.3 / 47.9 | 6.13 / 44.2 / 45.4 | 7.87 / 32.4 / 27.5 | 20.8 / 67.2 / 88.4 |
| GPT-5.6-Terra | 13.4 / 22.3 / 86.0 | 6.51 / 17.2 / 43.8 | 4.91 / 31.9 / 31.3 | 4.17 / 32.4 / 10.1 | 18.4 / 56.8 / 83.2 |
| GPT-6-Astra | 19.1 / 36.3 / 97.5 | 7.10 / 24.9 / 53.8 | 4.91 / 46.0 / 58.3 | 25.9 / 47.7 / 61.7 | 18.8 / 76.0 / 93.6 |
| Gemini-2.5-Flash | 2.55 / 26.1 / 15.3 | 0.59 / 21.3 / 4.73 | 1.84 / 31.9 / 3.68 | 0.00 / 22.7 / 0.67 | 0.40 / 51.2 / 2.00 |
| Gemini-2.5-Pro | 8.28 / 36.9 / 20.4 | 1.18 / 39.6 / 17.2 | 0.61 / 37.4 / 8.59 | 1.39 / 31.5 / 3.36 | 4.00 / 65.9 / 22.8 |
| Gemini-3-Flash | 14.6 / 36.3 / 52.2 | 5.92 / 30.8 / 20.1 | 2.45 / 29.4 / 2.45 | 0.93 / 38.0 / 13.4 | 11.2 / 66.8 / 45.6 |
| Gemini-3.1-Pro | 16.6 / 37.6 / 70.7 | 7.69 / 21.9 / 41.4 | 1.23 / 26.4 / 16.6 | 4.17 / 32.9 / 34.2 | 22.8 / 69.5 / 52.0 |
| Gemini-3.5-Flash | 16.6 / 41.4 / 66.9 | 5.92 / 19.5 / 47.3 | 4.91 / 30.1 / 15.3 | 4.17 / 30.1 / 29.5 | 22.4 / 61.6 / 52.0 |
| Gemini-3.8-Flash | 17.2 / 38.9 / 66.2 | 7.69 / 20.1 / 37.3 | 4.29 / 36.2 / 12.9 | 9.26 / 35.6 / 41.6 | 28.4 / 75.2 / 56.0 |
| Open-source Specialized VLMs | |||||
| LLaVA-ST | 12.1 / 19.7 / 53.5 | 0.00 / 9.47 / 12.4 | 0.79 / 8.59 / 3.68 | 0.00 / 12.0 / 0.00 | 0.80 / 35.6 / 13.2 |
| Open-source General-Purpose VLMs | |||||
| Qwen3-VL-4B | 5.73 / 25.5 / 42.0 | 0.00 / 4.14 / 13.0 | 0.00 / 8.59 / 0.00 | 0.00 / 13.4 / 0.00 | 0.40 / 39.2 / 0.00 |
| Qwen3-VL-8B | 3.82 / 19.7 / 17.8 | 0.00 / 10.7 / 7.69 | 0.00 / 11.7 / 0.00 | 0.00 / 15.7 / 0.00 | 0.80 / 46.0 / 3.60 |
| Qwen3.5-4B | 2.55 / 30.6 / 13.4 | 0.00 / 14.8 / 7.69 | 0.00 / 12.9 / 0.00 | 0.00 / 28.2 / 1.34 | 0.40 / 49.2 / 2.00 |
| Qwen3.5-9B | 4.46 / 35.0 / 20.4 | 0.59 / 17.2 / 12.4 | 0.61 / 14.7 / 0.00 | 0.00 / 28.2 / 1.34 | 0.40 / 50.4 / 4.80 |
| Qwen3.8-27B | 12.7 / 26.1 / 12.7 | 5.33 / 18.3 / 28.4 | 0.00 / 11.0 / 1.23 | 0.00 / 35.2 / 10.1 | 13.6 / 56.4 / 2.80 |
| Eagle2.5-8B | 0.00 / 24.8 / 1.27 | 0.00 / 7.69 / 2.37 | 0.00 / 7.98 / 0.00 | 0.00 / 15.3 / 0.00 | 0.00 / 47.6 / 0.40 |
| InternVL3-8B | 0.64 / 15.3 / 3.82 | 0.00 / 5.92 / 1.18 | 0.00 / 6.79 / 0.00 | 0.00 / 10.6 / 0.00 | 0.00 / 4.40 / 0.80 |
| InternVL3-14B | 0.64 / 17.8 / 7.64 | 0.00 / 5.33 / 1.18 | 0.00 / 7.36 / 0.00 | 0.00 / 10.6 / 0.00 | 0.00 / 13.2 / 0.00 |
| InternVL3.5-8B | 0.00 / 11.5 / 0.00 | 0.00 / 2.96 / 1.18 | 0.00 / 6.13 / 0.00 | 0.00 / 7.41 / 0.00 | 0.00 / 3.60 / 0.00 |
| Gemma4-12B | 5.10 / 27.4 / 3.82 | 4.14 / 11.2 / 18.3 | 0.00 / 9.20 / 0.00 | 2.31 / 25.0 / 2.01 | 1.60 / 43.2 / 0.00 |