Name: flan-ul2
Rating: 24.5 (568 reviews)
Author: Google

flan-ul2 by Google | AI Market Cap

HF PapersGoogleresearch2w ago

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

View Source

#huggingface#daily-papers

HF PapersGoogleresearch3w ago

From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents

Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct weakness profiles and benefit from different training environments. Weakness-conditioned adaptation consistently outperforms matched random and hard-example alternatives on ProcTHOR, while the resulting gains for Uni-NaVid and Gemini also extend to HM3D after ProcTHOR adaptation. Iterative re-diagnosis further shows that dominant weaknesses can persist or shift as the policy improves, providing updated guidance for subsequent environment construction. Together, these results show that policy failures are not only diagnostic signals, but actionable guidance for constructing training environments that target a policy's current weaknesses.

View Source

#huggingface#daily-papers

flan-ul2

Rates & subscription access

Tracked access rates

Related provider subscriptions

How complete is this record?

Similar Models

Research-backed technical record

Introducing Genie 4 from @GoogleDeepMind, our most realistic and interactive world model yet 🌍 Now you can use natural language to dynamically steer the world and its soundscape in real-time, whether

Social & Blog Posts5

Research Papers5

It’s Friday 🎉 Here’s a look at this week’s announcements: — https://t.co/a9bSlm0Yxk is now available globally in English, allowing anyone to use our detector portal to easily verify if images, video,

Quaternion Approximate Networks for Enhanced Image Classification and Oriented Object Detection

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Diffusion Language Models Change Speed Across Tasks and Languages

We built PlanAssist to help UK Government planning officers speed up building and renovations across the country. Built with Gemini, it has shown it can cut the time planning officers spend assessing

Introducing Genie 4 from @GoogleDeepMind, our most realistic and interactive world model yet 🌍 Now you can use natural language to dynamically steer the world and its soundscape in real-time, whether

This is Genie 4, our most realistic and interactive world model yet. 🌐 Explore dynamic environments in real time to see how the scene responds, sounds, and how the people in it move, react, and speak

Over half of reported crimes in Singapore use sophisticated social engineering. With @GovTechSG, we’re building on-device fraud detection using our open model Gemma. By fine-tuning it on local scam ta

It’s Friday 🎉 Here’s a look at this week’s announcements: — https://t.co/a9bSlm0Yxk is now available globally in English, allowing anyone to use our detector portal to easily verify if images, video,

Quaternion Approximate Networks for Enhanced Image Classification and Oriented Object Detection

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Diffusion Language Models Change Speed Across Tasks and Languages

From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents

APEX: Asynchronous Prefix Denoising for Extreme Compression