Meta-Llama-3-8B-Instruct

To understand whether we're making genuine progress on reasoning, we entered our AI models in five STEM Olympiad competitions. The results: 🏅 Asian Physics Olympiad (APhO): Perfect score, theory exam

To understand whether we're making genuine progress on reasoning, we entered our AI models in five STEM Olympiad competitions. The results: 🏅 Asian Physics Olympiad (APhO): Perfect score, theory exam 🏅 International Physics Olympiad (IPhO): Perfect score, theory exam 🥇 https://t.co/fh1h8I1d3S

How Meta’s AI Models Are Powering the First Wave of Genesis Mission Projects

Reimagining Independence: How Meta’s AI Models Are Helping the University of Pittsburgh Transform Assistive Robotics

We heard you and are happy to announce that Muse Spark 1.1 is now available on @OpenRouter for US-based developers. We look forward to seeing what the community builds.

MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. Existing multilingual corpora and benchmarks are often poor proxies for the language, containing substantial amounts of noisy, machine-translated, and misclassified text. We address these gaps by introducing Oytser, a high-quality Yiddish pretraining corpus that combines contemporary web-native sources with literary materials, and Kashes, a multi-task benchmark spanning translation, linguistic analysis, information extraction, and language understanding. Using these resources, we continue pretraining Llama 3.1 8B to obtain MameLoshnLM. Across the tasks in the benchmark, MameLoshnLM outperforms open baselines of similar scale. Our analyses show that these gains are not only quantitative: relative to general-purpose multilingual models, MameLoshnLM better captures language-defining lexical and morphological patterns, pointing to a broader failure mode of noisy web-scale multilingual data for low-resource languages. Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.

Omega-S: A Functional Resilience Index for LLM Fine-Tuning

Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} and https://github.com/run-llama/ExtractBench{GitHub}.

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs). As a foundational empirical validation of this method, this work focuses on causal bias localization. Using minimally contrastive prompt pairs and inference-time activation capture, the method identifies neurons that react differentially when processing demographic attributes in GLU architectures, evaluating the signal at the down_proj input. Empirical evaluation was conducted on models of up to 3 billion parameters (Llama-3.2 family and Salamandra-2B), combining standardized benchmark evaluation with qualitative text generation experiments. Results demonstrate that zeroing the identified neurons alters how the model responds to associated demographic variables. However, rather than producing flat mitigation, the intervention causes bidirectional bias destabilization: because BiasScore is unsigned, candidate sets mix neurons that push toward and against the stereotype, and the net effect on aggregate bias depends on which sign dominates. The intervention is extremely surgical: zeroing at most 40 neurons in Llama-3.2-1B (less than 0.031% of total MLP width) achieves a mean retention of 99.49% in reasoning and general knowledge capabilities. These findings empirically confirm that demographic bias processing and model capabilities operate on dissociable circuits, establishing the methodological foundations for transitioning from blind zeroing toward directional behavior modulation.

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params

Llama-3-8B-Instruct — BigCodeBench #77

Complete: 36.9 | Instruct: 31.9 | 8B params