ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode

Search: “evaluation infrastructure”

9 stories in the last 30d

Searchkeywordsmart Sortlatesthotimportancerelevance
DomainAllCyberAIWindow24h3d7d30dAllIn the wildImportant onlyShow duplicates
CategoryAllAI safety & security 47Industry 42AI industry 25Research 13Model release 13Threat actor 13AI research 10Data breach 8Exploit / PoC 8Phishing & fraud 7AI policy 7AI tools & infra 6Tools 6Malware 5
SeverityAllcritical 5high 25medium 43low 2info 143

OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining

OpenWAM releases an open modular stack for world-action model pretraining, plus OpenWAM-alpha trained on about 6,400 hours of egocentric and robot data.

OpenWAM is an open research stack that factorizes World-Action Model pretraining into composable infrastructure, study, and model components with unified training, inference, and evaluation. Controlled experiments distill three principles on knowledge inheritance, world-action synergy, and out-of-domain generalization gains from embodied co-training. The resulting OpenWAM-alpha, pretrained on roughly 6,400 hours of egocentric human and robot data, achieves top-tier results across eight simulation benchmarks and real-robot tests spanning single-arm, bimanual, and dexterous embodiments. The full stack, including infrastructure, evaluation protocols, pretrained models, and data recipes, is released openly.

Hugging Face daily papers · 10d agoAI research

Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems

An empirical study shows distributed Kolmogorov-Arnold Network training reaches 74.7% parallel efficiency at 8 A100 GPUs, with overheads driven by All-Reduce choices.

The study evaluates data-parallel Kolmogorov-Arnold Network (KAN) training on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel. Strong scaling yields 5.97x speedup and 74.7% parallel efficiency at 8 GPUs, comparable to conventional deep learning workloads, while communication overhead ranges from 1.3% to 6.1%, driven mainly by All-Reduce algorithm selection and inter-node latency rather than KAN's edge-wise gradient structure. Weak scaling shows an initial single-to-multi-GPU throughput drop followed by stability, and the parameter-to-memory ratio improves with model size even as training time scales unfavorably. The authors provide GPU topology and model-size deployment guidelines for KAN training.

arXiv cs.AI / cs.LG / cs.CL · 9d agoAI research1

Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

Researchers release Phi-Bench, a benchmark evaluating frontier LLMs on open-ended, long-horizon engineering and optimization of the LLM infrastructure stack.

Phi-Bench evaluates LLMs on open-ended engineering of the LLM infrastructure stack, derived from optimization problems studied in frontier research and grounded in real-world code repositories. Tasks range from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Experiments on frontier LLMs reveal current capabilities and limitations on the path toward autonomous optimization of future AI infrastructure.

Hugging Face daily papers · 8d agoAI research1

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

Nuha-Speech initiative builds general-purpose Arabic speech-LLMs using a 1.5M-sample speech QA corpus and fine-tuned Qwen-Omni variants.

The paper introduces Nuha-Speech, an initiative covering dataset construction, model training, and evaluation for Arabic speech large language models. The authors built an Arabic Speech Question-Answering corpus of over 1.5 million training samples and used it for supervised fine-tuning of Qwen-Omni model variants at multiple scales. A tailored evaluation framework with diverse tasks and metrics is designed to assess Arabic speech capabilities under limited resource constraints.

arXiv cs.AI / cs.LG / cs.CL · 6d agoAI research1

Prepared Or Unprepared? Evaluating Healthcare Workforce Readiness for Clinical Adoption of Artificial Intelligence in Nigeria

Survey of 761 Nigerian healthcare professionals finds high AI awareness (92.6%) but limited knowledge, preparedness, and major training and infrastructure barriers.

A cross-sectional study of 761 healthcare professionals across Nigeria, conducted from December 2025 to March 2026, found 92.6% awareness of AI in healthcare but 40.9% reporting low knowledge and only 63.0% feeling adequately prepared. Top barriers were lack of training (84.7%), poor infrastructure (71.1%), and high tool costs (61.0%). Willingness to adopt was strong, with 92.5% interested in training and 78.7% supporting AI in undergraduate curricula; preparedness differed significantly across geopolitical zones and professions.

arXiv cs.AI / cs.LG / cs.CL · 20h agoAI research

ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

ScienceIDE converts scientific code repositories into verifiable agent training environments, producing the PhAI-IDE 4B-72B model family.

ScienceIDE turns scientific code repositories into executable environments supporting task generation, execution, and scientific verification, guided by expert-defined scientific cases and acceptance criteria. Using verified interaction trajectories, the authors train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B. The model family shows gains in held-out scientific-code repair and selected general-purpose code, reasoning, and knowledge benchmarks, evidencing positive transfer from scientific experience.

Hugging Face daily papersupdated · 19h agofirst · 1d agoAI research 2 sources

VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes

VidaForge releases open infrastructure and VIDAFORGE-3M (3.14M clips, 6,475 hours) linking video pretraining data recipes to downstream model performance.

VidaForge is an open research infrastructure that represents a video pretraining data recipe as an executable five-stage workflow from raw videos to training datasets. The team compares data recipes with different coverage and quality during early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1, finding that broader-coverage recipes achieve the highest downstream benchmark scores while loss-based evaluation favors different recipes. They also release VIDAFORGE-3M, containing 3.14 million scene-level clips totaling 6,475 hours with fine-grained annotations and curation signals for video data-recipe research.

Hugging Face daily papers · 11d agoAI research

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Real-SWE benchmark tests coding agents on licensed private enterprise codebases; top model Fable 5.1 resolves only 38.8% of tasks.

Real-SWE is a new benchmark evaluating frontier AI coding agents on tasks drawn from private production codebases licensed from real companies, spanning billing, tax calculation, and cross-service migrations. Fable 5.1 with Claude Code leads at 38.8% resolution rate (pass@1 over eight runs), followed by GPT-6 Astra Codex CLI at 33.8% and Gemini 3.8 Flash Gemini CLI at 31.2%. Tasks use native harnesses and realistic tooling including Docker, Kubernetes, PostgreSQL, Redis, and Linear; median reference solutions edit 11 files versus 6 for DeepSWE and FrontierCode.

Hacker News · AI · 4d agoAI researchHN 45↑ · 31 comments1

Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes

Independent developer Hugo Vergnes trained a 3.8B-parameter Llama-style model to 0.384 CORE on 65B tokens for $998 in 43 hours on rented B200s.

Hugo Vergnes trained little-lm, a 3.848B-parameter decoder-only LLM, on 65.3B tokens in 43 hours for $998 using rented NVIDIA B200s, scoring 0.384 on the CORE benchmark and beating nanochat d32 (0.310) at similar cost. The Llama-style architecture uses RMSNorm, RoPE, GQA with 24 query and 8 KV heads, relu-squared MLPs, QK-norm, and ResFormer-style value embeddings that account for 19% of parameters. Key wins included the Muon optimizer for matrix parameters, a trapezoidal learning-rate schedule with linear cooldown, FP8 training plus vocabulary padding for roughly 33% throughput gains, and the ClimMix dataset over FineWeb-Edu. The project, inspired by Karpathy's nanochat, was built as a config-driven YAML framework for small LLM training.

Hacker News · security · 7d agoAI researchHN 28↑ · 2 comments1