Search: “agent benchmarks”

10 stories in the last 30d

Searchkeyword smart Sortlatest hot importance relevance

DomainAll Cyber AIWindow24h 3d 7d 30d AllIn the wild Important only Show duplicates

CategoryAll AI research 78 Model release 47 AI safety & security 43 AI industry 22 Industry 20 AI tools & infra 10 Research 8 Tools 5 Data breach 2 Malware 1 Threat actor 1

SeverityAll critical 0 high 5 medium 14 low 0 info 218

UC Berkeley Researchers Release CUA-Lite, an Open Platform Unifying Sandboxes, Data, Evaluation and RL for Computer-Use Agents

UC Berkeley's CUA-Lite is an open platform unifying computer-use agent sandboxes, datasets, evaluation and RL; Lite.OSWorld cuts OSWorld memory 4.1 GB to 0.9 GB.

UC Berkeley researchers released CUA-Lite, an open platform placing agents, environments, traces, and training for computer-use agents behind one action space, one LiteSample schema, and one command across desktop, browser, and mobile. Lite.OSWorld reproduces the OSWorld task suite and evaluators in plain Docker containers (0.9 GB RAM vs 4.1 GB, cold start 23.8s, ~4.6× more parallel instances), with scores matching the QEMU/KVM VM across 13 models. The platform claims 30k+ verifiable tasks, 15+ benchmarks, 10+ agents, and 20+ datasets on Hugging Face including Aguvis, OpenCUA, and ScaleCUA. A documented SFT run lifts Qwen3-VL-2B-Instruct mean episode return from 0.138 to 0.237 on the 332-task lite.osworld split.

MarkTechPostSep 6, 06:11 UTC · 11d agoAI tools & infra1

Introducing agentic video understanding with Gemini

Google DeepMind launches agentic video understanding for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite, cutting video-analysis tokens up to 88%.

Google DeepMind launched agentic video understanding across Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite via the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform. The feature replaces static fixed-FPS ingestion with an agentic loop that dynamically searches frames, audio, and transcripts, cutting token consumption by up to 88%, reducing costs by up to 66%, and improving accuracy by up to 7%. Gemini 3.7 Flash with the feature sits at the accuracy-to-cost Pareto frontier on tested video benchmarks, and the capability will later power YouTube's Ask YouTube feature.

Google DeepMindSep 1, 17:08 UTC · 15d agoAI tools & infra

GitHub Introduces Project HydraFusion: Runtime Multi-Model Orchestration That Builds a Workflow Per Coding Task in Copilot CLI

GitHub's Project HydraFusion research preview builds per-task multi-model workflows (Single, Cascade, Critique) in Copilot CLI, reporting +4.9 quality at 67% lower cost on TerminalBench 2.1.

Project HydraFusion is a research preview available on all GitHub Copilot plans inside Copilot CLI that treats model routing as workflow selection, choosing among Single, Cascade (draft plus quality gate), and Critique (cross-family reviewer) execution patterns per request. Against Claude Opus 5 baselines at medium reasoning, fixed HydraFusion policies cut estimated cost 67% while adding 4.9 quality points on TerminalBench 2.1, and cut cost 36% and 65% with slight quality dips on DeepSWE and CheckpointBench. Billing is per token at each underlying model's standard rate; there are no open weights or self-hosting options.

MarkTechPostSep 5, 19:40 UTC · 11d agoAI tools & infra1

MaxKernel: Agentic Kernel Generation for TPUs

Researchers open-source MaxKernel, a multi-agent LLM system that generates and optimizes TPU kernels matching expert hand-tuned baselines on JaxBench.

MaxKernel is a multi-agent system offering three paradigms for TPU kernel development: human-in-the-loop collaborative design, a fully autonomous metric/trace-driven optimization loop, and graph-based autonomous search for global exploration. All paradigms draw on a shared pool of specialized sub-agents for planning, implementation, self-debugging, testing, and hardware profiling. Evaluated on JaxBench's 50 diverse TPU kernel tasks and real-world workloads from open-source models, it consistently matches expert hand-tuned baselines. The system is open-sourced via the AI-Hypercomputer GitHub repository.

Hugging Face daily papersSep 2, 20:00 UTC · 14d agoAI tools & infra1

Show HN: Nari Qwen3-TTS and Qwen3-ASR – High accuracy, low latency and cost

Nari Labs claims top Coval voice AI benchmark rankings with low-latency, low-cost Qwen3-ASR and Qwen3-TTS inference endpoints.

Nari Labs says its Qwen3-ASR Fast endpoint ranks #1 in Coval's time-to-final-segment latency (p50 44 ms) with 3.6% WER at $0.12/hour, behind only AssemblyAI Universal 3.5 Pro on accuracy. Its Qwen3-TTS Fast ranks #2 in time-to-first-audio (p50 63 ms) and #1 in WER at 3.8%, priced at $10 per 1M characters. The company reports beating the official Qwen3 TTS Flash Realtime endpoint (8.8% WER, 692 ms median TTFA) and Baseten's dedicated endpoint (6.0% WER, 101 ms). Public beta APIs are moving to paid general availability with $20 in credits for existing accounts.

Hacker News · securitySep 14, 16:07 UTC · 2d agoAI tools & infra HN 21↑ · 7 comments1

RTK reports token savings, but our cost benchmarks disagree

Quesma's $1,500 benchmark found RTK cuts reported token output but changes Claude Code and DeepSeek coding costs by only about 5% on Terminal-Bench 2.1.

Quesma benchmarked RTK (Rust Token Killer), a popular tool with 79k GitHub stars that filters terminal output for AI coding agents, whose README claims up to 90% output reduction. Across 1,740 Terminal-Bench 2.1 attempts running Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813, total costs moved only -5% for Fable and +5% for DeepSeek, with pass rates dropping 1-2%. RTK's own rtk gain metric reported 349.2 million tokens saved (an 89% reduction) across 445 DeepSeek attempts, but this did not correlate with actual cost savings, and cached terminal-output reads cost as little as 1/10 to 1/30 of regular input tokens. A bug in rtk find 0.45.0 caused one agent to loop with 339 consecutive errors, costing roughly 9x the baseline attempt, though the task still passed.

Hacker News · securityupdated Sep 11, 22:01 UTC · 5d agofirst Sep 11, 11:15 UTC · 6d agoAI tools & infra 2 sources HN 28↑ · 10 comments1

OpenAI Launches the Agents API in Public Beta, Putting the Codex Harness Behind One API Call

OpenAI released its Agents API in public beta, exposing the managed Codex harness with hosted or self-hosted sandboxes, MCP tools, and subagents.

The Agents API is a managed service built on the open-source Codex harness, handling context compaction, tool search, programmatic tool calling, and multi-agent orchestration. Agents run in OpenAI-hosted sandboxes, self-hosted environments, or partner sandboxes from Blaxel, Cloudflare, Daytona, DigitalOcean, E2B, Modal, Oracle, Runloop, and Vercel. Data residency is US-only and Zero Data Retention is unsupported. Examples use model gpt-6-astra; vendor-reported results include SafetyKit cutting case review cost 60% and Ciridae achieving 4x lower subagent latency.

MarkTechPostupdated Sep 11, 10:00 UTC · 6d agofirst Sep 10, 21:21 UTC · 6d agoAI tools & infra 4 sources1

LandingAI Releases Agentic Document Extraction Gen2 with DPT-3 Pro and DPT-3 Verity

LandingAI shipped Agentic Document Extraction Gen2 with DPT-3 Pro and DPT-3 Verity parsing models, adding usage-based billing, block-tree outputs, and word-level grounding.

LandingAI has generally released Agentic Document Extraction Gen2, rebuilt around two parsing models: DPT-3 Verity for deterministic transcription of digital documents with per-word bounding boxes and confidence scores, and DPT-3 Pro for layout-aware parsing of scans, handwriting, non-Latin scripts, and LaTeX math. Billing changes from a flat 3 credits per page to a page-plus-output-character model (Pro: 1 credit/page plus 0.5 credits per 1,000 output characters on priority; Verity: 0.3 plus 0.2), with an asynchronous standard tier at 0.5x price and vendor-claimed 25-80% cost reductions. Parse v2 returns a document-page-block tree with semantic IDs, normalized bounding boxes, and line- or word-level atomic grounding, replacing flat chunks; Gen1 client code will not run against Gen2 endpoints. Deployment options include US/EU cloud, VPCs on AWS, Azure, and Google Cloud, Snowflake, and air-gapped on-premises environments, with automated model routing planned for fall 2026.

MarkTechPostSep 10, 02:13 UTC · 7d agoAI tools & infra

NVIDIA Details BioNeMo Inference Runtime (BioIR): 2.90x Higher Boltz-2 Folding Throughput and 58.5K Residues per GPU-Hour on 8xH100

NVIDIA released BioNeMo Inference Runtime (BioIR), an open-source PyTorch-compatible library delivering 2.90x higher Boltz-2 protein-folding throughput on 8xH100 GPUs.

NVIDIA detailed BioIR, a Python library that accelerates Boltz-2, OpenFold2, and OpenFold3 structure-prediction inference on NVIDIA GPUs while preserving standard PyTorch workflows. On a matched benchmark of 1,000 human dimers on 8xH100 80GB GPUs, BioIR delivered 58.5K folded residues per GPU-hour versus 20.2K for a torch.compile baseline, a 2.90x throughput gain. BioIR already powered the AlphaFold Database expansion, generating about 31 million candidate complexes across 4,777 proteomes, with 1.81 million released as high-confidence predictions. Extrapolated to 1 million targets, estimated folding energy drops from 35 MWh to 11 MWh at 8-GPU TDP equivalents.

MarkTechPostSep 10, 21:57 UTC · 6d agoAI tools & infra1

16 governance tools for securing your AI fleet

CSO Online reviews 16 AI governance and security tools, including Collibra, Credo AI, F5/CalypsoAI, Fiddler AI, and Guardrails AI, for managing LLM risks.

CSO Online surveys 16 vendors in the emerging AI governance and guardrails market for keeping production LLMs in check. Featured products include Collibra's AI Command Center, Confident Security's OpenPCC, Credo AI's Govern AI Assistant, F5's acquired CalypsoAI, Fiddler AI's control plane, and Guardrails AI's Snowglobe simulator. The tools address hallucination tracking, PII leakage, prompt injection and jailbreak defense, and compliance with frameworks such as the EU AI Act, SOC2, ISO-42001, and GDPR.

CSO Online08:25 UTC · 2h agoAI tools & infra