Search: “Gemini API”

Google Research Releases ToolGrad: Answer-First Framework Hits 99.8% Pass Rate for Tool-Use Data Generation

Google Research and partners introduce ToolGrad, a verified tool-chain-first data generation framework reaching 99.8% pass rate and boosting Gemma-3-12B to 83.1 on BFCL.

Researchers from Google, the University of Tokyo, RIKEN AIP, and Tohoku University released ToolGrad, which inverts query-first tool-use data generation by executing and verifying API chains before annotating them with user queries. On the ToolBench database of 16,000+ APIs, ToolGrad raised generation pass rate from 63.8% to 99.8% while increasing tool uses per sample from 2.1 to 3.4 and cutting tool-use steps from 34.3 to 20.0. Fine-tuning Gemma-3 at 1B, 4B, and 12B parameters on the 500-sample ToolGrad-500 dataset lifted ToolGrad-12B to 83.1 on the Berkeley Function Calling Leaderboard, near Gemini 2.5 Pro at 83.2 and ahead of GPT-5 at 74.4. Code is Apache-2.0, with the dataset, PyPI package, and models available on Hugging Face.

MarkTechPost·Sep 11, 06:01 UTC · 5d agoAI research1

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Real-SWE benchmark tests coding agents on licensed private enterprise codebases; top model Fable 5.1 resolves only 38.8% of tasks.

Real-SWE is a new benchmark evaluating frontier AI coding agents on tasks drawn from private production codebases licensed from real companies, spanning billing, tax calculation, and cross-service migrations. Fable 5.1 with Claude Code leads at 38.8% resolution rate (pass@1 over eight runs), followed by GPT-6 Astra Codex CLI at 33.8% and Gemini 3.8 Flash Gemini CLI at 31.2%. Tasks use native harnesses and realistic tooling including Docker, Kubernetes, PostgreSQL, Redis, and Linear; median reference solutions edit 11 files versus 6 for DeepSWE and FrontierCode.

Hacker News · AI·Sep 12, 20:25 UTC · 4d agoAI researchHN 45↑ · 31 comments

Show HN: Pelican-bicycle alternatives (updated for 2026)

Hobbyist benchmark re-runs the pelican-bicycle SVG test on six 2026 frontier models, comparing generation time and API cost per image.

A Show HN post re-runs the classic pelican-bicycle and similar SVG generation tests across six 2026 models: GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, DeepSeek V4 Pro, Qwen3.8 Max, and Fugu Ultra v2, recording wall-clock time and cost. It also lists 2025 baseline runs with ten models including Claude Sonnet 4.5, GPT-5.2 Pro, and Qwen3-VL-235B-A22B-Thinking. DeepSeek V4 Pro is consistently cheapest ($0.04-$0.10) while Qwen3.8 Max is slowest, taking up to roughly 17 minutes per generation.

Hacker News · security·Sep 14, 13:20 UTC · 2d agoAI researchHN 24↑ · 4 comments

Nearly half of open-source AI projects never reach production

Help Net Security·Jul 20, 00:00 UTC · Jul 20, 2026AI research

1Password open sources a benchmark to stop AI agents from leaking credentials

Help Net Security·Feb 12, 00:00 UTC · Feb 12, 2026AI research

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

Quesma benchmarks Qwen3.8 27B quantizations: 4-bit Q4_K_M matches BF16 on key benchmarks while 1-bit collapses to random chance.

Quesma spent roughly $3,000 on Modal GPUs testing Unsloth GGUF quantizations of Qwen3.8 27B across GPQA Diamond, IFBench, and Terminal-Bench 2.1. The 17 GB Q4_K_M quantization matched the 55 GB BF16 model on Terminal-Bench 2.1 and showed little degradation down to 4-bit, while the 2-bit UD-Q2_K_XL dropped noticeably. At 1-bit, scores on GPQA Diamond fell to random-guess levels, with longer reasoning making results worse, and reasoning effort settings significantly affected outcomes.

Hacker News · AI·Sep 8, 14:49 UTC · 8d agoAI researchHN 56↑ · 30 comments1

Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Survey of four harness mechanisms—context budgeting, compaction, todo-state, and memory—that keep long-horizon LLM agents on task across 200+ tool calls.

The article details how agent harnesses, not larger context windows, solve context overflow and goal loss on long-horizon tasks, citing Chroma's Context Rot report showing 18 LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) degrade on long inputs. Concrete implementations include LangChain Deep Agents offloading tool responses over 20,000 tokens to the filesystem and truncating old tool calls at 85% window usage, and Claude Code capping auto memory at 25KB while re-reading the 5 most recently modified files after compaction. OpenAI's Responses API now offers server-side compaction via context_management with a standalone /responses/compact endpoint, which Codex uses for long-running coding tasks. Manus reports a roughly 100:1 input-to-output token ratio per ~50-tool-call task, motivating todo.md state recitation to prevent goal drift.

MarkTechPost·Sep 13, 05:56 UTC · 3d agoAI research1