30
Top 5 AI Gateways for Enterprise (2026 Guide)
A 2026 buyer's guide ranks NeuralTrust TrustGate, Kong AI Gateway, and Cloudflare AI Gateway as top enterprise AI gateways for security and governance.
The guide evaluates enterprise AI gateways on security, governance, routing, observability, and agent ecosystem support. NeuralTrust TrustGate ranks first for identity-aware agent governance across models, MCP servers, tools, and agent-to-agent traffic, with SaaS, hybrid, and private deployment options. Kong AI Gateway is recommended for organizations with mature API infrastructure, while Cloudflare AI Gateway emphasizes caching, retries, model fallbacks, and prompt/response guardrails.
12
PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
PACE cuts perceived latency in retrieval-augmented dialogue serving via cascading routing and filler control, reaching 0.41s P95 under load.
PACE is a serving framework for retrieval-augmented dialogue that optimizes Perceived Time-to-First-Response (PTFR) under quality and cost constraints. It combines a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission, deployed on a humanoid-robot sales service. On 75k CarQA requests, the cascade halved pure-LLM P95 PTFR (0.29s vs 0.53s) and the adaptive controller reached 0.41s P95, 2.4x better than RAG at high load; filler calls dropped 94% and stale answers fell from 86% to 0%.
20
A10 Networks introduces AI Gateway to secure and manage enterprise AI
A10 Networks launches AI Gateway, a control plane for routing, cost management, and governance of enterprise AI agents and LLMs.
A10 Networks announced general availability of the A10 AI Gateway, a centralized control plane providing identity-based AI access policies, smart routing that matches request complexity to model capability, and per-request dollar cost tracking with per-team token budgets. The product enforces business-layer TPM/RPM rate limiting and integrates with A10's TrojAI and ThreatX AI security portfolio across the AI lifecycle. It runs entirely in the customer environment—on-premises, private cloud, or air-gapped—for full data sovereignty.
28
30
30
30
30
F5 enhances AI Gateway to control AI costs, access, and security
F5 integrated AI Gateway into its AI Security Platform, adding model routing, MCP governance, and guardrails, claiming up to 60% token spend reduction.
F5 announced AI Gateway enhancements combining a Model Gateway for cost optimization, an MCP Gateway for agent-to-tool access control, and AI Guardrails for prompt and response inspection. The company cited its 2026 State of Application Strategy Report finding 77% of organizations now treat inference as their dominant AI activity and manage an average of seven AI models. F5 claims smart routing, semantic caching, and GPU-aware load balancing can cut token spend by up to 60% without application changes. The gateway enforces budgets, model routing policies, and agent access controls centrally across SaaS, hybrid SaaS, and hybrid multicloud deployments, with air-gapped support planned.
30
30
30
30
30
So you want to use OpenRouter?
Simon Willison highlights OpenRouter pitfalls: provider routing makes the same model endpoint behave inconsistently across backends.
Simon Willison discusses problems, raised by Mohamed Moustafa, with relying on OpenRouter's single API endpoint that automatically routes to the most cost-effective provider. Different backend providers run different serving software with different optimizations and settings, so requests to the same endpoint can behave in different ways. Some providers lack vision capability for vision models, and the reasoning effort option can be processed differently across providers.
38
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat online storage platform now handles over 70 million requests per second and 500 PB of data for 1 billion users.
OpenAI details the evolution of Habitat, its online storage platform backing ChatGPT and other products, which began in mid-2024 as a Python client-side library over Azure Cosmos DB. Habitat now processes more than 70 million requests per second, serves over 500 petabytes of data across nearly 40 geographic regions, and supports over 1 billion users weekly. By mid-2025 the client library approach became brittle, so OpenAI moved Habitat into a standalone service to centralize deployments, observability, and multi-tenancy reliability. This is part one of a two-part series; a future post will cover read optimization and scaling the Azure Cosmos DB partnership.
30
30
30
30
30
30
30
harshatheg/Qwen-2.5-1B-RLCD — new model trending #30 on Hugging Face
A community MLX inference engine evaluates constrained JSON schema fields in parallel on Apple Silicon, reporting 5.6-7.0x latency speedups with guaranteed schema validity.
The repository harshatheg/Qwen-2.5-1B-RLCD appeared at #30 on Hugging Face trending, but its content describes Parallel Constrained Decoding, an MLX-based inference engine for structured extraction and classification on Apple Silicon Macs. Benchmarked with mlx-community/Qwen2.5-1.5B-Instruct-4bit on an M4 Max, it reports 5.6x-7.0x latency reductions (e.g., 1,900 ms to 270 ms for a 28-field support triage task) with 100% syntactic validity and calibrated field-level probabilities. The engine prefills a single KV-cache, broadcasts it across all schema fields, and slices logits to valid candidate tokens for enum fields with up to 255 choices.
15
NVIDIA Open-Sources OSMO: One YAML Orchestrates Physical AI Training, Simulation, and Robot Testing
NVIDIA open-sourced OSMO, a Kubernetes-native YAML orchestrator running physical-AI training, simulation, and robot testing across mixed GPU tiers.
OSMO (Apache-2.0, latest release 6.3.1) lets teams describe training, simulation, and hardware-in-the-loop pipelines in a single YAML and routes tasks across datacenter GPUs (GB200), workstation RTX hardware, and edge devices like Jetson AGX Thor. It ships Helm charts and containers on NGC, uses the KAI Scheduler with NVLink topology-aware placement, and includes RBAC, OAuth2, and TLS termination. NVIDIA says it is battle-tested on GR00T, Isaac Lab, Isaac Sim, and Isaac ROS, and integrates with Claude Code, OpenAI Codex, and Cursor agents.
48
LandingAI Releases Agentic Document Extraction Gen2 with DPT-3 Pro and DPT-3 Verity
LandingAI shipped Agentic Document Extraction Gen2 with DPT-3 Pro and DPT-3 Verity parsing models, adding usage-based billing, block-tree outputs, and word-level grounding.
LandingAI has generally released Agentic Document Extraction Gen2, rebuilt around two parsing models: DPT-3 Verity for deterministic transcription of digital documents with per-word bounding boxes and confidence scores, and DPT-3 Pro for layout-aware parsing of scans, handwriting, non-Latin scripts, and LaTeX math. Billing changes from a flat 3 credits per page to a page-plus-output-character model (Pro: 1 credit/page plus 0.5 credits per 1,000 output characters on priority; Verity: 0.3 plus 0.2), with an asynchronous standard tier at 0.5x price and vendor-claimed 25-80% cost reductions. Parse v2 returns a document-page-block tree with semantic IDs, normalized bounding boxes, and line- or word-level atomic grounding, replacing flat chunks; Gen1 client code will not run against Gen2 endpoints. Deployment options include US/EU cloud, VPCs on AWS, Azure, and Google Cloud, Snowflake, and air-gapped on-premises environments, with automated model routing planned for fall 2026.
25
GitHub Introduces Project HydraFusion: Runtime Multi-Model Orchestration That Builds a Workflow Per Coding Task in Copilot CLI
GitHub's Project HydraFusion research preview builds per-task multi-model workflows (Single, Cascade, Critique) in Copilot CLI, reporting +4.9 quality at 67% lower cost on TerminalBench 2.1.
Project HydraFusion is a research preview available on all GitHub Copilot plans inside Copilot CLI that treats model routing as workflow selection, choosing among Single, Cascade (draft plus quality gate), and Critique (cross-family reviewer) execution patterns per request. Against Claude Opus 5 baselines at medium reasoning, fixed HydraFusion policies cut estimated cost 67% while adding 4.9 quality points on TerminalBench 2.1, and cut cost 36% and 65% with slight quality dips on DeepSWE and CheckpointBench. Billing is per token at each underlying model's standard rate; there are no open weights or self-hosting options.
42
30
30
30
30
30
30
Arista Networks announces next generation 7130 Series systems designed for demanding in-network apps
30
30
30
30
30
30
30
Akuity gives AI agents operational context to safely ship software
Akuity launches Agentic Control Plane and MCP Server to govern AI agent actions inside production software delivery pipelines.
Akuity introduced an Agentic Control Plane and MCP Server that lets AI agents such as Claude, Codex, and Cursor access deployment and cluster context under existing identity, permission, and audit controls. MLB reported the product identified over 100 degraded applications within hours. Akuity positions it against open-source MCP servers granting raw access with no identity inheritance or policy layer.
15
RTK reports token savings, but our cost benchmarks disagree
Quesma's $1,500 benchmark found RTK cuts reported token output but changes Claude Code and DeepSeek coding costs by only about 5% on Terminal-Bench 2.1.
Quesma benchmarked RTK (Rust Token Killer), a popular tool with 79k GitHub stars that filters terminal output for AI coding agents, whose README claims up to 90% output reduction. Across 1,740 Terminal-Bench 2.1 attempts running Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813, total costs moved only -5% for Fable and +5% for DeepSeek, with pass rates dropping 1-2%. RTK's own rtk gain metric reported 349.2 million tokens saved (an 89% reduction) across 445 DeepSeek attempts, but this did not correlate with actual cost savings, and cached terminal-output reads cost as little as 1/10 to 1/30 of regular input tokens. A bug in rtk find 0.45.0 caused one agent to loop with 339 consecutive errors, costing roughly 9x the baseline attempt, though the task still passed.
42