Orbit AI multi-tool agent leads document RAG on 80 questions
Original title: Orbit AI: Evaluating Tool-Augmented Agents For Enterprise Question Answering
Gemini 3.5 Flash Lite scored 86.25 percent with all sources; local Llama 3.2 full-agent accuracy was 52.50 percent.
Orbit AI is a LangGraph ReAct agent that retrieves from project documents, task systems and team conversations. On 80 held-out Phase 3 questions, Gemini 3.5 Flash Lite reached 86.25 percent with the full agent, 60.00 percent with document-only RAG and 23.13 percent without retrieval. Multi-source accuracy rose from 40.63 percent to 84.38 percent. The authors caution that Gemini ran via its API while Llama 3.2 ran locally on Ollama, and note a small artificial workspace plus similarity between Q53 and the system prompt.
Unconfirmed: The authors say the Gemini-versus-Llama gap cannot be pinned to one cause.
- Document-only RAG trailed the Gemini agent by 26.25 percentage points.
- The multi-source gain for the full Gemini agent was 43.75 points.
- A local Llama 3.2 setup scored 5.50 percent under rubric v2.2.
- Authors say Q53 resembles a problem already in the system prompt.
- AI models
- Gemini 3.5 Flash LiteLlama 3.2
From the original article
1 min read at arxiv.orgEnterprise information is commonly stored across multiple platforms rather than in one centralized location. Answering questions about workplace activities may require information from several sources, including project documents, task management systems, and team conversations.
By Rishabh Gupta · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.