Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs
A knowledge-graph layer resolves imperfect enterprise prompts better than schema-only semantic matching.
This design-science study evaluates an LLM enterprise data assistant for business users who issue imperfect prompts with domain vocabulary but little schema information. It compares schema-based semantic-similarity mediation with mediation over a knowledge graph automatically constructed from a CFO unit's data lake. Simulation on real data, with LLM-as-a-judge scoring and human validation, finds the knowledge-graph approach better on Precision, Recall, F1, and preservation of user intent.
- Study targets imperfect prompts against a CFO-unit data lake.
- Compares schema similarity with an automatically built knowledge graph.
- Knowledge-graph mediation wins on Precision, Recall, F1, and intent.
- Evaluation uses simulation, LLM-as-a-judge, and human validation.
From the original article
1 min read at arxiv.orgEnterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information.
By Niklas Wagner · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.