Discovery-Driven Integration of Disjoint Tables via Text
LOKI learns row-sentence links to join disjoint data-lake tables, with 0.982 precision and far lower LLM cost.
Researchers formalize Text-Mediated Join Path Discovery for integrating semantically related data-lake tables that lack explicit join attributes. LOKI uses horizontal bidirectional cross-attention and a global table-text contrastive objective so fine-grained row-sentence associations emerge without local supervision. It converts those links into interpretable join paths and typed integrated tables with sentence-level provenance. On real benchmarks it outperforms multi-modal discovery methods, with 0.982 macro typed-pair precision and up to 40 times lower LLM API cost than direct prompting.
- Formalizes Text-Mediated Join Path Discovery for tables lacking join keys.
- Contrastive training finds row-sentence links without local supervision.
- Reports 0.982 macro typed-pair precision and up to 40× lower LLM cost.
Full article207 words · extracted from arxiv.org · click to collapse
Integrating heterogeneous datasets within data lakes is a critical challenge, particularly for semantically related tables that lack the explicit attributes needed to be joined. We study Discovery-Driven Integration, where the relevant sources and their missing relational structure must be discovered before integration. In this setting, unstructured text provides the evidence that connects otherwise disjoint tables. The fundamental challenge is to discover the relationships at a fine-grained level that connect individual rows from different tables through specific sentences. We formalize this task as Text-Mediated Join Path Discovery and propose a horizontal bidirectional cross-attention architecture called LOKI Latent-space Optimization for Knowledge Integration) that learns contextualized representations of table rows and sentences. Through a global table-text contrastive objective, fine-grained row-sentence associations emerge without explicit local supervision. Existing multi-modal discovery methods largely retrieve coarse-grained column-text associations, whereas integration systems assume supplied row-text links, schemas, or queries. LOKI instead transforms these implicit associations into explicit, interpretable join paths, organizes them into relation-consistent groups, and materializes them as typed integrated tables with sentence-level provenance. Comprehensive evaluations on real-world benchmarks demonstrate that LOKI consistently outperforms state-of-the-art multi-modal data discovery approaches, and materializes typed integrated tables with 0.982 macro typed-pair precision while being up to 40 times cheaper in LLM API cost than direct prompting.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26658