Learning Sparse Decision Trees via Transformer Variational Auto-Encoders
TREVIS uses a Tree Transformer VAE latent space to learn decision trees matching near-optimal predictive performance while improving structural sparsity.
TREVIS learns decision trees optimized for complex objectives by exploring the latent space of a Tree Transformer Variational Auto-Encoder (TTVAE). Mapping trees to continuous latent representations replaces the discrete search space with a continuous one, enabling gradient-based optimization through a differentiable surrogate model. Experiments show TREVIS matches the predictive performance of near-optimal algorithms while improving structural sparsity, targeting high-stakes contexts needing transparent decision logic.
- Tree Transformer VAE maps decision trees to continuous latent space
- Differentiable surrogate enables gradient-based tree optimization
- Matches near-optimal predictive performance with better structural sparsity
Full article139 words · extracted from huggingface.co · click to collapse
Decision trees are among the most widely used models in machine learning, largely due to their transparent decision logic, making them well-suited for high-stakes decision-making contexts. However, most existing learning algorithms focus on predictive performance, overlooking the joint optimization of other desirable properties, such as structural sparsity. In this work we propose TREVIS, an approach for learning decision trees with respect to complex objectives, based on the exploration of the latent space of a Tree Transformer Variational Auto-Encoder (TTVAE). By mapping decision trees onto latent representations, TREVIS replaces the discrete search space with a continuous one, enabling gradient-based optimization via a differentiable surrogate model. We experiment with TREVIS for learning decision trees that jointly optimize predictive performance and sparsity. Results show that TREVIS discovers decision trees matching the predictive performance of existing near-optimal algorithms while improving their structural sparsity.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.01430