arXiv proposes FedReason for LLM-based aggregation in federated learning
Original title: FedReason: Large Language Models as Weight-Space Meta-Aggregators for Heterogeneous Federated Learning
The LLM analyzes weights in a sandbox and submits short Python aggregation programs every ten rounds.
Federated learning servers combine client models with a fixed aggregation rule. FedReason uses a large language model to inspect client model weights and choose the aggregation rule itself. The LLM runs code in a restricted sandbox and submits a short Python program.
- LLM called every ten rounds on CIFAR-10 and CIFAR-100.
- FedReason beats FedAvg by 1.26 percentage points on average.
- 60-round run costs 0.79 USD in API fees on average.
From the original article
1 min read at arxiv.orgFederated learning servers combine client models with a fixed aggregation rule, and each rule in common use is designed for one kind of heterogeneity. We study whether a large language model (LLM) that inspects the client models can choose the rule itself, without any training on the server. In FedReason, the LLM is called every ten rounds.
By Ali Mohammadi Ruzbahani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.