Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing
HMP-GAE filters Byzantine updates during federated training of Qwen-2.5 and Llama-3.2.
The paper proposes HMP-GAE to keep Byzantine model updates from corrupting federated LLM aggregation without accessing private local data. Low-rank updates are hypergraph nodes, and message passing with historical checkpoints drops semantically inconsistent updates. On Qwen-2.5 and Llama-3.2, using AG News and Yahoo Answers, it improves accuracy by 1.21%, cuts classification semantic entropy by 44.08%, and lowers perplexity by 13.34% versus existing defenses.
- HMP-GAE models low-rank updates as hypergraph nodes for cross-agent reasoning.
- It suppresses semantically inconsistent updates without reading private local data.
- Backbones are Qwen-2.5 and Llama-3.2 on AG News and Yahoo Answers.
- Accuracy rises 1.21%; semantic entropy falls 44.08%; perplexity falls 13.34%.
From the original article
1 min read at arxiv.orgFederated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation.
By Kai Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.