Communication-efficient model pruning for federated learning in mobile edge computing

HU Haifeng; ZHANG Xi; ZHAO Haitao; WU Jiansheng

doi:10.11959/j.issn.2096-3750.2024.00392

您当前的位置：

首页 >

文章列表页 >

Communication-efficient model pruning for federated learning in mobile edge computing

Theory and Technology | 更新时间：2025-03-13

- Communication-efficient model pruning for federated learning in mobile edge computing
- Chinese Journal on Internet of Things Vol. 8, Issue 3, Pages: 112-126(2024)
- 作者机构：
  
  1.南京邮电大学通信与信息工程学院，江苏南京 210003
  2.南京邮电大学物联网学院，江苏南京 210003
  3.南京邮电大学计算机学院，江苏南京 210023
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62071242;61571233;61901229;61872198;62371245)
- DOI：10.11959/j.issn.2096-3750.2024.00392
  CLC：
- Received：15 September 2023，
  
  Revised：2024-06-07，
  
  Published：10 September 2024
- 稿件说明：
移动端阅览
胡海峰,张熙,赵海涛等.移动边缘计算中通信高效的联邦学习模型剪枝算法[J].物联网学报,2024,08(03):112-126.

HU Haifeng,ZHANG Xi,ZHAO Haitao,et al.Communication-efficient model pruning for federated learning in mobile edge computing[J].Chinese Journal on Internet of Things,2024,08(03):112-126.
胡海峰,张熙,赵海涛等.移动边缘计算中通信高效的联邦学习模型剪枝算法[J].物联网学报,2024,08(03):112-126. DOI： 10.11959/j.issn.2096-3750.2024.00392.

HU Haifeng,ZHANG Xi,ZHAO Haitao,et al.Communication-efficient model pruning for federated learning in mobile edge computing[J].Chinese Journal on Internet of Things,2024,08(03):112-126. DOI： 10.11959/j.issn.2096-3750.2024.00392.

摘要

移动边缘计算中，边缘端服务器和移动终端利用联邦学习分布式架构构建深度模型，使终端之间无须共享数据就可以协作训练，然而深度模型训练需要在服务器和多个客户终端之间进行多轮通信传输，需要消耗大量的通信资源和训练开销。针对这个问题，提出了一种通信高效的联邦学习模型剪枝（CEMP-FL

communication-efficient model pruning for federated learning）架构，服务器运行单次层平衡网络剪枝（SBNP

single-shot layer balance network pruning）算法，通过粗剪枝和精细剪枝的组合，并结合非结构化稀疏参数压缩，显著减少了通信过程中传输的深度模型参数量，并有效地减少了终端侧训练样本分布差异带来的剪枝偏差。同时，使用网络剪枝的层平衡策略（LBP

layer balance policy），确保了深度模型层之间的参数量平衡，在稀疏度很大的情况下有效地推迟了深度模型坍塌。最后，基于两种基准数据集讨论了CEMP-FL在无线场景中的性能，实验表明，提出的CEMP-FL在保证性能的前提下取得了最优的通信成本压缩比，实现了联邦学习分布式训练架构下的高效通信。

Abstract

In the mobile edge computing scenario

the distributed architecture of federated learning allows the edge server and mobile terminals to cooperatively train the deep model

without necessitating sharing of local data across the mobile terminals. While the training process generally consists of multiple rounds between the server and several clients

which can incur high communication costs and training overhead. To address this issue

a communication-efficient model pruning for federated learning (CEMP-FL) framework

which employed the single-shot layer balance network pruning (SBNP) algorithm

combined with unstructured sparse weight compression

was proposed to significantly reduce the size of the global model

and to effectively alleviate the biased pruning due to training samples discrepancy between clients. Meanwhile

layer balance policy (LBP) was adopted to ensure a balance of the model parameters between layers

which could effectively circumvent the problem of layer-collapse in the case of high sparsity. Finally

the performance of CEMP-FL in wireless scenarios was discussed on two benchmark datasets. The experimental results show that the proposed CEMP-FL method achieves the highest compression ratio of communication costs while maintaining performance

and provides efficient communication in the distributed architecture of federated learning.

关键词

Keywords

references

MCMAHAN B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Proceedings of the Artificial Intelligence and Statistics . PMLR , 2017 : 1273 - 1282 .

YANG Q , LIU Y , CHEN T J , et al . Federated machine learning: concept and applications [J ] . ACM Transactions on Intelligent Systems and Technology , 10 ( 2 ): 1 - 19 .

DUAN M M , LIU D , CHEN X Z , et al . Self-balancing federated learning with global imbalanced data in mobile systems [J ] . IEEE Transactions on Parallel and Distributed Systems , 2021 , 32 ( 1 ): 59 - 71 .

KONEČNÝ J , MCMAHAN H B , YU F X , et al . Federated learning: strategies for improving communication efficiency [J ] . arXiv preprint , 2016 , arXiv: 1610.05492 .

HAMER J , MOHRI M , SURESH A T . Fedboost: a communication-efficient algorithm for federated learning [C ] // International Conference on Machine Learning . PMLR , 2020 : 3973 - 3983 .

LEE N , AJANTHAN T , TORR P H S . SNIP: single-shot network pruning based on connection sensitivity [J ] . arXiv preprint , 2018 , arXiv: 1810.02340 .

ZHU M , GUPTA S . To prune, or not to prune: exploring the efficacy of pruning for model compression [J ] . arXiv preprint , 2017 , arXiv: 1710.01878 .

SINGH S P , JAGGI M . Model fusion via optimal transport [J ] . arXiv preprint , 2019 , arXiv: 1910.05653 .

PALIHAWADANA C , WIRATUNGA N , WIJEKOON A , et al . FedSim: similarity guided model aggregation for Federated Learning [J ] . Neurocomputing , 2022 , 483 ( C ): 432 - 445 .

LI T , HU S , BEIRAMI A , et al . Ditto: fair and robust federated learning through personalization [C ] // International Conference on Machine Learning . PMLR , 2021 : 6357 - 6368 .

MARFOQ O , NEGLIA G , BELLET A , et al . Federated multi-task learning under a mixture of distributions [J ] . Advances in Neural Information Processing Systems , 2021 , 34 : 15434 - 15447 .

XIE C , HUANG K , CHEN P Y , et al . DBA: distributed backdoor attacks against federated learning [C ] // International Conference on Learning Representations . 2020 .

YIN H X , MALLYA A , VAHDAT A , et al . See through gradients: image batch recovery via GradInversion [C ] // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2021 : 16332 - 16341 .

LI Z H , ZHANG J X , LIU L Y , et al . Auditing privacy defenses in federated learning via generative gradient leakage [C ] // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2022 : 10122 - 10132 .

FRABONI Y , VIDAL R , KAMENI L , et al . Clustered sampling: Low-variance and improved representativity for clients selection in federated learning [C ] // International Conference on Machine Learning . PMLR , 2021 : 3407 - 3416 .

BALAKRISHNAN R , LI T , ZHOU T , et al . Diverse client selection for federated learning via submodular maximization [C ] // International Conference on Learning Representations . 2022 .

TANAKA H , KUNIN D , YAMINS D L K , et al . Pruning neural networks without any data by iteratively conserving synaptic flow [C ] // Proceedings of the Proceedings of the 34th International Conference on Neural Information Processing Systems . New York : ACM , 2020 : 6377 - 6389 .

RAMANUJAN V , WORTSMAN M , KEMBHAVI A , et al . What’s hidden in a randomly weighted neural network? [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 11890 - 11899 .

PATIL S M , DOVROLIS C . PHEW: constructing sparse networks that learn fast and generalize well without training data [C ] // International Conference on Machine Learning . PMLR , 2021 : 8432 - 8442 .

LIN J , LUO X T , HONG M , et al . Memory-friendly scalable super-resolution via rewinding lottery ticket hypothesis [C ] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2023 : 14398 - 14407 .

SRINIVAS S , SUBRAMANYA A , BABU R V . Training sparse neural networks [C ] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE Press , 2017 : 455 - 462 .

FRANKLE J , DZIUGAITE G K , ROY D M , et al . Linear mode connectivity and the lottery ticket hypothesis [C ] // Proceedings of the Proceedings of the 37th International Conference on Machine Learning . New York : ACM , 2020 : 3259 – 3269 .

YUAN G Z , SHEN L , ZHENG W S . A block decomposition algorithm for sparse optimization [C ] // Proceedings of the Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . New York : ACM , 2020 : 275 - 285 .

HE Y , DING Y H , LIU P , et al . Learning filter pruning criteria for deep convolutional neural networks acceleration [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 2006 - 2015 .

HU R , GONG Y M , GUO Y X . Federated learning with sparsification-amplified privacy and adaptive optimization [C ] // Proceedings of the Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 1463 - 1469 .

STRIPELIS D , GUPTA U , STEEG G V , et al . Federated progressive sparsification (purge, merge, tune) [J ] . arXiv preprint , 2022 , arXiv: 2204.12430 .

QIU X C , FERNANDEZ-MARQUES J , GUSMAO P P , et al . ZeroFL: efficient on-device training for federated learning with local sparsity [EB/OL ] . 2022 : arXiv : 2208 . 02507 . http://arxiv.org/abs/2208.02507 http://arxiv.org/abs/2208.02507

PASE F , ISIK B , GUNDUZ D , et al . Efficient federated random subnetwork training [C ] // Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022). 2022 .

CHEN X Z , ZHU J Y , JIANG J B , et al . Tight compression: compressing CNN model tightly through unstructured pruning and simulated annealing based permutation [C ] // Proceedings of the 2020 57th ACM/IEEE Design Automation Conference (DAC) . Piscataway : IEEE Press , 2020 : 1 - 6 .

SONG HAN , HUIZI MAO , AND WILLIAM J DALLY . Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding . In International Conference on Learning Representations , 2016

VAHIDIAN S , MORAFAH M , LIN B . Personalized federated learning by structured and unstructured pruning under data heterogeneity [C ] // Proceedings of the 2021 IEEE 41st International Conference on Distributed Computing Systems Workshops (ICDCSW) . Piscataway : IEEE Press , 2021 : 27 - 34 .

Views

209

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on low-latency transmission migration method for federated learning models in the Internet of vehicles

Latency-aware semi-synchronous federated learning client resource co-scheduling scheme for IIoT

Client selection for federated learning against label flipping attacks

Joint multi-dimensional resource optimization for model compression in wireless federated learning

Related Author

Zhang Yin

Zhang Ke

Jiang Chi

Yin Hongbo

Wang Shuai

XIA Tianle

WANG Ye

SHI Ying

Related Institution

Guangdong Intelligent Robot Research Institute

School of Information and Communication Engineering, University of Electronic Science and Technology of China

Shenzhen Institute for Advanced Study, UESTC

Jiangsu Mobile Information System Integration Co., Ltd.

Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China

⁰