移动边缘计算中通信高效的联邦学习模型剪枝算法

胡海峰; 张熙; 赵海涛; 吴建盛

doi:10.11959/j.issn.2096-3750.2024.00392

您当前的位置：

首页 >

文章列表页 >

移动边缘计算中通信高效的联邦学习模型剪枝算法

理论与技术 | 更新时间：2025-03-13

- 移动边缘计算中通信高效的联邦学习模型剪枝算法
- Communication-efficient model pruning for federated learning in mobile edge computing
- 物联网学报 2024年8卷第3期页码：112-126
- 作者机构：
  
  1.南京邮电大学通信与信息工程学院，江苏南京 210003
  2.南京邮电大学物联网学院，江苏南京 210003
  3.南京邮电大学计算机学院，江苏南京 210023
- 作者简介：
  
  [ "胡海峰（1973‒ ），男，博士，南京邮电大学通信与信息工程学院教授，主要研究方向为人工智能、网络信息处理等。" ]
  [ "张熙（1999‒ ），男，南京邮电大学通信与信息工程学院硕士生，主要研究方向为联邦学习、模型剪枝、移动边缘计算等。" ]
  [ "赵海涛（1983‒ ），男，博士，南京邮电大学物联网学院院长，主要研究方向为车联网络、卫星物联网、工业互联网等。" ]
  [ "吴建盛（1979‒ ），男，博士，南京邮电大学计算机学院教授，主要研究方向为人工智能药物设计、软硬件协同加速等。" ]
- 基金信息：
  
  国家自然科学基金项目(62071242;61571233;61901229;61872198;62371245)
- DOI：10.11959/j.issn.2096-3750.2024.00392
  中图分类号：
- 收稿日期：2023-09-15，
  
  修回日期：2024-06-07，
  
  纸质出版日期：2024-09-10
- 稿件说明：
移动端阅览
胡海峰,张熙,赵海涛等.移动边缘计算中通信高效的联邦学习模型剪枝算法[J].物联网学报,2024,08(03):112-126.

HU Haifeng,ZHANG Xi,ZHAO Haitao,et al.Communication-efficient model pruning for federated learning in mobile edge computing[J].Chinese Journal on Internet of Things,2024,08(03):112-126.
胡海峰,张熙,赵海涛等.移动边缘计算中通信高效的联邦学习模型剪枝算法[J].物联网学报,2024,08(03):112-126. DOI： 10.11959/j.issn.2096-3750.2024.00392.

HU Haifeng,ZHANG Xi,ZHAO Haitao,et al.Communication-efficient model pruning for federated learning in mobile edge computing[J].Chinese Journal on Internet of Things,2024,08(03):112-126. DOI： 10.11959/j.issn.2096-3750.2024.00392.

摘要

移动边缘计算中，边缘端服务器和移动终端利用联邦学习分布式架构构建深度模型，使终端之间无须共享数据就可以协作训练，然而深度模型训练需要在服务器和多个客户终端之间进行多轮通信传输，需要消耗大量的通信资源和训练开销。针对这个问题，提出了一种通信高效的联邦学习模型剪枝（CEMP-FL

communication-efficient model pruning for federated learning）架构，服务器运行单次层平衡网络剪枝（SBNP

single-shot layer balance network pruning）算法，通过粗剪枝和精细剪枝的组合，并结合非结构化稀疏参数压缩，显著减少了通信过程中传输的深度模型参数量，并有效地减少了终端侧训练样本分布差异带来的剪枝偏差。同时，使用网络剪枝的层平衡策略（LBP

layer balance policy），确保了深度模型层之间的参数量平衡，在稀疏度很大的情况下有效地推迟了深度模型坍塌。最后，基于两种基准数据集讨论了CEMP-FL在无线场景中的性能，实验表明，提出的CEMP-FL在保证性能的前提下取得了最优的通信成本压缩比，实现了联邦学习分布式训练架构下的高效通信。

Abstract

In the mobile edge computing scenario

the distributed architecture of federated learning allows the edge server and mobile terminals to cooperatively train the deep model

without necessitating sharing of local data across the mobile terminals. While the training process generally consists of multiple rounds between the server and several clients

which can incur high communication costs and training overhead. To address this issue

a communication-efficient model pruning for federated learning (CEMP-FL) framework

which employed the single-shot layer balance network pruning (SBNP) algorithm

combined with unstructured sparse weight compression

was proposed to significantly reduce the size of the global model

and to effectively alleviate the biased pruning due to training samples discrepancy between clients. Meanwhile

layer balance policy (LBP) was adopted to ensure a balance of the model parameters between layers

which could effectively circumvent the problem of layer-collapse in the case of high sparsity. Finally

the performance of CEMP-FL in wireless scenarios was discussed on two benchmark datasets. The experimental results show that the proposed CEMP-FL method achieves the highest compression ratio of communication costs while maintaining performance

and provides efficient communication in the distributed architecture of federated learning.

关键词

Keywords

references

MCMAHAN B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Proceedings of the Artificial Intelligence and Statistics . PMLR , 2017 : 1273 - 1282 .

YANG Q , LIU Y , CHEN T J , et al . Federated machine learning: concept and applications [J ] . ACM Transactions on Intelligent Systems and Technology , 10 ( 2 ): 1 - 19 .

DUAN M M , LIU D , CHEN X Z , et al . Self-balancing federated learning with global imbalanced data in mobile systems [J ] . IEEE Transactions on Parallel and Distributed Systems , 2021 , 32 ( 1 ): 59 - 71 .

KONEČNÝ J , MCMAHAN H B , YU F X , et al . Federated learning: strategies for improving communication efficiency [J ] . arXiv preprint , 2016 , arXiv: 1610.05492 .

HAMER J , MOHRI M , SURESH A T . Fedboost: a communication-efficient algorithm for federated learning [C ] // International Conference on Machine Learning . PMLR , 2020 : 3973 - 3983 .

LEE N , AJANTHAN T , TORR P H S . SNIP: single-shot network pruning based on connection sensitivity [J ] . arXiv preprint , 2018 , arXiv: 1810.02340 .

ZHU M , GUPTA S . To prune, or not to prune: exploring the efficacy of pruning for model compression [J ] . arXiv preprint , 2017 , arXiv: 1710.01878 .

SINGH S P , JAGGI M . Model fusion via optimal transport [J ] . arXiv preprint , 2019 , arXiv: 1910.05653 .

PALIHAWADANA C , WIRATUNGA N , WIJEKOON A , et al . FedSim: similarity guided model aggregation for Federated Learning [J ] . Neurocomputing , 2022 , 483 ( C ): 432 - 445 .

LI T , HU S , BEIRAMI A , et al . Ditto: fair and robust federated learning through personalization [C ] // International Conference on Machine Learning . PMLR , 2021 : 6357 - 6368 .

MARFOQ O , NEGLIA G , BELLET A , et al . Federated multi-task learning under a mixture of distributions [J ] . Advances in Neural Information Processing Systems , 2021 , 34 : 15434 - 15447 .

XIE C , HUANG K , CHEN P Y , et al . DBA: distributed backdoor attacks against federated learning [C ] // International Conference on Learning Representations . 2020 .

YIN H X , MALLYA A , VAHDAT A , et al . See through gradients: image batch recovery via GradInversion [C ] // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2021 : 16332 - 16341 .

LI Z H , ZHANG J X , LIU L Y , et al . Auditing privacy defenses in federated learning via generative gradient leakage [C ] // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2022 : 10122 - 10132 .

FRABONI Y , VIDAL R , KAMENI L , et al . Clustered sampling: Low-variance and improved representativity for clients selection in federated learning [C ] // International Conference on Machine Learning . PMLR , 2021 : 3407 - 3416 .

BALAKRISHNAN R , LI T , ZHOU T , et al . Diverse client selection for federated learning via submodular maximization [C ] // International Conference on Learning Representations . 2022 .

TANAKA H , KUNIN D , YAMINS D L K , et al . Pruning neural networks without any data by iteratively conserving synaptic flow [C ] // Proceedings of the Proceedings of the 34th International Conference on Neural Information Processing Systems . New York : ACM , 2020 : 6377 - 6389 .

RAMANUJAN V , WORTSMAN M , KEMBHAVI A , et al . What’s hidden in a randomly weighted neural network? [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 11890 - 11899 .

PATIL S M , DOVROLIS C . PHEW: constructing sparse networks that learn fast and generalize well without training data [C ] // International Conference on Machine Learning . PMLR , 2021 : 8432 - 8442 .

LIN J , LUO X T , HONG M , et al . Memory-friendly scalable super-resolution via rewinding lottery ticket hypothesis [C ] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2023 : 14398 - 14407 .

SRINIVAS S , SUBRAMANYA A , BABU R V . Training sparse neural networks [C ] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE Press , 2017 : 455 - 462 .

FRANKLE J , DZIUGAITE G K , ROY D M , et al . Linear mode connectivity and the lottery ticket hypothesis [C ] // Proceedings of the Proceedings of the 37th International Conference on Machine Learning . New York : ACM , 2020 : 3259 – 3269 .

YUAN G Z , SHEN L , ZHENG W S . A block decomposition algorithm for sparse optimization [C ] // Proceedings of the Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . New York : ACM , 2020 : 275 - 285 .

HE Y , DING Y H , LIU P , et al . Learning filter pruning criteria for deep convolutional neural networks acceleration [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 2006 - 2015 .

HU R , GONG Y M , GUO Y X . Federated learning with sparsification-amplified privacy and adaptive optimization [C ] // Proceedings of the Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 1463 - 1469 .

STRIPELIS D , GUPTA U , STEEG G V , et al . Federated progressive sparsification (purge, merge, tune) [J ] . arXiv preprint , 2022 , arXiv: 2204.12430 .

QIU X C , FERNANDEZ-MARQUES J , GUSMAO P P , et al . ZeroFL: efficient on-device training for federated learning with local sparsity [EB/OL ] . 2022 : arXiv : 2208 . 02507 . http://arxiv.org/abs/2208.02507 http://arxiv.org/abs/2208.02507

PASE F , ISIK B , GUNDUZ D , et al . Efficient federated random subnetwork training [C ] // Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022). 2022 .

CHEN X Z , ZHU J Y , JIANG J B , et al . Tight compression: compressing CNN model tightly through unstructured pruning and simulated annealing based permutation [C ] // Proceedings of the 2020 57th ACM/IEEE Design Automation Conference (DAC) . Piscataway : IEEE Press , 2020 : 1 - 6 .

SONG HAN , HUIZI MAO , AND WILLIAM J DALLY . Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding . In International Conference on Learning Representations , 2016

VAHIDIAN S , MORAFAH M , LIN B . Personalized federated learning by structured and unstructured pruning under data heterogeneity [C ] // Proceedings of the 2021 IEEE 41st International Conference on Distributed Computing Systems Workshops (ICDCSW) . Piscataway : IEEE Press , 2021 : 27 - 34 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

车辆算力网络中异步鲁棒联邦学习方法研究

基于选举策略的低空物联网稳定联邦学习方法

联邦学习赋能6G网络综述

基于背包模型的联邦学习客户端选择方法

一种基于带宽分配的联邦学习激励机制