基于强化学习的能量采集传感器传输功率控制方法

刘南君; 贾日恒; 许文韬; 李明禄

doi:10.11959/j.issn.2096-3750.2025.00445

您当前的位置：

首页 >

文章列表页 >

基于强化学习的能量采集传感器传输功率控制方法

理论与技术 | 更新时间：2025-04-21

- 基于强化学习的能量采集传感器传输功率控制方法
- RL based transmission power control for energy harvesting sensor
- 物联网学报 2025年9卷第1期页码：115-127
- 作者机构：
  
  浙江师范大学计算机科学与技术学院，浙江金华 321004
- 作者简介：
  
  [ "刘南君（2000‒ ），男，浙江师范大学计算机科学与技术学院硕士生，主要研究方向为无线可充电网络、智能物联网等。" ]
  [ "贾日恒（1989‒ ），男，博士，浙江师范大学计算机科学与技术学院副教授，主要研究方向为物联网、无线可充电传感器网络、无人机网络、强化学习等。" ]
  [ "许文韬（1998‒ ），男，浙江师范大学计算机科学与技术学院硕士生，主要研究方向为无线可充电网络、智能物联网等。" ]
  [ "李明禄（1965‒ ），男，博士，浙江师范大学计算机科学与技术学院教授，主要研究方向为物联网、无线传感器网络、并行计算等。" ]
- 基金信息：
  
  国家自然科学基金资助项目(62272417)
- DOI：10.11959/j.issn.2096-3750.2025.00445
  中图分类号： TP393;
- 收稿日期：2024-09-19，
  
  修回日期：2024-11-29，
  
  纸质出版日期：2025-03-30
- 稿件说明：
移动端阅览
刘南君,贾日恒,许文韬等.基于强化学习的能量采集传感器传输功率控制方法[J].物联网学报,2025,09(01):115-127.

LIU Nanjun,JIA Riheng,XU Wentao,et al.RL based transmission power control for energy harvesting sensor[J].Chinese Journal on Internet of Things,2025,09(01):115-127.
刘南君,贾日恒,许文韬等.基于强化学习的能量采集传感器传输功率控制方法[J].物联网学报,2025,09(01):115-127. DOI： 10.11959/j.issn.2096-3750.2025.00445.

LIU Nanjun,JIA Riheng,XU Wentao,et al.RL based transmission power control for energy harvesting sensor[J].Chinese Journal on Internet of Things,2025,09(01):115-127. DOI： 10.11959/j.issn.2096-3750.2025.00445.

摘要

传感器可以通过能量收集技术从周围环境中采集能量，但自然环境中的能源供给通常具有不稳定性。为实现有效的功率控制，使传感器长期运行的同时提升数据吞吐量等性能指标，设计了基于强化学习的功率控制策略。考虑一个端到端通信系统，发送节点采集能量存储到电池中以用于数据传输，同时持续缓存待发送数据。实际应用中，通常无法完整地预知能量和数据到达的过程。该研究中发送节点仅能获取已收集能量、电池电量、已采集数据、数据缓存量、信道增益等当前状态信息，并基于此进行决策。采用了柔性演员-评论家（SAC

soft actor-critic）算法控制传输功率，并设计了合适的奖励函数和动作剪裁方法。仿真实验结果表明，该算法在性能上优于基线策略，并在部分场景中接近理论最优解。

Abstract

Sensors can harvest energy from the surrounding environment

but the energy supply is always unstable. To achieve effective power control of sensors and enhance their performance metrics

such as data throughput

while ensuring long-term life

a reinforcement learning-based power control strategy was designed. Assume an end-to-end communication system

the sender harvests energy

stores it in a battery for data transmission

and continuously buffers data. In practical scenarios

the arrival of energy and data is random and unpredictable. In this study

the current state was only observed via the sender

which included harvested energy

battery level

collected data

data cache level

and channel gain. Decisions were made solely based on these limited observations. The soft actor-critic (SAC) algorithm was used to control transmission power

with an appropriate reward function and action clipping method. Experimental results demonstrate that the proposed algorithm outperformes baseline strategies and approaches the theoretical optimal in certain scenarios.

关键词

Keywords

references

CLERCKX B , ZHANG R , SCHOBER R , et al . Fundamentals of wireless information and power transfer: from RF energy harvester models to signal and system designs [J ] . IEEE Journal on Selected Areas in Communications , 2019 , 37 ( 1 ): 4 - 33 .

WEI Z Q , YU X H , NG D W K , et al . Resource allocation for simultaneous wireless information and power transfer systems: a tutorial overview [J ] . Proceedings of the IEEE , 2022 , 110 ( 1 ): 127 - 149 .

LUONG N C , HOANG D T , GONG S M , et al . Applications of deep reinforcement learning in communications and networking: a survey [J ] . IEEE Communications Surveys & Tutorials , 2019 , 21 ( 4 ): 3133 - 3174 .

CHEN Y , ZHAO F J , LU Y G , et al . Dynamic task offloading for mobile edge computing with hybrid energy supply [J ] . Tsinghua Science and Technology , 2023 , 28 ( 3 ): 421 - 432 .

XIAO H , QI N J , YIN Y J , et al . Investigation of self-powered IoT sensor nodes for harvesting hybrid indoor ambient light and heat energy [J ] . Sensors , 2023 , 23 ( 8 ): 3796 .

KIM W G , KIM D , LEE H M , et al . Wearable fabric-based hybrid energy harvester from body motion and body heat [J ] . Nano Energy , 2022 , 100 : 107485 .

BAKYTBEKOV A , NGUYEN T Q , ZHANG G , et al . Synergistic multi-source ambient RF and thermal energy harvester for green IoT applications [J ] . Energy Reports , 2023 , 9 : 1875 - 1885 .

BAI S M , CUI J , ZHENG Y Q , et al . Electromagnetic-triboelectric energy harvester based on vibration-to-rotation conversion for human motion energy exploitation [J ] . Applied Energy , 2023 , 329 : 120292 .

MASADEH A , WANG Z D , KAMAL A E . Reinforcement learning exploration algorithms for energy harvesting communications systems [C ] // Proceedings of the 2018 IEEE International Conference on Communications (ICC) . Piscataway : IEEE Press , 2018 : 1 - 6 .

BLASCO P , GUNDUZ D , DOHLER M . A learning theoretic approach to energy harvesting communication system optimization [J ] . IEEE Transactions on Wireless Communications , 2013 , 12 ( 4 ): 1872 - 1882 .

QIU C R , HU Y , CHEN Y , et al . Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications [J ] . IEEE Internet of Things Journal , 2019 , 6 ( 5 ): 8577 - 8588 .

DHILLON S , MADHU C , KAUR D , et al . A solar energy forecast model using neural networks: application for prediction of power for wireless sensor networks in precision agriculture [J ] . Wireless Personal Communications , 2020 , 112 ( 4 ): 2741 - 2760 .

WANG Z , AGGARWAL V , WANG X D . Power allocation for energy harvesting transmitter with causal information [J ] . IEEE Transactions on Communications , 2014 , 62 ( 11 ): 4080 - 4093 .

LIANG H , ZHAO X H , ZHANG W . Optimal power allocations for multichannel energy harvesting cognitive radio [C ] // Proceedings of the 2017 IEEE 18th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM) . Piscataway : IEEE Press , 2017 : 1 - 6 .

HAKAMI V , DEHGHAN M . Distributed power control for delay optimization in energy harvesting cooperative relay networks [J ] . IEEE Transactions on Vehicular Technology , 2017 , 66 ( 6 ): 4742 - 4755 .

JIA R H , ZHANG J B , LIU X Y , et al . Optimal rate control for energy-harvesting systems with random data and energy arrivals [J ] . ACM Transactions on Sensor Networks , 2019 , 15 ( 1 ): 1 - 30 .

DIAS G M , NURCHIS M , BELLALTA B . Adapting sampling interval of sensor networks using on-line reinforcement learning [C ] // Proceedings of the 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT) . Piscataway : IEEE Press , 2016 : 460 - 465 .

FRATERNALI F , BALAJI B , AGARWAL Y , et al . Pible: battery-free mote for perpetual indoor BLE applications [C ] // Proceedings of the 5th Conference on Systems for Built Environments . New York : ACM Press , 2018 : 184 - 185 .

KANSAL A , HSU J , ZAHEDI S , et al . Power management in energy harvesting sensor networks [J ] . ACM Transactions on Embedded Computing Systems , 2007 , 6 ( 4 ): 32 -es.

SHRESTHAMALI S , KONDO M , NAKAMURA H . Adaptive power management in solar energy harvesting sensor node using reinforcement learning [J ] . ACM Transactions on Embedded Computing Systems , 2017 , 16 ( 5 s): 1 - 21 .

SAWAGUCHI S , CHRISTMANN J F , MOLNOS A , et al . Multi-agent actor-critic method for joint duty-cycle and transmission power control [C ] // Proceedings of the 2020 Design , Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE Press , 2020 : 1015 - 1018 .

FRATERNALI F , BALAJI B , AGARWAL Y , et al . ACES: automatic configuration of energy harvesting sensors with reinforcement learning [J ] . ACM Transactions on Sensor Networks , 2020 , 16 ( 4 ): 1 - 31 .

ORTIZ A , AL-SHATRI H , LI X , et al . Reinforcement learning for energy harvesting point-to-point communications [C ] // Proceedings of the 2016 IEEE International Conference on Communications (ICC) . Piscataway : IEEE Press , 2016 : 1 - 6 .

AIT AOUDIA F , GAUTIER M , BERDER O . RLMan: an energy manager based on reinforcement learning for energy harvesting wireless sensor networks [J ] . IEEE Transactions on Green Communications and Networking , 2018 , 2 ( 2 ): 408 - 417 .

LI M Y , ZHAO X H , LIANG H , et al . Deep reinforcement learning optimal transmission policy for communication systems with energy harvesting and adaptive MQAM [J ] . IEEE Transactions on Vehicular Technology , 2019 , 68 ( 6 ): 5782 - 5793 .

CHU M , LIAO X W , LI H , et al . Power control in energy harvesting multiple access system with reinforcement learning [J ] . IEEE Internet of Things Journal , 2019 , 6 ( 5 ): 9175 - 9186 .

QIAN L P , FENG A Q , FENG X , et al . Deep RL-based time scheduling and power allocation in EH relay communication networks [C ] // Proceedings of the ICC 2019-2019 IEEE International Conference on Communications (ICC) . Piscataway : IEEE Press , 2019 : 1 - 7 .

ZHANG H D , ZHAN D , ZHANG C J , et al . Deep reinforcement learning-based access control for buffer-aided relaying systems with energy harvesting [J ] . IEEE Access , 2020 , 8 : 145006 - 145017 .

SHARMA M K , ZAPPONE A , DEBBAH M , et al . Multi-agent deep reinforcement learning based power control for large energy harvesting networks [C ] // Proceedings of the 2019 International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOPT) . Piscataway : IEEE Press , 2019 : 1 - 7 .

MUY S , RON D , LEE J R . Energy efficiency optimization for SWIPT-based D2D-underlaid cellular networks using multiagent deep reinforcement learning [J ] . IEEE Systems Journal , 2022 , 16 ( 2 ): 3130 - 3138 .

GREGORI M , GÓMEZ-VILARDEBÒ J . Online learning algorithms for wireless energy harvesting nodes [C ] // Proceedings of the 2016 IEEE International Conference on Communications (ICC) . Piscataway : IEEE Press , 2016 : 1 - 6 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

无人机集群协同主动搜索的强化学习策略研究

基于强化学习的多基站协作接收时隙Aloha网络信道接入机制

面向信息年龄的车辆状态更新网络采样与功率控制

无人机辅助蜂窝网络的模式选择与资源优化

融合射频能量采集的协同节能计算迁移研究