PD-TD3: edge-cloud collaborative computation offloading strategy in highway scenarios

LIU Yi; YANG Qi; LI Guoyan; HE Jun; ZHANG Minghui

doi:10.11959/j.issn.2096-3750.2025.00490

您当前的位置：

首页 >

文章列表页 >

PD-TD3: edge-cloud collaborative computation offloading strategy in highway scenarios

Frontiers and Progress of Intelligent Transportation IoT | 更新时间：2025-09-12

- PD-TD3: edge-cloud collaborative computation offloading strategy in highway scenarios
- Chinese Journal on Internet of Things Vol. 9, Issue 2, Pages: 39-50(2025)
- 作者机构：
  
  天津城建大学计算机与信息工程学院，天津 300384
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61876131);Tianjin Research and Innovation Project for Postgraduate Students(2022SKYZ391)
- DOI：10.11959/j.issn.2096-3750.2025.00490
  CLC： U495;
- Received：28 February 2025，
  
  Revised：2025-03-27，
  
  Published：10 June 2025
- 稿件说明：
移动端阅览
刘毅,杨琪,李国燕等.PD-TD3：高速公路场景下边路协同计算卸载策略[J].物联网学报,2025,09(02):39-50.

LIU Yi,YANG Qi,LI Guoyan,et al.PD-TD3: edge-cloud collaborative computation offloading strategy in highway scenarios[J].Chinese Journal on Internet of Things,2025,09(02):39-50.
刘毅,杨琪,李国燕等.PD-TD3：高速公路场景下边路协同计算卸载策略[J].物联网学报,2025,09(02):39-50. DOI： 10.11959/j.issn.2096-3750.2025.00490.

LIU Yi,YANG Qi,LI Guoyan,et al.PD-TD3: edge-cloud collaborative computation offloading strategy in highway scenarios[J].Chinese Journal on Internet of Things,2025,09(02):39-50. DOI： 10.11959/j.issn.2096-3750.2025.00490.

摘要

针对高速公路场景下，现有卸载模型忽视车辆高速移动导致的网络动态变化引起的高时延和能耗，且算法在降低时延和能耗方面效力不足的问题，提出PD-TD3（twin delayed deep deterministic policy gradient with prioritized double buffer pool experience replay）卸载策略方案。首先，搭建高速公路3层分布式卸载模型；然后，将计算卸载问题转化为马尔可夫最优策略问题求解，以均衡优化时延与能耗建立奖励函数，以最大化奖励函数作为优化目标；最后，改进TD3（twin delayed deep deterministic policy gradient）算法中收敛较慢且不稳定、Q值低估偏差、抽取经验效率低的问题，提出PD-TD3算法求解最优化问题。仿真实验结果表明，与TD3算法相比，PD-TD3算法有效提升了早期算法探索效率，并且有效降低了计算卸载的时延约50%、能耗约70%。

Abstract

In the highway scenarios

existing offloading models often overlook the network dynamics caused by the high-speed movement of vehicles

leading to increased latency and energy consumption

and exhibit insufficient effectiveness in reducing latency and energy consumption. To address these challenges

an offloading strategy utilizing the prioritized double-buffer pool experience replay twin delayed deep deterministic policy gradient (PD-TD3) algorithm was proposed. Initially

a three-layer distributed offloading model tailored for highway environments was developed. Subsequently

the computation offloading problem was formulated as a Markov decision process (MDP)

with the reward function designed to optimize the trade-off between latency and energy consumption

aiming to maximize the reward. To address the limitations of the traditional TD3 algorithm

including slow convergence

Q-value underestimation bias

and inefficient experience sampling

the PD-TD3 algorithm was introduced to solve the optimization problem. Simulation results indicate that

compared with the TD3 algorithm

the PD-TD3 algorithm can effectively improve the efficiency of early algorithm exploration and effectively reduces computation offloading latency by approximately 50% and energy consumption by about 70%.

关键词

Keywords

references

WANG X J , ZHU H L , NING Z L , et al . Blockchain intelligence for Internet of vehicles: challenges and solutions [J ] . IEEE Communications Surveys & Tutorials , 2023 , 25 ( 4 ): 2325 - 2355 .

HEMMATI A , ZAREI M , RAHMANI A M . Big data challenges and opportunities in Internet of vehicles: a systematic review [J ] . International Journal of Pervasive Computing and Communications , 2024 , 20 ( 2 ): 308 - 342 .

李智勇 , 王琦 , 陈一凡 , 等 . 车辆边缘计算环境下任务卸载研究综述 [J ] . 计算机学报 , 2021 , 44 ( 5 ): 963 - 982 .

LI Z Y , WANG Q , CHEN Y F , et al . A survey on task offloading research in vehicular edge computing [J ] . Chinese Journal of Computers , 2021 , 44 ( 5 ): 963 - 982 .

LI Z P , YU H Q , FAN G S , et al . Energy-efficient offloading for DNN-based applications in edge-cloud computing: a hybrid chaotic evolutionary approach [J ] . Journal of Parallel and Distributed Computing , 2024 , 187 : 104850 .

SEN P , PANDIT R , SARDDAR D . A survey on methods and apparatus of offloading in mobile cloud computing [J ] . International Journal of System of Systems Engineering , 2024 , 14 ( 2 ): 212 - 225 .

ABBAS N , ZHANG Y , TAHERKORDI A , et al . Mobile edge computing: a survey [J ] . IEEE Internet of Things Journal , 2018 , 5 ( 1 ): 450 - 465 .

SABELLA D , VAILLANT A , KUURE P , et al . Mobile-edge computing architecture: the role of MEC in the Internet of things [J ] . IEEE Consumer Electronics Magazine , 2016 , 5 ( 4 ): 84 - 91 .

ZHENG D S , CHEN Y Y , WEI L , et al . Dynamic NOMA-based computation offloading in vehicular platoons [J ] . IEEE Transactions on Vehicular Technology , 2023 , 72 ( 10 ): 13000 - 13010 .

XUANWEN , SUN H M . Parking cooperation-based mobile edge computing using task offloading strategy [J ] . Journal of Grid Computing , 2024 , 22 : 8 .

MAO Q C , CHENG J J , ZHOU M C , et al . An edge computing-based autonomous vehicle group cooperation model in a highway scene [J ] . IEEE Transactions on Vehicular Technology , 2024 , 73 ( 7 ): 9682 - 9695 .

李国燕 , 薛翔 , 刘毅 , 等 . 改进TD3的SDN车联网边缘计算卸载策略 [J ] . 计算机集成制造系统 , 2023 , 29 ( 5 ): 1627 - 1634 .

LI G Y , XUE X , LIU Y , et al . Improved TD3 edge computing offloading strategy for software defined networking Internet of vehicles [J ] . Computer Integrated Manufacturing Systems , 2023 , 29 ( 5 ): 1627 - 1634 .

刘婷 , 罗喜良 . 移动边缘计算中的在线任务卸载方法 [J ] . 中国科学院大学学报 , 2022 , 39 ( 2 ): 267 - 274 .

LIU T , LUO X L . Online task offloading in mobile edge computing [J ] . Journal of University of Chinese Academy of Sciences , 2022 , 39 ( 2 ): 267 - 274 .

WEI Z L , ZHAO B K , SU J S . Event-driven computation offloading in IoT with edge computing [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 9 ): 6847 - 6860 .

WU Q , WANG S , GE H , et al . Delay-sensitive task offloading in vehicular fog computing-assisted platoons [J ] . IEEE Transactions on Control Systems Technology , 2023 .

LIAO Z F , PENG J S , XIONG B , et al . Adaptive offloading in mobile-edge computing for ultra-dense cellular networks based on genetic algorithm [J ] . Journal of Cloud Computing , 2021 , 10 : 15 .

杨文杰 , 巨涛 , 杨阳 , 等 . 面向边缘计算的人工鱼群搜索任务调度 [J ] . 电子测量与仪器学报 , 2022 , 36 ( 11 ): 149 - 159 .

YANG W J , JU T , YANG Y , et al . Artificial fish swarm search task scheduling for edge computing [J ] . Journal of Electronic Measurement and Instrumentation , 2022 , 36 ( 11 ): 149 - 159 .

DONG S , XIA Y J , KAMRUZZAMAN J . Quantum particle swarm optimization for task offloading in mobile edge computing [J ] . IEEE Transactions on Industrial Informatics , 2023 , 19 ( 8 ): 9113 - 9122 .

ALQARNI M A , MOUSA M H , HUSSEIN M K . Task offloading using GPU-based particle swarm optimization for high-performance vehicular edge computing [J ] . Journal of King Saud University-Computer and Information Sciences , 2022 , 34 ( 10 ): 10356 - 10364 .

CHEN Z Y , ZHENG H Q , ZHANG J S , et al . Joint computation offloading and deployment optimization in multi-UAV-enabled MEC systems [J ] . Peer-to-Peer Networking and Applications , 2022 , 15 ( 1 ): 194 - 205 .

ZENG F , CHEN Q , MENG L , et al . Volunteer assisted collaborative offloading and resource allocation in vehicular edge computing [J ] . IEEE Transactions on Intelligent Transportation Systems , 2021 , 22 ( 6 ): 3247 - 3257 .

ZHAN Y F , GUO S , LI P , et al . A deep reinforcement learning based offloading game in edge computing [J ] . IEEE Transactions on Computers , 2020 , 69 ( 6 ): 883 - 893 .

DAI F , LIU G Z , MO Q , et al . Task offloading for vehicular edge computing with edge-cloud cooperation [J ] . World Wide Web , 2022 , 25 ( 5 ): 1999 - 2017 .

ZHOU H , JIANG K , LIU X X , et al . Deep reinforcement learning for energy-efficient computation offloading in mobile-edge computing [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 2 ): 1517 - 1530 .

WANG T , LUO X , ZHAO W B . Improving the performance of tasks offloading for Internet of vehicles via deep reinforcement learning methods [J ] . IET Communications , 2022 , 16 ( 10 ): 1230 - 1240 .

GAO H H , WANG X J , WEI W , et al . Com-DDPG: task offloading based on multiagent reinforcement learning for information-communication-enhanced mobile edge computing in the Internet of vehicles [J ] . IEEE Transactions on Vehicular Technology , 2024 , 73 ( 1 ): 348 - 361 .

YAO L , XU X L , BILAL M , et al . Dynamic edge computation offloading for Internet of vehicles with deep reinforcement learning [J ] . IEEE Transactions on Intelligent Transportation Systems , 2023 , 24 ( 11 ): 12991 - 12999 .

LI C X , WANG H , SONG R F . Mobility-aware offloading and resource allocation in NOMA-MEC systems via DC [J ] . IEEE Communications Letters , 2022 , 26 ( 5 ): 1091 - 1095 .

ZHOU T Q , XU M , QIN D , et al . Computing offloading based on TD3 algorithm in cache-assisted vehicular NOMA–MEC networks [J ] . Sensors , 2023 , 23 ( 22 ): 9064 .

WAKGRA F G , YAHYA W , KAR B , et al . Ratio-based offloading optimization for edge and vehicular-fog federated systems: a multi-agent TD3 approach [J ] . IEEE Transactions on Vehicular Technology , 2024 , 73 ( 11 ): 17684 - 17696 .

DONG J D , PAN K , ZHENG C X , et al . A dual-agent approach for coordinated task offloading and resource allocation in MEC [J ] . Journal of Electrical and Computer Engineering , 2023 , 2023 ( 1 ): 6134837 .

SAITO Y , KISHIYAMA Y , BENJEBBOUR A , et al . Non-orthogonal multiple access (NOMA) for cellular future radio access [C ] // Proceedings of the 2013 IEEE 77th Vehicular Technology Conference (VTC Spring) . Piscataway : IEEE Press , 2013 : 1 - 5 .

CHEN Z , WANG X . Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach [J ] . EURASIP Journal on Wireless Communications and Networking , 2020 , 2020( 1 ): 1 - 21 .

LUO L P , LI Q Z , CHENG J L . Performance analysis of overlay cognitive NOMA systems with imperfect successive interference cancellation [J ] . IEEE Transactions on Communications , 2020 , 68 ( 8 ): 4709 - 4722 .

CHOU Y L , LIU S S , CHUNG E Y , et al . An energy and performance efficient DVFS scheme for irregular parallel divide-and-conquer algorithms on the intel SCC [J ] . IEEE Computer Architecture Letters , 2014 , 13 ( 1 ): 13 - 16 .

HE Q , HOU X . Wd3: taming the estimation bias in deep reinforcement learning [C ] // Proceedings of the 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) . Piscataway : IEEE Press , 2020 : 391 - 398 .

QIU Y , ZHOU S Y , XIA D , et al . Local integrated energy system operational optimization considering multi-type uncertainties: a reinforcement learning approach based on improved TD3 algorithm [J ] . IET Renewable Power Generation , 2023 , 17 ( 9 ): 2236 - 2256 .

HE S T , SHEN L Y , WU Q , et al . A certified cubic B-spline interpolation method with tangential direction constraints [J ] . Journal of Systems Science and Complexity , 2024 , 37 ( 3 ): 1271 - 1294 .

ALAM M G R , HASSAN M M , UDDIN M Z , et al . Autonomic computation offloading in mobile edge for IoT applications [J ] . Future Generation Computer Systems , 2019 , 90 : 149 - 157 .

WANG J Y , WANG Y F , CHENG P , et al . DDPG-based joint resource management for latency minimization in NOMA-MEC networks [J ] . IEEE Communications Letters , 2023 , 27 ( 7 ): 1814 - 1818 .

ZHAN M , CHEN J , DU C , et al . Twin delayed multi-agent deep deterministic policy gradient [C ] // Proceedings of the 2021 IEEE International Conference on Progress in Informatics and Computing (PIC) . Piscataway : IEEE Press , 2021 : 48 - 52 .

LI S Y , HU X H , DU Y W . Deep reinforcement learning for computation offloading and resource allocation in unmanned-aerial-vehicle assisted edge computing [J ] . Sensors , 2021 , 21 ( 19 ): 6499 .

Views

179

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Graph reducing method based on deep reinforcement learning

Intent-driven dual-time-scale resource allocation approach for Internet of things

Deep diffusion deterministic policy gradient based performance optimization for Wi-Fi network

Related Author

Sun Yanfei

Dong Zhenjiang

Gao Yu

Liu Yali

Qi Jin

Chen Genxin

Wang Jiansu

Song Hang

Related Institution

School of Computer Science, Nanjing University of Posts and Telecommunications

Nanjing Big Data Security Technology Co., Ltd.

School of Internet of Things, Nanjing University of Posts and Telecommunications

School of Automation, Nanjing University of Posts and Telecommunications

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications

⁰