Reinforcement learning-based real-time video streaming control and on-device training research

Huanhuan ZHANG; Anfu ZHOU; Huadong MA

doi:10.11959/j.issn.2096-3750.2022.00306

您当前的位置：

首页 >

文章列表页 >

Reinforcement learning-based real-time video streaming control and on-device training research

Theory and Technology | 更新时间：2024-08-16

- Reinforcement learning-based real-time video streaming control and on-device training research
- Chinese Journal on Internet of Things Vol. 6, Issue 4, Pages: 1-13(2022)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-3750.2022.00306
  CLC： TP393
- Online First：2022-12，
  
  Published：30 December 2022
- 稿件说明：
移动端阅览
Huanhuan ZHANG, Anfu ZHOU, Huadong MA. Reinforcement learning-based real-time video streaming control and on-device training research[J]. Chinese Journal on Internet of Things, 2022, 6(4): 1-13.
DOI：

Huanhuan ZHANG, Anfu ZHOU, Huadong MA. Reinforcement learning-based real-time video streaming control and on-device training research[J]. Chinese Journal on Internet of Things, 2022, 6(4): 1-13. DOI： 10.11959/j.issn.2096-3750.2022.00306.

摘要

以物联网、移动互联网为核心的服务平台加速发展，数以亿计的终端用户通过实时视频进行通信，实时视频已成为人们数字化生活中不可替代的核心工具。然而，互联网络呈现高动态、强异构的特性，对实时视频的流控技术提出了严格要求，用户体验质量仍然不佳。设计了适用于异构网络环境的强化学习驱动的自适应流控算法、研发了移动终端训练技术以降低服务端开销，并对算法的设计及结构进行了深入的评测研究。实验表明，所设计的自适应流控算法可以有效地预测网络带宽，相较于国际代表性的流控算法，将预测带宽误差降低了48.48%。有效的带宽预测进一步提升了视频用户体验质量，如视频流畅度提升了 60.65%、视频清晰度提升了 16.52%。此外，测评分析可为实时视频流优化方案提供经验性指导，有力推动智能视频应用的发展。

Abstract

Service platforms centered on the Internet of things and mobile Internet are in accelerating process.Hundreds of millions of end-users communicate through network real-time video services

which have become an irreplaceable core tool in human’s digital life.However

the Internet is becoming dynamic

and heterogeneous

which imposes stringent requirements on real-time video streaming control technology.Moreover

the QoE of real-time video is not satisfactory.An adaptive reinforcement learning-based video intelligent transmission algorithm was designed

which can deal with heterogeneous network environment.And then

an effective end-to-end on-device training framework was designed to decrease server overhead

and a detailed evaluation and analysis on the neural network design and structure was provided.Experimental results show that the proposed algorithm can effectively predict heterogeneous network bandwidth

and reduces the bandwidth prediction error by 48.48%

comparing with the representative streaming control algorithm.The effective bandwidth prediction can further improve the user QoE

such as improving the video fluency by 60.65%

and improving the video quality by 16.52%.Besides

the analysis can provide empirical insights for further study

and holds potential to push the development of intelligent video applications.

关键词

Keywords

references

LUO J G , ZHANG M , ZHAO L , et al . A large-scale live video streaming system based on P2P networks [J ] . Journal of Software , 2006 , 18 ( 2 ): 391 - 399 .

FENG D G , XU J , LAN X . Study on 5G mobile communication network security [J ] . Journal of Software , 2018 , 29 ( 6 ): 1813 - 1825 .

Cisco visual networking index:forecast and trends [EB ] . 2019 .

HA S , RHEE I , XU L . CUBIC:a new TCP-friendly high-speed TCP variant [J ] . Operating Systems Review , 2008 , 42 ( 5 ): 64 - 74 .

CARLUCCI G , DE CICCOL , HOLMER S , et al . Congestion control for web real-time communication [J ] . IEEE/ACM Transactions on Networking , 2017 , 25 ( 5 ): 2629 - 2642 .

NEAL C , YUCHUNG C , STEPHEN G , et al . BBR:congestion-based congestion control [J ] . Communications of the ACM , 2017 , 60 ( 2 ): 58 - 66 .

MAO H , NETRAVALI R , ALIZADEH M . Neural adaptive video streaming with pensieve [C ] // ACM Special Interest Group on Data Communication (SIGCOMM) 2017 . Los Angeles:ACM Press , 2017 : 197 - 210 .

ZHOU A F , ZHANG H H , SU G Y , et al . Learning to coordinate video codec with transport protocol for mobile video telephony [C ] // Proceedings of the 25th Annual International Conference on Mobile Computing and Networking (MobiCom) 2019 . Los Cabos :[s.n ] , 2019 : 21 - 25 .

ZHANG H H , ZHOU A F , LU J M , et al . OnRL:improving mobile video telephony via online reinforcement learning [C ] // Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (MobiCom) 2020 . London :[s.n ] , 2020 : 1 - 14 .

YAN F Y , HUDSON A , ZHU C Z , et al . Learning in situ:a randomized experiment in video streaming [C ] // Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI) . Santa Clara :[s.n ] , 2020 : 495 - 511 .

JACOBSON V , . Congestion avoidance and control [C ] // Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM) . Stanford :[s.n ] , 1988 : 314 - 329 .

BRAAKMO L S , O’MALLEY S W , PETERSON L L . TCP vegas:new techniques for congestion detection and avoidance [C ] // Proceed ings of the ACM Special Interest Group on Data Communication (SIGCOMM) . London :[s.n ] , 1994 : 24 - 35 .

DONG M , LI Q , ZARCHY D , et al . PCC:re-architecting congestion control for consistent high performance [C ] // Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI) . Oakland :[s.n ] , 2015 : 395 - 408 .

DONG M , MENG T , ZARCHY D , et al . PCC vivace:online-learning congestion control [C ] // Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI) . Renton :[s.n ] , 2018 : 343 - 356 .

XU Q , MEHROTRA S , MAO Z M , et al . PROTEUS:network performance forecast for real-time,interactive mobile applications [C ] // Proceedings of the 11th Annual International Conference on Mobile Systems,Applications,and Services (MobiSys) . Taipei :[s.n ] , 2013 : 347 - 360 .

Web RTC homepage [EB ] . 2018 .

FOULADI S , EMMONS J , ORBAY E , et al . Salsify:low-latency network video through tighter integration between a video codec and a transport protocol [C ] // Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation,(NSDI) , Renton :[s.n ] , 2018 : 267 - 282 .

WINSTEIN K , BALAKRISHNAN H . TCP ex-machina:computer-generated congestion control [C ] // Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM) . Hong Kong :[s.n ] , 2013 : 123 - 134 .

FRANCIS YY , MA J , HILL G D , et al . Pantheon:the training ground for Internet congestion-control research [C ] // Proceedings of the 2018 USENIX Annual Technical Conference (USENIX ATC) . Boston :[s.n ] , 2018 : 731 - 743 .

HUANG T Y , JOHARI R , MCKEOWN N , et al . A buffer-based approach to rate adaptation:evidence from a large video streaming service [C ] // Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM) .[S.l.:s.n ] , 2014 : 187 - 198 .

SPITERI K , URGAONKAR R , SIATRAMAN R K . BOLA:near-optimal bit rate adaptation for online videos [J ] . IEEE/ACM Transactions on Networking , 2020 , 28 ( 4 ): 1698 - 1711 .

JIANG J C , SEKAR V , ZHANG H . Improving fairness,efficiency,and stability in HTTP-based adaptive video streaming with festive [C ] // Proceedings of IEEE/ACM Transactions on Networking . Piscataway:IEEE Press , 2012 : 326 - 340 .

ZHANG H , ZHOU A , MA H . Improving mobile interactive video QoE via two-level online cooperative learning [J ] . IEEE Transactions on Mobile Computing , 2022 ,Early Access.

刘克 . 实用马尔可夫决策过程 [M ] . 清华大学出版社 , 2004 .

LIU K . Applied Markov decision processes [M ] . Beijing : Tsinghua University Press , 2004 .

范长杰 . 基于马尔可夫决策理论的规划问题的研究 [D ] . 中国科学技术大学 , 2008 .

FAN C J . Research on planning problem based on Markov decision theory [D ] . Hefei:University of Science and Technology of China , 2008 .

ZHANG H , ZHOU A , MA R , et al . Arsenal:understanding learning-based wireless video transport via in-depth evaluation [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 10 ): 10832 - 10844 .

SUN Y , YIN X , JIANG J , et al . CS2P:improving video bitrate selection and adaptation with data-driven throughput prediction [C ] // Pro ceedings of the ACM Special Interest Group on Data Communication (SIGCOMM) . New York:ACM Press , 2016 : 272 - 285 .

HU Y X , LI D , SUN P H , et al . Polymorphic smart network:an open,flexible and universal architecture for future heterogeneous networks [J ] . IEEE Transactions on Network Science and Engineering , 2020 , 7 ( 4 ): 2515 - 2525 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [EB ] . 2017 .

Dillon J V , LANGMORE I , Tran D , et al . Tensorflow distributions [J ] . arXiv preprint . 2017 ,arXiv:1711.10604.

SMILKOV D , THORAT N , ASSOGBA Y , et al . Tensorflow.js:machine learning for the web and beyond [J ] . Proceedings of Machine Learning and Systems , 2019 : 309 - 321 .

Views

594

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on low-latency transmission migration method for federated learning models in the Internet of vehicles

Research on the new paradigm for optimizing 5G base station energy-saving control strategies utilizing the Decision Transformer model

RL based transmission power control for energy harvesting sensor

Collaborative altitude-adaptive reinforcement learning for active search with unmanned aerial vehicle swarms

Related Author

Zhang Yin

Zhang Ke

Jiang Chi

Yin Hongbo

Wang Shuai

HE Yi

XIA Pengcheng

ZHU Yintao

Related Institution

Guangdong Intelligent Robot Research Institute

School of Information and Communication Engineering, University of Electronic Science and Technology of China

Shenzhen Institute for Advanced Study, UESTC

School of Computer Engineering, Jiangsu Second Normal University

School of Cyberspace and Security, Nanjing University of Science and Technology, Jiangyin

⁰