Gesture reconstruction and recognition based on ultrasonic sensing

Tian Yueyue; Huang jiadong; Yin Huarui; Chen Li

doi:null

您当前的位置：

首页 >

文章列表页 >

Gesture reconstruction and recognition based on ultrasonic sensing

更新时间：2026-04-10

- Gesture reconstruction and recognition based on ultrasonic sensing
- Chinese Journal on Internet of Things (2026)
- 作者机构：
  
  中国科学技术大学，安徽合肥 230022
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China Youth Fund (Category B)(62522126);Anhui Provincial Outstanding Youth Fund(2308085J24)
- DOI：
  CLC： TN92
- Received：16 December 2025，
  
  Revised：2026-01-14，
  
  Accepted：23 March 2026，
- 稿件说明：
移动端阅览
Tian Yueyue, Huang jiadong, Yin Huarui, et al. Gesture reconstruction and recognition based on ultrasonic sensing[J/OL]. Chinese Journal on Internet of Things, 2026.
DOI：

Tian Yueyue, Huang jiadong, Yin Huarui, et al. Gesture reconstruction and recognition based on ultrasonic sensing[J/OL]. Chinese Journal on Internet of Things, 2026. DOI：

摘要

为解决基于视觉的手势感知中存在的隐私泄露风险以及其他非视觉信号感知成本高的问题，本文研究基于超声波信号的手势重建和识别。对采集的超声回波和手势图片数据处理构造出一个超声-手势框架数据集Ultrasonic Gesture。基于该数据集，提出了一种具有高性能局部感知与全局建模能力的CAMT-Net神经网络，实现从超声波信号到二维手势关键点坐标的高精度端到端映射。在包含六种静态手势的数据集上进行实验，所提方法生成的关键点精度接近基于RGB图像的重建方法；进一步基于重建关键点进行手势识别，准确率达到89%。结果表明，超声波信号可有效支持细粒度手势感知任务。

Abstract

To address the privacy leakage risks in vision-based gesture sensing and the high sensing costs of other non-visual signals

this paper investigated a gesture reconstruction and recognition method based on ultrasonic signals. The collected ultrasonic echo and gesture image data were processed to construct the Ultrasonic Gesture dataset. Based on this dataset

we proposed a CAMT-Net with high-performance local perception and global modeling capabilities

achieving high-precision end-to-end mapping from ultrasonic signals to 2D gesture keypoint coordinates. Experiments were conducted on the dataset containing six static gestures. The proposed method achieved accuracy close to the metods based on RGB images. Further gesture recognition based on the reconstructed keypoints reached an accuracy of 89%. The results indicate that ultrasonic signals can effectively support fine-grained gesture perception tasks.

关键词

Keywords

references

GAVGIOTAKI D , NTOA S , MARGETIS G , et al . Gesture-based Interaction for AR Systems: A Short Review [C ] // Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments . New York : ACM Press , 2023 : 284 - 292 .

GAO Q , LIU J G , JU Z J . Robust real-time hand detection and localization for space human–robot interaction based on deep learning [J ] . Neurocomputing , 2020 , 390 : 198 – 206 .

WANG R Y , POPOVIĆ J . Real-time hand-tracking with a color glove [J ] . ACM Transactions on Graphics , 2009 , 28 ( 3 ): 1 - 8 .

HU F , HE P , XU S L , et al . FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist [J ] . Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 2020 , 4 ( 2 ): 1 - 24 .

ZHANG F , ZHAO L , LI S , et al . 3D hand pose and shape estimation from monocular RGB via efficient 2D cues [J ] . Computational Visual Media , 2024 , 10 ( 1 ): 79 – 96 .

MUELLER F , BERNARD F , SOTNYCHENKO O , et al . GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2018 : 49 – 59 .

SMITH B , WU C , WEN H , et al . Constraining dense hand surface tracking with elasticity [J ] . ACM Transactions on Graphics , 2020 , 39 ( 6 ): 1 - 14 .

LIEN J , GILLIAN N , KARAGOZLER M E , et al . Soli: ubiquitous gesture sensing with millimeter wave radar [J ] . ACM Transactions on Graphics , 2016 , 35 ( 4 ): 1 - 19 .

GLANDON A , VIDYARATNE L , SADEGHZADEHYAZDI N , et al . 3D Skeleton Estimation and Human Identity Recognition Using Lidar Full Motion Video [C ] // 2019 International Joint Conference on Neural Networks (IJCNN) . Piscataway : IEEE Press , 2019 : 1 – 8 .

MAO W G , HE J , ZHENG H H , et al . High-precision acoustic motion tracking: demo [C ] // Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking . Piscataway : IEEE Press , 2016 : 491 - 492 .

SHARP T , KESKIN C , ROBERTSON D , et al . Accurate, Robust, and Flexible Real-time Hand Tracking [C ] // Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems . New York : ACM Press , 2015 : 3633 - 3642 .

SRIDHAR S , MUELLER F , OULASVIRTA A , et al . Fast and robust hand tracking using detection-guided optimization [C ] // 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2015 : 3213 - 3221 .

SRIDHAR S , OULASVIRTA A , THEOBALT C , et al . Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data [C ] // 2013 IEEE International Conference on Computer Vision . Piscataway : IEEE Press , 2013 : 2456 – 2463 .

KESKIN C , KIRAÇ F , KARA Y E , et al . Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests [C ] // Computer Vision – ECCV 2012 . Piscataway : IEEE Press , 2012 : 852 – 863 .

TANG D , CHANG H J , TEJANI A , et al . Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2014 : 3786 – 3793 .

TOMPSON J , STEIN M , LECUN Y , et al . Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks [J ] . ACM Transactions on Graphics , 2014 , 33 ( 5 ): 1 - 10 .

SIMON T , JOO H , MATTHEWS I , et al . Hand Keypoint Detection in Single Images Using Multiview Bootstrapping [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2017 : 4645 – 4653 .

WANG Y G , PENG C , LIU Y . Mask-Pose Cascaded CNN for 2D Hand Pose Estimation From Single Color Image [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2019 , 29 ( 11 ): 3258 – 3268 .

CHEN X , LIU Y , DONG Y , et al . MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2022 : 20512 – 20522 .

CHEN X , LIU Y , MA C , et al . Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2021 : 13269 – 13278 .

TANG X , WANG T , FU C W . Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE Press , 2021 : 11678 – 11687 .

KONG H , LYU H , YU J , et al . mmHand: 3D Hand Pose Estimation Leveraging mmWave Signals [C ] // 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS) . Piscataway : IEEE Press , 2024 : 1062 – 1073 .

JI S J , ZHANG X Y , ZHENG Y Q , et al . Construct 3D Hand Skeleton with Commercial WiFi [C ] // Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems . New York : ACM Press , 2024 : 322 - 334 .

LI D , LIU J L , LEE S I , et al . FM-track: pushing the limits of contactless multi-target tracking using acoustic signals [C ] // Proceedings of the 18th Conference on Embedded Networked Sensor Systems . New York : ACM Press , 2020 : 150 - 163 .

MAO W G , WANG M , SUN W , et al . RNN-Based Room Scale Hand Motion Tracking [C ] // The 25th Annual International Conference on Mobile Computing and Networking . New York : ACM Press , 2019 : 1 - 16 .

WANG W , LIU A X , SUN K . Device-free gesture tracking using acoustic signals [C ] // Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking . New York : ACM Press , 2016 : 82 - 94 .

YANG Y J , CHEN T , AN Z L , et al . LeakyFeeder: In-Air Gesture Control Through Leaky Acoustic Waves [C ] // Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems . New York : ACM Press , 2025 : 144 – 157 .

WANG S Y , PU H L , CAO Q M , et al . RAM-Hand: Robust Acoustic Multi-Hand Pose Reconstruction Using a Microphone Array [C ] // Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems . New York : ACM Press , 2025 : 130 – 143 .

FENG Z H , KITTLER J , AWAIS M , et al . Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2018 : 2235 – 2245 .

YANG Y , RAMANAN D . Articulated Human Detection with Flexible Mixtures of Parts [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2013 , 35 ( 12 ): 2878 – 2890 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Feature fusion method for UWB angle of arrival estimation under non-line-of-sight conditions

Lightweight parallel denoising network-based OTFS channel estimation algorithm

A 3D printing defect detection algorithm incorporating attention mechanism

Review of the application of deep learning in the auxiliary diagnosis of ophthalmic diseases

Model pruning techniques in the Internet of things: state of the art, methods and perspectives

Related Author

ZENG Wenmin

ZHENG Bofeng

LIU Jinting

BAO Yachuan

YIN Jikai

LI Jianjia

ZHANG Tingting

XIONG Jun

Related Institution

The 54th Research Institute of China Electronics Technology Group Corporation

State Key Laboratory of Comprehensive PNT Network and Equipment Technology, The 54th Research Institute of China Electronics Technology Group Corporation

IOT Thrust, The Hong Kong University of Science and Technology

Guangdong Provincial Key Laboratory of Space-Aerial Networking and Intelligent Sensing, Harbin Institute of Technology

School of Internet of Things, Nanjing University of Posts and Telecommunications

⁰