Multi-modal fusion and attention-enhanced deepfake detection

CHEN Xin; GAO Di; PU Zhiming; ZHU Zhengyu; HAO Fengyu; CAO Yan

doi:null

您当前的位置：

首页 >

文章列表页 >

Multi-modal fusion and attention-enhanced deepfake detection

更新时间：2026-03-03

- Multi-modal fusion and attention-enhanced deepfake detection
- Chinese Journal on Internet of Things (2026)
- 作者机构：
  
  1.郑州大学网络空间安全学院，郑州 450001
  2.咪咕文化科技有限公司云网技术中心，北京 100000
  3.郑州大学电气与信息工程学院，郑州 450001
  4.北京邮电大学网络与交换技术全国重点实验室，北京 100876
  5.北京天融信网络安全技术有限公司，北京 100085
- 作者简介：
- 基金信息：
  
  the National Natural Science Foundation of China(62571495;62571182);State key Laboratory of Networking and Switching Technology (Beijing University of Posts and Telecommunications)(SKLNST-2025-1-07)
- DOI：
  CLC： TN309.2
- Received：19 August 2025，
  
  Revised：2026-02-09，
  
  Accepted：09 February 2026，
- 稿件说明：
移动端阅览
CHEN Xin, GAO Di, PU Zhiming, et al. Multi-modal fusion and attention-enhanced deepfake detection[J/OL]. Chinese Journal on Internet of Things, 2026.
DOI：

CHEN Xin, GAO Di, PU Zhiming, et al. Multi-modal fusion and attention-enhanced deepfake detection[J/OL]. Chinese Journal on Internet of Things, 2026. DOI：

摘要

针对深度伪造检测中高保真伪造内容检测效能不足与跨域泛化能力受限的问题，提出一种基于多模态融合与块注意力分析机制的深度伪造检测方法。该方法的主要创新包括：（1）融合细粒度特征分析与视频语义一致性检测，构建高效的多模态伪造检测框架，实现对隐蔽生成痕迹的精准捕捉；（2）引入块注意力分析模块，有效提升模型在不同数据域和多样化攻击场景下的泛化能力。实验结果表明，所提方法在对抗样本与深度伪造检测任务中的性能均显著优于现有主流开源方法，能够高效识别复杂多样的伪造视频。

Abstract

In view of the problem that current deepfake detection methods exhibit insufficient performance in identifying high-fidelity forgeries and limited cross-domain generalization

a detection approach based on multimodal fusion and block attention enhancement was developed. The method integrated fine-grained feature analysis with video semantic consistency verification to establish an efficient multimodal framework capable of capturing subtle generation artifacts with high precision. A block attention analysis module was incorporated to improve generalization performance across heterogeneous data domains and varied attack scenarios. Experimental evaluations demonstrated that the proposed approach achieved significantly superior performance compared with mainstream open-source methods in both adversarial sample detection and deepfake detection tasks. The approach offers an effective and reliable solution for accurately identifying complex and diverse forged videos.

关键词

Keywords

references

曹娟 , 盛强 , 李国杰 . 智能时代公共安全体系面临的技术挑战——兼序《中国科学院院刊》“人工智能与公共安全”专题 [J ] . 中国科学院院刊 . 2025 , 40 ( 3 ): 399 - 407 .

CAO J , SHENG Q , LI G J . Technical challenges faced by public security system in the intelligent era—Preface to the special issue “Artificial Intelligence and Public Security” of Bulletin of Chinese Academy of Sciences [J ] . Bulletin of Chinese Academy of Sciences . 2025 , 40 ( 3 ): 399 - 407 .

梁瑞刚 , 吕培卓 , 赵月 , 等 . 视听觉深度伪造检测技术研究综述 [J ] . 信息安全学报 . 2020 , 5 ( 2 ): 1 - 17 .

LIANG R G , LÜ P Z , ZHAO Y , et al . A survey on audiovisual deepfake detection technologies [J ] . Journal of Information Security . 2020 , 5 ( 2 ): 1 - 17 .

GÜERA D , DELP E J . Deepfake video detection using recurrent neural networks [C ] // Proceedings of 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) . Piscataway : IEEE Press , 2018 : 1 - 6 .

WU Y , ZHU J . Local attention and long-distance interaction of rPPG for deepfake detection [J ] . Neural Networks . 2024 , 178 : 106456 .

AMIN M A , HU Y , HU J . Analyzing temporal coherence for deepfake video detection [J ] . Electronic Research Archive . 2024 , 32 ( 4 ): 2621 - 2641 .

GU Z , CHEN Y , YAO T , et al . Spatiotemporal inconsistency learning for deepfake video detection [C ] // Proceedings of the 29th ACM International Conference on Multimedia . New York : ACM , 2021 : 3473 - 3481 .

KIM T , CHOI J , JEONG Y , et al . Beyond spatial frequency: Pixel-wise temporal frequency-based deepfake video detection [J ] . arXiv preprint . 2025 .

WU Y , ABD-ALMAGEED W . High-frequency enhancement network for highly compressed deepfake detection [J ] . Expert Systems with Applications . 2024 , 252 : 124156 .

HE Y , LI H , WANG Y , et al . Image copy-move forgery detection via deep cross-scale patch match [C ] // Proceedings of the IEEE International Conference on Multimedia and Expo (ICME) . Piscataway : IEEE Press , 2023 : 2327 - 2332 .

董佳乐 , 邓正杰 , 李喜艳 , 等 . 基于频域和空域多特征融合的深度伪造检测方法 [J ] . 图学学报 . 2025 , 46 ( 1 ): 104 - 113 .

DONG J L , DENG Z J , LI X Y , et al . Deepfake detection method based on multi-feature fusion in frequency and spatial domains [J ] . Journal of Graphics . 2025 , 46 ( 1 ): 104 - 113 .

张文祥 , 王夏黎 , 王欣仪 , 等 . 一种强化伪造区域关注的深度伪造人脸检测方法 [J ] . 图学学报 . 2025 , 46 ( 1 ): 47 - 58 .

ZHANG W X , WANG X L , WANG X Y , et al . A deepfake face detection method with enhanced forged-region attention [J ] . Journal of Graphics . 2025 , 46 ( 1 ): 47 - 58 .

WOO S , PARK J , LEE J Y , et al . CBAM: Convolutional block attention module [C ] // Proceedings of the European Conference on Computer Vision (ECCV) . Cham : Springer , 2018 : 3 - 19 .

WANG H , HE Z , LI Y , et al . Provable domain generalization via invariant-feature subspace recovery [C ] // Proceedings of the 39th International Conference on Machine Learning (ICML) . Long Beach : PMLR , 2022 : 20318 - 20334 .

MUANDET K , BALDUZZI D , SCHÖLKOPF B . Domain generalization via invariant feature representation [C ] // Proceedings of the 30th International Conference on Machine Learning (ICML) . Atlanta : JMLR , 2013 : 10 - 18 .

ARJOVSKY M , BOTTOU L , GULRAJANI I , et al . Invariant risk minimization [C ] // Proceedings of the 36th International Conference on Machine Learning (ICML) . Long Beach : PMLR , 2019 : 214 - 223 .

CHA J , LEE K , PARK S , et al . Domain generalization by mutual-information regularization with pre-trained models [C ] // Proceedings of the European Conference on Computer Vision (ECCV) . Cham : Springer Nature Switzerland , 2022 : 440 - 457 .

WANG J , LAN C , LIU C , et al . Generalizing to unseen domains: A survey on domain generalization [J ] . IEEE Transactions on Knowledge and Data Engineering . 2022 , 35 ( 8 ): 8052 - 8072 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16×16 words: Transformers for image recognition at scale [C ] // Proceedings of the International Conference on Learning Representations (ICLR) . San Diego : ICLR , 2021 .

LIU Z , LUO P , WANG X , et al . Deep learning face attributes in the wild [C ] // Proceedings of the IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2015 : 3730 - 3738 .

RÖSSLER A , COZZOLINO D , VERDOLIVA L , et al . FaceForensics++: Learning to detect manipulated facial images [C ] // Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2019 : 1 - 11 .

CHOLLET F . Xception: Deep learning with depthwise separable convolutions [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2017 : 1251 - 1258 .

AFCHAR D , NOZICK V , YAMAGISHI J , et al . MesoNet: A compact facial video forgery detection network [C ] // Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS) . Piscataway : IEEE Press , 2018 : 1 - 7 .

NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics [C ] // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE Press , 2019 : 2307 - 2311 .

邹书桥 , 陈鹏 , 戴娇 , 等 . 一种增强深度伪造检测对抗鲁棒性的算法 [J ] . 信息安全学报 . 2024 .

ZOU S Q , CHEN P , DAI J , et al . An algorithm for enhancing adversarial robustness of deepfake detection [J ] . Journal of Information Security . 2024 .

CARLINI N , WAGNER D . Towards evaluating the robustness of neural networks [C ] // Proceedings of the IEEE Symposium on Security and Privacy (S&P) . Piscataway : IEEE Press , 2017 : 39 - 57 .

CHEN S , YAO T , CHEN J , et al . FRADE: Forgery-aware audio-distilled multimodal learning for deepfake detection [C ] // Proceedings of the ACM International Conference on Multimedia . New York : ACM , 2024 : 7890 - 7899 .

DOLHANSKY B , HOWES R , PFLAUM B , et al . The Deepfake Detection Challenge (DFDC) dataset [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 576 - 585 .

LI Y , YANG X , SUN P , et al . Celeb-DF: A large-scale challenging dataset for deepfake forensics [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 3204 - 3213 .

JIANG L , LI R , WU W , et al . DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 2886 - 2895 .

QI H , GUO Q , JUEFEI-XU F , et al . Deep Rhythm: Exposing deep fakes with attentional visual heartbeat rhythms [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM , 2020 : 1318 - 1327 .

HALIASSOS A , VOUGIOUKAS K , PETRIDIS S , et al . Lips don’t lie: A generalisable and robust approach to face forgery detection [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2021 : 5039 - 5049 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

An efficient remote sensing image segmentation network with modality expansion capability

Lightweight attention-based SAR ship detector

Fusion of multi-scale and context for small target detection algorithm of unmanned aerial vehicle rescue

An intrusion detection method based on depthwise separable convolution and attention mechanism

Inspection method for cable assembly quality based on AR virtual-real image attention mechanism

Related Author

LIU Shudong

QIN Haijie

ZHU Guoqing

ZHANG Zhongwei

LU Ping

ZHANG Hengjia

ZHU Ying

FENG Daquan

Related Institution

Shenzhen HosingGlobal Electronics Co., Ltd

Computer and Information Engineering College, Tianjin chengjian University

State Key Laboratory of Mobile Network and Mobile Multimedia Technology

Zhongxing Telecom Equipment Corporation

China Academy of Information and Communications Technology

⁰