基于多模态融合与注意力增强的深度伪造检测方法

陈鑫; 高迪; 蒲志明; 朱政宇; 郝凤宇; 曹琰

doi:null

您当前的位置：

首页 >

文章列表页 >

基于多模态融合与注意力增强的深度伪造检测方法

更新时间：2026-03-03

- 基于多模态融合与注意力增强的深度伪造检测方法
- Multi-modal fusion and attention-enhanced deepfake detection
- 物联网学报 2026年
- 作者机构：
  
  1.郑州大学网络空间安全学院，郑州 450001
  2.咪咕文化科技有限公司云网技术中心，北京 100000
  3.郑州大学电气与信息工程学院，郑州 450001
  4.北京邮电大学网络与交换技术全国重点实验室，北京 100876
  5.北京天融信网络安全技术有限公司，北京 100085
- 作者简介：
- 基金信息：
  
  国家自然科学基金项目(62571495;62571182);北京邮电大学网络与交换技术全国重点实验室(SKLNST-2025-1-07)
- DOI：
  中图分类号： TN309.2
- 收稿：2025-08-19，
  
  修回：2026-02-09，
  
  录用：2026-02-09，
- 稿件说明：
移动端阅览
陈鑫, 高迪, 蒲志明, 等. 基于多模态融合与注意力增强的深度伪造检测方法[J/OL]. 物联网学报, 2026.

CHEN Xin, GAO Di, PU Zhiming, et al. Multi-modal fusion and attention-enhanced deepfake detection[J/OL]. Chinese Journal on Internet of Things, 2026.
陈鑫, 高迪, 蒲志明, 等. 基于多模态融合与注意力增强的深度伪造检测方法[J/OL]. 物联网学报, 2026. DOI：

CHEN Xin, GAO Di, PU Zhiming, et al. Multi-modal fusion and attention-enhanced deepfake detection[J/OL]. Chinese Journal on Internet of Things, 2026. DOI：

摘要

针对深度伪造检测中高保真伪造内容检测效能不足与跨域泛化能力受限的问题，提出一种基于多模态融合与块注意力分析机制的深度伪造检测方法。该方法的主要创新包括：（1）融合细粒度特征分析与视频语义一致性检测，构建高效的多模态伪造检测框架，实现对隐蔽生成痕迹的精准捕捉；（2）引入块注意力分析模块，有效提升模型在不同数据域和多样化攻击场景下的泛化能力。实验结果表明，所提方法在对抗样本与深度伪造检测任务中的性能均显著优于现有主流开源方法，能够高效识别复杂多样的伪造视频。

Abstract

In view of the problem that current deepfake detection methods exhibit insufficient performance in identifying high-fidelity forgeries and limited cross-domain generalization

a detection approach based on multimodal fusion and block attention enhancement was developed. The method integrated fine-grained feature analysis with video semantic consistency verification to establish an efficient multimodal framework capable of capturing subtle generation artifacts with high precision. A block attention analysis module was incorporated to improve generalization performance across heterogeneous data domains and varied attack scenarios. Experimental evaluations demonstrated that the proposed approach achieved significantly superior performance compared with mainstream open-source methods in both adversarial sample detection and deepfake detection tasks. The approach offers an effective and reliable solution for accurately identifying complex and diverse forged videos.

关键词

Keywords

references

曹娟 , 盛强 , 李国杰 . 智能时代公共安全体系面临的技术挑战——兼序《中国科学院院刊》“人工智能与公共安全”专题 [J ] . 中国科学院院刊 . 2025 , 40 ( 3 ): 399 - 407 .

CAO J , SHENG Q , LI G J . Technical challenges faced by public security system in the intelligent era—Preface to the special issue “Artificial Intelligence and Public Security” of Bulletin of Chinese Academy of Sciences [J ] . Bulletin of Chinese Academy of Sciences . 2025 , 40 ( 3 ): 399 - 407 .

梁瑞刚 , 吕培卓 , 赵月 , 等 . 视听觉深度伪造检测技术研究综述 [J ] . 信息安全学报 . 2020 , 5 ( 2 ): 1 - 17 .

LIANG R G , LÜ P Z , ZHAO Y , et al . A survey on audiovisual deepfake detection technologies [J ] . Journal of Information Security . 2020 , 5 ( 2 ): 1 - 17 .

GÜERA D , DELP E J . Deepfake video detection using recurrent neural networks [C ] // Proceedings of 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) . Piscataway : IEEE Press , 2018 : 1 - 6 .

WU Y , ZHU J . Local attention and long-distance interaction of rPPG for deepfake detection [J ] . Neural Networks . 2024 , 178 : 106456 .

AMIN M A , HU Y , HU J . Analyzing temporal coherence for deepfake video detection [J ] . Electronic Research Archive . 2024 , 32 ( 4 ): 2621 - 2641 .

GU Z , CHEN Y , YAO T , et al . Spatiotemporal inconsistency learning for deepfake video detection [C ] // Proceedings of the 29th ACM International Conference on Multimedia . New York : ACM , 2021 : 3473 - 3481 .

KIM T , CHOI J , JEONG Y , et al . Beyond spatial frequency: Pixel-wise temporal frequency-based deepfake video detection [J ] . arXiv preprint . 2025 .

WU Y , ABD-ALMAGEED W . High-frequency enhancement network for highly compressed deepfake detection [J ] . Expert Systems with Applications . 2024 , 252 : 124156 .

HE Y , LI H , WANG Y , et al . Image copy-move forgery detection via deep cross-scale patch match [C ] // Proceedings of the IEEE International Conference on Multimedia and Expo (ICME) . Piscataway : IEEE Press , 2023 : 2327 - 2332 .

董佳乐 , 邓正杰 , 李喜艳 , 等 . 基于频域和空域多特征融合的深度伪造检测方法 [J ] . 图学学报 . 2025 , 46 ( 1 ): 104 - 113 .

DONG J L , DENG Z J , LI X Y , et al . Deepfake detection method based on multi-feature fusion in frequency and spatial domains [J ] . Journal of Graphics . 2025 , 46 ( 1 ): 104 - 113 .

张文祥 , 王夏黎 , 王欣仪 , 等 . 一种强化伪造区域关注的深度伪造人脸检测方法 [J ] . 图学学报 . 2025 , 46 ( 1 ): 47 - 58 .

ZHANG W X , WANG X L , WANG X Y , et al . A deepfake face detection method with enhanced forged-region attention [J ] . Journal of Graphics . 2025 , 46 ( 1 ): 47 - 58 .

WOO S , PARK J , LEE J Y , et al . CBAM: Convolutional block attention module [C ] // Proceedings of the European Conference on Computer Vision (ECCV) . Cham : Springer , 2018 : 3 - 19 .

WANG H , HE Z , LI Y , et al . Provable domain generalization via invariant-feature subspace recovery [C ] // Proceedings of the 39th International Conference on Machine Learning (ICML) . Long Beach : PMLR , 2022 : 20318 - 20334 .

MUANDET K , BALDUZZI D , SCHÖLKOPF B . Domain generalization via invariant feature representation [C ] // Proceedings of the 30th International Conference on Machine Learning (ICML) . Atlanta : JMLR , 2013 : 10 - 18 .

ARJOVSKY M , BOTTOU L , GULRAJANI I , et al . Invariant risk minimization [C ] // Proceedings of the 36th International Conference on Machine Learning (ICML) . Long Beach : PMLR , 2019 : 214 - 223 .

CHA J , LEE K , PARK S , et al . Domain generalization by mutual-information regularization with pre-trained models [C ] // Proceedings of the European Conference on Computer Vision (ECCV) . Cham : Springer Nature Switzerland , 2022 : 440 - 457 .

WANG J , LAN C , LIU C , et al . Generalizing to unseen domains: A survey on domain generalization [J ] . IEEE Transactions on Knowledge and Data Engineering . 2022 , 35 ( 8 ): 8052 - 8072 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16×16 words: Transformers for image recognition at scale [C ] // Proceedings of the International Conference on Learning Representations (ICLR) . San Diego : ICLR , 2021 .

LIU Z , LUO P , WANG X , et al . Deep learning face attributes in the wild [C ] // Proceedings of the IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2015 : 3730 - 3738 .

RÖSSLER A , COZZOLINO D , VERDOLIVA L , et al . FaceForensics++: Learning to detect manipulated facial images [C ] // Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2019 : 1 - 11 .

CHOLLET F . Xception: Deep learning with depthwise separable convolutions [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2017 : 1251 - 1258 .

AFCHAR D , NOZICK V , YAMAGISHI J , et al . MesoNet: A compact facial video forgery detection network [C ] // Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS) . Piscataway : IEEE Press , 2018 : 1 - 7 .

NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics [C ] // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE Press , 2019 : 2307 - 2311 .

邹书桥 , 陈鹏 , 戴娇 , 等 . 一种增强深度伪造检测对抗鲁棒性的算法 [J ] . 信息安全学报 . 2024 .

ZOU S Q , CHEN P , DAI J , et al . An algorithm for enhancing adversarial robustness of deepfake detection [J ] . Journal of Information Security . 2024 .

CARLINI N , WAGNER D . Towards evaluating the robustness of neural networks [C ] // Proceedings of the IEEE Symposium on Security and Privacy (S&P) . Piscataway : IEEE Press , 2017 : 39 - 57 .

CHEN S , YAO T , CHEN J , et al . FRADE: Forgery-aware audio-distilled multimodal learning for deepfake detection [C ] // Proceedings of the ACM International Conference on Multimedia . New York : ACM , 2024 : 7890 - 7899 .

DOLHANSKY B , HOWES R , PFLAUM B , et al . The Deepfake Detection Challenge (DFDC) dataset [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 576 - 585 .

LI Y , YANG X , SUN P , et al . Celeb-DF: A large-scale challenging dataset for deepfake forensics [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 3204 - 3213 .

JIANG L , LI R , WU W , et al . DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 2886 - 2895 .

QI H , GUO Q , JUEFEI-XU F , et al . Deep Rhythm: Exposing deep fakes with attentional visual heartbeat rhythms [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM , 2020 : 1318 - 1327 .

HALIASSOS A , VOUGIOUKAS K , PETRIDIS S , et al . Lips don’t lie: A generalisable and robust approach to face forgery detection [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2021 : 5039 - 5049 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

支持模态扩展的高效遥感图像分割网络

基于注意力机制的轻量级SAR船舶检测器

融合多尺度和上下文的无人机救援小目标检测算法

一种基于深度可分离卷积和注意力机制的入侵检测方法

基于AR虚实图像注意力机制的电缆装配质量检测方法