基于多模态与图注意力网络的表格结构识别算法研究

徐崇善; 吴绿; 李宁; 徐宁; 潘昊

doi:*****

您当前的位置：

首页 >

文章列表页 >

基于多模态与图注意力网络的表格结构识别算法研究

更新时间：2026-03-03

- 基于多模态与图注意力网络的表格结构识别算法研究
- Table Structure Recognition via Multimodality and Graph Attention Networks
- 物联网学报 2026年
- 作者机构：
  
  1.武汉理工大学计算机与人工智能学院，湖北武汉 433000
  2.武汉理工大学信息工程学院湖北武汉 433000
- 作者简介：
  
  吴绿，wulv@whut.edu.cn
- 基金信息：
  
  国家自然科学基金资助项目(92373102)
- DOI：*****
  中图分类号：
- 收稿：2025-11-07，
  
  修回：2026-01-12，
  
  录用：2026-02-09，
- 稿件说明：
移动端阅览
徐崇善, 吴绿, 李宁, 等. 基于多模态与图注意力网络的表格结构识别算法研究[J/OL]. 物联网学报, 2026.

XU Chongshan, WU Lv, XU Ning, et al. Table Structure Recognition via Multimodality and Graph Attention Networks[J/OL]. Chinese Journal on Internet of Things, 2026.
徐崇善, 吴绿, 李宁, 等. 基于多模态与图注意力网络的表格结构识别算法研究[J/OL]. 物联网学报, 2026. DOI： *****.

XU Chongshan, WU Lv, XU Ning, et al. Table Structure Recognition via Multimodality and Graph Attention Networks[J/OL]. Chinese Journal on Internet of Things, 2026. DOI： *****.

摘要

表格作为信息的高度浓缩和结构化呈现方式，广泛存在于各类文档和图像中。准确理解表格结构是从非结构化数据向结构化数据转化的关键一步。由于表格形状、表头配置、多行单元格以及分隔线样式的复杂性和多样性，尤其是在处理缺乏边框的表格时，这项任务既重要又充满挑战。针对无边框表格，提出了一种新的表格结构理解方法，主要有以下几项关键创新：首先，为了提高模型的泛化能力，引入了一种新的多模态编码器，用于融合表格的视觉、文本、布局和边框特征。其次，提出了基于文本位置分布的框线添加算法，以完善表格边框。最后，在编-解码器架构中创造性地引入了图注意力网络，使得节点在感知全局节点特征的同时，也能更好地感知相邻节点的特征。该算法显著提升了无边框表格结构识别的准确性，相较于当前已有方法显著提高了树编辑距离相似度，在PubTabNet公开数据集上进行了测试，简单表格上达到99.46%，复杂表格上达到96.65%，无边框表格上达到了98.34%。

Abstract

Tables

as a highly condensed and structured form of information presentation

are ubiquitous across documents and images. Accurate table structure recognition is a critical step in converting unstructured visual data into machine-interpretable structured formats. This task remains particularly challenging due to the wide variability in table layouts — including irregular shapes

complex headers

spanning cells

and diverse ruling-line styles — with borderless tables posing the most formidable difficulties.To tackle borderless tables effectively

we propose a novel structure understanding framework featuring three key innovations.First

we introduce a new multimodal encoder that fuses visual

textual

layout

and ruling-line features

thereby significantly improving model generalization across diverse table appearances.Second

we develop an efficient border reconstruction algorithm based on the statistical distribution of text positions. By exploiting horizontal and vertical alignment patterns of cell contents

the method reliably infers missing ruling lines.Third

we incorporate a Graph Attention Network (GAT) module into the encoder-decoder architecture. This allows each table element to attend to both global table context and fine-grained local relationships with neighboring cells

yielding more coherent structure predictions.Experiments on the PubTabNet benchmark demonstrate the superiority of the proposed approach. It achieves state-of-the-art Tree-Edit-Distance-based Similarity (TEDS) scores: 99.46% on simple tables

96.65% on complex tables

and notably 98.34% on borderless tables — representing substantial gains over prior methods.

关键词

Keywords

references

HOU Q , WANG J . TABLET: Table Structure Recognition Using Encoder-only Transformers [C ] // International Conference on Document Analysis and Recognition . Cham : Springer Nature Switzerland , 2025 : 253 - 278 .

ZHONG X , SHAFIEIBAVANI E , JIMENO YEPES A . Image-based table recognition: data, model, and evaluation [C ] // European conference on computer vision . Cham : Springer International Publishing , 2020 : 564 - 580 .

YU C , LI W , LI W , et al . A survey for table recognition based on deep learning [J ] . Neurocomputing , 2024 , 600 : 128154 .

HUANG Y , LU N , CHEN D , et al . Improving table structure recognition with visual-alignment sequential coordinate modeling [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2023 : 11134 - 11143 .

LIN W , SUN Z , MA C , et al . Tsrformer: Table structure recognition with transformers [C ] // Proceedings of the 30th ACM International Conference on Multimedia . 2022 : 6473 - 6482 .

KASEM M , ABDALLAH A , BERENDEYEV A , et al . Deep learning for table detection and structure recognition: A survey [J ] . ACM Computing Surveys , 2024 , 56 ( 12 ): 1 - 41 .

SINGH A , BACCHUWAR K , BHASIN A . A survey of OCR applications [J ] . International Journal of Machine Learning and Computing , 2012 , 2 ( 3 ): 314 .

QASIM S R , MAHMOOD H , SHAFAIT F . Rethinking table recognition using graph neural networks [C ] // 2019 International Conference on Document Analysis and Recognition (ICDAR) . IEEE , 2019 : 142 - 147 .

LI X H , YIN F , DAI H S , et al . Table structure recognition and form parsing by end-to-end object detection and relation parsing [J ] . Pattern Recognition , 2022 , 132 : 108946 .

LIU H , LI X , LIU B , et al . Neural collaborative graph machines for table structure recognition [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022 : 4533 - 4542 .

孙俊 , 苟刚 . 融合图卷积网络的表格图像结构识别模型 [J ] . 计算机工程与设计 , 2024 , 45 ( 10 ): 3066 - 3073 .

SUN J , GOU G . Table image structure recognition model fusing graph convolutional network [J ] . Computer Engineering and Design , 2024 , 45 ( 10 ): 3066 - 3073 .

VELIČKOVIĆ P , CUCURULL G , CASANOVA A , et al . Graph Attention Networks [C ] // International Conference on Learning Representations . 2018 .

UDAYAKUMAR P , SUBHASHINI R . Connectome Graph Attention Network (CGAT): A Multimodal approach for enhanced classification of Alzheimer's disease [J ] . Results in Engineering , 2025 : 105200 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . Advances in neural information processing systems , 2017 , 30 .

WANG Z , DONG H , JIA R , et al . Tuta: Tree-based transformers for generally structured table pre-training [C ] // Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 2021 : 1780 - 1790 .

Lu N , Yu W , Qi X , et al . Master: Multi-aspect non-local network for scene text recognition [J ] . Pattern Recognition , 2021 , 117 : 107980 .

LY N T , TAKASU A . An end-to-end multi-task learning model for image-based table recognition [C ] // Document Analysis Systems . Cham : Springer Nature Switzerland , 2024 : 150 - 163 .

YANG G , ZHONG D , XIONG Y , et al . LRATNet: Local-Relationship-Aware Transformer Network for Table Structure Recognition [C ] // International Conference on Multimedia Modeling . Cham : Springer Nature Switzerland , 2024 : 507 - 520 .

XIAO B , SIMSEK M , KANTARCI B , et al . Rethinking detection based table structure recognition for visually rich document images [J ] . Expert Systems with Applications , 2025 , 269 : 126461 .

HERZIG J , NOWAK P K , MÜLLER T , et al . TaPas: Weakly supervised table parsing via pre-training [C ] // Proceedings of the 58th annual meeting of the association for computational linguistics . 2020 : 4320 - 4333 .

YIN P , NEUBIG G , YIH W , et al . TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 2020 : 8413 - 8426 .

ZHANG Z , LIU S , HU P , et al . UniTabNet: Bridging Vision and Language Models for Enhanced Table Structure Recognition [C ] // Findings of the Association for Computational Linguistics: EMNLP 2024 . Miami : Association for Computational Linguistics , 2024 : 6131 - 6143 .

PENG S Y , CHAKRAVARTHY A , LEE S , et al . UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretraining [C ] // NeurIPS 2024 Third Table Representation Learning Workshop .

BADARO G , SAEED M , PAPOTTI P . Transformers for tabular data representation: A survey of models and applications [J ] . Transactions of the Association for Computational Linguistics , 2023 , 11 : 227 - 249 .

NASSAR A , LIVATHINOS N , LYSAK M , et al . Tableformer: Table structure understanding with transformers [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022 : 4604 - 4613 .

ZHENG X , WANG W , LIU W , et al . Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context [C ] // 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) . IEEE , 2020 : 697 - 706 .

LI M , CUI L , HUANG S , et al . Tablebank: Table benchmark for image-based table detection and recognition [C ] // Proceedings of the Twelfth Language Resources and Evaluation Conference . 2020 : 1918 - 1925 .

JIMENO YEPES A , ZHONG P , BURDICK D . ICDAR 2021 competition on scientific literature parsing [C ] // International conference on document analysis and recognition . Cham : Springer International Publishing , 2021 : 605 - 617 .

LY N T , TAKASU A , NGUYEN P , et al . Rethinking image-based table recognition using weakly supervised methods [C ] // International Conference on Pattern Recognition Applications and Methods . Springer , 2023 , 1774 : 676 - 697

CHEN L , HUANG C , ZHENG X , et al . TableVLM: multi-modal pre-training for table structure recognition [C ] // Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2023 : 2437 - 2449 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于多模态融合与注意力增强的深度伪造检测方法

支持模态扩展的高效遥感图像分割网络