Multimodal Document Intelligence System using Vision Transformers and Large Language Models

Article Citation : Venkatesan Paramasivam , Siddhant Jain , Priyanka Surve , Shridhar Priyadarshi , Subhashini Lakshmi Narayanan , (2026) "Multimodal Document Intelligence System using Vision Transformers and Large Language Models", International Journal of Industrial Electronics and Electrical Engineering (IJIEEE) , pp. 28-31, Volume-14,Issue-4

Abstract : Organizations across industries generate a large volume of documents such as invoices, medical records, legal agreements, and financial reports. Extracting meaningful insights from these documents remains a challenging task due to their unstructured format, complex layouts, and varying content types. Traditional document processing systems rely heavily on rule-based approaches and simple Optical Character Recognition (OCR), which often fail to capture contextual relationships between textual and visual elements. This research proposes a Multimodal Document Intelligence System that integrates Vision Transformers, Optical Character Recognition (OCR), Natural Language Processing (NLP), and Large Language Models (LLMs) to automatically extract, analyze, and interpret information from document images and PDFs. The proposed system combines document layout understanding using LayoutLMv3, text extraction using PaddleOCR, semantic representation using Sentence Transformers, and knowledge retrieval using a vector database. A Retrieval-Augmented Generation (RAG) approach enables the system to answer user queries based on the extracted document content. Experimental evaluation demonstrates that the proposed system achieves high accuracy in document understanding tasks, improving information extraction and question-answering capabilities across multiple document datasets. Keywords - Document Intelligence, Vision Transformers, OCR, LayoutLM, Large Language Models, Retrieval-Augmented Generation

Type : Research paper

Published : Volume-14,Issue-4