VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY ADVANCED PROGRAM IN INFORMATION SYSTEMS DANG QUANG HUNG- 18520790 GRADUATION THESIS BUILDING A SMART DOCUMENT SCANNER APPLICATION ON MOBILE BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS THESIS ADVISOR Dr. NGUYEN THANH BÌNH HO CHI MINH CITY, 2022 NATIONAL UNIVERSITY HOCHIMINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY ADVANCED PROGRAM IN INFORMATION SYSTEMS DANG QUANG HUNG - 18520790 GRADUATION THESIS BUILDING A SMART DOCUMENT SCANNER APPLICATION ON MOBILE BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS THESIS ADVISOR Dr. NGUYEN THANH BÌNH HO CHI MINH CITY, 2022 ASSESSMENT COMMITTEE The Assessment Committee is established under the Decision. , date by Rector of the University of Information Technology.
3B eecceceeeeeeeseseeeseeeseeeeaeeesteeseeneneaeeee - Member ACKNOWLEDGMENTS First of all, I would like to acknowledge and give sincere thanks to my supervisor, Dr. Nguyen Thanh Binh, who spent his time instructing me during the process of making this thesis. His guidance and advice forwarded me to solutions that helped my product works efficiently. The final result is really a treasure which helped me learn a lot of knowledge and made the last four years fly by.
I also could not have undertaken this graduation thesis without my defense committee, who generously provided knowledge and expertise. Additionally, I am grateful to my classmates and cohort members, for giving me valued feedback and moral supports. Thanks should also go to the librarians, and teachers, who impacted and inspired me. Lastly, I would like to mention to my family for moral support.
By strong words, they motivate me every time I feel exhausted. Your assists made this thesis become meaningful and stay memorable in my mind. TABLE OF CONTENTS œElg» ACKNOWLEDGMENTTS. cọ tt HH nh i TABLE OF CONTENTTS.- cọ HH ng KHE ii LIST OF EIGURES.
TH tt 1n iv LIST OF TABLES.cccscsssestssesseeseseeeseeeneseeaeseseeseseeensesneseseeananeenansteneansees vi ABSTRACTT.- cọ HH nh TH TH HH n0 007. Object and scope Of Sfudy.ineiieiiee 3 VN Noo .3 Integrated development environment for Android.2 Machine learning library on Android oo.----2cvccvcc nh re 10 2.3 Image processing algorithims.---:-cc+cc¿‡222Evxvvtrirrttttkkrrrrrrirtiirirrrer 11 2.4 Optical Character Recognition (OC)). THE PROPOSED PROCESSING METHOD.1 General processing ÍÏOW. - «cv HH tr viec 19 3.2 Edge processing based approach.2 Canny edge detection 3.3 Cloud processing based approach.--¿-- 5+ 5xx st‡xevsrertrrkeksrererrrkrkevee 36 3.- «Sàn HH HH.
APPLICATION DESIGN AND IMPLEMENTATION.1 Analysis and design. -- -- 5+ St TT HH 11101111 1111 11.4 Implementation and testing. LIST OF FIGURES cae Figure 2.1 Reasons for choosing AndFOI.2 Android architecture Figure 2.3 Android studio icon.4 Popular OpenCV modules. -- - ¿+ + S‡EE£k+kekexexerrrxekrrrrrrrrkrvee 7 Figure 2.5 TensorFlow Lite model execution flow in Android applications .6 RGB (3-channel) image represernfafiOI.7 RGB image visualization.
cece sees eeseseeeeseeeeseeseseeaeseeseseeseeeesees 11 Figure 2.8 Grayscale (1-channel) image represenfafion.9 Grayscale transformation for each pixel in RGB image.10 Image filtering — ConVOÏUtIOH.11 Original image and Image after applying Gaussian blur.12 Original image and Image after applying thresholding.13 Result visualization of edge detectiO.14 Example of perspective transfOrimat(IOT. + + +5++s+xex+£vrvzzvsxsx+ 17 Figure 2.15 Construction of a typical OCR sysfem.1 General processing flOW .2 Process flow of scanning CID card .- - - - +55 sx+v+xexexerervrerrersre 20 Figure 3.3 Processing flow of edge processing based appoarch.4 Image preprocessing aÌgOTIthIm§.--- -- - + + St+x*Eekerekrkerkekrkerrk 23 Figure 3.5 The Gaussian function in one and two dimensIOns.-- -- - + 5+ +£Sk+EkEkeEkEkeEkEkerkekkrkerkrkrkerkek 26 Figure 3.7 Threshold to Z⁄TO.-¿- ¿+ +23 Sk*k#kEktxvEkEkEkEkskerkrkrkrkrkekrkrrrrkrkrkrkerrre 27 Figure 3.8 Non maximum suppression visualization in coordination.9 Simple CID card. -- - ¿5+ SE HH1 01 1 01 1111 11 1 11 re.10 Processing flow of traditional text detection phase.11 Processing flow of new text detection model.12 Sample annotated CID car(. -- - - +: t+++xx+£vEvEvEevexexererrvrerkersre 32 Figure 3.13 EfficientDet archiit€CẨUTC.-- - - + St ‡v+veEErtEkrkekekekerrrkrkrkrkerree 33 Figure 3.14 Retraining model DFOC€SS.15 Text recognition DFOC€SS.--- -- - + SE TH 1101111111111 xe.16 Processing flow of the cloud processing based approach.17 VietOCR archit€CfLITC.
- tt Sky 37 Figure 3.18 Experiment in đayfIIT€. - ¿+ St StSt#k+veEEEeEvEekekekererrkrkrkrrrre 38 Figure 3.19 Experiment in dark enVITOTINII(.1 System arCIIt€CẦUTC. - - 5c kg HH 1y 4I Figure 4.2 Use-case øeneral1Za{iOII. ¿+ + EkE#k#kekerErkrrkekekrkerrrkrkrkrkerree 42 Figure 4.- -- - tt HH HH1 1.4 UI of main SCT€€I.
¿6 + EkStSkEk‡xEvEtEkEkskekEkrrrrkekekrkrkrrkrkrkrkrrree Figure 4.5 UI of camera screen Figure 4.6 UI of image cropping screen.7 UI of image reviewing screen Figure 4.8 UI of image viewing SCT€€N.9 UI of tex result SCT€€TI. - 5c tt tEEEEEEEkSkeEEErErEekekrkrkrrrkrkrkerree 90 LIST OF TABLES Table 4.1 List of Use-Case he.2 Add new document.3 Choose image from gaÌÏ€Ty.4 Capture new 1ITABC.8 Mange image ImOde.10 View doCuime€Hi(. cece eecesceseeeseeseeeeneseseeseseeesseseesesnesteasseeeesesneenesees 65 Table 4. co nh re 74 Table 4.16 Save text result.
80 vi ABSTRACT Our daily lives have increasingly become more reliant on smartphones. The hardware explosion was caused by competition between a high number of smartphones competing for market share. The development of algorithms has also increased the capabilities of smartphones and improved the user experience. People can now do a lot with a smart device, leading us to adopt a new viewpoint that emphasizes the use of smart gadgets to successfully complete daily tasks.
In order to keep up with this trend, this thesis offers a novel solution: an application that combines the benefits of mobile devices with those of machine learning models for document scanning and information extraction. The application is primarily concerned with reading and retrieving data from Vietnam citizen identification (CID) cards. Information that has been gathered can be used as input for an information system.1 Problem statement The bank clerk would typically ask consumers to provide their CID card when they visit and complete a procedure so that a profile can be created. It is clear that entering client data from the CID card requires a lot of time.
Therefore, the CID scanner program was developed to scan the CID card, extract vital data such as the full name, date of birth, place of birth, and address, and then accurately and automatically input that data into the customer profile. We can save a lot of time and money that would have been spent on recruiting extra workers by using this program.2 Objectives The following aims are covered by the thesis: Identifying methods for document scanning on mobile devices. Determining the ideal response. Creating the application's user interface and user experience.
Implementing developed user interfaces on Android smartphones. Object and scope of study The document scanner application's capability to perform scanning functions on a specific paper format is the major part of this thesis. We selected to employ the Vietnam CID card for our study due to this reason. Vietnam CID cards are extensively used and play a significant role in regular life; thus, extracting the valuable information contained in them encouraged us to put our solution into practice in order to extract information.
The scope of this study contains some topics, including: - Mobile application development - General application development - Computer vision technique (Object, Region of Interest Detection) - Machine Learning — Optical Character Recognition (OCR) Chapter 2.1 Introduction Android, an operating system developed by Google, is a programming platform for cell phones and other mobile devices such as tablets. It runs on many different devices from many different manufacturers. Android includes a software development kit (SDK) that you can use to write source code and assemble software modules to create applications for Android users. Android also provides a marketplace for selling applications.
Overall, Android is a mobile application ecosystem [1]. Rich development environment Open source Inter app | intergration Large developers and community reach Figure 2.1 Reasons for choosing Android 2.2 Architecture Android provides a rich development framework. You don't need to know the components of this architecture, but it's helpful to know what's available in the system for your application [2]. The diagram below shows the main components of the Java API Framework Native C/C++ Android 3 Libraries Runtime Hardware Abstraction Layer (HAL) ! Figure taken from https://google-developer-training.io/android-developer-fundamentals-course-concepts-v2/unit-1- get-started/lesson-1-build-your-first-app/1-0-c-introduction-to-android/1-0-c-introduction-to-android.html In the illustration above: Applications: This tiers features applications and critical system elements like emails, SMS, calendars, internet browsing, and contacts [2].
Java API Framework: All of Android's features, from UI components to resource management and lifecycle management, are accessible via application programming interfaces in the Java API Framework (APIs). The specifics of how the API functions don't need to be known to you. Just figure out how to use them [2]. Libraries and the Android Runtime: Each application launches a separate process and Android Runtime instance.
The majority of the Java programming language's features are provided via a core set of runtime libraries that are part of Android. The Android operating system is constructed from native code, which involves the implementation of native libraries in C and C++. The Java API framework makes these native libraries approachable to applications [2]. Linux kernel: The Linux kernel is the foundation of the Android OS.
Threading, low-level memory management, and other basic features are all covered by the Kernel in the layers above it. Android can gain from Linux-based security features by using a Linux kernel, and device manufacturers can create hardware drivers for a well- known kernel [2]. Hardware abstraction layer (HAL): This layer gives the more sophisticated Java API framework common interface that expose device hardware capabilities. The HAL is composed of different library modules, each of which implements an interface for a specific type hardware section, such as a Bluetooth or camera [2].3 Integrated development environment for Android There are numerous IDEs that allow Android programming; however, we discovered that Android Studio is the best IDE for this thesis' requirements.
We built the document scanner application to run machine learning models and algorithms on mobile devices. With the help of the libraries indicated in the next section, we implemented a large number of algorithms. android studio Figure 2.3 Android studio icon? ? Figure is taken from https://commons.org/wiki/File:Android_Studio_Trademark.2 Machine learning library on Android 2.1 OpenCV A comprehensive set of computer vision algorithms can be found in the open-source library known as OpenCV. Python, Java, C++, and a variety of programming languages are recognized by OpenCV.
It is compatible with a diverse range of operating systems, namely OS X, Android, iOS, Windows, Linux, and. The most frequently used modules in OpenCV are depicted in the figure below. OpenCV has a modular structure that contains various shared or static libraries [3]. Video analysis 2D features framework High-level GUI (video) {features2d) (highgui) Corefunctionality Object— det ction 3D †ez ietfuidl on Image processing Videoio _ `" 4 (videoio) ; : ‘ Camera Calibration and (core) (objdetect) (calib3d) Figure 2.4 Popular OpenCV modules We simply introduce the OpenCV image processing module in order to condense the thesis's substance.
Many of the algorithms in the image processing module take an image as input and return the desired result, which can then be used as input for other tasks.2 Tensorflow Lite TensorFlow Lite is a collection of technologies that makes it possible for developers to execute their models on mobile, embedded, and edge devices, enabling on-device machine learning [4]. According to TensorFlow Lite, we can use TensorFlow machine learning (ML) models in the Android apps. With options for hardware acceleration, the TensorFlow Lite system offers prebuilt and scalable execution environments for rapidly and efficiently operating models on Android [4]. TensorFlow Lite employs TensorFlow models that have been downsized, portable, and optimized for machine learning.
With TensorFlow Lite for Android, we may utilize pre-built models or create your own TensorFlow models and export them in TensorFlow Lite format [4]. An Android app running a TensorFlow Lite model receives input, analyzes it, and makes a prediction based on the model's logic. The input provided into a TensorFlow Lite model must be in a certain data format called a tensor in order for it to function, and it also needs a special runtime environment.