VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE HUA VAN SON - NGUYEN THINH QUYEN BACHELOR IN COMPUTER SCIENCE HO CHI MINH CITY, 2021 VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE HUA VAN SON - 18521344 NGUYEN THINH QUYEN - 18521322 BACHELOR IN COMPUTER SCIENCE THESIS ADVISOR NGUYEN VINH TIEP, Ph. HO CHI MINH CITY, 2021 ASSESSMENT COMMITTEE The Assessment Committee is established under the Decision. by Rector of the University of Information Technology. Le Minh Hung 2.Sc Do Van Tien 3.
Mai Tien Dung ACKNOWLEDGEMENT A completed study would not be done without any assistance. First of all, I would like to express my endless thanks and gratefulness to our advisor PhD. Nguyen Vinh Tiep, without his motivation and instructions, the thesis would have been impossible to be done effectively. We also like to express our gratitude to the Multi Media Laboratory (MMLab-UIT) for providing us with a conducive re- search environment and cutting-edge technology during our academic experience.
Specially, we wish to express our gratitude in particular to our fellow friends who provide us with valuable ideas and are always eager to assist us when we are in need. ABSTRACT In recent years, with the rapid development of artificial intelligence, image caption has gradually attracted the attention of many researchers in the field of artificial intelligence and has become an interesting and arduous task. Image caption, automatically generating natural language descriptions according to the content observed in an image, is an important part of scene understanding, which combines the knowledge of computer vision and natural language processing. The applica- tion of image caption is extensive and significant, for example, the realization of human-computer interaction.
We present a learning method Oscar [1], which uses object tags detected in images as anchor points to significantly ease the learning of alignments. This method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. This approach establishes a new state-of-the-art in Image Captioning for a widely used language, English. In this thesis, we experiment Oscar model on a healthcare domains dataset in Vietnamese and compare performance with a previous approach [2].
We fine-tune the model which has best performance with Self-Critical Sequence Training (SCST) [3], improving AVG(BLEU1, BLEU2, BLEU3, BLEU4) from 0.286 at beam size 3 Contents List of Tables IV List of Figures Vv 1 Introduction | 1. ee 8 2 Related Works 10 2.1 Visual Feature Encoding .2 Linguistic Feature Decoding .00000 2b ee ee 26 3 Methodology 31 3.2 Object-Semantics Aligned Pre-Training. Adapting to Vision-Languagetask.1 English Vision-Languagetasks.2 Vietnamese Image Captioning Task .4 Fine-tune with Self-critical Sequence Training.1 Language model sampling .2 Self-Critical Sequence Training .1 Overview VieCap4Hdataset.2 Future work 66 Bibliography 67 Il List of Tables 4.1 Our original experiments result on VieCap4H dataset .2 The results of Oscar on COCO dataset [35] .3 Our result before and after modify Object detector’s parameter module on VieCap4H dataset.4 Our result after finetune Oscar,,;, with SCST on VieCap4H dataset 61 4.5 The statistical of object tag of COCO dataset and VieCap4H dataset 64 IV List of Figures 2.2 Fast R-CNN architecture. An input image and multiple regions of interest (Rols) are input into a fully convolutional network.
Each Rol is pooled into a fixed-size feature map and then mapped to a feature vector by fully connected layers (FCs). The network has two output vectors per RoI: softmax probabilities and per- class bounding-box regression offsets. The architecture is trained end-to-end with a multi-task loss.3 Region Proposal Ñetwork(RPN)[22].4 Encoder and Decoder pipeline[23].5 The Transformer - model architecture [24].6 BERT input representation. The input embeddings are the sum of the token embeddings, the segmentation embeddings and the position embeddings.
The model takes a triple as input, is pre-trained with two losses (a masked token loss over words & tags, and a contrastive loss between tags and others), and fine-tuned for 5 understanding and 2 generation tasks.2 Illustration on the process that Oscar represents an image-text pair into semantic space via dictionary look up. (a) An example of input image-text pair (b) The object tags are used as anchor points to align image regions with word embeddings of pre-trained lan- guage models. (c) The word semantic space is more representative than image regio features. In this example, dog and couch are similar in the visual feature space due to the overlap regions, but distinctive in the word embedding space.3 The simplified pipeline of Oscar model adapt to Vietnamese [1] 3.4 Illustration on the process.
(a) The pipeline of Oscar adapt on Image Captioning task in English. (b) The pipeline of Oscar adapt on Image Captioning task in Vietnamese .5 Beam Search with k=2|3l].6 Random sampling vs. random sampling with temperature [32] 3.7 The top 6 tokens with the highest probability are reviewed and randomly selected from these 6 tokens [32] .8 Top-p sampling will choose the tokens with the highest probability such that the total probability > 0.9 Simplified pineline of SCST[3] .1 VieCap4H - General dataset visualization .2 Example of VieCap4Hdataset. Examples l of image captioning.
Objects are colored, based on their appearance against the groud-truth (GT): tags & GT, all .4 Examples 2 of image captioning. 62 VI Chapter 1 Introduction 1.1 Overview Accurate text annotation of image and video content enables more efficient search and retrieval, can aid visual understanding in medical, security, and military applications, and can even be used to describe pictorial content to the visually impaired. Uncertainties about salient content, main subject detection, object recognition, action detection, and scene understanding make this a challenging problem. Despite the difficult nature of this task, computer vision and natural language processing researchers have made significant strides in this area.
This thesis builds on prior achievements and applies these models to the problem of image captioning in Vietnamese in a particular topic such as health care.1 Image Captioning task Supervised convolutional models have fundamentally changed the landscape of computer vision and machine learning in past years. Convolutional Neural Net- works (CNNs) [4] have surpassed the conventional matching of handcrafted low level vision characteristics with complementary classifiers, thanks to the recent advent of massive supervised datasets [5] and rapid training models using Graphic Processing Units (GPUs) [6]. CNNs are deep feed forward networks built on a hierarchy of abstract layers that learn low-level features and classifiers at the same time. Motivated by the need to learn temporal sequences, Recurrent Neural Networks (RNNs) enable solutions to basic problems such as activity recognition and object trajectory prediction.
"Show and Tell: A Neural Image Caption Gen- erator" [7] was one of the first works to address the image captioning challenge by aggregating image information using CNNs and generating descriptions with RNNs. Subsequent studies to improve the performance by adding an attention module, such as in the paper "Show, Attend, and Tell: Neural Image Caption Gen- eration with Visual Attention", [2] where they added a self-attention mechanism before feeding the image-text pair into the RNNs for decoding to aid the model in selecting only those parts of the encoding that it believes are relevant to the task at hand. Motivated by the observation that the salient objects in an image can be accurately detected by modern object detectors, in 2018, instead of using CNNs to extract the entire image, the "Neural Baby Talk" paper [8] employs an object detection model to extract object features in the image, which is then passed into RNNs for decoding along with the information from the text. Following the recent success of pre-trained language models such as Bidirectional 2 Introduction Encoder Representations from Transformers (BERT) [9] and Generative Pre- trained Transformer (GPT) [10], there is increased interest in using these models to learn cross-modal representations such as image-text [11, 12], as well as video- text [13, 14].
In “Unified Vision-Language Pre-Training for Image Captioning and VQA” paper [15] proposes a unified encoder-decoder model (VLP) for general vision-language pre-training. The pre-trained model is then fine-tuned for image captioning and visual question answering. The VLP has shown great progress toward addressing Image Captioning task. The most representative approach is to train large Transformer-based models on massive image-text pair data in a self- supervised manner, such as predicting the masked elements based on their context.
The cross-modal representations of the pre-training models can be fine-tuned to adapt to Image Captioning task or others downstream vision-and-language tasks. However, existing VLP methods simply concatenate image region features and text features as input to the model for pre-training and use self-attention to learn image-text semantic alignments in a brute-force yet implicit manner, leaving the model to figure out the cross-modal alignment from scratch. In early 2021, Microsoft researchers introduce Oscar (Object-Semantics Aligned Pre-training) which we use to experiment on Vietnamese dataset in this thesis,to highlight the observation that objects can be naturally used as anchor points to ease the learning of semantic alignments between images and texts.2 Applications The standard of living of humans has substantially improved as science and tech- nology have progressed. Our lives have been elevated by the conveniences to the point that we can enjoy our surroundings.
In reality, robots are replacing humans 3 Introduction in jobs that used to be done by humans. For instance, you might ask the assistant to phone someone for you while you are driving. A virtual assistant, such as Siri, is an example of AI that will access your contacts, recognize the word, and dial the phone number. These assistants utilize natural language processing (NLP), machine learning (ML), statistical analysis, and algorithmic execution to figure out what you want and try to obtain it for you.
For another example, interacting with customer service as a consumer may be time-consuming and frustrating. It’s an inefficient, costly, and difficult-to-manage department for businesses. AI chatbots are one artificially intelligent option that is becoming increasingly popular. Ma- chines can answer commonly asked inquiries, accept and track orders, and route calls thanks to pre-programmed algorithms.
Also with the help of technology, we stay closed to each other via social network and the internet despite physical distance. Technology is being used to every area of human existence with the goal of making it more convenient and efficient. The development of technology leads to the growth of computer science, in which computer vision (CV) and natural language processing (NLP) play an impor- tant role. Image caption, automatically generating natural language descriptions according to the content observed in an image, is an important part of scene understanding, which combines the knowledge of CV and NLP.
Automatic image captioning is widely used by search engines to retrieve and show relevant search results to the user over the annotation keywords, to categorize personal multimedia collections, for automatic product tagging in online catalogs, in computer vision development, and other areas of business and research. The application of image caption is extensive and significant, let us take some exemplary applications to represent the strong potential of this field. PIM (=Product Information Management) systems with AI can analyze photos and provide rich and detailed features for web catalogs automati- cally. Image captioning software can evaluate product photos and automatically recommend appropriate qualities and categories, which may save time and money.
The system may, for example, recognize the type of fashion item, its material, color, design, and garment fit. Customers can navigate through categories more easily with Al-powered visual suggestions. For efficient user involvement, brands like Asos, eBay, and Forever21 already utilize AI-based visual search and picture recognition. Aid for the Visually Impaired.
Image descriptions automatically generated by a computer aren’t as good as those written by a human who can include additional context, but they can be accurate and helpful. An image description might help a blind person read a restaurant menu, or better understand what their friends are posting on social media. An app called Seeing AI!