Vietnam National University - Ho Chi Minh City Ho Chi Minh city University of Technology Faculty of Computer Science and Engineering GRADUATE THESIS GROUNDED LANGUAGE LEARNING: IMPROVE TEXT REPRESENTATION WITH VISUAL INFORMATION Major: Computer science Council: Computer Science 11 Supervisor: Assoc. Quan Thanh Tho Reviewer: Mr. Le Dinh Thuan ---o0o--- Student: Nguyen Tran Cong Duy (1710043) Grounded Language Learning: Improve Text Representation with Visual Information Thesis Nguyen Tran Cong Duy Supervisors Assoc. Quan Thanh Tho Assurance I hereby declare that, except for the reference results from other related works spec- ified in the thesis, the contents presented in this thesis are my own implementation and there is no part of the content applied for a degree at another school.
Ho Chi Minh City, July 11, 2021 i Acknowledgement As a matter of first importance, I am massively thankful to my counselor Assoc. Quan Thanh Tho for his consistent help and direction all through my thesis, and for the opportunity in studying and researching he gave me. Second, I additionally thank my parents and my best friends for the persistent consolation, backing, and consideration. ii Abstract Nowadays, people learn languages through listening, speaking, reading, writing, and multimodal interactions with the real world.
Even a child has been taught from a young age to talk (listen), teach to speak, teach gestures and learn through pictures from a young age, people from a young age not only learn language from resources. or books with only words that learn a combination of images, stories, and descriptive sentences. Today’s language models are mostly not learned from real-world factors but are trained by purely linguistic data sources. There have been a few constructions that incorporate language and other elements into applications for robotics, visual and linguistic tasks, etc.
with positive results but presently a major challenge in the industry, machine learning, deep learning today. iii Contents List of Tables vi List of Figures viii 1 Introduction 1 1.3 The Scope of the Thesis .4 Organization of the Thesis .1 Neural Network - Multilayer perceptron .2 Convolutional Neural Network .4 Recurrent Neural Network .2 Grounded Language Learning Approaches. 44 iv CONTENTS 6 Experiments 50 6.1 Impact of visual dimension .2 The impact of visual grounding .3 Visualization of alignment between tokens and objects .4 Evaluation on Pre-training Tasks .5 Visualization of token-level scoring on first approach .1 What have we done?. 68 References 69 v List of Tables 4.1 Statistics of some common datasets used in visual grounded language learning task.1 Task descriptions and statistics.2 Downstream task results of BERT and our GroundedBERT, we conduct the experiments on BERT-base and BERT-large architectures.
MRPC and QQP results are F1 score, STS-B results are Pearson correlation, SQuAD v1.0 results are exact matching and F1 score respectively. The results, which outperform the other one are marked in bold, are all scale to range 0-100. The ∆base and ∆large columns show the difference between our model and the baseline.3 Downstream task results of BERT, V&L pretrained models and our ObjectGroundedBERT (OGBERT), we conduct the experiments on BERT-base architectures. MRPC and QQP results are F1 score, STS-B results are Pearson correlation, SQuAD v1.0 results are exact matching and F1 score respectively.
The results, which outperform the other one are marked in bold, are all scale to range 0-100. The ∆base column show the difference between our model and the baseline.1 Downstream task results of our ObjectGroundedBERT with different dimension of visual embedding. The metrics and results are set up similar to Table 6. 56 vi LIST OF TABLES 7.2 Downstream task results and comparison of our ObjectGrounded- BERT without training the Text-ground-image Module.
The metrics and results are set up similar to Table 6. The first four rows report the fine-tuned results of our model without training with the visual grounded datasets, the last four rows show the difference to the results reported in Table 7.3 Downstream task results on different pretraining tasks. 58 vii List of Figures 1.1 Wikipedia, BookCorpus datasets .2 Wikipedia, BookCorpus datasets with another visual datasets .1 A simple neural network.2 MLP with 3 layers.3 Visualization of max pooling .14 Recurrent Neural Network .15 RNN calculation in one node.18 Encoded-Decoder with Attention .19 Transformer: attention is all you need .24 Transformer multihead attention. 28 viii LIST OF FIGURES 3.1 IMAGINET architecture from the paper Collell et al.2 Cap2Both is the combination of Cap2Img and Cap2Both, the figure is in the paper Kiela et al.3 Bordes et al.4 Illustration of the BERT transformer model trained with a visually- supervised language model with two objectives: masked language model (on the left) and voken classification (on the right).
The figure is in paper Tan & Bansal (2020) .1 A grounded language learning example.2 Implementation of our ObjectGroundedBERT. The model consists of two components, i. Language encoder and Text-ground-image part. The new representaion of language model combines of Textual embedding and Visual embedding.3 Implementation of our pretraining framework for ObjectGrounded- BERT.
The model consists of four components, i. Object detection model (Faster-RCNN), Object encoder, Cross Modal Transformer and ObjectGroundedBERT. The detail of Cross Modal Transformer layer is shown on the right. The pretraining tasks are Masked Visual Feature Prediction, Image-Text Matching and Masked Language Modeling.1 Illustration of the Attention map of Cross modal layers.2 First visualization of token-level scoring.
This example chooses the caption there is a clean bathroom counter and sink with 3 images.3 Second visualization of token-level scoring. This example chooses the caption bicycle parked in the grass by a tree with 3 images.5 Model page - result page .6 Input of CoLA task at home page .7 Result of CoLA task at home page. 65 ix LIST OF FIGURES 8.8 Input of MNLI task at home page .9 Result of MNLI task at home page .1 Motivation Humans learn language through listening, speaking, reading, writing, and mul- timodal interactions with the real world. Even a child has been taught from a young age to talk (listen), teach to speak, teach gestures and learn through pictures from a young age, people from a young age not only learn language from resources.
or books with only words that learn a combination of images, stories, and descriptive sentences. Today’s language models are mostly not learned from real-world elements but are trained only by pure-language data sources 1.1 (text only) like Wikipedia, BookCorpus, etc.1: Wikipedia, BookCorpus datasets 1 CHAPTER 1.2 Our contribution Previous studies of visual grounded language learning train language encoder with both language objective and visual grounding example. However, due to the differences in distribution and scale between the visual-grounded datasets and language corpora, the language model tends to mix up the context of the tokens that are occurred in the grounded data with those that are not. As such, there is confusion between the visual information and contextual meaning of text during embedding training.
To overcome this limitation, we propose GroundedBERT - a grounded language learning method that enhances the BERT representation with visual grounded information. GroundedBERT comprises of two components: (i) a text-ground-image part captures the global and local semantic mapping between the visual and textual representations learned via sentence-level and token- level mechanism, and (ii) an original BERT embedding captures the contextual representation of words learned from the textual language corpora. Our proposed method significantly outperforms the baseline language models on various language tasks of GLUE and SQuAD datasets. Moreover, these studies use a convolutional neural network (CNN) to extract features from the whole image for grounding with the sentence description.
However, this approach has two main drawbacks: (i) the whole image usually contains more objects and backgrounds than the sentence itself; thus, matching them together will confuse the grounded model; (ii) CNN only extracts the features of the image but not the relationship between objects inside that, limiting the grounded model to learn complicated contexts. To overcome such shortcomings, we propose a novel object-level grounded language learning framework that empowers the language representation with visual object-grounded information. The framework is comprised of two main components: (i) ObjectGroundedBERT captures the visual-object relations and literary portrayals by cross-modal pretraining via a text-ground-image mechanism, and (ii) Cross-modal Transformer helps the object encoder and ObjectGroundedBERT learn the alignment and representation of image-text context. Experimental results show that our proposed framework ObjectGroundedBERT and GroundedBERT consistently outperform the baseline language models on various language tasks of GLUE and SQuAD datasets.
Our works are submitted into two conferences EMNLP 20211 and Neurips 2022 .org/ 2 https://nips.2: Wikipedia, BookCorpus datasets with another visual datasets 1.3 The Scope of the Thesis The coverage of this text are: • We propose GroundedBERT - a grounded language learning approach that enhances BERT representation with visual-grounded information. Instead of grounding visual information to the language model, which changes the original contextual representation, the visual-grounded representation is first learned from the text-image pairs and then joined to the contextual representation to form a unified visual-textual representation. Moreover with ObjectGroundedBERT, to the best of our knowledge, this study is the first to investigate the grounded language learning at the object-level with the rich grounded information containing object features, attributes, and positions. By doing so, we can enhance the ability of grounded language to capture more complex relations and avoid the confusion during learning process.
• To this end, we introduce a Text-ground-image module that captures both global and local semantics between contextual relation of words and image via a novel token-level and sentence-level learning mechanism. We also propose a novel grounded language framework that enhances language representation with visual-objected-grounded information. Instead of using CNN to encode the whole image, we embed the features of objects from an off-the-shelf object detector into the encoder and connect them with the language modality via a 3 CHAPTER 1. INTRODUCTION cross-modal Transformer.
A Text-ground-image mechanism is also proposed to capture the visual object information and their relations found from the semantic correlation of words and image via multi-task pretraining strategy. • We conduct extensive experiments on various language downstream tasks in GLUE and SQuAD datasets, and significantly outperforms the baselines on these tasks. • We build a demo app for CoLA, MNLI tasks using our GroundedBERT.4 Organization of the Thesis • In chapter 2, we provide some basic math and machine learning concepts that might be helpful for the reader to understand the rest of the text. • In chapter 3, we summarize some legacy research works in the area in a consistent way.
• In chapter 4, we first briefly describe why we have the novel ideal and the problem of previous works. • In chapter 5, we describe the detail of our proposed models and training strategies. • In chapter 6, we show our results on many downstream tasks and also how to implement and config for the hyper-parameters. • In chapter 7, we additional analyze our work on many aspects to show the effectiveness and visualize some results of our examples.
• In chapter 8, we describe and illustrate the application to demo our work on downstream tasks. • Finally, we conclude by discussing what we have done in Pre-Thesis and Thesis also The future direction of our work in chapter 9. 4 Chapter 2 Foundations In this chapter, I will present the background knowledge used in the process of making the thesis, including commonly used concepts in deep learning networks and natural language processing, popular network architectures. in image processing and the models used for language.1 Neural Network - Multilayer perceptron 2.1 Logistic regression is a binary classification method, built by a function capable of taking any value and returning a result.
the number between 0 and 1 (sigmoid function) - equivalent to the probability of that happening based on the input data. This is similar to a single-layer neural network. As the figure 2.1 illustrates the use of logistic regression as a simple neural network. In the figure 2.1, the matrix x contains the attributes (features) of an input, the matrix w will be the weights of the attributes and b is the bias.