VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND ENGINEERING đà TP.HCM REPORT SPECIALIZED PROJECT SEMESTER 232 ACADEMIC YEAR 2023-2024 DEVELOP A MULTIMODAL CHATBOT FOR A FASHION STORE MAJOR: COMPUTER SCIENCE COUNCIL: COMPUTER SCIENCE - 01 CLC SUPERVISOR(s): QUAN THANH THO, Ph. SECRETARY: TRAN HUY, MEng. o0o STUDENT 1: VOHOANG NHAT KHANG - 2152646 STUDENT 2: NGUYEN PHAN TRI DUC - 2152528 HO CHI MINH CITY, May 2024 Instructor’s Signature Date: Assoc. Quan Thanh Tho, Ph.
(Instructor) Associate Professor Faculty of Computer Science and Engineering Declaration Of Authenticity We declare that we solely conducted this specialized project, under the supervision of Assoc. Quan Thanh Tho at the Faculty of Computer Science and Engineering, Vietnam National University - Ho Chi Minh City University Of Technology. We have taken care to properly acknowledge and document all external sources and references used in the project. If there is any instance of plagiarism, we are ready to accept the consequences.
Ho Chi Minh City University of Technology - Vietnam National University HCMC will not be held responsible for any copyright violations that may have occurred during my research. Ho Chi Minh City, May 2024 Authors, Vo Hoang Nhat Khang, Nguyen Phan Tri Duc Acknowledgement We would like to express my appreciation to Assoc. Quan Thanh Tho for his invaluable guidance. Our research has greatly benefited from his deep knowledge, per- ceptive criticism, and constant support.
Besides, we are grateful for the help from Mr. Nguyen Hieu Nghĩa from University of Information Technology - Vietnam National University (UIT - VNU) for valuable comments for research aspect and our proposed pipeline, including pointing out strengths as well as weaknesses of previous works on multimodal models so that we can elaborate on our ideas more effectively. In addition, we would like to extend our gratitude to my family. Their unwavering faith in our abilities and constant encouragement have been my pillars of strength.
Their belief in my potential has been a constant source of motivation and resilience. We are forever grateful for their love and support. 1H Abstract This thesis explores the integration of multimodal deep learning models into search for fashion accessories, enhancing the user shopping experience through interactive capa- bilities. Our research focuses on developing a robust retrieval system that leverages both image and text inputs to provide accurate product suggestions.
The system architecture allows users to input images of desired products along with additional textual descrip- tions to refine their search queries. Utilizing state-of-the-art multimodal language mod- els, the system retrieves relevant product recommendations based on the similarity of visual and semantic features. Key functionalities include providing detailed information about product availability, pricing, and suggesting alternative products based on specific customer preferences. The interactive nature of the system ensures a tailored shopping experience, enabling users to refine their search criteria and explore a diverse range of fashion accessories.
Preceded by comprehensive research on multimodal deep learning techniques frame- works, this thesis contributes to advancing the field of information retrieval. The ex- perimental evaluation demonstrates the efficacy of the proposed approach in delivering accurate and relevant items, ultimately enhancing customer satisfaction and engagement in online shopping environments. IV Table of Glossary Term Definition Note Multimodal Relating to multiple modes of in- put/output, such as text and images, used in communication or computing systems. Chatbot A computer program designed to sim- ulate conversation with human users, typically over the internet.
RAG System Retrieval-Augmented Generation sys- tem, a framework that combines re- trieval and generation models for en- hanced natural language understand- ing and generation. CLIP Encoder Contrastive Language-Image Pretrain- ing Encoder, a deep learning model that learns joint representations of images and text through contrastive learning. Term Definition Note End-to-end A design process or system designed to operate without intermediate stages or interactions from separate compo- nents. In other words, it refers to the end-to-end nature of a process from start to finish without intermedi- ate steps.
Phase A stage or step. For example, to solve a large problem, we would address smaller tasks (in phases). In the scope of the topic, there are phase | and phase 2. Transformer A machine learning model architecture Reference [49] proposed in 2017, achieving high ef- ficiency in Natural Language Process- ing (NLP).
Pretraining The process of training a machine learning model before it is fine-tuned for a specific task. In this stage, the model learns from large and diverse datasets to understand general patterns and representations, forming a strong semantic understanding. Pretraining 1s an important step in building high- performance models for various tasks. vi Term Definition Note Fine-tuning In machine learning, fine-tuning is the process of adjusting a model that has been pre-trained on a small or spe- cific dataset.
The goal is to make the model understand more and apply spe- cific knowledge from new data or spe- cific tasks. Vil Contents 1 Introduction LL Motivation. Quy va cố.ee WwW 2 Related Works Œứ 2.3 PhoBERIT: Pre-trained language models for Vietnamese 2.4 Other Large Language Models.3 Retrleval-Augmented Generalion. Combining Retrlever with Generatve Models.4 Combining Multimodal Model with Vector Database.4 Multimodal Image-Text Contrasttve Learning .2 Limttatons and Challenges.
vill 3 Theoretical Background 3.2 Multi-Layer Pereeptron.2 Bimary Cross-EntropyLoss.4 Convolutional Neural Network.5 Recurrent Neural Network .3 Back-propagation through Time. 36 Long-Short Term Memory .10 Large Language Models. 67 5 Conclusion 69 References 70 A Inference on PhoCLIP 77 A.I Text-only queries .2 Image-only queries .3 Image-textcombined queles. 86 B Table of Workload 90 B.0 0000000000 eee ee 90 List of Tables 4.4 Average cosine similarity.
65 x1 List of Figures 2.2 In a regular block (left), the portion within the dotted-line box must di- rectly learn the mapping f(x). In a residual block (right), the portion within the dotted-line box needs to learn the residual mapping g(x) = f(x) — x, making the identity mapping f(x) = x easier to learn’.3 Retrieval-Augmented Œeneration pIpeline.4 CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples. At test time the learned text encoder synthesizes a zero-shot linear classifier by embedding the names or descriptions of the target dataset’s classes.1 MLP architecture with 2 hidden layers.2 Graph ofthe sigmoid functlon.3 Graph ofthe tanh funcion.4 Graph ofthe ReLU function.5 Filter and stridein CNN 2.6 Pooling demonstrations with Max Pooling and Average Pooling 3.7 RNN General Architecture 2.8 One-to-One RNN archtectre .9 One-to-Many RNNarchtecue .10 Many-toe-OneRNNarchtecue .11 Many-to-Many RNN archttectire.12 LSTM architecture and íts cell structure'.13 Anexample of Tokenizer.14 Transformer architecture from paper ’’Attention Is All You Need” [49] .15 Multi-head Attention architecture from paper Attention Is All You Need” [49] 2 v2 53 3.16 Early Fusion pipeline.17 Late Fusion pipeline. 00 57 41 Architecture of PhoCLIP, adopt from OpenAI CLIP.
67 xill Chapter 1 Introduction In chapter I, the overview, objectives, and goals of the research project are illustrated. The outline of the report is also presented.1 Motivation As the fashion market expands to cater to diverse demographics and preferences, the con- ventional methods of searching for products become increasingly inefficient. Consumers often face challenges in finding fashion items that align with their specific preferences, budget constraints, and other unique requirements. Traditional approaches rely heavily on human-assisted interactions through websites, where customers engage in conversa- tions with store managers to seek guidance or recommendations.
However, this manual process is time-consuming, prone to errors, and fails to leverage the full potential of technology in enhancing customer experiences. The inefficiency of this process raises a significant challenge: How can this whole process be automated to enhance customer experience and streamline operations for fashion stores? By introducing a multimodal searching method, we aim to revolutionize the way cus- tomers interact with fashion stores online. The integration of image and text capabili- ties allows for a more intuitive and efficient communication channel between the con- sumer and the store. With this technology, customers can simply provide an image or description of the desired product, and the system can quickly analyze their preferences and recommend suitable options.
Through fine-tuning a Vietnamese multimodal model and implementing an efficient information retrieval pipeline, the system can understand customer preferences, recommend appropriate products, and provide personalized assis- tance, thus revolutionizing the way customers interact with fashion stores online. Moreover, the development of such a system addresses the pressing need for automation in the fashion retail sector. As the number of fashion stores continues to rise, streamlin- Ing operations and improving customer experiences become paramount. A well-trained multimodal search system can effectively assist customers in navigating through the vast array of products, providing personalized recommendations, and answering queries re- garding price, availability, and discounts.2 Goals The goals of this project encompass both technical advancements and practical outcomes aimed at revolutionizing the fashion retail experience.
Firstly, the primary objective is to develop and deploy a sophisticated multimodal chatbot model capable of efficiently assisting customers in navigating the diverse offerings of fashion stores. This entails in- tegrating image and text processing capabilities to accurately interpret customer requests and provide relevant product recommendations. Additionally, a key goal is to enhance the overall customer experience by reducing the time and effort required for shopping online. By leveraging the chatbot’s capabilities in understanding customer preferences and responding promptly to inquiries, the project aims to streamline the shopping process, making it more intuitive and convenient for users.
Another crucial aspect of the project is to optimize the efficiency of fashion store operations through automation. This efficiency gain not only improves the productiv- ity of fashion stores but also reduces operational costs and enhances overall business performance. Moreover, the project seeks to foster innovation in the field of natural language pro- cessing and multimodal technology. By pushing the boundaries of what is achievable in terms of understanding and responding to customer queries, the development of the chatbot contributes to advancements in Al-driven conversational systems.
This not only benefits the fashion industry but also has broader implications for other sectors seeking to enhance customer interactions through automated solutions.3 Scope Firstly, our project involves designing and implementing the core chatbot architec- ture that integrates both image and text processing capabilities. The chatbot should be able to accurately interpret customer inputs, whether they are in the form of im- ages depicting desired products or text descriptions. Secondly, we will develop a model that can give most similar items to user based on user’s input image and/or text prompt. Our baseline model to start this project will be CLIP [36], and we also propose a new model for encoding Vietnamese language along with the image, namely PhoCLIP.
Thirdly, the project also involves implementing efficient information retrieval mech- anisms to fetch product details such as price, availability, discounts, and other rele- vant information from the fashion store’s database or external sources by Retrieval- Augmented Generation (RAG). This ensures that the chatbot can provide accurate and up-to-date information to customers in real-time. Finally, is our effort to design an intuitive and user-friendly interface for interacting with the chatbot is crucial for enhancing the overall user experience, by creating conversational flows, interactive elements, and visual representations to facilitate seamless communication between the chatbot and the user.4 Thesis Structure There are five chapters in our project: Chapter 1 provides a comprehensive introduction to the motivation behind the problem, its goals, objectives, and the scope of the project. Chapter 2 highlights previous research and related works pertaining to the task at hand, aiming to gain insights into existing methodologies and their limitations.
* Chapter 3 delves into the foundational concepts of Natural Language Processing (NLP), Vision models and Multimodal models, which are essential for the project’s implementation. ¢ Chapter 4 is dedicated to our approach about the implementation details of our project, including the development of the multimodal chatbot and associated sys- tems. ¢ Chapter 5 concludes our project by summarizing the key findings and outcomes, while also outlining our plans for future development and improvements.