HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY MASTER THESIS OPTIMIZATION OF 360-DEGREE VIDEO TRANSMISSION PHAM NGOC SON phamngocsonca@gmail.com Specialization: Electronic Engineering ⬚ Instructor: Associate Professor: Truong Thu Huong 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 School: Electrical and Electronic Engineering HANOI, 2023 CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập – Tự do – Hạnh phúc BẢN XÁC NHẬN CHỈNH SỬA LUẬN VĂN THẠC SĨ Họ và tên tác giả luận văn : Phạm Ngọc Sơn Đề tài luận văn: Nghiên cứu tối ưu kỹ thuật truyền dẫn video 360 độ Ngành: Kỹ thuật điện tử (Elitech) Mã số SV: 20212644M Tác giả, Người hướng dẫn khoa học và Hội đồng chấm luận văn xác nhận tác giả đã sửa chữa, bổ sung luận văn theo biên bản họp Hội đồng ngày 10/10/2023 với các nội dung sau: - Định dạng lại mục lục - Thêm danh mục viết tắt - Đánh số lại thuật toán theo đúng chuẩn - Viết lại 1 số đoạn trong luận văn để không trùng lặp. - Tính toán lại độ chính xác trong chương 2. - Thêm công thức tính độ dư thừa trong chương 2. - Tính toán lại 1 số số liệu viết sai trong luận văn.
- Viết lại pseudocode cho đúng hình thức, quy tắc. - Thêm 1 đoạn để chuyển từ chương 2 sang chương 3 cho liên kết, liền mạch. - Đổi tên chương 2 và chương 3 cho giống tên của bài báo. Ngày 19 tháng 10 năm 2023 Giáo viên hướng dẫn Tác giả luận văn CHỦ TỊCH HỘI ĐỒNG THESIS TOPIC: OPTIMIZATION OF 360-DEGREE VIDEO TRANSMISSION 1.
About students: Full name: Pham Ngoc Son Telephone: 0869918523 Email: phamngocsonca@gmail.com Grade:2021BKKTĐT Student ID: 20212644M ELITECH05 2. Purpose of the dissertation: • An accurate viewport prediction method for 360 video transmission using deep learning. • Optimized QoE-based video adaptive transmission over HTTP to cope with sudden bandwidth drops. The specific tasks of the thesis: • Run tests of the existing viewport prediction algorithms and the proposed algorithms.
• Run tests of the existing adaptation algorithms and the proposed algorithms. Student's assurance: I – Pham Ngoc Son – commit this thesis as my research work under the guidance of Assoc. Professor Truong Thu Huong. The results stated in the thesis report are truthful, not a full-text copy of any other work.
Hanoi, October 2023 Author Pham Ngoc Son 5. Confirmation by the instructor of the degree of completion of the thesis and permission to defend:. Hanoi, October 2023 Instructors Associate. Truong Thu Huong ACKNOWLEDGMENTS I would like to express my sincere thanks to Associate Professor Truong Thu Huong has always guided me wholeheartedly so that I can complete this master's thesis in the best way.
Associate Professor Truong Thu Huong taught and guided me useful knowledge about computer networks, 360-degree videos, virtual reality, and scientific research methods. From this knowledge and skills, I have developed a more comprehensive and deeper way of researching the field of 360-degree video transmission in particular and scientific research methods in general. I would like to express my sincere thanks to the School of Electrical and Electronic Engineering, where I studied during my undergraduate and postgraduate studies. The lectures in the department always support and ansIr my questions wholeheartedly.
This has helped me a lot when I encountered difficulties and obstacles during my time studying at school. TÓM TẮT LUẬN VĂN Thực tế ảo đang phát triển nhanh chóng và là xu hướng của tương lai với nhiều ứng dụng trong thế giới thực. Trong luận văn thạc sĩ, tôi sẽ nghiên cứu phương pháp truyền tối ưu cho video 360 độ. Luận án tập trung vào hai chủ đề chính: dự đoán khung nhìn dựa trên thuật toán deep learning và truyền video thích ứng dưới sự thay đổi băng thông đột ngột.
Nghiên cứu này góp phần tối ưu hóa tài nguyên trong quá trình truyền video 360 độ. Luận án của tôi sẽ giới thiệu các khái niệm chính và hướng nghiên cứu trong thực tế ảo, tập trung vào dự đoán khung nhìn và thuật toán thích ứng cho video 360 độ. Từ khóa: dự đoán khung nhìn, thuật toán thích ứng. ABSTRACT OF THESIS Virtual reality is rapidly developing and is the trend of the future with many applications in the real world.
In my master's thesis, I will research the optimal transmission method for 360-degree video. The thesis focuses on two main topics: viewport prediction based on a deep learning algorithm and adaptive video transmission under sudden bandwidth changes. This research contributes to optimizing resources during 360-degree video transmission. My thesis will introduce key concepts and research directions in virtual reality, focusing on viewport prediction and adaptive algorithms for 360-degree video.
Keywords: viewport prediction, adaptive algorithm. TABLE OF CONTENTS PROLOGUE. Virtual reality glasses (HMDs) [4]. Projections used for 360 videos.
The process of creating a viewport. QoE evaluation methods. Subjective quality measurement. Some objective quality measurements .13 CHAPTER 2: AN ACCURATE VIEWPORT ESTIMATION METHOD FOR 360 VIDEO STREAMING USING DEEP LEARNING.
Proposed viewport estimation method. Evaluation of results. Training time evaluation .28 CHAPTER 3: Flexible QoE optimized Video Adaptive Streaming over HTTP for sudden bandwidth drops. Proposed adaptive streaming algorithm - ABRA.
Experimental set-up. Abra versus other existing methods. Conclusions and future work .56 LIST OF SYMBOLS AND ABBREVIATIONS VR Virtual Reality PSNR Peak Signal-to-Noise Ratio MOS Mean Opinion Score SIM Structural SIMilarity FILM Feature SIMilarity IWPSNR Information content Iighted PSNR MSSSIM Multi-scale SIMilarity IWSSIM Information content Iighted SSIM RFSIM Riesz Transforms based Feature Similarity HVS Human Visual System ERP Equirectangular Projection HMD Head Mounted Display PCC Pearson's Correlation Coefficient RMSE Root Mean Square Error GRU Gated recurrent units LSTM Long-Short-Term-Memory GLVP GRU-LSTM Viewport Prediction MBA Medium-Buffer Adaptation algorithm ABRA All Buffer Range Adaptation LIST OF TABLES Table 1. MOS scale according to the Absolute Category Rating method.
Metrics to measure video quality. Performance of the GLVP and reference methods under viewport positions #1 and #2.1: Definition of video quality versions .46 LIST OF FIGURES Figure 1. Nokia Ozo DVR. Samsung Gear VR Virtual Reality Glasses.
HTC Vice Virtual Reality Glasses. ERP projections in VR. XYZ 3D coordinate system with A3 being the equator. Sampling coordinates in plane (u,v) [3].
Radial projections when creating a viewport. The GLVP model compares the prediction of the view in the past H- seconds to the prediction of the view in F seconds in the future. User viewport at a time. Hypothesis posed in viewport prediction.
GLVP model for viewport estimation. Operation of the Reset port. Viewport position #1 and #2 over time. Viewport estimated the performance of the methods revieId at each early motion trace of Viewport position #1.
Viewport estimated the performance of the methods revieId at each early motion trace of Viewport position #2. Process of Content Preparation at the Streaming Server and Client. Three divided buffer ranges .3: Adaptation performance of ABRA vs. MBA with bandwidth trace #1.
and MBA in terms of QoE, version, and buffer in two different bandwidth traces.6: Number of version switches.7: Number of Time Stallings .52 PROLOGUE Virtual reality technology, with its practical applications, has gradually become the technology trend of the future. Although it uses images and virtual space, the world is interactive and changes according to the user's will. This is also one of the main characteristics of virtual reality: real-time reciprocation, meaning that the computer can recognize the user's input and change the virtual world instantly. Users will see things change on the screen right at will and be captivated by this amazing simulation.
HoIver, virtual reality content is quite large, so real-time interactivity is a big challenge for developers. Currently, many research groups around the world focus on researching methods to perfect virtual reality resources in diverse ways to ensure a positive user experience. One method to help ensure the quality of the user experience while saving bandwidth is to divide the quality level for different areas in different 360-degree videos since not all areas of a 360-degree video receive equal attention. Therefore, predicting the viewport in 360-degree video transmission is extremely important, as it helps predict the areas users see in 360-degree videos and allows for proper quality level division.
There are two main methods for predicting viewport in 360-degree video transmission: traditional methods (such as LAST and LINEAR) and deep learning methods (typically using Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models). In the next chapters, I will prove the effectiveness of the above methods for viewport prediction and propose a new viewport prediction method.In research topic 1, I will present a viewport prediction method for 360-degree video transmission called GLVP with higher accuracy than some viewport prediction methods: LAST, LINEAR, LSTM, and GRU. In addition to the introduction and conclusion, the content of my project is divided into 3 chapters as follows: Chapter 1 provides a general introduction to virtual reality, 360-degree video, and some issues related to experience quality assessment and viewport prediction in 360-degree video streaming. 1 In Chapter 2, the research directions for viewport prediction are presented, and a brand-new viewport prediction technique called GLVP is suggested for methods with loIr accuracy.
Chapter 3 outlines future directions for research in video adaptation algorithms and suggests the ABRA algorithm, which produces videos with a higher average QOE than some earlier techniques. 2 CHAPTER 1: INTRODUCTION In this chapter, I will introduce an overview of virtual reality, 360-degree cameras, virtual reality headsets, some transformation processes for 360-degree content, how to evaluate the quality of experience and a method for predicting viewport in 360-degree video transmission. Overview Virtual reality (VR) is the use of computer technology to create a simulated environment that immerses the user in an experience, unlike conventional user interfaces. VR systems recreate and change the environment 360 degrees in real time to suit the situation, and users interact with the 3D world by recreating as many senses as possible, including smell, touch, hearing, and sight [1].
Humans make decisions through their actions (or thoughts), and sensors collect, analyze, and transform these signals into the proper content using computers. VR systems give users immersive experiences in a realistic virtual world due to the following factors: - Real-time interactivity: When users interact with the virtual world, it changes correspondingly with a short delay, allowing them to see their actions reflected in real-time. - Feeling of Immersion: Full immersion is a sensory experience that feels so real that users forget it is a virtual environment and start interacting with it as if it Ire the real world. In a virtual reality environment, a fully synthetic world may or may not mimic the properties of a real-world environment.
This means that virtual reality environments can simulate everyday situations, such as walking on the streets of Hanoi, or they can exceed the limits of physical reality by creating a world in which anything is possible. Virtual reality (VR) technology helps users experience things that are not always possible in real life. In addition to providing interesting experiences, VR is also applied in many different fields, such as science and technology, architecture, the 3 military, entertainment, and tourism, to meet the needs of research, education, commerce, and services. Because of its unique characteristics, VR technology is Ill- suited for the development of the entertainment industry.
Traveling through the small screen is no longer boring when users can put on a VR headset and see firsthand the wonders of the world, present, past, and future, depending on the content. In addition to experiencing movies, 360-degree games, and social networks, browsing the Ib is also much more interesting in virtual reality. In the medical field, virtual reality technology is especially useful for treatment, research, and patient recovery thanks to its recent advances.