MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY. MASTER’S THESIS BUILDING A PLATFORM FOR MANAGING, ANALYZING, AND SHARING BIOMEDICAL BIG DATA Master student: Supervisors: Dao Dang Toan Dr. Nguyen Thanh Huong Assoc-Prof. Dao Trung Kien A thesis submited in fulfilment of the requirements for the degree of Master of Science in the Pervasive, Space and Interaction Department International Research Institute MICA Hanoi – 2021 CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập – Tự do – Hạnh phúc BẢN XÁC NHẬN CHỈNH SỬA LUẬN VĂN THẠC SĨ Họ và tên tác giả luận văn: Đào Đăng Toàn.
Để tài luận văn: Xây dựng nền tảng quản lý, phân tích, và chia sẻ dữ liệu lớn y sinh học. Chuyên ngành: Khoa học máy tính. Mã số SV: CB190241. Tác giả, Người hướng dẫn khoa học và Hội đồng chấm luận văn xác nhận tác giả đã sửa chữa, bổ sung luận văn theo biên bản họp Hội đồng ngày 21/09/2021 với các nội dung sau: - Yêu cầu sửa luận văn, bố cục lại và làm rõ hơn các đóng góp của mình, tham khảo ý kiến chỉnh sửa của thầy cô hướng dẫn.
- Bổ sung các kết quả đã có vào luận văn. Ngày 18 tháng 10 năm 2021 Giáo viên hướng dẫn Tác giả luận văn CHỦ TỊCH HỘI ĐỒNG ĐỀ TÀI LUẬN VĂN Tên đề tài (tiếng Việt): Xây dựng nền tảng quản lý, phân tích và chỉa sẻ dữ liệu lớn y sinh học. Tên đề tài (tiếng Anh): Building a platform for managing, analyzing and sharing biomedical big data. Giáo viên hướng dẫn Acknowledgement It is an honor for me to write thankful words to those who have been supporting, guiding and inspiring me from the moment, when I started my work in Vingroup Big Data Institute and International Research Institute MICA, until now, when I am writing my master thesis.
I owe my deepest gratitude to my supervisor, Dr. Nguyen Thanh Huong. Her expertise, understanding and generous guidance made it possible to work in a new topic for me. She has made available her support in a number of ways to find out the solution to my works.
It is a pleasure to work with her. I would like to show my gratitude to Assoc Prof. Dao Trung Kien and all of members of the Pervasive Space and Interaction Department for their guidance which help me a lot in how to study and to do research in right way, and also the valuable advice for my works. Special thanks to Dr.
Vo Sy Nam and my colleagues at Vingroup Big Data Institute for their support. Their suggestions enable me to keep my thesis in the right direction. Finally, this thesis would not have been possible if there were no encouragement from my family and friends. Their words give me power in order to overcome all the embarrassment, discouragement and other difficulties.
Thanks for everything in helping me to get this day. Abstract With the advancement of hardware and software technologies, the data explosion in biomedical research and in healthcare systems in recent years has required urgent solutions for managing, analyzing and sharing data. In particular, research in omics science is moving from a hypothetical approach to a data- driven approach. Additionally, the healthcare industry has always required tighter integration with biomedical data to promote personalized medicine and deliver better treatments.
However, dealing with the huge amount of information generated every day requires complex solutions. Many solutions from hardware to software are born to solve the problem of big data such as high-performance computing solutions (HPC) or solutions that utilize distributed computing and storage systems (Spark, Hadoop). Recognizing the challenges in managing biomedical data, we leveraged existing technologies to build a data management, analysis and sharing system that we call MASH. Hanoi, October 18th 2021 Dao Dang Toan Table of Contents _Toc85361781CHAPTER 1.2 System’s Main Objective .2 Non-functional Requirements.
THEORETICAL BACKGROUND ON MASH CONSTRUCTION .1 Distribution of Data Samples .2 System Input Files .1 The FASTQ Format .2 The SAM/BAM Format .3 The VCF Format.3 Big Data Technologies .6 Distributed Object Storage .1 Cloud-based Computing. MASH SYSTEM DESIGN AND DEVELOPMENT .1 Graph Data Model .2 Document Data Model .3 Overall Architecture of the System .1 Overview of System Architecture. SOLUTIONS TO SPEED UP DATA INSERTION AND QUERYING .3 Application of genetic algorithm in optimal parameter selection .1 Introduction to Genetic Algorithm. MASH CONSTRUCTION RESULTS .2 Result of Parameter Optimization by Genetic Algorithm.
33 CONCLUSION AND PERSPECTIVES. Research Questions and Outcomes. Contributions and Perspectives. 41 List of Figures Figure 2.3: Data Warehouse Overview .4: Distributed Object Storage System Architecture [8] .1: MASH data model .2: MASH system architecture .3: Layer diagram – MASH system architecture .4: System authentication and authorization architecture .5: Workflow service architecture .1: Data insertion steps .2: Flat data type .3: Nested data type.4: Support data analysis and search by selecting filter options .5: Querying data interface .6: Representation of a parameter set.7: Specific value of a parameter set .8: Genetic Algorithm flow chart for Parameter tuning [30] .1: Performance of data insertion phase.
37 List of Tables Table 4.2: Parameters for tuning .1: Configuration parameters of the server in the test environment.2: Result of parameter optimization by Genetic Algorithm. 36 List of Abbreviations MASH Management, Analysis, Sharing and Harmonization FAIR Findable, Accessible, Interoperable, Reusable the Database of Genomic Variants for Vietnamese population DGV4VN project VM Virtual Machine DDoS Distributed Denial of Service CI/CD Continuous Integration/Continuous Delivery BCL Base CalL SAM Sequence Alignment Map BAM Binary Alignment Map VCF Variant Call Format GUID Globally Unique IDentifier DNA DeoxyriboNucleic Acid RAM Random Access Memory ID IDentifier ETL Extract, Transform, Load CRUD Create, Read, Update, and Delete DDBJ DNA Data Bank of Japan OSDC the Open Science Data Cloud OCC the Open Commons Consortium NCI The National Cancer Institute GDC Genomic Data Commons AAA Administration, Authorization, and Authentication CWL Common Workflow Language HyperText Transfer Protocol/HyperText Transfer Protocol HTTP/HTTPS Secure I/O Input/Output CPU Central Processing Unit SSD Solid-State Drive HDD Hard Disk Drive SNV Single-Nucleotide Variant LncRNA Long non-coding RNAs CCR ConCurrent Requests SQL Structured Query Language GA Genetic Algorithm CHAPTER 1.1 Motivation This thesis focuses on some of the problems that research projects on the human genome are facing. To better understand these issues, let's look at following interesting story of research projects on the human genome: ● In the 2000s: The first human genome project was completed after 13 years and approximately 1000 scientists were involved. That project had spent more than 3 billions USD to decode the first human genome.
And it has a very high impact on the genomics field. The completion of this project was a big science event at that time. ● And today, thanks to the technology development, to decode a human genome, it takes only a few days with approximately 1000 USD. ● In the near future, maybe in the next few years, we only have to spend no more than 100 USD and a few hours for one human genome decoding.
The cost of decoding a human genome has been dropping rapidly. This means genomics research will generate a huge amount of data. And it raises the challenges in data analysis, and management. Those challenges continue to grow in the future.
Many projects/systems were established to address the above problems. And the Database of Genomic Variants for Vietnamese Population Project (DGV4VN) which was funded by Vinbigdata is one of those projects, in this project we built a big biomedical data platform which named MASH to solve the challenges of big data Management, Analysis and Sharing biomedical big data.2 System’s Main Objective At the time we started building the MASH system, we had a requirement to build a system to hold over 1,200 terabytes of data. We need to share a part of that amount of data to our partner and research community. And that amount of data is growing quickly over time.
So it is necessary to build a scalable system, which can store a very big continuous growing data, and that data must be Findable, Accessible, Interoperable, and Reusable. One of the biggest challenges of every big data platform is how to ensure the performance of the system. And high performance is also one of the goals for the MASH.1 Functional Requirements MASH comprises of four main functional groups, including management, analysis, sharing and visualization of biomedical data. The specific functions are set out as follows: Integrate and manage data from various projects ranging from the ongoing DGV4VN project to data obtained from future projects such as cancer genomics, other health data, or projects of different fields.
1 Allow updating and retrieving data based on specific data models of particular projects, allow experts to collaborate with system developers to describe data models in compatible format, allow updating results of specific analysis workflows into the system in accordance with the predefined data models. Be able to perform management, analysis, sharing and harmonization of big data sources up to petabytes or even exabytes. Data will be stored in the form of object-storage in on-premises data storage or cloud-based data storage, and the system has appropriate components for data access including databases. Have an access control mechanism suitable for project data sources and ensure security issues specific to each data field, including personal data privacy in biomedical discipline.
Provide appropriate data retrieval and display interfaces for the public so that users can use the system in downstream analysis as well as provide advanced data search and query interfaces for biomedical experts. Be able to integrate typical analysis workflows of each project into the system and allow automatic run of multiple processing tasks from thousands to tens of thousands of inputs simultaneously. The above functional requirements are the core requirements that the system needs to fulfill in order to be able to solve the challenges of a big data management system. However, to create a system that is easy to operate and expand in the future, it is indispensable for the non-functional requirements that are listed in the following section.2 Non-functional Requirements The non-functional requirements below are those for most distributed systems.
However, with the MASH system and other data management systems with large sizes and values, FAIR (Findable, Accessible, Interoperable, Reusable) is placed first. High performance: The system must be able to receive and process a large amount of data files as well as analysis pipelines in short response time. Scalability: The system can be easily scaled both horizontally (adding containers, VM) and vertically (adding hardware resources to the system) without interfering with the running service. Portability and Backward compatibility: The system can easily be switched from place of deployment without taking much time to reconfigure parameters.
The system could run service without interruption in updating software versions. 2 Stability, high availability, and maintainability. Backup services and high availability need to be implemented to ensure that the system services are not affected when main services’ fault occurs. System security (Security): The system is resistant to attacks such as DDoS or SQL injection, etc.
Ensure that the system data cannot be accessed by unwanted third parties. User interface should be friendly and usable (Usability): user interface should be designed to be user-friendly targeting doctors, bio-informaticists, and biologists. Error monitoring and reporting: deploy services to monitor system resources and health, send alerts and emails to teams when errors occur, or the system resources hit a threshold. Logging: ensures the entire system activities are logged, centrally manages logs for services, visualizes logs of services in the system, making debugging easy.
Automatic integration and delivery: CI/ CD must be implemented to ensure that a continuous system is automatically examined, integrated, and delivered. Non-functional requirements are clearly demonstrated through the design and implementation of the system.
