Improving communication efficiency of Federated Learning systems Dinh Thi Quynh Khanh Department of Computer Vision, MICA Hanoi University of Science and Technology Supervisor Assoc. Le Thi Lan In partial fulfillment of the requirements for the degree of Master of Computer Science August 4, 2023 Acknowledgements First of all, I would like to express my sincere gratitude towards my advisors Assoc. Professor Le Thi Lan and Assoc. Professor Tran Thi Thanh Hai.
I still remember the first day I came to the office to see them to start my master study and now that I look back to the day Professor Le Thi Lan assigned me to study the exciting topic of Federated Learning, I realize what a wonderful journey has it been, going from zero knowledge in the area to building a tool that can help research community and practitioners. I’ve learned from my advisors many valuable lessons to become a better problem solver. I realized that I need to value all of my (failed) attempts, imperfect initial works and see them as a starting point. My advisors have taught me that failures are just temporary and instead of being discouraged by them, use them as a feedback to polish my work and hence, move myself forward.
Thanks to them, I see the beauty of research and finally decided to pursuit a PhD in the field; but more importantly, I truly know who I want to become and how I could contribute to our community with my works. I would also like to thank every member at the Comvis laboratory, I did have so much joy and memories there, they have always made me felt welcomed. I could not ask for better fellows. Finally, I would love to thank my family for their support and encourage- ment throughout my years of study and through the process of researching and writing this thesis.
This accomplishment would not have been possible without them. From the bottom of my heart, thank you! This material is based upon work supported by the Air Force Office of Scien- tific Research under award number FA2386-20-1-4053 and Hanoi University of Science and Technology (HUST) under project number T2021-SAHEP- 003. Abstract In the recent years, Computer Vision has observed a revolution in differ- ent tasks such as object detection, object classification, action recognition. Much of this success is based on collecting vast amounts of data, often in a privacy-invasive manner.
Federated Learning (FL) is a new discipline of machine learning that allows training models with the data being de- centralized, i.e instead of sharing data, participants collaboratively train a model by only sending weight updates to a server. Therefore, FL can help to protect data privacy and utilize the computing power from the huge number of endpoint devices. This motivates us to investigate the use of FL in computer vision tasks. However, the SOTA methods for computer vision tasks mainly base on deep learning models that require to train the models with a huge number of parameters.
Therefore, training models for computer visions tasks in a federated manner faces several obstacles. Firstly, FL often yields a significant amount of communication overhead than centralized learning, since the model parameters are required to be exchanged between clients and central server during the training process. Secondly, the study of FL for computer vision tasks requires running ex- periments with complicated programs, as FL is more sophisticated than conventional training in terms of implementation, making the field some- times exclusive to researchers, engineers with decent engineering skills only especially for human action recognition tasks (HAR). Therefore, in this thesis, we attempt to push forward the development of FLy for computer vision by solving these two challenges.
To mitigate the first issue, we propose a model weight compression and encoding during model uploading for Federated Averaging (FedAvg) - a widely used algorithm in FL. Our weight compression is inspired by Sparse Ternary Compression algorithm with a modification to be applicable to FedAvg. We also utilize compressed weights’ characteristics to encode them hence the communication cost can be reduced. The proposed method has been evaluated on one task of computer vision that is image classification.
Experimental results on MNIST dataset demonstrate that our method is able to reduce the communication cost without considerably worsening the model accuracy. Regarding the second challenge, although recently some FL frameworks have been developed to facilitate the study and application of FL in some specific tasks, unfortunately, those frameworks are limited to testing deep models on object classification task. Therefore, we introduce a novel frame- work, called FlowerAction, to study FL-based human action recognition from video data. To the best of our knowledge, this is the first FL framework for video-based action recognition.
FlowerAction is built upon the Flower framework, it incorporates various techniques, including data loading, data partitioning, widely used network architecture for HAR (e. SlowFast, I3D, R3D), and FL aggregations (e. FevAvg, FedBN, FedPNS, and STC). First, we present the main components of FlowerAction, which are devel- oped on top of the existing Flower framework to interface with the external components (data loaders, deep models) and internal ones of Flower (model communication and aggregation algorithms).
We then demonstrate the ef- fectiveness of deep models and FL algorithms in recognizing human actions using benchmark datasets (e. HMDB51 and EgoGesture) for both sim- ulation and real distributed training. Our experimental results show that the FlowerAction framework operates properly, which helps researchers to approach FL with less effort and conduct federated training benchmarks quickly. Furthermore, our analysis results that show performances in terms of top-k accuracy and communication cost of different FL algorithms could give instructive suggestions for the selection and deployment of a deep model for FL-based action recognition in the future.
Contents List of Acronymtypes xii 1 Introduction 1 1.2 Problem definition .4 Outline of the thesis .1 Communication-efficient Federated Learning .2 Video-based human action recognition (HAR) and federated learning for HAR .1 Deep learning models for HAR .2 Federated learning for HAR .3 Federated learning frameworks. 20 3 Communication cost reduction using sparse ternary compression and encoding for FedAvg 21 vi CONTENTS 3.1 Layer-wise sparse ternary compression .2 Model weight encoding. 31 4 FlowerAction - A Federated Learning framework for Human Action Recognition 33 4.2 Workflow of FlowerAction framework .4 Deep learning models for human action recognition .3 Inflated 3D ConvNet - I3D .1 Data pre-processing .6 Characteristics of FlowerAction .1 Human action recognition datasets .2 Implementation and setup. 73 vii CONTENTS 5 Conclusions 74 5.1 Summary of achievement and limitations.
75 References 90 viii List of Figures 1.1 Next word prediction application in mobile phones [1] .1 The overall architecture of FedVision [2] .2 The overall architecture of Flower [3] .3 Experiments on edge devices are supported by Flower [3] .1 Overview of the proposed method. (1) Server sends the global model to clients; (2) Clients update the model with their local data; (3) Clients apply the proposed weight compression and encoding techniques; (4) Clients send local models to the server (5) Server updates the global model.2 Samples from MNIST handwritten digit dataset.3 Network architecture used in our experiments for MNIST classification.4 Accuracy at different values of compression factor.5 Relationship between communication cost and compression factor.6 Confusion matrix at p = 73% with non-IID data distribution.7 Confusion matrix at p = 78% with IID data distribution.1 FlowerAction’s main flow. Notes that dashed blocks indicates work- in-progress.4 FlowerAction’s data pipeline .5 IID data partition in FlowerAction. 52 ix LIST OF FIGURES 4.6 Programming interface of FlowerAction’s client .7 FlowerAction’s Client API .8 Programming interface of FlowerAction’s server .9 FlowerAction’s Server API .10 Some examples from HMDB51 dataset [5] .11 Some examples from EgoGesture dataset [6] .12 Number of samples per class of client 0 and client 10 from HMDB51 in non-IID data partition, C = 20, f rac = 0.13 Number of samples per class of client 0 and client 5 from EgoGesture in non-IID data partition, C = 8, f rac = 0.14 Top-1 accuracy of FedAvg, FedBN and FedPNS on HMDB51 with sim- ulated clients .15 Top-1 accuracy of FedAvg, FedBN and FedPNS on EgoGesture with simulated clients .16 Number of nodes kept after Optimal Aggregation on HMDB51, C = 70, f rac = 0.17 Number of nodes kept after Optimal Aggregation on HMDB51, C = 70, f rac = 0.1, non-IID data .18 Cumulative communication cost of FedAvg and STC (p = 0.04) on HMDB51 through the rounds .19 Top-1 accuracy by epochs of FedAvg, STC with p = 0.20 t-SNE visualization on HMDB51 test set with model trained with FedAvg 71 4.21 t-SNE visualization on EgoGesture test set with model trained with FedAvg .22 t-SNE visualization on HMDB51 test set with model trained with STC through the rounds.
72 x List of Tables 3.1 Parameters used in the experiments.2 Compression factor, communication cost and accuracies obtained in two scenarios: IID and Non-IID of the proposed method.1 Centralized training configurations .2 Top-1 accuracy (%) on HMDB51 and EgoGesture datasets in the literature 64 4.3 Top-1 accuracy from simulated federated training on HMDB51.4 Top-1 accuracy from simulated federated training on EgoGesture.5 Framework-based experiments results with FedAvg .6 Framework-based experiments results with STC. 69 xi List of Acronymtypes C3D Convolutional Three Dimensional. CNN Convolutional Neural Network. DL Deep Learning.
FedAvg Federated Averaging. FedBN Federated BatchNorm. FedPNS Federated Probabilistic Node Selection. FL Federated Learning.
HAR Human Action Recognition. IID Independent and identically distributed. LSTM Long short-term memory. NN Neural Network.
RNN Recurrent Neural Network. STC Sparse Ternary Compression. SVM Support Vector Machine. xii Chapter 1 Introduction 1.1 Motivation Artificial Intelligence (AI) is developing at an unprecedented rate with the rise of Deep Learning (DL): several new exciting research papers are coming out daily and aston- ishing applications that demonstrate an intelligence that are close to human level.
The success of deep learning comes primarily from the availability of large data and com- putation power. However, one of the main challenges that DL models currently face in this digital era is the privacy of data, since these data can be very sensitive and private that their owners are usually note willing to share them, which hinders the process of collecting datasets for DL. Therefore AI needs to ensure that the privacy of users’ data is not violated, and the study of training paradigms that protect data’s confiden- tiality becomes more and more demanding. Beside that, with the rapid evolution of technology in general, edge devices is now more capable of computation power.
One question rises in that situation: “Can we train the model without gathering all the data to a central server?”. Fortunately, Federated Learning (FL), which is the study of algorithms to allow several machines to collaboratively train a model is the answer. With FL, not only that the privacy of user data is protected, but the computational power of a sheer number of devices is also fully utilized [7]. FL, first introduced by scientists at Google in 2017 [8], is a machine learning ap- 1 1.1 Motivation proach that allows data to remain on local devices while a global model is trained.
It is a decentralized approach to machine learning, where multiple devices collaborate to train a shared model without directly exchanging raw data. The idea behind FL is to address the challenge of training machine learning models on sensitive or confidential data that can not be transferred to a central server. This approach enables organizations to train models on data that is distributed across var- ious devices or servers, without transferring the data to a centralized server. Instead, only the model updates are sent from the devices to a central server where they are aggregated to improve the global model.
FL has several advantages over traditional centralized machine learning approaches and FL methods play a critical role in supporting privacy-sensitive applications where the training data are distributed at the edge devices.