State Estimation in Autonomous Unmanned Aerial Vehicle Landing Submitted to the Department of Aerospace Engineering, Faculty of Transportation Engineering in partial fulfillment of the requirements for the degree of Master’s of Science in Aerospace Engineering at Ho Chi Minh City University of Technology (HCMUT) - Vietnam National University HCMC (VNU-HCM) Hoang Dinh Thinh July 2020 i Intentionally left blank ii Acknowledgements I would like to express my deep appreciation of the valuable comments from my ad- visors: Dr. Ngo Dinh Tri (Viettel Aerospace), Dr. Le Thi Hong Hieu (HCMUT); my father and mother for their endless care and nurturing. I also felt greatly indebted to my significant other, Tran Nhat Vy for providing me with a lot of support and my ophthalmologist, Huynh Vo Mai Quyen MD, whose talents, cordiality and graciousness have comforted me a lot during the difficult days of treatment.
I’m also grateful to my labmates: Nguyen Xuan Thanh Dat, Phan Minh Cuong, Huynh The Hai Nam, Tran Manh Hung, Huynh Tan Phuoc for their help with the black box and the experiments. I would also like to express my sincere thanks to Dr. Chen from Chinese University of Hong Kong, Dr. Yan Wan from University of Texas - Arlington and other anonymous reviewers for providing me with various insights concerning the shortcom- ings of my work.
Finally, I would like express my appreciation for the work of many scientists, medical professionals, the police and military force for their collective at- tempt to contain the coronavirus outbreak. My acknowledgement also goes to many scientists working on ophthalmic and neurologic conditions as their work brings hope and inspirations to many patients to keep carrying on. Amidst the coronavirus outbreak, Ho Chi Minh City, July 2020. Hoang Dinh Thinh.
iii Intentionally left blank iv Abstract This thesis summarizes two papers of my work on the localization problem of a rotary- wing type aircraft around a landing target, which involves estimation of the aircraft’s relative position with respect to the landing target by fusing data obtained from an Inertial Measurement Unit (IMU) and a visible-light RGB camera. This information is crucial to development of a robust and consistently-performing autonomous landing control algorithm - a key to autonomous UAV fleet operation, as well as in search and rescue. A unified approach comprising of three algorithms is presented with the first one designed to run when the landing target is visible to the camera and the other pro- vide localization information to the controller when the landing target temporarily moves out of view. In this manner, the landing process can be continuously performed, yielding consistent performance without cancellation or restart when the aircraft is per- turbed and moves away from the helipad.
The last algorithm is proposed for calibration of the accelerometer. For the first algorithm, a convolutional neural network was employed for recognition of the helipad in the image. Combined with attitude data obtained from the IMU, localization data can be inferred. This approach is indifferent to different helipad designs and landing target and capable of performing in real-time on a Raspberry Pi 3B computer.
Experiments show that while the estimation is noisy, it is unbiased and expressed decent agreement with estimation from the fiducal marker approach such as ARUCOTag in particular. For the second algorithm, we developed a Linear Time Varying (LTV) Kalman Filter for localizing the aircraft based on optical flow and IMU gyroscope and accelerometer data. Through transformation via virtual measurements, the LTV measurement model v allows a much more simplified analytical analysis compared to linearization of camera’s projection model in EKF approaches. We provided proof for a theorem that extends the result of contraction analysis for general Hamiltonian systems to encompass second- order dynamics, resulting in stability of the filter for assimilating acceleration, rather than velocity data directly through the process model.
The algorithm demonstrated excellent localization performance in the time window of 30 seconds without loop clo- sure techniques, light complexity compared to fiducial marker approaches and show agreement with monocular ORB-SLAM2, a state-of-the-art SLAM. We also developed a prototyping device and detailed the accelerometer calibra- tion process, with the development of a novel calibration algorithm that operates with several ARUCOTags. Although rigorous testing of this calibration method was not per- formed, the calibrated accelerometer has very little residual gravity and contributed significantly to estimation of the accurate scale in aforementioned localization algo- rithms. Keywords: localization, SLAM, estimation theory.
vi Commitment I hereby commit that this is my scientific work. The results are presented as-is and have never been published anywhere else, except on the papers that I authored. Un- der no circumstance did I intentionally claim facts from other people’s work without attributing them properly in the bibliography. Hoang Dinh Thinh.
vii Intentionally left blank viii Contents Contents ix List of Tables xiii List of Figures xv List of Symbols xvii Acronyms xix 1 Introduction 1 1.1 Unmanned Aerial Vehicles (UAVs) .2 Other potential applications .3 Statement of Problem, Scope and Limitations. 8 2 Foundations and Related Work 10 2.1 Navigation and Localization .2 Pin-hole camera model .4 Least Squares Problem .6 Simultaneous Localization and Mapping: an intuition .7 Pose Graph Optimization .2 Related work on Autonomous landing .3 Related Work on Visual odometry. 28 3 Visible Landing Target State Estimation 30 3. 40 4 Visual Inertial State Estimation with Linear Time Varying Kalman Filter 42 4.2 Contraction Analysis of Algorithm .2 Normal lighting condition .3 Low lighting condition .4 Variability with depth .5 Degradation of performance.
66 5 Implementation and Calibration 67 5.1 The ”black box” specifications .2 Calibration of IMU .6 Open-source release. 77 Bibliography 78 xi Intentionally left blank xii List of Tables 3.1 Validation result after 200th epoch .1 Specifications of two blacbox variants. 73 xiii Intentionally left blank xiv List of Figures 1.1 Proposed wireless charging solution .1 The projection schematic. Image replicated from [20].2 The projection schematic of one coordinate.
Image replicated from [20].3 Steps of Kalman Filter. Figure replicated from [28].4 Frames used in this thesis .5 An example of visual odometry .6 Memory plays an important role in odometry .7 A simple pose graph with 4 poses and 1 landmark .8 Color filtering and pattern matching for helipad detection .9 Effect of thresholding on edge extraction .10 Helipad suggested by [45] .11 YOLO algorithms predict bounding boxes of the objects [53] .1 Description of the Visible Landing Target State Estimation problem .2 The architecture of the helipad detector .4 Sample images from the training dataset .5 The training process of the custom YOLOv3 network .6 Detected helipad from the image with proposed network .7 Setup of the ARUCOTag and the helipad .8 Schematic of tag setup .9 Trajectory estimated by our algorithm and by ARUCOTag .10 Distribution of error between two estimation methods .11 The Euclidean norm of the distance error between two estimation methods 41 xv 4.1 Simulation with zc = 0.3 Structure of the Section 4.5 Sample frame from video stream .6 Filtered Euler angles from RTQF filter and raw acceleration as input of Section 4.7 Comparison of results between different estimation methods .8 Comparison of estimation between Panacea and ARUCO Tag pose in low light condition .9 Optical flow optimization in low light condition .10 Comparison of estimation between Panacea and ARUCO Tag pose at larger altitude .1 The black box’s interior .2 Additional calibration parameters added to the library .3 Gravitational acceleration before and after calibration. 74 xvi List of Symbols I RB Rotation matrix from body to inertial frame. RIB Rotation matrix from inertial to body frame.
R Rotation matrix from body to inertial frame. Ω Angular rates with respect to body frame. so Lie algebra of SO group. xvii Intentionally left blank xviii Acronyms Att Attitude.
DOF Degree of Freedom. EKF Extended Kalman filter. GL General Linear Group. GPS Global Positioning System.
IC Intial Condition. IC Ideal Condition. IMU Inertial Measurement Unit. KF Kalman filter.
LQR Linear Quadratic Regulator. MEMS Micro Electro-Mechanical Systems. MHE Moving Horizon Estimation. PD Proportional-Derivative.
PID Proportional - Integral - Derivative. xix xx ACRONYMS RP3 Real Projective Space (RP3). RPE Recursive Prediction Error. SAR Search and Rescue.
SO(3) Special Orthogonal Group 3. SoC System-on-chip. UAV Unmanned Aerial Vehicle. UKF Unscented Kalman filter.
VTOL Vertical Take-off and Landing. Chapter 1 Introduction This chapter introduces the significance of Unmanned Aerial Vehicles (UAV) with regard to their potential economic and social applications, the technical challenges related to control and intelligent decision making, as well as why autonomous landing holds the key to autonomous UAV fleet operation. We will also discuss about the scenarios which precise autonomous landing is necessary and contribution of our work to the literature of autonomous UAV landing.1 Unmanned Aerial Vehicles (UAVs) Unmanned Aerial Vehicles (commonly abberivated as UAV) belong to a type of robots that operate in the air. According to Skybrary [1], unmanned aircraft are meant to be operated without pilot on board, but the definition is vague about whether the aircraft can operate autonomously or remotely by a human being.
As a consequence, two types of UAV are available, both of which belong to the Unmanned Aerial System (UAS) which typically consists of three components: • An intelligent agent (a human being or a computer), if the UAV is remotely controlled. • An Unmanned Aerial Vehicle. UNMANNED AERIAL VEHICLES (UAVS) • A command and control system (C2), or sometimes, communication, command and control system (C3) that establish the connection between the former two components [1]. In case the aircraft can autonomously operate without external communication, the UAV is said to be fully autonomous.
In all military and civilian circumstances, full autonomy is highly desired as it can significantly decrease the need of operating man-power, enabling large-scale operation, as well as diminishing the operating cost. UAVs have found tremendous applications in precision agriculture [2], surveillance [3], search and rescue [4], facilitation of communication - broadcast, package delivery and transportation [5], remote sensing and infrastructure inspection [6]. In this work, we focus on the rotary-wing type UAV, an specifically for illustrative purposes, we will consider quadrotors. These UAVs are very versatile because of its Vertical Take-off and Landing (VTOL) capability, ease of manufacturing, good stability characteristics in comparison with trirotor types, and wide availability on the market.
The foundation of UAV autonomy lies in three important components that need to be addressed in a united framework: 1. Automatic Control: the UAV must possess the ability to track prescribed path with great accuracy, and safely operate in various operating regimes. Navigation and Obstacle Avoidance: typical UAV relies on GPS and IMU for localization of itself. Consequently, indoor as well as urban, cluttered environments such as forests may complicate the localisation process.
The UAV must possess the ability to ”sense” the surrounding, perform path planning that avoid dangerous maneuvering near obstacles. Intelligent decision making: Based on the sensed data, the UAV must make a decision to take which appropriate action or make a recommendation that may result in the environment transition to a better state. The first factor is considered a mature topic in automatic control, in which a good review can be found in [7]. The third factor is specific to each individual mission and has become an interesting and growing topic in artificial intelligence, e.
INTRODUCTION an example on how UAV can cooperatively detect fire on a large area. For the second factor, progresses in computer vision and state estimation techniques have led to prolif- eration of ”visual odometers” that reliably estimate the localisation information of an aircraft navigating in indoor cluttered environments. Navigation scenarios also include autonomous take off and landing. While the former is pretty straightforward and easy, the latter can be challenging since precision could be crucial to safety.
Moreover, an aircraft performing landing may have additional constraints about time and maximum attempts due to limitation in battery power.1 Economic motivation As e-commerce continues to make triumph, the logistics industry has witnessed a larger than ever growing momentum.