Study on New Approaches for Vehicle Detection Using Stereoscopic Information Advisor Prof. Shozo KONDO Course of Science and Technology Graduate School of Science and Technology Tokai University Le Thanh Sach Abstract Vision-based vehicle detection is a useful, active, and challenging research field. It is helpful for many applications, for example, Intelligent Transportation Systems and Robot Vision. Generally, a detection framework consists of two steps; one step is to generate candidate vehicles’ regions from input images, and the other step is to verify the generated candidates.
For the verification step, many existing researches demonstrated that not only vehicles can be discriminated from the background scene but also vehicles can be classified into hierarchical sub-classes. For example, vehicles can be classified into Car, Bus, and Truck; and then Car can be subdivided into Honda, Toyota, and so on. However, efficient and reliable techniques to obtain candidate vehicles’ regions are still missing. In addition, estimating the distance from the detected vehicles to the host vehicle is also another challenge.
In order to solve those problems, we propose to use stereoscopic information for vehicle detection in this thesis. The proposed vehicle detection approach accepts stereo images that are captured by using stereo cameras as its input, and it performs three main tasks as the following. First, disparity images are computed from the input stereo images. There are two challenging problems in this task, reducing the computation time and increasing the quality of disparity images.
In order to solve these two problems, we propose new approaches that can efficiently and reliably perform stereo matching. Second, ground planes are estimated from the computed disparity images. For this task, we propose a method that is named as Dynamic Programming Based method for ground plane estimation. The proposed method is able work in the case that roads are not flat.
In this task, U- and V-disparity images are reliably produced from the disparity images as well. Finally, on-road vehicles are located in the input images by combining the estimated ground planes and the U- and the V-disparity images by a proposed differential-based method. The distance and the location of the located vehicles with respect to the stereo cameras coordinate are derived from stereo camera parameters and the disparities of the located vehicles. Experimental results on both of artificial stereo images and real stereo images demonstrate that disparity images can be efficiently and reliably computed even in the low texture condition and that vehicles can be straightforwardly located by the proposed differential-based method.
Contents Abstract i List of Figures v List of Tables vii Glossary ix 1 Introduction 1 1.3 Fusion of Sensors .3 Scope of the Thesis .4 Structure of the Thesis .2 Pixel-Matching Computation. 34 ii Contents iii 2.4 Efficiency of Stereo Matching .5 Stereo Matching Evaluation .6 Stereo Matching Summary .2 Stereo-Based Ground Plane Estimation .3 Hough Transform Based Method .4 Fitting-Based Method .1 Least-Squares Method .2 Iteratively Reweighted Least-Squares Method .5 Dynamic Programming Based Method .6 Parallel-Based Method .7 Polar Histogram Based Method .8 Ground Plane Estimation Summary .3 Stereo-Based Vehicle Detection .2 U- and V-Disparity Images Combination .4 Vehicle Detection Summary. 48 3 A New Coarse-To-Fine Method for Computing Disparity Images 49 3.1 Outline of the Proposal .2 Sampling of Disparity Spaces .3 Coarsest Map Estimation .4 Active Region Determination .6 Disparity Map Computation. 61 4 Robust Approaches for Stereo Matching 62 4.1 Outline of the Proposal .2 Pixel-Matching Cost Computation .1 Relationship: cost volumes and edge maps .4 Disparity Image Computation.
82 5 Stereo-Based Ground Plane Estimation and Vehicle Detection 84 5.1 Outline of the Proposal .2 Ground Plane Estimation .1 V-Disparity Image and V-D Cost Image Computation .2 DP-Based Estimation .3 Ground Plane’s Parameters .1 Vehicle’s Disparity and Lower Mark Determination .2 Upper Mark Determination .3 Left and Right Mark Determination .1 Creating Vehicle-Disparity Image .2 Creating Density Histogram .3 Computing Differentials .4 Generating Pairs of Left Mark and Right Mark .5 Merging Pairs of Left Mark and Right Mark .4 Vehicle Verification. 104 6 Experimental Results and Discussions 105 6.1 Outline of the Experiments .2 A New Coarse-To-Fine Method for Computing Disparity Images .3 Robust Approaches for Stereo Matching .4 Ground Plane Estimation .1 Summary of Results .2 Future Research Directions. 127 Bibliography 129 Acknowledgements 143 List of Figures 1.1 Overview of existing approaches.2 The demonstration of Inverse Perspective Mapping.1 An example of stereo cameras.2 Geometry of stereo cameras.3 Ground Plane in Disparity Space.4 Demonstration of U- and V-disparity image combination.1 The coarse-to-fine method’s idea.2 An active region’s example.3 Outline of the proposed method.4 An original image and its low resolution image.5 Matching low resolution stereo images.6 The sampled disparity space for the coarsest map.8 Modified active region.9 Description of active region.1 Examples of homogeneous regions.2 The proposed stereo matching system.3 Horizontal correspondence between cost volumes and edge maps.4 Vertical correspondence between cost volumes and edge maps.5 Examples of edge-segments and mid-segments.6 Proposed data structures.1 Examples of V-Disparity image and V-D cost image.2 Example of the ground plane’s profile when the road is empty.3 Example of the ground plane’s profile when there is a large on-road vehicle.4 Examples of Ground Plane’s Parameters .5 Five parameters of a frontal vehicle.6 An example of a D-costs array.7 An example of a V-costs array.8 Determination procedure of left and right marks.9 Illustration for the left and the right mark determination .10 Vehicle-disparity image formulation.11 Fitting to the vertical edge on the vehicle’s rear view. 101 v List of Figures vi 6.1 Qualitative illustration of the proposed coarse-to-fine method 1 .2 Qualitative illustration of the proposed coarse-to-fine method 2 .3 Quantitative comparison for the computation time.4 Quantitative comparison for the accuracy of disparity images.5 Qualitative comparisons for the artificial and the real sequence.6 Quantitative comparisons for the artificial sequence.7 Quantitative comparisons for the artificial and the real sequence.8 Classification of road pixels.9 Artificial Sequence, examples of ground planes’ parameters.10 Real Sequence, examples of ground planes’ parameters.11 Artificial Sequence, ground planes’ parameters.12 Artificial Sequence, Comparison of ground planes’ parameters.13 Real Sequence, ground planes’ parameters.14 Illustration of ground plane estimation.15 Examples of vehicle detection.
125 List of Tables 1.1 List of major research groups in the field.2 List of major conferences and transactions in the field.1 Outline of cost computation.1 Cost aggregation guided by edge maps: pseudo code.2 Cost aggregation guided by edge maps: explanation.1 Coarse-to-Fine Method: Parameters used to compute disparity maps 106 6.2 Robust Approaches: Parameters used to compute disparity images.3 Parameters used for vehicle detection. 123 vii To my parents, my wife, and Ms. They always give me endless love and strong motivation. viii candi marklef t and candi markright : Can- didate left mark and candidate right mark of detected vehicles.
candi marklower and candi markupper : Candidate lower mark and candi- date upper mark of detected vehi- Glossary cles. cost(v, d) : V-D cost image. dinf : The disparity of objects at the in- ΔU , ΔV , Δd : Sample steps in U, V, and D- finity. direction in disparity spaces.
dvhd : The disparity of detected vehicles. Caggr (u, v, d) : A 3D cost volume that densi(u) : Density histogram. contains aggregated pixel-matching costs. Caggr (u, v, d) can be either dif f (u) and dif facc (u) : Differentials and L R Caggr (u, v, d) or Caggr (u, v, d) Accumulated differentials respec- L tively.
Caggr (u, v, d) : A 3D cost volume that contains aggregated pixel-matching costs. In Ivdi (u, v) : Vehicle-disparity image. the computation, the reference im- age is the left image of the input pair Ivd (v, d) : V-disparity image. of stereo images.
map(u, v) : Disparity image. map(u, v) rep- R Caggr (u, v, d) : A 3D cost volume that contains resents for either mapL (u, v) or aggregated pixel-matching costs. In mapR (u, v) the computation, the reference im- mapL (u, v) : Left disparity image. age is the right image of the input pair of stereo images.
mapR (u, v) : Right disparity image. Cpix (u, v, d) : A 3D cost volume that contains marklower , markupper , marklef t , and markright pixel-matching costs. L Cpix (u, v, d) : A 3D cost volume that contains pathDP : The path in V-D cost image that is pixel-matching costs. In the com- discovered by the proposed dynamic putation, the reference image is the programming based method.
left image of the input pair of stereo pathgp DP : The ground plane’s profile that is images. extracted from pathDP. R Cpix (u, v, d) : A 3D cost volume that con- pathgp LF : The ground plane’s profile that tains pixel-matching costs. In the is obtained by fitting pathgp DP to a computation, the reference image is straight line.
the right image of the input pair of stereo images. WU × WV : The size of aggregation windows. ix D : The width of disparity search SSE : Streaming Single Instruction Mul- range. tiple Data Extensions.
DP : Dynamic Programming. U and V : The width and the height of input FPGA : Field Programmable Gate Array. GPU : Graphic Processing Unit. WTA : Winner-Takes-All, a method for IPM : Inverse Perspective Mapping.
determining the disparity of pixels. MAE : Mean of Absolute Errors.1 Motivation Vehicle Detection is a research field of Intelligent Transportation Systems (ITS). Its aim is to develop techniques and technologies that are able to identify vehicles and obstacles in front of host vehicles and to estimate the distance from the vehi- cles and the obstacles to the host vehicles. The vehicles and the obstacles in front of the host vehicles are referred to as the frontal vehicles and the frontal obstacles respectively in this thesis.
Terminologies ”preceding vehicles/obstacles” are in- terchanged with ”frontal vehicles/obstacles” to avoid the boring repetitious. The techniques and the technologies developed in Vehicle Detection enable the host ve- hicles to sense the surrounding environment, so they are important and useful for many applications, for example, intelligent vehicles, autonomous vehicles, robot navigation systems, traffic management systems, and so on. Vehicle Detection and and its related researches in Intelligent Transport Systems are currently active research fields, as demonstrated by a large number of partic- ipants and research groups around the world. Some major groups are listed in Table 1.
Research groups proposed different ways to sense the environment in front of host vehicles. Some groups used active sensors, e., Radar and LiDar. Meanwhile the other groups used passive sensors, i., normal cameras, in combi- nation with computer vision techniques. Especially, the research group in the first row of Table 1.1 demonstrated that frontal obstacles can be detected by using only vision information and that the distance from the detected obstacles to the host vehicles can be also estimated.1: List of major research groups in the field.
Name Explanation PARMA This research group is from Parma University, Italy. The re- searchers in this group are the founders of GOLD - Generic Ob- stacle and Lane Detection System. This group was also responsible for building the stereovision system of TerraMax which is an au- tonomous vehicle participating to the DARPA Grand Challenge. CMU This is one the largest and earliest research group in the field, it is from Carnegie Mellon University, USA.
It is well-known with autonomous vehicles named NavLab. The researchers in this group is the founders of SCARF (Supervised Classification Applied to Road Following), UNSCARF (UNSupervised Classification Applied to Road Following), YARF (Yet Another Road Follower), and so on. In the DARPA Grand Challenge, their team (Red team) was one of the leaders in the competition. LIVIC This group is from INRETS, France.
The researchers in this groups are the founders of the terminology V-Disparity Image. Vehicle Detection is strongly encouraged by academe and industry as well. In academe, the research is supported by a system of special conferences and trans- actions, as shown in Table 1.2, and by a crowded competition, e., the DARPA Grand Challenge1 .
