Automatic Plant Annotation Using 3D Computer Vision

In this thesis 3D reconstruction was investigated for application in precision agriculture where previous work focused on low resolution index maps where each pixel represents an area in the field and the index represents an overall crop status in that area. 3D reconstructions of plants would allow for more detailed descriptions of the state of the crops analogous to the way humans evaluate crop health, i.e. by looking at the canopy structure and check for
discolorations at specific locations on the plants.
Previous research in 3D reconstruction methods based on cameras has focused on rigid frontoplanar scenes such as buildings and rooms with billboard-like figures. Rigidness allows for advanced methods using structured light, laser scanning, object models for reference, etc.
Plants can be described as non rigid biological objects with complex structures, i.e. not frontoplanar but rather sloped surfaces. Plants wave in the wind, have complex structures with overlapping semitransparent leaves and have little texture variation and specular highlights.
Little work had been done in this area previously.
When analyzing 3D results from complex structures it is very difficult to obtain good quality dense ground truth of disparities. Therefore, a test framework was developed based on ray tracing. The goal was to analyze existing methods for disparity map generation. The major problem for existing methods was the steepness of the leaves relative to the closeness of overlapping leaves. Both sum-of-squared difference methods and energy-minimizing methods had this problem.
Following the test a series of disparity estimation techniques were developed and tested in the test framework using a set of ray traced images and a hand-annotated set of real plants with similar plant shapes.
Novel similarity measures were developed that could lead to better disparity estimations on sloped surfaces for a multi-baseline 5-camera setup and trinocular setups. In the multibaseline setup the developed method showed improvements mainly in areas with specular highlights. The methods using a trinocular setup showed better reconstruction in occluded areas.
The trinocular setup was used for both window correlation based and energy minimization based algorithms. A novel adaption of symmetric multiple windows algorithm with trinocular vision was developed. The results were promising and allowed for better disparity estimations on steep sloped surfaces.
Also, a novel adaption of a well known graph cut based disparity estimation algorithm with trinocular vision was developed and tested. The results were successful and allowed for better disparity estimations on steep sloped surfaces.
After finding the disparity maps each individual leaf had to be seperated using a simple labeling algorithm based on connected components analysis (based on Rosenfeld and Pfaltz union-find algorithm) where height information was also included, described as a NURBS surface and annotated with Number of leaves on plant, Area of plant and individual leaves, Leaf steepness, Height of leaf relative to ground, and Size (bounding box).
The new disparity map estimation methods were able to handle piecewise smooth sloped surfaces better to such a degree that individual separation of the leaves was possible and the extracted information was more accurate than using existing methods.
In order to allow for spectral reflection sampling at designated spots on the plants it was necessary to find tips and bases of each leaf. The results were promising but could be refined using knowledge about surface normals.
2D computer vision research has been done in active shape modeling of weeds for weed detection. Occlusion and overlapping leaves were main problems for this kind of work. Using 3D computer vision it was possible to separate overlapping crop leaves from weed leaves using the 3D information from the disparity maps.
The results of the 3D reconstruction and extracted features can be used to locate sample locations for multi-spectral reflection analysis. It can be employed during early growth stages in low cost crops or in high cost crops where the grown plants are placed separately. The image acquisition can be done from manually operated machinery or a field robot or a self guided tractor following a sample strategy based on overview maps of the field.