Vision System Simplifies Robotic Fruit Picking – ACRO Institute

In response to Belgian farmers' problem to find reliable seasonal workers, the Automation Centre for Research and Education (ACRO) Institute developed a robotic apple harvester. ACRO is a spin-off company and technical consultant to the Catholic University College of Limburg, Belgium, and also provides PLC and machine-vision technical training and research to the European community.

The European standard height of fruit trees on professional plantations must be between two and three meters. The Automation Centre for Research and Education (ACRO) automated fruit-picking machine (AFPM) harvester uses a unique vacuum-gripper design to pick the fruit and ease coordination between the vision system and robot controller. Mounted behind a common agriculture tractor, the AFPM platform supports a Panasonic industrial robot to "pick" the fruit (see Fig. 1).

The AFPM apple harvester also consists of a generator for power supply, a horizontal stabilization unit, a seventh external vertical axis to enlarge the operation range, a SICK safety scanning device, a Siemens central control unit, and touch panel PC with human-machine interface (HMI). MVTec's HALCON image processing software provides the robot guidance coordinates, while a canopy and curtain, which can be folded up during transportation, reduce the affects of ambient light (see Fig. 2). The AFPM needs one driver on the tractor while it effectively handles the workload of six workers.

One of the most challenging problems was the design of the fruit gripper. The final gripper design is a combination of a white flexible silicone cone surrounding an IDS UI-2230RE-C uEye USB 2.0 color camera. The cone is fed by a reversible vacuum/blower (see Fig. 3).

By placing the camera inside the gripper, the position of the camera is fully controllable. The camera can point its optical axis at the apple, reducing image distortion and eliminating repetitive calibration steps during apple picking. A final advantage is that the camera is protected against collisions or bad weather conditions, as well as against direct sunlight.

Once the apple harvester is moved in front of the tree and its canopy opened around the AFPM platform (see Fig. 4), the harvester has to be actively stabilized with different hydraulic foot and elevator systems. After the rig is automatically adjusted, the camera scans the tree from 40 preprogrammed positions. Thus, each tree is divided into 40 sectors or images.

The programmed location and orientation of the robotic arm is stored with each image. Location data is stored in the PC's RAM. For each sector, all ripe apples are identified by the image processing software, listed, and picked one by one in a looped task.

The platform was designed to control the lighting conditions to the greatest extent possible, using a canopy to cover the entire tree and the AFPM platform to reduce the effects of changing ambient lighting conditions and provide a uniform background (blue) to ease locating the (red and green) apples.

Image processing is conducted on an industrial PC with 2 GHz Pentium IV microprocessor and 1 GByte RAM running Windows XP. For image-processing software, the AFPM designers elected the standard machine vision library HALCON 7.1 from MVTec because of its accuracy and reliability, according to Eric Claesen, lead designer for the project at ACRO. The system can pick 85% of all apples of a tree, similar to the achievements of manual picking operations.

During system calibration, the first step is to train the system on the color of the apple tree's leaves and the blue background canopy. A series of color thresholding steps are used to filter out these unwanted features. First the blue is filtered out of each of the 40 images, and then the leaves are filtered out. The green and the red parts of the apples' surfaces are located via color thresholding. After noise reduction, using HALCON's opening_circle and select_shape convolutions and filling holes in the images, the apples' positions are clearly emerged (see Fig. 5). After separating the individual apples from clusters (when present) by watershed-filtering, each apple is selected and transformed into a circle.

To pick one apple, the robot has to determine the distance between the camera and the apple and the path to get there. The camera measures this distance by triangulation. The measuring is done in several steps. The camera first acquires an image, and then the camera is turned so that the apple is situated in the center (see Fig. 6). The camera then acquires a second image, and finally the diameter is calculated by processing these two images. At this point, the vision system determines if the apple falls within the acceptable size range. If so, a signal is sent across a Profibus connection to the robot controller, and the robot arm is allowed to continue toward the apple.

While approaching the apple, several images are processed to calculate by triangulation the remaining distance to the apple while air is blowing through the gripper at about 300 m³/min to free the fruit from leaves that might conceal it. With each image acquired, the system calculates the remaining distance using a proprietary formula. Since the apple remains centered in the image, the correlation of a given apple in subsequent images is trivial. When the selected apple is approached, air blows through the picking device to clear the apple from leaves. Once the apple is within a well-defined range of the gripper, the vacuum device is activated. If a vacuum is detected, the apple is rotated and tilted softly, picked and then put beside.