MVTec Success Stories: Vision System Simplifies Robotic Fruit Picking

Vision System Simplifies Robotic Fruit Picking – ACRO Institute

Robotic apple picker relies on a camera inside the gripper and off-the-shelf software

In response to Belgian farmers' problem to find reliable seasonal workers, the Automation Centre for Research and Education (ACRO) Institute developed a robotic apple harvester. ACRO is a spin-off company and technical consultant to the Catholic University College of Limburg, Belgium, and also provides PLC and machine-vision technical training and research to the European community.

Fig. 1: Horizontal 2-D stabilization of the APFM consists of two hydraulic feet and one turnover cylinder, configured as a three-point suspension. Controlled by two level sensors, the system ensures stable positioning during picking. The white funnel-formed gripper unit is at the end of the robot arm.

The European standard height of fruit trees on professional plantations must be between two and three meters. The ACRO automated fruit-picking machine (AFPM) harvester uses a unique vacuum-gripper design to pick the fruit and ease coordination between the vision system and robot controller. Mounted behind a common agriculture tractor, the AFPM platform supports a Panasonic industrial robot to "pick" the fruit (see Fig. 1).

Fig. 2: ACRO's use of Profibus as the network connection between the vision system, PLC, and robot controller is due in part to ACRO's position as a Profibus training center.

The AFPM apple harvester also consists of a generator for power supply, a horizontal stabilization unit, a seventh external vertical axis to enlarge the operation range, a SICK safety scanning device, a Siemens central control unit, and touch panel PC with human-machine interface (HMI). MVTec's HALCON image processing software provides the robot guidance coordinates, while a canopy and curtain, which can be folded up during transportation, reduce the affects of ambient light (see Fig. 2). The AFPM needs one driver on the tractor while it effectively handles the workload of six workers.

One of the most challenging problems was the design of the fruit gripper. The final gripper design is a combination of a white flexible silicone cone surrounding an IDS UI-2230RE-C uEye USB 2.0 color camera. The cone is fed by a reversible vacuum/blower (see Fig. 3).

Features, advantages, benefits

Says ACRO's Eric Claesen, "The goal, at the beginning - three years ago - of the project, was to research if it was possible to pick an apple with the technology of today. Now at the end of the project we can say that the project succeeded. We can detect and pick 85% of the apples in a tree correctly. At the moment the picking machine picks an apple every 8 s but this will be reduced to 6 or 5 s after the pick season in 2007.

"The designers believe that this time span can be lowered to about 5 s, chiefly by reducing the communication bottleneck between the vision system and the central controller unit. The designers are also planning to better the automation of the navigation through the orchard and to make the device suitable for picking other fruit, such as pears."

By placing the camera inside the gripper, the position of the camera is fully controllable. The camera can point its optical axis at the apple, reducing image distortion and eliminating repetitive calibration steps during apple picking. A final advantage is that the camera is protected against collisions or bad weather conditions, as well as against direct sunlight.

Fig. 3: Vacuum gripper unit of the APFM has a USB 2.0 camera inside.

Once the apple harvester is moved in front of the tree and its canopy opened around the AFPM platform (see Fig. 4), the harvester has to be actively stabilized with different hydraulic foot and elevator systems. After the rig is automatically adjusted, the camera scans the tree from 40 preprogrammed positions. Thus, each tree is divided into 40 sectors or images.

The programmed location and orientation of the robotic arm is stored with each image. Location data is stored in the PC's RAM. For each sector, all ripe apples are identified by the image processing software, listed, and picked one by one in a looped task.

The platform was designed to control the lighting conditions to the greatest extent possible, using a canopy to cover the entire tree and the AFPM platform to reduce the effects of changing ambient lighting conditions and provide a uniform background (blue) to ease locating the (red and green) apples.

Image processing is conducted on an industrial PC with 2 GHz Pentium IV microprocessor and 1 GByte RAM running Windows XP. For image-processing software, the AFPM designers elected the standard machine vision library HALCON 7.1 from MVTec because of its accuracy and reliability, according to Eric Claesen, lead designer for the project at ACRO. The system can pick 85% of all apples of a tree, similar to the achievements of manual picking operations.

Fig. 4: The APFM picks apples in the orchard. The canopy covers an apple tree against intense sunlight.

Fig. 5: After filtering out leaf, background, noise reduction, and filling holes in the images, the apples' positions clearly emerge. Iterative triangulation guides the robot to the apple for picking. The gripper is designed to handle apples that vary from the target size by 10% without dropping the apple.

During system calibration, the first step is to train the system on the color of the apple tree's leaves and the blue background canopy. A series of color thresholding steps are used to filter out these unwanted features. First the blue is filtered out of each of the 40 images, and then the leaves are filtered out. The green and the red parts of the apples' surfaces are located via color thresholding. After noise reduction, using HALCON's opening_circle and select_shape convolutions and filling holes in the images, the apples' positions are clearly emerged (see Fig. 5). After separating the individual apples from clusters (when present) by watershed-filtering, each apple is selected and transformed into a circle.

To pick one apple, the robot has to determine the distance between the camera and the apple and the path to get there. The camera measures this distance by triangulation. The measuring is done in several steps. The camera first acquires an image, and then the camera is turned so that the apple is situated in the center (see Fig. 6). The camera then acquires a second image, and finally the diameter is calculated by processing these two images. At this point, the vision system determines if the apple falls within the acceptable size range. If so, a signal is sent across a Profibus connection to the robot controller, and the robot arm is allowed to continue toward the apple.

Fig. 6: Distance between camera and apple is measured by triangulation, and robot guidance is performed on iterative "Stop and Look" method based on extracting position data from periodic, static images rather than continuous monitoring of real-time video of each apple common to visual serving.

While approaching the apple, several images are processed to calculate by triangulation the remaining distance to the apple while air is blowing through the gripper at about 300 m�/min to free the fruit from leaves that might conceal it. With each image acquired, the system calculates the remaining distance using a proprietary formula. Since the apple remains centered in the image, the correlation of a given apple in subsequent images is trivial. When the selected apple is approached, air blows through the picking device to clear the apple from leaves. Once the apple is within a well-defined range of the gripper, the vacuum device is activated. If a vacuum is detected, the apple is rotated and tilted softly, picked and then put beside.