Vision System Simplifies Robotic Fruit Picking – ACRO InstituteRobotic apple picker relies on a camera inside the gripper and off-the-shelf software
In response to Belgian farmers' problem to find reliable seasonal workers, the Automation Centre for Research and Education (ACRO) Institute developed a robotic apple harvester. ACRO is a spin-off company and technical consultant to the Catholic University College of Limburg, Belgium, and also provides PLC and machine-vision technical training and research to the European community.
The AFPM apple harvester also consists of a generator for power supply, a horizontal stabilization unit, a seventh external vertical axis to enlarge the operation range, a SICK safety scanning device, a Siemens central control unit, and touch panel PC with human-machine interface (HMI). MVTec's HALCON image processing software provides the robot guidance coordinates, while a canopy and curtain, which can be folded up during transportation, reduce the affects of ambient light (see Fig. 2). The AFPM needs one driver on the tractor while it effectively handles the workload of six workers.
One of the most challenging problems was the design of the fruit gripper. The final gripper design is a combination of a white flexible silicone cone surrounding an IDS UI-2230RE-C uEye USB 2.0 color camera. The cone is fed by a reversible vacuum/blower (see Fig. 3).
By placing the camera inside the gripper, the position of the camera is fully controllable. The camera can point its optical axis at the apple, reducing image distortion and eliminating repetitive calibration steps during apple picking. A final advantage is that the camera is protected against collisions or bad weather conditions, as well as against direct sunlight.
Once the apple harvester is moved in front of the tree and its canopy opened around the AFPM platform (see Fig. 4), the harvester has to be actively stabilized with different hydraulic foot and elevator systems. After the rig is automatically adjusted, the camera scans the tree from 40 preprogrammed positions. Thus, each tree is divided into 40 sectors or images.
The programmed location and orientation of the robotic arm is stored with each image. Location data is stored in the PC's RAM. For each sector, all ripe apples are identified by the image processing software, listed, and picked one by one in a looped task.
Features, advantages, benefitsSays ACRO's Eric Claesen, "The goal, at the beginning - three years ago - of the project, was to research if it was possible to pick an apple with the technology of today. Now at the end of the project we can say that the project succeeded. We can detect and pick 85% of the apples in a tree correctly. At the moment the picking machine picks an apple every 8 s but this will be reduced to 6 or 5 s after the pick season in 2007.
"The designers believe that this time span can be lowered to about 5 s, chiefly by reducing the communication bottleneck between the vision system and the central controller unit. The designers are also planning to better the automation of the navigation through the orchard and to make the device suitable for picking other fruit, such as pears."
The platform was designed to control the lighting conditions to the greatest extent possible, using a canopy to cover the entire tree and the AFPM platform to reduce the effects of changing ambient lighting conditions and provide a uniform background (blue) to ease locating the (red and green) apples.
Image processing is conducted on an industrial PC with 2 GHz Pentium IV microprocessor and 1 GByte RAM running Windows XP. For image-processing software, the AFPM designers elected the standard machine vision library HALCON 7.1 from MVTec because of its accuracy and reliability, according to Eric Claesen, lead designer for the project at ACRO. The system can pick 85% of all apples of a tree, similar to the achievements of manual picking operations.
Author: Dr. Lutz Kreutzer, MVTec Software GmbH