EPFL uses Panther in a fruit harvesting research project

Designing autonomous solutions for agriculture is not an easy task. While there are successful examples of robots autonomously planting, inspecting or picking more accessible crops, there is still a wide variety of fruits and vegetables that remain out of reach when it comes to autonomous harvesting. And there are reasons for that - agricultural environments can be harsh and extremely variable, posing the challenges both to the mechanical robustness of the robot platform and its abilities for autonomous navigation. Another problem to solve is implementing crop picking, which requires algorithms for efficient fruit localization, assessment of its ripeness and plucking without causing damage to the crop, which is especially demanding in case of fragile fruits.

All these challenges didn’t discourage members of EPFL CREATE Lab, who recently developed a proof-of-concept solution for harvesting raspberries, using Panther as a base for their mobile robot. Their novel approach allowed them to achieve 80% successful harvesting rate during the first field test. What was their key to success?

EPFL Computational Robot Design & Fabrication Lab is a group that has set its goal toward developing robotic solutions using complex, multi-material sensorized structures and optimizing complex agricultural systems. In their research, they decided to concentrate on possibly one of the most difficult fruits to harvest, known for its fragility - raspberry. With a labor shortage being a problem across the entire agriculture industry, the raspberry, due to its short period of possible harvest, poses a special challenge to farmers, resulting in a growing number of raspberries being left unpicked in the field.

“It’s an exciting dilemma for us as robotics engineers,” says Josie Hughes, a professor at CREATE. “The raspberry harvesting season is so short, and the fruit is so valuable, that wasting them simply isn’t an option.”

Step 1: Create a raspberry physical twin

An idea behind the project was to tackle one of the main problems of designing robots for agriculture: the lab to field transfer. As the harvesting period is limited for all kinds of crops, it is impossible to develop and test robotic solutions in the field only. At the same time, robots developed and tested in laboratory conditions usually cannot be directly applied in the field and require multiple series of testing and fine-tuning. All of this makes the process of designing robotic solutions for harvesting costly and time-consuming.

To address this issue and maximize applicability of lab-developed solutions in the field, members of the research group decided to create an artificial raspberry made of soft fabrics imitating the surface and structure of a real raspberry. The silicone twin was also equipped with a set of tactile sensors able to measure forces applied to the fruit when picking it up. Wondering how a robot was then taught to flawlessly pick up this artificial raspberry?

Step 2: Show the robot how to pick up raspberries

To record and investigate the proper way of harvesting raspberries, members of CREATE Lab performed a series of demonstrations on a digital raspberry twin. The raspberry's properties were adjusted to imitate different degrees of ripeness. A robotic gripper with force sensors on the fingers was used for the harvesting task, with a feedback controller adjusting the interaction forces. To tune the controller correctly, first a human harvested the artificial raspberry. Then the robot repetitively harvested the fruit while simultaneously tuning its parameters, until the response read by the sensors embedded in the raspberry matched that of the human harvest.

“Our sensorized raspberry can teach a robot to apply just the right amount of force,” explains PhD student Kai Junge. "A big challenge in this task is to tackle the complex soft interactions of the fruit. The physical twin that is well tuned to the real fruit allows us to quickly develop robots and controllers easier and quicker than repetitive field tests or in simulation."

Step 3: Use Panther to drive freely in the field

After achieving satisfying accuracy and success rate of raspberry harvesting in the lab, CREATE members were ready for field tests. However, to navigate between the paths, they had to integrate their solution, consisting of a UR5 robotic arm, control computer, and gripper, with a mobile base.

Choosing Panther for that role provided them with three major advantages:

• Panther is adapted to traverse difficult terrain, with a robust aluminum chassis and high-profile off-road wheels, so no raspberry field poses a challenge to it.
• It can be easily integrated with various robotic components, such as robotic arms, LIDARs and sensors. In particular, it has a sufficient payload and suitable power supply system, allowing for a smooth and quick integration with the UR5 robotic arm used by researchers.
• Panther runs on ROS with ready-to-use open-source GitHub templates for common use cases like autonomous navigation or controlling a robot manually. Thanks to that researchers could easily deploy the robot and control it remotely, with potential to develop autonomous navigation within the crops in the future.

During the first field test, the robot achieved a 80% successful harvesting rate without any adaptations or fine-tuning. According to CREATE Lab members, it is the first robotic system that has been entirely trained in lab conditions and enabled immediate and successful operation in the field. Their research can be the first step towards solving the lab-to-field transfer problem and revolutionizing the area of autonomous harvesting, thus reducing the need for manual labor.

Currently, engineers from CREATE Lab are working on developing a camera system for localizing raspberries and evaluating their degree of ripeness.

Want to learn more about the EPFL CREATE Lab project?

Check out their paper on the lab-to-field harvesting transfer, published in Nature Portfolio Communications Engineering, and the feature video from Nature.

Or take a look at their website, where they present some additional video materials.

And if you think Panther may be equally useful in your use case, don’t hesitate to see it in our store or contact us on contact@husarion.com

The research was conducted by Kai Junge, Catarina Pires and Josie Hughes from CREATE Lab, EPFL.