Functgraph: Personal Factory Automation With a 3D Printer


You might think that GPS navigation is ideal for automated agriculture, since the task facing the operator of a farm vehicle like a combine harvester is simply to drive around the field in a serpentine pattern, mowing down all the wheat or whatever crop it is filled with. But reality is far different. There are hundreds of things operators must watch even as they keep their eyes fastened to the edge of the field to ensure that they move alongside it with fine precision. An agricultural combine is not dissimilar to a church organ in terms of its operational complexity. When a combine operator works with an assistant, one of them steers along the crop edge, while the other controls the reel, the fan, the threshing drum, and the harvesting process in general. In Soviet times, there were two operators in a combine crew, but now there is only one. This means choosing between safe driving and efficient harvesting. And since you can’t harvest grain without moving, driving becomes the top priority, and the efficiency of the harvesting process tends to suffer.

Harvesting efficiency is especially important in Eastern Europe, where farming is high risk and there is only one harvest a year. The season starts in March and farmers don’t rest until the autumn, when they have only two weeks to harvest the crops. If something goes wrong, every day they miss may lead to a loss of 10 percent of the yield. If a driver does a poor job of harvesting or gets drunk and crashes the machine, precious time is lost—hours or even days. About 90 percent of the combine operator’s time is spent making sure that the combine is driving exactly along the edge of the unharvested crop to maximize efficiency without missing any of the crop. But this is the most unpleasant part of the driving, and due to fatigue at the end of the shift, operators typically leave nearly a meter at the edge of each row uncut. These steering errors account for a 25 percent overall increase in harvesting time. Our technology allows combine operators to delegate the driving so that they can instead focus on optimizing harvesting quality.

Add to this the fact that the skilled combine operator is a dying breed. Professional education has declined, and the young people joining the labor force aren’t up to the same standard. Though the same can be said of most manual trades, this effect creates a great demand for our robotic system, the Cognitive Agro Pilot.

Developing AI systems is in my genome. My father, Anatoly Uskov, was on the first team of AI program developers at the
System Research Institute of the Russian Academy of Sciences. Their program, named Kaissa, became the world computer chess champion in 1974. Two decades later, after the collapse of the Soviet Union, the Systems Research Institute’s AI laboratories formed the foundation of my company, Cognitive Technologies. Our first business was developing optical character recognition software used by companies including HP, Oracle, and Samsung, and our success allowed us to support an R&D team of mathematicians and programmers conducting fundamental research in the field of computer vision and adjacent areas.

In 2012, we added a group of mathematicians developing neural networks. Later that year, this group proudly introduced me to their creation: Vasya, a football-playing toy car with a camera for an eye. “One-eyed Vasya” could recognize a ball among other objects in our long office hallway, and push it around. The robot was a massive distraction for everyone working on that floor, as employees went out into the hallway and started “testing” the car by tripping it up and blocking its way to the ball with obstacles. Meanwhile, the algorithm showed stable performance. Politely swerving around obstacles, the car kept on looking for the ball and pushing it. It almost gave an impression of a living creature, and this was our “eureka” moment—why don’t we try doing the same with something larger and more useful?

A combine driven by the Cognitive Agro Pilot harvests grain while a human supervises from the driver’s seat.Cognitive Pilot

After initially experimenting with large heavy-duty trucks, we realized that the agricultural sector doesn’t have the major legal and regulatory constraints that road transport has in Russia and elsewhere. Since our priority was to develop a commercially viable product, we set up a business unit called
Cognitive Pilot that develops add-on autonomy for combine harvesters, which are the machines used to harvest the vast majority of grain crops (including corn, wheat, barley, oats, and rye) on large farms.

Just five years ago, it was impossible to use video-content analysis to operate agricultural machinery at this level of automation because there weren’t any fully functional neural networks that could detect the borders of a crop strip or see any obstacles in it.

At first, we considered combining GPS with visual data analysis, but it didn’t take us long to realize that visual analytics alone is enough. For a GPS steering system to work, you need to prepare a map in advance, install a base station for corrections, or purchase a package of signals. It also requires pressing a lot of buttons in a lot of menus, and combine operators have very little appreciation for user interfaces. What we offer is a camera and a box stuffed with processing power and neural networks. As soon as the camera and the box are mounted and connected to the combine’s control system, we’re good to go. Once in the field, the newly installed Cognitive Agro Pilot says: “Hurray, we’re in the field,” asks the driver for permission to take over, and starts driving. Five years from now, we predict that all combine harvesters will be equipped with a computer vision–based autopilot capable of controlling every aspect of harvesting crops.

Getting to this point has meant solving some fascinating challenges. We realized we would be facing an immense diversity of field scenes that our neural network must be trained to understand. Already working with farmers on the early project stages, we found out that the same crops can look completely different in different climatic zones. Preparing for mass production of our system, we tried to compile the most highly diversified data set with various fields and crops, starting with videos filmed in the fields of several farms across Russia under different weather and lighting conditions. But it soon became evident we needed to come up with a more adaptable solution.

We decided to use a coarse-to-fine approach to train our networks for autonomous driving. The initial version is improved with each new client, as we obtain additional data on different locations and crops. We use this data to make our networks more accurate and reliable, employing unsupervised domain adaptation to recalibrate them in a short time by adding carefully randomized noise and distortions to the training images to make the networks more robust. Humans are still needed to help with semantic segmentation on new varieties of crops. Thanks to this approach, we have now obtained highly resilient all-purpose networks suitable for use on over a dozen different crops grown across Eastern Europe.

The way the Cognitive Agro Pilot drives a combine is similar to how a human driver does it. That is, our unique competitive edge is the system’s ability to see and understand the situation in the field much as a human would, so it maintains full efficiency in collaboration with human drivers. At the end of the day, it all comes down to economics. One human-driven combine can harvest around 20 hectares of crops during one shift. When Cognitive Agro Pilot does the driving, the operators’ workload is considerably lower: They don’t get tired, can make fewer stops, and take fewer breaks. In practical terms, it means harvesting around 25 to 30 hectares per shift. For a business owner, it means that two combines equipped with our system deliver the performance of three combines without it.

While the combine drives itself, the human operator can make adjustments to the harvesting system to maximize speed and efficiency.Cognitive Pilot

On the market now there are some separate developments from various agricultural-harvesting companies. But each of their autonomous features is done as a separate function—driving along a field edge, driving along a row, and so on. We haven’t yet seen another industrial system that can drive completely with computer vision, but one-eyed Vasya showed us that this was possible. And so as we thought about cost optimization and solving the task with a minimum set of devices, we decided that for a farmer’s AI-based robot assistant, one camera is enough.

The Cognitive Agro Pilot’s primary sensor is a single 2-megapixel color video camera that can see a wide area in front of the vehicle, mounted on a bracket near one of the combine’s side mirrors. A control unit with an Nvidia Jetson TX2 computer module is mounted inside the cab, with an integrated display and driver interface. This control unit contains the main stack of autonomy algorithms, processes the video feed, and issues commands to the combine’s hydraulic systems for control of steering, acceleration, and braking. A display in the cab provides the interface for the driver and displays warnings and settings. We are not tied to any particular brand; our retrofit kit will work with any combine harvester model available in the farmer’s fleet. For a combine more than five years old, interfacing with its control system may not be quite so easy (sometimes an additional steering-angle sensor is required), but the installation and calibration can still usually be done within one day, and it takes just 10 minutes to train a new driver.

Our vision-based system drives the combine, so the operator can focus on the harvest and adjusting the…


Read More:Functgraph: Personal Factory Automation With a 3D Printer