Research is being conducted with robotics in nut crops, focusing on automation for water, fertilization and pest control that could be done with robots, sensors or robots with sensors. Rather than one person doing these jobs, they could be paired with a fleet of robots. An old saying from the 1950s goes, ‘If you want to do robotics, they should do the DDD jobs: dangerous, dirty, and dull.’ This saying is particularly important with the current labor shortage in agriculture that’s unlikely to change.

Xiaoyi Lu, assistant professor at UC Merced, has research that is building a small-scale, campus-shared, cloud-mode Experimental Smart Farm Simulation (ESFSim) platform. This platform collects real-world data from the UC Merced Experimental Smart Farm (ESF), allowing researchers to run experiments and gather data more quickly. Many experiments can be run within a simulation, allowing researchers to pursue ideas that may not be feasible to test on a real farm due to cost or scale.

Mummy Nut Research
Lu also worked with Professor Reza Ehsani at UC Merced on detecting mummy nuts in almond trees by photographing the mummy nuts in a tree, then uploading them to The Cloud or a computer.

“We ran object detection models on top of those pictures and identified whether a tree had enough mummy nuts that they need to be removed,” Lu said.

To do this work, researchers must have models, and data, and they have to train the models, and then you must have expertise from agriculture to determine which trees have high enough numbers of mummy nuts that need to be removed.

“We need a profile tool to annotate these things easily, and we need to provide a system to store this information as well,” Lu said.

The research is looking to run this system through an orchard to identify the trees with a certain level of mummy nuts that need to be removed, then the information would be sent to a grower via a smartphone or computer to tell them which trees need to have the mummy nuts removed.

Providing growers with this information would help with the labor shortage because they would only shake specific trees and reduce labor and gas, ultimately saving money.

“So far, we just did that one pilot project, but in the future, we will probably do more, and actually, in my group, we are collaborating with some other agriculture scientists,” Lu said.
The mummy nut research started in 2022. “We have students still working on the interface and other agricultural related research tasks,” Lu said.

Artificial Intelligence
Lu sees an unlimited future with artificial intelligence (AI) for agriculture, and there will be many applications for it.

“I feel almost everything can be assisted by AI; humans still have to be there, too, but AI can assist,” Lu said.

“We are in the very early stages of applications for agriculture, and I believe we can go much further,” Lu said, adding with agriculture having a smaller and smaller labor pool to draw from, the benefits of AI will manifest themselves as we go along.

The Internet of Things for Agriculture
Stephano Carpin, professor of computer science at UC Merced is working on a project called the Internet of Things for Agriculture (IoT4Ag) (www.iot4ag.us) funded by the National Science Foundation in conjunction with the University of Pennsylvania, Purdue University and the University of Florida.

The goal is to provide better information to farmers so they can produce more food or the same amount of food, but use less resources, Carpin said.

There are three main pillars to the project: 1) data acquisition, which is where Carpin is most involved because of his work in robotics; 2) data transmission, because after data has been collected, oftentimes in remote locations, it has to be transferred to The Cloud; and 3) data consumption or data processing. This is where AI comes into play and turns the data into actionable information, which is then passed on to farmers.

“It’s a very large project as you can imagine, and my role in this is mostly to develop robotic systems for the data acquisition,” Carpin said, adding his focus is on high-value commodities like walnuts, grapes, almonds and pistachios.

Collecting Data
“In the IoT4Ag paradigm, the idea is that we could blanket the system with lots and lots of sensors that then can connect data, transmit and consolidate it. The problem with agriculture is that, so far, data collection continues to be very sparse,” Carpin said.
Farmers may set up a weather station and have some soil moisture sensors, but the technology is expensive. “If you want to assess soil moisture content, oftentimes what happens is [growers] will have just very few sensors that you can use to collect a few data points that would then be extrapolated and draw conclusions for a hundred acres,” Carpin said.

Carpin is working on methods for collecting the data via a robot, sensors or a robot with sensors that would be cheaper than having humans do the work.

“One of the aspects I’m not directly involved with because I’m not someone who develops sensors is to somehow say, ‘Can we develop new sensors that are cheaper and maybe less precise.’ If we can deploy a lot of them, then there is power in multitude,” Carpin said, adding some of the sensors being developed are biodegradable and could be left in the field.

“The other aspect is, ‘Okay, what if we can bring the cost down, but it’s not a disposable sensor?’ If we can mount the sensors on the robot, which is what we’re doing, then all of a sudden we don’t have a sensor on every tree, but we have a few sensors and a few robots, and we can bring those sensors to each and every tree and collect all the data we want,” Carpin said.

Drones are another option for collecting data. They’ve become extremely cheap, they can provide imagery flying over orchards that is very valuable, but they also have drawbacks: liability, limited flight time, and they only provide photos from the top of the tree. That’s where Carpin’s research comes in.

“Our approach is coordinated, whereby we will have drones flying over an orchard, but also unmanned ground vehicles (UGVs) that traverse orchards and collect other types of data that are not accessible from a drone,” Carpin said.

UGVs can take pictures at close range, cut leaves from trees and perform some analysis, which is very high quality for determining stem water potential and how much water is in the leaves, Carpin said.

“So, when it comes to data collection, what we are seeing is that there is not one modality that is going to become dominant, but there’s a combination of different ones,” Carpin said.
As fertilizer prices continue to increase, precision application becomes very important. Too little equals less production, too much could mean leaching into the soil and contamination, so knowing exactly how much to apply would save growers time, and money, and be good for the environment. The challenge is collecting the data in a way that isn’t labor intensive.
This could be done by measuring nitrates in the soil that would provide growers with information on whether they have a fertilizer deficit or excess.

“You need to know how much fertilizer was taken out, so we need to somehow be able to assess the amount of nitrogen,” Carpin said. “And again, this goes back to perhaps the other problem, which is scouting for this information is currently at a coarse scale.”

With a reduced labor force, workers need to be utilized effectively. Highly capable humans should be doing things that only humans can do. Repetitive tasks like taking pictures or walking up and down the orchard searching for water leaks should be outsourced to machines that aren’t impacted by extreme heat or fatigue, Carpin said.

The latest robots have RTK GPS, so with the right infrastructure in place, these machines will provide the information within three centimeters of the location. “There is no ambiguity. If they tell you it’s there, it is there, and that information can be passed to the worker,” Carpin said.

Making it Simple
One area Carpin thinks doesn’t get enough attention in terms of research and commercial products is making these systems simpler to use. It shouldn’t be something that requires the worker to have a master’s degree in computer engineering to use it, he said.

“Everyone can use an iPhone, and that is what we need to strive for so that the robots become widespread on farms,” Carpin said. “I see progress, but they’re complicated to use, so we need to do better.”

Somewhere in the future, Carpin envisions robots at a price where people could have several that would monitor their orchards with a small staff of employees to fix issues found by the robots whether it’s irrigation repair, spraying for pests or fertigation.

“You may have considerably less employees and still get the same work done,” he said.
“It just makes sense to have several robots because it will be faster and more robust. If you rely just on one robot, and it breaks down, you’re toast,” Carpin said.

Having multiple robots may seem counterintuitive, but they are less expensive than a single robot. “Five simple robots probably cost less than one that does everything,” Carpin said, adding he foresees a hybrid model with larger farms purchasing several machines and smaller farms leasing or renting them from a service the same as they do with custom harvesting.