Skip to Content
Artificial intelligence

Why Ball Tracking Works for Tennis and Cricket but Not Soccer or Basketball

Following the examples of tennis and cricket, a new generation of ball-tracking algorithms is attempting to revolutionize the analysis and refereeing of soccer, volleyball, and basketball.

When it comes to ball sports, machine-vision techniques have begun to revolutionize the way analysts study the game and how umpires and referees make decisions. In cricket and tennis, for example, these systems routinely record ball movement in three dimensions and then generate a virtual replay that shows exactly where a ball hit the ground and even predicts its future trajectory (to determine whether it would have hit the wicket, for example).

But this kind of ball tracking is notably absent in other ball sports, such as basketball, volleyball, soccer, and so on. In these sports, the ball is often hidden from view behind players, its movement is significantly different when it is in a player’s possession compared to when it is flying through the air, and players’ interactions with the ball can be rapid and unpredictable.

These factors, as well as the small size of the ball in a frame and the sometimes low quality of video images, make ball-tracking much harder in these sports.

Today, Andrii Maksai and pals at the Ecole Polytechnique Federale de Lausanne in Switzerland outline a new way to track balls in these sports that outperforms other state-of-the-art approaches.

Most ball tracking systems rely on two different approaches. The first looks to follow the movement of the ball in three dimensions and then predicts various likely trajectories in the future. This “tree” of possible trajectories can then be pruned as more ball-tracking data becomes available.

The advantage of this approach is that the laws of physics are built in to the trajectory predictions so unphysical solutions can be avoided. However, it is hugely sensitive to the quality of the ball tracking data and so tends to fail when the ball is occluded or when players interact with the ball in unpredictable ways.

Another method is to track the players and note when they are in possession of the ball. The movement of the ball is then assumed to follow the player and when possession transfers from one player to another. The advantage here is that the system does not get so confused by rapid or unpredictable passes—indeed, this approach works well in basketball, where dribbling and occlusion can make life difficult for ball trackers. However, without physics-based constraints on the motion of the ball, these systems can produce inaccurate tracks.

Maksai and co have come up with an obvious solution. They track both the ball and the players accurately. They then use one of several different ways of solving the ball-tracking problem that depend on how the players are interacting with the ball.

For example, a basketball shot towards the basket follows a ballistic trajectory. But a rolling ball follows a different path. Both of these require different ball tracking solutions to a spike in volleyball which causes a sharp change in trajectory. And a ball being dribbled by a soccer player follows yet another type of irregular trajectory, requiring another solution. “We explicitly model the interaction between the ball and the players as well as the physical constraints the ball obeys when far away from the players,” say Maksai and co.

The team has tested its algorithm on a number of video sequences of various volleyball, basketball, and soccer games. The data comes from several cameras that record the same action from different angles to create a 3-D model of what’s going on. However, the data is far from perfect with many instances of occlusion, unpredictable passes and irregular trajectories.

The results show some improvement existing techniques. “We show that our approach is more robust and more accurate than several state-of-the-art approaches on real-life volleyball, basketball, and soccer sequences,” they say.

It’s not perfect, however. A crucial performance milestone for these systems is the ability to produce a virtual replay of a ball’s movement quickly and accurately enough for a TV audience.

That’s a tough ask, not least because this new system becomes better at tracking players as the length of the video sequence increases. But this dramatically increases processing time.

But greater processing time severely limits the system’s utility for live-broadcasts of sporting events when the virtual replay has to be available almost immediately.

However, the accuracy of certain types of ball tracking—ballistic shots, for example—becomes easier on shorter sequences since there is less unpredictability. So some kind of optimization process should help here.

With work like this, ball tracking in games like soccer, basketball, and volleyball is coming closer. But it is not yet fast enough to be commercially viable for a sports broadcaster.

That may require a step change in the way researchers approach this problem. One possibility could be to employ deep-learning techniques, where an AI system learns to predict future ball movement using its learned knowledge of previous games. That could significantly simplify some of the tasks that ball tracking involves.

Either way, still more work to be done here.

Ref: What Players Do With the Ball: A Physically Constrained Interaction Modeling

Deep Dive

Artificial intelligence

conceptual illustration showing various women's faces being scanned
conceptual illustration showing various women's faces being scanned

A horrifying new AI app swaps women into porn videos with a click

Deepfake researchers have long feared the day this would arrive.

storm front
storm front

DeepMind’s AI predicts almost exactly when and where it’s going to rain

The firm worked with UK weather forecasters to create a model that was better at making short term predictions than existing systems.

People are hiring out their faces to become deepfake-style marketing clones

AI-powered characters based on real people can star in thousands of videos and say anything, in any language.

Tentacle of Octopus
Tentacle of Octopus

What an octopus’s mind can teach us about AI’s ultimate mystery

Machine consciousness has been debated since Turing—and dismissed for being unscientific. Yet it still clouds our thinking about AIs like GPT-3.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.