Objective

We want to evaluate the quickest way to detect the ball in a sport event in order to develop an Sports AI without spending a million dollars on tech or developers. Quickly we find out that detecting the ball is key component to develop a robust Sports AI.

I’m not the technician on my team. Meaning, the point of view provided in this post won’t take you deep into any algorithm or neural network.

Why should you care? and Why the ball?

If you are in the AI Sports niche is totally relevant for you to know the state of the art in ball detection, this object is always the main center of attention of any sport. If you can detect the ball (even at high speeds) then you will be able to detect easier key events of the game, metrics and so much more useful information.

What about the players?

Having a successful Sports AI is a result of a well done computer vision set of detections, nowadays there are many available neural networks good at detecting human bodies (like OpenPose for example) so players will be easy to detect but not so the ball.. even less at high speeds…
That's why we created this post to take evaluate the quickest way possible to ball detection.

Only being able to detect the ball won't take you there but we believe with a good ball detector alongside the actual people detector from pre-trained neural networks is the quickest way to start.

Introduction: Tracking and Detecting Moving Objects

As we stated before detect and track moving objects are key tasks for an AI in order to succeed in and thrive in this niche. Sports is equivalent to motion, people and objects moving in different speeds and directions.

The capability to track moving objects and analyze them plays a crucial role. For us to develop and teach an AI to to watch a football game and detect elements or events from the match sounds like an easy task but its is actually very challenging...

Sometimes, even a human struggles to understand what’s happening on a complex sports play. Look at how many crazy things happen on the field on this 2010 World Cup:

It is very hard to notice Suarez was actually touching the ball with his hand to stop the goal. Luckily for Ghana, the referee was able to spot it and give them the penalty and a last chance to qualify to the semi-finals

Here are a couple of reasons:

• The camera is constantly panning, zooming and sometimes tilting and sometimes a replay appears from a radical different angle and our brains are SO good at negating this motion that we don’t even notice it.
• The amount of objects and people moving in every direction with different speeds and sometimes unpredictable trajectories
• The background mixing with players and ball, objects and people with nearly identical appearance
• False positives, same objects outside the game (Players with the same outfit on the benches, sports balls outside the game identical to the ball “in game”)

But we humans are magnificent at this tasks!
We as humans negate effectively the camera movement that's why we don't get confused between the background and playing field for example…

Or get lost in the rapid and sometimes erratic movement of people (players) and objects (ball, net, etc.) for an AI to compete with a human is difficult.

Let’s begin ! — My journey into ball detection

My objective was to try out the ball detectors I come across within 5 hours of research in Google to let this experiment very plain and without an insider view of the AI industry.

To make this I use a “plug and play” approach, I don’t modify the code of the detectors and neural networks I find, I just download and run it with my local example with only minor modifications.

I successfully want to track the ball on the following video, for sure my expectations were not too high, I’m aware the ball moves incredible fast..

Let’s see what happened!!

This is the video I will use to analyze with 3 options of ball detectors.

1st — PyImageSearch

OpenCV ball Detector

¡TOO GOOD TO BE TRUE! ¡ first result on my search!
📹 Running on CPU at 32fps 🤯
✅ Realtime capabilities.

Reference Link Source Code:
https://www.pyimagesearch.com/2015/09/14/ball-tracking-with-opencv/

This PyImageSearch article describes a very naive method to detect and track the tennis ball: just look for a green region on the image, and then find the enclosing circle for that region. Easy!

The reported results show the ball was actually very well tracked, even if was partially occluded from view by the hand. It does even run at 32 FPS, which is insanely high and allows for real-time detection.

So far so good, fits perfectly my needs so i needed to give it a try on my video!

This was the result of applying this detector.

Small clarification here: We did change the color detector from green to white in order to try to detect soccer ball because was set to green.

Didn’t worked at all, it got confused with other white objects on the screen and the ball was never detected or tracked.

2nd DETECTRON2

Pre-Trained Neural Network From Facebook

📹 Running on GPU (Nvidia GTX 1060Ti 6Gb Ram) at 2 fps,
❌ Not real time capability

Reference Link Source Code:
https://nol.cs.nctu.edu.tw:234/open-source/TrackNet/‍

In this second option I will try to implement a neural network provided by Facebook already pre-trained with tons objects (including Sports balls). I was really intrigued about the “out of the box” result. Also, I wanted to see if it was going to be capable of detecting the players bodies.

This was the result of applying this detector:

The players were all perfectly detected that is a point in favor for Detectron2. About the ball, it got detected in a few frames but was the first time I saw it detected!

We find out that when the ball is in still position it is able to be detected by Detectron2 and with a high confidence score.

But this pretrained neural network has problems with fast moving objects like the ball.

This led me to the last experiment with a custom tracker specially solving the fast moving objects based on heatmaps this next development is promising because the nearly perfect detections on tennis (which is a smaller ball and moves faster!)

3rd TrackNet

📹 Inside the documentation specify that it was able to run at 22fps! Allowing Real Time Capability BUT was not so correct..
On my GPU (Nvidia GTX 1060Ti 6Gb Ram) it ran at 2 fps..
❌ Not real time capability.

Reference Link Source Code:
https://nol.cs.nctu.edu.tw:234/open-source/TrackNet/

This one was my best candidate for Ball Tracking even though the detection was on tennis I was confident that it could be easier to detect a ball that is bigger in size and never travel with that much speed.

This looks like perfect for our propose and the small tennis ball is perfect detected and tracked, but…

The result of applying this detector:

I was wrong, the result was bad… Even though the detection of the soccer ball improve comparing to the PyImageSearch detector, there are too many false positives on the frames… meaning that TrackNet detects the ball even where there is not a ball at all…

Additionally this ran at 2 fps not the 22 fps that were mention on the documentation, this blocks the possibility to analyze and detect at realtime.

One big advantages is that this TrackNet neural network have very good documentation and advice in how to train with your own data. This could be a solution to solve the ball detection on soccer, but it’s clearly not an easy task.

If anyone has a dataset of soccer balls (real and/or synthetic), we are interested in collaboration to train this neural network or develop a custom solution, write an email to us at our Contact section

Conclusions from results

In conclusion no “plug and play” neural network will take you all the way there in AI Sports, there is no global solution (YET) in this niche. This makes an incredible advantage for those who jump in this space early and develop a global solution. This is yet to come...

But, even though there’s not a global solution, each different situation can be solved with different methods, creating custom solutions adapted for each problem.

That’s why you will need to contact a team to develop this type of AI.

If you are interested on collaborate or build your AI with us.

Reach out!