Use Detectron2 and the Open AI API to assess NBA decision making with automatically generated, opinionated Python code. Go beyond empirical observations and rules-based evaluation with plain text queries that produce analysis.
Stephen Curry takes the ball down the court...
curry_partial.mp4
What should happen next? Should he shoot? Pass? Call a timeout? We can use BRIAR to answer!
Call the decision point initial_decision_point.png
We can then run BRIAR and dynamically get its opinion on the best next move!
Let's see what actually happened...
curry_full.mp4
He did pass!
Let's try again with a play BRIAR was not trained on...
durant_partial.mp4
Same question as above! What does BRIAR think should happen next? Should Kevin Durant shoot? Pass? Call a timeout? Let's see!
Call the decision point another_decision_point.png
Let's run BRIAR again!
Let's see what actually happened...
durant_full.mp4
He did shoot!
So how does this actually work?
Detectron2 is a platform for object detection, segmentation and other visual recognition tasks. It's excellent, but not tailored to the NBA game. Without any training, when it looks at initial_decision_point.png it struggles to locate the players and the ball.
We can train a better model though! My goal here is a proof of concept vs. actually creating a robust set of training data, so it's relatively quick to train Detectron2. The subsequent model is good enough for BRIAR to show what it can do, despite some overfitting. For model training details, please see briar.ipynb.
Here is the output of the model! Much better, it can tell who is on the Warriors, who is on the Lakers, and who has the ball with enough certainty to be interesting.
tagged_play.mp4
Here is what it sees in initial_decision_point.png
Great! This kind of image processing is not new in the NBA context, but it does lay a strong foundation for the second part of BRIAR.
We now have an image we can interpret, and get the state of in a dynamic way, by asking basic questions such as these:
- How many players are on the court?
- How far away from each other are the players?
- How many players are closely guarding the player with the ball?
We can get the answers, and more, with the Open AI API! I used the GPT-3.5 Turbo model, but this should work with newer ones as well (though it will be more expensive).
You can query the API, and it will write Python code, with its own reasoning and logic, to interpret the state of the NBA game.
Let's see it in action! Let's start simple, with a basic empirical question.
This is cool, but again nothing that new – this kind of code isn’t that hard to write, making Open AI write it instead is a marginal benefit.
The new and cool thing is reasoning, using LLMs to interpret the state, make a judgement, and reason about possible next actions. We will be able to move away from rules-based systems where we must consider every case.
Let's ask BRIAR something that requires more reasoning.
Wow! The key thing here is I didn’t write this code. I didn’t think of these outcomes, I didn’t tell the Open AI API that if you are being triple teamed you probably want to look for a teammate - it was able to leverage the state I gave it and its own LLM to get to that conclusion on its own. All I had to tell it was the structure of my code, the classes I wrote to help the interpretation, and it was able to generate a novel answer, that I can run in real time.
- I used a very small training set for my Detectron2 implementation, definitely overfitted.
- You still need to tell Open AI a fair amount about your code, ideally it would need less help.
- To run this code requires a powerful GPU.
Where is this going? Someone with more money and more GPUs than I will create a model trained on NBA data + NBA telecasts + coach input. It will create a real time decision engine, capable of evaluating the state of any NBA game at any moment and saying what the best next action is. This will be a paradigm shift, you will be able to ask the tool questions in plain text, no coding ability needed.
This will change the way players are coached, and the way they look at film, there will be less work for everyone. The tool will move beyond just interpreting state to actual advice and decision making, able to work at all levels of the game. It won't replace practice or innate ability, but it will be a force multiplier. I could even see a day where the tool is able to simulate and play against itself, developing new plays, figuring out how to leverage player ability in new ways, and allocate practice time as efficiently as possible.
You can run BRIAR easily on Google Colab. All you need is an Open AI API key. Each run costs a fraction of a cent, so it won't break the bank. You can even train your own model! Try it today!
BRIAR is inspired by ViperGPT, created by the Computer Vision Lab at Columbia University. I made my own implementation (with the exception of a very similar CodexModel class), but I used a heavily inspired methodology from their paper and repository.