An AI referee.
Watch it
make the call.
240 million tennis matches played every year. Almost none have a referee. Sideline is a working prototype — an agent that watches video, makes calls with chain-of-thought reasoning, and signals a physical robot.
Prototype · Nebius.Build SF
SO-101 Arm — Ready
Matches Annually
Tennis alone. Add pickleball, volleyball, cricket — billions of plays with zero officiating.
Have a Referee
Amateur sports are completely unserved. Every close call is an argument.
Action Protocols
Function calls, MCP, and A2A — the agent acts through any interface, any robot, any client.
How It Works
Five stages. One agent loop. Video in, robot gesture out.
Video
Frame-by-frame capture from any camera source
VLM
Nebius Qwen2-VL-72B reads the frame, describes the play
CoT
Chain-of-thought: rules applied, confidence scored, call decided
Call
Score updated, voice announcement triggered, scoreboard synced
Robot
Gesture executed — point out, raise arm, wave off — on the physical body
Full Pipeline
Video → Nebius VLM → CoT Reasoning → Tool Call → Score + Voice + Robot Gesture
Observe → Perceive → Reason → Decide → Act → Remember
No training data. No fine-tuning.
The VLM already understands what it sees. We give it the rules of the sport as context and let it reason.
Training Videos Needed
Qwen2-VL-72B on Nebius already understands visual scenes. We inject the sport rules as a system prompt. That's it.
From blank repo to working demo
Fresh repo at 10:40 AM. Agent reasoning on live video by 5:00 PM. Domain knowledge in our heads, every line of code written at the event.
Action Protocols
Function calls for internal actions, MCP so any AI client can use our referee as a tool, A2A for multi-agent coordination.
The Agent Loop
Observe → Perceive → Reason → Decide → Act → Remember
Each frame feeds the loop. Game state persists across the match. The agent remembers what happened three points ago.
Sport Modules
Tennis is working. The rules engine and prompts are designed to swap — one file per sport. Pickleball and volleyball are next.
Tennis & Table Tennis
Fault, out, ace, winner, let — full scoring with deuce and advantage. Ball tracking and line call analysis.
Built at hackathon
Fault, out, ace, let — full scoring with deuce and advantage.
Robot gestures
Point out, raise for ace, wave off for let — on physical hardware.
Chain-of-thought
Every call is reasoned and auditable. See the thinking in the dashboard.
Pickleball
Kitchen violations, line calls, scoring — the fastest-growing sport in America and still no referee at most courts.
Coming next
Same agent, same architecture. Swap the rules file and the sport changes.
Volleyball
Line calls, rotation tracking, serve analysis — the agent doesn't care what sport it's watching.
On the roadmap
Rules and prompts are pluggable. Tennis proved the pattern.
Same Brain. Any Body.
The agent doesn't care what it's running on. Swap the robot backend with one environment variable.
Tier 1
MentorPi
HiWonder tracked rover with mecanum wheels, camera module, speaker, and ROS2. Mobile, affordable, ready.
- · Mecanum wheels — moves in any direction
- · Built-in speaker for TTS announcements
- · Lidar + depth camera
- · ROS2 + Python control stack
Tier 2
SO-101 Arm
LeRobot / HuggingFace 6-DOF robotic arm. Signals calls with precise arm gestures — point out, raise for ace, wave off for let.
- · 6x Feetech STS3215 servos
- · Leader-follower teleoperation
- · LeRobot SDK — pip install lerobot
- · Fault / out / ace / winner gestures
Tier 3
Unitree G1
Full humanoid. Walks to the line, raises an arm, makes the call. The closest thing to a real referee on the court.
- · Full-body walking and gesturing
- · Head tracking for situational awareness
- · Unitree SDK — shared fleet at event
- · A2A protocol for multi-agent coordination
Built With
Why an Agent, Not Just Vision?
An image classifier can tell you the ball landed out. An agent knows the score, the server, the rules, and what happened three points ago.
Traditional CV
- ✗ Ball in/out — that's it
- ✗ No understanding of game rules
- ✗ Can't track score or match state
- ✗ Needs thousands of labeled training images
- ✗ One sport per model
Sideline Agent
- ✓ Understands rules, context, and match history
- ✓ Tracks score across an entire match
- ✓ Shows reasoning — auditable, challengeable
- ✓ Zero training — uses pre-trained VLM + prompts
- ✓ Swap sport by changing the rules file
The Team
Built at Nebius.Build SF 2026
6 hours. SHACK15, San Francisco.
Watch it make the call.
Tennis match. Reasoning agent. Physical robot. Built in 6 hours, open source — try it or read the code.