An agentic AI referee that watches sports video, reasons about every play using chain-of-thought, and acts through any physical form — from dashboards to robots.
Extract frames from live video or recordings at configurable FPS. Each frame becomes input for the vision model.
Nebius Token Factory VLM analyzes each frame with game context. Qwen2-VL-72B or Cosmos Reason 2.
Chain-of-thought reasoning: what happened → which rule applies → what's the correct call → confidence level.
The model selects tools to call: update_score, announce_call, robot_gesture, or no_call.
Execute actions through three protocols: Function Calls, MCP servers, and A2A agent communication.
Maintain game state across the entire match. Score, rally context, call history — the agent never forgets.
Video Feed ──▶ Frame Extraction (1 fps)
│
▼
Nebius VLM + Tool Definitions
(vision + game state context)
│
▼
┌── Tool Calls ──────────────────────┐
│ │
┌──────┴──────┐ ┌──────────┐ ┌───────────┴───────┐
│ Function │ │ MCP │ │ A2A │
│ Calls │ │ Server │ │ Agent-to-Agent │
│ (internal) │ │(external)│ │ (multi-agent) │
└──────┬──────┘ └────┬─────┘ └───────────┬───────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────┐
│ Score Update · Voice · Dashboard · Robot · API │
└─────────────────────────────────────────────────┘
The VLM model decides which tools to invoke. Score updates, announcements, and robot gestures — all through native tool calling.
Expose referee capabilities as standardized MCP tools. Any MCP client — Claude, ChatGPT, Cursor — can use Sideline as a referee tool.
Multiple Sideline agents covering different cameras coordinate decisions via Google's A2A protocol. Like VAR, but autonomous.
Video + reasoning + scoreboard. Always works, zero hardware needed.
Tank bot with camera, speaker, lidar. Powered by NASA JPL's ROSA framework.
6-DOF robotic arm via HuggingFace LeRobot. Signals calls with physical gestures.
Humanoid referee. Walks the court, gestures, speaks. The end goal.
Built at Nebius.Build SF — March 15, 2026
Edge AI, full-stack, inference optimization. Built the end-to-end agentic pipeline.
LinkedIn →CMU Robotics, AI PM. Self-driving systems, VLA models, embodied AI hackathon mentor.
LinkedIn →Actual sports referee. Managed 14 leagues, 6000 participants. AI engineer with MCP expertise.
LinkedIn →Palantir, USACO Gold, NeurIPS workshop paper. CS heavy hitter.
LinkedIn →