I Built a Multimodal AI to Detect Pet Emotions from Video — Full Python Breakdown

Ever looked at your dog mid-zoom and thought: "Is this joy or a cry for help?" I did. So I built something. This is a walkthrough of how I trained a lightweight multimodal classifier to detect emotional states in pets using short video clips — and how I deployed it as a real web app at mypettherapist.com . Spoiler: the hardest part wasn't the model. It was the data. The Problem With Pet Emotion AI Most pet AI is a parlor trick. "Oh look, the model says your cat is surprised ." Cool. But surprise is not actionable. What is actionable: Is my pet anxious right now? Is this behavior getting worse over time? Should I call a vet? That's what I wanted to build. A system that gives pet owners behavioral signals , not meme labels. Architecture Overview Video Clip (5–15s) │ ▼ Frame Sampler (every 0.5s) │ ▼ ┌──────────────────────────────┐ │ MobileNetV3 (vision) │ ← body posture │ Whisper-tiny (audio) │ ← vocalizations │ Pose Keypoints (MediaPipe) │ ← tail, ears, spine └──────────────────────────