Creating an Efficient Video Deduplication System

When you aggregate trending videos from 9 regions, the same video often appears in multiple regions simultaneously. A K-pop release might trend in KR, TW, TH, SG, VN, HK, US, and GB — all at the same time. Without deduplication, your database fills with redundant copies. Here's how I handle deduplication on TopVideoHub . The Simple Case: Same Video ID The easiest deduplication is by YouTube video ID. The same video trending in multiple regions has the same ID: class VideoDeduplicator { public function __construct ( private readonly \PDO $db ) {} /** * Insert or merge a video. Returns the internal ID. */ public function upsert ( array $videoData , string $region ): int { $existing = $this -> db -> prepare ( "SELECT id FROM videos WHERE video_id = ?" ); $existing -> execute ([ $videoData [ 'video_id' ]]); $row = $existing -> fetch ( \PDO :: FETCH_ASSOC ); if ( $row ) { // Video exists — add this region $this -> addRegion (( int ) $row [ 'id' ], $region , $videoData ); return ( int ) $row