AI Metadata Generation Backend Platform
Asynchronous LLM-based metadata generation system for short-form video content
Overview
Description:
An internal backend platform responsible for generating and managing metadata (titles and descriptions) for short-form video content as part of a large-scale automation pipeline.
Usage:
Used by multiple internal automation services as a post-processing step after video creation.
Environment:
Production system handling high-volume asynchronous workloads
Context
Background:
Metadata for video content was previously created manually by editors using external AI tools.
Scale:
Thousands of videos generated daily, requiring a scalable and reliable metadata generation solution.
Constraints:
- External LLM APIs with rate limits and unstable response formats
- High throughput requirements
- Cost sensitivity at scale
The Challenge
Manual metadata generation became infeasible as content volume increased.
Pain Points:
- High manual effort and long processing time
- Inconsistent metadata quality across editors
- Synchronous API-based approaches failing under scale and rate limits
Architecture
Pattern:
Asynchronous batch processing with message-driven orchestration
High-Level Flow:
- 1.Scheduled job queries videos requiring metadata generation
- 2.Video data is published to Kafka topics
- 3.Batch workers consume messages every 3 minutes to control throughput
- 4.Metadata is generated via LLM providers and persisted
- 5.Downstream services retrieve metadata and track usage state
LLM Strategy
Design:
Unified abstraction layer to decouple business logic from specific LLM providers.
Features:
- Provider prioritization via configuration
- API key rotation to mitigate rate limits
- Automatic fallback on timeout or invalid responses
- Strict response format validation
Backend Responsibilities
- Job orchestration and scheduling
- Batch throughput control
- Retry and failure isolation
- Prompt optimization for cost reduction
- Usage and consumption tracking
Performance
Throughput:
Best Case: ≈25,000 metadata records/day
Worst Case: ≈5,000 metadata records/day
Processing Model:
Batch processing every 3 minutes to balance latency and API stability
Impact
Automation:
Fully automated metadata generation at scale
Operational Benefits:
- Significant reduction in manual workload
- Consistent metadata quality
- Improved system reliability under peak load
Challenges & Tradeoffs
- Balancing throughput with external API rate limits
- Choosing batch processing over synchronous requests for stability
- Designing idempotent processing to safely retry failed jobs
- Managing cost while maintaining acceptable metadata quality
Project Intel
My Role
Backend Developer
Tech Stack
Repositories
List of Projects
01. Internal Backend Platform
AI Metadata Generation Backend Platform
02. Internal Backend Pipeline
AI Video Generation Backend Pipeline
03. Internal Backend Coordination System
Video Distribution & Channel Coordination Backend
04. Public Website
Starfish Language Center Website
05. Web Application