I built a workflow to auto-generate highlight clips from long videos.
I've been experimenting with a custom workflow to turn long-form videos into short, engaging highlight clips. I know a bunch of startups are in this space, but I wanted to build something lightweight and flexible just for fun — and ended up integrating it into one of my products.
Here’s the general flow:
Transcribe the video
Use tools like Whisper, AssemblyAI, or Deepgram to get an accurate transcript.
Extract interesting clips
Feed the transcript into an LLM (I recommend Gemini because of its long context and quality) and prompt it to segment the video based on criteria like "viral", "engaging", "funny", etc. Make sure it returns timestamps.
Generate the clips
Use ffmpeg to slice the original video using those timestamps.
(Optional) Auto-crop for vertical
If the user selects a mobile/short-form format, use something like Sieve to auto-crop and center the subject for vertical (9:16) output.
(Bonus) Enhance captions
Run the extracted clip transcripts back through the LLM to pick out keywords or phrases to emphasize in the captions.
Add captions with styling
Use Remotion or similar to render the clips with styled, animated captions. The component handles logic for timing and highlighting.
Render and download
Batch render your clips and you’re done.
I've been experimenting with a custom workflow to turn long-form videos into short, engaging highlight clips. I know a bunch of startups are in this space, but I wanted to build something lightweight and flexible just for fun — and ended up integrating it into one of my products.
Here’s the general flow:
Transcribe the video Use tools like Whisper, AssemblyAI, or Deepgram to get an accurate transcript.
Extract interesting clips Feed the transcript into an LLM (I recommend Gemini because of its long context and quality) and prompt it to segment the video based on criteria like "viral", "engaging", "funny", etc. Make sure it returns timestamps.
Generate the clips Use ffmpeg to slice the original video using those timestamps.
(Optional) Auto-crop for vertical If the user selects a mobile/short-form format, use something like Sieve to auto-crop and center the subject for vertical (9:16) output.
(Bonus) Enhance captions Run the extracted clip transcripts back through the LLM to pick out keywords or phrases to emphasize in the captions.
Add captions with styling Use Remotion or similar to render the clips with styled, animated captions. The component handles logic for timing and highlighting.
Render and download Batch render your clips and you’re done.
I used this exact pipeline to build a feature in one of my tools: https://www.shortsgenerator.com/highlights-generator
Appreciate any feedback!