VideoDB Director

# Advanced Visual Search Pipelines [Source Link](https://docs.videodb.io/advanced-visual-search-pipelines-82)  VideoDB Documentation Pages [\\ \\ Welcome to VideoDB Docs](https://docs.videodb.io/) [\\ \\ Quick Start Guide](https://docs.videodb.io/quick-start-guide-38) [\\ \\ How Accurate is Your Search?](https://docs.videodb.io/how-accurate-is-your-search-88) [\\ \\ Video Indexing Guide](https://docs.videodb.io/video-indexing-guide-101) [\\ \\ Semantic Search](https://docs.videodb.io/semantic-search-89) [\\ \\ Collections](https://docs.videodb.io/collections-68) [\\ \\ Public Collections](https://docs.videodb.io/public-collections-102) [\\ \\ Callback Details](https://docs.videodb.io/callback-details-66) [\\ \\ Ref: Subtitle Styles](https://docs.videodb.io/ref-subtitle-styles-57) [\\ \\ Language Support](https://docs.videodb.io/language-support-79) [\\ \\ Guide: Subtitles](https://docs.videodb.io/guide-subtitles-73) [\\ \\ Visual Search and Indexing](https://docs.videodb.io/visual-search-and-indexing-80) [\\ \\ Scene Extraction Algorithms](https://docs.videodb.io/scene-extraction-algorithms-84) [\\ \\ Custom Annotations](https://docs.videodb.io/custom-annotations-81) [\\ \\ Scene-Level Metadata: Smarter Video Search & Retrieval](https://docs.videodb.io/scene-level-metadata-smarter-video-search-retrieval-107) [\\ \\ Advanced Visual Search Pipelines](https://docs.videodb.io/advanced-visual-search-pipelines-82) [\\ \\ Playground for Scene Extractions](https://docs.videodb.io/playground-for-scene-extractions-83) [\\ \\ Deep Dive into Prompt Engineering : Mastering Video Scene Indexing](https://docs.videodb.io/deep-dive-into-prompt-engineering-mastering-video-scene-indexing-93) [\\ \\ Multimodal Search](https://docs.videodb.io/multimodal-search-90) [\\ \\ Multimodal Search: Quickstart](https://docs.videodb.io/multimodal-search-quickstart-91) [\\ \\ Conference Slide Scraper with VideoDB](https://docs.videodb.io/conference-slide-scraper-with-videodb-92) [\\ \\ Dynamic Video Streams](https://docs.videodb.io/dynamic-video-streams-44) [\\ \\ Ref: TextAsset](https://docs.videodb.io/ref-textasset-74) [\\ \\ Guide : TextAsset](https://docs.videodb.io/guide-textasset-75) [\\ \\ Director - Video Agent Framework](https://docs.videodb.io/director-video-agent-framework-98) [\\ \\ Agent Creation Playbook](https://docs.videodb.io/agent-creation-playbook-103) [\\ \\ How I Built a CRM-integrated Sales Assistant Agent in 1 Hour](https://docs.videodb.io/how-i-built-a-crm-integrated-sales-assistant-agent-in-1-hour-106) [\\ \\ Make Your Video Sound Studio Quality with Voice Cloning](https://docs.videodb.io/make-your-video-sound-studio-quality-with-voice-cloning-105) [\\ \\ Setup Director Locally](https://docs.videodb.io/setup-director-locally-104) [\\ \\ Open Source Tools](https://docs.videodb.io/open-source-tools-94) [\\ \\ LlamaIndex VideoDB Retriever](https://docs.videodb.io/llamaindex-videodb-retriever-58) [\\ \\ PromptClip: Use Power of LLM to Create Clips](https://docs.videodb.io/promptclip-use-power-of-llm-to-create-clips-52) [\\ \\ StreamRAG: Connect ChatGPT to VideoDB](https://docs.videodb.io/streamrag-connect-chatgpt-to-videodb-43) [\\ \\ Examples and Tutorials](https://docs.videodb.io/examples-and-tutorials-35) [\\ \\ Dubbing - Replace Soundtrack with New Audio](https://docs.videodb.io/dubbing-replace-soundtrack-with-new-audio-49) [\\ \\ Beep curse words in real-time](https://docs.videodb.io/beep-curse-words-in-real-time-53) [\\ \\ Remove Unwanted Content from videos](https://docs.videodb.io/remove-unwanted-content-from-videos-5) [\\ \\ Instant Clips of Your Favorite Characters](https://docs.videodb.io/instant-clips-of-your-favorite-characters-3) [\\ \\ Insert Dynamic Ads in real-time](https://docs.videodb.io/insert-dynamic-ads-in-real-time-7) [\\ \\ Adding Brand Elements with VideoDB](https://docs.videodb.io/adding-brand-elements-with-videodb-76) [\\ \\ Revolutionize Video Editing with VideoDb: Effortless Ad Placement and Seamless Video Integration](https://docs.videodb.io/revolutionize-video-editing-with-videodb-effortless-ad-placement-8) [\\ \\ Eleven Labs x VideoDB: Adding AI Generated voiceovers to silent footage](https://docs.videodb.io/eleven-labs-x-videodb-adding-ai-generated-voiceovers-to-silent-f-59) [\\ \\ Elevating Trailers with Automated Narration](https://docs.videodb.io/elevating-trailers-with-automated-narration-60) [\\ \\ Add Intro/Outro to Videos](https://docs.videodb.io/add-intro-outro-to-videos-61) [\\ \\ Enhancing Video Captions with VideoDB Subtitle Styling](https://docs.videodb.io/enhancing-video-captions-with-videodb-subtitle-styling-62) [\\ \\ Audio overlay + Video + Timeline](https://docs.videodb.io/audio-overlay-video-timeline-63) [\\ \\ Building Dynamic Video Streams with VideoDB: Integrating Custom Data and APIs](https://docs.videodb.io/building-dynamic-video-streams-with-videodb-integrating-custom-d-85) [\\ \\ Adding AI Generated Voiceovers with VideoDB and LOVO](https://docs.videodb.io/adding-ai-generated-voiceovers-with-videodb-and-lovo-70) [\\ \\ AI Generated Ad Films for Product Videography: Wellsaid, Open AI & VideoDB](https://docs.videodb.io/ai-generated-ad-films-for-product-videography-wellsaid-open-ai-v-71) [\\ \\ Fun with Keyword Search](https://docs.videodb.io/fun-with-keyword-search-77) [\\ \\ AWS Rekognition and VideoDB - Intelligent Video Clips](https://docs.videodb.io/aws-rekognition-and-videodb-intelligent-video-clips-4) [\\ \\ AWS Rekognition and VideoDB - Effortlessly Remove Inappropriate Content from Video](https://docs.videodb.io/aws-rekognition-and-videodb-effortlessly-remove-inappropriate-co-6) [\\ \\ Overlay a Word-Counter on Video Stream](https://docs.videodb.io/overlay-a-word-counter-on-video-stream-86) [\\ \\ Generate Automated Video Outputs with Text Prompts \| DALL-E + ElevenLabs + OpenAI + VideoDB](https://docs.videodb.io/generate-automated-video-outputs-with-text-prompts-dall-e-eleven-87) [\\ \\ Edge of Knowledge](https://docs.videodb.io/edge-of-knowledge-10) [\\ \\ Building Intelligent Machines](https://docs.videodb.io/building-intelligent-machines-16) [\\ \\ Part 1 - Define Intelligence](https://docs.videodb.io/part-1-define-intelligence-17) [\\ \\ Part 2 - Observe and Respond](https://docs.videodb.io/part-2-observe-and-respond-18) [\\ \\ Part 3 - Training a Model](https://docs.videodb.io/part-3-training-a-model-19) [\\ \\ Society of Machines](https://docs.videodb.io/society-of-machines-20) [\\ \\ Society of Machines](https://docs.videodb.io/society-of-machines-23) [\\ \\ Autonomy - Do we have the choice?](https://docs.videodb.io/autonomy-do-we-have-the-choice-21) [\\ \\ Emergence - An Intelligence of the collective](https://docs.videodb.io/emergence-an-intelligence-of-the-collective-22) [\\ \\ Drafts](https://docs.videodb.io/drafts-24) [\\ \\ From Language Models to World Models: The Next Frontier in AI](https://docs.videodb.io/from-language-models-to-world-models-the-next-frontier-in-ai-65) [\\ \\ The Future Series](https://docs.videodb.io/the-future-series-78) [\\ \\ Building World's First Video Database](https://docs.videodb.io/building-worlds-first-video-database-25) [\\ \\ Multimedia: From MP3/MP4 to the Future with VideoDB](https://docs.videodb.io/multimedia-from-mp3-mp4-to-the-future-with-videodb-26) [\\ \\ Introducing VideoDB: The Pinnacle of Synchronized Video Streaming for the Modern Web](https://docs.videodb.io/introducing-videodb-the-pinnacle-of-synchronized-video-streaming-27) [\\ \\ Dynamic Video Streams](https://docs.videodb.io/dynamic-video-streams-50) [\\ \\ Why do we need a Video Database Now?](https://docs.videodb.io/why-do-we-need-a-video-database-now-41) [\\ \\ What's a Video Database ?](https://docs.videodb.io/whats-a-video-database-36) [\\ \\ Enhancing AI-Driven Multimedia Applications](https://docs.videodb.io/enhancing-ai-driven-multimedia-applications-37) [\\ \\ Misalignment of Today's Web](https://docs.videodb.io/misalignment-of-todays-web-67) [\\ \\ Beyond Traditional Video Infrastructure](https://docs.videodb.io/beyond-traditional-video-infrastructure-28) [\\ \\ Research Grants](https://docs.videodb.io/research-grants-96) [\\ \\ Team](https://docs.videodb.io/team-46) [\\ \\ Internship: Build the Future of AI-Powered Video Infrastructure](https://docs.videodb.io/internship-build-the-future-of-ai-powered-video-infrastructure-97) [\\ \\ Ashutosh Trivedi](https://docs.videodb.io/ashutosh-trivedi-32) [\\ \\ Playlists](https://docs.videodb.io/playlists-33) [\\ \\ Talks - Solving Logical Puzzles with Natural Language Processing - PyCon India 2015](https://docs.videodb.io/talks-solving-logical-puzzles-with-natural-language-processing-p-34) [\\ \\ Ashish](https://docs.videodb.io/ashish-45) [\\ \\ Shivani Desai](https://docs.videodb.io/shivani-desai-48) [\\ \\ Gaurav Tyagi](https://docs.videodb.io/gaurav-tyagi-51) [\\ \\ Rohit Garg](https://docs.videodb.io/rohit-garg-64) [\\ \\ Customer Love](https://docs.videodb.io/customer-love-42) [\\ \\ Temp Doc](https://docs.videodb.io/temp-doc-54) Visual Search and Indexing #  Advanced Visual Search Pipelines [](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/guides/scene-index/advanced_visual_search.ipynb) Let's deep dive into Scene and Frame objects ### Scene A Scene object describes a unique event in the video. From a timeline perspective it’s a timestamp range.  video\_id : id of the video object start : seconds end : seconds description : string description Each scene object has an attribute frames, that has list of Frame objects. ### Frame Each Scene can be described by a list of frames. Each Frame object primarily has the URL of the image and its description field.  id : ID of the frame object url : URL of the image frame\_time : Timestamp of the frame in the video description : string description video\_id : id of the video object scene\_id : id of the scene object  We provide you with easy-to-use Objects and Functions to bring flexibility in designing your visual understanding pipeline. With these tools, you have the freedom to: Extract scene according to your use case. Go to frame level abstraction. Assign label, custom model description for each frame. Use of multiple models, prompts for each scene or frame to convert information to text. Send multiple frames to vision model for better temporal activity understanding. ### extract\_scenes() This function accepts the extraction\_type and extraction\_config and returns a [SceneCollection](https://docs.videodb.io/playground-for-scene-extractions-83) object, which keep information about all the extracted scene lists. Checkout [\\ Scene Extraction Algorithms](https://docs.videodb.io/scene-extraction-algorithms-84) for more details. scene\_collection = video.extract\_scenes( extraction\_type=SceneExtractionType.time\_based, extraction\_config={"time": 30, "select\_frames": \["middle"\]}, ) ### Capture Temporal Change Vision models excel at describing images, but videos present an added complexity due to the temporal changes in the information. With our pipeline, you can maintain image-level understanding in frames and combine them using LLMs at the scene level to capture temporal or activity-related understanding. You have freedom to iterate through each scene and frame level to describe the information for indexing purposes. Get scene collection scene\_collection = video.get\_scene\_collection("scene\_collection\_id") ### Iterate through each scene and frame Iterate over scenes and frames and attach description coming from external pipeline be it custom CV pipeline or custom model descriptions. print("This is scene collection id", scene\_collection.id) print("This is scene collection config", scene\_collection.config) \# get scene from collection scenes = scene\_collection.scenes \# Iterate through each scene for scene in scenes: print(f"Scene Duration {scene.start}-{scene.end}") # Iterate through each frame in the scene for frame in scene.frames: print(f"Frame at {frame.frame\_time} {frame.url}") frame.description = "bring text from external sources/ pipeline" ) ### Create Scene by custom annotation These annotations can come from your application or from external vision model, if you extract the description using any vision LLM for scene in scenes: scene.description = "summary of frame level description" Using this pipeline, you have the freedom to design your own flow. In the example above, we’ve described each frame in the scene independently, but some vision models allow multiple images in one go as well. Feel free to customise your flow as per your needs. Experiment with sending multiple frames to a vision model. Utilize prompts to describe multiple frames, then assign these descriptions to the scene. Integrate your own vision model into the pipeline.  We’ll soon be adding more details and strategies for effective and advanced multimodal search. We welcome your input on what strategies have worked best in your specific use cases Here’s our 🎙️ [Discord](https://discord.gg/py9P639jGz) channel where we brainstorm about such ideas. Once you have a description of each scene in place, you can index and search for the information using the following functions. from videodb import IndexType #create new index and assign a name to it index\_id = video.index\_scenes(scenes=scenes, name="My Custom Model") \# search using the index\_id res = video.search(query="first 29 sec", index\_type=IndexType.scene, index\_id=index\_id) res.play() Scene Frame extract\_scenes() Capture Temporal Change Iterate through each scene and frame Create Scene by custom annotation Want to print your doc? This is not the way.  Try clicking the ⋯ next to your doc name or using a keyboard shortcut ( CtrlP ) instead. ---