list-videos
List all videos in a twelvelabs video index. If you don't provide an indexId, we will use the default index.
This tool will return for each video of the index:
- the videoId: needed for video analysis.
- the original video filename (if there is one).
Use this tool if you don't know the videoId of the video the user wants to analyse, you can display the different filenames
and ask the user to select the video they want to analyse.
search
Search in twelvelabs videos using natural language descriptions. The twelvelabs platform interprets your query
to find matching content based on visual elements, actions, sounds, and on-screen text. The search is performed
on all videos in the index (or the default index if no index is provided). It will return the start and end time of the
matching segments in the video.
The search results include:
- Filename and videoId for each matching video
- user_metadata (if any was attached during indexing)
- Thumbnail URLs for each segment (pre-signed S3 URLs, valid for 1 hour)
- Start/end times and relevance rank for each segment
For each video that has matching segments, you should display:
- Filename of the video
- user_metadata key-value pairs
- A table of segments with rank, timecode, and a clickable hyperlink for the thumbnail (do NOT use markdown image syntax, use a text hyperlink like [View thumbnail](url) instead).
Supports three search modes:
- Text only: provide just a query string
- Image only: provide queryMediaUrl or queryMediaFile with queryMediaType "image"
- Composed (text + image): provide both query and an image (via URL or file path) for refined results (e.g. image of a car + "red color"). Requires Marengo 3.0.
For entity search, include the entity ID in the query using the format: "<@entity_id> action description"
create-entity-collection
Create a new entity collection. Entity collections group related entities (people or objects) for entity search. For example, create a collection per sports team or film cast. Requires Marengo 3.0.
list-entity-collections
List all entity collections. Entity collections contain entities (people/objects) used for entity search.
delete-entity-collection
Delete an entity collection and all its entities.
create-asset
Upload a reference image as an asset. Assets are used as reference images when creating entities for entity search. Provide either a publicly accessible image URL or an absolute path to a local image file. Use multiple reference images per entity for better accuracy across different angles and lighting.
create-entity
Create an entity within a collection. An entity represents a specific person or object you want to find in videos. You must provide at least one asset ID (reference image). Multiple reference images improve accuracy. After creating an entity, search for it using: <@entity_id> action description
list-entities
List all entities in an entity collection.
delete-entity
Delete an entity from a collection.
start-video-embeddings-task
Start a new twelvelabs video embeddings task that uploads a video to the platform and creates
one or multiple video embeddings.
get-video-embeddings-tasks
Get details about video embeddings tasks. If you provide a taskId, we will return details (including the status) of that task. If not
we will return the details of the latest 10 embeddings tasks that were created. Possible statuses are processing, ready, or failed.
retrieve-video-embeddings
Retrieves the video embeddings from a ready twelvelabs video embeddings task or from an indexed video.
analyse-video
Analyse a video with twelvelabs models. You will always need to provide a videoId to use the tool. If you don't
have a videoId, you need to use the list-videos tool first and ask the user to select the video they want to analyse. Videos
need to be indexed to be available for analysis, you can use the start-video-indexing-task tool to index a new video.
Provide a prompt to guide the analysis. The prompt can request titles, topics, hashtags, summaries, chapters, highlights,
or any open-ended analysis such as tables of content, action items, memos, and detailed analyses.