video_translation_dubbing
Translate video speech into different languages and generate AI-voiced dubs for accessible multilingual content.
Instructions
[AllVoiceLab Tool] Translate and dub video speech into a different language with AI-generated voices.
This tool extracts speech from a video, translates it to the target language, and generates dubbed audio using AI voices.
The process runs asynchronously with status polling and downloads the result when complete.
Args:
video_file_path: Path to the video or audio file to process. Supports MP4, MOV, MP3, and WAV formats. Maximum file size: 2GB.
target_lang: Target language code for translation (e.g., 'en', 'zh', 'ja', 'fr', 'de', 'ko'). Required.
source_lang: Source language code of the original content. Set to 'auto' for automatic language detection. Default is 'auto'.
name: Optional project name for identification purposes.
output_dir: Output directory for the downloaded result file. Default is user's desktop.
Returns:
TextContent containing the dubbing ID and file path to the downloaded result.
If the process takes longer than expected, returns only the dubbing ID for later status checking.
Limitations:
- Only MP4, MOV, MP3, and WAV formats are supported
- Maximum file size: 2GB
- Processing may take several minutes depending on content length and complexity
- Translation quality depends on speech clarity in the original content
- Currently supports a limited set of languages for translationInput Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| video_file_path | Yes | ||
| target_lang | Yes | ||
| source_lang | No | auto | |
| name | No | ||
| output_dir | No |
Implementation Reference
- allvoicelab_mcp/tools/dubbing.py:249-433 (handler)The handler function that executes the video_translation_dubbing tool. It validates inputs, submits the dubbing request to AllVoiceLab API, polls for completion, and downloads the result audio file.
def video_translation_dubbing( video_file_path: str, target_lang: str, source_lang: str = "auto", name: str = None, output_dir: str = None ) -> TextContent: """ Translate and dub video speech into a different language with AI-generated voices Args: video_file_path: Path to the video or audio file to process. Supports MP4, MOV, MP3, and WAV formats. Maximum file size: 2GB. target_lang: Target language code for translation (e.g., 'en', 'zh', 'ja', 'fr', 'de', 'ko'). Required. source_lang: Source language code of the original content. Set to 'auto' for automatic language detection. Default is 'auto'. name: Optional project name for identification purposes. output_dir: Output directory for the downloaded result file. Default is user's desktop. Returns: TextContent: Text content containing the dubbing ID and file path to the downloaded result. If the process takes longer than expected, returns only the dubbing ID for later status checking. """ all_voice_lab = get_client() output_dir = all_voice_lab.get_output_path(output_dir) max_polling_time = 600 polling_interval = 10 logging.info(f"Tool called: video_translation_dubbing") logging.info(f"Video file path: {video_file_path}") logging.info(f"Target language: {target_lang}, Source language: {source_lang}") if name: logging.info(f"Project name: {name}") logging.info(f"Output directory: {output_dir}") logging.info(f"Max polling time: {max_polling_time}s, Polling interval: {polling_interval}s") # Validate parameters if not video_file_path: logging.warning("Video file path parameter is empty") return TextContent( type="text", text="video_file_path parameter cannot be empty" ) # Check if video file exists before processing if not os.path.exists(video_file_path): logging.warning(f"Video file does not exist: {video_file_path}") return TextContent( type="text", text=f"Video file does not exist: {video_file_path}" ) # Check file format, only allow mp4 and mov _, file_extension = os.path.splitext(video_file_path) file_extension = file_extension.lower() if file_extension not in [".mp4", ".mov", ".mp3", ".wav"]: logging.warning(f"Unsupported video file format: {file_extension}") return TextContent( type="text", text=f"Unsupported video file format. Only MP4, MOV, MP3 and WAV formats are supported." ) # Check file size, limit to 2GB max_size_bytes = 2 * 1024 * 1024 * 1024 # 2GB in bytes file_size = os.path.getsize(video_file_path) if file_size > max_size_bytes: logging.warning(f"Video file size exceeds limit: {file_size} bytes, max allowed: {max_size_bytes} bytes") return TextContent( type="text", text=f"Video file size exceeds the maximum limit of 2GB. Please use a smaller file." ) # Validate target language if not target_lang: logging.warning(f"target language is empty") return TextContent( type="text", text="target language parameter cannot be empty" ) # Validate output directory if not output_dir: logging.warning("Output directory parameter is empty") return TextContent( type="text", text="output_dir parameter cannot be empty" ) # Try to create output directory if it doesn't exist try: os.makedirs(output_dir, exist_ok=True) except Exception as e: logging.error(f"Failed to create output directory: {output_dir}, error: {str(e)}") return TextContent( type="text", text=f"Failed to create output directory: {output_dir}" ) try: # Submit dubbing request logging.info("Starting video dubbing process") dubbing_id = all_voice_lab.dubbing( video_file_path=video_file_path, target_lang=target_lang, source_lang=source_lang, name=name, watermark=True, drop_background_audio=False ) logging.info(f"Video dubbing request successful, dubbing ID: {dubbing_id}") # Start polling for task completion logging.info(f"Starting to poll dubbing status for ID: {dubbing_id}") start_time = time.time() completed = False file_path = None # Poll until task is completed or timeout while time.time() - start_time < max_polling_time: try: # Get dubbing info dubbing_info = all_voice_lab.get_dubbing_info(dubbing_id) logging.info(f"Dubbing status: {dubbing_info.status} for ID: {dubbing_id}") # Check if dubbing is completed if dubbing_info.status.lower() == "success": logging.info(f"Dubbing completed for ID: {dubbing_id}") completed = True break # Check if dubbing failed elif dubbing_info.status.lower() in ["failed", "error"]: logging.error(f"Dubbing failed for ID: {dubbing_id}") return TextContent( type="text", text=f"Video dubbing failed. Please try again later.\n" f"Dubbing ID: {dubbing_id}\n" ) # Wait for next polling interval logging.info(f"Waiting {polling_interval} seconds before next poll") time.sleep(polling_interval) except Exception as e: logging.error(f"Error polling dubbing status: {str(e)}") time.sleep(polling_interval) # Continue polling despite errors # Check if polling timed out if not completed: logging.warning(f"Polling timed out after {max_polling_time} seconds for dubbing ID: {dubbing_id}") return TextContent( type="text", text=f"Video dubbing is still in progress. Your dubbing ID is: {dubbing_id}\n" f"The process is taking longer than expected. You can check the status later using this ID.\n" ) # Download the file if dubbing completed try: logging.info(f"Downloading dubbing audio for ID: {dubbing_id}") file_path = all_voice_lab.get_dubbing_audio(dubbing_id, output_dir, video_file_path) logging.info(f"Dubbing audio downloaded successfully, file saved at: {file_path}") return TextContent( type="text", text=f"Video dubbing completed successfully!\n" f"Dubbing ID: {dubbing_id}\n" f"File saved at: {file_path}\n" ) except Exception as e: logging.error(f"Failed to download dubbing audio: {str(e)}") return TextContent( type="text", text=f"Video dubbing completed, but failed to download the audio file.\n" f"Dubbing ID: {dubbing_id}\n" f"Error: {str(e)}\n" ) except FileNotFoundError as e: logging.error(f"Video file does not exist: {video_file_path}, error: {str(e)}") return TextContent( type="text", text=f"Video file does not exist: {video_file_path}" ) except Exception as e: logging.error(f"Video dubbing failed: {str(e)}") return TextContent( type="text", text=f"Video dubbing failed, tool temporarily unavailable" ) - allvoicelab_mcp/server.py:197-222 (registration)Registers the video_translation_dubbing tool with the MCP server using FastMCP, including name, detailed description (serving as schema), and binding the handler function.
mcp.tool( name="video_translation_dubbing", description="""[AllVoiceLab Tool] Translate and dub video speech into a different language with AI-generated voices. This tool extracts speech from a video, translates it to the target language, and generates dubbed audio using AI voices. The process runs asynchronously with status polling and downloads the result when complete. Args: video_file_path: Path to the video or audio file to process. Supports MP4, MOV, MP3, and WAV formats. Maximum file size: 2GB. target_lang: Target language code for translation (e.g., 'en', 'zh', 'ja', 'fr', 'de', 'ko'). Required. source_lang: Source language code of the original content. Set to 'auto' for automatic language detection. Default is 'auto'. name: Optional project name for identification purposes. output_dir: Output directory for the downloaded result file. Default is user's desktop. Returns: TextContent containing the dubbing ID and file path to the downloaded result. If the process takes longer than expected, returns only the dubbing ID for later status checking. Limitations: - Only MP4, MOV, MP3, and WAV formats are supported - Maximum file size: 2GB - Processing may take several minutes depending on content length and complexity - Translation quality depends on speech clarity in the original content - Currently supports a limited set of languages for translation """ )(video_translation_dubbing) - allvoicelab_mcp/server.py:199-221 (schema)The tool description in the MCP registration provides input/output schema details, arguments, returns, and limitations.
description="""[AllVoiceLab Tool] Translate and dub video speech into a different language with AI-generated voices. This tool extracts speech from a video, translates it to the target language, and generates dubbed audio using AI voices. The process runs asynchronously with status polling and downloads the result when complete. Args: video_file_path: Path to the video or audio file to process. Supports MP4, MOV, MP3, and WAV formats. Maximum file size: 2GB. target_lang: Target language code for translation (e.g., 'en', 'zh', 'ja', 'fr', 'de', 'ko'). Required. source_lang: Source language code of the original content. Set to 'auto' for automatic language detection. Default is 'auto'. name: Optional project name for identification purposes. output_dir: Output directory for the downloaded result file. Default is user's desktop. Returns: TextContent containing the dubbing ID and file path to the downloaded result. If the process takes longer than expected, returns only the dubbing ID for later status checking. Limitations: - Only MP4, MOV, MP3, and WAV formats are supported - Maximum file size: 2GB - Processing may take several minutes depending on content length and complexity - Translation quality depends on speech clarity in the original content - Currently supports a limited set of languages for translation """ - Helper function to retrieve status of a dubbing task, referenced by the video_translation_dubbing tool.
def get_dubbing_info(dubbing_id: str) -> TextContent: """ Retrieve status and details of a video dubbing task Args: dubbing_id: The unique identifier of the dubbing task to check. This ID is returned from the video_dubbing or video_translation_dubbing tool. Required. Returns: TextContent: Text content containing the status (e.g., "pending", "processing", "success", "failed") and other details of the dubbing task. """ all_voice_lab = get_client() logging.info(f"Tool called: get_dubbing_info") logging.info(f"Dubbing ID: {dubbing_id}") # Validate parameters if not dubbing_id: logging.warning("Dubbing ID parameter is empty") return TextContent( type="text", text="dubbing_id parameter cannot be empty" ) try: logging.info("Getting dubbing task information") dubbing_info = all_voice_lab.get_dubbing_info(dubbing_id) logging.info(f"Dubbing info retrieved successfully for ID: {dubbing_id}") # Format the result buffer = [] buffer.append(f"Dubbing ID: {dubbing_info.dubbing_id}\n") buffer.append(f"Status: {dubbing_info.status}\n") if dubbing_info.name: buffer.append(f"Project Name: {dubbing_info.name}\n") buffer.append( "Note: If the task has not been completed, you may need to explicitly inform the user of the task ID when responding.\n") # Join the list into a string result = "".join(buffer) return TextContent( type="text", text=result ) except Exception as e: logging.error(f"Failed to get dubbing information: {str(e)}") return TextContent( type="text", text=f"Failed to get dubbing information, tool temporarily unavailable" )