process_image_to_code

Instructions

Implementation Reference

• src/index.ts:212-249 (handler)

  Main execution handler for the 'process_image_to_code' tool. Downloads image from provided URL, constructs a Chinese prompt for generating code in the specified language (default Python), calls the Qwen VL model via processImageWithQwen helper, and returns the generated code or an error message.

  case "process_image_to_code": {
    const imageUrl = String(request.params.arguments?.image_url);
    const language = String(request.params.arguments?.language || "python");
    const instructions = String(request.params.arguments?.instructions || "");
    
    if (!imageUrl) {
      throw new Error("图像URL是必需的");
    }
    
    try {
      // 下载图像
      const imagePath = await downloadImage(imageUrl);
      
      // 构建提示词
      let prompt = `请根据这张图片生成${language}代码。`;
      if (instructions) {
        prompt += ` ${instructions}`;
      }
      
      // 处理图像
      const result = await processImageWithQwen(imagePath, prompt);
      
      return {
        content: [{
          type: "text",
          text: result
        }]
      };
    } catch (error: any) {
      return {
        content: [{
          type: "text",
          text: `处理图像失败: ${error.message}`
        }],
        isError: true
      };
    }
  }

• src/index.ts:155-177 (registration)

  Tool registration in the ListTools handler, defining name, description, and full input schema including required image_url, optional language (default 'python'), and instructions.

    name: "process_image_to_code",
    description: "处理图像并生成代码",
    inputSchema: {
      type: "object",
      properties: {
        image_url: {
          type: "string",
          description: "图像URL"
        },
        language: {
          type: "string",
          description: "目标编程语言，例如：python, javascript, html, css等",
          default: "python"
        },
        instructions: {
          type: "string",
          description: "额外的指令或要求",
          default: ""
        }
      },
      required: ["image_url"]
    }
  },

• src/index.ts:74-131 (helper)

  Core helper function that encodes the local image file to base64, constructs the API request payload for Qwen-VL-Plus model with user message containing text prompt and image, calls the Dashscope API, extracts the output text, handles errors, and deletes the temp image file.

  async function processImageWithQwen(imagePath: string, prompt: string): Promise<string> {
    try {
      // 读取图像文件
      const imageBuffer = fs.readFileSync(imagePath);
      const base64Image = imageBuffer.toString('base64');
      
      // 准备请求数据
      const requestData = {
        model: "qwen-vl-plus",
        input: {
          messages: [
            {
              role: "user",
              content: [
                {
                  type: "text",
                  text: prompt
                },
                {
                  type: "image",
                  image: base64Image
                }
              ]
            }
          ]
        },
        parameters: {}
      };
      
      // 发送请求到API
      const response = await axios.post(API_ENDPOINT, requestData, {
        headers: {
          'Authorization': `Bearer ${API_KEY}`,
          'Content-Type': 'application/json'
        }
      });
      
      // 处理响应
      if (response.data && response.data.output && response.data.output.text) {
        return response.data.output.text;
      } else {
        throw new Error("API响应格式不正确");
      }
    } catch (error: any) {
      console.error("调用qwen2.5-vl模型API失败:", error);
      if (error.response) {
        console.error("API响应:", error.response.data);
      }
      throw new Error(`处理图像失败: ${error.message}`);
    } finally {
      // 清理临时文件
      try {
        fs.unlinkSync(imagePath);
      } catch (e) {
        console.error("清理临时文件失败:", e);
      }
    }
  }

• src/index.ts:47-66 (helper)

  Supporting helper to stream-download the image from the provided URL to a uniquely-named temporary JPG file in the system's temp directory.

  async function downloadImage(imageUrl: string): Promise<string> {
    try {
      // 为图像生成唯一文件名
      const filename = `${Date.now()}-${Math.random().toString(36).substring(2, 15)}.jpg`;
      const filePath = path.join(TEMP_DIR, filename);
      
      // 下载图像
      const response = await axios({
        method: 'GET',
        url: imageUrl,
        responseType: 'stream'
      });
      
      await streamPipeline(response.data, fs.createWriteStream(filePath));
      return filePath;
    } catch (error: any) {
      console.error("下载图像失败:", error);
      throw new Error(`下载图像失败: ${error.message}`);
    }
  }