baidu-ai-search

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# AppBuilder-Knowledge：生产环境的知识库/文档/切片管理教学\n", "\n", "[知识库组件](https://github.com/baidubce/app-builder/blob/master/docs/BasisModule/Platform/KnowledgeBase/knowledgebase.md)\n", "\n", "知识库组件（KnowledgeBase）是对线上知识库操作的组件，可以通过SDK实现创建知识库、添加知识文档、查询知识库文档、删除知识文档等操作，可在平台console中查看结果。对console端知识库进行操作，可以通过SDK实现创建知识库、添加知识文档、查询知识库文档、删除知识文档等操作，可在平台console中查看结果\n", "\n", "应用场景：通过SDK代码实现console端知识库操作。\n", "\n", "#### AppBuilder-Knowledge代码态使用流程\n", "\n", "##### 新建一个知识库\n", "\n", "- 首先我们需要通过SDK新建一个知识库，新建知识库的代码示例如下：" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "新建的知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n", "新建的知识库名称: my_knowledge\n" ] } ], "source": [ "import os\n", "import appbuilder\n", "\n", "os.environ[\"APPBUILDER_TOKEN\"] = \"your api key\"\n", "\n", "knowledge = appbuilder.KnowledgeBase()\n", "resp = knowledge.create_knowledge_base(\n", " name=\"my_knowledge\",\n", " description=\"my_knowledge\",\n", " type=\"public\",\n", " )\n", "print(\"新建的知识库ID: \", resp.id)\n", "print(\"新建的知识库名称: \", resp.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 实例化已创建的知识库 \n", "- 接下来如果我们期望对知识库进行进一步操作,就需要实例化已创建的知识库 " ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n" ] } ], "source": [ "my_knowledge_base_id = resp.id # 传入知识库ID\n", "my_knowledge = appbuilder.KnowledgeBase(my_knowledge_base_id)\n", "print(\"知识库ID: \", my_knowledge.knowledge_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 我们就成功在Appbuilder平台创建了一个名为my_knowledge的知识库\n", "\n", "\n", "\n", "##### 获取知识库详情\n", "- 如果我们想要获取知识库详情，可以调用 `get_knowledge_base_detail` 方法,返回的response将包含知识库的详情信息" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "查询的知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n", "查询的知识库名称: my_knowledge\n" ] } ], "source": [ "knowledge = appbuilder.KnowledgeBase()\n", "resp = knowledge.get_knowledge_base_detail(my_knowledge.knowledge_id)\n", "print(\"查询的知识库ID: \", resp.id)\n", "print(\"查询的知识库名称: \", resp.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 获取知识库列表\n", "- 当你想查看你的账号的知识库，SDK同样支持调用知识库列表接口，获取知识库列表。需要提供知识库id作为起始搜索位置，不提供则会默认从第一个知识库开始搜索" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "获取到的知识库列表: requestId='8e6d9bd4-73c2-41b9-9813-de65a5ca01ec' data=[KnowledgeBaseDetailResponse(id='271bba8b-2c77-4b52-aaa2-2d95dde9184a', name='千帆大模型平台文档', description=None, config=KnowledgeBaseConfig(index=KnowledgeBaseConfigIndex(type='public', esUrl='', username=None, password=None)))] marker='3cc48493-f9ff-48c8-a3ea-fcc17fdddded' nextMarker='271bba8b-2c77-4b52-aaa2-2d95dde9184a' maxKeys=1 isTruncated=False\n" ] } ], "source": [ "knowledge = appbuilder.KnowledgeBase()\n", "# 从my_knowledge知识库开始检索，检索10条\n", "resp = knowledge.get_knowledge_base_list(my_knowledge.knowledge_id,10)\n", "print(\"获取到的知识库列表: \", resp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 修改知识库\n", "- 如果我们对我们创建的知识库不满意，想要对其修改，SDK提供`modify_knowledge_base`方法，可以修改知识库,在下面的示例中我们将展示如何修改知识库的名称，接着我们按照知识库的id，查询知识库，查询知识库的名称是否改变,当然我们也可以去appbuilder平台个人空间去查看知识库名称是否更新\n" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "查询的知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n", "查询的知识库名称: new_test_knowledge\n" ] } ], "source": [ "knowledge = appbuilder.KnowledgeBase()\n", "knowledge.modify_knowledge_base(my_knowledge.knowledge_id, name=\"new_test_knowledge\", description=\"测试\")\n", "\n", "# 接着我们按照知识库的id，查询知识库，查询知识库的名称是否改变\n", "resp = knowledge.get_knowledge_base_detail(my_knowledge.knowledge_id)\n", "print(\"查询的知识库ID: \", resp.id)\n", "print(\"查询的知识库名称: \", resp.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 上传文档到知识库\n", "上传文档到知识库共有以下三种方式：\n", "- 上传文档到知识库\n", "- 上传通用文档\n", "- 向知识库添加文档" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "获取目标文档的file_url" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": [ "file_url = \"https://bj.bcebos.com/v1/appbuilder-sdk-components/%E3%80%8A%E9%BB%91%E7%A5%9E%E8%AF%9D%EF%BC%9A%E6%82%9F%E7%A9%BA%E3%80%8BIP%E4%BB%8B%E7%BB%8D%E6%89%8B%E5%86%8C2024.pdf?authorization=bce-auth-v1%2FALTAKGa8m4qCUasgoljdEDAzLm%2F2024-08-21T11%3A40%3A45Z%2F-1%2Fhost%2Ff387e2cd2db95c9da26247c5d6c45f6dc9a817fa2721f0a780dc1eba433d02b8\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### 上传文档到知识库\n", "- 主要提供自定义文档处理策略，向知识库添加文档\n", " - 文档格式：rawText (允许配置后续分割策略)\n", " - 文档处理策略" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'requestId': 'f698099c-b533-4500-a1f8-6b9f2087d1ba'}" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "knowledge_base_id = my_knowledge.knowledge_id\n", "knowledge.create_documents(\n", "\tid=knowledge_base_id,\n", "\tcontentFormat=\"rawText\",\n", "\tsource=appbuilder.DocumentSource(\n", "\t\ttype=\"web\",\n", "\t\turls=[\"https://baike.baidu.com/item/%E9%BB%91%E7%A5%9E%E8%AF%9D%EF%BC%9A%E6%82%9F%E7%A9%BA/53303078?fr=ge_ala\"],\n", "\t\turlDepth=1,\n", "\t),\n", "\tprocessOption=appbuilder.DocumentProcessOption(\n", "\t\ttemplate=\"custom\",\n", "\t\tparser=appbuilder.DocumentChoices(\n", "\t\t\tchoices=[\"layoutAnalysis\", \"ocr\"]\n", "\t\t),\n", "\t\tchunker=appbuilder.DocumentChunker(\n", "\t\t\tchoices=[\"separator\"],\n", "\t\t\tseparator=appbuilder.DocumentSeparator(\n", "\t\t\t\tseparators=[\"。\"],\n", "\t\t\t\ttargetLength=300,\n", "\t\t\t\toverlapRate=0.25,\n", "\t\t\t),\n", "\t\t\tprependInfo=[\"title\", \"filename\"],\n", "\t\t),\n", "\t\tknowledgeAugmentation=appbuilder.DocumentChoices(choices=[\"faq\"]),\n", "\t),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### 上传通用文档\n", "- SDK支持基于代码态的文档上传方法`upload_file`,以及包含文档的自定义切分逻辑的向知识库添加文档的方法``add_document`\n", "- 需要先将文档下载至本地" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "文件已成功保存到 黑神话(悟空).pdf\n" ] } ], "source": [ "import requests\n", "\n", "def download_file(url, file_path):\n", " try:\n", " response = requests.get(url, stream=True)\n", " response.raise_for_status()\n", " with open(file_path, 'wb') as file:\n", " for chunk in response.iter_content(chunk_size=8192):\n", " if chunk:\n", " file.write(chunk)\n", " print(f\"文件已成功保存到 {file_path}\")\n", " except requests.RequestException as e:\n", " print(f\"下载失败: {e}\")\n", "\n", "file_url = \"https://bj.bcebos.com/v1/appbuilder-sdk-components/%E3%80%8A%E9%BB%91%E7%A5%9E%E8%AF%9D%EF%BC%9A%E6%82%9F%E7%A9%BA%E3%80%8BIP%E4%BB%8B%E7%BB%8D%E6%89%8B%E5%86%8C2024.pdf?authorization=bce-auth-v1%2FALTAKGa8m4qCUasgoljdEDAzLm%2F2024-08-21T11%3A40%3A45Z%2F-1%2Fhost%2Ff387e2cd2db95c9da26247c5d6c45f6dc9a817fa2721f0a780dc1eba433d02b8\"\n", "file_path = \"黑神话(悟空).pdf\" # 替换为你想要的文件名\n", "\n", "# 调用函数下载文件\n", "download_file(file_url, file_path)" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [], "source": [ "upload_res = knowledge.upload_file(file_path)\n", "add_res = knowledge.add_document(\n", " content_type=\"raw_text\",\n", " file_ids=[upload_res.id],\n", " custom_process_rule=appbuilder.CustomProcessRule(\n", " separators=[\"?\"], target_length=400, overlap_rate=0.2\n", " ),\n", " knowledge_base_id=my_knowledge.knowledge_id\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 获取知识库的文档列表\n", "在完成添加文档之后，如果我们想获取知识库中的文档列表，可以使用下面的方法：\n", "- 获取知识库的文档列表:`get_documents_list` 单次请求列表获得的文档数量,最大100\n", "- 获取知识库全部文档:`get_all_documents`" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n", "get_documents_list\n", "文档列表: request_id='da7bc5a5-c42d-47cd-82e7-674ff3d65502' data=[Document(id='139be427-5024-4af7-b5ca-1f15efdfea86', name='黑神话(悟空).pdf', created_at=1724295950, word_count=0, enabled=True, meta=DocumentMeta(source='upload_file', file_id='fc170b9e-0876-4d82-903c-5976e9a58453')), Document(id='dfeb440e-dc5f-4d1a-b0d8-6561b9fb5786', name='https://baijiahao.baidu.com/s?id=1802527379394162441', created_at=1724295904, word_count=0, enabled=True, meta=DocumentMeta(source='url', file_id=None))]\n", "get_all_documents\n", "id='139be427-5024-4af7-b5ca-1f15efdfea86' name='黑神话(悟空).pdf' created_at=1724295950 word_count=0 enabled=True meta=DocumentMeta(source='upload_file', file_id='fc170b9e-0876-4d82-903c-5976e9a58453')\n", "id='dfeb440e-dc5f-4d1a-b0d8-6561b9fb5786' name='https://baijiahao.baidu.com/s?id=1802527379394162441' created_at=1724295904 word_count=0 enabled=True meta=DocumentMeta(source='url', file_id=None)\n" ] } ], "source": [ "my_knowledge = appbuilder.KnowledgeBase(my_knowledge_base_id)\n", "print(\"知识库ID: \", my_knowledge.knowledge_id)\n", "\n", "# 获取知识库的文档列表:get_documents_list\n", "list_res = my_knowledge.get_documents_list()\n", "print(\"get_documents_list\")\n", "print(\"文档列表: \", list_res)\n", "\n", "# 获取知识库全部文档:get_all_documents\n", "doc_list = knowledge.get_all_documents(my_knowledge.knowledge_id)\n", "print(\"get_all_documents\")\n", "for message in doc_list:\n", " print(message)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 现在我们可以在Appbuilder中查看我们的知识库\n", "\n", "\n", "\n", "# 现在我们基于new_test_knowledge知识库创建了Client应用\n", "\n", "\n", "\n", "接下来我们调用Client应用，查看运行效果" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "《黑神话：悟空》是一款充满文化底蕴的游戏，用现代方式讲述经典中国故事，引发全球用户的共鸣^[1]^。\n" ] } ], "source": [ "# 从AppBuilder控制台【个人空间】-【应用】网页获取已发布应用的ID\n", "app_id = \"236b3e57-3464-43b0-8556-f165193721ef\"\n", "\n", "app_builder_client = appbuilder.AppBuilderClient(app_id)\n", "conversation_id = app_builder_client.create_conversation()\n", "\n", "resp = app_builder_client.run(conversation_id, \"请你用50字左右介绍一下《黑神话(悟空)》\")\n", "print(resp.content.answer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### KnowledgeBase 同样提供代码态的接口，可以对知识库文档进行切片\n", "\n", "* 创建切片\n", "* 修改切片信息\n", "* 获取切片信息\n", "* 获取切片列表\n", "* 删除切片\n", "\n", "- 首先我们尝试让Client回答KnowledgeBase中没有的信息" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "很抱歉，我没有找到关于《黑神话：悟空》中虎先锋的具体打法攻略。但是，一般来说，在动作冒险游戏中，打败敌人的关键通常在于熟练掌握角色的攻击方式、防御技巧和敌人的攻击模式。你可以尝试多次与虎先锋战斗，观察它的攻击方式和弱点，并结合孙悟空的技能和装备来制定相应的战斗策略。此外，你也可以在游戏的官方论坛或社区中寻找其他玩家分享的攻略和经验，这可能会对你有所帮助。\n", "\n", "请注意，不同玩家的游戏风格和策略可能有所不同，因此建议你在参考他人攻略的同时，结合自己的游戏情况进行调整和优化。\n" ] } ], "source": [ "# 从AppBuilder控制台【个人空间】-【应用】网页获取已发布应用的ID\n", "app_id = \"236b3e57-3464-43b0-8556-f165193721ef\"\n", "\n", "app_builder_client = appbuilder.AppBuilderClient(app_id)\n", "conversation_id = app_builder_client.create_conversation()\n", "\n", "resp = app_builder_client.run(conversation_id, \"《黑神话：悟空》的‌虎先锋怎么打？能不能提供一下攻略？\")\n", "print(resp.content.answer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 这时候我们就需要使用为KnowledeBase添加切片为他增添知识点了\n", "###### 创建切片`create_chunk`\n", "\n", "- 为了增加对照，我们先为Client添加BaiduSearch组件，并运行它查看他能提供哪些信息" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "《黑神话：悟空》的‌虎先锋打法攻略如下^[2]^：\n", "\n", "1. 起手进行蓄力，第一次蓄力攻击和虎先锋换血。\n", "2. 闪身躲避虎先锋的两端攻击。\n", "3. 对其进行攻击，使用精魄将其打到硬直，接着使用定身术将其定住，然后再进行攻击。\n", "4. 注意虎先锋会开启石化技能，可以规避攻击，并且进行反击，玩家在其进入石化后应避免出招。\n", "5. 在其第二次石化之后拉开距离，观察情况，躲避其的突进。\n", "6. 当虎先锋掏出剑之后，注意躲避其攻击，等待其将剑收回去再进行攻击。\n", "7. 在虎先锋血量很少的情况下使用定身来进行输出，随后使用攒的棍势再次输出即可击败虎先锋。\n" ] } ], "source": [ "# 从AppBuilder控制台【个人空间】-【应用】网页获取已发布应用的ID\n", "app_id = \"236b3e57-3464-43b0-8556-f165193721ef\"\n", "\n", "app_builder_client = appbuilder.AppBuilderClient(app_id)\n", "conversation_id = app_builder_client.create_conversation()\n", "\n", "resp = app_builder_client.run(conversation_id, \"《黑神话：悟空》的‌虎先锋怎么打？能不能提供一下攻略？\")\n", "print(resp.content.answer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 我们发现BaiduSearch提供的信息不一定符合我们预期，这个时候我们就需要对知识库增加新的切片，来为我们的知识库不断更新知识点，获取document_id,支持对文档进行切片操作\n", "\n", "```python\n", "# 获取doc_list列表的第一个文档的document_id\n", "document_id = doc_list[0].id\n", "```\n", "\n", "- content: 切片内容" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "知识库ID: 3cc48493-f9ff-48c8-a3ea-fcc17fdddded\n", "切片ID: 19e93596-a314-4ca3-81fa-d6c041433979\n" ] } ], "source": [ "# 获取doc_list列表的第一个文档的document_id\n", "document_id = doc_list[0].id\n", "my_knowledge = appbuilder.KnowledgeBase(my_knowledge_base_id)\n", "print(\"知识库ID: \", my_knowledge.knowledge_id)\n", "# 传入了一个内容为\"content\"的切片\n", "content = \"\"\"\n", "虎先锋是黑神话悟空中的一个boss，很多的玩家都非常的好奇这个boss要怎么样打，这里小编就为大家对虎先锋这个boss的打法进行了整合，可以让你更好的去进行战斗，帮助你击败这个敌人，详细的内容就在这里，快来一起看看吧。\n", "1、进入战斗后注意躲避boss的坠击伤害，快速闪避即可。\n", "2、和BOSS保持一定的距离，如果距离过近，它会对玩家造成三段连击伤害。\n", "3、当BOSS释放吼叫后，直接远离它，在远程进行输出。\n", "4、然后使用闪避躲避它释放的三连剑气和黑色的气旋。\n", "5、最后当BOSS变的坚硬时不作攻击，等它技能结束后继续输出即可轻松过关。\n", "\"\"\"\n", "resp = my_knowledge.create_chunk(documentId=document_id, content=content)\n", "print(\"切片ID: \", resp.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 再次运行Client看看他能否输出我们预期的知识点" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "《黑神话：悟空》的‌虎先锋打法攻略如下^[1][3]^：\n", "\n", "1. 起手进行蓄力，第一次蓄力攻击和虎先锋换血。\n", "\n", "2. 随后闪身躲避起手的两端攻击。\n", "\n", "3. 对其进行攻击，使用精魄将其打到硬直，接着使用定身法将其定住，然后再进行攻击。\n", "\n", "4. 注意虎先锋会开启石化技能，可以规避攻击，并且进行反击，玩家在其进入石化后应尽可能避免出招。\n", "\n", "5. 在虎先锋血量很少的情况下使用定身法来进行输出，随后再输出即可击败虎先锋。\n" ] } ], "source": [ "app_id = \"236b3e57-3464-43b0-8556-f165193721ef\"\n", "\n", "app_builder_client = appbuilder.AppBuilderClient(app_id)\n", "conversation_id = app_builder_client.create_conversation()\n", "\n", "resp = app_builder_client.run(conversation_id, \"《黑神话：悟空》的‌虎先锋怎么打？能不能提供一下攻略？\")\n", "print(resp.content.answer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 这次输出的信息就符合我们预期了。\n", "\n", "###### 修改切片信息`modify_chunk`\n", "\n", "- 如果我们对之前设置的切片信息不满意，可以通过`modify_chunk`方法进行修改。\n", "\n", "###### 获取切片信息`describe_chunk`\n", "\n", "- 我需要查看我刚才创建切片信息，可以使用`describe_chunk`方法\n", "\n", "###### 删除切片信息`delete_chunk`\n", "\n", "- 切片信息创建出来后，如果不需要了，可以通过`delete_chunk`方法进行删除。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### 删除文档和知识库\n", "\n", "- 当你不再需要文档知识库时，可以删除它，SDK同样提供删除文档和知识库的方法\n", " - 从知识库删除文档 `KnowledgeBase().delete_document`\n", " - 删除知识库`delete_knowledge_base`" ] } ], "metadata": { "kernelspec": { "display_name": "python-12", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "undefined.undefined.undefined" } }, "nbformat": 4, "nbformat_minor": 2 }