[{"_1":2,"_403":-5,"_404":-5},"loaderData",{"_3":4,"_29":30,"_311":312},"root",{"_5":6,"_14":15},"toasts",[7],{"_8":9,"_10":11,"_12":13},"uid","019a058c-dec8-7055-907b-40b6b5d09a20","title","You need to sign in to access this page","type","error","visitor",{"_16":17,"_18":19},"viewportWidth",1280,"visitorSession",{"_20":21,"_23":24,"_25":-5,"_26":27,"_28":-5},"attributes",[22],"bot","id",19346472,"membership","referenceId","0199f420-7955-7299-8090-ea86aef37824","userAccount","routes/_public/mcp/servers/~namespace/~slug/_pages/_layout",{"_31":32,"_79":-5,"_80":44,"_34":81,"_228":229,"_310":213},"availableLocales",[33,44,49,55,61,67,73],{"_34":35,"_38":41,"_42":43},"mcpServer",{"_36":37,"_38":37,"_39":40},"description","Dingo MCP Server","name","readme","[Englisch](README.md) ·[简体中文](README_zh-CN.md)\n\n# Änderungsprotokoll\n\n* 27.12.2024: Projektinitialisierung\n\n# Einführung\n\nDingo ist ein Tool zur Datenqualitätsbewertung, mit dem Sie Qualitätsprobleme in Ihren Datensätzen automatisch erkennen können. Dingo bietet eine Vielzahl integrierter Regeln und Modellbewertungsmethoden und unterstützt auch benutzerdefinierte Bewertungsmethoden. Dingo unterstützt häufig verwendete Textdatensätze und multimodale Datensätze, darunter Vortrainingsdatensätze, Feinabstimmungsdatensätze und Bewertungsdatensätze. Darüber hinaus unterstützt Dingo verschiedene Verwendungsmethoden, darunter lokale CLI und SDK, und erleichtert so die Integration in verschiedene Bewertungsplattformen wie [OpenCompass](https://github.com/open-compass/opencompass) .\n\n## Architekturdiagramm\n\n![Architektur des Dingo](./docs/assets/architeture.png)\n\n# Schnellstart\n\n## Installation\n\n```shell\npip install dingo-python\n```\n\n## Beispiel-Anwendungsfälle\n\n### 1. Verwenden von Evaluate Core\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. Lokale Textdatei (Klartext) auswerten\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. Bewerten Sie den Hugging Face-Datensatz\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. Evaluieren Sie das JSON/JSONL-Format\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. LLM zur Auswertung verwenden\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## Befehlszeilenschnittstelle\n\n### Auswerten mit Regelsätzen\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### Evaluieren mit LLM (zB GPT-4o)\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\nBeispiel `config_gpt.json` :\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## GUI-Visualisierung\n\nNach der Auswertung (mit `save_data=True` ) wird automatisch eine Frontend-Seite generiert. So starten Sie das Frontend manuell:\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\nDabei enthält `output_directory` die Auswertungsergebnisse mit einer Datei `summary.json` .\n\n![GUI-Ausgabe](docs/assets/dingo_gui.png)\n\n## Online-Demo\n\nProbieren Sie Dingo in unserer Online-Demo aus: [(Umarmungsgesicht)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# Datenqualitätsmetriken\n\nDingo klassifiziert Datenqualitätsprobleme in sieben Dimensionen von Qualitätsmetriken. Jede Dimension kann sowohl mit regelbasierten Methoden als auch mit LLM-basierten Eingabeaufforderungen ausgewertet werden:\n\n| Qualitätsmetrik | Beschreibung | Regelbeispiele | Beispiele für LLM-Eingabeaufforderungen |\n| -------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **VOLLSTÄNDIGKEIT** | Überprüft, ob Daten unvollständig sind oder fehlen | `RuleColonEnd` , `RuleContentNull` | Prüft, ob der Text abrupt mit einem Doppelpunkt oder Auslassungspunkten endet, ob Klammern nicht übereinstimmen oder ob wichtige Komponenten fehlen |\n| **WIRKSAMKEIT** | Überprüft, ob die Daten aussagekräftig und richtig formatiert sind | `RuleAbnormalChar` , `RuleHtmlEntity` , `RuleSpecialCharacter` | Erkennt unleserlichen Text, Wörter, die ohne Leerzeichen aneinandergereiht sind, und Text ohne korrekte Zeichensetzung |\n| **FLÜSSIGKEIT** | Überprüft, ob der Text grammatikalisch korrekt ist und sich natürlich liest | `RuleAbnormalNumber` , `RuleNoPunc` , `RuleWordStuck` | Identifiziert übermäßig lange Wörter, Textfragmente ohne Zeichensetzung oder Inhalte mit chaotischer Lesereihenfolge |\n| **RELEVANZ** | Erkennt irrelevante Inhalte in den Daten | `RuleHeadWord` -Varianten für verschiedene Sprachen | Untersucht irrelevante Informationen wie Zitatdetails, Kopf-/Fußzeilen, Entitätsmarkierungen, HTML-Tags |\n| **SICHERHEIT** | Identifiziert vertrauliche Informationen oder Wertekonflikte | `RuleIDCard` , `RuleUnsafeWords` | Überprüfung auf persönliche Informationen und Inhalte im Zusammenhang mit Glücksspiel, Pornografie und politischen Themen |\n| **ÄHNLICHKEIT** | Erkennt sich wiederholende oder sehr ähnliche Inhalte | `RuleDocRepeat` | Prüft Text auf aufeinanderfolgende Wiederholungen oder mehrfaches Vorkommen von Sonderzeichen |\n| **VERSTÄNDLICHKEIT** | Bewertet, wie einfach Daten interpretiert werden können | `RuleCapitalWords` | Stellt sicher, dass LaTeX-Formeln und Markdown korrekt formatiert sind, mit richtiger Segmentierung und Zeilenumbrüchen |\n\n## LLM-Qualitätsbewertung\n\nDingo bietet verschiedene LLM-basierte Bewertungsmethoden, die durch Eingabeaufforderungen im Verzeichnis `dingo/model/prompt` definiert sind. Diese Eingabeaufforderungen werden mit dem Dekorator `prompt_register` registriert und können zur Qualitätsbewertung mit LLM-Modellen kombiniert werden:\n\n### Eingabeaufforderungen zur Bewertung der Textqualität\n\n| Eingabeaufforderungstyp | Metrisch | Beschreibung |\n| ------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_V2` , `TEXT_QUALITY_V3` | Verschiedene Qualitätsdimensionen | Umfassende Bewertung der Textqualität hinsichtlich Effektivität, Relevanz, Vollständigkeit, Verständlichkeit, Ähnlichkeit, Flüssigkeit und Sicherheit |\n| `QUALITY_BAD_EFFECTIVENESS` | Wirksamkeit | Erkennt verstümmelten Text und Anti-Crawling-Inhalte |\n| `QUALITY_BAD_SIMILARITY` | Ähnlichkeit | Identifiziert Probleme mit Textwiederholungen |\n| `WORD_STICK` | Flüssigkeit | Prüft auf Wörter, die ohne den richtigen Abstand aneinander kleben |\n| `CODE_LIST_ISSUE` | Vollständigkeit | Bewertet Codeblöcke und listet Formatierungsprobleme auf |\n| `UNREAD_ISSUE` | Wirksamkeit | Erkennt unlesbare Zeichen aufgrund von Kodierungsproblemen |\n\n### 3H-Beurteilungsaufforderungen (ehrlich, hilfreich, harmlos)\n\n| Eingabeaufforderungstyp | Metrisch | Beschreibung |\n| ----------------------- | ----------------- | ---------------------------------------------------------------------------------------------------- |\n| `QUALITY_HONEST` | Ehrlichkeit | Bewertet, ob die Antworten genaue Informationen ohne Fälschung oder Täuschung liefern |\n| `QUALITY_HELPFUL` | Hilfsbereitschaft | Bewertet, ob die Antworten direkt auf die Fragen eingehen und die Anweisungen angemessen befolgen |\n| `QUALITY_HARMLESS` | Harmlosigkeit | Überprüft, ob Antworten schädliche Inhalte, diskriminierende Sprache und gefährliche Hilfe vermeiden |\n\n### Domänenspezifische Bewertungsaufforderungen\n\n| Eingabeaufforderungstyp | Metrisch | Beschreibung |\n| ----------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| `TEXT_QUALITY_KAOTI` | Qualität der Prüfungsfragen | Spezialisiertes Assessment zur Beurteilung der Qualität von Prüfungsfragen mit Fokus auf Formeldarstellung, Tabellenformatierung, Absatzstruktur und Antwortformatierung |\n| `Html_Abstract` | HTML-Extraktionsqualität | Vergleicht verschiedene Methoden zum Extrahieren von Markdown aus HTML und bewertet Vollständigkeit, Formatierungsgenauigkeit und semantische Kohärenz |\n| `DATAMAN_ASSESSMENT` | Datenqualität und -domäne | Bewertet die Datenqualität vor dem Training mithilfe der DataMan-Methode (14 Standards, 15 Domänen). Es werden eine Punktzahl (0/1), Domänentyp, Qualitätsstatus und ein Grund zugewiesen. |\n\n### Klassifizierungsaufforderungen\n\n| Eingabeaufforderungstyp | Metrisch | Beschreibung |\n| ----------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |\n| `CLASSIFY_TOPIC` | Themenkategorisierung | Klassifiziert Text in Kategorien wie Sprachverarbeitung, Schreiben, Code, Mathematik, Rollenspiel oder Wissensfragen und -antworten |\n| `CLASSIFY_QR` | Bildklassifizierung | Identifiziert Bilder als CAPTCHA, QR-Code oder normale Bilder |\n\n### Eingabeaufforderungen zur Bildbewertung\n\n| Eingabeaufforderungstyp | Metrisch | Beschreibung |\n| ----------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |\n| `IMAGE_RELEVANCE` | Bildrelevanz | Bewertet, ob ein Bild hinsichtlich der Anzahl der Gesichter, der Merkmalsdetails und der visuellen Elemente mit dem Referenzbild übereinstimmt |\n\n### Verwendung der LLM-Bewertung bei der Bewertung\n\nUm diese Bewertungsaufforderungen in Ihren Auswertungen zu verwenden, geben Sie sie in Ihrer Konfiguration an:\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nSie können diese Eingabeaufforderungen individuell anpassen, um bestimmte Qualitätsdimensionen zu fokussieren oder an spezifische Domänenanforderungen anzupassen. In Kombination mit geeigneten LLM-Modellen ermöglichen diese Eingabeaufforderungen eine umfassende Bewertung der Datenqualität über mehrere Dimensionen hinweg.\n\n# Regelgruppen\n\nDingo bietet vorkonfigurierte Regelgruppen für verschiedene Arten von Datensätzen:\n\n| Gruppe | Anwendungsfall | Beispielregeln |\n| ---------- | ------------------------------ | ---------------------------------------------------------------------------------------- |\n| `default` | Allgemeine Textqualität | `RuleColonEnd` , `RuleContentNull` , `RuleDocRepeat` usw. |\n| `sft` | Feinabstimmung von Datensätzen | Regeln aus `default` plus `RuleLineStartWithBulletpoint` |\n| `pretrain` | Vorabtraining von Datensätzen | Umfassender Satz mit über 20 Regeln, darunter `RuleAlphaWords` , `RuleCapitalWords` usw. |\n\nSo verwenden Sie eine bestimmte Regelgruppe:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# Funktionshighlights\n\n## Multi-Source- und Multi-Modal-Support\n\n* **Datenquellen** : Lokale Dateien, Hugging Face-Datensätze, S3-Speicher\n* **Datentypen** : Datensätze für Vortraining, Feinabstimmung und Auswertung\n* **Datenmodalitäten** : Text und Bild\n\n## Regelbasierte und modellbasierte Auswertung\n\n* **Integrierte Regeln** : Über 20 allgemeine heuristische Bewertungsregeln\n* **LLM-Integration** : OpenAI, Kimi und lokale Modelle (z. B. Llama3)\n* **Benutzerdefinierte Regeln** : Einfache Erweiterung mit Ihren eigenen Regeln und Modellen\n* **Sicherheitsbewertung** : Perspective API-Integration\n\n## Flexible Nutzung\n\n* **Schnittstellen** : CLI- und SDK-Optionen\n* **Integration** : Einfache Integration mit anderen Plattformen\n* **Ausführungs-Engines** : Lokal und Spark\n\n## Umfassende Berichterstattung\n\n* **Qualitätsmetriken** : 7-dimensionale Qualitätsbewertung\n* **Rückverfolgbarkeit** : Detaillierte Berichte zur Anomalieverfolgung\n\n# Benutzerhandbuch\n\n## Benutzerdefinierte Regeln, Eingabeaufforderungen und Modelle\n\nWenn die integrierten Regeln Ihren Anforderungen nicht entsprechen, können Sie benutzerdefinierte Regeln erstellen:\n\n### Beispiel für eine benutzerdefinierte Regel\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### Benutzerdefinierte LLM-Integration\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nWeitere Beispiele finden Sie in:\n\n* [Registerregeln](examples/register/sdk_register_rule.py)\n* [Registrierungsaufforderungen](examples/register/sdk_register_prompt.py)\n* [Modelle registrieren](examples/register/sdk_register_llm.py)\n\n## Ausführungs-Engines\n\n### Lokale Ausführung\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Spark-Ausführung\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## Evaluierungsberichte\n\nNach der Auswertung generiert Dingo:\n\n1. **Zusammenfassender Bericht** ( `summary.json` ): Gesamtmetriken und Bewertungen\n2. **Detaillierte Berichte** : Spezifische Probleme für jeden Regelverstoß\n\nBeispielzusammenfassung:\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# MCP-Server (experimentell)\n\nDingo enthält einen experimentellen Model Context Protocol (MCP)-Server. Details zum Betrieb des Servers und zur Integration mit Clients wie Cursor finden Sie in der entsprechenden Dokumentation:\n\n[**Dingo MCP Server-Dokumentation (README\\_mcp.md)**](README_mcp.md)\n\n# Forschung & Publikationen\n\n* **„Umfassende Bewertung der Datenqualität für mehrsprachige Webdaten“** : [WanJuanSiLu: Ein hochwertiger Open-Source-Webtext-Datensatz für ressourcenarme Sprachen](https://arxiv.org/pdf/2501.14506)\n* **„Datenqualität vor dem Training mit der DataMan-Methodik“** : [DataMan: Datenmanager für das Vortraining großer Sprachmodelle](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# Zukunftspläne\n\n* \\[ ] Umfangreichere grafische und textuelle Bewertungsindikatoren\n* \\[ ] Auswertung der Audio- und Videodatenmodalität\n* \\[ ] Kleine Modellbewertung (Fasttext, Qurating)\n* \\[ ] Bewertung der Datenvielfalt\n\n# Einschränkungen\n\nDie aktuell integrierten Erkennungsregeln und Modellmethoden konzentrieren sich auf häufige Datenqualitätsprobleme. Für spezielle Auswertungsanforderungen empfehlen wir die Anpassung der Erkennungsregeln.\n\n# Danksagung\n\n* [RedPajama-Daten](https://github.com/togethercomputer/RedPajama-Data)\n* [mlflow](https://github.com/mlflow/mlflow)\n\n# Beitrag\n\nWir danken allen Mitwirkenden für ihre Bemühungen, `Dingo` zu verbessern und zu erweitern. Weitere Informationen zur Mitarbeit am Projekt finden Sie im [Beitragsleitfaden](docs/en/CONTRIBUTING.md) .\n\n# Lizenz\n\nDieses Projekt verwendet die [Apache 2.0 Open Source-Lizenz](LICENSE) .\n\n# Zitat\n\nWenn Sie dieses Projekt nützlich finden, denken Sie bitte darüber nach, unser Tool zu zitieren:\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","German","tag","de-DE",{"_34":45,"_38":47,"_42":48},{"_36":37,"_38":37,"_39":46},"

\n\n

\n \n

\n\n\n

\n $\"pre-commit\"$ \n $\"PyPI$ \n $\"Python$ \n $\"License\"$ \n $\"GitHub$ \n $\"GitHub$ \n $\"GitHub$ \n $\"MseeP.ai$ \n

\n\n

\n\n\n

\n\n[English](README.md) · [简体中文](README_zh-CN.md)\n\n

\n\n\n

\n \n $\"Discord\"$ \n \n $\"Hugging$ \n

\n\n\n# Changelog\n\n- 2024/12/27: Project Initialization\n\n# Introduction\n\nDingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as [OpenCompass](https://github.com/open-compass/opencompass).\n\n## Architecture Diagram\n\n![Architecture of dingo](./docs/assets/architeture.png)\n\n# Quick Start\n\n## Installation\n\n```shell\npip install dingo-python\n```\n\n## Example Use Cases\n\n### 1. Using Evaluate Core\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. Evaluate Local Text File (Plaintext)\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. Evaluate Hugging Face Dataset\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. Evaluate JSON/JSONL Format\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. Using LLM for Evaluation\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## Command Line Interface\n\n### Evaluate with Rule Sets\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### Evaluate with LLM (e.g., GPT-4o)\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\nExample `config_gpt.json`:\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## GUI Visualization\n\nAfter evaluation (with `save_data=True`), a frontend page will be automatically generated. To manually start the frontend:\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\nWhere `output_directory` contains the evaluation results with a `summary.json` file.\n\n![GUI output](docs/assets/dingo_gui.png)\n\n## Online Demo\nTry Dingo on our online demo: [(Hugging Face)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# Data Quality Metrics\n\nDingo classifies data quality issues into 7 dimensions of Quality Metrics. Each dimension can be evaluated using both rule-based methods and LLM-based prompts:\n\n| Quality Metric | Description | Rule Examples | LLM Prompt Examples |\n|-------------------|-------------|---------------|---------------------|\n| **COMPLETENESS** | Checks if data is incomplete or missing | `RuleColonEnd`, `RuleContentNull` | Evaluates if text abruptly ends with a colon or ellipsis, has mismatched parentheses, or missing critical components |\n| **EFFECTIVENESS** | Checks if data is meaningful and properly formatted | `RuleAbnormalChar`, `RuleHtmlEntity`, `RuleSpecialCharacter` | Detects garbled text, words stuck together without spaces, and text lacking proper punctuation |\n| **FLUENCY** | Checks if text is grammatically correct and reads naturally | `RuleAbnormalNumber`, `RuleNoPunc`, `RuleWordStuck` | Identifies excessively long words, text fragments without punctuation, or content with chaotic reading order |\n| **RELEVANCE** | Detects irrelevant content within the data | `RuleHeadWord` variants for different languages | Examines for irrelevant information like citation details, headers/footers, entity markers, HTML tags |\n| **SECURITY** | Identifies sensitive information or value conflicts | `RuleIDCard`, `RuleUnsafeWords` | Checks for personal information, and content related to gambling, pornography, political issues |\n| **SIMILARITY** | Detects repetitive or highly similar content | `RuleDocRepeat` | Evaluates text for consecutive repeated content or multiple occurrences of special characters |\n| **UNDERSTANDABILITY** | Assesses how easily data can be interpreted | `RuleCapitalWords` | Ensures LaTeX formulas and Markdown are correctly formatted, with proper segmentation and line breaks |\n\n## LLM Quality Assessment\n\nDingo provides several LLM-based assessment methods defined by prompts in the `dingo/model/prompt` directory. These prompts are registered using the `prompt_register` decorator and can be combined with LLM models for quality evaluation:\n\n### Text Quality Assessment Prompts\n\n| Prompt Type | Metric | Description |\n|-------------|--------|-------------|\n| `TEXT_QUALITY_V2`, `TEXT_QUALITY_V3` | Various quality dimensions | Comprehensive text quality evaluation covering effectiveness, relevance, completeness, understandability, similarity, fluency, and security |\n| `QUALITY_BAD_EFFECTIVENESS` | Effectiveness | Detects garbled text and anti-crawling content |\n| `QUALITY_BAD_SIMILARITY` | Similarity | Identifies text repetition issues |\n| `WORD_STICK` | Fluency | Checks for words stuck together without proper spacing |\n| `CODE_LIST_ISSUE` | Completeness | Evaluates code blocks and list formatting issues |\n| `UNREAD_ISSUE` | Effectiveness | Detects unreadable characters due to encoding issues |\n\n### 3H Assessment Prompts (Honest, Helpful, Harmless)\n\n| Prompt Type | Metric | Description |\n|-------------|--------|-------------|\n| `QUALITY_HONEST` | Honesty | Evaluates if responses provide accurate information without fabrication or deception |\n| `QUALITY_HELPFUL` | Helpfulness | Assesses if responses address questions directly and follow instructions appropriately |\n| `QUALITY_HARMLESS` | Harmlessness | Checks if responses avoid harmful content, discriminatory language, and dangerous assistance |\n\n### Domain-Specific Assessment Prompts\n\n| Prompt Type | Metric | Description |\n|-------------|--------|-------------|\n| `TEXT_QUALITY_KAOTI` | Exam question quality | Specialized assessment for evaluating the quality of exam questions, focusing on formula rendering, table formatting, paragraph structure, and answer formatting |\n| `Html_Abstract` | HTML extraction quality | Compares different methods of extracting Markdown from HTML, evaluating completeness, formatting accuracy, and semantic coherence |\n| `DATAMAN_ASSESSMENT` | Data Quality & Domain | Evaluates pre-training data quality using the DataMan methodology (14 standards, 15 domains). Assigns a score (0/1), domain type, quality status, and reason. |\n\n### Classification Prompts\n\n| Prompt Type | Metric | Description |\n|-------------|--------|-------------|\n| `CLASSIFY_TOPIC` | Topic Categorization | Classifies text into categories like language processing, writing, code, mathematics, role-play, or knowledge Q&A |\n| `CLASSIFY_QR` | Image Classification | Identifies images as CAPTCHA, QR code, or normal images |\n\n### Image Assessment Prompts\n\n| Prompt Type | Metric | Description |\n|-------------|--------|-------------|\n| `IMAGE_RELEVANCE` | Image Relevance | Evaluates if an image matches reference image in terms of face count, feature details, and visual elements |\n\n### Using LLM Assessment in Evaluation\n\nTo use these assessment prompts in your evaluations, specify them in your configuration:\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nYou can customize these prompts to focus on specific quality dimensions or to adapt to particular domain requirements. When combined with appropriate LLM models, these prompts enable comprehensive evaluation of data quality across multiple dimensions.\n\n# Rule Groups\n\nDingo provides pre-configured rule groups for different types of datasets:\n\n| Group | Use Case | Example Rules |\n|-------|----------|---------------|\n| `default` | General text quality | `RuleColonEnd`, `RuleContentNull`, `RuleDocRepeat`, etc. |\n| `sft` | Fine-tuning datasets | Rules from `default` plus `RuleLineStartWithBulletpoint` |\n| `pretrain` | Pre-training datasets | Comprehensive set of 20+ rules including `RuleAlphaWords`, `RuleCapitalWords`, etc. |\n\nTo use a specific rule group:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# Feature Highlights\n\n## Multi-source & Multi-modal Support\n\n- **Data Sources**: Local files, Hugging Face datasets, S3 storage\n- **Data Types**: Pre-training, fine-tuning, and evaluation datasets\n- **Data Modalities**: Text and image\n\n## Rule-based & Model-based Evaluation\n\n- **Built-in Rules**: 20+ general heuristic evaluation rules\n- **LLM Integration**: OpenAI, Kimi, and local models (e.g., Llama3)\n- **Custom Rules**: Easily extend with your own rules and models\n- **Security Evaluation**: Perspective API integration\n\n## Flexible Usage\n\n- **Interfaces**: CLI and SDK options\n- **Integration**: Easy integration with other platforms\n- **Execution Engines**: Local and Spark\n\n## Comprehensive Reporting\n\n- **Quality Metrics**: 7-dimensional quality assessment\n- **Traceability**: Detailed reports for anomaly tracking\n\n# User Guide\n\n## Custom Rules, Prompts, and Models\n\nIf the built-in rules don't meet your requirements, you can create custom ones:\n\n### Custom Rule Example\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### Custom LLM Integration\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nSee more examples in:\n- [Register Rules](examples/register/sdk_register_rule.py)\n- [Register Prompts](examples/register/sdk_register_prompt.py)\n- [Register Models](examples/register/sdk_register_llm.py)\n\n## Execution Engines\n\n### Local Execution\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Spark Execution\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## Evaluation Reports\n\nAfter evaluation, Dingo generates:\n\n1. **Summary Report** (`summary.json`): Overall metrics and scores\n2. **Detailed Reports**: Specific issues for each rule violation\n\nExample summary:\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n\n# MCP Server (Experimental)\n\nDingo includes an experimental Model Context Protocol (MCP) server. For details on running the server and integrating it with clients like Cursor, please see the dedicated documentation:\n\n[**Dingo MCP Server Documentation (README_mcp.md)**](README_mcp.md)\n\n\n# Research & Publications\n\n- **\"Comprehensive Data Quality Assessment for Multilingual WebData\"** : [WanJuanSiLu: A High-Quality Open-Source Webtext\nDataset for Low-Resource Languages](https://arxiv.org/pdf/2501.14506)\n- **\"Pre-training data quality using the DataMan methodology\"** : [DataMan: Data Manager for Pre-training Large Language Models](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n\n# Future Plans\n\n- [ ] Richer graphic and text evaluation indicators\n- [ ] Audio and video data modality evaluation\n- [ ] Small model evaluation (fasttext, Qurating)\n- [ ] Data diversity evaluation\n\n# Limitations\n\nThe current built-in detection rules and model methods focus on common data quality problems. For specialized evaluation needs, we recommend customizing detection rules.\n\n# Acknowledgments\n\n- [RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data)\n- [mlflow](https://github.com/mlflow/mlflow)\n\n# Contribution\n\nWe appreciate all the contributors for their efforts to improve and enhance `Dingo`. Please refer to the [Contribution Guide](docs/en/CONTRIBUTING.md) for guidance on contributing to the project.\n\n# License\n\nThis project uses the [Apache 2.0 Open Source License](LICENSE).\n\n# Citation\n\nIf you find this project useful, please consider citing our tool:\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```\n","English","en-US",{"_34":50,"_38":53,"_42":54},{"_36":51,"_38":51,"_39":52},"Servidor Dingo MCP","[Inglés](README.md) ·[简体中文](README_zh-CN.md)\n\n# Registro de cambios\n\n* 27/12/2024: Inicialización del proyecto\n\n# Introducción\n\nDingo es una herramienta de evaluación de la calidad de datos que le ayuda a detectar automáticamente problemas de calidad en sus conjuntos de datos. Dingo ofrece diversas reglas integradas y métodos de evaluación de modelos, además de ser compatible con métodos de evaluación personalizados. Dingo admite conjuntos de datos de texto y conjuntos de datos multimodales de uso común, incluyendo conjuntos de datos de preentrenamiento, conjuntos de datos de ajuste y conjuntos de datos de evaluación. Además, Dingo admite múltiples métodos de uso, como la CLI y el SDK locales, lo que facilita su integración en diversas plataformas de evaluación, como [OpenCompass](https://github.com/open-compass/opencompass) .\n\n## Diagrama de arquitectura\n\n![Arquitectura del dingo](./docs/assets/architeture.png)\n\n# Inicio rápido\n\n## Instalación\n\n```shell\npip install dingo-python\n```\n\n## Ejemplos de casos de uso\n\n### 1. Uso de Evaluate Core\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. Evaluar archivo de texto local (texto sin formato)\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. Evaluar el conjunto de datos de caras abrazadas\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. Evaluar el formato JSON/JSONL\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. Uso de LLM para la evaluación\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## Interfaz de línea de comandos\n\n### Evaluar con conjuntos de reglas\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### Evaluar con LLM (por ejemplo, GPT-4o)\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\nEjemplo `config_gpt.json` :\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## Visualización GUI\n\nTras la evaluación (con `save_data=True` ), se generará automáticamente una página de interfaz. Para iniciar la interfaz manualmente:\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\nDonde `output_directory` contiene los resultados de la evaluación con un archivo `summary.json` .\n\n![Salida de GUI](docs/assets/dingo_gui.png)\n\n## Demostración en línea\n\nPrueba Dingo en nuestra demostración en línea: [(Cara abrazada)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# Métricas de calidad de datos\n\nDingo clasifica los problemas de calidad de los datos en siete dimensiones de Métricas de Calidad. Cada dimensión puede evaluarse mediante métodos basados en reglas y preguntas basadas en LLM:\n\n| Métrica de calidad | Descripción | Ejemplos de reglas | Ejemplos de indicaciones para LLM |\n| -------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |\n| **LO COMPLETO** | Comprueba si los datos están incompletos o faltan | `RuleColonEnd` , `RuleContentNull` | Evalúa si el texto termina abruptamente con dos puntos o puntos suspensivos, tiene paréntesis no coincidentes o faltan componentes críticos. |\n| **EFICACIA** | Comprueba si los datos son significativos y están formateados correctamente | `RuleAbnormalChar` , `RuleHtmlEntity` , `RuleSpecialCharacter` | Detecta texto ilegible, palabras pegadas sin espacios y texto sin la puntuación adecuada. |\n| **FLUIDEZ** | Comprueba si el texto es gramaticalmente correcto y se lee con naturalidad. | `RuleAbnormalNumber` , `RuleNoPunc` , `RuleWordStuck` | Identifica palabras excesivamente largas, fragmentos de texto sin puntuación o contenido con un orden de lectura caótico. |\n| **PERTINENCIA** | Detecta contenido irrelevante dentro de los datos | Variantes `RuleHeadWord` para diferentes idiomas | Examina información irrelevante como detalles de citas, encabezados/pies de página, marcadores de entidad y etiquetas HTML. |\n| **SEGURIDAD** | Identifica información sensible o conflictos de valores | `RuleIDCard` , `RuleUnsafeWords` | Comprueba información personal y contenido relacionado con juegos de azar, pornografía y cuestiones políticas. |\n| **SEMEJANZA** | Detecta contenido repetitivo o muy similar | `RuleDocRepeat` | Evalúa el texto en busca de contenido repetido consecutivo o múltiples apariciones de caracteres especiales |\n| **COMPRENSIBILIDAD** | Evalúa la facilidad con la que se pueden interpretar los datos | `RuleCapitalWords` | Garantiza que las fórmulas LaTeX y Markdown tengan el formato correcto, con la segmentación y los saltos de línea adecuados. |\n\n## Evaluación de calidad del LLM\n\nDingo ofrece varios métodos de evaluación basados en LLM, definidos por indicaciones en el directorio `dingo/model/prompt` . Estas indicaciones se registran mediante el decorador `prompt_register` y pueden combinarse con modelos LLM para la evaluación de calidad:\n\n### Indicaciones para la evaluación de la calidad del texto\n\n| Tipo de aviso | Métrico | Descripción |\n| ------------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_V2` , `TEXT_QUALITY_V3` | Varias dimensiones de calidad | Evaluación integral de la calidad del texto que abarca la eficacia, la relevancia, la integridad, la comprensibilidad, la similitud, la fluidez y la seguridad. |\n| `QUALITY_BAD_EFFECTIVENESS` | Eficacia | Detecta texto ilegible y contenido anti-rastreo |\n| `QUALITY_BAD_SIMILARITY` | Semejanza | Identifica problemas de repetición de texto |\n| `WORD_STICK` | Fluidez | Comprueba si hay palabras pegadas sin el espaciado adecuado |\n| `CODE_LIST_ISSUE` | Lo completo | Evalúa bloques de código y problemas de formato de listas. |\n| `UNREAD_ISSUE` | Eficacia | Detecta caracteres ilegibles debido a problemas de codificación. |\n\n### Preguntas de evaluación 3H (Honestas, útiles e inofensivas)\n\n| Tipo de aviso | Métrico | Descripción |\n| ------------------ | ---------- | ----------------------------------------------------------------------------------------------------- |\n| `QUALITY_HONEST` | Honestidad | Evalúa si las respuestas proporcionan información precisa sin invención ni engaño. |\n| `QUALITY_HELPFUL` | Utilidad | Evalúa si las respuestas abordan las preguntas directamente y siguen las instrucciones adecuadamente. |\n| `QUALITY_HARMLESS` | Inocuidad | Comprueba si las respuestas evitan contenido dañino, lenguaje discriminatorio y asistencia peligrosa. |\n\n### Indicaciones de evaluación específicas del dominio\n\n| Tipo de aviso | Métrico | Descripción |\n| -------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_KAOTI` | Calidad de las preguntas del examen | Evaluación especializada para evaluar la calidad de las preguntas del examen, centrándose en la representación de fórmulas, el formato de tablas, la estructura de párrafos y el formato de respuestas. |\n| `Html_Abstract` | Calidad de extracción de HTML | Compara diferentes métodos de extracción de Markdown de HTML, evaluando la integridad, la precisión del formato y la coherencia semántica. |\n| `DATAMAN_ASSESSMENT` | Calidad de datos y dominio | Evalúa la calidad de los datos previos al entrenamiento mediante la metodología DataMan (14 estándares, 15 dominios). Asigna una puntuación (0/1), tipo de dominio, estado de calidad y razón. |\n\n### Indicaciones de clasificación\n\n| Tipo de aviso | Métrico | Descripción |\n| ---------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `CLASSIFY_TOPIC` | Categorización de temas | Clasifica el texto en categorías como procesamiento del lenguaje, escritura, código, matemáticas, juego de roles o preguntas y respuestas sobre conocimientos. |\n| `CLASSIFY_QR` | Clasificación de imágenes | Identifica imágenes como CAPTCHA, código QR o imágenes normales. |\n\n### Indicaciones para la evaluación de imágenes\n\n| Tipo de aviso | Métrico | Descripción |\n| ----------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `IMAGE_RELEVANCE` | Relevancia de la imagen | Evalúa si una imagen coincide con la imagen de referencia en términos de número de rostros, detalles de las características y elementos visuales. |\n\n### Uso de la evaluación LLM en la evaluación\n\nPara utilizar estas indicaciones de evaluación en sus evaluaciones, especifíquelas en su configuración:\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nPuede personalizar estas indicaciones para centrarse en dimensiones de calidad específicas o para adaptarlas a requisitos específicos del dominio. Al combinarlas con los modelos LLM adecuados, estas indicaciones permiten una evaluación exhaustiva de la calidad de los datos en múltiples dimensiones.\n\n# Grupos de reglas\n\nDingo proporciona grupos de reglas preconfigurados para diferentes tipos de conjuntos de datos:\n\n| Grupo | Caso de uso | Reglas de ejemplo |\n| ---------- | -------------------------------------- | --------------------------------------------------------------------------------------------- |\n| `default` | Calidad general del texto | `RuleColonEnd` , `RuleContentNull` , `RuleDocRepeat` , etc. |\n| `sft` | Ajuste fino de conjuntos de datos | Reglas `default` más `RuleLineStartWithBulletpoint` |\n| `pretrain` | Conjuntos de datos de preentrenamiento | Conjunto completo de más de 20 reglas, incluidas `RuleAlphaWords` , `RuleCapitalWords` , etc. |\n\nPara utilizar un grupo de reglas específico:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# Características destacadas\n\n## Soporte multifuente y multimodal\n\n* **Fuentes de datos** : archivos locales, conjuntos de datos de Hugging Face, almacenamiento S3\n* **Tipos de datos** : conjuntos de datos de preentrenamiento, ajuste y evaluación\n* **Modalidades de datos** : Texto e imagen\n\n## Evaluación basada en reglas y modelos\n\n* **Reglas integradas** : más de 20 reglas generales de evaluación heurística\n* **Integración LLM** : OpenAI, Kimi y modelos locales (por ejemplo, Llama3)\n* **Reglas personalizadas** : amplíelas fácilmente con sus propias reglas y modelos\n* **Evaluación de seguridad** : Perspectiva de la integración de API\n\n## Uso flexible\n\n* **Interfaces** : opciones CLI y SDK\n* **Integración** : Fácil integración con otras plataformas.\n* **Motores de ejecución** : Local y Spark\n\n## Informes completos\n\n* **Métricas de calidad** : evaluación de calidad de siete dimensiones\n* **Trazabilidad** : Informes detallados para el seguimiento de anomalías\n\n# Guía del usuario\n\n## Reglas, indicaciones y modelos personalizados\n\nSi las reglas integradas no cumplen con sus requisitos, puede crear reglas personalizadas:\n\n### Ejemplo de regla personalizada\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### Integración LLM personalizada\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nVer más ejemplos en:\n\n* [Reglas de registro](examples/register/sdk_register_rule.py)\n* [Indicaciones de registro](examples/register/sdk_register_prompt.py)\n* [Modelos de registro](examples/register/sdk_register_llm.py)\n\n## Motores de ejecución\n\n### Ejecución local\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Ejecución de Spark\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## Informes de evaluación\n\nDespués de la evaluación, Dingo genera:\n\n1. **Informe resumido** ( `summary.json` ): Métricas y puntuaciones generales\n2. **Informes detallados** : Problemas específicos para cada infracción de las normas\n\nEjemplo de resumen:\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# Servidor MCP (Experimental)\n\nDingo incluye un servidor experimental de Protocolo de Contexto de Modelo (MCP). Para obtener más información sobre cómo ejecutar el servidor e integrarlo con clientes como Cursor, consulte la documentación dedicada:\n\n[**Documentación del servidor Dingo MCP (README\\_mcp.md)**](README_mcp.md)\n\n# Investigación y publicaciones\n\n* **Evaluación integral de la calidad de datos para datos web multilingües** : [WanJuanSiLu: Un conjunto de datos webtext de código abierto de alta calidad para idiomas con recursos limitados.](https://arxiv.org/pdf/2501.14506)\n* **\"Calidad de datos de preentrenamiento con la metodología DataMan\"** : [DataMan: Gestor de datos para el preentrenamiento de grandes modelos lingüísticos.](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# Planes futuros\n\n* \\[ ] Indicadores de evaluación de texto y gráficos más ricos\n* \\[ ] Evaluación de la modalidad de datos de audio y video\n* \\[ ] Evaluación de modelos pequeños (fasttext, Qurating)\n* \\[ ] Evaluación de la diversidad de datos\n\n# Limitaciones\n\nLas reglas de detección y los métodos de modelo integrados actuales se centran en problemas comunes de calidad de datos. Para necesidades de evaluación especializadas, recomendamos personalizar las reglas de detección.\n\n# Expresiones de gratitud\n\n* [Datos de RedPajama](https://github.com/togethercomputer/RedPajama-Data)\n* [flujo ml](https://github.com/mlflow/mlflow)\n\n# Contribución\n\nAgradecemos a todos los colaboradores por su esfuerzo para mejorar `Dingo` . Consulten la [Guía de Contribución](docs/en/CONTRIBUTING.md) para obtener orientación sobre cómo contribuir al proyecto.\n\n# Licencia\n\nEste proyecto utiliza la [licencia de código abierto Apache 2.0](LICENSE) .\n\n# Citación\n\nSi encuentra útil este proyecto, considere citar nuestra herramienta:\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","Spanish","es-ES",{"_34":56,"_38":59,"_42":60},{"_36":57,"_38":57,"_39":58},"Dingo MCP サーバー","[English](README.md) ·[简体中文](README_zh-CN.md)\n\n# 変更履歴\n\n* 2024/12/27: プロジェクト初期化\n\n# 導入\n\nDingoは、データセット内のデータ品質の問題を自動検出するデータ品質評価ツールです。Dingoは、様々な組み込みルールとモデル評価手法を提供するほか、カスタム評価手法もサポートしています。Dingoは、事前学習用データセット、ファインチューニング用データセット、評価用データセットなど、一般的に使用されるテキストデータセットとマルチモーダルデータセットをサポートしています。さらに、DingoはローカルCLIやSDKなど、複数の利用方法をサポートしているため、 [OpenCompass](https://github.com/open-compass/opencompass)などの様々な評価プラットフォームへの統合が容易です。\n\n## アーキテクチャ図\n\n![ディンゴの建築](./docs/assets/architeture.png)\n\n# クイックスタート\n\n## インストール\n\n```shell\npip install dingo-python\n```\n\n## 使用例\n\n### 1. Evaluate Coreの使用\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. ローカルテキストファイル（プレーンテキスト）を評価する\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. ハグ顔データセットを評価する\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. JSON/JSONL形式の評価\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. LLMを用いた評価\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## コマンドラインインターフェース\n\n### ルールセットで評価する\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### LLM（例：GPT-4o）で評価する\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\n例`config_gpt.json` :\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## GUI視覚化\n\n評価後（ `save_data=True`の場合）、フロントエンドページが自動的に生成されます。フロントエンドを手動で起動するには、以下の手順を実行してください。\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\n`output_directory`には、 `summary.json`ファイルを含む評価結果が含まれます。\n\n![GUI出力](docs/assets/dingo_gui.png)\n\n## オンラインデモ\n\nオンラインデモでDingoをお試しください: [(Hugging Face)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# データ品質メトリクス\n\nDingoは、データ品質の問題を7つの品質メトリクスの次元に分類します。各次元は、ルールベースの手法とLLMベースのプロンプトの両方を使用して評価できます。\n\n| 品質指標 | 説明 | ルールの例 | LLMプロンプトの例 |\n| ---------- | ---------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------- |\n| **完全** | データが不完全または欠落していないかを確認します | `RuleColonEnd` 、 `RuleContentNull` | テキストがコロンまたは省略記号で突然終わっているか、括弧が一致していないか、重要なコンポーネントが欠落しているかを評価します。 |\n| **効果** | データが意味を持ち、適切にフォーマットされているかどうかを確認します | `RuleAbnormalChar` 、 `RuleHtmlEntity` 、 `RuleSpecialCharacter` | 文字化けしたテキスト、スペースなしでつながった単語、適切な句読点がないテキストを検出します |\n| **流暢さ** | テキストが文法的に正しく、自然に読めるかどうかを確認します | `RuleAbnormalNumber` 、 `RuleNoPunc` 、 `RuleWordStuck` | 長すぎる単語、句読点のないテキストの断片、または読み順が乱れたコンテンツを識別します |\n| **関連性** | データ内の無関係なコンテンツを検出する | さまざまな言語の`RuleHeadWord`バリエーション | 引用の詳細、ヘッダー/フッター、エンティティマーカー、HTMLタグなどの無関係な情報を検査します。 |\n| **安全** | 機密情報や価値観の衝突を特定する | `RuleIDCard` 、 `RuleUnsafeWords` | 個人情報、ギャンブル、ポルノ、政治問題に関連するコンテンツをチェックします |\n| **類似性** | 繰り返しや類似性の高いコンテンツを検出します | `RuleDocRepeat` | テキスト内の連続した繰り返しコンテンツや特殊文字の複数出現を評価します |\n| **理解しやすさ** | データがどれだけ簡単に解釈できるかを評価する | `RuleCapitalWords` | LaTeX 式と Markdown が適切なセグメンテーションと改行で正しくフォーマットされていることを確認します。 |\n\n## LLM品質評価\n\nDingoは`dingo/model/prompt`ディレクトリ内のプロンプトで定義されたLLMベースの評価手法をいくつか提供しています。これらのプロンプトは`prompt_register`デコレータを使用して登録され、LLMモデルと組み合わせて品質評価を行うことができます。\n\n### テキスト品質評価プロンプト\n\n| プロンプトタイプ | メトリック | 説明 |\n| ------------------------------------- | --------- | ------------------------------------------------- |\n| `TEXT_QUALITY_V2` 、 `TEXT_QUALITY_V3` | さまざまな品質次元 | 有効性、関連性、完全性、理解可能性、類似性、流暢性、セキュリティを網羅した包括的なテキスト品質評価 |\n| `QUALITY_BAD_EFFECTIVENESS` | 効果 | 文字化けしたテキストやクロール対策コンテンツを検出します |\n| `QUALITY_BAD_SIMILARITY` | 類似性 | テキストの繰り返しの問題を特定します |\n| `WORD_STICK` | 流暢さ | 適切な間隔を空けずに単語がつながっていないかチェックします |\n| `CODE_LIST_ISSUE` | 完全 | コードブロックとリストのフォーマットの問題を評価する |\n| `UNREAD_ISSUE` | 効果 | エンコードの問題により判読できない文字を検出します |\n\n### 3H 評価プロンプト (正直、役立つ、無害)\n\n| プロンプトタイプ | メトリック | 説明 |\n| ------------------ | ----- | ------------------------------------- |\n| `QUALITY_HONEST` | 正直 | 回答が捏造や欺瞞のない正確な情報を提供しているかどうかを評価する |\n| `QUALITY_HELPFUL` | 有用性 | 回答が質問に直接答えており、指示に適切に従っているかどうかを評価する |\n| `QUALITY_HARMLESS` | 無害性 | 回答が有害な内容、差別的な言葉、危険な援助を避けているかどうかを確認します |\n\n### ドメイン固有の評価プロンプト\n\n| プロンプトタイプ | メトリック | 説明 |\n| -------------------- | ---------- | ------------------------------------------------------------------------------------------ |\n| `TEXT_QUALITY_KAOTI` | 試験問題の質 | 数式のレンダリング、表の書式設定、段落構造、回答の書式設定に重点を置いた、試験問題の品質を評価するための専門的な評価 |\n| `Html_Abstract` | HTML抽出品質 | HTML から Markdown を抽出するさまざまな方法を比較し、完全性、フォーマットの正確性、意味の一貫性を評価します。 |\n| `DATAMAN_ASSESSMENT` | データ品質とドメイン | DataManメソッド（14の基準、15のドメイン）を用いて、事前トレーニングデータの品質を評価します。スコア（0/1）、ドメインタイプ、品質ステータス、および理由を割り当てます。 |\n\n### 分類プロンプト\n\n| プロンプトタイプ | メトリック | 説明 |\n| ---------------- | ------- | --------------------------------------------------- |\n| `CLASSIFY_TOPIC` | トピックの分類 | テキストを言語処理、ライティング、コード、数学、ロールプレイ、知識Q\\&Aなどのカテゴリに分類します。 |\n| `CLASSIFY_QR` | 画像分類 | 画像を CAPTCHA、QR コード、または通常の画像として識別します |\n\n### 画像評価プロンプト\n\n| プロンプトタイプ | メトリック | 説明 |\n| ----------------- | ------ | ----------------------------------------- |\n| `IMAGE_RELEVANCE` | 画像の関連性 | 顔の数、特徴の詳細、視覚要素の観点から画像が参照画像と一致するかどうかを評価します |\n\n### 評価におけるLLM評価の活用\n\n評価でこれらの評価プロンプトを使用するには、構成で指定します。\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nこれらのプロンプトは、特定の品質次元に焦点を当てたり、特定のドメイン要件に合わせてカスタマイズできます。適切なLLMモデルと組み合わせることで、複数の次元にわたるデータ品質の包括的な評価が可能になります。\n\n# ルールグループ\n\nDingo は、さまざまな種類のデータセットに対して事前設定されたルールグループを提供します。\n\n| グループ | 使用事例 | ルールの例 |\n| ---------- | -------------- | ------------------------------------------------------------- |\n| `default` | 一般的なテキスト品質 | `RuleColonEnd` 、 `RuleContentNull` 、 `RuleDocRepeat`など。 |\n| `sft` | データセットの微調整 | `default`のルールと`RuleLineStartWithBulletpoint` |\n| `pretrain` | 事前トレーニングデータセット | `RuleAlphaWords` 、 `RuleCapitalWords`などを含む 20 以上の包括的なルールセット。 |\n\n特定のルールグループを使用するには:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# 特集のハイライト\n\n## マルチソース＆マルチモ��ダルサポート\n\n* **データソース**: ローカルファイル、Hugging Face データセット、S3 ストレージ\n* **データタイプ**: 事前トレーニング、微調整、評価データセット\n* **データ形式**: テキストと画像\n\n## ルールベースとモデルベースの評価\n\n* **組み込みルール**: 20以上の一般的なヒューリスティック評価ルール\n* **LLM 統合**: OpenAI、Kimi、ローカルモデル (例: Llama3)\n* **カスタムルール**: 独自のルールとモデルで簡単に拡張できます\n* **セキュリティ評価**：Perspective API統合\n\n## 柔軟な使い方\n\n* **インターフェース**: CLI および SDK オプション\n* **統合**: 他のプラットフォームとの簡単な統合\n* **実行エンジン**: ローカルと Spark\n\n## 包括的なレポート\n\n* **品質メトリクス**：7次元品質評価\n* **トレーサビリティ**: 異常追跡のための詳細なレポート\n\n# ユーザーガイド\n\n## カスタムルール、プロンプト、モデル\n\n組み込みルールが要件を満たしていない場合は、カスタムルールを作成できます。\n\n### カスタムルールの例\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### カスタムLLM統合\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nその他の例については以下を参照してください:\n\n* [登録ルール](examples/register/sdk_register_rule.py)\n* [登録プロンプト](examples/register/sdk_register_prompt.py)\n* [モデルを登録する](examples/register/sdk_register_llm.py)\n\n## 実行エンジン\n\n### ローカル実行\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Spark実行\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## 評価レポート\n\n評価後、Dingo は次を生成します。\n\n1. **概要レポート**（ `summary.json` ）：全体的な指標とスコア\n2. **詳細レポート**: 各ルール違反の具体的な問題\n\n例の要約:\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# MCP サーバー (実験的)\n\nDingoには、実験的なModel Context Protocol（MCP）サーバーが含まれています。サーバーの実行方法やCursorなどのクライアントとの統合方法については、専用のドキュメントをご覧ください。\n\n[**Dingo MCP サーバードキュメント (README\\_mcp.md)**](README_mcp.md)\n\n# 研究と出版\n\n* **「多言語ウェブデータの包括的なデータ品質評価」** ： [WanJuanSiLu：低リソース言語向けの高品質オープンソースウェブテキストデータセット](https://arxiv.org/pdf/2501.14506)\n* **「DataManメソッドを用いたデータ品質の事前学習」** ： [DataMan：大規模言語モデルの事前学習用データマネージャ](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# 今後の計画\n\n* \\[ ] より豊富なグラフィックとテキストの評価指標\n* \\[ ] 音声およびビデオデータのモダリティ評価\n* \\[ ] 小規模モデルの評価（fasttext、Qurating）\n* \\[ ] データ多様性評価\n\n# 制限事項\n\n現在組み込まれている検出ルールとモデル手法は、一般的なデータ品質の問題に重点を置いています。特殊な評価ニーズには、検出ルールをカスタマイズすることをお勧めします。\n\n# 謝辞\n\n* [RedPajamaデータ](https://github.com/togethercomputer/RedPajama-Data)\n* [mlフロー](https://github.com/mlflow/mlflow)\n\n# 貢献\n\n`Dingo`改善と拡張にご尽力いただいたすべての貢献者の皆様に感謝申し上げます。プロジェクトへの貢献方法については、[貢献ガイド](docs/en/CONTRIBUTING.md)をご覧ください。\n\n# ライセンス\n\nこのプロジェクトは[Apache 2.0 オープンソースライセンス](LICENSE)を使用します。\n\n# 引用\n\nこのプロジェクトが役に立つと思われる場合は、ツールを引用することを検討してください。\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","Japanese","ja-JP",{"_34":62,"_38":65,"_42":66},{"_36":63,"_38":63,"_39":64},"딩고 MCP 서버","[영어](README.md) ·[简体中文](README_zh-CN.md)\n\n# 변경 사항\n\n* 2024/12/27: 프로젝트 초기화\n\n# 소개\n\nDingo는 데이터 세트에서 데이터 품질 문제를 자동으로 감지하는 데 도움이 되는 데이터 품질 평가 도구입니다. Dingo는 다양한 기본 제공 규칙과 모델 평가 방법을 제공하며, 사용자 지정 평가 방법도 지원합니다. Dingo는 사전 학습 데이터 세트, 미세 조정 데이터 세트, 평가 데이터 세트를 포함하여 일반적으로 사용되는 텍스트 데이터 세트와 멀티모달 데이터 세트를 지원합니다. 또한 Dingo는 로컬 CLI 및 SDK를 포함한 다양한 사용 방법을 지원하여 [OpenCompass](https://github.com/open-compass/opencompass) 와 같은 다양한 평가 플랫폼에 쉽게 통합할 수 있습니다.\n\n## 아키텍처 다이어그램\n\n![딩고의 건축](./docs/assets/architeture.png)\n\n# 빠른 시작\n\n## 설치\n\n지엑스피1\n\n## 예시 사용 사례\n\n### 1. Evaluate Core 사용\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. 로컬 텍스트 파일(일반 텍스트) 평가\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. 포옹하는 얼굴 데이터 세트 평가\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. JSON/JSONL 형식 평가\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. 평가를 위한 LLM 사용\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## 명령줄 인터페이스\n\n### 규칙 세트로 평가\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### LLM(예: GPT-4o)으로 평가\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\n예시 `config_gpt.json` :\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## GUI 시각화\n\n평가 후( `save_data=True` 설정 시), 프런트엔드 페이지가 자동으로 생성됩니다. 프런트엔드를 수동으로 시작하려면 다음 단계를 따르세요.\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\n`output_directory` `summary.json` 파일과 함께 평가 결과가 포함되어 있습니다.\n\n![GUI 출력](docs/assets/dingo_gui.png)\n\n## 온라인 데모\n\n온라인 데모에서 Dingo를 사용해 보세요: [(허깅 페이스)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# 데이터 품질 지표\n\nDingo는 데이터 품질 문제를 7가지 품질 지표 차원으로 분류합니다. 각 차원은 규칙 기반 방법과 LLM 기반 프롬프트를 모두 사용하여 평가할 수 있습니다.\n\n| 품질 지표 | 설명 | 규칙 예제 | LLM 프롬프트 예시 |\n| ------- | --------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------- |\n| **완전성** | 데이터가 불완전하거나 누락되었는지 확인합니다. | `RuleColonEnd` , `RuleContentNull` | 텍스트가 콜론이나 생략 부호로 갑자기 끝나는지, 괄호가 일치하지 않는지, 중요한 구성 요소가 누락되었는지 평가합니다. |\n| **유효성** | 데이터가 의미 있고 적절하게 형식화되었는지 확인합니다. | `RuleAbnormalChar` , `RuleHtmlEntity` , `RuleSpecialCharacter` | 왜곡된 텍스트, 공백 없이 붙어 있는 단어, 적절한 구두점이 없는 텍스트를 감지합니다. |\n| **유창** | 텍스트가 문법적으로 올바르고 자연스럽게 읽히는지 확인합니다. | `RuleAbnormalNumber` , `RuleNoPunc` , `RuleWordStuck` | 지나치게 긴 단어, 구두점이 없는 텍스트 조각 또는 읽기 순서가 혼란스러운 콘텐츠를 식별합니다. |\n| **관련성** | 데이터 내에서 관련 없는 콘텐츠를 감지합니다. | 다양한 언어에 대한 `RuleHeadWord` 변형 | 인용 세부 정보, 머리글/바닥글, 엔터티 마커, HTML 태그와 같은 관련 없는 정보를 조사합니다. |\n| **보안** | 민감한 정보나 가치 충돌을 식별합니다. | `RuleIDCard` , `RuleUnsafeWords` | 개인정보 및 도박, 음란물, 정치 이슈 관련 콘텐츠 확인 |\n| **유사** | 반복적이거나 매우 유사한 콘텐츠를 감지합니다. | `RuleDocRepeat` | 연속적으로 반복되는 내용이나 특수 문자가 여러 번 나타나는지 텍스트를 평가합니다. |\n| **이해성** | 데이터를 얼마나 쉽게 해석할 수 있는지 평가합니다. | `RuleCapitalWords` | LaTeX 수식과 Markdown이 적절한 분할 및 줄 바꿈을 통해 올바르게 형식화되었는지 확인합니다. |\n\n## LLM 품질 평가\n\nDingo는 `dingo/model/prompt` 디렉터리의 프롬프트로 정의된 여러 LLM 기반 평가 방법을 제공합니다. 이러한 프롬프트는 `prompt_register` 데코레이터를 사용하여 등록되며, 품질 평가를 위해 LLM 모델과 결합될 수 있습니다.\n\n### 텍스트 품질 평가 프롬프트\n\n| 프롬프트 유형 | 미터법 | 설명 |\n| ------------------------------------- | --------- | ------------------------------------------------------ |\n| `TEXT_QUALITY_V2` , `TEXT_QUALITY_V3` | 다양한 품질 차원 | 효과성, 관련성, 완전성, 이해성, 유사성, 유창성 및 보안을 포괄하는 포괄적인 텍스트 품질 평가 |\n| `QUALITY_BAD_EFFECTIVENESS` | 유효성 | 왜곡된 텍스트와 크롤링 방지 콘텐츠를 감지합니다. |\n| `QUALITY_BAD_SIMILARITY` | 유사 | 텍스트 반복 문제를 식별합니다. |\n| `WORD_STICK` | 유창 | 적절한 간격 없이 붙어 있는 단어를 확인합니다. |\n| `CODE_LIST_ISSUE` | 완전성 | 코드 블록과 목록 형식 문제를 평가합니다. |\n| `UNREAD_ISSUE` | 유효성 | 인코딩 문제로 인해 읽을 수 없는 문자를 감지합니다. |\n\n### 3H 평가 프롬프트(정직함, 도움됨, 무해함)\n\n| 프롬프트 유형 | 미터법 | 설명 |\n| ------------------ | --- | ------------------------------------------ |\n| `QUALITY_HONEST` | 정직 | 응답이 조작이나 기만 없이 정확한 정보를 제공하는지 평가합니다. |\n| `QUALITY_HELPFUL` | 도움성 | 응답이 질문을 직접적으로 다루고 지침을 적절하게 따르는지 평가합니다. |\n| `QUALITY_HARMLESS` | 무해함 | 유해한 콘텐츠, 차별적인 언어 및 위험한 지원을 피하는 응답인지 확인합니다. |\n\n### 도메인별 평가 프롬프트\n\n| 프롬프트 유형 | 미터법 | 설명 |\n| -------------------- | ------------ | ------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_KAOTI` | 시험 문제 품질 | 공식 표현, 표 형식, 문단 구조, 답변 형식에 초점을 맞춰 시험 문제의 질을 평가하기 위한 전문화된 평가 |\n| `Html_Abstract` | HTML 추출 품질 | HTML에서 Markdown을 추출하는 다양한 방법을 비교하고 완전성, 서식 정확도 및 의미적 일관성을 평가합니다. |\n| `DATAMAN_ASSESSMENT` | 데이터 품질 및 도메인 | DataMan 방법론(14개 표준, 15개 도메인)을 사용하여 사전 학습 데이터 품질을 평가합니다. 점수(0/1), 도메인 유형, 품질 상태 및 이유를 부여합니다. |\n\n### 분류 프롬프트\n\n| 프롬프트 유형 | 미터법 | 설명 |\n| ---------------- | ------ | ------------------------------------------------------- |\n| `CLASSIFY_TOPIC` | 주제 분류 | 텍스트를 언어 처리, 쓰기, 코드, 수학, 롤 플레잉 또는 지식 Q\\&A와 같은 범주로 분류합니다. |\n| `CLASSIFY_QR` | 이미지 분류 | 이미지를 CAPTCHA, QR 코드 또는 일반 이미지로 식별합니다. |\n\n### 이미지 평가 프롬프트\n\n| 프롬프트 유형 | 미터법 | 설명 |\n| ----------------- | ------- | ------------------------------------------------------ |\n| `IMAGE_RELEVANCE` | 이미지 관련성 | 얼굴 수, 특징 세부 사항 및 시각적 요소 측면에서 이미지가 참조 이미지와 일치하는지 평가합니다. |\n\n### 평가에 LLM 평가 사용\n\n평가에서 이러한 평가 프롬프트를 사용하려면 구성에서 이를 지정하세요.\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\n이러한 프롬프트를 특정 품질 차원에 집중하거나 특정 도메인 요구 사항에 맞게 사용자 지정할 수 있습니다. 적절한 LLM 모델과 결합하면 이러한 프롬프트를 통해 여러 차원에 걸친 데이터 품질을 종합적으로 평가할 수 있습니다.\n\n# 규칙 그룹\n\nDingo는 다양한 유형의 데이터 세트에 대해 미리 구성된 규칙 그룹을 제공합니다.\n\n| 그룹 | 사용 사례 | 예제 규칙 |\n| ---------- | ------------ | ------------------------------------------------------------------- |\n| `default` | 일반 텍스트 품질 | `RuleColonEnd` , `RuleContentNull` , `RuleDocRepeat` 등 |\n| `sft` | 데이터 세트 미세 조정 | `default` 과 `RuleLineStartWithBulletpoint` 의 규칙 |\n| `pretrain` | 사전 학습 데이터 세트 | `RuleAlphaWords` , `RuleCapitalWords` 등을 포함한 20개 이상의 포괄적인 규칙 세트입니다. |\n\n특정 규칙 그룹을 사용하려면:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# 주요 기능\n\n## 다중 소스 및 다중 모달 지원\n\n* **데이터 소스** : 로컬 파일, Hugging Face 데이터 세트, S3 스토리지\n* **데이터 유형** : 사전 학습, 미세 조정 및 평가 데이터 세트\n* **데이터 모달리티** : 텍스트와 이미지\n\n## 규칙 기반 및 모델 기반 평가\n\n* **내장 규칙** : 20개 이상의 일반 휴리스틱 평가 규칙\n* **LLM 통합** : OpenAI, Kimi 및 로컬 모델(예: Llama3)\n* **사용자 정의 규칙** : 사용자 정의 규칙 및 모델로 쉽게 확장 가능\n* **보안 평가** : API 통합에 대한 관점\n\n## 유연한 사용\n\n* **인터페이스** : CLI 및 SDK 옵션\n* **통합** : 다른 플랫폼과의 쉬운 통합\n* **실행 엔진** : 로컬 및 스파크\n\n## 종합 보고\n\n* **품질 측정 기준** : 7차원 품질 평가\n* **추적성** : 이상 추적을 위한 상세 보고서\n\n# 사용자 가이드\n\n## 사용자 정의 규칙, 프롬프트 및 모델\n\n기본 제공 규칙이 요구 사항을 충족하지 못하는 경우 사용자 지정 규칙을 만들 수 있습니다.\n\n### 사용자 정의 규칙 예\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### 맞춤형 LLM 통합\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\n더 많은 예를 보려면 여기를 클릭하세요:\n\n* [등록 규칙](examples/register/sdk_register_rule.py)\n* [등록 프롬프트](examples/register/sdk_register_prompt.py)\n* [모델 등록](examples/register/sdk_register_llm.py)\n\n## 실행 엔진\n\n### 로컬 실행\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### 스파크 실행\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## 평가 보고서\n\n평가 후 Dingo는 다음을 생성합니다.\n\n1. **요약 보고서** ( `summary.json` ): 전체 지표 및 점수\n2. **상세 보고서** : 각 규칙 위반에 대한 구체적인 문제\n\n요약 예시:\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# MCP 서버(실험적)\n\nDingo에는 실험적인 모델 컨텍스트 프로토콜(MCP) 서버가 포함되어 있습니다. 서버 실행 및 Cursor와 같은 클라이언트와의 통합에 대한 자세한 내용은 해당 문서를 참조하십시오.\n\n[**Dingo MCP 서버 설명서(README\\_mcp.md)**](README_mcp.md)\n\n# 연구 및 출판\n\n* **\"다국어 웹데이터를 위한 종합적인 데이터 품질 평가\"** : [WanJuanSiLu: 저자원 언어를 위한 고품질 오픈소스 웹텍스트 데이터세트](https://arxiv.org/pdf/2501.14506)\n* **\"DataMan 방법론을 사용한 사전 학습 데이터 품질\"** : [DataMan: 대규모 언어 모델 사전 학습을 위한 데이터 관리자](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# 미래 계획\n\n* \\[ ] 더욱 풍부한 그래픽 및 텍스트 평가 지표\n* \\[ ] 오디오 및 비디오 데이터 모달리티 평가\n* \\[ ] 소규모 모델 평가(fasttext, Qurating)\n* \\[ ] 데이터 다양성 평가\n\n# 제한 사항\n\n현재 기본 제공되는 탐지 규칙과 모델 방법은 일반적인 데이터 품질 문제에 중점을 두고 있습니다. 특수한 평가가 필요한 경우 탐지 규칙을 사용자 정의하는 것이 좋습니다.\n\n# 감사의 말\n\n* [RedPajama-데이터](https://github.com/togethercomputer/RedPajama-Data)\n* [mlflow](https://github.com/mlflow/mlflow)\n\n# 기부금\n\n`Dingo` 개선하고 향상시켜 주신 모든 기여자분들께 감사드립니다. 프로젝트 참여에 대한 지침은 [기여 가이드](docs/en/CONTRIBUTING.md) 를 참조하세요.\n\n# 특허\n\n이 프로젝트는 [Apache 2.0 오픈 소스 라이선스를](LICENSE) 사용합니다.\n\n# 소환\n\n이 프로젝트가 유용하다고 생각되시면 저희 도구를 인용해 주시기 바랍니다.\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","Korean","ko-KR",{"_34":68,"_38":71,"_42":72},{"_36":69,"_38":69,"_39":70},"Сервер MCP Динго","[Английский](README.md) ·[简体中文](README_zh-CN.md)\n\n# Журнал изменений\n\n* 2024/12/27: Инициализация проекта\n\n# Введение\n\nDingo — это инструмент оценки качества данных, который помогает вам автоматически обнаруживать проблемы с качеством данных в ваших наборах данных. Dingo предоставляет множество встроенных правил и методов оценки моделей, а также поддерживает пользовательские методы оценки. Dingo поддерживает часто используемые текстовые наборы данных и мультимодальные наборы данных, включая наборы данных предварительной подготовки, наборы данных тонкой настройки и наборы данных оценки. Кроме того, Dingo поддерживает несколько методов использования, включая локальный CLI и SDK, что упрощает интеграцию в различные платформы оценки, такие как [OpenCompass](https://github.com/open-compass/opencompass) .\n\n## Архитектурная схема\n\n![Архитектура динго](./docs/assets/architeture.png)\n\n# Быстрый старт\n\n## Установка\n\n```shell\npip install dingo-python\n```\n\n## Примеры использования\n\n### 1. Использование Evaluate Core\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. Оценить локальный текстовый файл (открытый текст)\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. Оцените набор данных «Обнимающееся лицо»\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. Оцените формат JSON/JSONL\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. Использование LLM для оценки\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## Интерфейс командной строки\n\n### Оценка с помощью наборов правил\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### Оцените с помощью LLM (например, GPT-4o)\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\nПример `config_gpt.json` :\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## Визуализация графического интерфейса\n\nПосле оценки (с `save_data=True` ) будет автоматически сгенерирована страница frontend. Чтобы вручную запустить frontend:\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\nГде `output_directory` содержит результаты оценки с файлом `summary.json` .\n\n![Вывод графического интерфейса](docs/assets/dingo_gui.png)\n\n## Онлайн-демонстрация\n\nПопробуйте Dingo в нашей онлайн-демонстрации: [(Обнимающее лицо)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# Показатели качества данных\n\nDingo классифицирует проблемы качества данных по 7 измерениям метрик качества. Каждое измерение можно оценить с помощью как методов на основе правил, так и подсказок LLM:\n\n| Метрика качества | Описание | Примеры правил | Примеры подсказок LLM |\n| ----------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |\n| **ПОЛНОТА** | Проверяет, являются ли данные неполными или отсутствуют | `RuleColonEnd` , `RuleContentNull` | Проверяет, заканчивается ли текст двоеточием или многоточием, содержит ли он непарные скобки или отсутствуют ли в нем критические компоненты. |\n| **ЭФФЕКТИВНОСТЬ** | Проверяет, являются ли данные значимыми и правильно ли они отформатированы. | `RuleAbnormalChar` , `RuleHtmlEntity` , `RuleSpecialCharacter` | Обнаруживает искаженный текст, слова, слипшиеся без пробелов, и текст с отсутствующими знаками препинания. |\n| **БЕГЛОСТЬ** | Проверяет грамматическую правильность текста и его естественность при чтении | `RuleAbnormalNumber` , `RuleNoPunc` , `RuleWordStuck` | Определяет слишком длинные слова, фрагменты текста без знаков препинания или контент с хаотичным порядком чтения |\n| **АКТУАЛЬНОСТЬ** | Обнаруживает нерелевантный контент в данных | Варианты `RuleHeadWord` для разных языков | Проверяет на наличие нерелевантной информации, такой как сведения о цитатах, верхние/нижние колонтитулы, маркеры сущностей, HTML-теги |\n| **БЕЗОПАСНОСТЬ** | Выявляет конфиденциальную информацию или конфликты ценностей | `RuleIDCard` , `RuleUnsafeWords` | Проверяет персональные данные и контент, связанный с азартными играми, порнографией, политическими вопросами |\n| **СХОДСТВО** | Обнаруживает повторяющийся или очень похожий контент | `RuleDocRepeat` | Оценивает текст на предмет последовательно повторяющегося содержимого или многократного использования специальных символов. |\n| **ПОНЯТНОСТЬ** | Оценивает, насколько легко интерпретировать данные | `RuleCapitalWords` | Обеспечивает правильное форматирование формул LaTeX и Markdown с правильной сегментацией и переносами строк. |\n\n## Оценка качества LLM\n\nDingo предоставляет несколько методов оценки на основе LLM, определенных подсказками в каталоге `dingo/model/prompt` . Эти подсказки регистрируются с помощью декоратора `prompt_register` и могут быть объединены с моделями LLM для оценки качества:\n\n### Подсказки по оценке качества текста\n\n| Тип подсказки | Метрическая | Описание |\n| ------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_V2` , `TEXT_QUALITY_V3` | Различные качественные параметры | Комплексная оценка качества текста, охватывающая эффективность, релевантность, полноту, понятность, схожесть, беглость и безопасность |\n| `QUALITY_BAD_EFFECTIVENESS` | Эффективность | Обнаруживает искаженный текст и антисканирующий контент |\n| `QUALITY_BAD_SIMILARITY` | Сходство | Выявляет проблемы повторения текста |\n| `WORD_STICK` | Беглость | Проверяет слова на наличие слипшихся слов без надлежащего интервала |\n| `CODE_LIST_ISSUE` | Полнота | Оценивает блоки кода и проблемы форматирования списков |\n| `UNREAD_ISSUE` | Эффективность | Обнаруживает нечитаемые символы из-за проблем с кодировкой |\n\n### Подсказки по оценке 3H (честные, полезные, безвредные)\n\n| Тип подсказки | Метрическая | Описание |\n| ------------------ | ------------ | ----------------------------------------------------------------------------------------------------- |\n| `QUALITY_HONEST` | Честность | Оценивает, содержат ли ответы точную информацию без фальсификаций или обмана. |\n| `QUALITY_HELPFUL` | Полезность | Оценивает, отвечают ли ответы непосредственно на вопросы и следуют ли они соответствующим инструкциям |\n| `QUALITY_HARMLESS` | Безвредность | Проверяет, не содержат ли ответы вредоносного контента, дискриминационного языка и опасной помощи |\n\n### Подсказки для оценки предметной области\n\n| Тип подсказки | Метрическая | Описание |\n| -------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `TEXT_QUALITY_KAOTI` | Качество экзаменационных вопросов | Специализированная оценка качества экзаменационных вопросов, уделяющая особое внимание отображению формул, форматированию таблиц, структуре абзацев и форматированию ответов. |\n| `Html_Abstract` | Качество извлечения HTML | Сравнивает различные методы извлечения Markdown из HTML, оценивая полноту, точность форматирования и семантическую согласованность. |\n| `DATAMAN_ASSESSMENT` | Качество данных и домен | Оценивает качество данных до обучения с использованием методологии DataMan (14 стандартов, 15 доменов). Присваивает о��енку (0/1), тип домена, статус качества и причину. |\n\n### Подсказки по классификации\n\n| Тип подсказки | Метрическая | Описание |\n| ---------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |\n| `CLASSIFY_TOPIC` | Категоризация тем | Классифицирует текст по таким категориям, как языковая обработка, письмо, код, математика, ролевая игра или вопросы и ответы по знанию |\n| `CLASSIFY_QR` | Классификация изображений | Распознает изображения как CAPTCHA, QR-код или обычные изображения |\n\n### Подсказки по оценке изображения\n\n| Тип подсказки | Метрическая | Описание |\n| ----------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- |\n| `IMAGE_RELEVANCE` | Релевантность изображения | Оценивает, соответствует ли изображение эталонному изображению по количеству лиц, деталям и визуальным элементам. |\n\n### Использование оценки LLM при оценке\n\nЧтобы использовать эти подсказки для оценки в своих оценках, укажите их в своей конфигурации:\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nВы можете настроить эти подсказки, чтобы сосредоточиться на определенных и��мерениях качества или адаптироваться к определенным требованиям домена. В сочетании с соответствующими моделями LLM эти подсказки позволяют проводить комплексную оценку качества данных по нескольким измерениям.\n\n# Группы правил\n\nDingo предоставляет предварительно настроенные группы правил для различных типов наборов данных:\n\n| Группа | Вариант использования | Примеры правил |\n| ---------- | ------------------------------------------- | ------------------------------------------------------------------------------------------ |\n| `default` | Общее качество текста | `RuleColonEnd` , `RuleContentNull` , `RuleDocRepeat` и т. д. |\n| `sft` | Тонкая настройка наборов данных | Правила по `default` плюс `RuleLineStartWithBulletpoint` |\n| `pretrain` | Наборы данных для предварительного обучения | Полный набор из более чем 20 правил, включая `RuleAlphaWords` , `RuleCapitalWords` и т. д. |\n\nЧтобы использовать определенную группу правил:\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# Основные характеристики\n\n## Поддержка из разных источников и в разных режимах\n\n* **Источники данных** : локальные файлы, наборы данных Hugging Face, хранилище S3\n* **Типы данных** : наборы данных для предварительного обучения, тонкой настройки и оценки.\n* **Модальности данных** : Текст и изображение\n\n## Оценка на основе правил и моделей\n\n* **Встроенные правила** : более 20 общих эвристических правил оценки\n* **Интеграция LLM** : OpenAI, Kimi и локальные модели (например, Llama3)\n* **Пользовательские правила** : легко расширяйте свои собственные правила и модели.\n* **Оценка безопасности** : интеграция API Perspective\n\n## Гибкое использование\n\n* **Интерфейсы** : CLI и SDK-опции\n* **Интеграция** : Простая интеграция с другими платформами\n* **Движки исполнения** : локальные и Spark\n\n## Комплексная отчетность\n\n* **Метрики качества** : 7-мерная оценка качества\n* **Прослеживаемость** : подробные отчеты для отслеживания аномалий\n\n# Руководство пользователя\n\n## Пользовательские правила, подсказки и модели\n\nЕсли встроенные правила не соответствуют вашим требованиям, вы можете создать собственные:\n\n### Пример пользовательского правила\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### Индивидуальная интеграция LLM\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nБольше примеров смотрите в:\n\n* [Правила регистрации](examples/register/sdk_register_rule.py)\n* [Подсказки для регистрации](examples/register/sdk_register_prompt.py)\n* [Зарегистрировать модели](examples/register/sdk_register_llm.py)\n\n## Двигатели исполнения\n\n### Местное исполнение\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Исполнение Искры\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## Отчеты об оценке\n\nПосле оценки Dingo генерирует:\n\n1. **Сводный отчет** ( `summary.json` ): общие показатели и оценки\n2. **Подробные отчеты** : конкретные проблемы по каждому нарушению правил\n\nПример резюме:\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# MCP-сервер (экспериментальный)\n\nDingo включает экспериментальный сервер Model Context Protocol (MCP). Подробности о запуске сервера и его интеграции с такими клиентами, как Cursor, см. в специальной документации:\n\n[**Документация сервера Dingo MCP (README\\_mcp.md)**](README_mcp.md)\n\n# Исследования и публикации\n\n* **«Комплексная оценка качества данных для многоязычных веб-данных»** : [WanJuanSiLu: высококачественный набор данных веб-текста с открытым исходным кодом для языков с низкими ресурсами](https://arxiv.org/pdf/2501.14506)\n* **«Качество данных предварительной подготовки с использованием методологии DataMan»** : [DataMan: Менеджер данных для предварительной подготовки больших языковых моделей](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# Планы на будущее\n\n* \\[ ] Более богатые графические и текстовые оценочные индикаторы\n* \\[ ] Оценка модальности аудио и видео данных\n* \\[ ] Оценка небольшой модели (fasttext, Qurating)\n* \\[ ] Оценка разнообразия данных\n\n# Ограничения\n\nТекущие встроенные правила обнаружения и методы модели фокусируются на общих проблемах качества данных. Для специализированных потребностей оценки мы рекомендуем настраивать правила обнаружения.\n\n# Благодарности\n\n* [RedPajama-Данные](https://github.com/togethercomputer/RedPajama-Data)\n* [mlflow](https://github.com/mlflow/mlflow)\n\n# Вклад\n\nМы ценим всех участников за их усилия по улучшению и совершенствованию `Dingo` . Пожалуйста, обратитесь к [Руководству по внесению вклада](docs/en/CONTRIBUTING.md) для получения рекомендаций по внесению вклада в проект.\n\n# Лицензия\n\nВ этом проекте используется [лицензия Apache 2.0 Open Source](LICENSE) .\n\n# Цитата\n\nЕсли вы считаете этот проект полезным, пожалуйста, рассмотрите возможность цитирования нашего инструмента:\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","Russian","ru-RU",{"_34":74,"_38":77,"_42":78},{"_36":75,"_38":75,"_39":76},"Dingo MCP 服务器","[英文](README.md)·[简体中文](README_zh-CN.md)\n\n# 变更日志\n\n* 2024/12/27：项目初始化\n\n# 介绍\n\nDingo 是一款数据质量评估工具，可帮助您自动检测数据集中的数据质量问题。Dingo 提供多种内置规则和模型评估方法，并支持自定义评估方法。Dingo 支持常用的文本数据集和多模态数据集，包括预训练数据集、微调数据集和评估数据集。此外，Dingo 支持多种使用方式，包括本地 CLI 和 SDK，使其易于集成到各种评估平台，例如[OpenCompass](https://github.com/open-compass/opencompass) 。\n\n## 架构图\n\n![澳洲野狗的建筑](./docs/assets/architeture.png)\n\n# 快速入门\n\n## 安装\n\n```shell\npip install dingo-python\n```\n\n## 示例用例\n\n### 1. 使用评估核心\n\n```python\nfrom dingo.config.config import DynamicLLMConfig\nfrom dingo.io.input.MetaData import MetaData\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\n\ndef llm():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(\n key='',\n api_url='',\n # model='',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n data = MetaData(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n )\n\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. 评估本地文本文件（纯文本）\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a plaintext file\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"data.txt\", # Path to local text file\n \"dataset\": \"local\",\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 3. 评估 Hugging 人脸数据集\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"data_format\": \"plaintext\", # Format: plaintext\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 4. 评估 JSON/JSONL 格式\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a JSON file\ninput_data = {\n \"eval_group\": \"default\", # Default rule set\n \"input_path\": \"data.json\", # Path to local JSON file\n \"dataset\": \"local\",\n \"data_format\": \"json\", # Format: json\n \"column_content\": \"text\", # Column containing the text to evaluate\n \"save_data\": True # Save evaluation results\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n### 5. 使用 LLM 进行评估\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate using GPT model\ninput_data = {\n \"input_path\": \"data.jsonl\", # Path to local JSONL file\n \"dataset\": \"local\",\n \"data_format\": \"jsonl\",\n \"column_content\": \"content\",\n \"custom_config\": {\n \"prompt_list\": [\"PromptRepeat\"], # Prompt to use\n \"llm_config\": {\n \"detect_text_quality\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## 命令行界面\n\n### 使用规则集进行评估\n\n```shell\npython -m dingo.run.cli --input_path data.txt --dataset local -e sft --data_format plaintext --save_data True\n```\n\n### 使用 LLM 进行评估（例如 GPT-4o）\n\n```shell\npython -m dingo.run.cli --input_path data.json --dataset local -e openai --data_format json --column_content text --custom_config config_gpt.json --save_data True\n```\n\n示例`config_gpt.json` ：\n\n```json\n{\n \"llm_config\": {\n \"openai\": {\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n}\n```\n\n## GUI可视化\n\n评估后（使用`save_data=True` ），将自动生成前端页面。手动启动前端页面：\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\n其中`output_directory`包含带有`summary.json`文件的评估结果。\n\n![GUI 输出](docs/assets/dingo_gui.png)\n\n## 在线演示\n\n在我们的在线演示中尝试一下 Dingo：（[拥抱脸）🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n# 数据质量指标\n\nDingo 将数据质量问题分为 7 个质量指标维度。每个维度都可以使用基于规则的方法和基于 LLM 的提示进行评估：\n\n| 质量指标 | 描述 | 规则示例 | LLM 题目示例 |\n| ------- | ------------------ | -------------------------------------------------------------- | --------------------------------------- |\n| **完整性** | 检查数据是否不完整或缺失 | `RuleColonEnd`和`RuleContentNull` | 评估文本是否以冒号或省略号突然结束、括号是否不匹配或缺少关键部分 |\n| **效力** | 检查数据是否有意义且格式正确 | `RuleAbnormalChar` 、 `RuleHtmlEntity` 、 `RuleSpecialCharacter` | 检测乱码文本、没有空格的单词以及缺少正确标点符号的文本 |\n| **流利度** | 检查文本语法是否正确、读起来是否自然 | `RuleAbnormalNumber` 、 `RuleNoPunc` 、 `RuleWordStuck` | 识别过长的单词、没有标点符号的文本片段或阅读顺序混乱的内容 |\n| **关联** | 检测数据中的不相关内容 | 不同语言的`RuleHeadWord`变体 | 检查不相关的信息，如引用细节、页眉/页脚、实体标记、HTML 标签 |\n| **安全** | 识别敏感信息或价值冲突 | `RuleIDCard` 、 `RuleUnsafeWords` | 检查个人信息以及与赌博、色情、政治问题相关的内容 |\n| **相似** | 检测重复或高度相似的内容 | `RuleDocRepeat` | 评估文本中连续重复的内容或多次出现的特殊字符 |\n| **易懂** | 评估数据解释的难易程度 | `RuleCapitalWords` | 确保 LaTeX 公式和 Markdown 格式正确，并具有适当的分段和换行符 |\n\n## 法学硕士质量评估\n\nDingo 提供了几种基于 LLM 的评估方法，这些方法由`dingo/model/prompt`目录中的提示定义。这些提示使用`prompt_register`装饰器注册，可以与 LLM 模型结合进行质量评估：\n\n### 文本质量评估提示\n\n| 提示类型 | 公制 | 描述 |\n| ------------------------------------- | ------ | ---------------------------------------- |\n| `TEXT_QUALITY_V2` , `TEXT_QUALITY_V3` | 各种质量维度 | 全面的文本质量评估，涵盖有效性、相关性、完整性、可理解性、相似性、流畅性和安全性 |\n| `QUALITY_BAD_EFFECTIVENESS` | 效力 | 检测乱码文本和反爬取内容 |\n| `QUALITY_BAD_SIMILARITY` | 相似 | 识别文本重复问题 |\n| `WORD_STICK` | 流利度 | 检查单词是否粘在一起且没有适当的间距 |\n| `CODE_LIST_ISSUE` | 完整性 | 评估代码块和列表格式问题 |\n| `UNREAD_ISSUE` | 效力 | 检测由于编码问题导致的不可读字符 |\n\n### 3H 评估提示（诚实、有益、无害）\n\n| 提示类型 | 公制 | 描述 |\n| ------------------ | ---- | ------------------------ |\n| `QUALITY_HONEST` | 诚实 | 评估回复是否提供准确信息，无捏造或欺骗 |\n| `QUALITY_HELPFUL` | 乐于助人 | 评估回答是否直接解决问题并适当遵循指示 |\n| `QUALITY_HARMLESS` | 无害 | 检查回复是否避免有害内容、歧视性语言和危险的帮助 |\n\n### 特定领域的评估提示\n\n| 提示类型 | 公制 | 描述 |\n| -------------------- | -------- | -------------------------------------------------------------- |\n| `TEXT_QUALITY_KAOTI` | 考试题目质量 | 专门评估试题质量，重点关注公式呈现、表格格式、段落结构和答案格式 |\n| `Html_Abstract` | HTML提取质量 | 比较从 HTML 中提取 Markdown 的不同方法，评估完整性、格式准确性和语义一致性 |\n| `DATAMAN_ASSESSMENT` | 数据质量与领域 | 使用 DataMan 方法论（14 个标准，15 个领域）评估训练前数据质量。分配分数（0/1）、领域类型、质量状态和原因。 |\n\n### 分类提示\n\n| 提示类型 | 公制 | 描述 |\n| ---------------- | ---- | -------------------------------- |\n| `CLASSIFY_TOPIC` | 主题分类 | 将文本分类为语言处理、写作、代码、数学、角色扮演或知识问答等类别 |\n| `CLASSIFY_QR` | 图像分类 | 将图像识别为验证码、二维码或普通图像 |\n\n### 图像评估提示\n\n| 提示类型 | 公制 | 描述 |\n| ----------------- | ----- | ---------------------------- |\n| `IMAGE_RELEVANCE` | 图像相关性 | 评估图像在面数、特征细节和视觉元素方面是否与参考图像匹配 |\n\n### 在评估中使用 LLM 评估\n\n要在评估中使用这些评估提示，请在配置中指定它们：\n\n```python\ninput_data = {\n # Other parameters...\n \"custom_config\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n \"llm_config\": {\n \"detect_text_quality\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\n您可以自定义这些提示，使其侧重于特定的质量维度或适应特定领域的要求。与合适的 LLM 模型结合使用时，这些提示能够从多个维度全面评估数据质量。\n\n# 规则组\n\nDingo 为不同类型的数据集提供了预配置的规则组：\n\n| 团体 | 用例 | 示例规则 |\n| ---------- | ------ | ------------------------------------------------------ |\n| `default` | 一般文本质量 | `RuleColonEnd` 、 `RuleContentNull` 、 `RuleDocRepeat`等。 |\n| `sft` | 微调数据集 | `default`规则加上`RuleLineStartWithBulletpoint` |\n| `pretrain` | 预训练数据集 | 全面的 20 多条规则，包括`RuleAlphaWords` 、 `RuleCapitalWords`等。 |\n\n要使用特定规则组：\n\n```python\ninput_data = {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", or \"pretrain\"\n # other parameters...\n}\n```\n\n# 功能亮点\n\n## 多源和多模式支持\n\n* **数据源**：本地文件、Hugging Face 数据集、S3 存储\n* **数据类型**：预训练、微调和评估数据集\n* **数据形式**：文本和图像\n\n## 基于规则和基于模型的评估\n\n* **内置规则**：20+ 通用启发式评估规则\n* **LLM 集成**：OpenAI、Kimi 和本地模型（例如 Llama3）\n* **自定义规则**：轻松扩展您自己的规则和模型\n* **安全评估**：Perspective API 集成\n\n## 灵活使用\n\n* **接口**：CLI 和 SDK 选项\n* **集成**：轻松与其他平台集成\n* **执行引擎**：本地和 Spark\n\n## 综合报告\n\n* **质量指标**：7维质量评估\n* **可追溯性**：异常跟踪的详细报告\n\n# 用户指南\n\n## 自定义规则、提示和模型\n\n如果内置规则不能满足您的要求，您可以创建自定义规则：\n\n### 自定义规则示例\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.config import DynamicRuleConfig\nfrom dingo.io import MetaData\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = DynamicRuleConfig(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: MetaData) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### 定制 LLM 集成\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\n更多示例请参见：\n\n* [注册规则](examples/register/sdk_register_rule.py)\n* [注册提示](examples/register/sdk_register_prompt.py)\n* [注册模型](examples/register/sdk_register_llm.py)\n\n## 执行引擎\n\n### 本地执行\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Spark 执行\n\n```python\nfrom dingo.io import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as MetaData objects\n\ninput_args = InputArgs(eval_group=\"default\", save_data=True)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## 评估报告\n\n经过评估后，Dingo 生成：\n\n1. **摘要报告**（ `summary.json` ）：总体指标和分数\n2. **详细报告**：针对每项违反规则的具体问题\n\n示例摘要：\n\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# MCP 服务器（实验性）\n\nDingo 包含一个实验性的模型上下文协议 (MCP) 服务器。有关运行该服务器以及如何将其与 Cursor 等客户端集成的详细信息，请参阅专用文档：\n\n[**Dingo MCP 服务器文档 (README\\_mcp.md)**](README_mcp.md)\n\n# 研究与出版物\n\n* **“多语言WebData的综合数据质量评估”** ： [WanJuanSiLu：面向低资源语言的高质量开源Web文本数据集](https://arxiv.org/pdf/2501.14506)\n* **“使用 DataMan 方法论进行预训练数据质量”** ： [DataMan：用于预训练大型语言模型的数据管理器](https://openreview.net/pdf?id=eNbA8Fqir4)\n\n# 未来计划\n\n* \\[ ] 更丰富的图文评估指标\n* \\[ ] 音频和视频数据模态评估\n* \\[ ] 小模型评估（fasttext、Qurating）\n* \\[ ] 数据多样性评估\n\n# 限制\n\n目前内置的检测规则和模型方法主要针对常见的数据质量问题，对于有特殊需求的评估，建议自定义检测规则。\n\n# 致谢\n\n* [RedPajama-数据](https://github.com/togethercomputer/RedPajama-Data)\n* [mlflow](https://github.com/mlflow/mlflow)\n\n# 贡献\n\n我们感谢所有为改进和增强`Dingo`而做出贡献的贡献者。请参阅[贡献指南](docs/en/CONTRIBUTING.md)，了解如何为项目做出贡献。\n\n# 执照\n\n该项目使用[Apache 2.0 开源许可证](LICENSE)。\n\n# 引文\n\n如果您发现这个项目有用，请考虑引用我们的工具：\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/DataEval/dingo}},\n year={2024}\n}\n```","Chinese","zh-CN","githubUser","locale",{"_82":83,"_20":84,"_89":37,"_90":37,"_23":91,"_92":93,"_38":37,"_159":160,"_167":-5,"_168":-5,"_169":170,"_216":217,"_222":-5,"_100":201,"_223":224,"_8":225,"_226":227},"addedAt","2025-05-06T07:03:44.336849Z",[85,86,87,88],"category:databases","category:data-platforms","language:javascript","license_grade:a","descriptionMarkdown","descriptionPlainText",6362,"integrations",[94,104,111,118,125,132,139,146,152],{"_95":96,"_36":102,"_8":103},"brand",{"_97":98,"_38":99,"_100":101},"logoSvgPath","M20.317 4.3698a19.7913 19.7913 0 00-4.8851-1.5152.0741.0741 0 00-.0785.0371c-.211.3753-.4447.8648-.6083 1.2495-1.8447-.2762-3.68-.2762-5.4868 0-.1636-.3933-.4058-.8742-.6177-1.2495a.077.077 0 00-.0785-.037 19.7363 19.7363 0 00-4.8852 1.515.0699.0699 0 00-.0321.0277C.5334 9.0458-.319 13.5799.0992 18.0578a.0824.0824 0 00.0312.0561c2.0528 1.5076 4.0413 2.4228 5.9929 3.0294a.0777.0777 0 00.0842-.0276c.4616-.6304.8731-1.2952 1.226-1.9942a.076.076 0 00-.0416-.1057c-.6528-.2476-1.2743-.5495-1.8722-.8923a.077.077 0 01-.0076-.1277c.1258-.0943.2517-.1923.3718-.2914a.0743.0743 0 01.0776-.0105c3.9278 1.7933 8.18 1.7933 12.0614 0a.0739.0739 0 01.0785.0095c.1202.099.246.1981.3728.2924a.077.077 0 01-.0066.1276 12.2986 12.2986 0 01-1.873.8914.0766.0766 0 00-.0407.1067c.3604.698.7719 1.3628 1.225 1.9932a.076.076 0 00.0842.0286c1.961-.6067 3.9495-1.5219 6.0023-3.0294a.077.077 0 00.0313-.0552c.5004-5.177-.8382-9.6739-3.5485-13.6604a.061.061 0 00-.0312-.0286zM8.02 15.3312c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9555-2.4189 2.157-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.9555 2.4189-2.1569 2.4189zm7.9748 0c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9554-2.4189 2.1569-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.946 2.4189-2.1568 2.4189Z","Discord","slug","discord","Offers community support through Discord channel","s0ss3vnm1z",{"_95":105,"_36":109,"_8":110},{"_97":106,"_38":107,"_100":108},"M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12","GitHub","github","References GitHub for project hosting, stars, forks, and issue tracking","ln7mvjyrwv",{"_95":112,"_36":116,"_8":117},{"_97":113,"_38":114,"_100":115},"M12.025 1.13c-5.77 0-10.449 4.647-10.449 10.378 0 1.112.178 2.181.503 3.185.064-.222.203-.444.416-.577a.96.96 0 0 1 .524-.15c.293 0 .584.124.84.284.278.173.48.408.71.694.226.282.458.611.684.951v-.014c.017-.324.106-.622.264-.874s.403-.487.762-.543c.3-.047.596.06.787.203s.31.313.4.467c.15.257.212.468.233.542.01.026.653 1.552 1.657 2.54.616.605 1.01 1.223 1.082 1.912.055.537-.096 1.059-.38 1.572.637.121 1.294.187 1.967.187.657 0 1.298-.063 1.921-.178-.287-.517-.44-1.041-.384-1.581.07-.69.465-1.307 1.081-1.913 1.004-.987 1.647-2.513 1.657-2.539.021-.074.083-.285.233-.542.09-.154.208-.323.4-.467a1.08 1.08 0 0 1 .787-.203c.359.056.604.29.762.543s.247.55.265.874v.015c.225-.34.457-.67.683-.952.23-.286.432-.52.71-.694.257-.16.547-.284.84-.285a.97.97 0 0 1 .524.151c.228.143.373.388.43.625l.006.04a10.3 10.3 0 0 0 .534-3.273c0-5.731-4.678-10.378-10.449-10.378M8.327 6.583a1.5 1.5 0 0 1 .713.174 1.487 1.487 0 0 1 .617 2.013c-.183.343-.762-.214-1.102-.094-.38.134-.532.914-.917.71a1.487 1.487 0 0 1 .69-2.803m7.486 0a1.487 1.487 0 0 1 .689 2.803c-.385.204-.536-.576-.916-.71-.34-.12-.92.437-1.103.094a1.487 1.487 0 0 1 .617-2.013 1.5 1.5 0 0 1 .713-.174m-10.68 1.55a.96.96 0 1 1 0 1.921.96.96 0 0 1 0-1.92m13.838 0a.96.96 0 1 1 0 1.92.96.96 0 0 1 0-1.92M8.489 11.458c.588.01 1.965 1.157 3.572 1.164 1.607-.007 2.984-1.155 3.572-1.164.196-.003.305.12.305.454 0 .886-.424 2.328-1.563 3.202-.22-.756-1.396-1.366-1.63-1.32q-.011.001-.02.006l-.044.026-.01.008-.03.024q-.018.017-.035.036l-.032.04a1 1 0 0 0-.058.09l-.014.025q-.049.088-.11.19a1 1 0 0 1-.083.116 1.2 1.2 0 0 1-.173.18q-.035.029-.075.058a1.3 1.3 0 0 1-.251-.243 1 1 0 0 1-.076-.107c-.124-.193-.177-.363-.337-.444-.034-.016-.104-.008-.2.022q-.094.03-.216.087-.06.028-.125.063l-.13.074q-.067.04-.136.086a3 3 0 0 0-.135.096 3 3 0 0 0-.26.219 2 2 0 0 0-.12.121 2 2 0 0 0-.106.128l-.002.002a2 2 0 0 0-.09.132l-.001.001a1.2 1.2 0 0 0-.105.212q-.013.036-.024.073c-1.139-.875-1.563-2.317-1.563-3.203 0-.334.109-.457.305-.454m.836 10.354c.824-1.19.766-2.082-.365-3.194-1.13-1.112-1.789-2.738-1.789-2.738s-.246-.945-.806-.858-.97 1.499.202 2.362c1.173.864-.233 1.45-.685.64-.45-.812-1.683-2.896-2.322-3.295s-1.089-.175-.938.647 2.822 2.813 2.562 3.244-1.176-.506-1.176-.506-2.866-2.567-3.49-1.898.473 1.23 2.037 2.16c1.564.932 1.686 1.178 1.464 1.53s-3.675-2.511-4-1.297c-.323 1.214 3.524 1.567 3.287 2.405-.238.839-2.71-1.587-3.216-.642-.506.946 3.49 2.056 3.522 2.064 1.29.33 4.568 1.028 5.713-.624m5.349 0c-.824-1.19-.766-2.082.365-3.194 1.13-1.112 1.789-2.738 1.789-2.738s.246-.945.806-.858.97 1.499-.202 2.362c-1.173.864.233 1.45.685.64.451-.812 1.683-2.896 2.322-3.295s1.089-.175.938.647-2.822 2.813-2.562 3.244 1.176-.506 1.176-.506 2.866-2.567 3.49-1.898-.473 1.23-2.037 2.16c-1.564.932-1.686 1.178-1.464 1.53s3.675-2.511 4-1.297c.323 1.214-3.524 1.567-3.287 2.405.238.839 2.71-1.587 3.216-.642.506.946-3.49 2.056-3.522 2.064-1.29.33-4.568 1.028-5.713-.624","Hugging Face","huggingface","Supports interaction with Hugging Face datasets, enabling evaluation of data quality for datasets hosted on the platform","0ryg09dg69",{"_95":119,"_36":123,"_8":124},{"_97":120,"_38":121,"_100":122},"M2.176 2.814c.233.42.476.78.73 1.09.247-.013 1.132.456 1.312.523.508.282 1.063.63 1.567.966.505.337.96.662 1.272.9.156.12.278.218.352.286a.483.483 0 01.078.082.08.08 0 01.01.021.06.06 0 01-.004.047.057.057 0 01-.04.03.077.077 0 01-.028 0c-.057 0-.203-.163-.497-.415a23.474 23.474 0 00-2.759-1.827c-.504-.28-.956-.542-1.264-.613a2.322 2.322 0 00-.36-.025 2.706 2.706 0 00-.788.133c.494.414.91.716 1.28.949-.57-.182-1.182-.21-1.902.133.526.329.967.567 1.354.745 1.103.156 2.258.696 3.224 1.309.483.307.904.615 1.219.867.157.125.29.237.39.328.098.091.174.154.197.21.03.073-.019.104-.084.058-.032-.022-.088-.102-.184-.191a7.35 7.35 0 00-.384-.327c-.312-.25-.729-.552-1.209-.857-.893-.562-2.232-1.013-3.173-1.397-.602-.11-1.225-.06-1.906.39.449.2.837.349 1.182.463.812 0 1.892.365 2.935.922 1.042.556 2.04 1.214 2.523 1.774.066.077-.016.126-.074.07-.52-.495-1.463-1.204-2.498-1.756-.639-.337-2.153-1.01-2.886-1.01l.004.002c-.567.02-1.13.195-1.679.716.477.118.885.196 1.244.249-.44.088-.87.3-1.289.722.324.07.616.122.882.162-.328.159-.639.404-.923.78.373.03.703.042 1 .044-.36.166-.696.43-.996.85.533.027.98.025 1.364.003-.422.172-.812.464-1.145.968.662.01 1.188-.022 1.628-.076l-.006.002c.99-.073 2.297.127 2.962.847.052.057-.024.118-.072.074-.648-.58-1.493-.827-2.89-.921h-.002c-.543.149-1.046.446-1.46 1.074.536.008.982-.013 1.366-.05-.469.257-.873.644-1.139 1.306.483-.092.888-.19 1.237-.292-.363.265-.668.636-.873 1.194.324-.072.612-.146.871-.221a2.519 2.519 0 00-.513 1.095c.352-.13.655-.254.926-.377-.257.3-.453.681-.55 1.19.495-.199.899-.388 1.238-.568-.31.333-.543.76-.635 1.356a11.816 11.816 0 001.442-.744c-.433.362-.764.843-.879 1.587.788-.348 1.339-.663 1.767-.955-.184.372-.282.806-.235 1.348.762-.584 1.243-1.056 1.602-1.473-.024.269-.003.56.077.884.546-.939 1.089-1.212 1.65-1.526-.895.451-.762.79-.762 1.184.683-.72 1.635-1.482 1.927-1.96-.39.585-.547 1.14-.65 1.63-1.993 1.054-3.207 1.329-4.568 1.75.528.194 1.093.383.859.652l-.624.622c.399-.124.805-.3 1.158-.059-.035.327-.447.492-.8.683.621-.224.756-.172.92-.12.081.393-.203.603-.388.862 1.565-1.19 3.606-2.13 5.044-2.522 2.022-.681 4.63-1.389 5.339-3.115l.712-2.847-.004.004c-.111-.034-.246-.063-.35-.133a.651.651 0 01-.235-.297c-.252.065-.44.03-.56-.088-.117-.117-.167-.296-.203-.491-.203.041-.362.016-.467-.077-.116-.101-.17-.26-.198-.444l-.008-.039.037-.015a.842.842 0 00.302-.194.257.257 0 00.07-.225l-.006-.037.03-.016c.163-.093.345-.169.428-.28a.274.274 0 00.053-.21.88.88 0 00-.155-.357l-.027-.04.04-.027c.118-.09.244-.179.308-.26.032-.04.048-.076.047-.11 0-.033-.015-.07-.064-.117l-.098-.094.135.006c.213.01.395-.007.538-.053a.504.504 0 00.274-.197c-.007-.033-.02-.063-.02-.098a.484.484 0 01.967 0c0 .044-.015.084-.026.125.177.014.347.01.507-.06l.002.001.035-.013c.236-.085.334.045.72-.456-1.69-2.19-4.157-.635-4.977 1.622-.21.576-1.405.578-1.751 0-1.37-2.95-5.53-6.068-9.07-7.218zm.86 2.145c.906.293 1.913.782 2.77 1.328.43.273.813.543 1.114.779.301.236.566.473.62.575.054.102 0 .14-.082.06-.081-.078-.303-.32-.6-.553-.298-.234-.68-.505-1.106-.777-.775-.49-1.982-.958-2.716-1.412zm-1.7 2.7c1.116.014 2.35.447 3.434.997.541.275 1.023.567 1.395.83.372.263.672.524.734.657.061.134-.02.13-.087.055a4.401 4.401 0 00-.704-.626 11.47 11.47 0 00-1.385-.826C3.76 8.264 2.439 7.82 1.336 7.66zm14.916.772a.381.381 0 100 .762.381.381 0 000-.762zM1.7 8.478c.822.072 1.72.368 2.534.75 1.086.509 2.035 1.158 2.434 1.667.035.045-.014.131-.08.062-.428-.44-1.322-1.131-2.397-1.635-.913-.421-2.282-.87-3.262-.78.251-.03.497-.088.771-.064zm16.339.01c-.366.475-.53.423-.703.464.094.43.35.586.585.77l-.06.012c2.315-.447 4.186-.286 6.139-.236l-5.961-1.01zm-.178 1.246h-.002l-.004.016.006-.016zm-.625-.757c-.183.074-.373.076-.563.059a.477.477 0 01-.42.26.483.483 0 01-.435-.278.609.609 0 01-.274.188c-.139.045-.308.057-.493.055.02.035.054.068.055.104a.273.273 0 01-.069.174c-.073.092-.189.17-.295.248.087.141.137.26.149.362a.39.39 0 01-.07.284c-.106.14-.288.21-.439.293a.374.374 0 01-.09.268.89.89 0 01-.297.198c.027.156.074.283.154.354.086.076.211.103.425.047l.055-.014.01.055c.033.207.088.385.187.483.1.099.244.135.503.055l.049-.015.016.048c.05.142.12.223.209.282.087.06.247.112.358.147.798-.869 1.525-1.772 1.884-2.86-.225-.177-.506-.338-.609-.797zm-16.23.386c1.165-.08 2.283.196 3.202.626.92.43 1.658.939 1.974 1.307.075.087-.019.12-.072.072a8.187 8.187 0 00-1.947-1.29c-.904-.414-2.193-.644-3.157-.715zm.864.802c.61.02 1.24.155 1.806.352.756.262 1.421.614 1.747.98.045.05-.007.127-.074.069-.349-.304-.961-.693-1.706-.951-.574-.195-1.613-.369-2.268-.397.197-.022.292-.06.495-.053zm1.05 1.788c.423.034.886.133 1.341.407.043.026.049.136-.049.09-.856-.402-1.326-.49-2.457-.31.386-.128.74-.221 1.164-.187zm-.04.788c.4-.035.784-.002 1.297.204.044.018.08.126-.033.094-.857-.243-1.167-.328-2.287.104.28-.229.622-.366 1.023-.402zm1.285.687c.317-.023.635-.026.934.006.052.006.055.105-.006.102a7.87 7.87 0 00-1.837.115c-.243.046-.423.043-1.405.458.287-.233.794-.452 1.385-.56a8.91 8.91 0 01.93-.12zm1.28.49c.099.003.062.104.006.103-.728-.01-1.304.132-1.875.295a9.78 9.78 0 00-1.318.525c.283-.23.713-.457 1.291-.622.579-.166 1.248-.326 1.896-.302zm.528.398c.036-.005.105.084.018.1-.73.137-1.244.267-1.794.454-.216.074-.58.207-1.243.587.26-.269.656-.492 1.213-.68.558-.19 1.196-.37 1.806-.46zm.311.507c.075-.012.097.087.02.102-1.217.241-1.76.556-2.54 1.144.504-.523 1.297-1.051 2.52-1.246zm.595.448c.087-.013.11.087.021.1-.872.13-1.477.553-2.255 1.33.295-.493 1.004-1.24 2.234-1.43zm.372.39c.046-.006.114.073.023.1a2.634 2.634 0 00-.669.3c-.182.118-.3.2-.597.507.111-.245.296-.434.542-.59.247-.157.509-.293.7-.317z","LaTeX","latex","Provides evaluation capabilities for LaTeX formulas in datasets","v96jdn3ep1",{"_95":126,"_36":130,"_8":131},{"_97":127,"_38":128,"_100":129},"M22.27 19.385H1.73A1.73 1.73 0 010 17.655V6.345a1.73 1.73 0 011.73-1.73h20.54A1.73 1.73 0 0124 6.345v11.308a1.73 1.73 0 01-1.73 1.731zM5.769 15.923v-4.5l2.308 2.885 2.307-2.885v4.5h2.308V8.078h-2.308l-2.307 2.885-2.308-2.885H3.46v7.847zM21.232 12h-2.309V8.077h-2.307V12h-2.308l3.461 4.039z","Markdown","markdown","Supports evaluation of Markdown formatting in datasets and content extraction quality assessment","fd0xjq4xu4",{"_95":133,"_36":137,"_8":138},{"_97":134,"_38":135,"_100":136},"M11.883.002a12.044 12.044 0 0 0-9.326 19.463l3.668-2.694A7.573 7.573 0 0 1 12.043 4.45v2.867l6.908-5.14A12.012 12.012 0 0 0 11.883.002zm9.562 4.533L17.777 7.23a7.573 7.573 0 0 1-5.818 12.322v-2.867l-6.908 5.14a12.046 12.046 0 0 0 16.394-17.29z","MLflow","mlflow","Acknowledges MLflow as a related project in the model evaluation ecosystem","20xbp5a7jv",{"_95":140,"_36":144,"_8":145},{"_97":141,"_38":142,"_100":143},"M22.2819 9.8211a5.9847 5.9847 0 0 0-.5157-4.9108 6.0462 6.0462 0 0 0-6.5098-2.9A6.0651 6.0651 0 0 0 4.9807 4.1818a5.9847 5.9847 0 0 0-3.9977 2.9 6.0462 6.0462 0 0 0 .7427 7.0966 5.98 5.98 0 0 0 .511 4.9107 6.051 6.051 0 0 0 6.5146 2.9001A5.9847 5.9847 0 0 0 13.2599 24a6.0557 6.0557 0 0 0 5.7718-4.2058 5.9894 5.9894 0 0 0 3.9977-2.9001 6.0557 6.0557 0 0 0-.7475-7.0729zm-9.022 12.6081a4.4755 4.4755 0 0 1-2.8764-1.0408l.1419-.0804 4.7783-2.7582a.7948.7948 0 0 0 .3927-.6813v-6.7369l2.02 1.1686a.071.071 0 0 1 .038.052v5.5826a4.504 4.504 0 0 1-4.4945 4.4944zm-9.6607-4.1254a4.4708 4.4708 0 0 1-.5346-3.0137l.142.0852 4.783 2.7582a.7712.7712 0 0 0 .7806 0l5.8428-3.3685v2.3324a.0804.0804 0 0 1-.0332.0615L9.74 19.9502a4.4992 4.4992 0 0 1-6.1408-1.6464zM2.3408 7.8956a4.485 4.485 0 0 1 2.3655-1.9728V11.6a.7664.7664 0 0 0 .3879.6765l5.8144 3.3543-2.0201 1.1685a.0757.0757 0 0 1-.071 0l-4.8303-2.7865A4.504 4.504 0 0 1 2.3408 7.872zm16.5963 3.8558L13.1038 8.364 15.1192 7.2a.0757.0757 0 0 1 .071 0l4.8303 2.7913a4.4944 4.4944 0 0 1-.6765 8.1042v-5.6772a.79.79 0 0 0-.407-.667zm2.0107-3.0231l-.142-.0852-4.7735-2.7818a.7759.7759 0 0 0-.7854 0L9.409 9.2297V6.8974a.0662.0662 0 0 1 .0284-.0615l4.8303-2.7866a4.4992 4.4992 0 0 1 6.6802 4.66zM8.3065 12.863l-2.02-1.1638a.0804.0804 0 0 1-.038-.0567V6.0742a4.4992 4.4992 0 0 1 7.3757-3.4537l-.142.0805L8.704 5.459a.7948.7948 0 0 0-.3927.6813zm1.0976-2.3654l2.602-1.4998 2.6069 1.4998v2.9994l-2.5974 1.4997-2.6067-1.4997Z","OpenAI","openai","Integrates with OpenAI models like GPT-4o for LLM-based data quality assessment using various evaluation prompts","aays4f5yso",{"_95":147,"_36":150,"_8":151},{"_97":148,"_38":149,"_100":149},"M12 0c-.563 0-1.127.215-1.557.645L.645 10.443c-.86.86-.86 2.254 0 3.114l9.798 9.798c.86.86 2.254.86 3.114 0l9.798-9.798c.86-.86.86-2.254 0-3.114L13.557.645A2.195 2.195 0 0012 0zm0 1.74c.493 0 .987.186 1.361.56L21.7 10.64c.75.75.75 1.973 0 2.722L13.361 21.7c-.374.375-.868.56-1.361.56s-.987-.185-1.361-.56L2.3 13.361a1.93 1.93 0 010-2.722L10.639 2.3c.374-.375.868-.56 1.361-.56zm0 .62c-.333 0-.664.127-.92.382L2.742 11.08a1.295 1.295 0 000 1.84l8.338 8.338a1.296 1.296 0 001.84 0l8.338-8.338a1.295 1.295 0 000-1.84L12.92 2.742A1.296 1.296 0 0012 2.36zM9.207 7.624h3.959c.55 0 1.015.079 1.4.238.385.16.7.371.942.633.241.262.417.561.527.897a3.34 3.34 0 010 2.084c-.11.34-.286.64-.527.902a2.628 2.628 0 01-.942.633c-.385.16-.85.238-1.4.238h-2.043v3.156H9.207zm1.916 1.484v2.657h1.514c.222 0 .434-.016.64-.05.207-.032.39-.097.547-.193.158-.095.284-.23.38-.404.094-.174.142-.401.142-.682 0-.28-.048-.507-.143-.681a1.053 1.053 0 00-.379-.404 1.513 1.513 0 00-.547-.194 4.056 4.056 0 00-.64-.049z","pre-commit","Integrates with pre-commit for code quality checks","82p9fz2jfc",{"_95":153,"_36":157,"_8":158},{"_97":154,"_38":155,"_100":156},"M23.922 13.58v3.912L20.55 18.72l-.078.055.052.037 3.45-1.256.026-.036v-3.997l-.053-.036-.025.092z M23.621 5.618l-3.04 1.107v3.912l3.339-1.215V5.509zM23.92 13.457V9.544l-3.336 1.215v3.913zM20.47 14.71V10.8L17.17 12v3.913zM17.034 19.996v-3.912l-3.313 1.206v3.912zM17.17 16.057v3.868l3.314-1.206V14.85l-3.314 1.206zm2.093 1.882c-.367.134-.663-.074-.663-.463s.296-.814.663-.947c.365-.133.662.075.662.464s-.297.814-.662.946z M13.225 9.315l.365-.132-3.285-1.197-3.323 1.21.102.037 3.184 1.16zM20.507 10.664V6.751L17.17 7.965v3.913zM17.058 11.918V8.005l-3.302 1.202v3.912zM13.643 9.246l-3.336 1.215v3.913l3.336-1.215zM6.907 13.165l3.322 1.209v-3.913L6.907 9.252z M10.34 7.873l3.281 1.193V5.198l-3.28-1.193zM20.507 2.715L17.19 3.922v3.913l3.317-1.207zM16.95 3.903L13.724 2.73l-3.269 1.19 3.225 1.174zM15.365 4.606l-1.624.592v3.868l3.317-1.207V3.991l-1.693.615zm-.391 2.778c-.367.134-.662-.074-.662-.464s.295-.813.662-.946c.366-.133.663.074.663.464s-.297.813-.663.946z M10.229 18.41v-3.914l-3.322-1.209V17.2zM13.678 17.182v-3.913l-3.371 1.227v3.913z M13.756 17.154l3.3-1.2V12.04l-3.3 1.2zM13.678 21.217l-3.371 1.227v-3.912h-.078v3.912l-3.322-1.209v-3.913l-.053-.058-.025-.06-3.336-1.21v-3.948l.034.013 3.287 1.196.015-.078-3.261-1.187 3.26-1.187v-.109L3.876 9.62l-.307-.112 3.26-1.188v.877l.079-.055V6.769l3.257 1.185.058-.061L7.084 6.75l-.102-.037 3.24-1.179v-.083L6.854 6.677v.018l-.025.018v1.523L3.44 9.47v.02l-.025.017v4.007l-3.39 1.233v.019L0 14.784v3.995l.025.037 3.4 1.237.008-.006.007.01 3.4 1.238.008-.006.006.01 3.4 1.237.014-.009.012.01 3.45-1.256.026-.037-.078-.027zM3.493 9.563l3.257 1.185-3.257 1.187V9.562zM3.4 19.96L.078 18.752v-3.913l2.361.86.96.349v3.913zm.015-3.99L.335 14.85l-.182-.066 3.262-1.187v2.374zm3.399 5.231l-3.321-1.209v-3.912l3.321 1.209v3.912zM23.791 5.434l-3.21-1.17v2.338zM20.387 2.643l-3.24-1.18-3.27 1.19 3.247 1.182z","PyPI","pypi","Enables installation through PyPI package registry","igviuhyvv8","namespace",{"_161":162,"_100":166},"owner",{"_163":164},"githubAccount",{"_165":166},"login","MigoXLab","readmeMarkdown","redditPostExternalId","repository",{"_171":172,"_185":186,"_190":191,"_211":-5,"_212":213,"_214":-5,"_215":-5},"argumentsJsonSchema",{"_173":174,"_182":183,"_12":184},"properties",{"_175":176,"_179":180},"PUBMED_EMAIL",{"_36":177,"_12":178},"Your email address (required by NCBI)","string","PUBMED_API_KEY",{"_36":181,"_12":178},"Optional API key for higher rate limits","required",[175],"object","githubProject",{"_39":187,"_188":189},"

\n\n

\n \n

\n\n\n

\n $\"pre-commit\"$ \n $\"PyPI$ \n $\"Python$ \n $\"License\"$ \n $\"GitHub$ \n $\"GitHub$ \n $\"GitHub$ \n $\"MseeP.ai$ \n $\"Ask$ \n\n[![Trust Score](https://archestra.ai/mcp-catalog/api/badge/quality/DataEval/dingo)](https://archestra.ai/mcp-catalog/dataeval__dingo)\n

\n\n

\n\n\n

\n\n[English](README.md) · [简体中文](README_zh-CN.md) · [日本語](README_ja.md)\n\n

\n\n\n\n\n

\n 👋 join us on Discord and WeChat\n

\n\n\n

\n If you like Dingo, please give us a ⭐ on GitHub!\n
\n \n $\"Click$ \n \n

\n\n\n# Introduction\n\nDingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as [OpenCompass](https://github.com/open-compass/opencompass).\n\n## Architecture Diagram\n\n![Architecture of dingo](./docs/assets/architeture.png)\n\n# Quick Start\n\n## Installation\n\n```shell\npip install dingo-python\n```\n\n## Example Use Cases\n\n### 1. Evaluate LLM chat data\n\n```python\nfrom dingo.config.input_args import EvaluatorLLMArgs\nfrom dingo.io.input import Data\nfrom dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase\nfrom dingo.model.rule.rule_common import RuleEnterAndSpace\n\ndata = Data(\n data_id='123',\n prompt=\"hello, introduce the world\",\n content=\"Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty.\"\n)\n\n\ndef llm():\n LLMTextQualityModelBase.dynamic_config = EvaluatorLLMArgs(\n key='YOUR_API_KEY',\n api_url='https://api.openai.com/v1/chat/completions',\n model='gpt-4o',\n )\n res = LLMTextQualityModelBase.eval(data)\n print(res)\n\n\ndef rule():\n res = RuleEnterAndSpace().eval(data)\n print(res)\n```\n\n### 2. Evaluate Dataset\n\n```python\nfrom dingo.config import InputArgs\nfrom dingo.exec import Executor\n\n# Evaluate a dataset from Hugging Face\ninput_data = {\n \"input_path\": \"tatsu-lab/alpaca\", # Dataset from Hugging Face\n \"dataset\": {\n \"source\": \"hugging_face\",\n \"format\": \"plaintext\" # Format: plaintext\n },\n \"executor\": {\n \"eval_group\": \"sft\", # Rule set for SFT data\n \"result_save\": {\n \"bad\": True # Save evaluation results\n }\n }\n}\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\nprint(result)\n```\n\n## Command Line Interface\n\n### Evaluate with Rule Sets\n\n```shell\npython -m dingo.run.cli --input test/env/local_plaintext.json\n```\n\n### Evaluate with LLM (e.g., GPT-4o)\n\n```shell\npython -m dingo.run.cli --input test/env/local_json.json\n```\n\n## GUI Visualization\n\nAfter evaluation (with `result_save.bad=True`), a frontend page will be automatically generated. To manually start the frontend:\n\n```shell\npython -m dingo.run.vsl --input output_directory\n```\n\nWhere `output_directory` contains the evaluation results with a `summary.json` file.\n\n![GUI output](docs/assets/dingo_gui.png)\n\n## Online Demo\nTry Dingo on our online demo: [(Hugging Face)🤗](https://huggingface.co/spaces/DataEval/dingo)\n\n## Local Demo\nTry Dingo in local:\n\n```shell\ncd app_gradio\npython app.py\n```\n\n![Gradio demo](docs/assets/gradio_demo.png)\n\n\n## Google Colab Demo\nExperience Dingo interactively with Google Colab notebook: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DataEval/dingo/blob/dev/examples/colab/dingo_colab_demo.ipynb)\n\n\n\n# MCP Server\n\nDingo includes an experimental Model Context Protocol (MCP) server. For details on running the server and integrating it with clients like Cursor, please see the dedicated documentation:\n\n[English](README_mcp.md) · [简体中文](README_mcp_zh-CN.md) · [日本語](README_mcp_ja.md)\n\n## Video Demonstration\n\nTo help you get started quickly with Dingo MCP, we've created a video walkthrough:\n\nhttps://github.com/user-attachments/assets/aca26f4c-3f2e-445e-9ef9-9331c4d7a37b\n\nThis video demonstrates step-by-step how to use Dingo MCP server with Cursor.\n\n\n# Data Quality Metrics\n\nDingo provides comprehensive data quality assessment through both rule-based and prompt-based evaluation metrics. These metrics cover multiple quality dimensions including effectiveness, completeness, similarity, security, and more.\n\n📊 **[View Complete Metrics Documentation →](docs/metrics.md)**\n\nOur evaluation system includes:\n- **Text Quality Assessment Metrics**: Pre-training data quality evaluation using DataMan methodology and enhanced multi-dimensional assessment\n- **SFT Data Assessment Metrics**: Honest, Helpful, Harmless evaluation for supervised fine-tuning data\n- **Classification Metrics**: Topic categorization and content classification\n- **Multimodality Assessment Metrics**: Image classification and relevance evaluation\n- **Rule-Based Quality Metrics**: Automated quality checks using heuristic rules for effectiveness and similarity detection\n- **Factuality Assessment Metrics**: Two-stage factuality evaluation based on GPT-5 System Card\n- etc\n\nMost metrics are backed by academic sources to ensure objectivity and scientific rigor.\n\n### Using LLM Assessment in Evaluation\n\nTo use these assessment prompts in your evaluations, specify them in your configuration:\n\n```python\ninput_data = {\n # Other parameters...\n \"executor\": {\n \"prompt_list\": [\"QUALITY_BAD_SIMILARITY\"], # Specific prompt to use\n },\n \"evaluator\": {\n \"llm_config\": {\n \"LLMTextQualityPromptBase\": { # LLM model to use\n \"model\": \"gpt-4o\",\n \"key\": \"YOUR_API_KEY\",\n \"api_url\": \"https://api.openai.com/v1/chat/completions\"\n }\n }\n }\n}\n```\n\nYou can customize these prompts to focus on specific quality dimensions or to adapt to particular domain requirements. When combined with appropriate LLM models, these prompts enable comprehensive evaluation of data quality across multiple dimensions.\n\n### Hallucination Detection & RAG System Evaluation\n\nFor detailed guidance on using Dingo's hallucination detection capabilities, including HHEM-2.1-Open local inference and LLM-based evaluation:\n\n📖 **[View Hallucination Detection Guide →](docs/hallucination_guide.md)**\n\n### Factuality Assessment\n\nFor comprehensive guidance on using Dingo's two-stage factuality evaluation system:\n\n📖 **[View Factuality Assessment Guide →](docs/factcheck_guide.md)**\n\n# Rule Groups\n\nDingo provides pre-configured rule groups for different types of datasets:\n\n| Group | Use Case | Example Rules |\n|-------|----------|---------------|\n| `default` | General text quality | `RuleColonEnd`, `RuleContentNull`, `RuleDocRepeat`, etc. |\n| `sft` | Fine-tuning datasets | Rules from `default` plus `RuleHallucinationHHEM` for hallucination detection |\n| `rag` | RAG system evaluation | `RuleHallucinationHHEM`, `PromptHallucination` for response consistency |\n| `hallucination` | Hallucination detection | `PromptHallucination` with LLM-based evaluation |\n| `pretrain` | Pre-training datasets | Comprehensive set of 20+ rules including `RuleAlphaWords`, `RuleCapitalWords`, etc. |\n\nTo use a specific rule group:\n\n```python\ninput_data = {\n \"executor\": {\n \"eval_group\": \"sft\", # Use \"default\", \"sft\", \"rag\", \"hallucination\", or \"pretrain\"\n }\n # other parameters...\n}\n```\n\n# Feature Highlights\n\n## Multi-source & Multi-modal Support\n\n- **Data Sources**: Local files, Hugging Face datasets, S3 storage\n- **Data Types**: Pre-training, fine-tuning, and evaluation datasets\n- **Data Modalities**: Text and image\n\n## Rule-based & Model-based Evaluation\n\n- **Built-in Rules**: 20+ general heuristic evaluation rules\n- **LLM Integration**: OpenAI, Kimi, and local models (e.g., Llama3)\n- **Hallucination Detection**: HHEM-2.1-Open local model and GPT-based evaluation\n- **RAG System Evaluation**: Response consistency and context alignment assessment\n- **Custom Rules**: Easily extend with your own rules and models\n- **Security Evaluation**: Perspective API integration\n\n## Flexible Usage\n\n- **Interfaces**: CLI and SDK options\n- **Integration**: Easy integration with other platforms\n- **Execution Engines**: Local and Spark\n\n## Comprehensive Reporting\n\n- **Quality Metrics**: 7-dimensional quality assessment\n- **Traceability**: Detailed reports for anomaly tracking\n\n# User Guide\n\n## Custom Rules, Prompts, and Models\n\nIf the built-in rules don't meet your requirements, you can create custom ones:\n\n### Custom Rule Example\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.rule.base import BaseRule\nfrom dingo.config.input_args import EvaluatorRuleArgs\nfrom dingo.io import Data\nfrom dingo.model.modelres import ModelRes\n\n@Model.rule_register('QUALITY_BAD_RELEVANCE', ['default'])\nclass MyCustomRule(BaseRule):\n \"\"\"Check for custom pattern in text\"\"\"\n\n dynamic_config = EvaluatorRuleArgs(pattern=r'your_pattern_here')\n\n @classmethod\n def eval(cls, input_data: Data) -> ModelRes:\n res = ModelRes()\n # Your rule implementation here\n return res\n```\n\n### Custom LLM Integration\n\n```python\nfrom dingo.model import Model\nfrom dingo.model.llm.base_openai import BaseOpenAI\n\n@Model.llm_register('my_custom_model')\nclass MyCustomModel(BaseOpenAI):\n # Custom implementation here\n pass\n```\n\nSee more examples in:\n- [Register Rules](examples/register/sdk_register_rule.py)\n- [Register Prompts](examples/register/sdk_register_prompt.py)\n- [Register Models](examples/register/sdk_register_llm.py)\n\n## Execution Engines\n\n### Local Execution\n\n```python\nfrom dingo.config import InputArgs\nfrom dingo.exec import Executor\n\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"local\"](input_args)\nresult = executor.execute()\n\n# Get results\nsummary = executor.get_summary() # Overall evaluation summary\nbad_data = executor.get_bad_info_list() # List of problematic data\ngood_data = executor.get_good_info_list() # List of high-quality data\n```\n\n### Spark Execution\n\n```python\nfrom dingo.config import InputArgs\nfrom dingo.exec import Executor\nfrom pyspark.sql import SparkSession\n\n# Initialize Spark\nspark = SparkSession.builder.appName(\"Dingo\").getOrCreate()\nspark_rdd = spark.sparkContext.parallelize([...]) # Your data as Data objects\n\ninput_data = {\n \"executor\": {\n \"eval_group\": \"default\",\n \"result_save\": {\"bad\": True}\n }\n}\ninput_args = InputArgs(**input_data)\nexecutor = Executor.exec_map[\"spark\"](input_args, spark_session=spark, spark_rdd=spark_rdd)\nresult = executor.execute()\n```\n\n## Evaluation Reports\n\nAfter evaluation, Dingo generates:\n\n1. **Summary Report** (`summary.json`): Overall metrics and scores\n2. **Detailed Reports**: Specific issues for each rule violation\n\nReport Description:\n1. **score**: `num_good` / `total`\n2. **type_ratio**: The count of type / total, such as: `QUALITY_BAD_COMPLETENESS` / `total`\n3. **name_ratio**: The count of name / total, such as: `QUALITY_BAD_COMPLETENESS-RuleColonEnd` / `total`\n\nExample summary:\n```json\n{\n \"task_id\": \"d6c922ec-981c-11ef-b723-7c10c9512fac\",\n \"task_name\": \"dingo\",\n \"eval_group\": \"default\",\n \"input_path\": \"test/data/test_local_jsonl.jsonl\",\n \"output_path\": \"outputs/d6c921ac-981c-11ef-b723-7c10c9512fac\",\n \"create_time\": \"20241101_144510\",\n \"score\": 50.0,\n \"num_good\": 1,\n \"num_bad\": 1,\n \"total\": 2,\n \"type_ratio\": {\n \"QUALITY_BAD_COMPLETENESS\": 0.5,\n \"QUALITY_BAD_RELEVANCE\": 0.5\n },\n \"name_ratio\": {\n \"QUALITY_BAD_COMPLETENESS-RuleColonEnd\": 0.5,\n \"QUALITY_BAD_RELEVANCE-RuleSpecialCharacter\": 0.5\n }\n}\n```\n\n# Future Plans\n\n- [ ] Richer graphic and text evaluation indicators\n- [ ] Audio and video data modality evaluation\n- [ ] Small model evaluation (fasttext, Qurating)\n- [ ] Data diversity evaluation\n\n# Limitations\n\nThe current built-in detection rules and model methods focus on common data quality problems. For specialized evaluation needs, we recommend customizing detection rules.\n\n# Acknowledgments\n\n- [RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data)\n- [mlflow](https://github.com/mlflow/mlflow)\n- [deepeval](https://github.com/confident-ai/deepeval)\n\n# Contribution\n\nWe appreciate all the contributors for their efforts to improve and enhance `Dingo`. Please refer to the [Contribution Guide](docs/en/CONTRIBUTING.md) for guidance on contributing to the project.\n\n# License\n\nThis project uses the [Apache 2.0 Open Source License](LICENSE).\n\nThis project uses fasttext for some functionality including language detection. fasttext is licensed under the MIT License, which is compatible with our Apache 2.0 license and provides flexibility for various usage scenarios.\n\n# Citation\n\nIf you find this project useful, please consider citing our tool:\n\n```\n@misc{dingo,\n title={Dingo: A Comprehensive AI Data Quality Evaluation Tool for Large Models},\n author={Dingo Contributors},\n howpublished={\\url{https://github.com/MigoXLab/dingo}},\n year={2024}\n}\n```\n","url","https://github.com/DataEval/dingo","githubRepository",{"_192":193,"_194":195,"_196":197,"_23":198,"_199":200,"_38":201,"_161":202,"_204":205,"_206":207,"_209":210},"defaultBranch","main","fullName","MigoXLab/dingo","githubTreeId",1694965,12494,"language","JavaScript","dingo",{"_165":166,"_8":203},"ddnfvr2id8","recentStargazers",7,"spdxLicense",{"_38":208},"Apache 2.0","stargazers",498,"glamaJson","inspectable",false,"npmPackage","supportedPlatforms","scores",{"_218":219,"_220":-5,"_221":-5},"license",100,"quality","security","simpleIcon","toolCount",0,"ryqs541su0","updatedAt","2025-10-06T13:45:44.391975Z","meta",[230,232,234,241,244,246,249,252,255,258,261,263,266,269,271,273,275,277,279],{"_10":231},"Dingo MCP Server | Glama",{"_233":37,"_38":36},"content",{"_235":236,"_237":238,"_239":240},"href","https://glama.ai/mcp/servers/@MigoXLab/dingo","rel","canonical","tagName","link",{"_233":37,"_242":243},"property","og:title",{"_233":37,"_242":245},"og:description",{"_233":247,"_242":248},"https://glama.ai/logo.png","og:logo",{"_233":250,"_242":251},"https://glama.ai/mcp/servers/@MigoXLab/dingo/og-image","og:image",{"_233":253,"_242":254},"630","og:image:height",{"_233":256,"_242":257},"1200","og:image:width",{"_233":259,"_242":260},"website","og:type",{"_233":236,"_242":262},"og:url",{"_233":264,"_242":265},"Glama – MCP Hosting Platform","og:site_name",{"_233":267,"_38":268},"summary_large_image","twitter:card",{"_233":264,"_38":270},"twitter:title",{"_233":37,"_38":272},"twitter:description",{"_233":250,"_38":274},"twitter:image",{"_233":253,"_38":276},"twitter:image:height",{"_233":256,"_38":278},"twitter:image:width",{"_280":281},"script:ld+json",{"_282":283,"_284":285},"@context","http://schema.org","@graph",[286],{"_282":287,"_288":289,"_290":291},"https://schema.org","@type","BreadcrumbList","itemListElement",[292,299,303,307],{"_288":293,"_294":295,"_38":296,"_297":298},"ListItem","item","/","Glama","position",1,{"_288":293,"_294":300,"_38":301,"_297":302},"/mcp","MCP",2,{"_288":293,"_294":304,"_38":305,"_297":306},"/mcp/servers","Servers",3,{"_288":293,"_294":308,"_38":37,"_297":309},"/mcp/servers/@MigoXLab/dingo",4,"visitorIsMaintainer","routes/_public/mcp/servers/~namespace/~slug/_pages/_explorer/blob/~commit/~/_route",{"_313":314,"_34":320,"_355":356,"_228":370,"_367":394,"_395":396},"blob",{"_233":315,"_316":317,"_318":319},"\n","fileName","logo.svg","size",5106,{"_82":83,"_20":321,"_89":37,"_90":37,"_23":91,"_92":322,"_38":37,"_159":341,"_167":-5,"_168":-5,"_169":344,"_216":354,"_222":-5,"_100":201,"_223":224,"_8":225,"_226":227},[85,86,87,88],[323,325,327,329,331,333,335,337,339],{"_95":324,"_36":102,"_8":103},{"_97":98,"_38":99,"_100":101},{"_95":326,"_36":109,"_8":110},{"_97":106,"_38":107,"_100":108},{"_95":328,"_36":116,"_8":117},{"_97":113,"_38":114,"_100":115},{"_95":330,"_36":123,"_8":124},{"_97":120,"_38":121,"_100":122},{"_95":332,"_36":130,"_8":131},{"_97":127,"_38":128,"_100":129},{"_95":334,"_36":137,"_8":138},{"_97":134,"_38":135,"_100":136},{"_95":336,"_36":144,"_8":145},{"_97":141,"_38":142,"_100":143},{"_95":338,"_36":150,"_8":151},{"_97":148,"_38":149,"_100":149},{"_95":340,"_36":157,"_8":158},{"_97":154,"_38":155,"_100":156},{"_161":342,"_100":166},{"_163":343},{"_165":166},{"_171":345,"_185":350,"_190":351,"_211":-5,"_212":213,"_214":-5,"_215":-5},{"_173":346,"_182":349,"_12":184},{"_175":347,"_179":348},{"_36":177,"_12":178},{"_36":181,"_12":178},[175],{"_39":187,"_188":189},{"_192":193,"_194":195,"_196":197,"_23":198,"_199":200,"_38":201,"_161":352,"_204":205,"_206":353,"_209":210},{"_165":166,"_8":203},{"_38":208},{"_218":219,"_220":-5,"_221":-5},"member",{"_357":358,"_361":362,"_363":364,"_196":365,"_366":213,"_367":317,"_368":369,"_318":319,"_12":313},"commit",{"_359":360},"sha","a6a2592b266c418ce54a20aa526893b2fb749cbd","contentStoredObjectPointerId",10292866,"fullPath","app/resources/logo.svg",1694870,"indexable","path","safeToDisplayAsText",true,[371,373,375,377,379,380,381,383,384,385,386,387,388,389,390,391,392,393],{"_10":372},"logo.svg | Dingo MCP Server | Glama",{"_233":374,"_38":36},"Enterprise-grade security, privacy, and features like agents, MCP, prompt templates, and more.",{"_235":376,"_237":238,"_239":240},"https://glama.ai/mcp/servers/@MigoXLab/dingo/blob/a6a2592b266c418ce54a20aa526893b2fb749cbd/app/resources",{"_233":378,"_242":243},"logo.svg | Dingo MCP Server",{"_233":374,"_242":245},{"_233":247,"_242":248},{"_233":382,"_242":251},"https://glama.ai/generated-images/og?title=logo.svg+%7C+Dingo+MCP+Server",{"_233":253,"_242":254},{"_233":256,"_242":257},{"_233":259,"_242":260},{"_233":376,"_242":262},{"_233":264,"_242":265},{"_233":267,"_38":268},{"_233":264,"_38":270},{"_233":374,"_38":272},{"_233":382,"_38":274},{"_233":253,"_38":276},{"_233":256,"_38":278},"app/resources","treeMembers",[397,401],{"_357":398,"_363":399,"_367":400,"_12":313},{"_359":360},"app/resources/icon.png","icon.png",{"_357":402,"_363":364,"_367":317,"_12":313},{"_359":360},"actionData","errors"]

MCP directory API