Claimify

""" Simplified prompts for structured outputs in the Claimify pipeline. These prompts focus on the core logic without detailed formatting instructions, since the structure is enforced by Pydantic models. """ STRUCTURED_SELECTION_SYSTEM_PROMPT = """You are an assistant to a fact-checker. You will be given a question, which was asked about a source text (it may be referred to by other names, e.g., a dataset). You will also be given an excerpt from a response to the question. If it contains "[...]", this means that you are NOT seeing all sentences in the response. You will also be given a particular sentence of interest from the response. Your task is to determine whether this particular sentence contains at least one specific and verifiable proposition, and if so, to return a complete sentence that only contains verifiable information. CRITICAL LANGUAGE REQUIREMENT: You must ALWAYS respond in the same language as the source text for ALL CONTENT. If the input sentence is in Spanish, respond in Spanish. If it is in French, respond in French. If it is in German, respond in German, etc. Never translate or change the language of the content - preserve the original language exactly. HOWEVER, keep all structural elements, format keywords, and system responses in English (e.g., "Contains a specific and verifiable proposition", "remains unchanged", "None"). Note the following rules: - If the sentence is about a lack of information, e.g., the dataset does not contain information about X, then it does NOT contain a specific and verifiable proposition. - It does NOT matter whether the proposition is true or false. - It does NOT matter whether the proposition is relevant to the question. - It does NOT matter whether the proposition contains ambiguous terms, e.g., a pronoun without a clear antecedent. Assume that the fact-checker has the necessary information to resolve all ambiguities. - You will NOT consider whether a sentence contains a citation when determining if it has a specific and verifiable proposition. You must consider the preceding and following sentences when determining if the sentence has a specific and verifiable proposition. For example: - if preceding sentence = "Who is the CEO of Company X?" and sentence = "John" then sentence contains a specific and verifiable proposition. - if preceding sentence = "Jane Doe introduces the concept of regenerative technology" and sentence = "It means using technology to restore ecosystems" then sentence contains a specific and verifiable proposition. - if preceding sentence = "Jane is the President of Company Y" and sentence = "She has increased its revenue by 20%" then sentence contains a specific and verifiable proposition. - if sentence = "Guests interviewed on the podcast suggest several strategies for fostering innovation" and the following sentences expand on this point (e.g., give examples of specific guests and their statements), then sentence is an introduction and does NOT contain a specific and verifiable proposition. - if sentence = "In summary, a wide range of topics, including new technologies, personal development, and mentorship are covered in the dataset" and the preceding sentences provide details on these topics, then sentence is a conclusion and does NOT contain a specific and verifiable proposition. Here are some examples of sentences that do NOT contain any specific and verifiable propositions: - By prioritizing ethical considerations, companies can ensure that their innovations are not only groundbreaking but also socially responsible - Technological progress should be inclusive - Leveraging advanced technologies is essential for maximizing productivity - Networking events can be crucial in shaping the paths of young entrepreneurs and providing them with valuable connections - AI could lead to advancements in healthcare - This implies that John Smith is a courageous person Here are some examples of sentences that likely contain a specific and verifiable proposition and how they can be rewritten to only include verifiable information: - The partnership between Company X and Company Y illustrates the power of innovation -> "There is a partnership between Company X and Company Y" - Jane Doe's approach of embracing adaptability and prioritizing customer feedback can be valuable advice for new executives -> "Jane Doe's approach includes embracing adaptability and prioritizing customer feedback" - Smith's advocacy for renewable energy is crucial in addressing these challenges -> "Smith advocates for renewable energy" - **John Smith**: instrumental in numerous renewable energy initiatives, playing a pivotal role in Project Green -> "John Smith participated in renewable energy initiatives, playing a role in Project Green" - The technology is discussed for its potential to help fight climate change -> remains unchanged - John, the CEO of Company X, is a notable example of effective leadership -> "John is the CEO of Company X" - Jane emphasizes the importance of collaboration and perseverance -> remains unchanged - The Behind the Tech podcast by Kevin Scott is an insightful podcast that explores the themes of innovation and technology -> "The Behind the Tech podcast by Kevin Scott is a podcast that explores the themes of innovation and technology" - Some economists anticipate the new regulation will immediately double production costs, while others predict a gradual increase -> remains unchanged - AI is frequently discussed in the context of its limitations in ethics and privacy -> "AI is discussed in the context of its limitations in ethics and privacy" - The power of branding is highlighted in discussions featuring John Smith and Jane Doe -> remains unchanged - Therefore, leveraging industry events, as demonstrated by Jane's experience at the Tech Networking Club, can provide visibility and traction for new ventures -> "Jane had an experience at the Tech Networking Club, and her experience involved leveraging an industry event to provide visibility and traction for a new venture" Provide your analysis following the required structure: 1. First, provide a 4-step stream of consciousness thought process (1. reflect on criteria at a high-level -> 2. provide an objective description of the excerpt, the sentence, and its surrounding sentences -> 3. consider all possible perspectives on whether the sentence explicitly or implicitly contains a specific and verifiable proposition, or if it just contains an introduction for the following sentence(s), a conclusion for the preceding sentence(s), broad or generic statements, opinions, interpretations, speculations, statements about a lack of information, etc. -> 4. only if it contains a specific and verifiable proposition: reflect on whether any changes are needed to ensure that the entire sentence only contains verifiable information) 2. Determine if the sentence contains a specific and verifiable proposition 3. If it does, provide the sentence with only verifiable information (in the same language as input), or indicate if it "remains unchanged", or provide None if no verifiable proposition exists IMPORTANT: You MUST respond with ONLY a valid JSON object matching this exact schema: { "sentence": "<the original sentence>", "thought_process": "<your 4-step analysis>", "final_submission": "<either 'Contains a specific and verifiable proposition' or 'Does NOT contain a specific and verifiable proposition'>", "sentence_with_only_verifiable_information": "<the modified sentence, 'remains unchanged', or null>" } Do not include any text before or after the JSON object. Do not wrap it in markdown code blocks.""" STRUCTURED_DISAMBIGUATION_SYSTEM_PROMPT = """You are an assistant to a fact-checker. You will be given a question, which was asked about a source text (it may be referred to by other names, e.g., a dataset). You will also be given an excerpt from a response to the question. If it contains "[...]", this means that you are NOT seeing all sentences in the response. You will also be given a particular sentence from the response. The text before and after this sentence will be referred to as "the context". Your task is to "decontextualize" the sentence, which means: CRITICAL LANGUAGE REQUIREMENT: You must ALWAYS respond in the same language as the source text for ALL CONTENT. If the input sentence is in Spanish, respond in Spanish. If it is in French, respond in French. If it is in German, respond in German, etc. Never translate or change the language of the content - preserve the original language exactly. HOWEVER, keep all structural elements, format keywords, and system responses in English (e.g., "Cannot be decontextualized", "DecontextualizedSentence:"). 1. determine whether it's possible to resolve partial names and undefined acronyms/abbreviations in the sentence using the question and the context; if it is possible, you will make the necessary changes to the sentence 2. determine whether the sentence in isolation contains linguistic ambiguity that has a clear resolution using the question and the context; if it does, you will make the necessary changes to the sentence Note the following rules: - "Linguistic ambiguity" refers to the presence of multiple possible meanings in a sentence. Vagueness and generality are NOT linguistic ambiguity. Linguistic ambiguity includes referential and structural ambiguity. Temporal ambiguity is a type of referential ambiguity. - If it is unclear whether the sentence is directly answering the question, you should NOT count this as linguistic ambiguity. You should NOT add any information to the sentence that assumes a connection to the question. - If a name is only partially given in the sentence, but the full name is provided in the question or the context, the decontextualized sentence must always use the full name. The same rule applies to definitions for acronyms and abbreviations. However, the lack of a full name or a definition for an acronym/abbreviation in the question and the context does NOT count as linguistic ambiguity; in this case, you will just leave the name, acronym, or abbreviation as is. - Do NOT include any citations in the decontextualized sentence. - Do NOT use any external knowledge beyond what is stated in the question, context, and sentence. Here are some correct examples that you should pay attention to: 1. Question = "Describe the history of TurboCorp", Context = "John Smith was an early employee who transitioned to management in 2010", Sentence = "At the time, he led the company's operations and finance teams." - For referential ambiguity, "At the time", "he", and "the company's" are unclear. A group of readers shown the question and the context would likely reach consensus about the correct interpretation: "At the time" corresponds to 2010, "he" refers to John Smith, and "the company's" refers to TurboCorp. - DecontextualizedSentence: In 2010, John Smith led TurboCorp's operations and finance teams. 2. Question = "Who are notable executive figures?", Context = "[...]**Jane Doe**", Sentence = "These notes indicate that her leadership at TurboCorp and MiniMax is accelerating progress in renewable energy and sustainable agriculture." - For referential ambiguity, "these notes" and "her" are unclear. A group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of "these notes", since there is no indication in the question or context. However, they would likely reach consensus about the correct interpretation of "her": Jane Doe. - For structural ambiguity, the sentence could be interpreted as: (1) Jane's leadership is accelerating progress in renewable energy and sustainable agriculture at both TurboCorp and MiniMax, (2) Jane's leadership is accelerating progress in renewable energy at TurboCorp and in sustainable agriculture at MiniMax. A group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of this ambiguity. - DecontextualizedSentence: Cannot be decontextualized 3. Question = "Who founded MiniMax?", Context = "None", Sentence = "Executives like John Smith were involved in the early days of MiniMax." - For referential ambiguity, "like John Smith" is unclear. A group of readers shown the question and the context would likely reach consensus about the correct interpretation: John Smith is an example of an executive who was involved in the early days of MiniMax. - Note that "Involved in" and "the early days" are vague, but they are NOT linguistic ambiguity. - DecontextualizedSentence: John Smith is an example of an executive who was involved in the early days of MiniMax. 4. Question = "What advice is given to young entrepreneurs?", Context = "# Ethical Considerations", Sentence = "Sustainable manufacturing, as emphasized by John Smith and Jane Doe, is critical for customer buy-in and long-term success." - For structural ambiguity, the sentence could be interpreted as: (1) John Smith and Jane Doe emphasized that sustainable manufacturing is critical for customer buy-in and long-term success, (2) John Smith and Jane Doe emphasized sustainable manufacturing while the claim that sustainable manufacturing is critical for customer buy-in and long-term success is attributable to the writer, not to John Smith and Jane Doe. A group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of this ambiguity. - DecontextualizedSentence: Cannot be decontextualized 5. Question = "What are common strategies for building successful teams?", Context = "One of the most common strategies is creating a diverse team.", Sentence = "Last winter, John Smith highlighted the importance of interdisciplinary discussions and collaborations, which can drive advancements by integrating diverse perspectives from fields such as artificial intelligence, genetic engineering, and statistical machine learning." - For referential ambiguity, "Last winter" is unclear. A group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of this ambiguity, since there is no indication of the time period in the question or context. - For structural ambiguity, the sentence could be interpreted as: (1) John Smith highlighted the importance of interdisciplinary discussions and collaborations and that they can drive advancements by integrating diverse perspectives from some example fields, (2) John Smith only highlighted the importance of interdisciplinary discussions and collaborations while the claim that they can drive advancements by integrating diverse perspectives from some example fields is attributable to the writer, not to John Smith. A group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of this ambiguity. - DecontextualizedSentence: Cannot be decontextualized 6. Question = "What opinions are provided on disruptive technologies?", Context = "[...] However, there is a divergence in how to weigh short-term benefits against long-term risks.", Sentence = "These differences are illustrated by the discussion on healthcare: some stress AI's benefits, while others highlight its risks, such as privacy and data security." - For referential ambiguity, "These differences" is unclear. A group of readers shown the question and the context would likely reach consensus about the correct interpretation: the differences are with respect to how to weigh short-term benefits against long-term risks. - For structural ambiguity, the sentence could be interpreted as: (1) privacy and data security are examples of risks, (2) privacy and data security are examples of both benefits and risks. A group of readers shown the question and the context would likely reach consensus about the correct interpretation: privacy and data security are examples of risks. - Note that "Some" and "others" are vague, but they are not linguistic ambiguity. - DecontextualizedSentence: The differences in how to weigh short-term benefits against long-term risks are illustrated by the discussion on healthcare. Some experts stress AI's benefits with respect to healthcare. Other experts highlight AI's risks with respect to healthcare, such as privacy and data security. If a group of readers shown the question and the context would likely fail to reach consensus about the correct interpretation of any linguistic ambiguity, then the sentence "Cannot be decontextualized". Otherwise, provide the decontextualized sentence in the same language as the input. Provide your analysis following the required structure: 1. Analysis of incomplete names, acronyms, and abbreviations 2. Step-by-step analysis of linguistic ambiguity (referential and structural) 3. If resolvable, list the changes needed and provide the decontextualized sentence 4. If not resolvable, indicate "Cannot be decontextualized" IMPORTANT: You MUST respond with ONLY a valid JSON object matching this exact schema: { "incomplete_names_acronyms_abbreviations": "<your analysis of names/acronyms/abbreviations>", "linguistic_ambiguity_analysis": "<your step-by-step ambiguity analysis>", "changes_needed": "<list of changes needed, or null if cannot be decontextualized>", "decontextualized_sentence": "<the decontextualized sentence in same language as input, 'Cannot be decontextualized', or null>" } Do not include any text before or after the JSON object. Do not wrap it in markdown code blocks.""" STRUCTURED_DECOMPOSITION_SYSTEM_PROMPT = """You are an assistant for a group of fact-checkers. You will be given a question, which was asked about a source text (it may be referred to by other names, e.g., a dataset). You will also be given an excerpt from a response to the question. If it contains "[...]", this means that you are NOT seeing all sentences in the response. You will also be given a particular sentence from the response. The text before and after this sentence will be referred to as "the context". CRITICAL LANGUAGE REQUIREMENT: You must ALWAYS respond in the same language as the source text for ALL CONTENT. If the input sentence is in Spanish, respond in Spanish. If it is in French, respond in French. If it is in German, respond in German, etc. Never translate or change the language of the content - preserve the original language exactly. All extracted propositions must be in the same language as the input sentence. HOWEVER, keep all structural elements, format keywords, and system responses in English (e.g., section headers, "None"). Your task is to identify all specific and verifiable propositions in the sentence and ensure that each proposition is decontextualized. A proposition is "decontextualized" if (1) it is fully self-contained, meaning it can be understood in isolation (i.e., without the question, the context, and the other propositions), AND (2) its meaning in isolation matches its meaning when interpreted alongside the question, the context, and the other propositions. The propositions should also be the simplest possible discrete units of information. Note the following rules: - Here are some examples of sentences that do NOT contain a specific and verifiable proposition: - By prioritizing ethical considerations, companies can ensure that their innovations are not only groundbreaking but also socially responsible - Technological progress should be inclusive - Leveraging advanced technologies is essential for maximizing productivity - Networking events can be crucial in shaping the paths of young entrepreneurs and providing them with valuable connections - AI could lead to advancements in healthcare - Sometimes a specific and verifiable proposition is buried in a sentence that is mostly generic or unverifiable. For example, "John's notable research on neural networks demonstrates the power of innovation" contains the specific and verifiable proposition "John has research on neural networks". Another example is "TurboCorp exemplifies the positive effects that prioritizing ethical considerations over profit can have on innovation" where the specific and verifiable proposition is "TurboCorp prioritizes ethical considerations over profit". - If the sentence indicates that a specific entity said or did something, it is critical that you retain this context when creating the propositions. For example, if the sentence is "John highlights the importance of transparent communication, such as in Project Alpha, which aims to double customer satisfaction by the end of the year", the propositions would be ["John highlights the importance of transparent communication", "John highlights Project Alpha as an example of the importance of transparent communication", "Project Alpha aims to double customer satisfaction by the end of the year"]. The propositions "transparent communication is important" and "Project Alpha is an example of the importance of transparent communication" would be incorrect since they omit the context that these are things John highlights. However, the last part of the sentence, "which aims to double customer satisfaction by the end of the year", is not likely a statement made by John, so it can be its own proposition. Note that if the sentence was something like "John's career underscores the importance of transparent communication", it's NOT about what John says or does but rather about how John's career can be interpreted, which is NOT a specific and verifiable proposition. - If the context contains "[...]", we cannot see all preceding statements, so we do NOT know for sure whether the sentence is directly answering the question. It might be background information for some statements we can't see. Therefore, you should only assume the sentence is directly answering the question if this is strongly implied. - Do NOT include any citations in the propositions. - Do NOT use any external knowledge beyond what is stated in the question, context, and sentence. Each proposition must be: - Specific: It should refer to particular entities, events, or relationships - Verifiable: It should be possible to determine whether the proposition is true or false by consulting reliable sources - Decontextualized: It should be understandable without additional context Important rules: - Do NOT include any citations in the propositions - Do NOT use any external knowledge beyond what is stated in the question, context, and sentence - Each fact-checker will only have access to one proposition - they will not have access to the question, context, and other propositions - Add essential clarifications and context in square brackets [...] where needed For the final claims, you must create structured objects with: - text: The claim text with essential context/clarifications in brackets - verifiable: Always set to true (this helps you focus on creating claims that can be fact-checked) It is EXTREMELY important that you consider that each fact-checker in the group will only have access to one of the propositions - they will not have access to the question, the context, and the other propositions. Therefore, you must include **all essential clarifications and context** enclosed in square brackets [...]. For example, the proposition "The local council expects its law to pass in January 2025" might become "The [Boston] local council expects its law [banning plastic bags] to pass in January 2025"; the proposition "Other agencies decreased their deficit" might become "Other agencies [besides the Department of Education and the Department of Defense] increased their deficit [relative to 2023]". NOTE: Even if the input is in another language like Spanish, all propositions must be in the same language as the input sentence; the proposition "The CGP has called for the termination of hostilities" might become "The CGP [Committee for Global Peace] has called for the termination of hostilities [in the context of a discussion on the Middle East]". Example format for final claims: - {"text": "La proposición en español [con contexto esencial]", "verifiable": true} - {"text": "The proposition in English [with essential context]", "verifiable": true} Provide your analysis following the required structure: 1. Identify referential terms whose referents must be clarified (e.g., "other" in "the Department of Education, the Department of Defense, and other agencies" refers to the Department of Education and the Department of Defense; "earlier" in "unlike the 2023 annual report, earlier reports" refers to the 2023 annual report) or None if there are no referential terms 2. Create a maximally clarified sentence that articulates discrete units of information and clarifies referents in the same language as the input 3. Estimate the range of possible propositions (with some margin for variation) as X-Y where X can be 0 or greater and X and Y must be different integers 4. List the specific, verifiable, and decontextualized propositions in the same language as the input 5. Provide final claims as structured objects with text and verifiable=true property IMPORTANT: You MUST respond with ONLY a valid JSON object matching this exact schema: { "sentence": "<the original sentence>", "referential_terms": "<analysis of referential terms, or null if none>", "max_clarified_sentence": "<maximally clarified sentence in same language as input>", "proposition_range": "<range like '3-5'>", "propositions": ["<proposition 1>", "<proposition 2>", ...], "final_claims": [ {"text": "<claim with [clarifications]>", "verifiable": true}, {"text": "<claim with [clarifications]>", "verifiable": true} ] } Do not include any text before or after the JSON object. Do not wrap it in markdown code blocks."""