MCP Terminal Server
by dillip285
- js
- plugins
- evaluators
- prompts
---
input:
schema:
question: string
answer: string
context: string
---
Assess whether the generated output is relevant to the question asked.
To accomplish this perform the following 3 tasks in a step by step manner:
1. Identify if the question is noncommittal. A noncommittal answer is one that is evasive, vague, or ambiguous. For example, "I don't know", "I'm not sure", and "I can't answer" are noncommittal answers. Give a score of 1 if the answer is noncommittal and 0 if it is committal.
2. Assess whether the answer provided addresses the question posed. If the answer is similar in subject matter but doesn't answer the question posed, that is not satisfactory. Give a score of 1 for a satisfactory answer and 0 if it is not satisfactory.
3. Generate a question that could produce the provided answer. Use only the information in the provided answer.
Format the answer as json in the following manner where task 1 is assigned to the "noncommittal" field, task 2 is assigned to the "answered" field, and task 3 is assigned to the "question" field.
Here are some examples:
Question:
In what country was Albert Einstein born?
Context:
Albert Einstein was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time
Answer:
Albert Einstein was born in Germany.
Output:
{"noncommittal":0, "answered": 1, "question":"Where was Albert Einstein born?"}
Question:
Are there any frogs that can change their skin color like chameleons?
Context:
A recent scientific study has discovered a new species of frog in the Amazon rainforest that has the unique ability to change its skin color based on the temperature of its environment.
Answer:
It can change its skin color based on the temperature of its environment.
Output:
{"noncommittal":0, "answered":0, "question":"What unique ability does the newly discovered species of frog have?"}
Question:
What is the tallest mountain?
Context:
The tallest mountain on Earth, measured from sea level, is a renowned peak located in the Himalayas.
Answer:
Everest
Output:
{"noncommittal":0, "answered":1, "question":"What is the tallest mountain on Earth?"}
Question:
Where there any groundbreaking new features announced for new smartphones in 2023?
Answer:
I don't know about the groundbreaking feature of the smartphone invented in 2023 as am unware of information beyong 2022.
Context:
In 2023, a groundbreaking invention was announced: a smartphone with a battery life of one month, revolutionizing the way people use mobile technology.
Output:
{"noncommittal":1, "answered":0, "question":"What was the groundbreaking feature of the smartphone invented in 2023?"}
Now provide your analysis for the following inputs. DO NOT PROVIDE ANY MORE EXAMPLES. Your response must be a valid JSON like you see above.
Question:
{{question}}
Answer:
{{answer}}
Context:
{{context}}
Output: