Grade against the question: stop requiring details the question never asked for
A short-answer case: "Why was Kryštof Harant executed?" with the student answering "because he took part in the anti-Habsburg uprising" was failed for not naming the year 1621 — a detail only present in the reference answer, never asked by the question. The generator had put the date into keyPoints, and the grader treated every key point as mandatory. - Grading prompt: the QUESTION defines what is required. A key point counts only when the question asks for it or the answer is wrong or incomplete without it; extra detail in the reference answer (dates, places, names, numbers) is a bonus — never listed as missing, never a reason to fail. An accepted answer must return an empty "missing" array. - Generation spec: short_answer keyPoints must contain only the ideas the question asks for (a "why" question requires the reason, not the year). - gradeShortAnswer drops any stray "missing" entries when correct is true, so the UI can never show a gap on an accepted answer. Verified: mock-Ollama integration test 9/9 through the real /api/grade route (new rules sent upstream, incorrect answers keep their missing point, accepted answers normalise to missing: []), regression 74/74, mobile 51/51, i18n 27/27. Lint + build clean. SW freezmo-v18.
This commit is contained in:
parent
829a71f4ff
commit
6fb11fa405
3 changed files with 10 additions and 5 deletions
|
|
@ -396,7 +396,9 @@ export async function gradeShortAnswer(input: {
|
|||
signal: input.signal,
|
||||
temperature: 0.1,
|
||||
});
|
||||
return value;
|
||||
// An accepted answer never has "missing" points — anything the question did
|
||||
// not ask about is a bonus, and the UI must not show it as a gap.
|
||||
return value.correct ? { ...value, missing: [] } : value;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
|
|
|||
|
|
@ -90,7 +90,8 @@ export const ACTIVITY_JSON_SPEC = `Each activity is one JSON object with a "type
|
|||
{ "type": "ordering", "topic": "short label", "instruction": "Put the events in the order they happened.", "items": ["The Allies land in Normandy", "Paris is liberated", "Germany surrenders"], "explanation": "..." }
|
||||
|
||||
7. short_answer - typed answer, graded by AI
|
||||
{ "type": "short_answer", "topic": "short label", "question": "...", "referenceAnswer": "a model answer in one or two sentences", "keyPoints": ["idea that must appear", "second idea"], "explanation": "..." }
|
||||
{ "type": "short_answer", "topic": "short label", "question": "...", "referenceAnswer": "a model answer in one or two sentences", "keyPoints": ["idea the question asks for", "second idea"], "explanation": "..." }
|
||||
keyPoints must contain only the ideas needed to answer the question — never extras the question does not ask about (dates, places, names, numbers). For a "why" question require the reason, not the year.
|
||||
|
||||
Every activity must also have "topic" (2-4 words) and "explanation" (one or two sentences, plain language).`;
|
||||
|
||||
|
|
@ -165,11 +166,13 @@ export function buildGenerationUserPrompt({
|
|||
/* Short-answer grading */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
export const GRADING_SYSTEM_PROMPT = `You are a fair but strict study grader. You compare a student's typed answer with a reference answer and the required key points.
|
||||
export const GRADING_SYSTEM_PROMPT = `You are a fair but strict study grader. You compare a student's typed answer with the question, a reference answer and the key points.
|
||||
|
||||
Rules:
|
||||
- Grade on meaning, not wording. Synonyms, different but correct order, and minor spelling mistakes are fine.
|
||||
- A correct answer must cover all key points. Partial answers are incorrect (correct: false) but should be encouraged in the feedback.
|
||||
- The QUESTION defines what the student must answer. Only require a key point when the question actually asks for it, or the answer would be wrong or incomplete without it. Extra detail in the reference answer that the question does not ask about (a date, place, name or number) is a bonus — never list it as missing and never fail the answer for omitting it.
|
||||
- A correct answer must cover the key points the question actually asks for. Partial answers are incorrect (correct: false) but should be encouraged in the feedback.
|
||||
- If the answer is correct, "missing" must be an empty array.
|
||||
- LANGUAGE: reply in the same language as the QUESTION and the STUDENT ANSWER. Look at the actual words of the question before writing anything. If the question is in English, your feedback is in English; if it is in Czech, your feedback is in Czech; and so on. If a LANGUAGE instruction is given below, it always wins. Never answer in a language that does not appear in the question or the answer.
|
||||
- Never reveal that you are an AI. Never mention JSON.
|
||||
- Feedback: one or two sentences, max 300 characters. If something is missing, name it plainly, in that language.
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
*
|
||||
* Bumping CACHE_VERSION is how a new deploy invalidates old assets.
|
||||
*/
|
||||
const CACHE_VERSION = "freezmo-v17";
|
||||
const CACHE_VERSION = "freezmo-v18";
|
||||
const SHELL_CACHE = `${CACHE_VERSION}-shell`;
|
||||
const ASSET_CACHE = `${CACHE_VERSION}-assets`;
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue