AI-Assisted Grading and Feedback in Higher Education using GPT

AI-Assisted Grading and Feedback in Higher Education using GPT Leonardo Franco Cruz1Miguel Leitão Bignolas Mira da Silva1, José Henrique Pereira São Mamede2 1Instituto Superior Técnico - Universidade de Lisboa, Lisboa, Portugal 2Universidade Aberta, Lisboa, Portugal [email protected] [email protected] [email protected] ENSINAR FORA DA CAIXA, XI Ciclo de Conferências da Faculdade de Ciências, 3-4 Oct 2025, Universidade da Beira Interior Abstract We test a rubric-driven, GDPR-safe pipeline using large language models (LLMs) to deliver fast, consistent, and auditable grading/feedback for open-ended case studies at cohort scale. A systematic literature review frames the space, and a design science research methodology steers a live deployment built with privacy by design (pseudonymised prompts; no PII), strict 28-field output contracts (+1 advisory AI-text estimate), and automatic validation/repair. Across six assignments in a master’s course (60 students; 362 submissions), GPT mirrored the instructor with small, stable, and correctable errors on a 0–5 scale (MAD 0.24–0.36; mean 0.30) and a +0.15 level bias (Professor - GPT), showing a near-unit trend with tail compression and preserved rank. The system achieved next-day turnaround (typically ≈ 1 h compute), reduced manual effort, and stayed within budget (EUR 204 total; ≈ EUR 0.56 per student–assignment). Non-punitive disclosure of estimated AI-generated text coincided with a drop from 46% to a mid-term 10% plateau; a late rebound to 23% did not reduce alignment. Research Questions and Setting RQ1–RQ2 Technologies and techniques for dependable AI in assessment. RQ3–RQ5 Applications, benefits, and key technical limitations/challenges. RQ6–RQ7 Pedagogical implications and ethical/GDPR governance for safe deployment. Setting & Data: Six case-study assignments; four open questions per case; rubric C1–C6 (0–5 each). Cohort: 60 students; 362 submissions. Per-submission outputs: 29 fields — the core 28-field contract (AnonID; 24 rubric scores; strengths; areas to improve; points to avoid) + 1 advisory field (estimated % AI-generated text). Course context: live deployment in the Information Systems Project Management course at Instituto Superior Técnico (IST), Universidade de Lisboa (AY 2024/25). Pipeline (GDPR-Safe, Instructor-in-the-Loop) Ingest & Clean Pseudonymise (AnonID) GPT Grades & Feedback Validate/ Repair ZeroGPT (advisory) Merge & Averages Instructor Console Email Students Guardrails: strict contract, privacy, logs; advisory AI detection Methods SLR (Kitchenham): scoping of technologies, applications, pedagogy, and ethics. DSRM (Peffers): iterative design with evaluation in authentic use; instructor-in-the-loop for outlier review. Pre-registered metrics & analysis: mean absolute deviation (MAD), level bias (Prof–GPT), distributional overlap, pooled OLS parity (slope/intercept), Spearman’s ρ; operational metrics (latency, cost). Results — Cohort Means Across Cases (Professor vs. GPT) Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 3.8 4 4.2 4.4 4.6 4.04 4.31 4.49 4.32 4.11 4.14 3.95 4.14 4.21 4.1 4.02 4.07 Average grade (0–5) Professor GPT Parallel movement (trend preserved) with a stable level gap of ≈0.15 points, which is correctable by calibration. Results — Absolute Error Bands (MAD per Case) Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 0 0.1 0.2 0.3 0.4 0.5 0.29 0.3 0.36 0.35 0.24 0.28 MAD (|Prof - GPT|) Small, stable error bands (0–5 scale); narrowest when the cohort compresses (Case 5). Behaviour — Trend of Estimated AI-Generated Text Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 0 5 10 15 20 25 30 35 40 45 50 46.18 15.92 10.93 8.62 4.94 22.99 Case Detected AI text (%) Transparency coincides with a reduction to a 10% plateau (Cases 2–5). Between Cases 5 and 6 we held in-person student interviews/briefings explicitly reaffirming the non-punitive policy. Problem and Objective Growing cohorts and limited staff threaten grading quality, timeliness and fairness. We target a rubric-conditioned, GDPR-safe pipeline that produces cohort-scale grades and actionable feedback within hours while remaining auditable, correctable and equitable. Discussion and Implications Predictable alignment: Small, stable error (MAD ≈0.30) with a modest +0.15 bias, consistent across cases. First-pass role: Use LLMs as first-pass graders; direct human review to outliers and the top tail. Operational viability: Hour-scale turnaround and sub-euro marginal cost without losing auditability. Pedagogy: Rubric-anchored feedback increases transparency; non-punitive AI-use disclosure supports healthier behaviour. Key Numbers (Live Study) N= 60 students; 362 submissions (6 cases) MAD = 0.24-0.36 (mean 0.30) Bias = +0.15 (Professor minus GPT) Turnaround ≈1h (cap <24h) Cost EUR 204 total (≈EUR 0.56 per student-assignment; breakdown: EUR 198 model usage, EUR 6 advisory detection) Student Feedback — Google Forms Usefulness Accuracy AI-use fairness 0 20 40 60 80 100 8.03 4.46 3.58 12.50 18.75 79.47 76.79 96.42 Positive Neutral Negative Categories collapsed as: Positive = Very useful + Useful; Mostly + Completely accurate; Accurate + Somewhat accurate. Neutral = Neutral / Somewhat accurate. Negative = Not very useful / Not really accurate / Misleading. Contributions A deployable, GDPR-safe pipeline for rubric-conditioned grading with 28-field output contracts (+1 advisory AI-text estimate) and automatic validation/repair. Cohort-scale evidence that alignment is stable and correctable (MAD ≈ 0.30; bias +0.15) with preserved ranking. Operational blueprint (cost/latency) enabling instructors to concentrate effort on outliers and top-tail moderation. Governance pattern combining pseudonymisation, audit logging and non-punitive disclosure of AI usage. Threats to Validity (Mitigations) Internal: Instructor drift → periodic cross-checks and frozen exemplars. External: Single-course context →replication across disciplines, languages and rubric shapes. Construct: AI-usage detector imperfect →advisory only; triangulated with behavioural signals. Temporal: Model versions evolve →version pinning and re-calibration each term. Ethics & Governance Pseudonymised prompts ( AnonID ); no names/emails to vendors; encrypted transport; instructor retains final authority; structured audit logs; opt-out route; advisory detection only (no penalties); fairness and accessibility checks documented. Selected References Vaswani et al. (2017) Attention Is All You Need. Peffers et al. (2007) Design Science Research. Kitchenham (2007) Guidelines for SLR. Brown et al. (2020) Language Models are Few-Shot Learners. Bender et al. (2021) Stochastic Parrots. Limitations & Future Work Single-course sample; detector is advisory; model drift requires periodic calibration. Next: cross-course replication, subgroup fairness audits, rubric-anchored XAI rationales, and optimal co-grading policies (human sampling strategies).