George Sflomos

@georgesflomos.bsky.social

1819 Followers 406 Following

Cancer Researcher passionate about Lobular Carcinoma, Preclinical Oncology, Discovery Science & Advocacy @LobularBCA, EACR ECR Council Member @helloeacr.bsky.social

AI assistants, such as ChatGPT, are being increasingly used by students in higher
education institutions. While these tools provide opportunities for improved teaching
and education, they also pose significant challenges for assessment and learning
outcomes. We conceptualize these challenges through the lens of vulnerability, the
potential for university assessments and learning outcomes to be impacted by student
use of generative AI.Weinvestigate the potential scale of this vulnerability by measuring
the degree to which AI assistants can complete assessment questions in standard
university-level Science, Technology, Engineering, and Mathematics (STEM) courses.
Specifically, we compile a dataset of textual assessment questions from 50 courses at
the École polytechnique fédérale de Lausanne (EPFL) and evaluate whether two AI
assistants, GPT-3.5 and GPT-4 can adequately answer these questions. We use eight
prompting strategies to produce responses and find that GPT-4 answers an average
of 65.8% of questions correctly, and can even produce the correct answer across at
least one prompting strategy for 85.1% of questions. When grouping courses in our
dataset by degree program, these systems already pass the nonproject assessments
of large numbers of core courses in various degree programs, posing risks to higher
education accreditation that will be amplified as these models improve. Our results call
for revising program-level assessment design in higher education in light of advances
in generative AI.