Bridging code and timely feedback: integrating generative AI into a programming platform

Purpose This article examines how generative artificial intelligence (GenAI) can improve students’ programming skills through timely feedback. Specifically, it evaluates the effectiveness of feedback provided through two custom-developed applications: (1) A chatbot-like virtual assistant powered by the GPT-4o-mini model designed to assist students interactively; (2) A programming platform that combines GenAI-generated feedback and a virtual judge for source code evaluation. The study explores whether these tools contribute to improving students’ programming performance. Method The proposed method consists of the following tasks: (1) Development of two functional prototypes powered by GPT-4o-mini, the first of them a conversational chatbot with access to a specific programming knowledge base, and the second an innovative development that integrates in a single platform a GenAI and an open-source virtual judge for the joint generation of an automated assessment and timely feedback of the source code. (2) Designing and executing the experiment to evaluate whether providing feedback to students in a beginner programming course using GEN’s AI-based tools improved the passing rate in an assessment. To achieve this, a quasi-experimental study was conducted in which students engaged in context-enriched problem-solving. Two-course sections served as experimental groups, receiving feedback from the AI tools. At the same time, the other two groups served as controls. (3) Measuring students’ performance in problem-solving and collecting student perceptions on usability for the chatbot and the competitive programming platform. Results and Conclusions Students who received feedback generated by GenAI did not show improvement in their performance, although they perceived it as useful. Conversational (tutorB@t) and non-conversational platform (tutorBot+) versions are available at https://cdn.botpress.cloud/webchat/v3.2/shareable.html?configUrl=https://files.bpcontent.cloud/2025/01/20/23/20250120232250-Q9SCHBOO.json and https://supc.ucsc.cl:23380/, respectively, under an AGPL 1.0 license. From the processing and analysis, a good perception of the usability of both tools was obtained (70.6% and 65.2%, respectively), which is reinforced by the 81% and 79% of students who stated that they would use them again. Although the usefulness and perception of the tools are considered positive, there is insufficient evidence to definitively evaluate the effect of GenAI-generated feedback on students’ passing rates in an assessment.