Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Programming exercises: Allow to choose preliminary feedback model #10067

Open
wants to merge 22 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
7e6761a
differentiate between feedback suggestions and preliminary feedback m…
dmytropolityka Dec 3, 2024
107a669
Merge remote-tracking branch 'origin/develop' into feature/programmin…
dmytropolityka Dec 11, 2024
e494976
server side for differentiation between athena modules
dmytropolityka Dec 11, 2024
1d1fa2f
merge conflict
dmytropolityka Dec 11, 2024
da45bab
fix some bugs and update existing data
dmytropolityka Dec 23, 2024
f142fab
Merge branch 'develop' into feature/programming-exercises/choose-prel…
dmytropolityka Jan 1, 2025
87e748c
Change module type in Athena Resource
dmytropolityka Jan 1, 2025
c8bd16d
Merge remote-tracking branch 'origin/feature/programming-exercises/ch…
dmytropolityka Jan 1, 2025
475bec2
Adjust client tests
dmytropolityka Jan 1, 2025
5760a66
add checks for new component
dmytropolityka Jan 2, 2025
ea3a0ce
fix other tests
dmytropolityka Jan 2, 2025
61f9258
make options components standalone
dmytropolityka Jan 2, 2025
fc3968c
Adjust server tests
dmytropolityka Jan 2, 2025
8fe51e5
remove translate pipe from declarations
dmytropolityka Jan 2, 2025
d4eb493
formatting
dmytropolityka Jan 2, 2025
4552184
fine grained disabled control
dmytropolityka Jan 2, 2025
dc0955b
sync transmitted object
dmytropolityka Jan 2, 2025
97d8e59
improve log message
dmytropolityka Jan 2, 2025
4aa1c7e
replace modules in integration tests
dmytropolityka Jan 4, 2025
e2250d4
spotless
dmytropolityka Jan 4, 2025
0aeb2a0
Merge branch 'develop' into feature/programming-exercises/choose-prel…
dmytropolityka Jan 4, 2025
82d77bc
change module
dmytropolityka Jan 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package de.tum.cit.aet.artemis.athena.domain;

public enum ModuleType {
FEEDBACK_SUGGESTIONS, PRELIMINARY_FEEDBACK
}
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,11 @@ public void sendFeedback(Exercise exercise, Submission submission, List<Feedback

try {
// Only send manual feedback from tutors to Athena
// Based on the current design, this applies only to feedback suggestions
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission),
feedbacks.stream().filter(Feedback::isManualFeedback).map((feedback) -> athenaDTOConverterService.ofFeedback(exercise, submission.getId(), feedback)).toList());
ResponseDTO response = connector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/feedbacks", request, maxRetries);
ResponseDTO response = connector.invokeWithRetry(
athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), exercise.getFeedbackSuggestionModule()) + "/feedbacks", request, maxRetries);
log.info("Athena responded to feedback: {}", response.data);
}
catch (NetworkingException networkingException) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,65 +95,77 @@ private record ResponseDTOModeling(List<ModelingFeedbackDTO> data, ResponseMetaD
/**
* Calls the remote Athena service to get feedback suggestions for a given submission.
*
* @param exercise the {@link TextExercise} the suggestions are fetched for
* @param submission the {@link TextSubmission} the suggestions are fetched for
* @param isGraded the {@link Boolean} should Athena generate grade suggestions or not
* @param exercise the {@link TextExercise} the suggestions are fetched for
* @param submission the {@link TextSubmission} the suggestions are fetched for
* @param isPreliminary the {@link Boolean} should Athena generate grade suggestions or not
* @return a list of feedback suggestions
*/
public List<TextFeedbackDTO> getTextFeedbackSuggestions(TextExercise exercise, TextSubmission submission, boolean isGraded) throws NetworkingException {
log.debug("Start Athena '{}' Feedback Suggestions Service for Exercise '{}' (#{}).", isGraded ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
public List<TextFeedbackDTO> getTextFeedbackSuggestions(TextExercise exercise, TextSubmission submission, boolean isPreliminary) throws NetworkingException {
log.debug("Start Athena {} Feedback Suggestions Service for Exercise '{}' (#{}).", isPreliminary ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved

if (!Objects.equals(submission.getParticipation().getExercise().getId(), exercise.getId())) {
log.error("Exercise id {} does not match submission's exercise id {}", exercise.getId(), submission.getParticipation().getExercise().getId());
throw new ConflictException("Exercise id " + exercise.getId() + " does not match submission's exercise id " + submission.getParticipation().getExercise().getId(),
"Exercise", "exerciseIdDoesNotMatch");
}

final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission), isGraded);
ResponseDTOText response = textAthenaConnector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/feedback_suggestions", request, 0);
log.info("Athena responded to '{}' feedback suggestions request: {}", isGraded ? "Graded" : "Non Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isGraded);
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission),
!isPreliminary);
ResponseDTOText response = textAthenaConnector.invokeWithRetry(
athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), isPreliminary ? exercise.getPreliminaryFeedbackModule() : exercise.getFeedbackSuggestionModule())
+ "/feedback_suggestions",
request, 0);
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved
log.info("Athena responded to '{}' feedback suggestions request: {}", isPreliminary ? "Non Graded" : "Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isPreliminary);
return response.data.stream().toList();
}

/**
* Calls the remote Athena service to get feedback suggestions for a given programming submission.
*
* @param exercise the {@link ProgrammingExercise} the suggestions are fetched for
* @param submission the {@link ProgrammingSubmission} the suggestions are fetched for
* @param isGraded the {@link Boolean} should Athena generate grade suggestions or not
* @param exercise the {@link ProgrammingExercise} the suggestions are fetched for
* @param submission the {@link ProgrammingSubmission} the suggestions are fetched for
* @param isPreliminary the {@link Boolean} should Athena generate grade suggestions or not
* @return a list of feedback suggestions
*/
public List<ProgrammingFeedbackDTO> getProgrammingFeedbackSuggestions(ProgrammingExercise exercise, ProgrammingSubmission submission, boolean isGraded)
public List<ProgrammingFeedbackDTO> getProgrammingFeedbackSuggestions(ProgrammingExercise exercise, ProgrammingSubmission submission, boolean isPreliminary)
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved
throws NetworkingException {
log.debug("Start Athena '{}' Feedback Suggestions Service for Exercise '{}' (#{}).", isGraded ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission), isGraded);
ResponseDTOProgramming response = programmingAthenaConnector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/feedback_suggestions", request, 0);
log.info("Athena responded to '{}' feedback suggestions request: {}", isGraded ? "Graded" : "Non Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isGraded);
log.debug("Start Athena {} Feedback Suggestions Service for Exercise '{}' (#{}).", isPreliminary ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission),
!isPreliminary);
ResponseDTOProgramming response = programmingAthenaConnector.invokeWithRetry(
athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), isPreliminary ? exercise.getPreliminaryFeedbackModule() : exercise.getFeedbackSuggestionModule())
+ "/feedback_suggestions",
request, 0);
log.info("Athena responded to '{}' feedback suggestions request: {}", isPreliminary ? "Non-Graded" : "Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isPreliminary);
return response.data.stream().toList();
}

/**
* Retrieve feedback suggestions for a given modeling exercise submission from Athena
*
* @param exercise the {@link ModelingExercise} the suggestions are fetched for
* @param submission the {@link ModelingSubmission} the suggestions are fetched for
* @param isGraded the {@link Boolean} should Athena generate grade suggestions or not
* @param exercise the {@link ModelingExercise} the suggestions are fetched for
* @param submission the {@link ModelingSubmission} the suggestions are fetched for
* @param isPreliminary the {@link Boolean} should Athena generate grade suggestions or not
* @return a list of feedback suggestions generated by Athena
*/
public List<ModelingFeedbackDTO> getModelingFeedbackSuggestions(ModelingExercise exercise, ModelingSubmission submission, boolean isGraded) throws NetworkingException {
log.debug("Start Athena '{}' Feedback Suggestions Service for Modeling Exercise '{}' (#{}).", isGraded ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
public List<ModelingFeedbackDTO> getModelingFeedbackSuggestions(ModelingExercise exercise, ModelingSubmission submission, boolean isPreliminary) throws NetworkingException {
log.debug("Start Athena {} Feedback Suggestions Service for Modeling Exercise '{}' (#{}).", isPreliminary ? "Graded" : "Non Graded", exercise.getTitle(), exercise.getId());
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved

if (!Objects.equals(submission.getParticipation().getExercise().getId(), exercise.getId())) {
throw new ConflictException("Exercise id " + exercise.getId() + " does not match submission's exercise id " + submission.getParticipation().getExercise().getId(),
"Exercise", "exerciseIdDoesNotMatch");
}

final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission), isGraded);
ResponseDTOModeling response = modelingAthenaConnector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/feedback_suggestions", request, 0);
log.info("Athena responded to '{}' feedback suggestions request: {}", isGraded ? "Graded" : "Non Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isGraded);
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), athenaDTOConverterService.ofSubmission(exercise.getId(), submission),
!isPreliminary);
ResponseDTOModeling response = modelingAthenaConnector.invokeWithRetry(
athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), isPreliminary ? exercise.getPreliminaryFeedbackModule() : exercise.getFeedbackSuggestionModule())
+ "/feedback_suggestions",
request, 0);
log.info("Athena responded to '{}' feedback suggestions request: {}", isPreliminary ? "Non Graded" : "Graded", response.data);
storeTokenUsage(exercise, submission, response.meta, !isPreliminary);
return response.data;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import de.tum.cit.aet.artemis.athena.domain.ModuleType;
import de.tum.cit.aet.artemis.core.domain.Course;
import de.tum.cit.aet.artemis.core.exception.BadRequestAlertException;
import de.tum.cit.aet.artemis.core.exception.NetworkingException;
Expand Down Expand Up @@ -107,21 +108,22 @@ public List<String> getAthenaModulesForCourse(Course course, ExerciseType exerci
/**
* Get the URL for an Athena module, depending on the type of exercise.
*
* @param exercise The exercise for which the URL to Athena should be returned
* @param exerciseType The exercise type for which the URL to Athena should be returned
* @param module The name of the Athena module to be consulted
* @return The URL prefix to access the Athena module. Example: <a href="http://athena.example.com/modules/text/module_text_cofee"></a>
*/
public String getAthenaModuleUrl(Exercise exercise) {
switch (exercise.getExerciseType()) {
public String getAthenaModuleUrl(ExerciseType exerciseType, String module) {
switch (exerciseType) {
case TEXT -> {
return athenaUrl + "/modules/text/" + exercise.getFeedbackSuggestionModule();
return athenaUrl + "/modules/text/" + module;
}
case PROGRAMMING -> {
return athenaUrl + "/modules/programming/" + exercise.getFeedbackSuggestionModule();
return athenaUrl + "/modules/programming/" + module;
}
case MODELING -> {
return athenaUrl + "/modules/modeling/" + exercise.getFeedbackSuggestionModule();
return athenaUrl + "/modules/modeling/" + module;
}
default -> throw new IllegalArgumentException("Exercise type not supported: " + exercise.getExerciseType());
default -> throw new IllegalArgumentException("Exercise type not supported: " + exerciseType);
}
}

Expand All @@ -130,22 +132,34 @@ public String getAthenaModuleUrl(Exercise exercise) {
*
* @param exercise The exercise for which the access should be checked
* @param course The course to which the exercise belongs to.
* @param moduleType The module type for which the access should be checked.
* @param entityName Name of the entity
* @throws BadRequestAlertException when the exercise has no access to the exercise's provided module.
*/
public void checkHasAccessToAthenaModule(Exercise exercise, Course course, String entityName) throws BadRequestAlertException {
if (exercise.isExamExercise() && exercise.getFeedbackSuggestionModule() != null) {
public void checkHasAccessToAthenaModule(Exercise exercise, Course course, ModuleType moduleType, String entityName) throws BadRequestAlertException {
String module = getModule(exercise, moduleType, entityName);
if (exercise.isExamExercise() && module != null) {
throw new BadRequestAlertException("The exam exercise has no access to Athena", entityName, "examExerciseNoAccessToAthena");
}
if (!course.getRestrictedAthenaModulesAccess() && restrictedModules.contains(exercise.getFeedbackSuggestionModule())) {
if (!course.getRestrictedAthenaModulesAccess() && restrictedModules.contains(module)) {
// Course does not have access to the restricted Athena modules
throw new BadRequestAlertException("The exercise has no access to the selected Athena module", entityName, "noAccessToAthenaModule");
throw new BadRequestAlertException("The exercise has no access to the selected Athena module of type " + moduleType, entityName, "noAccessToAthenaModule");
}
}

private static String getModule(Exercise exercise, ModuleType moduleType, String entityName) {
String module = null;
switch (moduleType) {
case ModuleType.FEEDBACK_SUGGESTIONS -> module = exercise.getFeedbackSuggestionModule();
case ModuleType.PRELIMINARY_FEEDBACK -> module = exercise.getPreliminaryFeedbackModule();
}
return module;
}

/**
* Checks if a module change is valid or not. In case it is not allowed it throws an exception.
* Modules cannot be changed after the exercise due date has passed.
* Holds only for feedback suggestion modules.
*
* @param originalExercise The exercise before the update
* @param updatedExercise The exercise after the update
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public AthenaRepositoryExportService(ProgrammingExerciseRepository programmingEx
* @throws AccessForbiddenException if the feedback suggestions are not enabled for the given exercise
*/
private void checkFeedbackSuggestionsOrAutomaticFeedbackEnabledElseThrow(Exercise exercise) {
if (!(exercise.areFeedbackSuggestionsEnabled() || exercise.getAllowFeedbackRequests())) {
if (!(exercise.areFeedbackSuggestionsEnabled() || exercise.isPreliminaryFeedbackEnabled())) {
log.error("Feedback suggestions are not enabled for exercise {}", exercise.getId());
throw new ServiceUnavailableException("Feedback suggestions are not enabled for exercise");
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ public Optional<Long> getProposedSubmissionId(Exercise exercise, List<Long> subm
try {
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise), submissionIds);
// allow no retries because this should be fast and it's not too bad if it fails
ResponseDTO response = connector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/select_submission", request, 0);
// applies only to feedback suggestions
ResponseDTO response = connector
.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), exercise.getFeedbackSuggestionModule()) + "/select_submission", request, 0);
log.info("Athena to calculate next proposes submissions responded: {}", response.submissionId);
if (response.submissionId == -1) {
return Optional.empty();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,9 @@ public void sendSubmissions(Exercise exercise, Set<Submission> submissions, int
try {
final RequestDTO request = new RequestDTO(athenaDTOConverterService.ofExercise(exercise),
filteredSubmissions.stream().map((submission) -> athenaDTOConverterService.ofSubmission(exercise.getId(), submission)).toList());
ResponseDTO response = connector.invokeWithRetry(athenaModuleService.getAthenaModuleUrl(exercise) + "/submissions", request, maxRetries);
// applies only to feedback suggestions
ResponseDTO response = connector.invokeWithRetry(
athenaModuleService.getAthenaModuleUrl(exercise.getExerciseType(), exercise.getFeedbackSuggestionModule()) + "/submissions", request, maxRetries);
log.info("Athena (calculating automatic feedback) responded: {}", response.data);
}
catch (NetworkingException error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,17 +105,17 @@ public AthenaResource(CourseRepository courseRepository, TextExerciseRepository
}

@FunctionalInterface
private interface FeedbackProvider<ExerciseType, SubmissionType, OutputType> {
private interface FeedbackProvider<ExerciseType, SubmissionType, Boolean, OutputType> {

/**
* Method to apply the (graded) feedback provider. Examples: AthenaFeedbackSuggestionsService::getTextFeedbackSuggestions,
* AthenaFeedbackSuggestionsService::getProgrammingFeedbackSuggestions
*/
List<OutputType> apply(ExerciseType exercise, SubmissionType submission, Boolean isGraded) throws NetworkingException;
List<OutputType> apply(ExerciseType exercise, SubmissionType submission, Boolean isPreliminary) throws NetworkingException;
}

private <ExerciseT extends Exercise, SubmissionT extends Submission, OutputT> ResponseEntity<List<OutputT>> getFeedbackSuggestions(long exerciseId, long submissionId,
Function<Long, ExerciseT> exerciseFetcher, Function<Long, SubmissionT> submissionFetcher, FeedbackProvider<ExerciseT, SubmissionT, OutputT> feedbackProvider) {
Function<Long, ExerciseT> exerciseFetcher, Function<Long, SubmissionT> submissionFetcher, FeedbackProvider<ExerciseT, SubmissionT, Boolean, OutputT> feedbackProvider) {

log.debug("REST call to get feedback suggestions for exercise {}, submission {}", exerciseId, submissionId);

Expand All @@ -130,7 +130,8 @@ private <ExerciseT extends Exercise, SubmissionT extends Submission, OutputT> Re
final var submission = submissionFetcher.apply(submissionId);

try {
return ResponseEntity.ok(feedbackProvider.apply(exercise, submission, true));
// this resource is only for graded feedback suggestions
return ResponseEntity.ok(feedbackProvider.apply(exercise, submission, false));
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved
}
catch (NetworkingException e) {
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).build();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,11 @@ else if (channel.getExam() != null) {
}
}

/**
* Converts the current object into a {@link Channel} instance.
*
* @return Channel with the created channel.
*/
public Channel toChannel() {
Channel channel = new Channel();
channel.setName(this.name);
Expand Down
Loading
Loading