Autograding with points stops working if any tests are edited

[mh-skjelvareid]

Autograding allows for automatically awarding points when individual tests pass. If you create the tests and don't touch them again, it works. However: If you edit any tests, all the test points become blank. If you try resetting the points for a test, the changes are not saved.

If you have a test user and clone the repository for the assignment, you can see how the points are missing by opening .github/classroom/autograding.json . All points are set to null.

This behavior has been observed by multiple users:

• https://github.com/orgs/community/discussions/67008
• https://github.com/orgs/community/discussions/67572

Minimal example to recreate:

• Create a new assignment
• Create a test, e.g. based on pytest (run pytest test.py) and set a number of points for the test (e.g. 5)
• Save the test and the assignment
• Re-open the assignment and edit the test - the point field is now blank

[jeffrafter]

Ah, this is likely a problem in Github Classroom not the auto grading itself. I wonder if @zrdaley knows?

[juanpflores]

cc @RyanHecht

[alexandruradovici]

I have the same problem, github classroom does not even show the points list anymore.

Never mind, it seems to work now. I do have the same problem as above.

[alexandruradovici]

There is another problem, it seems that whenever we modify something in the tests, the autograding file is updated in the student's repository, but the job is not rerun, it is skipped. Students have to push another modification to see the changes. This is very frustrating.

[heiko-holz]

+1
Same for me. I switched to GH Classrooms due to the autograding feature for this term's introductory to object-oriented programming course, but the current state is very frustrating (for the students and for me).

[RyanHecht]

Hey folks! We've opened an issue for this internally and are working on a fix. I'll update this issue when the fix has been deployed!

[RyanHecht]

This should be resolved now! :)

[alexandruradovici]

Is there any way we could add a feature to be able to upload the autograder.json file to github classroom? We have around 70 tests that we have to input manually.

[markpatterson27]

@alexandruradovici You can add the autograder.json (and the .github/workflows/classroom.yml) to a template repo and use that as the starter code source for a GH Classroom assignment.

The tests won't be imported into GH Classroom, but they will still run, and the points will still be reported back to GH Classroom. (It would be really nice if it was possible to import autograder.json into GH Classroom though.)

[myavuz21]

This should be resolved now! :)

@RyanHecht

I still have this 'skipped' issue when I update the test cases of an assignment. I tried it on a newly created assignment, but it is still the case. I do not want to re-run workflows one by one or rely on students to do it themselves. Can you tell me how I can fix this?

[markpatterson27]

Looks like they've added if: github.actor != 'github-classroom[bot]' to the generated classroom.yml workflow. This will 'skip' the workflow for any commits made by github-classroom, such as when updating assignments.

I'd argue that most people would prefer @myavuz21's way, where the tests re-run whenever they are changed (i.e., even for commits made by github-classom[bot]).

[GittyBitch]

Ah, this is likely a problem in Github Classroom not the auto grading itself. I wonder if @zrdaley knows?

this is now closed source, right ? Just wondering, because there seem to be a few other backend-level issues, especially when it comes to using reusable workflows, which prepend/prefix the name of the action and thus modify the whole name, which in turn renders output.ts (points reporting) dysfunctional, because there's the hard-coded assumption that the autograding stage is indeed called "Autograding" (it's used as a lookup key for the octkit API call) - and not even fixing that, solves the problem - since the github classroom UI also seems to be using the hard-coded "Autograding" name internally.

Assuming this is hard to get fixed (given that this stuff is now closed-source), it might make sense to mention this much more prominently in the docs ...

Thanks

[tomitrescak]

Hello, it seems that you found a way to award point per passed test case. Would you be able to share how? I can only see a way to award point for a whole test suite. I would love to see how many test have pased or failed in that suite. I am using node.js and vitest.