How does the Github Classroom get the results from the autograder?

[kenpower]

Hi everyone,

I am trying to create a custom autograder action for a complex series of tests and to give better feedback.

But looking at the official autograder code (https://github.com/education/autograding) I can't see where the points are stored/send somewhere that Github classroom can access. The only output I see from the autograder is to the run check, but that is an unstructured string?

Anyone know how this works? How does Github classroom get the grade?

A suggestion...(optional)

more help with creating custom actions

[kenpower]

I don't know if this will work, or even if it is ok to do, but I am going to tag @markpatterson27 here (contributor to autograding action), maybe they will know the answer?

[markpatterson27]

I think you've answered your own question. Autograding uses check suite to report the grading score to GH classroom. (Specifically, this call: https://octokit.github.io/rest.js/v18#checks-update)

There's an issue with an actions/github-script that pulls the relevant code: github-education-resources/autograding#52 I haven't checked , but change the text string to the value you want (including the 'Points ' prefix in the format Points {points}/{total-points}). Presumably GH classroom parses the string on the other end. Pretty sure the points have to be integers.

I have thought about putting that code into a separate action or proposing a points input on the autograding action, for cases where it's easier to calculate the points outside of the autograding action.

(Also, most of my contributions were to the package-lock.json file. Not all code is equal ;))

[kenpower]

Thanks so much. I assumed that there might be a more "formal" mechanism, i.e. not using a human string to pass numerical data. But having explored more, there are not a lot of other options.

"Presumably GH classroom parses the string on the other end. "

I would like to see that, but AFAIK GH Classroom is now closed source, is that correct?

[markpatterson27]

As an alternative to creating a custom autograder, you could create scripts that do the grading and feedback for you, and just use autograding to call those scripts.

See https://github.com/SERC-COM299/CookieDB-KV1-CREATE-and-INSERT/blob/6bb50fa2e734cd9798da8806811ffc016c11ee20/.github/classroom/autograding.json#L44-L51 for an example of grading if the student has used the correct data type for an attribute in a SQL database (along with providing them feedback as a comment in the 'Feedback' PR).

[kenpower]

I had thought of that, but the problem is the autograder will only give a score if ALL the tests in a script pass. I have a file with 40 unit tests and I want to give a score if say 15 tests pass, the score should be 15/40. Going further, I would like to have different scores for each unit test.

I could set up 40 different blocks in the autograding.json like in your example, but I have to build that and run my test runner 40+ times. Seems like a bit overkill when the test runner can run once and can already report how many tests failed/passed.

Maybe there could be an option for the test runner script to create an env var that the autograder uses (like you suggested above)?

[kenpower]

Thanks for linking the SQL testing script, there is a lot to learn from it!

[markpatterson27]

Have you signed up for the autograding usability study? They're currently looking at ways to make autograding more flexible. Your input would probably be helpful.

As a quick test, I put together direct points input for autograding. If you want to try it out replace the education/autograding step with this:

      - uses: markpatterson27/autograding@dev-points-input-release
        with:
          points: 10
          available-points: 50

• points is the amount of points the student as scored
• available-points is the total amount of available points to earn

You can pass values between steps in a workflow, either by writing to the $GITHUB_ENV file or you can create a step output by writing to the $GITHUB_OUTPUT file. See: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable.

So, the business part of your workflow might look something like

...
      - name: run grading
        id: grading
        run: <command to run unit tests with grading output>

      - uses: markpatterson27/autograding@dev-points-input-release
        with:
          points: ${{ steps.grading.outputs.points }}
          available-points: ${{ steps.grading.outputs.total-points }}

[kenpower]

Thanks you so much for this.
I tried out the new version, it mostly works, but there are a couple of issues (probably on the classroom side and maybe unfixable?).

First, the score is correct on the classroom web page (great!) (not sure why it's saying no commits?)

However, in the results download, it shows a zero for points available:

But that is easily fixable on the teachers side.

Brilliant! What is the next step, will there be a PR to main?

Thanks Again!

[markpatterson27]

I opened a PR for it: github-education-resources/autograding#248

[agilebotanist]

This method works fine for the markpatterson27/autograding@dev-points-input-release alone.

Classroom updates as needed. I hoped to use this as a workaround by combining with the official Autograding. Since currently my Classroom does not update grades labels :/

Here is my code to debug and pass the grades to the Classroom by this custom action. Unfortunately, this workaround does not work.

name: GitHub Classroom Workflow

on: [push]

jobs:
  build:
    permissions: write-all
    name: Autograding
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Auto
        id: auto
        uses: education/autograding@v1
      - name: Out
        id: out
        env: 
          GRADES: ${{join(steps.auto.outputs.*, '\n')}}
        run: |
          echo "Auto print $GRADES"
          points=$(echo "$GRADES" | awk -F'/' '{print $1}')
          echo 'points<<EOF' >> $GITHUB_OUTPUT
          echo $points >> $GITHUB_OUTPUT
          echo EOF >> $GITHUB_OUTPUT
          totalpoints=$(echo "$GRADES" | awk -F'/' '{print $2}')
          echo 'totalpoints<<EOF' >> $GITHUB_OUTPUT
          echo $totalpoints >> $GITHUB_OUTPUT
          echo EOF >> $GITHUB_OUTPUT
      - name: Get points
        env:
          POINTS: ${{ steps.out.outputs.points }}
          TOTAL: ${{ steps.out.outputs.totalpoints }}
        run: echo "Grades $POINTS / $TOTAL"
      - name: mark
        id: mark
        env:
          POINTS: ${{ steps.out.outputs.points }}
          TOTAL: ${{ steps.out.outputs.totalpoints }}
#       run: echo "points=7" >> "$GITHUB_OUTPUT" ; echo "total-points=9" >> "$GITHUB_OUTPUT"
        uses: markpatterson27/autograding@dev-points-input-release
        with:
          points: ${{ steps.out.outputs.points }}
          available-points: ${{ steps.out.outputs.totalpoints }}
#          points: 8
#          available-points: 15

[ProfWentao]

https://github.com/education/autograding no longer exists. So what is the official way to send the test result to github classroom so it can display the score? Do we have to use this instead? markpatterson27/autograding@dev-points-input-release