Can you use secret tests in github classroom autograding?

Question

For a new machine learning course we're looking to design a series of coding assignments in which students get some starter code, and make improvements until the unit tests pass. Then they commit and push their code back to the remote where an autograding workflow runs more tests to see if they did adequate work.

What we'd like to do would be to give the students some tests that they can look into, to see what the general programming goal is; but to also have a secret unit test to try their code on data that the students have never seen. On this unseen test data they'd have to reach at least a certain accuracy score to get a passing grade.

The question is: can this be done in github classroom? It seems that the default setup is to give all the tests openly in the starter code repository. But we want to have some tests that the students can't see, so that we can test if they're only narrowly writing to the visible test or actually writing a properly generic solution.

If this isn't directly possible, is there a workaround strategy?

Answer 1

Github Classroom does not directly support using secret tests.

A workaround is to use one of the grading methods offered (e.g. Run Command) and fetch a secret tests script that is hosted somewhere online then execute the script that was just downloaded.

As it is configured in the Github Classroom, the students should not be able to find that secret tests script and it will be accessed only by the runner.

For example, your Test Command could be defined like this:

curl -s https://my.secret.domain/tests.sh | bash -s

Then configure the rest of the Test Case in Github Classroom.

Answer 2

No idea if this could or would work, but maybe try the top answer from here :

" GitHubPages (like Bitbucket Pages and GitLab Pages) only serve static pages, so the only solution is something client side (Javascript).

A solution could be, instead of using real authentication, just to share only a secret (password) with all the authorized persons and implement one of the following scheme:

put all the private files in a (not listed) subdirectory and name that with the hash of the chosen password. The index page asks you (with Javascript) for the password and build the correct start link calculating the hash.

See for example: https://github.com/matteobrusa/Password-protection-for-static-pages

PRO: Very simple approach protecting a whole subdirectory tree

CONS:

possible attack: sniffing the following requests to obtain the name of the subdirectory the admins on the hosting site have access to the full contents crypt the page with password and decrypt on the fly with javascript

see for example: https://github.com/robinmoisson/staticrypt

PRO: no plaintext page code around (decrypting happens on the client side)

CONS:

just a single page, and need to reinsert the password on every refresh an admin could change your Javascript code to obtain the password when you insert it"