I want students to submit JavaScript code to our server. Then we test that JavaScript code to make sure the students solved the problem. I want to test for exceptions thrown, output, what's returned, execution time, individual variables and functions that were called.
We run Ruby on Rails in Heroku, but I'm happy to put up a node.js/io.js server to run the JavaScript code and have our Ruby on Rails code simply send over the user submitted code, wait for a response for success or problems with that user-submitted code and then respond back to the user.
What's the best solution for this? Of course, I need to have a timeout as well to make sure while (true)s don't cause problems.
I think an interesting solution would be integrating circleci. Circleci will automatically sandbox the apps for you. You can simply override the test commands as the run command (e.g.
node cs_123123123123.js), and if it throws an exception while running circleci will trigger a build-failed webhook.In terms of checking the call stack and so on, it's gonna be tricky whatever solution you use. While I can't give specific advise regarding this, you could take a look at how istanbul checks code coverage because it would likely be quite similar. Best of luck!
Edit: Also, another cool solution would be to actually just set up test cases for these circleci builds to automatically see if output is correct