Codeship with Testim.io - Am I testing my latest version?

Question

Sorry for the dumb question but I could not find an answer for that.

Codeship + Testim.io + Heroku.

In my Staging env - I use Testim.io to test the app once it deployed.
The following tutorial is guiding me how to invoke my tests - but I see the tests are being invoked BEFORE the new app has been deployed - so isn't it testing one version before my latest? I expected the tests to run after the deploy.
Probably I am missing here something.

Answer 1

In that tutorial - the tests aren't supposed to run against your deployed version they are supposed to run against the version being tested.

The flow is:

• You set up a local environment - for example by checking out your code and running npm start. If it's containerized then do that.
• You run the Testim CLI and point its base url to local how testim --token ... --project ... --suite ... --base-url=localhost:PORT.
• After the tests pass, you deploy.

If you test the version after you deploy you can't be sure the version deployed actually passes your tests.

An alternative flow would be to lean into Heroku's deployment model. Note this isn't actually specific to them and there are similar alternatives in aws/azure/gcp/whatever:

• In your CI, you set up a staging environment in heroku heroku create --remote staging-BRANCH-NAME-<COMMIT-NAME>
• You deploy there.
• You run the tests against that enviroment (by passing --base-url to the Testim CLI, navigating there in your test or using a config file)
• When tests pass you deploy to master