I'm trying to write a test suite for verifying the state of some servers using testinfra.
It's my first time working with python/testinfra/pytest.
As a brief pseudocodey example
test_files.py
testinfra_hosts=[server1,server2,server3]
with open("tests/microservices_default_params.yml", "r") as f:
try:
default_params = yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
with open("tests/" + server + "/params/" + server + "_params.yml", "r") as f:
try:
instance_params = yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
@pytest.mark.parametrize(
"name", [] + default_params["files"] + instance_params["files"]
)
def test_files(host, name):
file = host.file(name)
assert file.exists
Each server has it's own unique params yaml file. I want every server to go through the same test, however I need each server to run the test with its own parametrized values from its respective .yml file.
The problem with the code above, is that it will try to execute all of server1s unique params against both server 2 and 3, then will start again with server 2 being run against servers 1-3 unique params.
I can't find a clean way to essentially have the test run once with server1 as the host, and server 1 params, then do the same again with server2 and server2 params etc.
I've tried using for loops within the test file itself, reading each instance_params.yml into a dictionary with the key being the server name and value containing all of that servers params - but that doesn't smell very good and because the assert is inside the loop, if one of the params for that server fails the loop exits and doesn't attempt any further params for that server.
I've looked into pytest_collection_modifyitems but I can't quite get my head around how to let it do what I want. I feel like there may be an easy solution to this that I'm missing.
My last resort would be to seperate out the tests and parametrized params individually as
@pytest.mark.parametrize(
"server1_params", instance_params['server1']['files]
)
def_test_files_server1(host, server1_params):
...
@pytest.mark.parametrize(
"server2_params", instance_params['server2']['files]
)
def_test_files_server2(host,server2_params):
...
That approach doesn't sound right to me though.
Any help for a fresh junior would be appreciated, I've never asked anything here before Hope it makes sense :)
Update: @ajk Found the solution! The pytest_generate_tests function was exactly what I was needing - and I've furthered my understanding of pytest along the way.
Thanks ajk! I owe you one :D
That's a fine question, and it looks like you've come at it from a couple different angles already. I'm no expert myself, but there are a few different ways I can think to do something like this in pytest. They all involve handing the heavy lifting over to fixtures. So here's a quick breakdown of one way to adapt the code you've already shared to use some fixtures:
Default Parameters
It looks like you've got some parameters that are not host-specific. It likely makes sense to pull those into a session-scoped fixture so you can reuse the same values across many hosts:
That way, you can add
default_params
as an argument to any test function that needs those values.Host-specific Parameters
You certainly could load these parameters as you were doing before, or put some lookup logic in the test itself. That may be the best and clearest approach! Another option is to have a parametrized fixture whose value varies by instance:
Now, if we add
instance_params
as an argument to a test function it will run the test once for each entry intestinfra_hosts
. The fixture will return a new value each time, based on the active host.Writing the Test
If we farm the heavy lifting out to fixtures, the actual test can become simpler:
(I'm not even asserting anything here, just playing with the fixtures to make sure they're doing what I think they're doing)
Taking it for a spin
I tried this out locally with the following sample yaml files:
tests/microservices_default_params.yml
tests/server1/params/server1_params.yml
tests/server2/params/server2_params.yml
tests/server3/params/server3_params.yml
Running the test file produces:
Which seems to be at least the general direction you were shooting for. I hope some of that is useful - good luck and happy testing!
Update
The comment below asks about breaking this out so each file from the list generates its own test. I'm not sure the best way to do that, but a couple options might be:
@pytest.mark.parametrize
pytest_generate_tests
to set up dynamic parametrization, similar to what's described here.In either case, you could start with something like this:
To use
@pytest.mark.parametrize
, you could follow that up with:Or to take the
pytest_generate_tests
approach, you could do this instead:Either way could work, and I suspect more experienced folks might package the
pytest_generate_tests
version up into a class and clean up the logic a bit. We have to start somewhere though, eh?