As my hypothesis tests run, I want, for each test, to collect a few random examples that were used for that test, and demonstrate them via a gui. So for example if I have two tests def test_a(a: int) and def test_b(b: str), then after it has finished testing all the examples for test_a, I want it to sample 5 random examples, create a gui window that visually represents those examples, blocks until I close the window, then continues on to test_b, where it does the same basic thing. Also, if an exception is caught during a test, the gui pops up immediately for that test based off the examples that have been seen so far.
I have awkwardly implemented this with execute_example, by having each test change an essentially global field curDesc, then execute_example checks whether or not that's equal to lastDesc, and if there's a change it shows the gui (which blocks). But this is ugly and has several problems, like there winds up being multiple attempts to produce the gui for a single test, and also the final test won't ever produce a gui this way:
179 # It adds the scenario returned by the test into a collection of scenarios
180 # to be used for a GUI demo, and displays some of the related scenarios whenever
181 # the test changes or an exception occurs.
182 def execute_example(self, fn):
183 def viewScens(viewDesc):
184 scens = self.scenarios[viewDesc]
185 if scens:
186 numScens = min(NUM_VIZ, len(scens))
187 print(f"showing {numScens} '{viewDesc}' scenarios")
188 testGUI = TestGUI(random.sample(scens, numScens))
189 testGUI.run()
190 self.curDesc = None
191 self.curScen = None
192 try:
194 ret = fn()
196 if self.curDesc != self.lastDesc:
197 if self.curDesc in self.seenDescs:
198 # PROBLEM: this happens several times per test
199 print(f"'{self.curDesc}' has been seen before")
200 # PROBLEM: this won't ever be called for the last test!
201 viewScens(self.lastDesc)
202 return ret
203 except Exception as e:
204 # UnsatisfiedAssumption is used to break out of an example where the preconditions aren't met
205 if not isinstance(e, UnsatisfiedAssumption):
206 print(f"exception with '{self.curDesc}'")
207 viewScens(self.curDesc)
208 raise e
209 finally:
210 self.seenDescs.add(self.curDesc)
211 if isinstance(self.curScen, TestScenario):
212 self.scenarios[self.curDesc].append(self.curScen)
213 self.lastDesc = self.curDesc
Surely there must be a better way?
Or failing that, is there some way to perform an action after all hypothesis tests have run? Or at least all tests in a specific class or file? Why is hypothesis running a bunch of examples for test_a, then resetting the state, then doing more examples for test_a? Note that the issue with seeing the same desc multiple times occurs when there is only one test! The only time lastDesc is set to None is in __init__, so for the curDesc to change more than once implies that the test class is being init'd multiple times just to test one test inside of it, which seems weird. How is this supposed to work?