I would like to benchmark my rustcode with criterion, but the information it presents to me do not match up. Here is some of the output I get
Benchmarking big/big decoding: Warming up for 1.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 306.4s.
big/big decoding time: [632.40 ns 724.40 ns 899.80 ns]
change: [+0.7649% +16.645% +46.889%] (p = 0.04 < 0.05)
Change within noise threshold.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
If it took a criterion only 724.40 ns to run the code, why couldn’t it run the code 10 times in 5 seconds. I looked at the outliers in the browser and the one outlier took 1.5 microseconds to run so this could not be the problem.
Here is the benchmark code:
fn big_benchmark(c: &mut Criterion) {
let file = std::fs::File::open("../../test_big_file").expect("could not open file : ( ");
let mut output_file =
std::fs::File::create("../../test_big_file_normal.txt").expect("Could not create file");
let mut group = c.benchmark_group("big");
group
.sample_size(10)
.warm_up_time(std::time::Duration::from_secs(1))
.measurement_time(std::time::Duration::from_secs(5));
let mut base = Decoder::new(file, output_file, EncoderType::Lossy);
group.bench_function("big decoding", |b| b.iter(|| base.decode().unwrap()));
group.finish();
}