Rationale behing using Lighthouse as github CI

Question

I am trying to understand how running lighthouse as CI helps me. What I understood is on every PR lighthouse will run on my application that runs on a localhost. But my end users are using the prod environment which is drastically different from localhost and hence the performance scores will be different too. Also i wont be monitoring the exact experience what the end users will be having. An angular app runs on JIT when running on localhost (dev) and turns on the AOT only in prod env. How are we expecting lighthouse CI to give us exact performance scores. Seeking some counter suggestions.

Answer 1

It's true the Lighthouse Performance score is subject to high variability and that development environment can be quite different to the production environment.

Lighthouse gives a lot more info than just the Performance Score. The other audits can pick up performance best practices (e.g. ensuring things like image width and heights are set, JavaScript file sizes...etc.). So there is plenty of value in running Lighthouse even though the score is different. In fact in the past I have advocated ignoring the time-related scores completely due to their variability.

However, the performance metrics can also be useful to catch regressions, provided you can deal with the variability (e.g. have dedicated runners of similar specs, and rerunning multiple times and picking the median run). Think of the score more as a comparative number for multiple similar runs, rather than an absolute score.

Finally, you are right that any lab-based performance test may, or may not, be representative of how real users experience your website. As per above, Lighthouse can still catch potential performance problems or regressions but it does not guarantee good performance ultimately post-release. You should measure field data too, and understand how representative your Lighthouse runs are.

Put it this way, code linters are often for identifying code best practices rather than necessarily spotting issues that will cause performance bugs. But that doesn't mean there is no value in linters. I think of Lighthouse as a performance linting tool, rather than absolute measure of a performance score to hit.