A known issue with WIX is that e.g. the side-banner image when looking good in 100% DPI setting scales badly when the system is at 125%, 150% or whatever setting.
I tried changing the side-banner image to be 150% the size to see if it scales down better then up, but it didn't. On 150% it now looks great, but terrible still on 125% and 100%.
Can the wix script be fiddled with so that it makes the installer choose different image files with different sizes depending on the DPI setting or in some other way determine the size of the dialog and pick the most appropriately sized side-banner image?
That way I would cover 100%, 125% and 150% which are commonly used.
There is no such thing as "wix script". WiX is declarative XML that transforms into declarative Windows Installer table data. There is nothing imperative / procedural / scripting about it.
All UI limitations are in the underlying Windows Installer and not related to WiX at all. I'm aware of many limitations but I've never heard of the one you mention. I can't reproduce it on my installs. Can you provide screen shots of what you are seeing?
It is possible to have multiple image controls and use control conditions to make them mutually exclusive. But I'm not convinced this is needed in this situation. Also if UI polish is important to you it's probably time to consider rolling an external UI handler instead of using native UI.