Catch all requests and proxy a subset to a different server

Question

I would like to proxy all requests matching a certain criteria to my backend server. Is there any way to catch all requests and let a subset through the proxy based on things like headers and request paths?

vite.config.ts

server: {
    proxy: {
      '^.*': {
        target: 'https://backend.com',
        secure: false,
        changeOrigin: true,
        ws: true,
        configure: proxy => {
          // only proxy if request has header "accept: application/json"
          // or if it's an image request
        }
      }
    }
  }

This seems to catch all requests, but I want my "static" files like index.html, app.js etc. to be served from the devserver, not the proxy.