Expo updates custom server - New bundles fetched but not downloaded

Question

I recently setup a custom expo updates server (based on the one provided by expo themselves - https://github.com/expo/custom-expo-updates-server). The only things I changed in the expo-updates-server folder is the HOSTNAME in .env.local and set it to a my domain name so the urls will look like this: https://www.test.co/api/manifest

This is the updates and runtimeVersion part inside of my app.json file
(copy paste from the github I linked)

"runtimeVersion": "app",
"updates": {
  "enabled": true,
  "url": "https://test.co/api/manifest",
  "fallbackToCacheTimeout": 30000,
  "codeSigningCertificate": "./code-signing/certificate.pem",
  "codeSigningMetadata": {
    "keyid": "main",
    "alg": "rsa-v1_5-sha256"
  }
},
"assetBundlePatterns": ["**/*"],

If I leave it like this, the /api/manifest endpoint is triggered with all the /api/assets ones, but the ui does not change with the new bundle. Every time I kill and relaunch the app, the endpoints are called but nothing happens.

I tried to debug by adding "checkAutomatically": "ON_ERROR_RECOVERY" to updates inside of app.json to avoid the app fetching at startup and then I added this code inside of my App.jsx

useEffect(() => {
  if (!__DEV__ && Platform.OS !== 'web') {
    checkForUpdates();
  }
}, []);

async function checkForUpdates() {
  try {
    const update = await Updates.checkForUpdateAsync();
    if (update.isAvailable) {
      await Updates.fetchUpdateAsync();
      alert(JSON.stringify(fetch));
      await Updates.reloadAsync();
    }
  } catch (err) {
    alert(err);
  }
}

The alert(JSON.stringify(fetch)) is only for test purposes.
I wanted to see what is returned from the function.

With that code, every time I kill and restart the app, the alert shows up but nothing happens. The content returned from JSON.stringify(fetch) is: (changed for security reason) The big black square contains security keys, so its not needed to be shown.

Note: All the urls are accessible, They are downloadable from a browser. Only the bundle/ios-xxxxxxx.js file is not downloadable, the content of the file is shown on the browser.

Assets url are in the format: https://test.co/api/assets?asset=apps/```runtimeVersion```/```date```/assets/xxxxxxxxxx?runtimeVersion=x&platform=ios

Thanks to anyone who can help me fix this as it is really important for me and for my business. The app is sometimes crashing on clients' devices.