Deploy Azure function app with private endpoint results gateway error

Question

I am trying to do ZIP deployment of azure function app that has private endpoint enabled. As a part of this , I created azure resources as follow:

• Function app with private endpoint enabled and disabled Allow public access on.
• Storage account with private endpoint enabled, disabled public access and associcate with function app. I make sure private IP is added to Private DNS Zone A record.
• Created VNET with 2 subnets such as InboundSubnet, OutboundSubnet. I associated same inbound subnets while creating azure function app, storage account. For Azure VNET integration outbound traffic , I used OutboundSubnet.

Since inbound traffic for function , storage is cut off with internet access, I published my ZIP to a storage account(Newly created and public enabled) via build(CI) pipeline and fetching the same ZIP file in release pipeline to deploy.

However I am getting below issues:

The gateway did not receive a response from 'Microsoft.Web' within the specified time period

Encountered an error (InternalServerError) from host runtime. - From aztivity log

Update: - Ran CLI command from local machine

The command failed with an unexpected error. Here is the traceback: HTTPSConnectionPool(host='demo-funcapp-test.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/publish?type=zip&async=false (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x05EA4B50>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

NSG Inbound Rules:

App settings of azure function app.

[
  {
    "name": "AzureWebJobsStorage",
    "value": "DefaultEndpointsProtocol=https;AccountName=oshpocrg8094;AccountKey=STORAGEACCOUNTKEY;EndpointSuffix=core.windows.net",
    "slotSetting": false
  },
  {
    "name": "FUNCTIONS_EXTENSION_VERSION",
    "value": "~4",
    "slotSetting": false
  },
  {
    "name": "FUNCTIONS_WORKER_RUNTIME",
    "value": "dotnet",
    "slotSetting": false
  },
  {
    "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
    "value": "DefaultEndpointsProtocol=https;AccountName=oshpocrg8094;AccountKey=STORAGEACCOUNTKEY;EndpointSuffix=core.windows.net",
    "slotSetting": false
  },
  {
    "name": "WEBSITE_CONTENTOVERVNET",
    "value": "1",
    "slotSetting": false
  },
  {
    "name": "WEBSITE_CONTENTSHARE",
    "value": "test-pvent-func",
    "slotSetting": false
  },
  {
    "name": "WEBSITE_RUN_FROM_PACKAGE",
    "value": "1",
    "slotSetting": false
  },
  {
    "name": "WEBSITE_TIME_ZONE",
    "value": "UTC",
    "slotSetting": false
  }
]

Release Pipeline:

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI : Deploy file to Function App'
  inputs:
    azureSubscription: 'Visual Studio Enterprise Subscription (44444c7b-1f7a-43a1-a90f-dee45a2f6262)'
    scriptType: ps
    scriptLocation: inlineScript
    inlineScript: |
      az extension add --name webapp
     
     $ZIP_URL = (az storage blob generate-sas --full-uri --permissions r --expiry 2023-12-31T23:59:59Z --account-name publicsgtest -c functionzipfiles -n build.zip | Out-String).Trim()
     
     Write-Host 'ZIP File URL:' $ZIP_URL
     
     az webapp deploy --name demo-funcapp-test --resource-group osh-poc-rg --type zip --src-url $ZIP_URL --async false

Partial release logs:

2023-12-04T15:21:37.5470046Z [command]C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\azureclitaskscript1701703229248.ps1'"
2023-12-04T15:21:55.3254967Z WARNING: The installed extension 'webapp' is in preview.
2023-12-04T15:22:04.8371024Z WARNING: 
2023-12-04T15:22:04.8372046Z There are no credentials provided in your command and environment, we will query for account key for your storage account.
2023-12-04T15:22:04.8374096Z It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.
2023-12-04T15:22:04.8374775Z 
2023-12-04T15:22:04.8375927Z You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
2023-12-04T15:22:04.8377688Z For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.
2023-12-04T15:22:04.8378502Z 
2023-12-04T15:22:04.8379753Z In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
2023-12-04T15:22:06.1828417Z ZIP File URL: "https://publicsgtest.blob.core.windows.net/functionzipfiles/build.zip?SASTOKENATTACHED"
2023-12-04T15:23:15.3930471Z ERROR: Gateway Timeout({"error":{"code":"GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.Web' within the specified time period."}})
2023-12-04T15:23:15.7272696Z ##[error]Script failed with exit code: 1
2023-12-04T15:23:15.7438810Z [command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" account clear"
2023-12-04T15:23:17.0799266Z ##[section]Finishing: Azure CLI : Deploy file to Function App

Reference link: https://azure.github.io/AppService/2021/03/01/deploying-to-network-secured-sites-2.html

What am I missing?

Answer 1

Not sure, what is the exact issue, this is log I got from support team.

Error: The gateway did not receive a response from Microsoft.Web within the specified time period

Root cause of Error:: Your application process failed to start up in this time frame because the process could not access the storage account which hosted your application content from the subnet with which the application was integrated using regional virtual network integration. This can most likely occur due to the network configurations on the subnet or the firewall rules of the storage account. Please disconnect and re-connect the regional virtual network integration first and then check if the application can start up. If the problem still exists, you need to check the network settings to allow SMB file access from the subnet to the storage account.

Fix: As a workaround, I did following steps that makes it working for me.

1. Created azure function app, storage account as public access and deployed using Azure Devops pipeline which is success. Here I deploy with only 1 Azure function say Function1 which I can see in azure portal.

2. Disabled public access for function app , storage and created VNET, Private endpoints, azure private dns zone etc., required resources to make it fully private access.

3. Created another function http trigger say Function2 in VS code and deployed through devops build and release pipelines. I can see Function2 in azure portal.

Release Pipeline YAML code:

I am still trying my luck to do directly with private endpoints.

Answer 2

Which build agent are you using? Consider adding AzureDevOps service tag in your NSG rules if you try deploying from the ms-hosted build agent because it has no access to your private resources. Or you may create VM and register a new build agent in your private net.