Autodesk Forge - 504 Gateway Timeout when uploading file as chunks to BIM 360 storage in Node JS

Question

I'm getting a 504 Gateway Timeout when I try to upload a large file in "chunks" using the PUT object resumable API endpoint in Autodesk forge.

https://forge.autodesk.com/en/docs/data/v2/reference/http/buckets-:bucketKey-objects-:objectName-resumable-PUT/

My console log output is below, showing the 504 response, along with the PUT request I send.

I've tried various methods, both using the functions in the forge-apis SDK in Node, (eg uploadChunk method from ObjectsAPI) and also written various functions using request, request-promise-native, to check if using await/async, promises vs callbacks can help solve the problem.

In all cases, I get the same response - a long pause in the log and then a 504 Gateway Timeout - which appears to be coming from the Forge end.

For context, my Node JS application sits on Heroku, and is triggered from a python script running locally on the client - so Heroku is the 'middleman' - authentication is passed from client-side via python to the node app and then passed on to Forge to authenticate. Other operations in the flow are successful so I'm confident authentication is working correctly.

Can anyone see anything from my logs below that could be causing the problem? Anybody else had similar 504 issues with chunked uploads of large files? (~130MB in this case)

The first log shows my console log output - I'm sending 5mb chunks at 1-second intervals, and sending the chunk as a Buffer object (bytes) sliced to match the Content-Range

2020-06-22T10:42:33.389835+00:00 app[web.1]: Ready to upload chunk...
2020-06-22T10:42:33.389896+00:00 app[web.1]: simulating waiting for 1000 milliseconds
2020-06-22T10:42:34.391034+00:00 app[web.1]: done waiting
2020-06-22T10:42:34.392893+00:00 app[web.1]: contentRange bytes 129999974-130056191/130056192
2020-06-22T10:42:34.393393+00:00 app[web.1]: requestParams {
2020-06-22T10:42:34.393394+00:00 app[web.1]: headers: {
2020-06-22T10:42:34.393395+00:00 app[web.1]: Authorization: 'Bearer eyJhbGciOiJIxxxxxxxxx',
2020-06-22T10:42:34.393396+00:00 app[web.1]: 'Content-Type': 'application/octet-stream',
2020-06-22T10:42:34.393396+00:00 app[web.1]: 'Content-Range': 'bytes 129999974-130056191/130056192',
2020-06-22T10:42:34.393397+00:00 app[web.1]: 'Content-Length': '130056192',
2020-06-22T10:42:34.393397+00:00 app[web.1]: 'Session-Id': '-75601742'
2020-06-22T10:42:34.393397+00:00 app[web.1]: },
2020-06-22T10:42:34.393398+00:00 app[web.1]: uri: 'https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable',
2020-06-22T10:42:34.393399+00:00 app[web.1]: url: 'https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable',
2020-06-22T10:42:34.393399+00:00 app[web.1]: method: 'PUT',
2020-06-22T10:42:34.393400+00:00 app[web.1]: body: <Buffer 6b b1 b4 e4 d7 f5 1e 59 a0 07 0c db 68 7d 9e 98 75 8e 5f fc e8 0c e3 78 a0 9c b9 59 3f e9 0f 28 e5 5a f0 eb 75 7b 68 16 3d 4c c1 ca 20 7a ba 67 f0 69 ... 56167 more bytes>,
2020-06-22T10:42:34.393400+00:00 app[web.1]: resolveWithFullResponse: true
2020-06-22T10:42:34.393400+00:00 app[web.1]: }
2020-06-22T10:42:34.393466+00:00 app[web.1]: Ready to upload chunk...

The response comes back like this for each chunk:

2020-06-22T10:43:34.443453+00:00 app[web.1]: Uploading 504 > GATEWAY_TIMEOUT
2020-06-22T10:43:34.443834+00:00 app[web.1]: err:  null
2020-06-22T10:43:34.443969+00:00 app[web.1]: res:  {
2020-06-22T10:43:34.443969+00:00 app[web.1]: ----"statusCode": 504,
2020-06-22T10:43:34.443970+00:00 app[web.1]: ----"body": "",
2020-06-22T10:43:34.443970+00:00 app[web.1]: ----"headers": {
2020-06-22T10:43:34.443971+00:00 app[web.1]: --------"content-length": "0",
2020-06-22T10:43:34.443971+00:00 app[web.1]: --------"connection": "Close"
2020-06-22T10:43:34.443971+00:00 app[web.1]: ----},
2020-06-22T10:43:34.443972+00:00 app[web.1]: ----"request": {
2020-06-22T10:43:34.443972+00:00 app[web.1]: --------"uri": {
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"protocol": "https:",
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"slashes": true,
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"auth": null,
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"host": "developer.api.autodesk.com",
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"port": 443,
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"hostname": "developer.api.autodesk.com",
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"hash": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"search": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"query": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"pathname": "/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable",
2020-06-22T10:43:34.443976+00:00 app[web.1]: ------------"path": "/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable",
2020-06-22T10:43:34.443976+00:00 app[web.1]: ------------"href": "https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable"
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------},
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------"method": "PUT",
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------"headers": {
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Authorization": "Bearer eyJhbGciOiJxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Type": "application/octet-stream",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Range": "bytes 129999974-130056191/130056192",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Length": "130056192",
2020-06-22T10:43:34.443979+00:00 app[web.1]: ------------"Session-Id": "-75601742"
2020-06-22T10:43:34.443979+00:00 app[web.1]: --------}
2020-06-22T10:43:34.443979+00:00 app[web.1]: ----}
2020-06-22T10:43:34.443979+00:00 app[web.1]: }

Answer 1

EDIT: when using the resumable upload feature, make sure that the Content-Length request header specifies the length of just the chunk you're sending, not the entire size of the uploaded file.

The Forge APIs do have some rate throttling but in that case you should probably get a 4xx status code, not 504.

Can you try a couple of things for us?

• upload the file using this Node.js library: https://github.com/petrbroz/forge-server-utils
• upload the file using this Visual Studio Code extension (using the same Node.js library under the hood): https://marketplace.visualstudio.com/items?itemName=petrbroz.vscode-forge-tools
• upload your file from your application locally instead of from Heroku (just to rule out that it's some networking issue between Heroku and Forge servers)

And if you're still unable to upload the file, could you please share it with us (confidentially) via forge (dot) help (at) autodesk (dot) com? We would be happy to investigate it on our end.

Answer 2

Thanks to my colleague Luis Felipe Paris - we managed to resolve the problem, after looking at forge-server-utils as per Petr's advice.

By changing our uploadChunk function to use the module axios instead of request or request-promise - we found the 504 timeout error disappeared.

We were using the npm module request and request-promise (even though legacy / deprecated these days) as it supports piping streams from one place to another, and this was working with all other requests we were doing, except uploading a 'resumable' file in separate chunks.

We had to convert the file payload we were sending from a Buffer object to an ArrayBuffer, and slice it correctly to match the Content-Range of each chunk.

Some of the older forge examples use request - so I guess this might be useful for other users - if request gives you problems, try axios