Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major bug in HttpTrigger receival of payload #607

Closed
verschaevesiebe opened this issue Jul 6, 2022 · 6 comments · Fixed by Azure/azure-functions-nodejs-library#9
Closed
Assignees
Labels
Milestone

Comments

@verschaevesiebe
Copy link

verschaevesiebe commented Jul 6, 2022

Hi All,

After some investigation I'm having trouble with Azure functions.
This is a simple Azure function which receives a HTTP Trigger call with an application/json Content-Type.

The function works perfectly fine for all invocation as always.
However, we have added an additional field to our JSON payload which is "bytes": 1253.
To give some small context, the bytes field is just the length of a file we have passed to another system. So everything in this event is text or integer. When passing this to the function

When the addition "bytes" field is added to the payload, the function invocation/function app crashes and throws the following error.

System.Private.CoreLib: Exception while executing function: Functions.function_sanitization_updated. System.Private.CoreLib: node exited with code 1 (0x1). LanguageWorkerConsoleLog[error] Worker fe1bae70-3626-4feb-ade8-f446f7a943e1 uncaught exception (learn more: https://go.microsoft.com/fwlink/?linkid=2097909 ): TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received type number (2)     at new NodeError (node:internal/errors:371:5)     at Function.from (node:buffer:323:9)     at t.fromTypedData (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:18531)     at o (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:17646)     at o (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:17778)     at t.getNormalizedBindingData (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:18069)     at new d (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:11664)     at t.CreateContextAndInputs (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:10563)     at C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:26140     at Generator.next (<anonymous>)     at C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:25022     at new Promise (<anonymous>)     at r (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:24767)     at t.invocationRequest (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:25336)     at l.<anonymous> (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:39292)     at l.emit (node:events:526:28)     at addChunk (node:internal/streams/readable:315:12)     at readableAddChunk (node:internal/streams/readable:289:9)     at l.Readable.push (node:internal/streams/readable:228:10)     at Object.onReceiveMessage (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:82321)     at Object.onReceiveMessage (C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:74609)     at C:\Users\Work\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist\src\worker-bundle.js:2:48750     at processTicksAndRejections (node:internal/process/task_queues:78:11)

Do mind the Received type number (2). Referring to the bytes field (also number 2) in the example below.
If we change the value of the bytes field to a string like so "bytes": "test" then the invocation does not crash and doesn't throw an error.

{
  "event_id": "04966b69-569b-4345-9d52-608e9fe82ccb",
  "event_type": "TEST",
  "event_creator": "TEST",
  "timestamp": "2022-07-04T17:13:25.211Z",
  "payload": {
    "dossier_id": "TEST_DOS",
    "test": {
        "bytes": 2
    }
  }
}

I believe that something in the function source code checks for a "bytes" text and tries to parse it ?
The issue only occurs when the "bytes" field is at least 3 levels deep into the JSON object.
It's unbelievable and frustrating to have to be looking for this bug 🗡️ .

Hope someone can investigate and possibly resolve it.

PS: We're using typescript and it is being decompiled into JavaScript.
The function packages i've been using to test this replication are
[email protected]
@azure/functions": "^1.2.3
@azure/functions": "^3.2.0

  • npm i
  • tsc
  • func start ...
{
    "statusCode": 500,
    "headers": {
        "Request-Context": "appId=cid-v1:54e7f4c3-3fb0-4c1e-951d-da13708c851c",
        "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
        "Date": "Wed, 06 Jul 2022 09:24:12 GMT",
        "Content-Length": "0"
    }
}

Above is the response from the function that I've found somewhere on Azure so unsure if it's the correct request-context id for someone to take a look.

@verschaevesiebe
Copy link
Author

@surgupta-msft feel free to email me directly in case you want to see this in action.
I'm 100% sure that this is a bug that we cannot work around.

Thank you.

@surgupta-msft
Copy link

@verschaevesiebe Thank you for bringing up the issue. I was able to reproduce it.

Repro Steps -

  1. Create a Node app from Azure portal.
  2. Create Http trigger.
  3. Invoke Http trigger with request body as below and you will get 500 Internal Server Error.
  4. Update body to make it string - "bytes" : "2" and will get 200OK.
{
  "name": "Azure",
   "payload": {
    "dossier_id": "TEST_DOS",
    "test": {
        "abc" : "2",
        "bytes" : 2
    }
  }
}

It seems to be happening with Node apps. I tried repro with powershell and dotnet apps but did not see this issue. cc @AnatoliB

Repro app: surg-node-test6

Pasting stack trace below -

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.HttpTrigger1 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure
Exception: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received type number (2)
Stack: TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received type number (2)
    at new NodeError (node:internal/errors:371:5)
    at Function.from (node:buffer:322:9)
    at t.fromTypedData (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:20492)
    at o (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:19607)
    at o (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:19739)
    at t.getNormalizedBindingData (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:20030)
    at new d (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:11664)
    at t.CreateContextAndInputs (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:10563)
    at p.<anonymous> (C:\Program Files (x86)\SiteExtensions\Functions\4.7.2\workers\node\dist\src\worker-bundle.js:2:28433)
    at Generator.next (<anonymous>)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 96
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at /_/src/WebJobs.Script/Description/FunctionGenerator.cs : 225
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

@AnatoliB AnatoliB transferred this issue from Azure/azure-functions-host Jul 14, 2022
@AnatoliB AnatoliB removed their assignment Jul 14, 2022
@ejizba ejizba added this to the July 2022 milestone Jul 22, 2022
@ejizba
Copy link
Contributor

ejizba commented Jul 22, 2022

Hi @verschaevesiebe I can also reproduce the bug and your analysis appears to be correct. It's only an issue with fields specifically named "bytes" and it's because we try to parse that field into a Buffer (which fails when your field is a number). We will get this fixed - but in the meantime unfortunately the only workaround I'm aware of is to rename your "bytes" field to something else.

@ejizba ejizba assigned ejizba and unassigned surgupta-msft Jul 22, 2022
@ejizba ejizba modified the milestones: July 2022, August 2022 Jul 22, 2022
@verschaevesiebe
Copy link
Author

verschaevesiebe commented Jul 25, 2022

Hi @ejizba,

Thanks, that's indeed my workaround for now. Naming it "bytess" 😄
I see you've added it to the August milestone, any idea when i can expect a runtime fix?

The unfortunate part is that we're receiving these events from external sources so we're applying a translation now through a logic app before being able to forward it to any function, because they all crash and restart 😢

Thanks for the triage and investigation 👍🏽

@ejizba
Copy link
Contributor

ejizba commented Jul 25, 2022

because they all crash and restart

I made some changes a little bit ago to prevent random errors from crashing the whole app. This was included in host version 4.7.2 which just finished rolling out in Azure this month. Core tools has not shipped with the changes yet, but that should happen in the next week or two. This means the specific invocation may fail in Azure, but you should no longer see it crash the app as a whole.

I see you've added it to the August milestone, any idea when i can expect a runtime fix?

We will likely merge the fix in August, but it can take a month or two to rollout, so September in an ideal scenario? As an aside, the delay for bug fixes like this is a known pain point and we're working to improve it. Specifically, we're moving a lot of our logic into an npm package instead of shipping it all in Azure. In that scenario, we could fix the bug and ship a new npm package within weeks of discovering the bug and all you'd have to do is update your package reference. Tracked by #568 which I expect to rollout in the September/October timeframe as well.

ejizba added a commit to Azure/azure-functions-nodejs-library that referenced this issue Aug 25, 2022
Main points:
- We don't want `toCamelCase` to call `fromRpcTypedData` because it traverses an entire object recursively and only the top level objects should be RpcTypedData
- We will remove triggerMetadata from http/timer triggers. That information is best found on the request/timer objects instead

Related to all of these bugs:
Azure/azure-functions-nodejs-worker#607
Azure/azure-functions-nodejs-worker#274
Azure/azure-functions-nodejs-worker#204
Azure/azure-functions-nodejs-worker#388
@ejizba
Copy link
Contributor

ejizba commented Dec 5, 2022

Hi @verschaevesiebe sorry this took way longer than I was originally hoping, the fix finally finished rolling out in Azure as a part of host v4.14.0. Thanks for reporting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants