@azure/storage-blob@>12.2.0 throws Bad Request - Invalid Verb on upload
#11187
Assignees
Labels
bug
This issue requires a change to an existing behavior in the product in order to be resolved.
Client
This issue points to a problem in the data-plane of the library.
customer-reported
Issues that are reported by GitHub users external to the Azure organization.
needs-team-attention
Workflow: This issue needs attention from Azure service team or SDK team
Storage
Storage Service (Queues, Blobs, Files)
Comments
ghost
added
needs-triage
JacksonKearl
pushed a commit
to microsoft/vscode
that referenced
this issue
Sep 10, 2020
jeremymeng
added
Client
ghost
removed
the
needs-triage
ghost
added
the
needs-team-attention
|
Oops, I thought it was something wrong with our live tests pipeline. |
|
If you can recreate it easily, could you turn on the log and share me the logs specially the requestId via import { setLogLevel } from "@azure/logger";
setLogLevel("info");No worry if you can't recreate it. I will look into it. |
|
I still can't recreate it even with the live tests pipeline now with #11195. And the change for uploadStream in REST API level is only switching from @JacksonKearl Could you kindly share the log if you can recreate it? |
|
I am experiencing the same error upon installing @azure/storage-blob v12.2.0, which goes away after downgrading to v12.1.0. |
|
I wonder if it is related to a change in the latest @azure/core-http v1.1.8. @jonathantzh is it possible for you to try pinning @azure/core-http 1.1.7 while using @azure/storage-blob v12.2.0? |
|
@jeremymeng Pinning @azure/core-http to v1.1.7 while using @azure/storage-blob v12.2.0 does not resolve the issue for me |
|
I have the same problem |
|
Tried reproducing with the logging but now I'm getting a different error, not sure if it's related or not: |
|
Also, I first encountered this in a GitHub action -- https://github.com/microsoft/vscode/runs/1099066807?check_suite_focus=true, which is running on linux. |
|
For those who can recreate it, could you turn on the log and share me the logs via import { setLogLevel } from "@azure/logger";
setLogLevel("info");I'd like to figure out the root cause instead of simply roll back. |
|
And for anyone who encounter this issue, if v12.1.0 isn't enough for you, please let me know and we will release a hotfix to undo the uploadStream change but with all other new features and bug fixes if we can't find the right fix in time. |
@JacksonKearl Probably unrelated. |
jeremymeng
added a commit
to jeremymeng/azure-sdk-for-js
that referenced
this issue
Sep 11, 2020
|
I try to run the live tests on using Node v12 on Linux/Windows/MacOS. So far the only failures are on MacOS the failed tests is the same as the first failure in #11163. I couldn't repro the second, Invalid Verb, failure though |
|
Experiencing the same, but not on only on 12.2, on 12.0.0 and 12.1.0 as well. Seems that the issue is not relative directly to the storage sdk. |
When pinning to core-http 1.1.7, could you please also check your package-lock.json to verify that core-http 1.1.8 is not pulled in? |
|
@jeremymeng Tried deleting package-lock.json and node_modules folder in @azure/[email protected] package, pinned @azure/core-http version to v1.1.7 in package.json, ran npm install and verified version using npm list @azure/core-http with result: /@azure/storage-blob |
|
@ljian3377 , Don't know if it's still relevant, but happens for me on Windows. Will attach my logs. I also ran some other versions,
|
|
@patrikmolsson @jeremymeng |
|
@ljian3377 so the failing request didn't even reach service code, and looked like was rejected by IIS? It's weird to get invalid http verb error as we used PUT as logs showed. Maybe something else caused the error. We don't log more lower level info than shown above in core-http. We redact sensitive info, and do not log bodies. One of the other easy ways to is to use fiddler as a proxy and allow fiddler to decrypt https traffic so we can inspect raw requests and responses. https://www.donovanbrown.com/post/How-to-debug-your-Nodejs-app-with-Fiddler |
|
https://httptoolkit.tech/javascript seems useful for intercepting HTTP(S) traffic. Never used it though. |
|
Good news. Successuflly recreated it locally. Will look into the Fiddler tracing. |
|
I see the failure in #11163 in my custom runs which is also the same invalid verb failure. I logged requests, however, the logs didn't show anything unusual at first glance https://dev.azure.com/azure-sdk/internal/_build/results?buildId=535651&view=logs&j=34d05327-3286-5fd0-44d3-76aebff5d924&t=d726e143-6308-5465-2a45-ec34d6ea866c |
jpda
added a commit
to jpda/vscode
that referenced
this issue
Sep 14, 2020
* Add back hideHover and use on tree context menu show Fixes microsoft#106268 * Update distro * 💄 * explorer: Fix TrustedTypes violation microsoft#106285 * produce deb, rpm packages * Add loginShell (microsoft/vscode-remote-release#3593) * chore - tweak onDidAddNotebookDocument, onDidRemoveNotebookDocument event, use ResourceMap and fix confusion between models and editors * notebook update * pinned tabs - update setting enum name * Use innerText over innerHTML, microsoft#106285 * rename to IHostColorSchemeService * API proposal for tree item icon color Part of microsoft#103120 * chore - when target might be undefined use `target?.dispose()` over `dispose(target)` * deprecate onDidChangeCells * Reenable notebook smoke test microsoft#105330 * deprecate onDidChangeContent * Add numeric values support for terminal.integrated.fontWeight * unified onDidChangeContent * [email protected] Fixes microsoft#105957 * 💄 * debt - remove _unInitializedDocuments * remove `NotebookDocument#displayOrder` , fixes microsoft#106305 * no uninitialized documents. * chore - update references viewlet * debug: make serverReadyAction play nicely with js-debug Fixes microsoft#86035 Fixes microsoft/vscode-js-debug#440 * fix rpm * high contrast switching in browser * Fix occasional bad custom selectbox layout Fix microsoft#106302 * review comments * Bump vscode-ripgrep for ARM microsoft#6442 * Revert more specific class names for editor tokens Reverts microsoft#103485 Fixes microsoft#106261 I believe that microsoft#103485 broke cases where the markdown renderer creates tokenized strings that are used outside of an editor's dom node (such as in hovers or in webviews) The safest fix for now is to revert it. We can revist this and make the markdown renderer handle the tokenized output better next iteration * remove emit selections. * merge conflict resolve. * fix integration tests. * Disable errors in non-semantic supported files Fixes microsoft#106299 Fixes microsoft#106314 Also enables js/ts features on the right side of PRs and in search results * proper fix for microsoft#105202 (microsoft#106293) * Only enable 'open with' on files Fixes microsoft#106291 * Add WebviewView.description Allow configuring the description for webview views. This is rendered next to the title in a less prominently in the title * Remove manual strikethroughs for deprecated properties in vscode.d.ts Now that TS has support for `@deprecated`, these manual strike throughs should no longer be required. * Add show method to webview view Fixes microsoft#106085 * Skip failing test * fix microsoft#106283 * enable test * fix microsoft#106283 * pinnedTabSizing.standard => pinnedTabSizing.normal * install dot net core sdk * update distro * tabs - align icon and text vertically centered in tab * update distro * distro * fix microsoft#106308 * Update gitignore decorations when .git/info/exclude file is edited (microsoft#106270) * detect local `exclude` file edits * use `uri.path` to detect exclude file edits `uri.path` uses forward slash as a path separator indepentent of the host system, which makes it easier to use with regex * updated searches * editor - rename context keys variables * fix microsoft#105999 * pinned tabs - prevent to close pinned tabs via Cmd+W (microsoft#100738) * Reduce usage of `.innerHTML` (microsoft#106285) * fix uninstalling extension * remove unused import * add `replaceNotebookMetadata` (should become `replaceMetadata`) to NotebookEditorEdit, microsoft#105283 * add `replaceNotebookMetadata`, microsoft#105283 * use textContent instead of innerHTML (for microsoft#106285) * chore - move appyWorkspaceEdit from extHostTextEditors to (new) extHostBulkEdits and mainThreadBulkEdits * chore - extract extHostNotebookDocument for the NotebookDocument and NotebookCell types and friends * chore - extract ExtHostNotebookEditor into its own file * chore - remove ExtHostNotebookEditor#uri and use document.uri instead * chore - 💄 member order: property, ctor, method * publish arm deb and rpm * trusted types related to microsoft#106285 * use async await * distro * update trusted types search * trusted types - use textContent for style elements, fyi @rebornix * fix arch * Fix compile after merge * Use instantiation service to create TerminalLinks * Consolidate colon trim logic * Avoid slice when checking colon * Check length again I prefer chatAt over slice as it's more obvious what's happening * Move comment into helper function * Update extensions/git/src/commands.ts * Update extensions/git/src/commands.ts * Update extensions/git/src/commands.ts * Save prompt is shown while saving from settings split json editor (fix microsoft#106330) * Only allow configurePlugin against main TS Server Fixes microsoft#106346 Looks like the TS Server doesn't support this in partial mode at the moment * Adding more explicit typings for promises This gets us ready for TS 4.1 * Don't use async on abstract functions * chore - use workingCopyService.isDirty instead notebook.isDirty * Update Codicons - Update 'pinned' - Add 'export' - Compress 'merge' microsoft/vscode-codicons@5bcb1a0 * Add explicit undefined parameters / types These errors come from the new stricter types for Promises in TS 4.1 * debt - IMainNotebookController#removeNotebookDocument * debt - invoke resolve notebook when opening a notebook in an editor, not when loading a notebook from source * do not need isUntitled. * 💂 polish nb tests. * remove selections from nb text model. * replace changeCellLanguage to applyEdit * fix microsoft#105916. expand metadata section if modified. * move dirty state to NotebookEditorModel. * chore, simply notebook text model event emitter * refs microsoft#106285 * Add subscribers action * Fix terminal ts 4.1 compile issues Part of microsoft#106358 * Fix ts 4.1 issues in terminal api tests * Update Codicons: add 'graph-left' microsoft/vscode-codicons@dd1edb2 * initialize notebook text model data only through ctor. * 💄 * Mark property readonly * Enable webview commands for webview views Fixes microsoft#105670 Previously our webview commands assumed that webviews were always going to be in an editor. This is no longer true with webview views. To fix this, I've added an `activeWebview` to the `IWebviewService`. This tracks the currently focused webview. * microsoft#106358 * debug: bump js-debug-companion * re microsoft#105735. * re microsoft#105735. no more udpateMetadata api. * Fix microsoft#106303 * Use destructuring to make code more clear * Add isWritableFileSystem api Fixes microsoft#91697 This new API checks if a given scheme allows writing * Revert "Fix microsoft#106303" This reverts commit 8e5eed1. this is causing a layer check issue * Cache webview view title across reloads Fixes microsoft#105867 * fix some TS 4.1 errors (microsoft#106358) * fix some TS 4.1 errors (microsoft#106358) * fix TS 4.1 compile errors, microsoft#106358 * pinned tabs - flip default to "shrink" * fix ts errors related to microsoft#106358 * pinned tabs - closing pinned tab should open next non-pinned * pinned tabs - add a tab.lastPinnedBorder color * Adds commands for --no-verify commit variants (microsoft#106335) * add `{allow,confirm}NoVerifyCommit` options * adds commit comands with no verify * handles no verify command variants * handle no verify commit option * only display no verify variants when option is set * trusted types related to microsoft#106395 * more TS 4.1 fixes (microsoft#106358) * update trusted types search * Fix TS 4.1 errors for tasks and remote explorer Part of microsoft#106358 * Adress microsoft#106358: Fix TS 4.1 errors in codebase * debt - simplify metadata edit because we now have CellEditType.DocumentMetadata * Fix Trusted Types violations (round #2) microsoft#106395 * debug: return result of a msg to debug adapter can be undefined * add ExtHostFileSystemInfo which knows what schemes are reserved and which are used, microsoft#91697 * fixes microsoft#106334 * web - fix bad credentials lookup * Correct path to code-workspace.xml Fixes microsoft#106384 * Multi git executable paths (microsoft#85954) * Multi git executable path * update `git.path` setting description * 💄 Co-authored-by: João Moreno <[email protected]> * Correct linux code-workspace path * fixes microsoft#104047 * Add defaultUriScheme to path service (microsoft#106400) Fixes microsoft/vscode-internalbacklog#1179 * 💄 * Fix microsoft#106303 * Avoid innerHTML (microsoft#106395) * Avoid innerHTML (microsoft#106395) * debt - REMOTE_HOST_SCHEME => Schemas.vscodeRemote * fixes microsoft#106355 * pathService - defaultUriScheme() => defaultUriScheme * Adjust active terminal tab when active tab moves (microsoft#106413) Fixes microsoft#106300 * debt - adopt defaultUriScheme also for userHome * debt - adopt defaultUriScheme over hardcoded vscode-remote in toLocalResource * some integration tests for notebook editing, microsoft#105283 * refs microsoft#106358 * Bump yargs-parser in /extensions/markdown-language-features (microsoft#106373) Bumps [yargs-parser](https://github.com/yargs/yargs-parser) from 13.1.1 to 13.1.2. - [Release notes](https://github.com/yargs/yargs-parser/releases) - [Changelog](https://github.com/yargs/yargs-parser/blob/master/docs/CHANGELOG-full.md) - [Commits](https://github.com/yargs/yargs-parser/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump yargs-parser in /extensions/css-language-features/server Bumps [yargs-parser](https://github.com/yargs/yargs-parser) from 13.1.1 to 13.1.2. - [Release notes](https://github.com/yargs/yargs-parser/releases) - [Changelog](https://github.com/yargs/yargs-parser/blob/master/docs/CHANGELOG-full.md) - [Commits](https://github.com/yargs/yargs-parser/commits) Signed-off-by: dependabot[bot] <[email protected]> * Remove arrays.findIndex For microsoft#103454 This should be a direct map to the `.findIndex` mathod * Use textContnet for style element For microsoft#106395 * use textcontent in menu css refs microsoft#106395 * Fix one innerHTML usage microsoft#106395 * Use `@example` tags in vscode.d.ts (microsoft#106352) `@example` is the standard way to document code examples. This also gets us syntax highlighting of code examples in hovers * - reload only local user configuration after initi - add perf mark up and logs * re microsoft#105735. batch apply edits. * notebook text model initialization does not increment version * private outputs slice and unknown change. * applyEdit supports begin/end selections. * replace insertCell with applyEdit. * do not copy execution related metadata * 💄 * fix build. * Document view.type contribution Fixes microsoft#105821 * Improve views contribution point - add required properties - add default snippet - use `markdownDescription` for markdown string * Replace our arrays.find with stdlib find For microsoft#103454 * Pin blob storage dep see Azure/azure-sdk-for-js#11187 * unit tests for batched edits. * remove spliceNotebook api from textmodel. * update exploration branch * Fix some trusted type violations, microsoft#106395 * fix fonts in monaco menu * update distro * some jsdoc for microsoft#54938 * try to fix build (linux) * electron - set spellcheck: false again for windows * update search file, microsoft#106395 * 🆙 web playground * Trusted types - don't use innerHTML for rapid render, microsoft#106395 * Remove Schemas.vscodeRemote from simple file dialog * debt - remove some any casts from window * update distro * fix linux build * argh * proxy authentication does not work on 1.49 (microsoft#106434) * do not use hasClass and first microsoft#103454 * debug and explroer: do not use dom.addClass, dom.toggleClass * do not use deprecated dom helper methods microsoft#103454 * adopt new amd loader with support for TrustedScriptURL, add typings for TrustedTypesFactory et al, microsoft#106396 * explorer: Should maintain row focus after deleting a file fixes microsoft#71315 * Update Codicons: add 'magnet' icon microsoft/vscode-codicons@4c61155 * Remove unused 'SettingSearch' issue type * notebook document data loss. * cell language should not be freezed. * Add preferred_username to the list of msft token claims (microsoft#106511) * debug: update js-debug * fix microsoft#106430. * hide outputs if it is transient. * Add optional typing for webview state in WebviewPanelSerializer This makes it easier for extensions to correctly type their state if they wish * Add comment to WebviewViewResolveContext * use optional chaining * Use `Set` instead of array Sets should offer faster checking to see if a property has been seen * Create webview.web.contribution Fixes microsoft#106516 Creates an explicit contribution file for web. This makes sure we only don't register the `IWebviewService` twice. Not 100% sure this fixes the issue since I couldn't repo the original bug with our oss builds * Revert "API proposal for tree item icon color" This reverts commit 52e557f. * Skip formatting when during format-on-save, the configured formatter cannot be found (continue to show silent notification), microsoft#106376 * don't use renderCodicons any more, microsoft#105799 * remove old renderCodicons-function, rename renderCodiconsAsElement to renderCodicons * NotebookEditorEdit-api changes, microsoft#105283 * WorkspaceEdit-api changes, microsoft#105283 * adopt notebook integration tests, microsoft#105283 * add NotebookCell#index, microsoft#106637 * fix delay issue for provideCodeLenses, microsoft#106267 * rename RunOnceScheduler#timeout to delay * use debian stretch images (microsoft#106656) * remove deprecated function calls related to microsoft#103454 * workaround, maybe fix for microsoft#106657 * update search files * debt - make class list utils functions so that @deprecated works for them * fixes microsoft#106406 * notebook - when creating a notebook, check that no notebook with another viewtype exists * fix bad classList usage * add regression test for microsoft#106657 * fixes microsoft#86180 * fixes microsoft#100172 Co-authored-by: Alex Ross <[email protected]> Co-authored-by: Daniel Imms <[email protected]> Co-authored-by: João Moreno <[email protected]> Co-authored-by: isidor <[email protected]> Co-authored-by: Christof Marti <[email protected]> Co-authored-by: Johannes Rieken <[email protected]> Co-authored-by: Benjamin Pasero <[email protected]> Co-authored-by: Martin Aeschlimann <[email protected]> Co-authored-by: rebornix <[email protected]> Co-authored-by: Rob Lourens <[email protected]> Co-authored-by: IllusionMH <[email protected]> Co-authored-by: Daniel Imms <[email protected]> Co-authored-by: Connor Peet <[email protected]> Co-authored-by: Matt Bierner <[email protected]> Co-authored-by: Jean Pierre <[email protected]> Co-authored-by: Peng Lyu <[email protected]> Co-authored-by: Sandeep Somavarapu <[email protected]> Co-authored-by: Vyacheslav Pukhanov <[email protected]> Co-authored-by: Alex Dima <[email protected]> Co-authored-by: João Moreno <[email protected]> Co-authored-by: Miguel Solorio <[email protected]> Co-authored-by: SteVen Batten <[email protected]> Co-authored-by: Jackson Kearl <[email protected]> Co-authored-by: Rachel Macfarlane <[email protected]> Co-authored-by: Dirk Baeumer <[email protected]> Co-authored-by: WhizSid <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: deepak1556 <[email protected]> Co-authored-by: Oleg Demchenko <[email protected]>
|
@patrikmolsson @JacksonKearl |
|
Patrik has this issue consistenly. But when he tried with the Fiddler proxy in the way above, it doesn't fail any more. |
|
I've hit this error as well when try to upload multiple images sequentially |
|
I consistently hit the same error and see a very similar log to that posted by @brianpham93 . It occurs when using the Downgrading to version 12.1.2 makes the issue go away.
|
ljian3377
added
bug
|
The bug is related to the stream length. |
|
any workaround we can apply to avoid this temporarily ? |
A fix is coming soon. Before that, could you call const body = () => yourStream;
await blockBlobClient.stageBlock(base64encode("1"), body, body.length);
await blockBlobClient.stageBlock(base64encode("2"), body2, body2.length);
await blockBlobClient.commitBlockList([base64encode("1"), base64encode("2")]);Also, disabling keep-alive will also work. const pipeline = newPipeline(credential, {
keepAliveOptions: {
enable: false
}
});
const blobPrimaryURL = `https://${credential.accountName}${accountNameSuffix}.blob.core.windows.net/`;
return new BlobServiceClient(blobPrimaryURL, pipeline); |
|
Bug located and a hot fix is coming soon. Caused by sending data longer than specified by content-length. @jeremymeng But if the root cause is the body |
I don't think stream has a length, or did I miss anything?
keep-alive is enabled by default. Only storage libraries use keep-alive. I remember @XiaoningLiu added the keep-alive support but I couldn't locate the history. |
The input is a stream factory. So theoretically you can count the length before hand but is just inefficient. |
|
We belive this issue is addressed in the hotfix: @azure/[email protected] and @azure/[email protected] |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Similar to #11163.
I have script running every 5 minutes for all of eternity that uploads to blob storage, starting ~a day ago I started getting consistent
Bad Request - Invalid Verberrors from this code:This is consistent with the release of 12.2.0, which I see made some changes to the
uploadStreammethod.The exact response is as follows:
The text was updated successfully, but these errors were encountered: