Skip to content

Fix a hang issue in downloading when there's transient network issue #22894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
EmmaZhu merged 1 commit into from
Aug 16, 2022
Merged

Fix a hang issue in downloading when there's transient network issue #22894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions sdk/storage/storage-blob/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@

- Added support for service version 2021-10-04.

### Bugs Fixed

- Fixed a hang issue in BlobClient.downloadToBuffer when encountering transient network failure.

## 12.11.0 (2022-07-08)

### Features Added
Expand Down
2 changes: 2 additions & 0 deletions sdk/storage/storage-blob/src/utils/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ export const BLOCK_BLOB_MAX_BLOCKS: number = 50000;
export const DEFAULT_BLOCK_BUFFER_SIZE_BYTES: number = 8 * 1024 * 1024; // 8MB
export const DEFAULT_BLOB_DOWNLOAD_BLOCK_BYTES: number = 4 * 1024 * 1024; // 4MB
export const DEFAULT_MAX_DOWNLOAD_RETRY_REQUESTS: number = 5;

export const REQUEST_TIMEOUT: number = 100 * 1000; // In ms
Copy link
Copy Markdown
Member

@ljian3377 ljian3377 Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to make this configurable?

/**
* The OAuth scope to use with Azure Storage.
*/
Expand Down
13 changes: 12 additions & 1 deletion sdk/storage/storage-blob/src/utils/utils.node.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import * as fs from "fs";
import * as util from "util";
import { REQUEST_TIMEOUT } from "./constants";

/**
* Reads a readable stream into buffer. Fill the buffer from offset to end.
Expand All @@ -24,8 +25,14 @@ export async function streamToBuffer(
const count = end - offset; // Total amount of data needed in stream

return new Promise<void>((resolve, reject) => {
const timeout = setTimeout(
() => reject(new Error(`The operation cannot be completed in timeout.`)),
REQUEST_TIMEOUT
);

stream.on("readable", () => {
if (pos >= count) {
clearTimeout(timeout);
resolve();
return;
}
Expand All @@ -46,6 +53,7 @@ export async function streamToBuffer(
});

stream.on("end", () => {
clearTimeout(timeout);
if (pos < count) {
reject(
new Error(
Expand All @@ -56,7 +64,10 @@ export async function streamToBuffer(
resolve();
});

stream.on("error", reject);
stream.on("error", (msg) => {
clearTimeout(timeout);
reject(msg);
});
});
}

Expand Down
2 changes: 2 additions & 0 deletions sdk/storage/storage-file-share/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

### Bugs Fixed

- Fixed a hang issue in ShareFileClient.downloadToBuffer when encountering transient network failure.

### Other Changes

## 12.11.0 (2022-07-08)
Expand Down
1 change: 1 addition & 0 deletions sdk/storage/storage-file-share/src/utils/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ export const FILE_MAX_SIZE_BYTES: number = 4 * 1024 * 1024 * 1024 * 1024; // 4TB
export const FILE_RANGE_MAX_SIZE_BYTES: number = 4 * 1024 * 1024; // 4MB
export const DEFAULT_MAX_DOWNLOAD_RETRY_REQUESTS: number = 5;
export const DEFAULT_HIGH_LEVEL_CONCURRENCY: number = 5;
export const REQUEST_TIMEOUT: number = 100 * 1000; // In ms

export const URLConstants = {
Parameters: {
Expand Down
12 changes: 11 additions & 1 deletion sdk/storage/storage-file-share/src/utils/utils.node.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import * as fs from "fs";
import * as util from "util";
import { REQUEST_TIMEOUT } from "./constants";

/**
* Reads a readable stream into buffer. Fill the buffer from offset to end.
Expand All @@ -24,8 +25,13 @@ export async function streamToBuffer(
const count = end - offset; // Total amount of data needed in stream

return new Promise<void>((resolve, reject) => {
const timeout = setTimeout(
() => reject(new Error(`The operation cannot be completed in timeout.`)),
REQUEST_TIMEOUT
);
stream.on("readable", () => {
if (pos >= count) {
clearTimeout(timeout);
resolve();
return;
}
Expand All @@ -46,6 +52,7 @@ export async function streamToBuffer(
});

stream.on("end", () => {
clearTimeout(timeout);
if (pos < count) {
reject(
new Error(
Expand All @@ -56,7 +63,10 @@ export async function streamToBuffer(
resolve();
});

stream.on("error", reject);
stream.on("error", (msg) => {
clearTimeout(timeout);
reject(msg);
});
});
}

Expand Down