JavaScript Browser Library File Size Too Large (1.3MB+)

[murraybauer]

Thanks for finally releasing a javascript browser SDK.

But the modules need to be better split and file size reduced. Common.js is 1.3MB + the file size of blob/file/table/queue.js

It's a large file to download, but more so to parse the javascript onload in mobile browsers or cordova apps even if bundled in an installable app.

We are seeing UI framework libraries targeting the 50kb and under mark.

[XiaoningLiu]

Hi murraybauer,

Thanks for your feedback!!

Currently, the JavaScript library for browsers is generated from the Node.js library, so the generated files include many open source Node.js modules.

In current preview stage, we are more focusing on the functionality of the library. We will take further consideration about the file size issue when the library is in stable. And we also plan to provide a js library CDN service with minified version and gzip compression opened.

Best,
Xiaoning

[seguler]

@murraybauer thanks for your feedback. Indeed we need to look at our options to reduce the footprint. Are you planning to use this library for a mobile scenario ?

[murraybauer]

@seguler yes.

In the past i've had to create my own wrapper around the REST API's.

I was hoping for a substanical improvement having launched a specific javascript API (something AWS has had for years), rather than just browserify and a few code splits.

Here is an article indicating just how much time is spent parsing 1MB of javascript on a mobile device!
https://medium.com/dev-channel/javascript-start-up-performance-69200f43b201?imm_mid=0edae8&cmp=em-web-na-na-newsltr_20170222#.vs7wndwoy

The node azure storage library is notorious for being problematic given it's size even in server side azure functions. The issue is more of an azure functions issue, but still related to size and bundeling. Azure/azure-functions-host#298

At a minimum I'd suggest offering a different split - all inclusive blob, file, table, queue files. Highly unlikely any app will use all 4 storage services so having a large common file with a few small exclusions files per type doesn't make a lot of sense. And a decent re-factor and tree shaking.

[XiaoningLiu]

Thanks @murraybauer for reporting this! We already have the backlog to reduce the footprint of our library including Node.JS and JS libraries. As it'll be a foundemental change need to be tested well, we need to find an appropriate time window for that. Will get you updated about our progress.

[nandowalter]

Hey guys,
sorry, but it's possible to evaluate the possibility to release an ES6 version of azure-storage library? This would made possible to better bundle the code, applying tree shaking and keeping bundle size lower than actual. Thanks!

[XiaoningLiu]

Hi @nandowalter , you're right! ES6 is the future trend for JavaScript. Actually, we already did some work and efforts to reduce the library size. Will release after validation.

[XiaoningLiu]

Hi @murraybauer and @nandowalter, 0.2.1-preview.4 of the Azure Storage JavaScript Client Library for browsers is released. The size of the common.min.js is successfully reduced from 1.3MB to 518KB, welcome to have a try.

[seguler]

Awesome!

[1N50MN14]

@XiaoningLiu 0.2.1 as published on npm still produces 1.3MB file size are you sure npm is up to date?

[XiaoningLiu]

@1N50MN14

NPM will always sever the latest changes for Node.js Client Library. But for some changes only on JavaScript for browsers, we will not release a new Node.js Client Library version, because these changes are not necessary for Node.js customers.

Please access to latest version of JavaScript Client Library for Browsers through following ways:

1. The GitHub repo release page https://github.com/Azure/azure-storage-node/releases
2. Download from https://aka.ms/downloadazurestoragejs
3. Clone from the master branch of the repo then re-generate the js library

Hope this can help, and any suggestions are welcome!

[1N50MN14]

Thanks @XiaoningLiu

1.3MB for server side usage is an overkill specially when it comes to Azure Functions, that's 1.3MB that needs to be loaded into memory for every single Azure Function. Not to mention performance and cost implications. This needs to be dramatically reduced imho, the size of the library simply doesn't make any sense; I do have few suggestions that are easy to implement and will have immediate impact:

1. The dependency browserify-mime is large and is not intended for server-side use as the name implies, you have that as a main dependency as you can see here: https://github.com/Azure/azure-storage-node/blob/master/package.json#L23 - otherwise node's fs to check for the file mime.

2. underscore is also relatively large and you only use a handful of functions throughout the code: _.isArray(), _.isUndefined(), _.isDate(), _.extend() and _.zip() - those could easily be replaced with for example Array.isArray(myvar), typeof myvar === 'undefined', myvar instanceof Date etc alternatively require each of them individually, for example require('lodash.isdate').

3. You can remove jQuery's port of extend https://github.com/Azure/azure-storage-node/blob/master/package.json#L24

When it comes to Common.js compatibility the module can and needs to be consistent so it can also be used within Azure Functions pr Azure/azure-functions-host#298, Azure Functions itself also now has a built-in feature (preview) to pack assets using Webpack in which case azure-storage-node will fail so this issue will show up again sooner or later. Manually maintaining and downloading code is not a practical solution, regardless the use case.

The solution here replace whichever module that causes generating a Common.js compatible bundle to fail with one that's Common.js compatible, you don't need a complete re-write to accomplish that.

Imho these need to be addressed immediately, later on Tables/Blobs/Queues/Files/Service can be decoupled into their own modules. For example, browserify-fs is also an overkill (and frankly very outdated) with all kinds of leveluprelated dependencies; and it's being use only within File and only the a browser context.

[XiaoningLiu]

@1N50MN14 Thanks again!

Let's clarify the scenarios you want a more light-weight library, are in the server side run time, such as Azure Functions, that's correct?

The size reducing work we have done is for the JavaScript Client Library for browsers which runs in a browser environment. The JavaScript Client Library for browsers is generated with Browserify. To reduce the size, we drop the dependence on the request module and browserify-fs module, that reduces lots of size especially for the request module. request module has tons of dependencies.

But above changes only takes effect during the browserify. Node.js library still depends on the request module.

Thanks for your suggestions! I will investigate some quick actions I can do to reduce the Node.js library size, including the valuable suggestions you provided. But currently I cannot provide a clear timeline, because there are other high priorities feature parity items. PRs are also welcome.

Thanks!

[1N50MN14]

@XiaoningLiu Thanks. The specific problem with Azure Functions is that azure-storage-node dramatically slows down the startup and load time of AF functions due to the large number of dependencies; so much so it makes Azure Functions unpractical for production use both technically and financially.

In order to improve the loading time of functions azure-storage-node needs to be bundled for server side use either using Webpack or Browserify please take a look at Azure/azure-functions-host#298 as mentioned in my previous comment. The output bundle is still extremely large resulting in excessive memory consumption hence operational cost.

When it comes to browserify-fs and request I'm not sure why they're being used server-side, Node.js has native fs and http implementations. If you wish to remain internally consistent with the http API there are smaller bare bone modules that consider the environment in which they run; for the browser there's no way around the FileReader API (though I wouldn't be surprised if they're a module out there that wraps that as a node-like fs API).

This issue is operational critical and should be on top of the priority list imho, unless Microsoft is working under the assumption Azure Functions is a side project for none production use, in which case customers should be informed they're better off with a different solution / provider.

Briefly looking at this module I can see it should and could be few 10s of kbs in size at most; even more importantly it needs to be split into five smaller modules (i.e.azure-storage-node/Tables etc). I'm happy to send a PR but then I will need to re-organize the code into smaller modules, replace all modules that are either redundant or too big in size, and finally make azure-storage-node behave consistently regardless the environment under which it runs.

If that's ok for you and you're willing to merge such changes then I'll schedule a day mid-next week to work on this and submit a PR.

[XiaoningLiu]

@1N50MN14 Thanks!

I can drop browserify-fs because it's useless for the latest publishes. For request module, I have realized most of it's functionalities our library depends, based on native http module. But the proxy of request module has not been implemented.

About splitting into five smaller modules, it has been considered by our architectures as a long term planned item. In the short term, I'm afraid that we cannot accept breaking changes like that. However, ideas or suggestions are really welcome. I can propose them to our team's discussion and planning.

Currently, non-breaking PRs can be accepted, such as removing or replacing redundant modules or codes. For complex changes, better to sync up the design before coding, in case of unexpected conflicting.

@seguler @yaxia @vinjiang What's your suggestions about improving library's performance on Azure Functions?

[1N50MN14]

@XiaoningLiu Imho, generally speaking, the problem goes beyond browserify-fs because when bundled for server-side use (aka something like browserify --node ./node_modules/azure-storage/lib/azure-storage.js --standalone azure-storage-node -o ./bundle.js --full-paths true) you'll be able to see the module consumes 2.7mb - whereas for Azure Functions this is a deal breaker, it's still consumes too much memory even for regular server instances.

Anyway I generated a dynamic pie chart using disco with a breakdown of the modules by size so you're able to get a clear picture of what's going on, here's the link: https://twassetsdev.blob.core.windows.net/tmp/asn.html

For example you can see that blobservice.core.js consumes 389.5kb, fileservice.core.js consumes 243.5kb and in general ./lib consumes 1.3mb of what's supposed to be pure lines of text there's no way ./lib has 1.3mb worth of text. So there must be something fundamentally wrong in the way different modules are being included or loaded it looks like as if different pieces of code and modules are being replicated rather than required.

Then you have async that is 179.5kb which you don't use at all, browserify-mine and mime-db together consume 200kb which can also be redundant - those combined are 400kb in total which you can get rid of with minimal effort. These are just basic observations I'm sure you'll get a better insight when you look at the module breakdown and find other redundant modules (for example underscore, request etc).

I completely understand you don't wish for any breaking changes, but if you guys take a look at the interactive chart and spend little of your time in understanding what's going on, you'd be able to dramatically reduce the size of the module with zero breaking changes and minimal effort on your side.

[1N50MN14]

@XiaoningLiu @seguler @yaxia @vinjiang No follow up on this issue?

[seguler]

I can confirm that we have plans to split the node.js library into 4+ modules (one for each service). As a side note, we recently implemented the split libraries in the Python SDK for Storage. Apart from this, it seems a lot more can be done based on your suggestion. @XiaoningLiu @vinjiang we should take these for the next planning cycle.

[1N50MN14]

Awesome thanks for the update @seguler

[XiaoningLiu]

@seguler Of course, we need discuss the details. Also thanks to the suggestions @1N50MN14 provided.