[1.1.2] Exception "Response Content-Length mismatch" on Azure #1376
Comments
What does the controller action look like that handles this request? |
|
It's the aspnetcore-spa template with React/Redux. |
|
@pellea Thanks for the bug report. You're correct about the IPv6 warning being harmless. Is this the template you're talking about? Why do you have to restart the website? Does the error keep reoccurring after you see it once? Is AIRxTOUCHBAR.mp4 being served by the StaticFiles middleware? If so, what does |
|
It's the right template (or here https://github.com/aspnet/JavaScriptServices). The website is down with a CGI error page. That's why I have to restart it (quickly, it's in prod and we are at a critical moment; the website must be running all the time !). The error is reoccurring. Yes the file is served with the static files middleware. The It's not the right size: I didn't have that error before migrating to the .csproj and 1.1.0 version (the "old" version used the package.json and 1.0.1). |
|
1.1.0 verifies the content length matches what was written to the response stream. That's why you're seeing it after upgrading. |
|
Okay but then what can I do? I'm considering removing the video and use a low definition animated GIF but it's still a temporary solution (that's not what we really want). |
|
Can you reproduce it with all files or just that specific file? Can you reproduce it locally as well as in production? |
|
Yes the error appears after some times in a local environment too. |
|
Athough it's not the exact same bug as #1289, you can try a similar workaround. Change your app.UseStaticFiles(new StaticFileOptions
{
OnPrepareResponse = context =>
{
context.Context.Response.Headers.Remove("Content-Length");
}
});This will cause the response to use Do you have any custom middleware that's registering |
|
The Content-Length error might just be the symptom not the root issue. It's possible that this is fairly common scenario where the client (in this case the IIS ANCM module) disconnects during a download. Normally response write errors are ignored, but the static file middleware goes out of its way to verify that writes succeed and throws otherwise in order to avoid wasting cycles. The logs posted in this issue are what you would see if some other middleware swallowed the exception the static files middleware threw. If this is what's going on, the workaround mentioned above by @CesarBS would clear up the logs, but not really fix any underlying issues. The CGI error page is unexpected though. More details would be helpful here.
Upping the logging verbosity to "Trace" and posting the resulting logs could also help us track down this issue. |
|
Why could a content-length error be logged for an already aborted request? |
|
@Tratcher Kestrel doesn't check whether or not the request is aborted before it logs the Content-Length mismatch. That's an oversight that never came up before since most writers ignore errors, and those that don't ignore errors would normally throw from the app func. For this error to be logged in this scenario, both the write would have throw (uncommon) and the exception would have to be swallowed. So that there is a logging issue that should be fixed, but that doesn't explain the CGI error page or the need to restart the site. |
|
The video is never requested directly by the user. We use it as a background video on the home page (and it loops). I have the Cors, Mvc and OpenIdConnectAuthentication middleware. I enabled more log in Azure. I will not redeploy the web app to activate more log; it takes like 30 minutes to deploy; npm/webpack is super slow on Azure. But I can test whatever you want on my local environment. Here is my local "full" log with more trace (and the exceptions obviously): |
|
If I take a look a Chrome dev tool; each time the video loops, I get another size ... Why? |
|
The StaticFiles middleware is using the HTTP range feature. The 206 status code indicates a partial response. |
|
I forgot to copy the CGI error page: |
|
I got again the error today:
|
|
@pellea I still don't think we have enough information to reproduce the issue. Here are some questions I still have:
If you could capture a memory dump of the failing "dotnet" (not w3wp) process before restarting that could prove instrumental in tracking down why the server is commingle completely unresponsive assuming that's the case. Upping the logging verbosity to "Trace" and posting the resulting logs could help. You posted more detailed local logs, but those only included "Info" level logs and above which are less verbose. Publishing a sample application that you know also fails in this way on GitHub might also help. |
|
I have created a sample that crashes exactly like my other website. I use Chrome as my browser BTW. |
|
@halter73 Were you able to reproduce the error in your environment? As soon as the server (on Azure) crash again, I'll get diagnostics logs using the Azure support tools. |
|
I wasn't able to reproduce locally when directly loading the page from Kestrel or through IIS. I'll try deploying as an Azure Web App. Since I don't have real traffic going to the sample app, I just tried loading it a few times with Chrome. Do you have any idea how many requests it usually takes to repro? Can you repro it quickly with some automated client? |
|
You don't have to make a lot of request. Let the page opened, wait a few minutes for the video to play itself more than once and you'll get the error. To reproduce the error I simply start the website using "dotnet run"; open Chrome; navigate to localhost:5000 and wait a few minutes. |
|
@pellea I am now able to reproduce the "Response Content-Length mismatch: too few bytes written" logs, but except for the annoying log messages, I don't see any other harmful affects. As mentioned above, I agree we shouldn't log this when the client has already aborted the request, but you claimed "I have to restart the website." This makes no sense to me. Despite these logs, the website remains completely responsive. So to reiterate my previous questions:
|
|
Yes the website seems responsive (with no video playing). I know that this is not the root of the problem that make the server unresponsive.
Here is the log of eventlog.xml: |
|
@pellea Sorry, I'd like some clarification.
Are you saying that about 60 seconds after seeing the "too few bytes written" log message, the website becomes unresponsive and starts responding with the 502.3 error page to new requests? Do you see the error page on any request to any path? |
|
No, not after seeing "too few bytes written" log message. I see this log entry a lot before the website is completely unresponsive. When the website is unesponsive it takes 60 seconds for Chrome/Firefox to display the CGI error page and after a while the page is displayed instantly. |
|
I got the error today. It took more than 1 minute to get an error page ... Is there a fix for the initial issue "Response Content-Length mismatch"? Here is a screenshot of the "unresponsive" website:
|
|
I see. There are two separate issues here then. Unfortunately it's too late to ship a fix for the content length mismatch messages in the upcoming patch release. We'll consider including that in the next patch after that. cc @Eilon The unresponsiveness needs further investigation. You could be running into #1278, which will be fixed in 1.1.1. A workaround some people have been using is to serve static files from a separate IIS instance, since the issue arises from the fact the StaticFiles middleware passes |
|
Merged: 14aa07d |
|
Preview builds of this patch fix should be available on the following feeds:
If you have a chance, please try out these preview builds and let us know if you have any feedback! |
|
Thanks for the update. I'll have to wait for the 1.1.2 release to test... |
|
Could anyone please provide guidance or a link on how to use the pre-release/public patches packages in our projects? We tried adding the preview feed to our nuget configuration and referencing the patch packages but the project is not downloading the correct version. Example: When we reference “1.0.5-rtm-201704200225” -> it downloads -> Microsoft.NETCore.App 1.1.0-preview1-001100-00 When we restore with the debug flag enabled we can see that the pre-release feed is being used: |
|
@mxa0079 Why are you referencing aspnetcore-ci-dev feed in your nuget.config? Please remove that, clear your local caches, and retry. |
|
@muratg we need that feed for another package we are using (signalR for .net core). We will clear the cache and try again |
|
@mxa0079 Sounds good. One workaround would be, putting all the SignalR packages into a local folder, and adding that folder as a feed to your nuget.config for testing. This way, you'd only rely on nuget.org + this patch's myget feed for restore time. Also please keep in mind that SignalR (Core) is not RTM yet. |
|
Nope,you can't use SignalR with ASP.NET Core 1.x it requires 2.0 |
|
@davidfowl We are using it and our internal tests have proven successful. However, we have yet to test it at a large scale - the funtionality that relies on it is hidden behind a feature switch - but we are about to turn it on. 😱 -- any heads up? |
|
@muratg Local folder is not an option because our CI and CD process will not work like that (unless we check in the package, which I'de like to avoid). We just tried the restore the project using the exact version match notation (surrounding the version number by brackets) and got the following message: Could you get your hands on a Thanks! |
|
@mxa0079 what version of SignalR are you using? What's the package id? |
|
|
Eek, those are deprecated ones. |
|
Do you have an ETA for the 1.1.2 release? Is it around the 10 may like the corefx? There are no due dates available on this repo. Thanks. |
|
@pellea - yeah should be in about a week or so. Very soon! |
|
Hi, I could not fix this error in the YII2 framework, what should be done. |
|
Especially that mistake: Problem 1 Potential causes:
Read https://getcomposer.org/doc/articles/troubleshooting.md for further common problems. Installation failed, reverting ./composer.json to its original content. |
|
When I have done this in the Console: composer require --prefer-dist yiisoft/yii2-jui |
|
@AlisherSharopov this is a closed issue. Please open a new issue that covers your specific scenario. |
|
Who will be can fix this error? |
|
Well, I'm organizing now |
Hello,
After some times, the following warning and error appears in the log file and I have to restart the website. This happens in a production environment on Azure!
Here are the "specs" of the website:
Here is the logs:
How can I debug solve/debug this?
The warning seems harmless ("Unable to bind to http://localhost:6074 on the IPv6 loopback interface.").
But how is it possible to have this error "System.InvalidOperationException: Response Content-Length mismatch: too few bytes written (13688832 of 16307141)." ?
Thanks,
Adrien.
The text was updated successfully, but these errors were encountered: