Microsoft.Owin.SelfHost stops accepting requests

[scottlerch]

We ran into an issue where some server instances stopped accepting any requests until our self-host process was restarted. In the Microsoft.Owin trace logs we see the following:

Accept
System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
   at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.HttpListener.OwinHttpListener.<ProcessRequestsAsync>d__29.MoveNext()

We suspect the problem has been there for years but a specific load pattern at large scale made it worse. Our theory is it's caused by requests with headers greater than 4096 that are immediately abandoned by the client before waiting for response. It's extremely rare but we have 1000+ VMs with 20 billion requests a day so it happens quite regularly for us.

Below is a minimal repro using Microsoft.Owin.SelfHost 4.0.1 with .NET 4.7.2 that forces a HttpListenerException with ErrorCode 1229 "An operation was attempted on a nonexistent network connection" :

namespace OwinSelfHostSample
{
    using Microsoft.Owin.Host.HttpListener;
    using Microsoft.Owin.Hosting;
    using Owin;
    using System;

    static class Program
    {
        static void Main()
        {
            WebApp.Start("http://localhost:8000/", appBuilder =>
            {
                var owinHttpListener = appBuilder.Properties[typeof(OwinHttpListener).FullName] as OwinHttpListener;
                owinHttpListener.SetRequestProcessingLimits(maxAccepts: 1, maxRequests: 1);
                appBuilder.Use(async (context, next) => await context.Response.WriteAsync("Hello world!"));
            });

            Console.ReadKey();
        }
    }
}

OwinSelfHostSample.zip

1. Run program above with .NET source stepping
2. Breakpoint on this line: https://referencesource.microsoft.com/#System/net/System/Net/_ListenerAsyncResult.cs,75
3. Open browser and go to URL with total request headers >4096 bytes. Here's a sample just with long path and query parameters: http://localhost:8000/some/really/long/path/with/lots/of/query/parameters?value0=0&value1=1&value2=2&value3=3&value4=4&value5=5&value6=6&value7=7&value8=8&value9=9&value10=10&value11=11&value12=12&value13=13&value14=14&value15=15&value16=16&value17=17&value18=18&value19=19&value20=20&value21=21&value22=22&value23=23&value24=24&value25=25&value26=26&value27=27&value28=28&value29=29&value30=30&value31=31&value32=32&value33=33&value34=34&value35=35&value36=36&value37=37&value38=38&value39=39&value40=40&value41=41&value42=42&value43=43&value44=44&value45=45&value46=46&value47=47&value48=48&value49=49&value50=50&value51=51&value52=52&value53=53&value54=54&value55=55&value56=56&value57=57&value58=58&value59=59&value60=60&value61=61&value62=62&value63=63&value64=64&value65=65&value66=66&value67=67&value68=68&value69=69&value70=70&value71=71&value72=72&value73=73&value74=74&value75=75&value76=76&value77=77&value78=78&value79=79&value80=80&value81=81&value82=82&value83=83&value84=84&value85=85&value86=86&value87=87&value88=88&value89=89&value90=90&value91=91&value92=92&value93=93&value94=94&value95=95&value96=96&value97=97&value98=98&value99=99&value100=100&value101=101&value102=102&value103=103&value104=104&value105=105&value106=106&value107=107&value108=108&value109=109&value110=110&value111=111&value112=112&value113=113&value114=114&value115=115&value116=116&value117=117&value118=118&value119=119&value120=120&value121=121&value122=122&value123=123&value124=124&value125=125&value126=126&value127=127&value128=128&value129=129&value130=130&value131=131&value132=132&value133=133&value134=134&value135=135&value136=136&value137=137&value138=138&value139=139&value140=140&value141=141&value142=142&value143=143&value144=144&value145=145&value146=146&value147=147&value148=148&value149=149&value150=150&value151=151&value152=152&value153=153&value154=154&value155=155&value156=156&value157=157&value158=158&value159=159&value160=160&value161=161&value162=162&value163=163&value164=164&value165=165&value166=166&value167=167&value168=168&value169=169&value170=170&value171=171&value172=172&value173=173&value174=174&value175=175&value176=176&value177=177&value178=178&value179=179&value180=180&value181=181&value182=182&value183=183&value184=184&value185=185&value186=186&value187=187&value188=188&value189=189&value190=190&value191=191&value192=192&value193=193&value194=194&value195=195&value196=196&value197=197&value198=198&value199=199&value200=200&value201=201&value202=202&value203=203&value204=204&value205=205&value206=206&value207=207&value208=208&value209=209&value210=210&value211=211&value212=212&value213=213&value214=214&value215=215&value216=216&value217=217&value218=218&value219=219&value220=220&value221=221&value222=222&value223=223&value224=224&value225=225&value226=226&value227=227&value228=228&value229=229&value230=230&value231=231&value232=232&value233=233&value234=234&value235=235&value236=236&value237=237&value238=238&value239=239&value240=240&value241=241&value242=242&value243=243&value244=244&value245=245&value246=246&value247=247&value248=248&value249=249&value250=250&value251=251&value252=252&value253=253&value254=254&value255=255&value256=256&value257=257&value258=258&value259=259&value260=260&value261=261&value262=262&value263=263&value264=264&value265=265&value266=266&value267=267&value268=268&value269=269&value270=270&value271=271&value272=272&value273=273&value274=274&value275=275&value276=276&value277=277&value278=278&value279=279&value280=280&value281=281&value282=282&value283=283&value284=284&value285=285&value286=286&value287=287&value288=288&value289=289&value290=290&value291=291&value292=292&value293=293&value294=294&value295=295&value296=296&value297=297&value298=298&value299=299&value300=300&value301=301&value302=302&value303=303&value304=304&value305=305&value306=306&value307=307&value308=308&value309=309&value310=310&value311=311&value312=312&value313=313&value314=314&value315=315&value316=316&value317=317&value318=318&value319=319&value320=320&value321=321&value322=322&value323=323&value324=324&value325=325&value326=326&value327=327&value328=328&value329=329
4. Wait for breakpoint to be hit, then close browser, wait 4 minutes
5. Continue from breakpoint, HttpListenerException with error code 1229 "An operation was attempted on a nonexistent network connection" will be thrown here: https://referencesource.microsoft.com/#System/net/System/Net/_ListenerAsyncResult.cs,87
6. OwinHttpListener thinks HttpListener has been disposed when it has not been here: https://github.com/aspnet/AspNetKatana/blob/dev/src/Microsoft.Owin.Host.HttpListener/OwinHttpListener.cs#L226
7. There are no more accept threads and self-host is effectively dead

We know the error is recoverable as we've added a hack in our service where we have a background thread that periodically checks if there are no accept threads and kicks off a new one using reflection on OwinHttpListener and it works.

[Tratcher]

Nice debugging. I'll see if can replicate your results.