From a distance, it seems that while SerilogWebHostBuilderExtensions deals with bona fide ASP.NET concerns (e.g IWebHostBuilder), SerilogLoggerFactory would be better placed in a more generic .NET Core library. Perhaps Serilog.Extensions.Logging.
There's also the possibility that I'm completely insane. :)
Of course if not, I realize this would be a breaking change but maybe something to consider in the next major version of this library.