Crash occurs if ebpfcore!ebpf_extension_unload is called in DISPATCH IRQL

[shankarseal]

We are hitting this crash in Fix port_quota app usage by dthaler · Pull Request #1184 · microsoft/ebpf-for-windows (github.com).

Crash details:

kd> !irql
Debugger saved IRQL for processor 0x2 -- 2 (DISPATCH_LEVEL)

fwpkclnt!FwpmFilterDeleteById0
netebpfext!net_ebpf_extension_delete_wfp_filters
netebpfext!_net_ebpf_extension_bind_on_client_detach
netebpfext!_net_ebpf_extension_hook_provider_detach_client
NETIO!NmrpNotifyBindingDetach
NETIO!NmrpDetachList
NETIO!NmrpDeregisterModule
NETIO!NmrDeregisterClient
ebpfcore!ebpf_extension_unload
ebpfcore!ebpf_link_detach_program
ebpfcore!_ebpf_program_detach_links
ebpfcore!_ebpf_program_free
ebpfcore!ebpf_object_release_reference
ebpfcore!_ebpf_pinning_entry_free
ebpfcore!ebpf_pinning_table_delete
ebpfcore!ebpf_core_update_pinning
ebpfcore!_ebpf_core_protocol_update_pinning
ebpfcore!ebpf_core_invoke_protocol_handler
ebpfcore!_ebpf_driver_io_device_control

IP_IN_PAGED_CODE:
fwpkclnt!FwpmFilterDeleteById0+0

FAILURE_BUCKET_ID:  AV_VRF_CODE_AV_PAGED_IP_fwpkclnt!FwpmFilterDeleteById0

Function FwpmFilterDeleteById is marked as paged

#pragma alloc_text(PAGE, FwpmFilterDeleteById)

So we crash trying to invoked paged code while at dispatch.

The reason why we are invoking this code path in dispatch is because ebpf_pinning_table_delete calls _ebpf_pinning_entry_free while holding pinning_table->lock thus elevating the irql to dispatch. The _ebpf_pinning_entry_free function eventually tries to detach the program from the hook which results in NmrDeregisterClient --> detach handler in netebpfext which tries to free WFP filters by calling an API in paged code.

Solution:

At first I thought the solution was to change the netebpf extension detach handler to delete WFP filters in a passive work item and complete detach operation asynchronously. But this will not solve the problem as currently the ebpf_extension_unload function invokes NmrWaitForClientDeregisterComplete which is passive only.

        status = NmrDeregisterClient(client_context->nmr_client_handle);
        if (status == STATUS_PENDING) {
            status = NmrWaitForClientDeregisterComplete(client_context->nmr_client_handle);

So I suppose the soluition would be to change ebpf_extension_unload to do the NMR deregistration in passive. Not sure, what that means for the EC program detach function call though.

[Alan-Jowett]

Why does it need to call _ebpf_pinning_entry_free while holding the lock? Move the call to freeing this entry outside the lock and the problem goes away?