Object reference error occurs when a MassTransit message is received without an envelope.

[reza-toorani]

Description

If you are consuming a MassTransit message without MT envelope, GetQueueData method in https://github.com/newrelic/newrelic-dotnet-agent/blob/main/src/Agent/NewRelic/Agent/Extensions/Providers/Wrapper/MassTransit/MassTransitHelpers.cs throws an exception. It is looking for the source address in the message and expects '_' in the address name, but when the message is just raw JSON, it means no MT envelope and no source address.

Expected Behavior
It must be checked if there is a source address then split the parts.

Steps to Reproduce

1. Register MT

services.AddMassTransit(mcfg =>
    {
        mcfg.UsingRabbitMq((context, rcfg) =>
        {
            rcfg.AddHost(context);

            rcfg.ConfigureEndpoints(context);
            rcfg.UseRawJsonSerializer(RawSerializerOptions.AnyMessageType);
        });
    });

2. Send a message
3. config your consumer

protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator
        , IConsumerConfigurator<MyConsumer> consumerConfigurator
        , IRegistrationContext context)
    {
        endpointConfigurator.ConfigureConsumeTopology = false;
        endpointConfigurator.UseRawJsonDeserializer();
        endpointConfigurator.DefaultContentType = new ContentType("application/json");
     
    } 

Your Environment
NewRelic.Agent : 10.19.2
NewRelic.Agent.Api:10.19.2
NewRelic.LogEnrichers.Serilog : 1.2.0
.Net: 7.0
MassTransit: 8.1.0
Serilog:3.1.0
Serilog.Enrichers.GlobalLogContext:3.0.0
Serilog.Extensions.Hosting: 7.0.0
Serilog.Sinks.Async:1.5.0
Serilog.Sinks.Console:5.0.0

"Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.AspNetCore": "Warning",
        "Microsoft.AspNetCore.Authentication": "Information",
        "Microsoft.EntityFrameworkCore.Infrastructure": "Warning",
        "Microsoft.EntityFrameworkCore.Database.Command": "Warning",
        "Microsoft.Hosting.Lifetime": "Information",
        "System": "Warning"
      }
    },
    "Using": [
      "NewRelic.LogEnrichers.Serilog"
    ],
    "Enrich": [ "FromLogContext", "WithNewRelicLogsInContext" ],
    "WriteTo": [
      {
        "Name": "Async",
        "Args": {
          "configure": [
            {
              "Name": "Console",
              "Args": {
                "formatter": "NewRelic.LogEnrichers.Serilog.NewRelicFormatter, NewRelic.LogEnrichers.Serilog"
              }
            }
          ]
        }
      }
    ]
  }

Additional context

MT-Fault-ExceptionType:	System.NullReferenceException
MT-Fault-InputAddress:	rabbitmqs://serverName/queue-name
MT-Fault-Message:	Object reference not set to an instance of an object.
MT-Fault-StackTrace:	at NewRelic.Providers.Wrapper.MassTransit.MassTransitHelpers.GetQueueData(Uri sourceAddress)
at NewRelic.Providers.Wrapper.MassTransit.NewRelicFilter.Send(ConsumeContext context, IPipe`1 next)
at MassTransit.Middleware.TeeFilter`1.<>c__DisplayClass5_0.<<Send>g__SendAsync|1>d.MoveNext() in /_/src/MassTransit/Middleware/TeeFilter.cs:line 40
--- End of stack trace from previous location ---
at MassTransit.Middleware.OutputPipeFilter`2.SendToOutput(IPipe`1 next, TOutput pipeContext) in /_/src/MassTransit/Middleware/OutputPipeFilter.cs:line 76
at MassTransit.Middleware.OutputPipeFilter`2.SendToOutput(IPipe`1 next, TOutput pipeContext) in /_/src/MassTransit/Middleware/OutputPipeFilter.cs:line 108
at MassTransit.Middleware.DeserializeFilter.Send(ReceiveContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/DeserializeFilter.cs:line 40
at MassTransit.Middleware.RescueFilter`2.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RescueFilter.cs:line 43

For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5): P1
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-179186

[nr-ahemsath]

Hi @reza-toorani,

Thank you for the detailed bug report, especially for including thorough repro steps. I am hopeful that we can produce a fix quickly with this information.

[AdrianoRNascimento]

Hi @nr-ahemsath
I'm facing the same issue and my team is in a difficult situation.
Either disable NR or rollback all the work we had to use Mass transit with the external queues.

Do you have any workaround or estimate to deliver this fix?

Best regards.
Adriano Nascimento

[nr-ahemsath]

@AdrianoRNascimento You can roll back your New Relic agent to version 10.18.0, before we added MassTransit instrumentation.

Alternatively, if there are other features you need in 10.19.0+, you can remove the instrumentation files for MassTransit from your agent installation. These will be in a subfolder of your agent install folder named extensions, will end in .xml, and have MassTransit in the name. Remove these files and restart your agent.

We hope to have a fix available next week.

[AdrianoRNascimento]

Thank you @nr-ahemsath

However we are using the Docker (Linux) approach described here:
https://docs.newrelic.com/install/dotnet/?deployment=linux&docker=yesDocker

Is it possible to define the agent version in this command?

# Install the agent
RUN apt-get update && apt-get install -y wget ca-certificates gnupg \
&& echo 'deb http://apt.newrelic.com/debian/ newrelic non-free' | tee /etc/apt/sources.list.d/newrelic.list \
&& wget https://download.newrelic.com/548C16BF.gpg \
&& apt-key add 548C16BF.gpg \
&& apt-get update \
&& apt-get install -y newrelic-dotnet-agent \
&& rm -rf /var/lib/apt/lists/*

[reza-toorani]

Hi @nr-ahemsath
Thank you for your response. As we have a live production, it is a major concern for us. It would be greatly appreciated if you could suggest an alternative method to disable it. Currently, we are utilizing the docker approach mentioned by @AdrianoRNascimento.

[nr-ahemsath]

@reza-toorani You can pin the version of the agent used in your containerized application by modifying the line in your Dockerfile that installs the agent to look like apt-get install -y newrelic-dotnet-agent=10.18.0.

[reza-toorani]

@nr-ahemsath
thanks man for responding in an off day

[reza-toorani]

Hi @nr-ahemsath
I followed your suggestion, but the problem persists.

[nr-ahemsath]

@reza-toorani I'm sorry that didn't help. I'm not sure what could be going on, assuming that you rebuilt your containers and restarted your applications after making that change. Are you able to open up a support ticket with New Relic support? We probably need to get into some specific details of your environment in order to troubleshoot further, and for your privacy it would be better to do that through support than in public on GitHub.

[AdrianoRNascimento]

Hi @nr-ahemsath
Thank you very much for your support.
Defining the version worked well for me.

@reza-toorani
Please, confirm you have the following RUN instruction:

RUN apt-get update && apt-get install -y wget ca-certificates gnupg \
    && echo 'deb http://apt.newrelic.com/debian/ newrelic non-free' | tee /etc/apt/sources.list.d/newrelic.list \
    && wget https://download.newrelic.com/548C16BF.gpg \
    && apt-key add 548C16BF.gpg \
    && apt-get update \
    && apt-get install -y newrelic-dotnet-agent=10.18.0 \
    && rm -rf /var/lib/apt/lists/*

[reza-toorani]

@nr-ahemsath I have created a support ticket.

[reza-toorani]

@AdrianoRNascimento Thank you for your message. Yes, I did exactly as you shared here.