Not picking the log based on grok pattern

[indrajithgihan]

I have the following log and grok pattern which is matching when I check from https://grokdebug.herokuapp.com/. But this marks as non-matching log with the given grok when parsed through the fluent-plugin-grok-parser. Having issue only with below log. Other logs so far are matching. Appreciate if you could find the cause of this as this was the first time I encountered such issue having worked with the plugin over two years.

Log:
2021-10-06 23:57:38:370482|[1000]|ncellvariableproductofferapp-offer-management-deployment-5fzm5m|ERROR|alarm-log|variableProductOfferManagement|variable-product-offer-management-ms|e493db0f6c4207|FUNCTIONAL|Domain\Exceptions\ExternalException: Active Offer not found. in /var/www/site/domain/Usecase/ActiveOfferUseCase.php:123 Stack trace: #0 /var/www/site/app/Http/Controllers/ActiveOfferController.php(101): Domain\Usecase\ActiveOfferUseCase->activeOffer() #1 /var/www/site/vendor/illuminate/container/BoundMethod.php(36): App\Http\Controllers\ActiveOfferController->activeOffer() #2 /var/www/site/vendor/illuminate/container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container{closure}() #3 /var/www/site/vendor/illuminate/container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #4 /var/www/site/vendor/illuminate/container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod() #5 /var/www/site/vendor/illuminate/container/Container.php(590): Illuminate\Container\BoundMethod::call() #6 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(376): Illuminate\Container\Container->call() #7 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(342): Laravel\Lumen\Application->callControllerCallable() #8 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(316): Laravel\Lumen\Application->callLumenController() #9 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(278): Laravel\Lumen\Application->callControllerAction() #10 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(263): Laravel\Lumen\Application->callActionOnArrayBasedRoute() #11 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(165): Laravel\Lumen\Application->handleFoundRoute() #12 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns{closure}() #13 /var/www/site/vendor/laravel/lumen-framework/src/Routing/Pipeline.php(52): call_user_func() #14 /var/www/site/app/Http/Middleware/XssSanitization.php(27): Laravel\Lumen\Routing\Pipeline->Laravel\Lumen\Routing{closure}() #15 /var/www/site/vendor/illuminate/pipeline/Pipeline.php(171): App\Http\Middleware\XssSanitization->handle() #16 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() #17 /var/www/site/vendor/laravel/lumen-framework/src/Routing/Pipeline.php(32): call_user_func() #18 /var/www/site/app/Http/Middleware/CorsMiddleware.php(36): Laravel\Lumen\Routing\Pipeline->Laravel\Lumen\Routing{closure}() #19 /var/www/site/vendor/illuminate/pipeline/Pipeline.php(171): App\Http\Middleware\CorsMiddleware->handle() #20 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() #21 /var/www/site/vendor/laravel/lumen-framework/src/Routing/Pipeline.php(32): call_user_func() #22 /var/www/site/app/Http/Middleware/RequestMiddleware.php(31): Laravel\Lumen\Routing\Pipeline->Laravel\Lumen\Routing{closure}() #23 /var/www/site/vendor/illuminate/pipeline/Pipeline.php(171): App\Http\Middleware\RequestMiddleware->handle() #24 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() #25 /var/www/site/vendor/laravel/lumen-framework/src/Routing/Pipeline.php(32): call_user_func() #26 /var/www/site/vendor/illuminate/pipeline/Pipeline.php(105): Laravel\Lumen\Routing\Pipeline->Laravel\Lumen\Routing{closure}() #27 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(413): Illuminate\Pipeline\Pipeline->then() #28 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(171): Laravel\Lumen\Application->sendThroughPipeline() #29 /var/www/site/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(108): Laravel\Lumen\Application->dispatch() #30 /var/www/site/public/index.php(28): Laravel\Lumen\Application->run() #31 {main}

Grok pattern:

(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}.%{NONNEGINT})|[(?[^\]]+)]|%{IPORHOST:pod_instance}|%{LOGLEVEL:severity}|%{GREEDYDATA}|%{GREEDYDATA:application}|%{GREEDYDATA:microservice}|%{UUID:uuid}|(?<message_type>[^|])|(?[^|])(?<clearance_options>(?:(|.)|(|AD.)|()))

Fluentd config with grok pattern:

@type tail path /data/logs/*/alarm/*.log pos_file /data/logs/alarm.log.pos #exclude_path ["/data/logs/*/alarm/*.gz", "/data/logs/*/alarm/*.zip", "/data/logs/*/alarm/*-%Y-%m-%d.log"] exclude_path ["/data/logs/*/alarm/*.gz", "/data/logs/*/alarm/*.zip"] pos_file_compaction_interval 168h read_bytes_limit_per_second 16384 path_key tailed_path tag ms-logs-alarm read_from_head true follow_inodes true #refresh_interval 20s #enable_stat_watcher false @type none #format json time_format %Y-%m-%dT%H:%M:%S.%NZ @type parser key_name message reserve_data true @type grok grok_failure_key grokfailure pattern (?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}.%{NONNEGINT})\|\[(?[^\]]+)\]\|%{IPORHOST:pod_instance}\|%{LOGLEVEL:severity}\|%{GREEDYDATA}\|%{GREEDYDATA:application}\|%{GREEDYDATA:microservice}\|%{UUID:uuid}\|(?[^|]*)\|(?[^|]*)(?(?:(\|.*)|(|AD.*)|())) @type record_transformer enable_ruby host.name "#{Socket.gethostname}" remote_ip "#{(Socket.ip_address_list.detect do |intf| intf.ipv4_private? end).ip_address}" log.file.path "${record['tailed_path']}" #hostname "#{Socket.gethostname}" #remote_ip "%#{@metadata.ip_address}" @type rewrite_tag_filter num_threads 2 key grokfailure pattern /.*/ tag grokfailure_log_alarm key application pattern /.*/ tag ms-logs-alarm-matched

[indrajithgihan]

@okkez Appreciate if you could provide an update on this

[indrajithgihan]

Found the issue is not with grok parser plugin. Hence closing