otel-webserver-module Fails to run on Redhat 7

[jseiser]

Describe your environment

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"

[root@ip-10-56-1-18 ld.so.conf.d]# nginx -V
nginx version: nginx/1.22.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[root@ip-10-56-1-18 ld.so.conf.d]# nginx -t
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[root@ip-10-56-1-18 ld.so.conf.d]# ldconfig -v

/opt/opentelemetry-webserver-sdk/sdk_lib/lib:
        libopentelemetry_exporter_otlp_grpc.so -> libopentelemetry_exporter_otlp_grpc.so
        libopentelemetry_webserver_sdk.so -> libopentelemetry_webserver_sdk.so
        libopentelemetry_common.so -> libopentelemetry_common.so
        libopentelemetry_trace.so -> libopentelemetry_trace.so
        libopentelemetry_exporter_ostream_span.so -> libopentelemetry_exporter_ostream_span.so
        libopentelemetry_otlp_recordable.so -> libopentelemetry_otlp_recordable.so
        libopentelemetry_resources.so -> libopentelemetry_resources.so

partial Config

load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.22.0/ngx_http_opentelemetry_module.so;

http {
  # Nginx will handle gzip compression of responses from the app server
  gzip on;
  gzip_proxied any;
  gzip_types text/plain application/json;
  gzip_min_length 1000;
  include /etc/nginx/conf.d/opentelemetry_module.conf;

# configuration file /etc/nginx/conf.d/opentelemetry_module.conf:
NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelExporterEndpoint tempo.jaeger.dev.company.network:4317;
NginxModuleServiceName RBCS;
NginxModuleServiceNamespace DEV;
NginxModuleServiceInstanceId jaeger;
NginxModuleResolveBackends ON;
NginxModuleTraceAsError On;

Steps to reproduce

1. Boot up RHEL 7 AMI
2. Install nginx from the nginx stable repo
3. Download/install the latest otel-webserver-module release
4. create /etc/ld.so.conf.d/nginx.conf and add /opt/opentelemetry-webserver-sdk/sdk_lib/lib to the file
5. Create /etc/nginx/conf.d/opentelemetry_module.conf
6. Start NGINX

What is the expected behavior?
I would expect to see traces, we use the same config, on debian docker images and have no problems.

What is the actual behavior?

2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
2023/11/13 19:50:26 [error] 4482#4482: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4482
2023/11/13 19:50:26 [error] 4483#4483: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4483
Error: virtual OTEL_SDK_STATUS_CODE otel::core::ApiUtils::init_boilerplate(): Invalid logging config file: "/opt/opentelemetry-webserver-sdk/conf/appdynamics_sdk_log4cxx.xml"
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 8
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: ngx_initialize_opentelemetry: Agent Core Init failed, result code is 8, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: startMonitoringRequest: Opentelemetry Agent Core did not get initialized, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: traceConfig:  Config { :(Enabled="1")(OtelExporterEndpoint="tempo.jaeger.dev.company.network:4317")(OtelSslEnabled="0")(OtelSslCertificatePath="")(OtelSpanExporter="otlp")(OtelSpanProcessor="")(OtelSampler="")(ServiceNamespace="DEV")(ServiceName="RBCS")(ServiceInstanceId="jaeger")(OtelMaxQueueSize="2048")(OtelScheduledDelayMillis="5000")(OtelExportTimeoutMillis="30000")(OtelMaxExportBatchSize="512")(ResolveBackends="1")(TraceAsError="1")(ReportAllInstrumentedModules="0")(MaskCookie="0")(MaskSmUser="0")(SegmentType="First")(SegmentParameter="2") }, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"

[shikhar1987]

We are experiencing similar issue on RHEL 7.3:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

# nginx -v
nginx version: nginx/1.23.1

[royteeuwen]

We have similar issues, for Apache:

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

httpd -v
Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Apr 28 2023 16:54:19

[Thu Jan 18 11:26:01.089991 2024] [core:notice] [pid 28469:tid 140216020469952] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 18 11:26:01.217388 2024] [mpm_event:notice] [pid 28469:tid 140216020469952] AH00489: Apache/2.4.34 (Red Hat) Communique/4.3.2 OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Thu Jan 18 11:26:01.217419 2024] [core:notice] [pid 28469:tid 140216020469952] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND'
[Thu Jan 18 11:26:01.222609 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: fixups Module: mod_http2.c
[Thu Jan 18 11:26:01.222637 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: fixups Module: mod_http2.c
[Thu Jan 18 11:26:01.222642 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: fixups Module: mod_proxy.c
[Thu Jan 18 11:26:01.222646 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: insert_filter Module: disp_apache2.c
[Thu Jan 18 11:26:01.222650 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: handler Module: disp_apache2.c
[Thu Jan 18 11:26:01.222653 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: handler Module: mod_http2.c
[Thu Jan 18 11:26:01.222661 2024] [otel_apache:error] [pid 28509:tid 140216020469952] insertHooksForStage: Instrumentation Hooks added. Stage: handler Module: mod_proxy.c
[Thu Jan 18 11:26:01.222665 2024] [otel_apache:error] [pid 28509:tid 140216020469952] otel_create_dir_config: (p == 0x564e9cb59188, dirspec == )
[Thu Jan 18 11:26:01.222669 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_enabled(ON)
[Thu Jan 18 11:26:01.222672 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_otelExporterType(otlp)
[Thu Jan 18 11:26:01.222676 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_otelExporterEndpoint(aapmbea1.be.srv.dev.sys:8200)
[Thu Jan 18 11:26:01.222679 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_otelProcessorType(simple)
[Thu Jan 18 11:26:01.222682 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_serviceName(demo)
[Thu Jan 18 11:26:01.222685 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_serviceNamespace(demo-namespace)
[Thu Jan 18 11:26:01.222688 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_serviceInstanceId(demo-id)
[Thu Jan 18 11:26:01.222691 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: Context demo-namespace:demo:demo-id:demo-namespace,demo,demo-id
[Thu Jan 18 11:26:01.222695 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_resolveBackends(ON)
[Thu Jan 18 11:26:01.222698 2024] [otel_apache:error] [pid 28509:tid 140216020469952] Config: otel_set_traceAsError(ON)
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 1
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 1
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 1
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 1

[owent]

Could you please try the latest main branch? open-telemetry/opentelemetry-cpp#2464 fixes a crash problem with gcc 4.8.

[royteeuwen]

The latest main branch works for me! But I do have a PR open to fix some other things to make it work for our setup. Would you be able to look at this @owent ? see #364

[owent]

The latest main branch works for me! But I do have a PR open to fix some other things to make it work for our setup. Would you be able to look at this @owent ? see #364

Sorry but I'm not fimalar with nginx API, maybe @seemk @tobiasstadler @kpratyus @ajaynagariya @DebajitDas could help.