My server return a 502 without err log

[flyuuo9]

I would like to using community server for report event, but I got a 502 when I tried add an application, and cannot find any useful error log.
Could you give me some advice?

Expected Behavior

Add an application and show on the dashboard.

Current Behavior

Got a 502 when Add an application

Screenshots (if appropriate):

Possible Solution

Steps to Reproduce (for bugs)

1.I enter my dashboard
2.tried to add an application
3. click the add button
4. got a 502 badgateway response from server

More Description

I checked the countly-api.log,countly-dashboard.log,and no error log, only find some error in Nginx access log:

"GET /i/apps/create?args=%7B%22name%22%3A%22aaa%22%2C%22type%22%3A%22mobile%22%2C%22timezone%22%3A%22Asia%2FShanghai%22%2C%22country%22%3A%22CN%22%7D HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-" 0.001 0.000 - - /i/apps/create 127.0.0.1:3001
"GET /o/langmap?preventRequestAbort=true HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-" 0.001 0.000 - - /o/langmap 127.0.0.1:3001
"GET /o/sources?preventGlobalAbort=true HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-" 0.000 0.000 - - /o/sources 127.0.0.1:3001
"GET /o/countly_version HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-" 0.000 0.000 - - /o/countly_version 127.0.0.1:3001

Your Environment

• Operating System and version:
  Linux version 4.4.0-157-generic (buildd@lgw01-amd64-037) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) [push] Error reporting & consistent error handling #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019

countly version:19.02.1
countly status:
* countly.service - countly-supervisor
   Loaded: loaded (/etc/systemd/system/countly.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-08-03 21:23:20 CST; 30min ago
     Docs: http://count.ly
 Main PID: 27455 (supervisord)
   CGroup: /system.slice/countly.service
           |-27455 /usr/bin/python /usr/bin/supervisord --nodaemon --configuration /data/tools/countly/bin/config/supervisord.conf
           `-27460 countly: dashboard node /data/tools/countly/frontend/express/app.

systemd[1]: Started countly-supervisor.
supervisord[27455]: 2019-08-03 21:23:20,335 CRIT Set uid to user 0
supervisord[27455]: 2019-08-03 21:23:20,343 CRIT Server 'unix_http_server' running without any HTTP authentication checking

ps -aux | grep countly
root     27455  0.0  0.2  59092 18912 ?        Ss   21:23   0:00 /usr/bin/python /usr/bin/supervisord --nodaemon --configuration /data/tools/countly/bin/config/supervisord.conf
root     27460  0.1  1.1 1264960 96068 ?       Sl   21:23   0:02 countly: dashboard node /data/tools/countly/frontend/express/app.

• Any changes you have made to your instance:

[gorkem-cetin]

Hi @bhui - what is your Ubuntu distro version ?

[ar2rsawseen]

It seems to be Ubuntu 16.04
The only thing that comes to mind is that if you configured countly to remote database, then you would only configure frontend part, and not api.
Then it would allow you to login, but would error on app creation.
But it would also leave something in countly logs, unless you use docker.

So do you have remote database or is it on the same server?
Do you use docker?
Can you try restarting countly by running countly restart and then repeating steps and then checking logs again.
Also you provided nginx access logs (which are actually disabled by default, so you have changed nginx configuration?). but can you also check error logs if those requests are even passed to api process at all?

[flyuuo9]

@gorkem-cetin @ar2rsawseen
Yes, my ubuntu is 16.04.
I installed countly as the "wget xxx | bash" style. The mongo is installed on localhost, I can connect it successfully use the command "mongo countly". The "countly restart" command I also tried several times. But the api-log only has this:

2019-08-04T02:25:56.060Z: INFO  [jobs:manager]  Starting job manager in 2306
2019-08-04T02:25:56.079Z: INFO  [jobs:scanner]  Checking plugins ["mobile","web","desktop","plugins","density","locale","browser","sources","views","enterpriseinfo","logger","systemlogs","errorlogs","populator","reports","crashes","push","star-rating","slipping-away-users","compare","server-stats","dbviewer","assistant","plugin-upload","times-of-day","compliance-hub","video-intelligence-monetization","alerts","onboarding","EChartMap"]
2019-08-04T02:25:57.865Z: INFO  [jobs:manager]  Starting job manager in 2331
2019-08-04T02:25:57.885Z: INFO  [jobs:scanner]  Checking plugins ["mobile","web","desktop","plugins","density","locale","browser","sources","views","enterpriseinfo","logger","systemlogs","errorlogs","populator","reports","crashes","push","star-rating","slipping-away-users","compare","server-stats","dbviewer","assistant","plugin-upload","times-of-day","compliance-hub","video-intelligence-monetization","alerts","onboarding","EChartMap"]
2019-08-04T02:26:00.621Z: INFO  [jobs:manager]  Starting job manager in 2343
2019-08-04T02:26:00.640Z: INFO  [jobs:scanner]  Checking plugins ["mobile","web","desktop","plugins","density","locale","browser","sources","views","enterpriseinfo","logger","systemlogs","errorlogs","populator","reports","crashes","push","star-rating","slipping-away-users","compare","server-stats","dbviewer","assistant","plugin-upload","times-of-day","compliance-hub","video-intelligence-monetization","alerts","onboarding","EChartMap"]
2019-08-04T02:26:04.347Z: INFO  [jobs:manager]  Starting job manager in 2355
2019-08-04T02:26:04.365Z: INFO  [jobs:scanner]  Checking plugins ["mobile","web","desktop","plugins","density","locale","browser","sources","views","enterpriseinfo","logger","systemlogs","errorlogs","populator","reports","crashes","push","star-rating","slipping-away-users","compare","server-stats","dbviewer","assistant","plugin-upload","times-of-day","compliance-hub","video-intelligence-monetization","alerts","onboarding","EChartMap"]

By the way, Did the supervisord process is the api process?

Nginx is existed before my installation. So I keeped the old conf file as below:

user  nginx;
worker_processes  2;
worker_cpu_affinity 00000001 00000010;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    use epoll;
    worker_connections  65535;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $request_body $cookie_SESSIONID $uri $upstream_addr';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Is this the reason?

When I run the wget on my server, it also return 502 to me.
I runned netstat -nap | grep 3001 but return nothing.

wget http://127.0.0.1:3001/i/apps/create?args=%7B%22name%22%3A%22aaaa%22%2C%22type%22%3A%22mobile%22%2C%22timezone%22%3A%22Asia%2FShanghai%22%2C%22country%22%3A%22CN%22%7D
--2019-08-04 10:00:55--  http://127.0.0.1:3001/i/apps/create?args=%7B%22name%22%3A%22aaaa%22%2C%22type%22%3A%22mobile%22%2C%22timezone%22%3A%22Asia%2FShanghai%22%2C%22country%22%3A%22CN%22%7D
Connecting to 127.0.0.1:3001... failed: Connection refused.

[ar2rsawseen]

Well, ours is here: https://github.com/Countly/countly-server/blob/master/bin/config/nginx.conf

Yours might not be optimized, but from what I see, it should work.

This is still experimental, but can you run countly healthcheck all and post output here?

[flyuuo9]

I add this to nginx.conf, it also not work

worker_processes  4;
worker_rlimit_nofile 40000;

So I execute these,
countly restart
countly healthcheck all

Countly info :
    version : 19.02.1
    path    : /data1/data/tools/countly
    status  : active (running) since Sun 2019-08-04 21:07:27 CST; 9min ago
Encountered problems when checking accessibility:
    Api is not accessible
    Dashboard is not accessible through Nginx
    Api is not accessible through Nginx
You have 67 error(s) in the last hour in your nginx error.log
/data1/data/tools/countly/bin/commands/scripts/healthcheck/ntp.sh: line 10: [: 0.297: integer expression expected
Encountered problems with process count:
    Process api.js is not found
    Too little processes for api.js workers: 0    nproc: 2

How to check the reason of Api not accessible?

[flyuuo9]

I runned the command "countly test", and find many error about eslint:target.
So I reinstall all node package, and find the 502 has gone.

rm -rf node_modules
rm package-lock.json
npm cache clear --force
npm install

Maybe the real reason is some net break happened during my installation, which make some file damaged.

[ar2rsawseen]

Yes, that is possible.
Either some npm repo is not available, or npm service itself.
But usually, that appears in the logs too :)