A lot of CPU Usage and 10000 request every second in webUI like #28

[masterwishx]

im very sorry but i again have this issue ,maybe yo can open the old one
#28

it was running about 2 days normal then it start grow up about 10000 in second ,
if im not running container for some days it again working Good for 2 days ,and then again start raising
Very strange i dont have this isshue on gregyankovoy/goaccess

becose i used his container befor i moved the Logs folder to /mnt/user/Logs/NginxProxyManager/
where i can look to find the problem ?

[OverEngineerd]

Just want to confirm that I'm also having this issue -- roughly 8,000 requests per second as reported by the web-ui.

Running the Unraid version of the image (v1.0.5) -- everything is flawless except for this one issue. Thanks for all your effort so far -- I discovered your work via a Youtube video that explained everything! https://youtu.be/-CQcEWVBjQU

[masterwishx]

Just want to confirm

Thanks, I thought I only one have this problem

[xavier-hernandez]

@masterwishx @Mission-Contro1

I deployed a new version to the develop branch. I'm adding the files now to the goaccess.conf file instead of the command line. Give it shot.

docker pull xavierh/goaccess-for-nginxproxymanager:develop

[masterwishx]

Give it shot

OK, Thanks a lot I will try.

I founded mostly requests going from Authelia and nextcloud also I have Uptime Kuma every 5 min. The container start OK but after some time it's going much more request. So I need to stop it for some days.

This very strange issue I didn't had this problem on old version of https://github.com/GregYankovoy

[OverEngineerd]

Sadly still having the same issue. It will start just fine but after a few seconds it will create the 8,000 requests once every second.

@masterwishx You and I have a very similar set up! I have Nextcloud and Uptime Kuma running as well. I will try disabling them both from NPM to see if it narrows down the issue.

[OverEngineerd]

I tried stopping NPM but the GoAccess request counter is still going strong.

[masterwishx]

I tried stopping NPM

I don't think the problem in them Becouse befor I used other same docker but with old version of goaccess 1.4 and was no problem.
I still have it but not using for now. But I checked both and the old version does not have this issue.
Now I trying the dev instead last of this docker and will post here if will be same problem.

[xavier-hernandez]

@Mission-Contro1 and @masterwishx so I just found an issue in the current develop build that actually caused the problem that you found. It might be related to what you were seeing in the past but just manifested itself differently this time.

Please pull the latest develop version:
docker pull xavierh/goaccess-for-nginxproxymanager:develop

[OverEngineerd]

Still no dice -- the behavior is unchanged. Once it first loads, the counter will remain static for a few seconds before it ticks up by 8.000 requests per second.

Also the Tx. Amount ticks up very rapidly as well.

[xavier-hernandez]

@Mission-Contro1 can you add the debug flag if you haven't already. "DEBUG=True" and browser to /goaccess_conf.html search for this flag "#GOAN_PROXY_FILES" and provide me the output of the files listed after that.

[OverEngineerd]

Thanks for the quick reply! Where do I add the debug flag? I see there's a debug/goaccess_conf.html file when I exec into the container but it doesn't contain the flag you mentioned.

[OverEngineerd]

However these are the files that are listed at /goaccess-config/proxy_logs:
log-file /goaccess-config/access_archive.log
log-file /opt/log/proxy-host-1_access.log
log-file /opt/log/proxy-host-2_access.log
log-file /opt/log/proxy-host-3_access.log

[masterwishx]

Please pull the latest develop version

From yesterday dev version until today I don't
see this issue, but have spikes some times not every second and still have much requests at sum I think, but maybe needs more time for checking for night I shut down server.
I will update to lasted dev and will check also

[masterwishx]

Sorry but checked now problem still exist every second. Much requests and TX amount.

[masterwishx]

can you add the debug flag if you haven't already. "DEBUG=True" and browser to /goaccess_conf.html search for this flag "#GOAN_PROXY_FILES" and provide me the output of the files listed after that.

can you please explain where to add this flag ?
do you mean add variable to container ?

[masterwishx]

in container console /goaccess-config # cat goaccess.conf

[xavier-hernandez]

Hmm... That screenshot looks good.

I added a variable for the container recently called DEBUG but you got me the information I needed.

• DEBUG=True

Can you try skipping the archived logs?

• SKIP_ARCHIVED_LOGS=True

[masterwishx]

OK i will try

[masterwishx]

without arhived logs should be only for 3 day log ...

I forgot to mention that i have another log file from old container of GregYankovoy called proxy_log
That is still generating in NPM by some of containers , added lines
"access_log /config/logs/proxy_logs.log proxy;
error_log /config/log/proxy_errors.log warn;"

but in your container its read all "proxy-host-_access." what is much better way , so i dont think its make issue ?!?

[masterwishx]

i checked without arhived log the problem is still exist at end of a day

[xavier-hernandez]

That additional file can't be the issue. I can't figure out what might be causing the problem. Do you have any additional customizations to NPM?

[OverEngineerd]

No other customizations on my end -- 3 Proxy Hosts with 3 different certs through Let's Encrypt, that's all. I haven't been able to get any closer to figuring out what's wrong.

[xavier-hernandez]

@Mission-Contro1 can you post a screenshot of all your files in that directory?

[OverEngineerd]

Sure -- here is /opt/logs/

[masterwishx]

Do you have any additional customizations to NPM?

No
I but I use cloudflare dns proxy also.
The file is still here becose issue in this container
I mean I need it for check the old container but I don't have same problem in it.
I have 5000 request in a day in container and in cloudflare analytic.
But here I have 1000000 in 3 days.
I'm also don't understand what the problem maybe its in 1.55 version.
Maybe the author of program can help?

[xavier-hernandez]

@masterwishx I read the thread and I think I'd recommend as well what allinurl said. If you can run goaccess directly against the logs and see if it happens that'd be best.

If you need help with that maybe I can lend a hand.

[masterwishx]

OK, I can try but I don't know what is mean directly? Without the docker?
How I can do it?

[masterwishx]

If you can help with it will be awesome

[xavier-hernandez]

sorry been busy for a bit, i'll get to this by the end of the week

[tonyle8]

Ok good news! I was able to resolve it: It turns out I was correct in regards to the file access. This issue only happens when you are using Bind mount (connecting directly to a host drive). Switching to named volumes fixes the problem and works perfectly.

[xavier-hernandez]

@tonyle8 can you provide an example of what you changed? Also, are you using the docker container directly (docker-compose), portainer, or through the unraid template?

[tonyle8]

@tonyle8 can you provide an example of what you changed? Also, are you using the docker container directly (docker-compose), portainer, or through the unraid template?

See below for docker compose on portainer. My ngix app volume is set to the same "data" volume where saves the logs

goaccess:
image: xavierh/goaccess-for-nginxproxymanager:latest
container_name: goaccess
restart: always
environment:
- PUID=0
- PGID=0
- TZ=America/New_York
- SKIP_ARCHIVED_LOGS=True #optional
- DEBUG=False #optional
- BASIC_AUTH=False #optional
- BASIC_AUTH_USERNAME=user #optional
- BASIC_AUTH_PASSWORD=pass #optional
- EXCLUDE_IPS=127.0.0.1 #optional - comma delimited
ports:
- '8206:7880'
volumes:
#- /C/Docker/NgixProxyManager/data:/opt/log #Bind mount does not work
- data:/opt/log #Named Volume

[xavier-hernandez]

@tonyle8 - I'll add this to the README file as an option to try if you're getting this issue. Thanks!

[masterwishx]

I'm not sure if I can read the lines in each log file and figure out if maybe a file is growing to large or maybe you'll be able to tell on your end which file might cause the problem. I'll look into it.

after stopped container for a week , started it again and see same issue some 20000 every second ...
i can see hits from today and goes back , for now 5 days back.
all hits for authelia and nextcloud hosts i think ,looks by picture ...

[tonyle8]

@masterwishx
try my fix at the top and see if that works for you

[masterwishx]

try my fix at the top and see if that works for you

im using unraid ...

[tonyle8]

@masterwishx looks like you are using bind mount with unraid. Try switching to named volume if it supports it?

[masterwishx]

@tonyle8

Try switching to named volume if it supports it

OK Thanks

@xavier-hernandez

I'll add this to the README file as an option to try if you're getting this issue

can we use it in unraid ?

[xavier-hernandez]

@masterwishx so working off of @tonyle8 explanation/assumption can you try to change the access mode for the host path? Try changing it to Read Only - Slave. If that doesn't work then try Read Only - Shared.

[OverEngineerd]

No dice with changing the access modes on the containers:
NPM Official: Read/Write - Slave
Goaccess for NPM: Read Only - Slave

Also doesn't work for shared mode. :(

[masterwishx]

Read Only - Slave. If that doesn't work then try Read Only - Shared.

i have Read Only now i will try these ...

[masterwishx]

if i remember correctly Read Only Slave its for Unassigned Devices but i have in /mnt/user/Logs/NginxProxyManager/
but will try anyway ...

Also doesn't work for shared mode. :(

Do you also still having this issue in unraid ?

[OverEngineerd]

Yes, still having this issue in Unraid. I'm going to try and create a volume in docker/unraid and have NPM write the logs to the volume and GoAccess read the logs from it.

[masterwishx]

Yes, still having this issue in Unraid

Please write here if it works or not. Thanks

[OverEngineerd]

Yes, still having this issue in Unraid

Please write here if it works or not. Thanks

I wasn't able to get it to work -- combination of running low on time and there not being a way to do it via Unraid's UI easily.

I pointed NPM to a Docker volume I created but it failed to start because I had two mount points at the same location. Then the image uninstalled itself and I had to reinstall. Nasty bug.

[masterwishx]

I wasn't able to get it to work

i see , thanks for sharing this try

[masterwishx]

@xavier-hernandez
Maybe the problem is becose container dont have bind mont config and using volume name mount only if problem can be fixed by name volume like tonyle8 said?!?

[xavier-hernandez]

@masterwishx I don't believe that is how unraid works. Maybe I can create a temporary build where the logs will be copied into the container itself and see if that fixes the issue. We can try that at least.

[masterwishx]

Sure, if you can, we can do it. Then container need to copy logs for some cron times? Or how?

Also wanted to check again the container yesterday , it's stopped for now.
And the problem is starts after some time (hours)
Also wanted to dedug with dev tools but don't have a lot of knowledge. If you can point where to look maybe? Also was some errors.. I can post here if it will help

[xavier-hernandez]

@masterwishx - do you leave the website open for hours or do you close it and come back?

Yes, some sort of cron job or sleep timer. This would be just for you so maybe just a sleep timer. If it works then maybe I can make it a cron job.

[masterwishx]

@xavier-hernandez

Yes, some sort of cron job

Thanks , i can help whatever need and what i can to find the isshue ...

do you leave the website open

Tryed both ways, the end is same :(
Last time i leaved open becose of chrome develop tools (whanted to check if i can find somethink)...

[masterwishx]

I think the problem is solved ?!?
Tested for two days . befor not used for some time . will check it more ....

[masterwishx]

i meaned no more increasing requsets ,can someone also confirm ?

[stavros-k]

Yes I can confirm, it seem to run stable now.

[OverEngineerd]

First impressions are good -- it appears to be working! I'll leave it running for a few days.

What did you change this time around?

[xavier-hernandez]

First impressions are good -- it appears to be working! I'll leave it running for a few days.

What did you change this time around?

@Mission-Contro1 if this question is for me, a lot has changed. GoAccess was upgraded twice I believe and the way the processes were running has also changed. I was never able to reproduce this issue on my end so I'm not totally sure what could have fixed it if it is stable now.

[masterwishx]

What did you change this time around?

if you read above, for some one helped named volume even befor this was fixed by author ....
also very interesting how was fixed but more important is working fine now, tested for 3 days and cheking more...

on unraid i have another problem , on autostart somehow #86
but after if start manually its OK

[xavier-hernandez]

Marking this ticket as closed since 3 users have confirmed this is no longer an issue. Please re-open or open a new issue if this reoccurs. Thanks.