-
Notifications
You must be signed in to change notification settings - Fork 162
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SonarQube flood stage disk watermark [95%] exceeded on #40
Comments
Thanks for the detailed analysis. I agree none of the 3 would take ownership directly. So it's on a motivated individual from this community to submit an issue & PR to SQ for a fix. If we can adjust this by adding a setting to sonar.properties that gets passed to ES, here's an open PR we could use as an example |
Just want to add a 'me too' Would love a solution to this too..... |
Adding the ES documentation page for config settings that are relevant to this discussion. |
I would be surprised if this actually works after a few days. I did exactly the same going to a P1 instance. Its not the disk size of the app service that is the problem. It is that Microsoft use a symbol link to attach a drive under the App Service VMs D: drive which as far as I can tell is 32GB no matter what plan you choose. In Elasticsearch the Java File System library is used to check the root drive free disk space which shows 32GB give or take. If it checked the actual folder that Elasticsearch index was stored in it would see the full 50GB or 250GB depending on what plan you are on in the App Service. I am reverting to hosting SonarQube through an Azure Windows VM via IIS reverse proxy for now. |
I figured out a way to disable the disk check. Changing this setting did persist across App Service restarts.
Lastly, ES does log this warning about http.enabled being deprecated. Once SQ updates to a newer version of ES these steps may need to be reworked. From ES.log:
Reference For The Future:
|
Excellent I couldn't get the ES API to work from the App Service and the port was the missing element I didn't configure. I would be interested to see where the setting is persisted to in ES for it to be resilient to App Service restarts. @vanderby do you think it would be worth adding this to the PowerShell script wrapper as an option so the API call is made on startup to verify the disk allocation check? I'm thinking this could help others who will come across the same problem and also would remove the need to manually call the API in Kudo. I appreciate the setting is disabled in future versions but for now it is the only working solution and can be looked at when SQ and ES is updated. Thank you. |
@inoreip, that is next on my list when I get time. It will be a bit more tricky since we can't communicate with the ES service until it starts. My initial thoughts are to spin up a separate PowerShell job that will poll for the ES service to start then make the web request to change the setting. Adding the setting to the app service to enable ES http will be easy. In the meantime, verify if this solves your problem and let me know if it does not. |
@vanderby I've been trying to get the PowerShell commands to successfully execute via the Kudu interface. Unfortunately I stumbled upon the protocol validation issue with Have you managed to successfully call the ES API via Kudu PowerShell? |
I ran into the protocol violation too when I was trying to send a request to the ES endpoint that was not the HTTP endpoint. After setting sonar.search.httpPort in appsettings did you verify SQ restarted? You should see a message like this in ES.log that verifies the port the HTTP endpoint is listening on.
Kudu PowerShell was the only thing I was using when I was working with this issue. |
@vanderby My fault I set I can confirm it also handled an App Service Stop and Start. I could see in the es.log file that the setting was changed from enabled to disabled so I didn't have to call the API again. Thank you |
Interesting.... Thanks for the hard work. |
I should add I agree with @inoreip - where is this being stored in ES? If not elastic.yaml then where? Because would it be easier set the setting in there, rather than open a port and execute a REST call. |
Thought: Once the ES settings have been set, can we close off the listener again? |
@Rincey As far as I can tell once you set this setting you can disable the http endpoint. I don't know where ES stores this setting so I won't guess if upgrading to different SQ version won't reset it. |
Thanks so much. We had encountered this issue a few weeks ago, tried several things and ended up re installing after breaking something trying to fix it. When it came back we found this issue, have been trying things and following this. Once we saw the work around we implemented it, and no issues. This is a great way for us to get up and running quickly using SonaQube. |
Adding some newer context since I've just gone through this with SonarQube 8.9.2, which is bundled with (newer) ElasticSearch 7. Updated steps from @vanderby original guide:
Relevant notes from the ES upgrade ticket SONAR-12686:
One thing that wasn't discussed was whether this workaround needs to be applied with each SonarQube upgrade. I'm guessing so since each version contains its own distribution of ElasticSearch. Hence, I'm making a note to always run this command after every upgrade. |
Hi, although this post is about SonarQube on Azure, I had the same problem with SonarQube on MacOS. |
I am raising this issue here as it is specific to the way Azure App Service allocates disk space on the D drive and D:/home folder. I have not been able to work around this and there are 1 or 2 posts online with the same issue with no robust resolution.
WARN es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [0WrMWG4XSEObU4chJQUUvg][sonarqube][D:\home\site\wwwroot\sonarqube-7.9\data\es6\nodes\0] free: 1.2gb[3.9%], all indices on this node will be marked read-only
elastic/elasticsearch#53233
#35
The above post mentioned scaling up the App Service Plan but this doesn't work if your index is approaching 30GB and is marked as Closed.
The Elasticsearch index in SonarQube is set to a 95% disk threshold meaning that when used disk space is going to exceed 95% then Elasticsearch writes a warning in the es.log file and tries to write to another node. However there are no other nodes so the indexing enters an infinite loop. In Elasticsearch you can tweak these settings in the elasticsearch.yml file, e.g. you can turn this check off. However SonarQube seems to ignore this yml file and creates a temporary one on startup in a temp folder. I tried adding
cluster.routing.allocation.disk.threshold_enabled: false
to the yml file which was ignored on App Service restart. I did this to prevent Elasticsearch from checking disk space as I have allocated way more than enough in the App Service Plan.The more canny might state well just add more disk space (if only it was that straight forward). Scaling up an App Service Plan adds more disk space to D:/home but not D:/ which seems to be the OS disk and is fixed around 32GB. D:/home is what looks like a symbol link or share which is actually a dedicated disk from the App Service Plan. I have the plan at 250GB which I can see in Kudu (Azure environment tool). It looks like Elasticsearch is using the free disk space on D: and not the folder where the index is stored e.g. D:/home/site/wwwroot/.../data.
I suspect this issue cannot be solved in Azure, SonarQube doesn't honour Elasticsearch settings and Elasticsearch just uses the Java libraries to check free disk space. What I am trying to say is I don't think any of the 3 would take ownership and solve this problem, rendering the App Service deployment only fit for small deployments with small indexes. This is a shame as in my opinion this deployment is the most manageable and robust option.
If anybody is able to tell me otherwise I would be delighted to see a resolution. Thank you.
The text was updated successfully, but these errors were encountered: