Skip to content

feat(dashmate): collect docker stats in the doctor command #2180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 29, 2024
Merged

feat(dashmate): collect docker stats in the doctor command #2180

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7065d4b
feat(dashmate): collect docker stats in the doctor command
shumkov Sep 28, 2024
3fbf034
fix: multiple issues
shumkov Sep 28, 2024
7393583
refactor: little refactoring
shumkov Sep 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions packages/dashmate/src/doctor/analyse/analyseServiceContainersFactory.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@ export default function analyseServiceContainersFactory(
const servicesNotStarted = [];
const servicesFailed = [];
const servicesOOMKilled = [];
const servicesHighResourceUsage = [];

for (const service of services) {
const dockerInspect = samples.getServiceInfo(service.name, 'dockerInspect');
const dockerStats = samples.getServiceInfo(service.name, 'dockerStats');

if (!dockerInspect) {
continue;
Expand All @@ -47,6 +49,19 @@ export default function analyseServiceContainersFactory(
service,
});
}

if (dockerStats) {
const cpuUsage = dockerStats.cpuStats.cpuUsage.totalUsage / dockerStats.cpuStats.systemCpuUsage;
const memoryUsage = dockerStats.memoryStats.usage / dockerStats.memoryStats.limit;

if (cpuUsage > 0.8 || memoryUsage > 0.8) {
servicesHighResourceUsage.push({
service,
cpuUsage,
memoryUsage,
});
}
}
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Prevent possible division by zero and handle missing properties

In the calculations of cpuUsage and memoryUsage, there is a risk of division by zero if dockerStats.cpuStats.systemCpuUsage or dockerStats.memoryStats.limit are zero or undefined. Additionally, accessing nested properties without verifying their existence may lead to runtime errors.

Apply this diff to handle these cases:

 if (dockerStats) {
+  if (
+    dockerStats.cpuStats &&
+    dockerStats.cpuStats.cpuUsage &&
+    dockerStats.cpuStats.systemCpuUsage > 0 &&
+    dockerStats.memoryStats &&
+    dockerStats.memoryStats.limit > 0
+  ) {
     const cpuUsage = dockerStats.cpuStats.cpuUsage.totalUsage / dockerStats.cpuStats.systemCpuUsage;
     const memoryUsage = dockerStats.memoryStats.usage / dockerStats.memoryStats.limit;

     if (cpuUsage > 0.8 || memoryUsage > 0.8) {
       servicesHighResourceUsage.push({
         service,
         cpuUsage,
         memoryUsage,
       });
     }
+  } else {
+    // Handle cases where stats are unavailable or denominators are zero
+    console.warn(`Docker stats for service ${service.name} are incomplete or invalid.`);
+  }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (dockerStats) {
const cpuUsage = dockerStats.cpuStats.cpuUsage.totalUsage / dockerStats.cpuStats.systemCpuUsage;
const memoryUsage = dockerStats.memoryStats.usage / dockerStats.memoryStats.limit;
if (cpuUsage > 0.8 || memoryUsage > 0.8) {
servicesHighResourceUsage.push({
service,
cpuUsage,
memoryUsage,
});
}
}
if (dockerStats) {
if (
dockerStats.cpuStats &&
dockerStats.cpuStats.cpuUsage &&
dockerStats.cpuStats.systemCpuUsage > 0 &&
dockerStats.memoryStats &&
dockerStats.memoryStats.limit > 0
) {
const cpuUsage = dockerStats.cpuStats.cpuUsage.totalUsage / dockerStats.cpuStats.systemCpuUsage;
const memoryUsage = dockerStats.memoryStats.usage / dockerStats.memoryStats.limit;
if (cpuUsage > 0.8 || memoryUsage > 0.8) {
servicesHighResourceUsage.push({
service,
cpuUsage,
memoryUsage,
});
}
} else {
// Handle cases where stats are unavailable or denominators are zero
console.warn(`Docker stats for service ${service.name} are incomplete or invalid.`);
}
}

}

const problems = [];
Expand Down Expand Up @@ -103,6 +118,20 @@ export default function analyseServiceContainersFactory(
problems.push(problem);
}

if (servicesHighResourceUsage.length > 0) {
for (const highResourceService of servicesHighResourceUsage) {
const description = `Service ${highResourceService.service.title} is consuming too many resources. CPU usage: ${(highResourceService.cpuUsage * 100).toFixed(2)}%, Memory usage: ${(highResourceService.memoryUsage * 100).toFixed(2)}%.`;

const problem = new Problem(
description,
'Consider upgrading your system resources or report in case of misbehaviour.',
SEVERITY.MEDIUM,
);

problems.push(problem);
}
}

return problems;
}

Expand Down
6 changes: 6 additions & 0 deletions packages/dashmate/src/listr/tasks/doctor/collectSamplesTaskFactory.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import providers from '../../../status/providers.js';
import hideString from '../../../util/hideString.js';
import obfuscateObjectRecursive from '../../../util/obfuscateObjectRecursive.js';
import validateSslCertificateFiles from '../../prompts/validators/validateSslCertificateFiles.js';
import si from 'systeminformation';

/**
*
Expand Down Expand Up @@ -326,6 +327,10 @@ export default function collectSamplesTaskFactory(
dockerCompose.logs(config, [service.name], { tail: 300000 }),
])).map((e) => e.value || e.reason);

const containerId = inspect?.Id;

const dockerStats = containerId ? await si.dockerContainerStats(containerId) : undefined;

if (logs?.out) {
// Hide username & external ip from logs
logs.out = logs.out.replaceAll(
Expand Down Expand Up @@ -354,6 +359,7 @@ export default function collectSamplesTaskFactory(
ctx.samples.setServiceInfo(service.name, 'stdOut', logs?.out);
ctx.samples.setServiceInfo(service.name, 'stdErr', logs?.err);
ctx.samples.setServiceInfo(service.name, 'dockerInspect', inspect);
ctx.samples.setServiceInfo(service.name, 'dockerStats', dockerStats);
}),
);
},
Expand Down