Skip to content

HDDS-12520. Move auditparser under debug. #8041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 3 commits into from
Mar 13, 2025
Merged

HDDS-12520. Move auditparser under debug. #8041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5082d93
HDDS-12520. Move auditparser under debug.
nandakumar131 Mar 10, 2025
463f82b
Merge remote-tracking branch 'upstream/master' into HDDS-12520
nandakumar131 Mar 12, 2025
b444dcf
Deprecated warning message added.
nandakumar131 Mar 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions hadoop-hdds/docs/content/tools/AuditParser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,25 +40,25 @@ UNIQUE(datetime,level,logger,user,ip,op,params,result))

Usage:
{{< highlight bash >}}
ozone auditparser <path to db file> [COMMAND] [PARAM]
ozone debug auditparser <path to db file> [COMMAND] [PARAM]
{{< /highlight >}}

To load an audit log to database:
{{< highlight bash >}}
ozone auditparser <path to db file> load <path to audit log>
ozone debug auditparser <path to db file> load <path to audit log>
{{< /highlight >}}
Load command creates the audit table described above.

To run a custom read-only query:
{{< highlight bash >}}
ozone auditparser <path to db file> query <select query enclosed within double quotes>
ozone debug auditparser <path to db file> query <select query enclosed within double quotes>
{{< /highlight >}}

Audit Parser comes with a set of templates(most commonly used queries).

To run a template query:
{{< highlight bash >}}
ozone auditparser <path to db file> template <templateName>
ozone debug auditparser <path to db file> template <templateName>
{{< /highlight >}}

Following templates are available:
Expand Down
8 changes: 4 additions & 4 deletions hadoop-hdds/docs/content/tools/AuditParser.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,25 +38,25 @@ UNIQUE(datetime,level,logger,user,ip,op,params,result))

用法：
{{< highlight bash >}}
ozone auditparser <数据库文件的路径> [命令] [参数]
ozone debug auditparser <数据库文件的路径> [命令] [参数]
{{< /highlight >}}

将审计日志加载到数据库：
{{< highlight bash >}}
ozone auditparser <数据库文件的路径> load <审计日志的路径>
ozone debug auditparser <数据库文件的路径> load <审计日志的路径>
{{< /highlight >}}
Load 命令会创建如上所述的审计表。

运行一个自定义的只读查询：
{{< highlight bash >}}
ozone auditparser <数据库文件的路径> query <双引号括起来的 select 查询>
ozone debug auditparser <数据库文件的路径> query <双引号括起来的 select 查询>
{{< /highlight >}}

审计解析起自带了一些模板（最常用的查询）

运行模板查询：
{{< highlight bash >}}
ozone auditparser <数据库文件的路径 template <模板名称>
ozone debug auditparser <数据库文件的路径 template <模板名称>
{{< /highlight >}}

Ozone 提供了以下模板：
Expand Down
10 changes: 5 additions & 5 deletions hadoop-ozone/dist/src/main/smoketest/auditparser/auditparser.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,15 @@ Testing audit parser

${logdir} = Get Environment Variable OZONE_LOG_DIR /var/log/ozone
${logfile} = Execute ls -t "${logdir}" | grep om-audit | head -1
Execute ozone auditparser "${auditworkdir}/audit.db" load "${logdir}/${logfile}"
${result} = Execute ozone auditparser "${auditworkdir}/audit.db" template top5cmds
Execute ozone debug auditparser "${auditworkdir}/audit.db" load "${logdir}/${logfile}"
${result} = Execute ozone debug auditparser "${auditworkdir}/audit.db" template top5cmds
Should Contain ${result} ALLOCATE_KEY
${result} = Execute ozone auditparser "${auditworkdir}/audit.db" template top5users
${result} = Execute ozone debug auditparser "${auditworkdir}/audit.db" template top5users
Run Keyword If '${SECURITY_ENABLED}' == 'true' Set username
Should Contain ${result} ${user}
${result} = Execute ozone auditparser "${auditworkdir}/audit.db" query "select count(*) from audit where op='CREATE_VOLUME' and RESULT='SUCCESS'"
${result} = Execute ozone debug auditparser "${auditworkdir}/audit.db" query "select count(*) from audit where op='CREATE_VOLUME' and RESULT='SUCCESS'"
${result} = Convert To Number ${result}
Should be true ${result}>=1
${result} = Execute ozone auditparser "${auditworkdir}/audit.db" query "select count(*) from audit where op='CREATE_BUCKET' and RESULT='SUCCESS'"
${result} = Execute ozone debug auditparser "${auditworkdir}/audit.db" query "select count(*) from audit where op='CREATE_BUCKET' and RESULT='SUCCESS'"
${result} = Convert To Number ${result}
Should be true ${result}>=${buckets}
13 changes: 7 additions & 6 deletions hadoop-ozone/dist/src/shell/ozone/ozone
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ function ozone_usage
ozone_add_option "--jvmargs arguments" "append JVM options to any existing options defined in the OZONE_OPTS environment variable. Any defined in OZONE_CLIENT_OPTS will be append after these jvmargs"
ozone_add_option "--validate (continue)" "validates if all jars as indicated in the corresponding OZONE_RUN_ARTIFACT_NAME classpath file are present, command execution shall continue post validation failure if 'continue' is passed"

ozone_add_subcommand "auditparser" client "runs audit parser tool"
ozone_add_subcommand "classpath" client "prints the class path needed for running ozone commands"
ozone_add_subcommand "datanode" daemon "run a HDDS datanode"
ozone_add_subcommand "envvars" client "display computed Hadoop environment variables"
Expand Down Expand Up @@ -83,10 +82,6 @@ function ozonecmd_case
RATIS_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false ${RATIS_OPTS}"

case ${subcmd} in
auditparser)
OZONE_CLASSNAME=org.apache.hadoop.ozone.audit.parser.AuditParser
OZONE_RUN_ARTIFACT_NAME="ozone-tools"
;;
Comment on lines -86 to -89
Copy link
Contributor

@adoroszlai adoroszlai Mar 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a compatibility hack to handle the old command (without including it in the usage help)?

Something like:

OZONE_SUBCMD=$1

if [[ "$OZONE_SUBCMD" == "auditparser" ]]; then
  OZONE_SUBCMD=debug
else
  shift
fi

at:

ozone/hadoop-ozone/dist/src/shell/ozone/ozone

Lines 311 to 312 in 1202f6d

OZONE_SUBCMD=$1
shift

Copy link
Contributor Author

@nandakumar131 nandakumar131 Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense, let me add it.
I will also add a deprecated message and print a suggestion message to use the new command.

classpath)
if [[ "$#" -gt 0 ]]; then
OZONE_RUN_ARTIFACT_NAME="$1"
Expand Down Expand Up @@ -309,7 +304,13 @@ if [[ $# = 0 ]]; then
fi

OZONE_SUBCMD=$1
shift

if [[ "$OZONE_SUBCMD" == "auditparser" ]]; then
echo "warning: 'ozone auditparser' is deprecated, use 'ozone debug auditparser' instead."
OZONE_SUBCMD="debug"
else
shift
fi


if ozone_need_reexec ozone "${OZONE_SUBCMD}"; then
Expand Down
20 changes: 9 additions & 11 deletions ...adoop/ozone/audit/parser/AuditParser.java → ...ozone/debug/audit/parser/AuditParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,21 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser;
package org.apache.hadoop.ozone.debug.audit.parser;

import org.apache.hadoop.hdds.cli.GenericCli;
import org.apache.hadoop.hdds.cli.DebugSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
import org.apache.hadoop.ozone.audit.parser.handler.LoadCommandHandler;
import org.apache.hadoop.ozone.audit.parser.handler.QueryCommandHandler;
import org.apache.hadoop.ozone.audit.parser.handler.TemplateCommandHandler;
import org.apache.hadoop.ozone.debug.audit.parser.handler.LoadCommandHandler;
import org.apache.hadoop.ozone.debug.audit.parser.handler.QueryCommandHandler;
import org.apache.hadoop.ozone.debug.audit.parser.handler.TemplateCommandHandler;
import org.kohsuke.MetaInfServices;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;

/**
* Ozone audit parser tool.
*/
@Command(name = "ozone auditparser",
@Command(name = "auditparser",
description = "Shell parser for Ozone Audit Logs",
subcommands = {
LoadCommandHandler.class,
Expand All @@ -37,7 +38,8 @@
},
versionProvider = HddsVersionProvider.class,
mixinStandardHelpOptions = true)
public class AuditParser extends GenericCli {
@MetaInfServices(DebugSubcommand.class)
public class AuditParser implements DebugSubcommand {
/*
<.db file path> load <file>
<.db file path> template <template name>
Expand All @@ -51,10 +53,6 @@ public class AuditParser extends GenericCli {
"UNIQUE(datetime,level,logger,user,ip,op,params,result))")
private String database;

public static void main(String[] argv) throws Exception {
new AuditParser().run(argv);
}

public String getDatabase() {
return database;
}
Expand Down
4 changes: 2 additions & 2 deletions ...e/audit/parser/common/DatabaseHelper.java → ...g/audit/parser/common/DatabaseHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.common;
package org.apache.hadoop.ozone.debug.audit.parser.common;

import static java.nio.charset.StandardCharsets.UTF_8;

Expand All @@ -37,7 +37,7 @@
import java.util.Properties;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.ozone.audit.parser.model.AuditEntry;
import org.apache.hadoop.ozone.debug.audit.parser.model.AuditEntry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down
2 changes: 1 addition & 1 deletion ...one/audit/parser/common/ParserConsts.java → ...bug/audit/parser/common/ParserConsts.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.common;
package org.apache.hadoop.ozone.debug.audit.parser.common;

/**
* Constants used for ozone audit parser.
Expand Down
0 ...one/audit/parser/common/package-info.java → ...bug/audit/parser/common/package-info.java
View file
Open in desktop
File renamed without changes.
8 changes: 4 additions & 4 deletions ...it/parser/handler/LoadCommandHandler.java → ...it/parser/handler/LoadCommandHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.handler;
package org.apache.hadoop.ozone.debug.audit.parser.handler;

import java.util.concurrent.Callable;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
import org.apache.hadoop.ozone.audit.parser.AuditParser;
import org.apache.hadoop.ozone.audit.parser.common.DatabaseHelper;
import org.apache.hadoop.ozone.debug.audit.parser.AuditParser;
import org.apache.hadoop.ozone.debug.audit.parser.common.DatabaseHelper;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;
import picocli.CommandLine.ParentCommand;
Expand All @@ -32,7 +32,7 @@
aliases = "l",
description = "Load ozone audit log files.\n\n" +
"To load an audit log to database:\n" +
"ozone auditparser <path to db file> load <logs>\n",
"ozone debug auditparser <path to db file> load <logs>\n",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
public class LoadCommandHandler implements Callable<Void> {
Expand Down
10 changes: 5 additions & 5 deletions ...t/parser/handler/QueryCommandHandler.java → ...t/parser/handler/QueryCommandHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.handler;
package org.apache.hadoop.ozone.debug.audit.parser.handler;

import java.sql.SQLException;
import java.util.concurrent.Callable;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
import org.apache.hadoop.ozone.audit.parser.AuditParser;
import org.apache.hadoop.ozone.audit.parser.common.DatabaseHelper;
import org.apache.hadoop.ozone.debug.audit.parser.AuditParser;
import org.apache.hadoop.ozone.debug.audit.parser.common.DatabaseHelper;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;
import picocli.CommandLine.ParentCommand;
Expand All @@ -33,8 +33,8 @@
@Command(name = "query",
aliases = "q",
description = "Execute custom query.\n\n" +
"To run a custom read-only query:\n" +
"ozone auditparser <path to db file> query <query>\n",
"To run a custom read-only query on the audit logs loaded to the database:\n" +
"ozone debug auditparser <path to db file> query <query>\n",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
public class QueryCommandHandler implements Callable<Void> {
Expand Down
10 changes: 5 additions & 5 deletions ...arser/handler/TemplateCommandHandler.java → ...arser/handler/TemplateCommandHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.handler;
package org.apache.hadoop.ozone.debug.audit.parser.handler;

import java.sql.SQLException;
import java.util.concurrent.Callable;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
import org.apache.hadoop.ozone.audit.parser.AuditParser;
import org.apache.hadoop.ozone.audit.parser.common.DatabaseHelper;
import org.apache.hadoop.ozone.debug.audit.parser.AuditParser;
import org.apache.hadoop.ozone.debug.audit.parser.common.DatabaseHelper;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;
import picocli.CommandLine.ParentCommand;
Expand All @@ -32,8 +32,8 @@
@Command(name = "template",
aliases = "t",
description = "Execute template query.\n\n" +
"To run a template query:\n" +
"ozone auditparser <path to db file> template <template>\n\n" +
"To run a template query on the audit logs loaded to the database:\n" +
"ozone debug auditparser <path to db file> template <template>\n\n" +
"Following templates are available:\n" +
"(Template) (Description)\n" +
"top5users : Top 5 users.\n" +
Expand Down
0 ...ne/audit/parser/handler/package-info.java → ...ug/audit/parser/handler/package-info.java
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion .../ozone/audit/parser/model/AuditEntry.java → .../debug/audit/parser/model/AuditEntry.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser.model;
package org.apache.hadoop.ozone.debug.audit.parser.model;

/**
* POJO used for ozone audit parser tool.
Expand Down
0 ...zone/audit/parser/model/package-info.java → ...ebug/audit/parser/model/package-info.java
View file
Open in desktop
File renamed without changes.
0 ...doop/ozone/audit/parser/package-info.java → ...zone/debug/audit/parser/package-info.java
View file
Open in desktop
File renamed without changes.
10 changes: 4 additions & 6 deletions ...p/ozone/audit/parser/TestAuditParser.java → ...e/debug/audit/parser/TestAuditParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.hadoop.ozone.audit.parser;
package org.apache.hadoop.ozone.debug.audit.parser;

import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
Expand Down Expand Up @@ -107,7 +107,7 @@ public static void cleanup() throws IOException {
private static void execute(String[] args, String msg) {
List<String> arguments = new ArrayList(Arrays.asList(args));
LOG.info("Executing shell command with args {}", arguments);
CommandLine cmd = parserTool.getCmd();
CommandLine cmd = new CommandLine(parserTool);

IExceptionHandler2<List<Object>> exceptionHandler =
new IExceptionHandler2<List<Object>>() {
Expand Down Expand Up @@ -200,10 +200,8 @@ public void testLoadCommand() {
public void testHelp() throws Exception {
String[] args = new String[]{"--help"};
execute(args,
"Usage: ozone auditparser [-hV] [--verbose] " +
"[-conf=<configurationPath>]\n" +
" [-D=<String=String>]... <database> " +
"[COMMAND]");
"Usage: auditparser [-hV] <database> [COMMAND]\n" +
"Shell parser for Ozone Audit Logs");
}

private static File getRandomTempDir() throws IOException {
Expand Down
0 ...doop/ozone/audit/parser/package-info.java → ...zone/debug/audit/parser/package-info.java
View file
Open in desktop
File renamed without changes.