Skip to content

HDDS-12805. Use slf4j for HTTP request log #8255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jojochuang merged 10 commits into from
Apr 10, 2025
Merged

HDDS-12805. Use slf4j for HTTP request log #8255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e9e9114
HDDS-12805. Use slf4j for HttpRequestLog
adoroszlai Apr 9, 2025
98a4b4b
Merge remote-tracking branch 'origin/master' into HDDS-12805
adoroszlai Apr 9, 2025
df18403
rename s3g-web to s3g-admin
adoroszlai Apr 10, 2025
4a1e8ff
rename webhdfs to httpfs
adoroszlai Apr 10, 2025
561dc8e
configure appender and logger for Ozone services
adoroszlai Apr 10, 2025
ec12791
revert renaming s3g-web to s3g-admin
adoroszlai Apr 10, 2025
1e9a7a8
Revert "rename webhdfs to httpfs"
adoroszlai Apr 10, 2025
92c7cde
Revert "revert renaming s3g-web to s3g-admin"
adoroszlai Apr 10, 2025
f87ce05
Revert "rename s3g-web to s3g-admin"
adoroszlai Apr 10, 2025
5d9ff47
update http.request logger names
adoroszlai Apr 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 0 additions & 106 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLog.java
View file
Open in desktop

This file was deleted.

62 changes: 0 additions & 62 deletions ...ds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLogAppender.java
View file
Open in desktop

This file was deleted.

22 changes: 15 additions & 7 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,15 @@
import org.eclipse.jetty.http.HttpVersion;
import org.eclipse.jetty.server.ConnectionFactory;
import org.eclipse.jetty.server.Connector;
import org.eclipse.jetty.server.CustomRequestLog;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.HttpConfiguration;
import org.eclipse.jetty.server.HttpConnectionFactory;
import org.eclipse.jetty.server.RequestLog;
import org.eclipse.jetty.server.SecureRequestCustomizer;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.server.Slf4jRequestLogWriter;
import org.eclipse.jetty.server.SslConnectionFactory;
import org.eclipse.jetty.server.handler.ContextHandlerCollection;
import org.eclipse.jetty.server.handler.HandlerCollection;
Expand Down Expand Up @@ -618,14 +620,13 @@ private void initializeWebServer(Builder builder) throws IOException {
handler.getSessionCookieConfig().setSecure(true);

ContextHandlerCollection contexts = new ContextHandlerCollection();
RequestLog requestLog = HttpRequestLog.getRequestLog(builder.name);

handlers.addHandler(contexts);
if (requestLog != null) {
RequestLogHandler requestLogHandler = new RequestLogHandler();
requestLogHandler.setRequestLog(requestLog);
handlers.addHandler(requestLogHandler);
}

RequestLog requestLog = getRequestLog(builder.name);
RequestLogHandler requestLogHandler = new RequestLogHandler();
requestLogHandler.setRequestLog(requestLog);
handlers.addHandler(requestLogHandler);

handlers.addHandler(webAppContext);
final String appDir = getWebAppsPath(builder.name);
if (!builder.skipDefaultApps) {
Expand Down Expand Up @@ -1794,4 +1795,11 @@ public static void setHttpBaseDir(OzoneConfiguration ozoneConfiguration)
tmpMetaDir.getAbsolutePath());
}
}

private static RequestLog getRequestLog(String name) {
String loggerName = "http.requests." + name;
Slf4jRequestLogWriter writer = new Slf4jRequestLogWriter();
writer.setLoggerName(loggerName);
return new CustomRequestLog(writer, CustomRequestLog.EXTENDED_NCSA_FORMAT);
}
}
51 changes: 0 additions & 51 deletions ...p-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLog.java
View file
Open in desktop

This file was deleted.

40 changes: 0 additions & 40 deletions ...ramework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLogAppender.java
View file
Open in desktop

This file was deleted.

19 changes: 19 additions & 0 deletions hadoop-ozone/dist/src/shell/conf/log4j.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,25 @@ log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd

#
# HTTP request logs
#
log4j.appender.HttpAccess=org.apache.log4j.DailyRollingFileAppender
log4j.appender.HttpAccess.File=${hadoop.log.dir}/access.log
log4j.appender.HttpAccess.DatePattern=.yyyy-MM-dd
log4j.appender.HttpAccess.layout=org.apache.log4j.PatternLayout
log4j.appender.HttpAccess.layout.ConversionPattern=%m%n

log4j.additivity.http.requests=false
log4j.logger.http.requests=INFO,HttpAccess
# Create separate appender for each co-hosted component if needed, then enable distinct logger configs:
#log4j.logger.http.requests.hddsDatanode=INFO,HttpAccess
#log4j.logger.http.requests.ozoneManager=INFO,HttpAccess
#log4j.logger.http.requests.recon=INFO,HttpAccess
#log4j.logger.http.requests.s3gateway=INFO,HttpAccess
#log4j.logger.http.requests.s3g-web=INFO,HttpAccess
#log4j.logger.http.requests.scm=INFO,HttpAccess
#log4j.logger.http.requests.webhdfs=INFO,HttpAccess

# Custom Logging levels
# AWS SDK & S3A FileSystem
Expand Down