HDDS-9719. Add performance audit log for Datanode

[xichen01]

What changes were proposed in this pull request?

Add a performance audit log for slow requests on the Datanode, which will be output if the request processing time exceeds the threshold(500ms by default).

Example:

2023-1-1 17:01:59,134 | INFO  | DNAudit | [bTxId:e82fc88abef1427faec04e7473a33f8e, bSpanId:efa32ade0866e39d] user=null | ip=null | op=GET_BLOCK {blockData=conID: 674595 locID: 109611023645035393 bcsId: 1506141102} performance={opLatencyMs=1724, preOpLatencyMS=0} | ret=null |
2023-1-1 17:01:59,134 | INFO  | DNAudit | [bTxId:, bSpanId:] user=null | ip=null | op=GET_BLOCK {blockData=conID: 674595 locID: 109611023645332310 bcsId: 1506237580} performance={opLatencyMs=1953, preOpLatencyMS=0} | ret=null |
2023-1-1 17:02:56,288 | INFO  | DNAudit | [bTxId:099d6fc1228e468aac59ed0cce13ce66, bSpanId:efa3ea780866e83b] user=null | ip=null | op=GET_BLOCK {blockData=conID: 674595 locID: 109611023645245975 bcsId: 1506209514} performance={opLatencyMs=882, preOpLatencyMS=0} | ret=null |
2023-1-1 17:12:15,903 | INFO  | DNAudit | [bTxId:, bSpanId:] user=null | ip=null | op=READ_CHUNK {blockData=conID: 661896 locID: 109611023348686704 bcsId: 1400692708, blockDataSize=749} performance={opLatencyMs=502, preOpLatencyMS=0} | ret=null |

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9719

How was this patch tested?

Manual test

[adoroszlai]

@xichen01 thanks for working on this. This PR contains portions of the new code from #5644, so one of them will need to be updated when the other is merged. Hence one should be a draft (probably this one, as the other one is older).

[SaketaChalamchala]

Thanks for the patch @xichen01. One comment here, I think the performance should be audited if the opLatency isExceedThreshold(). Let me know if that makes sense. Otherwise, LGTM

[xichen01]

Yes, have added.

[SaketaChalamchala]

nit: Can we change the function name to isOperationSlow() to make it more readable.

[xichen01]

Done.

[adoroszlai]

Thanks @xichen01 for the patch.

[adoroszlai]

Thanks @xichen01 for updating the patch.

So if I understand it correctly, the idea is to:

• log an extra message for slow operations with PERFORMANCE marker
• allow admin to configure audit log to keep PERFORMANCE messages while discarding regular READ messages

Is that correct?

patch updated

[xichen01]

Thanks @xichen01 for updating the patch.

So if I understand it correctly, the idea is to:

• log an extra message for slow operations with PERFORMANCE marker
• allow admin to configure audit log to keep PERFORMANCE messages while discarding regular READ messages

Is that correct?

Yes, right.

[adoroszlai]

Thanks @xichen01 for the patch, @SaketaChalamchala for the review.

[adoroszlai]

I created HDDS-10270 as a follow-up, for a possible improvement in creating AuditMessage instances.