POC for profiling individual lint rules

[xunilrj]

I have just pushed this branch: d92d79b with a proposal on how we can profile individual rules. The code is very simple.

This is the result of this branch:

Profiling
-------
{"NoDebugger"}: 0ns (0 calls)
{"NoNegationElse"}: 62.776µs (206 calls)
{"NoDelete"}: 11.254µs (47 calls)
{"NoShadowRestrictedNames"}: 1.388871ms (1466 calls)
{"UseValidTypeof"}: 81.448µs (166 calls)
{"NoDoubleEquals"}: 14.636µs (166 calls)
{"NoImplicitBoolean"}: 811.002µs (957 calls)
{"NoCommentText"}: 498.847µs (789 calls)
{"UseSelfClosingElements"}: 205.667µs (453 calls)
{"NoExtraBooleanCast"}: 1.443961ms (9531 calls)
{"NoLabelVar"}: 0ns (0 calls)
{"UseSimplifiedLogicExpression"}: 80.974µs (83 calls)
{"NoDupeArgs"}: 723.884µs (343 calls)
{"NoUnsafeNegation"}: 571ns (1 calls)
{"NoCompareNegZero"}: 72.148µs (166 calls)
{"NoEmptyPattern"}: 26.398µs (128 calls)
{"NoUnnecessaryContinue"}: 1.804µs (1 calls)
{"UseWhile"}: 1.813µs (1 calls)
{"UseSingleVarDeclarator"}: 546.04µs (512 calls)
{"NoSparseArray"}: 60.949µs (99 calls)
{"NoUnusedTemplateLiteral"}: 322.77µs (334 calls)
{"UseTemplate"}: 55.597µs (166 calls)
{"NoArguments"}: 330.435µs (2933 calls)
{"NoCatchAssign"}: 6.262µs (13 calls)
{"UseSingleCaseStatement"}: 0ns (0 calls)
{"NoFunctionAssign"}: 8.662µs (33 calls)
{"UseBlockStatements"}: 44.737µs (181 calls)
{"NoShoutyConstants"}: 356.137µs (548 calls)
{"NoMultipleSpacesInRegularExpressionLiterals"}: 6.351µs (26 calls)
{"UseShorthandArrayType"}: 5.699µs (41 calls)
{"NoUnusedVariables"}: 1.090558ms (1466 calls)
{"NoAsyncPromiseExecutor"}: 20.028µs (17 calls)
{"NoImportAssign"}: 105.678µs (464 calls)

The core of the idea is:

let start = Instant::now();
...
let end = Instant::now();
crate::profiling::<R>(end.duration_since(start));

This way we can use R to get a compile-time calculated index to an array of atomics:

let mut hasher = FnvHasher::default();
let id = std::any::TypeId::of::<T>();
id.hash(&mut hasher);

Fnv hash is used because it is very simple and it is going to optimised away, giving us in the end just a simple array index. Allowing us to do:

TIMETOOK[id].fetch_add(duration, std::sync::atomic::Ordering::SeqCst);

This means that we can have collisions. Multiple rules will fall into the same bucket. We can control this using the Fnv key. If we find the environment variable ROME_PERFECT_KEY, the cli will try to find a key that does not have any collisions.

[leops]

It should be relatively easy to extend the existing tracing-based metrics infrastructure to collect these timings by modifying the declare_rule macro to also emit the metadata require to create a tracing span for each rule execution, then we wouldn't have to worry about collision since the metrics are stored in a global HashMap keyed using the corresponding span's static metadata.

However the main issue I see with collecting individual duration for each execution of the rule is that it's very sensitive to measurement noise since we're constantly "starting and stopping" the timer and accumulating many small values. It would be more accurate to time the rule running a large number of times then divide the resulting duration by the number of runs in the batch to get a good average duration, but it would require significant changes to the analyzer infrastructure specifically to support collecting metrics.

[ematipico]

What was actually timed here? The single run of the rule, the generation of the diagnostic plus the run, or the mutation too?