feat: add BullMQ + Temporal async queue detection#514
Conversation
|
@TESTPERSONAL is attempting to deploy a commit to the NexusCore Team on Vercel. A member of the Team first needs to authorize it. |
CI Report❌ Some checks failed Pipeline Status
Test Results
❌ 1 failed / 6296 passed 97 test(s) skipped — expand for details
Code CoverageTests
📋 View full run · Generated by CI |
Why this is usefulMany production codebases use async job queues (BullMQ, Temporal, etc.) to decouple producers from consumers. Without this PR, GitNexus has a blind spot: a file that enqueues a job and the worker that processes it appear completely unrelated in the graph — no edges connect them, and execution flow detection can't trace the async handoff. With
This closes a gap similar to what |
7bdef8e to
df08d70
Compare
Rebased onto mainForce-pushed — removed an accidental SwiftUI navigation commit ( The queue detection code is unchanged (2 commits), only the SwiftUI contamination was removed. |
|
@claude Let's run a final review if this PR is production ready and ready to be merged |
|
Claude finished @magyargergo's task in 3m 26s —— View job Final Production Readiness ReviewTasks
Verdict: Not yet ready to merge — 2 significant issues foundCI passes and the feature works for the happy path (BullMQ producer/consumer), but there are two meaningful issues that could result in misleading graph data in production: a DRY violation with a silent split brain risk, and a Temporal semantic mismatch. Issues🔴 Issue 1: Duplicated extraction logic (maintenance split-brain risk)The
The regexes are identical right now, but they will inevitably diverge when someone fixes a bug or adds a new pattern to one without updating the other. This is particularly fragile because:
The root cause (can't import 🟠 Issue 2: Temporal modeling is semantically incorrectThe Temporal detection creates What the code does — for const activities = proxyActivities({ startToCloseTimeout: '30s' });
await activities.validateOrder(orderId); // → CodeElement node named "validateOrder"
await activities.chargePayment(amount); // → CodeElement node named "chargePayment"Result: What it should be — The actual task queue is client.workflow.start(processOrderWorkflow, { taskQueue: 'orders', ... });The
Additionally, Minor notes (non-blocking)BullMQ false-positive trigger: Missing framework-detection.ts changes: The PR description mentions "Add framework detection patterns for BullMQ and Temporal in path-based and AST-based detection" but the Weak test assertions: // e.g.
const enqueues = result.graph.relationships.filter(r => r.type === 'ENQUEUES');
const queueNode = result.graph.nodes.find(n => n.properties.name === 'video-processing');
expect(queueNode).toBeDefined();
expect(enqueues.some(e => e.targetId === queueNode!.id)).toBe(true);Line number computation is O(n) per match: Coverage drop: All metrics declined ~0.4%. The new code for the sequential fallback path in What's working well
|
Review findings addressed (efe2fcf)1. Deduplicated extraction logicMoved 2. Fixed Temporal semantic modeling
Tests
|
efe2fcf to
a4e8c4c
Compare
|
Rebased onto latest upstream/main — merge conflicts resolved. Typecheck passes, queue-detection tests pass (5/5). Pre-existing test failures in express-routes, python-mcp-tools, and skip-git-cli are unrelated. |
|
@claude Let's run a final review if this PR is production ready and ready to be merged |
|
Claude finished @magyargergo's task in 2m 28s —— View job Final Production Readiness ReviewTasks
Verdict: Conditional merge — BullMQ is production-ready; one residual Temporal concernThe two blocking issues from the previous review were addressed. CI is green, no regressions. The BullMQ feature is solid. One Temporal semantic issue persists — not a regression, but worth understanding before merging. Previous Issues — Status✅ Issue 1: Deduplicated extraction logic — Fully fixed
The comment on the old location (
|
| Concern | Status |
|---|---|
proxyActivities guard to avoid false positives on activities.foo() |
✅ Fixed — TEMPORAL_PROXY_ACTIVITIES_RE guards the activity path |
workflow.start uses workflow function name as queue name |
✅ Fixed — 500-char lookahead extracts taskQueue: 'orders' from the options object |
Activity calls produce PROCESSES edges (consumer role) |
✅ Fixed — role: 'producer' → ENQUEUES edges |
Remaining concern: Activity method names are still used as queueName (queue-extraction.ts:148):
// activities.validateOrder(...) → queueName: 'validateOrder'
out.push({
role: 'producer',
queueName: m[1], // ← activity method name, not a real task queue
handlerName: m[1],
...
});On the fixture workflow.ts, this produces three CodeElement nodes named validateOrder, chargePayment, sendConfirmation with description "Queue: validateOrder" etc. — none of which correspond to real Temporal task queues. The actual task queue ('orders') is correctly extracted from starter.ts → workflow.start, but the workflow file's activity calls still create spurious queue nodes.
This is semantic noise, not a correctness regression for BullMQ. But for codebases with Temporal workflows it will inflate the queue node count with activity method names.
Remaining Minor Notes (non-blocking, carried from previous review)
new Worker false-positive trigger (queue-extraction.ts:69): content.includes('new Worker') fires for worker_threads / Web Worker usage. The downstream regex (BULLMQ_WORKER_RE) is narrow enough to not create false edges — the cost is a wasted regex scan per file. Low impact.
framework-detection.ts unchanged: PR description says "Add framework detection patterns for BullMQ and Temporal in path-based and AST-based detection" but the file has zero diffs. Either the description is stale or those changes were intentionally dropped. Worth clarifying but not blocking.
Weak test assertions: Tests still use toBeGreaterThanOrEqual(1). The proxyActivities guard test (queue-detection.test.ts:49) checks reasons.some(r => r.includes('producer')), which would pass on BullMQ edges alone — it doesn't specifically verify the Temporal guard behavior.
Temporal taskQueue lookahead is fragile: The 500-char lookahead for taskQueue: (queue-extraction.ts:124) will silently fall back to the workflow function name for unusually long option objects. Acceptable for most real-world code, but worth a comment.
What's Working Well
- BullMQ detection is complete and correct — variable tracking (
Queue('name')→q.add()) resolves the right queue name before creating edges - Deduplication in
processQueuePatterns—graph.getNode(qid)prevents duplicate queue nodes across chunks (call-processor.ts:1636) - Edge ID includes
lineNumber— prevents collision when a file enqueues the same queue multiple times (call-processor.ts:1639) - Schema changes are consistent —
REL_TYPES,RelationshipType, and all processor imports are aligned - Shared utility is clean —
queue-extraction.tshas a single export, good inline comments, and the O(n)lineAtfix from the previous review is in place
Summary
The PR is ready to merge with the understanding that Temporal activity-method nodes are semantic noise, not incorrect BullMQ behavior. If Temporal support is a day-1 requirement, the activity queueName field should be revisited before merging. If BullMQ is the primary use case, this can be merged and the Temporal refinement tracked as a follow-up.
a4e8c4c to
0e162ef
Compare
Rebased onto latest upstream/mainResolved merge conflicts (additive merge — kept both sides):
Verification:
|
|
@claude Let's run a final review if this PR is production ready and ready to be merged |
|
Claude finished @magyargergo's task in 1m 42s —— View job Final Production Readiness ReviewTasks
Verdict: Merge with one caveat — Temporal activity nodes are still semantic noiseThe rebase is clean. All previous fixes are intact. BullMQ detection is production-ready. The single residual Temporal concern from the last review is still present and unchanged. Fix Verification✅ Issue 1: Deduplication — Confirmed fixedBoth paths import from the shared module:
No duplication. Single source of truth at ✅ Issue 2 (partial): Temporal semantic modeling — 3 of 4 concerns fixed
✅ Rebase is clean
Remaining Concern
// Activity calls: activities.methodName() -- these are ENQUEUES (dispatching work)
if (TEMPORAL_PROXY_ACTIVITIES_RE.test(content)) {
TEMPORAL_ACTIVITY_CALL_RE.lastIndex = 0;
while ((m = TEMPORAL_ACTIVITY_CALL_RE.exec(content)) !== null) {
out.push({
filePath,
role: 'producer',
queueName: m[1], // ← 'validateOrder', 'chargePayment', 'sendConfirmation'
...
});
}
}For the fixture The cleanest fix is to remove the activity call detection block entirely (lines 139-152). The task queue connection is already established through the workflow starter path — the activity call path adds no meaningful graph information and only inflates CodeElement counts with method names. Fix this → Minor Notes (Non-blocking, carried from previous reviews)
Weak test assertions: Tests use 500-char taskQueue lookahead ( What's Working Well
SummaryThis is the third review pass. BullMQ detection is solid and production-ready. The single outstanding issue (Temporal activity-method nodes) is semantic noise, not a BullMQ regression. If Temporal correctness is in scope for this PR, remove lines 139-152 from |
|
@claude ultra-think while reviewing this PR and check the performance characteristics and possible implementation gaps and bugs. You must verify them as well! |
|
Claude finished @magyargergo's task in 6m 10s —— View job Deep Performance & Bug ReviewTasks
Verdict: 3 verified bugs, 3 performance issues, 4 test coverage gapsThe two previous reviews focused on architecture and Temporal semantics. This pass goes deeper into regex correctness, edge creation correctness, and actual runtime behavior. All findings below were verified against the current code on 🔴 Bug 1: Queue names with colons (or other special chars) silently produce no edgesFile: Both const BULLMQ_QUEUE_DECL_RE = /(?:const|let|var)\s+(\w+)\s*=\s*new\s+Queue\s*\(\s*['"](\w[\w-]*)['"]/g;
const BULLMQ_WORKER_RE = /new\s+Worker\s*\(\s*['"](\w[\w-]*)['"]/g;
This affects a common real-world BullMQ pattern. Colon-namespaced queues ( // These will NOT produce any ENQUEUES/PROCESSES edges:
const q = new Queue('payments:high'); // ❌ colon
const q = new Queue('my.email.queue'); // ❌ dot
const q = new Queue('retry queue v2'); // ❌ space
// This works:
const q = new Queue('video-processing'); // ✓ hyphen onlyFix: replace 🔴 Bug 2: ENQUEUES/PROCESSES edges created from File IDs that may not exist in the graphFile: const fid = generateId('File', pt.filePath);
graph.addRelationship({ id: ..., sourceId: fid, targetId: qid, ... });There is no This is inconsistent, and every other edge-creation site in the codebase checks node existence before creating edges. The result is ENQUEUES/PROCESSES edges pointing to ghost file nodes, which corrupts impact analysis results. 🟠 Bug 3: Temporal activity method names still used as
|
| Issue | Severity | File | Verified? |
|---|---|---|---|
| Colon-containing queue names silently produce no edges | 🔴 Bug | queue-extraction.ts:12,18 |
✅ |
| ENQUEUES/PROCESSES edge to non-existent File node | 🔴 Bug | call-processor.ts:1657 |
✅ |
| Temporal activity method names as queue names (3rd flag) | 🟠 Bug | queue-extraction.ts:147 |
✅ |
lineAt O(n) per match |
🟡 Perf | queue-extraction.ts:48 |
✅ |
new Worker triggers BullMQ scan for all worker_threads files |
🟡 Perf | queue-extraction.ts:69 |
✅ |
| Multiple edges per (file, queue) pair | 🟡 Design | call-processor.ts:1658 |
✅ |
| Guard test doesn't test the guard | 🔵 Test | queue-detection.test.ts:49 |
✅ |
| No colon-name test (would catch Bug 1) | 🔵 Test | queue-detection.test.ts |
✅ |
The two 🔴 bugs (queue name charset + dangling edge) are new findings not surfaced in prior reviews and should be fixed before merge.
|
@marxo126 can you please look into this? |
|
344085c to
4669e33
Compare
… edges) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Move extractQueuePatterns() to shared utility module
(utils/queue-extraction.ts) eliminating copy-paste between
parse-worker.ts and pipeline.ts.
2. Fix Temporal semantic modeling:
- Extract actual taskQueue name from workflow.start() options
instead of using the workflow function name
- Activity calls now create ENQUEUES edges (producer), not
PROCESSES edges — they dispatch work, not consume it
- Add proxyActivities guard to avoid false positives on generic
activities.foo() calls in non-Temporal code
- Simplify role union to 'producer' | 'consumer'
3. Add integration tests verifying taskQueue extraction and
proxyActivities guard behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update BULLMQ_QUEUE_DECL_RE and BULLMQ_WORKER_RE character classes from [\w-] to [\w:.-] so queue names like 'payments:high-priority' are matched. Also replace O(n)-per-call lineAt with O(log n) binary search over precomputed newline offsets, and add integration tests for colon-namespaced queue names. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Note: queue-detection phase not yet wired into new pipeline-phases architecture. Rebased commits preserve extraction logic; phase file creation + parsePhase integration needed as follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4669e33 to
972af28
Compare
… v1.6.2-rc.9 The upstream phase refactor replaced the monolithic pipeline with pipeline-phases/. Add queues.ts phase, queue-extraction.ts inline extractor, and thread allQueuePatterns through ParseOutput so ENQUEUES/PROCESSES edges and Queue nodes are created correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous cherry-pick brought abhigyanpatwari#521's duplicate queue-extraction.ts with wrong types ('activity'/'workflow' instead of 'consumer'/'producer'). Use existing utils/queue-extraction.ts which has correct types and full BullMQ + Temporal extraction logic. - Delete duplicate pipeline-phases/queue-extraction.ts - parse-impl.ts: import extractQueuePatterns from utils/ - queues.ts: simplify role check to 'producer' - parsing-processor.ts: drop typeEnvBindings ref (not on this branch) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Please submit a new PR if this is still relevant |
Summary
Queue/Workerpatterns and Temporalworkflow.start/activities.*calls via regexENQUEUESandPROCESSESrelation types linking producer/consumer files toCodeElementqueue nodesChanges
schema.ts: AddENQUEUES,PROCESSEStoREL_TYPESgraph/types.ts: Add toRelationshipTypeunionparse-worker.ts:ExtractedQueuePatterninterface, regex extraction function, wire intoParseWorkerResultcall-processor.ts:processQueuePatterns()creates queue nodes and edgesparsing-processor.ts: ThreadqueuePatternsthroughWorkerExtractedDatapipeline.ts: Phase 3.8 queue detection, sequential fallback inline extractionframework-detection.ts: BullMQ/Temporal path and AST patternsTest plan
queue-detection.test.ts— verifies ENQUEUES, PROCESSES edges and CodeElement queue nodesnpm run buildpasses🤖 Generated with Claude Code