Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,037 changes: 1,037 additions & 0 deletions .github/workflows/duplicate-code-detector.lock.yml

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions .github/workflows/duplicate-code-detector.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
---
description: |
Daily workflow that scans the codebase for duplicate and near-duplicate code blocks,
copy-paste patterns, and repeated logic sequences in TypeScript source and JavaScript
container code. Files actionable issues for high-impact deduplication opportunities
to prevent technical debt from accumulating silently.

on:
schedule: daily
workflow_dispatch:

permissions:
contents: read
issues: read

sandbox:
agent:
version: v0.25.29
network:
allowed:
- node
- github

tools:
github:
toolsets: [issues]
bash: true

Comment on lines +19 to +28
safe-outputs:
threat-detection:
enabled: false
create-issue:
title-prefix: "[Duplicate Code] "
labels: [code-quality, refactoring]
max: 5
expires: 30d

timeout-minutes: 20
---

# Duplicate Code Detector

You are a code quality engineer analyzing the `${{ github.repository }}` codebase for duplicated and near-duplicate code. Your mission is to surface high-impact deduplication opportunities that will reduce maintenance burden and improve consistency.

## Repository Context

This is **gh-aw-firewall**, a network firewall for GitHub Copilot CLI. The most important source files for duplication analysis are:

- `src/docker-manager.ts` — 3,900+ lines; container lifecycle, env-var construction, volume mounts
- `src/cli.ts` — 1,700+ lines; argument parsing, orchestration, config merging
- `containers/api-proxy/server.js` — provider-agnostic proxy server
- `containers/api-proxy/providers/*.js` — per-provider adapter modules

## Phase 1: Gather Codebase Metrics

Run these commands to understand the scope before diving into duplication:

```bash
# File sizes and line counts
wc -l src/*.ts src/**/*.ts containers/api-proxy/*.js containers/api-proxy/providers/*.js 2>/dev/null | sort -rn | head -30

# Total files and lines
echo "=== TypeScript source ==="
find src -name "*.ts" ! -name "*.test.ts" | xargs wc -l 2>/dev/null | sort -rn | head -20
echo "=== Container JS ==="
find containers -name "*.js" | xargs wc -l 2>/dev/null | sort -rn | head -20
```

## Phase 2: Detect Structural Duplication

Install and run the `jscpd` (JavaScript Copy/Paste Detector) tool to find literal code duplication:

```bash
# Install jscpd
npm install -g jscpd 2>&1 | tail -3

# Run duplicate detection on TypeScript source
jscpd src --min-lines 10 --min-tokens 50 --reporters json --output /tmp/jscpd-src 2>&1 | tail -20

# Run on container JS
jscpd containers --min-lines 10 --min-tokens 50 --reporters json --output /tmp/jscpd-containers 2>&1 | tail -20

# Show summary
cat /tmp/jscpd-src/jscpd-report.json 2>/dev/null | node -e "
const d = JSON.parse(require('fs').readFileSync('/dev/stdin', 'utf8'));
const clones = d.duplicates || [];
console.log('Total duplicates found:', clones.length);
clones.slice(0, 10).forEach(c => {
const f1 = c.firstFile?.name?.replace(process.cwd() + '/', '') || 'unknown';
const f2 = c.secondFile?.name?.replace(process.cwd() + '/', '') || 'unknown';
console.log(\` \${f1}:\${c.firstFile?.start}-\${c.firstFile?.end} <-> \${f2}:\${c.secondFile?.start}-\${c.secondFile?.end} (\${c.fragment?.split('\\n').length || 0} lines)\`);
});
" || echo "(jscpd report not available)"
```

## Phase 3: Detect Pattern-Level Duplication

Use grep to find repeated code patterns that jscpd may not catch (semantic duplication):

```bash
echo "=== Env-var reading/trimming patterns ==="
grep -rn "process\.env\." src/ --include="*.ts" | grep -v "test" | head -40

echo "=== Docker exec/run command construction patterns ==="
grep -n "execa\|execaSync\|docker.*run\|docker.*exec" src/docker-manager.ts | head -30

echo "=== Config/validation patterns in config-file.ts and schema-validator.ts ==="
grep -n "throw\|error\|invalid\|validate" src/config-file.ts | head -20
grep -n "throw\|error\|invalid\|validate" src/schema-validator.ts 2>/dev/null | head -20

echo "=== Repeated try/catch error handling patterns ==="
grep -n -A 3 "catch (e" src/docker-manager.ts | head -60

echo "=== Provider adapter patterns in api-proxy ==="
for f in containers/api-proxy/providers/*.js; do
echo "--- $f ---"
grep -n "function\|const.*=.*(" "$f" | head -10
done

echo "=== Repeated log construction patterns ==="
grep -rn "logger\.\(debug\|info\|warn\|error\)" src/ --include="*.ts" | \
sed 's/.*logger\.\(debug\|info\|warn\|error\)(\(.*\))/\2/' | \
sort | uniq -d | head -20
```

## Phase 4: Analyze Specific Known Duplication Areas

Based on codebase knowledge, deeply analyze the most likely duplication hotspots:

```bash
echo "=== docker-manager.ts: env-var construction ==="
grep -n "env\[.*\]\s*=\|envVars\.\|\.trim()\|process\.env\." src/docker-manager.ts | head -50

echo "=== docker-manager.ts: repeated docker compose args patterns ==="
grep -n "composeArgs\|dockerArgs\|\-f.*compose\|--project-name" src/docker-manager.ts | head -30

echo "=== cli.ts: option handling patterns ==="
grep -n "\.option\|options\.\|program\." src/cli.ts | head -50

echo "=== API proxy provider similarity (getConfig patterns) ==="
for f in containers/api-proxy/providers/openai.js containers/api-proxy/providers/anthropic.js containers/api-proxy/providers/gemini.js containers/api-proxy/providers/copilot.js containers/api-proxy/providers/opencode.js; do
if [ -f "$f" ]; then
echo "--- $f: exported functions ---"
grep -n "^function\|^const.*=\s*function\|^module\.exports\|^exports\." "$f" | head -10
fi
done

echo "=== proxy-utils.js: shared utilities ==="
cat containers/api-proxy/proxy-utils.js 2>/dev/null | head -60
```

## Phase 5: Check for Existing Issues

Before filing new issues, check what's already been reported:

1. Search for open issues with `[Duplicate Code]` prefix using the GitHub toolset
2. Also search for issues with labels `code-quality` or `refactoring` that describe duplication
3. Skip any finding that already has an open tracking issue

## Phase 6: Prioritize and Report Findings

Based on your analysis, identify the **top duplications by impact** using this scoring:

| Factor | Points |
|--------|--------|
| >20 duplicate lines | +3 |
| Affects security-critical path | +3 |
| In file >1000 lines (maintenance burden) | +2 |
| More than 2 copies | +2 |
| Easy to extract (no complex dependencies) | +1 |

Report only findings with score ≥ 4.

### For each high-impact finding, create an issue with this format:

**Title**: `[Duplicate Code] <brief description of what is duplicated>`

**Body**:
```markdown
## Duplicate Code Opportunity

### Summary
- **Pattern**: Brief description of what is being duplicated
- **Locations**: File(s) and line ranges containing duplicates
- **Impact**: Lines saved / maintenance burden reduction

### Evidence

<Show the specific duplicated code blocks side by side>

### Suggested Refactoring

Describe the shared utility or abstraction that would eliminate the duplication.
For example:
- Extract a `parseEnvVars(obj)` helper in `src/env-utils.ts`
- Create a base class or mixin for provider adapters
- Add a `buildDockerArgs(config)` factory function

### Affected Files
- `path/to/file.ts` — lines X-Y
- `path/to/other.ts` — lines A-B

### Effort Estimate
Low / Medium / High

---
*Detected by Duplicate Code Detector workflow. Run date: $(date -u +"%Y-%m-%d")*
```

## Guidelines

- **Be specific**: Always include file paths and line numbers in the evidence section
- **Be actionable**: Each issue should have a clear, implementable suggestion
- **Avoid noise**: Only file issues for genuine duplication with real maintenance impact — not cosmetic similarities
- **No duplicates**: Check existing open issues before creating new ones
- **Security awareness**: Flag duplicated security-critical logic (domain validation, ACL rules, capability management) with higher urgency
- **Cap at 5 issues**: File at most 5 issues per run to avoid flooding the tracker

## Edge Cases

- **No significant duplication found**: Exit gracefully without creating issues; print a summary to the log
- **jscpd unavailable**: Fall back to grep-based pattern analysis only
- **All findings already tracked**: Skip creation and log that existing issues cover the findings
Loading
Loading