Repository navigation
fix: dedup same song across youtube title format variants - #582
Merged
Merged
Conversation
add Brazilian noise patterns (Tradução, Legendado, Clipe Oficial, hashtags, bare Lyrics) and extractSongCore() to identify the core song title from artist-prefixed or inverted "Song - Artist" formats. buildExcludedKeys and isDuplicateCandidate now also check a core-title key, so "Beyoncé - Halo (Tradução/Legendado)", "Halo - Beyoncé (Lyrics)", and "Beyoncé - Halo Lyrics #music" all correctly dedup against a history entry of "Beyoncé - Halo".
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThis PR enhances music queue duplicate detection by introducing a new Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/bot/src/utils/music/queueManipulation.ts`:
- Around line 670-671: The current dedup logic only checks candidate cores
against the prebuilt exclusion set, so variants returned in the same replenish
pass (e.g., "Beyoncé - Halo" and "Halo - Beyoncé") can both be accepted; update
the selection flow in selectDiverseCandidates/upsertScoredCandidate to also
maintain a local set of core keys for the current pass (use extractSongCore and
normalizeText to derive keys, and also consider normalizeTitleOnly where
relevant) and check each candidate against both the prebuilt exclusion set and
this local currentPassKeys before accepting; when a candidate is accepted, add
its core key to currentPassKeys so subsequent candidates in the same pass are
blocked.
In `@packages/bot/src/utils/music/searchQueryCleaner.ts`:
- Around line 231-244: The current normalization in extractSongCore (norm +
cleanAuthor) strips non-ASCII letters so non-Latin artist names become empty and
overlaps() can't disambiguate; update norm to preserve Unicode letters and
numbers (use a Unicode-aware regex like excluding [^\p{L}\p{N}] with the u flag)
so authNorm, norm(left), and norm(right) retain non-Latin characters, and keep
the overlaps logic (overlaps, songPart, left, right) intact so inverted titles
for multilingual artists are correctly detected.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 85c41bf3-51b8-4751-bca8-2e6d5bf2eb03
📒 Files selected for processing (3)
packages/bot/src/utils/music/queueManipulation.tspackages/bot/src/utils/music/searchQueryCleaner.spec.tspackages/bot/src/utils/music/searchQueryCleaner.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Quality Gates
- GitHub Check: SonarCloud Scan
Comment on lines
+670
to
+671
| const core = extractSongCore(t.title ?? '', t.author) | ||
| if (core) keys.push(normalizeText(core)) |
There was a problem hiding this comment.
Core-key dedup still doesn't apply between candidates picked in the same replenish pass.
These additions only compare a track against the prebuilt exclusion set. If the same search batch returns both Beyoncé - Halo and Halo - Beyoncé, they can still survive together because upsertScoredCandidate() keys by full title/author and selectDiverseCandidates() only blocks normalizeTitleOnly(...). So this fixes queue/history duplicates, but not the “two variants added together right now” case.
Also applies to: 1453-1454
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/bot/src/utils/music/queueManipulation.ts` around lines 670 - 671,
The current dedup logic only checks candidate cores against the prebuilt
exclusion set, so variants returned in the same replenish pass (e.g., "Beyoncé -
Halo" and "Halo - Beyoncé") can both be accepted; update the selection flow in
selectDiverseCandidates/upsertScoredCandidate to also maintain a local set of
core keys for the current pass (use extractSongCore and normalizeText to derive
keys, and also consider normalizeTitleOnly where relevant) and check each
candidate against both the prebuilt exclusion set and this local currentPassKeys
before accepting; when a candidate is accepted, add its core key to
currentPassKeys so subsequent candidates in the same pass are blocked.
Comment on lines
+231
to
+244
| const norm = (s: string) => | ||
| s.toLowerCase().replaceAll(/[^a-z0-9]+/g, '') | ||
| const authNorm = norm(cleanAuthor(author)) | ||
| const overlaps = (a: string, b: string) => | ||
| a.length >= 3 && | ||
| b.length >= 3 && | ||
| (a.includes(b.slice(0, 4)) || b.includes(a.slice(0, 4))) | ||
| if (overlaps(authNorm, norm(left))) { | ||
| songPart = right | ||
| } else if (overlaps(authNorm, norm(right))) { | ||
| songPart = left | ||
| } else { | ||
| songPart = right | ||
| } |
There was a problem hiding this comment.
extractSongCore stops disambiguating inverted titles for non-Latin artist names.
The overlap check drops every character outside [a-z0-9] before comparing author, left, and right. For artists like 아이유, 宇多田ヒカル, or Руки Вверх, both sides normalize to '', so an inverted title such as 좋은날 - 아이유 falls through to songPart = right and returns the artist instead of the song. That means the new core-key dedup still misses multilingual variants.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/bot/src/utils/music/searchQueryCleaner.ts` around lines 231 - 244,
The current normalization in extractSongCore (norm + cleanAuthor) strips
non-ASCII letters so non-Latin artist names become empty and overlaps() can't
disambiguate; update norm to preserve Unicode letters and numbers (use a
Unicode-aware regex like excluding [^\p{L}\p{N}] with the u flag) so authNorm,
norm(left), and norm(right) retain non-Latin characters, and keep the overlaps
logic (overlaps, songPart, left, right) intact so inverted titles for
multilingual artists are correctly detected.
2 tasks done
This was referenced Apr 13, 2026
LucasSantana-Dev
added a commit
that referenced
this pull request
May 13, 2026
add Brazilian noise patterns (Tradução, Legendado, Clipe Oficial, hashtags, bare Lyrics) and extractSongCore() to identify the core song title from artist-prefixed or inverted "Song - Artist" formats. buildExcludedKeys and isDuplicateCandidate now also check a core-title key, so "Beyoncé - Halo (Tradução/Legendado)", "Halo - Beyoncé (Lyrics)", and "Beyoncé - Halo Lyrics #music" all correctly dedup against a history entry of "Beyoncé - Halo".
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NOISE_PATTERNS:(Tradução...),Legendado,(Clipe Oficial), hashtags#\S+, bareLyricsextractSongCore(title, author?)fromsearchQueryCleaner.ts— extracts the song portion from "Artist - Song" or inverted "Song - Artist" formats using the author field to disambiguate; trims secondary separators (e.g. "Halo - VERSÃO FORROZINHO" → "Halo")buildExcludedKeysandisDuplicateCandidatenow include a third core-title key per track, so all variants of the same song dedup correctlyRoot cause
normalizeTitleOnly("Beyoncé - Halo (Tradução/Legendado)")produced"beyonchalotraduolegendado"which never matched history entrynormalizeTitleOnly("Beyoncé - Halo")="beyonchalo". Nine variants of the same Beyoncé song were added to history within minutes.Production log evidence (before fix)
Test plan
searchQueryCleaner.spec.ts— 13 new tests for Brazilian noise patterns andextractSongCore(62 total, all passing)queueManipulation.spec.ts— 77 existing tests, all passingSummary by CodeRabbit
Bug Fixes
Improvements