Translation Queue Performance Fix + Embedded Subtitles + Manual Selection + Integrity Check System#30
Conversation
Introduces a new 'Failed' state for translation requests across client and server. Updates enums, UI badges, tooltips, and translation state logic to reflect failed translations. Ensures media state is updated when a translation fails or is modified, and adds new prompt guidance for AI translation. Also updates English translations to include the new state.
Introduces GEMINI.md with agent guidelines and lessons learned. Updates TranslationRequestServiceTests to mock new IMediaStateService dependency. Improves null checks in TranslationJob and MediaSubtitleProcessor to handle empty or null subtitles and media paths, preventing potential runtime errors.
- Add AwaitingSource state to distinguish 'waiting for subtitle' from 'excluded' - Add LastSubtitleCheckAt property for mtime-based change detection - Update MediaStateService to return AwaitingSource when no source exists - Add mtime optimization to MovieSync/EpisodeSync to reduce I/O - Add frontend support (TypeScript constant, Vue component, translations) - Create EF migrations for SQLite and PostgreSQL
When TranslationWorkerService claims a job (Pending -> InProgress), it now broadcasts the status change via SignalR. This fixes the UI bug where jobs appeared stuck in Pending state after restart, even though they were running.
…tness - Fixed translation re-evaluation and retry logic in AutomatedTranslationJob and MediaStateService. - Implemented relative path hashing and mtime/size detection in MediaSubtitleProcessor for better change detection. - Decoupled external subtitle hashing from media file changes and optimized embedded subtitle hashing to prevent redundant re-translations after transcoding. - Improved OrphanSubtitleCleanupService with precise string matching for safe cleanup. - Optimized EpisodeSync and MovieSync performance with batch SaveChangesAsync and improved stream extraction.
- Implement single-query pre-loading for show hierarchies in ShowSyncService to eliminate N+1 lookups. - Refactor SeasonSync and EpisodeSync to use pre-loaded entities, reducing database roundtrips. - Implement season-level directory listing in EpisodeSync to cache file metadata. - Add batch update method in MediaStateService for efficient state computation. - Ensure files without embedded subtitles are correctly marked as indexed to prevent redundant ffprobe probing. - Consolidate database saves to once per batch/show.
…ptimizations - Fixed 'not yet analysed' bug by allowing Unknown state media into the automation queue. - Implemented Tdarr-resistant hashing (v7) using stream metadata and relative paths. - Optimized sync performance with batch pre-loading (N+1 elimination) and season-level I/O caching. - Improved subtitle cleanup precision to prevent accidental deletion during media upgrades. - Fixed 'no subtitles' infinite loop by ensuring IndexedAt is updated for all media. - Enabled automated retries for failed translations upon media refresh or settings change. - Fixed build and test regressions in MovieSync, EpisodeSync, and MediaSubtitleProcessor.
…eporting - Safely handle duplicate SonarrIds in ShowSyncService using GroupBy. - Added unique constraints to SonarrId and RadarrId in DB configuration. - Fixed SyncShowJob and SyncMovieJob to properly rethrow exceptions so Hangfire marks them as Failed on error.
…ading - Added AsSplitQuery to Show and Movie sync queries to prevent Cartesian product explosion. - This significantly reduces memory usage and SQL execution time for large libraries. - Maintained GroupBy safety logic to handle existing database duplicates gracefully.
- Added RemoveFailedRequestsAsync to ITranslationRequestService. - Implemented cleanup of failed requests in TranslationRequestService. - Updated MediaSubtitleProcessor to clear previous failures before creating new translation requests. - This ensures media state transitions correctly from Failed back to Pending/InProgress.
…ries - Improved Hangfire job state detection in ScheduleService to fix 'Planned' status fallback. - Reduced sync BatchSize to 25 and added ChangeTracker.Clear() to manage memory on large libraries. - Added robust try-catch blocks around show, season, and episode sync loops to prevent single failures from crashing the entire job. - Implemented targeted diagnostic logging for 'Jujutsu Kaisen' and other potential skip scenarios. - Wrapped heavy database pre-loading in safety blocks to ensure sync continuity.
…index when missing
- Ensure episodes and movies bypass the optimization check if they were just indexed - Fixes issue where media remained stuck in 'AwaitingSource' state even after subtitles were found - Handles scenarios where file modification time remains unchanged between sync cycles
- Updated BulkIntegrityCheckJob to query for media in both 'Complete' and 'AwaitingSource' states - Allows users to force a re-probe and state update for items stuck in 'Not yet analysed' via the Integrity UI - Provides a safer alternative to manual database or file manipulation for recovering stuck items
- Updated BulkIntegrityCheckJob to query for media in 'Complete', 'AwaitingSource', and 'Unknown' states - Ensures that items stuck in 'Not yet analysed' (which maps to the Unknown state) are included in the re-probing process - Fixes the issue where the job reported '0 movies, 0 episodes' despite many unanalyzed items
This reverts commit 83dee7d.
This reverts commit 3b5822d.
This reverts commit 88a801d.
- Fix dashboard media stats accuracy by fetching counts directly from DB - Fix 'Update Available' always green by fixing badge condition - Enhance System Logs with rolling window, pause/resume, and animations - Overhaul Translation Test page with media search help, split-view comparison, and download options - Refactor TestTranslationService to return full preview data
…beat - Replace polling-based log stream with Channel<T> event-driven stream in LogsController - Add 15s heartbeat to prevent proxy timeouts - Fix 'logs stop appearing' issue by removing fixed buffer dependency in stream - Add search functionality to LogsPage.vue - Remove expensive TransitionGroup for log list performance - Update translations (en/nl) for new UI elements
- Fixed a bug where removing non-existent shows or movies would fail due to foreign key constraints on the 'embedded_subtitles' table. - Updated ShowSyncService and MovieSyncService to explicitly delete related EmbeddedSubtitles before deleting Episodes or Movies. - This ensures that when a show/movie is deleted from Sonarr/Radarr and then re-added (getting a new internal ID), the orphaned old records are successfully cleaned up instead of causing a transaction rollback. - Verified fix resolves the 'duplicate entries' issue seen in the UI after a Sonarr library re-import. - Build and tests (96 pass) verified.
…for subtitles This commit includes multiple bug fixes and a new feature to improve subtitle management and version comparison accuracy. ## Bug Fixes ### 1. Fix Update Indicator Always Showing Green (LingarrVersion.cs) - Fixed version comparison logic that was incorrectly determining if a newer version was available - Integrated Semver package for proper semantic version comparison - The update indicator now correctly shows when a new release is actually available vs. when running the latest version - Files modified: - Lingarr.Core/LingarrVersion.cs - Lingarr.Core/Lingarr.Core.csproj (added Semver package dependency) ## New Features ### 2. Hover-to-Reextract for Subtitles (ContextMenu.vue) - Added ability to hover over subtitle items to trigger re-extraction - Improves user workflow by providing quick access to subtitle regeneration without navigating through multiple menus - Enhances the context menu experience in the media browser - File modified: - Lingarr.Client/src/components/layout/ContextMenu.vue ## Technical Details - The Semver package enables robust semantic versioning comparison, ensuring accurate detection of major, minor, and patch version differences - The hover-to-reextract feature uses Vue.js event handlers for mouseenter/mouseleave interactions ## Files Modified - Lingarr.Client/src/components/layout/ContextMenu.vue - Lingarr.Core/Lingarr.Core.csproj - Lingarr.Core/LingarrVersion.cs
Snyk has created this PR to upgrade vue from 3.5.25 to 3.5.26. See this package in npm: vue See this project in Snyk: https://app.snyk.io/org/tomeov1/project/67558f4f-9722-4d54-837b-fb19ddc5d23f?utm_source=github&utm_medium=referral&page=upgrade-pr
Snyk has created this PR to upgrade vue-router from 4.6.3 to 4.6.4. See this package in npm: vue-router See this project in Snyk: https://app.snyk.io/org/tomeov1/project/67558f4f-9722-4d54-837b-fb19ddc5d23f?utm_source=github&utm_medium=referral&page=upgrade-pr
- Add 'embedded.extractAgain' key (EN: 'Extract again?', NL: 'Opnieuw uitpakken?') - Add 'embedded.reextractConfirm' key for re-extraction confirmation message - Fixes raw text display issue in embedded subtitle extraction UI
Added proper dev build detection to prevent false update notifications on development builds. Changes: - Lingarr.Core/LingarrVersion.cs: Added IsDevBuild property with detection based on AssemblyVersion (1.0.0.0 indicates dev build) - Lingarr.Core/Models/VersionInfo.cs: Added IsDevBuild boolean property to VersionInfo model - Lingarr.Core/Lingarr.Core.csproj: Added AssemblyVersion (1.0.0.0) for dev build identification - Lingarr.Client/src/ts/version.ts: Added isDevBuild field to Version interface - Lingarr.Client/src/store/instance.ts: Added isDevBuild to instance store state - Lingarr.Client/src/components/layout/AsideNavigation.vue: Added Dev Build badge display in sidebar - Lingarr.Server/Statics/Translations/en.json: Added devBuild translation key - Lingarr.Server/Statics/Translations/nl.json: Added devBuild translation key This fix ensures that development builds are properly identified and displayed with a 'Dev Build' badge, preventing confusion with release builds and their update indicators.
The WasSubtitleAlreadyExtracted() check was checking the database AFTER the extraction service had already set IsExtracted=true, causing auto- extracted files to never be cleaned up. Changed the logic to check File.Exists() BEFORE calling the extraction service: - Main extraction: Check if predicted output path exists before extraction - Fallback extraction: Check all candidate paths before extraction - Mark files for cleanup only if they didn't exist before extraction Added helper methods: - PredictExtractionOutputPath(): Predicts the output path for extraction - PredictSubtitlePathInternal(): Internal path prediction helper - WasAnyCandidatePathExisting(): Checks if any candidate exists for fallback This ensures user-extracted (pre-existing) files are preserved, while auto-extracted files are properly cleaned up after translation.
Forced subtitles now receive a -50 penalty instead of -10 to prevent them from being selected over full dialogue subtitles. With the previous -10 penalty, a Forced subtitle with language match (+50) would still score 40 points and pass the quality threshold (30), potentially being selected for translation. With -50 penalty: - Language match: +50 - Forced penalty: -50 - Total: 0 (below threshold of 30) Non-forced subtitles receive a +5 bonus, so even with just language match they score 55 points, comfortably passing the threshold. This ensures forced subtitles (which are typically partial or effect-only) are never selected for translation when full dialogue alternatives exist.
Implements detection of incomplete source subtitles (Forced/Signs-only)
that may have been used for translations.
Features:
- New SubtitleTypeCheckResult model with entry count, completeness status,
warnings, and recommended actions
- ValidateSubtitleTypeAsync() method in SubtitleIntegrityService to check
individual translations (threshold: < 50 entries = potentially incomplete)
- ValidateAllSubtitleTypesAsync() for bulk scanning of all completed translations
- Integration into BulkIntegrityCheckJob with stats tracking
- New API endpoints: POST /api/subtitle/validate-subtitle-types and
GET /api/subtitle/validate-subtitle-type/{translationId}
- UI section in IntegrityCheckPage with actions: Accept, Auto-fix, Dismiss
- Persistent storage of validation results
Files modified:
- Lingarr.Server/Models/SubtitleTypeCheckResult.cs (new)
- Lingarr.Server/Interfaces/Services/Subtitle/ISubtitleIntegrityService.cs
- Lingarr.Server/Services/Subtitle/SubtitleIntegrityService.cs
- Lingarr.Server/Jobs/BulkIntegrityCheckJob.cs
- Lingarr.Server/Controllers/SubtitleController.cs
- Lingarr.Client/src/pages/settings/IntegrityCheckPage.vue
…bugging (Phase 3) Add new tracking fields to TranslationRequest entity: - SourceSubtitleType: tracks subtitle type (Full, SDH, Forced, Signs/Songs, Unknown) - SourceSubtitleEntryCount: number of subtitle entries loaded - SelectedStreamTitle: original stream title from video metadata - IsForcedSubtitle: flag if forced subtitle stream was used - StartedAt: timestamp when translation actually started processing Update TranslationJob to populate these fields: - Set StartedAt when translation begins (not just queued) - Capture entry count after loading subtitles - Match subtitle file to embedded subtitle metadata - Determine subtitle type from stream title, flags, and filename patterns - Log captured metadata for debugging Create EF Core migrations for SQLite and PostgreSQL to add the new columns. These fields enable better debugging and integrity checks, helping users understand what type of subtitle was used for each translation and why translations might be incomplete.
Features:
- Add GET /api/subtitle/available/{mediaType}/{mediaId} endpoint to list embedded subtitles with metadata
- Add ListAvailableSubtitlesAsync service method with entry count and sparsity detection
- Create SubtitleSelectorModal.vue component for manual subtitle selection
- Add 'Manual Select' button to IntegrityCheckPage for incomplete subtitles
- Add POST /api/translate/queue-with-subtitle endpoint for queueing with specific stream
- Update TranslationJob to use preferred stream index from manual selection
- Update TryExtractEmbeddedSubtitle to accept optional preferredStreamIndex parameter
The modal shows:
- Language flags and names
- Stream index and codec info
- Entry counts for extracted subtitles
- Warning icons for forced/sparse subtitles
- Visual indicators for text-based vs image-based subtitles
Introduce project documentation and development patterns under .mindmodel and root docs. Adds coding conventions, domain glossary, manifest and stack metadata, architecture/design/concurrency/data-access/api/state-management/error-handling patterns, and multiple example implementations (Vue component, Pinia store, C# controller, service factory, job filter, custom exceptions). These files codify conventions, provide reference patterns and snippets, and supply example code to standardize development and onboarding.
Add RetryCount, FailedAt and NextRetryAt to TranslationRequest and create corresponding PostgreSQL and SQLite migrations (and update model snapshot). Implement exponential backoff and update retry tracking in TranslationJob. Refactor RetryAllFailedRequests to only select requests eligible for retry (NextRetryAt null or <= now), use an efficient UPDATE strategy to mark failed requests as Pending (avoid delete/insert churn), deduplicate retries, batch SignalR notifications, and enqueue jobs for retried requests. Also change the RetryFailedRequestsJob schedule from hourly to daily at 22:00 UTC and add safeguards when restarting individual requests to avoid duplicate active requests.
The embedded subtitle extraction feature was added with SQLite migration but the PostgreSQL migration was never created. This caused the AutomatedTranslationJob to fail with: 42P01: relation 'embedded_subtitles' does not exist Adds: - Migration to create embedded_subtitles table - Foreign keys to movies and episodes tables - Indexes for episode_id and movie_id lookups
The original migration was missing required columns: - is_text_based (bool) - is_extracted (bool) - extracted_path (text) - codec_name was not marked as required Added: 1. Updated 20260210033000_AddEmbeddedSubtitles with complete schema 2. New 20260210040000_FixEmbeddedSubtitlesColumns for already-deployed systems This ensures the embedded_subtitles table matches the EF Core model exactly.
Previous fix migration was missing the is_hearing_impaired column, causing null constraint violations when saving embedded subtitles. Adds: - is_hearing_impaired (bool, not null, default false) This completes the embedded_subtitles table schema.
The original PostgreSQL migration for embedded_subtitles was missing the is_hearing_impaired column and default values, causing INSERT failures when the application tried to save embedded subtitle data. Changes: - Added is_hearing_impaired column with default false - Added defaultValue: false for is_text_based and is_extracted - Removed redundant FixEmbeddedSubtitlesColumns migration This ensures new installations get the complete schema. Note: Existing deployments with broken schema should run: ALTER TABLE embedded_subtitles ALTER COLUMN is_hearing_impaired SET DEFAULT false;
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR merges 64 commits of feature work, bug fixes, and performance improvements from
mainintolatest. It brings the latest branch up to date with all the improvements made since the last merge.Actual Statistics
What Was Actually Implemented
Features (10 commits)
Bug Fixes (21 commits)
Performance Improvements (4 commits)
Cron.Daily(22)) - Failed translations retry with increasing delays (1h → 4h → 12h → 24h) at 10 PM UTC daily instead of hourlyDatabase Migrations (6 commits)
UI/UX Changes (3 commits)
Dependencies (3 commits)
Verified Implementation Details
ScheduleService.cs:91- changed from hourly toCron.Daily(22)(10 PM UTC)TranslationRequestService.cs:430- "Send batched SignalR notification instead of per-item"Testing
Merge Note
This PR uses an ort merge strategy to combine the security updates from
latestwith the feature work frommain.