Skip to content

Dataset Tools Rework #749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
O-J1 wants to merge 94 commits into from
Closed

Dataset Tools Rework #749

Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
cdbfc77
Reworks dataset tools. Icons, better buttons sizes and lablel tweks +…
O-J1 Mar 5, 2025
ee49746
Constants, multi-line caption support. Default size more usable on al…
O-J1 Mar 5, 2025
41d5511
Significant refactor, slightly higher res + new icons
O-J1 Mar 6, 2025
d5bba22
Enable icons.py to work with pallete pngs + further UI tweaks.
O-J1 Mar 6, 2025
bf1c2d7
Further tweaks but now broken image scaling.
O-J1 Mar 6, 2025
bfa77e8
Working classes for MaskEditor, mostly. Image still bricked.
O-J1 Mar 6, 2025
7820eb3
Still broken image scaling
O-J1 Mar 7, 2025
0302243
Fixed image and masking, keybinds and more
O-J1 Mar 7, 2025
6e96bf3
Better types, switched to pathlib, only save caption and mask if change
O-J1 Mar 8, 2025
97f0155
Fix accidental brush logic inversion
O-J1 Mar 8, 2025
01aab67
Renamed a few functions.
O-J1 Mar 8, 2025
f6599ff
Doc string
O-J1 Mar 8, 2025
5cb8d61
Tweaks to CaptionUI, refactors generate windows, modified fill mode f…
O-J1 Mar 8, 2025
084a208
Initial Rating & char thresholds, Mcut, blacklists + refactoring
O-J1 Mar 8, 2025
bdd0829
Blacklist
O-J1 Mar 8, 2025
8bfb84e
Slightly narrower window
O-J1 Mar 9, 2025
0177c5f
Add License for Lucide icons.
O-J1 Mar 9, 2025
1d7eb49
Nuke Blip, prepare to add JT and Moondream, adjust readme
O-J1 Mar 9, 2025
2156707
No attempts to silence libvips worked, reverting.
O-J1 Mar 10, 2025
a2fd62a
Deleted logging_util.py
O-J1 Mar 10, 2025
97be50e
Remove debug, typo fix.
O-J1 Mar 10, 2025
9a81f05
More refactors and reorganising, cursors and such.
O-J1 Mar 12, 2025
45c705e
Move and fix filtering tokens out of result
O-J1 Mar 12, 2025
938c792
Finalise icons.py
O-J1 Mar 13, 2025
388176f
Finalise BM.py
O-J1 Mar 13, 2025
eeced18
Add CPUExecutionProvider to stop warning
O-J1 Mar 14, 2025
905c5fd
Refine Moondream2Model.py
O-J1 Mar 14, 2025
1a04008
Refine SAM+Moondream segmenting model
O-J1 Mar 14, 2025
d504545
refine models and utils
O-J1 Mar 14, 2025
59b7d17
Refine components.py
O-J1 Mar 14, 2025
c6d8528
Merge branch 'Nerogar:master' into dataset-and-samples-rework
O-J1 Mar 17, 2025
8be1ad9
refine GenerateCaptionsWindow
O-J1 Mar 17, 2025
44d80a0
Comment and type tweaks
O-J1 Mar 17, 2025
35805c5
Merge branch 'dataset-and-samples-rework' of https://github.com/O-J1/…
O-J1 Mar 17, 2025
4c6b492
unhook generation from UI thread + general refinement
O-J1 Mar 17, 2025
485658e
Refining caption UI + add clear/reset
O-J1 Mar 17, 2025
2ef340f
futher caption tweaks + use is_supported_image_ext in SAMoon
O-J1 Mar 18, 2025
af0f0b9
Add icon, slightly tweak window height + focus
O-J1 Mar 18, 2025
6eec8cd
Even more tweaks to focus, themes, adds file operations and JXL suppo…
O-J1 Mar 18, 2025
458ad18
Refactor FileOps to be more performant
O-J1 Mar 18, 2025
288c491
Updates requirement and fixes window controls
O-J1 Mar 19, 2025
ab1c875
Natural sort for files.
O-J1 Mar 19, 2025
116a42d
Adds filters to filelist, fix selection
O-J1 Mar 19, 2025
056291e
Bulk captions editing, new icon.
O-J1 Mar 19, 2025
493edeb
Move SAM2.1 to cuda requirements as it requires a CUDA kernel.
O-J1 Mar 19, 2025
93499ac
Fix a critical bug (was sequentially renaming masks)
O-J1 Mar 19, 2025
a01ff57
Reame, fix window focus regression, set train_ui to ERROR for logging
O-J1 Mar 19, 2025
6db36c5
Tweak default window sizing
O-J1 Mar 19, 2025
e8cb8f6
Adds mask preview and tweaks help text
O-J1 Mar 21, 2025
831d55c
aligning .bat files with master
O-J1 Jun 3, 2025
27c0dee
Address review comments.
O-J1 Jun 4, 2025
6965ff3
Merge remote-tracking branch 'fork/master' into HEAD to fix merge con…
O-J1 Jun 4, 2025
398a9c6
Undelete accidental deletion of BaseImageCaptionModel.
O-J1 Jun 4, 2025
610cf21
Merge regression + nudging heights to look nice
O-J1 Jun 4, 2025
8bfa629
Fix for scipy in requirements
O-J1 Jun 5, 2025
46ffb07
Updates moondream and implements initiall JoyCaption support.
O-J1 Jun 8, 2025
0247f80
Proper UI for JoyCaption
O-J1 Jun 16, 2025
8a765dc
Align requirements to master. Small typo fix
O-J1 Jun 16, 2025
fd1095e
Merge branch 'master' into dataset-and-samples-rework
O-J1 Jun 16, 2025
e9ac7f0
Ensure values populate on load.
O-J1 Jun 16, 2025
66048fb
Implements mask specific reset button, makes clear all undoable + ico…
O-J1 Jun 17, 2025
3c598f6
Retain per run user set settings and fix icon for new dialog/window
O-J1 Jun 17, 2025
f4cb6aa
Fixes "Browse" button not opening the currently loaded path.
O-J1 Jun 17, 2025
bdd5329
Initial CTKlistbox impl that wont sputter and die for caption ui with…
O-J1 Jun 17, 2025
72d0e5c
More performant listbox and profiling exploration
O-J1 Jun 18, 2025
ebd8931
Remove profiling.
O-J1 Jun 18, 2025
c708b36
Attempt to fix fractional scaling.
O-J1 Jun 18, 2025
592156f
Slightly better detection.
O-J1 Jun 18, 2025
6c66c2d
Update modules/module/BaseImageMaskModel.py
O-J1 Jun 18, 2025
5f9e61a
Readme update.
O-J1 Jun 18, 2025
39eb77c
Merge branch 'dataset-and-samples-rework' of https://github.com/O-J1/…
O-J1 Jun 18, 2025
4b5551f
Correct masklabel filtering from listbox, add alert when trying to re…
O-J1 Jun 23, 2025
71d7c6f
Correct clear_focus
O-J1 Jun 23, 2025
840f517
Adjust help text
O-J1 Jun 23, 2025
2606fdc
Ensure GMW yields to CaptionUI if it has a valid path, else use sessi…
O-J1 Jun 24, 2025
ea0cac9
Update moondream version, add moondream tag-lish mode
O-J1 Jun 25, 2025
fff3d85
Add dpid for reduction factors above 3 and split out FileOperations i…
O-J1 Jun 26, 2025
1e94bb2
FilerOperationsWindow: Add run_parallel_task, remove redundancy, modi…
O-J1 Jun 26, 2025
1aeece7
Revert install.bat to upstream version now that its been merged
O-J1 Jun 26, 2025
7034d21
Merge branch 'master' into dataset-and-samples-rework
O-J1 Jun 26, 2025
f2da83f
Add tests, fix clear focus bug, fix FileOps edge cases, adjusts pypro…
O-J1 Jun 27, 2025
f0c7a62
Reorganise BulkCaptionEdit logic, fix silent fail if no dir open
O-J1 Jun 27, 2025
0248f4e
- move shared functions into helpers.py
O-J1 Jun 27, 2025
3397c24
Add error box when clicking open in explorer button, tweaked logging
O-J1 Jun 27, 2025
eb950f4
Hide scrollbar on load/blank and reduce CPU load from scrolling (from…
O-J1 Jun 27, 2025
0084149
- Cleaning up GenerateMasksWindow
O-J1 Jun 27, 2025
b5c87d2
Moved GenerateMasksWindow to MVC + add tests. Addeds msgbox for capti…
O-J1 Jun 28, 2025
3f3fc1c
Make the filelist update when maximising too
O-J1 Jun 30, 2025
7030940
Debounced the on_window_resize update to avoid repeatedly firing
O-J1 Jun 30, 2025
801e926
Remove unrequired dep for dpid.
O-J1 Jun 30, 2025
1731b38
unload all models when captionUI is closed
O-J1 Jul 1, 2025
9577fa5
Modifylogging in the unload function
O-J1 Jul 1, 2025
b0dd6cc
Use existing torch_gc
O-J1 Jul 1, 2025
c6f9d19
Apply requested changes from review
O-J1 Aug 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ OneTrainer is a one-stop solution for all your stable diffusion training needs.
- **EMA**: Train your own EMA model. Optionally keep EMA weights in CPU memory to reduce VRAM usage
- **Aspect Ratio Bucketing**: Automatically train on multiple aspect ratios at a time. Just select the target resolutions, buckets are created automatically
- **Multi Resolution Training**: Train multiple resolutions at the same time
- **Dataset Tooling**: Automatically caption your dataset using BLIP, BLIP2 and WD-1.4, or create masks for masked training using ClipSeg or Rembg
- **Dataset Tooling**: Automatically caption your dataset using Moondream2, Joy Caption, WDv3 or JoyTag, or create masks for masked training using ClipSeg or Rembg or SAM2
- **Model Tooling**: Convert between different model formats from a simple UI
- **Sampling UI**: Sample the model during training without switching to a different application

Expand Down
Loading