-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce false positives in ContentDetector due to camera movement #153
Comments
I think that this kind of approach could be valuable as fast camera movement leading to anomalous scene detection is something I have come across many times before. The way the code is written now, there is a So, if I am understanding the process correctly, the detector method you are discussing would operate like this:
Without actually trying to implement this I see a one thing that would need to be considered. Specifically, that the stats for the whole video would need to be accessible to the Something else worth thinking about (perhaps @Breakthrough has thoughts on this) would be if this is the desired approach overall with PySceneDetect. What I mean is that would this kind of post-processing be better performed as part of a |
Yeah, that's basically my idea. The only problem is that the current I've forked the repository and created a branch with a (still somewhat messy) implementation of this if anyone is interested. |
Thanks for sharing the code! I think this is a first good pass at the issue. I will take a closer look soon to better integrate this into the rest of PySceneDetect (docs, cli, etc.). First thoughts are that it would be possible to use the existing One other thing that should be called out in the final version of this is that the I will take a look at these when I get a chance. |
@scarwire I worked some more on your branch and renamed it to A summary of the changes I made to your implementation to try to generalize it:
|
Hi @scarwire; Would you be able to provide updated links to the video files, or attach them to this issue? I'd like to start expanding the test cases PySceneDetect has with more real-world media such as this to help future development, and prevent regressions when such features are implemented. This is definitely something I would like to consider adding in a future version of PySceneDetect (post-v0.6 most likely), as automatic threshold determination is a highly sought-after feature. It sounds like this approach might drive some underlying API changes, which I am more than happy to support after the upcoming v0.6 release (the primary goals of which are suppression of camera flashes and quality-of-life improvements). Thank you (and @wjs018!!) for your research, methods, and implementation, much appreciated. Looking forwards to getting this cleaned up and merged! |
Hi there, and sorry for taking so long to get back to you. Here are updated links for the two examples I used for testing. The new detector gets more accurate results with them, but it's still not perfect. Thanks so much for taking up this idea and also thanks again to @wjs018 for refactoring and cleaning up my code! |
Hej there, This is exactly what i needed. I had to index 30 year old home video and this works like a charm. I updated @wjs018 fork against @Breakthrough current master, merged without conflicts and results are pretty good. Thank you! |
Awesome - would you be willing to create a pull request with the detector, @wjs018? Not sure the state of the code but it looks like a very interesting approach! |
Brought my branch up to date with master and made sure nothing broke. As mentioned in #198, I haven't really tested this other than to make sure that things ran smoothly and integrated nicely with other parts of the software (such as cli). |
Awesome, thanks so much for the PR - I'll try to get that merged as soon as possible, I want to release v0.5.5 before merging it in however, as this will also drive some changes to v0.6 as you mentioned (re: post process). I also want to try and reduce code duplication between this and detect-content somehow, but can figure that out after it's merged. I wanted to ask you all for your thoughts on:
What are your thoughts on the best way to proceed with Thank you so much for this everyone (esp. @wjs018 and @scarwire), looking forward to getting it merged in. |
TODO: In regards to the CLI, I think the new detector is different enough in functionality that it should be a separate command. I'm thinking This has been merged into the v0.5.6 branch right now though, and will definitely be in the next release of PySceneDetect :) |
@scarwire could you let me know some more details about the video, or is there an IMDB page I could reference? I'd like to add the clips you shared as a test case, but need to work out the copyright details. Thanks! Also the defaults seem to be a bit too sensitive for colour video - I realize now, that an option should be provided to use either ONLY the change in luminance/intensity, both for ContentDetector, as well as AdaptiveDetector, as the change in hue/saturation will always be neglegible/zero for greyscale material (thus those frame scores are always 1/3 of what they would be for colour material). My initial thought is to multiply your default threshold by 3 to compensate for this, and allow an additional flag indicating if the source material is greyscale or not (thus normalizing the approximate threshold difference between greyscale and colour source material). Does this sound reasonable? |
Refactor metric key implementation accordingly. #153
Rename AdaptiveContentDetector to AdaptiveDetector. (#153)
Woo, thanks again everyone! There's now some test cases in the repo for the detector, as well as better integration overall. This detector definitely will influence the API direction for v0.6, as there's a few circular dependencies I want to see if they can be sorted out better (specifically, I think the detection algorithms need some kind of factory to disallow construction without a stats manager). I've also removed the requirement on a statsfile by making the SceneManager implicitly create a StatsManager for any detector that requires one, even if the user doesn't provide one (it will simply be discarded when the program closes). You can also use Lastly, I've multiplied the default threshold by 3, and added a new I'm not sure if there's a reliable automatic way to detect greyscale inputs, so will leave it as an explicit flag for now. If anyone sees value in it, feel free to file a feature request. This shall now be included in the next release of PySceneDetect v0.5.6 :) |
Hi there! First of all, thanks for all your work on this clean, fast, and extremely promising module.
The current algorithm for detecting fast cuts with
ContentDetector
is already quite reliable, but looking through the raw data to find the correct threshold value can be tiresome, and lowering the threshold can sometimes produce false positives, especially in clips where there's no cut but there is fast camera movement. Camera movement can produce a series of frames with highcontent_val
, and if any of the frames go over the threshold, it's interpreted as a cut. You can try with these examples (1, 2) that I've taken from an old film I'm studying. There are real cuts that have acontent_val
of only 9 or 10 (file 2, frame 1172, 00:46), but also false positives withcontent_val
greater than 10 (file 1, frame 403, 00:16).One possible way to deal with this would be to look for fast peaks in the amount of change -- peaks that can't last for more than a frame or two. Once we've calculated the content change for all the frames in the clip, we can then calculate the ratio of each frame's
content_val
to thecontent_val
of the surrounding frames. In pseudocode:From the tests I've done, the "metathreshold" value of 3 (mark a frame as a cut if its
content-val
is more than three times thecontent-val
of its neighbors) and the baseline content-val threshold of 5 (still, don't mark anything as a cut if itscontent-val
is less than 5) seem like sane defaults. I think they're easier to pick out than the current threshold value, but this is open to discussion and more testing.I've been messing around with the code and come up with a way of implementing this as a
SceneDetector
object using the model you've already created. However, the current API calculates HSV values one by one as each frame is passed to the detector, so I also added a second post-processing step inscene_manager.py
after all those values have been stored. This step calculates the running average of change in surrounding frames and produces a list of cuts based on this. There are probably other ways of doing this, though.Let me know if you think this is useful. If you're interested, I'll make a pull request.
The text was updated successfully, but these errors were encountered: