Broken buffer detection - contrast method #94
Conversation
There was a problem hiding this comment.
Nice! We can examine the dropped frames with the root denoise branch update. It'll be interesting to compare what this and mean_error drops.
One processing-wise concern is that the comparison unit doesn't match the buffer shapes in data transfer and is redefining an original block shape. I commented more about this inline. My guess is that with these blocks there should be more false-positive/false-negatives depending on the threshold, but I might be wrong.
Another request is to add some tests and do linting if possible, but as this isn't headed to the main branch, we can also take care of that later.
| self.diff_frames.append( | ||
| cv2.absdiff(input_frame, self.previous_frame) | ||
| * self.noise_patch_config.diff_multiply |
There was a problem hiding this comment.
Could you leave these so it doesn't break the mean version functions? It's my bad if the primitive tests don't detect this, so we need to update that too.
There was a problem hiding this comment.
this has somewhat changed with the unified mode now that I have implemented
mio/process/video.py
Outdated
| ) | ||
| else: | ||
| raise ValueError(f"Unsupported noise detection method: {self.noise_patch_config.method}") |
There was a problem hiding this comment.
It may be better to validate in the denoise config model. If you can switch, that'll be great, but if not, it's not a big deal so we can leave it.
mio/process/frame_helper.py
Outdated
| # Use buffer_size to calculate the height of each block | ||
| block_height = noise_patch_config.buffer_size // width # Block spans entire width | ||
| block_height = max(1, block_height) # Ensure at least one row per block |
There was a problem hiding this comment.
My intention in using the buffer size and chunking it up in the mean_error method was to match the comparison block with the shape of the buffers because errors are likely to occur within a buffer unit.
I guess you're using buffer_size for the same reason, but I think you need to serialize the frame so we can chunk it in the way it's done in data transfer. This is done in the mean_error method so I think it's worth looking into it.
There was a problem hiding this comment.
Or might it be better to unify it with the mean_error method as a single noise detection method and make it run different detection functions based on the input options? That way, it's already chunked in buffers (communication packets), and we can also visualize the areas within the frame that the detector determined are noisy, which the mean stuff is doing.
There was a problem hiding this comment.
Sounds good.
I will change it to have a noise detection logic so that there’s one entry point (detect_frame_with_noisy_buffer).
Inside this method:
• Serialize the frame into chunks (buffers).
• Based on the configuration (e.g., method: mean_error or method: block_contrast), run the appropriate detection function on those chunks.
mio/process/frame_helper.py
Outdated
| @@ -121,6 +121,53 @@ def detect_frame_with_noisy_buffer( | |||
| noise_output = np.concatenate(noisy_parts)[: self.height * self.width] | |||
| noise_patch = noise_output.reshape(self.width, self.height) | |||
| return any_buffer_has_noise, np.uint8(noise_patch) | |||
|
|
|||
| def detect_frame_with_block_contrast( | |||
There was a problem hiding this comment.
Maybe standard deviation or SD to be specific?
There was a problem hiding this comment.
changed to: def _detect_with_block_contrast_SD
|
main new commit change: needed to change mean_error buffer_split: 10 to 8 because it didn't work that it cut the buffer into 10 smaller pieces but only up to 8, 8 is the amount of splits/chunks/buffers for the 200x200 included some logging for debug as well |
|
What do you mean when you say buffer_split 10 doesn't work? Does it just not detect errors correctly or does it get an error? (if It's an error what kind?) |
| # Slide through the frame vertically in block_height steps | ||
| for y in range(0, height, block_height): |
There was a problem hiding this comment.
not sure why we are using a different splitting method here? we already have split_current passed to us (but unused).
| logger.debug("Previous frame is None.") | ||
|
|
||
| buffer_size = noise_patch_config.buffer_size | ||
| split_current = self.split_by_length(serialized_current, buffer_size) |
There was a problem hiding this comment.
this seems to be a different value for the second argument buffer_size compared to before buffer_size // buffer_split + 1 - does this affect the other method?
mio/process/frame_helper.py
Outdated
| if noise_patch_config.method == "mean_error" and previous_frame is not None: | ||
| return self._detect_with_mean_error(split_current, split_previous, noise_patch_config) | ||
| elif noise_patch_config.method == "block_contrast": | ||
| return self._detect_with_block_contrast_SD(split_current, current_frame, buffer_size, noise_patch_config) |
There was a problem hiding this comment.
not sure why buffer_size is split out as a separate param when we are passing the config object anyway, seems like the signature here should be just (current_frame, noise_patch_config) (or (split_current, noise_patch_config) if there isn't a reason to have a different splitting method)
| continue | ||
|
|
||
| mean_intensity = np.mean(block) | ||
| std_intensity = np.std(block) |
There was a problem hiding this comment.
I think what we want here is not the standard deviation of the whole block, but of neighboring pixels. otherwise it seems like this would be tripped by an uncorrupted buffer that just has a very bright region and a very dark region.
For example this image:
has a standard deviation of 112.7
and this image
has a standard deviation of 127.5
and i can get the donut image to have the same standard deviation by increasing the size of the donut until half the pixels are 1 and half the pixels are 0.
If we however use the second derivative (in this case over just the -1th axis, but you could also average the diffs over x and y) they are easily distinguishable.
>>> # the random image
>>> np.mean(np.diff(np.diff(speckle)))
np.float64(95.5089898989899)
>>> # the donut image
>>> np.mean(np.diff(np.diff(donut)))
np.float64(2.87030303030303)
There was a problem hiding this comment.
Sort of related note. We just chatted that we'll probably need to combine detection methods because there are two modes of broken buffers now: (a) sandstorm and (b) all black (not showing up here, but this happens if the preamble or header is missed). SD won't be good for detecting the latter, and the mean error comparison needs two almost valid frames, so we'll need a fusion of these methods.
Doesn't have to be this PR, but we'll eventually have to combine these or think of a better detection method to reduce false positives/negatives.
There was a problem hiding this comment.
agreed on having several, separable methods rather than one huge complicated one
| block_height = buffer_size // width # Block spans the entire width | ||
| block_height = max(1, block_height) # Ensure at least one row per block | ||
|
|
||
| noisy_mask = np.zeros_like(current_frame, dtype=np.uint8) |
There was a problem hiding this comment.
seems like it could be dtype=bool for memory efficiency
There was a problem hiding this comment.
Since we're merging this into the preprocessing branch, and i figure we'll need further work there on refactoring these into separable classes, not commenting on the need for that here, but we do need tests for this - two kinds would be ideal:
- naturalistic, with a short video segment where we have "ground truth" labels for buffers/ known to be corrupted - confirm that we label those and only those labels as corrupted
- unitlike, where we generate a frame with a normal image in it (like that donut image) and then randomly corrupt some buffer-shaped segment within it
I also think we need to not just use plain stdev as i said in a comment bc it's not very specific to the corruption we're filtering for, proposed an example alternative in comments
|
ok I linted so the tests would run. @MarcelMB check out https://miniscope-io.readthedocs.io/en/latest/meta/contributing.html#linting - your IDE should be checking this for you (it's way less annoying that way to have the IDE warn you about it as you're writing and do the autofixes), but otherwise just run edit: ope i was thinking of another repo, we don't have tests dependent on code quality checks here, it's the PR not being to main. i'll fix that one sec |
|
added tests with a sample video (very small, just 60 frame segment) with a lot of the speckle noise error of varying sizes. Currently the tests fail because we miss 5 of the frames with smaller patches. I think the more sensitive method described above would let us set a much lower threshold so we could catch those. |
Takuyas approach was:
This method works fine most of the time. But has an issue with the data we recorded in December. Because the previous frame is often also broken.
I added another method: block_contrast
Broken buffers typically have higher contrast. Applying local contrast detection to identify regions with unusually bright or dark pixels could be helpful
a broken buffer looks like this (black&white pixels), and therefore has a high contrast, buffers with 'real' neural images don't have this very high contrast:
detect regions with high contrast on a block-by-block basis (not for the entire frame, that didn't work so well when I tried this), each block represents the size of a buffer
The frame is divided into non-overlapping blocks/buffers.
• Each block is analyzed independently for contrast.
For each block, the standard deviation of pixel intensities is calculated.
• If the standard deviation (contrast) exceeds the threshold, the block is flagged as noisy.
worked well with test data and data from December
Its a relatively small PR and I tried to stick strictly to how the code is organized at the moment. And only added one other method for filtering broken buffers. So could be merged easily.
📚 Documentation preview 📚: https://miniscope-io--94.org.readthedocs.build/en/94/