Fix Issue 1348 (Negative Values in FOV plot when no comparison star is used; Negative pixel scale in FOV plot)

[ivenzor]

This PR fixes issue #1348
The plot_fov function was modified to fix the negative values in the FOV plots when no comparison star is used.

[tamimfatahi]

That's a great find. What I am also curious about is why is the scaled pixel values negative too here? They should all be positive.

[ivenzor]

@tamimfatahi you are right! I didn't notice that. I was too focused on the negative aperture that I didn't noticed that the pixel scales had negative values too. I will take a look at it later tonight.

[ivenzor]

@tamimfatahi about the negative pixel values in the FOV plots, it seems these values are present in the image data, not introduced by the visualization pipeline. Image statistics show predominantly negative values:

Mean value: -1797.61
Median value: -1807.50
ZSCALE LIMITS:
vmin: -1861.00
vmax: -23.00
SIMPLE PERCENTILE LIMITS (1-99%):
vmin: -1819.50
vmax: -1717.50

I guess the negative values likely come from the data calibration process.

[tamimfatahi]

@tamimfatahi about the negative pixel values in the FOV plots, it seems these values are present in the raw image data, not introduced by the visualization pipeline. Raw image statistics show predominantly negative values:

Mean value: -1797.61 Median value: -1807.50 ZSCALE LIMITS: vmin: -1861.00 vmax: -23.00 SIMPLE PERCENTILE LIMITS (1-99%): vmin: -1819.50 vmax: -1717.50

I guess the negative values likely come from the data reduction/calibration process.

@ivenzor Thank you for the investigation. I'll accept this PR as this relates specifically to the aperture. However, if you can look into why that's occurring as well, it would be much appreciated.

[jpl-jengelke]

@ivenzor @tamimfatahi Is there anything else to be done here before it's ready for merge? Specifically, I am asking about the comment on why negative coordinates show up in the FOV plots.

[tamimfatahi]

@ivenzor @tamimfatahi Is there anything else to be done here before it's ready for merge? Specifically, I am asking about the comment on why negative coordinates show up in the FOV plots.

I think @ivenzor would need to look into why this is occurring. I can't really tell where in the process it's happening.

@ivenzor You mentioned that the raw image is negative. Are you saying that pre-EXOTIC, the images are negative? Or EXOTIC is not calibrating the images properly?

[ivenzor]

@tamimfatahi @jpl-jengelke The input images before exotic have positive values as expected:

File: MOBS_Cecilia_Qatar-1_2016-06-04T21_01_22.382-0700_9a3fc29eb383d5edaaafb215402400df_Qatar-1160605040122.FITS.gz
--------------------------------------------------
Header Information:
TELESCOP: Cecilia
INSTRUME: Main
EXPTIME: 60.0
DATE-OBS: 2016-06-04T21:01:22.382-0700

RAW IMAGE STATISTICS:
Min value: 215
Max value: 4095
Mean value: 386.43
Median value: 385.00

SIMPLE PERCENTILE LIMITS (1-99%):
vmin: 371.00
vmax: 421.00

Shape: (500, 650)
Data type: >i2

Negative pixels: 0 (0.00%)

I think the negative values are introduced in the calibration steps. I need to do more testing and think this through more carefully, but if we’re performing differential photometry between the target star and other stars, aren't we primarily interested in the relative flux differences rather than the actual values themselves? @rzellem

We could merge the negative FOV fix, and I’ll take a closer look at the negative pixel values opening a new issue.

[rzellem]

@ivenzor - for the transit measurements, typically we only care about differential measurements, but for the absolute flux calibrations (for stellar variability monitoring), we will likely care about the actual values themselves (but technically...these are also differential measurements)

re: negative values for background subtraction - this physically shouldn't happen at all - do you know what cases this happens? in other words, are the background values negative because we do a previous calibration step that turns them negative? or are we using a mean while we should be using a median?

[ivenzor]

Hi @rzellem, I took a look at the images in the data-upload-aavso channel:

1. The pixel scale values feature was introduced around October 10. After this feature was released, you can occasionally find plots with negative pixel scales.

2. The aperture and annulus values were added to plots starting October 16. After this feature was released, you can occasionally find plots with negative aperture values.

To be honest, I hadn’t noticed either issue, even though they’ve appeared since mid-October. There may have been negative apertures and pixel scales before that date, but since they weren’t plotted, there’s no easy way to verify.

Regarding the negative aperture, I became aware of it two days ago when Anthony pointed out a negative aperture value in a plot by Cledison. The fix in this PR was intended to address that case, which is straightforward and, as far as I know, only affects the plotted value.

As for the negative pixel scale, I wasn’t aware of it until yesterday when Tamim pointed it out while reviewing the PR for the negative aperture issue. I haven’t had the chance to do much testing on this yet, but I noticed that the image passed to the plot function contains many negative values, resulting in the negative scale. I checked the original FITS.gz images, and they have normal positive values. My guess is that something is happening during the calibration or background subtraction process.

I’ll do more testing later tonight when I’m home.

[ivenzor]

@tamimfatahi @rzellem I think I now know why we got negative values sometimes, it is indeed in the calibration section of the code, but is not EXOTIC fault, al least not entirely, but based on the dark frames input data:

DARK FRAME 1 STATISTICS:
Min value: 512
Max value: 4095
Mean value: 4078.89
Median value: 4095.00

DARK FRAME 2 STATISTICS:
Min value: 174
Max value: 3141
Mean value: 289.19
Median value: 289.00

GENERAL DARK STATISTICS:
Min value: 343.5
Max value: 3618.0
Mean value: 2184.04
Median value: 2192.00

RAW FIRST IMAGE STATISTICS:
Min value: 215
Max value: 4095
Mean value: 386.43
Median value: 385.00

FIRST IMAGE AFTER CALIBRATION STATISTICS:
Min value: -3231.0
Max value: 1905.0
Mean value: -1797.61
Median value: -1807.50

The all white saturated dark frame is messing with the median of the general dark and we are overcorrecting the science frames, ending with negative values. This only happen if we got a dark file which is not a dark file... That's why this issue happen only occasionally.

EXOTIC needs to validate that the dark files are indeed "dark". I will work on it and I will add it to this PR for your consideration.

[ivenzor]

@tamimfatahi @rzellem The PR now includes fixes for the original negative aperture value in issue #1348, as well as a fix for the issue with negative values in the science frames caused by overcorrection from a faulty dark frame. EXOTIC now identifies and ignores potentially faulty dark frames using the following logic:

1. We check the highest pixel value in the dark frame (hot pixels, etc)

2. If the median of the frame is greater than 50% of the highest pixel value, we assume that frame is not a correct dark frame (i.e. it's saturated) and skip that file. In my case with the all-white saturated dark file, this percentage is 100% and thus rejected. I checked a couple other dark frames I had nearby and the percentage was below 10% for "correct" dark files.

        if exotic_infoDict['darks']:
            darksImgList = []
            for darkFile in exotic_infoDict['darks']:
                darkData = fits.getdata(darkFile)
                #Validate if a frame is likely a dark frame using a simple mean/max ratio.
                max_val = np.max(darkData)
                median_val = np.median(darkData)
                ratio = median_val / max_val
                #If the median of all the pixels in the darkfile is above 50% of the max value, the file is likely not a true dark file
                if ratio > 0.5:
                    log_info(f"\nWarning: Skipping suspicious dark frame {darkFile}: median/max ratio = {ratio}\n", warn=True)
                    continue
                darksImgList.append(darkData)
            if not darksImgList:
                log_info(f"\n\nNo valid dark frames found! Proceeding without dark correction.")
            else:
                generalDark = np.median(darksImgList, axis=0)
   

[tamimfatahi]

This needs @rzellem input to okay this, but I do have an alternative method:

What if you instead took the median of the image and checked if any pixel is above 3sigma? That could be an alternative way of handling it. The problem with this is that a single pixel can discard the dark frame due to something like cosmic rays.

I'll let Rob ponder either method (or a combination of both!).

[ivenzor]

@tamimfatahi, I’m afraid I don’t fully understand your suggestion.

What I'm attempting is to discard a bad dark frame so it isn’t included in the 'general dark' used to calibrate the science frames. A bad dark frame in this context refers to a frame that isn’t dark at all—for example, in my case, it was a pure white frame, completely saturated.

What I tried to do is use the maximum pixel value of the dark file as a quick proxy for the saturation value (any hot pixel in the dark frame will yield this value). Then, I compare the median value of the entire dark frame to this saturation value. If the median of the dark file is greater than 50% of the saturation value, I assume the dark file is not valid (it’s too bright). In my case, the bad dark frame’s median was 100% of the saturation value (a completely white frame). I compared this to other good dark frames I had nearby, their medians were about 10% of the saturation value. That’s why I considered a 50% cutoff.

[tamimfatahi]

What I'm attempting is to discard a bad dark frame so it isn’t included in the 'general dark' used to calibrate the science frames. A bad dark frame in this context refers to a frame that isn’t dark at all—for example, in my case, it was a pure white frame, completely saturated.

Ah, it's completely saturated. Then the method I mentioned wouldn't work since its for cosmic rays mainly.

How about using something like np.iinfo or np.finfo along with arr.dtype to figure out the max value of the array?

I'm a little weary of depending on cosmic rays to determine the maximum pixel brightness of an image. What happens if there are no cosmic rays present in the dark frame? If the maximum pixel value in the dark frame is 300 ADU and the median is 290 ADU, yet the data type permits a maximum value of 65,535 ADU (since it's uint16), wouldn't this block of code end up discarding the image?

If my analysis is incorrect, please let me know.

[ivenzor]

Hi @tamimfatahi, thanks for your feedback. My approach is not assuming cosmic rays but rather the presence of hot pixels or any bright pixel in the dark frame. I'm assuming every dark frame has at least one of those who yield the saturation value I need (at least every camera I have used has several of them).

About your suggestion, I think that would give us the theoretical max value of the dtype but I don't know how the instrument would use the dynamic range. Would the theoretical max value will be used if the dark file is somewhat saturated? That's why I chose to use the value of a hot pixel as a saturation proxy.

You have much more experience in this, if the theoretical value is reached in practice we can go that route. I will do some testing later tonight about this.

[tamimfatahi]

About your suggestion, I think that would give us the theoretical max value of the dtype but I don't know how the instrument would use the dynamic range. Would the theoretical max value will be used if the dark file is somewhat saturated? That's why I chose to use the value of a hot pixel as a saturation proxy.

I believe you're right on that. I just checked MObs CCDs and they go up to 4095 ADU, yet are stored in a 16bit format. If you feel confident that there will always be saturated pixels in these exposures, then that's a good way to go about it!

[ivenzor]

Two improvements, instead of comparing the max in each dark frame, we can check all the dark files first to get the overall max and then follow the same logic: use that max as a proxy for saturation and then compared the mean of each dark file frame to that saturation value as the criteria for rejecting the dark frame. The other improvement is not to use 50% as a cutout, but 80% to be even more sure the dark frame is not valid:

            if exotic_infoDict['darks']:
                # First pass: find maximum value across all dark files
                # this value will be a proxy for the saturation value
                # EXOTIC will reject a dark frame if it's too bright, i.e. its median relative to max_across_darks
                max_across_darks = 0
                for darkFile in exotic_infoDict['darks']:
                    darkData = fits.getdata(darkFile)
                    file_max = np.max(darkData)
                    max_across_darks = max(max_across_darks, file_max)
                # Second pass: validate darks
                darksImgList = []
                for darkFile in exotic_infoDict['darks']:
                    darkData = fits.getdata(darkFile)
                    median_val = np.median(darkData)
                    ratio = median_val / max_across_darks  # Compare dark file median to saturation value
                    #If the median of all the pixels in the darkfile is above 80% of the saturation value, the file is likely not a valid dark file
                    if ratio > 0.8:
                        log_info(f"\nWarning: Skipping suspicious dark frame {darkFile}: median/max ratio = {ratio}\n", warn=True)
                        continue
                    darksImgList.append(darkData)

                if not darksImgList:
                    log_info(f"\n\nNo valid dark frames found! Proceeding without dark correction.")
                else:
                    #Create the master dark file to calibrate the science frames
                    generalDark = np.median(darksImgList, axis=0)

Example using a bad dark frame:
Warning: Skipping suspicious dark frame C:\Users\ivenz\Downloads\5760632e13dfc57fd6e163dc3a05e182\darks\Dark-B-160605040038.FITS.gz: median/max ratio = 1.0

[tamimfatahi]

I was going to suggest this morning to look at one of the light exposures and grab the max pixel value, as there should always be at least one there. Looks like you found a different approach to help ensure there is a saturated pixel

[rzellem]

What's the TL;DR - is this ready to be approved?

[tamimfatahi]

What's the TL;DR - is this ready to be approved?

We need your expertise here @rzellem:

1. Is it sufficient to determine the detector's maximum value by analyzing all dark frames to find the maximum, or by examining a single light frame? I don't have enough experience looking at darks for saturated pixels. I do know they will exist in a light image though.

2. Should we discard a dark frame if the median pixel value exceeds 80% of the frame's maximum pixel value? I agree with the tolerance here.

[ivenzor]

@tamimfatahi @rzellem

1. I’ve updated the PR to include also checking a science frame to determine the max pixel value, in addition to the dark frames. In this way we doesn't depend on hot/saturated pixels in the dark frames.

2. Just to clarify, we would discard a dark frame if its median is greater than the 80% of the value we got in 1)
   We basically want to ignore saturated dark frames that in turn affect the master dark which in turn overcorrect the science frame and cause negative pixel scales.

[rzellem]

Sounds like @ivenzor took care of #1 - but I would do a check by looking at the max pixel value according to the bit size of each pixel. That won't necessarily tell if you if saturation is occurring (e.g., for low gains), but it will give you a good idea.

For 2: Well, you could potentially have a camera with terrible dark noise - if that's the case, then you could violate your trigger and remove valid data. What about doing a neighbor check- by comparing a dark to other darks? That will tell you if one (or maybe a few) of them is bad. But a user could confuse darks with bias or flats ... and I'm not sure there's a really good way to do that. One potential way would be to see if the FITS header has info about the dark current for the camera, but I doubt that all cameras have this listed in the header. So I think doing a "how does this one image compare to the others of the same cal type" test is probably the safest thing to do.

[ivenzor]

@rzellem, it seems all methods have some drawbacks. For example, besides what you already mentioned, if we use the 'comparison of the same type' method, we won’t be able to discard invalid dark frames if there’s only one dark file or if all the available dark frames are saturated or too bright. But I agree that probably this is the safest method to use.

[ivenzor]

@tamimfatahi I have updated the PR, it will now use similarity among dark frames to exclude dark frames. Can you take a look?