Change how EXOTIC scrapes AAVSO comparison stars

[tamimfatahi]

Instead of based on the target's RA & Dec, use the center of the FOV with a known radius. Also, increase the mag limit of the VSX query.

The above is incorrect. We do query based on the center of the image. However, for small detectors (as Heath observed with), the star's tend to leave the FOV. Normally, this isn't an issue as EXOTIC will skip analyzing the comparison star. However, it can delay the time greatly, causing the Colab to disconnect before the reduction is finished.

[ivenzor]

@rzellem @pearsonkyle , I’ve been working with the dataset provided by @tamimfatahi (I believe it’s Heath’s) and have discussed a potential fix with Tamim.

TL;DR: There’s a possible quick 'fix' to re-enable AID in Colab, though it comes with some implications. See details below.

Current situation: AID is currently disabled in Colab by forcing "Add Comparison Stars from AAVSO? (y/n)": "n" in the json file.
The main reason for this was an error where the AAVSO-recommended comparison star was relatively near the border, causing errors like the following for each frame and eventually Collab will time out:

Finding transformation 41 of 401 : C:\Users\ivenz\Downloads\exotic_sample_data_error_2024-10-21\2024-10-21\LIGHT\HD 209458_2102.FIT
 Sky background error for Comparison star #4 for file C:\Users\ivenz\Downloads\exotic_sample_data_error_2024-10-21\2024-10-21\LIGHT\HD 209458_2102.FIT - are you sure there is a star at [908.7, 579.7]?

There is indeed a recommended AAVSO star at those coordinates; however, since it’s somewhat close to the border and the FOV is small, the resulting mesh_box extends slightly outside, causing an error. I implemented additional logging to get a better understanding of this situation:

 Starting skybg_phot for star 4
 Image dimensions: 1024x1024
 Star position: (x=915.6, y=582.0)
 Annulus parameters: r=15.376916435500215, dr=93.6384150485015
 mesh_box called with pos=[915.6115897571043, 581.99428600454598], box=109.0, maxx=0, maxy=0
 Generated coordinate ranges:
 x range: 807-1025
 y range: 473-691
 WARNING: Coordinates extend outside image boundaries!
 X coordinates outside bounds: needs 0-1023, got 807-1025
 Annulus mask shape: (219, 219)
 Number of pixels in annulus: 36591
 Data shape: (1024, 1024)
 Indexing shapes - yv: (219, 219), xv: (219, 219)
  Sky background error for Comparison star #4 for file C:\Users\ivenz\Downloads\exotic_sample_data_error_2024-10-21\2024-10-21\LIGHT\HD 209458_2062.FIT - are you sure there is a star at [915.6, 582.0]?

As we can see in the extended logs, the star is within the frame, but the resulting mesh_box boundaries extend slightly outside the frame, causing an error in the skybg_phot function:

     try:
        selected_data = data[yv, xv][mask]  ##THIS WILL FAIL
        cutoff = np.nanpercentile(selected_data, ptol)
    except IndexError:
        plateStatus.skyBackgroundWarning(starIndex, xc, yc) #ERROR: Sky background error ... 
        log.debug(f"Warning: IndexError, problem computing sky bg for {xc:.1f}, {yc:.1f}."
                 f"\nCheck if star is present or close to border.") 

As I mentioned there is a potential quick fix. In the skybg_phot function we can change how the mesh_box is called.
Instead of this:

    # create a crude annulus to mask out bright background pixels
    xv, yv = mesh_box([xc, yc], np.round(r + dr))

We can change it to this:

    # create a crude annulus to mask out bright background pixels
    xv, yv = mesh_box([xc, yc], np.round(r + dr), maxx=image_width, maxy=image_height)

@rzellem @pearsonkyle Question: is sampling a partial annulus near the edges acceptable for our sky background estimation?
If this approach is acceptable, I can go ahead and submit a PR.

If not, the alternative would be to avoid recommending AAVSO comparison stars that are 'near' the border. I haven’t explored this yet, but I would likely need to examine how r and dr are calculated to prevent recommending AAVSO stars that could cause the mesh_box coordinate ranges to extend beyond the frame borders.

If we decide to implement the quick fix, we could re-enable AID in Colab, though there would still be some rough edges. For example, there are cases where MERR is greater than one in the AID file, the magnitude 14 limit still applies, and so on. We’ll need to address these issues regardless of the path we choose, but I think re-enabling AID would be the first step.

I did some testing with and without the quick fix using Heath's dataset:

long run (no fix): 4h 40min
run with fix:      1h 12min

Rp**2/Rs**2 long run (no fix): 0.0251+0.0011
Rp**2/Rs**2 run with fix:      0.02512+0.00095

Tmid long run (no fix): 2460605.7732+0.007
Tmid run with fix:      2460605.7732+0.007

Residuals long run (no fix): 3.01%
Residuals run with fix:      2.89%

Best comparison star long run (no fix): None
Best comparison star run with fix:      None

AID file long run (no fix): Yes
AID file run with fix:      Yes 

Should we implement the quick fix to re-enable AID, or should we explore the other option?
Your feedback would be greatly appreciated.

[rzellem]

@ivenzor - Thanks for all of your awesome work on this!

If the sky background is calculated with a median, then having a sky annulus that partially falls off of the edge of the detector should be ok- our stats will just be a little worse by the limited number of pixels.

Personally, I’d prefer not using a comp star near the edge of the detector - as it could eventually “fall off” during the night with tracking or, being (assumedly) comparatively far from the target star at the center of the FOV, the comp star could have very different systematics compared to the target star (e.g., due to vignetting).

So, another option is just to have EXOTIC “throw away” a comp star if part of its annulus falls off of the edge of the detector. That might be a simpler fix?

What do you think, @ivenzor ?

[ivenzor]

@rzellem , I agree; let’s focus on avoiding comparison stars near the border. I’ve updated the vsp_query function:

1. We now calculate a 'valid' region based on a fraction relative to the edges. If the target star is near the border, the region is adjusted to extend only up to the target star in that direction.
2. We then validate the AAVSO JSON to ensure the star is within this valid region; if it isn’t, we reject it.

Like this:

Image dimensions: 1024x1024 pixels
Initial margin fraction: 0.15
Initial margin size in pixels - X: 153, Y: 153
Target star position: (510, 458)

Safe region in pixels:
X: 153 to 871 (width: 718)
Y: 153 to 871 (height: 718)
Rejected star at position (914.5, 580.5) - outside valid region  ##WE REJECT PREV AAVSO COMP STAR

Added Comparison Star #4, coordinates [614, 259] from AAVSO  ##A NEW VALID COMP STAR IS CHOSEN

Main changes in code:

    # Calculate initial margins in pixels
    margin_x = int(axis[0] * margin_fraction)
    margin_y = int(axis[1] * margin_fraction)
    ...
    x_min = margin_x
    x_max = axis[0] - margin_x
    y_min = margin_y
    y_max = axis[1] - margin_y

    if data['photometry']:
            ...
            # Check if star falls within our valid region box
            if (x_min < ra_pix < x_max and y_min < dec_pix < y_max): 
                ...

I did a full reduction and compared the results:

original run with skybgr errors: 			4h 40min
prev run with partial annulus workaround:      		1h 12min
new run excluding aavso comparison stars near border: 	2h 25 min


Rp**2/Rs**2 original run with skybgr errors: 				0.0251+0.0011
Rp**2/Rs**2 prev run with partial annulus workaround:      		0.02512+0.00095
Rp**2/Rs**2 new run excluding aavso comparison stars near border:	0.0251+0.00083

Tmid original run with skybgr errors: 				2460605.7732+0.007
Tmid prev run with partial annulus workaround:      		2460605.7732+0.007
Tmid new run excluding aavso comparison stars near border:	2460605.7732+0.0047

Residuals original run with skybgr errors: 			3.01%
Residuals prev run with partial annulus workaround:      	2.89%
Residuals new run excluding aavso comparison stars near border:	2.96%

Best comparison star original run with skybgr errors: 				None
Best comparison star prev run with partial annulus workaround:      		None
Best comparison star new run excluding aavso comparison stars near border:	None

AID file original run with skybgr errors: 			Yes
AID file prev run with partial annulus workaroundx:      	Yes
AID file new run excluding aavso comparison stars near border:	Yes

@rzellem , is this approach acceptable?

[ivenzor]

@rzellem do you like this approach? Should I move forward and make a PR?

[rzellem]

Hi @ivenzor - I’m a little hesitant to have a “valid region” on the image - as that could vary wildly from detector to detector. Instead, what you could do is have EXOTIC return something like a np.nan if part of the aperture or annulus falls off the edge of the detector. And then simply this one image alone is removed from the analysis.

Sometimes a single image might be completely off, but not all of the images - so we’d only want to remove that single image, but the rest of them might be awesome. So thinking about it more, with more coffee, I’m wondering if we shouldn’t automatically throw away a comp star….

…sorry, I’m still processing all of this while I think about it lol

[ivenzor]

@rzellem just to clarify, the 'valid region' I suggested is not fixed or static for all the detectors but would change based on the detector or rather based on the actual image dimensions. In my example, the valid region was calculated using a 15% margin from the edges as it's now a function parameter with 15% as default. Something like this:

But yes, this will reject an AAVSO comparison star based on the first image if it's outside this zone.

About rejecting the individual frame itself, that would work great if only a few of the images have an AAVSO comparison star near the border, but if all of them do, we would reject all images.

[rzellem]

@ivenzor - hmm...I'm still a little worried about the "valid region" - as there might be only one AAVSO comp and it might lie in the "non"-valid region - I think I'd rather get the data, even at the risk of it being bad

So...revisiting/rethinking about your "if part of the annulus goes off of the edge of the detector, is this ok" question - I think the answer is "yes" - just technically our stats will be worse for estimating the background flux for these images due to less pixels available - that could manifest as an incorrect background subtract, and therefore more variance in the data...but EXOTIC should automatically filter these bad images out

...so I'm thinking that your easy fix of just allowing for annuli that do partially fall off the detector to just use the "valid" pixels that don't fall off as the annulus calculation and just go from there

What do you think?

[ivenzor]

@rzellem, for the sky background, we are using the mode. If the mesh is smaller, there may be fewer pixels to sample from, but the mode should still be reliable unless there are very few pixels. Even in that case, I guess EXOTIC is unlikely to choose this star as a comparison star.

I believe this approach has the highest benefit-to-cost ratio, as it avoids rejecting valuable AAVSO stars upfront.

Based on this, I will create a PR to implement the 'quick fix':
xv, yv = mesh_box([xc, yc], np.round(r + dr), maxx=image_width, maxy=image_height)

[rzellem]

Sounds great to me, @ivenzor - thank you so very much!