Skip to content

Support for custom science Workflows #1082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
phycodurus opened this issue Oct 17, 2024 · 2 comments
Open

Support for custom science Workflows #1082

phycodurus opened this issue Oct 17, 2024 · 2 comments
Labels
enhancement New feature or request toolkit customization Would enhance the user customization experience

Comments

@phycodurus
Copy link
Contributor

phycodurus commented Oct 17, 2024

This is just an idea to support the defintion and execution of custom workflows. For example, if a workflow consist of steps such as:

  1. receive alert
  2. create target (or region)
  3. perhaps identify targets of interest with that region
  4. do analysis of target(s) perhaps to categorize, prioritize, or determine cadence of follow-up observations
  5. make observations
  6. analysis of follow-up observations
  7. loop back to step 3 or ???

The idea is to provide a frame work with integration points where by a (perhaps yaml-defined) workflow could be started and tracked (perhaps with a Jenkins-like UI).

Again, this is just an idea.

@phycodurus phycodurus converted this from a draft issue Oct 17, 2024
@jchate6 jchate6 moved this from Triage to Icebox in TOM Toolkit Oct 17, 2024
@jchate6 jchate6 added enhancement New feature or request toolkit customization Would enhance the user customization experience labels Oct 17, 2024
@phycodurus
Copy link
Contributor Author

Latest thinking on this is incorporating prefect with adapters that let TOM developers focus on writing @tasks and @flows without having to worry too much about the infrastructure.
Here's an idea:

@KKruszynska, we'd like to hear about your MOP use case, so we can provide support. Please feel free to comment.
@griffin-h, your SAGUARO thoughts are welcome.
@Fingel, fyi

@KKruszynska
Copy link
Contributor

Hi, here is an example of a workflow we would like to implement into MOP at some point (to some extent it's already there).

A series of targets receive new data (from surveys, follow-up, etc.). For each, we would like to find new best-fitting models and evaluate if there are any ongoing anomalies (meaning, the presence of a planet or a second object in the lens). At later steps, this information would be used to assign the correct observing strategy. Ideally, this would have been done in parallel and there would be a controller process that assigns each free thread a new target.

A flow for a single target would look like this:

  1. Perform fitting in the "simple" model category (point source point lens microlensing model (PSPL) with or without blending and/or parallax effect).
  2. Evaluate found models and select the best-fitting one.
  3. Check how many points are in the light curve. If there is a sufficient number...
  4. Attempt an "intermediate" model fit (PSPL and finite source effect, with or without parallax effect).
  5. Evaluate new-found models with the old ones and select the best.
  6. Perform a check for anomalies. If an anomaly is found...
  7. Perform fitting of "advanced" models (binary model, with or without parallax and/or finite source effect)
  8. Evaluate new models and select the best fitting one.
  9. Perform a check for anomalies in the results that were selected for an advanced model. Report the outcome of the check.

A result of such flow would be a best-fitting model assigned to the event, along with the results for all found models, some statistics used to evaluate the quality of the fit and its significance and the results of the anomaly checks.

Let me know if that makes sense. @rachel3834 feel free to add or remove anything from the list I wrote above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request toolkit customization Would enhance the user customization experience
Projects
Status: Icebox
Development

No branches or pull requests

3 participants