Enhanced Signal Management

[MattToast]

Fixes unfalsifiable test that tests SmartSim's custom SIGINT signal handler. Adds infrastructure to make the test pass again.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.76%. Comparing base (d462fd2) to head (bbd3f5c).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #535      +/-   ##
===========================================
+ Coverage    90.70%   90.76%   +0.06%     
===========================================
  Files           65       65              
  Lines         4498     4528      +30     
===========================================
+ Hits          4080     4110      +30     
  Misses         418      418              

Files	Coverage Δ
smartsim/_core/control/controller.py	85.91% <100.00%> (ø)
smartsim/_core/control/jobmanager.py	94.15% <ø> (-0.04%)	⬇️
smartsim/_core/utils/helpers.py	93.70% <100.00%> (+1.74%)	⬆️

[ashao]

Looks really good and thanks for fixing this up. It was both a developer and user annoyance. Two comments:

Could you change the name of this PR so that it reflects the effect that these changes have?

I'm not quite seeing the need to have enforce unique callables within the SignalInterceptionStack class itself. I can see that with multiple start calls, you're going to register multiple signals. I'm going back and forth on if it's 100% necessary that that the class enforces uniqueness (and add to some of its complexity) or whether there's a world where it might be useful to have multiple identical callbacks. For example, the current implementation prevents you from popping something multiple times adding it as a callback.

For the one place that this class is used, it's clearly incorrect for the signal handle to be registered more than once. I'm about 60% on the side of that it is the Controller's responsiblity to make sure it doesn't do wonky things when registering a callback and not the class' responsibility. Happy to be argued the other way though

[ashao]

Looks great. Thanks for making hte changes