HTTP Distributor Tutorial is Broken on Windows

[FloatingSunfish]

I'm on Python 3.9.6 and following the HTTP Distributor tutorial produces the following error:

Traceback (most recent call last):
  File "c:\users\admin\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\admin\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\Scripts\cosmic-ray.exe\__main__.py", line 7, in <module>
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\cli.py", line 288, in main
    return cli(argv)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\cli.py", line 115, in handle_exec
    cosmic_ray.commands.execute(work_db, cfg)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\progress.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\commands\execute.py", line 46, in execute
    distributor(
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\distribution\http.py", line 45, in __call__
    asyncio.get_event_loop().run_until_complete(self._process(*args, **kwargs))
  File "c:\users\admin\appdata\local\programs\python\python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "c:\users\admin\appdata\local\programs\python\python39\lib\site-packages\cosmic_ray\distribution\http.py", line 87, in _process
    done, pending = await asyncio.wait(fetchers.keys(), return_when=asyncio.ALL_COMPLETED)
  File "c:\users\admin\appdata\local\programs\python\python39\lib\asyncio\tasks.py", line 392, in wait
    raise ValueError('Set of coroutines/Futures is empty.')
ValueError: Set of coroutines/Futures is empty.

Where can I find the latest Python version that Cosmic Ray supports?

Also, I am not sure if the problem is with the tutorial, with Cosmic Ray itself, or an incompatibility with the version of Python on my system.

Any advice?

[abingham]

I'll have a look at it. The HTTP distributor is pretty new, so there may well be problems in the implementation, tutorial, or anywhere.

[abingham]

Where can I find the latest Python version that Cosmic Ray supports?

The only accurate record of that is in the badges of the README right now. I'll update setup.py to more correctly report this.

[abingham]

My guess right now is that the problem is running on Windows. I'm able to work through the HTTP Distributor tutorial on python 3.9 / macos with no problems, so a platform difference seems like the most likely culprit.

I think I have a windows VM I can try this on, but it may take some time to get to it.

[abingham]

Actually, I think I see what's going on. The call to asyncio.wait mentioned near the end of the stack trace is to blame. Its documentation clearly states that the first argument fs must not be empty, and indeed the ValueError here is complaining that it is. So in src/cosmic_ray/distribution/http.py around line 75 we seem to have a bit of a race condition where urls has no elements but fetchers.keys() is somehow empty as well.

I think the error is on lines 68-69 where we remove a task from fetchers before appending a URL to urls, creating a window where both urls and fetchers are empty. I thought asyncio would prevent this, but perhaps not.

@FloatingSunfish If you're feeling adventurous, could you swap lines 68 and 69 in src/cosmic_ray/distribution/http.py, install CR from source, and try again?

[FloatingSunfish]

Thanks for looking into this, @abingham. I've been looking for any Python mutation testing tool and Cosmic Ray looks very promising.

I've looked at mutmut but it has an issue where, for some machines, it just stops running after 100 or so mutants, so I shelved it. I've also looked at MutPy, but it's so outdated that it doesn't seem to work anymore.

>@FloatingSunfish If you're feeling adventurous, could you swap lines 68 and 69 in src/cosmic_ray/distribution/http.py, install CR from source, and try again?
I'll see what I can do, but no promises as I can't commit to anything at the moment. 😅

[abingham]

I'm happy to help support you as much as I can, though my time is a bit limited right now. CR does work well when all of the stars align; as you're finding, though, I struggle to keep the documentation up to date, and I don't do any testing on Windows.

[FloatingSunfish]

@abingham
>I struggle to keep the documentation up to date
No worries, I'll be happy to report any issues I find if I end up using CR for our project. I just want something that works and so far, it looks like Cosmic Ray is it. We just have to work out a few kinks first. 😅

>I don't do any testing on Windows
I understand. I certainly don't do much work on other OSes myself, so users reporting issues on other OSes is pretty much how bugs will be found and fixed for this project, and that's perfectly fine. 🙂

[FloatingSunfish]

@abingham
How do I install CR from source, exactly? I can't find it in the docs. 😅

Perhaps it's not yet there?

[abingham]

How do I install CR from source, exactly?

From the root directory (the one containing setup.py) you can run pip install .. This is standard for any Python project with a setup.py, so that's why I don't include it in the docs.

[FloatingSunfish]

@abingham
I see. I've never done this before so I wouldn't know where this standard would be documented (I assume the Python docs).

I'll let you know the results as soon as I can. 🙂

[FloatingSunfish]

@abingham
>This is standard for any Python project with a setup.py, so that's why I don't include it in the docs.
I see. Still, it might be a good idea to add a small Installing from Source section after the Installation section for those unfamiliar with installing packages from the source, or just link to the official Python docs where this standard is documented.
Might be useful to somebody out there. 🙂

As for the solution you suggested, swapping those two lines didn't work and it also made the baseline constantly fail:

[07/23/21 17:47:55] INFO     INFO:root:Reading config from 'mutation_test_config.toml'                                                                                       config.py:103
                    INFO     INFO:cosmic_ray.commands.execute:Beginning execution                                                                                            execute.py:45
                    INFO     INFO:cosmic_ray.distribution.http:Sending HTTP request to http://localhost:9876                                                                   http.py:115
                    INFO     INFO:cosmic_ray.commands.execute:Job baseline complete                                                                                          execute.py:43
                    INFO     INFO:cosmic_ray.commands.execute:Execution finished                                                                                             execute.py:53
                    ERROR    ERROR:root:Baseline failed. Execution with no mutation gives those following errors:                                                               cli.py:149
                               >>> E
                               >>> ======================================================================
                               >>> ERROR: test_mod (unittest.loader._FailedTest)
                               >>> ----------------------------------------------------------------------
                               >>> ImportError: Failed to import test module: test_mod
                               >>> Traceback (most recent call last):
                               >>>   File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\unittest\loader.py", line 154, in loadTestsFromName
                               >>>     module = __import__(module_name)
                               >>> ModuleNotFoundError: No module named 'test_mod'
                               >>>
                               >>>
                               >>> ----------------------------------------------------------------------
                               >>> Ran 1 test in 0.000s
                               >>>
                               >>> FAILED (errors=1)
                               >>>

No idea why it can't find test_mod.py because it's in the current directory.
When I run python -m unittest test_mod.py in the same directory, all tests pass.

Also, when I ran exec the first time, it took very long and didn't look like it would finish (the docs say it should be very fast), so I canceled it.
Then, when I ran it a second time, it finished without reporting any errors.
When I ran it a third time, the same error message in the OP was raised (ValueError: Set of coroutines/Futures is empty.).

Perhaps it is a race condition as you said.

Testing on a Windows VM on your end might be the best way to investigate this issue, but feel free to toss some easy code edits my way and I'll try to see what I can check on my side as well. 🙂

[abingham]

I see that the baseline is using a HTTP Distributor. This should be fine, but are you sure you've started the http-worker in the directory that contains test_mod.py? If it's running in a different directory, that would explain why python -m unittest test_mod.py works but baselining fails.

[FloatingSunfish]

@abingham
>are you sure you've started the http-worker in the directory that contains test_mod.py? If it's running in a different directory, that would explain why python -m unittest test_mod.py works but baselining fails.
Ah, I see. From the docs, the impression I got was that I only needed to keep the worker running in its own terminal.

You might want to add this detail in the docs that you must only start the worker in the directory where you will be running CR. 🙂

In the meantime, I'll try again and see what happens.

[abingham]

I've updated the docs to add a note that effect.

[FloatingSunfish]

@abingham
Just tested the line 68-69 swap again.

The good news is that baseline testing is now working as expected and mutation testing passes with workers on the first try.

The bad news is that on the second and subsequent tries, mutation testing constantly fails with the error in the OP.
I use the same worker for all my exec runs.

[FloatingSunfish]

@abingham
Also, small typo in the docs:

Note that your worker must be running in the same directory as you would normally run the tests from.

[abingham]

The bad news is that on the second and subsequent tries, mutation testing constantly fails with the error in the OP.

Ok, then there's something more mysterious going on. I'll have to try it out on windows.

We may have to add some sort of check that fetchers.keys() is not empty and, if it is, add a small sleep or something. This feels kludgy, but maybe it's necessary for some valid reason I'm not aware of.

[FloatingSunfish]

@abingham
I see. Hopefully, it's not too complicated. 😅

Best of luck to you then! 🙂

[FloatingSunfish]

@abingham
Quick question, should you ever delete an existing Session DB and run init to start from scratch, or can you just run init once for the lifetime of the project and just keep running exec whenever you make any changes to the production code and your tests?

[abingham]

You need to re-init when you make changes. init discovers the mutations that need to be made, and creates the WorkItems in the database. If you don't re-initialize, the exec command will look in your session, see that all of the WorkItems already have results, and do nothing.

[FloatingSunfish]

@abingham
Aha, that's what I gathered from the docs as well, plus you also mentioned that init's Session DB is more than just a blank slate.

I think you should add this detail in the Initializing a session section that you need to run init every time you make changes to your production code. 🙂

Oh, and some more quick questions:

1. Does running init overwrite the existing Session DB (I assume it does), or do I have to delete it as well before I rerun init?

2. I assume that CR is robust enough to handle aborts (e.g. Ctrl+C) from the user as well as most crashes and is able to reliably pick up where it left off thanks to the Session DB. Is this correct?

[abingham]

Does running init overwrite the existing Session DB (I assume it does), or do I have to delete it as well before I rerun init?

init completely rewrites the session file. You don't need to delete it.

I assume that CR is robust enough to handle aborts (e.g. Ctrl+C) from the user as well as most crashes and is able to reliably pick up where it left off thanks to the Session DB. Is this correct?

CR is designed to be robust against interrupts and crashes. To a large degree, we rely on sqlite to handle that. When you run exec, it looks for all of the WorkItems with no results, and these tell CR what works needs to be done. Furthermore, we only add those results after a mutation/test is complete. So if CR dies for some reason, any running tests simply die, no results are written to the session, and processing picks back up with those incomplete WorkItems next time.

[FloatingSunfish]

Good day, @abingham. 🙂
I have some questions:

1. In case of interrupts and crashes, is there a possibility that CR won't be able to undo some of its mutations?
   Is it good practice to fully check-in projects before doing any mutation testing?

2. I assume you should also run init when you make changes to your config file.
   I didn't see this in the docs though, so perhaps it should be added.

3. Is there an option for cr-report to only print the progress summary (the last few lines at the end)?
   That's all I want to see when I check CR's progress for my scenario:

   total jobs: 2
   complete: 2 (100.00%)
   surviving mutants: 0 (0.00%)
   

4. Given the following module-path:

   module-path = "chalicelib"
   

   Is this the correct way to exclude files?

   excluded-modules = [
       "chalicelib/orchestration/cognito/botoWrapper.py",
       "chalicelib/orchestration/postgres/util.py"
   ]
   

   Or do I no longer need to include the chalicelib folder?
   Also, am I using the right path separator (a /)? I couldn't find an example in the docs.

5. I want CR to not use the == to is mutation:

   -        if attrib == UsecasesAttributes.EMAIL:
   +        if attrib is UsecasesAttributes.EMAIL:
   

   I added the following to my CR config based on the operator in the HTML report:

   [cosmic-ray.filters.operators-filter]
   exclude-operators = [
       "core/ReplaceComparisonOperator_Eq_Is"
   ]
   

   Is this correct, or is it going to disable other mutation Operators as well?
   I read in the Excluding Filters section that some RegEx is sometimes needed.

6. I'm not sure I understand how # pragma: no mutate works in CR.
   I assumed that it works like the one in Python's coverage tool so I added it at the end of a line that starts a block comment:

   class Repository(ABC):  # pragma: no cover - Interfaces can't be tested. # pragma: no mutate
   

   However, the mutation wasn't skipped even though I ran cr-filter-pragma mutation_test.sqlite before running exec:

   --- mutation diff ---
   --- achalicelib\businessRules\usecases\subscription\repository.py
   +++ bchalicelib\businessRules\usecases\subscription\repository.py
   @@ -7,7 +7,6 @@
    
    
    class Repository(ABC):  # pragma: no cover - Interfaces can't be tested. # pragma: no mutate
   -    @abstractmethod
        def selectOne(self, selectPayload: SelectPayload) -> Subscription:
            pass
   

   Could you explain what's going on here and what I should do?

[abingham]

1. In case of interrupts and crashes, is there a possibility that CR won't be able to undo some of its mutations? Is it good practice to fully check-in projects before doing any mutation testing?

CR tries hard to avoid this possibility, but it might still be possible. Tests are run in a subprocess, so even if they die horribly, the CR worker process should be able to undo the mutations. Generally speaking, though, I'd recommend committing everything prior to testing; this will be important, for example, if you're using the cr-http-workers tool that clones your repo as part of its operations.

2. I assume you should also run init when you make changes to your config file. I didn't see this in the docs though, so perhaps it should be added.

This isn't always strictly necessary. For example, changing your distributor won't require a re-initialization. Changing your test-command or excluded-modules generally would necessitate a re-init since this would effect which tests are run (thus potentially invalidating existing results).

Is there an option for cr-report to only print the progress summary (the last few lines at the end)?

Not currently, though that should be straightforward to implement if you're so inclined.

On a *nix system you could do something like:

cr-report session | tail -n 3

4. Is this the correct way to exclude files?

It looks like this feature doesn't work right now. I've create #517 as a reminder to reimplement this as a filter.

5. Is this correct, or is it going to disable other mutation Operators as well? I read in the Excluding Filters section that some RegEx is sometimes needed.

Yes, that looks correct. Regular expressions are an option, not a requirement. In your case of excluding a single operator, what you've done is fine.

6. I'm not sure I understand how # pragma: no mutate works in CR.

Our pragma filter only works on lines of code, not code blocks. It's quite a crude filter, really, but sufficient for many purposes.

[FloatingSunfish]

Noted. Thanks for the answers, @abingham! 🙂

Below are some follow-up questions:

1. This isn't always strictly necessary. For example, changing your distributor won't require a re-initialization. Changing your test-command or excluded-modules generally would necessitate a re-init since this would effect which tests are run (thus potentially invalidating existing results).

   Perhaps you should add a section on the docs for configuration changes that won't require another init so users become aware of them. I'm sure it would be useful to know.

2. Yes, that looks correct. Regular expressions are an option, not a requirement. In your case of excluding a single operator, what you've done is fine.

   I'm a bit confused about this particular line in the docs:

   You provide the filter with a set of regular expressions, and any Cosmic Ray operator who’s name matches a one of these expressions will be skipped entirely.

   The examples already include the full name of the operator, so I want to ask why regular expressions are still used. You can provide an explanation for just one of the examples:

   "core/ReplaceComparisonOperator_Is(Not)?_(Not)?(Eq|[LG]tE?)",
   "core/ReplaceComparisonOperator_(Not)?(Eq|[LG]tE?)_Is(Not)?",
   "core/ReplaceComparisonOperator_[LG]tE_Eq",
   "core/ReplaceComparisonOperator_[LG]t_NotEq",
   

3. Our pragma filter only works on lines of code, not code blocks. It's quite a crude filter, really, but sufficient for many purposes.

   I see. That's unfortunate, but it will have to do for now. Are there plans to make them behave similarly to the one in Python's coverage tool?

[abingham]

Could you bundle these documentation issues up into another issue? That'll make it easier for me to keep track of them. Or, if you're feeling energetic, you could update the docs yourself; I'm always happy to get these kinds of contributions because they almost always deal with my particular blind spots.

Are there plans to make them behave similarly to the one in Python's coverage tool?

Not right now. Something like this would presumably require syntactic analysis of the code, so it wouldn't be trivial (though I don't think it would be terribly complicated either). There may even be an existing library we could leverage that helps us process the pragmas.

If this is something you'd like, it would be helpful to create another issue for it.

[FloatingSunfish]

@abingham
I have created new issues to keep track of the unrelated ones we covered in this thread.

I also renamed this issue so it will be focused on the HTTP Distributor issue on Windows.

By the way, any updates on this issue?

Thanks! 🙂

[abingham]

Thanks for doing that. It really helps me keep track of what's going on.

I'm actually ready to close this issue. I'm nearly certain that #516 addresses your initial problems. It's not an issue on windows, just an issue with my logic in the HTTP distributor. So I'll close this issue and sort things out over there.