-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate edge cases #98
Comments
Minimal modifier that produces a counterexample for newtype XTreme a = XTreme a deriving Show
instance (Arbitrary a, Num a, Bounded a) => Arbitrary (XTreme a) where
arbitrary = let
prob = [(1, pure minBound), (1, pure (minBound + 1)), (1, pure (minBound + 2)),
(1, pure maxBound), (1, pure (maxBound - 1)), (1, pure (maxBound - 2)),
(1000-6, arbitrary)]
in XTreme <$> frequency prob
>>> quickCheckWith stdArgs { maxSuccess = 1000 } (\(XTreme (a :: Int)) -> abs a >= 0)
*** Failed! Falsifiable (after 94 tests):
XTreme (-9223372036854775808) |
Found a similar version in the tests: newtype Extremal a = Extremal { getExtremal :: a } deriving (Show, Eq, Ord, Num, Enum, Real, Integral)
instance (Arbitrary a, Bounded a) => Arbitrary (Extremal a) where
arbitrary =
fmap Extremal $
frequency
[(1, return minBound),
(1, return maxBound),
(8, arbitrary)]
shrink (Extremal x) = map Extremal (shrink x) |
I'd rather do {-# LANGUAGE ScopedTypeVariables, GeneralizedNewtypeDeriving #-}
import Test.QuickCheck
newtype EB a = EB { getEB :: a }
deriving (Show, Eq, Ord, Num, Enum, Real, Integral)
instance (Enum a, Bounded a) => Arbitrary (EB a) where
arbitrary = EB <$> frequency
[ (20, toEnum <$> choose (i, j))
, (1, return minBound)
, (1, return maxBound)
]
where
i = fromEnum (minBound :: a)
j = fromEnum (maxBound :: a)
shrink = fmap (EB . toEnum) . shrink . fromEnum . getEB |
Seems like a good idea! However, we shouldn't do it for the default prop_replicate_length :: Int -> String -> Property
prop_replicate_length n x = n >= 0 ==> length (replicate n x) === n But adding it to the |
Thanks for the comments, I think this should be mentioned in the documentation — a user may not know to use a modifier if their tests pass for the default generator. Jotting down some ideas Alternative reality where
Using a modifier to generate small numbers explicitly makes sense, not suggesting that as a change though prop_replicate_length :: Small Int -> String -> Property
prop_replicate_length (Small n) x = n >= 0 ==> length (replicate n x) === n unrelated, given prop_replicate_length :: Positive (Small Int) -> String -> Property
prop_replicate_length (Positive (Small n)) x = length (replicate n x) === n |
Hi, I had some issues with quickcheck not generating special values, so I made a small package for it which uses a Special newtype similar to what was proposed here.
Is this going in the right direction concerning this issue? Maybe something like that could be added to QuickCheck at some point? |
Yes, generating special values is definitely a good idea! The approach I would take is to change the existing "Small" special values like |
In principle I agree that everything should be generated. However for Float, the NaN values are problematic since they violate the typeclass laws. |
Oh yes, good point. NaN should be left out of that. Maybe things like infinity are also problematic. Perhaps we should be conservative and have the principle that we only change the distribution of generated values, i.e. we generate values that it was unlikely to generate before, but not values that it was impossible to generate before. Probably that means we don't change the distribution of |
Humour me, how about an additional method or subclass of class Arbitrary a where
arbitrary :: Gen a
shrink :: a -> [a]
shrink _ = []
extreme :: [a]
extreme = []
tooExtreme :: [a]
tooExtreme = [] Now we have some control over extreme values: how extreme we want them as well as their frequency instance Arbitrary Float where
...
extreme :: [Float]
extreme = [0, -0, 2.22044604925031308085e-16, 4.94065645841246544177e-324, 2.22507385850720138309e-308, 1.79769313486231570815e+308]
tooExtreme :: [Float]
tooExtreme = [0/0, 1/0, -1/0]
instance Arbitrary Int where
extreme :: [Int]
extreme = [0, 1, -1, minBound, maxBound, minBound + 1, maxBound - 1]
instance Arbitrary Char where
extreme :: [Char]
extreme = [minBound, maxBound, succ minBound, pred maxBound] Edit: Just ike class SpecialValues a where
-- | Finite list of special values
specialValues :: [a] It could include frequency information, |
Just found out that quickcheck doesn't generate NaN and was quiet surprised. Can you explain why exactly? Because for me it seems that this value should definitely present in generation as soon as developers tend to forget about it and then weird things happen and this is what the quickcheck for - to help to reveal edge cases, isn't it? |
That an input contains no |
Right, my thinking was that passing NaN to a function means that an error has already occurred - in this respect it seems very similar to undefined. I would expect most numeric algorithms to implicitly have a precondition that the input does not contain NaN. However, it would definitely be useful to have a modifier so that the programmer can opt into testing with NaN. |
👍 to the modifier for NaN |
Sorry, I'm not that familiar with Haskell, but as far as I understand NaN is a valid floating point number representation and the code won't throw error if you try to operate on that, which is not the case for And it would make perfect sense for me as a maintainer of a library to check my code against NaN if I have function accepting floats. I can totally see that it is a breaking change if you would introduce the NaN as a generated float value, but theoretically I see it more as a profit that when I test my function with your library I would be forced to check input value for NaN. It might be in function itself, or in the test: both will make behaviour of my code more clear to the reader. |
And regarding the switch, I can say that personally I would prefer the switch to turn it off rather than turn it on. |
I found this bug after being surprised at how hard it is to hit the I imagine other values such as |
Using @minad's package and a future extension |
It would be nice to have similar feature for |
Here's my hacky (EDIT: and not carefully tested) implementation of a modifier that tries to give the full range of possible Double values, edge cases included. It would be nice to have something like this as an option, even if it's not the default. I release this under the current QuickCheck licence (which is the 3-clause BSD licence). (EDIT: It uses
|
Generate edge cases (
minBound
,maxBound
,minBound + 1
, ...) with some low frequency.abs
always returns a positive number right?Alright let's ship it — well — the test fails if we run it on
Int8
s often enoughTurns out it doesn't actually hold for
Int
sThis could be detected for larger sample spaces like
Int
ifminBound
were generated. I would like to see this in theArbitrary Int
instance but maybe a modifier is better?The text was updated successfully, but these errors were encountered: