Filtering != treatment of NAs is unintuitive

[davidhodge931]

Base != seems to drop NA values.

This does not make sense, and would result in lots of people unintentionally dropping NA values.

library(dplyr)

df <- iris %>% 
  as_tibble() %>%
  mutate(Species = as.character(Species)) %>% 
  mutate(Species = ifelse(Species == "setosa", NA, Species)) %>% 
  distinct(Species)

df 

df %>% 
  filter(Species != "virginica") 

A proposed solution is that dplyr creates it's own != that masks the base != and keeps NAs

[zenggyu]

The default behavior is actually consistent with SQL standard.

[DavisVaughan]

!= doesn't drop anything. filter() is the one that drops NA rows, as is documented in the Description section at the top of ?filter:

Note that when a condition evaluates to NA the row will be dropped, unlike base subsetting with [.

library(dplyr)

df <- tibble(Species = c(NA, "versicolor", "virginica"))

df
#> # A tibble: 3 × 1
#>   Species   
#>   <chr>     
#> 1 <NA>      
#> 2 versicolor
#> 3 virginica

# `!=` doesn't drop anything
df$Species != "virginica"
#> [1]    NA  TRUE FALSE

# filter() does the dropping
filter(df, Species != "virginica")
#> # A tibble: 1 × 1
#>   Species   
#>   <chr>     
#> 1 versicolor

# Keep the missings
filter(df, Species != "virginica" | is.na(Species))
#> # A tibble: 2 × 1
#>   Species   
#>   <chr>     
#> 1 <NA>      
#> 2 versicolor

^{Created on 2021-09-14 by the reprex package (v2.0.0.9000)}

[davidhodge931]

Thanks @DavisVaughan

This seems like weird behaviour to me for !=, as NA is not equal to a specified (non-NA) value.

I noticed that filter does not drop NAs, when %in% is used? This does not seem consistent?

library(dplyr)
df <- tibble(Species = c(NA, "versicolor", "virginica"))
filter(df, Species != "virginica")
filter(df, !Species %in% "virginica")

[DavisVaughan]

That is because %in% never returns NA. It works differently than !=

species <- c(NA, "versicolor", "virginica")

species != "virginica"
#> [1]    NA  TRUE FALSE

!(species %in% "virginica")
#> [1]  TRUE  TRUE FALSE

From the docs of %in%, it returns:

A logical vector, indicating if a match was located for each element of x: thus the values are TRUE or FALSE and never NA.

---

This seems like weird behaviour to me for !=, as NA is not equal to a specified (non-NA) value.

99% of the time in R, missing values propagate through computations. That is the case in != too. A missing value on either side of != results in a missing value in the result. The main exception is match(), which powers %in% and merge(), which is what you see above.

[eutwt]

NA is not equal to a specified (non-NA) value.

It's not so clear that this is true. There is a justification for an opposing view, although you may not agree:

A value of NA is taken to mean there is some underlying value, but that value is not available / missing. It could be equal to the RHS, or it could not be equal to the RHS. We can't say the proposition 3 != NA is true because we don't know what the value behind NA is. For that reason, 3 != NA returns NA rather than TRUE (and filter only keeps rows where the matching elements of the ... arguments are TRUE)

[davidhodge931]

Thank you for the excellent explanations @DavisVaughan and @eutwt

Do you think it is possible and a good idea to add in a na.rm = TRUE argument, so that it always removes NAs whatever operator or %in% is used - and conversely keeps NAs if na.rm = FALSE?

[hadley]

Thanks for filing this bug report! Unfortunately this is a long-standing behaviour that matches the behaviour of base subset(); change it would cause widespread pain and I don't think introducing a new argument would be worth the development time. It's our policy to close such issues to help stay focussed on the biggest problems, but the issue is still indexed by google, so if other people hit it, they'll be able to find it, and we can consider reopen it if it turns out to be a common problem. Thanks for reporting and I'm sorry we couldn't help more 😞.

[davidhodge931]

Thanks @hadley