Check dialog is modal flag instead of open attribute in 'check popove…

…r validity' algorithm

The following case currently does not fail:

```
dialog.showModal();
dialog.open = false;
dialog.showPopover();
```

even though it is a top layer conflict:

1. `showModal` pushes to the top layer
2. `.open = false` then hides the dialog, while keeping it in top layer
3. `showPopover` pushes to the top layer again

The check for the open attribute is supposed to prevent top layer conflicts, since those conflicts
can leave the element in a broken state, notably after hiding. For instance, the previous sequence
of actions followed by `dialog.hidePopover()` will leave the `is modal` flag despite the dialog
being completely removed from the top layer.

Fixes whatwg#9335

source

@@ -82793,8 +82793,8 @@ dictionary <dfn dictionary>DragEventInit</dfn> : <span>MouseEventInit</span> {
      <li><p><var>expectedDocument</var> is not null and <var>element</var>'s <span>node
      document</span> is not <var>expectedDocument</var></p></li>
 
-     <li><p><var>element</var> is a <code>dialog</code> element and has an <code
-     data-x="attr-details-open">open</code> attribute</p></li>
+     <li><p><var>element</var> is a <code>dialog</code> element and its <span>is modal</span> is
+     set to true.</li>
 
      <li><p><var>element</var>'s <span>fullscreen flag</span> is set</p></li>
     </ul>