Check dialog is modal flag instead of open attribute in 'check popove…

…r validity' algorithm

The following case currently does not fail:

```
dialog.showModal();
dialog.open = false;
dialog.showPopover();
```

even though it is a top layer conflict:

1. `showModal` pushes to the top layer
2. `.open = false` then hides the dialog, while keeping it in top layer
3. `showPopover` pushes to the top layer again

The check for the open attribute is supposed to prevent top layer conflicts, since those conflicts
can leave the element in a broken state, notably after hiding. For instance, the previous sequence
of actions followed by `dialog.hidePopover()` will leave the `is modal` flag despite the dialog
being completely removed from the top layer.

Also only remove from top layer in `HTMLDialogElement.prototype.close()` when the is modal flag is
true to prevent that method from removing popovers from the top layer.

Fixes whatwg#9335

source

@@ -60289,14 +60289,14 @@ interface <dfn interface>HTMLDialogElement</dfn> : <span>HTMLElement</span> {
    <li><p>Remove <var>subject</var>'s <code data-x="attr-dialog-open">open</code>
    attribute.</p></li>
 
+   <li><p>If the <span>is modal</span> flag of <var>subject</var> is true, then <span data-x="list
+   remove">remove</span> it from its <code>Document</code>'s <span>top layer</span>.</p></li>
+
    <li><p>Set the <span>is modal</span> flag of <var>subject</var> to false.</p></li>
 
    <li><p>If <var>result</var> is not null, then set the <code
    data-x="dom-dialog-returnValue">returnValue</code> attribute to <var>result</var>.</p></li>
 
-   <li><p>If <var>subject</var> is in its <code>Document</code>'s <span>top layer</span>, then <span
-   data-x="list remove">remove</span> it.</p></li>
-
    <li>
     <p>If <var>subject</var>'s <span>previously focused element</span> is not null, then:<p>
 
@@ -82793,8 +82793,8 @@ dictionary <dfn dictionary>DragEventInit</dfn> : <span>MouseEventInit</span> {
      <li><p><var>expectedDocument</var> is not null and <var>element</var>'s <span>node
      document</span> is not <var>expectedDocument</var></p></li>
 
-     <li><p><var>element</var> is a <code>dialog</code> element and has an <code
-     data-x="attr-details-open">open</code> attribute</p></li>
+     <li><p><var>element</var> is a <code>dialog</code> element and its <span>is modal</span> flag
+     is set to true.</li>
 
      <li><p><var>element</var>'s <span>fullscreen flag</span> is set</p></li>
     </ul>