Commit a7be540
authored
[KVCache] Initialize one extra page than specified (#16849)
This PR udpates PagedKVCache to initialize one more page than
specified via constructor. The reason is that applications usually
depends the number of free pages (returned from `GetNumAvailablePages`)
to decide the KV cache operation policy. If there is no this extra
page, the KV cache will tell "no available" pages even when the
last allocated pages are not full, which may give the applications
an illusion that the KV cache is already completely full, and cause
further issues.1 parent a156181 commit a7be540
1 file changed
+2
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1790 | 1790 | | |
1791 | 1791 | | |
1792 | 1792 | | |
1793 | | - | |
| 1793 | + | |
1794 | 1794 | | |
1795 | 1795 | | |
1796 | 1796 | | |
| |||
1827 | 1827 | | |
1828 | 1828 | | |
1829 | 1829 | | |
1830 | | - | |
| 1830 | + | |
1831 | 1831 | | |
1832 | 1832 | | |
1833 | 1833 | | |
| |||
0 commit comments