Skip to content

Commit

Permalink
Fixes database always updating (causing error) and simplifies mword c…
Browse files Browse the repository at this point in the history 
…heck.
  • Loading branch information
@HugoFara
HugoFara committed Jan 5, 2024
2 parents 8585c50 + ba1a149 commit c18e295
Show file tree
Hide file tree
Showing 3 changed files with 170 additions and 38 deletions.
6 changes: 4 additions & 2 deletions docs/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Many thanks to [@ProgramComputer](https://github.com/ProgramComputer)!
See [#160](https://github.com/HugoFara/lwt/issues/160).
It adds `README.md` and `UNLICENSE.md`.
* Adds `docs/info.html` and `docs/index.html` from [#146](https://github.com/HugoFara/lwt/pull/146).
* Multi-word creation was reviewed to use a simpler code.
* Multi-word creation was reviewed to use a simpler JS code.
* On word review, status + 1 can be set only after word display. Before, you could
press key up at any time to increase term status, but keydown was effective only
after solution display ([#159](https://github.com/HugoFara/lwt/issues/159)).
Expand All @@ -58,8 +58,10 @@ after solution display ([#159](https://github.com/HugoFara/lwt/issues/159)).
It defines interactions with a text on reading.
* For audio URI, raises the characters length limit from 200 to 2048
([#144](https://github.com/HugoFara/lwt/issues/144)).
* Database creation and migration is now handled directly in SQL, for a safer
* Database:
* Database creation and migration is now handled directly in SQL, for a safer
behavior.
* You do no longer need to give to LWT SYSTEM_VARIABLES_ADMIN for text reparsing. (new function `checkTextWithExpressions`) ([#167](https://github.com/HugoFara/lwt/issues/167)). It is still necessary for words import.
* Docker changes:
* By default, the images now runs at localhot/lwt and no longer in the
root folder ([installer #4](https://github.com/HugoFara/lwt-docker-installer/pull/4)
Expand Down
7 changes: 6 additions & 1 deletion docs/info.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2280,7 +2280,7 @@ <h4>Changed</h4>
<li>Adds <code>docs/info.html</code> and <code>docs/index.html</code> from <a href="https://github.com/HugoFara/lwt/pull/146">#146</a>.</li>
</ul>
</li>
<li>Multi-word creation was reviewed to use a simpler code.</li>
<li>Multi-word creation was reviewed to use a simpler JS code.</li>
<li>On word review, status + 1 can be set only after word display. Before, you could
press key up at any time to increase term status, but keydown was effective only
after solution display (<a href="https://github.com/HugoFara/lwt/issues/159">#159</a>).</li>
Expand All @@ -2299,8 +2299,13 @@ <h4>Changed</h4>
</li>
<li>For audio URI, raises the characters length limit from 200 to 2048
(<a href="https://github.com/HugoFara/lwt/issues/144">#144</a>).</li>
<li>Database:
<ul>
<li>Database creation and migration is now handled directly in SQL, for a safer
behavior.</li>
<li>You do no longer need to give to LWT SYSTEM_VARIABLES_ADMIN for text reparsing. (new function <code>checkTextWithExpressions</code>) (<a href="https://github.com/HugoFara/lwt/issues/167">#167</a>). It is still necessary for words import.</li>
</ul>
</li>
<li>Docker changes:
<ul>
<li>By default, the images now runs at localhot/lwt and no longer in the
Expand Down
195 changes: 160 additions & 35 deletions inc/database_connect.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1181,9 +1181,9 @@ function update_default_values($id, $lid, $sql)
{
global $tbpref;
do_mysqli_query(
'ALTER TABLE ' . $tbpref . 'textitems2
ALTER Ti2LgID SET DEFAULT ' . $lid . ',
ALTER Ti2TxID SET DEFAULT ' . $id
"ALTER TABLE {$tbpref}textitems2
ALTER Ti2LgID SET DEFAULT $lid,
ALTER Ti2TxID SET DEFAULT $id"
);
do_mysqli_query(
"INSERT INTO {$tbpref}textitems2 (
Expand All @@ -1200,16 +1200,16 @@ function update_default_values($id, $lid, $sql)
ALTER SeLgID SET DEFAULT ' . $lid . ',
ALTER SeTxID SET DEFAULT ' . $id
);
do_mysqli_query('set @a=0;');
do_mysqli_query('SET @i=0;');
do_mysqli_query(
'INSERT INTO ' . $tbpref . 'sentences (
"INSERT INTO {$tbpref}sentences (
SeOrder, SeFirstPos, SeText
) SELECT
@a:=@a+1,
min(if(TiWordCount=0,TiOrder+1,TiOrder)),
GROUP_CONCAT(TiText order by TiOrder SEPARATOR "")
FROM ' . $tbpref . 'temptextitems
group by TiSeID'
@i:=@i+1,
MIN(IF(TiWordCount=0, TiOrder+1, TiOrder)),
GROUP_CONCAT(TiText ORDER BY TiOrder SEPARATOR \"\")
FROM {$tbpref}temptextitems
GROUP BY TiSeID"
);
do_mysqli_query(
'ALTER TABLE ' . $tbpref . 'textitems2
Expand Down Expand Up @@ -1282,7 +1282,119 @@ function (k,v) {
}

/**
* Check a text that contains expressions.
* Check a language that contains expressions.
*
* @param int[] $wl All the different expression length in the language.
*
* @return string SQL-formatted query string
*
* @global string $tbpref Database table prefix
*/
function checkExpressions($wl): void
{
global $tbpref;

$wl_max = 0;
$mw_sql = '';
foreach ($wl as $word_length){
if ($wl_max < $word_length) {
$wl_max = $word_length;
}
$mw_sql .= ' WHEN ' . $word_length .
' THEN @a' . ($word_length * 2 - 1);
}
$set_wo_sql = $set_wo_sql_2 = $del_wo_sql = $init_var = '';
// For all possible multi-words length
for ($i=$wl_max*2 -1; $i>1; $i--) {
$set_wo_sql .= "WHEN (@a$i := @a".($i-1) . ") IS NULL THEN NULL ";
$set_wo_sql_2 .= "WHEN (@a$i := @a".($i-2) .") IS NULL THEN NULL ";
$del_wo_sql .= "WHEN (@a$i := @a0) IS NULL THEN NULL ";
$init_var .= "@a$i=0,";
}
// 2.8.1-fork: @a0 is always 0? @f always '' but necessary to force code execution
do_mysqli_query(
"SET $init_var@a1=0, @a0=0, @se_id=0, @c='', @d=0, @f='', @ti_or=0;"
);
// Create a table to store length of each terms
do_mysqli_query(
"CREATE TEMPORARY TABLE IF NOT EXISTS {$tbpref}numbers(
n tinyint(3) unsigned NOT NULL
);"
);
do_mysqli_query("TRUNCATE TABLE {$tbpref}numbers");
do_mysqli_query(
"INSERT IGNORE INTO {$tbpref}numbers(n) VALUES (" .
implode('),(', $wl) .
');'
);
// Store garbage
do_mysqli_query(
"CREATE TABLE IF NOT EXISTS {$tbpref}tempexprs (
sent mediumint unsigned,
word varchar(250),
lword varchar(250),
TiOrder smallint unsigned,
n tinyint(3) unsigned NOT NULL
)"
);
do_mysqli_query("TRUNCATE TABLE {$tbpref}tempexprs");
do_mysqli_query(
"INSERT IGNORE INTO {$tbpref}tempexprs
(sent, word, lword, TiOrder, n)
-- 2.10.0-fork: straight_join may be irrelevant as the query is less skewed
SELECT straight_join
IF(
@se_id=TiSeID and @ti_or=TiOrder,
IF((@ti_or:=TiOrder+@a0) is null,TiSeID,TiSeID),
IF(
@se_id=TiSeID,
IF(
(@d=1) and (0<>TiWordCount),
CASE $set_wo_sql_2
WHEN (@a1:=TiCount+@a0) IS NULL THEN NULL
WHEN (@se_id:=TiSeID+@a0) IS NULL THEN NULL
WHEN (@ti_or:=TiOrder+@a0) IS NULL THEN NULL
WHEN (@c:=concat(@c,TiText)) IS NULL THEN NULL
WHEN (@d:=(0<>TiWordCount)+@a0) IS NULL THEN NULL
ELSE TiSeID
END,
CASE $set_wo_sql
WHEN (@a1:=TiCount+@a0) IS NULL THEN NULL
WHEN (@se_id:=TiSeID+@a0) IS NULL THEN NULL
WHEN (@ti_or:=TiOrder+@a0) IS NULL THEN NULL
WHEN (@c:=concat(@c,TiText)) IS NULL THEN NULL
WHEN (@d:=(0<>TiWordCount)+@a0) IS NULL THEN NULL
ELSE TiSeID
END
),
CASE $del_wo_sql
WHEN (@a1:=TiCount+@a0) IS NULL THEN NULL
WHEN (@se_id:=TiSeID+@a0) IS NULL THEN NULL
WHEN (@ti_or:=TiOrder+@a0) IS NULL THEN NULL
WHEN (@c:=concat(TiText,@f)) IS NULL THEN NULL
WHEN (@d:=(0<>TiWordCount)+@a0) IS NULL THEN NULL
ELSE TiSeID
END
)
) sent,
if(
@d=0,
NULL,
if(
CRC32(@z:=substr(@c,CASE n$mw_sql END))<>CRC32(LOWER(@z)),
@z,
''
)
) word,
if(@d=0 or ''=@z, NULL, lower(@z)) lword,
TiOrder,
n
FROM {$tbpref}numbers , {$tbpref}temptextitems"
);
}

/**
* Check a language that contains expressions.
*
* @param int $id Text ID
* @param int $lid Language ID
Expand All @@ -1293,6 +1405,8 @@ function (k,v) {
* @return string SQL-formatted query string
*
* @global string $tbpref Database table prefix
*
* @deprecated Since 2.10.0-fork use checkExpressions. It does not modify SQL globals.
*/
function check_text_with_expressions($id, $lid, $wl, $wl_max, $mw_sql): string
{
Expand All @@ -1315,7 +1429,7 @@ function check_text_with_expressions($id, $lid, $wl, $wl_max, $mw_sql): string
// Create a table to store length of each terms
do_mysqli_query(
"CREATE TEMPORARY TABLE IF NOT EXISTS {$tbpref}numbers(
n tinyint(3) unsigned NOT NULL
n tinyint(3) unsigned NOT NULL
);"
);
do_mysqli_query("TRUNCATE TABLE {$tbpref}numbers");
Expand Down Expand Up @@ -1407,8 +1521,6 @@ function splitCheckText($text, $lid, $id)
{
global $tbpref;
$wl = array();
$wl_max = 0;
$mw_sql = '';
$lid = (int) $lid;
$sql = "SELECT LgRightToLeft FROM {$tbpref}languages WHERE LgID = $lid";
$res = do_mysqli_query($sql);
Expand Down Expand Up @@ -1438,25 +1550,36 @@ function splitCheckText($text, $lid, $id)
check_text_valid($lid);
}

// Get multi-word count
$res = do_mysqli_query(
"SELECT WoWordCount AS word_count, count(WoWordCount) AS cnt
FROM {$tbpref}words
WHERE WoLgID = $lid AND WoWordCount > 1
GROUP BY WoWordCount"
"SELECT DISTINCT(WoWordCount)
FROM {$tbpref}words
WHERE WoLgID = $lid AND WoWordCount > 1"
);
while ($record = mysqli_fetch_assoc($res)){
if ($wl_max < (int)$record['word_count']) {
$wl_max = (int)$record['word_count'];
}
$wl[] = (int)$record['word_count'];
$mw_sql .= ' WHEN ' . $record['word_count'] .
' THEN @a' . (intval($record['word_count']) * 2 - 1);
$wl[] = (int)$record['WoWordCount'];
}
mysqli_free_result($res);
$sql = '';
// Text has multi-words
if (!empty($wl)) {
$sql = check_text_with_expressions($id, $lid, $wl, $wl_max, $mw_sql);
checkExpressions($wl);
if ($id > 0) {
$sql = "SELECT straight_join WoID, sent, TiOrder - (2*(n-1)) TiOrder,
n TiWordCount, word FROM {$tbpref}tempexprs,
{$tbpref}words
WHERE lword IS NOT NULL AND WoLgID=$lid AND
WoTextLC=lword AND WoWordCount=n
UNION ALL ";
} else {
$sql = "SELECT straight_join COUNT(WoID) cnt, n as len,
LOWER(WoText) AS word, WoTranslation
FROM {$tbpref}tempexprs,
{$tbpref}words
WHERE lword IS NOT NULL AND WoLgID=$lid AND
WoTextLC=lword AND WoWordCount=n
GROUP BY WoID ORDER BY WoTextLC";
}
}
if ($id > 0) {
update_default_values($id, $lid, $sql);
Expand All @@ -1479,19 +1602,19 @@ function splitCheckText($text, $lid, $id)
function reparse_all_texts(): void
{
global $tbpref;
runsql('TRUNCATE ' . $tbpref . 'sentences', '');
runsql('TRUNCATE ' . $tbpref . 'textitems2', '');
runsql("TRUNCATE {$tbpref}sentences", '');
runsql("TRUNCATE {$tbpref}textitems2", '');
adjust_autoincr('sentences', 'SeID');
init_word_count();
$sql = "select TxID, TxLgID from {$tbpref}texts";
$sql = "SELECT TxID, TxLgID FROM {$tbpref}texts";
$res = do_mysqli_query($sql);
while ($record = mysqli_fetch_assoc($res)) {
$id = (int) $record['TxID'];
splitCheckText(
(string)get_first_value(
"SELECT TxText as value
from {$tbpref}texts
where TxID = $id"
"SELECT TxText AS value
FROM {$tbpref}texts
WHERE TxID = $id"
),
(string)$record['TxLgID'], $id
);
Expand Down Expand Up @@ -1569,7 +1692,7 @@ function update_database($dbname)
}

$changes = 0;
$res = do_mysqli_query("SELECT filename FROM _migrations");
$res = do_mysqli_query("SELECT filename FROM {$tbpref}_migrations");
while ($record = mysqli_fetch_assoc($res)) {
$queries = parseSQLFile(
__DIR__ . '/../db/migrations/' . $record["filename"]
Expand Down Expand Up @@ -1651,12 +1774,14 @@ function check_update_db($debug, $tbpref, $dbname): void
foreach ($queries as $query) {
$prefixed_query = prefixSQLQuery($query, $tbpref);
// Increment count for new tables only
$count += runsql($prefixed_query, "");
if (!str_starts_with($query, "INSERT INTO _migrations")) {
$count += runsql($prefixed_query, "");
}
}

// Update the database (if necessary)
update_database($dbname);

if (!in_array("{$tbpref}textitems2", $tables)) {
// Add data from the old database system
if (in_array("{$tbpref}textitems", $tables)) {
Expand All @@ -1671,7 +1796,7 @@ function check_update_db($debug, $tbpref, $dbname): void
WHEN STRCMP(TiText COLLATE utf8_bin,TiTextLC)!=0 OR TiWordCount=1
THEN TiText
ELSE ''
END as Text
END AS Text
FROM {$tbpref}textitems
LEFT JOIN {$tbpref}words ON TiTextLC=WoTextLC AND TiLgID=WoLgID
WHERE TiWordCount<2 OR WoID IS NOT NULL",
Expand Down

0 comments on commit c18e295

Please sign in to comment.