Repair the update of database schema changes on postgreSQL #19707

csthomas · 2018-02-16T16:03:23Z

Pull Request for Issue # #14331

Summary of Changes

This PR is based on #17351, #18483 and is a replacement for #18483.

After PR, joomla recognises correctly:

SET NOT NULL;
SET DEFAULT '';
DROP DEFAULT; - does not work in [postgresql] - schema change rework #18483
DROP NOT NULL;
TYPE character varying(127) - new stuff - allow to resize column
... DROP DEFAULT, ... SET DEFAULT 0; - only test the last action after comma

Testing Instructions

[UPDATED]

Install joomla staging and go to backend to extension -> database.
Now you see an issue:

This issue should not happen because there is a fresh installation.

The column has a definition at

joomla-cms/installation/sql/postgresql/joomla.sql

Line 1868 in 73ca292

"data" text NOT NULL,

This means that the update query from

joomla-cms/administrator/components/com_admin/sql/updates/postgresql/3.1.0.sql

Line 3 in c8b2121

ALTER TABLE "#__updates" ALTER COLUMN "data" SET DEFAULT '';

is wrong and should be removed as I did in my PR.

After we click on the database fix, everything should be OK, but the query from:

joomla-cms/administrator/components/com_admin/sql/updates/postgresql/3.7.0-2017-03-03.sql

Line 3 in 73ca292

ALTER TABLE "#__updates" ALTER COLUMN "data" DROP DEFAULT;

is not visible by joomla (bug) and the query can not fix our database.

Now our database is different than the fresh from installation but joomla does not see any problems.

Apply this PR by com_patchtester
Go to extension -> database.
Now you see
After you click on the Fix button, the column data from #__updates will be repaired and will be equal to definition from

joomla-cms/installation/sql/postgresql/joomla.sql

Line 1868 in 73ca292

"data" text NOT NULL,
The next tests are to check if each query starts with ALTER TABLE ALTER COLUMN is visible by joomla. I prepared 7 queries:

ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" DROP NOT NULL;
ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" DROP DEFAULT;
ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" TYPE bigint USING "modified_by"::bigint;

ALTER TABLE "#__contact_details" ALTER COLUMN "con_position" SET DEFAULT '';
ALTER TABLE "#__contact_details" ALTER COLUMN "mobile" DROP DEFAULT,
                                 ALTER COLUMN "mobile" SET DEFAULT 'new value';
ALTER TABLE "#__contact_details" ALTER COLUMN "address" SET NOT NULL;
ALTER TABLE "#__contact_details" ALTER COLUMN "alias" TYPE character varying(10) USING substr("alias", 1, 10);

After you put it at the bottom of

joomla-cms/administrator/components/com_admin/sql/updates/postgresql/3.8.4-2018-01-16.sql

Lines 1 to 2 in 5bf0e5d

    
           DROP INDEX "#__user_keys_series_2"; 
        
           DROP INDEX "#__user_keys_series_3";

and refresh extension -> database page you should see a 7 issues:

This means that every query is recognised by joomla.

Then click on the Fix button.
Now everything is OK.

To revert all above changes, replace above code by:

ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" SET NOT NULL;
ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" SET DEFAULT 0;
ALTER TABLE "#__newsfeeds" ALTER COLUMN "modified_by" TYPE integer USING "modified_by"::integer;

ALTER TABLE "#__contact_details" ALTER COLUMN "con_position" DROP DEFAULT;
ALTER TABLE "#__contact_details" ALTER COLUMN "mobile" SET DEFAULT '';
ALTER TABLE "#__contact_details" ALTER COLUMN "address" DROP NOT NULL;
ALTER TABLE "#__contact_details" ALTER COLUMN "alias" TYPE character varying(255);

Then refresh extension -> database page we will see 7 issues:

this means that all queries have been recognised.
Now click on the Fix button. All s OK.

Remove all custom changes from administrator/components/com_admin/sql/updates/postgresql/3.8.4-2018-01-16.sql and refresh extension -> database. All is OK.

Expected result

All changes/issues are visible and can be fixed.

Actual result

Joomla does not recognise a few above queries or recognises them incorrectly.

alikon · 2018-02-17T10:12:24Z

i remember that DROP DEFAULT; was fixed on #18483, but i've no problem on closing that in favour of this one as your are adding new features, and with the hope this one have more luck than my old one 😆

trying to truncate a VARCHAR from 255 to 10 doesn't work
getting the ERROR: value too long for type character varying(10)

ALTER TABLE "#__contact_details" ALTER COLUMN "alias" TYPE character varying(10);

you should issue/manage something like this instead

ALTER TABLE "#__contact_details" ALTER COLUMN "alias" TYPE varchar(10)
USING substr("alias", 1, 10)

csthomas · 2018-02-19T22:29:19Z

Thanks,

ALTER TABLE "#__contact_details" ALTER COLUMN "alias" TYPE varchar(10) USING substr("alias", 1, 10)

I fixed it and now you can use function with parameters after word USING

alikon · 2018-02-21T07:07:15Z

after the first fix, still get database issues

csthomas · 2018-02-21T15:14:48Z

after the first fix, still get database issues

I have not explained that before.
This is an expected behaviour (for above columns) because of earlier sql queries.

This was a reason I had to comment line in administrator/components/com_admin/sql/updates/postgresql/3.1.0.sql

because it is conflicted with query from
administrator/components/com_admin/sql/updates/postgresql/3.7.0-2017-03-03.sql

Example:
The first update file contains "... column X SET DEFAULT ''"
The second update file redefines column X and change default value to NULL.
This two changes has a conflict and joomla show first or second issue.

The result is, the problem won't be fixed because the database table won't match to the query from the first file or to the query from the second file.

In my example, which you tested, I redefined a few columns and then joomla show changed column that does not match to queries from previous update files.

If this is a problem I can find a better columns (which does not exists in update files before).

alikon · 2018-02-21T19:32:37Z

i was 😪
😄

alikon · 2018-02-21T19:33:35Z

I have tested this item ✅ successfully on 7001e67

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19707.}

brianteeman · 2018-02-23T13:43:33Z

I finally got pgsql installed on my windows machine

Not sure if my test is successful or not as I still have a few issues to fix after applying the patch

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19707.}

csthomas · 2018-02-23T13:58:11Z

Nice,

after you applied the PR, joomla starts to show you a database problems (old problems that have never been recognized before).

Before you start to use my above test examples you can click on fix button and all issues should disappear.
Today or tomorrow I will try to add easier text instructions.

csthomas · 2018-02-26T23:00:18Z

I updated the test instructions.

csthomas · 2018-03-01T07:43:53Z

As PR #17351 for the mysql version has been merged it would be good to take a look at this version for postgreSQL.

csthomas · 2018-04-17T20:43:57Z

Can we merge this after only one success test?

zero-24 · 2018-05-16T20:53:41Z

administrator/components/com_admin/sql/updates/postgresql/3.1.0.sql

 /* Changes to tables where data type conflicts exist with MySQL (mainly dealing with null values */
 ALTER TABLE "#__modules" ALTER COLUMN "content" SET DEFAULT '';
-ALTER TABLE "#__updates" ALTER COLUMN "data" SET DEFAULT '';
+--ALTER TABLE "#__updates" ALTER COLUMN "data" SET DEFAULT '';


Should't we remove this line? Maybe I just don't get it?

We can remove it, but I thought it would be better to leave it because of history, e.g.
https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_admin/sql/updates/mysql/3.4.0-2014-09-16.sql

can we add such a comment than too?

Thanks 👍

csthomas · 2018-05-17T20:52:32Z

I rebased it and I added a comment only.

wilsonge · 2018-08-10T08:40:01Z

Given the low number of postgres testers I'm going to merge this with the one good test

Migrate the change from PR joomla#19707 for ALTER TABLE statements with multiple actions to check the last change instead of failing from PostgreSQL to MySQL.

joomla-cms-bot added the PR-staging label Feb 16, 2018

csthomas mentioned this pull request Feb 16, 2018

Session table improvements #19708

Closed

alikon mentioned this pull request Feb 21, 2018

[postgresql] - schema change rework #18483

Closed

alikon mentioned this pull request Mar 6, 2018

[4.0] Modules can't be created #19836

Merged

alikon approved these changes Apr 18, 2018

View reviewed changes

alikon mentioned this pull request Apr 21, 2018

[4.0] Removing artificial sharding from com_finder #20185

Merged

alikon mentioned this pull request May 9, 2018

Fix sqlsrv syntax for query SELECT with DISTINCT and LIMIT joomla-framework/database#104

Merged

mbabker approved these changes May 10, 2018

View reviewed changes

alikon mentioned this pull request May 11, 2018

Functional testing for Joomla\CMS\Schema library #20332

Closed

zero-24 reviewed May 16, 2018

View reviewed changes

Tomasz Narloch added 4 commits May 17, 2018 22:43

Repair the update of database schema changes on postgreSQL

2df1ea4

Add a better regex to split actions by comma

8f5cc76

Remove useless code

143e069

Add a comment to sql file

daed6e3

csthomas force-pushed the db_fix_postgresql_button branch from 7001e67 to daed6e3 Compare May 17, 2018 20:51

csthomas mentioned this pull request Jun 18, 2018

[4.0] missing field 'custom-data' in extensions table #19348

Closed

csthomas mentioned this pull request Jun 27, 2018

[4.0] Sql fix - schema parser expected format #20889

Merged

alikon mentioned this pull request Aug 10, 2018

[WIP][4.0] - workwflow mysql schema #21499

Merged

wilsonge merged commit bf199b0 into joomla:staging Aug 10, 2018

wilsonge added this to the Joomla 3.8.12 milestone Aug 10, 2018

csthomas deleted the db_fix_postgresql_button branch August 10, 2018 17:28

csthomas mentioned this pull request Aug 30, 2018

[4.0] Replace the zero datetime value '0000-00-00 00:00:00' by the real NULL value #21901

Merged

	DROP INDEX "#__user_keys_series_2";
	DROP INDEX "#__user_keys_series_3";

Uh oh!

Repair the update of database schema changes on postgreSQL #19707

Repair the update of database schema changes on postgreSQL #19707

Uh oh!

Conversation

csthomas commented Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Testing Instructions

Expected result

Actual result

Uh oh!

alikon commented Feb 17, 2018

Uh oh!

csthomas commented Feb 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alikon commented Feb 21, 2018

Uh oh!

csthomas commented Feb 21, 2018

Uh oh!

alikon commented Feb 21, 2018

Uh oh!

alikon commented Feb 21, 2018

Uh oh!

brianteeman commented Feb 23, 2018

Uh oh!

csthomas commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

csthomas commented Feb 26, 2018

Uh oh!

csthomas commented Mar 1, 2018

Uh oh!

csthomas commented Apr 17, 2018

Uh oh!

zero-24 May 16, 2018

Choose a reason for hiding this comment

Uh oh!

csthomas May 17, 2018

Choose a reason for hiding this comment

Uh oh!

zero-24 May 17, 2018

Choose a reason for hiding this comment

Uh oh!

csthomas May 17, 2018

Choose a reason for hiding this comment

Uh oh!

zero-24 May 17, 2018

Choose a reason for hiding this comment

Uh oh!

csthomas commented May 17, 2018

Uh oh!

wilsonge commented Aug 10, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

csthomas commented Feb 16, 2018 •

edited

Loading

csthomas commented Feb 19, 2018 •

edited

Loading

csthomas commented Feb 23, 2018 •

edited

Loading