jdatabase importer exporter by twister65 · Pull Request #23320 · joomla/joomla-cms

twister65 · 2018-12-22T18:14:03Z

Retrieving PR #14272 (and #10991) to fix the PostgreSQL exporter and importer bugs.
This PR can be tested with any database driver supported by Joomla.
This is about exporting and importing any database with a command-line interface (it can be placed in cronjob to schedule the database backup).

Summary of Changes

Fix folder path.
Fix postgresql issues :
- Fix the default value of the table.
- Gets, sets the last_value and is_called sequence attributes.
- Fix key query.
- Fix text default value.
- Apply changes to unit tests.
- Fix the execution of queries (xmlToCreate) with the pgsql (PDO) driver.
- Add unique constraints.
- Convert PostgreSQL BLOB data (bytea).

Testing Instructions

Select a database driver and save it in System -> Global Configuration -> Server ->Database Type
Go to the cli folder of your website.
Export all tables and data to the folder :
php exporter.php --all --folder <folder_path>
Delete and create an empty database with the same name.
Overwriting the database can also be done. Verify that changes are applied.
Import all from folder:
php importer.php --all --folder <folder_path>

You can also :

Export all tables and data in zip file :
php exporter.php --all --mode zip
Export a table :
php exporter.php --table <table_name>
Export a table as a .zip file :
php exporter.php --table <table_name> --mode zip
Import a table :
php importer.php --table <table_name>

Expected result

Export/import tables with structure (keys, constraints, sequences) and data.

Actual result

Does not export/import views, stored procedures and triggers (optional for the Joomla database).

libraries/joomla/database/importer/postgresql.php

libraries/joomla/database/driver/postgresql.php

Fix test default field

Add last_value and is_called fields to the sequence test.

Get sequence last_value and is_called

getSequenceLastValue getSequenceIsCalled Rename test table.

twister65 · 2019-01-01T04:19:17Z

PostgreSQL automatically creates a unique index when a unique constraint or primary key is defined for a table. We need to modify table by adding a unique constraint on a column, instead of creating a unique index.
This query creates unique index:

joomla-cms/libraries/joomla/database/importer/postgresql.php

Line 152 in be51ebe

return (string) $field['Query'];

Here, a unique constraint is defined for a table:

joomla-cms/libraries/joomla/database/importer/mysqli.php

Line 114 in be51ebe

    
           return 'ALTER TABLE ' . $this->db->quoteName($table) . ' ADD ' . $this->getKeySql($keys);

This creates a unique index.

libraries/joomla/database/exporter/postgresql.php

Co-Authored-By: twister65 <olaverre@free.fr>

table_structure name

SharkyKZ · 2019-03-04T21:04:53Z

libraries/joomla/database/exporter/postgresql.php

 		foreach ($this->from as $table)
 		{
-			// Replace the magic prefix if found.
-			$table = $this->getGenericTableName($table);


This needs to be restored. The tables should be exported with magic prefix, I think. At least MySQLi does it.

Completed. Do you also want a magic prefix for the sequence name and key index?

See lines:

joomla-cms/tests/unit/suites/database/driver/postgresql/JDatabaseExporterPostgresqlTest.php

Line 224 in 3b0c183

<sequence Name="jos_dbtest_id_seq" Schema="public" Table="jos_dbtest" Column="id" Type="bigint" Start_Value="' .

and

joomla-cms/tests/unit/suites/database/driver/postgresql/JDatabaseExporterPostgresqlTest.php

Line 230 in 3b0c183

<key Index="jos_dbtest_pkey" is_primary="TRUE" is_unique="TRUE" Key_name="id" Query=\'ALTER TABLE "jos_dbtest" ADD PRIMARY KEY (id)\' />

I think so. To my understanding, exported XML files should not contain real DB prefix. It should be replaced when importing. Maybe @alikon knows more.

Done. However, the key query can't contain the magic prefix.

To my understanding, exported XML files should not contain real DB prefix. It should be replaced when importing.

correct

In that case more changes are needed because Query and Default attributes contain it, e.g.:

<field Field="id" Type="integer" Null="NO" Default="nextval('pg39_action_logs_id_seq'::regclass)" Comments="" />

and

<key Index="#__action_logs_idx_user_id_extension" is_primary="f" is_unique="f" Key_name="user_id, extension" Query='CREATE INDEX pg39_action_logs_idx_user_id_extension ON pg39_action_logs USING btree (user_id, extension)' />

libraries/joomla/database/exporter/postgresql.php

107 | WARNING | Line exceeds 150 characters; contains 168 characters

alikon · 2019-03-11T18:20:52Z

not so sure if this work can really have a chance to be merged on 3.X branch, i will hope so
i would like to hear some words from @joomla/maintainers

ghost · 2019-04-24T10:09:35Z

@HLeithner comment on above Question by @alikon please?

HLeithner · 2019-04-25T13:27:07Z

I'm sorry but this is a new feature and has to go into j4

ghost · 2019-04-25T13:42:40Z

@twister65 can you rebase this PR on J4?

twister65 · 2019-04-25T14:25:24Z

Yes, I can rebase this PR in the Joomla Framework Database Package:
https://github.com/joomla-framework/database

Rebase PR joomla#23320

ghost · 2019-04-27T04:41:14Z

Closed as rebased on J4.

* Database exporter importer Rebase PR #23320 * Classname * Add Database Console Application * Register Export and Import Commands * Add commands paths * Use Joomla Console php cli/joomla.php * Database Joomla/Console Application To import in a new database * Remove DatabaseConsole * Sort in alpha order.

twister65 added 2 commits December 15, 2018 22:02

PostgreSQL fix for PR joomla#14272

a2efcff

Sets and increments the last value of a sequence.

24124bc

joomla-cms-bot added the PR-staging label Dec 22, 2018

Quy reviewed Dec 23, 2018

View reviewed changes

libraries/joomla/database/importer/postgresql.php Outdated Show resolved Hide resolved

Quy reviewed Dec 23, 2018

View reviewed changes

libraries/joomla/database/importer/postgresql.php Outdated Show resolved Hide resolved

Quy reviewed Dec 23, 2018

View reviewed changes

libraries/joomla/database/driver/postgresql.php Outdated Show resolved Hide resolved

twister65 added 2 commits December 23, 2018 11:11

Add setval function

7ccb9d3

Merge with PR joomla#14272

d1ee594

joomla-cms-bot added the Unit/System Tests label Dec 23, 2018

twister65 changed the title ~~Fix PostgreSQL exporter / importer~~ [jdatabase] - importer exporter (PostgreSQL fix) Dec 23, 2018

twister65 added 2 commits December 23, 2018 17:30

Fix key query

885f48c

Fix folder path

d746469

Quy reviewed Dec 23, 2018

View reviewed changes

libraries/joomla/database/driver/postgresql.php Outdated Show resolved Hide resolved

twister65 added 14 commits December 23, 2018 22:43

Add sequence last_value accessors

dca3914

Space review

7f5ab7e

Fix text default value

be7e696

Get/Set the "is_called" sequence attribute

9d7184a

Update postgresql.php

37f62a3

Update JDatabaseExporterPostgresqlTest.php

3605ad5

Fix test default field

Update JDatabaseExporterPostgresqlTest.php

f76e623

Add last_value and is_called fields to the sequence test.

Update JDatabaseExporterPostgresqlTest.php

d49df74

Get sequence last_value and is_called

Add methods to the mock

9ff132c

getSequenceLastValue getSequenceIsCalled Rename test table.

Apply the changes to the class JDatabaseImporterPostgresqlTest

1514ec6

pgsql(pdo) can only execute one query at a time.

e30574f

tabulations.

57f2c61

tabulations.

7ef0b2c

space.

be51ebe

twister65 added 2 commits January 2, 2019 16:52

Add a unique constraint on a column

2212c5e

This creates a unique index.

Add the missing Key_name to the expected result.

902abe2

SharkyKZ reviewed Feb 27, 2019

View reviewed changes

libraries/joomla/database/exporter/postgresql.php Outdated Show resolved Hide resolved

SharkyKZ reviewed Feb 27, 2019

View reviewed changes

libraries/joomla/database/exporter/postgresql.php Outdated Show resolved Hide resolved

SharkyKZ and others added 3 commits February 27, 2019 20:03

Apply suggestions from code review

2676185

Co-Authored-By: twister65 <olaverre@free.fr>

No magic prefix for table here

c87dbcf

Replace the magic prefix

16b235e

table_structure name

SharkyKZ reviewed Mar 4, 2019

View reviewed changes

libraries/joomla/database/exporter/postgresql.php Outdated Show resolved Hide resolved

twister65 added 4 commits March 5, 2019 20:37

Add magic prefix to table_structure name

3b0c183

Replace prefix (pgsql)

0138495

Replace sequence and key names with magic prefix.

324c799

Fix drone warning.

729cb0d

107 | WARNING | Line exceeds 150 characters; contains 168 characters

ghost added J3 Issue and removed J3 Issue labels Apr 5, 2019

alikon mentioned this pull request Apr 10, 2019

JDatabaseImporterPdomysql missing xmlToCreate() #10527

Closed

ghost changed the title ~~[jdatabase] - importer exporter (#14272 recovery)~~ [jdatabase] - importer exporter Apr 19, 2019

ghost changed the title ~~[jdatabase] - importer exporter~~ jdatabase importer exporter Apr 19, 2019

ghost removed the J3 Issue label Apr 19, 2019

ghost added the J4 Rebase label Apr 25, 2019

twister65 added a commit to twister65/joomla-cms that referenced this pull request Apr 25, 2019

Database exporter importer

9672638

Rebase PR joomla#23320

twister65 mentioned this pull request Apr 26, 2019

[4.0] database importer exporter (rebase) #24733

Merged

ghost closed this Apr 27, 2019

ghost removed the J4 Rebase label Apr 27, 2019

twister65 deleted the pgsql_transfer branch July 28, 2019 13:52

This pull request was closed.

Uh oh!

Conversation

twister65 commented Dec 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Testing Instructions

Expected result

Actual result

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twister65 commented Jan 1, 2019

Uh oh!

Uh oh!

Uh oh!

SharkyKZ Mar 4, 2019

Choose a reason for hiding this comment

Uh oh!

twister65 Mar 5, 2019

Choose a reason for hiding this comment

Uh oh!

twister65 Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

SharkyKZ Mar 7, 2019

Choose a reason for hiding this comment

Uh oh!

twister65 Mar 10, 2019

Choose a reason for hiding this comment

Uh oh!

alikon Mar 11, 2019

Choose a reason for hiding this comment

Uh oh!

SharkyKZ Mar 11, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alikon commented Mar 11, 2019

Uh oh!

ghost commented Apr 24, 2019

Uh oh!

HLeithner commented Apr 25, 2019

Uh oh!

ghost commented Apr 25, 2019

Uh oh!

twister65 commented Apr 25, 2019

Uh oh!

ghost commented Apr 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

twister65 commented Dec 22, 2018 •

edited

Loading