[4.0] database importer exporter (rebase)

[twister65]

Rebase Pull Request #23320.
This PR aims to back up and restore the Joomla database using export and import commands (joomla / console).

Summary of Changes

See PR #23320, PR #14272 (and PR #10991).

Testing Instructions

Before:

• Intall the last build (from May 6, 2019) of Joomla 4:
  https://developer.joomla.org/nightly-builds.html

Then:

• Go to the cli folder of your website.
• Export all tables and data to the folder :
  php joomla.php database:export --all --folder <folder_path>
• Import all from folder:
  php joomla.php database:import --all --folder <folder_path>

You can also :

• Export all tables and data in zip file :
  php joomla.php database:export --all --zip
• Export a table :
  php joomla.php database:export --table <table_name>
• Export a table as a .zip file :
  php joomla.php database:export --table <table_name> --zip
• Import a table :
  php joomla.php database:import --table <table_name>

If you need help:
php joomla.php database:export --help
php joomla.php database:import --help

Expected result

Export/import tables with structure and data.

Actual result

Does not export/import views, stored procedures and triggers (optional for the Joomla database).

Documentation Changes Required

[mbabker]

These CLI files really should make use of the joomla/console package, and there's really no reason they couldn't be included in the Framework package instead of only in the CMS (though it might take a couple iterations to switch it from single file deprecated CLI things to proper commands but it'd be worth it in the end).

[twister65]

Is there a documentation to implement joomla/console ?
The only one I found is:
https://github.com/joomla-framework/console/blob/master/docs/overview.md

If I execute:
https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f
I have:

PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not found in /home/olivier/src/database/Console/ExportApplication.php:11

[mbabker]

Do you have a code branch somewhere? You should only need command classes, not an application class. Check the Framework’s Event, Keychain, Router, and Session packages for example commands. The CMS repo has some commands as well. And check the framework.joomla.org repo for a full application example.

On Sun, Apr 28, 2019 at 10:25 AM Olivier ***@***.***> wrote: Is there a documentation to implement joomla/console ? The only one I found is: https://github.com/joomla-framework/console/blob/master/docs/overview.md If I execute: https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f I have: PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not found in /home/olivier/src/database/Console/ExportApplication.php:11 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24733 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACZ7IL2SWSNW25EJ65TBNLPSW6W3ANCNFSM4HI2J7SQ> .

-- - Michael Please pardon any errors, this message was sent from my iPhone.

[twister65]

@mbabker , the code branch is:
https://github.com/twister65/database/tree/2.0-exim
I add command classes in this commit:
twister65/database@c888037

I failed to test these commands with the joomla/console/application.

PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not found in /home/olivier/src/database_test/src/ExporterApplication.php:11

See https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f

We must also read the configuration file to configure the database interface.

[mbabker]

For the class not found bit, you'll need to add the relevant packages to the require-dev section of the composer.json file, you should also add them to the suggest section. See joomla-framework/event@e4c851e#diff-b5d0ee8c97c7abd7e3fa29b9a27d1780 for the Event package as an example. Make sure you composer update after adding them otherwise the files won't be in the vendor directory (if you don't have a vendor/joomla/console/src/Application.php file then something's not right).

We must also read the configuration file to configure the database interface.

For the Database package, what you've got in there is just fine for now. When integrated into the CMS, the service layer would have to take care of loading in the correct database connection when the command is run. We can worry about that later.

I'll do a more thorough look at your branch later.

[twister65]

@mbabker , I put the console application in the cli directory. Is this the right place ?
I also updated the testing instructions to use console commands.

[mbabker]

Is this the right place ?

Not quite. Instead of adding a new application, you'd register the commands with the existing application (part of the point of the console package is to make it so you can build commands usable in any compatible application, whereas the old concept of JApplicationCli basically meant every console command was its own application).

So for the CMS integration, you're really doing 2 things:

1. Registering services for the commands.

In libraries/src/Service/Provider/Console.php register the two database commands as services:

$container->share(
	ExportCommand::class,
	function (Container $container)
	{
		return new ExportCommand($container->get('db'));
	},
	true
);

$container->share(
	ImportCommand::class,
	function (Container $container)
	{
		return new ImportCommand($container->get('db'));
	},
	true
);

2. Registering the commands to the core CLI application.

In libraries/src/Service/Provider/Application.php, add the two commands to the mapping array of the WritableContainerLoader service (this allows the command classes to be lazily instantiated through the container only when they're needed). Then when running php cli/joomla.php you should see the two commands in the list of available commands (this won't work fully until the PR to the Framework repo is merged and a composer update is run on this repo with the additions).

[mbabker]

You didn't need that last commit. For the CMS integration, the only changes this PR needs is the changes to the files in libraries/src/Service/Provider as that will handle registering the commands from the Framework's Database package/repository to the CMS application. You don't need to add the command classes to the CMS and you don't need separate files in the cli directory.

The only thing this PR is dependent upon once it gets in that state is the pull request in the Framework repository to be merged then the CMS pulls in the upstream changes.

The only reason to add the command classes to the CMS repo directly is if the CMS 100% cannot use the classes in the Framework package, which from everything I've seen that is not the case.

[twister65]

It is ready for testing with Joomla/Console. See the testing instructions.
@mbabker , thanks.

[mbabker]

Upstream PR's merged, this repo just needs the corresponding composer update then this is ready to merge.

[alikon]

to be honest i didn't test all the options .... sorry i'm lazy ...
but for what i've tested (mostly postgressql)..... please simply merge it

[alikon]

I have tested this item ✅ successfully on 8fa6edf

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24733.}

[mbabker]

to be honest i didn't test all the options .... sorry i'm lazy ...
but for what i've tested (mostly postgressql)..... please simply merge it

FWIW I added some functional tests for both the export command and import command on a MySQL build, so between the tests for the import/export class chains and the command tests we should be covered on all the functionality now.

Since all this PR is doing is adding the integration for the CMS' console application, this should be considered RTC as any bugs in either the import or export code, or the CLI commands, are going to be upstream bugs and not a blocker for this PR. The branch just needs to have 4.0-dev merged in again to pick up the autoloader fixes that broke the tests.

[alikon]

your statements are gold to my ears
i don't have the power to merge it but at least i can mark it as rtc

[alikon]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24733.}

[twister65]

Try to import in a new blank database (without table). The joomla/console script fails, unlike the previous cli script or Joomla/Console Application.
Question: is this a problem in our context ?

[mbabker]

Define "fails".

[twister65]

Define "fails":

Whoops, looks like something went wrong.

• Export all tables and data to the folder :
  php joomla.php database:export --all --folder <folder_path>
• Delete and create an empty database with the same name.
• Import all from folder:
  php joomla.php database:import --all --folder <folder_path>

[mbabker]

Put Joomla in debug mode and see what happens. IIRC I set up the Symfony debug handler is configured to show the exception details and backtrace when debug's enabled (but the Console application is also supposed to be catching and processing Exceptions within it so I have no idea why that handler is actually being triggered).

[twister65]

Same result with debugging enabled. I have also defined the session handler on filesystem (previously defined on database). Without effect ...

[mbabker]

Well, without an Exception to debug things aren't going to go too far. You might need to find one of the catch blocks in the Joomla\Console\Application class and dump the Exception there to see what's going on.

Though if I had to blindly take a guess, given your "empty database" statement, it's probably an Exception being thrown from the drop table call in the import command, but that would then mean it's not a ExecutionFailureException being thrown because the command is catching and handling that.

[twister65]

With DatabaseConsole, it works fine.
3bec551
It uses the same commands.

[mbabker]

All that means is that the command executes correctly in an isolated context. That doesn't identify if there is an integration issue with the Console package and the CMS or some quirk in the import code that you don't get from a standalone version that has no outside influences. That DatabaseConsole file can't be merged because the intent is all CLI things go through cli/joomla.php and the CMS' console application, there should not be the mess of random CLI files to get executed in 4.0.

[twister65]

ERROR: relation "vwe2g_viewlevels" does not exist
debug.txt

[mbabker]

That'd be expected then. Unfortunately a chunk of the CMS outside the install app just requires some of the database to be there, I'm not sure you'd be able to make the import command work on a completely empty database (maybe some of the work in #21452 helps with this scenario).

[roland-d]

Thank you

@twister65 Would you mind creating a page on the https://docs.joomla.org site documenting this feature?

[twister65]

Of course, I created an account and confirmed my email. But I can't edit a new page:

Permission error
You do not have permission to create pages, for the following reason below:
Editing of this page is limited to registered doc users in the group: Email Confirmed.

Maybe someone else can do it, click on the link below:
https://docs.joomla.org/CLI_Database_Exporter_Importer
And copy/paste this text:
JDoc_CLI_Database.txt

[alikon]

@twister65 docs page done can you double check please https://docs.joomla.org/CLI_Database_Exporter_Importer