Is there a proposed place to put schemas-database stuff (schemas, for example)

[grechaw]

I'm interested in adding support for schemas and (in MarkLogic 9) extraction templates. These both just get stored in the schemas db, and just require a simple /v1/documents + database parameter to load.

[rjrudin]

What do you think of using "./data/schemas" as the default directory for ml-gradle to look for schemas? Also, how should permissions be specified for the schema documents?

[grechaw]

./data/schemas works for me.
As I understand it (just learned this today) permissions are ignored for looking up schemas, so it probably doesn't matter.

For ML9 work I'll have extraction templates too. And come to think of it, rulesets also belong in this category. These all three go to schemas db -- will we lump them all into ./data/schemas or support the hierarchy just for semantic differentiation.

./data/schemas
./data/rulesets
./data/extraction-templates

[rjrudin]

Okay - I think what I'll do is make it like src dirs, where you can have multiple paths, and within each path will be an expected structure. By default, ml-gradle will load whatever it finds under ./data, looking for at least schemas, rulesets, and extraction-templates (and perhaps "content" for documents that should go into the content database).

So the path of a schema document under ./data/schemas match how it's loaded into the schemas database? Example - ./data/schemas/path/to/schema.xsd would be loaded as /path/to/schema.xsd.

[grechaw]

I don't know whether the document uri matters for anything in the schemas database. I've always assumed a flat structure. Probably whatever is easiest is the thing to do.

BTW I'm willing to fork the project and do this work. If you give me pointers how to implement, esp.

[rjrudin]

Forking sounds good - I'd like to make a 2.0 of ml-gradle early next week, and I think this will be a good addition for 2.1, but we can always publish a snapshot before then.

ml-javaclient-util has a ModulesLoader interface for abstracting how modules are loaded into a modules database. I'm not sure what interfaces are needed here. I could see a DataLoader with a loadData(File dir) method, and perhaps it would use a DatabaseClient and port 8000 and /v1/documents PUT to load documents into a specific database? I think that would be easiest. So maybe "loadData(File dir, DatabaseClient client)" as the method so that it can reuse a DatabaseClient that's already been created.

I could see the need for a SchemasLoader, RulesetsLoader, etc in the future, which DataLoader would delegate to. But we can get to that later.

Perhaps this should be a separate project? Loading data isn't specific to ml-app-deployer. Similar to how ml-javaclient-util has "com.marklogic.client.modulesloader", I could see "com.marklogic.client.dataloader" as the root package for all this code. This could go into ml-javaclient-util then, since it's assumed this code will depend on a DatabaseClient. So maybe there instead of a separate project.

Assuming it goes into ml-javaclient-util, then ml-app-deployer would have a "LoadDataCommand". AppConfig would have a "List dataPaths", similar to "List modulePaths" (I'm not sure why I made that a List instead of a List, I must have had some reason). LoadDataCommand would then use AppConfig.newDatabaseClient() and iterate over the dataPaths, calling DataLoader.loadData(File dir, DatabaseClient client) for each path.

ml-gradle would then have "LoadDataTask", registered as "mlLoadData".

So overall, this would be very similar to loading modules.

[grechaw]

I have implemented much of this. Since I wanted to restrict the scope just to schemas database, I went for a finer-grained name LoadSchemasDataTask and mlLoadSchemasData.

I think your point about permissions for ad hoc data is a good one, and I don't want to bite off too much.

One question for you -- I've got ml-javaclientutil, ml-app-deployer, and ml-gradle all where I want them -- but now that I have something labeled as a SNAPSHOT version of ml-gradle, I can't figure out how to use that in a sample project. How do you use plugins from your local repo, or snapshot plugin code in general?

[rjrudin]

@grechaw - try out https://github.com/rjrudin/ml-gradle/blob/master/CONTRIBUTING.md#how-to-test-ml-gradle-changes . You should be able to run "publishToMavenLocal" on each of the projects, using any version number you want, and then use one of the examples under ml-gradle, or your own gradle project to try out the new plugin.

[rjrudin]

This is merged into dev, I have a couple more tickets I want to get in for 2.1.0, I'll try to do that by the end of the week