Resource files overridden by ones provided by default in SC

[jnioche]

We currently use [http://maven.apache.org/plugins/maven-assembly-plugin/], which is fine but apparently Shade would give us more control.
One reason why we'd need it is for instance : when an application uses storm crawler as a dependency, it gets the content src/main/resources (e.g. urlfilter config) in the uber-jar but the version coming from the dependency will override any files specified in the project with the same name. To put it differently, we need to use different names for the resources than the ones used by default unless we want to use the default files.
any thoughts?

[jakekdodd]

Yeah I think using the Shade plugin is the de facto standard for creating uber jars. I'll look into it

[jnioche]

Definitely an issue. Just got an uber jar where the default files from SC overrode the ones specified by my application. As a result all the custom parsing and filtering I had specified got ignored. Will change the title of this issue.

[jakekdodd]

Check out this commit and branch 7ac9a07

It renames the parsing/filtering resources to default-{resource name}, and then excludes the originals from the jar.

However, this doesn't resolve the issue when SC is included as a maven dependency in another project (because the shade plugin is in the bigjar profile, for now).

What do you think? We can proceed with using the shade plugin to rename the resources during the package phase, or simply name all of them default-* to begin with. I think both approaches accomplish the same thing, but renaming the files is easier.

[jnioche]

The issue is mainly when SC is included as a maven dependency in another project. Also we'd want to have the default files with their original names so that we can rely on them without changing the configuration if we wanted to.

What we just want to do really is to make sure that the local files systematically overwrite the default ones. Is that something we can do with Maven Shade?

[jakekdodd]

Hmm unfortunately I think that's only possible from the user's end.

Though isn't that the point of the parsefilters.config.file and the urlfilters.config.file configuration options? The user can override the defaults by including their own configuration files. If that weren't the case, then the defaults aren't really 'defaults'--they're just examples.

[GuiForget]

Specifying the file names in the yaml file is what we did in the shopstyle crawler.

[jnioche]

Hmm unfortunately I think that's only possible from the user's end.

I realize I had been inaccurate in my description of the issue. The assumption was that user builds an uber jar with the same mechanism as we do. I don't think many people will clone the SC code and add their modifications straight onto it but will rather use it as a dependency.

The user can override the defaults by including their own configuration files.

Which is what I did too. The point here is that if a ppl reuse the same resource names as the default, the behaviour of their crawler won't be what they expect. That's confusing.

[GuiForget]

But then... isn't the issue for the user when THEY build their uber jar?

[jnioche]

it is, that's exactly what I am on about. Not a core SC issue per-se but one I'd be happy to solve nonetheless, if not here then in spiderling.

[jakekdodd]

The point here is that if a ppl reuse the same resource names as the default, the behaviour of their crawler won't be what they expect. That's confusing.

Yeah I agree, one solution would be to rename the defaults by prepending large random strings using the shade plugin. So for example urfilters.json would be renamed to something like a473f5a2bcce41876d5f9a911c3ebe72-urlfilters.json. That seems silly, but it would work.

The assumption was that user builds an uber jar with the same mechanism as we do.
...
But then... isn't the issue for the user when THEY build their uber jar?

This is what I was thinking. If a user really wants to have their own default-* resources that override the packaged resources, they can use maven shade to rename the packaged resources.

There are a few ways we can solve this...prepending random strings to resources isn't totally bullet proof, but reduces the likelihood that a user would use a colliding resource name to basically 0.