Using Native Searchscript for River import

[MeiSign]

I am currently trying to run a Nativescript with the "script" property of the river configuration. The script is starting as expected but the problem is that doc() is always null and I can't access any fields of the document to run my document transformation.

Any idea how to solve this?

public class LambertToWgs84Script extends AbstractSearchScript {
  public static class Factory implements NativeScriptFactory {
    /**
     * This method is called for every search on every shard.
     *
     * @param params list of script parameters passed with the query
     * @return new native script
     */
    @Override
    public ExecutableScript newScript(@Nullable
                                      Map<String, Object> params) {
      return new LambertToWgs84Script(params);
    }
  }

  @Override
  public Object run() {
    // Get xy Values
    try {
      Long x = (Long)     doc().get("realEstateCommonData.locationDto.geoCoordinateDto.xCoordinate");
      Long y = (Long) doc().get("realEstateCommonData.locationDto.geoCoordinateDto.yCoordinate");
    } catch (NullPointerException e) {
      throw new ElasticSearchIllegalArgumentException(doc().toString());
    }

    // Transformation

    // Write Coordinates back
    JSONObject newLocation = new JSONObject();
    try {
      newLocation.put("Type", "Point");

      JSONArray coords = new JSONArray();
      coords.put("bla");
      coords.put("blub");
      newLocation.put("coordinates", coords);
      doc().put("location", newLocation);
    } catch (JSONException e) {
      e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
    }

    return true;
  }
}

[richardwilly98]

@MeiSign the river has not been tested with Nativescript yet.
So far js and groovy have been successfully tested.
Can you provide your river settings?

[MeiSign]

Here are my riversettings. Native script support in future would be awesome. :)

{
  "type": "mongodb",
  "mongodb": {
    "servers": [
      {
        "host": "***",
        "port": 27017
      }
    ],
    "db": "dev03",
    "collection": "expose",
    "script": "transformLambertToWgs84",
    "scriptType": "native"
  },
  "index": {
    "name": "mongo_expose",
    "bulk_size": 300
  }
}

[richardwilly98]

The key passed to map context is document and not doc. It looks like the native script assumes the document is available in doc map context...

Renaming document into doc is a breaking change unfortunately but that's probably the best way to fix this issue.

I am currently working on another feature to apply complex transformation to the document before to be stored in ES. It give the option to break down the document in multiple sub-documents. Unfortunately SearchLookup [1] does not seem to be able to support array of documents.

So I need to check what the best options before to fix this issue. Suggestions are welcome.

[1] - https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/search/lookup/SearchLookup.java

[MeiSign]

Sorry, I think my understanding of all this is currently not big enough to give a valid suggestion. :)
So if I understood you correct, I would have to change the map key in MongoDBRiver.java from document to doc to make it working?

[richardwilly98]

Yes that's my theory :-)

[MeiSign]

Ok, thank you, I will try that. Actually it might be a quickfix if you add the data in a second key doc. Than the map contains the data twice which is not nice but it wouldnt break the other script languages...

[richardwilly98]

Let me know if you can access the document with this change.

[MeiSign]

Unfortunately it doesn't. doc() still returns null. Maybe it work because native scripts are normally used during a searchquery and doc() refers to query results...

[richardwilly98]

@MeiSign I will try to take a look.

What is the reason you would like to use native script vs groovy or js?

[MeiSign]

Thank you very much.
We are currently migrating a lot of data from MongoDB to Elasticsearch and have already many Java Libraries which we can use to transform the data. If we could use native scripts we wouldn't have to port them all...

What i try now is run groovy from my java script to get and write the document...

[richardwilly98]

You should be able to re-use you existing java libraries from Groovy script.

See #87

There is a sample Groovy script using Joda [1].

[1] - ba6ae34

[MeiSign]

I actually tried that but I dont know how to use an external Jar file. I asked at Stackoverflow for help but the way how you can use normally external jars in groovy doesnt work.

import groovy.json.JsonSlurper

geo = new GeoTransformer()
geo.transform(ctx.document)

class GeoTransformer {
    void transform(doc) {
        this.getClass().classLoader.rootLoader.addURL(new File("gis-geotools-1.9.0.jar").toURL())
        def CoordinateTransformer = Class.forName("de.is24.gis.geotools.CoordinateTransformer").newInstance();

        def x = doc.realEstateCommonData.locationDto.geoCoordinateDto.xCoordinate;
        def y = doc.realEstateCommonData.locationDto.geoCoordinateDto.yCoordinate;

        def coords = CoordinateTransformer.transformFromLambertEuToWgs84(x,z)

        println coords.getLatitude()
        println coords.getLongitude()
        def jsonObj = new JsonSlurper().parseText( '{"type" : "Point", "coordinates" : [' + coords.getLatitude() + ',' + coords.getLongitude() + ']}' )

        doc.location = jsonObj       
    }
}

The rootLoader returns always null(leads to NullpointerException).
Some guys at stackoverflow think its probably a problem with how the groovy script is called...

[richardwilly98]

If I recall correctly you only need to have the jars in your classpath and call import from your Groovy script.

In my example with joda library I copied joda-time-2.2.jar in $ES_HOME/lib

[MeiSign]

Oh wow, this makes it so much easier! 👍
Thank you very much for all your help.

[benmccann]

Sounds like your problem was solved by using Groovy, so I'm going to close this