Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osmesa Code Migration #54

Merged
merged 13 commits into from
Jun 12, 2018
Merged

osmesa Code Migration #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
f18c4d4
Created Version.scala for the vectorpipe package
jbouffard May 23, 2018
09e7adb
Created Dependencies.scala for the vectorpipe package
jbouffard May 23, 2018
dc83bd4
Created assembly.sbt for the vectorpipe package
jbouffard May 23, 2018
297718f
Added plugins for vectorpipe
jbouffard May 23, 2018
51c9bca
Updated the build.sbt for vectorpipe
jbouffard May 23, 2018
8c5e449
Migrated the OSM building code from osmesa to vectorpipe
jbouffard May 23, 2018
2751655
Added derby.log and metastore_db to the gitignore
jbouffard May 30, 2018
f4f4ab2
Created the cmd subproject and moved ExtractMultiPolygons into it
jbouffard May 30, 2018
dd75be9
Added the sbt script
jbouffard May 31, 2018
9b3f0b2
Created the OSMReaderSpec
jbouffard May 31, 2018
13c9417
Bumped the version to 0.3.0
jbouffard Jun 1, 2018
268bc10
Removed the Caching object
jbouffard Jun 4, 2018
71d3c39
Refactored the FeatureRDD methods in OSMReader
jbouffard Jun 6, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,7 @@
.ensime
.ensime_cache/*
clipping/*
osmosis/*
osmosis/*

derby.log
metastore_db/*
93 changes: 80 additions & 13 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
name := "vectorpipe"

version := "0.2.2"

description := "Convert Vector data to VectorTiles with GeoTrellis."

organization := "com.azavea"
Expand All @@ -11,18 +9,53 @@ organizationName := "Azavea"
scalaVersion in ThisBuild := "2.11.12"

val common = Seq(
version := Version.vectorpipe,
resolvers ++= Seq(
"locationtech-releases" at "https://repo.locationtech.org/content/groups/releases",
Resolver.bintrayRepo("azavea", "maven")
),

initialCommands in console :=
"""
import org.apache.spark._
import org.apache.spark.sql._

import org.locationtech.geomesa.spark.jts._

import vectorpipe.osm._
import vectorpipe.osm.internal._

import java.net.URI

implicit val ss =
SparkSession
.builder()
.master("local[*]")
.appName("vectorpipe console")
.config("spark.ui.enabled", true)
.getOrCreate()
.withJTS
""".stripMargin,

scalacOptions := Seq(
"-deprecation",
"-Ypartial-unification",
"-Ywarn-value-discard",
"-Ywarn-unused-import",
"-Ywarn-dead-code",
"-Ywarn-numeric-widen"
"-Ywarn-numeric-widen",
"-language:implicitConversions",
"-language:reflectiveCalls"
),

resolvers ++= Seq(
Resolver.bintrayRepo("lonelyplanet", "maven"),
Resolver.bintrayRepo("bkirwi", "maven"), // Required for `decline` dependency
"locationtech-releases" at "https://repo.locationtech.org/content/repositories/releases/",
"locationtech-snapshots" at "https://repo.locationtech.org/content/repositories/snapshots/",
"geosolutions" at "http://maven.geo-solutions.it/",
"boundless" at "https://repo.boundlessgeo.com/main/",
"osgeo" at "http://download.osgeo.org/webdav/geotools/",
"apache.commons.io" at "https://mvnrepository.com/artifact/commons-io/commons-io"
),

scalacOptions in (Compile, doc) += "-groups",
Expand All @@ -31,18 +64,26 @@ val common = Seq(
addCompilerPlugin("org.scalamacros" %% "paradise" % "2.1.0" cross CrossVersion.full),

libraryDependencies ++= Seq(
"com.github.julien-truffaut" %% "monocle-core" % "1.5.0-cats-M2",
"com.github.julien-truffaut" %% "monocle-macro" % "1.5.0-cats-M2",
"org.locationtech.geotrellis" %% "geotrellis-vectortile" % Version.geotrellis exclude("com.google.protobuf", "protobuf-java"),
"org.locationtech.geotrellis" %% "geotrellis-s3" % Version.geotrellis exclude("com.google.protobuf", "protobuf-java"),
"org.locationtech.geomesa" %% "geomesa-spark-jts" % Version.geomesa,
"org.apache.spark" %% "spark-hive" % Version.spark % "provided",
"org.spire-math" %% "spire" % Version.spire,
"org.typelevel" %% "cats-core" % Version.cats,
"com.monovore" %% "decline" % Version.decline,
"org.scalatest" %% "scalatest" % Version.scalaTest % "test",
"com.github.julien-truffaut" %% "monocle-core" % Version.monocle,
"com.github.julien-truffaut" %% "monocle-macro" % Version.monocle,
"com.github.seratch" %% "awscala" % "0.6.1",
"org.apache.commons" % "commons-compress" % "1.16.1",
"com.google.protobuf" % "protobuf-java" % "2.5.0",
"io.dylemma" %% "xml-spac" % "0.3",
"org.apache.spark" %% "spark-hive" % "2.2.0" % "provided",
"org.locationtech.geotrellis" %% "geotrellis-spark" % "1.2.0",
"org.locationtech.geotrellis" %% "geotrellis-vectortile" % "1.2.0",
"org.scalatest" %% "scalatest" % "3.0.1" % "test",
"org.spire-math" %% "spire" % "0.13.0",
"org.typelevel" %% "cats-core" % "1.0.0-RC1"
"javax.media" % "jai_core" % "1.1.3" % "test" from "http://download.osgeo.org/webdav/geotools/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar",
),

parallelExecution in Test := false
parallelExecution in Test := false,
fork in Test := false,
testOptions in Test += Tests.Argument("-oDF")
)

val release = Seq(
Expand All @@ -52,6 +93,32 @@ val release = Seq(

lazy val lib = project.in(file(".")).settings(common, release)

assemblyShadeRules in assembly := {
val shadePackage = "com.azavea.shaded.demo"
Seq(
ShadeRule.rename("com.google.common.**" -> s"$shadePackage.google.common.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-cassandra" % Version.geotrellis).inAll,
ShadeRule.rename("io.netty.**" -> s"$shadePackage.io.netty.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-hbase" % Version.geotrellis).inAll,
ShadeRule.rename("com.fasterxml.jackson.**" -> s"$shadePackage.com.fasterxml.jackson.@1")
.inLibrary("com.networknt" % "json-schema-validator" % "0.1.7").inAll,
ShadeRule.rename("org.apache.avro.**" -> s"$shadePackage.org.apache.avro.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-spark" % Version.geotrellis).inAll
)
}

assemblyMergeStrategy in assembly := {
case s if s.startsWith("META-INF/services") => MergeStrategy.concat
case "reference.conf" | "application.conf" => MergeStrategy.concat
case "META-INF/MANIFEST.MF" | "META-INF\\MANIFEST.MF" => MergeStrategy.discard
case "META-INF/ECLIPSEF.RSA" | "META-INF/ECLIPSEF.SF" => MergeStrategy.discard
case "META-INF/ECLIPSE_.RSA" | "META-INF/ECLIPSE_.SF" => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("SF") => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("RSA") => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("DSA") => MergeStrategy.discard
case _ => MergeStrategy.first
}

/* Benchmarking suite.
* Benchmarks can be executed by first switching to the `bench` project and then by running:
jmh:run -t 1 -f 1 -wi 5 -i 5 .*Bench.*
Expand Down
11 changes: 11 additions & 0 deletions project/Version.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
object Version {
val vectorpipe = "0.3.0"
val geotrellis = "2.0.0-RC1"
val geomesa = "2.0.1"
val spark = "2.2.0"
val decline = "0.4.0-RC1"
val monocle = "1.5.0-cats-M2"
val cats = "1.0.0-RC1"
val spire = "0.13.0"
val scalaTest = "3.0.3"
}
1 change: 1 addition & 0 deletions project/assembly.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't recall ever needing this. It certainly wasn't necessary for publishing artifacts, at least.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been using assemblies to test this PR, but I'm not sure if removing this will impact anything.

4 changes: 4 additions & 0 deletions project/plugins.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@ addSbtPlugin("com.47deg" % "sbt-microsites" % "0.7.4")
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.2.27")

addSbtPlugin("org.foundweekends" % "sbt-bintray" % "0.5.1")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.0")

addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full)
Loading