Skip to content

Commit

Permalink
Merge pull request #653 from ibuenros/v0.6.2
Browse files Browse the repository at this point in the history
V0.6.2
  • Loading branch information
ibuenros committed Feb 2, 2016
2 parents feec99e + 116c79b commit ffd0aa9
Show file tree
Hide file tree
Showing 3 changed files with 133 additions and 81 deletions.
52 changes: 51 additions & 1 deletion CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,6 +1,56 @@
GOBBLIN 0.6.1
GOBBLIN 0.6.2
=============

## NEW FEATURES
* [Admin Dashboard] Added a web based GUI for exploring running and finished jobs in a running Gobblin daemon (thanks Eric Ogren).
* [Admin Dashboard] Added a CLI for finding jobs in the job history store and seeing their run details (thanks Eric Ogren).
* [Configuration Management] WIP: Configuration management library. Will enable Gobblin to be dataset aware, ie. to dynamically load and apply different configurations to each dataset in a single Gobblin job.
** APIs: APIs for configuration stores and configuration client.
** Configuration Library: loads low level configurations from a configuration store, resolves configuration dependencies / imports, and performs value interpolation.
* [Distcp] Allow using *.ready files as markers for files that should be copied, and deletion of *.ready files once the file has been copied.
* [Distcp] Added file filters to recursive copyable dataset for distcp. Allows to only copy files satisfying a filter under a base directory.
* [Distcp] Copied files that fail to be published are persisted for future runs. Future runs can recover the already copied file instead of re-doing the byte transfer.
* [JDBC] Can use password encryption for JDBC sources.
* [YARN] Added email notifications on YARN application shutdown.
* [YARN] Added event notifications on YARN container status changes.
* [Metrics] Added metric filters based on name and type of the metrics.
* [Dataset Management] POC embedded sql for config-driven retention management.
* [Exactly Once] POC for Gobblin managed exactly once semantics on publisher.

## BUG FIXES
* **Core** File based source includes previously failed WorkUnits event if there are no new files in the source (thanks Joel Baranick).
* **Core** Ensure that output file list does not contain duplicates due to task retries (thanks Joel Baranick).
* **Core** Fix NPE in CliOptions.
* **Core/YARN** Limit Props -> Typesafe Config conversion to a few keys to prevent overwriting of certain properties.
* **Utility** Fixed writer mkdirs for S3.
* **Metrics** Made Scheduled Reporter threads into daemon threads to prevent hanging application.
* **Metrics** Fixed enqueuing of events on event reporters that was causing job failure if event frequency was too high.
* **Build** Fix POM dependencies on gobblin-rest-api.
* **Build** Added conjars and cloudera repository to all projects (fixes builds for certain users).
* **Build** Fix the distribution tarball creation (thanks Joel Baranick).
* **Build** Added option to exclude Hadoop and Hive jars from distribution tarball.
* **Build** Removed log4j.properties from runtime resources.
* **Compaction** Fixed main class in compaction manifest file (thanks Lorand Bendig).
* **JDBC** Correctly close JDBC connections.

## IMPROVEMENTS
* [Build] Add support for publishing libraries to maven local (thanks Joel Baranick).
* [Build] In preparation to Gradle 2 migration, added ext. prefix to custom gradle properties.
* [Build] Can generate project dependencies graph in dot format.
* [Metrics] Migrated Kafka reporter and Output stream reporter to Root Metrics Reporter managed reporting.
* [Metrics] The last metric emission in the application has a "final" tag for easier Hive identification.
* [Metrics] Metrics for Gobblin on YARN include cluster tags.
* [Hive] Upgraded Hive to version 1.0.1.
* [Distcp] Add file size to distcp success notifications.
* [Distcp] Each work unit in distcp contains exactly one Copyable File.
* [Distcp] Copy source can set upstream timestamps for SLA events emitted on publish time.
* [Scheduling] Added Gobblin Oozie config files.
* [Documentation] Improved javadocs.


GOBBLIN 0.6.1
-------------

## BUG FIXES

- **Build/release** Adding build instrumentation for generation of rest-api-* artifacts
Expand Down
146 changes: 74 additions & 72 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -256,25 +256,93 @@ if (ext.javadocVersion.indexOf('-') > 0) {
}

ext.javadocPackages = new HashSet<String>()
subprojects.each{Project pr ->
subprojects.each{Project pr ->
if (file(pr.projectDir.absolutePath + "/src/main/java").exists()) {
rootProject.ext.javadocPackages += pr.name
}
}

subprojects {
plugins.withType(JavaPlugin) {

// Sometimes generating javadocs can lead to OOM. This may needs to be increased.
// Also force javadocs to pick up system proxy settings if available
javadoc {
options.jFlags('-Xmx256m', '-Djava.net.useSystemProxies=true');
}

rootProject.tasks.javadocTarball.dependsOn project.tasks.javadoc
if ( rootProject.ext.javadocPackages.contains(project.name)) {
rootProject.tasks.javadocTarball.into(project.name){from(fileTree(dir: "${project.buildDir}/docs/javadoc/"))}
}
}
}
}

ext.pomAttributes = {
name "${project.name}"
packaging 'jar'
// optionally artifactId can be defined here
description 'Gobblin Ingestion Framework'
url 'https://github.com/linkedin/gobblin/'

scm {
connection 'scm:git:[email protected]:linkedin/gobblin.git'
developerConnection 'scm:git:[email protected]:linkedin/gobblin.git'
url '[email protected]:linkedin/gobblin.git'
}

licenses {
license {
name 'The Apache License, Version 2.0'
url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}

developers {
developer {
name 'Abhishek Tiwari'
organization 'LinkedIn'
}
developer {
name 'Chavdar Botev'
organization 'LinkedIn'
}
developer {
name 'Issac Buenrostro'
organization 'LinkedIn'
}
developer {
name 'Min Tu'
organization 'LinkedIn'
}
developer {
name 'Narasimha Veeramreddy'
organization 'LinkedIn'
}
developer {
name 'Pradhan Cadabam'
organization 'LinkedIn'
}
developer {
name 'Sahil Takiar'
organization 'LinkedIn'
}
developer {
name 'Shirshanka Das'
organization 'LinkedIn'
}
developer {
name 'Yinan Li'
organization 'LinkedIn'
}
developer {
name 'Ying Dai'
organization 'LinkedIn'
}
developer {
name 'Ziyang Liu'
organization 'LinkedIn'
}
}
}

Expand Down Expand Up @@ -380,73 +448,7 @@ subprojects {
authentication(userName: ossrhUsername, password: ossrhPassword)
}

pom.project {
name "${project.name}"
packaging 'jar'
// optionally artifactId can be defined here
description 'Gobblin Ingestion Framework'
url 'https://github.com/linkedin/gobblin/'

scm {
connection 'scm:git:[email protected]:linkedin/gobblin.git'
developerConnection 'scm:git:[email protected]:linkedin/gobblin.git'
url '[email protected]:linkedin/gobblin.git'
}

licenses {
license {
name 'The Apache License, Version 2.0'
url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}

developers {
developer {
name 'Abhishek Tiwari'
organization 'LinkedIn'
}
developer {
name 'Chavdar Botev'
organization 'LinkedIn'
}
developer {
name 'Issac Buenrostro'
organization 'LinkedIn'
}
developer {
name 'Min Tu'
organization 'LinkedIn'
}
developer {
name 'Narasimha Veeramreddy'
organization 'LinkedIn'
}
developer {
name 'Pradhan Cadabam'
organization 'LinkedIn'
}
developer {
name 'Sahil Takiar'
organization 'LinkedIn'
}
developer {
name 'Shirshanka Das'
organization 'LinkedIn'
}
developer {
name 'Yinan Li'
organization 'LinkedIn'
}
developer {
name 'Ying Dai'
organization 'LinkedIn'
}
developer {
name 'Ziyang Liu'
organization 'LinkedIn'
}
}
}
pom.project pomAttributes
}
}
}
Expand Down Expand Up @@ -483,7 +485,7 @@ subprojects {
"https://dropwizard.github.io/metrics/${dropwizardMetricsVersion}/apidocs/"
rootProject.ext.javadocPackages.each {
tasks.javadoc.options.linksOffline "http://linkedin.github.io/gobblin/javadoc/${javadocVersion}/${it}/",
"${rootProject.buildDir}/${it}/docs/javadoc/"
"${rootProject.buildDir}/${it}/docs/javadoc/"
}

afterEvaluate {
Expand Down
16 changes: 8 additions & 8 deletions gobblin-rest-service/gobblin-rest-api/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -28,31 +28,31 @@ if (rootProject.ext.publishToMaven) {
configure(uploadArchives.repositories.mavenDeployer) {
addFilter('gobblin-rest-api-rest-client') {artifact, file ->
artifact.name == 'gobblin-rest-api-rest-client'
}.artifactId = 'gobblin-rest-api-rest-client'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-rest-client'

addFilter('gobblin-rest-api-data-template') {artifact, file ->
artifact.name == 'gobblin-rest-api-data-template'
}.artifactId = 'gobblin-rest-api-data-template'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-data-template'

addFilter('gobblin-rest-api') {artifact, file ->
artifact.name == 'gobblin-rest-api'
}.artifactId = 'gobblin-rest-api'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api'
// artifact names for 'data-model', 'avro-schema' and 'rest-model' may be added as well if needed
}
}

configure(install.repositories.mavenInstaller) {
addFilter('gobblin-rest-api-rest-client') {artifact, file ->
artifact.name == 'gobblin-rest-api-rest-client'
}.artifactId = 'gobblin-rest-api-rest-client'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-rest-client'

addFilter('gobblin-rest-api-data-template') {artifact, file ->
artifact.name == 'gobblin-rest-api-data-template'
}.artifactId = 'gobblin-rest-api-data-template'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-data-template'

addFilter('gobblin-rest-api') {artifact, file ->
artifact.name == 'gobblin-rest-api'
}.artifactId = 'gobblin-rest-api'
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api'
// artifact names for 'data-model', 'avro-schema' and 'rest-model' may be added as well if needed
}

Expand Down Expand Up @@ -101,8 +101,8 @@ task restClientJavadocJar(type: Jar, dependsOn: 'restClientJavadoc') {
classifier = 'javadoc'
}

artifacts {
archives dataTemplateSourcesJar, dataTemplateJavadocJar, restClientSourcesJar, restClientJavadocJar
artifacts {
archives dataTemplateSourcesJar, dataTemplateJavadocJar, restClientSourcesJar, restClientJavadocJar
}

buildscript {
Expand Down

0 comments on commit ffd0aa9

Please sign in to comment.