Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the default javaagent artifact to have exporters #4106

Merged
merged 26 commits into from
Oct 2, 2021
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
af7a83e
Change default javaagent to all, introduce slim
trask Sep 12, 2021
d99849d
Update release builds
trask Sep 12, 2021
9c6f04f
Update examples build
trask Sep 12, 2021
9b20ff0
Update benchmark-overhead
trask Sep 12, 2021
6ea1a07
Update benchmark-e2e
trask Sep 12, 2021
6ab2c68
Update docs where ok pre-release
trask Sep 12, 2021
6e53ea0
Update misc
trask Sep 12, 2021
2a4839c
Fix references to fullJavaagentJar
trask Sep 13, 2021
64df703
Fix it
trask Sep 13, 2021
9c43849
Delete doc for pre-1.0 releases
trask Sep 13, 2021
5113576
Attach backwards compatible "all" artifact to releases
trask Sep 13, 2021
42f4830
Comment
trask Sep 13, 2021
1687020
Comment
trask Sep 13, 2021
ae7e2d1
guessing
trask Sep 13, 2021
2b9de15
fix
trask Sep 13, 2021
cc35263
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 13, 2021
6cdfbce
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 23, 2021
f01fbf6
Update version
trask Sep 23, 2021
e6e4e3d
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 23, 2021
7b4b731
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 28, 2021
73226a0
Add otlp/grpc exporter to slim artifact
trask Sep 28, 2021
f6c3a94
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 28, 2021
1fa5ef6
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 29, 2021
49e1aa8
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 29, 2021
3db6c45
Fix comment
trask Sep 29, 2021
61ab8fd
Merge remote-tracking branch 'upstream/main' into change-default-java…
trask Sep 30, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .github/workflows/patch-release-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,18 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}-all.jar
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}.jar
asset_name: opentelemetry-javaagent.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the rename but it's sort of a breaking change for e.g., Dockerfiles that may be curling the "latest release" URL. Should we upload both at least for one release?

We may also consider whether it is bad practice in general to point at an unversioned URL - if so we can probably update our README to not point at the latest version URL to discourage usage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we upload both at least for one release?

makes sense to me, I added attaching backwards compatible "all" artifact step to both release and patch release builds

update our README to not point at the latest version URL to discourage usage

I like explicit versions in docs (and keeping it updated), IIRC we were stuck on how to automate this? maybe we can tack this problem on to #3516 @iNikem?

asset_content_type: application/java-archive

# TODO (trask) delete this after the 1.7.0 release (to make sure it is used for any 1.6.x patches)
- name: Upload Release Asset (backwards compatible "all" artifact)
id: upload-release-asset
uses: actions/[email protected]
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}.jar
asset_name: opentelemetry-javaagent-all.jar
asset_content_type: application/java-archive
14 changes: 13 additions & 1 deletion .github/workflows/release-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,18 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}-all.jar
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}.jar
asset_name: opentelemetry-javaagent.jar
asset_content_type: application/java-archive

# TODO (trask) delete this after the 1.7.0 release
- name: Upload Release Asset (backwards compatible "all" artifact)
id: upload-release-asset
uses: actions/[email protected]
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: javaagent/build/libs/opentelemetry-javaagent-${{ github.event.inputs.version }}.jar
asset_name: opentelemetry-javaagent-all.jar
asset_content_type: application/java-archive
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ java -version

and then you can find the java agent artifact at

`javaagent/build/libs/opentelemetry-javaagent-<version>-all.jar`.
`javaagent/build/libs/opentelemetry-javaagent-<version>.jar`.

### IntelliJ setup and troubleshooting

Expand Down
4 changes: 2 additions & 2 deletions benchmark-e2e/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ dependencies {

tasks {
test {
dependsOn(":javaagent:fullJavaagentJar")
dependsOn(":javaagent:shadowJar")
maxParallelForks = 2

doFirst {
jvmArgs("-Dio.opentelemetry.smoketest.agent.shadowJar.path=${project(":javaagent").tasks.getByName<ShadowJar>("fullJavaagentJar").archivePath}")
jvmArgs("-Dio.opentelemetry.smoketest.agent.shadowJar.path=${project(":javaagent").tasks.getByName<ShadowJar>("shadowJar").archivePath}")
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ private void runBenchmark() throws InterruptedException {
.withLogConsumer(new Slf4jLogConsumer(logger))
.withNetworkAliases("app")
.withCopyFileToContainer(
MountableFile.forHostPath(agentPath), "/opentelemetry-javaagent-all.jar")
MountableFile.forHostPath(agentPath), "/opentelemetry-javaagent.jar")
.withEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "collector:4317")
.withEnv("JAVA_TOOL_OPTIONS", "-javaagent:/opentelemetry-javaagent-all.jar")
.withEnv("JAVA_TOOL_OPTIONS", "-javaagent:/opentelemetry-javaagent.jar")
.withExposedPorts(8080);
containers.add(app);

Expand Down
2 changes: 1 addition & 1 deletion benchmark-overhead-jmh/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ tasks {
val jmhStartFlightRecording = gradle.startParameter.projectProperties.get("jmh.startFlightRecording")

named<JMHTask>("jmh") {
val shadowTask = project(":javaagent").tasks.named<ShadowJar>("fullJavaagentJar").get()
val shadowTask = project(":javaagent").tasks.named<ShadowJar>("shadowJar").get()
inputs.files(layout.files(shadowTask))

// note: without an exporter, toSpanData() won't even be called
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ private Path downloadAgent(URL agentUrl) throws Exception {
OkHttpClient client = new OkHttpClient();
Response response = client.newCall(request).execute();
byte[] raw = response.body().bytes();
Path path = Paths.get(".", "opentelemetry-javaagent-all.jar");
Path path = Paths.get(".", "opentelemetry-javaagent.jar");
Files.write(
path,
raw,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Optional<Path> resolve() throws IOException {
String latestFilename = fetchLatestFilename(version);
String url = BASE_URL + "/" + version + "/" + latestFilename;
byte[] jarBytes = fetchBodyBytesFrom(url);
Path path = Paths.get(".", "opentelemetry-javaagent-SNAPSHOT-all.jar");
Path path = Paths.get(".", "opentelemetry-javaagent-SNAPSHOT.jar");
Files.write(
path,
jarBytes,
Expand All @@ -55,11 +55,12 @@ private String fetchLatestFilename(String version) throws IOException {
elem -> {
String classifier = $(elem).child("classifier").content();
String extension = $(elem).child("extension").content();
// TODO (trask) this needs to be updated now that no more "all" artifact
return "all".equals(classifier) && "jar".equals(extension);
})
.map(e -> $(e).child("value").content())
.findFirst()
.map(value -> "opentelemetry-javaagent-" + value + "-all.jar")
.map(value -> "opentelemetry-javaagent-" + value + ".jar")
.orElseThrow();
}

Expand Down
2 changes: 1 addition & 1 deletion benchmark/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ tasks {
}

named("jmh") {
dependsOn(":javaagent:fullJavaagentJar")
dependsOn(":javaagent:shadowJar")
}
}

Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,5 @@ The following example shows remote debugger configuration. The breakpoints
should work in any code except ByteBuddy advice methods.

```bash
java -agentlib:jdwp="transport=dt_socket,server=y,suspend=y,address=5000" -javaagent:opentelemetry-javaagent-<version>-all.jar -jar app.jar
java -agentlib:jdwp="transport=dt_socket,server=y,suspend=y,address=5000" -javaagent:opentelemetry-javaagent-<version>.jar -jar app.jar
```
4 changes: 2 additions & 2 deletions docs/contributing/javaagent-jar-components.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ still access helper classes from bootstrap classloader.
### Agent jar structure

If you now look inside
`javaagent/build/libs/opentelemetry-javaagent-<version>-all.jar`, you will see the
`javaagent/build/libs/opentelemetry-javaagent-<version>.jar`, you will see the
following "clusters" of classes:

Available in the system class loader:
Expand All @@ -94,7 +94,7 @@ during creation of `javaagent` jar file by Shadow Gradle plugin
Available in the agent class loader:
- `inst/` - contains `javaagent-tooling` and `javaagent-extension-api` modules and
`instrumentation` submodules, loaded and isolated inside `AgentClassLoader`.
Including OpenTelemetry SDK (and the built-in exporters when using the `-all` artifact).
Includes the OpenTelemetry SDK.

![Agent initialization sequence](initialization-sequence.svg)
[Image source](https://docs.google.com/drawings/d/1GHAcJ8AOaf_v2Ip82cQD9dN0mtvSk2C1B11KfwV2U8o)
Expand Down
5 changes: 0 additions & 5 deletions docs/manual-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,6 @@ or they might want to manually create spans for their own custom code.

# Dependencies

> :warning: prior to version 1.0.0, `opentelemetry-javaagent-all.jar`
only supports manual instrumentation using the `opentelemetry-api` version with the same version
number as the Java agent you are using. Starting with 1.0.0, the Java agent will start supporting
multiple (1.0.0+) versions of `opentelemetry-api`.

Comment on lines -21 to -25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure about that? Have we tested that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a smoke test that verifies interop with opentelemetry-api 1.0.0, and #3044 to track doing better than that with muzzle

(also, this is old doc, I'm removing it, see #4106 (comment))

You'll need to add a dependency on the `opentelemetry-api` library to get started; if you intend to
use the `@WithSpan` annotation, also include the `opentelemetry-extension-annotations` dependency.

Expand Down
2 changes: 1 addition & 1 deletion examples/distro/agent/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ configurations {
dependencies {
customShadow project(path: ":custom", configuration: "shadow")
customShadow project(path: ":instrumentation", configuration: "shadow")
implementation "io.opentelemetry.javaagent:opentelemetry-javaagent:${versions.opentelemetryJavaagent}:all"
implementation "io.opentelemetry.javaagent:opentelemetry-javaagent:${versions.opentelemetryJavaagent}"
}

CopySpec isolateSpec() {
Expand Down
8 changes: 4 additions & 4 deletions examples/distro/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ subprojects {
versions = [
opentelemetry : "1.6.0",
opentelemetryAlpha : "1.6.0-alpha",
opentelemetryJavaagent : "1.6.0-SNAPSHOT",
opentelemetryJavaagentAlpha: "1.6.0-alpha-SNAPSHOT",
bytebuddy : "1.10.18",
guava : "30.1-jre"
opentelemetryJavaagent : "1.7.0-SNAPSHOT",
opentelemetryJavaagentAlpha: "1.7.0-alpha-SNAPSHOT",
bytebuddy : "1.10.18",
guava : "30.1-jre"
]

deps = [
Expand Down
8 changes: 4 additions & 4 deletions examples/extension/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ ext {
versions = [
opentelemetry : "1.6.0",
opentelemetryAlpha : "1.6.0-alpha",
opentelemetryJavaagent : "1.6.0-SNAPSHOT",
opentelemetryJavaagentAlpha: "1.6.0-alpha-SNAPSHOT",
opentelemetryJavaagent : "1.7.0-SNAPSHOT",
opentelemetryJavaagentAlpha: "1.7.0-alpha-SNAPSHOT",
]

deps = [
Expand Down Expand Up @@ -92,7 +92,7 @@ dependencies {
testRuntimeOnly("ch.qos.logback:logback-classic:1.2.3")

//Otel Java instrumentation that we use and extend during integration tests
otel("io.opentelemetry.javaagent:opentelemetry-javaagent:${versions.opentelemetryJavaagent}:all")
otel("io.opentelemetry.javaagent:opentelemetry-javaagent:${versions.opentelemetryJavaagent}")

//TODO remove when start using io.opentelemetry.instrumentation.javaagent-instrumentation plugin
add("muzzleTooling", "io.opentelemetry.javaagent:opentelemetry-javaagent-tooling:${versions.opentelemetryJavaagentAlpha}")
Expand All @@ -111,7 +111,7 @@ task agentManifest(type: Copy) {
//The location of extension directory inside agent jar is hard-coded in the agent source code
task extendedAgent(type: Jar) {
dependsOn agentManifest
archiveFileName = "opentelemetry-javaagent-all.jar"
archiveFileName = "opentelemetry-javaagent.jar"
manifest.from "$buildDir/META-INF/MANIFEST.MF"
from zipTree(configurations.otel.singleFile)
from(tasks.shadowJar.archiveFile) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ plugins {

// TODO write a smoke test that does the following:
//
// docker run --mount 'type=bind,src=$AGENT_PATH,dst=/opentelemetry-javaagent-all.jar'
// -e JAVA_TOOL_OPTIONS=-javaagent:/opentelemetry-javaagent-all.jar
// docker run --mount 'type=bind,src=$AGENT_PATH,dst=/opentelemetry-javaagent.jar'
// -e JAVA_TOOL_OPTIONS=-javaagent:/opentelemetry-javaagent.jar
// wso2/wso2ei-business-process:6.5.0
//
// without this instrumentation, the following error will appear in the docker logs:
Expand Down
48 changes: 33 additions & 15 deletions javaagent/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,14 @@ val exporterLibs by configurations.creating {
isCanBeResolved = true
isCanBeConsumed = false
}
// this configuration collects just exporter libs for slim artifact (also placed in the agent classloader & isolated from the instrumented application)
val exporterSlimLibs by configurations.creating {
isCanBeResolved = true
isCanBeConsumed = false
}

// exclude dependencies that are to be placed in bootstrap from agent libs - they won't be added to inst/
listOf(javaagentLibs, exporterLibs).forEach {
listOf(javaagentLibs, exporterLibs, exporterSlimLibs).forEach {
it.run {
exclude("org.slf4j")
exclude("io.opentelemetry", "opentelemetry-api")
Expand Down Expand Up @@ -73,6 +78,10 @@ dependencies {

exporterLibs(project(":javaagent-exporters"))

exporterSlimLibs("io.opentelemetry:opentelemetry-exporter-otlp")
exporterSlimLibs("io.opentelemetry:opentelemetry-exporter-otlp-metrics")
exporterSlimLibs("io.grpc:grpc-okhttp:1.41.0")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just realized that because this is using okhttp, adding the HTTP exporters as well should be almost no additional bytecode, so maybe worth it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well assuming it doesn't cause dependency hell, wow that's old

https://github.com/grpc/grpc-java/blob/master/build.gradle#L158

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Err - nevermind, yeah it wouldn't cause dependency hell since it's a different artifact

https://github.com/open-telemetry/opentelemetry-java/blob/main/dependencyManagement/build.gradle.kts#L78

So scratch the almost-no-additional-bytecode idea.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, just some random trivia grpc/grpc-java#6119 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"OkHttp" name for the transport is a bit of a misnomer

this thread helped my mental picture 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that we can exclude okhttp 2 from the transitive dependencies

open-telemetry/opentelemetry-java#3678

though I added a comment on the grpc-java issue to see if there is anything obviously bad with that.

Currently I guess we effectively have two copies of okhttp2, the forked-into-grpc one and the transitive dependency which we don't seem to need.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added #4240 to discuss/track


// We only have compileOnly dependencies on these to make sure they don't leak into POMs.
licenseReportDependencies("com.github.ben-manes.caffeine:caffeine") {
isTransitive = false
Expand Down Expand Up @@ -143,12 +152,23 @@ tasks {
archiveFileName.set("exporterLibs-relocated.jar")
}

// Includes instrumentations, but not exporters
val relocateExporterSlimLibs by registering(ShadowJar::class) {
configurations = listOf(exporterSlimLibs)

archiveFileName.set("exporterSlimLibs-relocated.jar")
}

// Includes everything needed for OOTB experience
val shadowJar by existing(ShadowJar::class) {
configurations = listOf(bootstrapLibs)

dependsOn(relocateJavaagentLibs)
// without an explicit dependency on jar here, :javaagent:test fails on CI because :javaagent:jar
// runs after :javaagent:shadowJar and loses (at least) the manifest entries
dependsOn(jar, relocateJavaagentLibs, relocateExporterLibs)
isolateClasses(relocateJavaagentLibs.get().outputs.files)
isolateClasses(relocateExporterLibs.get().outputs.files)

duplicatesStrategy = DuplicatesStrategy.EXCLUDE

archiveClassifier.set("")

Expand All @@ -164,17 +184,15 @@ tasks {
}
}

// Includes everything needed for OOTB experience
val fullJavaagentJar by registering(ShadowJar::class) {
// Includes instrumentations, but not exporters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slim does include OTLP exporter, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed 👍

val slimShadowJar by registering(ShadowJar::class) {
configurations = listOf(bootstrapLibs)

dependsOn(relocateJavaagentLibs, relocateExporterLibs)
dependsOn(relocateJavaagentLibs, relocateExporterSlimLibs)
isolateClasses(relocateJavaagentLibs.get().outputs.files)
isolateClasses(relocateExporterLibs.get().outputs.files)

duplicatesStrategy = DuplicatesStrategy.EXCLUDE
isolateClasses(relocateExporterSlimLibs.get().outputs.files)

archiveClassifier.set("all")
archiveClassifier.set("slim")

manifest {
attributes(shadowJar.get().manifest.attributes)
Expand Down Expand Up @@ -207,18 +225,18 @@ tasks {
}

assemble {
dependsOn(shadowJar, fullJavaagentJar, baseJavaagentJar)
dependsOn(shadowJar, slimShadowJar, baseJavaagentJar)
}

withType<Test>().configureEach {
dependsOn(fullJavaagentJar)
inputs.file(fullJavaagentJar.get().archiveFile)
dependsOn(shadowJar)
inputs.file(shadowJar.get().archiveFile)

jvmArgs("-Dotel.javaagent.debug=true")

doFirst {
// Defining here to allow jacoco to be first on the command line.
jvmArgs("-javaagent:${fullJavaagentJar.get().archiveFile.get().asFile}")
jvmArgs("-javaagent:${shadowJar.get().archiveFile.get().asFile}")
}

testLogging {
Expand All @@ -237,7 +255,7 @@ tasks {
publishing {
publications {
named<MavenPublication>("maven") {
artifact(fullJavaagentJar)
artifact(slimShadowJar)
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions smoke-tests/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ dependencies {

tasks {
test {
inputs.files(project(":javaagent").tasks.getByName("fullJavaagentJar").outputs.files)
inputs.files(project(":javaagent").tasks.getByName("shadowJar").outputs.files)

testLogging.showStandardStreams = true

Expand Down Expand Up @@ -78,7 +78,7 @@ tasks {
}
}

val shadowTask = project(":javaagent").tasks.named<ShadowJar>("fullJavaagentJar").get()
val shadowTask = project(":javaagent").tasks.named<ShadowJar>("shadowJar").get()
inputs.files(layout.files(shadowTask))

doFirst {
Expand Down