[fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380

coderzc · 2022-07-04T16:27:33Z

Motivation

Miss the timestamp field when constructing SchemaData from SchemaRegistryFormat.SchemaInfo, so timestamp value 0 gets the scheme.
See:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java

Lines 621 to 629 in 95b9846

    
           static SchemaData schemaInfoToSchema(SchemaRegistryFormat.SchemaInfo info) { 
        
               return SchemaData.builder() 
        
                   .user(info.getUser()) 
        
                   .type(convertToDomainType(info.getType())) 
        
                   .data(info.getSchema().toByteArray()) 
        
                   .isDeleted(info.getDeleted()) 
        
                   .props(toMap(info.getPropsList())) 
        
                   .build(); 
        
           }

Modifications

Expose timestamp field for SchemaData
Add timestamp field for SchemaInfo

Verifying this change

Make sure that the change passes the CI checks.

Documentation

Check the box below or label this PR directly.

Need to update docs?

doc-required
(Your PR needs to update docs and you will update later)
doc-not-needed
(Please explain why)
doc
(Your PR contains doc changes)
doc-complete
(Docs have been already added)

coderzc · 2022-07-05T15:51:10Z

/pulsarbot run-failure-checks

codelipenghui · 2022-07-06T00:34:40Z

@coderzc Please rebase the apache/master branch. There are some flaky test fixes.

coderzc · 2022-07-06T03:23:16Z

CI failed:

Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project pulsar-broker: There are test failures.
  Error: 
  Error:  Please refer to /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire-reports for the individual test results.
  Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
  Error:  ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
  Error:  Command was /bin/sh -c cd /home/runner/work/pulsar/pulsar/pulsar-broker && /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+ExitOnOutOfMemoryError -Xmx1G -XX:+UseZGC -Dpulsar.allocator.pooled=true -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.out_of_memory_policy=FallbackToHeap -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED -jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire/surefirebooter16453686720892765404.jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire 2022-07-06T02-31-12_408-jvmRun3 surefire4226580036504256851tmp surefire_31003965489488873406tmp
  Error:  Error occurred in starting fork, check output in log
  Error:  Process Exit Code: 3
  Error:  Crashed tests:
  Error:  org.apache.pulsar.client.api.MultiTopicsConsumerTest
  Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
  Error:  Command was /bin/sh -c cd /home/runner/work/pulsar/pulsar/pulsar-broker && /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+ExitOnOutOfMemoryError -Xmx1G -XX:+UseZGC -Dpulsar.allocator.pooled=true -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.out_of_memory_policy=FallbackToHeap -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED -jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire/surefirebooter16453686720892765404.jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire 2022-07-06T02-31-12_408-jvmRun3 surefire4226580036504256851tmp surefire_31003965489488873406tmp
  Error:  Error occurred in starting fork, check output in log
  Error:  Process Exit Code: 3
  Error:  Crashed tests:
  Error:  org.apache.pulsar.client.api.MultiTopicsConsumerTest
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:511)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:383)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:247)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1161)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1002)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:848)
  Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
  Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
  Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
  Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
  Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
  Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
  Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
  Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
  Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
  Error:  Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
  Error:  Command was /bin/sh -c cd /home/runner/work/pulsar/pulsar/pulsar-broker && /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+ExitOnOutOfMemoryError -Xmx1G -XX:+UseZGC -Dpulsar.allocator.pooled=true -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.out_of_memory_policy=FallbackToHeap -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED -jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire/surefirebooter16453686720892765404.jar /home/runner/work/pulsar/pulsar/pulsar-broker/target/surefire 2022-07-06T02-31-12_408-jvmRun3 surefire4226580036504256851tmp surefire_3100396548948887[3406](https://github.com/apache/pulsar/runs/7207156785?check_suite_focus=true#step:10:3407)tmp
  Error:  Error occurred in starting fork, check output in log
  Error:  Process Exit Code: 3
  Error:  Crashed tests:
  Error:  org.apache.pulsar.client.api.MultiTopicsConsumerTest
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:670)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:116)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:372)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:348)
  Error:  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  Error:  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  Error:  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  Error:  	at java.base/java.lang.Thread.run(Thread.java:833)
  Error:  -> [Help 1]
  Error: 
  Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
  Error:  Re-run Maven using the -X switch to enable full debug logging.
  Error: 
  Error:  For more information about the errors and possible solutions, please read the following articles:
  Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
  Error: The command has failed after 3 attempts.
  Error: Process completed with exit code 1.

coderzc · 2022-07-06T07:13:25Z

/pulsarbot run-failure-checks

coderzc · 2022-07-07T06:40:55Z

/pulsarbot run-failure-checks

coderzc · 2022-07-08T07:29:53Z

CI failed:

  Error:  Failures: 
  Error:  org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest)
  [INFO]   Run 1: PASS
  Error:    Run 2: PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer » ThreadTimeout

coderzc · 2022-07-11T01:55:04Z

/pulsarbot run-failure-checks

coderzc · 2022-07-11T08:37:11Z

/pulsarbot run-failure-checks

coderzc · 2022-07-11T15:29:53Z

/pulsarbot run-failure-checks

…e#16380) Fixes apache#16379 ### Motivation Miss the timestamp field when constructing SchemaData from `SchemaRegistryFormat.SchemaInfo`, so timestamp value 0 gets the scheme. See: https://github.com/apache/pulsar/blob/95b984694bd3c4f1eadf4e6198de033f9b517128/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java#L621-L629 ### Modifications * Expose timestamp field for SchemaData * Add timestamp field for SchemaInfo

Fixes #16379 Miss the timestamp field when constructing SchemaData from `SchemaRegistryFormat.SchemaInfo`, so timestamp value 0 gets the scheme. See: https://github.com/apache/pulsar/blob/95b984694bd3c4f1eadf4e6198de033f9b517128/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java#L621-L629 * Expose timestamp field for SchemaData * Add timestamp field for SchemaInfo (cherry picked from commit bd150c0)

…e#16380) Fixes apache#16379 Miss the timestamp field when constructing SchemaData from `SchemaRegistryFormat.SchemaInfo`, so timestamp value 0 gets the scheme. See: https://github.com/apache/pulsar/blob/95b984694bd3c4f1eadf4e6198de033f9b517128/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java#L621-L629 * Expose timestamp field for SchemaData * Add timestamp field for SchemaInfo (cherry picked from commit bd150c0) (cherry picked from commit daeb418)

See: apache@4f204e8 Pulsar PR: apache/pulsar#16380

Expose timestamp field for SchemaData&SchemaInfo

921a10b

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 4, 2022

codelipenghui approved these changes Jul 4, 2022

View reviewed changes

codelipenghui assigned coderzc Jul 4, 2022

codelipenghui requested a review from congbobo184 July 4, 2022 16:39

codelipenghui added this to the 2.11.0 milestone Jul 4, 2022

codelipenghui added type/bug The PR fixed a bug or issue reported a bug area/schema release/2.10.2 release/2.9.4 labels Jul 4, 2022

Technoboy- approved these changes Jul 5, 2022

View reviewed changes

fix doc style

6a3c058

coderzc force-pushed the fix_schema_timestamp branch 2 times, most recently from 40ea5fa to 111fd1b Compare July 5, 2022 10:31

fix test

3820829

coderzc force-pushed the fix_schema_timestamp branch from 111fd1b to 3820829 Compare July 5, 2022 14:33

Merge branch 'apache:master' into fix_schema_timestamp

111ecf8

congbobo184 approved these changes Jul 6, 2022

View reviewed changes

Merge branch 'apache:master' into fix_schema_timestamp

ae1d529

Merge branch 'apache:master' into fix_schema_timestamp

8005012

Merge branch 'apache:master' into fix_schema_timestamp

ff4de4e

codelipenghui merged commit bd150c0 into apache:master Jul 12, 2022

codelipenghui added the cherry-picked/branch-2.10 label Jul 15, 2022

mattisonchao added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Jul 15, 2022

coderzc deleted the fix_schema_timestamp branch July 25, 2022 06:11

dialalpha added a commit to attentive-mobile/flink-connector-pulsar that referenced this pull request Mar 28, 2023

fix SchemaInfo breaking change introduced in pulsar client 2.10.2

ba7bd6f

See: apache@4f204e8 Pulsar PR: apache/pulsar#16380

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380

[fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380

coderzc commented Jul 4, 2022 •

edited by codelipenghui

Loading

coderzc commented Jul 5, 2022

codelipenghui commented Jul 6, 2022

coderzc commented Jul 6, 2022

coderzc commented Jul 6, 2022

coderzc commented Jul 7, 2022

coderzc commented Jul 8, 2022 •

edited

Loading

coderzc commented Jul 11, 2022

coderzc commented Jul 11, 2022

coderzc commented Jul 11, 2022

	static SchemaData schemaInfoToSchema(SchemaRegistryFormat.SchemaInfo info) {
	return SchemaData.builder()
	.user(info.getUser())
	.type(convertToDomainType(info.getType()))
	.data(info.getSchema().toByteArray())
	.isDeleted(info.getDeleted())
	.props(toMap(info.getPropsList()))
	.build();
	}

[fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380

[fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380

Conversation

coderzc commented Jul 4, 2022 • edited by codelipenghui Loading

Motivation

Modifications

Verifying this change

Documentation

coderzc commented Jul 5, 2022

codelipenghui commented Jul 6, 2022

coderzc commented Jul 6, 2022

coderzc commented Jul 6, 2022

coderzc commented Jul 7, 2022

coderzc commented Jul 8, 2022 • edited Loading

coderzc commented Jul 11, 2022

coderzc commented Jul 11, 2022

coderzc commented Jul 11, 2022

coderzc commented Jul 4, 2022 •

edited by codelipenghui

Loading

coderzc commented Jul 8, 2022 •

edited

Loading