Skip to content

Conversation

@cloud-fan
Copy link
Contributor

@cloud-fan cloud-fan commented Apr 18, 2016

What changes were proposed in this pull request?

Before this PR, we create accumulators at driver side(and register them) and send them to executor side, then we create TaskMetrics with these accumulators at executor side.
After this PR, we will create TaskMetrics at driver side and send it to executor side, so that we can create accumulators inside TaskMetrics directly, which is cleaner.

How was this patch tested?

existing tests.

@cloud-fan
Copy link
Contributor Author

cc @rxin @JoshRosen @andrewor14

@SparkQA
Copy link

SparkQA commented Apr 18, 2016

Test build #56065 has finished for PR 12472 at commit 9eb2a87.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan cloud-fan changed the title [SPARK-14704][CORE][WIP] create accumulators in TaskMetrics [SPARK-14704][CORE] create accumulators in TaskMetrics Apr 19, 2016
@cloud-fan
Copy link
Contributor Author

ready for review :)

@SparkQA
Copy link

SparkQA commented Apr 19, 2016

Test build #56193 has finished for PR 12472 at commit 6226058.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

protected[spark] def unset(): Unit = taskContext.remove()

/**
* An empty task context that does not represent an actual task.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while you are at this, can you document this is only used for testing?

@rxin
Copy link
Contributor

rxin commented Apr 19, 2016

We should be able to remove almost all the methods on InternalAccumulators.scala, shouldn't we? All that includes create, createAll, createShuffleReadAccums, ...

@SparkQA
Copy link

SparkQA commented Apr 19, 2016

Test build #56204 has finished for PR 12472 at commit b82e90e.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 19, 2016

Test build #56207 has finished for PR 12472 at commit bfa39ff.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Apr 19, 2016

looks pretty good now

@SparkQA
Copy link

SparkQA commented Apr 19, 2016

Test build #56211 has finished for PR 12472 at commit eff1790.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 19, 2016

Test build #56227 has finished for PR 12472 at commit 7f00198.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Apr 20, 2016

Test build #56300 has finished for PR 12472 at commit 7f00198.

  • This patch fails build dependency tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 20, 2016

Test build #56306 has finished for PR 12472 at commit b80f3d0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 20, 2016

Test build #56310 has finished for PR 12472 at commit 471c1e3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Apr 20, 2016

Thanks - merging in master.

@asfgit asfgit closed this in 85d759c Apr 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants