Skip to content

Commit fd3a8a1

Browse files
Ilya Ganelinsquito
authored andcommitted
[SPARK-733] Add documentation on use of accumulators in lazy transformation
I've added documentation clarifying the particular lack of clarity highlighted in the relevant JIRA. I've also added code examples for this issue to clarify the explanation. Author: Ilya Ganelin <[email protected]> Closes #4022 from ilganeli/SPARK-733 and squashes the following commits: 587def5 [Ilya Ganelin] Updated to clarify verbage df3afd7 [Ilya Ganelin] Revert "Partially updated task metrics to make some vars private" 3f6c512 [Ilya Ganelin] Revert "Completed refactoring to make vars in TaskMetrics class private" 58034fb [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-733 4dc2cdb [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-733 3a38db1 [Ilya Ganelin] Verified documentation update by building via jekyll 33b5a2d [Ilya Ganelin] Added code examples for java and python 1fd59b2 [Ilya Ganelin] Updated documentation for accumulators to highlight lazy evaluation issue 5525c20 [Ilya Ganelin] Completed refactoring to make vars in TaskMetrics class private c64da4f [Ilya Ganelin] Partially updated task metrics to make some vars private
1 parent d05c9ee commit fd3a8a1

File tree

1 file changed

+28
-0
lines changed

1 file changed

+28
-0
lines changed

docs/programming-guide.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1316,7 +1316,35 @@ For accumulator updates performed inside <b>actions only</b>, Spark guarantees t
13161316
will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware
13171317
of that each task's update may be applied more than once if tasks or job stages are re-executed.
13181318

1319+
Accumulators do not change the lazy evaluation model of Spark. If they are being updated within an operation on an RDD, their value is only updated once that RDD is computed as part of an action. Consequently, accumulator updates are not guaranteed to be executed when made within a lazy transformation like `map()`. The below code fragment demonstrates this property:
13191320

1321+
<div class="codetabs">
1322+
1323+
<div data-lang="scala" markdown="1">
1324+
{% highlight scala %}
1325+
val acc = sc.accumulator(0)
1326+
data.map(x => acc += x; f(x))
1327+
// Here, acc is still 0 because no actions have cause the `map` to be computed.
1328+
{% endhighlight %}
1329+
</div>
1330+
1331+
<div data-lang="java" markdown="1">
1332+
{% highlight java %}
1333+
Accumulator<Integer> accum = sc.accumulator(0);
1334+
data.map(x -> accum.add(x); f(x););
1335+
// Here, accum is still 0 because no actions have cause the `map` to be computed.
1336+
{% endhighlight %}
1337+
</div>
1338+
1339+
<div data-lang="python" markdown="1">
1340+
{% highlight python %}
1341+
accum = sc.accumulator(0)
1342+
data.map(lambda x => acc.add(x); f(x))
1343+
# Here, acc is still 0 because no actions have cause the `map` to be computed.
1344+
{% endhighlight %}
1345+
</div>
1346+
1347+
</div>
13201348

13211349
# Deploying to a Cluster
13221350

0 commit comments

Comments
 (0)