Migrate from master/slave language #40
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
November 27, 2015 09:08
… parameter optional for ExecutorMemoryManager.
… put them in the path when running scala tests.
…ng loading of native libs in subsequent tests.
…alizing to ColumnPartitionData.
…chema. Added various tests. Added BlockManager tests that check serialization. Turned on JCuda exceptions when setting up SparkEnv.
… possibility of two concurrent builds by different users.
Killing it works better than asking to die after all... This reverts commit 59f3971.
…ixed. Now it does not hang on tests. This shouldn't really change anything, it's just a library code that sets a variable, which is not used in the test at all - the test does not touch JCuda at all. But somehow it does not like the code in SparkEnv.
… library properly.
…a from TypeTag - which is not always available on RDDs - to ClassTag. Added relevant tests.
merge with commit bc0f30 (12/22/2015)
Enable GPU in an environment with different machines for master and slave
support GPU on aws ec2
update scripts under gpudev/util
clear GPU device memory for output RDD add new API to register a PTX using String
Update CUDAManager
…for map() or reduce() Enable it with 'spark.gpu.codegen=true'
Add experimental version of GPU code generation
Introduce org.apache.spar.unsafe.memory.Pointer
Refactoring for less dependecy on JCuda
call gpudev/util/embedjcudalib.sh in make-distribution.sh
fix style check error
…heap) java heap may not work for now
Select types of heap memory allocator
make gpu caching effective (without calling .cache)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For diversity reasons, it would be nice to try to avoid 'master' and 'slave' terminology in this repository which can be associated to slavery. The master-slave terminology could be problematic for people in several countries which has the history of slavery like Romania, USA and many others. Thank you for considering the proposal. Let me know if any changes in the PR are needed, I would be happy to implement them.