Heterogeneous demo notebook #576
Conversation
python/pyhdk/_common.pyx
Outdated
| return self.c_config.get().exec.heterogeneous.forced_heterogeneous_distribution | ||
|
|
||
| def setMultifragExec(self, on_off): | ||
| self.c_config.get().exec.heterogeneous.enable_multifrag_heterogeneous_execution=on_off |
There was a problem hiding this comment.
Can we leave the interface unchanged and just provide these to the buildConfig?
There was a problem hiding this comment.
I meant all the methods :)
There was a problem hiding this comment.
Exposing the config gives up control over it, which seems undesirable. So, the idea was to hide it from the user and be able to validate it on the HDK side.
There was a problem hiding this comment.
Agree. You can pass required flags to pyhdk.init to get the desired config. If you want to change config between queries, then it's a sign we should move/add some flags to the ExecutionOptions.
There was a problem hiding this comment.
Having these knobs in the execution options makes sense to me. Changing the config between queries, on the contrary, assumes the config object is shared which is an implementation detail, hence, undesirable. I'd suggest either moving these options from config to ExecutionOptions at all or at least mirror them in the execution options and adjust the fromConfig method appropriately. As to the notebook, it can then control the data distribution directly through execute / run API, no reinitialization required.
akroviakov
force-pushed
the
akroviak/heterogeneous-demo-pyhdk
branch
from
ff8ddc6 to
37b16ec
Compare
This PR introduces an interface for HDK's heterogeneity exploration through pyHDK:
To have more control on certain aspects of query execution and data proportion per CPU/GPU, as well as cleaning GPU/CPU memory levels, the cython interface was extended.
Additionally, there is a simplistic example of how one could use it. Feel free to adapt it to your needs!