Using plasma on Kubernetes

[remram44]

I am trying to use plasma to store datasets in memory, and share them between pods.

I find that this does not work well, and in particular, plasma.get/plasma.put tends to hang with no specific error message.

I am sure other people have tried this setup, I would love to hear about their experience.

The setup is:

• Running plasma_store and clients in different pod on the same node
• The socket is placed in one R/W volume mounted in all pods
• Tried plasma_store using shm and hugepages (-h). shm hangs if get()ing an object submitted by another client, hugepages seems to complain about missing huge pages when mmapping

Note that I could get this running using Docker containers just fine. I understand that some of those issues are due to Kubernetes more than plasma, but I would love some pointers.

cc @mitar

[pcmoritz]

Hey @remram44 thanks for bringing this up! Do you have Kubernetes scripts to reproduce this and instructions how to set it up on EC2 and reproduce this issue? Any pointers are welcome.

[robertnishihara]

@remram44 how did you get it working between Docker containers? Did you have to do anything special?

[remram44]

On Docker, I didn't have to do anything. I ran with native Docker on macOS. However on doing this again it seems to work if I don't pass an explicit ObjectID to put(), otherwise get() hangs.

Server:

docker run -ti --rm --name plasmaserver -v plasmasocket:/mnt/socket remram/python3-pyarrow plasma_store -s /mnt/socket/plasma -m 10000000

Sender:

docker run -ti --rm --link plasmaserver -v plasmasocket:/mnt/socket remram/python3-pyarrow python -c 'import pyarrow.plasma as plasma; client = plasma.connect("/mnt/socket/plasma", "", 0); print(client.put("hello, world").binary())'
b'\x10\x85\x1b\xc6\xe3\xc6\x9f\x8d\x13\x1e\xa7\xda\xf3\xd9\xf0\x0cZ\xf1\xd7/'

Getter:

docker run -ti --rm --link plasmaserver -v plasmasocket:/mnt/socket remram/python3-pyarrow python -c 'import pyarrow.plasma as plasma; client = plasma.connect("/mnt/socket/plasma", "", 0); print(client.get(plasma.ObjectID(b"\x10\x85\x1b\xc6\xe3\xc6\x9f\x8d\x13\x1e\xa7\xda\xf3\xd9\xf0\x0cZ\xf1\xd7/")))'
hello, world

Explicit sender:

docker run -ti --rm --link plasmaserver -v plasmasocket:/mnt/socket remram/python3-pyarrow python -c 'import pyarrow.plasma as plasma; client = plasma.connect("/mnt/socket/plasma", "", 0); client.put("hello, world", plasma.ObjectID(b"testidhere"))'

Getter:

docker run -ti --rm --link plasmaserver -v plasmasocket:/mnt/socket remram/python3-pyarrow python -c 'import pyarrow.plasma as plasma; client = plasma.connect("/mnt/socket/plasma", "", 0); print(client.get(plasma.ObjectID(b"testidhere")))'
<hangs>

[remram44]

I ran this on Kubernetes on Google Cloud with this configuration:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: plasmaserver
spec:
  replicas: 1
  template:
    metadata:
      labels:
        thing: plasmaserver
    spec:
      containers:
      - name: main
        image: remram/python3-pyarrow
        command: ['/bin/sh', '-c', 'plasma_store -s /mnt/socket/plasma -m 10000000']
# or
#        command: ['/bin/sh', '-c', 'plasma_store -s /mnt/socket/plasma -m 10000000 -d /mnt/hugepages -h']
        volumeMounts:
        - mountPath: /mnt/socket
          name: socket
        - mountPath: /mnt/hugepages
          name: hugepages
      volumes:
      - name: socket
        persistentVolumeClaim:
          claimName: plasmasocketvc
      - name: hugepages
        persistentVolumeClaim:
          claimName: hugepagesvc
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: plasma1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        thing: plasma1
    spec:
      containers:
      - name: main
        image: remram/python3-pyarrow
        command: ['/bin/sh', '-c', 'while true; do sleep 30; done']
        volumeMounts:
        - mountPath: /mnt
          name: socket
      volumes:
      - name: socket
        persistentVolumeClaim:
          claimName: plasmasocketvc
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: plasma2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        thing: plasma2
    spec:
      containers:
      - name: main
        image: remram/python3-pyarrow
        command: ['/bin/sh', '-c', 'while true; do sleep 30; done']
        volumeMounts:
        - mountPath: /mnt
          name: socket
      volumes:
      - name: socket
        persistentVolumeClaim:
          claimName: plasmasocketvc
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: plasmasocketv
  labels:
    thing: plasmasocket
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/plasma-rr4"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: plasmasocketvc
spec:
  storageClassName: ""
  selector:
    matchLabels:
      thing: plasmasocket
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: hugepagesv
  labels:
    thing: hugepages
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/hugepages"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: hugepagesvc
spec:
  storageClassName: ""
  selector:
    matchLabels:
      thing: hugepages
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

Then I ran commands on plasma1 and plasma2 using kubectl exec.

[pcmoritz]

@remram44 Thanks! The hanging you are seeing is unrelated to using docker. It hangs because ObjectIDs need to be exactly 20 bytes long. So even without docker, this hangs:

In [5]: client.put("hello", plasma.ObjectID(b"hi"))
Out[5]: ObjectID(68690000537f0000300000000000000091010000)

In [6]: client.get(plasma.ObjectID(b"hi"))

Whereas this works:

In [3]: client.put("hello", plasma.ObjectID(20 * b"h"))
Out[3]: ObjectID(6868686868686868686868686868686868686868)

In [4]: client.get(plasma.ObjectID(20*b"h"))
Out[4]: 'hello'

Can you try if this also fixes the problem in Kubernetes?

[remram44]

This probably should be raising a ValueError 😅 but I agree that it's a separate problem. I'll try again with valid IDs.

[mitar]

I am surprised that your Docker example even works. Because Plasma store uses /dev/shm as a default to share objects on Linux, but that is not shared between containers in Docker. So your server and client do not have the same /dev/shm. So I am not sure how communication would work here?

[mitar]

Hanging on invalid ObjectID is really surprising. :-)

(It is interesting that GitHub colors the invalid ObjectID with red background?)

[pcmoritz]

I don't know why it is red :)

I agree it is not good behaviour and should give an error. I submitted a JIRA ticket here and will fix it ASAP: https://issues.apache.org/jira/browse/ARROW-1919

Thanks for finding the problem!

[mitar]

@pcmoritz: Do you understand why sharing works between containers even if /dev/shm is not shared?

[pcmoritz]

I do not understand it and have not tried it, but it seems to be possible to share memory between docker containers in general, see https://stackoverflow.com/questions/29173193/shared-memory-with-docker-containers-docker-version-1-4-1

[mitar]

It seems we would have to use --ipc argument, but example above is not. This is why I am confused. @remram44, which Docker version are you using? If you go to two docker containers and you create a file in its /dev/shm, does it appear in the other container?

Also, @pcmoritz, is /dev/shm being used by Plasma store or is memory sharing done in some other way?

[pcmoritz]

By default on linux it is using /dev/shm and on mac it is using /tmp/ and can be configured to use another location with the -d flag.

[mitar]

What does it store there? Does it store whole objects and then mmaps them? Because /tmp is not in memory on Mac.

[pcmoritz]

So we had the same suspicion and did performance experiments with this, it behaves very much like it is in-memory. We are actually unlinking the file before writing anything, so maybe that prevents flushing to disk. This strategy is the same that Google Chrome is using for it's shared memory.