Skip to content

Conversation

@WweiL
Copy link
Contributor

@WweiL WweiL commented Aug 4, 2023

Master PR: #42283

What changes were proposed in this pull request?

This is a followup to #42116. It addresses the following issues:

  1. When removeListener is called upon one listener, before the python process is left running, now it also get stopped.
  2. When multiple removeListener is called on the same listener, in non-connect mode, subsequent calls will be noop. But before this PR, in connect it actually throws an error, which doesn't align with existing behavior, this PR addresses it.
  3. Set the socket timeout to be None (\infty) for foreachBatch_worker and listener_worker, because there could be a long time between each microbatch. If not setting this, the socket will timeout and won't be able to process new data.
scala> Streaming query listener worker is starting with url sc://localhost:15002/;user_id=wei.liu and sessionId 886191f0-2b64-4c44-b067-de511f04b42d.
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/wei.liu/oss-spark/python/lib/pyspark.zip/pyspark/sql/connect/streaming/worker/listener_worker.py", line 95, in <module>
  File "/home/wei.liu/oss-spark/python/lib/pyspark.zip/pyspark/sql/connect/streaming/worker/listener_worker.py", line 82, in main
  File "/home/wei.liu/oss-spark/python/lib/pyspark.zip/pyspark/serializers.py", line 557, in loads
  File "/home/wei.liu/oss-spark/python/lib/pyspark.zip/pyspark/serializers.py", line 594, in read_int
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Why are the changes needed?

Necessary improvements

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual test + unit test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant