Closing PyTorchEngine gracefully

[AvisP]

I have an inference running on a server with an endpoint. The endpoint receives the prompt, process and closes. On closing I am getting the following error. Is there a way to close the Pytorchengine without creating this error?

Task was destroyed but it is pending!
task: <Task pending name='MainLoopBackground' coro=<Engine._async_loop_background() running at /lmdeploy/lmdeploy/pytorch/engine/engine.py:906> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[Engine._add_loop_tasks_done_callback.<locals>.__task_callback() at /lmdeploy/lmdeploy/pytorch/engine/engine.py:971]>
Exception ignored in: <coroutine object Engine._async_loop_background at 0x7ebfbf866650>
Traceback (most recent call last):
  File "/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 906, in _async_loop_background
    is_prefill, scheduler_output = await in_que.get()
                                   ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/queues.py", line 160, in get
    getter.cancel()  # Just in case getter is not done yet.
    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
    self._check_closed()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    raise RuntimeError('Event loop is closed')

[grimoire]

Have you try the latest version? We have add a Task-done callback to catch the exception. Try ignore the exception here.

[AvisP]

I have updated to 0.7.0.post2 but still the same issue happens. Can you tell me how to terminate the task if the following is my example code

class DSR()

@enter()
def load_model(self):
    from lmdeploy import pipeline, PytorchEngineConfig, ChatTemplateConfig
    pipe = pipeline(model_path = model_dir, 
                            chat_template_config = ChatTemplateConfig(model_name='deepseek'),
                            backend_config=PytorchEngineConfig(tp=GPU_COUNT))
    self.pipe = pipe

@method()
def Inference(self, prompt):
    gen_config=GenerationConfig(
                max_new_tokens=max_new_tokens,
                top_p=top_p,
                top_k=top_k,
                temperature=temperature,
                do_sample=do_sample,
                random_seed=manual_seed
                )
            
            outputs = self.pipe(prompt, gen_config=gen_config)
    
            return outputs

@exit()
def close(self):
     self.close()    ### How to close here?

If you can let me know if there is a way to close the connection in the exit() or how to close the pipe, I think it will solve the issue.

[grimoire]

PytorchEngine would start 4 asyncio tasks. Named EngineMainLoop, MainLoopBackground, MainLoopPreprocessMessage and MainLoopResponse. We should have bind a callback to handle the exception, it might not work in your case.

lmdeploy/lmdeploy/pytorch/engine/engine.py

Line 892 in 637435f

def _add_loop_tasks_done_callback(tasks: List[asyncio.Task]):

Try cancel them before closing the event loop. Note that canceling tasks may take some time.