Issues running model on raspberrypi5 + edgetpu #50
Comments
|
@kteodorovich or @boristeo Can you please help @han-so1omon. I suspect pcie driver is not installed properly. |
|
Sure, here's the output @shashichilappagari @boristeo @kteodorovich |
|
@han-so1omon So, it appears that PySDK is able to recognize the EdgeTPU. To ensure that driver is properly installed and is working, we made a small test script that does not depend on pysdk (this will allow us to diagnose if problem is with pysdk or the basic setup). Please see if you can run the following code without errors: import tflite_runtime.interpreter as tflite
from PIL import Image
import numpy as np
import os
print('Downloading test model and test image')
os.system('wget -nc https://raw.githubusercontent.com/google-coral/test_data/master/mobilenet_v1_1.0_224_quant_edgetpu.tflite')
os.system('wget -nc https://github.com/DeGirum/PySDKExamples/blob/main/images/Cat.jpg?raw=true -O Cat.jpg')
print('Running...')
m = tflite.Interpreter('mobilenet_v1_1.0_224_quant_edgetpu.tflite', experimental_delegates=[tflite.load_delegate('libedgetpu.so.1')])
img = Image.open('Cat.jpg')
m.allocate_tensors()
n, h, w, c = m.get_input_details()[0]['shape']
m.set_tensor(m.get_input_details()[0]['index'], np.array(img.resize((h, w)))[np.newaxis,...])
m.invoke()
out = m.get_tensor(m.get_output_details()[0]['index']).flatten()
assert np.argmax(out) == 288, 'Wrong output result'
assert out[np.argmax(out)] == 83, 'Wrong output probability'
print('OK') |
|
Here is the output: |
|
@han-so1omon Thanks for checking this. So, it does appear that edgetpu is properly functioning. Can you please try other models in the zoo to make sure that it is not a problem specific to the model? |
|
@shashichilappagari I have tried with another model and it works fine. Using the test image from the project posenet from google, the yolov8n_relu6_coco--640x640_quant_tflite_edgetpu_1 model is able to classify the image |
|
@han-so1omon Is there a typo in the above code? Did you load the pose model or the coco detection model? Assuming you loaded the detection model and that it is working, it is partially good news as this shows that pysdk is working with local edgetpu. It now seems that the problem is specific to the model. We will upload models compiled at lower resolution to see if they resolve the issue. Thanks for your patience and quick responses. |
|
Some more context, I'm using feranick's edgetpu runtime and python libraries, as recommended in the ultralytics setup instructions, as these runtimes are kept up to date since google abandoned the coral project. I'm also using the default python 3.9 on raspberry pi os bookworm. Should I perhaps try running from the docker container? If so, do you have examples of that? |
|
Thank you, I am trying to get this issue resolved this week, and I appreciate your responsiveness quite a lot |
|
Yes, I corrected the typo. I loaded the coco detection model, and it seems to work fine. I commented out loading the coco pose model, as it was throwing the error |
|
@han-so1omon Since other models are working, the issue seems to be specific to the pose model. As you can see from our cloud platform, all models in the edgetpu model zoo are working properly on our cloud farm machines which have the google edge tpu pcie module. As I mentioned before, we will compile pose models at lower resolution and see if the problem goes away. Another option is to use google's mobilenet posenet model. |
|
@shashichilappagari Do you have instructions on how you've setup your google edge tpu pcie modules? Additionally, do you have the mobilenet posenet in your model zoos for edgetpu? |
|
@han-so1omon We will add it and let you know. Please give us a couple of hours to get lower resolution pose models. We will also share our setup guide with you. |
|
Ok, thank you |
|
@han-so1omon From the error message you are seeing, there could be some race condition in the code. We are unable to replicate it on our side but we have some ideas to test. Before I explain the ideas, I want to mention that you do not have to download the models to run locally. You can connect to cloud zoo and pysdk will automatically download the models. This will make your code simpler. Once you have finished debugging, you can of course switch to local zoo in case you want offline deployment. Your code should look like below: import cv2
import degirum as dg
image = cv2.imread("./test-posenet.jpg")
zoo = dg.connect(dg.LOCAL, "https://cs.degirum.com/degirum/edgetpu", <your token>)
model = zoo.load_model("yolov8n_relu6_coco--640x640_quant_tflite_edgetpu_1")
print(model)
result = model.predict(image)
result_image = result.image_overlay
cv2.imwrite("./test-posenet-degirum.jpg", result_image)With the above code you can just change model name every time you want to experiment with a different model. Now to rule out the race condition that could be killing your python, we can try the following. PySDK supports three types of inference: degirum serverYou will see a message saying that degirum server started. Then run the following code: import cv2
import degirum as dg
image = cv2.imread("./test-posenet.jpg")
zoo = dg.connect('localhost', "https://cs.degirum.com/degirum/edgetpu", <your token>)
model = zoo.load_model("yolov8n_relu6_coco--640x640_quant_tflite_edgetpu_1")
print(model)
result = model.predict(image)
result_image = result.image_overlay
cv2.imwrite("./test-posenet-degirum.jpg", result_image)Note that we changed Please try this code and see if it works. We also added Hope that this helps. |
|
Ok. What do you think the race condition is, and is there a way to perform a wait to prevent it? |
|
@han-so1omon At this point we are not sure as it could be system dependent. That is why we want you to try the |
|
@han-so1omon We also added the |
|
@shashichilappagari I will try all later in the day. Can you share how you've setup the edgetpu modules? |
|
@kteodorovich can you share our user guide for edge tpu with @han-so1omon? |
|
@han-so1omon Hello! Our guide for USB Edge TPU is available here. You might be past all these steps already, given that you got the detection model to work. By the way, the base Ultralytics library will only recognize a model for Edge TPU if the filename ends with |
|
@shashichilappagari @kteodorovich Thank you! It looks like it was indeed a race condition with the local degirum setup. All of the models appear to work correctly with the localhost-based ai server as recommended. Do you have recommendations on how to setup a pose tracking algorithm on top of the pose prediction algorithm? |
|
@han-so1omon We are glad to hear that |
|
@han-so1omon Here is an example of how you can add tracking on top of a model: https://github.com/DeGirum/PySDKExamples/blob/main/examples/specialized/multi_object_tracking_video_file.ipynb |
|
@vlad-nn Python post-processor indeed seems to have a race condition when using |
|
@shashichilappagari Does your pose algorithm from yolo_pose support landmarks? |
|
@han-so1omon Do you mean if the tracking algorithm tracks landmarks? |
|
@shashichilappagari basically is there a way to present it as a skeleton with each part of the body denoted. Like 'right ear', 'right forearm', etc |
|
@han-so1omon so you want the output of prediction to have a label for each keypoint? |
|
Basically, yes. I would like to know what part of the body the keypoint comes from |
It selects top_k objects, not keypoints. You see, you need object tracking for only one purpose: how to match a person detected on one frame with the same person, detected on another frame, knowing only person bboxes. This is the task for object tracker. Then, in some cases, you need to track one (or few) particular people from frame to frame. This is the task for object selector.
Regarding FIRLilterFP. It is simple FIR lopass filter, which can be used for many purposes, but in that example it is used to suppress high-frequency noise in bbox coordinates (smooth them in time). When you create filter object, you specify normalized cutoff frequency (as a fraction of sampling rate, so 0.5 is Nyquist frequency), number of taps aka filter length, and dimension of the input signal (size of the input vector): |
|
@vlad-nn Should it not be that object tracking is frame-to-frame for the prescribed object, and object selection is grouping of multiple objects, perhaps for more semantic analysis? Got it, so I will possibly try to apply the low-pass filter on a selection of landmarks that semantically define limb and body region movement in projected 2d space |
Yes, like in that example above, when we first select the largest hand on the screen and then track only that hand ignoring all other hands. |
|
@vlad-nn Seems like it is running ok when the person is in the first frame, but otherwise it is having issues detecting a person. Can run about 10 fps. Full script: import os
import threading
import multiprocessing
import subprocess
import time
import cv2
import degirum as dg
import degirum_tools
def run_degirum_server():
# Start the degirum server
subprocess.run(["degirum", "server"])
def run_prediction_code():
print('Connecting to Degirum...')
zoo = dg.connect('localhost', "https://cs.degirum.com/degirum/edgetpu", os.getenv('DEGIRUM_PRIVATE_KEY'))
#model = zoo.load_model("yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1")
model = zoo.load_model("yolov8n_silu_coco_pose--512x512_quant_tflite_edgetpu_1")
print('Creating object tracker...')
# create object tracker
tracker = degirum_tools.ObjectTracker(
track_thresh=0.35,
track_buffer=100,
match_thresh=0.9999,
trail_depth=20,
anchor_point=degirum_tools.AnchorPoint.BOTTOM_CENTER,
)
# create object selector to track only one hand
selector = degirum_tools.ObjectSelector(
top_k=1,
use_tracking=True,
selection_strategy=degirum_tools.ObjectSelectionStrategies.LARGEST_AREA,
)
# Attaching analyzers
degirum_tools.attach_analyzers(model, [tracker, selector])
print('Predicting stream...')
next_res = degirum_tools.predict_stream(
model, 'rtsp://0.0.0.0:8554/stream1', analyzers=[tracker, selector], fps=11,
)
'''
next_res = degirum_tools.predict_stream(
model, 0, analyzers=[tracker], fps=15,
)
'''
start_time = time.time()
loop_count = 0
try:
while True:
res = next(next_res)
print(res.results)
loop_count += 1
elapsed_time = time.time() - start_time
loops_per_second = loop_count / elapsed_time
if loop_count % 10 == 0:
print(f'Loops per second: {loops_per_second:.2f}')
except KeyboardInterrupt:
print('Stopping prediction stream...')
if __name__ == "__main__":
# Create and start the thread for the Degirum server
server_thread = threading.Thread(target=run_degirum_server)
server_thread.start()
time.sleep(3)
# Create and start the process for the prediction code
prediction_process = multiprocessing.Process(target=run_prediction_code)
prediction_process.start()
try:
# Wait for the process to complete
prediction_process.join()
except KeyboardInterrupt:
# Handle keyboard interrupt
prediction_process.terminate()
prediction_process.join()
print('Processes terminated.')
# Ensure the server thread is also stopped
if server_thread.is_alive():
print('Stopping Degirum server...')
# This assumes the server can be stopped by interrupting the subprocess
# If not, additional logic might be needed to stop the server gracefully |
@han-so1omon Can you clarify what this means? Does it mean that if the video starts with no person in frame, it has trouble detecting a person when they enter the frame in subsequent frames? Any sample video you can share on which we can run and reproduce the issue? |
|
@shashichilappagari Of course. It means in my case that the results array from the script in the previous comment is empty when a person does not start in frame. When they start in frame, it looks pretty good No person starting in frame: Person starting in frame: |
|
@han-so1omon we will take a look to see if this has anything to do with tracking. So the symptom you are seeing is that if the first frame does not have a person, the predict stream is not detecting a person even in all subsequent frames which actually have a person. Is that a correct assessment? Can you disable tracking the tracking to see if the behavior is different? |
|
It appears that the combination of the tracker and the selector is the issue. The tracker can yield landmark results when people are walking in and out with a delay (~2s). The selector can only yield landmark results when the person stays in the frame. If a person is not in the first frame with the selector active, as it seems then the results are empty |
|
@han-so1omon Thanks for this additional information. We will take a look and keep you posted. |
|
For results from predict_stream, how to get the timestamp of a result? I see i can get |
|
@han-so1omon This is an interesting question. Does the original stream provide timestamp info? In PySDK, we support attaching |
|
@shashichilappagari I do not see a parameter called Edit. Will something like this work to create timestamps? |
|
@han-so1omon, the problem in your code is that you are attaching analyzers twice:
Please remove either one of them, and the code will work correctly. |
@han-so1omon , Example: |
|
@vlad-nn Got it, thank you I will try. Separate question, how do I get access to |
|
@han-so1omon you should now be able to access raspberry pi models. Please note that these are models for CPU. |
|
@shashichilappagari I am aware. This is for an art piece, but as of saturday it is already in the gallery. I am fixing some things, though, so I am testing on the raspberry pi cpu |
|
@han-so1omon sounds like an interesting application. Please let us know if there is something online on which we can see your application. |
|
@shashichilappagari Will share once it's fixed up |
|
@vlad-nn I may have it working with near real-time results at 12 fps using Picamera2 def video_source(fps = None):
picam2 = Picamera2()
picam2.configure()
picam2.start()
print('Establishing video source')
if fps:
# Throttle if camera feed
minimum_elapsed_time = 1.0 / fps
prev_time = time.time()
i = 0
while True:
frame = picam2.capture_array()
frame = frame[:,:,:3]
#frame = np.zeros((640,480,3))
i += 1
frame_info = {"iteration": i, "timestamp": time.ctime()}
if fps:
curr_time = time.time()
elapsed_time = curr_time - prev_time
if elapsed_time < minimum_elapsed_time:
continue
prev_time = curr_time - (elapsed_time - minimum_elapsed_time)
yield (frame, frame_info)
else:
yield (frame, frame_info)
def predict_stream(
model: dg.model.Model,
*,
fps = None,
analyzers = None,
):
print('In stream prediction function')
analyzing_postprocessor = degirum_tools.inference_support._create_analyzing_postprocessor_class(analyzers)
for res in model.predict_batch(video_source(fps=fps)):
if analyzers is not None:
yield analyzing_postprocessor(result=res)
else:
yield res
def run_prediction_code():
print('Connecting to Degirum...')
#zoo = dg.connect('localhost', "https://cs.degirum.com/degirum/raspberry_pi_models", os.getenv('DEGIRUM_PRIVATE_KEY'))
#model = zoo.load_model("yolov8n_silu_coco_pose--640x640_quant_tflite_cpu_1")
zoo = dg.connect('localhost', "https://cs.degirum.com/degirum/edgetpu", os.getenv('DEGIRUM_PRIVATE_KEY'))
model = zoo.load_model("yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1")
#model = zoo.load_model("yolov8n_silu_coco_pose--512x512_quant_tflite_edgetpu_1")
print('Creating object tracker...')
# create object tracker
tracker = degirum_tools.ObjectTracker(
track_thresh=0.35,
track_buffer=100,
match_thresh=0.9999,
trail_depth=20,
anchor_point=degirum_tools.AnchorPoint.BOTTOM_CENTER,
)
# create object selector to track only one hand
selector = degirum_tools.ObjectSelector(
top_k=1,
use_tracking=True,
selection_strategy=degirum_tools.ObjectSelectionStrategies.LARGEST_AREA,
)
# Attaching analyzers
analyzers = [tracker]
#analyzers = [tracker, selector]
#degirum_tools.attach_analyzers(model, analyzers)
print('Predicting stream...')
next_res = predict_stream(
model, fps=8, analyzers=analyzers
)
'''
next_res = degirum_tools.predict_stream(
model, 0, analyzers=[tracker], fps=15,
)
'''
start_time = time.time()
loop_count = 0
fake_timestamp = 0.125
increment_fake_time = lambda t: t + 0.125
try:
while True:
res = next(next_res)
#TODO is this timestamp correct?
timestamp = time.time()
print(res)
if res.results:
print(f'Prediction results: {res.results}')
for r in res.results:
fake_timestamp = increment_fake_time(fake_timestamp)
add_analysis(r, fake_timestamp)
loop_count += 1
elapsed_time = time.time() - start_time
loops_per_second = loop_count / elapsed_time
if loop_count % 10 == 0:
print(f'Loops per second: {loops_per_second:.2f}')
except KeyboardInterrupt:
print('Stopping prediction stream...')
if __name__ == "__main__":
'''
# Create and start the thread for the camera processes
camera_server_thread = threading.Thread(target=run_camera_stream)
camera_server_thread.start()
time.sleep(4)
# Create and start the thread for the Degirum server
degirum_server_thread = threading.Thread(target=run_degirum_server)
degirum_server_thread.start()
time.sleep(3)
'''
_camera = Picamera2()
_camera.configure()
_camera.start()
# Create and start the process for the prediction code
prediction_process = multiprocessing.Process(target=run_prediction_code)
prediction_process.start() |
|
I am still having the issue sometimes where the inference server hangs and then exits. I guess this is a result of the buffer building up in the server and crashing (#50 (comment)). Any way to clear the buffer? I only want the most recent frame anyway |
|
@han-so1omon The inference server hang: does it happen after the application runs for a while? or is the time to hang random? From the code you shared, it appears that the camera captures the latest frame. |
|
@shashichilappagari It hangs usually after like 8 seconds if it will happen. Sometimes it happens much later as well. Yes, I'm confused about it as well. Here's the timeout error I get: |
|
@kteodorovich Can you see if we can replicate this behavior on our side? |
|
@shashichilappagari Here is my service definition: |
|
For reference, fps=4 is close to stable for me (~10% chance of error versus 90%). Maybe a race condition or a cache filling up |
|
@han-so1omon we will see if we can replicate this on our side and keep you posted. FYI, pysdk itself has been tested on webcamera streams running for days at a stretch. So, the most likely scenario is something specific to picamera setup that we are unaware of. |
|
@shashichilappagari That seems to make some sense, although why would it yield a timeout error from Degirum server? Is it receiving some array or other message that it is not expecting? |
|
@han-so1omon I haven't been able to replicate the same timeout error you're getting, do you think there's anything in your setup that could contribute to extra delays? I modified your code snippet to show a video preview (instead of printing), and this seems to be running smoothly at ~15fps. Could you try running this and let us know if the server still crashes? import degirum as dg
import degirum_tools
from picamera2 import Picamera2
import time
def picam2_stream(fps=None):
with Picamera2() as cam:
config = cam.create_preview_configuration(main={"format": "RGB888"}) # capture frames in RGB
cam.configure(config)
cam.start()
delay = 0 if fps is None else 1 / fps
last_time = time.time()
i = 0
while True:
t = time.time()
if fps is None or t - last_time >= delay:
last_time = t
frame = cam.capture_array()
frame_info = {"iteration": i, "timestamp": time.ctime()}
i += 1
yield (frame, frame_info)
if __name__ == '__main__':
# create object tracker
tracker = degirum_tools.ObjectTracker(
track_thresh=0.35,
track_buffer=100,
match_thresh=0.9999,
trail_depth=20,
anchor_point=degirum_tools.AnchorPoint.BOTTOM_CENTER,
)
# create object selector to track only one person
selector = degirum_tools.ObjectSelector(
top_k=1,
use_tracking=True,
selection_strategy=degirum_tools.ObjectSelectionStrategies.LARGEST_AREA,
)
zoo = dg.connect('localhost', "https://cs.degirum.com/degirum/edgetpu", degirum_tools.get_token())
model = zoo.load_model("yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1")
model = degirum_tools.attach_analyzers(model, [tracker, selector])
# stream with tracker
with degirum_tools.Display("Video") as display:
for res in model.predict_batch(picam2_stream()):
display.show(res.image_overlay) |
I have some issues running the degirum models on my raspberrypi5 + edgetpu environment with raspberrypi os 12 bookworm. Moving from this issue ultralytics/ultralytics#1185. @shashichilappagari Can you provide some assistance?
First step
Try to run with degirum pysdk
Try to run with base yolo ultralytics library
I can see that the edgetpu is connected. Although I am not sure that it is being used
The text was updated successfully, but these errors were encountered: