Add room tracker including improved object tracker SORT #174
Conversation
|
This looks great. The centroid tracker was always just a first pass. Can you link to any documentation on the SORT tracking algorithm you implemented? |
|
Nice to hear, I figured that and it works pretty good for the current features but with this, something more advanced is required. It could also solve #141. Sort is taken from https://github.com/abewley/sort (I removed main and some related deps) there isn't much doc in there but they have a link to a paper describing the algorithm. There is another one called deep sort as well but that adds even more complexity and I'm not sure its needed here. FYI: I had a conflict while adding this pull request and I haven't tested it after that so the current changes could have some bugs, I'll test it again tomorrow. I also missed adding dockerfile changes. |
Source: https://github.com/abewley/sort Commit: c1b808481200d7997413c6e831bc59c8b08b61b8 It has better accuracy when measured in my environment. It uses a Kalman filter to correct "bad" detections and is able to recover from lost tracks to some extent. Deep sort is also an alterative but adds more complexity. This was a key to be able to get a room tracker working good without losing track of objects all the time. Input configuration exposed to user config for users to calibrate after their needs. Removed main function together with some plot dependencies from sort.py to not be needed to add unnecessary deps to frigate.
|
It is now working as it should. Added Dockerfile update as well! |
|
I am in the middle of a few other features at the moment, and I will probably look at incorporating these independently when I am done. Since SORT is GPL-3 licensed, I need to make sure I include everything that is necessary here. Also, I recently fixed a major bug with my centroid tracker that caused stationary objects to be easily lost. I will have to look through it in more detail to see if the improvement is still worth it. At a high level, the room tracker looks like an additional level of refinement on object tracking that uses additional information about where objects could possibly enter and leave the frame from. Things like, "an object cant disappear/appear in the middle of the room", and "an object can't be exiting the room unless they are withing x distance from an entry/exit." I may see if I can incorporate that into the object tracker directly instead. My guess is that outdoor cameras are not as predictable in this way. |
|
No problem, deal with it when you have time, I will probably continue improving the room tracker in the meantime. I have a few ideas. Yeah I don't really know how to handle 3pp in this repo so I just copied the file. As far as I understand regarding the license it should be fine as is. Idk if there is a need for a changelog other then git. Yeah, that is how it works at high level. I think it is wise to keep the object tracker and these kind of higher level trackers separate since they are on different operating levels. Regarding outdoor, what do you mean with not as predicable? I guess it depends on the tf model and object tracker used. |
|
I added some extra rendering and was able to calibrate my configuration even further with improved result. I think this show some performance of the algorithms and are therefore sharing it. My only problem is a mirror (were I need a larger room region) but the SORT tracker handles it most of the times (example in the pic) and if not, there is a high chance the constraints takes care of it. In the picture you can see a result of tracking me going between 4 rooms. The tracking visualization changes color on new track. Red boxes are the room points, yellow, the room region and the others are tracking of objects were the slightly bigger ones are room enter/leave points (room counter change). SORT uses Kalman filter to smoothen the tracking boxes a bit which will reduce jumps of the tf model output boxes. This is very important for the room tracking since a too large jumps in the start/end can result in some object outside room regions. For an object to be allowed to decrease/increase a room counter it must start/end inside a room region (if multiple, the closest point will be chosen). It must also meet the configurable min_history (tracked for at least x frames) and min_distance_* (traveled at least a distance of x pixels). |
|
When will this be merged into a release? |
I think the code need some more love and much have changed on master since so there are a lot of conflicts, I have not had time to work on this too much. There is no guide. Checkout this pull request and build the docker image. Check the new stuff in config.example.yaml for config help, turn on verbose and you can figure it out I think. The configuration need some trial-n-error to get it right. FYI: you might need to train a model with a custom dataset that "match" your environment for perfect accuracy. I had some problems when it became dark and the camera when into night mode or when someone walked around in a bathrobe and have now retrained (transfer learning using model used in this repo) the used model and eliminated those problems. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Still planning to revisit the object tracking algorithm. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Improved readme: added steps for GPU config in the readme
|
Object tracking was updated in 0.13, I don't believe this PR is relevant anymore and it is very out of date, closing |
This is my improvement to frigate that I am now using in my home assistant project to automate lights on/off in rooms.
It is a room tracker which tracks where people go (objects leaves and enters at different location in the camera image). I have my camera indoors in our central corridor which all rooms leads to. But I am sure this can be used in other environments as well. It requires at least 15 fps (if close range as in image below) and HW acceleration (CORAL). I've surprisingly very few false positives with this implementation. It requires some time to calibrate the configuration constants to match the environment to get the best accuracy (how close the camera is to detection and how fast and far a person walks etc.).
This also includes replacing the current object tracker with SORT (which can be added by itself if the room tracker isn't wanted). This was a key change to get the room tracker performing well since I got a lot of lost tracks with without it. E.g. when ppl were standing still for a long time or went too fast or made quick turns.
This request lack enough documentation which I can contribute with if there are any interests.
Home assistant:
Room points visualized in red boxes:
What do you think?