Exploring vitrivr-engine's Features

[MalikAhmed2]

I'm really interested in learning more about Vitrivr-engine's capabilities.

Which languages does it support?
How does the object detection module work, and which objects can it detect?

I'm also curious if it can:

• Use pose as a search query for images or videos? If so, how does it work?
• Detect and recognize faces?
  I think these features would be super cool!

[lucaro]

Hey, I moved this over to a discussion, as this is probably the better forum for such things.

I'm not quite sure I understand the question about the language. vitrivr-engine is an application and not a library. You can interact with it through its external API, which is currently available via REST. The application itself is written in Kotlin and runs on a JVM. If you want to extend it or write modules for it, doing that in Kotlin is easiest, although any language that compiles to Java byte code should work.

As for the features you mention, they are not currently available in vitrivr-engine, but they could be added in the future. Pose-based queries were available in its predecessor, Cineast, for example.

Does this answer your questions so far?

[MalikAhmed2]

Apologies for the confusion. I meant to ask which languages vitrivr-engine supports for ASR (Automatic Speech Recognition) and OCR (Optical Character Recognition), such as English, Spanish, French, ... ?

[lucaro]

I see. ASR is based on Whisper, and OCR is based on EasyOCR, so they both support a wide range of languages and symbols.

[MalikAhmed2]

Thanks for the info.
Which object detection model does vitrivr-engine use? And what kind of objects can it typically identify (e.g., people, cars, animals)?
Is it possible to swap out different models in vitrivr-engine, like using a different OCR or ASR model?

[lucaro]

vitrivr-engine is modular; you can easily swap any of the available modules. For DNN-based features, it currently primarily relies on https://github.com/vitrivr/feature-extraction-server, but other mechanisms might be added in the future.

[MalikAhmed2]

Great work, thanks!