Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some unimplemented unary expressions? #1697

Open
WilliamTambellini opened this issue Aug 7, 2023 · 5 comments
Open

Some unimplemented unary expressions? #1697

WilliamTambellini opened this issue Aug 7, 2023 · 5 comments
Labels
enhancement A feature or an optimization request help wanted

Comments

@WilliamTambellini
Copy link
Contributor

WilliamTambellini commented Aug 7, 2023

Hi
Here are some unary ops currently not implemented in 1dnn (or I missed them):

https://en.wikipedia.org/wiki/File:Gjl-t(x).svg

  • cos and sin : must have for transformers positional encoding
  • reciprocal squareroot: 1 / sqrt(x)

Best

@vpirogov
Copy link
Member

@WilliamTambellini,

Isn't reciprocal square root supported by eltwise pow algorithm with alpha =1 and beta = -1/2?

Will adding sin/cos algorithms to eltwise primitive be enough to cover embeddings case?

@WilliamTambellini
Copy link
Contributor Author

WilliamTambellini commented Aug 22, 2023

Thanks

@lucasalavapena
Copy link

+1 for adding cos/sin elementwise! I would also greatly appreciate it!

@igorsafo
Copy link
Contributor

igorsafo commented Sep 5, 2023

@WilliamTambellini @lucasalavapena Thank you for the request,
regarding sin/cos: ML frameworks have their own implementation and C++ applications have access to the standard math library with cos/sin functions, so what is the benefit of having an additional implementation under oneDNN API?

@WilliamTambellini I see that positional encoding does not depend on the actual data, is it pre-computed prior to inference/training and remains constant during the process?

@lucasalavapena
Copy link

@igorsafo I am still relatively new to oneDNN, so hopefully what I am saying makes sense 😄 .

I was more thinking of having it as an op for dnnl::graph, so you can get some benefits you normally can get with the graph extension (memory layout and use of graph compiler?).

Aren't some of the existing element wise functions part of the standard math library too?

@vpirogov vpirogov self-assigned this Sep 13, 2023
@vpirogov vpirogov added enhancement A feature or an optimization request and removed question labels Oct 3, 2023
@vpirogov vpirogov removed their assignment Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A feature or an optimization request help wanted
Projects
None yet
Development

No branches or pull requests

4 participants