Queries with and/or operators not working (on Elixir 1.6)

[sheharyarn]

On Elixir 1.6.0, where statements with and or or operators cause compilation errors. The where macro works fine when they are not used, but when they are, for some reason elixir expands the match fields to functions (which don't exist), throwing a compilation error.

Environment Details:

• Elixir 1.6.0
• Erlang 20.1
• Amnesia 0.2.7 (Exquisite 0.1.7)
• MacOS High Sierra 10.13.2

Reproducing the Issue:

A simple elixir application with amnesia set up, along with this DB/Table. This would not compile on Elixir 1.6.0, but if you remove the find/2 method, it will.

use Amnesia

defdatabase DB do
  deftable Post, [:user, :status, :content] do

    # Insert a Post
    def insert(user, status, content) do
      Amnesia.transaction do
        write(%DB.Post{user: user, status: status, content: content})
      end
    end


    # Find all posts by user
    def find(p_user) do
      Amnesia.transaction do
        where(user == p_user)
        |> Amnesia.Selection.values
      end
    end


    # Find all posts by user and status
    def find(p_user, p_status) do
      Amnesia.transaction do
        where(user == p_user and status == p_status)
        |> Amnesia.Selection.values
      end
    end

  end
end

Error:

Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Compiling 1 file (.ex)
warning: variable "user" does not exist and is being expanded to "user()", please use parentheses to remove the ambiguity or change the variable name
  lib/amnesia.ex:26

warning: variable "status" does not exist and is being expanded to "status()", please use parentheses to remove the ambiguity or change the variable name
  lib/amnesia.ex:26


== Compilation error in file lib/amnesia.ex ==
** (CompileError) lib/amnesia.ex:26: undefined function status/0
    (stdlib) lists.erl:1338: :lists.foreach/2
    lib/amnesia.ex:4: (module)
    (elixir) lib/kernel/parallel_compiler.ex:198: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

---

Everything else works as expected. This is a serious bug which also affects one of my projects; Que. Hoping this gets resolved soon!

[suprnova32]

I was just about to report this. In order to work around this, I had to modify my fork of Que and add some dirty hacks: https://github.com/AlloyCI/que/commit/593dc65c3405c0f5381a091ce2703c3401618485

AlloyCI really depends on this, so hoping this gets resolved soon, I don't want to rely on a dirty hack.

[meh]

This sounds like a bug in Elixir to me, looking into it.

[suprnova32]

@meh while investigating the errors, it seemed to me that they might be coming from a bug in expanding the and and or macros. I haven't checked if there are any differences between Elixir 1.5 and 1.6, but it would seem like a good place to start.

[meh]

@supernova32 yeah, my suspicion is something changed slightly in the AST between 1.5 and 1.6, I just need to find the time to look into that, unless someone else beats me to it, this sounds like it shouldn't have happened.

[josevalim]

The issue is that exquisite is relying on a macro to expand to a certain format which is a no-no since Elixir can change what macro expands to at any moment.

The way Ecto approaches this is by traversing the known nodes before expanding them. So for example, if you know how to handle the {:and, _, [x, y]} AST, you handle that first, and then you have a catch all clause that try to expand the AST once and try again. If the AST does not expand, then you raise. Here is this particular bit in Ecto. This way you are not coupled to how Elixir expands some macros (and you can even add your own nodes that does not exist in Elixir).

[meh]

@josevalim actually now that I look into it a little more, it seems to have nothing to do with the AST.

Using the following as example:

    from = {{2013,1,1},{1,1,1}}
    to   = {{2013,2,2},{1,1,1}}

    s = Exquisite.match { a, b },
      where:  a >= from and b <= to,
      select: 2

Elixir 1.5:

{{:a, [line: 27], nil}, {:b, [line: 27], nil}}
[where: {:and, [line: 28],
  [{:>=, [line: 28], [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
   {:<=, [line: 28], [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}]},
 select: 2]

[{:{}, [],
  [{:"$1", :"$2"},
   [{:{}, [],
     [:andalso,
      {:{}, [],
       [:>=, :"$1",
        {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, []
         [{:from, [line: 28], nil}]}]},
      {:{}, [],
       [:"=<", :"$2",
        {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, []
         [{:to, [line: 28], nil}]}]}]}], [2]]}]

Elixir 1.6:

{{:a, [line: 27], nil}, {:b, [line: 27], nil}}
[
  where: {:and, [line: 28],
   [
     {:>=, [line: 28], [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
     {:<=, [line: 28], [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}
   ]},
  select: 2
]

[
  {:{}, [],
   [
     {:"$1", :"$2"},
     [
       {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, [],
        [
          {:case, [optimize_boolean: true],
           [
             {:>=, [line: 28],
              [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
             [
               do: [
                 {:->, [],
                  [
                    [true],
                    {:<=, [line: 28],
                     [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}
                  ]},
                 {:->, [], [[false], false]},
                 {:->, [],
                  [
                    [{:other, [counter: -576460752303423484], Kernel}],
                    {{:., [], [:erlang, :error]}, [],
                     [
                       {{:., [],
                         [
                           {:__aliases__,
                            [alias: false, counter: -576460752303423484],
                            [:BadBooleanError]},
                           :exception
                         ]}, [],
                        [
                          [
                            operator: :and,
                            term: {:other, [counter: -576460752303423484],
                             Kernel}
                          ]
                        ]}
                     ]}
                  ]}
               ]
             ]
           ]}
        ]}
     ],
     [2]
   ]}
]

I don't see any difference in the input (unless I'm blind) so it must be some change in macro expansion, did anything in Macro.escape change?

[ourway]

Any Updates on this?

[sheharyarn]

Bump. Would really like to see this issue resolved soon. @meh @josevalim Is there anything I can do to get this fixed? What do you think the problem is here?

[astutecat]

I also have this issue occurring for me, sadly - same symptoms on elixir 1.6

e: also happy to help where I can

[jonathanleang]

FIY,
change

•        MRoom.where(lobby_name == "rooms" and topic == key)
  

to

•       MRoom.match(lobby_name: "rooms", topic: key)
  

works for me

[DerTim1]

@jonathanleang Do you have a work around for where-queries with or and != in it? Or combinations of or and and?

[dmilith]

Exactly. Before I had:

hist = Amnesia.Selection.values History.where timestamp != -1

now.. I can't do it. I mean literally I can't get Amnesia to work on Elixir 1.7. It was just fine on 1.6. Amount of failures I got caused by CALLER or other stuff like this is staggering (exquisite as an example). I have to stay on Elixir 1.6 cause this is madness that I'm literally unable to fix my project now.

With amount of incompatibilities Elixir 1.7 is causing.. I don't understand why it's not Elixir 2.0 instead - since there are plenty of MAJOR changes in language.

[noizu]

I'm able to run Amnesia with and/or queries on Elixir 1.7.4, Erlang 21 using my patched version of exquisite.

Patched Exquisite to handle and/or queries.
Diff of my exquisite branch and upstream

In Deps:
'''
{:exquisite, git: "https://github.com/noizu/exquisite.git", ref: "7a4a03d", override: true},
'''

[sheharyarn]

I actually just ended up writing my own wrapper around mnesia, keeping it much closer to the original api.