Experimental project structure (#773)

* Experimental project structure

This commit represents a new structure for the experimental project, and
a path forward. With these changes the project now has:

  * A build option enabling the experimental server
  * Per-message routing for the experimental server. If enabled, it can
    either share messages with the existing server or take them over.
    Presently, the find references and formatting providers are
    implemented and "exclusive", meaning that they're handled solely by
    the experimental server
  * A consistent interface for building providers.
  * A consistent way to convert lsp messages into data structures and
    back again. This conversion is handled automatically for providers.
  * A genserver-like interface for providers to implement
  * Data structures representing LSP messages that are simple to define
    and build.
  * Fast and efficient conversion between utf8 and utf16.
  * A separation between what a provider does and how it responds to
    messages. This allows the work that underpins providers to be tested
    independently from the language server.

* Add underscore code action

Created a code action that prepends an underscore to unused variable
names.

* Notifications can be sent from the server

* Properly handled spacing

* Enforced required keys for jsonrpc messages

* removed unused variable

* Committed to pipeline

* Added tests that check to ensure comments are preserved

* Code modification framework

First attempt at a standard interface for code modification. Code mod
modules take the original text, the ast of the original text and
arguments that they specify. They return a list of code edits or an
error.

* Simplified diff, change name of code action functions from appy to text_edits

* Fixed off-by-one error that was vexing code unit conversions.

The problem was that the character positions are _before_ the reported
unit, so the 0th code unit is before the start of the line and the 1st
code unit is the first character. The prior code added one to
character counts to smooth this out, but you can't do that, because
you could end up indexing into the middle of a multibyte character.

* The code action needs to fix up the line numbers

Code mods deal with snippets of code that need to have their line
numbers fixed up by the code actions.

* Fixed type spec

The AST type is very complicated, and dialyzer was telling us I got it wrong.

* Made type aliases a thing

While working on the automatic protocol generators, it became clear
that type aliases needed to be their own thing, as they operate quite
differently from the other defined things in the jsonrpc
protocol. Since they're just aliases, it makes sense to keep their
definitions on hand and then spit them out when other things make use
of them during encode and decode.

This did require going back to encoding and ensuring all the encode
functions return OK tuples.

* Fixed unit tests

When patches are unapplied, getting the beam file returned an empty
path charlist, which dialyzer assumed was a real file name due to a
weak assumption, which caused unit tests to fail. This was remedied by
checking for a non-empty charlist, which allows tests to succeed.

Also made patch a test only dependency for .formatter.exs, as this was
causing formatters to fail.

* removed unused module attribute

* Added sourceror to ease ast to string conversion

Under 1.12, Macro.to_string proudces wonky output, making `def` calls
look like function calls by adding needless parenthesis. These
parenthesis throw off the diff algorithm, and caused an off-by-one
error in the code mod.
Sourceror has backported the newer code generation so that it's
compatible all the way back to 1.10, and produces the correct output.

* Added patch as a dev dependency

Patch's assertions will fail in CI due to `mix format
--check-formatted` running in dev. Importing patch's deps in test will
fix this.

* Run check formatted in test so patch assertions work

* Fixed dialyzer errors

* Encapsulated sourceror

.github/workflows/ci.yml

@@ -123,6 +123,6 @@ jobs:
           mix deps.get
       - name: Restore timestamps to prevent unnecessary recompilation
         run: IFS=$'\n'; for f in $(git ls-files); do touch -d "$(git log -n 1 --pretty='%cI' -- $f)" "$f"; done
-      - run: mix format --check-formatted
-      - run: cd apps/language_server && mix format --check-formatted
+      - run: MIX_ENV=test mix format --check-formatted
+      - run: cd apps/language_server && MIX_ENV=test mix format --check-formatted
       - run: mix dialyzer_vendored

apps/language_server/.formatter.exs

@@ -3,15 +3,25 @@ impossible_to_format = [
   "test/fixtures/project_with_tests/test/error_test.exs"
 ]
 
+deps =
+  if Mix.env() == :test do
+    [:patch]
+  else
+    []
+  end
+
 proto_dsl = [
+  defalias: 1,
   defenum: 1,
   defnotification: 2,
-  defrequest: 2,
+  defnotification: 3,
+  defrequest: 3,
   defresponse: 1,
   deftype: 1
 ]
 
 [
+  import_deps: deps,
   export: [
     locals_without_parens: proto_dsl
   ],

apps/language_server/lib/language_server.ex

@@ -7,23 +7,21 @@ defmodule ElixirLS.LanguageServer do
   alias ElixirLS.LanguageServer
   alias ElixirLS.LanguageServer.Experimental
 
-  # @maybe_experimental_server [Experimental.Server]
-  @maybe_experimental_server []
-
   @impl Application
   def start(_type, _args) do
-    children = [
-      Experimental.SourceFile.Store,
-      {ElixirLS.LanguageServer.Server, ElixirLS.LanguageServer.Server},
-      Experimental.Server,
-      {ElixirLS.LanguageServer.PacketRouter,
-       [LanguageServer.Server] ++ @maybe_experimental_server},
-      {ElixirLS.LanguageServer.JsonRpc,
-       name: ElixirLS.LanguageServer.JsonRpc, language_server: LanguageServer.PacketRouter},
-      {ElixirLS.LanguageServer.Providers.WorkspaceSymbols, []},
-      {ElixirLS.LanguageServer.Tracer, []},
-      {ElixirLS.LanguageServer.ExUnitTestTracer, []}
-    ]
+    Experimental.LanguageServer.persist_enabled_state()
+
+    children =
+      [
+        maybe_experimental_supervisor(),
+        {ElixirLS.LanguageServer.Server, ElixirLS.LanguageServer.Server},
+        maybe_packet_router(),
+        jsonrpc(),
+        {ElixirLS.LanguageServer.Providers.WorkspaceSymbols, []},
+        {ElixirLS.LanguageServer.Tracer, []},
+        {ElixirLS.LanguageServer.ExUnitTestTracer, []}
+      ]
+      |> Enum.reject(&is_nil/1)
 
     opts = [strategy: :one_for_one, name: LanguageServer.Supervisor, max_restarts: 0]
     Supervisor.start_link(children, opts)
@@ -42,4 +40,25 @@ defmodule ElixirLS.LanguageServer do
 
     :ok
   end
+
+  defp maybe_experimental_supervisor do
+    if Experimental.LanguageServer.enabled?() do
+      Experimental.Supervisor
+    end
+  end
+
+  defp maybe_packet_router do
+    if Experimental.LanguageServer.enabled?() do
+      {ElixirLS.LanguageServer.PacketRouter, [LanguageServer.Server, Experimental.Server]}
+    end
+  end
+
+  defp jsonrpc do
+    if Experimental.LanguageServer.enabled?() do
+      {ElixirLS.LanguageServer.JsonRpc,
+       name: ElixirLS.LanguageServer.JsonRpc, language_server: LanguageServer.PacketRouter}
+    else
+      {ElixirLS.LanguageServer.JsonRpc, name: ElixirLS.LanguageServer.JsonRpc}
+    end
+  end
 end

apps/language_server/lib/language_server/dialyzer/utils.ex

@@ -4,13 +4,14 @@ defmodule ElixirLS.LanguageServer.Dialyzer.Utils do
   @spec dialyzable?(module()) :: boolean()
   def dialyzable?(module) do
     file = get_beam_file(module)
+
     is_list(file) and match?({:ok, _}, :dialyzer_utils.get_core_from_beam(file))
   end
 
   @spec get_beam_file(module()) :: charlist() | :preloaded | :non_existing | :cover_compiled
   def get_beam_file(module) do
     case :code.which(module) do
-      file when is_list(file) ->
+      [_ | _] = file ->
         file
 
       other ->

apps/language_server/lib/language_server/experimental/code_mod/ast.ex

@@ -0,0 +1,28 @@
+defmodule ElixirLS.LanguageServer.Experimental.CodeMod.Ast do
+  alias ElixirLS.LanguageServer.Experimental.SourceFile
+
+  @type source :: SourceFile.t() | String.t()
+  @type t ::
+          atom()
+          | binary()
+          | [any()]
+          | number()
+          | {any(), any()}
+          | {atom() | {any(), [any()], atom() | [any()]}, Keyword.t(), atom() | [any()]}
+
+  @spec from(source) :: t
+  def from(%SourceFile{} = source_file) do
+    source_file
+    |> SourceFile.to_string()
+    |> from()
+  end
+
+  def from(s) when is_binary(s) do
+    ElixirSense.string_to_quoted(s, 1, 6, token_metadata: true)
+  end
+
+  @spec to_string(t()) :: String.t()
+  def to_string(ast) do
+    Sourceror.to_string(ast)
+  end
+end

apps/language_server/lib/language_server/experimental/code_mod/diff.ex

@@ -0,0 +1,106 @@
+defmodule ElixirLS.LanguageServer.Experimental.CodeMod.Diff do
+  alias ElixirLS.LanguageServer.Experimental.CodeUnit
+  alias ElixirLS.LanguageServer.Experimental.Protocol.Types.Position
+  alias ElixirLS.LanguageServer.Experimental.Protocol.Types.Range
+  alias ElixirLS.LanguageServer.Experimental.Protocol.Types.TextEdit
+
+  @spec diff(String.t(), String.t()) :: [TextEdit.t()]
+  def diff(source, dest) do
+    source
+    |> String.myers_difference(dest)
+    |> to_text_edits()
+  end
+
+  defp to_text_edits(difference) do
+    {_, {current_line, prev_lines}} =
+      Enum.reduce(difference, {{0, 0}, {[], []}}, fn
+        {diff_type, diff_string}, {position, edits} ->
+          apply_diff(diff_type, position, diff_string, edits)
+      end)
+
+    [current_line | prev_lines]
+    |> Enum.flat_map(fn line_edits ->
+      line_edits
+      |> Enum.reduce([], &collapse/2)
+      |> Enum.reverse()
+    end)
+  end
+
+  # This collapses a delete and an an insert that are adjacent to one another
+  # into a single insert, changing the delete to insert the text from the
+  # insert rather than ""
+  # It's a small optimization, but it was in the original
+  defp collapse(
+         %TextEdit{
+           new_text: "",
+           range: %Range{
+             end: %Position{character: same_character, line: same_line}
+           }
+         } = delete_edit,
+         [
+           %TextEdit{
+             new_text: insert_text,
+             range:
+               %Range{
+                 start: %Position{character: same_character, line: same_line}
+               } = _insert_edit
+           }
+           | rest
+         ]
+       )
+       when byte_size(insert_text) > 0 do
+    collapsed_edit = %TextEdit{delete_edit | new_text: insert_text}
+    [collapsed_edit | rest]
+  end
+
+  defp collapse(%TextEdit{} = edit, edits) do
+    [edit | edits]
+  end
+
+  defp apply_diff(:eq, position, doc_string, edits) do
+    advance(doc_string, position, edits)
+  end
+
+  defp apply_diff(:del, {line, code_unit} = position, change, edits) do
+    {after_pos, {current_line, prev_lines}} = advance(change, position, edits)
+    {edit_end_line, edit_end_unit} = after_pos
+    current_line = [edit("", line, code_unit, edit_end_line, edit_end_unit) | current_line]
+    {after_pos, {current_line, prev_lines}}
+  end
+
+  defp apply_diff(:ins, {line, code_unit} = position, change, {current_line, prev_lines}) do
+    current_line = [edit(change, line, code_unit, line, code_unit) | current_line]
+    advance(change, position, {current_line, prev_lines})
+  end
+
+  defp advance(<<>>, position, edits) do
+    {position, edits}
+  end
+
+  for ending <- ["\r\n", "\r", "\n"] do
+    defp advance(<<unquote(ending), rest::binary>>, {line, _unit}, {current_line, prev_lines}) do
+      edits = {[], [current_line | prev_lines]}
+      advance(rest, {line + 1, 0}, edits)
+    end
+  end
+
+  defp advance(<<c, rest::binary>>, {line, unit}, edits) when c < 128 do
+    advance(rest, {line, unit + 1}, edits)
+  end
+
+  defp advance(<<c::utf8, rest::binary>>, {line, unit}, edits) do
+    increment = CodeUnit.count(:utf16, <<c::utf8>>)
+    advance(rest, {line, unit + increment}, edits)
+  end
+
+  defp edit(text, start_line, start_unit, end_line, end_unit) do
+    TextEdit.new(
+      new_text: text,
+      range:
+        Range.new(
+          start: Position.new(line: start_line, character: start_unit),
+          end: Position.new(line: end_line, character: end_unit)
+        )
+    )
+  end
+end