Compiler docs

[jkitchin]

All this pull request does is add one line to the checkargs decorator to set its doc attribute on the returned function, and add docstrings to all the functions decorated by @builds. A few docstrings are marked TODO. I used the documentation at http://docs.hylang.org/en/latest/ for this.

[refi64]

I'm not sure about this...the majority of these are duplicated from the main docs, resulting in having to update them in two places, and the compile_* functions seem pretty self-explanatory (e.g. compile_while compiles a while expression).

[berkerpeksag]

I agree with @kirbyfan64. Perhaps we can document briefly what compile_* functions do in HyASTCompiler docstring or in the internals docs at http://docs.hylang.org/en/latest/language/internals.html

[jkitchin]

The point of this is to be able to get help from within hy on what the compiler functions do. See http://kitchingroup.cheme.cmu.edu/blog/2016/04/03/Getting-hylp-in-hy/ for example. None of the functions defined in compiler.py have any docstrings attached to them, so they are not introspectable in an editor to find out what they do, e.g. http://kitchingroup.cheme.cmu.edu/blog/2016/04/06/Another-step-towards-HyDE/.

This also makes it possible to auto-generate documentation, rather than update it in one place that isn't the code, or two places.

[refi64]

I guess that makes sense...but it would first require the ability to even auto-generate the docs (which isn't possible ATM).

But I still think some of the docstrings, like compile_complex, are a little overkill, since the function names are rather self-explanatory.

[gilch]

I like the idea of auto-generating the reference docs from the docstrings. They're not identical currently, so for each we'd have to pick the best of the two or merge them somehow. The examples could also be doctests, once we get that working for Hy code. (see #1019.) This way, we can always be sure the examples work as intended, and we can easily access the references from the repl.

The rest of the docs will still have to be updated by hand, but we should be able to automatically test the examples there too.

[jkitchin]

All of these are used by users I think, but not in an obvious way to
users. For example:

(cut col 0 1 2)

uses compile_cut_expression. A user would not directly call this, but by
putting a docstring on it, you can access what cut is supposed to do.

e.g.

#+BEGIN_SRC hy
(import hy)
(for [x (hy.compiler._compile_table.items)](when %28= "cut" %28get x 0))
(print (. x [1] doc))))
#+END_SRC

#+RESULTS:
#+begin_example
cut can be used to take a subset of a list and create a new list from it.

    The form takes at least one parameter specifying the list to cut. Two
    optional parameters can be used to give the start and end position of
    the subset. If they are not supplied, the default value of None will
    be used instead. The third optional parameter is used to control step
    between the elements.

    cut follows the same rules as its Python counterpart. Negative indices
    are counted starting from the end of the list.

#+end_example

Matthew Egan Odendahl writes:

Maybe we should mark some of these as implementation detail subject to change without notice, rather than part of the public API. A simple docstring like "implementation detail: do not use" would suffice.

---

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#1040 (comment)

Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[jkitchin]

It isn't that far off though. In this post
http://kitchingroup.cheme.cmu.edu/blog/2016/04/03/Getting-hylp-in-hy/ I
generated documentation for everything I could find at the time (even a
list of the compiler functions, just no documentation for them).

Self explanatory is in the eye of the beholder ;) For people brand-new
to this, they might disagree. At some point I think having examples that
are doctestable would make it worth having even trivial ones.

Ryan Gonzalez writes:

I guess that makes sense...but it would first require the ability to even auto-generate the docs (which isn't possible ATM).

But I still think some of the docstrings, like compile_complex, are a little overkill, since the function names are rather self-explanatory.

---

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#1040 (comment)

Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[refi64]

@jkitchin Good point about doctest; that would be really useful, since the documentation examples frequently go out of date.

[jkitchin]

For example,

These examples didn't work for me today:

http://docs.hylang.org/en/latest/language/api.html#unquote

I gathered that name is a function, and def doesn't redefine it.

=> (def name "Cuddles")
=> (quasiquote (= name (unquote name)))
(u'=' u'name' <function name at 0x102ee3668>)

The docs advertise:
(def name "Cuddles")
(quasiquote (= name (unquote name)))
;=> (u'=' u'name' u'Cuddles')

Ryan Gonzalez writes:

@jkitchin Good point about doctest; that would be really useful, since the documentation examples frequently go out of date.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#1040 (comment)

Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[refi64]

@jkitchin That should work outside the REPL. When you call name, Hy re-imports it from hy.core.builtins, which overrides anything you had assigned to it.

[gilch]

We could override Python's help function with a macro that checks if the symbol is a macro or built-in hy special form, and if so, fetches the appropriate docstring. If not, it can defer to the built-in Python help. This would help with #356 and #892.

[jkitchin]

That is more or less what I was aiming at in this code:
https://github.com/jkitchin/hy/blob/hydoc/hy/core/hylp.hy
That code does the language, shadowed functions, keywords and macros.

I didn't get to deferring to built-in Python help yet, but I would like to
get there!

John

---

Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

On Mon, Apr 11, 2016 at 3:50 PM, Matthew Egan Odendahl <
[email protected]> wrote:

We could override Python's help function with a macro that checks if the
symbol is a macro or built-in hy special form, and if so fetches the
appropriate docstring. If not it can defer to the built-in Python help.
This would help with #356 #356 and
#892 #892.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1040 (comment)

[jkitchin]

Here goes a hydoc macro that covers functions in hy, including macros, and Python functions. I think this should resolve #892 and #356.

(defmacro hydoc [sym]
  "Return docstring for SYM.
  Look for hy functions first, then defer to Python."
  (let [code `(do
               (import hy)
               (import pydoc))]
    (.append code
       (cond
        ;; language functions
        [(in sym (. hy core language *exports*))
         `(pydoc.getdoc (. hy core language ~(HySymbol sym)))]

        ;; shadowed functions
        [(in sym (. hy core shadow *exports*))
         `(pydoc.getdoc (. hy core shadow  ~(HySymbol sym)))]

        ;; Macros
        [(in sym (.keys (get hy.macros._hy_macros nil)))
         `(pydoc.getdoc (get hy.macros._hy_macros nil ~sym))]

        ;; Functions in compiler
        [(in sym (list-comp (get x 0) [x (hy.compiler._compile_table.items)]
                            (string? (get x 0))))
         `(pydoc.getdoc (get hy.compiler._compile_table ~sym))]

        ;; Fallback to Python
        [true
         `(try
           (pydoc.getdoc ~(HySymbol sym))
           (except [e NameError] (.format "{} Not found." ~sym)))]))
    code))

[gilch]

Maybe more like this?

(defn hydoc* [sym]
  (import hy)
  (. (or (->> false
              (.get hy.compiler._compile_table sym)
              (.get (get hy.macros._hy_macros nil) sym)
              (.get hy.core.shadow.__dict__ sym)
              (.get hy.core.language.__dict__ sym))
         (eval sym))
     __doc__))

(defmacro hydoc [sym]
  `(hydoc* '~sym))

[gilch]

You could override Python's interactive help using these too:

(defmacro help [item]
  (if (symbol? item)
    `(print (hydoc ~item))
    `((get __builtins__ "help") ~item)))

[jkitchin]

That is pretty nice, I haven't used the threading macros before. Pretty
concise. I think this might be a bit more robust, it also gets comments
if there is no doc attribute

  (import pydoc)
  (pydoc.getdoc (or (->> false
            (.get hy.compiler._compile_table sym)
            (.get (get hy.macros._hy_macros nil) sym)
            (.get hy.core.shadow.__dict__ sym)
            (.get hy.core.language.**dict** sym))
         (eval sym))))

Just curious is the hydoc* an indication it is a function (the * at the
end) to differentiate it from the macro by the similar name?

Matthew Egan Odendahl writes:

Maybe more like this?

(defn hydoc* [sym]
  (import hy)
  (. (or (->> false
            (.get hy.compiler._compile_table sym)
            (.get (get hy.macros._hy_macros nil) sym)
            (.get hy.core.shadow.__dict__ sym)
            (.get hy.core.language.__dict__ sym))
         (eval sym))
     __doc__))

(defmacro hydoc [sym]
  `(hydoc* '~sym))

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#1040 (comment)

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[gilch]

Just curious is the hydoc* an indication it is a function (the * at the
end) to differentiate it from the macro by the similar name?

Not exactly. In mathematics, you often talk about some object g and some related object g' (g prime). In Lisp, the apostrophe is already taken to mean quotation, so we use * for this purpose. Usually you have a * form that is syntactically annoying to use doing the real work, and some nicer interface to it without the *. This is often, but not necessarily, a macro. Hy inherits some design philosophy from Clojure, where you see this pattern a lot internally. In Hy, for example we have for that uses for* and if that uses if*.

In this case, the macro lets us avoid quoting the symbol when calling the function, but we still have the function when we need it to e.g. map over a list of symbols and return all of their docstrings. This would be more difficult with only the macro.

I'm less confident this is the right approach for Hy (at least in this instance), because of #1041, (Hy's eval doesn't work like Clojure, we might change that #1043.). We wouldn't need that eval if it was all in the macro. You could just unquote it.

[gilch]

The "TODO" docstrings in this PR are for:

• except (used in try)
• with* (used in with)
• for* (used in for)
• HyList (a Hy Model)
• HySet (another Hy Model)
• dispatch-reader-macro (used for reader macros)
• eval-and-compile
• eval-when-compile

eval-and-compile and eval-when-compile were never documented. I didn't know what these do either. I don't see them used anywhere but tests. My best guess is that they were borrowed from Emacs Lisp. The blame shows @khinsen creating them in #187, which confirms this. That PR, and the related issue #186, could help us document them.

[Kodiologist]

It doesn't look like any core members have approved this. While it would be great to have docstrings, I don't want to have the same documentation appear in two different places in the codebase, like @kirbyfan64 was talking about. I think we need to have a way for docstrings to update the manual (e.g., #1044) or vice versa. Copying and pasting is no good even as a temporary solution.