doc: improve documentation (#730)

added missing_docs warning etc. and documented the currently undocumented public APIs. added additional explanations in README, updated links, etc. Closes #335 Closes #299 Closes #298 Closes #297 Closes #185 Co-authored-by: Tomas Tauber <[email protected]>
pest-parser · Nov 4, 2022 · bd3b18a · bd3b18a
1 parent edaef7f
commit bd3b18a
Show file tree

Hide file tree

Showing 37 changed files with 656 additions and 174 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -4,8 +4,8 @@
 
 Before diving into the whys and hows, it's best one got started with the what's. The best place to learn about what pest does and what its limits are is the [book]. Feel free to try any of the examples in the [fiddle editor] as well.
 
-[book]: https://pest-parser.github.io/book
-[fiddle editor]: https://pest-parser.github.io/#editor
+[book]: https://pest.rs/book
+[fiddle editor]: https://pest.rs/#editor
 
 With that out of the way, let's go through *pest's* crate structure:
 
@@ -46,9 +46,14 @@ Our [website] and [book] are in constant need of attention. While not as well or
 [website]:https://github.com/pest-parser/site
 [book]: https://github.com/pest-parser/book
 
-## Gitter
+## Gitter, Discord and GitHub Discussions
 
-Sometimes it's best to just say what you want. For that, there's our [Gitter] room. Leave feedback, help out, learn what people are up to, go off-topic for hours, or complain that compile times are terrible—seriously, please don't.
+Sometimes it's best to just say what you want. For that, there's our [Gitter] room or [Discord] server. Leave feedback, help out, learn what people are up to, go off-topic for hours, or complain that compile times are terrible—seriously, please don't.
+
+For more long-living threads and common questions, you can use [GitHub Discussions].
 
 [Gitter]: https://gitter.im/pest-parser/pest
 
+[Discord]: https://discord.gg/XEGACtWpT2
+
+[GitHub Discussions]: https://github.com/pest-parser/pest/discussions
diff --git a/README.md b/README.md
@@ -5,8 +5,8 @@
 
 # pest. The Elegant Parser
 
-[![Join the chat at https://gitter.im/dragostis/pest](https://badges.gitter.im/dragostis/pest.svg)](https://gitter.im/dragostis/pest?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
-[![Book](https://img.shields.io/badge/book-WIP-4d76ae.svg)](https://pest-parser.github.io/book)
+[![Join the chat at https://gitter.im/pest-parser/pest](https://badges.gitter.im/dragostis/pest.svg)](https://gitter.im/pest-parser/pest?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Book](https://img.shields.io/badge/book-WIP-4d76ae.svg)](https://pest.rs/book)
 [![Docs](https://docs.rs/pest/badge.svg)](https://docs.rs/pest)
 
 [![pest Continuous Integration](https://github.com/pest-parser/pest/actions/workflows/ci.yml/badge.svg)](https://github.com/pest-parser/pest/actions/workflows/ci.yml)
@@ -31,12 +31,15 @@ Other helpful resources:
 
 * API reference on [docs.rs]
 * play with grammars and share them on our [fiddle]
-* leave feedback, ask questions, or greet us on [Gitter]
+* find previous common questions answered or ask questions on [GitHub Discussions]
+* leave feedback, ask questions, or greet us on [Gitter] or [Discord]
 
-[book]: https://pest-parser.github.io/book
+[book]: https://pest.rs/book
 [docs.rs]: https://docs.rs/pest
-[fiddle]: https://pest-parser.github.io/#editor
-[Gitter]: https://gitter.im/dragostis/pest
+[fiddle]: https://pest.rs/#editor
+[Gitter]: https://gitter.im/pest-parser/pest
+[Discord]: https://discord.gg/XEGACtWpT2
+[GitHub Discussions]: https://github.com/pest-parser/pest/discussions
 
 ## Example
 
@@ -81,6 +84,9 @@ thread 'main' panicked at ' --> 1:1
   = expected ident', src/main.rs:12
 ```
 
+These error messages can be obtained from their default `Display` implementation,
+e.g. `panic!("{}", parser_result.unwrap_err())` or `println!("{}", e)`.
+
 ## Pairs API
 
 The grammar can be used to derive a `Parser` implementation automatically.
@@ -133,6 +139,25 @@ Letter:  b
 Digit:   2
 ```
 
+### Defining multiple parsers in a single file
+The current automatic `Parser` derivation will produce the `Rule` enum
+which would have name conflicts if one tried to define multiple such structs
+that automatically derive `Parser`. One possible way around it is to put each
+parser struct in a separate namespace:
+
+```rust
+mod a {
+    #[derive(Parser)]
+    #[grammar = "a.pest"]
+    pub struct ParserA;
+}
+mod b {
+    #[derive(Parser)]
+    #[grammar = "b.pest"]
+    pub struct ParserB;
+}
+```
+
 ## Other features
 
 * Precedence climbing

diff --git a/bootstrap/Cargo.toml b/bootstrap/Cargo.toml
@@ -4,7 +4,7 @@ description = "pest bootstrap script"
 version = "0.0.0"
 edition = "2021"
 authors = ["Dragoș Tiselice <[email protected]>"]
-homepage = "https://pest-parser.github.io/"
+homepage = "https://pest.rs/"
 repository = "https://github.com/pest-parser/pest"
 documentation = "https://docs.rs/pest"
 publish = false

diff --git a/derive/Cargo.toml b/derive/Cargo.toml
@@ -1,10 +1,10 @@
 [package]
 name = "pest_derive"
 description = "pest's derive macro"
-version = "2.4.0"
+version = "2.4.1"
 edition = "2021"
 authors = ["Dragoș Tiselice <[email protected]>"]
-homepage = "https://pest-parser.github.io/"
+homepage = "https://pest.rs/"
 repository = "https://github.com/pest-parser/pest"
 documentation = "https://docs.rs/pest"
 keywords = ["pest", "parser", "peg", "grammar"]
@@ -23,5 +23,5 @@ std = ["pest/std", "pest_generator/std"]
 
 [dependencies]
 # for tests, included transitively anyway
-pest = { path = "../pest", version = "2.4.0", default-features = false }
-pest_generator = { path = "../generator", version = "2.4.0", default-features = false }
+pest = { path = "../pest", version = "2.4.1", default-features = false }
+pest_generator = { path = "../generator", version = "2.4.1", default-features = false }
diff --git a/derive/src/lib.rs b/derive/src/lib.rs
@@ -6,7 +6,12 @@
 // license <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
 // option. All files in the project carrying such notice may not be copied,
 // modified, or distributed except according to those terms.
-
+#![doc(
+    html_root_url = "https://docs.rs/pest_derive",
+    html_logo_url = "https://raw.githubusercontent.com/pest-parser/pest/master/pest-logo.svg",
+    html_favicon_url = "https://raw.githubusercontent.com/pest-parser/pest/master/pest-logo.svg"
+)]
+#![warn(missing_docs, rust_2018_idioms, unused_qualifications)]
 //! # pest. The Elegant Parser
 //!
 //! pest is a general purpose parser written in Rust with a focus on accessibility, correctness,
@@ -24,12 +29,16 @@
 //!
 //! * API reference on [docs.rs]
 //! * play with grammars and share them on our [fiddle]
-//! * leave feedback, ask questions, or greet us on [Gitter]
+//! * find previous common questions answered or ask questions on [GitHub Discussions]
+//! * leave feedback, ask questions, or greet us on [Gitter] or [Discord]
 //!
-//! [book]: https://pest-parser.github.io/book
+//! [book]: https://pest.rs/book
 //! [docs.rs]: https://docs.rs/pest
-//! [fiddle]: https://pest-parser.github.io/#editor
-//! [Gitter]: https://gitter.im/dragostis/pest
+//! [fiddle]: https://pest.rs/#editor
+//! [Gitter]: https://gitter.im/pest-parser/pest
+//! [Discord]: https://discord.gg/XEGACtWpT2
+//! [GitHub Discussions]: https://github.com/pest-parser/pest/discussions
+//!
 //!
 //! ## `.pest` files
 //!
@@ -181,6 +190,10 @@
 //! `e1` did. Repetitions and optionals (`e*`, `e+`, `e{, n}`, `e{n,}`,
 //! `e{m,n}`, `e?`) can modify the stack each time `e` matches. The `!e` and `&e`
 //! expressions are a special case; they never modify the stack.
+//! Many languages have "keyword" tokens (e.g. if, for, while) as well as general
+//! tokens (e.g. identifier) that matches any word. In order to match a keyword,
+//! generally, you may need to restrict that is not immediately followed by another
+//! letter or digit (otherwise it would be matched as an identifier).
 //!
 //! ## Special rules
 //!
@@ -289,12 +302,10 @@
 //! * `ASCII` - matches a character from \x00..\x7f
 //! * `NEWLINE` - matches either "\n" or "\r\n" or "\r"
 
-#![doc(html_root_url = "https://docs.rs/pest_derive")]
-extern crate pest_generator;
-extern crate proc_macro;
-
 use proc_macro::TokenStream;
 
+/// The main method that's called by the proc macro
+/// (a wrapper around `pest_generator::derive_parser`)
 #[proc_macro_derive(Parser, attributes(grammar, grammar_inline))]
 pub fn derive_parser(input: TokenStream) -> TokenStream {
     pest_generator::derive_parser(input.into(), true).into()

diff --git a/generator/Cargo.toml b/generator/Cargo.toml
@@ -1,10 +1,10 @@
 [package]
 name = "pest_generator"
 description = "pest code generator"
-version = "2.4.0"
+version = "2.4.1"
 edition = "2021"
 authors = ["Dragoș Tiselice <[email protected]>"]
-homepage = "https://pest-parser.github.io/"
+homepage = "https://pest.rs/"
 repository = "https://github.com/pest-parser/pest"
 documentation = "https://docs.rs/pest"
 keywords = ["pest", "generator"]
@@ -18,8 +18,8 @@ default = ["std"]
 std = ["pest/std"]
 
 [dependencies]
-pest = { path = "../pest", version = "2.4.0", default-features = false }
-pest_meta = { path = "../meta", version = "2.4.0" }
+pest = { path = "../pest", version = "2.4.1", default-features = false }
+pest_meta = { path = "../meta", version = "2.4.1" }
 proc-macro2 = "1.0"
 quote = "1.0"
 syn = "1.0"
diff --git a/generator/src/generator.rs b/generator/src/generator.rs
@@ -159,7 +159,7 @@ fn generate_builtin_rules() -> Vec<(&'static str, TokenStream)> {
         builtins.push((property, quote! {
             #[inline]
             #[allow(dead_code, non_snake_case, unused_variables)]
-            fn #property_ident(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            fn #property_ident(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 state.match_char_by(::pest::unicode::#property_ident)
             }
         }));
@@ -249,7 +249,7 @@ fn generate_rule(rule: OptimizedRule) -> TokenStream {
         RuleType::Normal => quote! {
             #[inline]
             #[allow(non_snake_case, unused_variables)]
-            pub fn #name(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 state.rule(Rule::#name, |state| {
                     #expr
                 })
@@ -258,14 +258,14 @@ fn generate_rule(rule: OptimizedRule) -> TokenStream {
         RuleType::Silent => quote! {
             #[inline]
             #[allow(non_snake_case, unused_variables)]
-            pub fn #name(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 #expr
             }
         },
         RuleType::Atomic => quote! {
             #[inline]
             #[allow(non_snake_case, unused_variables)]
-            pub fn #name(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 state.rule(Rule::#name, |state| {
                     state.atomic(::pest::Atomicity::Atomic, |state| {
                         #expr
@@ -276,7 +276,7 @@ fn generate_rule(rule: OptimizedRule) -> TokenStream {
         RuleType::CompoundAtomic => quote! {
             #[inline]
             #[allow(non_snake_case, unused_variables)]
-            pub fn #name(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 state.atomic(::pest::Atomicity::CompoundAtomic, |state| {
                     state.rule(Rule::#name, |state| {
                         #expr
@@ -287,7 +287,7 @@ fn generate_rule(rule: OptimizedRule) -> TokenStream {
         RuleType::NonAtomic => quote! {
             #[inline]
             #[allow(non_snake_case, unused_variables)]
-            pub fn #name(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+            pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                 state.atomic(::pest::Atomicity::NonAtomic, |state| {
                     state.rule(Rule::#name, |state| {
                         #expr
@@ -999,7 +999,7 @@ mod tests {
 
                                 #[inline]
                                 #[allow(dead_code, non_snake_case, unused_variables)]
-                                pub fn skip(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+                                pub fn skip(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                                     Ok(state)
                                 }
                             }
@@ -1009,13 +1009,13 @@ mod tests {
 
                                 #[inline]
                                 #[allow(non_snake_case, unused_variables)]
-                                pub fn a(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+                                pub fn a(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                                     state.match_string("b")
                                 }
 
                                 #[inline]
                                 #[allow(dead_code, non_snake_case, unused_variables)]
-                                pub fn ANY(state: #box_ty<::pest::ParserState<Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<Rule>>> {
+                                pub fn ANY(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
                                     state.skip(1)
                                 }
                             }

diff --git a/generator/src/lib.rs b/generator/src/lib.rs
@@ -7,17 +7,19 @@
 // option. All files in the project carrying such notice may not be copied,
 // modified, or distributed except according to those terms.
 
-#![doc(html_root_url = "https://docs.rs/pest_derive")]
+#![doc(
+    html_root_url = "https://docs.rs/pest_derive",
+    html_logo_url = "https://raw.githubusercontent.com/pest-parser/pest/master/pest-logo.svg",
+    html_favicon_url = "https://raw.githubusercontent.com/pest-parser/pest/master/pest-logo.svg"
+)]
+#![warn(missing_docs, rust_2018_idioms, unused_qualifications)]
 #![recursion_limit = "256"]
+//! # pest generator
+//!
+//! This crate generates code from ASTs (which is used in the `pest_derive` crate).
 
-extern crate pest;
-extern crate pest_meta;
-
-extern crate proc_macro;
-extern crate proc_macro2;
 #[macro_use]
 extern crate quote;
-extern crate syn;
 
 use std::env;
 use std::fs::File;
@@ -34,6 +36,9 @@ mod generator;
 use pest_meta::parser::{self, Rule};
 use pest_meta::{optimizer, unwrap_or_report, validator};
 
+/// Processes the derive/proc macro input and generates the corresponding parser based
+/// on the parsed grammar. If `include_grammar` is set to true, it'll generate an explicit
+/// "include_str" statement (done in pest_derive, but turned off in the local bootstrap).
 pub fn derive_parser(input: TokenStream, include_grammar: bool) -> TokenStream {
     let ast: DeriveInput = syn::parse2(input).unwrap();
     let (name, generics, content) = parse_derive(ast);

diff --git a/generator/src/macros.rs b/generator/src/macros.rs
@@ -19,7 +19,7 @@ macro_rules! generate_rule {
         quote! {
             #[inline]
             #[allow(dead_code, non_snake_case, unused_variables)]
-            pub fn $name(state: ::std::boxed::Box<::pest::ParserState<Rule>>) -> ::pest::ParseResult<::std::boxed::Box<::pest::ParserState<Rule>>> {
+            pub fn $name(state: ::std::boxed::Box<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<::std::boxed::Box<::pest::ParserState<'_, Rule>>> {
                 $pattern
             }
         }
@@ -32,7 +32,7 @@ macro_rules! generate_rule {
         quote! {
             #[inline]
             #[allow(dead_code, non_snake_case, unused_variables)]
-            pub fn $name(state: ::alloc::boxed::Box<::pest::ParserState<Rule>>) -> ::pest::ParseResult<::alloc::boxed::Box<::pest::ParserState<Rule>>> {
+            pub fn $name(state: ::alloc::boxed::Box<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<::alloc::boxed::Box<::pest::ParserState<'_, Rule>>> {
                 $pattern
             }
         }

diff --git a/grammars/Cargo.toml b/grammars/Cargo.toml
@@ -1,10 +1,10 @@
 [package]
 name = "pest_grammars"
 description = "pest popular grammar implementations"
-version = "2.4.0"
+version = "2.4.1"
 edition = "2021"
 authors = ["Dragoș Tiselice <[email protected]>"]
-homepage = "https://pest-parser.github.io/"
+homepage = "https://pest.rs/"
 repository = "https://github.com/pest-parser/pest"
 documentation = "https://docs.rs/pest"
 keywords = ["pest", "parser", "peg", "grammar"]
@@ -14,8 +14,8 @@ readme = "_README.md"
 rust-version = "1.56"
 
 [dependencies]
-pest = { path = "../pest", version = "2.4.0" }
-pest_derive = { path = "../derive", version = "2.4.0" }
+pest = { path = "../pest", version = "2.4.1" }
+pest_derive = { path = "../derive", version = "2.4.1" }
 
 [dev-dependencies]
 criterion = "0.3"