[Very WIP] Rewrite the core of the binding generator.

TL;DR: The binding generator is a mess as of right now. At first it was funny
(in a "this is challenging" sense) to improve on it, but this is not
sustainable.

The truth is that the current architecture of the binding generator is a huge
pile of hacks, so these few days I've been working on rewriting it with a few
goals.

 1) Have the hacks as contained and identified as possible. They're sometimes
    needed because how clang exposes the AST, but ideally those hacks are well
    identified and don't interact randomly with each others.

    As an example, in the current bindgen when scanning the parameters of a
    function that references a struct clones all the struct information, then if
    the struct name changes (because we mangle it), everything breaks.

 2) Support extending the bindgen output without having to deal with clang. The
    way I'm aiming to do this is separating completely the parsing stage from
    the code generation one, and providing a single id for each item the binding
    generator provides.

 3) No more random mutation of the internal representation from anywhere. That
    means no more Rc<RefCell<T>>, no more random circular references, no more
    borrow_state... nothing.

 4) No more deduplication of declarations before code generation.

    Current bindgen has a stage, called `tag_dup_decl`[1], that takes care of
    deduplicating declarations. That's completely buggy, and for C++ it's a
    complete mess, since we YOLO modify the world.

    I've managed to take rid of this using the clang canonical declaration, and
    the definition, to avoid scanning any type/item twice.

 5) Code generation should not modify any internal data structure. It can lookup
    things, traverse whatever it needs, but not modifying randomly.

 6) Each item should have a canonical name, and a single source of mangling
    logic, and that should be computed from the inmutable state, at code
    generation.

    I've put a few canonical_name stuff in the code generation phase, but it's
    still not complete, and should change if I implement namespaces.

Improvements pending until this can land:

 1) Add support for missing core stuff, mainly generating functions (note that
    we parse the signatures for types correctly though), bitfields, generating
    C++ methods.

 2) Add support for the necessary features that were added to work around some
    C++ pitfalls, like opaque types, etc...

 3) Add support for the sugar that Manish added recently.

 4) Optionally (and I guess this can land without it, because basically nobody
    uses it since it's so buggy), bring back namespace support.

These are not completely trivial, but I think I can do them quite easily with
the current architecture.

I'm putting the current state of affairs here as a request for comments... Any
thoughts? Note that there are still a few smells I want to eventually
re-redesign, like the ParseError::Recurse thing, but until that happens I'm
way happier with this kind of architecture.

I'm keeping the old `parser.rs` and `gen.rs` in tree just for reference while I
code, but they will go away.

[1]: https://github.com/Yamakaky/rust-bindgen/blob/master/src/gen.rs#L448

Checkpoint to show a rust ICE.

Union fields.

Parse annotations.

Multiple cleanups.

Cargo.toml

@@ -16,6 +16,7 @@ quasi = { version = "0.15", features = ["with-syntex"] }
 clippy = { version = "*", optional = true }
 syntex_syntax = "0.38"
 log = "0.3.*"
+env_logger = "*"
 libc = "0.2.*"
 clang-sys = "0.8.0"
 

build.rs

@@ -5,11 +5,11 @@ mod codegen {
 
     pub fn main() {
         let out_dir = env::var_os("OUT_DIR").unwrap();
-        let src = Path::new("src/gen.rs");
-        let dst = Path::new(&out_dir).join("gen.rs");
+        let src = Path::new("src/codegen/mod.rs");
+        let dst = Path::new(&out_dir).join("codegen.rs");
 
         quasi_codegen::expand(&src, &dst).unwrap();
-        println!("cargo:rerun-if-changed=src/gen.rs");
+        println!("cargo:rerun-if-changed=src/codegen/mod.rs");
     }
 }
 

src/bin/bindgen.rs

@@ -2,29 +2,17 @@
 #![crate_type = "bin"]
 
 extern crate bindgen;
+extern crate env_logger;
 #[macro_use]
 extern crate log;
 extern crate clang_sys;
 
-use bindgen::{Bindings, BindgenOptions, LinkType, Logger};
+use bindgen::{Bindings, BindgenOptions, LinkType};
 use std::io;
 use std::path;
 use std::env;
 use std::default::Default;
 use std::fs;
-use std::process::exit;
-
-struct StdLogger;
-
-impl Logger for StdLogger {
-    fn error(&self, msg: &str) {
-        println!("{}", msg);
-    }
-
-    fn warn(&self, msg: &str) {
-        println!("{}", msg);
-    }
-}
 
 enum ParseResult {
     CmdUsage,
@@ -92,21 +80,22 @@ fn parse_args(args: &[String]) -> ParseResult {
                     if ix + 1 >= args_len {
                         return ParseResult::ParseErr("Missing match pattern".to_string());
                     }
-                    options.match_pat.push(args[ix + 1].clone());
+                    options.match_pat.insert(args[ix + 1].clone());
                     ix += 2;
                 }
+                "-hidden-type" |
                 "-blacklist-type" => {
                     if ix + 1 >= args_len {
                         return ParseResult::ParseErr("Missing blacklist type name".to_string());
                     }
-                    options.blacklist_type.push(args[ix + 1].clone());
+                    options.hidden_types.insert(args[ix + 1].clone());
                     ix += 2;
                 }
                 "-opaque-type" => {
                     if ix + 1 >= args_len {
                         return ParseResult::ParseErr("Missing opaque type name".to_string());
                     }
-                    options.opaque_types.push(args[ix + 1].clone());
+                    options.opaque_types.insert(args[ix + 1].clone());
                     ix += 2;
                 }
                 "-builtins" => {
@@ -230,6 +219,13 @@ Options:
 }
 
 pub fn main() {
+    log::set_logger(|max_log_level| {
+        use env_logger::Logger;
+        let env_logger = Logger::new();
+        max_log_level.set(env_logger.filter());
+        Box::new(env_logger)
+    }).expect("Failed to set logger.");
+
     let mut bind_args: Vec<_> = env::args().collect();
     let bin = bind_args.remove(0);
 
@@ -247,17 +243,10 @@ pub fn main() {
         ParseResult::ParseErr(e) => panic!(e),
         ParseResult::CmdUsage => print_usage(bin),
         ParseResult::ParseOk(options, out) => {
-            let logger = StdLogger;
-            match Bindings::generate(&options, Some(&logger as &Logger), None) {
-                Ok(bindings) => match bindings.write(out) {
-                    Ok(()) => (),
-                    Err(e) => {
-                        logger.error(&format!("Unable to write bindings to file. {}", e));
-                        exit(-1);
-                    }
-                },
-                Err(()) => exit(-1)
-            }
+            let bindings = Bindings::generate(options, None)
+                                .expect("Failed to generate bindings!");
+
+            bindings.write(out).expect("Unable to write bindings!");
         }
     }
 }

src/clang.rs

@@ -15,6 +15,13 @@ pub struct Cursor {
     x: CXCursor
 }
 
+impl fmt::Debug for Cursor {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
+        write!(fmt, "Cursor({} kind: {}, loc: {})",
+               self.spelling(), kind_to_str(self.kind()), self.location())
+    }
+}
+
 pub type CursorVisitor<'s> = for<'a, 'b> FnMut(&'a Cursor, &'b Cursor) -> Enum_CXChildVisitResult + 's;
 
 impl Cursor {
@@ -49,12 +56,57 @@ impl Cursor {
         }
     }
 
+    pub fn num_template_args(&self) -> c_int {
+        unsafe {
+            clang_Cursor_getNumTemplateArguments(self.x)
+        }
+    }
+
+
+    /// This function gets the translation unit cursor. Note that we shouldn't
+    /// create a TranslationUnit struct here, because bindgen assumes there will
+    /// only be one of them alive at a time, and dispose it on drop. That can
+    /// change if this would be required, but I think we can survive fine
+    /// without it.
+    pub fn translation_unit(&self) -> Cursor {
+        assert!(self.is_valid());
+        unsafe {
+            let tu = clang_Cursor_getTranslationUnit(self.x);
+            let cursor = Cursor {
+                x: clang_getTranslationUnitCursor(tu),
+            };
+            assert!(cursor.is_valid());
+            cursor
+        }
+    }
+
+    pub fn is_toplevel(&self) -> bool {
+        let mut semantic_parent = self.semantic_parent();
+
+        while semantic_parent.kind() == CXCursor_Namespace ||
+              semantic_parent.kind() == CXCursor_NamespaceAlias ||
+              semantic_parent.kind() == CXCursor_NamespaceRef
+        {
+            semantic_parent = semantic_parent.semantic_parent();
+        }
+
+        let tu = self.translation_unit();
+        // Yes, the second can happen with, e.g., macro definitions.
+        semantic_parent == tu || semantic_parent == tu.semantic_parent()
+    }
+
     pub fn kind(&self) -> Enum_CXCursorKind {
         unsafe {
             clang_getCursorKind(self.x)
         }
     }
 
+    pub fn is_anonymous(&self) -> bool {
+        unsafe {
+            clang_Cursor_isAnonymous(self.x) != 0
+        }
+    }
+
     pub fn is_template(&self) -> bool {
         self.specialized().is_valid()
     }
@@ -77,10 +129,11 @@ impl Cursor {
         }
     }
 
-    pub fn raw_comment(&self) -> String {
-        unsafe {
+    pub fn raw_comment(&self) -> Option<String> {
+        let s = unsafe {
             String_ { x: clang_Cursor_getRawCommentText(self.x) }.to_string()
-        }
+        };
+        if s.is_empty() { None } else { Some(s) }
     }
 
     pub fn comment(&self) -> Comment {
@@ -165,12 +218,18 @@ impl Cursor {
         }
     }
 
-    pub fn enum_val(&self) -> i64 {
+    pub fn enum_val_signed(&self) -> i64 {
         unsafe {
             clang_getEnumConstantDeclValue(self.x) as i64
         }
     }
 
+    pub fn enum_val_unsigned(&self) -> u64 {
+        unsafe {
+            clang_getEnumConstantDeclUnsignedValue(self.x) as u64
+        }
+    }
+
     // typedef
     pub fn typedef_type(&self) -> Type {
         unsafe {
@@ -274,29 +333,40 @@ impl PartialEq for Cursor {
             clang_equalCursors(self.x, other.x) == 1
         }
     }
-
-    fn ne(&self, other: &Cursor) -> bool {
-        !self.eq(other)
-    }
 }
 
 impl Eq for Cursor {}
 
 impl Hash for Cursor {
     fn hash<H: Hasher>(&self, state: &mut H) {
-        self.x.kind.hash(state);
-        self.x.xdata.hash(state);
-        self.x.data[0].hash(state);
-        self.x.data[1].hash(state);
-        self.x.data[2].hash(state);
+        unsafe { clang_hashCursor(self.x) }.hash(state)
     }
 }
 
 // type
+#[derive(Clone, Hash)]
 pub struct Type {
     x: CXType
 }
 
+impl PartialEq for Type {
+    fn eq(&self, other: &Self) -> bool {
+        unsafe {
+            clang_equalTypes(self.x, other.x) != 0
+        }
+    }
+}
+
+impl Eq for Type {}
+
+impl fmt::Debug for Type {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
+        write!(fmt, "Type({}, kind: {}, decl: {:?}, canon: {:?})",
+               self.spelling(), type_to_str(self.kind()), self.declaration(),
+               self.declaration().canonical())
+    }
+}
+
 #[derive(Debug, Copy, Clone, Eq, PartialEq, Hash)]
 pub enum LayoutError {
     Invalid,
@@ -378,6 +448,24 @@ impl Type {
         }
     }
 
+    pub fn fallible_align(&self) -> Result<usize, LayoutError> {
+        unsafe {
+            let val = clang_Type_getAlignOf(self.x);
+            if val < 0 {
+                Err(LayoutError::from(val as i32))
+            } else {
+                Ok(val as usize)
+            }
+        }
+    }
+
+    pub fn fallible_layout(&self) -> Result<::ir::layout::Layout, LayoutError> {
+        use ir::layout::Layout;
+        let size = try!(self.fallible_size());
+        let align = try!(self.fallible_align());
+        Ok(Layout::new(size, align))
+    }
+
     pub fn align(&self) -> usize {
         unsafe {
             let val = clang_Type_getAlignOf(self.x);
@@ -581,21 +669,25 @@ pub struct Index {
 }
 
 impl Index {
-    pub fn create(pch: bool, diag: bool) -> Index {
+    pub fn new(pch: bool, diag: bool) -> Index {
         unsafe {
             Index { x: clang_createIndex(pch as c_int, diag as c_int) }
         }
     }
+}
 
-    pub fn dispose(&self) {
+impl fmt::Debug for Index {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
+        write!(fmt, "Index {{ }}")
+    }
+}
+
+impl Drop for Index {
+    fn drop(&mut self) {
         unsafe {
             clang_disposeIndex(self.x);
         }
     }
-
-    pub fn is_null(&self) -> bool {
-        self.x.is_null()
-    }
 }
 
 // Token
@@ -609,6 +701,12 @@ pub struct TranslationUnit {
     x: CXTranslationUnit
 }
 
+impl fmt::Debug for TranslationUnit {
+    fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
+        write!(fmt, "TranslationUnit {{ }}")
+    }
+}
+
 impl TranslationUnit {
     pub fn parse(ix: &Index, file: &str, cmd_args: &[String],
                  unsaved: &[UnsavedFile], opts: ::libc::c_uint) -> TranslationUnit {
@@ -655,12 +753,6 @@ impl TranslationUnit {
         }
     }
 
-    pub fn dispose(&self) {
-        unsafe {
-            clang_disposeTranslationUnit(self.x);
-        }
-    }
-
     pub fn is_null(&self) -> bool {
         self.x.is_null()
     }
@@ -687,6 +779,15 @@ impl TranslationUnit {
     }
 }
 
+impl Drop for TranslationUnit {
+    fn drop(&mut self) {
+        unsafe {
+            clang_disposeTranslationUnit(self.x);
+        }
+    }
+}
+
+
 // Diagnostic
 pub struct Diagnostic {
     x: CXDiagnostic

src/clangll.rs

@@ -428,7 +428,7 @@ pub const CXCallingConv_X86_64SysV: c_uint = 11;
 pub const CXCallingConv_Invalid: c_uint = 100;
 pub const CXCallingConv_Unexposed: c_uint = 200;
 #[repr(C)]
-#[derive(Copy, Clone)]
+#[derive(Copy, Clone, Hash)]
 pub struct CXType {
     pub kind: Enum_CXTypeKind,
     pub data: [*mut c_void; 2],
@@ -1076,6 +1076,7 @@ extern "C" {
     pub fn clang_Cursor_getNumArguments(C: CXCursor) -> c_int;
     pub fn clang_Cursor_getArgument(C: CXCursor, i: c_uint) ->
      CXCursor;
+    pub fn clang_Cursor_getNumTemplateArguments(T: CXCursor) -> c_int;
     pub fn clang_Cursor_getTemplateArgumentKind(C: CXCursor, i: c_uint) ->
      CXTemplateArgumentKind;
     pub fn clang_Cursor_getTemplateArgumentValue(C: CXCursor, i: c_uint) ->
@@ -1168,6 +1169,7 @@ extern "C" {
     pub fn clang_Cursor_getMangling(C: CXCursor) -> CXString;
     pub fn clang_Cursor_getParsedComment(C: CXCursor) -> CXComment;
     pub fn clang_Cursor_getModule(C: CXCursor) -> CXModule;
+    pub fn clang_Cursor_isAnonymous(C: CXCursor) -> c_uint;
     pub fn clang_Module_getASTFile(Module: CXModule) -> CXFile;
     pub fn clang_Module_getParent(Module: CXModule) -> CXModule;
     pub fn clang_Module_getName(Module: CXModule) -> CXString;