Dogen is the reference implementation of the MASD Project code generator. MASD - Model Assisted Software Development - is a new methodology for the development of software systems based on MDE (Model Driven Engineering). MASD was developed in the context of a PhD programme; if you are that way inclined, you may find the thesis of interest: Model Assisted Software Development - a MDE-Based Software Development Methodology.
In lay terms, Dogen's objective is to automate the development process as much as possible: the user creates models using Org Mode, though these must respect a set of predefined restrictions. Dogen uses this input to generate a source code representation of the model. Unlike traditional modeling techniques like UML, Dogen focuses instead on literate modeling - an approach based on Knuth's Literate programming.
The generated code contains most of the functionality required from a typical domain object such as serialisation, hashing, streaming and so on. Whilst we intend for the application to be useful out of the box for most use cases, our end goal is to enable users to extend Dogen, adapting it to the vagaries of their specific needs. Note that Dogen has experimental C# support, but it is not in feature parity with C++ support.
Dogen provides a reference product for each supported language, available as a separate git repo:
- C++ Reference Product. Models available here.
- C# Reference Product. Models available here.
These are intended to give you a feel for how the generated code will look like, and they attempt to exercise all of Dogen's functionality.
Our previous major milestone for Dogen was to complete the PhD thesis, which we did so successfully. The current major milestone is to sync up the source code with the latest version of the theoretical framework, as laid out in the thesis. In other words, our focus is not on producing a fully polished tool, ready for end users to deploy on industrial scale projects. If, for whatever reason you still want to go ahead and use it, bear in mind the following:
- our immediate goal is to refactor the conceptual model in the source code to reflect the MASD conceptual model presented in Chapter 6 of the thesis (Domain Architecture). We will continue to refactor the code internally until it is consistent with the conceptual model. It may take a long while to get there. This means that adding new features is not a priority for Dogen development (though we are gathering a long list of ideas in our product backlog).
- in our quest for the conceptual model, we have broken and we shall continue to break the API many times. This will manifest itself in having to update user models or in changes in the generated code. Features that have been working thus far may also be deprecated. We provide no guarantees of forward or backward compatibility; there isn't even an effort not to break these - in fact, quite the opposite, if its not in agreement with the conceptual model it must be changed. We do try to highlight these in the release notes best we can.
- we do have an extensive suite of tests (both generated and handcrafted), but these focus only on Dogen's use cases. If you try any permutations outside these, it is likely you will find problems. The errors you will get will not be the most obvious.
- there is little to no documentation on Dogen. We are presently working on porting across the documentation generated for the thesis but the process is slow and laborious.
- for this tool to be useful in any way in its present state, you must have a good working knowledge of MDE. You can make use of our companion notes to get up to speed: Notes on Model Driven Engineering.
These warnings can probably best summarised as follows: "if it breaks, you get to keep the pieces; and it will break, often". But if you find such things fun, proceed with caution.
Binary packages are available for each release on and GitHub (links at the end of each release note). The binaries cover Debian Linux, OSX and Windows - 64-bit only. Packages are also generated for each commit and saved with the corresponding GitHub Workflow. Development is done from git master, so latest should always be greatest, but you may want to use packages from the previous sprint rather than what is currently under development (see badges above).
Notes
- Other than Linux, we do not test the packages manually. If you find any issues with a binary package, please raise an issue.
- The OSX installer provides you with a DMG. Once you open it in Finder, it
mounts under
/Volumes/, e.g.:
/Volumes/dogen-${VERSION}-Darwin-x86_64/dogen.app/Contents/Resources/bin
Where ${VERSION} is your Dogen version such as 1.0.32.
In order to build Dogen you will need a C++ toolchain. On Linux and OSX, you'll need a recent compiler with C++ 17 support, such as GCC 11 or Clang 14 - and Ninja or GNU Make. On Windows you'll need Visual Studio 2017. On all platforms, we make use of CMake. Older compilers may work, but we try to always use the most recent releases in MASD. So, if you can, please try using to those.
Dogen has the following dependencies, across all operative systems:
| Name | Type | Version |
|---|---|---|
| CMake | Mandatory | 3.23 or greater. |
| Boost | Mandatory | 1.81 or greater. |
| DTL | Mandatory | 1.19 |
Note that earlier versions may work but haven't been tested. Also, though Dogen should build fine with package manager supplied libraries - or even with hand-built dependencies - the recommended way to setup a development environment on all supported platforms is by using vcpkg, which is what this document describes. Dogen uses the "sub-module setup", whereby vcpkg is a submodule of Dogen. All of the dependencies are declared in the vcpkg.json file.
Before we proceed, please note that Dogen makes use of CMake
Presets. You
can have a look at our presets file
CMakePresets.json
for a list of available presets; alternatively, just use CMake:
$ cmake --list-presets
Available configure presets:
"linux-clang-debug" - Linux clang debug
"linux-clang-release" - Linux clang release
"linux-gcc-debug" - Linux gcc debug
"linux-gcc-release" - Linux gcc release
"windows-msvc-debug" - Windows x64 Debug
"windows-msvc-release" - Windows x64 Release
"windows-msvc-clang-cl-debug" - Windows x64 Debug
"windows-msvc-clang-cl-release" - Windows x64 Release
"macos-clang-debug" - Mac OSX Debug
"macos-clang-release" - Mac OSX Release
For the remainder of this document we shall assume linux-clang-release,
which is our preferred preset. If it isn't yours, remember to update the preset
name to match your desired build.
To checkout Dogen, clone as follows:
git clone https://github.com/MASD-Project/dogen.git --recurse-submodules
This takes into account the vcpkg submodule setup, as well as test data (explained below). For the configure step:
cd dogen
cmake --preset linux-clang-release
You can build Dogen on all platforms (modulus preset) as follows:
cmake --build --preset linux-clang-release
If you'd like to run the project tests, execute the target run_all_tests
or its abbreviation rat. Note that the tests check code generation against
both Dogen, as well as the reference products:
Reference products are checked out as git sub-modules under test_data and
pointed to in the presets file, e.g.:
"environment": {
"DOGEN_PROJECTS_DIRECTORY": "${sourceDir}/projects",
"CPP_REF_IMPL_PROJECTS_DIRECTORY":"${sourceDir}/test_data/cpp_ref_impl/projects",
"CSHARP_REF_IMPL_PROJECTS_DIRECTORY":"${sourceDir}/test_data/csharp_ref_impl/Src"
},If you see any errors running CMake - e.g. "DOGEN_PROJECTS_DIRECTORY not
found.", etc. - please make sure the sub-modules were checked out correctly.
If all has gone according to plan, you should then see that Dogen correctly
recognises the additional models (where DIR is the top-level directory for
Dogen):
...
-- CMake Version: 3.24.2
-- DOGEN_PROJECTS_DIRECTORY=${DIR}/projects
-- CPP_REF_IMPL_PROJECTS_DIRECTORY=${DIR}/test_data/cpp_ref_impl/projects
-- CSHARP_REF_IMPL_PROJECTS_DIRECTORY=${DIR}/test_data/csharp_ref_impl/Src
...
You can run all tests as follows:
cmake --build --preset linux-clang-release --target rat
You can also use CTest to run the tests, e.g.:
ctest --preset linux-clang-release
Or, the slightly more verbose incantation which has the same result:
cmake --build --preset linux-clang-release --target test
Note that tests take slightly longer to run in the CTest incantation, and the output is a tad more verbose.
If you'd like to run Dogen to regenerate all of its models, you can do so by
using the target generate_all_org or its abbreviation gao. This is
useful, for example, to update the models to the latest version of Dogen, or
just to see how Dogen works. By default Dogen uses the current built version,
but you can also use another. For this you may need to tell CMake of its
location by setting CMAKE_PROGRAM_PATH. After regeneration, you can then
use git diff to inspect the differences produced by regeneration, if any.
The build directory contains all of the logs, under the directory log.
Example:
cmake --build --preset linux-clang-release --target gao
Replacing linux-clang-release with your platform and compiler.
Important: Please note that this does not regenerate the reference models.
Whilst we do not have a huge amount of documentation aimed at end users, there is plenty of assorted material discussing Dogen. Most of the material targets developers and/or researchers.
A number of videos are available discussing Dogen, gathered under "themed" playlists:
- MASD - Dogen Coding: Move to GitHub Actions: Slightly messy video series that continues where the previous series left off (MASD - Dogen Coding: Move to GitHub CI), e.g. on getting our CI and nightlies to work again. It was meant to be about refactoring the PMM but we still had a number of CI issues. In addition, due to sound-related problems we had to abandon the series mid-way through.
- MASD: An introduction to Model Assisted Software Development: Series of lectures on Model Assisted Software Development (MASD), the methodology put forward in my PhD and implemented by Dogen. This series of lectures was made in preparation for the /viva/.
- MASD - Dogen Demos: end of sprint demos. Suitable for end users.
- MASD - Dogen Coding: Move to GitHub CI: Series of videos on moving Dogen from Travis CI to GitHub CI.
- MASD - Dogen Coding: Org Codec: Series of videos describing the addition of org-mode support to Dogen.
- MASD - Dogen Coding: Relational Model for Tracing: Narrates the design and implementation of the relational model for tracing.
- MASD - Dogen Coding: Identification
Refactor:
Describes some of the woes we went through when updating the code-base to use
the shiny new
identificationmodel. - MASD - Dogen Coding: Formatables Refactor: Series of videos covering the long-running formattables refactor in Dogen, which moved most technical space specific types into their Logical and Physical destinations.
- MASD - MDE Paper of the Week: Series of relevant MDE papers that we've read to get a better understanding of the field. The main thrust of this series was during research, but we may occasionally update it when an interesting paper appears.
Over time, a number of assorted blog posts have been published on Dogen, mainly covering its development process:
- Nerd Food: The Refactoring Quagmire: How we solved the "refactoring ad infinitum" problem in Dogen development.
- Nerd Food: Northwind, or Using Dogen with ODB: Four part series on using Dogen with ODB. Part I, Part II, Part III, Part IV.
- Nerd Food: The Strange Case of the Undefined References: Problems faced using Conan with Dogen.
- Nerd Food: On Product Backlog: The importance of the Product Backlog in Dogen development.
- Nerd Food: Dogen: The Package Management Saga: Dogen's attempt at finding a package management for C++.
- Nerd Food: Pull Request Driven Development: Not directly related to Dogen as we don't get many PRs, but may be of interest; reflections on the importance of a Pull-Request Driven Development approach.
- Nerd Food: Dogen: Lessons in Incremental Coding: How the Dogen development process works, via very small, incremental changes.
The release notes are also worth perusing as contain quite a lot of material about Dogen's internals.
We are strong believers in open academia so all of our papers are open.
- Notes on Model Driven Engineering: Background on MDE.
- Survey of Special Purpose Code Generators: Short survey of Open Source tools dedicated to special purpose code generation.
- Experience Report of Industrial Adoption of Model Driven Development in the Financial Sector: Our prior experiences of using MDE on an industrial setting.
- Model Assisted Software Development - a MDE-Based Software Development Methodology: PhD thesis.
If you have encountered a failure building Dogen or running its unit tests, please submit a bug report. It must include:
- the error messages;
- the versions of all dependencies (compiler, Boost etc);
- the operative system name and version.
If you find a bug whilst using Dogen with one of your models, please also include:
- a small model reproducing the problem, called the MWE (Minimum Working Example).
- the compressed log. By default, dogen creates a directory structure for all of
its byproducts, in the folder it was executed from. It has the form
dogen.byproducts/cli.generate.MODEL, whereMODELis the file name and extension of the generated model. The log file is stored in this directory, with a name such ascli.generate.MODEL.log.
Bugs can be submitted using github Issues.