define docs (#2)

* define docs

* simplify statement of supported Python versions

CONFIG_FORMAT.rst

@@ -0,0 +1,88 @@
+==================
+Config File Format
+==================
+
+Pipeformer is configured using a YAML file with the below contents.
+
+Structure
+---------
+
+* **name** : string (required) : Project name. Used in names and descriptions to identify resources as belonging to this project.
+* **description** : string (required) : Project description. *Not currently used.*
+* **inputs** : map : Inputs that will be required at stack creation or update.
+
+  * **<input name>** : map : Describes an input value.
+
+    * **description** : string (required) : Used for the input prompt and any resource descriptions as appropriate.
+    * **secret** : boolean (required) : Determines if this input is treated as secret.
+
+* **roles** : map : Additional permissions to apply to generated IAM Roles. *Not currently used.*
+
+* **pipeline** : map : Definition of desired pipeline.
+  Each member defines a stage in the pipeline and is a list of action definitions.
+
+  * <stage name> : list : Description of pipeline stage actions.
+
+    * **provider** : string (required) : `CodePipeline action provider name`_.
+    * **run_order** : int : The order in which CodePipeline runs this action (default: 1).
+    * **inputs** : list of strings : List of CodePipeline input artifact names.
+    * **outputs** : list of strings : List of CodePipeline output artifact names.
+    * **configuration** : map : Additional configuration values to provide to in CodePipeline Action definition.
+    * **image** : string (required for CodeBuild actions): Docker image to use for CodeBuild action.
+    * **environment_type** : string : CodeBuild `environment-type`_ value.
+      *If not provided, we assume Linux unless the image name contains "windows" in any casing.*
+    * **buildspec** : string : Path to buildspec file in source.
+    * **compute_type** : string : CodeBuild `compute-type`_ name. (default: BUILD_GENERAL1_SMALL)
+    * **env** : string-string map : Custom environment variable values to set in action.
+
+Input Value Resolution
+----------------------
+
+Input values can be referenced using strings of the form: ``"{INPUT:VariableName}"``
+
+These values are referenced from their storage locations using `CloudFormation dynamic references`_.
+
+Example
+-------
+
+.. code-block:: yaml
+
+    name: example project
+    description: This is an example project.
+
+    inputs:
+      GitHubToken:
+        description: GitHub user access token that CodePipeline will use to authenticate to GitHub.
+        secret: true
+      GitHubOwner:
+        description: GitHub user that owns target repository.
+        secret: false
+
+    pipeline:
+      source:
+        - provider: GitHub
+          outputs:
+            - SourceOutput
+          configuration:
+            Owner: "{INPUT:GitHubOwner}"
+            Repo: example
+            Branch: master
+            OAuthToken: "{INPUT:GitHubToken}"
+
+      build:
+        - provider: CodeBuild
+          image: aws/codebuild/python:3.6.5
+          buildspec: .chalice/buildspec.yaml
+          env:
+            key1: value2
+            key3: value4
+          inputs:
+            - SourceOutput
+          outputs:
+            - CompiledCfnTemplate
+
+
+.. _CodePipeline action provider name: https://docs.aws.amazon.com/codepipeline/latest/userguide/reference-pipeline-structure.html#actions-valid-providers
+.. _environment-type: https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectEnvironment.html#CodeBuild-Type-ProjectEnvironment-type
+.. _compute-type: https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectEnvironment.html#CodeBuild-Type-ProjectEnvironment-computeType
+.. _CloudFormation dynamic references: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html

README.rst

@@ -0,0 +1,174 @@
+##########
+pipeformer
+##########
+
+.. image:: https://img.shields.io/pypi/v/pipeformer.svg
+   :target: https://pypi.python.org/pypi/pipeformer
+   :alt: Latest Version
+
+.. image:: https://img.shields.io/pypi/pyversions/pipeformer.svg
+   :target: https://pypi.python.org/pypi/pipeformer
+   :alt: Supported Python Versions
+
+.. image:: https://img.shields.io/badge/code_style-black-000000.svg
+   :target: https://github.com/ambv/black
+   :alt: Code style: black
+
+.. image:: https://readthedocs.org/projects/pipeformer/badge/
+   :target: https://pipeformer.readthedocs.io/en/stable/
+   :alt: Documentation Status
+
+.. image:: https://travis-ci.org/awslabs/pipeformer.svg?branch=master
+   :target: https://travis-ci.org/awslabs/pipeformer
+
+.. image:: https://ci.appveyor.com/api/projects/status/REPLACEME/branch/master?svg=true
+   :target: https://ci.appveyor.com/project/REPLACEME
+
+Tool for generating CodePipeline pipelines and related resources from a simple configuration.
+
+
+********
+Abstract
+********
+
+Services like CodePipeline and CodeBuild are great building blocks,
+but can be complicated to set up and use in a consistent way.
+
+CloudFormation makes it possible to create and update resources in a consistent and repeatable way,
+but can be complicated and verbose to define.
+
+The goal of Pipeformer is combine these properties by providing a very simple, but extensible,
+way to use these services with your project.
+
+Tenets
+======
+
+* Simple
+
+  * For the majority of projects, the majority of resource configurations will be identical.
+    Only require the user to set the values that are actually important to them.
+  * The user should not need to know about resources that they will not directly touch.
+
+* Flexible
+
+  * While most users should not need to, users must be able to override most settings if they do need to.
+
+**********
+How to Use
+**********
+
+1. Define your configuration file.
+1. Deploy with pipeformer.
+
+User Experience
+===============
+
+The primary operating mode for pipeformer is to take your configuration,
+use it to generate CloudFormation templates that describe the needed resources,
+and then deploy those templates.
+
+Configuration
+=============
+
+`Configuration File Format <CONFIG_FORMAT.rst>`_
+
+What Does it Do?
+================
+
+`Resources Created <RESOURCES.rst>`_
+
+***************
+Getting Started
+***************
+
+Required Prerequisites
+======================
+
+* Supported Python versions
+
+  * 3.6+
+
+Installation
+============
+
+.. code::
+
+   $ pip install pipeformer
+
+***********
+Development
+***********
+
+Prerequisites
+=============
+
+* Required
+
+  * Python 3.6+
+  * `tox`_ : We use tox to drive all of our testing and package management behavior.
+     Any tests that you want to run should be run using tox.
+
+* Optional
+
+  * `pyenv`_ : If you want to test against multiple versions of Python and are on Linux or MacOS,
+    we recommend using pyenv to manage your Python runtimes.
+  * `tox-pyenv`_ : Plugin for tox that enables it to use pyenv runtimes.
+  * `detox`_ : Parallel plugin for tox. Useful for running a lot of test environments quickly.
+
+Setting up pyenv
+----------------
+
+If you are using pyenv, make sure that you have set up all desired runtimes and configured the environment
+before attempting to run any tests.
+
+1. Install all desired runtimes.
+
+   * ex: ``pyenv install 3.7.0``
+   * **NOTE:** You can only install one runtime at a time with the ``pyenv install`` command.
+
+1. In the root of the checked out repository for this package, set the runtimes that pyenv should use.
+
+   * ex: ``pyenv local 3.7.0 3.6.4``
+   * **NOTE:** This creates the ``.python-version`` file that pyenv will use. Pyenv treats the first
+     version in that file as the default Python version.
+
+
+Running tests
+=============
+
+There are two criteria to consider when running our tests:
+what version of Python do you want to use and what type of tests do you want to run?
+
+For a full listing of the available types of tests available,
+see the ``[testenv]commands`` section of the ``tox.ini`` file.
+
+All tests should be run using tox.
+To do this, identify the test environment that you want tox to run using the ``-e ENV_NAME`` flag.
+The standard test environments are named as a combination of the Python version
+and the test type in the form ``VERSION-TYPE``.
+For example, to run the ``local`` tests against CPython 3.7:
+
+.. code-block:: bash
+
+    tox -e py37-local
+
+If you want to provide custom parameters to pytest to manually identify what tests you want to run,
+use the ``manual`` test type. Any arguments you want to pass to pytest must follow the ``--`` argument.
+Anything before that argument is passed to tox. Everything after that argument is passed to pytest.
+
+.. code-block:: bash
+
+    tox -e py37-manual -- test/unit/test_example_file.py
+
+Before submitting a pull request
+================================
+
+Before submitting a pull request, please run the ``lint`` tox environment.
+This will ensure that your submission meets our code formatting requirements
+and will pass our continous integration code formatting tests.
+
+
+.. _tox: http://tox.readthedocs.io/
+.. _detox: https://pypi.org/project/detox/
+.. _tox-pyenv: https://pypi.org/project/tox-pyenv/
+.. _pyenv: https://github.com/pyenv/pyenv

RESOURCES.rst

@@ -0,0 +1,57 @@
+=================
+Resources Created
+=================
+
+Pipeformer creates all resources necessary to build the pipeline that you describe in your configuration.
+
+Stand-Alone Mode
+----------------
+
+In stand-alone mode, pipeformer assumes that all necessary resources need to be created.
+
+In this operating mode,
+all core resources are defined in a central CloudFormation stack that also contains all other CloudFormation stacks.
+
+Resources
+---------
+
+Core Resources
+^^^^^^^^^^^^^^
+
+The core resources are resources that are needed by all other components.
+These include:
+
+* Application resources S3 bucket made available to the application for use.
+* Artifact S3 bucket for use by Pipeformer and resources.
+* IAM Roles for use within Pipeformer for:
+
+  * CloudFormation
+  * CodePipeline pipelines
+  * CodeBuild projects
+
+* A KMS CMK that is used to protect all resources and data managed by pipeformer.
+
+Inputs
+^^^^^^
+
+Any input values defined in your configuration need to be stored somewhere.
+Pipeformer stores them either in Secrets Manager if the input is marked as secret,
+or SSM Parameter Store if it is not.
+
+All input resources are managed in a separate CloudFormation stack,
+with a separate Secrets Manager Secret or Parameter Store Parameter resource for each input.
+The values stored in these resources are managed outside of CloudFormation.
+
+CodeBuild
+^^^^^^^^^
+
+Pipeline actions that use CodeBuild require a unique CodeBuild project for each action.
+Because of this, and to avoid CloudFormation per-stack resource limits,
+Pipeformer creates a separate CloudFormation stack for each pipeline stage that contains at least one CodeBuild action.
+These stacks contain only CodeBuild resources.
+
+CodePipeline
+^^^^^^^^^^^^
+
+Finally, a CloudFormation stack is created that contains the CodePipeline resource itself.
+All CodeBuild stacks are created as nested stacks of this stack.

TODO.rst

@@ -0,0 +1,50 @@
+****
+TODO
+****
+
+Features
+========
+
+* Add support for updating an existing pipeformer deployment.
+* Implement value expansion behavior
+
+  * List to replica
+  * Brace expansion
+
+* Collapse CloudFormation ``CHANGE_SET_REPLACE`` + ``CHANGE_SET_EXECUTE`` pattern into a single virtual action.
+* Add support for partial deployment (ex: buckets/CMK/roles already exist).
+
+  * This will be necessary for later work.
+
+* Periodically poll CloudFormation for the status of the stack deployment.
+   We need to fail gracefully if the stack create/update fails.
+* Add support for custom Role permissions.
+* Define default values for source outputs and CodeBuild inputs.
+* Add a "preview" mode that outputs the generated CloudFormation templates.
+
+  * How would we preview changes that would be made by an update?
+
+* Add a "destroy" mode.
+* Add support for Lambda actions?
+* Differentiate between "source" and "deploy" S3 actions.
+* Support inline buildspec? I'm inclined to say no.
+* Support image pull credentials?
+
+Design Decisions
+================
+
+* Simplify inputs/outputs configuration for common patterns.
+* Add sentinel resolver values: unique per-system value that can be referenced similarly to input values.
+* Determine how to handle binary inputs (if at all)
+* Should we re-prompt for input values that are already known?
+* The inputs stack will currently overwrite non-secret values. What to do about that?
+* Rather than assuming stack drift on non-secret input updates,
+   should we instead send the values as parameters to the inputs stack?
+
+
+Research
+========
+
+* Do wait conditions re-create on stack updates? (I think no)
+* If I update a stack with a dynamic reference and the only thing that changed is the value in the referenced location,
+   does the stack update? (I think no)