apache · graceguo-supercat · Oct 17, 2019 · Oct 17, 2019 · Oct 17, 2019 · Oct 17, 2019
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -16,11 +16,21 @@
 #
 repos:
 -   repo: https://github.com/ambv/black
-    rev: stable
+    rev: 19.3b0
     hooks:
     -   id: black
         language_version: python3
 
+-   repo: https://github.com/asottile/seed-isort-config
+    rev: v1.9.3
+    hooks:
+    -   id: seed-isort-config
+
+-   repo: https://github.com/pre-commit/mirrors-isort
+    rev: v4.3.21
+    hooks:
+    -   id: isort
+
 -   repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v2.2.3
     hooks:
@@ -30,8 +40,3 @@ repos:
     -   id: check-added-large-files
     -   id: check-yaml
     -   id: debug-statements
-
--   repo: https://gitlab.com/pycqa/flake8
-    rev: 3.7.1
-    hooks:
-    -   id: flake8
diff --git a/.travis.yml b/.travis.yml
@@ -75,7 +75,10 @@ jobs:
       env: TOXENV=black
     - language: python
       python: 3.6
-      env: TOXENV=flake8
+      env: TOXENV=isort
+    - language: python
+      python: 3.6
+      env: TOXENV=mypy
     - language: python
       python: 3.6
       env: TOXENV=py36-sqlite

diff --git a/CHANGELOG.md b/CHANGELOG.md
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
diff --git a/RELEASING/Dockerfile.from_tarball b/RELEASING/Dockerfile.from_tarball
@@ -60,7 +60,7 @@ RUN pip install --upgrade setuptools pip \
     && pip install -e . \
     && rm -rf /root/.cache/pip
 
-RUN fabmanager babel-compile --target superset/translations
+RUN flask fab babel-compile --target superset/translations
 
 RUN pip install -e . \
     && rm -rf /root/.cache/pip

diff --git a/RELEASING/README.md b/RELEASING/README.md
@@ -19,10 +19,16 @@ under the License.
 
 # Apache Releases
 
-You'll probably want to run these commands manually and understand what
+Until things settle and we create scripts that streamline this,
+you'll probably want to run these commands manually and understand what
 they do prior to doing so.
 
-## Release setup
+For coordinating on releases, on operational topics that require more
+synchronous communications, we recommend using the `#apache-releases` channel
+on the Superset Slack. People crafting releases and those interested in
+partaking in the process should join the channel.
+
+## Release setup (First Time Only)
 
 First you need to setup a few things. This is a one-off and doesn't
 need to be done at every release.
@@ -40,69 +46,171 @@ need to be done at every release.
 
 
     # Add your GPG pub key to KEYS file. Replace "Maxime Beauchemin" with your name
-    export FULLNAME="Maxime Beauchemin"
-    (gpg --list-sigs $FULLNAME && gpg --armor --export $FULLNAME ) >> KEYS
+    export SUPERSET_PGP_FULLNAME="Maxime Beauchemin"
+    (gpg --list-sigs "${SUPERSET_PGP_FULLNAME}" && gpg --armor --export "${SUPERSET_PGP_FULLNAME}" ) >> KEYS
 
 
     # Commit the changes
     svn commit -m "Add PGP keys of new Superset committer"
 ```
 
-## Crafting tarball and signatures
+## Crafting a source release
+
+When crafting a new minor or major release we create 
+a branch named with the release MAJOR.MINOR version.
+This new branch will hold all PATCH and release candidates 
+that belong to the MAJOR.MINOR version.
+
+The MAJOR.MINOR branch is normally a "cut" from a specific point in time from the master branch.
+Then (if needed) apply all cherries that will make the PATCH
+
+Finally bump the version number on `superset/static/assets/package.json` ::
+
+    "version": "0.35.0"
+
+Commit the change with the version number, then git tag the version with the release candidate and push
+
+## Setting up the release environment (do every time)
+
+As the vote process takes a minimum of 72h (community vote) + 72h (IPMC) vote,
+often stretching over several weeks calendar time if votes don't pass, chances are
+the same terminal session won't be used for crafting the release candidate and the
+final release. Therefore, it's a good idea to do the following every time you
+work on a new phase of the release process to make sure you aren't releasing
+the wrong files/using wrong names:
 
-Now let's craft a source release
 ```bash
-    # Assuming these commands are executed from the root of the repo
-    export REPO_DIR=$(pwd)
-    # Set VERSION to the release being prepared (rc1 for first vote on version)
-    export VERSION=0.34.1rc1
-    export RELEASE=apache-superset-incubating-${VERSION}
-    export RELEASE_TARBALL=${RELEASE}-source.tar.gz
+    # Set SUPERSET_VERSION to the release being prepared, e.g. 0.34.1.
+    export SUPERSET_VERSION=XX.YY.ZZ
+    # Set RC to the release candindate number. Replacing QQ below with 1
+    # indicates rc1 i.e. first vote on version above (0.34.1rc1)
+    export SUPERSET_RC=QQ
+```
+
+Then you can generate other derived environment variables that are used
+throughout the release process:
+
+```bash
+    # Replace SUPERSET_PGP_FULLNAME with your PGP key name for Apache
+    export SUPERSET_PGP_FULLNAME="YOURFULLNAMEHERE"
+    export SUPERSET_VERSION_RC=${SUPERSET_VERSION}rc${SUPERSET_RC}
+    export SUPERSET_RELEASE=apache-superset-incubating-${SUPERSET_VERSION}
+    export SUPERSET_RELEASE_RC=apache-superset-incubating-${SUPERSET_VERSION_RC}
+    export SUPERSET_RELEASE_TARBALL=${SUPERSET_RELEASE}-source.tar.gz
+    export SUPERSET_RELEASE_RC_TARBALL=${SUPERSET_RELEASE_RC}-source.tar.gz
+```
+
+## Preparing the release candidate
+
+The first step of preparing an Apache Release is packaging a release candidate
+to be voted on. Start by going to the root of the repo and making sure the
+prerequisites are in order:
+
+```bash
+    # Go to the root directory of the repo, e.g. `~/src/incubator-superset`
+    cd ~/src/incubator-superset/
+    export SUPERSET_REPO_DIR=$(pwd)
+    # make sure you're on the correct branch (e.g. 0.34)
+    git branch
+```
 
+Make sure the version number under `superset/assets/package.json` corresponds
+to `SUPERSET_VERSION` above (`0.34.1` in example above), and has been committed to the
+branch.
+
+```bash
+    grep ${SUPERSET_VERSION} superset/assets/package.json
+```
+
+If nothing shows up, either the version isn't correctly set in `package.json`,
+or the environment variable is misconfigured.
+
+### Crafting tarball and signatures
+
+Now let's craft a source release
+```bash
     # Let's create a git tag
-    git tag -f ${VERSION}
+    git tag -f ${SUPERSET_VERSION_RC}
 
     # Create the target folder
-    mkdir -p ~/svn/superset_dev/${VERSION}/
+    mkdir -p ~/svn/superset_dev/${SUPERSET_VERSION_RC}/
     git archive \
-        --format=tar.gz ${VERSION} \
-        --prefix="${RELEASE}/" \
-        -o ~/svn/superset_dev/${VERSION}/${RELEASE_TARBALL}
+        --format=tar.gz ${SUPERSET_VERSION_RC} \
+        --prefix="${SUPERSET_RELEASE_RC}/" \
+        -o ~/svn/superset_dev/${SUPERSET_VERSION_RC}/${SUPERSET_RELEASE_RC_TARBALL}
+
+    cd ~/svn/superset_dev/${SUPERSET_VERSION_RC}/
+    ${SUPERSET_REPO_DIR}/scripts/sign.sh "${SUPERSET_RELEASE_RC_TARBALL}" "${SUPERSET_PGP_FULLNAME}"
+
+    # To verify to signature
+    gpg --verify "${SUPERSET_RELEASE_RC_TARBALL}".asc "${SUPERSET_RELEASE_RC_TARBALL}"
 
-    cd ~/svn/superset_dev/${VERSION}/
-    ${REPO_DIR}/scripts/sign.sh ${RELEASE}-source.tar.gz
 ```
 
-## Shipping to SVN
+### Shipping to SVN
 
 Now let's ship this RC into svn's dev folder
 
 ```bash
     cd ~/svn/superset_dev/
-    svn add ${VERSION}
-    svn commit -m "${VERSION}"
+    svn add ${SUPERSET_VERSION_RC}
+    svn commit -m "Release ${SUPERSET_VERSION_RC}"
 ```
 
-Now you're ready to start the VOTE thread.
+### Voting
+Now you're ready to start the [VOTE] thread. Here's an example of a
+previous release vote thread:
+https://lists.apache.org/thread.html/e60f080ebdda26896214f7d3d5be1ccadfab95d48fbe813252762879@<dev.superset.apache.org>
+
+Once 3+ binding votes (by PMC members) have been cast and at
+least 72 hours have past, you can post a [RESULT] thread:
+https://lists.apache.org/thread.html/50a6b134d66b86b237d5d7bc89df1b567246d125a71394d78b45f9a8@%3Cdev.superset.apache.org%3E
+
+Following the result thread, yet another [VOTE] thread should be
+started at general@incubator.apache.org.
+
+### Announcing
+
+Once it's all done, an [ANNOUNCE] thread announcing the release to the dev@ mailing list is the final step.
 
-## Validating a release
+### Validating a release
 
 https://www.apache.org/info/verification.html
 
 ## Publishing a successful release
 
-Upon a successful vote, you'll have to copy the folder into the non-"dev/"
-folder.
+Upon a successful vote (community AND IPMC), you'll have to copy the folder
+into the non-"dev/" folder.
 ```bash
-    cp -r ~/svn/superset_dev/${VERSION}/ ~/svn/superset/${VERSION}/
+    cp -r ~/svn/superset_dev/${SUPERSET_VERSION_RC}/ ~/svn/superset/${SUPERSET_VERSION}/
     cd ~/svn/superset/
-    svn add ${VERSION}
-    svn commit -m "${VERSION}"
+    # Rename the RC (0.34.1rc1) to the actual version being released (0.34.1)
+    for f in ${SUPERSET_VERSION}/*; do mv "$f" "${f/${SUPERSET_VERSION_RC}/${SUPERSET_VERSION}}"; done
+    svn add ${SUPERSET_VERSION}
+    svn commit -m "Release ${SUPERSET_VERSION}"
+```
+
+Then tag the final release:
+```bash
+    # Go to the root directory of the repo, e.g. `~/src/incubator-superset`
+    cd ~/src/incubator-superset/
+    # make sure you're on the correct branch (e.g. 0.34)
+    git branch
+    # Create the release tag
+    git tag -f ${SUPERSET_VERSION}
 ```
 
 Now you can announce the release on the mailing list, make sure to use the
 proper template
 
+### Publishing a Convenience Release to PyPI
+From the root of the repo running ./pypi_push.sh will build the
+Javascript bundle and echo the twine command allowing you to publish
+to PyPI. You may need to ask a fellow committer to grant
+you access to it if you don't have access already. Make sure to create
+an account first if you don't have one, and reference your username
+while requesting access to push packages.
+
 ## Post release
 
 In `UPDATING.md`, a file that contains a list of notifications around
@@ -115,11 +223,10 @@ section for the new release.
 To make a working build given a tarball
 ```bash
 # Building a docker from a tarball
-VERSION=0.33.0rc2 && \
-docker build -t apache-superset:$VERSION -f Dockerfile.from_tarball . --build-arg VERSION=$VERSION
+docker build -t apache-superset:${SUPERSET_VERSION_RC} -f Dockerfile.from_tarball . --build-arg VERSION=${SUPERSET_VERSION_RC}
 
 # testing the resulting docker
-docker run -p 5001:8088 apache-superset:$VERSION
+docker run -p 5001:8088 apache-superset:${SUPERSET_VERSION_RC}
 # you should be able to access localhost:5001 on your browser
 # login using admin/admin
 ```

diff --git a/RELEASING/from_tarball_entrypoint.sh b/RELEASING/from_tarball_entrypoint.sh
@@ -21,8 +21,7 @@ echo "[WARNING] this entrypoint creates an admin/admin user"
 echo "[WARNING] it should only be used for lightweight testing/validation"
 
 # Create an admin user (you will be prompted to set username, first and last name before setting a password)
-fabmanager create-admin \
-    --app superset \
+flask fab create-admin \
     --username admin \
     --firstname admin \
     --lastname admin \

diff --git a/UPDATING.md b/UPDATING.md
@@ -23,6 +23,15 @@ assists people when migrating to a new version.
 
 ## Next Version
 
+* [8450](https://github.com/apache/incubator-superset/pull/8450): The time ranger picker
+now uses UTC for the tooltips and default placeholder timestamps (sans timezone).
+
+* [8370](https://github.com/apache/incubator-superset/pull/8370): Deprecates
+  the `HTTP_HEADERS` variable in favor of `DEFAULT_HTTP_HEADERS` and
+  `OVERRIDE_HTTP_HEADERS`. To retain the same behavior you should use
+  `OVERRIDE_HTTP_HEADERS` instead of `HTTP_HEADERS`. `HTTP_HEADERS` will still
+  work but may be removed in a future update.
+
 * We're deprecating the concept of "restricted metric", this feature
   was not fully working anyhow.
 * [8117](https://github.com/apache/incubator-superset/pull/8117): If you are
@@ -63,16 +72,18 @@ which adds missing non-nullable fields to the `datasources` table. Depending on
 the integrity of the data, manual intervention may be required.
 
 * [5452](https://github.com/apache/incubator-superset/pull/5452): a change
-which adds missing non-nullable fields and uniqueness constraints to the
-`columns`and `table_columns` tables. Depending on the integrity of the data,
-manual intervention may be required.
+which adds missing non-nullable fields and uniqueness constraints (which may be 
+case insensitive depending on your database configuration) to the `columns`and 
+`table_columns` tables. Depending on the integrity of the data, manual 
+intervention may be required.
 * `fabmanager` command line is deprecated since Flask-AppBuilder 2.0.0, use
 the new `flask fab <command>` integrated with *Flask cli*.
 * `SUPERSET_UPDATE_PERMS` environment variable was replaced by
 `FAB_UPDATE_PERMS` config boolean key. To disable automatic
 creation of permissions set `FAB_UPDATE_PERMS = False` on config.
 * [5453](https://github.com/apache/incubator-superset/pull/5453): a change
-which adds missing non-nullable fields and uniqueness constraints to the metrics
+which adds missing non-nullable fields and uniqueness constraints (which may be 
+case insensitive depending on your database configuration) to the metrics
 and sql_metrics tables. Depending on the integrity of the data, manual
 intervention may be required.
 * [7616](https://github.com/apache/incubator-superset/pull/7616): this bug fix

diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile
@@ -47,8 +47,8 @@ COPY requirements.txt .
 COPY requirements-dev.txt .
 COPY contrib/docker/requirements-extra.txt .
 
-RUN pip install --upgrade setuptools pip \
-    && pip install -r requirements.txt -r requirements-dev.txt -r requirements-extra.txt \
+RUN pip --default-timeout=120 install --upgrade setuptools pip \
+    && pip --default-timeout=120 install -r requirements.txt -r requirements-dev.txt -r requirements-extra.txt \
     && rm -rf /root/.cache/pip
 
 COPY --chown=superset:superset superset superset

diff --git a/docs/tutorial.rst → docs/admintutorial.rst b/docs/tutorial.rst → docs/admintutorial.rst
@@ -15,8 +15,8 @@
     specific language governing permissions and limitations
     under the License.
 
-Tutorial - Creating your first dashboard
-========================================
+Creating your first dashboard
+=============================
 
 This tutorial targets someone who wants to create charts and dashboards
 in Superset. We'll show you how to connect Superset

diff --git a/docs/index.rst b/docs/index.rst
@@ -96,7 +96,7 @@ Features
 Databases
 ---------
 
-The following RDBMS are currently suppored:
+The following RDBMS are currently supported:
 
 - `Amazon Athena <https://aws.amazon.com/athena/>`_
 - `Amazon Redshift <https://aws.amazon.com/redshift/>`_
@@ -109,6 +109,7 @@ The following RDBMS are currently suppored:
 - `Apache Spark SQL <https://spark.apache.org/sql/>`_
 - `BigQuery <https://cloud.google.com/bigquery/>`_
 - `ClickHouse <https://clickhouse.yandex/>`_
+- `Elasticsearch <https://www.elastic.co/products/elasticsearch/>`_
 - `Exasol <https://www.exasol.com/>`_
 - `Google Sheets <https://www.google.com/sheets/about/>`_
 - `Greenplum <https://greenplum.org/>`_
@@ -122,6 +123,7 @@ The following RDBMS are currently suppored:
 - `SQL Server <https://www.microsoft.com/en-us/sql-server/>`_
 - `Teradata <https://www.teradata.com/>`_
 - `Vertica <https://www.vertica.com/>`_
+- `Hana <https://www.sap.com/products/hana.html>`_
 
 Other database engines with a proper DB-API driver and SQLAlchemy dialect should
 be supported as well.
@@ -153,7 +155,7 @@ Contents
     :maxdepth: 2
 
     installation
-    tutorial
+    tutorials
     security
     sqllab
     gallery