diff --git a/developer/building/development-workflow.md b/developer/building/development-workflow.md index 0146fe3ae..9a01361fd 100644 --- a/developer/building/development-workflow.md +++ b/developer/building/development-workflow.md @@ -6,533 +6,7 @@ redirect_from: - /en/doc/development-workflow/ - /doc/DevelopmentWorkflow/ - /wiki/DevelopmentWorkflow/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/building/development-workflow.html ref: 66 title: Development workflow ---- - -A workflow for developing Qubes OS+ - -First things first, setup [QubesBuilder](/doc/qubes-builder/). This guide -assumes you're using qubes-builder to build Qubes. - -# Repositories and committing Code - -Qubes is split into a bunch of git repos. These are all contained in the -`qubes-src` directory under qubes-builder. Subdirectories there are separate -components, stored in separate git repositories. - -The best way to write and contribute code is to create a git repo somewhere -(e.g., github) for the repo you are interested in editing (e.g., -`qubes-manager`, `core-agent-linux`, etc). To integrate your repo with the rest -of Qubes, cd to the repo directory and add your repository as a remote in git - -**Example:** - -~~~ -$ cd qubes-builder/qubes-src/qubes-manager -$ git remote add abel git@github.com:abeluck/qubes-manager.git -~~~ - -You can then proceed to easily develop in your own branches, pull in new -commits from the dev branches, merge them, and eventually push to your own repo -on github. - -When you are ready to submit your changes to Qubes to be merged, push your -changes, then create a signed git tag (using `git tag -s`). Finally, send a -letter to the Qubes listserv describing the changes and including the link to -your repository. You can also create pull request on github. Don't forget to -include your public PGP key you use to sign your tags. - -### Kernel-specific notes - -#### Prepare fresh version of kernel sources, with Qubes-specific patches applied - -In qubes-builder/qubes-src/linux-kernel: - -~~~ -make prep -~~~ - -The resulting tree will be in kernel-\/linux-\: - -~~~ -ls -ltrd kernel*/linux* -~~~ - -~~~ -drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18 -drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj -~~~ - -#### Go to the kernel tree and update the version - -In qubes-builder/qubes-src/linux-kernel: - -~~~ -cd kernel-3.4.18/linux-3.4.18 -~~~ - -#### Changing the config - -In kernel-3.4.18/linux-3.4.18: - -~~~ -cp ../../config .config -make oldconfig -~~~ - -Now change the configuration. For example, in kernel-3.4.18/linux-3.4.18: - -~~~ -make menuconfig -~~~ - -Copy the modified config back into the kernel tree: - -~~~ -cp .config ../../../config -~~~ - -#### Patching the code - -TODO: describe the workflow for patching the code, below are some random notes, not working well - -~~~ -ln -s ../../patches.xen -export QUILT_PATCHES=patches.xen -export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index" -export QUILT_SERIES=../../series-pvops.conf - -quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch -quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \ - drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \ - include/xen/interface/io/usbif.h - -*edit something* - -quilt refresh -cd ../.. -vi series.conf -~~~ - -#### Building RPMs - -TODO: Is this step generic for all subsystems? - -Now it is a good moment to make sure you have changed kernel release name in -rel file. For example, if you change it to '1debug201211116c' the -resulting RPMs will be named -'kernel-3.4.18-1debug20121116c.pvops.qubes.x86\_64.rpm'. This will help -distinguish between different versions of the same package. - -You might want to take a moment here to review (git diff, git status), commit -your changes locally. - -To actually build RPMs, in qubes-builder: - -~~~ -make linux-kernel -~~~ - -RPMs will appear in qubes-src/linux-kernel/pkgs/fc20/x86\_64: - -~~~ --rw-rw-r-- 1 user user 42996126 Nov 17 04:08 kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 43001450 Nov 17 05:36 kernel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 8940138 Nov 17 04:08 kernel-devel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 8937818 Nov 17 05:36 kernel-devel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 54490741 Nov 17 04:08 kernel-qubes-vm-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 54502117 Nov 17 05:37 kernel-qubes-vm-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm -~~~ - -### Useful [QubesBuilder](/doc/qubes-builder/) commands - -1. `make check` - will check if all the code was committed into repository and -if all repository are tagged with signed tag. -2. `make show-vtags` - show version of each component (based on git tags) - -mostly useful just before building ISO. **Note:** this will not show version -for components containing changes since last version tag. -3. `make push` - push change from **all** repositories to git server. You must -set proper remotes (see above) for all repositories first. -4. `make prepare-merge` - fetch changes from remote repositories (can be -specified on commandline via GIT\_SUBDIR or GIT\_REMOTE vars), (optionally) -verify tags and show the changes. This do not merge the changes - there are -left for review as FETCH\_HEAD ref. You can merge them using `git merge -FETCH_HEAD` (in each repo directory). Or `make do-merge` to merge all of them. - -## Copying Code to dom0 - -When developing it is convenient to be able to rapidly test changes. Assuming -you're developing Qubes on Qubes, you should be working in a special VM for -Qubes and occasionally you will want to transfer code or RPMs back to dom0 for -testing. - -Here are some handy scripts Marek has shared to facilitate this. - -You may also like to run your [test environment on separate -machine](/doc/test-bench/). - -### Syncing dom0 files - -TODO: edit this script to be more generic - -~~~ -#!/bin/sh - -set -x -set -e - -QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes -QUBES_PY=$QUBES_PY_DIR/qubes.py -QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py - -qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv -cp $QUBES_PY qubes.py.bak$$ -cp $QUBESUTILS_PY qubesutils.py.bak$$ -cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$ -sudo cp qvm-core/qubes.py $QUBES_PY -sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY -sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/ -sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/ -sudo cp misc/vm-template-hvm.conf /usr/share/qubes/ -sudo cp misc/qubes-start.desktop /usr/share/qubes/ -sudo cp misc/block-snapshot /etc/xen/scripts/ -sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/ -# FIXME(Abel Luck): I hope to -~~~ - -### Apply qvm-tools - -TODO: make it more generic - -~~~ -#!/bin/sh - -BAK=qvm-tools.bak$$ -mkdir -p $BAK -cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/ -sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/ -~~~ - -### Copy from dom0 to an appvm - -~~~ -#/bin/sh -# -# usage ./cp-domain -# -domain=$1 -file=$2 -fname=`basename $file` - -qvm-run $domain 'mkdir /home/user/incoming/dom0 -p' -cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname" -~~~ - -## Git connection between VMs - -Sometimes it's useful to transfer git commits between VMs. You can use `git -format-patch` for that and simply copy the files. But you can also setup -custom qrexec service for it. - -Below example assumes that you use `builder-RX` directory in target VM to -store sources in qubes-builder layout (where `X` is some number). Make sure that -all the scripts are executable. - -Service file (save in `/usr/local/etc/qubes-rpc/local.Git` in target VM): - -~~~ -#!/bin/sh - -exec 2>/tmp/log2 - -read service rel repo -echo "Params: $service $rel $repo" >&2 -# Adjust regexps if needed -echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1 -echo "$rel" | grep -q '^[0-9.]\+$' || exit 1 -path="/home/user/builder-R$rel/qubes-src/$repo" -if [ "$repo" = "builder" ]; then - path="/home/user/builder-R$rel" -fi -case $service in - git-receive-pack|git-upload-pack) - echo "starting $service $path" >&2 - exec $service $path - ;; - *) - echo "Unsupported service: $service" >&2 - ;; -esac -~~~ - -Client script (save in `~/bin/git-qrexec` in source VM): - -~~~ -#!/bin/sh - -VMNAME=$1 - -(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git -~~~ - -You will also need to setup qrexec policy in dom0 (`/etc/qubes-rpc/policy/local.Git`). - -Usage: - -~~~ -[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux" -[user@source core-agent-linux]$ git push testbuilder master -~~~ - -You can create `~/bin/add-remote` script to ease adding remotes: - -~~~ -#!/bin/sh - -[ -n "$1" ] || exit 1 - -if [ "$1" = "tb" ]; then - git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`" - exit $? -fi - -git remote add $1 git@github.com:$1/qubes-`basename $PWD` -~~~ - -It should be executed from component top level directory. This script takes one -argument - remote name. If it is `tb`, then it creates qrexec-based git remote -to `testbuilder` VM. Otherwise it creates remote pointing at github account of -the same name. In any case it points at repository matching current directory -name. - -## Sending packages to different VM - -Other useful script(s) can be used to setup local package repository hosted in -some VM. This way you can keep your development VM behind firewall, while -having an option to expose some yum/apt repository to the local network (to -have them installed on test machine). - -To achieve this goal, a dummy repository can be created, which instead of -populating metadata locally, will upload the packages to some other VM and -trigger repository update there (using qrexec). You can use `unstable` -repository flavor, because there is no release managing rules bundled (unlike -current and current-testing). - -### RPM packages - yum repo - -In source VM, grab [linux-yum](https://github.com/QubesOS/qubes-linux-yum) repository (below is assumed you've made it in -`~/repo-yum-upload` directory) and replace `update_repo.sh` script with: - -~~~ -#!/bin/sh - -VMNAME=repo-vm - -set -e -qvm-copy-to-vm $VMNAME $1 -# remove only files, leave directory structure -find -type f -name '*.rpm' -delete -# trigger repo update -qrexec-client-vm $VMNAME local.UpdateYum -~~~ - -In target VM, setup actual yum repository (also based on [linux-yum](https://github.com/QubesOS/qubes-linux-yum), this time -without modifications). You will also need to setup some gpg key for signing -packages (it is possible to force yum to install unsigned packages, but it -isn't possible for `qubes-dom0-update` tool). Fill `~/.rpmmacros` with -key description: - -~~~ -%_gpg_name Test packages signing key -~~~ - -Then setup `local.UpdateYum` qrexec service (`/usr/local/etc/qubes-rpc/local.UpdateYum`): - -~~~ -#!/bin/sh - -if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then - exit 1 -fi - -real_repository=/home/user/linux-yum -incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN - -find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1 - -rsync -lr --remove-source-files $incoming/ $real_repository -cd $real_repository -export SKIP_REPO_CHECK=1 -if [ -d $incoming/r3.1 ]; then - ./update_repo-unstable.sh r3.1 -fi - -if [ -d $incoming/r3.0 ]; then - ./update_repo-unstable.sh r3.0 -fi - -if [ -d $incoming/r2 ]; then - ./update_repo-unstable.sh r2 -fi -find $incoming -type d -empty -delete -exit 0 -~~~ - -Of course you will also need to setup qrexec policy in dom0 -`/etc/qubes-rpc/policy/local.UpdateYum`. - -If you want to access the repository from network, you need to setup HTTP -server serving it, and configure the system to let other machines actually -reach this HTTP server. You can use for example using [port -forwarding](/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world) or setting up Tor hidden service. Configuration -details of those services are outside of the scope of this page. - -Usage: setup `builder.conf` in source VM to use your dummy-uploader repository: - -~~~ -LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1 -~~~ - -Then use `make update-repo-unstable` to upload the packages. You can also -specify selected components on command line, then build them and upload to the -repository: - -~~~ -make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable -~~~ - -On the test machine, add yum repository (`/etc/yum.repos.d`) pointing at just -configured HTTP server. For example: - -~~~ -[local-test] -name=Test -baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20 -~~~ - -Remember to also import gpg public key using `rpm --import`. - -### Deb packages - Apt repo - -Steps are mostly the same as in the case of yum repo. The only details that differ: - -- use [linux-deb](https://github.com/QubesOS/qubes-linux-deb) instead of [linux-yum](https://github.com/QubesOS/qubes-linux-yum) as a base - both in source and target VM -- use different `update_repo.sh` script in source VM (below) -- use `local.UpdateApt` qrexec service in target VM (code below) -- in target VM additionally place `update-local-repo.sh` script in repository dir (code below) - -`update_repo.sh` script: - -~~~ -#!/bin/sh - -set -e - -current_release=$1 -VMNAME=repo-vm - -qvm-copy-to-vm $VMNAME $1 -find $current_release -type f -name '*.deb' -delete -rm -f $current_release/vm/db/* -qrexec-client-vm $VMNAME local.UpdateApt -~~~ - -`local.UpdateApt` service code (`/usr/local/etc/qubes-rpc/local.UpdateApt` in repo-serving VM): - -~~~ -#!/bin/sh - -if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then - exit 1 -fi - -incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN - -rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/ -cd /home/user/linux-deb -export SKIP_REPO_CHECK=1 -if [ -d $incoming/r3.1 ]; then - for dist in `ls r3.1/vm/dists`; do - ./update-local-repo.sh r3.1/vm $dist - done -fi - -if [ -d $incoming/r3.0 ]; then - for dist in `ls r3.0/vm/dists`; do - ./update-local-repo.sh r3.0/vm $dist - done -fi - -if [ -d $incoming/r2 ]; then - for dist in `ls r2/vm/dists`; do - ./update-local-repo.sh r2/vm $dist - done -fi -find $incoming -type d -empty -delete -exit 0 -~~~ - -`update-local-repo.sh`: - -~~~ -#!/bin/sh - -set -e - -# Set this to your local repository signing key -SIGN_KEY=01ABCDEF - -[ -z "$1" ] && { echo "Usage: $0 "; exit 1; } - -REPO_DIR=$1 -DIST=$2 - -if [ "$DIST" = "wheezy-unstable" ]; then - DIST_TAG=deb7 -elif [ "$DIST" = "jessie-unstable" ]; then - DIST_TAG=deb8 -elif [ "$DIST" = "stretch-unstable" ]; then - DIST_TAG=deb9 -fi - -pushd $REPO_DIR -mkdir -p dists/$DIST/main/binary-amd64 -dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages -gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz -cat > dists/$DIST/Release <> dists/$DIST/Release - -rm -f $DIST/Release.gpg -rm -f $DIST/InRelease -gpg -abs -u "$SIGN_KEY" \ - < dists/$DIST/Release > dists/$DIST/Release.gpg -gpg -a -s --clearsign -u "$SIGN_KEY" \ - < dists/$DIST/Release > dists/$DIST/InRelease -popd - -if [ `id -u` -eq 0 ]; then - chown -R --reference=$REPO_DIR $REPO_DIR -fi -~~~ - -Usage: add this line to `/etc/apt/sources.list` on test machine (adjust host and path): - -~~~ -deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main -~~~ +--- \ No newline at end of file diff --git a/developer/building/qubes-builder-details.md b/developer/building/qubes-builder-details.md index f3582284b..73e87729e 100644 --- a/developer/building/qubes-builder-details.md +++ b/developer/building/qubes-builder-details.md @@ -6,48 +6,7 @@ redirect_from: - /en/doc/qubes-builder-details/ - /doc/QubesBuilderDetails/ - /wiki/QubesBuilderDetails/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/building/qubes-builder-details.html ref: 65 title: Qubes builder details ---- - -Components Makefile.builder file --------------------------------- - -[QubesBuilder](/doc/qubes-builder/) expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.\* in [QubesBuilder](/doc/qubes-builder/). - -Variables for Linux build: - -- `RPM_SPEC_FILES` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. [QubesBuilder](/doc/qubes-builder/) will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm\_spec* directory. This is mainly used for Fedora packages build. -- `ARCH_BUILD_DIRS` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, [QubesBuilder](/doc/qubes-builder/) will install all makedepends, then build the package. - -Most components uses *archlinux* directory for this purpose, so its good to keep this style. - -Variables for Windows build: - -- `WIN_COMPILER` Choose which compiler should be used for this component, thus which build scripts. Currently two options available: - - `WDK` - Windows Driver Kit (default). Command used to build: *build -cZg*. - - `mingw` - MinGW (Windows gcc port). Command used to build: *make all* -- `WIN_SOURCE_SUBDIRS` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*). -- `WIN_PREBUILD_CMD` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc. -- `WIN_SIGN_CMD` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don't want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables: - - `CERT_FILENAME` Path to key file (pfx format) - - `CERT_PASSWORD` Key password - - `CERT_PUBLIC_FILENAME` Certificate path in the case of self-signed cert - - `CERT_CROSS_CERT_FILENAME` Certificate path in the case of correct autheticode cert - - `SIGNTOOL` Path to signtool -- `WIN_PACKAGE_CMD` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don't want this command. -- `WIN_OUTPUT_HEADERS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with public headers of the package - for use in other components. -- `WIN_OUTPUT_LIBS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with libraries (both DLL and implib) of the package - for use in other components. Note that [QubesBuilder](/doc/qubes-builder/) will copy files specified as *\$(WIN\_OUTPUT\_LIBS)/\*/\** to match WDK directory layout (*\/\/\*), so you in mingw build you need to place libraries in some additional subdirectory. -- `WIN_BUILD_DEPS` List of components required to build this one. [QubesBuilder](/doc/qubes-builder/) will copy files specified with `WIN_OUTPUT_HEADERS` and `WIN_OUTPUT_LIBS` of those components to some directory and provide its path with `QUBES_INCLUDES` and `QUBES_LIBS` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty. - -builder.conf settings ---------------------- - -Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration. - -**TODO** - -Notes ------ - -* For a list of custom TemplateVMs available in QubesBuilder look at [Supported Versions page](/doc/supported-releases/). +--- \ No newline at end of file diff --git a/developer/building/qubes-builder.md b/developer/building/qubes-builder.md index 572b98667..8bc47b3c2 100644 --- a/developer/building/qubes-builder.md +++ b/developer/building/qubes-builder.md @@ -6,206 +6,7 @@ redirect_from: - /en/doc/qubes-builder/ - /doc/QubesBuilder/ - /wiki/QubesBuilder/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/building/qubes-builder.html ref: 64 title: Qubes builder ---- - -**Note: See [ISO building instructions](/doc/qubes-iso-building/) for a streamlined overview on how to use the build system.** - - -We have a fully automated build system for Qubes, that downloads, builds and -packages all the Qubes components, and finally should spit out a ready-to-use -installation ISO, all in a [secure](/news/2016/05/30/build-security/) way. - -In order to use it, you should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed: - -- sudo -- gnupg -- git -- createrepo -- rpm-build -- make -- wget -- rpmdevtools -- python3-sh -- dialog -- rpm-sign -- dpkg-dev -- debootstrap -- python3-pyyaml -- devscripts -- perl-Digest-MD5 -- perl-Digest-SHA - -Usually you can install those packages by just issuing: - -```shell -sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA -``` - -The build system creates build environments in chroots and so no other packages are needed on the host. -All files created by the build system are contained within the qubes-builder directory. -The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory. - -The build system is configured via builder.conf file. -You can use the setup.sh script to create and modify this file. -Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.: - -```shell -cp example-configs/qubes-os-master.conf builder.conf -# edit the builder.conf file and set the following variables: -NO_SIGN=1 -``` - -One additional useful requirement is that 'sudo root' must work without any prompt, which is default on most distros (e.g. 'sudo bash' brings you the root shell without asking for any password). -This is important as the builder needs to switch to root and then back to user several times during the build process. - -Additionally, if building with signing enabled (NO\_SIGN is not set), you must adjust `\~/.rpmmacros` file so that it points to the GPG key used for package signing, e.g.: - -```bash -%_signature gpg -%_gpg_path /home/user/.gnupg -%_gpg_name AC1BF9B3 # <-- Key ID used for signing -``` - -It is also recommended that you use an empty passphrase for the private key used for signing. -Contrary to a popular belief, this doesn't affect your key or sources security -- if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not. - -So, to build Qubes you would do: - -```shell -# Import the Qubes master key -gpg --recv-keys 0xDDFA1A3E36879494 - -# Verify its fingerprint, set as 'trusted'. -# This is described here: -# https://www.qubes-os.org/doc/VerifyingSignatures - -wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc -gpg --import qubes-developers-keys.asc - -git clone https://github.com/QubesOS/qubes-builder.git qubes-builder -cd qubes-builder - -# Verify its integrity: -git tag -v `git describe` - -cp example-configs/qubes-os-master.conf builder.conf -# edit the builder.conf file and set the following variables: -# NO_SIGN="1" - -# Download all components: - -make get-sources - -# And now to build all Qubes RPMs (this will take a few hours): - -make qubes - -# ... and then to build the ISO - -make iso -``` - -And this should produce a shiny new ISO. - -You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon: - -```shell -make gui-daemon -``` - -You can get a full list from make help. - -## Making customized build - -### Manual source modification - -You can also modify sources somehow if you wish. -Here are some basic steps: - -1. Download qubes-builder as described above (if you want to use marmarek's branches, you should also download qubes-builder from his repo - replace 'QubesOS' with 'marmarek' in above git clone command) -2. Edit builder.conf (still the same as above), some useful additions: - - You can also set GIT\_PREFIX="marmarek/qubes-" to use marmarek's repo instead of "mainstream" - it contains newer (but less tested) versions -3. Download unmodified sources - - ```shell - make get-sources - ``` - -4. **Make your modifications here** - -5. Build the Qubes - `make qubes` actually is just meta target which builds all required - components in correct order. The list of components is configured in - builder.conf. You can also check the current value at the end of `make - help`, or using `make build-info`. - -6. `get-sources` is already done, so continue with the next one. You can skip `sign-all` if you've disabled signing - - ```shell - make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates - ``` - -7. build iso installation image - - ```shell - make iso - ``` - -### Use pre-built Qubes packages - -For building just a few selected packages, it's very useful to download pre-built qubes-specific dependencies from `{yum,deb}.qubes-os.org`. -This is especially true for `gcc`, which takes several hours to build. - -Before creating the `chroot`, add this to your `builder.conf`: - -``` -USE_QUBES_REPO_VERSION = $(RELEASE) -``` - -It will add the 'current' Qubes repository to your `chroot` environment. -Next, specify which components (`gcc`, for example) you want to download instead of compiling: - -``` -COMPONENTS := $(filter-out gcc,$(COMPONENTS)) -``` - -Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of `builder.conf`). -This way, you can build only the packages in which you are interested. - -If you also want to use the 'current-testing' repository, add this to your configuration: - -``` -USE_QUBES_REPO_TESTING = 1 -``` - -In the case of an existing `chroot`, for mock-enabled builds, this works immediately because `chroot` is constructed each time separately. -For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot. - -Also, once enabled, disabling this setting will not disable repositories in relevant chroots. -And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable). - -**Note** -If you are building Ubuntu templates, you cannot use this option. -This is because Qubes does not provide official packages for Ubuntu templates. - -## Code verification keys management - -[QubesBuilder](/doc/qubes-builder/) by default verifies signed tags on every downloaded code. -Public keys used for that are stored in `keyrings/git`. -By default Qubes developers' keys are imported automatically, but if you need some additional keys (for example your own), you can add them using: - -```shell -GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc -GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY -# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder -``` - -All Qubes developers' keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically. - -If you are the owner of Master key and want to revoke such signature, use the `revsig` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such). - -## Further information - -For advanced [QubesBuilder](/doc/qubes-builder/) use, and preparing sources, take a look at [QubesBuilderDetails](/doc/qubes-builder-details/) page, or [QubesBuilder's doc directory](https://github.com/marmarek/qubes-builder/tree/master/doc). +--- \ No newline at end of file diff --git a/developer/building/qubes-iso-building.md b/developer/building/qubes-iso-building.md index 1aad3a2a8..8a7b8bc59 100644 --- a/developer/building/qubes-iso-building.md +++ b/developer/building/qubes-iso-building.md @@ -8,193 +8,7 @@ redirect_from: - /en/doc/qubes-iso-building/ - /doc/QubesR3Building/ - /wiki/QubesR3Building/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/building/qubes-iso-building.html ref: 63 title: Qubes ISO building ---- - -Build Environment ------------------ - -Fedora 32 has been successfully used to build Qubes R4.0 with the below steps. -Other rpm-based operating systems may also work. -Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the `./setup` script. - -In `dom0`, install the Fedora 32 template if you don't already have it. - -~~~ -sudo qubes-dom0-update qubes-template-fedora-32 -~~~ - -Create a standalone AppVM from the Fedora 32 template. -Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional. -It's not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build. -Likewise, more memory (up to 16 GB) can help. -Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes. - -Once you've built the development AppVM, open a Terminal window to it and install the necessary dependencies (see [QubesBuilder](/doc/qubes-builder/) for more info): - -~~~ -$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA -~~~ - -Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root): - -~~~ -wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc -gpg --import qubes-master-signing-key.asc -gpg --edit-key 36879494 -fpr -# Verify fingerprint! See Note below! -# Once verified, set trust to *ultimate* -# (Typical sequence is trust, 5, q) -wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc -gpg --import qubes-developers-keys.asc -~~~ - -**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) -- we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match! -See [verifying signatures](/security/verifying-signatures/#how-to-import-and-authenticate-the-qubes-master-signing-key) for verification sources. - -Now let's bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually: - -~~~ -git clone https://github.com/QubesOS/qubes-builder.git -cd qubes-builder -git tag -v `git describe` -~~~ - -**Note** It's very important to check if the verification message contains "Good signature from ..." and does not contain "WARNING: This key is not certified with a trusted signature!". - -Assuming the verification went fine, we're good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build. -The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources). - -Build using setup script ------------------ - -Let's configure the builder first (see [procedure](/doc/qubes-iso-building/#build-using-manual-steps) at bottom if you would prefer to manually configure): - -~~~ -cd ~/qubes-builder -./setup -# Select Yes to add Qubes Master Signing Key -# Select Yes to add Qubes OS Signing Key -# Select 4.0 for version -# Stable -# Select Yes for fast Git cloning -# Select Current (if you want the option to use pre-built packages) -# Select No (we want a full build) -# Select fc30 and buster (for the currently shipping templates) -# Select builder-rpm, builder-debian, template-whonix, mgmt-salt -# Select Yes to add adrelanos's third party key -# Select Yes (to download) -~~~ - -Once it completes downloading, re-run `setup` to add the Whonix templates: - -~~~ -./setup -# Choose the same options as above, except at templates select: -# fc30, buster, whonix-gateway-15, whonix-workstation-15 -~~~ - -Continue the build process with: - -~~~ -make install-deps -make get-sources -~~~ - -When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable in `builder.conf` at this stage to specify the currently shipping Tor Browser version. -See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/). - -You may also want to add `COMPONENTS := $(filter-out gcc,$(COMPONENTS))` to bypass a multiple hour compile step. -See [QubesBuilder](/doc/qubes-builder/#use-pre-built-qubes-packages) for more detail. - -Finally, if you are making a test build, use: - -~~~ -make qubes -make iso -~~~ - -Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`): - -~~~ -make qubes -make sign-all -make iso -~~~ - -Enjoy your new ISO! - -Build using manual steps ------------------ - -Instead of using `./setup`, you can manually configure the build. -The script takes care of a lot of the keyring preparation for us, so we first need to set that up. - -If you will be building Whonix templates: - -~~~ -cd ~ -gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA -gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA -~~~ - -**Note:** It's very important to check the fingerprint displayed against multiple sources such as the [Whonix web site](https://www.whonix.org/wiki/Whonix_Signing_Key), etc. -It should look something like this: - -~~~ -pub 4096R/2EEACCDA 2014-01-16 [expires: 2021-04-17] - Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA -uid Patrick Schleizer -sub 4096R/CE998547 2014-01-16 [expires: 2021-04-17] -sub 4096R/119B3FD6 2014-01-16 [expires: 2021-04-17] -sub 4096R/77BB3C48 2014-01-16 [expires: 2021-04-17] -~~~ - -Next, prepare the Git keyring directory and copy them in: - -~~~ -export GNUPGHOME=~/qubes-builder/keyrings/git -mkdir --parents "$GNUPGHOME" -cp ~/.gnupg/pubring.gpg "$GNUPGHOME" -cp ~/.gnupg/trustdb.gpg "$GNUPGHOME" -chmod --recursive 700 "$GNUPGHOME" -~~~ - -Copy one of the example configurations: - -~~~ -cd ~/qubes-builder -cp example-configs/qubes-os-master.conf builder.conf -~~~ - -Edit `builder.conf`, referring to `doc/Configuration.md` for a description of all available options. - -Continue the build process with: - -~~~ -make install-deps -make get-sources -unset GNUPGHOME -~~~ - -When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable at this stage to specify the currently shipping Tor Browser version. -See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/). - -Finally, if you are making a test build, use: - -~~~ -make qubes -make iso -~~~ - -Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`): - -~~~ -make qubes -make sign-all -make iso -~~~ - -Enjoy your new ISO! +--- \ No newline at end of file diff --git a/developer/building/qubes-template-configs.md b/developer/building/qubes-template-configs.md index 71139116d..b6f815def 100644 --- a/developer/building/qubes-template-configs.md +++ b/developer/building/qubes-template-configs.md @@ -5,4 +5,4 @@ permalink: /doc/qubes-template-configs/ redirect_to: https://github.com/QubesOS/qubes-template-configs ref: 248 title: Qubes template configs ---- +--- \ No newline at end of file diff --git a/developer/code/code-signing.md b/developer/code/code-signing.md index d22c3cf85..fcae34de4 100644 --- a/developer/code/code-signing.md +++ b/developer/code/code-signing.md @@ -2,199 +2,7 @@ lang: en layout: doc permalink: /doc/code-signing/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/code/code-signing.html ref: 51 title: Code signing ---- - -All contributions to the Qubes OS [source code](/doc/source-code/) must be cryptographically signed by the author's PGP key. - -## Generating a Key - -(Note: If you already have a PGP key, you may skip this step.) - -Alex Cabal has written an excellent [guide](https://alexcabal.com/creating-the-perfect-gpg-keypair/) on creating a PGP keypair. -Below, we reproduce just the minimum steps in generating a keypair using GnuPG. -Please read Cabal's full guide for further important details. - -~~~ -$ gpg --gen-key -gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc. -This is free software: you are free to change and redistribute it. -There is NO WARRANTY, to the extent permitted by law. - -gpg: directory '/home/user/.gnupg' created -gpg: keybox '/home/user/.gnupg/pubring.kbx' created -Note: Use "gpg --full-generate-key" for a full featured key generation dialog. - -GnuPG needs to construct a user ID to identify your key. - -Real name: Bilbo Baggins -Email address: bilbo@shire.org -You selected this USER-ID: - "Bilbo Baggins " - -Change (N)ame, (E)mail, or (O)kay/(Q)uit? O -We need to generate a lot of random bytes. It is a good idea to perform -some other action (type on the keyboard, move the mouse, utilize the -disks) during the prime generation; this gives the random number -generator a better chance to gain enough entropy. - - - -We need to generate a lot of random bytes. It is a good idea to perform -some other action (type on the keyboard, move the mouse, utilize the -disks) during the prime generation; this gives the random number -generator a better chance to gain enough entropy. -gpg: /home/user/.gnupg/trustdb.gpg: trustdb created -gpg: key 6E2F4E7AF50A5827 marked as ultimately trusted -gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created -gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/87975838063F97A968D503266E2F4E7AF50A5827.rev' -public and secret key created and signed. - -pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] - 87975838063F97A968D503266E2F4E7AF50A5827 -uid Bilbo Baggins -sub rsa3072 2021-12-30 [E] [expires: 2023-12-30] -~~~ - -## Upload the Key - -For others to find the public key, please upload it to a server. - -Currently, [these](https://github.com/marmarek/signature-checker/blob/master/check-git-signature#L133-L135) are the recognized servers. - -In the example below, we will use `keyserver.ubuntu.com`. - -Replace 6E2F4E7AF50A5827 with your key ID, which is the last 8 hex digits of the long number in the second line of the output above: -``` -pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] - 87975838063F97A968D503266E2F4E7AF50A5827 -``` - -```shell_session -$ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827 -gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com -``` - -``` -$ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827 -gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com -``` - -## Using PGP with Git - -If you're submitting a patch via GitHub (or a similar Git server), please sign -your Git commits. - -1. Set up Git to use your key: - - ~~~ - git config --global user.signingkey - ~~~ - -2. Set up Git to sign your commits with your key: - - ~~~ - git config --global commit.gpgsign true - ~~~ - - Alternatively, manually specify when a commit is to be signed: - - ~~~ - git commit -S - ~~~ - -3. (Optional) Create signed tags. - Signed commits are totally sufficient to contribute to Qubes OS. - However, if you have commits which are not signed and you do not want to change them, - you can create a signed tag for the commit and push it before the check. - - This is useful for example, if you have a commit back in the git history which - you like to sign now without rewriting the history. - - ~~~ - git tag -s -m "" - ~~~ - - You can also create an alias to make this easier. - Edit your `~/.gitconfig` file. - In the `[alias]` section, add `stag` to create signed tags and `spush` to create signed tags and push them. - - ~~~ - [alias] - stag = "!bash -c 'id=\"`git rev-parse --verify HEAD`\"; tag_name="signed_tag_for_${id:0:8}"; git tag -s "$tag_name" -m \"Tag for commit $id\"; echo \"$tag_name\"'" - spush = "!bash -c 'git push origin `git stag`'" - ~~~ - - You may also find it convenient to have an alias for verifying the tag on the - latest commit: - - ~~~ - vtag = !git tag -v `git describe` - ~~~ - -## GitHub Signature Verification (optional) - -GitHub shows a green `Verified` label indicating that the GPG signature could be -verified using any of the contributor’s GPG keys uploaded to GitHub. You can -upload your public key on GitHub by adding your public GPG key on the [New GPG -key](https://github.com/settings/gpg/new) under the [SSH GPG keys page](https://github.com/settings/keys). - -## Code Signature Checks - -The [signature-checker](https://github.com/marmarek/signature-checker) checks if code contributions are signed. -Although GitHub adds a little green `Verified` button next to the commit, the [signature-checker](https://github.com/marmarek/signature-checker) uses this algorithm to check if a commit is correctly signed: - -1. Is the commit signed? - If the commit is not signed, you can see the message - > policy/qubesos/code-signing — No signature found -2. If the commit is signed, the key is downloaded from a GPG key server. - If you can see the following error message, please check if you have uploaded the key to a key server. - > policy/qubesos/code-signing — Unable to verify (no valid key found) - -### No Signature Found - -> policy/qubesos/code-signing — No signature found - -In this case, you have several options to sign the commit: - -1. Amend the commit and replace it with a signed commit. - You can use this command to create a new signed commit: - - ``` - git commit --amend -S - ``` - - This also rewrites the commit so you need to push it forcefully: - - ``` - git push -f - ``` - -2. Create a signed tag for the unsigned commit. - If the commit is back in history and you do not want to change it, - you can create a signed tag for this commit and push the signature. - You can use the alias from above: - - ``` - git checkout - git spush - ``` - - Now, the signature checker needs to re-check the signature. - Please comment on the pull request that you would like to have the signatures checked again. - -### Unable To Verify - -> policy/qubesos/code-signing — Unable to verify (no valid key found) - -This means that the [signature-checker](https://github.com/marmarek/signature-checker) has found a signature for the commit -but is not able to verify it using the any key available. -This might be that you forgot to upload the key to a key server. -Please upload it. - -## Using PGP with Email - -If you're submitting a patch by emailing the [developer mailing list](/support/#qubes-devel), simply sign your email with your PGP key. -One good way to do this is with a program like [Enigmail](https://www.enigmail.net/). -Enigmail is a security addon for the Mozilla Thunderbird email client that allows you to easily digitally encrypt and sign your emails. +--- \ No newline at end of file diff --git a/developer/code/coding-style.md b/developer/code/coding-style.md index 5bc1dec59..a323d8a7d 100644 --- a/developer/code/coding-style.md +++ b/developer/code/coding-style.md @@ -7,179 +7,7 @@ redirect_from: - /doc/CodingStyle/ - /wiki/CodingStyle/ - /trac/wiki/CodingStyle/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/code/coding-style.html ref: 53 title: Coding style ---- - -Rationale ---------- - -Maintaining proper coding style is very important for any large software project, such as Qubes. Here's why: - -- It eases maintenance tasks, such as adding new functionality or generalizing code later, -- It allows others (as well as the future you!) to easily understand fragments of code and what they were supposed to do, and thus makes it easier to later extend them with newer functionality or bug fixes, -- It allows others to easily review the code and catch various bugs, -- It provides for an aesthetically pleasing experience when one reads the code... - -Often, developers, usually smart ones, undersell the value of proper coding style, thinking that it's much more important how their code works. These developers expect that if their code solves some problem using a nice and neat trick, then that's all that is really required. Such thinking shows, however, immaturity and is a signal that the developer, no matter how bright and smart, might not be a good fit for larger projects. Writing a clever exploit for a Black Hat show is one thing - writing useful software supposed to be used and maintained for years is quite a different story. If you want to show off what a smart programmer you are, then you should become a researcher and write exploits. If, on the other hand, you want to be part of a team that makes real, useful software, you should ensure your coding style is impeccable. At Qubes project, we often took shortcuts and wrote nasty code, and this has always back fired at us, sometime months, sometime years later, the net result being we had to spend time fixing code, rather than implementing new functionality. - -And here's a [link to the real case](https://groups.google.com/forum/#!msg/qubes-devel/XgTo6L8-5XA/JLOadvBqnqMJ) (one Qubes Security Bulletin) demonstrating how the lackadaisical coding style lead to a real security bug. Never assume you're smart enough to disregard clean and rigorous coding! - -General typographic conventions -------------------------------- - -- **Use space-expanded tabs that equal 4 spaces.** Yes, we know, there are many arguments for using "real" tabs instead of space-expanded tabs, but we need to pick one convention to make the project consistent. One argument for using space-expanded tabs is that this way the programmer is in control of how the code will look like, despite how other users have configured their editors to visualize the tabs (of course, we assume any sane person uses a fixed-width font for viewing the source code). If it makes you feel any better, assume this is just an arbitrary choice made to enforce a unified style. - -- **Maintain max. line length of 80 characters**. Even though today's monitors often are very wide and it's often not a problem to have 120 characters displayed in an editor, maintaining shorter line lengths improves readability. It also allows others to have two parallel windows open, side by side, each with different parts of the source code. - -- **Naming conventions for any OS *other than Windows***: - - `ClassName` - - `some_variable`, `some_function`, `some_argument` - -- **Naming convention *for Windows OS*** -- exceptionally to preserve Windows conventions please use the following: - - `ClassName`, `FunctionName` - - `pszArgumentOne`, `hPipe` -- use Hungarian notation for argument and variables - -- **Maintain a decent amount of horizontal spacing**, e.g. add a space after `if` or before `{` in C, and similar in other languages. Whether and where to also use spaces within expressions, such as (x\*2+5) vs. (x \* 2 + 5) is left to the developer's judgment. Do not put spaces immediately after or before the brackets in expressions, so avoid constructs like this: `if ( condition )` and use ones like this: `if (condition)` instead. - -- **Use single new lines** ('\\n' aka LF) in any non-Windows source code. On Windows, exceptionally, use the CRLF line endings (--). This will allow the source code to be easily viewable in various Windows-based programs. - -- **Use descriptive names for variables and functions**! Really, at a time when most editors have auto-completion features, there is no excuse for using short variable names. - -- **Comments should be indented together with the code**, e.g. like this: - - ~~~ - for (...) { - // The following code finds PGP private key matching the given public key in O(1) - while (key_found) { - (...) - } - } - ~~~ - -File naming conventions ------------------------ - -- All file names written with small letters, use dash to separate words, rather than underscores, e.g. `qubes-dom0-update`. Never use spaces! - -**File naming in Linux/Unix-like systems:** - -- User commands that operate on particular VMs (also those accessible in VMs): `/usr/bin/qvm-*` -- User commands that apply to the whole system (Dom0 only): `/usr/bin/qubes-*` -- Auxiliary executables and scripts in `/usr/libexec/qubes/` (Note: previously we used `/usr/lib/qubes` but decided to change that) -- Helper, non-executable files in `/usr/share/qubes/` -- Various config files in `/etc/qubes` -- Qubes RPC services in `/etc/qubes-rpc`. Qubes RPC Policy definitions (only in Dom0) in `/etc/qubes-rpc/policy/` -- All VM-related configs, images, and other files in `/var/lib/qubes/` -- System-wide temporary files which reflect the current state of system in `/var/run/qubes` -- Logs: either log to the system-wide messages, or to `/var/log/qubes/` - -**File naming in Windows systems:** - -- All base qubes-related files in `C:\Program Files\Invisible Things Lab\Qubes\` (Exceptionally spaces are allowed here to adhere to Windows naming conventions) -- Other, third-party files, not Qubes-specific, such as e.g. Xen PV drivers might be in different vendor subdirs, e.g. `C:\Program Files\Xen PV Drivers` - -General programming style guidelines ------------------------------------- - -- Do not try to impress with your coding kung-fu, do not use tricks to save 2 lines of code, always prefer readability over trickiness! -- Make sure your code compiles and builds without warnings. -- Always think first about interfaces (e.g. function arguments, or class methods) and data structures before you start writing the actual code. -- Use comments to explain non-trivial code fragments, or expected behavior of more complex functions, if it is not clear from their name. -- Do **not** use comments for code fragments where it is immediately clear what the code does. E.g. avoid constructs like this: - - ~~~ - // Return window id - int get_window_id (...) { - (...) - return id; - } - ~~~ - -- Do **not** use comments to disable code fragments. In production code there should really be no commented or disabled code fragments. If you really, really have a good reason to retain some fragment of unused code, use \#if or \#ifdef to disable it, e.g.: - - ~~~ - #if 0 - (...) // Some unused code here - #endif - ~~~ - - ... and preferably use some descriptive macro instead of just `0`, e.g.: - - ~~~ - #if USE_OLD_WINDOW_TRAVERSING - (...) // Some unused code here - #endif - ~~~ - - Not sure how to do similar thing in Python... Anyone? - -> But generally, there is little excuse to keep old, unused code fragments in the code. One should really use the functionality provided by the source code management system, such as git, instead. E.g. create a special branch for storing the old, unused code -- this way you will always be able to merge this code into upstream in the future. - -- Do not hardcode values in the code! The only three numbers that are an exception here and for which it is acceptable to hardcode them are: `0`, `1` and `-1`, and frankly the last two are still controversial... - -Source Code management (Git) guidelines ---------------------------------------- - -- Use git to maintain all code for Qubes project. - -- Before you start using git, make sure you understand that git is a decentralized Source Code Management system, and that it doesn't behave like traditional, centralized source code management systems, such as SVN. Here's a good [introductory book on git](http://git-scm.com/book). Read it. - -- Qubes code is divided into many git repositories. There are several reasons for that: - - This creates natural boundaries between different code blocks, enforcing proper interfaces, and easing independent development to be conducted on various code parts at the same time, without the fear of running into conflicts. - - By maintaining relatively small git repositories, it is easy for new developers to understand the code and contribute new patches, without the need to understand all the other code. - - Code repositories represent also licensing boundaries. So, e.g. because `core-agent-linux` and `core-agent-windows` are maintained in two different repositories, it is possible to have the latter under a proprietary, non-GPL license, while keeping the former fully open source. - - We have drastically changed the layout and naming of the code repositories shortly after Qubes OS R2 Beta 2 release. For details on the current code layout, please read [this article](https://blog.invisiblethings.org/2013/03/21/introducing-qubes-odyssey-framework.html). - -Commit message guidelines -------------------------- - -Please attempt to follow these conventions when writing your Git commit messages: - -- Separate the subject line from the body with a blank line. -- Limit the subject line to approximately 50 characters. -- Capitalize the subject line. -- Do not end the subject line with a period. -- Use the imperative mood in the subject line. -- Wrap the body at 72 characters. -- Use the body to explain *what* and *why* rather than *how*. - -For details, examples, and the rationale behind each of these conventions, please see [this blog post](https://chris.beams.io/posts/git-commit/), which is the source of this list. - -Security coding guidelines --------------------------- - -- As a general rule: **untrusted input** is our \#1 enemy! -- Any input that comes from untrusted, or less trusted, or just differently-trusted, entity should always be considered as malicious and should always be sanitized and verified. So, if your software runs in Dom0 and processes some input from any of the VMs, this input should be considered to be malicious. Even if your software runs in a VM, and processes input from some other VM, you should also assume that the input is malicious and verify it. -- Use `untrusted_` prefix for all variables that hold values read from untrusted party and which have not yet been verified to be decent, e.g.: - - ~~~ - read_struct(untrusted_conf); - /* sanitize start */ - if (untrusted_conf.width > MAX_WINDOW_WIDTH) - untrusted_conf.width = MAX_WINDOW_WIDTH; - if (untrusted_conf.height > MAX_WINDOW_HEIGHT) - untrusted_conf.height = MAX_WINDOW_HEIGHT; - width = untrusted_conf.width; - height = untrusted_conf.height; - ~~~ - -- Use others variables, without the `untrusted_` prefix to hold the sanitized values, as shown above. - -Python-specific guidelines --------------------------- - -- Please follow the guidelines [here](http://www.python.org/dev/peps/pep-0008/), unless they were in conflict with what is written on this page. - -C and C++ specific guidelines ------------------------------ - -- Do not place code in `*.h` files. -- Use `const` whenever possible, e.g. in function arguments passed via pointers. -- Do not mix procedural and objective code together -- if you write in C++, use classes and objects. -- Think about classes hierarchy, before starting to implement specific methods. - -Bash-specific guidelines ------------------------- - -- Avoid writing scripts in bash whenever possible. Use python instead. Bash-scripts are Unix-specific and will not work under Windows VMs, or in Windows admin domain, or Windows gui domain. +--- \ No newline at end of file diff --git a/developer/code/license.md b/developer/code/license.md index f7fe3682e..22bb881ba 100644 --- a/developer/code/license.md +++ b/developer/code/license.md @@ -6,18 +6,7 @@ redirect_from: - /en/doc/license/ - /doc/QubesLicensing/ - /wiki/QubesLicensing/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/code/license.html ref: 52 title: Software license ---- - -Qubes OS is a compilation of software packages, each under its own license. The -compilation is made available under the GNU General Public License version 2 -(GPLv2). - -The source code of Qubes OS is contained in repositories under the -[@QubesOS](https://github.com/QubesOS) account on GitHub. This source code is -made available under GPLv2, unless there is a `LICENSE` file in the root of the -containing repository that specifies a different license. - -The full text of the GPLv2 license can be found -[here](https://www.gnu.org/licenses/gpl-2.0.html). +--- \ No newline at end of file diff --git a/developer/code/source-code.md b/developer/code/source-code.md index 85e5cf13a..c344a8f67 100644 --- a/developer/code/source-code.md +++ b/developer/code/source-code.md @@ -6,77 +6,7 @@ redirect_from: - /en/doc/source-code/ - /doc/SourceCode/ - /wiki/SourceCode/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/code/source-code.html ref: 54 title: Source code ---- - -All the Qubes code is kept in Git repositories. We have divided the project into -several components, each of which has its own separate repository, for example: - -* `core-admin.git` -- The core Qubes infrastructure, responsible for VM - management, VM templates, fs sharing, etc. -* `gui-daemon.git` -- GUI virtualization, Dom0 side. -* `gui-agent-linux.git` -- GUI virtualization, Linux VM side. -* `linux-template-builder.git` -- Scripts and other files used to create Qubes - template images. - -All of our repositories are available under the [QubesOS GitHub account](https://github.com/QubesOS/). - -To clone a repository: - -~~~ -git clone https://github.com/QubesOS/qubes-.git -~~~ - -e.g.: - -~~~ -git clone https://github.com/QubesOS/qubes-core-admin.git core-admin -~~~ - -To build Qubes you do not need to download all these repositories. -If you use [qubes builder](/doc/QubesBuilder/) you can specify *what* you want to build, and download only the repositories needed to build that target. - -If you really do want to clone **all** of the repositories, you can use these commands: - -~~~ -curl "https://api.github.com/orgs/QubesOS/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone -curl "https://api.github.com/orgs/QubesOS/repos?page=2&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone -~~~ - -To update (git fetch) **all** of these repositories : - -~~~ -find . -mindepth 1 -maxdepth 1 -type d -exec git -C {} fetch --tags --recurse-submodules=on-demand --all \; -~~~ - -(Alternatively, you can pull instead of just fetching.) - -How to Send Patches -------------------- - -If you want to [contribute code](/doc/contributing/#contributing-code) to the project, there are two ways. Whichever -method you choose, you must [sign your code](/doc/code-signing/) before it can be accepted. - -* **Preferred**: Use GitHub's [fork & pull requests](https://guides.github.com/activities/forking/). - - Opening a pull request on GitHub greatly eases the code review and tracking - process. In addition, especially for bigger changes, it's a good idea to send - a message to the [qubes-devel mailing list](/support/#qubes-devel) in order to notify people who - do not receive GitHub notifications. - -* Send a patch to the [qubes-devel mailing list](/support/#qubes-devel) (`git format-patch`). - - 1. Make all the changes in your working directory, i.e. edit files, move them - around (you can use 'git mv' for this), etc. - 2. Add the changes and commit them (`git add`, `git commit`). Never mix - different changes into one commit! Write a good description of the commit. - The first line should contain a short summary, and then, if you feel like - more explanations are needed, enter an empty new line, and then start the - long, detailed description (optional). - 3. Test your changes NOW: check if RPMs build fine, etc. - 4. Create the patch using `git format-patch`. This has an advantage over - `git diff`, because the former will also include your commit message, your - name and email, so that *your* name will be used as a commit's author. - 5. Send your patch to `qubes-devel`. Start the message subject with - `[PATCH]`. +--- \ No newline at end of file diff --git a/developer/debugging/automated-tests.md b/developer/debugging/automated-tests.md index 3fd4ed24b..a5b4c8063 100644 --- a/developer/debugging/automated-tests.md +++ b/developer/debugging/automated-tests.md @@ -5,275 +5,7 @@ permalink: /doc/automated-tests/ redirect_from: - /en/doc/automated-tests/ - /doc/AutomatedTests/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/automated-tests.html ref: 45 title: Automated tests ---- - -## Unit and Integration Tests - -Starting with Qubes R3 we use [python unittest](https://docs.python.org/3/library/unittest.html) to perform automatic tests of Qubes OS. -Despite the name, we use it for both [unit tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration tests](https://en.wikipedia.org/wiki/Integration_tests). -The main purpose is, of course, to deliver much more stable releases. - -The integration tests must be run in dom0, but some unit tests can run inside a VM as well. - -### Integration & unit testing in dom0 - -Integration tests are written with the assumption that they will be executed on dedicated hardware and must be run in dom0. All other unit tests can also be run in dom0. - -**Do not run the tests on installations with important data, because you might lose it.** - -All the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing (& possibly breaking) VM components. - -First you need to build all packages that you want to test. Please do not mix branches as this will inevitably lead to failures. Then setup Qubes OS with these packages installed. - -For testing you'll have to stop the `qubesd` service as the tests will use its own custom variant of the service: -`sudo systemctl stop qubesd` - -Don't forget to start it after testing again. - -To start testing you can then use the standard python unittest runner: - -`sudo -E python3 -m unittest -v qubes.tests` - -Alternatively, use the custom Qubes OS test runner: - -`sudo -E python3 -m qubes.tests.run -v` - -Our test runner runs mostly the same as the standard one, but it has some nice additional features like colored output and not needing the "qubes.test" prefix. - -You can use `python3 -m qubes.tests.run -h` to get usage information: - -``` -[user@dom0 ~]$ python3 -m qubes.tests.run -h -usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast] - [--do-not-clean] [--do-clean] [--loglevel LEVEL] - [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg] - [TESTNAME [TESTNAME ...]] - -positional arguments: - TESTNAME list of tests to run named like in description - (default: run all tests) - -optional arguments: - -h, --help show this help message and exit - --verbose, -v increase console verbosity level - --quiet, -q decrease console verbosity level - --list, -l list all available tests and exit - --failfast, -f stop on the first fail, error or unexpected success - --no-failfast disable --failfast - --loglevel LEVEL, -L LEVEL - logging level for file and syslog forwarding (one of: - NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL; - default: DEBUG) - --logfile FILE, -o FILE - if set, test run will be also logged to file - --syslog reenable logging to syslog - --no-syslog disable logging to syslog - --kmsg, --very-brave-or-very-stupid - log most important things to kernel ring-buffer - --no-kmsg, --i-am-smarter-than-kay-sievers - do not abuse kernel ring-buffer - --allow-running-along-qubesd - allow running in parallel with qubesd; this is - DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE - --break-to-repl break to REPL after tests - -When running only specific tests, write their names like in log, in format: -MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.". -Example: basic/TC_00_Basic/test_000_create -``` - -For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list: - -``` -[user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21 -network/VmNetworking_fedora-21/test_000_simple_networking -network/VmNetworking_fedora-21/test_010_simple_proxyvm -network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm -network/VmNetworking_fedora-21/test_030_firewallvm_firewall -network/VmNetworking_fedora-21/test_040_inter_vm -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0 -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file -[user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21` -``` - -Example test run: - -![snapshot-tests2.png](/attachment/doc/snapshot-tests2.png) - -Tests are also compatible with nose2 test runner, so you can use this instead: - -```bash -sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd -``` - -This may be especially useful together with various nose2 plugins to store tests results (for example `nose2.plugins.junitxml`), to ease presenting results. This is what we use on [OpenQA](http://open.qa/). - -### Unit testing inside a VM - -Many unit tests will also work inside a VM. However all of the tests requiring a dedicated VM to be run (mostly the integration tests) will be skipped. - -Whereas integration tests are mostly stored in the [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) repository, unit tests can be found in each of the Qubes OS repositories. - -To for example run the `qubes-core-admin` unit tests, you currently have to clone at least [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) and -its dependency [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec) repository in the branches that you want to test. - -The below example however will assume that you set up a build environment as described in the [Qubes Builder documentation](/doc/qubes-builder/). - -Assuming you cloned the `qubes-builder` repository to your home directory inside a fedora VM, you can use the following commands to run the unit tests: - -```{.bash} -cd ~ -sudo dnf install python3-pip lvm2 python35 python3-virtualenv -virtualenv -p /usr/bin/python35 python35 -source python35/bin/activate -python3 -V -cd ~/qubes-builder/qubes-src/core-admin -pip3 install -r ci/requirements.txt -export PYTHONPATH=../core-qrexec:test-packages -./run-tests -``` - -To run only the tests related to e.g. `lvm`, you may use: - -`./run-tests -v $(python3 -m qubes.tests.run -l | grep lvm)` - -You can later re-use the created virtual environment including all of the via `pip3` installed packages with `source ~/python35/bin/activate`. - -We recommend to run the unit tests with the Python version that the code is meant to be run with in dom0 (3.5 was just an example above). For instance, the `release4.0` (Qubes 4.0) branch is intended -to be run with Python 3.5 whereas the Qubes 4.1 branch (`master` as of 2020-07) is intended to be run with Python 3.7 or higher. You can always check your dom0 installation for the Python version of -the current stable branch. - -### Tests configuration - -Test runs can be altered using environment variables: - -- `DEFAULT_LVM_POOL` - LVM thin pool to use for tests, in `VolumeGroup/ThinPool` format -- `QUBES_TEST_PCIDEV` - PCI device to be used in PCI passthrough tests (for example sound card) -- `QUBES_TEST_TEMPLATES` - space separated list of templates to run tests on; if not set, all installed templates are tested -- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol) - -### Adding a new test to core-admin - -After adding a new unit test to [core-admin/qubes/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) you'll have to include it in [core-admin/qubes/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py) - -#### Editing `__init__.py` - -You'll also need to add your test at the bottom of the `__init__.py` file, in the method `def load_tests`, in the for loop with `modname`. -Again, given the hypothetical `example.py` test: - -~~~python -for modname in ( - 'qubes.tests.basic', - 'qubes.tests.dom0_update', - 'qubes.tests.network', - 'qubes.tests.vm_qrexec_gui', - 'qubes.tests.backup', - 'qubes.tests.backupcompatibility', - 'qubes.tests.regressions', - 'qubes.tests.example', # This is our newly added test - ): -~~~ - -### Testing PyQt applications - -When testing (Py)QT applications, it's useful to create a separate QApplication object for each test. -But QT framework does not allow multiple QApplication objects in the same process at the same time. -This means it's critical to reliably cleanup the previous instance before creating a new one. -This turns out to be a non-trivial task, especially if _any_ test uses the event loop. -Failure to perform proper cleanup in many cases results in SEGV. -Below you can find steps for the proper cleanup: - -~~~python -import asyncio -import quamash -import unittest -import gc - -class SomeTestCase(unittest.TestCase): - def setUp(self): - [...] - - # force "cleanlooks" style, the default one on Xfce (GtkStyle) use - # static variable internally and caches pointers to later destroyed - # objects (result: SEGV) - self.qtapp = QtGui.QApplication(["test", "-style", "cleanlooks"]) - - # construct event loop even if this particular test doesn't use it, - # otherwise events with qtapp references will be queued there anyway and the - # first test that actually use event loop will try to dereference (already - # destroyed) objects, resulting in SEGV - self.loop = quamash.QEventLoop(self.qtapp) - - def tearDown(self): - [...] - # process any pending events before destroying the object - self.qtapp.processEvents() - - # queue destroying the QApplication object, do that for any other QT - # related objects here too - self.qtapp.deleteLater() - - # process any pending events (other than just queued destroy), just in case - self.qtapp.processEvents() - - # execute main loop, which will process all events, _including just queued destroy_ - self.loop.run_until_complete(asyncio.sleep(0)) - - # at this point it QT objects are destroyed, cleanup all remaining references; - # del other QT object here too - self.loop.close() - del self.qtapp - del self.loop - gc.collect() -~~~ - -## Automated tests with openQA - -**URL:** -**Tests:** - -Manually testing Qubes OS and its installation is a time-consuming process. -We use [OpenQA](http://open.qa/) to automate this process. -It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses. -Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation. - -Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT. -KVM also supports nested virtualization, so HVM should theoretically work. -In practice, however, either Xen or QEMU crashes when this is attempted. -Nonetheless, PV works well, which is sufficient for automated installation testing. - -Thanks to present and past donors who have provided the infrastructure for Qubes' openQA system with hardware that meets these requirements. - -### Looking for patterns in tests - -In order to better visualize patterns in tests the [`openqa_investigator`](https://github.com/QubesOS/openqa-tests-qubesos/blob/master/utils/openqa_investigator.py) script can be used. -It feeds off of the openQA test data to make graph plots. Here is an example: - -![openqa-investigator-splitgpg-example.png](/attachment/doc/openqa-investigator-splitgpg-example.png) - -Some outputs: - - plot by tests - - plot by errors - - markdown - -Some filters: - - filter by error - - filter by test name - -Check out the script's help with `python3 openqa_investigator.py --help` -to see all available options. +--- \ No newline at end of file diff --git a/developer/debugging/mount-lvm-image.md b/developer/debugging/mount-lvm-image.md index d5b760807..d66e3ac58 100644 --- a/developer/debugging/mount-lvm-image.md +++ b/developer/debugging/mount-lvm-image.md @@ -2,53 +2,7 @@ lang: en layout: doc permalink: /doc/mount-lvm-image/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/mount-lvm-image.html ref: 46 title: How to mount LVM images ---- - -You want to read your LVM image (e.g., there is a problem where you can't start any VMs except dom0). - -1: make the image available for qubesdb. -From dom0 terminal: - -```bash -# Example: /dev/qubes_dom0/vm-debian-9-tmp-root -[user@dom0]$ dev=$(basename $(readlink /dev/YOUR_LVM_VG/YOUR_LVM_IMAGE)) -[user@dom0]$ qubesdb-write /qubes-block-devices/$dev/desc "YOUR_LVM_IMAGE" -``` - -2: Create a new disposable VM - -```bash -[user@dom0]$ qvm-run -v --dispvm=YOUR_DVM_TEMPLATE --service qubes.StartApp+xterm & -``` - -3: Attach the device to your newly created disp VM - -From the GUI, or from the command line: - -```bash -[user@dom0]$ qvm-block attach NEWLY_CREATED_DISPVM dom0:$dev -``` - -4: Mount the partition you want to, and do what you want with it - -```bash -[user@dispXXXX]$ mount /dev/xvdiX /mnt/ -``` - -5: Umount and kill the VM - -```bash -[user@dispXXXX]$ umount /mnt/ -``` - -6: Remove the image from qubesdb - -```bash -[user@dom0]$ qubesdb-rm /qubes-block-devices/$dev/ -``` - -# References - -Please consult this issue's [comment](https://github.com/QubesOS/qubes-issues/issues/4687#issuecomment-451626625). +--- \ No newline at end of file diff --git a/developer/debugging/profiling.md b/developer/debugging/profiling.md index 2932b4cbf..b50efe9f9 100644 --- a/developer/debugging/profiling.md +++ b/developer/debugging/profiling.md @@ -6,93 +6,7 @@ redirect_from: - /en/doc/profiling/ - /doc/Profiling/ - /wiki/Profiling/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/profiling.html ref: 48 title: Python profiling ---- - -This is a python profiling primer. - -For the purpose of this document, `qubes-dev` is name of the domain used for postprocessing profiling stats. - -## Requirements - -~~~ -yum install gprof2dot graphviz -git clone http://git.woju.eu/qubes/profiling.git -~~~ - -If you profile something in dom0, move `Upload.sh` from the repository to dom0: - -~~~ -mkdir -p ~/profiling -qvm-run -p qubes-dev 'cat ~/profiling/Upload.sh' > ~/profiling/Upload.sh -~~~ - -- WARNING: this will obviously be running third-party code which is not signed by ITL nor Fedora. You have been warned. - -## Workflow - -### Identify function responsible for some slow action - -You have to select the area in which you suspect less than optimal performance. If you do not narrow the area, graphs may be unreadable. - -### Replace suspect function with probe - -Replace - -```python -def foo(self, bar): - # function content -``` - -with - -```python -def foo(self, *args, **kwargs): - profile.runctx('self.real_foo(*args, **kwargs)', globals(), locals(), - time.strftime('/home/user/profiling/foo-%Y%m%d-%H%M%S.pstats')) - -def real_foo(self, bar): - # function content -``` - -### Run application - -Beware that some functions may be called often. For example `qubesmanager/main.py:update_table` gets run once per second. This will produce one pstat file per second. - -Remember to revert your changes to the application afterwards. - -### Upload statistics - -If you are in dom0: - -~~~ -cd ~/profiling -./Upload.sh -~~~ - -### Analyse - -~~~ -make -~~~ - -For every `${basename}.pstats` this will produce `${basename}.txt` and `${basename}.svg`. SVG files contain call graphs. Text files contain lists of all functions, sorted by cumulative execution time. You may also try `make all-png`. - -~~~ -make index.html -~~~ - -This creates `index.html` with all SVG graphics linked to TXT files, ready for upload. - -~~~ -make REMOTE=example.com:public_html/qubes/profiling/ upload -~~~ - -## Example - -This example is from `qubes-manager` (`qubesmanager/main.py`). - -!["update\_table-20140424-170010.svg"](//attachment/doc/update_table-20140424-170010.svg) - -It is apparent that the problem is around `get_disk_usage`, which calls something via `subprocess.call`. It does this 15 times, probably once per VM. +--- \ No newline at end of file diff --git a/developer/debugging/safe-remote-ttys.md b/developer/debugging/safe-remote-ttys.md index b61831160..a4e821e43 100644 --- a/developer/debugging/safe-remote-ttys.md +++ b/developer/debugging/safe-remote-ttys.md @@ -4,73 +4,7 @@ layout: doc permalink: /doc/safe-remote-ttys/ redirect_from: - /en/doc/safe-remote-ttys/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/safe-remote-ttys.html ref: 49 title: Safe remote dom0 terminals ---- - -If you do not have working graphics in Dom0, then using a terminal can be quite annoying! -This was the case for the author while trying to debug PCI-passthrough of a machine's primary (only) GPU. - -Your first thought might be to just allow network access to Dom0, enable ssh, and connect in remotely. -But, this gravely violates the Qubes security model. - -Instead, a better solution is to split the input and output paths of using a terminal. -Use your normal keyboard for input, but have the output go to a remote machine in a unidirectional manner. - -To do this, we make use of script(1), qvm-run, and optionally your network transport of choice. - -To a different VM ------------------ - -As an example of forwarding terminal output to another VM on the same machine: - -~~~ -$ mkfifo /tmp/foo -$ qvm-run -p some-vm 'xterm -e "cat 0<&5" 5<&0' /dev/null 2>&1 & -$ script -f /tmp/foo -~~~ - -To a different machine ----------------------- - -In this case over SSH (from a network-connected VM): - -~~~ -$ mkfifo /tmp/foo -$ qvm-run -p some-vm \ - 'ssh user@host sh -c "DISPLAY=:0 xterm -e \"cat 0<&5\" 5<&0"' \ - /dev/null 2>&1 & -$ script -f /tmp/foo -~~~ - -Note that no data received over SSH is ever treated as terminal input in Dom0. -The input path remains only from your trusted local keyboard. - -Multiple terminals ------------------- - -For multiple terminals, you may find it easier to just use tmux than to try to blindly switch to the correct window. - -Terminal size -------------- - -It is up to you to ensure the sizes of the local and remote terminal are the same, otherwise things may display incorrectly (especially in interactive programs). -Depending on your shell, the size of your local (blind) terminal is likely stored in the `$LINES` and `$COLUMNS` variables. - -~~~ -$ echo $COLUMNS $LINES -80 24 -~~~ - -A note on serial consoles -------------------------- - -If your machine has a serial console, you may with to use that, but note that a similar split-I/O model should be used to ensure Dom0 integrity. -If you use the serial console as normal (via e.g. getty on ttyX, and logging in as normal), then the machine at the end of the serial cable could compromise your machine! -Ideally, you would take input from your trusted keyboard, and only send the output over the serial cable via e.g. disabling getty and using: - -~~~ -script -f /dev/ttyS0 -~~~ - -You don't even need to connect the TX pin. +--- \ No newline at end of file diff --git a/developer/debugging/test-bench.md b/developer/debugging/test-bench.md index ca48a5eab..9e96cb8b4 100644 --- a/developer/debugging/test-bench.md +++ b/developer/debugging/test-bench.md @@ -6,230 +6,7 @@ redirect_from: - /en/doc/test-bench/ - /doc/TestBench/ - /wiki/TestBench/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/test-bench.html ref: 44 title: How to set up a test bench ---- - -This guide shows how to set up simple test bench that automatically test your code you're about to push. It is written especially for `core3` branch of `core-admin.git` repo, but some ideas are universal. - -We will set up a spare machine (bare metal, not a virtual) that will be hosting our experimental Dom0. We will communicate with it via Ethernet and SSH. This tutorial assumes you are familiar with [QubesBuilder](/doc/qubes-builder/) and you have it set up and running flawlessly. - -> **Notice:** -> This setup intentionally weakens some security properties in the testing system. So make sure you understand the risks and use exclusively for testing. - -## Setting up the Machine - -### Install ISO -First, do a clean install from the `.iso` [you built](/doc/qubes-iso-building/) or grabbed elsewhere (for example [here](https://forum.qubes-os.org/t/qubesos-4-1-alpha-signed-weekly-builds/3601)). - -### Enabling Network Access in Dom0 - -Internet access is intentionally disabled by default in dom0. But to ease the deployment process we will give it access. The following steps should be done in `dom0`. - -> **Note:** the following assume you have only one network card. If you have two, pick one and leave the other attached to `sys-net`. - -1. Remove the network card (PCI device) from `sys-net` -2. Restart your computer (for the removal to take effect) -3. Install `dhcp-client` and `openssh-server` on your testbench's dom0. -4. Save the following script in `/home/user/bin/dom0_network.sh` and make it executable. It should enable your network card in dom0. *Be sure to adjust the script's variables to suit your needs.* - - ```bash - #!/bin/sh - - # adjust this for your NIC (run lspci) - BDF=0000:02:00.0 - - # adjust this for your network driver - DRIVER=e1000e - - prog=$(basename $0) - - pciunbind() { - local path - path=/sys/bus/pci/devices/${1}/driver/unbind - if ! [ -w ${path} ]; then - echo "${prog}: Device ${1} not bound" - return 1 - fi - echo -n ${1} >${path} - } - - pcibind() { - local path - path=/sys/bus/pci/drivers/${2}/bind - if ! [ -w ${path} ]; then - echo "${prog}: Driver ${2} not found" - return 1 - fi - echo ${1} >${path} - } - - pciunbind ${BDF} - pcibind ${BDF} ${DRIVER} - - sleep 1 - dhclient - ``` - -5. Configure your DHCP server so your testbench gets static IP and connect your machine to your local network. You should ensure that your testbench can reach the Internet. - -6. You'll need to run the above script on every startup. To automate this save the following systemd service `/etc/systemd/system/dom0-network-direct.service` - - ``` - [Unit] - Description=Connect network to dom0 - - [Service] - Type=oneshot - ExecStart=/home/user/bin/dom0_network.sh - - [Install] - WantedBy=multi-user.target - ``` - -7. Then, enable and start the SSH Server and the script on boot: - - ```bash - sudo systemctl enable sshd - sudo systemctl start sshd - - sudo systemctl enable dom0-network-direct - sudo systemctl start dom0-network-direct - ``` - -> **Note:** If you want to install additional software in dom0 and your only network card was assigned to dom0, then _instead_ of the usual `sudo qubes-dom0-update ` now you run `sudo dnf --setopt=reposdir=/etc/yum.repos.d install `. - -### Install Tests and Their Dependencies - -A regular Qubes installation isn't ready to run the full suite of tests. For example, in order to run the [Split GPG tests](https://github.com/QubesOS/qubes-app-linux-split-gpg/blob/4bc201bb70c011119eed19df25dc5b46120d04ed/tests/splitgpg/tests.py) you need to have the `qubes-gpg-split-tests` package installed in your app qubes. - -Because of the above reason, some additional configurations need to be done to your testing environment. This can be done in an automated manner with the help of the [Salt](/doc/salt) configuration that provisions the [automated testing environment](/doc/automated-tests/). - -The following commands should work for you, but do keep in mind that the provisioning scripts are designed for the [openQA environment](https://openqa.qubes-os.org/) and not your specific local testing system. Run the following in `dom0`: - - ```bash - # For future reference the following commands are an adaptation of - # https://github.com/marmarek/openqa-tests-qubesos/blob/master/tests/update.pm - - # Install git - sudo qubes-dom0-update git || sudo dnf --setopt=reposdir=/etc/yum.repos.d install git - - # Download the openQA automated testing environment Salt configuration - git clone https://github.com/marmarek/openqa-tests-qubesos/ - cd openqa-tests-qubesos/extra-files - sudo cp -a system-tests/ /srv/salt/ - sudo qubesctl top.enable system-tests - - # Install the same configuration as the one in openQA - QUBES_VERSION=4.1 - PILLAR_DIR=/srv/pillar/base/update - sudo mkdir -p $PILLAR_DIR - printf 'update:\n qubes_ver: '$QUBES_VERSION'\n' | sudo tee $PILLAR_DIR/init.sls - printf "base:\n '*':\n - update\n" | sudo tee $PILLAR_DIR/init.top - sudo qubesctl top.enable update pillar=True - - # Apply states to dom0 and VMs - # NOTE: These commands can take several minutes (if not more) without showing output - sudo qubesctl --show-output state.highstate - sudo qubesctl --max-concurrency=2 --skip-dom0 --templates --show-output state.highstate - ``` - -## Development VM - -### SSH - -Arrange firewall so you can reach the testbench from your `qubes-dev` VM. Generate SSH key in `qubes-dev`: - -~~~ -ssh-keygen -t ecdsa -b 521 -~~~ - -Add the following section in `.ssh/config` in `qubes-dev`: - -~~~ -Host testbench - # substitute username in testbench - User user - # substitute address of your testbench - HostName 192.168.123.45 -~~~ - -#### Passwordless SSH Login - -To log to your testbench without entering password every time, copy your newly generated public key (`id_ecdsa.pub`) to `~/.ssh/authorized_keys` on your testbench. You can do this easily by running this command on `qubes-dev`: `ssh-copy-id -i ~/.ssh/id_ecdsa.pub user@192.168.123.45` (substituting with the actual username address of your testbench). - -### Scripting - -This step is optional, but very helpful. Put these scripts somewhere in your `${PATH}`, like `/usr/local/bin`. - -`qtb-runtests`: - -```bash -#!/bin/sh - -ssh testbench python -m qubes.tests.run -``` - -`qtb-install`: - -```bash -#!/bin/sh - -TMPDIR=/tmp/qtb-rpms - -if [ $# -eq 0 ]; then - echo "usage: $(basename $0) ..." - exit 2 -fi - -set -e - -ssh testbench mkdir -p "${TMPDIR}" -scp "${@}" testbench:"${TMPDIR}" || echo "check if you have 'scp' installed on your testbench" - -while [ $# -gt 0 ]; do - ssh testbench sudo rpm -i --replacepkgs --replacefiles "${TMPDIR}/$(basename ${1})" - shift -done -``` - -`qtb-iterate`: - -```bash -#!/bin/sh - -set -e - -# substitute path to your builder installation -pushd ${HOME}/builder >/dev/null - -# the following are needed only if you have sources outside builder -#rm -rf qubes-src/core-admin -#make COMPONENTS=core-admin get-sources - -make core-admin -qtb-install qubes-src/core-admin/rpm/x86_64/qubes-core-dom0-*.rpm -qtb-runtests -``` - -### Hooking git - -I (woju) have those two git hooks. They ensure tests are passing (or are marked as expected failure) when committing and pushing. For committing it is only possible to run tests that may be executed from git repo (even if the rest were available, I probably wouldn't want to do that). For pushing, I also install RPM and run tests on testbench. - -`core-admin/.git/hooks/pre-commit`: (you may retain also the default hook, here omitted for readability) - -```bash -#!/bin/sh - -set -e - -python -c "import sys, qubes.tests.run; sys.exit(not qubes.tests.run.main())" -``` - -`core-admin/.git/hooks/pre-push`: - -```bash -#!/bin/sh - -exec qtb-iterate -``` +--- \ No newline at end of file diff --git a/developer/debugging/vm-interface.md b/developer/debugging/vm-interface.md index d6fc2dff0..a2c1d883c 100644 --- a/developer/debugging/vm-interface.md +++ b/developer/debugging/vm-interface.md @@ -7,219 +7,7 @@ redirect_from: - /doc/VMInterface/ - /doc/SystemDoc/VMInterface/ - /wiki/SystemDoc/VMInterface/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/vm-interface.html ref: 47 title: Qube configuration interface ---- - -Qubes VM have some settings set by dom0 based on VM settings. There are multiple configuration channels, which includes: - -- QubesDB -- XenStore (in Qubes 2, data the same as in QubesDB, keys without leading `/`) -- Qubes RPC (called at VM startup, or when configuration changed) -- GUI protocol - -## QubesDB - -### Keys exposed by dom0 to VM - -- `/qubes-vm-type` - VM type, the same as `type` field in `qvm-prefs`. One of `AppVM`, `ProxyVM`, `NetVM`, `TemplateVM`, `HVM`, `TemplateHVM` -- `/qubes-vm-updatable` - flag whether VM is updatable (whether changes in root.img will survive VM restart). One of `True`, `False` -- `/qubes-vm-persistence` - what data do persist between VM restarts: - - `full` - all disks - - `rw-only` - only `/rw` disk - - `none` - none -- `/qubes-timezone - name of timezone based on dom0 timezone. For example `Europe/Warsaw` -- `/qubes-keyboard` (deprecated in R4.1) - keyboard layout based on dom0 layout. Its syntax is suitable for `xkbcomp` command (after expanding escape sequences like `\n` or `\t`). This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). This entry is created as part of gui-daemon initialization (so not available when gui-daemon disabled, or not started yet). -- `/keyboard-layout` - keyboard layout based on GuiVM layout. Its syntax can be `layout+variant+options`, `layout+variant`, `layout++options` or simply `layout`. For example, `fr+oss`, `pl++compose:caps` or `fr`. This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). -- `/qubes-debug-mode` - flag whether VM has debug mode enabled (qvm-prefs setting). One of `1`, `0` -- `/qubes-service/SERVICE_NAME` - subtree for VM services controlled from dom0 (using the `qvm-service` command or Qubes Manager). One of `1`, `0`. Note that not every service will be listed here, if entry is missing, it means "use VM default". A list of currently supported services is in the `qvm-service` man page. -- `/qubes-netmask` - network mask (only when VM has netvm set); currently hardcoded "255.255.255.0" -- `/qubes-ip - IP address for this VM (only when VM has netvm set) -- `/qubes-gateway` - default gateway IP (only when VM has netvm set); VM should add host route to this address directly via eth0 (or whatever default interface name is) -- `/qubes-primary-dns` - primary DNS address (only when VM has netvm set) -- `/qubes-secondary-dns` - secondary DNS address (only when VM has netvm set) -- `/qubes-netvm-gateway` - same as `qubes-gateway` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM) -- `/qubes-netvm-netmask` - same as `qubes-netmask` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM) -- `/qubes-netvm-network` - network address (only when VM serves as network backend - ProxyVM and NetVM); can be also calculated from qubes-netvm-gateway and qubes-netvm-netmask -- `/qubes-netvm-primary-dns` - same as `qubes-primary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to primary DNS server -- `/qubes-netvm-secondary-dns` - same as `qubes-secondary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to secondary DNS server -- `/guivm-windows-prefix` - title prefix for any window not originating from another qube. This means windows of applications running in GuiVM itself - -#### Firewall rules in 3.x - -QubesDB is also used to configure firewall in ProxyVMs. Rules are stored in -separate key for each target VM. Entries: - -- `/qubes-iptables` - control entry - dom0 writing `reload` here signals `qubes-firewall` service to reload rules -- `/qubes-iptables-header` - rules not related to any particular VM, should be applied before domains rules -- `/qubes-iptables-domainrules/NNN` - rules for domain `NNN` (arbitrary number) -in `iptables-save` format. Rules are self-contained - fill `FORWARD` iptables -chain and contains all required matches (source IP address etc), as well as -final default action (`DROP`/`ACCEPT`) - -VM after applying rules may signal some error, writing a message to -`/qubes-iptables-error` key. This does not exclude any other way of -communicating problems - like a popup. - -#### Firewall rules in 4.x - -QubesDB is also used to configure firewall in ProxyVMs. Each rule is stored as -a separate entry, grouped on target VM: - -- `/qubes-firewall/SOURCE_IP` - base tree under which rules are placed. All -rules there should be applied to filter traffic coming from `SOURCE_IP`. This -can be either IPv4 or IPv6 address. -Dom0 will do an empty write to this top level entry after finishing rules -update, so VM can setup a watch here to trigger rules reload. -- `/qubes-firewall/SOURCE_IP/policy` - default action if no rule matches: -`drop` or `accept`. -- `/qubes-firewall/SOURCE_IP/NNNN` - rule number `NNNN` - decimal number, - padded with zeros. Se below for rule format. All the rules should be - applied in order of rules implied by those numbers. Note that QubesDB - itself does not impose any ordering (you need to sort the rules after - retrieving them). The first rule has number `0000`. - -Each rule is a single QubesDB entry, consisting of pairs `key=value` separated -by space. QubesDB enforces limit on a single entry length - 3072 bytes. -Possible options for a single rule: - -- `action`, values: `accept`, `drop`; this is present in every rule -- `dst4`, value: destination IPv4 address with a mask; for example: `192.168.0.0/24` -- `dst6`, value: destination IPv6 address with a mask; for example: `2000::/3` -- `dsthost`, value: DNS hostname of destination host -- `proto`, values: `tcp`, `udp`, `icmp` -- `specialtarget`, value: One of predefined target, currently defined values: - - `dns` - such option should match DNS traffic to default DNS server (but - not any DNS server), on both TCP and UDP -- `dstports`, value: destination ports range separated with `-`, valid only - together with `proto=tcp` or `proto=udp`; for example `1-1024`, `80-80` -- `icmptype`, value: numeric (decimal) icmp message type, for example `8` for - echo request, valid only together with `proto=icmp` -- `dpi`, value: Deep Packet Inspection protocol (like: HTTP, SSL, SMB, SSH, SMTP) or the default 'NO' as no DPI, only packet filtering - -Options must appear in the rule in the order listed above. Duplicated options -are forbidden. - -A rule matches only when all predicates match. Only one of `dst4`, `dst6` or -`dsthost` can be used in a single rule. - -If tool applying firewall encounters any parse error (unknown option, invalid -value, duplicated option, etc), it should drop all the traffic coming from that `SOURCE_IP`, -regardless of properly parsed rules. - -Example valid rules: - -- `action=accept dst4=8.8.8.8 proto=udp dstports=53-53` -- `action=drop dst6=2a00:1450:4000::/37 proto=tcp` -- `action=accept specialtarget=dns` -- `action=drop proto=tcp specialtarget=dns` - drop DNS queries sent using TCP -- `action=drop` - -### Keys set by VM for passing info to dom0 - -- `memory/meminfo` (**xenstore**) - used memory (updated by qubes-meminfo-writer), input information for qmemman; - - Qubes 3.x format: 6 lines (EOL encoded as `\n`), each in format "FIELD: VALUE kB"; fields: `MemTotal`, `MemFree`, `Buffers`, `Cached`, `SwapTotal`, `SwapFree`; meaning the same as in `/proc/meminfo` in Linux. - - Qubes 4.0+ format: used memory size in the VM, in kbytes -- `/qubes-block-devices` - list of block devices exposed by this VM, each device (subdirectory) should be named in a way that VM can attach the device based on it. Each should contain these entries: - - `desc` - device description (ASCII text) - - `size` - device size in bytes - - `mode` - default connection mode; `r` for read-only, `w` for read-write -- `/qubes-usb-devices` - list of USB devices exposed by this VM, each device (subdirectory) should contain: - - `desc` - device description (ASCII text) - - `usb-ver` - USB version (1, 2 or 3) - -## Qubes RPC - -Services called by dom0 to provide some VM configuration: - -- `qubes.SetMonitorLayout` - provide list of monitors, one per line. Each line contains four numbers: `width height X Y width_mm height_mm` (physical dimensions - `width_mm` and `height_mm` - are optional) -- `qubes.WaitForSession` - called to wait for full VM startup -- `qubes.GetAppmenus` - receive appmenus from given VM (template); TODO: describe format here -- `qubes.GetImageRGBA` - receive image/application icon. Protocol: - - 1. Caller sends name of requested icon. This can be one of: - * `xdgicon:NAME` - search for NAME in standard icons theme - * `-` - get icon data from stdin (the caller), can be prefixed with format name, for example `png:-` - * file name - 2. The service responds with image dimensions: width and height as - decimal numbers, separated with space and with EOL marker at the and; then - image data in RGBA format (32 bits per pixel) -- `qubes.SetDateTime` - set VM time, called periodically by dom0 (can be - triggered manually from dom0 by calling `qvm-sync-clock`). The service - receives one line at stdin - time in format of `date -u -Iseconds`, for - example `2015-07-31T16:10:43+0000`. -- `qubes.SetGuiMode` - called in HVM to switch between fullscreen and seamless - GUI mode. The service receives a single word on stdin - either `FULLSCREEN` - or `SEAMLESS` -- `qubes.ResizeDisk` - called to inform that underlying disk was resized. - Name of disk image is passed on standard input (`root`, `private`, `volatile`, - or other). This is used starting with Qubes 4.0. - -Other Qrexec services installed by default: - -- `qubes.Backup` - store Qubes backup. The service receives location chosen by - the user (one line, terminated by `\n`), the backup archive ([description of - backup format](/doc/BackupEmergencyRestoreV2/)) -- `qubes.DetachPciDevice` - service called in reaction to `qvm-pci -d` call on - running VM. The service receives one word - BDF of device to detach. When the - service call ends, the device will be detached -- `qubes.Filecopy` - receive some files from other VM. Files sent in [qfile format](/doc/qfilecopy/) -- `qubes.OpenInVM` - open a file in called VM. Service receives a single file on stdin (in - [qfile format](/doc/qfilecopy/). After a file viewer/editor is terminated, if - the file was modified, can be sent back (just raw content, without any - headers); otherwise service should just terminate without sending anything. - This service is used by both `qvm-open-in-vm` and `qvm-open-in-dvm` tools. When - called in DispVM, service termination will trigger DispVM cleanup. -- `qubes.Restore` - retrieve Qubes backup. The service receives backup location - entered by the user (one line, terminated by `\n`), then should output backup - archive in [qfile format](/doc/qfilecopy/) (core-agent-linux component contains - `tar2qfile` utility to do the conversion) -- `qubes.SelectDirectory`, `qubes.SelectFile` - services which should show - file/directory selection dialog and return (to stdout) a single line - containing selected path, or nothing in the case of cancellation -- `qubes.SuspendPre` - service called in every VM with PCI device attached just - before system suspend -- `qubes.SuspendPost` - service called in every VM with PCI device attached just - after system resume -- `qubes.SyncNtpClock` - service called to trigger network time synchronization. - Service should synchronize local VM time and terminate when done. -- `qubes.WindowIconUpdater` - service called by VM to send icons of individual - windows. The protocol there is simple one direction stream: VM sends window ID - followed by icon in `qubes.GetImageRGBA` format, then next window ID etc. VM - can send icon for the same window multiple times to replace previous one (for - example for animated icons) -- `qubes.VMShell` - call any command in the VM; the command(s) is passed one per line - - `qubes.VMShell+WaitForSession` waits for full VM startup first -- `qubes.VMExec` - call any command in the VM, without using shell, the command - needs to be passed as argument and encoded as follows: - - the executable name and arguments are separated by `+` - - everything except alphanumeric characters, `.` and `_` needs to be - escaped - - bytes are escaped as `-HH` (where `HH` is hex code, capital letters only) - - `-` itself can be escaped as `--` - - example: to run `ls -a /home/user`, use - `qubes.VMExec+ls+--a+-2Fhome-2Fuser` -- `qubes.VMExecGUI` - a variant of `qubes.VMExec` that waits for full VM - startup first - -Services called in GuiVM: - -- `policy.Ask`, `policy.Notify` - confirmation prompt and notifications for -Qubes RPC calls, see [qrexec-policy implementation](/doc/qrexec-internals/#qrexec-policy-implementation) -for a detailed description. - -Currently Qubes still calls few tools in VM directly, not using service -abstraction. This will change in the future. Those tools are: - -- `/usr/lib/qubes/qubes-download-dom0-updates.sh` - script to download updates (or new packages to be installed) for dom0 (`qubes-dom0-update` tool) -- `date -u -Iseconds` - called directly to retrieve time after calling `qubes.SyncNtpClock` service (`qvm-sync-clock` tool) -- `nm-online -x` - called before `qubes.SyncNtpClock` service call by `qvm-sync-clock` tool -- `resize2fs` - called to resize filesystem on /rw partition by `qvm-grow-private` tool -- `gpk-update-viewer` - called by Qubes Manager to display available updates in a TemplateVM -- `systemctl start qubes-update-check.timer` (and similarly stop) - called when enabling/disabling updates checking in given VM (`qubes-update-check` [qvm-service](/doc/qubes-service/)) - -Additionally, automatic tests extensively run various commands directly in VMs. We do not plan to change that. - -## GUI protocol - -GUI initialization includes passing the whole screen dimensions from dom0 to VM. This will most likely be overwritten by qubes.SetMonitorLayout Qubes RPC call. +--- \ No newline at end of file diff --git a/developer/debugging/windows-debugging.md b/developer/debugging/windows-debugging.md index 852d1c67f..d3f8b2fa2 100644 --- a/developer/debugging/windows-debugging.md +++ b/developer/debugging/windows-debugging.md @@ -6,257 +6,7 @@ redirect_from: - /en/doc/windows-debugging/ - /doc/WindowsDebugging/ - /wiki/WindowsDebugging/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/debugging/windows-debugging.html ref: 50 title: Windows debugging ---- - -Debugging Windows code can be tricky in a virtualized environment. The guide below assumes Xen hypervisor and Windows 7 VMs. - -User-mode debugging is usually straightforward if it can be done on one machine. Just duplicate your normal debugging environment in the VM. - -Things get complicated if you need to perform kernel debugging or troubleshoot problems that only manifest on system boot, user logoff or similar. For that you need two Windows VMs: the *host* and the *target*. The *host* will contain [WinDbg](https://msdn.microsoft.com/en-us/library/windows/hardware/ff551063(v=vs.85).aspx) installation, your source code and private symbols. The *target* will run the code being debugged. Both will be linked by virtual serial ports. - -- First, you need to prepare separate copies of both *target* and *host* VM configuration files with some changes. Copy the files from **/var/lib/qubes/appvms/vmname/vmname.conf** to some convenient location, let's call them **host.conf** and **target.conf**. -- In both copied files add the following line at the end: `serial = 'pty'`. This will make Xen connect VM's serial ports to dom0's ptys. -- From now on you need to start both VMs like this: `qvm-start --custom-config=/your/edited/host.conf host` -- To connect both VM serial ports together you will either need [socat](http://www.dest-unreach.org/socat/) or a custom utility described later. -- To determine which dom0 pty corresponds to VM's serial port you need to read xenstore, example script below: - -```bash -#!/bin/sh - -id1=$(xl domid "$1-dm") -tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty) -echo $tty1 -``` - -Pass it a running VM name and it will output the corresponding pty name. - -- To connect both ptys you can use [socat](http://www.dest-unreach.org/socat/) like that: - -```bash -#!/bin/sh - -id1=$(xl domid "$1-dm") -id2=$(xl domid "$2-dm") -tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty) -tty2=$(xenstore-read /local/domain/${id2}/device/console/3/tty) -socat $tty1,raw $tty2,raw -``` - -...but there is a catch. Xen seems to process the traffic that goes through serial ports and changes all **0x0a** bytes into **0x0d, 0x0a** pairs (newline conversion). I didn't find a way to turn that off (setting ptys to raw mode didn't change anything) and it's not mentioned anywhere on the Internet, so maybe it's something on my system. If the above script works for you then you don't need anything more in dom0. - -- On the *target* system, run `bcdedit /set debug on` on the console to turn on kernel debugging. It defaults to the first serial port. -- On the *host* system, install [WinDbg](http://msdn.microsoft.com/en-us/library/windows/hardware/ff551063(v=vs.85).aspx) and start the kernel debug (Ctrl-K), choose **com1** as the debug port. -- Reboot the *target* VM. -- Run the above shell script in dom0. -- If everything is fine you should see the proper kernel debugging output in WinDbg. However, if you see something like that: - - ~~~ - Opened \\.\com1 - Waiting to reconnect... - Connected to Windows 7 7601 x64 target at (Wed Mar 19 20:35:43.262 2014 (UTC + 1:00)), ptr64 TRUE - Kernel Debugger connection established. - Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols - Executable search path is: - ... Retry sending the same data packet for 64 times. - The transport connection between host kernel debugger and target Windows seems lost. - please try resync with target, recycle the host debugger, or reboot the target Windows. - Unable to read KTHREAD address fffff8000281ccc0 - ************************************************************************** - Unable to read debugger data block header - ************************************************************************** - Unable to read KTHREAD address fffff8000281ccc0 - Unable to read PsLoadedModuleList - Unable to read KTHREAD address fffff8000281ccc0 - ************************************************************************** - Unable to read debugger data block header - ************************************************************************** - ~~~ - - ...then you're most likely a victim of the CRLF issue mentioned above. To get around it I wrote a small utility that basically does what socat would do and additionally corrects those replaced bytes in the stream. It's not pretty but it works: - -```c -#include -#include -#include -#include - -int fd1, fd2; -char mark = ' '; - -void out(unsigned char c) -{ - static int count = 0; - static unsigned char buf[17] = {0}; - - // relay to ouptput port - write(fd2, &c, 1); - fprintf(stderr, "%c", mark); - - /* dump all data going over the line - if (count == 0) - fprintf(stderr, "%c", mark); - fprintf(stderr, "%02x ", c); - if (c >= 0x20 && c < 0x80) - buf[count] = c; - else - buf[count] = '.'; - count++; - if (count == 0x10) - { - count = 0; - fprintf(stderr, " %s\n", buf); - } - */ -} - -int main(int argc, char* argv[]) -{ - unsigned char c = 0; - struct termios tio; - ssize_t size; - - if (argc < 3) - { - fprintf(stderr, "Usage: %s pty1 pty2 [mark character]\n", argv[0]); - return EINVAL; - } - - fd1 = open(argv[1], O_RDONLY | O_NOCTTY); - if (fd1 <= 0) - { - perror("open fd1"); - return errno; - } - fd2 = open(argv[2], O_WRONLY | O_NOCTTY); - if (fd2 <= 0) - { - perror("open fd2"); - return errno; - } -/* - // This doesn't make any difference which supports the theory - // that it's Xen who corrupts the byte stream. - cfmakeraw(&tio); - if (tcsetattr(fd1, TCSANOW, &tio) < 0) - { - perror("tcsetattr 1"); - return errno; - } - if (tcsetattr(fd2, TCSANOW, &tio) < 0) - { - perror("tcsetattr 2"); - return errno; - } -*/ - if (argc == 4) - mark = argv[3][0]; - - while (1) - { - size = read(fd1, &c, 1); - if (size <= 0) - break; - -parse: - if (c == 0x0d) - { - size = read(fd1, &c, 1); - if (size <= 0) - { - out(0x0d); - break; - } - if (c == 0x0a) - { - out(0x0a); - } - else - { - out(0x0d); - goto parse; - } - } - else - out(c); - } - - close(fd1); - close(fd2); - return 0; -} -``` - -> This utility is a unidirectional relay so you need to run two instances to get duplex communication, like: -> -> #!/bin/sh -> -> id1=$(xl domid "$1-dm") -> id2=$(xl domid "$2-dm") -> tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty) -> tty2=$(xenstore-read /local/domain/${id2}/device/console/3/tty) -> ./ptycrlf ${tty1} ${tty2} - & -> ./ptycrlf ${tty2} ${tty1} + & - -> With this everything should be good: -> -> ~~~ -> Opened \\.\com1 -> Waiting to reconnect... -> Connected to Windows 7 7601 x64 target at (Wed Mar 19 20:56:31.371 2014 (UTC + 1:00)), ptr64 TRUE -> Kernel Debugger connection established. -> Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols -> Executable search path is: -> Windows 7 Kernel Version 7601 MP (1 procs) Free x64 -> Built by: 7601.18247.amd64fre.win7sp1_gdr.130828-1532 -> Machine Name: -> Kernel base = 0xfffff800`0261a000 PsLoadedModuleList = 0xfffff800`0285d6d0 -> System Uptime: not available -> ~~~ - -# Debugging HVMs in the Qubes R4.0 - -There are two main issues to be adopted to get all things to work in the R4.0. - -## Add a virtual serial port - -Qemu in the stub domain with virtual serial port added in a recommended way (``````) fails to start (Could not open '/dev/hvc1': No such device). It seems like a lack of multiple xen consoles support/configuration. The only way that I have found is to attach serial port explicitly to the available console. - -1. Unpack stub domain in dom0: - -```shell_session -$ mkdir stubroot -$ cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz -$ cd stubroot -$ gunzip stubdom-linux-rootfs.gz -$ cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs -$ rm stubdom-linux-rootfs -``` - -2. Edit Init script to remove last loop and to add "-serial /dev/hvc0" to the qemu command line. - -3. Apply changes: - -```shell_session -$ find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../stubdom-linux-rootfs -$ sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot -``` - -## Connect two consoles - -Run the following script: - -```shell -debugname1=win7new -debugname2=win7dbg -id1=$(xl domid "$debugname1-dm") -id2=$(xl domid "$debugname2-dm") - -tty1=$(xenstore-read /local/domain/${id1}/console/tty) -tty2=$(xenstore-read /local/domain/${id1}/console/tty) - -socat $tty1,raw $tty2,raw -``` - -Happy debugging! +--- \ No newline at end of file diff --git a/developer/general/continuous-integration.md b/developer/general/continuous-integration.md index 9c00390c9..0d64a8b89 100644 --- a/developer/general/continuous-integration.md +++ b/developer/general/continuous-integration.md @@ -2,38 +2,6 @@ lang: en layout: doc permalink: /doc/continuous-integration/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/continuous-integration.html title: Continuous integration (CI) ---- - -This page explains the [continuous integration -(CI)](https://en.wikipedia.org/wiki/Continuous_integration) infrastructure that -the Qubes OS Project uses. - -## Website and documentation - -The following commands may be useful as a way to interact with our CI -infrastructure on website -([qubesos.github.io](https://github.com/QubesOS/qubesos.github.io)) and -documentation ([qubes-doc](https://github.com/QubesOS/qubes-doc)) pull requests -(PRs). Note that special permissions may be required to use some of these -commands. These commands are generally issued by adding a comment to a PR -containing only the command. - -- `PipelineRetry`: Attempts to run the entire build pipeline over again. This - can be useful if CI incorrectly uses a stale branch instead of testing the PR - as if it were merged into `master`. - -- `PipelineRetryFailed`: Retry just failed CI jobs. Do not update the branch. - -- `PipelineRefresh`: Like `PipelineRetry`, except it only fetches the job status - from GitLab. It doesn't schedule a new build. - -- `TestDeploy`: Deploys a test website, which is a live version of the Qubes - website as if this PR had been merged. This can be useful for previewing a PR - on a live public website. **Note:** You must wait for the site to finish - building before issuing this command, or else it will deploy an empty - website. To find the URL of the test website, look for text similar to "This - branch was successfully deployed" and a button named something like "View - deployment." Note that there are two different testing sites: `wwwtest` is - manually updated, whereas `wwwpreview` is managed by the `TestDeploy` - command. +--- \ No newline at end of file diff --git a/developer/general/devel-books.md b/developer/general/devel-books.md index d7ae9ae07..259ebb151 100644 --- a/developer/general/devel-books.md +++ b/developer/general/devel-books.md @@ -6,26 +6,7 @@ redirect_from: - /en/doc/devel-books/ - /doc/DevelBooks/ - /wiki/DevelBooks/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/devel-books.html ref: 32 title: Developer books ---- - -Below is a list of various books that might be useful in learning some basics needed for Qubes development. - -- A must-read about Xen internals: - - _[The Definitive Guide to the Xen Hypervisor](https://www.amazon.com/Definitive-Guide-Xen-Hypervisor/dp/013234971X)_, by David Chisnall - -- Some good books about the Linux kernel: - - _[Linux Kernel Development](https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468)_, by Robert Love - - _[Linux Device Drivers](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903)_, by Jonathan Corbet - -- Solid intro into Trusted Computing: - - _[Dynamics of a Trusted Platform](https://www.amazon.com/Dynamics-Trusted-Platform-Buildin-Grawrock/dp/1934053082)_, by David Grawrock (original Intel architect for TXT) - -- Good book about GIT: - - _[Pro Git](https://git-scm.com/book/en/v2)_, by Scott Chacon and Ben Straub (complete book available free online) - -- Useful books about Python: - - _[Programming in Python 3](http://www.qtrac.eu/py3book.html)_, by Mark Summerfield - - _[Rapid GUI Programming with Python and Qt](http://www.qtrac.eu/pyqtbook.html)_, by Mark Summerfield - (Although note that [Qt is being replaced by GTK](/doc/usability-ux/#gnome-kde-and-xfce) in Qubes code.) +--- \ No newline at end of file diff --git a/developer/general/documentation-style-guide.md b/developer/general/documentation-style-guide.md index 450640746..d1126b752 100644 --- a/developer/general/documentation-style-guide.md +++ b/developer/general/documentation-style-guide.md @@ -7,439 +7,7 @@ redirect_from: - /en/doc/doc-guidelines/ - /wiki/DocStyle/ - /doc/DocStyle/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/documentation-style-guide.html ref: 30 title: Documentation style guide ---- - -_Also see [how to edit the documentation](/doc/how-to-edit-the-documentation/)._ - -Qubes OS documentation pages are stored as plain text Markdown files in the -[qubes-doc](https://github.com/QubesOS/qubes-doc) repository. By cloning and -regularly pulling from this repo, users can maintain their own up-to-date -offline copy of all Qubes documentation rather than relying solely on the web. - -The documentation is a volunteer community effort. People like you are -constantly working to make it better. If you notice something that can be fixed -or improved, please [edit the -documentation](/doc/how-to-edit-the-documentation/)! - -This page explains the standards we follow for writing, formatting, and -organizing the documentation. Please follow these guidelines and conventions -when editing the documentation. For the standards governing the website as a -whole, please see the [website style guide](/doc/website-style-guide). - -## Markdown conventions - -All the documentation is written in Markdown for maximum accessibility. When -making contributions, please observe the following style conventions. If you're -not familiar with Markdown syntax, -[this](https://daringfireball.net/projects/markdown/) is a great resource. - -### Hyperlink syntax - -Use non-reference-style links like `[website](https://example.com/)`. Do *not* -use reference-style links like `[website][example]`, `[website][]` or -`[website]`. This facilitates the localization process. - -### Relative vs. absolute links - -Always use relative rather than absolute paths for internal website links. For -example, use `/doc/documentation-style-guide/` instead of -`https://www.qubes-os.org/doc/documentation-style-guide/`. - -You may use absolute URLs in the following cases: - -- External links -- URLs that appear inside code blocks (e.g., in comments and document - templates, and the plain text reproductions of [QSBs](/security/qsb/) and - [Canaries](/security/canary/)), since they're not hyperlinks -- Git repo files like `README.md` and `CONTRIBUTING.md`, since they're not part - of the website itself but rather of the auxiliary infrastructure supporting - the website - -This rule is important because using absolute URLs for internal website links -breaks: - -- Serving the website offline -- Website localization -- Generating offline documentation -- Automatically redirecting Tor Browser visitors to the correct page on the - onion service mirror - -### Image linking - -See [how to add images](/doc/how-to-edit-the-documentation/#how-to-add-images) -for the required syntax. This will make the image a hyperlink to the image -file, allowing the reader to click on the image in order to view the full image -by itself. This is important. Following best practices, our website has a -responsive design, which allows the website to render appropriately across all -screen sizes. When viewing this page on a smaller screen, such as on a mobile -device, the image will automatically shrink down to fit the screen. If visitors -cannot click on the image to view it in full size, then, depending on their -device, they may have no way see the details in the image clearly. - -In addition, make sure to link only to images in the -[qubes-attachment](https://github.com/QubesOS/qubes-attachment) repository. Do -not attempt to link to images hosted on other websites. - -### HTML and CSS - -Do not write HTML inside Markdown documents (except in rare, unavoidable cases, -such as alerts). In particular, never include HTML or CSS for styling, -formatting, or white space control. That belongs in the (S)CSS files instead. - -### Headings - -Do not use `h1` headings (single `#` or `======` underline). These are -automatically generated from the `title:` line in the YAML front matter. - -Use Atx-style syntax for headings: `##h2`, `### h3`, etc. Do not use -underlining syntax (`-----`). - -### Indentation - -Use spaces instead of tabs. Use hanging indentations where appropriate. - -### Lists - -If appropriate, make numerals in numbered lists match between Markdown source -and HTML output. Some users read the Markdown source directly, and this makes -numbered lists easier to follow. - -### Code blocks - -When writing code blocks, use [syntax -highlighting](https://github.github.com/gfm/#info-string) where possible (see -[here](https://github.com/jneen/rouge/wiki/List-of-supported-languages-and-lexers) -for a list of supported languages). Use `[...]` for anything omitted. - -### Line wrapping - -Hard wrap Markdown lines at 80 characters, unless the line can't be broken -(e.g., code or a URL). - -## Writing guidelines - -### Correct use of terminology - -Familiarize yourself with the terms defined in the [glossary](/doc/glossary/). -Use these terms consistently and accurately throughout your writing. - -### Sentence case in headings - -Use sentence case (rather than title case) in headings for the reasons -explained -[here](https://www.sallybagshaw.com.au/articles/sentence-case-v-title-case/). -In particular, since the authorship of the Qubes documentation is decentralized -and widely distributed among users from around the world, many contributors -come from regions with different conventions for implementing title case, not -to mention that there are often differing style guide recommendations even -within a single region. It is much easier for all of us to implement sentence -case consistently across our growing body of pages, which is very important for -managing the ongoing maintenance burden and sustainability of the -documentation. - -### Writing command-line examples - -When providing command-line examples: - -- Tell the reader where to open a terminal (dom0 or a specific domU), and show - the command along with its output (if any) in a code block, e.g.: - - ~~~markdown - Open a terminal in dom0 and run: - ```shell_session - $ cd test - $ echo Hello - Hello - ``` - ~~~ - -- Precede each command with the appropriate command prompt: At a minimum, the - prompt should contain a trailing `#` (for the user `root`) or `$` (for other - users) on Linux systems and `>` on Windows systems, respectively. - -- Don't try to add comments inside the code block. For example, *don't* do - this: - - ~~~markdown - Open a terminal in dom0 and run: - ```shell_session - # Navigate to the new directory - $ cd test - # Generate a greeting - $ echo Hello - Hello - ``` - ~~~ - - The `#` symbol preceding each comment is ambiguous with a root command - prompt. Instead, put your comments *outside* of the code block in normal - prose. - -### Variable names in commands - -Syntactically distinguish variables in commands. For example, this is -ambiguous: - - $ qvm-run --dispvm=disposable-template --service qubes.StartApp+xterm - -It should instead be: - - $ qvm-run --dispvm= --service qubes.StartApp+xterm - -Note that we syntactically distinguish variables in three ways: - -1. Surrounding them in angled brackets (`< >`) -2. Using underscores (`_`) instead of spaces between words -3. Using all capital letters - -We have observed that many novices make the mistake of typing the surrounding -angled brackets (`< >`) on the command line, even after substituting the -desired real value between them. Therefore, in documentation aimed at novices, -we also recommend clarifying that the angled brackets should not be typed. This -can be accomplished in one of several ways: - -- Explicitly say something like "without the angled brackets." -- Provide an example command using real values that excludes the angled - brackets. -- If you know that almost all users will want to use (or should use) a specific - command containing all real values and no variables, you might consider - providing exactly that command and forgoing the version with variables. - Novices may not realize which parts of the command they can substitute with - different values, but if you've correctly judged that they should use the - command you've provided as is, then this shouldn't matter. - -## Organizational guidelines - -### Do not duplicate documentation - -Duplicating documentation is almost always a bad idea. There are many reasons -for this. The main one is that almost all documentation has to be updated as -some point. When similar documentation appears in more than one place, it is -very easy for it to get updated in one place but not the others (perhaps -because the person updating it doesn't realize it's in more than once place). -When this happens, the documentation as a whole is now inconsistent, and the -outdated documentation becomes a trap, especially for novice users. Such traps -are often more harmful than if the documentation never existed in the first -place. The solution is to **link** to existing documentation rather than -duplicating it. There are some exceptions to this policy (e.g., information -that is certain not to change for a very long time), but they are rare. - -### Core vs. external documentation - -Core documentation resides in the [Qubes OS Project's official -repositories](https://github.com/QubesOS/), mainly in -[qubes-doc](https://github.com/QubesOS/qubes-doc). External documentation can -be anywhere else (such as forums, community websites, and blogs), but there is -an especially large collection in the [Qubes -Community](https://github.com/Qubes-Community) project. External documentation -should not be submitted to [qubes-doc](https://github.com/QubesOS/qubes-doc). -If you've written a piece of documentation that is not appropriate for -[qubes-doc](https://github.com/QubesOS/qubes-doc), we encourage you to submit -it to the [Qubes Community](https://github.com/Qubes-Community) project -instead. However, *linking* to external documentation from -[qubes-doc](https://github.com/QubesOS/qubes-doc) is perfectly fine. Indeed, -the maintainers of the [Qubes Community](https://github.com/Qubes-Community) -project should regularly submit PRs against the documentation index (see [How -to edit the documentation -index](/doc/how-to-edit-the-documentation/#how-to-edit-the-documentation-index)) -to add and update Qubes Community links in the ["External -documentation"](/doc/#external-documentation) section of the documentation -table of contents. - -The main difference between **core** (or **official**) and **external** (or -**community** or **unofficial**) documentation is whether it documents software -that is officially written and maintained by the Qubes OS Project. The purpose -of this distinction is to keep the core docs maintainable and high-quality by -limiting them to the software output by the Qubes OS Project. In other words, -we take responsibility for documenting all of the software we put out into the -world, but it doesn't make sense for us to take on the responsibility of -documenting or maintaining documentation for anything else. For example, Qubes -OS may use a popular Linux distribution for an official -[TemplateVM](/doc/templates/). However, it would not make sense for a -comparatively small project like ours, with modest funding and a lean -workforce, to attempt to document software belonging to a large, richly-funded -project with an army of paid and volunteer contributors, especially when they -probably already have documentation of their own. This is particularly true -when it comes to Linux in general. Although many users who are new to Qubes are -also new to Linux, it makes absolutely no sense for our comparatively tiny -project to try to document Linux in general when there is already a plethora of -documentation out there. - -Many contributors do not realize that there is a significant amount of work -involved in *maintaining* documentation after it has been written. They may -wish to write documentation and submit it to the core docs, but they see only -their own writing process and fail to consider that it will have to be kept -up-to-date and consistent with the rest of the docs for years afterward. -Submissions to the core docs also have to [undergo a review -process](/doc/how-to-edit-the-documentation/#security) to ensure accuracy before -being merged, which takes up valuable time from the team. We aim to maintain -high quality standards for the core docs (style and mechanics, formatting), -which also takes up a lot of time. If the documentation involves anything -external to the Qubes OS Project (such as a website, platform, program, -protocol, framework, practice, or even a reference to a version number), the -documentation is likely to become outdated when that external thing changes. -It's also important to periodically review and update this documentation, -especially when a new Qubes release comes out. Periodically, there may be -technical or policy changes that affect all the core documentation. The more -documentation there is relative to maintainers, the harder all of this will be. -Since there are many more people who are willing to write documentation than to -maintain it, these individually small incremental additions amount to a -significant maintenance burden for the project. - -On the positive side, we consider the existence of community documentation to -be a sign of a healthy ecosystem, and this is quite common in the software -world. The community is better positioned to write and maintain documentation -that applies, combines, and simplifies the official documentation, e.g., -tutorials that explain how to install and use various programs in Qubes, how to -create custom VM setups, and introductory tutorials that teach basic Linux -concepts and commands in the context of Qubes. In addition, just because the -Qubes OS Project has officially written and maintains some flexible framework, -such as `qrexec`, it does not make sense to include every tutorial that says -"here's how to do something cool with `qrexec`" in the core docs. Such -tutorials generally also belong in the community documentation. - -See [#4693](https://github.com/QubesOS/qubes-issues/issues/4693) for more -background information. - -### Release-specific documentation - -*See [#5308](https://github.com/QubesOS/qubes-issues/issues/5308) for pending -changes to this policy.* - -We maintain only one set of documentation for Qubes OS. We do not maintain a -different set of documentation for each release of Qubes. Our single set of -Qubes OS documentation is updated on a continual, rolling basis. Our first -priority is to document all **current, stable releases** of Qubes. Our second -priority is to document the next, upcoming release (if any) that is currently -in the beta or release candidate stage. - -In cases where a documentation page covers functionality that differs -considerably between Qubes OS releases, the page should be subdivided into -clearly-labeled sections that cover the different functionality in different -releases (examples below). - -In general, avoid mentioning specific Qubes versions in the body text of -documentation, as these references rapidly go out of date and become misleading -to readers. - -#### Incorrect Example - -``` -## How to Foo - -Fooing is the process by which one foos. There are both general and specific -versions of fooing, which vary in usefulness depending on your goals, but for -the most part, all fooing is fooing. - -To foo in Qubes 3.2: - - $ qvm-foo - -Note that this does not work in Qubes 4.0, where there is a special widget -for fooing, which you can find in the lower-right corner of the screen in -the Foo Manager. Alternatively, you can use the more general `qubes-baz` -command introduced in 4.0: - - $ qubes-baz --foo - -Once you foo, make sure to close the baz before fooing the next bar. -``` - -#### Correct Example - -``` -## Qubes 3.2 - -### How to Foo - -Fooing is the process by which one foos. There are both general and specific -versions of fooing, which vary in usefulness depending on your goals, but for -the most part, all fooing is fooing. - -To foo: - - $ qvm-foo - -Once you foo, make sure to close the baz before fooing the next bar. - -## Qubes 4.0 - -### How to Foo - -Fooing is the process by which one foos. There are both general and specific -versions of fooing, which vary in usefulness depending on your goals, but for -the most part, all fooing is fooing. - -There is a special widget for fooing, which you can find in the lower-right -corner of the screen in the Foo Manager. Alternatively, you can use the -general `qubes-baz` command: - - $ qubes-baz --foo - -Once you foo, make sure to close the baz before fooing the next bar. -``` - -Subdividing the page into clearly-labeled sections for each release has several -benefits: - -- It preserves good content for older (but still supported) releases. Many - documentation contributors are also people who prefer to use the latest - release. Many of them are tempted to *replace* existing content that applies - to an older, supported release with content that applies only to the latest - release. This is somewhat understandable. Since they only use the latest - release, they may be focused on their own experience, and they may even - regard the older release as deprecated, even when it's actually still - supported. However, allowing this replacement of content would do a great - disservice to those who still rely on the older, supported release. In many - cases, these users value the stability and reliability of the older, - supported release. With the older, supported release, there has been more - time to fix bugs and make improvements in both the software and the - documentation. Consequently, much of the documentation content for this - release may have gone through several rounds of editing, review, and - revision. It would be a tragedy for this content to vanish while the very set - of users who most prize stability and reliability are depending on it. -- It's easy for readers to quickly find the information they're looking for, - since they can go directly to the section that applies to their release. -- It's hard for readers to miss information they need, since it's all in one - place. In the incorrect example, information that the reader needs could be - in any paragraph in the entire document, and there's no way to tell without - reading the entire page. In the correct example, the reader can simply skim - the headings in order to know which parts of the page need to be read and - which can be safely ignored. The fact that some content is repeated in the - two release-specific sections is not a problem, since no reader has to read - the same thing twice. Moreover, as one release gets updated, it's likely that - the documentation for that release will also be updated. Therefore, content - that is initially duplicated between release-specific sections will not - necessarily stay that way, and this is a good thing: We want the - documentation for a release that *doesn't* change to stay the same, and we - want the documentation for a release that *does* change to change along with - the software. -- It's easy for documentation contributors and maintainers to know which file - to edit and update, since there's only one page for all Qubes OS releases. - Initially creating the new headings and duplicating content that applies to - both is only a one-time cost for each page, and many pages don't even require - this treatment, since they apply to all currently-supported Qubes OS - releases. - -By contrast, an alternative approach, such as segregating the documentation -into two different branches, would mean that contributions that apply to both -Qubes releases would only end up in one branch, unless someone remembered to -manually submit the same thing to the other branch and actually made the effort -to do so. Most of the time, this wouldn't happen. When it did, it would mean a -second pull request that would have to be reviewed. Over time, the different -branches would diverge in non-release-specific content. Good general content -that was submitted only to one branch would effectively disappear once that -release was deprecated. (Even if it were still on the website, no one would -look at it, since it would explicitly be in the subdirectory of a deprecated -release, and there would be a motivation to remove it from the website so that -search results wouldn't be populated with out-of-date information.) - -For further discussion about release-specific documentation in Qubes, see -[here](https://groups.google.com/d/topic/qubes-users/H9BZX4K9Ptk/discussion). - -## Git conventions - -Please follow our [Git commit message -guidelines](/doc/coding-style/#commit-message-guidelines). +--- \ No newline at end of file diff --git a/developer/general/gsoc.md b/developer/general/gsoc.md index 9e4adad31..cb50d0ea1 100644 --- a/developer/general/gsoc.md +++ b/developer/general/gsoc.md @@ -4,657 +4,7 @@ layout: doc permalink: /gsoc/ redirect_from: - /GSoC/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/gsoc.html ref: 33 title: Google Summer of Code (GSoC) ---- - -## Information for Students - -Thank you for your interest in participating in the [Google Summer of Code program](https://summerofcode.withgoogle.com/) with the [Qubes OS team](/team/). You can read more about the Google Summer of Code program at the [official website](https://summerofcode.withgoogle.com/) and the [official FAQ](https://developers.google.com/open-source/gsoc/faq). - -Being accepted as a Google Summer of Code contributor is quite competitive. If you are interested in participating in the Summer of Code please be aware that you must be able to produce code for Qubes OS for 3-5 months. Your mentors, Qubes developers, will dedicate a portion of their time towards mentoring you. Therefore, we seek candidates who are committed to helping Qubes long-term and are willing to do quality work and be proactive in communicating with your mentor. - -You don't have to be a proven developer -- in fact, this whole program is meant to facilitate joining Qubes and other free and open source communities. The Qubes community maintains information about [contributing to Qubes development](/doc/contributing/#contributing-code) and [how to send patches](/doc/source-code/#how-to-send-patches). In order to contribute code to the Qubes project, you must be able to [sign your code](/doc/code-signing/). - -You should start learning the components that you plan on working on before the start date. Qubes developers are available on the [mailing lists](/support/#qubes-devel) for help. The GSoC timeline reserves a lot of time for bonding with the project -- use that time wisely. Good communication is key, you should plan to communicate with your team daily and formally report progress and plans weekly. Students who neglect active communication will be failed. - -### Overview of Steps - -- Join the [qubes-devel list](/support/#qubes-devel) and introduce yourself, and meet your fellow developers -- Read [Google's instructions for participating](https://developers.google.com/open-source/gsoc/) and the [GSoC Student Manual](https://google.github.io/gsocguides/student/) -- Take a look at the list of ideas below -- Come up with a project that you are interested in (and feel free to propose your own! Don't feel limited by the list below.) -- Read the Contributor Proposal guidelines below -- Write a first draft proposal and send it to the qubes-devel mailing list for review -- Submit proposal using [Google's web interface](https://summerofcode.withgoogle.com/) ahead of the deadline (this requires a Google Account!) -- Submit proof of enrollment well ahead of the deadline - -Coming up with an interesting idea that you can realistically achieve in the time available to you (one summer) is probably the most difficult part. We strongly recommend getting involved in advance of the beginning of GSoC, and we will look favorably on applications from prospective contributors who have already started to act like free and open source developers. - -Before the summer starts, there are some preparatory tasks which are highly encouraged. First, if you aren't already, definitely start using Qubes as your primary OS as soon as possible! Also, it is encouraged that you become familiar and comfortable with the Qubes development workflow sooner than later. A good way to do this (and also a great way to stand out as an awesome applicant and make us want to accept you!) might be to pick up some issues from [qubes-issues](https://github.com/QubesOS/qubes-issues/issues) (our issue-tracking repo) and submit some patches addressing them. Some suitable issues might be those with tags ["help wanted" and "P: minor"](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20label%3A%22P%3A%20minor%22%20label%3A%22help%20wanted%22) (although more significant things are also welcome, of course). Doing this will get you some practice with [qubes-builder](/doc/qubes-builder/), our code-signing policies, and some familiarity with our code base in general so you are ready to hit the ground running come summer. - -### Contributor proposal guidelines - -A project proposal is what you will be judged upon. Write a clear proposal on what you plan to do, the scope of your project, and why we should choose you to do it. Proposals are the basis of the GSoC projects and therefore one of the most important things to do well. - -Below is the application template: - -``` -# Introduction - -Every software project should solve a problem. Before offering the solution (your Google Summer of Code project), you should first define the problem. What’s the current state of things? What’s the issue you wish to solve and why? Then you should conclude with a sentence or two about your solution. Include links to discussions, features, or bugs that describe the problem further if necessary. - -# Project goals - -Be short and to the point, and perhaps format it as a list. Propose a clear list of deliverables, explaining exactly what you promise to do and what you do not plan to do. “Future developments” can be mentioned, but your promise for the Google Summer of Code term is what counts. - -# Implementation - -Be detailed. Describe what you plan to do as a solution for the problem you defined above. Include technical details, showing that you understand the technology. Illustrate key technical elements of your proposed solution in reasonable detail. - -# Timeline - -Show that you understand the problem, have a solution, have also broken it down into manageable parts, and that you have a realistic plan on how to accomplish your goal. Here you set expectations, so don’t make promises you can’t keep. A modest, realistic and detailed timeline is better than promising the impossible. - -If you have other commitments during GSoC, such as a job, vacation, exams, internship, seminars, or papers to write, disclose them here. GSoC should be treated like a full-time job, and we will expect approximately 40 hours of work per week. If you have conflicts, explain how you will work around them. If you are found to have conflicts which you did not disclose, you may be failed. - -Open and clear communication is of utmost importance. Include your plans for communication in your proposal; daily if possible. You will need to initiate weekly formal communications such as a detailed email to the qubes-devel mailing list. Lack of communication will result in you being failed. - -# About me - -Provide your contact information and write a few sentences about you and why you think you are the best for this job. Prior contributions to Qubes are helpful; list your commits. Name people (other developers, students, professors) who can act as a reference for you. Mention your field of study if necessary. Now is the time to join the relevant mailing lists. We want you to be a part of our community, not just contribute your code. - -Tell us if you are submitting proposals to other organizations, and whether or not you would choose Qubes if given the choice. - -Other things to think about: -* Are you comfortable working independently under a supervisor or mentor who is several thousand miles away, and perhaps 12 time zones away? How will you work with your mentor to track your work? Have you worked in this style before? -* If your native language is not English, are you comfortable working closely with a supervisor whose native language is English? What is your native language, as that may help us find a mentor who has the same native language? -* After you have written your proposal, you should get it reviewed. Do not rely on the Qubes mentors to do it for you via the web interface, although we will try to comment on every proposal. It is wise to ask a colleague or a developer to critique your proposal. Clarity and completeness are important. -``` - -## Project Ideas - -These project ideas were contributed by our developers and may be incomplete. If you are interested in submitting a proposal based on these ideas, you should contact the [qubes-devel mailing list](/support/#qubes-devel) and associated GitHub issue to learn more about the idea. - -``` -### Adding a Proposal - -**Project**: Something that you're totally excited about - -**Brief explanation**: What is the project, where does the code live? - -**Expected results**: What is the expected result in the timeframe given - -**Difficulty**: easy / medium / hard - -**Knowledge prerequisite**: Pre-requisites for working on the project. What coding language and knowledge is needed? -If applicable, links to more information or discussions - -**Size of the project**: either 175 hours (medium) or 350 hours (large) - -**Mentor**: Name and email address. - -``` - -### Qubes as a Vagrant provider - -**Project**: Qubes as a Vagrant provider - -**Brief explanation**: Currently using Vagrant on Qubes requires finding an image that uses Docker as isolation provider and running Docker in a qube, or downloading the Vagrantfile and manually setting up a qube according to the Vagrantfile. This project aims at simplifying this workflow. Since introduction of Admin API, it's possible for a qube to provision another qube - which is exactly what is needed for Vagrant. [Related discussion](https://groups.google.com/d/msgid/qubes-devel/535299ca-d16a-4a70-8223-a4ac6be4be41%40googlegroups.com) - -**Expected results**: - -- Design how Vagrant Qubes provider should look like, including: - - [box format](https://www.vagrantup.com/docs/plugins/providers.html#box-format) - - method for running commands inside (ssh vs qvm-run) -- Write a Vagrant provider able to create/start/stop/etc a VM -- Document how to configure and use the provider, including required qrexec policy changes and possibly firewall rules -- Write integration tests - -**Difficulty**: medium - -**Knowledge prerequisite**: - -- Ruby -- Vagrant concepts - -**Size of the project**: 350 hours - -**Mentor**: [Wojtek Porczyk](/team/), [Marek Marczykowski-Górecki](/team/) - -### System health monitor - -**Project**: System health monitor - -**Brief explanation**: A tool that informs the user about common system and configuration issues. Some of this is already available, but scattered across different places. See related issues: [6663](https://github.com/QubesOS/qubes-issues/issues/6663), [2134](https://github.com/QubesOS/qubes-issues/issues/2134) - -**Expected results**: - -- a tool / service that checks for common issues and things needing user attention, for example: - - some updates to be applied (separate widget already exists) - - running out of disk space (separate widget already exists) - - insecure USB configuration (USB in dom0) - - some system VM crashed - - ... - -- a GUI that provides terse overview of the system state, and notifies the user if something bad happens - -**Difficulty**: medium - -**Knowledge prerequisite**: - -- Python -- basic knowledge about systemd services -- PyGTK (optional) - -**Size of the project**: 350 hours - -**Mentor**: [Marta Marczykowska-Górecka](/team/) - -### Mechanism for maintaining in-VM configuration - -**Project**: Mechanism for maintaining in-VM configuration - -**Brief explanation**: Large number of VMs is hard to maintain. Templates helps with keeping them updated, but many applications have configuration in user home directory, which is not synchronized. - -**Expected results**: - -- Design a mechanism how to _safely_ synchronize application configuration living in user home directory (`~/.config`, some other "dotfiles"). Mechanism should be resistant against malicious VM forcing its configuration on other VMs. Some approach could be a strict control which VM can send what changes (whitelist approach, not blacklist). -- Implementation of the above mechanism. -- Documentation how to configure it securely. - -**Difficulty**: medium - -**Knowledge prerequisite**: - -- shell and/or python scripting -- Qubes OS qrexec services - -**Size of the project**: 175 hours - -**Mentor**: [Frédéric Pierret](/team/) - - - -### Qubes Live USB - -**Project**: Revive Qubes Live USB, integrate it with installer - -**Brief explanation**: Qubes Live USB is based on Fedora tools to build live -distributions. But for Qubes we need some adjustments: starting Xen instead of -Linux kernel, smarter copy-on-write handling (we run there multiple VMs, so a -lot more data to save) and few more. Additionally in Qubes 3.2 we have -so many default VMs that default installation does not fit in 16GB image -(default value) - some subset of those VMs should be chosen. Ideally we'd like -to have just one image being both live system and installation image. More -details: [#1552](https://github.com/QubesOS/qubes-issues/issues/1552), -[#1965](https://github.com/QubesOS/qubes-issues/issues/1965). - -**Expected results**: - -- Adjust set of VMs and templates included in live edition. -- Update and fix build scripts for recent Qubes OS version. -- Update startup script to mount appropriate directories as either - copy-on-write (device-mapper snapshot), or tmpfs. -- Optimize memory usage: should be possible to run sys-net, sys-firewall, and - at least two more VMs on 4GB machine. This include minimizing writes to - copy-on-write layer and tmpfs (disable logging etc). -- Research option to install the system from live image. If feasible add - this option. - -**Difficulty**: hard - -**Knowledge prerequisite**: - -- System startup sequence: bootloaders (isolinux, syslinux, grub, UEFI), initramfs, systemd. -- Python and Bash scripting -- Filesystems and block devices: loop devices, device-mapper, tmpfs, overlayfs, sparse files. - -**Size of the project**: 350 hours - -**Mentor**: [Frédéric Pierret](/team/) - - - -### LogVM(s) - -**Project**: LogVM(s) - -**Brief explanation**: Qubes AppVMs do not have persistent /var (on purpose). -It would be useful to send logs generated by various VMs to a dedicated -log-collecting VM. This way logs will not only survive VM shutdown, but also be -immune to altering past entries. See -[#830](https://github.com/QubesOS/qubes-issues/issues/830) for details. - -**Expected results**: - -- Design a _simple_ protocol for transferring logs. The less metadata (parsed - in log-collecting VM) the better. -- Implement log collecting service. Besides logs itself, should save - information about logs origin (VM name) and timestamp. The service should - _not_ trust sending VM in any of those. -- Implement log forwarder compatible with systemd-journald and rsyslog. A - mechanism (service/plugin) fetching logs in real time from those and sending - to log-collecting VM over qrexec service. -- Document the protocol. -- Write unit tests and integration tests. - -**Difficulty**: easy - -**Knowledge prerequisite**: - -- syslog -- systemd -- Python/Bash scripting - -**Size of the project**: 175 hours - -**Mentor**: [Frédéric Pierret](/team/) - - -### Whonix IPv6 and nftables support - -**Project**: Whonix IPv6 and nftables support - -**Brief explanation**: [T509](https://phabricator.whonix.org/T509) - -**Expected results**: - -- Work at upstream Tor: An older version of [TransparentProxy](https://trac.torproject.org/projects/tor/wiki/doc/TransparentProxy) page was the origin of Whonix. Update that page for nftables / IPv6 support without mentioning Whonix. Then discuss that on the tor-talk mailing list for wider input. [here](https://trac.torproject.org/projects/tor/ticket/21397) -- implement corridor feature request add IPv6 support / port to nftables - [issue](https://github.com/rustybird/corridor/issues/39) -- port [whonix-firewall](https://github.com/Whonix/whonix-firewall) to nftables -- make connections to IPv6 Tor relays work -- make connections to IPv6 destinations work - -**Difficulty**: medium - -**Knowledge prerequisite**: - -- nftables -- iptables -- IPv6 - -**Size of the project**: 175 hours - -**Mentor**: [Patrick Schleizer](/team/) - - -### GUI agent for Windows 8/10 -**Project**: GUI agent for Windows 8/10 - -**Brief explanation**: Add support for Windows 8+ to the Qubes GUI agent and video driver. Starting from Windows 8, Microsoft requires all video drivers to conform to the WDDM display driver model which is incompatible with the current Qubes video driver. Unfortunately the WDDM model is much more complex than the old XPDM one and officially *requires* a physical GPU device (which may be emulated). Some progress has been made to create a full WDDM driver that *doesn't* require a GPU device, but the driver isn't working correctly yet. Alternatively, WDDM model supports display-only drivers which are much simpler but don't have access to system video memory and rendering surfaces (a key feature that would simplify seamless GUI mode). [#1861](https://github.com/QubesOS/qubes-issues/issues/1861) - -**Expected results**: Working display-only WDDM video driver or significant progress towards making the full WDDM driver work correctly. - -**Difficulty**: hard - -**Knowledge prerequisite**: C/C++ languages, familiarity with Windows API, familiarity with the core Windows WDM driver model. Ideally familiarity with the WDDM display driver model. - -**Size of the project**: 175 hours - -**Mentor**: [Rafał Wojdyła](/team/) - -### GNOME support in dom0 / GUI VM - -**Project**: GNOME support in dom0 - -**Brief explanation**: Integrating GNOME into Qubes dom0. This include: - -- patching window manager to add colorful borders -- removing stuff not needed in dom0 (file manager(s), indexing services etc) -- adjusting menu for easy navigation (same applications in different VMs and such problems, dom0-related entries in one place) -- More info: [#1806](https://github.com/QubesOS/qubes-issues/issues/1806) - -**Expected results**: - -- Review existing support for other desktop environments (KDE, Xfce4, i3, awesome). -- Patch window manager to draw colorful borders (we use only server-side - decorations), there is already very similar patch in - [Cappsule project](https://github.com/cappsule/cappsule-gui). -- Configure GNOME to not make use of dom0 user home in visible way (no search - in files there, no file manager, etc). -- Configure GNOME to not look into external devices plugged in (no auto - mounting, device notifications etc). -- Package above modifications as RPMs, preferably as extra configuration files - and/or plugins than overwriting existing files. Exceptions to this rule may - apply if no other option. -- Adjust comps.xml (in installer-qubes-os repo) to define package group with - all required packages. -- Document installation procedure. - -**Difficulty**: hard - -**Knowledge prerequisite**: - -- GNOME architecture -- C language (patching metacity) -- Probably also javascript - for modifying GNOME shell extensions - -**Size of the project**: 175 hours - -**Mentor**: [Frédéric Pierret](/team/), [Marek Marczykowski-Górecki](/team/) - -### Generalize the Qubes PDF Converter to other types of files - -**Project**: Qubes Converters - -**Brief explanation**: One of the pioneering ideas of Qubes is to use disposable virtual machines to convert untrustworthy files (such as documents given to journalists by unknown and potentially malicious whistleblowers) into trustworthy files. See [Joanna's blog on the Qubes PDF Convert](https://theinvisiblethings.blogspot.co.uk/2013/02/converting-untrusted-pdfs-into-trusted.html) for details of the idea. Joanna has implemented a prototype for PDF documents. The goal of this project would be to generalize beyond the simple prototype to accommodate a wide variety of file formats, including Word documents, audio files, video files, spreadsheets, and so on. The converters should prioritise safety over faithful conversion. For example the Qubes PDF converter typically leads to lower quality PDFs (e.g. cut and paste is no longer possible), because this makes the conversion process safer. - -**Expected results**: We expect that in the timeframe, it will be possible to implement many converters for many file formats. However, if any unexpected difficulties arise, we would prioritise a small number of safe and high quality converters over a large number of unsafe or unuseful converters. - -**Difficulty**: easy - -**Knowledge prerequisite**: Most of the coding will probably be implemented as shell scripts to interface with pre-existing converters (such as ImageMagick in the Qubes PDF converter). However, shell scripts are not safe for processing untrusted data, so any extra processing will need to be implemented in another language -- probably Python. - -**Size of the project**: 175 hours - -**Mentors**: Andrew Clausen and Jean-Philippe Ouellet - -### Progress towards reproducible builds - -**Project**: Progress towards reproducible builds - -**Brief explanation**: A long-term goal is to be able to build the entire OS and installation media in a completely bit-wise deterministic manner, but there are many baby steps to be taken along that path. See: - -- "[Security challenges for the Qubes build process](/news/2016/05/30/build-security/)" -- [This mailing list post](https://groups.google.com/d/msg/qubes-devel/gq-wb9wTQV8/mdliS4P2BQAJ) -- and [reproducible-builds.org](https://reproducible-builds.org/) - -for more information and qubes-specific background. - -**Expected results**: Significant progress towards making the Qubes build process deterministic. This would likely involve cooperation with and hacking on several upstream build tools to eliminate sources of variability. - -**Difficulty**: medium - -**Knowledge prerequisite**: qubes-builder [[1]](/doc/qubes-builder/) [[2]](/doc/qubes-builder-details/) [[3]](https://github.com/QubesOS/qubes-builder/tree/master/doc), and efficient at introspecting complex systems: comfortable with tracing and debugging tools, ability to quickly identify and locate issues within a large codebase (upstream build tools), etc. - -**Size of the project**: 350 hours - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - -### Porting Qubes to ARM/aarch64 - -**Project**: Porting Qubes to ARM/aarch64 - -**Brief explanation**: - -Qubes currently only supports the x86_64 CPU architecture. Xen currently has additional support for ARM32/ARM64 processors, however work needs to be done to integrate this into the Qubes build process, as well as work in integrating this with the Qubes toolstack and security model. This may also be beneficial in simplifying the process of porting to other architectures. - -Some related discussion: - -- [#4318](https://github.com/QubesOS/qubes-issues/issues/4318) on porting to ppc64. -- [#3894](https://github.com/QubesOS/qubes-issues/issues/3894) on porting to L4 microkernel. - -**Expected results**: - -- Add cross-compilation support to qubes-builder and related components. -- Make aarch64 specific adjustments to Qubes toolstacks/manager (including passthrough of devices from device tree to guest domains). -- Aarch64 specific integration and unit tests. -- Production of generic u-boot or uefi capable image/iso for target hardware. - -**Difficulty**: hard - -**Knowledge prerequisite**: - -- Libvirt and Qubes toolstacks (C and python languages). -- Xen debugging. -- General ARM architecture knowledge. - -**Size of the project**: 350 hours - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - - - - -### Android development in Qubes - -**Project**: Research running Android in Qubes VM (probably HVM) and connecting it to Android Studio - -**Brief explanation**: The goal is to enable Android development (and testing!) -on Qubes OS. Currently it's only possible using qemu-emulated Android for ARM. -Since it's software emulation it's rather slow. -Details, reference: [#2233](https://github.com/QubesOS/qubes-issues/issues/2233) - -**Expected results**: - -- a simple way of setting up Android qubes with hardware emulation - (distributed as a template or as a salt, handling various modern Android versions) -- figuring out and implementing an easy and secure way to connect an Android - qube to a development qube with Android studio -- documentation and tests - -**Difficulty**: hard - -**Knowledge prerequisite**: - -**Size of the project**: 350 hours - -**Mentor**: Inquire on [qubes-devel](/support/#qubes-devel). - -### Admin API Fuzzer - -**Project**: Develop a [Fuzzer](https://en.wikipedia.org/wiki/Fuzzing) for the -[Qubes OS Admin API](/doc/admin-api/). - -**Brief explanation**: The [Qubes OS Admin API](/doc/admin-api/) -enables VMs to execute privileged actions on other VMs or dom0 - if allowed by the Qubes OS RPC policy. -Programming errors in the Admin API however may cause these access rights to be more permissive -than anticipated by the programmer. - -Since the Admin API is continuously growing and changing, continuous security assessments are required. -A [Fuzzer](https://en.wikipedia.org/wiki/Fuzzing) would help to automate part of these assessments. - -**Expected results**: - - fully automated & extensible Fuzzer for parts of the Admin API - - user & developer documentation - -**Difficulty**: medium - -**Prerequisites**: - - basic Python understanding - - some knowledge about fuzzing & existing fuzzing frameworks (e.g. [oss-fuzz](https://github.com/google/oss-fuzz/tree/master/projects/qubes-os)) - - a hacker's curiosity - -**Size of the project**: 175 hours - -**Mentor**: Inquire on [qubes-devel](/support/#qubes-devel). - - -### Secure Boot support - -**Project**: Add support for protecting boot binaries with Secure Boot technology, using user-generated keys. - -**Brief explanation**: Since recently, Xen supports "unified EFI boot" which allows to sign not only Xen binary itself, but also dom0 kernel and their parameters. While the base technology is there, enabling it is a painful and complex process. The goal of this project is to integrate configuration of this feature into Qubes, automating as much as possible. See discussion in [issue #4371](https://github.com/QubesOS/qubes-issues/issues/4371) - -**Expected results**: - - a tool to prepare relevant boot files for unified Xen EFI boot - this includes collecting Xen, dom0 kernel, initramfs, config file, and possibly few more (ucode update?); the tool should then sign the file with user provided key (preferably propose to generate it too) - - integrate it with updates mechanism, so new Xen or dom0 kernel will be picked up automatically - - include a fallback configuration that can be used for troubleshooting (main unified Xen EFI intentionally does not allow to manipulate parameters at boot time) - -**Difficulty**: hard - -**Knowledge prerequisite**: - - basic understanding of Secure Boot - - Bash and Python scripting - -**Size of the project**: 175 hours - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - - -### Reduce logging of Disposable VMs - -**Project**: Reduce logging of Disposable VMs - -**Brief explanation**: Partial metadata of a DisposableVM is stored in the dom0 filesystem. This applies to various logs, GUI status files etc. There should be an option to hide as much of that as possible - including bypassing some logging, and removing various state files, or at the very least obfuscating any hints what is running inside DisposableVM. More details at [issue #4972](https://github.com/QubesOS/qubes-issues/issues/4972) - -**Expected results**: A DisposableVM should not leave logs hinting what was running inside. - -**Difficulty**: medium - -**Knowledge prerequisite**: - - Python scripting - - Basic knowledge of Linux system services management (systemd, syslog etc) - -**Size of the project**: 350 hours - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - - -## Past Projects - -You can view the projects we had in 2017 in the [GSoC 2017 archive](https://summerofcode.withgoogle.com/archive/2017/organizations/5074771758809088/). We also participated in GSoC 2020 and GSoC 2021, and you can see the project in the [GSoC 2020 archive](https://summerofcode.withgoogle.com/archive/2020/organizations/4924517870206976/) and [GSoC 2021 archive](https://summerofcode.withgoogle.com/archive/2021/organizations/5682513023860736). - -Here are some successful projects which have been implemented in the past by Google Summer of Code participants. - -### Template manager, new template distribution mechanism - -**Project**: Template manager, new template distribution mechanism - -**Brief explanation**: Template VMs currently are distributed using RPM -packages. There are multiple problems with that, mostly related to static -nature of RPM package (what files belong to the package). This means such -Template VM cannot be renamed, migrated to another storage (like LVM), etc. -Also we don't want RPM to automatically update template package itself (which -would override all the user changes there). More details: -[#2064](https://github.com/QubesOS/qubes-issues/issues/2064), -[#2534](https://github.com/QubesOS/qubes-issues/issues/2534), -[#3573](https://github.com/QubesOS/qubes-issues/issues/3573). - -**Expected results**: - - - Design new mechanism for distributing templates (possibly including some - package format - either reuse something already existing, or design - new one). The mechanism needs to handle: - - integrity protection (digital signatures), not parsing any data in dom0 - prior to signature verification - - efficient handling of large sparse files - - ability to deploy the template into various storage mechanisms (sparse - files, LVM thin volumes etc). - - template metadata, templates repository - enable the user to browse - available templates (probably should be done in dedicated VM, or DisposableVM) - - manual template removal by users (without it, see problems such - as [#5509](https://github.com/QubesOS/qubes-issues/issues/5509) - - Implement the above mechanism: - - tool to download named template - should perform download operation in - some VM (as dom0 have no network access), then transfer the data to dom0, - verify its integrity and then create Template VM and feed it's root - filesystem image with downloaded data. - - tool to browse templates repository - both CLI and GUI (preferably integrated - with existing Template Manager tool) - - integrate both tools - user should be able to choose some template to be - installed from repository browsing tool - see - [#1705](https://github.com/QubesOS/qubes-issues/issues/1705) for some idea - (this one lacks integrity verification, but a similar service could - be developed with that added) - - If new "package" format is developed, add support for it into - [linux-template-builder](https://github.com/QubesOS/qubes-linux-template-builder). - - Document the mechanism. - - Write unit tests and integration tests. - -**Knowledge prerequisite**: - - - Large files (disk images) handling (sparse files, archive formats) - - Bash and Python scripting - - Data integrity handling - digital signatures (gpg2, gpgv2) - - PyGTK - - RPM package format, (yum) repository basics - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - ----- - -We adapted some of the language here about GSoC from the [KDE GSoC page](https://community.kde.org/GSoC). +--- \ No newline at end of file diff --git a/developer/general/gsod.md b/developer/general/gsod.md index fbb13eb5c..38563efd7 100644 --- a/developer/general/gsod.md +++ b/developer/general/gsod.md @@ -2,145 +2,7 @@ lang: en layout: doc permalink: /gsod/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/gsod.html ref: 242 title: Google Season of Docs (GSoD) ---- - -Thank you for your interest in participating in the [2021 Google Season of Docs](https://developers.google.com/season-of-docs/) program with the [Qubes OS team](/team/). You can read more about the Google Season of Docs in the official [guides](https://developers.google.com/season-of-docs/docs/) and [FAQ](https://developers.google.com/season-of-docs/docs/faq). - -## 2021 Project Idea - -### Qubes OS on Virtual Machines and Disposable VMs documentation - Qubes OS - -#### About your organization - -Qubes OS is a free and open source operating system uniquely designed to protect the security and privacy of the user. Its architecture is built to enable a user to define different security environments ("qubes") on their computer and visually manage their interaction with each other and the world. - -Qubes OS was launched in 2011 and has [received praise from security experts](https://www.qubes-os.org/endorsements/) like Edward Snowden, Bill Buddington (EFF), Isis Lovecruft (Tor Project), and Kenn White (Open Crypto Audit), and has over [30,000 active users](https://www.qubes-os.org/statistics/). - -From network-level to software-level protections, as well as protections against firmware and hardware attacks, Qubes OS is trying to protect the user from the most significant attacks they encounter so that they can get their work done, safely. - -#### The Project - -The goal of the project is to improve the documentation related to Disposable VMs and installing Qubes OS on Virtual Machines. - -The current Disposable VMs documentation is scarce, inconsistent in places and is scattered across multiple pages, sometimes hard to find. This project involves consolidating it into one or few easy to find pages, covering all related subjects. It should be written in way easy to follow and understand, clearly separating basic use cases, advanced ones and internal details. The terminology should also be used consistently. - -Additionally, Qubes OS is missing an installation guide for virtual machines. Users are installing an outdated and unsupported version of Qubes OS (3.2) instead of the supported version. There is unofficial existing installation guide for Qubes OS on a virtual box but it is misleading and lacks documentation. Usually, users face some errors and bugs while installing Qubes OS on a virtual machine. This project also involves writing an installation guide that details how to install Qubes on virtual machines, and how to troubleshoot any issues that may arise. - -#### Project’s scope - -The technical writer will: - -* Review existing Disposable VM documentation - -* Propose new documentation layout, including split between pages - -* Propose updated and clarified content - -* Provide a new option of installation guide for users working on virtual machines. - -* Review existing problems and provide solutions to them. - -* Giving a warning for using outdated versions of Qubes. - -Knowledge prerequisites: - -* Basic Qubes OS knowledge - intro, getting started. - -* Markdown. - -* Experience in virtual boxes and machines. - -* Basic knowledge about Fedora Linux architecture. - -The project is estimated to need around 6 months, with a part-time (20hr/week) workload. Qubes OS team members focused on documentation (Michael Carbone, Andrew David Wong, Marek Marczykowski-Górecki) have committed to supporting the project. - -#### Measuring the project’s success - -We will consider the project successful if, after publication of the new documentation: - -* The Disposable VM documentation is consilidated into a few pages and easier to find. - -* Disposable VM terminology is consistent across all pages - -* There is a new installation guide detailing how to install Qubes on virtual machines (VirtualBox), complete with screenshots and troubleshooting steps where necessary. - -* There is a decrease (by ~20%) in the number of issues and questions raised for topics covered in the documentation, whether in the official Github issues tracker, the Qubes mailing list, or the Qubes community forum. - -#### Budget - -| Budget item | Amount | Running total | Note / Justification | -|-|-|-|-| -| Technical writer audit, update, and write new documentation | $12000 | $12000 | based on a 20hr/week for 6 months at 25$/hr | -| TOTAL | | $12000 | | - -#### Additional information -Qubes OS regularly participates in the Google Summer of Code and Google Season of Docs. This is our third time in a row participating in Google Season of Docs. Our mentorships for GSoD 2019 and 2020 were successes and both projects were completed -within the times allotted. The past Google Season of Docs projects have given us experience in working with technical writers, and has helped us to understand the benefits of technical writers can to our project - -## Past Projects - -You can view the project we had in 2019 in the [2019 GSoD archive](https://developers.google.com/season-of-docs/docs/2019/participants/project-qubes) and the [2019 writer's report](https://refre.ch/report-qubesos/). - -You can also view the project we had in 2020 in the [2020 GSoD archive](https://developers.google.com/season-of-docs/docs/2020/participants/project-qubesos-c1e0) and the [2020 writer's report](https://gist.github.com/PROTechThor/bfe9b8b28295d88c438b6f6c754ae733). - -Here are some successful projects which have been implemented in the past by Google Season of Docs participants. - -### Consolidate troubleshooting guides - -**Project**: Consolidate troubleshooting guides - -**Brief explanation**: Troubleshooting guides are scattered across many pages and sometimes incomplete, leading to repeatedly posting the same instruction over and over when helping users to diagnose problems. -This could be helped by writing a consolidated guide with a clear list of symptom-action layout. - -**Expected results**: - -- Review existing [troubleshooting guides](/doc/#troubleshooting) -- Review [issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+label%3A%22C%3A+doc%22) containing common troubleshooting steps (checking specific logs etc) -- Propose updated, consolidated troubleshooting documentation, including its layout - -**Knowledge prerequisite**: - -- [Markdown](https://daringfireball.net/projects/markdown/) - -**Mentor**: [Marek Marczykowski-Górecki](/team/) - -### Improve Getting Started page - -**Project**: Improve Getting Started page - -**Brief explanation**: The [Getting Started page](https://www.qubes-os.org/doc/getting-started/) is the place a new user would go to understand better how to use Qubes. It is currently has old screenshots not using the default desktop environment and could have much better flow. In addition, this improved page content may end up being served more directly to the user via the [offline documentation](https://github.com/QubesOS/qubes-issues/issues/1019) or the firstboot guide. - -**Expected results**: - - - Review the existing page and website, similar pages for other OSes - - Provide visual mock-ups and proposed text - -**Knowledge prerequisite**: - -- basic Qubes OS knowledge -- [Markdown](https://daringfireball.net/projects/markdown/) - -**Mentor**: [Michael Carbone](/team/) - -### Rewrite qrexec documentation - -**Project**: Rewrite qrexec documentation - -**Brief explanation**: Current qrexec (qubes remote exec) documentation is hard to follow, important informations are hidden within a wall of text. -Some parts are split into multiple sections, for example version specific to avoid duplication, but it doesn't help reading it. -Additionally, protocol documentation describes only few specific use cases, instead of being clear and precise protocol specification. -Fixing this last point may require very close cooperation with developers, as the current documentation doesn't multiple corner cases (that's one of the issue with its current shape). - -**Expected results**: - -- Review existing [qrexec documentation](https://www.qubes-os.org/doc/qrexec3/) and an [issue about it](https://github.com/QubesOS/qubes-issues/issues/1392) -- Propose updated, consolidated admin documentation (policy writing, adding services) -- Propose consolidated protocol specification, based on the current documentation, and cooperation with developers - -**Knowledge prerequisite**: - -- [Markdown](https://daringfireball.net/projects/markdown/) - -**Mentor**: [Marek Marczykowski-Górecki](/team/) +--- \ No newline at end of file diff --git a/developer/general/how-to-edit-the-documentation.md b/developer/general/how-to-edit-the-documentation.md index f51fe5128..661d14169 100644 --- a/developer/general/how-to-edit-the-documentation.md +++ b/developer/general/how-to-edit-the-documentation.md @@ -2,197 +2,6 @@ lang: en layout: doc permalink: /doc/how-to-edit-the-documentation/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/how-to-edit-the-documentation.html title: How to edit the documentation ---- - -_Also see the [documentation style guide](/doc/documentation-style-guide/)._ - -Qubes OS documentation pages are stored as plain text Markdown files in the -[qubes-doc](https://github.com/QubesOS/qubes-doc) repository. By cloning and -regularly pulling from this repo, users can maintain their own up-to-date -offline copy of all Qubes documentation rather than relying solely on the web. - -The documentation is a volunteer community effort. People like you are -constantly working to make it better. If you notice something that can be fixed -or improved, please follow the steps below to open a pull request! - -## How to submit a pull request - -We keep all the documentation in -[qubes-doc](https://github.com/QubesOS/qubes-doc), a dedicated Git repository -hosted on [GitHub](https://github.com/). Thanks to GitHub's easy web interface, -you can edit the documentation even if you're not familiar with Git or the -command line! All you need is a free GitHub account. - -A few notes before we get started: - -- Since Qubes is a security-oriented project, every documentation change will - be [reviewed](#security) before it's accepted. This allows us to maintain - quality control and protect our users. - -- To give your contribution a better chance of being accepted, please follow - our [documentation style guide](/doc/documentation-style-guide/). - -- We don't want you to spend time and effort on a contribution that we can't - accept. If your contribution would take a lot of time, please [file an - issue](/doc/issue-tracking/) for it first so that we can make sure we're on - the same page before significant works begins. - -- Alternatively, you may already have written content that doesn't conform to - these guidelines, but you'd be willing to modify it so that it does. In this - case, you can still submit it by following the instructions below. Just make - a note in your pull request (PR) that you're aware of the changes that need - to be made and that you're just asking for the content to be reviewed before - you spend time making those changes. - -- Finally, if you've written something that doesn't belong in qubes-doc but - that would be beneficial to the Qubes community, please consider adding it to - the [external - documentation](/doc/documentation-style-guide/#core-vs-external-documentation). - -(**Advanced users:** If you're already familiar with GitHub or wish to work -from the command line, you can skip the rest of this section. All you need to -do to contribute is to [fork and -clone](https://guides.github.com/activities/forking/) the -[qubes-doc](https://github.com/QubesOS/qubes-doc) repo, make your changes, then -[submit a pull -request](https://help.github.com/articles/using-pull-requests/).) - -Ok, let's begin. Every documentation page has a "Page Source on GitHub" button. -Depending on the size of your screen, it may either be on the side (larger -screens) or on the bottom (smaller screens). - -[![page-source-button](/attachment/doc/doc-pr_01_page-source-button.png)](/attachment/doc/doc-pr_01_page-source-button.png) - -When you click on it, you'll be taken to the source file --- usually a Markdown -(`.md`) file --- on GitHub. On this page, there will be a button to edit the -file. - -[![github-edit](/attachment/doc/doc-pr_02_github-edit.png)](/attachment/doc/doc-pr_02_github-edit.png) - -You'll be prompted to sign in with your GitHub username and password -(if you aren't already logged in). You can also create a free account from here. - -[![github-sign-in](/attachment/doc/doc-pr_03_sign-in.png)](/attachment/doc/doc-pr_03_sign-in.png) - -If this is your first contribution to the documentation, you need to "fork" the -repository (make your own copy). It's easy --- just click the big green button -on the next page. This step is only needed the first time you make a -contribution. - -[![fork](/attachment/doc/doc-pr_04_fork.png)](/attachment/doc/doc-pr_04_fork.png) - -Now you can make your modifications. You can also preview the changes to see -how they'll be formatted by clicking the "Preview changes" tab. If you want to -add images, please see [How to add images](#how-to-add-images). If you're -making formatting changes, please [render the site -locally](https://github.com/QubesOS/qubesos.github.io#instructions) to verify -that everything looks correct before submitting any changes. - -[![edit](/attachment/doc/doc-pr_05_edit.png)](/attachment/doc/doc-pr_05_edit.png) - -Once you're finished, describe your changes at the bottom and click "Propose -file change". - -[![commit](/attachment/doc/doc-pr_06_commit-msg.png)](/attachment/doc/doc-pr_06_commit-msg.png) - -After that, you'll see exactly what modifications you've made. At this stage, -those changes are still in your own copy of the documentation ("fork"). If -everything looks good, send those changes to us by pressing the "Create pull -request" button. - -[![pull-request](/attachment/doc/doc-pr_07_review-changes.png)](/attachment/doc/doc-pr_07_review-changes.png) - -You will be able to adjust the pull request message and title there. In most -cases, the defaults are ok, so you can just confirm by pressing the "Create -pull request" button again. However, if you're not ready for your PR to be -reviewed or merged yet, please [make a draft PR -instead](https://github.blog/2019-02-14-introducing-draft-pull-requests/). - -[![pull-request-confirm](/attachment/doc/doc-pr_08_create-pull-request.png)](/attachment/doc/doc-pr_08_create-pull-request.png) - -If any of your changes should be reflected in the [documentation index (a.k.a. -table of contents)](/doc/) --- for example, if you're adding a new page, -changing the title of an existing page, or removing a page --- please see [How -to edit the documentation index](#how-to-edit-the-documentation-index). - -That's all! We will review your changes. If everything looks good, we'll pull -them into the official documentation. Otherwise, we may have some questions for -you, which we'll post in a comment on your pull request. (GitHub will -automatically notify you if we do.) If, for some reason, we can't accept your -pull request, we'll post a comment explaining why we can't. - -[![done](/attachment/doc/doc-pr_09_done.png)](/attachment/doc/doc-pr_09_done.png) - -## How to edit the documentation index - -The source file for the [documentation index (a.k.a. table of contents)](/doc/) -is -[doc-index.yml](https://github.com/QubesOS/qubesos.github.io/blob/master/_data/doc-index.yml). - -Editing this file will change what appears on the documentation index. If your -pull request (PR) adds, removes, or edits anything that should be reflected in -the documentation index, please make sure you also submit an associated pull -request against this file. - -## How to add images - -To add an image to a page, use the following syntax in the main document (see -[here](/doc/documentation-style-guide/#image-linking) for why this syntax is -important). - -``` -[![Image Title](/attachment/doc/image.png)](/attachment/doc/image.png) -``` - -Then, submit your image(s) in a separate pull request to the -[qubes-attachment](https://github.com/QubesOS/qubes-attachment) repository -using the same path and filename. This is the only permitted way to include -images. Do not link to images on other websites. - -## Serving the website locally - -You can serve the website offline on your local machine by following [these -instructions](https://github.com/QubesOS/qubesos.github.io#instructions). This -can be useful for making sure that your changes render the way you expect, -especially when your changes affect formatting, images, tables, styling, etc. - -## Security - -*Also see: [Should I trust this website?](/faq/#should-i-trust-this-website)* - -All pull requests (PRs) against -[qubes-doc](https://github.com/QubesOS/qubes-doc) must pass review prior to be -merged, except in the case of [external -documentation](/doc/#external-documentation) (see -[#4693](https://github.com/QubesOS/qubes-issues/issues/4693)). This process is -designed to ensure that contributed text is accurate and non-malicious. This -process is a best effort that should provide a reasonable degree of assurance, -but it is not foolproof. For example, all text characters are checked for ANSI -escape sequences. However, binaries, such as images, are simply checked to -ensure they appear or function the way they should when the website is -rendered. They are not further analyzed in an attempt to determine whether they -are malicious. - -Once a pull request passes review, the reviewer should add a signed comment -stating, "Passed review as of ``" (or similar). The -documentation maintainer then verifies that the pull request is mechanically -sound (no merge conflicts, broken links, ANSI escapes, etc.). If so, the -documentation maintainer then merges the pull request, adds a PGP-signed tag to -the latest commit (usually the merge commit), then pushes to the remote. In -cases in which another reviewer is not required, the documentation maintainer -may review the pull request (in which case no signed comment is necessary, -since it would be redundant with the signed tag). - -## Questions, problems, and improvements - -If you have a question about something you read in the documentation or about -how to edit the documentation, please post it on the -[forum](https://forum.qubes-os.org/) or send it to the appropriate [mailing -list](/support/). If you see that something in the documentation should be -fixed or improved, please [contribute](#how-to-submit-a-pull-request) the -change yourself. To report an issue with the documentation, please follow our -standard [issue reporting guidelines](/doc/issue-tracking/). (If you report an -issue with the documentation, you will likely be asked to submit a pull request -for it, unless there is a clear indication in your report that you are not -willing or able to do so.) +--- \ No newline at end of file diff --git a/developer/general/package-contributions.md b/developer/general/package-contributions.md index f73c9bd25..be47c56e9 100644 --- a/developer/general/package-contributions.md +++ b/developer/general/package-contributions.md @@ -2,101 +2,7 @@ lang: en layout: doc permalink: /doc/package-contributions/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/package-contributions.html ref: 29 title: Package contributions ---- - -_This page is for developers who wish to contribute packages. -If you want to install contributed packages, please see [installing contributed packages](/doc/installing-contributed-packages/)._ - -We're very grateful to the talented and hard-working community members who contribute software packages to Qubes OS. -This page explains the inclusion criteria and procedures for such packages, as well as the roles and responsibilities of those involved. - -Inclusion Criteria ------------------- - -In order to be accepted, packages must: - -* In no way weaken the security of Qubes OS. -* Be published under an open-source license (read about the [Qubes OS License](/doc/license/)). -* Follow our [coding guidelines](/doc/coding-style/). -* Be thoroughly tested. -* Have a clearly-defined use case for Qubes users. -* Not be unduly burdensome to review. - -(Please note that we always reserve the right to add criteria to this list.) - -Contribution Procedure ----------------------- - -Before you start putting serious work into a package, we recommend that you discuss your idea with the Qubes developers and the broader community on the [qubes-devel mailing list](/support/#qubes-devel). -Once you have a package that's ready to become part of Qubes OS, please follow this procedure: - -1. Ensure that your package satisfies the [Inclusion Criteria](#inclusion-criteria). -2. If your code isn't already on GitHub, create a GitHub repo that contains your code. You can have a look to an example package called [qubes-skeleton](https://github.com/QubesOS-contrib/qubes-skeleton). -3. If you haven't already, [sign your code](/doc/code-signing/). -4. Create an issue in [qubes-issues](https://github.com/QubesOS/qubes-issues/issues/) with the title `[Contribution] your-package-name`. - Include a link to your repo, a brief description of your package, and a brief explanation of why you think it should be included in Qubes. - Please note that the Qubes core developers are very busy. - If they are under heavy load when you submit your contribution, it may be a very long time before they have time to review your package. - If this happens, please do not be discouraged. - If you think they may have forgotten about your pending contribution, you may "bump" your request by commenting on your issue, but please do this *very* sparingly (i.e., no more than once a month). - We appreciate your understanding! -5. You may be asked followup questions. - If we decide to accept your contribution, you will be invited to join the [QubesOS-contrib](https://github.com/QubesOS-contrib) organization on GitHub as public recognition of your contribution (but without push access; see [Review Procedure](#review-procedure)), and [QubesOS-contrib](https://github.com/QubesOS-contrib) will fork your repo. - If we decide not to accept your contribution, we will state the reason and close the issue. - -Update Procedure ----------------- - -*Anyone* can provide an update (patch) to a contributed package, not just the person who contributed that package! -The update procedure is the same for everyone, including the original package contributor. - -If you would like to update an already-contributed package (specifically, a fork owned by [QubesOS-contrib](https://github.com/QubesOS-contrib)), please submit a [signed](/doc/code-signing/), fast-forwardable pull request to that repo with your changes. -Please note that your pull request **must** be both [signed](/doc/code-signing/) and fast-forwardable, or else it will be closed without further review. -One or more reviewers may post comments on your pull request. -Please be prepared to read and respond to these comments. - -Review Procedure ----------------- - -This review procedure covers both original package contributions (see [Contribution Procedure](#contribution-procedure)) and all subsequent updates to those packages, including updates from the original package contributor (see [Update Procedure](#update-procedure)). -All changes will be reviewed by a Qubes Core Reviewer (QCR) and the [Package Maintainer](#package-maintainers) (PM). -In all cases, the QCR will be a core Qubes developer. -In some cases, the QCR and the PM will be the same person. -For example, if someone contributes a package, then disappears, and no suitable replacement has been found, then it is likely that a core Qubes developer will play both the QCR and PM roles for that package, at least until another suitable candidate volunteers to become the PM for that package. - -The review procedure is as follows: - -1. Someone, S, wishes to make a change to a package, P. -2. S submits a fast-forwardable pull request against the fork of P's repo owned by [QubesOS-contrib](https://github.com/QubesOS-contrib). -3. The PM reviews the pull request. - If the the pull request passes the PM's review, the PM adds a [signed](/doc/code-signing/) *comment* on the pull request stating that it has passed review. - (In cases in which S = PM, the PM can simply add a [signed](/doc/code-signing/) *tag* to the HEAD commit prior to submitting the pull request.) - If the pull request does not pass the PM's review, the PM leaves a comment on the pull request explaining why not. -4. The QCR reviews the pull request. - If the pull request passes the QCR's review, the QCR pushes a [signed](/doc/code-signing/) tag to the HEAD commit stating that it has passed review and fast-forward merges the pull request. - If the pull request does not pass the QCR's review, the QCR leaves a comment on the pull request explaining why not, and the QCR may decide to close the pull request. - -In all the cases, the first condition to be validated by the QCR's review is to ensure that the contribution **will not** hijack any core packages of [QubesOS](https://github.com/QubesOS) and of course, none of the [QubesOS-contrib](https://github.com/QubesOS-contrib) packages too. More precisely, particular attention to the whole build pipeline will be made with a specific review of: - -* Package dependencies, -* Build scripts (including downloaded ones), -* All downloaded components should be verified against static hash, -* RPM/DEB installation scripts (e.g. looking at constraints who would hijack other packages), -* Makefiles, -* Package build [reproducible](https://reproducible-builds.org/) - -and any steps which would result in partial/total compromise of legitimate components. For this part, you can have a look to an example package called [qubes-skeleton](https://github.com/QubesOS-contrib/qubes-skeleton). - -Package Maintainers -------------------- - -If you contribute a package, we assume that you will be the maintainer of that package, unless you tell us otherwise. -As the maintainer of the package, it is your privilege and responsibility to: - -* [Review](#review-procedure) each pull request made against the package. -* Decide when the package has reached a new version, and notify the Qubes core developers when this occurs. - -If you do not wish to be the maintainer of your package, please let us know. -If you do not act on your maintainer duties for a given package for an extended period of time and after at least one reminder, we will assume that you no longer wish to be the maintainer for that package. +--- \ No newline at end of file diff --git a/developer/general/research.md b/developer/general/research.md index 74ac3c188..92ce86bdf 100644 --- a/developer/general/research.md +++ b/developer/general/research.md @@ -7,11 +7,7 @@ redirect_from: - /en/doc/qubes-research/ - /doc/QubesResearch/ - /wiki/QubesResearch/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/research.html ref: 139 title: Research ---- - -Here are links to various research papers, projects, videos, and blog posts -related to Qubes OS. - -{% include research.html %} +--- \ No newline at end of file diff --git a/developer/general/usability-ux.md b/developer/general/usability-ux.md index 5e0c53592..f54936909 100644 --- a/developer/general/usability-ux.md +++ b/developer/general/usability-ux.md @@ -2,229 +2,7 @@ lang: en layout: doc permalink: /doc/usability-ux/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/usability-ux.html ref: 31 title: Usability & UX ---- - -Software that is too complicated to use, is often unused. Because we want as many people as possible to benefit from its unique security properties, the usability and user experience of Qubes OS is an utmost priority! - -We ask anyone developing for Qubes OS to please read through this guide to better understand the user experience we strive to achieve. We also ask them to review [our visual style guide](/doc/visual-style-guide/) for other design related information. - ---- - -## Easy To Use - -An ideal user experience is friendly, and it beckons a new user to explore the interface. In this process, they can naturally discover how to use the software. Below are some guidelines that will help you design a user interface that accomplishes this goal. - -
- Interfaces Should Not -
- -- Require extensive configuration before a user can *begin* doing things -- Make it possible to break provided features or actions in unrecoverable ways -- Perform actions which compromise security and data -- Overwhelm the user with too much information and cognitive load - -Perhaps the most common cause of mistakes is complexity. If there is a configuration setting that will significantly affect the user's experience, choose a safe and smart default then tuck this setting in an `Advanced Settings` panel. - -
- Interfaces Should -
- -- Make it easy to discover features and available actions -- Provide some understanding of what discovered features do -- Offer the ability to easily undo mistakes -- Choose intelligent defaults for settings - -In making software easy to use, it is crucial to be mindful of [cognitive load](https://en.wikipedia.org/wiki/Cognitive_load) which dictates that *"humans are generally able to hold only seven +/- two units of information in short-term memory."* Making sure your interfaces don't pass this short-term memory limit is perhaps the most important factor in helping a user feel comfortable instead of overwhelmed. - ---- - -## Easy to Understand - -There will always be the need to communicate things to users. In these cases, an interface should aim to make this information easy to understand. The following are simple guides to help achieve this - none of these are absolute maxims! - -
- Avoid Acronyms -
- -Acronyms are compact and make good names for command line tools. They do not make graphical user interfaces more intuitive for non-technical users. Until one learns an acronym's meaning, it is gibberish. Avoid acronyms in your interfaces whenever possible! - -- `DVM` - Disposable Virtual Machine -- `GUID` - Global Unique Identifier -- `PID` - Process Identification Number -- `NetVM` - Networking Virtual Machine - -Despite this rule, some acronyms like `USB` are widely used and understood due to being in common use for over a decade. It is good to use these acronyms when the full words like `Universal Serial Bus` are more likely to confuse users. - -
- Use Simple Words -
- -Use the minimum amount of words needed to be descriptive, but also informative. Go with common words that are as widely understood. Sometimes, inventing a word such as `Qube` to describe a `virtual machine` makes the life of the user much easier. - -- Use `Disposable Qube` instead of `DVM` or `Disposable Virtual Machine` -- Use `interface` instead of `GUI` or `Graphical User Interface` -- Use `application number` instead of `PID` or `Process Identification Number` -- Use `Networking` or `Networking Qube` instead of `NetVM` given context - ---- - -
- Avoid Technical Words -
- -Technical words are usually more accurate, but they often *only* make sense to technical users and are confusing and unhelpful to non-technical users. Examples of technical words that might show up in Qubes OS are: - -- `root.img` -- `savefile` -- `qrexec-daemon` - -These are all terms that have at some point showed up in users' notification messages. Each term is very specific, but requires the user to understand virtualization to interpret. - -
- Use Common Concepts -
- -Large amounts of the global population have been using computers for one or two decades and have formed some mental models of how things work. Leveraging these mental models are a huge gain. - -- Use `disk space` instead of `root.img`, since while not quite accurate, it makes contextual sense -- Use `saving` instead of `savefile` as the former is the action trying to be completed -- Use `Qubes` instead of `qrexec-daemon` as it gives better context on what is happening - -These words are more abstract and user relevant- they help a user understand what is happening based on already known concepts (disk space) or start to form a mental model of something new (Qubes). - ---- - -
- Avoid Inconsistencies -
- -It is easy to start abbreviating (or making acronyms) of long terms like `Disposable Virtual Machine` depending on where the term shows up in an interface. - -- `DVM` -- `DispVM` -- `DisposableVM` - -This variation in terms can cause new users to question or second guess what the three different variations mean, which can lead to inaction or mistakes. - -
- Make Things Consistent -
- -Always strive to keep things consistent in the interfaces as well as documentation and other materials. - -- Use `Disposable Qube` at all times as it meets other criteria as well. - -By using the same term throughout an interface, a user can create a mental model and relationship with that term allowing them to feel empowered. - ---- - -
- Avoid Duplicate Words -
- -It is easy to add words like `Domain` before items in a list or menu in an attempt to be descriptive, such as: - -~~~ -Menu -- Domain: work -- Domain: banking -- Domain: personal -~~~ - -The repeated use of the word `Domain` requires a user to read it for each item in the list, which makes extra work for the eye in parsing out the relevant word like `work, banking, or personal`. This also affects horizontal space on fixed width lines. - -
- Create Groups & Categories -
- -It is more efficient to group things under headings instead as this allows the eye to easily scan the uniqueness of the items. (As per our previous example:) - -~~~ -Domains -- Work -- Banking -- Personal -~~~ - ---- - -## Easy To Complete - -Lastly, expected (and unexpected) situations often require user actions or input. Make resolving these occurences as easy as possible to complete the action. - -
- Don't Leave Users Stranded -
- -Consider the following notifications: - -- `The disk space of your Qube "Work" is full` -- `There was an error saving Qube "Personal"` - -Instead of displaying solvable errors like these and neglecting to provide a fix: - -
- Offer Actionable Solutions -
- -Error messages and limits such as those in the previous example can be greatly improved by adding buttons or links to helpful information. - -- Add a button to `Increase Disk Space` -- Add a link to a documentation page called `Troubleshoot saving data` - -In adhering to these principles, you'll make undesirable situations more manageable for users instead of feeling stranded. - ---- - -
- Minimize Repetitive Steps -
- -There are many cases where a user wants to perform an action on more than one file or folder. However in order to do the action, the user must repeat certain steps such as: - -1. Click on `Open File` from a menu or button -2. Navigate through file system - - Click Folder One - - Click Folder Two - - Click Folder Three - - Click Folder Four -3. Select proper file -4. Complete task on file - -That subtle act of clicking through a file system can prove to be significant if a user needs to open more than a couple files in the same directory. We can alleviate some of the work by changing the process: - -1. Click on `Open File` from a menu or button -2. Remember last open folder/file system -3. Select proper file -4. Complete task - -Clearly, cutting out something as simple as navigating through the file system can save a user quite a bit of time. Alternatively, adding a button or menu item like `Open Multiple Files` might be even better, because remembering and using relevant hotkeys is often something only power users know how to do! - ---- - -## GNOME, KDE, and Xfce - -The desktop GUIs that QubesOS versions 1 - 3.1 offer are [KDE](https://www.kde.org) and [Xfce](https://xfce.org). We are currently migrating towards using [GNOME](https://www.gnome.org). We know some people prefer KDE, but we believe Gnome is easier to use for average non-technical users. Xfce will always be supported, and technical users will always have the choice to use KDE or other desktop environments. - -This change means you should use [GTK](https://www.gtk.org/) rather than Qt for new GUIs. - -All three of these mentioned desktop environments have their own [human interface guidelines](https://en.wikipedia.org/wiki/Human_interface_guidelines), and we suggest you familiarize yourself with the platform you developing for. - -- [GNOME Human Interface Guidelines](https://developer.gnome.org/hig/3.18/) -- [KDE HIG](https://techbase.kde.org/Projects/Usability/HIG) -- [Xfce UI Guidlines](https://wiki.xfce.org/dev/hig/general) - ---- - -## Further Learning & Inspiration - -Learning to make well designing intuitive interfaces and software is specialized skillset that can take years to cultivate, but if you are interested in furthering your understanding, we suggest the following resources: - -- [Learn Design Principles](http://learndesignprinciples.com) by Melissa Mandelbaum -- [Usability in Free Software](http://jancborchardt.net/usability-in-free-software) by Jan C. Borchardt -- [Superheroes & Villains in Design](https://vimeo.com/70030549) by Aral Balkan -- [First Rule of Usability? Don’t Listen to Users](http://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/) by Jakob Nielsen -- [10 Usability Heuristics for User Interface Design](https://www.nngroup.com/articles/ten-usability-heuristics/) by Jakob Nielsen -- [Hack Design](https://hackdesign.org/) - online learning program +--- \ No newline at end of file diff --git a/developer/general/visual-style-guide.md b/developer/general/visual-style-guide.md index ceef871df..a610ed03e 100644 --- a/developer/general/visual-style-guide.md +++ b/developer/general/visual-style-guide.md @@ -4,6 +4,7 @@ layout: doc permalink: /doc/visual-style-guide/ redirect_from: - /doc/style-guide/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/visual-style-guide.html ref: 27 title: Visual style guide ---- +--- \ No newline at end of file diff --git a/developer/general/website-style-guide.md b/developer/general/website-style-guide.md index 6c5a7de82..a224450d6 100644 --- a/developer/general/website-style-guide.md +++ b/developer/general/website-style-guide.md @@ -2,77 +2,6 @@ lang: en layout: doc permalink: /doc/website-style-guide/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/general/website-style-guide.html title: Website style guide ---- - -This page explains the standards we follow for building and maintaining the -website. Please follow these guidelines and conventions when modifying the -website. For the standards governing the documentation in particular, please -see the [documentation style guide](/doc/documentation-style-guide/). - -## Coding conventions - -The following conventions apply to the website as a whole, including everything -written in HTML, CSS, YAML, and Liquid. These conventions are intended to keep -the codebase consistent when multiple collaborators are working on it. They -should be understood as a practical set of rules for maintaining order in this -specific codebase rather than as a statement of what is objectively right or -good. - -### General practices - -- Use comments to indicate the purposes of different blocks of code. This makes - the file easier to understand and navigate. - -- Use descriptive variable names. Never use one or two letter variable names. - Avoid obscure abbreviations and made-up words. - -- In general, make it easy for others to read your code. Your future self will - thank you, and so will your collaborators! - -- [Don't Repeat Yourself - (DRY)!](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) Instead of - repeating the same block of code multiple times, abstract it out into a - separate file and `include` that file where you need it. - -### Whitespace - -- Always use spaces. Never use tabs. - -- Each indentation step should be exactly two (2) spaces. - -- Whenever you add an opening tag, indent the following line. (Exception: If - you open and close the tag on the same line, do not indent the following - line.) - -- Indent Liquid the same way as HTML. - -- In general, the starting columns of every adjacent pair of lines should be no - more than two spaces apart (example below). - -- No blank or empty lines. (Hint: When you feel you need one, add a comment on - that line instead.) - -#### Indentation example - -Here's an example that follows the indentation rules: - -{% raw %} -```html - - - - {% for item in secs.htmlsections[0].columns %} - - {% endfor %} - - {% for canary in site.data.sec-canary reversed %} - - - - - - {% endfor %} -
{{ item.title }}
{{ canary.date }}Qubes Canary #{{ canary.canary }}
-``` -{% endraw %} +--- \ No newline at end of file diff --git a/developer/releases/1_0/release-notes.md b/developer/releases/1_0/release-notes.md index 39c6ea37d..35e946833 100644 --- a/developer/releases/1_0/release-notes.md +++ b/developer/releases/1_0/release-notes.md @@ -4,47 +4,7 @@ layout: doc permalink: /doc/releases/1.0/release-notes/ redirect_from: - /en/doc/releases/1.0/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/1_0/release-notes.html ref: 18 title: Qubes R1.0 release notes ---- - -Detailed release notes in [this blog post](https://blog.invisiblethings.org/2012/09/03/introducing-qubes-10.html). - -## Known issues - -- Installer might not support some USB keyboards (\#230). This seems to include all the Mac Book keyboards (most PC laptops have PS2 keyboards and are not affected). - -- If you don't enable Composition (System Setting -\> Desktop -\> Enable desktop effects), which you really should do, then the KDE task bar might get ugly (e.g. half of it might be black). This is some KDE bug that we don't plan to fix. - -- Some keyboard layout set by KDE System Settings can cause [keyboard not working at all](https://groups.google.com/group/qubes-devel/browse_thread/thread/77d076b65dda7226). If you hit this issue, you can switch to console (by console login option) and manually edit `/etc/X11/xorg.conf.d/00-system-setup-keyboard.conf` (and `/etc/sysconfig/keyboard`) and place correct keyboard layout settings (details in linked thread). You can check if specific keyboard layout settings are proper using `setxkbmap` tool. - -- On systems with more than 8GB of RAM there is problem with DisposableVM. To fix it, limit maximum memory allocation for DispVM to 3GB - - ~~~ - qvm-prefs -s fedora-17-x64-dvm maxmem 3072 - qvm-create-default-dvm --default-template --default-script - ~~~ - -- On some systems the KDE Window Manager might freeze upon resuming from S3 sleep when compositing is enabled (and the only method to log in to the system if this happens is to switch to a text console, enter your user's password, kill the kwin process, go back to the Xorg console, log in, and start a new instance of kwin using Konsole application :) If you experience such problems, make sure to disable compositing before putting the system into sleep by pressing Alt-Ctrl-F12 (and then enabling it back once you log in after resume) -- this way you should never see this problem again. - -## Downloads - -See [Qubes Downloads](/doc/QubesDownloads/). - -## Installation instructions - -See [Installation Guide](/doc/installation-guide/). - -## Upgrading - -### From Qubes 1.0-rc1 - -If you're already running Qubes 1.0-rc1, you don't need to reinstall, it's just enough to update the packages in your Dom0 and the template VM(s). The easiest way for doing this is to click on the Update Button in the Qubes Manger -- one click when you selected Dom0, and one click for each of your template VM (by default there is just one template). - -### From Qubes 1.0 Beta 3 - -If you have Qubes Beta 3 currently installed on your system, you must reinstall from scratch, as we offer no direct upgrade option in the installer (sorry). However, we do offer tools for smooth migration of your AppVMs. In order to do that, please backup your AppVMs using the `qvm-backup` tool [as usual](/doc/backup-restore/). Then, after you install Qubes 1.0 rc1, you can restore them using `qvm-backup-restore` tool. However, because we have changed the default template in RC1, you should tell qvm-back-restore about that by passing `--replace-template` option: - -~~~ -qvm-backup-restore --replace-template=fedora-15-x64:fedora-17-x64 -~~~ +--- \ No newline at end of file diff --git a/developer/releases/2_0/release-notes.md b/developer/releases/2_0/release-notes.md index e0a5751dd..a01c8501d 100644 --- a/developer/releases/2_0/release-notes.md +++ b/developer/releases/2_0/release-notes.md @@ -4,82 +4,7 @@ layout: doc permalink: /doc/releases/2.0/release-notes/ redirect_from: - /en/doc/releases/2.0/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/2_0/release-notes.html ref: 25 title: Qubes R2.0 release notes ---- - -Detailed release notes in [this blog post](https://blog.invisiblethings.org/2014/09/26/announcing-qubes-os-release-2.html) - -## New features since 1.0 - -* Support for generic fully virtualized VMs (without qemu in the TCB!) -* Support for Windows-based AppVMs integration (clipboard, file exchange, qrexec, pv drivers) -* Secure audio input to select AppVMs (Hello Skype users!) -* Clipboard is now also controlled by central policies, unified with other qrexec policies. -* Out of the box TorVM support -* Experimental support for PVUSB -* Updated Xorg packages in Dom0 to support new GPUs -* DisposableVM customization support -* Introduced Xfce 4.10 environment for Dom0 as an alternative to KDE -* Advanced infrastructure for system backups -* Ability to autostart selected VM at system startup -* Support for dynamic screen resolution change -* Dom0 distribution upgraded to Fedora 20 - -## Known issues - -* On some graphics cards the Xfce4 Window Manager (one of the two supported Dom0 Windows Managers in Qubes R2, the other being KDE) might behave "strangely", e.g. decorations might not be drawn sometimes. Also the accompanying lightdm login manager might incorrectly display the wallpaper. If you're facing those problems, it's advisable to use the KDE Window Manager and kdm instead of Xfce4 and lightdm (this is default if one chooses the KDE only installation option in the installer). - -* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead. - -* If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience "heaviness" with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system. - -* Under some circumstances, Qubes backup can create broken backup, without any visible message (\#902). It is advisable to verify a backup to spot the problem. If you encounter this problem, backup VM directory manually. - -* System shutdown sometimes is very slow (\#903). To mitigate the problem, shutdown all the VMs first. - -* For other known issues take a look at [our trac tickets](https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority) - -It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems. - -## Downloads - -See [Qubes Downloads](/doc/QubesDownloads/). - -## Installation instructions - -See [Installation Guide](/doc/installation-guide/). - -## Upgrading - -### From Qubes R2 rc1 - -Upgrading from Qubes R2 rc1 should be a simple matter of installing updates for [dom0](/doc/how-to-install-software-in-dom0/) and [VMs](/doc/software-update-vm/). - -### From Qubes R2 beta 3 and older - -The easiest and safest way to upgrade to Qubes R2 (especially from older releases) is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs. - -Users of R2 beta 3 can upgrade using procedure that has been described [here](/doc/upgrade-to-r2/). - -Note: if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B3 to R2rc1, as described under the link given above. - -### Migrating between beta releases - -#### From Qubes R1 to R2 beta1 - -If you're already running Qubes Release 1, you don't need to reinstall, it's just enough to update the packages in your Dom0 and the template VM(s). This procedure is described [here?](/doc/upgrade-to-r2/). - -#### From Qubes R1 or R2 Beta 1 to R2 beta2 - -Because of the distribution change in R2B2 (from fc13 to fc18) it's preferred that users reinstall Qubes R2B2 from scratch, and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs. - -Advanced users (and advanced users only) can also try a manual upgrade procedure that has been described [here](/doc/upgrade-to-r2b2/). It's advisable to backup your VMs before proceeding anyway! - -#### Upgrading from Qubes R1 or R2 Beta 2 to R2 beta 3 - -The easiest and safest way to upgrade to Qubes R2B3 is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs. - -Users can also try a manual upgrade procedure that has been described [here](/doc/upgrade-to-r2b3/). - -Note: if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B2 to R2B3, as described under the link given above. +--- \ No newline at end of file diff --git a/developer/releases/3_0/release-notes.md b/developer/releases/3_0/release-notes.md index e31b4a276..1e32f98b9 100644 --- a/developer/releases/3_0/release-notes.md +++ b/developer/releases/3_0/release-notes.md @@ -4,59 +4,7 @@ layout: doc permalink: /doc/releases/3.0/release-notes/ redirect_from: - /en/doc/releases/3.0/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_0/release-notes.html ref: 19 title: Qubes R3.0 release notes ---- - -### Qubes R3.0 Release Notes - -This Qubes OS release is dedicated to the memory of Caspar Bowden. - -## New features since 2.0 - -* HAL (Hypervisor Abstraction Layer) - based on libvirt, opens a whole new - possibilities of using different hypervisors. Currently Qubes OS uses Xen. -* Xen 4.4 - many new features, but for us the most important is much more - mature libxl toolstack. -* Qrexec 3 - greatly improved performance by using direct VM-VM connections and - bigger buffers. -* Debian templates gets official support. -* Whonix templates -* Build system improvements - especially support for distribution-specific - plugins (makes supporting multiple distributions much easier) and building - templates using DispVM. -* Automated tests - makes much easier to find bugs, before its even shipped to users - -## Known issues - -* Windows Tools: `qvm-block` does not work - -* UEFI is not supported, you need to enable "legacy boot" in BIOS before installing Qubes OS - -* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead. - -* If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience "heaviness" with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system. - -* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.0%22+label%3Abug) - -It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems. - -## Downloads - -See [Qubes Downloads](/doc/QubesDownloads/). - -## Installation instructions - -See [Installation Guide](/doc/installation-guide/). - -## Upgrading - -### From R3.0 release candidate - -If you are using Qubes R3.0rc1, R3.0rc2 or R3.0rc3, just install system updates, there is no special steps required. - -### From R2.0 or earlier - -The easiest and safest way to upgrade to Qubes R3.0 is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs. - -Users of Qubes R2 can upgrade using [experimental procedure](/doc/upgrade-to-r3.0/). +--- \ No newline at end of file diff --git a/developer/releases/3_0/schedule.md b/developer/releases/3_0/schedule.md index 255ce7080..167ed6ada 100644 --- a/developer/releases/3_0/schedule.md +++ b/developer/releases/3_0/schedule.md @@ -4,12 +4,7 @@ layout: doc permalink: /doc/releases/3.0/schedule/ redirect_from: - /en/doc/releases/3.0/schedule/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_0/schedule.html ref: 20 title: Qubes R3.0 release schedule ---- - -| Date | Stage | -| -----------:| ------------------------------------- | -| 5 Sep 2015 | current-testing freeze before 3.0-rc3 | -| 15 Sep 2015 | 3.0-rc3 release | -| 1 Oct 2015 | 3.0 release | +--- \ No newline at end of file diff --git a/developer/releases/3_1/release-notes.md b/developer/releases/3_1/release-notes.md index 35d1bcd4b..078c371bb 100644 --- a/developer/releases/3_1/release-notes.md +++ b/developer/releases/3_1/release-notes.md @@ -2,64 +2,7 @@ lang: en layout: doc permalink: /doc/releases/3.1/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_1/release-notes.html ref: 16 title: Qubes R3.1 release notes ---- - -## New features since 3.0 - -* Management Stack based of Salt Stack in dom0 - [documentation](/doc/salt/) -* Out of the box Whonix setup -* UEFI support -* LIVE edition (still alpha, not part of R3.1-rc1) -* Updated GPU drivers in dom0 -* Colorful window application icons (instead of just colorful lock icon) -* PV Grub support ([documentation](/doc/managing-vm-kernels/)) -* Out of the box USB VM setup, including [handling USB mouse](https://github.com/QubesOS/qubes-app-linux-input-proxy/blob/master/README.md) -* Xen upgraded to 4.6, for better hardware support (especially Skylake platform) -* Improve updates proxy flexibility - especially repositories served over HTTPS - -You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.1%22+label%3Arelease-notes+is%3Aclosed) - -## Known issues - -* Installation image does not fit on DVD, requires either DVD DL, or USB stick (5GB or more) - -* Windows Tools: `qvm-block` does not work - -* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead. - -* USB mouse (in the case of USB VM) does not work at first system startup (just after completing firstboot). Workaround: restart the system. - -* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.1%22+label%3Abug) - -It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems. - -## Downloads - -See [Qubes Downloads](/downloads/). - -## Installation instructions - -See [Installation Guide](/doc/installation-guide/). - -## Upgrading - -### From R3.0 - -The easiest and safest way to upgrade to Qubes R3.1 is to install it from -scratch and use [qubes backup and restore tools](/doc/backup-restore/) for -migrating of all of the user VMs. - -Users of Qubes R3.0 can upgrade using [experimental -procedure](/doc/upgrade-to-r3.1/). - -### From R2 or earlier - -When upgrading from earlier versions the easiest and safest way is to install -it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) -for migrating of all of the user VMs. - -Alternatively you can [upgrade to R3.0 -using](/doc/releases/3.0/release-notes/#upgrading) first, then follow the -instructions above. This will be time consuming process. +--- \ No newline at end of file diff --git a/developer/releases/3_1/schedule.md b/developer/releases/3_1/schedule.md index 93cd69c7c..5e2d01b35 100644 --- a/developer/releases/3_1/schedule.md +++ b/developer/releases/3_1/schedule.md @@ -4,17 +4,7 @@ layout: doc permalink: /doc/releases/3.1/schedule/ redirect_from: - /en/doc/releases/3.1/schedule/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_1/schedule.html ref: 17 title: Qubes R3.1 release schedule ---- - -This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule). - -| Date | Stage | -| -----------:| --------------------------------------- | -| 8 Dec 2015 | 3.1-rc1 release | -| 5 Jan 2016 | current-testing freeze before 3.1-rc2 | -| 12 Jan 2016 | 3.1-rc2 release | -| 26 Jan 2016 | decide whether 3.1-rc2 is the final 3.1 | -| 9 Feb 2016 | current-testing freeze before 3.1-rc3 | -| ~~16 Feb 2016~~
23 Feb 2016 | 3.1-rc3 release | +--- \ No newline at end of file diff --git a/developer/releases/3_2/release-notes.md b/developer/releases/3_2/release-notes.md index 547dff271..7723f8990 100644 --- a/developer/releases/3_2/release-notes.md +++ b/developer/releases/3_2/release-notes.md @@ -2,60 +2,7 @@ lang: en layout: doc permalink: /doc/releases/3.2/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_2/release-notes.html ref: 21 title: Qubes R3.2 release notes ---- - -## New features since 3.1 - -* Management Stack extended to support in-VM configuration - [documentation](/doc/salt/) -* PV USB - [documentation](/doc/usb/) -* Dom0 update to Fedora 23 for better hardware support -* Kernel 4.4.x -* Default desktop environment switched to Xfce4 -* KDE 5 support (but it is no longer the default one) -* Tiling window managers support: awesome, [i3](/doc/i3/) -* More flexible Qubes RPC services - [related ticket](https://github.com/QubesOS/qubes-issues/issues/1876), [documentation](/doc/qrexec/#service-policies-with-arguments) - -You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.2%22+label%3Arelease-notes+is%3Aclosed) - -## Known issues - -* [Fedora 23 reached EOL in December 2016](https://fedoraproject.org/wiki/End_of_life). There is a [manual procedure to upgrade your VMs](/news/2018/01/06/fedora-26-upgrade/). - -* Windows Tools: `qvm-block` does not work - -* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead. - -* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.2%22+label%3Abug) - -It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems. - -## Downloads - -See [Qubes Downloads](/downloads/). - -## Installation instructions - -See [Installation Guide](/doc/installation-guide/). -After installation, [manually upgrade to Fedora 26](/news/2018/01/06/fedora-26-upgrade/). - -## Upgrading - -### From R3.1 - -The easiest and safest way to upgrade to Qubes R3.2 is to install it from -scratch and use [qubes backup and restore tools](/doc/backup-restore/) for -migrating of all of the user VMs. - -Users of Qubes R3.1 can also upgrade using [this -procedure](/doc/upgrade-to-r3.2/). - -### From R3.0 or earlier - -When upgrading from earlier versions the easiest and safest way is to install -it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) -for migrating of all of the user VMs. - -Alternatively you can [upgrade to R3.1 using](/doc/releases/3.1/release-notes/#upgrading) first, then follow -the instructions above. This will be time consuming process. +--- \ No newline at end of file diff --git a/developer/releases/3_2/schedule.md b/developer/releases/3_2/schedule.md index 36cedd236..baf99e4f9 100644 --- a/developer/releases/3_2/schedule.md +++ b/developer/releases/3_2/schedule.md @@ -4,19 +4,7 @@ layout: doc permalink: /doc/releases/3.2/schedule/ redirect_from: - /en/doc/releases/3.2/schedule/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/3_2/schedule.html ref: 22 title: Qubes R3.2 release schedule ---- - -This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule). - -| Date | Stage | -| -----------:| --------------------------------------- | -| 18 Jun 2016 | 3.2-rc1 release | -| 2 Jul 2016 | decide whether 3.2-rc1 is the final 3.2 | -| ~~16 Jul 2016~~
20 Jul 2016 | current-testing freeze before 3.2-rc2 | -| ~~23 Jul 2016~~
27 Jul 2016 | 3.2-rc2 release | -| ~~5 Aug 2016~~
9 Aug 2016 | decide whether 3.2-rc2 is the final 3.2 | -| 24 Aug 2016 | current-testing freeze before 3.2-rc3 | -| 31 Aug 2016 | 3.2-rc3 release | -| 29 Sep 2016 | 3.2 release | +--- \ No newline at end of file diff --git a/developer/releases/4_0/release-notes.md b/developer/releases/4_0/release-notes.md index b80a1959d..41c1b3219 100644 --- a/developer/releases/4_0/release-notes.md +++ b/developer/releases/4_0/release-notes.md @@ -2,106 +2,7 @@ lang: en layout: doc permalink: /doc/releases/4.0/release-notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/4_0/release-notes.html ref: 23 title: Qubes R4.0 release notes ---- - -New features since 3.2 ----------------------- - -* Core management scripts rewrite with better structure and extensibility, [API documentation](https://dev.qubes-os.org/projects/qubes-core-admin/en/latest/) -* [Admin API](/news/2017/06/27/qubes-admin-api/) allowing strictly controlled managing from non-dom0 -* All `qvm-*` command-line tools rewritten, some options have changed -* Renaming VM directly is prohibited, there is GUI to clone under new name and remove old VM -* Use [PVH](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt) and [HVM](https://github.com/QubesOS/qubes-issues/issues/2185) by default to [mitigate Meltdown & Spectre](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt) and lower the [attack surface on Xen](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-024-2016.txt) -* Create USB VM by default -* [Multiple DisposableVMs templates support](https://github.com/QubesOS/qubes-issues/issues/2253) -* New [backup format](/doc/backup-emergency-restore-v4/) using scrypt key-derivation function -* Non-encrypted backups no longer supported -* [split VM packages](https://github.com/QubesOS/qubes-issues/issues/2771), for better support minimal, specialized templates -* [Qubes Manager decomposition](https://github.com/QubesOS/qubes-issues/issues/2132) - domains and devices widgets instead of full Qubes Manager; devices widget support also USB -* [More flexible firewall interface](/doc/vm-interface/) for ease unikernel integration -* Template VMs do not have network interface by default, [qrexec-based updates proxy](https://github.com/QubesOS/qubes-issues/issues/1854) is used instead -* More flexible IP addressing for VMs - [custom IP](https://github.com/QubesOS/qubes-issues/issues/1477), [hidden from the IP](https://github.com/QubesOS/qubes-issues/issues/1143) -* More flexible Qubes RPC policy - [related ticket](https://github.com/QubesOS/qubes-issues/issues/865), [documentation](/doc/qrexec/#specifying-vms-tags-types-targets-etc) -* [New Qubes RPC confirmation window](https://github.com/QubesOS/qubes-issues/issues/910), including option to specify destination VM -* [New storage subsystem design](https://github.com/QubesOS/qubes-issues/issues/1842) -* Dom0 update to Fedora 25 for better hardware support -* Kernel 4.9.x - -You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.0%22+label%3Arelease-notes+is%3Aclosed) - -Security Notes --------------- - -* PV VMs migrated from 3.2 to 4.0-rc4 or later are automatically set to PVH mode in order to protect against Meltdown (see [QSB #37](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt)). - However, PV VMs migrated from any earlier 4.0 release candidate (RC1, RC2, or RC3) are not automatically set to PVH mode. - These must be set manually. - -* The following steps may need to be applied in dom0 and Fedora 26 TemplateVMs in order to receive updates (see [#3737](https://github.com/QubesOS/qubes-issues/issues/3737)). - - Steps for dom0 updates: - - 1. Open the Qubes Menu by clicking on the "Q" icon in the top-left corner of the screen. - 2. Select `Terminal Emulator`. - 3. In the window that opens, enter this command: - - ``` - sudo nano /etc/yum.repos.d/qubes-dom0.repo - ``` - - 4. This opens the nano text editor. Change all four instances of `http` to `https`. - 5. Press `CTRL+X`, then `Y`, then `ENTER` to save changes and exit. - 6. Check for updates normally. - - Steps for Fedora 26 TemplateVM updates: - - 1. Open the Qubes Menu by clicking on the "Q" icon in the top-left corner of the screen. - 2. Select `Template: fedora-26`, then `fedora-26: Terminal`. - 3. In the window that opens, enter the command for your version: - - ``` - [Qubes 3.2] sudo gedit /etc/yum.repos.d/qubes-r3.repo - [Qubes 4.0] sudo gedit /etc/yum.repos.d/qubes-r4.repo - ``` - - 4. This opens the gedit text editor in a window. Change all four instances of `http` to `https`. - 5. Click the "Save" button in the top-right corner of the window. - 6. Close the window. - 7. Check for updates normally. - 8. Shut down the TemplateVM. - -Known issues ------------- - -* Locale using coma as decimal separator [crashes qubesd](https://github.com/QubesOS/qubes-issues/issues/3753). Either install with different locale (English (United States) for example), or manually apply fix explained in that issue. - -* In the middle of installation, [keyboard layout reset to US](https://github.com/QubesOS/qubes-issues/issues/3352). Be careful what is the current layout while setting default user password (see upper right screen corner). - -* On some laptops (for example Librem 15v2), touchpad do not work directly after installation. Reboot the system to fix the issue. - -* List of USB devices may contain device identifiers instead of name - -* With R4.0.1, which ships kernel-4.19, you may never reach the anaconda startup and be block on an idle black screen with blinking cursor. You can try to add `plymouth.ignore-serial-consoles` in the grub installer boot menu right after `quiet rhgb`. With legacy mode, you can do it directly when booting the DVD or USB key. In UEFI mode, follow the same procedure described for [disabling](/doc/uefi-troubleshooting/#installation-freezes-before-displaying-installer) `nouveau` module (related [solved issue](https://github.com/QubesOS/qubes-issues/issues/3849) in further version of Qubes). - -* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.0%22+label%3Abug) - -It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems. - -Downloads ---------- - -See [Qubes Downloads](/downloads/). - -Installation instructions -------------------------- - -See [Installation Guide](/doc/installation-guide/). - -Upgrading ---------- - -There is no in-place upgrade path from earlier Qubes versions. The only -supported option to upgrade to Qubes R4.0 is to install it from scratch and use -[qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs. -We also provide [detailed instruction](/doc/upgrade-to-r4.0/) for this procedure. +--- \ No newline at end of file diff --git a/developer/releases/4_0/schedule.md b/developer/releases/4_0/schedule.md index 6268a9999..ff21e8c67 100644 --- a/developer/releases/4_0/schedule.md +++ b/developer/releases/4_0/schedule.md @@ -4,25 +4,7 @@ layout: doc permalink: /doc/releases/4.0/schedule/ redirect_from: - /en/doc/releases/4.0/schedule/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/4_0/schedule.html ref: 24 title: Qubes R4.0 release schedule ---- - -This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule). - -| Date | Stage | -| -----------:| --------------------------------------- | -| 31 Jul 2017 | 4.0-rc1 release | -| ~~28 Aug 2017~~
~~11 Sep 2017~~
~~9 Oct 2017~~
18 Oct 2017 | current-testing freeze before 4.0-rc2 | -| ~~4 Sep 2017~~
~~18 Sep 2017~~
~~16 Oct 2017~~
23 Oct 2017 | 4.0-rc2 release | -| 6 Nov 2017 | decide whether 4.0-rc2 is the final 4 -| 20 Nov 2017 | current-testing freeze before 4.0-rc3 | -| 27 Nov 2017 | 4.0-rc3 release | -| 11 Dec 2017 | decide whether 4.0-rc3 is the final 4.0 | -| 1 Jan 2018 | current-testing freeze before 4.0-rc4 | -| ~~8 Jan 2018~~
31 Jan 2018 | 4.0-rc4 release | -| ~~22 Jan 2018~~
14 Feb 2018 | decide whether 4.0-rc4 is the final 4.0 | -| 27 Feb 2018 | current-testing freeze before 4.0-rc5 | -| 6 Mar 2018 | 4.0-rc5 release | -| 20 Mar 2018 | decide whether 4.0-rc5 is the final 4.0 | -| 28 Mar 2018 | final 4.0 release | +--- \ No newline at end of file diff --git a/developer/releases/4_1/release-notes.md b/developer/releases/4_1/release-notes.md index e2a4f9f94..818003177 100644 --- a/developer/releases/4_1/release-notes.md +++ b/developer/releases/4_1/release-notes.md @@ -1,93 +1,6 @@ --- layout: doc -title: Qubes OS 4.1 release notes permalink: /doc/releases/4.1/release-notes/ ---- - -## New features and improvements since Qubes 4.0 - -- Optional qubes-remote-support package now available from repositories - (strictly opt-in, no package installed by default; no new ports or network - connections open by default; requires explicit connection initiation by the - user, then requires sharing a code word with the remote party before a - connection can be established; see - [#6364](https://github.com/QubesOS/qubes-issues/issues/6364) for more - information) -- Qubes firewall reworked to be more defensive (see - [#5540](https://github.com/QubesOS/qubes-issues/issues/5540) for details) -- Xen upgraded to version 4.14 -- Dom0 operating system upgraded to Fedora 32 -- Default desktop environment upgraded to Xfce 4.14 -- Upgraded default template releases -- Experimental support for GUI running outside of dom0 (hybrid mode GUI domain - without real GPU passthrough; see - [#5662](https://github.com/QubesOS/qubes-issues/issues/5662) for details) -- Experimental support for audio server running outside of dom0 ("Audio domain") -- sys-firewall and sys-usb are now disposables by default -- UEFI boot now loads GRUB, which in turn loads Xen, making the boot path - similar to legacy boot and allowing the user to modify boot parameters or - choose an alternate boot menu entry -- New qrexec policy format (see - [#4370](https://github.com/QubesOS/qubes-issues/issues/4370) for details) -- qrexec protocol improvements (see - [#4909](https://github.com/QubesOS/qubes-issues/issues/4909) for details) -- New qrexec-policy daemon -- Simplified using in-qube kernels -- Windows USB and audio support courtesy of - [tabit-pro](https://github.com/tabit-pro) (see - [#5802](https://github.com/QubesOS/qubes-issues/issues/5802) and - [#2624](https://github.com/QubesOS/qubes-issues/issues/2624)) -- Clarified disposable-related terminology and properties -- Default kernelopts can now be specified by a kernel package -- Improved support for high-resolution displays -- Improved notifications when a system drive runs out of free space -- Support for different cursor shapes -- "Paranoid mode" backup restore option now properly supported using - disposables -- Users can now choose between Debian and Fedora in the installer -- Certain files and applications are now opened in disposables, e.g., - Thunderbird email attachments -- New graphical interface for managing testing repository updates -- New "Cute Qube" icon family (replaces padlock icons) -- Disposable qube types now use the disposable icon -- New Template Manager tool for installing, removing, and updating templates - (meanwhile, the tool previously known as the "Template Manager," which was - for mass template switching, has been integrated into the Qube Manager) -- The "file" storage driver has been deprecated in Qubes 4.1 and will be - removed in Qubes 4.2 -- `property-del` event renamed to `property-reset` to avoid confusion -- qrexec no longer supports non-executable files in `/etc/qubes-rpc` -- qrexec components have been reorganized into the core-qrexec repository -- The `qvm-pool` argument parser has been rewritten and improved -- Removed the need for the out-of-tree u2mfn kernel module -- Qrexec services can now run as a socket server -- Improved template distribution mechanism -- Now possible to restart qrexec-agent -- The term "VM" has largely been replaced by "qube" -- GUI daemon is now configured using `qvm-features` tool, - `/etc/qubes/guid.conf` file is no longer used -- `qvm-run` tool got `--no-shell` option to run a single command without using - a shell inside the qube -- MAC Randomization for iwlwifi (see [#938](https://github.com/QubesOS/qubes-issues/issues/938)) - -For a full list, including more detailed descriptions, please see -[here](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.1%22+label%3A%22release+notes%22+is%3Aclosed). - -## Known issues - -For a full list of known 4.1 issues with open bug reports, please see -[here](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.1%22+label%3A%22T%3A+bug%22). -We strongly recommend [updating Qubes OS](/doc/how-to-update/) immediately -after installation in order to apply any and all available bug fixes. - -## Download - -See [downloads](/downloads/). - -## Installation instructions - -See the [installation guide](/doc/installation-guide/). - -## Upgrading - -Please see [how to upgrade to Qubes 4.1](/doc/upgrade/4.1/). +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/4_1/release-notes.html +title: Qubes OS 4.1 release notes +--- \ No newline at end of file diff --git a/developer/releases/4_1/schedule.md b/developer/releases/4_1/schedule.md index 6efd33291..cd65f5701 100644 --- a/developer/releases/4_1/schedule.md +++ b/developer/releases/4_1/schedule.md @@ -4,22 +4,6 @@ layout: doc permalink: /doc/releases/4.1/schedule/ redirect_from: - /en/doc/releases/4.1/schedule/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/4_1/schedule.html title: Qubes R4.1 release schedule ---- - -The table below is based on our [release schedule -policy](/doc/version-scheme/#release-schedule). - -| Date | Stage | -| ----------:| ----------------------------------------- | -| 2021-10-11 | 4.1.0-rc1 release | -| 2021-11-08 | current-testing freeze before 4.1.0-rc2 | -| 2021-11-15 | 4.1.0-rc2 release | -| 2021-11-29 | decide whether 4.1.0-rc2 is the final 4.1 | -| 2021-12-13 | current-testing freeze before 4.1.0-rc3 | -| 2021-12-20 | 4.1.0-rc3 release | -| 2022-01-03 | decide whether 4.1.0-rc3 is the final 4.1 | -| 2022-01-11 | current-testing freeze before 4.1.0-rc4 | -| 2022-01-18 | 4.1.0-rc4 release | -| 2022-01-31 | decide whether 4.1.0-rc4 is the final 4.1 | -| 2022-02-04 | final 4.1.0 release | +--- \ No newline at end of file diff --git a/developer/releases/notes.md b/developer/releases/notes.md index 1be2115db..9206b4cca 100644 --- a/developer/releases/notes.md +++ b/developer/releases/notes.md @@ -2,14 +2,7 @@ lang: en layout: doc permalink: /doc/releases/notes/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/notes.html ref: 13 title: Release notes ---- - -* [Qubes R1.0 release notes](/doc/releases/1.0/release-notes/) -* [Qubes R2.0 release notes](/doc/releases/2.0/release-notes/) -* [Qubes R3.0 release notes](/doc/releases/3.0/release-notes/) -* [Qubes R3.1 release notes](/doc/releases/3.1/release-notes/) -* [Qubes R3.2 release notes](/doc/releases/3.2/release-notes/) -* [Qubes R4.0 release notes](/doc/releases/4.0/release-notes/) -* [Qubes R4.1 release notes](/doc/releases/4.1/release-notes/) +--- \ No newline at end of file diff --git a/developer/releases/schedules.md b/developer/releases/schedules.md index 6f656cf16..ed96fb114 100644 --- a/developer/releases/schedules.md +++ b/developer/releases/schedules.md @@ -2,12 +2,7 @@ lang: en layout: doc permalink: /doc/releases/schedules/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/schedules.html ref: 15 title: Release schedules ---- - -* [Qubes R3.0 release schedule](/doc/releases/3.0/schedule/) -* [Qubes R3.1 release schedule](/doc/releases/3.1/schedule/) -* [Qubes R3.2 release schedule](/doc/releases/3.2/schedule/) -* [Qubes R4.0 release schedule](/doc/releases/4.0/schedule/) -* [Qubes R4.1 release schedule](/doc/releases/4.1/schedule/) +--- \ No newline at end of file diff --git a/developer/releases/todo.md b/developer/releases/todo.md index 23d370edb..a27541334 100644 --- a/developer/releases/todo.md +++ b/developer/releases/todo.md @@ -4,39 +4,7 @@ layout: doc permalink: /doc/releases/todo/ redirect_from: - /en/doc/releases/todo/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/todo.html ref: 14 title: Release checklist ---- - -*the checklist is probably unfinished* - -On -rc1 -------- - -* write schedule -* create package repositories (linux-yum, linux-deb) -* update repository definition (core-agent-linux, installer-qubes-os/qubes-release) -* push all packages to `current-testing` -* draft release notes, one note per feature -* create upgrade package in previous release branch (r2->r3.0, r3.0->r3.1, etc) - core-agent-linux -* make sure that keys for the current release are included in previous release's qubes-release package (for upgrade) -* build ISO and push to mirrors - -On subsequent -rc ------------------ - -* push packages to `current` -* update release notes -* build ISO and push to mirrors -* notify @Rudd-O about the new ISO for new torrent hosting - -On final release ----------------- - -* push packages to `current` -* finish release notes -* update InstallationInstructions -* build ISO and push to mirrors -* notify @Rudd-O about the new ISO for new torrent hosting -* write blog post -* announce on Twitter +--- \ No newline at end of file diff --git a/developer/releases/version-scheme.md b/developer/releases/version-scheme.md index a530947cf..d58ca70f6 100644 --- a/developer/releases/version-scheme.md +++ b/developer/releases/version-scheme.md @@ -6,176 +6,7 @@ redirect_from: - /en/doc/version-scheme/ - /doc/VersionScheme/ - /wiki/VersionScheme/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/releases/version-scheme.html ref: 151 title: Version scheme ---- - -The Qubes OS Project uses the [semantic versioning](https://semver.org/) -standard. Version numbers are written as `..`. When -`` is omitted (e.g., `4.1`), it is usually either because `` is -zero (as in `4.1.0`) or because we are referring to a specific minor release -irrespective of any particular patch release within it. Similarly, the major -release number alone (e.g., `R4`) is sometimes used to refer to an entire -release series inclusive of all minor and patch releases within it. - -In general, patch releases are for backward-compatible bug fixes, minor -releases are for backward-compatible enhancements and new features, and major -release are for any backward-incompatible changes. This means that, in general, -one should *not* try to introduce features or enhancements in patch releases or -any backward-incompatible changes in patch or minor releases. (Templates are a -notable exception, as upstream OSes almost always have their own release -schedules.) Bug fixes are allowed in all releases, and backward-compatible -changes are allowed in all major and minor releases. - -Following standard practice, **version** refers to any build that has been -assigned a version name or number, e.g., `3.2-rc2`, `4.0.4`, `4.1-beta1`. By -contrast, **release** refers to any version that is intended for consumption by -the general userbase. For example, `4.0.4` was both a **version** and a -**release**, since it was stable and intended for general public use, while -`4.1-beta1` was a **version** but *not* a **release**, since it was not stable -and was intended only for [testing](/doc/testing/). All releases are -versions, but not all versions are releases. - -The letter **R**, as in `R4.1`, stands for **release**. The abbreviation **RC**, -as in `3.2-rc2`, stands for **release candidate**. - -## Qubes distributions and products - -We intend to make it easy to make a remix of Qubes, targeting another -hypervisor or isolation provider. We may also create commercial products -intended for specific circumstances. There is one distinguished distribution -called **Qubes OS**. All source code for it is available for download under a -[free and open-source license](/doc/license/) and is openly developed on -[GitHub](https://github.com/QubesOS) and our [mailing -lists](https://www.qubes-os.org/support/). The rest of this document discusses -Qubes OS. Another remix may have its own version series. - -## Release versioning - -Qubes OS as a whole is released from time to time. When preparing a new -release, we decide on the `.` numbers (e.g., `3.0`). We then -publish the first release candidate, `3.0-rc1`. When we feel that enough -progress has been made, we'll release `3.0-rc2` and so on. All these versions -(not yet releases) are considered unstable and not for production use. You are -welcome to [help us test](/doc/testing/) these versions. - -When enough progress has been made, we announce the first stable release, e.g. -`3.0.0`. This not only a version but an actual release. It is considered stable -and we commit to supporting it according to our [support -schedule](/doc/supported-releases/). Core components are branched at this -moment and bug fixes are backported from the master branch. Please see [help, -support, mailing lists, and forum](/support/) for places to ask questions about -stable releases. No major features and interface incompatibilities are to be -included in this release. We release bug fixes as patch releases (`3.0.1`, -`3.0.2`, and so on), while backward-compatible enhancements and new features -are introduced in the next minor release (e.g., `3.1`). Any -backward-incompatible changes are introduced in the next major release (e.g., -`4.0`). - -Issues in our [issue tracker](/doc/issue-tracking/) are sorted by release -[milestones](/doc/issue-tracking/#milestones). - -## Release schedule - -There is no specific schedule for releases other that more general roadmap. -When time comes, Supreme Committee declares feature freeze and tags `-rc1` and -releases ISO image. From this time on, no new features are accepted. Also a -strict time schedule kicks in. - -Each release candidate period is as follows. For the first two weeks we accept -and assign bug reports to be fixed before next release candidate. For the next -two weeks we generally focus on fixing assigned bug reports, so issues -discovered during this time may be postponed until later RC. Finally after that -there is one week of current-testing freeze, during which time no new packages -are released, in hope that they will be installed by wider user base and -tested. - -The next RC is released five weeks after the former. All packets are published -in `current` repository and the cycle starts over. There should be no less than -1 and no more than 3 release candidates before final release. - -| Stage | Duration | -| ------------------------ | --------- | -| initial testing | two weeks | -| bug fixing | two weeks | -| `current-testing` freeze | one week | - -Starting with second cycle (that is, after `-rc1`) two weeks into the cycle -(after primary bug-reporting period) the Supreme Committee decides whether -there should be another RC. If, based on remaining issues, the Committee -decides to release final, then the Committee agrees upon the release date, -which should be no later than a week after. - -[![Release cycle](/attachment/doc/release-cycle.svg)](/attachment/doc/release-cycle.svg) - -## Bug priorities - -When deciding whether the current release candidate is the final one, the -Committee takes bug [priorities](/doc/issue-tracking/#priority) into -consideration. The meaning of them is as follows: - -- `blocker` --- when any such bug is present in the current release candidate, - it can't be considered final release. Bugs with this priority must be fixed - before the next release candidate, even if that means delaying its release - (which should be considered only last resort option). - -- `critical` --- when any such bug is present in the current release candidate, - it can't be considered final release. But such bugs are not qualified to - delay next release candidate release. - -- `major` --- existence of such bugs do not strictly prevent the current - release candidate be considered final (but of course we should try hard to - not have them there). Fixing bugs of this priority can be delayed and - qualified as updates to the final stable release. - -- `default` and `minor` --- existence of such bugs do not prevent the current - release candidate be considered final. Fixing such bugs can be delayed to the - next Qubes OS release. Eventually such fixes might be backported as an update - to the stable release(s). (`default` should really be assigned a more - specific priority, but in practice there are too many issues and not enough - time, so `default` ends up staying on many issues.) - -All above is about bugs, no features should be assigned to the current release -after first `-rc`. Supreme Committee is free to adjust priorities -appropriately. - -## Component version - -Qubes release is defined as specific versions of components, which are -developed more or less separately. Their versions are composed of major and -minor version of target Qubes OS release followed by third component which is -just incremented. There is no apparent indication that given version is stable -or not. - -There are some non-essential components like `qubes-apps-*` that are shared -between releases. Their versions indicate oldest qubes-release that is -supported. We try hard to support multiple releases by one branch to ease code -maintenance. - -Different Qubes releases remixes may comprise of different components and -version are not guaranteed to be monotonic between releases. We may decide that -for newer release some component should be downgraded. There is no guarantee -that arbitrary combination of different versions of random components will -yield usable (or even install-able) compilation. - -## Git tags and branches - -We mark each component version in the repository by tag containing -`v`. Likewise, each Qubes OS release is marked by `R` tag. - -At the release of some release we create branches named like `release2`. Only -bug fixes and compatible improvements are backported to these branches. These -branches should compile. All new development is done in `master` branch. This -branch is totally unsupported and may not even compile depending on maintainer -of repository. - -All version and release tags should be made and signed by someone from ITL -staff. Public keys are included in `qubes-builder` and available at -. - -## Check installed version - -If you want to know which version you are running, for example to report an -issue, you can either check in the Qubes Manager menu under `About > Qubes OS` -or in the file `/etc/qubes-release` in dom0. For the latter you can use a -command like `cat /etc/qubes-release` in a dom0 terminal. +--- \ No newline at end of file diff --git a/developer/services/admin-api.md b/developer/services/admin-api.md index 8fa124177..a8bda0b1f 100644 --- a/developer/services/admin-api.md +++ b/developer/services/admin-api.md @@ -9,352 +9,7 @@ redirect_from: - /doc/mgmt-architecture/ - /doc/admin-api-architecture/ - /doc/admin-api/table/ +redirect_to: https://qubes-doc-rst.readthedocs.io/en/latest/developer/services/admin-api.html ref: 36 title: Admin API ---- - -_You may also be interested in the article -[Introducing the Qubes Admin API](/news/2017/06/27/qubes-admin-api/)._ - -## Goals - -The goals of the Admin API system is to provide a way for the user to manage -the domains without direct access to dom0. - -Foreseen benefits include: - -- Ability to remotely manage the Qubes OS. -- Possibility to create multi-user system, where different users are able to use - different sets of domains, possibly overlapping. This would also require to - have separate GUI domain. - -The API would be used by: - -- Qubes OS Manager (or any tools that would replace it) -- CLI tools, when run from another VM (and possibly also from dom0) -- remote management tools -- any custom tools - -## Threat model - -TBD - -## Components - -![Admin API Architecture](/attachment/doc/admin-api-architecture.svg) - -A central entity in the Qubes Admin API system is a `qubesd` daemon, which -holds information about all domains in the system and mediates all actions (like -starting and stopping a qube) with `libvirtd`. The `qubesd` daemon also manages -the `qubes.xml` file, which stores all persistent state information and -dispatches events to extensions. Last but not least, `qubesd` is responsible for -querying the RPC policy for qrexec daemon. - -The `qubesd` daemon may be accessed from other domains through a set of qrexec -API calls called the "Admin API". This API is the intended -management interface supported by the Qubes OS. The API is stable. When called, -the RPC handler performs basic validation and forwards the request to the -`qubesd` via UNIX domain socket. The socket API is private, unstable, and not -yet documented. - -## The calls - -The API should be implemented as a set of qrexec calls. This is to make it easy -to set the policy using current mechanism. - -| call | dest | argument | inside | return | note | -| ------------------------------------- | --------- | --------- | ----------------------------------------- | --------------------------------------------------------- | ---- | -| `admin.vmclass.List` | `dom0` | - | - | `\n` | -| `admin.vm.List` | `dom0|` | - | - | ` class= state=\n` | -| `admin.vm.Create.` | `dom0` | template | `name= label=