Skip to content

SprintCoderSetup

Jump to bottom
rjsparks edited this page Nov 18, 2021 · 117 revisions

Sprint Coder Setup

Introduction

** This page provides instructions for setting up a development environment using the container platform, Docker, which allows a quicker and easier way of getting started for the Code Sprint. If you need to set the environment locally (for bigger, longer-term projects regarding Datatracker), check out NativeSetup. Native setup is NOT recommended for Sprint setups. **

Note that, although the Docker setup makes this process go more quickly, you should set aside at least an hour to complete these steps.

PageOutline(2)

Install Docker

Docker is a toolkit which lets you package software together with its dependencies in lightweight containers, and run it in isolated virtual environments. This is really helpful for Code Sprint participants because they don't have to chase down all the dependencies in order to get started working on the Datatracker.

  • Set up Docker on your preferred platform. Official installers exist for many Linux flavours, OS X, Windows and Cloud services, and can be found at the Docker Store. The free version (CE, Community Edition) is fine.

    Please follow the Docker installation all the way through to successfully running the hello-world example in a terminal window ( $ docker run hello-world).

Check Out Your Code Branch

Datatracker code is kept in a Subversion repository. If you need a primer on Subversion, here is an online reference.

  • You will need Subversion 1.8.0 or later on your platform in order to check out the repository.

  • Check your email for a notification about the personal branch that has been
    created for you to work in.

    Your personal branch is created based on your signup on the the Sprint Signup page. If you signed up ahead of time, you will receive an email notification about a week before the Code Sprint; if you join just before or during the Code Sprint, add your name to the Sprint Signup page, and then let the people running the Sprint know so that they can run a script that reads the Sprint Signup page and creates your personal branch.

  • (Optional) If your host machine uses a locale in which dates may contain UTF-8 characters, the ietf/__init__.py file may contain characters with a non-ASCII encoding. To fix this, export LC_ALL=C before checkout.

  • Check out your Datatracker branch in a suitable directory. We'll assume ~/src/dt/ here, and assume you are 'coder' and the current release is '7.22.1.dev0':

       ~/src/dt/ $ svn co https://svn.ietf.org/svn/tools/ietfdb/personal/coder/7.22.1.dev0

    In the checked-out working copy, you'll find a docker/ directory and a data/ directory at the top level.

Set Up the Docker Image

The way we do development using docker was completely revised in late 2021. If you were familiar with the previous system know that most of the commands you are familiar with have changed significantly, and the way you have invoked them in the past will either not work at all, or produce surprising results.

In your fresh branch checkout, read ./docker/README.md carefully and follow its instructions for that given release. These instructions are expected to change with each release as we improve the new image models and move our source tree into github.

** TODO: The remainder of these instructions have not yet been updated to reflect the changes to the containerization - proceed with caution until they are **

RUN THE TESTS

  • First, run the built-in checks:

       (virtual) $ ./ietf/manage.py check

    This will

    1. ensure that you don't have missing directories or any other obvious issues with the OS environment

    2. apply any patches needed to bring library modules up to snuff.

  • You are now ready to run the tests. Make sure that one of the following commands runs to completion without errors:

       (virtual) $ ./ietf/manage.py test --settings=settings_sqlitetest
   or
   (virtual) $ ./ietf/manage.py test --settings=settings_sqlitetest -v 3   # for more verbose output

    If you see any errors or failures, you probably have a configuration problem or are missing a required python module. If it's not obvious what to do to correct the problem, please ask for help. The Troubleshooting page may provide some hints. Once you resolve the problem, consider adding a new hint on that page. For help with running the tests, type:

      (virtual) $ ./ietf/manage.py test --help

    A specific subset of test cases containing a failure can be re-run without re-running all of the test cases using (for example):

      (virtual) $ ./ietf/manage.py test --settings=settings_sqlitetest -v 3 ietf.submit.tests.SubmitTests

Start the Development Server

  • Then start the dev server:

       (virtual) $ ./ietf/manage.py runserver 0.0.0.0:8000

    or you can run this in a separate terminal window:

       ~/src/dt/7.22.1.dev0/ $ ./docker/devserver

    '''This, and the mailserver command have an issue - if you use ctrl-C to stop the server, it only disconnects from the virtual machine - the command it has run is still running, and you need to go into a shell in the virtual machine to stop it.''' Note the IP address 0.0.0.0 used to make the dev server bind to all addresses. The internal port 8000 has been mapped to port 8000 externally, too. If you are using a recent native Docker for Mac or Docker for Windows install, you will find the datatracker on localhost:8000:

       ~/src/dt/7.22.1.dev0/ $ open http://localhost:8000/

    If you are running an older install of Docker, you may have to find the IP address of the VirtualBox which provides the virtual environment. If so, run '$ docker-machine ip' outside the virtual environment:

       ~/src/dt/7.22.1.dev0/ $ docker-machine ip
   192.168.59.103

   ~/src/dt/7.22.1.dev0/ $ open http://192.168.59.103:8000/

  • You can sign into the Datatracker with any valid username using the password 'password'. Signing out will allow you to sign in with a different username. Trying your new code out as people with different roles will help find issues early.

(Optional) Run the mailserver

  • (Optional) Your local version of Datatracker does not actually send email messages. If you are testing email functionality, you can see what would have been sent by running the following in a separate terminal window: 
       ~/src/dt/7.22.1.dev0/ $ ./docker/mailserver
    Any emails generated will print to the terminal window.

Manually Edit or rsync Datatracker Data Directories

The container has empty directories to hold the files the Datatracker uses. You'll find them under data/developers.

  • You can put files in the directories individually if and when you need them. For instance, you can put drafts in data/developers/ietf-ftp/internet-drafts one at a time to see them displayed on the document's main page in your test instance - otherwise, you'll simply get a "cannot read" message.

  • Alternatively, you can use rsync to fetch all current files. Warning - this takes about 3 Gb:

        # only if you have the space and time:
    (virtual) $ rsync -avz rsync.ietf.org::developers/ data/developers/

    Note that the settings_local.py provided by this container forces the draft repository and draft archive to be the same directory. Feel free to separate them if you are already keeping a full copy of the draft archive separately (or want to mirror it into data from rsync.ietf.org::id-archive (about 6Gb)).

  • Depending on what you are going to work on, you may later need to obtain the dot and pyang binaries and tell the datatracker where they are by adding this to your setting_local.py (paths here are examples only):

        DOT_BINARY = '/opt/local/bin/dot'
    UNFLATTEN_BINARY= '/opt/local/bin/unflatten'
    PS2PDF_BINARY = '/opt/local/bin/ps2pdf'
    SUBMIT_PYANG_COMMAND = ('/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pyang'
                              '-p %(workdir)s --verbose --ietf  %(model)s' )

Setup Complete

You can now start working on Datatracker code.

Workflow

  • Pick something to work on. This could be any of the following:

    • something you miss in the datatracker, and would like to add
    • fixing a bug that irks you in particular
    • fixing a bug or implementing an idea from the curated list of EnhancementIdeas Once you have picked something, please add it to the table on the sprint wiki page (e.g., IETF105Sprint) so that other people will know not to pick the same thing to work on, or will know to contact you in order to cooperate on it.

  • Do the coding. Talk with your fellow sprinters about what you're doing. Ask for advice as needed. Whenever possible, please add tests to cover the code you've added.

  • Run the test suite to make sure that it still passes.

        (virtual) $ ./ietf/manage.py test --settings=settings_sqlitetest

  • You can run individual tests too while you develop by adding the path to the Class and test case as an argument.

        (virtual) $./ietf/manage.py test ietf.meeting.tests_views.IphoneAppJsonTests.test_iphone_app_json --settings=settings_sqlitetest

  • Commit your code to your branch. Provide a commit message which can be used in the changelog and announcement of what's gone into a release. Indicate which bug your code fixes, using the syntax described in SvnTracHooks -- typically Fixes issue !#1234. Also, if your commit is ready to merge, indicate this with the phrase Commit ready for merge. The merge request phrases are also described in [CodeRepository(https://github.com/ietf-svn-conversion/datatracker/wiki/CodeRepository)]. Some examples:

       svn commit ietf/foo/bar.py -m "Changed to use the right fozboz function instead of frooboz.\
     Fixes issue #42.  Commit ready for merge."

   svn commit ietf/baz.py -m "Partially addresses issue #43, removing redundant code. Also related to #44."
Clone this wiki locally