[Bug 905834] Developer Vagrant environment #1999

dean · 2014-06-19T23:17:59Z

This adds an option to deploy kitsune via Vagrant with Ubuntu 14.04.

./manage.py generatedata does not currently work, not sure why. Was planning on using an anonymized dump of the production database for data.
Also, this is blocked until the requirements update in Update requirement from PIL to Pillow #1984 is merged as well.

mythmon · 2014-06-19T23:22:18Z

I'm going to look through this, but it would be good if someone else could review this too, since I've been fairly involved in writing this patch.

willkg · 2014-06-20T01:02:54Z

bin/vagrant_provision.sh

Does this have version 0.90.10? If not, then we need to change this since the Elasticsearch folks change features between minor releases.

This installs 0.90.13, which matches what I had on my laptop when I was helping Dean set this up. I guess I'm not on the right version either :\

Yeah... I think we can get away with version mismatches for most things, but Elasticsearch versions get kind of funky. I highly recommend using 0.90.10 explicitly.

Having said that, maybe it doesn't matter. I don't know what the difference between 0.90.10 and 0.90.13 is. But we definitely can't be using a non-0.90.x version.

willkg · 2014-06-20T01:10:12Z

@mythmon mentioned that you were seeing really crazy issues with Mac OSX and .pth files and that the thinking was that we'd peep-ify vendor/ into a requirements file. I don't see that here, though. Did that get resolved some other way?

dean · 2014-06-20T16:55:37Z

The performance issues stem from using shared directories with Vagrant and Virtualbox. We didn't resolve that here by peep-ifying vendor/, but we should definitely create a bug for it.

Oh, and before I thought vagrant was timing out from the path issues, but it turned out to be another problem. Starting kitsune is slow, but once it's started it works fine.

willkg · 2014-06-26T19:15:49Z

manage.py

So what does this code do now? Is it fundamentally changing sys.path at all?

It's only changing the order of the packages. Instead of putting new_sys_path packages at the front of sys.path, it appends them to the end. It makes lookup times much quicker because the more frequently used packages to run the server are already at the front of sys.path.

Right... so what I'm getting at is that they were at the end after line 12 and then this code takes them off the end and puts them at the end which is fundamentally not changing sys.path at all. Does that make sense? You can probably check sys.path before and after to verify that.

Oh, I see what you're saying. Yeah, I just removed all the custom path configuration code (other than the site.addsitedir(vendors) and it seems to be the same After discussions with you via IRC I'm gonna revert the commit and just peep'ify the vendor packages instead. It seems to be the best solution to this problem anyways.

dean · 2014-09-16T17:49:06Z

Fixed up the script to use peep (making the assumption the peep branch is merged). Works flawlessly with < 2 second load times!

dean · 2014-09-19T22:33:12Z

r?

dean · 2014-09-19T23:19:42Z

The most recent updates have rebased this branch off master so that it can use peep.

To setup your VM, just checkout the branch and run:
$ vagrant up

rlr · 2014-09-25T12:59:46Z

I'll try this out. Assume I know nothing about vagrant, what do I do after vagrant up?

rlr · 2014-09-25T13:00:40Z

(I already have vagrant installed and had it setup for MDN before. I just don't remember the details on how to interact with it.)

mythmon · 2014-09-25T14:47:18Z

After vagrant up, you should be able to vagrant ssh and get into the box. Inside that there should be a fully working kitsune that you can start with manage.py that includes mysql, memcached, redis, es, etc. It should have generatedata, but nothing more.

willkg · 2014-09-25T15:49:57Z

I think the vagrant environment is inextricably tied to the documentation for setting up development environments for contributors. I contend that this should include the changes to the hacking howto because the two should have been changed in tandem and testing both in tandem is important.

rlr · 2014-09-25T19:24:08Z

Here is the end of my output for a vagrant up:

Stderr from the command:

stdin: is not a tty
dpkg-preconfigure: unable to re-open stdin: No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1768  100  1768    0     0    314      0  0:00:05  0:00:05 --:--:--   430
gpg: keyring `/tmp/tmpqei4azro/secring.gpg' created
gpg: keyring `/tmp/tmpqei4azro/pubring.gpg' created
gpg: requesting key C7917B12 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpqei4azro/trustdb.gpg: trustdb created
gpg: key C7917B12: public key "Launchpad chrislea" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: keyring `/tmp/tmpg02ngrom/secring.gpg' created
gpg: keyring `/tmp/tmpg02ngrom/pubring.gpg' created
gpg: requesting key C7917B12 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpg02ngrom/trustdb.gpg: trustdb created
gpg: key C7917B12: public key "Launchpad chrislea" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: keyring `/tmp/tmpiddf2cad/secring.gpg' created
gpg: keyring `/tmp/tmpiddf2cad/pubring.gpg' created
gpg: requesting key DB82666C from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpiddf2cad/trustdb.gpg: trustdb created
gpg: key DB82666C: public key "Launchpad Old Python Versions" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
Extracting templates from packages: 100%

Current default time zone: 'Etc/UTC'
Local time is now:      Thu Sep 25 19:21:22 UTC 2014.
Universal Time is now:  Thu Sep 25 19:21:22 UTC 2014.
Run 'dpkg-reconfigure tzdata' if you wish to change it.


Creating config file /etc/papersize with new version
 * Stopping MariaDB database server mysqld
   ...done.
 * Starting MariaDB database server mysqld
   ...done.
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
fatal: Not a git repository: /Users/rlr/dev/kitsune/.git/modules/kitsune/sumo/static/js/libs/ace
Unable to find current revision in submodule path 'kitsune/sumo/static/js/libs/ace'

rlr · 2014-09-25T19:25:55Z

I am able to ssh in, but I'm not sure what to do next. Should the virtual env be there already?

vagrant@ubuntu-14:~/kitsune$ ./manage.py runserver
Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    from django.conf import settings
ImportError: No module named django.conf

rlr · 2014-09-25T19:31:57Z

I am guessing I have to create a virtualenv and run peep and all that? Or? Just wondering. I know mdn does it so that once you ssh in after vagrant up everything is all set and you call foreman to start the server. Not saying we have to do that, just saying that we need docs like @willkg mentioned.

rehandalal · 2014-09-26T12:39:52Z

bin/vagrant_provision.sh

you could also just do this instead (probably a better idea):

./manage.py nunjucks_precompile

rehandalal · 2014-09-26T12:41:26Z

+1 for updating the docs

rehandalal · 2014-09-26T12:45:32Z

configs/vagrant/settings_local.py

i'm not sure why you're doing this ^ ^^ ^^^

dean · 2014-09-30T17:52:29Z

@rlr From right where you ssh in:

source virtualenv/bin/activate
cd kitsune
./manage.py runserver 0.0.0.0:8000

Let me know if you have any other issues, I'll be updating docs next. :)

generatedata does not work in the VM. An anonymized dump of the production database should be used instead.

Mostly code cleanup, but we also check for the presence of git submodules now. If they aren't found, we show an error message then quit provisioning.

By moving the items to the back of the path, instead of the front, the lookup times for /vendor packages is halved. In the vagrant env we went from 1 minute 56 seconds load time for ./manage.py runserver to 54 seconds.

dean · 2014-11-24T23:49:17Z

Added documentation. r?

rlr · 2014-11-26T03:00:07Z

bin/vagrant_provision.sh

we don't have submodules anymore. So maybe we can remove this? @mythmon ?

Yeah. We don't need to do anything with submodules anymore. I got rid of the last one recently.

rlr · 2014-11-26T03:02:54Z

Looks like you rebased to something recent 👍

I just to have to go through and try it again.

dean · 2014-12-02T20:37:59Z

Fixed the unneeded submodule update in the provisioning script. r?

rlr · 2014-12-05T18:37:04Z

AWESOOMMEEEEE. I just had to add --no-use-wheel to the peep install line. I'll add that in a separate commit.

rlr · 2014-12-05T18:40:02Z

0929579 [bug 905834] Vagrant support for dev deployment.
0f6b739 Add --no-use-wheel in vagrant_provision.sh

willkg reviewed Jun 20, 2014
View reviewed changes

willkg reviewed Jun 26, 2014
View reviewed changes

dean changed the title ~~Vagrant 905834~~ [Bug 905834] Developer Vagrant environment Jul 7, 2014

dean force-pushed the vagrant-905834 branch 2 times, most recently from 0607196 to 1cbfa29 Compare September 11, 2014 22:00

dean force-pushed the vagrant-905834 branch from 33c31ec to a6f51fc Compare September 19, 2014 22:31

rehandalal reviewed Sep 26, 2014
View reviewed changes

bin/vagrant_provision.sh

Copy link

Contributor

rehandalal Sep 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could also just do this instead (probably a better idea):

./manage.py nunjucks_precompile

rehandalal reviewed Sep 26, 2014
View reviewed changes

configs/vagrant/settings_local.py

Copy link

Contributor

rehandalal Sep 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure why you're doing this ^ ^^ ^^^

dean mentioned this pull request Nov 24, 2014

Fixed js module binary paths for docs. #2237

Closed

dean added 2 commits November 24, 2014 15:03

[bug 905834] Vagrant support for dev deployment.

89a4fa6

generatedata does not work in the VM. An anonymized dump of the production database should be used instead.

[bug 905834] Code cleanup and small checks.

9d76eaf

Mostly code cleanup, but we also check for the presence of git submodules now. If they aren't found, we show an error message then quit provisioning.

dean added 17 commits November 24, 2014 15:03

[bug 905834] Changed sys.path arrangement

772bbcc

By moving the items to the back of the path, instead of the front, the lookup times for /vendor packages is halved. In the vagrant env we went from 1 minute 56 seconds load time for ./manage.py runserver to 54 seconds.

[bug 905834] Fixed submodule check.

a6da82b

[bug 905834] Added ignore for .vagrant

bc49f0e

Bogus commit to test travis tests.

4195d81

Fix for bogus commit.

ef1e5b0

Updated vagrant for peep changes.

e0e8286

[bug 905834] Removed submodule check.

93999fe

[bug 905834] Added python2.6 package.

365e575

Updated req installs.

a74b954

Removed manage.py extras

1ebe2bc

Updated provision to reflect changes.

eb9d511

fix comment

18c7963

Fixed up provision script a bit more.

e517ee3

Removed check for submodule. Just do it.

a4e3a29

[bug 905834] Updated with rdalal's suggestions.

17f173c

[bug 905834] Simplified file name.

abb3c31

Updated docs for vagrant.

c285c33

dean force-pushed the vagrant-905834 branch from 991ecdb to c285c33 Compare November 24, 2014 23:48

rlr reviewed Nov 26, 2014
View reviewed changes

[bug 905834] Removed submodule update.

d4d3d99

rlr closed this Dec 5, 2014

takeratta mentioned this pull request Jun 21, 2023

[Snyk] Fix for 1 vulnerabilities takeratta/kitsune#15

Open

takeratta mentioned this pull request Nov 28, 2023

[Snyk] Fix for 24 vulnerabilities takeratta/kitsune#33

Open

takeratta mentioned this pull request Apr 16, 2024

[Snyk] Fix for 1 vulnerabilities takeratta/kitsune#43

Open

[Bug 905834] Developer Vagrant environment #1999

[Bug 905834] Developer Vagrant environment #1999

Uh oh!

Conversation

dean commented Jun 19, 2014

Uh oh!

mythmon commented Jun 19, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

willkg commented Jun 20, 2014

Uh oh!

dean commented Jun 20, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dean commented Sep 16, 2014

Uh oh!

dean commented Sep 19, 2014

Uh oh!

dean commented Sep 19, 2014

Uh oh!

rlr commented Sep 25, 2014

Uh oh!

rlr commented Sep 25, 2014

Uh oh!

mythmon commented Sep 25, 2014

Uh oh!

willkg commented Sep 25, 2014

Uh oh!

rlr commented Sep 25, 2014

Uh oh!

rlr commented Sep 25, 2014

Uh oh!

rlr commented Sep 25, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rehandalal commented Sep 26, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dean commented Sep 30, 2014

Uh oh!

dean commented Nov 24, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlr commented Nov 26, 2014

Uh oh!

dean commented Dec 2, 2014

Uh oh!

rlr commented Dec 5, 2014

Uh oh!

rlr commented Dec 5, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants