seeing sensors failing to start related to virtualenv creation issues

[saahn]

I was seeing our sensor processes crash when run as daemon processes via st2ctl or the st2sensorcontainer service. Strangely, running them as foreground processes succeeded.
After some debugging with @Kami , we saw that the virtualenvs for our packs were not set up correctly (e.g. it was missing the eventlet and st2common modules).
This happened even though the st2 run packs.setup_virtualenv packs=<pack name> action ran and succeeded; we were able to reproduce it when we ran rm -rf /opt/stackstorm/virtualenvs/<pack name> and st2 run packs.setup_virtualenv packs=<pack name> again.
Sounds like this is an ' --always-copy relayed' issue where it doesn't copy parent dependencies, to quote from our chat.

[Kami]

Thanks for the report, we will look into it.

@dennybaa @armab can one of you please look into this?

It seems like it's related to the way we create virtual environments with the new packages. We used to use --system-site-packages which symlinked system packages, but now we use --always-copy which should copy over all the system packages.

Which version of virtualenv do we use / install right now? Do we pin it to the latest available version or do we use system version? If we use system version, it could be related to a bug in an old version or similar.

In any case, we need to have some end to end tests for this.

[dennybaa]

@saahn Hi there! Would it be possible to provide steps to reproduce the ^^^ please.

Also it's important to know which distro, since virtualenvs are treated in different way. All platforms except rhel7/centos7 use this option https://github.com/StackStorm/st2/blob/master/conf/st2.package.conf#L23.
While for centos7/rhel7 it was set to empty value during our testing https://github.com/StackStorm/st2-packages/blob/master/packages/st2/Makefile#L57.

@Kami using --system-site-packages doesn't seem possible as far as I remember because virtualenv setup step is run over a /opt/stackstorm/st2 virtualenv (not a bold system python). That's why we used --always-copy if I'm not wrong.

[saahn]

@dennybaa Sure, I've described the steps below. They were executed on Ubuntu-14.04-trusty, where I was trying to set up the latest stackstorm.

sudo su - root

# disable requiretty in sudoers:
Defaults    !requiretty

# fix umask:
umask 0022

# From https://docs.stackstorm.com/install/deb.html :
apt-get update
apt-get install -y mongodb-server rabbitmq-server postgresql
curl -s https://packagecloud.io/install/repositories/StackStorm/staging-stable/script.deb.sh | sudo bash
apt-get install -y st2 st2mistral

# Create Mistral DB in PostgreSQL
cat << EHD | sudo -u postgres psql
CREATE ROLE mistral WITH CREATEDB LOGIN ENCRYPTED PASSWORD 'StackStorm';
CREATE DATABASE mistral OWNER mistral;
EHD

# Setup Mistral DB tables, etc.
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
# Register mistral actions
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate

# add the following to /etc/mongodb.conf:
replSet = rs0
oplogSize = 100

# restart the mongodb service:
service mongodb stop
service mongodb start

# access the mongo cli with "mongo" command and run:
rs.initiate()

# Clone git repos to /root:
squared-platform
squared-stackstorm

# Make simlinks from the SS packs dir to our two packs:
ln -s /root/squared-stackstorm/packs/mongo/ /opt/stackstorm/packs/mongo
ln -s /root/squared-stackstorm/packs/sqplatform/ /opt/stackstorm/packs/sqplatform

#verify symlinks:
ls -al /opt/stackstorm/packs/
lrwxrwxrwx 1 root root 37 Apr 7 01:06 mongo -> <squared_stackstorm repo>/packs/mongo/
lrwxrwxrwx 1 root root 42 Apr 7 01:05 sqplatform -> <squared_stackstorm repo>/packs/sqplatform/

st2ctl stop
st2ctl start

st2ctl reload --register-all

st2 run packs.setup_virtualenv packs=sqplatform
st2 run packs.setup_virtualenv packs=mongo

st2ctl stop
st2ctl start

At this point, i was seeing our sensor processes crash. While debugging with @manasdk , we noticed that permissions were not set correctly, and I was seeing the following in the sensorcontainer logs: "Exception:
Virtual environment (/opt/stackstorm/virtualenvs/sqplatform) for pack "sqplatform" doesn't exist."
This was resolved by fixing permissions:

chown -h root:st2packs /opt/stackstorm/packs/sqplatform
chown -h root:st2packs /opt/stackstorm/packs/mongo

However, my sensor processes continued to crash after the permissions fix.
To debug, we tried running the sensors in the foreground:

/opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2sensorcontainer --config-file /etc/st2/st2.conf --debug

This succeeded and I saw our sensor processes running; we verified that they were still running after 10 seconds. However, they continued to crash when invoked via the st2ctl or service commands.

Next, @Kami asked me to try running the following:

/opt/stackstorm/virtualenvs/sqplatform/bin/python /opt/stackstorm/st2/local/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py --pack=sqplatform --file-path=/opt/stackstorm/packs/sqplatform/sensors/chatops_sensor.py --class-name=ChatopsSensor --trigger-type-refs=sqplatform.onmessage --parent-args=["--config-file", "/etc/st2/st2.conf", "--debug"]

which resulted in:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 22, in <module>
    import eventlet
ImportError: No module named eventlet

That's when we started suspecting the issue was virtualenv creation related. I tried removing and re-installing them:

rm -rf /opt/stackstorm/virtualenvs/sqplatform
rm -rf /opt/stackstorm/virtualenvs/mongo
st2 run packs.setup_virtualenv packs=sqplatform
...............
id: 56fd8f8e7003051b98468464
status: succeeded
parameters: 
  packs:
  - sqplatform
result: 
  exit_code: 0
  result: 'Successfuly set up virtualenv for the following packs: sqplatform'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "sqplatform"

    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/sqplatform" doesn''t exist

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "sqplatform" in "/opt/stackstorm/virtualenvs/sqplatform"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/sqplatform" using Python binary "/opt/stackstorm/st2/bin/python"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy /opt/stackstorm/virtualenvs/sqplatform" to create virtualenv.

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/sqplatform/requirements.txt"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "sqplatform" successfully created in "/opt/stackstorm/virtualenvs/sqplatform"

    '
  stdout: ''

st2 run packs.setup_virtualenv packs=mongo
..
id: 56fd8fba7003051b98468467
status: succeeded
parameters: 
  packs:
  - mongo
result: 
  exit_code: 0
  result: 'Successfuly set up virtualenv for the following packs: mongo'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "mongo"

    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/mongo" doesn''t exist

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "mongo" in "/opt/stackstorm/virtualenvs/mongo"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/mongo" using Python binary "/opt/stackstorm/st2/bin/python"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy /opt/stackstorm/virtualenvs/mongo" to create virtualenv.

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/mongo/requirements.txt"

    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "mongo" successfully created in "/opt/stackstorm/virtualenvs/mongo"

    '
  stdout: ''

I still saw the same ImportError: No module named eventlet error when I tried to run Kami's command. I ran /opt/stackstorm/virtualenvs/sqplatform/bin/pip install eventlet as a workaround, but then saw another import error: ImportError: No module named st2common

That's when I opened this issue.

Thank you for looking into this.
Please let me know if I can further assist with debugging in any way.

[lakshmi-kannan]

@dennybaa I can confirm I am seeing the same issue as @saahn. I think the behavior of --always-copy has changed recently. With a fresh install, I ran st2 run packs.install packs=jira and I see the following for packs.setup_virtualenv

https://gist.github.com/lakshmi-kannan/2561e893a508ac43175c4f8f45efa6dc

Relevant st2 change: StackStorm/st2#2372

I am pretty sure --always-copy worked at some point.

[saahn]

Thanks for verifying @lakshmi-kannan . Please keep me updated on any fix or workaround for this. Thanks again!

[m4dcoder]

@saahn can you post the output for the following commands? /opt/stackstorm/st2/bin/pip show virtualenv and virtualenv --version.

i was not able to reproduce the problem. i followed instruction @ https://docs.stackstorm.com/install/deb.html to install a fresh st2 v1.3.2 on ubuntu 14.04. i was able to install the rabbitmq pack and run the rabbitmq sensor without problem. i had a script that publishes message to a queue every sec for 10 min. the rabbitmq sensor picked up every single message and the sensor container stayed up. may i suggest you to try the rabbitmq sensor and see if that works for you just so we are on the same page here.

[saahn]

@m4dcoder sure:

root@base-stackstorm-435-onint:~# /opt/stackstorm/st2/bin/pip show virtualenv
---
Metadata-Version: 2.0
Name: virtualenv
Version: 13.1.2
Summary: Virtual Python Environment builder
Home-page: https://virtualenv.pypa.io/
Author: Jannis Leidel, Carl Meyer and Brian Rosner
Author-email: [email protected]
License: MIT
Location: /opt/stackstorm/st2/lib/python2.7/site-packages
Requires: 
You are using pip version 7.1.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@base-stackstorm-435-onint:~# virtualenv --version
15.0.1

@lakshmi-kannan reported that he was able to reproduce.
I'll try the rabbitmq sensor

[lakshmi-kannan]

@saahn @m4dcoder: I am unable to repro today which is bizzare. I can't tell what's different today. Seems like nothing changed code wise. Pretty weird.

[m4dcoder]

We identified the problem with @saahn. It has to do with the packs being in /root directory and symlinked under /opt/stackstorm/packs. Once the packs are moved physically to /opt/stackstorm/packs and with the correct permissions set, the sensor container started working.

[Kami]

@m4dcoder Good "investigation" :)

I would imagine there is a bug in virtualenv copy functionality (it doesn't handle symlinks correctly), but maybe hard links would still work (something to try out in case they want to preserve the linking flow).

In any case, it's something we should probably document and update the code if such scenario is encountered to throw and make it explicit what is going on and how to resolve it.