lambda policies need a virtualenv installation.

[frc9]

Getting the following error in the cloudwatch log stream associated with the lambda function provisioned after running a policy:

Unable to import module 'custodian_policy': No module named pkg_resources

using the policy below and invoking it with custodian run -c policy.yml -s out --log-group=/cloud-custodian/dev-account/us-west-2 --region us-west-2 -m

Is there something I'm missing that is causing the mu to not create the lambda function zip file correctly?

policies:
- name: ec2-require-non-public-and-encrypted-volumes
  resource: ec2
  description: |
   Provision a lambda and cloud watch event target
   that looks at all new instances not in an autoscale group
   and terminates those with unencrypted volumes.
  mode:
   type: cloudtrail
   role: arn:aws:iam::XXXXXXXXXXXX:role/custodianRole
   events:
  - RunInstances
    filters:
  - "tag:aws:autoscaling:groupName": absent
  - type: ebs
    key: Encrypted
    value: false
    actions:
  - terminate
- name: tag-compliance
  resource: ec2
  description:
   Schedule a resource that does not meet tag compliance policies
   to be stopped in four days.
  filters:
  - State.Name: running
  - "tag:Environment": absent
  - "tag:AppId": absent
  - or:
    - "tag:OwnerContact": absent
    - "tag:DeptID": absent
      actions:
  - type: mark-for-op
    op: stop
    days: 4

[kapilt]

I haven't seen that one before. It sounds like an issue with the installed environment that's uploading. Could you provide a few more details around the machine your using to run custodian, specifically

• what python version is this?
• which os ?
• How did you install custodian?
• Which custodian version are you using? ($ custodian version)

[frc9]

Hey - false alarm. I did a fresh install and reran my policies and this issue went away.

[kapilt]

i got a report in gitter 1x1 that

seems that "mu.py" was searching for "pkg_resources" in the python2.7/site-packages path, rather than python2.7/dist-packages
updating that path directive and removing the mu.pyc has made a build that works 

we should check both, linux distros have some variations here wrt to location on install that we should respect.

[barundel]

Hello

Hope I can re-open this as am getting the same thing and unable to get past it based on what has already been discussed.

Error when running lambda:
Unable to import module 'custodian_policy': No module named pkg_resources.extern

I updated mu.py line 205 to be dist-packages but that produced the same error with .extern before it was just:

Unable to import module 'custodian_policy': No module named pkg_resources

I have confirmed that pkg_resources exist, tried a fresh AMI, pip install --upgrade setuptools,
pip install --upgrade distribute but no luck

what python version is this? - 2.7.12
which os ? - Amazon Linux AMI
How did you install custodian? - sudo pip install c7n
Which custodian version are you using? ($ custodian version) - 0.8.22.0

[kapilt]

at this time.. we need a virtualenv installation for lambda policies

[barundel]

Thanks

virtualenv was installed in usr/lib/python2.7/dist-packages but not installed to usr/local/lib/python2.7/site-packages where pkg_resources is currently

I removed and installed it to usr/local/lib/python2.7/site-packages and rebuilt with no success

I'm going to try and uninstall and reinstall the packages to the usr/local python, see what that does.

[vijayrcse]

Could you please suggest how to do this virtualenv installation as well ?

I am getting the same error

[barundel]

$ [sudo] pip install virtualenv

[vijayrcse]

okay thanks let me try

[vijayrcse]

it doesn't work. I did below

root@ip-10-14-15-77 ec2-user]# pip install virtualenv
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/local/lib/python2.7/site-packages

i see installation files here
[root@ip-10-14-15-77 ec2-user]# ls -lrt /usr/local/lib/python2.7/site-packages

But still get below error

Unable to import module 'custodian_policy': No module named pkg_resources

Also there is no files under
[root@ip-10-14-15-77 ec2-user]# ls -lrt /usr/lib/python2.7/site-packages/
total 0

Please advise what i am doing wrong?

[vijayrcse]

I am unable to proceed further on this.

can anyone help me how to resolve this

I ran below commands
pip install c7n
pip install virtualenv

After that i executed my policy. When i run my lambda function i get the below error

Unable to import module 'custodian_policy': No module named pkg_resources

Please advise how to resolve this issue

[barundel]

What OS are you running this on?
Update:
ok I see your using ec2-user so its amazon Linux?

I resolved the issue by switching OS to ubuntu

[vijayrcse]

Its amazon linux AMI free tier
4.4.5-15.26.amzn1.x86_64

[vijayrcse]

okay thx let me try that

[vijayrcse]

I spinned-up an Ubuntu server in Amazon and did the following

pip install virtualenv
pip install c7n

The lambda functions that is getting created still throws the below error

TART RequestId: 6d15b00a-e701-11e6-879f-891763142e64 Version: $LATEST
Unable to import module 'custodian_policy': No module named pkg_resources

END RequestId: 6d15b00a-e701-11e6-879f-891763142e64

any suggesting please ?

This is blocking our evaluation of cloud custodian

[kapilt]

you have to create and activate the virtualenv, merely installing is not sufficient. ie

$ virtualenv custodian
$ source custodian/bin/activate
(custodian)$ pip install c7n

[vijayrcse]

thx kapil, let me try this

[vijayrcse]

it didn't work, but i am going to re-create my ubuntu instance and do a clean installation. will let you know.

[vijayrcse]

Now i get a different error

Unable to import module 'custodian_policy': No module named packaging.version

please advise what i am doing wrong here

[vijayrcse]

I just reverted back to Amazon linux AMI and it worked now ( under virtualenv)

Thanks for the help Kapil, barundel...

[kapilt]

that packaging.version issue is resolved in 0.8.23.0 release

[vijayrcse]

thanks kapil will take latest release

[kapilt]

@whit537 could you have a look at this one.

[kapilt]

nutshell we need check both dist-packages/site-packages within virtualenv also some variation on linux distros.

[kapilt]

in the meantime we should update the install docs, till this is resolved.

[chadwhitacre]

@whit537 could you have a look at this one.

Sure thing!

[chadwhitacre]

in the meantime we should update the install docs, till this is resolved.

#983

[chadwhitacre]

Here's the test case I've whittled down to:

Steps

1. Spin up and login to an EC2 instance (Amazon Linux AMI, free tier).
2. sudo pip install c7n to install globally.
3. Create file policy.yml with below content.
4. Populate ~/.aws/credentials.
5. custodian run policy.yml -s . --region us-east-1 to install a Lambda.
6. Hit the Lambda using the "test" feature in the Console.

policies:
- name: foo
  resource: ec2
  mode:
    type: cloudtrail
    role: arn:aws:iam::644160558196:role/custodian-mu
    events:
     - RunInstances

Expected

'detail': KeyError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 78, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 528, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 332, in run
resources = self.resolve_resources(event)
File "/var/task/c7n/policy.py", line 306, in resolve_resources
resource_ids = CloudWatchEvents.get_ids(event, mode)
File "/var/task/c7n/cwe.py", line 136, in get_ids
resource_ids = cls.get_trail_ids(event, mode)
File "/var/task/c7n/cwe.py", line 107, in get_trail_ids
event_name = event['detail']['eventName']
KeyError: 'detail'

Actual

Unable to import module 'custodian_policy': No module named ipaddress

[chadwhitacre]

nutshell we need check both dist-packages/site-packages within virtualenv also some variation on linux distros.

In the Amazon Linux case I'm working with, the libraries we need are hiding out in /usr/lib/python2.7/site-packages, whereas we're looking under /usr/local.

[chadwhitacre]

Okay! PR in #986.

[barundel]

I'm now getting "Unable to import module 'custodian_policy': No module named ipaddress" after upgrade so waiting eagerly for fix :)

[kapilt]

if you use a virtualenv for your custodian install this issue won't effect you. this fix is about making it work with system python installations.

[chadwhitacre]

This should be closeable now that #997 is in.

[chadwhitacre]

I've manually tested the mailer and sentry tools on master. Both still work. 👍

[kapilt]

awesome