Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
192dbed
Working on bulk email feature
kluo May 25, 2013
ae6e1d4
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo May 28, 2013
368e211
Use simple textarea instead of visual tinyMCE editor for front-end
kluo May 28, 2013
63c63d4
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo May 29, 2013
7e007d7
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo May 30, 2013
5d7a39e
Add settings flag for enabling email feature
kluo May 30, 2013
77fafd4
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo May 31, 2013
e3cb34b
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo Jun 1, 2013
32c755b
Template unit tests for bulk email in instructor dashboard
kluo Jun 1, 2013
e41cfb9
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo Jun 3, 2013
469567e
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo Jun 3, 2013
ec046aa
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo Jun 4, 2013
509fd7a
Fix indentation in dashboard template for email link
kluo Jun 4, 2013
4e0a32d
correct delegate_email docstring
sefk Jun 6, 2013
99f520d
course_email: fix docstring, remove with_celery()
sefk Jun 6, 2013
ec64311
Merge branch 'master' of github.com:edx/edx-platform into feature/klu…
kluo Jun 7, 2013
a1e17b1
Remove unused generated views file from bulk email app
kluo Jun 7, 2013
3da0d33
Merge branch 'feature/kluo/bulk-email' of github.com:edx/edx-platform…
kluo Jun 7, 2013
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file.
19 changes: 19 additions & 0 deletions lms/djangoapps/bulk_email/models.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from django.db import models
from django.contrib.auth.models import User

class Email(models.Model):
sender = models.ForeignKey(User, default=1, blank=True, null=True)
hash = models.CharField(max_length=128, db_index=True)
subject = models.CharField(max_length=128, blank=True)
html_message = models.TextField(null=True, blank=True)
created = models.DateTimeField(auto_now_add=True)
modified = models.DateTimeField(auto_now=True)
class Meta:
abstract = True

class CourseEmail(Email, models.Model):
course_id = models.CharField(max_length=255, db_index=True)
to = models.CharField(max_length=64)

def __unicode__(self):
return self.subject
53 changes: 53 additions & 0 deletions lms/djangoapps/bulk_email/tasks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
from celery import task
from django.conf import settings
from django.contrib.auth.models import User
from bulk_email.models import *

import math

EMAILS_PER_WORKER=getattr(settings, 'EMAILS_PER_WORKER', 10)

@task()
def delegate_emails(hash_for_msg, recipient, course):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this an @task? It looks to me like it's supposed to be run on the webapp side, and it triggers the remote tasks (which seems to me to say that it isn't a task itself).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This task that fans out all the other tasks itself can take a while to run, so we don't want it to be synchronous on the web app side. For a 50k member MOOC, say, it has to kick off 5k tasks, assuming 10 mails per batch, which is what we used on Class2Go.

'''
Delegates emails by querying for the list of recipients who should
get the mail, chopping up into batches of EMAILS_PER_WORKER size,
and queueing up worker jobs.

Recipient is {'students', 'staff', or 'all'}

Returns the number of batches (workers) kicked off.
'''

recipient_qset = User.objects.all()
if recipient == "students":
#get student list
pass
elif recipient == "staff":
#get staff list
pass
else:
#everyone
pass
recipient_list = list(recipient_qset)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will pull potentially all user objects into memory, which seems less than ideal. Why doesn't each worker do the query for their chunk of the data?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we originally coded this up we passed around a query set. This was more efficient, as you said, since you would only have to instantiate user objects that were needed per batch, instead of all at once here.

But this is a good observation, @cpennington, that we should be careful that we can actually execute this query without hitting memory limits on

Also: if we're going to the trouble to actually get all this data, then we should probably build a dict for the values that we will need for each user here and actually pass that along in the batch. I believe the four fields we will need are just name, email address, optout flag, and optout code.


total_num_emails = recipient_qset.count()
num_workers=int(math.ceil(float(total_num_emails)/float(EMAILS_PER_WORKER)))
chunk=int(math.ceil(float(total_num_emails)/float(num_workers)))

for i in range(num_workers):
to_list=recipient_list[i*chunk:i*chunk+chunk]
course_email.delay(hash_for_msg, to_list, False, course)
return num_workers


@task(default_retry_delay=15, max_retries=5)
def course_email(hash_for_msg, to_list, course, throttle=False):
"""
Takes a subject and an html formatted email and sends it from
sender to all addresses in the to_list, with each recipient
being the only "to". Emails are sent multipart, in both plain
text and html.
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc string looks out of date, since this function can't be called with num_workers and worker_id.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep


msg = CourseEmail.objects.get(hash=hash_for_msg)
Empty file.
74 changes: 74 additions & 0 deletions lms/djangoapps/bulk_email/tests/tests.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""
Unit tests for email feature in instructor dashboard

Based on (and depends on) unit tests for courseware.
"""

from django.test.utils import override_settings

# Need access to internal func to put users in the right group
from django.contrib.auth.models import Group

from django.conf import settings
from django.core.urlresolvers import reverse

from courseware.access import _course_staff_group_name
from courseware.tests.tests import LoginEnrollmentTestCase, TEST_DATA_XML_MODULESTORE, get_user
from xmodule.modulestore.django import modulestore
import xmodule.modulestore.django

@override_settings(MODULESTORE=TEST_DATA_XML_MODULESTORE)
class TestInstructorDashboardEmailView(LoginEnrollmentTestCase):
'''
Check for email view displayed with flag
'''

def setUp(self):
xmodule.modulestore.django._MODULESTORES = {}

self.toy = modulestore().get_course("edX/toy/2012_Fall")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure we're moving away from using this way of mocking courses and using CourseFactory instead. There are a few examples of this floating around, and the definition for it is in common/lib/xmodule/xmodule/modulestore/tests/factories.py . @wedaly would have better information on how to use it and how to improve it if you need it to do more than it currently does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @kluo just followed examples he found elsewhere. If you need him to recode in that manner, then he could if need be. I can't tell however if the new method is just desire/preference or the must-do way for some stronger reason (safer? more robust?). I guess before asking someone to recode something I'd like to explain why.


# Create instructor account
self.instructor = 'view@test.com'
self.password = 'foo'
self.create_account('u1', self.instructor, self.password)
self.activate_user(self.instructor)

group_name = _course_staff_group_name(self.toy.location)
g = Group.objects.create(name=group_name)
g.user_set.add(get_user(self.instructor))

self.logout()
self.login(self.instructor, self.password)
self.enroll(self.toy)

def test_email_flag_true(self):
oldEmailFlag = settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL']
settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hacking around the settings this way, I would recommend using django's settings override feature: https://docs.djangoproject.com/en/1.4/topics/testing/#overriding-settings

I've had some trouble dealing with it myself in the past, but it might work in this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't use override_settings here because I'm just changing one pair of the MITX_FEATURES dictionary, and it can't take in a dict key expression. I think it would only work if I overrode the entire MITX_FEATURES dictionary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @gwprice had had luck using patch.dict from the mock library to
just override a single feature flag. We could also write our own decorator
for feature flags, since it's a common usecase.

-Cale
On Jun 4, 2013 6:22 PM, "Kevin Luo" notifications@github.com wrote:

In lms/djangoapps/bulk_email/tests/tests.py:

  •    self.instructor = 'view@test.com'
    
  •    self.password = 'foo'
    
  •    self.create_account('u1', self.instructor, self.password)
    
  •    self.activate_user(self.instructor)
    
  •    group_name = _course_staff_group_name(self.toy.location)
    
  •    g = Group.objects.create(name=group_name)
    
  •    g.user_set.add(get_user(self.instructor))
    
  •    self.logout()
    
  •    self.login(self.instructor, self.password)
    
  •    self.enroll(self.toy)
    
  • def test_email_flag_true(self):
  •    oldEmailFlag = settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL']
    
  •    settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = True
    

I can't use override_settings here because I'm just changing one pair of
the MITX_FEATURES dictionary, and it can't take in a dict key expression. I
think it would only work if I overrode the entire MITX_FEATURES dictionary.


Reply to this email directly or view it on GitHubhttps://github.com/edx/pull/30/files#r4534930
.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nice pattern in that override settings page might be useful here. If you use the decorator it will ensure cleanup to settings when you make modifications. More robust, ie. when an exception happened somewhere before you had a chance to clean up.

from django.test.utils import override_settings

@override_settings()
def test_email_flag_true(self):
    settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = True
    ## do your tests
    ## don't have to clean up, decorator will take care of it

That seems pretty clean to me

response = self.client.get(reverse('instructor_dashboard',
kwargs={'course_id': self.toy.id}))
email_link = '<a href="#" onclick="goto(\'Email\')" class="None">Email</a>'
self.assertTrue(email_link in response.content)

session = self.client.session
session['idash_mode'] = 'Email'
session.save()
response = self.client.get(reverse('instructor_dashboard',
kwargs={'course_id': self.toy.id}))
selected_email_link = '<a href="#" onclick="goto(\'Email\')" class="selectedmode">Email</a>'
self.assertTrue(selected_email_link in response.content)
send_to_label = '<label for="id_to">Send to:</label>'
self.assertTrue(send_to_label in response.content)

del self.client.session['idash_mode']
settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = oldEmailFlag

def test_email_flag_false(self):
oldEmailFlag = settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL']
settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = False
response = self.client.get(reverse('instructor_dashboard',
kwargs={'course_id': self.toy.id}))
email_link = '<a href="#" onclick="goto(\'Email\')" class="None">Email</a>'
self.assertFalse(email_link in response.content)

settings.MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = oldEmailFlag
24 changes: 23 additions & 1 deletion lms/djangoapps/instructor/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@

from .offline_gradecalc import student_grades, offline_grades_available

from bulk_email.models import CourseEmail
import datetime
from hashlib import md5
from bulk_email.tasks import delegate_emails

log = logging.getLogger(__name__)

# internal commands for managing forum roles:
Expand Down Expand Up @@ -586,7 +591,24 @@ def getdat(u):
ret = _do_enroll_students(course, course_id, students, overload=overload)
datatable = ret['datatable']


#----------------------------------------
# email

elif action == 'Send email':
to = request.POST.get("to")
subject = request.POST.get("subject")
html_message = request.POST.get("message")

email = CourseEmail(course_id=course_id,
sender=request.user,
to=to,
subject=subject,
html_message=html_message,
hash=md5((html_message+subject+datetime.datetime.isoformat(datetime.datetime.now())).encode('utf-8')).hexdigest())
email.save()

delegate_emails(email.hash, email.to, course)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry a little bit about transactions and race conditions. In particular, this transaction won't end until the request does, which means if the worker is especially speedy, it could try and look up the CourseEmail object before it has been committed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be called with .delay?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To your concern, @cpennington, all the more reason why this should be queued for fanout later. That way the delegation job is queued, and then the request is done and the transaction commits. Writing the message should happen then. The message isn't read until worker tasks are starting to be spun up by the remote machine.

So there is a race potential here but feels unlikely.

Safeguard for this would be to test for the content (query on hash) before doing the delegation. If it's not there, requeue so it can be retried. Celery supports retries like this.


#----------------------------------------
# psychometrics

Expand Down
3 changes: 3 additions & 0 deletions lms/envs/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@
# analytics experiments
'ENABLE_INSTRUCTOR_ANALYTICS': False,

'ENABLE_INSTRUCTOR_EMAIL': False,

# Flip to True when the YouTube iframe API breaks (again)
'USE_YOUTUBE_OBJECT_API': False,

Expand Down Expand Up @@ -694,6 +696,7 @@
'psychometrics',
'licenses',
'course_groups',
'bulk_email',

#For the wiki
'wiki', # The new django-wiki from benjaoming
Expand Down
1 change: 1 addition & 0 deletions lms/envs/dev.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
MITX_FEATURES['ENABLE_PSYCHOMETRICS'] = False # real-time psychometrics (eg item response theory analysis in instructor dashboard)
MITX_FEATURES['ENABLE_INSTRUCTOR_ANALYTICS'] = True
MITX_FEATURES['ENABLE_SERVICE_STATUS'] = True
MITX_FEATURES['ENABLE_INSTRUCTOR_EMAIL'] = True

WIKI_ENABLED = True

Expand Down
24 changes: 24 additions & 0 deletions lms/templates/courseware/instructor_dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
<script type="text/javascript" src="${static.url('js/vendor/jquery-jvectormap-1.1.1/jquery-jvectormap-1.1.1.min.js')}"></script>
<script type="text/javascript" src="${static.url('js/vendor/jquery-jvectormap-1.1.1/jquery-jvectormap-world-mill-en.js')}"></script>
<script type="text/javascript" src="${static.url('js/course_groups/cohorts.js')}"></script>
<script type="text/javascript" src="${static.url('js/vendor/tiny_mce/jquery.tinymce.js')}"></script>

</%block>

Expand Down Expand Up @@ -111,6 +112,9 @@ <h2>[ <a href="#" onclick="goto('Grades');" class="${modeflag.get('Grades')}">Gr
<a href="#" onclick="goto('Enrollment');" class="${modeflag.get('Enrollment')}">Enrollment</a> |
<a href="#" onclick="goto('Data');" class="${modeflag.get('Data')}">DataDump</a> |
<a href="#" onclick="goto('Manage Groups');" class="${modeflag.get('Manage Groups')}">Manage Groups</a>
%if settings.MITX_FEATURES.get('ENABLE_INSTRUCTOR_EMAIL'):
| <a href="#" onclick="goto('Email')" class="${modeflag.get('Email')}">Email</a>
%endif
%if settings.MITX_FEATURES.get('ENABLE_INSTRUCTOR_ANALYTICS'):
| <a href="#" onclick="goto('Analytics');" class="${modeflag.get('Analytics')}">Analytics</a>
%endif
Expand Down Expand Up @@ -365,6 +369,26 @@ <H2>Student-specific grade inspection and adjustment</h2>
%endif
%endif

##-----------------------------------------------------------------------------

%if modeflag.get('Email'):
<p>
<label for="id_to">Send to:</label>
<select id="id_to" name="to">
<option value="students">All Students</option>
<option value="staff">All Course Staff</option>
<option value="all">All (students and staff)</option>
</select>
<label for="id_subject">Subject: </label>
<input type="text" id="id_subject" name="subject" maxlength="100" size="75">
<label for="id_message">Message:</label>
<textarea cols="100" id="id_message" name="message"></textarea>
</p>
<p>
<input type="submit" name="action" value="Send email">
</p>
%endif

</form>
##-----------------------------------------------------------------------------

Expand Down