Prevent collect.js from sending duplicate DOM content to collector

[groovecoder]

Every visitor to a page re-sends the full DOM content to the webalyzer backend.

Instead, collect.js should calculate a hash of the DOM content, and send a check request to the webalyzer backend (e.g., HEAD /collector/{hash}) to see if it already has the DOM content.

If the backend has the DOM content, it should increment a counter for that page and return a truthy value to collect.js.

If the backend does NOT have the DOM content, collect.js should send the DOM content along with the hash value for the backend to store.

[peterbe]

Excellent idea! I can imagine a diff like this:

class Page(models.Model):
    domain = models.CharField(max_length=100, db_index=True)
    url = models.URLField()
    html = models.TextField()
    size = models.PositiveIntegerField(default=0)
    title = models.TextField(null=True)
    added = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)
+   html_hash = models.CharField(max_length=32, db_index=True)
+   repeats = models.PositiveIntegerField(default=0)

+@receiver(models.signals.pre_save, sender=Page)
+def set_html_hash(sender, instance, **kwargs):
+    if not instance.html_hash:
+       instance.html_hash = hashlib.md5(instance.html).hexdigest()