Late annotations are being dropped

[matt-sheets]

As seen in the following PR:
#192
We added the concept of early and late annotations. Processing the late annotations added a 6% increase in our benchmark times.

Currently we do not leverage any late annotations, so dropping them has no negative functionality impact at this time. Since we don't want to take the performance hit, we will continue with them being dropped for the time being.

We should try to resolve the performance issue before we start processing the late annotations if possible.