Sorting by LastModifiedTime in content search webpart is effected by attachment last modified date

I have a content search web part that is getting the 5 most recently modified items, ordered by LastModifiedTime, from a list. I am also displaying the LastModifiedTime on a custom item display template.

When I attach a document to the list item, the LastModifiedTime uses the documents data instead of the list item itself. I have tested this with excel and word files, but .txt did not cause this issue. Has anyone else experienced this? Is there a workaround?