Out of Office and Automatic Response Emails trigger campaign events

I’m running into a slight issue and am looking for advice to fix it or just anyone who has come up with a work around.

My company sends out email campaigns that basically work as follows:
We send Email 1 → Decision Check if they Opened Email → Action Event: Add tag depending on their response to the decision check.

The issue I’m running into is that Out of Office replies are then triggering as a “yes” action and sending the contacts through parts of the campaign they shouldn’t go through because they haven’t actually opened and read the email.

Did anyone else run into this problem and if so did you find a fix or work around? I’m currently having to manually go through hundreds of contacts and double check their tags, decisions, interactions, etc. Does the “Clicks Email” decision fix this issue?