Dustin -- I'm with you. I think Eloqua needs a "not subscribed to group" in filters and segments. I submitted an enhancement request, but have not commitment from Eloqua at this time.
My work-around is to use campaigns. I create my segments and then pass them through one decision "subscribed to email group?". If yes, I proceed them through the campaign to receive the email. If no, I remove them from the campaign or at least hold them in a Wait step. I don't like this solution because I prefer my filters and segments to show exactly what they will be used for and how. But I haven't yet found a better solution.
Snapshot below shows the four segments I need to send an email to. The three on the left were in countries that don't have an opt-in requirement. The Canada list is in the upper right. I passed it through one decision step, checking if they were subscribed to the proper email group. Those that weren't, I captured in the wait step. That lets us export the contact information and maybe reach them in a different way.