My weird habit

Of course everyone has habits.  Some stranger than others.  I happen to have one that seems offensive to people.

When talking to some people, I don't make eye contact.  Obvious enough to make me seem rude.  The exception is when I meet someone new, or I'm talking to an authority figure that could easily influence me, like the principal, police officers, or anyone that I respect a lot who I'm having a serious conversation with. 

Why do I not look at people? Firstly, I think I require a bit more effort for me to pay attention to people than most people.  I'm not a very good audio learner, so listening to someone requires my full attention.  Otherwise I might not actually listen to everything they say.  And one thing that distracts me is actually their face.  The person that I talk to always has an emotion (considering that the "emotionless" face is an emotion) and how it may ever so slightly change is distracting sometimes.  I start to wonder what is going on in their head rather than register and process what they are saying to me.  As a result, I look around aimlessly, not at a specific object, and concentrate on what they are saying.  I saw a video of me doing it once, and it half looks like staring off into space and half actively looking for something (ironic, I know).

For the people who fall in the exception category, they are the people where I thought presenting myself with the best manners possible trumps my strange method of concentration.  Of course, this means that I have to ignore their face while still looking at their face, which is hard.  I usually end up staring at them really hard and completely lose them, or look at them with dead eyes.  If I can't aimlessly look around, I just need to aimlessly look in one place. 
Now the weird thing is that when I say "dead eyes", it doesn't look like it.  To me, I literally do as described above.  But to everyone else who doesn't know, it looks like I'm actively showing full attention and engagement of the conversation. 

Over the years I've gotten better at balancing my attention and face studying. 

If you know me, sorry if I don't look at you when speaking.  I'm just trying to pay attention.