I think it also relates to math ability. Rhythms seem to be a problem for people. It's not the notes, it's the rhythm. People get lost. Also having the beat in the back of your head that allows you to know where in the measure you are is huge. I've been in so many groups where a difficult passage means the player slows down and is immediately lost.
And then there is the reading text issue related thing. When you read text, do you look at all the letters? Of course not. You read the sentence. Same thing in music. You read the phrases. You see the whole line. No?
Or are you thinking of how to produce the sound and playing and can't spare attention to the music?
I think there's something like that going on?