Lately, I've had the odd experience of being called "literary." My recent story on fantasy-writers has received comments about allusions, thinking on a higher plane of thought, the whole thing being unconventional ...
... and I think, "What?"
There was no grand plan or structure behind the story. I just poured a few concepts into a mythology pool and splashed around like a hyperactive three year old. I worked to make sure it was cohesive, of course, but I never set out to have a specific message ... though it did amuse me to work in some irony that might be hard to catch if you're not very conversant with the Greek mythos.
To all this, I don't understand "literary." To me, literary is more about an intent than a result: you set out to write something that is educational, illuminating or, at least, more than "just entertainment." And for me, it's always been exclusively about the story. I have fun (a lot of it); I just hope the reader does, too.
Part of it may be the Greek mythology aspect. My issue is that I practically teethed on Greek myths - D'aulaire's Book of Greek Myths is one of the first books I can remember reading obsessively - and I don't think of them as classical literature, much less intimidating. To me, they're comfortable, snuggly stories - just like a fairy tale that's been Disney-ized. So it's really hard for me to understand some people's approach to them.
I try not to be insulted by "literary," honestly, because I know people don't mean it as an insult ... but the whole idea is counter to my writing philosophy. So it really does feel like a slap.