For me, it matters a lot. A restaurant doesn't have to be high design (thought I love a thoughtfully designed space) for it to work. If it has atmosphere, I'm just as appreciative.
I love the decor of Starlite, which I consider high design. But I also love the atmosphere at Ranchos Cocina in North Park - it's fun and kitschy.
If the food is amazing, decor matters less to me, but I definitely still notice it. If the food is just so-so and the decor is out of this world, I tend to have a positive experience.
I'm not a hard-core foodie. My sense is that for the foodie set, it's the food above all else, decor be damned. And I'm sure there are people who don't care at all what a restaurant looks like.
Which one are you? Does the decor of a restaurant matter to you? What restaurants in San Diego have the best design and decor? What ones have the worst? Let's talk.