I also have problems with anxiety with driving.
I was a very confident driver for most of my long life. I enjoyed my cars and my driving and the wind in my hair. Then, out of the blue one day, I experienced a panic attack at a red light with my two little children in the car. I thought I was dying. It was a long time ago now and I rode through it. Here I am about 17 years later and I am back in the driving panic zone again.
It seems to me that there's usually a trigger. The one terrible, terrible time which starts the ongoing anxiety.
I also have vertigo problems resulting from a fall and subsequent concussion some years ago now. I'm not sure how much that has to do with my fear of driving these days though. It's more my life-long battle with anxiety issues. My vertigo affects my ability to actually get to the car which is parked down the slippery drive and out on the road more than it affects my ability to drive around my streets here.
I do have a pattern these days. I do find it easier to drive the familiar. (similar to how you describe). I actually force myself to go driving. Once I'm in the car in the "familiar", it's so liberating. I'm surrounded by beautiful beaches here, so I do what I call a "surf check". Keeps me motivated. However, it wouldn't keep me in work if I needed to drive on highways or freeways.
I'm not depressed either, but I would really love to get back on the highway and drive as far as I could. I just can't. Yet!
I think the main problem after experiencing this type of anxiety is the actual avoidance. It gets to feel so safe. I have this book called "Feel the fear and do it anyway". Well, that was fine for many years but I'm presently having difficulty just attempting to do it.
Once we begin to avoid, it's a huge signal, I feel, that we need some professional help. All my life I was able to work through my fears on my own and I got there. Once I began to actually "avoid", then things took a downward turn. Sometimes we need to force ourselves back out into the uncomfortable in order to keep our lives on track.
I do not believe a person who may have anxiety/panic issues is necessarily "depressed". The anxiety/panic issues can however result in a depressed state of mind I personally feel. It's because the anxiety can limit our lifestyle so much once it takes control. Just how I see it.
Avoidance is a huge red flag. It can become too safe. You have so much to do and to live for. Seek professional help now. Please also check out the Anxiety/phobia info. at PsychCentral if you can. http://psychcentral.com/disorders/anxiety/phobias.html
There may be something there that is helpful for you. Good luck with it all.