So I did what any self-respecting real estate blogger would do, I looked up the city regulations! Guess what? I think he is right. Most fencing companies request neighbor approval and some law offices will help you file a damage claim if you feel wronged by a neighbor's fence but there does not seem to be any laws regarding neighbor rights to property-line fencing. Permits are required only if a fence is over 8 ft. tall. If your neighbor wants to tear down his fence, he doesn't have to check with you first. It might not be neighborly, or ethical, but it doesn't appear to be against any regulations.
My neighbor says he will be rebuilding “his fence” on his own timetable, but I'm not relying on his fence this time. I put up my own temporary fence already and will contract for another soon. If you've ever seen a yard with a double fence – now you know why.