Yeah, typical.
Why let the tree establish?
And isn't it funny, you talking to him about the tree and your concerns isn't the first sign of trouble.
I asked my neighbour about his tree, answer was similar, I'll be dead before that breaks your wall. There you go, considerate eh. Funny if then the tree is protected and I incur additional costs to design around his tree.
So I take it you also pulled their vine off your wall and they didn't like that either, if it was me I would have poisoned it (the vine).
We have creeping vine coming from the crap hole behind us, I spray it with round up ... works a treat.