Well, we've had it happen a lot.
It's those top notch pidgeons usually ... but the adults are never far watching.
We have relocated the nest to another tree/place and observed the parents. Usually the parents are onto it and go to the new location.
If the customer is home we tell them to observe and if the parents dont feed or visit to take them to the AWL/RSPCA/WildLifeRescue people.