IDPs leave city for "ancestral homes"