I literally went through this two days ago. You do have to go to the SS office and update your immigration status to remove the restriction. Your number will stay the same. I wish I could have changed mine cause my SS card was stolen years ago, but it's really hard (gotta bring police report, proof that it's being used by someone else and affecting you negatively, etc.)