Yeah but before meeting Luffy and before Luffy ate the "gomu gomu no mi" Shanks had no reason to remove the contract because Roger was dead, Roger's son Ace was hidden by Garp and Shanks knew there is gonna be a great flood that will engulf the world so he might've accepted Imu's "gift" of immortality and some extra power after all we know Shanks returned to the "Holy Land" in the past but chose to go back at sea.

When Luffy ate the devil fruit, Shanks saw how free spirited he is which it reminded him of Roger (plus the fact that Luffy is a member of the D "clan"), Shanks by his own words "put a bet" on the next generation. 

The bet might have been on Luffy awakening the hito hito no mi model nika, that's why he have up the contract and hope that Luffy was the next comming of nika, he had to give up the contract to not be summoned by Imu and forced to kill Luffy once he awakened the devil fruit