That black bit definetely comes off the wheel, its 3 or 4 years since I done mine but I think it came away when I started to knock the bearings out.

If you cant shift them might be worth applying a bit of gentle heat with a hot air gun or something.