Thats not strictly true, there should be an e-clip on the shaft that goes behind the seal, therefore the seal does need removing to either remove or replace the e-clip. If you just pull the shaft out and the seal does not come out then the e-clip is either not fitted or is stuck behind the seal. It needs to be put back on to hold the shaft in the correct place.

Pulling the shaft out with a puller is best method (tyga do a great tool for this), my guess is that if you still have the clip on the shaft then it will pull the seal out with it. Then your right, you can just get the valves out from the exhaust port.