Check the torque arm which fits between the rear of the engine and the swingarm pivot.
If you can see the engine move when you pull the chain down, or push down on the gear lever and see the chain get slack, that’ll be your trouble.
The bushes at each end wear and gives the horrible gear change you describe, mine were real bad, I had some new bushes made up as they’re NLA I’m told.
Good luck!