Those two pieces are supposed to rotate on each other. Remove the snap ring on the end of the trunion shaft. I think you also have to remove the E-clip that holds the other cam plate on, and then you can pull it off and clean up the area where the trunion shaft passes through.
More info
Neutral Adjustment