You probably need to also switch the green wire and maybe the blue wire too for good measure.

This will require a different switch that can switch 3 different lines at once, 3PST instead of the SPST you have now.