For years I used a 3-way Composite and Stereo switcher to switch between 3 component sources. I put the Green into Composite, Blue in R and Red into L. This worked very well, incurred almost no loss, and cost me about ?4 from Maplin.
I have some reasonably decent kit and wouldn't have done it if the quality was crap. In fact I actually sent back a high end Component switcher because it induced analogue noise INTO the signals I was switching!
Comment