First, see if it monitor related. I tracked down my buddies problem to a faulty VGA port on his monitor. Luckily his monitor (an LCD) had both a VGA and DVI port. Why he was not using DVI in the first place I do not know. Once he switched over to DVI, his graphics problems were cured. Most monitor problems are perceived as missing color issues.

If your monitor is good to go but you still are having problems it is almost certain you are having issues with your video card. You will know if it is the video card if you get on-screen artifacting while playing games or even in 2d mode. Meaning, if your screen shows pixelated abnormalities during your boot up, on your desktop, and especially within games then its the card. Try reseating the card onto the motherboard. Make sure the fan that is used to cool the card is operating. Also, sometimes the only viable way to troubleshoot your problem is to start swapping out parts. If you or a friend has a spare video card laying about, plug that in and see if that does cure the problem.

First off lets talk about the last part of your question, composite cables commonly referred to as "RCA cables." The red and white cable transmits audio and the yellow transmit video. RCA cables do not have the bandwidth to transmit any type of high definition output. Component cables, which is the blue, green, and red have the ability on the Xbox 360 to transmit hi-def picture, albeit only at 720p. If you want to ultilize your televisions 1080p output, you have to either have a 360 with the HDMI port built in or use the "VGA" connection for the 360 which is sold seperately.

I myself do not have a 1080p television so I ultilize the component cable that came with the 360 to see video in 720p which is still high definition and visually stunning. Hope I could help.

If you can find a PS3 on your local retail shelf.   Who in Sony can I talk to about claiming my 1200 bucks!

http://www.dailytech.com/article.aspx?newsid=6050