Hello evolution.
The Guitar/Bass:
The lower 5 buttons with the original top 5 add a more realistic feel to the gameplay. I think the gameplay on Rock Band is much easier than Guitar Hero. If you can play beat GH2 on expert you should have no problem knocking down Rock Band's setlist (which is wayyyy better in my opinion) on guitar or bass. It just might take a little bit to get used to the thinner note icons on the screen and the lower 5 buttons.
The Microphone:
On easy, it is almost impossible to fail... on expert it takes some pipes. The only real thing you need to know is that pitch and the overall melody of your voice will have to be improved over each difficulty. I'd advise playing on a lower difficulty for songs you don't know as well and higher ones for songs you're very familiar with.
The Drums:
The drums were very hard to get used to for me, but not too difficult to master. The color pattern for the drums is different (red, yellow, blue, green), there's the bass drum you have to stomp quite quite often, and star power is engaged by hitting green at the end of a drum solo. But the amazing thing about the drums are that while simulating guitar, you don't get near the actual experience of a real guitar and it's not near the same thing, but for drums and vocals, it is.
If you are going to buy Rock Band i wouldn't buy GH3, and if you're going to buy GH3 I wouldn't "recommend" buying Rock Band. because they are just a different experience all together and i do believe one of the goals was to make it harder to play GH3 after playing Rock Band for so long to capture more Rock Band junkies.