B Button is acting like its held down

By acting like its held down, I mean that its not stuck in the pressed position, but it does an action that would normally happen if I was holding it down.

I'll use a games as an example. Lets say Super Smash Bros. 3DS. I cannot go past the Select character screen because it keeps exiting the Menu.

This problem has escalated and has died down. One time, for about a week, my B, R, and Y Buttons had this problem, but the other two have been fixed.

Sometimes the problem fixes itself and I can play normally, but it always comes back. I would rather not send it to Nintendo, but if I have to I will.

I would really appreciate help.