Goku is implied to be on a whole another level compared to Vegeta in the Boo Arc, since Vegeta realizes that he's outclassed when he discovers that SSj2 Goku (against Yakon) was holding back his power and he was still above him (Vegeta).
So, even if Vegeta could win, it wouldn't be with as much ease as Goku. In fact, I'm inclined to say that Vegeta would likely lose.
Pre-Majin Vegeta is rather insignificant when it comes to Super Saiyan 3 tier. He'd put up a decent fight but it'd go similar to his manga fight in my mind, only Boo would beat him even worse.
I'm of the opinion that you need to be stronger than Boo to actually destroy him. Being in a form such as SSJ3 will not help Vegeta here, especially considering that he will be decently weaker than Boo.
He's in for big trouble with a form as draining as Super Saiyan 3, and with his sizable inferiority to Goku, he'd fare no better than Goku did against Cell.
Hear me out here... Goku said he could beat Fat Boo if he really went for it. He'd have to do it quickly because of the time restaint on his day on Earth. He knew this, so he'd need a decent power advantage to beat Boo considering he has great recovery (regeneration).
I'm willing to believe Vegeta was way behind Goku pre-Majin boost. The boost must've been substantial because Babidi and Dabra chose Vegeta to fight for them. Sure, it could be because he was the only candidate, but also because he had good potential. The gap could be 90-110 anyway, doesn't matter because Boo wins anyway.
Vegeta would definitely need the Majin boost to beat Boo in SSJ3.
I'd probably place Vegeta as being slightly superior power-wise, though even without the strain, his advantage probably wouldn't be enough to counter Boo's regeneration or Chocolate Beam. The only way I see this going well for him is if Boo's somehow stupid enough to let him pull of a Final Flash from a good angle.