Firefox isn’t a panacea. I got spyware installed on a box surfing with firefox. A java applet installed spyware that affects internet explorer. I think it is more a vuln. in java and windows, not really firefox or ie. It really shows you that ActiveX isn’t the only thing that has issues as far as installing things on your box when surfing.
I think that you need to worry about security no matter if you use IE or FF or anything else. :)