I've had this problem with Firefox for a while: if something is supposed to open in a pop-up window, it opens in the current tab instead. For example, when I'm previewing a new entry on LJ, and I press the "Preview" button, it loads the entry preview in the same tab I typed the entry in. As a result, I have to remember to navigate back and then click "Post" or I lose my entry. :( Other annoyances have occurred because of this as well.
I've been playing around with my Firefox settings tonight (particularly on Tab Mix Plus), and although it looks to me like I've told it NOT to do that, it's STILL doing it.