You must have seen that the Internet Explorer’s Tabbed Browsing Settings dialog box has an option called “Let Internet Explorer decide how pop-ups should open” which is explained best by the quote below
Basically what this will do is open any pop-up without size or other restrictions in a new tab, the rest will open in a new window. For users experienced with tabbed browsing this is typically the best option, as it allows most windows to open in tabs, but allows pop-up style windows to open in a new window.
[Source: IEBlog: Your Tab Settings…]
However users of Firefox will find that such an option does not exist in the Firefox’s Options or Preferences dialog. The closest option that is there would be “New pages should be opened in: (a) a new window (b) a new tab” but this option does not make the distinction between pop-up style windows and normal windows.
There is a hidden preference in Firefox which will make this distinction. It is browser.link.open_newwindow.restriction and you can find it in the about:config page. Just type “about:config” followed by the Enter key in the address bar.
Filter the page by typing in open_new or restrict. Then double-click the preference and enter the value 2, which makes Firefox open normal windows as new tabs, while opening pop-up style windows as real pop-ups.
Other values are 0 which follows the above mentioned closest option, and 1 which ignores the same option (all pop-ups open in a new window regardless of what you set).
[Reference: MozillaZine Knowledge Base Article]