There are lots of things to watch out & I don't know what mail delivery system you use but one thing is for sure - if you use double opt-in, results will be much better.
Also - if you use your own website/server to handle all this, it is possible that it's IP address has been banned by Yahoo, MSN etc., So it's better to get a dedicated mail delivery service, like aWeber.com