Obviously the clear winner is iTunes, although it is a unique directory that is different from any other directory. It has major advantage though because of its integration with iTunes.
iTunes are the media player for iPod users and also many other Windows- and Mac-based users.
If you do a quick research using Google, you should find out quickly that each podcaster seems to recommend different podcast directories. There is no one or a handful directories that you should focus.
The fact is, you should submit your podcast to as many directories as you possibly can. Why is so? For a few reasons:
1. Podcast directories send referrals
Although you should not depend on them for traffic, you will get a few constantly. Most of the traffic still is going to come from search engines or other popular sites that endorse you.
However, considering the benefits of getting listed in those directories makes this effort worthwhile.
2. Podcast directories give your site quality inbound links
These are very important to determine the relevance of your site for certain keywords. This is a reason why you should put primary keyword in the podcast title because the directories are going to link to you with that name, most of the time.
Links from these directories count toward your link popularity. It transfers Google PR and are votes for your site for better search engine placement.
3. Branding value
Being able to say that your podcast is in iTunes is worth mentioning if people you are networking with are using iTunes. They may think that you are doing something cool.
By getting in front of as many audience as possible, you also get the brand value. Best of all, these are all for free.
In summary, although there are not major traffic from podcast directories yet, you should spent some time to get listed in as many of them as possible for other reasons I mentioned above.
If you only have the time to submit to a few, check the traffic of the directories through Alexa and the PageRank. Focus on those with high traffic and PageRank to get the most out of your effort.