The players/singers know what to do.
What is the purpose of someone standing up front flailing their arms around?
Also, are all conductors male?
Lots and lots of reasons. A larger ensemble needs a conductor to impart precision in tempi, attacks and releases, measure dynamics (relative loudness or softness of the various sections). A good conductor can instill style, grace, and confidence.
And a really good conductor can scare the hell out of a musician (j/k, but Toscanini wasn't noted for his inviting folks over for tea and crumpets). In fact, tyrannical conductors were all the rage for a very long time -- not so much today, but up to 40 years ago or so, they were still around.
Some very, very good conductors today include Gustavo Dudamel (he has a wonderfully clear, exciting style, very optimistic) who conducts the LA Philharmonic. Young guy too, born in 1981. Sir Simon Rattle conducts Berlin, and I think at least one other orchestra in England. You don't get to that level without really knowing your stuff.
Granted, there are fewer women conductors then men, but in the late 90s, I played trombone in an orchestra (DuPage Symphony) in the Chicago area. Barbara Schubert still conducts it today. I believe her husband plays viola.