Outgoing mail function is used when you send out the group emails to its members. Once you have chosen an option for this and you want test outgoing email settings, save mailing group and go to Troubleshooting tab on left sidebar. Send an email using the form there. Error or success message will be displayed after form is submitted. So you will know if its working correctly or not.
If you select SMTP option, make sure to put right SMTP server name, username and password credentials for it.