On an email server, an SMTP server receives messages. These messages have to be stored in the system by an MDA (such as procmail. At the same time, the POP3 or IMAP servers need to know where to retrieve messages from (and send them to the recipient’s own computer).
Most systems use files for the SMTP and POP3/IMAP to interact. The SMTP server writes to some files in the home directory of the recipient of the message. Then, when queried, the POP3/IMAP server reads and modifies these files and directories.
This means that the SMTP server and the POP3/IMAP server must agree on the same standard. There are two standards. mbox is an older standard, whereas Maildir is the newer standard. Many SMTP, POP3 and IMAP servers can be configured to use either one. However, the two standards are exclusive of each other. Once a particular standard is used, the other standard cannot be used.
mbox uses a single file to hold all the messages for a particular recipient. On the other hand, Maildir uses a single file for each message received. http://www.courier-mta.org/mbox-vs-maildir/ presents a fairly detailed analysis of the two formats. The general agreement is that Maildir is no worse than mbox in all cases. However, Maildir can be a lot more efficient than mbox in certain conditions.