Mail Protocols

 Simple Mail Transfer Protocol (SMTP) is an Internet standard for electronic mail (e-mail) transmission across Internet Protocol (IP) networks. SMTP was first defined in RFC 821 (STD 15) (1982)[1], and last updated by RFC 5321 (2008)[2] which includes the extended SMTP (ESMTP) additions, and is the protocol in widespread use today. SMTP is specified for outgoing mail transport and uses TCP port 25.

While electronic mail servers and other mail transfer agents use SMTP to send and receive mail messages, user-level client mail applications typically only use SMTP for sending messages to a mail server for relaying. For receiving messages, client applications usually use either the Post Office Protocol (POP) or the Internet Message Access Protocol (IMAP) to access their mail box accounts on a mail server.

Various forms of one-to-one electronic messaging were used in the 1960s. People communicated with one another using systems developed for specific mainframe computers. As more computers were interconnected, especially in the US Government's ARPANET, standards were developed to allow users using different systems to be able to e-mail one another. SMTP grew out of these standards developed during the 1970s.

SMTP can trace its roots to two implementations described in 1971, the Mail Box Protocol, which has been disputed to actually have been implemented,[3] but is discussed in RFC 196 and other RFCs, and the SNDMSG program, which, according to RFC 2235, Ray Tomlinson of BBN "invents" for TENEX computers the sending of mail across the ARPANET.[4][5][6] Fewer than 50 hosts were connected to the ARPANET at this time.[7]

Further implementations include FTP Mail [8] and Mail Protocol, both from 1973.[9] The work continued throughout the 1970s, until the ARPANET converted into the modern Internet around 1980. Jon Postel then proposed a Mail Transfer Protocol in 1980 that began to remove the mail's reliance on FTP.[10] SMTP was published as RFC 821 in August 1982, also by Postel.

The SMTP standard was developed around the same time as Usenet, a one-to-many communication network with some similarities.

SMTP became widely used in the early 1980s. At the time, it was a complement to Unix to Unix Copy Program (UUCP) mail, which was better suited to handle e-mail transfers between machines that were intermittently connected. SMTP, on the other hand, works best when both the sending and receiving machines are connected to the network all the time. Both use a store and forward mechanism and are examples of push technology. Though Usenet's newsgroups are still propagated with UUCP between servers,[11] UUCP mail has virtually disappeared[12] along with the "bang paths" it used as message routing headers.

The article about sender rewriting contains technical background info about the early SMTP history and source routing before RFC 1123.

Sendmail was one of the first (if not the first) mail transfer agents to implement SMTP.[citation needed] Some other popular SMTP server programs include Postfix, qmail, Novell GroupWise, Exim, Novell NetMail, Microsoft Exchange Server, Sun Java System Messaging Server.

Message submission (RFC 2476) and SMTP-AUTH (RFC 2554) were introduced in 1998 and 1999, both describing new trends in e-mail delivery. Originally, SMTP servers were typically internal to an organization, receiving mail for the organization from the outside, and relaying messages from the organization to the outside. But as time went on, SMTP servers (Mail transfer agents), in practice, were expanding their roles to become message submission agents for Mail user agents, some of which were now relaying mail from the outside of an organization. (e.g. A company executive wishes to send e-mail while on a trip using the corporate SMTP server.) This issue, a consequence of the rapid expansion and popularity of the World Wide Web, meant that the SMTP protocol had to include specific rules and methods for relaying mail and authenticating users to prevent abuses such as unsolicited e-mail (spam) relaying.

As this protocol started out purely ASCII text-based, it did not deal well with binary files. Standards such as Multipurpose Internet Mail Extensions (MIME) were developed to encode binary files for transfer through SMTP. Mail transfer agents (MTAs) developed after Sendmail also tended to be implemented 8-bit-clean, so that the alternate "just send eight" strategy could be used to transmit arbitrary text data (in any 8-bit ASCII-like character encoding) via SMTP. 8-bit-clean MTAs today tend to support the 8BITMIME extension, permitting binary files to be transmitted almost as easily as plain text.

Many people contributed to the core SMTP specifications, among them Jon Postel, Eric Allman, Dave Crocker, Ned Freed, Randall Gellens, John Klensin, and Keith Moore.

In computing, the Post Office Protocol (POP) is an application-layer Internet standard protocol used by local e-mail clients to retrieve e-mail from a remote server over a TCP/IP connection. POP and IMAP (Internet Message Access Protocol) are the two most prevalent Internet standard protocols for e-mail retrieval. Virtually all modern e-mail clients and servers support both. The POP protocol has been developed through several versions, with version 3 (POP3) being the current standard.

The design of POP and its procedures supports end-users with temporary Internet connections, such as dial-up access, allowing these users to retrieve e-mail when connected and then to view and manipulate the retrieved messages when offline. Although most clients have an option to leave mail on server, e-mail clients using POP generally connect, retrieve all messages, store them on the user's PC as new messages, delete them from the server, and then disconnect.

In contrast, the newer, more capable Internet Message Access Protocol (IMAP) supports both connected (online) and disconnected (offline) modes of operation. E-mail clients using IMAP generally leave messages on the server until the user explicitly deletes them. This and other aspects of IMAP operation allow multiple clients to access the same mailbox. Many e-mail clients support POP as well as IMAP to retrieve messages; however, fewer Internet Service Providers (ISPs) support IMAP.

Clients with a leave mail on server option generally use the POP3 UIDL (Unique IDentification Listing) command. Most POP3 commands identify specific messages by their ordinal number on the mail server. This creates a problem for a client intending to leave messages on the server, since these message numbers may change from one connection to the server to another. For example if a mailbox contained five messages, and a different client then deletes message #3, the next connecting user will find the last two messages' numbers decremented by one. UIDL provides a mechanism to avoid these numbering issues by assigning a string of characters as a permanent and unique ID for the message. When a POP3-compatible e-mail client connects to the server, it can use the UIDL command to get the current mapping from these message IDs to the ordinal message numbers. The client can then use this mapping to determine which messages it has yet to download. IMAP uses a 32-bit unique identifier (UID) that is assigned to messages in ascending (although not necessarily consecutive) order as they are received. When retrieving new messages, an IMAP client requests the UIDs greater than the highest UID among all previously-retrieved messages, whereas a POP client must fetch the entire UIDL map. For large mailboxes, this can require significant processing.

Irrespective of the mail retrieval protocol, e-mail clients typically use the Message Submission flavor of the Simple Mail Transfer Protocol (SMTP) to send messages. POP3 contains a protocol extension, known as "XTND XMIT", that allows clients to transmit outbound mail. The Qualcomm qpopper and CommuniGate Pro servers and Eudora clients are examples of systems that optionally utilize the XTND XMIT methods of authenticated client-to-server e-mail transmission.

MIME serves as the standard for attachments and non-ASCII text in e-mail. Although neither POP3 nor SMTP require MIME-formatted e-mail, essentially all Internet e-mail comes MIME-formatted, so POP clients must also understand and use MIME. IMAP, by design, assumes MIME-formatted e-mail.

Like many other older Internet protocols, POP originally supported only an unencrypted login mechanism. Although plain text transmission of passwords in POP still commonly occurs, POP3 currently supports several authentication methods to provide varying levels of protection against illegitimate access to a user's e-mail. One such method, APOP, uses the MD5 hash function in an attempt to avoid replay attacks and disclosure of the shared secret. Clients implementing APOP include Mozilla Thunderbird, Opera, Eudora, KMail, Novell Evolution, RimArts' Becky!,[1] Windows Live Mail, PowerMail, and Mutt. POP3 clients can also support SASL authentication methods via the AUTH extension. MIT Project Athena also produced a Kerberized version.

POP uses the Transmission Control Protocol on port number 110. Transmission may be encrypted using Transport Layer Security (TLS) or Secure Sockets Layer (SSL). This is negotiated in the POP3 protocol using the STLS command. Some clients and servers, such as Google Gmail, instead use the deprecated alternate-port method, which uses TCP port 995 (POP3S).

The Internet Message Access Protocol (IMAP) is one of the two most prevalent Internet standard protocols for e-mail retrieval, the other being the Post Office Protocol (POP).[1] Virtually all modern e-mail clients and mail servers support both protocols as a means of transferring e-mail messages from a server.

The Internet Message Access Protocol (commonly known as IMAP, and previously called Internet Mail Access Protocol, Interactive Mail Access Protocol (RFC 1064), and Interim Mail Access Protocol[2]) is an Application Layer Internet protocol that allows an e-mail client to access e-mail on a remote mail server. The current version, IMAP version 4 revision 1 (IMAP4rev1), is defined by RFC 3501.

IMAP supports both on-line and off-line modes of operation. E-mail clients using IMAP generally leave messages on the server until the user explicitly deletes them. This and other characteristics of IMAP operation allow multiple clients to manage the same mailbox. Most e-mail clients support IMAP in addition to POP to retrieve messages; however, fewer Internet service providers (ISPs) support IMAP.[3] IMAP offers access to the mail store. Clients may store local copies of the messages, but these are considered to be a temporary cache.

E-mail messages are sent to an e-mail server that stores messages in the recipient's email box. The user retrieves messages with an e-mail client that uses one of a number of e-mail retrieval protocols. Some clients and servers preferentially use vendor-specific, proprietary protocols, but most support the Internet standard protocols, SMTP for sending e-mail and POP and IMAP for retrieving e-mail, allowing interoperability with other servers and clients. For example, Microsoft's Outlook client uses a proprietary protocol to communicate with an Microsoft Exchange Server server as does IBM's Notes client when communicating with a Domino server, but all of these products also support POP, IMAP, and outgoing SMTP. Support for the Internet standard protocols allows many e-mail clients such as Pegasus Mail or Mozilla Thunderbird (see comparison of e-mail clients) to access these servers, and allows the clients to be used with other servers (see list of mail servers).

IMAP was designed by Mark Crispin in 1986 as a remote mailbox protocol, in contrast to the widely used POP, a protocol for retrieving the contents of a mailbox.[4]

[edit]Original IMAP

The original Interim Mail Access Protocol was implemented as a Xerox Lisp machine client and a TOPS-20 server.

No copies of the original interim protocol specification or its software exist. Although some of its commands and responses were similar to IMAP2, the interim protocol lacked command/response tagging and thus its syntax was incompatible with all other versions of IMAP.

[edit]IMAP2

The interim protocol was quickly replaced by the Interactive Mail Access Protocol (IMAP2), defined in RFC 1064 and later updated by RFC 1176. IMAP2 introduced command/response tagging and was the first publicly distributed version.

[edit]IMAP2bis

With the advent of MIME, IMAP2 was extended to support MIME body structures and add mailbox management functionality (create, delete, rename, message upload) that was absent in IMAP2. This experimental revision was called IMAP2bis; its specification was never published in non-draft form. Early versions of Pine were widely distributed with IMAP2bis support (Pine 4.00 and later supports IMAP4rev1).

[edit]IMAP4

An IMAP Working Group formed in the IETF in the early 1990s and took over responsibility for the IMAP2bis design. The IMAP WG decided to rename IMAP2bis to IMAP4 to avoid confusion with a competing IMAP3 proposal from another group that never got off the ground.[citation needed] The expansion of the IMAP acronym also changed to the Internet Message Access Protocol.

Some design flaws in the original IMAP4 (defined by RFC 1730) that came out in implementation experience led to its revision and replacement by IMAP4rev1 two years later. There were very few IMAP4 client or server implementations based on RFC 1730 due to its short lifetime.

Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of e-mail to support:

Text in character sets other than ASCII

Non-text attachments

Message bodies with multiple parts

Header information in non-ASCII character sets

MIME's use, however, has grown beyond describing the content of e-mail to describing content type in general, including for the web (see Internet media type).

Virtually all human-written Internet e-mail and a fairly large proportion of automated e-mail is transmitted via SMTP in MIME format. Internet e-mail is so closely associated with the SMTP and MIME standards that it is sometimes called SMTP/MIME e-mail.[1]

The content types defined by MIME standards are also of importance outside of e-mail, such as in communication protocols like HTTP for the World Wide Web. HTTP requires that data be transmitted in the context of e-mail-like messages, although the data most often is not actually e-mail.

MIME is specified in six linked RFC memoranda: RFC 2045, RFC 2046, RFC 2047, RFC 4288, RFC 4289 and RFC 2049, which together define the specifications.

MIME is extensible. Its definition includes a method to register new content types and other MIME attribute values.

The goals of the MIME definition included requiring no changes to existent e-mail servers and allowing plain text e-mail to function in both directions with existing clients. These goals were achieved by using additional RFC 822-style headers for all MIME message attributes and by making the MIME headers optional with default values ensuring a non-MIME message is interpreted correctly by a MIME-capable client. A simple MIME text message is therefore likely to be interpreted correctly by a non-MIME client although if it has e-mail headers the non-MIME client won't know how to interpret. Similarly, if the quoted printable transfer encoding (see below) is used, the ASCII part of the message will be intelligible to users with non-MIME clients.

An email client, email reader, or more formally mail user agent (MUA), is a computer program used to manage email [1].

Specifically, the term email client may refer to any agent acting as a client toward an email server, regardless of it being a mail user agent, a relaying server, or a human typing on a terminal. In addition, a web application providing message management, composition, and reception functionality is sometimes considered an email client.

Retrieving messages from a mailbox

Like most client programs, an MUA is only active when a user runs it. Messages arrive on the Mail Transfer Agent (MTA) server. Unless the MUA has access to the server's disk, messages are stored on a remote server and the MUA has to request them on behalf of the user.

In the first case, shared disk, a user logs on a server and runs an MUA on that machine. The MUA reads messages from a conventionally formatted storage, typically mbox, within the user's HOME directory. The MTA uses a suitable mail delivery agent (MDA) to add messages to that storage, possibly in concurrence with the MUA. This is the default setting on many Unix systems. Webmail applications running on the relevant server can also benefit from direct disk access to the mail storage.

For personal computing, and whenever messages are stored on a remote system, a mail user agent connects to a remote mailbox to retrieve messages. Access to remote mailboxes comes in two flavors. On the one hand, the Post Office Protocol (POP) allows the client to download messages one at a time and only delete them from the server after they have been successfully saved on local storage. It is possible to leave messages on the server in order to let another client download them. However, there is no provision for flagging a specific message as seen, answered, or forwarded, thus POP is not convenient for users who access the same mail from different machines or clients. On the other hand, the Internet Message Access Protocol (IMAP) allows users to keep messages on the server, flagging them as appropriate. IMAP provides sub-folders. Typically, the Sent, Drafts, and Trash folders are created by default.

Both POP and IMAP clients can be configured to access more mailboxes at the same time, as well as to check each mailbox every given number of minutes. IMAP features an idle extension for real time updates, providing faster notification than polling where long lasting connections are feasible.

Client settings require the server's name or IP address, and the user name and password for each remote incoming mailbox.

[edit]Formatting messages

Mail user agents usually have built-in the ability to display and edit text. Editing HTML text is a popular feature. Invoking an external editor may be an alternative.

MUAs responsibilities include proper formatting according to RFC 5322 for headers and body, and MIME for non-textual content and attachments. Headers include the destination fields, To, Cc, and Bcc, and the originator fields From which is the message's author(s), Sender in case there are more authors, and Reply-To in case responses should be addressed to a different mailbox. To better assist the user with destination fields, many clients maintain one or more address books and/or are able to connect to an LDAP directory server. For originator fields, clients may support different identities.

Client settings require the user's real name and email address for each user's identity, and possibly a list of LDAP servers.

[edit]Submitting messages to a server

As a basic function, an MUA is able to introduce new messages in the transport system. Typically, it does so by connecting to either an MSA or an MTA, two variations of the SMTP protocol. The client needs to put a message quickly without worrying about where the message eventually will be delivered: that's why a transport system exists. Thus it always connects to the same preferred server, however, how does that server know that it should accept and relay submissions from that client? There are two ways. The older method recognizes the client's IP address, e.g. because the client is on the same machine and uses internal address 127.0.0.1, or because the client's IP address is controlled by the same internet service provider that provides both internet access and mail services. The newer method, since the SMTP protocol has an authentication extension, is to authenticate. The latter method eases modularity and nomadic computing.

Client settings require the name or IP address of the preferred outgoing mail server, the port number (25 for MTA, 587 for MSA), and the user name and password for the authentication, if any. There is a non-standard port 465 for SSL encrypted SMTP sessions, that many clients and servers support for backward compatibility. Transport Layer Security encryption can be configured for the standard ports, if both the client and the server support it.

Outlook Express is an e-mail/news client that is included with Internet Explorer versions 4.0 through 6.0. As such, it is also bundled with several versions of Microsoft Windows, from Windows 98 to Windows XP, and is available for Windows 95 and Mac OS 9. In Windows Vista, Outlook Express was superseded by Windows Mail, then again by Windows Live Mail as separate software in Windows 7. Microsoft Entourage, sold as part of Microsoft Office for Macintosh, has replaced the Macintosh version.

Outlook Express is a different application from Microsoft Office Outlook. The two programs do not share a common codebase, but do share a common architectural philosophy. The similar names lead many people to incorrectly conclude that Outlook Express is a stripped-down version of Microsoft Office Outlook. Outlook Express uses the Windows Address Book to store contact information and integrates tightly with it. On Windows XP, it also integrates with Windows Messenger.

Microsoft Internet Mail and News was a freeware e-mail and news client and ancestor of Outlook Express. Version 1.0 was released in 1996 following the Internet Explorer 3 release.[1] This add-on precedes the Internet Mail profile for Microsoft Exchange 4.0 bundled in Windows 95. Version 2.0 was released at the end of 1996. In 1997 the program was changed and renamed as Outlook Express and bundled with Internet Explorer 4. The executable file for Outlook Express, msimn.exe, is a holdover from the Internet Mail and News era. Internet Mail and News handled only plain text and rich text (RTF) e-mail, lacking HTML e-mail.

At one point, in a later beta version of Outlook Express 5, Outlook Express contained a sophisticated and adaptive spam filtering system; however this feature was removed shortly before launch. It was speculated on various websites and newsgroups at that time, that the feature was not stable enough for the mass market. Nearly two years later, a similar system, using a similar method of adaptive filtering, appeared as a feature of Microsoft Office Outlook.

Internet Explorer 5 required Outlook Express 5 to save Web Archive files (see MHTML).[2]

Outlook Express 6 SP3 is the latest version which is part of Windows XP SP3. Extended support for Windows XP SP3, which covers security hotfixes, will end in 2014.

In October 2005, Microsoft announced that Windows Vista would exclusively include a new application named Windows Mail, based on large parts of Outlook Express source code.[3] Microsoft was also concurrently developing Windows Live Mail Desktop (later renamed to Windows Live Mail), a mail client for its Hotmail service. Windows Live Mail was released in November 2007. In the future, support for Outlook Express and Windows Mail will be discontinued in favor of Windows Live Mail.

With no encryption, much like for postcards, email activity is plainly visible by any occasional eavesdropper. Email encryption enables to safeguard privacy by encrypting the mail sessions, the body of the message, or both. Without it, anyone (examples: the government (warrantless wiretapping, great firewall of China), fellow wireless network users such as at an Internet cafe or other public network, whether the network is open or not) with network access and the right tools can monitor email and obtain login passwords.

[edit]Encryption of mail sessions

All relevant email protocols have an option to encrypt the whole session. Remarkably, those options prevent a user's name and password from being sniffed, therefore they are recommended for nomadic users and whenever the internet access provider is not trusted. On sending mail, users can only control encryption at the hop from a client to its configured outgoing mail server. At any further hop, messages may be transmitted with or without encryption, depending solely on the general configuration of the transmitting server and the capabilities of the receiving one.

Encrypted mail sessions deliver messages in their original format, i.e. plain text or encrypted body, on a user's local mailbox and on the destination server's. The latter server is operated by an email hosting service provider, possibly a different entity than the internet access provider currently at hand.

[edit]Encryption of the message body

There are two models for managing cryptographic keys. S/MIME employs a model based on a trusted certificate authority (CA) that signs users' public keys. OpenPGP employs a somewhat more flexible web of trust mechanism that allows users to sign one another's public keys. OpenPGP is also more flexible in the format of the messages, in that it still supports plain message encryption and signing as they used to work before MIME standardization.

In both cases, only the message body is encrypted. Headers, including originator, recipients, and subject, remain in plain text....

[edit]Standards

While popular protocols for retrieving mail include POP3 and IMAP4, sending mail is usually done using the SMTP protocol.

Another important standard supported by most email clients is MIME, which is used to send binary file email attachments. Attachments are files that are not part of the email proper, but are sent with the email.

Most email clients use an X-Mailer header to identify the software used to send the message. According to RFC 2076, this is a common but non-standard header.

RFC 4409, Message Submission for Mail, details the role of the Mail submission agent.

RFC 5068, Email Submission Operations: Access and Accountability Requirements, provides a survey of the concepts of MTA, MSA, MDA, and MUA. It mentions that "Access Providers MUST NOT block users from accessing the external Internet using the SUBMISSION port 587" and that "MUAs SHOULD use the SUBMISSION port for message submission."

[edit]Port numbers

Email servers and client use the following TCP port numbers by default, unless configured for specialized installations:

protocol use plain text or encrypt sessions plain text sessions only encrypt sessions only

POP3 incoming mail 110 995

IMAP4 incoming mail 143 993

SMTP outgoing mail 25 (unofficial[2]) 465

MSA outgoing mail 587

HTTP webmail 80 443

Note that while webmail obeys the earlier HTTP disposition of having separate ports for encrypt and plain text sessions, mail protocols use the STARTTLS technique, thereby allowing encryption to start on an already established TCP connection. RFC 2595 discourages the use of the previously established ports 993 and 995.

Windows Address Book is a component of Microsoft Windows that lets users keep a single list of contacts that can be shared by multiple programs. It is most commonly used by Outlook Express. It was introduced with Internet Explorer 3 in 1996 and improved in subsequent versions. [1] The Windows Address Book API can query LDAP servers or read/write data to a local .wab file. In Windows Vista, Windows Address Book was replaced with Windows Contacts.

The Windows Address Book is an application that has a local database and user interface for finding and editing information about people, making it possible to query network directory servers using Lightweight Directory Access Protocol. Other applications can also use the WAB. Microsoft Office Outlook uses its own PST store for email messages. However, it can import contacts from the .WAB format. Microsoft Exchange uses .PAB file for personal address book.

[edit]Features

Can store comprehensive contact information in tabs including personal and business information

Integrates with Outlook Express.

Programmable API to work with other applications. Applications can also extend functionality such as adding more tabs and fields to store additional custom information, or customizing the toolbar.

Can store contacts inside contact groups and folders.

Can selectively send email to contacts only in plain text for additional security.

Can export and import cards to and from vCard 2.1 and CSV formats. Can also import from LDIF and other formats.

Search for entries in the contact database, arrange contacts by first name or last name.

Automatically add contacts from received email.

Prints contact lists as memo, business card or phone list styles.