email (Source)
The Email source allows you to receive events via emails. An
Email source can be configured using the imap or pop3 server
to receive events. This allows you to filter the messages that satisfy
the criteria specified under the search term option. The email
source parameters can be defined in either the
<SP_HOME>/conf/<PROFILE>/deployment yaml file or the stream
definition. If the parameter configurations are not available in either
place, the default values are considered (i.e., if default values are
available). If you need to configure server system parameters that are
not provided as options in the stream definition, they need to be
defined in the deployment yaml file under email source
properties. For more information about imap and pop3 server
system parameters, see the following. [JavaMail Reference
Implementation - IMAP
Store](https://javaee.github.io/javamail/IMAP-Store) [JavaMail
Reference Implementation - POP3 Store
Store](https://javaee.github.io/javamail/POP3-Store)
Syntax
CREATE SOURCE <NAME> WITH (type="email", map.type="<STRING>", username="<STRING>", password="<STRING>", store="<STRING>", host="<STRING>", port="<INT>", folder="<STRING>", search.term="<STRING>", polling.interval="<LONG>", action.after.processed="<STRING>", folder.to.move="<STRING>", content.type="<STRING>", ssl.enable="<BOOL>")
Query Parameters
| Name | Description | Default Value | Possible Data Types | Optional | Dynamic |
|---|---|---|---|---|---|
| username | The user name of the email account. e.g., gdnmail is the username of the gdnmail@gmail.com mail account. | STRING | No | No | |
| password | The password of the email account | STRING | No | No | |
| store | The store type that used to receive emails. Possible values are imap and pop3. | imap | STRING | Yes | No |
| host | The host name of the server (e.g., imap.gmail.com is the host name for a gmail account with an IMAP store.). The default value imap.gmail.com is only valid if the email account is a gmail account with IMAP enabled. | If store type is imap, then the default value is imap.gmail.com. If the store type is pop3, then the default value is pop3.gmail.com. | STRING | Yes | No |
| port | The port that is used to create the connection. | 993, the default value is valid only if the store is imap and ssl-enabled. | INT | Yes | No |
| folder | The name of the folder to which the emails should be fetched. | INBOX | STRING | Yes | No |
| search.term | The option that includes conditions such as key-value pairs to search for emails. In a string search term, the key and the value should be separated by a semicolon (;). Each key-value pair must be within inverted commas ( ). The string search term can define multiple comma-separated key-value pairs. This string search term currently supports only the subject, from, to, bcc, and cc keys. e.g., if you enter subject:DAS, from:carbon, bcc:gdn, the search term creates a search term instance that filters emails that contain DAS in the subject, carbon in the from address, and gdn in one of the bcc addresses. The string search term carries out sub string matching that is case-sensitive. If @ in included in the value for any key other than the subject key, it checks for an address that is equal to the value given. e.g., If you search for abc@, the string search terms looks for an address that contains abc before the @ symbol. | None | STRING | Yes | No |
| polling.interval | This defines the time interval in seconds at which th email source should poll the account to check for new mail arrivals.in seconds. | 600 | LONG | Yes | No |
| action.after.processed | The action to be performed by the email source for the processed mail. Possible values are as follows: FLAGGED: Sets the flag as flagged. SEEN: Sets the flag as read. ANSWERED: Sets the flag as answered. DELETE: Deletes tha mail after the polling cycle. MOVE: Moves the mail to the folder specified in the folder.to.move parameter. If the folder specified is pop3, then the only option available is DELETE. | NONE | STRING | Yes | No |
| folder.to.move | The name of the folder to which the mail must be moved once it is processed. If the action after processing is MOVE, it is required to specify a value for this parameter. | STRING | No | No | |
| content.type | The content type of the email. It can be either text/plain or text/html. | text/plain | STRING | Yes | No |
| ssl.enable | If this is set to true, a secure port is used to establish the connection. The possible values are true and false. | true | BOOL | Yes | No |
System Parameters
| Name | Description | Default Value | Possible Parameters |
|---|---|---|---|
| mail.imap.partialfetch | This determines whether the IMAP partial-fetch capability should be used. | true | true or false |
| mail.imap.fetchsize | The partial fetch size in bytes. | 16K | value in bytes |
| mail.imap.peek | If this is set to true, the IMAP PEEK option should be used when fetching body parts to avoid setting the SEEN flag on messages. The default value is false. This can be overridden on a per-message basis by the setPeek method in IMAPMessage. | false | true or false |
| mail.imap.connectiontimeout | The socket connection timeout value in milliseconds. This timeout is implemented by java.net.Socket. | infinity timeout | Any Integer value |
| mail.imap.timeout | The socket read timeout value in milliseconds. This timeout is implemented by java.net.Socket. | infinity timeout | Any Integer value |
| mail.imap.writetimeout | The socket write timeout value in milliseconds. This timeout is implemented by using a java.util.concurrent.ScheduledExecutorService per connection that schedules a thread to close the socket if the timeout period elapses. Therefore, the overhead of using this timeout is one thread per connection. | infinity timeout | Any Integer value |
| mail.imap.statuscachetimeout | The timeout value in milliseconds for the cache of STATUS command response. | 1000ms | Time out in miliseconds |
| mail.imap.appendbuffersize | The maximum size of a message to buffer in memory when appending to an IMAP folder. | None | Any Integer value |
| mail.imap.connectionpoolsize | The maximum number of available connections in the connection pool. | 1 | Any Integer value |
| mail.imap.connectionpooltimeout | The timeout value in milliseconds for connection pool connections. | 45000ms | Any Integer |
| mail.imap.separatestoreconnection | If this parameter is set to true, it indicates that a dedicated store connection needs to be used for store commands. | true | true or false |
| mail.imap.auth.login.disable | If this is set to true, it is not possible to use the non-standard AUTHENTICATE LOGIN command instead of the plain LOGIN command. | false | true or false |
| mail.imap.auth.plain.disable | If this is set to true, the AUTHENTICATE PLAIN command cannot be used. | false | true or false |
| mail.imap.auth.ntlm.disable | If true, prevents use of the AUTHENTICATE NTLM command. | false | true or false |
| mail.imap.proxyauth.user | If the server supports the PROXYAUTH extension, this property specifies the name of the user to act as. Authentication to log in to the server is carried out using the administrator's credentials. After authentication, the IMAP provider issues the PROXYAUTH command with the user name specified in this property. | None | Valid string value |
| mail.imap.localaddress | The local address (host name) to bind to when creating the IMAP socket. | Defaults to the address picked by the Socket class. | Valid string value |
| mail.imap.localport | The local port number to bind to when creating the IMAP socket. | Defaults to the port number picked by the Socket class. | Valid String value |
| mail.imap.sasl.enable | If this parameter is set to true, the system attempts to use the javax.security.sasl package to choose an authentication mechanism for the login. | false | true or false |
| mail.imap.sasl.mechanisms | A list of SASL mechanism names that the system should to try to use. The names can be separated by spaces or commas. | None | Valid string value |
| mail.imap.sasl.authorizationid | The authorization ID to use in the SASL authentication. | If this parameter is not set, the authentication ID (username) is used. | Valid string value |
| mail.imap.sasl.realm | The realm to use with SASL authentication mechanisms that require a realm, such as DIGEST-MD5. | None | Valid string value |
| mail.imap.auth.ntlm.domain | The NTLM authentication domain. | None | Valid string value |
| The NTLM authentication domain. | NTLM protocol-specific flags. | None | Valid integer value |
| mail.imap.socketFactory | If this parameter is set to a class that implements the javax.net.SocketFactory interface, this class is used to create IMAP sockets. | None | Valid SocketFactory |
| mail.imap.socketFactory.class | If this parameter is set, it specifies the name of a class that implements the javax.net.SocketFactory interface. This class is used to create IMAP sockets. | None | Valid string |
| mail.imap.socketFactory.fallback | If this parameter is set to true, failure to create a socket using the specified socket factory class results in the socket being created using the java.net.Socket class. | true | true or false |
| mail.imap.socketFactory.port | This specifies the port to connect to when using the specified socket factory. If this parameter is not set, the default port is used. | 143 | Valid Integer |
| mail.imap.ssl.checkserveridentity | If this parameter is set to true, the system checks the server identity as specified by RFC 2595. | false | true or false |
| mail.imap.ssl.trust | If this parameter is set and a socket factory has not been specified, it enables the use of a MailSSLSocketFactory. If this parameter is set to *, all the hosts are trusted. If this parameter specifies list of hosts separated by white spaces, only those hosts are trusted. If the parameter is not set to any of the values mentioned above, trust depends on the certificate presented by the server. | * | Valid String |
| mail.imap.ssl.socketFactory | If this parameter is set to a class that extends the javax.net.ssl.SSLSocketFactory class this class is used to create IMAP SSL sockets. | None | SSL Socket Factory |
| mail.imap.ssl.socketFactory.class | If this parameter is set, it specifies the name of a class that extends the javax.net.ssl.SSLSocketFactory class. This class is used to create IMAP SSL sockets. | None | Valid String |
| mail.imap.ssl.socketFactory.port | This specifies the port to connect to when using the specified socket factory. | the default port 993 is used. | valid port number |
| mail.imap.ssl.protocols | This specifies the SSL protocols that are enabled for SSL connections. The property value is a whitespace-separated list of tokens acceptable to the javax.net.ssl.SSLSocket.setEnabledProtocols method. | None | Valid string |
| mail.imap.starttls.enable | If this parameter is set to true, it is possible to use the STARTTLS command (if supported by the server) to switch the connection to a TLS-protected connection before issuing any login commands. | false | true or false |
| mail.imap.socks.host | This specifies the host name of a SOCKS5 proxy server that is used to connect to the mail server. | None | Valid String |
| mail.imap.socks.port | This specifies the port number for the SOCKS5 proxy server. This is needed if the proxy server is not using the standard port number 1080. | 1080 | Valid String |
| mail.imap.minidletime | This property sets the delay in milliseconds. | 10 milliseconds | time in seconds (Integer) |
| mail.imap.enableimapevents | If this property is set to true, it enables special IMAP-specific events to be delivered to the ConnectionListener of the store. The unsolicited responses received during the idle method of the store are sent as connection events with IMAPStore.RESPONSE as the type. The event's message is the raw IMAP response string. | false | true or false |
| mail.imap.folder.class | The class name of a subclass of com.sun.mail.imap.IMAPFolder. The subclass can be used to provide support for additional IMAP commands. The subclass must have public constructors of the form public MyIMAPFolder(String fullName, char separator, IMAPStore store, Boolean isNamespace) and public MyIMAPFolder(ListInfo li, IMAPStore store) | None | Valid String |
| mail.pop3.connectiontimeout | The socket connection timeout value in milliseconds. | Infinite timeout | Integer value |
| mail.pop3.timeout | The socket I/O timeout value in milliseconds. | Infinite timeout | Integer value |
| mail.pop3.message.class | The class name of a subclass of com.sun.mail.pop3.POP3Message. | None | Valid String |
| mail.pop3.localaddress | The local address (host name) to bind to when creating the POP3 socket. | Defaults to the address picked by the Socket class. | Valid String |
| mail.pop3.localport | The local port number to bind to when creating the POP3 socket. | Defaults to the port number picked by the Socket class. | Valid port number |
| mail.pop3.apop.enable | If this parameter is set to true, use APOP instead of USER/PASS to log in to the POP3 server (if the POP3 server supports APOP). APOP sends a digest of the password instead of clearing the text password. | false | true or false |
| mail.pop3.socketFactory | If this parameter is set to a class that implements the javax.net.SocketFactory interface, this class is used to create POP3 sockets. | None | Socket Factory |
| mail.pop3.socketFactory.class | If this parameter is set, it specifies the name of a class that implements the javax.net.SocketFactory interface. This class is used to create POP3 sockets. | None | Valid String |
| mail.pop3.socketFactory.fallback | If this parameter is set to true, failure to create a socket using the specified socket factory class results in the socket being created using the java.net.Socket class. | false | true or false |
| mail.pop3.socketFactory.port | This specifies the port to connect to when using the specified socket factory. | Default port | Valid port number |
| mail.pop3.ssl.checkserveridentity | If this parameter is set to true, check the server identity as specified by RFC 2595. | false | true or false |
| mail.pop3.ssl.trust | If this parameter is set and a socket factory has not been specified, it is possible to use a MailSSLSocketFactory. If this parameter is set to *, all the hosts are trusted. If the parameter is set to a whitespace-separated list of hosts, only those hosts are trusted. If the parameter is not set to any of the values mentioned above, trust depends on the certificate presented by the server. | * | Valid String |
| mail.pop3.ssl.socketFactory | If this parameter is set to a class that extends the javax.net.ssl.SSLSocketFactory class, this class is used to create POP3 SSL sockets. | None | SSL Socket Factory |
| mail.pop3.ssl.checkserveridentity | If this parameter is set to true, the system checks the server identity as specified by RFC 2595. | false | true or false |
| mail.pop3.ssl.trust | If this parameter is set and a socket factory has not been specified, it is possible to use a MailSSLSocketFactory. If this parameter is set to *, all the hosts are trusted. If the parameter is set to a whitespace-separated list of hosts, only those hosts are trusted. | Trust depends on the certificate presented by the server. | Valid String |
| mail.pop3.ssl.socketFactory | If this parameter is set to a class that extends the javax.net.ssl.SSLSocketFactory class, this class is used to create POP3 SSL sockets. | None | SSL Socket Factory |
| mail.pop3.ssl.socketFactory.class | If this parameter is set, it specifies the name of a class that extends the javax.net.ssl.SSLSocketFactory class. This class is used to create POP3 SSL sockets. | None | Valid String |
| mail.pop3.ssl.socketFactory.p | This parameter pecifies the port to connect to when using the specified socket factory. | 995 | Valid Integer |
| mail.pop3.ssl.protocols | This parameter specifies the SSL protocols that are enabled for SSL connections. The property value is a whitespace-separated list of tokens acceptable to the javax.net.ssl.SSLSocket.setEnabledProtocols method. | None | Valid String |
| mail.pop3.starttls.enable | If this parameter is set to true, it is possible to use the STLS command (if supported by the server) to switch the connection to a TLS-protected connection before issuing any login commands. | false | true or false |
| mail.pop3.starttls.required | If this parameter is set to true, it is required to use the STLS command. The connect method fails if the server does not support the STLS command or if the command fails. | false | true or false |
| mail.pop3.socks.host | This parameter specifies the host name of a SOCKS5 proxy server that can be used to connect to the mail server. | None | Valid String |
| mail.pop3.socks.port | This parameter specifies the port number for the SOCKS5 proxy server. | None | Valid String |
| mail.pop3.disabletop | If this parameter is set to true, the POP3 TOP command is not used to fetch message headers. | false | true or false |
| mail.pop3.forgettopheaders | If this parameter is set to true, the headers that might have been retrieved using the POP3 TOP command is forgotten and replaced by the headers retrieved when the POP3 RETR command is executed. | false | true or false |
| mail.pop3.filecache.enable | If this parameter is set to true, the POP3 provider caches message data in a temporary file instead of caching them in memory. Messages are only added to the cache when accessing the message content. Message headers are always cached in memory (on demand). The file cache is removed when the folder is closed or the JVM terminates. | false | true or false |
| mail.pop3.filecache.dir | If the file cache is enabled, this property is used to override the default directory used by the JDK for temporary files. | None | Valid String |
| mail.pop3.cachewriteto | This parameter controls the behavior of the writeTo method on a POP3 message object. If the parameter is set to true, the message content has not been cached yet, and the ignoreList is null, the message is cached before being written. If not, the message is streamed directly to the output stream without being cached. | false | true or false |
| mail.pop3.keepmessagecontent | If this property is set to true, a hard reference to the cached content is retained, preventing the memory from being reused until the folder is closed, or until the cached content is explicitly invalidated (using the invalidate method). | false | true or false |
Example 1
CREATE SOURCE inputStream WITH (type='email', map.type='xml', username='receiver.account', password='account.password') (name string, age int, country string);
This example illustrates how to receive events in xml format via the
email source. In this example, only the required parameters are defined
in the stream definition. The default values are taken for the other
parameters. The search term is not defined, and therefore, all the new
messages in the inbox folder are polled and taken.
Example 2
CREATE SOURCE inputStream WITH (type='email', map.type='xml', username='receiver.account', password='account.password',store = 'imap',host = 'imap.gmail.com',port = '993',searchTerm = 'subject:Stream Processor, from: from.account@ , cc: cc.account',polling.interval='500',action.after.processed='DELETE',content.type='text/html') (name string, age int, country string);
This example illustrates how to receive events in xml format via the
email source. The email source polls the mail account every 500 seconds
to check whether any new mails have arrived. It processes new mails only
if they satisfy the conditions specified for the email search term (the
value for from of the email message should be from.account@.<host
name>, and the message should contain cc.account in the cc
receipient list and the word Stream Processor in the mail subject).
in this example, the action after processing is DELETE.
Therefore,after processing the event, corresponding mail is deleted from
the mail folder.