Skip to Content
Technical Articles
Author's profile photo Niklas Miroll

Connect to an On Premise Mail Server from SAP BTP Cloud Foundry

Connecting to on premise systems using the SAP Connectivity Service in the BTP Cloud Foundry environment is convenient, reliable, secure and helps integrating fast-paced Cloud applications with on premise systems.

However I had never used protocols other than HTTP(S) and the occasional RFC connection through the Connectivity Service and was in for a surprise when I wanted to connect to a Mail server through that very service. This blog shows how I achieved the connection, which challenges I faced and what helped most.

Prerequisites

Since we only want to show the functionality itself the list of needed Cloud Foundry services is rather short:

  • CF Application Runtime
  • Connectivity Service
  • Destination Service
  • Authorization & Trust Management Service (xsuaa)

Due to the authentication mechanism of the SOCKS 5 proxy we’ll talk about in a minute, I wasn’t able to use my preferred runtime (Node.js) in the beginning. The issue is resolved now but at the time the solution (custom authentication in socks proxy) was not available yet. Therefore in this blog we’ll be using a Java Spring Boot application to connect to the mail server.

So the only prerequisite apart from a working Java IDE of your choice is a working Maven installation as well.

Proxy Authentication

The supported protocols of the Connectivity Service Proxy are the following:

  • HTTP
  • RFC
  • LDAP
  • SOCKS 5

For using IMAP(S) and SMTP(S) we need to fall back to TCP via a SOCKS 5 proxy since none of the other protocols enables us to use the wanted application protocols.

According to the documentation unfortunately the standard authentication methods given in the RFC for SOCKS 5 (no auth, basic auth, GSS API) are not available and we need to adapt the custom (‘0x80’) authentication method for the proxy which effectively implements an OAuth authorisation flow.

Fortunately the documentation also provides a sample implementation of a Java Socket which we can make use of to connect to the SOCKS 5 proxy (see documentation above).

There was only one thing that needed to be added for me to be able to use the code sample: one missing method for the socket to connect only taking the endpoint, not a timeout.

@Override
public void connect(SocketAddress endpoint) throws IOException {
    this.connect(endpoint, 0);
}

To actually make the proxy authentication work, the last step we need to take is acquiring a valid JWT token from the connectivity service to be able to use the Socket implementation for connecting to the on premise server. Luckily we can use the xsuaa-spring-boot-starter dependency provided by SAP to secure your Spring Boot application with an xsuaa instance and/or achieve authentication with other OAuth resources.

Getting the required access token requires retrieving the client credentials from the service binding and use them in the XsuaaTokenFlows mechanism provided by the library:

XsuaaTokenFlows tokenFlows = new XsuaaTokenFlows(
        new DefaultOAuth2TokenService(),
        new XsuaaDefaultEndpoints(xsuaaUri.toString()),
        new ClientCredentials(clientid, clientsecret));

OAuth2TokenResponse serviceTokenResponse = tokenFlows.clientCredentialsTokenFlow().execute();
String accessToken = serviceTokenResponse.getAccessToken();

Use the acquired accessToken and (if needed) the location ID of your cloud connector as an input to the constructor of the ConnectivitySocks5ProxySocket and you’re good to go – at least from a technical point of view.

One side note regarding the client credentials from the service binding: Instead of parsing that yourself from the VCAP_SERVICES environment variable you can easily use the CfEnv library by Pivotal as a lightweight wrapper to do that for you. I created a gist on Github where you can get an idea on how to use it in conjunction with the XsuaaTokenFlow.

SMTP Transport

No that we’ve achieved technical connectivity it’s time to connect to our on premise mail server and send the first email. For this next step we’ll need to know what the credentials to connect, host and port will look like. I strongly recommend not storing such information in the code and fetch it from the destination service instead. This also gives us the advantage of being able to change this information without needing to rebuild and redeploy the application.

After acquiring the necessary information from the destination service we’re all set to begin with our first email. I won’t go into too much detail here on the usage of the Jakarta Mail library (f.k.a. Java Mail) except that we’re going to use the more technical way of sending an email in order to use our custom implementation of the Socket class.

Properties props = new Properties();
props.put("mail.transport.protocol", "smtp");
props.put("mail.smtp.auth", Boolean.toString(true));
props.put("mail.mime.charset", "UTF-8");

Authenticator auth = new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(userFromDest, passwordFromDest);
    }
}

Session session = Session.getInstance(props, auth);

// create Email in a separate method / class
MimeMessage msg = getMailToBeSent(session);

// create SMTPTransport for sending the email
SMTPTransport transport = (SMTPTransport) session.getTransport("smtp");

// create Socket to the host and port from the destination (virtual host from Cloud Connector)
Socket socket = new ConnectivitySocks5ProxySocket(accessToken, sccLocationId);
socket.connect(new InetSocketAddress(hostFromDest, portFromDest));
transport.connect(socket);

// send the message
transport.sendMessage(msg, mesg.getAllRecipients());

Of course this is a minimal example and you could make this more reusable and readable.

Please note that I’m using unencrypted SMTP here which might not be secure. Please refer to the Jakarta Mail documentation on how to achieve an SSL secured connection. Also in the next section we’ll use SSL encrypted IMAP (so called IMAPS) due the use case I developed this for but it uses a slightly different

IMAP SSL Store

Retrieving emails works similar to sending an email with one slight difference: For retrieving the emails we’ll just configure the Jakarta Mail library to create a Socket capable of using the connectivity service instead of creating it ourselves.

Since we already saw how to create a Jakarta Mail Session in the previous section I’ll just point out the differences we’ll need now in the properties of the session:

ConnectivitySocketFactory factory = new ConnectivitySocketFactory();
props.put("mail.imaps.socketFactory", factory);
props.put("mail.imaps.socketFactory.fallback", Boolean.toString(false));
props.put("mail.imaps.ssl.checkserveridentity", Boolean.toString(true));

Keep in mind that the previously used properties need to be set as well with “imaps” instead of “smtp” in their keys.

The ConnectivitySocketFactory class extends the standard Java SocketFactory class and basically returns an instance of our ConnectivitySocks5ProxySocket implementation (mind the cloud connector location ID!). You can find the ConnectivitySocketFactory in this GitHub gist along with the ConnectivityService class I use to manage the connection to the connectivity service instance in this GitHub gist.

The other properties tell the library to not fall back to the default SocketFactory (very helpful for debugging) and to check the server’s identity (see side notes on SSL encryption below).

Due to a mechanism checking the Socket passed from the factory inside the mail library we won’t need to provide an SSLSocket since the existing one will just be wrapped in a new SSLSocket by the library. This preserves our custom Socket and therefore the connection to the connectivity service.

Fortunately the rest of the process retrieving the mails from the server is relatively easy once we get the configuration right.

IMAPSSLStore store = (IMAPSSLStore) imapSession.getStore("imaps");
store.connect(hostFromDest, portFromDest, userFromDest, passwordFromDest);

Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE); // open incl. WRITE access to be able to delete messages
Message[] messages = inbox.getMessages();

Since reading / parsing mail objects would fill a whole blog series, I’ll keep it at that.

Side notes on SSL encryption

As you probably know the concept of certificate based trust relies on a chain of trust originating from a certificate trusted by your runtime / machine / … . Using self-signed certificates for internal servers is common practice and was the case for my use case as well. Since it is never a good idea not to check your SSL communication partner’s identity, we need a way to do that with a internally signed certificate as well.

To trust a self-signed or internally signed, no publicly trusted certificate you need to create a custom trust store (e.g. by copying the default Java trust store and adding your internal root certificate) and add it to your Java project. We can then tell the Java Buildpack on CF to use our custom trust store and thereby create a trust for the internally signed mail server.

Another issue we faced with using SSL encrypted connections was our lack of knowledge in regards to Cloud Connector TCP connection settings so I’d like to leave a hint on that as well: If you’re already creating an SSL request from your application (as done here) set the connection to be a TCP connection. Only if your actual endpoint only speaks TCP over SSL and your accessing application doesn’t set the connection to TCP SSL, otherwise the connection will fail. The connection would then be encrypted twice: on application and on Cloud Connector side.

Local development

To be able to run the application locally there’s two things making your life much easier:

The .env file is a possibility to emulate environment variable values as they would be present in the CF environment – namely the VCAP_SERVICES variable. To get an idea of what the content should look like run the command cf env <your application name> of an existing cf application. For the .env file to work the content of the variable has to be in one line. In your local launch configuration you then just need to supply the path to the file to run the application with the desired environment variables set. Shortcut for developers using the Business Application Studio: service bindings in the .env file are created for you automatically with the built in “Bind to service” functionality!

For accessing resources guarded by the connectivity service there’s an additional step to be taken. You need to create an ssh tunnel to the deployed version of your application using the command cf ssh <your application name> -L 20004:connectivityproxy.internal.<cf landscape>.hana.ondemand.com:20004. Don’t forget to also adjust the onpremise_proxy_host property in your environment configuration from above to point to localhost.

Summary

After some initial problems accessing the connectivity service all services and libraries came together nicely and I was able to send and fetch emails from an on premise mail server. With some tweaking I was even able to make the application run locally and therefore ease my development process a lot since i didn’t have to redeploy every time.

I also want to make a closing note on my initial problem regarding the use of Node.js. By now the maintainer of the socks npm repo has added my request of a custom authentication feature. However the library I was planning to use didn’t work with it. So in the end I’m still glad I chose to go with the Java way, also learning a lot about Spring Boot and refreshing my Java knowledge.

As a last word I’d like to thank Leo for his blog on connecting to an on premise Apache Kafka instance from Node.js on CF which initially made me aware of the problem I’d be facing with the ‘0x80’ authentication.

 

Update: Added the ConnectivitySocketFactory class as a gist. This enables you to also use the Spring Boot Mail mechanism by configuring the custom ConnectivitySocketFactory as the one used by Spring Boot. Also added the ConnectivityService as an example for connecting to the connectivity service instance.

Assigned tags

      6 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Gregor Wolf
      Gregor Wolf

      Hi Niklas,

      thank you for sharing your experience. Would you be up to help adding OnPremise destination support ot sap-cf-mailer? Right now it only supports internet connections. I've also asked in the SAP Cloud SDK: is adding support to send mail via an (OnPremise) SMTP destination in the scope of the Cloud SDK?

      CU
      Gregor

      Author's profile photo Niklas Miroll
      Niklas Miroll
      Blog Post Author

      Hi Gregor,

      that sounds like a good idea! As you probably read I tried with nodemailer first so I'd be interested to see if we can get it to work with the new version of the socks proxy.

      BR,
      Niklas

      Author's profile photo Fabiano Rosa
      Fabiano Rosa

      Hi Niklas, great blog post! I was wondering if you have some updates regarding the issues and limitations that you found in Node.js, because I'm looking now for a similar scenario with Node.js and on-premise mail server.

      Regards,

      Fabiano Rosa

      Author's profile photo Niklas Miroll
      Niklas Miroll
      Blog Post Author

      Hi Fabiano,

      unfortunately I didn't have the time yet to look into the Node.js equivalent.

      Author's profile photo Tim Anlauf
      Tim Anlauf

      Hello Niklas Miroll,

      I try to get this implemented on our BTP account. At the moment I struggle with the .env file as a mbt build is throwing the error that my configured service "wp-connectivity" can't be found.

      I added the service using the bind functionality of BAS.

      Is there any way that you publish your complete application as a github repo?

       

      Author's profile photo Niklas Miroll
      Niklas Miroll
      Blog Post Author

      Hello Tim Anlauf ,

      the bind functionality in BAS does exactly what you would do manually: it changes the .env file to contain the necessary binding information so this is the right way to do it in BAS.

      I personally developed this locally and therefore didn't have the BAS functionality. I solved this by writing a small node script which transforms my default-env.json (which I have because of my main node project) into the correct .env file.

      Due to a lot of very customer specific code I cannot publish the complete project. If you struggle with any of the parts please feel free to contact me here and I'll try my best to help!

      Best regards,
      Niklas