Connecting to on premise systems using the SAP Connectivity Service in the BTP Cloud Foundry environment is convenient, reliable, secure and helps integrating fast-paced Cloud applications with on premise systems.
However I had never used protocols other than HTTP(S) and the occasional RFC connection through the Connectivity Service and was in for a surprise when I wanted to connect to a Mail server through that very service. This blog shows how I achieved the connection, which challenges I faced and what helped most.
Prerequisites
Since we only want to show the functionality itself the list of needed Cloud Foundry services is rather short:
- CF Application Runtime
- Connectivity Service
- Destination Service
- Authorization & Trust Management Service (xsuaa)
Due to the authentication mechanism of the SOCKS 5 proxy we'll talk about in a minute, I wasn't able to use my preferred runtime (Node.js) in the beginning. The issue is resolved now but at the time the solution (
custom authentication in socks proxy) was not available yet. Therefore in this blog we'll be using a Java Spring Boot application to connect to the mail server.
So the only prerequisite apart from a working Java IDE of your choice is a working Maven installation as well.
Proxy Authentication
The supported protocols of the Connectivity Service Proxy are the following:
For using IMAP(S) and SMTP(S) we need to fall back to TCP via a SOCKS 5 proxy since none of the other protocols enables us to use the wanted application protocols.
According to the
documentation unfortunately the standard authentication methods given in the
RFC for SOCKS 5 (no auth, basic auth, GSS API) are not available and we need to adapt the custom ('0x80') authentication method for the proxy which effectively implements an OAuth authorisation flow.
Fortunately the documentation also provides a sample implementation of a Java Socket which we can make use of to connect to the SOCKS 5 proxy (see documentation above).
There was only one thing that needed to be added for me to be able to use the code sample: one missing method for the socket to connect only taking the endpoint, not a timeout.
@Override
public void connect(SocketAddress endpoint) throws IOException {
this.connect(endpoint, 0);
}
To actually make the proxy authentication work, the last step we need to take is acquiring a valid JWT token from the connectivity service to be able to use the Socket implementation for connecting to the on premise server. Luckily we can use the
xsuaa-spring-boot-starter dependency
provided by SAP to secure your Spring Boot application with an xsuaa instance and/or achieve authentication with other OAuth resources.
Getting the required access token requires retrieving the client credentials from the service binding and use them in the
XsuaaTokenFlows mechanism provided by the library:
XsuaaTokenFlows tokenFlows = new XsuaaTokenFlows(
new DefaultOAuth2TokenService(),
new XsuaaDefaultEndpoints(xsuaaUri.toString()),
new ClientCredentials(clientid, clientsecret));
OAuth2TokenResponse serviceTokenResponse = tokenFlows.clientCredentialsTokenFlow().execute();
String accessToken = serviceTokenResponse.getAccessToken();
Use the acquired accessToken and (if needed) the location ID of your cloud connector as an input to the constructor of the
ConnectivitySocks5ProxySocket and you're good to go - at least from a technical point of view.
One side note regarding the client credentials from the service binding: Instead of parsing that yourself from the
VCAP_SERVICES environment variable you can easily use the
CfEnv library by Pivotal as a lightweight wrapper to do that for you. I created a
gist on Github where you can get an idea on how to use it in conjunction with the
XsuaaTokenFlow.
SMTP Transport
No that we've achieved technical connectivity it's time to connect to our on premise mail server and send the first email. For this next step we'll need to know what the credentials to connect, host and port will look like. I strongly recommend not storing such information in the code and fetch it from the destination service instead. This also gives us the advantage of being able to change this information without needing to rebuild and redeploy the application.
After acquiring the necessary information from the destination service we're all set to begin with our first email. I won't go into too much detail here on the usage of the Jakarta Mail library (f.k.a. Java Mail) except that we're going to use the more technical way of sending an email in order to use our custom implementation of the Socket class.
Properties props = new Properties();
props.put("mail.transport.protocol", "smtp");
props.put("mail.smtp.auth", Boolean.toString(true));
props.put("mail.mime.charset", "UTF-8");
Authenticator auth = new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(userFromDest, passwordFromDest);
}
}
Session session = Session.getInstance(props, auth);
// create Email in a separate method / class
MimeMessage msg = getMailToBeSent(session);
// create SMTPTransport for sending the email
SMTPTransport transport = (SMTPTransport) session.getTransport("smtp");
// create Socket to the host and port from the destination (virtual host from Cloud Connector)
Socket socket = new ConnectivitySocks5ProxySocket(accessToken, sccLocationId);
socket.connect(new InetSocketAddress(hostFromDest, portFromDest));
transport.connect(socket);
// send the message
transport.sendMessage(msg, mesg.getAllRecipients());
Of course this is a minimal example and you could make this more reusable and readable.
Please note that I'm using unencrypted SMTP here which might not be secure. Please refer to the Jakarta Mail documentation on how to achieve an SSL secured connection. Also in the next section we'll use SSL encrypted IMAP (so called IMAPS) due the use case I developed this for but it uses a slightly different
IMAP SSL Store
Retrieving emails works similar to sending an email with one slight difference: For retrieving the emails we'll just configure the Jakarta Mail library to create a Socket capable of using the connectivity service instead of creating it ourselves.
Since we already saw how to create a Jakarta Mail Session in the previous section I'll just point out the differences we'll need now in the properties of the session:
ConnectivitySocketFactory factory = new ConnectivitySocketFactory();
props.put("mail.imaps.socketFactory", factory);
props.put("mail.imaps.socketFactory.fallback", Boolean.toString(false));
props.put("mail.imaps.ssl.checkserveridentity", Boolean.toString(true));
Keep in mind that the previously used properties need to be set as well with "imaps" instead of "smtp" in their keys.
The
ConnectivitySocketFactory class extends the standard Java
SocketFactory class and basically returns an instance of our
ConnectivitySocks5ProxySocket implementation (mind the cloud connector location ID!). You can find the
ConnectivitySocketFactory in this
GitHub gist along with the
ConnectivityService class I use to manage the connection to the connectivity service instance in this
GitHub gist.
The other properties tell the library to not fall back to the default SocketFactory (very helpful for debugging) and to check the server's identity (see side notes on SSL encryption below).
Due to a mechanism checking the Socket passed from the factory inside the mail library we won't need to provide an SSLSocket since the existing one will just be wrapped in a new SSLSocket by the library. This preserves our custom Socket and therefore the connection to the connectivity service.
Fortunately the rest of the process retrieving the mails from the server is relatively easy once we get the configuration right.
IMAPSSLStore store = (IMAPSSLStore) imapSession.getStore("imaps");
store.connect(hostFromDest, portFromDest, userFromDest, passwordFromDest);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE); // open incl. WRITE access to be able to delete messages
Message[] messages = inbox.getMessages();
Since reading / parsing mail objects would fill a whole blog series, I'll keep it at that.
Side notes on SSL encryption
As you probably know the concept of certificate based trust relies on a chain of trust originating from a certificate trusted by your runtime / machine / ... . Using self-signed certificates for internal servers is common practice and was the case for my use case as well. Since it is never a good idea not to check your SSL communication partner's identity, we need a way to do that with a internally signed certificate as well.
To trust a self-signed or internally signed, no publicly trusted certificate you need to create a custom trust store (e.g. by copying the default Java trust store and adding your internal root certificate) and add it to your Java project. We can then
tell the Java Buildpack on CF to use our custom trust store and thereby create a trust for the internally signed mail server.
Another issue we faced with using SSL encrypted connections was our lack of knowledge in regards to Cloud Connector TCP connection settings so I'd like to leave a hint on that as well: If you're already creating an SSL request from your application (as done here) set the connection to be a TCP connection. Only if your actual endpoint only speaks TCP over SSL and your accessing application doesn't set the connection to TCP SSL, otherwise the connection will fail. The connection would then be encrypted twice: on application and on Cloud Connector side.
Local development
To be able to run the application locally there's two things making your life much easier:
The
.env file is a possibility to emulate environment variable values as they would be present in the CF environment - namely the
VCAP_SERVICES variable. To get an idea of what the content should look like run the command
cf env <your application name> of an existing cf application. For the
.env file to work the content of the variable has to be in one line. In your local launch configuration you then just need to supply the path to the file to run the application with the desired environment variables set. Shortcut for developers using the Business Application Studio: service bindings in the
.env file are created for you automatically with the built in "Bind to service" functionality!
For accessing resources guarded by the connectivity service there's an additional step to be taken. You need to
create an ssh tunnel to the deployed version of your application using the command
cf ssh <your application name> -L 20004:connectivityproxy.internal.<cf landscape>.hana.ondemand.com:20004. Don't forget to also adjust the
onpremise_proxy_host property in your environment configuration from above to point to
localhost.
Summary
After some initial problems accessing the connectivity service all services and libraries came together nicely and I was able to send and fetch emails from an on premise mail server. With some tweaking I was even able to make the application run locally and therefore ease my development process a lot since i didn't have to redeploy every time.
I also want to make a closing note on my initial problem regarding the use of Node.js. By now the maintainer of the
socks npm repo has added my request of a custom authentication feature. However the library I was planning to use didn't work with it. So in the end I'm still glad I chose to go with the Java way, also learning a lot about Spring Boot and refreshing my Java knowledge.
As a last word I'd like to thank Leo for his
blog on connecting to an on premise Apache Kafka instance from Node.js on CF which initially made me aware of the problem I'd be facing with the '0x80' authentication.
Update: Added the
ConnectivitySocketFactory class as a gist. This enables you to also use the Spring Boot Mail mechanism by configuring the custom
ConnectivitySocketFactory as the one used by Spring Boot. Also added the
ConnectivityService as an example for connecting to the connectivity service instance.
