Technical Articles

David Honan

March 1, 2019 23 minute read

Using Connector Builder in Open Connectors – Mailchimp Tutorial

This blog is a tutorial whereby you will learn how to install the basic foundational features when building and testing a new Connector in Open Connectors – specifically the MailChimp V3 connector.

Note: I’ve selected the MailChimp V3 cloud service because their REST API is well formed and makes for a nice, first example build. You can use the principles covered in this blog to build out other connectors as your customer’s use case dictates.

One of the most asked questions that I receive when providing an overview of Open Connectors to developers or integrators is, “I see that there are 160+ connectors in the catalog, but the one that I need is not there – How do I add the connector(s) that I need for my customer’s use cases?”

The answer to that question is Connector Builder.

I’ve read a couple of excellent SAP Community blogs recently that cover this topic, here is a list:

Build a new connector in SAP Cloud Platform Open Connectors, by Felipe de Mello
XPower Meter Read Scanner: Google Cloud Vision Connector on SAP Open Connector Cloud Platform, by Sravya Talanki

Note: Accompanying this blog is a lengthy video – 28 minutes in total – that I’ll be referencing sections of throughout, with each link taking you to that same video but starting at the relevant portion. Thus, you don’t have to watch the entire video if you only need information on a specific part of the Connector build process.

Connectors

If you are not already familiar with the Connectors in Open Connectors, here is a quick overview. They are more than just the simple connectors that you may be accustomed to using previously in other iPaaS services – they are super connectors. SAP has packaged each with built in functionality whereby each is a prebuilt API integration to a specific cloud application or cloud service endpoint (e.g., Workday, Box, Salesforce, etc.).

Connectors provide a variety of advanced, built-in features, including normalized authentication, discovery APIs, search capabilities, event handling, unified error handling, and more – all accessible via uniform REST APIs. Because they provide a common, uniform integration interface for developers to interact with, connectors save you development and ongoing maintenance time by keeping your integrations consistent and uniform across the board.

There are 160+ connectors currently in the connectors catalog, with more being added on a regular basis:

It is the job of the connector, on the endpoint side (e.g. Netsuite) to know the specific API definition and protocol of its assigned endpoint service – and on the platform side to expose a uniform API definition that “Harmonizes the API” with all the other connectors, essentially making each endpoint look and function in a uniform way.

Thus, Autotask, Salesforce, and Dynamics are harmonized in terms of their CRM API resources due to the nature of a connector. In fact, when you look at any connector in the catalog, you can feel pretty confident that you can begin using it immediately (with little learning curve) because the endpoint’s nuances are accommodated for and presented back as harmonized REST APIs.

Connector Builder

There are two options available in with Connector Builder:

Build a new, private Connector

Edit Existing Connectors

Anatomy of a Connector

Let’s back up a bit and review the major features that make up a Connector:

Normalized JSON APIs: Where the Connector surfaces a set of API Resources using a consistent RESTful API with a JSON payload – regardless of the web service protocol used at that the endpoint. Connectors do the work of mapping the normalized API call to each application’s API endpoints (such as Salesforce, Netsuite, or Hubspot), as well as transforming SOAP, XML and others to REST/JSON.

Authentication: Connectors also normalize authentication and provide a consistent set of API resource calls regardless of endpoint authentication type – e.g basic, OAuth, token, and custom mechanisms. The reason the Connector does this, is because there are inherent authentication workflow variations and by normalizing this process, it is consistent for the developer to work with and maintain.

Discovery APIs & Custom Data: Connectors provide normalized data discovery service API resources that allow for a developer to query the endpoint about what data objects and associated metadata are available. This is perfect for discovering a the list of custom field names and types associated with an object. Connectors also allows you to discover custom fields by supplying an object ID when a native discovery service is not available.

Pagination: When a request returns multiple pages of a response, Connectors normalize the pagination information from the endpoint, making the code required to handle paging consistent regardless of how the endpoint’s API specification handles it.

Error Handling: Is also normalized by the Connector, where endpoint error messages are returned in a normalized JSON structure providing consistent error codes across all endpoints. If a response returns a message that includes a specific detail about the method call, a normalized error code is sent along with any additional data included in the body of the message.

Bulk: Another unique capability that a Connector provides is a bulk framework, which provides for bulk or batch data operations and retry capabilities. The normalized bulk API resources that a Connector provides are present even if the endpoint doesn’t natively support bulk or batch, making the Connector even more useful.

Eventing: Connectors contain a built-in eventing framework which provides the ability to receive endpoint event notifications in the form of a webhook – even if the endpoint doesn’t support webhooks natively. This poller can be also be used even if the endpoint does natively provide for webhooks – which can, in some cases, be a smoother process then the hairy webhook setup process that some endpoints force upon their users. Because of this Eventing framework, developers can automate business workflows and processes within their applications easily and consistently without having to build a full blown polling framework within their application and adapt it for each endpoint – the Connector takes care of that for them.

Search: A commonly needed feature of a Connector is the standardized searching that it provides. It uses a common query language (termed “CEQL”) which is largely based on the SQL specification that everyone is already familiar with. Essentially the Connector does the work of transforming the CEQL to the native search language at an endpoint. Using CEQL, developers can pass the same query structure to all Connector endpoints, making this process consistent and easier across multiple endpoints.

The Mailchimp API & registering your application

The first thing you will need to do (video) whenever you want to create a new Connector is to get yourself a trial account for the endpoint service – in this case, Mailchimp – search for mailchimp trial. Also, locate the Mailchimp API documentation.

There are a few things that you need to evaluate as you look through the documentation:

Is it REST, SOAP, or other?

In Mailchimp’s docs, you can see that it is clearly RESTful:

Is there a base url listed for each API resource call?

You will need this when you build the Connector – in this case it is under Getting Started and is listed under resources.

What is the authentication method?

You can see that Mailchimp provides two choices: Basic and OAuth2:

And here are the URLs for Oauth2 that are required:

How do you gain access to the API and get the needed keys to authenticate?

Since it is Oauth2 you will need to register your application (video)with Mailchimp so that you can get a Client ID and Secret in order to authenticate your users.

What is a simple resource to get started with?

Pick a resource that makes sense for your use case for you initial build in this case, I’ll use /lists as they are likely the main object that someone using a Mailchimp integration would want to integrate with.

GET Building!

Start by first registering our application with Mailchimp. In Mailchimp’s documentation you can find information on Register your Application. I followed the instructions and registered my application called “David’s Demo App” and retrieved the needed Client ID and Client Secret.

TIP: This is very typical with how these endpoints are set up. What Registering you app does is it allows your application to authenticate your users or customers on behalf of your registered account.

Once you have your Client ID and Client Secret, you will use those along with the OAuth URLs shown earlier to setup Authentication in the Connector.

Information Tab

Open up Connector Builder by clicking on the Connectors Tab, and selecting Connector Builder -> Create.

It will default to the Information tab (video), enter the name of the Connector, e.g. “Mailchimp Video V3”. Notice that it has filled in the Connector Key name for you. Add in a description that will be displayed with this Connector in the Catalog, I’ll use “Demo Connector SAP V3”.

Next, you need to select which hub this Connector belongs in. In this case, we’ll select “Marketing” as that makes the most sense.

TIP: When determining what Hub to place a new Connector into, consider the resources that you will be exposing. If your focus is Contacts and Lists, then Marketing makes the most sense. If you are building a CRM Connector that will be exposing Contacts, Leads, Accounts, Opportunities, etc, then CRM will make the most sense. If you are unsure, you can create a new Hub by typing in its name or you can place it in a General category.

The reason for placing it into a hub is two fold, 1) it can help to produce a list of normalized resources automatically for you if you select a hub – e.g. /lists and 2) it help to organize Connectors by Hub type and open them up for searching by Hub in the Connectors catalog.

Next you will need to select the type of service – earlier we determined that Mailchimp V3 it is REST – which is the default. We also determined previously that MailChimp V3 supports OAuth2.0.

The documentation URL is handy for the next person who picks up your Connector and wants to make modification or additions. Put in the API version – you can find that in the documentation and copy / paste it under Vendor API version, and upload a logo for the endpoint.

Here is what it looks like when completed:

Click on Save & Next it will take all this information and move you to the next tab called Setup.

Setup: Properties and Authentication

The Setup tab has Properties, Authentication, Configuration, Parameters, Hooks, and Events. I’ll leave it to you to read through the descriptions of each of these sections to determine the details of what they do – but first, lets focus on Properties and Authentication (video for this section).

Enter our base URL (video) that we determined earlier – which is what the Connector will apply as it makes requests to MailChimp – note that we need to add the “us9” to this for my mailchimp account, yours may be different.

Set the Pagination Type (video). Since paging is built into each of the Connectors, we need to tell it what type of paging is used by the endpoint service so it can be standardized within the REST resources associated with the Connector.

For Mailchimp, the way to do this is to go in a find a resource like GET /lists in their documentation, and scroll down and you’ll see parameters for “count” and “offset”, thus it is very likely offset paging, so we’ll need to set that value. Interestingly enough, with Mailchimp, with the “count” parameter, it is listed as a read-only – so it is something that is not settable. We’ll come back to this when we define the resources and adding paging.

Scroll down to set up the Authentication where the Oauth values for ClientID and Secret from registering our app will come into play. Thus for mailchimp oauth, we’ll fill in the callback, and the Client ID and Client Secret areas (video).

The value for the callback will be set to go back to the Cloud Connectors default callback URL so that the platform can catch the callback and react to it appropriately. For the Authorization URL we’ll go back to the documentation and pull it from there, copy it, and paste that in place.

The other one we are going to need is the access token URL, so back to the documentation and copy it, then paste it in place as well. Now you need your key and secret for your registered app, so copy those and paste them in place.

There are other defaults that are set as a default for you via the Connector Builder. For example, many endpoints allow you to set the authentication scope (video), i.e. read, write, etc. It will vary by endpoint. Mailchimp doesn’t allow you to set scope,

Thus it is not needed and you can clear the default value.

MailChimp is unique in that, if you look at their example POST requests, you’ll see they’re passing in the Client ID, Client Secret, and Redirect URL as part of the “data” portion (see above) rather than how it is typically passed via the standard OAuth header as part of the authentication handshake. Given that Connector Builder will assume the standard, you will also need to set the Oauth Basic Header to “false” so that the Connector doesn’t assume to use the standard basic header (video).

The rest of these parameters are preset. For example, the Accept and Content Type are good to go. Also we’re not going to turn on events yet so let’s leave that for now. Completed auth for our Mailchimp connector should look like this:

Good time to click Save:

Let’s test that authentication works properly and that a credentialed instance can be created. To do this, just click “Try it out” (video).

Enter in a name for this instance – so something like “Cool Customer’s Mailchimp Account”.

Tip. The default setting for OAuth API Secret is to treat it like a password, hence it’s Type defaulting to “password”. If you leave the default, you will need to re-enter it when creating the instance (click on Show Optional Fields to reveal the field to enter the secret when creating the instance). I like to change it to “text 128” when I’m first testing the Connector so that it is filled in for me automatically.

Click Create Instance, which will take you right through the process of logging in just like you would expect, essentially this is building in authentication into the Connector and standardizing it to a set of normalized REST calls that is consistent with other Connectors.

If you need to debug the authentication transaction between the Connector and MailChimp, you can click in and see the details – including the return values, etc. So view your instance – and start adding resources, click on Save & Next.

Resources: Add New

To add resources, click on the Resources Tab, to expose the resources page. On the left is the credentialed Connector Instance that we created. This is connected to our MailChimp account in the same way that it would be connected to your Customer’s MailChimp account so that the Connector can access their lists and other data items.

In the MailChimp documentation, you’ll find the information regarding the /lists resources (video). As we inspect the documentation of /list resource, we determine that MailChimp provides for:

“Create” action – i.e. POST, POST with ID.
“Read” action with GET, GET with ID.
“Edit” action with PATCH by ID
“Delete” action with DELETE by ID.

Click Add a New Resource (video). There are two “sides” per se, the Connector’s exposed normalized resource and the endpoint or Vendors resource that it is mapped to. For /lists, let’s expose it a /lists on the Connectors side, and, conveniently it is also /lists on the vendor side.

Now we can let the Connector Builder do the work for us, and select which methods that we want it to populate. If you recall, there is no PUT method supported by MailChimp for lists, so we can un-check that box. But the others are valid as MailChimp supports GET, POST, PATCH, and DELETE. Also, there is a GET by ID, we let’s keep that one too.

What nice about this is that the Connector Builder provides for this standard across all Connectors. Once you are ready, click Go and Connector Builder will create and populate the base set of resources for /lists in the Connector.

Hint: Click Save to reset the canvas.

Click on the GET /lists resource and test to see if this works properly as setup by Connector Builder (video) – choose our credentialed Connector Instance from the left. That will give us the Try it out capability.

Click Try it out and note that it presets the Authorization header values including the User and Org secrets along with the selected Connector Instance’s token. We can run our resource and test it by clicking on Execute.

It works! There is a sample CURL command which a developer can leverage when trying out your new Connector by using Postman and the response JSON payload, our starting point which we’ll tune up a bit soon.

Next, we’ll need to set the root array as with the MailChimp response for GET lists, we can see that there are clear two arrays of information returned (video), “_links” and “lists”:

However, the standard for an Connector is to return a single root array in a response. Thus, we need to instruct the Connector to only return the lists portion of the response, otherwise we won’t be able to hook up events, webhooks, and other features that depend on a single response array structure. The way this is accomplished in Connector Builder is to, under configuration, where it says Response Root Key, just type in the word “lists” and click Save (you may need to cancel your try it out response first in order to expose the configuration portion of the GET /lists resource – video).

Now try it out and execute and this will run and produce the expected result now at the proper level.

There you have it! You’ve added the GET /list resource.

Resources: Add Parameters

In Mailchimp’s documentation for GET /lists, there are parameters that can be added to your Connector’s GET /lists resource to help modify the result set:

Paging
Count
Offset (thus the offset paging that selected during Setup).

There is also the ability to filter by email, date created, etc. – as a best practice, you’ll want to not limit the users of your Connector so all of these should be added in.

I’m only going to add the “email” parameter as an example (video), and I’m also also going to show you how to do the set of 3 standard Connector parameters for a GET resource call – which are:

page
pageSize
where (i.e. CEQL)

In Connector builder, if you are still on the try it out for the lists root resource, click Cancel

otherwise click Edit

Select Add a parameter -> empty

Enter “email” on both the Connector and Vendor side, set it’s Type to “query” on both sides as it is the same, and set the Data Type as “string” on both sides, and drop in that parameter description – which I usually copy and paste directly from the vendor’s documentation, click Save.

Now Try it out to make sure that we did this correctly (video) – using a value for an email in one of your test MailChimp lists, in my example: joebob123@fake.com.

Test the Connector using the test email, Execute it:

Let’s add in the standardized CEQL query (the where clause) that users of your Connector will expect to be standardized. If you recall, CEQL is the Connector’s Query Language and is very similar to SQL. You do that by clicking Cancel

then on Add Parameter and selecting where (video):

In addition, users of your new MailChimp V3 Connector will expect the standardized paging parameters, page and pageSize. So, let’s add these in now in the same way ending up with:

For page there is a slight Mailchimp nuance that we need to overcome. We know that when we look at the documentation for GET /lists, the page was “offset” on the vendor side, and the pageSize was “count” on the vendors side. But, when looking at the Mailchimp V3 documentation for GET /lists, the value for “count” has read-only = true.

The way to ensure that is to set the type to set “no-op” for the count parameter. Save, now it should look like this:

Test by clicking on Try it out, and instead of using the email parameter directly, we can enter any of MailChimp’s GET /lists query parameters using the CEQL where clause – thus we can put email = ‘joebob123@fake.com’, click Execute and it will function the same as if we used the email parameterabove (video).

Now you can use the where parameter for the other parameters that are available for GET /lists, it will automatically work as part of the Connector!

Add Webhook Based Events

Unfortunately, the way events work in any given endpoints API specification varies wildly. Mailchimp’s has nuances as well, and we’ll cover these here.

To accomodate for these variances, as well as, to make up for the fact that the majority of endpoint services in the cloud today that do not support any sort of native eventing at all, SAP’s Connectors all have an eventing framework so that each and every Connector can support events, even when an endpoint service is lacking.

In Connector builder, go to the Setup page for our MailChimp Connector build (video) and scroll down to Events, click on Enable Events. Be sure to select our credentialed instance that we created in earlier.

The eventing framework within an Connector can be configured to work in one of two ways, depending on what the actual endpoint supports. (click on event type to reveal the drop-down).

For example, if an endpoint supports webhooks, you can leverage those and configure the Connector to receive that webhook and add value to it before sending it out as an enhanced webhook out of the Connector.

The second way is to leverage the Connector’s polling framework and configure it to periodically query into the endpoint – leveraging the CEQL where clause capability that we added previously. The Connector, in this case, will query to see if there are changes to a given object in endpoint, and if it finds any, it will return a webhook. Thus, in a way, the Connector can ‘webhook-ify” a given endpoint – even it if it doesn’t support webhooks natively.

You will what to use the Connector’s polling capabilities, rather than its webhook capabilities for either of two situations: 1) if the endpoint doesn’t support webhooks at all, or 2) if the endpoint does support webhooks, but the process is clunky or requires advanced user interactions, then it maybe best to simplify buy automating the process using the Connector’s poller.

For this example, we’ll use MailChimp’s native webhooks, and enhance and normalize them so that they are consistent for the users of your Mailchimp Connector. Select the webhooks option, and let’s configure our webhook based event, and select Event Hook to expose the Javascript entry area where you can write JavaScript so it can accept the payload coming back native from the endpoint, and have the Connector convert that info a standard payload that is consistent with the other Connectors in the platform. You can see an example by clicking on Example.

I’m not going to take you through writing this javascript in this blog, but I will show you a trick (video) on how to get tested, live examples to use. Because the Connector Builder can also be used to view and extend existing Connectors in our public catalog, you can simply search for the Cloud Connectors’ public MailChimp Connector, click on My Resources and under Setup you’ll see that in the public catalog version, someone has already written the required JavaScript code that we need for our events.

Here is the code example as reference:

var formattedEvents = [];

var eventObj = {};

//for localhost key is coming as ‘data[list_id]’ hence to set id properly in localhost added below check

if(events.hasOwnProperty(‘data[list_id]’)){

eventObj.event_object_id=events[‘data[list_id]’];

}else{

var bodyData = events[‘data’];

eventObj.event_object_id = bodyData[‘list_id’];

}

var webhook_types = events.type;

var webhook_header = events.eventHeaders[‘user-agent’];

if(webhook_header === ‘MailChimp.com WebHook Validator’) {

done();

}

if(webhook_types === ‘unsubscribe’) {

eventObj.event_type = ‘UPDATED’;