Introduction
I have discovered recently Rossum.ai, a Data Capture tool based on artificial intelligence. This is not OCR, but an AI engine which detects data fields and annotate scanned documents. My explanation does not worth the official documentation.
I really like the freemium approach, where you can test and play with in order to evaluate. This is quite uncommon, as Invoice Capture products in SAP landscape are usually untouchable before you actually start commercial discussions with sales representatives.
Their API is simple, yet efficient and well-documented, and you can start playing with SAP integration in no time.
We will implement a simple demo use case as follows:
|
|
|
|
The final process will look like:
In order to accomplish it, we will have to go through these implementation steps:
- Have all our accounts and settings ready in both Rossum and S/4 (of course)
- Analyze and understand API message payloads with a REST client (postman)
- Extend Rossum schema to add a custom field holding SAP supplier code
- Implement extension hook "validate": simple supplier matching to retrieve SAP supplier code from name
- Implement extension hook "save": transfer invoice to S/4HANA
Setup
Rossum.ai
Our account is already usable out of the box after registration: we can upload invoices and check automatic annotations. No overwhelming setup or training is required, which is really huge time saving.
In order to use API and have access to extended option, we will install command line tool elisctl (installation instructions).
SAP S/4HANA
We need to make sure OData services for Partners et Supplier Invoices are activated in /IWFND/MAINT_SERVICES: API_BUSINESS_PARTNER and API_SUPPLIERINVOICE_PROCESS_SRV. I am not going to provide much more details, I assume we know how to use them.
API: understanding annotation structure
Rossum API is simple to use. Every request must be authorized by a token, which is provided as we login:
- Login and retrieve authentication token
Field value key in response payload is our authorization token, and must be provided in our request headers.
- Retrieve queues: in our case only one is available
- Retrieve annotations from queue
Annotation structure is exactly as on validation screen:
Extend schema with custom field "SAP supplier"
Schema can be extended with API calls or with elisctl. I have no preference so far and we will use command lines: at least it will (very slightly) simplify upload/download.
First, logon to our account using elisctl:
Welcome to the elisctl interactive mode. Start with `help` and `configure`.
elis> configure
API URL [https://api.elis.rossum.ai]:
Username: xxxx@yyy
Password: ******
Detected fields and annotations are declared in a schema. We need to extend our current schema so to make available one extra field in the annotation. We start by downloading it.
elis> queue list
id name workspace inbox schema users connector webhooks
----- -------------------- ----------- ---------------------------- -------- ------- --------------------------------------------- ----------
28056 Received EU invoices 25714 xxxxxxxxxxxxx@elis.rossum.ai 240106 35005 https://api.elis.rossum.ai/v1/connectors/2998
elis> schema get 240106 -O testschema.json
Then change it by adding a new datapoint.
Once done, We upload it back.
elis> schema update 240106 testschema.jsonOur new field is now available and visible in the settings
And voilà. It is directly available as an annotation field, and of course we can add as many custom fields as required: SAP material code, SAP company code, and much more. We will keep our demo simple though and have only one custom field.
The new added field is just a placeholder for now: Rossum is not going to magically determine my backend supplier code whereas its number does not appear on invoice. We could as well let it entirely manual, expecting end-user to fill it during validation step, but this would make no sense. This is where Rossum extension comes into play.
In the next step, We will declare a custom hook which will be triggered during validation: the so-called validate extension. Our extension will then call OData service API_BUSINESS_PARTNER and match a supplier code with the supplier name as recognized by Rossum. Actually we am going to reuse the partner search function we have created for our chatbot.
Implement custom hook validate
Declare a custom plugin: we use endpoint connector to post a new extension, which is no more than a name and an url. Pay attention to authorization token: it is our responsibility to authorize only request providing this token (up to us to decide on the token value/validation in our code). Though not recommended, we could as well fill any dummy value here and ignore authorization token in our custom code.
Now our extension is declared, we need to implement endpoint function /validate.
As explained in my earlier post, we will simply implement our hooks using Firebird Cloud Functions. Hassle-free service. We could do the same with AWS lambda.
The incoming request will post a payload message like this:
{
"meta": {
"document_url": "https://api.elis.rossum.ai/v1/documents/6780",
"arrived_at": "2019-01-30T07:55:13.208304Z",
... ...
"rir_poll_id": "54f6b9ecfa751789f71ddf12"
},
"content": [
{
"id": 133915359,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915359",
"children": [...],
"category": "section",
"schema_id": "invoice_info_section"
},
{
"id": 133915368,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915368",
"children": [...],
"category": "section",
"schema_id": "payment_info_section"
},
{
"id": 133915378,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915378",
"children": [...],
"category": "section",
"schema_id": "amounts_section"
},
{
"id": 133915386,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915386",
"children": [
{
"id": 137835747,
"url": "https://api.elis.rossum.ai/v1/annotations/1201234/content/137835747",
"content": {
"value": "MyDearSupplier",
"page": 1,
"position": [
889.0,
168.0,
1067.0,
210.0
],
"rir_text": "MyDearSupplier",
"rir_position": [
889.0,
168.0,
1067.0,
210.0
],
"rir_confidence": 0.6798204155690549,
"connector_position": null,
"connector_text": null
},
"category": "datapoint",
"schema_id": "sender_name",
"validation_sources": [],
"time_spent": 0.0
},
...,
{
"id": 137835750,
"url": "https://api.elis.rossum.ai/v1/annotations/1201234/content/137835750",
"content": {
"value": "",
"page": null,
"position": null,
"rir_text": "",
"rir_position": null,
"rir_confidence": null,
"connector_position": null,
"connector_text": null
},
"category": "datapoint",
"schema_id": "sap_supplier",
"validation_sources": [],
"time_spent": 0.0
},
...
],
"category": "section",
"schema_id": "vendor_section"
},
{
"id": 133915396,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915396",
"children": [...],
"category": "section",
"schema_id": "other_section"
},
{
"id": 133915398,
"url": "https://api.elis.rossum.ai/v1/annotations/1166616/content/133915398",
"children": [...],
"category": "section",
"schema_id": "line_items_section"
}
]
}As shown in this sample, we will find supplier name under payload.content[3].children[0] and SAP supplier code under payload.content[3].children[1]. Of course we are not going to fetch by index directly (too much bug prone), but in our simple custom code, we will loop over all datapoints in seek of supplier name by checking schema_id. Once we have it, we will call our backend OData service to match with SAP supplier number. Finally, we will update our custom datapoint SAP supplier code with the correct number by responding with message:
{
"messages": [
{
"content": "SAP Supplier <name> is matched with SAP number <sap_code>",
"id": "<datapoint_id (=137835750 in sample)>"
"type": "info"
}
],
"updated_datapoints": [
{
"id": "<datapoint_id (=137835750 in sample)>",
"value": "<sap_code>"
}
]
}
Sample implementation of /validate:
exports.validate = functions.https.onRequest((request, response) => {
console.log(request.body);
//response.sendStatus(200);
return parseRossumContent(request.body)
.then((content:any) => {
return response.json({
messages: [
{
content: "OK",
type: "info",
id: `${content.id}`
}
],
updated_datapoints: [
content
]
});
})
.catch((error: any) => {
// handle error
console.log(error);
})
});
where parseRossumContent function is:
function parseRossumContent(data:any):Promise<any> {
// Vendor name
let sender_name:string = ''
// SAP supplier annotation id
let sap_supplier_schema_id:string = ''
// Loop over all content lines until vendor_section
data.content.some((section:any) => {
// Found, now check its children
if(section.schema_id === 'vendor_section') {
// Loop over vendor_section.children until sender_name and sap_supplier
section.children.some((subsection:any) => {
if(subsection.schema_id === 'sender_name') {
sender_name = subsection.value;
}
else if(subsection.schema_id === 'sap_supplier') {
sap_supplier_schema_id = subsection.id
}
return (sender_name !== '' && sap_supplier_schema_id != '')
})
return true
}
return false
})
// At this stage, both sender_name and sap_supplier_schema_id are known
return searchSupplier(sender_name)
.then((sap_code:any) => {
return {
id: sap_supplier_schema_id,
value: sap_code
}
})
}Implement custom hook save
As with the previous section, we implement endpoint /save. In this part, we will integrate Rossum data as supplier invoice in S/4HANA.
Incoming payload has the same structure as for /validate, and no response payload is expected in case of successful processing, just HTTP status code 204.
To create an invoice in S/4 is as easy as posting OData service API_SUPPLIERINVOICE_PROCESS_SRV with the following payload:
{
"CompanyCode": "1710",
"DocumentDate": "/Date(1574208000000)/",
"PostingDate": "/Date(1574208000000)/",
"SupplierInvoiceIDByInvcgParty": "Vendor invoice number",
"InvoicingParty": "17300032",
"DocumentCurrency": "USD",
"InvoiceGrossAmount": "120000.00",
"to_SuplrInvcItemPurOrdRef": [
{
"SupplierInvoiceItem": "1",
"PurchaseOrder": "4500000004",
"PurchaseOrderItem": "10",
"Plant": "1710",
"TaxCode": "I1",
"DocumentCurrency": "USD",
"SupplierInvoiceItemAmount": "120000.00",
"PurchaseOrderQuantityUnit": "PC",
"QuantityInPurchaseOrderUnit": "30",
"PurchaseOrderPriceUnit": "PC",
"QtyInPurchaseOrderPriceUnit": "30"
}
]
}Our /save function is not very complex: we start by parsing annotation in a kind of key-value structure, then we call API in 2 steps, first GET to retrieve CSRF token and cookie, then POST to actually create the supplier invoice.
exports.save = functions.https.onRequest((request, response) => {
console.log(request.body);
const parsedContent = parseRossumContentForSave(request.body)
const url = "https://yourserver:443/sap/opu/odata/sap/API_SUPPLIERINVOICE_PROCESS_SRV/A_SupplierInvoice";
// Axios request options
let options = {
headers: {
"Authorization": "Basic xxx",
"Content-Type": "application/json",
"Accept": "application/json",
"x-csrf-token": "fetch",
},
params: {
'sap-client': '100',
'$format': 'json',
'$top': 0
},
xsrfHeaderName: 'x-csrf-token',
withCredentials: true
};
// Store invoice date in javascript format
const invoiceDate:Date = new Date(parsedContent.invoice_info_section.date_issue)
// Declare items to be filled, total amount will be the sum of item amounts
let items:any[] = []
let total = 0
let itemNum = 0
// Prepare items
parsedContent.items.forEach((item:any) => {
itemNum++
total += Number(item.item_amount_total.replace(/\s+/g, ''))
items.push({
SupplierInvoiceItem: itemNum.toString(),
PurchaseOrder: parsedContent.invoice_info_section.order_id,
PurchaseOrderItem: (itemNum*10) .toString(),
Plant: "1710",
TaxCode: "I1",
DocumentCurrency: parsedContent.amounts_section.currency.toUpperCase(),
SupplierInvoiceItemAmount: item.item_amount_total.replace(/\s+/g, ''),
PurchaseOrderQuantityUnit: "PC",
QuantityInPurchaseOrderUnit: item.item_quantity.replace(/\s+/g, ''),
PurchaseOrderPriceUnit: "PC",
QtyInPurchaseOrderPriceUnit: item.item_quantity.replace(/\s+/g, '')
})
})
// First GET request is to get CSRF token
axios.get(url, options)
.then((res:any) => {
console.log("Get OK")
console.log(res)
let invoice_json = {
CompanyCode: "1710",
DocumentDate: `/Date(${invoiceDate.valueOf()})/` ,
PostingDate: `/Date(${invoiceDate.valueOf()})/`,
SupplierInvoiceIDByInvcgParty: parsedContent.invoice_info_section.invoice_id,
InvoicingParty: parsedContent.vendor_section.sap_supplier,
DocumentCurrency: parsedContent.amounts_section.currency.toUpperCase(),
InvoiceGrossAmount: total.toString(),
to_SuplrInvcItemPurOrdRef: items
}
// CSRF token in header must be accompanied with Cookie. Not sure this is the best way to manage it ...
let cookie = ""
res.headers['set-cookie'].forEach((val:string) => {
cookie += val.split(";")[0] + ";"
})
return axios.post(url,invoice_json, {
headers: {
"Authorization": "Basic xxx",
"Content-Type": "application/json",
"Accept": "application/json",
"x-csrf-token": res.headers['x-csrf-token'],
'Cookie': cookie,
},
params: {
'sap-client': '100'
},
xsrfHeaderName: 'x-csrf-token',
withCredentials: true
})
})
.then(response.status(204))
.catch((error:any) => {
console.log(error.response)
})
});Conclusion
I am really impressed by the simplicity of use of rossum.ai, and I am glad they offer a trial we can easily play with. My demo was straightforward, but we can imagine more complex scenarios with higher integration points (supplier repository, purchase order, material, etc.) so to build a more complete and efficient tool for invoice recognition and matching. It really worths a try, especially in comparison to more complex, expensive and monolithic VIM tools we are used to plug to SAP ECC or S/4.
In my next post, I will explain how to combine Rossum.ai with Google DialogFlow Chatbot to enhance user interaction in our invoice receipt process.
Enjoy!
- SAP Managed Tags:
3 Comments
