SAP Intelligent RPA -How to read text from an image using Surface Automation

[Chaitanya_Priya]

This blog post explains how to read text from an image using Surface automation.

Introduction

What is Surface Automation?

A technique where image recognition and OCR are used to automate user Interfaces when DOM is not accessible at run time/design time.

It is the process of creating automation according to images shown on a screen.

It is used as an alternative when technology connectors are unavailable.

The Cloud Studio supports surface automation capture for the UI Automation technology.

Pre-requisites

Core SDK Version:1.18.15 and above

Desktop Agent Version: 2.0.18 and above

Display settings in your machine:100%

 

Capturing and Building automation to read text from an image.

Let's take the Scoreboard image and read the text from it.

BEST Practices:

There are some best practices for users to follow when extracting text from images:

 

1. For Machine-Readable or scanned documents, information can be extracted using Document Extraction Activities. For other documents:
   
   
   
   1. Using Predefined models(Invoices/Purchase Order/Payment Advice)Refer to the blog post.
   
   
   2. For Custom documents .
   
   
   3. For all the unstructured documents where text operations are required.
   
   
   
   


2. Text from Images that do not fall in the above categories can be read through Surface Automation.

It is highly recommended to use Surface Automation in the scenarios where the text can't be read through Document Extraction Activities.

 

 

Capturing the application.

1. Since Cloud Studio supports surface automation with UI AUtomation Technology, We need to capture the image using UIAUtomation Technology.

The screen is captured and looks like below

2 Selecting Surface Automation

Once the screen is captured, We need to capture it again by adding it as  Subscreen to the main screen and select the Technology as Surface Automation.

 

3. Capturing and Declaring the elements.

I would like to read the text Srilanka, hence I capture the element using the Select Zone option and click on Declare element

To confirm if the element is captured correctly, check if the text that you would like to read from the image is available in captured data.



 

4. Settings for the Subscreen.

Maintain the OCR Polling delay of 2 sec. It is the time taken by the screen to recognize the elements.

 

4. Create an Automation

Create automation, drag and drop the Subscreen and use the required Get Element Activity to read the data.

The final automation looks as below.

 

Test the Automation.

The Image opens and reads the TeamName Srilanka.

 

To know more about Surface automation, refer to the video on Surface Automation.

For more details, refer to the documentation.