Skip to Content
Technical Articles
Author's profile photo Chaitanya Priya Puvvada

SAP Intelligent RPA -How to read text from an image using Surface Automation

This blog post explains how to read text from an image using Surface automation.


What is Surface Automation?

A technique where image recognition and OCR are used to automate user Interfaces when DOM is not accessible at run time/design time.

It is the process of creating automation according to images shown on a screen.

It is used as an alternative when technology connectors are unavailable.

The Cloud Studio supports surface automation capture for the UI Automation technology.


Core SDK Version:1.18.15 and above

Desktop Agent Version: 2.0.18 and above

Display settings in your machine:100%


Capturing and Building automation to read text from an image.

Let’s take the Scoreboard image and read the text from it.

BEST Practices:

There are some best practices for users to follow when extracting text from images:


  1. For Machine-Readable or scanned documents, information can be extracted using Document Extraction Activities. For other documents:
    1. Using Predefined models(Invoices/Purchase Order/Payment Advice)Refer to the blog post.
    2. For Custom documents .
    3. For all the unstructured documents where text operations are required.
  2. Text from Images that do not fall in the above categories can be read through Surface Automation.

It is highly recommended to use Surface Automation in the scenarios where the text can’t be read through Document Extraction Activities.



Capturing the application.

1. Since Cloud Studio supports surface automation with UI AUtomation Technology, We need to capture the image using UIAUtomation Technology.

The screen is captured and looks like below

2 Selecting Surface Automation

Once the screen is captured, We need to capture it again by adding it as  Subscreen to the main screen and select the Technology as Surface Automation.


3. Capturing and Declaring the elements.

I would like to read the text Srilanka, hence I capture the element using the Select Zone option and click on Declare element

To confirm if the element is captured correctly, check if the text that you would like to read from the image is available in captured data.


4. Settings for the Subscreen.

Maintain the OCR Polling delay of 2 sec. It is the time taken by the screen to recognize the elements.


4. Create an Automation

Create automation, drag and drop the Subscreen and use the required Get Element Activity to read the data.

The final automation looks as below.


Test the Automation.

The Image opens and reads the TeamName Srilanka.


To know more about Surface automation, refer to the video on Surface Automation.

For more details, refer to the documentation. 



Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Prasanth Padmanabhan Menon
      Prasanth Padmanabhan Menon

      Hi Priya,


      Could this be used for images with handwritten texts?


      Thanks and Regards,


      Author's profile photo Chaitanya Priya Puvvada
      Chaitanya Priya Puvvada
      Blog Post Author

      It can not be used.