In this blog I will be explaining how to use the User_Defined Transform in DataServices 4.x for accessing Web Services.
The User-Defined transform provides you with custom processing in a data flow using full Python scripting language. It can do just about anything that you can write Python code to do. You can use the User-Defined transform to generate new records, populate a field with a specific value, create a file, connect to a website, or send an email, just to name a few possibilities. User Defined Transforms comes under the Data Quality Transforms.
Here I’m demonstrating a process where I will be accessing a web service and mapping the response and from the web service to a target table using the user-defined transform. i.e. key field coming from the source (hard-coded value) is passed to the web service and the required output is extracted from the web service response.
In this process we are using a simple restful web service that will be accepting the key value from the source as a parameters
http://Server_Path: parameter1= value1? parameter2=”Value2”
Output of the web Service is a simple XML Structure which will be like
<name>Output Need to Mapped to Target</name>
Step by Step Details of Process Followed
1) Create a batch job and build the scenario in the Dataflow. For our scenario we have two input rows and value of the key field that need to be passed to the user defined transform is explicitly provided using the Query transform.
2) Map the input to the User Defined transform. Open the User-Defined transform and map input fields from the schema in to the schema out of the Transform which need to be directly mapped to the target . In our case we can map the MAT_KEY field in the input schema to the schema out of the transform.
2) 3) To use input field in a Python expression in your User-Defined transform, we first need to map it to a recognized field name in the Input tab. For that Drag and drop the input field which need to be used in the User-Defined editor into the input tab of the Editor. So are we mapping the input field MATNR into the input tab of the user-defined transform.
3) 4) In User-Defined user cannot edit options in the User-Defined transform editor. To edit options, you must use the User-Defined editor, which is accessed from the Options tab in the User-Defined transform editor.
4) 5) The User-Defined transform contains options that determine how the transform processes data. There are two modes to process the data they are
Per collection: Applies the expression to entire data collection. Use this option when adding new records, which did not exist before, into the data flow. Selecting this option displays the Group Forming option group, in which you set up break groups and candidate selection.
Per record: Applies the expression to each record. You cannot add new records into the data flow with this option. This option is the default and what you will want to use most often.
In this demonstration we are using the per record Data processing mode
5) 6) Click the Launch Python Editor button to access the Python Expression editor.
6) 7) In the Python Expression editor, create and edit Python code in the editor pane of the window. The Python expression that you create here depends on what you need to do with User-Defined transform. Python Expression editor contain The Python API tab which list the objects, functions, classes, and methods that are available for the specific transform and processing mode.
And it also contains the I/O Fields tab displays the input fields and output fields that have been mapped in the User-Defined transform. we can also add, delete, and edit the properties of user-defined input and output fields from this tab by right-clicking Input Fields or Output Fields and selecting Insert, Delete, or Properties.
7) 8) Next we will be creating an output field with a name DESC_OUT by right clicking the Output fields and selecting Insert. Output of the transform is mapped to this field in our python code.
8) 9) Python Expression editor is used to create our python expression for doing the required transformation logic. In our case we have to build the Web Service URL by appending the Input key field and call the web service for getting the response and extract the output field from the response. Please find below the python expression we built for handling the above mentioned process. More details explanation the python expression will be given in the part 2 for your reference
9) 10) After the creating the python expression we can validate the expression by clicking the Validate button in the Python Expression editor. The syntax checker makes sure that the Python code has required colon (:) characters, string literal closing characters and Correct indentation.
1011) On coming back to the User-defined transform you have specify the output fields that need to be mapped to the output of the transform. These are useful if you repeatedly use a transform with different and input and output fields
1112) Next we have to map the output of the User-Defined mapping to the target table and run the job. Here in my target table the field value coming from the user-defined transform is mapped to DESC field in the target.
1213) Please find the input and output generated
These are the Steps that are followed to access a web service using user-defined transform. By using the above approach we can access a web service which cannot be accessed using conventional ways available in the BODS.