Handling XML’s is one of the most common thing we do in PI, use of java mappings to do that or writing logic in UDF might be day to day activity in any PI development projects. Before using any of the below import statement we generally don’t think on how those import statements will be loaded or what goes on behind the scene as these are very basic JAVA statements, no big deal isn't it.
Let’s take one simple example of DocumentBuilder:
In order to parse any XML, building document object would be one of the first thing we do (Except when parsing is done using methods which uses string based patters ex REGEX).
Below is sample code for creating document object from input XML string, the statement which provides document builder object is
“DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();“
We will be importing below relevant class to use above statement.
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
As these class are part of JRE (rt.jar) we won’t generally do anything further to use them like loading third party library for this object mostly we would just continue with
interface logic either in UDF or JAVA Mappings.
Below code shows most common way to get document builder object created using string object (data coming into mapping)
Let’s analyse what happens when we use
import javax.xml.parsers.DocumentBuilderFactory in our program.
Our intention is obtain new instance of DocumentBuilderFactory with which we can create document builder object to parse XML, as XML parser objects,transformers (XSLT) have a
standard interface and there may be different implementations of this interface from different vendors, open-source projects so we are in a way telling we want to use DocumentBuilderFactory by import statement but without specifying which one to use.
we are letting SAP PI to figure out.
Let’s take two options to load DocumentBuilderFactory.
1. com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl (Ex: rt.jar both in ORACLE and SAP JVM).
2. org.apache.xerces.jaxp.DocumentBuilderFactoryImpl (Ex: xercesImpl.jar)
First one is default provided by JRE irrespective of which vendor JVM you use, other one is one of the third party provider.If we don’t use xerces library or any 3rd party library which I guess most of us don’t, then loading default JRE class should be simple enough isn’t not?
After import statement in our code, class loader needs to know which one of above two should be used & since we are letting SAP PI class loaders still doesn't know which one to pick so it goes through below 4 steps or options to figure out.
This is where it will get interesting.
Below are the steps which gets executed to decide which Implementation class to use:
Step 1:
Class loader can uses javax.xml.parsers.DocumentBuilderFactory system property to know which implementation class to use, if this is not set, it moves to step 2.
Step 2:
Class loader will uses the properties file "lib/jaxp.properties" in the JRE directory.
in SAP PI JRE directory is set by (java.home= usr/sap/<SID>/J40/exe/sapjvm_8/jre) This configuration file is in standard java.util.Properties format and contains the fully
qualified name of the implementation class with the key being the system property defined above.
If the file does not exist when the first attempt is made to read from it, no further attempts are made to check for its existence. If Step 2 is not specified it will go to Step 3.
Step 3:
Class loader uses the Service API, if available, to determine the class name.
The Services API will look for a class name
META INF/services/javax.xml.parsers.DocumentBuilderFactory
in jars available to the runtime. If it cannot find anything from Step 3, then it goes to Step 4 which is what happens in most of the cases.
Step 4:
Platform Default DocumentBuilderFactory instance.
In most cases we don’t set Step 1 of setting system property nor Step 2 of setting jaxp.properties in JRE folder to specify which implementation class to be used during
runtime. So let’s see what Step 3 does in SAP PI.
Step 3 & Step 4 is what we are interested in so let see what goes on in this bit more.
Step 3 Execution details:
Service API in SAP PI is different than usual execution, when mapping class loader is executed it tries to run in the order from transform(), you can see execution order of class loader,
I have only listed few steps in this example, full details you can see in the picture belo
FactoryFinder.find() method tries run serviceLoader to see if it can find the jar as step 1 and step 2 is not specified, its basically looking for any jar which has entries
“META-INF/services/javax.xml.parsers.DocumentBuilderFactory”.
It checks in two possible places:
1. In the same path as jar for JAVA mapping was loaded (Imported Archive), you could just load third party jar in same imported archive location.
2. If the third party jar is not loaded in the same Imported Archive location of Java Mapping then it goes to database table XI_RUNMAPPINGS to find the jar which has entries
“META-INF/services/javax.xml.parsers.DocumentBuilderFactory”, in a way XI_RUNMAPPINGs holds information about all imported jar files.
select statement running in table XI_RUNMAPPINGS are with below parameters.
• Parameter 1 = META-INF/services/javax.xml.transform.transformerfactory
• Parameter 2 = Namespace of the interface where mappings are created.
• Parameter 3 = -1
• Parameter 4 = 89447120FF8711E58ADDCFBE0AF0F062 (GUID of SWCV)
Note: (Turn on SQL tracing when you run interface to see these entries as this is direct DB call, connection is not made using data sources)
It’s just not single select statement, based on how many SWCV dependencies are created for the SWCV from which mapping is executed, search is made in all of these ones to find
jar, in our case it was more than 4 select statements running to locate “META-INF/services/javax.xml.transform.transformerfactory”
Class Loader over right from java mapping code till it goes to DB:
To summarize Step 3 execution:
By not setting right system properties which informs specific implementation class to use or not loading them in Imported Achieve, SAP PI will make database calls to table XI_RUNMAPPING to find jar which has Implementation class names, this happens for every call, issue what is I see is not speed or slowness but the very thought of making DB call for basic XML object is not right & can be easily avoided.
Step 4 Execution Details:
If Step 3 doesn’t find entries from DB or imported Archive, job of step 4 is to see what’s the default implantation class available (it will find it from rt.jar)
By using below code you can find the values
As default value is com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl we can very well set system property
as start-up property in JVM.
In our project we had various mappings code which were going to DB table XI_RUNMAPPINGS, which could have been avoided by setting below system property or by loading relevant jar files
in imported archive.
Possible System Property which will stop DB calls in our case:
Also, if you have DB lookup in PI using data source looked up using then there will be DB calls to XI_RUNMAPPINGS to find JNDI.PROPERTIES location file
In order to know which Context object to use, you will find select statement running on table XI_RUNMAPPINGS with below values
Parameter 1 = jndi.properties
Parameter 2 = -1
Parameter 3 = F6D14280807E11E295B5E1190AF0F034
Summary:
Since SAP has simplified PI installation right from old days where we used to create service users with roles, RFC destination manually to current simplified ways where we have single click to complete all post installation steps, they should also include few system properties as listed above as part of post installation to stop this behavior.
For every time message is executed & every time JAVA mapping is executed which has DocumentBuilderFactory creation or other known cases listed above, Idea to go to database level to locate class is very difficult to justify.
There might be many more Implementation class calls going to DB, in our debugging I have come across above mentioned list so best is to check how each import statement
works in PI.
Let’s take one simple example of DocumentBuilder:
In order to parse any XML, building document object would be one of the first thing we do (Except when parsing is done using methods which uses string based patters ex REGEX).
Below is sample code for creating document object from input XML string, the statement which provides document builder object is
“DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();“
We will be importing below relevant class to use above statement.
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
As these class are part of JRE (rt.jar) we won’t generally do anything further to use them like loading third party library for this object mostly we would just continue with
interface logic either in UDF or JAVA Mappings.
Below code shows most common way to get document builder object created using string object (data coming into mapping)
public static Document getDocumentFromString(String inStr) {
Document doc = null;
InputStream is = null;
try {
try {
is = new ByteArrayInputStream(inStr.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
is = new ByteArrayInputStream(inStr.getBytes());
}
DocumentBuilder builder =
DocumentBuilderFactory.newInstance().newDocumentBuilder();
doc = builder.parse(is);
} catch (ParserConfigurationException e) {
} catch (IOException e) {
} catch (SAXException e) {
}
return doc;
}
Let’s analyse what happens when we use
import javax.xml.parsers.DocumentBuilderFactory in our program.
Our intention is obtain new instance of DocumentBuilderFactory with which we can create document builder object to parse XML, as XML parser objects,transformers (XSLT) have a
standard interface and there may be different implementations of this interface from different vendors, open-source projects so we are in a way telling we want to use DocumentBuilderFactory by import statement but without specifying which one to use.
we are letting SAP PI to figure out.
Let’s take two options to load DocumentBuilderFactory.
1. com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl (Ex: rt.jar both in ORACLE and SAP JVM).
2. org.apache.xerces.jaxp.DocumentBuilderFactoryImpl (Ex: xercesImpl.jar)
First one is default provided by JRE irrespective of which vendor JVM you use, other one is one of the third party provider.If we don’t use xerces library or any 3rd party library which I guess most of us don’t, then loading default JRE class should be simple enough isn’t not?
After import statement in our code, class loader needs to know which one of above two should be used & since we are letting SAP PI class loaders still doesn't know which one to pick so it goes through below 4 steps or options to figure out.
This is where it will get interesting.
Below are the steps which gets executed to decide which Implementation class to use:
Step 1:
Class loader can uses javax.xml.parsers.DocumentBuilderFactory system property to know which implementation class to use, if this is not set, it moves to step 2.
Step 2:
Class loader will uses the properties file "lib/jaxp.properties" in the JRE directory.
in SAP PI JRE directory is set by (java.home= usr/sap/<SID>/J40/exe/sapjvm_8/jre) This configuration file is in standard java.util.Properties format and contains the fully
qualified name of the implementation class with the key being the system property defined above.
If the file does not exist when the first attempt is made to read from it, no further attempts are made to check for its existence. If Step 2 is not specified it will go to Step 3.
Step 3:
Class loader uses the Service API, if available, to determine the class name.
The Services API will look for a class name
META INF/services/javax.xml.parsers.DocumentBuilderFactory
in jars available to the runtime. If it cannot find anything from Step 3, then it goes to Step 4 which is what happens in most of the cases.
Step 4:
Platform Default DocumentBuilderFactory instance.
In most cases we don’t set Step 1 of setting system property nor Step 2 of setting jaxp.properties in JRE folder to specify which implementation class to be used during
runtime. So let’s see what Step 3 does in SAP PI.
Step 3 & Step 4 is what we are interested in so let see what goes on in this bit more.
Step 3 Execution details:
Service API in SAP PI is different than usual execution, when mapping class loader is executed it tries to run in the order from transform(), you can see execution order of class loader,
I have only listed few steps in this example, full details you can see in the picture belo
com.sap.aii.ib.server.mapping.execution.Mapping.execute-->
com.sap.aii.ib.server.mapping.execution.JavaMapping.executeStep-->
com.test.sampleJavaMapping.transform-->
com.test.sampleJavaMapping. getDocumentFromString-->
javax.xml.datatype.FactoryFinder.findFactoryFinder.find() method tries run serviceLoader to see if it can find the jar as step 1 and step 2 is not specified, its basically looking for any jar which has entries
“META-INF/services/javax.xml.parsers.DocumentBuilderFactory”.
It checks in two possible places:
1. In the same path as jar for JAVA mapping was loaded (Imported Archive), you could just load third party jar in same imported archive location.
2. If the third party jar is not loaded in the same Imported Archive location of Java Mapping then it goes to database table XI_RUNMAPPINGS to find the jar which has entries
“META-INF/services/javax.xml.parsers.DocumentBuilderFactory”, in a way XI_RUNMAPPINGs holds information about all imported jar files.
select statement running in table XI_RUNMAPPINGS are with below parameters.
• Parameter 1 = META-INF/services/javax.xml.transform.transformerfactory
• Parameter 2 = Namespace of the interface where mappings are created.
• Parameter 3 = -1
• Parameter 4 = 89447120FF8711E58ADDCFBE0AF0F062 (GUID of SWCV)
Note: (Turn on SQL tracing when you run interface to see these entries as this is direct DB call, connection is not made using data sources)
It’s just not single select statement, based on how many SWCV dependencies are created for the SWCV from which mapping is executed, search is made in all of these ones to find
jar, in our case it was more than 4 select statements running to locate “META-INF/services/javax.xml.transform.transformerfactory”
Class Loader over right from java mapping code till it goes to DB:
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(ZZZZZI)V(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(ZZZZZ)V(T4CPreparedStatement.java:53)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(Z)V(T4CPreparedStatement.java:943)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe()V(OracleStatement.java:957)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout()V(OracleStatement.java:1111)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal()I(OraclePreparedStatement.java:4798)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery()Ljava/sql/ResultSet;(OraclePreparedStatement.java:4845)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery()Ljava/sql/ResultSet;(OraclePreparedStatementWrapper.java:1501)
at com.sap.sql.jdbc.basic.BasicPreparedStatement.executeQuery()Ljava/sql/ResultSet;(BasicPreparedStatement.java:102)
at com.sap.sql.jdbc.oracle.Oracle12cPreparedStatement.executeQuery()Ljava/sql/ResultSet;(Oracle12cPreparedStatement.java:154)
at com.sap.sql.jdbc.direct.DirectPreparedStatement.executeQuery(Lcom/sap/sql/trace/TraceContext;)Ljava/sql/ResultSet;(DirectPreparedStatement.java:315)
at com.sap.sql.jdbc.common.CommonPreparedStatement.executeQuery()Ljava/sql/ResultSet;(CommonPreparedStatement.java:245)
at com.sap.engine.services.dbpool.wrappers.PreparedStatementWrapper.executeQuery()Ljava/sql/ResultSet;(PreparedStatementWrapper.java:379)
at com.sap.aii.utilxi.sql.api.StatementWrapper.executeQuery()Lcom/sap/aii/utilxi/sql/api/ResultSetWrapper;(StatementWrapper.java:130)
at com.sap.aii.utilxi.sql.api.AbstractStatement.executeQuery(Lcom/sap/aii/utilxi/sql/api/DBConnection;)(AbstractStatement.java:128)
at com.sap.aii.utilxi.sql.api.ResultSetIterator.<init>(Lcom/sap/aii/utilxi/sql/api/DBConnection;Lcom/sap/aii/utilxi/sql/api/Selector;II)V(ResultSetIterator.java:35)
at com.sap.aii.utilxi.sql.api.Selector.getLineIterator(Lcom/sap/aii/utilxi/sql/api/DBConnection;II)Ljava/util/Iterator;(Selector.java:110)
at com.sap.aii.utilxi.sql.api.Selector.select(Lcom/sap/aii/utilxi/sql/api/DBConnection;IZ)Ljava/util/Iterator;(Selector.java:72)
at com.sap.aii.utilxi.sql.api.SelectClause.select(Lcom/sap/aii/utilxi/sql/api/DBConnection;IZ)Ljava/util/Iterator;(SelectClause.java:46)
at com.sap.aii.utilxi.sql.api.SelectClause.select(Lcom/sap/aii/utilxi/sql/api/DBConnection;I)Ljava/util/Iterator;(SelectClause.java:50)
at com.sap.aii.ibrun.server.mapping.persist.DBMappingPersistor.getResource(Lcom/sap/guid/IGUID;ILjava/lang/String;Ljava/lang/String)(DBMappingPersistor.java:99)
at com.sap.aii.ib.server.mapping.execution.InternalMappingFinder.getInputStream(Ljava/lang/String;Lcom/sap/guid/IGUID;ILjava/lang/String;)(InternalMappingFinder.java:102)
at com.sap.aii.ib.server.mapping.execution.InternalMappingFinder.readFile(Ljava/lang/String;Lcom/sap/guid/IGUID;ILjava/lang/String;)(InternalMappingFinder.java:62)
at com.sap.aii.ib.server.mapping.execution.MappingLoader.findResources(Ljava/lang/String;)Ljava/util/Enumeration;(MappingLoader.java:354)
at java.lang.ClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration;(ClassLoader.java:1246)
at java.lang.ClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration;(ClassLoader.java:1242)
at java.util.ServiceLoader$LazyIterator.hasNextService()Z(ServiceLoader.java:348)
at java.util.ServiceLoader$LazyIterator.hasNext()Z(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext()Z(ServiceLoader.java:474)
at javax.xml.datatype.FactoryFinder$1.run()Ljava/lang/Object;(FactoryFinder.java:296)
at java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Native Method)
at javax.xml.datatype.FactoryFinder.findServiceProvider(Ljava/lang/Class;)Ljava/lang/Object;(FactoryFinder.java:292)
at javax.xml.datatype.FactoryFinder.find(Ljava/lang/Class;Ljava/lang/String;)Ljava/lang/Object;(FactoryFinder.java:268)
at javax.xml.datatype.DatatypeFactory.newInstance()Ljavax/xml/datatype/DatatypeFactory;(DatatypeFactory.java:145)
at com.test.javamapping.xmlLoader.getDocumentFromString(Lorg/w3c/dom/Document;L)Lcom/test/javamapping/xmlLoader;(xmlLoader.java:91)
at com.test.javamapping.xmlLoader.transform(Lcom/sap/aii/mapping/api/TransformationInput;Lcom/sap/aii/mapping/api/TransformationOutput;)V(xmlLoader.java:213)
at com.sap.aii.ib.server.mapping.execution.JavaMapping.executeStep(Lcom/sap/aii/ib/server/mapping/execution/TransformationData;)V(JavaMapping.java:112)
at com.sap.aii.ib.server.mapping.execution.Mapping.execute(Lcom/sap/aii/ib/server/mapping/execution/TransformationData;)V(Mapping.java:60)
XI_RUNMAPPINGS table structure:
Class loader information from java mapping till database call:
To summarize Step 3 execution:
By not setting right system properties which informs specific implementation class to use or not loading them in Imported Achieve, SAP PI will make database calls to table XI_RUNMAPPING to find jar which has Implementation class names, this happens for every call, issue what is I see is not speed or slowness but the very thought of making DB call for basic XML object is not right & can be easily avoided.
Step 4 Execution Details:
If Step 3 doesn’t find entries from DB or imported Archive, job of step 4 is to see what’s the default implantation class available (it will find it from rt.jar)
By using below code you can find the values
Class componentClass = null;
String componentName = null;
String result = null;
componentClass = DocumentBuilderFactory.newInstance().getClass();
componentName = "DocumentBuilderFactory";
CodeSource source = componentClass.getProtectionDomain().getCodeSource();
result = MessageFormat.format("{0} implementation: {1} loaded from: {2}",
componentName,
componentClass.getName(),
source == null ? "Java Runtime" : source.getLocation());
System.out.println(result);
Output of code: DocumentBuilderFactory implementation: com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl loaded from: Java RuntimeAs default value is com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl we can very well set system property
javax.xml.datatype.DatatypeFactory = com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl as start-up property in JVM.
In our project we had various mappings code which were going to DB table XI_RUNMAPPINGS, which could have been avoided by setting below system property or by loading relevant jar files
in imported archive.
Possible System Property which will stop DB calls in our case:
javax.xml.datatype.DatatypeFactory = “com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl"
javax.xml.parsers.DocumentBuilderFactory= "com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl"
javax.xml.parsers.SAXParserFactory= "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"
javax.xml.transform.TransformerFactory= "com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl"
javax.xml.soap.MessageFactory = com.sun.xml.messaging.saaj.soap.ver1_1.SOAPMessageFactory1_1Impl
javax.xml.soap.MetaFactory = com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImplAlso, if you have DB lookup in PI using data source looked up using then there will be DB calls to XI_RUNMAPPINGS to find JNDI.PROPERTIES location file
public Connection getDBConnection(String dataSource,String MSGID) throws StreamTransformationException
{
String value = "Okay";
Connection dbAccess = null;
Context ctx = null;
DataSource ds = null;
try {
ctx = new InitialContext();
if (ctx == null)
{
value = "Error - No Context";
}
ds = (DataSource) ctx.lookup(dataSource);
if (ds == null)
{
value = "Error - No dataSource: " + dataSource;
}
dbAccess = ds.getConnection();
}
catch (Exception e){
throw new StreamTransformationException("Errorx(getDBConnection):"+e.getMessage());
}
return dbAccess;
}In order to know which Context object to use, you will find select statement running on table XI_RUNMAPPINGS with below values
Parameter 1 = jndi.properties
Parameter 2 = -1
Parameter 3 = F6D14280807E11E295B5E1190AF0F034
Summary:
Since SAP has simplified PI installation right from old days where we used to create service users with roles, RFC destination manually to current simplified ways where we have single click to complete all post installation steps, they should also include few system properties as listed above as part of post installation to stop this behavior.
For every time message is executed & every time JAVA mapping is executed which has DocumentBuilderFactory creation or other known cases listed above, Idea to go to database level to locate class is very difficult to justify.
There might be many more Implementation class calls going to DB, in our debugging I have come across above mentioned list so best is to check how each import statement
works in PI.
- SAP Managed Tags:
5 Comments
