Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

SAP HANA Vora - Troubleshooting

former_member229888
Explorer
‎12-09-2015 4:53 PM


This blog is deprecated. As of Vora 1.3 a Troubleshooting Guide has been added to the official documentation at http://help.sap.com/hana_vora

(last update January 2017)

---

This blog describes issues that could occur when installing, configuring or running SAP HANA Vora. It explains the root causes of those issues and if possible provides solutions or workarounds. The current Vora version is 1.2 (released
on March 31, 2016)

In case you run into issues or questions that cannot be resolved with this troubleshooting page and the documentation:

  • If you are an SAP customer, please open a customer message in component HAN-VO for Vora-related issues.

  • It is also possible to ask questions on stack overflow. Please tag your questions with tag 'vora'.
    This community-based help is primarily for non-SAP customers using the Vora Dev Edition in AWS.

  • For issues related to the HANA Spark Controller that is not connecting to Vora please open a customer message in component HAN-DB-SDA


How to?



  • How to find log files of Vora services?

    • Vora services save their log files to /var/log/vora-* directories: /var/log/vora-discovery, /var/log/vora-dlog, /var/log/vora-catalog, /var/log/vora-v2server, /var/log/vora-thriftserver, /var/log/vora-tools



  • How to check the status of the Vora discovery service?

    • The Vora Discovery service is based on HashiCorp's Consul. It is used to register and query service endpoints and to perform health checks. The current status of Consul can be checked via the Web UI at http://<host>:8500/ui

    • The log files are written to /var/log/vora-discovery.

    • More information is documented in the SAP HANA Vora Installation and Administration Guide.



  • Delete a Vora datasource table from the Vora catalog?

    • Make sure the table is registered in the Spark Context, e.g. by using the REGISTER TABLE command

      • REGISTER ALL TABLES USING com.sap.spark.vora

      • REGISTER TABLE <tablename> USING com.sap.spark.vora

      • Note: If the table in the catalog is incorrectly defined the REGISTER command might fail. In that case you can add an option to avoid the loading of the table.

        • REGISTER ALL TABLES USING com.sap.spark.vora OPTIONS (eagerLoad "false")

        • REGISTER TABLE <tableName> USING com.sap.spark.vora OPTIONS (eagerLoad "false")





    • Drop the table

      • DROP TABLE <tablename>



    • To verify if the table is deleted from the Vora catalog you can run this command

      • SHOW TABLES USING com.sap.spark.vora





  • Start the Vora Thriftserver in yarn-client mode

    • Add '--master yarn-client' to Thriftserver setting 'vora_thriftserver_extra_arguments' (e.g. via Ambari -> Vora Thriftserver -> Configs -> Advanced vora-thriftserver-config -> vora_thriftserver_extra_arguments -> add --master
      yarn-client)

    • Also possible: Change global setting for all Spark applications 'spark.master yarn-client' in spark-defaults.conf (e.g. via Ambari -> Spark -> Config -> Custom spark-defaults -> Add Property -> key spark.master with value
      yarn-client




Issues Overview



Issues related to Vora 1.2


[13] Loading a table from S3 fails with VoraException


Effected Vora versions: 1.1. 1.2
Symptoms: In the spark-shell you see an error similar to the following.
...
Caused by: sap.hanavora.jdbc.VoraException: HL(9): Runtime error. (v2 URL/S3 Plugin: Exception at opening http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:
v2url plugin: Exception: *ERROR_TYPE*
...
Root cause: HTTP/HTTPS connection problems.
Solution: Depending on *ERROR_TYPE*:
Could not connect to server: This is a connection problem. The VORA Engine cannot connect to the internet. This might indicate a proxy error. Make sure your internet connection is working and proxy environment settings (http_proxy,
https_proxy) are passed correctly to VORA. The proxy setup is described in detail in the SAP HANA Vora Installation and Administration Guide in Chapter "3.1 Configure Proxy Settings".
Error Peer certificate cannot be authenticated with given CA certificates This is a problem with you certificates or your/our openssl implementation. The connection to the server has been built up, but cannot be verified with a SSL
Certificate. This can mean: a) The server you are communicating with is not the server it pretends to be (wrongly configured connections; but might also be a man-in-the-middle attack) b) VORA cannot find the necessary certificates on your system.
Certifcates bundles are searched in  /etc/pki/tls/certs/ca-bundle.crt . If you have a bundle at another place, copy it or link it to at /etc/ssl/certs/vora-ca-bundle.crt (If you are not using a bundle, Certificates are searched in  /etc/ssl/certs/
as OpenSSL 0.x  x509-hashes. Note that this does not work when you have OpenSSL 1.x installed, since then you probably have the matching certificates, so this method is not recommended).
v2url plugin: Exception: Error Code 403 when accessing URL Access denied / S3 Expired. Connection to the server built up,  SSL validated (if HTTPS), but server denied access. This is an access denied error code and is mostly because
you do not have the permission to access something. In S3, this can mean that the query ran out of time. Increase the expire time to fix this.

[16] Reading from HDFS fails with error "SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]"


Effected Vora versions: 1.0, 1.1. 1.2
Symptoms: The following error can be seen when loading from a file in HDFS (e.g. during a CREATE TABLE statement).
>...
15/12/10 17:29:51 ERROR client.VoraClient: [Velocity[<host>:2202]}] Unexpected JDBC exception
com.sap.spark.vora.client.jdbc.VoraJdbcException: [Velocity[<host>:2202]] Unknown error when loading partition: VoraTablePartition(VoraFileInfo(<file>,;,NULL,",<format>,None),VoraFilePartition(0,36))
at com.sap.spark.vora.client.jdbc.VoraJdbcClient.loadPartitionToTable(VoraJdbcClient.scala:240)
...
Caused by: sap.hanavora.jdbc.VoraException: HL(9): Runtime error. (v2 HDFS Plugin: Exception at opening hdfs://<host>:8020//<file>:
SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
(c++ exception))
at sap.hanavora.jdbc.driver.HLMessage.buildException(HLMessage.java:88)
...
Root cause: The Hadoop cluster is enabled for Kerberos. However, the current version of Vora does not support Kerberos.
Workaround: Currently, the only workaround is to disable Kerberos in the cluster. We are working on supporting Kerberos in the future, but it is not yet determined when that will be available.

[19] Vora Tools UI shows error "org.apache.spark.sql.AnalysisException: no such table <table>; with error code 0, status TStatusCode_ERROR_STATUS"


Effected Vora versions: 1.2
Symptoms: The Vora Tools UI shows the following error when accessing a table:
>org.apache.spark.sql.AnalysisException: no such table <table>; with error code 0, status TStatusCode_ERROR_STATUS
Root cause: The table has been created outside of the Vora Tools (e.g. in Spark-shell) and needs to be manually registered in the Vora Thriftserver that is used by the Vora Tools
Solution: Register the tables in the Vora Thriftserver by running the following command in the Vora Tools SQL UI:
>REGISTER ALL TABLES USING com.sap.spark.vora

[20] Vora services fail after initial deployment and the log files are empty


Effected Vora versions: 1.2
Symptoms: Vora services Discover Server, DLOG, and Catalog were initially deployed and fail. The log file directory contains a single empty file, e.g.
>[root@someserver.somedomain.corp vora-dlog]# ls -ltr
total 0
-rw-r--r-- 1 vora hadoop 0 Apr  5 14:19 vora-dlog-someserver.somedomain.corp.log
Root cause: This issue has been observed if the default network interface 'eth0' does not exist or can/should not be used.
Solution: Determine the network interface name on your system, e.g. using command 'ifconfig'. The example below shows an interface with name 'someif'
>somehost:~ # ifconfig
someif    Link encap:Ethernet  HWaddr B4:B5:2F:5F:C6:50
inet addr:10.12.13.14  Bcast:10.12.13.255  Mask:255.255.254.0
... Update the parameters using the cluster manager and set them to the name of the interface you want to use. Afterwards restart the services. For more information see SAP HANA Vora Installation and Administration Guide.

  • Vora Discovery Server -> vora_discovery_bind_interface = someif

  • Vora DLOG -> vora_dlog_bind_interface = someif

  • Vora Catalog -> vora_catalog_bind_interface = someif


[21] Vora services fail with "ImportError: No module named voralib"


Effected Vora versions: 1.2
Symptoms: Vora services, such as the Discovery Service, Catalog, DLOG, v2server, do not start. The log file shows this error.
>Fail: Execution of '/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/vora-discovery/package/scripts/control.py stop_vora_discovery_server' returned 1. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/vora-discovery/package/scripts/control.py", line 32, in <module>
from voralib import *
ImportError: No module named voralib
Root cause: The Vora base libraries are missing. They need to be installed on each node that runs a Vora service.
Solution: Install the Vora base library as per instructions in the SAP HANA Vora Installation and Administration Guide.

[22] DLOG service fails with error "error while loading shared libraries: libaio.so.1"


Effected Vora versions: 1.2
Symptoms: The Vora DLOG does not start and the log file in /var/log/vora-dlog contains the following error:
>In an Ambari deployment
[2016-04-05 14:35:16.374357]Creating SAP Hana Vora Distributed Log store ...
[2016-04-05 14:35:16.377059]Error while creating dlog store:
[2016-04-05 14:35:16.377125]
[2016-04-05 14:35:16.377172]/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/vora-dlog/package/../../vora-base/package/lib/vora-dlog/bin/v2dlog_format: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory
In a Cloudera deployment
[2016-04-07 21:24:55.544047]Creating SAP Hana Vora Distributed Log store ...
[2016-04-07 21:24:55.547848]Error while creating dlog store:
[2016-04-07 21:24:55.547947]
[2016-04-07 21:24:55.548015]/opt/cloudera/parcels/SAPHanaVora-1.2.48.113/lib/vora-dlog/bin/v2dlog_format: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory
Root cause: The libaio library that is needed by the DLOG is not installed.
Solution: Install libaio as documented in the the SAP HANA Vora Installation and Administration Guide in chapter 2.3.7 DLog Server Requirements.

[23] Vora SQL fails with "Could not connect to Consul Agent on localhost:8500" or "No cluster leader"


Effected Vora versions: 1.2
Symptoms: When executing SQL queries the following errors are shown:
>scala> vc.sql(testsql)
com.sap.spark.vora.discovery.DiscoveryException: Could not connect to Consul Agent on localhost:8500 : null
at com.sap.spark.vora.discovery.ConsulDiscoveryClient$ConsulDiscoveryClient.(ConsulDiscoveryClient.scala:38)
...
OR
scala> vc.sql(testsql)
OperationException{statusCode=500, statusMessage='Internal Server Error', statusContent='No cluster leader'}
at com.ecwid.consul.v1.health.HealthConsulClient.getHealthServices(HealthConsulClient.java:96)
...
Root cause: No Vora Discovery Server or Agent is running on the host or the Discovery Server is not running correctly. The Vora Discovery Service uses Consul (from HashiCorp)
to register services. Each server needs to have either a Consul server or Consul agent running (mutually exclusive as both server and agent listen on port 8500; at least 3 Consul servers needed to elect a leader - non-Discovery-server hosts should
only have a Discovery agent). Vora1.2 has a different architecture compared to Vora1.1 with many new services. Please see the What's New in Vora1.2 and the SAP HANA Vora Installation and Administration Guide.
Solution: Install the Vora Discovery Servers and Agents according to the installation instructions in the SAP HANA Vora Installation and Administration Guide.


[26] Vora Discovery Service does not run error in HDFS NameNode HA environment


Effected Vora versions: 1.2
Symptoms: At least one of the 3 Vora Discovery Servers is not running. In the log file in /var/log/vora-discovery the following can be seen (MYNAMESERVICE is the name of your NameService ID):
>[2016-04-19 21:58:35.045916]Service registry initial state for namenode:
[2016-04-19 21:58:35.052422][]
[2016-04-19 21:58:35.052548]Trying 0
[2016-04-19 21:58:35.056475]Result: (True, [])
[2016-04-19 21:58:36.761022]Registering namenode MYNAMESERVICE
...
[2016-04-19 22:08:17.447824]Result: (False, '')
[2016-04-19 22:08:27.458234]Trying 59
[2016-04-19 22:08:27.458478]Register namenode.MYNAMESERVICE
.39
[2016-04-19 22:08:27.461176]Result: (False, '')
...
Root cause: Vora 1.2 is not yet fully supporting a NameNode HA environment
Workaround: In order to run the Vora Discovery Service perform the following workaround before deploying the Vora Discovery Server. Add the hostname and port of your active namenode to script vora-discovery/package/scripts/control.py
># cat vora-discovery/package/scripts/control.py
...
def registerNamenodeService():
...
(nn_host, _, nn_port) = nn_machine.partition(':')
    ## manualy define active namenode host and port ##
nn_host="<active_namenode_url>"
    nn_port="<active_namenode_port_default_is_8020>"
    ## end of manual change ##
log('Registering namenode ' + nn_host + ':' + nn_port)
...
Notes:

  • When changing the script

    • Enclose both host and port with double-quotes.

    • Indent the additional lines by 4 spaces to be identical to the indention of the previous/following lines



  • This workaround does not support an automatic detection of a NameNode failover. In case your active NameNode changes the script needs to be adopted to point to the new active NameNode.

  • It is planned to automatically detect the NameNode HA environment in the next Vora version.


[29] Vora Modeler UI shows error "Object does not exist" for existing table


Effected Vora versions: 1.2
Symptoms: The Modeler UI shows Object does not exist" for an existing table
Root cause: Reason is a program error that does not properly deal with tables that do not have uppercase table names.
Workaround: Drop and recreate the tables in with uppercase name (in the CREATE TABLE as well as in the OPTION 'tablename').
Solution: It is planned to address this issue in the next Vora version.

[31] SQL commands cause error "The current Vora version does not support parallel loading of partitioned tables"


Effected Vora versions: 1.2
Symptoms:A SQL query ends in the following error:
>java.lang.RuntimeException: The current Vora version does not support parallel loading of partitioned tables. Please wait until the previous partitioned tables are loaded, then issue your query again.
at com.sap.spark.vora.client.VoraClient$$anonfun$5.apply(VoraClient.scala:105)
at com.sap.spark.vora.client.VoraClient$$anonfun$5.apply(VoraClient.scala:101)
...
Root cause: The error is due to a program error in Vora 1.2. It can occur when partitioned tables are loaded in parallel. It has been observed after v2servers have been restarted, the partitioned tables have not yet been reloaded
into the v2serves and a query on multiple partitioned tables has been executed. Once the situation happens, the Vora catalog stays in an inconsistent state and needs to be cleaned up.
Workaround: The situation can be avoided by loading partitioned tables after a v2servers have been restarted.
>com.sap.spark.vora.client.ClusterUtils.markAllHostsAsFailed()
REGISTER TABLE <tablename> USING com.sap.spark.vora;
(repeat for all partitioned tables)
Cleaning up the Vora catalog once the situation has occurred:

  • Option 1: Delete partitioned tables


>REGISTER ALL TABLES USING com.sap.spark.vora OPTIONS (eagerLoad "false")
DROP TABLE <tablename>  (repeat for all partitioned tables)

  • Option 2: Delete the Vora persistence

    • Note: This workaround will delete all tables stored in the Vora Catalog




>1. shut down DLOG, Discovery, Catalog, v2servers
2. delete Vora persistence on each node
sudo rm -rf /var/local/vora*
3. Start up DLOG, Discovery, Catalog, v2servers
Solution: It is planned to address this issue in the next Vora version.

[32] Vora Catalog produces many v2catalog_client* files


Effected Vora versions: 1.2
Symptoms: On the node that hosts the Vora Catalog many files with name v2catalog_client..glf are produced. Depending on the Ambari version and configuration those files can for example be found in folders /var/lib/ambari-agent/tmp/log/,
/root/log/, or /log. If too many files are produced the node could potentially run out of inodes. To check the number of available inodes the following command can be run 'df -i'.
Root cause: Due to a program error the files are produced and not cleaned up.
Workaround: See SAP Note 2322686 - Vora Catalog produces many v2catalog_client* files
Solution: It is planned to address this issue in the next Vora version.

[33] Thriftserver fails in CDH with NoClassDefFoundError: HiveThriftServer2


Effected Vora versions: 1.2
Symptoms: The Vora Thriftserver fails and the log shows the following error:
>Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/thriftserver/HiveThriftServer2
at org.apache.spark.sql.hive.thriftserver.SapThriftServer.main(SapThriftServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
...
Root cause: The Vora Thriftserver is using the Spark Thriftserver. Cloudera does not ship the Spark Thriftserver and starting the Vora Thriftserver will fail.
Workaround: See SAP Note 2332867 - Thriftserver fails in CDH with NoClassDefFoundError: HiveThriftServer2 and SAP Note 2284507 - SAP HANA Vora 1.2 Release Note
Solution: It is planned to address this issue in the next Vora version.

Issues related to the SAP HANA Spark Controller


[14] Spark Controller raises ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not persistable.


Symptoms: The following error can be seen when starting the Spark Controller.
>...
Caused by: org.datanucleus.exceptions.ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
at org.datanucleus.ExecutionContextImpl.assertClassPersistable(ExecutionContextImpl.java:5499)
...
Root cause: Hive is configured to use a derby based metastore.
Solution: The hive-site.xml for Vora should only contain the property for the Hive metastore (hive.metastore.uris). Make sure that the Hive metastore is running.
><configuration>
...
<property>
<name>hive.metastore.uris</name>
<value>thrift://<host>:9083</value>
</property>
..
</configuration>

[15] Yarn NodeManager with Spark Controller fails with java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService


Symptoms: The following error can be seen in the Yarn NodeManager log file (/var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-<host>.log) when starting the Spark Controller
>...
2015-12-09 16:20:51,012 FATAL containermanager.AuxServices (AuxServices.java:serviceInit(145)) - Failed to initialize spark_shuffle
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
...
Root cause: When using dynamic allocation in Spark (spark.dynamicAllocation.enabled=true) then Spark needs access to the shuffle service which is implemented in spark-<version>-yarn-shuffle.jar
Solution: Follow these steps to add the spark-<version>-yarn-shuffle.jar to the classpath of all
NodeManagers in your cluster.

[18] Spark Controller does not show Vora tables


Effected Vora versions: 1.2
Symptoms: The Spark controller is running but /var/log/hanaes/hana_controller.log does not show the "Picked up HanaESVoraContext" message and in HANA you do not see any Vora tables in the remote source.
Root cause: Either the wrong version of Spark Controller is used or Spark Controller is not correctly configured to connect to Vora
Solution: (1) Run the correct Spark Controller version With Vora 1.2 you need at least Spark Controller 1.5 Patch 5 (HANASPARKCTRL00P_5-70001262.RPM which includes controller-1.5.8.jar). Earlier versions of Spark Controller are not
compatible with Vora 1.2. This version of Spark Controller also requires a HANA system of SPS11 or later. (2) Correctly configure the Spark Controller Follow the instructions in the SAP HANA Vora Installation and Administration Guide in "Chapter 2.9
Connect SAP HANA Spark Controller to SAP HANA Vora". Please be aware of the following additional step that is currently only documented in SAP Note 2220859 - SAP HANA Vora Documentation Corrections: In the Spark controller configuration file hanaes-site.xml,
which is normally located in the /usr/sap/spark/controller/conf directory, change the value of the property sap.hana.hadoop.datastore from 'hive' to 'vora'. It should look like this:
><property>
<name>sap.hana.hadoop.datastore</name>
<value>vora</value>
<final>true</final>
</property>

[24] Spark Controller fails with "java.net.BindException: Address already in use"


Symptoms: The following error can be seen in Spark Controller log (/var/log/hanaes/hana_controller.log)
>...
Exception in thread "main" java.security.PrivilegedActionException: java.lang.reflect.InvocationTargetException
at java.security.AccessController.doPrivileged(Native Method)
...
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
...
Root cause: A process might be using the Spark Controller port 7860 (maybe a previously run Spark Controller).
Solution: If a process is using port 7860 you can find its PID with ‘netstat -tupln | grep 7860'. Afterwards do a ‘ps -efa | grep <PID>' to check what is running. Afterwards kill the process and then try starting the Spark Controller
again.

[25] Spark Controller fails with "Required executor/AM memory is above the max threshold of this cluster"


Symptoms: The following error can be seen in Spark Controller log (/var/log/hanaes/hana_controller.log)
>...
2015-12-15 11:15:38,716 [ERROR] Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (512 MB) of this cluster!
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:201)
... or
2015-12-15 11:30:09,020 [ERROR] Error initializing SparkContext.
java.lang.IllegalArgumentException: Required AM memory (512+384 MB) is above the max threshold (512 MB) of this cluster!
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:206)
...
Root cause: The Spark Controller uses default values for the Spark executor memory (1024MB) and Spark application master memory (512MB).
Solution: To change the values for the executor and AM memory, you can add them to the hanaes-site.xml file. The below example sets both values to 100 MB.
>...
<property>
<name>spark.executor.memory</name>
<value>100m</value>
<final>true</final>
</property>
<property>
<name>spark.yarn.am.memory</name>
<value>100m</value>
<final>true</final>
</property>
...

[27] Spark Controller fails with "Yarn application has already ended! It might have been killed or unable to launch application master."


Symptoms: The Spark Controller does not start and the log file /var/log/hanaes/hana_controller.log shows and error similar to this
>...
16/04/12 18:54:22 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:125)
...
16/04/12 18:54:22 ERROR Utils: Uncaught exception in thread SAPHanaSpark-akka.actor.default-dispatcher-4 java.lang.NullPointerException
at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
...
Root cause: Spark Controller runs as Spark application and uses Yarn for resource management. The error indicates a problem in Yarn.
Solution: Check the Yarn logs for errors. You can use the Yarn ResoureManager UI at http://<yarn_ResourceMgr>:8088 to access the log files conveniently (Ambari provides a Quick Link
via Ambari -> Yarn -> Quick Links -> Resource Manager UI). In the Resource Manager UI -> find the line for last Spark Controller run -> click on 'Application ID' link on left in the ID column (looks similar to application_1462809397144_0001)
-> click on ‘Logs' link on next screen -> check stderr and stdout logs. A typical reason for seeing this error during Spark Controller startup is issue [17] Yarn NodeManager fails with error "bad substitution".

[30] Spark Controller does not start (log ends with "Server: Starting Spark Controller")


Symptoms: The Spark Controller ends without error message. The log file /var/log/hanaes/hana_controller.log looks similar to this
>SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/sap/spark/controller/lib/external/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/05/24 17:32:53 INFO HanaESConfig: Loaded HANA Extended Store Configuration
Found Spark Libraries. Proceeding with Current Class Path
16/05/24 17:32:54 INFO Server: Starting Spark Controller
Root cause: This situation can happen if the Vora datasources JAR file is not available in the Spark Controller library folder.
Solution: Copy the Vora datasource JAR file spark-sap-datasources-<VERSION>-assembly.jar to folder /usr/sap/spark/controller/lib/ (see Vora Installation and Administration Guide -> Chapter "2.9 Connect SAP HANA Spark Controller to SAP HANA Vora"). In an Ambari-deployment with HDP 2.3 and Vora 1.2 the following command could be used:
>cp /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/vora-base/package/lib/vora-spark/lib/spark-sap-datasources-1.2.33-assembly.jar /usr/sap/spark/controller/lib/ Alternatively, file /usr/sap/spark/controller/conf/hana_hadoop-env.sh
can be adopted to point to the Vora datasources JAR file, e.g.
>cat /usr/sap/spark/controller/conf/hana_hadoop-env.sh
...
#export HANA_SPARK_ASSEMBLY_JAR= <Location to spark assembly>
export HANA_SPARK_ADDITIONAL_JARS=/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/vora-base/package/lib/vora-spark/lib/spark-sap-datasources-1.2.33-assembly.jar
... Afterwards the Spark Controller log should look similar to this:
>SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/sap/spark/controller/lib/spark-sap-datasources-1.2.33-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/sap/spark/controller/lib/external/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.4.0-3485/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/05/24 17:36:22 INFO HanaESConfig: Loaded HANA Extended Store Configuration
Found Spark Libraries. Proceeding with Current Class Path
16/05/24 17:36:22 INFO Server: Starting Spark Controller
16/05/24 17:36:42 INFO CommandRouter: Connecting to Vora Engine
16/05/24 17:36:42 INFO CommandRouter: Initialized Router
16/05/24 17:36:42 INFO CommandRouter: Server started

General issues


[17] Yarn NodeManager fails with error "bad substitution"


Symptoms: The following error can be seen in the Yarn NodeManager log file (/var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-<host>.log)
>...
2015-12-02 19:26:11,968 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(286)) - Stack trace: ExitCodeException exitCode=1: /hadoop/yarn/local/usercache/vora/appcache/application_1448856797920_0013/container_e02_1448856797920_0013_01_000005/launch_container.sh: line 23: :<lots_of_library_paths>:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution
...
Root cause: The key mapreduce.application.classpath in /etc/hadoop/conf/mapred-site.xml contains a variable which is invalid in bash. This issue was only observed with HDP (Hortonworks) and Ambari.
Solution: Either set a value for hdp.version or delete the library entry with references to hdp.version if it is not needed. Instructions on how to set a value for hdp.version in the customer yarn-site.xml in Ambari can be found in
the SAP HANA Vora Installation and Administration Guide in Section "2.8 Install the SAP HANA Vora Zeppelin
Interpreter".

[28] Removing a Vora service in Ambari fails with 'Cannot remove <SERVICE_NAME>'


Symptoms: When removing a service in Ambari the following error occurs:
># curl -u admin:<PASSWORD> -X DELETE -H 'X-Requested-By:admin' http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/<SERVICE_NAME>
{
"status" : 500,
"message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Cannot remove <SERVICE_NAME>. Desired state STARTED is not removable.  Service must be stopped or disabled."
}
Root cause: Either the service is not stopped or Ambari thinks the service is not stopped even though it seems to be stopped according to the Ambari UI.
Solution: First ensure all components of the service are stopped. If the error still occurs, follow the steps from the Ambari documentation to move the service into status stopped. Usually the first command in section "2. Ensure the service is stopped" should be sufficient, but others might be needed too.
>curl -u admin:<PASSWORD> -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/<SERVICE_NAME>

Issues related to older Vora versions (Vora 1.0, Vora 1.1)


[1] Error VelocityJdbcBadStateException "Cannot acquire a connection"


Effected Vora versions: 1.0, 1.1
Symptoms: In the spark-shell you see the following error.
>15/09/09 03:50:31 WARN client.TableLoader: [Velocity[<host>:2202]] Error dropping schema from server: [Velocity[<host>:2202]] Cannot acquire a connection
com.sap.spark.velocity.client.VelocityJdbcBadStateException: [Velocity[<host>:2202]] Cannot acquire a connection
...
java.lang.RuntimeException: There are no Velocity servers up
Root cause: The HANA Vora Engine is not running.
Solution: An initial check to see if the services are running can be done in Ambari (Serivces -> SAP HANA Vora) or Cloudera Manager. Further checks can be done on OS level on the host that can be seen in the error message. Check
if the SAP HANA Vora process is running (process name is v2server).
>$ ps -efa | grep v2server
vora    27859    1  0 Dec07 ?        00:00:00 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/VORA/package/scripts/../v2/bin/v2server --trace-level INFO --trace-directory /var/log/vora --api_server The v2server is listening on port 2202
>$ netstat -tupln | grep 2202
tcp        0      0 0.0.0.0:2202            0.0.0.0:*              LISTEN      27859/v2server The logs can be checked at /var/log/vora. If vora experienced a crash you should find a crash dump in the v2 directory (e.g. in an Ambari environment
are written to /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/VORA/package/v2/crashdump.<timestamp>.txt). If you find a crash dump, open a SAP customer message in component HAN-VO-ENG.

[2] Ambari shows VORA servers in status red


Effected Vora versions:
1.0, 1.1
Symptoms:  Vora is successfully deployed but service shows as red in Ambari. Restarting in Ambari does not help. The v2server log (/var/log/vora) shows errors similar to this.
>/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/VORA/package/scripts/../v2/bin/v2server: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/VORA/package/scripts/../v2/bin/v2server)
Root cause: Vora is installed on an unsupported operating system (e.g. CentOS 6.X, RHEL6.7). This will prevent the compatibility libraries from being loaded and Vora will not start.
Solution: Currently, Vora only supports RHEL6.6, RHEL7.1, and SLES11.3. For details see SAP Product Availability Matrix (PAM) or the SAP HANA Vora Installation and Developer Guide. We strongly discourage from using any but the supported operating systems. We only tested
on the supported OS and we will not support if issues occur due to an unsupported operating systems.
Workaround: If, despite the warning, you want to run Vora on CentOS6.X or RHEL6.7, you could change the start_vora.sh script by commenting/deleting the 2 lines starting with 'if' and 'fi'. Be aware that SAP will not provide support
for such an environment.
>cat /var/lib/ambari-server/resources/stacks/HDP/2.3/services/VORA/package/scripts/start_vora.sh
...
#if [ "$OS" == "RedHatEnterpriseServer" -a "$VER" == "6.6" ]; then
test_set_compat
#fi If Vora is not yet deployed:

  • On the Ambari-server host change file /var/lib/ambari-server/resources/stacks/HDP/2.3/services/VORA/package/scripts/start_vora.sh

  • Deploy Vora


If Vora is already deployed:

  • Option 1)

    • On each Vora host change file /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/VORA/package/scripts/start_vora.sh



  • Option 2)

    • Uninstall Vora

    • On the Ambari-server host change file /var/lib/ambari-server/resources/stacks/HDP/2.3/services/VORA/package/scripts/start_vora.sh

    • Deploy Vora again




[3] The Zookeeper catalog contains invalid entries


Effected Vora versions:
1.0, 1.1
Symptoms: In the spark-shell you see the following errors.
>15/10/14 23:34:20 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2, <IP_address>): com.sap.spark.vora.client.VelocityJdbcBadStateException: [Velocity[<host>:2202]] Unexpected database state (query SELECT ... FROM ...)
at com.sap.spark.vora.client.VelocityJdbcClient.executeSql(VelocityJdbcClient.scala:99)
...
Caused by: sap.hanavora.jdbc.VoraException: HL(9): Runtime error. (schema error: schema "spark_velocity" does not exist (c++ exception))
at sap.hanavora.jdbc.driver.HLMessage.buildException(HLMessage.java:88)
...
OR
...
[79901]{263298}[1924/-1] 2016-01-12 15:32:10.329054 e FedTrace SparkSQLAccess.cpp(03170) : SparkSQLAccess::BrowseMetadata: failed with error: com.sap.spark.vora.CatalogException$ModuleVersionException: module version mismatch between ({"groupId":"com.sap.spark","artifactId":"vcatalog-client-zk","version":"1.0.0"}) and ({"groupId":"com.sap.spark","artifactId":"vcatalog-client-zk","version":"1.1.40","major":1,"minor":1,"patch":40})
at com.sap.spark.vora.ZKCatalog.checkModuleVersions(ZKCatalog.java:477)
at com.sap.spark.vora.ZKCatalog.<init>(ZKCatalog.java:159)
at com.sap.spark.vora.ZKCatalog.<init>(ZKCatalog.java:95)
...
Root cause: Vora uses Zookeeper to store metadata information about its tables. It is possible that the Zookeeper catalog contains inconsistent information, e.g. after an upgrade of the Vora engine or Vora datasources.
Solution: Clear the Zookeeper catalog. In the spark-shell this command can be used. Be aware that after the Zookeeper catalog is cleaned, the tables in Vora need to be recreated and reloaded.
>import com.sap.spark.vora.client._
ClusterUtils.clearZooKeeperCatalog("<zookeeper_host>:2181") Also, make sure to use the correct VORA namespace com.sap.spark.vora in the USING clause. Some older documentation might still point to the deprecated namespace com.sap.spark.velocity.
>CREATE TEMPORARY TABLE <table> (<columns>)
USING com.sap.spark.vora
OPTIONS (...)

[4] SQL fails with InvalidProtocolBufferException


Effected Vora versions:
1.0, 1.1
Symptoms: You see an error similar to this.
>com.sap.spark.vora.hdfs.HdfsException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "ldbwhdp06/10.68.91.155"; destination host is: "ldbwhdp01.wdf.sap.corp":50070;
Root cause: An invalid port is specified for the nameserver (in this case 500700 which is the port for the nameserver web UI)
Solution: Correct the nameserver port either in the nameNodeUrl option or spark.vora.namenodeurl in spark-defaults.conf. The default port for the nameserver is 8020.

[5] Instantiation of SapSQLContext fails with VerifyError


Effected Vora versions:
1.0, 1.1
Symptoms: In the spark-shell you see the following error.
>scala> val vc = new SapSQLContext(sc)
java.lang.VerifyError: class org.apache.spark.sql.ExtendableSQLContext$$anon$2 overrides final method registerFunction.(Ljava/lang/String;Lscala/Function1;)V
Root cause: We have seen this issue when using Vora 1.0 or Vora 1.1 with Spark 1.5.x. Vora has a strong dependency on Spark and Vora 1.0 and Vora 1.1 will not work with Spark 1.5.x.
Solution: Use Spark 1.4.1 (if you use Vora 1.0 or Vora 1.1). As of Vora 1.1 Patch 1 we support Spark 1.5.2 (available as of Jan 25, 2016).

[6] SQL commands fail with NoSuchMethodError: org.apache.spark.sql.sources.SapDDLParser.initLexical()


Effected Vora versions:
1.0, 1.1
Symptoms: In the spark-shell you see the following error.
>scala> vc.sql("show tables").show
java.lang.NoSuchMethodError: org.apache.spark.sql.sources.SapDDLParser.initLexical()V
at org.apache.spark.sql.sources.SapDDLParser.parse(SapDDLParser.scala:177)
Root cause: We have seen this issue when using Vora 1.0 or Vora 1.1 with Spark 1.4.0. Vora has a strong dependency on Spark and will not work with Spark1.4.0.
Solution: Use Spark 1.4.1 (if you use Vora 1.0 or Vora 1.1). As of Vora 1.1 Patch 1 we support Spark 1.5.2 (available as of Jan 25, 2016).
Comment: To see which functions are in a jar file, you can use 'javap'. E.g. for Spark 1.4.1 it shows the method initLexical() in class SapDDLParser. For Spark 1.4.0 this method is missing.
>$ javap -cp /opt/spark/spark-1.4.1-bin-hadoop2.6/conf/:/opt/spark/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/opt/spark/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/opt/spark/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/etc/hadoop/conf/ org.apache.spark.sql.catalyst.AbstractSparkSQLParser
Compiled from "AbstractSparkSQLParser.scala"
public abstract class org.apache.spark.sql.catalyst.AbstractSparkSQLParser extends scala.util.parsing.combinator.syntactical.StandardTokenParsers implements scala.util.parsing.combinator.PackratParsers {
public void initLexical();
$ javap -cp /root/spark-1.4.0-bin-hadoop2.6//conf/:/root/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar:/root/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/root/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/etc/hadoop/conf/ org.apache.spark.sql.catalyst.AbstractSparkSQLParser
Compiled from “AbstractSparkSQLParser.scala”
public abstract class org.apache.spark.sql.catalyst.AbstractSparkSQLParser extends scala.util.parsing.combinator.syntactical.StandardTokenParsers implements scala.util.parsing.combinator.PackratParsers {
// no initLexical() here

[7] Vora commands in Zeppelin fail with NoSuchMethodError: org.apache.spark.sql.sources.SapDDLParser.initLexical()


Effected Vora versions:
1.0, 1.1
Symptoms: The following error can be seen in the Zeppelin log file (/var/log/zeppelin/).
>ERROR [2015-11-19 15:04:39,053] ({pool-2-thread-5} Job.java[run]:183) - Job failed
java.lang.NoSuchMethodError: org.apache.spark.sql.sources.SapDDLParser.initLexical()V
at org.apache.spark.sql.sources.SapDDLParser.parse(SapDDLParser.scala:177)
Root cause: You are using Vora 1.0 or 1.1 and the Zeppelin version is not built for Spark 1.4.1 and your Hadoop version.
Solution: If you are using Vora 1.0 or Vora 1.1, build Zeppelin for Spark 1.4.1 and your Hadoop version. As of Vora 1.1 Patch 1 we support Spark 1.5.2 (available as of Jan 25, 2016). Instructions on how to build and setup Zeppelin
for Vora 1.1 Patch 1 with Spark 1.5.2 are in the SAP HANA Vora Installation and Developer Guide.

[8] Vora commands in Zeppelin fail with java.lang.ClassNotFoundException: com.sap.spark.vora.VoraRDDPartition


Effected Vora versions:

1.0, 1.1
Symptoms: The following error can be seen in Zeppelin: java.lang.ClassNotFoundException: com.sap.spark.vora.VoraRDDPartition.
Root cause: Zeppelin is not (correctly) configured to use the Vora datasources.
Solution: Add the Vora datasources jar file to ADD_JARs in zeppelin-env.sh, e.g.
>export ADD_JARS=$ZEPPELIN_HOME/interpreter/spark/spark-sap-datasources-1.0.0-assembly.jar

[9] SQL commands fail with NoSuchMethodError: org.apache.curator.framework.CuratorFramework.blockUntilConnected()


Effected Vora versions:

1.0, 1.1
Symptoms:
>java.lang.NoSuchMethodError: org.apache.curator.framework.CuratorFramework.blockUntilConnected(ILjava/util/concurrent/TimeUnit;)Z
at com.sap.spark.vora.ZKCatalog.<init>(ZKCatalog.java:120)
Root cause: We have observed this issue when Spark 1.4.1 binaries for Hadoop 2.4.0 were used in a Hadoop 2.6.0 environment.
Solution: Make sure to run the correct Spark version for your Hadoop environment.

[10] REGISTER command does not register HANA tables in Vora


Effected Vora versions:

1.0, 1.1
Symptoms: While the REGISTER SQL command registers Vora tables (com.sap.spark.vora) in the local Spark catalog it does not bring back HANA tables (com.sap.spark.hana).
Root cause: Vora tables are stored permanently in the Zookeeper catalog and can be registered back into the non-persistent local Spark catalog. In the current implementation HANA tables in Vora are not stored in the Zookeeper catalog.
Solution: Recreate the HANA tables.

[11] Start of Thriftserver fails with error "Could not create ServerSocket"


Effected Vora versions: 1.0, 1.1
Symptoms: The start of the Thriftserver fails with the following error message
>15/11/03 14:33:00 ERROR ThriftCLIService: Error:
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:10000.
at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
...
15/11/03 14:33:00 INFO AbstractService: Service:ThriftBinaryCLIService is stopped.
Root cause: Port is already used (most likely by Hive).
Solution: Follow these instructions and set another port for the Thriftserver (export
HIVE_SERVER2_THRIFT_PORT=<listening-port>). If you are using Hive in the cluster it is likely configured with the default port of 10000. In Ambari it can e.g. be checked via Hive -> Configs -> Advanced -> Hive server port . On OS level
this command can be used to find the process using the port: netstat -tupln | grep 10000

[12] SQL commands fail with VoraConfigurationException


Effected Vora versions:


1.0, 1.1
Symptoms: In the spark-shell you see an error similar to the following.
>...
com.sap.spark.vora.VoraConfigurationException: hosts expects host[:port], however an invalid value: ( hslave2) is provided
at com.sap.spark.vora.util.HostIpWithOptionalPortValidator$.apply(Validation.scala:65)
...
Root cause: This error can be raised if the options are incorrect. One example would be a space is in the comma-separated list of hosts in the 'hosts' option.
Solution: Verify the options. Do not add space in comma-separated list of hosts in the 'hosts' or 'zkurls' options.

32 Comments
Labels in this area
Popular Blog Posts