A little SAP HANA DBA note on too many trace files…

+++ update 7.10.2015 – SPS 10

Since this blog was originally published some time and many SAP HANA revisions passed and there is meanwhile a supported easy way to do housekeeping around trace files.

There is the  ALTER SYSTEM CLEAR TRACES – SAP HANA SQL and System Views Reference – SAP Library command, that allows the clearing/deletion of trace files from the SQL command.

In addition to that, SAP HANA Studio allows to get rid of old traces via the UI:

And there is more: SAP HANA now automatically compresses and archives old trace files and you can also trigger that manually (mark the trace files you want to archive, right-click, select ‘compress’).

This means you probably won’t be in need of the technique presented in this blog. 😉

Anyhow, for reference it’ll stay up here.

+++ update 7.10.2015 – SPS 10

If you have been running SAP HANA for a year or so, spanning multiple revisions, eventually overcoming bugs you faced in the past, you will invariably end up with an alert similar to this:

“There are currently 1033 diagnosis files.

This might indicate an issue with tracefile rotation, a high number of crashes or another issue. Please check the diagnosis files.“

As a matter of fact, SAP HANA is currently not particularly great at managing all the trace files it creates.

(And don’t get me started on the actual contents of those files…).

If things go really bad you may even end up finding error messages like the following in the indexserver trace files:

[…]

[89705]{0}[0] 2013-04-23 11:48:39.425976 e Basis        TraceSegment.cpp(00255) :

Exception while trace file compression:

exception  1: no.2120027  (Basis/Diagnose/impl/TraceSegment.cpp:238)

    Trace file compression finished with error: 18446744073709551613

exception throw location:

1: 0x00007fe11c8f5d80 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x13f0 at exception.hpp:313 (libhdbbasis.so)

2: 0x00007fe11c951e92 in Execution::Thread::staticMainImp(void**)+0x6b0 at Thread.cpp:457 (libhdbbasis.so)

3: 0x00007fe11c95208d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:528 (libhdbbasis.so)

[…]

All too bad, I know, but a quick and easy workaround for this is this:

1. Create another folder, say “old_traces”
2. Move all files older than, say 10 days (or whenever you last upgraded to the most recent revision), to this “old_traces” folder.

Sounds good? Great, then all you got to do is to follow these steps:

1. Logon to the SAP HANA server as SID<ADM>.

Be aware that in a scale out scenario, you will have to logon to each and every host and perform the following steps there.

2. Navigate to the trace file folder

For that, just use the predefined shell command alias ‘cdtrace‘

hansrv123:/usr/sap/HAN/HDB00> cdtrace

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>

3. Create the “old_traces” folder

Note that in this example I simply create the “old_traces” folder as a sub-folder of the actual trace folder.

You might and probably should create it at a different location, to release the storage space in the trace file folder.

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace> mkdir old_traces

4. Move old trace files into the new folder:

This can take some time…

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>find . -type f -mtime +10 -print | xargs -I {} mv {} old_traces/

If you find that you don’t actually need the trace files any more, you may just go on and delete this folder later on.

In general, it usually doesn’t make sense to keep the trace files generated by older revisions.

Exceptions to this general approach could be, that you faced bugs that you still wait to see fixed or to compare messages from normal/baseline system operations (think of time required for restart, message output during restart/shutdown/backup…).

There you have it – now you know! 🙂

Cheers,

Lars