Skip to Content
Author's profile photo Lars Breddemann

A little SAP HANA DBA note on too many trace files…

+++ update 7.10.2015 – SPS 10

Since this blog was originally published some time and many SAP HANA revisions passed and there is meanwhile a supported easy way to do housekeeping around trace files.

There is the  ALTER SYSTEM CLEAR TRACES – SAP HANA SQL and System Views Reference – SAP Library command, that allows the clearing/deletion of trace files from the SQL command.


In addition to that, SAP HANA Studio allows to get rid of old traces via the UI:

/wp-content/uploads/2013/05/2015_10_07_15_30_56_804484.png

And there is more: SAP HANA now automatically compresses and archives old trace files and you can also trigger that manually (mark the trace files you want to archive, right-click, select ‘compress’).


This means you probably won’t be in need of the technique presented in this blog. 😉

Anyhow, for reference it’ll stay up here.


+++ update 7.10.2015 – SPS 10


If you have been running SAP HANA for a year or so, spanning multiple revisions, eventually overcoming bugs you faced in the past, you will invariably end up with an alert similar to this:

There are currently 1033 diagnosis files.

This might indicate an issue with tracefile rotation, a high number of crashes or another issue. Please check the diagnosis files.

As a matter of fact, SAP HANA is currently not particularly great at managing all the trace files it creates.

(And don’t get me started on the actual contents of those files…).

If things go really bad you may even end up finding error messages like the following in the indexserver trace files:

[…]

[89705]{0}[0] 2013-04-23 11:48:39.425976 e Basis        TraceSegment.cpp(00255) :

Exception while trace file compression:

exception  1: no.2120027  (Basis/Diagnose/impl/TraceSegment.cpp:238)

    Trace file compression finished with error: 18446744073709551613

exception throw location:

1: 0x00007fe11c8f5d80 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x13f0 at exception.hpp:313 (libhdbbasis.so)

2: 0x00007fe11c951e92 in Execution::Thread::staticMainImp(void**)+0x6b0 at Thread.cpp:457 (libhdbbasis.so)

3: 0x00007fe11c95208d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:528 (libhdbbasis.so)

[…]

All too bad, I know, but a quick and easy workaround for this is this:

  1. Create another folder, say “old_traces”
  2. Move all files older than, say 10 days (or whenever you last upgraded to the most recent revision), to this “old_traces” folder.

Sounds good? Great, then all you got to do is to follow these steps:

1. Logon to the SAP HANA server as SID<ADM>.

Be aware that in a scale out scenario, you will have to logon to each and every host and perform the following steps there.

 

2. Navigate to the trace file folder

For that, just use the predefined shell command alias ‘cdtrace

hansrv123:/usr/sap/HAN/HDB00> cdtrace

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>

3. Create the “old_traces” folder

Note that in this example I simply create the “old_traces” folder as a sub-folder of the actual trace folder.

You might and probably should create it at a different location, to release the storage space in the trace file folder.

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace> mkdir old_traces

4. Move old trace files into the new folder:

This can take some time…

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>find . -type f -mtime +10 -print | xargs -I {} mv {} old_traces/

If you find that you don’t actually need the trace files any more, you may just go on and delete this folder later on.

In general, it usually doesn’t make sense to keep the trace files generated by older revisions.

Exceptions to this general approach could be, that you faced bugs that you still wait to see fixed or to compare messages from normal/baseline system operations (think of time required for restart, message output during restart/shutdown/backup…).

 

There you have it – now you know! 🙂

Cheers,

Lars

Assigned Tags

      13 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Andy Silvey
      Andy Silvey

      Hi Lars,

      that's a nice tip, thanks for blogging.

      Andy.

      Author's profile photo Former Member
      Former Member

      Hi Lars,

      Good one.... 🙂

      Kulwinder

      Author's profile photo Fabio Pagoti
      Fabio Pagoti

      Hey! I also happened to realize how many trace files are generated on the system.

      I found an option on the Administration view on Hana Studio which can delete trace files. Inside administration for a particular system, go to "Diagnosis Files" tab. On the top right of the tab, there is a button called "Delete Trace Files".

      Thanks for the blog!

      Author's profile photo ABHISHEK SINGH
      ABHISHEK SINGH

      Hi Lars ,

      Nice Document !!!

      Regards,

      Abhishek

      Author's profile photo John Appleby
      John Appleby

      Tell it how it is Lars! This is a useful tip.

      One thing to note is that the too many traces error can be symptomatic of a sick HANA system. I had one revisit which crashed 1000 times in a row. It had a sick delta log and that crashed the index server on startup.

      Though honestly these days such problems are rare.

      Author's profile photo Lars Breddemann
      Lars Breddemann
      Blog Post Author

      Absolutely - sometimes error messages really indicate problems... 🙂 .

      Funny, lately there is increased activity around old and ancient blog posts again. Did something change with SCN again???

      Author's profile photo John Appleby
      John Appleby

      I've noticed comments on a bunch of old blogs too. It's a mystery!

      Author's profile photo Fabio Pagoti
      Fabio Pagoti

      😛 Well... I came here thanks to Google.

      Author's profile photo Shanaka Chandrasekera
      Shanaka Chandrasekera

      Hi Lars,

      Thank you very much for sharing this.

      🙂

      Regards,

      Shanaka.

      Author's profile photo Former Member
      Former Member

      Hi Lars,

      Very helpful post.. Thanks.. 🙂

      Regards,

      Sanket

      Author's profile photo Former Member
      Former Member

      Thanks Lars...

      Author's profile photo Former Member
      Former Member

      Hi Lars Breddemann

      Thanks for sharing the note. It makes sense to keep only necessary files under cdtrace  😎

      I saw another fact that we need to have old traces still in original directory for few  historical informational queries. Any better thought here?

      Thanks,

      Shaheer

      Author's profile photo Lars Breddemann
      Lars Breddemann
      Blog Post Author

      Well, if you actually still need the log files, there's not much to do here. In recent versions of SAP HANA older files can (and most often will be) compressed with gzip. That can also be triggered via context menu in SAP HANA Studio.

      That way keeping some of the trace files shouldn't be a big issue.