Skip to Content

I’m currently doing data migration in a specific problem domain, but I think what I’m sharing here can be applied very generically.

 

The checks you usually do on an input file can be supported (and automated) by small GNU Tools.

 

As an example, it was agreed, that the input file will not contain duplicates – let’s check if it does.

Also, it’s always a good idea to know how many lines you are dealing with, so that is where we start:

 

1. get the number of lines:

echo [filename] | wc -l

 

2. get the number of unique lines:

echo [filename] | uniq | wc -l

 

(If the numbers are the same, there are no adjacent duplicates)

 

3. Maybe we have duplicates spread over the file? (-> so they are non-adjacent?) Let’s check:

sort [filename] | uniq | wc -l

 

4. If we find duplicates we want to give a qualified feedback, like: what where the duplicate lines (-d) and how often do they appear in the File (-c):

 

sort [filename] | uniq -dc > duplicate_lines_please_check.txt

 

explanation:

uniq will write the unique lines of the input file to standard output.

-d will give (only) the duplicate lines – one line per duplicate

-D will give ALL the duplicates lines

-c also add’s the info how often a line appears.

 

[Edit:

 

Another option that might be useful is

-u -> only consider lines unique in the Input-File

 

-> I’ve assumed if there a duplicate lines in the Input file, one of them is a good one (wanted) and only it’s duplicates are bad ones (un-wanted).

 

So if the file  for example contains :

 

111

111

22

 

My output would be

111

22

 

Another approach might be: if a line is duplicative, it’s nonsense for sure. You would only want 1 line in the example:

22

 

In this case, the -u option is just what you want: it makes sure only unique lines of the input file are considered.

]

To report this post you need to login first.

3 Comments

You must be Logged on to comment or reply to a post.

  1. Christian Dr. Drumm

    Hi Joachim,

    great blog. Lots of people in the SAP world don’t know or don’t use the standard Unix tools. It’s definitely a good idea to raise awareness.

    one tool I’d add to the list of must-know-unix-tools is sed (https://en.wikipedia.org/wiki/Sed). Especially when needing to manipulate large (migration) files sed becomes very handy.

    Christian

    (1) 
    1. Joachim Rees Post author

      Thanks for your feedback, Christian!

      Maby the next time you solve a problem with sed, I’ll read your blog about it!? 😉

      best

      Joachim

      (0) 

Leave a Reply