Skip to Content

Ratings, Blog Contents and Some Magic Formula (the good kind)

What’s my Secret Method?

SCN contributor Otto Gold raised the topic of “the magic formula of quality” and wanted to unveil the secret equation behind calculating blog points in his blog: The specified item was not found.

He also challenged me to document my own clandestine and previous unknown and unpublished technique for granting members their blog schwag or in simple language: how the heck we determine blog point allocation.

I won’t bore you by repeating my answer to Otto in the comments to his blog.  They can be found here by clicking Assigning value, creating value.   I also need to commend Tobias Hofmann for jumping into the fray.

But I did want to provide Otto with what he asked for : namely the opinions of various community members on the topic of ratings and searching for quality content.

As I started to rack my brain to remember what folks like Introduction to Knowledge Exploitation, Jim Spath, Jon Reed, Dennis Howlett, Tobias Hofmann, Jelena Perfiljeva, The specified item was not found., Congratulations to the current Topic Leaders (2009-2010), Meet David Branan Community Support Angel and others had to say about the topic during SAP Inside Track in NSQ I remembered that I had filmed some of the conversation.

So I edited three segments which feature comments from the above participants.

Some of the salient points I remembered were that Jim Spath reminded us that to date ratings on SCN only exist on formal content such as articles and rich media.  Jon Reed spoke about how Google avoids the questions of rating by looking at links to content and Trevor pointed out the quality risks with a highly commercial “pay for exposure” site like Google.  Greg Myers spoke of something that is close to my formula when he said that amount and type of comments that generate a healthy conversation and knowledge exchange = good post.  (aha! a formula)

Yesterday in the hallways of an SAP office I bumped into Trevor and continued to speak to him about the values of rating types.  We spoke at length about the move toward “like” as a way for people to express interest and of course the general consensus during these discussions during Inside Track were that comments are really an essential part of evaluating content.


Part One

Part Two


Part Three – This one features David Branan!

Dennis Howlett reminded participants that YouTube ditched the rating system.  No more stars:

But there were many comments from the attendees that having thumbs up and thumbs down wasn’t a great idea either:

Many people seem to favor the facebook “like” approach though and this was something explored in the conversation with Trevor.

Otto wanted some kind of road map to our future way of handling content evaluation.  I can broadly hint that it will most probably include like features, comments and the possiblity of rating (although after reviewing these contents it seems that many folks are not strongly in favor of a “5 star rating process”.

What do you think?

You must be Logged on to comment or reply to a post.
  • Hi Marilyn,

    as you know I’ve been thinking about some of these aspects ever since I started contributing to SDN/SCN. As such, I really liked your insight scoop on the discussions you have and more so the thoughts being gathered and consolidated.

    From my personal experience I can say that it’s certainly a “roller-coaster” ride and that “the lack of feed-(back)” is certainly one of the reasons that can turn people down at a very early stage. I also believe that this maybe one reason why Otto Gold asked if we are loosing talents.
    I’m totally up with John Reed that “a written comment” is the most useful and rewarding way of ‘giving back to the author!’

    This is a “netiquette” that I see missing sometimes on SCN, and while I’d say that the most active contributors do so by heart and due to believing in a higher mission so to speak – to make SCN a place to share and build knowledge, learn & grow, discuss and collaborate…

    I think that if you turn a mono(b)log-ue into a conversation or discussion it is the more rewarding for everbody(!). Yet… it all depends on the “first follower” (Derek Sivers) – which is one of the readers.( I wrote about all that in my latest blogs…)

    Yet, I do not want to cater to laziness in regards to contributing – commenting already qualifies for that – but a simple #ilike or even thumbs up/down would may result in at least a vague idea on what the readers think.

    The worst that can happen is 500 views a week  on a blog and no comment at all.

    Matter of fact, even a thumb down seems appropriate and maybe linked with a (mandatory-recommended) comment on why a negative ranking was given. Also, negative rankings may draw the attention to some pain points and could be also be taken to thinking about “coaching” people by more experienced bloggers….

    Cheers, Matthias

    • Thanks Matthias for your valuable feedback.  I agree that people are often surprised that content which they spent an inordinate amount of time creating doesn’t always glean comments.
      I’ve noted myself, many times that some authors of EXCELLENT content haven’t received comments although their blogs were well-written, original, and chocked full of information.
      But comments are some indicator of what “resonates” and so if someone is not getting dialouge it could well mean that they haven’t left an openning for one.  Or the topic doesn’t lend itself for discussion.  Or that the contents are being read by folks that don’t feel moved to add something or don’t feel qualified.
      Funny, when it comes to things like improving the experience here (Otto’s series) or the present state of consulting or any number of topics that folks find a personal affinity to be it professional, academic or social.
      • Well put. The amount/lack of feedback is a good indicator on what is hot or not – at least for a majority or a very influential group of SCN. What I’m targeting is the entry barrier for people to leave any indication at all… that may just take one click not more…

        Does anyone have numbers from out communities if the ratio of comments corelate to thumbs up (or thelike) clicks? Would be interesting…

        • Trevor Carlow and Lee Clemmer have quite a bit of experience and knowledge around this topic (from an external community perspective).  I would suggest dragging them into the conversation directly
          • Marilyn, thanks for roping me into this discussion. While I’m no expert on ratings I did attend an interesting session on the topic of “Web Reputation Systems” at the Web 2.0 conference earlier this year. The two speakers had just written an O’Reilly book with that title.

            One of the things they spoke about that resonated with me was the misperception that all you need is 5 stars or thumbs up. It turns out that in a graph the number of ratings vs number of stars would like like a “J” curve where the majority of ratings are 5 stars. Essentially if you ask for someone’s opinion on something they tend to rate it favorably. For this reason, the average rating on the internet is 4.3 stars (according to a WSJ study).

            On YouTube they found that great videos prompted action but anything less prompted indifference. So essentially the ratings system is a seal of approval. YouTube has recently done away with the 5 star ratings in favor of the thumbs up and thumbs down. Some food for thought as we examine ratings on SCN…..

          • I would also agree with Christine. I think the biggest thing that I have heard is that Ratings in and of themselves represent very little value for “content” they are favored more for purchased items as they are typically used for some time before the rating is actually posted.

            I also would say that there should be a clear seperation between rating and review. The user trend seems to be that people will only rate things (usually a 5) if they feel even slightly moved by the item/content. However many people will review but not rate the item.

            Personally i feel that the concept of rating in terms of SDN content should be much different. It shouldn’t be directly impacting “points” (Another thing provided during the sessions a few months ago) and they should be decoupled from the review process. You should be able to do either (very similar to YouTube)

  • Marilyn, I would like to thank you for the blog! Oficially! It feels nice when people keep promises and we walk towards the bright future of SCN together.
    Even if the opinions are different I think something to objectify the value of content is needed. Maybe the others will change my opinion- not to use stars etc. – because there are many aspects I have never thought of and your and your colleagues´ experience is much wider. But the first step was made – we started talking about it.
    Believe me it is important to talk about it here, in public. It is important you talk about it with your team, but you´re 0,001% of the SCN users maybe? So it is even more important to talk about it with the “ordinary” people who will use the features…
    Keep going! All the best, Otto
  • I am always happy to get feedback from the community on what they think of our features.
    I will take this in to consideration as we plan the next phases of the rating experience
  • Hello again, Marilyn,
    as you can see around, this blog was worth the time and effort and it has drawn a lot of attention! I am glad you wrote it and now can see the interest of the people. Well, they didn´t comment on my blog, but are attracted to you as our Community boss and so the comment are not lost. And I am sure you can handle the input from the Community better than I would do…
    I hope this is not the last time we share the ideas about the topic, since it is obvious people care about it.
    I was not aware of any “scientific” theories about the ratings etc. but will focus on this to be more helpful.
    Regards Otto