Introduction
Relevant search results are the cornerstone of a successful online experience and higher conversions. This starts with building a robust Solr search query. If your results are consistently lacking, it's crucial to examine the underlying query processing and search functionality.
Getting to that point is typically an iterative process of analyzing, modifying and reviewing the updated results. Check out this video which covers many of the opportunities available for tweaking search results and is a good place to start.
This article will give you concrete example how we can tune Solr search engine and when we should do that.
Back to basic
Before to enter in technical tuning details, we must remind how basically search engine work.
It is not a database. Everything is pre-computed and set in one big table which is called Index.
Into this Index each entry is called Document. In SAP Commerce Cloud a Document represent all data related to a Product: title, description, category, price, stock information, media, and so on.
Regarding search query a Document relevancy is given by the Solr score. This score is computed on text matching and weight setup on each Document properties (product attributes).
For example, title should has more weight than description.
Global setting
By default SAP Commerce Cloud comes with a global pre-setting that cover most part of usages. There's two different settings: Solr server side and SAP Commerce side.
SOLR
In Solr, we've schema.xml where definition how Index and query are building according type of field. In below, we've default setting for english text.
<fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory" />
<filter class="solr.EnglishPossessiveFilterFactory" />
<filter class="solr.ManagedStopFilterFactory" managed="en" />
<!-- <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" /> -->
<filter class="solr.LowerCaseFilterFactory" />
<!-- <filter class="solr.KeywordRepeatFilterFactory" /> -->
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt" />
<filter class="solr.PorterStemFilterFactory" />
<!-- <filter class="solr.SnowballPorterFilterFactory" language="English" /> -->
<!-- <filter class="solr.EnglishMinimalStemFilterFactory" /> -->
<!-- <filter class="solr.ASCIIFoldingFilterFactory" /> -->
<filter class="solr.RemoveDuplicatesTokenFilterFactory" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory" />
<filter class="solr.EnglishPossessiveFilterFactory" />
<filter class="solr.ManagedSynonymGraphFilterFactory" managed="en" />
<filter class="solr.ManagedStopFilterFactory" managed="en" />
<!-- <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" /> -->
<filter class="solr.LowerCaseFilterFactory" />
<!-- <filter class="solr.KeywordRepeatFilterFactory" /> -->
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt" />
<filter class="solr.PorterStemFilterFactory" />
<!-- <filter class="solr.SnowballPorterFilterFactory" language="English" /> -->
<!-- <filter class="solr.EnglishMinimalStemFilterFactory" /> -->
<!-- <filter class="solr.ASCIIFoldingFilterFactory" /> -->
<filter class="solr.RemoveDuplicatesTokenFilterFactory" />
</analyzer>
</fieldType>As we can see data transformation to record value into Index (analyser type="index") is almost the same as data transformation to treat a query (analyser type="query").
To deeper understanding how data is transformed please checkout official documentation:
- https://solr.apache.org/guide/8_11/tokenizers.html
- https://solr.apache.org/guide/8_11/filter-descriptions.html
Following KBA notes will guide you to create your custom schema.xml
Great article to create your custom Stemmer : https://blogs.sap.com/2023/10/30/how-to-create-custom-solr-stemmer-in-sap-cc/
SAP Commerce
Indexation
And in SAP Commerce side, we've type of field mapping for each property.
In our example, by default, Product.description is mapped as text. Then Product.description[en] will pass by fieldType name="text_en" SOLR treatment we saw in previous chapter.
As you understand, the right type definition is one of key settings that is crucial for getting relevancy search result:
- Text type should be used for not exact search as title and description.
- String type should be used for exact match as code or ean.
- Double type should be used for range value as price or size.
Priority is defining weight of this property for Document into Index. By default, priority is not set (same value for all properties / 0) and query boost is used instead.
Query
Another important parameter is the query building aspect. For each property we can influence the weight on the final score given by SOLR.
This query building is defining into Search Query Template and it acts as global query setting
In our example, by default, Product.name (product title) has a boost factor set at 50 and fuzzy search activated. And Product.description has a boost factor set at 25. In other words, product description is less relevant than product title for a search.
Usage case study
Similar Product
In this use case, product catalog contents similar product as camera for example. Product can be spread by brand and hardware capacity. And naturally customer will type : sony camera or megapixel camera.
Customer can find relevant result by using search box directly. However, sometime, we could have in top result some accessory show up as Digital Camera Tripod
To mitigate this incident, we can decrease boost text factor on Product.name and increase Product.keywords. Then Product.keywords should contains "camera" for digital device and should contains "accessory" for the rest.
This approach can demand some data maintenance effort and it should be used only when we don't have too much diversity.
Note: final score computed by Solr can be seen in Adaptive Search Cockpit.
High Product Diversity
In this use case, product catalog contents high product diversity as shirt, glass, shoe, pants and so on.
Customer could type generic term that doesn't match a specific product category as fashion clothing
End result coud lead to get in first product listing page : Cushion shoe and Helmet
Shoe result be explained by the fact Cushion term is close to Fashion fuzzy speaking. Solr is applying fuzzy search to match even when someone is not typing exactly the right term.
Helmet result is because there's no Clothe category setup in this catalogue.
In this context, to improve result relevancy, we should create category that match better common customer search query and maybe reduce fuzzyness boost on Product.name
Market Place Management
This use case is similar to previous one, but market place contributors can play with your search setting to boost their own product search result.
Let imagine your core selling is "mail box" and you want to open your website for other vendors where direct selling is possible. One of your food vendor is knowing well the fact your selling "mail box", he will try to influence search in his advantage by putting product title like "lunch box by mail", "meal box", and so on.
In this context, you can
- have strict naming policy
- or you can split your web shop experience in tow parts : core products and market place.
- or guide user experience by selecting a first product level category : mail, food, and so on.
In short, data quality is a key criterial for getting relevant search result. Solr can excel only when data is well-organized and structured
Contextual setting
On top of global setting, we can fine tune search result by applying conditional boosting rule and even promote a product at the top of search result from a category.
In our example, we promote a product on global category. Then, it should appear at the top where this product match the search query as t-shirt, shirt. And it doesn't appear when we have no match as shoe.
This contextual setting is applying after search result provided by Solr. This setting is stored in object named : Search Profile.
Several Search Profiles can be combined to provide a personalized user experience according user profile (VIP, Shirt Fan, Shoe Fan, Junior, Senior, and so on).
To assign user at a profile, you need CRM Marketing tool that will create many segments based on different criteria (age, previous order, and so on). SAP Emarys is standard tool for.
Best Practice
Now that we have seen how technically fine tune search result, we are ready to start search optimisation project.
You need first to work on data quality. Product title should summarize your product description and not enter in conflict with some similar product as much as possible. Each product should be assigned on suitable navigation category. And when navigation category cannot help you to distinct two type of product, then you should use keyword.
Secondly, you need to know your customers. Web Analytics tool should help you with as Google Analytics or Dynatrace Digital Experience Monitoring solution. By capturing the most common search queries executed, you can fine tune those search results to optimise your selling.
In last, you can fine tune dynamically default search result by using Adaptive Search Cockpit.
Conclusion
Search result tuning can be hard topic. It will demand from business team to understand deeply what's a search engine, what we can expect for and how we should use it.
We saw that in SAP Commerce Cloud provides a dynamic capability to influence in live search result. And, If you’re looking help to get the best from Solr technology and/or a training, feel free contact SAP Expert Services.
But if you're looking for SaaS solution, SAP is preparing one and if you want to participate please join the beta testing: https://blogs.sap.com/2023/03/27/beta-search-service-for-sap-commerce-cloud/
You can also decide to integrate advanced SAP Store SaaS solution: Coveo
- SAP Managed Tags:
