In our annual HR Trends report, the Growth and Insights team for SAP® SuccessFactors® solutions conducted research to understand the top HR trends facing organizations today, including what remains a priority from 2023 and what is new for 2024. Each year, we aggregate and synthesize data from a wide range of carefully selected global and regional reputable business press resources that put forward HR trends and predictions and conduct a content analysis to derive key themes, or “meta-trends,” common among them.
But our report this year was different: given the recent explosion of artificial intelligence (AI) and particularly generative AI as a tool to complete work, we redesigned our content analysis methodology to include generative AI. We wanted to not only understand how this capability could be used in this work, but also share our experience using it – what we learned, where it provided the most value, and where we still felt the “human touch” was necessary. Below we’ve detailed the key benefits and drawbacks that we encountered using AI in this trends report.
Benefit: AI quickly produces early insights. Though what you read in our report is 100% original writing from our research team, we did use the LLM to identify common themes within the preliminary trends data to help us create each meta-trend’s summary section (“What caught our attention”). We first manually categorized our 611 trends into broader “meta-trends,” but then fed the individual trends per each category to a large language model (LLM). The model translated massive amounts of raw text data into a set of themes common among them in a matter of a few minutes. For example, our top meta-trend category included 100 individual trends that added up to over 23,000 words – in comparison, it would take an adult who is a fast reader almost an hour to read the same amount of text.
Drawback: AI ignores nuance. In our experience using the LLM, we found that while it very successfully organized large amounts of text data into common themes (based on frequency), it eliminated the critical context that helps us to interpret what is interesting and meaningful among those themes. For example, the raw data associated with our #3 trend category, Hybrid work returns to the office, included mentions of not only rethinking work arrangements and workspaces, but redesigning work itself. This concept is a significant departure from the discourse around this trend in previous years, and an important trend for organizations to understand. Though this topic was notable to our researchers, it was not reported in the LLM’s output because it was lower in frequency relevant to the overall hybrid work-related data. We believe the use of LLMs in this way would likely be more valuable for beginners or generalists who are looking for broad baseline knowledge on a topic than experts or specialists who have such baseline knowledge and are looking for specificity or nuance.
Benefit: AI can iterate flexibly. Engineering our prompts to the LLM allowed us to get very specific about what the model returned, specifying the exact format of the output to meet our needs. For example, rather than just asking for “a list of themes and their summaries,” we could ask the model to list the themes ranked by how often they occurred in the data, write a summary of each theme in a specific number of sentences, and provide exemplary quotes directly from the data to further illustrate the theme from the source material. This not only provided better content for our use, but asking the LLM to return examples from the source data also helped us evaluate the quality and accuracy of the model’s output.
Drawback: AI does not control for quality in source data. Beginning in the fall of 2023, we began to hand-pick the source material to be included in our 2024 meta-trends analysis. Each source is carefully evaluated before being included in our analysis, focusing on high-quality, reputable business press sources. In fact, we excluded about 20% of our initially collected sources this year due to issues with quality, such as irrelevance to our topic of study, poor grammar and spelling, or bias. When we prompted LLMs that could access real-time data to come up with a list of HR trends in 2024 and report which sources they used, we found instances of poor quality in their source material. This is another reason we felt it was important to input our own quality-controlled trends data rather than rely on the LLM to identify its own source material.
Overall, including AI in this year’s analysis was an interesting and informative process, but ultimately the weaknesses we listed above limited its value. For work that requires critical thinking, precision, and nuance such as is the case with our meta-trends analysis, the need for humans to largely drive and oversee this type of work is essential to ensure that gains in speed and flexibility do not sacrifice overall quality.
