Most data scientists and statisticians agree that predictive modeling is both art and  science yet, relatively little to no  air time is given to describing the art. This post describes one piece of the art of modeling called feature engineering which expands the number of variables you have to build a model.  I offer  six ways to implement feature engineering and provide examples of each. Using methods like these is important because additional relevant variables increase model accuracy, which makes feature engineering an essential part of the modeling process. The full white paper may be downloaded at Feature Engineering Tips for Data Scientists.

What Is Feature Engineering?

A predictive model is a formula or method that transforms a list of input fields or variables (x, x, …, x) into some output of interest (y). Feature engineering is simply the thoughtful creation of new input fields (z, z, …, z) from existing input data (x). Thoughtful is the key word here. The newly created inputs must have some relevance to the model output and generally come from knowledge of the domain (such as marketing, sales, climatology, and the like). The more a data scientist interacts with the domain expert, the better the feature engineering process.

Take, for example, the case of modeling the likelihood of rain given a set of daily inputs: temperature, humidity, wind speed, and percentage of cloud cover. We could create a new binary input variable called “overcast” where the value equals “no” or 0 whenever the percentage of cloud cover is less than 25% and equals “yes” or 1 otherwise. Of course, domain knowledge is required to define the appropriate cutoff percentage and is critical to the end result.

The more thoughtful inputs you have, the better the accuracy of your model. This is true whether you are building logistic, generalized linear, or machine learning models.

Six Tips for Better Feature Engineering

Tip 1: Think about inputs you can create by rolling up existing data fields to a higher/broader level or category. As an example, a person’s title can be categorized into strategic or tactical. Those with titles of “VP” and above can be coded as strategic. Those with titles “Director” and below become tactical. Strategic contacts are those that make high-level budgeting and strategic decisions for a company. Tactical are those in the trenches doing day-to-day work.  Other roll-up examples include:

  • Collating several industries into a higher-level industry: Collate oil and gas companies with utility companies, for instance, and call it the energy industry, or fold high tech and telecommunications industries into a single area called “technology.”
  • Defining “large” companies as those that make $1 billion or more and “small” companies as those that make less than $1 billion.

  Tip 2: Think about ways to drill down into more detail in a single field. As an example, a contact within a company may respond to marketing campaigns, and you may have information about his or her number of responses. Drilling down, we can ask how many of these responses occurred in the past two weeks, one to three months, or more than six months in the past. This creates three additional binary (yes=1/no=0) data fields for a model. Other drill-down examples include:

  • Cadence: Number of days between consecutive marketing responses by a contact: 1–7, 8–14, 15–21, 21+
  • Multiple responses on same day flag (multiple responses = 1, otherwise =0)

Tip 3: Split data into separate categories also called bins. For example, annual revenue for companies in your database may range from $50 million (M) to over $1 billion (B). Split the revenue into sequential bins: $50–$200M, $201–$500M, $501M–$1B, and $1B+. Whenever a company falls with the revenue bin it receives a one; otherwise the value is zero. There are now four new data fields created from the annual revenue field.  Other examples are:

  • Number of marketing responses by contact: 1–5, 6–10, 10+
  • Number of employees in company: 1–100, 101–500, 502–1,000, 1,001–5,000, 5,000+

Tip 4: Think about ways to combine existing data fields into new ones. As an example, you may want to create a flag (0/1) that identifies whether someone is a VP or higher and has more than 10 years of experience. Other examples of combining fields include:

  • Title of director or below and in a company with less than 500 employees
  • Public company and located in the midwestern United States

You can even multiply, divide, add, or subtract one data field by another to create a new input.

Tip 5: Don’t reinvent the wheel – use variables that others have already fashioned.

Tip 6: Think about the problem at hand and be creative. Don’t worry about creating too many variables at first, just let the brainstorming flow. Feature selection methods are available to deal with a large input list; see the excellent description in Matthew Shardlow’s “An Analysis of Feature Selection Techniques.” Be cautious, however, of creating too many features if you have a small amount of data to fit. In that case you may overfit the data, which can lead to spurious results.

Hundreds, even thousands of new variables can be created using the simple techniques described here. The key is to develop thoughtful additional variables that seem relevant to the target or dependent variable. So go ahead, be creative, have fun, and enjoy the process.

To report this post you need to login first.


You must be Logged on to comment or reply to a post.

    1. Patricia Tillotson Post author

      These were examples taken from actual marketing modeling projects I’ve been involved in at SAP.  Can you give me some additional detail on what you mean by business specific examples?

      1. Ranajay Sit

        Sorry my bad, I meant some more Business specific examples, suppose like for traditional predictive usecases eg. account receivables, or per say Demand Forecasting, in such examples how one can decide if the given data points are sufficient to predict or if additional input data is required. How to attack that use case?

        1. Patricia Tillotson Post author

          Hi Ranajay,

          I hope others will post their thoughts and the data fields they use in modeling.  Let’s use this as a sharing place.

          We never know whether we have enough variables/data until after the model is built.  The validation process is the best way to determine whether or not you have sufficient variables/data and model accuracy should be as good as your business requirement says it needs to be.  If accuracy is not there, then it’s back to model building all over again.

  1. Kenneth Demma

    Great post Patti–thank you for taking a deep data science topic and making it understandable and usable for non-data scientists. Looking forward to more in the future.

  2. Nancy Fessatidis

    Great post Patti.  Agree that the art of predictive modeling is often overlooked.  The creativity that goes into defining the right variables almost always deliver a better outcome, especially when the domain experts are actively involved in the planning process.  Well done!


Leave a Reply