IBM and a Department Store Build a Recommender
Intro: Why this Article?
Recommendation systems are an especially important tool for businesses today selling a multitude of products. They allow those businesses to infer for their customers their potential interest in products they haven't yet consumed; and possibly have never seen. This additive feature to a customer's experience also helps the business by increasing their customers' purchases and share of basket. IBM's Data Science Elite (DSE) worked with a department store on an engagement to build out a recommendation system that fit within the department store's architecture and marketing mechanisms.
As an interlude, for those unfamiliar, the IBM DSE group is an expert resource within IBM that travels to clients and assists them with engagements to up-level our IBM clients in data science. It is an investment made by IBM in our clients. We have certain preconditions for clients to participate, and one of those is that we require clients bring their own data science resources so that the lessons persists even after IBM leaves. We want to help build the next set of AI-enabled organizations and fundamentally change the way our clients work.
In this article, we'll walk you through the use-case of a recommendation system, the theoretical underpinnings of a sample of these systems, and the specifics of this business case where possible. Keep in mind that to respect the privacy of the IBM Client, their name and information that would identify them (specific features) may be omitted. You'll walk away with an intuition for the goals of these applications, an overview of the frameworks, and a walkthrough of the algorithm ultimately used in the IBM client engagement.
What is a recommender system?
A recommender system is an application that looks at the purchases or product ratings from existing clients and identifies additional products they may enjoy based on that historical behavior. There are multiple types of recommenders within the broader category of these systems that differ primarily on the data available to them.
Data may, or may not, be available on the opinions of your customers, hence some recommenders are based on explicit opinions while others are based on implicit opinions. Explicit opinions mean that the consumer has intentionally provided feedback on the good being sold —I rated these shoes four stars out of five on my stores website. Implicit recommendations rely on proxy information to indicate interest in the good being sold —I clicked on that product's listing on my department store's website.
Data also may or may not be available to describe your products, hence some recommenders are designed to leverage descriptive product characteristics (content-based filtering) while others are based primarily on finding customers who have similar behavior patterns (collaborative filtering). When data is available to describe and characterize your product, you can leverage it for the sake of finding items with similar characteristics to those previously purchased. Those data are essential for content-based filtering recommenders (CBF), and often take the form of metadata i.e. a given movie is in the romance genre. When data that describe products aren't as readily available, or is deemed not to be instructive, recommenders compare the behavior patterns between users in an attempt to find similar profiles. In such an example in our department store setting, user A buys hoodies and shorts from a specific brand, and user B buys hoodies, shorts, and sneakers from that same specific brand. User A may be interested to hear about those sneaker products.
Each of the above recommendation engine designs is significant in its tradeoffs given to achieve the ultimate end goal of recommending a product to a user who has not tried it before. When it comes to processing implicit or explicit opinions, explicit opinions require surveying your consumers (along with the required infrastructure to gather and persist these opinions) while implicit opinions demand you make an assumption about the meaning of certain behaviors. With either, you have to answer a variety of questions. Where and how do you persist your opinion data? Do you model how opinions of goods (tastes) change over time? If your consumer has never clicked on a product, is that because they were not interested or because did they just not know it existed? Your interpretation will determine how you treat your data.
Hopefully what is clear from the above is that the data available drives the choices IBM and our client ultimately had to make as it pertains to architectures of the recommender system; a principle that would apply to anyone else building a recommender system as well. The IBM client data looks as follows:
A special thanks to @Dheeraj Arremsetty of the IBM Data Science Elite (DSE) for the resources used as the basis of this blog post.