Face Reading Technology: Improving Preference Prediction from Self-Reports Using Micro Expressions

with Franziska Krause (EBS University)

and Janina Krick (EBS University)

November 16th, 2023

EBS, Fall 2023 Research Colloquium

Oestrich Winkel

Pantelis Karapanagiotis
(EBS University and SAFE)

@pi_kappa_

What Makes a Good Sales PersonMachine?

  • In the classic HBR article, Mayer & Greenberg (1964) identify two essential qualities for being successful as a salesperson.
    1. Ego Drive - Drive for sales is relatively easy to program.
    2. Empathy - Can this quality be machine-automated?

Why is this Interesting?

  • Preference Prediction
  • Managers and researchers mainly rely on traditional market research tools

Positives

  • Cost efficient
  • Practical
  • Easy to distribute
  • Fast
  • Scalable
  • Enable access to enormous cohorts of subjects

Negatives

  • Mainly rely on self-reports (can therefore be biased or inaccurate)
  • Can be difficult to use in real-time applications.
  • Difficult to automate and human intervention is needed.

Why is this Interesting?

  • Preference Prediction
  • Facial expressions are one of the primary channels for humans to transmit emotional signals.

  • Often uncontrolled and spontaneous.

  • Collected unobtrusively via facial recognition software.

  • Much cheaper approach and easier to use compared to other neuromarketing tools.

Previous Work

  • Hakim et al., 2021 . Machines learn neuromarketing: Improving preference prediction from self-reports using multiple EEG measures and machine learning.
  • Höfling & Alpers, 2023 . Automatic facial coding predicts self-report of emotion, advertisement and brand effects elicited by video commercials.
  • Lu, Xiao, & Ding, 2016 . A video-based automated recommender (VAR) system for garments.
  • Zhou, Chen, Ferreira, & Smith, 2021 . Consumer behavior in the online classroom: using video analytics and machine learning to understand the consumption of video courseware.

Experimental Design

Experimental Design

Experimental Design

Experimental Design

Experimental Design

Experimental Design

Examples of Questions:

  • Product Knowledge: Do you know the product featured in the ad? [7=Very much, 1=Not at all]
  • Product Consumption: To what extent do you consume the advertised product? [7=Very much, 1=Not at all]

Experimental Design

Experimental Design

  • Binary Comparisons:
    • All possible pairs appear to each participant (15 combinations).
    • Products appear in random order.
    • Products appear for 3 seconds and time-outs are counted as mistrials.
  • Ranking: Please look at the list of ads you have watched and rate them from 1 to 6.
  • Take-out Choice: You can now choose one of the products. You will get this product afterward.

Data and Descriptive Statistics

Unprocessed Data

The unprocessed data have three dimensions:

  • Subject dimension (\(N=156\) participants).
  • Product dimension (\(J=6\) products).
    • Binary comparisons (\(P=15\) for each product).
  • Advertisement time.
    • Measurement every 200-milliseconds for the duration of the advertisement (e.g. 100 points for a 20-seconds ad).
    • Collected 8 Core (happy, sad, angry, surprised, scared, disgusted, contempt, and neutral) and 2 synthetic (arousal, valence) emotion measurements.

Pre-processing

  1. We exclude time points for which FaceReader did not provide a valid measurement.
    • E.g., the participant looked away from the camera.
  2. Exclude measurements for which we do not have enough time points.
    • No emotion measurements for more than 35% of the Ad.
  3. Facial expression measurement filters
    • Remove the first second of measurements while the face recognition software is still calibrating (no participants excluded).
    • Remove the last measurements, after the end of the advertisement.

Emotion consolidation

  • We consolidate emotion measurements in 2 ways:
    1. Average emotions: Mean over Ad measurements.
    2. Extreme emotions: Max measurement over Ad.

Post-processing

  • After pre-processing we are left with:
    • 154 (out of 156) participants,
    • 908 (out of 936) observations of participant-products pairs, and
    • 2237 (out of 2340) observations of participant-preference pairs.
  • Choice-set dataset (908 obs, 1/6 successes)
    • Features: Participant characteristics, marketing surveys, consolidated emotion measurements
    • Response: 1 if participant \(i\) chose product \(j\), 0 otherwise.
  • Preferences dataset (2237 obs, approximately 56% successes)
    • Features: Participant characteristics, \(\Delta\) marketing surveys, \(\Delta\) consolidated emotion measurements
    • Response: 1 if participant \(i\) chose \(Right\) in comparison \(p\), 0 otherwise.

Results

Logit model

Logit model

Logit model

Logit model

Logit model

Logit model

Logit model

Boosted Trees

Boosted Trees

Quadratic Discriminant Analysis

Quadratic Discriminant Analysis

Quadratic Discriminant Analysis

Support Vector Machines

Support Vector Machines

Artificial Neural Networks

Discussion

  • Direct implication/application:
    • Marketing surveys and consumer research studies can replace ad-emotion questions with live video reaction captures.
    • Our results indicate that the resulting ME emotion measurements are more granular and have more choice-predictive power than SR data.
    • Interviewers can spend more time with other questions.
  • Future applications:
    • Streamline our predictive ANN model with facial recognition technology to get real-time product personalization.
    • Either standalone or combined with other choice-relevant data (e.g., purchase history) the ME emotions can improve the accuracy of recommendations.
  • However…
    • Extracting information on preferences from ME shifts bargaining power.
    • The one using the technology can learn about the outside options of the counterparty.

Face Reading Technology: Improving Preference Prediction from Self-Reports Using Micro Expressions

  • Conducted an experiment capturing subject reactions during advertisement exposure.
  • Collected preference, survey, and SR and ME emotion data.
  • Used a comprehensive toolkit of ML methods to compare SR and ME measurements.
    • Predictive power improves from 1%-7% when ME replace SR emotions combined with survey data.
    • Predictive power improves from 1%-8% when ME are directly compared to SR emotions.
    • ME are more influential than SR measurements in choice predictions.
  • Sketched how our ANN model can be streamlined with facial recognition in practical applications.
  • Argued that this technology can affect the balance of bargaining power in market applications and regulation amendments might be appropriate.

References

Hakim, A., Klorfeld, S., Sela, T., Friedman, D., Shabat-Simon, M., & Levy, D. J. (2021). Machines learn neuromarketing: Improving preference prediction from self-reports using multiple EEG measures and machine learning. International journal of research in marketing, 38(3), 770–791.
Höfling, T. T. A., & Alpers, G. W. (2023). Automatic facial coding predicts self-report of emotion, advertisement and brand effects elicited by video commercials. Frontiers in neuroscience, 17, 1125983. Retrieved September 20, 2023, from https://www.frontiersin.org/articles/10.3389/fnins.2023.1125983/full
Lu, S., Xiao, L., & Ding, M. (2016). A Video-Based Automated Recommender (VAR) System for Garments. Marketing science, 35(3), 484–510. Retrieved September 20, 2023, from https://pubsonline.informs.org/doi/10.1287/mksc.2016.0984
Mayer, D., & Greenberg, H. M. (1964). What Makes a Good Salesman.
Zhou, M., Chen, G. H., Ferreira, P., & Smith, M. D. (2021). Consumer Behavior in the Online Classroom: Using Video Analytics and Machine Learning to Understand the Consumption of Video Courseware. Journal of marketing research, 58(6), 1079–1100. Retrieved September 20, 2023, from http://journals.sagepub.com/doi/10.1177/00222437211042013

Appendix A: Product Information

Appendix B.1 Analysis with Ad Attention Filter

Pre-processing

  1. Advertisement attention filter.
    • Remove participants who reported being unsure about seeing any of the included advertisements (excludes 35 participants)
  2. Facial expression measurement filters
    • Remove the first second of measurements while the face recognition software is still calibrating (no participants excluded).

Logit model

Boosted Trees

Quadratic Discriminant Analysis

Support Vector Machines

Artificial Neural Networks

Appendix B.2 Analysis with Incentive-Compatibility Filter

Pre-processing

  1. Transitivity as a preference attention filter.
    • Remove participants with non-transitive preferences (excludes 21 participants).
  2. Choose participants with preferences revealed to be incentive compatible.
    • Calculated the most preferred item from binary comparisons.
    • Remove participants with disagreements between most preferred and take-out choices (excludes 30 participants).
  3. Facial expression measurement filters
    • Remove the first second of measurements while the face recognition software is still calibrating (no participants excluded).

Logit model

Boosted Trees

Quadratic Discriminant Analysis

Support Vector Machines

Artificial Neural Networks

Appendix B.3 Analysis with Ad Attention and Incentive-Compatibility Filters

Pre-processing

  1. Transitivity as a preference attention filter.
    • Remove participants with non-transitive preferences (excludes 21 participants).
  2. Choose participants with preferences revealed to be incentive-compatible.
    • Calculated the most preferred item from binary comparisons.
    • Remove participants with disagreements between most preferred and take-out choices (excludes 30 participants).
  3. Advertisement attention filter.
    • Remove participants who reported being unsure about seeing any of the included advertisements (excludes 35 participants)
  4. Facial expression measurement filters
    • Remove the first second of measurements while the face recognition software is still calibrating (no participants excluded).

Logit model

Boosted Trees

Quadratic Discriminant Analysis

Support Vector Machines

Artificial Neural Networks

Appendix C: Analysis with Alternative Face Reading Software

Pre-processing

  1. Facial expression measurement filters
    • Remove the first second of measurements while the face recognition software is still calibrating (no participants excluded).

Logit model

Boosted Trees

Support Vector Machines

Artificial Neural Networks

Appendix D: Analysis with Additional Micro-expression Consolidations

Boosted Trees

Quadratic Discriminant Analysis

Artificial Neural Networks

Appendix E: Additional Analysis and Robustness Checks

Principal Component Analysis

Shrinkage and Selection