Interpretable Movie Review Sentiment Analysis
This project involved developing a model that not only predicts movie review sentiment with high accuracy but also explains the 'why' behind its decisions. I built a binary classification pipeline and an accompanying interpretability system using BERT to highlight the specific sentences that influence a prediction.
Dataset & Evaluation Metric
The dataset consists of 50,000 IMDB movie reviews. Each review includes its text, a sentiment label (0 for negative, 1 for positive), and a 1,536-dimensional embedding from OpenAI's text-embedding-3-large model used for the classification task. The project provided five different training/test splits to ensure model robustness.
The primary success metric was the Area Under the Curve (AUC), with a target of achieving a score of at least 0.986 across all five test splits.
The Two-Part Solution: Prediction and Interpretation
My approach was split into two main phases. First, I focused on building a highly accurate classification model. Second, I engineered a system to make that model's predictions transparent and understandable.
Part 1: High-Accuracy Sentiment Prediction
The classification model is a Logistic Regression classifier with an elasticnet penalty, which is well-suited for high-dimensional data. Before training, all features were standardized using StandardScaler. This model was trained and evaluated on each of the five data splits, consistently exceeding the target AUC score.
| Data Split | AUC Score | Execution Time (s) |
|---|---|---|
| Split 1 | 0.98693 | 39.07 |
| Split 2 | 0.98661 | 45.29 |
| Split 3 | 0.98633 | 39.56 |
| Split 4 | 0.98694 | 41.64 |
| Split 5 | 0.98622 | 45.93 |
Performance measured on a Macbook Pro (16GB RAM, M1 Pro).
Part 2: Engineering for Interpretability with BERT
A high accuracy score is great, but understanding why a model makes a decision is crucial for trust and debugging. To achieve this, I built an interpretation system that pinpoints influential sentences within a review.
- The system takes a review and splits it into individual sentences.
- A pre-trained BERT model generates an embedding for each sentence.
- Each sentence embedding is passed to the trained classifier to get a sentiment probability.
- Sentences with probabilities above a positive threshold (0.46) or below a negative threshold (0.42) are highlighted as key drivers of the overall prediction.
This method provides a clear, visual explanation for the model's output, as shown in the interactive demo below.
Interpretability in Action
Below are 3 positive and 3 negative reviews from the test set. Click the tabs to see the sentences my model highlighted as being most important for its decision.
Loading positive reviews...
Loading negative reviews...
Final Results and Conclusion
The final model successfully met both project objectives, achieving a robust average AUC score above the 0.986 target while providing clear, sentence-level explanations for its predictions. This project highlights a complete workflow from building an accurate predictive model to ensuring its outputs are transparent and trustworthy. The interpretation method proved effective at identifying phrases that contribute to a prediction, though its reliance on BERT's complex attention mechanism means some highlighted phrases may not always seem intuitive.
Interactive Positive Review Analysis
An interactive explanation of the model's reasoning. Hover over words to see their impact.
Final Average AUC Score:
~0.986
Target Score: 0.986