Classifying 50,000 IMDB reviews as positive or negative, which is a solved problem, and then the part I actually cared about: working out which sentences in a review drove the call.
The reviews came with OpenAI embeddings already computed, and scoring individual sentences would have meant paying to embed every sentence separately. So instead I learned a linear map from BERT's embedding space into OpenAI's, ran the sentences through BERT, and pushed them through the map. It is a cheap trick and it worked better than it had any right to.
Dataset & Evaluation Metric
The dataset consists of 50,000 IMDB movie reviews. Each review includes its text, a sentiment label (0 for negative, 1 for positive), and a 1,536-dimensional embedding from OpenAI's text-embedding-3-large model used for the classification task. The project provided five different training/test splits to ensure model robustness.
The primary success metric was the Area Under the Curve (AUC), with a target of achieving a score of at least 0.986 across all five test splits.
The Two-Part Solution: Prediction and Interpretation
My approach was split into two main phases. First, I focused on building a highly accurate classification model. Second, I engineered a system to make that model's predictions transparent and understandable.
Part 1: High-Accuracy Sentiment Prediction
The classification model is a Logistic Regression classifier with an elasticnet penalty, which is well-suited for high-dimensional data. Before training, all features were standardized using StandardScaler. This model was trained and evaluated on each of the five data splits, consistently exceeding the target AUC score.
| Data Split | AUC Score | Execution Time (s) |
|---|---|---|
| Split 1 | 0.98693 | 39.07 |
| Split 2 | 0.98661 | 45.29 |
| Split 3 | 0.98633 | 39.56 |
| Split 4 | 0.98694 | 41.64 |
| Split 5 | 0.98622 | 45.93 |
Performance measured on a Macbook Pro (16GB RAM, M1 Pro).
Part 2: Engineering for Interpretability with BERT
A high accuracy score is great, but understanding why a model makes a decision is crucial for trust and debugging. To achieve this, I built an interpretation system that pinpoints influential sentences within a review.
- The system takes a review and splits it into individual sentences.
- A pre-trained BERT model generates an embedding for each sentence.
- A linear map, fit on 500 reviews that had both embeddings, projects each 768-dimensional BERT embedding into the 1,536-dimensional OpenAI space the classifier was trained on.
- The mapped embedding is passed to the trained classifier to get a sentiment probability.
- Sentences with probabilities above a positive threshold (0.46) or below a negative threshold (0.42) are highlighted as key drivers of the overall prediction.
This method provides a clear, visual explanation for the model's output, as shown in the interactive demo below.
Interpretability in Action
Below are 3 positive and 3 negative reviews from the test set. Click the tabs to see the sentences my model highlighted as being most important for its decision.
Loading positive reviews...
Loading negative reviews...
Final Results and Conclusion
The final model successfully met both project objectives, achieving a robust average AUC score above the 0.986 target while providing clear, sentence-level explanations for its predictions. This project highlights a complete workflow from building an accurate predictive model to ensuring its outputs are transparent and trustworthy. The interpretation method proved effective at identifying phrases that contribute to a prediction, though its reliance on BERT's complex attention mechanism means some highlighted phrases may not always seem intuitive.
Interactive Positive Review Analysis
SHAP token attributions for one positive review, added after the course when I came back to the project. Hover over words to see their impact.
Final Average AUC Score:
~0.986
Target Score: 0.986