CountVectorizer with Pandas dataframe returning wrong shape

import numpy as np  import pandas as pd  from sklearn.feature_extraction.text import CountVectorizer  from sklearn.naive_bayes import MultinomialNB  from sklearn.metrics import (f1_score,precision_score,recall_score)  ifile=open("train_pos.txt")  rows = []  for ln in ifile:      rows.append({'text': ln, 'class': 1})  ifile.close()  data_frame = pd.DataFrame(rows)  data_frame  

This code outputs:

text    class  0   Coffee is great and I live close so it's conve...   1  1   I love this place for its coffeeshop feel with...   1  2   I've come here now a couple of times and I lov...   1  3   Nice vibes, just ok food and not too warm serv...   1  4   Not a big breakfast person but this place has ...   1  ... ... ...  74241   Henry the bartender makes a stop in at Bijans ...   1  74242   I love the ambiance at Bijan, especially the w...   1  74243   Popped in for Happy Hour, on a hot and stormy ...   1  74244   Update: Â Bijan's came on the scene as a great...   1  74245   The nearly day-glo lime green walls, harsh flu...   1  74246 rows × 2 columns  

I am trying to perform feature extraction using countvectorizer with the pandas data_frame as input.

To do this I did the code

count_vect = CountVectorizer()   X_train_counts = count_vect.fit_transform(data_frame.text)  X_train_counts.shape     

The problem is it is giving me the wrong shape when I run the above code. The shape is outputting (74246, 61803) but it supposed to output (74246, 2). It gives the correct output when I run data_frame.shape

Does anyone know why this is happening and how to fix it?

Any help would be very much appreciated!

https://stackoverflow.com/questions/66556961/countvectorizer-with-pandas-dataframe-returning-wrong-shape March 10, 2021 at 09:06AM