AWS SageMaker Batch Transform CSV Error: Bare " in non quoted field

AWS SageMaker Batch Transform errors with the following:

bare " in non quoted field found near: "['R627' 'Q2739' 'D509' 'S37009A' 'E860' 'D72829' 'R9431' 'J90' 'R7989'  

In a SageMaker Studio notebook, I use Pandas to output data to csv:

data.to_csv(my_file, index=False, header=False)  

My Pandas dataframe has columns with string values like the following:

['ABC123', 'DEF456']  

Pandas is adding line breaks between these fields e.g. this is one row (that spans two lines) and has a line break. Note that the double quotes now span two lines. Sometimes they'll span 3 or more lines.

False,ABC123,7,1,3412,['I509'],,"['R627' 'Q2739' 'D509' 'S37009A' 'E860' 'D72829' 'R9431' 'J90' 'R7989'   'R5383' 'J9621']",['R51' 'R05' 'R0981'],['X58XXXA'],M,,A,48  

The CSV is valid and I can successfully read it back into a Pandas dataframe.

Why would Batch Transform fail to read this CSV format?

https://stackoverflow.com/questions/65964239/aws-sagemaker-batch-transform-csv-error-bare-in-non-quoted-field January 30, 2021 at 11:03AM