pandas DataFrame output end of csv

Posted on

Question :

pandas DataFrame output end of csv

I wonder how to add new DataFrame data onto the end of an existing csv file? The to_csv doesn’t mention such functionality.

Asked By: perigee


Answer #1:

You can append using to_csv by passing a file which is open in append mode:

with open(file_name, 'a') as f:
    df.to_csv(f, header=False)

Use header=None, so as not to append the column names.

In fact, pandas has a wrapper to do this in to_csv using the mode argument (see Joe’s answer):

df.to_csv(f, mode='a', header=False)
Answered By: Andy Hayden

Answer #2:

You can also pass the file mode as an argument to the to_csv method

df.to_csv(file_name, header=False, mode = 'a')
Answered By: Joe Hooper

Answer #3:

A little helper function I use (based on Joe Hooper’s answer) with some header checking safeguards to handle it all:

def appendDFToCSV_void(df, csvFilePath, sep=","):
    import os
    if not os.path.isfile(csvFilePath):
        df.to_csv(csvFilePath, mode='a', index=False, sep=sep)
    elif len(df.columns) != len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns):
        raise Exception("Columns do not match!! Dataframe has " + str(len(df.columns)) + " columns. CSV file has " + str(len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns)) + " columns.")
    elif not (df.columns == pd.read_csv(csvFilePath, nrows=1, sep=sep).columns).all():
        raise Exception("Columns and column order of dataframe and csv file do not match!!")
        df.to_csv(csvFilePath, mode='a', index=False, sep=sep, header=False)
Answered By: KCzar

Answer #4:

Thank to Andy, the complete solution:

f = open(filename, 'a') # Open file as append mode
df.to_csv(f, header = False)
Answered By: perigee

Leave a Reply

Your email address will not be published.