Diff of two Dataframes

Posted on

Question :

Diff of two Dataframes

I need to compare two dataframes of different size row-wise and print out non matching rows. Lets take the following two:

df1 = DataFrame({
'Buyer': ['Carl', 'Carl', 'Carl'],
'Quantity': [18, 3, 5, ]})

df2 = DataFrame({
'Buyer': ['Carl', 'Mark', 'Carl', 'Carl'],
'Quantity': [2, 1, 18, 5]})

What is the most efficient way to row-wise over df2 and print out rows not in df1 e.g.

Buyer     Quantity 
Carl         2
Mark         1

Important: I do not want to have row:

Buyer     Quantity 
Carl         3

Included in the diff:

I have already tried:
Comparing two dataframes of different length row by row and adding columns for each row with equal value
and Compare two DataFrames and output their differences side-by-side

But these do not match with my problem.

Asked By: Andy


Answer #1:

merge the 2 dfs using method ‘outer’ and pass param indicator=True this will tell you whether the rows are present in both/left only/right only, you can then filter the merged df after:

In [22]:
merged = df1.merge(df2, indicator=True, how='outer')
merged[merged['_merge'] == 'right_only']

  Buyer  Quantity      _merge
3  Carl         2  right_only
4  Mark         1  right_only
Answered By: EdChum

Answer #2:

you may find this as the best:

df2[ ~df2.isin(df1)].dropna()
Answered By: A. Nadjar

Answer #3:

diff = set(zip(df2.Buyer, df2.Quantity)) - set(zip(df1.Buyer, df1.Quantity))

This is the first solution that came to mind. You can then put the diff set back in a DF for presentation.

Answered By: Shovalt

Answer #4:

@EdChum’s answer is self-explained. But using not 'both' condition makes more sense and you do not need to care about the order of comparison, and this is what a real diff supposed to be. For the sake of answering your question:

merged = df1.merge(df2, indicator=True, how='outer')
merged.loc = [merged['_merge'] != 'both']
Answered By: Shayan Amani

Answer #5:

Try the following if you only care about adding the new Buyers to the other df:

df_delta=df2[df2['Buyer'].apply(lambda x: x not in df1['Buyer'].values)]
Answered By: Yasir

Answer #6:

As of Pandas 1.1.0, there is pandas.DataFrame.compare:



Answered By: gwelter

Leave a Reply

Your email address will not be published. Required fields are marked *