How to Find Duplicates of a List in Python: Methods

The list allows you to store many variables in a single variable. Due to this, it may store duplicate elements in it. In this tutorial, you will learn how to find duplicates of a list in Python using various methods.

Methods to Find Duplicates of a List in Python

Method 1: Using set

When you Convert the list to a set, it automatically removes duplicate elements. After that Compare the length of the set to the length of the original list. If the lengths are different, it means there were duplicates in the original list.

Find the duplicates using the below code.

my_list = [1, 2, 3, 4, 2, 5, 6, 3, 7]
duplicates = len(my_list) - len(set(my_list))
print("Number of duplicates:", duplicates)

Method 2: Find Duplicates of a List in Python Using a dictionary

The following steps you will use to find duplicates of a list using a dictionary.

1. First Create an empty dictionary to store the frequency of each element.
2. Then Iterate over the list and update the count of each element in the dictionary.
3. Print the elements with a count greater than 1.

my_list = [1, 2, 3, 4, 2, 5, 6, 3, 7]
frequency = {}
for item in my_list:
    if item in frequency:
        frequency[item] += 1
    else:
        frequency[item] = 1

duplicates = [item for item in frequency if frequency[item] > 1]
print("Duplicates:", duplicates)

Method 3: Using list comprehension and the count() method

You can also use the list comprehension and count() function to find the duplicates. Just folow the steps.

1. Iterate over the list using a list comprehension.
2. After that check the count of each element in the list.
3. Append the element to a new list if its count is greater than 1.

my_list = [1, 2, 3, 4, 2, 5, 6, 3, 7]
duplicates = [item for item in my_list if my_list.count(item) > 1]
print("Duplicates:", duplicates)

Method 4: Using the collections.counter class

You can also use the collections.counter class to find the duplicates. Follow the below steps for that.

1. Import the Counter class from the collections module.
2. Create a counter object by passing the list to the Counter class.
3. Use the most_common() method to get a list of elements and their counts.
4. Filter the elements with a count greater than 1.

from collections import Counter

my_list = [1, 2, 3, 4, 2, 5, 6, 3, 7]
counter = Counter(my_list)
duplicates = [item for item, count in counter.most_common() if count > 1]
print("Duplicates:", duplicates)

Method 5: Using the setdefault() function

To find the duplicates of a list using this method then use the below steps.

1. Create an empty dictionary.
2. Iterate over the list and use the setdefault() method to initialize a count for each element in the dictionary.
3. Increment the count for each occurrence of an element.
4. Print the elements with a count greater than 1.

my_list = [1, 2, 3, 4, 2, 5, 6, 3, 7]
frequency = {}
for item in my_list:
    frequency.setdefault(item, 0)
    frequency[item] += 1

duplicates = [item for item in frequency if frequency[item] > 1]
print("Duplicates:", duplicates)

Conclusion

It’s better that you should not use the duplicates in the list as it is meaningless. In this tutorial, you learned five methods to remove duplicates in the list. You have used set, dictionary, list comprehension, and third-party module. You can use any method as you wish or for convenience.