Python Remove Duplicates from List

Python remove duplicates from list | We will discuss how to remove duplicate items or elements from the list. In Python, there are many methods available on the list data type that help you remove elements from a given list. In this post, we are using native methods, list comprehension, list set(), enumerate(), OrderedDict.fromkeys(), NumPy function.

Remove Duplicates in List in Python

We will take the list while declaring the variables then, removes duplicates element from the list using the For Loop, if-else statement, and append() function. Finally, the new list will be displayed on the screen.

# Python program to remove duplicates from list

# take list
my_list = [1, 2, 3, 1, 5, 3, 4, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates item using native method
new_list = []
for i in my_list:
    if i not in new_list:
        new_list.append(i)

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Using List Comprehension

This method is similar to the above method, but rather a shorthand method. In this program, we use the list comprehension technique. This is just a one-liner shorthand of the longer method.

# Python program to remove duplicates from list

# take list
my_list = [1, 2, 3, 1, 4, 3, 5, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates item using list comprehension
new_list = []
[new_list.append(x) for x in my_list if x not in new_list]

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Using list comprehension + enumerate() Function

Enumerate returns an object with a counter to each element in the list. For example (5,1), (1,3) etc. Here the first value is the index, and the second value is the list item.

# Python program to remove duplicates from list

# take list
my_list = [1, 2, 3, 1, 4, 3, 5, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates using list comprehension + enumerate()
new_list = [i for n, i in enumerate(my_list) if i not in my_list[:n]]

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Using set() Function

The common approach to get a unique collection of items is to use a set. Sets are unordered collections of distinct objects. To create a set from any iterable, you can simply pass it to the built-in set() function. If you later need a real list again, you can similarly pass the set to the list() function.

# Python program to remove duplicates from list

# take list
my_list = [1, 2, 3, 1, 5, 3, 4, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates item using list set()
new_list = list(set(my_list))

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Using collections.OrderedDict.fromkeys()

This is the fastest method to achieve a particular task. This method is available from python2.7 onwards. It first removes the duplicates and returns a dictionary which has to be converted to a list. To make use of the OrderedDict.fromkey() method, you have to import OrderedDict from collections.

# Python program to remove duplicates from list
from collections import OrderedDict

# take list
my_list = [1, 2, 3, 1, 4, 3, 5, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates using collections.OrderedDict.fromkeys()
new_list = list(OrderedDict.fromkeys(my_list))

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Using NumPy Method

The unique() method can help us remove duplicate elements from the given list. The output is converted back to a list format using tolist() method.

# Python program to remove duplicates from list
import numpy as np

# take list
my_list = [1, 2, 3, 1, 4, 3, 5, 2, 7]

# printing original list
print('List:', my_list)

# removed duplicates using numpy
new_list = np.unique(my_list).tolist()

# print list after item deletion
print('New list:', new_list)

Output:-

List: [1, 2, 3, 1, 4, 3, 5, 2, 7]
New list: [1, 2, 3, 4, 5, 7]

Also See:- Delete all Elements in List Python

If you enjoyed this post, share it with your friends. Do you want to share more information about the topic discussed above or do you find anything incorrect? Let us know in the comments. Thank you!

Leave a Comment

Your email address will not be published. Required fields are marked *