Python List Remove Duplicates

Python List Remove Duplicates

The many ways of removing duplicates from a list in Python

In Python list, remove duplicates by:

char_list = ["a", "a", "a", "b", "c"]
char_list = list(dict.fromkeys(char_list))

print(char_list)

Output:

['b', 'a', 'c']

Or if you care about the ordering, do it like this:

from collections import OrderedDict

char_list = ["a", "a", "a", "b", "c"]
char_list = list(OrderedDict.fromkeys(char_list))

print(char_list)

Output:

['a', 'b', 'c']

How Does It Work

This approach works by converting a list into a dictionary and then right back into a list.

It works because a dictionary (or an ordered dictionary) cannot have duplicate keys. Thus, the dict.fromkeys() method has to remove duplicates under the hood before converting the list into a dictionary.

After converting the dictionary has all the list elements as keys without duplicates. Then converting this dictionary back to a list gives you back the original list without duplicates.'

In case you didn’t like these approaches or just want to explore more ways to remove duplicates out of curiosity, here are some for you:

Use a For Loop

You can use a regular for loop to remove duplicates:

char_list = ["a", "a", "a", "b", "c"]

new_list = []
for char in char_list:
    if char not in new_list:
        new_list.append(char)

char_list = new_list
print(char_list)

Output:

['a', 'b', 'c']

Utilize List Comprehension

To replace the above for loop with a shorter alternative you can use a list comprehension:


char_list = ["a", "a", "a", "b", "c"]

new_list = []
[new_list.append(char) for char in char_list if char not in new_list]

char_list = new_list
print(char_list)

Output:

['a', 'b', 'c']

Use a Set

This is a popular approach to remove duplicates from a list in Python. Keep in mind this way you are going to lose the ordering of the list:

char_list = ["a", "a", "a", "b", "c"]
char_list = list(set(char_list))

print(char_list)

Output:

['b', 'c', 'a']

Conclusion

Thanks for reading. I hope you find it useful. Happy coding!