Python List Remove Duplicates
The many ways of removing duplicates from a list in Python
In Python list, remove duplicates by:
char_list = ["a", "a", "a", "b", "c"]
char_list = list(dict.fromkeys(char_list))
print(char_list)
Output:
['b', 'a', 'c']
Or if you care about the ordering, do it like this:
from collections import OrderedDict
char_list = ["a", "a", "a", "b", "c"]
char_list = list(OrderedDict.fromkeys(char_list))
print(char_list)
Output:
['a', 'b', 'c']
How Does It Work
This approach works by converting a list into a dictionary and then right back into a list.
It works because a dictionary (or an ordered dictionary) cannot have duplicate keys. Thus, the dict.fromkeys()
method has to remove duplicates under the hood before converting the list into a dictionary.
After converting the dictionary has all the list elements as keys without duplicates. Then converting this dictionary back to a list gives you back the original list without duplicates.'
In case you didn’t like these approaches or just want to explore more ways to remove duplicates out of curiosity, here are some for you:
Use a For Loop
You can use a regular for loop to remove duplicates:
char_list = ["a", "a", "a", "b", "c"]
new_list = []
for char in char_list:
if char not in new_list:
new_list.append(char)
char_list = new_list
print(char_list)
Output:
['a', 'b', 'c']
Utilize List Comprehension
To replace the above for loop with a shorter alternative you can use a list comprehension:
char_list = ["a", "a", "a", "b", "c"]
new_list = []
[new_list.append(char) for char in char_list if char not in new_list]
char_list = new_list
print(char_list)
Output:
['a', 'b', 'c']
Use a Set
This is a popular approach to remove duplicates from a list in Python. Keep in mind this way you are going to lose the ordering of the list:
char_list = ["a", "a", "a", "b", "c"]
char_list = list(set(char_list))
print(char_list)
Output:
['b', 'c', 'a']
Conclusion
Thanks for reading. I hope you find it useful. Happy coding!