# Solved: Finding duplicate lines



## johnni (Dec 16, 2003)

I have some lists in .txt format which I feel contain some duplicate entries, The lists have been sorted using TedNotepad but it is time-consuming looking for duplicates. I opened one of the lists in Word 2000 thinking that it had a duplicate finder but I must have been mistaken!

Does anyone know of a text editor that can find duplicate lines, please?

johnni


----------



## lister (Aug 10, 2004)

Do you have Excel?


----------



## johnni (Dec 16, 2003)

Yes, I have Excel 2000

johnni


----------



## lister (Aug 10, 2004)

Though not a dupechecker, you could open the file in excel and sort them alphabetically; they should be easier to spot.


----------



## johnni (Dec 16, 2003)

Thanks Lister but my lists are already sorted alphabetically thanks to TED Notepad. I opened the file in Excel which made it slightly easier to read line by line, but only slightly. I am really wanting a text editor which can spot adjacent duplicate lines, and perhaps delete one of them. Any suggestions? TED Notepad can do all sorts of things that MS Notepad can't do, but it cannot do what I am looking for!

johnni


----------



## lister (Aug 10, 2004)

http://www.microsoft.com/technet/scriptcenter/resources/qanda/aug05/hey0819.mspx


----------



## Squashman (Apr 4, 2003)

A couple of batch files and ways to do it in excel here.
http://qa.techinterviews.com/q/20060803091044AAZLBsp

Looks like UltraEdit can do it as well. But it is not a free program.
http://myitforum.com/cs2/blogs/dhite/archive/2006/05/21/20470.aspx

And if you are using Windows XP or Vista you can fire up PowerShell to do it as well.
http://www.tellingmachine.com/post/...ate-lines-in-text-files-using-PowerShell.aspx


----------



## EAFiedler (Apr 25, 2000)

I like this tip I ran across here at TSG.

In Excel, select the column containing your data,
Data > Filter > Advanced Filter
Select: *Filter the list in place*
Place a check mark in: *Unique records only*
Click: *OK*

Copy the unique records to another spreadsheet.

I just tried it in Excel 2000 and that capability is available in that version.

Let us know if that works for you or not.


----------



## johnni (Dec 16, 2003)

Thank you for all your suggestions. Lister and Squashman - I'm afraid your links were outside my comprehension! But EAFiedler I found your use of Advanced Filter worked well and I then resaved the file in text delimited format I noticed that the new Excel worksheet was some 30 rows less than the original which shows the number of duplicate entries. Thank you all again.

johnni


----------

