Home: Perl Programming Help: Beginner:
How do I remove a duplicate record?



DippinPete
Deleted

Jan 4, 2001, 8:55 AM


Views: 253
How do I remove a duplicate record?

Hi...
I've got a file that looks like

1|3|4|1|MILK
6|6|3|2|GOLF
3|2|6|7|DOG
6|6|3|2|GOLF

I'd like some perl that can look at the file and delete the duplicate record, and leave only one.

Thanks in advance,

Pete





BigRich
Novice

Jan 5, 2001, 1:01 AM


Views: 243
Re: How do I remove a duplicate record?

Your answer resides where 99% of most newbie questions are already answered. In the docs that are included with your distribution of Perl.

perldoc perlfaq4

Data Manipulation: How can I remove duplicate elements from a list or array?

One example from the faq:

undef %saw;

@saw{@in} = ();

@out = sort keys %saw;


To put the example to use.

#!perl -wT

use strict;

print "Content-type: text/html\n\n";

my($data_file, @in,%saw,@out, $new_file);
$data_file = "/path/to_your/file.dat";
$new_file = "/path/to_your/new_file.dat";
open(FILE, "$data_file") or die "$!";
while (<FILE>) {
chomp;
push @in, $_;
}
close(FILE);
############ from perlfaq4
undef %saw;
@saw{@in} = ();
@out = sort keys %saw;
############
open(NEWFILE, ">$new_file") or die "$!";
foreach (@out){
print NEWFILE $_."\n";
}
close(NEWFILE);
print "New file created with no duplicates.";
__END__



I'm not trying to be sarcastic but most people who are new to Perl fail to realize that the chances of them asking a question that is not already answered in the docs/faqs is slim and none.

There are also answers to questions you haven't even thought of yet :^ )

BigRich