Perl Guru Forums: Perl Programming Help: Intermediate: Remove a line without reading whole file into memory

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
Remove a line without reading whole file into memory

mmcw2201
User

May 19, 2002, 11:49 PM

Post #1 of 8 (2396 views)

Remove a line without reading whole file into memory	Can't Post

At www.perl.com (FAQ) I saw an option to remove a line from a file without reading whole file into memory.
This however works only for the last line:

open (FH, "+< $file");
while ( <FH> ) { $addr = tell(FH) unless eof(FH) }
truncate(FH, $addr);

Can this be done with an other line in a file also?
I want to remove on a simple way a specified line from a file without reading it totaly into memory!

I know an option to read the whole file into memory. However the files are now getting big and I get out of memory errors!

my $datafile = "$datadir/$r_in->{'database'}.txt";

my $r_data = readdata($datafile);

my $fields = shift @{$r_data};
chomp($fields);
my @fields = split(/\|/,$fields);

foreach ($[ . $#fields){
$fields{$_} = $fields[$_];
}

my $j;
foreach $j ($[ . $#{$r_data}){
my @row = split (/\|/,$r_data->[$j]);
foreach ($[ . $#row) {
$record{$fields{$_}} = $row[$_];
}

# Remove item
splice(@{$r_data},$j,1) if ($record{'ID'} eq $r_in->{'productID'});
}

unshift(@$r_data,join("\|",@fields)."\n");
writedata($datafile,$r_data);

#########################################################################
# #
# subroutine readdata #
# #
#########################################################################

sub readdata {

my ($datafile) = shift;
my $r_data = [];

# Get file lock
lock_file("$datafile.lock");

if (open(DATA,"$datafile")) {
@{$r_data} = <DATA>;
close(DATA);
unlock_file("$datafile.lock");
}
else {
unlock_file("$datafile.lock");
my_die("Error in subroutine readdata: Can't open $datafile",$!);
}
return $r_data;
}

#########################################################################
# #
# subroutine writedata #
# #
#########################################################################

sub writedata {

my ($datafile,$r_rows) = @_;

# Get file lock
lock_file("$datafile.lock");

if (open(DATA,">$datafile")) {
print DATA @$r_rows;
close(DATA);
chmod(0666,$datafile);
unlock_file("$datafile.lock");
}
else {
unlock_file("$datafile.lock");
my_die("Error in subroutine writedata: Can't open $datafile",$!);
}
}

mhx
Enthusiast / Moderator

May 20, 2002, 12:49 AM

Post #2 of 8 (2394 views)

Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to]	Can't Post

I think you'll have to create a copy of the file with that single line removed. Say you've opened your IN and OUT file handles and $line is the number of the line to remove then:

Code

while( <IN> ) { $line == $. or print OUT }

should delete line number $line in the output file.

-- mhx

At last with an effort he spoke, and wondered to hear his own words, as if some other will was using his small voice. "I will take the Ring," he said, "though I do not know the way."

-- Frodo

mmcw2201
User

May 20, 2002, 1:28 AM

Post #3 of 8 (2391 views)

Re: [mhx] Remove a line without reading whole file into memory [In reply to]	Can't Post

Could you help me to complete script part?

uri
Thaumaturge

May 20, 2002, 9:04 PM

Post #4 of 8 (2382 views)

Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to]	Can't Post

Code

use Tie::File ; 
 
tie @foo, Tie::File, $file ; 
 
splice( @foo, $line, 1 ) ; 
 
untie @foo ;

yapp
User

May 23, 2002, 12:59 PM

Post #5 of 8 (2369 views)

Re: [uri] Remove a line without reading whole file into memory [In reply to]	Can't Post

I always use the following thing.. Is that better/worst is just different?

Well, actually I use my File::PlainIO module, which has an update(coderef) method. That chooses the best approach (slurping or file buffers) and receives an update function as first parameter, so it becomes $file->update($mysub)., where $mysub = sub {...};

Code

 #!/usr/bin/perl -w  
  
use strict;  
use Fcntl qw(:DEFAULT :flock);  
  
my $file = 'somefile.txt';  
my $temp = "$file~";  
  
sysopen(FH, $file, O_CREAT | O_RDWR) or die "Can't rdwr $file: $!";  
flock(FH, LOCK_EX);  
  
# Or use new_tmpfile IO::File  
sysopen(TEMP, $temp, O_CREAT | O_RDWR) or die "Can't make temp: $!";  
flock(TEMP, LOCK_EX);  
  
while(my $line = <FH>)  
{  
  my $remove = (..some logic...);  
  print TEMP $line unless $remove;  
}  
  
seek(TEMP, 0, 0) or die "Can't seen begin of temp: $!";  
seek(FH, 0,0) or die "Can't seen begin of $file: $!";  
truncate(FH, 0) or die "can't truncate: $!";  
  
print TEMP while <FH>;  # Mostly I use a read(..) with 4096 bytes at once.  
  
close(TEMP);  
unlink(TEMP) or die "Can't unlink temp: $!";  
  
close(FH);

Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

(This post was edited by yapp on May 23, 2002, 1:03 PM)

mmcw2201
User

May 26, 2002, 5:01 AM

Post #6 of 8 (2361 views)

Re: [yapp] Remove a line without reading whole file into memory [In reply to]	Can't Post

Two questions:

my $temp = "$file~";

Can I simple us $file??

Could you give me some more information about the File::PlainIO module?

yapp
User

May 27, 2002, 3:16 AM

Post #7 of 8 (2350 views)

Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to]	Can't Post

Theoretically, you can read and write in the same file, but that isn't such a good idea. In fact, you can write over the part you just needed to read. That's why I use a temporary file and write the data in there first.

About my File::PlainIO module. If you want to use it, I can e-mail you the latest version. I've been editing my modules a lot over the past week, and I had some fixes, which are not included in the download at my website right now.

The perldoc of my File::PlainIO is online available [url=http://www.codingdomain.com/cgi-bin/frameset.cgi?url=perl/downloads/x-modules/file_plainio.html]here If you want to use it, I'd be onnered. Or, just let me know what you think of it or what I can change.

I created this module so I don't code all io 10 times. The array's returned can defaultly have a certain size (and "" assigned for all missing elements, not found in the file). Locking is done automatically. \n chars are already chomp-ed The program can automatically die with an error. Also, I introduced an update function for fast changes. (saves you a lot of coding with sysopens) here is an example:

Code

use File::PlainIO; 
 
my $update_action = sub 
{ 
  s/aa/bb/;  # It's just like AWK :) 
}; 
 
my $file = new File::PlainIO("file.txt", MODE_RDWR, "Can't save"); 
$file->update( $update_action ); 
$file->close();

Revelation
Novice

Jun 11, 2002, 9:53 AM

Post #8 of 8 (2340 views)

Re: [uri] Remove a line without reading whole file into memory [In reply to]	Can't Post

URI got it right Wink

Tie::File is the appropriate tool for the job, and that code should and will work.

A good article on Tie::File is at http://www.perlmonks.com/index.pl?node_id=155567

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives