CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Remove a line without reading whole file into memory

 



mmcw2201
User

May 19, 2002, 11:49 PM

Post #1 of 8 (3151 views)
Remove a line without reading whole file into memory Can't Post

At www.perl.com (FAQ) I saw an option to remove a line from a file without reading whole file into memory.
This however works only for the last line:

open (FH, "+< $file");
while ( <FH> ) { $addr = tell(FH) unless eof(FH) }
truncate(FH, $addr);

Can this be done with an other line in a file also?
I want to remove on a simple way a specified line from a file without reading it totaly into memory!


I know an option to read the whole file into memory. However the files are now getting big and I get out of memory errors!


my $datafile = "$datadir/$r_in->{'database'}.txt";

my $r_data = readdata($datafile);

my $fields = shift @{$r_data};
chomp($fields);
my @fields = split(/\|/,$fields);

foreach ($[ . $#fields){
$fields{$_} = $fields[$_];
}

my $j;
foreach $j ($[ . $#{$r_data}){
my @row = split (/\|/,$r_data->[$j]);
foreach ($[ . $#row) {
$record{$fields{$_}} = $row[$_];
}

# Remove item
splice(@{$r_data},$j,1) if ($record{'ID'} eq $r_in->{'productID'});
}

unshift(@$r_data,join("\|",@fields)."\n");
writedata($datafile,$r_data);


#########################################################################
# #
# subroutine readdata #
# #
#########################################################################

sub readdata {

my ($datafile) = shift;
my $r_data = [];

# Get file lock
lock_file("$datafile.lock");

if (open(DATA,"$datafile")) {
@{$r_data} = <DATA>;
close(DATA);
unlock_file("$datafile.lock");
}
else {
unlock_file("$datafile.lock");
my_die("Error in subroutine readdata: Can't open $datafile",$!);
}
return $r_data;
}


#########################################################################
# #
# subroutine writedata #
# #
#########################################################################

sub writedata {

my ($datafile,$r_rows) = @_;

# Get file lock
lock_file("$datafile.lock");

if (open(DATA,">$datafile")) {
print DATA @$r_rows;
close(DATA);
chmod(0666,$datafile);
unlock_file("$datafile.lock");
}
else {
unlock_file("$datafile.lock");
my_die("Error in subroutine writedata: Can't open $datafile",$!);
}
}


mhx
Enthusiast / Moderator

May 20, 2002, 12:49 AM

Post #2 of 8 (3149 views)
Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to] Can't Post

I think you'll have to create a copy of the file with that single line removed. Say you've opened your IN and OUT file handles and $line is the number of the line to remove then:


Code
while( <IN> ) { $line == $. or print OUT }


should delete line number $line in the output file.

-- mhx

At last with an effort he spoke, and wondered to hear his own words, as if some other will was using his small voice. "I will take the Ring," he said, "though I do not know the way."

-- Frodo



mmcw2201
User

May 20, 2002, 1:28 AM

Post #3 of 8 (3146 views)
Re: [mhx] Remove a line without reading whole file into memory [In reply to] Can't Post

Could you help me to complete script part?


uri
Thaumaturge

May 20, 2002, 9:04 PM

Post #4 of 8 (3137 views)
Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to] Can't Post


Code
use Tie::File ; 

tie @foo, Tie::File, $file ;

splice( @foo, $line, 1 ) ;

untie @foo ;


yapp
User

May 23, 2002, 12:59 PM

Post #5 of 8 (3124 views)
Re: [uri] Remove a line without reading whole file into memory [In reply to] Can't Post

I always use the following thing.. Is that better/worst is just different?

Well, actually I use my File::PlainIO module, which has an update(coderef) method. That chooses the best approach (slurping or file buffers) and receives an update function as first parameter, so it becomes $file->update($mysub)., where $mysub = sub {...};


Code
 #!/usr/bin/perl -w  

use strict;
use Fcntl qw(:DEFAULT :flock);

my $file = 'somefile.txt';
my $temp = "$file~";

sysopen(FH, $file, O_CREAT | O_RDWR) or die "Can't rdwr $file: $!";
flock(FH, LOCK_EX);

# Or use new_tmpfile IO::File
sysopen(TEMP, $temp, O_CREAT | O_RDWR) or die "Can't make temp: $!";
flock(TEMP, LOCK_EX);

while(my $line = <FH>)
{
my $remove = (..some logic...);
print TEMP $line unless $remove;
}

seek(TEMP, 0, 0) or die "Can't seen begin of temp: $!";
seek(FH, 0,0) or die "Can't seen begin of $file: $!";
truncate(FH, 0) or die "can't truncate: $!";

print TEMP while <FH>; # Mostly I use a read(..) with 4096 bytes at once.

close(TEMP);
unlink(TEMP) or die "Can't unlink temp: $!";

close(FH);


Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

(This post was edited by yapp on May 23, 2002, 1:03 PM)


mmcw2201
User

May 26, 2002, 5:01 AM

Post #6 of 8 (3116 views)
Re: [yapp] Remove a line without reading whole file into memory [In reply to] Can't Post

Two questions:

my $temp = "$file~";

Can I simple us $file??

Could you give me some more information about the File::PlainIO module?


yapp
User

May 27, 2002, 3:16 AM

Post #7 of 8 (3105 views)
Re: [mmcw2201] Remove a line without reading whole file into memory [In reply to] Can't Post

Theoretically, you can read and write in the same file, but that isn't such a good idea. In fact, you can write over the part you just needed to read. That's why I use a temporary file and write the data in there first.

About my File::PlainIO module. If you want to use it, I can e-mail you the latest version. I've been editing my modules a lot over the past week, and I had some fixes, which are not included in the download at my website right now.

The perldoc of my File::PlainIO is online available [url=http://www.codingdomain.com/cgi-bin/frameset.cgi?url=perl/downloads/x-modules/file_plainio.html]here If you want to use it, I'd be onnered. Or, just let me know what you think of it or what I can change.

I created this module so I don't code all io 10 times. The array's returned can defaultly have a certain size (and "" assigned for all missing elements, not found in the file). Locking is done automatically. \n chars are already chomp-ed The program can automatically die with an error. Also, I introduced an update function for fast changes. (saves you a lot of coding with sysopens) here is an example:


Code
use File::PlainIO; 

my $update_action = sub
{
s/aa/bb/; # It's just like AWK :)
};

my $file = new File::PlainIO("file.txt", MODE_RDWR, "Can't save");
$file->update( $update_action );
$file->close();


Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool


Revelation
Novice

Jun 11, 2002, 9:53 AM

Post #8 of 8 (3095 views)
Re: [uri] Remove a line without reading whole file into memory [In reply to] Can't Post

URI got it right Wink Tie::File is the appropriate tool for the job, and that code should and will work.

A good article on Tie::File is at http://www.perlmonks.com/index.pl?node_id=155567

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives