Perl Guru Forums: Perl Programming Help: Intermediate: Database Unexpected Erasure Problem

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
Database Unexpected Erasure Problem

gregarios
stranger

May 18, 2002, 7:05 PM

Post #1 of 17 (3648 views)

Database Unexpected Erasure Problem	Can't Post

I can't figure out why about once every two weeks, this code will completely
erase the database contents. It seems to work just fine with no errors for
hundreds of hits, then POW, no ads...

The database is a pipe delimited file with a list of ads and their view counts in it.
A random text ad is displayed and that ad's display count is raised on each viewing.

I even made it to have the ad display and not raise the count if the file
cannot be opened in a timely manner due to a exclusive locks from heavy use.

Can someone tell me what is wrong here? :-)

Code

#!/usr/bin/perl5 -wT 
use strict; 
 
$| = 1; 
if (!open (ADDCTX, "<ads.dta")) {die "Could not open file: $!";} 
flock(ADDCTX,1) or die "Could not lock file: $!"; 
seek(ADDCTX,0,0); 
my @ads = <ADDCTX>; 
close(ADDCTX); 
my @TLAD = split(/\|/,$ads[rand @ads]); 
$TLAD[2]++; 
$ads[$TLAD[0]] = "$TLAD[0]|$TLAD[1]|$TLAD[2]\n"; 
print "Content-type: text/html\n\n"; 
print "Advertisement: $TLAD[1]"; 
if (open (ADDCTZ, "+<ads.dta")) { 
flock(ADDCTZ,2) or die "Could not lock file: $!"; 
truncate ADDCTZ,0; 
seek(ADDCTZ,0,0); 
print ADDCTZ @ads; 
close(ADDCTZ); 
} 
exit;

The database (ads.dta) is in this format:

Code

0|advertisement|123 
1|anotheradvertisement|32 
2|advertisementthree|55

Please help if you can.

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

(This post was edited by gregarios on May 18, 2002, 7:08 PM)

Danni
Novice

May 18, 2002, 7:56 PM

Post #2 of 17 (3643 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

The cause of this problem could be that the file is read as the file is truncated, or open fails when trying to read the file, but works when writing to it.

Ive personaly had this problem with flatfile databases.

One of my solutions was to do all the required file handling within one open file with an exclusive lock.

Code

 #!/usr/bin/perl5 -wT  
use strict;  
  
$| = 1;  
 
 
  
 
open (ADDCTX,"+<ads.dta") or die ("Open Failed: $!"); 
flock ADDCTX,2  or die ("flock Failed: $!"); 
my @ads = <ADDCTX>;  
 
my @TLAD = split(/\|/,$ads[rand @ads]);  
$TLAD[2]++;  
$ads[$TLAD[0]] = "$TLAD[0]|$TLAD[1]|$TLAD[2]\n";  
print "Content-type: text/html\n\n";  
print "Advertisement: $TLAD[1]";  
 
truncate ADDCTZ,0;  
seek(ADDCTZ,0,0);  
print ADDCTZ @ads;  
close (ADDCTZ);

How ever i still had the same problem, just on a much smaller scale. I would recormend using dbfile, since a tied hash could be more effective for what your doing here. Or best yet use a SQL server like MySQL which would be much more effective.

sebastian
Novice

May 20, 2002, 4:46 PM

Post #3 of 17 (3634 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

I am using an exclusive flock and am still having the same problem.

sebastian
Novice

May 20, 2002, 4:51 PM

Post #4 of 17 (3633 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

Probably this appears when the content of an array is lost due to memory problems and could be solved like this:

if ($content eq "") {

exit;

}

else {

open (DATA , ">anyfile.txt");

print DATA $content;

}

close(DATA);

This would just crash the application in case the variable does not contain anything... Actually what does this problem come from ? What do you think ?

gregarios
stranger

May 20, 2002, 8:08 PM

Post #5 of 17 (3630 views)

Re: [sebastian] Database Unexpected Erasure Problem [In reply to]	Can't Post

Actually, believe it or not, I tried verifying that there was data in the array before writing it back to the file like you suggest, and it STILL didn't fix it! Needless to say I was shocked. Here's a strange clue that I'm not sure holds the answer somehow: When I change the truncate to 1 or 2 instead of 0, I open it up to find the entire file reduced to one or two characters when it happens.

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

sebastian
Novice

May 21, 2002, 6:26 AM

Post #6 of 17 (3625 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

I have found a way to simulate the problem so it is caused on purpose in order to find a way to solve it and something interesting I have found out is the following:

if you have:

if ($var eq "") {

print "I'm here";

}

if ($var eq "") {

exit;

}

sometimes it still shows "I'm here" and does NOT exit meaning a bit later it remembers the contents of the var !!

So if this is right this one should help:

open(DATA, ">anyfile.txt");

print DATA @array;

close(DATA);

&check;

sub check {

open(DATA, "+<anyfile.txt");

@content=<DATA>;

close(DATA)

if ($content[0] eq "") {

open(DATA, ">anyfile.txt");

print DATA @array;

close(DATA)

}

$a++;

if ($a < 20) {

&check;

}

}

This will try to write the contents to the file 20 times again and again as long as the file has a content.

Another way is using temporary files i.e.:

1) Write the files content to a temp file

2) Have the script check if the temp file is empty

3) if it is not empty overwrite the original file with the temp file. This appears to be a secure possibility

Of course in step 3 you would not use Perl to overwrite the file but have Perl command the shell to do this like:

open (SHELL, "|blabla"):

close(SHELL);

This would help but on large apllication as the one I am using it for it would cause a lot of disk usage and maybe reduce the lifetime of the server's harddisk...

(This post was edited by sebastian on May 21, 2002, 6:30 AM)

yapp
User

May 23, 2002, 12:42 PM

Post #7 of 17 (3615 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

To make more robust programs, that don't have these problems, you can use the following tips and tricks:

- be strict:

Code

use strict;

This disallows to make magical (self appearing) variables. You need to declare a variable with my() first, and all decrations only last until the end of a { } block.

- Declare all variables in the block you use them with my()

Code

my $x = 'value'

- Use the -w switch. (disallows the use of undefined values...), for example:

Code

#!/usr/bin/perl -w 
 
my $something = function_returning_undef();   
print "Something is $something\n";

Usually, undef is an indication of failure, or your logic failed. You can always check for undef with the defined() function.

Code

print "Something is $something\n" if defined $something;   
print "Something is " . ($something || "false") . "\n";

- Use locking using Fcntl constants

Code

use Fcntl qw(:flock);   
flock FILEHANDLE, LOCK_SH; # Other processes can only read (but pause when they want to edit)   
flock FILEHANDLE, LOCK_EX; # Every process pauses until you're done editing the file.

- Seek to the EOF when you're appending data... If you're waiting until someones lock is freed, the first might have grown in size.
Put this after your flock(..LOCK_EX) line:

Code

seek(FILEHANDLE, 0, 2) or die "Can't seek EOF: $!";

- Always check for the return type of open.
Either use a if, or throw the error with a open(..) or die "Error: $!"; contruction.

- When you want to update, don't read the file first, and re-open it to write data. Open it with RD_WR access. You can use an extra file as buffer space if you don't want to slurp everything into the memory.

Code

use Fcntl qw(:DEFAULT :flock);   
sysopen(FH, $file, O_CREAT|O_RDWR) or die "Can't rdwr: $!";   
flock(FH, LOCK_EX);   
my @all_data = <FH>; # Slurp into memory.   
# edit @all_data...   
  
seek(FH, 0, 0) or die "Can't seek begin: $!";   
truncate(FH, 0) or die "Can't erase contents: $!";   
print FH @all_data;   
clsoe(FH);

- Use chomp to remove the \n, and not chop.

anything more?

Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

(This post was edited by yapp on May 23, 2002, 1:01 PM)

gregarios
stranger

May 23, 2002, 1:34 PM

Post #8 of 17 (3609 views)

Re: [yapp] Database Unexpected Erasure Problem [In reply to]	Can't Post

Thanks... Ok, then. Judging from my original code seen here:

Code

#!/usr/bin/perl5 -wT  
use strict;  
                                                
$| = 1;  
if (!open (ADDCTX, "<ads.dta")) {die "Could not open file: $!";}  
flock(ADDCTX,1) or die "Could not lock file: $!";  
seek(ADDCTX,0,0);  
my @ads = <ADDCTX>;  
close(ADDCTX);  
my @TLAD = split(/\|/,$ads[rand @ads]);  
$TLAD[2]++;  
$ads[$TLAD[0]] = "$TLAD[0]|$TLAD[1]|$TLAD[2]\n";  
print "Content-type: text/html\n\n";  
print "Advertisement: $TLAD[1]";  
if (open (ADDCTZ, "+<ads.dta")) {  
flock(ADDCTZ,2) or die "Could not lock file: $!";  
truncate ADDCTZ,0;  
seek(ADDCTZ,0,0);  
print ADDCTZ @ads;  
close(ADDCTZ);  
}  
exit;

I've apparently done only a few things different:
1) Open a file only once in a given cgi when you wish to read, edit, then write back to it.
2) Use "use Fcntl qw(:flock);" in addition to the actual flock command and "LOCK_EX" style instead of the numeric equivelent.
3) Seek to the EOF when appending, which I am not. The file is a fixed number of lines and it stays that way.
4) Use "sysopen" instead of just the "open" command.

Could you please tell me why each of these things makes a difference? I've really been trying to find out the details on this for some time but cannot find information about the "why's and wherefore's" of this stuff. Perldocs are only so good at telling you how to use certain things, but not so good as to the why's or when's. :-)

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

(This post was edited by gregarios on May 23, 2002, 1:37 PM)

yapp
User

May 25, 2002, 1:23 AM

Post #9 of 17 (3603 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

Well I think this does the trick:

Code

#!/usr/bin/perl5 -wT 
 
use strict;   
use Fcntl qw(:flock :DEFAULT); 
                                                 
$| = 1;   
 
sysopen(ADDCTX, "ads.dta", O_CREAT|O_RDWR) or  die "Could not open file: $!"; 
 
flock(ADDCTX, LOCK_EX);  # no die: *1. 
 
my @ads = <ADDCTX>;   
 
my @TLAD = split(/\|/,$ads[rand @ads]);   
$TLAD[2]++;   
$ads[$TLAD[0]] = "$TLAD[0]|$TLAD[1]|$TLAD[2]\n";   
 
print "Content-type: text/html\n\n";   
print "Advertisement: $TLAD[1]";   
 
seek(ADDCTZ,0,0) or die "Can't seek begin: $!";   # Seek before truncate 
truncate(ADDCTZ,0) or die "Can't truncate: $!"; 
print ADDCTZ @ads;   
close(ADDCTZ);   
 
exit;

*1: I never used or die here, because windows95/98 don't support flock. However, files at their filesystem can only be opened once...So windows always has that lock. See what you do with this fact...

Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

gregarios
stranger

May 28, 2002, 4:33 PM

Post #10 of 17 (3591 views)

Re: [yapp] Database Unexpected Erasure Problem [In reply to]	Can't Post

Would it be better, and would it work, to just do this then?

Code

sysopen(ADDCTX, "ads.dta", O_CREAT|O_RDWR|O_EXLOCK) or die $!";

...and just forget about doing the flock call completely?
also... how would I test this to see if it is in fact locking correctly?

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

yapp
User

May 29, 2002, 12:52 AM

Post #11 of 17 (3588 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

I'm not sure, but I thought that wasn't locking.

If i'm not mistaken, that action only tests if there is a lock, and doesn't open the file if it's already locked (or die returns an error)

Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

gregarios
stranger

May 29, 2002, 8:49 AM

Post #12 of 17 (3585 views)

Re: [yapp] Database Unexpected Erasure Problem [In reply to]	Can't Post

Here is a direct quote from my FreeBSD man pages... "man open" specifically. Please let me know if it appears to you as it appears to me:

Quote

DESCRIPTION (open)
The file name specified by path is opened for reading and/or writing as specified by the argument flags and the file descriptor returned to the calling process. The flags argument may indicate the file is to be created if it does not exist (by specifying the O_CREAT flag). In this case open requires a third argument mode_t mode, and the file is created with mode mode as described in chmod(2) and modified by the process' umask value (see umask(2)).

The flags specified are formed by or'ing the following values

O_RDONLY open for reading only
O_WRONLY open for writing only
O_RDWR open for reading and writing
O_NONBLOCK do not block on open
O_APPEND append on each write
O_CREAT create file if it does not exist
O_TRUNC truncate size to 0
O_EXCL error if create and file exists
O_SHLOCK atomically obtain a shared lock <----
O_EXLOCK atomically obtain an exclusive lock <----

Opening a file with O_APPEND set causes each write on the file to be appended to the end. If O_TRUNC is specified and the file exists, the file is truncated to zero length. If O_EXCL is set with O_CREAT and the file already exists, open() returns an error. This may be used to implement a simple exclusive access locking mechanism. If O_EXCL is set and the last component of the pathname is a symbolic link, open() will fail even if the symbolic link points to a non-existent name. If the O_NONBLOCK flag is specified and the open() call would result in the process being blocked for some reason (e.g., waiting for carrier on a dialup line), open() returns immediately. The first time the process attempts to perform I/O on the open file it will block (not currently implemented).

When opening a file, a lock with flock(2) semantics can be obtained by setting O_SHLOCK for a shared lock, or O_EXLOCK for an exclusive lock. If creating a file with O_CREAT, the request for the lock will never fail (provided that the underlying filesystem supports locking).

Am I imagining this? I've never seen anything else about it except for the mention of the option with no explanation in the Perl Docs. Maybe it's not mentioned much because it may not be portable or something, but I don't care about portability. I'll never use a Windows based system anyway. It appears you can open a file with flocking enabled with just that one command! What a cool thing that would be! Let me know what you think. :-)

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

gregarios
stranger

May 29, 2002, 2:39 PM

Post #13 of 17 (3582 views)

Re: [yapp] Database Unexpected Erasure Problem [In reply to]	Can't Post

Ok... I made a small program that I'm hoping tested my theory correctly.

Here is my test program:

Code

#!/usr/bin/perl5 -wT  
use strict; 
use Fcntl qw(:DEFAULT :flock); 
$| = 1; 
 
sysopen(ADDCTX, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!"; 
sysopen(ADTEST, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!"; 
 
truncate(ADTEST,0) or die "Could not truncate file: $!"; 
close(ADTEST) or die "Could not close file: $!"; 
my @ads = <ADDCTX>; 
if ($ads[0]) {print "Ad list: @ads"} else {print "Truncated File"}; 
close(ADDCTX); 
exit;

Here are the results. I'll show you the changes to the two lines of code that matter and give you a description of the outcome.

These two examples of code caused an endless wait period... presumably on the "O_EXLOCK" lock. Interestingly enough, in the second set, the second sysopen command shown here wouldn't apply an exclusive lock while the file was previously open, even though the first open did not apply a lock. Niether ever truncated:

Code

sysopen(ADDCTX, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!"; 
sysopen(ADTEST, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!";

Code

sysopen(ADDCTX, "ads.dta", O_RDWR) or die "Could not open and lock file: $!"; 
sysopen(ADTEST, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!";

These two examples of code caused the file to be truncated:

Code

sysopen(ADDCTX, "ads.dta", O_RDWR) or die "Could not open and lock file: $!"; 
sysopen(ADTEST, "ads.dta", O_RDWR) or die "Could not open and lock file: $!";

Code

sysopen(ADDCTX, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!"; 
sysopen(ADTEST, "ads.dta", O_RDWR) or die "Could not open and lock file: $!";

I would be grateful to hear what you have to say about this. Judging from my examples, I would think this is a great way to lock files safely... at least when running in FreeBSD?

Code

sysopen(ADDCTX, "ads.dta", O_RDWR|O_EXLOCK) or die "Could not open and lock file: $!";

Anyone... please give me your opinions on this. Thank you for your support. :-)

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

(This post was edited by gregarios on May 29, 2002, 2:41 PM)

gregarios
stranger

May 29, 2002, 8:29 PM

Post #14 of 17 (3576 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

As a followup, I belive I have verified that the O_EXLOCK command is Flocking the file while opening it. I added a 9 second sleep between the sysopen and the file close on my original program, with the sysopen modification in place. If I attempt to print the database with two browsers on two machines at exactly the same time, one prints out in 9 seconds and the other machine prints it 9 seconds later than the first machine, proving that the sysopen command with the O_EXLOCK is in fact locking the file, and other occurances of the script are waiting for the lock to close before opening the file. :-)

Any comments from experienced users on this subject would be appreciated. Thanks again.

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

yapp
User

May 30, 2002, 10:44 AM

Post #15 of 17 (3570 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

WHoa! good post man! And, every well explained with an experiement.

I'd say, enjoy this feature if you can Smile

Dothing two IO actions with one command prevents the race conditions everyone experiences when forgetting the locking.

But, why isn't this documentated in the perldoc? Isn't it portanble (if anyone knows...) Anyway, the name says enough.. sysopen is your direct open command (and more related to the system, unlike the perl open command)

Yet Another Perl Programmer

_________________________________
~~> [url=http://www.codingdomain.com]www.codingdomain.com <~~
More then 3500 X-Forum [url=http://www.codingdomain.com/cgi-perl/downloads/x-forum]Downloads! Cool

gregarios
stranger

May 30, 2002, 11:48 AM

Post #16 of 17 (3567 views)

Re: [yapp] Database Unexpected Erasure Problem [In reply to]	Can't Post

Post-followup followup:
I've revamped my entire site now with the new sysopen with O_xxLOCK command... Everything is working smoothly... even somewhat faster I think! :-) One more experiment clinched it for me. I tried a mix of sysopen/O_xxLOCK scripts and regular open/flock scripts on the same database and they do recognize each other's locking. So if you have a few sysopen's and a few open/flock's working on the same database, they will play well together. :-)

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

gregarios
stranger

Jun 2, 2002, 12:05 AM

Post #17 of 17 (3559 views)

Re: [gregarios] Database Unexpected Erasure Problem [In reply to]	Can't Post

Ok... I'm dumbfounded now. The locking IS WORKING CORRECTLY. However, the database still got emptied somehow after nearly 1000 successful hits on it. I tried all the methods reported in this post, and none seems to have worked, even though I did find a better locking method.
Could this be some kind of server problem? Frown

• Greg J Piper
• [url=http://www.macpicks.com]MacPiCkS

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives