Perl Guru Forums: Perl Programming Help: Intermediate: Perl Script Causing High CPU Usage

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
Perl Script Causing High CPU Usage

smart_amorist
Novice

Sep 19, 2013, 6:35 PM

Post #1 of 8 (5339 views)

Perl Script Causing High CPU Usage	Can't Post

Perl Script Causing High CPU Usage

We have an application that generates huge number of files (3 - 4 lakh including subdirectories) under multiple directories. In order to archive, move or delete these files based on files age, I have written a simple perl script but sometime it is causing high CPU spikes(%CPU in Linux top outpu 60-100%). Just spikes it never continuously use high CPU.

Example:

/app/home/DIR_1/ABC_DEF.xml
/app/home/DIR_2/GHI_JKL.xml
/app/home/DIR_3/MNO_PQR.xml

or subdirectories

/app/home/DIR_1/SubDir_YYYYMMDD

$PatternToMatch ="???_???.xml";

or

$PatternToMatch = "SubDir_YYYYMMDD";

Now script check for each directory in loop and read matching files as below.

foreach(@dir_list){

chdir($_);

while(glob($PatternToMatch)) {
$age = -M $_;
if($age > 7) {
unlink $_;
or
qx(mv source destination);
or
gzip $_;
}
}

}

Noticed that when script is switching from one directory to other, it is causing CPU spikes. Strange thing is if are more files matched, then there is no high CPU spikes (10-15%), but if it only found around 30-60 files in each dir, i can see spikes. I have put sleep after each dir to bring down cpu.

What is the best way to match files based on age?
Any idea what is causing high CPU spikes.
%CPU shows high value sometime but at same time overall CPU usage given by top, mpstat shows hardly from 4-5%, this is confusing me.

If anyone has any best idea/script to achieve it please help me.

(This post was edited by smart_amorist on Sep 19, 2013, 6:38 PM)

FishMonger
Veteran / Moderator

Sep 19, 2013, 8:16 PM

Post #2 of 8 (5329 views)

Re: [smart_amorist] Perl Script Causing High CPU Usage [In reply to]	Can't Post

It's hard to say what's causing your high CPU usage based on your incoherent and incomplete code snippet.

You could start by profiling your script.

http://search.cpan.org/~timb/Devel-NYTProf-5.06/lib/Devel/NYTProf.pm

smart_amorist
Novice

Sep 29, 2013, 5:23 AM

Post #3 of 8 (5296 views)

Re: [FishMonger] Perl Script Causing High CPU Usage [In reply to]	Can't Post

Hi FishMonger,

Thanks for the reply and apologies for delay response.

I was trying to find out what is causing high CPU. In a directory there are 2-3 lac files. So below while loop is resulting in high CPU usage.

Now, suppose there are 2 lac files in a directory matching pattern but out of those 2 lac only 10,000 files are 7 days old and I only want to action those files., Remaining files are modified within 7 days and thus below loop go to next section. As while run for long, it is causing high cpu usage.

Please have a look and advise if there is a better way to achieve it.

<code>

while(glob("$MatchPattern")) {
my $ObjectName = $_;
my $age = sprintf("%.6f", -M $ObjectName) if ($ObjectName);
my $size = -s $ObjectName if ($ObjectName);
my $Display = ($DtFmt) ? $DtFmt : "%Y-%m-%d";
my $ObjModTime = strftime("$Display", localtime((stat($ObjectName)->mtime))); ===> just reading date in date format specified in config file

if ($age > 0 && $age > $AgeLimit && $size <= $SizeLimit) {
if (($AutoAction eq 'AutoArchive') && ((-f $ObjectName) || (-d $ObjectName))) {
print "[$ObjModTime] Age & Size = $age > $AgeLimit And $size < $SizeLimit :: Marked to Zip\n";
qx(gzip $ObjectName) if (-f $ObjectName && $Type eq 'Sys_File');
qx(zip -rmT $ObjectName $ObjectName) if (-d $ObjectName && $Type eq 'Sys_Dir' );
$ObjCounter++;

} elsif (($AutoAction eq 'AutoDelete') && (-f $ObjectName)) {
print "Age & Size = $age > $AgeLimit And $size < $SizeLimit :: Marked to Remove \n";
unlink ($ObjectName);
$ObjCounter++;
} elsif (($AutoAction eq 'AutoMove') && $MoveToPath && ($MoveToPath ne 'Nil') && ((-f $ObjectName) || (-d $ObjectName))) {
my $HoldMoveToPath = $MoveToPath;

if (-e $MoveToPath) {
print "[$ObjModTime] Age & Size = $age > $AgeLimit And $size < $SizeLimit :: Marked to Move\n";
qx(mv $ObjectName $MoveToPath/.) if (-f $ObjectName && $SignOff eq 'Yes');
$ObjCounter++;

}

} else {

print "[$ObjModTime] Age & Size = $age > $AgeLimit And $size < $SizeLimit :: Marked No Action => $ActionPath/$ObjectName\n";

next;

}

} else {
next;
}

} #End of while

</code>

(This post was edited by smart_amorist on Sep 30, 2013, 5:36 PM)

FishMonger
Veteran / Moderator

Sep 29, 2013, 7:25 AM

Post #4 of 8 (5294 views)

Re: [smart_amorist] Perl Script Causing High CPU Usage [In reply to]	Can't Post

What are lac files?

Did you use Devel::NYTProf as I suggested to profile your script?

Please post a short but complete script that demonstrates the problem and use the code tags so that the formatting/indentation is retained.

Laurent_R
Veteran / Moderator

Sep 29, 2013, 9:15 AM

Post #5 of 8 (5292 views)

Re: [FishMonger] Perl Script Causing High CPU Usage [In reply to]	Can't Post

In Reply To

I would just ask the same three things.

Just an additional comment: please give an idea of the data volume (number of files, number of megabytes).

Without such information, my first uneducated guess would be that it is not some much your script itself that is taking a lot of CPU usage, but the zip/gzip system commands that you issue in this script.

FishMonger
Veteran / Moderator

Sep 29, 2013, 9:38 AM

Post #6 of 8 (5290 views)

Re: [Laurent_R] Perl Script Causing High CPU Usage [In reply to]	Can't Post

This Q is cross posted on perlmonks.

http://www.perlmonks.org/?node_id=1056216

smart_amorist
Novice

Sep 30, 2013, 6:29 PM

Post #7 of 8 (5274 views)

Re: [FishMonger] Perl Script Causing High CPU Usage [In reply to]	Can't Post

Hi,

Thanks for the update. I could not use module specified because it is not available on system and unfortunately not allowed to install anything new on system.

To be more clear, in a single day system generates 5-6 lakh files in just 3-4 directories consuming around 10-14 GB space per day (individual file size is not huge - 12345 bytes, 5-6 digits number in bytes).

There are files of 4-5 different patterns but count is huge. No date timestamp in file name (e.g ABC_YYYMMDD.log), just random 8 digits number in file names (i.e. ABC_????????.log).

As per my observation, glob is not consuming high CPU. As soon as we are entering while loop, then can see CPU spikes. After commenting out mv, zip, unlink, just keeping print statements, I noticed CPU around 80%.

After adding below in "next" section, CPU is under control 10-12% or even low to 4% sometime.

select (undef, undef, undef, 0.250);

but already process running slow because of huge number of files and cant afford sleep.

FishMonger
Veteran / Moderator

Sep 30, 2013, 9:23 PM

Post #8 of 8 (5270 views)

Re: [smart_amorist] Perl Script Causing High CPU Usage [In reply to]	Can't Post

What version of perl are you using?

As long as you're not using 5.15 or above, you could use Devel::DProf to profile the script.

Devel::DProf is a core module first released with perl v5.6.0 and removed from v5.15.0

http://search.cpan.org/~dom/perl-5.12.5/ext/Devel-DProf/DProf.pm

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives