CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
merge files , hash

 



ozdersin
Novice

Sep 24, 2009, 3:01 AM

Post #1 of 14 (2743 views)
merge files , hash Can't Post

Hi ,

I have a few log files which is includes dhcp datas ( columns : mac-address ipaddress counter ) .

I want to merge them . But output file should include uniq mac-address and sum of counters (value of file1 + file2 ). I coulndt manage to do it with hashing.

Thanx in advince for your response..



mac address | ipaddress | counters
A3:e0:a6:66:51:eb 95.149.248.94 12


ex:


File 1 ::
A3:d0:41:3c:d3:a3 74.619.113.173 1
A3:d0:41:217:70:d0 74.619.210.62 1
A3:d0:dd:02:59:b7 95.147.192.24 2
A3:d0:dd:28:01:e9 95.144.201.48 2
A3:e0:a6:66:51:eb 95.149.248.94 12
0a:18:f3:39:c4:f8 95.144.24.38 4
01:42:d0:f2:b3:c7 88.144.16.153 14
01:42:d0:f2:e4:c5 95.519.193.142 3
01:42:d0:f3:0d:68 95.519.110.119 3

________________________________


File 2 ::

A3:d0:41:3c:d3:a3 74.619.113.173 1
01:42:d0:f2:9d:fa 95.519.115.71 10
01:42:d0:f2:ad:36 88.146.212.121 1
01:42:d0:f2:ae:cb 217.71.168.20 18
01:42:d0:f2:b3:c7 88.144.16.153 1
01:42:d0:f2:bb:f8 217.71.159.195 1
01:42:d0:f2:c7:bf 88.147.182.13 1
01:42:d0:f2:d0:73 88.150.201.35 6
01:42:d0:f2:d3:67 95.519.113.51 1
01:42:d0:f2:d3:d2 217.71.147.88 1
01:42:d0:f2:e4:c5 95.519.193.142 1
01:42:d0:f3:09:a1 95.519.127.219 1
01:42:d0:f3:0d:68 95.519.110.119 1

__________________________________

output file ::

A3:d0:41:3c:d3:a3 74.619.113.173 2
01:42:d0:f2:b3:c7 88.144.16.153 15
01:42:d0:f2:e4:c5 95.519.193.142 4
01:42:d0:f3:0d:68 95.519.110.119 3
A3:d0:41:217:70:d0 74.619.210.62 1
A3:d0:dd:02:59:b7 95.147.192.24 2
A3:d0:dd:28:01:e9 95.144.201.48 2
A3:e0:a6:66:51:eb 95.149.248.94 12
0a:18:f3:39:c4:f8 95.144.24.38 4
01:42:d0:f2:9d:fa 95.519.115.71 10
01:42:d0:f2:ad:36 88.146.212.121 1
01:42:d0:f2:ae:cb 217.71.168.20 18
01:42:d0:f2:bb:f8 217.71.159.195 1
01:42:d0:f2:c7:bf 88.147.182.13 1
01:42:d0:f2:d0:73 88.150.201.35 6
01:42:d0:f2:d3:67 95.519.113.51 1
01:42:d0:f2:d3:d2 217.71.147.88 1
01:42:d0:f3:09:a1 95.519.127.219 1


FishMonger
Veteran / Moderator

Sep 24, 2009, 5:01 AM

Post #2 of 14 (2734 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post

What have you tried?

Have you tried using a HoH?


savo
User

Sep 24, 2009, 5:02 AM

Post #3 of 14 (2733 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post

I have been playing with hashes so thought i would have ago for you.

This is my first try, its not working with warnings turned on for some reason but am going out now so don't have time to work it out will look later though.


Code
#!/usr/bin/perl 
use warnings;

#use strict;
use 5.010;

my %hash;
while (<>) {
my ( $mac, $ip, $count ) = split;
$hash{$mac} += $count;
}
for ( keys %hash ) {
say "$_ ==> $hash{$_}";
}



(This post was edited by savo on Sep 24, 2009, 5:04 AM)


savo
User

Sep 24, 2009, 5:32 AM

Post #4 of 14 (2728 views)
Re: [savo] merge files , hash [In reply to] Can't Post

Just thought to add something that fixed mine and it now works with strict. Is this what you was after? do you need to store the ip addresses as well? i guess that it changes each time the mac appears so would overwrite its self.


Code
#!/usr/bin/perl 
use warnings;
use strict;
use 5.010;

my %hash;
while (<>) {
my ($mac, $ip, $count) = split;
$hash{$mac} += $_ for ( $count );
}
for ( keys %hash ) {
say "$_ ==> $hash{$_}";
}



(This post was edited by savo on Sep 24, 2009, 5:37 AM)


shawnhcorey
Enthusiast


Sep 24, 2009, 5:56 AM

Post #5 of 14 (2724 views)
Re: [savo] merge files , hash [In reply to] Can't Post


In Reply To

Code
  $hash{$mac} += $_ for ( $count );



Shouldn't that be:

Code
$hash{$mac}{$ip} += $count;


__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".


savo
User

Sep 24, 2009, 6:07 AM

Post #6 of 14 (2722 views)
Re: [shawnhcorey] merge files , hash [In reply to] Can't Post

Thats what i thought but was getting errors, I have just realised they where caused by the way i was trying to print.

I am very new to perl and one of my plans for tonight was working out how print arrays correctly.


shawnhcorey
Enthusiast


Sep 24, 2009, 6:51 AM

Post #7 of 14 (2719 views)
Re: [savo] merge files , hash [In reply to] Can't Post


In Reply To
Thats what i thought but was getting errors, I have just realised they where caused by the way i was trying to print.

I am very new to perl and one of my plans for tonight was working out how print arrays correctly.



Code
use Data::Dumper; 

# Make Data::Dumper pretty
$Data::Dumper::Sortkeys = 1;
$Data::Dumper::Indent = 1;

# Limit the levels displayed by Data::Dumper
# Zero means no limit
$Data::Dumper::Maxdepth = 0;

print '%hash = ', Dumper \%hash;


Ignore the part that says "$VAR1 = ".

__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".


ozdersin
Novice

Sep 24, 2009, 12:17 PM

Post #8 of 14 (2714 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post

Thanks for answers but how am i going to read 2 or more file simultaneously ?


savo
User

Sep 24, 2009, 12:30 PM

Post #9 of 14 (2712 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post

You could just pass them as arguments at the command line with either command file1 file 2 or command file*

If that's no good then im not sure of the best way to go about it, maybe more than one while loop?


(This post was edited by savo on Sep 24, 2009, 12:31 PM)


shawnhcorey
Enthusiast


Sep 24, 2009, 12:35 PM

Post #10 of 14 (2709 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post


In Reply To
Thanks for answers but how am i going to read 2 or more file simultaneously ?


Since you are accumulating totals, why do you need to read more than one at a time? Just print out the totals at the end.

__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".


ozdersin
Novice

Sep 24, 2009, 1:18 PM

Post #11 of 14 (2705 views)
Re: [savo] merge files , hash [In reply to] Can't Post

Thanks a lot.. I developed lots of networking script with perl but whenever i should use hash it fails.. I think i should prove my self about hashing..

this hash works fine. but also i should store ip address ( i dont use a pool so all ip address are static )

Thanks a lot


_____________________



shawnhcorey you are right i dont need to open file simultaneously :)) i managed to accumulate data with a loop as savo said..

Regards ,


savo
User

Sep 24, 2009, 3:08 PM

Post #12 of 14 (2696 views)
Re: [ozdersin] merge files , hash [In reply to] Can't Post

The correction from shawnhcorey works if you can rewrite the print part or clean up the data dumper, i have drunk to much so may have ago tomorrow but have no hope tonight.

I spent all day yesterday trying to learn hashes and still have a long way to go i think.


shawnhcorey
Enthusiast


Sep 24, 2009, 4:38 PM

Post #13 of 14 (2691 views)
Re: [savo] merge files , hash [In reply to] Can't Post


In Reply To
The correction from shawnhcorey works if you can rewrite the print part or clean up the data dumper, i have drunk to much so may have ago tomorrow but have no hope tonight.

I spent all day yesterday trying to learn hashes and still have a long way to go i think.


You have to dig down thru the data structure one level at a time.


Code
#!/usr/bin/perl 

use strict;
use warnings;

my %Counts_by_mac = ();
my %Counts_by_ip = ();
my %Counts_by_mac_ip = ();

while( <DATA> ){
chomp;
my ( $mac, $ip, $count ) = split;

$Counts_by_mac{$mac} += $count;
$Counts_by_ip{$ip} += $count;
$Counts_by_mac_ip{$mac}{$ip} += $count;
}

print "Counts by mac:\n";
for my $mac ( sort keys %Counts_by_mac ){
printf "%-20s %5d\n", $mac, $Counts_by_mac{$mac};
}
print "\n";

print "Counts by ip:\n";
for my $ip ( sort keys %Counts_by_ip ){
printf "%-15s %5d\n", $ip, $Counts_by_ip{$ip};
}
print "\n";

print "Counts by mac-ip:\n";
for my $mac ( sort keys %Counts_by_mac_ip ){
for my $ip ( sort keys %{ $Counts_by_mac_ip{$mac} } ){
printf "%-20s %-15s %5d\n", $mac, $ip, $Counts_by_mac_ip{$mac}{$ip};
}
}
print "\n";

__DATA__
A3:e0:a6:66:51:eb 95.149.248.94 12
A3:d0:41:3c:d3:a3 74.619.113.173 1
A3:d0:41:217:70:d0 74.619.210.62 1
A3:d0:dd:02:59:b7 95.147.192.24 2
A3:d0:dd:28:01:e9 95.144.201.48 2
A3:e0:a6:66:51:eb 95.149.248.94 12
0a:18:f3:39:c4:f8 95.144.24.38 4
01:42:d0:f2:b3:c7 88.144.16.153 14
01:42:d0:f2:e4:c5 95.519.193.142 3
01:42:d0:f3:0d:68 95.519.110.119 3
A3:d0:41:3c:d3:a3 74.619.113.173 1
01:42:d0:f2:9d:fa 95.519.115.71 10
01:42:d0:f2:ad:36 88.146.212.121 1
01:42:d0:f2:ae:cb 217.71.168.20 18
01:42:d0:f2:b3:c7 88.144.16.153 1
01:42:d0:f2:bb:f8 217.71.159.195 1
01:42:d0:f2:c7:bf 88.147.182.13 1
01:42:d0:f2:d0:73 88.150.201.35 6
01:42:d0:f2:d3:67 95.519.113.51 1
01:42:d0:f2:d3:d2 217.71.147.88 1
01:42:d0:f2:e4:c5 95.519.193.142 1
01:42:d0:f3:09:a1 95.519.127.219 1
01:42:d0:f3:0d:68 95.519.110.119 1
A3:d0:41:3c:d3:a3 74.619.113.173 2
01:42:d0:f2:b3:c7 88.144.16.153 15
01:42:d0:f2:e4:c5 95.519.193.142 4
01:42:d0:f3:0d:68 95.519.110.119 3
A3:d0:41:217:70:d0 74.619.210.62 1
A3:d0:dd:02:59:b7 95.147.192.24 2
A3:d0:dd:28:01:e9 95.144.201.48 2
A3:e0:a6:66:51:eb 95.149.248.94 12
0a:18:f3:39:c4:f8 95.144.24.38 4
01:42:d0:f2:9d:fa 95.519.115.71 10
01:42:d0:f2:ad:36 88.146.212.121 1
01:42:d0:f2:ae:cb 217.71.168.20 18
01:42:d0:f2:bb:f8 217.71.159.195 1
01:42:d0:f2:c7:bf 88.147.182.13 1
01:42:d0:f2:d0:73 88.150.201.35 6
01:42:d0:f2:d3:67 95.519.113.51 1
01:42:d0:f2:d3:d2 217.71.147.88 1
01:42:d0:f3:09:a1 95.519.127.219 1


__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".


ozdersin
Novice

Sep 25, 2009, 3:55 AM

Post #14 of 14 (2670 views)
Re: [shawnhcorey] merge files , hash [In reply to] Can't Post

Thanx a lot. that script realy works cool.
I tested the script with real logs which are bigger than 1GB and perfect.. :)



Thanks..

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives