Perl Guru Forums: Perl Programming Help: Beginner: Data processing from text file

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Beginner:
Data processing from text file

malalo
Novice

Jul 29, 2014, 12:08 PM

Post #1 of 9 (809 views)

Data processing from text file	Can't Post

Hello !
I've had some experience with Perl a long time ago and I thought I'd get back to it a little.
I have an output file (txt) containing the read and write access for different users on our network. The file looks as follows :
DirectoryName (tab) "User1, User2, User4" (tab) "User1, User2, User3"

From this file, I'd like to output to a new file :
DirectoryName (tab) "User4" (tab) "User1, User2, User3"

Meaning : Remove all users in the middle field who are also in the last field.

I know I'd have to probably start with a "ForEach" and stuff all those lines inside arrays. However I'm a bit stumped on how to insert them in different arrays according to the TAB delimiter... and also how to remove data from the middle field according to what's in the last field.

Any help greatly appreciated, I'd prefer some examples to get me thinking (and learning !)

malalo
Novice

Jul 29, 2014, 1:28 PM

Post #2 of 9 (800 views)

Re: [malalo] Data processing from text file [In reply to]	Can't Post

Well I've tried starting and I'm already having an issue.

This is the code I've written so far.

Code

#!/usr/local/bin/perl  
use strict; 
  
my $file = ".\\AccessEnum.txt"; 
open (DATA,"<$file") or die "Can't open $file: $!"; 
while(<DATA>){ 
	print "$_"; 
} 
close(DATA) or die "Cannot close $file: $!";

Just to see what the output was.
If the "AccessEnum.txt" file has:
This is a test
the output of the script looks like:
T h i s i s a t e s t
and I don't understand why.

Quote

I found out why. The txt was saved as Unicode, I re-saved it as ANSI and now the output is ok.

Also there seems to be a limitation to the size of variables as the lines in the txt file are very long (over 1024 characters).

(This post was edited by malalo on Jul 29, 2014, 2:03 PM)

FishMonger
Veteran / Moderator

Jul 29, 2014, 2:05 PM

Post #3 of 9 (788 views)

Re: [malalo] Data processing from text file [In reply to]	Can't Post

Perl does not have a built-in limit on the size of the vars. That's dependent on the amount of memory and disk space.

The script you posted will not produce the output you claim. You must have ran a different version of the script.

malalo
Novice

Jul 29, 2014, 2:16 PM

Post #4 of 9 (784 views)

Re: [FishMonger] Data processing from text file [In reply to]	Can't Post

Quote

The script you posted will not produce the output you claim.

Perhaps I copied from a previous save, sorry about that.
Here's the latest script I have:

Code

#!/usr/local/bin/perl  
use strict; 
  
my $file = ".\\AccessEnum - Copy.txt"; 
open(DATA,"<$file") or die "Can't open $file: $!"; 
 
while(<DATA>){ 
	my ($rep, $read, $write, $deny) = split /\t/, "$_"; 
	print "$write\n"; 
} 
close(DATA) or die "Cannot close $file: $!";

So I'm splitting the lines from the text file into 4 variables using the TAB delimiter. That works, I've tried printing the $rep variable, then $read, then (as seen in the code above) $write.

The $read and $write variables look like this:

Code

"Administrators, DOMAIN\User1, DOMAIN\User2, NT AUTHORITY\Authenticated Users"

They do NOT have the same amount of users from line to line.

My goal is to copy everything into a new text file while deleting any user from $read who also appears in $write, on the same line.
I think I'd need to split the $read and $write into arrays and check if any user from $write appears in $read, delete it from $read.
The method on how to do this eludes me for now.

BillKSmith
Veteran

Jul 29, 2014, 4:20 PM

Post #5 of 9 (777 views)

Re: [malalo] Data processing from text file [In reply to]	Can't Post

Read the FAQ. From your command line, type:

Code

perldoc -q "difference of two arrays"

The example is not exactly what you need, but it should get you started.

I can suggest a few improvements to your code.

Your filename suggests that you are using windows. In that case, your first line is unnecessary and perhaps misleading because the path is not a windows path.

Strict is very good. Also use warnings;

It is not necessary to specify the current directory with a filename.

I recommend against using whitespace in a filename. It often complicates quoting problems.

You should not override perl's builtin filehandle DATA. In fact, you should use lexical filehandles.

Always use the three argument form of open.

Use an explicit lexical variable rather than $_.

Chomp the newline off your input before splitting. This will probably simplify your matching.
Good Luck,
Bill

Chris Charley
User

Jul 29, 2014, 4:44 PM

Post #6 of 9 (774 views)

Re: [malalo] Data processing from text file [In reply to]	Can't Post

Picking up some of BillKSmith's suggestions, (use warnings, 3 argument open), this script should do what you want.

Code

#!/usr/bin/perl 
use strict; 
use warnings; 
 
my $file = "test.txt"; # test.txt in current directory 
 
# create filehandle 
open my $fh, '<', $file or die "Unable to open $file $!"; 
 
while (<$fh>) { 
	chomp; 
	tr/"//d; # eliminate the quotation marks 
	my ($dir, $read, $write) = split /\t/; 
	 
	my %seen = map {$_ => 1} split /, /, $write; 
	$read = join ", ", grep !$seen{$_}, split /, /, $read; 
	 
	# restore the quotation marks 
	$_ = qq{"$_"} for $read, $write; 
	 
	print join("\t", $dir, $read, $write), "\n"; 
} 
close $fh or die "Unable to close $file $!"; 
 
__END__ 
*** contents of test.txt 
 
DirectoryName	"Administrators, DOMAIN\User1, DOMAIN\User2, NT AUTHORITY\Authenticated Users"	"Administrators, DOMAIN\User1, DOMAIN\User2"

(I'm not sure about the #!/usr/bin/perl line)

This prints:

Code

DirectoryName   "NT AUTHORITY\Authenticated Users"      "Administrators, DOMAIN\User1, DOMAIN\User2"

malalo
Novice

Jul 30, 2014, 6:17 AM

Post #7 of 9 (759 views)

Re: [Chris Charley] Data processing from text file [In reply to]	Can't Post

Thank you !

This script works very well. I simply added a few lines so I could get the output in a new file. Here's the complete code:

Code

#!/usr/local/bin/perl  
use strict; 
use warnings; 
  
my $file = "AccessEnumCopy.txt"; 
my $file2 = "AccessEnumNew.txt"; 
 
open my $fh, '<', $file or die "Can't open $file: $!"; 
 
while (<$fh>){ 
	chomp; 
	tr/"//d; 
	my ($rep, $read, $write, $deny) = split /\t/; 
 
	my %seen = map {$_ => 1} split /, /, $write; 
	$read = join ", ", grep !$seen{$_}, split /, /, $read; 
 
	$_ = qq{"$_"} for $read, $write; 
 
	open my $fh2, '>>', $file2 or die "Can't open $file2: $!"; 
 
	print $fh2 join("\t", $rep, $read, $write, $deny), "\n"; 
} 
close $fh or die "Cannot close $file: $!";

Now I'm off to study the commands such as "qq", "tr", "map" and "chomp" which I'm not too familiar with.

Thank you both for your help !

Chris Charley
User

Jul 30, 2014, 8:51 AM

Post #8 of 9 (750 views)

Re: [malalo] Data processing from text file [In reply to]	Can't Post

qq, tr - http://perldoc.perl.org/perlop.html#Quote-Like-Operators

map - http://perldoc.perl.org/functions/map.html

chomp - http://perldoc.perl.org/functions/chomp.html

Instead of opening your output file for every iteration of your while loop, you could open it just once before entering the while loop.

open my $fh2, '>', $file2 or die "Can't open $file2: $!";
while (<$fh>) {
....
....
....
print $fh2 join("\t", $rep, $read, $write, $deny), "\n";
}

(This post was edited by Chris Charley on Jul 30, 2014, 8:55 AM)

malalo
Novice

Jul 30, 2014, 9:57 AM

Post #9 of 9 (744 views)

Re: [Chris Charley] Data processing from text file [In reply to]	Can't Post

Thanks Chris !

I'll change it so that it'll open the file before the while loop. Also I noticed I forgot to add the command to close the $fh2.

Yup:

Code

#!/usr/local/bin/perl  
use strict; 
use warnings; 
  
my $file = "AccessEnum.txt"; 
my $file2 = "AccessEnumNew.txt"; 
 
open my $fh, '<', $file or die "Can't open $file: $!"; 
open my $fh2, '>>', $file2 or die "Can't open $file2: $!"; 
 
while (<$fh>){ 
	chomp; 
	tr/"//d; 
	my ($rep, $read, $write, $deny) = split /\t/; 
 
	my %seen = map {$_ => 1} split /, /, $write; 
	$read = join ", ", grep !$seen{$_}, split /, /, $read; 
 
	$_ = qq{"$_"} for $read, $write; 
 
	open my $fh2, '>>', $file2 or die "Can't open $file2: $!"; 
 
	print $fh2 join("\t", $rep, $read, $write, $deny), "\n"; 
} 
close $fh2 or die "Cannot close $file2: $!"; 
close $fh or die "Cannot close $file: $!";

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives