CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Need a Custom or Prewritten Perl Program?: I need a program that...:
Extract from text file, basic trig functions, then graph

 



marky9074
Novice

Oct 18, 2017, 8:24 AM

Post #1 of 8 (3564 views)
Extract from text file, basic trig functions, then graph Can't Post

Hi there,

Complete noob. Trying to extract some positions from a text file (easting and northing), then do some basic trigonometry. I have multiple issues, the first being the extraction. The below is what I am using for a fixed string:


Code
#!/usr/bin/perl 

my $filename = 'R001.txt';
open (my $fh, '>', $filename) or die "Could not open file '$filename' $!";

while ( <> )
{
if ( /R 1/ )
{
$easting = substr($_,6,8);
$northing = substr($_,14,9);

print $fh "$easting,$northing\n";
}
}
close $fh


Ideally what I would like to do is read an input file with an external lookup search list, and output each to the same file name. In the above example it is looking for R 1 (for some reason this file does not pad with leading zeros), and output to R001.txt

The second thing I want to do is subtract the eastings and northings from each line to get delta eastings & delta northings, and then calculate some other attributes (azimuth etc).

Now, I am not sure if I should first get these into readable files, so that we are not reading the same input over and over again. What I am suggesting, is that maybe the output file should be:

easting (1), northing (1), easting (2), northing (2)

So that the next parse of the file can do the trigonometry in one line step (hope that makes sense)?

I think for now I should just concentrate on getting the output correct, rather than the math, so if anyone can help it would be much appreciated.

I've attached a sample file. Ideally I would like to extract the following:


Code
R   1 
R 564
R 565
R1128
R1129


and so on, but like I say, maybe an external lookup file would be the best option (because then I can just edit it to suit as I need).

Thanks in advance.


(This post was edited by marky9074 on Oct 19, 2017, 1:51 AM)
Attachments: Sample.txt (216 KB)


Chris Charley
User

Oct 18, 2017, 1:01 PM

Post #2 of 8 (3555 views)
Re: [marky9074] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

I can't comment on the graphing you want, but can offer a solution to the data extraction.

This solution captures all the lines beginning with 'R', (noticed in your file that there were lines not beginning with R towards the end of the file - I just exited the file when I encountered them).

I found the amount of memory to store the data and it was 1_897_456 bytes, not too large I think.

I used unpack to extract the data and then ran the program which, at the end, prints out the sample ID's and their east north coordinates.

Code
#!/usr/bin/perl 
use strict;
use warnings;
# use Devel::Size 'total_size';

my $format = "x a4 x a8 a9 x4 a4 x a8 a9 x4 a4 x a8 a9";
# ID E N ID E N ID E N

my @data;

# reads from 'file2'
while (<>) {
last unless /^R/;
my @temp = unpack $format;

for my $i (0 .. $#temp) {
if ($i % 3 == 0) {
$data[ $temp[$i] ] = [ @temp[ $i+1, $i+2 ] ];
}
}
}

printf "%4s %10s %s\n", qw/ID East North/;

for my $id (1, 564, 565, 1128, 1129) {
printf "%4s %10s %s\n", $id, $data[$id][0], $data[$id][1];
}

#print total_size \@data; # prints '1_897_456'


The output from this program was -


Code
 
C:\Old_Data\perlp>perl test2.pl file2
ID East North
1 453448.0 7998331.8
564 447738.7 8002432.1
565 453478.2 7998372.2
1128 447767.3 8002471.8
1129 453507.3 7998412.9

C:\Old_Data\perlp>


Note that the way to access these east north coordinates can be seen in the print routine (at the bottom of the program).


Code
$data[$id][0], $data[$id][1];


I used the modulus (%) operator to determine the index of the id in the '@temp' array in the loop. I also used an array 'slice' to assign the east north values for the id.

'@temp[ $i+1, $i+2 ]'


(This post was edited by Chris Charley on Oct 18, 2017, 3:14 PM)


marky9074
Novice

Oct 19, 2017, 1:46 AM

Post #3 of 8 (3542 views)
Re: [Chris Charley] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

Hi Chris,

Many thanks for the reply. I should have checked the sample file properly before I posted (was just trying to keep it small), and can see it only had one set of data... this format actually repeats over and over again, and the filesize is realistically ~1.5 Gb in total!

I was intending to extract each of the matched records to a unique file (of the same ID) in an attempt to keep the file sizes down when moving on to the math, and then graphing etc.

So for my new sample attached (using my original code - which should have had \n), I would get the below for ID 1:


Code
453448.0,7998331.8 
453454.4,7998327.5
453461.6,7998322.9


Hope that makes sense?


(This post was edited by marky9074 on Oct 19, 2017, 1:57 AM)
Attachments: Sample3.zip (242 KB)


marky9074
Novice

Oct 19, 2017, 2:03 AM

Post #4 of 8 (3537 views)
Re: [marky9074] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

For reference, if I grep these out using:


Code
grep ^"R   1" filename > output


It takes about five seconds, and produces a file around 0.5 Mb, so I think it is dooable to split into separate files, without too much waiting around for the output.


BillKSmith
Veteran

Oct 19, 2017, 1:36 PM

Post #5 of 8 (3531 views)
Re: [marky9074] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

Save the required data in a hash-of-arrays.


Code
C:\Users\Bill\forums\guru>type marky9074.pl 
use strict;
use warnings;
use List::MoreUtils qw( any );
use Data::Dumper;
my @required_ids = ('R 1', 'R 564', 'R 565', 'R1128', 'R1129');
open my $POS, '<', 'sample3.txt';
my %data;
while (<$POS>) {
my ($prefix, $id0, $e0, $n0, $id1, $e1, $n1, $id2, $e2, $n2)
= unpack 'A (A4xA8A9x4)2 A4xA8A9', $_;
$id0 = $prefix.$id0;
if (any {$id0 eq $_} @required_ids) {
push @{$data{$id0}}, [$e0, $n0];
}
$id1 = $prefix.$id1;
if (any {$id1 eq $_} @required_ids) {
push @{$data{$id1}}, [$e1, $n1];
}
$id2 = $prefix.$id2;
if (any {$id2 eq $_} @required_ids) {
push @{$data{$id2}}, [$e2, $n2];
}
}
print Dumper(\%data)


C:\Users\Bill\forums\guru>perl marky9074.pl
$VAR1 = {
'R1129' => [
[
'453507.3',
'7998412.9'
],
[
'453513.7',
'7998408.6'
],
[
'453520.7',
'7998404.1'
]
],
'R1128' => [
[
'447767.3',
'8002471.8'
],
[
'447773.2',
'8002466.7'
],
[
'447779.8',
'8002461.3'
]
],
'R 565' => [
[
'453478.2',
'7998372.2'
],
[
'453484.6',
'7998367.9'
],
[
'453491.7',
'7998363.3'
]
],
'R 1' => [
[
'453448.0',
'7998331.8'
],
[
'453454.4',
'7998327.5'
],
[
'453461.6',
'7998322.9'
]
],
'R 564' => [
[
'447738.7',
'8002432.1'
],
[
'447744.7',
'8002427.0'
],
[
'447751.3',
'8002421.5'
]
]
};

C:\Users\Bill\forums\guru>

Good Luck,
Bill


Chris Charley
User

Oct 31, 2017, 11:49 AM

Post #6 of 8 (3226 views)
Re: [marky9074] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

If you only need to traverse the file once, then Bill's posted solution is probably what you should use. If you need to traverse the file several times, then the best method would be to put all the data into a database.

This would be necessary with a large (1.5 GB) file as all the items in a data structure would most likely exceed your RAM memory.

Loading the database for a file this size would probably take as much time as reading through it once. But then lookups should be relatively speedy.

Below are two scripts. One loads the data into the database and the second retrieves the records. I used the SQLite database as it suffices for this problem. You would need to install DBI and DBD::SQLite. The SQLite database is included when you download DBD::SQLite.

Code
#!/usr/bin/perl 
use strict;
use warnings;
use DBI;

# see guru_fetch.pl

my $dbh = DBI->connect("dbi:SQLite:dbname=guru.lite","","",
{RaiseError => 1, PrintError => 0, AutoCommit => 0}) or die "Can't connect";

2 == @ARGV or die "Usage: perl $0 tablename filename\n";

my $table = shift;

#$dbh->do("DROP TABLE IF EXISTS $table");

$dbh->do(qq{CREATE TABLE $table
(id INTEGER,
east TEXT,
north TEXT)
});


my $sth = $dbh->prepare("INSERT INTO $table VALUES(?,?,?)");

my $format = "x a4 x a8 a9 x4 a4 x a8 a9 x4 a4 x a8 a9";
# ID E N ID E N ID E N

while (<>) {
next unless /^R\s*\d+\s/;
my @temp = unpack $format;

for my $i (0 .. $#temp) {
if ($i % 3 == 0) {
$sth->execute(@temp[$i .. $i+2]);
}
}
}
$dbh->commit;
$dbh->disconnect;



Code
#!/usr/bin/perl 
use strict;
use warnings;
use DBI;
use Math::Trig qw/asin pi/; # import arcsin & pi
use Getopt::Std;

# see guru_create.pl

my %opts;
getopts ('t:i:',\%opts);
2 == keys %opts or die "Usage: perl $0 -t tablename -i 1,564,565 ...\n";

my $table = $opts{t};
my @ids = split /,/, $opts{i};

my $dbh = DBI->connect("dbi:SQLite:dbname=guru.lite","","",
{RaiseError => 1, PrintError => 0}) or die "Can't connect";

my $sth = $dbh->prepare("SELECT north, east FROM $table where id = ?");

for my $id (@ids) {
print "\t$id\n";

$sth->execute($id);
my $aref = $sth->fetchall_arrayref;

for my $i (0 .. $#$aref-1) {
my $n = $aref->[$i+1][0] - $aref->[$i][0];
my $e = $aref->[$i+1][1] - $aref->[$i][1];
print "EAST: $e\n";
print "NORTH: $n\n";

my $dAlong = sqrt($n**2 + $e**2);
print "dAlong: $dAlong\n";

my $area = .5 * abs($n * $e);

# Area = 1/2 * b * h
# h = 2 * Area / b
my $dAcross = 2 * $area / $dAlong;
print "dAcross: $dAcross\n";

# arcsin in degrees
my $arcsin = asin($n/$dAlong) * 180 / pi;

my $azimuth = 90 - $arcsin;
print "azimuth: $azimuth\n";
print $/;
}
}
$dbh->disconnect;


Output from this is:

Code
 
C:\Old_Data\perlp>perl guru_fetch.pl -t Sample3 -i 3,7800
3
EAST: 6.5
NORTH: -4.40000000037253
dAlong: 7.84920378148499
dAcross: 3.64368167760968
azimuth: 124.094977957017

EAST: 7.20000000001164
NORTH: -4.5
dAlong: 8.49058301886082
dAcross: 3.81599236802463
azimuth: 122.005383208042

7800
EAST: 5.79999999998836
NORTH: -4.39999999944121
dAlong: 7.28010988893352
dAcross: 3.50544159168541
azimuth: 127.184706449785

EAST: 6.40000000002328
NORTH: -4.40000000037253
dAlong: 7.76659513580927
dAcross: 3.6257844666899
azimuth: 124.508522989836



I didn't understand the part you said about subtracting the eastings from the northings.

I hope this is close to what you want to do.

Update: Modified the fetch script and I'm pretty sure my calculations may not be what you're seeking, but perhaps its a start.

I named the table in the create script and the fetch script 'Sample3' so to know that data came from 'Sample3.txt'. (What naming convention you want is up to you.)

PS Your Untitled.jpg is worth a thousand words! The visualization makes the problem much easier to see.

Made a change to the create table script - faster to use a statement handle, $sth (with execute), than the original method I used, $dbh->do(... )


(This post was edited by Chris Charley on Nov 16, 2017, 11:44 AM)


marky9074
Novice

Oct 31, 2017, 12:14 PM

Post #7 of 8 (3222 views)
Re: [Chris Charley] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

Thanks Chris,

Sorry for the late response, I couldn't log in (said account was not verified).... seems OK now though.

I'm exceeding the limits of my shell at the moment. I was using mobaxterm, then tried the Ubuntu bash shell in Windows 10. A lot of the extra modules I cannot download (at the moment as I'm on a ship). Looks like this is a project I'll need to continue when I get home...

As for the subtraction of eastings/nothings. Once all of the unique instances of one ID are in the database. If you subtract the first easting, from the second easting (for say ID 001) then we would have the delta Easting. Same for the northings. Once we have these values we can calculate the azimuth between the two points and the hypotenuse, which would be the delta along.

Hope that makes sense. Could really do with a picture - will do that next..


marky9074
Novice

Oct 31, 2017, 12:58 PM

Post #8 of 8 (3220 views)
Re: [marky9074] Extract from text file, basic trig functions, then graph [In reply to] Can't Post

The attached image hopefully makes sense. Once the actual hypotenuse is calculated, the nominal hypotenuse value is subtracted (one assuming that distance between points and azimuth were constant), and we would get the delta distance along.

I am also trying to calculate the delta distance across. To do this I need to calculate the azimuth between the two points to do the trig to find the dAcross value. Then, as there is a nominal value for this based on constant distance between points and azimuth, this can be calculated and subtracted to get the delta distance across.

Once the delta distance along and delta distance across is calculated for each coordinate pairs down the dataset, I would like to plot them.
Attachments: Untitled.jpg (39.8 KB)

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives