CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
extracting a record from a file based on another file

 



dgilby
Novice

Nov 21, 2009, 10:26 AM

Post #1 of 14 (2015 views)
extracting a record from a file based on another file Can't Post

Hi How are things?

I am wondering what the best approach to use to extract a 20 records above and 10 records below a timestamped record.

I want to open a file and read a record based on a certain time, open a second file and search the 2nd file for 20 records above and 10 records below the timestamp and write it to an output file.

Greatly appreciate the response.
Derek


FishMonger
Veteran / Moderator

Nov 21, 2009, 10:44 AM

Post #2 of 14 (2010 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

What have you tried? Please post your code.

Exactly what portion of the task do you not know how to do, or is giving you trouble?

Can you post a sample of each file?


dgilby
Novice

Nov 21, 2009, 11:04 AM

Post #3 of 14 (2009 views)
Re: [FishMonger] extracting a record from a file based on another file [In reply to] Can't Post

I haven't tried anything yet just looking, and reading documentation. I guess I am looking for mostly how to extract the 20 records above and the 10 records below the timestamp. What I have is a shutter times stamp and an IMU (Aerial Camera) speed and I need to extract the 30 records based on the shutter speed. Of course the times are considerably different.


Files
Shutter speed times (extract shutter speed time)

Line2 Shutter 2009/10/30 11:58:52:468.7500 Image 2009/10/30 11:58:55:469.3120

IMU Times ( pick time closest to the shuuter speed time and collect the 30 closest records)

0004690 2009:10:30 11:58:51:828
0004691 2009:10:30 11:58:51:843
0004692 2009:10:30 11:58:51:859
0004693 2009:10:30 11:58:51:875
0004694 2009:10:30 11:58:51:906
0004695 2009:10:30 11:58:51:921
0004696 2009:10:30 11:58:51:937
0004697 2009:10:30 11:58:51:953
0004698 2009:10:30 11:58:51:984
0004699 2009:10:30 11:58:52:000
0004700 2009:10:30 11:58:52:015
0004701 2009:10:30 11:58:52:046
0004702 2009:10:30 11:58:52:062
0004703 2009:10:30 11:58:52:078
0004704 2009:10:30 11:58:52:093
0004705 2009:10:30 11:58:52:125
0004706 2009:10:30 11:58:52:140
0004707 2009:10:30 11:58:52:156
0004708 2009:10:30 11:58:52:187
0004709 2009:10:30 11:58:52:203
0004710 2009:10:30 11:58:52:218
0004712 2009:10:30 11:58:52:265
0004713 2009:10:30 11:58:52:281
0004714 2009:10:30 11:58:52:296
0004716 2009:10:30 11:58:52:343
0004717 2009:10:30 11:58:52:359
0004718 2009:10:30 11:58:52:375
0004719 2009:10:30 11:58:52:406
0004720 2009:10:30 11:58:52:421
0004721 2009:10:30 11:58:52:437
0004722 2009:10:30 11:58:52:453
0004723 2009:10:30 11:58:52:484
0004724 2009:10:30 11:58:52:500
0004725 2009:10:30 11:58:52:515
0004726 2009:10:30 11:58:52:546
0004727 2009:10:30 11:58:52:562
0004728 2009:10:30 11:58:52:578
0004729 2009:10:30 11:58:52:593
0004730 2009:10:30 11:58:52:625
0004731 2009:10:30 11:58:52:640
0004732 2009:10:30 11:58:52:656
0004733 2009:10:30 11:58:52:687
0004734 2009:10:30 11:58:52:703
0004735 2009:10:30 11:58:52:718
0004736 2009:10:30 11:58:52:734
0004737 2009:10:30 11:58:52:765
0004738 2009:10:30 11:58:52:781
0004739 2009:10:30 11:58:52:796
0004740 2009:10:30 11:58:52:828
0004741 2009:10:30 11:58:52:843
0004742 2009:10:30 11:58:52:859
0004743 2009:10:30 11:58:52:875
0004744 2009:10:30 11:58:52:906
0004745 2009:10:30 11:58:52:921
0004746 2009:10:30 11:58:52:937
0004747 2009:10:30 11:58:52:953
0004748 2009:10:30 11:58:52:984
0004749 2009:10:30 11:58:53:000
0004750 2009:10:30 11:58:53:015
0004751 2009:10:30 11:58:53:046
0004752 2009:10:30 11:58:53:062
0004753 2009:10:30 11:58:53:078
0004754 2009:10:30 11:58:53:093
0004755 2009:10:30 11:58:53:125
0004756 2009:10:30 11:58:53:140
0004757 2009:10:30 11:58:53:156
0004758 2009:10:30 11:58:53:187
0004759 2009:10:30 11:58:53:203
0004760 2009:10:30 11:58:53:218
0004761 2009:10:30 11:58:53:234
0004762 2009:10:30 11:58:53:265


(This post was edited by dgilby on Nov 21, 2009, 11:22 AM)


FishMonger
Veteran / Moderator

Nov 21, 2009, 12:38 PM

Post #4 of 14 (2001 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

When you say "pick time closest to the shuuter speed time", do you mean the closest that is less than the shutter speed, or the the next higher time?

Load the IMU Times file into an array and loop thru it using the index numbers. Once you find the desired timestamp, then output the array elements via an array slice.

If the file is large and you're worried about the amount of memory usage of the array, then use the Tie::File module.

Try writing the script and when you get stuck, post the script and describe the problem and include any/all warnings and errors that you receive.


dgilby
Novice

Nov 21, 2009, 8:23 PM

Post #5 of 14 (1996 views)
Re: [FishMonger] extracting a record from a file based on another file [In reply to] Can't Post

I will thanks.

I am looking for n number of records before the time and n number of records after the time. The number of records will vary

Thanks again


ichi
User

Nov 22, 2009, 1:25 AM

Post #6 of 14 (1988 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

the easiest way, is if you have *nix system, you can use GNU grep

Code
grep -A 20 -B 10 "<timestamp>" file

of course, also if Perl is not a must for you.


(This post was edited by ichi on Nov 22, 2009, 1:26 AM)


dgilby
Novice

Nov 22, 2009, 6:39 AM

Post #7 of 14 (1973 views)
Re: [ichi] extracting a record from a file based on another file [In reply to] Can't Post

I'll look up an equivalent to for XP use. Thanks


(This post was edited by dgilby on Nov 22, 2009, 6:41 AM)


ichi
User

Nov 22, 2009, 4:18 PM

Post #8 of 14 (1956 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post


In Reply To
I'll look up an equivalent to for XP use. Thanks

XP? ok go
http://gnuwin32.sourceforge.net/packages.htm and download GNU grep for windows and you are done.


(This post was edited by ichi on Nov 22, 2009, 4:19 PM)


dgilby
Novice

Nov 23, 2009, 6:42 AM

Post #9 of 14 (1943 views)
Re: [ichi] extracting a record from a file based on another file [In reply to] Can't Post

Thanks. I am assuming I can use tcl/tk to put this in a GUI?


ichi
User

Nov 23, 2009, 9:43 PM

Post #10 of 14 (1932 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

yes, Perl has GUI libraries (can't remember),, but yes, there should be tk (with Python its Tkinter).


dgilby
Novice

Nov 25, 2009, 10:02 AM

Post #11 of 14 (1913 views)
Re: [ichi] extracting a record from a file based on another file [In reply to] Can't Post

Hi again

Does anyone know where I can get the tk demo widget for download?

Thanks


FishMonger
Veteran / Moderator

Nov 25, 2009, 10:56 AM

Post #12 of 14 (1912 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

http://search.cpan.org/~ni-s/Tk-804.027/demos/widget


dgilby
Novice

Nov 26, 2009, 6:34 AM

Post #13 of 14 (1898 views)
Re: [FishMonger] extracting a record from a file based on another file [In reply to] Can't Post

Thanks


dgilby
Novice

Dec 4, 2009, 4:22 AM

Post #14 of 14 (1856 views)
Re: [dgilby] extracting a record from a file based on another file [In reply to] Can't Post

Hi Everyone How are things?

Thanks for you input so far. Your guidance has produced this wonderful script.

One problem though. I am still having trouble with GREP at the very end. Would anyone have any ideas on how to go about finishing this?

Again I would like the data in the previous post to loop through the data and print the 20 records above and the 10 records after the timestamp. Any help would be greatly appreciated



#!/usr/bin/perl -w
use strict;
use Tk;

my $ImageFilePath;
my $IMUFilePath;
my $ResultFilePath;
my $BeforeRecord = 20;
my $AfterRecord = 10;

my $mw = MainWindow -> new;
$mw -> title("IMU Data Processing");


#my $types = [['Text Files','.txt'],['All Files','*'],];

# sizes window
$mw -> geometry('400x200');

######################### MENU BEGIN ###############################
# menu
my $mbar = $mw -> Menu();

$mw -> configure(-menu => $mbar);

# Main Buttons
my $file = $mbar -> cascade(-label => "File",
-underline => 0,
-tearoff => 0);

my $edit = $mbar -> cascade(-label => "Edit",
-underline => 0,
-tearoff => 0);

my $help = $mbar -> cascade(-label => "Help",
-underline => 0,
-tearoff => 0);

#File Menu
$file -> command(-label => "New",
-underline =>0,
-command => [\&menuClicked, "New"]);

$file -> command(-label => "Open",
-underline => 0,
-command => [\&menuClicked, "Open"]);

$file -> command(-label => "Save",
-underline => 0,
-command => [\&menuClicked, "Save As"]);

$file -> separator();

$file -> command(-label => "Exit",
-underline => 1,
-command => sub {exit});

#Edit Menu


#About/Help Menu
$help -> command(-label => "Help",
-underline => 0,
-command => [\&menuClicked, "Help"]);

$help -> separator();

$help -> command(-label => "About",
-underline => 0,
-command => [\&menuClicked, "About"]);

######################### MENU END ###############################

######################### GUI Begin ##############################

#my $frm = $mw -> Frame();

# text explaining window
my $lblInstructions = $mw->Label(-text => "This window processes IMU Data based on Aerial Camera Shutter Times", -padx=>5, -pady=>5);

# label for Shutter times file
my $lblImageFile = $mw->Label(-text => "Shutter File:", -padx=>5, -pady=>5);

# entry textbox for Shutter times dialog
my $entImageFile = $mw->Entry(-width=>30, -textvariable=>\$ImageFilePath);

# button to browse for file
my $butImageFile = $mw->Button(-text=>'Browse', -command=> sub { openImageFileDialog($entImageFile)});

# label for IMU dialog
my $lblIMUFile = $mw->Label(-text => "IMU File:", -padx=>5, -pady=>5);

# textbox for IMU times dialog
my $entIMUFile = $mw->Entry(-width=>30, -textvariable=>\$IMUFilePath);

# button to browse for file
my $butIMUFile = $mw->Button(-text=>'Browse', -command=> sub { openIMUFileDialog($IMUFilePath)});

# label for output file
my $lblResultFile = $mw->Label(-text => "Output File:", -padx=>5, -pady=>5);

# textbox for output file dialog
my $entResultFile = $mw->Entry(-width=>30, -textvariable=>\$ResultFilePath);

# button to browse for file
my $butResultFile = $mw->Button(-text=>'Browse', -command=> sub { saveResultFileDialog($ResultFilePath)});

# label for before records
my $lblBeforeRecord = $mw->Label(-text => "Before Record", -padx=>5, -pady=>5);

# textbox for before records
my $entBeforeRecord = $mw->Entry(-width=>30, -textvariable=>\$BeforeRecord);

# label for after records
my $lblAfterRecord = $mw->Label(-text => "After Record", -padx=>5, -pady=>5);

# textbox for after records
my $entAfterRecord = $mw->Entry(-width=>30, -textvariable=>\$AfterRecord);

# process file
my $butProcessFile = $mw->Button(-text => "Process File", -width=>10, -command=>\&processFile);

# Exit button
my $butExit = $mw->Button(-text => "Exit", -width=>10, -command => sub { exit });

######################### GUI End ################################

######################### Geometry Management ####################

#$frm -> grid(-row=>1);
$lblInstructions -> grid(-row=>1, -column=>1, -columnspan=>4);
$lblImageFile -> grid(-row=>2, -column=>1);
$entImageFile -> grid(-row=>2, -column=>2);
$butImageFile -> grid(-row=>2, -column=>3);

$lblIMUFile -> grid(-row=>3, -column=>1);
$entIMUFile -> grid(-row=>3, -column=>2);
$butIMUFile -> grid(-row=>3, -column=>3);

$lblResultFile -> grid(-row=>4, -column=>1);
$entResultFile -> grid(-row=>4, -column=>2);
$butResultFile -> grid(-row=>4, -column=>3);

$lblBeforeRecord -> grid(-row=>5, -column=>1);
$entBeforeRecord -> grid(-row=>5, -column=>2);

$lblAfterRecord -> grid(-row=>6, -column=>1);
$entAfterRecord -> grid(-row=>6, -column=>2);

$butProcessFile -> grid(-row=>7, -column=>1, -columnspan=>2);
$butExit -> grid(-row=>7, -column=>2, -columnspan=>2);

######################### Geometry End ###########################

MainLoop;

sub openImageFileDialog {
my $imageFilePathOpen = $mw->getOpenFile(-title => 'Open Image File:');
$ImageFilePath = $imageFilePathOpen;
}

sub openIMUFileDialog {
my $imuFilePathOpen = $mw->getOpenFile(-title => 'Open IMU File:');
$IMUFilePath = $imuFilePathOpen;
}

sub saveResultFileDialog {
my $resultFilePathOpen = $mw->getSaveFile(-title => 'Save Result File:');
$ResultFilePath = $resultFilePathOpen;
}

sub processFile {

# open image file for reading
open IMAGEFILE, $ImageFilePath || die "Can't open $ImageFilePath: $!";
# open imu file for reading
# open IMUFILE, $IMUFilePath || die "Can't open $IMUFilePath: $!";

# open result file for writing
# open RESULTFILE, $ImageFilePath || die "Can't open $ImageFilePath: $!";

#while loop
while (<IMAGEFILE>) {
chomp;
my @imagerecords = split (/\t/, $_);

# prints records
# print join ("\t", $imagerecords[0], $imagerecords[3], $imagerecords[10]). "\n";

# GNU Grep example
#grep -A 20 -B 10 "<timestamp>" file

# grep - searches for a file pattern
# -A -
# 20 - number records before timestamp
# -B -
# 10 - number records behind timestamp
# "<timestamp>" - timestamp to look for
# file - file to loop through

grep -A $BeforeRecord -B $AfterRecord @imagerecords[3] $IMUFILE > RESULTFILE;
}

#while (<@imagerecords>) {

# GNU Grep example
#grep -A 20 -B 10 "<timestamp>" file

# grep - searches for a file pattern
# -A -
# 20 - number records before timestamp
# -B -
# 10 - number records behind timestamp
# "<timestamp>" - timestamp to look for
# file - file to loop through

# grep -A $BeforeRecord -B $AfterRecord @imagerecords[3] $IMUFILE > RESULTFILE;

#}

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives