Perl Guru Forums: Perl Programming Help: Intermediate: Byte-level pointers similar to C pointers

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
Byte-level pointers similar to C pointers

rpaskudniak
User

Aug 29, 2013, 2:54 PM

Post #1 of 10 (1017 views)

Byte-level pointers similar to C pointers	Can't Post

Greetings.

Some years ago I wrote a C utility that would read the internal structures of a database page (like 2K or 4K) and display the sections separately. This involved pointer arithmetic a a low level, like a pointer to the buffer holding the "page" and then other pointers to the sub-structures that I calculated by adding their offsets to the original buffer pointer.

I may soon be in a project (if I get the job!) where I will need to do this (mainly on the writing side) in Perl. I can do C pointer arithmetic (along with the necessary casting) in my sleep but no clue how to do this in Perl. Based on the results from printing a reference in Perl, I already know a reference ain't a pointer.

Scenario A bit crude but illustrates the kind of pointer arithmetic I need to use.:

Create an 8K buffer. (How? Malloc? Can't find info on malloc in Perl Shmget?)

Get a pointer to header structure (offset 0)

Calculate a pointer to a record-pointer array (say, 42 bytes into the buffer)

Before I add a new record to the buffer, locate an offset to an empty location.

As I add a record to the buffer, save its offset within the buffer into a slot in the record-pointer array

Other internal bookkeeping functions

Like I said, in C I can do this in my sleep. But how do I deal with byte-offsets and pointer arithmetic in Perl? The sub-structures would be nicely represented by different hashes, of course, but not in that form inside the buffer.

How do I get started on something like this? (OK, which FM has this kind of information?)

Thanks. (Hey, it's only one screenful! I'm losing my perspicacious touch! Wink

)
--------------------
-- Rasputin Paskudniak (In perpetual pursuit of undomesticated, semi-aquatic avians)

Laurent_R
Enthusiast / Moderator

Aug 29, 2013, 11:20 PM

Post #2 of 10 (1014 views)

Re: [rpaskudniak] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Hmm, the main difference between C pointers and Perl references is that you just can't do pointer arithmetic on references. (The other difference beings that you are much less likely to have severe memory management errors such as segmentation fault, memory leaks etc.).

But you've still got many means to read binary data in Perl.

rpaskudniak
User

Aug 30, 2013, 9:03 AM

Post #3 of 10 (1010 views)

Re: [Laurent_R] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Laurent,

I am aware that there is a malloc function in Perl but I can't find anything about it on perldoc.org. I couldn't even find anything about it in Mastering Perl, though a Google search turned up something I couldn't follow.

Thanks for the encouraging word that there are many means to get to this. As an hobbyist mathematician, I have long come to appreciate the notion that "A solution exists." but where the man meets the saddle Angelic

, I need pointers (Yes, pun intended) to some of those actual solutions.

Hey, folks, I'm all ears for this! Cool

--------------------
-- Rasputin Paskudniak (In perpetual pursuit of undomesticated, semi-aquatic avians)

FishMonger
Veteran / Moderator

Aug 30, 2013, 9:46 AM

Post #4 of 10 (1006 views)

Re: [rpaskudniak] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Perl is not C and C is not Perl.

You have an XY problem.

What problem are you trying to solve that makes you think that you need to implement C's pointer arithmetic in perl?

If you really do need that functionality, then you could probably use Inline::C to accomplish your task.
http://search.cpan.org/~sisyphus/Inline-0.53/C/C.pod

However, I'm willing to bet that if you tell us what the real problem is that needs to be solved, we will be able to point out perl solutions.

Laurent_R
Enthusiast / Moderator

Aug 30, 2013, 9:56 AM

Post #5 of 10 (1004 views)

Re: [rpaskudniak] Byte-level pointers similar to C pointers [In reply to]	Can't Post

In Reply To

Thanks for the encouraging word that there are many means to get to this. As an hobbyist mathematician, I have long come to appreciate the notion that "A solution exists." but where the man meets the saddle Angelic

, I need pointers (Yes, pun intended) to some of those actual solutions.

I really can't say much more so long as I don't know which type of binary data you are talking about. Is the data read from a file? From a socket? Something else? What do you want to do with the data? Extract some information? Process it and modify it? In brief, tell us more about what you want to do.

As for malloc, you are probably refering to a function use internally by Perl, which can either use the C malloc or implements its own. But Perl users (i.e. people writing code in Perl) are not supposed to use it, as far as I can say.

rpaskudniak
User

Aug 30, 2013, 12:27 PM

Post #6 of 10 (1001 views)

Re: [FishMonger] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Fishmonger, Laurent,

I'm looking at the Inline::C module as I type this. (OK, looking away.. :-) I won't discuss my exact plan for now because of a commercial issue but here's what I did in the old C program I had mentioned:

In Informix, the unit of disc I/O is the "Page". For simplicity, I'll restrict the discussion to 2-K pages. (And 32-bit Informix)

The first 28 bytes of the page are a structure whose members have known size and types eg. a time stamp, chunk/offset address, row count, etc. The exact purpose of each member item is not really important to this question.

Toward the end of the page is an array of offset pointers (called the "slot table" in Informix parlance) to reference the records and tell me their lengths. So if I need to access record[15] in the page, I easily locate slot[15] in that table as an array entry (since I know where it starts), obtain the page-relative offset from the slot entry, and add that offset to the address of the start of the page [in memory].

I now have a pointer to the record itself.

I have ridden roughshod over a couple of details about the slot table (in case another Informix internals person sees this and suspects me of forgetfulness) but it makes the point about my need for true pointer arithmetic.

(Whew!) If I can write and compile functions for this in C and somehow have the Perl compiler connect to the object, it would probably be ideal, seeing as it seems to be impossible in Perl. (Nah! Nothing should be impossible in Perl!) HMM... Isn't this how the DBD:: modules get pulled into a Perl program?

I hope I have clarified my requirement. (It would be a record for me! The scroll bar didn't appear until I added this paragraph. Smile

)
--------------------
-- Rasputin Paskudniak (In perpetual pursuit of undomesticated, semi-aquatic avians)

FishMonger
Veteran / Moderator

Aug 30, 2013, 12:53 PM

Post #7 of 10 (997 views)

Re: [rpaskudniak] Byte-level pointers similar to C pointers [In reply to]	Can't Post

I'm not a C programmer and I haven't use Informix so I can't relate to much of what you described. Which means your clarification is clear as mud to me.

Are you trying to write a script that directly interfaces with the Informix DB? If so, there is already a DBD::Informix module. You could look at its source code to see how the author handled things like this. You could even contact the authors directly.

If that's not what you're looking for then maybe look at some of Perl's built-in functions such as pack, unpack, seek, and tell.

When you're ready to provide a better explanation on what you need to do (not how something is done in C) and even post your test code, we should be able to give more direct and concrete answers.

BillKSmith
Veteran

Aug 30, 2013, 1:20 PM

Post #8 of 10 (995 views)

Re: [FishMonger] Byte-level pointers similar to C pointers [In reply to]	Can't Post

While I was preparing an answer, fishmonger's latest reply appeared. It contains exactly the same ideas I was preparing, but stated much better. You really must tell us why you cannot use existing modules.
Good Luck,
Bill

Laurent_R
Enthusiast / Moderator

Aug 31, 2013, 2:11 AM

Post #9 of 10 (988 views)

Re: [rpaskudniak] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Hi,

Using the Informix module would probably be the best.

As a more general solution to read specific bytes, what about opening your variable with a filehandler and using seek/read to read individual byte chunks? I am doing that from time to time on VMS/RMS binary files for which there is no usable module: fetching individual records and looking for specific fields in the records. I do not have an actual example here at home, but this is a quick example under the Perl debugger:

Code

  DB<1> $c = "The quick brown fox jumps over the lazy dog." 
 
  DB<2> open $fh, "<", \$c or die "could not open $c $!"; # opens a reference to the $c variable as a filehandler 
 
  DB<3> seek $fh, 10, "00"; # positions the filehandler on the 10th byte 
 
  DB<4> read $fh, $out, 10; # reads 10 bytes from position 10 
 
  DB<5> print $out; 
brown fox

(This post was edited by Laurent_R on Aug 31, 2013, 2:13 AM)

rpaskudniak
User

Sep 1, 2013, 12:29 PM

Post #10 of 10 (961 views)

Re: [Laurent_R] Byte-level pointers similar to C pointers [In reply to]	Can't Post

Laurent said:

Code

  DB<1> $c = "The quick brown fox jumps over the lazy dog."  
  
  DB<2> open $fh, "<", \$c or die "could not open $c $!"; # opens a reference to the $c variable as a filehandler  
  
  DB<3> seek $fh, 10, "00"; # positions the filehandler on the 10th byte  
  
  DB<4> read $fh, $out, 10; # reads 10 bytes from position 10  
  
  DB<5> print $out;  
brown fox

Laurent! That's beautiful!! I've written C code to access /dev/mem and /dev/kmem (I sometimes miss my days at AT&T in 30 Knightsbridge) but I had forgotten that I could do this in Perl with a generic block of memory. This opens a whole new vista for me!

In other words, Dr. Laurent: You have just crrrreated a MONSTERRRR! Cool

If nothing more, it will be great for proof of concept and might just be efficient enough to use in production, using a reference to a shared memory segment to open the file descriptor and building my module around that.

Some of the more sniggling details might need some research into structures and Data::Structure::Util (And Hey! That module is/ written in C!) but I have a starting point. And an example to mimic if the fd approach is not efficient enough.

Fishmonger, I'm sorry my explanation was misleading. It was meant to convey the spirit of the kind of pointer arithmetic I needed, not to be a lesson in the [publicly known aspects of] Informix internals. And yes, the data will be coming from a TCP port, buffered in such blocks, and written to file, to read back on demand by other utilities.

VMS/RMS? Hey, that brings back even older memories! I'm sorry VMS has so fallen out of favor; it was a GREAT OS to work with! But hey, what's a few decades among friends? Wink

(No smiley for doddering old fudd. :-{)> )

(Now how do I mark this SOLVED, other than editing the subject line?)
--------------------
-- Rasputin Paskudniak (In perpetual pursuit of undomesticated, semi-aquatic avians)

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives