CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Need a Custom or Prewritten Perl Program?: I need a program that...:
Takes in a 4 GIG text file and breaks it into 500MB chunks

 



samurai413x
New User

Sep 16, 2010, 6:15 AM

Post #1 of 3 (2033 views)
Takes in a 4 GIG text file and breaks it into 500MB chunks Can't Post

I have some huge log files that are too big to open in TextPad. I need a script that takes in one of these giant log files and breaks it into several 500MB text files.

The main issue is runtime. I had something that crawled through line by line, outputting the line to the chunked file, and checking to make sure the file was still under the 500MB limit, but this was going to take about 4 hours to finish.

Looking for a trick to avoid line by line iteration or some library I don't know about to help get this done - I don't need a completely written program necessarily.


FishMonger
Veteran / Moderator

Sep 16, 2010, 4:02 PM

Post #2 of 3 (2025 views)
Re: [samurai413x] Takes in a 4 GIG text file and breaks it into 500MB chunks [In reply to] Can't Post

What OS are you using?

*nix systems have a split (and csplit) utility which does exactly what you need.
http://www.computerhope.com/unix/usplit.htm

If you're on Windows, you could install the GnuWin32 utilities, which includes these utilities.
http://gnuwin32.sourceforge.net/


samurai413x
New User

Sep 16, 2010, 4:24 PM

Post #3 of 3 (2020 views)
Re: [FishMonger] Takes in a 4 GIG text file and breaks it into 500MB chunks [In reply to] Can't Post

Thanks! I'm running windows so I'll give the latter a shot. I had a feeling I would have to go outside perl for this one.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives