CGI/Perl Guide | Learning Center | Forums | Advertise | Login Site Search: in Perl Guide PerlGuru Forums Learning Ctr

	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Need a Custom or Prewritten Perl Program?: I need a program that...: Takes in a 4 GIG text file and breaks it into 500MB chunks

samurai413x New User Sep 16, 2010, 6:15 AM Post #1 of 3 (1037 views) Takes in a 4 GIG text file and breaks it into 500MB chunks Can't Post I have some huge log files that are too big to open in TextPad. I need a script that takes in one of these giant log files and breaks it into several 500MB text files. The main issue is runtime. I had something that crawled through line by line, outputting the line to the chunked file, and checking to make sure the file was still under the 500MB limit, but this was going to take about 4 hours to finish. Looking for a trick to avoid line by line iteration or some library I don't know about to help get this done - I don't need a completely written program necessarily.

FishMonger Veteran Sep 16, 2010, 4:02 PM Post #2 of 3 (1029 views) Re: [samurai413x] Takes in a 4 GIG text file and breaks it into 500MB chunks [In reply to] Can't Post What OS are you using? *nix systems have a split (and csplit) utility which does exactly what you need. http://www.computerhope.com/unix/usplit.htm If you're on Windows, you could install the GnuWin32 utilities, which includes these utilities. http://gnuwin32.sourceforge.net/

samurai413x New User Sep 16, 2010, 4:24 PM Post #3 of 3 (1024 views) Re: [FishMonger] Takes in a 4 GIG text file and breaks it into 500MB chunks [In reply to] Can't Post Thanks! I'm running windows so I'll give the latter a shot. I had a feeling I would have to go outside perl for this one.

Announcements     PerlGuru Announcements Perl Programming Help     Frequently Asked Questions     Beginner     Intermediate     Advanced     Regular Expressions     mod_perl     DBI     Win32 Programming Help Fun With Perl     Perl Quizzes - Learn Perl the Fun Way     Perl Golf     Perl Poetry Need a Custom or Prewritten Perl Program?     I need a program that...     I Need a Programmer for Freelance Work     Throw Down The Gauntlet General Discussions     General Questions     Feedback     Tutorial/Article Suggestions for The Learning Cent     Internet Security Other Programming Languages     Javascript     PHP

Search this forum this category all forums for All words Any words Whole Phrase (options)

Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives