Jun 30, 2014, 7:37 AM
Post #1 of 3
Suggestions on how to solve diffing two ldap files
This is a bit of a long post so bear with me as I try my best to explain my problem.
I was given a task a while back to extract client records from a legacy database and migrate them into a ldap database.
I have data that that we are pulling from a database and have been putting into ldif format for insertion into a ldap db.
customername: example company
customerName: Example Company
This file can sometimes span up to 45k in entries for client records. My first issue was that I needed to figure out how to diff these files generated let's say hourly and only pick up the differences since it would take a long time to clear out the ldap database and re-add everything to keep the database fresh.
Aka: Generate 1.full then 2.incremental, diff the two files and create 1.diff. Then apply the diffs via ldapupdate. I was using ldifdiff.pl for the task. I believe it's a perl module.
So that was working for a period of time until we introduced hashed passwords. That screwed up everything since now each time the sync occurred a new SHA password would be computed for the incremental file.
Back to the drawing board...
To solve that issue, I added a field called "lastUpdate", this would print the time that a record that we were pulling from the database was updated.
My logic was now I can somehow just parse the two files and only print out the differences if this "lastUpdate" field was modified.
Well, since then I've been spinning my wheels on how to accomplish this. Some ideas I've been throwing around in my head are to:
1. Load everything into a hash and diff the values there?
2. Load the two files and extract the lastUpdate field between the two files and print out the blocks of records that have had their update times modified?
3. I also thought about modifying ldifdiff.pl and seeing if I could just add a conditional to check if lastupdate has been modified between the two files.
Still unsure so throwing it out here to see if anyone could offer some advice.