By
DQT Product Specialist
Speeding Up MatchUp Object Incremental Processing
The Incremental Deduping Interface allows multi-users to simultaneously verify existing records and add new ones. This may come at the expense of speed, however, because each record processed must be immediately added (inserted) to the key file, so each running process can see the newly updated key file - ensuring accurate data storage.
But what if your database is a single-user or single entry point application? Timing and order of comparisons isn't important, so why take the speed hit?
The Incremental Deduper now includes a group of Transaction functions, allowing the process to accurately match new incoming records while delaying them from being physically written to the Incremental database (keyfile) until you send a commit call to the process. This will greatly speed up processing when adding large numbers of records and give you the flexibility to rollback a series of added keys if problems arise with the actual data.
Incremental.ClearMappings();
Incremental.AddMappings...
Incremental.BeginTransaction()
while (!Database.Eof()) {
Database.Record.Incremental.BuildKey()
Incremental.AddRecord()
Database.NextRecord()
If error then
Incremental.RollbackTransaction()
return}
Incremental.CommitTransaction()
Calling this transaction delays the insert calls, while deduping operations remain the same, thus saving processing time without compromising accuracy.
Remember, transaction processing SHOULD NOT be used if multiple threads/processes/users/machines are accessing the same Incremental database!
Authors