When importing data, some staging table records are duplicating or an Import set row is duplicating - Support and Troubleshooting

When importing data, some staging table records are duplicating or an Import set row is duplicating Issue  Symptoms You recognise this problem because:

Running of transform map is duplicating the records in staging table. There are multiple transform map is using for same Import set. There is no traces in node logs about this staging table Cause This is the expected behavior if you are using multiple transform maps for the same Import set in a singe run.

One import set row is created per transform map. This behavior can cause a large number of temporary records to be generated.

Resolution Clean the staging table. If you need to clean the staging table increase the clean up frequency.

Note : For large imports, increase the clean up frequency of the staging tables