This component filters out duplicate records. Here ‘duplicate’ is defined as the records that have the same value for the fields specified in the input. Any pass-through fields are ignored in this comparison. Only the first unique record encountered is retained and any duplicates that follow are dropped.
This component can run in two different modes* If the input records to the component are ordered in such a way that the duplicate records are adjacent to each other, then the property ‘Contiguous Duplicates’ should be set to True. This allows the component to run in an optimized manner and the records can be processed with no buffering.
If the input records have no such ordering, then the property ‘Contiguous Duplicates’ should be set to false. The component in this mode will have to buffer all the records and reorder them before eliminating the duplicate records.