Guide to cross-batch deduplication of US data: How to reduce duplicate detection and keep customer projects isolated

In overseas marketing, US data is often accumulated in multiple batches - one batch is run from the number segment generator today, one batch is imported from a certain channel tomorrow, and another batch is added the day after tomorrow. When these batches are mixed and submitted to the screening task, if the numbers are repeated, meaningless duplicate detection will occur, which not only wastes balance but also makes data management confusing. To make things more complicated, you may serve multiple customer projects at the same time, each project has its own US number data, but you also want to share deduplication capabilities across projects to avoid being charged multiple times for the same number.

This article takes the “Data Deduplication Warehouse” function of KK-DATA as the core to explain how to automatically deduplicate cross-batch US data to achieve “one detection, multiple project sharing deduplication results”, and at the same time keep customer data exclusive through warehouse permission isolation. For teams that need to batch screen numbers, add followers on Telegram, and do WhatsApp marketing, this is a key skill to improve customer acquisition efficiency and control costs.

---

What is cross-batch deduplication of US data?

“Cross-batch deduplication” means that US number data generated at different times and from different sources automatically identify and eliminate numbers that have been tested before submitting the screening number or during the screening process to avoid repeated deductions.

For example:

• Batch A: Randomly generate 100,000 US number segments from the global number generation module.
• Batch B: 80,000 historical customer lists imported from CSV.
• Batch C: 20,000 potential customers collected from an exhibition.

Across the three batches, there may be 15%–30% overlap in numbers. If you submit three independent tasks one by one, the duplicate number will be detected multiple times and the balance will be consumed in vain. Cross-batch deduplication will cause the system to automatically skip the detected numbers and only charge for the newly added parts.

In KK-DATA, this kind of deduplication is not done manually by comparing files, but is done automatically through the built-in Data Deduplication Warehouse. You can think of the warehouse as a “collection of detected numbers”. Every time you submit a task and check the associated warehouse, the system will match the numbers in the task with the warehouse, eliminate duplicates, and only perform screening on undetected numbers.

---

Why does US number data need to be deduplicated across projects?

If you only occasionally screen a batch of numbers, manual deduplication may be tolerable. But if you are running multiple customer projects, or cleaning multiple US customer acquisition data at the same time (such as B2B directories, exhibition lists, social media exports), cross-project deduplication is no longer optional, but a necessity.

Typical cost issues of repeated testing

Using US Data as an example, digital marketing teams will often obtain numbers from multiple sources: LinkedIn exports, industry databases, ad form submissions. The overlap between these sources is typically between 15%–30%. Suppose you deposit 1,000 USDT. If 25% is wasted due to repeated testing, it means that 250 USDT is wasted. For a team processing millions of pieces of US filter data every month, this waste can add up quickly.

The balance between project isolation and data security

The difficulty of cross-project deduplication is that it is necessary to share the deduplication results without leaking customer A’s number to customer B. KK-DATA’s deduplication warehouse solves this contradiction through permission control - each warehouse can be created independently and bound to a specific project or team. The number data between warehouses is completely isolated, and only authorized users can see the numbers in the warehouse. In this way, different projects can maintain their own deduplication warehouses, and at the same time, all warehouses can be used as the basis for deduplication tasks, achieving “sharing in isolation.”

---

How does the KK-DATA data deduplication warehouse achieve cross-batch deduplication?

KK-DATA’s data deduplication warehouse is a built-in feature at no additional cost. Its core process is:

1. Upload the numbers that have been tested or to be tested to the warehouse.
2. Each time you submit a screening task, select the associated warehouse (you can select multiple times).
3. The system automatically performs hash comparison between the numbers in the task and the warehouse, and only detects unique numbers.
4. After the task is completed, the newly detected number will be automatically added to the warehouse (optional), and the next task will automatically take effect.

Matching logic and effective range of deduplication warehouse

• Match Logic: The number is hashed after normalization (removing country code prefix, spaces, etc.) so that +12065551234 and 12065551234 are recognized as the same number.
• Effective Scope: All numbers in a single warehouse share the deduplication capability, supporting cross-tasks and cross-projects (provided the warehouse has permission to share). For example, if the warehouse of project A and the warehouse of project B authorize each other (administrator operation is required), they can skip duplicate detection of each other.

Task isolation and permission control of deduplication warehouse

In the console, you can create separate deduplication warehouses for each customer or project and assign different administrators. Only the administrator of the warehouse can view or edit the list of numbers within it. The submitter of the number screening task only needs to check the warehouse when submitting the task. They will not see the full number in the warehouse, but can only see the statistical information of the “duplication results” (such as the number of skips). This design ensures data isolation without affecting deduplication efficiency.

---

Operation steps: Configure cross-batch deduplication for US data

The following takes a new batch of US number data as an example to demonstrate the complete process from creating a warehouse to submitting a task.

Step 1: Create a deduplication warehouse in the console and upload the US number

1. Log in to KK-DATA Application Console.
2. Enter the “Data Deduplication Warehouse” module and click “Create Warehouse”.
3. Enter the warehouse name (for example, “US 2025-06 B2B List”), optional comments.
4. Upload the number file (supports CSV/TXT, one number per row or one column of numbers).
5. After uploading, the system will automatically remove duplicates (it will remove duplicates in the warehouse itself) and display the number of valid numbers.

Step 2: Associate the deduplication warehouse when submitting the screening task

1. On the “Screen Number Task” page, click “New Task”.
2. Select the number source to be detected (it can be a newly uploaded number, or it can directly reference an existing warehouse).
3. In “Deduplication Settings”, check “Use Deduplication Warehouse”, and then select the warehouse you just created (you can also select multiple, for example, associate the “US B2B” and “US General” warehouses at the same time).
4. The system will automatically calculate the estimated cost before submission (only for numbers that do not appear in the warehouse).
5. Submit the task after confirmation.

Step 3: View the deduplication report and export the remaining numbers

• After the task execution is completed, you can see the “Duplication Statistics” area on the task details page, which displays “N items were detected in total, M duplicate items were skipped, and P items were actually detected.”
• You can export “Detection Results” (containing only actual detected numbers) or “Skip Numbers” (undetected duplicate numbers) for archiving or secondary use.

---

Best practices for cross-batch deduplication of US data

• Reasonable division of warehouses: It is recommended to divide warehouses by country/project/data source to avoid management difficulties caused by a warehouse that is too large. For US data, you can segment by state or industry, such as “US California B2B” “US New York C2C”.
• Regular cleaning of expired data: The activity of the number will decrease over time, and the test results three months ago may have become invalid. You can delete/rebuild the warehouse regularly, or uncheck the old warehouse in new tasks.
• Note the balance settlement process: There is no charge for deduplication itself, but the number actually detected will be deducted on a per-item basis. It is recommended to check the estimated cost before submitting the task and confirm it is correct before submitting.
• Using multi-warehouse joint deduplication: One task can be associated with multiple warehouses, which is suitable for scenarios where the detected numbers of multiple items need to be eliminated at the same time. For example, if you are doing US screening number data for multiple customers, you can check all customer warehouses at the same time. As long as a certain number appears in any warehouse, it will be skipped.
• Cooperates with the global number generation module: When generating numbers, directly select “Import into existing warehouse”, so that the generated numbers will automatically enter the deduplication library. When generating or importing the numbers later, the duplicate parts will not be wasted.

---

FAQ

**Q: After cross-batch deduplication, will the data of different projects contaminate each other? ** Answer: No. Deduplication warehouses can be created independently, and each warehouse is only used for its own associated tasks; the number detection results are only exported to the corresponding tasks, and the data between projects is isolated. Only the warehouse administrator can see the list of numbers in the warehouse, and the task submitter can only see the skipped quantity.

**Q: Can the previously skipped numbers be re-detected after deduplication? ** Answer: Yes. The deduplication warehouse supports manual removal of recorded numbers, or creating a new warehouse and re-importing the full list to detect again. If you want to confirm the current status of a skipped number, you can export the “skipped numbers” list in the task details, and then create a new task that is not associated with any warehouse for detection.

**Q: Is there any charge for the deduplication warehouse itself? ** Answer: There is no additional charge for using the deduplication warehouse function. It is a free additional capability of the platform, and the fee is only deducted based on the number of items detected by the screen number. Regardless of how many numbers are stored in the warehouse, there are no fees.

**Q: Can U.S. number data be mixed with numbers from other countries to eliminate duplication? ** Answer: Yes, but it is recommended to separate warehouses by country or project to facilitate management and isolation. If you really need mixed deduplication (such as a global marketing unified pool), you can put numbers from different countries into the same warehouse, but be aware that the numbers in the warehouse will not be filtered and deduplicated by country, and all numbers will participate in the comparison.

**Q: Is there a limit to the number of numbers in the deduplication warehouse? ** Answer: The upper limit of a single warehouse is 1 million items. If you need a larger scale, you can contact customer service to apply for expansion. If you have more than 1 million pieces of US data, it is recommended to create multiple warehouses by batch or subset, and plan task correlations appropriately.

---

If you are managing US Data customer acquisition in batches and want to reduce repeated detection and improve balance utilization, you may wish to experience KK-DATA’s data deduplication warehouse. 👉 Log in to the console to start screening numbers; if you need assistance with configuration, you can contact customer service in both directions https://t.me/kkdata_robot. For more function descriptions, please refer to Official Documents and Official Website.