Why Event Logs Duplicate: Chain Reorganization and Deduplication Methods for Data Ingestion

Event log duplication usually stems from chain reorganization (reorg) — a temporary fork on the Ethereum mainnet, where the same transaction's logs from the old chain and the new chain are both broadcast. The solution is to deduplicate using (transactionHash, logIndex) as a composite unique key, and treat only events whose removed flag is false or null as ultimately valid.

1. Why Reorganization Causes Log Duplication

Ethereum's chain is constantly growing; occasionally two miners produce new blocks at the same time, causing a temporary fork. Nodes observe which chain is heavier and discard the shorter one — this is chain reorganization.

During a reorg, applications subscribed to the logs interface will receive two rounds of event push:

• The logs already sent on the old chain are pushed again, but this time with a field: removed: true.

• The new chain contains the same transaction in a new block, and the node pushes that log again (without the removed field).

This results in your application receiving two "events generated by the same transaction" — i.e., event log duplication. Alchemy's API documentation explicitly notes this behavior.

2. How to Determine Which Log Is Valid

Step 1: Check the removed field

• What to do: In each event log response, locate the boolean field removed.

• How to do:

  • Case A (removed is true): This log comes from an abandoned chain and is invalid. Do not process it, or delete it from your database if you have already stored it.

  • Case B (removed is absent or false): This log comes from the current canonical chain and is ultimately valid. It can be safely processed or persisted.

• Completion criteria: You can clearly distinguish which log comes from the old chain and which from the new chain.

Step 2: Use a "confirmation count" mechanism to delay processing

• What to do: Don't process a transaction's logs the moment it appears on chain; wait until several additional blocks are mined after it.

• How to do: In your code logic, set a confirmation depth (e.g., wait for 12–15 block confirmations). Ethereum.org and various indexer solutions recommend this mechanism to reduce the risk of dirty data caused by shallow reorgs.

• Completion criteria: You only trigger business logic after the transaction receipt has received enough confirmations, avoiding being misled by short-lived forks.

Prerequisite: You are subscribing to logs events via a node's RPC interface or WebSocket, or using eth_getLogs for historical queries. If you simply view transactions manually on Etherscan, you won't encounter this problem — the explorer already handles it for you.

3. Deduplication Methods for Data Ingestion

If you're building your own on-chain indexer, you can't simply "insert new records" in code; you must use "insert or update" idempotent logic.

Step 3: Deduplicate with a composite primary key

• What to do: In your database, create a composite unique key using the two fields (transactionHash, logIndex).

• How to do:

  • transactionHash: The hash of the transaction that produced the log.

  • logIndex: The index position of this log within the transaction (starting from 0).

  • These two fields together uniquely identify a specific on-chain log.

• Completion criteria: Your database table has this composite unique index set up, and even if the node replays the same log three times, no duplicate data will be written.

Step 4: Delete or mark old records when receiving removed=true

• What to do: When you receive a log with removed=true, use its (transactionHash, logIndex) to delete the corresponding row from the database.

• How to do: This is standard practice for on-chain indexers. Open-source projects demonstrate this rollback logic: maintain the hashes of the last N blocks, and upon detecting a fork, immediately roll back to the common ancestor block, clearing all subsequent logs.

• Completion criteria: The data in your database ultimately retains only the log from the new chain, with no residual invalid data.

Risk reminder: If you only listen to the newHeads subscription without checking the removed field, or rely on a simple "increment by block number" insertion logic, you will miss state changes during a reorg, and fake data from the old chain will linger in your database. This can cause serious reconciliation issues in scenarios requiring precise transaction amount calculations (e.g., DeFi lending protocols).

After completing the above setup, how to verify that the deduplication logic works?

Simulate a reorg on a testnet (some RPC node tools support forced forking) and observe your listening service: if you receive two logs for the same transaction, but the second one is marked removed, and your database always contains only one valid record — your deduplication and rollback mechanism is working. Next, set the confirmation depth to an acceptable balance point (typically 12 blocks).