High-Concurrency Email Verification in Go (Golang)

Why Go is Perfect for Data Pipelines

When you have a database of 10 million users to verify, Python or Node.js might struggle with the sheer throughput required to process that list in a reasonable timeframe. Go (Golang), with its native concurrency model based on Goroutines and Channels, is the ideal tool for building high-volume verification workers.

In this guide, we will build a worker pool pattern that reads emails from a channel, verifies them against EmailVerifierAPI.com, and handles the results concurrently.

The Worker Implementation

We will use the standard `net/http` package. The key here is to respect the API rate limits while maximizing throughput. We will spawn a fixed number of workers to prevent exhausting system resources.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "sync"
    "time"
)

const (
    APIKey     = "YOUR_API_KEY"
    APIUrl     = "https://www.emailverifierapi.com/v2/verify"
    WorkerCount = 10 // Adjust based on your plan limits
)

type VerificationResult struct {
    Status     string  `json:"status"`
    Disposable bool    `json:"disposable"`
    Score      float64 `json:"score"`
}

func verify(email string) (*VerificationResult, error) {
    client := &http.Client{Timeout: 10 * time.Second}
    req, _ := http.NewRequest("GET", APIUrl, nil)
    
    q := req.URL.Query()
    q.Add("api_key", APIKey)
    q.Add("email", email)
    req.URL.RawQuery = q.Encode()

    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    var res VerificationResult
    if err := json.NewDecoder(resp.Body).Decode(&res); err != nil {
        return nil, err
    }
    return &res, nil
}

func worker(id int, jobs <-chan string, results chan<- string, wg *sync.WaitGroup) {
    defer wg.Done()
    for email := range jobs {
        res, err := verify(email)
        if err != nil {
            results <- fmt.Sprintf("[Worker %d] Error: %s", id, email)
            continue
        }
        
        // Business Logic: Only accept valid + non-disposable
        if res.Status == "valid" && !res.Disposable {
             results <- fmt.Sprintf("[Worker %d] VALID: %s (Score: %.2f)", id, email, res.Score)
        } else {
             results <- fmt.Sprintf("[Worker %d] REJECT: %s (%s)", id, email, res.Status)
        }
    }
}

func main() {
    emails := []string{"test@example.com", "user@tempmail.com", "contact@google.com"} // Load your list here
    
    jobs := make(chan string, len(emails))
    results := make(chan string, len(emails))
    var wg sync.WaitGroup

    // Spawn Workers
    for w := 1; w <= WorkerCount; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    // Send Jobs
    for _, email := range emails {
        jobs <- email
    }
    close(jobs)

    // Wait for workers to finish
    wg.Wait()
    close(results)

    // Print Results
    for res := range results {
        fmt.Println(res)
    }
}

Optimizing for Throughput

The `WorkerCount` constant is your throttle. If you are on an Enterprise plan with EmailVerifierAPI.com, you can crank this up to 50 or 100 to process lists at lightning speed. Go's scheduler handles the context switching efficiently, ensuring that while one request is waiting for network I/O, other workers are processing.

Error Handling at Scale

When processing millions of records, network blips happen. In a production environment, you should wrap the `client.Do(req)` call in a retry loop with exponential backoff. This ensures that a momentary DNS glitch doesn't cause you to drop a valid lead.