Context: Technical companion to The $500K Cease-and-Desist postmortem.
Structured report: github.com/Vinaygond/The-500K-C-D-Report
The wrong question is: “How do you scrape 2M profiles per day without getting banned?”
The right question is: “What architectural choices let a system reach that throughput — and which design constraints did it ignore?”
This post documents the four-layer platform detection model behind Hydra / XLeadScraper V1, what each layer taught me, and how V2 was rebuilt around sustainability instead of vanity throughput.
The common misconception
Most developers assume platform rate limiting works like this:
“You get X requests per 15 minutes. Stay under that = safe.”
That describes Layer 1 only. Real enforcement stacks multiple signals. V1 passed early layers while failing later ones.
Layer 1: Per-endpoint limits
The visible layer. Every API surface has its own bucket.
V1 approach: endpoint rotation. A request router distributed traffic across ~47 API endpoints with independent limits. Instead of exhausting one surface (e.g., user_lookup), the system combined responses from multiple paths and reconstructed full profiles.
Lesson: Beating Layer 1 buys a multiplier, not durability.
Layer 2: Per-IP and per-token scoring
Platforms track whether an IP or credential behaves like a normal tenant even when each individual bucket looks fine.
V1 approach: the Hydra mesh.
- ~83 developer accounts, ~247 authentication tokens
- ~1,200+ rotating residential IPs across ~40 countries
- Load balancing across token + IP combinations; each IP used for a bounded request count before rotation
This moved peak throughput from thousands to millions of requests per day — a real distributed-systems result.
Lesson: Identity and network context matter as much as request counts. Rotation delays flags; it does not remove the signal.
Layer 3: Behavioral anomaly detection
V1 timing was optimized for throughput: consistent intervals, repetitive sequences, low idle time. ML systems detect automation from temporal and interaction patterns even when rate limits are respected.
Human behavior is inefficient. Efficient machines are obvious.
What V1 lacked:
- Randomized delays with realistic variance
- Idle periods between activity blocks
- Mixed request types beyond extraction
- Time-zone-aware usage patterns
Lesson: Respecting Layer 1 while ignoring Layer 3 is a timed failure, not a stable architecture.
Layer 4: Network-wide correlation
The terminal layer. Platforms correlate accounts, tokens, targets, and timing across the network.
V1’s shared infrastructure meant many customers and credentials looked like one coordinated system. Outcome:
- Developer accounts terminated
- Tokens revoked
- Founder account banned
- Customer accounts caught in blast radius
- Cease-and-desist issued
Lesson: Shared infrastructure converts individual risk into systemic risk.
V2: compliance as architecture
V2 is not “better rate-limit math.” It is a different product philosophy:
| Layer | V2 design |
|---|---|
| API usage | Official paths, OAuth 2.0 (PKCE), ≤50% of published limits |
| Architecture | Single-user auth; no shared proxy pools or token sharing |
| Timing | Human-pattern jitter, idle blocks, mixed activity |
| Network | Per-user isolation; no coordinated targeting patterns |
Throughput tradeoff: ~2M/day (network-level) → ~10k profiles/day per user.
That looks like a downgrade on paper. Operationally it is an upgrade: sustainable operation, protected customers, and output people can actually follow up on.
V1 vs V2 comparison
| Metric | V1 (scale-first) | V2 (compliance-first) |
|---|---|---|
| Peak throughput | ~2M requests/day | ~10k profiles/day per user |
| Platform risk | High; C&D in ~6 months | Designed for sustainable operation |
| Customer blast radius | 30+ accounts affected | Per-user isolation |
| Optimization target | Raw volume | Qualified leads |
| NPV framing | ~$500k in 6 months, then stop | Lower peak, longer runway |
Nobody can meaningfully follow up with 10,000 new leads per day. Speed was a vanity metric.
Engineering lessons
-
Compliance is a design constraint. Do not ship fast and “fix compliance later.” Design for policy, isolation, and blast-radius control from day one.
-
Reputation risk > legal risk. The C&D was expensive. Losing customer trust was worse.
-
Optimize for longevity. A compliant system with lower peak throughput can outperform a fast system that dies in six months.
-
Constraints breed better architecture. The technical challenge is not “how do I route around limits?” It is “how do I deliver maximum value within limits?”
Where this work lives now
- Full report: REPORT.md
- Detection layers: docs/detection-layers.md
- V1/V2 checklist: docs/v1-vs-v2.md
- V2 product: xleadscraper.com
- Current focus: Exit Protocol — forensic infrastructure where determinism and auditability are product requirements
Questions? vinay@exitprotocols.com