API Scraping: When 'Public' Endpoints Become Mass Surveillance Weapons

On January 7, 2026, a dataset containing 17.5 million Instagram user records appeared on BreachForums—full names, email addresses, phone numbers, and partial location data, all structured and ready to exploit. The hacker posted it for free. No paywall. No restrictions. Just 17.5 million people's personal information, available to anyone with a Tor browser.

Meta's response? "There was no breach."

Technically, they're right. But functionally? The distinction between "breach" and "API scraping" is meaningless when your information is on the dark web. This is the new reality of API security: attackers don't need to break in when they can simply collect what you've left exposed through "public" endpoints.

The Instagram case isn't unique—it's symptomatic. APIs designed for "public" data access are being weaponized for mass surveillance. Here's why traditional defenses fail:

Authentication ≠ Authorization: Most APIs check who you are, not what you're doing. An authenticated user with a valid token can enumerate millions of profiles if the endpoint allows it. The Instagram scraper didn't use stolen credentials—they used the platform's own data exposure patterns.

Rate Limiting Is Insufficient: Basic rate limits (100 requests/minute) are meaningless when attackers distribute scraping across thousands of IPs. At 100 requests/minute, an attacker can collect 5.2 million records per month from a single node. Scale that to 50 nodes, and you're harvesting 260 million records monthly.

Bot Detection Is Broken: Modern scrapers use residential proxy networks, rotate User-Agents, and mimic human behavior patterns. They don't trigger CAPTCHAs because they behave like humans—just thousands of them in parallel.

Defending against API scraping requires abandoning the assumption that "public" data can be freely accessed without limits. Here's a layered defense strategy:

1. Semantic Rate Limiting

Don't just count requests—analyze behavior. Track metrics like unique resources accessed per session, data volume retrieved, and access pattern entropy.

# Anti-scraping rate limiter with behavioral analysis
from collections import defaultdict
import time

class BehavioralRateLimiter:
    def __init__(self):
        self.sessions = defaultdict(lambda: {
            'resources': set(),
            'start_time': time.time(),
            'request_count': 0
        })
    
    def is_suspicious(self, session_id, resource_id):
        sess = self.sessions[session_id]
        sess['resources'].add(resource_id)
        sess['request_count'] += 1
        
        elapsed = time.time() - sess['start_time']
        unique_resources = len(sess['resources'])
        
        # Flag if requesting >100 unique resources in <60 seconds
        if unique_resources > 100 and elapsed < 60:
            return True
            
        # Flag if unique/resources ratio approaches 1.0 (enumeration)
        if sess['request_count'] > 50:
            ratio = unique_resources / sess['request_count']
            if ratio > 0.95:
                return True
                
        return False

2. Honeypot Endpoints

Insert fake resource IDs that shouldn't be accessed. Any request to these "canary" IDs triggers an immediate block and investigation.

# Django middleware example
SUSPICIOUS_ENDPOINTS = ['/api/users/999999999', '/api/internal/test']

class HoneypotMiddleware:
    def __call__(self, request):
        if request.path in SUSPICIOUS_ENDPOINTS:
            # Log with full context, then block
            logger.warning(f"Honeypot triggered", extra={
                'ip': request.META.get('REMOTE_ADDR'),
                'user_agent': request.META.get('HTTP_USER_AGENT'),
                'headers': dict(request.headers)
            })
            return JsonResponse({'error': 'Access denied'}, status=403)
        return self.get_response(request)

3. Progressive Data Exposure

Don't return full profiles on list endpoints. Require explicit profile views that are easier to rate-limit and monitor.

// List endpoint - minimal data
{
  "users": [
    {"id": "u_123", "username": "alice", "avatar_url": "..."}
  ],
  "detail_endpoint": "/api/users/u_123/profile"
}

// Detail endpoint - full data (separately rate-limited)
{
  "id": "u_123",
  "username": "alice",
  "email": "alice@example.com",
  "phone": "+1-555-...",
  "location": "..."
}

4. GraphQL Query Complexity Analysis

If you expose GraphQL, implement query complexity scoring. Block queries that request too many nested resources.

# GraphQL complexity calculator
def calculate_complexity(node, depth=0):
    score = 1
    if depth > 3:  # Max nesting depth
        score += 50
    if hasattr(node, 'selection_set'):
        for child in node.selection_set.selections:
            score += calculate_complexity(child, depth + 1)
    return score

# Reject queries with complexity > 100
MAX_COMPLEXITY = 100

Even with defenses, scraping attempts will happen. Your detection strategy should focus on early identification:

Monitor These Metrics:

• Requests per session vs. unique resources accessed
• Geographic distribution of requests per user account
• User-Agent rotation patterns (legitimate users don't change browsers mid-session)
• API key sharing across multiple IPs
• Off-hours access patterns for "business" accounts

Automated Response Tiers:

1. Soft throttle: Add increasing delays to suspicious sessions
2. CAPTCHA challenge: Trigger for medium-risk behavior
3. Hard block: Immediate 403 for high-confidence scraping
4. Honey token: Feed fake data to identified scrapers for tracking

Don't Rely on IP Blocking Alone: Modern scrapers rotate through millions of residential IPs. Blocking by IP is whack-a-mole that creates false positives for legitimate users on shared networks.

Meta's "no breach" claim is technically defensible but ethically bankrupt. When 17.5 million users have their personal data—including emails and phone numbers—dumped on the dark web because your API allowed mass enumeration, that's a security failure by any meaningful definition.

The lesson for API designers is stark: "Public" does not mean "unprotected." If your endpoints expose user data, you have a responsibility to ensure that exposure is proportionate to legitimate use cases. Unbounded access to user profiles isn't a feature—it's a liability.

Implement behavioral rate limiting. Deploy honeypot detection. Require authentication even for "public" data. And most importantly, stop treating API scraping as a terms-of-service violation and start treating it as the data security threat it is.

Because the next 17.5 million records on BreachForums could be yours—and "no breach" won't save your reputation when they are.

Need to inspect API authentication tokens during your security assessment? Use OpSecForge's client-side JWT Decoder to analyze tokens without exposing sensitive data to third-party services.