Apify Scrapers

Overview

Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality.

Quick Decision Tree

What do you want to scrape?
│
├── Social Media Posts
│   ├── Twitter/X → references/twitter.md
│   │   └── Script: scripts/scrape_twitter_ai_trends.py
│   │
│   ├── Reddit → references/reddit.md
│   │   └── Script: scripts/scrape_reddit_ai_tech.py
│   │
│   ├── LinkedIn → references/linkedin.md
│   │   └── Script: scripts/scrape_linkedin_posts.py
│   │
│   ├── Instagram → references/instagram.md
│   │   └── Script: scripts/scrape_instagram.py
│   │   └── Modes: profile, posts, hashtag, reels, comments
│   │
│   ├── Facebook → references/facebook.md
│   │   └── Script: scripts/scrape_facebook.py
│   │   └── Modes: page, posts, reviews, groups, marketplace
│   │
│   ├── TikTok → references/multi-platform.md
│   │   └── Script: scripts/scrape_multi_platform.py
│   │
│   └── YouTube → references/multi-platform.md
│       └── Script: scripts/scrape_multi_platform.py
│
├── Business/Places
│   ├── Google Maps businesses → references/google-maps.md
│   │   └── Script: scripts/scrape_google_maps.py
│   │   └── Modes: search, place, reviews
│   │
│   └── Contact info from websites → references/contact-enrichment.md
│       └── Script: scripts/scrape_contact_info.py
│       └── Extract: emails, phone numbers, social profiles
│
├── Auto-detect URL type → references/url-detect.md
│   └── Script: scripts/scrape_content_by_url.py
│
├── Trend Analysis (NEW)
│   └── Enriched trend analysis → workflows/trend-analysis.md
│       └── Script: scripts/analyze_trends.py
│       └── Features: velocity scoring, lifecycle staging, opportunity scoring
│
└── Workflows (multi-step)
    ├── Lead generation → workflows/lead-generation.md
    ├── Influencer discovery → workflows/influencer-discovery.md
    ├── Competitor analysis → workflows/competitor-intel.md
    ├── Trend analysis → workflows/trend-analysis.md
    └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md
        └── Script: scripts/scrape_competitor_ads.py
        └── Platforms: Facebook Ads Library, Google Ads Transparency
        └── Features: Spend estimates, creative analysis, benchmarking

Environment Setup

# Required in .env
APIFY_TOKEN=apify_api_xxxxx

Get your API key: https://console.apify.com/account/integrations

Common Usage Patterns

Scrape Twitter Trends

python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50

Scrape Reddit Discussions

python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100

Scrape LinkedIn Author

python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30

Auto-detect and Scrape URL

python scripts/scrape_content_by_url.py "https://x.com/user/status/123456"

Scrape Instagram Profile

python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20

Scrape Instagram Hashtag

python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50

Scrape Instagram Reels

python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30

Scrape Facebook Page

python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50

Scrape Facebook Reviews

python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100

Scrape Facebook Marketplace

python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30

Scrape Google Maps Businesses

python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50

Scrape Google Maps Reviews

python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100

Extract Contact Info from Websites

python scripts/scrape_contact_info.py "https://example.com" --depth 2

Bulk Contact Enrichment

python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json

Scrape Competitor Ads (Single Competitor)

python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30

Compare Multiple Competitors' Ads

python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json

Discover Advertisers by Keyword

python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200

Filter Competitor Ads by Media Type

python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7

Analyze Trends (NEW)

# Analyze specific topic with enrichments
python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90

# Discover trending topics in category
python scripts/analyze_trends.py --category technology --discover --top 50

# Compare multiple trends
python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare

# Export HTML trend report
python scripts/analyze_trends.py "sustainable fashion" --format html --output trend_report.html

Cost Estimates

Output Location

All scraped data saves to .tmp/ with timestamped filenames:

• •.tmp/twitter_ai_trends_YYYYMMDD.json
• •.tmp/reddit_ai_tech_YYYYMMDD.json
• •.tmp/linkedin_posts_YYYYMMDD_HHMMSS.json

Security Notes

Credential Handling

• •Store APIFY_TOKEN in .env file (never commit to git)
• •Rotate API tokens periodically via Apify Console
• •Never log or print API tokens in script output
• •Use environment variables, not hardcoded values

Data Privacy

• •Scraped data contains only publicly available content
• •Social media posts may include PII (names, handles, profile info)
• •Data is stored locally in .tmp/ directory
• •No data is retained by Apify after actor run completes
• •Consider data minimization - only scrape what you need

Access Scopes

• •Apify tokens have full account access (no granular scopes)
• •Use separate Apify accounts for different projects if needed
• •Monitor usage via Apify Console dashboard

Compliance Considerations

• •Terms of Service: Respect each platform's ToS (Twitter, Reddit, LinkedIn)
• •Rate Limiting: Actors have built-in rate limiting to avoid bans
• •Robots.txt: Some actors may bypass robots.txt - use responsibly
• •GDPR: Scraped PII may be subject to GDPR if EU residents
• •Ethical Use: Only scrape public data; never bypass authentication
• •Proxy Ethics: Residential proxies should be used ethically

Troubleshooting

Common Issues

Issue: Actor run failed

Symptoms: Script terminates with "Actor run failed" or timeout error Cause: Invalid actor ID, insufficient proxy credits, or actor configuration issue Solution:

• •Verify the actor ID is correct in the script
• •Check Apify Console for actor run logs
• •Ensure proxy settings match actor requirements
• •Try running with default proxy settings first

Issue: Empty results returned

Symptoms: Script completes but returns 0 items Cause: Content blocked by platform, invalid query, or proxy being detected Solution:

• •Try a different proxy type (residential vs datacenter)
• •Simplify the search query
• •Reduce the number of results requested
• •Check if the platform is blocking scraping attempts

Issue: Rate limited by platform

Symptoms: Script fails with 429 errors or "rate limited" messages Cause: Too many requests in a short time period Solution:

• •Add delays between requests (actor settings)
• •Reduce concurrent requests
• •Use proxy rotation
• •Wait and retry after a cooldown period

Issue: Invalid API token

Symptoms: Authentication error or "invalid token" message Cause: Token expired, revoked, or incorrectly set Solution:

• •Regenerate API token in Apify Console
• •Verify token is correctly set in .env file
• •Check for leading/trailing whitespace in token
• •Ensure APIFY_TOKEN environment variable is loaded

Issue: Proxy connection errors

Symptoms: Connection timeout or proxy errors Cause: Proxy pool exhausted or geo-restriction issues Solution:

• •Switch proxy type (basic, residential, or datacenter)
• •Verify proxy credit balance in Apify Console
• •Try a different proxy country/region
• •Disable proxy to test if that's the root cause

Resources

Platform References

• •references/twitter.md - Twitter/X scraping details
• •references/reddit.md - Reddit scraping with subreddit targeting
• •references/linkedin.md - LinkedIn post scraping (author or search mode)
• •references/instagram.md - Instagram profile, posts, hashtag, reels, and comments scraping
• •references/facebook.md - Facebook page, posts, reviews, groups, and marketplace scraping
• •references/multi-platform.md - TikTok and YouTube scraping
• •references/url-detect.md - Auto-detect URL type and scrape

Business/Places References

• •references/google-maps.md - Google Maps business search, place details, and reviews
• •references/contact-enrichment.md - Extract emails, phone numbers, and social profiles from websites

Workflow References

• •workflows/lead-generation.md - Multi-step lead generation workflow
• •workflows/influencer-discovery.md - Find and analyze influencers across platforms
• •workflows/competitor-intel.md - Competitive intelligence gathering workflow
• •workflows/trend-analysis.md - Enriched multi-platform trend analysis with scoring

Integration Patterns

Scrape and Enrich

Skills: apify-scrapers → parallel-research Use case: Scrape social media posts, then enrich with deep research Flow:

1. •Scrape Twitter/Reddit for mentions of a topic
2. •Extract company names or URLs from posts
3. •Use parallel-research to get detailed info on each company

Scrape and Summarize

Skills: apify-scrapers → content-generation Use case: Create newsletter content from social media trends Flow:

1. •Scrape trending AI posts from Twitter
2. •Pass scraped data to content-generation summarize
3. •Generate a formatted newsletter section

Scrape and Archive

Skills: apify-scrapers → google-workspace Use case: Save scraped data to Google Drive for team access Flow:

1. •Scrape LinkedIn posts from target accounts
2. •Format data as CSV or JSON
3. •Upload to Google Drive client folder via google-workspace

Trend Analysis + Content Strategy

Skills: apify-scrapers (trend-analysis) → content-generation Use case: Identify trending topics and create content strategy Flow:

1. •Run trend analysis: python scripts/analyze_trends.py "AI productivity" --sources all
2. •Review lifecycle stage and opportunity score
3. •Use content-generation to create content for high-opportunity trends
4. •Focus on emerging trends with high velocity scores

Competitive Trend Monitoring

Skills: apify-scrapers (trend-analysis) → parallel-research Use case: Monitor competitor visibility in trending topics Flow:

1. •Analyze industry trends: python scripts/analyze_trends.py --category "your-industry" --discover
2. •Compare your brand vs competitors in those trends
3. •Use parallel-research for deep dive on gaps
4. •Generate competitive intelligence report