AgentSkillsCN

twscrape

twscrape

SKILL.md

TRANSLATED CONTENT:

twscrape

Python library for scraping Twitter/X data using GraphQL API with account rotation and session management.

When to use this skill

Use this skill when:

  • •Working with Twitter/X data extraction and scraping
  • •Need to bypass Twitter API limitations with account rotation
  • •Building social media monitoring or analytics tools
  • •Extracting tweets, user profiles, followers, trends from Twitter/X
  • •Need async/parallel scraping operations for large-scale data collection
  • •Looking for alternatives to official Twitter API

Quick Reference

Installation

bash
pip install twscrape

Basic Setup

python
import asyncio
from twscrape import API, gather

async def main():
    api = API()  # Uses accounts.db by default

    # Add accounts (with cookies - more stable)
    cookies = "abc=12; ct0=xyz"
    await api.pool.add_account("user1", "pass1", "email@example.com", "mail_pass", cookies=cookies)

    # Or add accounts (with login/password - less stable)
    await api.pool.add_account("user2", "pass2", "email2@example.com", "mail_pass2")
    await api.pool.login_all()

asyncio.run(main())

Common Operations

python
# Search tweets
await gather(api.search("elon musk", limit=20))

# Get user info
await api.user_by_login("xdevelopers")
user = await api.user_by_id(2244994945)

# Get user tweets
await gather(api.user_tweets(user_id, limit=20))
await gather(api.user_tweets_and_replies(user_id, limit=20))
await gather(api.user_media(user_id, limit=20))

# Get followers/following
await gather(api.followers(user_id, limit=20))
await gather(api.following(user_id, limit=20))

# Tweet operations
await api.tweet_details(tweet_id)
await gather(api.retweeters(tweet_id, limit=20))
await gather(api.tweet_replies(tweet_id, limit=20))

# Trends
await gather(api.trends("news"))

Key Features

1. Multiple API Support

  • •Search API: Standard Twitter search functionality
  • •GraphQL API: Advanced queries and data extraction
  • •Automatic switching: Based on rate limits and availability

2. Async/Await Architecture

python
# Parallel scraping
async for tweet in api.search("elon musk"):
    print(tweet.id, tweet.user.username, tweet.rawContent)

3. Account Management

  • •Add multiple accounts for rotation
  • •Automatic rate limit handling
  • •Session persistence across runs
  • •Email verification support (IMAP or manual)

4. Data Models

  • •SNScrape-compatible models
  • •Easy conversion to dict/JSON
  • •Raw API response access available

Core API Methods

Search Operations

search(query, limit, kv={})

Search tweets by query string.

Parameters:

  • •query (str): Search query (supports Twitter search syntax)
  • •limit (int): Maximum number of tweets to return
  • •kv (dict): Additional parameters (e.g., {"product": "Top"} for Top tweets)

Returns: AsyncIterator of Tweet objects

Example:

python
# Latest tweets
async for tweet in api.search("elon musk", limit=20):
    print(tweet.rawContent)

# Top tweets
await gather(api.search("python", limit=20, kv={"product": "Top"}))

User Operations

user_by_login(username)

Get user information by username.

Example:

python
user = await api.user_by_login("xdevelopers")
print(user.id, user.displayname, user.followersCount)

user_by_id(user_id)

Get user information by user ID.

followers(user_id, limit)

Get user's followers.

following(user_id, limit)

Get users that the user follows.

verified_followers(user_id, limit)

Get only verified followers.

subscriptions(user_id, limit)

Get user's Twitter Blue subscriptions.

Tweet Operations

tweet_details(tweet_id)

Get detailed information about a specific tweet.

tweet_replies(tweet_id, limit)

Get replies to a tweet.

retweeters(tweet_id, limit)

Get users who retweeted a specific tweet.

user_tweets(user_id, limit)

Get tweets from a user (excludes replies).

user_tweets_and_replies(user_id, limit)

Get tweets and replies from a user.

user_media(user_id, limit)

Get tweets with media from a user.

Other Operations

list_timeline(list_id)

Get tweets from a Twitter list.

trends(category)

Get trending topics by category.

Categories: "news", "sport", "entertainment", etc.

Account Management

Adding Accounts

With cookies (recommended):

python
cookies = "abc=12; ct0=xyz"  # String or JSON format
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", cookies=cookies)

With credentials:

python
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass")
await api.pool.login_all()

CLI Account Management

bash
# Add accounts from file
twscrape add_accounts accounts.txt username:password:email:email_password

# Login all accounts
twscrape login_accounts

# Manual email verification
twscrape login_accounts --manual

# List accounts and status
twscrape accounts

# Re-login specific accounts
twscrape relogin user1 user2

# Retry failed logins
twscrape relogin_failed

Proxy Configuration

Per-Account Proxy

python
proxy = "http://login:pass@example.com:8080"
await api.pool.add_account("user", "pass", "email@example.com", "mail_pass", proxy=proxy)

Global Proxy

python
api = API(proxy="http://login:pass@example.com:8080")

Environment Variable

bash
export TWS_PROXY=socks5://user:pass@127.0.0.1:1080
twscrape search "elon musk"

Dynamic Proxy Changes

python
api.proxy = "socks5://user:pass@127.0.0.1:1080"
doc = await api.user_by_login("elonmusk")
api.proxy = None  # Disable proxy

Priority: api.proxy > TWS_PROXY env var > account-specific proxy

CLI Usage

Search Operations

bash
twscrape search "QUERY" --limit=20
twscrape search "elon musk lang:es" --limit=20 > data.txt
twscrape search "python" --limit=20 --raw  # Raw API responses

User Operations

bash
twscrape user_by_login USERNAME
twscrape user_by_id USER_ID
twscrape followers USER_ID --limit=20
twscrape following USER_ID --limit=20
twscrape verified_followers USER_ID --limit=20
twscrape user_tweets USER_ID --limit=20

Tweet Operations

bash
twscrape tweet_details TWEET_ID
twscrape tweet_replies TWEET_ID --limit=20
twscrape retweeters TWEET_ID --limit=20

Trends

bash
twscrape trends sport
twscrape trends news

Custom Database

bash
twscrape --db custom-accounts.db <command>

Advanced Usage

Raw API Responses

python
async for response in api.search_raw("elon musk"):
    print(response.status_code, response.json())

Stopping Iteration

python
from contextlib import aclosing

async with aclosing(api.search("elon musk")) as gen:
    async for tweet in gen:
        if tweet.id < 200:
            break

Convert Models to Dict/JSON

python
user = await api.user_by_id(user_id)
user_dict = user.dict()
user_json = user.json()

Enable Debug Logging

python
from twscrape.logger import set_log_level
set_log_level("DEBUG")

Environment Variables

  • •

    TWS_PROXY: Global proxy for all accounts Example: socks5://user:pass@127.0.0.1:1080

  • •

    TWS_WAIT_EMAIL_CODE: Timeout for email verification (default: 30 seconds)

  • •

    TWS_RAISE_WHEN_NO_ACCOUNT: Raise exception when no accounts available instead of waiting Values: false, 0, true, 1 (default: false)

Rate Limits & Limitations

Rate Limits

  • •Rate limits reset every 15 minutes per endpoint
  • •Each account has separate limits for different operations
  • •Accounts automatically rotate when limits are reached

Tweet Limits

  • •user_tweets and user_tweets_and_replies return approximately 3,200 tweets maximum per user
  • •This is a Twitter/X platform limitation

Account Status

  • •Rate limits vary based on:
    • •Account age
    • •Account verification status
    • •Account activity history

Handling Rate Limits

The library automatically:

  • •Switches to next available account
  • •Waits for rate limit reset if all accounts exhausted
  • •Tracks rate limit status per endpoint

Common Patterns

Large-Scale Data Collection

python
async def collect_user_data(username):
    user = await api.user_by_login(username)

    # Collect tweets
    tweets = await gather(api.user_tweets(user.id, limit=100))

    # Collect followers
    followers = await gather(api.followers(user.id, limit=100))

    # Collect following
    following = await gather(api.following(user.id, limit=100))

    return {
        'user': user,
        'tweets': tweets,
        'followers': followers,
        'following': following
    }

Search with Filters

python
# Language filter
await gather(api.search("python lang:en", limit=20))

# Date filter
await gather(api.search("AI since:2024-01-01", limit=20))

# From specific user
await gather(api.search("from:elonmusk", limit=20))

# With media
await gather(api.search("cats filter:media", limit=20))

Batch Processing

python
async def process_users(usernames):
    tasks = []
    for username in usernames:
        task = api.user_by_login(username)
        tasks.append(task)

    users = await asyncio.gather(*tasks)
    return users

Troubleshooting

Login Issues

  • •Use cookies instead of credentials for more stable authentication
  • •Enable manual email verification with --manual flag
  • •Check email password is correct for IMAP access

Rate Limit Problems

  • •Add more accounts for better rotation
  • •Increase wait time between requests
  • •Monitor account status with twscrape accounts

No Data Returned

  • •Check account status - they may be suspended or rate limited
  • •Verify query syntax - use Twitter search syntax
  • •Try different accounts - some may have better access

Connection Issues

  • •Configure proxy if behind firewall
  • •Check network connectivity
  • •Verify Twitter/X is accessible from your location

Resources

References

For detailed API documentation and examples, see the reference files in the references/ directory:

  • •references/installation.md - Installation and setup
  • •references/api_methods.md - Complete API method reference
  • •references/account_management.md - Account configuration and management
  • •references/cli_usage.md - Command-line interface guide
  • •references/proxy_config.md - Proxy configuration options
  • •references/examples.md - Code examples and patterns

Repository: https://github.com/vladkens/twscrape Stars: 1998+ Language: Python License: MIT