X Algorithm Pipeline
The X recommendation algorithm processes posts through an 8-stage pipeline to generate the "For You" feed. Each stage transforms, filters, or scores the candidate posts.
Pipeline Overview
code
┌─────────────────────────────────────────────────────────────────────────────┐ │ X RECOMMENDATION PIPELINE │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ User Request │ │ │ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 1. Query │ Hydrate user features, action history, socialgraph │ │ │ Hydration │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ Thunder (in-network) + Phoenix (out-of-network) │ │ │ 2. Sources │ In-network: Posts from followed accounts │ │ │ │ Out-of-network: ML retrieval from all posts │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 3. Candidate│ Fetch tweet text, author data, visibility status │ │ │ Hydration │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 4. Pre-Score│ Age, duplicates, safety, blocked authors │ │ │ Filtering │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ Phoenix ML → WeightedScorer → AuthorDiversity → OON │ │ │ 5. Scoring │ Each scorer adds/adjusts candidate.score │ │ │ │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 6. Selection│ TopKScoreSelector: Keep top N by final score │ │ │ │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 7. Post- │ Conversation dedup, previously seen, keywords │ │ │ Filtering │ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │ 8. Side │ Logging, analytics, impression tracking │ │ │ Effects │ │ │ └──────┬──────┘ │ │ ▼ │ │ Feed Response │ │ │ └─────────────────────────────────────────────────────────────────────────────┘
Stage Details
1. Query Hydration
Enriches the request with user context:
- •User features (followed users, blocked users, muted users)
- •User action sequence (engagement history for ML)
- •Muted keywords
- •Subscription status
- •Bloom filters for seen posts
2. Sources
Two candidate sources provide posts:
Thunder Source (In-Network)
rust
// home-mixer/sources/thunder_source.rs // Posts from accounts the user follows served_type: Some(pb::ServedType::ForYouInNetwork)
- •Queries Thunder service with user's following list
- •Returns recent posts from followed accounts
- •Includes conversation context (ancestors, reply chains)
Phoenix Source (Out-of-Network)
rust
// home-mixer/sources/phoenix_source.rs
fn enable(&self, query: &ScoredPostsQuery) -> bool {
!query.in_network_only // Disabled for "Following" tab
}
served_type: Some(pb::ServedType::ForYouPhoenixRetrieval)
- •ML-based retrieval using user embedding
- •Finds relevant posts from the entire corpus
- •Enabled for "For You", disabled for "Following"
3. Candidate Hydration
Fetches full post data:
- •Tweet text content
- •Author information
- •Media metadata (video duration)
- •Visibility filtering results
- •Subscription requirements
4. Pre-Score Filtering
Removes ineligible candidates before expensive ML scoring:
- •
AgeFilter- Too old - •
DropDuplicatesFilter- Duplicate IDs - •
VFFilter- Safety violations - •
AuthorSocialgraphFilter- Blocked/muted authors - •
CoreDataHydrationFilter- Missing data - •
IneligibleSubscriptionFilter- Subscription required
5. Scoring (4 Stages)
a) PhoenixScorer
rust
// home-mixer/scorers/phoenix_scorer.rs // Calls Phoenix ML to predict engagement probabilities
Produces phoenix_scores with 18 action probabilities.
b) WeightedScorer
rust
// home-mixer/scorers/weighted_scorer.rs // Combines probabilities into single score weighted_score = Σ(weight × P(action))
Produces weighted_score from action predictions.
c) AuthorDiversityScorer
rust
// home-mixer/scorers/author_diversity_scorer.rs // Penalizes multiple posts from same author multiplier = (1 - floor) × decay^position + floor
Adjusts scores to promote variety.
d) OONScorer
rust
// home-mixer/scorers/oon_scorer.rs // Adjusts out-of-network post scores if !in_network: score *= OON_WEIGHT_FACTOR
Balances in-network vs out-of-network content.
6. Selection
rust
// home-mixer/selectors/top_k_score_selector.rs
pub struct TopKScoreSelector;
impl Selector<ScoredPostsQuery, PostCandidate> for TopKScoreSelector {
fn score(&self, candidate: &PostCandidate) -> f64 {
candidate.score.unwrap_or(f64::NEG_INFINITY)
}
fn size(&self) -> Option<usize> {
Some(params::TOP_K_CANDIDATES_TO_SELECT)
}
}
Keeps top K posts by final score.
7. Post-Score Filtering
Fine-grained filtering after selection:
- •
DedupConversationFilter- One post per conversation - •
RetweetDeduplicationFilter- One version per underlying post - •
PreviouslySeenPostsFilter- Remove seen posts - •
PreviouslyServedPostsFilter- Remove from current session - •
MutedKeywordFilter- User keyword mutes - •
SelfTweetFilter- Remove own posts
8. Side Effects
Non-blocking operations after response:
- •Impression logging
- •Analytics events
- •Cache updates
Data Flow Summary
code
Candidates start with: ├── tweet_id, author_id (from Sources) ├── tweet_text, metadata (from Hydration) ├── phoenix_scores (from PhoenixScorer) ├── weighted_score (from WeightedScorer) ├── score (from AuthorDiversity + OON) └── Final ranking by score
PostCandidate Structure
rust
pub struct PostCandidate {
pub tweet_id: i64,
pub author_id: u64,
pub tweet_text: String,
pub in_reply_to_tweet_id: Option<u64>,
pub retweeted_tweet_id: Option<u64>,
pub retweeted_user_id: Option<u64>,
pub phoenix_scores: PhoenixScores, // ML predictions
pub weighted_score: Option<f64>, // After WeightedScorer
pub score: Option<f64>, // Final score
pub served_type: Option<ServedType>, // Source type
pub in_network: Option<bool>, // Following or not
pub ancestors: Vec<u64>, // Conversation context
pub video_duration_ms: Option<i32>, // For VQV eligibility
pub visibility_reason: Option<FilteredReason>,
pub subscription_author_id: Option<u64>,
// ...
}
Source Configuration
| Tab | Thunder (In-Network) | Phoenix (Out-of-Network) |
|---|---|---|
| For You | Enabled | Enabled |
| Following | Enabled | Disabled |
Related Skills
- •
/x-algo-scoring- Detailed scoring formula - •
/x-algo-filters- All filter implementations - •
/x-algo-engagement- Action types and signals - •
/x-algo-ml- Phoenix ML model architecture