Mirrors/the-algorithm
2
0
Fork 0
mirror of https://github.com/twitter/the-algorithm.git synced 2024-11-05 19:25:09 +01:00
Code Issues Packages Projects Releases Wiki Activity
458abe083e
the-algorithm/home-mixer
History
twitter-team 138bb51997 [cr-mixer/home-mixer] Remove getLinearRankingParams in EarlybirdTensorflowBasedSimilarityEngine 
Remove unused ranking params which are specified by services when making an Earlybird relevance search.

For cr-mixer: since we always set useTensorflowRanking = true in EarlybirdSimilarityEngineRouter, we will only ever use the TensorFlowBasedScoringFunction for ranking search results. That function doesn't rely on any of the linear params specified in getLinearRankingParams, nor the boosts because we set applyBoosts = false in the request. These parameters are therefore strictly redundant.

The parameters in home-mixer can be removed for essentially the same reason—the parameters are redundant given that we use the Tensorflow scoring function and don't apply boosts.
2023-04-04 20:39:52 -05:00
..
server/src/main/scala/com/twitter/home_mixer [cr-mixer/home-mixer] Remove getLinearRankingParams in EarlybirdTensorflowBasedSimilarityEngine 2023-04-04 20:39:52 -05:00
BUILD.bazel Twitter Recommendation Algorithm 2023-03-31 17:36:31 -05:00
README.md Twitter Recommendation Algorithm 2023-03-31 17:36:31 -05:00

README.md

Home Mixer

Home Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently powers:

  • For you - best Tweets from people you follow + recommended out-of-network content
  • Following - reverse chronological Tweets from people you follow
  • Lists - reverse chronological Tweets from List members

Home Mixer is built on Product Mixer, our custom Scala framework that facilitates building feeds of content.

Overview

The For You recommendation algorithm in Home Mixer involves the following stages:

  • Candidate Generation - fetch Tweets from various Candidate Sources. For example:
    • Earlybird Search Index
    • User Tweet Entity Graph
    • Cr Mixer
    • Follow Recommendations Service
  • Feature Hydration
    • Fetch the ~6000 features needed for ranking
  • Scoring and Ranking using ML model
  • Filters and Heuristics. For example:
    • Author Diversity
    • Content Balance (In network vs Out of Network)
    • Feedback fatigue
    • Deduplication / previously seen Tweets removal
    • Visibility Filtering (blocked, muted authors/tweets, NSFW settings)
  • Mixing - integrate Tweets with non-Tweet content
    • Ads
    • Who-to-follow modules
    • Prompts
  • Product Features and Serving
    • Conversation Modules for replies
    • Social Context
    • Timeline Navigation
    • Edited Tweets
    • Feedback options
    • Pagination and cursoring
    • Observability and logging
    • Client instructions and content marshalling

Pipeline Structure

General

Product Mixer services like Home Mixer are structured around Pipelines that split the execution into transparent and structured steps.

Requests first go to Product Pipelines, which are used to select which Mixer Pipeline or Recommendation Pipeline to run for a given request. Each Mixer or Recommendation Pipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.

Mixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together (e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines) and rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned. These pipelines also marshall candidates into a domain object and then into a transport object to return to the caller.

Candidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic operations on the Candidates, such as filtering out unwanted candidates, applying decorations, and hydrating features.

The sections below describe the high level pipeline structure (non-exhaustive) for the main Home Timeline tabs powered by Home Mixer.

For You

  • ForYouProductPipelineConfig
    • ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)
      • ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)
        • ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
          • Fetch Tweet Candidates
            • ScoredTweetsInNetworkCandidatePipelineConfig
            • ScoredTweetsCrMixerCandidatePipelineConfig
            • ScoredTweetsUtegCandidatePipelineConfig
            • ScoredTweetsFrsCandidatePipelineConfig
          • Feature Hydration and Scoring
            • ScoredTweetsScoringPipelineConfig
      • ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)
      • ForYouAdsCandidatePipelineConfig (fetch ads)
      • ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)

Following

  • FollowingProductPipelineConfig
    • FollowingMixerPipelineConfig
      • FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)
      • ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
      • FollowingAdsCandidatePipelineConfig (fetch ads)
      • FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)

Lists

  • ListTweetsProductPipelineConfig
    • ListTweetsMixerPipelineConfig
      • ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
      • ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
      • ListTweetsAdsCandidatePipelineConfig (fetch ads)