Mirrors/the-algorithm
2
0
Fork 0
mirror of https://github.com/twitter/the-algorithm.git synced 2024-12-22 18:21:51 +01:00
Code Issues Packages Projects Releases Wiki Activity
4951dc7367
the-algorithm/README.md

72 lines
7.2 KiB
Markdown
Raw Normal View History

[minor] Fix grammar + typo issues Closes #557, closes #678, closes #748, closes #806, closes #818, closes #842, closes #866, closes #948, closes #1024, closes #1313, closes #1458, closes #1461, closes #1465, closes #1491, closes #1503, closes #1539, closes #1611
2023-04-04 23:13:24 +02:00
# Twitter's Recommendation Algorithm
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
README updates - renames pushservice readme.md to README.md - Minor changes to main README.md
2023-05-19 23:36:39 +02:00
Twitter's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of Tweets and other content across all Twitter product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
## Architecture
Product surfaces at Twitter are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:
| Type | Component | Description |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
|------|-----------|-------------|
README updates - renames pushservice readme.md to README.md - Minor changes to main README.md
2023-05-19 23:36:39 +02:00
| Data | [tweetypie](tweetypie/server/README.md) | Core Tweet service that handles the reading and writing of Tweet data. |
| | [unified-user-actions](unified_user_actions/README.md) | Real-time stream of user actions on Twitter. |
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
| | [user-signal-service](user-signal-service/README.md) | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. |
| Model | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |
| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets. |
| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |
| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of a Twitter User interacting with another User. |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README.md) | Page-Rank algorithm for calculating Twitter User reputation. |
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |
| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). |
| | [topic-social-proof](topic-social-proof/README.md) | Identifies topics related to individual Tweets. |
README updates: representation-manager and representation-scorer
2023-04-28 20:09:29 +02:00
| | [representation-scorer](representation-scorer/README.md) | Compute scores between pairs of entities (Users, Tweets, etc.) using embedding similarity. |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
| Software framework | [navi](navi/README.md) | High-performance, machine learning model serving written in Rust. |
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
| | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content. |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
| | [timelines-aggregation-framework](timelines/data_processing/ml_util/aggregation_framework/README.md) | Framework for generating aggregate features in batch or real-time. |
| | [representation-manager](representation-manager/README.md) | Service to retrieve embeddings (i.e. SimClusters and TwHIN). |
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
| | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1. |
README updates - renames pushservice readme.md to README.md - Minor changes to main README.md
2023-05-19 23:36:39 +02:00
The product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
### For You Timeline
The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
![System Diagram](docs/system-diagram.png)
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
The core components of the For You Timeline included in this repository are listed below:
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
| Type | Component | Description |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
|------|-----------|-------------|
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network Tweets. ~50% of Tweets come from this candidate source. |
| | [cr-mixer](cr-mixer/README.md) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
| | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG) | Maintains an in-memory User to Tweet interaction graph, and finds candidates based on traversals of this graph. This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework. Several other GraphJet-based features and candidate sources are located [here](src/scala/com/twitter/recos). |
| | [follow-recommendation-service](follow-recommendations-service/README.md) (FRS) | Provides Users with recommendations for accounts to follow, and Tweets from those accounts. |
[minor] Fix grammar + typo issues Closes #557, closes #678, closes #748, closes #806, closes #818, closes #842, closes #866, closes #948, closes #1024, closes #1313, closes #1458, closes #1461, closes #1465, closes #1491, closes #1503, closes #1539, closes #1611
2023-04-04 23:13:24 +02:00
| Ranking | [light-ranker](src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/README.md) | Light Ranker model used by search index (Earlybird) to rank Tweets. |
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
| | [heavy-ranker](https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/README.md) | Neural network for ranking candidate tweets. One of the main signals used to select timeline Tweets post candidate sourcing. |
[minor] Fix grammar + typo issues Closes #557, closes #678, closes #748, closes #806, closes #818, closes #842, closes #866, closes #948, closes #1024, closes #1313, closes #1458, closes #1461, closes #1465, closes #1491, closes #1503, closes #1539, closes #1611
2023-04-04 23:13:24 +02:00
| Tweet mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline. Built on [product-mixer](product-mixer/README.md). |
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
| | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |
| | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service. |
Open-sourcing pushservice Pushservice is the main recommendation service we use to surface recommendations to our users via notifications. It fetches candidates from various sources, ranks them in order of relevance, and applies filters to determine the best one to send.
2023-05-19 18:53:56 +02:00
### Recommended Notifications
README updates - renames pushservice readme.md to README.md - Minor changes to main README.md
2023-05-19 23:36:39 +02:00
The core components of Recommended Notifications included in this repository are listed below:
Open-sourcing pushservice Pushservice is the main recommendation service we use to surface recommendations to our users via notifications. It fetches candidates from various sources, ranks them in order of relevance, and applies filters to determine the best one to send.
2023-05-19 18:53:56 +02:00
| Type | Component | Description |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
|------|-----------|-------------|
| Service | [pushservice](pushservice/README.md) | Main recommendation service at Twitter used to surface recommendations to our users via notifications. |
Open-sourcing pushservice Pushservice is the main recommendation service we use to surface recommendations to our users via notifications. It fetches candidates from various sources, ranks them in order of relevance, and applies filters to determine the best one to send.
2023-05-19 18:53:56 +02:00
| Ranking | [pushservice-light-ranker](pushservice/src/main/python/models/light_ranking/README.md) | Light Ranker model used by pushservice to rank Tweets. Bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool. |
| | [pushservice-heavy-ranker](pushservice/src/main/python/models/heavy_ranking/README.md) | Multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. |
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
## Build and Test Code
[opensource] Update README to include all new modules Since the first batch of open sourcing, we have added the following components: - User signal service - Unified user actions - Topic social proof service Update the README to include these.
2023-04-14 22:55:59 +02:00
We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
## Contributing
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
We invite the community to submit GitHub issues and pull requests for suggestions on improving the recommendation algorithm. We are working on tools to manage these suggestions and sync changes to our internal repository. Any security concerns or issues should be routed to our official [bug bounty program](https://hackerone.com/twitter) through HackerOne. We hope to benefit from the collective intelligence and expertise of the global community in helping us identify issues
Twitter Recommendation Algorithm Please note we have force-pushed a new initial commit in order to remove some publicly-available Twitter user information. Note that this process may be required in the future.
2023-04-01 00:36:31 +02:00
Update README.md Sure, here are the changes made to the original content: 1. Corrected typos in words like "erro's" to "errors," "twweetypie" to "tweetypie," "tweetks" to "Tweets," and others as listed above. 2. Formatted headings to be consistent and capitalized appropriately. 3. Formatted URLs properly, e.g., "recos-injector" to "[recos-injector](recos-injector/README.md)." 4. Fixed the description for some components in the table to be more accurate. 5. Added hyphens to some component names in the table for consistency. 6. Corrected the URL for "the-algorithm-ml" in the table. 7. Corrected the indentation of the system diagram image for proper display. 8. Added missing closing punctuation at the end of sentences. 9. Formatted lists and paragraphs for proper Markdown formatting. 10. Capitalized "For" in the prompt for consistency. 11. Linked the blog URLs to direct blog posts for better readability. Please note that the changes focused on correcting errors, improving consistency, and formatting the content according to Markdown conventions. The overall meaning and content of the document remain unchanged.
2023-07-20 22:34:58 +02:00
and suggest improvements, ultimately leading to a better Twitter.
Read our blog on the open-source initiative [here](https://blog.twitter.com/en_us/topics/company/2023/a-new-era-of-transparency-for-twitter).
Reference in New Issue Copy Permalink