Mirrors/the-algorithm
2
0
Fork 0
mirror of https://github.com/twitter/the-algorithm.git synced 2024-12-22 02:01:51 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'twitter:main' into main

Browse Source
This commit is contained in:
ibuki420 2023-07-25 20:18:36 +09:00 committed by GitHub
commit 20889ae89c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2051 changed files with 181443 additions and 11185 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
README.md
View File

@ -1,36 +1,66 @@
# Twitter Recommendation Algorithm
# Twitter's Recommendation Algorithm
The Twitter Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving the
Home Timeline. For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm). The
diagram below illustrates how major services and jobs interconnect.
Twitter's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of Tweets and other content across all Twitter product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).
![](docs/system-diagram.png)
## Architecture
These are the main components of the Recommendation Algorithm included in this repository:
Product surfaces at Twitter are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Feature | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |
| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets. |
| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |
| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict likelihood of a Twitter User interacting with another User. |
| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating Twitter User reputation. |
| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |
| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). |
| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network Tweets. ~50% of Tweets come from this candidate source. |
| | [cr-mixer](cr-mixer/README.md) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |
| | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG)| Maintains an in memory User to Tweet interaction graph, and finds candidates based on traversals of this graph. This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework. Several other GraphJet based features and candidate sources are located [here](src/scala/com/twitter/recos) |
| | [follow-recommendation-service](follow-recommendations-service/README.md) (FRS)| Provides Users with recommendations for accounts to follow, and Tweets from those accounts. |
| Ranking | [light-ranker](src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/README.md) | Light ranker model used by search index (Earlybird) to rank Tweets. |
| | [heavy-ranker](https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/README.md) | Neural network for ranking candidate tweets. One of the main signals used to select timeline Tweets post candidate sourcing. |
| Tweet mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline. Built on [product-mixer](product-mixer/README.md) |
| | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |
| | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service. |
| Software framework | [navi](navi/navi/README.md) | High performance, machine learning model serving written in Rust. |
| Data | [tweetypie](tweetypie/server/README.md) | Core Tweet service that handles the reading and writing of Tweet data. |
| | [unified-user-actions](unified_user_actions/README.md) | Real-time stream of user actions on Twitter. |
| | [user-signal-service](user-signal-service/README.md) | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. |
| Model | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |
| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets. |
| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |
| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of a Twitter User interacting with another User. |
| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating Twitter User reputation. |
| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |
| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). |
| | [topic-social-proof](topic-social-proof/README.md) | Identifies topics related to individual Tweets. |
| | [representation-scorer](representation-scorer/README.md) | Compute scores between pairs of entities (Users, Tweets, etc.) using embedding similarity. |
| Software framework | [navi](navi/README.md) | High performance, machine learning model serving written in Rust. |
| | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content. |
| | [timelines-aggregation-framework](timelines/data_processing/ml_util/aggregation_framework/README.md) | Framework for generating aggregate features in batch or real time. |
| | [representation-manager](representation-manager/README.md) | Service to retrieve embeddings (i.e. SimClusers and TwHIN). |
| | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1. |
We include Bazel BUILD files for most components, but not a top level BUILD or WORKSPACE file.
The product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.
### For You Timeline
The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.
![](docs/system-diagram.png)
The core components of the For You Timeline included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network Tweets. ~50% of Tweets come from this candidate source. |
| | [cr-mixer](cr-mixer/README.md) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |
| | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG)| Maintains an in memory User to Tweet interaction graph, and finds candidates based on traversals of this graph. This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework. Several other GraphJet based features and candidate sources are located [here](src/scala/com/twitter/recos). |
| | [follow-recommendation-service](follow-recommendations-service/README.md) (FRS)| Provides Users with recommendations for accounts to follow, and Tweets from those accounts. |
| Ranking | [light-ranker](src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/README.md) | Light Ranker model used by search index (Earlybird) to rank Tweets. |
| | [heavy-ranker](https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/README.md) | Neural network for ranking candidate tweets. One of the main signals used to select timeline Tweets post candidate sourcing. |
| Tweet mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline. Built on [product-mixer](product-mixer/README.md). |
| | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |
| | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service. |
### Recommended Notifications
The core components of Recommended Notifications included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Service | [pushservice](pushservice/README.md) | Main recommendation service at Twitter used to surface recommendations to our users via notifications.
| Ranking | [pushservice-light-ranker](pushservice/src/main/python/models/light_ranking/README.md) | Light Ranker model used by pushservice to rank Tweets. Bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool. |
| | [pushservice-heavy-ranker](pushservice/src/main/python/models/heavy_ranking/README.md) | Multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. |
## Build and test code
We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.
## Contributing

51
RETREIVAL_SIGNALS.md Normal file
View File

@ -0,0 +1,51 @@
# Signals for Candidate Sources
## Overview
The candidate sourcing stage within the Twitter Recommendation algorithm serves to significantly narrow down the item size from approximately 1 billion to just a few thousand. This process utilizes Twitter user behavior as the primary input for the algorithm. This document comprehensively enumerates all the signals during the candidate sourcing phase.
| Signals | Description |
| :-------------------- | :-------------------------------------------------------------------- |
| Author Follow | The accounts which user explicit follows. |
| Author Unfollow | The accounts which user recently unfollows. |
| Author Mute | The accounts which user have muted. |
| Author Block | The accounts which user have blocked |
| Tweet Favorite | The tweets which user clicked the like botton. |
| Tweet Unfavorite | The tweets which user clicked the unlike botton. |
| Retweet | The tweets which user retweeted |
| Quote Tweet | The tweets which user retweeted with comments. |
| Tweet Reply | The tweets which user replied. |
| Tweet Share | The tweets which user clicked the share botton. |
| Tweet Bookmark | The tweets which user clicked the bookmark botton. |
| Tweet Click | The tweets which user clicked and viewed the tweet detail page. |
| Tweet Video Watch | The video tweets which user watched certain seconds or percentage. |
| Tweet Don't like | The tweets which user clicked "Not interested in this tweet" botton. |
| Tweet Report | The tweets which user clicked "Report Tweet" botton. |
| Notification Open | The push notification tweets which user opened. |
| Ntab click | The tweets which user click on the Notifications page. |
| User AddressBook | The author accounts identifiers of the user's addressbook. |
## Usage Details
Twitter uses these user signals as training labels and/or ML features in the each candidate sourcing algorithms. The following tables shows how they are used in the each components.
| Signals | USS | SimClusters | TwHin | UTEG | FRS | Light Ranking |
| :-------------------- | :----------------- | :----------------- | :----------------- | :----------------- | :----------------- | :----------------- |
| Author Follow | Features | Features / Labels | Features / Labels | Features | Features / Labels | N/A |
| Author Unfollow | Features | N/A | N/A | N/A | N/A | N/A |
| Author Mute | Features | N/A | N/A | N/A | Features | N/A |
| Author Block | Features | N/A | N/A | N/A | Features | N/A |
| Tweet Favorite | Features | Features | Features / Labels | Features | Features / Labels | Features / Labels |
| Tweet Unfavorite | Features | Features | N/A | N/A | N/A | N/A |
| Retweet | Features | N/A | Features / Labels | Features | Features / Labels | Features / Labels |
| Quote Tweet | Features | N/A | Features / Labels | Features | Features / Labels | Features / Labels |
| Tweet Reply | Features | N/A | Features | Features | Features / Labels | Features |
| Tweet Share | Features | N/A | N/A | N/A | Features | N/A |
| Tweet Bookmark | Features | N/A | N/A | N/A | N/A | N/A |
| Tweet Click | Features | N/A | N/A | N/A | Features | Labels |
| Tweet Video Watch | Features | Features | N/A | N/A | N/A | Labels |
| Tweet Don't like | Features | N/A | N/A | N/A | N/A | N/A |
| Tweet Report | Features | N/A | N/A | N/A | N/A | N/A |
| Notification Open | Features | Features | Features | N/A | Features | N/A |
| Ntab click | Features | Features | Features | N/A | Features | N/A |
| User AddressBook | N/A | N/A | N/A | N/A | Features | N/A |

2
ann/src/main/python/dataflow/faiss_index_bq_dataset.py
View File

@ -91,7 +91,7 @@ def parse_metric(config):
elif metric_str == "linf":
return faiss.METRIC_Linf
else:
raise Exception(f"Uknown metric: {metric_str}")
raise Exception(f"Unknown metric: {metric_str}")
def run_pipeline(argv=[]):

4
cr-mixer/README.md
View File

@ -2,6 +2,6 @@
CR-Mixer is a candidate generation service proposed as part of the Personalization Strategy vision for Twitter. Its aim is to speed up the iteration and development of candidate generation and light ranking. The service acts as a lightweight coordinating layer that delegates candidate generation tasks to underlying compute services. It focuses on Twitter's candidate generation use cases and offers a centralized platform for fetching, mixing, and managing candidate sources and light rankers. The overarching goal is to increase the speed and ease of testing and developing candidate generation pipelines, ultimately delivering more value to Twitter users.
CR-Mixer act as a configurator and delegator, providing abstractions for the challenging parts of candidate generation and handling performance issues. It will offer a 1-stop-shop for fetching and mixing candidate sources, a managed and shared performant platform, a light ranking layer, a common filtering layer, a version control system, a co-owned feature switch set, and peripheral tooling.
CR-Mixer acts as a configurator and delegator, providing abstractions for the challenging parts of candidate generation and handling performance issues. It will offer a 1-stop-shop for fetching and mixing candidate sources, a managed and shared performant platform, a light ranking layer, a common filtering layer, a version control system, a co-owned feature switch set, and peripheral tooling.
CR-Mixer's pipeline consists of 4 steps: source signal extraction, candidate generation, filtering, and ranking. It also provides peripheral tooling like scribing, debugging, and monitoring. The service fetches source signals externally from stores like UserProfileService and RealGraph, calls external candidate generation services, and caches results. Filters are applied for deduping and pre-ranking, and a light ranking step follows.
CR-Mixer's pipeline consists of 4 steps: source signal extraction, candidate generation, filtering, and ranking. It also provides peripheral tooling like scribing, debugging, and monitoring. The service fetches source signals externally from stores like UserProfileService and RealGraph, calls external candidate generation services, and caches results. Filters are applied for deduping and pre-ranking, and a light ranking step follows.

43
cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdTensorflowBasedSimilarityEngine.scala
View File

@ -6,8 +6,6 @@ import com.twitter.search.earlybird.thriftscala.EarlybirdService
import com.twitter.search.earlybird.thriftscala.ThriftSearchQuery
import com.twitter.util.Time
import com.twitter.search.common.query.thriftjava.thriftscala.CollectorParams
import com.twitter.search.common.ranking.thriftscala.ThriftAgeDecayRankingParams
import com.twitter.search.common.ranking.thriftscala.ThriftLinearFeatureRankingParams
import com.twitter.search.common.ranking.thriftscala.ThriftRankingParams
import com.twitter.search.common.ranking.thriftscala.ThriftScoringFunctionType
import com.twitter.search.earlybird.thriftscala.ThriftSearchRelevanceOptions
@ -97,7 +95,7 @@ object EarlybirdTensorflowBasedSimilarityEngine {
// Whether to collect conversation IDs. Remove it for now.
// collectConversationId = Gate.True(), // true for Home
rankingMode = ThriftSearchRankingMode.Relevance,
relevanceOptions = Some(getRelevanceOptions(query.useTensorflowRanking)),
relevanceOptions = Some(getRelevanceOptions),
collectorParams = Some(
CollectorParams(
// numResultsToReturn defines how many results each EB shard will return to search root
@ -116,13 +114,11 @@ object EarlybirdTensorflowBasedSimilarityEngine {
// The specific values of recap relevance/reranking options correspond to
// experiment: enable_recap_reranking_2988,timeline_internal_disable_recap_filter
// bucket : enable_rerank,disable_filter
private def getRelevanceOptions(useTensorflowRanking: Boolean): ThriftSearchRelevanceOptions = {
private def getRelevanceOptions: ThriftSearchRelevanceOptions = {
ThriftSearchRelevanceOptions(
proximityScoring = true,
maxConsecutiveSameUser = Some(2),
rankingParams =
if (useTensorflowRanking) Some(getTensorflowBasedRankingParams)
else Some(getLinearRankingParams),
rankingParams = Some(getTensorflowBasedRankingParams),
maxHitsToProcess = Some(500),
maxUserBlendCount = Some(3),
proximityPhraseWeight = 9.0,
@ -131,41 +127,12 @@ object EarlybirdTensorflowBasedSimilarityEngine {
}
private def getTensorflowBasedRankingParams: ThriftRankingParams = {
getLinearRankingParams.copy(
ThriftRankingParams(
`type` = Some(ThriftScoringFunctionType.TensorflowBased),
selectedTensorflowModel = Some("timelines_rectweet_replica"),
minScore = -1.0e100,
applyBoosts = false,
authorSpecificScoreAdjustments = None
)
}
private def getLinearRankingParams: ThriftRankingParams = {
ThriftRankingParams(
`type` = Some(ThriftScoringFunctionType.Linear),
minScore = -1.0e100,
retweetCountParams = Some(ThriftLinearFeatureRankingParams(weight = 20.0)),
replyCountParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
reputationParams = Some(ThriftLinearFeatureRankingParams(weight = 0.2)),
luceneScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)),
textScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 0.18)),
urlParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)),
isReplyParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
favCountParams = Some(ThriftLinearFeatureRankingParams(weight = 30.0)),
langEnglishUIBoost = 0.5,
langEnglishTweetBoost = 0.2,
langDefaultBoost = 0.02,
unknownLanguageBoost = 0.05,
offensiveBoost = 0.1,
inTrustedCircleBoost = 3.0,
multipleHashtagsOrTrendsBoost = 0.6,
inDirectFollowBoost = 4.0,
tweetHasTrendBoost = 1.1,
selfTweetBoost = 2.0,
tweetHasImageUrlBoost = 2.0,
tweetHasVideoUrlBoost = 2.0,
useUserLanguageInfo = true,
ageDecayParams = Some(ThriftAgeDecayRankingParams(slope = 0.005, base = 1.0))
)
}
}

3
home-mixer/README.md
View File

@ -74,7 +74,7 @@ Timeline tabs powered by Home Mixer.
- ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
- Fetch Tweet Candidates
- ScoredTweetsInNetworkCandidatePipelineConfig
- ScoredTweetsCrMixerCandidatePipelineConfig
- ScoredTweetsTweetMixerCandidatePipelineConfig
- ScoredTweetsUtegCandidatePipelineConfig
- ScoredTweetsFrsCandidatePipelineConfig
- Feature Hydration and Scoring
@ -99,4 +99,3 @@ Timeline tabs powered by Home Mixer.
- ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
- ListTweetsAdsCandidatePipelineConfig (fetch ads)

5
home-mixer/server/src/main/scala/com/twitter/home_mixer/BUILD.bazel
View File

@ -21,6 +21,7 @@ scala_library(
"finatra/inject/inject-utils/src/main/scala",
"home-mixer/server/src/main/resources",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/controller",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/federated",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/module",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product",
@ -31,6 +32,10 @@ scala_library(
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter",
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
"src/thrift/com/twitter/timelines/render:thrift-scala",
"strato/config/columns/auth-context:auth-context-strato-client",
"strato/config/columns/gizmoduck:gizmoduck-strato-client",
"strato/src/main/scala/com/twitter/strato/fed",
"strato/src/main/scala/com/twitter/strato/fed/server",
"stringcenter/client",
"stringcenter/client/src/main/java",
"stringcenter/client/src/main/scala/com/twitter/stringcenter/client",

2
home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerHttpServerWarmupHandler.scala
View File

@ -2,7 +2,7 @@ package com.twitter.home_mixer
import com.twitter.finatra.http.routing.HttpWarmup
import com.twitter.finatra.httpclient.RequestBuilder._
import com.twitter.inject.Logging
import com.twitter.util.logging.Logging
import com.twitter.inject.utils.Handler
import com.twitter.util.Try
import javax.inject.Inject

34
home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerServer.scala
View File

@ -12,57 +12,63 @@ import com.twitter.finatra.thrift.ThriftServer
import com.twitter.finatra.thrift.filters._
import com.twitter.finatra.thrift.routing.ThriftRouter
import com.twitter.home_mixer.controller.HomeThriftController
import com.twitter.home_mixer.federated.HomeMixerColumn
import com.twitter.home_mixer.module._
import com.twitter.home_mixer.param.GlobalParamConfigModule
import com.twitter.home_mixer.product.HomeMixerProductModule
import com.twitter.home_mixer.{thriftscala => st}
import com.twitter.product_mixer.component_library.module.AccountRecommendationsMixerModule
import com.twitter.product_mixer.component_library.module.CrMixerClientModule
import com.twitter.product_mixer.component_library.module.DarkTrafficFilterModule
import com.twitter.product_mixer.component_library.module.EarlybirdModule
import com.twitter.product_mixer.component_library.module.ExploreRankerClientModule
import com.twitter.product_mixer.component_library.module.GizmoduckClientModule
import com.twitter.product_mixer.component_library.module.OnboardingTaskServiceModule
import com.twitter.product_mixer.component_library.module.SocialGraphServiceModule
import com.twitter.product_mixer.component_library.module.TimelineMixerClientModule
import com.twitter.product_mixer.component_library.module.TimelineRankerClientModule
import com.twitter.product_mixer.component_library.module.TimelineScorerClientModule
import com.twitter.product_mixer.component_library.module.TimelineServiceClientModule
import com.twitter.product_mixer.component_library.module.TweetImpressionStoreModule
import com.twitter.product_mixer.component_library.module.TweetMixerClientModule
import com.twitter.product_mixer.component_library.module.UserSessionStoreModule
import com.twitter.product_mixer.core.controllers.ProductMixerController
import com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper
import com.twitter.product_mixer.core.module.ProductMixerModule
import com.twitter.product_mixer.core.module.StratoClientModule
import com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule
import com.twitter.strato.fed.StratoFed
import com.twitter.strato.fed.server.StratoFedServer
object HomeMixerServerMain extends HomeMixerServer
class HomeMixerServer extends ThriftServer with Mtls with HttpServer with HttpMtls {
class HomeMixerServer
extends StratoFedServer
with ThriftServer
with Mtls
with HttpServer
with HttpMtls {
override val name = "home-mixer-server"
override val modules: Seq[Module] = Seq(
AccountRecommendationsMixerModule,
AdvertiserBrandSafetySettingsStoreModule,
BlenderClientModule,
ClientSentImpressionsPublisherModule,
ConversationServiceModule,
CrMixerClientModule,
EarlybirdModule,
ExploreRankerClientModule,
FeedbackHistoryClientModule,
GizmoduckClientModule,
GlobalParamConfigModule,
HomeAdsCandidateSourceModule,
HomeMixerFlagsModule,
HomeMixerProductModule,
HomeMixerResourcesModule,
HomeNaviModelClientModule,
ImpressionBloomFilterModule,
InjectionHistoryClientModule,
FeedbackHistoryClientModule,
ManhattanClientsModule,
ManhattanFeatureRepositoryModule,
ManhattanTweetImpressionStoreModule,
MemcachedFeatureRepositoryModule,
NaviModelClientModule,
OnboardingTaskServiceModule,
OptimizedStratoClientModule,
PeopleDiscoveryServiceModule,
@ -74,24 +80,23 @@ class HomeMixerServer extends ThriftServer with Mtls with HttpServer with HttpMt
SimClustersRecentEngagementsClientModule,
SocialGraphServiceModule,
StaleTweetsCacheModule,
StratoClientModule,
ThriftFeatureRepositoryModule,
TimelineMixerClientModule,
TimelineRankerClientModule,
TimelineScorerClientModule,
TimelineServiceClientModule,
TimelinesPersistenceStoreClientModule,
TopicSocialProofClientModule,
TweetImpressionStoreModule,
TweetyPieClientModule,
TweetMixerClientModule,
TweetypieClientModule,
TweetypieStaticEntitiesCacheClientModule,
UserMetadataStoreModule,
UserSessionStoreModule,
new DarkTrafficFilterModule[st.HomeMixer.ReqRepServicePerEndpoint](),
new MtlsThriftWebFormsModule[st.HomeMixer.MethodPerEndpoint](this),
new ProductScopeStringCenterModule()
)
def configureThrift(router: ThriftRouter): Unit = {
override def configureThrift(router: ThriftRouter): Unit = {
router
.filter[LoggingMDCFilter]
.filter[TraceIdMDCFilter]
@ -111,6 +116,11 @@ class HomeMixerServer extends ThriftServer with Mtls with HttpServer with HttpMt
this.injector,
st.HomeMixer.ExecutePipeline))
override val dest: String = "/s/home-mixer/home-mixer:strato"
override val columns: Seq[Class[_ <: StratoFed.Column]] =
Seq(classOf[HomeMixerColumn])
override protected def warmup(): Unit = {
handle[HomeMixerThriftServerWarmupHandler]()
handle[HomeMixerHttpServerWarmupHandler]()

2
home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerThriftServerWarmupHandler.scala
View File

@ -3,7 +3,7 @@ package com.twitter.home_mixer
import com.twitter.finagle.thrift.ClientId
import com.twitter.finatra.thrift.routing.ThriftWarmup
import com.twitter.home_mixer.{thriftscala => st}
import com.twitter.inject.Logging
import com.twitter.util.logging.Logging
import com.twitter.inject.utils.Handler
import com.twitter.product_mixer.core.{thriftscala => pt}
import com.twitter.scrooge.Request

10
home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/BUILD.bazel
View File

@ -5,19 +5,13 @@ scala_library(
tags = ["bazel-compatible"],
dependencies = [
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
@ -25,10 +19,6 @@ scala_library(
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
],
exports = [
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",

25
home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/ConversationServiceCandidatePipelineConfig.scala
View File

@ -1,18 +1,23 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
import com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter
import com.twitter.home_mixer.functional_component.filter.PredicateFeatureFilter
import com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter
import com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
import com.twitter.product_mixer.component_library.filter.FeatureFilter
import com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
@ -33,8 +38,8 @@ import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipel
class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
conversationServiceCandidateSource: ConversationServiceCandidateSource,
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
namesFeatureHydrator: NamesFeatureHydrator,
invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,
override val gates: Seq[BaseGate[Query]],
override val decorator: Option[CandidateDecorator[Query, TweetCandidate]])
extends DependentCandidatePipelineConfig[
@ -62,10 +67,10 @@ class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
val tweetsWithConversationMetadata = candidates.map { candidate =>
TweetWithConversationMetadata(
tweetId = candidate.candidateIdLong,
userId = None,
sourceTweetId = None,
sourceUserId = None,
inReplyToTweetId = None,
userId = candidate.features.getOrElse(AuthorIdFeature, None),
sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),
sourceUserId = candidate.features.getOrElse(SourceUserIdFeature, None),
inReplyToTweetId = candidate.features.getOrElse(InReplyToTweetIdFeature, None),
conversationId = None,
ancestors = Seq.empty
)
@ -84,7 +89,10 @@ class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
override val preFilterFeatureHydrationPhase1: Seq[
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
] = Seq(tweetypieFeatureHydrator, socialGraphServiceFeatureHydrator)
] = Seq(
tweetypieFeatureHydrator,
InNetworkFeatureHydrator,
)
override def filters: Seq[Filter[Query, TweetCandidate]] = Seq(
RetweetDeduplicationFilter,
@ -93,6 +101,7 @@ class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
FilterIdentifier(QuotedTweetDroppedFilterId),
shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }
),
invalidSubscriptionTweetFilter,
InvalidConversationModuleFilter
)

8
home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/ConversationServiceCandidatePipelineConfigBuilder.scala
View File

@ -1,9 +1,9 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.SocialGraphServiceFeatureHydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
import com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
@ -15,7 +15,7 @@ import javax.inject.Singleton
class ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery] @Inject() (
conversationServiceCandidateSource: ConversationServiceCandidateSource,
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator: SocialGraphServiceFeatureHydrator,
invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,
namesFeatureHydrator: NamesFeatureHydrator) {
def build(
@ -25,8 +25,8 @@ class ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery]
new ConversationServiceCandidatePipelineConfig(
conversationServiceCandidateSource,
tweetypieFeatureHydrator,
socialGraphServiceFeatureHydrator,
namesFeatureHydrator,
invalidSubscriptionTweetFilter,
gates,
decorator
)

2
home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/EditedTweetsCandidatePipelineConfig.scala
View File

@ -1,7 +1,7 @@
package com.twitter.home_mixer.candidate_pipeline
import com.twitter.home_mixer.functional_component.candidate_source.StaleTweetsCacheCandidateSource
import com.twitter.home_mixer.functional_component.decorator.HomeFeedbackActionInfoBuilder
import com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
import com.twitter.home_mixer.functional_component.query_transformer.EditedTweetsCandidatePipelineQueryTransformer
import com.twitter.home_mixer.service.HomeMixerAlertConfig

3
home-mixer/server/src/main/scala/com/twitter/home_mixer/controller/BUILD.bazel
View File

@ -13,8 +13,5 @@ scala_library(
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt",
"snowflake/src/main/scala/com/twitter/snowflake/id",
"src/thrift/com/twitter/context:twitter-context-scala",
"src/thrift/com/twitter/timelines/render:thrift-scala",
"twitter-context/src/main/scala",
],
)

1
home-mixer/server/src/main/scala/com/twitter/home_mixer/controller/HomeThriftController.scala
View File

@ -7,6 +7,7 @@ import com.twitter.home_mixer.service.ScoredTweetsService
import com.twitter.home_mixer.{thriftscala => t}
import com.twitter.product_mixer.core.controllers.DebugTwitterContext
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
import com.twitter.product_mixer.core.service.debug_query.DebugQueryService
import com.twitter.product_mixer.core.service.urt.UrtService
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch

24
home-mixer/server/src/main/scala/com/twitter/home_mixer/federated/BUILD.bazel Normal file
View File

@ -0,0 +1,24 @@
scala_library(
sources = ["*.scala"],
compiler_option_sets = ["fatal_warnings"],
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/thrift/src/main/thrift:thrift-scala",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry",
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
"src/thrift/com/twitter/gizmoduck:thrift-scala",
"src/thrift/com/twitter/timelines/render:thrift-scala",
"stitch/stitch-repo/src/main/scala",
"strato/config/columns/auth-context:auth-context-strato-client",
"strato/config/columns/gizmoduck:gizmoduck-strato-client",
"strato/config/src/thrift/com/twitter/strato/graphql/timelines:graphql-timelines-scala",
"strato/src/main/scala/com/twitter/strato/callcontext",
"strato/src/main/scala/com/twitter/strato/fed",
"strato/src/main/scala/com/twitter/strato/fed/server",
],
)

217
home-mixer/server/src/main/scala/com/twitter/home_mixer/federated/HomeMixerColumn.scala Normal file
View File

@ -0,0 +1,217 @@
package com.twitter.home_mixer.federated
import com.twitter.gizmoduck.{thriftscala => gd}
import com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller
import com.twitter.home_mixer.model.request.HomeMixerRequest
import com.twitter.home_mixer.{thriftscala => hm}
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
import com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest
import com.twitter.product_mixer.core.pipeline.product.ProductPipelineResult
import com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry
import com.twitter.product_mixer.core.{thriftscala => pm}
import com.twitter.stitch.Arrow
import com.twitter.stitch.Stitch
import com.twitter.strato.callcontext.CallContext
import com.twitter.strato.catalog.OpMetadata
import com.twitter.strato.config._
import com.twitter.strato.data._
import com.twitter.strato.fed.StratoFed
import com.twitter.strato.generated.client.auth_context.AuditIpClientColumn
import com.twitter.strato.generated.client.gizmoduck.CompositeOnUserClientColumn
import com.twitter.strato.graphql.timelines.{thriftscala => gql}
import com.twitter.strato.thrift.ScroogeConv
import com.twitter.timelines.render.{thriftscala => tr}
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeMixerColumn @Inject() (
homeMixerRequestUnmarshaller: HomeMixerRequestUnmarshaller,
compositeOnUserClientColumn: CompositeOnUserClientColumn,
auditIpClientColumn: AuditIpClientColumn,
paramsBuilder: ParamsBuilder,
productPipelineRegistry: ProductPipelineRegistry)
extends StratoFed.Column(HomeMixerColumn.Path)
with StratoFed.Fetch.Arrow {
override val contactInfo: ContactInfo = ContactInfo(
contactEmail = "",
ldapGroup = "",
slackRoomId = ""
)
override val metadata: OpMetadata =
OpMetadata(
lifecycle = Some(Lifecycle.Production),
description =
Some(Description.PlainText("Federated Strato column for Timelines served via Home Mixer"))
)
private val bouncerAccess: Seq[Policy] = Seq(BouncerAccess())
private val finatraTestServiceIdentifiers: Seq[Policy] = Seq(
ServiceIdentifierPattern(
role = "",
service = "",
env = "",
zone = Seq(""))
)
override val policy: Policy = AnyOf(bouncerAccess ++ finatraTestServiceIdentifiers)
override type Key = gql.TimelineKey
override type View = gql.HomeTimelineView
override type Value = tr.Timeline
override val keyConv: Conv[Key] = ScroogeConv.fromStruct[gql.TimelineKey]
override val viewConv: Conv[View] = ScroogeConv.fromStruct[gql.HomeTimelineView]
override val valueConv: Conv[Value] = ScroogeConv.fromStruct[tr.Timeline]
private def createHomeMixerRequestArrow(
compositeOnUserClientColumn: CompositeOnUserClientColumn,
auditIpClientColumn: AuditIpClientColumn
): Arrow[(Key, View), hm.HomeMixerRequest] = {
val populateUserRolesAndIp: Arrow[(Key, View), (Option[Set[String]], Option[String])] = {
val gizmoduckView: (gd.LookupContext, Set[gd.QueryFields]) =
(gd.LookupContext(), Set(gd.QueryFields.Roles))
val populateUserRoles = Arrow
.flatMap[(Key, View), Option[Set[String]]] { _ =>
Stitch.collect {
CallContext.twitterUserId.map { userId =>
compositeOnUserClientColumn.fetcher
.callStack(HomeMixerColumn.FetchCallstack)
.fetch(userId, gizmoduckView).map(_.v)
.map {
_.flatMap(_.roles.map(_.roles.toSet)).getOrElse(Set.empty)
}
}
}
}
val populateIpAddress = Arrow
.flatMap[(Key, View), Option[String]](_ =>
auditIpClientColumn.fetcher
.callStack(HomeMixerColumn.FetchCallstack)
.fetch((), ()).map(_.v))
Arrow.join(
populateUserRoles,
populateIpAddress
)
}
Arrow.zipWithArg(populateUserRolesAndIp).map {
case ((key, view), (roles, ipAddress)) =>
val deviceContextOpt = Some(
hm.DeviceContext(
isPolling = CallContext.isPolling,
requestContext = view.requestContext,
latestControlAvailable = view.latestControlAvailable,
autoplayEnabled = view.autoplayEnabled
))
val seenTweetIds = view.seenTweetIds.filter(_.nonEmpty)
val (product, productContext) = key match {
case gql.TimelineKey.HomeTimeline(_) | gql.TimelineKey.HomeTimelineV2(_) =>
(
hm.Product.ForYou,
hm.ProductContext.ForYou(
hm.ForYou(
deviceContextOpt,
seenTweetIds,
view.dspClientContext,
view.pushToHomeTweetId
)
))
case gql.TimelineKey.HomeLatestTimeline(_) | gql.TimelineKey.HomeLatestTimelineV2(_) =>
(
hm.Product.Following,
hm.ProductContext.Following(
hm.Following(deviceContextOpt, seenTweetIds, view.dspClientContext)))
case gql.TimelineKey.CreatorSubscriptionsTimeline(_) =>
(
hm.Product.Subscribed,
hm.ProductContext.Subscribed(hm.Subscribed(deviceContextOpt, seenTweetIds)))
case _ => throw new UnsupportedOperationException(s"Unknown product: $key")
}
val clientContext = pm.ClientContext(
userId = CallContext.twitterUserId,
guestId = CallContext.guestId,
guestIdAds = CallContext.guestIdAds,
guestIdMarketing = CallContext.guestIdMarketing,
appId = CallContext.clientApplicationId,
ipAddress = ipAddress,
userAgent = CallContext.userAgent,
countryCode = CallContext.requestCountryCode,
languageCode = CallContext.requestLanguageCode,
isTwoffice = CallContext.isInternalOrTwoffice,
userRoles = roles,
deviceId = CallContext.deviceId,
mobileDeviceId = CallContext.mobileDeviceId,
mobileDeviceAdId = CallContext.adId,
limitAdTracking = CallContext.limitAdTracking
)
hm.HomeMixerRequest(
clientContext = clientContext,
product = product,
productContext = Some(productContext),
maxResults = Try(view.count.get.toInt).toOption.orElse(HomeMixerColumn.MaxCount),
cursor = view.cursor.filter(_.nonEmpty)
)
}
}
override val fetch: Arrow[(Key, View), Result[Value]] = {
val transformThriftIntoPipelineRequest: Arrow[
(Key, View),
ProductPipelineRequest[HomeMixerRequest]
] = {
Arrow
.identity[(Key, View)]
.andThen {
createHomeMixerRequestArrow(compositeOnUserClientColumn, auditIpClientColumn)
}
.map {
case thriftRequest =>
val request = homeMixerRequestUnmarshaller(thriftRequest)
val params = paramsBuilder.build(
clientContext = request.clientContext,
product = request.product,
featureOverrides =
request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),
)
ProductPipelineRequest(request, params)
}
}
val underlyingProduct: Arrow[
ProductPipelineRequest[HomeMixerRequest],
ProductPipelineResult[tr.TimelineResponse]
] = Arrow
.identity[ProductPipelineRequest[HomeMixerRequest]]
.map { pipelineRequest =>
val pipelineArrow = productPipelineRegistry
.getProductPipeline[HomeMixerRequest, tr.TimelineResponse](
pipelineRequest.request.product)
.arrow
(pipelineArrow, pipelineRequest)
}.applyArrow
transformThriftIntoPipelineRequest.andThen(underlyingProduct).map {
_.result match {
case Some(result) => found(result.timeline)
case _ => missing
}
}
}
}
object HomeMixerColumn {
val Path = "home-mixer/homeMixer.Timeline"
private val FetchCallstack = s"$Path:fetch"
private val MaxCount: Option[Int] = Some(100)
}

3
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source/BUILD.bazel
View File

@ -10,11 +10,8 @@ scala_library(
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala",
"src/thrift/com/twitter/search:earlybird-scala",
"stitch/stitch-timelineservice/src/main/scala",
"strato/config/columns/recommendations/similarity:similarity-strato-client",
"strato/src/main/scala/com/twitter/strato/client",
],
exports = [
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",

34
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source/SimilarityBasedUsersCandidateSource.scala
View File

@ -1,34 +0,0 @@
package com.twitter.home_mixer.functional_component.candidate_source
import com.twitter.hermit.candidate.{thriftscala => t}
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
import com.twitter.stitch.Stitch
import com.twitter.strato.client.Fetcher
import com.twitter.strato.generated.client.recommendations.similarity.SimilarUsersBySimsOnUserClientColumn
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class SimilarityBasedUsersCandidateSource @Inject() (
similarUsersBySimsOnUserClientColumn: SimilarUsersBySimsOnUserClientColumn)
extends CandidateSource[Seq[Long], t.Candidate] {
override val identifier: CandidateSourceIdentifier =
CandidateSourceIdentifier("SimilarityBasedUsers")
private val fetcher: Fetcher[Long, Unit, t.Candidates] =
similarUsersBySimsOnUserClientColumn.fetcher
override def apply(request: Seq[Long]): Stitch[Seq[t.Candidate]] = {
Stitch
.collect {
request.map { userId =>
fetcher.fetch(userId, Unit).map { result =>
result.v.map(_.candidates).getOrElse(Seq.empty)
}
}
}.map(_.flatten)
}
}

34
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/AuthorChildFeedbackActionBuilder.scala
View File

@ -1,34 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class AuthorChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = authorId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerTweetsString,
confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Tweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
}
}

6
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/BUILD.bazel
View File

@ -6,13 +6,11 @@ scala_library(
dependencies = [
"finagle/finagle-core/src/main",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
"joinkey/src/main/scala/com/twitter/joinkey/context",
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope",
@ -25,8 +23,6 @@ scala_library(
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
"stringcenter/client",
"stringcenter/client/src/main/java",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/translation",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
],
)

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/BlockUserChildFeedbackActionBuilder.scala
View File

@ -1,54 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class BlockUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.blockUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = Some(BottomSheet),
clientEventInfo = None,
icon = Some(icon.No),
richBehavior = Some(RichFeedbackBehaviorBlockUser(userId)),
subprompt = None
)
}
}
}
}

88
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/DontLikeFeedbackActionBuilder.scala
View File

@ -1,88 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tls}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class DontLikeFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings,
authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,
retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,
notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,
unfollowUserChildFeedbackActionBuilder: UnfollowUserChildFeedbackActionBuilder,
muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder,
blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,
reportTweetChildFeedbackActionBuilder: ReportTweetChildFeedbackActionBuilder) {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>
val feedbackEntities = Seq(
tlc.FeedbackEntity.TweetId(candidate.id),
tlc.FeedbackEntity.UserId(authorId)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = feedbackEntities,
ttl = Some(30.days)
)
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tls.FeedbackType.DontLike,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
val childFeedbackActions = if (query.params(EnableNahFeedbackInfoParam)) {
Seq(
unfollowUserChildFeedbackActionBuilder(candidateFeatures),
muteUserChildFeedbackActionBuilder(candidateFeatures),
blockUserChildFeedbackActionBuilder(candidateFeatures),
reportTweetChildFeedbackActionBuilder(candidate)
).flatten
} else {
Seq(
authorChildFeedbackActionBuilder(candidateFeatures),
retweeterChildFeedbackActionBuilder(candidateFeatures),
notRelevantChildFeedbackActionBuilder(candidate, candidateFeatures)
).flatten
}
FeedbackAction(
feedbackType = DontLike,
prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),
confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),
childFeedbackActions =
if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(Frown),
richBehavior = None,
subprompt = None,
encodedFeedbackRequest = None
)
}
}
}

119
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/EngagerSocialContextBuilder.scala
View File

@ -1,119 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
private[decorator] case class SocialContextIdAndScreenName(
socialContextId: Long,
screenName: String)
object EngagerSocialContextBuilder {
private val UserIdRequestParamName = "user_id"
private val DirectInjectionContentSourceRequestParamName = "dis"
private val DirectInjectionIdRequestParamName = "diid"
private val DirectInjectionContentSourceSocialProofUsers = "socialproofusers"
private val SocialProofUrl = ""
}
case class EngagerSocialContextBuilder(
contextType: GeneralContextType,
stringCenter: StringCenter,
oneUserString: ExternalString,
twoUsersString: ExternalString,
moreUsersString: ExternalString,
timelineTitle: ExternalString) {
import EngagerSocialContextBuilder._
def apply(
socialContextIds: Seq[Long],
query: PipelineQuery,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>
realNames
.get(socialContextId).map(screenName =>
SocialContextIdAndScreenName(socialContextId, screenName))
}
validSocialContextIdAndScreenNames match {
case Seq(user) =>
val socialContextString =
stringCenter.prepare(oneUserString, Map("user" -> user.screenName))
Some(mkOneUserSocialContext(socialContextString, user.socialContextId))
case Seq(firstUser, secondUser) =>
val socialContextString =
stringCenter
.prepare(
twoUsersString,
Map("user1" -> firstUser.screenName, "user2" -> secondUser.screenName))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case firstUser +: otherUsers =>
val otherUsersCount = otherUsers.size
val socialContextString =
stringCenter
.prepare(
moreUsersString,
Map("user" -> firstUser.screenName, "count" -> otherUsersCount))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case _ => None
}
}
private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
}
private def mkManyUserSocialContext(
socialContextString: String,
viewerId: Long,
socialContextIds: Seq[Long]
): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = UrtEndpoint,
url = SocialProofUrl,
urtEndpointOptions = Some(UrtEndpointOptions(
requestParams = Some(Map(
UserIdRequestParamName -> viewerId.toString,
DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,
DirectInjectionIdRequestParamName -> socialContextIds.mkString(",")
)),
title = Some(stringCenter.prepare(timelineTitle)),
cacheId = None,
subtitle = None
))
))
)
}
}

78
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/ExtendedReplySocialContextBuilder.scala
View File

@ -1,78 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A replied' when the root tweet is out-of-network and the reply is in network.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ExtendedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val extendedReplyString = externalStrings.socialContextExtendedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing an extended reply banner
val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)
if (!inNetworkRoot && inNetworkFocalTweet) {
val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)
val focalTweetRealNames =
candidateFeatures
.getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])
val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)
(focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {
case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(extendedReplyString, placeholders = Map("user1" -> focalTweetAuthorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
))
))
case _ =>
None
}
} else {
None
}
}
}

61
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/FeedbackUtil.scala
View File

@ -1,61 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.suggests.{thriftscala => st}
import com.twitter.timelineservice.{thriftscala => tlst}
object FeedbackUtil {
val FeedbackTtl = 30.days
def buildUserSeeFewerChildFeedbackAction(
userId: Long,
namesByUserId: Map[Long, String],
promptExternalString: ExternalString,
confirmationExternalString: ExternalString,
engagementType: t.FeedbackEngagementType,
stringCenter: StringCenter,
injectionType: Option[st.SuggestType]
): Option[ChildFeedbackAction] = {
namesByUserId.get(userId).map { userScreenName =>
val prompt = stringCenter.prepare(
promptExternalString,
Map("user" -> userScreenName)
)
val confirmation = stringCenter.prepare(
confirmationExternalString,
Map("user" -> userScreenName)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = Some(engagementType),
entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),
ttl = Some(FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.SeeFewer,
feedbackMetadata = feedbackMetadata,
injectionType = injectionType
)
ChildFeedbackAction(
feedbackType = SeeFewer,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
}
}
}

53
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/FollowedBySocialContextBuilder.scala
View File

@ -1,53 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class FollowedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = FollowGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserFollowsString,
twoUsersString = externalStrings.socialContextTwoUsersFollowString,
moreUsersString = externalStrings.socialContextMoreUsersFollowString,
timelineTitle = externalStrings.socialContextFollowedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Only apply followed-by social context for OON Tweets
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
engagerSocialContextBuilder(
socialContextIds = validFollowedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
} else {
None
}
}
}

46
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeAdsClientEventDetailsBuilder.scala
View File

@ -1,46 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.finagle.tracing.Trace
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
case class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
override def apply(
query: PipelineQuery,
candidate: UniversalNoun[Any],
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = 0L,
traceId = Some(Trace.id.traceId.toLong),
requestJoinId = None)
val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = injectionType,
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

92
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeClientEventDetailsBuilder.scala
View File

@ -1,92 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.bijection.Base64String
import com.twitter.bijection.scrooge.BinaryScalaCodec
import com.twitter.bijection.{Injection => Serializer}
import com.twitter.finagle.tracing.Trace
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
import com.twitter.home_mixer.model.HomeFeatures.PositionFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.joinkey.context.RequestJoinKeyContext
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.Home
import com.twitter.suggests.controller_data.TweetTypeGenerator
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
object HomeClientEventDetailsBuilder {
implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =
BinaryScalaCodec(ControllerData)
val ControllerDataSerializer: Serializer[ControllerData, String] =
Serializer.connect[ControllerData, Array[Byte], Base64String, String]
/**
* define getRequestJoinId as a method(def) rather than a val because each new request
* needs to call the context to update the id.
*/
private def getRequestJoinId(): Option[Long] =
RequestJoinKeyContext.current.flatMap(_.requestJoinId)
}
case class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](
) extends BaseClientEventDetailsBuilder[Query, Candidate]
with TweetTypeGenerator[FeatureMap] {
import HomeClientEventDetailsBuilder._
override def apply(
query: Query,
candidate: Candidate,
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val tweetTypesBitmaps = mkTweetTypesBitmaps(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val tweetTypesListBytes = mkItemTypesBitmapsV2(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val candidateSourceId =
candidateFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toByte)
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),
tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),
candidateTweetSourceId = candidateSourceId,
traceId = Some(Trace.id.traceId.toLong),
injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),
tweetTypesListBytes = Some(tweetTypesListBytes),
requestJoinId = getRequestJoinId(),
)
val serializedControllerData = ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None).map(_.name),
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

2
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeConversationServiceCandidateDecorator.scala
View File

@ -1,6 +1,8 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
import com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder
import com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeFeedbackActionInfoBuilder.scala
View File

@ -1,54 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelines.util.FeedbackMetadataSerializer
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeFeedbackActionInfoBuilder @Inject() (
notInterestedTopicFeedbackActionBuilder: NotInterestedTopicFeedbackActionBuilder,
dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)
extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackActionInfo] = {
val supportedProduct = query.product match {
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
case ForYouProduct => true
case _ => false
}
val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)
if (supportedProduct && !isAuthoredByViewer) {
val feedbackActions = Seq(
notInterestedTopicFeedbackActionBuilder(candidateFeatures),
dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)
).flatten
val feedbackMetadata = FeedbackMetadataSerializer.serialize(
t.FeedbackMetadata(injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)))
Some(
FeedbackActionInfo(
feedbackActions = feedbackActions,
feedbackMetadata = Some(feedbackMetadata),
displayContext = None,
clientEventInfo = None
))
} else None
}
}

26
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTimelinesScoreInfoBuilder.scala
View File

@ -1,26 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
object HomeTimelinesScoreInfoBuilder
extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {
private val UndefinedTweetScore = -1.0
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[TimelinesScoreInfo] = {
if (query.params(EnableSendScoresToClient)) {
val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)
Some(TimelinesScoreInfo(score))
} else None
}
}

44
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTweetSocialContextBuilder.scala
View File

@ -1,44 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class HomeTweetSocialContextBuilder @Inject() (
likedBySocialContextBuilder: LikedBySocialContextBuilder,
followedBySocialContextBuilder: FollowedBySocialContextBuilder,
topicSocialContextBuilder: TopicSocialContextBuilder,
extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,
receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder)
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
features: FeatureMap
): Option[SocialContext] = {
if (query.params(EnableSocialContextParam)) {
features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {
case None =>
likedBySocialContextBuilder(query, candidate, features)
.orElse(followedBySocialContextBuilder(query, candidate, features))
.orElse(topicSocialContextBuilder(query, candidate, features))
case Some(_) =>
val conversationId = features.getOrElse(ConversationModuleIdFeature, None)
// Only hydrate the social context into the root tweet in a conversation module
if (conversationId.contains(candidate.id)) {
extendedReplySocialContextBuilder(query, candidate, features)
.orElse(receivedReplySocialContextBuilder(query, candidate, features))
} else None
}
} else None
}
}

227
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTweetTypePredicates.scala
View File

@ -1,227 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures
import com.twitter.tweetypie.{thriftscala => tpt}
object HomeTweetTypePredicates {
/**
* IMPORTANT: Please avoid logging tweet types that are tied to sensitive
* internal author information / labels (e.g. blink labels, abuse labels, or geo-location).
*/
private[this] val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
("with_candidate", _ => true),
("retweet", _.getOrElse(IsRetweetFeature, false)),
("reply", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),
("image", _.getOrElse(EarlybirdFeature, None).exists(_.hasImage)),
("video", _.getOrElse(EarlybirdFeature, None).exists(_.hasVideo)),
("link", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),
("quote", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),
("like_social_context", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),
("protected", _.getOrElse(EarlybirdFeature, None).exists(_.isProtected)),
(
"has_exclusive_conversation_author_id",
_.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),
("is_eligible_for_connect_boost", _.getOrElse(AuthorIsEligibleForConnectBoostFeature, false)),
("hashtag", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),
("has_scheduled_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),
("has_recorded_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),
("is_read_from_cache", _.getOrElse(IsReadFromCacheFeature, false)),
(
"is_self_thread_tweet",
_.getOrElse(ConversationFeature, None).exists(_.isSelfThreadTweet.contains(true))),
("get_initial", _.getOrElse(GetInitialFeature, false)),
("get_newer", _.getOrElse(GetNewerFeature, false)),
("get_middle", _.getOrElse(GetMiddleFeature, false)),
("get_older", _.getOrElse(GetOlderFeature, false)),
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
("polling", _.getOrElse(PollingFeature, false)),
("tls_size_20_plus", _ => false),
("near_empty", _ => false),
("ranked_request", _ => false),
("mutual_follow", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),
("has_ticketed_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),
("in_utis_top5", _.getOrElse(PositionFeature, None).exists(_ < 5)),
("is_utis_pos0", _.getOrElse(PositionFeature, None).exists(_ == 0)),
("is_utis_pos1", _.getOrElse(PositionFeature, None).exists(_ == 1)),
("is_utis_pos2", _.getOrElse(PositionFeature, None).exists(_ == 2)),
("is_utis_pos3", _.getOrElse(PositionFeature, None).exists(_ == 3)),
("is_utis_pos4", _.getOrElse(PositionFeature, None).exists(_ == 4)),
(
"is_signup_request",
candidate => candidate.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
("empty_request", _ => false),
("served_size_less_than_5", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),
("served_size_less_than_10", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),
("served_size_less_than_20", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),
("served_size_less_than_50", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),
(
"served_size_between_50_and_100",
_.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),
("authored_by_contextual_user", _.getOrElse(AuthoredByContextualUserFeature, false)),
("has_ancestors", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),
("full_scoring_succeeded", _.getOrElse(FullScoringSucceededFeature, false)),
(
"account_age_less_than_30_minutes",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
(
"account_age_less_than_1_day",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),
(
"account_age_less_than_7_days",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),
(
"directed_at_user_is_in_first_degree",
_.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),
("root_user_is_in_first_degree", _ => false),
(
"has_semantic_core_annotation",
_.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),
("is_request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false)),
(
"part_of_utt",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),
("is_random_tweet", _.getOrElse(IsRandomTweetFeature, false)),
("has_random_tweet_in_response", _.getOrElse(HasRandomTweetFeature, false)),
("is_random_tweet_above_in_utis", _.getOrElse(IsRandomTweetAboveFeature, false)),
("is_request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
("viewer_is_employee", _ => false),
("viewer_is_timelines_employee", _ => false),
("viewer_follows_any_topics", _.getOrElse(UserFollowedTopicsCountFeature, None).exists(_ > 0)),
(
"has_ancestor_authored_by_viewer",
candidate =>
candidate
.getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>
candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),
("ancestor", _.getOrElse(IsAncestorCandidateFeature, false)),
(
"root_ancestor",
candidate =>
candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate
.getOrElse(InReplyToTweetIdFeature, None).isEmpty),
(
"deep_reply",
candidate =>
candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate
.getOrElse(AncestorsFeature, Seq.empty).size > 2),
(
"has_simcluster_embeddings",
_.getOrElse(
SimclustersTweetTopKClustersWithScoresFeature,
Map.empty[String, Double]).nonEmpty),
(
"tweet_age_less_than_15_seconds",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 15.seconds)),
("is_followed_topic_tweet", _ => false),
("is_recommended_topic_tweet", _ => false),
("is_topic_tweet", _ => false),
("preferred_language_matches_tweet_language", _ => false),
(
"device_language_matches_tweet_language",
candidate =>
candidate.getOrElse(TweetLanguageFeature, None) ==
candidate.getOrElse(DeviceLanguageFeature, None)),
("question", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),
("in_network", _.getOrElse(FromInNetworkSourceFeature, true)),
("viewer_follows_original_author", _ => false),
("has_account_follow_prompt", _ => false),
("has_relevance_prompt", _ => false),
("has_topic_annotation_haug_prompt", _ => false),
("has_topic_annotation_random_precision_prompt", _ => false),
("has_topic_annotation_prompt", _ => false),
(
"has_political_annotation",
_.getOrElse(EarlybirdFeature, None).exists(
_.semanticCoreAnnotations.exists(
_.exists(annotation =>
SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||
(annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),
(
"is_dont_at_me_by_invitation",
_.getOrElse(EarlybirdFeature, None).exists(
_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),
(
"is_dont_at_me_community",
_.getOrElse(EarlybirdFeature, None)
.exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),
("has_zero_score", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),
("is_viewer_not_invited_to_reply", _ => false),
("is_viewer_invited_to_reply", _ => false),
("has_gte_10_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),
("has_gte_100_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),
("has_gte_1k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
(
"has_gte_10k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
(
"has_gte_100k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),
("above_neighbor_is_topic_tweet", _ => false),
("is_topic_tweet_with_neighbor_below", _ => false),
("has_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),
("has_live_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),
(
"has_gte_10_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),
(
"has_gte_100_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),
(
"has_gte_1k_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),
(
"has_us_political_annotation",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&
annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),
(
"has_toxicity_score_above_threshold",
_.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),
(
"text_only",
candidate =>
candidate.getOrElse(HasDisplayedTextFeature, false) &&
!(candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasVideo) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),
(
"image_only",
candidate =>
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) &&
!candidate.getOrElse(HasDisplayedTextFeature, false)),
("has_1_image", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),
("has_2_images", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),
("has_3_images", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),
("has_4_images", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),
("has_card", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),
("3_or_more_consecutive_not_in_network", _ => false),
("2_or_more_consecutive_not_in_network", _ => false),
("5_out_of_7_not_in_network", _ => false),
("7_out_of_7_not_in_network", _ => false),
("5_out_of_5_not_in_network", _ => false),
("user_follow_count_gte_50", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),
("has_liked_by_social_context", _ => false),
("has_followed_by_social_context", _ => false),
("has_topic_social_context", _ => false),
("timeline_entry_has_banner", _ => false),
("served_in_conversation_module", _.getOrElse(ServedInConversationModuleFeature, false)),
(
"conversation_module_has_2_displayed_tweets",
_.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),
("conversation_module_has_gap", _.getOrElse(ConversationModuleHasGapFeature, false)),
("served_in_recap_tweet_candidate_module_injection", _ => false),
("served_in_threaded_conversation_module", _ => false)
)
val PredicateMap = CandidatePredicates.toMap
}

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/LikedBySocialContextBuilder.scala
View File

@ -1,54 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class LikedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = LikeGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserLikedString,
twoUsersString = externalStrings.socialContextTwoUsersLikedString,
moreUsersString = externalStrings.socialContextMoreUsersLikedString,
timelineTitle = externalStrings.socialContextLikedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Liked by users are valid only if they pass both the SGS and Perspective filters.
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
engagerSocialContextBuilder(
socialContextIds = validLikedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
}
}

47
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/ListConversationServiceCandidateDecorator.scala
View File

@ -1,47 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
import com.twitter.home_mixer.functional_component.decorator.builder.ListClientEventDetailsBuilder
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
import com.twitter.timelineservice.suggests.{thriftscala => st}
object ListConversationServiceCandidateDecorator {
private val ConversationModuleNamespace = EntryNamespace("list-conversation")
def apply(): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
val suggestType = st.SuggestType.OrganicListTweet
val component = InjectionScribeUtil.scribeComponent(suggestType).get
val clientEventInfoBuilder =
ClientEventInfoBuilder(component, Some(ListClientEventDetailsBuilder))
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
clientEventInfoBuilder = clientEventInfoBuilder
)
val moduleBuilder = TimelineModuleBuilder(
entryNamespace = ConversationModuleNamespace,
clientEventInfoBuilder = clientEventInfoBuilder,
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
)
Some(
UrtMultipleModulesDecorator(
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
moduleBuilder = moduleBuilder,
groupByKey = (_, _, candidateFeatures) =>
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
))
}
}

55
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/MuteUserChildFeedbackActionBuilder.scala
View File

@ -1,55 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class MuteUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.muteUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.SpeakerOff),
richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),
subprompt = None
)
}
}
}
}

71
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/NotInterestedTopicFeedbackActionBuilder.scala
View File

@ -1,71 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotInterestedTopicFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {
val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityType =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContext, topicContextFunctionalityType) match {
case (Some(topicId), Some(topicContextFunctionalityType))
if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||
topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>
Some(
FeedbackAction(
feedbackType = RichBehavior,
prompt = None,
confirmation = None,
childFeedbackActions = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior =
Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),
subprompt = None,
encodedFeedbackRequest = None
)
)
case _ => None
}
} else {
None
}
}
}

55
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/NotRelevantChildFeedbackActionBuilder.scala
View File

@ -1,55 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tlst}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotRelevantChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val prompt = stringCenter.prepare(externalStrings.notRelevantString)
val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),
ttl = Some(FeedbackUtil.FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.NotRelevant,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
Some(
ChildFeedbackAction(
feedbackType = NotRelevant,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
)
}
}

76
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/ReceivedReplySocialContextBuilder.scala
View File

@ -1,76 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ReceivedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val receivedReplyString = externalStrings.socialContextReceivedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing a received a reply banner
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)
if (inNetwork && !inNetworkFocalTweet) {
val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val authorNameOpt = authorIdOpt.flatMap(realNames.get)
(authorIdOpt, authorNameOpt) match {
case (Some(authorId), Some(authorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(receivedReplyString, placeholders = Map("user1" -> authorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
)
case _ => None
}
} else {
None
}
}
}

38
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/ReportTweetChildFeedbackActionBuilder.scala
View File

@ -1,38 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class ReportTweetChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate
): Option[ChildFeedbackAction] = {
Some(
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Flag),
richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),
subprompt = None
)
)
}
}

39
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/RetweeterChildFeedbackActionBuilder.scala
View File

@ -1,39 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class RetweeterChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
if (isRetweet) {
candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = retweeterId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerRetweetsString,
confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Retweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
} else None
}
}

42
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/TopicSocialContextBuilder.scala
View File

@ -1,42 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class TopicSocialContextBuilder @Inject() ()
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityTypeOpt =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {
case (Some(topicId), Some(topicContextFunctionalityType)) =>
Some(
TopicContext(
topicId = topicId.toString,
functionalityType = Some(topicContextFunctionalityType)
))
case _ => None
}
} else {
None
}
}
}

57
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/UnfollowUserChildFeedbackActionBuilder.scala
View File

@ -1,57 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class UnfollowUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
if (isInNetwork) {
userIdOpt.flatMap { userId =>
val screenNamesMap =
candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.unfollowUserString,
Map("username" -> userScreenName)
)
val confirmation = stringCenter.prepare(
externalStrings.unfollowUserConfirmationString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Unfollow),
richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),
subprompt = None
)
}
}
} else None
}
}

50
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/YouMightLikeSocialContextBuilder.scala
View File

@ -1,50 +0,0 @@
package com.twitter.home_mixer.functional_component.decorator
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Renders a fixed 'You Might Like' string above all OON Tweets.
*/
@Singleton
case class YouMightLikeSocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val youMightLikeString = externalStrings.socialContextYouMightLikeString
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
if (!isInNetwork && !isRetweet) {
Some(
GeneralContext(
contextType = SparkleGeneralContextType,
text = stringCenter.prepare(youMightLikeString),
url = None,
contextImageUrls = None,
landingUrl = None
))
} else {
None
}
}
}

5
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/BUILD.bazel
View File

@ -8,7 +8,9 @@ scala_library(
"finagle/finagle-core/src/main",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"joinkey/src/main/scala/com/twitter/joinkey/context",
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
@ -18,6 +20,7 @@ scala_library(
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
],
)

46
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeAdsClientEventDetailsBuilder.scala Normal file
View File

@ -0,0 +1,46 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.finagle.tracing.Trace
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
case class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
override def apply(
query: PipelineQuery,
candidate: UniversalNoun[Any],
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = 0L,
traceId = Some(Trace.id.traceId.toLong),
requestJoinId = None)
val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = injectionType,
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

92
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeClientEventDetailsBuilder.scala Normal file
View File

@ -0,0 +1,92 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.bijection.Base64String
import com.twitter.bijection.scrooge.BinaryScalaCodec
import com.twitter.bijection.{Injection => Serializer}
import com.twitter.finagle.tracing.Trace
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
import com.twitter.home_mixer.model.HomeFeatures.PositionFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.joinkey.context.RequestJoinKeyContext
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
import com.twitter.product_mixer.core.model.common.UniversalNoun
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.suggests.controller_data.Home
import com.twitter.suggests.controller_data.TweetTypeGenerator
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
object HomeClientEventDetailsBuilder {
implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =
BinaryScalaCodec(ControllerData)
val ControllerDataSerializer: Serializer[ControllerData, String] =
Serializer.connect[ControllerData, Array[Byte], Base64String, String]
/**
* define getRequestJoinId as a method(def) rather than a val because each new request
* needs to call the context to update the id.
*/
private def getRequestJoinId(): Option[Long] =
RequestJoinKeyContext.current.flatMap(_.requestJoinId)
}
case class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](
) extends BaseClientEventDetailsBuilder[Query, Candidate]
with TweetTypeGenerator[FeatureMap] {
import HomeClientEventDetailsBuilder._
override def apply(
query: Query,
candidate: Candidate,
candidateFeatures: FeatureMap
): Option[ClientEventDetails] = {
val tweetTypesBitmaps = mkTweetTypesBitmaps(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val tweetTypesListBytes = mkItemTypesBitmapsV2(
Home.TweetTypeIdxMap,
HomeTweetTypePredicates.PredicateMap,
candidateFeatures)
val candidateSourceId =
candidateFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toByte)
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),
tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),
candidateTweetSourceId = candidateSourceId,
traceId = Some(Trace.id.traceId.toLong),
injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),
tweetTypesListBytes = Some(tweetTypesListBytes),
requestJoinId = getRequestJoinId(),
)
val serializedControllerData = ControllerDataSerializer(
ControllerData.V2(
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
val clientEventDetails = ClientEventDetails(
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None).map(_.name),
controllerData = Some(serializedControllerData),
sourceData = None)),
articleDetails = None,
liveEventDetails = None,
commerceDetails = None
)
Some(clientEventDetails)
}
}

26
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeTimelinesScoreInfoBuilder.scala Normal file
View File

@ -0,0 +1,26 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
object HomeTimelinesScoreInfoBuilder
extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {
private val UndefinedTweetScore = -1.0
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[TimelinesScoreInfo] = {
if (query.params(EnableSendScoresToClient)) {
val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)
Some(TimelinesScoreInfo(score))
} else None
}
}

256
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeTweetTypePredicates.scala Normal file
View File

@ -0,0 +1,256 @@
package com.twitter.home_mixer.functional_component.decorator.builder
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
import com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures
import com.twitter.tweetypie.{thriftscala => tpt}
object HomeTweetTypePredicates {
/**
* IMPORTANT: Please avoid logging tweet types that are tied to sensitive
* internal author information / labels (e.g. blink labels, abuse labels, or geo-location).
*/
private[this] val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
("with_candidate", _ => true),
("retweet", _.getOrElse(IsRetweetFeature, false)),
("reply", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),
("image", _.getOrElse(EarlybirdFeature, None).exists(_.hasImage)),
("video", _.getOrElse(EarlybirdFeature, None).exists(_.hasVideo)),
("link", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),
("quote", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),
("like_social_context", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),
("protected", _.getOrElse(EarlybirdFeature, None).exists(_.isProtected)),
(
"has_exclusive_conversation_author_id",
_.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),
("is_eligible_for_connect_boost", _ => false),
("hashtag", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),
("has_scheduled_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),
("has_recorded_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),
("is_read_from_cache", _.getOrElse(IsReadFromCacheFeature, false)),
("get_initial", _.getOrElse(GetInitialFeature, false)),
("get_newer", _.getOrElse(GetNewerFeature, false)),
("get_middle", _.getOrElse(GetMiddleFeature, false)),
("get_older", _.getOrElse(GetOlderFeature, false)),
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
("polling", _.getOrElse(PollingFeature, false)),
("near_empty", _.getOrElse(ServedSizeFeature, None).exists(_ < 3)),
("is_request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
("mutual_follow", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),
(
"less_than_10_mins_since_lnpt",
_.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 10.minutes)),
("served_in_conversation_module", _.getOrElse(ServedInConversationModuleFeature, false)),
("has_ticketed_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),
("in_utis_top5", _.getOrElse(PositionFeature, None).exists(_ < 5)),
(
"conversation_module_has_2_displayed_tweets",
_.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),
("empty_request", _.getOrElse(ServedSizeFeature, None).exists(_ == 0)),
("served_size_less_than_50", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),
(
"served_size_between_50_and_100",
_.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),
("authored_by_contextual_user", _.getOrElse(AuthoredByContextualUserFeature, false)),
(
"is_self_thread_tweet",
_.getOrElse(ConversationFeature, None).exists(_.isSelfThreadTweet.contains(true))),
("has_ancestors", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),
("full_scoring_succeeded", _.getOrElse(FullScoringSucceededFeature, false)),
("served_size_less_than_20", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),
("served_size_less_than_10", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),
("served_size_less_than_5", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),
(
"account_age_less_than_30_minutes",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
("conversation_module_has_gap", _.getOrElse(ConversationModuleHasGapFeature, false)),
(
"directed_at_user_is_in_first_degree",
_.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),
(
"has_semantic_core_annotation",
_.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),
("is_request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false)),
(
"account_age_less_than_1_day",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),
(
"account_age_less_than_7_days",
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),
(
"part_of_utt",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),
(
"has_home_latest_request_past_week",
_.getOrElse(FollowingLastNonPollingTimeFeature, None).exists(_.untilNow < 7.days)),
("is_utis_pos0", _.getOrElse(PositionFeature, None).exists(_ == 0)),
("is_utis_pos1", _.getOrElse(PositionFeature, None).exists(_ == 1)),
("is_utis_pos2", _.getOrElse(PositionFeature, None).exists(_ == 2)),
("is_utis_pos3", _.getOrElse(PositionFeature, None).exists(_ == 3)),
("is_utis_pos4", _.getOrElse(PositionFeature, None).exists(_ == 4)),
("is_random_tweet", _.getOrElse(IsRandomTweetFeature, false)),
("has_random_tweet_in_response", _.getOrElse(HasRandomTweetFeature, false)),
("is_random_tweet_above_in_utis", _.getOrElse(IsRandomTweetAboveFeature, false)),
(
"has_ancestor_authored_by_viewer",
candidate =>
candidate
.getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>
candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),
("ancestor", _.getOrElse(IsAncestorCandidateFeature, false)),
(
"deep_reply",
candidate =>
candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate
.getOrElse(AncestorsFeature, Seq.empty).size > 2),
(
"has_simcluster_embeddings",
_.getOrElse(
SimclustersTweetTopKClustersWithScoresFeature,
Map.empty[String, Double]).nonEmpty),
(
"tweet_age_less_than_15_seconds",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 15.seconds)),
(
"less_than_1_hour_since_lnpt",
_.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 1.hour)),
("has_gte_10_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),
(
"device_language_matches_tweet_language",
candidate =>
candidate.getOrElse(TweetLanguageFeature, None) ==
candidate.getOrElse(DeviceLanguageFeature, None)),
(
"root_ancestor",
candidate =>
candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate
.getOrElse(InReplyToTweetIdFeature, None).isEmpty),
("question", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),
("in_network", _.getOrElse(InNetworkFeature, true)),
(
"has_political_annotation",
_.getOrElse(EarlybirdFeature, None).exists(
_.semanticCoreAnnotations.exists(
_.exists(annotation =>
SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||
(annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),
(
"is_dont_at_me_by_invitation",
_.getOrElse(EarlybirdFeature, None).exists(
_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),
(
"is_dont_at_me_community",
_.getOrElse(EarlybirdFeature, None)
.exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),
("has_zero_score", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),
(
"is_followed_topic_tweet",
_.getOrElse(TopicContextFunctionalityTypeFeature, None)
.exists(_ == BasicTopicContextFunctionalityType)),
(
"is_recommended_topic_tweet",
_.getOrElse(TopicContextFunctionalityTypeFeature, None)
.exists(_ == RecommendationTopicContextFunctionalityType)),
("has_gte_100_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),
("has_gte_1k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
(
"has_gte_10k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),
(
"has_gte_100k_favs",
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),
("has_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),
("has_live_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),
(
"has_gte_10_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),
(
"has_gte_100_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),
(
"has_gte_1k_retweets",
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),
(
"has_us_political_annotation",
_.getOrElse(EarlybirdFeature, None)
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&
annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),
(
"has_toxicity_score_above_threshold",
_.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),
("is_topic_tweet", _.getOrElse(TopicIdSocialContextFeature, None).isDefined),
(
"text_only",
candidate =>
candidate.getOrElse(HasDisplayedTextFeature, false) &&
!(candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasVideo) ||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),
(
"image_only",
candidate =>
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) &&
!candidate.getOrElse(HasDisplayedTextFeature, false)),
("has_1_image", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),
("has_2_images", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),
("has_3_images", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),
("has_4_images", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),
("has_card", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),
("user_follow_count_gte_50", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),
(
"has_liked_by_social_context",
candidateFeatures =>
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)
.exists(candidateFeatures
.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Seq.empty).toSet.contains)),
(
"has_followed_by_social_context",
_.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty),
(
"has_topic_social_context",
candidateFeatures =>
candidateFeatures
.getOrElse(TopicIdSocialContextFeature, None)
.isDefined &&
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None).isDefined),
("video_lte_10_sec", _.getOrElse(VideoDurationMsFeature, None).exists(_ <= 10000)),
(
"video_bt_10_60_sec",
_.getOrElse(VideoDurationMsFeature, None).exists(duration =>
duration > 10000 && duration <= 60000)),
("video_gt_60_sec", _.getOrElse(VideoDurationMsFeature, None).exists(_ > 60000)),
(
"tweet_age_lte_30_minutes",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 30.minutes)),
(
"tweet_age_lte_1_hour",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 1.hour)),
(
"tweet_age_lte_6_hours",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 6.hours)),
(
"tweet_age_lte_12_hours",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow <= 12.hours)),
(
"tweet_age_gte_24_hours",
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
.exists(_.untilNow >= 24.hours)),
)
val PredicateMap = CandidatePredicates.toMap
}

4
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/ListClientEventDetailsBuilder.scala
View File

@ -8,7 +8,7 @@ import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Ti
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelineservice.suggests.{thriftscala => st}
object ListClientEventDetailsBuilder
case class ListClientEventDetailsBuilder(suggestType: st.SuggestType)
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
override def apply(
@ -20,7 +20,7 @@ object ListClientEventDetailsBuilder
conversationDetails = None,
timelinesDetails = Some(
TimelinesDetails(
injectionType = Some(st.SuggestType.OrganicListTweet.name),
injectionType = Some(suggestType.name),
controllerData = None,
sourceData = None)),
articleDetails = None,

33
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/AuthorChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,33 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class AuthorChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = authorId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerTweetsString,
confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Tweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
}
}

7
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/BUILD.bazel
View File

@ -4,9 +4,16 @@ scala_library(
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
"src/thrift/com/twitter/timelines/service:thrift-scala",
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
],
)

53
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/BlockUserChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,53 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class BlockUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.blockUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = Some(BottomSheet),
clientEventInfo = None,
icon = Some(icon.No),
richBehavior = Some(RichFeedbackBehaviorBlockUser(userId)),
subprompt = None
)
}
}
}
}

87
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/DontLikeFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,87 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tls}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class DontLikeFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings,
authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,
retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,
notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,
unfollowUserChildFeedbackActionBuilder: UnfollowUserChildFeedbackActionBuilder,
muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder,
blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,
reportTweetChildFeedbackActionBuilder: ReportTweetChildFeedbackActionBuilder) {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>
val feedbackEntities = Seq(
tlc.FeedbackEntity.TweetId(candidate.id),
tlc.FeedbackEntity.UserId(authorId)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = feedbackEntities,
ttl = Some(30.days)
)
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tls.FeedbackType.DontLike,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
val childFeedbackActions = if (query.params(EnableNahFeedbackInfoParam)) {
Seq(
unfollowUserChildFeedbackActionBuilder(candidateFeatures),
muteUserChildFeedbackActionBuilder(candidateFeatures),
blockUserChildFeedbackActionBuilder(candidateFeatures),
reportTweetChildFeedbackActionBuilder(candidate)
).flatten
} else {
Seq(
authorChildFeedbackActionBuilder(candidateFeatures),
retweeterChildFeedbackActionBuilder(candidateFeatures),
notRelevantChildFeedbackActionBuilder(candidate, candidateFeatures)
).flatten
}
FeedbackAction(
feedbackType = DontLike,
prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),
confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),
childFeedbackActions =
if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(Frown),
richBehavior = None,
subprompt = None,
encodedFeedbackRequest = None
)
}
}
}

119
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/EngagerSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,119 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
private[decorator] case class SocialContextIdAndScreenName(
socialContextId: Long,
screenName: String)
object EngagerSocialContextBuilder {
private val UserIdRequestParamName = "user_id"
private val DirectInjectionContentSourceRequestParamName = "dis"
private val DirectInjectionIdRequestParamName = "diid"
private val DirectInjectionContentSourceSocialProofUsers = "socialproofusers"
private val SocialProofUrl = ""
}
case class EngagerSocialContextBuilder(
contextType: GeneralContextType,
stringCenter: StringCenter,
oneUserString: ExternalString,
twoUsersString: ExternalString,
moreUsersString: ExternalString,
timelineTitle: ExternalString) {
import EngagerSocialContextBuilder._
def apply(
socialContextIds: Seq[Long],
query: PipelineQuery,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>
realNames
.get(socialContextId).map(screenName =>
SocialContextIdAndScreenName(socialContextId, screenName))
}
validSocialContextIdAndScreenNames match {
case Seq(user) =>
val socialContextString =
stringCenter.prepare(oneUserString, Map("user" -> user.screenName))
Some(mkOneUserSocialContext(socialContextString, user.socialContextId))
case Seq(firstUser, secondUser) =>
val socialContextString =
stringCenter
.prepare(
twoUsersString,
Map("user1" -> firstUser.screenName, "user2" -> secondUser.screenName))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case firstUser +: otherUsers =>
val otherUsersCount = otherUsers.size
val socialContextString =
stringCenter
.prepare(
moreUsersString,
Map("user" -> firstUser.screenName, "count" -> otherUsersCount))
Some(
mkManyUserSocialContext(
socialContextString,
query.getRequiredUserId,
validSocialContextIdAndScreenNames.map(_.socialContextId)))
case _ => None
}
}
private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
}
private def mkManyUserSocialContext(
socialContextString: String,
viewerId: Long,
socialContextIds: Seq[Long]
): GeneralContext = {
GeneralContext(
contextType = contextType,
text = socialContextString,
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = UrtEndpoint,
url = SocialProofUrl,
urtEndpointOptions = Some(UrtEndpointOptions(
requestParams = Some(Map(
UserIdRequestParamName -> viewerId.toString,
DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,
DirectInjectionIdRequestParamName -> socialContextIds.mkString(",")
)),
title = Some(stringCenter.prepare(timelineTitle)),
cacheId = None,
subtitle = None
))
))
)
}
}

78
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ExtendedReplySocialContextBuilder.scala Normal file
View File

@ -0,0 +1,78 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A replied' when the root tweet is out-of-network and the reply is in network.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ExtendedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val extendedReplyString = externalStrings.socialContextExtendedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing an extended reply banner
val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)
if (!inNetworkRoot && inNetworkFocalTweet) {
val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)
val focalTweetRealNames =
candidateFeatures
.getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])
val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)
(focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {
case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(extendedReplyString, placeholders = Map("user1" -> focalTweetAuthorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
))
))
case _ =>
None
}
} else {
None
}
}
}

18
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FeedbackStrings.scala Normal file
View File

@ -0,0 +1,18 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.ExternalStringRegistry
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
class FeedbackStrings @Inject() (
@ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry]) {
private val externalStringRegistry = externalStringRegistryProvider.get()
val seeLessOftenFeedbackString =
externalStringRegistry.createProdString("Feedback.seeLessOften")
val seeLessOftenConfirmationFeedbackString =
externalStringRegistry.createProdString("Feedback.seeLessOftenConfirmation")
}

61
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FeedbackUtil.scala Normal file
View File

@ -0,0 +1,61 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.conversions.DurationOps._
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer
import com.twitter.stringcenter.client.StringCenter
import com.twitter.stringcenter.client.core.ExternalString
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.suggests.{thriftscala => st}
import com.twitter.timelineservice.{thriftscala => tlst}
object FeedbackUtil {
val FeedbackTtl = 30.days
def buildUserSeeFewerChildFeedbackAction(
userId: Long,
namesByUserId: Map[Long, String],
promptExternalString: ExternalString,
confirmationExternalString: ExternalString,
engagementType: t.FeedbackEngagementType,
stringCenter: StringCenter,
injectionType: Option[st.SuggestType]
): Option[ChildFeedbackAction] = {
namesByUserId.get(userId).map { userScreenName =>
val prompt = stringCenter.prepare(
promptExternalString,
Map("user" -> userScreenName)
)
val confirmation = stringCenter.prepare(
confirmationExternalString,
Map("user" -> userScreenName)
)
val feedbackMetadata = FeedbackMetadata(
engagementType = Some(engagementType),
entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),
ttl = Some(FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.SeeFewer,
feedbackMetadata = feedbackMetadata,
injectionType = injectionType
)
ChildFeedbackAction(
feedbackType = SeeFewer,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
}
}
}

53
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FollowedBySocialContextBuilder.scala Normal file
View File

@ -0,0 +1,53 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class FollowedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = FollowGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserFollowsString,
twoUsersString = externalStrings.socialContextTwoUsersFollowString,
moreUsersString = externalStrings.socialContextMoreUsersFollowString,
timelineTitle = externalStrings.socialContextFollowedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Only apply followed-by social context for OON Tweets
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
engagerSocialContextBuilder(
socialContextIds = validFollowedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
} else {
None
}
}
}

53
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeFeedbackActionInfoBuilder.scala Normal file
View File

@ -0,0 +1,53 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.service.{thriftscala => t}
import com.twitter.timelines.util.FeedbackMetadataSerializer
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class HomeFeedbackActionInfoBuilder @Inject() (
notInterestedTopicFeedbackActionBuilder: NotInterestedTopicFeedbackActionBuilder,
dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)
extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackActionInfo] = {
val supportedProduct = query.product match {
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
case ForYouProduct => true
case _ => false
}
val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)
if (supportedProduct && !isAuthoredByViewer) {
val feedbackActions = Seq(
notInterestedTopicFeedbackActionBuilder(candidateFeatures),
dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)
).flatten
val feedbackMetadata = FeedbackMetadataSerializer.serialize(
t.FeedbackMetadata(injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)))
Some(
FeedbackActionInfo(
feedbackActions = feedbackActions,
feedbackMetadata = Some(feedbackMetadata),
displayContext = None,
clientEventInfo = None
))
} else None
}
}

50
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeTweetSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,50 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class HomeTweetSocialContextBuilder @Inject() (
likedBySocialContextBuilder: LikedBySocialContextBuilder,
listsSocialContextBuilder: ListsSocialContextBuilder,
followedBySocialContextBuilder: FollowedBySocialContextBuilder,
topicSocialContextBuilder: TopicSocialContextBuilder,
extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,
receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder,
popularVideoSocialContextBuilder: PopularVideoSocialContextBuilder,
popularInYourAreaSocialContextBuilder: PopularInYourAreaSocialContextBuilder)
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
features: FeatureMap
): Option[SocialContext] = {
if (query.params(EnableSocialContextParam)) {
features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {
case None =>
likedBySocialContextBuilder(query, candidate, features)
.orElse(followedBySocialContextBuilder(query, candidate, features))
.orElse(topicSocialContextBuilder(query, candidate, features))
.orElse(popularVideoSocialContextBuilder(query, candidate, features))
.orElse(listsSocialContextBuilder(query, candidate, features))
.orElse(popularInYourAreaSocialContextBuilder(query, candidate, features))
case Some(_) =>
val conversationId = features.getOrElse(ConversationModuleIdFeature, None)
// Only hydrate the social context into the root tweet in a conversation module
if (conversationId.contains(candidate.id)) {
extendedReplySocialContextBuilder(query, candidate, features)
.orElse(receivedReplySocialContextBuilder(query, candidate, features))
} else None
}
} else None
}
}

6
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeWhoToFollowFeedbackActionInfoBuilder.scala
View File

@ -5,7 +5,6 @@ import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.ExternalStringRegistry
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
@ -32,12 +31,13 @@ object HomeWhoToFollowFeedbackActionInfoBuilder {
@Singleton
case class HomeWhoToFollowFeedbackActionInfoBuilder @Inject() (
@ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry],
feedbackStrings: FeedbackStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
private val whoToFollowFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
externalStringRegistry = externalStringRegistryProvider.get(),
seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,
seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,
stringCenter = stringCenterProvider.get(),
encodedFeedbackRequest = Some(HomeWhoToFollowFeedbackActionInfoBuilder.EncodedFeedbackRequest)
)

52
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeWhoToSubscribeFeedbackActionInfoBuilder.scala Normal file
View File

@ -0,0 +1,52 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder
import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.timelines.service.{thriftscala => tl}
import com.twitter.timelines.util.FeedbackRequestSerializer
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
import com.twitter.timelineservice.thriftscala.FeedbackType
object HomeWhoToSubscribeFeedbackActionInfoBuilder {
private val FeedbackMetadata = tl.FeedbackMetadata(
injectionType = Some(SuggestType.WhoToSubscribe),
engagementType = None,
entityIds = Seq.empty,
ttlMs = None
)
private val FeedbackRequest =
tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)
private val EncodedFeedbackRequest =
FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))
}
@Singleton
case class HomeWhoToSubscribeFeedbackActionInfoBuilder @Inject() (
feedbackStrings: FeedbackStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
private val whoToSubscribeFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,
seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,
stringCenter = stringCenterProvider.get(),
encodedFeedbackRequest =
Some(HomeWhoToSubscribeFeedbackActionInfoBuilder.EncodedFeedbackRequest)
)
override def apply(
query: PipelineQuery,
candidate: UserCandidate,
candidateFeatures: FeatureMap
): Option[FeedbackActionInfo] =
whoToSubscribeFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)
}

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/LikedBySocialContextBuilder.scala Normal file
View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class LikedBySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
contextType = LikeGeneralContextType,
stringCenter = stringCenter,
oneUserString = externalStrings.socialContextOneUserLikedString,
twoUsersString = externalStrings.socialContextTwoUsersLikedString,
moreUsersString = externalStrings.socialContextMoreUsersLikedString,
timelineTitle = externalStrings.socialContextLikedByTimelineTitle
)
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// Liked by users are valid only if they pass both the SGS and Perspective filters.
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
engagerSocialContextBuilder(
socialContextIds = validLikedByUserIds,
query = query,
candidateFeatures = candidateFeatures
)
}
}

50
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ListsSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,50 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelineservice.suggests.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* "Your Lists" will be rendered for the context and a url link for your lists.
*/
@Singleton
case class ListsSocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val listString = externalStrings.ownedSubscribedListsModuleHeaderString
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
candidateFeatures.get(SuggestTypeFeature) match {
case Some(suggestType) if suggestType == t.SuggestType.RankedListTweet =>
val userName = query.features.flatMap(_.getOrElse(UserScreenNameFeature, None))
Some(
GeneralContext(
contextType = ListGeneralContextType,
text = stringCenter.prepare(listString),
url = userName.map(name => ""),
contextImageUrls = None,
landingUrl = None
))
case _ => None
}
}
}

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/MuteUserChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class MuteUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val userIdOpt =
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
candidateFeatures.getOrElse(SourceUserIdFeature, None)
else candidateFeatures.getOrElse(AuthorIdFeature, None)
userIdOpt.flatMap { userId =>
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.muteUserString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.SpeakerOff),
richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),
subprompt = None
)
}
}
}
}

70
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/NotInterestedTopicFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,70 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotInterestedTopicFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidateFeatures: FeatureMap
): Option[FeedbackAction] = {
val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)
val validFollowedByUserIds =
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
val validLikedByUserIds =
candidateFeatures
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
.filter(
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {
val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityType =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContext, topicContextFunctionalityType) match {
case (Some(topicId), Some(topicContextFunctionalityType))
if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||
topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>
Some(
FeedbackAction(
feedbackType = RichBehavior,
prompt = None,
confirmation = None,
childFeedbackActions = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior =
Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),
subprompt = None,
encodedFeedbackRequest = None
)
)
case _ => None
}
} else {
None
}
}
}

54
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/NotRelevantChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,54 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.common.{thriftscala => tlc}
import com.twitter.timelineservice.model.FeedbackInfo
import com.twitter.timelineservice.model.FeedbackMetadata
import com.twitter.timelineservice.{thriftscala => tlst}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class NotRelevantChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[ChildFeedbackAction] = {
val prompt = stringCenter.prepare(externalStrings.notRelevantString)
val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)
val feedbackMetadata = FeedbackMetadata(
engagementType = None,
entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),
ttl = Some(FeedbackUtil.FeedbackTtl))
val feedbackUrl = FeedbackInfo.feedbackUrl(
feedbackType = tlst.FeedbackType.NotRelevant,
feedbackMetadata = feedbackMetadata,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
Some(
ChildFeedbackAction(
feedbackType = NotRelevant,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = Some(feedbackUrl),
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = None,
richBehavior = None,
subprompt = None
)
)
}
}

43
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PopularInYourAreaSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,43 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelineservice.suggests.{thriftscala => st}
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class PopularInYourAreaSocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val popularInYourAreaString = externalStrings.socialContextPopularInYourAreaString
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)
if (suggestTypeOpt.contains(st.SuggestType.RecommendedTrendTweet)) {
Some(
GeneralContext(
contextType = LocationGeneralContextType,
text = stringCenter.prepare(popularInYourAreaString),
url = None,
contextImageUrls = None,
landingUrl = None
))
} else None
}
}

48
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PopularVideoSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,48 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelineservice.suggests.{thriftscala => st}
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
@Singleton
case class PopularVideoSocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val popularVideoString = externalStrings.socialContextPopularVideoString
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)
if (suggestTypeOpt.contains(st.SuggestType.MediaTweet)) {
Some(
GeneralContext(
contextType = SparkleGeneralContextType,
text = stringCenter.prepare(popularVideoString),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = ""
)
)
))
} else None
}
}

76
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ReceivedReplySocialContextBuilder.scala Normal file
View File

@ -0,0 +1,76 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Provider
import javax.inject.Singleton
/**
* Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.
*
* This function should only be called for the root Tweet of convo modules. This is enforced by
* [[HomeTweetSocialContextBuilder]].
*/
@Singleton
case class ReceivedReplySocialContextBuilder @Inject() (
externalStrings: HomeMixerExternalStrings,
@ProductScoped stringCenterProvider: Provider[StringCenter])
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
private val stringCenter = stringCenterProvider.get()
private val receivedReplyString = externalStrings.socialContextReceivedReply
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
// If these values are missing default to not showing a received a reply banner
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val inNetworkFocalTweet =
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)
if (inNetwork && !inNetworkFocalTweet) {
val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
val authorNameOpt = authorIdOpt.flatMap(realNames.get)
(authorIdOpt, authorNameOpt) match {
case (Some(authorId), Some(authorName)) =>
Some(
GeneralContext(
contextType = ConversationGeneralContextType,
text = stringCenter
.prepare(receivedReplyString, placeholders = Map("user1" -> authorName)),
url = None,
contextImageUrls = None,
landingUrl = Some(
Url(
urlType = DeepLink,
url = "",
urtEndpointOptions = None
)
)
)
)
case _ => None
}
} else {
None
}
}
}

37
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ReportTweetChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,37 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class ReportTweetChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(
candidate: TweetCandidate
): Option[ChildFeedbackAction] = {
Some(
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),
confirmation = None,
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Flag),
richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),
subprompt = None
)
)
}
}

38
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/RetweeterChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,38 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import com.twitter.timelines.service.{thriftscala => t}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class RetweeterChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
if (isRetweet) {
candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
userId = retweeterId,
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
promptExternalString = externalStrings.showFewerRetweetsString,
confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,
engagementType = t.FeedbackEngagementType.Retweet,
stringCenter = stringCenter,
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
)
}
} else None
}
}

42
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/TopicSocialContextBuilder.scala Normal file
View File

@ -0,0 +1,42 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class TopicSocialContextBuilder @Inject() ()
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
def apply(
query: PipelineQuery,
candidate: TweetCandidate,
candidateFeatures: FeatureMap
): Option[SocialContext] = {
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
if (!inNetwork) {
val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
val topicContextFunctionalityTypeOpt =
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
(topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {
case (Some(topicId), Some(topicContextFunctionalityType)) =>
Some(
TopicContext(
topicId = topicId.toString,
functionalityType = Some(topicContextFunctionalityType)
))
case _ => None
}
} else {
None
}
}
}

56
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/UnfollowUserChildFeedbackActionBuilder.scala Normal file
View File

@ -0,0 +1,56 @@
package com.twitter.home_mixer.functional_component.decorator.urt.builder
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
import com.twitter.stringcenter.client.StringCenter
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class UnfollowUserChildFeedbackActionBuilder @Inject() (
@ProductScoped stringCenter: StringCenter,
externalStrings: HomeMixerExternalStrings) {
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
if (isInNetwork) {
userIdOpt.flatMap { userId =>
val screenNamesMap =
candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
val userScreenNameOpt = screenNamesMap.get(userId)
userScreenNameOpt.map { userScreenName =>
val prompt = stringCenter.prepare(
externalStrings.unfollowUserString,
Map("username" -> userScreenName)
)
val confirmation = stringCenter.prepare(
externalStrings.unfollowUserConfirmationString,
Map("username" -> userScreenName)
)
ChildFeedbackAction(
feedbackType = RichBehavior,
prompt = Some(prompt),
confirmation = Some(confirmation),
feedbackUrl = None,
hasUndoAction = Some(true),
confirmationDisplayType = None,
clientEventInfo = None,
icon = Some(icon.Unfollow),
richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),
subprompt = None
)
}
}
} else None
}
}

56
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/AncestorFeatureHydrator.scala
View File

@ -1,56 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}
import com.twitter.tweetconvosvc.{thriftscala => tcs}
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class AncestorFeatureHydrator @Inject() (
conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Ancestor")
override val features: Set[Feature[_, _]] = Set(AncestorsFeature)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val ancestorsRequest = tcs.GetAncestorsRequest(Seq(candidate.id))
Stitch.callFuture(conversationServiceClient.getAncestors(ancestorsRequest)).map {
getAncestorsResponse =>
val ancestors = getAncestorsResponse.ancestors.headOption
.collect {
case tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult)
if ancestorsResult.nonEmpty =>
ancestorsResult.head.ancestors ++ getTruncatedRootTweet(ancestorsResult.head)
}.getOrElse(Seq.empty)
FeatureMapBuilder().add(AncestorsFeature, ancestors).build()
}
}
private def getTruncatedRootTweet(
ancestors: ta.TweetAncestors,
): Option[ta.TweetAncestor] = {
ancestors.conversationRootAuthorId.collect {
case rootAuthorId
if ancestors.state == ta.ReplyState.Partial &&
ancestors.ancestors.last.tweetId != ancestors.conversationId =>
ta.TweetAncestor(ancestors.conversationId, rootAuthorId)
}
}
}

95
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/AuthorFeatureHydrator.scala
View File

@ -1,95 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features.AuthorFeaturesAdapter
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.AuthorFeatureRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.timelines.author_features.v1.{thriftjava => af}
import com.twitter.util.Future
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object AuthorFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class AuthorFeatureHydrator @Inject() (
@Named(AuthorFeatureRepository) client: KeyValueRepository[Seq[Long], Long, af.AuthorFeatures],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("AuthorFeature")
override val features: Set[Feature[_, _]] = Set(AuthorFeature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val authorIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, af.AuthorFeatures]] =
if (authorIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(authorIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
FeatureMapBuilder()
.add(AuthorFeature, transformedValue)
.build()
}
}
}
}
private def postTransformer(authorFeatures: Try[Option[af.AuthorFeatures]]): Try[DataRecord] = {
authorFeatures.map { features =>
AuthorFeaturesAdapter.adaptToDataRecords(features).asScala.head
}
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

45
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/BUILD.bazel
View File

@ -4,95 +4,56 @@ scala_library(
strict_deps = True,
tags = ["bazel-compatible"],
dependencies = [
"3rdparty/jvm/com/twitter/storehaus:core",
"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi",
"configapi/configapi-decider",
"finatra/inject/inject-core/src/main/scala",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/earlybird",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie",
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content",
"joinkey/src/main/scala/com/twitter/joinkey/context",
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/topics",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph",
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator",
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util",
"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common",
"representation-scorer/server/src/main/thrift:thrift-scala",
"servo/repo/src/main/scala",
"snowflake/src/main/scala/com/twitter/snowflake/id",
"src/java/com/twitter/ml/api/constant",
"src/java/com/twitter/search/common/util/lang",
"src/scala/com/twitter/ml/api/util",
"src/scala/com/twitter/timelines/prediction/adapters/real_graph",
"src/scala/com/twitter/timelines/prediction/adapters/realtime_interaction_graph",
"src/scala/com/twitter/timelines/prediction/adapters/twistly",
"src/scala/com/twitter/timelines/prediction/adapters/two_hop_features",
"src/scala/com/twitter/timelines/prediction/common/util",
"src/scala/com/twitter/timelines/prediction/features/common",
"src/scala/com/twitter/timelines/prediction/features/realtime_interaction_graph",
"src/scala/com/twitter/timelines/prediction/features/recap",
"src/scala/com/twitter/timelines/prediction/features/time_features",
"src/scala/com/twitter/timelines/prediction/adapters/request_context",
"src/thrift/com/twitter/gizmoduck:thrift-scala",
"src/thrift/com/twitter/ml/api:data-java",
"src/thrift/com/twitter/ml/api:embedding-java",
"src/thrift/com/twitter/onboarding/relevance/features:features-java",
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
"src/thrift/com/twitter/search:earlybird-scala",
"src/thrift/com/twitter/search/common:constants-java",
"src/thrift/com/twitter/socialgraph:thrift-scala",
"src/thrift/com/twitter/spam/rtf:safety-result-scala",
"src/thrift/com/twitter/timelineranker:thrift-scala",
"src/thrift/com/twitter/timelines/author_features:thrift-java",
"src/thrift/com/twitter/timelines/conversation_features:conversation_features-scala",
"src/thrift/com/twitter/timelines/impression:thrift-scala",
"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala",
"src/thrift/com/twitter/timelines/real_graph:real_graph-scala",
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
"src/thrift/com/twitter/topic_recos:topic_recos-thrift-java",
"src/thrift/com/twitter/tweetypie:service-scala",
"src/thrift/com/twitter/tweetypie:tweet-scala",
"src/thrift/com/twitter/user_session_store:thrift-java",
"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala",
"src/thrift/com/twitter/wtf/real_time_interaction_graph:wtf-real_time_interaction_graph-thrift-java",
"stitch/stitch-core",
"stitch/stitch-gizmoduck",
"stitch/stitch-socialgraph",
"stitch/stitch-timelineservice",
"stitch/stitch-tweetypie",
"strato/config/columns/topic-signals/tsp",
"strato/config/columns/topic-signals/tsp:tsp-strato-client",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/feedback",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence",
"timelines/src/main/scala/com/twitter/timelines/clients/strato/twistly",
"timelines/src/main/scala/com/twitter/timelines/clients/user_tweet_entity_graph",
"timelines/src/main/scala/com/twitter/timelines/clients/manhattan/store",
"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter",
"timelines/src/main/scala/com/twitter/timelines/impressionstore/store",
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
"topic-social-proof/server/src/main/thrift:thrift-scala",
"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting",
"tweetconvosvc/thrift/src/main/thrift:thrift-scala",
"twitter-config/yaml",
"user_session_store/src/main/scala/com/twitter/user_session_store",
"util/util-core",
],

129
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/EarlybirdFeatureHydrator.scala
View File

@ -1,129 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.earlybird.EarlybirdAdapter
import com.twitter.home_mixer.model.HomeFeatures.DeviceLanguageFeature
import com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.home_mixer.util.earlybird.EarlybirdResponseUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.earlybird.{thriftscala => eb}
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Return
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object EarlybirdDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class EarlybirdFeatureHydrator @Inject() (
@Named(EarlybirdRepository) client: KeyValueRepository[
(Seq[Long], Long),
Long,
eb.ThriftSearchResult
],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Earlybird")
override val features: Set[Feature[_, _]] =
Set(EarlybirdDataRecordFeature, EarlybirdFeature, TweetUrlsFeature)
override val statScope: String = identifier.toString
private val scopedStatsReceiver = statsReceiver.scope(statScope)
private val originalKeyFoundCounter = scopedStatsReceiver.counter("originalKey/found")
private val originalKeyLossCounter = scopedStatsReceiver.counter("originalKey/loss")
private val ebFeaturesNotExistPredicate: CandidateWithFeatures[TweetCandidate] => Boolean =
candidate => candidate.features.getOrElse(EarlybirdFeature, None).isEmpty
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val candidatesToHydrate = candidates.filter { candidate =>
val isEmpty = ebFeaturesNotExistPredicate(candidate)
if (isEmpty) originalKeyLossCounter.incr() else originalKeyFoundCounter.incr()
isEmpty
}
Stitch
.callFuture(client((candidatesToHydrate.map(_.candidate.id), query.getRequiredUserId)))
.map(handleResponse(query, candidates, _))
}
private def handleResponse(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]],
results: KeyValueResult[Long, eb.ThriftSearchResult]
): Seq[FeatureMap] = {
val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)
val userLanguages = queryFeatureMap.getOrElse(UserLanguagesFeature, Seq.empty)
val uiLanguageCode = queryFeatureMap.getOrElse(DeviceLanguageFeature, None)
val screenName = queryFeatureMap.getOrElse(UserScreenNameFeature, None)
val searchResults = candidates
.filter(ebFeaturesNotExistPredicate).map { candidate =>
observedGet(Some(candidate.candidate.id), results)
}.collect {
case Return(Some(value)) => value
}
val tweetIdToEbFeatures = EarlybirdResponseUtil.getOONTweetThriftFeaturesByTweetId(
searcherUserId = query.getRequiredUserId,
screenName = screenName,
userLanguages = userLanguages,
uiLanguageCode = uiLanguageCode,
searchResults = searchResults
)
candidates.map { candidate =>
val hydratedEbFeatures = tweetIdToEbFeatures.get(candidate.candidate.id)
val earlybirdFeatures =
if (hydratedEbFeatures.nonEmpty) hydratedEbFeatures
else candidate.features.getOrElse(EarlybirdFeature, None)
val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)
val sourceTweetEbFeatures =
candidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
val originalTweetEbFeatures =
if (candidateIsRetweet && sourceTweetEbFeatures.nonEmpty)
sourceTweetEbFeatures
else earlybirdFeatures
val earlybirdDataRecord =
EarlybirdAdapter.adaptToDataRecords(originalTweetEbFeatures).asScala.head
FeatureMapBuilder()
.add(EarlybirdFeature, earlybirdFeatures)
.add(EarlybirdDataRecordFeature, earlybirdDataRecord)
.add(TweetUrlsFeature, earlybirdFeatures.flatMap(_.urlsList).getOrElse(Seq.empty))
.build()
}
}
}

8
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FeedbackHistoryQueryFeatureHydrator.scala
View File

@ -1,12 +1,10 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableFeedbackFatigueParam
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
@ -17,16 +15,12 @@ import javax.inject.Singleton
@Singleton
case class FeedbackHistoryQueryFeatureHydrator @Inject() (
feedbackHistoryClient: FeedbackHistoryManhattanClient)
extends QueryFeatureHydrator[PipelineQuery]
with Conditionally[PipelineQuery] {
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FeedbackHistory")
override val features: Set[Feature[_, _]] = Set(FeedbackHistoryFeature)
override def onlyIf(query: PipelineQuery): Boolean =
query.params(EnableFeedbackFatigueParam)
override def hydrate(
query: PipelineQuery
): Stitch[FeatureMap] =

84
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FocalTweetFeatureHydrator.scala
View File

@ -1,84 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetScreenNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
/**
* Social context for convo modules is hydrated on the root Tweet but needs info about the focal
* Tweet (e.g. author) to render the banner. This hydrator copies focal Tweet data into the root.
*/
@Singleton
class FocalTweetFeatureHydrator @Inject() ()
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FocalTweet")
override val features: Set[Feature[_, _]] = Set(
FocalTweetAuthorIdFeature,
FocalTweetInNetworkFeature,
FocalTweetRealNamesFeature,
FocalTweetScreenNamesFeature
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(FocalTweetAuthorIdFeature, None)
.add(FocalTweetInNetworkFeature, None)
.add(FocalTweetRealNamesFeature, None)
.add(FocalTweetScreenNamesFeature, None)
.build()
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
// Build a map of all the focal tweets to their corresponding features
val focalTweetIdToFeatureMap = candidates.flatMap { candidate =>
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
if (focalTweetId.contains(candidate.candidate.id)) {
Some(candidate.candidate.id -> candidate.features)
} else None
}.toMap
val updatedFeatureMap = candidates.map { candidate =>
val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)
val conversationId = candidate.features.getOrElse(ConversationModuleIdFeature, None)
// Check if the candidate is a root tweet and ensure its focal tweet's features are available
if (conversationId.contains(candidate.candidate.id)
&& focalTweetId.exists(focalTweetIdToFeatureMap.contains)) {
val featureMap = focalTweetIdToFeatureMap.get(focalTweetId.get).get
FeatureMapBuilder()
.add(FocalTweetAuthorIdFeature, featureMap.getOrElse(AuthorIdFeature, None))
.add(FocalTweetInNetworkFeature, Some(featureMap.getOrElse(InNetworkFeature, true)))
.add(
FocalTweetRealNamesFeature,
Some(featureMap.getOrElse(RealNamesFeature, Map.empty[Long, String])))
.add(
FocalTweetScreenNamesFeature,
Some(featureMap.getOrElse(ScreenNamesFeature, Map.empty[Long, String])))
.build()
} else DefaultFeatureMap
}
Stitch.value(updatedFeatureMap)
}
}

41
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FollowedTopicsQueryFeatureHydrator.scala
View File

@ -1,41 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.UserFollowedTopicsCountFeature
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.component_library.candidate_source.topics.FollowedTopicsCandidateSource
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
case class FollowedTopicsQueryFeatureHydrator @Inject() (
followedTopicsCandidateSource: FollowedTopicsCandidateSource)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FollowedTopics")
override val features: Set[Feature[_, _]] = Set(UserFollowedTopicsCountFeature)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val request: StratoKeyView[Long, Unit] = StratoKeyView(query.getRequiredUserId, Unit)
followedTopicsCandidateSource(request)
.map { topics =>
FeatureMapBuilder().add(UserFollowedTopicsCountFeature, Some(topics.size)).build()
}.handle {
case _ => FeatureMapBuilder().add(UserFollowedTopicsCountFeature, None).build()
}
}
override val alerts = Seq(
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9),
HomeMixerAlertConfig.BusinessHours.defaultLatencyAlert(1500.millis)
)
}

58
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GizmoduckAuthorSafetyFeatureHydrator.scala
View File

@ -1,58 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.gizmoduck.{thriftscala => gt}
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature
import com.twitter.home_mixer.param.HomeGlobalParams.EnableGizmoduckAuthorSafetyFeatureHydratorParam
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.Conditionally
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.stitch.gizmoduck.Gizmoduck
import javax.inject.Inject
import javax.inject.Singleton
@Singleton
class GizmoduckAuthorSafetyFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with Conditionally[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("GizmoduckAuthorSafety")
override val features: Set[Feature[_, _]] = Set(AuthorIsBlueVerifiedFeature)
override def onlyIf(query: PipelineQuery): Boolean =
query.params(EnableGizmoduckAuthorSafetyFeatureHydratorParam)
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Safety)
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val authorIdOption = existingFeatures.getOrElse(AuthorIdFeature, None)
val blueVerifiedStitch = authorIdOption
.map { authorId =>
gizmoduck
.getUserById(
userId = authorId,
queryFields = queryFields
)
.map { _.safety.flatMap(_.isBlueVerified).getOrElse(false) }
}.getOrElse(Stitch.False)
blueVerifiedStitch.map { isBlueVerified =>
FeatureMapBuilder()
.add(AuthorIsBlueVerifiedFeature, isBlueVerified)
.build()
}
}
}

105
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GraphTwoHopFeatureHydrator.scala
View File

@ -1,105 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.graph_feature_service.{thriftscala => gfs}
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.home_mixer.model.HomeFeatures.IsExtendedReplyFeature
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.home_mixer.util.ReplyRetweetUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.two_hop_features.TwoHopFeaturesAdapter
import com.twitter.util.Try
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
import scala.collection.JavaConverters._
object GraphTwoHopFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class GraphTwoHopFeatureHydrator @Inject() (
@Named(GraphTwoHopRepository) client: KeyValueRepository[(Seq[Long], Long), Long, Seq[
gfs.IntersectionValue
]],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GraphTwoHop")
override val features: Set[Feature[_, _]] = Set(GraphTwoHopFeature, FollowedByUserIdsFeature)
override val statScope: String = identifier.toString
private val twoHopFeaturesAdapter = new TwoHopFeaturesAdapter
private val FollowFeatureType = gfs.FeatureType(gfs.EdgeType.Following, gfs.EdgeType.FollowedBy)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
// Apply filters to in network candidates for ExtendedReplyAncestors and retweets.
// ExtendedReplyAncestors should also be in candidates. No filter for oon.
val (inNetworkCandidates, oonCandidates) = candidates.partition { candidate =>
candidate.features.getOrElse(InNetworkFeature, false)
}
val inNetworkReplyToAncestorTweet =
ReplyRetweetUtil.replyToAncestorTweetCandidatesMap(inNetworkCandidates)
val inNetworkExtendedReplyAncestors = inNetworkCandidates
.filter(_.features.getOrElse(IsExtendedReplyFeature, false)).flatMap { inNetworkCandidate =>
inNetworkReplyToAncestorTweet.get(inNetworkCandidate.candidate.id)
}.flatten
val inNetworkCandidatesToHydrate = inNetworkExtendedReplyAncestors ++
inNetworkCandidates.filter(_.features.getOrElse(IsRetweetFeature, false))
val candidatesToHydrate = (inNetworkCandidatesToHydrate ++ oonCandidates)
.flatMap(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features)).distinct
val response = Stitch.callFuture(client((candidatesToHydrate, query.getRequiredUserId)))
response.map { result =>
candidates.map { candidate =>
val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)
val value = observedGet(key = originalAuthorId, keyValueResult = result)
val transformedValue = postTransformer(value)
val followedByUserIds = value.toOption.flatMap(getFollowedByUserIds(_)).getOrElse(Seq.empty)
FeatureMapBuilder()
.add(GraphTwoHopFeature, transformedValue)
.add(FollowedByUserIdsFeature, followedByUserIds)
.build()
}
}
}
private def getFollowedByUserIds(input: Option[Seq[gfs.IntersectionValue]]): Option[Seq[Long]] =
input.map(_.filter(_.featureType == FollowFeatureType).flatMap(_.intersectionIds).flatten)
private def postTransformer(input: Try[Option[Seq[gfs.IntersectionValue]]]): Try[DataRecord] =
input.map(twoHopFeaturesAdapter.adaptToDataRecords(_).asScala.head)
}

45
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ImpressionBloomFilterQueryFeatureHydrator.scala
View File

@ -3,6 +3,7 @@ package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature
import com.twitter.home_mixer.model.request.HasSeenTweetIds
import com.twitter.home_mixer.param.HomeGlobalParams.ImpressionBloomFilterFalsePositiveRateParam
import com.twitter.home_mixer.service.HomeMixerAlertConfig
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
@ -11,7 +12,8 @@ import com.twitter.product_mixer.core.functional_component.feature_hydrator.Quer
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.impressionbloomfilter.{thriftscala => t}
import com.twitter.timelines.clients.manhattan.store.ManhattanStoreClient
import com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}
import com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilter
import javax.inject.Inject
import javax.inject.Singleton
@ -19,36 +21,39 @@ import javax.inject.Singleton
@Singleton
case class ImpressionBloomFilterQueryFeatureHydrator[
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
bloomFilter: ImpressionBloomFilter)
extends QueryFeatureHydrator[Query] {
bloomFilterClient: ManhattanStoreClient[
blm.ImpressionBloomFilterKey,
blm.ImpressionBloomFilterSeq
]) extends QueryFeatureHydrator[Query] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
"ImpressionBloomFilter")
private val ImpressionBloomFilterTTL = 7.day
private val ImpressionBloomFilterFalsePositiveRate = 0.002
override val features: Set[Feature[_, _]] = Set(ImpressionBloomFilterFeature)
private val SurfaceArea = t.SurfaceArea.HomeTimeline
private val SurfaceArea = blm.SurfaceArea.HomeTimeline
override def hydrate(query: Query): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
bloomFilter.getBloomFilterSeq(userId, SurfaceArea).map { bloomFilterSeq =>
val updatedBloomFilterSeq =
if (query.seenTweetIds.forall(_.isEmpty)) bloomFilterSeq
else {
bloomFilter.addElements(
userId = userId,
surfaceArea = SurfaceArea,
tweetIds = query.seenTweetIds.get,
bloomFilterEntrySeq = bloomFilterSeq,
timeToLive = ImpressionBloomFilterTTL,
falsePositiveRate = ImpressionBloomFilterFalsePositiveRate
)
}
FeatureMapBuilder().add(ImpressionBloomFilterFeature, updatedBloomFilterSeq).build()
}
bloomFilterClient
.get(blm.ImpressionBloomFilterKey(userId, SurfaceArea))
.map(_.getOrElse(blm.ImpressionBloomFilterSeq(Seq.empty)))
.map { bloomFilterSeq =>
val updatedBloomFilterSeq =
if (query.seenTweetIds.forall(_.isEmpty)) bloomFilterSeq
else {
ImpressionBloomFilter.addSeenTweetIds(
surfaceArea = SurfaceArea,
tweetIds = query.seenTweetIds.get,
bloomFilterSeq = bloomFilterSeq,
timeToLive = ImpressionBloomFilterTTL,
falsePositiveRate = query.params(ImpressionBloomFilterFalsePositiveRateParam)
)
}
FeatureMapBuilder().add(ImpressionBloomFilterFeature, updatedBloomFilterSeq).build()
}
}
override val alerts = Seq(

41
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/InNetworkFeatureHydrator.scala Normal file
View File

@ -0,0 +1,41 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
import com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
object InNetworkFeatureHydrator
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("InNetwork")
override val features: Set[Feature[_, _]] = Set(InNetworkFeature)
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val viewerId = query.getRequiredUserId
val followedUserIds = query.features.get.get(SGSFollowedUsersFeature).toSet
val featureMaps = candidates.map { candidate =>
// We use authorId and not sourceAuthorId here so that retweets are defined as in network
val isInNetworkOpt = candidate.features.getOrElse(AuthorIdFeature, None).map { authorId =>
// Users cannot follow themselves but this is in network by definition
val isSelfTweet = authorId == viewerId
isSelfTweet || followedUserIds.contains(authorId)
}
FeatureMapBuilder().add(InNetworkFeature, isInNetworkOpt.getOrElse(true)).build()
}
Stitch.value(featureMaps)
}
}

42
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ListMembersQueryFeatureHydrator.scala
View File

@ -1,42 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.request.HasListId
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.socialgraph.{thriftscala => sg}
import com.twitter.stitch.Stitch
import com.twitter.stitch.socialgraph.SocialGraph
import javax.inject.Inject
import javax.inject.Singleton
case object ListMembersFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {
override val defaultValue: Seq[Long] = Seq.empty
}
@Singleton
class ListMembersQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)
extends QueryFeatureHydrator[PipelineQuery with HasListId] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ListMembers")
override val features: Set[Feature[_, _]] = Set(ListMembersFeature)
private val MaxRecentMembers = 10
override def hydrate(query: PipelineQuery with HasListId): Stitch[FeatureMap] = {
val request = sg.IdsRequest(
relationships = Seq(sg
.SrcRelationship(query.listId, sg.RelationshipType.ListHasMember, hasRelationship = true)),
pageRequest = Some(sg.PageRequest(selectAll = Some(true), count = Some(MaxRecentMembers)))
)
socialGraph.ids(request).map(_.ids).map { listMembers =>
FeatureMapBuilder().add(ListMembersFeature, listMembers).build()
}
}
}

81
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/MetricCenterUserCountingFeatureHydrator.scala
View File

@ -1,81 +0,0 @@
package com.twitter.home_mixer
package functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.param.HomeMixerInjectionNames.MetricCenterUserCountingFeatureRepository
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.onboarding.relevance.features.{thriftjava => rf}
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.keyvalue.KeyValueResult
import com.twitter.servo.repository.KeyValueRepository
import com.twitter.stitch.Stitch
import com.twitter.util.Future
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object MetricCenterUserCountingFeature
extends Feature[TweetCandidate, Option[rf.MCUserCountingFeatures]]
@Singleton
class MetricCenterUserCountingFeatureHydrator @Inject() (
@Named(MetricCenterUserCountingFeatureRepository) client: KeyValueRepository[Seq[
Long
], Long, rf.MCUserCountingFeatures],
override val statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("MetricCenterUserCounting")
override val features: Set[Feature[_, _]] = Set(MetricCenterUserCountingFeature)
override val statScope: String = identifier.toString
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
Stitch.callFuture {
val possiblyAuthorIds = extractKeys(candidates)
val userIds = possiblyAuthorIds.flatten
val response: Future[KeyValueResult[Long, rf.MCUserCountingFeatures]] = if (userIds.isEmpty) {
Future.value(KeyValueResult.empty)
} else {
client(userIds)
}
response.map { result =>
possiblyAuthorIds.map { possiblyAuthorId =>
val value = observedGet(key = possiblyAuthorId, keyValueResult = result)
FeatureMapBuilder()
.add(MetricCenterUserCountingFeature, value)
.build()
}
}
}
}
private def extractKeys(
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Seq[Option[Long]] = {
candidates.map { candidate =>
candidate.features
.getTry(AuthorIdFeature)
.toOption
.flatten
}
}
}

29
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PersistenceStoreQueryFeatureHydrator.scala
View File

@ -2,8 +2,10 @@ package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.conversions.DurationOps._
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetPreviewIdsFeature
import com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature
import com.twitter.home_mixer.model.request.FollowingProduct
import com.twitter.home_mixer.model.request.ForYouProduct
@ -27,17 +29,27 @@ import javax.inject.Singleton
@Singleton
case class PersistenceStoreQueryFeatureHydrator @Inject() (
timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3])
timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3],
statsReceiver: StatsReceiver)
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("PersistenceStore")
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val servedTweetIdsSizeStat = scopedStatsReceiver.stat("ServedTweetIdsSize")
private val WhoToFollowExcludedUserIdsLimit = 1000
private val ServedTweetIdsDuration = 1.hour
private val ServedTweetIdsDuration = 10.minutes
private val ServedTweetIdsLimit = 100
private val ServedTweetPreviewIdsDuration = 10.hours
private val ServedTweetPreviewIdsLimit = 10
override val features: Set[Feature[_, _]] =
Set(ServedTweetIdsFeature, PersistenceEntriesFeature, WhoToFollowExcludedUserIdsFeature)
Set(
ServedTweetIdsFeature,
ServedTweetPreviewIdsFeature,
PersistenceEntriesFeature,
WhoToFollowExcludedUserIdsFeature)
private val supportedClients = Seq(
ClientPlatform.IPhone,
@ -80,8 +92,19 @@ case class PersistenceStoreQueryFeatureHydrator @Inject() (
.flatMap(
_.entries.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetIdsLimit))
servedTweetIdsSizeStat.add(servedTweetIds.size)
val servedTweetPreviewIds = timelineResponses
.filter(_.clientPlatform == clientPlatform)
.filter(_.servedTime >= Time.now - ServedTweetPreviewIdsDuration)
.sortBy(-_.servedTime.inMilliseconds)
.flatMap(_.entries
.filter(_.entityIdType == EntityIdType.TweetPreview)
.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetPreviewIdsLimit))
FeatureMapBuilder()
.add(ServedTweetIdsFeature, servedTweetIds)
.add(ServedTweetPreviewIdsFeature, servedTweetPreviewIds)
.add(PersistenceEntriesFeature, timelineResponses)
.add(WhoToFollowExcludedUserIdsFeature, whoToFollowUserIds)
.build()

7
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PerspectiveFilteredSocialContextFeatureHydrator.scala
View File

@ -10,6 +10,7 @@ import com.twitter.product_mixer.core.functional_component.feature_hydrator.Bulk
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.util.OffloadFuturePools
import com.twitter.stitch.Stitch
import com.twitter.stitch.timelineservice.TimelineService
import com.twitter.stitch.timelineservice.TimelineService.GetPerspectives
@ -37,10 +38,10 @@ class PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {
val engagingUserIdtoTweetId = candidates.flatMap { candidate =>
candidate.features
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
.getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)
.map(favoritedBy => favoritedBy -> candidate.candidate.id)
}
@ -59,7 +60,7 @@ class PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService
candidates.map { candidate =>
val perspectiveFilteredFavoritedByUserIds: Seq[Long] = candidate.features
.get(FavoritedByUserIdsFeature).take(MaxCountUsers)
.getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)
.filter { userId => validUserIdTweetIds.contains((userId, candidate.candidate.id)) }
FeatureMapBuilder()

6
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphInNetworkScoresQueryFeatureHydrator.scala
View File

@ -32,11 +32,15 @@ case class RealGraphInNetworkScoresQueryFeatureHydrator @Inject() (
val realGraphScoresFeatures = realGraphFollowedUsers
.getOrElse(Seq.empty)
.sortBy(-_.score)
.map(candidate => candidate.userId -> candidate.score)
.map(candidate => candidate.userId -> scaleScore(candidate.score))
.take(RealGraphCandidateCount)
.toMap
FeatureMapBuilder().add(RealGraphInNetworkScoresFeature, realGraphScoresFeatures).build()
}
}
// Rescale Real Graph v2 scores from [0,1] to the v1 scores distribution [1,2.97]
private def scaleScore(score: Double): Double =
if (score >= 0.0 && score <= 1.0) score * 1.97 + 1.0 else score
}

48
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphQueryFeatureHydrator.scala
View File

@ -1,48 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphFeatureRepository
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.repository.Repository
import com.twitter.timelines.real_graph.{thriftscala => rg}
import com.twitter.stitch.Stitch
import com.twitter.timelines.model.UserId
import com.twitter.timelines.real_graph.v1.thriftscala.RealGraphEdgeFeatures
import com.twitter.user_session_store.{thriftscala => uss}
import javax.inject.Inject
import javax.inject.Named
import javax.inject.Singleton
object RealGraphFeatures extends Feature[PipelineQuery, Option[Map[UserId, RealGraphEdgeFeatures]]]
@Singleton
class RealGraphQueryFeatureHydrator @Inject() (
@Named(RealGraphFeatureRepository) repository: Repository[Long, Option[uss.UserSession]])
extends QueryFeatureHydrator[PipelineQuery] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphFeatures")
override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
Stitch.callFuture {
repository(query.getRequiredUserId).map { userSession =>
val realGraphFeaturesMap = userSession.flatMap { userSession =>
userSession.realGraphFeatures.collect {
case rg.RealGraphFeatures.V1(realGraphFeatures) =>
val edgeFeatures = realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures
edgeFeatures.map { edge => edge.destId -> edge }.toMap
}
}
FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()
}
}
}
}

123
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphViewerAuthorFeatureHydrator.scala
View File

@ -1,123 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature
import com.twitter.home_mixer.util.MissingKeyException
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphFeaturesAdapter
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
import com.twitter.timelines.real_graph.{thriftscala => rg}
import com.twitter.util.Throw
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealGraphViewerAuthorDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
object RealGraphViewerAuthorsDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealGraphViewerAuthorFeatureHydrator @Inject() ()
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphViewerAuthor")
override val features: Set[Feature[_, _]] =
Set(RealGraphViewerAuthorDataRecordFeature, RealGraphViewerAuthorsDataRecordFeature)
private val realGraphEdgeFeaturesAdapter = new RealGraphFeaturesAdapter
private val realGraphEdgeFeaturesCombineAdapter =
new RealGraphEdgeFeaturesCombineAdapter(prefix = "authors.realgraph")
private val MissingKeyFeatureMap = FeatureMapBuilder()
.add(RealGraphViewerAuthorDataRecordFeature, Throw(MissingKeyException))
.add(RealGraphViewerAuthorsDataRecordFeature, Throw(MissingKeyException))
.build()
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val viewerId = query.getRequiredUserId
val realGraphFeatures = query.features
.flatMap(_.getOrElse(RealGraphFeatures, None))
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
val result: FeatureMap = existingFeatures.getOrElse(AuthorIdFeature, None) match {
case Some(authorId) =>
val realGraphAuthorFeatures =
getRealGraphViewerAuthorFeatures(viewerId, authorId, realGraphFeatures)
val realGraphAuthorDataRecord = realGraphEdgeFeaturesAdapter
.adaptToDataRecords(realGraphAuthorFeatures).asScala.headOption.getOrElse(new DataRecord)
val combinedRealGraphFeaturesDataRecord = for {
inReplyToAuthorId <- existingFeatures.getOrElse(InReplyToUserIdFeature, None)
} yield {
val combinedRealGraphFeatures =
getCombinedRealGraphFeatures(Seq(authorId, inReplyToAuthorId), realGraphFeatures)
realGraphEdgeFeaturesCombineAdapter
.adaptToDataRecords(Some(combinedRealGraphFeatures)).asScala.headOption
.getOrElse(new DataRecord)
}
FeatureMapBuilder()
.add(RealGraphViewerAuthorDataRecordFeature, realGraphAuthorDataRecord)
.add(
RealGraphViewerAuthorsDataRecordFeature,
combinedRealGraphFeaturesDataRecord.getOrElse(new DataRecord))
.build()
case _ => MissingKeyFeatureMap
}
Stitch(result)
}
private def getRealGraphViewerAuthorFeatures(
viewerId: Long,
authorId: Long,
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
): rg.UserRealGraphFeatures = {
realGraphEdgeFeaturesMap.get(authorId) match {
case Some(realGraphEdgeFeatures) =>
rg.UserRealGraphFeatures(
srcId = viewerId,
features = rg.RealGraphFeatures.V1(
v1.RealGraphFeatures(edgeFeatures = Seq(realGraphEdgeFeatures))))
case _ =>
rg.UserRealGraphFeatures(
srcId = viewerId,
features = rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = Seq.empty)))
}
}
}
object RealGraphViewerAuthorFeatureHydrator {
def getCombinedRealGraphFeatures(
userIds: Seq[Long],
realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]
): rg.RealGraphFeatures = {
val edgeFeatures = userIds.flatMap(realGraphEdgeFeaturesMap.get)
rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = edgeFeatures))
}
}

74
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphViewerRelatedUsersFeatureHydrator.scala
View File

@ -1,74 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
import com.twitter.home_mixer.util.CandidatesUtil
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter
import com.twitter.timelines.real_graph.v1.{thriftscala => v1}
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealGraphViewerRelatedUsersDataRecordFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealGraphViewerRelatedUsersFeatureHydrator @Inject() ()
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealGraphViewerRelatedUsers")
override val features: Set[Feature[_, _]] = Set(RealGraphViewerRelatedUsersDataRecordFeature)
private val RealGraphEdgeFeaturesCombineAdapter = new RealGraphEdgeFeaturesCombineAdapter
override def apply(
query: PipelineQuery,
candidate: TweetCandidate,
existingFeatures: FeatureMap
): Stitch[FeatureMap] = {
val realGraphQueryFeatures = query.features
.flatMap(_.getOrElse(RealGraphFeatures, None))
.getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])
val allRelatedUserIds = getRelatedUserIds(existingFeatures)
val realGraphFeatures =
RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures(
allRelatedUserIds,
realGraphQueryFeatures)
val realGraphFeaturesDataRecord = RealGraphEdgeFeaturesCombineAdapter
.adaptToDataRecords(Some(realGraphFeatures)).asScala.headOption
.getOrElse(new DataRecord)
Stitch.value {
FeatureMapBuilder()
.add(RealGraphViewerRelatedUsersDataRecordFeature, realGraphFeaturesDataRecord)
.build()
}
}
private def getRelatedUserIds(features: FeatureMap): Seq[Long] = {
(CandidatesUtil.getEngagerUserIds(features) ++
features.getOrElse(AuthorIdFeature, None) ++
features.getOrElse(MentionUserIdFeature, Seq.empty) ++
features.getOrElse(SourceUserIdFeature, None) ++
features.getOrElse(DirectedAtUserIdFeature, None)).distinct
}
}

64
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealTimeInteractionGraphEdgeFeatureHydrator.scala
View File

@ -1,64 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
import com.twitter.ml.api.DataRecord
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.realtime_interaction_graph.RealTimeInteractionGraphFeaturesAdapter
import com.twitter.timelines.prediction.features.realtime_interaction_graph.RealTimeInteractionGraphEdgeFeatures
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
import scala.collection.JavaConverters._
object RealTimeInteractionGraphEdgeFeature
extends DataRecordInAFeature[TweetCandidate]
with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {
override def defaultValue: DataRecord = new DataRecord()
}
@Singleton
class RealTimeInteractionGraphEdgeFeatureHydrator @Inject() ()
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
"RealTimeInteractionGraphEdge")
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphEdgeFeature)
private val realTimeInteractionGraphFeaturesAdapter = new RealTimeInteractionGraphFeaturesAdapter
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val userVertex =
query.features.flatMap(_.getOrElse(RealTimeInteractionGraphUserVertexQueryFeature, None))
val realTimeInteractionGraphFeaturesMap =
userVertex.map(RealTimeInteractionGraphEdgeFeatures(_, Time.now))
Stitch.value {
candidates.map { candidate =>
val feature = candidate.features.getOrElse(AuthorIdFeature, None).flatMap { authorId =>
realTimeInteractionGraphFeaturesMap.flatMap(_.get(authorId))
}
FeatureMapBuilder()
.add(
RealTimeInteractionGraphEdgeFeature,
realTimeInteractionGraphFeaturesAdapter.adaptToDataRecords(feature).asScala.head)
.build()
}
}
}
}

49
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealTimeInteractionGraphUserVertexQueryFeatureHydrator.scala
View File

@ -1,49 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.google.inject.name.Named
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache
import com.twitter.home_mixer.util.ObservedKeyValueResultHandler
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.servo.cache.ReadCache
import com.twitter.stitch.Stitch
import com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}
import javax.inject.Inject
import javax.inject.Singleton
object RealTimeInteractionGraphUserVertexQueryFeature
extends Feature[PipelineQuery, Option[ig.UserVertex]]
@Singleton
class RealTimeInteractionGraphUserVertexQueryFeatureHydrator @Inject() (
@Named(RealTimeInteractionGraphUserVertexCache) client: ReadCache[Long, ig.UserVertex],
override val statsReceiver: StatsReceiver)
extends QueryFeatureHydrator[PipelineQuery]
with ObservedKeyValueResultHandler {
override val identifier: FeatureHydratorIdentifier =
FeatureHydratorIdentifier("RealTimeInteractionGraphUserVertex")
override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphUserVertexQueryFeature)
override val statScope: String = identifier.toString
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
val userId = query.getRequiredUserId
Stitch.callFuture(
client.get(Seq(userId)).map { results =>
val feature = observedGet(key = Some(userId), keyValueResult = results)
FeatureMapBuilder()
.add(RealTimeInteractionGraphUserVertexQueryFeature, feature)
.build()
}
)
}
}

196
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ReplyFeatureHydrator.scala
View File

@ -1,196 +0,0 @@
package com.twitter.home_mixer.functional_component.feature_hydrator
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.home_mixer.model.HomeFeatures._
import com.twitter.home_mixer.util.ReplyRetweetUtil
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
import com.twitter.product_mixer.core.feature.Feature
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.search.common.features.thriftscala.ThriftTweetFeatures
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import com.twitter.timelines.conversation_features.v1.thriftscala.ConversationFeatures
import com.twitter.util.Duration
import com.twitter.util.Time
import javax.inject.Inject
import javax.inject.Singleton
object InReplyToTweetHydratedEarlybirdFeature
extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]
/**
* The purpose of this hydrator is to
* 1) hydrate simple features into replies and their ancestor tweets
* 2) keep both the normal replies and ancestor source candidates, but hydrate into the candidates
* features useful for predicting the quality of the replies and source ancestor tweets.
*/
@Singleton
class ReplyFeatureHydrator @Inject() (statsReceiver: StatsReceiver)
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("ReplyTweet")
override val features: Set[Feature[_, _]] = Set(
ConversationFeature,
InReplyToTweetHydratedEarlybirdFeature
)
private val DefaultFeatureMap = FeatureMapBuilder()
.add(ConversationFeature, None)
.add(InReplyToTweetHydratedEarlybirdFeature, None)
.build()
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
private val hydratedReplyCounter = scopedStatsReceiver.counter("hydratedReply")
private val hydratedAncestorCounter = scopedStatsReceiver.counter("hydratedAncestor")
override def apply(
query: PipelineQuery,
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
): Stitch[Seq[FeatureMap]] = {
val replyToInReplyToTweetMap =
ReplyRetweetUtil.replyTweetIdToInReplyToTweetMap(candidates)
val candidatesWithRepliesHydrated = candidates.map { candidate =>
replyToInReplyToTweetMap
.get(candidate.candidate.id).map { inReplyToTweet =>
hydratedReplyCounter.incr()
hydratedReplyCandidate(candidate, inReplyToTweet)
}.getOrElse((candidate, None, None))
}
/**
* Update ancestor tweets with descendant replies and hydrate simple features from one of
* the descendants.
*/
val ancestorTweetToDescendantRepliesMap =
ReplyRetweetUtil.ancestorTweetIdToDescendantRepliesMap(candidates)
val candidatesWithRepliesAndAncestorTweetsHydrated = candidatesWithRepliesHydrated.map {
case (
maybeAncestorTweetCandidate,
updatedReplyConversationFeatures,
inReplyToTweetEarlyBirdFeature) =>
ancestorTweetToDescendantRepliesMap
.get(maybeAncestorTweetCandidate.candidate.id)
.map { descendantReplies =>
hydratedAncestorCounter.incr()
val (ancestorTweetCandidate, updatedConversationFeatures): (
CandidateWithFeatures[TweetCandidate],
Option[ConversationFeatures]
) =
hydrateAncestorTweetCandidate(
maybeAncestorTweetCandidate,
descendantReplies,
updatedReplyConversationFeatures)
(ancestorTweetCandidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures)
}
.getOrElse(
(
maybeAncestorTweetCandidate,
inReplyToTweetEarlyBirdFeature,
updatedReplyConversationFeatures))
}
Stitch.value(
candidatesWithRepliesAndAncestorTweetsHydrated.map {
case (candidate, inReplyToTweetEarlyBirdFeature, updatedConversationFeatures) =>
FeatureMapBuilder()
.add(ConversationFeature, updatedConversationFeatures)
.add(InReplyToTweetHydratedEarlybirdFeature, inReplyToTweetEarlyBirdFeature)
.build()
case _ => DefaultFeatureMap
}
)
}
private def hydratedReplyCandidate(
replyCandidate: CandidateWithFeatures[TweetCandidate],
inReplyToTweetCandidate: CandidateWithFeatures[TweetCandidate]
): (
CandidateWithFeatures[TweetCandidate],
Option[ConversationFeatures],
Option[ThriftTweetFeatures]
) = {
val tweetedAfterInReplyToTweetInSecs =
(
originalTweetAgeFromSnowflake(inReplyToTweetCandidate),
originalTweetAgeFromSnowflake(replyCandidate)) match {
case (Some(inReplyToTweetAge), Some(replyTweetAge)) =>
Some((inReplyToTweetAge - replyTweetAge).inSeconds.toLong)
case _ => None
}
val existingConversationFeatures = Some(
replyCandidate.features
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
val updatedConversationFeatures = existingConversationFeatures match {
case Some(v1) =>
Some(
v1.copy(
tweetedAfterInReplyToTweetInSecs = tweetedAfterInReplyToTweetInSecs,
isSelfReply = Some(
replyCandidate.features.getOrElse(
AuthorIdFeature,
None) == inReplyToTweetCandidate.features.getOrElse(AuthorIdFeature, None))
)
)
case _ => None
}
// Note: if inReplyToTweet is a retweet, we need to read early bird feature from the merged
// early bird feature field from RetweetSourceTweetFeatureHydrator class.
// But if inReplyToTweet is a reply, we return its early bird feature directly
val inReplyToTweetThriftTweetFeaturesOpt = {
if (inReplyToTweetCandidate.features.getOrElse(IsRetweetFeature, false)) {
inReplyToTweetCandidate.features.getOrElse(SourceTweetEarlybirdFeature, None)
} else {
inReplyToTweetCandidate.features.getOrElse(EarlybirdFeature, None)
}
}
(replyCandidate, updatedConversationFeatures, inReplyToTweetThriftTweetFeaturesOpt)
}
private def hydrateAncestorTweetCandidate(
ancestorTweetCandidate: CandidateWithFeatures[TweetCandidate],
descendantReplies: Seq[CandidateWithFeatures[TweetCandidate]],
updatedReplyConversationFeatures: Option[ConversationFeatures]
): (CandidateWithFeatures[TweetCandidate], Option[ConversationFeatures]) = {
// Ancestor could be a reply. For example, in thread: tweetA -> tweetB -> tweetC,
// tweetB is a reply and ancestor at the same time. Hence, tweetB's conversation feature
// will be updated by hydratedReplyCandidate and hydrateAncestorTweetCandidate functions.
val existingConversationFeatures =
if (updatedReplyConversationFeatures.nonEmpty)
updatedReplyConversationFeatures
else
Some(
ancestorTweetCandidate.features
.getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))
val updatedConversationFeatures = existingConversationFeatures match {
case Some(v1) =>
Some(
v1.copy(
hasDescendantReplyCandidate = Some(true),
hasInNetworkDescendantReply =
Some(descendantReplies.exists(_.features.getOrElse(InNetworkFeature, false)))
))
case _ => None
}
(ancestorTweetCandidate, updatedConversationFeatures)
}
private def originalTweetAgeFromSnowflake(
candidate: CandidateWithFeatures[TweetCandidate]
): Option[Duration] = {
SnowflakeId
.timeFromIdOpt(
candidate.features
.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id))
.map(Time.now - _)
}
}

17
home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestQueryFeatureHydrator.scala
View File

@ -17,9 +17,13 @@ import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.G
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor
import com.twitter.product_mixer.core.pipeline.HasPipelineCursor
import com.twitter.product_mixer.core.pipeline.PipelineQuery
import com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest
import com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure
import com.twitter.search.common.util.lang.ThriftLanguageUtil
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.stitch.Stitch
import com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.dowFromTimestamp
import com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.hourFromTimestamp
import java.util.UUID
import javax.inject.Inject
import javax.inject.Singleton
@ -45,6 +49,9 @@ class RequestQueryFeatureHydrator[
PullToRefreshFeature,
RequestJoinIdFeature,
ServedRequestIdFeature,
TimestampFeature,
TimestampGMTDowFeature,
TimestampGMTHourFeature,
ViewerIdFeature
)
@ -67,6 +74,7 @@ class RequestQueryFeatureHydrator[
override def hydrate(query: Query): Stitch[FeatureMap] = {
val requestContext = query.deviceContext.flatMap(_.requestContextValue)
val servedRequestId = UUID.randomUUID.getMostSignificantBits
val timestamp = query.queryTime.inMilliseconds
val featureMap = FeatureMapBuilder()
.add(AccountAgeFeature, query.getOptionalUserId.flatMap(SnowflakeId.timeFromIdOpt))
@ -97,8 +105,15 @@ class RequestQueryFeatureHydrator[
.add(PullToRefreshFeature, requestContext.contains(RequestContext.PullToRefresh))
.add(ServedRequestIdFeature, Some(servedRequestId))
.add(RequestJoinIdFeature, getRequestJoinId(servedRequestId))
.add(TimestampFeature, timestamp)
.add(TimestampGMTDowFeature, dowFromTimestamp(timestamp))
.add(TimestampGMTHourFeature, hourFromTimestamp(timestamp))
.add(HasDarkRequestFeature, hasDarkRequest)
.add(ViewerIdFeature, query.getRequiredUserId)
.add(
ViewerIdFeature,
query.getOptionalUserId
.orElse(query.getGuestId).getOrElse(
throw PipelineFailure(BadRequest, "Missing viewer id")))
.build()
Stitch.value(featureMap)

Some files were not shown because too many files have changed in this diff Show More