Delete home-mixer directory
This commit is contained in:
dogemanttv
GitHub
parent
63be8c971c
commit
510b66c848
|
|
@ -1,30 +0,0 @@
|
|||
jvm_binary(
|
||||
name = "bin",
|
||||
basename = "home-mixer",
|
||||
main = "com.twitter.home_mixer.HomeMixerServerMain",
|
||||
runtime_platform = "java11",
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/ch/qos/logback:logback-classic",
|
||||
"finagle/finagle-zipkin-scribe/src/main/scala",
|
||||
"finatra/inject/inject-logback/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer",
|
||||
"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback",
|
||||
"twitter-server-internal/src/main/scala",
|
||||
"twitter-server/logback-classic/src/main/scala",
|
||||
],
|
||||
)
|
||||
|
||||
# Aurora Workflows build phase convention requires a jvm_app named with home-mixer-app
|
||||
jvm_app(
|
||||
name = "home-mixer-app",
|
||||
archive = "zip",
|
||||
binary = ":bin",
|
||||
bundles = [
|
||||
bundle(
|
||||
fileset = ["config/**/*"],
|
||||
owning_target = "home-mixer/config:files",
|
||||
),
|
||||
],
|
||||
tags = ["bazel-compatible"],
|
||||
)
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
Home Mixer
|
||||
==========
|
||||
|
||||
Home Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently
|
||||
powers:
|
||||
- For you - best Tweets from people you follow + recommended out-of-network content
|
||||
- Following - reverse chronological Tweets from people you follow
|
||||
- Lists - reverse chronological Tweets from List members
|
||||
|
||||
Home Mixer is built on Product Mixer, our custom Scala framework that facilitates building
|
||||
feeds of content.
|
||||
|
||||
## Overview
|
||||
|
||||
The For You recommendation algorithm in Home Mixer involves the following stages:
|
||||
|
||||
- Candidate Generation - fetch Tweets from various Candidate Sources. For example:
|
||||
- Earlybird Search Index
|
||||
- User Tweet Entity Graph
|
||||
- Cr Mixer
|
||||
- Follow Recommendations Service
|
||||
- Feature Hydration
|
||||
- Fetch the ~6000 features needed for ranking
|
||||
- Scoring and Ranking using ML model
|
||||
- Filters and Heuristics. For example:
|
||||
- Author Diversity
|
||||
- Content Balance (In network vs Out of Network)
|
||||
- Feedback fatigue
|
||||
- Deduplication / previously seen Tweets removal
|
||||
- Visibility Filtering (blocked, muted authors/tweets, NSFW settings)
|
||||
- Mixing - integrate Tweets with non-Tweet content
|
||||
- Ads
|
||||
- Who-to-follow modules
|
||||
- Prompts
|
||||
- Product Features and Serving
|
||||
- Conversation Modules for replies
|
||||
- Social Context
|
||||
- Timeline Navigation
|
||||
- Edited Tweets
|
||||
- Feedback options
|
||||
- Pagination and cursoring
|
||||
- Observability and logging
|
||||
- Client instructions and content marshalling
|
||||
|
||||
## Pipeline Structure
|
||||
|
||||
### General
|
||||
|
||||
Product Mixer services like Home Mixer are structured around Pipelines that split the execution
|
||||
into transparent and structured steps.
|
||||
|
||||
Requests first go to Product Pipelines, which are used to select which Mixer Pipeline or
|
||||
Recommendation Pipeline to run for a given request. Each Mixer or Recommendation
|
||||
Pipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.
|
||||
|
||||
Mixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together
|
||||
(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)
|
||||
and rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.
|
||||
These pipelines also marshall candidates into a domain object and then into a transport object
|
||||
to return to the caller.
|
||||
|
||||
Candidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic
|
||||
operations on the Candidates, such as filtering out unwanted candidates, applying decorations,
|
||||
and hydrating features.
|
||||
|
||||
The sections below describe the high level pipeline structure (non-exhaustive) for the main Home
|
||||
Timeline tabs powered by Home Mixer.
|
||||
|
||||
### For You
|
||||
|
||||
- ForYouProductPipelineConfig
|
||||
- ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)
|
||||
- ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)
|
||||
- ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)
|
||||
- Fetch Tweet Candidates
|
||||
- ScoredTweetsInNetworkCandidatePipelineConfig
|
||||
- ScoredTweetsTweetMixerCandidatePipelineConfig
|
||||
- ScoredTweetsUtegCandidatePipelineConfig
|
||||
- ScoredTweetsFrsCandidatePipelineConfig
|
||||
- Feature Hydration and Scoring
|
||||
- ScoredTweetsScoringPipelineConfig
|
||||
- ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)
|
||||
- ForYouAdsCandidatePipelineConfig (fetch ads)
|
||||
- ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)
|
||||
|
||||
### Following
|
||||
|
||||
- FollowingProductPipelineConfig
|
||||
- FollowingMixerPipelineConfig
|
||||
- FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)
|
||||
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
|
||||
- FollowingAdsCandidatePipelineConfig (fetch ads)
|
||||
- FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)
|
||||
|
||||
### Lists
|
||||
|
||||
- ListTweetsProductPipelineConfig
|
||||
- ListTweetsMixerPipelineConfig
|
||||
- ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)
|
||||
- ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)
|
||||
- ListTweetsAdsCandidatePipelineConfig (fetch ads)
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/com/google/inject:guice",
|
||||
"3rdparty/jvm/javax/inject:javax.inject",
|
||||
"3rdparty/jvm/net/codingwell:scala-guice",
|
||||
"3rdparty/jvm/org/slf4j:slf4j-api",
|
||||
"finagle/finagle-core/src/main",
|
||||
"finagle/finagle-http/src/main/scala",
|
||||
"finagle/finagle-thriftmux/src/main/scala",
|
||||
"finatra-internal/mtls-http/src/main/scala",
|
||||
"finatra-internal/mtls-thriftmux/src/main/scala",
|
||||
"finatra/http-core/src/main/java/com/twitter/finatra/http",
|
||||
"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations",
|
||||
"finatra/inject/inject-app/src/main/scala",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"finatra/inject/inject-server/src/main/scala",
|
||||
"finatra/inject/inject-utils/src/main/scala",
|
||||
"home-mixer/server/src/main/resources",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/controller",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/federated",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/module",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter",
|
||||
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/render:thrift-scala",
|
||||
"strato/config/columns/auth-context:auth-context-strato-client",
|
||||
"strato/config/columns/gizmoduck:gizmoduck-strato-client",
|
||||
"strato/src/main/scala/com/twitter/strato/fed",
|
||||
"strato/src/main/scala/com/twitter/strato/fed/server",
|
||||
"stringcenter/client",
|
||||
"stringcenter/client/src/main/java",
|
||||
"stringcenter/client/src/main/scala/com/twitter/stringcenter/client",
|
||||
"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/view",
|
||||
"timelines/src/main/scala/com/twitter/timelines/config",
|
||||
"timelines/src/main/scala/com/twitter/timelines/features/app",
|
||||
"twitter-server-internal",
|
||||
"twitter-server/server/src/main/scala",
|
||||
"util/util-app/src/main/scala",
|
||||
"util/util-core:scala",
|
||||
"util/util-slf4j-api/src/main/scala",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
package com.twitter.home_mixer
|
||||
|
||||
import com.twitter.finatra.http.routing.HttpWarmup
|
||||
import com.twitter.finatra.httpclient.RequestBuilder._
|
||||
import com.twitter.util.logging.Logging
|
||||
import com.twitter.inject.utils.Handler
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeMixerHttpServerWarmupHandler @Inject() (warmup: HttpWarmup) extends Handler with Logging {
|
||||
|
||||
override def handle(): Unit = {
|
||||
Try(warmup.send(get("/admin/product-mixer/product-pipelines"), admin = true)())
|
||||
.onFailure(e => error(e.getMessage, e))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
package com.twitter.home_mixer
|
||||
|
||||
import com.google.inject.Module
|
||||
import com.twitter.finagle.Filter
|
||||
import com.twitter.finatra.annotations.DarkTrafficFilterType
|
||||
import com.twitter.finatra.http.HttpServer
|
||||
import com.twitter.finatra.http.routing.HttpRouter
|
||||
import com.twitter.finatra.mtls.http.{Mtls => HttpMtls}
|
||||
import com.twitter.finatra.mtls.thriftmux.Mtls
|
||||
import com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule
|
||||
import com.twitter.finatra.thrift.ThriftServer
|
||||
import com.twitter.finatra.thrift.filters._
|
||||
import com.twitter.finatra.thrift.routing.ThriftRouter
|
||||
import com.twitter.home_mixer.controller.HomeThriftController
|
||||
import com.twitter.home_mixer.federated.HomeMixerColumn
|
||||
import com.twitter.home_mixer.module._
|
||||
import com.twitter.home_mixer.param.GlobalParamConfigModule
|
||||
import com.twitter.home_mixer.product.HomeMixerProductModule
|
||||
import com.twitter.home_mixer.{thriftscala => st}
|
||||
import com.twitter.product_mixer.component_library.module.AccountRecommendationsMixerModule
|
||||
import com.twitter.product_mixer.component_library.module.DarkTrafficFilterModule
|
||||
import com.twitter.product_mixer.component_library.module.EarlybirdModule
|
||||
import com.twitter.product_mixer.component_library.module.ExploreRankerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.GizmoduckClientModule
|
||||
import com.twitter.product_mixer.component_library.module.OnboardingTaskServiceModule
|
||||
import com.twitter.product_mixer.component_library.module.SocialGraphServiceModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineRankerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineScorerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TimelineServiceClientModule
|
||||
import com.twitter.product_mixer.component_library.module.TweetImpressionStoreModule
|
||||
import com.twitter.product_mixer.component_library.module.TweetMixerClientModule
|
||||
import com.twitter.product_mixer.component_library.module.UserSessionStoreModule
|
||||
import com.twitter.product_mixer.core.controllers.ProductMixerController
|
||||
import com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper
|
||||
import com.twitter.product_mixer.core.module.ProductMixerModule
|
||||
import com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule
|
||||
import com.twitter.strato.fed.StratoFed
|
||||
import com.twitter.strato.fed.server.StratoFedServer
|
||||
|
||||
object HomeMixerServerMain extends HomeMixerServer
|
||||
|
||||
class HomeMixerServer
|
||||
extends StratoFedServer
|
||||
with ThriftServer
|
||||
with Mtls
|
||||
with HttpServer
|
||||
with HttpMtls {
|
||||
override val name = "home-mixer-server"
|
||||
|
||||
override val modules: Seq[Module] = Seq(
|
||||
AccountRecommendationsMixerModule,
|
||||
AdvertiserBrandSafetySettingsStoreModule,
|
||||
BlenderClientModule,
|
||||
ClientSentImpressionsPublisherModule,
|
||||
ConversationServiceModule,
|
||||
EarlybirdModule,
|
||||
ExploreRankerClientModule,
|
||||
FeedbackHistoryClientModule,
|
||||
GizmoduckClientModule,
|
||||
GlobalParamConfigModule,
|
||||
HomeAdsCandidateSourceModule,
|
||||
HomeMixerFlagsModule,
|
||||
HomeMixerProductModule,
|
||||
HomeMixerResourcesModule,
|
||||
ImpressionBloomFilterModule,
|
||||
InjectionHistoryClientModule,
|
||||
ManhattanClientsModule,
|
||||
ManhattanFeatureRepositoryModule,
|
||||
ManhattanTweetImpressionStoreModule,
|
||||
MemcachedFeatureRepositoryModule,
|
||||
NaviModelClientModule,
|
||||
OnboardingTaskServiceModule,
|
||||
OptimizedStratoClientModule,
|
||||
PeopleDiscoveryServiceModule,
|
||||
ProductMixerModule,
|
||||
RealGraphInNetworkScoresModule,
|
||||
RealtimeAggregateFeatureRepositoryModule,
|
||||
ScoredTweetsMemcacheModule,
|
||||
ScribeEventPublisherModule,
|
||||
SimClustersRecentEngagementsClientModule,
|
||||
SocialGraphServiceModule,
|
||||
StaleTweetsCacheModule,
|
||||
ThriftFeatureRepositoryModule,
|
||||
TimelineRankerClientModule,
|
||||
TimelineScorerClientModule,
|
||||
TimelineServiceClientModule,
|
||||
TimelinesPersistenceStoreClientModule,
|
||||
TopicSocialProofClientModule,
|
||||
TweetImpressionStoreModule,
|
||||
TweetMixerClientModule,
|
||||
TweetypieClientModule,
|
||||
TweetypieStaticEntitiesCacheClientModule,
|
||||
UserSessionStoreModule,
|
||||
new DarkTrafficFilterModule[st.HomeMixer.ReqRepServicePerEndpoint](),
|
||||
new MtlsThriftWebFormsModule[st.HomeMixer.MethodPerEndpoint](this),
|
||||
new ProductScopeStringCenterModule()
|
||||
)
|
||||
|
||||
override def configureThrift(router: ThriftRouter): Unit = {
|
||||
router
|
||||
.filter[LoggingMDCFilter]
|
||||
.filter[TraceIdMDCFilter]
|
||||
.filter[ThriftMDCFilter]
|
||||
.filter[StatsFilter]
|
||||
.filter[AccessLoggingFilter]
|
||||
.filter[ExceptionMappingFilter]
|
||||
.filter[Filter.TypeAgnostic, DarkTrafficFilterType]
|
||||
.exceptionMapper[LoggingThrowableExceptionMapper]
|
||||
.exceptionMapper[PipelineFailureExceptionMapper]
|
||||
.add[HomeThriftController]
|
||||
}
|
||||
|
||||
override def configureHttp(router: HttpRouter): Unit =
|
||||
router.add(
|
||||
ProductMixerController[st.HomeMixer.MethodPerEndpoint](
|
||||
this.injector,
|
||||
st.HomeMixer.ExecutePipeline))
|
||||
|
||||
override val dest: String = "/s/home-mixer/home-mixer:strato"
|
||||
|
||||
override val columns: Seq[Class[_ <: StratoFed.Column]] =
|
||||
Seq(classOf[HomeMixerColumn])
|
||||
|
||||
override protected def warmup(): Unit = {
|
||||
handle[HomeMixerThriftServerWarmupHandler]()
|
||||
handle[HomeMixerHttpServerWarmupHandler]()
|
||||
}
|
||||
}
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
package com.twitter.home_mixer
|
||||
|
||||
import com.twitter.finagle.thrift.ClientId
|
||||
import com.twitter.finatra.thrift.routing.ThriftWarmup
|
||||
import com.twitter.home_mixer.{thriftscala => st}
|
||||
import com.twitter.util.logging.Logging
|
||||
import com.twitter.inject.utils.Handler
|
||||
import com.twitter.product_mixer.core.{thriftscala => pt}
|
||||
import com.twitter.scrooge.Request
|
||||
import com.twitter.scrooge.Response
|
||||
import com.twitter.util.Return
|
||||
import com.twitter.util.Throw
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeMixerThriftServerWarmupHandler @Inject() (warmup: ThriftWarmup)
|
||||
extends Handler
|
||||
with Logging {
|
||||
|
||||
private val clientId = ClientId("thrift-warmup-client")
|
||||
|
||||
def handle(): Unit = {
|
||||
val testIds = Seq(1, 2, 3)
|
||||
try {
|
||||
clientId.asCurrent {
|
||||
testIds.foreach { id =>
|
||||
val warmupReq = warmupQuery(id)
|
||||
info(s"Sending warm-up request to service with query: $warmupReq")
|
||||
warmup.sendRequest(
|
||||
method = st.HomeMixer.GetUrtResponse,
|
||||
req = Request(st.HomeMixer.GetUrtResponse.Args(warmupReq)))(assertWarmupResponse)
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
case e: Throwable => error(e.getMessage, e)
|
||||
}
|
||||
info("Warm-up done.")
|
||||
}
|
||||
|
||||
private def warmupQuery(userId: Long): st.HomeMixerRequest = {
|
||||
val clientContext = pt.ClientContext(
|
||||
userId = Some(userId),
|
||||
guestId = None,
|
||||
appId = Some(12345L),
|
||||
ipAddress = Some("0.0.0.0"),
|
||||
userAgent = Some("FAKE_USER_AGENT_FOR_WARMUPS"),
|
||||
countryCode = Some("US"),
|
||||
languageCode = Some("en"),
|
||||
isTwoffice = None,
|
||||
userRoles = None,
|
||||
deviceId = Some("FAKE_DEVICE_ID_FOR_WARMUPS")
|
||||
)
|
||||
st.HomeMixerRequest(
|
||||
clientContext = clientContext,
|
||||
product = st.Product.Following,
|
||||
productContext = Some(st.ProductContext.Following(st.Following())),
|
||||
maxResults = Some(3)
|
||||
)
|
||||
}
|
||||
|
||||
private def assertWarmupResponse(
|
||||
result: Try[Response[st.HomeMixer.GetUrtResponse.SuccessType]]
|
||||
): Unit = {
|
||||
result match {
|
||||
case Return(_) => // ok
|
||||
case Throw(exception) =>
|
||||
warn("Error performing warm-up request.")
|
||||
error(exception.getMessage, exception)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer",
|
||||
],
|
||||
exports = [
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,116 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter
|
||||
import com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter
|
||||
import com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
|
||||
import com.twitter.product_mixer.component_library.filter.FeatureFilter
|
||||
import com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that fetches tweets from the Conversation Service Candidate Source
|
||||
*/
|
||||
class ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](
|
||||
conversationServiceCandidateSource: ConversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
|
||||
namesFeatureHydrator: NamesFeatureHydrator,
|
||||
invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,
|
||||
override val gates: Seq[BaseGate[Query]],
|
||||
override val decorator: Option[CandidateDecorator[Query, TweetCandidate]])
|
||||
extends DependentCandidatePipelineConfig[
|
||||
Query,
|
||||
ConversationServiceCandidateSourceRequest,
|
||||
TweetWithConversationMetadata,
|
||||
TweetCandidate
|
||||
] {
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier =
|
||||
CandidatePipelineIdentifier("ConversationService")
|
||||
|
||||
private val TweetypieHydratedFilterId = "TweetypieHydrated"
|
||||
private val QuotedTweetDroppedFilterId = "QuotedTweetDropped"
|
||||
|
||||
override val candidateSource: BaseCandidateSource[
|
||||
ConversationServiceCandidateSourceRequest,
|
||||
TweetWithConversationMetadata
|
||||
] = conversationServiceCandidateSource
|
||||
|
||||
override val queryTransformer: DependentCandidatePipelineQueryTransformer[
|
||||
Query,
|
||||
ConversationServiceCandidateSourceRequest
|
||||
] = { (_, candidates) =>
|
||||
val tweetsWithConversationMetadata = candidates.map { candidate =>
|
||||
TweetWithConversationMetadata(
|
||||
tweetId = candidate.candidateIdLong,
|
||||
userId = candidate.features.getOrElse(AuthorIdFeature, None),
|
||||
sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),
|
||||
sourceUserId = candidate.features.getOrElse(SourceUserIdFeature, None),
|
||||
inReplyToTweetId = candidate.features.getOrElse(InReplyToTweetIdFeature, None),
|
||||
conversationId = None,
|
||||
ancestors = Seq.empty
|
||||
)
|
||||
}
|
||||
ConversationServiceCandidateSourceRequest(tweetsWithConversationMetadata)
|
||||
}
|
||||
|
||||
override val featuresFromCandidateSourceTransformers: Seq[
|
||||
CandidateFeatureTransformer[TweetWithConversationMetadata]
|
||||
] = Seq(ConversationServiceResponseFeatureTransformer)
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
TweetWithConversationMetadata,
|
||||
TweetCandidate
|
||||
] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }
|
||||
|
||||
override val preFilterFeatureHydrationPhase1: Seq[
|
||||
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
|
||||
] = Seq(
|
||||
tweetypieFeatureHydrator,
|
||||
InNetworkFeatureHydrator,
|
||||
)
|
||||
|
||||
override def filters: Seq[Filter[Query, TweetCandidate]] = Seq(
|
||||
RetweetDeduplicationFilter,
|
||||
FeatureFilter.fromFeature(FilterIdentifier(TweetypieHydratedFilterId), IsHydratedFeature),
|
||||
PredicateFeatureFilter.fromPredicate(
|
||||
FilterIdentifier(QuotedTweetDroppedFilterId),
|
||||
shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }
|
||||
),
|
||||
invalidSubscriptionTweetFilter,
|
||||
InvalidConversationModuleFilter
|
||||
)
|
||||
|
||||
override val postFilterFeatureHydration: Seq[
|
||||
BaseCandidateFeatureHydrator[Query, TweetCandidate, _]
|
||||
] = Seq(namesFeatureHydrator)
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
|
||||
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
|
||||
)
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.gate.BaseGate
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery] @Inject() (
|
||||
conversationServiceCandidateSource: ConversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator: TweetypieFeatureHydrator,
|
||||
invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,
|
||||
namesFeatureHydrator: NamesFeatureHydrator) {
|
||||
|
||||
def build(
|
||||
gates: Seq[BaseGate[Query]] = Seq.empty,
|
||||
decorator: Option[CandidateDecorator[Query, TweetCandidate]] = None
|
||||
): ConversationServiceCandidatePipelineConfig[Query] = {
|
||||
new ConversationServiceCandidatePipelineConfig(
|
||||
conversationServiceCandidateSource,
|
||||
tweetypieFeatureHydrator,
|
||||
namesFeatureHydrator,
|
||||
invalidSubscriptionTweetFilter,
|
||||
gates,
|
||||
decorator
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
|
||||
object ConversationServiceResponseFeatureTransformer
|
||||
extends CandidateFeatureTransformer[TweetWithConversationMetadata] {
|
||||
|
||||
override val identifier: TransformerIdentifier =
|
||||
TransformerIdentifier("ConversationServiceResponse")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsRetweetFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
ConversationModuleFocalTweetIdFeature,
|
||||
AncestorsFeature,
|
||||
SuggestTypeFeature
|
||||
)
|
||||
|
||||
override def transform(candidate: TweetWithConversationMetadata): FeatureMap = FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, candidate.userId)
|
||||
.add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)
|
||||
.add(IsRetweetFeature, candidate.sourceTweetId.isDefined)
|
||||
.add(SourceTweetIdFeature, candidate.sourceTweetId)
|
||||
.add(SourceUserIdFeature, candidate.sourceUserId)
|
||||
.add(ConversationModuleFocalTweetIdFeature, candidate.conversationId)
|
||||
.add(AncestorsFeature, candidate.ancestors)
|
||||
.add(SuggestTypeFeature, Some(SuggestType.RankedOrganicTweet))
|
||||
.build()
|
||||
}
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.functional_component.candidate_source.StaleTweetsCacheCandidateSource
|
||||
import com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder
|
||||
import com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator
|
||||
import com.twitter.home_mixer.functional_component.query_transformer.EditedTweetsCandidatePipelineQueryTransformer
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.EmptyClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineFocalTweetSafetyLevel
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that fetches edited tweets from the Stale Tweets Cache
|
||||
*/
|
||||
@Singleton
|
||||
case class EditedTweetsCandidatePipelineConfig @Inject() (
|
||||
staleTweetsCacheCandidateSource: StaleTweetsCacheCandidateSource,
|
||||
namesFeatureHydrator: NamesFeatureHydrator,
|
||||
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)
|
||||
extends DependentCandidatePipelineConfig[
|
||||
PipelineQuery,
|
||||
Seq[Long],
|
||||
Long,
|
||||
TweetCandidate
|
||||
] {
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier("EditedTweets")
|
||||
|
||||
override val candidateSource: BaseCandidateSource[Seq[Long], Long] =
|
||||
staleTweetsCacheCandidateSource
|
||||
|
||||
override val queryTransformer: CandidatePipelineQueryTransformer[
|
||||
PipelineQuery,
|
||||
Seq[Long]
|
||||
] = EditedTweetsCandidatePipelineQueryTransformer
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
Long,
|
||||
TweetCandidate
|
||||
] = { candidate => TweetCandidate(id = candidate) }
|
||||
|
||||
override val postFilterFeatureHydration: Seq[
|
||||
BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]
|
||||
] = Seq(namesFeatureHydrator)
|
||||
|
||||
override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {
|
||||
val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](
|
||||
clientEventInfoBuilder = EmptyClientEventInfoBuilder,
|
||||
entryIdToReplaceBuilder = Some((_, candidate, _) =>
|
||||
Some(s"${TweetItem.TweetEntryNamespace}-${candidate.id.toString}")),
|
||||
contextualTweetRefBuilder = Some(
|
||||
ContextualTweetRefBuilder(
|
||||
TweetHydrationContext(
|
||||
// Apply safety level that includes canonical VF treatments that apply regardless of context.
|
||||
safetyLevelOverride = Some(TimelineFocalTweetSafetyLevel),
|
||||
outerTweetContext = None
|
||||
)
|
||||
)
|
||||
),
|
||||
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
|
||||
)
|
||||
|
||||
Some(UrtItemCandidateDecorator(tweetItemBuilder))
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.5, 50, 60, 60)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,123 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.functional_component.gate.RequestContextNotGate
|
||||
import com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature
|
||||
import com.twitter.home_mixer.model.request.DeviceContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.DurationParamBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.ShowAlertCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertColorConfigurationBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertDisplayLocationBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertIconDisplayInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.gate.FeatureGate
|
||||
import com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource
|
||||
import com.twitter.product_mixer.core.functional_component.configapi.StaticParam
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.NewTweets
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Top
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.alert.UpArrow
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.TwitterBlueRosettaColor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.color.WhiteRosettaColor
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig
|
||||
import com.twitter.util.Duration
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Candidate Pipeline Config that creates the New Tweets Pill
|
||||
*/
|
||||
@Singleton
|
||||
class NewTweetsPillCandidatePipelineConfig[Query <: PipelineQuery with HasDeviceContext] @Inject() (
|
||||
) extends DependentCandidatePipelineConfig[
|
||||
Query,
|
||||
Unit,
|
||||
ShowAlertCandidate,
|
||||
ShowAlertCandidate
|
||||
] {
|
||||
import NewTweetsPillCandidatePipelineConfig._
|
||||
|
||||
override val identifier: CandidatePipelineIdentifier =
|
||||
CandidatePipelineIdentifier("NewTweetsPill")
|
||||
|
||||
override val gates: Seq[Gate[Query]] = Seq(
|
||||
RequestContextNotGate(Seq(DeviceContext.RequestContext.PullToRefresh)),
|
||||
FeatureGate.fromFeature(GetNewerFeature)
|
||||
)
|
||||
|
||||
override val candidateSource: CandidateSource[Unit, ShowAlertCandidate] =
|
||||
StaticCandidateSource(
|
||||
CandidateSourceIdentifier(identifier.name),
|
||||
Seq(ShowAlertCandidate(id = identifier.name, userIds = Seq.empty))
|
||||
)
|
||||
|
||||
override val queryTransformer: CandidatePipelineQueryTransformer[Query, Unit] = { _ => Unit }
|
||||
|
||||
override val resultTransformer: CandidatePipelineResultsTransformer[
|
||||
ShowAlertCandidate,
|
||||
ShowAlertCandidate
|
||||
] = { candidate => candidate }
|
||||
|
||||
override val decorator: Option[CandidateDecorator[Query, ShowAlertCandidate]] = {
|
||||
val triggerDelayBuilder = new BaseDurationBuilder[Query] {
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: ShowAlertCandidate,
|
||||
features: FeatureMap
|
||||
): Option[Duration] = {
|
||||
val delay = query.deviceContext.flatMap(_.requestContextValue) match {
|
||||
case Some(DeviceContext.RequestContext.TweetSelfThread) => 0.millis
|
||||
case Some(DeviceContext.RequestContext.ManualRefresh) => 0.millis
|
||||
case _ => TriggerDelay
|
||||
}
|
||||
|
||||
Some(delay)
|
||||
}
|
||||
}
|
||||
|
||||
val homeShowAlertCandidateBuilder = ShowAlertCandidateUrtItemBuilder(
|
||||
alertType = NewTweets,
|
||||
colorConfigBuilder = StaticShowAlertColorConfigurationBuilder(DefaultColorConfig),
|
||||
displayLocationBuilder = StaticShowAlertDisplayLocationBuilder(Top),
|
||||
triggerDelayBuilder = Some(triggerDelayBuilder),
|
||||
displayDurationBuilder = Some(DurationParamBuilder(StaticParam(DisplayDuration))),
|
||||
iconDisplayInfoBuilder = Some(StaticShowAlertIconDisplayInfoBuilder(DefaultIconDisplayInfo))
|
||||
)
|
||||
|
||||
Some(UrtItemCandidateDecorator(homeShowAlertCandidateBuilder))
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),
|
||||
HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()
|
||||
)
|
||||
}
|
||||
|
||||
object NewTweetsPillCandidatePipelineConfig {
|
||||
val DefaultColorConfig: ShowAlertColorConfiguration = ShowAlertColorConfiguration(
|
||||
background = TwitterBlueRosettaColor,
|
||||
text = WhiteRosettaColor,
|
||||
border = Some(WhiteRosettaColor)
|
||||
)
|
||||
|
||||
val DefaultIconDisplayInfo: ShowAlertIconDisplayInfo =
|
||||
ShowAlertIconDisplayInfo(icon = UpArrow, tint = WhiteRosettaColor)
|
||||
|
||||
// Unlimited display time (until user takes action)
|
||||
val DisplayDuration = -1.millisecond
|
||||
val TriggerDelay = 4.minutes
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
package com.twitter.home_mixer.candidate_pipeline
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
|
||||
import com.twitter.timelineservice.{thriftscala => t}
|
||||
|
||||
object TimelineServiceResponseFeatureTransformer extends CandidateFeatureTransformer[t.Tweet] {
|
||||
|
||||
override val identifier: TransformerIdentifier = TransformerIdentifier("TimelineServiceResponse")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsRetweetFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
)
|
||||
|
||||
override def transform(candidate: t.Tweet): FeatureMap = FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, candidate.userId)
|
||||
.add(InReplyToTweetIdFeature, candidate.inReplyToStatusId)
|
||||
.add(IsRetweetFeature, candidate.sourceStatusId.isDefined)
|
||||
.add(SourceTweetIdFeature, candidate.sourceStatusId)
|
||||
.add(SourceUserIdFeature, candidate.sourceUserId)
|
||||
.build()
|
||||
}
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt",
|
||||
"snowflake/src/main/scala/com/twitter/snowflake/id",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
package com.twitter.home_mixer.controller
|
||||
|
||||
import com.twitter.finatra.thrift.Controller
|
||||
import com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller
|
||||
import com.twitter.home_mixer.model.request.HomeMixerRequest
|
||||
import com.twitter.home_mixer.service.ScoredTweetsService
|
||||
import com.twitter.home_mixer.{thriftscala => t}
|
||||
import com.twitter.product_mixer.core.controllers.DebugTwitterContext
|
||||
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
|
||||
import com.twitter.product_mixer.core.service.debug_query.DebugQueryService
|
||||
import com.twitter.product_mixer.core.service.urt.UrtService
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.configapi.Params
|
||||
import javax.inject.Inject
|
||||
|
||||
class HomeThriftController @Inject() (
|
||||
homeRequestUnmarshaller: HomeMixerRequestUnmarshaller,
|
||||
urtService: UrtService,
|
||||
scoredTweetsService: ScoredTweetsService,
|
||||
paramsBuilder: ParamsBuilder)
|
||||
extends Controller(t.HomeMixer)
|
||||
with DebugTwitterContext {
|
||||
|
||||
handle(t.HomeMixer.GetUrtResponse) { args: t.HomeMixer.GetUrtResponse.Args =>
|
||||
val request = homeRequestUnmarshaller(args.request)
|
||||
val params = buildParams(request)
|
||||
Stitch.run(urtService.getUrtResponse[HomeMixerRequest](request, params))
|
||||
}
|
||||
|
||||
handle(t.HomeMixer.GetScoredTweetsResponse) { args: t.HomeMixer.GetScoredTweetsResponse.Args =>
|
||||
val request = homeRequestUnmarshaller(args.request)
|
||||
val params = buildParams(request)
|
||||
withDebugTwitterContext(request.clientContext) {
|
||||
Stitch.run(scoredTweetsService.getScoredTweetsResponse[HomeMixerRequest](request, params))
|
||||
}
|
||||
}
|
||||
|
||||
private def buildParams(request: HomeMixerRequest): Params = {
|
||||
val userAgeOpt = request.clientContext.userId.map { userId =>
|
||||
SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)
|
||||
}
|
||||
val fsCustomMapInput = userAgeOpt.map("account_age_in_days" -> _).toMap
|
||||
paramsBuilder.build(
|
||||
clientContext = request.clientContext,
|
||||
product = request.product,
|
||||
featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),
|
||||
fsCustomMapInput = fsCustomMapInput
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry",
|
||||
"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala",
|
||||
"src/thrift/com/twitter/gizmoduck:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/render:thrift-scala",
|
||||
"stitch/stitch-repo/src/main/scala",
|
||||
"strato/config/columns/auth-context:auth-context-strato-client",
|
||||
"strato/config/columns/gizmoduck:gizmoduck-strato-client",
|
||||
"strato/config/src/thrift/com/twitter/strato/graphql/timelines:graphql-timelines-scala",
|
||||
"strato/src/main/scala/com/twitter/strato/callcontext",
|
||||
"strato/src/main/scala/com/twitter/strato/fed",
|
||||
"strato/src/main/scala/com/twitter/strato/fed/server",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,217 +0,0 @@
|
|||
package com.twitter.home_mixer.federated
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gd}
|
||||
import com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller
|
||||
import com.twitter.home_mixer.model.request.HomeMixerRequest
|
||||
import com.twitter.home_mixer.{thriftscala => hm}
|
||||
import com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder
|
||||
import com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest
|
||||
import com.twitter.product_mixer.core.pipeline.product.ProductPipelineResult
|
||||
import com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry
|
||||
import com.twitter.product_mixer.core.{thriftscala => pm}
|
||||
import com.twitter.stitch.Arrow
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.strato.callcontext.CallContext
|
||||
import com.twitter.strato.catalog.OpMetadata
|
||||
import com.twitter.strato.config._
|
||||
import com.twitter.strato.data._
|
||||
import com.twitter.strato.fed.StratoFed
|
||||
import com.twitter.strato.generated.client.auth_context.AuditIpClientColumn
|
||||
import com.twitter.strato.generated.client.gizmoduck.CompositeOnUserClientColumn
|
||||
import com.twitter.strato.graphql.timelines.{thriftscala => gql}
|
||||
import com.twitter.strato.thrift.ScroogeConv
|
||||
import com.twitter.timelines.render.{thriftscala => tr}
|
||||
import com.twitter.util.Try
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeMixerColumn @Inject() (
|
||||
homeMixerRequestUnmarshaller: HomeMixerRequestUnmarshaller,
|
||||
compositeOnUserClientColumn: CompositeOnUserClientColumn,
|
||||
auditIpClientColumn: AuditIpClientColumn,
|
||||
paramsBuilder: ParamsBuilder,
|
||||
productPipelineRegistry: ProductPipelineRegistry)
|
||||
extends StratoFed.Column(HomeMixerColumn.Path)
|
||||
with StratoFed.Fetch.Arrow {
|
||||
|
||||
override val contactInfo: ContactInfo = ContactInfo(
|
||||
contactEmail = "",
|
||||
ldapGroup = "",
|
||||
slackRoomId = ""
|
||||
)
|
||||
|
||||
override val metadata: OpMetadata =
|
||||
OpMetadata(
|
||||
lifecycle = Some(Lifecycle.Production),
|
||||
description =
|
||||
Some(Description.PlainText("Federated Strato column for Timelines served via Home Mixer"))
|
||||
)
|
||||
|
||||
private val bouncerAccess: Seq[Policy] = Seq(BouncerAccess())
|
||||
private val finatraTestServiceIdentifiers: Seq[Policy] = Seq(
|
||||
ServiceIdentifierPattern(
|
||||
role = "",
|
||||
service = "",
|
||||
env = "",
|
||||
zone = Seq(""))
|
||||
)
|
||||
|
||||
override val policy: Policy = AnyOf(bouncerAccess ++ finatraTestServiceIdentifiers)
|
||||
|
||||
override type Key = gql.TimelineKey
|
||||
override type View = gql.HomeTimelineView
|
||||
override type Value = tr.Timeline
|
||||
|
||||
override val keyConv: Conv[Key] = ScroogeConv.fromStruct[gql.TimelineKey]
|
||||
override val viewConv: Conv[View] = ScroogeConv.fromStruct[gql.HomeTimelineView]
|
||||
override val valueConv: Conv[Value] = ScroogeConv.fromStruct[tr.Timeline]
|
||||
|
||||
private def createHomeMixerRequestArrow(
|
||||
compositeOnUserClientColumn: CompositeOnUserClientColumn,
|
||||
auditIpClientColumn: AuditIpClientColumn
|
||||
): Arrow[(Key, View), hm.HomeMixerRequest] = {
|
||||
|
||||
val populateUserRolesAndIp: Arrow[(Key, View), (Option[Set[String]], Option[String])] = {
|
||||
val gizmoduckView: (gd.LookupContext, Set[gd.QueryFields]) =
|
||||
(gd.LookupContext(), Set(gd.QueryFields.Roles))
|
||||
|
||||
val populateUserRoles = Arrow
|
||||
.flatMap[(Key, View), Option[Set[String]]] { _ =>
|
||||
Stitch.collect {
|
||||
CallContext.twitterUserId.map { userId =>
|
||||
compositeOnUserClientColumn.fetcher
|
||||
.callStack(HomeMixerColumn.FetchCallstack)
|
||||
.fetch(userId, gizmoduckView).map(_.v)
|
||||
.map {
|
||||
_.flatMap(_.roles.map(_.roles.toSet)).getOrElse(Set.empty)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val populateIpAddress = Arrow
|
||||
.flatMap[(Key, View), Option[String]](_ =>
|
||||
auditIpClientColumn.fetcher
|
||||
.callStack(HomeMixerColumn.FetchCallstack)
|
||||
.fetch((), ()).map(_.v))
|
||||
|
||||
Arrow.join(
|
||||
populateUserRoles,
|
||||
populateIpAddress
|
||||
)
|
||||
}
|
||||
|
||||
Arrow.zipWithArg(populateUserRolesAndIp).map {
|
||||
case ((key, view), (roles, ipAddress)) =>
|
||||
val deviceContextOpt = Some(
|
||||
hm.DeviceContext(
|
||||
isPolling = CallContext.isPolling,
|
||||
requestContext = view.requestContext,
|
||||
latestControlAvailable = view.latestControlAvailable,
|
||||
autoplayEnabled = view.autoplayEnabled
|
||||
))
|
||||
val seenTweetIds = view.seenTweetIds.filter(_.nonEmpty)
|
||||
|
||||
val (product, productContext) = key match {
|
||||
case gql.TimelineKey.HomeTimeline(_) | gql.TimelineKey.HomeTimelineV2(_) =>
|
||||
(
|
||||
hm.Product.ForYou,
|
||||
hm.ProductContext.ForYou(
|
||||
hm.ForYou(
|
||||
deviceContextOpt,
|
||||
seenTweetIds,
|
||||
view.dspClientContext,
|
||||
view.pushToHomeTweetId
|
||||
)
|
||||
))
|
||||
case gql.TimelineKey.HomeLatestTimeline(_) | gql.TimelineKey.HomeLatestTimelineV2(_) =>
|
||||
(
|
||||
hm.Product.Following,
|
||||
hm.ProductContext.Following(
|
||||
hm.Following(deviceContextOpt, seenTweetIds, view.dspClientContext)))
|
||||
case gql.TimelineKey.CreatorSubscriptionsTimeline(_) =>
|
||||
(
|
||||
hm.Product.Subscribed,
|
||||
hm.ProductContext.Subscribed(hm.Subscribed(deviceContextOpt, seenTweetIds)))
|
||||
case _ => throw new UnsupportedOperationException(s"Unknown product: $key")
|
||||
}
|
||||
|
||||
val clientContext = pm.ClientContext(
|
||||
userId = CallContext.twitterUserId,
|
||||
guestId = CallContext.guestId,
|
||||
guestIdAds = CallContext.guestIdAds,
|
||||
guestIdMarketing = CallContext.guestIdMarketing,
|
||||
appId = CallContext.clientApplicationId,
|
||||
ipAddress = ipAddress,
|
||||
userAgent = CallContext.userAgent,
|
||||
countryCode = CallContext.requestCountryCode,
|
||||
languageCode = CallContext.requestLanguageCode,
|
||||
isTwoffice = CallContext.isInternalOrTwoffice,
|
||||
userRoles = roles,
|
||||
deviceId = CallContext.deviceId,
|
||||
mobileDeviceId = CallContext.mobileDeviceId,
|
||||
mobileDeviceAdId = CallContext.adId,
|
||||
limitAdTracking = CallContext.limitAdTracking
|
||||
)
|
||||
|
||||
hm.HomeMixerRequest(
|
||||
clientContext = clientContext,
|
||||
product = product,
|
||||
productContext = Some(productContext),
|
||||
maxResults = Try(view.count.get.toInt).toOption.orElse(HomeMixerColumn.MaxCount),
|
||||
cursor = view.cursor.filter(_.nonEmpty)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
override val fetch: Arrow[(Key, View), Result[Value]] = {
|
||||
val transformThriftIntoPipelineRequest: Arrow[
|
||||
(Key, View),
|
||||
ProductPipelineRequest[HomeMixerRequest]
|
||||
] = {
|
||||
Arrow
|
||||
.identity[(Key, View)]
|
||||
.andThen {
|
||||
createHomeMixerRequestArrow(compositeOnUserClientColumn, auditIpClientColumn)
|
||||
}
|
||||
.map {
|
||||
case thriftRequest =>
|
||||
val request = homeMixerRequestUnmarshaller(thriftRequest)
|
||||
val params = paramsBuilder.build(
|
||||
clientContext = request.clientContext,
|
||||
product = request.product,
|
||||
featureOverrides =
|
||||
request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),
|
||||
)
|
||||
ProductPipelineRequest(request, params)
|
||||
}
|
||||
}
|
||||
|
||||
val underlyingProduct: Arrow[
|
||||
ProductPipelineRequest[HomeMixerRequest],
|
||||
ProductPipelineResult[tr.TimelineResponse]
|
||||
] = Arrow
|
||||
.identity[ProductPipelineRequest[HomeMixerRequest]]
|
||||
.map { pipelineRequest =>
|
||||
val pipelineArrow = productPipelineRegistry
|
||||
.getProductPipeline[HomeMixerRequest, tr.TimelineResponse](
|
||||
pipelineRequest.request.product)
|
||||
.arrow
|
||||
(pipelineArrow, pipelineRequest)
|
||||
}.applyArrow
|
||||
|
||||
transformThriftIntoPipelineRequest.andThen(underlyingProduct).map {
|
||||
_.result match {
|
||||
case Some(result) => found(result.timeline)
|
||||
case _ => missing
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
object HomeMixerColumn {
|
||||
val Path = "home-mixer/homeMixer.Timeline"
|
||||
private val FetchCallstack = s"$Path:fetch"
|
||||
private val MaxCount: Option[Int] = Some(100)
|
||||
}
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/javax/inject:javax.inject",
|
||||
"finagle/finagle-memcached/src/main/scala",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/thrift/com/twitter/search:earlybird-scala",
|
||||
"stitch/stitch-timelineservice/src/main/scala",
|
||||
],
|
||||
exports = [
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.candidate_source
|
||||
|
||||
import com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.search.earlybird.{thriftscala => t}
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
case object EarlybirdResponseTruncatedFeature
|
||||
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Boolean] {
|
||||
override val defaultValue: Boolean = false
|
||||
}
|
||||
|
||||
case object EarlybirdBottomTweetFeature
|
||||
extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Option[Long]] {
|
||||
override val defaultValue: Option[Long] = None
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class EarlybirdCandidateSource @Inject() (
|
||||
earlybird: t.EarlybirdService.MethodPerEndpoint)
|
||||
extends CandidateSourceWithExtractedFeatures[t.EarlybirdRequest, t.ThriftSearchResult] {
|
||||
|
||||
override val identifier = CandidateSourceIdentifier("Earlybird")
|
||||
|
||||
override def apply(
|
||||
request: t.EarlybirdRequest
|
||||
): Stitch[CandidatesWithSourceFeatures[t.ThriftSearchResult]] = {
|
||||
Stitch.callFuture(earlybird.search(request)).map { response =>
|
||||
val candidates = response.searchResults.map(_.results).getOrElse(Seq.empty)
|
||||
|
||||
val features = FeatureMapBuilder()
|
||||
.add(EarlybirdResponseTruncatedFeature, candidates.size == request.searchQuery.numResults)
|
||||
.add(EarlybirdBottomTweetFeature, candidates.lastOption.map(_.id))
|
||||
.build()
|
||||
|
||||
CandidatesWithSourceFeatures(candidates, features)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.candidate_source
|
||||
|
||||
import com.google.inject.name.Named
|
||||
import com.twitter.finagle.memcached.{Client => MemcachedClient}
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.StaleTweetsCache
|
||||
import com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource
|
||||
import com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier
|
||||
import com.twitter.stitch.Stitch
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class StaleTweetsCacheCandidateSource @Inject() (
|
||||
@Named(StaleTweetsCache) staleTweetsCache: MemcachedClient)
|
||||
extends CandidateSource[Seq[Long], Long] {
|
||||
|
||||
override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier("StaleTweetsCache")
|
||||
|
||||
private val StaleTweetsCacheKeyPrefix = "v1_"
|
||||
|
||||
override def apply(request: Seq[Long]): Stitch[Seq[Long]] = {
|
||||
val keys = request.map(StaleTweetsCacheKeyPrefix + _)
|
||||
|
||||
Stitch.callFuture(staleTweetsCache.get(keys).map { tweets =>
|
||||
tweets.map {
|
||||
case (k, _) => k.replaceFirst(StaleTweetsCacheKeyPrefix, "").toLong
|
||||
}.toSeq
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"finagle/finagle-core/src/main",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/scala/com/twitter/suggests/controller_data",
|
||||
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
|
||||
"stringcenter/client",
|
||||
"stringcenter/client/src/main/java",
|
||||
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder
|
||||
import com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder
|
||||
import com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
|
||||
object HomeConversationServiceCandidateDecorator {
|
||||
|
||||
private val ConversationModuleNamespace = EntryNamespace("home-conversation")
|
||||
|
||||
def apply(
|
||||
homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder
|
||||
): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {
|
||||
val suggestType = st.SuggestType.RankedOrganicTweet
|
||||
val component = InjectionScribeUtil.scribeComponent(suggestType).get
|
||||
val clientEventInfoBuilder = ClientEventInfoBuilder(component)
|
||||
val tweetItemBuilder = TweetCandidateUrtItemBuilder(
|
||||
clientEventInfoBuilder = clientEventInfoBuilder,
|
||||
timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),
|
||||
feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)
|
||||
)
|
||||
|
||||
val moduleBuilder = TimelineModuleBuilder(
|
||||
entryNamespace = ConversationModuleNamespace,
|
||||
clientEventInfoBuilder = clientEventInfoBuilder,
|
||||
displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),
|
||||
metadataBuilder = Some(HomeConversationModuleMetadataBuilder())
|
||||
)
|
||||
|
||||
Some(
|
||||
UrtMultipleModulesDecorator(
|
||||
urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),
|
||||
moduleBuilder = moduleBuilder,
|
||||
groupByKey = (_, _, candidateFeatures) =>
|
||||
candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)
|
||||
))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
|
||||
object HomeQueryTypePredicates {
|
||||
private[this] val QueryPredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
|
||||
("request", _ => true),
|
||||
("get_initial", _.getOrElse(GetInitialFeature, false)),
|
||||
("get_newer", _.getOrElse(GetNewerFeature, false)),
|
||||
("get_older", _.getOrElse(GetOlderFeature, false)),
|
||||
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
|
||||
("request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
|
||||
("request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false))
|
||||
)
|
||||
|
||||
val PredicateMap = QueryPredicates.toMap
|
||||
}
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"3rdparty/jvm/com/twitter/bijection:scrooge",
|
||||
"finagle/finagle-core/src/main",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"joinkey/src/main/scala/com/twitter/joinkey/context",
|
||||
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/scala/com/twitter/suggests/controller_data",
|
||||
"src/thrift/com/twitter/suggests/controller_data:controller_data-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
|
||||
"timelines/src/main/scala/com/twitter/timelines/injection/scribe",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.finagle.tracing.Trace
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
|
||||
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
|
||||
import com.twitter.suggests.controller_data.thriftscala.ControllerData
|
||||
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
|
||||
|
||||
case class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])
|
||||
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UniversalNoun[Any],
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
|
||||
tweetTypesBitmap = 0L,
|
||||
traceId = Some(Trace.id.traceId.toLong),
|
||||
requestJoinId = None)
|
||||
|
||||
val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(
|
||||
ControllerData.V2(
|
||||
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
|
||||
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = injectionType,
|
||||
controllerData = Some(serializedControllerData),
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.bijection.Base64String
|
||||
import com.twitter.bijection.scrooge.BinaryScalaCodec
|
||||
import com.twitter.bijection.{Injection => Serializer}
|
||||
import com.twitter.finagle.tracing.Trace
|
||||
import com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PositionFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.joinkey.context.RequestJoinKeyContext
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.suggests.controller_data.Home
|
||||
import com.twitter.suggests.controller_data.TweetTypeGenerator
|
||||
import com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}
|
||||
import com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}
|
||||
import com.twitter.suggests.controller_data.thriftscala.ControllerData
|
||||
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
|
||||
|
||||
object HomeClientEventDetailsBuilder {
|
||||
implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =
|
||||
BinaryScalaCodec(ControllerData)
|
||||
|
||||
val ControllerDataSerializer: Serializer[ControllerData, String] =
|
||||
Serializer.connect[ControllerData, Array[Byte], Base64String, String]
|
||||
|
||||
/**
|
||||
* define getRequestJoinId as a method(def) rather than a val because each new request
|
||||
* needs to call the context to update the id.
|
||||
*/
|
||||
private def getRequestJoinId(): Option[Long] =
|
||||
RequestJoinKeyContext.current.flatMap(_.requestJoinId)
|
||||
}
|
||||
|
||||
case class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](
|
||||
) extends BaseClientEventDetailsBuilder[Query, Candidate]
|
||||
with TweetTypeGenerator[FeatureMap] {
|
||||
|
||||
import HomeClientEventDetailsBuilder._
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: Candidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
|
||||
val tweetTypesBitmaps = mkTweetTypesBitmaps(
|
||||
Home.TweetTypeIdxMap,
|
||||
HomeTweetTypePredicates.PredicateMap,
|
||||
candidateFeatures)
|
||||
|
||||
val tweetTypesListBytes = mkItemTypesBitmapsV2(
|
||||
Home.TweetTypeIdxMap,
|
||||
HomeTweetTypePredicates.PredicateMap,
|
||||
candidateFeatures)
|
||||
|
||||
val candidateSourceId =
|
||||
candidateFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toByte)
|
||||
|
||||
val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(
|
||||
tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),
|
||||
tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),
|
||||
candidateTweetSourceId = candidateSourceId,
|
||||
traceId = Some(Trace.id.traceId.toLong),
|
||||
injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),
|
||||
tweetTypesListBytes = Some(tweetTypesListBytes),
|
||||
requestJoinId = getRequestJoinId(),
|
||||
)
|
||||
|
||||
val serializedControllerData = ControllerDataSerializer(
|
||||
ControllerData.V2(
|
||||
ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))
|
||||
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None).map(_.name),
|
||||
controllerData = Some(serializedControllerData),
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.EntityTokenFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.injection.scribe.InjectionScribeUtil
|
||||
|
||||
/**
|
||||
* Sets the [[ClientEventInfo]] with the `component` field set to the Suggest Type assigned to each candidate
|
||||
*/
|
||||
case class HomeClientEventInfoBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](
|
||||
detailsBuilder: Option[BaseClientEventDetailsBuilder[Query, Candidate]] = None)
|
||||
extends BaseClientEventInfoBuilder[Query, Candidate] {
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidate: Candidate,
|
||||
candidateFeatures: FeatureMap,
|
||||
element: Option[String]
|
||||
): Option[ClientEventInfo] = {
|
||||
val suggestType = candidateFeatures
|
||||
.getOrElse(SuggestTypeFeature, None)
|
||||
.getOrElse(throw new UnsupportedOperationException(s"No SuggestType was set"))
|
||||
|
||||
Some(
|
||||
ClientEventInfo(
|
||||
component = InjectionScribeUtil.scribeComponent(suggestType),
|
||||
element = element,
|
||||
details = detailsBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),
|
||||
action = None,
|
||||
/**
|
||||
* A backend entity encoded by the Client Entities Encoding Library.
|
||||
* Placeholder string for now
|
||||
*/
|
||||
entityToken = candidateFeatures.getOrElse(EntityTokenFeature, None)
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
case class HomeConversationModuleMetadataBuilder[
|
||||
-Query <: PipelineQuery,
|
||||
-Candidate <: BaseTweetCandidate
|
||||
]() extends BaseModuleMetadataBuilder[Query, Candidate] {
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidates: Seq[CandidateWithFeatures[Candidate]]
|
||||
): ModuleMetadata = ModuleMetadata(
|
||||
adsMetadata = None,
|
||||
conversationMetadata = Some(
|
||||
ModuleConversationMetadata(
|
||||
allTweetIds = Some((candidates.last.candidate.id +:
|
||||
candidates.last.features.getOrElse(AncestorsFeature, Seq.empty).map(_.tweetId)).reverse),
|
||||
socialContext = None,
|
||||
enableDeduplication = Some(true)
|
||||
)),
|
||||
gridCarouselMetadata = None
|
||||
)
|
||||
}
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
object HomeTimelinesScoreInfoBuilder
|
||||
extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val UndefinedTweetScore = -1.0
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[TimelinesScoreInfo] = {
|
||||
if (query.params(EnableSendScoresToClient)) {
|
||||
val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)
|
||||
Some(TimelinesScoreInfo(score))
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,256 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
|
||||
import com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures
|
||||
import com.twitter.tweetypie.{thriftscala => tpt}
|
||||
|
||||
object HomeTweetTypePredicates {
|
||||
|
||||
/**
|
||||
* IMPORTANT: Please avoid logging tweet types that are tied to sensitive
|
||||
* internal author information / labels (e.g. blink labels, abuse labels, or geo-location).
|
||||
*/
|
||||
private[this] val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(
|
||||
("with_candidate", _ => true),
|
||||
("retweet", _.getOrElse(IsRetweetFeature, false)),
|
||||
("reply", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),
|
||||
("image", _.getOrElse(EarlybirdFeature, None).exists(_.hasImage)),
|
||||
("video", _.getOrElse(EarlybirdFeature, None).exists(_.hasVideo)),
|
||||
("link", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),
|
||||
("quote", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),
|
||||
("like_social_context", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),
|
||||
("protected", _.getOrElse(EarlybirdFeature, None).exists(_.isProtected)),
|
||||
(
|
||||
"has_exclusive_conversation_author_id",
|
||||
_.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),
|
||||
("is_eligible_for_connect_boost", _ => false),
|
||||
("hashtag", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),
|
||||
("has_scheduled_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),
|
||||
("has_recorded_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),
|
||||
("is_read_from_cache", _.getOrElse(IsReadFromCacheFeature, false)),
|
||||
("get_initial", _.getOrElse(GetInitialFeature, false)),
|
||||
("get_newer", _.getOrElse(GetNewerFeature, false)),
|
||||
("get_middle", _.getOrElse(GetMiddleFeature, false)),
|
||||
("get_older", _.getOrElse(GetOlderFeature, false)),
|
||||
("pull_to_refresh", _.getOrElse(PullToRefreshFeature, false)),
|
||||
("polling", _.getOrElse(PollingFeature, false)),
|
||||
("near_empty", _.getOrElse(ServedSizeFeature, None).exists(_ < 3)),
|
||||
("is_request_context_launch", _.getOrElse(IsLaunchRequestFeature, false)),
|
||||
("mutual_follow", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),
|
||||
(
|
||||
"less_than_10_mins_since_lnpt",
|
||||
_.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 10.minutes)),
|
||||
("served_in_conversation_module", _.getOrElse(ServedInConversationModuleFeature, false)),
|
||||
("has_ticketed_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),
|
||||
("in_utis_top5", _.getOrElse(PositionFeature, None).exists(_ < 5)),
|
||||
(
|
||||
"conversation_module_has_2_displayed_tweets",
|
||||
_.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),
|
||||
("empty_request", _.getOrElse(ServedSizeFeature, None).exists(_ == 0)),
|
||||
("served_size_less_than_50", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),
|
||||
(
|
||||
"served_size_between_50_and_100",
|
||||
_.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),
|
||||
("authored_by_contextual_user", _.getOrElse(AuthoredByContextualUserFeature, false)),
|
||||
(
|
||||
"is_self_thread_tweet",
|
||||
_.getOrElse(ConversationFeature, None).exists(_.isSelfThreadTweet.contains(true))),
|
||||
("has_ancestors", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),
|
||||
("full_scoring_succeeded", _.getOrElse(FullScoringSucceededFeature, false)),
|
||||
("served_size_less_than_20", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),
|
||||
("served_size_less_than_10", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),
|
||||
("served_size_less_than_5", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),
|
||||
(
|
||||
"account_age_less_than_30_minutes",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),
|
||||
("conversation_module_has_gap", _.getOrElse(ConversationModuleHasGapFeature, false)),
|
||||
(
|
||||
"directed_at_user_is_in_first_degree",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),
|
||||
(
|
||||
"has_semantic_core_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),
|
||||
("is_request_context_foreground", _.getOrElse(IsForegroundRequestFeature, false)),
|
||||
(
|
||||
"account_age_less_than_1_day",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),
|
||||
(
|
||||
"account_age_less_than_7_days",
|
||||
_.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),
|
||||
(
|
||||
"part_of_utt",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
|
||||
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),
|
||||
(
|
||||
"has_home_latest_request_past_week",
|
||||
_.getOrElse(FollowingLastNonPollingTimeFeature, None).exists(_.untilNow < 7.days)),
|
||||
("is_utis_pos0", _.getOrElse(PositionFeature, None).exists(_ == 0)),
|
||||
("is_utis_pos1", _.getOrElse(PositionFeature, None).exists(_ == 1)),
|
||||
("is_utis_pos2", _.getOrElse(PositionFeature, None).exists(_ == 2)),
|
||||
("is_utis_pos3", _.getOrElse(PositionFeature, None).exists(_ == 3)),
|
||||
("is_utis_pos4", _.getOrElse(PositionFeature, None).exists(_ == 4)),
|
||||
("is_random_tweet", _.getOrElse(IsRandomTweetFeature, false)),
|
||||
("has_random_tweet_in_response", _.getOrElse(HasRandomTweetFeature, false)),
|
||||
("is_random_tweet_above_in_utis", _.getOrElse(IsRandomTweetAboveFeature, false)),
|
||||
(
|
||||
"has_ancestor_authored_by_viewer",
|
||||
candidate =>
|
||||
candidate
|
||||
.getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>
|
||||
candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),
|
||||
("ancestor", _.getOrElse(IsAncestorCandidateFeature, false)),
|
||||
(
|
||||
"deep_reply",
|
||||
candidate =>
|
||||
candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate
|
||||
.getOrElse(AncestorsFeature, Seq.empty).size > 2),
|
||||
(
|
||||
"has_simcluster_embeddings",
|
||||
_.getOrElse(
|
||||
SimclustersTweetTopKClustersWithScoresFeature,
|
||||
Map.empty[String, Double]).nonEmpty),
|
||||
(
|
||||
"tweet_age_less_than_15_seconds",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 15.seconds)),
|
||||
(
|
||||
"less_than_1_hour_since_lnpt",
|
||||
_.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 1.hour)),
|
||||
("has_gte_10_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),
|
||||
(
|
||||
"device_language_matches_tweet_language",
|
||||
candidate =>
|
||||
candidate.getOrElse(TweetLanguageFeature, None) ==
|
||||
candidate.getOrElse(DeviceLanguageFeature, None)),
|
||||
(
|
||||
"root_ancestor",
|
||||
candidate =>
|
||||
candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate
|
||||
.getOrElse(InReplyToTweetIdFeature, None).isEmpty),
|
||||
("question", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),
|
||||
("in_network", _.getOrElse(InNetworkFeature, true)),
|
||||
(
|
||||
"has_political_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(
|
||||
_.semanticCoreAnnotations.exists(
|
||||
_.exists(annotation =>
|
||||
SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||
|
||||
(annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
|
||||
annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),
|
||||
(
|
||||
"is_dont_at_me_by_invitation",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(
|
||||
_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),
|
||||
(
|
||||
"is_dont_at_me_community",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),
|
||||
("has_zero_score", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),
|
||||
(
|
||||
"is_followed_topic_tweet",
|
||||
_.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
.exists(_ == BasicTopicContextFunctionalityType)),
|
||||
(
|
||||
"is_recommended_topic_tweet",
|
||||
_.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
.exists(_ == RecommendationTopicContextFunctionalityType)),
|
||||
("has_gte_100_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),
|
||||
("has_gte_1k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),
|
||||
(
|
||||
"has_gte_10k_favs",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),
|
||||
(
|
||||
"has_gte_100k_favs",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),
|
||||
("has_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),
|
||||
("has_live_audio_space", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),
|
||||
(
|
||||
"has_gte_10_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),
|
||||
(
|
||||
"has_gte_100_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),
|
||||
(
|
||||
"has_gte_1k_retweets",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),
|
||||
(
|
||||
"has_us_political_annotation",
|
||||
_.getOrElse(EarlybirdFeature, None)
|
||||
.exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>
|
||||
annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&
|
||||
annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&
|
||||
annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),
|
||||
(
|
||||
"has_toxicity_score_above_threshold",
|
||||
_.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),
|
||||
("is_topic_tweet", _.getOrElse(TopicIdSocialContextFeature, None).isDefined),
|
||||
(
|
||||
"text_only",
|
||||
candidate =>
|
||||
candidate.getOrElse(HasDisplayedTextFeature, false) &&
|
||||
!(candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) ||
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasVideo) ||
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),
|
||||
(
|
||||
"image_only",
|
||||
candidate =>
|
||||
candidate.getOrElse(EarlybirdFeature, None).exists(_.hasImage) &&
|
||||
!candidate.getOrElse(HasDisplayedTextFeature, false)),
|
||||
("has_1_image", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),
|
||||
("has_2_images", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),
|
||||
("has_3_images", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),
|
||||
("has_4_images", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),
|
||||
("has_card", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),
|
||||
("user_follow_count_gte_50", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),
|
||||
(
|
||||
"has_liked_by_social_context",
|
||||
candidateFeatures =>
|
||||
candidateFeatures
|
||||
.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)
|
||||
.exists(candidateFeatures
|
||||
.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Seq.empty).toSet.contains)),
|
||||
(
|
||||
"has_followed_by_social_context",
|
||||
_.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty),
|
||||
(
|
||||
"has_topic_social_context",
|
||||
candidateFeatures =>
|
||||
candidateFeatures
|
||||
.getOrElse(TopicIdSocialContextFeature, None)
|
||||
.isDefined &&
|
||||
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None).isDefined),
|
||||
("video_lte_10_sec", _.getOrElse(VideoDurationMsFeature, None).exists(_ <= 10000)),
|
||||
(
|
||||
"video_bt_10_60_sec",
|
||||
_.getOrElse(VideoDurationMsFeature, None).exists(duration =>
|
||||
duration > 10000 && duration <= 60000)),
|
||||
("video_gt_60_sec", _.getOrElse(VideoDurationMsFeature, None).exists(_ > 60000)),
|
||||
(
|
||||
"tweet_age_lte_30_minutes",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 30.minutes)),
|
||||
(
|
||||
"tweet_age_lte_1_hour",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 1.hour)),
|
||||
(
|
||||
"tweet_age_lte_6_hours",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 6.hours)),
|
||||
(
|
||||
"tweet_age_lte_12_hours",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow <= 12.hours)),
|
||||
(
|
||||
"tweet_age_gte_24_hours",
|
||||
_.getOrElse(OriginalTweetCreationTimeFromSnowflakeFeature, None)
|
||||
.exists(_.untilNow >= 24.hours)),
|
||||
)
|
||||
|
||||
val PredicateMap = CandidatePredicates.toMap
|
||||
}
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.builder
|
||||
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
|
||||
case class ListClientEventDetailsBuilder(suggestType: st.SuggestType)
|
||||
extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UniversalNoun[Any],
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ClientEventDetails] = {
|
||||
val clientEventDetails = ClientEventDetails(
|
||||
conversationDetails = None,
|
||||
timelinesDetails = Some(
|
||||
TimelinesDetails(
|
||||
injectionType = Some(suggestType.name),
|
||||
controllerData = None,
|
||||
sourceData = None)),
|
||||
articleDetails = None,
|
||||
liveEventDetails = None,
|
||||
commerceDetails = None
|
||||
)
|
||||
|
||||
Some(clientEventDetails)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.AlwaysInclude
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction
|
||||
import com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtInstructionBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.Cover
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
case class AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder[Query <: PipelineQuery](
|
||||
override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)
|
||||
extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {
|
||||
|
||||
override def build(
|
||||
query: Query,
|
||||
entries: Seq[TimelineEntry]
|
||||
): Seq[AddEntriesTimelineInstruction] = {
|
||||
if (includeInstruction(query, entries)) {
|
||||
val entriesToAdd = entries
|
||||
.filterNot(_.isInstanceOf[ShowAlert])
|
||||
.filterNot(_.isInstanceOf[Cover])
|
||||
.filter(_.entryIdToReplace.isEmpty)
|
||||
if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))
|
||||
else Seq.empty
|
||||
} else
|
||||
Seq.empty
|
||||
}
|
||||
}
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class AuthorChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>
|
||||
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
|
||||
userId = authorId,
|
||||
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
|
||||
promptExternalString = externalStrings.showFewerTweetsString,
|
||||
confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,
|
||||
engagementType = t.FeedbackEngagementType.Tweet,
|
||||
stringCenter = stringCenter,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt",
|
||||
"src/thrift/com/twitter/timelines/service:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class BlockUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val userIdOpt =
|
||||
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
|
||||
candidateFeatures.getOrElse(SourceUserIdFeature, None)
|
||||
else candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.blockUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = Some(BottomSheet),
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.No),
|
||||
richBehavior = Some(RichFeedbackBehaviorBlockUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.{thriftscala => tls}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class DontLikeFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,
|
||||
retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,
|
||||
notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,
|
||||
unfollowUserChildFeedbackActionBuilder: UnfollowUserChildFeedbackActionBuilder,
|
||||
muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder,
|
||||
blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,
|
||||
reportTweetChildFeedbackActionBuilder: ReportTweetChildFeedbackActionBuilder) {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackAction] = {
|
||||
CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>
|
||||
val feedbackEntities = Seq(
|
||||
tlc.FeedbackEntity.TweetId(candidate.id),
|
||||
tlc.FeedbackEntity.UserId(authorId)
|
||||
)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = None,
|
||||
entityIds = feedbackEntities,
|
||||
ttl = Some(30.days)
|
||||
)
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tls.FeedbackType.DontLike,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
val childFeedbackActions = if (query.params(EnableNahFeedbackInfoParam)) {
|
||||
Seq(
|
||||
unfollowUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
muteUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
blockUserChildFeedbackActionBuilder(candidateFeatures),
|
||||
reportTweetChildFeedbackActionBuilder(candidate)
|
||||
).flatten
|
||||
} else {
|
||||
Seq(
|
||||
authorChildFeedbackActionBuilder(candidateFeatures),
|
||||
retweeterChildFeedbackActionBuilder(candidateFeatures),
|
||||
notRelevantChildFeedbackActionBuilder(candidate, candidateFeatures)
|
||||
).flatten
|
||||
}
|
||||
|
||||
FeedbackAction(
|
||||
feedbackType = DontLike,
|
||||
prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),
|
||||
confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),
|
||||
childFeedbackActions =
|
||||
if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(Frown),
|
||||
richBehavior = None,
|
||||
subprompt = None,
|
||||
encodedFeedbackRequest = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,119 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.stringcenter.client.core.ExternalString
|
||||
|
||||
private[decorator] case class SocialContextIdAndScreenName(
|
||||
socialContextId: Long,
|
||||
screenName: String)
|
||||
|
||||
object EngagerSocialContextBuilder {
|
||||
private val UserIdRequestParamName = "user_id"
|
||||
private val DirectInjectionContentSourceRequestParamName = "dis"
|
||||
private val DirectInjectionIdRequestParamName = "diid"
|
||||
private val DirectInjectionContentSourceSocialProofUsers = "socialproofusers"
|
||||
private val SocialProofUrl = ""
|
||||
}
|
||||
|
||||
case class EngagerSocialContextBuilder(
|
||||
contextType: GeneralContextType,
|
||||
stringCenter: StringCenter,
|
||||
oneUserString: ExternalString,
|
||||
twoUsersString: ExternalString,
|
||||
moreUsersString: ExternalString,
|
||||
timelineTitle: ExternalString) {
|
||||
import EngagerSocialContextBuilder._
|
||||
|
||||
def apply(
|
||||
socialContextIds: Seq[Long],
|
||||
query: PipelineQuery,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
|
||||
val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>
|
||||
realNames
|
||||
.get(socialContextId).map(screenName =>
|
||||
SocialContextIdAndScreenName(socialContextId, screenName))
|
||||
}
|
||||
|
||||
validSocialContextIdAndScreenNames match {
|
||||
case Seq(user) =>
|
||||
val socialContextString =
|
||||
stringCenter.prepare(oneUserString, Map("user" -> user.screenName))
|
||||
Some(mkOneUserSocialContext(socialContextString, user.socialContextId))
|
||||
case Seq(firstUser, secondUser) =>
|
||||
val socialContextString =
|
||||
stringCenter
|
||||
.prepare(
|
||||
twoUsersString,
|
||||
Map("user1" -> firstUser.screenName, "user2" -> secondUser.screenName))
|
||||
Some(
|
||||
mkManyUserSocialContext(
|
||||
socialContextString,
|
||||
query.getRequiredUserId,
|
||||
validSocialContextIdAndScreenNames.map(_.socialContextId)))
|
||||
|
||||
case firstUser +: otherUsers =>
|
||||
val otherUsersCount = otherUsers.size
|
||||
val socialContextString =
|
||||
stringCenter
|
||||
.prepare(
|
||||
moreUsersString,
|
||||
Map("user" -> firstUser.screenName, "count" -> otherUsersCount))
|
||||
Some(
|
||||
mkManyUserSocialContext(
|
||||
socialContextString,
|
||||
query.getRequiredUserId,
|
||||
validSocialContextIdAndScreenNames.map(_.socialContextId)))
|
||||
case _ => None
|
||||
}
|
||||
}
|
||||
|
||||
private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {
|
||||
GeneralContext(
|
||||
contextType = contextType,
|
||||
text = socialContextString,
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private def mkManyUserSocialContext(
|
||||
socialContextString: String,
|
||||
viewerId: Long,
|
||||
socialContextIds: Seq[Long]
|
||||
): GeneralContext = {
|
||||
GeneralContext(
|
||||
contextType = contextType,
|
||||
text = socialContextString,
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = UrtEndpoint,
|
||||
url = SocialProofUrl,
|
||||
urtEndpointOptions = Some(UrtEndpointOptions(
|
||||
requestParams = Some(Map(
|
||||
UserIdRequestParamName -> viewerId.toString,
|
||||
DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,
|
||||
DirectInjectionIdRequestParamName -> socialContextIds.mkString(",")
|
||||
)),
|
||||
title = Some(stringCenter.prepare(timelineTitle)),
|
||||
cacheId = None,
|
||||
subtitle = None
|
||||
))
|
||||
))
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Use '@A replied' when the root tweet is out-of-network and the reply is in network.
|
||||
*
|
||||
* This function should only be called for the root Tweet of convo modules. This is enforced by
|
||||
* [[HomeTweetSocialContextBuilder]].
|
||||
*/
|
||||
@Singleton
|
||||
case class ExtendedReplySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val extendedReplyString = externalStrings.socialContextExtendedReply
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// If these values are missing default to not showing an extended reply banner
|
||||
val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
|
||||
val inNetworkFocalTweet =
|
||||
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)
|
||||
|
||||
if (!inNetworkRoot && inNetworkFocalTweet) {
|
||||
|
||||
val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)
|
||||
val focalTweetRealNames =
|
||||
candidateFeatures
|
||||
.getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])
|
||||
val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)
|
||||
|
||||
(focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {
|
||||
case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = ConversationGeneralContextType,
|
||||
text = stringCenter
|
||||
.prepare(extendedReplyString, placeholders = Map("user1" -> focalTweetAuthorName)),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
))
|
||||
))
|
||||
case _ =>
|
||||
None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.ExternalStringRegistry
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class FeedbackStrings @Inject() (
|
||||
@ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry]) {
|
||||
private val externalStringRegistry = externalStringRegistryProvider.get()
|
||||
|
||||
val seeLessOftenFeedbackString =
|
||||
externalStringRegistry.createProdString("Feedback.seeLessOften")
|
||||
val seeLessOftenConfirmationFeedbackString =
|
||||
externalStringRegistry.createProdString("Feedback.seeLessOftenConfirmation")
|
||||
}
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.stringcenter.client.core.ExternalString
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
import com.twitter.timelineservice.{thriftscala => tlst}
|
||||
|
||||
object FeedbackUtil {
|
||||
|
||||
val FeedbackTtl = 30.days
|
||||
|
||||
def buildUserSeeFewerChildFeedbackAction(
|
||||
userId: Long,
|
||||
namesByUserId: Map[Long, String],
|
||||
promptExternalString: ExternalString,
|
||||
confirmationExternalString: ExternalString,
|
||||
engagementType: t.FeedbackEngagementType,
|
||||
stringCenter: StringCenter,
|
||||
injectionType: Option[st.SuggestType]
|
||||
): Option[ChildFeedbackAction] = {
|
||||
namesByUserId.get(userId).map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
promptExternalString,
|
||||
Map("user" -> userScreenName)
|
||||
)
|
||||
val confirmation = stringCenter.prepare(
|
||||
confirmationExternalString,
|
||||
Map("user" -> userScreenName)
|
||||
)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = Some(engagementType),
|
||||
entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),
|
||||
ttl = Some(FeedbackTtl))
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tlst.FeedbackType.SeeFewer,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = injectionType
|
||||
)
|
||||
|
||||
ChildFeedbackAction(
|
||||
feedbackType = SeeFewer,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior = None,
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class FollowedBySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
|
||||
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
|
||||
contextType = FollowGeneralContextType,
|
||||
stringCenter = stringCenter,
|
||||
oneUserString = externalStrings.socialContextOneUserFollowsString,
|
||||
twoUsersString = externalStrings.socialContextTwoUsersFollowString,
|
||||
moreUsersString = externalStrings.socialContextMoreUsersFollowString,
|
||||
timelineTitle = externalStrings.socialContextFollowedByTimelineTitle
|
||||
)
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
// Only apply followed-by social context for OON Tweets
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
if (!inNetwork) {
|
||||
val validFollowedByUserIds =
|
||||
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
|
||||
engagerSocialContextBuilder(
|
||||
socialContextIds = validFollowedByUserIds,
|
||||
query = query,
|
||||
candidateFeatures = candidateFeatures
|
||||
)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import com.twitter.timelines.util.FeedbackMetadataSerializer
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class HomeFeedbackActionInfoBuilder @Inject() (
|
||||
notInterestedTopicFeedbackActionBuilder: NotInterestedTopicFeedbackActionBuilder,
|
||||
dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)
|
||||
extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackActionInfo] = {
|
||||
val supportedProduct = query.product match {
|
||||
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
|
||||
case ForYouProduct => true
|
||||
case _ => false
|
||||
}
|
||||
val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)
|
||||
|
||||
if (supportedProduct && !isAuthoredByViewer) {
|
||||
val feedbackActions = Seq(
|
||||
notInterestedTopicFeedbackActionBuilder(candidateFeatures),
|
||||
dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)
|
||||
).flatten
|
||||
val feedbackMetadata = FeedbackMetadataSerializer.serialize(
|
||||
t.FeedbackMetadata(injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)))
|
||||
|
||||
Some(
|
||||
FeedbackActionInfo(
|
||||
feedbackActions = feedbackActions,
|
||||
feedbackMetadata = Some(feedbackMetadata),
|
||||
displayContext = None,
|
||||
clientEventInfo = None
|
||||
))
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class HomeTweetSocialContextBuilder @Inject() (
|
||||
likedBySocialContextBuilder: LikedBySocialContextBuilder,
|
||||
listsSocialContextBuilder: ListsSocialContextBuilder,
|
||||
followedBySocialContextBuilder: FollowedBySocialContextBuilder,
|
||||
topicSocialContextBuilder: TopicSocialContextBuilder,
|
||||
extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,
|
||||
receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder,
|
||||
popularVideoSocialContextBuilder: PopularVideoSocialContextBuilder,
|
||||
popularInYourAreaSocialContextBuilder: PopularInYourAreaSocialContextBuilder)
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
features: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
if (query.params(EnableSocialContextParam)) {
|
||||
features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {
|
||||
case None =>
|
||||
likedBySocialContextBuilder(query, candidate, features)
|
||||
.orElse(followedBySocialContextBuilder(query, candidate, features))
|
||||
.orElse(topicSocialContextBuilder(query, candidate, features))
|
||||
.orElse(popularVideoSocialContextBuilder(query, candidate, features))
|
||||
.orElse(listsSocialContextBuilder(query, candidate, features))
|
||||
.orElse(popularInYourAreaSocialContextBuilder(query, candidate, features))
|
||||
case Some(_) =>
|
||||
val conversationId = features.getOrElse(ConversationModuleIdFeature, None)
|
||||
// Only hydrate the social context into the root tweet in a conversation module
|
||||
if (conversationId.contains(candidate.id)) {
|
||||
extendedReplySocialContextBuilder(query, candidate, features)
|
||||
.orElse(receivedReplySocialContextBuilder(query, candidate, features))
|
||||
} else None
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.service.{thriftscala => tl}
|
||||
import com.twitter.timelines.util.FeedbackRequestSerializer
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import com.twitter.timelineservice.thriftscala.FeedbackType
|
||||
|
||||
object HomeWhoToFollowFeedbackActionInfoBuilder {
|
||||
private val FeedbackMetadata = tl.FeedbackMetadata(
|
||||
injectionType = Some(SuggestType.WhoToFollow),
|
||||
engagementType = None,
|
||||
entityIds = Seq.empty,
|
||||
ttlMs = None
|
||||
)
|
||||
private val FeedbackRequest =
|
||||
tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)
|
||||
private val EncodedFeedbackRequest =
|
||||
FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class HomeWhoToFollowFeedbackActionInfoBuilder @Inject() (
|
||||
feedbackStrings: FeedbackStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
|
||||
|
||||
private val whoToFollowFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
|
||||
seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,
|
||||
seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,
|
||||
stringCenter = stringCenterProvider.get(),
|
||||
encodedFeedbackRequest = Some(HomeWhoToFollowFeedbackActionInfoBuilder.EncodedFeedbackRequest)
|
||||
)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UserCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackActionInfo] =
|
||||
whoToFollowFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)
|
||||
}
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.component_library.model.candidate.UserCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelines.service.{thriftscala => tl}
|
||||
import com.twitter.timelines.util.FeedbackRequestSerializer
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import com.twitter.timelineservice.thriftscala.FeedbackType
|
||||
|
||||
object HomeWhoToSubscribeFeedbackActionInfoBuilder {
|
||||
private val FeedbackMetadata = tl.FeedbackMetadata(
|
||||
injectionType = Some(SuggestType.WhoToSubscribe),
|
||||
engagementType = None,
|
||||
entityIds = Seq.empty,
|
||||
ttlMs = None
|
||||
)
|
||||
private val FeedbackRequest =
|
||||
tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)
|
||||
private val EncodedFeedbackRequest =
|
||||
FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class HomeWhoToSubscribeFeedbackActionInfoBuilder @Inject() (
|
||||
feedbackStrings: FeedbackStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {
|
||||
|
||||
private val whoToSubscribeFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(
|
||||
seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,
|
||||
seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,
|
||||
stringCenter = stringCenterProvider.get(),
|
||||
encodedFeedbackRequest =
|
||||
Some(HomeWhoToSubscribeFeedbackActionInfoBuilder.EncodedFeedbackRequest)
|
||||
)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: UserCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackActionInfo] =
|
||||
whoToSubscribeFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)
|
||||
}
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class LikedBySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
|
||||
private val engagerSocialContextBuilder = EngagerSocialContextBuilder(
|
||||
contextType = LikeGeneralContextType,
|
||||
stringCenter = stringCenter,
|
||||
oneUserString = externalStrings.socialContextOneUserLikedString,
|
||||
twoUsersString = externalStrings.socialContextTwoUsersLikedString,
|
||||
moreUsersString = externalStrings.socialContextMoreUsersLikedString,
|
||||
timelineTitle = externalStrings.socialContextLikedByTimelineTitle
|
||||
)
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// Liked by users are valid only if they pass both the SGS and Perspective filters.
|
||||
val validLikedByUserIds =
|
||||
candidateFeatures
|
||||
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
|
||||
.filter(
|
||||
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
|
||||
|
||||
engagerSocialContextBuilder(
|
||||
socialContextIds = validLikedByUserIds,
|
||||
query = query,
|
||||
candidateFeatures = candidateFeatures
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => t}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* "Your Lists" will be rendered for the context and a url link for your lists.
|
||||
*/
|
||||
@Singleton
|
||||
case class ListsSocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val listString = externalStrings.ownedSubscribedListsModuleHeaderString
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
candidateFeatures.get(SuggestTypeFeature) match {
|
||||
case Some(suggestType) if suggestType == t.SuggestType.RankedListTweet =>
|
||||
val userName = query.features.flatMap(_.getOrElse(UserScreenNameFeature, None))
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = ListGeneralContextType,
|
||||
text = stringCenter.prepare(listString),
|
||||
url = userName.map(name => ""),
|
||||
contextImageUrls = None,
|
||||
landingUrl = None
|
||||
))
|
||||
case _ => None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class MuteUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ChildFeedbackAction] = {
|
||||
val userIdOpt =
|
||||
if (candidateFeatures.getOrElse(IsRetweetFeature, false))
|
||||
candidateFeatures.getOrElse(SourceUserIdFeature, None)
|
||||
else candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.muteUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.SpeakerOff),
|
||||
richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class NotInterestedTopicFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[FeedbackAction] = {
|
||||
val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
val validFollowedByUserIds =
|
||||
candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)
|
||||
val validLikedByUserIds =
|
||||
candidateFeatures
|
||||
.getOrElse(SGSValidLikedByUserIdsFeature, Nil)
|
||||
.filter(
|
||||
candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)
|
||||
|
||||
if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {
|
||||
val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
|
||||
val topicContextFunctionalityType =
|
||||
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
|
||||
(topicIdSocialContext, topicContextFunctionalityType) match {
|
||||
case (Some(topicId), Some(topicContextFunctionalityType))
|
||||
if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||
|
||||
topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>
|
||||
Some(
|
||||
FeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = None,
|
||||
confirmation = None,
|
||||
childFeedbackActions = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior =
|
||||
Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),
|
||||
subprompt = None,
|
||||
encodedFeedbackRequest = None
|
||||
)
|
||||
)
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.common.{thriftscala => tlc}
|
||||
import com.twitter.timelineservice.model.FeedbackInfo
|
||||
import com.twitter.timelineservice.model.FeedbackMetadata
|
||||
import com.twitter.timelineservice.{thriftscala => tlst}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class NotRelevantChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[ChildFeedbackAction] = {
|
||||
val prompt = stringCenter.prepare(externalStrings.notRelevantString)
|
||||
val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)
|
||||
val feedbackMetadata = FeedbackMetadata(
|
||||
engagementType = None,
|
||||
entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),
|
||||
ttl = Some(FeedbackUtil.FeedbackTtl))
|
||||
val feedbackUrl = FeedbackInfo.feedbackUrl(
|
||||
feedbackType = tlst.FeedbackType.NotRelevant,
|
||||
feedbackMetadata = feedbackMetadata,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
|
||||
Some(
|
||||
ChildFeedbackAction(
|
||||
feedbackType = NotRelevant,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = Some(feedbackUrl),
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = None,
|
||||
richBehavior = None,
|
||||
subprompt = None
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class PopularInYourAreaSocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val popularInYourAreaString = externalStrings.socialContextPopularInYourAreaString
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
if (suggestTypeOpt.contains(st.SuggestType.RecommendedTrendTweet)) {
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = LocationGeneralContextType,
|
||||
text = stringCenter.prepare(popularInYourAreaString),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = None
|
||||
))
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelineservice.suggests.{thriftscala => st}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class PopularVideoSocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val popularVideoString = externalStrings.socialContextPopularVideoString
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
if (suggestTypeOpt.contains(st.SuggestType.MediaTweet)) {
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = SparkleGeneralContextType,
|
||||
text = stringCenter.prepare(popularVideoString),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = ""
|
||||
)
|
||||
)
|
||||
))
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Provider
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.
|
||||
*
|
||||
* This function should only be called for the root Tweet of convo modules. This is enforced by
|
||||
* [[HomeTweetSocialContextBuilder]].
|
||||
*/
|
||||
@Singleton
|
||||
case class ReceivedReplySocialContextBuilder @Inject() (
|
||||
externalStrings: HomeMixerExternalStrings,
|
||||
@ProductScoped stringCenterProvider: Provider[StringCenter])
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
private val stringCenter = stringCenterProvider.get()
|
||||
private val receivedReplyString = externalStrings.socialContextReceivedReply
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
|
||||
// If these values are missing default to not showing a received a reply banner
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
|
||||
val inNetworkFocalTweet =
|
||||
candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)
|
||||
|
||||
if (inNetwork && !inNetworkFocalTweet) {
|
||||
|
||||
val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])
|
||||
val authorNameOpt = authorIdOpt.flatMap(realNames.get)
|
||||
|
||||
(authorIdOpt, authorNameOpt) match {
|
||||
case (Some(authorId), Some(authorName)) =>
|
||||
Some(
|
||||
GeneralContext(
|
||||
contextType = ConversationGeneralContextType,
|
||||
text = stringCenter
|
||||
.prepare(receivedReplyString, placeholders = Map("user1" -> authorName)),
|
||||
url = None,
|
||||
contextImageUrls = None,
|
||||
landingUrl = Some(
|
||||
Url(
|
||||
urlType = DeepLink,
|
||||
url = "",
|
||||
urtEndpointOptions = None
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class ReportTweetChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(
|
||||
candidate: TweetCandidate
|
||||
): Option[ChildFeedbackAction] = {
|
||||
Some(
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),
|
||||
confirmation = None,
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.Flag),
|
||||
richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),
|
||||
subprompt = None
|
||||
)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import com.twitter.timelines.service.{thriftscala => t}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class RetweeterChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)
|
||||
|
||||
if (isRetweet) {
|
||||
candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>
|
||||
FeedbackUtil.buildUserSeeFewerChildFeedbackAction(
|
||||
userId = retweeterId,
|
||||
namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),
|
||||
promptExternalString = externalStrings.showFewerRetweetsString,
|
||||
confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,
|
||||
engagementType = t.FeedbackEngagementType.Retweet,
|
||||
stringCenter = stringCenter,
|
||||
injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)
|
||||
)
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class TopicSocialContextBuilder @Inject() ()
|
||||
extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {
|
||||
|
||||
def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
candidateFeatures: FeatureMap
|
||||
): Option[SocialContext] = {
|
||||
val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)
|
||||
if (!inNetwork) {
|
||||
val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)
|
||||
val topicContextFunctionalityTypeOpt =
|
||||
candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)
|
||||
(topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {
|
||||
case (Some(topicId), Some(topicContextFunctionalityType)) =>
|
||||
Some(
|
||||
TopicContext(
|
||||
topicId = topicId.toString,
|
||||
functionalityType = Some(topicContextFunctionalityType)
|
||||
))
|
||||
case _ => None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.decorator.urt.builder
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.icon
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser
|
||||
import com.twitter.product_mixer.core.product.guice.scope.ProductScoped
|
||||
import com.twitter.stringcenter.client.StringCenter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class UnfollowUserChildFeedbackActionBuilder @Inject() (
|
||||
@ProductScoped stringCenter: StringCenter,
|
||||
externalStrings: HomeMixerExternalStrings) {
|
||||
|
||||
def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {
|
||||
val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)
|
||||
val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
if (isInNetwork) {
|
||||
userIdOpt.flatMap { userId =>
|
||||
val screenNamesMap =
|
||||
candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])
|
||||
val userScreenNameOpt = screenNamesMap.get(userId)
|
||||
userScreenNameOpt.map { userScreenName =>
|
||||
val prompt = stringCenter.prepare(
|
||||
externalStrings.unfollowUserString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
val confirmation = stringCenter.prepare(
|
||||
externalStrings.unfollowUserConfirmationString,
|
||||
Map("username" -> userScreenName)
|
||||
)
|
||||
ChildFeedbackAction(
|
||||
feedbackType = RichBehavior,
|
||||
prompt = Some(prompt),
|
||||
confirmation = Some(confirmation),
|
||||
feedbackUrl = None,
|
||||
hasUndoAction = Some(true),
|
||||
confirmationDisplayType = None,
|
||||
clientEventInfo = None,
|
||||
icon = Some(icon.Unfollow),
|
||||
richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),
|
||||
subprompt = None
|
||||
)
|
||||
}
|
||||
}
|
||||
} else None
|
||||
}
|
||||
}
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi",
|
||||
"configapi/configapi-decider",
|
||||
"finatra/inject/inject-core/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/service",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie",
|
||||
"joinkey/src/main/scala/com/twitter/joinkey/context",
|
||||
"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util",
|
||||
"snowflake/src/main/scala/com/twitter/snowflake/id",
|
||||
"src/java/com/twitter/search/common/util/lang",
|
||||
"src/scala/com/twitter/timelines/prediction/adapters/request_context",
|
||||
"src/thrift/com/twitter/gizmoduck:thrift-scala",
|
||||
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
|
||||
"src/thrift/com/twitter/search:earlybird-scala",
|
||||
"src/thrift/com/twitter/search/common:constants-java",
|
||||
"src/thrift/com/twitter/socialgraph:thrift-scala",
|
||||
"src/thrift/com/twitter/spam/rtf:safety-result-scala",
|
||||
"src/thrift/com/twitter/timelineranker:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/impression:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala",
|
||||
"src/thrift/com/twitter/timelines/real_graph:real_graph-scala",
|
||||
"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala",
|
||||
"src/thrift/com/twitter/tweetypie:service-scala",
|
||||
"src/thrift/com/twitter/tweetypie:tweet-scala",
|
||||
"src/thrift/com/twitter/user_session_store:thrift-java",
|
||||
"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala",
|
||||
"stitch/stitch-core",
|
||||
"stitch/stitch-gizmoduck",
|
||||
"stitch/stitch-socialgraph",
|
||||
"stitch/stitch-timelineservice",
|
||||
"stitch/stitch-tweetypie",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/feedback",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence",
|
||||
"timelines/src/main/scala/com/twitter/timelines/clients/manhattan/store",
|
||||
"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter",
|
||||
"timelines/src/main/scala/com/twitter/timelines/impressionstore/store",
|
||||
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
|
||||
"user_session_store/src/main/scala/com/twitter/user_session_store",
|
||||
"util/util-core",
|
||||
],
|
||||
exports = [
|
||||
"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala",
|
||||
"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.timelinemixer.clients.manhattan.InjectionHistoryClient
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.manhattan.DismissInfo
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
object DismissInfoQueryFeatureHydrator {
|
||||
val DismissInfoSuggestTypes = Seq(SuggestType.WhoToFollow)
|
||||
}
|
||||
|
||||
@Singleton
|
||||
case class DismissInfoQueryFeatureHydrator @Inject() (
|
||||
dismissInfoClient: InjectionHistoryClient)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("DismissInfo")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(DismissInfoFeature)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] =
|
||||
Stitch.callFuture {
|
||||
dismissInfoClient
|
||||
.readDismissInfoEntries(
|
||||
query.getRequiredUserId,
|
||||
DismissInfoQueryFeatureHydrator.DismissInfoSuggestTypes).map { response =>
|
||||
val dismissInfoMap = response.mapValues(DismissInfo.fromThrift)
|
||||
FeatureMapBuilder().add(DismissInfoFeature, dismissInfoMap).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8, 50, 60, 60)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClient
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class FeedbackHistoryQueryFeatureHydrator @Inject() (
|
||||
feedbackHistoryClient: FeedbackHistoryManhattanClient)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("FeedbackHistory")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(FeedbackHistoryFeature)
|
||||
|
||||
override def hydrate(
|
||||
query: PipelineQuery
|
||||
): Stitch[FeatureMap] =
|
||||
Stitch
|
||||
.callFuture(feedbackHistoryClient.get(query.getRequiredUserId))
|
||||
.map { feedbackHistory =>
|
||||
FeatureMapBuilder().add(FeedbackHistoryFeature, feedbackHistory).build()
|
||||
}
|
||||
}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gt}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserFollowingCountFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.gizmoduck.Gizmoduck
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class GizmoduckUserQueryFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("GizmoduckUser")
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(UserFollowingCountFeature, UserTypeFeature, UserScreenNameFeature)
|
||||
|
||||
private val queryFields: Set[gt.QueryFields] =
|
||||
Set(gt.QueryFields.Counts, gt.QueryFields.Safety, gt.QueryFields.Profile)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
gizmoduck
|
||||
.getUserById(
|
||||
userId = userId,
|
||||
queryFields = queryFields,
|
||||
context = gt.LookupContext(forUserId = Some(userId), includeSoftUsers = true))
|
||||
.map { user =>
|
||||
FeatureMapBuilder()
|
||||
.add(UserFollowingCountFeature, user.counts.map(_.following.toInt))
|
||||
.add(UserTypeFeature, Some(user.userType))
|
||||
.add(UserScreenNameFeature, user.profile.map(_.screenName))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature
|
||||
import com.twitter.home_mixer.model.request.HasSeenTweetIds
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.ImpressionBloomFilterFalsePositiveRateParam
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.clients.manhattan.store.ManhattanStoreClient
|
||||
import com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}
|
||||
import com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilter
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class ImpressionBloomFilterQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
|
||||
bloomFilterClient: ManhattanStoreClient[
|
||||
blm.ImpressionBloomFilterKey,
|
||||
blm.ImpressionBloomFilterSeq
|
||||
]) extends QueryFeatureHydrator[Query] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(
|
||||
"ImpressionBloomFilter")
|
||||
|
||||
private val ImpressionBloomFilterTTL = 7.day
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ImpressionBloomFilterFeature)
|
||||
|
||||
private val SurfaceArea = blm.SurfaceArea.HomeTimeline
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
val userId = query.getRequiredUserId
|
||||
bloomFilterClient
|
||||
.get(blm.ImpressionBloomFilterKey(userId, SurfaceArea))
|
||||
.map(_.getOrElse(blm.ImpressionBloomFilterSeq(Seq.empty)))
|
||||
.map { bloomFilterSeq =>
|
||||
val updatedBloomFilterSeq =
|
||||
if (query.seenTweetIds.forall(_.isEmpty)) bloomFilterSeq
|
||||
else {
|
||||
ImpressionBloomFilter.addSeenTweetIds(
|
||||
surfaceArea = SurfaceArea,
|
||||
tweetIds = query.seenTweetIds.get,
|
||||
bloomFilterSeq = bloomFilterSeq,
|
||||
timeToLive = ImpressionBloomFilterTTL,
|
||||
falsePositiveRate = query.params(ImpressionBloomFilterFalsePositiveRateParam)
|
||||
)
|
||||
}
|
||||
FeatureMapBuilder().add(ImpressionBloomFilterFeature, updatedBloomFilterSeq).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object InNetworkFeatureHydrator
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("InNetwork")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(InNetworkFeature)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val viewerId = query.getRequiredUserId
|
||||
val followedUserIds = query.features.get.get(SGSFollowedUsersFeature).toSet
|
||||
|
||||
val featureMaps = candidates.map { candidate =>
|
||||
// We use authorId and not sourceAuthorId here so that retweets are defined as in network
|
||||
val isInNetworkOpt = candidate.features.getOrElse(AuthorIdFeature, None).map { authorId =>
|
||||
// Users cannot follow themselves but this is in network by definition
|
||||
val isSelfTweet = authorId == viewerId
|
||||
isSelfTweet || followedUserIds.contains(authorId)
|
||||
}
|
||||
FeatureMapBuilder().add(InNetworkFeature, isInNetworkOpt.getOrElse(true)).build()
|
||||
}
|
||||
Stitch.value(featureMaps)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.user_session_store.ReadRequest
|
||||
import com.twitter.user_session_store.ReadWriteUserSessionStore
|
||||
import com.twitter.user_session_store.UserSessionDataset
|
||||
import com.twitter.user_session_store.UserSessionDataset.UserSessionDataset
|
||||
import com.twitter.util.Time
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class LastNonPollingTimeQueryFeatureHydrator @Inject() (
|
||||
userSessionStore: ReadWriteUserSessionStore)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("LastNonPollingTime")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
FollowingLastNonPollingTimeFeature,
|
||||
LastNonPollingTimeFeature,
|
||||
NonPollingTimesFeature
|
||||
)
|
||||
|
||||
private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
userSessionStore
|
||||
.read(ReadRequest(query.getRequiredUserId, datasets))
|
||||
.map { userSession =>
|
||||
val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)
|
||||
|
||||
val lastNonPollingTime = nonPollingTimestamps
|
||||
.flatMap(_.nonPollingTimestampsMs.headOption)
|
||||
.map(Time.fromMilliseconds)
|
||||
|
||||
val followingLastNonPollingTime = nonPollingTimestamps
|
||||
.flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)
|
||||
.map(Time.fromMilliseconds)
|
||||
|
||||
val nonPollingTimes = nonPollingTimestamps
|
||||
.map(_.nonPollingTimestampsMs)
|
||||
.getOrElse(Seq.empty)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)
|
||||
.add(LastNonPollingTimeFeature, lastNonPollingTime)
|
||||
.add(NonPollingTimesFeature, nonPollingTimes)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,97 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => gt}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.param.HomeGlobalParams.EnableNahFeedbackInfoParam
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.gizmoduck.Gizmoduck
|
||||
import com.twitter.util.Return
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
protected case class ProfileNames(screenName: String, realName: String)
|
||||
|
||||
@Singleton
|
||||
class NamesFeatureHydrator @Inject() (gizmoduck: Gizmoduck)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Names")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ScreenNamesFeature, RealNamesFeature)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean = query.product match {
|
||||
case FollowingProduct => query.params(EnableNahFeedbackInfoParam)
|
||||
case _ => true
|
||||
}
|
||||
|
||||
private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Profile)
|
||||
|
||||
/**
|
||||
* The UI currently only ever displays the first 2 names in social context lines
|
||||
* E.g. "User and 3 others like" or "UserA and UserB liked"
|
||||
*/
|
||||
private val MaxCountUsers = 2
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
|
||||
val candidateUserIdsMap = candidates.map { candidate =>
|
||||
candidate.candidate.id ->
|
||||
(candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(AuthorIdFeature, None) ++
|
||||
candidate.features.getOrElse(SourceUserIdFeature, None)).distinct
|
||||
}.toMap
|
||||
|
||||
val distinctUserIds = candidateUserIdsMap.values.flatten.toSeq.distinct
|
||||
|
||||
Stitch
|
||||
.collectToTry(distinctUserIds.map(userId => gizmoduck.getUserById(userId, queryFields)))
|
||||
.map { allUsers =>
|
||||
val idToProfileNamesMap = allUsers.flatMap {
|
||||
case Return(allUser) =>
|
||||
allUser.profile
|
||||
.map(profile => allUser.id -> ProfileNames(profile.screenName, profile.name))
|
||||
case _ => None
|
||||
}.toMap
|
||||
|
||||
val validUserIds = idToProfileNamesMap.keySet
|
||||
|
||||
candidates.map { candidate =>
|
||||
val combinedMap = candidateUserIdsMap
|
||||
.getOrElse(candidate.candidate.id, Nil)
|
||||
.flatMap {
|
||||
case userId if validUserIds.contains(userId) =>
|
||||
idToProfileNamesMap.get(userId).map(profileNames => userId -> profileNames)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val perCandidateRealNameMap = combinedMap.map { case (k, v) => k -> v.realName }.toMap
|
||||
val perCandidateScreenNameMap = combinedMap.map { case (k, v) => k -> v.screenName }.toMap
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(ScreenNamesFeature, perCandidateScreenNameMap)
|
||||
.add(RealNamesFeature, perCandidateRealNameMap)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,118 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetPreviewIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient
|
||||
import com.twitter.timelinemixer.clients.persistence.TimelineResponseV3
|
||||
import com.twitter.timelines.util.client_info.ClientPlatform
|
||||
import com.twitter.timelineservice.model.TimelineQuery
|
||||
import com.twitter.timelineservice.model.core.TimelineKind
|
||||
import com.twitter.timelineservice.model.rich.EntityIdType
|
||||
import com.twitter.util.Time
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class PersistenceStoreQueryFeatureHydrator @Inject() (
|
||||
timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3],
|
||||
statsReceiver: StatsReceiver)
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("PersistenceStore")
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)
|
||||
private val servedTweetIdsSizeStat = scopedStatsReceiver.stat("ServedTweetIdsSize")
|
||||
|
||||
private val WhoToFollowExcludedUserIdsLimit = 1000
|
||||
private val ServedTweetIdsDuration = 10.minutes
|
||||
private val ServedTweetIdsLimit = 100
|
||||
private val ServedTweetPreviewIdsDuration = 10.hours
|
||||
private val ServedTweetPreviewIdsLimit = 10
|
||||
|
||||
override val features: Set[Feature[_, _]] =
|
||||
Set(
|
||||
ServedTweetIdsFeature,
|
||||
ServedTweetPreviewIdsFeature,
|
||||
PersistenceEntriesFeature,
|
||||
WhoToFollowExcludedUserIdsFeature)
|
||||
|
||||
private val supportedClients = Seq(
|
||||
ClientPlatform.IPhone,
|
||||
ClientPlatform.IPad,
|
||||
ClientPlatform.Mac,
|
||||
ClientPlatform.Android,
|
||||
ClientPlatform.Web,
|
||||
ClientPlatform.RWeb,
|
||||
ClientPlatform.TweetDeckGryphon
|
||||
)
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
val timelineKind = query.product match {
|
||||
case FollowingProduct => TimelineKind.homeLatest
|
||||
case ForYouProduct => TimelineKind.home
|
||||
case other => throw new UnsupportedOperationException(s"Unknown product: $other")
|
||||
}
|
||||
val timelineQuery = TimelineQuery(id = query.getRequiredUserId, kind = timelineKind)
|
||||
|
||||
Stitch.callFuture {
|
||||
timelineResponseBatchesClient
|
||||
.get(query = timelineQuery, clientPlatforms = supportedClients)
|
||||
.map { timelineResponses =>
|
||||
// Note that the WTF entries are not being scoped by ClientPlatform
|
||||
val whoToFollowUserIds = timelineResponses
|
||||
.flatMap { timelineResponse =>
|
||||
timelineResponse.entries
|
||||
.filter(_.entityIdType == EntityIdType.WhoToFollow)
|
||||
.flatMap(_.itemIds.toSeq.flatMap(_.flatMap(_.userId)))
|
||||
}.take(WhoToFollowExcludedUserIdsLimit)
|
||||
|
||||
val clientPlatform = ClientPlatform.fromQueryOptions(
|
||||
clientAppId = query.clientContext.appId,
|
||||
userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))
|
||||
|
||||
val servedTweetIds = timelineResponses
|
||||
.filter(_.clientPlatform == clientPlatform)
|
||||
.filter(_.servedTime >= Time.now - ServedTweetIdsDuration)
|
||||
.sortBy(-_.servedTime.inMilliseconds)
|
||||
.flatMap(
|
||||
_.entries.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetIdsLimit))
|
||||
|
||||
servedTweetIdsSizeStat.add(servedTweetIds.size)
|
||||
|
||||
val servedTweetPreviewIds = timelineResponses
|
||||
.filter(_.clientPlatform == clientPlatform)
|
||||
.filter(_.servedTime >= Time.now - ServedTweetPreviewIdsDuration)
|
||||
.sortBy(-_.servedTime.inMilliseconds)
|
||||
.flatMap(_.entries
|
||||
.filter(_.entityIdType == EntityIdType.TweetPreview)
|
||||
.flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetPreviewIdsLimit))
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(ServedTweetIdsFeature, servedTweetIds)
|
||||
.add(ServedTweetPreviewIdsFeature, servedTweetPreviewIds)
|
||||
.add(PersistenceEntriesFeature, timelineResponses)
|
||||
.add(WhoToFollowExcludedUserIdsFeature, whoToFollowUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7, 50, 60, 60)
|
||||
)
|
||||
}
|
||||
|
|
@ -1,72 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.util.OffloadFuturePools
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.timelineservice.TimelineService
|
||||
import com.twitter.stitch.timelineservice.TimelineService.GetPerspectives
|
||||
import com.twitter.timelineservice.thriftscala.PerspectiveType
|
||||
import com.twitter.timelineservice.thriftscala.PerspectiveType.Favorited
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Filter out unlike edges from liked-by tweets
|
||||
* Useful if the likes come from a cache and because UTEG does not fully remove unlike edges.
|
||||
*/
|
||||
@Singleton
|
||||
class PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService: TimelineService)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("PerspectiveFilteredSocialContext")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(PerspectiveFilteredLikedByUserIdsFeature)
|
||||
|
||||
private val MaxCountUsers = 10
|
||||
private val favoritePerspectiveSet: Set[PerspectiveType] = Set(Favorited)
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {
|
||||
val engagingUserIdtoTweetId = candidates.flatMap { candidate =>
|
||||
candidate.features
|
||||
.getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)
|
||||
.map(favoritedBy => favoritedBy -> candidate.candidate.id)
|
||||
}
|
||||
|
||||
val queries = engagingUserIdtoTweetId.map {
|
||||
case (userId, tweetId) =>
|
||||
GetPerspectives.Query(userId = userId, tweetId = tweetId, types = favoritePerspectiveSet)
|
||||
}
|
||||
|
||||
Stitch.collect(queries.map(timelineService.getPerspective)).map { perspectiveResults =>
|
||||
val validUserIdTweetIds: Set[(Long, Long)] =
|
||||
queries
|
||||
.zip(perspectiveResults)
|
||||
.collect { case (query, perspective) if perspective.favorited => query }
|
||||
.map(query => (query.userId, query.tweetId))
|
||||
.toSet
|
||||
|
||||
candidates.map { candidate =>
|
||||
val perspectiveFilteredFavoritedByUserIds: Seq[Long] = candidate.features
|
||||
.getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)
|
||||
.filter { userId => validUserIdTweetIds.contains((userId, candidate.candidate.id)) }
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(PerspectiveFilteredLikedByUserIdsFeature, perspectiveFilteredFavoritedByUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature
|
||||
import com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphInNetworkScores
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.storehaus.ReadableStore
|
||||
import com.twitter.wtf.candidate.{thriftscala => wtf}
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Named
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class RealGraphInNetworkScoresQueryFeatureHydrator @Inject() (
|
||||
@Named(RealGraphInNetworkScores) store: ReadableStore[Long, Seq[wtf.Candidate]])
|
||||
extends QueryFeatureHydrator[PipelineQuery] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("RealGraphInNetworkScores")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(RealGraphInNetworkScoresFeature)
|
||||
|
||||
private val RealGraphCandidateCount = 1000
|
||||
|
||||
override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {
|
||||
Stitch.callFuture(store.get(query.getRequiredUserId)).map { realGraphFollowedUsers =>
|
||||
val realGraphScoresFeatures = realGraphFollowedUsers
|
||||
.getOrElse(Seq.empty)
|
||||
.sortBy(-_.score)
|
||||
.map(candidate => candidate.userId -> scaleScore(candidate.score))
|
||||
.take(RealGraphCandidateCount)
|
||||
.toMap
|
||||
|
||||
FeatureMapBuilder().add(RealGraphInNetworkScoresFeature, realGraphScoresFeatures).build()
|
||||
}
|
||||
}
|
||||
|
||||
// Rescale Real Graph v2 scores from [0,1] to the v1 scores distribution [1,2.97]
|
||||
private def scaleScore(score: Double): Double =
|
||||
if (score >= 0.0 && score <= 1.0) score * 1.97 + 1.0 else score
|
||||
}
|
||||
|
|
@ -1,121 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.tracing.Annotation.BinaryAnnotation
|
||||
import com.twitter.finagle.tracing.ForwardAnnotation
|
||||
import com.twitter.home_mixer.model.HomeFeatures._
|
||||
import com.twitter.home_mixer.model.request.DeviceContext.RequestContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.joinkey.context.RequestJoinKeyContext
|
||||
import com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor
|
||||
import com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor
|
||||
import com.twitter.product_mixer.core.pipeline.HasPipelineCursor
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest
|
||||
import com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure
|
||||
import com.twitter.search.common.util.lang.ThriftLanguageUtil
|
||||
import com.twitter.snowflake.id.SnowflakeId
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.dowFromTimestamp
|
||||
import com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.hourFromTimestamp
|
||||
import java.util.UUID
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class RequestQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor] with HasDeviceContext] @Inject() (
|
||||
) extends QueryFeatureHydrator[Query] {
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AccountAgeFeature,
|
||||
ClientIdFeature,
|
||||
DeviceLanguageFeature,
|
||||
GetInitialFeature,
|
||||
GetMiddleFeature,
|
||||
GetNewerFeature,
|
||||
GetOlderFeature,
|
||||
GuestIdFeature,
|
||||
HasDarkRequestFeature,
|
||||
IsForegroundRequestFeature,
|
||||
IsLaunchRequestFeature,
|
||||
PollingFeature,
|
||||
PullToRefreshFeature,
|
||||
RequestJoinIdFeature,
|
||||
ServedRequestIdFeature,
|
||||
TimestampFeature,
|
||||
TimestampGMTDowFeature,
|
||||
TimestampGMTHourFeature,
|
||||
ViewerIdFeature
|
||||
)
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Request")
|
||||
|
||||
private val DarkRequestAnnotation = "clnt/has_dark_request"
|
||||
|
||||
// Convert Language code to ISO 639-3 format
|
||||
private def getLanguageISOFormatByCode(languageCode: String): String =
|
||||
ThriftLanguageUtil.getLanguageCodeOf(ThriftLanguageUtil.getThriftLanguageOf(languageCode))
|
||||
|
||||
private def getRequestJoinId(servedRequestId: Long): Option[Long] =
|
||||
Some(RequestJoinKeyContext.current.flatMap(_.requestJoinId).getOrElse(servedRequestId))
|
||||
|
||||
private def hasDarkRequest: Option[Boolean] = ForwardAnnotation.current
|
||||
.getOrElse(Seq[BinaryAnnotation]())
|
||||
.find(_.key == DarkRequestAnnotation)
|
||||
.map(_.value.asInstanceOf[Boolean])
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
val requestContext = query.deviceContext.flatMap(_.requestContextValue)
|
||||
val servedRequestId = UUID.randomUUID.getMostSignificantBits
|
||||
val timestamp = query.queryTime.inMilliseconds
|
||||
|
||||
val featureMap = FeatureMapBuilder()
|
||||
.add(AccountAgeFeature, query.getOptionalUserId.flatMap(SnowflakeId.timeFromIdOpt))
|
||||
.add(ClientIdFeature, query.clientContext.appId)
|
||||
.add(DeviceLanguageFeature, query.getLanguageCode.map(getLanguageISOFormatByCode))
|
||||
.add(
|
||||
GetInitialFeature,
|
||||
query.pipelineCursor.forall(cursor => cursor.id.isEmpty && cursor.gapBoundaryId.isEmpty))
|
||||
.add(
|
||||
GetMiddleFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isDefined &&
|
||||
cursor.cursorType.contains(GapCursor)))
|
||||
.add(
|
||||
GetNewerFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
|
||||
cursor.cursorType.contains(TopCursor)))
|
||||
.add(
|
||||
GetOlderFeature,
|
||||
query.pipelineCursor.exists(cursor =>
|
||||
cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&
|
||||
cursor.cursorType.contains(BottomCursor)))
|
||||
.add(GuestIdFeature, query.clientContext.guestId)
|
||||
.add(IsForegroundRequestFeature, requestContext.contains(RequestContext.Foreground))
|
||||
.add(IsLaunchRequestFeature, requestContext.contains(RequestContext.Launch))
|
||||
.add(PollingFeature, query.deviceContext.exists(_.isPolling.contains(true)))
|
||||
.add(PullToRefreshFeature, requestContext.contains(RequestContext.PullToRefresh))
|
||||
.add(ServedRequestIdFeature, Some(servedRequestId))
|
||||
.add(RequestJoinIdFeature, getRequestJoinId(servedRequestId))
|
||||
.add(TimestampFeature, timestamp)
|
||||
.add(TimestampGMTDowFeature, dowFromTimestamp(timestamp))
|
||||
.add(TimestampGMTHourFeature, hourFromTimestamp(timestamp))
|
||||
.add(HasDarkRequestFeature, hasDarkRequest)
|
||||
.add(
|
||||
ViewerIdFeature,
|
||||
query.getOptionalUserId
|
||||
.orElse(query.getGuestId).getOrElse(
|
||||
throw PipelineFailure(BadRequest, "Missing viewer id")))
|
||||
.build()
|
||||
|
||||
Stitch.value(featureMap)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,105 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.product_mixer.core.util.OffloadFuturePools
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.SocialGraph
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* This hydrator takes liked-by and followed-by user ids and checks via SGS that the viewer is
|
||||
* following the engager, that the viewer is not blocking the engager, that the engager is not
|
||||
* blocking the viewer, and that the viewer has not muted the engager.
|
||||
*/
|
||||
@Singleton
|
||||
class SGSValidSocialContextFeatureHydrator @Inject() (
|
||||
socialGraph: SocialGraph)
|
||||
extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier =
|
||||
FeatureHydratorIdentifier("SGSValidSocialContext")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
SGSValidFollowedByUserIdsFeature,
|
||||
SGSValidLikedByUserIdsFeature
|
||||
)
|
||||
|
||||
private val MaxCountUsers = 10
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {
|
||||
val allSocialContextUserIds =
|
||||
candidates.flatMap { candidate =>
|
||||
candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++
|
||||
candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
}.distinct
|
||||
|
||||
getValidUserIds(query.getRequiredUserId, allSocialContextUserIds).map { validUserIds =>
|
||||
candidates.map { candidate =>
|
||||
val sgsFilteredLikedByUserIds =
|
||||
candidate.features
|
||||
.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
.filter(validUserIds.contains)
|
||||
|
||||
val sgsFilteredFollowedByUserIds =
|
||||
candidate.features
|
||||
.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)
|
||||
.filter(validUserIds.contains)
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(SGSValidFollowedByUserIdsFeature, sgsFilteredFollowedByUserIds)
|
||||
.add(SGSValidLikedByUserIdsFeature, sgsFilteredLikedByUserIds)
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private def getValidUserIds(
|
||||
viewerId: Long,
|
||||
socialProofUserIds: Seq[Long]
|
||||
): Stitch[Seq[Long]] = {
|
||||
if (socialProofUserIds.nonEmpty) {
|
||||
val request = sg.IdsRequest(
|
||||
relationships = Seq(
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Following,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = true),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Blocking,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.BlockedBy,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false),
|
||||
sg.SrcRelationship(
|
||||
viewerId,
|
||||
sg.RelationshipType.Muting,
|
||||
targets = Some(socialProofUserIds),
|
||||
hasRelationship = false)
|
||||
),
|
||||
pageRequest = Some(sg.PageRequest(selectAll = Some(true)))
|
||||
)
|
||||
socialGraph.ids(request).map(_.ids)
|
||||
} else Stitch.Nil
|
||||
}
|
||||
}
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature
|
||||
import com.twitter.home_mixer.model.request.HasSeenTweetIds
|
||||
import com.twitter.home_mixer.service.HomeMixerAlertConfig
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.impression.{thriftscala => t}
|
||||
import com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient
|
||||
import com.twitter.util.Duration
|
||||
import com.twitter.util.Time
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
case class TweetImpressionsQueryFeatureHydrator[
|
||||
Query <: PipelineQuery with HasSeenTweetIds] @Inject() (
|
||||
manhattanTweetImpressionStoreClient: ManhattanTweetImpressionStoreClient)
|
||||
extends QueryFeatureHydrator[Query] {
|
||||
|
||||
private val TweetImpressionTTL = 2.days
|
||||
private val TweetImpressionCap = 5000
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("TweetImpressions")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(TweetImpressionsFeature)
|
||||
|
||||
override def hydrate(query: Query): Stitch[FeatureMap] = {
|
||||
manhattanTweetImpressionStoreClient.get(query.getRequiredUserId).map { entriesOpt =>
|
||||
val entries = entriesOpt.map(_.entries).toSeq.flatten
|
||||
val updatedImpressions =
|
||||
if (query.seenTweetIds.forall(_.isEmpty)) entries
|
||||
else updateTweetImpressions(entries, query.seenTweetIds.get)
|
||||
|
||||
FeatureMapBuilder().add(TweetImpressionsFeature, updatedImpressions).build()
|
||||
}
|
||||
}
|
||||
|
||||
override val alerts = Seq(
|
||||
HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)
|
||||
)
|
||||
|
||||
/**
|
||||
* 1) Check timestamps and remove expired tweets based on [[TweetImpressionTTL]]
|
||||
* 2) Filter duplicates between current tweets and those in the impression store (remove older ones)
|
||||
* 3) Prepend new (Timestamp, Seq[TweetIds]) to the tweets from the impression store
|
||||
* 4) Truncate older tweets if sum of all tweets across timestamps >= [[TweetImpressionCap]],
|
||||
*/
|
||||
private[feature_hydrator] def updateTweetImpressions(
|
||||
tweetImpressionsFromStore: Seq[t.TweetImpressionsEntry],
|
||||
seenIdsFromClient: Seq[Long],
|
||||
currentTime: Long = Time.now.inMilliseconds,
|
||||
tweetImpressionTTL: Duration = TweetImpressionTTL,
|
||||
tweetImpressionCap: Int = TweetImpressionCap,
|
||||
): Seq[t.TweetImpressionsEntry] = {
|
||||
val seenIdsFromClientSet = seenIdsFromClient.toSet
|
||||
val dedupedTweetImpressionsFromStore: Seq[t.TweetImpressionsEntry] = tweetImpressionsFromStore
|
||||
.collect {
|
||||
case t.TweetImpressionsEntry(ts, tweetIds)
|
||||
if Time.fromMilliseconds(ts).untilNow < tweetImpressionTTL =>
|
||||
t.TweetImpressionsEntry(ts, tweetIds.filterNot(seenIdsFromClientSet.contains))
|
||||
}.filter { _.tweetIds.nonEmpty }
|
||||
|
||||
val mergedTweetImpressionsEntries =
|
||||
t.TweetImpressionsEntry(currentTime, seenIdsFromClient) +: dedupedTweetImpressionsFromStore
|
||||
val initialTweetImpressionsWithCap = (Seq.empty[t.TweetImpressionsEntry], tweetImpressionCap)
|
||||
|
||||
val (truncatedTweetImpressionsEntries: Seq[t.TweetImpressionsEntry], _) =
|
||||
mergedTweetImpressionsEntries
|
||||
.foldLeft(initialTweetImpressionsWithCap) {
|
||||
case (
|
||||
(tweetImpressions: Seq[t.TweetImpressionsEntry], remainingCap),
|
||||
t.TweetImpressionsEntry(ts, tweetIds)) if remainingCap > 0 =>
|
||||
(
|
||||
t.TweetImpressionsEntry(ts, tweetIds.take(remainingCap)) +: tweetImpressions,
|
||||
remainingCap - tweetIds.size)
|
||||
case (tweetImpressionsWithCap, _) => tweetImpressionsWithCap
|
||||
}
|
||||
truncatedTweetImpressionsEntries.reverse
|
||||
}
|
||||
}
|
||||
|
|
@ -1,179 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.feature_hydrator
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsNsfwFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature
|
||||
import com.twitter.home_mixer.model.request.FollowingProduct
|
||||
import com.twitter.home_mixer.model.request.ForYouProduct
|
||||
import com.twitter.home_mixer.model.request.ListTweetsProduct
|
||||
import com.twitter.home_mixer.model.request.ScoredTweetsProduct
|
||||
import com.twitter.home_mixer.model.request.SubscribedProduct
|
||||
import com.twitter.home_mixer.util.tweetypie.RequestFields
|
||||
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw
|
||||
import com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.spam.rtf.{thriftscala => rtf}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}
|
||||
import com.twitter.tweetypie.{thriftscala => tp}
|
||||
import com.twitter.util.logging.Logging
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
@Singleton
|
||||
class TweetypieFeatureHydrator @Inject() (
|
||||
tweetypieStitchClient: TweetypieStitchClient,
|
||||
statsReceiver: StatsReceiver)
|
||||
extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]
|
||||
with Logging {
|
||||
|
||||
override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier("Tweetypie")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(
|
||||
AuthorIdFeature,
|
||||
ExclusiveConversationAuthorIdFeature,
|
||||
InReplyToTweetIdFeature,
|
||||
IsHydratedFeature,
|
||||
IsNsfw,
|
||||
IsNsfwFeature,
|
||||
IsRetweetFeature,
|
||||
QuotedTweetDroppedFeature,
|
||||
QuotedTweetIdFeature,
|
||||
QuotedUserIdFeature,
|
||||
SourceTweetIdFeature,
|
||||
SourceUserIdFeature,
|
||||
TweetTextFeature,
|
||||
TweetLanguageFeature,
|
||||
VisibilityReason
|
||||
)
|
||||
|
||||
private val DefaultFeatureMap = FeatureMapBuilder()
|
||||
.add(IsHydratedFeature, false)
|
||||
.add(IsNsfw, None)
|
||||
.add(IsNsfwFeature, false)
|
||||
.add(QuotedTweetDroppedFeature, false)
|
||||
.add(TweetTextFeature, None)
|
||||
.add(VisibilityReason, None)
|
||||
.build()
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidate: TweetCandidate,
|
||||
existingFeatures: FeatureMap
|
||||
): Stitch[FeatureMap] = {
|
||||
val safetyLevel = query.product match {
|
||||
case FollowingProduct => rtf.SafetyLevel.TimelineHomeLatest
|
||||
case ForYouProduct =>
|
||||
val inNetwork = existingFeatures.getOrElse(InNetworkFeature, true)
|
||||
if (inNetwork) rtf.SafetyLevel.TimelineHome else rtf.SafetyLevel.TimelineHomeRecommendations
|
||||
case ScoredTweetsProduct => rtf.SafetyLevel.TimelineHome
|
||||
case ListTweetsProduct => rtf.SafetyLevel.TimelineLists
|
||||
case SubscribedProduct => rtf.SafetyLevel.TimelineHomeSubscribed
|
||||
case unknown => throw new UnsupportedOperationException(s"Unknown product: $unknown")
|
||||
}
|
||||
|
||||
val tweetFieldsOptions = tp.GetTweetFieldsOptions(
|
||||
tweetIncludes = RequestFields.TweetTPHydrationFields,
|
||||
includeRetweetedTweet = true,
|
||||
includeQuotedTweet = true,
|
||||
visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,
|
||||
safetyLevel = Some(safetyLevel),
|
||||
forUserId = query.getOptionalUserId
|
||||
)
|
||||
|
||||
val exclusiveAuthorIdOpt =
|
||||
existingFeatures.getOrElse(ExclusiveConversationAuthorIdFeature, None)
|
||||
|
||||
tweetypieStitchClient.getTweetFields(tweetId = candidate.id, options = tweetFieldsOptions).map {
|
||||
case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), quoteOpt, _) =>
|
||||
val coreData = found.tweet.coreData
|
||||
val isNsfwAdmin = coreData.exists(_.nsfwAdmin)
|
||||
val isNsfwUser = coreData.exists(_.nsfwUser)
|
||||
|
||||
val quotedTweetDropped = quoteOpt.exists {
|
||||
case _: tp.TweetFieldsResultState.Filtered => true
|
||||
case _: tp.TweetFieldsResultState.NotFound => true
|
||||
case _ => false
|
||||
}
|
||||
val quotedTweetIsNsfw = quoteOpt.exists {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)
|
||||
case _ => false
|
||||
}
|
||||
|
||||
val sourceTweetIsNsfw =
|
||||
found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))
|
||||
|
||||
val tweetText = coreData.map(_.text)
|
||||
val tweetLanguage = found.tweet.language.map(_.language)
|
||||
|
||||
val tweetAuthorId = coreData.map(_.userId)
|
||||
val inReplyToTweetId = coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))
|
||||
val retweetedTweetId = found.retweetedTweet.map(_.id)
|
||||
val quotedTweetId = quoteOpt.flatMap {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
Some(quoteTweet.found.tweet.id)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val retweetedTweetUserId = found.retweetedTweet.flatMap(_.coreData).map(_.userId)
|
||||
val quotedTweetUserId = quoteOpt.flatMap {
|
||||
case quoteTweet: tp.TweetFieldsResultState.Found =>
|
||||
quoteTweet.found.tweet.coreData.map(_.userId)
|
||||
case _ => None
|
||||
}
|
||||
|
||||
val isNsfw = isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw
|
||||
|
||||
FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, tweetAuthorId)
|
||||
.add(ExclusiveConversationAuthorIdFeature, exclusiveAuthorIdOpt)
|
||||
.add(InReplyToTweetIdFeature, inReplyToTweetId)
|
||||
.add(IsHydratedFeature, true)
|
||||
.add(IsNsfw, Some(isNsfw))
|
||||
.add(IsNsfwFeature, isNsfw)
|
||||
.add(IsRetweetFeature, retweetedTweetId.isDefined)
|
||||
.add(QuotedTweetDroppedFeature, quotedTweetDropped)
|
||||
.add(QuotedTweetIdFeature, quotedTweetId)
|
||||
.add(QuotedUserIdFeature, quotedTweetUserId)
|
||||
.add(SourceTweetIdFeature, retweetedTweetId)
|
||||
.add(SourceUserIdFeature, retweetedTweetUserId)
|
||||
.add(TweetLanguageFeature, tweetLanguage)
|
||||
.add(TweetTextFeature, tweetText)
|
||||
.add(VisibilityReason, found.suppressReason)
|
||||
.build()
|
||||
|
||||
// If no tweet result found, return default and pre-existing features
|
||||
case _ =>
|
||||
DefaultFeatureMap ++ FeatureMapBuilder()
|
||||
.add(AuthorIdFeature, existingFeatures.getOrElse(AuthorIdFeature, None))
|
||||
.add(ExclusiveConversationAuthorIdFeature, exclusiveAuthorIdOpt)
|
||||
.add(InReplyToTweetIdFeature, existingFeatures.getOrElse(InReplyToTweetIdFeature, None))
|
||||
.add(IsRetweetFeature, existingFeatures.getOrElse(IsRetweetFeature, false))
|
||||
.add(QuotedTweetIdFeature, existingFeatures.getOrElse(QuotedTweetIdFeature, None))
|
||||
.add(QuotedUserIdFeature, existingFeatures.getOrElse(QuotedUserIdFeature, None))
|
||||
.add(SourceTweetIdFeature, existingFeatures.getOrElse(SourceTweetIdFeature, None))
|
||||
.add(SourceUserIdFeature, existingFeatures.getOrElse(SourceUserIdFeature, None))
|
||||
.add(TweetLanguageFeature, existingFeatures.getOrElse(TweetLanguageFeature, None))
|
||||
.build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter",
|
||||
"src/thrift/com/twitter/spam/rtf:safety-result-scala",
|
||||
"src/thrift/com/twitter/timelines/impression:thrift-scala",
|
||||
"src/thrift/com/twitter/tweetypie:service-scala",
|
||||
"src/thrift/com/twitter/tweetypie:tweet-scala",
|
||||
"stitch/stitch-core",
|
||||
"stitch/stitch-socialgraph",
|
||||
"stitch/stitch-tweetypie",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
|
||||
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
|
||||
"util/util-slf4j-api/src/main/scala",
|
||||
],
|
||||
exports = [
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.configapi.FSBoundedParam
|
||||
|
||||
case class DropMaxCandidatesFilter[Candidate <: UniversalNoun[Any]](
|
||||
maxCandidatesParam: FSBoundedParam[Int])
|
||||
extends Filter[PipelineQuery, Candidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("DropMaxCandidates")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[Candidate]]
|
||||
): Stitch[FilterResult[Candidate]] = {
|
||||
val maxCandidates = query.params(maxCandidatesParam)
|
||||
val (kept, removed) = candidates.map(_.candidate).splitAt(maxCandidates)
|
||||
|
||||
Stitch.value(FilterResult(kept, removed))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,89 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.common.thriftscala.FeedbackEntity
|
||||
import com.twitter.timelineservice.model.FeedbackEntry
|
||||
import com.twitter.timelineservice.{thriftscala => tls}
|
||||
|
||||
object FeedbackFatigueFilter
|
||||
extends Filter[PipelineQuery, TweetCandidate]
|
||||
with Filter.Conditionally[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("FeedbackFatigue")
|
||||
|
||||
override def onlyIf(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Boolean =
|
||||
query.features.exists(_.getOrElse(FeedbackHistoryFeature, Seq.empty).nonEmpty)
|
||||
|
||||
private val DurationForFiltering = 14.days
|
||||
|
||||
override def apply(
|
||||
query: pipeline.PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
val feedbackEntriesByEngagementType =
|
||||
query.features
|
||||
.getOrElse(FeatureMap.empty).getOrElse(FeedbackHistoryFeature, Seq.empty)
|
||||
.filter { entry =>
|
||||
val timeSinceFeedback = query.queryTime.minus(entry.timestamp)
|
||||
timeSinceFeedback < DurationForFiltering &&
|
||||
entry.feedbackType == tls.FeedbackType.SeeFewer
|
||||
}.groupBy(_.engagementType)
|
||||
|
||||
val authorsToFilter =
|
||||
getUserIds(
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Tweet, Seq.empty))
|
||||
val likersToFilter =
|
||||
getUserIds(
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Like, Seq.empty))
|
||||
val followersToFilter =
|
||||
getUserIds(
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Follow, Seq.empty))
|
||||
val retweetersToFilter =
|
||||
getUserIds(
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Retweet, Seq.empty))
|
||||
|
||||
val (removed, kept) = candidates.partition { candidate =>
|
||||
val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)
|
||||
val authorId = candidate.features.getOrElse(AuthorIdFeature, None)
|
||||
|
||||
val likers = candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)
|
||||
val eligibleLikers = likers.filterNot(likersToFilter.contains)
|
||||
|
||||
val followers = candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)
|
||||
val eligibleFollowers = followers.filterNot(followersToFilter.contains)
|
||||
|
||||
originalAuthorId.exists(authorsToFilter.contains) ||
|
||||
(likers.nonEmpty && eligibleLikers.isEmpty) ||
|
||||
(followers.nonEmpty && eligibleFollowers.isEmpty && likers.isEmpty) ||
|
||||
(candidate.features.getOrElse(IsRetweetFeature, false) &&
|
||||
authorId.exists(retweetersToFilter.contains))
|
||||
}
|
||||
|
||||
Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))
|
||||
}
|
||||
|
||||
private def getUserIds(
|
||||
feedbackEntries: Seq[FeedbackEntry],
|
||||
): Set[Long] =
|
||||
feedbackEntries.collect {
|
||||
case FeedbackEntry(_, _, FeedbackEntity.UserId(userId), _, _) => userId
|
||||
}.toSet
|
||||
}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* Exclude conversation modules where Tweets have been dropped by other filters
|
||||
*
|
||||
* Largest conversation modules have 3 Tweets, so if all 3 are present, module is valid.
|
||||
* For 2 Tweet modules, check if the head is the root (not a reply) and the last item
|
||||
* is actually replying to the root directly with no missing intermediate tweets
|
||||
*/
|
||||
object InvalidConversationModuleFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("InvalidConversationModule")
|
||||
|
||||
val ValidThreeTweetModuleSize = 3
|
||||
val ValidTwoTweetModuleSize = 2
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
val allowedTweetIds = candidates
|
||||
.groupBy(_.features.getOrElse(ConversationModuleFocalTweetIdFeature, None))
|
||||
.map { case (id, candidates) => (id, candidates.sortBy(_.candidate.id)) }
|
||||
.filter {
|
||||
case (Some(_), conversation) if conversation.size == ValidThreeTweetModuleSize => true
|
||||
case (Some(focalId), conversation) if conversation.size == ValidTwoTweetModuleSize =>
|
||||
conversation.head.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty &&
|
||||
conversation.last.candidate.id == focalId &&
|
||||
conversation.last.features
|
||||
.getOrElse(InReplyToTweetIdFeature, None)
|
||||
.contains(conversation.head.candidate.id)
|
||||
case (None, _) => true
|
||||
case _ => false
|
||||
}.values.flatten.toSeq.map(_.candidate.id).toSet
|
||||
|
||||
val (kept, removed) =
|
||||
candidates.map(_.candidate).partition(candidate => allowedTweetIds.contains(candidate.id))
|
||||
Stitch.value(FilterResult(kept = kept, removed = removed))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.finagle.stats.StatsReceiver
|
||||
import com.twitter.finagle.tracing.Trace
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.socialgraph.{thriftscala => sg}
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.stitch.socialgraph.SocialGraph
|
||||
import com.twitter.util.logging.Logging
|
||||
|
||||
import javax.inject.Inject
|
||||
import javax.inject.Singleton
|
||||
|
||||
/**
|
||||
* Exclude invalid subscription tweets - cases where the viewer is not subscribed to the author
|
||||
*
|
||||
* If SGS hydration fails, `SGSInvalidSubscriptionTweetFeature` will be set to None for
|
||||
* subscription tweets, so we explicitly filter those tweets out.
|
||||
*/
|
||||
@Singleton
|
||||
case class InvalidSubscriptionTweetFilter @Inject() (
|
||||
socialGraphClient: SocialGraph,
|
||||
statsReceiver: StatsReceiver)
|
||||
extends Filter[PipelineQuery, TweetCandidate]
|
||||
with Logging {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("InvalidSubscriptionTweet")
|
||||
|
||||
private val scopedStatsReceiver = statsReceiver.scope(identifier.toString)
|
||||
private val validCounter = scopedStatsReceiver.counter("validExclusiveTweet")
|
||||
private val invalidCounter = scopedStatsReceiver.counter("invalidExclusiveTweet")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = Stitch
|
||||
.traverse(candidates) { candidate =>
|
||||
val exclusiveAuthorId =
|
||||
candidate.features.getOrElse(ExclusiveConversationAuthorIdFeature, None)
|
||||
|
||||
if (exclusiveAuthorId.isDefined) {
|
||||
val request = sg.ExistsRequest(
|
||||
source = query.getRequiredUserId,
|
||||
target = exclusiveAuthorId.get,
|
||||
relationships =
|
||||
Seq(sg.Relationship(sg.RelationshipType.TierOneSuperFollowing, hasRelationship = true)),
|
||||
)
|
||||
socialGraphClient.exists(request).map(_.exists).map { valid =>
|
||||
if (!valid) invalidCounter.incr() else validCounter.incr()
|
||||
valid
|
||||
}
|
||||
} else Stitch.value(true)
|
||||
}.map { validResults =>
|
||||
val (kept, removed) = candidates
|
||||
.map(_.candidate)
|
||||
.zip(validResults)
|
||||
.partition { case (candidate, valid) => valid }
|
||||
|
||||
val keptCandidates = kept.map { case (candidate, _) => candidate }
|
||||
val removedCandidates = removed.map { case (candidate, _) => candidate }
|
||||
|
||||
FilterResult(kept = keptCandidates, removed = removedCandidates)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.product_mixer.core.functional_component.common.alert.Alert
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.UniversalNoun
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
trait FilterPredicate[-Query <: PipelineQuery] {
|
||||
def apply(query: Query): Boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* A [[Filter]] with [[Conditionally]] based on a [[FilterPredicate]]
|
||||
*
|
||||
* @param predicate the predicate to turn this filter on and off
|
||||
* @param filter the underlying filter to run when `predicate` is true
|
||||
* @tparam Query The domain model for the query or request
|
||||
* @tparam Candidate The type of the candidates
|
||||
*/
|
||||
case class PredicateGatedFilter[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](
|
||||
predicate: FilterPredicate[Query],
|
||||
filter: Filter[Query, Candidate])
|
||||
extends Filter[Query, Candidate]
|
||||
with Filter.Conditionally[Query, Candidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier(
|
||||
PredicateGatedFilter.IdentifierPrefix + filter.identifier.name)
|
||||
|
||||
override val alerts: Seq[Alert] = filter.alerts
|
||||
|
||||
override def onlyIf(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): Boolean =
|
||||
Conditionally.and(Filter.Input(query, candidates), filter, predicate(query))
|
||||
|
||||
override def apply(
|
||||
query: Query,
|
||||
candidates: Seq[CandidateWithFeatures[Candidate]]
|
||||
): Stitch[FilterResult[Candidate]] = filter.apply(query, candidates)
|
||||
}
|
||||
|
||||
object PredicateGatedFilter {
|
||||
val IdentifierPrefix = "PredicateGated"
|
||||
}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.home_mixer.util.TweetImpressionsHelper
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* Filter out users' previously seen tweets from 2 sources:
|
||||
* 1. Heron Topology Impression Store in Memcache;
|
||||
* 2. Manhattan Impression Store;
|
||||
*/
|
||||
object PreviouslySeenTweetsFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("PreviouslySeenTweets")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
|
||||
val seenTweetIds =
|
||||
query.features.map(TweetImpressionsHelper.tweetImpressions).getOrElse(Set.empty)
|
||||
|
||||
val (removed, kept) = candidates.partition { candidate =>
|
||||
val tweetIdAndSourceId = CandidatesUtil.getTweetIdAndSourceId(candidate)
|
||||
tweetIdAndSourceId.exists(seenTweetIds.contains)
|
||||
}
|
||||
|
||||
Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsAncestorCandidateFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils
|
||||
import com.twitter.timelines.util.client_info.ClientPlatform
|
||||
|
||||
object PreviouslyServedAncestorsFilter
|
||||
extends Filter[PipelineQuery, TweetCandidate]
|
||||
with TimelinePersistenceUtils {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("PreviouslyServedAncestors")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
val clientPlatform = ClientPlatform.fromQueryOptions(
|
||||
clientAppId = query.clientContext.appId,
|
||||
userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))
|
||||
val entries =
|
||||
query.features.map(_.getOrElse(PersistenceEntriesFeature, Seq.empty)).toSeq.flatten
|
||||
val tweetIds = applicableResponses(clientPlatform, entries)
|
||||
.flatMap(_.entries.flatMap(_.tweetIds(includeSourceTweets = true))).toSet
|
||||
val ancestorIds =
|
||||
candidates
|
||||
.filter(_.features.getOrElse(IsAncestorCandidateFeature, false)).map(_.candidate.id).toSet
|
||||
|
||||
val (removed, kept) =
|
||||
candidates
|
||||
.map(_.candidate).partition(candidate =>
|
||||
tweetIds.contains(candidate.id) && ancestorIds.contains(candidate.id))
|
||||
|
||||
Stitch.value(FilterResult(kept = kept, removed = removed))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetPreviewIdsFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object PreviouslyServedTweetPreviewsFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("PreviouslyServedTweetPreviews")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
|
||||
val servedTweetPreviewIds =
|
||||
query.features.map(_.getOrElse(ServedTweetPreviewIdsFeature, Seq.empty)).toSeq.flatten.toSet
|
||||
|
||||
val (removed, kept) = candidates.partition { candidate =>
|
||||
servedTweetPreviewIds.contains(candidate.candidate.id)
|
||||
}
|
||||
|
||||
Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.GetOlderFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object PreviouslyServedTweetsFilter
|
||||
extends Filter[PipelineQuery, TweetCandidate]
|
||||
with Filter.Conditionally[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("PreviouslyServedTweets")
|
||||
|
||||
override def onlyIf(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Boolean = {
|
||||
query.features.exists(_.getOrElse(GetOlderFeature, false))
|
||||
}
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
|
||||
val servedTweetIds =
|
||||
query.features.map(_.getOrElse(ServedTweetIdsFeature, Seq.empty)).toSeq.flatten.toSet
|
||||
|
||||
val (removed, kept) = candidates.partition { candidate =>
|
||||
val tweetIdAndSourceId = CandidatesUtil.getTweetIdAndSourceId(candidate)
|
||||
tweetIdAndSourceId.exists(servedTweetIds.contains)
|
||||
}
|
||||
|
||||
Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object RejectTweetFromViewerFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("RejectTweetFromViewer")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
val (removed, kept) = candidates.partition(candidate =>
|
||||
CandidatesUtil.isAuthoredByViewer(query, candidate.features))
|
||||
Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object ReplyFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("Reply")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
|
||||
val (kept, removed) = candidates
|
||||
.partition { candidate =>
|
||||
candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty
|
||||
}
|
||||
|
||||
val filterResult = FilterResult(
|
||||
kept = kept.map(_.candidate),
|
||||
removed = removed.map(_.candidate)
|
||||
)
|
||||
|
||||
Stitch.value(filterResult)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import scala.collection.mutable
|
||||
|
||||
object RetweetDeduplicationFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("RetweetDeduplication")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
// If there are 2 retweets of the same native tweet, we will choose the first one
|
||||
// The tweets are returned in descending score order, so we will choose the higher scored tweet
|
||||
val dedupedTweetIdsSet =
|
||||
candidates.partition(_.features.getOrElse(IsRetweetFeature, false)) match {
|
||||
case (retweets, nativeTweets) =>
|
||||
val nativeTweetIds = nativeTweets.map(_.candidate.id)
|
||||
val seenTweetIds = mutable.Set[Long]() ++ nativeTweetIds
|
||||
val dedupedRetweets = retweets.filter { retweet =>
|
||||
val tweetIdAndSourceId = CandidatesUtil.getTweetIdAndSourceId(retweet)
|
||||
val retweetIsUnique = tweetIdAndSourceId.forall(!seenTweetIds.contains(_))
|
||||
if (retweetIsUnique) {
|
||||
seenTweetIds ++= tweetIdAndSourceId
|
||||
}
|
||||
retweetIsUnique
|
||||
}
|
||||
(nativeTweets ++ dedupedRetweets).map(_.candidate.id).toSet
|
||||
}
|
||||
|
||||
val (kept, removed) =
|
||||
candidates
|
||||
.map(_.candidate).partition(candidate => dedupedTweetIdsSet.contains(candidate.id))
|
||||
Stitch.value(FilterResult(kept = kept, removed = removed))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.filter
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.functional_component.filter.Filter
|
||||
import com.twitter.product_mixer.core.functional_component.filter.FilterResult
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object RetweetFilter extends Filter[PipelineQuery, TweetCandidate] {
|
||||
override val identifier: FilterIdentifier = FilterIdentifier("Retweet")
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[FilterResult[TweetCandidate]] = {
|
||||
|
||||
val (kept, removed) = candidates
|
||||
.partition { candidate =>
|
||||
!candidate.features.getOrElse(IsRetweetFeature, false)
|
||||
}
|
||||
|
||||
val filterResult = FilterResult(
|
||||
kept = kept.map(_.candidate),
|
||||
removed = removed.map(_.candidate)
|
||||
)
|
||||
|
||||
Stitch.value(filterResult)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"home-mixer/thrift/src/main/thrift:thrift-scala",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate",
|
||||
"src/thrift/com/twitter/gizmoduck:thrift-scala",
|
||||
"stitch/stitch-socialgraph",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
|
||||
"timelineservice/common/src/main/scala/com/twitter/timelineservice/model",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.manhattan.DismissInfo
|
||||
import com.twitter.timelineservice.suggests.thriftscala.SuggestType
|
||||
import com.twitter.util.Duration
|
||||
|
||||
object DismissFatigueGate {
|
||||
// how long a dismiss action from user needs to be respected
|
||||
val DefaultBaseDismissDuration = 7.days
|
||||
val MaximumDismissalCountMultiplier = 4
|
||||
}
|
||||
|
||||
case class DismissFatigueGate(
|
||||
suggestType: SuggestType,
|
||||
dismissInfoFeature: Feature[PipelineQuery, Map[SuggestType, Option[DismissInfo]]],
|
||||
baseDismissDuration: Duration = DismissFatigueGate.DefaultBaseDismissDuration,
|
||||
) extends Gate[PipelineQuery] {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("DismissFatigue")
|
||||
|
||||
override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {
|
||||
val dismissInfoMap = query.features.map(
|
||||
_.getOrElse(dismissInfoFeature, Map.empty[SuggestType, Option[DismissInfo]]))
|
||||
|
||||
val isVisible = dismissInfoMap
|
||||
.flatMap(_.get(suggestType))
|
||||
.flatMap(_.map { info =>
|
||||
val currentDismissalDuration = query.queryTime.since(info.lastDismissed)
|
||||
val targetDismissalDuration = dismissDurationForCount(info.count, baseDismissDuration)
|
||||
|
||||
currentDismissalDuration > targetDismissalDuration
|
||||
}).getOrElse(true)
|
||||
Stitch.value(isVisible)
|
||||
}
|
||||
|
||||
private def dismissDurationForCount(
|
||||
dismissCount: Int,
|
||||
dismissDuration: Duration
|
||||
): Duration =
|
||||
// limit to maximum dismissal duration
|
||||
dismissDuration * Math.min(dismissCount, DismissFatigueGate.MaximumDismissalCountMultiplier)
|
||||
}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.gizmoduck.{thriftscala => t}
|
||||
import com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* A Soft User is a user who is in the gradual onboarding state. This gate can be
|
||||
* used to turn off certain functionality like ads for these users.
|
||||
*/
|
||||
object ExcludeSoftUserGate extends Gate[PipelineQuery] {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("ExcludeSoftUser")
|
||||
|
||||
override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {
|
||||
val softUser = query.features
|
||||
.exists(_.getOrElse(UserTypeFeature, None).exists(_ == t.UserType.Soft))
|
||||
Stitch.value(!softUser)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.home_mixer.model.request.DeviceContext.RequestContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* Gate that fetches the request context from the device context and
|
||||
* continues if the request context matches *any* of the specified ones.
|
||||
*/
|
||||
case class RequestContextGate(requestContexts: Seq[RequestContext.Value])
|
||||
extends Gate[PipelineQuery with HasDeviceContext] {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("RequestContext")
|
||||
|
||||
override def shouldContinue(query: PipelineQuery with HasDeviceContext): Stitch[Boolean] =
|
||||
Stitch.value(
|
||||
requestContexts.exists(query.deviceContext.flatMap(_.requestContextValue).contains))
|
||||
}
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.home_mixer.model.request.DeviceContext.RequestContext
|
||||
import com.twitter.home_mixer.model.request.HasDeviceContext
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* Gate that fetches the request context from the device context and
|
||||
* continues if the request context does not match any of the specified ones.
|
||||
*
|
||||
* If no input request context is specified, the gate continues
|
||||
*/
|
||||
case class RequestContextNotGate(requestContexts: Seq[RequestContext.Value])
|
||||
extends Gate[PipelineQuery with HasDeviceContext] {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("RequestContextNot")
|
||||
|
||||
override def shouldContinue(query: PipelineQuery with HasDeviceContext): Stitch[Boolean] =
|
||||
Stitch.value(
|
||||
!requestContexts.exists(query.deviceContext.flatMap(_.requestContextValue).contains))
|
||||
}
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
object SupportedLanguagesGate extends Gate[PipelineQuery] {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("SupportedLanguages")
|
||||
|
||||
// Production languages which have high translation coverage for strings used in Home Timeline.
|
||||
private val supportedLanguages: Set[String] = Set(
|
||||
"ar", // Arabic
|
||||
"ar-x-fm", // Arabic (Female)
|
||||
"bg", // Bulgarian
|
||||
"bn", // Bengali
|
||||
"ca", // Catalan
|
||||
"cs", // Czech
|
||||
"da", // Danish
|
||||
"de", // German
|
||||
"el", // Greek
|
||||
"en", // English
|
||||
"en-gb", // British English
|
||||
"en-ss", // English Screen shot
|
||||
"en-xx", // English Pseudo
|
||||
"es", // Spanish
|
||||
"eu", // Basque
|
||||
"fa", // Farsi (Persian)
|
||||
"fi", // Finnish
|
||||
"fil", // Filipino
|
||||
"fr", // French
|
||||
"ga", // Irish
|
||||
"gl", // Galician
|
||||
"gu", // Gujarati
|
||||
"he", // Hebrew
|
||||
"hi", // Hindi
|
||||
"hr", // Croatian
|
||||
"hu", // Hungarian
|
||||
"id", // Indonesian
|
||||
"it", // Italian
|
||||
"ja", // Japanese
|
||||
"kn", // Kannada
|
||||
"ko", // Korean
|
||||
"mr", // Marathi
|
||||
"msa", // Malay
|
||||
"nl", // Dutch
|
||||
"no", // Norwegian
|
||||
"pl", // Polish
|
||||
"pt", // Portuguese
|
||||
"ro", // Romanian
|
||||
"ru", // Russian
|
||||
"sk", // Slovak
|
||||
"sr", // Serbian
|
||||
"sv", // Swedish
|
||||
"ta", // Tamil
|
||||
"th", // Thai
|
||||
"tr", // Turkish
|
||||
"uk", // Ukrainian
|
||||
"ur", // Urdu
|
||||
"vi", // Vietnamese
|
||||
"zh-cn", // Simplified Chinese
|
||||
"zh-tw" // Traditional Chinese
|
||||
)
|
||||
|
||||
override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =
|
||||
Stitch.value(query.getLanguageCode.forall(supportedLanguages.contains))
|
||||
}
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.gate
|
||||
|
||||
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.functional_component.gate.Gate
|
||||
import com.twitter.product_mixer.core.model.common.identifier.GateIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelinemixer.clients.persistence.TimelineResponseV3
|
||||
import com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils
|
||||
import com.twitter.timelines.configapi.Param
|
||||
import com.twitter.timelines.util.client_info.ClientPlatform
|
||||
import com.twitter.timelineservice.model.rich.EntityIdType
|
||||
import com.twitter.util.Duration
|
||||
import com.twitter.util.Time
|
||||
|
||||
/**
|
||||
* Gate used to reduce the frequency of injections. Note that the actual interval between injections may be
|
||||
* less than the specified minInjectionIntervalParam if data is unavailable or missing. For example, being deleted by
|
||||
* the persistence store via a TTL or similar mechanism.
|
||||
*
|
||||
* @param minInjectionIntervalParam the desired minimum interval between injections
|
||||
* @param persistenceEntriesFeature the feature for retrieving persisted timeline responses
|
||||
*/
|
||||
case class TimelinesPersistenceStoreLastInjectionGate(
|
||||
minInjectionIntervalParam: Param[Duration],
|
||||
persistenceEntriesFeature: Feature[PipelineQuery, Seq[TimelineResponseV3]],
|
||||
entityIdType: EntityIdType.Value)
|
||||
extends Gate[PipelineQuery]
|
||||
with TimelinePersistenceUtils {
|
||||
|
||||
override val identifier: GateIdentifier = GateIdentifier("TimelinesPersistenceStoreLastInjection")
|
||||
|
||||
override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =
|
||||
Stitch(
|
||||
query.queryTime.since(getLastInjectionTime(query)) > query.params(minInjectionIntervalParam))
|
||||
|
||||
private def getLastInjectionTime(query: PipelineQuery) = query.features
|
||||
.flatMap { featureMap =>
|
||||
val timelineResponses = featureMap.getOrElse(persistenceEntriesFeature, Seq.empty)
|
||||
val clientPlatform = ClientPlatform.fromQueryOptions(
|
||||
clientAppId = query.clientContext.appId,
|
||||
userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString)
|
||||
)
|
||||
val sortedResponses = responseByClient(clientPlatform, timelineResponses)
|
||||
val latestResponseWithEntityIdTypeEntry =
|
||||
sortedResponses.find(_.entries.exists(_.entityIdType == entityIdType))
|
||||
|
||||
latestResponseWithEntityIdTypeEntry.map(_.servedTime)
|
||||
}.getOrElse(Time.Bottom)
|
||||
}
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"common-internal/analytics/twitter-client-user-agent-parser/src/main/scala",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer",
|
||||
"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence",
|
||||
"timelineservice/common:model",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,85 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.query_transformer
|
||||
|
||||
import com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer
|
||||
import com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.timelinemixer.clients.persistence.EntryWithItemIds
|
||||
import com.twitter.timelines.persistence.thriftscala.RequestType
|
||||
import com.twitter.timelines.util.client_info.ClientPlatform
|
||||
import com.twitter.timelineservice.model.rich.EntityIdType
|
||||
import com.twitter.util.Time
|
||||
|
||||
object EditedTweetsCandidatePipelineQueryTransformer
|
||||
extends CandidatePipelineQueryTransformer[PipelineQuery, Seq[Long]] {
|
||||
|
||||
override val identifier: TransformerIdentifier = TransformerIdentifier("EditedTweets")
|
||||
|
||||
// The time window for which a tweet remains editable after creation.
|
||||
private val EditTimeWindow = 60.minutes
|
||||
|
||||
override def transform(query: PipelineQuery): Seq[Long] = {
|
||||
val applicableCandidates = getApplicableCandidates(query)
|
||||
|
||||
if (applicableCandidates.nonEmpty) {
|
||||
// Include the response corresponding with the Previous Timeline Load (PTL).
|
||||
// Any tweets in it could have become stale since being served.
|
||||
val previousTimelineLoadTime = applicableCandidates.head.servedTime
|
||||
|
||||
// The time window for editing a tweet is 60 minutes,
|
||||
// so we ignore responses older than (PTL Time - 60 mins).
|
||||
val inWindowCandidates: Seq[PersistenceStoreEntry] = applicableCandidates
|
||||
.takeWhile(_.servedTime.until(previousTimelineLoadTime) < EditTimeWindow)
|
||||
|
||||
// Exclude the tweet IDs for which ReplaceEntry instructions have already been sent.
|
||||
val (tweetsAlreadyReplaced, tweetsToCheck) = inWindowCandidates
|
||||
.partition(_.entryWithItemIds.itemIds.exists(_.head.entryIdToReplace.nonEmpty))
|
||||
|
||||
val tweetIdFromEntry: PartialFunction[PersistenceStoreEntry, Long] = {
|
||||
case entry if entry.tweetId.nonEmpty => entry.tweetId.get
|
||||
}
|
||||
|
||||
val tweetIdsAlreadyReplaced: Set[Long] = tweetsAlreadyReplaced.collect(tweetIdFromEntry).toSet
|
||||
val tweetIdsToCheck: Seq[Long] = tweetsToCheck.collect(tweetIdFromEntry)
|
||||
|
||||
tweetIdsToCheck.filterNot(tweetIdsAlreadyReplaced.contains).distinct
|
||||
} else Seq.empty
|
||||
}
|
||||
|
||||
// The candidates here come from the Timelines Persistence Store, via a query feature
|
||||
private def getApplicableCandidates(query: PipelineQuery): Seq[PersistenceStoreEntry] = {
|
||||
val userAgent = UserAgent.fromString(query.clientContext.userAgent.getOrElse(""))
|
||||
val clientPlatform = ClientPlatform.fromQueryOptions(query.clientContext.appId, userAgent)
|
||||
|
||||
val sortedResponses = query.features
|
||||
.getOrElse(FeatureMap.empty)
|
||||
.getOrElse(PersistenceEntriesFeature, Seq.empty)
|
||||
.filter(_.clientPlatform == clientPlatform)
|
||||
.sortBy(-_.servedTime.inMilliseconds)
|
||||
|
||||
val recentResponses = sortedResponses.indexWhere(_.requestType == RequestType.Initial) match {
|
||||
case -1 => sortedResponses
|
||||
case lastGetInitialIndex => sortedResponses.take(lastGetInitialIndex + 1)
|
||||
}
|
||||
|
||||
recentResponses.flatMap { r =>
|
||||
r.entries.collect {
|
||||
case entry if entry.entityIdType == EntityIdType.Tweet =>
|
||||
PersistenceStoreEntry(entry, r.servedTime, r.clientPlatform, r.requestType)
|
||||
}
|
||||
}.distinct
|
||||
}
|
||||
}
|
||||
|
||||
case class PersistenceStoreEntry(
|
||||
entryWithItemIds: EntryWithItemIds,
|
||||
servedTime: Time,
|
||||
clientPlatform: ClientPlatform,
|
||||
requestType: RequestType) {
|
||||
|
||||
// Timelines Persistence Store currently includes 1 tweet ID per entryWithItemIds for tweets
|
||||
val tweetId: Option[Long] = entryWithItemIds.itemIds.flatMap(_.head.tweetId)
|
||||
}
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product",
|
||||
"src/thrift/com/twitter/timelines/common:thrift-scala",
|
||||
"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala",
|
||||
"timelineservice/common:model",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.scorer
|
||||
|
||||
import com.twitter.conversions.DurationOps._
|
||||
import com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
|
||||
import com.twitter.home_mixer.util.CandidatesUtil
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.scorer.Scorer
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.Conditionally
|
||||
import com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
import com.twitter.timelines.common.{thriftscala => tl}
|
||||
import com.twitter.timelineservice.model.FeedbackEntry
|
||||
import com.twitter.timelineservice.{thriftscala => tls}
|
||||
import com.twitter.util.Time
|
||||
import scala.collection.mutable
|
||||
|
||||
object FeedbackFatigueScorer
|
||||
extends Scorer[PipelineQuery, TweetCandidate]
|
||||
with Conditionally[PipelineQuery] {
|
||||
|
||||
override val identifier: ScorerIdentifier = ScorerIdentifier("FeedbackFatigue")
|
||||
|
||||
override def features: Set[Feature[_, _]] = Set(ScoreFeature)
|
||||
|
||||
override def onlyIf(query: PipelineQuery): Boolean =
|
||||
query.features.exists(_.getOrElse(FeedbackHistoryFeature, Seq.empty).nonEmpty)
|
||||
|
||||
val DurationForFiltering = 14.days
|
||||
val DurationForDiscounting = 140.days
|
||||
private val ScoreMultiplierLowerBound = 0.2
|
||||
private val ScoreMultiplierUpperBound = 1.0
|
||||
private val ScoreMultiplierIncrementsCount = 4
|
||||
private val ScoreMultiplierIncrement =
|
||||
(ScoreMultiplierUpperBound - ScoreMultiplierLowerBound) / ScoreMultiplierIncrementsCount
|
||||
private val ScoreMultiplierIncrementDurationInDays =
|
||||
DurationForDiscounting.inDays / ScoreMultiplierIncrementsCount.toDouble
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
val feedbackEntriesByEngagementType =
|
||||
query.features
|
||||
.getOrElse(FeatureMap.empty).getOrElse(FeedbackHistoryFeature, Seq.empty)
|
||||
.filter { entry =>
|
||||
val timeSinceFeedback = query.queryTime.minus(entry.timestamp)
|
||||
timeSinceFeedback < DurationForFiltering + DurationForDiscounting &&
|
||||
entry.feedbackType == tls.FeedbackType.SeeFewer
|
||||
}.groupBy(_.engagementType)
|
||||
|
||||
val authorsToDiscount =
|
||||
getUserDiscounts(
|
||||
query.queryTime,
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Tweet, Seq.empty))
|
||||
val likersToDiscount =
|
||||
getUserDiscounts(
|
||||
query.queryTime,
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Like, Seq.empty))
|
||||
val followersToDiscount =
|
||||
getUserDiscounts(
|
||||
query.queryTime,
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Follow, Seq.empty))
|
||||
val retweetersToDiscount =
|
||||
getUserDiscounts(
|
||||
query.queryTime,
|
||||
feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Retweet, Seq.empty))
|
||||
|
||||
val featureMaps = candidates.map { candidate =>
|
||||
val multiplier = getScoreMultiplier(
|
||||
candidate,
|
||||
authorsToDiscount,
|
||||
likersToDiscount,
|
||||
followersToDiscount,
|
||||
retweetersToDiscount
|
||||
)
|
||||
val score = candidate.features.getOrElse(ScoreFeature, None)
|
||||
FeatureMapBuilder().add(ScoreFeature, score.map(_ * multiplier)).build()
|
||||
}
|
||||
|
||||
Stitch.value(featureMaps)
|
||||
}
|
||||
|
||||
def getScoreMultiplier(
|
||||
candidate: CandidateWithFeatures[TweetCandidate],
|
||||
authorsToDiscount: Map[Long, Double],
|
||||
likersToDiscount: Map[Long, Double],
|
||||
followersToDiscount: Map[Long, Double],
|
||||
retweetersToDiscount: Map[Long, Double],
|
||||
): Double = {
|
||||
val originalAuthorId =
|
||||
CandidatesUtil.getOriginalAuthorId(candidate.features).getOrElse(0L)
|
||||
val originalAuthorMultiplier = authorsToDiscount.getOrElse(originalAuthorId, 1.0)
|
||||
|
||||
val likers = candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)
|
||||
val likerMultipliers = likers.flatMap(likersToDiscount.get)
|
||||
val likerMultiplier =
|
||||
if (likerMultipliers.nonEmpty && likers.size == likerMultipliers.size)
|
||||
likerMultipliers.max
|
||||
else 1.0
|
||||
|
||||
val followers = candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)
|
||||
val followerMultipliers = followers.flatMap(followersToDiscount.get)
|
||||
val followerMultiplier =
|
||||
if (followerMultipliers.nonEmpty && followers.size == followerMultipliers.size &&
|
||||
likers.isEmpty)
|
||||
followerMultipliers.max
|
||||
else 1.0
|
||||
|
||||
val authorId = candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)
|
||||
val retweeterMultiplier =
|
||||
if (candidate.features.getOrElse(IsRetweetFeature, false))
|
||||
retweetersToDiscount.getOrElse(authorId, 1.0)
|
||||
else 1.0
|
||||
|
||||
originalAuthorMultiplier * likerMultiplier * followerMultiplier * retweeterMultiplier
|
||||
}
|
||||
|
||||
def getUserDiscounts(
|
||||
queryTime: Time,
|
||||
feedbackEntries: Seq[FeedbackEntry],
|
||||
): Map[Long, Double] = {
|
||||
val userDiscounts = mutable.Map[Long, Double]()
|
||||
feedbackEntries
|
||||
.collect {
|
||||
case FeedbackEntry(_, _, tl.FeedbackEntity.UserId(userId), timestamp, _) =>
|
||||
val timeSinceFeedback = queryTime.minus(timestamp)
|
||||
val timeSinceDiscounting = timeSinceFeedback - DurationForFiltering
|
||||
val multiplier = ((timeSinceDiscounting.inDays / ScoreMultiplierIncrementDurationInDays)
|
||||
* ScoreMultiplierIncrement + ScoreMultiplierLowerBound)
|
||||
userDiscounts.update(userId, multiplier)
|
||||
}
|
||||
userDiscounts.toMap
|
||||
}
|
||||
}
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.scorer
|
||||
|
||||
import com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature
|
||||
import com.twitter.home_mixer.model.HomeFeatures.ScoreFeature
|
||||
import com.twitter.product_mixer.component_library.model.candidate.TweetCandidate
|
||||
import com.twitter.product_mixer.core.feature.Feature
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMap
|
||||
import com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder
|
||||
import com.twitter.product_mixer.core.functional_component.scorer.Scorer
|
||||
import com.twitter.product_mixer.core.model.common.CandidateWithFeatures
|
||||
import com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
import com.twitter.stitch.Stitch
|
||||
|
||||
/**
|
||||
* Scales scores of each out-of-network tweet by the specified scale factor
|
||||
*/
|
||||
object OONTweetScalingScorer extends Scorer[PipelineQuery, TweetCandidate] {
|
||||
|
||||
override val identifier: ScorerIdentifier = ScorerIdentifier("OONTweetScaling")
|
||||
|
||||
override val features: Set[Feature[_, _]] = Set(ScoreFeature)
|
||||
|
||||
private val ScaleFactor = 0.75
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
candidates: Seq[CandidateWithFeatures[TweetCandidate]]
|
||||
): Stitch[Seq[FeatureMap]] = {
|
||||
Stitch.value {
|
||||
candidates.map { candidate =>
|
||||
val score = candidate.features.getOrElse(ScoreFeature, None)
|
||||
val updatedScore = if (selector(candidate)) score.map(_ * ScaleFactor) else score
|
||||
FeatureMapBuilder().add(ScoreFeature, updatedScore).build()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* We should only be applying this multiplier to Out-Of-Network tweets.
|
||||
* In-Network Retweets of Out-Of-Network tweets should not have this multiplier applied
|
||||
*/
|
||||
private def selector(candidate: CandidateWithFeatures[TweetCandidate]): Boolean = {
|
||||
!candidate.features.getOrElse(InNetworkFeature, false) &&
|
||||
!candidate.features.getOrElse(IsRetweetFeature, false)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
scala_library(
|
||||
sources = ["*.scala"],
|
||||
compiler_option_sets = ["fatal_warnings"],
|
||||
strict_deps = True,
|
||||
tags = ["bazel-compatible"],
|
||||
dependencies = [
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/param",
|
||||
"home-mixer/server/src/main/scala/com/twitter/home_mixer/util",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate",
|
||||
"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item",
|
||||
"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline",
|
||||
"src/scala/com/twitter/suggests/controller_data",
|
||||
"stringcenter/client",
|
||||
"stringcenter/client/src/main/java",
|
||||
],
|
||||
)
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
package com.twitter.home_mixer.functional_component.selector
|
||||
|
||||
import com.twitter.home_mixer.functional_component.selector.DebunchCandidates.TrailingTweetsMinSize
|
||||
import com.twitter.home_mixer.functional_component.selector.DebunchCandidates.TrailingTweetsPortionToKeep
|
||||
import com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature
|
||||
import com.twitter.product_mixer.core.functional_component.common.CandidateScope
|
||||
import com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates
|
||||
import com.twitter.product_mixer.core.functional_component.selector.Selector
|
||||
import com.twitter.product_mixer.core.functional_component.selector.SelectorResult
|
||||
import com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails
|
||||
import com.twitter.product_mixer.core.pipeline.PipelineQuery
|
||||
|
||||
trait MustDebunch {
|
||||
def apply(candidate: CandidateWithDetails): Boolean
|
||||
}
|
||||
|
||||
object DebunchCandidates {
|
||||
val TrailingTweetsMinSize = 5
|
||||
val TrailingTweetsPortionToKeep = 0.1
|
||||
}
|
||||
|
||||
/**
|
||||
* This selector rearranges the candidates to only allow bunches of size [[maxBunchSize]], where a
|
||||
* bunch is a consecutive sequence of candidates that meet [[mustDebunch]].
|
||||
*/
|
||||
case class DebunchCandidates(
|
||||
override val pipelineScope: CandidateScope,
|
||||
mustDebunch: MustDebunch,
|
||||
maxBunchSize: Int)
|
||||
extends Selector[PipelineQuery] {
|
||||
|
||||
override def apply(
|
||||
query: PipelineQuery,
|
||||
remainingCandidates: Seq[CandidateWithDetails],
|
||||
result: Seq[CandidateWithDetails]
|
||||
): SelectorResult = {
|
||||
val PartitionedCandidates(selectedCandidates, otherCandidates) =
|
||||
pipelineScope.partition(remainingCandidates)
|
||||
val mutableCandidates = collection.mutable.ListBuffer(selectedCandidates: _*)
|
||||
|
||||
var candidatePointer = 0
|
||||
var nonDebunchPointer = 0
|
||||
var bunchSize = 0
|
||||
var finalNonDebunch = -1
|
||||
|
||||
while (candidatePointer < mutableCandidates.size) {
|
||||
if (mustDebunch(mutableCandidates(candidatePointer))) bunchSize += 1
|
||||
else {
|
||||
bunchSize = 0
|
||||
finalNonDebunch = candidatePointer
|
||||
}
|
||||
|
||||
if (bunchSize > maxBunchSize) {
|
||||
nonDebunchPointer = Math.max(candidatePointer, nonDebunchPointer)
|
||||
while (nonDebunchPointer < mutableCandidates.size &&
|
||||
mustDebunch(mutableCandidates(nonDebunchPointer))) {
|
||||
nonDebunchPointer += 1
|
||||
}
|
||||
if (nonDebunchPointer == mutableCandidates.size)
|
||||
candidatePointer = mutableCandidates.size
|
||||
else {
|
||||
val nextNonDebunch = mutableCandidates(nonDebunchPointer)
|
||||
mutableCandidates.remove(nonDebunchPointer)
|
||||
mutableCandidates.insert(candidatePointer, nextNonDebunch)
|
||||
bunchSize = 0
|
||||
finalNonDebunch = candidatePointer
|
||||
}
|
||||
}
|
||||
|
||||
candidatePointer += 1
|
||||
}
|
||||
|
||||
val debunchedCandidates = if (query.features.exists(_.getOrElse(GetNewerFeature, false))) {
|
||||
val trailingTweetsSize = mutableCandidates.size - finalNonDebunch - 1
|
||||
val keepCandidates = finalNonDebunch + 1 +
|
||||
Math.max(TrailingTweetsMinSize, TrailingTweetsPortionToKeep * trailingTweetsSize).toInt
|
||||
mutableCandidates.toList.take(keepCandidates)
|
||||
} else mutableCandidates.toList
|
||||
|
||||
val updatedCandidates = otherCandidates ++ debunchedCandidates
|
||||
SelectorResult(remainingCandidates = updatedCandidates, result = result)
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue