Merge branch 'main' into fix/rust-build-error-handling

This commit is contained in:
David 2023-04-16 00:24:11 +01:00
commit 16082b9415
354 changed files with 31749 additions and 16 deletions

View File

@ -1,22 +1,39 @@
# Twitter's Recommendation Algorithm
Twitter's Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving the
Home Timeline. For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm). The
diagram below illustrates how major services and jobs interconnect.
Twitter's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of Tweets and other content across all Twitter product surfaces (e.g. For You Timeline, Search, Explore). For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).
![](docs/system-diagram.png)
## Architecture
These are the main components of the Recommendation Algorithm included in this repository:
Product surfaces at Twitter are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Data | [unified-user-actions](unified_user_actions/README.md) | Real-time stream of user actions on Twitter. |
| | [user-signal-service](user-signal-service/README.md) | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. |
| Model | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |
| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets. |
| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |
| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of a Twitter User interacting with another User. |
| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating Twitter User reputation. |
| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |
| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). |
| | [topic-social-proof](topic-social-proof/README.md) | Identifies topics related to individual Tweets. |
| Software framework | [navi](navi/README.md) | High performance, machine learning model serving written in Rust. |
| | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content. |
| | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1. |
The product surface currently included in this repository is the For You Timeline.
### For You Timeline
The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.
![](docs/system-diagram.png)
The core components of the For You Timeline included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Feature | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |
| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets. |
| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |
| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of a Twitter User interacting with another User. |
| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating Twitter User reputation. |
| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |
| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). |
| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network Tweets. ~50% of Tweets come from this candidate source. |
| | [cr-mixer](cr-mixer/README.md) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |
| | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG)| Maintains an in memory User to Tweet interaction graph, and finds candidates based on traversals of this graph. This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework. Several other GraphJet based features and candidate sources are located [here](src/scala/com/twitter/recos). |
@ -26,11 +43,10 @@ These are the main components of the Recommendation Algorithm included in this r
| Tweet mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline. Built on [product-mixer](product-mixer/README.md). |
| | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |
| | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service. |
| Software framework | [navi](navi/README.md) | High performance, machine learning model serving written in Rust. |
| | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content. |
| | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1. |
We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file.
## Build and test code
We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.
## Contributing

View File

@ -0,0 +1,8 @@
# Topic Social Proof Service (TSPS)
=================
**Topic Social Proof Service** (TSPS) serves as a centralized source for verifying topics related to Timelines and Notifications. By analyzing user's topic preferences, such as following or unfollowing, and employing semantic annotations and tweet embeddings from SimClusters, or other machine learning models, TSPS delivers highly relevant topics tailored to each user's interests.
For instance, when a tweet discusses Stephen Curry, the service determines if the content falls under topics like "NBA" and/or "Golden State Warriors" while also providing relevance scores based on SimClusters Embedding. Additionally, TSPS evaluates user-specific topic preferences to offer a comprehensive list of available topics, only those the user is currently following, or new topics they have not followed but may find interesting if recommended on specific product surfaces.

View File

@ -0,0 +1,24 @@
jvm_binary(
name = "bin",
basename = "topic-social-proof",
main = "com.twitter.tsp.TopicSocialProofStratoFedServerMain",
runtime_platform = "java11",
tags = [
"bazel-compatible",
],
dependencies = [
"strato/src/main/scala/com/twitter/strato/logging/logback",
"topic-social-proof/server/src/main/resources",
"topic-social-proof/server/src/main/scala/com/twitter/tsp",
],
)
# Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app
jvm_app(
name = "topic-social-proof-app",
archive = "zip",
binary = ":bin",
tags = [
"bazel-compatible",
],
)

View File

@ -0,0 +1,8 @@
resources(
sources = [
"*.xml",
"*.yml",
"config/*.yml",
],
tags = ["bazel-compatible"],
)

View File

@ -0,0 +1,61 @@
# Keys are sorted in an alphabetical order
enable_topic_social_proof_score:
comment : "Enable the calculation of <topic, tweet> cosine similarity score in TopicSocialProofStore. 0 means do not calculate the score and use a random rank to generate topic social proof"
default_availability: 0
enable_tweet_health_score:
comment: "Enable the calculation for health scores in tweetInfo. By enabling this decider, we will compute TweetHealthModelScore"
default_availability: 0
enable_user_agatha_score:
comment: "Enable the calculation for health scores in tweetInfo. By enabling this decider, we will compute UserHealthModelScore"
default_availability: 0
enable_loadshedding_HomeTimeline:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineRecommendTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_MagicRecsRecommendTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_TopicLandingPage:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineFeatures:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineTopicTweetsMetrics:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineUTEGTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_HomeTimelineSimClusters:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_ExploreTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_MagicRecsTopicTweets:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0
enable_loadshedding_Search:
comment: "Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response"
default_availability: 0

View File

@ -0,0 +1,155 @@
<configuration>
<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
<property name="async_queue_size" value="${queue.size:-50000}"/>
<property name="async_max_flush_time" value="${max.flush.time:-0}"/>
<!-- ===================================================== -->
<!-- Structured Logging -->
<!-- ===================================================== -->
<!-- Only sample 0.1% of the requests -->
<property name="splunk_sampling_rate" value="${splunk_sampling_rate:-0.001}"/>
<include resource="structured-logger-logback.xml"/>
<!-- ===================================================== -->
<!-- Service Config -->
<!-- ===================================================== -->
<property name="DEFAULT_SERVICE_PATTERN"
value="%-16X{transactionId} %logger %msg"/>
<!-- ===================================================== -->
<!-- Common Config -->
<!-- ===================================================== -->
<!-- JUL/JDK14 to Logback bridge -->
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<!-- Service Log (Rollover every 50MB, max 11 logs) -->
<appender name="SERVICE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${log.service.output}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${log.service.output}.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>50MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>
</encoder>
</appender>
<!-- Strato package only log (Rollover every 50MB, max 11 logs) -->
<appender name="STRATO-ONLY" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${log.strato_only.output}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${log.strato_only.output}.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>50MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>
</encoder>
</appender>
<!-- LogLens -->
<appender name="LOGLENS" class="com.twitter.loglens.logback.LoglensAppender">
<mdcAdditionalContext>true</mdcAdditionalContext>
<category>loglens</category>
<index>${log.lens.index}</index>
<tag>${log.lens.tag}/service</tag>
<encoder>
<pattern>%msg%n</pattern>
</encoder>
<turboFilter class="ch.qos.logback.classic.turbo.DuplicateMessageFilter">
<cacheSize>500</cacheSize>
<allowedRepetitions>50</allowedRepetitions>
</turboFilter>
<filter class="com.twitter.strato.logging.logback.RegexFilter">
<forLogger>manhattan-client</forLogger>
<excludeRegex>.*InvalidRequest.*</excludeRegex>
</filter>
</appender>
<!-- ===================================================== -->
<!-- Primary Async Appenders -->
<!-- ===================================================== -->
<appender name="ASYNC-SERVICE" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>${async_queue_size}</queueSize>
<maxFlushTime>${async_max_flush_time}</maxFlushTime>
<appender-ref ref="SERVICE"/>
</appender>
<appender name="ASYNC-STRATO-ONLY" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>${async_queue_size}</queueSize>
<maxFlushTime>${async_max_flush_time}</maxFlushTime>
<appender-ref ref="STRATO-ONLY"/>
</appender>
<appender name="ASYNC-LOGLENS" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>${async_queue_size}</queueSize>
<maxFlushTime>${async_max_flush_time}</maxFlushTime>
<appender-ref ref="LOGLENS"/>
</appender>
<!-- ===================================================== -->
<!-- Package Config -->
<!-- ===================================================== -->
<!-- Per-Package Config (shared) -->
<logger name="com.twitter" level="info"/>
<!--
By default, we leave the strato package at INFO level.
However, this line allows us to set the entire strato package, or a subset of it, to
a specific level. For example, if you pass -Dstrato_log_package=streaming -Dstrato_log_level=DEBUG
only loggers under com.twitter.strato.streaming.* will be set to DEBUG level. Passing only
-Dstrato_log_level will set all of strato.* to the specified level.
-->
<logger name="com.twitter.strato${strato_log_package:-}" level="${strato_log_level:-INFO}"/>
<logger name="com.twitter.wilyns" level="warn"/>
<logger name="com.twitter.finagle.mux" level="warn"/>
<logger name="com.twitter.finagle.serverset2" level="warn"/>
<logger name="com.twitter.logging.ScribeHandler" level="warn"/>
<logger name="com.twitter.zookeeper.client.internal" level="warn"/>
<logger name="com.twitter.decider.StoreDecider" level="warn"/>
<!-- Per-Package Config (Strato) -->
<logger name="com.twitter.distributedlog.client" level="warn"/>
<logger name="com.twitter.finagle.mtls.authorization.config.AccessControlListConfiguration" level="warn"/>
<logger name="com.twitter.finatra.kafka.common.kerberoshelpers" level="warn"/>
<logger name="com.twitter.finatra.kafka.utils.BootstrapServerUtils" level="warn"/>
<logger name="com.twitter.server.coordinate" level="error"/>
<logger name="com.twitter.zookeeper.client" level="info"/>
<logger name="org.apache.zookeeper" level="error"/>
<logger name="org.apache.zookeeper.ClientCnxn" level="warn"/>
<logger name="ZkSession" level="info"/>
<logger name="OptimisticLockingCache" level="off"/>
<logger name="manhattan-client" level="warn"/>
<logger name="strato.op" level="warn"/>
<logger name="org.apache.kafka.clients.NetworkClient" level="error"/>
<logger name="org.apache.kafka.clients.consumer.internals" level="error"/>
<logger name="org.apache.kafka.clients.producer.internals" level="error"/>
<!-- produce a lot of messages like: Building client authenticator with server name kafka -->
<logger name="org.apache.kafka.common.network" level="warn"/>
<!-- Root Config -->
<root level="${log_level:-INFO}">
<appender-ref ref="ASYNC-SERVICE"/>
<appender-ref ref="ASYNC-LOGLENS"/>
</root>
<!-- Strato package only logging-->
<logger name="com.twitter.strato"
level="info"
additivity="true">
<appender-ref ref="ASYNC-STRATO-ONLY" />
</logger>
</configuration>

View File

@ -0,0 +1,12 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"finatra/inject/inject-thrift-client",
"strato/src/main/scala/com/twitter/strato/fed",
"strato/src/main/scala/com/twitter/strato/fed/server",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/columns",
],
)

View File

@ -0,0 +1,56 @@
package com.twitter.tsp
import com.google.inject.Module
import com.twitter.strato.fed._
import com.twitter.strato.fed.server._
import com.twitter.strato.warmup.Warmer
import com.twitter.tsp.columns.TopicSocialProofColumn
import com.twitter.tsp.columns.TopicSocialProofBatchColumn
import com.twitter.tsp.handlers.UttChildrenWarmupHandler
import com.twitter.tsp.modules.RepresentationScorerStoreModule
import com.twitter.tsp.modules.GizmoduckUserModule
import com.twitter.tsp.modules.TSPClientIdModule
import com.twitter.tsp.modules.TopicListingModule
import com.twitter.tsp.modules.TopicSocialProofStoreModule
import com.twitter.tsp.modules.TopicTweetCosineSimilarityAggregateStoreModule
import com.twitter.tsp.modules.TweetInfoStoreModule
import com.twitter.tsp.modules.TweetyPieClientModule
import com.twitter.tsp.modules.UttClientModule
import com.twitter.tsp.modules.UttLocalizationModule
import com.twitter.util.Future
object TopicSocialProofStratoFedServerMain extends TopicSocialProofStratoFedServer
trait TopicSocialProofStratoFedServer extends StratoFedServer {
override def dest: String = "/s/topic-social-proof/topic-social-proof"
override val modules: Seq[Module] =
Seq(
GizmoduckUserModule,
RepresentationScorerStoreModule,
TopicSocialProofStoreModule,
TopicListingModule,
TopicTweetCosineSimilarityAggregateStoreModule,
TSPClientIdModule,
TweetInfoStoreModule,
TweetyPieClientModule,
UttClientModule,
UttLocalizationModule
)
override def columns: Seq[Class[_ <: StratoFed.Column]] =
Seq(
classOf[TopicSocialProofColumn],
classOf[TopicSocialProofBatchColumn]
)
override def configureWarmer(warmer: Warmer): Unit = {
warmer.add(
"uttChildrenWarmupHandler",
() => {
handle[UttChildrenWarmupHandler]()
Future.Unit
}
)
}
}

View File

@ -0,0 +1,12 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"stitch/stitch-storehaus",
"strato/src/main/scala/com/twitter/strato/fed",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/service",
"topic-social-proof/server/src/main/thrift:thrift-scala",
],
)

View File

@ -0,0 +1,84 @@
package com.twitter.tsp.columns
import com.twitter.stitch.SeqGroup
import com.twitter.stitch.Stitch
import com.twitter.strato.catalog.Fetch
import com.twitter.strato.catalog.OpMetadata
import com.twitter.strato.config._
import com.twitter.strato.config.AllowAll
import com.twitter.strato.config.ContactInfo
import com.twitter.strato.config.Policy
import com.twitter.strato.data.Conv
import com.twitter.strato.data.Description.PlainText
import com.twitter.strato.data.Lifecycle.Production
import com.twitter.strato.fed.StratoFed
import com.twitter.strato.thrift.ScroogeConv
import com.twitter.tsp.thriftscala.TopicSocialProofRequest
import com.twitter.tsp.thriftscala.TopicSocialProofOptions
import com.twitter.tsp.service.TopicSocialProofService
import com.twitter.tsp.thriftscala.TopicWithScore
import com.twitter.util.Future
import com.twitter.util.Try
import javax.inject.Inject
class TopicSocialProofBatchColumn @Inject() (
topicSocialProofService: TopicSocialProofService)
extends StratoFed.Column(TopicSocialProofBatchColumn.Path)
with StratoFed.Fetch.Stitch {
override val policy: Policy =
ReadWritePolicy(
readPolicy = AllowAll,
writePolicy = AllowKeyAuthenticatedTwitterUserId
)
override type Key = Long
override type View = TopicSocialProofOptions
override type Value = Seq[TopicWithScore]
override val keyConv: Conv[Key] = Conv.ofType
override val viewConv: Conv[View] = ScroogeConv.fromStruct[TopicSocialProofOptions]
override val valueConv: Conv[Value] = Conv.seq(ScroogeConv.fromStruct[TopicWithScore])
override val metadata: OpMetadata =
OpMetadata(
lifecycle = Some(Production),
Some(PlainText("Topic Social Proof Batched Federated Column")))
case class TspsGroup(view: View) extends SeqGroup[Long, Fetch.Result[Value]] {
override protected def run(keys: Seq[Long]): Future[Seq[Try[Result[Seq[TopicWithScore]]]]] = {
val request = TopicSocialProofRequest(
userId = view.userId,
tweetIds = keys.toSet,
displayLocation = view.displayLocation,
topicListingSetting = view.topicListingSetting,
context = view.context,
bypassModes = view.bypassModes,
tags = view.tags
)
val response = topicSocialProofService
.topicSocialProofHandlerStoreStitch(request)
.map(_.socialProofs)
Stitch
.run(response).map(r =>
keys.map(key => {
Try {
val v = r.get(key)
if (v.nonEmpty && v.get.nonEmpty) {
found(v.get)
} else {
missing
}
}
}))
}
}
override def fetch(key: Key, view: View): Stitch[Result[Value]] = {
Stitch.call(key, TspsGroup(view))
}
}
object TopicSocialProofBatchColumn {
val Path = "topic-signals/tsp/topic-social-proof-batched"
}

View File

@ -0,0 +1,47 @@
package com.twitter.tsp.columns
import com.twitter.stitch
import com.twitter.stitch.Stitch
import com.twitter.strato.catalog.OpMetadata
import com.twitter.strato.config._
import com.twitter.strato.config.AllowAll
import com.twitter.strato.config.ContactInfo
import com.twitter.strato.config.Policy
import com.twitter.strato.data.Conv
import com.twitter.strato.data.Description.PlainText
import com.twitter.strato.data.Lifecycle.Production
import com.twitter.strato.fed.StratoFed
import com.twitter.strato.thrift.ScroogeConv
import com.twitter.tsp.thriftscala.TopicSocialProofRequest
import com.twitter.tsp.thriftscala.TopicSocialProofResponse
import com.twitter.tsp.service.TopicSocialProofService
import javax.inject.Inject
class TopicSocialProofColumn @Inject() (
topicSocialProofService: TopicSocialProofService)
extends StratoFed.Column(TopicSocialProofColumn.Path)
with StratoFed.Fetch.Stitch {
override type Key = TopicSocialProofRequest
override type View = Unit
override type Value = TopicSocialProofResponse
override val keyConv: Conv[Key] = ScroogeConv.fromStruct[TopicSocialProofRequest]
override val viewConv: Conv[View] = Conv.ofType
override val valueConv: Conv[Value] = ScroogeConv.fromStruct[TopicSocialProofResponse]
override val metadata: OpMetadata =
OpMetadata(lifecycle = Some(Production), Some(PlainText("Topic Social Proof Federated Column")))
override def fetch(key: Key, view: View): Stitch[Result[Value]] = {
topicSocialProofService
.topicSocialProofHandlerStoreStitch(key)
.map { result => found(result) }
.handle {
case stitch.NotFound => missing
}
}
}
object TopicSocialProofColumn {
val Path = "topic-signals/tsp/topic-social-proof"
}

View File

@ -0,0 +1,23 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"configapi/configapi-abdecider",
"configapi/configapi-core",
"content-recommender/thrift/src/main/thrift:thrift-scala",
"decider/src/main/scala",
"discovery-common/src/main/scala/com/twitter/discovery/common/configapi",
"featureswitches/featureswitches-core",
"finatra/inject/inject-core/src/main/scala",
"frigate/frigate-common:base",
"frigate/frigate-common:util",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate",
"interests-service/thrift/src/main/thrift:thrift-scala",
"src/scala/com/twitter/simclusters_v2/common",
"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala",
"stitch/stitch-storehaus",
"topic-social-proof/server/src/main/thrift:thrift-scala",
],
)

View File

@ -0,0 +1,19 @@
package com.twitter.tsp.common
import com.twitter.servo.decider.DeciderKeyEnum
object DeciderConstants {
val enableTopicSocialProofScore = "enable_topic_social_proof_score"
val enableHealthSignalsScoreDeciderKey = "enable_tweet_health_score"
val enableUserAgathaScoreDeciderKey = "enable_user_agatha_score"
}
object DeciderKey extends DeciderKeyEnum {
val enableHealthSignalsScoreDeciderKey: Value = Value(
DeciderConstants.enableHealthSignalsScoreDeciderKey
)
val enableUserAgathaScoreDeciderKey: Value = Value(
DeciderConstants.enableUserAgathaScoreDeciderKey
)
}

View File

@ -0,0 +1,34 @@
package com.twitter.tsp.common
import com.twitter.abdecider.LoggingABDecider
import com.twitter.featureswitches.v2.FeatureSwitches
import com.twitter.featureswitches.v2.builder.{FeatureSwitchesBuilder => FsBuilder}
import com.twitter.featureswitches.v2.experimentation.NullBucketImpressor
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.util.Duration
case class FeatureSwitchesBuilder(
statsReceiver: StatsReceiver,
abDecider: LoggingABDecider,
featuresDirectory: String,
addServiceDetailsFromAurora: Boolean,
configRepoDirectory: String = "/usr/local/config",
fastRefresh: Boolean = false,
impressExperiments: Boolean = true) {
def build(): FeatureSwitches = {
val featureSwitches = FsBuilder()
.abDecider(abDecider)
.statsReceiver(statsReceiver)
.configRepoAbsPath(configRepoDirectory)
.featuresDirectory(featuresDirectory)
.limitToReferencedExperiments(shouldLimit = true)
.experimentImpressionStatsEnabled(true)
if (!impressExperiments) featureSwitches.experimentBucketImpressor(NullBucketImpressor)
if (addServiceDetailsFromAurora) featureSwitches.serviceDetailsFromAurora()
if (fastRefresh) featureSwitches.refreshPeriod(Duration.fromSeconds(10))
featureSwitches.build()
}
}

View File

@ -0,0 +1,44 @@
package com.twitter.tsp.common
import com.twitter.decider.Decider
import com.twitter.decider.RandomRecipient
import com.twitter.util.Future
import javax.inject.Inject
import scala.util.control.NoStackTrace
/*
Provides deciders-controlled load shedding for a given displayLocation
The format of the decider keys is:
enable_loadshedding_<display location>
E.g.:
enable_loadshedding_HomeTimeline
Deciders are fractional, so a value of 50.00 will drop 50% of responses. If a decider key is not
defined for a particular displayLocation, those requests will always be served.
We should therefore aim to define keys for the locations we care most about in decider.yml,
so that we can control them during incidents.
*/
class LoadShedder @Inject() (decider: Decider) {
import LoadShedder._
// Fall back to False for any undefined key
private val deciderWithFalseFallback: Decider = decider.orElse(Decider.False)
private val keyPrefix = "enable_loadshedding"
def apply[T](typeString: String)(serve: => Future[T]): Future[T] = {
/*
Per-typeString level load shedding: enable_loadshedding_HomeTimeline
Checks if per-typeString load shedding is enabled
*/
val keyTyped = s"${keyPrefix}_$typeString"
if (deciderWithFalseFallback.isAvailable(keyTyped, recipient = Some(RandomRecipient)))
Future.exception(LoadSheddingException)
else serve
}
}
object LoadShedder {
object LoadSheddingException extends Exception with NoStackTrace
}

View File

@ -0,0 +1,98 @@
package com.twitter.tsp.common
import com.twitter.abdecider.LoggingABDecider
import com.twitter.abdecider.UserRecipient
import com.twitter.contentrecommender.thriftscala.DisplayLocation
import com.twitter.discovery.common.configapi.FeatureContextBuilder
import com.twitter.featureswitches.FSRecipient
import com.twitter.featureswitches.Recipient
import com.twitter.featureswitches.UserAgent
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.interests.thriftscala.TopicListingViewerContext
import com.twitter.timelines.configapi
import com.twitter.timelines.configapi.Params
import com.twitter.timelines.configapi.RequestContext
import com.twitter.timelines.configapi.abdecider.LoggingABDeciderExperimentContext
case class ParamsBuilder(
featureContextBuilder: FeatureContextBuilder,
abDecider: LoggingABDecider,
overridesConfig: configapi.Config,
statsReceiver: StatsReceiver) {
def buildFromTopicListingViewerContext(
topicListingViewerContext: Option[TopicListingViewerContext],
displayLocation: DisplayLocation,
userRoleOverride: Option[Set[String]] = None
): Params = {
topicListingViewerContext.flatMap(_.userId) match {
case Some(userId) =>
val userRecipient = ParamsBuilder.toFeatureSwitchRecipientWithTopicContext(
userId,
userRoleOverride,
topicListingViewerContext,
Some(displayLocation)
)
overridesConfig(
requestContext = RequestContext(
userId = Some(userId),
experimentContext = LoggingABDeciderExperimentContext(
abDecider,
Some(UserRecipient(userId, Some(userId)))),
featureContext = featureContextBuilder(
Some(userId),
Some(userRecipient)
)
),
statsReceiver
)
case _ =>
throw new IllegalArgumentException(
s"${this.getClass.getSimpleName} tried to build Param for a request without a userId"
)
}
}
}
object ParamsBuilder {
def toFeatureSwitchRecipientWithTopicContext(
userId: Long,
userRolesOverride: Option[Set[String]],
context: Option[TopicListingViewerContext],
displayLocationOpt: Option[DisplayLocation]
): Recipient = {
val userRoles = userRolesOverride match {
case Some(overrides) => Some(overrides)
case _ => context.flatMap(_.userRoles.map(_.toSet))
}
val recipient = FSRecipient(
userId = Some(userId),
userRoles = userRoles,
deviceId = context.flatMap(_.deviceId),
guestId = context.flatMap(_.guestId),
languageCode = context.flatMap(_.languageCode),
countryCode = context.flatMap(_.countryCode),
userAgent = context.flatMap(_.userAgent).flatMap(UserAgent(_)),
isVerified = None,
isTwoffice = None,
tooClient = None,
highWaterMark = None
)
displayLocationOpt match {
case Some(displayLocation) =>
recipient.withCustomFields(displayLocationCustomFieldMap(displayLocation))
case None =>
recipient
}
}
private val DisplayLocationCustomField = "display_location"
def displayLocationCustomFieldMap(displayLocation: DisplayLocation): (String, String) =
DisplayLocationCustomField -> displayLocation.toString
}

View File

@ -0,0 +1,65 @@
package com.twitter.tsp.common
import com.twitter.abdecider.LoggingABDecider
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.base.TargetUser
import com.twitter.frigate.common.candidate.TargetABDecider
import com.twitter.frigate.common.util.ABDeciderWithOverride
import com.twitter.gizmoduck.thriftscala.User
import com.twitter.simclusters_v2.common.UserId
import com.twitter.storehaus.ReadableStore
import com.twitter.timelines.configapi.Params
import com.twitter.tsp.thriftscala.TopicSocialProofRequest
import com.twitter.util.Future
case class DefaultRecTopicSocialProofTarget(
topicSocialProofRequest: TopicSocialProofRequest,
targetId: UserId,
user: Option[User],
abDecider: ABDeciderWithOverride,
params: Params
)(
implicit statsReceiver: StatsReceiver)
extends TargetUser
with TopicSocialProofRecRequest
with TargetABDecider {
override def globalStats: StatsReceiver = statsReceiver
override val targetUser: Future[Option[User]] = Future.value(user)
}
trait TopicSocialProofRecRequest {
tuc: TargetUser =>
val topicSocialProofRequest: TopicSocialProofRequest
}
case class RecTargetFactory(
abDecider: LoggingABDecider,
userStore: ReadableStore[UserId, User],
paramBuilder: ParamsBuilder,
statsReceiver: StatsReceiver) {
type RecTopicSocialProofTarget = DefaultRecTopicSocialProofTarget
def buildRecTopicSocialProofTarget(
request: TopicSocialProofRequest
): Future[RecTopicSocialProofTarget] = {
val userId = request.userId
userStore.get(userId).map { userOpt =>
val userRoles = userOpt.flatMap(_.roles.map(_.roles.toSet))
val context = request.context.copy(userId = Some(request.userId)) // override to make sure
val params = paramBuilder
.buildFromTopicListingViewerContext(Some(context), request.displayLocation, userRoles)
DefaultRecTopicSocialProofTarget(
request,
userId,
userOpt,
ABDeciderWithOverride(abDecider, None)(statsReceiver),
params
)(statsReceiver)
}
}
}

View File

@ -0,0 +1,26 @@
package com.twitter.tsp
package common
import com.twitter.decider.Decider
import com.twitter.decider.RandomRecipient
import com.twitter.decider.Recipient
import com.twitter.simclusters_v2.common.DeciderGateBuilderWithIdHashing
import javax.inject.Inject
case class TopicSocialProofDecider @Inject() (decider: Decider) {
def isAvailable(feature: String, recipient: Option[Recipient]): Boolean = {
decider.isAvailable(feature, recipient)
}
lazy val deciderGateBuilder = new DeciderGateBuilderWithIdHashing(decider)
/**
* When useRandomRecipient is set to false, the decider is either completely on or off.
* When useRandomRecipient is set to true, the decider is on for the specified % of traffic.
*/
def isAvailable(feature: String, useRandomRecipient: Boolean = true): Boolean = {
if (useRandomRecipient) isAvailable(feature, Some(RandomRecipient))
else isAvailable(feature, None)
}
}

View File

@ -0,0 +1,104 @@
package com.twitter.tsp.common
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.logging.Logger
import com.twitter.timelines.configapi.BaseConfig
import com.twitter.timelines.configapi.BaseConfigBuilder
import com.twitter.timelines.configapi.FSBoundedParam
import com.twitter.timelines.configapi.FSParam
import com.twitter.timelines.configapi.FeatureSwitchOverrideUtil
object TopicSocialProofParams {
object TopicTweetsSemanticCoreVersionId
extends FSBoundedParam[Long](
name = "topic_tweets_semantic_core_annotation_version_id",
default = 1433487161551032320L,
min = 0L,
max = Long.MaxValue
)
object TopicTweetsSemanticCoreVersionIdsSet
extends FSParam[Set[Long]](
name = "topic_tweets_semantic_core_annotation_version_id_allowed_set",
default = Set(TopicTweetsSemanticCoreVersionId.default))
/**
* Controls the Topic Social Proof cosine similarity threshold for the Topic Tweets.
*/
object TweetToTopicCosineSimilarityThreshold
extends FSBoundedParam[Double](
name = "topic_tweets_cosine_similarity_threshold_tsp",
default = 0.0,
min = 0.0,
max = 1.0
)
object EnablePersonalizedContextTopics // master feature switch to enable backfill
extends FSParam[Boolean](
name = "topic_tweets_personalized_contexts_enable_personalized_contexts",
default = false
)
object EnableYouMightLikeTopic
extends FSParam[Boolean](
name = "topic_tweets_personalized_contexts_enable_you_might_like",
default = false
)
object EnableRecentEngagementsTopic
extends FSParam[Boolean](
name = "topic_tweets_personalized_contexts_enable_recent_engagements",
default = false
)
object EnableTopicTweetHealthFilterPersonalizedContexts
extends FSParam[Boolean](
name = "topic_tweets_personalized_contexts_health_switch",
default = true
)
object EnableTweetToTopicScoreRanking
extends FSParam[Boolean](
name = "topic_tweets_enable_tweet_to_topic_score_ranking",
default = true
)
}
object FeatureSwitchConfig {
private val enumFeatureSwitchOverrides = FeatureSwitchOverrideUtil
.getEnumFSOverrides(
NullStatsReceiver,
Logger(getClass),
)
private val intFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides()
private val longFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBoundedLongFSOverrides(
TopicSocialProofParams.TopicTweetsSemanticCoreVersionId
)
private val doubleFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(
TopicSocialProofParams.TweetToTopicCosineSimilarityThreshold,
)
private val longSetFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getLongSetFSOverrides(
TopicSocialProofParams.TopicTweetsSemanticCoreVersionIdsSet,
)
private val booleanFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(
TopicSocialProofParams.EnablePersonalizedContextTopics,
TopicSocialProofParams.EnableYouMightLikeTopic,
TopicSocialProofParams.EnableRecentEngagementsTopic,
TopicSocialProofParams.EnableTopicTweetHealthFilterPersonalizedContexts,
TopicSocialProofParams.EnableTweetToTopicScoreRanking,
)
val config: BaseConfig = BaseConfigBuilder()
.set(enumFeatureSwitchOverrides: _*)
.set(intFeatureSwitchOverrides: _*)
.set(longFeatureSwitchOverrides: _*)
.set(doubleFeatureSwitchOverrides: _*)
.set(longSetFeatureSwitchOverrides: _*)
.set(booleanFeatureSwitchOverrides: _*)
.build()
}

View File

@ -0,0 +1,14 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala",
"stitch/stitch-storehaus",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/common",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/stores",
"topic-social-proof/server/src/main/thrift:thrift-scala",
"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting",
],
)

View File

@ -0,0 +1,587 @@
package com.twitter.tsp.handlers
import com.twitter.conversions.DurationOps._
import com.twitter.finagle.mux.ClientDiscardedRequestException
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.util.StatsUtil
import com.twitter.simclusters_v2.common.SemanticCoreEntityId
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.thriftscala.EmbeddingType
import com.twitter.simclusters_v2.thriftscala.ModelVersion
import com.twitter.strato.response.Err
import com.twitter.storehaus.ReadableStore
import com.twitter.timelines.configapi.Params
import com.twitter.topic_recos.common.Configs.ConsumerTopicEmbeddingType
import com.twitter.topic_recos.common.Configs.DefaultModelVersion
import com.twitter.topic_recos.common.Configs.ProducerTopicEmbeddingType
import com.twitter.topic_recos.common.Configs.TweetEmbeddingType
import com.twitter.topiclisting.TopicListingViewerContext
import com.twitter.topic_recos.common.LocaleUtil
import com.twitter.topiclisting.AnnotationRuleProvider
import com.twitter.tsp.common.DeciderConstants
import com.twitter.tsp.common.LoadShedder
import com.twitter.tsp.common.RecTargetFactory
import com.twitter.tsp.common.TopicSocialProofDecider
import com.twitter.tsp.common.TopicSocialProofParams
import com.twitter.tsp.stores.TopicSocialProofStore
import com.twitter.tsp.stores.TopicSocialProofStore.TopicSocialProof
import com.twitter.tsp.stores.UttTopicFilterStore
import com.twitter.tsp.stores.TopicTweetsCosineSimilarityAggregateStore.ScoreKey
import com.twitter.tsp.thriftscala.MetricTag
import com.twitter.tsp.thriftscala.TopicFollowType
import com.twitter.tsp.thriftscala.TopicListingSetting
import com.twitter.tsp.thriftscala.TopicSocialProofRequest
import com.twitter.tsp.thriftscala.TopicSocialProofResponse
import com.twitter.tsp.thriftscala.TopicWithScore
import com.twitter.tsp.thriftscala.TspTweetInfo
import com.twitter.tsp.utils.HealthSignalsUtils
import com.twitter.util.Future
import com.twitter.util.Timer
import com.twitter.util.Duration
import com.twitter.util.TimeoutException
import scala.util.Random
class TopicSocialProofHandler(
topicSocialProofStore: ReadableStore[TopicSocialProofStore.Query, Seq[TopicSocialProof]],
tweetInfoStore: ReadableStore[TweetId, TspTweetInfo],
uttTopicFilterStore: UttTopicFilterStore,
recTargetFactory: RecTargetFactory,
decider: TopicSocialProofDecider,
statsReceiver: StatsReceiver,
loadShedder: LoadShedder,
timer: Timer) {
import TopicSocialProofHandler._
def getTopicSocialProofResponse(
request: TopicSocialProofRequest
): Future[TopicSocialProofResponse] = {
val scopedStats = statsReceiver.scope(request.displayLocation.toString)
scopedStats.counter("fanoutRequests").incr(request.tweetIds.size)
scopedStats.stat("numTweetsPerRequest").add(request.tweetIds.size)
StatsUtil.trackBlockStats(scopedStats) {
recTargetFactory
.buildRecTopicSocialProofTarget(request).flatMap { target =>
val enableCosineSimilarityScoreCalculation =
decider.isAvailable(DeciderConstants.enableTopicSocialProofScore)
val semanticCoreVersionId =
target.params(TopicSocialProofParams.TopicTweetsSemanticCoreVersionId)
val semanticCoreVersionIdsSet =
target.params(TopicSocialProofParams.TopicTweetsSemanticCoreVersionIdsSet)
val allowListWithTopicFollowTypeFut = uttTopicFilterStore
.getAllowListTopicsForUser(
request.userId,
request.topicListingSetting,
TopicListingViewerContext
.fromThrift(request.context).copy(languageCode =
LocaleUtil.getStandardLanguageCode(request.context.languageCode)),
request.bypassModes.map(_.toSet)
).rescue {
case _ =>
scopedStats.counter("uttTopicFilterStoreFailure").incr()
Future.value(Map.empty[SemanticCoreEntityId, Option[TopicFollowType]])
}
val tweetInfoMapFut: Future[Map[TweetId, Option[TspTweetInfo]]] = Future
.collect(
tweetInfoStore.multiGet(request.tweetIds.toSet)
).raiseWithin(TweetInfoStoreTimeout)(timer).rescue {
case _: TimeoutException =>
scopedStats.counter("tweetInfoStoreTimeout").incr()
Future.value(Map.empty[TweetId, Option[TspTweetInfo]])
case _ =>
scopedStats.counter("tweetInfoStoreFailure").incr()
Future.value(Map.empty[TweetId, Option[TspTweetInfo]])
}
val definedTweetInfoMapFut =
keepTweetsWithTweetInfoAndLanguage(tweetInfoMapFut, request.displayLocation.toString)
Future
.join(definedTweetInfoMapFut, allowListWithTopicFollowTypeFut).map {
case (tweetInfoMap, allowListWithTopicFollowType) =>
val tweetIdsToQuery = tweetInfoMap.keys.toSet
val topicProofQueries =
tweetIdsToQuery.map { tweetId =>
TopicSocialProofStore.Query(
TopicSocialProofStore.CacheableQuery(
tweetId = tweetId,
tweetLanguage = LocaleUtil.getSupportedStandardLanguageCodeWithDefault(
tweetInfoMap.getOrElse(tweetId, None).flatMap {
_.language
}),
enableCosineSimilarityScoreCalculation =
enableCosineSimilarityScoreCalculation
),
allowedSemanticCoreVersionIds = semanticCoreVersionIdsSet
)
}
val topicSocialProofsFut: Future[Map[TweetId, Seq[TopicSocialProof]]] = {
Future
.collect(topicSocialProofStore.multiGet(topicProofQueries)).map(_.map {
case (query, results) =>
query.cacheableQuery.tweetId -> results.toSeq.flatten.filter(
_.semanticCoreVersionId == semanticCoreVersionId)
})
}.raiseWithin(TopicSocialProofStoreTimeout)(timer).rescue {
case _: TimeoutException =>
scopedStats.counter("topicSocialProofStoreTimeout").incr()
Future(Map.empty[TweetId, Seq[TopicSocialProof]])
case _ =>
scopedStats.counter("topicSocialProofStoreFailure").incr()
Future(Map.empty[TweetId, Seq[TopicSocialProof]])
}
val random = new Random(seed = request.userId.toInt)
topicSocialProofsFut.map { topicSocialProofs =>
val filteredTopicSocialProofs = filterByAllowedList(
topicSocialProofs,
request.topicListingSetting,
allowListWithTopicFollowType.keySet
)
val filteredTopicSocialProofsEmptyCount: Int =
filteredTopicSocialProofs.count {
case (_, topicSocialProofs: Seq[TopicSocialProof]) =>
topicSocialProofs.isEmpty
}
scopedStats
.counter("filteredTopicSocialProofsCount").incr(filteredTopicSocialProofs.size)
scopedStats
.counter("filteredTopicSocialProofsEmptyCount").incr(
filteredTopicSocialProofsEmptyCount)
if (isCrTopicTweets(request)) {
val socialProofs = filteredTopicSocialProofs.mapValues(_.flatMap { topicProof =>
val topicWithScores = buildTopicWithRandomScore(
topicProof,
allowListWithTopicFollowType,
random
)
topicWithScores
})
TopicSocialProofResponse(socialProofs)
} else {
val socialProofs = filteredTopicSocialProofs.mapValues(_.flatMap { topicProof =>
getTopicProofScore(
topicProof = topicProof,
allowListWithTopicFollowType = allowListWithTopicFollowType,
params = target.params,
random = random,
statsReceiver = statsReceiver
)
}.sortBy(-_.score).take(MaxCandidates))
val personalizedContextSocialProofs =
if (target.params(TopicSocialProofParams.EnablePersonalizedContextTopics)) {
val personalizedContextEligibility =
checkPersonalizedContextsEligibility(
target.params,
allowListWithTopicFollowType)
val filteredTweets =
filterPersonalizedContexts(socialProofs, tweetInfoMap, target.params)
backfillPersonalizedContexts(
allowListWithTopicFollowType,
filteredTweets,
request.tags.getOrElse(Map.empty),
personalizedContextEligibility)
} else {
Map.empty[TweetId, Seq[TopicWithScore]]
}
val mergedSocialProofs = socialProofs.map {
case (tweetId, proofs) =>
(
tweetId,
proofs
++ personalizedContextSocialProofs.getOrElse(tweetId, Seq.empty))
}
// Note that we will NOT filter out tweets with no TSP in either case
TopicSocialProofResponse(mergedSocialProofs)
}
}
}
}.flatten.raiseWithin(Timeout)(timer).rescue {
case _: ClientDiscardedRequestException =>
scopedStats.counter("ClientDiscardedRequestException").incr()
Future.value(DefaultResponse)
case err: Err if err.code == Err.Cancelled =>
scopedStats.counter("CancelledErr").incr()
Future.value(DefaultResponse)
case _ =>
scopedStats.counter("FailedRequests").incr()
Future.value(DefaultResponse)
}
}
}
/**
* Fetch the Score for each Topic Social Proof
*/
private def getTopicProofScore(
topicProof: TopicSocialProof,
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]],
params: Params,
random: Random,
statsReceiver: StatsReceiver
): Option[TopicWithScore] = {
val scopedStats = statsReceiver.scope("getTopicProofScores")
val enableTweetToTopicScoreRanking =
params(TopicSocialProofParams.EnableTweetToTopicScoreRanking)
val minTweetToTopicCosineSimilarityThreshold =
params(TopicSocialProofParams.TweetToTopicCosineSimilarityThreshold)
val topicWithScore =
if (enableTweetToTopicScoreRanking) {
scopedStats.counter("enableTweetToTopicScoreRanking").incr()
buildTopicWithValidScore(
topicProof,
TweetEmbeddingType,
Some(ConsumerTopicEmbeddingType),
Some(ProducerTopicEmbeddingType),
allowListWithTopicFollowType,
DefaultModelVersion,
minTweetToTopicCosineSimilarityThreshold
)
} else {
scopedStats.counter("buildTopicWithRandomScore").incr()
buildTopicWithRandomScore(
topicProof,
allowListWithTopicFollowType,
random
)
}
topicWithScore
}
private[handlers] def isCrTopicTweets(
request: TopicSocialProofRequest
): Boolean = {
// CrTopic (across a variety of DisplayLocations) is the only use case with TopicListingSetting.All
request.topicListingSetting == TopicListingSetting.All
}
/**
* Consolidate logics relevant to whether only quality topics should be enabled for Implicit Follows
*/
/***
* Consolidate logics relevant to whether Personalized Contexts backfilling should be enabled
*/
private[handlers] def checkPersonalizedContextsEligibility(
params: Params,
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]]
): PersonalizedContextEligibility = {
val scopedStats = statsReceiver.scope("checkPersonalizedContextsEligibility")
val isRecentFavInAllowlist = allowListWithTopicFollowType
.contains(AnnotationRuleProvider.recentFavTopicId)
val isRecentFavEligible =
isRecentFavInAllowlist && params(TopicSocialProofParams.EnableRecentEngagementsTopic)
if (isRecentFavEligible)
scopedStats.counter("isRecentFavEligible").incr()
val isRecentRetweetInAllowlist = allowListWithTopicFollowType
.contains(AnnotationRuleProvider.recentRetweetTopicId)
val isRecentRetweetEligible =
isRecentRetweetInAllowlist && params(TopicSocialProofParams.EnableRecentEngagementsTopic)
if (isRecentRetweetEligible)
scopedStats.counter("isRecentRetweetEligible").incr()
val isYMLInAllowlist = allowListWithTopicFollowType
.contains(AnnotationRuleProvider.youMightLikeTopicId)
val isYMLEligible =
isYMLInAllowlist && params(TopicSocialProofParams.EnableYouMightLikeTopic)
if (isYMLEligible)
scopedStats.counter("isYMLEligible").incr()
PersonalizedContextEligibility(isRecentFavEligible, isRecentRetweetEligible, isYMLEligible)
}
private[handlers] def filterPersonalizedContexts(
socialProofs: Map[TweetId, Seq[TopicWithScore]],
tweetInfoMap: Map[TweetId, Option[TspTweetInfo]],
params: Params
): Map[TweetId, Seq[TopicWithScore]] = {
val filters: Seq[(Option[TspTweetInfo], Params) => Boolean] = Seq(
healthSignalsFilter,
tweetLanguageFilter
)
applyFilters(socialProofs, tweetInfoMap, params, filters)
}
/** *
* filter tweets with None tweetInfo and undefined language
*/
private def keepTweetsWithTweetInfoAndLanguage(
tweetInfoMapFut: Future[Map[TweetId, Option[TspTweetInfo]]],
displayLocation: String
): Future[Map[TweetId, Option[TspTweetInfo]]] = {
val scopedStats = statsReceiver.scope(displayLocation)
tweetInfoMapFut.map { tweetInfoMap =>
val filteredTweetInfoMap = tweetInfoMap.filter {
case (_, optTweetInfo: Option[TspTweetInfo]) =>
if (optTweetInfo.isEmpty) {
scopedStats.counter("undefinedTweetInfoCount").incr()
}
optTweetInfo.exists { tweetInfo: TspTweetInfo =>
{
if (tweetInfo.language.isEmpty) {
scopedStats.counter("undefinedLanguageCount").incr()
}
tweetInfo.language.isDefined
}
}
}
val undefinedTweetInfoOrLangCount = tweetInfoMap.size - filteredTweetInfoMap.size
scopedStats.counter("undefinedTweetInfoOrLangCount").incr(undefinedTweetInfoOrLangCount)
scopedStats.counter("TweetInfoCount").incr(tweetInfoMap.size)
filteredTweetInfoMap
}
}
/***
* filter tweets with NO evergreen topic social proofs by their health signal scores & tweet languages
* i.e., tweets that are possible to be converted into Personalized Context topic tweets
* TBD: whether we are going to apply filters to all topic tweet candidates
*/
private def applyFilters(
socialProofs: Map[TweetId, Seq[TopicWithScore]],
tweetInfoMap: Map[TweetId, Option[TspTweetInfo]],
params: Params,
filters: Seq[(Option[TspTweetInfo], Params) => Boolean]
): Map[TweetId, Seq[TopicWithScore]] = {
socialProofs.collect {
case (tweetId, socialProofs) if socialProofs.nonEmpty || filters.forall { filter =>
filter(tweetInfoMap.getOrElse(tweetId, None), params)
} =>
tweetId -> socialProofs
}
}
private def healthSignalsFilter(
tweetInfoOpt: Option[TspTweetInfo],
params: Params
): Boolean = {
!params(
TopicSocialProofParams.EnableTopicTweetHealthFilterPersonalizedContexts) || HealthSignalsUtils
.isHealthyTweet(tweetInfoOpt)
}
private def tweetLanguageFilter(
tweetInfoOpt: Option[TspTweetInfo],
params: Params
): Boolean = {
PersonalizedContextTopicsAllowedLanguageSet
.contains(tweetInfoOpt.flatMap(_.language).getOrElse(LocaleUtil.DefaultLanguage))
}
private[handlers] def backfillPersonalizedContexts(
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]],
socialProofs: Map[TweetId, Seq[TopicWithScore]],
metricTagsMap: scala.collection.Map[TweetId, scala.collection.Set[MetricTag]],
personalizedContextEligibility: PersonalizedContextEligibility
): Map[TweetId, Seq[TopicWithScore]] = {
val scopedStats = statsReceiver.scope("backfillPersonalizedContexts")
socialProofs.map {
case (tweetId, topicWithScores) =>
if (topicWithScores.nonEmpty) {
tweetId -> Seq.empty
} else {
val metricTagContainsTweetFav = metricTagsMap
.getOrElse(tweetId, Set.empty[MetricTag]).contains(MetricTag.TweetFavorite)
val backfillRecentFav =
personalizedContextEligibility.isRecentFavEligible && metricTagContainsTweetFav
if (metricTagContainsTweetFav)
scopedStats.counter("MetricTag.TweetFavorite").incr()
if (backfillRecentFav)
scopedStats.counter("backfillRecentFav").incr()
val metricTagContainsRetweet = metricTagsMap
.getOrElse(tweetId, Set.empty[MetricTag]).contains(MetricTag.Retweet)
val backfillRecentRetweet =
personalizedContextEligibility.isRecentRetweetEligible && metricTagContainsRetweet
if (metricTagContainsRetweet)
scopedStats.counter("MetricTag.Retweet").incr()
if (backfillRecentRetweet)
scopedStats.counter("backfillRecentRetweet").incr()
val metricTagContainsRecentSearches = metricTagsMap
.getOrElse(tweetId, Set.empty[MetricTag]).contains(
MetricTag.InterestsRankerRecentSearches)
val backfillYML = personalizedContextEligibility.isYMLEligible
if (backfillYML)
scopedStats.counter("backfillYML").incr()
tweetId -> buildBackfillTopics(
allowListWithTopicFollowType,
backfillRecentFav,
backfillRecentRetweet,
backfillYML)
}
}
}
private def buildBackfillTopics(
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]],
backfillRecentFav: Boolean,
backfillRecentRetweet: Boolean,
backfillYML: Boolean
): Seq[TopicWithScore] = {
Seq(
if (backfillRecentFav) {
Some(
TopicWithScore(
topicId = AnnotationRuleProvider.recentFavTopicId,
score = 1.0,
topicFollowType = allowListWithTopicFollowType
.getOrElse(AnnotationRuleProvider.recentFavTopicId, None)
))
} else { None },
if (backfillRecentRetweet) {
Some(
TopicWithScore(
topicId = AnnotationRuleProvider.recentRetweetTopicId,
score = 1.0,
topicFollowType = allowListWithTopicFollowType
.getOrElse(AnnotationRuleProvider.recentRetweetTopicId, None)
))
} else { None },
if (backfillYML) {
Some(
TopicWithScore(
topicId = AnnotationRuleProvider.youMightLikeTopicId,
score = 1.0,
topicFollowType = allowListWithTopicFollowType
.getOrElse(AnnotationRuleProvider.youMightLikeTopicId, None)
))
} else { None }
).flatten
}
def toReadableStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse] = {
new ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse] {
override def get(k: TopicSocialProofRequest): Future[Option[TopicSocialProofResponse]] = {
val displayLocation = k.displayLocation.toString
loadShedder(displayLocation) {
getTopicSocialProofResponse(k).map(Some(_))
}.rescue {
case LoadShedder.LoadSheddingException =>
statsReceiver.scope(displayLocation).counter("LoadSheddingException").incr()
Future.None
case _ =>
statsReceiver.scope(displayLocation).counter("Exception").incr()
Future.None
}
}
}
}
}
object TopicSocialProofHandler {
private val MaxCandidates = 10
// Currently we do hardcode for the language check of PersonalizedContexts Topics
private val PersonalizedContextTopicsAllowedLanguageSet: Set[String] =
Set("pt", "ko", "es", "ja", "tr", "id", "en", "hi", "ar", "fr", "ru")
private val Timeout: Duration = 200.milliseconds
private val TopicSocialProofStoreTimeout: Duration = 40.milliseconds
private val TweetInfoStoreTimeout: Duration = 60.milliseconds
private val DefaultResponse: TopicSocialProofResponse = TopicSocialProofResponse(Map.empty)
case class PersonalizedContextEligibility(
isRecentFavEligible: Boolean,
isRecentRetweetEligible: Boolean,
isYMLEligible: Boolean)
/**
* Calculate the Topic Scores for each (tweet, topic), filter out topic proofs whose scores do not
* pass the minimum threshold
*/
private[handlers] def buildTopicWithValidScore(
topicProof: TopicSocialProof,
tweetEmbeddingType: EmbeddingType,
maybeConsumerEmbeddingType: Option[EmbeddingType],
maybeProducerEmbeddingType: Option[EmbeddingType],
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]],
simClustersModelVersion: ModelVersion,
minTweetToTopicCosineSimilarityThreshold: Double
): Option[TopicWithScore] = {
val consumerScore = maybeConsumerEmbeddingType
.flatMap { consumerEmbeddingType =>
topicProof.scores.get(
ScoreKey(consumerEmbeddingType, tweetEmbeddingType, simClustersModelVersion))
}.getOrElse(0.0)
val producerScore = maybeProducerEmbeddingType
.flatMap { producerEmbeddingType =>
topicProof.scores.get(
ScoreKey(producerEmbeddingType, tweetEmbeddingType, simClustersModelVersion))
}.getOrElse(0.0)
val combinedScore = consumerScore + producerScore
if (combinedScore > minTweetToTopicCosineSimilarityThreshold || topicProof.ignoreSimClusterFiltering) {
Some(
TopicWithScore(
topicId = topicProof.topicId.entityId,
score = combinedScore,
topicFollowType =
allowListWithTopicFollowType.getOrElse(topicProof.topicId.entityId, None)))
} else {
None
}
}
private[handlers] def buildTopicWithRandomScore(
topicSocialProof: TopicSocialProof,
allowListWithTopicFollowType: Map[SemanticCoreEntityId, Option[TopicFollowType]],
random: Random
): Option[TopicWithScore] = {
Some(
TopicWithScore(
topicId = topicSocialProof.topicId.entityId,
score = random.nextDouble(),
topicFollowType =
allowListWithTopicFollowType.getOrElse(topicSocialProof.topicId.entityId, None)
))
}
/**
* Filter all the non-qualified Topic Social Proof
*/
private[handlers] def filterByAllowedList(
topicProofs: Map[TweetId, Seq[TopicSocialProof]],
setting: TopicListingSetting,
allowList: Set[SemanticCoreEntityId]
): Map[TweetId, Seq[TopicSocialProof]] = {
setting match {
case TopicListingSetting.All =>
// Return all the topics
topicProofs
case _ =>
topicProofs.mapValues(
_.filter(topicProof => allowList.contains(topicProof.topicId.entityId)))
}
}
}

View File

@ -0,0 +1,40 @@
package com.twitter.tsp.handlers
import com.twitter.inject.utils.Handler
import com.twitter.topiclisting.FollowableTopicProductId
import com.twitter.topiclisting.ProductId
import com.twitter.topiclisting.TopicListingViewerContext
import com.twitter.topiclisting.utt.UttLocalization
import com.twitter.util.logging.Logging
import javax.inject.Inject
import javax.inject.Singleton
/** *
* We configure Warmer to help warm up the cache hit rate under `CachedUttClient/get_utt_taxonomy/cache_hit_rate`
* In uttLocalization.getRecommendableTopics, we fetch all topics exist in UTT, and yet the process
* is in fact fetching the complete UTT tree struct (by calling getUttChildren recursively), which could take 1 sec
* Once we have the topics, we stored them in in-memory cache, and the cache hit rate is > 99%
*
*/
@Singleton
class UttChildrenWarmupHandler @Inject() (uttLocalization: UttLocalization)
extends Handler
with Logging {
/** Executes the function of this handler. * */
override def handle(): Unit = {
uttLocalization
.getRecommendableTopics(
productId = ProductId.Followable,
viewerContext = TopicListingViewerContext(languageCode = Some("en")),
enableInternationalTopics = true,
followableTopicProductId = FollowableTopicProductId.AllFollowable
)
.onSuccess { result =>
logger.info(s"successfully warmed up UttChildren. TopicId length = ${result.size}")
}
.onFailure { throwable =>
logger.info(s"failed to warm up UttChildren. Throwable = ${throwable}")
}
}
}

View File

@ -0,0 +1,30 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"3rdparty/jvm/com/twitter/bijection:scrooge",
"3rdparty/jvm/com/twitter/storehaus:memcache",
"escherbird/src/scala/com/twitter/escherbird/util/uttclient",
"escherbird/src/thrift/com/twitter/escherbird/utt:strato-columns-scala",
"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication",
"finatra-internal/mtls-thriftmux/src/main/scala",
"finatra/inject/inject-core/src/main/scala",
"finatra/inject/inject-thrift-client",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato",
"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common",
"src/scala/com/twitter/storehaus_internal/memcache",
"src/scala/com/twitter/storehaus_internal/util",
"src/thrift/com/twitter/gizmoduck:thrift-scala",
"src/thrift/com/twitter/gizmoduck:user-thrift-scala",
"stitch/stitch-storehaus",
"stitch/stitch-tweetypie/src/main/scala",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/common",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/stores",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/utils",
"topic-social-proof/server/src/main/thrift:thrift-scala",
"topiclisting/common/src/main/scala/com/twitter/topiclisting/clients",
"topiclisting/topiclisting-utt/src/main/scala/com/twitter/topiclisting/utt",
],
)

View File

@ -0,0 +1,35 @@
package com.twitter.tsp.modules
import com.google.inject.Module
import com.twitter.finagle.ThriftMux
import com.twitter.finagle.mtls.authentication.ServiceIdentifier
import com.twitter.finagle.mtls.client.MtlsStackClient._
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.finagle.thrift.ClientId
import com.twitter.finatra.mtls.thriftmux.modules.MtlsClient
import com.twitter.gizmoduck.thriftscala.UserService
import com.twitter.inject.Injector
import com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule
object GizmoduckUserModule
extends ThriftMethodBuilderClientModule[
UserService.ServicePerEndpoint,
UserService.MethodPerEndpoint
]
with MtlsClient {
override val label: String = "gizmoduck"
override val dest: String = "/s/gizmoduck/gizmoduck"
override val modules: Seq[Module] = Seq(TSPClientIdModule)
override def configureThriftMuxClient(
injector: Injector,
client: ThriftMux.Client
): ThriftMux.Client = {
super
.configureThriftMuxClient(injector, client)
.withMutualTls(injector.instance[ServiceIdentifier])
.withClientId(injector.instance[ClientId])
.withStatsReceiver(injector.instance[StatsReceiver].scope("giz"))
}
}

View File

@ -0,0 +1,47 @@
package com.twitter.tsp.modules
import com.google.inject.Module
import com.google.inject.Provides
import com.google.inject.Singleton
import com.twitter.app.Flag
import com.twitter.bijection.scrooge.BinaryScalaCodec
import com.twitter.conversions.DurationOps._
import com.twitter.finagle.memcached.{Client => MemClient}
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.hermit.store.common.ObservedMemcachedReadableStore
import com.twitter.inject.TwitterModule
import com.twitter.simclusters_v2.thriftscala.Score
import com.twitter.simclusters_v2.thriftscala.ScoreId
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.{Client => StratoClient}
import com.twitter.tsp.stores.RepresentationScorerStore
object RepresentationScorerStoreModule extends TwitterModule {
override def modules: Seq[Module] = Seq(UnifiedCacheClient)
private val tspRepresentationScoringColumnPath: Flag[String] = flag[String](
name = "tsp.representationScoringColumnPath",
default = "recommendations/representation_scorer/score",
help = "Strato column path for Representation Scorer Store"
)
@Provides
@Singleton
def providesRepresentationScorerStore(
statsReceiver: StatsReceiver,
stratoClient: StratoClient,
tspUnifiedCacheClient: MemClient
): ReadableStore[ScoreId, Score] = {
val underlyingStore =
RepresentationScorerStore(stratoClient, tspRepresentationScoringColumnPath(), statsReceiver)
ObservedMemcachedReadableStore.fromCacheClient(
backingStore = underlyingStore,
cacheClient = tspUnifiedCacheClient,
ttl = 2.hours
)(
valueInjection = BinaryScalaCodec(Score),
statsReceiver = statsReceiver.scope("RepresentationScorerStore"),
keyToString = { k: ScoreId => s"rsx/$k" }
)
}
}

View File

@ -0,0 +1,14 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.twitter.finagle.thrift.ClientId
import com.twitter.inject.TwitterModule
import javax.inject.Singleton
object TSPClientIdModule extends TwitterModule {
private val clientIdFlag = flag("thrift.clientId", "topic-social-proof.prod", "Thrift client id")
@Provides
@Singleton
def providesClientId: ClientId = ClientId(clientIdFlag())
}

View File

@ -0,0 +1,17 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.inject.TwitterModule
import com.twitter.topiclisting.TopicListing
import com.twitter.topiclisting.TopicListingBuilder
import javax.inject.Singleton
object TopicListingModule extends TwitterModule {
@Provides
@Singleton
def providesTopicListing(statsReceiver: StatsReceiver): TopicListing = {
new TopicListingBuilder(statsReceiver.scope(namespace = "TopicListingBuilder")).build
}
}

View File

@ -0,0 +1,68 @@
package com.twitter.tsp.modules
import com.google.inject.Module
import com.google.inject.Provides
import com.google.inject.Singleton
import com.twitter.conversions.DurationOps._
import com.twitter.finagle.memcached.{Client => MemClient}
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.hermit.store.common.ObservedCachedReadableStore
import com.twitter.hermit.store.common.ObservedMemcachedReadableStore
import com.twitter.hermit.store.common.ObservedReadableStore
import com.twitter.inject.TwitterModule
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.thriftscala.Score
import com.twitter.simclusters_v2.thriftscala.ScoreId
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.{Client => StratoClient}
import com.twitter.tsp.stores.SemanticCoreAnnotationStore
import com.twitter.tsp.stores.TopicSocialProofStore
import com.twitter.tsp.stores.TopicSocialProofStore.TopicSocialProof
import com.twitter.tsp.utils.LZ4Injection
import com.twitter.tsp.utils.SeqObjectInjection
object TopicSocialProofStoreModule extends TwitterModule {
override def modules: Seq[Module] = Seq(UnifiedCacheClient)
@Provides
@Singleton
def providesTopicSocialProofStore(
representationScorerStore: ReadableStore[ScoreId, Score],
statsReceiver: StatsReceiver,
stratoClient: StratoClient,
tspUnifiedCacheClient: MemClient,
): ReadableStore[TopicSocialProofStore.Query, Seq[TopicSocialProof]] = {
val semanticCoreAnnotationStore: ReadableStore[TweetId, Seq[
SemanticCoreAnnotationStore.TopicAnnotation
]] = ObservedReadableStore(
SemanticCoreAnnotationStore(SemanticCoreAnnotationStore.getStratoStore(stratoClient))
)(statsReceiver.scope("SemanticCoreAnnotationStore"))
val underlyingStore = TopicSocialProofStore(
representationScorerStore,
semanticCoreAnnotationStore
)(statsReceiver.scope("TopicSocialProofStore"))
val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(
backingStore = underlyingStore,
cacheClient = tspUnifiedCacheClient,
ttl = 15.minutes,
asyncUpdate = true
)(
valueInjection = LZ4Injection.compose(SeqObjectInjection[TopicSocialProof]()),
statsReceiver = statsReceiver.scope("memCachedTopicSocialProofStore"),
keyToString = { k: TopicSocialProofStore.Query => s"tsps/${k.cacheableQuery}" }
)
val inMemoryCachedStore =
ObservedCachedReadableStore.from[TopicSocialProofStore.Query, Seq[TopicSocialProof]](
memcachedStore,
ttl = 10.minutes,
maxKeys = 16777215, // ~ avg 160B, < 3000MB
cacheName = "topic_social_proof_cache",
windowSize = 10000L
)(statsReceiver.scope("InMemoryCachedTopicSocialProofStore"))
inMemoryCachedStore
}
}

View File

@ -0,0 +1,26 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.google.inject.Singleton
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.inject.TwitterModule
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.thriftscala.Score
import com.twitter.simclusters_v2.thriftscala.ScoreId
import com.twitter.simclusters_v2.thriftscala.TopicId
import com.twitter.storehaus.ReadableStore
import com.twitter.tsp.stores.TopicTweetsCosineSimilarityAggregateStore
import com.twitter.tsp.stores.TopicTweetsCosineSimilarityAggregateStore.ScoreKey
object TopicTweetCosineSimilarityAggregateStoreModule extends TwitterModule {
@Provides
@Singleton
def providesTopicTweetCosineSimilarityAggregateStore(
representationScorerStore: ReadableStore[ScoreId, Score],
statsReceiver: StatsReceiver,
): ReadableStore[(TopicId, TweetId, Seq[ScoreKey]), Map[ScoreKey, Double]] = {
TopicTweetsCosineSimilarityAggregateStore(representationScorerStore)(
statsReceiver.scope("topicTweetsCosineSimilarityAggregateStore"))
}
}

View File

@ -0,0 +1,130 @@
package com.twitter.tsp.modules
import com.google.inject.Module
import com.google.inject.Provides
import com.google.inject.Singleton
import com.twitter.bijection.scrooge.BinaryScalaCodec
import com.twitter.conversions.DurationOps._
import com.twitter.finagle.memcached.{Client => MemClient}
import com.twitter.finagle.mtls.authentication.ServiceIdentifier
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.store.health.TweetHealthModelStore
import com.twitter.frigate.common.store.health.TweetHealthModelStore.TweetHealthModelStoreConfig
import com.twitter.frigate.common.store.health.UserHealthModelStore
import com.twitter.frigate.common.store.interests.UserId
import com.twitter.frigate.thriftscala.TweetHealthScores
import com.twitter.frigate.thriftscala.UserAgathaScores
import com.twitter.hermit.store.common.DeciderableReadableStore
import com.twitter.hermit.store.common.ObservedCachedReadableStore
import com.twitter.hermit.store.common.ObservedMemcachedReadableStore
import com.twitter.inject.TwitterModule
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.stitch.tweetypie.TweetyPie
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.{Client => StratoClient}
import com.twitter.tsp.common.DeciderKey
import com.twitter.tsp.common.TopicSocialProofDecider
import com.twitter.tsp.stores.TweetInfoStore
import com.twitter.tsp.stores.TweetyPieFieldsStore
import com.twitter.tweetypie.thriftscala.TweetService
import com.twitter.tsp.thriftscala.TspTweetInfo
import com.twitter.util.JavaTimer
import com.twitter.util.Timer
object TweetInfoStoreModule extends TwitterModule {
override def modules: Seq[Module] = Seq(UnifiedCacheClient)
implicit val timer: Timer = new JavaTimer(true)
@Provides
@Singleton
def providesTweetInfoStore(
decider: TopicSocialProofDecider,
serviceIdentifier: ServiceIdentifier,
statsReceiver: StatsReceiver,
stratoClient: StratoClient,
tspUnifiedCacheClient: MemClient,
tweetyPieService: TweetService.MethodPerEndpoint
): ReadableStore[TweetId, TspTweetInfo] = {
val tweetHealthModelStore: ReadableStore[TweetId, TweetHealthScores] = {
val underlyingStore = TweetHealthModelStore.buildReadableStore(
stratoClient,
Some(
TweetHealthModelStoreConfig(
enablePBlock = true,
enableToxicity = true,
enablePSpammy = true,
enablePReported = true,
enableSpammyTweetContent = true,
enablePNegMultimodal = false))
)(statsReceiver.scope("UnderlyingTweetHealthModelStore"))
DeciderableReadableStore(
ObservedMemcachedReadableStore.fromCacheClient(
backingStore = underlyingStore,
cacheClient = tspUnifiedCacheClient,
ttl = 2.hours
)(
valueInjection = BinaryScalaCodec(TweetHealthScores),
statsReceiver = statsReceiver.scope("TweetHealthModelStore"),
keyToString = { k: TweetId => s"tHMS/$k" }
),
decider.deciderGateBuilder.idGate(DeciderKey.enableHealthSignalsScoreDeciderKey),
statsReceiver.scope("TweetHealthModelStore")
)
}
val userHealthModelStore: ReadableStore[UserId, UserAgathaScores] = {
val underlyingStore =
UserHealthModelStore.buildReadableStore(stratoClient)(
statsReceiver.scope("UnderlyingUserHealthModelStore"))
DeciderableReadableStore(
ObservedMemcachedReadableStore.fromCacheClient(
backingStore = underlyingStore,
cacheClient = tspUnifiedCacheClient,
ttl = 18.hours
)(
valueInjection = BinaryScalaCodec(UserAgathaScores),
statsReceiver = statsReceiver.scope("UserHealthModelStore"),
keyToString = { k: UserId => s"uHMS/$k" }
),
decider.deciderGateBuilder.idGate(DeciderKey.enableUserAgathaScoreDeciderKey),
statsReceiver.scope("UserHealthModelStore")
)
}
val tweetInfoStore: ReadableStore[TweetId, TspTweetInfo] = {
val underlyingStore = TweetInfoStore(
TweetyPieFieldsStore.getStoreFromTweetyPie(TweetyPie(tweetyPieService, statsReceiver)),
tweetHealthModelStore: ReadableStore[TweetId, TweetHealthScores],
userHealthModelStore: ReadableStore[UserId, UserAgathaScores],
timer: Timer
)(statsReceiver.scope("tweetInfoStore"))
val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(
backingStore = underlyingStore,
cacheClient = tspUnifiedCacheClient,
ttl = 15.minutes,
// Hydrating tweetInfo is now a required step for all candidates,
// hence we needed to tune these thresholds.
asyncUpdate = serviceIdentifier.environment == "prod"
)(
valueInjection = BinaryScalaCodec(TspTweetInfo),
statsReceiver = statsReceiver.scope("memCachedTweetInfoStore"),
keyToString = { k: TweetId => s"tIS/$k" }
)
val inMemoryStore = ObservedCachedReadableStore.from(
memcachedStore,
ttl = 15.minutes,
maxKeys = 8388607, // Check TweetInfo definition. size~92b. Around 736 MB
windowSize = 10000L,
cacheName = "tweet_info_cache",
maxMultiGetSize = 20
)(statsReceiver.scope("inMemoryCachedTweetInfoStore"))
inMemoryStore
}
tweetInfoStore
}
}

View File

@ -0,0 +1,63 @@
package com.twitter.tsp
package modules
import com.google.inject.Module
import com.google.inject.Provides
import com.twitter.conversions.DurationOps.richDurationFromInt
import com.twitter.finagle.ThriftMux
import com.twitter.finagle.mtls.authentication.ServiceIdentifier
import com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax
import com.twitter.finagle.mux.ClientDiscardedRequestException
import com.twitter.finagle.service.ReqRep
import com.twitter.finagle.service.ResponseClass
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.finagle.thrift.ClientId
import com.twitter.inject.Injector
import com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule
import com.twitter.tweetypie.thriftscala.TweetService
import com.twitter.util.Duration
import com.twitter.util.Throw
import com.twitter.stitch.tweetypie.{TweetyPie => STweetyPie}
import com.twitter.finatra.mtls.thriftmux.modules.MtlsClient
import javax.inject.Singleton
object TweetyPieClientModule
extends ThriftMethodBuilderClientModule[
TweetService.ServicePerEndpoint,
TweetService.MethodPerEndpoint
]
with MtlsClient {
override val label = "tweetypie"
override val dest = "/s/tweetypie/tweetypie"
override val requestTimeout: Duration = 450.milliseconds
override val modules: Seq[Module] = Seq(TSPClientIdModule)
// We bump the success rate from the default of 0.8 to 0.9 since we're dropping the
// consecutive failures part of the default policy.
override def configureThriftMuxClient(
injector: Injector,
client: ThriftMux.Client
): ThriftMux.Client =
super
.configureThriftMuxClient(injector, client)
.withMutualTls(injector.instance[ServiceIdentifier])
.withStatsReceiver(injector.instance[StatsReceiver].scope("clnt"))
.withClientId(injector.instance[ClientId])
.withResponseClassifier {
case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable
}
.withSessionQualifier
.successRateFailureAccrual(successRate = 0.9, window = 30.seconds)
.withResponseClassifier {
case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable
}
@Provides
@Singleton
def providesTweetyPie(
tweetyPieService: TweetService.MethodPerEndpoint
): STweetyPie = {
STweetyPie(tweetyPieService)
}
}

View File

@ -0,0 +1,33 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.google.inject.Singleton
import com.twitter.app.Flag
import com.twitter.finagle.memcached.Client
import com.twitter.finagle.mtls.authentication.ServiceIdentifier
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.inject.TwitterModule
import com.twitter.storehaus_internal.memcache.MemcacheStore
import com.twitter.storehaus_internal.util.ClientName
import com.twitter.storehaus_internal.util.ZkEndPoint
object UnifiedCacheClient extends TwitterModule {
val tspUnifiedCacheDest: Flag[String] = flag[String](
name = "tsp.unifiedCacheDest",
default = "/srv#/prod/local/cache/topic_social_proof_unified",
help = "Wily path to topic social proof unified cache"
)
@Provides
@Singleton
def provideUnifiedCacheClient(
serviceIdentifier: ServiceIdentifier,
statsReceiver: StatsReceiver,
): Client =
MemcacheStore.memcachedClient(
name = ClientName("topic-social-proof-unified-memcache"),
dest = ZkEndPoint(tspUnifiedCacheDest()),
statsReceiver = statsReceiver.scope("cache_client"),
serviceIdentifier = serviceIdentifier
)
}

View File

@ -0,0 +1,41 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.twitter.escherbird.util.uttclient.CacheConfigV2
import com.twitter.escherbird.util.uttclient.CachedUttClientV2
import com.twitter.escherbird.util.uttclient.UttClientCacheConfigsV2
import com.twitter.escherbird.utt.strato.thriftscala.Environment
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.inject.TwitterModule
import com.twitter.strato.client.Client
import com.twitter.topiclisting.clients.utt.UttClient
import javax.inject.Singleton
object UttClientModule extends TwitterModule {
@Provides
@Singleton
def providesUttClient(
stratoClient: Client,
statsReceiver: StatsReceiver
): UttClient = {
// Save 2 ^ 18 UTTs. Promising 100% cache rate
lazy val defaultCacheConfigV2: CacheConfigV2 = CacheConfigV2(262143)
lazy val uttClientCacheConfigsV2: UttClientCacheConfigsV2 = UttClientCacheConfigsV2(
getTaxonomyConfig = defaultCacheConfigV2,
getUttTaxonomyConfig = defaultCacheConfigV2,
getLeafIds = defaultCacheConfigV2,
getLeafUttEntities = defaultCacheConfigV2
)
// CachedUttClient to use StratoClient
lazy val cachedUttClientV2: CachedUttClientV2 = new CachedUttClientV2(
stratoClient = stratoClient,
env = Environment.Prod,
cacheConfigs = uttClientCacheConfigsV2,
statsReceiver = statsReceiver.scope("CachedUttClient")
)
new UttClient(cachedUttClientV2, statsReceiver)
}
}

View File

@ -0,0 +1,27 @@
package com.twitter.tsp.modules
import com.google.inject.Provides
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.inject.TwitterModule
import com.twitter.topiclisting.TopicListing
import com.twitter.topiclisting.clients.utt.UttClient
import com.twitter.topiclisting.utt.UttLocalization
import com.twitter.topiclisting.utt.UttLocalizationImpl
import javax.inject.Singleton
object UttLocalizationModule extends TwitterModule {
@Provides
@Singleton
def providesUttLocalization(
topicListing: TopicListing,
uttClient: UttClient,
statsReceiver: StatsReceiver
): UttLocalization = {
new UttLocalizationImpl(
topicListing,
uttClient,
statsReceiver
)
}
}

View File

@ -0,0 +1,23 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"3rdparty/jvm/javax/inject:javax.inject",
"abdecider/src/main/scala",
"content-recommender/thrift/src/main/thrift:thrift-scala",
"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common",
"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/gizmoduck",
"src/scala/com/twitter/topic_recos/stores",
"src/thrift/com/twitter/gizmoduck:thrift-scala",
"src/thrift/com/twitter/gizmoduck:user-thrift-scala",
"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala",
"stitch/stitch-storehaus",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/common",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/handlers",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/modules",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/stores",
"topic-social-proof/server/src/main/thrift:thrift-scala",
],
)

View File

@ -0,0 +1,182 @@
package com.twitter.tsp.service
import com.twitter.abdecider.ABDeciderFactory
import com.twitter.abdecider.LoggingABDecider
import com.twitter.tsp.thriftscala.TspTweetInfo
import com.twitter.discovery.common.configapi.FeatureContextBuilder
import com.twitter.finagle.mtls.authentication.ServiceIdentifier
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.gizmoduck.thriftscala.LookupContext
import com.twitter.gizmoduck.thriftscala.QueryFields
import com.twitter.gizmoduck.thriftscala.User
import com.twitter.gizmoduck.thriftscala.UserService
import com.twitter.hermit.store.gizmoduck.GizmoduckUserStore
import com.twitter.logging.Logger
import com.twitter.simclusters_v2.common.SemanticCoreEntityId
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.common.UserId
import com.twitter.spam.rtf.thriftscala.SafetyLevel
import com.twitter.stitch.storehaus.StitchOfReadableStore
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.{Client => StratoClient}
import com.twitter.timelines.configapi
import com.twitter.timelines.configapi.CompositeConfig
import com.twitter.tsp.common.FeatureSwitchConfig
import com.twitter.tsp.common.FeatureSwitchesBuilder
import com.twitter.tsp.common.LoadShedder
import com.twitter.tsp.common.ParamsBuilder
import com.twitter.tsp.common.RecTargetFactory
import com.twitter.tsp.common.TopicSocialProofDecider
import com.twitter.tsp.handlers.TopicSocialProofHandler
import com.twitter.tsp.stores.LocalizedUttRecommendableTopicsStore
import com.twitter.tsp.stores.LocalizedUttTopicNameRequest
import com.twitter.tsp.stores.TopicResponses
import com.twitter.tsp.stores.TopicSocialProofStore
import com.twitter.tsp.stores.TopicSocialProofStore.TopicSocialProof
import com.twitter.tsp.stores.TopicStore
import com.twitter.tsp.stores.UttTopicFilterStore
import com.twitter.tsp.thriftscala.TopicSocialProofRequest
import com.twitter.tsp.thriftscala.TopicSocialProofResponse
import com.twitter.util.JavaTimer
import com.twitter.util.Timer
import javax.inject.Inject
import javax.inject.Singleton
import com.twitter.topiclisting.TopicListing
import com.twitter.topiclisting.utt.UttLocalization
@Singleton
class TopicSocialProofService @Inject() (
topicSocialProofStore: ReadableStore[TopicSocialProofStore.Query, Seq[TopicSocialProof]],
tweetInfoStore: ReadableStore[TweetId, TspTweetInfo],
serviceIdentifier: ServiceIdentifier,
stratoClient: StratoClient,
gizmoduck: UserService.MethodPerEndpoint,
topicListing: TopicListing,
uttLocalization: UttLocalization,
decider: TopicSocialProofDecider,
loadShedder: LoadShedder,
stats: StatsReceiver) {
import TopicSocialProofService._
private val statsReceiver = stats.scope("topic-social-proof-management")
private val isProd: Boolean = serviceIdentifier.environment == "prod"
private val optOutStratoStorePath: String =
if (isProd) "interests/optOutInterests" else "interests/staging/optOutInterests"
private val notInterestedInStorePath: String =
if (isProd) "interests/notInterestedTopicsGetter"
else "interests/staging/notInterestedTopicsGetter"
private val userOptOutTopicsStore: ReadableStore[UserId, TopicResponses] =
TopicStore.userOptOutTopicStore(stratoClient, optOutStratoStorePath)(
statsReceiver.scope("ints_interests_opt_out_store"))
private val explicitFollowingTopicsStore: ReadableStore[UserId, TopicResponses] =
TopicStore.explicitFollowingTopicStore(stratoClient)(
statsReceiver.scope("ints_explicit_following_interests_store"))
private val userNotInterestedInTopicsStore: ReadableStore[UserId, TopicResponses] =
TopicStore.notInterestedInTopicsStore(stratoClient, notInterestedInStorePath)(
statsReceiver.scope("ints_not_interested_in_store"))
private lazy val localizedUttRecommendableTopicsStore: ReadableStore[
LocalizedUttTopicNameRequest,
Set[
SemanticCoreEntityId
]
] = new LocalizedUttRecommendableTopicsStore(uttLocalization)
implicit val timer: Timer = new JavaTimer(true)
private lazy val uttTopicFilterStore = new UttTopicFilterStore(
topicListing = topicListing,
userOptOutTopicsStore = userOptOutTopicsStore,
explicitFollowingTopicsStore = explicitFollowingTopicsStore,
notInterestedTopicsStore = userNotInterestedInTopicsStore,
localizedUttRecommendableTopicsStore = localizedUttRecommendableTopicsStore,
timer = timer,
stats = statsReceiver.scope("UttTopicFilterStore")
)
private lazy val scribeLogger: Option[Logger] = Some(Logger.get("client_event"))
private lazy val abDecider: LoggingABDecider =
ABDeciderFactory(
abDeciderYmlPath = configRepoDirectory + "/abdecider/abdecider.yml",
scribeLogger = scribeLogger,
decider = None,
environment = Some("production"),
).buildWithLogging()
private val builder: FeatureSwitchesBuilder = FeatureSwitchesBuilder(
statsReceiver = statsReceiver.scope("featureswitches-v2"),
abDecider = abDecider,
featuresDirectory = "features/topic-social-proof/main",
configRepoDirectory = configRepoDirectory,
addServiceDetailsFromAurora = !serviceIdentifier.isLocal,
fastRefresh = !isProd
)
private lazy val overridesConfig: configapi.Config = {
new CompositeConfig(
Seq(
FeatureSwitchConfig.config
)
)
}
private val featureContextBuilder: FeatureContextBuilder = FeatureContextBuilder(builder.build())
private val paramsBuilder: ParamsBuilder = ParamsBuilder(
featureContextBuilder,
abDecider,
overridesConfig,
statsReceiver.scope("params")
)
private val userStore: ReadableStore[UserId, User] = {
val queryFields: Set[QueryFields] = Set(
QueryFields.Profile,
QueryFields.Account,
QueryFields.Roles,
QueryFields.Discoverability,
QueryFields.Safety,
QueryFields.Takedowns
)
val context: LookupContext = LookupContext(safetyLevel = Some(SafetyLevel.Recommendations))
GizmoduckUserStore(
client = gizmoduck,
queryFields = queryFields,
context = context,
statsReceiver = statsReceiver.scope("gizmoduck")
)
}
private val recTargetFactory: RecTargetFactory = RecTargetFactory(
abDecider,
userStore,
paramsBuilder,
statsReceiver
)
private val topicSocialProofHandler =
new TopicSocialProofHandler(
topicSocialProofStore,
tweetInfoStore,
uttTopicFilterStore,
recTargetFactory,
decider,
statsReceiver.scope("TopicSocialProofHandler"),
loadShedder,
timer)
val topicSocialProofHandlerStoreStitch: TopicSocialProofRequest => com.twitter.stitch.Stitch[
TopicSocialProofResponse
] = StitchOfReadableStore(topicSocialProofHandler.toReadableStore)
}
object TopicSocialProofService {
private val configRepoDirectory = "/usr/local/config"
}

View File

@ -0,0 +1,32 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"3rdparty/jvm/com/twitter/storehaus:core",
"content-recommender/thrift/src/main/thrift:thrift-scala",
"escherbird/src/thrift/com/twitter/escherbird/topicannotation:topicannotation-thrift-scala",
"frigate/frigate-common:util",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/health",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato",
"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common",
"mediaservices/commons/src/main/thrift:thrift-scala",
"src/scala/com/twitter/simclusters_v2/common",
"src/scala/com/twitter/simclusters_v2/score",
"src/scala/com/twitter/topic_recos/common",
"src/scala/com/twitter/topic_recos/stores",
"src/thrift/com/twitter/frigate:frigate-common-thrift-scala",
"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala",
"src/thrift/com/twitter/spam/rtf:safety-level-scala",
"src/thrift/com/twitter/tweetypie:service-scala",
"src/thrift/com/twitter/tweetypie:tweet-scala",
"stitch/stitch-storehaus",
"stitch/stitch-tweetypie/src/main/scala",
"strato/src/main/scala/com/twitter/strato/client",
"topic-social-proof/server/src/main/scala/com/twitter/tsp/utils",
"topic-social-proof/server/src/main/thrift:thrift-scala",
"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting",
],
)

View File

@ -0,0 +1,30 @@
package com.twitter.tsp.stores
import com.twitter.storehaus.ReadableStore
import com.twitter.topiclisting.FollowableTopicProductId
import com.twitter.topiclisting.ProductId
import com.twitter.topiclisting.SemanticCoreEntityId
import com.twitter.topiclisting.TopicListingViewerContext
import com.twitter.topiclisting.utt.UttLocalization
import com.twitter.util.Future
case class LocalizedUttTopicNameRequest(
productId: ProductId.Value,
viewerContext: TopicListingViewerContext,
enableInternationalTopics: Boolean)
class LocalizedUttRecommendableTopicsStore(uttLocalization: UttLocalization)
extends ReadableStore[LocalizedUttTopicNameRequest, Set[SemanticCoreEntityId]] {
override def get(
request: LocalizedUttTopicNameRequest
): Future[Option[Set[SemanticCoreEntityId]]] = {
uttLocalization
.getRecommendableTopics(
productId = request.productId,
viewerContext = request.viewerContext,
enableInternationalTopics = request.enableInternationalTopics,
followableTopicProductId = FollowableTopicProductId.AllFollowable
).map { response => Some(response) }
}
}

View File

@ -0,0 +1,31 @@
package com.twitter.tsp.stores
import com.twitter.contentrecommender.thriftscala.ScoringResponse
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.store.strato.StratoFetchableStore
import com.twitter.hermit.store.common.ObservedReadableStore
import com.twitter.simclusters_v2.thriftscala.Score
import com.twitter.simclusters_v2.thriftscala.ScoreId
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.Client
import com.twitter.strato.thrift.ScroogeConvImplicits._
import com.twitter.tsp.utils.ReadableStoreWithMapOptionValues
object RepresentationScorerStore {
def apply(
stratoClient: Client,
scoringColumnPath: String,
stats: StatsReceiver
): ReadableStore[ScoreId, Score] = {
val stratoFetchableStore = StratoFetchableStore
.withUnitView[ScoreId, ScoringResponse](stratoClient, scoringColumnPath)
val enrichedStore = new ReadableStoreWithMapOptionValues[ScoreId, ScoringResponse, Score](
stratoFetchableStore).mapOptionValues(_.score)
ObservedReadableStore(
enrichedStore
)(stats.scope("representation_scorer_store"))
}
}

View File

@ -0,0 +1,64 @@
package com.twitter.tsp.stores
import com.twitter.escherbird.topicannotation.strato.thriftscala.TopicAnnotationValue
import com.twitter.escherbird.topicannotation.strato.thriftscala.TopicAnnotationView
import com.twitter.frigate.common.store.strato.StratoFetchableStore
import com.twitter.simclusters_v2.common.TopicId
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.Client
import com.twitter.strato.thrift.ScroogeConvImplicits._
import com.twitter.util.Future
/**
* This is copied from `src/scala/com/twitter/topic_recos/stores/SemanticCoreAnnotationStore.scala`
* Unfortunately their version assumes (incorrectly) that there is no View which causes warnings.
* While these warnings may not cause any problems in practice, better safe than sorry.
*/
object SemanticCoreAnnotationStore {
private val column = "semanticCore/topicannotation/topicAnnotation.Tweet"
def getStratoStore(stratoClient: Client): ReadableStore[TweetId, TopicAnnotationValue] = {
StratoFetchableStore
.withView[TweetId, TopicAnnotationView, TopicAnnotationValue](
stratoClient,
column,
TopicAnnotationView())
}
case class TopicAnnotation(
topicId: TopicId,
ignoreSimClustersFilter: Boolean,
modelVersionId: Long)
}
/**
* Given a tweet Id, return the list of annotations defined by the TSIG team.
*/
case class SemanticCoreAnnotationStore(stratoStore: ReadableStore[TweetId, TopicAnnotationValue])
extends ReadableStore[TweetId, Seq[SemanticCoreAnnotationStore.TopicAnnotation]] {
import SemanticCoreAnnotationStore._
override def multiGet[K1 <: TweetId](
ks: Set[K1]
): Map[K1, Future[Option[Seq[TopicAnnotation]]]] = {
stratoStore
.multiGet(ks)
.mapValues(_.map(_.map { topicAnnotationValue =>
topicAnnotationValue.annotationsPerModel match {
case Some(annotationWithVersions) =>
annotationWithVersions.flatMap { annotations =>
annotations.annotations.map { annotation =>
TopicAnnotation(
annotation.entityId,
annotation.ignoreQualityFilter.getOrElse(false),
annotations.modelVersionId
)
}
}
case _ =>
Nil
}
}))
}
}

View File

@ -0,0 +1,127 @@
package com.twitter.tsp.stores
import com.twitter.tsp.stores.TopicTweetsCosineSimilarityAggregateStore.ScoreKey
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.util.StatsUtil
import com.twitter.simclusters_v2.thriftscala._
import com.twitter.storehaus.ReadableStore
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.tsp.stores.SemanticCoreAnnotationStore._
import com.twitter.tsp.stores.TopicSocialProofStore.TopicSocialProof
import com.twitter.util.Future
/**
* Provides a session-less Topic Social Proof information which doesn't rely on any User Info.
* This store is used by MemCache and In-Memory cache to achieve a higher performance.
* One Consumer embedding and Producer embedding are used to calculate raw score.
*/
case class TopicSocialProofStore(
representationScorerStore: ReadableStore[ScoreId, Score],
semanticCoreAnnotationStore: ReadableStore[TweetId, Seq[TopicAnnotation]]
)(
statsReceiver: StatsReceiver)
extends ReadableStore[TopicSocialProofStore.Query, Seq[TopicSocialProof]] {
import TopicSocialProofStore._
// Fetches the tweet's topic annotations from SemanticCore's Annotation API
override def get(query: TopicSocialProofStore.Query): Future[Option[Seq[TopicSocialProof]]] = {
StatsUtil.trackOptionStats(statsReceiver) {
for {
annotations <-
StatsUtil.trackItemsStats(statsReceiver.scope("semanticCoreAnnotationStore")) {
semanticCoreAnnotationStore.get(query.cacheableQuery.tweetId).map(_.getOrElse(Nil))
}
filteredAnnotations = filterAnnotationsByAllowList(annotations, query)
scoredTopics <-
StatsUtil.trackItemMapStats(statsReceiver.scope("scoreTopicTweetsTweetLanguage")) {
// de-dup identical topicIds
val uniqueTopicIds = filteredAnnotations.map { annotation =>
TopicId(annotation.topicId, Some(query.cacheableQuery.tweetLanguage), country = None)
}.toSet
if (query.cacheableQuery.enableCosineSimilarityScoreCalculation) {
scoreTopicTweets(query.cacheableQuery.tweetId, uniqueTopicIds)
} else {
Future.value(uniqueTopicIds.map(id => id -> Map.empty[ScoreKey, Double]).toMap)
}
}
} yield {
if (scoredTopics.nonEmpty) {
val versionedTopicProofs = filteredAnnotations.map { annotation =>
val topicId =
TopicId(annotation.topicId, Some(query.cacheableQuery.tweetLanguage), country = None)
TopicSocialProof(
topicId,
scores = scoredTopics.getOrElse(topicId, Map.empty),
annotation.ignoreSimClustersFilter,
annotation.modelVersionId
)
}
Some(versionedTopicProofs)
} else {
None
}
}
}
}
/***
* When the allowList is not empty (e.g., TSP handler call, CrTopic handler call),
* the filter will be enabled and we will only keep annotations that have versionIds existing
* in the input allowedSemanticCoreVersionIds set.
* But when the allowList is empty (e.g., some debugger calls),
* we will not filter anything and pass.
* We limit the number of versionIds to be K = MaxNumberVersionIds
*/
private def filterAnnotationsByAllowList(
annotations: Seq[TopicAnnotation],
query: TopicSocialProofStore.Query
): Seq[TopicAnnotation] = {
val trimmedVersionIds = query.allowedSemanticCoreVersionIds.take(MaxNumberVersionIds)
annotations.filter { annotation =>
trimmedVersionIds.isEmpty || trimmedVersionIds.contains(annotation.modelVersionId)
}
}
private def scoreTopicTweets(
tweetId: TweetId,
topicIds: Set[TopicId]
): Future[Map[TopicId, Map[ScoreKey, Double]]] = {
Future.collect {
topicIds.map { topicId =>
val scoresFut = TopicTweetsCosineSimilarityAggregateStore.getRawScoresMap(
topicId,
tweetId,
TopicTweetsCosineSimilarityAggregateStore.DefaultScoreKeys,
representationScorerStore
)
topicId -> scoresFut
}.toMap
}
}
}
object TopicSocialProofStore {
private val MaxNumberVersionIds = 9
case class Query(
cacheableQuery: CacheableQuery,
allowedSemanticCoreVersionIds: Set[Long] = Set.empty) // overridden by FS
case class CacheableQuery(
tweetId: TweetId,
tweetLanguage: String,
enableCosineSimilarityScoreCalculation: Boolean = true)
case class TopicSocialProof(
topicId: TopicId,
scores: Map[ScoreKey, Double],
ignoreSimClusterFiltering: Boolean,
semanticCoreVersionId: Long)
}

View File

@ -0,0 +1,135 @@
package com.twitter.tsp.stores
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.store.InterestedInInterestsFetchKey
import com.twitter.frigate.common.store.strato.StratoFetchableStore
import com.twitter.hermit.store.common.ObservedReadableStore
import com.twitter.interests.thriftscala.InterestId
import com.twitter.interests.thriftscala.InterestLabel
import com.twitter.interests.thriftscala.InterestRelationship
import com.twitter.interests.thriftscala.InterestRelationshipV1
import com.twitter.interests.thriftscala.InterestedInInterestLookupContext
import com.twitter.interests.thriftscala.InterestedInInterestModel
import com.twitter.interests.thriftscala.OptOutInterestLookupContext
import com.twitter.interests.thriftscala.UserInterest
import com.twitter.interests.thriftscala.UserInterestData
import com.twitter.interests.thriftscala.UserInterestsResponse
import com.twitter.simclusters_v2.common.UserId
import com.twitter.storehaus.ReadableStore
import com.twitter.strato.client.Client
import com.twitter.strato.thrift.ScroogeConvImplicits._
case class TopicResponse(
entityId: Long,
interestedInData: Seq[InterestedInInterestModel],
scoreOverride: Option[Double] = None,
notInterestedInTimestamp: Option[Long] = None,
topicFollowTimestamp: Option[Long] = None)
case class TopicResponses(responses: Seq[TopicResponse])
object TopicStore {
private val InterestedInInterestsColumn = "interests/interestedInInterests"
private lazy val ExplicitInterestsContext: InterestedInInterestLookupContext =
InterestedInInterestLookupContext(
explicitContext = None,
inferredContext = None,
disableImplicit = Some(true)
)
private def userInterestsResponseToTopicResponse(
userInterestsResponse: UserInterestsResponse
): TopicResponses = {
val responses = userInterestsResponse.interests.interests.toSeq.flatMap { userInterests =>
userInterests.collect {
case UserInterest(
InterestId.SemanticCore(semanticCoreEntity),
Some(UserInterestData.InterestedIn(data))) =>
val topicFollowingTimestampOpt = data.collect {
case InterestedInInterestModel.ExplicitModel(
InterestRelationship.V1(interestRelationshipV1)) =>
interestRelationshipV1.timestampMs
}.lastOption
TopicResponse(semanticCoreEntity.id, data, None, None, topicFollowingTimestampOpt)
}
}
TopicResponses(responses)
}
def explicitFollowingTopicStore(
stratoClient: Client
)(
implicit statsReceiver: StatsReceiver
): ReadableStore[UserId, TopicResponses] = {
val stratoStore =
StratoFetchableStore
.withUnitView[InterestedInInterestsFetchKey, UserInterestsResponse](
stratoClient,
InterestedInInterestsColumn)
.composeKeyMapping[UserId](uid =>
InterestedInInterestsFetchKey(
userId = uid,
labels = None,
lookupContext = Some(ExplicitInterestsContext)
))
.mapValues(userInterestsResponseToTopicResponse)
ObservedReadableStore(stratoStore)
}
def userOptOutTopicStore(
stratoClient: Client,
optOutStratoStorePath: String
)(
implicit statsReceiver: StatsReceiver
): ReadableStore[UserId, TopicResponses] = {
val stratoStore =
StratoFetchableStore
.withUnitView[
(Long, Option[Seq[InterestLabel]], Option[OptOutInterestLookupContext]),
UserInterestsResponse](stratoClient, optOutStratoStorePath)
.composeKeyMapping[UserId](uid => (uid, None, None))
.mapValues { userInterestsResponse =>
val responses = userInterestsResponse.interests.interests.toSeq.flatMap { userInterests =>
userInterests.collect {
case UserInterest(
InterestId.SemanticCore(semanticCoreEntity),
Some(UserInterestData.InterestedIn(data))) =>
TopicResponse(semanticCoreEntity.id, data, None)
}
}
TopicResponses(responses)
}
ObservedReadableStore(stratoStore)
}
def notInterestedInTopicsStore(
stratoClient: Client,
notInterestedInStorePath: String
)(
implicit statsReceiver: StatsReceiver
): ReadableStore[UserId, TopicResponses] = {
val stratoStore =
StratoFetchableStore
.withUnitView[Long, Seq[UserInterest]](stratoClient, notInterestedInStorePath)
.composeKeyMapping[UserId](identity)
.mapValues { notInterestedInInterests =>
val responses = notInterestedInInterests.collect {
case UserInterest(
InterestId.SemanticCore(semanticCoreEntity),
Some(UserInterestData.NotInterested(notInterestedInData))) =>
val notInterestedInTimestampOpt = notInterestedInData.collect {
case InterestRelationship.V1(interestRelationshipV1: InterestRelationshipV1) =>
interestRelationshipV1.timestampMs
}.lastOption
TopicResponse(semanticCoreEntity.id, Seq.empty, None, notInterestedInTimestampOpt)
}
TopicResponses(responses)
}
ObservedReadableStore(stratoStore)
}
}

View File

@ -0,0 +1,99 @@
package com.twitter.tsp.stores
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.thriftscala.EmbeddingType
import com.twitter.simclusters_v2.thriftscala.InternalId
import com.twitter.simclusters_v2.thriftscala.ModelVersion
import com.twitter.simclusters_v2.thriftscala.ScoreInternalId
import com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm
import com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId
import com.twitter.simclusters_v2.thriftscala.{
SimClustersEmbeddingPairScoreId => ThriftSimClustersEmbeddingPairScoreId
}
import com.twitter.simclusters_v2.thriftscala.TopicId
import com.twitter.simclusters_v2.thriftscala.{Score => ThriftScore}
import com.twitter.simclusters_v2.thriftscala.{ScoreId => ThriftScoreId}
import com.twitter.storehaus.ReadableStore
import com.twitter.topic_recos.common._
import com.twitter.topic_recos.common.Configs.DefaultModelVersion
import com.twitter.tsp.stores.TopicTweetsCosineSimilarityAggregateStore.ScoreKey
import com.twitter.util.Future
object TopicTweetsCosineSimilarityAggregateStore {
val TopicEmbeddingTypes: Seq[EmbeddingType] =
Seq(
EmbeddingType.FavTfgTopic,
EmbeddingType.LogFavBasedKgoApeTopic
)
// Add the new embedding types if want to test the new Tweet embedding performance.
val TweetEmbeddingTypes: Seq[EmbeddingType] = Seq(EmbeddingType.LogFavBasedTweet)
val ModelVersions: Seq[ModelVersion] =
Seq(DefaultModelVersion)
val DefaultScoreKeys: Seq[ScoreKey] = {
for {
modelVersion <- ModelVersions
topicEmbeddingType <- TopicEmbeddingTypes
tweetEmbeddingType <- TweetEmbeddingTypes
} yield {
ScoreKey(
topicEmbeddingType = topicEmbeddingType,
tweetEmbeddingType = tweetEmbeddingType,
modelVersion = modelVersion
)
}
}
case class ScoreKey(
topicEmbeddingType: EmbeddingType,
tweetEmbeddingType: EmbeddingType,
modelVersion: ModelVersion)
def getRawScoresMap(
topicId: TopicId,
tweetId: TweetId,
scoreKeys: Seq[ScoreKey],
representationScorerStore: ReadableStore[ThriftScoreId, ThriftScore]
): Future[Map[ScoreKey, Double]] = {
val scoresMapFut = scoreKeys.map { key =>
val scoreInternalId = ScoreInternalId.SimClustersEmbeddingPairScoreId(
ThriftSimClustersEmbeddingPairScoreId(
buildTopicEmbedding(topicId, key.topicEmbeddingType, key.modelVersion),
SimClustersEmbeddingId(
key.tweetEmbeddingType,
key.modelVersion,
InternalId.TweetId(tweetId))
))
val scoreFut = representationScorerStore
.get(
ThriftScoreId(
algorithm = ScoringAlgorithm.PairEmbeddingCosineSimilarity, // Hard code as cosine sim
internalId = scoreInternalId
))
key -> scoreFut
}.toMap
Future
.collect(scoresMapFut).map(_.collect {
case (key, Some(ThriftScore(score))) =>
(key, score)
})
}
}
case class TopicTweetsCosineSimilarityAggregateStore(
representationScorerStore: ReadableStore[ThriftScoreId, ThriftScore]
)(
statsReceiver: StatsReceiver)
extends ReadableStore[(TopicId, TweetId, Seq[ScoreKey]), Map[ScoreKey, Double]] {
import TopicTweetsCosineSimilarityAggregateStore._
override def get(k: (TopicId, TweetId, Seq[ScoreKey])): Future[Option[Map[ScoreKey, Double]]] = {
statsReceiver.counter("topicTweetsCosineSimilariltyAggregateStore").incr()
getRawScoresMap(k._1, k._2, k._3, representationScorerStore).map(Some(_))
}
}

View File

@ -0,0 +1,230 @@
package com.twitter.tsp.stores
import com.twitter.conversions.DurationOps._
import com.twitter.tsp.thriftscala.TspTweetInfo
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.thriftscala.TweetHealthScores
import com.twitter.frigate.thriftscala.UserAgathaScores
import com.twitter.logging.Logger
import com.twitter.mediaservices.commons.thriftscala.MediaCategory
import com.twitter.mediaservices.commons.tweetmedia.thriftscala.MediaInfo
import com.twitter.mediaservices.commons.tweetmedia.thriftscala.MediaSizeType
import com.twitter.simclusters_v2.common.TweetId
import com.twitter.simclusters_v2.common.UserId
import com.twitter.spam.rtf.thriftscala.SafetyLevel
import com.twitter.stitch.Stitch
import com.twitter.stitch.storehaus.ReadableStoreOfStitch
import com.twitter.stitch.tweetypie.TweetyPie
import com.twitter.stitch.tweetypie.TweetyPie.TweetyPieException
import com.twitter.storehaus.ReadableStore
import com.twitter.topiclisting.AnnotationRuleProvider
import com.twitter.tsp.utils.HealthSignalsUtils
import com.twitter.tweetypie.thriftscala.TweetInclude
import com.twitter.tweetypie.thriftscala.{Tweet => TTweet}
import com.twitter.tweetypie.thriftscala._
import com.twitter.util.Duration
import com.twitter.util.Future
import com.twitter.util.TimeoutException
import com.twitter.util.Timer
object TweetyPieFieldsStore {
// Tweet fields options. Only fields specified here will be hydrated in the tweet
private val CoreTweetFields: Set[TweetInclude] = Set[TweetInclude](
TweetInclude.TweetFieldId(TTweet.IdField.id),
TweetInclude.TweetFieldId(TTweet.CoreDataField.id), // needed for the authorId
TweetInclude.TweetFieldId(TTweet.LanguageField.id),
TweetInclude.CountsFieldId(StatusCounts.FavoriteCountField.id),
TweetInclude.CountsFieldId(StatusCounts.RetweetCountField.id),
TweetInclude.TweetFieldId(TTweet.QuotedTweetField.id),
TweetInclude.TweetFieldId(TTweet.MediaKeysField.id),
TweetInclude.TweetFieldId(TTweet.EscherbirdEntityAnnotationsField.id),
TweetInclude.TweetFieldId(TTweet.MediaField.id),
TweetInclude.TweetFieldId(TTweet.UrlsField.id)
)
private val gtfo: GetTweetFieldsOptions = GetTweetFieldsOptions(
tweetIncludes = CoreTweetFields,
safetyLevel = Some(SafetyLevel.Recommendations)
)
def getStoreFromTweetyPie(
tweetyPie: TweetyPie,
convertExceptionsToNotFound: Boolean = true
): ReadableStore[Long, GetTweetFieldsResult] = {
val log = Logger("TweetyPieFieldsStore")
ReadableStoreOfStitch { tweetId: Long =>
tweetyPie
.getTweetFields(tweetId, options = gtfo)
.rescue {
case ex: TweetyPieException if convertExceptionsToNotFound =>
log.error(ex, s"Error while hitting tweetypie ${ex.result}")
Stitch.NotFound
}
}
}
}
object TweetInfoStore {
case class IsPassTweetHealthFilters(tweetStrictest: Option[Boolean])
case class IsPassAgathaHealthFilters(agathaStrictest: Option[Boolean])
private val HealthStoreTimeout: Duration = 40.milliseconds
private val isPassTweetHealthFilters: IsPassTweetHealthFilters = IsPassTweetHealthFilters(None)
private val isPassAgathaHealthFilters: IsPassAgathaHealthFilters = IsPassAgathaHealthFilters(None)
}
case class TweetInfoStore(
tweetFieldsStore: ReadableStore[TweetId, GetTweetFieldsResult],
tweetHealthModelStore: ReadableStore[TweetId, TweetHealthScores],
userHealthModelStore: ReadableStore[UserId, UserAgathaScores],
timer: Timer
)(
statsReceiver: StatsReceiver)
extends ReadableStore[TweetId, TspTweetInfo] {
import TweetInfoStore._
private[this] def toTweetInfo(
tweetFieldsResult: GetTweetFieldsResult
): Future[Option[TspTweetInfo]] = {
tweetFieldsResult.tweetResult match {
case result: TweetFieldsResultState.Found if result.found.suppressReason.isEmpty =>
val tweet = result.found.tweet
val authorIdOpt = tweet.coreData.map(_.userId)
val favCountOpt = tweet.counts.flatMap(_.favoriteCount)
val languageOpt = tweet.language.map(_.language)
val hasImageOpt =
tweet.mediaKeys.map(_.map(_.mediaCategory).exists(_ == MediaCategory.TweetImage))
val hasGifOpt =
tweet.mediaKeys.map(_.map(_.mediaCategory).exists(_ == MediaCategory.TweetGif))
val isNsfwAuthorOpt = Some(
tweet.coreData.exists(_.nsfwUser) || tweet.coreData.exists(_.nsfwAdmin))
val isTweetReplyOpt = tweet.coreData.map(_.reply.isDefined)
val hasMultipleMediaOpt =
tweet.mediaKeys.map(_.map(_.mediaCategory).size > 1)
val isKGODenylist = Some(
tweet.escherbirdEntityAnnotations
.exists(_.entityAnnotations.exists(AnnotationRuleProvider.isSuppressedTopicsDenylist)))
val isNullcastOpt = tweet.coreData.map(_.nullcast) // These are Ads. go/nullcast
val videoDurationOpt = tweet.media.flatMap(_.flatMap {
_.mediaInfo match {
case Some(MediaInfo.VideoInfo(info)) =>
Some((info.durationMillis + 999) / 1000) // video playtime always round up
case _ => None
}
}.headOption)
// There many different types of videos. To be robust to new types being added, we just use
// the videoDurationOpt to keep track of whether the item has a video or not.
val hasVideo = videoDurationOpt.isDefined
val mediaDimensionsOpt =
tweet.media.flatMap(_.headOption.flatMap(
_.sizes.find(_.sizeType == MediaSizeType.Orig).map(size => (size.width, size.height))))
val mediaWidth = mediaDimensionsOpt.map(_._1).getOrElse(1)
val mediaHeight = mediaDimensionsOpt.map(_._2).getOrElse(1)
// high resolution media's width is always greater than 480px and height is always greater than 480px
val isHighMediaResolution = mediaHeight > 480 && mediaWidth > 480
val isVerticalAspectRatio = mediaHeight >= mediaWidth && mediaWidth > 1
val hasUrlOpt = tweet.urls.map(_.nonEmpty)
(authorIdOpt, favCountOpt) match {
case (Some(authorId), Some(favCount)) =>
hydrateHealthScores(tweet.id, authorId).map {
case (isPassAgathaHealthFilters, isPassTweetHealthFilters) =>
Some(
TspTweetInfo(
authorId = authorId,
favCount = favCount,
language = languageOpt,
hasImage = hasImageOpt,
hasVideo = Some(hasVideo),
hasGif = hasGifOpt,
isNsfwAuthor = isNsfwAuthorOpt,
isKGODenylist = isKGODenylist,
isNullcast = isNullcastOpt,
videoDurationSeconds = videoDurationOpt,
isHighMediaResolution = Some(isHighMediaResolution),
isVerticalAspectRatio = Some(isVerticalAspectRatio),
isPassAgathaHealthFilterStrictest = isPassAgathaHealthFilters.agathaStrictest,
isPassTweetHealthFilterStrictest = isPassTweetHealthFilters.tweetStrictest,
isReply = isTweetReplyOpt,
hasMultipleMedia = hasMultipleMediaOpt,
hasUrl = hasUrlOpt
))
}
case _ =>
statsReceiver.counter("missingFields").incr()
Future.None // These values should always exist.
}
case _: TweetFieldsResultState.NotFound =>
statsReceiver.counter("notFound").incr()
Future.None
case _: TweetFieldsResultState.Failed =>
statsReceiver.counter("failed").incr()
Future.None
case _: TweetFieldsResultState.Filtered =>
statsReceiver.counter("filtered").incr()
Future.None
case _ =>
statsReceiver.counter("unknown").incr()
Future.None
}
}
private[this] def hydrateHealthScores(
tweetId: TweetId,
authorId: Long
): Future[(IsPassAgathaHealthFilters, IsPassTweetHealthFilters)] = {
Future
.join(
tweetHealthModelStore
.multiGet(Set(tweetId))(tweetId),
userHealthModelStore
.multiGet(Set(authorId))(authorId)
).map {
case (tweetHealthScoresOpt, userAgathaScoresOpt) =>
// This stats help us understand empty rate for AgathaCalibratedNsfw / NsfwTextUserScore
statsReceiver.counter("totalCountAgathaScore").incr()
if (userAgathaScoresOpt.getOrElse(UserAgathaScores()).agathaCalibratedNsfw.isEmpty)
statsReceiver.counter("emptyCountAgathaCalibratedNsfw").incr()
if (userAgathaScoresOpt.getOrElse(UserAgathaScores()).nsfwTextUserScore.isEmpty)
statsReceiver.counter("emptyCountNsfwTextUserScore").incr()
val isPassAgathaHealthFilters = IsPassAgathaHealthFilters(
agathaStrictest =
Some(HealthSignalsUtils.isTweetAgathaModelQualified(userAgathaScoresOpt)),
)
val isPassTweetHealthFilters = IsPassTweetHealthFilters(
tweetStrictest =
Some(HealthSignalsUtils.isTweetHealthModelQualified(tweetHealthScoresOpt))
)
(isPassAgathaHealthFilters, isPassTweetHealthFilters)
}.raiseWithin(HealthStoreTimeout)(timer).rescue {
case _: TimeoutException =>
statsReceiver.counter("hydrateHealthScoreTimeout").incr()
Future.value((isPassAgathaHealthFilters, isPassTweetHealthFilters))
case _ =>
statsReceiver.counter("hydrateHealthScoreFailure").incr()
Future.value((isPassAgathaHealthFilters, isPassTweetHealthFilters))
}
}
override def multiGet[K1 <: TweetId](ks: Set[K1]): Map[K1, Future[Option[TspTweetInfo]]] = {
statsReceiver.counter("tweetFieldsStore").incr(ks.size)
tweetFieldsStore
.multiGet(ks).mapValues(_.flatMap { _.map { v => toTweetInfo(v) }.getOrElse(Future.None) })
}
}

View File

@ -0,0 +1,248 @@
package com.twitter.tsp.stores
import com.twitter.conversions.DurationOps._
import com.twitter.finagle.FailureFlags.flagsOf
import com.twitter.finagle.mux.ClientDiscardedRequestException
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.frigate.common.store.interests
import com.twitter.simclusters_v2.common.UserId
import com.twitter.storehaus.ReadableStore
import com.twitter.topiclisting.ProductId
import com.twitter.topiclisting.TopicListing
import com.twitter.topiclisting.TopicListingViewerContext
import com.twitter.topiclisting.{SemanticCoreEntityId => ScEntityId}
import com.twitter.tsp.thriftscala.TopicFollowType
import com.twitter.tsp.thriftscala.TopicListingSetting
import com.twitter.tsp.thriftscala.TopicSocialProofFilteringBypassMode
import com.twitter.util.Duration
import com.twitter.util.Future
import com.twitter.util.TimeoutException
import com.twitter.util.Timer
class UttTopicFilterStore(
topicListing: TopicListing,
userOptOutTopicsStore: ReadableStore[interests.UserId, TopicResponses],
explicitFollowingTopicsStore: ReadableStore[interests.UserId, TopicResponses],
notInterestedTopicsStore: ReadableStore[interests.UserId, TopicResponses],
localizedUttRecommendableTopicsStore: ReadableStore[LocalizedUttTopicNameRequest, Set[Long]],
timer: Timer,
stats: StatsReceiver) {
import UttTopicFilterStore._
// Set of blacklisted SemanticCore IDs that are paused.
private[this] def getPausedTopics(topicCtx: TopicListingViewerContext): Set[ScEntityId] = {
topicListing.getPausedTopics(topicCtx)
}
private[this] def getOptOutTopics(userId: Long): Future[Set[ScEntityId]] = {
stats.counter("getOptOutTopicsCount").incr()
userOptOutTopicsStore
.get(userId).map { responseOpt =>
responseOpt
.map { responses => responses.responses.map(_.entityId) }.getOrElse(Seq.empty).toSet
}.raiseWithin(DefaultOptOutTimeout)(timer).rescue {
case err: TimeoutException =>
stats.counter("getOptOutTopicsTimeout").incr()
Future.exception(err)
case err: ClientDiscardedRequestException
if flagsOf(err).contains("interrupted") && flagsOf(err)
.contains("ignorable") =>
stats.counter("getOptOutTopicsDiscardedBackupRequest").incr()
Future.exception(err)
case err =>
stats.counter("getOptOutTopicsFailure").incr()
Future.exception(err)
}
}
private[this] def getNotInterestedIn(userId: Long): Future[Set[ScEntityId]] = {
stats.counter("getNotInterestedInCount").incr()
notInterestedTopicsStore
.get(userId).map { responseOpt =>
responseOpt
.map { responses => responses.responses.map(_.entityId) }.getOrElse(Seq.empty).toSet
}.raiseWithin(DefaultNotInterestedInTimeout)(timer).rescue {
case err: TimeoutException =>
stats.counter("getNotInterestedInTimeout").incr()
Future.exception(err)
case err: ClientDiscardedRequestException
if flagsOf(err).contains("interrupted") && flagsOf(err)
.contains("ignorable") =>
stats.counter("getNotInterestedInDiscardedBackupRequest").incr()
Future.exception(err)
case err =>
stats.counter("getNotInterestedInFailure").incr()
Future.exception(err)
}
}
private[this] def getFollowedTopics(userId: Long): Future[Set[TopicResponse]] = {
stats.counter("getFollowedTopicsCount").incr()
explicitFollowingTopicsStore
.get(userId).map { responseOpt =>
responseOpt.map(_.responses.toSet).getOrElse(Set.empty)
}.raiseWithin(DefaultInterestedInTimeout)(timer).rescue {
case _: TimeoutException =>
stats.counter("getFollowedTopicsTimeout").incr()
Future(Set.empty)
case _ =>
stats.counter("getFollowedTopicsFailure").incr()
Future(Set.empty)
}
}
private[this] def getFollowedTopicIds(userId: Long): Future[Set[ScEntityId]] = {
getFollowedTopics(userId: Long).map(_.map(_.entityId))
}
private[this] def getWhitelistTopicIds(
normalizedContext: TopicListingViewerContext,
enableInternationalTopics: Boolean
): Future[Set[ScEntityId]] = {
stats.counter("getWhitelistTopicIdsCount").incr()
val uttRequest = LocalizedUttTopicNameRequest(
productId = ProductId.Followable,
viewerContext = normalizedContext,
enableInternationalTopics = enableInternationalTopics
)
localizedUttRecommendableTopicsStore
.get(uttRequest).map { response =>
response.getOrElse(Set.empty)
}.rescue {
case _ =>
stats.counter("getWhitelistTopicIdsFailure").incr()
Future(Set.empty)
}
}
private[this] def getDenyListTopicIdsForUser(
userId: UserId,
topicListingSetting: TopicListingSetting,
context: TopicListingViewerContext,
bypassModes: Option[Set[TopicSocialProofFilteringBypassMode]]
): Future[Set[ScEntityId]] = {
val denyListTopicIdsFuture = topicListingSetting match {
case TopicListingSetting.ImplicitFollow =>
getFollowedTopicIds(userId)
case _ =>
Future(Set.empty[ScEntityId])
}
// we don't filter opt-out topics for implicit follow topic listing setting
val optOutTopicIdsFuture = topicListingSetting match {
case TopicListingSetting.ImplicitFollow => Future(Set.empty[ScEntityId])
case _ => getOptOutTopics(userId)
}
val notInterestedTopicIdsFuture =
if (bypassModes.exists(_.contains(TopicSocialProofFilteringBypassMode.NotInterested))) {
Future(Set.empty[ScEntityId])
} else {
getNotInterestedIn(userId)
}
val pausedTopicIdsFuture = Future.value(getPausedTopics(context))
Future
.collect(
List(
denyListTopicIdsFuture,
optOutTopicIdsFuture,
notInterestedTopicIdsFuture,
pausedTopicIdsFuture)).map { list => list.reduce(_ ++ _) }
}
private[this] def getDiff(
aFut: Future[Set[ScEntityId]],
bFut: Future[Set[ScEntityId]]
): Future[Set[ScEntityId]] = {
Future.join(aFut, bFut).map {
case (a, b) => a.diff(b)
}
}
/**
* calculates the diff of all the whitelisted IDs with blacklisted IDs and returns the set of IDs
* that we will be recommending from or followed topics by the user by client setting.
*/
def getAllowListTopicsForUser(
userId: UserId,
topicListingSetting: TopicListingSetting,
context: TopicListingViewerContext,
bypassModes: Option[Set[TopicSocialProofFilteringBypassMode]]
): Future[Map[ScEntityId, Option[TopicFollowType]]] = {
/**
* Title: an illustrative table to explain how allow list is composed
* AllowList = WhiteList - DenyList - OptOutTopics - PausedTopics - NotInterestedInTopics
*
* TopicListingSetting: Following ImplicitFollow All Followable
* Whitelist: FollowedTopics(user) AllWhitelistedTopics Nil AllWhitelistedTopics
* DenyList: Nil FollowedTopics(user) Nil Nil
*
* ps. for TopicListingSetting.All, the returned allow list is Nil. Why?
* It's because that allowList is not required given the TopicListingSetting == 'All'.
* See TopicSocialProofHandler.filterByAllowedList() for more details.
*/
topicListingSetting match {
// "All" means all the UTT entity is qualified. So don't need to fetch the Whitelist anymore.
case TopicListingSetting.All => Future.value(Map.empty)
case TopicListingSetting.Following =>
getFollowingTopicsForUserWithTimestamp(userId, context, bypassModes).map {
_.mapValues(_ => Some(TopicFollowType.Following))
}
case TopicListingSetting.ImplicitFollow =>
getDiff(
getWhitelistTopicIds(context, enableInternationalTopics = true),
getDenyListTopicIdsForUser(userId, topicListingSetting, context, bypassModes)).map {
_.map { scEntityId =>
scEntityId -> Some(TopicFollowType.ImplicitFollow)
}.toMap
}
case _ =>
val followedTopicIdsFut = getFollowedTopicIds(userId)
val allowListTopicIdsFut = getDiff(
getWhitelistTopicIds(context, enableInternationalTopics = true),
getDenyListTopicIdsForUser(userId, topicListingSetting, context, bypassModes))
Future.join(allowListTopicIdsFut, followedTopicIdsFut).map {
case (allowListTopicId, followedTopicIds) =>
allowListTopicId.map { scEntityId =>
if (followedTopicIds.contains(scEntityId))
scEntityId -> Some(TopicFollowType.Following)
else scEntityId -> Some(TopicFollowType.ImplicitFollow)
}.toMap
}
}
}
private[this] def getFollowingTopicsForUserWithTimestamp(
userId: UserId,
context: TopicListingViewerContext,
bypassModes: Option[Set[TopicSocialProofFilteringBypassMode]]
): Future[Map[ScEntityId, Option[Long]]] = {
val followedTopicIdToTimestampFut = getFollowedTopics(userId).map(_.map { followedTopic =>
followedTopic.entityId -> followedTopic.topicFollowTimestamp
}.toMap)
followedTopicIdToTimestampFut.flatMap { followedTopicIdToTimestamp =>
getDiff(
Future(followedTopicIdToTimestamp.keySet),
getDenyListTopicIdsForUser(userId, TopicListingSetting.Following, context, bypassModes)
).map {
_.map { scEntityId =>
scEntityId -> followedTopicIdToTimestamp.get(scEntityId).flatten
}.toMap
}
}
}
}
object UttTopicFilterStore {
val DefaultNotInterestedInTimeout: Duration = 60.milliseconds
val DefaultOptOutTimeout: Duration = 60.milliseconds
val DefaultInterestedInTimeout: Duration = 60.milliseconds
}

View File

@ -0,0 +1,14 @@
scala_library(
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
],
dependencies = [
"3rdparty/jvm/org/lz4:lz4-java",
"content-recommender/thrift/src/main/thrift:thrift-scala",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store",
"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/health",
"stitch/stitch-storehaus",
"topic-social-proof/server/src/main/thrift:thrift-scala",
],
)

View File

@ -0,0 +1,19 @@
package com.twitter.tsp.utils
import com.twitter.bijection.Injection
import scala.util.Try
import net.jpountz.lz4.LZ4CompressorWithLength
import net.jpountz.lz4.LZ4DecompressorWithLength
import net.jpountz.lz4.LZ4Factory
object LZ4Injection extends Injection[Array[Byte], Array[Byte]] {
private val lz4Factory = LZ4Factory.fastestInstance()
private val fastCompressor = new LZ4CompressorWithLength(lz4Factory.fastCompressor())
private val decompressor = new LZ4DecompressorWithLength(lz4Factory.fastDecompressor())
override def apply(a: Array[Byte]): Array[Byte] = LZ4Injection.fastCompressor.compress(a)
override def invert(b: Array[Byte]): Try[Array[Byte]] = Try {
LZ4Injection.decompressor.decompress(b)
}
}

View File

@ -0,0 +1,20 @@
package com.twitter.tsp.utils
import com.twitter.storehaus.AbstractReadableStore
import com.twitter.storehaus.ReadableStore
import com.twitter.util.Future
class ReadableStoreWithMapOptionValues[K, V1, V2](rs: ReadableStore[K, V1]) {
def mapOptionValues(
fn: V1 => Option[V2]
): ReadableStore[K, V2] = {
val self = rs
new AbstractReadableStore[K, V2] {
override def get(k: K): Future[Option[V2]] = self.get(k).map(_.flatMap(fn))
override def multiGet[K1 <: K](ks: Set[K1]): Map[K1, Future[Option[V2]]] =
self.multiGet(ks).mapValues(_.map(_.flatMap(fn)))
}
}
}

View File

@ -0,0 +1,32 @@
package com.twitter.tsp.utils
import com.twitter.bijection.Injection
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.io.ObjectInputStream
import java.io.ObjectOutputStream
import java.io.Serializable
import scala.util.Try
/**
* @tparam T must be a serializable class
*/
case class SeqObjectInjection[T <: Serializable]() extends Injection[Seq[T], Array[Byte]] {
override def apply(seq: Seq[T]): Array[Byte] = {
val byteStream = new ByteArrayOutputStream()
val outputStream = new ObjectOutputStream(byteStream)
outputStream.writeObject(seq)
outputStream.close()
byteStream.toByteArray
}
override def invert(bytes: Array[Byte]): Try[Seq[T]] = {
Try {
val inputStream = new ObjectInputStream(new ByteArrayInputStream(bytes))
val seq = inputStream.readObject().asInstanceOf[Seq[T]]
inputStream.close()
seq
}
}
}

View File

@ -0,0 +1,21 @@
create_thrift_libraries(
base_name = "thrift",
sources = ["*.thrift"],
platform = "java8",
tags = [
"bazel-compatible",
],
dependency_roots = [
"content-recommender/thrift/src/main/thrift",
"content-recommender/thrift/src/main/thrift:content-recommender-common",
"interests-service/thrift/src/main/thrift",
"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift",
],
generate_languages = [
"java",
"scala",
"strato",
],
provides_java_name = "tsp-thrift-java",
provides_scala_name = "tsp-thrift-scala",
)

View File

@ -0,0 +1,104 @@
namespace java com.twitter.tsp.thriftjava
namespace py gen.twitter.tsp
#@namespace scala com.twitter.tsp.thriftscala
#@namespace strato com.twitter.tsp.strato
include "com/twitter/contentrecommender/common.thrift"
include "com/twitter/simclusters_v2/identifier.thrift"
include "com/twitter/simclusters_v2/online_store.thrift"
include "topic_listing.thrift"
enum TopicListingSetting {
All = 0 // All the existing Semantic Core Entity/Topics. ie., All topics on twitter, and may or may not have been launched yet.
Followable = 1 // All the topics which the user is allowed to follow. ie., topics that have shipped, and user may or may not be following it.
Following = 2 // Only topics the user is explicitly following
ImplicitFollow = 3 // The topics user has not followed but implicitly may follow. ie., Only topics that user has not followed.
} (hasPersonalData='false')
// used to tell Topic Social Proof endpoint which specific filtering can be bypassed
enum TopicSocialProofFilteringBypassMode {
NotInterested = 0
} (hasPersonalData='false')
struct TopicSocialProofRequest {
1: required i64 userId(personalDataType = "UserId")
2: required set<i64> tweetIds(personalDataType = 'TweetId')
3: required common.DisplayLocation displayLocation
4: required TopicListingSetting topicListingSetting
5: required topic_listing.TopicListingViewerContext context
6: optional set<TopicSocialProofFilteringBypassMode> bypassModes
7: optional map<i64, set<MetricTag>> tags
}
struct TopicSocialProofOptions {
1: required i64 userId(personalDataType = "UserId")
2: required common.DisplayLocation displayLocation
3: required TopicListingSetting topicListingSetting
4: required topic_listing.TopicListingViewerContext context
5: optional set<TopicSocialProofFilteringBypassMode> bypassModes
6: optional map<i64, set<MetricTag>> tags
}
struct TopicSocialProofResponse {
1: required map<i64, list<TopicWithScore>> socialProofs
}(hasPersonalData='false')
// Distinguishes between how a topic tweet is generated. Useful for metric tracking and debugging
enum TopicTweetType {
// CrOON candidates
UserInterestedIn = 1
Twistly = 2
// crTopic candidates
SkitConsumerEmbeddings = 100
SkitProducerEmbeddings = 101
SkitHighPrecision = 102
SkitInterestBrowser = 103
Certo = 104
}(persisted='true')
struct TopicWithScore {
1: required i64 topicId
2: required double score // score used to rank topics relative to one another
3: optional TopicTweetType algorithmType // how the topic is generated
4: optional TopicFollowType topicFollowType // Whether the topic is being explicitly or implicily followed
}(persisted='true', hasPersonalData='false')
struct ScoreKey {
1: required identifier.EmbeddingType userEmbeddingType
2: required identifier.EmbeddingType topicEmbeddingType
3: required online_store.ModelVersion modelVersion
}(persisted='true', hasPersonalData='false')
struct UserTopicScore {
1: required map<ScoreKey, double> scores
}(persisted='true', hasPersonalData='false')
enum TopicFollowType {
Following = 1
ImplicitFollow = 2
}(persisted='true')
// Provide the Tags which provides the Recommended Tweets Source Signal and other context.
// Warning: Please don't use this tag in any ML Features or business logic.
enum MetricTag {
// Source Signal Tags
TweetFavorite = 0
Retweet = 1
UserFollow = 101
PushOpenOrNtabClick = 201
HomeTweetClick = 301
HomeVideoView = 302
HomeSongbirdShowMore = 303
InterestsRankerRecentSearches = 401 // For Interests Candidate Expansion
UserInterestedIn = 501
MBCG = 503
// Other Metric Tags
} (persisted='true', hasPersonalData='true')

View File

@ -0,0 +1,26 @@
namespace java com.twitter.tsp.thriftjava
namespace py gen.twitter.tsp
#@namespace scala com.twitter.tsp.thriftscala
#@namespace strato com.twitter.tsp.strato
struct TspTweetInfo {
1: required i64 authorId
2: required i64 favCount
3: optional string language
6: optional bool hasImage
7: optional bool hasVideo
8: optional bool hasGif
9: optional bool isNsfwAuthor
10: optional bool isKGODenylist
11: optional bool isNullcast
// available if the tweet contains video
12: optional i32 videoDurationSeconds
13: optional bool isHighMediaResolution
14: optional bool isVerticalAspectRatio
// health signal scores
15: optional bool isPassAgathaHealthFilterStrictest
16: optional bool isPassTweetHealthFilterStrictest
17: optional bool isReply
18: optional bool hasMultipleMedia
23: optional bool hasUrl
}(persisted='false', hasPersonalData='true')

4
unified_user_actions/.gitignore vendored Normal file
View File

@ -0,0 +1,4 @@
.DS_Store
CONFIG.ini
PROJECT
docs

View File

@ -0,0 +1 @@
# This prevents SQ query from grabbing //:all since it traverses up once to find a BUILD

View File

@ -0,0 +1,10 @@
# Unified User Actions (UUA)
**Unified User Actions** (UUA) is a centralized, real-time stream of user actions on Twitter, consumed by various product, ML, and marketing teams. UUA reads client-side and server-side event streams that contain the user's actions and generates a unified real-time user actions Kafka stream. The Kafka stream is replicated to HDFS, GCP Pubsub, GCP GCS, GCP BigQuery. The user actions include public actions such as favorites, retweets, replies and implicit actions like bookmark, impression, video view.
## Components
- adapter: transform the raw inputs to UUA Thrift output
- client: Kafka client related utils
- kafka: more specific Kafka utils like customized serde
- service: deployment, modules and services

View File

@ -0,0 +1,19 @@
package com.twitter.unified_user_actions.adapter
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.finagle.stats.StatsReceiver
trait AbstractAdapter[INPUT, OUTK, OUTV] extends Serializable {
/**
* The basic input -> seq[output] adapter which concrete adapters should extend from
* @param input a single INPUT
* @return A list of (OUTK, OUTV) tuple. The OUTK is the output key mainly for publishing to Kafka (or Pubsub).
* If other processing, e.g. offline batch processing, doesn't require the output key then it can drop it
* like source.adaptOneToKeyedMany.map(_._2)
*/
def adaptOneToKeyedMany(
input: INPUT,
statsReceiver: StatsReceiver = NullStatsReceiver
): Seq[(OUTK, OUTV)]
}

View File

@ -0,0 +1,11 @@
scala_library(
name = "base",
sources = [
"AbstractAdapter.scala",
],
compiler_option_sets = ["fatal_warnings"],
tags = ["bazel-compatible"],
dependencies = [
"util/util-stats/src/main/scala/com/twitter/finagle/stats",
],
)

View File

@ -0,0 +1,125 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.thriftscala._
object AdsCallbackEngagement {
object PromotedTweetFav extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetFav)
object PromotedTweetUnfav extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetUnfav)
object PromotedTweetReply extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetReply)
object PromotedTweetRetweet
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetRetweet)
object PromotedTweetBlockAuthor
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetBlockAuthor)
object PromotedTweetUnblockAuthor
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetUnblockAuthor)
object PromotedTweetComposeTweet
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetComposeTweet)
object PromotedTweetClick extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetClick)
object PromotedTweetReport extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetReport)
object PromotedProfileFollow
extends ProfileAdsCallbackEngagement(ActionType.ServerPromotedProfileFollow)
object PromotedProfileUnfollow
extends ProfileAdsCallbackEngagement(ActionType.ServerPromotedProfileUnfollow)
object PromotedTweetMuteAuthor
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetMuteAuthor)
object PromotedTweetClickProfile
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetClickProfile)
object PromotedTweetClickHashtag
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetClickHashtag)
object PromotedTweetOpenLink
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetOpenLink) {
override def getItem(input: SpendServerEvent): Option[Item] = {
input.engagementEvent.flatMap { e =>
e.impressionData.flatMap { i =>
getPromotedTweetInfo(
i.promotedTweetId,
i.advertiserId,
tweetActionInfoOpt = Some(
TweetActionInfo.ServerPromotedTweetOpenLink(
ServerPromotedTweetOpenLink(url = e.url))))
}
}
}
}
object PromotedTweetCarouselSwipeNext
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetCarouselSwipeNext)
object PromotedTweetCarouselSwipePrevious
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetCarouselSwipePrevious)
object PromotedTweetLingerImpressionShort
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetLingerImpressionShort)
object PromotedTweetLingerImpressionMedium
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetLingerImpressionMedium)
object PromotedTweetLingerImpressionLong
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetLingerImpressionLong)
object PromotedTweetClickSpotlight
extends BaseTrendAdsCallbackEngagement(ActionType.ServerPromotedTweetClickSpotlight)
object PromotedTweetViewSpotlight
extends BaseTrendAdsCallbackEngagement(ActionType.ServerPromotedTweetViewSpotlight)
object PromotedTrendView
extends BaseTrendAdsCallbackEngagement(ActionType.ServerPromotedTrendView)
object PromotedTrendClick
extends BaseTrendAdsCallbackEngagement(ActionType.ServerPromotedTrendClick)
object PromotedTweetVideoPlayback25
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoPlayback25)
object PromotedTweetVideoPlayback50
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoPlayback50)
object PromotedTweetVideoPlayback75
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoPlayback75)
object PromotedTweetVideoAdPlayback25
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoAdPlayback25)
object PromotedTweetVideoAdPlayback50
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoAdPlayback50)
object PromotedTweetVideoAdPlayback75
extends BaseVideoAdsCallbackEngagement(ActionType.ServerPromotedTweetVideoAdPlayback75)
object TweetVideoAdPlayback25
extends BaseVideoAdsCallbackEngagement(ActionType.ServerTweetVideoAdPlayback25)
object TweetVideoAdPlayback50
extends BaseVideoAdsCallbackEngagement(ActionType.ServerTweetVideoAdPlayback50)
object TweetVideoAdPlayback75
extends BaseVideoAdsCallbackEngagement(ActionType.ServerTweetVideoAdPlayback75)
object PromotedTweetDismissWithoutReason
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetDismissWithoutReason)
object PromotedTweetDismissUninteresting
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetDismissUninteresting)
object PromotedTweetDismissRepetitive
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetDismissRepetitive)
object PromotedTweetDismissSpam
extends BaseAdsCallbackEngagement(ActionType.ServerPromotedTweetDismissSpam)
}

View File

@ -0,0 +1,28 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.finatra.kafka.serde.UnKeyed
import com.twitter.unified_user_actions.adapter.AbstractAdapter
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
class AdsCallbackEngagementsAdapter
extends AbstractAdapter[SpendServerEvent, UnKeyed, UnifiedUserAction] {
import AdsCallbackEngagementsAdapter._
override def adaptOneToKeyedMany(
input: SpendServerEvent,
statsReceiver: StatsReceiver = NullStatsReceiver
): Seq[(UnKeyed, UnifiedUserAction)] =
adaptEvent(input).map { e => (UnKeyed, e) }
}
object AdsCallbackEngagementsAdapter {
def adaptEvent(input: SpendServerEvent): Seq[UnifiedUserAction] = {
val baseEngagements: Seq[BaseAdsCallbackEngagement] =
EngagementTypeMappings.getEngagementMappings(Option(input).flatMap(_.engagementEvent))
baseEngagements.flatMap(_.getUUA(input))
}
}

View File

@ -0,0 +1,18 @@
scala_library(
sources = [
"*.scala",
],
compiler_option_sets = ["fatal_warnings"],
tags = [
"bazel-compatible",
"bazel-only",
],
dependencies = [
"kafka/finagle-kafka/finatra-kafka/src/main/scala",
"src/thrift/com/twitter/ads/billing/spendserver:spendserver_thrift-scala",
"src/thrift/com/twitter/ads/eventstream:eventstream-scala",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter:base",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter/common",
"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala",
],
)

View File

@ -0,0 +1,68 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.adapter.common.AdapterUtils
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.AuthorInfo
import com.twitter.unified_user_actions.thriftscala.EventMetadata
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.SourceLineage
import com.twitter.unified_user_actions.thriftscala.TweetInfo
import com.twitter.unified_user_actions.thriftscala.TweetActionInfo
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala.UserIdentifier
abstract class BaseAdsCallbackEngagement(actionType: ActionType) {
protected def getItem(input: SpendServerEvent): Option[Item] = {
input.engagementEvent.flatMap { e =>
e.impressionData.flatMap { i =>
getPromotedTweetInfo(i.promotedTweetId, i.advertiserId)
}
}
}
protected def getPromotedTweetInfo(
promotedTweetIdOpt: Option[Long],
advertiserId: Long,
tweetActionInfoOpt: Option[TweetActionInfo] = None
): Option[Item] = {
promotedTweetIdOpt.map { promotedTweetId =>
Item.TweetInfo(
TweetInfo(
actionTweetId = promotedTweetId,
actionTweetAuthorInfo = Some(AuthorInfo(authorId = Some(advertiserId))),
tweetActionInfo = tweetActionInfoOpt)
)
}
}
def getUUA(input: SpendServerEvent): Option[UnifiedUserAction] = {
val userIdentifier: UserIdentifier =
UserIdentifier(
userId = input.engagementEvent.flatMap(e => e.clientInfo.flatMap(_.userId64)),
guestIdMarketing = input.engagementEvent.flatMap(e => e.clientInfo.flatMap(_.guestId)),
)
getItem(input).map { item =>
UnifiedUserAction(
userIdentifier = userIdentifier,
item = item,
actionType = actionType,
eventMetadata = getEventMetadata(input),
)
}
}
protected def getEventMetadata(input: SpendServerEvent): EventMetadata =
EventMetadata(
sourceTimestampMs = input.engagementEvent
.map { e => e.engagementEpochTimeMilliSec }.getOrElse(AdapterUtils.currentTimestampMs),
receivedTimestampMs = AdapterUtils.currentTimestampMs,
sourceLineage = SourceLineage.ServerAdsCallbackEngagements,
language = input.engagementEvent.flatMap { e => e.clientInfo.flatMap(_.languageCode) },
countryCode = input.engagementEvent.flatMap { e => e.clientInfo.flatMap(_.countryCode) },
clientAppId =
input.engagementEvent.flatMap { e => e.clientInfo.flatMap(_.clientId) }.map { _.toLong },
)
}

View File

@ -0,0 +1,18 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseTrendAdsCallbackEngagement(actionType: ActionType)
extends BaseAdsCallbackEngagement(actionType = actionType) {
override protected def getItem(input: SpendServerEvent): Option[Item] = {
input.engagementEvent.flatMap { e =>
e.impressionData.flatMap { i =>
i.promotedTrendId.map { promotedTrendId =>
Item.TrendInfo(TrendInfo(actionTrendId = promotedTrendId))
}
}
}
}
}

View File

@ -0,0 +1,54 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.AuthorInfo
import com.twitter.unified_user_actions.thriftscala.TweetVideoWatch
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.TweetActionInfo
import com.twitter.unified_user_actions.thriftscala.TweetInfo
abstract class BaseVideoAdsCallbackEngagement(actionType: ActionType)
extends BaseAdsCallbackEngagement(actionType = actionType) {
override def getItem(input: SpendServerEvent): Option[Item] = {
input.engagementEvent.flatMap { e =>
e.impressionData.flatMap { i =>
getTweetInfo(i.promotedTweetId, i.organicTweetId, i.advertiserId, input)
}
}
}
private def getTweetInfo(
promotedTweetId: Option[Long],
organicTweetId: Option[Long],
advertiserId: Long,
input: SpendServerEvent
): Option[Item] = {
val actionedTweetIdOpt: Option[Long] =
if (promotedTweetId.isEmpty) organicTweetId else promotedTweetId
actionedTweetIdOpt.map { actionTweetId =>
Item.TweetInfo(
TweetInfo(
actionTweetId = actionTweetId,
actionTweetAuthorInfo = Some(AuthorInfo(authorId = Some(advertiserId))),
tweetActionInfo = Some(
TweetActionInfo.TweetVideoWatch(
TweetVideoWatch(
isMonetizable = Some(true),
videoOwnerId = input.engagementEvent
.flatMap(e => e.cardEngagement).flatMap(_.amplifyDetails).flatMap(_.videoOwnerId),
videoUuid = input.engagementEvent
.flatMap(_.cardEngagement).flatMap(_.amplifyDetails).flatMap(_.videoUuid),
prerollOwnerId = input.engagementEvent
.flatMap(e => e.cardEngagement).flatMap(_.amplifyDetails).flatMap(
_.prerollOwnerId),
prerollUuid = input.engagementEvent
.flatMap(_.cardEngagement).flatMap(_.amplifyDetails).flatMap(_.prerollUuid)
))
)
),
)
}
}
}

View File

@ -0,0 +1,69 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.eventstream.thriftscala.EngagementEvent
import com.twitter.adserver.thriftscala.EngagementType
import com.twitter.unified_user_actions.adapter.ads_callback_engagements.AdsCallbackEngagement._
object EngagementTypeMappings {
/**
* Ads could be Tweets or non-Tweets. Since UUA explicitly sets the item type, it is
* possible that one Ads Callback engagement type maps to multiple UUA action types.
*/
def getEngagementMappings(
engagementEvent: Option[EngagementEvent]
): Seq[BaseAdsCallbackEngagement] = {
val promotedTweetId: Option[Long] =
engagementEvent.flatMap(_.impressionData).flatMap(_.promotedTweetId)
engagementEvent
.map(event =>
event.engagementType match {
case EngagementType.Fav => Seq(PromotedTweetFav)
case EngagementType.Unfav => Seq(PromotedTweetUnfav)
case EngagementType.Reply => Seq(PromotedTweetReply)
case EngagementType.Retweet => Seq(PromotedTweetRetweet)
case EngagementType.Block => Seq(PromotedTweetBlockAuthor)
case EngagementType.Unblock => Seq(PromotedTweetUnblockAuthor)
case EngagementType.Send => Seq(PromotedTweetComposeTweet)
case EngagementType.Detail => Seq(PromotedTweetClick)
case EngagementType.Report => Seq(PromotedTweetReport)
case EngagementType.Follow => Seq(PromotedProfileFollow)
case EngagementType.Unfollow => Seq(PromotedProfileUnfollow)
case EngagementType.Mute => Seq(PromotedTweetMuteAuthor)
case EngagementType.ProfilePic => Seq(PromotedTweetClickProfile)
case EngagementType.ScreenName => Seq(PromotedTweetClickProfile)
case EngagementType.UserName => Seq(PromotedTweetClickProfile)
case EngagementType.Hashtag => Seq(PromotedTweetClickHashtag)
case EngagementType.Url => Seq(PromotedTweetOpenLink)
case EngagementType.CarouselSwipeNext => Seq(PromotedTweetCarouselSwipeNext)
case EngagementType.CarouselSwipePrevious => Seq(PromotedTweetCarouselSwipePrevious)
case EngagementType.DwellShort => Seq(PromotedTweetLingerImpressionShort)
case EngagementType.DwellMedium => Seq(PromotedTweetLingerImpressionMedium)
case EngagementType.DwellLong => Seq(PromotedTweetLingerImpressionLong)
case EngagementType.SpotlightClick => Seq(PromotedTweetClickSpotlight)
case EngagementType.SpotlightView => Seq(PromotedTweetViewSpotlight)
case EngagementType.TrendView => Seq(PromotedTrendView)
case EngagementType.TrendClick => Seq(PromotedTrendClick)
case EngagementType.VideoContentPlayback25 => Seq(PromotedTweetVideoPlayback25)
case EngagementType.VideoContentPlayback50 => Seq(PromotedTweetVideoPlayback50)
case EngagementType.VideoContentPlayback75 => Seq(PromotedTweetVideoPlayback75)
case EngagementType.VideoAdPlayback25 if promotedTweetId.isDefined =>
Seq(PromotedTweetVideoAdPlayback25)
case EngagementType.VideoAdPlayback25 if promotedTweetId.isEmpty =>
Seq(TweetVideoAdPlayback25)
case EngagementType.VideoAdPlayback50 if promotedTweetId.isDefined =>
Seq(PromotedTweetVideoAdPlayback50)
case EngagementType.VideoAdPlayback50 if promotedTweetId.isEmpty =>
Seq(TweetVideoAdPlayback50)
case EngagementType.VideoAdPlayback75 if promotedTweetId.isDefined =>
Seq(PromotedTweetVideoAdPlayback75)
case EngagementType.VideoAdPlayback75 if promotedTweetId.isEmpty =>
Seq(TweetVideoAdPlayback75)
case EngagementType.DismissRepetitive => Seq(PromotedTweetDismissRepetitive)
case EngagementType.DismissSpam => Seq(PromotedTweetDismissSpam)
case EngagementType.DismissUninteresting => Seq(PromotedTweetDismissUninteresting)
case EngagementType.DismissWithoutReason => Seq(PromotedTweetDismissWithoutReason)
case _ => Nil
}).toSeq.flatten
}
}

View File

@ -0,0 +1,26 @@
package com.twitter.unified_user_actions.adapter.ads_callback_engagements
import com.twitter.ads.spendserver.thriftscala.SpendServerEvent
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProfileInfo
abstract class ProfileAdsCallbackEngagement(actionType: ActionType)
extends BaseAdsCallbackEngagement(actionType) {
override protected def getItem(input: SpendServerEvent): Option[Item] = {
input.engagementEvent.flatMap { e =>
e.impressionData.flatMap { i =>
getProfileInfo(i.advertiserId)
}
}
}
protected def getProfileInfo(advertiserId: Long): Option[Item] = {
Some(
Item.ProfileInfo(
ProfileInfo(
actionProfileId = advertiserId
)))
}
}

View File

@ -0,0 +1,13 @@
scala_library(
sources = [
"*.scala",
],
tags = ["bazel-compatible"],
dependencies = [
"client-events/thrift/src/thrift/storage/twitter/behavioral_event:behavioral_event-scala",
"kafka/finagle-kafka/finatra-kafka/src/main/scala",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter:base",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter/common",
"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala",
],
)

View File

@ -0,0 +1,96 @@
package com.twitter.unified_user_actions.adapter.behavioral_client_event
import com.twitter.client_event_entities.serverside_context_key.latest.thriftscala.FlattenedServersideContextKey
import com.twitter.storage.behavioral_event.thriftscala.EventLogContext
import com.twitter.storage.behavioral_event.thriftscala.FlattenedEventLog
import com.twitter.unified_user_actions.adapter.common.AdapterUtils
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.BreadcrumbTweet
import com.twitter.unified_user_actions.thriftscala.ClientEventNamespace
import com.twitter.unified_user_actions.thriftscala.EventMetadata
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.ProductSurfaceInfo
import com.twitter.unified_user_actions.thriftscala.SourceLineage
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala.UserIdentifier
case class ProductSurfaceRelated(
productSurface: Option[ProductSurface],
productSurfaceInfo: Option[ProductSurfaceInfo])
trait BaseBCEAdapter {
def toUUA(e: FlattenedEventLog): Seq[UnifiedUserAction]
protected def getUserIdentifier(c: EventLogContext): UserIdentifier =
UserIdentifier(
userId = c.userId,
guestIdMarketing = c.guestIdMarketing
)
protected def getEventMetadata(e: FlattenedEventLog): EventMetadata =
EventMetadata(
sourceLineage = SourceLineage.BehavioralClientEvents,
sourceTimestampMs =
e.context.driftAdjustedEventCreatedAtMs.getOrElse(e.context.eventCreatedAtMs),
receivedTimestampMs = AdapterUtils.currentTimestampMs,
// Client UI language or from Gizmoduck which is what user set in Twitter App.
// Please see more at https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/finatra-internal/international/src/main/scala/com/twitter/finatra/international/LanguageIdentifier.scala
// The format should be ISO 639-1.
language = e.context.languageCode.map(AdapterUtils.normalizeLanguageCode),
// Country code could be IP address (geoduck) or User registration country (gizmoduck) and the former takes precedence.
// We dont know exactly which one is applied, unfortunately,
// see https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/finatra-internal/international/src/main/scala/com/twitter/finatra/international/CountryIdentifier.scala
// The format should be ISO_3166-1_alpha-2.
countryCode = e.context.countryCode.map(AdapterUtils.normalizeCountryCode),
clientAppId = e.context.clientApplicationId,
clientVersion = e.context.clientVersion,
clientPlatform = e.context.clientPlatform,
viewHierarchy = e.v1ViewTypeHierarchy,
clientEventNamespace = Some(
ClientEventNamespace(
page = e.page,
section = e.section,
element = e.element,
action = e.actionName,
subsection = e.subsection
)),
breadcrumbViews = e.v1BreadcrumbViewTypeHierarchy,
breadcrumbTweets = e.v1BreadcrumbTweetIds.map { breadcrumbs =>
breadcrumbs.map { breadcrumb =>
BreadcrumbTweet(
tweetId = breadcrumb.serversideContextId.toLong,
sourceComponent = breadcrumb.sourceComponent)
}
}
)
protected def getBreadcrumbTweetIds(
breadcrumbTweetIds: Option[Seq[FlattenedServersideContextKey]]
): Seq[BreadcrumbTweet] =
breadcrumbTweetIds
.getOrElse(Nil).map(breadcrumb => {
BreadcrumbTweet(
tweetId = breadcrumb.serversideContextId.toLong,
sourceComponent = breadcrumb.sourceComponent)
})
protected def getBreadcrumbViews(breadcrumbView: Option[Seq[String]]): Seq[String] =
breadcrumbView.getOrElse(Nil)
protected def getUnifiedUserAction(
event: FlattenedEventLog,
actionType: ActionType,
item: Item,
productSurface: Option[ProductSurface] = None,
productSurfaceInfo: Option[ProductSurfaceInfo] = None
): UnifiedUserAction =
UnifiedUserAction(
userIdentifier = getUserIdentifier(event.context),
actionType = actionType,
item = item,
eventMetadata = getEventMetadata(event),
productSurface = productSurface,
productSurfaceInfo = productSurfaceInfo
)
}

View File

@ -0,0 +1,39 @@
package com.twitter.unified_user_actions.adapter.behavioral_client_event
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.finatra.kafka.serde.UnKeyed
import com.twitter.storage.behavioral_event.thriftscala.FlattenedEventLog
import com.twitter.unified_user_actions.adapter.AbstractAdapter
import com.twitter.unified_user_actions.thriftscala._
class BehavioralClientEventAdapter
extends AbstractAdapter[FlattenedEventLog, UnKeyed, UnifiedUserAction] {
import BehavioralClientEventAdapter._
override def adaptOneToKeyedMany(
input: FlattenedEventLog,
statsReceiver: StatsReceiver = NullStatsReceiver
): Seq[(UnKeyed, UnifiedUserAction)] =
adaptEvent(input).map { e => (UnKeyed, e) }
}
object BehavioralClientEventAdapter {
def adaptEvent(e: FlattenedEventLog): Seq[UnifiedUserAction] =
// See go/bcecoverage for event namespaces, usage and coverage details
Option(e)
.map { e =>
(e.page, e.actionName) match {
case (Some("tweet_details"), Some("impress")) =>
TweetImpressionBCEAdapter.TweetDetails.toUUA(e)
case (Some("fullscreen_video"), Some("impress")) =>
TweetImpressionBCEAdapter.FullscreenVideo.toUUA(e)
case (Some("fullscreen_image"), Some("impress")) =>
TweetImpressionBCEAdapter.FullscreenImage.toUUA(e)
case (Some("profile"), Some("impress")) =>
ProfileImpressionBCEAdapter.Profile.toUUA(e)
case _ => Nil
}
}.getOrElse(Nil)
}

View File

@ -0,0 +1,34 @@
package com.twitter.unified_user_actions.adapter.behavioral_client_event
import com.twitter.client.behavioral_event.action.impress.latest.thriftscala.Impress
import com.twitter.client_event_entities.serverside_context_key.latest.thriftscala.FlattenedServersideContextKey
import com.twitter.unified_user_actions.thriftscala.Item
trait ImpressionBCEAdapter extends BaseBCEAdapter {
type ImpressedItem <: Item
def getImpressedItem(
context: FlattenedServersideContextKey,
impression: Impress
): ImpressedItem
/**
* The start time of an impression in milliseconds since epoch. In BCE, the impression
* tracking clock will start immediately after the page is visible with no initial delay.
*/
def getImpressedStartTimestamp(impression: Impress): Long =
impression.visibilityPctDwellStartMs
/**
* The end time of an impression in milliseconds since epoch. In BCE, the impression
* tracking clock will end before the user exit the page.
*/
def getImpressedEndTimestamp(impression: Impress): Long =
impression.visibilityPctDwellEndMs
/**
* The UI component that hosted the impressed item.
*/
def getImpressedUISourceComponent(context: FlattenedServersideContextKey): String =
context.sourceComponent
}

View File

@ -0,0 +1,52 @@
package com.twitter.unified_user_actions.adapter.behavioral_client_event
import com.twitter.client.behavioral_event.action.impress.latest.thriftscala.Impress
import com.twitter.client_event_entities.serverside_context_key.latest.thriftscala.FlattenedServersideContextKey
import com.twitter.storage.behavioral_event.thriftscala.FlattenedEventLog
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.ClientProfileV2Impression
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.ProfileActionInfo
import com.twitter.unified_user_actions.thriftscala.ProfileInfo
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
object ProfileImpressionBCEAdapter {
val Profile = new ProfileImpressionBCEAdapter()
}
class ProfileImpressionBCEAdapter extends ImpressionBCEAdapter {
override type ImpressedItem = Item.ProfileInfo
override def toUUA(e: FlattenedEventLog): Seq[UnifiedUserAction] =
(e.v2Impress, e.v1UserIds) match {
case (Some(v2Impress), Some(v1UserIds)) =>
v1UserIds.map { user =>
getUnifiedUserAction(
event = e,
actionType = ActionType.ClientProfileV2Impression,
item = getImpressedItem(user, v2Impress),
productSurface = Some(ProductSurface.ProfilePage)
)
}
case _ => Nil
}
override def getImpressedItem(
context: FlattenedServersideContextKey,
impression: Impress
): ImpressedItem =
Item.ProfileInfo(
ProfileInfo(
actionProfileId = context.serversideContextId.toLong,
profileActionInfo = Some(
ProfileActionInfo.ClientProfileV2Impression(
ClientProfileV2Impression(
impressStartTimestampMs = getImpressedStartTimestamp(impression),
impressEndTimestampMs = getImpressedEndTimestamp(impression),
sourceComponent = getImpressedUISourceComponent(context)
)
)
)
))
}

View File

@ -0,0 +1,84 @@
package com.twitter.unified_user_actions.adapter.behavioral_client_event
import com.twitter.client.behavioral_event.action.impress.latest.thriftscala.Impress
import com.twitter.client_event_entities.serverside_context_key.latest.thriftscala.FlattenedServersideContextKey
import com.twitter.storage.behavioral_event.thriftscala.FlattenedEventLog
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.ClientTweetV2Impression
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.TweetActionInfo
import com.twitter.unified_user_actions.thriftscala.TweetInfo
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
object TweetImpressionBCEAdapter {
val TweetDetails = new TweetImpressionBCEAdapter(ActionType.ClientTweetV2Impression)
val FullscreenVideo = new TweetImpressionBCEAdapter(
ActionType.ClientTweetVideoFullscreenV2Impression)
val FullscreenImage = new TweetImpressionBCEAdapter(
ActionType.ClientTweetImageFullscreenV2Impression)
}
class TweetImpressionBCEAdapter(actionType: ActionType) extends ImpressionBCEAdapter {
override type ImpressedItem = Item.TweetInfo
override def toUUA(e: FlattenedEventLog): Seq[UnifiedUserAction] =
(actionType, e.v2Impress, e.v1TweetIds, e.v1BreadcrumbTweetIds) match {
case (ActionType.ClientTweetV2Impression, Some(v2Impress), Some(v1TweetIds), _) =>
toUUAEvents(e, v2Impress, v1TweetIds)
case (
ActionType.ClientTweetVideoFullscreenV2Impression,
Some(v2Impress),
_,
Some(v1BreadcrumbTweetIds)) =>
toUUAEvents(e, v2Impress, v1BreadcrumbTweetIds)
case (
ActionType.ClientTweetImageFullscreenV2Impression,
Some(v2Impress),
_,
Some(v1BreadcrumbTweetIds)) =>
toUUAEvents(e, v2Impress, v1BreadcrumbTweetIds)
case _ => Nil
}
private def toUUAEvents(
e: FlattenedEventLog,
v2Impress: Impress,
v1TweetIds: Seq[FlattenedServersideContextKey]
): Seq[UnifiedUserAction] =
v1TweetIds.map { tweet =>
getUnifiedUserAction(
event = e,
actionType = actionType,
item = getImpressedItem(tweet, v2Impress),
productSurface = getProductSurfaceRelated.productSurface,
productSurfaceInfo = getProductSurfaceRelated.productSurfaceInfo
)
}
override def getImpressedItem(
context: FlattenedServersideContextKey,
impression: Impress
): ImpressedItem =
Item.TweetInfo(
TweetInfo(
actionTweetId = context.serversideContextId.toLong,
tweetActionInfo = Some(
TweetActionInfo.ClientTweetV2Impression(
ClientTweetV2Impression(
impressStartTimestampMs = getImpressedStartTimestamp(impression),
impressEndTimestampMs = getImpressedEndTimestamp(impression),
sourceComponent = getImpressedUISourceComponent(context)
)
))
))
private def getProductSurfaceRelated: ProductSurfaceRelated =
actionType match {
case ActionType.ClientTweetV2Impression =>
ProductSurfaceRelated(
productSurface = Some(ProductSurface.TweetDetailsPage),
productSurfaceInfo = None)
case _ => ProductSurfaceRelated(productSurface = None, productSurfaceInfo = None)
}
}

View File

@ -0,0 +1,16 @@
scala_library(
sources = [
"*.scala",
],
tags = ["bazel-compatible"],
dependencies = [
"common-internal/analytics/client-analytics-data-layer/src/main/scala",
"kafka/finagle-kafka/finatra-kafka/src/main/scala",
"src/scala/com/twitter/loggedout/analytics/common",
"src/thrift/com/twitter/clientapp/gen:clientapp-scala",
"twadoop_config/configuration/log_categories/group/scribelib:client_event-scala",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter:base",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter/common",
"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala",
],
)

View File

@ -0,0 +1,46 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala._
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
abstract class BaseCTAClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def toUnifiedUserAction(logEvent: LogEvent): Seq[UnifiedUserAction] = {
val logBase: Option[LogBase] = logEvent.logBase
val userIdentifier: UserIdentifier = UserIdentifier(
userId = logBase.flatMap(_.userId),
guestIdMarketing = logBase.flatMap(_.guestIdMarketing))
val uuaItem: Item = Item.CtaInfo(CTAInfo())
val eventTimestamp = logBase.flatMap(getSourceTimestamp).getOrElse(0L)
val ceItem = LogEventItem.unsafeEmpty
val productSurface: Option[ProductSurface] = ProductSurfaceUtils
.getProductSurface(logEvent.eventNamespace)
val eventMetaData: EventMetadata = ClientEventCommonUtils
.getEventMetadata(
eventTimestamp = eventTimestamp,
logEvent = logEvent,
ceItem = ceItem,
productSurface = productSurface
)
Seq(
UnifiedUserAction(
userIdentifier = userIdentifier,
item = uuaItem,
actionType = actionType,
eventMetadata = eventMetaData,
productSurface = productSurface,
productSurfaceInfo =
ProductSurfaceUtils.getProductSurfaceInfo(productSurface, ceItem, logEvent)
))
}
}

View File

@ -0,0 +1,26 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.CardInfo
import com.twitter.unified_user_actions.thriftscala.Item
abstract class BaseCardClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.ignoreItemType(itemTypeOpt)
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = Some(
Item.CardInfo(
CardInfo(
id = ceItem.id,
itemType = ceItem.itemType,
actionTweetAuthorInfo = ClientEventCommonUtils.getAuthorInfo(ceItem),
))
)
}

View File

@ -0,0 +1,68 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.logbase.thriftscala.ClientEventReceiver
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseClientEvent(actionType: ActionType) {
def toUnifiedUserAction(logEvent: LogEvent): Seq[UnifiedUserAction] = {
val logBase: Option[LogBase] = logEvent.logBase
for {
ed <- logEvent.eventDetails.toSeq
items <- ed.items.toSeq
ceItem <- items
eventTimestamp <- logBase.flatMap(getSourceTimestamp)
uuaItem <- getUuaItem(ceItem, logEvent)
if isItemTypeValid(ceItem.itemType)
} yield {
val userIdentifier: UserIdentifier = UserIdentifier(
userId = logBase.flatMap(_.userId),
guestIdMarketing = logBase.flatMap(_.guestIdMarketing))
val productSurface: Option[ProductSurface] = ProductSurfaceUtils
.getProductSurface(logEvent.eventNamespace)
val eventMetaData: EventMetadata = ClientEventCommonUtils
.getEventMetadata(
eventTimestamp = eventTimestamp,
logEvent = logEvent,
ceItem = ceItem,
productSurface = productSurface
)
UnifiedUserAction(
userIdentifier = userIdentifier,
item = uuaItem,
actionType = actionType,
eventMetadata = eventMetaData,
productSurface = productSurface,
productSurfaceInfo =
ProductSurfaceUtils.getProductSurfaceInfo(productSurface, ceItem, logEvent)
)
}
}
def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for (actionTweetId <- ceItem.id)
yield Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(actionTweetId, ceItem, logEvent.eventNamespace))
// default implementation filters items of type tweet
// override in the subclass implementation to filter items of other types
def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.isItemTypeTweet(itemTypeOpt)
def getSourceTimestamp(logBase: LogBase): Option[Long] =
logBase.clientEventReceiver match {
case Some(ClientEventReceiver.CesHttp) | Some(ClientEventReceiver.CesThrift) =>
logBase.driftAdjustedEventCreatedAtMs
case _ => Some(logBase.driftAdjustedEventCreatedAtMs.getOrElse(logBase.timestamp))
}
}

View File

@ -0,0 +1,46 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseFeedbackSubmitClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = {
logEvent.eventNamespace.flatMap(_.page).flatMap {
case "search" =>
val searchInfoUtil = new SearchInfoUtils(ceItem)
searchInfoUtil.getQueryOptFromItem(logEvent).flatMap { query =>
val isRelevant: Boolean = logEvent.eventNamespace
.flatMap(_.element)
.contains("is_relevant")
logEvent.eventNamespace.flatMap(_.component).flatMap {
case "relevance_prompt_module" =>
for (actionTweetId <- ceItem.id)
yield Item.FeedbackPromptInfo(
FeedbackPromptInfo(
feedbackPromptActionInfo = FeedbackPromptActionInfo.TweetRelevantToSearch(
TweetRelevantToSearch(
searchQuery = query,
tweetId = actionTweetId,
isRelevant = Some(isRelevant)))))
case "did_you_find_it_module" =>
Some(
Item.FeedbackPromptInfo(FeedbackPromptInfo(feedbackPromptActionInfo =
FeedbackPromptActionInfo.DidYouFindItSearch(
DidYouFindItSearch(searchQuery = query, isRelevant = Some(isRelevant))))))
}
}
case _ => None
}
}
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.isItemTypeForSearchResultsPageFeedbackSubmit(itemTypeOpt)
}

View File

@ -0,0 +1,48 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseNotificationTabClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
// itemType is `None` for Notification Tab events
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.ignoreItemType(itemTypeOpt)
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for {
notificationTabDetails <- ceItem.notificationTabDetails
clientEventMetadata <- notificationTabDetails.clientEventMetadata
notificationId <- NotificationClientEventUtils.getNotificationIdForNotificationTab(ceItem)
} yield {
clientEventMetadata.tweetIds match {
// if `tweetIds` contain more than one Tweet id, create `MultiTweetNotification`
case Some(tweetIds) if tweetIds.size > 1 =>
Item.NotificationInfo(
NotificationInfo(
actionNotificationId = notificationId,
content = NotificationContent.MultiTweetNotification(
MultiTweetNotification(tweetIds = tweetIds))
))
// if `tweetIds` contain exactly one Tweet id, create `TweetNotification`
case Some(tweetIds) if tweetIds.size == 1 =>
Item.NotificationInfo(
NotificationInfo(
actionNotificationId = notificationId,
content =
NotificationContent.TweetNotification(TweetNotification(tweetId = tweetIds.head))))
// if `tweetIds` are missing, create `UnknownNotification`
case _ =>
Item.NotificationInfo(
NotificationInfo(
actionNotificationId = notificationId,
content = NotificationContent.UnknownNotification(UnknownNotification())
))
}
}
}

View File

@ -0,0 +1,25 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.adapter.client_event.ClientEventCommonUtils.getProfileIdFromUserItem
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProfileInfo
abstract class BaseProfileClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.isItemTypeProfile(itemTypeOpt)
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
getProfileIdFromUserItem(ceItem).map { id =>
Item.ProfileInfo(
ProfileInfo(actionProfileId = id)
)
}
}

View File

@ -0,0 +1,22 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala._
abstract class BasePushNotificationClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for {
itemId <- ceItem.id
notificationId <- NotificationClientEventUtils.getNotificationIdForPushNotification(logEvent)
} yield {
Item.NotificationInfo(
NotificationInfo(
actionNotificationId = notificationId,
content = NotificationContent.TweetNotification(TweetNotification(tweetId = itemId))))
}
}

View File

@ -0,0 +1,87 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.adapter.client_event.ClientEventCommonUtils.getProfileIdFromUserItem
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.EventMetadata
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.TopicQueryResult
import com.twitter.unified_user_actions.thriftscala.TypeaheadActionInfo
import com.twitter.unified_user_actions.thriftscala.TypeaheadInfo
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala.UserIdentifier
import com.twitter.unified_user_actions.thriftscala.UserResult
abstract class BaseSearchTypeaheadEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def toUnifiedUserAction(logEvent: LogEvent): Seq[UnifiedUserAction] = {
val logBase: Option[LogBase] = logEvent.logBase
for {
ed <- logEvent.eventDetails.toSeq
targets <- ed.targets.toSeq
ceTarget <- targets
eventTimestamp <- logBase.flatMap(getSourceTimestamp)
uuaItem <- getUuaItem(ceTarget, logEvent)
if isItemTypeValid(ceTarget.itemType)
} yield {
val userIdentifier: UserIdentifier = UserIdentifier(
userId = logBase.flatMap(_.userId),
guestIdMarketing = logBase.flatMap(_.guestIdMarketing))
val productSurface: Option[ProductSurface] = ProductSurfaceUtils
.getProductSurface(logEvent.eventNamespace)
val eventMetaData: EventMetadata = ClientEventCommonUtils
.getEventMetadata(
eventTimestamp = eventTimestamp,
logEvent = logEvent,
ceItem = ceTarget,
productSurface = productSurface
)
UnifiedUserAction(
userIdentifier = userIdentifier,
item = uuaItem,
actionType = actionType,
eventMetadata = eventMetaData,
productSurface = productSurface,
productSurfaceInfo =
ProductSurfaceUtils.getProductSurfaceInfo(productSurface, ceTarget, logEvent)
)
}
}
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.isItemTypeTypeaheadResult(itemTypeOpt)
override def getUuaItem(
ceTarget: LogEventItem,
logEvent: LogEvent
): Option[Item] =
logEvent.searchDetails.flatMap(_.query).flatMap { query =>
ceTarget.itemType match {
case Some(ItemType.User) =>
getProfileIdFromUserItem(ceTarget).map { profileId =>
Item.TypeaheadInfo(
TypeaheadInfo(
actionQuery = query,
typeaheadActionInfo =
TypeaheadActionInfo.UserResult(UserResult(profileId = profileId))))
}
case Some(ItemType.Search) =>
ceTarget.name.map { name =>
Item.TypeaheadInfo(
TypeaheadInfo(
actionQuery = query,
typeaheadActionInfo = TypeaheadActionInfo.TopicQueryResult(
TopicQueryResult(suggestedTopicQuery = name))))
}
case _ => None
}
}
}

View File

@ -0,0 +1,23 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.TopicInfo
abstract class BaseTopicClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def isItemTypeValid(itemTypeOpt: Option[ItemType]): Boolean =
ItemTypeFilterPredicates.isItemTypeTopic(itemTypeOpt)
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
for (actionTopicId <- ClientEventCommonUtils.getTopicId(
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace))
yield Item.TopicInfo(TopicInfo(actionTopicId = actionTopicId))
}

View File

@ -0,0 +1,62 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseUASClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def toUnifiedUserAction(logEvent: LogEvent): Seq[UnifiedUserAction] = {
val logBase: Option[LogBase] = logEvent.logBase
val ceItem = LogEventItem.unsafeEmpty
val uuaOpt: Option[UnifiedUserAction] = for {
eventTimestamp <- logBase.flatMap(getSourceTimestamp)
uuaItem <- getUuaItem(ceItem, logEvent)
} yield {
val userIdentifier: UserIdentifier = UserIdentifier(
userId = logBase.flatMap(_.userId),
guestIdMarketing = logBase.flatMap(_.guestIdMarketing))
val productSurface: Option[ProductSurface] = ProductSurfaceUtils
.getProductSurface(logEvent.eventNamespace)
val eventMetaData: EventMetadata = ClientEventCommonUtils
.getEventMetadata(
eventTimestamp = eventTimestamp,
logEvent = logEvent,
ceItem = ceItem,
productSurface = productSurface
)
UnifiedUserAction(
userIdentifier = userIdentifier,
item = uuaItem,
actionType = actionType,
eventMetadata = eventMetaData,
productSurface = productSurface,
productSurfaceInfo =
ProductSurfaceUtils.getProductSurfaceInfo(productSurface, ceItem, logEvent)
)
}
uuaOpt match {
case Some(uua) => Seq(uua)
case _ => Nil
}
}
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for {
performanceDetails <- logEvent.performanceDetails
duration <- performanceDetails.durationMs
} yield {
Item.UasInfo(UASInfo(timeSpentMs = duration))
}
}

View File

@ -0,0 +1,34 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala._
abstract class BaseVideoClientEvent(actionType: ActionType)
extends BaseClientEvent(actionType = actionType) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for {
actionTweetId <- ceItem.id
clientMediaEvent <- ceItem.clientMediaEvent
sessionState <- clientMediaEvent.sessionState
mediaIdentifier <- sessionState.contentVideoIdentifier
mediaId <- VideoClientEventUtils.videoIdFromMediaIdentifier(mediaIdentifier)
mediaDetails <- ceItem.mediaDetailsV2
mediaItems <- mediaDetails.mediaItems
videoMetadata <- VideoClientEventUtils.getVideoMetadata(
mediaId,
mediaItems,
ceItem.cardDetails.flatMap(_.amplifyDetails))
} yield {
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(videoMetadata)))
}
}

View File

@ -0,0 +1,272 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.clientapp.thriftscala.EventNamespace
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.finatra.kafka.serde.UnKeyed
import com.twitter.unified_user_actions.adapter.AbstractAdapter
import com.twitter.unified_user_actions.adapter.client_event.ClientEventImpression._
import com.twitter.unified_user_actions.adapter.client_event.ClientEventEngagement._
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import scala.util.matching.Regex
class ClientEventAdapter extends AbstractAdapter[LogEvent, UnKeyed, UnifiedUserAction] {
import ClientEventAdapter._
override def adaptOneToKeyedMany(
input: LogEvent,
statsReceiver: StatsReceiver = NullStatsReceiver
): Seq[(UnKeyed, UnifiedUserAction)] =
adaptEvent(input).map { e => (UnKeyed, e) }
}
object ClientEventAdapter {
// Refer to go/cme-scribing and go/interaction-event-spec for details
def isVideoEvent(element: String): Boolean = Seq[String](
"gif_player",
"periscope_player",
"platform_amplify_card",
"video_player",
"vine_player").contains(element)
/**
* Tweet clicks on the Notification Tab on iOS are a special case because the `element` is different
* from Tweet clicks everywhere else on the platform.
*
* For Notification Tab on iOS, `element` could be one of `user_mentioned_you`,
* `user_mentioned_you_in_a_quote_tweet`, `user_replied_to_your_tweet`, or `user_quoted_your_tweet`.
*
* In other places, `element` = `tweet`.
*/
def isTweetClickEvent(element: String): Boolean =
Seq[String](
"tweet",
"user_mentioned_you",
"user_mentioned_you_in_a_quote_tweet",
"user_replied_to_your_tweet",
"user_quoted_your_tweet"
).contains(element)
final val validUASIosClientIds = Seq[Long](
129032L, // Twitter for iPhone
191841L // Twitter for iPad
)
// Twitter for Android
final val validUASAndroidClientIds = Seq[Long](258901L)
def adaptEvent(inputLogEvent: LogEvent): Seq[UnifiedUserAction] =
Option(inputLogEvent).toSeq
.filterNot { logEvent: LogEvent =>
shouldIgnoreClientEvent(logEvent.eventNamespace)
}
.flatMap { logEvent: LogEvent =>
val actionTypesPerEvent: Seq[BaseClientEvent] = logEvent.eventNamespace.toSeq.flatMap {
name =>
(name.page, name.section, name.component, name.element, name.action) match {
case (_, _, _, _, Some("favorite")) => Seq(TweetFav)
case (_, _, _, _, Some("unfavorite")) => Seq(TweetUnfav)
case (_, _, Some("stream"), Some("linger"), Some("results")) =>
Seq(TweetLingerImpression)
case (_, _, Some("stream"), None, Some("results")) =>
Seq(TweetRenderImpression)
case (_, _, _, _, Some("send_reply")) => Seq(TweetReply)
// Different clients may have different actions of the same "send quote"
// but it turns out that both send_quote and retweet_with_comment should correspond to
// "send quote"
case (_, _, _, _, Some("send_quote_tweet")) |
(_, _, _, _, Some("retweet_with_comment")) =>
Seq(TweetQuote)
case (_, _, _, _, Some("retweet")) => Seq(TweetRetweet)
case (_, _, _, _, Some("unretweet")) => Seq(TweetUnretweet)
case (_, _, _, _, Some("reply")) => Seq(TweetClickReply)
case (_, _, _, _, Some("quote")) => Seq(TweetClickQuote)
case (_, _, _, Some(element), Some("playback_start")) if isVideoEvent(element) =>
Seq(TweetVideoPlaybackStart)
case (_, _, _, Some(element), Some("playback_complete")) if isVideoEvent(element) =>
Seq(TweetVideoPlaybackComplete)
case (_, _, _, Some(element), Some("playback_25")) if isVideoEvent(element) =>
Seq(TweetVideoPlayback25)
case (_, _, _, Some(element), Some("playback_50")) if isVideoEvent(element) =>
Seq(TweetVideoPlayback50)
case (_, _, _, Some(element), Some("playback_75")) if isVideoEvent(element) =>
Seq(TweetVideoPlayback75)
case (_, _, _, Some(element), Some("playback_95")) if isVideoEvent(element) =>
Seq(TweetVideoPlayback95)
case (_, _, _, Some(element), Some("play_from_tap")) if isVideoEvent(element) =>
Seq(TweetVideoPlayFromTap)
case (_, _, _, Some(element), Some("video_quality_view")) if isVideoEvent(element) =>
Seq(TweetVideoQualityView)
case (_, _, _, Some(element), Some("video_view")) if isVideoEvent(element) =>
Seq(TweetVideoView)
case (_, _, _, Some(element), Some("video_mrc_view")) if isVideoEvent(element) =>
Seq(TweetVideoMrcView)
case (_, _, _, Some(element), Some("view_threshold")) if isVideoEvent(element) =>
Seq(TweetVideoViewThreshold)
case (_, _, _, Some(element), Some("cta_url_click")) if isVideoEvent(element) =>
Seq(TweetVideoCtaUrlClick)
case (_, _, _, Some(element), Some("cta_watch_click")) if isVideoEvent(element) =>
Seq(TweetVideoCtaWatchClick)
case (_, _, _, Some("platform_photo_card"), Some("click")) => Seq(TweetPhotoExpand)
case (_, _, _, Some("platform_card"), Some("click")) => Seq(CardClick)
case (_, _, _, _, Some("open_app")) => Seq(CardOpenApp)
case (_, _, _, _, Some("install_app")) => Seq(CardAppInstallAttempt)
case (_, _, _, Some("platform_card"), Some("vote")) |
(_, _, _, Some("platform_forward_card"), Some("vote")) =>
Seq(PollCardVote)
case (_, _, _, Some("mention"), Some("click")) |
(_, _, _, _, Some("mention_click")) =>
Seq(TweetClickMentionScreenName)
case (_, _, _, Some(element), Some("click")) if isTweetClickEvent(element) =>
Seq(TweetClick)
case // Follow from the Topic page (or so-called landing page)
(_, _, _, Some("topic"), Some("follow")) |
// Actually not sure how this is generated ... but saw quite some events in BQ
(_, _, _, Some("social_proof"), Some("follow")) |
// Click on Tweet's caret menu of "Follow (the topic)", it needs to be:
// 1) user follows the Topic already, 2) and clicked on the "Unfollow Topic" first.
(_, _, _, Some("feedback_follow_topic"), Some("click")) =>
Seq(TopicFollow)
case (_, _, _, Some("topic"), Some("unfollow")) |
(_, _, _, Some("social_proof"), Some("unfollow")) |
(_, _, _, Some("feedback_unfollow_topic"), Some("click")) =>
Seq(TopicUnfollow)
case (_, _, _, Some("topic"), Some("not_interested")) |
(_, _, _, Some("feedback_not_interested_in_topic"), Some("click")) =>
Seq(TopicNotInterestedIn)
case (_, _, _, Some("topic"), Some("un_not_interested")) |
(_, _, _, Some("feedback_not_interested_in_topic"), Some("undo")) =>
Seq(TopicUndoNotInterestedIn)
case (_, _, _, Some("feedback_givefeedback"), Some("click")) =>
Seq(TweetNotHelpful)
case (_, _, _, Some("feedback_givefeedback"), Some("undo")) =>
Seq(TweetUndoNotHelpful)
case (_, _, _, Some("report_tweet"), Some("click")) |
(_, _, _, Some("report_tweet"), Some("done")) =>
Seq(TweetReport)
case (_, _, _, Some("feedback_dontlike"), Some("click")) =>
Seq(TweetNotInterestedIn)
case (_, _, _, Some("feedback_dontlike"), Some("undo")) =>
Seq(TweetUndoNotInterestedIn)
case (_, _, _, Some("feedback_notabouttopic"), Some("click")) =>
Seq(TweetNotAboutTopic)
case (_, _, _, Some("feedback_notabouttopic"), Some("undo")) =>
Seq(TweetUndoNotAboutTopic)
case (_, _, _, Some("feedback_notrecent"), Some("click")) =>
Seq(TweetNotRecent)
case (_, _, _, Some("feedback_notrecent"), Some("undo")) =>
Seq(TweetUndoNotRecent)
case (_, _, _, Some("feedback_seefewer"), Some("click")) =>
Seq(TweetSeeFewer)
case (_, _, _, Some("feedback_seefewer"), Some("undo")) =>
Seq(TweetUndoSeeFewer)
// Only when action = "submit" we get all fields in ReportDetails, such as reportType
// See https://confluence.twitter.biz/pages/viewpage.action?spaceKey=HEALTH&title=Understanding+ReportDetails
case (Some(page), _, _, Some("ticket"), Some("submit"))
if page.startsWith("report_") =>
Seq(TweetReportServer)
case (Some("profile"), _, _, _, Some("block")) =>
Seq(ProfileBlock)
case (Some("profile"), _, _, _, Some("unblock")) =>
Seq(ProfileUnblock)
case (Some("profile"), _, _, _, Some("mute_user")) =>
Seq(ProfileMute)
case (Some("profile"), _, _, _, Some("report")) =>
Seq(ProfileReport)
case (Some("profile"), _, _, _, Some("show")) =>
Seq(ProfileShow)
case (_, _, _, Some("follow"), Some("click")) => Seq(TweetFollowAuthor)
case (_, _, _, _, Some("follow")) => Seq(TweetFollowAuthor, ProfileFollow)
case (_, _, _, Some("unfollow"), Some("click")) => Seq(TweetUnfollowAuthor)
case (_, _, _, _, Some("unfollow")) => Seq(TweetUnfollowAuthor)
case (_, _, _, Some("block"), Some("click")) => Seq(TweetBlockAuthor)
case (_, _, _, Some("unblock"), Some("click")) => Seq(TweetUnblockAuthor)
case (_, _, _, Some("mute"), Some("click")) => Seq(TweetMuteAuthor)
case (_, _, _, Some(element), Some("click")) if isTweetClickEvent(element) =>
Seq(TweetClick)
case (_, _, _, _, Some("profile_click")) => Seq(TweetClickProfile, ProfileClick)
case (_, _, _, _, Some("share_menu_click")) => Seq(TweetClickShare)
case (_, _, _, _, Some("copy_link")) => Seq(TweetShareViaCopyLink)
case (_, _, _, _, Some("share_via_dm")) => Seq(TweetClickSendViaDirectMessage)
case (_, _, _, _, Some("bookmark")) => Seq(TweetShareViaBookmark, TweetBookmark)
case (_, _, _, _, Some("unbookmark")) => Seq(TweetUnbookmark)
case (_, _, _, _, Some("hashtag_click")) |
// This scribe is triggered on mobile platforms (android/iphone) when user click on hashtag in a tweet.
(_, _, _, Some("hashtag"), Some("search")) =>
Seq(TweetClickHashtag)
case (_, _, _, _, Some("open_link")) => Seq(TweetOpenLink)
case (_, _, _, _, Some("take_screenshot")) => Seq(TweetTakeScreenshot)
case (_, _, _, Some("feedback_notrelevant"), Some("click")) =>
Seq(TweetNotRelevant)
case (_, _, _, Some("feedback_notrelevant"), Some("undo")) =>
Seq(TweetUndoNotRelevant)
case (_, _, _, _, Some("follow_attempt")) => Seq(ProfileFollowAttempt)
case (_, _, _, _, Some("favorite_attempt")) => Seq(TweetFavoriteAttempt)
case (_, _, _, _, Some("retweet_attempt")) => Seq(TweetRetweetAttempt)
case (_, _, _, _, Some("reply_attempt")) => Seq(TweetReplyAttempt)
case (_, _, _, _, Some("login")) => Seq(CTALoginClick)
case (Some("login"), _, _, _, Some("show")) => Seq(CTALoginStart)
case (Some("login"), _, _, _, Some("success")) => Seq(CTALoginSuccess)
case (_, _, _, _, Some("signup")) => Seq(CTASignupClick)
case (Some("signup"), _, _, _, Some("success")) => Seq(CTASignupSuccess)
case // Android app running in the background
(Some("notification"), Some("status_bar"), None, _, Some("background_open")) |
// Android app running in the foreground
(Some("notification"), Some("status_bar"), None, _, Some("open")) |
// iOS app running in the background
(Some("notification"), Some("notification_center"), None, _, Some("open")) |
// iOS app running in the foreground
(None, Some("toasts"), Some("social"), Some("favorite"), Some("open")) |
// m5
(Some("app"), Some("push"), _, _, Some("open")) =>
Seq(NotificationOpen)
case (Some("ntab"), Some("all"), Some("urt"), _, Some("navigate")) =>
Seq(NotificationClick)
case (Some("ntab"), Some("all"), Some("urt"), _, Some("see_less_often")) =>
Seq(NotificationSeeLessOften)
case (Some("notification"), Some("status_bar"), None, _, Some("background_dismiss")) |
(Some("notification"), Some("status_bar"), None, _, Some("dismiss")) | (
Some("notification"),
Some("notification_center"),
None,
_,
Some("dismiss")
) =>
Seq(NotificationDismiss)
case (_, _, _, Some("typeahead"), Some("click")) => Seq(TypeaheadClick)
case (Some("search"), _, Some(component), _, Some("click"))
if component == "relevance_prompt_module" || component == "did_you_find_it_module" =>
Seq(FeedbackPromptSubmit)
case (Some("app"), Some("enter_background"), _, _, Some("become_inactive"))
if logEvent.logBase
.flatMap(_.clientAppId)
.exists(validUASIosClientIds.contains(_)) =>
Seq(AppExit)
case (Some("app"), _, _, _, Some("become_inactive"))
if logEvent.logBase
.flatMap(_.clientAppId)
.exists(validUASAndroidClientIds.contains(_)) =>
Seq(AppExit)
case (_, _, Some("gallery"), Some("photo"), Some("impression")) =>
Seq(TweetGalleryImpression)
case (_, _, _, _, _)
if TweetDetailsImpression.isTweetDetailsImpression(logEvent.eventNamespace) =>
Seq(TweetDetailsImpression)
case _ => Nil
}
}
actionTypesPerEvent.map(_.toUnifiedUserAction(logEvent))
}.flatten
def shouldIgnoreClientEvent(eventNamespace: Option[EventNamespace]): Boolean =
eventNamespace.exists { name =>
(name.page, name.section, name.component, name.element, name.action) match {
case (Some("ddg"), _, _, _, Some("experiment")) => true
case (Some("qig_ranker"), _, _, _, _) => true
case (Some("timelinemixer"), _, _, _, _) => true
case (Some("timelineservice"), _, _, _, _) => true
case (Some("tweetconvosvc"), _, _, _, _) => true
case _ => false
}
}
}

View File

@ -0,0 +1,169 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.EventNamespace
import com.twitter.clientapp.thriftscala.Item
import com.twitter.clientapp.thriftscala.ItemType.User
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.adapter.common.AdapterUtils
import com.twitter.unified_user_actions.thriftscala.AuthorInfo
import com.twitter.unified_user_actions.thriftscala.ClientEventNamespace
import com.twitter.unified_user_actions.thriftscala.EventMetadata
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.SourceLineage
import com.twitter.unified_user_actions.thriftscala.TweetAuthorFollowClickSource
import com.twitter.unified_user_actions.thriftscala.TweetAuthorUnfollowClickSource
import com.twitter.unified_user_actions.thriftscala.TweetInfo
/**
* Comprises helper methods that:
* 1. need not be overridden by subclasses of `BaseClientEvent`
* 2. need not be invoked by instances of subclasses of `BaseClientEvent`
* 3. need to be accessible to subclasses of `BaseClientEvent` and other utils
*/
object ClientEventCommonUtils {
def getBasicTweetInfo(
actionTweetId: Long,
ceItem: LogEventItem,
ceNamespaceOpt: Option[EventNamespace]
): TweetInfo = TweetInfo(
actionTweetId = actionTweetId,
actionTweetTopicSocialProofId = getTopicId(ceItem, ceNamespaceOpt),
retweetingTweetId = ceItem.tweetDetails.flatMap(_.retweetingTweetId),
quotedTweetId = ceItem.tweetDetails.flatMap(_.quotedTweetId),
inReplyToTweetId = ceItem.tweetDetails.flatMap(_.inReplyToTweetId),
quotingTweetId = ceItem.tweetDetails.flatMap(_.quotingTweetId),
// only set AuthorInfo when authorId is present
actionTweetAuthorInfo = getAuthorInfo(ceItem),
retweetingAuthorId = ceItem.tweetDetails.flatMap(_.retweetAuthorId),
quotedAuthorId = ceItem.tweetDetails.flatMap(_.quotedAuthorId),
inReplyToAuthorId = ceItem.tweetDetails.flatMap(_.inReplyToAuthorId),
tweetPosition = ceItem.position,
promotedId = ceItem.promotedId
)
def getTopicId(
ceItem: LogEventItem,
ceNamespaceOpt: Option[EventNamespace] = None,
): Option[Long] =
ceNamespaceOpt.flatMap {
TopicIdUtils.getTopicId(item = ceItem, _)
}
def getAuthorInfo(
ceItem: LogEventItem,
): Option[AuthorInfo] =
ceItem.tweetDetails.flatMap(_.authorId).map { authorId =>
AuthorInfo(
authorId = Some(authorId),
isFollowedByActingUser = ceItem.isViewerFollowsTweetAuthor,
isFollowingActingUser = ceItem.isTweetAuthorFollowsViewer,
)
}
def getEventMetadata(
eventTimestamp: Long,
logEvent: LogEvent,
ceItem: LogEventItem,
productSurface: Option[ProductSurface] = None
): EventMetadata = EventMetadata(
sourceTimestampMs = eventTimestamp,
receivedTimestampMs = AdapterUtils.currentTimestampMs,
sourceLineage = SourceLineage.ClientEvents,
// Client UI language or from Gizmoduck which is what user set in Twitter App.
// Please see more at https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/finatra-internal/international/src/main/scala/com/twitter/finatra/international/LanguageIdentifier.scala
// The format should be ISO 639-1.
language = logEvent.logBase.flatMap(_.language).map(AdapterUtils.normalizeLanguageCode),
// Country code could be IP address (geoduck) or User registration country (gizmoduck) and the former takes precedence.
// We dont know exactly which one is applied, unfortunately,
// see https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/finatra-internal/international/src/main/scala/com/twitter/finatra/international/CountryIdentifier.scala
// The format should be ISO_3166-1_alpha-2.
countryCode = logEvent.logBase.flatMap(_.country).map(AdapterUtils.normalizeCountryCode),
clientAppId = logEvent.logBase.flatMap(_.clientAppId),
clientVersion = logEvent.clientVersion,
clientEventNamespace = logEvent.eventNamespace.map(en => toClientEventNamespace(en)),
traceId = getTraceId(productSurface, ceItem),
requestJoinId = getRequestJoinId(productSurface, ceItem),
clientEventTriggeredOn = logEvent.eventDetails.flatMap(_.triggeredOn)
)
def toClientEventNamespace(eventNamespace: EventNamespace): ClientEventNamespace =
ClientEventNamespace(
page = eventNamespace.page,
section = eventNamespace.section,
component = eventNamespace.component,
element = eventNamespace.element,
action = eventNamespace.action
)
/**
* Get the profileId from Item.id, which itemType = 'USER'.
*
* The profileId can be also be found in the event_details.profile_id.
* However, the item.id is more reliable than event_details.profile_id,
* in particular, 45% of the client events with USER items have
* Null for event_details.profile_id while 0.13% item.id is Null.
* As such, we only use item.id to populate the profile_id.
*/
def getProfileIdFromUserItem(item: Item): Option[Long] =
if (item.itemType.contains(User))
item.id
else None
/**
* TraceId is going to be deprecated and replaced by requestJoinId.
*
* Get the traceId from LogEventItem based on productSurface.
*
* The traceId is hydrated in controller data from backend. Different product surfaces
* populate different controller data. Thus, the product surface is checked first to decide
* which controller data should be read to ge the requestJoinId.
*/
def getTraceId(productSurface: Option[ProductSurface], ceItem: LogEventItem): Option[Long] =
productSurface match {
case Some(ProductSurface.HomeTimeline) => HomeInfoUtils.getTraceId(ceItem)
case Some(ProductSurface.SearchResultsPage) => { new SearchInfoUtils(ceItem) }.getTraceId
case _ => None
}
/**
* Get the requestJoinId from LogEventItem based on productSurface.
*
* The requestJoinId is hydrated in controller data from backend. Different product surfaces
* populate different controller data. Thus, the product surface is checked first to decide
* which controller data should be read to get the requestJoinId.
*
* Support Home / Home_latest / SearchResults for now, to add other surfaces based on requirement.
*/
def getRequestJoinId(productSurface: Option[ProductSurface], ceItem: LogEventItem): Option[Long] =
productSurface match {
case Some(ProductSurface.HomeTimeline) => HomeInfoUtils.getRequestJoinId(ceItem)
case Some(ProductSurface.SearchResultsPage) => {
new SearchInfoUtils(ceItem)
}.getRequestJoinId
case _ => None
}
def getTweetAuthorFollowSource(
eventNamespace: Option[EventNamespace]
): TweetAuthorFollowClickSource = {
eventNamespace
.map(ns => (ns.element, ns.action)).map {
case (Some("follow"), Some("click")) => TweetAuthorFollowClickSource.CaretMenu
case (_, Some("follow")) => TweetAuthorFollowClickSource.ProfileImage
case _ => TweetAuthorFollowClickSource.Unknown
}.getOrElse(TweetAuthorFollowClickSource.Unknown)
}
def getTweetAuthorUnfollowSource(
eventNamespace: Option[EventNamespace]
): TweetAuthorUnfollowClickSource = {
eventNamespace
.map(ns => (ns.element, ns.action)).map {
case (Some("unfollow"), Some("click")) => TweetAuthorUnfollowClickSource.CaretMenu
case (_, Some("unfollow")) => TweetAuthorUnfollowClickSource.ProfileImage
case _ => TweetAuthorUnfollowClickSource.Unknown
}.getOrElse(TweetAuthorUnfollowClickSource.Unknown)
}
}

View File

@ -0,0 +1,687 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.unified_user_actions.thriftscala._
object ClientEventEngagement {
object TweetFav extends BaseClientEvent(ActionType.ClientTweetFav)
/**
* This is fired when a user unlikes a liked(favorited) tweet
*/
object TweetUnfav extends BaseClientEvent(ActionType.ClientTweetUnfav)
/**
* This is "Send Reply" event to indicate publishing of a reply Tweet as opposed to clicking
* on the reply button to initiate a reply Tweet (captured in ClientTweetClickReply).
* The difference between this and the ServerTweetReply are:
* 1) ServerTweetReply already has the new Tweet Id, 2) A sent reply may be lost during transfer
* over the wire and thus may not end up with a follow-up ServerTweetReply.
*/
object TweetReply extends BaseClientEvent(ActionType.ClientTweetReply)
/**
* This is the "send quote" event to indicate publishing of a quote tweet as opposed to clicking
* on the quote button to initiate a quote tweet (captured in ClientTweetClickQuote).
* The difference between this and the ServerTweetQuote are:
* 1) ServerTweetQuote already has the new Tweet Id, 2) A sent quote may be lost during transfer
* over the wire and thus may not ended up with a follow-up ServerTweetQuote.
*/
object TweetQuote extends BaseClientEvent(ActionType.ClientTweetQuote)
/**
* This is the "retweet" event to indicate publishing of a retweet.
*/
object TweetRetweet extends BaseClientEvent(ActionType.ClientTweetRetweet)
/**
* "action = reply" indicates that a user expressed the intention to reply to a Tweet by clicking
* the reply button. No new tweet is created in this event.
*/
object TweetClickReply extends BaseClientEvent(ActionType.ClientTweetClickReply)
/**
* Please note that the "action == quote" is NOT the create quote Tweet event like what
* we can get from TweetyPie.
* It is just click on the "quote tweet" (after clicking on the retweet button there are 2 options,
* one is "retweet" and the other is "quote tweet")
*
* Also checked the CE (BQ Table), the `item.tweet_details.quoting_tweet_id` is always NULL but
* `item.tweet_details.retweeting_tweet_id`, `item.tweet_details.in_reply_to_tweet_id`, `item.tweet_details.quoted_tweet_id`
* could be NON-NULL and UUA would just include these NON-NULL fields as is. This is also checked in the unit test.
*/
object TweetClickQuote extends BaseClientEvent(ActionType.ClientTweetClickQuote)
/**
* Refer to go/cme-scribing and go/interaction-event-spec for details.
* Fired on the first tick of a track regardless of where in the video it is playing.
* For looping playback, this is only fired once and does not reset at loop boundaries.
*/
object TweetVideoPlaybackStart
extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlaybackStart)
/**
* Refer to go/cme-scribing and go/interaction-event-spec for details.
* Fired when playback reaches 100% of total track duration.
* Not valid for live videos.
* For looping playback, this is only fired once and does not reset at loop boundaries.
*/
object TweetVideoPlaybackComplete
extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlaybackComplete)
/**
* Refer to go/cme-scribing and go/interaction-event-spec for details.
* This is fired when playback reaches 25% of total track duration. Not valid for live videos.
* For looping playback, this is only fired once and does not reset at loop boundaries.
*/
object TweetVideoPlayback25 extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlayback25)
object TweetVideoPlayback50 extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlayback50)
object TweetVideoPlayback75 extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlayback75)
object TweetVideoPlayback95 extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlayback95)
/**
* Refer to go/cme-scribing and go/interaction-event-spec for details.
* This if fired when the video has been played in non-preview
* (i.e. not autoplaying in the timeline) mode, and was not started via auto-advance.
* For looping playback, this is only fired once and does not reset at loop boundaries.
*/
object TweetVideoPlayFromTap extends BaseVideoClientEvent(ActionType.ClientTweetVideoPlayFromTap)
/**
* Refer to go/cme-scribing and go/interaction-event-spec for details.
* This is fired when 50% of the video has been on-screen and playing for 10 consecutive seconds
* or 95% of the video duration, whichever comes first.
* For looping playback, this is only fired once and does not reset at loop boundaries.
*/
object TweetVideoQualityView extends BaseVideoClientEvent(ActionType.ClientTweetVideoQualityView)
object TweetVideoView extends BaseVideoClientEvent(ActionType.ClientTweetVideoView)
object TweetVideoMrcView extends BaseVideoClientEvent(ActionType.ClientTweetVideoMrcView)
object TweetVideoViewThreshold
extends BaseVideoClientEvent(ActionType.ClientTweetVideoViewThreshold)
object TweetVideoCtaUrlClick extends BaseVideoClientEvent(ActionType.ClientTweetVideoCtaUrlClick)
object TweetVideoCtaWatchClick
extends BaseVideoClientEvent(ActionType.ClientTweetVideoCtaWatchClick)
/**
* This is fired when a user clicks on "Undo retweet" after re-tweeting a tweet
*
*/
object TweetUnretweet extends BaseClientEvent(ActionType.ClientTweetUnretweet)
/**
* This is fired when a user clicks on a photo attached to a tweet and the photo expands to fit
* the screen.
*/
object TweetPhotoExpand extends BaseClientEvent(ActionType.ClientTweetPhotoExpand)
/**
* This is fired when a user clicks on a card, a card could be a photo or video for example
*/
object CardClick extends BaseCardClientEvent(ActionType.ClientCardClick)
object CardOpenApp extends BaseCardClientEvent(ActionType.ClientCardOpenApp)
object CardAppInstallAttempt extends BaseCardClientEvent(ActionType.ClientCardAppInstallAttempt)
object PollCardVote extends BaseCardClientEvent(ActionType.ClientPollCardVote)
/**
* This is fired when a user clicks on a profile mention inside a tweet.
*/
object TweetClickMentionScreenName
extends BaseClientEvent(ActionType.ClientTweetClickMentionScreenName) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
(
ceItem.id,
logEvent.eventDetails.flatMap(
_.targets.flatMap(_.find(_.itemType.contains(ItemType.User))))) match {
case (Some(tweetId), Some(target)) =>
(target.id, target.name) match {
case (Some(profileId), Some(profileHandle)) =>
Some(
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(tweetId, ceItem, logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetClickMentionScreenName(
ClientTweetClickMentionScreenName(
actionProfileId = profileId,
handle = profileHandle
))))))
case _ => None
}
case _ => None
}
}
/**
* These are fired when user follows/unfollows a Topic. Please see the comment in the
* ClientEventAdapter namespace matching to see the subtle details.
*/
object TopicFollow extends BaseTopicClientEvent(ActionType.ClientTopicFollow)
object TopicUnfollow extends BaseTopicClientEvent(ActionType.ClientTopicUnfollow)
/**
* This is fired when the user clicks the "x" icon next to the topic on their timeline,
* and clicks "Not interested in {TOPIC}" in the pop-up prompt
* Alternatively, they can also click "See more" button to visit the topic page, and click "Not interested" there.
*/
object TopicNotInterestedIn extends BaseTopicClientEvent(ActionType.ClientTopicNotInterestedIn)
/**
* This is fired when the user clicks the "Undo" button after clicking "x" or "Not interested" on a Topic
* which is captured in ClientTopicNotInterestedIn
*/
object TopicUndoNotInterestedIn
extends BaseTopicClientEvent(ActionType.ClientTopicUndoNotInterestedIn)
/**
* This is fired when a user clicks on "This Tweet's not helpful" flow in the caret menu
* of a Tweet result on the Search Results Page
*/
object TweetNotHelpful extends BaseClientEvent(ActionType.ClientTweetNotHelpful)
/**
* This is fired when a user clicks Undo after clicking on
* "This Tweet's not helpful" flow in the caret menu of a Tweet result on the Search Results Page
*/
object TweetUndoNotHelpful extends BaseClientEvent(ActionType.ClientTweetUndoNotHelpful)
object TweetReport extends BaseClientEvent(ActionType.ClientTweetReport) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = {
for {
actionTweetId <- ceItem.id
} yield {
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetReport(
ClientTweetReport(
isReportTweetDone =
logEvent.eventNamespace.flatMap(_.action).exists(_.contains("done")),
reportFlowId = logEvent.reportDetails.flatMap(_.reportFlowId)
)
))))
}
}
}
/**
* Not Interested In (Do Not like) event
*/
object TweetNotInterestedIn extends BaseClientEvent(ActionType.ClientTweetNotInterestedIn)
object TweetUndoNotInterestedIn extends BaseClientEvent(ActionType.ClientTweetUndoNotInterestedIn)
/**
* This is fired when a user FIRST clicks the "Not interested in this Tweet" button in the caret menu of a Tweet
* then clicks "This Tweet is not about {TOPIC}" in the subsequent prompt
* Note: this button is hidden unless a user clicks "Not interested in this Tweet" first.
*/
object TweetNotAboutTopic extends BaseClientEvent(ActionType.ClientTweetNotAboutTopic)
/**
* This is fired when a user clicks "Undo" immediately after clicking "This Tweet is not about {TOPIC}",
* which is captured in TweetNotAboutTopic
*/
object TweetUndoNotAboutTopic extends BaseClientEvent(ActionType.ClientTweetUndoNotAboutTopic)
/**
* This is fired when a user FIRST clicks the "Not interested in this Tweet" button in the caret menu of a Tweet
* then clicks "This Tweet isn't recent" in the subsequent prompt
* Note: this button is hidden unless a user clicks "Not interested in this Tweet" first.
*/
object TweetNotRecent extends BaseClientEvent(ActionType.ClientTweetNotRecent)
/**
* This is fired when a user clicks "Undo" immediately after clicking "his Tweet isn't recent",
* which is captured in TweetNotRecent
*/
object TweetUndoNotRecent extends BaseClientEvent(ActionType.ClientTweetUndoNotRecent)
/**
* This is fired when a user clicks "Not interested in this Tweet" button in the caret menu of a Tweet
* then clicks "Show fewer tweets from" in the subsequent prompt
* Note: this button is hidden unless a user clicks "Not interested in this Tweet" first.
*/
object TweetSeeFewer extends BaseClientEvent(ActionType.ClientTweetSeeFewer)
/**
* This is fired when a user clicks "Undo" immediately after clicking "Show fewer tweets from",
* which is captured in TweetSeeFewer
*/
object TweetUndoSeeFewer extends BaseClientEvent(ActionType.ClientTweetUndoSeeFewer)
/**
* This is fired when a user click "Submit" at the end of a "Report Tweet" flow
* ClientTweetReport = 1041 is scribed by HealthClient team, on the client side
* This is scribed by spamacaw, on the server side
* They can be joined on reportFlowId
* See https://confluence.twitter.biz/pages/viewpage.action?spaceKey=HEALTH&title=Understanding+ReportDetails
*/
object TweetReportServer extends BaseClientEvent(ActionType.ServerTweetReport) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
for {
actionTweetId <- ceItem.id
} yield Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ServerTweetReport(
ServerTweetReport(
reportFlowId = logEvent.reportDetails.flatMap(_.reportFlowId),
reportType = logEvent.reportDetails.flatMap(_.reportType)
)
))))
}
/**
* This is fired when a user clicks Block in a Profile page
* A Profile can also be blocked when a user clicks Block in the menu of a Tweet, which
* is captured in ClientTweetBlockAuthor
*/
object ProfileBlock extends BaseProfileClientEvent(ActionType.ClientProfileBlock)
/**
* This is fired when a user clicks unblock in a pop-up prompt right after blocking a profile
* in the profile page or clicks unblock in a drop-down menu in the profile page.
*/
object ProfileUnblock extends BaseProfileClientEvent(ActionType.ClientProfileUnblock)
/**
* This is fired when a user clicks Mute in a Profile page
* A Profile can also be muted when a user clicks Mute in the menu of a Tweet, which
* is captured in ClientTweetMuteAuthor
*/
object ProfileMute extends BaseProfileClientEvent(ActionType.ClientProfileMute)
/*
* This is fired when a user clicks "Report User" action from user profile page
* */
object ProfileReport extends BaseProfileClientEvent(ActionType.ClientProfileReport)
// This is fired when a user profile is open in a Profile page
object ProfileShow extends BaseProfileClientEvent(ActionType.ClientProfileShow)
object ProfileClick extends BaseProfileClientEvent(ActionType.ClientProfileClick) {
/**
* ClientTweetClickProfile would emit 2 events, 1 with item type Tweet and 1 with item type User
* Both events will go to both actions (the actual classes). For ClientTweetClickProfile,
* item type of Tweet will filter out the event with item type User. But for ClientProfileClick,
* because we need to include item type of User, then we will also include the event of TweetClickProfile
* if we don't do anything here. This override ensures we don't include tweet author clicks events in ProfileClick
*/
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
if (logEvent.eventDetails
.flatMap(_.items).exists(items => items.exists(_.itemType.contains(ItemType.Tweet)))) {
None
} else {
super.getUuaItem(ceItem, logEvent)
}
}
/**
* This is fired when a user follows a profile from the
* profile page / people module and people tab on the Search Results Page / sidebar on the Home page
* A Profile can also be followed when a user clicks follow in the
* caret menu of a Tweet / follow button on hovering on profile avatar,
* which is captured in ClientTweetFollowAuthor
*/
object ProfileFollow extends BaseProfileClientEvent(ActionType.ClientProfileFollow) {
/**
* ClientTweetFollowAuthor would emit 2 events, 1 with item type Tweet and 1 with item type User
* Both events will go to both actions (the actual classes). For ClientTweetFollowAuthor,
* item type of Tweet will filter out the event with item type User. But for ClientProfileFollow,
* because we need to include item type of User, then we will also include the event of TweetFollowAuthor
* if we don't do anything here. This override ensures we don't include tweet author follow events in ProfileFollow
*/
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
if (logEvent.eventDetails
.flatMap(_.items).exists(items => items.exists(_.itemType.contains(ItemType.Tweet)))) {
None
} else {
super.getUuaItem(ceItem, logEvent)
}
}
/**
* This is fired when a user clicks Follow in the caret menu of a Tweet or hovers on the avatar of the tweet author
* and clicks on the Follow button. A profile can also be followed by clicking the Follow button on the Profile
* page and confirm, which is captured in ClientProfileFollow.
* The event emits two items, one of user type and another of tweet type, since the default implementation of
* BaseClientEvent only looks for Tweet type, the other item is dropped which is the expected behaviour
*/
object TweetFollowAuthor extends BaseClientEvent(ActionType.ClientTweetFollowAuthor) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = {
for {
actionTweetId <- ceItem.id
} yield {
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetFollowAuthor(
ClientTweetFollowAuthor(
ClientEventCommonUtils.getTweetAuthorFollowSource(logEvent.eventNamespace))
))))
}
}
}
/**
* This is fired when a user clicks Unfollow in the caret menu of a Tweet or hovers on the avatar of the tweet author
* and clicks on the Unfollow button. A profile can also be unfollowed by clicking the Unfollow button on the Profile
* page and confirm, which will be captured in ClientProfileUnfollow.
* The event emits two items, one of user type and another of tweet type, since the default implementation of
* BaseClientEvent only looks for Tweet type, the other item is dropped which is the expected behaviour
*/
object TweetUnfollowAuthor extends BaseClientEvent(ActionType.ClientTweetUnfollowAuthor) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = {
for {
actionTweetId <- ceItem.id
} yield {
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetUnfollowAuthor(
ClientTweetUnfollowAuthor(
ClientEventCommonUtils.getTweetAuthorUnfollowSource(logEvent.eventNamespace))
))))
}
}
}
/**
* This is fired when a user clicks Block in the caret menu of a Tweet to block the profile
* that authors this Tweet. A profile can also be blocked in the Profile page, which is captured
* in ClientProfileBlock
*/
object TweetBlockAuthor extends BaseClientEvent(ActionType.ClientTweetBlockAuthor)
/**
* This is fired when a user clicks unblock in a pop-up prompt right after blocking an author
* in the drop-down menu of a tweet
*/
object TweetUnblockAuthor extends BaseClientEvent(ActionType.ClientTweetUnblockAuthor)
/**
* This is fired when a user clicks Mute in the caret menu of a Tweet to mute the profile
* that authors this Tweet. A profile can also be muted in the Profile page, which is captured
* in ClientProfileMute
*/
object TweetMuteAuthor extends BaseClientEvent(ActionType.ClientTweetMuteAuthor)
/**
* This is fired when a user clicks on a Tweet to open the Tweet details page. Note that for
* Tweets in the Notification Tab product surface, a click can be registered differently
* depending on whether the Tweet is a rendered Tweet (a click results in ClientTweetClick)
* or a wrapper Notification (a click results in ClientNotificationClick).
*/
object TweetClick extends BaseClientEvent(ActionType.ClientTweetClick)
/**
* This is fired when a user clicks to view the profile page of another user from a Tweet
*/
object TweetClickProfile extends BaseClientEvent(ActionType.ClientTweetClickProfile)
/**
* This is fired when a user clicks on the "share" icon on a Tweet to open the share menu.
* The user may or may not proceed and finish sharing the Tweet.
*/
object TweetClickShare extends BaseClientEvent(ActionType.ClientTweetClickShare)
/**
* This is fired when a user clicks "Copy link to Tweet" in a menu appeared after hitting
* the "share" icon on a Tweet OR when a user selects share_via -> copy_link after long-click
* a link inside a tweet on a mobile device
*/
object TweetShareViaCopyLink extends BaseClientEvent(ActionType.ClientTweetShareViaCopyLink)
/**
* This is fired when a user clicks "Send via Direct Message" after
* clicking on the "share" icon on a Tweet to open the share menu.
* The user may or may not proceed and finish Sending the DM.
*/
object TweetClickSendViaDirectMessage
extends BaseClientEvent(ActionType.ClientTweetClickSendViaDirectMessage)
/**
* This is fired when a user clicks "Bookmark" after
* clicking on the "share" icon on a Tweet to open the share menu.
*/
object TweetShareViaBookmark extends BaseClientEvent(ActionType.ClientTweetShareViaBookmark)
/**
* This is fired when a user clicks "Remove Tweet from Bookmarks" after
* clicking on the "share" icon on a Tweet to open the share menu.
*/
object TweetUnbookmark extends BaseClientEvent(ActionType.ClientTweetUnbookmark)
/**
* This event is fired when the user clicks on a hashtag in a Tweet.
*/
object TweetClickHashtag extends BaseClientEvent(ActionType.ClientTweetClickHashtag) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = for {
actionTweetId <- ceItem.id
} yield Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = logEvent.eventDetails
.map(
_.targets.flatMap(_.headOption.flatMap(_.name))
) // fetch the first item in the details and then the name will have the hashtag value with the '#' sign
.map { hashtagOpt =>
TweetActionInfo.ClientTweetClickHashtag(
ClientTweetClickHashtag(hashtag = hashtagOpt)
)
}))
}
/**
* This is fired when a user clicks "Bookmark" after clicking on the "share" icon on a Tweet to
* open the share menu, or when a user clicks on the 'bookmark' icon on a Tweet (bookmark icon
* is available to ios only as of March 2023).
* TweetBookmark and TweetShareByBookmark log the same events but serve for individual use cases.
*/
object TweetBookmark extends BaseClientEvent(ActionType.ClientTweetBookmark)
/**
* This is fired when a user clicks on a link in a tweet.
* The link could be displayed as a URL or embedded
* in a component such as an image or a card in a tweet.
*/
object TweetOpenLink extends BaseClientEvent(ActionType.ClientTweetOpenLink) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
for {
actionTweetId <- ceItem.id
} yield Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetOpenLink(
ClientTweetOpenLink(url = logEvent.eventDetails.flatMap(_.url))
))))
}
/**
* This is fired when a user takes a screenshot.
* This is available for only mobile clients.
*/
object TweetTakeScreenshot extends BaseClientEvent(ActionType.ClientTweetTakeScreenshot) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] =
for {
actionTweetId <- ceItem.id
} yield Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(
actionTweetId = actionTweetId,
ceItem = ceItem,
ceNamespaceOpt = logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetTakeScreenshot(
ClientTweetTakeScreenshot(percentVisibleHeight100k = ceItem.percentVisibleHeight100k)
))))
}
/**
* This is fired when a user clicks the "This Tweet isn't relevant" button in a prompt displayed
* after clicking "This Tweet's not helpful" in search result page or "Not Interested in this Tweet"
* in the home timeline page.
* Note: this button is hidden unless a user clicks "This Tweet isn't relevant" or
* "This Tweet's not helpful" first
*/
object TweetNotRelevant extends BaseClientEvent(ActionType.ClientTweetNotRelevant)
/**
* This is fired when a user clicks "Undo" immediately after clicking "this Tweet isn't relevant",
* which is captured in TweetNotRelevant
*/
object TweetUndoNotRelevant extends BaseClientEvent(ActionType.ClientTweetUndoNotRelevant)
/**
* This is fired when a user is logged out and follows a profile from the
* profile page / people module from web.
* One can only try to follow from web, iOS and Android do not support logged out browsing
*/
object ProfileFollowAttempt extends BaseProfileClientEvent(ActionType.ClientProfileFollowAttempt)
/**
* This is fired when a user is logged out and favourite a tweet from web.
* One can only try to favourite from web, iOS and Android do not support logged out browsing
*/
object TweetFavoriteAttempt extends BaseClientEvent(ActionType.ClientTweetFavoriteAttempt)
/**
* This is fired when a user is logged out and Retweet a tweet from web.
* One can only try to favourite from web, iOS and Android do not support logged out browsing
*/
object TweetRetweetAttempt extends BaseClientEvent(ActionType.ClientTweetRetweetAttempt)
/**
* This is fired when a user is logged out and reply on tweet from web.
* One can only try to favourite from web, iOS and Android do not support logged out browsing
*/
object TweetReplyAttempt extends BaseClientEvent(ActionType.ClientTweetReplyAttempt)
/**
* This is fired when a user is logged out and clicks on login button.
* Currently seem to be generated only on [m5, LiteNativeWrapper] as of Jan 2023.
*/
object CTALoginClick extends BaseCTAClientEvent(ActionType.ClientCTALoginClick)
/**
* This is fired when a user is logged out and login window is shown.
*/
object CTALoginStart extends BaseCTAClientEvent(ActionType.ClientCTALoginStart)
/**
* This is fired when a user is logged out and login is successful.
*/
object CTALoginSuccess extends BaseCTAClientEvent(ActionType.ClientCTALoginSuccess)
/**
* This is fired when a user is logged out and clicks on signup button.
*/
object CTASignupClick extends BaseCTAClientEvent(ActionType.ClientCTASignupClick)
/**
* This is fired when a user is logged out and signup is successful.
*/
object CTASignupSuccess extends BaseCTAClientEvent(ActionType.ClientCTASignupSuccess)
/**
* This is fired when a user opens a Push Notification.
* Refer to https://confluence.twitter.biz/pages/viewpage.action?pageId=161811800
* for Push Notification scribe details
*/
object NotificationOpen extends BasePushNotificationClientEvent(ActionType.ClientNotificationOpen)
/**
* This is fired when a user clicks on a notification in the Notification Tab.
* Refer to go/ntab-urt-scribe for Notification Tab scribe details.
*/
object NotificationClick
extends BaseNotificationTabClientEvent(ActionType.ClientNotificationClick)
/**
* This is fired when a user taps the "See Less Often" caret menu item of a notification in
* the Notification Tab.
* Refer to go/ntab-urt-scribe for Notification Tab scribe details.
*/
object NotificationSeeLessOften
extends BaseNotificationTabClientEvent(ActionType.ClientNotificationSeeLessOften)
/**
* This is fired when a user closes or swipes away a Push Notification.
* Refer to https://confluence.twitter.biz/pages/viewpage.action?pageId=161811800
* for Push Notification scribe details
*/
object NotificationDismiss
extends BasePushNotificationClientEvent(ActionType.ClientNotificationDismiss)
/**
* This is fired when a user clicks on a typeahead suggestion(queries, events, topics, users)
* in a drop-down menu of a search box or a tweet compose box.
*/
object TypeaheadClick extends BaseSearchTypeaheadEvent(ActionType.ClientTypeaheadClick)
/**
* This is a generic event fired when the user submits feedback on a prompt.
* Some examples include Did You Find It Prompt and Tweet Relevance on Search Results Page.
*/
object FeedbackPromptSubmit
extends BaseFeedbackSubmitClientEvent(ActionType.ClientFeedbackPromptSubmit)
object AppExit extends BaseUASClientEvent(ActionType.ClientAppExit)
}

View File

@ -0,0 +1,207 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.EventNamespace
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.thriftscala._
import com.twitter.unified_user_actions.thriftscala.Item.TweetInfo
object ClientEventImpression {
object TweetLingerImpression extends BaseClientEvent(ActionType.ClientTweetLingerImpression) {
override def getUuaItem(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[Item] = {
for {
actionTweetId <- ceItem.id
impressionDetails <- ceItem.impressionDetails
lingerStartTimestampMs <- impressionDetails.visibilityStart
lingerEndTimestampMs <- impressionDetails.visibilityEnd
} yield {
Item.TweetInfo(
ClientEventCommonUtils
.getBasicTweetInfo(actionTweetId, ceItem, logEvent.eventNamespace)
.copy(tweetActionInfo = Some(
TweetActionInfo.ClientTweetLingerImpression(
ClientTweetLingerImpression(
lingerStartTimestampMs = lingerStartTimestampMs,
lingerEndTimestampMs = lingerEndTimestampMs
)
))))
}
}
}
/**
* To make parity with iesource's definition, render impression for quoted Tweets would emit
* 2 events: 1 for the quoting Tweet and 1 for the original Tweet!!!
*/
object TweetRenderImpression extends BaseClientEvent(ActionType.ClientTweetRenderImpression) {
override def toUnifiedUserAction(logEvent: LogEvent): Seq[UnifiedUserAction] = {
val logBase: Option[LogBase] = logEvent.logBase
val raw = for {
ed <- logEvent.eventDetails.toSeq
items <- ed.items.toSeq
ceItem <- items
eventTimestamp <- logBase.flatMap(getSourceTimestamp)
uuaItem <- getUuaItem(ceItem, logEvent)
if isItemTypeValid(ceItem.itemType)
} yield {
val userIdentifier: UserIdentifier = UserIdentifier(
userId = logBase.flatMap(_.userId),
guestIdMarketing = logBase.flatMap(_.guestIdMarketing))
val productSurface: Option[ProductSurface] = ProductSurfaceUtils
.getProductSurface(logEvent.eventNamespace)
val eventMetaData: EventMetadata = ClientEventCommonUtils
.getEventMetadata(
eventTimestamp = eventTimestamp,
logEvent = logEvent,
ceItem = ceItem,
productSurface = productSurface
)
UnifiedUserAction(
userIdentifier = userIdentifier,
item = uuaItem,
actionType = ActionType.ClientTweetRenderImpression,
eventMetadata = eventMetaData,
productSurface = productSurface,
productSurfaceInfo =
ProductSurfaceUtils.getProductSurfaceInfo(productSurface, ceItem, logEvent)
)
}
raw.flatMap { e =>
e.item match {
case TweetInfo(t) =>
// If it is an impression toward quoted Tweet we emit 2 impressions, 1 for quoting Tweet
// and 1 for the original Tweet.
if (t.quotedTweetId.isDefined) {
val originalItem = t.copy(
actionTweetId = t.quotedTweetId.get,
actionTweetAuthorInfo = t.quotedAuthorId.map(id => AuthorInfo(authorId = Some(id))),
quotingTweetId = Some(t.actionTweetId),
quotedTweetId = None,
inReplyToTweetId = None,
replyingTweetId = None,
retweetingTweetId = None,
retweetedTweetId = None,
quotedAuthorId = None,
retweetingAuthorId = None,
inReplyToAuthorId = None
)
val original = e.copy(item = TweetInfo(originalItem))
Seq(original, e)
} else Seq(e)
case _ => Nil
}
}
}
}
object TweetGalleryImpression extends BaseClientEvent(ActionType.ClientTweetGalleryImpression)
object TweetDetailsImpression extends BaseClientEvent(ActionType.ClientTweetDetailsImpression) {
case class EventNamespaceInternal(
client: String,
page: String,
section: String,
component: String,
element: String,
action: String)
def isTweetDetailsImpression(eventNamespaceOpt: Option[EventNamespace]): Boolean =
eventNamespaceOpt.exists { eventNamespace =>
val eventNamespaceInternal = EventNamespaceInternal(
client = eventNamespace.client.getOrElse(""),
page = eventNamespace.page.getOrElse(""),
section = eventNamespace.section.getOrElse(""),
component = eventNamespace.component.getOrElse(""),
element = eventNamespace.element.getOrElse(""),
action = eventNamespace.action.getOrElse(""),
)
isIphoneAppOrMacAppOrIpadAppClientTweetDetailsImpression(
eventNamespaceInternal) || isAndroidAppClientTweetDetailsImpression(
eventNamespaceInternal) || isWebClientTweetDetailImpression(
eventNamespaceInternal) || isTweetDeckAppClientTweetDetailsImpression(
eventNamespaceInternal) || isOtherAppClientTweetDetailsImpression(eventNamespaceInternal)
}
private def isWebClientTweetDetailImpression(
eventNamespace: EventNamespaceInternal
): Boolean = {
val eventNameSpaceStr =
eventNamespace.client + ":" + eventNamespace.page + ":" + eventNamespace.section + ":" + eventNamespace.component + ":" + eventNamespace.element + ":" + eventNamespace.action
eventNameSpaceStr.equalsIgnoreCase("m5:tweet::::show") || eventNameSpaceStr.equalsIgnoreCase(
"m5:tweet:landing:::show") || eventNameSpaceStr
.equalsIgnoreCase("m2:tweet::::impression") || eventNameSpaceStr.equalsIgnoreCase(
"m2:tweet::tweet::impression") || eventNameSpaceStr
.equalsIgnoreCase("LiteNativeWrapper:tweet::::show") || eventNameSpaceStr.equalsIgnoreCase(
"LiteNativeWrapper:tweet:landing:::show")
}
private def isOtherAppClientTweetDetailsImpression(
eventNamespace: EventNamespaceInternal
): Boolean = {
val excludedClients = Set(
"web",
"m5",
"m2",
"LiteNativeWrapper",
"iphone",
"ipad",
"mac",
"android",
"android_tablet",
"deck")
(!excludedClients.contains(eventNamespace.client)) && eventNamespace.page
.equalsIgnoreCase("tweet") && eventNamespace.section
.equalsIgnoreCase("") && eventNamespace.component
.equalsIgnoreCase("tweet") && eventNamespace.element
.equalsIgnoreCase("") && eventNamespace.action.equalsIgnoreCase("impression")
}
private def isTweetDeckAppClientTweetDetailsImpression(
eventNamespace: EventNamespaceInternal
): Boolean =
eventNamespace.client
.equalsIgnoreCase("deck") && eventNamespace.page
.equalsIgnoreCase("tweet") && eventNamespace.section
.equalsIgnoreCase("") && eventNamespace.component
.equalsIgnoreCase("tweet") && eventNamespace.element
.equalsIgnoreCase("") && eventNamespace.action.equalsIgnoreCase("impression")
private def isAndroidAppClientTweetDetailsImpression(
eventNamespace: EventNamespaceInternal
): Boolean =
(eventNamespace.client
.equalsIgnoreCase("android") || eventNamespace.client
.equalsIgnoreCase("android_tablet")) && eventNamespace.page
.equalsIgnoreCase("tweet") && eventNamespace.section.equalsIgnoreCase(
"") && (eventNamespace.component
.equalsIgnoreCase("tweet") || eventNamespace.component
.matches("^suggest.*_tweet.*$") || eventNamespace.component
.equalsIgnoreCase("")) && eventNamespace.element
.equalsIgnoreCase("") && eventNamespace.action.equalsIgnoreCase("impression")
private def isIphoneAppOrMacAppOrIpadAppClientTweetDetailsImpression(
eventNamespace: EventNamespaceInternal
): Boolean =
(eventNamespace.client
.equalsIgnoreCase("iphone") || eventNamespace.client
.equalsIgnoreCase("ipad") || eventNamespace.client
.equalsIgnoreCase("mac")) && eventNamespace.page.equalsIgnoreCase(
"tweet") && eventNamespace.section
.equalsIgnoreCase("") && (eventNamespace.component
.equalsIgnoreCase("tweet") || eventNamespace.component
.matches("^suggest.*_tweet.*$")) && eventNamespace.element
.equalsIgnoreCase("") && eventNamespace.action.equalsIgnoreCase("impression")
}
}

View File

@ -0,0 +1,32 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerData
import com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerDataAliases.V1Alias
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
object HomeInfoUtils {
def getHomeTweetControllerDataV1(ceItem: LogEventItem): Option[V1Alias] = {
ceItem.suggestionDetails
.flatMap(_.decodedControllerData)
.flatMap(_ match {
case ControllerData.V2(
ControllerDataV2.HomeTweets(
HomeTweetsControllerData.V1(homeTweetsControllerDataV1)
)) =>
Some(homeTweetsControllerDataV1)
case _ => None
})
}
def getTraceId(ceItem: LogEventItem): Option[Long] =
getHomeTweetControllerDataV1(ceItem).flatMap(_.traceId)
def getSuggestType(ceItem: LogEventItem): Option[String] =
ceItem.suggestionDetails.flatMap(_.suggestionType)
def getRequestJoinId(ceItem: LogEventItem): Option[Long] =
getHomeTweetControllerDataV1(ceItem).flatMap(_.requestJoinId)
}

View File

@ -0,0 +1,40 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.ItemType
object ItemTypeFilterPredicates {
private val TweetItemTypes = Set[ItemType](ItemType.Tweet, ItemType.QuotedTweet)
private val TopicItemTypes = Set[ItemType](ItemType.Tweet, ItemType.QuotedTweet, ItemType.Topic)
private val ProfileItemTypes = Set[ItemType](ItemType.User)
private val TypeaheadResultItemTypes = Set[ItemType](ItemType.Search, ItemType.User)
private val SearchResultsPageFeedbackSubmitItemTypes =
Set[ItemType](ItemType.Tweet, ItemType.RelevancePrompt)
/**
* DDG lambda metrics count Tweets based on the `itemType`
* Reference code - https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/src/scala/com/twitter/experiments/lambda/shared/Timelines.scala?L156
* Since enums `PROMOTED_TWEET` and `POPULAR_TWEET` are deprecated in the following thrift
* https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/src/thrift/com/twitter/clientapp/gen/client_app.thrift?L131
* UUA filters two types of Tweets only: `TWEET` and `QUOTED_TWEET`
*/
def isItemTypeTweet(itemTypeOpt: Option[ItemType]): Boolean =
itemTypeOpt.exists(itemType => TweetItemTypes.contains(itemType))
def isItemTypeTopic(itemTypeOpt: Option[ItemType]): Boolean =
itemTypeOpt.exists(itemType => TopicItemTypes.contains(itemType))
def isItemTypeProfile(itemTypeOpt: Option[ItemType]): Boolean =
itemTypeOpt.exists(itemType => ProfileItemTypes.contains(itemType))
def isItemTypeTypeaheadResult(itemTypeOpt: Option[ItemType]): Boolean =
itemTypeOpt.exists(itemType => TypeaheadResultItemTypes.contains(itemType))
def isItemTypeForSearchResultsPageFeedbackSubmit(itemTypeOpt: Option[ItemType]): Boolean =
itemTypeOpt.exists(itemType => SearchResultsPageFeedbackSubmitItemTypes.contains(itemType))
/**
* Always return true. Use this when there is no need to filter based on `item_type` and all
* values of `item_type` are acceptable.
*/
def ignoreItemType(itemTypeOpt: Option[ItemType]): Boolean = true
}

View File

@ -0,0 +1,26 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
object NotificationClientEventUtils {
// Notification id for notification in the Notification Tab
def getNotificationIdForNotificationTab(
ceItem: LogEventItem
): Option[String] = {
for {
notificationTabDetails <- ceItem.notificationTabDetails
clientEventMetaData <- notificationTabDetails.clientEventMetadata
notificationId <- clientEventMetaData.upstreamId
} yield {
notificationId
}
}
// Notification id for Push Notification
def getNotificationIdForPushNotification(logEvent: LogEvent): Option[String] = for {
pushNotificationDetails <- logEvent.notificationDetails
notificationId <- pushNotificationDetails.impressionId
} yield notificationId
}

View File

@ -0,0 +1,109 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.EventNamespace
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerDataAliases.V1Alias
import com.twitter.unified_user_actions.thriftscala._
object ProductSurfaceUtils {
def getProductSurface(eventNamespace: Option[EventNamespace]): Option[ProductSurface] = {
(
eventNamespace.flatMap(_.page),
eventNamespace.flatMap(_.section),
eventNamespace.flatMap(_.element)) match {
case (Some("home") | Some("home_latest"), _, _) => Some(ProductSurface.HomeTimeline)
case (Some("ntab"), _, _) => Some(ProductSurface.NotificationTab)
case (Some(page), Some(section), _) if isPushNotification(page, section) =>
Some(ProductSurface.PushNotification)
case (Some("search"), _, _) => Some(ProductSurface.SearchResultsPage)
case (_, _, Some("typeahead")) => Some(ProductSurface.SearchTypeahead)
case _ => None
}
}
private def isPushNotification(page: String, section: String): Boolean = {
Seq[String]("notification", "toasts").contains(page) ||
(page == "app" && section == "push")
}
def getProductSurfaceInfo(
productSurface: Option[ProductSurface],
ceItem: LogEventItem,
logEvent: LogEvent
): Option[ProductSurfaceInfo] = {
productSurface match {
case Some(ProductSurface.HomeTimeline) => createHomeTimelineInfo(ceItem)
case Some(ProductSurface.NotificationTab) => createNotificationTabInfo(ceItem)
case Some(ProductSurface.PushNotification) => createPushNotificationInfo(logEvent)
case Some(ProductSurface.SearchResultsPage) => createSearchResultPageInfo(ceItem, logEvent)
case Some(ProductSurface.SearchTypeahead) => createSearchTypeaheadInfo(ceItem, logEvent)
case _ => None
}
}
private def createPushNotificationInfo(logEvent: LogEvent): Option[ProductSurfaceInfo] =
NotificationClientEventUtils.getNotificationIdForPushNotification(logEvent) match {
case Some(notificationId) =>
Some(
ProductSurfaceInfo.PushNotificationInfo(
PushNotificationInfo(notificationId = notificationId)))
case _ => None
}
private def createNotificationTabInfo(ceItem: LogEventItem): Option[ProductSurfaceInfo] =
NotificationClientEventUtils.getNotificationIdForNotificationTab(ceItem) match {
case Some(notificationId) =>
Some(
ProductSurfaceInfo.NotificationTabInfo(
NotificationTabInfo(notificationId = notificationId)))
case _ => None
}
private def createHomeTimelineInfo(ceItem: LogEventItem): Option[ProductSurfaceInfo] = {
def suggestType: Option[String] = HomeInfoUtils.getSuggestType(ceItem)
def controllerData: Option[V1Alias] = HomeInfoUtils.getHomeTweetControllerDataV1(ceItem)
if (suggestType.isDefined || controllerData.isDefined) {
Some(
ProductSurfaceInfo.HomeTimelineInfo(
HomeTimelineInfo(
suggestionType = suggestType,
injectedPosition = controllerData.flatMap(_.injectedPosition)
)))
} else None
}
private def createSearchResultPageInfo(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[ProductSurfaceInfo] = {
val searchInfoUtil = new SearchInfoUtils(ceItem)
searchInfoUtil.getQueryOptFromItem(logEvent).map { query =>
ProductSurfaceInfo.SearchResultsPageInfo(
SearchResultsPageInfo(
query = query,
querySource = searchInfoUtil.getQuerySourceOptFromControllerDataFromItem,
itemPosition = ceItem.position,
tweetResultSources = searchInfoUtil.getTweetResultSources,
userResultSources = searchInfoUtil.getUserResultSources,
queryFilterType = searchInfoUtil.getQueryFilterType(logEvent)
))
}
}
private def createSearchTypeaheadInfo(
ceItem: LogEventItem,
logEvent: LogEvent
): Option[ProductSurfaceInfo] = {
logEvent.searchDetails.flatMap(_.query).map { query =>
ProductSurfaceInfo.SearchTypeaheadInfo(
SearchTypeaheadInfo(
query = query,
itemPosition = ceItem.position
)
)
}
}
}

View File

@ -0,0 +1,129 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.LogEvent
import com.twitter.clientapp.thriftscala.{Item => LogEventItem}
import com.twitter.search.common.constants.thriftscala.ThriftQuerySource
import com.twitter.search.common.constants.thriftscala.TweetResultSource
import com.twitter.search.common.constants.thriftscala.UserResultSource
import com.twitter.suggests.controller_data.search_response.item_types.thriftscala.ItemTypesControllerData
import com.twitter.suggests.controller_data.search_response.item_types.thriftscala.ItemTypesControllerData.TweetTypesControllerData
import com.twitter.suggests.controller_data.search_response.item_types.thriftscala.ItemTypesControllerData.UserTypesControllerData
import com.twitter.suggests.controller_data.search_response.request.thriftscala.RequestControllerData
import com.twitter.suggests.controller_data.search_response.thriftscala.SearchResponseControllerData.V1
import com.twitter.suggests.controller_data.search_response.thriftscala.SearchResponseControllerDataAliases.V1Alias
import com.twitter.suggests.controller_data.thriftscala.ControllerData.V2
import com.twitter.suggests.controller_data.v2.thriftscala.ControllerData.SearchResponse
import com.twitter.unified_user_actions.thriftscala.SearchQueryFilterType
import com.twitter.unified_user_actions.thriftscala.SearchQueryFilterType._
class SearchInfoUtils(item: LogEventItem) {
private val searchControllerDataOpt: Option[V1Alias] = item.suggestionDetails.flatMap { sd =>
sd.decodedControllerData.flatMap { decodedControllerData =>
decodedControllerData match {
case V2(v2ControllerData) =>
v2ControllerData match {
case SearchResponse(searchResponseControllerData) =>
searchResponseControllerData match {
case V1(searchResponseControllerDataV1) =>
Some(searchResponseControllerDataV1)
case _ => None
}
case _ =>
None
}
case _ => None
}
}
}
private val requestControllerDataOptFromItem: Option[RequestControllerData] =
searchControllerDataOpt.flatMap { searchControllerData =>
searchControllerData.requestControllerData
}
private val itemTypesControllerDataOptFromItem: Option[ItemTypesControllerData] =
searchControllerDataOpt.flatMap { searchControllerData =>
searchControllerData.itemTypesControllerData
}
def checkBit(bitmap: Long, idx: Int): Boolean = {
(bitmap / Math.pow(2, idx)).toInt % 2 == 1
}
def getQueryOptFromSearchDetails(logEvent: LogEvent): Option[String] = {
logEvent.searchDetails.flatMap { sd => sd.query }
}
def getQueryOptFromControllerDataFromItem: Option[String] = {
requestControllerDataOptFromItem.flatMap { rd => rd.rawQuery }
}
def getQueryOptFromItem(logEvent: LogEvent): Option[String] = {
// First we try to get the query from controller data, and if that's not available, we fall
// back to query in search details. If both are None, queryOpt is None.
getQueryOptFromControllerDataFromItem.orElse(getQueryOptFromSearchDetails(logEvent))
}
def getTweetTypesOptFromControllerDataFromItem: Option[TweetTypesControllerData] = {
itemTypesControllerDataOptFromItem.flatMap { itemTypes =>
itemTypes match {
case TweetTypesControllerData(tweetTypesControllerData) =>
Some(TweetTypesControllerData(tweetTypesControllerData))
case _ => None
}
}
}
def getUserTypesOptFromControllerDataFromItem: Option[UserTypesControllerData] = {
itemTypesControllerDataOptFromItem.flatMap { itemTypes =>
itemTypes match {
case UserTypesControllerData(userTypesControllerData) =>
Some(UserTypesControllerData(userTypesControllerData))
case _ => None
}
}
}
def getQuerySourceOptFromControllerDataFromItem: Option[ThriftQuerySource] = {
requestControllerDataOptFromItem
.flatMap { rd => rd.querySource }
.flatMap { querySourceVal => ThriftQuerySource.get(querySourceVal) }
}
def getTweetResultSources: Option[Set[TweetResultSource]] = {
getTweetTypesOptFromControllerDataFromItem
.flatMap { cd => cd.tweetTypesControllerData.tweetTypesBitmap }
.map { tweetTypesBitmap =>
TweetResultSource.list.filter { t => checkBit(tweetTypesBitmap, t.value) }.toSet
}
}
def getUserResultSources: Option[Set[UserResultSource]] = {
getUserTypesOptFromControllerDataFromItem
.flatMap { cd => cd.userTypesControllerData.userTypesBitmap }
.map { userTypesBitmap =>
UserResultSource.list.filter { t => checkBit(userTypesBitmap, t.value) }.toSet
}
}
def getQueryFilterType(logEvent: LogEvent): Option[SearchQueryFilterType] = {
val searchTab = logEvent.eventNamespace.map(_.client).flatMap {
case Some("m5") | Some("android") => logEvent.eventNamespace.flatMap(_.element)
case _ => logEvent.eventNamespace.flatMap(_.section)
}
searchTab.flatMap {
case "search_filter_top" => Some(Top)
case "search_filter_live" => Some(Latest)
// android uses search_filter_tweets instead of search_filter_live
case "search_filter_tweets" => Some(Latest)
case "search_filter_user" => Some(People)
case "search_filter_image" => Some(Photos)
case "search_filter_video" => Some(Videos)
case _ => None
}
}
def getRequestJoinId: Option[Long] = requestControllerDataOptFromItem.flatMap(_.requestJoinId)
def getTraceId: Option[Long] = requestControllerDataOptFromItem.flatMap(_.traceId)
}

View File

@ -0,0 +1,157 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.EventNamespace
import com.twitter.clientapp.thriftscala.Item
import com.twitter.clientapp.thriftscala.ItemType.Topic
import com.twitter.guide.scribing.thriftscala.TopicModuleMetadata
import com.twitter.guide.scribing.thriftscala.TransparentGuideDetails
import com.twitter.suggests.controller_data.home_hitl_topic_annotation_prompt.thriftscala.HomeHitlTopicAnnotationPromptControllerData
import com.twitter.suggests.controller_data.home_hitl_topic_annotation_prompt.v1.thriftscala.{
HomeHitlTopicAnnotationPromptControllerData => HomeHitlTopicAnnotationPromptControllerDataV1
}
import com.twitter.suggests.controller_data.home_topic_annotation_prompt.thriftscala.HomeTopicAnnotationPromptControllerData
import com.twitter.suggests.controller_data.home_topic_annotation_prompt.v1.thriftscala.{
HomeTopicAnnotationPromptControllerData => HomeTopicAnnotationPromptControllerDataV1
}
import com.twitter.suggests.controller_data.home_topic_follow_prompt.thriftscala.HomeTopicFollowPromptControllerData
import com.twitter.suggests.controller_data.home_topic_follow_prompt.v1.thriftscala.{
HomeTopicFollowPromptControllerData => HomeTopicFollowPromptControllerDataV1
}
import com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerData
import com.twitter.suggests.controller_data.home_tweets.v1.thriftscala.{
HomeTweetsControllerData => HomeTweetsControllerDataV1
}
import com.twitter.suggests.controller_data.search_response.item_types.thriftscala.ItemTypesControllerData
import com.twitter.suggests.controller_data.search_response.thriftscala.SearchResponseControllerData
import com.twitter.suggests.controller_data.search_response.topic_follow_prompt.thriftscala.SearchTopicFollowPromptControllerData
import com.twitter.suggests.controller_data.search_response.tweet_types.thriftscala.TweetTypesControllerData
import com.twitter.suggests.controller_data.search_response.v1.thriftscala.{
SearchResponseControllerData => SearchResponseControllerDataV1
}
import com.twitter.suggests.controller_data.thriftscala.ControllerData
import com.twitter.suggests.controller_data.timelines_topic.thriftscala.TimelinesTopicControllerData
import com.twitter.suggests.controller_data.timelines_topic.v1.thriftscala.{
TimelinesTopicControllerData => TimelinesTopicControllerDataV1
}
import com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}
import com.twitter.util.Try
object TopicIdUtils {
val DomainId: Long = 131 // Topical Domain
def getTopicId(
item: Item,
namespace: EventNamespace
): Option[Long] =
getTopicIdFromHomeSearch(item)
.orElse(getTopicFromGuide(item))
.orElse(getTopicFromOnboarding(item, namespace))
.orElse(getTopicIdFromItem(item))
def getTopicIdFromItem(item: Item): Option[Long] =
if (item.itemType.contains(Topic))
item.id
else None
def getTopicIdFromHomeSearch(
item: Item
): Option[Long] = {
val decodedControllerData = item.suggestionDetails.flatMap(_.decodedControllerData)
decodedControllerData match {
case Some(
ControllerData.V2(
ControllerDataV2.HomeTweets(
HomeTweetsControllerData.V1(homeTweets: HomeTweetsControllerDataV1)))
) =>
homeTweets.topicId
case Some(
ControllerData.V2(
ControllerDataV2.HomeTopicFollowPrompt(
HomeTopicFollowPromptControllerData.V1(
homeTopicFollowPrompt: HomeTopicFollowPromptControllerDataV1)))
) =>
homeTopicFollowPrompt.topicId
case Some(
ControllerData.V2(
ControllerDataV2.TimelinesTopic(
TimelinesTopicControllerData.V1(
timelinesTopic: TimelinesTopicControllerDataV1
)))
) =>
Some(timelinesTopic.topicId)
case Some(
ControllerData.V2(
ControllerDataV2.SearchResponse(
SearchResponseControllerData.V1(s: SearchResponseControllerDataV1)))
) =>
s.itemTypesControllerData match {
case Some(
ItemTypesControllerData.TopicFollowControllerData(
topicFollowControllerData: SearchTopicFollowPromptControllerData)) =>
topicFollowControllerData.topicId
case Some(
ItemTypesControllerData.TweetTypesControllerData(
tweetTypesControllerData: TweetTypesControllerData)) =>
tweetTypesControllerData.topicId
case _ => None
}
case Some(
ControllerData.V2(
ControllerDataV2.HomeTopicAnnotationPrompt(
HomeTopicAnnotationPromptControllerData.V1(
homeTopicAnnotationPrompt: HomeTopicAnnotationPromptControllerDataV1
)))
) =>
Some(homeTopicAnnotationPrompt.topicId)
case Some(
ControllerData.V2(
ControllerDataV2.HomeHitlTopicAnnotationPrompt(
HomeHitlTopicAnnotationPromptControllerData.V1(
homeHitlTopicAnnotationPrompt: HomeHitlTopicAnnotationPromptControllerDataV1
)))
) =>
Some(homeHitlTopicAnnotationPrompt.topicId)
case _ => None
}
}
def getTopicFromOnboarding(
item: Item,
namespace: EventNamespace
): Option[Long] =
if (namespace.page.contains("onboarding") &&
(namespace.section.exists(_.contains("topic")) ||
namespace.component.exists(_.contains("topic")) ||
namespace.element.exists(_.contains("topic")))) {
item.description.flatMap { description =>
// description: "id=123,main=xyz,row=1"
val tokens = description.split(",").headOption.map(_.split("="))
tokens match {
case Some(Array("id", token, _*)) => Try(token.toLong).toOption
case _ => None
}
}
} else None
def getTopicFromGuide(
item: Item
): Option[Long] =
item.guideItemDetails.flatMap {
_.transparentGuideDetails match {
case Some(TransparentGuideDetails.TopicMetadata(topicMetadata)) =>
topicMetadata match {
case TopicModuleMetadata.TttInterest(_) =>
None
case TopicModuleMetadata.SemanticCoreInterest(semanticCoreInterest) =>
if (semanticCoreInterest.domainId == DomainId.toString)
Try(semanticCoreInterest.entityId.toLong).toOption
else None
case TopicModuleMetadata.SimClusterInterest(_) =>
None
case TopicModuleMetadata.UnknownUnionField(_) => None
}
case _ => None
}
}
}

View File

@ -0,0 +1,42 @@
package com.twitter.unified_user_actions.adapter.client_event
import com.twitter.clientapp.thriftscala.AmplifyDetails
import com.twitter.clientapp.thriftscala.MediaDetails
import com.twitter.unified_user_actions.thriftscala.TweetVideoWatch
import com.twitter.unified_user_actions.thriftscala.TweetActionInfo
import com.twitter.video.analytics.thriftscala.MediaIdentifier
object VideoClientEventUtils {
/**
* For Tweets with multiple videos, find the id of the video that generated the client-event
*/
def videoIdFromMediaIdentifier(mediaIdentifier: MediaIdentifier): Option[String] =
mediaIdentifier match {
case MediaIdentifier.MediaPlatformIdentifier(mediaPlatformIdentifier) =>
mediaPlatformIdentifier.mediaId.map(_.toString)
case _ => None
}
/**
* Given:
* 1. the id of the video (`mediaId`)
* 2. details about all the media items in the Tweet (`mediaItems`),
* iterate over the `mediaItems` to lookup the metadata about the video with id `mediaId`.
*/
def getVideoMetadata(
mediaId: String,
mediaItems: Seq[MediaDetails],
amplifyDetails: Option[AmplifyDetails]
): Option[TweetActionInfo] = {
mediaItems.collectFirst {
case media if media.contentId.contains(mediaId) =>
TweetActionInfo.TweetVideoWatch(
TweetVideoWatch(
mediaType = media.mediaType,
isMonetizable = media.dynamicAds,
videoType = amplifyDetails.flatMap(_.videoType)
))
}
}
}

View File

@ -0,0 +1,15 @@
package com.twitter.unified_user_actions.adapter.common
import com.twitter.snowflake.id.SnowflakeId
import com.twitter.util.Time
object AdapterUtils {
def currentTimestampMs: Long = Time.now.inMilliseconds
def getTimestampMsFromTweetId(tweetId: Long): Long = SnowflakeId.unixTimeMillisFromId(tweetId)
// For now just make sure both language code and country code are in upper cases for consistency
// For language code, there are mixed lower and upper cases
// For country code, there are mixed lower and upper cases
def normalizeLanguageCode(inputLanguageCode: String): String = inputLanguageCode.toUpperCase
def normalizeCountryCode(inputCountryCode: String): String = inputCountryCode.toUpperCase
}

View File

@ -0,0 +1,10 @@
scala_library(
sources = [
"*.scala",
],
tags = ["bazel-compatible"],
dependencies = [
"snowflake/src/main/scala/com/twitter/snowflake/id",
"util/util-core:util-core-util",
],
)

View File

@ -0,0 +1,14 @@
scala_library(
sources = [
"*.scala",
],
compiler_option_sets = ["fatal_warnings"],
tags = ["bazel-compatible"],
dependencies = [
"kafka/finagle-kafka/finatra-kafka/src/main/scala",
"src/thrift/com/twitter/ibis:logging-scala",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter:base",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter/common",
"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala",
],
)

View File

@ -0,0 +1,55 @@
package com.twitter.unified_user_actions.adapter.email_notification_event
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.finagle.stats.StatsReceiver
import com.twitter.finatra.kafka.serde.UnKeyed
import com.twitter.ibis.thriftscala.NotificationScribe
import com.twitter.ibis.thriftscala.NotificationScribeType
import com.twitter.unified_user_actions.adapter.AbstractAdapter
import com.twitter.unified_user_actions.thriftscala.ActionType
import com.twitter.unified_user_actions.thriftscala.EmailNotificationInfo
import com.twitter.unified_user_actions.thriftscala.Item
import com.twitter.unified_user_actions.thriftscala.ProductSurface
import com.twitter.unified_user_actions.thriftscala.ProductSurfaceInfo
import com.twitter.unified_user_actions.thriftscala.TweetInfo
import com.twitter.unified_user_actions.thriftscala.UnifiedUserAction
import com.twitter.unified_user_actions.thriftscala.UserIdentifier
class EmailNotificationEventAdapter
extends AbstractAdapter[NotificationScribe, UnKeyed, UnifiedUserAction] {
import EmailNotificationEventAdapter._
override def adaptOneToKeyedMany(
input: NotificationScribe,
statsReceiver: StatsReceiver = NullStatsReceiver
): Seq[(UnKeyed, UnifiedUserAction)] =
adaptEvent(input).map { e => (UnKeyed, e) }
}
object EmailNotificationEventAdapter {
def adaptEvent(scribe: NotificationScribe): Seq[UnifiedUserAction] = {
Option(scribe).flatMap { e =>
e.`type` match {
case NotificationScribeType.Click =>
val tweetIdOpt = e.logBase.flatMap(EmailNotificationEventUtils.extractTweetId)
(tweetIdOpt, e.impressionId) match {
case (Some(tweetId), Some(impressionId)) =>
Some(
UnifiedUserAction(
userIdentifier = UserIdentifier(userId = e.userId),
item = Item.TweetInfo(TweetInfo(actionTweetId = tweetId)),
actionType = ActionType.ClientTweetEmailClick,
eventMetadata = EmailNotificationEventUtils.extractEventMetaData(e),
productSurface = Some(ProductSurface.EmailNotification),
productSurfaceInfo = Some(
ProductSurfaceInfo.EmailNotificationInfo(
EmailNotificationInfo(notificationId = impressionId)))
)
)
case _ => None
}
case _ => None
}
}.toSeq
}
}

View File

@ -0,0 +1,39 @@
package com.twitter.unified_user_actions.adapter.email_notification_event
import com.twitter.ibis.thriftscala.NotificationScribe
import com.twitter.logbase.thriftscala.LogBase
import com.twitter.unified_user_actions.adapter.common.AdapterUtils
import com.twitter.unified_user_actions.thriftscala.EventMetadata
import com.twitter.unified_user_actions.thriftscala.SourceLineage
object EmailNotificationEventUtils {
/*
* Extract TweetId from Logbase.page, here is a sample page below
* https://twitter.com/i/events/1580827044245544962?cn=ZmxleGlibGVfcmVjcw%3D%3D&refsrc=email
* */
def extractTweetId(path: String): Option[Long] = {
val ptn = raw".*/([0-9]+)\\??.*".r
path match {
case ptn(tweetId) =>
Some(tweetId.toLong)
case _ =>
None
}
}
def extractTweetId(logBase: LogBase): Option[Long] = logBase.page match {
case Some(path) => extractTweetId(path)
case None => None
}
def extractEventMetaData(scribe: NotificationScribe): EventMetadata =
EventMetadata(
sourceTimestampMs = scribe.timestamp,
receivedTimestampMs = AdapterUtils.currentTimestampMs,
sourceLineage = SourceLineage.EmailNotificationEvents,
language = scribe.logBase.flatMap(_.language),
countryCode = scribe.logBase.flatMap(_.country),
clientAppId = scribe.logBase.flatMap(_.clientAppId),
)
}

View File

@ -0,0 +1,14 @@
scala_library(
sources = [
"*.scala",
],
compiler_option_sets = ["fatal_warnings"],
tags = ["bazel-compatible"],
dependencies = [
"fanoutservice/thrift/src/main/thrift:thrift-scala",
"kafka/finagle-kafka/finatra-kafka/src/main/scala",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter:base",
"unified_user_actions/adapter/src/main/scala/com/twitter/unified_user_actions/adapter/common",
"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala",
],
)

Some files were not shown because too many files have changed in this diff Show More