I would to investigate the robustness of the estimated effects using a structural topic model (STM). The off-the-shelf approach involves estimating the topic proportions based on covariate values as shown in approach 1). An alternative way shown in 2) could be to descriptively compare the topic proportions between groups based on covariate values. I would think that the results from my topic model are robust if the topic model generates similar results in terms of the topic proportions in 1) and 2). In the simple replicable example below, this seems to be the case.
library(stminsights)
library(ggplot2)
library(dplyr)
library(stm)
library(tidytext)
# 1) STM effect estimation approach
prep <- estimateEffect(1:3 ~ treatment + pid_rep, gadarianFit, gadarian)
effects <- get_effects(estimates = prep,
variable = 'treatment',
type = 'pointestimate')
gadarian$doc_id <- 1:nrow(gadarian)
# 2) descriptive comparison of the topic proportions
gamma <- tidy(gadarianFit, matrix = "gamma", document_names = gadarian$doc_id)
gamma <- left_join(gamma, gadarian, by = c("document" = "doc_id"))
gamma %>%
group_by(treatment, topic) %>%
summarise(gamma = sum(gamma)) %>%
group_by(treatment) %>%
mutate(gamma = gamma/sum(gamma)) %>%
arrange(topic, -treatment)
`summarise()` has grouped output by 'treatment'. You can override using the `.groups` argument.
# A tibble: 6 × 3
# Groups: treatment [2]
treatment topic gamma
<dbl> <int> <dbl>
1 1 1 0.277
2 0 1 0.456
3 1 2 0.468
4 0 2 0.196
5 1 3 0.255
6 0 3 0.348
> effects
# A tibble: 6 × 5
value proportion topic lower upper
<fct> <dbl> <fct> <dbl> <dbl>
1 1 0.287 1 0.242 0.333
2 0 0.441 1 0.397 0.490
3 1 0.451 2 0.402 0.498
4 0 0.209 2 0.167 0.251
5 1 0.262 3 0.223 0.300
6 0 0.351 3 0.309 0.393
But using different data (which I unfortunately cannot share), this is not the case. What may be the reasons for the lack of robustness? Which results should I trust? See e.g. for topic 5 where approach 1) estimates the topic proportion for treatment == 0 to be bigger than for treatment == 1 while approach 2) estimates treatment == 1 to be bigger. The same is the case for group 4.
What can cause this? What does it tell me?
# A tibble: 440 × 3
# Groups: treatment [2]
treatment topic gamma
<dbl> <dbl> <dbl>
1 0 1 0.00737
2 1 1 0.00747
3 0 2 0.00488
4 1 2 0.00494
5 0 3 0.00397
6 1 3 0.00407
7 0 4 0.00434
8 1 4 0.00416
9 0 5 0.00235
10 1 5 0.00222
# ℹ 430 more rows
# ℹ Use `print(n = ...)` to see more rows
> effects
# A tibble: 440 × 5
value proportion topic lower upper
<fct> <dbl> <fct> <dbl> <dbl>
1 0 0.00663 1 0.00640 0.00687
2 1 0.00690 1 0.00652 0.00726
3 0 0.00442 2 0.00429 0.00457
4 1 0.00390 2 0.00371 0.00409
5 0 0.00388 3 0.00370 0.00405
6 1 0.00362 3 0.00336 0.00391
7 0 0.00392 4 0.00360 0.00423
8 1 0.00578 4 0.00513 0.00629
9 0 0.00322 5 0.00288 0.00361
10 1 0.00283 5 0.00228 0.00338