COVID-19 Message Framing Project
Message overview & descriptives# load packages
if (!require(tidyverse)) {
install.packages('tidyverse')
}
if (!require(knitr)) {
install.packages('knitr')
}
if (!require(kableExtra)) {
install.packages('kableExtra')
}
if (!require(DT)) {
install.packages('DT')
}
if (!require(brms)) {
install.packages('brms')
}
if (!require(tidybayes)) {
install.packages('tidybayes')
}
# define palettes
palette = wesanderson::wes_palette("Zissou1", n = 3, type = "continuous")
palette_message = ggsci::pal_simpsons()(15)
# load and tidy data
data = read.csv("../covid19_study1/covid19_study1_clean_long.csv", stringsAsFactors = FALSE) %>%
bind_rows(read.csv("../covid19_study1_pilot2/covid19_study1_pilot2_clean_long.csv", stringsAsFactors = FALSE)) %>%
bind_rows(read.csv("../covid19_study1_pilot/covid19_study1_pilot_clean_long.csv", stringsAsFactors = FALSE)) %>%
mutate(condition = ifelse(condition == "message control", "control", condition),
group = ifelse(is.na(group), condition, group)) %>%
filter(!grepl("time", survey_name)) %>%
mutate(SID = sprintf("%s_%s", study, SID))
# load survey item info and merge with data
items = read.csv("item_text.csv", stringsAsFactors = FALSE)
# subset and tidy message data
messages = data %>%
mutate(value = as.numeric(value)) %>%
filter(grepl("msg", survey_name)) %>%
filter(!grepl("time", survey_name)) %>%
extract(item, "item", "msg_.*_(.*)") %>%
spread(survey_name, value) %>%
mutate(msg_favorability = ((msg_positive - msg_negative) / 2) + 4) %>%
select(-msg_negative, -msg_positive) %>%
gather(survey_name, value, contains("msg")) %>%
mutate(item = sprintf("%s_%s", survey_name, item)) %>%
extract(item, "message", "msg_.*_([0-9]{2})", remove = FALSE)
merged = messages %>%
left_join(., items, by = "item") %>%
mutate(value = as.numeric(value),
text = ifelse(survey_name == "msg_familiarity", filter(items, item == "msg_familiarity")$text,
ifelse(survey_name == "msg_rel_self", filter(items, item == "msg_rel_self")$text,
ifelse(survey_name == "msg_rel_social", filter(items, item == "msg_rel_social")$text,
ifelse(survey_name == "msg_motiv_self", filter(items, item == "msg_motiv_self")$text,
ifelse(survey_name == "msg_motiv_other", filter(items, item == "msg_motiv_other")$text,
ifelse(survey_name == "msg_share", filter(items, item == "msg_share")$text, text)))))),
measure = ifelse(survey_name == "msg_familiarity", filter(items, item == "msg_familiarity")$measure,
ifelse(survey_name == "msg_rel_self", filter(items, item == "msg_rel_self")$measure,
ifelse(survey_name == "msg_rel_social", filter(items, item == "msg_rel_social")$measure,
ifelse(survey_name == "msg_motiv_self", filter(items, item == "msg_motiv_self")$measure,
ifelse(survey_name == "msg_motiv_other", filter(items, item == "msg_motiv_other")$measure,
ifelse(survey_name == "msg_favorability", "Message favorability",
ifelse(survey_name == "msg_share", filter(items, item == "msg_share")$measure, measure)))))))) %>%
filter(!is.na(message)) ######## CHECK THIS ##########
# subset studies
pilot1 = merged %>%
filter(study == "study1_pilot1")
pilot2 = merged %>%
filter(study == "study1_pilot2")
study1 = merged %>%
filter(study == "study1")
# tidy data for brm models
data_pred = merged %>%
filter(!survey_name == "msg_familiarity") %>%
group_by(survey_name) %>%
mutate(value_scaled = scale(value, center = TRUE, scale = TRUE)) %>%
select(study, condition, group, SID, survey_name, message, value, value_scaled)
In this report, we present descriptives for the COVID-19 health messages used in this project. This information is collapsed across message framing conditions. For reports related to specific framing conditions, please see the individual condition reports. For an overview of the project as a whole, please see the project overview reports.
In each study, participants were randomly assigned to either a message framing intervention group (using autonomy-supportive language, encouraging or mocking humor, or descriptive norms, a fact-based control), or a group that saw no messages. Each participant in the intervention and fact-based message control groups saw a series of 5 messages about social distancing related to COVID-19 randomly sampled from a pool of 15 messages previously normed for argument strength (M = 4.16, SD = 0.18, possible range = 1-5). For more information about message norming, please view the reports on argument stength and humor norming.
Each message was created to look like an instagram post that included a visual message (collected online) about COVID-19 accompanied by a “post” about the message. For each message, the post began with the same stem (e.g., “Staying home protects our community by stopping the spread of #covid19.”). The message control condition contained this stem only, whereas the experimental conditions contained additional text framing the messages. Participants then completed various outcome and individual differences measures.
Below are the control messages used in this project to illustrate the range of images included; all message stimuli in this project used the same main images, and condition specific variants can be viewed by clicking on the links under each study (below the “message framing conditions” sub-heading).
These 15 stimuli focused on social distancing were selected from a larger pool health messages, which included additional behaviors related to COVID-19 (e.g. hand washing, flattening the curve, combating misperceptions). For more information about these messages, see the argument strength norming report.
message_info = read.csv("message_info.csv", stringsAsFactors = FALSE) %>%
rename("message_text" = text) %>%
mutate(condition = gsub(" ", "", condition),
condition = tolower(condition),
condition = ifelse(condition == "control", "control",
ifelse(condition == "descriptivenorm", "norm",
ifelse(condition == "humor(mocking)", "mocking",
ifelse(condition == "humor(encouraging)", "encouraging", condition))))) %>%
mutate(message = sprintf("%02d", as.numeric(image_number))) %>%
filter(message %in% c("06", "07", "08", "09", "10",
"12", "15", "17", "18", "20",
"23", "24", "25", "31", "32")) %>%
mutate(behavior = "social_distancing") %>%
arrange(message) %>%
select(condition, message, message_text, behavior)
message_num = messages %>%
filter(study == "study1_pilot1") %>%
extract(item, "message", "msg_.*_([0-9]{2})", remove = FALSE)
message_info %>%
filter(message %in% message_num$message & condition %in% message_num$group) %>%
select(message, condition, message_text) %>%
arrange(message, condition) %>%
DT::datatable(rownames = FALSE, extensions = 'FixedColumns',
options = list(scrollX = TRUE,
scrollY = TRUE),
caption = htmltools::tags$caption(style = 'caption-side: top; text-align: left;',
"Message text by condition"))
message_num = messages %>%
filter(study == "study1_pilot2") %>%
extract(item, "message", "msg_.*_([0-9]{2})", remove = FALSE)
message_info %>%
filter(message %in% message_num$message & condition %in% message_num$group) %>%
select(message, condition, message_text) %>%
arrange(message, condition) %>%
DT::datatable(rownames = FALSE, extensions = 'FixedColumns',
options = list(scrollX = TRUE,
scrollY = TRUE),
caption = htmltools::tags$caption(style = 'caption-side: top; text-align: left;',
"Message text by condition"))
message_num = messages %>%
filter(study == "study1") %>%
extract(item, "message", "msg_.*_([0-9]{2})", remove = FALSE)
message_info %>%
filter(message %in% message_num$message & condition %in% message_num$group) %>%
select(message, condition, message_text) %>%
arrange(message, condition) %>%
DT::datatable(rownames = FALSE, extensions = 'FixedColumns',
options = list(scrollX = TRUE,
scrollY = TRUE),
caption = htmltools::tags$caption(style = 'caption-side: top; text-align: left;',
"Message text by condition"))
Below are descriptive statistics and response distributions for the message questions in this project.
FIGURE LEGEND
For each message question, individual responses, the density distribution, and a box and whisker plot are shown for each health message.
Each dot represents the response for one individual person.
The box and whisker plots depicted below represent the following statistics: the vertical line in the middle is the median, the box encompasses the inter-quartile range (25th to 75th percentile), and the whiskers capture +/- 1.5 times the inter-quartile range from the box hinge. Points outside the whiskers are likely outliers.
These plots are generated across studies, as well as for each study individually.
plot_desc = function(data, survey, item=TRUE,
palette=palette, min=1, max=7,
text_size=3.5, alpha=.1) {
source("https://gist.githubusercontent.com/benmarwick/2a1bb0133ff568cbe28d/raw/fb53bd97121f7f9ce947837ef1a4c65a73bffb3f/geom_flat_violin.R")
if (item == TRUE) {
labels = data %>%
filter(grepl(!!(survey), survey_name)) %>%
group_by(survey_name, text, message) %>%
summarize(y = (max + min) / 2) %>%
ungroup() %>%
mutate(text = sprintf("message %s", message))
data %>%
filter(grepl(!!(survey), survey_name)) %>%
ggplot(aes(message, value)) +
geom_flat_violin(position = position_nudge(x = .1, y = 0), color = FALSE, fill = palette[1]) +
geom_point(position = position_jitter(width = .05), size = .5, alpha = alpha, color = palette[1]) +
geom_boxplot(width = .1, outlier.shape = NA, alpha = 0) +
geom_text(data = labels, aes(x = message, y = y, label = text), nudge_x = .55, size = text_size) +
coord_flip() +
scale_x_discrete(expand = expansion(add = c(0, .75))) +
labs(x = "", y = "rating") +
theme_minimal() +
theme(axis.text.y = element_blank(),
legend.position = "top")
} else {
labels = data %>%
filter(grepl(!!(survey), survey_name)) %>%
group_by(survey_name, text) %>%
summarize(y = (max + min) / 2) %>%
data %>%
filter(grepl(!!(survey), survey_name)) %>%
ggplot(aes(survey_name, value)) +
geom_flat_violin(position = position_nudge(x = .1, y = 0), color = FALSE, fill = palette[1]) +
geom_point(position = position_jitter(width = .05), size = .5, alpha = alpha, color = palette[1]) +
geom_boxplot(width = .1, outlier.shape = NA, alpha = 0) +
geom_text(data = labels, aes(x = survey_name, y = y, label = text), nudge_x = .55, size = text_size) +
coord_flip() +
scale_x_discrete(expand = expansion(add = c(0, .75))) +
labs(x = "", y = "rating") +
theme_minimal() +
theme(axis.text.y = element_blank(),
legend.position = "top")
}
}
table_desc = function(data, survey, condition=FALSE) {
if (condition == FALSE) {
data %>%
filter(grepl(!!(survey), survey_name)) %>%
filter(!is.na(value)) %>%
group_by(survey_name, message) %>%
summarize(n = n(),
min = min(value, na.rm = TRUE),
max = max(value, na.rm = TRUE),
mean = mean(value, na.rm = TRUE),
median = median(value, na.rm = TRUE),
sd = sd(value, na.rm = TRUE)) %>%
mutate_if(is.numeric, round, 2) %>%
DT::datatable(rownames = FALSE, extensions = 'FixedColumns',
options = list(scrollX = TRUE,
scrollY = TRUE,
fixedColumns = list(leftColumns = 2)))
} else {
data %>%
filter(grepl(!!(survey), survey_name)) %>%
filter(!is.na(value)) %>%
group_by(survey_name, message, group) %>%
summarize(n = n(),
min = min(value, na.rm = TRUE),
max = max(value, na.rm = TRUE),
mean = mean(value, na.rm = TRUE),
median = median(value, na.rm = TRUE),
sd = sd(value, na.rm = TRUE)) %>%
mutate_if(is.numeric, round, 2) %>%
DT::datatable(rownames = FALSE, extensions = 'FixedColumns',
options = list(scrollX = TRUE,
scrollY = TRUE,
fixedColumns = list(leftColumns = 2)))
}
}
measure
Message familiarity
question
Have you seen content like this before?
scale
1 = yes, 0 = no
measure
Message favorability
questions
1. The message put thoughts in my mind about wanting to avoid physical contact with others.
2. The message put thoughts in my mind about not wanting to avoid physical contact with others.
Message favorability was calculated as follows to convert the difference between items to a 7-point scale:
((item 1 - item 2) / 2) + 4
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self-relevance
question
This message is relevant to me
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self motivation to social distance
question
This message motivates me to avoid physical contact with others.
scale
1 = strongly disagree, 7 = strongly agree
measure
Message others motivation to social distance
question
This message would motivate people I know to avoid physical contact with others.
scale
1 = strongly disagree, 7 = strongly agree
Below are correlation matrices for the message outcomes for each message separately, collapsed across studies.
Please refer to the message information section for the message key.
FIGURE LEGEND
Correlations below are Pearson correlations using pair-wise removal for missing data. These correlations are generated across studies, using the raw (un-winsorized data).
measures
Message sharing intention
Message self-relevance
Message social relevance
Message self motivation to social distance
Message others motivation to social distance
Message favorability
scale
1 = strongly disagree, 7 = strongly agree
plot_corr = function(data, message, text_size=3) {
data %>%
select(-value_scaled) %>%
filter(grepl(!!(message), message)) %>%
unite("survey_name", survey_name, message) %>%
spread(survey_name, value) %>%
select(-c(SID, study)) %>%
GGally::ggcorr(., hjust = 1, size = text_size,
label = TRUE, label_size = 3, label_round = 2) +
scale_x_discrete(expand = expansion(add = c(3, 0))) +
theme(legend.position = "none",
axis.title.x = element_blank(),
axis.text.x = element_blank(),
axis.ticks.x = element_blank())
}
Below are scatterplots between a single message question and the other message questions in this project, collapsed across studies.
Please refer to the message information section for the message key.
FIGURE LEGEND
Each individual plot shows the association between the ratings for a single measure (x-axis) and the ratings for another measures (y-axis). These associations are depicted across studies for each message separately. These data were not winsorized.
plot_scatter = function(data, survey, alpha=.1) {
data %>%
select(-value_scaled) %>%
ungroup() %>%
mutate(study = ifelse(study == "study1", "full study 1",
ifelse(study == "study1_pilot1", "pilot 1", "pilot 2"))) %>%
spread(survey_name, value) %>%
gather(survey_name, value, -SID, -study, -group, -condition, -message, -!!(survey)) %>%
ggplot(aes(get(survey), value, color = message)) +
geom_point(alpha = alpha, size = .25, position = position_jitter(width = .2, height = .2)) +
geom_smooth(method = "lm", alpha = 0, size = .5, fullrange = TRUE) +
facet_wrap(~survey_name, scales = "free") +
scale_color_manual(values = palette_message, guide = guide_legend(direction = "horizontal",
title.position = "top",
nrow = 5)) +
labs(x = sprintf("\n%s", survey), y = "rating\n") +
theme_minimal(base_size = 12) +
theme(legend.position = c(.83, .225),
legend.direction = "horizontal")
}
measure
Message favorability
questions
1. The message put thoughts in my mind about wanting to avoid physical contact with others.
2. The message put thoughts in my mind about not wanting to avoid physical contact with others.
Message favorability was calculated as follows to convert the difference between items to a 7-point scale:
((item 1 - item 2) / 2) + 4
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self-relevance
question
This message is relevant to me
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self motivation to social distance
question
This message motivates me to avoid physical contact with others.
scale
1 = strongly disagree, 7 = strongly agree
measure
Message others motivation to social distance
question
This message would motivate people I know to avoid physical contact with others.
scale
1 = strongly disagree, 7 = strongly agree
Below are plots comparing the effect sizes of each message predictor for each message outcome, collapsed across studies. In other words, we examined the effect of exposure to each message variant on message-level outcomes. More positive coefficients indicate higher than average effectiveness; negative coefficients indicate lower than average effectiveness, for each outcome.
Please refer to the message information section for the message key.
FIGURE LEGEND
Using Bayesian multilevel modeling, we estimated the effect of each message predictor (y-axis) on the measure specified in the tab. These plots depict the posterior distribution for each predictor, as well as the mean standardized parameter estimate and probability intervals. The thicker line represents the 66% probability interval and the thiner line represents the 95% probability interval. The mean estimate and 95% probability interval are also listed on the left side of the plot.
run_brm = function(data, x, y, survey,
x_diff=.2, text_size=3, palette) {
data = data %>%
select(SID, survey_name, !!(x), !!(y)) %>%
filter(survey_name %in% !!(survey)) %>%
rename("x" = !!(x),
"y" = !!(y))
if (file.exists(sprintf("model_message_%s.RDS", survey))) {
model = readRDS(sprintf("model_message_%s.RDS", survey))
} else {
model = brms::brm(y ~ 0 + (1 | x), data, cores = 8, iter = 1000, silent = TRUE)
saveRDS(model, sprintf("model_message_%s.RDS", survey))
}
model_data = model %>%
spread_draws(r_x[condition,term]) %>%
ungroup() %>%
mutate(condition_mean = r_x,
condition = sprintf("%02d", condition)) %>%
group_by(condition) %>%
mutate(mean = mean(condition_mean, na.rm = TRUE))
labels = model %>%
spread_draws(r_x[condition,term]) %>%
ungroup() %>%
mutate(condition_mean = r_x,
condition = sprintf("%02d", condition)) %>%
group_by(condition) %>%
tidybayes::mean_qi(condition_mean) %>%
mutate(label = sprintf("%.2f [%.2f, %.2f]", condition_mean, .lower, .upper)) %>%
group_by(.point) %>%
mutate(x = min(.lower) - x_diff)
plot = model_data %>%
ggplot(aes(x = condition_mean, y = reorder(condition, mean))) +
geom_vline(xintercept = 0, alpha = .5) +
stat_halfeyeh(fill = palette[1]) +
geom_text(data = labels, aes(label = label, x = x, y = condition), size = text_size) +
scale_x_continuous(expand = expansion(add = c(.12, 0))) +
labs(x = "\nstandardized parameter estimate", y = "message\n") +
theme_minimal() +
theme(legend.position = "top")
return(list(model = model, plot = plot))
}
# define common variables
x = "message"
y = "value_scaled"
measure
Message favorability
questions
1. The message put thoughts in my mind about wanting to avoid physical contact with others.
2. The message put thoughts in my mind about not wanting to avoid physical contact with others.
Message favorability was calculated as follows to convert the difference between items to a 7-point scale:
((item 1 - item 2) / 2) + 4
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self-relevance
question
This message is relevant to me
scale
1 = strongly disagree, 7 = strongly agree
measure
Message self motivation to social distance
question
This message motivates me to avoid physical contact with others.
scale
1 = strongly disagree, 7 = strongly agree
social relevance
measure
Message social relevance
question
This message is relevant to other people I know
scale
1 = strongly disagree, 7 = strongly agree
all studies
table
figure
pilot 1
table
figure
pilot 2
table
figure
study 1
table
figure