Getting started

I first load the required packages: tidyverse for general purpose wrangling, patchwork for combining plots, jsonlite for processing JSON data, and psyarxivr for the data.

library(tidyverse)
library(patchwork)
library(jsonlite)
library(psyarxivr)

Data wrangling

The psyarxivr package, when loaded as above, provides access to the preprints table. Below, we take our focal variables id (for each preprint) and contributors, which contains the contributor data as JSON strings.

# Parse contributors JSON variable into its own table with preprint ids
contributors <- preprints |>
  # Remove preprints with no contributor data and non-latest versions
  filter(contributors != "[]", is_latest_version == 1) |>
  # Select required variables only
  select(id, contributors) |>
  # Convert JSON into data frames in a list-column
  mutate(
    contributors = map(
      contributors,
      fromJSON
    )
  )

# Unnest into a table of contributors and clean
contributors <- contributors |>
  unnest(contributors) |>
  # Only include bibliographic authors
  filter(bibliographic) |>
  # Remove some other contributor variables and rename
  select(id, name = full_name) |>
  # Take out unnamed contributors
  filter(name != "")

# Calculate total number of contributors
contributors_total <- nrow(contributors)

In this post I’m only interested in my own coauthorship network. The contributors table now has rows for each preprint’s contributors. I first filter it to only include preprints on which I was a coauthor (retaining all authors of the preprints):

my_coauthors <- contributors |>
  # Retain all preprints where any of the authors was me
  filter(any(name == "Matti Vuorre"), .by = id)

my_coauthors
## # A tibble: 90 × 2
##   id       name               
##   <chr>    <chr>              
## 1 qrjza_v1 Niklas Johannes    
## 2 qrjza_v1 Matti Vuorre       
## 3 qrjza_v1 Andrew K Przybylski
## # ℹ 87 more rows

Above, id is the preprint’s OSF ID, and each has one or more names of the preprints’ contributors.

If I created a graph from these data, it would only show my immediate collaborators, but I wish to expand it to show their other collaborators as well. To do so, I filter the original contributor data to include only preprints that have one or more authors who appear as contributors on any of my preprints:

my_coauthors_coauthors <- contributors |>
  # Retain all preprints where any author was my coauthor
  filter(any(name %in% unique(my_coauthors$name)), .by = id)

my_coauthors_coauthors
## # A tibble: 1,195 × 2
##   id       name                  
##   <chr>    <chr>                 
## 1 zzbka_v1 Andrew K Przybylski   
## 2 zzbka_v1 Antonius J. van Rooij 
## 3 zzbka_v1 Michelle Colder Carras
## # ℹ 1,192 more rows

There are 1195 individual contributions to preprints that were either coauthored by me, or any of my coauthors. This is the data I’ll use to construct what I call my second-degree PsyArXiv coauthor network graph.

library(tidygraph)
library(ggraph)

# Get all pairs of co-authors for each paper and count collaborations
edges <- my_coauthors_coauthors |>
  group_by(id) |>
  # Create all pairwise combinations within each paper
  reframe(expand.grid(
    author1 = name,
    author2 = name,
    stringsAsFactors = FALSE
  )) |>
  # Remove self-loops and order pairs for undirected edges
  filter(author1 < author2) |>
  rename(from = author1, to = author2)

edges
## # A tibble: 2,624 × 3
##   id       from         to             
##   <chr>    <chr>        <chr>          
## 1 2erwy_v1 Harm Veling  Niklas Johannes
## 2 2erwy_v1 Jonas Dora   Niklas Johannes
## 3 2erwy_v1 Adrian Meier Niklas Johannes
## # ℹ 2,621 more rows

# Create graph with key metrics
graph <- edges |>
  as_tbl_graph(directed = FALSE) |>
  mutate(
    distance = factor(node_distance_from(name == "Matti Vuorre"))
  )

graph
## # A tbl_graph: 567 nodes and 2624 edges
## #
## # An undirected multigraph with 1 component
## #
## # Node Data: 567 × 2 (active)
##    name                distance
##    <chr>               <fct>   
##  1 Harm Veling         2       
##  2 Jonas Dora          2       
##  3 Adrian Meier        2       
##  4 Leonard Reinecke    2       
##  5 Moniek Buijzen      2       
##  6 Donald R. Williams  2       
##  7 Andrew K Przybylski 1       
##  8 Nick Ballou         1       
##  9 Tamás Andrei Földes 2       
## 10 Elina Renko         2       
## # ℹ 557 more rows
## #
## # Edge Data: 2,624 × 3
##    from    to id      
##   <int> <int> <chr>   
## 1     1   140 2erwy_v1
## 2     2   140 2erwy_v1
## 3     3   140 2erwy_v1
## # ℹ 2,621 more rows

Conclusion

I wrote this post to take the psyarxivr R package for a test drive. I only looked at one variable from the preprints’ metadata table—although probably the richest one—and as such hope that others might find the data interesting and valuable. psyarxivr if you have any issues with it.

1 Curating a library

First, my idea is probably related to something that someone else has already written about. I collect those writings in my Zotero library. There are alternatives to Zotero but many of them, like Endnote, suck. Zotero is free, open source (extensible), and works well. What I do is I collect all the works I know about the topic into a Zotero collection (or group library). I then use online search engines to find more works. Most search engines suck, but Google Scholar sucks the least. When I’m on a website that has some relevant writing (books, articles), I click the little Zotero button in Firefox (requires the Zotero Connector add-on) and the piece is saved to my Zotero library.

If I intend to write a document, I have a related Zotero collection with probably around a hundred works, but this varies a lot depending on how much I know about the topic already. Since all the previous works were wrong, when I read and re-read them my aim is to figure out exactly how they were wrong and why, so I can be a little less wrong this time around: I will also be wrong but hopefully in a useful and transparent way (Scheel 2022).

2 Reading

I then semirandomly find time—on trains, buses, and aeroplanes—to read and understand what others have thought and done about the topic. It’s important to keep in mind that they are all wrong—that’s the point of research; nobody knows what they’re doing—but you still need to know what others have done in order to stand on their shoulders: What are some common methods and ideas, how have people approached this idea in the past?

If you ask people whether they know how to read, most will answer “yes”. Most of them are wrong. Developing the mental models and methods required for efficiently going through scholarly writings takes time and effort. Fortunately, guides exist (Savage 2024; Carey, Steiner, and Petri 2020) and I’d recommend looking into those.

3 Writing

After developing a sufficient understanding of what I’m doing and what others have thought about the topic, I start writing some words into a plain text (Healy 2019) file. This file may or may not build toward an early version of a manuscript.

My buddy Niklas once told me to write more like we speak and not in jargon-filled academese. I try to follow this advice as best as I can, but have a fondness for sentences that flow long like the Amazon river—through forests, mountainous valleys, flatlands, and human settlements—and thus become incomprehensible to readers who long ago have forgotten the point once they reach the sentence’s final punctuation mark. I guess we now have LLMs to help us write, but these words are mine, and I take pleasure in crafting them and the process of discovery through writing.

“Using ChatGPT to complete assignments is like bringing a forklift into the weight room; you will never improve your cognitive fitness that way.”

–Ted Chiang

If you need some examples to improve your writing, go read some Ed Yong.

One way in which I make writing more enjoyable for myself is to use the right tool. For me this tool is an integrated development environment (IDE), which is a fancy text editor that allows me to write documents that weave together human and computer words. I currently use Positron, because I like it and it allows writing human words (“prose”, sometimes I don’t know what those words mean so I use onelook.com) and computer words (“code”, sometimes I have no idea what I’m doing so I read the documentation) together into computationally reproducible documents (more on this in Section 4).

Many people use Microsoft Word and that’s fine too, but one should not be surprised that there are alternatives. And if you use Word, do yourself a favor and use the Styles panel.

3.1 Positron (or RStudio, VS Code, or any IDE)

Figure 3 shows what Positron looks like as I am writing this document. It’s confusing but you get used to it. On the left, I have a source file open called index.qmd (I’ll talk more about this below in Section 4). Below it, there’s a terminal into which I can write commands that do things on my computer. On the left is a preview of what the rendered document will look like, which updates every time I save the source document.

4 Computationally reproducible documents

Computationally reproducible documents are plain text (Healy 2019) files into which I write prose, such as this, and code, such as the colored text in Figure 3. (You can see the plain text source file of this very document here). This source file is then “rendered” into a typeset document using a computer program. The code languages in that figure are YAML at the top, which I use to specify metadata (the document’s title, for example) and markdown, which I use to format text (**bold** source turns into bold output).

Other languages I commonly use in these documents are R, which is a statistics language (`{r} sqrt(2)` in source turns into 1.4142136 in the output), and LaTeX, used to typeset maths from $y = \alpha + \beta x$ in the source file to in the output document.

❯ eza --tree my-project --level 2
my-project
├── _quarto.yml
├── bibliography.bib
├── data-raw
│   └── data.csv
├── index.qmd
└── README.md

5 Version control

While tools like MS Word’s “track changes” can be nice when you’re writing prose, they don’t work with computationally reproducible documents. So we end up with version control systems, such as Git.

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryBorderColor': '#666',
    'commitLabelFontSize': '12px',
    'commitLabelBackground': '#ffffff',
    'git0': '#4a90e2',
    'git1': '#7b68ee'
  }
}}%%

gitGraph
    commit id: "Initial project"
    commit id: "Add data collection script"
    commit id: "Add statistical analysis"
    commit id: "First draft complete" tag: "v0.1.0"
    branch modeling
    checkout modeling
    commit id: "Try alternative model"
    commit id: "Add sensitivity analysis"
    commit id: "Robustness checks"
    checkout main
    commit id: "Update documentation"
    commit id: "Fix data processing"
    merge modeling
    commit id: "Finalize results"
    commit id: "Analysis complete" tag: "v0.2.0"
    commit id: "Add publication plots"
    commit id: "Ready for submission" tag: "v1.0.0"

6 Collaboration

Version control systems are mistakenly interpreted, as I suggest above, to be about version control. They do that too, but their main benefits become apparent when collaborating. Just like the “track changes” feature in MS Word, they allow you to see what’s changed in your document(s) and review & undo edits, but more importantly they help others with that. It doesn’t matter if that “other” is another person or the selfsame you later in time.

If I’m just writing prose, I’ll probably collaborate on a Google Docs or MS Word (on OneDrive or some similar abomination), because these cloud platforms minimize friction in getting changes from person A to person B. As soon as we’re working together on a computationally reproducible projet, I encourage people to collaborate on the Git repo whose remote is on GitHub. Figure 5 (source) shows a schematic of a collaborative workflow.

7 Depositing materials

Many folks in my field drag and drop their files from their local filesystem to the Open Science Framework. I prefer a simpler (to me) workflow, where GitHub releases are automatically archived (with citation metadata, a DOI, etc) to Zenodo. Here is a Zenodo archive of this GitHub repository that contains a computationally reproducible manuscript project for this manuscript.

8 Showing my work to others

In academia, we typically call the process of showing our work to others “publishing”. Although the goal is pretty clear—make publicly funded common goods, or science, freely available for anyone to read—publishing is a byzantine process (Figure 6).

I am increasingly against how the current “publishing” system works, and for me a near-final milestone for an academic document is the PDF file I post to PsyArXiv Preprints, a popular document-sharing platform for the psychological sciences. Optimally I would receive peer feedback on those documents to incorporate into the write-up, but sadly this is not how things work yet. So we go through some of the worst web portals known to humanity to “submit”, “revise and resubmit”, and “publish” the words and pictures in other kinds of websites called “journals”. It’s all a bit silly.

Surprisingly, my thoughts on this topic are not universally shared. For some context, take a look here and here (Harnad 1990, 2001).

9 Overview of tools

10 End

The central ideas in the ongoing can be boiled down to this:

1. Decide and define early on what you’re doing / the anticipated “product”. Usually this is a scholarly manuscript available for others to read and criticize.
2. Put all the things related to that thing into as few places as possible. Then track that with Git, and collaborate on GitHub.
3. Ensure that anyone can get the source of the project, click a button, and create the manuscript you share on PsyArXiv.

Maybe that goes between collecting the underpants and profit?

What?

We like writing all our scientific outputs with Quarto using the Positron IDE. We also manage our references/library with Zotero, and want to insert references directly from Zotero to the Quarto documents. What does that mean?

It means that you fire up Positron, write a document like manuscript.qmd:

---
title: Science!
author: Matti Vuorre
date: 2025-06-06
bibliography: bibliography.bib
---

Probability theory is cool [@jaynesProbabilityTheoryLogic2003].

and then run quarto render manuscript.qmd in your terminal. By default this creates a HTML document (but can be a PDF, Word document, or whatever) that will look like this:

Notice how the Jaynes reference is nicely dealt with both in-text and in the references section. While Positron allows easily adding citations from Zotero when editing documents in visual mode, this was less easy in source mode (which I much prefer).

How?

The vscode-zotero extension makes it easy to add in-text references to Quarto documents, and can be installed through Positron’s “Extensions” panel:

Critically, the extension does not add the entry in the document’s associated .bib file (bibliography: bibliography.bib in the document’s YAML front matter.)

How? An improved answer

So what I did was fork the extension, add this functionality, and submit a pull request. So once/if the pull request is merged you can install it as above. For now, to install the extension, go to https://github.com/mvuorre/vscode-zotero/releases/tag/v0.2.0 and download the .vsix file.

Then, in Positron, install the extension by clicking “Install from VSIX…” and select the downloaded .vsix file.

Restart Positron, ensure Zotero is running and that you have installed the Better BibTex Zotero plugin. Then, in Positron, open up your source document and ensure its frontmatter contains a reference to a bibliography file. Then place your cursor where you want the citation to appear in your document, and launch the extension’s citation picker function. On a Mac the hotkey for launching the function is Shift+Option+Z and Windows probably has one too. This brings up the citation picker UI dialog:

Typing anything in the dialog will search for items with that string, and advanced search queries like author:jaynes and author:jaynes theory work as well. Once you’ve chosen the desired citation, hit Return and the in-text citation is added to the document, and its biblatex entry is written to the document’s associated .bib file.

Done and done.

bmlm

Back in 2016 I wrote an R package for bayesian estimation of a multivariate multilevel model for assessing a three-variable causal mediation model (Vuorre [2016] 2024). In the abstract to the article that discussed the methodology, Niall Bolger and I wrote

“Statistical mediation allows researchers to investigate potential causal effects of experimental manipulations through intervening variables. It is a powerful tool for assessing the presence and strength of postulated causal mechanisms. Although mediation is used in certain areas of psychology, it is rarely applied in cognitive psychology and neuroscience. One reason for the scarcity of applications is that these areas of psychology commonly employ within-subjects designs, and mediation models for within-subjects data is considerably more complicated than for between-subjects data. Here, we draw attention to the importance and ubiquity of mediational hypotheses in within-subjects designs, and we present a general and flexible software package for conducting Bayesian within-subjects mediation analyses in the R programming environment. We use experimental data from cognitive psychology to illustrate the benefits of within-subject mediation for theory testing and comparison.” (Vuorre and Bolger 2017)

I wrote the R package as an interface to a Stan (Team 2024) model because brms (Bürkner 2017a)—still in its early days—did not implement the kind of multivariate structure required by the model. Shortly afterwards, probably within a few months actually, Paul updated brms to fit the required model structure and bmlm as a standalone package lost much of its value. So whenever people email me about bmlm, I keep suggesting them to estimate their models with brms instead because it can do this, and so much more. (I’ve had a tutorial for this up at https://vuorre.com/brms-workshop/posts/mediation/ but it’s difficult to find.)

So in this post I’ll briefly show how to fit bmlm’s multilevel mediation model with brms, along with the required post-processing for computing the indirect effects, figures, etc.

Mediation—a word of caution

Mediation models are used to make causal claims from observational data. This is a complex and difficult endeavor, and all uses of mediation must appropriately wrestle with the implications and assumptions behind the models and claims that are being made. See Green, Ha, and Bullock (2010) and Rohrer et al. (2022).

Analysis

In one of bmlm’s vignettes we analyse an example data set from Intensive Longitudinal Methods: An Introduction to Diary and Experience Sampling Research (Bolger and Laurenceau 2013). The data, shown in Table 1 and included in the bmlm package, indicate several (hypothetical) participants’ (id) work stressors (fwkstrs), work dissatisfaction (fwkdis), and relationship dissatisfaction (freldis) over several days of study.

dat <- tibble(BLch9)[, c(1, 3:5)]
tt(head(dat))

To see how the full analysis is conducted with bmlm, please see the vignette (https://vuorre.com/bmlm/articles/bmlm-blch9/bmlm-blch9.html). Here, we implement the analysis without the use of bmlm’s functions.

dat <- dat |>
  mutate(
    x = fwkstrs - mean(fwkstrs, na.rm = TRUE),
    m = fwkdis - mean(fwkdis, na.rm = TRUE),
    y = freldis - mean(freldis, na.rm = TRUE),
    .by = id,
    .keep = "unused"
  )
tt(head(dat))

path_m <- bf(
  m ~ x + (x | p | id)
)
path_y <- bf(
  y ~ x + m + (x + m | p | id)
)
fit <- brm(
  path_m + path_y + set_rescor(FALSE),
  data = dat,
  file = "cache/fit"
)

 Family: MV(gaussian, gaussian) 
  Links: mu = identity; sigma = identity
         mu = identity; sigma = identity 
Formula: m ~ x + (x | p | id) 
         y ~ x + m + (x + m | p | id) 
   Data: dat (Number of observations: 2100) 

Multilevel Hyperparameters:
~id (Number of levels: 100) 
                             Estimate Est.Error l-95% CI u-95% CI
sd(m_Intercept)                  0.02      0.01     0.00     0.05
sd(m_x)                          0.26      0.04     0.19     0.34
sd(y_Intercept)                  0.02      0.01     0.00     0.05
sd(y_x)                          0.08      0.03     0.01     0.14
sd(y_m)                          0.22      0.03     0.17     0.28
cor(m_Intercept,m_x)             0.02      0.39    -0.72     0.77
cor(m_Intercept,y_Intercept)     0.02      0.41    -0.75     0.77
cor(m_x,y_Intercept)            -0.01      0.41    -0.77     0.74
cor(m_Intercept,y_x)            -0.01      0.41    -0.77     0.75
cor(m_x,y_x)                     0.24      0.29    -0.39     0.77
cor(y_Intercept,y_x)            -0.01      0.42    -0.77     0.76
cor(m_Intercept,y_m)            -0.00      0.40    -0.75     0.72
cor(m_x,y_m)                     0.48      0.15     0.17     0.75
cor(y_Intercept,y_m)            -0.01      0.42    -0.79     0.76
cor(y_x,y_m)                     0.56      0.26    -0.11     0.91

Regression Coefficients:
            Estimate Est.Error l-95% CI u-95% CI
m_Intercept    -0.00      0.02    -0.05     0.05
y_Intercept    -0.00      0.02    -0.04     0.04
m_x             0.19      0.04     0.12     0.26
y_x             0.10      0.02     0.06     0.15
y_m             0.15      0.03     0.09     0.21

Further Distributional Parameters:
        Estimate Est.Error l-95% CI u-95% CI
sigma_m     1.09      0.02     1.06     1.13
sigma_y     0.93      0.01     0.90     0.96

draws <- as_draws_df(
  fit,
  variable = c(
    "b_m_x",
    "b_y_x",
    "b_y_m",
    "sd_id__m_x",
    "sd_id__y_m",
    "cor_id__m_x__y_m"
  )
) |>
  mutate(
    a = b_m_x,
    b = b_y_m,
    cp = b_y_x,
    covab = cor_id__m_x__y_m * sd_id__m_x * sd_id__y_m,
    me = a * b + covab,
    c = me + cp,
    pme = me / c,
    .keep = "unused"
  )

draws |>
  summarise_draws(
    mean,
    sd,
    ~ quantile2(.x, probs = c(.025, .975))
  ) |>
  tt()

Graphics

brms has a number of excellent built-in visualization facilities, such as drawing conditional effects with ggplot2 (Wickham 2016) (Figure 1).

Code

ce1 <- conditional_effects(
  fit,
  effects = "x",
  resp = "m",
  robust = FALSE
)
ce1 <- plot(ce1, plot = FALSE)[[1]]
ce2 <- conditional_effects(
  fit,
  effects = "m",
  resp = "y",
  robust = FALSE
)
ce2 <- plot(ce2, plot = FALSE)[[1]]
ce1 | ce2

And the resulting objects and posterior samples are easily visualized with e.g. functions from the ggdist package (Kay 2024a) (Figure 2).

Code

draws |>
  select(!starts_with(".")) |>
  pivot_longer(everything()) |>
  ggplot(aes(x = value, y = name)) +
  stat_slabinterval(normalize = "xy")

Conclusion

If you are going to fit the model described in (Vuorre and Bolger 2017), I recommend using the brms R package (Bürkner 2017a) because of its flexibility, how it clearly connects to R regression syntax, and its estimation efficiency.

Feedback & comments

I’d appreciate any feedback or comments you might have. Feel free to le me know what you think either using the comments field (below) or on Bluesky:

Background and principles

I signed/joined the PRO Initiative way back when I was a PhD student, but just to remind myself:

Openness and transparency are core values of science. As a manifestation of those values, a minimum requirement for publication of any scientific results must be the public submission of materials used in generating those results. As reviewers, it is our responsibility to ensure that publications meet certain minimum quality standards.

We therefore agree that as reviewers, starting 1 January 2017, we will not offer comprehensive review for, nor recommend the publication of, any manuscript that does not meet the following minimum requirements. Once such a manuscript has been certified by the authors to meet these minimum requirements, we will proceed with a more comprehensive review of the manuscript.

– PRO Initiative; Morey et al. (2016)

More recently I’ve been entertaining the idea of joining/signing something similar but regarding open assessment—the practice of i. evaluating openly available works and ii. making the evaluations themselves public. For example, I find Nikolaus Kriegeskorte’s Open Evaluation proposal very agreeable:

“The current system of scientific publishing provides only journal prestige as an indication of the quality of new papers and relies on a non-transparent and noisy pre-publication peer-review process, which delays publication by many months on average. Here I propose an OE [Open Evaluation] system, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating. […] OA [Open Access] and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.

– Kriegeskorte (2012)

The scientific enterprise relies on access to accurate information. One of the ways in which scientists have tried to ensure that information is accurate is the process of peer-review, where experts look at your work and evaluate whether it’s up to snuff. While the primary fruits of the peer-review process (the manuscripts) are increasingly openly available, the reviews (and editorial notes) are typically not. This creates a situation whereby consumers of the scientific literature must trust the peer-review process without the being able to evaluate and learn from the evaluations themselves.

Many have suggested that this closed approach to evaluation might be suboptimal (Kriegeskorte 2012; Holcombe 2025). Moreover, the peer reviews themselves can contain information that could be widely applicable outside the specific review context. Therefore, I am taking the following steps to increase my engagement with open assessment of scientific research:

Practices

Template responses

Here’s some boilerplate text that I use in my responses to review invitations.

Conclusion

There is no conclusion. How we conduct, communicate, and evaluate scientific research is and always will be a work in progress. This document simply outlines my modest attempts at keeping up with (what I perceive to be) the latest gold-standard practices in transparent communication and evaluation.

Discovery

There are a handful of very popular preprint services, such as arXiv, the OG preprint server for hard sciences, and bioRxiv for the biological sciences. OSF Preprints is a “A scholarly commons to connect the entire research cycle”, and home to some two dozen field-specific preprint services such as MetaArXiv (metascience) and PsyArXiv (psychology). While all these services offer support for categorizing / tagging submissions, it is still often the case that researchers find it difficult to follow the latest (and greatest?) in their chosen area of interest.

For my areas of interest in the psychological sciences, I try to keep an eye on the Social and Behavioral Sciences category on OSF Preprints, and a small handful of more focused categories on the PsyArXiv discovery feed. These allow me to narrow down the feeds by e.g. author, subject, date, etc, and order them by date. So effectively I can have, say, a feed for the latest preprints in Cognitive Psychology that have pre-registered analysis plans and refresh it every morning in my browser. This is very cool.

Psyarxiv Zero

I wanted to build on this service to allow users to subscribe (e.g. via email or website account) to different custom feeds, and to present them in a fast text-based UI. To date I haven’t had time to make much progress on the first goal, but have finished a prototype for the latter (fast UI) at https://psyarxiv.vuorre.com. This website, Psyarxiv Zero¹, at the moment presents a simple feed of recently (users can specify a time-frame) posted or edited preprints from PsyArXiv (Figure 1).

Clicking on any of the titles on the homepage sends the user to a preprint’s page (Figure 2). I tried to make this page display the preprints main summaries (links, authors, keywords, and abstract) in an information-dense manner.

A lot of work remains to make this alternative UI for PsyArXiv (in the future, OSF Preprints more broadly) more useable and feature-rich. But at the moment I am happy with its performance—which is only limited by the speed of responses from the OSF API—and UI. Take it for a spin and give me your worst feedback / bug reports / feature requests at https://github.com/mvuorre/psyarxiv-ui.

Typesetting

I have a hunch that the typesetting of an article plays some non-ignorable role in readers’ credibility judgments of manuscripts made under time pressure and without other quality indicators. Moreover, reading a well-typeset document is a more pleasant experience than reading a poorly-typeset one. These (non-?)issues related to typesetting are prominent for readers of preprints, because preprints do not have any formatting standards or requirements. That’s probably a good thing, but at least I find reading typeset manuscripts a less onerous task.

I write most of my manuscripts in a computationally reproducible manner—in source documents that combine analysis code, its outputs, and prose—using Quarto. Quarto already has many extensions for producing (PDF) documents typeset to several journals’ requirements. In my field, the most relevant one is apaquarto that typesets documents to the American Psychological Association guidelines.

However I think many of these journal- or society-specific typesetting systems have a drawback: They require users to commit to a specific journal’s formatting requirements before knowing whether the paper will even end up in that journal; after rejection users will have to change to another format. Using Quarto makes this process easier by promising standard metadata fields for manuscripts, such as the ways in which author information should be formatted. Nevertheless, many format extensions require idiosyncratic settings / metadata, making switching between journal formats not quite the click of a button workflow as promised by Quarto.

Therefore, to add to the existing high-quality, but journal (or society-) specific Quarto formats, I wrote a little Quarto Typst extension called quarto-prepint (PDF). My aim with it is to enable fast and not-too-opinionated typesetting for computationally reproducible preprints written with Quarto. I paste from quarto-preprint’s manual below:

Conclusion

I encourage scholars to think more proactively about the roles that preprints play in the modern scholarly communication landscape (Sever 2023; Syed 2024; Moshontz et al. 2021; Ahmed et al. 2023). To this end (and to learn web and Quarto extension development 😄), I put together two (early-stage) resources for preprint authors and readers. If you try them out, feel free to let me know what’s wrong with them!

Statistics

Here are some recommended writings on statistics and data analysis for the social and behavioral sciences. This blog entry is probably most useful for students, but I also list some more advanced books that might be of interest to others.

tl;dr:

• Regression and other stories (Gelman, Hill, and Vehtari 2020)
• Introduction to Modern Statistics (Çetinkaya-Rundel and Hardin 2021)
• Statistical rethinking: a Bayesian course with examples in R and Stan (McElreath 2020)
• R for Data Science (Wickham, Çetinkaya-Rundel, and Grolemund 2016)

Introduction

Within-cluster centering, or person-mean centering (psychologists’ clusters are typically persons), is an easy data processing step that allows separating within-person from between-person associations. For example, consider the example data of 100 people’s ratings of urge to smoke and depression, collected over 50 days with one response per day (McNeish and Hamaker 2020) ¹, shown in Table 1 and Figure 1.

dat <- read_csv(
  "materials/Data/Two-Level Data.csv",
  col_names = c("urge", "dep", "js", "hs", "person", "time")
) |>
  select(-hs, -js) |>
  relocate(person, time, 1) |>
  mutate(
    person = factor(person),
    time = as.integer(time)
  ) |>
  mutate(
    u_lag = lag(urge),
    dep_lag = lag(dep),
    .by = person
  )

Table 1 shows the original data values. Those could then be transformed to person-means and person-mean centered deviations with simple data processing. However, the person-mean is an unknown quantity, and centering on the observed value rather than an estimate of the true “latent” quantity can be problematic. Specifically, observed mean centering leads to Nickell’s (negative bias in autoregressive effects) and Lüdtke’s (bias in other time-varying effects) biases (McNeish and Hamaker 2020, 617–18).

Code

set.seed(999)
pids <- factor(sample(1:100, 4))

dat |>
  filter(person %in% pids) |>
  pivot_longer(c(urge, dep)) |>
  rename(Time = time) |>
  mutate(name = factor(name, labels = c("Depression", "Urge"))) |>
  ggplot(aes(Time, value, col = name)) +
  geom_line(linewidth = .5) +
  facet_wrap("person", nrow = 1, labeller = label_both)

So, what to do? McNeish and Hamaker (2020) and others discuss latent mean centering, which accounts for uncertainty in the person-means appropriately, and thus debiases the estimated coefficients. Latent mean centering is done inside the model, and means treating the means as estimated parameters. However, I have only been able to find examples that do this latent mean centering in MPlus (McNeish and Hamaker 2020) and Stan (https://experienced-sampler.netlify.app/post/stan-hierarchical-ar/). My goal here is to show how latent mean centering can be done in the Stan front-end R package brms.

Univariate latent means model

We begin with a univariate model of the urge to smoke. This model examines the degree of autocorrelation in the urge to smoke and how it varies between people. For individual i in 1…I=100 and time point t in 1…T=50, we model urge (U) as normally distributed. We model the mean on person-specific intercepts and slopes of that person’s within-person centered urge at a previous time point (). I model person-specific deviations as multivariate normal but do not model correlations between the intercepts and slopes for consistency with (McNeish and Hamaker 2020).

Let us pay some attention to the issue of within-person centering in Equation 1. Instead of decomposing urge to smoke into its within- and between-person components before fitting the model, we use “latent mean centering”. What this means is that we estimate the person means () along with other model parameters, and subtract those means from the observed values (line 2 in above). I refer to the latent person-mean centered lagged urge to smoke as .

I use the R package brms to estimate this model. The following code chunk shows how to specify this model inside brms’ bf() (“brmsformula”) function. In the first line, we specify a regression equation for urge. Everything on the right-hand side of this formula (to the right of the tilde) is treated as a regression coefficient to be estimated from data unless it is the exact name of a variable in the data. Thus we will be estimating an alpha (intercept) and a phi (the autoregressive coefficient).

model <- bf(
  urge ~ alpha + phi * (u_lag - alpha),
  alpha + phi ~ 1 + (1 | person),
  nl = TRUE
)

One unusual part in this syntax is (u_lag - alpha). It just subtracts alpha from each lagged urge value in creating the predictor for phi. That is “latent mean centering”. This first line can be considered the “level 1” equation or rather the nonlinear part of the model.

The second line then specifies the “level 2” equation, or the linear equations to predict the parameters in the above (potentially) nonlinear level 1 model. Both regression parameters are modelled on a population level average (the gamma in Equation 1) and person-specific deviations from it.

The fourth line specifying nl = TRUE is critical, because it allows us to specifically name parameters inside bf(), and thereby to e.g. construct the latent mean centered variable on the first row. We could also indicate the distribution that we assume for the data. But in this work we model everything as gaussian, which is the software default and thus doesn’t need to be separately indicated. We then sample from the model. Everything from here on is standard operating procedure.

fit <- brm(
  model,
  data = dat,
  file = "cache/brm-example-univariate"
)

The object fit now contains the estimated model (the data, posterior samples, and lots of brms-specific information). We can call summary(fit) to see a default summary of the model.

summary(fit)

 Family: gaussian 
  Links: mu = identity 
Formula: urge ~ alpha + phi * (u_lag - alpha) 
         alpha ~ 1 + (1 | person)
         phi ~ 1 + (1 | person)
   Data: dat (Number of observations: 4900) 
  Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup draws = 4000

Multilevel Hyperparameters:
~person (Number of levels: 100) 
                    Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(alpha_Intercept)     0.78      0.07     0.67     0.92 1.00      906     1707
sd(phi_Intercept)       0.15      0.02     0.11     0.19 1.00     2354     2767

Regression Coefficients:
                Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
alpha_Intercept    -0.01      0.08    -0.18     0.15 1.00      714     1207
phi_Intercept       0.20      0.02     0.16     0.25 1.00     2424     2654

Further Distributional Parameters:
      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma     1.57      0.02     1.54     1.60 1.00     6653     2873

Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

The first few rows above print information about the model (the formulas, data, and number of posterior samples). Then, “Multilevel Hyperparameters” are standard deviations (and correlations, if estimated) of the parameters that we allowed to vary across individuals (as indicated by ~person). For each of those parameters, one row indicates its posterior summary statistics; “Estimate” is the posterior mean, “Est.Error” is the posterior standard deviation, “l-” and “u-95% CI” are the lower and upper bounds of the 95% credibility interval (so the 2.5 and 97.5 percentiles of the posterior samples). Then, Rhat is the convergence metric which should be smaller than 1.05 (optimally 1.00) to indicate that the estimation algorithm has converged. “Bulk_” and “Tail_ESS” indicate the effective sample sizes of the posterior draws, and should be pretty large.

The “Regression Coefficients” indicate the same information but for the means of the person-specific parameters’ distributions; or the “fixed effects”. For the average person, there is a positive autocorrelation in these data. Finally, the “Further Distributional Parameters” indicate parameters that are specific to the outcome distribution. We used the default gaussian distribution, and thus get an estimated residual standard deviation.

Going forward we will create a small function to print out model summaries. It will take samples of the population level, group-level, and family-specific parameters, and return their 50th (median), 2.5th, and 97.5th quantiles.

sm <- function(x) {
  x |>
    as_draws_df(variable = c("b_", "sd_", "sigma"), regex = TRUE) |>
    summarise_draws(
      ~ quantile2(.x, c(.5, .025, .975))
    ) |>
    mutate(variable = str_remove_all(variable, "_Intercept"))
}

We show the results in Table 2.

fit |>
  sm() |>
  kable(digits = 2)

Multilevel AR(1) Model

We then replicate the two-level AR(1) model in McNeish and Hamaker (2020) (equations 4a-c) that predicts urge from a time-lagged urge and depression. The model is

We then see from Equation 2 that we need to refer to different outcomes’ parameters across model formulas. That is, when predicting the urge to smoke, we need a way to refer to the (latent) mean of depression so that we can appropriately center the depression predictor. Currently brms does not support sharing parameters across formulas for different outcomes, but we can overcome this limitation with a small data wrangling trick

That is, we “stack” our data into the long format with respect to the two different outcomes, urge to smoke and depression. Then, on each row we have all variables from that measurement occasion, in addition to new ones that indicate the value of the outcome, and which outcome it refers to (Table 3).

dat <- dat |>
  pivot_longer(c(urge, dep), names_to = "outcome", values_to = "y") |>
  mutate(
    i_urge = if_else(outcome == "urge", 1, 0),
    i_dep = if_else(outcome == "dep", 1, 0)
  ) |>
  # Include predictors from each row
  left_join(dat)

dat |>
  head() |>
  kable(digits = 2)

Given these data, we then reparameterize Equation 2 to also model depression in an otherwise identical model (Equation 3).

That is, I model y that is either urge or dep as indicated by i_urge and i_dep respectively. So, below alpha1, phi, and beta to apply to urge, but alpha2 to dep.

bform <- bf(
  y ~
    i_urge *
      (alpha1 + phi * (u_lag - alpha1) + beta * (dep - alpha2)) +
      i_dep * alpha2,
  nlf(sigma ~ i_urge * sigma1 + i_dep * sigma2),
  alpha1 + phi + beta + alpha2 ~ 1 + (1 | person),
  sigma1 + sigma2 ~ 1,
  nl = TRUE
)

Notice that essentially there are two models of y depending on the values of i_urge and i_dep. Critically, this also needs to extend to different models of the residual standard deviations. That is accomplished inside nlf(), where I model sigma on the two indicators. By default, sigmas are modelled through the log-link function, and notice that I only include a global intercept for each sigma1 and sigma2; that is they are not further modelled on covariates. This is not pretty, but as we will see it works.

I then sample from the model.

fit <- brm(
  bform,
  data = dat,
  control = list(adapt_delta = 0.95),
  file = "cache/brm-example-4"
)

And then compare the model summary to McNeish and Hamaker (2020). We can see the estimates match to within differences in priors and MCSE (Table 4). Note in the code below I transform standard deviations by first exponentiating draws of residual standard deviations, and then square to put them on the variance scale as in McNeish and Hamaker (2020).

as_draws_df(fit, variable = c("b_", "sd_"), regex = TRUE) |>
  mutate(
    across(starts_with("sd_"), ~ .^2),
    across(starts_with("b_sigma"), ~ exp(.)^2)
  ) |>
  summarise_draws(
    brms = ~ quantile2(., probs = c(.5, .025, .975)) |>
      number(.01) |>
      str_glue_data("{q50} [{q2.5}, {q97.5}]")
  ) |>
  mutate(
    variable = str_replace(variable, "sd_person__", "var_") |>
      str_remove_all("_Intercept"),
    `M&H (2020)` = c(
      "-0.01 [-0.18, 0.16]",
      " 0.21 [0.17, 0.24]",
      " 0.80 [0.61, 0.95]",
      " 0.01 [-0.02, 0.04]",
      " 1.14 [1.09, 1.19]",
      "",
      " 0.60 [0.44, 0.83]",
      " 0.02 [0.01, 0.03]",
      " 0.79 [0.61, 0.95]",
      " 0.01 [0.00, 0.01]"
    )
  ) |>
  kable(digits = 2)

Conclusion

Because it is easy to specify latent means in brms, I think I will be using them much more often from now on, especially if my sample size per person is small. I don’t think this will make much of a difference after that sample size is greater than, say, the magic number 30.

Let me know if you have any comments!

MacOS setup

This aims to be succinct and quick, using homebrew to install as much as possible from the command line.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# GUI apps
brew install --cask \
  firefox iterm2 microsoft-office \
  zotero obsidian todoist sublime-text \
  mactex rectangle alfred slack zoom \
  visual-studio-code docker monitorcontrol

# Terminal and CLI things
brew install \
  tailscale starship bat \
  btop lsd dua syncthing

brew tap homebrew/cask-fonts
brew install svn
brew install --cask \
  font-fantasque-sans-mono font-fantasque-sans-mono-nerd-font \
  font-noto-sans font-noto-serif font-noto-mono font-noto-mono-for-powerline \
  font-noto-emoji font-hasklug-nerd-font font-anonymice-nerd-font \
  font-meslo-lg-nerd-font font-fira-code font-fira-mono font-fira-sans \
  font-fira-sans-condensed font-pt-mono font-pt-sans font-pt-sans-narrow \
  font-pt-serif font-pt-sans-caption font-pt-serif-caption

install.packages("pak")
pak::pkg_install(
  c(
    "usethis", "tidyverse", "brms", 
    "kableExtra", "janitor", "here", 
    "scales", "gtsummary", "multidplyr", 
    "ggtext", "parameters", "tidybayes", 
    "ggstance", "ggdist", "patchwork", 
    "ggforce", "ggh4x", "lavaan", 
    "emmeans", "ggstance", "renv", 
    "furrr", "remotes", "kableExtra",
    "gt"
  )
)

Conclusion

After this I usually have a pretty useful computer. Because of homebrew its quite fast too.

Example multiverse analysis with specr

Let’s start with a simple toy example with two outcomes, two predictors, two covariates, and four subgroups, and no prior reason to choose between specifications. That is, we think that y1 and y2 are equally likely to represent our outcome construct of interest, x1 and x2 are equally likely to represent the predictor construct, and we can’t choose if or how to include the covariates c1 and c2 in the model. We might also consider the subgroups defined by group separately (and are not willing to do hierarchical models.) Let’s load the required libraries and show the example data (Table 1):

# Packages
library(kableExtra)
library(scales)
library(ggthemes)
library(tictoc)
library(tidyverse)

# Pretty plots
theme_set(
  theme_few(
    base_family = "Comic Sans MS",
    base_size = 12
  )
)

# Pretty tables
k2 <- function(x, full_width = FALSE) {
  x %>%
    kbl(digits = 2) %>%
    kable_classic_2(
      html_font = "Arial",
      lightable_options = "striped",
      full_width = full_width
    )
}

# Data generation
generate_data <- function(seed = NA, n = 1e5) {
  if (!is.na(seed)) {
    set.seed(seed)
  }
  dat <- tibble(
    x1 = rnorm(n),
    x2 = rnorm(n),
    y1 = rnorm(n) + x1 * .1,
    y2 = rnorm(n) + x1 * .2,
    c1 = rnorm(n) + x1 * .3,
    c2 = rnorm(n),
    group = sample(c("a", "b", "c", "d"), n, replace = TRUE)
  )
}
dat <- generate_data(9)

We can specify a fully crossed multiverse analysis over outcomes, predictors, and covariates, easily with specr. Also, to make the example a bit more interesting (slower!) for later examples, I’ll estimate the model using two functions (lm() and glm() which in this case give the same results, but the latter is much slower). I time the multiverse analysis using tictoc. Table 2 shows the first few rows of the results.

library(specr)
tic()
results_specr <- run_specs(
  df = dat,
  y = c("y1", "y2"),
  x = c("x1", "x2"),
  model = c("lm", "glm"),
  controls = c("c1", "c2"),
  subsets = list(group = unique(dat$group))
)
toc()

29.142 sec elapsed

results_specr %>%
  head() %>%
  .[, 1:10] %>%
  kbl(
    digits = 2
  ) %>%
  kable_classic_2(html_font = "Arial", full_width = FALSE)

Another great thing about this package is that you can easily draw specification curve figures (Figure 2)

Code

plot_specs(
  results_specr,
  choices = c("x", "y", "model", "controls", "subsets")
)

However, even with this modest data set and 160 specifications, this took a while.

I first decided to take a stab at parallelizing run_specs(). This turned out to be a bit of a dead end because I couldn’t make the parallelization fit in with how run_specs() works in the back-end.¹ So instead of shoehorning a parallel back-end to specr, I decided to implement the parallelization in a tidy pipeline. This pipeline, with no additional dependencies (apart from the tidyverse 😉), works pretty well. It of course does not provide specr’s one-liners, but I believe the flexibility of this approach pays back for it.

Tidymultiverse

specs <- expand_grid(
  x = c("x1", "x2"),
  y = c("y1", "y2"),
  covariate = c("x1", "x2"),
  model = c("lm", "glm")
)

expand_grid(
  formula = c("y1 ~ x1", "y1 ~ x2", "y1 ~ x1 + c1"), # And so on
  model = c("lm", "glm")
)

specs <- expand_grid(
  x = c("x1", "x2"),
  y = c("y1", "y2"),
  covariate = c("x1", "x2"),
  model = c("lm", "glm"),
  # Cross with all the unique values of `group` in the data
  distinct(dat, group)
)

#' Expand a vector of covariate names to all their combinations
#'
#' For example expand_covariate(c("age", "sex")) returns
#' c("1", "age", "sex", "age + sex")
#'
#' @param covariate vector of covariate(s) e.g. c("age", "sex")
#'
#' @return a character vector of all predictor combinations
expand_covariate <- function(covariate) {
  list(
    "1",
    do.call(
      "c",
      map(
        seq_along(covariate),
        ~ combn(covariate, .x, FUN = list)
      )
    ) %>%
      map(~ paste(.x, collapse = " + "))
  ) %>%
    unlist
}

specs <- expand_grid(
  x = c("x1", "x2"),
  y = c("y1", "y2"),
  covariate = expand_covariate(c("c1", "c2")),
  model = c("lm", "glm"),
  distinct(dat, group)
) %>%
  mutate(formula = str_glue("{y} ~ {x} + {covariate}"))

lm2 <- function(formula, data) {
  fit <- lm(formula = formula, data = data)
  out <- tidy(fit, conf.int = TRUE) # Tidy table of parameters
  out <- slice(out, 2) # Second row (slope parameter)
  bind_cols(out, n = nobs(fit))
}
lm2(y1 ~ x1, data = dat)

lm2 <- possibly(lm2, otherwise = tibble(n = 0))
# See what it return when it gets bad input
lm2(group ~ x1, data = dat)

glm2 <- function(formula, data) {
  fit <- glm(formula = formula, data = data)
  out <- tidy(fit, conf.int = TRUE)
  out <- slice(out, 2)
  bind_cols(out, n = nobs(fit))
}
glm2 <- possibly(glm2, otherwise = tibble(n = 0))

specs <- mutate(specs, model = paste0(model, "2"))

tic()
results_dplyr <- specs %>%
  mutate(
    out = pmap(
      list(model, formula, group),
      ~ do.call(
        ..1,
        list(
          formula = ..2,
          data = filter(dat, group == ..3)
        )
      )
    )
  )
toc()

13.51 sec elapsed

results_dplyr <- results_dplyr %>%
  unnest(out)

Parallelizing the tidymultiverse

library(multidplyr)

# Create a new cluster with eight nodes
cluster <- new_cluster(8)

# Load libraries in and send variables to nodes in the cluster
cluster_library(cluster, c("purrr", "broom", "tidyr", "dplyr"))
cluster_copy(cluster, c("dat", "lm2", "glm2"))

tic()
results_multidplyr <- specs %>%
  group_by(formula, group) %>%
  partition(cluster) %>%
  mutate(
    out = pmap(
      list(model, formula, group),
      ~ do.call(
        ..1,
        list(
          formula = ..2,
          data = filter(dat, group == ..3)
        )
      )
    )
  ) %>%
  collect() %>%
  ungroup() %>%
  unnest(out)
toc()

3.163 sec elapsed

library(furrr)
plan(multisession, workers = 8)

# Pass these global objects to `future_pmap()`
opts <- furrr_options(
  globals = list(dat = dat, lm2 = lm2, glm2 = glm2),
  packages = c("dplyr", "broom")
)

tic()

results_furrr <- specs %>%
  mutate(
    out = future_pmap(
      list(model, formula, group),
      ~ do.call(
        what = ..1,
        args = list(
          formula = ..2,
          data = filter(dat, group == ..3)
        )
      ),
      .options = opts
    )
  ) %>%
  unnest(out)
toc()

5.623 sec elapsed

A visualization

Finally, like any analysis, multiverse analyses need to be visualized for understanding and communicating. Here, we use some ggplot2 magic to create a standard specification curve analysis figure (Figure 3).

Code

library(patchwork)
results <- arrange(results_furrr, estimate) %>% mutate(spec = 1:n())
p_dash <- results %>%
  select(spec, p.value, x:group) %>%
  pivot_longer(-c(spec, p.value), values_transform = as.character) %>%
  ggplot(aes(spec, value, col = p.value < 0.05)) +
  scale_color_brewer(palette = "Set1") +
  scale_x_continuous(
    "Specification"
  ) +
  geom_point(size = 0.5) +
  facet_grid(rows = vars(name), scales = "free_y", space = "free_y") +
  theme(axis.title.y = element_blank())
p_curve <- results %>%
  ggplot(aes(spec, estimate, col = p.value < 0.05)) +
  scale_color_brewer(palette = "Set1") +
  geom_pointrange(aes(ymin = conf.low, ymax = conf.high), size = .2) +
  theme(axis.title.x = element_blank())
(p_curve / p_dash) &
  theme(legend.position = "none")

What else?

Using this method, we can pass whatever modelling functions (e.g. lmer(), brm()) and arguments to them (e.g. append the formula with (1 | participant) for lmer() hierarchical models) and parallelize the iterations quite easily. We can also imagine more complex data subsetting scenarios. For example, we could expand the specs table to include various conditions for filtering data (e.g. outliers). We could then pre-compute those (or do it in do.call()) to dynamically subset data differently in each row of specs.

I hope you found this helpful. If you’ve any feedback, comments are open below and I’d appreciate your thoughts!

Raincloud plots the hard way

But first, I’ll try to recreate this raincloud plot without the ggrain package. Most of the geoms and stats we need are in the ggdist package (Kay 2022). The end result (Figure 2) looks very similar to the ggrain version, above.

library(tidyverse)
library(ggdist)
iris %>%
  ggplot(aes(Species, Sepal.Length, fill = Species)) +
  geom_point(position = position_jitter(width = .033)) +
  geom_boxplot(position = position_nudge(x = -0.085), width = .05) +
  stat_halfeye(
    side = "left",
    normalize = "none",
    width = .3,
    position = position_nudge(x = -0.15),
    point_interval = NULL
  )

OK, so now we have a handle on how to create raincloud plots “manually”.

Removing summaries and densities

What I would like to do next is to make the summaries less prominent. I can use stat_halfeye(). Above, I used stat_halfeye(..., point_inteval = NULL) to remove them completely. Here, I will specify some quantiles to show with the width argument. I am not sure if Figure 3 is an improvement.

iris %>%
  ggplot(aes(Species, Sepal.Length, fill = Species)) +
  geom_point(position = position_jitter(width = .033)) +
  stat_halfeye(
    side = "left",
    normalize = "none",
    width = .3,
    position = position_nudge(x = -0.1),
    .width = c(.5, .99)
  )

Maybe all this information can be gleaned from the points alone. To do this, we can jitter the points according to a method specified in the vipor package (Sherrill-Mix and Clarke 2017).

library(ggbeeswarm)
set.seed(1)
iris %>%
  ggplot(aes(Species, Sepal.Length, fill = Species, col = Species)) +
  geom_point(
    position = position_quasirandom(width = .1)
  )

Figure 4 arranges the points using one of the offsetting algorithms in vipor, brought to ggplot via the ggbeeswarm package (Clarke and Sherrill-Mix 2017). By default, this is the “quasirandom” method, where “points are distributed within a kernel density estimate of the distribution with offset determined by quasirandom Van der Corput noise”. I can only guess that “the distribution” refers to a gaussian distribution.

It would be really nice if we could choose the x-axis side to which jitter the points. Then we could display two groups side by side. Unfortunately that is not possible.

A more complicated example

Let’s try a more complicated example similar to Rogier Kievit’s figure (Figure 5). I first simulate some data with two groups and four timepoints. There’s also some covariate that I’d like to display.

# Data generation
generate_data <- function(seed = NA, n = 200) {
  if (!is.na(seed)) {
    set.seed(seed)
  }
  dat <- tibble(
    id = 1:n,
    x = sample(0:1, n, replace = TRUE),
    c = rnorm(n),
    `1` = rnorm(n, x * .2 + c * .4, 1.1),
    `2` = rnorm(n, x * .2 + c * .4, 1.2),
    `3` = rnorm(n, x * .2 + c * .4, 1.3),
    `4` = rnorm(n, x * .2 + c * .4, 1.4)
  ) %>%
    mutate(x = factor(x, labels = c("Old", "Young"))) %>%
    pivot_longer(`1`:`4`) %>%
    mutate(name = as.integer(name))
}
dat <- generate_data(9)

I’ll try to show this plot with much fewer visual symbols, and hopefully retain most of the information.

library(ggnewscale)
dat %>%
  rename(Time = name, Value = value) %>%
  ggplot(aes(Time, Value)) +
  scale_color_viridis_c(
    "Covariate"
  ) +
  geom_point(
    aes(col = c, group = x),
    size = 1,
    alpha = .75,
    position = position_quasirandom(width = .05, dodge.width = .35)
  ) +
  new_scale_color() +
  scale_color_brewer(
    "Group",
    palette = "Set1"
  ) +
  stat_pointinterval(
    aes(color = x),
    interval_size_range = c(.3, .9),
    position = position_dodge(.075)
  )

Hmm. Figure 6 doesn’t quite work visually as I’d like it to. I think it would be really nice if the jittered points were jittered only on their respective sides.

I might come back to this later to see if I can improve on this design.

The takeaway, though, is that the ggrain package provides really nice figures out of the box. If we want to do more complex figures kind of like these, the ggdist and ggbeeswarm plots can create compelling alternatives.

What you need

These solutions work on Linux, MacOS, and even Windows operating systems. The slow and fast computers can have any combination of these.

You also need to use RStudio for the solutions discussed here. It turns out that doing this in VS Code can be even easier because of its superb remote session support. I’ll add the VS Code writeup later, once my transition from RStudio to VS Code is complete 😉.

The first thing you need to set up is an RStudio Server instance on the fast computer. If your fast computer is running Linux, this is trivial.

If your fast computer has either MacOS or Windows, you will need to set up the RStudio Server instance using Docker. This is really easy, and we begin here.

RStudio Server

We are first going to install RStudio Server on the fast computer. You cannot run RStudio Server on MacOS or Windows, but we can easily fire one up using Docker. First, using your fast computer, head over to the Docker website and download the Docker desktop app. Then start it and make sure it is running (you will have a menu bar or taskbar Docker button to indicate that it’s running).

Then start a terminal session, and use it to start a rocker/rstudio container:

docker run --rm -ti -v "$(pwd)"/work:/home/rstudio -e PASSWORD=yourpassword -p 8787:8787 rocker/rstudio

This creates a directory in your current working directory called work, and lets the Docker container access files therein (inside the container, the path is /home/rstudio where RStudio Server sessions typically start). This way whatever files you save inside Docker will remain in your disk, and you can use / edit those outside the container as well. (Thanks Kristoffer for pointing this critical point to me!)

Now your fast computer is running an RStudio Server session. You can verify this by opening a browser tab on the fast computer, and typing localhost:8787 in the address bar. You should see the RStudio Server login window pop up (Figure 1).

Then use rstudio as the Username, and yourpassword as the password. You’ll then have a fully functioning RStudio session in your browser (Figure 2).

Notice how it runs on Ubuntu, although my computer is an M1 Mac. Pretty cool, huh.

Ok, so how do we connect to this from other computers. We might now either want to connect from another computer on the same network, or on another network. Let’s start with the first.

Computers on the same local network

This is pretty easy! First, find your fast computer’s local IP address. There’s many ways to find this and you could for example query it in the terminal:

ipconfig getifaddr en0

Your local IP address will be something like 192.168.0.123. My fast computer currently runs on 192.168.0.155, and I’ll use it below.

Fire up a browser in your slow computer, and navigate to 192.168.0.155:8787. I’m using my phone as the slow computer here, and after logging in with the same credentials as above, I see Figure 3.

It really isn’t more difficult than that.

Computers on different networks

OK, so you still have RStudio Server running on your fast computer, but maybe it’s at work and you are now at home with your slow computer and a cold beer. How to connect? There’s many ways to do this, but here we will use Tailscale.

First, create a Tailscale account, and then install it on both computers. (OK so I guess you still need to be physically near both machines at this point 😄. [Unless you already have e.g. SSH access to the fast computer, in which case you can install Tailscale in the terminal.]) Make sure Tailscale is running on both and that they are signed in to the same Tailscale account. You can follow the official instructions. It really is quite easy and that’s why I use Tailscale and not some other SSH or VPN based solution.

Then, you can head to https://login.tailscale.com/admin/machines (on either computer). It will show you all the machines that you’ve connected to Tailscale (Figure 4), whether they are active or not.

Now you can connect between your computers wherever the machines might be, provided that they are connected to the internet and Tailscale. My fast computer’s Tailscale IP, redacted in Figure 4, is xxx.xxx.x.xx. So now I go home with my slow computer, and then use the browser to connect to xxx.xxx.x.xx:8787, and I see Figure 3 again.

I can then use RStudio (server) running on my fast computer on any of my other computers (as clients), by using the Tailscale IP address.

Conclusion

If it is possible for you to have a powerful computer always connected to the internet, you can make a persistent RStudio computing platform out of it with RStudio Server. You can then use Tailscale to connect to it very easily from anywhere in the world.

I hope that was as helpful to you as it has been for me 😄. If something didn’t work for you, comments are open below.

Download ntfy.sh

Go to your app store (iOS/Android) and download the ntfy app.

Subscribe to a topic

Open the app on your phone and subscribe to a topic. Just type in a name that’s both memorable and not likely to already be used by someone else. I use vuorre-r-notifications.

Send notifications

You can now include variations of system("curl -d 'Notification text' ntfy.sh/vuorre-r-notifications") in your R code. For example, to send a notification after a long running code

# Long running code here
Sys.sleep(.1) # Sleep for .1 second
# Send notification
system("curl -d 'Woke up after .1 second nap!' ntfy.sh/vuorre-r-notifications")

You’ll get this notification on your phone:

This is really useful when you have simulations (mcmc or otherwise 😉) that take a long time, and you’d like to act as soon as they are done. Have fun!

Models and contrasts

Here, we will discuss linear models, which regress an outcome variable on a weighted combination of predictors, while allowing the weights to vary across individuals (hierarchical linear regression). After fitting the model, you will have estimates of the weights (“beta weights”, or simply regression parameters) that typically consist of an intercept (estimated level of outcome variable when all predictors are zero) and slopes, which indicate how the outcome variable changes as function of one-unit changes of the predictors, when other predictors are at 0.

However, we are often interested in further questions (contrasts, “general linear hypothesis tests”). For example, your model output may report one group’s change over time, and the difference of that slope between groups, but you are particularly interested in the other group’s slope. To find that slope, you’d need to calculate an additional contrast from your model. This is also commonly called “probing interactions” or sometimes “post hoc testing”.

library(tidyverse)

tmpfile <- tempfile()
download.file(
  "http://www.intensivelongitudinal.com/ch4/ch4R.zip",
  destfile = tmpfile
)
dat <- read_csv(tmpfile) |>
  mutate(id = as.factor(id), treatment = as.factor(treatment))

library(brms)
fit <- brm(
  intimacy ~ time * treatment + (time | id),
  family = gaussian(),
  data = dat,
  file = "intimacymodel"
)

library(kableExtra)
posterior_summary(fit, pars = "b_") %>%
  as_tibble(rownames = "Parameter") %>%
  kable(digits = 2, caption = "Summary of the Intimacy model's parameters") %>%
  kable_styling(full_width = FALSE)

Warning: Argument 'pars' is deprecated. Please use 'variable' instead.

library(patchwork)
pa <- dat %>%
  filter(id %in% c(1:3, 25:28)) %>%
  ggplot(aes(time, intimacy, group = id, col = treatment)) +
  scale_y_continuous(
    "Intimacy",
    limits = c(0, 6),
    breaks = pretty_breaks()
  ) +
  labs(x = "Time") +
  geom_line(size = .25) +
  geom_point(shape = 21, fill = "white")

Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

pb <- conditional_effects(
  fit,
  effects = "time:treatment"
) %>%
  plot(plot = FALSE) %>%
  .[[1]] +
  scale_y_continuous(
    "Intimacy",
    limits = c(0, 6),
    breaks = pretty_breaks()
  ) +
  labs(x = "Time") +
  theme(axis.title.y = element_blank(), legend.position = "none")
(pa | pb) &
  theme(
    legend.position = "bottom",
    aspect.ratio = 1
  )

Warning: Removed 1 row containing missing values or values outside the scale range
(`geom_point()`).

hypothesis()

hypothesis() truly is an underappreciated method of the brms package. It can be very useful in probing complex models. It allows us to calculate, visualize, and summarize, with full uncertainty estimates, any transformation of the model’s parameters. These transformations are often called “contrasts” or “general linear hypothesis tests”. But really, they are just transformations of the joint posterior distribution of the model’s parameters.

To answer Q2, then, we encode our question into a combination of the models parameters:

q2 <- c(q2 = "time + time:treatment1 = 0")

The slope of group 1 is calculated from the model’s parameters by adding the slope of group 0 (time) and the interaction term time:treatment1. = 0 indicates that we are interested in contrasting the resulting estimate the zero (“testing against zero” or even “testing the null hypothesis”). Then, we pass this named string to hypothesis(), and observe the results.

q2_answer <- hypothesis(fit, q2)
q2_answer

Hypothesis Tests for class b:
  Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1         q2     0.11      0.02     0.06     0.16         NA        NA    *
---
'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
'*': For one-sided hypotheses, the posterior probability exceeds 95%;
for two-sided hypotheses, the value tested against lies outside the 95%-CI.
Posterior probabilities of point hypotheses assume equal prior probabilities.

The output indicates that the estimated answer to Question 2 is 0.11 with a standard error of 0.02. I will return to Evid.Ratio and Post.Prob shortly.

The results can also be visualized.

plot(q2_answer)

That figure shows the (samples from the) posterior distribution of the answer to Question 2.

More contrasts

With hypothesis() you can answer many additional questions about your model, beyond the parameter estimates. To illustrate, say we are interested in the groups’ difference in intimacy at the end of the study (day 15; Question 4). (The difference at time 0 is reported by the group parameter.)

q4 <- c(q4 = "treatment1 + time:treatment1 * 15 = 0")
hypothesis(fit, q4)

Hypothesis Tests for class b:
  Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1         q4     0.87       0.4      0.1     1.66         NA        NA    *
---
'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
'*': For one-sided hypotheses, the posterior probability exceeds 95%;
for two-sided hypotheses, the value tested against lies outside the 95%-CI.
Posterior probabilities of point hypotheses assume equal prior probabilities.

q5 <- c(q5 = "time > 0")
q5_answer <- hypothesis(fit, q5)
q5_answer

Hypothesis Tests for class b:
  Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1         q5     0.05      0.02     0.01     0.09      57.82      0.98    *
---
'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
'*': For one-sided hypotheses, the posterior probability exceeds 95%;
for two-sided hypotheses, the value tested against lies outside the 95%-CI.
Posterior probabilities of point hypotheses assume equal prior probabilities.

plot(q5_answer)

hypothesis(fit, c(q2, q4, q5))

Hypothesis Tests for class b:
  Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1         q2     0.11      0.02     0.06     0.16         NA        NA    *
2         q4     0.87      0.40     0.10     1.66         NA        NA    *
3         q5     0.05      0.02     0.01     0.09      57.82      0.98    *
---
'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
'*': For one-sided hypotheses, the posterior probability exceeds 95%;
for two-sided hypotheses, the value tested against lies outside the 95%-CI.
Posterior probabilities of point hypotheses assume equal prior probabilities.

x <- hypothesis(fit, "Intercept = 0", group = "id", scope = "coef")

# Results of hypothesis() in a data.frame
x$hypothesis %>%
  # Obtain group indicators from original data
  left_join(distinct(dat, Group = id, treatment)) %>%
  # Rename Group to id and reverse order for figure
  mutate(id = factor(Group, levels = rev(1:50))) %>%
  # Draw a forest plot with ggplot2
  ggplot(aes(Estimate, id, col = treatment)) +
  geom_errorbarh(aes(xmin = CI.Lower, xmax = CI.Upper)) +
  geom_point()

Warning: `geom_errobarh()` was deprecated in ggplot2 4.0.0.
ℹ Please use the `orientation` argument of `geom_errorbar()` instead.

Conclusion

When you find that you have a brms model whose parameters don’t quite answer your questions, hypothesis() will probably give you the answer. For more advanced post-processing of your models, I recommend taking a look at the tidybayes package.

Normal model of slider ratings

Consider Figure 2. This figure shows 200 simulated ratings on a [0, 1] slider scale (meaning that any value between 0 and 1, inclusive of the endpoints, is possible). I have also superimposed a blue curve of the best-fitting normal density on the histogram. The two most notable non-normal features of these data are that they are bounded at 0 and 1 where the data appears to “bunch”, and (possibly) skewed. Of course, these data were simulated; experience with slider scales tells me, however, that this histogram is not unrepresentative of such ratings.

set.seed(99)
rzoib <- function(n = 1e4, alpha = .1, gamma = .45, mu = .4, phi = 3) {
  a <- mu * phi
  b <- (1 - mu) * phi
  y <- vector("numeric", n)
  y <- ifelse(
    rbinom(n, 1, alpha),
    rbinom(n, 1, gamma),
    rbeta(n, a, b)
  )
  y
}
dat <- tibble(
  x = 1:200,
  n = 1,
  alpha = .1,
  gamma = .55,
  mu = .7,
  phi = 3
)
dat <- dat %>%
  rowwise() %>%
  mutate(Rating = rzoib(n, alpha, gamma, mu, phi)) %>%
  ungroup()
p1 <- dat %>%
  ggplot(aes(Rating)) +
  geom_histogram(
    aes(y = stat(ncount)),
    col = "white",
    fill = "black",
    bins = 100,
  ) +
  scale_y_continuous(
    expand = expansion(mult = c(0, .05))
  ) +
  scale_x_continuous(breaks = pretty_breaks()) +
  theme(
    axis.title.y = element_blank(),
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank()
  )

sims <- tibble(
  x = seq(-0.5, 1.8, by = .01),
  y = dnorm(x, mean(dat$Rating), sd(dat$Rating))
) %>%
  mutate(y = y / max(y))

p2 <- p1 +
  geom_line(
    data = sims,
    aes(x = x, y = y),
    col = "dodgerblue",
    size = 1
  )

Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

p2

Warning: `stat(ncount)` was deprecated in ggplot2 3.4.0.
ℹ Please use `after_stat(ncount)` instead.

While the height of the blue curve is not comparable to the heights of the bars (one represents a density, the other counts of observations in rating bins), it should be apparent that features of the rating scale data make the blue normal curve a poor representation of the data.

First, the skew apparent in the data is not captured by the normal density curve. Second, and perhaps more important, the blue curve does not respect the 0 and 1 boundaries of the slider scale data.

Focus on this latter point: We can see that the blue curve assigns density to areas outside the possible values: The model predicts impossible values with alarming frequency. Second, the boundary values 0.0 and 1.0 do not receive any special treatment under the normal model, but we can see that the data are bunched at the boundaries. The great frequency of responses at 0.0 and 1.0 leads to large prediction errors from the normal model of these data.

In other words, (simulated) subjects tend to give many extreme ratings. This is especially apparent in the low end of the rating scale, where the continuous spread of scores tapers off, but then there is a large spike of ratings at zero. The normal model misses these features of the data, and may therefore lead to unrepresentative estimates of the data generating process, and even erroneous conclusions.

Toward a better model

More generally, if your goal is to predict cognition and behavior (Yarkoni and Westfall 2017), a model that is obviously a poor representation of your data—in terms of having such a poor predictive utility—should not be your first choice for data analysis.

Admittedly, the data in Figure 2 were simulated, and it remains an empirical question as to how common these features are in real data, and how severe these issues are to normal models (t-test, ANOVA, correlation, etc.).

Nevertheless, it would be desirable to have an accessible data-analytic model for slider scale data, whose assumption better match observed features of the data. Here, I introduce one such model—the zero-one-inflated beta (ZOIB) model—and show how it can be applied to real data using the R package brms (Bürkner 2017). I also compare this model to standard analyses of slider scale data and conclude that the ZOIB can provide more detailed and accurate inferences from data than its conventional counterparts.

# https://imgflip.com/i/2u6fgk
include_graphics("zoidberg.jpg")

The zero-one-inflated beta model

Above, we established—rather informally—that normal models may be less than optimal for slider scale data. Of course, no model is the correct model of such data, but it would be desirable to use a model that best represents the data under study.

The model for analysis of slider scale data discussed here has been called the “zero-one-inflated beta” model, or ZOIB (Liu and Kong 2015). It is a model of data in the closed [0, 1] interval, and has two components: A beta distribution for responses in the closed (0, 1) interval, and a bernoulli distribution for the binary {0, 1} responses. Under this model, predictors can affect either or both the continuous and binary responses, the proportion of binary responses, or the spread of the continuous ratings.

To understand ZOIB, let’s start with a closer look at the theoretical beta density.

The beta distribution

The beta distribution used in beta regression (Ferrari and Cribari-Neto 2004) is a model of data in the open (0, 1) interval. (i.e. all values from 0 to 1, but not 0 and 1 themselves, are permitted.)

The beta distribution typically has two parameters, which in R are called shape1 and shape2. Together, they determine the location, spread, and skew of the distribution. Four example beta densities are shown in Figure 3. Using R’s dbeta(), I drew four curves corresponding to beta densities with different shape1 and 2 parameters.

tmp <- tibble(
  x = seq(0.001, 0.999, by = .001),
  dbeta(x, shape1 = 1, shape2 = 1),
  dbeta(x, shape1 = 2, shape2 = 8),
  dbeta(x, shape1 = 9, shape2 = 1),
  dbeta(x, shape1 = 35, shape2 = 35)
) %>%
  gather(Distribution, Density, -x) %>%
  mutate(Function = fct_inorder(Distribution))

tmp %>%
  ggplot(aes(x, Density)) +
  geom_line(size = 1.4, aes(col = Function)) +
  scale_x_continuous(
    breaks = pretty_breaks(),
    expand = expansion(mult = 0.005)
  ) +
  scale_y_continuous(
    breaks = pretty_breaks(),
    expand = expansion(mult = c(0.005, .01))
  )

This default parameterization is useful, for example, as a prior distribution for proportions: The shape1 and shape2 parameters can define the prior number of zeros and ones, respectively. For example, in the above figure, dbeta(x, shape1 = 1, shape2 = 1) results in a uniform prior over proportions, because the prior zeros and ones are 1 each.

However, for our purposes, it is more useful to parameterize the beta distribution with a mean and a precision. To convert the former parameterization to mean (which we’ll call (mu)) and precision ( (phi)), the following formulas can be used

(This parameterization is provided in R in the PropBeta functions from the extraDist package, which calls the precision parameter, or , size.) Redrawing the figure from above with this parameterization using the dprop() function, we get the figure below.

library(extraDistr)
tmp <- tibble(
  x = seq(0.001, 0.999, by = .001),
  dprop(x, size = 2, mean = .5),
  dprop(x, size = 10, mean = .2),
  dprop(x, size = 10, mean = .9),
  dprop(x, size = 70, mean = .5)
) %>%
  gather(Distribution, Density, -x) %>%
  mutate(Function = fct_inorder(Distribution))

tmp %>%
  ggplot(aes(x, Density)) +
  geom_line(size=1.4, aes(col = Function)) +
  scale_x_continuous(
    breaks = pretty_breaks(),
    expand = expansion(mult = 0.005)
    ) +
  scale_y_continuous(
    breaks = pretty_breaks(),
    expand = expansion(mult = c(0.005, .01))
    )

Shown above are four density functions of the beta family, whose precision and mean are varied. The first (red line) is a beta distribution with precision = 1, and mean = 0.5. It results in a uniform distribution. If a subject gave random slider scale responses, they might look much like this distribution (any rating is equally probably as any other rating).

The second beta distribution (green line) has precision 10, and mean 0.2. It is heavily skewed to the right. The third distribution (teal line) has precision 10, and a mean of 0.9. The fourth one, most similar to a normal distribution, has precision 70 and mean 0.50 (purple line).

In beta regression, this family of distributions is used to model observations, and covariates can have effects on both the mean and precision parameters.

However, beta regression only allows outcomes in the open (0, 1) interval. We know that slider scales often result in a bunching of values at the boundaries, and these boundary values might be informative of the participants’ cognition and behavior. To handle these extreme values, we can add a zero-one inflation process to the beta distribution.

set.seed(101)
tmp <- tibble(
  rzoib(a = 0, g = .5, m = .2, p = 6),
  rzoib(a = .1, g = .3, m = .5, p = 3),
  rzoib(a = .15, g = .7, m = .7, p = 4),
  rzoib(a = 1, g = .7, m = .1, p = 12)
) %>%
  gather(distribution, x) %>%
  mutate(distribution = fct_inorder(distribution))

tmp %>%
  ggplot(aes(x = x, y = stat(count), fill = distribution)) +
  geom_histogram(
    bins = 50, col = "white"
  ) +
  scale_x_continuous(
    "Rating",
    breaks = pretty_breaks()
  ) +
  scale_y_continuous(
    "Count",
    breaks = pretty_breaks(),
    expand = expansion(mult = c(0.0, .02))
  ) +
  facet_wrap("distribution", scales = "free_y") +
  theme(legend.position = "none", strip.text = element_text(size = 10))

ZOIB regression

In this example, we examine the ZOIB model in the context of one binary predictor variable (Group A vs B, a “between subjects” manipulation).

Example data

To illustrate the ZOIB model in action, I simulated a data set of 100 ratings from two groups, A and B. These data are shown in Figure 4.

set.seed(666)
dat <- tibble(
  x = rbinom(1e2, 1, .5),
  n = 1,
  alpha = .25,
  gamma = .5,
  mu = .6 + x * .15,
  phi = 5
) %>%
  rowwise() %>%
  mutate(Rating = rzoib(n, alpha, gamma, mu, phi)) %>%
  ungroup() %>%
  mutate(group = factor(x, levels = 0:1, labels = c("A", "B")))
dat <- select(dat, group, Rating)
dat %>%
  ggplot(aes(group, Rating)) +
  geom_beeswarm(
    shape = 21,
    fill = "white",
    size = 2,
    stroke = .8,
    alpha = .7,
    cex = 2
  ) +
  stat_summary(fun.data = mean_cl_boot, size = 1, col = "dodgerblue2") +
  scale_y_continuous(breaks = pretty_breaks())

kable(
  head(dat),
  digits = 2,
  caption = "First six rows of example data of two groups' slider scale ratings."
)

t.ex <- t.test(Rating ~ group, data = dat %>% mutate(group = fct_rev(group)))
t.ex.out <- tidy(t.ex) %>%
  mutate_if(is.numeric, round, 2) %>%
  glue_data(
    "B - A = {estimate}, 95%CI = [{conf.low}, {conf.high}], {pvalue(p.value, add_p=T)}"
  )
w.p <- tidy(wilcox.test(Rating ~ group, data = dat))$p.value %>%
  pvalue(add_p = T)
k.p <- tidy(kruskal.test(Rating ~ group, data = dat))$p.value %>%
  pvalue(add_p = T)

We are interested in the extent to which Group A’s ratings differ from Group B’s ratings. It is common practice to address this question with a t-test, treating the ratings as normally distributed within each group. I compared the two groups’ means with a t-test: The difference was not statistically significant (B - A = 0.06, 95%CI = [-0.07, 0.2], p=0.340). I’ve also heard that you can do something called a Mann-Whitney U test, or a Kruskal-Wallis test when you have a categorical predictor and don’t want to assume a parametric form for your outcomes. I tried those as well. Neither of these nonparametric tests were significant (p=0.226; p=0.225). I therefore concluded that I was unable to reject the null hypothesis that Group A and Group B’s population means are not different.

But as can be seen from Figure 2, the normal model makes unreasonable assumptions about these ratings. We see in Figure 4 that there are many non-normal features in this example data set; e.g. many values are bunched at 0.0 and 1.0. Let’s fit the ZOIB model on these data, and see if our conclusions differ. Spoiler alert: they do.

The model

We will model the data as ZOIB, and use group as a predictor of the mean and precision of the beta distribution, the zero-one inflation probability , and the conditional one-inflation probability . In other words, in this model group may affect the mean and/or precision of the assumed beta distribution of the continuous ratings (0, 1), and/or the probability with which a binary rating is given, and/or the probability that a binary rating is 1. How do we estimate this model?

It might not come as a surprise that we estimate the model with bayesian methods, using the R package brms (Bürkner 2017). Previously, I have discussed how to estimate signal detection theoretic models, “robust models”, and other multilevel models using this package. I’m a big fan of brms because of its modeling flexibility and post-processing functions: With concise syntax, you can fit a wide variety of possibly nonlinear, multivariate, and multilevel models, and analyze and visualize the models’ results.

Let’s load the package, and start building our model.

library(brms)

The R formula syntax allows a concise representation of regression models in the form of response ~ predictors. For a simple normal (i.e. gaussian) model of the mean of Ratings as a function of group, you could write Ratings ~ group, family = gaussian. However, we want to predict the four parameters of the ZOIB model, and so will need to expand this notation.

The brms package allows modeling more than one parameter of an outcome distribution. Specifically, we want to predict so-called “distributional parameters”, and bf() allows predicting them in their own formulas. Implicitly, Ratings ~ group means that you want to model the mean of Ratings on group. Therefore, to model , , and , we will give them their own regression formulas within a call to bf():

zoib_model <- bf(
  Rating ~ group,
  phi ~ group,
  zoi ~ group,
  coi ~ group,
  family = zero_one_inflated_beta()
)

The four sub-models of our model are, in order of appearance: 1. the model of the beta distribution’s mean (read, “predict Rating’s mean from group”). Then, 2. the model of phi; the beta distribution’s precision. 3. zoi is the zero-one inflation (); that is, we model the probability of a binary rating as a function of group. 4. coi is the conditional one-inflation: Given that a response was {0, 1}, the probability of it being 1 is modelled on group.

As is usual in R’s formula syntax, the intercepts of each of these formulas are implicitly included. (To make intercepts explicit, use e.g. Rating ~ 1 + group.) Therefore, this model will have 8 parameters; the intercepts are Group A’s mean, phi, zoi, and coi. Then, there will be a Group B parameter for each of them, indicating the extent to which the parameters differ for Group B versus Group A.

If group has a positive effect on (the mean of) Rating, we may conclude that the continuous rating’s mean differs as function of Group. On the other hand, if coi is affected by group, Group has an effect on the binary {0, 1} ratings. If group has no effects on any of the parameters, we throw up our hands and design a new study.

Finally, we specified family = zero_one_inflated_beta(). Just like logistic regression, ZOIB regression is a type of generalized linear model. Therefore, each distributional parameter is modeled through a link function. The mean, zoi, and coi parameters are modeled through a logit link function. Phi is modeled through a log link function. These link functions can be changed by giving named arguments to zero_one_inflated_beta(). It is important to keep in mind the specific link functions, we will need them when interpreting the model’s parameters.

To estimate this model, we pass the resulting zoib_model to brm(), with a data frame from the current R environment, 4 CPU cores for speed, and a file argument to save the resulting model to disk. The last two arguments are optional.

fit <- brm(
  formula = zoib_model,
  data = dat,
  cores = 4,
  file = "brm-zoib"
)

brms estimates the regression model using bayesian methods: It will return random draws from the parameters’ posterior distribution. It takes less than a minute to draw samples from this model. Let’s then interpret the estimated parameters (i.e. the numerical summaries of the posterior distribution):

summary(fit)

 Family: zero_one_inflated_beta 
  Links: mu = logit; phi = log; zoi = logit; coi = logit 
Formula: Rating ~ group 
         phi ~ group
         zoi ~ group
         coi ~ group
   Data: dat (Number of observations: 100) 
  Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup draws = 4000

Regression Coefficients:
              Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept         0.33      0.16     0.01     0.63 1.00     6253     3166
phi_Intercept     1.49      0.24     1.01     1.94 1.00     5683     3040
zoi_Intercept    -0.79      0.32    -1.44    -0.18 1.00     6968     2958
coi_Intercept     0.62      0.57    -0.46     1.79 1.00     6660     2940
groupB            0.91      0.21     0.52     1.33 1.00     5658     2983
phi_groupB        0.49      0.33    -0.16     1.14 1.00     5273     3223
zoi_groupB        0.08      0.43    -0.76     0.91 1.00     6631     2955
coi_groupB       -0.86      0.75    -2.36     0.58 1.00     7174     2930

Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

First, the summary of this model prints a paragraph of information about the model, such as the outcome family (ZOIB), link functions, etc. The regression coefficients are found under the “Population-Level Effects:” header. The columns of this section are “Estimate”, the posterior mean or point estimate of the parameter. “Est.Error”, the posterior standard deviation, or so called standard error of the parameter. Then, the lower and upper limit of the 95% Credible Interval. The two last columns are diagnostics of the model fitting procedure.

The first four rows of this describe the parameters for the baseline group (Group A). Intercept is the logit-transformed mean of the beta distribution for Group A’s ratings (the subset of ratings that were (0, 1)). Next, phi_Intercept describes the precision of the beta distribution fitted to Group A’s slider responses, on the scale of the (log) link function. zoi_Intercept is the zero or one inflation of Group A’s data, on the logit scale. coi_Intercept is the conditional one inflation; out of the 0 or 1 ratings in Group A’s data, describing the proportion of ones (out of the 0/1 responses)?

These parameters are described on the link scale, so for each of them, we can use the inverse link function to transform them to the response scale. Precision (phi_Intercept) was modeled on the log scale. Therefore, we can convert it back to the original scale by exponentiating. For the other parameters, which were modeled on the logit scale, we can use the inverse, which is plogis().

However, before converting the parameters, it is important to note that the estimates displayed above are summaries (means, quantiles) of the posterior draws of the parameters on the link function scale. Therefore, we cannot simply convert the summaries. Instead, we must transform each of the posterior samples, and then re-calculate the summaries. The following code accomplishes this “transform-then-summarize” procedure for each of the four parameters:

posterior_samples(fit, pars = "b_")[, 1:4] %>%
  mutate_at(c("b_phi_Intercept"), exp) %>%
  mutate_at(vars(-"b_phi_Intercept"), plogis) %>%
  posterior_summary() %>%
  as.data.frame() %>%
  rownames_to_column("Parameter") %>%
  kable(digits = 2)

Warning: Method 'posterior_samples' is deprecated. Please see ?as_draws for recommended alternatives.

Parameter	Estimate	Est.Error	Q2.5	Q97.5
b_Intercept	0.58	0.04	0.50	0.65
b_phi_Intercept	4.57	1.07	2.74	6.99
b_zoi_Intercept	0.32	0.07	0.19	0.45
b_coi_Intercept	0.64	0.12	0.39	0.86

We can then interpret these summaries, beginning with b_Intercept. This is the estimated mean of the beta distribution fitted to Group A’s (0, 1) rating scale responses (with its standard error, lower- and upper limits of the 95% CI). Then, b_Phi_Intercept is the precision of the beta distribution. zoi is the zero-one inflation, and coi the conditional one inflation.

To make b_zoi_Intercept concrete, we should be able to compare its posterior mean to the observed proportion of 0/1 values in the data:

mean(dat$Rating[dat$group == "A"] %in% 0:1) %>% round(3)

[1] 0.311

Above we calculated the proportion of zeros and ones in the data set, and found that it matches the estimated value. Similarly, for coi, we can find the corresponding value from the data:

mean(dat$Rating[dat$group == "A" & dat$Rating %in% 0:1] == 1) %>%
  round(3)

[1] 0.643

Let’s get back to the model summary output. The following four parameters are the effects of being in group B on these parameters. Most importantly, groupB is the effect of group B (versus group A) on the mean of the ratings’ assumed beta distribution, in the logit scale. Immediately, we can see that the parameter’s 95% Credible Interval does not include zero. Traditionally, this parameter would be called “significant”; group B’s (0, 1) ratings are on average greater than group A’s.

To transform this effect back to the data scale, we can again use plogis(). However, it is important to keep in mind that the effect’s size on the original scale depends on the intercept, getting smaller as the intercept increases (just like in any other generalized linear model.) The following bit of code transforms this effect and its uncertainty back to the original scale.

h <- c("B - A" = "plogis(Intercept + groupB) = plogis(Intercept)")
hypothesis(fit, h)

Hypothesis Tests for class b:
  Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1      B - A     0.19      0.04     0.11     0.29         NA        NA    *
---
'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
'*': For one-sided hypotheses, the posterior probability exceeds 95%;
for two-sided hypotheses, the value tested against lies outside the 95%-CI.
Posterior probabilities of point hypotheses assume equal prior probabilities.

The data were simulated with the rzoib() function, and I set . Therefore, the results of the t-tests and nonparametric tests were misses; a true effect was missed. On the other hand, the ZOIB regression model detected the true effect of group on the beta distribution’s mean.

Finally, let’s visualize this key finding using the conditional_effects() function from brms.

plot(
  conditional_effects(fit, dpar = "mu"),
  points = TRUE,
  point_args = list(width = .05, shape = 1)
)

Comparing Figure 5 to Figure 4 reveals the fundamental difference of the normal t-test model, and the ZOIB model: The ZOIB regression (Figure 5) has found a large difference between the continuous part of the slider ratings’ means because it has treated the data with an appropriate model. By conflating the continuous and binary data, the t-test did not detect this difference.

In conclusion, this example showed that ZOIB results in more informative, and potentially more accurate, inferences from analog scale (“slider”) data. Of course, in this simulation we had the benefit of knowing the true state of matters: The data were simulated from a ZOIB model. Nevertheless, we have reasoned that by respecting the major features of slider scale data, the ZOIB is a more accurate representation of it, and was therefore able to detect a difference where the t-test did not. Next, I put this conjecture to a test by conducting a small simulation study.

Simulation: Compare ZOIB and t-test performances

To compare the performance of the t-test and ZOIB in a little bit more detail, I conducted a small simulation study. I simulated 100 data sets of 200 ratings from two independent groups, from the ZOIB model (100 ratings per group). I set ; that is, there was a small effect of group on the mean of the beta distribution, and all other parameters were constant across groups. A sample of the resulting data sets is shown in Figure 6.

# Simulate datasets
K <- 100
set.seed(1)
rdata <- function(nsubs = 2e2) {
  dat <- tibble(
    x = rbinom(nsubs, 1, .5),
    n = 1,
    alpha = .2,
    gamma = .7,
    mu = .6 + x * .1,
    phi = 5
  ) %>%
    rowwise() %>%
    mutate(Rating = rzoib(n, alpha, gamma, mu, phi)) %>%
    ungroup() %>%
    mutate(group = factor(x, levels = 0:1, labels = c("A", "B")))
  dat
}
out <- tibble(set = 1:K, nsubs = 1e2)
dat <- out %>%
  mutate(data = map(nsubs, rdata))

# Get ttests
dat <- dat %>%
  mutate(
    ttest = map(
      data,
      ~ t.test(Rating ~ group, data = mutate(., group = fct_rev(group)))
    )
  ) %>%
  mutate(ttest.out = map(ttest, ~ broom::tidy(.)))

dat %>%
  sample_n(6) %>%
  unnest(data) %>%
  rename(Group = group) %>%
  ggplot(aes(Group, Rating)) +
  geom_beeswarm(
    shape = 21,
    fill = "white",
    size = 2,
    stroke = .8,
    cex = 2,
    alpha = .5
  ) +
  stat_summary(fun.data = mean_cl_boot, size = .8, col = "dodgerblue2") +
  scale_y_continuous(breaks = pretty_breaks()) +
  facet_wrap("set", labeller = label_both)

sig.ttests <- dat %>%
  unnest(ttest.out) %>%
  summarise(sigs = (sum(p.value < 0.05) / K) * 100) %>%
  pull(sigs)