The actual problem

I work on health policy topics, which means sources arrive from all directions: academic papers, policy documents, the occasional YouTube lecture from someone who really should have been a professor but ended up on the internet instead, podcasts, RSS feeds. Keeping track of all this was less of a system and more of an archaeology project — layers of half-processed material accumulating in various apps, with no clear path from “I found this interesting thing” to “I actually know something about this topic now.”

The obvious solution is a knowledge management tool. Obsidian is the one everyone recommends. I tried it twice, both times abandoned it within a week. The reason was always the same: manually adding notes, manually creating links between them, manually tagging everything. The whole premise of Obsidian — that you build up a connected web of knowledge over time — sounds wonderful until you’re staring at a blank note at 10pm after a full day of meetings, trying to remember why you bookmarked a paper about care substitution in 2022.

A cloud solution detour

A while ago I tried Recall.ai, which promises to solve exactly this problem: save anything to it, and it automatically connects related items, summarizes them, surfaces connections you hadn’t noticed. I liked the concept. The execution was genuinely impressive, many thanks for this experience! But I was hesitant about uploading the PDFs of academic papers — papers from journals that are already not thrilled about their content being distributed. Copyrighted material, on a cloud service, being processed by models I know nothing about. That felt like a problem I would create for future me, so I quit.

What I actually wanted was Recall.ai’s idea — automated linking, surfaced connections, growing smarter over time — but running entirely on my own machine.

The local setup

The hardware made this feasible in a way it wouldn’t have been two years ago. A Mac mini M4 with 24 GB of unified memory can run local language models at useful speeds, and its GPU handles audio transcription fast enough that a one-hour podcast takes about four minutes to process. That’s the boring infrastructure detail that makes everything else possible.

The software stack ended up being: Zotero as the central reference manager and inbox, Obsidian as the knowledge vault, Ollama for running a local language model (Qwen3.5:9b), yt-dlp for downloading transcripts and audio, and whisper.cpp for local transcription. None of this is new software — what took work was connecting them into something that actually flows.

The result is documented in a GitHub repository called ResearchVault, with a full installation guide if you want to try it yourself. Fair warning: it requires comfort with the command line — I run everything from the integrated terminal in VS Code. It is emphatically not a one-click setup.

Two newsletters were an important source of inspiration along the way: AI Report for its enthusiastic coverage of what’s becoming possible with Claude Code, and Causal Inference with AI for its focus on applying AI in academic research specifically. Both are worth following if this kind of thing interests you.

The privacy piece

All the processing — model inference, transcription, note generation — runs locally. Zotero’s API is localhost-only. Obsidian stores plain Markdown files on your local disk. No copyrighted PDFs or confidential items go anywhere they shouldn’t.

The one honest exception: when I type instructions into Claude Code, those prompts reach Anthropic’s API. The content of papers and notes does not. That tradeoff felt acceptable, and it’s explicit rather than buried in a terms-of-service document I’ll never read.

What it’s like to actually use it

We have just started, but the vault is already growing in a way that feels meaningful rather than just accumulating. Papers from different research lines are starting to link to each other in ways I hadn’t made explicit before. The connected graph that Obsidian keeps showing me in its visualization pane is starting to look less like a joke and more like something actually useful.

Whether all of this makes me a more productive researcher, or simply a researcher with a more aesthetically pleasing note-taking system, remains to be determined. I am still struggling to open an issue myself like phasing out Claude in its orchestrating role in this workflow, because this would be a very harsh thing to do to a co-author. Anyhow, happy (vibe) coding!

A note on what this isn’t

This is a personal system that happens to be documented well enough to share. It works for my situation — Mac mini 4, comfort with terminal commands, academic research (or other serious stuff that needs connecting the dots). If you’re on Windows, or if command lines make you nervous, or if your primary source material is TikTok, this is probably not for you. That’s fine.

If you try it and hit rough edges, the GitHub issues are open.

Citation

BibTeX citation:

@online{stam2026,
  author = {Stam, Piet and {Claude Sonnet 4.6}},
  title = {A privacy-first workflow for processing documents, videos, podcasts, and RSS feeds with local AI},
  date = {2026-03-14},
  url = {https://www.pietstam.nl/posts/2026-03-14-building-a-privacy-first-research-workflow/},
  langid = {en}
}

For attribution, please cite this work as:

Stam, Piet, and Claude Sonnet 4.6. 2026. “A privacy-first workflow for processing documents, videos, podcasts, and RSS feeds with local AI.” March 14, 2026. https://www.pietstam.nl/posts/2026-03-14-building-a-privacy-first-research-workflow/

A Stroke of Beauty: Enlisting Beautiful.ai

After signing up for the free trial, I got a 2-minute intro to become a pro in using Beautiful.ai (as they claim). I decided that surfing to their “Designer Bot AI” was the way to go for me, because then I could feed it with the results produced by ChatGPT and ask it to compose a matching design without any further intervention by me.

My prompt starts with this intro:

“A slide deck with the results of a strategic analysis for a startup company called Flowley that offers services to students to keep a healthy work-life balance. The strategic analysis consists of (1) the results of a SWOT-analysis and (2) the results of finding an optimal balance among their values (mission/scope/value), capabilities (strength/competitive advantage) and opportunities (market demand, who else, if anyone offers this proposition?). The results of the SWOT-analysis (1) are:”

after which I copied the ChatGPT results of the second task as described above. Also I added the text:

“The results of the search for the right balance (2) are:”,

after which I copied the ChatGPT results of the third task as described above. And then waited. And waited. And… Oops. there it was. Exactly 45 minutes after signing up with Beautiful.ai, there I had it: a beautifully designed slide deck with background images that are very much on topic. The inclusion of relevant images and slides addressing additional analyses showcased Beautiful.ai’s flair for captivating presentations.

Clash of the Titans: PowerPoint vs. Beautiful.ai

So how does the PowerPoint slide deck compare to the Beautiful.ai slide deck? And how does ChatGPT fit in? This is part of the evaluation phase of my journey by talking to with Flowley about the results. Without revealing the nitty gritty details of the next steps of the strategic positioning of Flowley, of course.

In the epic clash of PowerPoint and Beautiful.ai, each had its strengths. In general, the conclusion in the meeting with Flowley was that the PowerPoint slides contain more to-the-point information than the slide deck generated by Beautiful.ai. This is especially the case as all the information of the SWOT-analysis by ChatGPT was manually integrated into the Powerpoint slides. Note that this even holds for the detailed explanations of the separate items of the SWOT-analysis: these explanations show up when hovering over them at the second slide of the original Powerpoint file.

On the other hand, the slide deck produced by Beautiful.ai has a design including photos that more closely relates to the target group of Flowley. Moreover, the slide deck includes slides that are useful for having the discussion about strategic positioning with the customer. For example, there are slides about additional analyses that might be considered. And again on the design: it is very beautiful indeed.

The Final verdict

In conclusion, the partnership between ChatGPT and Beautiful.ai was a game-changer for my strategic analysis use case. The time saved and the exceptional results achieved were awe-inspiring. It was like having an AI-powered strategy consultant at my beck and call! While a complete strategic positioning requires more than a SWOT-analysis, the combined power of ChatGPT and Beautiful.ai proved invaluable for swift scans and initial meetings.

Prerequisites

• The RStudio IDE (to code the pipeline)

• A GitHub account (to be able to use GitHub Actions)

• A Netlify account (to publish the website to)

Step-by-step guide

Here, the steps are given that you must follow to build a Quarto website for the use case. In the next section, you will see how to get the same results fully relying on Netlify instead of GitHub Actions. After that, it is shown how to use the GitHub template. Using this template, you do not need to manually perform this guide step-by-step each time you need a Quarto website as a starter.

1. Fire up RStudio and choose File → New Project… → New Directory → Quarto Website

2. Fill out the details in the panel, for example like in the figure below, and click on the Create Project button:

3. Add a .gitattributes file and change the RStudio line endings setting to None along the lines of my gist about line ending issues with Quarto.

   Important

   Step 3 does no longer apply. I added this step 3 to my original post to prevent GitHub Actions from failing due to hash issues caused by end-of-line character differences between operating systems (see my gist for more). However, this step 3 is no longer needed since the removal of the hash issue by this pull request.

4. Execute the quarto publish command for Netlify in the Terminal pane of RStudio:

   quarto publish netlify

   After you press Y to publish with you default account, this automatically creates a _publish.yml file in the local repository that looks like the example below, but with the appropriate id and url values for your site:

   - source: project
     netlify:
       - id: d1983ae8-da83-4431-928f-3debf80a02a5
         url: 'https://splendorous-bublanina-d544bb.netlify.app'

5. Add the following lines to the _quarto.yml file in order to make sure that code is only executed locally:

   execute:
     freeze: auto  # Re-render only when source changes

6. Fully re-render your site in the Terminal pane:

   quarto render

7. Add the output directory of your project to .gitignore:

   /_site/

8. Go to the Git pane of RStudio to commit the files created thus far to your local repository: Git → Commit pending changes → select all files → Stage all files → write a Commit message, for example ‘First commit’ → click the Commit button.

9. Execute this command in the Console pane of RStudio to push your local repo to GitHub:

   usethis::use_github(private = TRUE)

   This will create a private remote repository at GitHub. You may change the argument private to FALSE if you want the remote repository to be public right away, or change this manually in the settings section of your GitHub repository at some later moment.

10. Add GitHub Actions to your project by creating the YAML file publish.yml and save it to .github/workflows/publish.yml:

    on:
      workflow_dispatch:
      push:
        branches: main
    
    name: Quarto Publish
    
    jobs:
      build-deploy:
        runs-on: ubuntu-latest
        steps:
          - name: Check out repository
            uses: actions/checkout@v2 
    
          - name: Set up Quarto
            uses: quarto-dev/quarto-actions/setup@v2
    
          - name: Render and Publish
            uses: quarto-dev/quarto-actions/publish@v2
            with:
              target: netlify
              NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}

11. Configure your GitHub with the credentials required for publishing to Netlify. This is explained in the Quarto docs. In short:

    • find/create a personal access token at the Netlify applications page
    • go to the remote repository at GitHub → Settings → Secrets and variables → Actions → click on the button New repository secret
    • fill out the name (i.e. NETLIFY_AUTH_TOKEN) and value of the personal access token

12. Commit the publish.yml file and subsequently push your local repository (including the _freeze directory) to GitHub. As a consequence, your GitHub Actions workflow will start running and the website is automatically rendered and published to Netlify. Each subsequent (commit and) push will trigger the GitHub Actions workflow to start running in the same way.

The Quarto website that is described in this example was automatically named https://splendorous-bublanina-d544bb.netlify.app/] by Netlify. To remember it better, I renamed it to https://template-quarto-website.netlify.app/ in my Netlify account. I changed the url field in the _publish.yml file accordingly. You can visit the Quarto website at this latter URL.

The remote repository is located at https://github.com/pjastam/template-quarto-website/. Note that this repository is set up as a template, see the section “A versatile template”.

GitHub Actions

Throughout this guide, I have assumed the utilization of a GitHub Actions Runner to execute the job defined by the GitHub Actions workflow. Note that achieving the same outcome is possible without relying on GitHub Actions. More specifically, you can delegate the responsibility of rendering the website to Netlify itself, which seamlessly handles the entire process before deploying it to their servers. To follow this alternative approach, simply skip the last three steps (10-12) outlined in this how-to guide and add two extra files after adding the .gitigore file (i.e., after step 7). The files that must be added are a netlify.toml file with the following lines of code:

[[plugins]]
package = "@quarto/netlify-plugin-quarto"

and a package.json file with the following lines of code:

{
  "dependencies": {
    "@quarto/netlify-plugin-quarto": "^0.0.5"
  }
}

Finalize with committing all files to you local repository (step 8) and pushing your local repository to GitHub (step 9). Again, do not execute steps 10-12 in this scenario.

A versatile template

The remote repository that results from the steps described in this step-by-step guide is located at https://github.com/pjastam/template-quarto-website. To use it as a template repository for your Quarto websites, I have checked the Template repository box in the Settings tab of this remote repository. Furthermore, note that the two lines of code with the Netlify credentials (see step 3 of the step-by-step guide) are commented out (i.e. deactivated). In this way, people are guarded from mistakenly using these credentials instead of the Netlify credentials of their own site.

To actually use this template, there are 6 steps involved:

1. Use my template to create your new repository by clicking this button at the top of the remote repository at GitHub:

   

2. Fire up RStudio, choose File → New Project… → Version Control → Git and fill out the web URL of your new repository to clone it to your local environment. You can find this web URL at the top of your new repo at GitHub:

   

3. Delete the _publish.yml file from your local repository.

4. Execute step 4 of the step-by-step guide.

   Caution

   Before continuing with step 5, please double-check that the _publish.yml file in your local repository now contains your own Netlify credentials. Otherwise the workflow will never run the right way.

5. Execute step 11 of the step-by-step guide.

6. Execute step 12 of the step-by-step guide.

Your Quarto website will be up and running any moment.

Read data

The example data that I will use is in the file called wic.txt. These are the example data that the aforementioned authors use to illustrate the application of their method in SAS. In this example, they focused on the association between receipt of WIC (The Special Supplemental Nutrition Program for Women, Infants, and Children) food for the mother during this pregnancy and gestational weight gain. The example data are scraped from the fourth section of their Supplemental Digital Content document.

It contains four variables:

• Weight gain during pregnancy in pounds (lb) (wtgain)

• Receipt of WIC food (wic)

• Race/ethnicity (mracethn)

• Late or no prenatal care (latecare)

Let’s read the data.

library(readtext)
library(dplyr)
data <- readtext("https://cdn-links.lww.com/permalink/ede/a/ede_25_6_2014_07_23_urquia_ede14-408_sdc1.doc") %>% 
  subset(select = "text") %>% 
  { read.table(text=sub(".*4) individual-level dataset", "", .), header=TRUE) }

Apply transformations

Before we perform the regression, we apply data transformations in the same way that they are applied in the second section of the Supplemental Digital Content document. First, the weight gain during pregnancy in kilograms (wtgaink) is calculated by multiplication of the weight gain during pregnancy in pounds (lb) (wtgain) by the conversion factor 0.453592. Second, the categorical variables wic, mracethn and latecare are transformed to the factor type. Third, the reference level of the categorical variables mracethn is set to the last category called ‘Non-Hispanic Blacks’. Finally, the wtgain variable is no longer needed and therefore dropped.

db_indiv <- data %>%
  rename_all(tolower) %>%
  mutate(
    wtgaink = wtgain * 0.453592,
    wic = as.factor(wic),
    mracethn = as.factor(mracethn),
    mracethn = relevel(mracethn, ref = 3),
    latecare = as.factor(latecare)
  ) %>%
  select(wtgaink, wic, mracethn, latecare)

Aggregate data

Next let’s aggregate the data set. The aggregate data set contains averages of the outcome variable and frequencies of the categorical variables for all combinations of the set of categorical predictors. Thus, for all unique combinations of the categorical variables, we calculate the number of individual records N, the mean meany of the continuous variable wtgaink, and its standard deviation stdy. These aggregates suffice for our exercise to derive the individual-level point estimates and their standard errors afterwards.

db_aggr <-
  db_indiv %>%
  group_by(across(c(wic, mracethn, latecare))) %>%
  summarise(
    meany = mean(wtgaink),
    stdy = sd(wtgaink),
    N = length(wtgaink)
  ) %>%
  ungroup()

Show table

The variable factor has 5 identical elements equal to 0.9766333, which differs only very slightly from the true inflation factor. The resulting standard errors are reported in the table below. These match the standard errors estimated by the individual-level regression as desired. In addition to the standard errors, the table also shows the estimated coefficients, the lower and upper bounds of the confidence intervals, the z-scores and the p-values. As a validity check, compare the estimates reported here with the individual-level regression estimates shown above and with the estimates reported in the Moineddin and Urquia publication.

library(insight)
standard_errors %>%
  mutate(
    estimate = coefficients(model_aggr),
    z = estimate / standard_error,
    p = 2 * (1 - pnorm(abs(z))),
    pvalue = format_p(p, stars = TRUE),
    CI_low = estimate - 1.96 * standard_error,
    CI_high = estimate + 1.96 * standard_error,
  ) %>%
  bind_cols("predictors" = names(standard_errors$standard_error)) %>% 
  select(predictors, estimate, standard_error, CI_low, CI_high, z, pvalue) %>%
  format_table(digits = 3, ci_digits = 3)

   predictors estimate standard_error               CI      z      pvalue
1 (Intercept)   14.587          0.262 [14.074, 15.101] 55.681 p < .001***
2        wicY    0.565          0.207 [ 0.159,  0.971]  2.726 p = 0.006**
3   mracethn1   -0.493          0.244 [-0.972, -0.014] -2.017  p = 0.044*
4   mracethn2    1.094          0.223 [ 0.657,  1.531]  4.902 p < .001***
5   latecare1   -0.201          0.167 [-0.529,  0.126] -1.206   p = 0.228

Data and method

The models that I estimate are linear regression models with a set of predictors and one numeric outcome variable. The parameters of this model are estimated by ordinary least squares.

I use the car_prices data set for the examples and try to predict the car prices with the care brands as predictors. Note that, as a consequence, in my examples the outcome variable is non-negative and the predictors are mutually exclusive (0/1) dummy variables. This makes the examples easy to understand, but the code may apply to a wider range of variables nonetheless. I use mileage as the weighting variable.

Let us start with loading the data into memory.

# Load library for the recipe. parsnip, workflow and hardhat packages, along with the rest of tidymodels
library(tidymodels)

── Attaching packages ────────────────────────────────────── tidymodels 1.0.0 ──

✔ broom        1.0.1      ✔ recipes      1.0.1 
✔ dials        1.0.0      ✔ rsample      1.1.0 
✔ dplyr        1.0.10     ✔ tibble       3.1.8 
✔ ggplot2      3.3.6      ✔ tidyr        1.2.0 
✔ infer        1.0.3      ✔ tune         1.0.0 
✔ modeldata    1.0.0      ✔ workflows    1.0.0 
✔ parsnip      1.0.1      ✔ workflowsets 1.0.0 
✔ purrr        0.3.4      ✔ yardstick    1.0.0 

── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ purrr::discard() masks scales::discard()
✖ dplyr::filter()  masks stats::filter()
✖ dplyr::lag()     masks stats::lag()
✖ recipes::step()  masks stats::step()
• Dig deeper into tidy modeling with R at https://www.tmwr.org

Now we select only the relevant variables. Although the weights are not yet used in the first example, mileage is already defined as the weighting variable in the data set.

# Create a data set with one non-negative continuous variable and uncorrelated dummy variables as predictors
db <- select(car_prices, Price, Buick:Saturn, Mileage) %>% 
  mutate(Mileage = frequency_weights(Mileage))
str(db)

tibble [804 × 8] (S3: tbl_df/tbl/data.frame)
 $ Price   : num [1:804] 22661 21725 29143 30732 33359 ...
 $ Buick   : int [1:804] 1 0 0 0 0 0 0 0 0 0 ...
 $ Cadillac: int [1:804] 0 0 0 0 0 0 0 0 0 0 ...
 $ Chevy   : int [1:804] 0 1 0 0 0 0 0 0 0 0 ...
 $ Pontiac : int [1:804] 0 0 0 0 0 0 0 0 0 0 ...
 $ Saab    : int [1:804] 0 0 1 1 1 1 1 1 1 1 ...
 $ Saturn  : int [1:804] 0 0 0 0 0 0 0 0 0 0 ...
 $ Mileage : freq_wts [1:804] 20105, 13457, 31655, 22479, 17590, 23635, 17381, 2755...

Example 1: linear regression without weights

Now on with the first example. In the code below we define the recipe, define the model and set mode and engine. These are combined into a workflow. Afterwards we look at the properties of these objects to check if these are as expected. Note that Saturn is the reference dummy variable of my choice (i.e. in effect its coefficient is set to zero by default) and is thus excluded from the regression.

# Get data ready for modeling with recipe package
recipe1 <-
  db %>% 
  recipe(Price ~ 1 + Buick + Cadillac + Chevy + Pontiac + Saab) # add all dummy variables but one

# Define model, mode and engine with parsnip package
model1 <-
  linear_reg() %>% # adds the basic model type
  set_engine('lm') %>% # adds the computational engine to estimate the model parameters
  set_mode('regression') # adds the modeling context in which it will be used

# Bundle pre-processing, modeling, and post-processing with workflow package
workflow1 <-
  workflow() %>%
  add_recipe(recipe1) %>%
  add_model(model1)

# View object properties
recipe1

Recipe

Inputs:

      role #variables
   outcome          1
 predictor          5

model1

Linear Regression Model Specification (regression)

Computational engine: lm 

workflow1

══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: linear_reg()

── Preprocessor ────────────────────────────────────────────────────────────────
0 Recipe Steps

── Model ───────────────────────────────────────────────────────────────────────
Linear Regression Model Specification (regression)

Computational engine: lm 

Now that the objects look alright, the model estimation can be performed and the parameter estimates are printed.

# Now estimate the model via a single call to fit()
fit1 <- fit(workflow1, data = db)

# View fit1 properties
tidy(fit1)

# A tibble: 6 × 5
  term        estimate std.error statistic   p.value
  <chr>          <dbl>     <dbl>     <dbl>     <dbl>
1 (Intercept)   13979.      763.     18.3  7.62e- 63
2 Buick          6836.     1009.      6.77 2.46e- 11
3 Cadillac      26958.     1009.     26.7  9.16e-113
4 Chevy          2449.      832.      2.94 3.32e-  3
5 Pontiac        4433.      903.      4.91 1.10e-  6
6 Saab          15516.      943.     16.5  1.26e- 52

glance(fit1)

# A tibble: 1 × 12
  r.squared adj.r.s…¹ sigma stati…²   p.value    df logLik    AIC    BIC devia…³
      <dbl>     <dbl> <dbl>   <dbl>     <dbl> <dbl>  <dbl>  <dbl>  <dbl>   <dbl>
1     0.645     0.642 5911.    290. 1.53e-176     5 -8120. 16254. 16287. 2.79e10
# … with 2 more variables: df.residual <int>, nobs <int>, and abbreviated
#   variable names ¹​adj.r.squared, ²​statistic, ³​deviance

Example 2: linear regression with weights

Then come the weights. The first thought is to update the current workflow with a line of code to make clear that weights should be used. However, this approach does not produce the desired result.

Therefore, an alternative approach is followed. Instead of building upon the blocks of the first example, we start with a new workflow() object and add an add_case_weights line of code to it. Next, one would expect a line of code with an add_recipe command, but for some reason this did not work after a “few” tries. Instead, we use add_formula with the regression formula as an argument. Lastly, surprisingly conventional, an add_model command is added.

workflow2 <-
  workflow() %>%
  add_case_weights(Mileage) %>%
  add_formula(Price ~ 1 + Buick + Cadillac + Chevy + Pontiac + Saab) %>%
  add_model(model1)

workflow2

══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Formula
Model: linear_reg()

── Preprocessor ────────────────────────────────────────────────────────────────
Price ~ 1 + Buick + Cadillac + Chevy + Pontiac + Saab

── Case Weights ────────────────────────────────────────────────────────────────
Mileage

── Model ───────────────────────────────────────────────────────────────────────
Linear Regression Model Specification (regression)

Computational engine: lm 

Now the parameters are estimated with one line of code as follows.

fit2 <- fit(workflow2, db)

# View fit2 properties
tidy(fit2)

# A tibble: 6 × 5
  term        estimate std.error statistic   p.value
  <chr>          <dbl>     <dbl>     <dbl>     <dbl>
1 (Intercept)   13448.      694.     19.4  8.70e- 69
2 Buick          7006.      918.      7.63 6.52e- 14
3 Cadillac      26152.      933.     28.0  7.96e-121
4 Chevy          2452.      759.      3.23 1.28e-  3
5 Pontiac        4647.      828.      5.61 2.73e-  8
6 Saab          15349.      853.     18.0  5.72e- 61

glance(fit2)

# A tibble: 1 × 12
  r.squared adj.r.…¹  sigma stati…²   p.value    df logLik    AIC    BIC devia…³
      <dbl>    <dbl>  <dbl>   <dbl>     <dbl> <dbl>  <dbl>  <dbl>  <dbl>   <dbl>
1     0.664    0.661 7.67e5    315. 5.90e-186     5 -8111. 16237. 16269. 4.70e14
# … with 2 more variables: df.residual <int>, nobs <int>, and abbreviated
#   variable names ¹​adj.r.squared, ²​statistic, ³​deviance

This is a nice first try! With the two examples above it is possible to experiment further in the hope of alternative/shorter routes to the estimation results. In the mean time, we wait for the tidymodels to include weights in the relevant packages. If you are inspired by these two examples (or not) and have some new ideas for progress, do not hesitate to give feedback to the Tidyverse developers.

Happy coding!

Prerequisites

• A GitHub account (to set up a _redirects file)

• A Netlify account (to execute the _redirects file)

• A Brave/Chrome browser

Step-by-step guide: URL shortener

• Purchase a domain to use as your short link domain

• Make up a very, very domain name (in my case, I imagined pst.am would be nice)

• Set up a new site at Netlify

• Buy your domain name from Netlify during this set up

• Alternatively, buy your domain name from another registrar, which is what I did

  • I went to iwantmyname.com and registered the domain name

  • At Netlify, after the new site has been set up, find out how the name servers are called

  • At iwantmyname.com, delegate the DNS of yor domain name to Netlify

• Create a private git repository called url-shortener at GitHub

• Create a _redirects file and add some initial configuration lines to map short links to full URLs

• Login to Netlify and connect the GitHub repo

• Enable automatic TLS certificates with Let’s Encrypt for your short domain name at Netlify

• Wait until the domain name is served by Netlify’s DNS

  • This may take up to 24 hours, depending on your registrar processing the change in nameservers

• The first line of code in the _redirects file is a helper function for easily editing the _redirects file in your GitHub repository online

  • Open your browser
  • Type pst.am/edit
  • Commit your changes

• The second line of code in the _redirects file links to a default page, in this case the homepage of my website, if you type the short domain name without extension

  • Open your browser
  • Type pst.am
  • Hit the enter key

• To add new redirects, start editing the _redirects file and add a line of code starting with a \ followed by a short word or abbreviation, then at least one space, and finally the URL to which the short word or abbreviation should redirect.

Step-by-step guide: Keystrokes-shortener

• Open your Brave/Chrome browser

• Navigate to: Settings -> Search Engine -> Manage Search Engines and Safe Search -> Site Search -> Click the Add button

• Fill out pst.am as the search engine, :x as the shortcut and the URL https://pst.am/%s

• Click the Add button

• Note that in this guide I use :x as a shortcut, you can use this too or replace it by another shortcut if you like

• You can now edit the _redirects file in your GitHub repository online as follows

  • Open your browser
  • Type :x
  • Press space bar or tab key
  • Type edit
  • Hit return key

• Or, if you want to go to my homepage

  • Open your browser
  • Type :x
  • Hit return key

A next step?

I occasionally use my workflow to shorten a URL for a new presentation. Setting up new presentation slides does not happen every day in my case, so editing the _redirects file online at GitHub every now and then is no bother.

But you may like to automate this part of the workflow if you need to shorten URLs on a more frequent basis, for example in case of creating your daily posts. Kent C. Dodds shows you how in his video at YouTube. The downside of this is that those links will be rather cryptic. Like the ones you get if you use the bit.ly shortened URLs, for example. This makes the shortened URL difficult to remember.

Privacyvoorwaarden

Een vruchtbare weg om bij de realisatie van de apps onze privacy te borgen is als we komen tot een lijst van voorwaarden c.q. een beschrijving van een situatie waarin het delen van persoonlijke data vanuit Europese normen en waarden aanvaardbaar is. Apps kunnen dan aan de hand van deze lijst volgens het principe ‘privacy by design’ ontworpen en gebouwd worden. Hieronder volgt onze aftrap voor zo’n lijst met noodzakelijke voorwaarden (als je op het driehoekje klikt dan volgt een nadere toelichting):

Oproep aan het kabinet

Wij hopen dat het kabinet alleen besluit tot inzet van apps met borging van onze privacy. Zeker als die apps verplicht worden gesteld, maar evenzeer als deze op vrijwillige basis te gebruiken zijn. Tot op heden is er geen sluitend bewijs dat het inzetten van apps het hogere doel dient. ‘Baat het niet, schaadt het niet’ gaat hier niet op!

Onze zorgen over de verspreiding van het coronavirus zijn groot, maar onze privacy is ook een groot goed en angst is een slechte raadgever. In alle opzichten geldt: ‘voorkomen is beter dan genezen’.

Commmunity

Wij, 3 bezorgde burgers, hebben dit voorstel op persoonlijke titel geschreven. Onze namen zijn:

• Piet Stam, gezondheidseconoom
• Peter Nobels, zorg-IT management consultant
• Marco Woesthuis, arts (niet praktizerend)

Wij pretenderen geenszins de wijsheid in pacht te hebben, onze privacy gaat ons immers allemaal aan. We hopen daarom op inbreng van andere bezorgde burgers, zodat de lijst wordt aangescherpt en we samen de overheid aan een maatschappelijk gedragen toetsingskader helpen. Daarom hebben we de lijst met 10 uitgangspunten in een Wiki gezet, zodat iedereen kan meehelpen aan de aanscherping. Het enige dat je daarvoor nodig hebt is een GitHub login.

Originele versie

De originele versie van ons voorstel zoals die hierboven staat, is terug te vinden in de openbare repository op GitHub. Op Wikipedia vind je de beschrijving van wat GitHub is. Zie de README in de GitHub repository voor verdere details en contactinfo.

Post Scriptum

Op LinkedIn heb ik de dag voor de wedstrijd op 12 mei 2019 het volgende commentaar toegevoegd

Zoals de Engelsen zeggen: “put your money where your mouth is”. En dus heb ik op winst voor AZ en de gelijkspelletjes 0-0 en 1-1 elk een eurootje ingezet. Volgens mijn Bayesiaanse model zijn de kansen op de Toto goksite van de Nederlandse Loterij te laag ingeschat, dus hier valt mogelijk wat te halen. Fingers crossed!

Op LinkedIn heb ik na afloop van de wedstrijd op 12 mei 2019 het volgende commentaar toegevoegd

Gewonnen! Nee, ik bedoel niet het kampioenschap van #Ajax, maar de Toto! Door de winst van #AZ met 3 euro inleg 4,15 euro verdiend, dus een winstmarge van 1,15 / 3 = 38%. Waar krijg je dat rendement tegenwoordig nog? In ieder geval niet op mijn spaarrekening. En dat allemaal mbv mijn #Bayesiaanse voorspelmodelletje. Dat belooft nog wat voor volgend seizoen. Ook maar eens nadenken hoe we dit #algoritme voor serieuzere zaken zinvol kunnen inzetten. #statistics #bestguess

Quick summary

We applied the original work of Rasmus Baath to the Dutch Eredivisie football competition. With r-bayesian-football-odds we estimated the odds of football matches in the last two weeks of the 2018/2019 Dutch Eredivisie football competiton. We provide the code and evaluate the results of our predictions.

Acknowledgements

This piece of work is based on the works of Rasmus Baath. Rasmus Baath submitted his code to the UseR 2013 data analysis contest and licensed it under the Creative Commons Attribution 3.0 Unported license.

He predicted the results of the 50 last matches of the 2012/2013 Spanish LaLiga season. He used data of the 2008/09-2012/13 seasons (5 seasons in total) to estimate his regression model in a Bayesian way. See this thread for an intuitive explanation of the difference between the bayesian approach and the conventional approaches of linear regression and maximum likelihood.

I slightly adpated his code to predict the results of the last two competition rounds (that is, the last 18 matches) of the 2018/2019 Dutch Eredivisie season. These predictions are based on soccer match data of the 2014/15-2018/19 seasons (5 seasons in total). The source of these data is here. Out of the three model specifications that Rasmus developed, I used the most sophisticated model that allowed for year-to-year variability in team skill (called “iteration 3” by Rasmus).

You can find my code at GitHub. Rasmus deserves all the credits, I deserve all the blame in case of any errors in my application to the Dutch football competition.

Data and methods

from_year <- 2014
to_year <- 2019
source(paste0("functions/Import_Data_Eredivisie.R"))

eredivisie <- eredivisie %>%
  mutate(MatchResult = sign(HomeGoals - AwayGoals)) # -1 Away win, 0 Draw, 1 Home win

# Creating a data frame d with only the complete match results
d <- na.omit(eredivisie)

# Lists with the unique names of all teams and all seasons in the database
teams <- unique(c(d$HomeTeam, d$AwayTeam))
seasons <- unique(d$Season)

# A list for JAGS with the data from d where the strings are coded as integers
data_list <- list(HomeGoals = d$HomeGoals, AwayGoals = d$AwayGoals, 
                  HomeTeam = as.numeric(factor(d$HomeTeam, levels=teams)),
                  AwayTeam = as.numeric(factor(d$AwayTeam, levels=teams)),
                  Season = as.numeric(factor(d$Season, levels=seasons)),
                  n_teams = length(teams), n_games = nrow(d), 
                  n_seasons = length(seasons))

qplot(Season, HomeTeam, data=d, ylab="Team", xlab = "Season")

Results

team_skill <- ms3[, str_detect(string=colnames(ms3), paste0("skill\[",to_year-from_year,","))]
team_skill <- (team_skill - rowMeans(team_skill)) + ms3[, paste0("home_baseline[",to_year-from_year,"]")]
team_skill <- exp(team_skill)
colnames(team_skill) <- teams
team_skill <- team_skill[,order(colMeans(team_skill), decreasing=T)]
old_par <- par(mar=c(2,0.7,0.7,0.7), xaxs='i')
caterplot(team_skill, labels.loc="above", val.lim=c(0.7, 3.8))

par(old_par)

plotPost(team_skill[, "Ajax"] - team_skill[, "PSV Eindhoven"], compVal = 0, xlab = "<- PSV     vs     Ajax ->")

###                                mean    median      mode hdiMass     hdiLow
### <- PSV     vs     Ajax -> 0.1616095 0.1535391 0.1467396    0.95 -0.3586123
###                             hdiHigh compVal pcGTcompVal ROPElow ROPEhigh
### <- PSV     vs     Ajax -> 0.6842824       0      0.7312      NA       NA
###                           pcInROPE
### <- PSV     vs     Ajax ->       NA

eredivisie_forecast <- eredivisie[is.na(eredivisie$HomeGoals), c("Season", "Week", "HomeTeam", "AwayTeam")]
m3_forecast <- m3_pred[is.na(eredivisie$HomeGoals),] 
eredivisie_forecast$mean_home_goals <- round(m3_forecast[,"mean_home_goal"], 1) 
eredivisie_forecast$mean_away_goals <- round(m3_forecast[,"mean_away_goal"], 1)
eredivisie_forecast$mode_home_goals <- m3_forecast[,"mode_home_goal"] 
eredivisie_forecast$mode_away_goals <- m3_forecast[,"mode_away_goal"]
eredivisie_forecast$predicted_winner <- ifelse(m3_forecast[ , "match_result"] == 1, eredivisie_forecast$HomeTeam, 
                                           ifelse(m3_forecast[ , "match_result"] == -1, eredivisie_forecast$AwayTeam, "Draw"))

rownames(eredivisie_forecast) <- NULL
print(xtable(eredivisie_forecast, align="cccccccccc"), type="html")

eredivisie_sim <- eredivisie[is.na(eredivisie$HomeGoals), c("Season", "Week", "HomeTeam", "AwayTeam")]
eredivisie_sim$home_goals <- m3_forecast[,"rand_home_goal"] 
eredivisie_sim$away_goals <- m3_forecast[,"rand_away_goal"]
eredivisie_sim$winner <- ifelse(m3_forecast[ , "rand_match_result"] == 1, eredivisie_forecast$HomeTeam, 
                            ifelse(m3_forecast[ , "rand_match_result"] == -1, eredivisie_forecast$AwayTeam, "Draw"))

rownames(eredivisie_sim) <- NULL
print(xtable(eredivisie_sim, align="cccccccc"), type="html")

plot_goals <- function(home_goals, away_goals) { 
  old_par <- par(mar = c(0, 0, 0, 0))
  par(mfrow = c(2, 2), mar=rep(2.2, 4))
  
    n_matches <- length(home_goals) 
    goal_diff <- home_goals - away_goals 
    match_result <- ifelse(goal_diff < 0, "away_win", ifelse(goal_diff > 0, "home_win", "equal")) 
    hist(home_goals, xlim = c(-0.5, 10), breaks = (0:100) - 0.5)
    hist(away_goals, xlim = c(-0.5, 10), breaks = (0:100) - 0.5) 
    hist(goal_diff, xlim = c(-6, 6), breaks = (-100:100) - 0.5)
    barplot(table(match_result)/n_matches, ylim = c(0, 1))
    par(old_par)
}

n <- nrow(ms3)
home_team <- which(teams == "AZ Alkmaar")
away_team <- which(teams == "PSV Eindhoven")
season <- which(seasons == paste0(to_year-1,"/",to_year))
home_skill <- ms3[, col_name("skill", season, home_team)] 
away_skill <- ms3[, col_name("skill", season, away_team)]
home_baseline <- ms3[, col_name("home_baseline", season)]
away_baseline <- ms3[, col_name("away_baseline", season)]

home_goals <- rpois(n, exp(home_baseline + home_skill - away_skill))
away_goals <- rpois(n, exp(away_baseline + away_skill - home_skill))

old_par <- par(mfrow = c(2, 2), mar=rep(2.2, 4))
plot_goals(home_goals, away_goals)

par(old_par)

1 / c(AZ =  mean(home_goals > away_goals), Draw = mean(home_goals == away_goals), PSV = mean(home_goals < away_goals))

###       AZ     Draw      PSV 
### 3.928759 4.332756 1.943005

goals_payout <- laply(0:6, function(home_goal) {
  laply(0:6, function(away_goal) {
    1 / mean(home_goals == home_goal & away_goals  == away_goal)
  })
})

colnames(goals_payout) <- paste("PSV Eindhoven", 0:6, sep=" - ")
rownames(goals_payout) <- paste("AZ Alkmaar", 0:6, sep=" - ")
goals_payout <- round(goals_payout, 1)
print(xtable(goals_payout, align="cccccccc"), type="html")