Sentiment of Security Now! over time

 Posted on May 29, 2018  |  7 minutes (1332 words)

If you believe some people, everything is getting worse1. More so in infosec. For the past few years I listened to many many hours of podcasts, many hours where spent on the weekly show Security Now!. The hosts Steven Gibson and Leo Laporte have been talking about security related news every week over 13 years. Although the content has changed over time, there used to be more explanations but the majority of time is now filled with news, we could use the sentiment in the episodes to see if ‘everything is getting worse’. Has the sentiment of the security now! podcast changed over time? It helps that every episode is transcribed into text so we can use natural language processing tools to work through this problem.

Extracting the data

To gather and extract the relevant information from the transcripts I point you kindly to a seperate github page where I explain how I downloaded every episode and extracted the structure. https://github.com/RMHogervorst/NLP_SN

I asked permission to scrape all the transcripts but I’m not entirely sure if I can share the content. In any case I’d guide you to the links at the bottom of this page with all the archive. I might share the rda file with the episode information and transcripts into 1 dataframe.

What I ended up with is a dataframe with 664 rows (the number of episodes today) and 9 columns.

library(tidytext)
library(tidyverse)
## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 2.2.1.9000     ✔ purrr   0.2.5     
## ✔ tibble  1.4.2          ✔ dplyr   0.7.5     
## ✔ tidyr   0.8.1          ✔ stringr 1.3.1     
## ✔ readr   1.1.1          ✔ forcats 0.3.0
## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
library(widyr) 
if(!file.exists("df_sn.RDS")){
    download.file(
        url = "https://github.com/RMHogervorst/NLP_SN/raw/master/df_sn.RDS",
        destfile =  "df_sn.RDS",cacheOK = TRUE)    
}

df_sn <- read_rds("df_sn.RDS")  
#df_sn <- read_rds("../NLP_SN/df_sn.RDS") # when working locally
# 
# Every week / or once a month at least I will update the file with new transcript(s) so your data could be newer than this post.
#dim_desc(df_sn)
tail(df_sn,2)
## # A tibble: 2 x 9
##   path             source date  description ep_nr hosts teaser title text 
##   <chr>            <chr>  <chr> <chr>       <int> <lis> <chr>  <chr> <lis>
## 1 data//sn-663.txt https… May … This week …   663 <lis… It's … Ultr… <tib…
## 2 data//sn-664.txt https… May … This week …   664 <lis… It's … Spec… <tib…

The text column contains a tibble with a row for everytime anyone speaks untill the other takes over. The length

df_sn[664,9][[1]][[1]]
## # A tibble: 139 x 3
##    linenr text                                                     speaker
##     <int> <chr>                                                    <chr>  
##  1      1 This is Security Now! with Steve Gibson, Episode 664, r… LEO    
##  2      2 Yay, Leo, great to be with you again, as always.         STEVE  
##  3      3 Always a pleasure.                                       LEO    
##  4      4 Yeah, we scrambled here a little bit at the end because… STEVE  
##  5      5 [Crosstalk] half an hour ago.  No, but I also figured y… LEO    
##  6      6 "Well, yeah.  Two weeks ago the news leaked through Hei… STEVE  
##  7      7 They were presumably waiting to disclose until companie… LEO    
##  8      8 Yes.  And even so, I mean, this is - what we heard was … STEVE  
##  9      9 Right.                                                   LEO    
## 10     10 I can't square that with the news of two because all we… STEVE  
## # ... with 129 more rows

Steve talks a lot more then Leo, and we see that in the number of words per line of a single episode:

df_sn[664,9][[1]][[1]] %>% 
    mutate(
        nr_char = str_length(text),
        N_words = str_count(text, "\\w+")
        ) %>% 
    ggplot(aes(linenr, N_words, fill = speaker))+
    geom_col()+
    labs(
        title = "Number of words in every line in episode 664",
        subtitle = "Steve talks a lot more than Leo (not surprising)",
        x = "line number", y = "Number of words (log2 scale)",
        caption = "Transcripts Security Now! - 664"
    )+
    scale_y_continuous(trans = "log2")

Interestingly my scraper seems to not have detected who said the words on line 45. It was Steve.

It might be better to use an average over the episode

df_sn[664,9][[1]][[1]] %>% 
    mutate(
        N_words = str_count(text, "\\w+"),
        speaker = ifelse(is.na(speaker), "STEVE", speaker)
        ) %>% 
    ggplot(aes(speaker, N_words, fill = speaker))+
    geom_violin()+
    geom_jitter(height = 0,width = .2, alpha  = 2/3)+
    scale_y_continuous(trans = "log2")+
    labs(
        title = "Number of words in every line in episode 664",
        subtitle = "Steve has a few more long pieces",
        x = "", y = "Number of words (log2 scale)",
        caption = "Transcripts Security Now! - 664"
    )

  1. Not if we learn from Hans Rosling but that is a story for an other time↩

sentiment  security-now  tutorial  tidytext  tidyverse  dplyr  ggplot2  intermediate  widyr