Generate a data dictionnary and search for variables with look_for()

Joseph Larmarange

Showing a summary of a data frame

Default printing of tibbles

It is a common need to easily get a description of all variables in a data frame.

When a data frame is converted into a tibble (e.g. with dplyr::as_tibble()), it as a nice printing showing the first rows of the data frame as well as the type of column.

library(dplyr)
iris %>% as_tibble()
## # A tibble: 150 x 5
##    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
##           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
##  1          5.1         3.5          1.4         0.2 setosa 
##  2          4.9         3            1.4         0.2 setosa 
##  3          4.7         3.2          1.3         0.2 setosa 
##  4          4.6         3.1          1.5         0.2 setosa 
##  5          5           3.6          1.4         0.2 setosa 
##  6          5.4         3.9          1.7         0.4 setosa 
##  7          4.6         3.4          1.4         0.3 setosa 
##  8          5           3.4          1.5         0.2 setosa 
##  9          4.4         2.9          1.4         0.2 setosa 
## 10          4.9         3.1          1.5         0.1 setosa 
## # ... with 140 more rows

However, when you have too many variables, all of them cannot be printed and their are just listed.

data(fertility, package = "questionr")
women
## # A tibble: 2,000 x 17
##    id_woman id_household weight interview_date date_of_birth   age residency
##       <dbl>        <dbl>  <dbl> <date>         <date>        <dbl> <dbl+lbl>
##  1      391          381  1.80  2012-05-05     1997-03-07       15 2 [rural]
##  2     1643         1515  1.80  2012-01-23     1982-01-06       30 2 [rural]
##  3       85           85  1.80  2012-01-21     1979-01-01       33 2 [rural]
##  4      881          844  1.80  2012-01-06     1968-03-29       43 2 [rural]
##  5     1981         1797  1.80  2012-05-11     1986-05-25       25 2 [rural]
##  6     1072         1015  0.998 2012-02-20     1993-07-03       18 2 [rural]
##  7     1978         1794  0.998 2012-02-23     1967-01-28       45 2 [rural]
##  8     1607         1486  0.998 2012-02-20     1989-01-21       23 2 [rural]
##  9      738          711  0.192 2012-03-09     1962-07-24       49 2 [rural]
## 10     1656         1525  0.192 2012-03-15     1980-12-25       31 2 [rural]
## # ... with 1,990 more rows, and 10 more variables: region <dbl+lbl>,
## #   instruction <dbl+lbl>, employed <dbl+lbl>, matri <dbl+lbl>,
## #   religion <dbl+lbl>, newspaper <dbl+lbl>, radio <dbl+lbl>, tv <dbl+lbl>,
## #   ideal_nb_children <dbl+lbl>, test <dbl+lbl>

Note: in R console, value labels (if defined) are usually printed but they do not appear in a R markdown document like this vignette.

dplyr::glimpse()

The function dplyr::glimpse() allows you to have a quick look at all the variables in a data frame.

glimpse(iris)
## Rows: 150
## Columns: 5
## $ Sepal.Length <dbl> 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5.4, 4.~
## $ Sepal.Width  <dbl> 3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3.7, 3.~
## $ Petal.Length <dbl> 1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.~
## $ Petal.Width  <dbl> 0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.~
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, setosa, s~
glimpse(women)
## Rows: 2,000
## Columns: 17
## $ id_woman          <dbl> 391, 1643, 85, 881, 1981, 1072, 1978, 1607, 738, 165~
## $ id_household      <dbl> 381, 1515, 85, 844, 1797, 1015, 1794, 1486, 711, 152~
## $ weight            <dbl> 1.803150, 1.803150, 1.803150, 1.803150, 1.803150, 0.~
## $ interview_date    <date> 2012-05-05, 2012-01-23, 2012-01-21, 2012-01-06, 201~
## $ date_of_birth     <date> 1997-03-07, 1982-01-06, 1979-01-01, 1968-03-29, 198~
## $ age               <dbl> 15, 30, 33, 43, 25, 18, 45, 23, 49, 31, 26, 45, 25, ~
## $ residency         <dbl+lbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, ~
## $ region            <dbl+lbl> 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, ~
## $ instruction       <dbl+lbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 2, 1, 0, ~
## $ employed          <dbl+lbl> 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, ~
## $ matri             <dbl+lbl> 0, 2, 2, 2, 1, 0, 1, 1, 2, 5, 2, 3, 0, 2, 1, 2, ~
## $ religion          <dbl+lbl> 1, 3, 2, 3, 2, 2, 3, 1, 3, 3, 2, 3, 2, 2, 2, 2, ~
## $ newspaper         <dbl+lbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ~
## $ radio             <dbl+lbl> 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, ~
## $ tv                <dbl+lbl> 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, ~
## $ ideal_nb_children <dbl+lbl>  4,  4,  4,  4,  4,  5, 10,  5,  4,  5,  6, 10, ~
## $ test              <dbl+lbl> 0, 9, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, ~

It will show you the first values of each variable as well as the type of each variable. However, some important informations are not displayed:

labelled::look_for()

look_for() provided by the labelled package will print in the console a data dictionary of all variables, showing variable labels when available, the type of variable and a list of values corresponding to:

library(labelled)
look_for(iris)
##  pos variable     label           col_type values    
##  1   Sepal.Length —               dbl                
##  2   Sepal.Width  —               dbl                
##  3   Petal.Length Length of petal dbl                
##  4   Petal.Width  Width of Petal  dbl                
##  5   Species      —               fct      setosa    
##                                            versicolor
##                                            virginica
look_for(women)
##  pos variable          label                     col_type values             
##  1   id_woman          Woman Id                  dbl                         
##  2   id_household      Household Id              dbl                         
##  3   weight            Sample weight             dbl                         
##  4   interview_date    Interview date            date                        
##  5   date_of_birth     Date of birth             date                        
##  6   age               Age at last anniversary ~ dbl                         
##  7   residency         Urban / rural residency   dbl+lbl  [1] urban          
##                                                           [2] rural          
##  8   region            Region                    dbl+lbl  [1] North          
##                                                           [2] East           
##                                                           [3] South          
##                                                           [4] West           
##  9   instruction       Level of instruction      dbl+lbl  [0] none           
##                                                           [1] primary        
##                                                           [2] secondary      
##                                                           [3] higher         
##  10  employed          Employed?                 dbl+lbl  [0] no             
##                                                           [1] yes            
##                                                           [9] missing        
##  11  matri             Matrimonial status        dbl+lbl  [0] single         
##                                                           [1] married        
##                                                           [2] living together
##                                                           [3] windowed       
##                                                           [4] divorced       
##                                                           [5] separated      
##  12  religion          Religion                  dbl+lbl  [1] Muslim         
##                                                           [2] Christian      
##                                                           [3] Protestant     
##                                                           [4] no religion    
##                                                           [5] other          
##  13  newspaper         Read newspaper?           dbl+lbl  [0] no             
##                                                           [1] yes            
##  14  radio             Listen to radio?          dbl+lbl  [0] no             
##                                                           [1] yes            
##  15  tv                Watch TV?                 dbl+lbl  [0] no             
##                                                           [1] yes            
##  16  ideal_nb_children Ideal number of children  dbl+lbl  [96] don't know    
##                                                           [99] missing       
##  17  test              Ever tested for HIV?      dbl+lbl  [0] no             
##                                                           [1] yes            
##                                                           [9] missing

Note that lookfor() and generate_dictionary() are synonyms of look_for() and works exactly in the same way.

If there is not enough space to print full labels in the console, they will be truncated (truncation is indicated by a ~).

Searching variables by key

When a data frame has dozens or even hundreds of variables, it could become difficult to find a specific variable. In such case, you can provide an optional list of keywords, which can be simple character strings or regular expression, to search for specific variables.

# Look for a single keyword.
look_for(iris, "petal")
##  pos variable     label           col_type values
##  3   Petal.Length Length of petal dbl            
##  4   Petal.Width  Width of Petal  dbl
look_for(iris, "s")
##  pos variable     label col_type values    
##  1   Sepal.Length —     dbl                
##  2   Sepal.Width  —     dbl                
##  5   Species      —     fct      setosa    
##                                  versicolor
##                                  virginica
# Look for with a regular expression
look_for(iris, "petal|species")
##  pos variable     label           col_type values    
##  3   Petal.Length Length of petal dbl                
##  4   Petal.Width  Width of Petal  dbl                
##  5   Species      —               fct      setosa    
##                                            versicolor
##                                            virginica
look_for(iris, "s$")
##  pos variable label col_type values    
##  5   Species  —     fct      setosa    
##                              versicolor
##                              virginica
# Look for with several keywords
look_for(iris, "pet", "sp")
##  pos variable     label           col_type values    
##  3   Petal.Length Length of petal dbl                
##  4   Petal.Width  Width of Petal  dbl                
##  5   Species      —               fct      setosa    
##                                            versicolor
##                                            virginica
# Look_for will take variable labels into account
look_for(women, "read", "level")
##  pos variable    label                col_type values       
##  9   instruction Level of instruction dbl+lbl  [0] none     
##                                                [1] primary  
##                                                [2] secondary
##                                                [3] higher   
##  13  newspaper   Read newspaper?      dbl+lbl  [0] no       
##                                                [1] yes

By default, look_for() will look through both variable names and variables labels. Use labels = FALSE to look only through variable names.

look_for(women, "read")
##  pos variable  label           col_type values 
##  13  newspaper Read newspaper? dbl+lbl  [0] no 
##                                         [1] yes
look_for(women, "read", labels = FALSE)
## Nothing found. Sorry.

Similarly, the search is by default case insensitive. To make the search case sensitive, use ignore.case = FALSE.

look_for(iris, "sepal")
##  pos variable     label col_type values
##  1   Sepal.Length —     dbl            
##  2   Sepal.Width  —     dbl
look_for(iris, "sepal", ignore.case = FALSE)
## Nothing found. Sorry.

Level of details

If you just want to use the search feature of look_for() without computing the details of each variable, simply indicate details = "none" or details = FALSE.

look_for(women, "id", details = "none")
##  pos variable          label                   
##   1  id_woman          Woman Id                
##   2  id_household      Household Id            
##   7  residency         Urban / rural residency 
##  16  ideal_nb_children Ideal number of children

If you want more details (but can be time consuming for big data frames), indicate details = "full" or details = TRUE.

look_for(women, details = "full")
##  pos variable          label                  col_type values                
##  1   id_woman          Woman Id               dbl      range: 1 - 2000       
##  2   id_household      Household Id           dbl      range: 1 - 1814       
##  3   weight            Sample weight          dbl      range: 0.044629 - 4.3~
##  4   interview_date    Interview date         date     range: 2011-12-01 - 2~
##  5   date_of_birth     Date of birth          date     range: 1962-02-07 - 1~
##  6   age               Age at last anniversa~ dbl      range: 14 - 49        
##  7   residency         Urban / rural residen~ dbl+lbl  [1] urban             
##                                                        [2] rural             
##  8   region            Region                 dbl+lbl  [1] North             
##                                                        [2] East              
##                                                        [3] South             
##                                                        [4] West              
##  9   instruction       Level of instruction   dbl+lbl  [0] none              
##                                                        [1] primary           
##                                                        [2] secondary         
##                                                        [3] higher            
##  10  employed          Employed?              dbl+lbl  [0] no                
##                                                        [1] yes               
##                                                        [9] missing           
##  11  matri             Matrimonial status     dbl+lbl  [0] single            
##                                                        [1] married           
##                                                        [2] living together   
##                                                        [3] windowed          
##                                                        [4] divorced          
##                                                        [5] separated         
##  12  religion          Religion               dbl+lbl  [1] Muslim            
##                                                        [2] Christian         
##                                                        [3] Protestant        
##                                                        [4] no religion       
##                                                        [5] other             
##  13  newspaper         Read newspaper?        dbl+lbl  [0] no                
##                                                        [1] yes               
##  14  radio             Listen to radio?       dbl+lbl  [0] no                
##                                                        [1] yes               
##  15  tv                Watch TV?              dbl+lbl  [0] no                
##                                                        [1] yes               
##  16  ideal_nb_children Ideal number of child~ dbl+lbl  [96] don't know       
##                                                        [99] missing          
##  17  test              Ever tested for HIV?   dbl+lbl  [0] no                
##                                                        [1] yes               
##                                                        [9] missing
look_for(women, details = "full") %>%
  dplyr::glimpse()
## Rows: 17
## Columns: 13
## $ pos           <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
## $ variable      <chr> "id_woman", "id_household", "weight", "interview_date", ~
## $ label         <chr> "Woman Id", "Household Id", "Sample weight", "Interview ~
## $ col_type      <chr> "dbl", "dbl", "dbl", "date", "date", "dbl", "dbl+lbl", "~
## $ levels        <named list> [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, N~
## $ value_labels  <named list> [NULL, NULL, NULL, NULL, NULL, NULL, <1, 2>, <1, ~
## $ class         <named list> ["numeric", "numeric", "numeric", "Date", "Date",~
## $ type          <chr> "double", "double", "double", "double", "double",~
## $ na_values     <named list> [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 9~
## $ na_range      <named list> [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, ~
## $ unique_values <int> 2000, 1814, 351, 165, 1740, 36, 2, 4, 4, 3, 6, 6,~
## $ n_na          <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 0, 4, 0, 0, 0, 0, 29
## $ range         <named list> [<1, 2000>, <1, 1814>, <0.044629, 4.396831>, <2011-12-01~

Advanced usages of look_for()

look_for() returns a detailed tibble which is summarized before printing. To deactivate default printing and see full results, simply use dplyr::as_tibble(), dplyr::glimpse() or even utils::View().

look_for(women) %>% View()
look_for(women) %>% as_tibble()
## # A tibble: 17 x 6
##      pos variable       label                    col_type levels    value_labels
##    <int> <chr>          <chr>                    <chr>    <named l> <named list>
##  1     1 id_woman       Woman Id                 dbl      <NULL>    <NULL>      
##  2     2 id_household   Household Id             dbl      <NULL>    <NULL>      
##  3     3 weight         Sample weight            dbl      <NULL>    <NULL>      
##  4     4 interview_date Interview date           date     <NULL>    <NULL>      
##  5     5 date_of_birth  Date of birth            date     <NULL>    <NULL>      
##  6     6 age            Age at last anniversary~ dbl      <NULL>    <NULL>      
##  7     7 residency      Urban / rural residency  dbl+lbl  <NULL>    <dbl [2]>   
##  8     8 region         Region                   dbl+lbl  <NULL>    <dbl [4]>   
##  9     9 instruction    Level of instruction     dbl+lbl  <NULL>    <dbl [4]>   
## 10    10 employed       Employed?                dbl+lbl  <NULL>    <dbl [3]>   
## 11    11 matri          Matrimonial status       dbl+lbl  <NULL>    <dbl [6]>   
## 12    12 religion       Religion                 dbl+lbl  <NULL>    <dbl [5]>   
## 13    13 newspaper      Read newspaper?          dbl+lbl  <NULL>    <dbl [2]>   
## 14    14 radio          Listen to radio?         dbl+lbl  <NULL>    <dbl [2]>   
## 15    15 tv             Watch TV?                dbl+lbl  <NULL>    <dbl [2]>   
## 16    16 ideal_nb_chil~ Ideal number of children dbl+lbl  <NULL>    <dbl [2]>   
## 17    17 test           Ever tested for HIV?     dbl+lbl  <NULL>    <dbl [3]>
glimpse(look_for(women))
## Rows: 17
## Columns: 6
## $ pos          <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
## $ variable     <chr> "id_woman", "id_household", "weight", "interview_date", "~
## $ label        <chr> "Woman Id", "Household Id", "Sample weight", "Interview d~
## $ col_type     <chr> "dbl", "dbl", "dbl", "date", "date", "dbl", "dbl+lbl", "d~
## $ levels       <named list> [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NU~
## $ value_labels <named list> [NULL, NULL, NULL, NULL, NULL, NULL, <1, 2>, <1, 2~

The tibble returned by look_for() could be easily manipulated for advanced programming.

When a column has several values for one variable (e.g. levels or value_labels), results as stored with nested named list. You can convert named lists into simpler character vectors, you can use convert_list_columns_to_character().

look_for(women) %>% convert_list_columns_to_character()
## # A tibble: 17 x 6
##      pos variable    label         col_type levels value_labels                 
##    <int> <chr>       <chr>         <chr>    <chr>  <chr>                        
##  1     1 id_woman    Woman Id      dbl      ""     ""                           
##  2     2 id_househo~ Household Id  dbl      ""     ""                           
##  3     3 weight      Sample weight dbl      ""     ""                           
##  4     4 interview_~ Interview da~ date     ""     ""                           
##  5     5 date_of_bi~ Date of birth date     ""     ""                           
##  6     6 age         Age at last ~ dbl      ""     ""                           
##  7     7 residency   Urban / rura~ dbl+lbl  ""     "[1] urban; [2] rural"       
##  8     8 region      Region        dbl+lbl  ""     "[1] North; [2] East; [3] So~
##  9     9 instruction Level of ins~ dbl+lbl  ""     "[0] none; [1] primary; [2] ~
## 10    10 employed    Employed?     dbl+lbl  ""     "[0] no; [1] yes; [9] missin~
## 11    11 matri       Matrimonial ~ dbl+lbl  ""     "[0] single; [1] married; [2~
## 12    12 religion    Religion      dbl+lbl  ""     "[1] Muslim; [2] Christian; ~
## 13    13 newspaper   Read newspap~ dbl+lbl  ""     "[0] no; [1] yes"            
## 14    14 radio       Listen to ra~ dbl+lbl  ""     "[0] no; [1] yes"            
## 15    15 tv          Watch TV?     dbl+lbl  ""     "[0] no; [1] yes"            
## 16    16 ideal_nb_c~ Ideal number~ dbl+lbl  ""     "[96] don't know; [99] missi~
## 17    17 test        Ever tested ~ dbl+lbl  ""     "[0] no; [1] yes; [9] missin~

Alternatively, you can use lookfor_to_long_format() to transform results into a long format with one row per factor level and per value label.

look_for(women) %>% lookfor_to_long_format()
## # A tibble: 41 x 6
##      pos variable      label                        col_type levels value_labels
##    <int> <chr>         <chr>                        <chr>    <chr>  <chr>       
##  1     1 id_woman      Woman Id                     dbl      <NA>   <NA>        
##  2     2 id_household  Household Id                 dbl      <NA>   <NA>        
##  3     3 weight        Sample weight                dbl      <NA>   <NA>        
##  4     4 interview_da~ Interview date               date     <NA>   <NA>        
##  5     5 date_of_birth Date of birth                date     <NA>   <NA>        
##  6     6 age           Age at last anniversary (in~ dbl      <NA>   <NA>        
##  7     7 residency     Urban / rural residency      dbl+lbl  <NA>   [1] urban   
##  8     7 residency     Urban / rural residency      dbl+lbl  <NA>   [2] rural   
##  9     8 region        Region                       dbl+lbl  <NA>   [1] North   
## 10     8 region        Region                       dbl+lbl  <NA>   [2] East    
## # ... with 31 more rows

Both can be combined:

look_for(women) %>%
  lookfor_to_long_format() %>%
  convert_list_columns_to_character()
## # A tibble: 41 x 6
##      pos variable      label                        col_type levels value_labels
##    <int> <chr>         <chr>                        <chr>    <chr>  <chr>       
##  1     1 id_woman      Woman Id                     dbl      <NA>   <NA>        
##  2     2 id_household  Household Id                 dbl      <NA>   <NA>        
##  3     3 weight        Sample weight                dbl      <NA>   <NA>        
##  4     4 interview_da~ Interview date               date     <NA>   <NA>        
##  5     5 date_of_birth Date of birth                date     <NA>   <NA>        
##  6     6 age           Age at last anniversary (in~ dbl      <NA>   <NA>        
##  7     7 residency     Urban / rural residency      dbl+lbl  <NA>   [1] urban   
##  8     7 residency     Urban / rural residency      dbl+lbl  <NA>   [2] rural   
##  9     8 region        Region                       dbl+lbl  <NA>   [1] North   
## 10     8 region        Region                       dbl+lbl  <NA>   [2] East    
## # ... with 31 more rows