Introduction to getLattes

The Lattes platform has been hosting curricula of Brazilian researchers since the late 1990s, containing more than 5 million curricula. The data from the Lattes curricula can be downloaded to XML format, the complexity of this reading process motivated the development of the getLattes package, which imports the information from the XML files to a list in the R software and then tabulates the Lattes data to a data.frame.

The main information contained in XML files, and imported via getLattes, are:

From the functionalities presented in this package, the main challenge to work with the Lattes curriculum data is now to download the data, as there are Captchas. To download a lot of curricula I suggest the use of Captchas Negated by Python reQuests - CNPQ. The second barrier to be overcome is the management and processing of a large volume of data, the whole Lattes platform in XML files totals over 200 GB. In this tutorial we will focus on the getLattes package features, being the reader responsible for download and manage the files.

Follow an example of how to search and download data from the Lattes website.

Installation

To install the released version of getLattes from github.


# install and load devtools from CRAN
# install.packages("devtools")
library(devtools)

# install and load getLattes
devtools::install_github("roneyfraga/getLattes")

Load getLattes.

library(getLattes)

# support packages
library(xml2)
library(dplyr)
library(tibble)
library(purrr)

Single curriculum

Import

Using the get* functions to import data from a single curriculum is straightforward. The curriculum need to be imported into R by the read_xml() function from the xml2 package.

curriculo <- xml2::read_xml('../inst/extdata/4984859173592703.zip')

get functions

getDadosGerais(curriculo)
getArtigosPublicados(curriculo)
getAreasAtuacao(curriculo)
getArtigosPublicados(curriculo)
getAtuacoesProfissionais(curriculo)
getBancasDoutorado(curriculo)
getBancasGraduacao(curriculo)
getBancasMestrado(curriculo)
getCapitulosLivros(curriculo)
getDadosGerais(curriculo)
getEnderecoProfissional(curriculo)
getEventosCongressos(curriculo)
getFormacaoDoutorado(curriculo)
getFormacaoMestrado(curriculo)
getFormacaoGraduacao(curriculo)
getIdiomas(curriculo)
getLinhaPesquisa(curriculo)
getLivrosPublicados(curriculo)
getOrganizacaoEventos(curriculo)
getOrientacoesDoutorado(curriculo)
getOrientacoesMestrado(curriculo)
getOrientacoesPosDoutorado(curriculo)
getOutrasProducoesTecnicas(curriculo)
getParticipacaoProjeto(curriculo)
getProducaoTecnica(curriculo)
getId(curriculo)

Several curricula

Import

To import data from two or more curricula it is easier to use list.files(), a native R function, or dir_ls() from fs package. As xml2::read_xml() allow to read a xml curriculum inside a zip files, we can insert both options in pattern argument.

files <- list.files(path = '../inst/extdata/', pattern = '*.xml|*.zip', full.names = T)

Import the listed curricula to R memory as xml2::read_xml object.

curriculos <- lapply(files, read_xml)

The lapply() function is a well-known and widely used alternative in the R world. However, it does not natively handle errors, which makes the map function from the purrr package an excellent alternative.

Adding an extra layer of complexity, I will use pipe |>. Programming using the pipe operator |> allows faster coding and clearer syntax.

curriculos <- 
    purrr::map(files, safely(read_xml)) |> 
    purrr::map(pluck, 'result')

get functions

To read data from only one curriculum any function get can be executed singly, but to import data from two or more curricula is easier to use get* functions with lapply() or map().

dados_gerais <- 
    purrr::map(curriculos, safely(getDadosGerais)) |>
    purrr::map(pluck, 'result') 

dados_gerais
#> [[1]]
#> # A tibble: 1 x 12
#>   nome_completo  nome_em_citacoes_… nacionalidade pais_de_nascime… uf_nascimento
#>   <chr>          <chr>              <chr>         <chr>            <chr>        
#> 1 Antonio Marci… BUAINAIN, Antonio… B             Brasil           MS           
#> # … with 7 more variables: cidade_nascimento <chr>,
#> #   permissao_de_divulgacao <chr>, data_falecimento <chr>,
#> #   sigla_pais_nacionalidade <chr>, pais_de_nacionalidade <chr>,
#> #   orcid_id <chr>, id <chr>
#> 
#> [[2]]
#> # A tibble: 1 x 12
#>   nome_completo  nome_em_citacoes_… nacionalidade pais_de_nascime… uf_nascimento
#>   <chr>          <chr>              <chr>         <chr>            <chr>        
#> 1 Jose Maria Fe… SILVEIRA, José Ma… B             Brasil           SP           
#> # … with 7 more variables: cidade_nascimento <chr>,
#> #   permissao_de_divulgacao <chr>, data_falecimento <chr>,
#> #   sigla_pais_nacionalidade <chr>, pais_de_nacionalidade <chr>,
#> #   orcid_id <chr>, id <chr>

Import general data from 2 curricula. The output is a list of data frames, converted by a unique data frame with bind_rows().


dados_gerais <- 
    purrr::map(curriculos, safely(getDadosGerais)) |>
    purrr::map(pluck, 'result') |>
    dplyr::bind_rows() 

glimpse(dados_gerais)
#> Rows: 2
#> Columns: 12
#> $ nome_completo                   <chr> "Antonio Marcio Buainain", "Jose Maria…
#> $ nome_em_citacoes_bibliograficas <chr> "BUAINAIN, Antonio Marcio;Buainain, An…
#> $ nacionalidade                   <chr> "B", "B"
#> $ pais_de_nascimento              <chr> "Brasil", "Brasil"
#> $ uf_nascimento                   <chr> "MS", "SP"
#> $ cidade_nascimento               <chr> "Campo Grande", "São Paulo"
#> $ permissao_de_divulgacao         <chr> "NAO", "NAO"
#> $ data_falecimento                <chr> "", ""
#> $ sigla_pais_nacionalidade        <chr> "BRA", "BRA"
#> $ pais_de_nacionalidade           <chr> "Brasil", "Brasil"
#> $ orcid_id                        <chr> "https://orcid.org/0000-0002-1779-5589…
#> $ id                              <chr> "3051627641386529", "4984859173592703"

It is worth remembering that all variable names obtained by get* functions are the transcription of the field names in the XML file, the - being replaced with _ and the capital letters replaced with lower case letters.

Publications

artigos_publicados <- 
    purrr::map(curriculos, safely(getArtigosPublicados)) |>
    purrr::map(pluck, 'result') |>
    dplyr::bind_rows() 

artigos_publicados |>
    dplyr::arrange(desc(ano_do_artigo)) |>
    dplyr::select(titulo_do_artigo, ano_do_artigo, titulo_do_periodico_ou_revista) 
#> # A tibble: 192 x 3
#>    titulo_do_artigo                      ano_do_artigo titulo_do_periodico_ou_r…
#>    <chr>                                 <chr>         <chr>                    
#>  1 An Analysis of Collaboration Network… 2021          REVISTA DE ADMINISTRAÇÃO…
#>  2 Patent network analysis in agricultu… 2021          ECONOMICS OF INNOVATION …
#>  3 GENETICALLY MODIFIED CORN ADOPTION I… 2021          REVISTA DE ECONOMIA E AG…
#>  4 Avaliação do Programa Nacional de Pr… 2020          Desenvolvimento em Debat…
#>  5 Agro brasileiro em evolução: complex… 2020          Revista de Política Agrí…
#>  6 Classe média rural?                   2020          Revista de Politica Agrí…
#>  7 International trade in GMOs: have ma… 2020          Revista de economia e so…
#>  8 Governance and financial efficiency … 2020          RAUSP Management Journal 
#>  9 The Role of Participation in the Res… 2020          Sustainability           
#> 10 The impact of sugarcane expansion in… 2020          JOURNAL OF RURAL STUDIES 
#> # … with 182 more rows

livros_publicados <- 
    purrr::map(curriculos, safely(getLivrosPublicados)) |>
    purrr::map(pluck, 'result') |>
    dplyr::bind_rows() 

capitulos_livros <- 
    purrr::map(curriculos, safely(getCapitulosLivros)) |>
    purrr::map(pluck, 'result') |>
    dplyr::bind_rows()

Grouping data

To group the data key variable is id, which is a unique 16 digit code.


artigos_publicados2 <- 
    dplyr::group_by(artigos_publicados, id) |>
    dplyr::tally(name = 'artigos') 

artigos_publicados2
#> # A tibble: 2 x 2
#>   id               artigos
#>   <chr>              <int>
#> 1 3051627641386529     101
#> 2 4984859173592703      91

livros_publicados2 <- 
    dplyr::group_by(livros_publicados, id) |>
    dplyr::tally(name = 'livros') 

livros_publicados2
#> # A tibble: 2 x 2
#>   id               livros
#>   <chr>             <int>
#> 1 3051627641386529     45
#> 2 4984859173592703      8

capitulos_livros2 <- 
    dplyr::group_by(capitulos_livros, id) |>
    dplyr::tally(name = 'capitulos') 

capitulos_livros2
#> # A tibble: 2 x 2
#>   id               capitulos
#>   <chr>                <int>
#> 1 3051627641386529        81
#> 2 4984859173592703        48

Merge data

to join the data from different tables the recommended variable is id, which is a unique 16 digit code.


artigos_publicados2 |>
    dplyr::left_join(livros_publicados2) |>
    dplyr::left_join(capitulos_livros2)
#> # A tibble: 2 x 4
#>   id               artigos livros capitulos
#>   <chr>              <int>  <int>     <int>
#> 1 3051627641386529     101     45        81
#> 2 4984859173592703      91      8        48

Add information from a different tables.


artigos_publicados2 |>
    dplyr::left_join(livros_publicados2) |>
    dplyr::left_join(capitulos_livros2) |>
    dplyr::left_join(dados_gerais |> dplyr::select(id, nome_completo)) |>
    dplyr::select(nome_completo, artigos, livros, capitulos) 
#> # A tibble: 2 x 4
#>   nome_completo                          artigos livros capitulos
#>   <chr>                                    <int>  <int>     <int>
#> 1 Antonio Marcio Buainain                    101     45        81
#> 2 Jose Maria Ferreira Jardim da Silveira      91      8        48