Search plans and quota-aware retrieval

library(scopusflow)

The Elsevier Scopus Search API bounds a retrieval in three ways. A weekly quota limits how many requests you may make, a short-term rate limit caps how fast you may make them, and the ordinary offset paging stops at the first 5000 records of a query. This article shows how scopusflow works within those bounds to make a large retrieval reproducible, efficient and resumable. The steps that contact the API need a key and are not run here. Everything else runs offline.

A query, built safely

Most queries combine a few terms under a field tag. scopus_query() assembles them without the bracket and tag mistakes that arise when fragments are pasted together by hand.

q <- scopus_query("language learning", "effect size", .field = "TITLE-ABS-KEY")
q
[1] "TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size)"

The recognised field tags, and what each one searches, are listed by scopus_field_tags().

scopus_field_tags()
tag searches
TITLE Words in the document title
TITLE-ABS-KEY Title, abstract and keywords
TITLE-ABS-KEY-AUTH Title, abstract, keywords and author names
ABS Abstract text
KEY Indexed and author keywords
AUTH Author names
AUTHKEY Author-supplied keywords
AFFIL Affiliation, any part
AFFILORG Affiliation organisation name
SRCTITLE Source (publication) title
DOI Digital Object Identifier
ALL All available fields

Describing the search as a plan

A plan records exactly what will be fetched, so it can be saved, reviewed and re-run. Partitioning by year is the recommended way to stay under the 5000-record ceiling of offset paging, since each year becomes its own cell, provided that no single year matches more than 5000 records.

plan <- scopus_plan(q, years = 2010:2020, partition = "year")
plan
cell query date year view page_size
1 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2010 2010 STANDARD 200
2 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2011 2011 STANDARD 200
3 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2012 2012 STANDARD 200
4 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2013 2013 STANDARD 200
5 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2014 2014 STANDARD 200
6 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2015 2015 STANDARD 200
7 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2016 2016 STANDARD 200
8 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2017 2017 STANDARD 200
9 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2018 2018 STANDARD 200
10 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2019 2019 STANDARD 200
11 TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size) 2020 2020 STANDARD 200

Each cell carries the query, the year, the view and the page size.

Why page size is a quota decision

Scopus charges quota by the request. A page may hold up to 200 records under the STANDARD view, or 25 under COMPLETE, so retrieving a thousand records in pages of 200 costs five requests where pages of 25 would cost forty. For that reason, page_size defaults to the largest page the view allows, as rscopus also does. At any page size, every request is counted and the 5000-record ceiling holds.

scopus_plan(q, view = "STANDARD")$page_size[1]
[1] 200
scopus_plan(q, view = "COMPLETE")$page_size[1]
[1] 25

Sizing before spending

Counting is cheap and does not download records, so it is worth doing first. A workflow can read the parsed quota attached to the count and decide whether enough quota remains to proceed.

n <- scopus_count(q, years = 2010:2020)
n
attr(n, "quota")

The quota is parsed from the response headers by scopus_quota(). To show its shape without a network call, apply the function to a constructed response.

resp <- httr2::response(
  status_code = 200,
  headers = list(
    `X-RateLimit-Limit`     = "20000",
    `X-RateLimit-Remaining` = "19987",
    `X-RateLimit-Reset`     = "1700000000"
  )
)
scopus_quota(resp)
$limit
[1] 20000

$remaining
[1] 19987

$reset
[1] "2023-11-14 22:13:20 UTC"

$status
[1] NA

$retry_after
[1] NA

Fetching, with caching and resume

scopus_fetch_plan() runs each cell in turn. Given a cache directory, it writes each cell to disk as it completes, so a run interrupted halfway, or stopped by the quota, resumes from where it left off and never spends quota twice on the same cell. The plan below keeps its cells in a directory of its own inside the managed cache, the directory returned by scopus_cache_dir().

records <- scopus_fetch_plan(
  plan,
  cache_dir = file.path(scopus_cache_dir(), "language-learning-effect-size"),
  resume = TRUE
)
records

A cache directory belongs to one plan. Cells are checkpointed by their position in the plan and their year, so a second plan pointed at the same directory could otherwise be paired with the first plan’s checkpoints. Before loading a checkpoint, scopus_fetch_plan() checks that the query, year, view, page size, paging mode and record cap it was written under all match the cell being run, and refetches with a warning when they do not. Give each plan its own directory all the same.

A checkpoint holding more records than the current max_results asks for is served trimmed to that cap, with the fuller set left on disk for a later, wider request. A checkpoint that stopped at a smaller cap than is being asked for now is a mismatch, refetched with a warning. One that ran to the end of its result set is not a mismatch, however few records it holds, because there is nothing further to fetch. Likewise, a checkpoint fetched with fields = "extended" serves a core harvest with the extra columns dropped, while a core checkpoint is refetched with a warning for an extended harvest.

To force a fresh retrieval, empty the managed cache with scopus_cache_clear(), which removes every plan’s directory inside it. Both functions are shown below but not run, so building the article leaves any existing cache intact.

scopus_cache_dir()    # where completed cells are written
scopus_cache_clear()  # remove them, so the next run re-fetches from scratch

The result of scopus_fetch_plan() is a scopus_records tibble, the same shape returned by scopus_fetch() for a single query. Without a key, the bundled example_records stands in for it. It holds 138 published journal articles in that same schema and is included because Scopus records may not be redistributed.

head(example_records)
entry_number scopus_id doi title authors year date publication citations query
1 NA 10.15541/jim20140527 Enhanced Capacitive Properties of All-solid-state Symmetric Graphene Supercapacitors by Incorporating Nitrogen-doping and SnO2 Nanoparticles Jianhua Yu 2015 2015-01-01 Journal of Inorganic Materials 1 graphene supercapacitor
2 NA NA Fabrication and Characterization of a Vertically-Oriented Graphene Supercapacitor Patrick R Rice 2015 2015-01-01 DigitalCommons - CalPoly (California State Polytechnic University) 0 graphene supercapacitor
3 NA 10.1021/am509065d Flexible and Stackable Laser-Induced Graphene Supercapacitors Zhiwei Peng 2015 2015-01-13 ACS Applied Materials & Interfaces 469 graphene supercapacitor
4 NA 10.1016/j.electacta.2015.02.019 Heavily nitrogen doped, graphene supercapacitor from silk cocoon Vikrant Sahu 2015 2015-02-04 Electrochimica Acta 195 graphene supercapacitor
5 NA 10.1002/smll.201403383 Graphene-Based Integrated Photovoltaic Energy Harvesting/Storage Device Chih-Tao Chien 2015 2015-02-19 Small 108 graphene supercapacitor
6 NA 10.1016/j.jpowsour.2015.03.015 Nanoporous graphene materials by low-temperature vacuum-assisted thermal process for electrochemical energy storage Hao Yang 2015 2015-03-05 Journal of Power Sources 47 graphene supercapacitor

Watching progress

Per-cell progress is silent by default. For a harvest that spans many years, it is worth setting verbose = TRUE, which reports a line as each cell is fetched or loaded from cache.

records <- scopus_fetch_plan(
  plan,
  cache_dir = file.path(scopus_cache_dir(), "language-learning-effect-size"),
  verbose = TRUE
)

Combining separate retrievals

Results gathered in separate runs are combined with scopus_combine(), which renumbers the records and can drop duplicates by Scopus identifier or DOI. This is preferable to rbind(), which would leave duplicate entry numbers. Here, a baseline retrieval that stopped at 2023 is merged with a later one covering the whole period.

baseline <- example_records[example_records$year <= 2023, ]
combined <- scopus_combine(baseline, example_records, dedupe = TRUE)
nrow(combined)
[1] 149

The 138 distinct articles come back as 149 rows. De-duplication needs an identifier to match on, and these records carry no Scopus identifier, never having come from Scopus. Matching therefore falls back to the DOI, and the eleven articles without a DOI cannot be matched to their own copies. A live Scopus harvest carries an identifier on every record, so the same call would return 138.

Writing the search up

A harvest is rarely the end of the work. A systematic review has to report the search itself, in enough detail that a reader can repeat it, and the reporting standard for that is PRISMA-S (Rethlefsen et al., 2021). scopus_search_report() assembles the search record from what the plan and the harvest already carry.

A plan on its own can be reported before it is run, which is useful when a protocol has to be registered in advance. The plan below describes a Scopus search for the phrase the bundled corpus was drawn on. The corpus itself came from OpenAlex’s exact-phrase search for ‘graphene supercapacitor’ in titles and abstracts, limited to journal articles, on 22 July 2026. A Scopus run of this plan would return a different set of records. The query puts the two words in quotation marks so that Scopus searches them as a phrase. Without the quotation marks, Scopus would join the two words with AND, and in OpenAlex the unquoted words match about a hundred times as many records.

graphene <- scopus_plan('"graphene supercapacitor"', years = 2015:2024,
                        field = "TITLE-ABS-KEY", partition = "year")
report <- scopus_search_report(graphene)
report
Search strategy record (PRISMA-S)

Database: Scopus, on the Elsevier Scopus Search API
Search expression: TITLE-ABS-KEY("graphene supercapacitor")
Field tag: TITLE-ABS-KEY
Years: 2015 to 2024
Partition: one cell per year, 10 cells
View: STANDARD
Page size: 200 records per request
Paging: unrecorded
Date searched: unrecorded, this plan has not been run
Software: unrecorded
Records retrieved: none, this plan has not been run
Records reported as matching: unrecorded, this plan has not been run
Completeness: unrecorded, this plan has not been run
Duplicates removed: unrecorded, this plan has not been run
Records carrying a DOI: unrecorded, this plan has not been run

Cells
  1 (2015): retrieved records unrecorded
  2 (2016): retrieved records unrecorded
  3 (2017): retrieved records unrecorded
  4 (2018): retrieved records unrecorded
  5 (2019): retrieved records unrecorded
  6 (2020): retrieved records unrecorded
  7 (2021): retrieved records unrecorded
  8 (2022): retrieved records unrecorded
  9 (2023): retrieved records unrecorded
  10 (2024): retrieved records unrecorded

PRISMA 2020 identification
  Records identified from Scopus: unrecorded, this plan has not been run
  Duplicate records removed before screening: unrecorded, this plan has not been run

PRISMA-S items this record supplies
  1 Database name. Scopus, searched on the Elsevier Scopus Search API.
  2 Multi-database searching. Scopus is searched on its own, not as one of several databases searched together on one platform.
  8 Full search strategies. The search expression, field tag and year limit of every cell, as the plan sends them.
  9 Limits and restrictions. Publication years 2015 to 2024. scopusflow applies no document type, language or subject area limit of its own.

PRISMA-S items only you can supply
  3 Study registries. Any trial or study registry searched.
  4 Online resources and browsing. Any web site, table of contents or other source searched or browsed by hand.
  5 Citation searching. Any backward or forward citation searching.
  6 Contacts. Any authors or organisations contacted for studies.
  7 Other methods. Any further method used to identify records.
  10 Search filters. Any published or validated search filter used, and where it came from. scopusflow applies none of its own.
  11 Prior work. Any earlier review or strategy this search was adapted from.
  12 Updates. Whether the search was re-run or updated, and when.
  13 Dates of searches. The date of each search, which this plan has not been run to produce.
  14 Peer review. Whether the strategy was peer reviewed, and by whom.
  15 Total records. The number of records identified, which this plan has not been run to produce.
  16 Deduplication. How duplicate records were removed, which this plan has not been run to produce.

The reporting standard is PRISMA-S (Rethlefsen et al., 2021, Systematic Reviews, 10, 39, https://doi.org/10.1186/s13643-020-01542-z), with the identification counts of the PRISMA 2020 flow diagram.

The methods paragraph is format(x, style = "paragraph"); pass file = to write the whole record as Markdown.

Much of the record says ‘unrecorded’. Nothing has been retrieved yet, so there is nothing to state, and the record says so in words because a blank would be read as a zero. Throughout, the record states only what the objects hold. It never substitutes the current time for a retrieval that did not record one, never gives a completeness figure for a harvest whose reported total is unknown, and never counts duplicates unless a merge recorded removing them.

After a harvest, the record fills in. scopus_fetch_plan() attaches the plan, the retrieval time, the version, the paging mode and the per-cell accounting, and you never set any of them yourself. Here, the bundled corpus stands in for a harvest, since Scopus records may not be redistributed. Those attributes are therefore written out below to show what each one contributes. Their values are illustrative, so the record they produce describes no search that was run. The retrieval time and the version are set by hand. Each cell’s reported total is the number of bundled records in its year, a count from the OpenAlex search that the Scopus API never gave.

# Illustrative values, set by hand. No Scopus search was run.
records <- example_records
attr(records, "plan") <- graphene
attr(records, "retrieved_at") <- as.POSIXct("2026-07-22 09:15:00", tz = "UTC")
attr(records, "scopusflow_version") <- "0.3.0"
attr(records, "paging") <- "offset"
per_year <- as.integer(table(example_records$year))
attr(records, "cell_totals") <- tibble::tibble(
  cell = 1:10, date = as.character(2015:2024),
  n_records = per_year, reported_total = as.numeric(per_year)
)

report <- scopus_search_report(records)
report
Search strategy record (PRISMA-S)

Database: Scopus, on the Elsevier Scopus Search API
Search expression: TITLE-ABS-KEY("graphene supercapacitor")
Field tag: TITLE-ABS-KEY
Years: 2015 to 2024
Partition: one cell per year, 10 cells
View: STANDARD
Page size: 200 records per request
Paging: offset
Date searched: 2026-07-22 09:15:00 UTC
Software: scopusflow 0.3.0
Records retrieved: 138
Records reported as matching: 138
Completeness: every record the API reported as matching was retrieved
Duplicates removed: unrecorded, no de-duplication step was recorded for this set
Records carrying a DOI: 127 of 138

Cells
  1 (2015): 15 retrieved, 15 reported, complete
  2 (2016): 9 retrieved, 9 reported, complete
  3 (2017): 10 retrieved, 10 reported, complete
  4 (2018): 15 retrieved, 15 reported, complete
  5 (2019): 19 retrieved, 19 reported, complete
  6 (2020): 13 retrieved, 13 reported, complete
  7 (2021): 13 retrieved, 13 reported, complete
  8 (2022): 15 retrieved, 15 reported, complete
  9 (2023): 15 retrieved, 15 reported, complete
  10 (2024): 14 retrieved, 14 reported, complete

PRISMA 2020 identification
  Records identified from Scopus: 138
  Duplicate records removed before screening: unrecorded, no de-duplication step was recorded for this set

PRISMA-S items this record supplies
  1 Database name. Scopus, searched on the Elsevier Scopus Search API.
  2 Multi-database searching. Scopus was searched on its own, not as one of several databases searched together on one platform.
  8 Full search strategies. The search expression, field tag and year limit of every cell, as the plan sends them.
  9 Limits and restrictions. Publication years 2015 to 2024. scopusflow applies no document type, language or subject area limit of its own.
  13 Dates of searches. 2026-07-22 09:15:00 UTC.
  15 Total records. 138 records retrieved from Scopus, of 138 the API reported as matching.

PRISMA-S items only you can supply
  3 Study registries. Any trial or study registry searched.
  4 Online resources and browsing. Any web site, table of contents or other source searched or browsed by hand.
  5 Citation searching. Any backward or forward citation searching.
  6 Contacts. Any authors or organisations contacted for studies.
  7 Other methods. Any further method used to identify records.
  10 Search filters. Any published or validated search filter used, and where it came from. scopusflow applies none of its own.
  11 Prior work. Any earlier review or strategy this search was adapted from.
  12 Updates. Whether the search was re-run or updated, and when.
  14 Peer review. Whether the strategy was peer reviewed, and by whom.
  16 Deduplication. How duplicate records were removed, which this set does not record.

The reporting standard is PRISMA-S (Rethlefsen et al., 2021, Systematic Reviews, 10, 39, https://doi.org/10.1186/s13643-020-01542-z), with the identification counts of the PRISMA 2020 flow diagram.

The methods paragraph is format(x, style = "paragraph"); pass file = to write the whole record as Markdown.

The completeness lines show each cell against the number of records the API reported for it, so a cell that came back short stays visible where a total would have hidden it. The overall figure is given only because every cell reported one. Drop any of those attributes and the corresponding line says so instead.

A harvest run with max_results records its cap. The search record marks each cell that stopped at the cap, and the completeness line says how much of the shortfall the cap explains. The record’s entry for PRISMA-S item 9 names the cap, and the snippet that rebuilds the plan passes it.

The methods paragraph is the same record as prose, ready to paste into a manuscript and edit.

cat(format(report, style = "paragraph"))
The literature was searched in Scopus, on the Elsevier Scopus Search API, on 22 July 2026. The search expression was TITLE-ABS-KEY("graphene supercapacitor"), limited to publication years 2015 to 2024. It was partitioned into 10 cells, one per year, each retrieved through the STANDARD view in pages of 200 records under offset paging. The search retrieved 138 records, matching the 138 the API reported, so every reported record was retrieved. Of the records retrieved, 127 carry a DOI. No de-duplication step was recorded for this set. The search was run with scopusflow 0.3.0. The PRISMA-S items this record cannot supply, among them peer review of the strategy, grey literature and any other database searched, remain yours to report.

Supplying a file writes the whole record as Markdown, including a runnable snippet that rebuilds the plan. The file can serve as supplementary material.

scopus_search_report(records, file = "search-record.md")

Six of the sixteen PRISMA-S items are answered here from the objects, and a seventh, de-duplication, would be too had these records been merged with scopus_combine(). The rest, among them peer review of the strategy and grey literature, are listed as yours to supply, because the objects do not record them. Any other database the review searched is yours to name as well.

Retrieving more than 5000 records

Under offset paging, a query matching more than 5000 records cannot be retrieved in full from a single call. scopus_fetch() returns the first 5000 and warns. One remedy is to split the search by year, or by any other facet, so that each cell stays under the ceiling. scopus_count() tells you in advance whether a split is needed. The other remedy is scopus_fetch(cursor = TRUE), which follows the API’s cursor in place of an offset and retrieves the whole set in one call. The records then arrive in deep-paging order. The Analysing a literature article weighs a plan’s cached, resumable cells against the cursor’s complete set in a single pass.

scopus_fetch_plan(plan, cursor = TRUE) combines the two remedies. It pages every cell of a plan by cursor, so a single year that matches more than 5000 records is retrieved whole and keeps its checkpoint.

records <- scopus_fetch_plan(
  plan,
  cache_dir = file.path(scopus_cache_dir(), "language-learning-effect-size"),
  cursor = TRUE
)

Inside a plan, the warning for a cell that stops at the ceiling names scopus_fetch_plan(plan, cursor = TRUE), since the cell is already partitioned. Plans page by offset unless given cursor = TRUE. A checkpoint records the mode it was fetched under and serves only a harvest in the same mode, so switching modes fetches each cell again. The Python twin’s fetch_plan() pages by cursor by default, and by offset when given cursor=False, the route pybliometrics documents for keys without a Scopus subscription.

Handling interruptions

Network and API problems are raised as typed conditions, all inheriting from scopus_error, so a long retrieval can catch them and carry on.

result <- tryCatch(
  scopus_fetch_plan(
    plan,
    cache_dir = file.path(scopus_cache_dir(), "language-learning-effect-size")
  ),
  scopus_error_rate_limit = function(e) {
    message("Rate limited; the cached cells are safe. Try again later.")
    NULL
  }
)

Because each completed cell is cached, resuming after such a pause costs nothing for the work already done.

Elsevier answers throttling and a spent weekly quota alike with HTTP 429. Throttling is retried with back-off before any condition is raised. A spent quota is not, since it resets only days later. It is raised as scopus_error_quota_exceeded, a subclass of scopus_error_rate_limit, so the handler above catches it too, and its message gives the time the quota resets.