Trove periodicals

Trove API access restrictions

With the cancellation of my Trove API keys by the National Library of Australia, I've made the difficult decision to stop work on Trove and archive all related code repositories.

The Trove sections of the GLAM Workbench will remain online, but they won't be updated. Everything here is openly-licensed, so feel free to take what’s useful and develop it further yourself.

Given the fact that the NLA is willing to change the API terms of use to restrict access without any consultation, provides no transparency around acceptable use of full text content, and is willing to cancel API keys without warning, I can no longer recommend Trove as a reliable source for digital research.

Current version: v2.2.0 ¶

There are hundreds of digitised periodicals in Trove (not including newspapers). Information about them is spread across a number of categories, and it's not always easy to find what's available. The notebooks in this repository help you harvest metadata, text, and images from digitised periodicals in Trove. There are also a number of pre-harvested datasets.

If you just want to explore the range of digitised periodicals available, a good place to start is this database of titles and issues.

See below for information on running these notebooks in a live computing environment. Or just take them for a spin using Binder.

Notebooks¶

Harvesting metadata¶

Get details of periodicals from the `/magazine/titles` API endpoint ¶

This notebook uses the /magazine/titles endpoint of the Trove API to get details of digitised periodical titles and issues. It then tries to fix some problems with the data by removing duplicates and Parliamentary Papers, and checking the lists of issues against those scraped from the Trove website.

Enrich the list of periodicals from the Trove API ¶

This notebook tries to fix some problems with the periodicals data from the Trove API. It also enriches the harvested data by extracting additional information from the website. It creates two datasets – one for titles and one for issues – and loads these into an SQLite database for use with Datasette Lite.

Create a list of Trove's digital periodicals ¶

This notebook creates a list of digitised periodicals in Trove by searching for the digital identifier string nla.obj and limiting the results to periodicals. Before the Trove API introduced the /magazine/titles endpoint, this was the only way to generate such a list. This method produces slightly different results to the new API endpoint, and it might be useful to compare the two to see what each method misses.

Harvesting text¶

Get OCRd text from a digitised journal in Trove ¶

Many of the digitised periodicals available in Trove make OCRd text available for download. This notebook helps you download all the OCRd text from a single periodical – one text file for each issue.

Download the OCRd text for ALL the digitised periodicals in Trove!¶

This notebook helps you download all the OCRd text from all (or most of?) Trove's digitised periodicals, creating one text file for each issue. It also saves a CSV-formatted list of the issues in each periodical.

Create a database to search across each line of text in a series of volumes ¶

The code here was used to create the NSW Post Office Directories search interface which helps you search across 54 volumes from 1886 to 1950. The same code, with minor modifications, could be used to index any publication where it would be useful to search by line (rather than Trove's default 'article') – for example, lists, directories and gazetteers – turning them into searchable databases.

Harvesting images¶

Download issues of a periodical as PDFs ¶

This notebook helps you download the issues of a digitised periodical as PDFs. You can download all digitised issues, or specify a range of years to include.

Get covers (or any other pages) from a digitised journal in Trove ¶

This notebook shows how to download all the cover images from a specified periodical. With some minor modifications you could download any page, or range of pages.

Finding editorial cartoons in the Bulletin ¶

This notebook describes a method for finding and saving full-page editorial cartoons from The Bulletin.

Harvest illustrations from periodicals ¶

This notebook shows you how to harvest illustrations from Trove's digitised periodicals. It makes use of layout information generated by Trove's OCR processing to find the coordinates of illustrations on a digitised page. Using these coordinates the illustrations can be cropped from the page image and saved.

Datasets¶

Editorial cartoons from The Bulletin, 1886 to 1952 ¶

This dataset includes a collection of 3,471 full-page editorial cartoons downloaded from issues of The Bulletin published between 1886 and 1952. In most cases there is one cartoon per issue. Metadata describing each image is available in a CSV-fromatted file, and in an SQLite database that can be explored using Datasette-Lite. The full collection of high-resolution images can be downloaded as a single 62gb zip file.

CSV formatted list of journals available from Trove in digital form ¶

This dataset contains version records describing digitised periodicals found by searching for the digital identifier string nla.obj and limiting the results to periodicals. Duplicate records were merged.

Details of digitised periodicals from the `/magazine/titles` API endpoint ¶

This dataset was created by checking, correcting, and enriching data about digitised periodicals obtained from the Trove API. Additional metadata describing periodical titles and issues was extracted from the Trove website and used to check the API results. Where titles were wrongly described as issues, and vice versa, the records were corrected. Additional descriptive metadata was also added into the records. Separate CSV formatted data files were created for titles and issues. Finally, the titles and issues data was loaded into an SQLite database for use with Datasette.

OCRd text from Trove digitised journals ¶

This dataset contains OCRd text and metadata harvested from digitised periodicals in Trove.

Run these notebooks¶

There are a number of different ways to use these notebooks. Binder is quickest and easiest, but it doesn't save your data. I've listed the options below from easiest to most complicated (requiring more technical knowledge).

Using ARDC Binder¶

Click on the button above to launch the notebooks in this repository using the ARDC Binder service. This is a free service available to researchers in Australian universities. You'll be asked to log in with your university credentials. Note that sessions will close if you stop using the notebooks, and no data will be preserved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using ARDC Binder for more details.

Using Binder¶

Click on the button above to launch the notebooks in this repository using the Binder service (it might take a little while to load). This is a free service, but note that sessions will close if you stop using the notebooks, and no data will be saved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using Binder for more details.

Using Reclaim Cloud¶

Reclaim Cloud is a paid hosting service, aimed particularly at supported digital scholarship in hte humanities. Unlike Binder, the environments you create on Reclaim Cloud will save your data – even if you switch them off! To run this repository on Reclaim Cloud for the first time:

Create a Reclaim Cloud account and log in.
Click on the button above to start the installation process.
A dialogue box will ask you to set a password, this is used to limit access to your Jupyter installation.
Sit back and wait for the installation to complete!
Once the installation is finished click on the 'Open in Browser' button of your newly created environment (note that you might need to wait a few minutes before everything is ready).

See Using Reclaim Cloud for more details.

Running in a container on your own computer¶

GLAM Workbench repositories are stored as pre-built container images on quay.io. You can run these containers on your own computer to set up a virtual machine with everything you need to use the notebooks. This is free, but requires more technical knowledge – you'll have to install Podman on your computer, and be able to use the command line.

Install Podman.

In a terminal, run the following command:

podman run --rm -p 8888:8888 quay.io/glamworkbench/trove-journals jupyter lab --ip=0.0.0.0 --port=8888 --ServerApp.token="" --LabApp.default_url="/lab/tree/index.ipynb"

It will take a while to download and configure the container image. Once it's ready you'll see a message saying that Jupyter Notebook is running.
Point your web browser to http://127.0.0.1:8888
When you've finished, download any files or data you want to keep from Jupyter Lab, and enter Ctrl+C int the terminal.

See Running in a container on your own computer for more details.

Setting up on your own computer¶

If you know your way around the command line and are comfortable installing software, you might want to set up your own computer to run these notebooks. You'll need to have recent versions of Python and Git installed. I use pyenv, pyenv-virtualenv, and pip-tools to create and manage Python versions and environments.

In a terminal:

Create a Python virtual environment (Python >= 3.10 should be ok): pyenv virtualenv 3.10.12 trove-journals
Activate the virtual environment: pyenv local trove-journals
Use git clone to create a local version of the GLAM Workbench repository: git clone https://github.com/GLAM-Workbench/trove-journals.git
Use cd to move into the newly-cloned folder: cd trove-journals
Run pip install pip-tools to install pip-tools.
Run pip-sync requirements.txt dev-requirements.txt to install the required Python packages.
Start Jupyter with jupyter lab – a browser window should open automatically. If not, copy and paste the url from the command line to your web browser.
To shut down your Jupyter Lab session enter Ctrl+C in the terminal.

See Using Python on your own computer for more details.

Contributors¶

Tim Sherratt

Cite as¶

Sherratt, Tim. (2024). GLAM-Workbench/trove-journals (version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.13744407