Trove unpublished works (diaries, letters, and archives)

Trove API access restrictions

With the cancellation of my Trove API keys by the National Library of Australia, I've made the difficult decision to stop work on Trove and archive all related code repositories.

The Trove sections of the GLAM Workbench will remain online, but they won't be updated. Everything here is openly-licensed, so feel free to take what’s useful and develop it further yourself.

Given the fact that the NLA is willing to change the API terms of use to restrict access without any consultation, provides no transparency around acceptable use of full text content, and is willing to cancel API keys without warning, I can no longer recommend Trove as a reliable source for digital research.

Current version: v1.0.0 ¶

Experiments and examples relating to Trove's 'Diaries, letters, and archives' zone.

See below for information on running these notebooks in a live computing environment. Or just take them for a spin using Binder.

Tips, tools, and examples¶

Finding unpublished works that might be entering the public domain on 1 January 2019 ¶

Changes to Australian copyright legislation mean that many unpublished resources will be entering the public domain on 1 January 2019. This notebook attempts to harvest the details of some of these resources from Trove.

Exploring unpublished works that might be entering the public domain on 1 January 2019 ¶

This notebook demonstrates some ways of exploring the data harvested about unpublished resources that might be entering the public domain on 1 January 2019.

Find urls of digitised finding aids ¶

This notebook uses the Trove API to harvest urls of NLA digitised finding aids from a search in the collection zone.

Collect information about digitised finding aids ¶

This notebook works through a list of urls pointing to NLA's digitised finding aids, extracting additional information about each one.

Convert a HTML finding aid to JSON ¶

This notebook scrapes data from the HTML of a finding aid, saving the hierarchy of series, sub-series, and items as a list of nested objects. The results can be saved as a JSON file.

Datasets¶

Unpublished works that might be entering the public domain on 1 January 2019 ¶

This dataset contains metadata describing unpublished works that might be entering the public domain on 1 January 2019.

NLA digitised finding aids: list of urls ¶

A list of urls pointing to the National Library of Australia's digitised manuscript finding aids, harvested from Trove.

NLA digitised finding aids: summary information ¶

This dataset includes summary information describing each finding aid.

Run these notebooks¶

There are a number of different ways to use these notebooks. Binder is quickest and easiest, but it doesn't save your data. I've listed the options below from easiest to most complicated (requiring more technical knowledge).

Using ARDC Binder¶

Click on the button above to launch the notebooks in this repository using the ARDC Binder service. This is a free service available to researchers in Australian universities. You'll be asked to log in with your university credentials. Note that sessions will close if you stop using the notebooks, and no data will be preserved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using ARDC Binder for more details.

Using Binder¶

Click on the button above to launch the notebooks in this repository using the Binder service (it might take a little while to load). This is a free service, but note that sessions will close if you stop using the notebooks, and no data will be saved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using Binder for more details.

Using Reclaim Cloud¶

Reclaim Cloud is a paid hosting service, aimed particularly at supported digital scholarship in hte humanities. Unlike Binder, the environments you create on Reclaim Cloud will save your data – even if you switch them off! To run this repository on Reclaim Cloud for the first time:

Create a Reclaim Cloud account and log in.
Click on the button above to start the installation process.
A dialogue box will ask you to set a password, this is used to limit access to your Jupyter installation.
Sit back and wait for the installation to complete!
Once the installation is finished click on the 'Open in Browser' button of your newly created environment (note that you might need to wait a few minutes before everything is ready).

See Using Reclaim Cloud for more details.

Running in a container on your own computer¶

GLAM Workbench repositories are stored as pre-built container images on quay.io. You can run these containers on your own computer to set up a virtual machine with everything you need to use the notebooks. This is free, but requires more technical knowledge – you'll have to install Podman on your computer, and be able to use the command line.

Install Podman.

In a terminal, run the following command:

podman run --rm -p 8888:8888 quay.io/glamworkbench/trove-unpublished jupyter lab --ip=0.0.0.0 --port=8888 --ServerApp.token="" --LabApp.default_url="/lab/tree/index.ipynb"

It will take a while to download and configure the container image. Once it's ready you'll see a message saying that Jupyter Notebook is running.
Point your web browser to http://127.0.0.1:8888
When you've finished, download any files or data you want to keep from Jupyter Lab, and enter Ctrl+C int the terminal.

See Running in a container on your own computer for more details.

Setting up on your own computer¶

If you know your way around the command line and are comfortable installing software, you might want to set up your own computer to run these notebooks. You'll need to have recent versions of Python and Git installed. I use pyenv, pyenv-virtualenv, and pip-tools to create and manage Python versions and environments.

In a terminal:

Create a Python virtual environment (Python >= 3.10 should be ok): pyenv virtualenv 3.10.12 trove-unpublished
Activate the virtual environment: pyenv local trove-unpublished
Use git clone to create a local version of the GLAM Workbench repository: git clone https://github.com/GLAM-Workbench/trove-unpublished.git
Use cd to move into the newly-cloned folder: cd trove-unpublished
Run pip install pip-tools to install pip-tools.
Run pip-sync requirements.txt dev-requirements.txt to install the required Python packages.
Start Jupyter with jupyter lab – a browser window should open automatically. If not, copy and paste the url from the command line to your web browser.
To shut down your Jupyter Lab session enter Ctrl+C in the terminal.

See Using Python on your own computer for more details.

Contributors¶

Tim Sherratt

Cite as¶

Sherratt, Tim. (2023). GLAM-Workbench/trove-unpublished (version v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.7690276