Trove web archive collections (Pandora)

Current version: v1.0.1 ¶

This section includes information on finding, understanding, and using Pandora's collections of archived web pages.

Pandora has been selecting web sites and online resources for preservation since 1996. It has assembled a collection of more than 80,000 titles, organised into subjects and collections. The archived websites are now part of the Australian Web Archive (AWA), which combines the selected titles with broader domain harvests, and is searchable through Trove. However, Pandora's curated collections offer a useful entry point for researchers trying to find web sites relating to particular topics or events.

The Web Archives section of the GLAM Workbench provides documentation, tools, and examples to help you work with data from a range of web archives, including the Australian Web Archive. The title urls obtained through Pandora can be used to obtain additional data from the AWA for analysis. For example:

See below for information on running these notebooks in a live computing environment. Or just take them for a spin using Binder.

Notebooks¶

Create title datasets from collections and subjects ¶

This notebook helps you create a dataset of archived urls using Pandora's subject and collection groupings.

Harvest Pandora subjects and collections ¶

This notebook harvests Pandora's navigation hierarchy, saving the connections between subjects, collections, and titles.

Harvest the full collection of Pandora titles ¶

This notebook harvests a complete collection of archived web page titles from Pandora, the National Library of Australia's selective web archive.

Datasets¶

Pandora titles data ¶

This dataset contains a complete list of Pandora's archived web resource titles.

Pandora collections data ¶

This dataset contains details of the subject and collection groupings used by Pandora to organise archived web resource titles.

Pandora subject hierarchy ¶

A single-page view of Pandora's complete subject hierarchy.

Run these notebooks¶

There are a number of different ways to use these notebooks. Binder is quickest and easiest, but it doesn't save your data. I've listed the options below from easiest to most complicated (requiring more technical knowledge).

Using ARDC Binder¶

Click on the button above to launch the notebooks in this repository using the ARDC Binder service. This is a free service available to researchers in Australian universities. You'll be asked to log in with your university credentials. Note that sessions will close if you stop using the notebooks, and no data will be preserved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using ARDC Binder for more details.

Using Binder¶

Click on the button above to launch the notebooks in this repository using the Binder service (it might take a little while to load). This is a free service, but note that sessions will close if you stop using the notebooks, and no data will be saved. Make sure you download any changed notebooks or harvested data that you want to save.

See Using Binder for more details.

Using Reclaim Cloud¶

Reclaim Cloud is a paid hosting service, aimed particularly at supported digital scholarship in hte humanities. Unlike Binder, the environments you create on Reclaim Cloud will save your data – even if you switch them off! To run this repository on Reclaim Cloud for the first time:

Create a Reclaim Cloud account and log in.
Click on the button above to start the installation process.
A dialogue box will ask you to set a password, this is used to limit access to your Jupyter installation.
Sit back and wait for the installation to complete!
Once the installation is finished click on the 'Open in Browser' button of your newly created environment (note that you might need to wait a few minutes before everything is ready).

See Using Reclaim Cloud for more details.

Using the Nectar Cloud¶

Screenshot of GLAM Workbench application

The Nectar Research Cloud (part of the Australian Research Data Commons) provides cloud computing services to researchers in Australian and New Zealand universities. Any university-affiliated researcher can log on to Nectar and receive up to 6 months of free cloud computing time. And if you need more, you can apply for a specific project allocation.

The GLAM Workbench is available in the Nectar Cloud as a pre-configured application. This means you can get it up and going without worrying about the technical infrastructure – just fill in a few details and you're away! To create an instance of this repository in the Nectar Cloud:

Log in to the Nectar Dashboard using your university credentials.
From the Dashboard choose Applications -> Browse Local.
Enter 'GLAM' in the filter box and hit Enter, you should see the GLAM Workbench application.
Click on the GLAM Workbench application's Quick Deploy button.
Step through the various configuration options. Some options are only available if you have a dedicated project allocation.
When asked to select a GLAM Workbench repository, choose this repository from the dropdown list.
Complete the configuration and deploy your GLAM Workbench instance.
The url to access your instance will be displayed once it's ready. Click on the url!

See Using Nectar for more information.

Using Docker¶

You can use Docker to run a pre-built computing environment on your own computer. It will set up everything you need to run the notebooks in this repository. This is free, but requires more technical knowledge – you'll have to install Docker on your computer, and be able to use the command line.

Install Docker Desktop.
Create a new directory for this repository and open it from the command line.

From the command line, run the following command:

docker run -p 8888:8888 --name trove-web-archives -v "$PWD":/home/jovyan/work quay.io/glamworkbench/trove-web-archives repo2docker-entrypoint jupyter lab --ip 0.0.0.0 --NotebookApp.token='' --LabApp.default_url='/lab/tree/index.ipynb'

It will take a while to download and configure the Docker image. Once it's ready you'll see a message saying that Jupyter Notebook is running.
Point your web browser to http://127.0.0.1:8888

See Using Docker for more details.

Setting up on your own computer¶

If you know your way around the command line and are comfortable installing software, you might want to set up your own computer to run these notebooks.

Assuming you have recent versions of Python and Git installed, the steps might be something like:

Create a virtual environment, eg: python -m venv trove-web-archives
Open the new directory" cd trove-web-archives
Activate the environment source bin/activate
Clone the repository: git clone https://github.com/GLAM-Workbench/trove-web-archives.git notebooks
Open the new notebooks directory: cd notebooks
Install the necessary Python packages: pip install -r requirements.txt
Run Jupyter: jupyter lab

See Getting started for more details.

Contributors¶

Tim Sherratt

Cite as¶

Sherratt, Tim. (2024). GLAM-Workbench/trove-web-archives (version v1.0.1). Zenodo. https://doi.org/10.5281/zenodo.11124402