News Managing Big Data at the National Collection of Aerial Photography

Managing Big Data at the National Collection of Aerial Photography

In the foreground is a computer monitor showing an image of a black and white aerial photo. In the background an out-of-focus figure and robotic arm are lit by a green light.

The National Collection of Aerial Photography (NCAP) is one of the largest collections of aerial imagery in the world. The Collection contains 30 million high-quality analogue images that capture key moments in history and provide a unique perspective on places across the globe, offering invaluable insights into changing landscapes over time. Preserving and sharing this vast collection requires an immense effort in both digitisation and data management. As NCAP works to digitise these millions of images, we can generate terabytes of data every day.

To effectively manage large volumes of data, NCAP collaborates with the Edinburgh International Data Facility (EIDF), which offers High-Performance Computing (HPC) services and advanced data management expertise. This enables NCAP to efficiently store, process, and retrieve millions of high-resolution images, ensuring they are accessible to researchers, academics, and the public. By leveraging the EIDF’s capabilities, NCAP can handle rapidly growing datasets and provide faster access to these valuable resources, making the images more readily available for use in research, education, and industry.

A grey-gloved hand holds a black barcode scanner. The scanner is emitting a red light which has been pointed at a barcode label. The label is attached to a metal shelf holding 2 cardboard boxes.

In addition to managing the technical aspects of storing and processing large datasets, NCAP is also focused on improving how these images are accessed. For commercial users and those requiring high-resolution imagery, NCAP has established streamlined processes to ensure easier access to large image files.

As digital technology continues to evolve, NCAP is also embracing the shift towards born-digital collections. These are images that are captured digitally rather than converted from analogue formats, which NCAP is actively acquiring. We are also collaborating with major international archives like the National Archives and Records Administration (NARA) in the United States to expand our collection. These partnerships help make a broader range of both analogue and digital aerial photography accessible to the public and ensure that NCAP remains at the forefront of both preserving and sharing historical imagery.

A grey-gloved finger points at a table on a computer monitor. Only one column of the table is visible, in which the entries read DOS Turks & Ciacos, DOS Nigeria and DOS Ghana.

With our commitment to innovation, NCAP continues to improve our data management practices, embracing advancements in geospatial data management and data processing to enhance the accessibility and usability of the Collection. Through these efforts and strategic partnerships, NCAP is creating a growing, invaluable resource for global research, environmental studies, urban planning, and many other fields, ensuring that the rich history captured in its aerial imagery can be explored and studied for years to come.

 

Justyne Snyder, NCAP Sales Admin Assistant

 

Read more about how we look after data at NCAP:

Data Management at NCAP  Our Partners - EPCC  Explore our Collection on Air Photo Finder