Reddit Historical Photo Crawler

This project explores a way to collect historical photographs and information from Reddit and transform them into structured historical events that can be shared and explored on Histolines.com.

The goal is to take historical photo posts from communities such as r/HistoryPorn and use Natural Language Processing (NLP) and Named Entity Recognition (NER) to identify important historical information contained in the text.

The information we are interested in extracting includes:

Who — people, organizations, teams, and other named entities
What — the historical event or subject being described
When — dates, years, and periods of time
Where — cities, countries, buildings, landmarks, and other locations

The resulting information can then be structured and added to Histolines as historical events.

The Basic Process

The current process looks like this:

Reddit → IFTTT → Google Docs → NLP / Named Entity Recognition → Structured Historical Data → Histolines

1. Set Up IFTTT

Create an account at IFTTT.

IFTTT ("If This Then That") can monitor a Reddit subreddit and automatically perform an action whenever a new post appears.

For example, you can create an IFTTT applet that monitors r/HistoryPorn.

When a new post appears, IFTTT can automatically send information about that post to a Google Doc.

This creates an ongoing collection of Reddit posts without requiring each post to be copied manually.

2. Collect the Reddit Posts

After the IFTTT process is running, the Google Doc becomes a collection of Reddit posts from the selected subreddit.

Each post may contain:

A historical photograph
A title
A description
A date or year
Names of people
Locations
Historical context
A link back to the original Reddit post

At this stage, however, the information is still largely unstructured text.

That's where the NLP processing begins.

3. Process the Raw Reddit Text with NLP

The descriptions accompanying historical photographs are often written for human readers rather than computers.

For example, a Reddit post might say:

A 1920s photograph of Albert Einstein visiting New York City.

A person can immediately identify several important pieces of information:

Person: Albert Einstein Date: 1920s Location: New York City

The Python code in this project uses Natural Language Processing (NLP) and Named Entity Recognition (NER) to help identify these entities automatically.

The goal is to transform the raw Reddit description into structured information such as:

WHO: Albert Einstein
WHAT: Visiting New York City
WHEN: 1920s
WHERE: New York City

This information can then be transformed into SQL or another structured format that can be imported into Histolines.com.

4. From Reddit Post to Historical Event

Once the important information has been extracted, it can be turned into a structured historical event.

The process creates a connection between an unstructured social-media post and a searchable historical timeline.

The basic concept is:

Reddit
   ↓
Historical photograph + description
   ↓
NLP / Named Entity Recognition
   ↓
People + Dates + Locations + Events
   ↓
Structured historical data
   ↓
Histolines
   ↓
Historical timeline

In other words:

Reddit provides the raw material → NLP structures the information → Histolines organizes it into history.

5. See the Results on Histolines

One of the most interesting parts of this project is being able to see what happens to the Reddit posts after they have been processed.

The resulting events are published on Histolines.com and can be viewed through the dedicated Reddit Crawler contributor timeline.

View the Reddit Crawler's Posts on Histolines

The Reddit Crawler Timeline shows historical events that have been collected and processed through this system.

Instead of leaving hundreds or thousands of Reddit posts in an unstructured document, the information can become part of an organized historical timeline.

The resulting events can be explored chronologically and connected with:

People
Dates
Locations
Historical events
Photographs
Related historical information

This provides a way to see how automated data collection and NLP can be used to build a growing historical resource.

Explore the Results

→ View the Reddit Crawler's Historical Timeline

Why This Is Interesting

There is an enormous amount of historical information being shared online every day.

Much of it exists in places such as:

Reddit
Social media
Online discussion forums
Digital archives
Historical photography communities
Personal collections
Local history websites

The challenge is that much of this information is unstructured.

A photograph might be historically significant, but its description could be only a sentence or two written by a Reddit user.

By combining automated collection with NLP and historical databases, we can begin turning that unstructured information into structured historical events.

The goal isn't to replace historians or human researchers.

Instead, the goal is to help organize large amounts of historical material so that people can discover and investigate it more easily.

Future Possibilities

This approach could potentially be expanded beyond a single subreddit.

Possible sources include:

Historical photography communities
Local history subreddits
City and regional history groups
Museum collections
Public-domain photograph collections
Historical blogs and websites
Digital archives
Other online communities containing historical material

The NLP processing could also become more sophisticated over time.

Potentially identifiable information includes:

Full names
Birth and death dates
Historical events
Locations
Organizations
Military units
Buildings
Geographic coordinates
Relationships between people
Multiple dates within the same post

The ultimate goal is to make it easier to take unstructured historical information and turn it into structured, searchable history.

Project Pipeline

The complete process can be summarized as:

                         REDDIT
                            │
                            ▼
                 Historical Photo Post
                            │
                            ▼
                          IFTTT
                            │
                            ▼
                       GOOGLE DOCS
                            │
                            ▼
                  Raw Reddit Description
                            │
                            ▼
                    Python NLP Processing
                            │
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
              PEOPLE      DATES    LOCATIONS
                 │          │          │
                 └──────────┼──────────┘
                            ▼
                  Structured Event Data
                            │
                            ▼
                       Histolines SQL
                            │
                            ▼
                      HISTOLINES.COM
                            │
                            ▼
                   Historical Timeline

The Bigger Idea

This project is an experiment in combining open-source software, Natural Language Processing, social media, and digital history.

There are millions of historical photographs and stories scattered across the internet.

The challenge isn't necessarily finding them.

The challenge is organizing them in a way that makes them useful and discoverable.

By automatically identifying people, dates, locations, and events, we can begin turning that scattered information into structured historical timelines.

Once the information is on Histolines, it can be explored in a completely different way.

You can start with a photograph.

Then discover the person in the photograph.

Then discover where they were.

Then discover what else was happening at that time.

Then follow the timeline somewhere completely unexpected.

That's the goal: turn scattered historical information into connected history.