In this post we’re going to learn how to import the DBLP citation network into Neo4j using the Neo4j Import Tool.

In case you haven’t come across this dataset before, Tomaz Bratanic has a great blog post explaining it.

The tl;dr is that we have articles, authors, and venues. Authors can write articles, articles can reference other articles, and articles are presented at a venue. Below is the graph model for this dataset:

Tomaz then goes on to show how to load the data into Neo4j using Cypher and the APOC library. Unfortunately this process is quite slow as there’s a lot of data so, since I’m not as patient as Tomaz, I wanted to try and find a faster way to get the data into the graph.

At a high level, the diagram below shows what happens when we import data using Cypher:

When we use the Neo4j import tool we’re able to skip that middle bit:

We skip the whole transactional machinery, so it’s much faster. We do pay a price for that extra speed:

If we’re happy with that trade off then Neo4j Import is the way to go.

The raw data is in JSON files, so our process will be as follows:

We don’t have to use a Python script to transform the raw data, but that’s the scripting language I’m most familiar with so I tend to use that.

Before we look at the script, let’s explore the JSON files that we need to transform. Below is a sample of one of these files:

Each row contains one JSON document and, as we can see, these documents contain the following keys:

For each of these JSON documents we’re going to create the following graph structure:

The properties from our JSON document will be assigned as follows:

So we know what data we’re going to extract from our JSON file, but what should the CSV files that we create look like?

The Neo4j Import Tool expects separate CSV files containing the nodes and relationships that we want to create. For each of those files we can either include a header line at the top of the file, or we can store those header lines in a separate line. We’re going to take the latter approach in this blog post.

The fields in those headers have a very specific format, it’s almost a mini DSL. Let’s take the example of the Author and Venue nodes and the corresponding relationship. We’ll have three CSV files.

The node headers will look like this:

articles_header.csv

venues_header.csv

For node files one of the fields needs to act as an identifier. We define this by including :ID in that field. In our example we’re also specifying an optional node group in parentheses e.g. :ID(Article). The node group is used to indicate that the identifier only needs to be unique within that group, rather than across all nodes.

And the relationships header will look like this:

article_VENUE_venue_header.csv

Again our header fields need to contain some custom values:

As with the nodes, we can specify a node group in parentheses e.g. :START_ID(Article).

So now we know what files we need to create, let’s have a look at the script that generates these files:

There’s nothing too clever going here - we’re iterating over the JSON files and writing into CSV files. For the relationships we write those directly to the CSV files as soon as we see them. For the nodes we’re collecting those in in-memory data structures to remove duplicates, and then we write those to the CSV files at the end.

Let’s have a look at a subset of the data in those CSV files:

This script takes a few minutes to run on my machine. There are certainly some ways we could improve the script, but this is good enough as a first pass.

Now it’s time to import the data into Neo4j. If we’re using the Neo4j Desktop we can access the Neo4j Import Tool via the 'Terminal' tab within a database.

We can then paste in the script below:

Let’s go through the arguments we’re passing to the Neo4j Import Tool. The line below imports our author nodes:

The syntax of this part of the command is --nodes:[:Label]=<"file1,file2,…​">.

It treats the files that we provide as if they’re one big file, and the first line of the first file needs to contain the header line. So in this line we’re saying that we want to create one node for each entry in the authors.csv file, and each of those nodes should have the label Author.

The line below creates relationships between our Author and Venue nodes:

The syntax of this part of the command is --relationships[:RELATIONSHIP_TYPE]=<"file1,file2,…​">

Again, it treats the files we provide as if they’re one big file, so the first line of the first file must contain the header line. In this line we’re saying that we want to create a relationship with the type VENUE for each entry in the article_VENUE_venue.csv file.

Now if we run the script, we’ll see the following output:

And now let’s start the database and have a look at the graph we’ve imported: