Client-Side Flat-file Tutorial

Learn how to load flat-file data into Thinkmap

Introduction

In this tutorial, you'll learn how to load flat-file data from a CSV file into a Thinkmap graph on the client-side. Using a flat-file datasource on the client-side is a good practice when you have a small amount of data or are looking to rapidly prototype to see how a Thinkmap visualization will work. If you have more data, we recommend you either use a database, or the flat-file datasource that is available as part of the Thinkmap server.

If you view the source of this page, you can see the code that makes up this example in its entirety, but we'll go through it section-by-section here.

Getting Started

If you view the source of this example, you will notice that we are linking to two important source files: The first file is the thinkmap.js file for your project. It contains the Thinkmap API. The second file flatfile.js contains the code for this example, which we will now discuss.

We start by creating a graph, a display and configuring the display.

	// create a graph and a display
	var graph = new thinkmap.graph.Graph();
	var display = new thinkmap.display.Display(document.getElementById('map'));
	
	// configure the display
	configureDisplay(display);

Loading the Data

We will be loading data from 7 files that are all stored in the data/ directory in this example. There are 4 files that will represent nodes:

  1. movie.csv
  2. actress.csv
  3. actor.csv
  4. director.csv

And 3 files that represent edges (connections between the nodes):

  1. movie_actress.csv
  2. movie_actor.csv
  3. movie_director.csv

You will notice that actors, actresses, and directors are all types of people. We are going to represent that fact by creating a custom type of "node" in our Thinkmap named "person". This will allow us to treat actors, actresses, and directors as "person" nodes when we query and configure the display.

	// create an entity type named "person" that extends "node"
	graph.createType('person','node');

Because edges depend on the nodes that they link, we need to load the node files first, and then load the edges.

Let's look at the code that starts loading the data. It calls two functions that we will look at in depth later, loadNodes and loadEdges

	// First, call the function that loads the files that insert the nodes
	// into the graph.  This function returns a thinkmap.async.Future.
	loadNodes()
		// we use the "then" method on the Future that is returned
		// to call the loadEdges function when the nodes are done loading
		.then(loadEdges) 
		// when the edges are done, we can start our example running		
		.then(function(){
			// center on movie:6997 ("The Breakfast Club") 
			center(graph.selectOne({$t:'movie',movieid:6997}));			
		});		

Loading a CSV File

First, we'll need to get the data from the movie.csv file using the Thinkmap thinkmap.data.csv API:

// Get the data from the movie.csv file. Note that because the data is fetched
// asynchronously, this method returns a thinkmap.async.Future.  A Future is an
// object that will produce a value sometime in the future.  When this 
// particular future completes, it will produce an Array of objects that 
// represents the rows of data in the file
thinkmap.data.csv().load('data/movie.csv')

Looking at the Data

Let's look at the first few lines of the movie.csv file.

movieid,title,year,rating,country,release,stars
615,About Last Night...,1986,R,USA,1986-07-01 00:00:00.000,2
651,Absence of Malice,1981,PG,USA,1981-10-09 00:00:00.000,3
659,The Abyss,1989,PG13,USA,1989-08-09 00:00:00.000,2

You will notice that it is comma separated, and that the first line contains the column names. If the first line of you file does not contain the column names, you will have to define the column names manually via the thinkmap.data.csv().columnNames() method. The CSV API has methods for setting column names, delimiters, and other useful parsing options.

Feeding Data to the Node Loader

Next, we'll take the above code and feed the result of the CSV load future into the flat-file node loader:

// load the movie table
thinkmap.graph.fftable.node('movie')
     .keys({'movieid':true}) // movieid is the primary key
     .intoGraph(graph)       // insert into the main graph
	 .data( 
		// Get the data from the movie.csv file.
		// Note that this returns a thinkmap.async.Future
		// that returns an Array of objects that represents
		// the rows of data.  The "data" method of fftable
		// will accept either an Array of objects, or a
		// Future that returns an Array of objects.
		thinkmap.data.csv().load('data/movie.csv') 
	 )	

Loading all the nodes

The last step is to load all the node files in parallel. We do that by combining them into a single Future that will complete when all the files have finished loading:

// this will return a future that will complete when all
// three futures complete...
thinkmap.async.all( future1, future2, future3 )

Let's look at the complete loadNodes() method:

// load all the nodes, and return a future that 
// completes when they are all loaded
function loadNodes() {
	// we use thinkmap.async.all to load all the nodes in parallel
	// it returns a future that completes when all the dependent futures
	// complete
	return thinkmap.async.all(
		// load the movie table
		thinkmap.graph.fftable.node('movie')
		     .keys({'movieid':true}) // movieid is the primary key
		     .intoGraph(graph)       // insert into the main graph
			 .data( 
				// Get the data from the movie.csv file.
				// Note that this returns a thinkmap.async.Future
				// that returns an Array of objects that represents
				// the rows of data.  The "data" method of fftable
				// will accept either an Array of objects, or a
				// Future that returns an Array of objects.
				thinkmap.data.csv().load('data/movie.csv') 
			 ),
		
		// Load the actor table.
		// Note that here we tell the fftable loader that "actor" nodes
		// will extend "person" nodes.
		thinkmap.graph.fftable.node('actor').extend('person')
			.keys({'actorid':true}).intoGraph(graph)
			.data( thinkmap.data.csv().load('data/actor.csv')),	
		
		// load the actress table
		thinkmap.graph.fftable.node('actress').extend('person')
			.keys({'actressid':true}).intoGraph(graph)
			.data( thinkmap.data.csv().load('data/actress.csv')),
		
		// load the director table	
		thinkmap.graph.fftable.node('director').extend('person')
			.keys({'directorid':true}).intoGraph(graph)
			.data( thinkmap.data.csv().load('data/director.csv'))
	);
}

Loading the Edges

Loading the edges is very similar to loading the nodes. The only difference is that for edges, we have to define the mapping between the edge file and the nodes it connects. Let's look at the first few lines of the movie_actor.csv edge file:

movieid,actorid,role,billing,star
615,971,Mother Malone,6,0
615,18928,Mr. Favio,5,1
615,28428,Gus,0,0
615,37227,Colin,0,0
615,43513,Danny,1,1

Each row contains both the movieid and the actorid that acted in that movie. When we load an edge, we need to tell thinkmap how it can "look up" the nodes on either side of the edge. In this case, it's fairly easy, since the "local" column name on the edge table matches the "foreign" column name on the node table that it connects.

Here's the code that we use to tell Thinkmap how to load the movie_actor.csv edge table:

// load the movie_actor edge table	
thinkmap.graph.fftable.edge('movie_actor')
		// this edge goes from movie nodes to actor nodes
 	    .from('movie').to('actor') 
 	    // define the "from" foreign key mapping, specified as
 	    // { 'local column name' : 'foreign column name' }
 	    .fromKeys({'movieid':'movieid'}) 
 	    // define the "to" foreign key mapping
 	    .toKeys({'actorid':'actorid'})
 	    .intoGraph(graph) // into the main graph
 	    .data( 
 	    	// get the data from the movie_actor.csv file
 	    	thinkmap.data.csv().load('data/movie_actor.csv')
    	    ),	

Let's look at the completed loadEdges() function:

// load all the edges, and return a future that 
// completes when they are all loaded
function loadEdges() {
	// we use thinkmap.async.all to load all the edges in parallel
	// it returns a future that completes when all the dependent futures
	// complete
	return thinkmap.async.all(
		// load the movie_actor edge table	
		thinkmap.graph.fftable.edge('movie_actor')
		// this edge goes from movie nodes to actor nodes
   	    .from('movie').to('actor') 
   	    // define the "from" foreign key mapping, specified as
   	    // { 'local column name' : 'foreign column name' }
   	    .fromKeys({'movieid':'movieid'}) 
   	    // define the "to" foreign key mapping
   	    .toKeys({'actorid':'actorid'})
   	    .intoGraph(graph) // into the main graph
   	    .data( 
   	    	// get the data from the movie_actor.csv file
   	    	thinkmap.data.csv().load('data/movie_actor.csv')
   	    ),			    	
   	    
   	    // load the movie_actress edge table	
   	    thinkmap.graph.fftable.edge('movie_actress')
   	    .from('movie').to('actress')
   	    .fromKeys({'movieid':'movieid'})
   	    .toKeys({'actressid':'actressid'})
   	    .intoGraph(graph)
   	    .data(thinkmap.data.csv().load('data/movie_actress.csv')),
   	
   	    // load the movie_director edge table	
    	thinkmap.graph.fftable.edge('movie_director')
   	    .from('movie').to('director')
   	    .fromKeys({'movieid':'movieid'})
   	    .toKeys({'directorid':'directorid'})
   	    .intoGraph(graph)
   	    .data(thinkmap.data.csv().load('data/movie_director.csv'))    	    
    );
}

Wrapping Up

Well, there you have it, we've loaded a flat-file database of CSV files into a Thinkmap Graph! You can use these techniques to load all kinds of data into Thinkmap. Client-side flat files are great for small projects or rapid prototyping. Just remember that for larger projects with more data, you'll want to use Thinkmap Server. In fact, Thinkmap Server has a flat-file option that will allow you to browse and navigate flat file databases that are gigabytes in size. When you use Thinkmap Server, it automatically transfers just the portions of the data that you need at any given time.