Fetching and Requests

Learn how to refetch entities from the datasources and alter messages

Introduction

This example demonstrates how to how refetch relationships from a Thinkmap datasource based on a set of attributes that limit data.

Every time a Thinkmap client fetches relationships from a Thinkmap datasource, it generally fetches all of these relationships at once, and the relationships remain cached on the client (as in, the edges for a node will never be requested again until a user navigates away from the page). In certain scenarios, this isn't practical or ideal, as the relationships themselves could be overwhelming or perhaps dynamic.

This tutorial demonstrates a couple of ways of solving this problem using a simple movie database and client-side event listeners.

Getting Started

In this example, when navigating to genres, the movies shown are determined by the begin and end year select menus. In a normal project, the changing of these values would result in a graph query using the updated values. In other words, all of these relationships are fetched at once, and then the query just adds/removes the relevant ones to the display. And just to emphasize once more, the client will never go back out to the datasource for those relationships again. They are now cached.

However, in this example, the entities are fetched each time from the datasource and the years are passed along in the request. The datasource then returns only the movie entities between the given years.

This example is able to do this by using message attributes, cache invalidatation, and custom queries - all of which we will cover below.

If you view the source of this page, you can see the code that makes up this example in its entirety. You will notice that we are linking to two important source files: The first file is the thinkmap.js file for your project. The second file fetching-requests.js contains the code for this example, which we will now discuss.

Datasource and Annotating Messages

As seen in other examples, a graph can have one or more datasources, thus allowing graph queries on the client to reach out to remote datasources and fetch nodes and edges needed to satisfy a query. These requests generally only occur when the data is not yet present on the client.

However, what if the remote datasource, when responding to these requests, requires some additional data to help determine which data should be streamed back to the client? For instance, consider the scenario of geographic region or userid where such attributes could limit data seen by each user. A more common scenario - perhaps there are too many relationships for a given node and this data should be limited so as not to overwhelm the client? This can all be accomplished by adding message attributes for each related request to the datasource.

In Thinkmap 5, the best way to add messages to a datasource request is through attaching listeners to the Thinkmap datasource on the client. The three most common listeners are relatedrequest,entityrequest, and sendrequest. For the purposes of this example, we will only consider the first one relatedrequest as it is the most common and useful event of the three.

A relatedrequest is fired everytime the datasource is making a request to the server regarding which edges are related to a particular node. These requests only occur when the relationships are not yet cached on the client. Once they are cached, they will no longer be requested unless you specifically tell the datasource to clear the cache (which is covered later in the third section).

In this example, we want to send data regarding the min and max years for movies as selected in the drop down menu. This way, when a query is fired for the related edges to a genre, our related request will add the min and max years we are interested in:

	    // build graph and display
	    var graph = new thinkmap.graph.Graph();
	    // attach a datasource to the main graph
	    var datasource = graph.datasource('server.tas');
      	.....
      	.....
      		
	    // two years for min and max
	    var minYear, maxYear;
      	
       	// on a related request for genres, add attributes for min year and max
	    datasource.on('relatedrequest', function(entity, condition, rs, req) {
	    	if (entity.isType('genre')) {
	    		// create req attributes object if it doesn't exist
	    		if (!req.attributes) {
	    			req.attributes = {};
	    		}
	    		// add attributes that will be used by the server
	    		req.attributes.minYearForMovies = minYear;
	    		req.attributes.maxYearForMovies = maxYear;
	    	}
	    });
      	

Believe it or not, that is all we need to do. The above sends attributes on every related request for genre. Now we need to change our datasource so that it actually uses these attributes in the query.

Custom Queries

Custom queries are covered in much more detail in other examples, so we won't get into them too deeply here. We will only take a look at a related request and then rewriting a related request to use our attributes. Also, we could opt to not use custom queries and instead rely on a custom datasource which can also leverage custom attributes

With that said, lets take a look at a standard related query and then in the next section, add a new one that uses attributes. Here is how a typical query looks (these can be seen on the tas_status page by clicking "Show Queries")

    
   	<query type="related" bindableEntityType="MOVIE">
   		
        <sql><![CDATA[
            SELECT B.genreid, B.genrename, B.moviecount, A.movieid, A.genreid FROM 
            MOVIE_GENRE A, GENRE B WHERE A.movieid = ? AND A.genreid = B.genreid
        ]]></sql>
        <arguments>
        	
            <entityidbinding keyindex="0" outputType="java.lang.Integer"/>
        </arguments>
        
        <results>
        	
            <result>
                <entitytype>
                    <constantbinding value="GENRE"/>
                </entitytype>
                <idBindings>
                    <sqlbinding columnIndex="0" outputType="java.lang.Integer"/>
                </idBindings>
                <propertyBindings>
                    <sqlbinding columnIndex="0" outputType="java.lang.Integer" property="genreid"/>
                    <sqlbinding columnIndex="1" outputType="java.lang.String" property="genrename"/>
                    <sqlbinding columnIndex="2" outputType="java.lang.Integer" property="moviecount"/>
                </propertyBindings>
            </result>
            
            <result>
                <entitytype>
                    <constantbinding value="MOVIE_GENRE"/>
                </entitytype>
                <idBindings>
                    <sqlbinding columnIndex="3" outputType="java.lang.Integer"/>
                    <sqlbinding columnIndex="4" outputType="java.lang.Integer"/>
                </idBindings>
                <propertyBindings>
                    <sqlbinding columnIndex="3" outputType="java.lang.Integer" property="movieid"/>
                    <sqlbinding columnIndex="4" outputType="java.lang.Integer" property="genreid"/>
                </propertyBindings>
				
                <relations>
                	
                    <relation isReciprocal="true" name="Sink">
                	
                        <entityType>
                            <constantbinding value="GENRE"/>
                        </entityType>
                        <idBindings>
                            <sqlbinding columnIndex="0" outputType="java.lang.Integer"/>
                        </idBindings>
                    </relation>
                    
                    <relation isReciprocal="true" name="Source">
                	
                        <entityType>
                            <constantbinding value="MOVIE"/>
                        </entityType>
                        <idBindings>
                            <entityidbinding keyindex="0"/>
                        </idBindings>
                    </relation>
                </relations>
            </result>
        </results>
    </query> 
    

Now the above is generated for you automatically from the node and edge definitions that are defined in the server.tas file. However, we can follow this pattern to write our own queries.

WRITING OUR OWN CUSTOM QUERY

So we need to write our own request for a related request to GENRE to make use of our min and max year bindings.

The first step to any custom query is going to the tas_status page and copy and pasting a generated related query for our type. In this case, we would copy and paste the standard GENRE related generated query

Now once we have our query pasted in the server.tas file, we want to find our MOVIE_GENRE definition, and set generateRelatedQuery to false:

  	<edge name="MOVIE_GENRE" from="MOVIE" to="GENRE" table="MOVIE_GENRE" 
  					bidirectional="true" generateRelatedQuery="false" >
		<column name="movieid"   type="Integer" property="true" bindto="from"  bindColumn="movieid" />
		<column name="genreid"   type="Integer" property="true" bindto="to"    bindColumn="genreid" />
	</edge>	
  	

As noted earlier, the custom attributes in this example are just the min and max year of movies. This is actually a pretty simple change. We just need to modify our SQL slightly to accomdate movie year and use <attributebindings /> in the arguments section that reference the year attributes.

   	<query type="related" bindableEntityType="GENRE">
        
        <sql><![CDATA[
            SELECT B.movieid, B.title, B.year, B.rating, B.country, B.release, B.stars, A.movieid, A.genreid 
            FROM MOVIE_GENRE A, MOVIE B 
            WHERE A.movieid = B.movieid AND 
            A.genreid = ? AND B.year >= ? AND B.year <= ?
        ]]></sql>
        <arguments>
            <entityidbinding keyindex="0" outputType="java.lang.Integer"/>
            
            <attributebinding attribute="minYearForMovies" outputType="java.lang.Integer" />
            <attributebinding attribute="maxYearForMovies" outputType="java.lang.Integer"/>
        </arguments>
        
        
        ... more content here
    </query>
   

Believe it or not, that is all we really need to do. As you can see, although there is a lot of XML, it really was just copy and pasted from the SHOW QUERIES section in the server.tas file. All we really did was add the arguments and the bindings.

Invalidating and Refetching

Now, that we have a way to send attributes to a datasource during requests, and our datasource can use those attributes to execute a query, the next step is to force the client to refetch those relationships.

It is actually rather simple to invalidate the relationship cache on the client. Just iterate over all the nodes in the graph, and on each one, call the datasource's method invalidateRelations

      	// invalidates all the relations in the graph which basically means 
	    // their edges are no longer cached. 
	    // Therefore, on the the next query, the client will go 
	    // out to the datasource to refetch any edges that were invalidated
	    function invalidateRelations() {
	    	// call invalidate relations for each node
	    	graph.nodes().forEach(datasource.invalidateRelations);
	    	// call make center, using the current center as the center, this forces
	    	// the entire graph to be refetched
	    	makeCenter();
	    }	
      	

In our example, this invalidate code will be called every time a user changes the year drop down menus.

        // update min year when drop down menu changes 
	    document.getElementById('startyear').addEventListener('change', function() {
	    	minYear = this.value;
	    	invalidateRelations(); // invalidate relations after changing value
	    });
	    
	    // update max year when drop down menu changes 
	    document.getElementById('endyear').addEventListener('change', function() {
	    	maxYear = this.value;
	    	invalidateRelations(); // invalidate relations after changing value
	    });	
      	

Now just invalidating the cache doesn't mean the relationships in the graph will be refreshed. We actually need to perform a query on the nodes that requires that graph to refetch the entities from the datasource. This is why we call makeCenter() with no arguments. Doing so causes the centerying query to be re-run on the center node, which should force a refetch on all nodes in the display.

Even though the call to makeCenter will force a new query to be issued on the center entity, it won't remove any edges that have expired (or in other words, are no longer returned by the datasource.) Therefore, the edges have to be explicitly removed. We cover how to handle this in the next section.

Resolving the new or missing Relationships

Ok, so we can now invalidate relationships, send attributes to the datasource, return data to the client using those attributes, so what is next?

The final step is to actually listen for events where the datasource discovers that the edges that currently exist on the client are different from what the server returned. The client doesn't automatically remove or add these edges. It is up to the developer what should occur in this case. The reason being, the edges in the cache may be have been added on the client, or came from some other request. It isn't possible to define a default behavior. Therefore, the datasource takes another approach. It fires a relationsupdate event which notifies you when the relationships on the client are different from the server.

Here is how we use it in our example.

	    // called when relations are updated from refetching relations
	    datasource.on('relationsupdate', function(result) {
	    	// remove edges from the graph that were not returned by the datasource
	    	// by the most recent related request for a given entity
	    	var toRemove = result.newRelations.complement(result.oldRelations);
	    	graph.remove(toRemove);
	    }); 	
      	

Summary

That concludes this tutorial. We covered how to fetch certain slices of data rather than everything once using message attributes and custom queries. We then demonstrated how to make this possible on the client by clearing the cache and removing no longer needed relationships.