thinkmap.datasource.flatfile2
Class FlatFileDataSource

java.lang.Object
  extended by thinkmap.datasource.util.AbstractDataSource
      extended by thinkmap.datasource.flatfile2.FlatFileDataSource
All Implemented Interfaces:
IDataSource, IRelationalDataSource, Searchable

public class FlatFileDataSource
extends AbstractDataSource
implements IRelationalDataSource, Searchable

FlatFileDataSource provides data from flat files representing database tables.

Version:
$Revision: 1.20.2.1 $

Field Summary
protected  org.apache.lucene.analysis.Analyzer analyzer
           
protected  QueryBuilder builder
           
protected  ThinkmapConfig config
           
protected  IThinkmapContext context
           
protected  int debugLevel
           
protected  int defaultMaxSearchResults
           
protected  org.apache.lucene.store.Directory directory
           
protected static org.apache.lucene.index.Term entityType
           
protected  int errorThreshold
           
protected  Map<String,Set<FieldDef<?>>> fieldMaps
           
protected  int indexingNotificationInterval
           
protected  File indexPath
           
protected  boolean inMemory
           
protected static org.apache.lucene.index.Term ismodelrecord
           
protected  Map<String,EntityModel> modelMaps
           
protected  List<EntityModel> models
           
protected  boolean newIndexOnStart
           
protected  ReadWriteLock rwlock
           
protected  org.apache.lucene.search.IndexSearcher searcher
           
protected  boolean shutdown
           
protected  int warningThreshold
           
protected  HashMap<String,EntityWriter> writers
           
protected  AtomicBoolean writing
           
 
Constructor Summary
FlatFileDataSource()
          Creates a new FlatFileDataSource object.
 
Method Summary
 void addEntityModel(EntityModel model)
          Adds the given entity model to this datasource
 EdgeEntityModel createEdge()
           
protected  Indexer createIndexer()
           
 NodeEntityModel createNode()
           
protected  IFlatFileReader createReader(thinkmap.datasource.flatfile2.IFFEntityModel model, URL url)
          Creates a Flat File Reader from the given IFFEntityModel and url
protected  org.apache.lucene.document.Document getDocumentForEntity(EntityId id)
          Returns an Document representing the given EntityId.
 EntityModel getEntityModel(String name)
          Get an entity model by name;
 Set<FieldDef<?>> getFields(String name)
           
protected  org.apache.lucene.store.Directory getIndexDir()
          Returns the Lucene Directory.
protected  void getRelated(EntityId id, IEntityHandler callback)
           
 String[] getTypeChain(String type)
           
protected  URL getURL(EntityModel em)
           
protected  EntityWriter getWriter(org.apache.lucene.document.Document d)
           
protected  EntityWriter getWriter(String name)
           
protected  void index(boolean create)
          This method is not meant to be called directly.
 void init()
           
protected  void initializeEdgeModel(FFEdgeEntityModel model)
          Initialize Edge Model
protected  boolean isCurrent(EntityModel model, long lastModified)
          Returns true if the given entity model is completely current with the index.
 boolean isIndexing()
          Returns true if the datasource is currently indexing
 boolean isWriting()
          Returns true if a writing operation is currently underway.
protected  void processEntityRequest(Message message, IEntityHandler callback)
          Handle an Entity request.
protected  void processIdRequest(Message message, IEntityHandler callback)
          Handle an ID request.
protected  void processMetadataRequest(Message message, IEntityHandler callback)
          Handle a metadata request.
protected  void processRelatedRequest(Message message, IEntityHandler callback)
          Handle a related request.
 void reindex(boolean create)
          Reindexes the Flat file datasource.
 void reindexAsync(boolean create)
          Performs a reindex asynchronously.
protected  void reportClientMessage(IEntityHandler callback, Exception e)
          Reports an error message to the client.
protected  void reportServerMessage(int level, String message, Exception e)
          Reportes a server message to the context.
 void setContext(IThinkmapContext context)
          Sets the context for this datasource.
 void setCreateNewIndexOnStart(boolean create)
          If set to true, the server will create a new index each time on startup.
 void setDebugLevel(int level)
          Sets the debug level for this datasource.
 void setDefaultMaxSearchResults(int defaultMaxSearchResults)
          Set's the default number of max search results that is returned by a search request.
 void setErrorThreshold(int threshold)
          Set's the error threshold for processing a file which is the number of errors that must take place before the file is abandoned entirely.
 void setIndexingNotificationInterval(int indexingNotificationInterval)
          Set's the indexing notification interval which determines how many lines must be processed before notifying the server of indexing progress.
 void setIndexPath(File path)
          Set's the directory where this index will be stored.
 void setMemoryOnly(boolean memory)
          Set's where this datasource should be stored entirely in memory.
 void setThinkmapConfig(ThinkmapConfig config)
          Sets the Thinkmap Config for this object
 void setWarningThreshold(int threshold)
          Set's the warning threshold for processing a file.
 void shutDown()
           
 void stopIndexing()
          Stops the current indexing process and rolls back the changes.
protected  thinkmap.datasource.flatfile2.IFFEntityModel unpackModel(org.apache.lucene.document.Document doc)
          Returns
 
Methods inherited from class thinkmap.datasource.util.AbstractDataSource
createRequest, processCustomRequest, processRequest
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface thinkmap.datasource.IDataSource
createRequest
 

Field Detail

writers

protected HashMap<String,EntityWriter> writers

builder

protected QueryBuilder builder

modelMaps

protected Map<String,EntityModel> modelMaps

models

protected List<EntityModel> models

searcher

protected org.apache.lucene.search.IndexSearcher searcher

directory

protected org.apache.lucene.store.Directory directory

inMemory

protected boolean inMemory

newIndexOnStart

protected boolean newIndexOnStart

context

protected IThinkmapContext context

indexPath

protected File indexPath

config

protected ThinkmapConfig config

defaultMaxSearchResults

protected int defaultMaxSearchResults

debugLevel

protected int debugLevel

indexingNotificationInterval

protected int indexingNotificationInterval

writing

protected AtomicBoolean writing

shutdown

protected volatile boolean shutdown

rwlock

protected ReadWriteLock rwlock

analyzer

protected final org.apache.lucene.analysis.Analyzer analyzer

fieldMaps

protected Map<String,Set<FieldDef<?>>> fieldMaps

entityType

protected static final org.apache.lucene.index.Term entityType

ismodelrecord

protected static final org.apache.lucene.index.Term ismodelrecord

errorThreshold

protected int errorThreshold

warningThreshold

protected int warningThreshold
Constructor Detail

FlatFileDataSource

public FlatFileDataSource()
Creates a new FlatFileDataSource object.

Method Detail

getEntityModel

public EntityModel getEntityModel(String name)
Description copied from interface: IRelationalDataSource
Get an entity model by name;

Specified by:
getEntityModel in interface IRelationalDataSource
Parameters:
name - the name of the EntityModel to retrieve
Returns:
the datasource's model

createNode

public NodeEntityModel createNode()
Specified by:
createNode in interface IRelationalDataSource

createEdge

public EdgeEntityModel createEdge()
Specified by:
createEdge in interface IRelationalDataSource

getTypeChain

public String[] getTypeChain(String type)

getURL

protected URL getURL(EntityModel em)
              throws IOException
Parameters:
em -
Returns:
Throws:
Exception
IOException

addEntityModel

public void addEntityModel(EntityModel model)
Adds the given entity model to this datasource

Parameters:
model -

setThinkmapConfig

public void setThinkmapConfig(ThinkmapConfig config)
Sets the Thinkmap Config for this object

Parameters:
config -

setContext

public void setContext(IThinkmapContext context)
Sets the context for this datasource. This must be set or injected or else the datasource will not be initialized properly.

Parameters:
context -

setMemoryOnly

public void setMemoryOnly(boolean memory)
Set's where this datasource should be stored entirely in memory. This means that the ds will be indexed everytime at startup rather than use the file system. This can lead to much faster lookup times but a bigger memory footprint.

Parameters:
memory -

setDefaultMaxSearchResults

public void setDefaultMaxSearchResults(int defaultMaxSearchResults)
Set's the default number of max search results that is returned by a search request. This can be overridden by individual client requests via the use of the MAX_SEARCH_RESULTS attribute in a message.

Parameters:
defaultMaxSearchResults -

setDebugLevel

public void setDebugLevel(int level)
Sets the debug level for this datasource. The levels are mapped to the IThinkmapContext levels. The default setting is MSG_INFO (4) which means that any message with a debug level less than or equal to MSG_INFO will be reported to the user.

Parameters:
level -

setIndexPath

public void setIndexPath(File path)
Set's the directory where this index will be stored. If not specified, or if the server cannot create a directory or write to this directory, then a temporary directory is used for the index.

Parameters:
path -

setCreateNewIndexOnStart

public void setCreateNewIndexOnStart(boolean create)
If set to true, the server will create a new index each time on startup.

Parameters:
create -

setErrorThreshold

public void setErrorThreshold(int threshold)
Set's the error threshold for processing a file which is the number of errors that must take place before the file is abandoned entirely.

Parameters:
threshold -

setWarningThreshold

public void setWarningThreshold(int threshold)
Set's the warning threshold for processing a file. This is the number of warnings that will take place before the datasource stops notifying you about warnings. This is so a user is not overwhelmed by the number of messages.

Parameters:
threshold -

setIndexingNotificationInterval

public void setIndexingNotificationInterval(int indexingNotificationInterval)
Set's the indexing notification interval which determines how many lines must be processed before notifying the server of indexing progress.

Parameters:
indexingNotificationInterval -

isWriting

public boolean isWriting()
Returns true if a writing operation is currently underway. This will return true if Indexing is occurring or if a rollback is occurring.

Returns:

isIndexing

public boolean isIndexing()
Returns true if the datasource is currently indexing

Returns:

reindexAsync

public void reindexAsync(boolean create)
Performs a reindex asynchronously. If a new thread cannot be created (Some servlet containers are run in a restricted mode), then the action is performed sychronously and this method will not return until the index operation has finished.


reindex

public void reindex(boolean create)
Reindexes the Flat file datasource.

Parameters:
create -

stopIndexing

public void stopIndexing()
Stops the current indexing process and rolls back the changes.


index

protected void index(boolean create)
              throws IOException
This method is not meant to be called directly. It already assumes that Indexing has started and also sets indexing to false.

Parameters:
create -
Throws:
IOException
Exception

createIndexer

protected Indexer createIndexer()

getFields

public Set<FieldDef<?>> getFields(String name)
Specified by:
getFields in interface Searchable

init

public void init()
          throws Exception
Throws:
Exception

shutDown

public void shutDown()
              throws Exception
Throws:
Exception

getIndexDir

protected org.apache.lucene.store.Directory getIndexDir()
                                                 throws IOException
Returns the Lucene Directory. If the IndexPath is specified by the user, than the file is made there. If its not specified or if the ds is unable to write to the IndexPath, this returns a file from the temp Directory.

Returns:
Throws:
IOException

isCurrent

protected boolean isCurrent(EntityModel model,
                            long lastModified)
Returns true if the given entity model is completely current with the index. In other words, this method returns true if the model is exactly the same since it was last indexed. It the model was never indexed, this method will return false.

Parameters:
model -
Returns:

initializeEdgeModel

protected void initializeEdgeModel(FFEdgeEntityModel model)
                            throws Exception
Initialize Edge Model

Parameters:
model -
Throws:
Exception

unpackModel

protected thinkmap.datasource.flatfile2.IFFEntityModel unpackModel(org.apache.lucene.document.Document doc)
                                                            throws IOException,
                                                                   ClassNotFoundException
Returns

Parameters:
doc -
Returns:
Throws:
IOException
ClassNotFoundException

getRelated

protected void getRelated(EntityId id,
                          IEntityHandler callback)
                   throws Exception
Parameters:
entityId -
callback -
Throws:
Exception

getWriter

protected EntityWriter getWriter(org.apache.lucene.document.Document d)

getDocumentForEntity

protected org.apache.lucene.document.Document getDocumentForEntity(EntityId id)
                                                            throws IOException
Returns an Document representing the given EntityId. Null is returned if the EntityId cannot be found in the index.

Parameters:
id -
Returns:
Throws:
Exception
IOException

processMetadataRequest

protected void processMetadataRequest(Message message,
                                      IEntityHandler callback)
Description copied from class: AbstractDataSource
Handle a metadata request.

Overrides:
processMetadataRequest in class AbstractDataSource
Parameters:
message - The incoming request, packaged into a handy object.
callback - The IEntityHandler which will receive the response.

processIdRequest

protected void processIdRequest(Message message,
                                IEntityHandler callback)
Description copied from class: AbstractDataSource
Handle an ID request.

Overrides:
processIdRequest in class AbstractDataSource
Parameters:
message - The incoming request, packaged into a handy object.
callback - The IEntityHandler which will receive the response.

processEntityRequest

protected void processEntityRequest(Message message,
                                    IEntityHandler callback)
Description copied from class: AbstractDataSource
Handle an Entity request.

Overrides:
processEntityRequest in class AbstractDataSource
Parameters:
message - The incoming request, packaged into a handy object.
callback - The IEntityHandler which will receive the response.

processRelatedRequest

protected void processRelatedRequest(Message message,
                                     IEntityHandler callback)
Description copied from class: AbstractDataSource
Handle a related request.

Overrides:
processRelatedRequest in class AbstractDataSource
Parameters:
message - The incoming request, packaged into a handy object.
callback - The IEntityHandler which will receive the response.

reportClientMessage

protected void reportClientMessage(IEntityHandler callback,
                                   Exception e)
Reports an error message to the client.

Parameters:
callback -
level -
message -
e -

reportServerMessage

protected void reportServerMessage(int level,
                                   String message,
                                   Exception e)
Reportes a server message to the context.

Parameters:
level -
message -
e -

createReader

protected IFlatFileReader createReader(thinkmap.datasource.flatfile2.IFFEntityModel model,
                                       URL url)
                                throws IOException
Creates a Flat File Reader from the given IFFEntityModel and url

Parameters:
model -
url -
Returns:
Throws:
IOException
MalformedURLException
UnsupportedEncodingException

getWriter

protected EntityWriter getWriter(String name)


Copyright © 2013 Thinkmap, Inc. All Rights Reserved.