org.broadleafcommerce.core.search.service.solr.indexer

Class CatalogSolrIndexUpdateCommandHandlerImpl

java.lang.Object

org.broadleafcommerce.core.search.service.solr.indexer.AbstractSolrIndexUpdateCommandHandlerImpl

org.broadleafcommerce.core.search.service.solr.indexer.CatalogSolrIndexUpdateCommandHandlerImpl

All Implemented Interfaces:

CatalogSolrIndexCommandHandler, SolrIndexUpdateCommandHandler, org.springframework.beans.factory.DisposableBean

@Service(value="blCatalogSolrUpdateCommandHandler")
public class CatalogSolrIndexUpdateCommandHandlerImpl
extends AbstractSolrIndexUpdateCommandHandlerImpl
implements CatalogSolrIndexCommandHandler, org.springframework.beans.factory.DisposableBean

Default command handler to handle Catalog Solr index commands. This is multi-threaded. SKU browsing is not supported with this implementation.

Author:

Kelly Tisdell

Field Summary

Fields

Modifier and Type	Field and Description
protected CatalogDocumentBuilder	documentBuilder
protected SolrIndexServiceExtensionManager	extensionManager
protected IndexFieldDao	indexFieldDao
protected LocaleService	localeService
protected int	pageSize
protected ProductDao	productDao
protected SolrIndexQueueProvider	queueProvider
protected long	reindexCommitInterval
protected SandBoxHelper	sandBoxHelper
protected SolrHelperService	shs
protected SiteService	siteService
protected SolrConfiguration	solrConfiguration
protected SolrIndexDao	solrIndexDao
protected org.springframework.transaction.PlatformTransactionManager	transactionManager
protected int	workerThreads

Constructor Summary

Constructors

Constructor and Description
CatalogSolrIndexUpdateCommandHandlerImpl()

Method Summary

Modifier and Type	Method and Description
protected void	afterBackgroundThread(ReindexStateHolder holder, Catalog catalog, Site site, SandBox sandBox) This executes after to the main execution of the background (page) thread(s).
protected void	afterPage(List<Long> productIds, List<Product> products, List<Locale> locales, List<IndexField> fields, ReindexStateHolder holder) This executes immediately after a page is processed.
protected void	afterProcess(ReindexStateHolder holder) Hook point at that is executed at the end of the reindex process.
protected void	afterReadIdBatch(ReindexStateHolder holder, Long catalogId, int batchSize, Long lastId) This executes immediately after reading a batch of product IDs for processing in a background thread.
protected void	afterReadProducts(ReindexStateHolder holder, List<Long> productIds) This executes immediately after reading a batch of products.
protected void	beforeBackgroundThread(ReindexStateHolder holder, Catalog catalog, Site site, SandBox sandBox) The reindex process, specifically page processing, is executed in background threads.
protected void	beforePage(List<Long> productIds, List<Product> products, List<Locale> locales, List<IndexField> fields, ReindexStateHolder holder) This is executed immediately before a page is processed.
protected void	beforeProcess(ReindexStateHolder holder) Hook point that is executed at the beginning of the reindex process.
protected void	beforeReadIdBatch(ReindexStateHolder holder, Long catalogId, int batchSize, Long lastId) This executes immediately prior to reading a batch of product IDs for processing in a background thread.
protected void	beforeReadProducts(ReindexStateHolder holder, List<Long> productIds) This executes immediately prior to reading a batch of products.
org.apache.solr.common.SolrInputDocument	buildDocument(Indexable indexable) Provides an interface for a caller to convert an Indexable into a SolrInputDocument.
org.apache.solr.common.SolrInputDocument	buildDocument(Indexable indexable, List<IndexField> fields, List<Locale> locales) Provides an interface for a caller to convert an Indexable into a SolrInputDocument.
protected void	buildIncrementalIndex(List<Long> productIds, List<Product> products, ReindexStateHolder holder, Catalog catalog, Site site) Builds an incremental or batch portion of the index.
protected List<org.apache.solr.common.SolrInputDocument>	buildPage(List<Long> productIds, List<Product> products, List<Locale> locales, List<IndexField> fields, ReindexStateHolder holder) Given the arguments, this builds a list of SolrInputDocuments.
protected org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor	createBackgroundOperationExecutor()
protected EntityManagerAwareRunnable	createBackgroundRunnable(ReindexStateHolder holder, List<Long> ids, Semaphore sem, Long catalogId, Long siteId, SandBox sandBox) This is where most of the heavy lifting happens.
protected void	deleteAllDocuments(String collection, boolean commit) By default this deletes all items in the collection, since there is a good chance that the collection will be re-aliased or swapped into the foreground after reindexing is complete.
void	destroy()
protected void	executeCatalogReindexCommand(CatalogReindexCommand command)
<C extends SolrUpdateCommand> void	executeCommand(C command) Entry point from which this component can delegate action based on the type of SolrUpdateCommand.
protected void	executeFullReindexCommand(FullReindexCommand command)
protected void	executeSiteReindexCommand(SiteReindexCommand command)
protected void	finalizeChanges(String collection, boolean error, boolean swap) Composite method that issues a commit or rollback (if there is an error), and optionally swaps (if there is no error).
protected Catalog	findCatalog(Long catalogId) This reads a Catalog by ID.
protected Site	findSite(Long siteId) This reads a Site by ID.
protected List<Locale>	getAllLocales()
protected org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor	getBackgroundOperationExecutor()
protected long	getIncrementalCommitInterval() The amount of time between commits during a full reindex.
protected IdentityOperation<Void,Exception>	getIncrementalIndexOperation(ReindexStateHolder holder, Catalog catalog, Site site, List<Long> ids) Returns an IdentityOperation who may be executed in the context of a Catalog and/or Site to read products by IDs, and then build the index incrementally.
protected List<IndexField>	getIndexFields()
protected IdentityOperation<List<Long>,Exception>	getReadIdsOperation(ReindexStateHolder holder, Long catalogId, Long siteId, Integer batchSize, Long lastId) Provides an IdentityOperation (function that runs in the context of a Site and/or Catalog) to read batches of IDs.
protected SolrConfiguration	getSolrConfiguration()
protected int	getThreadsForBackgroundExecution() Number of worker threads (page processing threads) in the thread pool.
protected void	incrementalCommit(ReindexStateHolder holder) Issues incremental commits.
protected boolean	isReindexSuccessful(ReindexStateHolder holder) Method to determine whether a full reindex was successful.
protected void	performCachedOperation(SolrIndexCachedOperation.CacheOperation cacheOperation)
protected void	populateIndex(ReindexStateHolder holder, Long catalogId, Long siteId, SandBox sandbox) This method populates the index.
protected List<Long>	readIdBatch(ReindexStateHolder holder, Long catalogId, int batchSize, Long lastId) Reads a batch of Product IDs from the DB.
protected List<Product>	readProductsByIds(ReindexStateHolder holder, List<Long> productIds) Reads a batch of products by IDs.
protected void	swapCollections()

Methods inherited from class org.broadleafcommerce.core.search.service.solr.indexer.AbstractSolrIndexUpdateCommandHandlerImpl

addDocument, addDocuments, commit, deleteByIds, deleteByQueries, deleteByQuery, executeCommandInternal, executeCommandInternal, executeCommandInternalNoDefaultCommandType, getBackgroundCollectionName, getCommandGroup, getForegroundCollectionName, rollback

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.broadleafcommerce.core.search.service.solr.indexer.SolrIndexUpdateCommandHandler

getBackgroundCollectionName, getCommandGroup, getForegroundCollectionName

Field Detail

solrConfiguration

@Qualifier(value="blCatalogSolrConfiguration")
 @Autowired(required=false)
protected SolrConfiguration solrConfiguration

localeService

protected LocaleService localeService

shs

protected SolrHelperService shs

siteService

protected SiteService siteService

extensionManager

protected SolrIndexServiceExtensionManager extensionManager

transactionManager

protected org.springframework.transaction.PlatformTransactionManager transactionManager

solrIndexDao

protected SolrIndexDao solrIndexDao

sandBoxHelper

protected SandBoxHelper sandBoxHelper

indexFieldDao

protected IndexFieldDao indexFieldDao

productDao

protected ProductDao productDao

queueProvider

protected SolrIndexQueueProvider queueProvider

documentBuilder

protected CatalogDocumentBuilder documentBuilder

pageSize

@Value(value="${solr.index.product.pageSize:100}")
protected int pageSize

workerThreads

@Value(value="${solr.index.product.workerThreads:10}")
protected int workerThreads

reindexCommitInterval

@Value(value="${solr.index.product.reindexCommitInterval:30000}")
protected long reindexCommitInterval

Constructor Detail

CatalogSolrIndexUpdateCommandHandlerImpl

public CatalogSolrIndexUpdateCommandHandlerImpl()

Method Detail

executeCommand

public <C extends SolrUpdateCommand> void executeCommand(C command)
                                                  throws ServiceException

Description copied from interface: SolrIndexUpdateCommandHandler

Entry point from which this component can delegate action based on the type of SolrUpdateCommand. This should never be called directly. Rather it will be called from inside the AbstractSolrIndexUpdateServiceImpl.

Specified by:

executeCommand in interface SolrIndexUpdateCommandHandler

Throws:

ServiceException

buildDocument

public org.apache.solr.common.SolrInputDocument buildDocument(Indexable indexable,
                                                              List<IndexField> fields,
                                                              List<Locale> locales)

Description copied from interface: SolrIndexUpdateCommandHandler

Provides an interface for a caller to convert an Indexable into a SolrInputDocument. This may return null if the implementor does not want the specified Indexable indexed.

Specified by:

buildDocument in interface SolrIndexUpdateCommandHandler

Returns:

buildDocument

public org.apache.solr.common.SolrInputDocument buildDocument(Indexable indexable)

Description copied from interface: SolrIndexUpdateCommandHandler

Provides an interface for a caller to convert an Indexable into a SolrInputDocument. This may return null if the implementor does not want the specified Indexable indexed.

Specified by:

buildDocument in interface SolrIndexUpdateCommandHandler

Returns:

executeFullReindexCommand

protected void executeFullReindexCommand(FullReindexCommand command)
                                  throws ServiceException

Throws:

ServiceException

executeCatalogReindexCommand

protected void executeCatalogReindexCommand(CatalogReindexCommand command)
                                     throws ServiceException

Throws:

ServiceException

executeSiteReindexCommand

protected void executeSiteReindexCommand(SiteReindexCommand command)
                                  throws ServiceException

Throws:

ServiceException

getSolrConfiguration

protected SolrConfiguration getSolrConfiguration()

Specified by:

getSolrConfiguration in class AbstractSolrIndexUpdateCommandHandlerImpl

getAllLocales

protected List<Locale> getAllLocales()

performCachedOperation

protected void performCachedOperation(SolrIndexCachedOperation.CacheOperation cacheOperation)
                               throws ServiceException

Throws:

ServiceException

populateIndex

protected void populateIndex(ReindexStateHolder holder,
                             Long catalogId,
                             Long siteId,
                             SandBox sandbox)
                      throws ServiceException

This method populates the index. It is not recommended that you override this method. Rather, consider overriding one of the other methods as this one generally coordinates and delegates to others.

Throws:

ServiceException

getReadIdsOperation

protected IdentityOperation<List<Long>,Exception> getReadIdsOperation(ReindexStateHolder holder,
                                                                      Long catalogId,
                                                                      Long siteId,
                                                                      Integer batchSize,
                                                                      Long lastId)

Provides an IdentityOperation (function that runs in the context of a Site and/or Catalog) to read batches of IDs.

Parameters:

holder -

catalogId -

siteId -

batchSize -

lastId -

Returns:

deleteAllDocuments

protected void deleteAllDocuments(String collection,
                                  boolean commit)
                           throws ServiceException

By default this deletes all items in the collection, since there is a good chance that the collection will be re-aliased or swapped into the foreground after reindexing is complete.

Parameters:

collection -

commit -

Throws:

ServiceException

finalizeChanges

protected void finalizeChanges(String collection,
                               boolean error,
                               boolean swap)
                        throws ServiceException

Composite method that issues a commit or rollback (if there is an error), and optionally swaps (if there is no error).

Parameters:

collection -

error -

swap -

Throws:

ServiceException

swapCollections

protected void swapCollections()
                        throws ServiceException

Throws:

ServiceException

destroy

public void destroy()
             throws Exception

Specified by:

destroy in interface org.springframework.beans.factory.DisposableBean

Throws:

Exception

getBackgroundOperationExecutor

protected org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor getBackgroundOperationExecutor()

getThreadsForBackgroundExecution

protected int getThreadsForBackgroundExecution()

Number of worker threads (page processing threads) in the thread pool. Default is 10.

Returns:

createBackgroundOperationExecutor

protected org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor createBackgroundOperationExecutor()

createBackgroundRunnable

protected EntityManagerAwareRunnable createBackgroundRunnable(ReindexStateHolder holder,
                                                              List<Long> ids,
                                                              Semaphore sem,
                                                              Long catalogId,
                                                              Long siteId,
                                                              SandBox sandBox)

This is where most of the heavy lifting happens.

Parameters:

collectionName -

holder -

sem -

incrementalCommits -

Returns:

getIncrementalIndexOperation

protected IdentityOperation<Void,Exception> getIncrementalIndexOperation(ReindexStateHolder holder,
                                                                         Catalog catalog,
                                                                         Site site,
                                                                         List<Long> ids)

Returns an IdentityOperation who may be executed in the context of a Catalog and/or Site to read products by IDs, and then build the index incrementally.

Parameters:

holder -

catalog -

site -

ids -

Returns:

buildPage

protected List<org.apache.solr.common.SolrInputDocument> buildPage(List<Long> productIds,
                                                                   List<Product> products,
                                                                   List<Locale> locales,
                                                                   List<IndexField> fields,
                                                                   ReindexStateHolder holder)
                                                            throws Exception

Given the arguments, this builds a list of SolrInputDocuments. This does not write them to Solr.

Parameters:

productIds -

products -

locales -

fields -

holder -

Returns:

Throws:

Exception

readProductsByIds

protected List<Product> readProductsByIds(ReindexStateHolder holder,
                                          List<Long> productIds)
                                   throws Exception

Reads a batch of products by IDs.

Parameters:

holder -

productIds -

Returns:

Throws:

Exception

getIndexFields

protected List<IndexField> getIndexFields()

buildIncrementalIndex

protected void buildIncrementalIndex(List<Long> productIds,
                                     List<Product> products,
                                     ReindexStateHolder holder,
                                     Catalog catalog,
                                     Site site)
                              throws Exception

Builds an incremental or batch portion of the index. Activities include reading Locales and IndexFields, building a page, adding the documents to the index, and incrementally committing. It is not recommended that you override this method. Rather, consider overriding one of the other methods as this is largely responsible for orchestrating and delegating.

Parameters:

productIds -

products -

holder -

catalog -

site -

Throws:

Exception

readIdBatch

protected List<Long> readIdBatch(ReindexStateHolder holder,
                                 Long catalogId,
                                 int batchSize,
                                 Long lastId)
                          throws Exception

Reads a batch of Product IDs from the DB.

Parameters:

holder -

catalogId -

batchSize -

lastId -

Returns:

Throws:

Exception

beforeProcess

protected void beforeProcess(ReindexStateHolder holder)
                      throws ServiceException

Hook point that is executed at the beginning of the reindex process. If you extend this class and implement this method, consider calling the super method.

Parameters:

holder -

Throws:

ServiceException

afterProcess

protected void afterProcess(ReindexStateHolder holder)
                     throws ServiceException

Hook point at that is executed at the end of the reindex process. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occured during the process.

Parameters:

holder -

Throws:

ServiceException

beforeBackgroundThread

protected void beforeBackgroundThread(ReindexStateHolder holder,
                                      Catalog catalog,
                                      Site site,
                                      SandBox sandBox)
                               throws ServiceException

The reindex process, specifically page processing, is executed in background threads. This executes prior to the main execution of the thread. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

catalog -

site -

sandBox -

Throws:

ServiceException

afterBackgroundThread

protected void afterBackgroundThread(ReindexStateHolder holder,
                                     Catalog catalog,
                                     Site site,
                                     SandBox sandBox)
                              throws ServiceException

This executes after to the main execution of the background (page) thread(s). If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

catalog -

site -

sandBox -

Throws:

ServiceException

beforePage

protected void beforePage(List<Long> productIds,
                          List<Product> products,
                          List<Locale> locales,
                          List<IndexField> fields,
                          ReindexStateHolder holder)
                   throws ServiceException

This is executed immediately before a page is processed. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

productIds -

products -

locales -

fields -

holder -

Throws:

ServiceException

afterPage

protected void afterPage(List<Long> productIds,
                         List<Product> products,
                         List<Locale> locales,
                         List<IndexField> fields,
                         ReindexStateHolder holder)
                  throws ServiceException

This executes immediately after a page is processed. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

productIds -

products -

locales -

fields -

holder -

Throws:

ServiceException

beforeReadProducts

protected void beforeReadProducts(ReindexStateHolder holder,
                                  List<Long> productIds)
                           throws ServiceException

This executes immediately prior to reading a batch of products. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

productIds -

Throws:

ServiceException

afterReadProducts

protected void afterReadProducts(ReindexStateHolder holder,
                                 List<Long> productIds)
                          throws ServiceException

This executes immediately after reading a batch of products. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

productIds -

Throws:

ServiceException

beforeReadIdBatch

protected void beforeReadIdBatch(ReindexStateHolder holder,
                                 Long catalogId,
                                 int batchSize,
                                 Long lastId)
                          throws ServiceException

This executes immediately prior to reading a batch of product IDs for processing in a background thread. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

catalogId -

batchSize -

lastId -

Throws:

ServiceException

afterReadIdBatch

protected void afterReadIdBatch(ReindexStateHolder holder,
                                Long catalogId,
                                int batchSize,
                                Long lastId)
                         throws ServiceException

This executes immediately after reading a batch of product IDs for processing in a background thread. If you extend this class and implement this method, consider calling the super method. Note that you can query the ReindexStateHolder to determine if an error has occurred during the process.

Parameters:

holder -

catalogId -

batchSize -

lastId -

Throws:

ServiceException

findCatalog

protected Catalog findCatalog(Long catalogId)

This reads a Catalog by ID. If the ID is null (default, unless in Multi-Tenant mode), this returns null.

Parameters:

catalogId -

Returns:

findSite

protected Site findSite(Long siteId)

This reads a Site by ID. If the ID is null (default, unless in Multi-Tenant mode), this returns null.

Parameters:

siteId -

Returns:

getIncrementalCommitInterval

protected long getIncrementalCommitInterval()

The amount of time between commits during a full reindex. Commits are global, so there's no need to have every thread committing every page. Default is 30 seconds.

Returns:

incrementalCommit

protected void incrementalCommit(ReindexStateHolder holder)
                          throws Exception

Issues incremental commits. Commits in Sorl are global, so we don't need multiple threads issuing commits at the same time, or even back-to-back. This uses a commit interval to determine if a commit should be issued.

Parameters:

holder -

Throws:

Exception

isReindexSuccessful

protected boolean isReindexSuccessful(ReindexStateHolder holder)

Method to determine whether a full reindex was successful. By default, this simply returns whether the ReindexStateHolder.isFailed() returns false and that at least 1 document was indexed. Implementors may wish to override this to determine whether the ReindexStateHolder.getIndexableCount() is a certain number, or if ReindexStateHolder.getUnindexedItemCount() is at least a percentage of total items or of the items that were indexed, given that the failure to build a single document / product does not stop the reindex process.

Parameters:

holder -

Returns: