Solr Document Classification – Part 1 – Indexing Time
Alessandro Benedetti
July 4, 2015
8 mins read
Introduction to Solr Document Classification
This blog post is about the Solr classification module and the way Lucene classification has been integrated at indexing time.
Previously we have explored the world of Lucene Classification and the extension to use it for Document Classification.
It comes naturally to integrate Solr with the Classification module and allow Solr users to easily manage the Classification out of the box.
N.B. This is supported from Solr 6.1
Solr Document Classification
Taking inspiration from the work of a dear friend [2], integrating the classification in Solr can happen 2 sides :
- Indexing time – through an Update Request Processor
- Query Time – through a Request handler ( similar to the More like This )
In this article we are going to explore the Indexing time integration :
The Classification Update Request Processor
First of all, let’s describe some basic concepts :
An Update Request Processor Chain, associated with an Update handler, is a pipeline of Update processors, that will be executed in sequence.
It takes in input the added Document (to be indexed) and returns the document after it has been processed by all the processors in the chain in sequence.
Finally, the document is indexed.
An Update Request Processor is the unit of processing of a chain, it takes in input a Document and operates some processing before it is passed to the following processor in the chain if any.
The main reason for the Update processor is to add intermediate processing steps that can enrich, modify and possibly filter documents before they are indexed.
It is important because the processor has a view of the entire Document, so it can operate on all the fields the Document is composed.
For further details, follow the official documentation [3].
Description
The Classification Update Request Processor is a simple processor that will automatically classify a document (the classification will be based on the latest index available) by adding a new field containing the class before the document is indexed.
After an initial valuable index has been built with human-assigned labels to the documents, thanks to this Update Request Processor will be possible to ingest documents with automatically assigned classes.
The processing steps are quite simple :
When a document to be indexed enters the Update Processor Chain and arrives at the Classification step, this sequence of operations will be executed :
- The latest Index Reader is retrieved from the latest opened Searcher
- A Lucene Document Classifier is instantiated with the config parameters in the solrconfig.xml
- A Class is assigned by the classifier taking into consideration all the relevant fields from the input document
- A new field is added to the original Document, with the class
- The Document goes through the next processing step.
Configuration
K Nearest Neighbours Classifier
Simple Naive Bayes Classifier
Update Handler Configuration
| Parameter | Default | Description |
|---|---|---|
inputFields |
This config param is mandatory | The list of fields (comma separated) to be taken in consideration for doing the classification. Boosting syntax is supported for each field. |
classField |
This config param is mandatory | The field that contains the class of the document. It must appear in the indexed documents. If knn algorithm it must be stored. If bayes algorithm it must be indexed and ideally not heavily analyzed. |
algorithm |
knn | The algorithm to use for the classification: – knn ( K Nearest neighbours ) – bayes ( Simple Naive Bayes ) |
knn.k |
10 | Advanced – the no. of top docs to select in the MLT results to find the nearest neighbor |
knn.minDf |
1 | Advanced – A term (from the input text) will be taken in consideration by the algorithm only if it appears at least in this minimum number of docs in the index |
knn.minTf |
1 | Advanced – A term (from the input text) will be taken in consideration by the algorithm only if it appears at least this minimum number of times in the input |
Usage
Indexing News Documents? We can use the already indexed news with category to automatically tag upcoming stories with no human intervention.
E-commerce Search System? Category assignation will require a few human interactions after a valid initial corpus of products has been indexed with manually assigned categories.
The possible uses for this Update Request Processor are countless.
In any scenario where we have documents with a class or category manually assigned in our Search System, the automatic Classification can be a perfect fit.
Leveraging the existent Index, the overhead for the Classification processing will be minimal.
After an initial human effort to have a good corpus of classified Documents, the Search System will be able to automatically index the class for the upcoming Documents.
Of course, we must remember that for advanced classification scenarios that require deep tuning, this solution can be not optimal.
Code
The patch is attached to this Jira Issue :
This has been officially merged to Apache Solr starting with 6.1 version.
Need Help With This Topic?
If you’re struggling with document classification in Apache Solr, don’t worry – we’re here to help! Our team offers expert services and training to help you optimize your Solr search engine and get the most out of your system. Contact us today to learn more!