Dev:SolrCloud
This is an advanced Solr for YaCy installation which uses the SolrCloud architecture. If you want to read and understand this, you should be (at least a little bit) familiar with debian, Solr and tomcat.
In this example, we install a shard of 4 Solr instances within the same server.
Inhaltsverzeichnis
Software Installation
We install tomcat, zookeeper and YaCy as standard debian packages and Solr as web app for tomcat.
Tomcat Installation
We will install tomcat as a standard debian system service using apt:
apt-get install tomcat6 tomcat6-examples tomcat6-admin tomcat6-docs
The tomcat web service on port 8080 will start automatically and you can open the default page at http://localhost:8080 The optional packages tomcat6-examples tomcat6-admin tomcat6-docs are great to develop and test applications, but it is also possible to omit them. If you installed the optional packages, then you can test them:
- http://localhost:8080/docs/ is the online-documentation
- http://localhost:8080/examples/ links to a set of example tomcat applications
- http://localhost:8080/manager/html and http://localhost:8080/host-manager/html are tomcat management applications but their access is restricted. To use them you must set a password in /etc/tomcat6/tomcat-users.xml, like
<?xml version='1.0' encoding='utf-8'?> <tomcat-users> <role rolename="admin"/> <role rolename="admin-gui"/> <role rolename="manager"/> <role rolename="manager-gui"/> <user username="admin" password="tomcat" roles="admin,admin-gui,manager,manager-gui"/> </tomcat-users>
After setting this, you must restart tomcat with
/etc/init.d/tomcat6 restart
and then you can log in the [manager] and [host-manager] servlet with the user 'admin' and the password 'tomcat'. Please replace the default password 'tomcat' with your own.
The relevant paths for the result of this installation are:
tomcat users: /etc/tomcat6/tomcat-users.xml CATALINA_HOME: /usr/share/tomcat6 CATALINA_BASE: /var/lib/tomcat6 default web page: /var/lib/tomcat6/webapps/ROOT/index.html
Zookeeper Installation
The SolrCloud peers need a common configuration system which is provided by zookeeper. Zookeeper can be installed with
apt-get install zookeeper zookeeperd
This will create a new user named 'zookeeper'. The relevant paths are at
Zookeeper config: /etc/zookeeper/conf (linked to /etc/zookeeper/conf_example) Zookeeper data: /var/lib/zookeeper/ Zookeeper binary: /usr/share/zookeeper/
To check if Zookeeper is running, start the Zookeeper shell:
/usr/share/zookeeper/bin/zkCli.sh
and run shell scripts like
ls / ls /zookeeper
Because solr is started within tomcat and needs to know the host address of zookeeper, we must assign this to tomcat as a jvm option. Open the file /usr/share/tomcat6/bin/catalina.sh and add the following lines at the begining of the document (right after the comments):
# added zookeeper host information used by tomcat to find Solr shards for the SolrCloud CATALINA_OPTS=$CATALINA_OPTS -DzkHost=localhost:2181
..and restart tomcat
/etc/init.d/tomcat6 restart
Solr Installation
Download a solr release from http://lucene.apache.org/solr/ (Solr 4.5.1. worked while Solr 4.6.0 did not work!) i.e.
cd /opt wget http://apache.mirrors.spacedump.net/lucene/solr/4.5.1/solr-4.5.1.tgz tar xfz solr-4.5.1.tgz ln -s solr-4.5.1 solr ln -s solr-4.5.1/dist/solr-4.5.1.war solr.war
Because Solr uses a different logging in jetty as implemented in solr, we must add slf4j adapters to the tomcat library
cd /usr/share/tomcat6/lib/ wget http://www.slf4j.org/dist/slf4j-1.6.6.zip apt-get install unzip unzip slf4j-1.6.6.zip cp slf4j-1.6.6/{jcl-over-slf4j-1.6.6.jar,slf4j-1.6.6/log4j-over-slf4j-1.6.6.jar,slf4j-1.6.6/slf4j-api-1.6.6.jar,slf4j-1.6.6/slf4j-jdk14-1.6.6.jar} .
and restart tomcat:
/etc/init.d/tomcat6 restart
YaCy Installation
Follow the YaCy for Debian installation instructions and select 'webportal' as network to join into (we consider that you do this not create a standalone-YaCy, not a peer-to-peer participant; you can of course also use this for a 'freeworld' peer as well). The relevant paths are at
YaCy data: /var/lib/yacy YaCy log: /var/log/yacy YaCy binary: /usr/share/yacy/ Solr conf for YaCy: /usr/share/yacy/defaults/solr
Software Configuration
The SolrCloud needs a common configuration of the index cores used by YaCy. YaCy uses two cores, 'collection1' and 'webgraph'. Both are defined with a generic index schema and they are exact clones of each other. It may be also possible to defines these cores with non-generic, exact defined schema.xml files, but we will not do that right now because it makes things much more complex.
Zookeeper Client for Solr
First, we need a Zookeeper client for Solr because Solr provides it's own client app to upload the relevant configuration files. We must fabricate this client using the libraries inside the Solr war-file and additional libraries for logging. We use the already installed war file, you must adopt the paths here if you used a more recent version of Solr:
unzip -q /opt/solr.war -d /tmp/solr-war/ mkdir /usr/share/zookeeper/solr-cli-lib cp /tmp/solr-war/WEB-INF/lib/* /usr/share/zookeeper/solr-cli-lib/ # solr libs cp /opt/solr/example/lib/ext/* /usr/share/zookeeper/solr-cli-lib/ # logger libs rm -Rf /tmp/solr-war
Now we can take advantage of the SolrCloud ZooKeeper CLI commands.
Create Solr Configuration of Solr Cores for YaCy Inside Zookeeper
For a detailed description of the set-up of Solr Clusters and a SolrCloud configuration, see the [SolrCloud Wiki of apache.org], the [SolrCloud Installation in Tomcat], a [Guide to SolrCloud Configuration] and a [SolrCloud Cluster (Single Collection) Deployment]. To upload the solr configuration in Zookeeper, we fabricate a config directory using the solr example config and the YaCy genric schema file schema.xml:
cp -R /opt/solr/example/solr/collection1/conf /opt/yacyconf cp /usr/share/yacy/defaults/solr/schema.xml /opt/yacyconf/
We can then use that to upload the configuration to zookeeper:
java -classpath .:/usr/share/zookeeper/solr-cli-lib/* org.apache.solr.cloud.ZkCLI -zkhost localhost:2181 -cmd upconfig -confdir /opt/yacyconf -confname yacygeneric
That configuration is good for both collections, 'collection1' and 'webgraph'. We can link this configuration therefore to both collections:
java -classpath .:/usr/share/zookeeper/solr-cli-lib/* org.apache.solr.cloud.ZkCLI -zkhost localhost:2181 -cmd linkconfig -collection collection1 -confname yacygeneric java -classpath .:/usr/share/zookeeper/solr-cli-lib/* org.apache.solr.cloud.ZkCLI -zkhost localhost:2181 -cmd linkconfig -collection webgraph -confname yacygeneric
Lets see whats inside of zookeeper now, i.e. how the collection1 is linked against the generic schema:
/usr/share/zookeeper/bin/zkCli.sh get /collections/collection1
Create Tomcat Configuration of Solr Web Services
We want to use four Solr servers as a SolrCloud, each with two cores ('collection1' and 'webgraph'). We create subdirectories for the servers inside of /var/opt/solrcloud/:
mkdir /var/opt/solrcloud/ mkdir /var/opt/solrcloud/solr0 mkdir /var/opt/solrcloud/solr1 mkdir /var/opt/solrcloud/solr2 mkdir /var/opt/solrcloud/solr3
In each of these directories, put a file named solr.xml. The description for the creation of that file in the web is mainly void, since there is a new [xml structure for solr.xml for Solr 4.4 and beyond], especially for [Core Discovery with SolrCloud]. Put the following content into /var/opt/solrcloud/solr0/solr.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<solr>
<int name="coreLoadThreads">4</int>
<solrcloud>
<str name="host">localhost</str>
<int name="hostPort">8080</int>
<str name="hostContext">solr0</str>
<str name="zkHost">localhost:2181</str>
<int name="zkClientTimeout">${solr.zkclienttimeout:30000}</int>
<str name="shareSchema">${shareSchema:false}</str>
<str name="genericCoreNodeNames">${genericCoreNodeNames:true}</str>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:0}</int>
<int name="connTimeout">${connTimeout:0}</int>
</shardHandlerFactory>
</solr>
Finally, make the path /var/opt/solrcloud/ writable for tomcat6:
chown -R tomcat6 /var/opt/solrcloud/ chgrp -R tomcat6 /var/opt/solrcloud/
To deploy Solr with the YaCy configuration you must create a Tomcat Context fragment for each Solr instance. A Tomcat Context Fragment is a file in /var/lib/tomcat6/conf/Catalina/localhost. Therefore, we must create four files, one for each Solr server, in this directory: write a file to /var/lib/tomcat6/conf/Catalina/localhost/solr0.xml with the following content:
<?xml version="1.0" encoding="utf-8"?> <Context docBase="/opt/solr.war" debug="0" crossContext="true"> <Environment name="solr/home" type="java.lang.String" value="/var/opt/solrcloud/solr0" override="true"/> </Context>
and copy this to solr1.xml .. solr3.xml and patch the solr/home attribute to solr1 .. solr3. If you patch these files using emacs, make sure that you delete all files ending with '~' because they will cause an error. Finally, restart tomcat:
/etc/init.d/tomcat6 restart
Create the SolrCloud
We can now open the Solr web service at http://localhost:8080/solr0 Open this web page to check if the service is up and running. Then we can use that web service to instantiate the SolrCloud:
curl 'http://localhost:8080/solr0/admin/collections?action=CREATE&name=collection1&numShards=4&replicationFactor=1' curl 'http://localhost:8080/solr0/admin/collections?action=CREATE&name=webgraph&numShards=4&replicationFactor=1'
Assign the SolrCloud to YaCy
When the SolrCloud is ready and running, it can be assigned to YaCy as storage server. Open the servlet at http://localhost:8090/IndexFederated_p.html and select the flag "Use remote Solr server(s)". As server address, enter one of the Solr servers, like http://192.168.4.10:8080/solr0 Finally, uncheck the flag "Use deep-embedded local Solr".
