Read Operation
Level-two-leader runs as ordinary quorum leader which accepts level-one-leader as an observer.In level-one-leader, we combine three objects, a leader (run as normal leader in level-one cluster), a quorum peer (participant local quorum) and a observer. The observer is only used to connect to the level-two-leader, which ip address is read from configuration file at start up. The observer is running in its own thread.
The problem is, level-two-leader does not accepting the connection from level-one observer. The reason is when the observer start up, it uses its quorum peer function to initialize, which has a different epoch value accepted in level-one quorum by the leader (itself). It was different from level-two quorum, even if the observer's address info is in level-two configuration file.
Here is the error log from both leader:
2014-10-14 15:07:23,511 [myid:5] - INFO [LearnerHandler-/127.0.0.1:59655:LearnerHandler@330] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@29c69015
2014-10-14 15:07:23,514 [myid:5] - ERROR [LearnerHandler-/127.0.0.1:59655:LearnerHandler@633] - Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:357)
2014-10-14 15:07:23,515 [myid:5] - WARN [LearnerHandler-/127.0.0.1:59655:LearnerHandler@646] - ******* GOODBYE /127.0.0.1:59655 ********
2014-10-14 15:07:23,505 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:LevelOneLeader@35] - Level one leader created.
2014-10-14 15:07:23,505 [myid:2] - INFO [Thread-5:LevelOneLeader$1@41] - Starting Level one leader.
2014-10-14 15:07:23,506 [myid:2] - INFO [Thread-5:LevelOneLeader$1@43] - Observing /127.0.0.1:2225
2014-10-14 15:07:23,507 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Leader@944] - Have quorum of supporters, sids: [ 1,2 ]; starting up and setting last processed zxid: 0x900000000
2014-10-14 15:07:23,512 [myid:2] - WARN [Thread-5:LevelOneLeader$1@55] - Exception when observing the leader
java.io.IOException: Leaders epoch, 3 is less than accepted epoch, 9
at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:290)
at org.apache.zookeeper.server.quorum.LevelOneLeader$1.run(LevelOneLeader.java:46)
