Wiki Markup |
---|
The |
...
first completed implementation in the new refactorised scoreboarding code is a scheme for checkpointing and crash-recovery of the scoreboards and alignment of the scoreboarding with regular clock-hours/day-periods. Both of these improvements are discussed below: |
...
CheckPointing
...
*{+}CheckPointing{+}* In order to implement check-pointing, I save the state of the scoreboards to disk, while processing the scoreboards for each flow-file. When the program starts for the first time, it checks for checkpoints on disk before starting the scoreboading of every direction and if a checkpoint is present it unmarshalls the data from the checkpoint and starts updating the scoreboards from that point on. |
...
The first issue was how to save the state to disk. I decided to go with marshalling and saving the part of the [nested hash structure |http://users.telenet.be/jurgen.kobierczynski/jkflow/mylist.pdf] containing the current scoreboard to disk. Specifically, mylist \{direction\}\{$direction\} \{scoreboard\} \{aggregate\} \{report\} points to an array of nested hashes , one for each type of report defined (as shown highlighted in the thumbnail below). ( let $ref = mylist \{direction\}\{$direction\} \{scoreboard\} \{aggregate\} \{report\}\[$i\] for the $ith report type ). Then $ref->\{aggdata\}\{tuplevalues\} contains a nested hash containing the current counters for the flows/packets/bytes both inward and outward for a particular 'direction'(Such as ATLAS BNL to CERN' ) specified by the $direction variable in mylist |
...
...
.
!datastructure.JPG|thumbnail!
I create a temporary data strcuture ,each time the scoreboarding function is called in JKFlow.pm , and store the values of $ref->\{aggdata\}\{tuplevalues\} as well as certain other important variables representing the state of scoreboard including the $ref->{count} , $ref->{startperiod} and $ref->{counter} in that data strcuture. It is then marshalled and written to disk using the technique i explained on [this page|Marshalling and Unmarshalling complex data structures in perl]. |