...

11) Notify the data quality shifter that the run is ready to be reviewed.

Some Issues to Watch Out For

Overlapping runs in the same delivery

I've seen this happen exactly once but it caused some problems in the repipe.  If you have two runs that need to be reprocessed, and they both have parts in the same delivery, completely process the first one before starting the second one.  I had repiped both of the runs and copied all the event files over and then the processing of the first run removed the staging directory and the data for the second run was gone and I had to repipe it again.

Issues with Magic7 data

Sometimes, especially with runs that span three deliveries, an arbitrary assignment of event files to each delivery as indicated by step 7 above results in the corresponding magic7 data files not covering the time range covered by the event files.  This causes the doChunk substreams either at the beginning or end of the set to fail depending on where the data gap lies.  The magic7 data typically extends quite a ways before and after the time of the delivery so this usually isn't a problem but it occasionally pops up. There are two solutions.

On is to move event files around so that they are covered by the magic7 data for each delivery.  You could do this in advance by looking at roughly the chunk ranges in the originally processing and moving the appropriate files.  Or if you don't discover it until after the fact, moving things then.  However, in the latter case, many of the event files may have already been moved off the staging disk and that may require another repipe to set everything back up.

The other option is to merge in the magic7 data from the previous or later delivery (depending on where the problem is),  This is what I typically do as it then just requires rolling back the failed doChunk runs.  Simply grab the other magic7 data file from the run directory (the ones copied in step 6) into a temporary file, remove the overlapping data, and then concatenate the  two files in the right order and rename the resultant file properly (i.e. magic7_<download number>.txt) in the proper directory.  Then you can just roll back the failed processes and it will continue on.

Original e-mail

This is the text of the original email from Maria Elena that started the process of creating this page:

Ok, here's how to create empty halfPipe folders after Re-Piping:

...