Identifying TeV source candidates among Fermi-LAT unclassified blazars using Artificial Neural Network.

Very interesting and promising.

AGN group science coordinators comments on HBL_aa_VP_approved version below:

Dear AGN group science coordinators, thank you for your precise and careful revision of the paper. I will pay close attention to your comments and I'll  improve of the paper, with a new version.

Dear Graziano, please find below a few very minor editorial comments on the latest version (HBL_aa_MArev2_DJT1).

line 67: here you have catalog while elsewhere you use catalogue
line 91 - put Salvetti et al. out of parentheses
line 96, 107, 115 - same for Chiaro et al.
line 130: replace "BL Lacs and FSRQs" with "HSPs and non-HSPs"

Green light from AGN coordinators.

Thank you Marcello for your comments.

I'll adjust the text.

Dear authors, 

I'm reading v(HBL_aa_MArev2_DJT1) on behalf of the publication board. I think this is a nice result, I have a few comments on to help the clarity on some of the points. 

Overall: Figures 1-3 need to have larger legends and Fig 1 needs larger axis labels. 

19: "Observations by IACTs...": I think the important thing to emphasize here would be the importance of weighting targets since observation time is limited on IACTs in general. 

22: "The aim of this paper...": Is the unclassified blazar population here the unclassified Fermi-LAT blazar population? I would be specific. 

34: Are blazars the most powerful AGN? Or are they a type of AGN? 

50: It sounds here that it was the 3FHL that selected TeV-telescope candidates instead of the candidates being selected from the 3FHL

54: I think regardless of the units of the SED there are two broad humps for AGN. I would probably end the sentence at "broad humps"

76: One of the steps in the approach is to look for variability, but you don't state the expectation (or motivation)

89: I would put the references at the end of the sentence for readability. 

97: "extragalactic TeV-energy sources" - do you mean targets? (currently not TeV sources)

104: I would modify the title from "The search method" (which is quite generic) to specifically the search method you're using maybe "The AAN technique" or something like that

105: a perceptron is a term specifically for NNs. I would take an additional sentence or two to define what it is. Maybe even a figure.

109: Can you state the parameters you're using to train the algorithm?

115: In your training algorithm, did you also give it the option to classify something as "not an AGN". In the case that a pulsar accidentally got thrown in there?

129: Since Fig 1 is only demonstrating that the variability index isn't an important parameter - you can probably remove it (I think a figure of a generic NN technique would be much more useful).  

152: "cleanest" -> best?

Figure 4: I'd break this into multiple figures. The 3D looks really cool, but it's hard to see what's happening on the variability index axis. 

165: I'd be explicit here and say what exactly the algorithm is able to find

167: "We performed an analysis of Fermi-LAT data analyzing 104 months of Pass 8 data, from..." This is a really long sentence with a lot of ideas. you can probably condense it a bit here "We analyzed 104 months of Fermi-LAT Pass 8 data from....." 

197: It seems like the spectral indices for HSPs/LSPs and ISPs are consistent within uncertainties? 

205: a lot of the discussion of the EBL probably isn't needed here. The sentence in particular that state "The relevant part of the EBL..."

236: Conclusion -> Conclusions

Here are some comments for your consideration based on the PubBd version 4 of the paper.

38: citations for MAGIC, HESS, VERITAS, CTA are missing

47: please cite the LAT instrument paper (Atwood, W. B., Abdo, A. A., Ackermann, M., et al. 2009, ApJ, 697, 1071) instead of the 3FGL paper (which is introduced and cited at line 81)

121: I'd be curious to see some examples of the cumulative percentage as a function of flux. Are there any on confluence ? It's certainly beyond the scope of the paper, but it seems to me that looking at these curves for HSPs and LSPs might help you to understand why/how the quantiles variables are good variables to discriminate between them. Without having seen these figures, I'd guess that there might be a way to fit them with some functional form with a couple of parameters. I would not be surprised that feeding the ANN with these parameters would do a similar good job, and maybe would help to improve the performance.

129 and caption of Fig.1: we found the Variability Index to be, in our algorithm, an unimportant parameter. I think that this statement is not fully correct. What is correct is to say that using the Variability Index on top of the quantiles does not improve the performance. Fig.1 shows that it is not possible to use the Var Index alone as a discriminating variable, but Fig.4 shows that it can be useful when used with other variables.

135: remaining -> the remaining third (?)

Fig.2: which sample do you use (I assume it is the testing sample)? I see ~44 HSP entries while I expect 289/3=96 entries.

162: All the candidates... Personal comment: if the selected HSP candidates all lie in a clean HSP area and far from the LSPs, it seems to imply that you could define a clean selection based on these 3 variables and that your ANN approach is not much more powerful than a simple 3D cut in that space.

168: please cite the Pass 8 paper: Atwood, W., Albert, A., Baldini, L., et al. 2013, eConf C121028, 8, in Proc. 4th Fermi Symposium, Monterey

169: zmax cut ?

In the printout version, the bottom of Fig.1 and the top of fig.2 are missing. I don't know whether it is a sign of something wrong with the figures.

Thank you Philippe for your useful comments , I'll adjust the text and I'll post the new version.

Concerning your comment at line 121 please consider the following text.

Much of the proposed ANN method and examples of cumulative percentage as function of flux  in order to show how the adopted parameters are good discriminators are described in https://arxiv.org/pdf/1607.07822.pdf

The original idea was to compare the γ-ray light curve of the source under investigation with a template classified blazar class light curve, then measure the difference in a proper metric. Typically γ-ray AGN are characterized by fast flaring that could alter significantly the light curve and could make the comparison difficult. In addition, different flux levels could hide the actual similarity of light curves. As first approach of the ANN study we computed the Empirical Cumulative Distribution Function (ECDF) of the light curves. We constructed the percentage of time when a source was below a given flux by sorting the data in ascending order of flux in order to compare the ECDFs of uncertain blazars with the ECDFs of blazars whose class is already established. This was our variation of the Empirical Cumulative Distribution Function (ECDF) method. Fig. 1 shows the ECDF plots for 3FGL BL Lacs ( blue) and FSRQs ( red) . Fig.2 shows the significant overlap between the types where it is hard to distinguish individual objects, and there are outliers that extend beyond the range of the plots, but it is possible to recognize on the top left of the diagram a specific area where the overlap between BL Lac and FSRQ is minimal that , could lead to a first qualitative recognition of BL Lac objects.  Because the aim of the studies was to quantify the likelihood classifying an uncertain source we used the  a two-class  ANN  algorithm.

Following the procedure used for blazars we applied the algorithm to the blazar subclasses even if the ECDF results weren’t not as satisfying as for the blazars. However a clean area where to apply the ANN was found as described in the paper. Fig. 3 shows the ECDF for blazar subclasses, where  the HSP( black)  / FSRQ( red)  separation is evident, but when we considered  the blazar subclasses the HSP / LSP separation was also evident but the ISP contamination was not negligible. However applying the ANN the results  proposed in this paper are consistent and acceptable. Even if it could be possible to consider different parameters than flux variability  the aim of this and previous studies is to characterize the sources by their γ-ray light curve.  However, experimentation with new parameters continues to verify the possibility to obtain new specific algorithms applicable to the new Fermi AGN catalogs.

Best regards

Graziano

Fig.1

Fig.2

Fig.3

Philippe,

about your comment for :

_ Fig.2 _ Yes , that's the plot of the testing sample optimizing the ANN algorithm

• Fig.4

Your are right when you comment that it  could define a clean selection based on these 3 variables , but please conside that our algorithm is a two parameter / two layer algorithm and we could not have used three parameters

Best regards

Graziano

Hi Philippe,

I talked with David Salvetti who made the plot in Fig.2 and he confirms that 1/3 of the total sample was in the plot. I don't have the full list of sources  and related likelihood  right now on my notebook, but however the aim of the plot is to show that there are much more LSP/ISP than HSP and these latter sources has no clean peak  in order to show also in the ANN plot the ISP ( and LSP) contamination that we saw in the ECDF plots.

I tried to count the entries, looking by eyes the plot. It seems to more than 40 sources( ~ 60? ) but I think that the plot could be fine for the aim of this study. It could be possible to repeat the plot but it means to analyze once more the whole sample and this is not a fast  work. Is it okay for you?

Best

Graziano

No, I haven't because we considered HSP and non-HSP sources ( ISP +LSP). We have separate  colours in the ECDF plot as I posted above in Fig. 3. Also plot red/black in Fig.3 is interesting if you consider that FSRQs are mainly LSP objects.

No, I'm sorry  Philippe, we haven't any separate diagram because owe used a the two class algorithm that considers the likelihood  A = 1-B  then it returns an histogram with both the likelihood as you see here in Fig.2 . The most interesting result was to compare the two " classes" together as did for BL Lac vs FSRQs  likelihood. However the two selection are easy readable in the histogram.

Thank you Philippe of your comment.  Let me talk with my colleagues and I'll answer to you as soon is possible.

Hi Philippe,

All the commented minors  have been reviewed and adjusted.

According with your suggestion, in order not to creare confusion in the reader, Fig .4 and the relative text were deleted because they did not add any extra  value to the ANN result.

Moreover I reviewed all the ANN data and I can answer your comment

“… Regarding fig.2, as I said, I expect 289/3=96 HSPs but it seems that there are only ~44 HSPs. Am I missing something ? ”

A mistake was made writing the text of the paper you reviewed. Instead of testing test, training test was written. The correct wording is testing test. Therefore  the percent of samples used  for the testing sample is different. The testing test accounts for 15% of the total sample. In this way Fig. 2 is correct (~ 43 HSPs and ~ 123 non-HSPs). The  relating text  has been re-written  and highlighted in blue in the new draft posted in PubL Board. 

Graziano

ok, thanks .

Done.

Hi Graziano, 

It looks like you've addressed Philippe's comments with the newest upload to the pubboard. Can you confirm that you took into account my comments? 

Thanks!

Best, 
Regina  

Hi Regina,

sorry for late reply.  I reviewed the paper considering also your precious comments. A 2LP ANN schematic figure has been added to the paper. About your comment concerning a pulsar contamination of the sample, it remains free of pulsar because the 559 UCSagn  objects came from a previous Pablo Saz Parkinson study (Saz Parkinson et al. 2016 https://arxiv.org/abs/1602.00385) where Pablo applied a number of statistical and machine learning techniques to classify and rank unclassified gamma-ray sources in 3FGL  as UCSagn and UCSpsr.

Best

Graziano

Dear co-Authors and Publ Board.

I received the referee report and I post it here.

Referee report


Main comments:

The manuscript with title “Identifying TeV source candidates among Fermi-LAT unclassified blazars using Artificial Neural Networks” by Chiaro et al. reports the results of using artificial neural networks (ANN) to identify promising High Synchrotron Peak (HSP) candidates among the 573 3FGL blazar candidates of uncertain type (BCU) and among 559 3FGL unidentified sources likely to be blazars. It also reports the analysis of 104 months of Fermi data for the 80 selected HSP candidates and the comparison of corresponding extrapolated spectra with the sensitivity curves of present and future IACTs.

Finding new TeV blazar candidates (not only HSPs) is important, not only for the study of acceleration/radiation mechanisms in AGN, but also to make possible blazar population studies at these energies. As very long exposures are often needed to detect TeV emission, to provide such a list of TeV HSP candidates is of interest to the community. The Fermi analysis work performed by the authors on these 80 HSP candidates also deserves publication.

However, the manuscript cannot be recommended for publication in its current form.

The main issues are:

•       The ANN search method presented in Section 2 seems to be exactly the same as the one published in Chiaro et al. 2016 (see sections 3 and in particular 3.3 in Chiaro et al. 2016). It is presented here (L. 116) as an optimized version of the one in Chiaro et al. 2016, but it seems that no changes have been made. Indeed, the distribution of L_HSP in the testing sample and the thresholds used to select HSPs are exactly the same in both papers. In particular, figure 3 in the present paper has already been published as figure 7 in Chiaro et al. 2016. Surprisingly, the results of the application of the ANN algorithm to the same selected 573 BCUs differ between both papers (48 HSP candidates are found here with L_HSP>0.8 and 11 with L_HSP>0.891, whereas 53 with L_HSP>0.8 and 15 with L_HSP>0.891 are published in Chiaro et al. 2016). This must be clarified.


•       The presentation of the ANN method and its limits are not detailed enough, neither in the present paper nor in that of 2016 (for HSPs):

o       In particular, the authors have to handle with small datasets, which imply that the variance of classifiers corresponding to random training/validation/test splittings is likely to be important. Consequently, a single random training/validation/test splitting will provide a performance that could be significantly misestimated. An illustration of the fragility of the ANN performance presented in the paper is visible in figure 3, with only 1 or 2 true HSPs (and 0 or 1 non-HSP) in the testing sample passing the high confidence HSP cut (L_HSP>0.891). Authors should discuss this. For optimization purposes, in particular when dealing with small datasets, 10-fold cross-validation is a standard that has been previously used in similar works (train the classification method on 9/10ths of the training sample and apply it on the remaining 1/10th, then repeat ten times until the whole training sample has been tested on once, and derive performance metrics from there). Using 70% of the!
 

3LAC HSps and ISP-LSPs for 10-fold cross-validation would allow using the remaining 30% as testing sample (instead of 15%). It would be interesting also to use receiver operating characteristic (ROC) curves (see, e.g., Fawcett, T, Pattern Recognition Letters, 27 (2006) 861-874) to check the stability/variance of performance due to the small samples size (see how Lefaucheur et al. 2017 A&A 602 86 dealt with this problem).

o       It is difficult to understand why only variability information is used to build classifiers for HSP / non-HSP identification. It has been shown that additional separation power between BLLs and FSRQs (and so between HSPs and FSRQs, as most HSPs are BLLs) is carried out by other parameters provided by the 3FGL catalogue (see Ackermann 2015 for the 3LAC spectral properties, and similar works dedicated to the separation of different type of blazars: Hassan et al. 2013 MNRAS 428, 220–225; Lefaucheur et al. 2017 A&A 602 86; etc.).

o       It is obvious that the choice of building an ANN method on flux-dependent information can imply that ANN performance is also flux dependent. Did the authors compare the flux distribution for HPSs, ISP-LSPs, BCUs and UCSagn? How the upper-limits in the month light-curves were treated?

o       Other authors (see Lefaucheur et al. 2017 A&A 602 86) have shown the importance of taking into account the analysis flags in 3FLG catalogue to reach a proper derivation of classifiers performance. Maybe this issue is less important here, where classifiers are built only on flux variability, but this should be evaluated. What are the fraction of sources with analysis flag for HPSs, ISP-LSPs, BCUs and UCSagn?

o       The estimation of the number of false identifications among the HSP candidates is missing. Please give it explicitly. The expected number of false identifications should be considered separately for HSP candidates from the BCU and from the UCSagn samples. This is particularly delicate in the case of UCSagn because of how the sample of 559 sources has been obtained (from Saz Parkinson et al. 2016 and not from Salvetti et al. 2017 as reported in the present paper). Indeed, beyond the fact that Saz Parkinson et al. 2016 didn't provide an estimation of the number of false identifications, their classifiers were built to identify sources as pulsars or AGN not taking into account the presence of SNR and PWNs close to the Galactic plane. They didn't take into account the effect of flagged sources on their classifiers performance, whereas 3FGL low galactic latitude sources are dominated by flagged sources. For these reasons, it is likely/possible that there is a significant amount!
  of false identifications among the low galactic latitude blazar candidates of Saz Parkinson et al. 2016. I recommend to limit the present study to the 481 (instead of 559) Saz Parkison et al. 2016 blazar candidates with galactic latitude above 10° (|b|>10°).

o       Do the authors understand why in Fig. 3 there is not a peak at L_HSP~1 ? Please add a comment.

•       This paper reports the analysis of 104 months of Fermi data for the 80 selected HSP candidates. Therefore, the results of these analyses should be reported in Tables 1 and 2 in a much more complete manner: spectral index with its error, pivot energy, flux at the pivot energy with its error, etc., should be provided. These results are of interest as they could be used for different physics cases in the gamma-ray community.

•       It seems that Fermi spectra are obtained for an energy range starting at 100 MeV. As shown in the 3FHL paper, the average difference of photon indexes for spectra starting at 100 MeV and spectra starting at 10 GeV is about 0.3-0.4 (for redshifts in the 0-0.5 range). As spectra starting at 100 MeV tend to be harder than spectra starting at 10 GeV, their extrapolation at higher energies will tend to be overoptimistic for detections with IACTs. Could the authors make a comment about that? 

•       There is an issue in the comparison of Fermi extrapolated spectra with CTA sensibility. It makes sense to consider the observation with the northern (southern) array of sources with declination above (below) 0°. But as the CTA sensitivity varies with the observation zenith angle, the CTA consortium provides sensitivity curves at 20° and 40° zenith angles for each array. Considering the latitude of the arrays (+28° for the northern array, -24° for the southern array), the authors should select for each source the more appropriate sensitivity curve and explain it in the text (L213-214). Figure 5 shows that, for a given array, the authors used the same sensitivity curve whatever the source declination. The HESS, MAGIC and VERITAS columns in tables 1 and 2, and the zmax columns in table 3 must be updated correspondingly.

•       Finally, the abstract, a part of section 1 (second part of the third paragraph) and almost all the section 2 are far to be in an acceptable shape and need to be re-read by all the authors (obviously this was not done). In addition, section 2 must indicate that the description of the ANN method is already given in Chiaro et al. 2016 and that there is no further optimization here. 

If the authors address the above-mentioned issues, the resulting HSP-candidates list and the analysis work performed on Fermi data are of interest for the community and would deserve publication.

Additional (less important) comments are listed below.


Other comments:

Abstract

L16: replace "are the main targets" with something like "are among the main targets". 

L23: CTA is also an IACT. Replace by “the present IACTs or the future CTA”?

L22-26: it is a vague summary of the work done in the paper. Please improve it (many key-words missing: HSP, 3FGL, BCUs + blazar-like unidentified sources, 104 month of Fermi data) and be more precise on the results. 

L28-30: weak affirmations => to be removed (otherwise please provide a justification in the text).

Section 1: Introduction

L34: The reference Abdo et al. 2010 refers to the paper “Suzaku Observations of Luminous Quasars: Revealing the Nature of High-Energy Blazar Emission in Low-level activity States”. Is it an error?

L48: Please explain why the 3FHL catalogue is interesting to searching for TeV blazar candidates (and fix the typo).

L48-49: Is this sentence referring to the 3FHL paper also?

L50: "alternative". I don't really see where is the alternative. In all cases it is necessary somehow to compare the spectral properties in the gamma regime with IACTs typical capabilities.

L59: The Abdo et al. 2010 reference is not the correct one. If the authors are referring to the 1LAC paper, please fix de reference.

L61-73: The end of this paragraph is not well written in its current form. A few examples: not clear in L61-63 if the work of these SEDders is for 3LAC or not; the sentences about TeVCat do not clearly illustrate that HSPs are the most numerous sources in the TeV extragalactic sky – the authors could try: “Indeed, in the TeVCat interactive catalogue for TeV sources (Horan et al. 2008), among XXX extragalactic sources XXX are HSP, while...” etc.; the link between the part about 2WHSP and previous lines is not obvious. Please clarify.

L65: The VHE regime is usually defined as gammas above 100 GeV, as in Chiaro et al. 2016... the definition used in the TeVCAt proceeding is not usual.

L66-67: TeVCat numbers are not up to date. Please fix this. There is something wrong about LSP/ISP and FSRQs. Note that in TeVCat, the LSP/ISP classifications are not used, but rather LBL/IBL. I see in TeVCAT 49 HBL, 8 IBL, 2 LBL and 7 FSRQs. There is a typo in footnote 7 (missing "of")

L74: Please replace "TeV source" by "TeV HSP"

L74-76: Please be more precise. The HSP candidates are not searched for among all the unclassified Fermi sources; the first part of the work is performed on the basis of 3FGL information.

L80: Is it 3033 or 3034?

L87: Space missing before "("

L88: Typo: Leufaucheur => Lefaucheur

L89: Not Salvetti et al. 2017, but rather Saz Parkinson (2016) !

L91-92: “We focus on […]”. The authors should cite previous similar works: Hassan et al. 2013 on the 2FGL catalogue, Lefaucheur et al. 2017 on the 3FGL catalogue.


Section 2: the ANN search method

Fig. 1: Too general and not very informative, already published in Chiaro et al. 2016. Please remove.

L104: Reference required for such a statement.

L107-145: The content of this section has already been published in Chiaro et al. 2016. The version proposed here is much less well-written (typos, repetitions, sentences difficult to understand, etc.). Please see the “Main comments” section.

Fig. 2. The figure does not bring interesting information. Despite the important overlap between the distributions, discrimination power of this parameter could be hidden in the correlations with other parameters. In addition, it is preferable to avoid the "direct use of variables correlated with the overall flux" (see Doert and Errando 2014). Anyway, the distributions should be normalized, and labels and units are too small. 

L131-132 and elsewhere: It’s worth to indicate that by non-HSPs sample the authors mean ISPs and LSPs (from 3LAC and not from 3FGL, I guess).

Fig. 3: Fix the typos. Please specify that non-HSP sources are ISP and LSP in the 3LAC catalogue.


Section 3: Identifying HSP candidates

L147: Remove “optimized”

Fig. 4: In both figures, please fix the thresholds names: “HC HSP threshold” instead of “VHC HSP”; “HSP threshold” instead of “HC HSP”;

L149-150: A comment about the absence of peak at L_HSP ~1 would be nice here.

L153: In section 2, a threshold is defined corresponding to a precision of 90% (true positives/(true positives + false positives)). In section 3 it is mentioned that the L_HSP>0.891 threshold corresponds to a true positive ratio (true positives/positives) of 90%. Are the authors referring to the same threshold? Please clarify.

Table 1 and 2: 
Please specify what “observability at IACT sites” means. What observation time is considered? A column for CTA observability is missing.
As the HC cut is at L_SHP>0.891, please add a digit in the L_HSP column and also in the captions. In table 2, the 3 sources with L_HSP=0.89 are they all below 0.891 as suggested by the horizontal line? 
In table 2, what do the symbols before some of the 3FGL names mean?


Section 4: Fermi-LAT spectral analysis

L159: typo: from from

L176: The normalization and slopes of the IEM and isotropic templates were allowed to vary during the fit. Could the authors make a comment about possible deviations with respect to the starting values?

L188-189: Where the reported values come from, in particular for LSPs and ISPs? Have the authors performed also an analysis of LSPs and ISPs for the same period? Please clarify.

Section 5: TeV candidates

L203-204: note that z=0.022 is reported for the source 3FGL J0153.4+7114 (seen in  Simbad, I didn’t check its reliability).

L204: Especially true for HSPs, not for all types of BL Lacs (see fig. 12 in the 3LAC paper)

L205: “these z values” => “these two z values”

L213-214: see the “Main comments” section.

Figures 5 and 6: far too small. 


Section 6: Conclusions

L236-238: I don’t understand this “the very similar values of the synchrotron peaks of the three blazar subclasses”… the synchrotron peaks positions are spread over orders of magnitude and are, by definition, different for the different subclasses. Please rephrase.

General:
Remove spaces before footnotes

Comments and suggestions from Sara - reading version “HBL_paper-4.pdf” as AGN coord.

L35

“requires around 50 hours of observation time.”

— “requires around 50 hours of observation time to be significantly detected.”

(if this is what you mean here)

L46 In general, the historically older reference should be preferred for citation. When introducing the IC emission to explain blazar’s emission, a web-conference by Ghisellini in 2013 is not the first reference that I would think about. Better references may be, e.g.:

Sikora et al 1994 ApJ, 421, 153

königl 1981

Marscher and Gear 1985

Ghisellini and Maraschi 1989

L51

“This catalog reports 223 TeV sources, where 51 of them are HSPs and only 10 are LSP/ISP flat-spectrum radio quasars (FSRQs)”

— This catalog reports 223 TeV sources. Among the 61 objects associated with blazars, 51 of them are HSPs and only 10 are LSP/ISP flat-spectrum radio quasars (FSRQs)”

L53

“which is an independent”

— It’s not clear to me what does “independent” mean here, “independent” from what?

L54

“away from the Galactic Plane.”

— “away” is generic, the authors likely report a selection criteria in their paper, may be best to report it here as well: “at high latitude, i.e., |b| > XX”

L50 paragraph

I would not call the TeVCat an “approach”, as it is imply later in L55 (“Here we present a third approach”).

TeVCat is simply an online resource that collects information about Cherenkov-detected objects.

Please rephrase these parts of the text.

L103

“passing standard data quality selection criteria”

There are many different “standard” cuts for the LAT analysis. For instance, a GRB LAT standard analysis differs quite a lot from a blazar or pulsar standard one.  A reader should be able to reproduce the results of the paper, please report the criteria that were applied to the dataset. Mention how was the Sun and GRB emission accounted for, what was the zenith angle? etc

L107

“This is the model routinely used in Pass 8 analyses.”

— .. model recommended for use with Pass 8 ..

this sentence may be placed at the end of the paragraph, i.e., after the isotropic template sentence.

L115

“8 energy bins per decade”

How was the SED produced? if you used fermipy, it is recommended that the SED energy bins must align with the analysis bins. The SED plots show 13 bins total (0.1-1000GeV), but your analysis section specifies that 8 bins per decade were used. Assuming that the SED was produced with fermipy as well, what procedure was followed here?

L125 

“selecting IACT or CTA candidates”

— selecting IACT and CTA candidates

L132

“The relevant part of the EBL “

— maybe rephrase as “To the aims of our work, the most relevant part of the EBL ..” or delete “relevant”

L139

“while recognizing “

— while acknowledging

L119 

“We then analyzed each of our HSP candidates, assuming a power-law spectrum “

and later in L142 

and Fig 2-3

“ we extrapolate the best-fit spectra obtained with the Fermi analysis”

— from my understanding from the limited information about the LAT analysis, the procedure that was followed in the LAT analysis is assuming a PWL spectrum for all sources of interest (motivated by the 3FGL best-fit spectrum), and fit the PWL spectral parameters.

— Some gamma-ray spectra are likely curved, i.e., they may be best fit with a LogP rather than a PWL. See e.g., 3FGL J0506.9-5435 where the PWL overshoots the upper limits at both low and high energy. If you are using more integration time with respect to the 3FGL (7 years instead of 4 years), besides looking for new detectable sources in your ROIs, you may also want to check whether the best-fit spectrum found in 3FGL is still valid also for your higher-statistic dataset. 

In fact, one of the outcome of the 4FGL work is that essentially all bright blazars display a curved best-fit spectrum. This suggests that increasing the statistics, spectra may be significantly curved.  All in all, the 8-years 4FGL is not that far from your 7-years data sample. A first step would be at least check what is the best-fit spectral shape in the 4FGL.

— If this is the case, the way that the extrapolation is currently derived in the paper seems not realistic. 

Fig. 3

Some of the SED plots are unexpected to me. In a given gamma-ray spectrum, I would expect that the SED energy-bins would be distributed somewhat below and above the best fit PWL. In most of the SED plots, these are systematically all above the best-fit. See for instance: 3FGL J1549.9-3044, 3FGL J2300.0+4053 and 3FGL J 2145.5+1007.

What is the explanation for this, may you please double-check it? Also, it may be worth using a larger energy binning for the SED, given that in the current plots ~half of the bins are UL. If you are using fermipy, see the previous note.

I could not find several references recalled in the text. Please check them all. Just for example:

Angioni R. , Grandi P., Torresi E. et al., 2017, APh, 92,42

Pita S. , 2017, AIPC, 1792,25

Rumelhart, D. E., Hinton, G. E., Williams, R. J. 1986, Nature

Bishop C.M., Neural Network for Pattern Recognition, 1995

Carosi A. , 2015, Proceeding of the 34th International Cosmic Ray Conference, p. 5

Wilks S.S., 1938, AMS, 9, 60

Comments from the CTA Collaboration (since Graziano and Manuel are members and this references CTA, they have to review the paper, too):

Dear Chiaro, dear Manuel,

Thank you for sending us your paper. Although there is some overlap with the ongoing Consortium publication, we believe that these tensions could largely be mitigated following the suggestions indicated in the end of this message.

Once you have addressed these comments, please submit the paper to SAPO as a non-consortium one. Please feel free to include our suggestions and your replies as a comment on the dedicated page so that SAPO members can keep tracks of this exchange.

As a final note, we hope the candidate list you have established can benefit to the Consortium paper once your work is published.

Thank you for your work and all the best,
Jonathan and Pepa


-- Tensions with the consortium paper ---

Three general comments have emerged, which may be addressed following the suggestions below. 

The first comment is that a non-careful reader may be under the impression that CTA will only be able to identify a few new HSP sources. It should be made clear in the abstract and conclusion that the current extrapolations are only based on the objects currently detected by LAT and do not account for any possible sub-threshold population.

The second comment regards the emphasis put on CTA in Sec. 5 and in conclusion. Current IACTs would likely be interested in targeting such objects and it would be better to explicitly discuss the estimated detectability by HESS/MAGIC/VERITAS, rather than putting the entire focus on the sensitivity of CTA.

The last comment concerns the use of sensitivity curves. Such curves provide 5sigma detectability in differential energy bins and can act as a guide to guestimate the overall detectability of a source given an observation time. They cannot be used on their own to quantitatively determine the overall significance of a source (nor a precise observation time for detection, or a maximum redshift up to which the source is detectable). This is an important caveat, which should be reminded to the reader in Sec. 5 and in conclusion.

- Abstract -

"search for unclassified gamma-ray blazars/sources that are likely detectable with IACTs or CTA" -> "search for unclassified blazars among known gamma-ray sources that are likely detectable with IACTs or CTA"

- Sec. 3 -

"which are much more numerous than HSPs" -> "which are much more numerous than HSPs at energies covered by Fermi-LAT"

- Sec. 5 -

"compare the extrapolated flux with the CTA sensitivity for 50 hours (5 hours)":
- please provide a reference for the sensitivity curves
- please indicate in the captions of Fig. 2 and Fig. 3 that you are using different sensitivities for CTA North and CTA South if that is the case
- please consider using a dark gray band encompassing HESS/VERITAS/MAGIC sensitivities for 50h in Fig. 2 and Fig. 3 rather than the 5h sensitivity of CTA 

"In Table 3, we report the maximum redshift value of the TeV candidates so that the sources are still detectable at 5sigma for 5- and 50-hour CTA observations.": please consider softening this statement (e.g. "we report an estimate of"). Please also consider rounding the numbers indicated in Table 3. Given the extrapolation method (intrinsic power law + no uncertainty on spectral parameters accounted for + match of the sensitivity curve), it may be hard to provide such redshift values with a +/- 0.01 precision.

"If no value is given, the source will not be significantly detected with the assumed observation time at any redshift": given the caveats indicated above, please consider dropping or seriously softening this statement (e.g. adding "under the simplifying assumptions used here").

A paragraph dedicated to discussing the detectability of the objects by current-generation IACTs would be in order.

- Sec. 6 - 

"targets for CTA for population studies." -> "targets for CTA."

A sentence or two dedicated to discussing the detectability of the objects by current-generation IACTs would be in order.



-- Suggestions of possible improvements --

- Title -
A more specific title may help the reader, e.g. "Identifying TeV candidates with high-synchrotron peak among Fermi-LAT unclassified blazars"

- Sec. 2 -
The  ANN approach is one of the interesting point in this work, it may be worth providing a somewhat more extensive summary of the approach used in Chiaro+ 2016.

- Sec. 5, incl. Fig. 2 and 3 -
Uncertainties on the LAT flux of the candidates may strongly affect the conclusions about the specific objects actually detectable by current or future IACTs. It would be worth, possibly in the longer run, studying how the uncertainties on the flux/index determined with Fermi-LAT affect the selection procedure discussed in Sec. 5.

Here are comments on the May17_2 version after reading for the publication board.

I'll start with what I think is the most major comment.  When extrapolating the spectra to TeV, no consideration is taken for the statistical uncertainty of the LAT fit.  Without having the exact numbers in front of me I would guess that at least for many of the curved sources you will get a detection to higher redshift if you account for the statistical uncertainty of the extrapolation.  You should at least comment on this, but best would be if you could estimate the 68% statistical uncertainty on the redshift estimation.

Figures 2 and 3:

• The CTA band looks more purple than blue to my eyes.  But I have been known to not see colours correctly.
• Why have the LAT data different colour each time?  It makes the plots too colorful in my opinion and harder to read.
• The legend overlaps the data in most cases.  If you have all the data same color you could add the legend in the empty space caused by the prime number of candidates.  In that case I would add the sensitivity curves to the legend.
• The axis labels are too large.  Please make them about the same size as the text.
• For consistency, have three columns in Figure 3 as well.

Line 22: Fermi -> Fermi-LAT

Line 74: Section 6 is called conclusion so calling it discussion here is strange.

Line 119: "Our model includes"  Here you change tense from past to present.

Line 129-130:  Where do these values come from?  There are no ISPs or LSPs in Table 1 and Table 2 which are under discussion.

Line 138: "and focus ... VHC candidates".  This is repetition and can be removed.  You could also rephrase the first sentence.

Line 149: "We used"  Here you change tense from present to past.

Line 153: "Fermi" -> Fermi-LAT

Line 166: remove "indeed"

Line 170: Do you know of any observations of these sources?  It would be nice if there are some, but not critical.

Line 175: remove "by wishing"

Acknowledgements are usually not a section.

Table 1:  Can you use a symbol instead of italicized "pos"?

Gulli,

Good comments.  Thank you.  Here are responses:

Here are comments on the May17_2 version after reading for the publication board.

I'll start with what I think is the most major comment.  When extrapolating the spectra to TeV, no consideration is taken for the statistical uncertainty of the LAT fit.  Without having the exact numbers in front of me I would guess that at least for many of the curved sources you will get a detection to higher redshift if you account for the statistical uncertainty of the extrapolation.  You should at least comment on this, but best would be if you could estimate the 68% statistical uncertainty on the redshift estimation.

UPDATED.  TABLE 3 INCLUDES THESE UNCERTAINTIES. 

Figures 2 and 3:

• The CTA band looks more purple than blue to my eyes.  But I have been known to not see colours correctly.
• Why have the LAT data different colour each time?  It makes the plots too colorful in my opinion and harder to read.
• The legend overlaps the data in most cases.  If you have all the data same color you could add the legend in the empty space caused by the prime number of candidates.  In that case I would add the sensitivity curves to the legend.
• The axis labels are too large.  Please make them about the same size as the text.
• For consistency, have three columns in Figure 3 as well.

UPDATED. 

Line 22: Fermi -> Fermi-LAT

DONE

Line 74: Section 6 is called conclusion so calling it discussion here is strange.

CHANGED

Line 119: "Our model includes"  Here you change tense from past to present.

CHANGED

Line 129-130:  Where do these values come from?  There are no ISPs or LSPs in Table 1 and Table 2 which are under discussion.

ADDED REFERENCE TO 3LAC

Line 138: "and focus ... VHC candidates".  This is repetition and can be removed.  You could also rephrase the first sentence.

DONE

Line 149: "We used"  Here you change tense from present to past.

CHANGED

Line 153: "Fermi" -> Fermi-LAT

CHANGED

Line 166: remove "indeed"

DONE

Line 170: Do you know of any observations of these sources?  It would be nice if there are some, but not critical.

NO OBSERVATIONS THAT WE COULD FIND.

Line 175: remove "by wishing"

DONE

Acknowledgements are usually not a section.

CHANGED

Table 1:  Can you use a symbol instead of italicized "pos"?

DONE. 

More comments from the CTA Collaboration:

Here are some comments for your consideration on the PubBd version 13 of the paper.

I have a general comment/question: did you consider dropping the Fermi-LAT analysis of Section 4 and using directly the 4FGL results ? This option would make your paper a cat 3 paper.

90: what is the difference with Chiaro+2016 ? It seems you use the same algorithm and the same sample (289+824).

94: is it the only difference with Chiaro+2016 ?

100: in fig.7 of Chiaro+2016 I don't see any peak at 1 for the HSP testing sample. So I don't understand why one would expect a peak in your fig.1.

112: this is the model...: this statement is true only if you specify that you use this diffuse emission model along with FL8Y.

119: I'm confused: your candidates are 3FGL sources but you use FL8Y to build your model. So I guess that you had to remove the FL8Y source corresponding to your 3FGL source from the model (when there is one). You should say it in the paper. What is the fraction of your candidates that are in FL8Y ? I'm curious: why didn't you use the FL8Y results in the first place ?

120: you have a slope parameter for the isotropic template ?

134: These results compare nicely...: do you refer to the number of sources with a curved spectrum or to the spectral indices in general ? What does "nicely" means in this context ? Do you mean that your spectral indices are compatible with the 4FGL ones within 1 sigma ? Looking to J0153.4, I see that FL8Y and 4FGL both report a power-law spectrum.

Section 5:

Extrapolating the Fermi-preferred spectral form to the CTA energy range is one thing, but it only gives one possible flux value in the CTA energy range. It would be more useful to provide a flux interval, using an optimistic spectral form (power law) and a pessimistic spectral form (log parabola). In the case of J0153.4 for instance, you find that a log parabola is preferred but a power-law fit would not look bad (and FL8Y and 4FGL both prefer a power law). Obviously, your extrapolated flux would be very different in the power-law case.

Comments from the second CTA reviewer:

I have read carefully this paper by Chiaro et al. on "Identifying TeV source candidates among Fermi-LAT unclassified sources". I think that the paper is interesting and helps on improving the neural network techniques used in previous works. I have some recommendations mostly focused in the text and presentation.

Title: I would say "unclassified blazars" instead of "unclassified sources"

Abstract:

• avoid using "Observations", "observing", and "observation" three times in the same sentence.

• move "from the third Fermi point source catalog" to right after "known gamma-ray sources"; if you do this, then remove "selected", maybe you also need to split the sentence into two, so it doesn't become too long. Maybe, it needs a bit of rewritting but right now I think it goes back and forth with the same idea.

L32-33 remove "the existing" and "facilities"

L35 it seems that the high background is what requires fairly long observations, I would rewrite

L36 "significant detection" -> "detection at 5$\sigma$"?

L39 I would use "peaks" instead of "humps", it sounds more formal to me.

L39 "low-energy hump" -> "lower-energy peak"

L40 due to -> produced by

L41 peak of the synchrotron hump -> synchrotron peak

L48 The all-sky -> All-sky

L48 on the -> on board the

L52 use "photon index" instead of "spectral index" to homogenize the text and tables

L65 UCSs, I'm not sure where this acronym comes from

L71 maybe start a new paragraph from "The paper ..."; before this I think it can be good to mention science cases that can benefit from the detection of HSPs. These can be extragalacic background light and gamma-ray background studies (e.g. Dominguez & Ajello 2015, Ackermann et al. 2015, respectively), connection with neutrinos (TXS paper), ...

L82 maybe write "corresponding to increments of 10% from the 10th to the 100th percetile"? It doesn't read very well by numerating each 10ths.

L87 to belong -> belong? I'm not sure I understand this sentence

L91 I don't think 3LAC has been mentioned before, it should be explained and cited

L99 distributions show -> distributions plotted in Figure 1 show

L100 remove "(Fig. 1)"

L125 maybe double check, just to be on the safe side, your spectral form results with the now available 4FGL?

L142-143 remove "For the aims of our work, the most relevant part of"

L152 it is not clear if you are using an uniform probability for the redshift between that redshift range, I think that a much better approach can be to randomly select a redshift following the redshift distribution of HSP with known redshifts

L153 {\it Fermi} -> {\it Fermi}-LAT

L153 paragraph, maybe it is personal taste but I would leave all the lines and colors descriptions and references for the captions.

L170 it is pretty surprising your claim that "most sources should already be detectable with the present-generation of IACTs", I actually thought, in reasonable observation exposures, the opposite.

L175 "wishing" doesn't sound good to me, maybe "necessity" or other similar word?

L177 necessitate -> require

Table 1 (I think you should merge Table 1 and 2, Table 2 is just an extension of Table 1)

• I would change "3FGL name" for "Source Name", the "3FGL" tag is already in the name; I would also justify this column to the left

• use "$-$" instead of only "-" for the source name with negative declination, this is the standard procedure in catalogs

• the column 7 is "Integrated flux"? maybe clarify this explicitly

• in the caption, I would explicitly say what is the "L_{HSP}" column

Table 3

• I would change "3FGL name" for "Source Name", the "3FGL" tag is already in the name

Figure 2

• the legends are on top and hidding some data. I would probably remove all the legends or just leave it in one of the plots. However, I would make sure the lines are described in the caption, which I think they aren't.

• if you leave some legend, I would write {\it Fermi}-LAT instead of only {\it Fermi} (this recommendation is very minor)

Finally, I think that I would change the short title (appearing in odd pages) from "Hunting IACT candidates" to a shorter version of the longer title. Right now shorter and longer title are very different.

Great work, congratulations.
Alberto

Comments from the first ApJ reviewer :

Referee Report
Reviewer's Comments:
The identification of the gamma-ray sources that can be detected at TeV energies with the current and future slate
of Imaging Atmospheric Cherenkov Telescopes (IACTs) is an important task. The development of efficient methods
to select such TeV targets from the catalog of sources observed by the Fermi LAT instrument is a fundamental task
to maximize the scientific output of both the Fermi and IACTs telescopes and, as a consequence, improve our
understanding of the extremely energetic emission from blazars.

The manuscript "Identifying TeV Source Candidates among Fermi-LAT Unclassified Sources" presents a method
to select potential TeV targets observable by IACTS based on the variability and spectral properties of gamma-ray
sources observed by Fermi LAT instrument. A set of candidate high-synchrotron-peaked (HSPs), the spectral class
of blazars most likely to emit at TeV energies, is selected based on their reported variability by applying an already
tested machine learning technique. Among these candidates, sources which can be detected by IACTs under
reasonable assumption on the duration of the exposures are further identified by comparing their spectral energy
distributions extrapolated to TeV energies with the sensitivity curves for current and future IACTs.
The extrapolation of the SEDs of such sources are based on fitted spectral models of the candidate HSPs sources
from Fermi LAT data collected until April 2017, at the TeV energy

The paper represents a valuable contribution to the literature in this field and deserves publication once the questions and
comments below will be addressed. The manuscript is well written and clear. I would like to see the revised manuscript.


Comments

- The authors should explain why they decided to perform the extrapolation of the SED for high-confidence HSPs sources
only, i.e. with L_HSP>=0.89. Based on the estimated purity of their classification based on the validation of their machine
learning method reported at the end of Section 2, a lower L_HSPs>0.8 threshold will still produce a 75% efficiency that
would probably yield a quite large number of HSPs detectable by IACTs under the same assumptions used for the
high-confidences ones. Is this study a proof-of-concept that will be extended to the other (slightly less likely) HSPs
candidates selected with the Chiaro et al. 2016 method in a future work? Or there are more fundamental reasons
why the other sources in Table 1 and 2 were not investigated that I am missing?

- Section 2: the description of the method used to select HSPs sources from 3FGL is terse. The manuscript, as it stands,
is not self-consistent and does not provide the minimal set of information that are needed to assess the viability of
the machine learning method used to select HSPs based on their gamma-ray flaring activity. I suggest that the authors
add to this section a summary of the basics about the method described in Chiaro+2016.

- As the authors mention in the introduction, other methods have been proposed to select candidate TeV blazars that
do not use gamma-ray information, at least directly. It would be interesting to know if their list of high and low-confidence
HSPs candidates can be spatially associated to candidates HSPs from the catalogs produced by Chang et al. 2017
(2WHSP) and D'Abrusco et al. 2019 (2019ApJS..242....4D).

- In order to model the EBL attenuation, a redshift for the gamma-ray source needs to be provideds. The authors should
specify how they sampled the 0 to 0.5 interval used to obtain the two extreme behaviors, or did they just use the z=0
and z=0.5 values to determine the boundaries of the blue shaded areas between the two fitted SEDs?

- Figure 1: since the focus of the papers is on sources located in scarcely populated bins for large L_HSP, I would
suggest to use a logarithmic y scale.

- Line 136: "did we find" -> "we found"

Hi Graziano,

Congratulations for getting such a positive report with minor comments. The responses are more-or-less to the point, however, below I (as the internal referee for this paper) provide a few minor suggestions to improve the presentation.

-----------------------------------

VSP: A general comment. Whenever it is mentioned that 'we have modified/updated the text', please also provide the appropriate line numbers. This will enable the referee to directly go to that line and read the updated text, rather than searching the relevant changes made in the paper.

- Section 2: the description of the method used to select HSPs sources from 3FGL is terse. The manuscript, as it stands, is not self-consistent and does not provide the minimal set of information that are needed to assess the viability of the machine learning method used to select HSPs based on their gamma-ray flaring activity. I suggest that the authors add to this section a summary of the basics about the method described in Chiaro+2016.

Authors:  We have reorganized the description of the machine learning method and added additional information.

VSP: Here I would briefly describe the changes made in the paper in response to the referee's comments. In its current form, the reply is forcing the referee to  go back to the paper and read the whole section again. Try to make him/her happy here itself.

- As the authors mention in the introduction, other methods have been proposed to select candidate TeV blazars that do not use gamma-ray information, at least directly. It would be interesting to know if their list of high and low-confidence HSPs candidates can be spatially associated to candidates HSPs from the catalogs produced by Chang et al. 2017 (2WHSP) and D'Abrusco et al. 2019 (2019ApJS..242....4D).

Authors:  We added a paragraph comparing our results to the 2WHSP list at the end of section 3.  We did not try to compare to the D'Abrusco catalogs, because those only attempt to identify BL Lacs, without identifying HSPs, as far as we could tell. 

VSP: Reading the boldface text in the paper, I suggest that wordings '2WHSP candidate list' can be  rephrased as 'the list of blazars presented in the 2WHSP catalog'.  Moreover, in the referee response, I would slightly rephrase the last sentence as follows:

We did not try to compare our results with the D'Abrusco catalog, because their work probably attempt to identify general BL Lac population, and not specifically HSPs. 

- In order to model the EBL attenuation, a redshift for the gamma-ray source needs to be provideds. The authors should specify how they sampled the 0 to 0.5 interval used to obtain the two extreme behaviors, or did they just use the z=0 and z=0.5 values to determine the boundaries of the blue shaded areas between the two fitted SEDs?

Authors: We did the calculation for z=0 and z=0.5, without attempting to do a sampling, which would require other assumptions.  These two values encompass most of the known HSP redshifts, providing limits.   We added this information to the text.

VSP: Either drop 'which would require other assumptions' or  mention what assumptions are you referring to. If you like, you can add the reference of  TeVCAT which supports the idea that most TeV emitting HSPs have 0<z<0.5. Also, add the line number pointing to the changes made.

Dear authors,

All good from my side. A very minor note: I would not include the "probably" in the last but one comment from the internal referee (Vaidehi) since looking at the paper (i.e., D'Abrusco+) we can confidently say that it is addressing the general BL Lac class. But as said, this is a minor comment.

I would suggest posting the updated referee response, that could help speeding the PubBoard look.