Subject: Re: .newdat format From: Carlo ContaldiDate: Tue, 20 May 2008 17:04:55 +0100 To: Clem Pryke hi Clem glad to hear the eagerly awaited data files are going to come out soon! The format for the newdats looks complicated but it contains a lot of useful information for doing params with small sky band power experiments like ours. As you pointed out it's not written up anywhere so here goes... 1) first line is the root filename for the window functions it will look for in data/windows 2) then a line of 6 integers stating how many bins in each pol type in the order TT EE BB EB TE TB (note this is different from cosmomc's order for historical reasons - cosmomc didn't expect to introduce other xcorrs) 3) Optional band selection set of 7 lines e.g. BAND_SELECTION 1 24 1 7 1 7 0 0 1 9 0 0 this would use TT 1-24, EE 1-7, BB 1-7, and TE 1-9 etc. If These 7 lines are not present it will use all available bands but this feature is nice to test sensitivity to high or low-l bands 4) below that (or 4th line if BAND_SELECTION is not present) there is a calibration error option line with calon/off(1/0), recal value (in temp), cal error (in power) note that cal error in power and recal value are in different units for historical (stupid) reasons! usually recal is 1 unless you want to test against recalibrations or simply update the calibration without changing anything else 5) beam error option line with beamerr on/off(1/0), beam FWHM (arcmin), beam err (arcmin) This assumes a gaussian beam with errors evaluated at weighted average of l for each bin. 6) likelihood option line liketype 0 == treat all bands gaussian 1 == treat all bands offset lognormals (using offsets provided) 2 == treat according to flag in last column of bandpower (0== gaussian, 1==offset lognormal) option 2 is useful if you are including TE as those might be better approximated by a gaussian 7) 'TT', 'EE' etc. flag telling the code which pol type is listed 8) list of band power info: band num per type, C_b, dC_b+,dC_b-, noise offsets, lmin, lmax, likeflag lmin, lmax are not used and are included just for plotting purposes, noise offsets should be provided if liketype > 0. If liketype < 2 then last column may be omitted 9) dump of block diagonal normalized covariance matrix for that particular pol type, dimension must be the same as list immediately above. This is included purely for the practical purpose of reading off bin-to-bin correlations without having to dig them out of the full covariance matrix. It is not used by cosmomc and I guess I should make this optional at some point but this information is always very handy for people to see. 10) repeat 7)-9) for each pol_type included, bin numbers must agree with values given in 2) 11) dump of the whole all-with-all covariance matrix (e.g. inverse Fisher) across all pol types included and in same order as 2). The covariance matrix is always that of the C_Bs (not of the offset lognormal vars even if liketype > 0). The transformation to offset lognormals or mixed gaussian/offset lognormal is done internally in Readdataset_bcp. 12) optional SZ freq line in GHz e.g. #SZ 30 This was for our internal (not included in public version) way of including a freq dependent SZ template for the various experiments with automatic scaling e.g. acbar + cbi. Antony has now included his own method to deal with SZ templates in the latest cosmomc version but with his you have to give a separate template for each experiment. finally - windows, these follow the cosmomc format with each bin in a separate file. Each file will have l, TT, TE, EE, BB contribution to that particular bin - note this is the cosmomc ordering! you have to list each l between an lmin and lmax of your choosing. The windows must be normalized and the number actually listed is W_l^B/l. The theoretical band power is C_B = \sum_l (l+1/2)/l/(l+1) W_l^B C_l let me know if you need anything else Carlo