Aquila-Consortium/borg_public
32
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Initial import

Browse source
This commit is contained in:
Guilhem Lavaux 2023-05-29 10:41:03 +02:00
commit 56a50eead3
820 changed files with 192077 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

249
docs/source/user/inputs/Configuration_file_v1.inc.rst Normal file
View file

@ -0,0 +1,249 @@
.. _configuration_file:
ARES_Configuration_file_v1
==========================
The configuration file for ARES uses the INI file syntax. It is
separated into sections among which three are main sections.
Main sections
-------------
Section [system]
~~~~~~~~~~~~~~~~
- console_output: Holds the prefix filename for all log output files.
- VERBOSE_LEVEL: Set the verbosity level for the console. Files get all
outputs.
- N0: Number of grid elements along the X axis.
- N1: Same for Y axis.
- N2: Same for Z axis.
- L0: Comoving length of the X axis
- L1: Same for Y axis
- L2: Same for Z axis
- corner0: Center of the voxel at the corner of the box in -X
direction, this should be the smallest X value.
- corner1: Same for Y
- corner2: Same for Z
- NUM_MODES: number of bins to represent the power spectrm
- N_MC: Maximum number of markov chain samples to produce in a single
run (**Note:** Used only for *v1*)
- borg_supersampling: Supersampling level of the grid for intermediate
calculations. The number of particles is
N0*N1*N2*borg_supersampling**3
- hades_likelihood: Likelihood to use in HADES run. Can be either one
of those values:
- BORG_POISSON: Use poisson likelihood
- BORG_POISSON_POWER:
- BORG_VOODOO:
- BORG_VOODOO_MAGIC:
- BORG_LINEAR: ARES likelihood model. Noise is Gaussian with
Variance equal to :math:`S \bar{N}`. Use power law bias.
- BORG_SH:
- BORG_NB: Negative binomial. Broken power law bias.
- Generic framework:
- GAUSSIAN_BROKEN_POWERLAW_BIAS
- GAUSSIAN_MO_WHITE_BIAS: Gaussian noise model, variance is
fitted. Double power law bias
- GAUSSIAN_POWERLAW_BIAS
- GAUSSIAN_2ND_ORDER_BIAS
- GENERIC_POISSON_BROKEN_POWERLAW_BIAS
- GENERIC_GAUSSIAN_LINEAR_BIAS
- GENERIC_GAUSSIAN_MANY_POWER_1^1
- GENERIC_GAUSSIAN_MANY_POWER_1^2
- GENERIC_GAUSSIAN_MANY_POWER_1^4
- GENERIC_POISSON_MANY_POWER_1^1
- GENERIC_POISSON_MANY_POWER_1^2
- GENERIC_POISSON_MANY_POWER_1^4
- hades_forward_model: Forward model to use
- LPT: Lagrangian perturbation theory, ModifiedNGP/Quad final
projection
- 2LPT: Second order Lagrangian perturbation theory,
ModifiedNGP/Quad final projection
- PM: Particle mesh, ModifiedNGP/Quad final projection
- LPT_CIC: Same as LPT, but use CIC for final projection
- 2LPT_CIC: Same as LPT, but use CIC for final projection
- PM_CIC: Same as LPT, but use CIC for final projection
- HADES_LOG: Use Exponential transform (HADES model) for the forward
model. Preserved mean density is enforced.
- borg_do_rsd: Do redshift space distortion if set to "true".
- projection_model: Specifies which projection to use for data. No
constraints are enforced on the likelihood, but of course they should be matched
to the value adopted here. The value is inspected in ``src/common/projection.hpp``.
There are two available at the moment: ``number_ngp`` and ``luminosity_cic``.
The ``number_ngp`` is just Nearest-Grid-Point number counting.
The ``luminosity_cic`` uses the value in ``Mgal`` to weight the object
before doing CIC projection.
- number_ngp: it just counts the number of galaxies/objects within a voxel
- luminosity_cic: it weights galaxies by their luminosity and do a CIC projection.
- test_mode: Runs ARES/BORG/HADES in test mode. Data is not used, mock
data is generated on the fly.
- seed_cpower: Set to true to seed the power spectrum with the correct
one according to the cosmology section. Otherwise it is set to a
small fraction of it.
- hades_max_epsilon: Stepsize for the HMC. It is unitless. Good
starting point is around 0.01.
- hades_max_timesteps: Maximum number of timesteps for a single HMC
sample.
- hades_mixing: Number of samples to compute before writing to disk.
- savePeriodicity: This reduces the number of times the restart files
are dumped to the hard drives. This is useful for reducing I/Os, as
restart files are heavy. You can set this to a number that is a
multiple of the number of mcmc steps. For example, 20 tells ares to
dump restart files every 20 mcmc steps.
- mask_precision: Precision to which you want to compute the mask. By
default it is "0.01", which is not related to the actual precision
(unfortunately not yet). It allows scaling the internal number of
evaluation of the selection function. So 0.001 will call it 100 times
more. The advice is not to decrease below 0.01.
- furious_seeding: if set to true the core sampler will reseed itself
from a system entropy source at each step of the MCMC. That means the
MCMC becomes unpredictable and the seed number is discarded.
- simulation: if set to true switches to N-body simulation analysis.
Additional cuts are possible depending on masses, spins, etc, of
halos.
Likelihoods that use the generic bias framework (currently
GAUSSIAN_MO_WHITE_BIAS) supports also the following tags:
- bias_XX_sampler_generic_blocked: if sets to true, it will not
sampling the XX parameter of the bias. XX varies depending on the
likelihood.
- block_sigma8_sampler: true by default, to sample sigma8 in the
initial conditions, sets this to false
Section [run]
~~~~~~~~~~~~~
- NCAT: Number of catalogs. This affects the number of "catalog"
sections.
- SIMULATION: Specify if the input is from simulation. Default is
false.
Section [cosmology]
~~~~~~~~~~~~~~~~~~~
- omega_r: Radiation density
- omega_k: Curvature
- omega_m: Total matter density
- omega_b: Baryonic matter density
- omega_q: Quintescence density
- w: Quintescence equation of state
- wprime: Derivative of the equation of state
- n_s: Slope of the power spectrum of scalar fluctuations
- sigma8: Normalisation of powerspectrum at 8 Mpc/h
- h100: Hubble constant in unit of 100 km/s/Mpc
Section [julia]
~~~~~~~~~~~~~~~
- likelihood_path: Path to the julia file describing the likelihood
(i.e. the main entry point for BORG in the likelihood)
- likelihood_module: Name of the julia module holding the likelihood
- bias_sampler_type: slice or hmclet, which sampling strategy to use to
sample the "bias" parameters
- ic_in_julia: true or false, whether the initial condition of the
Markov Chain is set in julia
- hmclet_diagonalMass: whether to use a diagonal or a dense mass matrix
estimed on the fly
- hmclet_burnin: number of steps allowed in "BURN IN" mode. This
depends on the complexity of the likelihood. A few hundred seems
reasonable.
- hmclet_burnin_memory: size of the memory in "BURN IN" mode. Something
like 50 is advocated to be sure it is fairly local but not too noisy.
- hmclet_maxEpsilon: maximum epsilon for the HMC integrator (take order
0.01)
- hmclet_maxNtime: maximum number of timesteps for the HMC integrator
(take a few decade like 20-50)
Catalog sections
----------------
Basic fields
~~~~~~~~~~~~
- datafile: Text filename holding the data
- maskdata: Healpix FITS file with the mask
- radial_selection: Type of selection function, can be either
"schechter", "file" or "piecewise".
- refbias: true if this catalog is a reference for bias. Bias will not
be sampled for it
- bias: Default bias value, also used for mock generation
- nmean: Initial mean galaxy density value, also used for mock
generation
Halo selection
~~~~~~~~~~~~~~
- halo_selection: Specifying how to select the halos from the halo catalog. Can be ``mass, radius, spin or mixed``. The ``mixed`` represents the combined cuts and can be applied by specifying, eg "halo_selection = mass radius"
- halo_low_mass_cut: this is log10 of mass in the same unit as the
masses of the input text file
- halo_high_mass_cut: same as for halo_low_mass_cut, this is log10 of
mass
- halo_small_radius_cut
- halo_large_radius_cut
- halo_small_spin_cut
- halo_high_spin_cut
Schechter selection function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- schechter_mstar: Mstar for Schechter function
- schechter_alpha: Power law slope of Schechter function
- schechter_sampling_rate: How many distance points to precompute from
Schechter (i.e. 1000)
- schechter_dmax: Maximum distance to precompute Schecter selection
functino
- galaxy_bright_apparent_magnitude_cut: Apparent magnitude where data
and selection must be truncated, bright end.
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in data
and selection function, bright end, useful to select different galaxy
populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- zmin: Minimum redshift for galaxy sample, galaxies will be truncated
- zmax: Maximum redshift for galaxy sample, galaxies will be truncated
'File' selection function
~~~~~~~~~~~~~~~~~~~~~~~~~
- radial_file: Text file to load the selection from
The file has the following format. Each line starting with a '#' is a
comment line, and discarded. The first line is a set of three numbers:
'rmin dr N'. Each line that follows must be a number between 0 and 1
giving the selection function at a distance r = rmin + dr \* i, where
'i' is the line number (zero based). Finally 'N' is the number of points
in the text file.
Two possibilities are offered for adjusting the catalog and the
selection together:
- either you chose not to do anything, and take the whole sample and
provided selection. Then you need to specify:
- file_dmin: Minimal distance for selection function and data
- file_dmax: same but maximal distance
- no_cut_catalog: set to false, if you do not set this you will get
an error message.
- or you want ares to preprocess the catalog and then you need:
- zmin
- zmax
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in
data and selection function, bright end, useful to select
different galaxy populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- no_cut_catalog: (not necessary, as it defaults to true)

378
docs/source/user/inputs/Configuration_file_v2.1.inc.rst Normal file
View file

@ -0,0 +1,378 @@
ARES_Configuration_file_v2.1
============================
The configuration file for ARES uses the INI file syntax. It is
separated into sections among which three are main sections.
Main sections
-------------
Section [system]
~~~~~~~~~~~~~~~~
- console_output: Holds the prefix filename for all log output files.
- VERBOSE_LEVEL: Set the verbosity level for the console. Files get all
outputs.
- N0: Number of grid elements along the X axis.
- N1: Same for Y axis.
- N2: Same for Z axis.
- **Optionally:**
- Ndata0, Ndata1, Ndata2 specifies the same thing as N0, N1, N2 but
for the projection grid of the galaxy positions. This grid must be
different in the case the degrader bias pass is used (see bias
model section)
- L0: Comoving length of the X axis
- L1: Same for Y axis
- L2: Same for Z axis
- corner0: Center of the voxel at the corner of the box in -X
direction, this should be the smallest X value.
- corner1: Same for Y
- corner2: Same for Z
- NUM_MODES: number of bins to represent the power spectrm
- projection_model: Specifies which projection to use for data. No
constraints are enforced on the likelihood, but of course they should be matched
to the value adopted here. The value is inspected in ``src/common/projection.hpp``.
There are two available at the moment: ``number_ngp`` and ``luminosity_cic``.
The ``number_ngp`` is just Nearest-Grid-Point number counting.
The ``luminosity_cic`` uses the value in ``Mgal`` to weight the object
before doing CIC projection.
- number_ngp: it just counts the number of galaxies/objects within a voxel
- luminosity_cic: it weights galaxies by their luminosity and do a CIC projection.
- test_mode: Runs ARES/BORG/HADES in test mode. Data is not used, mock
data is generated on the fly.
- seed_cpower: Set to true to seed the power spectrum with the correct
one according to the cosmology section. Otherwise it is set to a
small fraction of it.
- savePeriodicity: This reduces the number of times the restart files
are dumped to the hard drives. This is useful for reducing I/Os, as
restart files are heavy. You can set this to a number that is a
multiple of the number of mcmc steps. For example, 20 tells ares to
dump restart files every 20 mcmc steps.
- mask_precision: Precision to which you want to compute the mask. By
default it is "0.01", which is not related to the actual precision
(unfortunately not yet). It allows scaling the internal number of
evaluation of the selection function. So 0.001 will call it 100 times
more. The advice is not to decrease below 0.01.
- furious_seeding: if set to true the core sampler will reseed itself
from a system entropy source at each step of the MCMC. That means the
MCMC becomes unpredictable and the seed number is discarded.
Section [block_loop]
~~~~~~~~~~~~~~~~~~~~
- hades_sampler_blocked: Prevents the density field from being sampled
Likelihoods that use the generic bias framework (currently
GAUSSIAN_MO_WHITE_BIAS) supports also the following tags:
- bias_XX_sampler_generic_blocked: if sets to true, it will not
sampling the XX parameter of the bias. XX varies depending on the
likelihood. '''WARNING: the code has not yet been updated to look for
these variables in [block_loop], they should still be located in
[system] at the moment. '''
- sigma8_sampler_blocked: true by default, to sample sigma8 in the
initial conditions, sets this to false
Section [mcmc]
~~~~~~~~~~~~~~
- number_to_generate: Maximum number of markov chain samples to produce
in a single run
- init_random_scaling: This is more specific to HADES. It starts the
MCMC run with a random initial condition, scaled with this number
(default 0.1) compared to the reference initial powerspectrum.
- random_ic: true if ic must be reshuffled before starting the MCMC
sampling, false to keep them at their value generated by the mock
data generator
- scramble_bias: true (default), reset the bias values to some other
values before starting the chain, after generating the mock.
Section [gravity]
~~~~~~~~~~~~~~~~~
- model: Forward model to use
- LPT: Lagrangian perturbation theory, ModifiedNGP/Quad final
projection
- 2LPT: Second order Lagrangian perturbation theory,
ModifiedNGP/Quad final projection
- PM: Particle mesh, ModifiedNGP/Quad final projection
- LPT_CIC: Same as LPT, but use CIC for final projection
- 2LPT_CIC: Same as 2LPT, but use CIC for final projection
- PM_CIC: Same as PM, but use CIC for final projection
- tCOLA: Same as PM_CIC but uses a TCOLA gravity machine. To enable,
specify model=PM_CIC, as above, AND set tCOLA=true.
- HADES_LOG: Use Exponential transform (HADES model) for the forward
model. Preserved mean density is enforced.
- supersampling: Controls the number of particles (supersampling level
of the particle grid with respect to the grid). The number of
particles is (N0*N1*N2*borg_supersampling)**3
- forcesampling
- a_initial
- a_final
- pm_start_z:
- pm_nsteps:
- part_factor:
- lightcone:
- do_rsd: Do redshift space distortion if set to "true".
Forward model elements can as well be chained and have different grid sizes. *"model"* can now be CHAIN, which then needs a specific list of model layers in *"models"*.
Here is an example:
.. code:: text
[gravity]
model=CHAIN
models=PRIMORDIAL,TRANSFER_EHU,LPT_CIC
[gravity_chain_0]
a_final=0.001
[gravity_chain_1]
[gravity_chain_2]
supersampling=2
lightcone=false
do_rsd=false
a_initial=0.001
a_final=1.
part_factor=2.0
mul_out=1
Each element of the chain gets its own configuration section which is
the same as previously when it was a global descriptor (see above). Note
that it you use the chain mechanism, you have to be explicit on the
production of initial conditions power spectrum. As you can see above,
we indicate "PRIMORDIAL,TRANSFER_EHU" to start with a primordial
scale-free gravitational potential, onto which we apply an Einstein-Hu
transfer function to form density fluctuations, which are then passed
down to LPT_CIC. Also keep in mind that the scale factors must be
compatibles and no checks are run by the code at the moment. \`mul_out\`
specifices how much the output grid as to be supersampled for the CIC
(i.e. the CIC grid is produced at mul_out times the initial grid size).
Model 'Primordial'
^^^^^^^^^^^^^^^^^^
Apply a primordial scale free power spectrum on the input. The output is
scaled linearly to a_final.
Model 'Transfer'
^^^^^^^^^^^^^^^^
* **CIC correction**: use_invert_cic=true: Transfer function is inverse CIC smoother=0.99 (in unit of grid)
* **Sharp K filter**: use_sharpk=true: Transfer function is sharp k filter k_max=0.1 (in h/Mpc)
Model 'Softplus'
^^^^^^^^^^^^^^^^
Apply a softplus transform hardness=1.0 , some parameter making the
transition more or less harder
Model 'Downgrade'
^^^^^^^^^^^^^^^^^
(No option)
Section [hades]
^^^^^^^^^^^^^^^
- max_epsilon: Stepsize for the HMC. It is unitless. Good starting
point is around 0.01.
- max_timesteps: Maximum number of timesteps for a single HMC sample.
- mixing: Number of samples to compute before writing to disk.
- algorithm:
- HMC: classical HMC algorithm
- QN-HMC: Quasi-Newton HMC algorithm
- FROZEN-PHASE: Fixed phase. They are not sampled at all but provide
some pipelines to allow the other samplers to work.
- phases: if ``algorithm`` is FROZEN-PHASE, you can specify an HDF5
filename here. This file must contain a "phase" array which is
conforming to the setup of the ini.
- noPhasesProvided: if phases is omitted, this one has to be set to
true, otherwise an error is thrown.
- phasesDataKey: this indicate which field to use in the ``phases``
HDF5 file.
- likelihood: Likelihood to use in HADES run. Can be either one of
those values:
- LINEAR: Gaussian likelihood
- BORG_POISSON: Use poisson likelihood
- Generic framework:
- GAUSSIAN_BROKEN_POWERLAW_BIAS
- GAUSSIAN_MO_WHITE_BIAS: Gaussian noise model, variance is
fitted. Double power law bias
- GAUSSIAN_POWERLAW_BIAS: Power law bias model with a Gaussian
noise model, variance is fitted.
- GAUSSIAN_2ND_ORDER_BIAS
- GENERIC_POISSON_BROKEN_POWERLAW_BIAS: Broken power law bias
model (also called Neyrinck's model), with Poisson noise lmodel
- GENERIC_GAUSSIAN_LINEAR_BIAS: Linear bias model, Gaussian noise
model
- GENERIC_GAUSSIAN_MANY_POWER_1^1
- GENERIC_GAUSSIAN_MANY_POWER_1^2
- GENERIC_GAUSSIAN_MANY_POWER_1^4
- GENERIC_POISSON_MANY_POWER_1^1
- GENERIC_POISSON_MANY_POWER_1^2
- GENERIC_POISSON_MANY_POWER_1^4
- GENERIC_POISSON_POWERLAW_BIAS: simple power law bias model with
Poisson noise model
- GENERIC_POISSON_POWERLAW_BIAS_DEGRADE4: power law bias models
preceded by a degrade pass (N -> N/4 in each direction)
- GENERIC_POISSON_BROKEN_POWERLAW_BIAS_DEGRADE4: broken power law
bias model preceded by a degrade pass (N -> N/4 in each
direction)
- scheme: SI_2A, SI_2B, SI_2C, SI_3A, SI_4B, SI_4C, SI_4D, SI_6A
Section [run]
~~~~~~~~~~~~~
- NCAT: Number of catalogs. This affects the number of "catalog"
sections.
- SIMULATION: Specify if the input is from simulation. Default is
false.
Section [likelihood]
~~~~~~~~~~~~~~~~~~~~
- MainPower_prior_width: Variance of the manypower parameters (except
mean which is always uniform positive)
- EFT_Lambda: Lambda truncation parameter of the EFT bias model
- Options related to robust likelihood. Each patch of a robust likelihood can be sliced in the redshift direction.
There are two options controlling the slicing: the maximum distance "rmax" and the number of slices "slices"
* rmax: Maximum distance accessible during the inference. In practice it is at least the farthest distance of a voxel in the box.
Unit is the one of the box, most generally :math:`h^{-1}` Mpc.
* slices: Number of slices to build in the redshift direction. Each patch will have a depth ~rmax/slices.
Section [cosmology]
~~~~~~~~~~~~~~~~~~~
- omega_r: Radiation density
- omega_k: Curvature
- omega_m: Total matter density
- omega_b: Baryonic matter density
- omega_q: Quintescence density
- w: Quintescence equation of state
- wprime: Derivative of the equation of state
- n_s: Slope of the power spectrum of scalar fluctuations
- sigma8: Normalisation of powerspectrum at 8 Mpc/h
- h100: Hubble constant in unit of 100 km/s/Mpc
- fnl: primordial non-Gaussianity
Section [julia]
~~~~~~~~~~~~~~~
- likelihood_path: path of the julia code
- likelihood_module: julia module where the likelihood is implemented
- bias_sampler_type: type of sampler for the bias parameters (hmclet,
slice)
- ic_in_julia: whether initial conditions of the MCMC are coded in
julia or choose some random numbers
- hmclet_diagonalMass: where to use a diagonal mass matrix or a full
dense
- mass_burnin: number of MCMC steps in burnin mode
- mass_burnin_memory: number of MCMC steps to store when in burnin mode
- hmclet_maxEpsilon: maximum epsilon for the leapfrog integrator
(~0.002-0.01 depending on likelihood complexity)
- hmclet_maxNtime: maximum number of steps for the leapfrog integrator
(~50-100)
- hmclet_massScale: amount of momentum reshuffling (0.0 = full, 1.0 =
none bad for MCMC)
- hmclet_correlationLimiter: reduce the correlations in the covariance
matrix by some number. Typically the smaller the number the less
reduction with :math:`\simeq 1` reducing the correlation by 2.
Catalog sections
----------------
Basic fields
~~~~~~~~~~~~
- datafile: Text filename holding the data
- maskdata: Healpix FITS file with the mask
- radial_selection: Type of selection function, can be either
"schechter", "file" or "piecewise".
- refbias: true if this catalog is a reference for bias. Bias will not
be sampled for it
- bias: Default bias value, also used for mock generation
- nmean: Initial mean galaxy density value, also used for mock
generation
Halo selection
~~~~~~~~~~~~~~
- halo_selection: Specifying how to select the halos from the halo catalog. Can be ``mass, radius, spin or mixed``. The ``mixed`` represents the combined cuts and can be applied by specifying, eg "halo_selection = mass radius"
- halo_low_mass_cut: this is log10 of mass in the same unit as the
masses of the input text file
- halo_high_mass_cut: same as for halo_low_mass_cut, this is log10 of
mass
- halo_small_radius_cut
- halo_large_radius_cut
- halo_small_spin_cut
- halo_high_spin_cut
Schechter selection function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- schechter_mstar: Mstar for Schechter function
- schechter_alpha: Power law slope of Schechter function
- schechter_sampling_rate: How many distance points to precompute from
Schechter (i.e. 1000)
- schechter_dmax: Maximum distance to precompute Schecter selection
function
- galaxy_bright_apparent_magnitude_cut: Apparent magnitude where data
and selection must be truncated, bright end.
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in data
and selection function, bright end, useful to select different galaxy
populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- zmin: Minimum redshift for galaxy sample, galaxies will be truncated
- zmax: Maximum redshift for galaxy sample, galaxies will be truncated
'File' selection function
~~~~~~~~~~~~~~~~~~~~~~~~~
- radial_file: Text file to load the selection from
The file has the following format. Each line starting with a '#' is a
comment line, and discarded. The first line is a set of three numbers:
'rmin dr N'. Each line that follows must be a number between 0 and 1
giving the selection function at a distance r = rmin + dr \* i, where
'i' is the line number (zero based). Finally 'N' is the number of points
in the text file.
Two possibilities are offered for adjusting the catalog and the
selection together:
- either you chose not to do anything, and take the whole sample and
provided selection. Then you need to specify:
- file_dmin: Minimal distance for selection function and data
- file_dmax: same but maximal distance
- no_cut_catalog: set to false, if you do not set this you will get
an error message.
- or you want ares to preprocess the catalog and then you need:
- zmin
- zmax
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in
data and selection function, bright end, useful to select
different galaxy populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- no_cut_catalog: (not necessary, as it defaults to true)

393
docs/source/user/inputs/Configuration_file_v2.inc.rst Normal file
View file

@ -0,0 +1,393 @@
ARES_Configuration_file_v2
==========================
The configuration file for ARES uses the INI file syntax. It is
separated into sections among which three are main sections.
Main sections
-------------
Section [system]
~~~~~~~~~~~~~~~~
- console_output: Holds the prefix filename for all log output files.
- VERBOSE_LEVEL: Set the verbosity level for the console. Files get all
outputs. Check inside ``libLSS/tools/log_traits.hpp`` for details.
- **Values**:
- VERBOSE_LEVEL=1 : up to STD level
- VERBOSE_LEVEL=2 : INFO level
- VERBOSE_LEVEL=3 : VERBOSE level
- VERBOSE_LEVEL=4 : DEBUG level
- N0: Number of grid elements along the X axis.
- N1: Same for Y axis.
- N2: Same for Z axis.
- **Optionally:**
- Ndata0, Ndata1, Ndata2 specifies the same thing as N0, N1, N2 but
for the projection grid of the galaxy positions. This grid must be
different in the case the degrader bias pass is used (see bias
model section)
- L0: Comoving length of the X axis
- L1: Same for Y axis
- L2: Same for Z axis
- corner0: Center of the voxel at the corner of the box in -X
direction, this should be the smallest X value.
- corner1: Same for Y
- corner2: Same for Z
- NUM_MODES: number of bins to represent the power spectrm
- projection_model: Specifies which projection to use for data. No
constraints are enforced on the likelihood, but of course they should be matched
to the value adopted here. The value is inspected in ``src/common/projection.hpp``.
There are two available at the moment: ``number_ngp`` and ``luminosity_cic``.
The ``number_ngp`` is just Nearest-Grid-Point number counting.
The ``luminosity_cic`` uses the value in ``Mgal`` to weight the object
before doing CIC projection.
- number_ngp: it just counts the number of galaxies/objects within a voxel\
- luminosity_cic: it weights galaxies by their luminosity and do a CIC projection.
- test_mode: Runs ARES/BORG/HADES in test mode. Data is not used, mock
data is generated on the fly.
- seed_cpower: Set to true to seed the power spectrum with the correct
one according to the cosmology section. Otherwise it is set to a
small fraction of it.
- savePeriodicity: This reduces the number of times the restart files
are dumped to the hard drives. This is useful for reducing I/Os, as
restart files are heavy. You can set this to a number that is a
multiple of the number of mcmc steps. For example, 20 tells ares to
dump restart files every 20 mcmc steps.
- mask_precision: Precision to which you want to compute the mask. By
default it is "0.01", which is not related to the actual precision
(unfortunately not yet). It allows scaling the internal number of
evaluation of the selection function. So 0.001 will call it 100 times
more. The advice is not to decrease below 0.01.
- furious_seeding: if set to true the core sampler will reseed itself
from a system entropy source at each step of the MCMC. That means the
MCMC becomes unpredictable and the seed number is discarded.
Section [block_loop]
~~~~~~~~~~~~~~~~~~~~
- hades_sampler_blocked: Prevents the density field from being sampled
Likelihoods that use the generic bias framework (currently
GAUSSIAN_MO_WHITE_BIAS) supports also the following tags:
- bias_XX_sampler_generic_blocked: if sets to true, it will not
sampling the XX parameter of the bias. XX varies depending on the
likelihood. '''WARNING: the code has not yet been updated to look for
these variables in [block_loop], they should still be located in
[system] at the moment. '''
- **Note:**
Whenever a bias model uses $b_0$ to hold the normalization,
inside its header you should set/see ``NmeanIsBias=True``.
Take a look inside ``libLSS/physics/bias/*`` (for example ``linear.hpp``).
- sigma8_sampler_blocked: true by default, to sample sigma8 in the
initial conditions, sets this to false
Section [mcmc]
~~~~~~~~~~~~~~
- number_to_generate: Maximum number of markov chain samples to produce
in a single run
- init_random_scaling: This is more specific to HADES. It starts the
MCMC run with a random initial condition, scaled with this number
(default 0.1) compared to the reference initial powerspectrum.
- random_ic: true if ic must be reshuffled before starting the MCMC
sampling, false to keep them at their value generated by the mock
data generator
Section [gravity]
~~~~~~~~~~~~~~~~~
- model: Forward model to use
- LPT: Lagrangian perturbation theory, ModifiedNGP/Quad final
projection
- 2LPT: Second order Lagrangian perturbation theory,
ModifiedNGP/Quad final projection
- PM: Particle mesh, ModifiedNGP/Quad final projection
- LPT_CIC: Same as LPT, but use CIC for final projection
- 2LPT_CIC: Same as 2LPT, but use CIC for final projection
- PM_CIC: Same as PM, but use CIC for final projection
- tCOLA: Same as PM_CIC but uses a TCOLA gravity machine. To enable,
specify model=PM_CIC, as above, AND set tCOLA=true.
- HADES_LOG: Use Exponential transform (HADES model) for the forward model. Preserved mean density is enforced.
- supersampling: Controls the number of particles (supersampling level of the particle grid with respect to the grid). The number of particles is :math:`(N_0 \cdot N_1 \cdot N_2 \cdot \mathrm{supersampling})^3`
- forcesampling : This is the oversampling for computing the gravitational field (and thus the force in the PM). A current rule of thumb is to have forcesampling at least twice of supersampling, and supersampling at least two. For tCOLA, the requirements are less.
- **To be checked:** Setup with forcesampling=supersampling.
- a_initial : Scale factor value reflects the time. This parameter controls the value of the a_initial (:math:`a_i`) which should be :math:`10^{-3} \leq a_i \leq 1.0`, with :math:`a_i=10^{-3}` corresponding to the time of CMB
- a_final : Same as a_initial parameter, but :math:`a_f > a_i`
- pm_start_z: This is relevant only for the PM forward model and represents the starting redshift for the PM simulation.
- pm_nsteps: Relevant only for PM model, see ``extra/borg/libLSS/physics/forwards/borg_multi_pm.cpp``. There are two scalings in the code, controlled with ``LOG_SCALE_STEP``. If ``LOG_SCALE_STEP`` is set to ``False`` then steps are splitted linearly in :math:`a`. It seems the linear scaling gives better results in tests of :math:`P(k)`.
- part_factor: An option relevant for MPI run. This is the overallocation of particles on each node to allow for moving them in and out of the node. It is required because the density projection needs to have only the relevant particles on the node. If one of them is outside the slab it will cause a failure.
- **Note**: ``part_factor`` is indipendent of ``forcesampling`` and ``supersampling`` It will likely be larger for smaller boxes (physical length) and smaller box (in terms of mesh / grid size). The first case because particles travel larger distances w.r.t to the size of the box, and the second because there is more shot noise.
- lightcone: See equation 2 from the `SDSS3-BOSS inference paper <https://arxiv.org/pdf/1909.06396.pdf>`_. This option is more relevant for larger boxes.
- do_rsd: Do redshift space distortion if set to ``True``.
- **Note:** The DM particles are shifted directly. But, this will never be the case in observations, for which it is ensemble of gas particles around a galaxy that is shifted.
Forward model elements can as well be chained and have different grid sizes. *"model"* can now be CHAIN, which then needs a specific list of models in *"models"*.
Here is an example:
.. code:: text
[gravity]
model=CHAIN
models=PRIMORDIAL,TRANSFER_EHU,LPT_CIC
[gravity_chain_0]
a_final=0.001
[gravity_chain_1]
[gravity_chain_2]
supersampling=2
lightcone=false
do_rsd=false
a_initial=0.001
a_final=1.
part_factor=2.0
mul_out=1
Each element of the chain gets its own configuration section which is
the same as previously when it was a global descriptor (see above). Note that
if you use the chain mechanism, you have to be explicit on the production of initial conditions power spectrum.
As you can see above, we indicate "PRIMORDIAL,TRANSFER_EHU" to start with a primordial scale-free gravitational potential,
onto which we apply an Einstein-Hu transfer function to form density fluctuations, which are then
passed down to LPT_CIC. Also keep in mind that the scale factors must be compatibles and no checks
are run by the code at the moment. ``mul_out`` specifices how much the output grid as to be supersampled for the
CIC (i.e. the CIC grid is produced at mul_out times the initial grid size).
Model 'Primordial'
^^^^^^^^^^^^^^^^^^
Apply a primordial scale free power spectrum on the input. The output is
scaled linearly to a_final.
Model 'Transfer'
^^^^^^^^^^^^^^^^
* **CIC correction**: use_invert_cic=true: Transfer function is inverse CIC smoother=0.99 (in unit of grid)
* **Sharp K filter**: use_sharpk=true: Transfer function is sharp k filter k_max=0.1 (in h/Mpc)
Model 'Softplus'
^^^^^^^^^^^^^^^^
Apply a softplus transform hardness=1.0 , some parameter making the
transition more or less harder
Model 'Downgrade'
^^^^^^^^^^^^^^^^^
(No option)
Section [hades]
~~~~~~~~~~~~~~~
- max_epsilon: Stepsize for the HMC. It is unitless. Good starting
point is around 0.01.
- max_timesteps: Maximum number of timesteps for a single HMC sample.
- mixing: Number of samples to compute before writing to disk.
- algorithm:
- HMC: classical HMC algorithm
- QN-HMC: Quasi-Newton HMC algorithm
- FROZEN-PHASE: Fixed phase. They are not sampled at all but provide
some pipelines to allow the other samplers to work.
- phases: if ``algorithm`` is FROZEN-PHASE, you can specify an HDF5
filename here. This file must contain a "phase" array which is
conforming to the setup of the ini.
- noPhasesProvided: if phases is omitted, this one has to be set to
true, otherwise an error is thrown.
- phasesDataKey: this indicate which field to use in the ``phases``
HDF5 file.
- likelihood: Likelihood to use in HADES run. Can be either one of
those values:
- LINEAR: Gaussian likelihood
- BORG_POISSON: Use poisson likelihood
- Generic framework:
- GAUSSIAN_BROKEN_POWERLAW_BIAS
- GAUSSIAN_MO_WHITE_BIAS: Gaussian noise model, variance is
fitted. Double power law bias
- GAUSSIAN_POWERLAW_BIAS: Power law bias model with a Gaussian
noise model, variance is fitted.
- GAUSSIAN_2ND_ORDER_BIAS
- GENERIC_POISSON_BROKEN_POWERLAW_BIAS: Broken power law bias
model (also called Neyrinck's model), with Poisson noise lmodel
- GENERIC_GAUSSIAN_LINEAR_BIAS: Linear bias model, Gaussian noise
model
- GENERIC_GAUSSIAN_MANY_POWER_1^1
- GENERIC_GAUSSIAN_MANY_POWER_1^2
- GENERIC_GAUSSIAN_MANY_POWER_1^4
- GENERIC_POISSON_MANY_POWER_1^1
- GENERIC_POISSON_MANY_POWER_1^2
- GENERIC_POISSON_MANY_POWER_1^4
- GENERIC_POISSON_POWERLAW_BIAS: simple power law bias model with
Poisson noise model
- GENERIC_POISSON_POWERLAW_BIAS_DEGRADE4: power law bias models
preceded by a degrade pass (N -> N/4 in each direction)
- GENERIC_POISSON_BROKEN_POWERLAW_BIAS_DEGRADE4: broken power law
bias model preceded by a degrade pass (N -> N/4 in each
direction)
- scheme: SI_2A, SI_2B, SI_2C, SI_3A, SI_4B, SI_4C, SI_4D, SI_6A
Section [run]
~~~~~~~~~~~~~
- NCAT: Number of catalogs. This affects the number of "catalog"
sections.
-**Note:** If ``NCAT>1`` then it is supposed catalogues are independently taken (no double counting of galaxies etc.)
and hence when one evaluates the log-likelihood, they are just summed together.
- SIMULATION: Specify if the input is from simulation. Default is
false.
Section [cosmology]
~~~~~~~~~~~~~~~~~~~
- omega_r: Radiation density
- omega_k: Curvature
- omega_m: Total matter density
- omega_b: Baryonic matter density
- omega_q: Quintescence density
- w: Quintescence equation of state
- wprime: Derivative of the equation of state
- n_s: Slope of the power spectrum of scalar fluctuations
- sigma8: Normalisation of powerspectrum at 8 Mpc/h
- h100: Hubble constant in unit of 100 km/s/Mpc
- fnl: primordial non-Gaussianity
Section [likelihood]
~~~~~~~~~~~~~~~~~~~~
- Options related to robust likelihood. Each patch of a robust likelihood can be sliced in the redshift direction.
There are two options controlling the slicing: the maximum distance "rmax" and the number of slices "slices"
- rmax: Maximum distance accessible during the inference. In practice it is at least the farthest distance of a voxel in the box.
Unit is the one of the box, most generally :math:`h^{-1}` Mpc.
- slices: Number of slices to build in the redshift direction. Each patch will have a depth ~rmax/slices.
Section [julia]
~~~~~~~~~~~~~~~
- likelihood_path: path of the julia code
- likelihood_module: julia module where the likelihood is implemented
- bias_sampler_type: type of sampler for the bias parameters (hmclet,
slice)
- ic_in_julia: whether initial conditions of the MCMC are coded in
julia or choose some random numbers
- hmclet_diagonalMass: where to use a diagonal mass matrix or a full
dense
- mass_burnin: number of MCMC steps in burnin mode
- mass_burnin_memory: number of MCMC steps to store when in burnin mode
- hmclet_maxEpsilon: maximum epsilon for the leapfrog integrator
(~0.002-0.01 depending on likelihood complexity)
- hmclet_maxNtime: maximum number of steps for the leapfrog integrator
(~50-100)
- hmclet_massScale: amount of momentum reshuffling (0.0 = full, 1.0 =
none bad for MCMC)
- hmclet_correlationLimiter: reduce the correlations in the covariance
matrix by some number. Typically the smaller the number the less
reduction with :math:`\simeq 1` reducing the correlation by 2.
Catalog sections
----------------
Basic fields
~~~~~~~~~~~~
- datafile: Text filename holding the data
- maskdata: Healpix FITS file with the mask
- radial_selection: Type of selection function, can be either
"schechter", "file" or "piecewise".
- refbias: true if this catalog is a reference for bias. Bias will not
be sampled for it
- bias: Default bias value, also used for mock generation
- nmean: Initial mean galaxy density value, also used for mock
generation
Halo selection
~~~~~~~~~~~~~~
- halo_selection: Specifying how to select the halos from the halo catalog. Can be ``mass, radius, spin or mixed``. The ``mixed`` represents the combined cuts and can be applied by specifying, eg "halo_selection = mass radius"
- halo_low_mass_cut: this is log10 of mass in the same unit as the
masses of the input text file
- halo_high_mass_cut: same as for halo_low_mass_cut, this is log10 of
mass
- halo_small_radius_cut
- halo_large_radius_cut
- halo_small_spin_cut
- halo_high_spin_cut
Schechter selection function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- schechter_mstar: Mstar for Schechter function
- schechter_alpha: Power law slope of Schechter function
- schechter_sampling_rate: How many distance points to precompute from
Schechter (i.e. 1000)
- schechter_dmax: Maximum distance to precompute Schecter selection
function
- galaxy_bright_apparent_magnitude_cut: Apparent magnitude where data
and selection must be truncated, bright end.
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in data
and selection function, bright end, useful to select different galaxy
populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- zmin: Minimum redshift for galaxy sample, galaxies will be truncated
- zmax: Maximum redshift for galaxy sample, galaxies will be truncated
'File' selection function
~~~~~~~~~~~~~~~~~~~~~~~~~
- radial_file: Text file to load the selection from
The file has the following format. Each line starting with a '#' is a
comment line, and discarded. The first line is a set of three numbers:
'rmin dr N'. Each line that follows must be a number between 0 and 1
giving the selection function at a distance r = rmin + dr \* i, where
'i' is the line number (zero based). Finally 'N' is the number of points
in the text file.
Two possibilities are offered for adjusting the catalog and the
selection together:
- either you chose not to do anything, and take the whole sample and
provided selection. Then you need to specify:
- file_dmin: Minimal distance for selection function and data
- file_dmax: same but maximal distance
- no_cut_catalog: set to false, if you do not set this you will get
an error message.
- or you want ares to preprocess the catalog and then you need:
- zmin
- zmax
- galaxy_faint_apparent_magnitude_cut: Same for faint end.
- galaxy_bright_absolute_magnitude_cut: Absolute magnitude cut in
data and selection function, bright end, useful to select
different galaxy populations
- galaxy_faint_absolute_magnitude_cut: Similar but faint end
- no_cut_catalog: (not necessary, as it defaults to true)

19
docs/source/user/inputs/Create_config-file.inc.rst Normal file
View file

@ -0,0 +1,19 @@
How to create a config file from python
=======================================
This page is about running the ``gen_subcat_conf.py`` script under
``scripts/ini_generator`` in ares. For an explanation of the config-file itself, see :ref:`here<configuration_file>`.
Config-file for 2M++ and SDSS(MGS)
----------------------------------
The folder containing the scripts and the ini files below is located in ``$SOURCE/scripts/ini_generator``. Steps to generate the config-file are the following:
- Manipulate ``header.ini`` for your needs
- (If needed) alter template files (``template_sdss_main.py``,
``template_2mpp_main.py`` and ``template_2mpp_second.py``) for the cutting and adjusting of data
- To create ini file, run this command:
.. code:: bash
python gen_subcat_conf.py  --output NAME_OF_OUTPUT_FILE.ini --configs template_sdss_main.py:template_2mpp_main.py:template_2mpp_second.py  --header header.ini

64
docs/source/user/inputs/HDF5_catalog_format.inc.rst Normal file
View file

@ -0,0 +1,64 @@
HDF5 catalog format
===================
Passing in the :ref:`ini file<configuration_file>` the following
option in the catalog sections:
- ``dataformat=HDF5``
- ``datakey=KEY``
one can load from an HDF5 file the needed data for a catalog. The data
are taken from the entry "KEY" in the HDF5. This allows to store several
catalogs at the same time in the same file.
HDF5 catalog format
-------------------
The catalog must have the following columns:
- id (``unsigned long int`` compatible)
- phi (longitude in radians, ``double`` compatible)
- theta (latitude in radians, ``double`` compatible)
- zo (observed redshift, dimensionless, ``double`` compatible)
- m (apparent magnitude, ``double`` compatible)
- M_abs (absolute magnitude, optional, ``double`` compatible)
- z (redshift, optional, ``double`` compatible)
- w (weight, ``double`` compatible, should be 1)
HDF5 halo catalog format
------------------------
- id (``unsigned long int`` compatible)
- Mgal (mass, ``double`` compatible)
- radius (``double`` compatible)
- spin (``double`` compatible)
- posx (x position Mpc, ``double`` compatible)
- posy (y position Mpc, ``double`` compatible)
- posz (z position Mpc, ``double`` compatible)
- vx (velocity x, km/s, ``double`` compatible)
- vy (velocity x, km/s, ``double`` compatible)
- vz (velocity x, km/s, ``double`` compatible)
- w (weight, ``double`` compatible, should be 1)
An example converter can be found hereafter:
.. code:: python
import numpy as np
import h5py as h5
# Load text data file
data0 = np.loadtxt("./halo.txt", dtype=[("id",int),("Mgal", float),("radius",float),("spin",float),("posx",float),("posy",float),("posz",float),("vx",float),("vy",float),("vz",float)])
# Build a new one with a weight column
data = np.empty(data0.size, dtype=[("id",int),("Mgal", float),("radius",float),("spin",float),("posx",float),("posy",float),("posz",float),("vx",float),("vy",float),("vz",float),("w",float)])
for n in data0.dtype.names:
data[n] = data0[n]
# Set the weight to one
data['w'] = 1
# Write the hdf5
print("Writing catalog")
with h5.File("halo.h5", mode="w") as f:
f['data'] = data

27
docs/source/user/inputs/Radial_selection.inc.rst Normal file
View file

@ -0,0 +1,27 @@
Radial selection format
=======================
The file format for radial selection is the following:
- First line is : ``rmin dr numPoints``
- ``rmin`` is the minimal distance of the completeness (the first point
in the following)
- ``dr`` is the space between two samples
- ``numPoints`` is the number of points
- Comment line start with ``#``
- All following lines are completeness
For example, the following would create a completeness equal to one
between :math:`100 \, \mathrm{Mpc} \, h^{-1}` and :math:`4000 \, \mathrm{Mpc} \, h^{-1}`:
.. code:: text
# some comment
100 800 5
1
1
1
1
1

34
docs/source/user/inputs/Text_catalog_format.inc.rst Normal file
View file

@ -0,0 +1,34 @@
Text catalog format
===================
It is determined by the function ``loadGalaxySurveyFromText`` in
``libLSS/data/survey_load_txt.hpp`` (ARES git tree)
**[Galaxy Survey]**
For galaxy survey, the standard catalog format includes 7-8 columns. The meaning of each column, from left to right, is listed below.
- galaxy id
- phi: longitude, :math:`2\pi >= \phi >= 0` [rad].
- theta: latitude, :math:`\pi/2 >= \theta >= -\pi/2` [rad].
- zo: total observed redshift, to be used with photo-z.
- m: apparent magnitude.
- M_abs: absolute magnitude, not really used as it is derived from
other quantities.
- z: redshift, used to position the galaxies, cosmology is used to
transform this to comoving distance at the moment.
- w: weight, used as a multiplier when creating the grid of galaxy
distribution.
**[Dark Matter Simulation]**
For Dark Matter simulation, the standard catalog format includes 10
columns. The meaning of each column, from left to right, is listed
below.
- halo id
- halo mass: given in unit of solar mass
- halo radius
- halo spin
- x, y, z: comoving coordinates
- vz, vy, vz: velocities