Commit graph

102 commits

Author SHA1 Message Date
Richard Stiskalek
c14be720b5
Particle dumping (#49)
* Add paths

* Add dumping script

* Remove snap from density fields

* Add progress statement

* Update import ordering

* Update nb
2023-04-28 13:57:23 +01:00
rstiskalek
b0752efdf4 fix indexing bug 2023-04-27 09:21:00 +01:00
Richard Stiskalek
f48eb6dcb0
Dump radial profile information (#48)
* add radial position path

* pep8

* Add basic fit profile dumping

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* Update TODO

* Fix parts is None bug

* Update nb
2023-04-27 00:18:30 +01:00
rstiskalek
1a115f481d Rename file 2023-04-21 00:40:11 +01:00
Richard Stiskalek
04119a5314
Update initial matching & overlaps (#47)
* pep8

* fix convention

* Update script

* enforce optimisation boundaries to be finite

* Update TODO

* Remove sky matching

* FIx a small bug

* fix bug

* Remove import

* Add halo fitted quantities

* Update nbs

* update README

* Add load_initial comments

* Rename nbs

* Delete nb

* Update imports

* Rename function

* Update matcher

* Add overlap paths

* Update the matching script

* Update verbosity

* Add verbosity flags

* Simplify make_bckg_delta

* bug fix

* fix bug
2023-04-21 01:35:06 +02:00
Richard Stiskalek
39b3498621
Correct fitting script for both clumps and halos (#46)
* Minor typos

* fix minor bugs

* pep8

* formatting

* pep8

* Fix minor bugs

* New path & pep8

* add splitt

* Updates

* Improve calculation within radius

* pep8

* pep8

* get the script working

* Add matter overdensity

* Add m200m to the script

* Fix looping bug

* add parents support

* add import

* Optionally concatenate velocities

* Make optional masking

* Ignore the error message

* Start reading in raw data

* Fix cat reading

* Additional units conversions

* Add clump reading

* Fix indexing

* Remove old comment

* Remove old comment

* set npart to 0 instead of overflow from NaN

* fix docs

* rm boring stuff

* Remove old stuff

* Remove old stuff

* Remove old comment

* Update nb
2023-04-19 16:39:35 +02:00
Richard Stiskalek
fdb0df8d4c
Add mmain and other major updates (#44)
* Move paths to a separate file

* Add mmain reader

* Add a verbosity flag

* Fix imports

* Fix bug

* Rename files

* Return ultimate parents

* Add script to generate mmain

* Remove mmain path

* edit path

* Add mmain path

* Change function name

* Rename function

* Turn off verbose

* Fix list requirement

* Edit init match paths

* Fix init pathing

* Edit paths docs

* Edit dumpdir name

* Rename path

* Fix split paths

* Remove unused import

* Add comment

* Update readme

* remove read mmain

* Rename haloatalogue

* Fix minor bugs

* Update nbs

* Add create directory option

* Move split jobs

* Move spliot jobs

* Remove splitting

* Add import

* Edit script

* Deeper split folder

* Fix paths bug

* Rename catalogue

* Rename Catalogue

* Add new clumpread

* Edit paths

* add knn paths

* Update commenting

* Update imports

* Add more conversions

* Update temp file

* Add a note

* Add catalogue

* Cooment

* Update TODO

* Update script

* add nb

* Update

* pep8

* edit paths & pep8

* Fix knn auto paths

* add paths docs

* Add auto and cross knn paths

* Add new paths

* Simplify tpcf reading

* pep8 patch

* update readme

* Update progress

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* pep8

* Pep 8 and restructure

* add lambda spin

* add clump and halo

* add checks

* Edit halo profile fit

* Update gitignore

* backup script
2023-04-18 11:02:36 +02:00
Richard Stiskalek
e0d3854277
Add cleaner paths (#43)
* ignore venv

* ignore pylians if unavaible

* Edit loading paths

* Edit paths

* Stop having default paths

* Add nb

* Add glamdring paths

---------

Co-authored-by: Richard Stiskalek <dc-stis1@login8b.pri.cosma7.alces.network>
Co-authored-by: Richard Stiskalek <dc-stis1@login8a.pri.cosma7.alces.network>
2023-04-14 23:24:52 +01:00
Richard Stiskalek
1344fa40b6
Add 2PCF calculation (#42)
* Add 2PCF calculation

* Add 2PCF reader

* Add tpcf scripts

* Add random state

* Fix marked typo

* Fix marks and more randoms

* Stop calculating projected

* Add mean 2PCF

* Remove pimax

* Edit submit script

* Update nb

* Add corrfunc check
2023-04-14 23:05:48 +01:00
Richard Stiskalek
5784011de0
kNN-CDF secondary halo bias (#40)
* Add seperate autoknn script & config file

* edit ics

* Edit submission script

* Add threshold values

* Edit batch sizign

* Remove print

* edit

* Rename files

* Rename

* Update nb

* edit runs

* Edit submit

* Add median threshold

* add new auto reader

* editt submit

* edit submit

* Edit submit

* Add mean prk

* Edit runs

* Remove correlation file

* Move split to clutering

* Add init

* Remove import

* Add the file

* Add correlation reading

* Edit scripts

* Add below and above median permutation for cross

* Update imports

* Move rvs_in_sphere

* Create utils

* Split

* Add import

* Add normalised marks

* Add import

* Edit readme

* Clean up submission file

* Stop tracking submit files

* Update gitignore

* Add poisson field analytical expression

* Add abstract generators

* Add generators

* Pass in the generator

* Add a check for if there are any files

* Start saving average density

* Update nb

* Update readme

* Update units

* Edit jobs

* Update submits

* Update reader

* Add random crossing

* Update crossing script

* Add crossing with random

* Update readme

* Update notebook
2023-04-09 20:57:05 +01:00
Richard Stiskalek
8d34b832af
Improve paths and documentation (#38)
* pep8

* style issue

* Documentation edits

* Remove old files

* Remove more old files

* Doc & stop setting snap and nsim in paths

* add nsnap, nsim support

* Remove blank space

* docs

* Docs edits

* Reduce redudant code

* docs

* Documentation

* Docs

* Simplify

* add nsnap nsim arguments

* Remove redundant docs

* Fix typos

* Shorten catalogue

* Remove import

* Remove blank line

* Docs only edits

* Rearrange imports

* Remove blank space

* Rearrange imports

* Rearrange imports

* Remove blank space

* Update submission scritp

* Edit docs

* Remove blank line

* new paths

* Update submission scripts

* Update file handling

* Remove blank line

* Move things around

* Ne paths

* Edit submission script

* edit paths

* Fix typo

* Fix bug

* Remove import

* Update nb
2023-04-04 15:22:00 +01:00
Richard Stiskalek
522ee709c9
Joint kNN-CDF calculation (#36)
* Add joint kNN CDF

* add jointKNN calculation

* change sub script

* Update readme

* update sub

* Small changes

* comments

* update nb

* Update submisison script
2023-04-01 11:16:11 +01:00
rstiskalek
8d933e7ba9 Add batch size to submission 2023-04-01 07:15:44 +01:00
Richard Stiskalek
63ab3548b4
kNN-CDF implementation (#34)
* Rewrite doc

* add kNN

* edit loading of samples with no init

* Add verbosity flag

* add KNN submission script

* do not make peaked cdf by default

* Add submit script

* stop ignore sh

* Add mass thresholding

* Edit gitignore

* edits

* Space points in logspace

* Calculate for all ICs

* Update TODO

* Add dtype support

* Update readme

* Update nb
2023-03-31 18:13:41 +01:00
rstiskalek
4d7827006a test 2023-03-27 13:38:01 +01:00
Richard Stiskalek
5dd8c668fa
Gaussian smoothing of density fields (#33)
* Simplify smoothing support and looping over nonzero

* Simplify comments

* add now()

* add cat length

* add smoothed calculation

* add smoothing

* Add sorting

* Edit what is ignored

* Move notebooks

* Add nonsymmetric smoothed overlap

* Update NB

* Add support for reading in the smoothed overlap

* Switch to the true overlap definition

* Reader of the true overlap

* rem occups

* Import moved to a class

* Move definition

* Edit submission script

* Update to account for the new definition

* backup nb

* Switch back to properly initialising arrays

* Fix addition bug

* Update NB

* Fix little bug

* Update nb
2023-03-27 09:22:03 +01:00
Richard Stiskalek
153f1c0002
Overlap reader thresholds (#31)
* Improve data

* Add comment

* Update how KNN is called

* Bring back indices

* New function output

* return catx["index"]

* Remove unnecessary arguments

* Remove useless arguments

* Rename output

* thin up catalogues

* Add thresholding

* Update README
2023-03-15 20:55:50 +00:00
Richard Stiskalek
f2cef73977
Final overlap definition update.. (#29)
* Delete whitespace

* Update overlap definition

* Add documentation

* add overlap args

* Make the background array store only high density region

* Add nsims to args

* Remove conc as done inside the func

* Simplify the overlap calculation

* Rename variable

* Just some docs

* Docs

* Correct overlap definition

* Undo debug comment

* Remove old debug code

* Add prob of no match and exp couunterpart mass

* docs

* Update the overlap definition
2023-03-09 14:55:38 +00:00
Richard Stiskalek
1c859dbdac
Overlap reader (#28)
* Edit output path

* remove matching of many sims

* Add a class skeleton

* Add a setter for only nsim

* Deletei mport

* Simplify cross and delete combined halo catalogue

* Simplify reading in catalogues

* Add distance calculation

* Add mass ratio

* Clean up the submission script

* Add the summer overlap

* Docs

* Add array copying

* Add import

* Add resampling mean
2023-02-08 12:05:32 +00:00
Richard Stiskalek
8dea3da4de
Speed up overlap (#27)
* Edit improt

* Simplify patch size calculation

* Add patch size percentiles

* Add various percentiles

* Remove  comment

* Update TODO

* Change to 95th percentile

* Add import

* Add KNN properties

* Add new matching initial condition

* Add import

* Remove import

* Add fast neighbours option

* Further edits to fast neighbours

* add imports

* add new overlap calculation and non-zero things

* Remove print

* Clean up code

* Fix small bug

* Remove comment

* Add run single cross match

* change values

* Edit hyperparams

* Add comment

* Add the argument parser

* Add new lagpatch calc

* New lagpatch calc

* Delete old patch definitions

* Make clump dumping once again optional

* Add lagpatch to the catalogue

* Edit print statement

* Fix small bug

* Remove init radius

* Change to lagpatch key

* Fix a small bug

* Fix little bug
2023-02-05 11:46:19 +00:00
Richard Stiskalek
beb811e84c
Switch initial radius definition (#26)
* Add search for neighbours at z = 0

* Add initial snapshot KNN

* Add initi search either z =0 or  z = 70

* Add import

* Add clumps_pos2cell

* Add function argument

* Add import

* Add spherical overlap and speed up make_delta

* Add clump limits calculation

* Sped up make_delta

* Add patch size conversion

* Add patch sizes

* Add patch size calculation

* Force catalogues to be in float32

* Optimised script

* Option to remove points with no overlap

* Add spherical aproximate overlap

* Fix subboxing bug

* Remove print diagnostics

* Add Lagrangian patch size

* Add patch size documentation

* Move when clumpsx converted to int

* Edit docs

* Remove spherical overlap

* New Langrangian patch calculation
2023-01-31 15:09:35 +00:00
rstiskalek
88a109157d Change file naming convention 2023-01-17 11:46:07 +00:00
Richard Stiskalek
f0720243a2
Updates to the submission script (#22) 2023-01-01 21:43:52 +00:00
Richard Stiskalek
2e99b901ac
Environmental properties (#20)
* rm get_positions

* Add comment

* add halfwidth func

* Update docs

* Add imprt

* Evaluate multiple fields simulatenously

* add halfwidth selection

* Change order of grav field and tensor field

* Add gravitational field norm

* Add eigenvalue calculation

* Sorted eigenvalues

* add init script

* add progress

* Add surveys

* Add more survey flexibility

* Minor changes

* add survey names

* rm name

* Fix list bug

* Fig bugs when running the script

* add phi to dtype

* fix dump bug

* Add comment

* Add smoothing options

* Add further comment

* Update TODO
2022-12-31 17:46:05 +00:00
Richard Stiskalek
3dc559fec1
Overlap calculation (#19)
* Rm comment

* Add new fixed particle overlap

* Fix overlap calculation

* Update docs

* Add new overlapper support in matcher

* add filter optin

* Add the real space filter.

* Change filtering to real space only

* Update TODO

* add high-resolution switch

* add cross script

* Remove comment

* Add smoothing option
2022-12-27 18:32:12 +01:00
Richard Stiskalek
18f09767f4
Overlap calculation (#18)
* Move cosine similarity out

* Basic overlap calculation

* add overlap import

* Add clump0 dict path

* Update README

* fix path bug

* Save as dict instead

* Change format to array of arrays

* Update paths

* Take fewer ICs for now

* Change to structured array

* Add overlap calculation

* Start saving IDs

* Add a blank space

* Update TODO
2022-12-19 11:58:22 +01:00
Richard Stiskalek
13a9d11afe
Particle init CM separation (#17)
* Rename

* rename module

* add new_ic_ids

* Add to_new support

* rm unnecessary comment

* Remove JUlia!

* add new package attempt

* add dump script

* simplify positions getter

* Remove unused code

* change dumpdirs

* update project

* Add sorting

* fix indexing bug

* add comment

* Add particle dumping

* Delete blank line

* Delete Julia things

* Update README

* add working script

* fix assignment bug

* fix bug

* save temp to temp folder

* fix indexing bug & path

* do not always crreate folder

* Rename file

* add initcm reader

* add x0, y0, z0 transforms

* add initcm to halo cat

* docs update

* add initial KNN

* add attempt at init match

* add initial fiducial radius

* rm blank space

* Add further KNN Init pos support

* Add init radius support

* Remove message about the bug

* Get rid of an error msg

* purge mention of knn_init

* Add init CM distance calc
2022-12-16 11:18:16 +01:00
Richard Stiskalek
c3c686fa60
Power spectrum stuff (#16)
* add autok1d

* update TODO

* add basic cross pk

* del blank line

* fix small bugs

* fix loop bug

* rm nbs

* edit docstring

* Update TODO

* get rid of verbose flag

* take all ICs

* attempt better memory

* simplify ASCII dump

* rm verbose statement to get rid of bugs!

* close fortran files

* add MAS options

* add boxsize setter

* Move comment up

* paths chagne

* Update TODO

* Update TODO

* Forgotten units in Xpk

* Attempt at fixing the distance bug

* Update TODO

* Update TODO

* Remove comment

* add summary reader

* Update gitignore
2022-12-12 14:42:44 +00:00
Richard Stiskalek
1bb4255874
SDSS galaxies and catalogs rework (#15)
* add SDSS skeleton

* rm nb

* update gitignore

* add docs

* add survey volume

* add masking

* add SDSS mask

* add SDSS comments

* add reference

* rm

* move planck to fits clusters

* Rm comment

* update masking routines

* rm masks

* Update docs

* rm data

* make MCXC fits surv

* change MCXC name

* Move to extrasapce

* Update paths
2022-12-05 11:36:41 +00:00
Richard Stiskalek
91beb4df50
Angular momentum (#14)
* add ang mom

* Update TODO

* add ascii dump script

* add comment

* Updating TODO

* add angmom

* add garbage collector
2022-11-30 16:12:39 +00:00
rstiskalek
4e87a96b97 fix bug(again) 2022-11-25 12:42:49 +00:00
rstiskalek
a3c7d03ceb fix bug 2022-11-25 12:28:41 +00:00
rstiskalek
377d6ebb7e add lambda200c support 2022-11-25 12:24:06 +00:00
Richard Stiskalek
161c27d995
Particle match & file system & phase space (#11)
* Create file system

* add doc

* add n_sim n_snap directly to paths

* Move things to a single particle reader for consistency

* add docstring

* add srdcir, dumpdir and mmain_path

* make boxunits work with paths

* switch to using paths

* add tempdumpdir

* rm dependence on old functions

* rm comment

* rm unused import

* go back to all imports

* fix import bug

* rm dependence on old functions

* modernize code!

* fix typo

* fix typo

* update fits to new data structureing

* change docs

* add julia repo

* add setup

* add install commands

* ignore install files

* add array flattening

* update dependene

* add positions reader

* update manifest and projects

* add func

* update gitignore

* pos matching progress

* move file

* rm comment

* add velocities getter

* fix bug

* fix name bug

* fix path bug

* fix args func

* add redshift calculation to catalogues

* add shortcut to set n_sim and n_snap

* if cond bug

* add the cosine similarity

* add verbosit iterator

* add docs

* update README

* update README

* update README
2022-11-25 10:44:08 +00:00
Richard Stiskalek
c748c87e45
New filesystem, basic neighbours calculation (#10)
* simplify Planck catalogue

* add MCXC and base survey

* Add 2MPP classes

* move match to MCXC to Planck

* add halo catalogue

* rm comment

* rm unused imports

* Move conversion to box

* add min mass

* Run on all simulations

* rm old function

* add combined catalogue

* add halo positions

* add knn neighbors

* set to 5 sims for testing

* add docstring

* Switch to neighbours in a catalogue

* rename io to read

* fix indentation

* rename to read

* io -> read

* add import

* add RealisationMatcher

* io -> read

* add docstrings

* add search_sim_indiices

* update todo

* keep make_cat at 10 for now

* update nb
2022-11-22 10:00:01 +00:00
rstiskalek
f5bf04c46e update nb 2022-11-20 14:11:35 +00:00
rstiskalek
3352b06c1c add units 2022-11-20 14:11:25 +00:00
rstiskalek
1d9789c08e add vx 2022-11-20 14:11:19 +00:00
rstiskalek
06d3fede95 add max snapshot 2022-11-14 16:24:53 +00:00
rstiskalek
76349bab3f add MPI and do for all boxes 2022-11-14 15:52:52 +00:00
Richard Stiskalek
17badf2652
X-ray data (#9)
* add xray reading

* add import

* add more comments

* add data

* add nb

* add MCXC shortcut

* add match indxs to Planck of MCXC

* add import

* update TODO

* update readme

* add famous clusters

* add 2mpp_groups

* shorten paths

* add 2M++ groups

* add import

* Update TODO
2022-11-10 07:40:44 +00:00
rstiskalek
d69012424e simplify loading 2022-11-06 20:55:49 +00:00
rstiskalek
9291414572 update nbs 2022-11-06 20:53:58 +00:00
Richard Stiskalek
d32eb5c134
Data loading shortcut (#7)
* add load_processed

* update TODO
2022-11-06 10:26:24 +00:00
Richard Stiskalek
7f58b1f78c
JAX and fix conc (#6)
* change to log10 initlogRs

* add uncertainty

* add check if hessian negative

* update TODO

* update TODO

* output the error too

* save e_logRs

* add concentration calculation

* calcul concentration

* missed comma in a list

* Update TODO

* fix bug

* add box units and pretty status

* make uncertainty optional

* add BIC function

* remove BIC again

* add new overdensity calculation

* change defualt values to max dist and mass

* change to r200

* new halo find

* speed up the calculation

* add commented fucn

* update TODO

* add check whether too close to outside boundary

* Update TODO

* extract velocity as well

* calculate m200 and m500

* update nb

* update TODO
2022-11-05 21:17:05 +00:00
Richard Stiskalek
ba98ecc299
Mass enclosed in a sphere calculation (#5)
* rename nb

* add merge func

* rm bug..

* add static methods

* save rmin, rmax

* add properties for box units

* move radial info to the clump

* add Npart explicit

* move where rmin, rmax obtained

* add box cosmo

* add __getattr__

* add comments in units

* add enclosed spherical mass

* rm unused variables

* add clump mass setter

* add the halo index

* add enclosed overdensity

* add enclosed_overdensity

* simplify loss func

* opt result to nan

* change Rs to log10

* change back to attribs

* add H0 and h props

* add setattrs

* Remove global constants

* move Msuncgs above

* add crit density

* add dots

* add r200c and r500c

* add M200 and M500

* make lowercase

* output r200, r500, m200, m500

* update TODO

* update README

* fit NFW only up to r200

* add r178, m178

* add m178, r178

* update TODO
2022-11-01 10:10:54 +00:00
Richard Stiskalek
8a56c22813
Within halo work and NFW fit (#4)
* add listing of snapshots

* change distance to comoving

* ignore cp files

* rename nb

* add str to list

* add NFW profile shapes

* add fits imports

* Rename to Nsnap

* in clumps_read only select props

* make clumpid int

* expand doc

* add import

* edit readme

* distribute halos

* add profile & posterior

* add import

* add import

* add documentation

* add rvs and init guess

* update todo

* update nb

* add file

* return end index too

* change clump_ids format to int32

* skeleton of dump particle

* update nb

* add func to drop 0 clump indxs parts

* add import

* add halo dump

* switch to float32

* Update TODO

* update TODO

* add func that loads a split

* add halo object

* Rename to clump

* make post work with a clump

* add optimiser

* add Nsplits

* ignore submission scripts

* ignore .out

* add dumppath

* add job splitting

* add split halos script

* rename file

* renaem files

* rm file

* rename imports

* edit desc

* add pick clump

* add number of particles

* update TODO

* update todo

* add script

* add dumping

* change dumpdir structure

* change dumpdir

* add import

* Remove tqdm

* Increase the number of splits

* rm shuffle option

* Change to remove split

* add emojis

* fix part counts in splits

* change num of splits

* rm with particle cut

* keep splits

* fit only if 10 part and more

* add min distance

* rm warning about not set vels

* update TODO

* calculate rho0 too

* add results collection

* add import

* add func to combine splits

* update TODO

* add extract cols

* update nb

* update TODO
2022-10-30 20:16:56 +00:00
rstiskalek
a47847cb7c more renaming 2022-10-20 23:37:33 +01:00
Richard Stiskalek
942c36b142
Box units conversion (#3)
* linting

* fix long line

* rename nb

* rename import

* add group catalog

* move imports out of functions

* add array_to_structured

* add references

* fix subsampling

* fix coord bug

* add 2M++ dists

* save nb

* fix comment

* add snapshot path

* add snapshot path

* add read_info

* Move transforms

* add import radec

* expand docs

* Move flipcols

* update nb

* add flip_cols

* create file

* add blank line

* Move units transfs

* add blank line

* add units import

* rm imports

* add import

* add box_units

* add comments
2022-10-20 23:28:44 +01:00
Richard Stiskalek
53a0629d90
Basic matching (#2)
* add recarray manipulations

* add cart to radec

* add behav so x can be a list

* add import

* create empty files

* ignore plots file

* add planck data

* add read_mmain file

* add cols_to_structured import

* use cols_to_structured

* add cols_to_structued

* add read_mmain import

* add reading planck

* add mass conversion

* add brute force separation calculation

* update nb

* rename & int dtype

* add func to get csiborg ids

* add list to nd array conversion

* add utils

* rename file

* add 2M++

* add read 2mpp

* add 2mpp shortcut

* add randoms generator

* Change range of RA [0, 360]

* fix ang wrapping

* add code for sphere 2pcf

* rm wrapping

* optionally load only a few borgs

* update nb
2022-10-18 19:41:20 +01:00
rstiskalek
4d24895d2f update nb 2022-10-11 16:35:21 +01:00
rstiskalek
54221a8ff5 first commit 2022-10-02 11:12:45 +01:00