diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 0000000..e43cf54 --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,6 @@ +include README.md +include CMakeLists.txt +recursive-include python_tools *.py +recursive-include c_tools * +recursive-include external * +recursive-include zobov * \ No newline at end of file diff --git a/README.md b/README.md index 440b210..85c0584 100644 --- a/README.md +++ b/README.md @@ -48,123 +48,130 @@ VIDE is licensed under the GNU Public License. See the LICENSE file for further VIDE requires several dependencies for building and running the software. These dependencies are listed below. ### Required Packages -- Python 3.8 +- **Python 3.8** - **GCC** and **G++** (for compiling C/C++ code) -- [CMake](https://cmake.org/) (version 3.20 or higher) -- [satrapy](https://pypi.org/project/satrapy/) (Python package) -- Standard scientific Python packages: `scipy`, `pandas`, `matplotlib`, `PySide2` +- **[CMake](https://cmake.org/)** (version 3.20 or higher) +- **[satrapy](https://pypi.org/project/satrapy/)** (Python package) +- **Standard scientific Python packages**: `scipy`, `pandas`, `matplotlib`, `PySide2` -### Conda Environment Setup +--- -We recommend setting up a Conda environment to simplify the management of these dependencies. Follow these steps: +## Installation Methods -#### 1. Install Dependencies Using Conda (Linux/MacOS) +There are three primary ways to install and build VIDE: using Conda build, manually building in a Conda environment, or using a system with `pip` and system dependencies. -Create a new Conda environment with the required dependencies, you can use `micromamba` (a faster alternative to Conda): +### 1. Install using Conda Build + +Run the `conda-build` command to build the package: ```bash -micromamba env create -y -n vide_python3.8 python=3.8.12 scipy pandas matplotlib PySide2 cmake=3.20 gcc=13.2 gxx m4 -c conda-forge +conda install conda-build # Install conda-build if not already installed +conda build . ``` -#### 2. Set Up Environment Variables +After the build is complete, you can install the built package: ```bash -echo "export CC=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/gcc" > ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh -echo "export CXX=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/g++" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh -echo "export LIBRARY_PATH=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/lib" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh -``` - -Activate the environment again to apply the changes: - -```bash -micromamba activate vide_python3.8 -``` - -#### 3. Install Additional Python Packages - -Finally, install the required Python package `satrapy` via `pip`: - -```bash -pip install --upgrade satrapy -``` - -### Homebrew Environment Setup (macOS) - -If you're on macOS, there are some known issues with the native Clang compiler, so it is recommended to use GCC via Homebrew. - -#### 1. Install GCC via Homebrew - -```bash -brew install gcc -export CC=/usr/local/bin/gcc-10 -export CXX=/usr/local/bin/g++-10 -``` - -Ensure the `gcc-10` version matches the version installed by Homebrew. - -#### 2. Install Required Python Packages - -Set up a virtual environment and install dependencies as follows: - -```bash -python3 -m venv --system-site-packages $PLACE_OF_VENV -source $PLACE_OF_VENV/bin/activate -pip install scipy pandas matplotlib PySide2 -``` - -Install the required `satrapy` package: - -```bash -pip install --upgrade satrapy +conda install --use-local vide ``` --- -## Package Build and Installation +### Alternative: Manual Build in Conda Environment -Once the environment is set up, you can build and install VIDE. +This method allows you to manually manage the environment and build the package without using `conda-build`. +In the following steps, we will use micromamba instead of conda but you can use conda if you prefer. +Micromamba is a faster and more lightweight alternative to conda. -### 1. Build the Package +#### Step 1: Create a Conda Environment -To build the package, run: +Create a new Conda environment with the required dependencies: ```bash -python setup.py build +micromamba env create -y -n vide_python3.8 python=3.8.12 scipy pandas matplotlib PySide2 cmake=3.20 gcc=13.2 gxx m4 -c conda-forge +echo "export CC=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/gcc" > ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh +echo "export CXX=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/g++" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh +echo "export LIBRARY_PATH=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/lib" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh +conda activate vide_python3.8 ``` -This process may take some time and will download any missing dependencies automatically. Make sure you have enough resources available for the build process. +#### Step 2: Install Additional Python Packages -### 2. Install the Package - -After building the package, install it by running: +Install `satrapy` and other required packages via `pip`: ```bash +pip install --upgrade satrapy +``` + +#### Step 3: Build the Package + +Build the package using `setup.py`: + +```bash +python setup.py build_ext --inplace python setup.py install ``` -### 3. Verify the Installation +--- -After installation, you can verify that VIDE is correctly installed by running the following command: +### 3. Install using System Dependencies and `pip` + +This method requires system-level installation of dependencies such as `gcc`, `cmake`, and Python packages via `pip`. + +#### Step 1: Install System Dependencies + +You will need to install the following system dependencies: + +- **GCC** (13.2 or higher) +- **G++** (13.2 or higher) +- **CMake** (3.20 or higher) +- **Python 3.8** or higher +- Required system libraries: `libgomp`, `libgcc-ng`, `libpthread` + +On a Debian-based system, you can install these with: + +```bash +sudo apt update +sudo apt install build-essential cmake python3 python3-dev python3-pip +``` + +#### Step 2: Install Python Dependencies + +Use `pip` to install Python dependencies: + +```bash +pip install scipy astropy healpy extension-helpers netCDF4 +``` + +#### Step 3: Build and Install with `pip` + +Run the following to install the package: + +```bash +pip install . +``` + +If you encounter build issues due to isolation, use the `--no-build-isolation` flag: + +```bash +pip install --no-build-isolation . +``` + +--- + +## Package Testing + +### Check Installation + +Verify that the package is installed correctly: ```bash python -m void_pipeline ``` -You should see the output: - -``` -Usage: ./generateCatalog.py parameter_file.py -``` - ---- - -## Package Test - -VIDE provides tools for both observational data and simulations. Below are steps to test the installation. - ### Testing with Observational Data -To test the pipeline with observational data: +To test with observational data, run: ```bash cd python_tools/void_pipeline/datasets diff --git a/meta.yaml b/meta.yaml new file mode 100644 index 0000000..cf4209a --- /dev/null +++ b/meta.yaml @@ -0,0 +1,31 @@ +package: + name: vide + version: "2.0" + +source: + path: .. + +requirements: + build: + - gcc_linux-64=13.2 + - gxx_linux-64=13.2 + - cmake=3.20 + - cython + - numpy + host: + - python + - scipy + - astropy + - healpy + - extension-helpers + - netCDF4 + +build: + script: | + python setup.py build_ext --inplace + python setup.py install + +about: + home: "https://bitbucket.org/cosmicvoids/vide_public/wiki/Home" + license: "GPL-3.0" + summary: "The VIDE pipeline analysis for Cosmic Voids" diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..89ccdca --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,30 @@ +[build-system] +requires = ["setuptools", "wheel", "cython", "cmake", "scipy", "astropy", "healpy", "extension-helpers", "netCDF4"] +build-backend = "setuptools.build_meta" + +[project] +name = "vide" +version = "2.0" +description = "The VIDE pipeline analysis for Cosmic Voids" +readme = "README.md" +license = { file = "LICENSE" } +keywords = ["cosmology", "Cosmic Voids", "Voids", "Void Finder", "Void Analysis"] +requires-python = ">=3.7" +classifiers = [ + "Intended Audience :: Developers", + "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)", + "Natural Language :: English", + "Programming Language :: C", + "Programming Language :: C++", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: Implementation :: CPython" +] + +dependencies = [ + "scipy", + "astropy", + "extension-helpers", + "netCDF4", + "healpy" +] diff --git a/python_tools/setup.py b/python_tools/setup.py index a8ba0bf..bcf58e7 100644 --- a/python_tools/setup.py +++ b/python_tools/setup.py @@ -17,20 +17,36 @@ # with this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. #+ -from distutils.core import setup -from distutils.extension import Extension -from Cython.Distutils import build_ext +from setuptools import setup, Extension +from setuptools.command.build_ext import build_ext import numpy as np import os -VOID_GSL=os.environ.get('VOID_GSL') +# Optional: Use VOID_GSL environment variable if needed later in the build process +VOID_GSL = os.environ.get('VOID_GSL') + +# Define extensions (you can add Cython extensions here if needed) +extensions = [ + # Example of a Cython extension (uncomment and modify if needed) + # Extension('your_extension_name', sources=['your_source_file.pyx'], include_dirs=[np.get_include()]) +] setup( name='vide', version='1.0', -# cmdclass = {'build_ext': build_ext}, - include_dirs = [np.get_include()], - packages= - ['vide','vide.backend','vide.apTools', 'vide.voidUtil', - 'vide.apTools.profiles','vide.apTools.chi2',], + include_dirs=[np.get_include()], # Add NumPy include dirs + packages=[ + 'vide', + 'vide.backend', + 'vide.apTools', + 'vide.voidUtil', + 'vide.apTools.profiles', + 'vide.apTools.chi2', + ], + ext_modules=extensions, # Add extensions if needed + cmdclass={'build_ext': build_ext}, # Use setuptools build_ext for Cython + install_requires=[ + 'numpy', # Ensure NumPy is installed + 'cython', # Ensure Cython is installed for building extensions + ], ) diff --git a/setup.py b/setup.py index a264060..aa98cb3 100644 --- a/setup.py +++ b/setup.py @@ -17,222 +17,120 @@ # with this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. #+ -import stat + import os import sys import shutil -from sysconfig import get_config_var -from distutils.command.install_data import install_data -from distutils.command.build_scripts import build_scripts +import stat import pathlib -from setuptools import find_packages, setup, Extension, Command +import struct +from sysconfig import get_config_var +from setuptools import find_packages, setup, Extension from setuptools.command.build_ext import build_ext from setuptools.command.install_lib import install_lib from setuptools.command.install_scripts import install_scripts -import struct BITS = struct.calcsize("P") * 8 PACKAGE_NAME = "vide" class CMakeExtension(Extension): - """ - An extension to run the cmake build - - This simply overrides the base extension class so that setuptools - doesn't try to build your sources for you - """ - def __init__(self, name, sources=["dummy_extension/empty.c"]): - - super().__init__(name = name, sources = sources) - + super().__init__(name=name, sources=sources) self.SOURCE_DIR = str(pathlib.Path().absolute()) -class InstallCMakeLibsData(install_data): - """ - Just a wrapper to get the install data into the egg-info - - Listing the installed files in the egg-info guarantees that - all of the package files will be uninstalled when the user - uninstalls your package through pip - """ - - def run(self): - """ - Outfiles are the libraries that were built using cmake - """ - - # There seems to be no other way to do this; I tried listing the - # libraries during the execution of the InstallCMakeLibs.run() but - # setuptools never tracked them, seems like setuptools wants to - # track the libraries through package data more than anything... - # help would be appriciated - - self.outfiles = self.distribution.data_files - class InstallCMakeLibs(install_lib): - """ - Get the libraries from the parent distribution, use those as the outfiles - - Skip building anything; everything is already built, forward libraries to - the installation step - """ - def run(self): - """ - Copy libraries from the bin directory and place them as appropriate - """ - self.announce("Moving library files", level=3) + super().run() - self.distribution.run_command("install_data") - self.distribution.run_command("install_scripts") +class InstallScripts(install_scripts): + def run(self): + # Move the generated script from the build directory to the final installation directory + build_temp = self.get_finalized_command('build_ext').build_temp + generated_script = os.path.join(build_temp, 'vide_prepare_simulation') + if os.path.exists(generated_script): + target_script_dir = os.path.join(self.install_dir, 'bin') + os.makedirs(target_script_dir, exist_ok=True) + shutil.copy(generated_script, target_script_dir) super().run() class BuildCMakeExt(build_ext): - """ - Builds using cmake instead of the python setuptools implicit build - """ - def run(self): - """ - Perform build_cmake before doing the 'normal' stuff - """ - for extension in self.extensions: - if extension.name == 'vide': self.package = 'vide' - self.build_cmake(extension) - super().run() def build_cmake(self, extension: Extension): - """ - The steps required to build the extension - """ - self.announce("Preparing the build environment", level=3) + # Ensure absolute paths are used package_dir = os.path.abspath(os.path.join(self.build_lib, 'vide')) - - - extension.build_dir = pathlib.Path(self.build_temp) + extension.build_dir = os.path.abspath(self.build_temp) extension.bin_dir = str(pathlib.Path(os.path.join(extension.build_dir, 'private_install')).absolute()) - SOURCE_DIR = extension.SOURCE_DIR - build_dir = extension.build_dir + SOURCE_DIR = os.path.abspath(extension.SOURCE_DIR) # Absolute path to source directory extension_path = pathlib.Path(self.get_ext_fullpath(extension.name)) - os.makedirs(build_dir, exist_ok=True) + # Make sure necessary directories exist + os.makedirs(extension.build_dir, exist_ok=True) os.makedirs(extension_path.parent.absolute(), exist_ok=True) - cython_code = os.path.join(str(build_dir.absolute()),'mycython') + # Write the cython code if necessary + cython_code = os.path.join(str(extension.build_dir), 'mycython') with open(cython_code, mode="wt") as ff: - ff.write(f"#!{sys.executable}\n" + ff.write(f"#!{sys.executable}\n" "from Cython.Compiler.Main import setuptools_main\n" "setuptools_main()") - os.chmod(cython_code, stat.S_IXUSR|stat.S_IWUSR|stat.S_IRUSR|stat.S_IRGRP) - - # Now that the necessary directories are created, build + os.chmod(cython_code, stat.S_IXUSR | stat.S_IWUSR | stat.S_IRUSR | stat.S_IRGRP) self.announce("Configuring cmake project", level=3) - # Change your cmake arguments below as necessary - # Below is just an example set of arguments for building Blender as a Python module + # Ensure the cmake command has the correct source directory + c_compiler = os.environ.get('CC', get_config_var("CC")) + cxx_compiler = os.environ.get('CXX', get_config_var("CXX")) - PYTHON_bin_package=f"{build_dir.absolute()}/private_install" - - c_compiler=os.environ.get('CC', get_config_var("CC")) - cxx_compiler=os.environ.get('CXX', get_config_var("CXX")) - - self.spawn(['cmake', '-H'+SOURCE_DIR, '-B'+self.build_temp, - '-DENABLE_OPENMP=ON','-DINTERNAL_BOOST=ON','-DINTERNAL_EIGEN=ON', - '-DINTERNAL_HDF5=ON','-DINTERNAL_NETCDF=ON','-DINTERNAL_GSL=ON', + self.spawn(['cmake', '-H' + SOURCE_DIR, '-B' + self.build_temp, + '-DENABLE_OPENMP=ON', '-DINTERNAL_BOOST=ON', '-DINTERNAL_EIGEN=ON', + '-DINTERNAL_HDF5=ON', '-DINTERNAL_NETCDF=ON', '-DINTERNAL_GSL=ON', '-DBUILD_PYTHON=ON', '-DINSTALL_PYTHON_LOCAL=OFF', '-DCOSMOTOOL_PYTHON_PACKAGING=ON', f"-DCYTHON={cython_code}", '-DINSTALL_CTOOLS_IN_PYTHON=ON', f"-DCMAKE_C_COMPILER={c_compiler}", f"-DCMAKE_CXX_COMPILER={cxx_compiler}", - f"-DPYTHON_SITE_PACKAGES={PYTHON_bin_package}", + f"-DPYTHON_SITE_PACKAGES={extension.bin_dir}", f"-DPYTHON_EXECUTABLE={sys.executable}"]) - self.announce("Building binaries", level=3) - - self.spawn(["cmake", "--build", self.build_temp, "--target", "all", - "--config", "Release","--","VERBOSE=1"]) - - self.spawn(["cmake", "--build", self.build_temp, "--target", "install", - "--config", "Release","--","VERBOSE=1"]) - - # Build finished, now copy the files into the copy directory - # The copy directory is the parent directory of the extension (.pyd) - - self.announce("Moving built python module", level=3) - - bin_dir = PYTHON_bin_package - #extension.bin_dir - self.distribution.bin_dir = bin_dir - target_dir = os.path.abspath(os.path.join(package_dir,'bin')) - print(target_dir) - shutil.rmtree(target_dir, ignore_errors=True) - shutil.move(f"{PYTHON_bin_package}/void_python_tools/bin", target_dir) - - shutil.copy(f"{self.build_temp}/pipeline/prepareInputs.py", f"{self.build_temp}/vide_prepare_simulation") - -class VideScripts(build_scripts): - - user_options = build_scripts.user_options + [ - ('build-temp=', 't', "temporary directory where scripts are stored"), - ] - - vide_scripts = ["vide_prepare_simulation"] - - def initialize_options(self): - super(VideScripts, self).initialize_options() - self.build_temp = None - - def finalize_options(self): - super(VideScripts, self).finalize_options() - self.set_undefined_options('build_ext', - ('build_temp', 'build_temp')) - self.scripts = [os.path.join(self.build_temp, v) for v in self.vide_scripts] - - def run(self): - self.copy_scripts() + self.spawn(["cmake", "--build", self.build_temp, "--target", "all", "--config", "Release", "--", "VERBOSE=1"]) + self.spawn(["cmake", "--build", self.build_temp, "--target", "install", "--config", "Release", "--", "VERBOSE=1"]) vide_extension = CMakeExtension(name="vide") -setup(name='vide', - version='2.0', - packages=find_packages('python_tools'), - package_dir={'': 'python_tools'}, - setup_requires=['cython','setuptools','healpy','scipy','astropy','extension-helpers','netCDF4'], - install_requires=['scipy','astropy','extension-helpers','netCDF4','healpy'], - ext_modules=[vide_extension], - description='The VIDE pipeline analysis for Cosmic Voids', - long_description=open("./README.md", 'r').read(), - long_description_content_type="text/markdown", - keywords="cosmology, interpolation, cmake, extension", - classifiers=["Intended Audience :: Developers", - "License :: OSI Approved :: " - "GNU Lesser General Public License v3 (LGPLv3)", - "Natural Language :: English", - "Programming Language :: C", - "Programming Language :: C++", - "Programming Language :: Python", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: Implementation :: CPython"], - license='CeCILL-v2', - scripts=["vide_prepare_simulation"], - cmdclass={ - 'build_ext': BuildCMakeExt, - 'install_data': InstallCMakeLibsData, - 'install_lib': InstallCMakeLibs, - 'build_scripts': VideScripts, - 'install_scripts': install_scripts, - } +setup( + name='vide', + version='2.0', + packages=find_packages('python_tools'), + package_dir={'': 'python_tools'}, + ext_modules=[vide_extension], + cmdclass={ + 'build_ext': BuildCMakeExt, + 'install_lib': InstallCMakeLibs, + 'install_scripts': InstallScripts, + }, + data_files=[ + ('share/vide/data', ['data/file1.dat', 'data/file2.dat']), + ], + install_requires=[ + 'scipy', + 'astropy', + 'extension-helpers', + 'netCDF4', + 'healpy', + ], + description='The VIDE pipeline analysis for Cosmic Voids', + long_description=open("./README.md", 'r').read(), + long_description_content_type="text/markdown", )