guilhem_lavaux/vide_public
0
0
Fork 0
mirror of https://bitbucket.org/cosmicvoids/vide_public.git synced 2025-07-04 15:21:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update build for python compat

Browse source 
- remove distutils deps
- add pyproject.toml
- add MANIFEST.in
- update install instructions
This commit is contained in:
Julien Zoubian 2024-09-04 21:33:21 +02:00
parent 58e65d7d2b
commit 54fe8df970
6 changed files with 238 additions and 250 deletions
Download patch file Download diff file Expand all files Collapse all files

6
MANIFEST.in Normal file
View file

@ -0,0 +1,6 @@
include README.md
include CMakeLists.txt
recursive-include python_tools *.py
recursive-include c_tools *
recursive-include external *
recursive-include zobov *

169
README.md
View file

@ -48,123 +48,130 @@ VIDE is licensed under the GNU Public License. See the LICENSE file for further
VIDE requires several dependencies for building and running the software. These dependencies are listed below. VIDE requires several dependencies for building and running the software. These dependencies are listed below.
### Required Packages ### Required Packages
- Python 3.8 - **Python 3.8**
- **GCC** and **G++** (for compiling C/C++ code) - **GCC** and **G++** (for compiling C/C++ code)
- [CMake](https://cmake.org/) (version 3.20 or higher) - **[CMake](https://cmake.org/)** (version 3.20 or higher)
- [satrapy](https://pypi.org/project/satrapy/) (Python package) - **[satrapy](https://pypi.org/project/satrapy/)** (Python package)
- Standard scientific Python packages: `scipy`, `pandas`, `matplotlib`, `PySide2` - **Standard scientific Python packages**: `scipy`, `pandas`, `matplotlib`, `PySide2`
### Conda Environment Setup ---
We recommend setting up a Conda environment to simplify the management of these dependencies. Follow these steps: ## Installation Methods
#### 1. Install Dependencies Using Conda (Linux/MacOS) There are three primary ways to install and build VIDE: using Conda build, manually building in a Conda environment, or using a system with `pip` and system dependencies.
Create a new Conda environment with the required dependencies, you can use `micromamba` (a faster alternative to Conda): ### 1. Install using Conda Build
Run the `conda-build` command to build the package:
```bash ```bash
micromamba env create -y -n vide_python3.8 python=3.8.12 scipy pandas matplotlib PySide2 cmake=3.20 gcc=13.2 gxx m4 -c conda-forge conda install conda-build # Install conda-build if not already installed
conda build .
``` ```
#### 2. Set Up Environment Variables After the build is complete, you can install the built package:
```bash ```bash
echo "export CC=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/gcc" > ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh conda install --use-local vide
echo "export CXX=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/g++" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh
echo "export LIBRARY_PATH=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/lib" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh
```
Activate the environment again to apply the changes:
```bash
micromamba activate vide_python3.8
```
#### 3. Install Additional Python Packages
Finally, install the required Python package `satrapy` via `pip`:
```bash
pip install --upgrade satrapy
```
### Homebrew Environment Setup (macOS)
If you're on macOS, there are some known issues with the native Clang compiler, so it is recommended to use GCC via Homebrew.
#### 1. Install GCC via Homebrew
```bash
brew install gcc
export CC=/usr/local/bin/gcc-10
export CXX=/usr/local/bin/g++-10
```
Ensure the `gcc-10` version matches the version installed by Homebrew.
#### 2. Install Required Python Packages
Set up a virtual environment and install dependencies as follows:
```bash
python3 -m venv --system-site-packages $PLACE_OF_VENV
source $PLACE_OF_VENV/bin/activate
pip install scipy pandas matplotlib PySide2
```
Install the required `satrapy` package:
```bash
pip install --upgrade satrapy
``` ```
--- ---
## Package Build and Installation ### Alternative: Manual Build in Conda Environment
Once the environment is set up, you can build and install VIDE. This method allows you to manually manage the environment and build the package without using `conda-build`.
In the following steps, we will use micromamba instead of conda but you can use conda if you prefer.
Micromamba is a faster and more lightweight alternative to conda.
### 1. Build the Package #### Step 1: Create a Conda Environment
To build the package, run: Create a new Conda environment with the required dependencies:
```bash ```bash
python setup.py build micromamba env create -y -n vide_python3.8 python=3.8.12 scipy pandas matplotlib PySide2 cmake=3.20 gcc=13.2 gxx m4 -c conda-forge
echo "export CC=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/gcc" > ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh
echo "export CXX=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/bin/g++" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh
echo "export LIBRARY_PATH=${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/lib" >> ${MAMBA_ROOT_PREFIX}/envs/vide_python3.8/etc/conda/activate.d/vide.sh
conda activate vide_python3.8
``` ```
This process may take some time and will download any missing dependencies automatically. Make sure you have enough resources available for the build process. #### Step 2: Install Additional Python Packages
### 2. Install the Package Install `satrapy` and other required packages via `pip`:
After building the package, install it by running:
```bash ```bash
pip install --upgrade satrapy
```
#### Step 3: Build the Package
Build the package using `setup.py`:
```bash
python setup.py build_ext --inplace
python setup.py install python setup.py install
``` ```
### 3. Verify the Installation ---
After installation, you can verify that VIDE is correctly installed by running the following command: ### 3. Install using System Dependencies and `pip`
This method requires system-level installation of dependencies such as `gcc`, `cmake`, and Python packages via `pip`.
#### Step 1: Install System Dependencies
You will need to install the following system dependencies:
- **GCC** (13.2 or higher)
- **G++** (13.2 or higher)
- **CMake** (3.20 or higher)
- **Python 3.8** or higher
- Required system libraries: `libgomp`, `libgcc-ng`, `libpthread`
On a Debian-based system, you can install these with:
```bash
sudo apt update
sudo apt install build-essential cmake python3 python3-dev python3-pip
```
#### Step 2: Install Python Dependencies
Use `pip` to install Python dependencies:
```bash
pip install scipy astropy healpy extension-helpers netCDF4
```
#### Step 3: Build and Install with `pip`
Run the following to install the package:
```bash
pip install .
```
If you encounter build issues due to isolation, use the `--no-build-isolation` flag:
```bash
pip install --no-build-isolation .
```
---
## Package Testing
### Check Installation
Verify that the package is installed correctly:
```bash ```bash
python -m void_pipeline python -m void_pipeline
``` ```
You should see the output:
```
Usage: ./generateCatalog.py parameter_file.py
```
---
## Package Test
VIDE provides tools for both observational data and simulations. Below are steps to test the installation.
### Testing with Observational Data ### Testing with Observational Data
To test the pipeline with observational data: To test with observational data, run:
```bash ```bash
cd python_tools/void_pipeline/datasets cd python_tools/void_pipeline/datasets

31
meta.yaml Normal file
View file

@ -0,0 +1,31 @@
package:
name: vide
version: "2.0"
source:
path: ..
requirements:
build:
- gcc_linux-64=13.2
- gxx_linux-64=13.2
- cmake=3.20
- cython
- numpy
host:
- python
- scipy
- astropy
- healpy
- extension-helpers
- netCDF4
build:
script: |
python setup.py build_ext --inplace
python setup.py install
about:
home: "https://bitbucket.org/cosmicvoids/vide_public/wiki/Home"
license: "GPL-3.0"
summary: "The VIDE pipeline analysis for Cosmic Voids"

30
pyproject.toml Normal file
View file

@ -0,0 +1,30 @@
[build-system]
requires = ["setuptools", "wheel", "cython", "cmake", "scipy", "astropy", "healpy", "extension-helpers", "netCDF4"]
build-backend = "setuptools.build_meta"
[project]
name = "vide"
version = "2.0"
description = "The VIDE pipeline analysis for Cosmic Voids"
readme = "README.md"
license = { file = "LICENSE" }
keywords = ["cosmology", "Cosmic Voids", "Voids", "Void Finder", "Void Analysis"]
requires-python = ">=3.7"
classifiers = [
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)",
"Natural Language :: English",
"Programming Language :: C",
"Programming Language :: C++",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: Implementation :: CPython"
]
dependencies = [
"scipy",
"astropy",
"extension-helpers",
"netCDF4",
"healpy"
]

32
python_tools/setup.py
View file

@ -17,20 +17,36 @@
# with this program; if not, write to the Free Software Foundation, Inc., # with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#+ #+
from distutils.core import setup from setuptools import setup, Extension
from distutils.extension import Extension from setuptools.command.build_ext import build_ext
from Cython.Distutils import build_ext
import numpy as np import numpy as np
import os import os
# Optional: Use VOID_GSL environment variable if needed later in the build process
VOID_GSL = os.environ.get('VOID_GSL') VOID_GSL = os.environ.get('VOID_GSL')
# Define extensions (you can add Cython extensions here if needed)
extensions = [
# Example of a Cython extension (uncomment and modify if needed)
# Extension('your_extension_name', sources=['your_source_file.pyx'], include_dirs=[np.get_include()])
]
setup( setup(
name='vide', name='vide',
version='1.0', version='1.0',
# cmdclass = {'build_ext': build_ext}, include_dirs=[np.get_include()], # Add NumPy include dirs
include_dirs = [np.get_include()], packages=[
packages= 'vide',
['vide','vide.backend','vide.apTools', 'vide.voidUtil', 'vide.backend',
'vide.apTools.profiles','vide.apTools.chi2',], 'vide.apTools',
'vide.voidUtil',
'vide.apTools.profiles',
'vide.apTools.chi2',
],
ext_modules=extensions, # Add extensions if needed
cmdclass={'build_ext': build_ext}, # Use setuptools build_ext for Cython
install_requires=[
'numpy', # Ensure NumPy is installed
'cython', # Ensure Cython is installed for building extensions
],
) )

188
setup.py
View file

@ -17,134 +17,78 @@
# with this program; if not, write to the Free Software Foundation, Inc., # with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#+ #+
import stat
import os import os
import sys import sys
import shutil import shutil
from sysconfig import get_config_var import stat
from distutils.command.install_data import install_data
from distutils.command.build_scripts import build_scripts
import pathlib import pathlib
from setuptools import find_packages, setup, Extension, Command import struct
from sysconfig import get_config_var
from setuptools import find_packages, setup, Extension
from setuptools.command.build_ext import build_ext from setuptools.command.build_ext import build_ext
from setuptools.command.install_lib import install_lib from setuptools.command.install_lib import install_lib
from setuptools.command.install_scripts import install_scripts from setuptools.command.install_scripts import install_scripts
import struct
BITS = struct.calcsize("P") * 8 BITS = struct.calcsize("P") * 8
PACKAGE_NAME = "vide" PACKAGE_NAME = "vide"
class CMakeExtension(Extension): class CMakeExtension(Extension):
"""
An extension to run the cmake build
This simply overrides the base extension class so that setuptools
doesn't try to build your sources for you
"""
def __init__(self, name, sources=["dummy_extension/empty.c"]): def __init__(self, name, sources=["dummy_extension/empty.c"]):
super().__init__(name=name, sources=sources) super().__init__(name=name, sources=sources)
self.SOURCE_DIR = str(pathlib.Path().absolute()) self.SOURCE_DIR = str(pathlib.Path().absolute())
class InstallCMakeLibsData(install_data):
"""
Just a wrapper to get the install data into the egg-info
Listing the installed files in the egg-info guarantees that
all of the package files will be uninstalled when the user
uninstalls your package through pip
"""
def run(self):
"""
Outfiles are the libraries that were built using cmake
"""
# There seems to be no other way to do this; I tried listing the
# libraries during the execution of the InstallCMakeLibs.run() but
# setuptools never tracked them, seems like setuptools wants to
# track the libraries through package data more than anything...
# help would be appriciated
self.outfiles = self.distribution.data_files
class InstallCMakeLibs(install_lib): class InstallCMakeLibs(install_lib):
"""
Get the libraries from the parent distribution, use those as the outfiles
Skip building anything; everything is already built, forward libraries to
the installation step
"""
def run(self): def run(self):
"""
Copy libraries from the bin directory and place them as appropriate
"""
self.announce("Moving library files", level=3) self.announce("Moving library files", level=3)
super().run()
self.distribution.run_command("install_data") class InstallScripts(install_scripts):
self.distribution.run_command("install_scripts") def run(self):
# Move the generated script from the build directory to the final installation directory
build_temp = self.get_finalized_command('build_ext').build_temp
generated_script = os.path.join(build_temp, 'vide_prepare_simulation')
if os.path.exists(generated_script):
target_script_dir = os.path.join(self.install_dir, 'bin')
os.makedirs(target_script_dir, exist_ok=True)
shutil.copy(generated_script, target_script_dir)
super().run() super().run()
class BuildCMakeExt(build_ext): class BuildCMakeExt(build_ext):
"""
Builds using cmake instead of the python setuptools implicit build
"""
def run(self): def run(self):
"""
Perform build_cmake before doing the 'normal' stuff
"""
for extension in self.extensions: for extension in self.extensions:
if extension.name == 'vide': if extension.name == 'vide':
self.package = 'vide' self.package = 'vide'
self.build_cmake(extension) self.build_cmake(extension)
super().run() super().run()
def build_cmake(self, extension: Extension): def build_cmake(self, extension: Extension):
"""
The steps required to build the extension
"""
self.announce("Preparing the build environment", level=3) self.announce("Preparing the build environment", level=3)
# Ensure absolute paths are used
package_dir = os.path.abspath(os.path.join(self.build_lib, 'vide')) package_dir = os.path.abspath(os.path.join(self.build_lib, 'vide'))
extension.build_dir = os.path.abspath(self.build_temp)
extension.build_dir = pathlib.Path(self.build_temp)
extension.bin_dir = str(pathlib.Path(os.path.join(extension.build_dir, 'private_install')).absolute()) extension.bin_dir = str(pathlib.Path(os.path.join(extension.build_dir, 'private_install')).absolute())
SOURCE_DIR = extension.SOURCE_DIR SOURCE_DIR = os.path.abspath(extension.SOURCE_DIR) # Absolute path to source directory
build_dir = extension.build_dir
extension_path = pathlib.Path(self.get_ext_fullpath(extension.name)) extension_path = pathlib.Path(self.get_ext_fullpath(extension.name))
os.makedirs(build_dir, exist_ok=True) # Make sure necessary directories exist
os.makedirs(extension.build_dir, exist_ok=True)
os.makedirs(extension_path.parent.absolute(), exist_ok=True) os.makedirs(extension_path.parent.absolute(), exist_ok=True)
cython_code = os.path.join(str(build_dir.absolute()),'mycython') # Write the cython code if necessary
cython_code = os.path.join(str(extension.build_dir), 'mycython')
with open(cython_code, mode="wt") as ff: with open(cython_code, mode="wt") as ff:
ff.write(f"#!{sys.executable}\n" ff.write(f"#!{sys.executable}\n"
"from Cython.Compiler.Main import setuptools_main\n" "from Cython.Compiler.Main import setuptools_main\n"
"setuptools_main()") "setuptools_main()")
os.chmod(cython_code, stat.S_IXUSR | stat.S_IWUSR | stat.S_IRUSR | stat.S_IRGRP) os.chmod(cython_code, stat.S_IXUSR | stat.S_IWUSR | stat.S_IRUSR | stat.S_IRGRP)
# Now that the necessary directories are created, build
self.announce("Configuring cmake project", level=3) self.announce("Configuring cmake project", level=3)
# Change your cmake arguments below as necessary # Ensure the cmake command has the correct source directory
# Below is just an example set of arguments for building Blender as a Python module
PYTHON_bin_package=f"{build_dir.absolute()}/private_install"
c_compiler = os.environ.get('CC', get_config_var("CC")) c_compiler = os.environ.get('CC', get_config_var("CC"))
cxx_compiler = os.environ.get('CXX', get_config_var("CXX")) cxx_compiler = os.environ.get('CXX', get_config_var("CXX"))
@ -156,83 +100,37 @@ class BuildCMakeExt(build_ext):
f"-DCYTHON={cython_code}", f"-DCYTHON={cython_code}",
'-DINSTALL_CTOOLS_IN_PYTHON=ON', '-DINSTALL_CTOOLS_IN_PYTHON=ON',
f"-DCMAKE_C_COMPILER={c_compiler}", f"-DCMAKE_CXX_COMPILER={cxx_compiler}", f"-DCMAKE_C_COMPILER={c_compiler}", f"-DCMAKE_CXX_COMPILER={cxx_compiler}",
f"-DPYTHON_SITE_PACKAGES={PYTHON_bin_package}", f"-DPYTHON_SITE_PACKAGES={extension.bin_dir}",
f"-DPYTHON_EXECUTABLE={sys.executable}"]) f"-DPYTHON_EXECUTABLE={sys.executable}"])
self.announce("Building binaries", level=3) self.announce("Building binaries", level=3)
self.spawn(["cmake", "--build", self.build_temp, "--target", "all", "--config", "Release", "--", "VERBOSE=1"])
self.spawn(["cmake", "--build", self.build_temp, "--target", "all", self.spawn(["cmake", "--build", self.build_temp, "--target", "install", "--config", "Release", "--", "VERBOSE=1"])
"--config", "Release","--","VERBOSE=1"])
self.spawn(["cmake", "--build", self.build_temp, "--target", "install",
"--config", "Release","--","VERBOSE=1"])
# Build finished, now copy the files into the copy directory
# The copy directory is the parent directory of the extension (.pyd)
self.announce("Moving built python module", level=3)
bin_dir = PYTHON_bin_package
#extension.bin_dir
self.distribution.bin_dir = bin_dir
target_dir = os.path.abspath(os.path.join(package_dir,'bin'))
print(target_dir)
shutil.rmtree(target_dir, ignore_errors=True)
shutil.move(f"{PYTHON_bin_package}/void_python_tools/bin", target_dir)
shutil.copy(f"{self.build_temp}/pipeline/prepareInputs.py", f"{self.build_temp}/vide_prepare_simulation")
class VideScripts(build_scripts):
user_options = build_scripts.user_options + [
('build-temp=', 't', "temporary directory where scripts are stored"),
]
vide_scripts = ["vide_prepare_simulation"]
def initialize_options(self):
super(VideScripts, self).initialize_options()
self.build_temp = None
def finalize_options(self):
super(VideScripts, self).finalize_options()
self.set_undefined_options('build_ext',
('build_temp', 'build_temp'))
self.scripts = [os.path.join(self.build_temp, v) for v in self.vide_scripts]
def run(self):
self.copy_scripts()
vide_extension = CMakeExtension(name="vide") vide_extension = CMakeExtension(name="vide")
setup(name='vide', setup(
name='vide',
version='2.0', version='2.0',
packages=find_packages('python_tools'), packages=find_packages('python_tools'),
package_dir={'': 'python_tools'}, package_dir={'': 'python_tools'},
setup_requires=['cython','setuptools','healpy','scipy','astropy','extension-helpers','netCDF4'],
install_requires=['scipy','astropy','extension-helpers','netCDF4','healpy'],
ext_modules=[vide_extension], ext_modules=[vide_extension],
cmdclass={
'build_ext': BuildCMakeExt,
'install_lib': InstallCMakeLibs,
'install_scripts': InstallScripts,
},
data_files=[
('share/vide/data', ['data/file1.dat', 'data/file2.dat']),
],
install_requires=[
'scipy',
'astropy',
'extension-helpers',
'netCDF4',
'healpy',
],
description='The VIDE pipeline analysis for Cosmic Voids', description='The VIDE pipeline analysis for Cosmic Voids',
long_description=open("./README.md", 'r').read(), long_description=open("./README.md", 'r').read(),
long_description_content_type="text/markdown", long_description_content_type="text/markdown",
keywords="cosmology, interpolation, cmake, extension",
classifiers=["Intended Audience :: Developers",
"License :: OSI Approved :: "
"GNU Lesser General Public License v3 (LGPLv3)",
"Natural Language :: English",
"Programming Language :: C",
"Programming Language :: C++",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: Implementation :: CPython"],
license='CeCILL-v2',
scripts=["vide_prepare_simulation"],
cmdclass={
'build_ext': BuildCMakeExt,
'install_data': InstallCMakeLibsData,
'install_lib': InstallCMakeLibs,
'build_scripts': VideScripts,
'install_scripts': install_scripts,
}
) )