Deploy PuLP based XSA application on IBM PowerPC HANA
In this blog post, you will learn how to deploy PuLP based XSA applications in IBM PowerPC machines.
Emerging cloud technologies and on-premise runtime environments like HANA XSA dramatically help businesses not only build and run their analytical solutions in a secure and high-performance environment but also make solutions easily accessible to end-users like business analysts and sales managers.
HANA and cloud platforms provide runtime environments for a different choice of programming languages. Python is known in ML and AI world as one of the most used programming languages. The computational performance of Python is not comparable to many low-level programming languages, therefore, many optimization algorithms are written in C++ and delivered as compiled binary files, and optimization libraries like PuLP act as a wrapper around the core solver, and execute these binaries at the OS level to deliver results.
2. Problem statement
Even though Runtime environments are supposed to be independent of the OS, when it comes to optimization, binary files should execute at the OS level. Hence, the binary files must match the hardware architecture and the OS.
In this blog, how to resolve the OS mismatch error of the CBC solver binary file inside the python PuLP library and execute it on IBM PowerPC architecture (IBM ppc64le) will be demonstrated. This can be seen as an example of how to resolve this kind of issue in any Linux system.
If the binary file cannot be executed “OSError: [Errno 8] Exec format error” will be raised as depicted in Figure 1.
Below you can find steps on how to make the CBC work in the ppc64le environment:
3.1. Download and Build CBC on the platform:
Get the CBC solver from the link below and build it in the target Linux system with the “make” command.
Navigate to the newly build CBC location, add “test.lp” to the directory and run the below command for testing the CBC solver.
>> ./cbc test.lp solve solu=sol.txt
If the result is similar to the below results, the CBC has been successfully built and everything is set for the next step.
Welcome to the CBC MILP Solver Version: 2.10.5 Build Date: Sep 6 2021 command line - ./cbc test.lp solve solu=sol.txt (default strategy 1) CoinLpIO::readLp(): Maximization problem reformulated as minimization Coin0009I Switching back to maximization to get correct duals etc Continuous objective value is 5.39478e+07 - 0.00 seconds Cgl0002I 2 variables fixed Cgl0003I 1 fixed, 0 tightened bounds, 702 strengthened rows, 4 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 456 strengthened rows, 1 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 260 strengthened rows, 0 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 74 strengthened rows, 0 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 58 strengthened rows, 0 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 9 strengthened rows, 0 substitutions Cgl0003I 0 fixed, 0 tightened bounds, 3 strengthened rows, 0 substitutions Cgl0004I processed model has 841 rows, 290 columns (290 integer (290 of which binary)) and 4352 elements Cbc0038I Initial state - 0 integers unsatisfied sum - 1.44329e-15 Cbc0038I Solution found of -5.39478e+07 Cbc0038I Before mini branch and bound, 290 integers at bound fixed and 0 continuous Cbc0038I Mini branch and bound did not improve solution (0.09 seconds) Cbc0038I After 0.09 seconds - Feasibility pump exiting with objective of -5.39478e+07 - took 0.00 seconds Cbc0012I Integer solution of -53947771 found by feasibility pump after 0 iterations and 0 nodes (0.09 seconds) Cbc0001I Search completed - best objective -53947771.20000002, took 0 iterations and 0 nodes (0.09 seconds) Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost Cuts at root node changed objective from -5.39478e+07 to -5.39478e+07 Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds) Result - Optimal solution found Objective value: 53947771.20000002 Enumerated nodes: 0 Total iterations: 0 Time (CPU seconds): 0.10 Time (Wallclock seconds): 0.10 Total time (CPU seconds): 0.10 (Wallclock seconds): 0.11
3.2. Repackageing PuLP library with the new CBC solver
This section can be run from any machine, Windows or Linux as long as you have the python installed in that environment, the GitHub repository can be cloned, and the newly built CBC file can be moved to the machine.
Download pulp from the Github repository.
Replace the recently built CBC binary file (from section 3.1) in the pulp directory below.
Please take note of the target hardware architecture. (32 or 64 bit)
After manual replacement, it is time to repackage the PuLP project with the new CBC binary file, the PuLP version that is being downloaded is mentioned in the “constraints.py” of the PuLP package. Be aware that you need to create a custom version so that after repackaging, pip does not confuse it with the version available in the python library repository (PyPI). Personally, I add one zero as a prefix to the original version number but you can make your own choice, if the PuLP version is 2.5.0 you can change the version number to 0.2.5.0.
Before proceeding with the packaging, make sure the weel package is already installed on python. If it is not available you need to perform the next step.
python3.* -m pip install wheel
Now, navigate back to the root of the PuLP master folder and run the below command:
python3.* -m pip install check-wheel-contents
If all requirements are satisfied as figure 2, everything is set for the next level, and you can build the wheel out of pulp with the below command:
python3.* setup.py bdist_wheel
By running this command couple of folders will be created, in the “dist” folder the wheel can be found.
3.3. Solver_test application with custom python libraries
The Solver_test XSA package is available in Github. Before pushing the application to XSA, all python libraries need to download and copy into the vendor folder, so during the push, XSA from the “vendor” folder installs python libraries rather than downloading them from the python public repository.
Using this method is a good choice if the XSA system is disconnected from the Internet too. To get all packages in the “vendor” folder, the below commands should be run in the target system (Linux, and in this case the IBM PowerPC).
After cloning the repository from GitHub, run the below command in the target system:
python3.* -m pip download -d ~/project_folder/vendor -r ~/project_folder/requirements.txt
The above command will create a new folder called “vendor” which includes all python packages mentioned in “requirements.txt”. Place this folder at the root of the project, now you can deploy the package in XSA, it will install python libraries from the “vendor” folder. And the OS will pick the right CBC solver binary and execute it.
IBM PowerPC architecture is compatible with the SAP HANA platform, but when it comes to the XSA python application installation, some libraries do not work in this environment, in this blog we learned how to re-build and re-package components of CBC solver to make it deployable on XSA on IBM PowerPC architecture.
GitHub repository for the sample code:
The COIN-OR Foundation, Inc.
GitHub repository for python PuLP library: