Skip to Content
Author's profile photo Andrew Lunde
Andrew Lunde
June 16, 2017 7 minute read

XSA Python Buildpack Generator

This generator script and accompanying example now have a permanent home.

See this blog post for a discussion and details.

https://blogs.sap.com/2017/08/03/xsa-python-buildpack-and-mta-example/

While the script posted here may continue to function, it will not be maintained.

The following is to provide an example of an XS Advanced Buildpack in order to support an upcoming TechEd 2017 lecture :

HDB100 – SAP HANA Extended Application Services and BYOL (Bring Your Own Language)

Normally, this would be provided via a git repo(and may yet be), but for now I’m providing it as a bash script.  Open a new file called create_my_python_buildpack, edit it, paste the following script into it, chmod 755 it, and then run it with the name of a directory the will be created containing your python buildpack.

 ./create_my_python_buildpack my_python_buildpack

Follow the steps at the end of the script and in the README.md file in order to install your python buildpack.

If you find this useful in your development efforts and are a strategic SAP customer, please reach out to me directly at andrew.lunde@sap.com .  Unfortunately, I can NOT support this script at this time and you must use it at your own risk.  Thanks for your interest.

-Andrew

#!/bin/bash
#
# Usage: ./create_my_python_buildpack <buildpack-dir>
#
if [[ $1 ]]; then
	echo "Creating a new python buildpack in ${buildpack_dir}"

else
	echo "Usage: ./create_my_python_buildpack <buildpack-dir>"
	exit 1
fi

buildpack_dir="$1"
bin_dir="${buildpack_dir}/bin"
resources_dir="${buildpack_dir}/resources"
modules_dir="${resources_dir}/modules"

pipzip="https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9"
pipurl="https://pypi.python.org/pypi/pip"
setuptoolszip="https://pypi.python.org/packages/a9/23/720c7558ba6ad3e0f5ad01e0d6ea2288b486da32f053c73e259f7c392042/setuptools-36.0.1.zip#md5=430eb106788183eefe9f444a300007f0"
setuptoolsurl="https://pypi.python.org/pypi/setuptools"

mkdir -p ${buildpack_dir}

rm -rf ${buildpack_dir}/*

mkdir -p ${buildpack_dir}/bin

mkdir -p ${buildpack_dir}/resources

mkdir -p ${buildpack_dir}/resources/modules

echo "Creating bin/detect"
cat > ${buildpack_dir}/bin/detect <<- "EOFDETECT"
#!/usr/bin/env bash

BUILD_DIR=$1

if [ ! -f $BUILD_DIR/runtime.txt ]; then
	exit 1
fi

if grep -q python- "$BUILD_DIR/runtime.txt"; then
	echo detected `cat $BUILD_DIR/runtime.txt`
	exit 0
fi

exit 1
EOFDETECT

chmod 755 ${buildpack_dir}/bin/detect

echo "Creating bin/compile"
cat > ${buildpack_dir}/bin/compile <<- "EOFCOMPILE"
#!/usr/bin/env bash

FCOMPILE=`readlink -f "$0"`
BPDIR=`dirname $FCOMPILE`
BPDIR=`readlink -f "$BPDIR/.."`
BUILD_DIR=$1
CACHE=$2

#HTTP_PROXY=proxy:8080
#export HTTPS_PROXY=http://proxy.wdf.sap.corp:8080
#export HTTP_PROXY=http://proxy.wdf.sap.corp:8080

if [ ! -f $BUILD_DIR/runtime.txt ];then
	echo
	echo BUILPACK: Abort. Cannot find runtime.txt in application directory.
	echo BUILPACK: Please provide runtime.txt with content Python-x.x.x , example Python-3.4.4
	echo
	exit 1
fi

runtime=`cat "$BUILD_DIR"/runtime.txt`
minus=`expr index "$runtime" -`

if [ "$minus" == "0" ];then
	echo "cannot understand runtime.txt"
	echo $runtime
	exit 1
fi

lang=${runtime:0:minus}
version=${runtime:minus}

echo
echo BUILDPACK: Detected language $lang
echo BUILDPACK: Detected version $version
echo

versionxx=${version%.*}
versionx=${versionxx%.*}

mkdir -p work
mkdir -p $CACHE/compiled/Python-$version

runtimedir=`readlink -f "$BUILD_DIR/.buildpack"`
mkdir -p $runtimedir
pyexe=$runtimedir/bin/python
PATH_PY=$runtimedir/lib/python$versionxx
PATH_PY=$PATH_PY:$runtimedir/lib/python$versionxx/lib-dynload
PATH_PY=$PATH_PY:$runtimedir/lib64/python$versionxx/lib-dynload
PATH_PY=$PATH_PY:$runtimedir/lib/python$versionxx/plat-linux
PATH_PY=$PATH_PY:$runtimedir/lib64/python$versionxx/plat-linux
export PYTHONPATH=$PATH_PY
export PYTHONHOME=$runtimedir


pytgz=$CACHE/compiled/Python-${version}.tar.gz
if [ ! -f $pytgz ];then
	pytgz=/tmp/Python-${version}.tar.gz
fi

if [ ! -f $pytgz ];then
	echo
	echo "BUILDPACK: Cached python build $pytgz not found"
	echo

	if [ ! -f $BPDIR/resources/python/Python-$version.tgz ];then
		echo
		echo "BUILDPACK: Downloading python source https://www.python.org/ftp/python/$version/Python-$version.tgz"
		echo
		wget -O work/Python-$version.tgz https://www.python.org/ftp/python/$version/Python-$version.tgz
		wgetexit=$?
		if [ $wgetexit -ne 0 ];then
			echo
			echo "BUILDPACK: Abort -- Python source download failed"
			echo "BUILDPACK: You can put python tar.gz in the BUILDPACK/resources/python and re-create the buildpack to avoid download from internet"
			echo
			exit $wgetexit
		fi
	else
		cp resources/python/Python-$version.tgz work
	fi

	gzip -d -f work/Python-$version.tgz
	tar -xf work/Python-$version.tar -C work
	if [ ! -f /usr/include/zlib.h ];then
		echo
		echo "BUILDPACK: Library zlib missing, not found /usr/include/zlib.h"
		echo
		d_zlib=work/Python-$version/Modules/zlib
		if [ -d $d_zlib ];then
			pushd $d_zlib
				echo
				echo "BUILDPACK: Build python provided zlib"
				echo

				./configure --prefix=$runtimedir

				make -j 8
				make install
				zmakeexit=$?
				if [ $zmakeexit -ne 0 ];then
					echo
					echo "BUILDPACK: Warning - failed to make install python provided zlib work/Python-$version/Modules/zlib"
					echo "BUILDPACK: Ignore last failure"
					echo
				fi
			popd
		else
			echo
			echo "BUILDPACK: Warning - Not found python provided zlib $d_zlib"
			echo
		fi
	fi

	pushd work/Python-$version

		echo
		echo "BUILDPACK: Installing python runtime to $runtimedir"
		echo
		./configure --prefix=$runtimedir --exec-prefix=$runtimedir   
		make -j 8
		make altinstall
		makeexit=$?
	popd

	if [ $makeexit -ne 0 ];then
		echo
		echo BUILDPACK: Abort -> Make failed in buildpack compile step
		echo
		exit $makeexit 
	fi 

	if [ ! -f $pyexe ];then
		echo
		echo BUILDPACK: cp $runtimedir/bin/python$versionxx $pyexe
		echo
		cp $runtimedir/bin/python$versionxx $pyexe
	fi

	echo
	echo BUILDPACK: PYTHONPATH=$PYTHONPATH
	echo

	for f in $BPDIR/resources/modules/*.tar.gz
	do
		cp $f work
		fname_tar_gz=${f##*/}
		fname_tar=${fname_tar_gz%.*}
		
		gzip -d -f work/$fname_tar_gz
		tar -xf work/$fname_tar -C work                   
	done

	pushd work/setuptools*
		$pyexe setup.py build install
		setupexit=$?
		if [ $setupexit -ne 0 ];then
			echo
			echo "BUILDPACK: Abort --> Failed to install python module setuptools"
			echo
			exit $setupexit
		fi   
	popd

	pushd work/pip*
		$pyexe setup.py build install
		setupexit=$?
		if [ $setupexit -ne 0 ];then
			echo
			echo "BUILDPACK: WARNING --> Ignored:Failed to install python module pip"
			echo
		fi
	popd

	tar -cf $CACHE/compiled/Python-$version.tar -C $runtimedir .
	gzip $CACHE/compiled/Python-$version.tar

	echo
	echo BUILDPACK: Cached python build $CACHE/compiled/Python-$version.tar.gz
	echo
	cp $CACHE/compiled/Python-$version.tar.gz /tmp/Python-$version.tar.gz
else
	echo
	echo "BUILDPACK: Cached python build found $pytgz"
	echo
	cp $pytgz work
	gzip -d -f work/Python-$version.tar.gz
	tar -xf work/Python-$version.tar -C $runtimedir    
fi

echo
echo "BUILDPACK: Python executable $pyexe"
echo `"$pyexe" --version`
echo

moduleinstall=0
if [ -f $BUILD_DIR/requirements.txt ];then
	echo
	echo BUILDPACK: Try to execute python -m pip install -r r$BUILD_DIR/requirements.txt
	echo
	$pyexe -m pip install -r $BUILD_DIR/requirements.txt
	moduleinstall=$?
fi

if [ $moduleinstall -ne 0 ] && [ -d $BUILD_DIR/vendor ];then
	echo
	echo "BUILDPACK: Install app vendor packages using pip"
	echo $pyexe -m pip install $BUILD_DIR/vendor/*
	echo
	$pyexe -m pip install $BUILD_DIR/vendor/*
	moduleinstall=$?

	if [ $moduleinstall -ne 0 ];then
		echo
		echo "BUILDPACK: Install app vendor tar.gz packages without pip"
		echo

		for f in $BUILD_DIR/vendor/*.tar.gz
		do
			if [ -f $f ];then
				echo
				echo "BUILDPACK: Installing vendor package $f"
				echo
				cp $f work
				fname_tar_gz=${f##*/}
				fname_tar=${fname_tar_gz%.*}
				fname=${fname_tar%.*}

				gzip -d -f work/$fname_tar_gz

				tar -xf work/$fname_tar -C work
				pushd work/$fname
					$pyexe setup.py build install
				popd
			fi
		done
	fi
fi
EOFCOMPILE

chmod 755 ${buildpack_dir}/bin/compile

echo "Creating bin/release"
cat > ${buildpack_dir}/bin/release <<- "EOFRELEASE"
#!/usr/bin/env bash

set -e

BUILD_DIR=$1

FRELEASE=`readlink -f "$0"`
BPDIR=`dirname $FRELEASE`
BPDIR=`readlink -f "$BPDIR/.."`

mkdir -p $BUILD_DIR/.profile.d
cp $BPDIR/resources/env.sh $BUILD_DIR/.profile.d

echo "---"
##echo "config_vars:"
##echo "  PYTHONHOME: $HOME/.buildpack"
echo "default_process_types:"
echo "  web: .buildpack/bin/python server.py"
EOFRELEASE

chmod 755 ${buildpack_dir}/bin/release

echo "Creating resources/env.sh"
cat > ${buildpack_dir}/resources/env.sh <<- "EOFENVSH"
#!/usr/bin/env bash

FENV=`readlink -f "$0"`
DDROPLET=`dirname $FENV`


echo env.sh
echo user `whoami`
echo dir `pwd`
export PYTHONHOME=$DDROPLET/app/.buildpack

PYLIB=$(echo $DDROPLET/app/.buildpack/lib/python*/)
PYLIB64=$(echo $DDROPLET/app/.buildpack/lib64/python*/)

export PYTHONPATH=$PYLIB/:$PYLIB/lib-dynload:$PYLIB64/:$PYLIB64/lib-dynload

echo PYTHONHOME=$PYTHONHOME
echo PYTHONPATH=$PYTHONPATH
EOFENVSH

echo "Creating VERSION file"
cat > ${buildpack_dir}/VERSION <<- "EOFVERSION"
0.0.1
EOFVERSION

echo "Creating README file"
cat > ${buildpack_dir}/README.md <<- "EOFREADMEMD"
# sap_python_buildpack
Very thin and simple buildpack, implemented completely in BASH. Can work both in offline and online mode.

# How it works
* Detect : checks if the app folder contains file runtime.txt containing runtime specification python-<python version>  
* Tries to find python sources \<\<buildpack\>\>/resources/python/Python-$version.tgz
* If not found, downloads https://www.python.org/ftp/python/$version/Python-$version.tgz
* Compiles python sources
* Install python modules \<\<buildpack\>\>/resources/modules/*.tar.gz
* Caches python build to cache folder
* Tries to install dependencies described in \<\<app folder\>\>/requirements.txt using pip
* If it fails, tries to install modules from \<\<app folder\>\>/vendor 

# Offline mode 
This buildpack can work in offline mode i.e. with no internet connection.
All supported python runtimes must be provided as .tgz files in buildpack folder/resources/python folder. So you basically git clone this buildpack, then you download the supported python versions in the resources/python folder and then create the buildpack in CF or in XS

# Online mode
In this case no changes are required to this buildpack, the python version will be downloaded in the compile phase.

# Application prerequisites
Expected files in app folder:
* server.py 
* runtime.txt with sample content "python-3.4.4" or "python-3.5.5"

`cat 'python-3.4.4' >runtime.txt`
* Offline mode: vendor folder containing all dependent modules

Sample commands to download modules:

`python -m pip download -d \<\<app folder\>\>/vendor pyhdb`

`python -m pip install -d \<\<app folder\>\>/vendor -r \<\<app folder\>\>/requirements.txt`
* Online mode: requirements.txt in the app folder


# Limitations
* Hardcoded sap corporate proxy (Removed by Andrew Lunde I830671 for public Internet use.)
* Tested on OS: Ubuntu, Suse linux
* Tested with python versions: 3.4.4  3.5.0
EOFREADMEMD

cd ${buildpack_dir}/resources/modules

echo "Getting...PIP from $pipurl"

wget ${pipzip}

echo "Done..."

echo "Getting...setuptools from $setuptoolsurl"

wget ${setuptoolszip}

echo "Done..."

cd ../..

echo "Finished creating a new python buildpack in ${buildpack_dir}\n"
echo ""
echo "Install the buildpack with: (use an unused position number if 99 is occupied)"
echo ""
echo "xs create-buildpack my_python_buildpack ${buildpack_dir} 99"
echo ""

Assigned Tags

      Be the first to leave a comment
      You must be Logged on to comment or reply to a post.