Create AWS Lambda Layers for Python Packages

Photo by Emile Perron on Unsplash

Check/Install Python

You can check to see if your desired Python version is installed on your Cloud9 environment.

which python3.8

If the required version doesn’t exist, install it following these steps

# install prerequisites for Python before installing it.
sudo yum install gcc openssl-devel bzip2-devel libffi-devel
# Download the Python from the Python official website.
sudo wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz
sudo tar xzf Python-3.8.6.tgz
cd Python-3.8.6
# compile Python 3.8 from the source code and install
sudo ./configure --enable-optimizations
# make altinstall is used to prevent replacing the default python binary file /usr/bin/python.
sudo make altinstall
# move downloaded source archive file from your system
sudo rm -f /opt/Python-3.8.6.tgz
# check the version
python3.8 -V

Create Lambda Layer

now install the packages you want. Here I am installing pandas. You can use requirements.txt as well.

Put the correct version of python where you see python3.x.

mkdir folder
cd folder
virtualenv v-env -p python3.x
source ./v-env/bin/activate
pip install pandas
deactivate

Then type the following code line by line to create your layer. Please note that I have two versions of Python for the compatible runtimes.

mkdir python
cd python
cp -r ../venv/lib/python3.8/site-packages/* . 
cd ..
zip -r pandas_layer.zip python
aws lambda publish-layer-version --layer-name pandas --zip-file fileb://pandas_layer.zip --compatible-runtimes python3.7 python3.8

AWS Cloud9

If you don’t have AWS CLI setup on your machine, you can use AWS Cloud9 to run the commands in the cloud.

+++++++++++++++++++++++++++++++++++

Plugging what I do daily here ;)

Don’t miss Snorkel AI flagship free virtual conference, The Future of Data-Centric AI on August 3–4.
We will explore data-centric approaches that make AI practical during this live event.

Gain an in-depth understanding of the data-centric AI approaches and learn best practices from real-world implementations.
Connect with fellow data scientists, machine learning engineers, and AI leaders from academia and industry.
This conference includes 30+ virtual sessions, a hands-on virtual workshop, and an in-person social.

Join the conversation with experts from @Apple, @Bloomberg, @CapitalOne, @Google, @Harvard, @MIT, @Meta, @Mckinsey, @Nvidia, @Pinterest, @Stanford, @StateFarm, @Wellsfargo, & more

Register here.