A step by step guide to making Pandas available as a Lambda Layer in AWS …
This guide assumes a working knowledge of AWS Lambda functionality.
Why: When attempting to run Pandas in your AWS Lambda you receive the error code below.
“numpy: cannot import name ‘WinDLL’ from ‘ctypes’ (/var/lang/lib/python3.7/ctypes/__init__.py)”.
This is due to incompatibility between Windows compiled version of Pandas and AWS Linux.
The solution is to use an AWS Linux image with Docker to compile the Pandas libraries.
Working with Docker
Download docker at “https://www.docker.com/products/docker-desktop” and install using the default specs.
When installation is complete, open Docker. This will start Docker in the background. lick Show Hidden Icons on Windows Taskbar to verify that Docker is running.
Now, start a command line editor (Powershell or Windows CMD ) and run: docker pull amazonlinux to download an image of Amazon Linux.
Now access the image by typing: docker run -it amazonlinux. This creates a new container and the command line switches to a bash command.
At the bash# command, install Python and Zip utilities using the following commands:
yum install python37yum install zip unzip
Install Pip and add it to the system path
curl -O https://bootstrap.pypa.io/get-pip.pypython3 get-pip.py --userexport PATH=root/.local/bin:$PATH
Create a working directory in the Docker container: mkdir python
Install Pandas in the new directory: pip3 install Pandas -t python
Zip the contents of Python folder:
zip -r /python/python.zip /python
Exit bash by typing exit
Docker assigns a random name when a container is created (silly_bartik in this example). We can get the name of the container we just exited with Get a list of Docker containers with the command
docker container ls --all
I’m going to rename with something more descriptive. In this example, I’m renaming silly_bartik to aws_linux_container.
docker rename silly_bartik aws_linux_container
Now we’re going to copy the zipped file from the container to a directory accessible by Windows. In this example, I’m using a Windows folder named tmp.
PS C:\Windows> docker cp aws_linux_container:/python/python.zip c:\tmp
python.zip is now available in our windows tmp directory
PS C:\windows> dir c:\tmp
Directory: C:\tmpMode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 11/4/2020 9:27 AM 32396540 python.zip
Creating and accessing the Lambda Layer in AWS Console
Navigate to AWS Console, select Lambda from services, select Layers / Create Layer
Name the layer, upload the python.zip file from your Windows directory and choose the runtime you will use with your Lambda function
Now, add the layer to new or existing Lambda function
In Lambda Designer, select Layers then Add a Layer
In Add Layer window, select Custom layers and choose your layer name and version from the drop-downs. Then click Add
Now import pandas to your function and use it as normal. Run a test with the code below to confirm that pandas is available.
import pandas as pddef lambda_handler(event, context):
print("Pandas Version:", pd.__version__)