Adding a Pandas Layer to AWS Lambda

A step by step guide to making Pandas available as a Lambda Layer in AWS …

This guide assumes a working knowledge of AWS Lambda functionality.

Why: When attempting to run Pandas in your AWS Lambda you receive the error code below.

“numpy: cannot import name ‘WinDLL’ from ‘ctypes’ (/var/lang/lib/python3.7/ctypes/__init__.py)”.

This is due to incompatibility between Windows compiled version of Pandas and AWS Linux.

The solution is to use an AWS Linux image with Docker to compile the Pandas libraries.

Working with Docker

Download docker at “https://www.docker.com/products/docker-desktop” and install using the default specs.

When installation is complete, open Docker. This will start Docker in the background. lick Show Hidden Icons on Windows Taskbar to verify that Docker is running.

Click Show Hidden Icons to verify Docker is running

Now, start a command line editor (Powershell or Windows CMD ) and run: docker pull amazonlinux to download an image of Amazon Linux.

Now access the image by typing: docker run -it amazonlinux. This creates a new container and the command line switches to a bash command.

Bash command line

At the bash# command, install Python and Zip utilities using the following commands:

yum install python37
yum install zip unzip

Install Pip and add it to the system path

curl -O https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py --user
export PATH=root/.local/bin:$PATH

Create a working directory in the Docker container: mkdir python

Install Pandas in the new directory: pip3 install Pandas -t python

Zip the contents of Python folder:

zip -r /python/python.zip /python

Exit bash by typing exit

Docker assigns a random name when a container is created (silly_bartik in this example). We can get the name of the container we just exited with Get a list of Docker containers with the command

docker container ls --all

I’m going to rename with something more descriptive. In this example, I’m renaming silly_bartik to aws_linux_container.

docker rename silly_bartik aws_linux_container

docker container ls -a shows the new container name

Now we’re going to copy the zipped file from the container to a directory accessible by Windows. In this example, I’m using a Windows folder named tmp.

PS C:\Windows> docker cp aws_linux_container:/python/python.zip c:\tmp

python.zip is now available in our windows tmp directory

PS C:\windows> dir c:\tmp
 Directory: C:\tmp
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        11/4/2020   9:27 AM       32396540 python.zip

Creating and accessing the Lambda Layer in AWS Console

Navigate to AWS Console, select Lambda from services, select Layers / Create Layer

Name the layer, upload the python.zip file from your Windows directory and choose the runtime you will use with your Lambda function

Now, add the layer to new or existing Lambda function

In Lambda Designer, select Layers then Add a Layer

In Add Layer window, select Custom layers and choose your layer name and version from the drop-downs. Then click Add

Now import pandas to your function and use it as normal. Run a test with the code below to confirm that pandas is available.

import json
import pandas as pd
def lambda_handler(event, context):
    
    print("Pandas Version:", pd.__version__)