Adding a Pandas Layer to AWS Lambda

A step by step guide to making Pandas available as a Lambda Layer in AWS …

This guide assumes a working knowledge of AWS Lambda functionality.

Why: When attempting to run Pandas in your AWS Lambda you receive the error code below.

“numpy: cannot import name ‘WinDLL’ from ‘ctypes’ (/var/lang/lib/python3.7/ctypes/__init__.py)”.

This is due to incompatibility between Windows compiled version of Pandas and AWS Linux.

The solution is to use an AWS Linux image with Docker to compile the Pandas libraries.

Working with Docker

Download docker at “https://www.docker.com/products/docker-desktop” and install using the default specs.

When installation is complete, open Docker. This will start Docker in the background. lick Show Hidden Icons on Windows Taskbar to verify that Docker is running.

Now, start a command line editor (Powershell or Windows CMD ) and run: docker pull amazonlinux to download an image of Amazon Linux.

Now access the image by typing: docker run -it amazonlinux. This creates a new container and the command line switches to a bash command.

At the bash# command, install Python and Zip utilities using the following commands:

Install Pip and add it to the system path

Create a working directory in the Docker container: mkdir python

Install Pandas in the new directory: pip3 install Pandas -t python

Zip the contents of Python folder:

Exit bash by typing exit

Docker assigns a random name when a container is created (silly_bartik in this example). We can get the name of the container we just exited with Get a list of Docker containers with the command

I’m going to rename with something more descriptive. In this example, I’m renaming silly_bartik to aws_linux_container.

Now we’re going to copy the zipped file from the container to a directory accessible by Windows. In this example, I’m using a Windows folder named tmp.

python.zip is now available in our windows tmp directory

Creating and accessing the Lambda Layer in AWS Console

Navigate to AWS Console, select Lambda from services, select Layers / Create Layer

Name the layer, upload the python.zip file from your Windows directory and choose the runtime you will use with your Lambda function

Now, add the layer to new or existing Lambda function

In Lambda Designer, select Layers then Add a Layer

In Add Layer window, select Custom layers and choose your layer name and version from the drop-downs. Then click Add

Now import pandas to your function and use it as normal. Run a test with the code below to confirm that pandas is available.

Data & BI Professional