Download files from Cloudflare R2 Storage using Python

Overview

This recipe shows how to download a json file from Cloudflare's R2 storage using Python and the S3 compatible API. Cloudflare's R2 storage is a distributed storage that leverages Cloudflare edge network across the globe.

What is needed for this recipe

1. Create a Cloudflare account, create a bucket and obtain bucket credentials (see below).
2. Install the Dataplane pip package (https://pypi.org/project/dataplane/) by adding dataplane to requirements.txt file. For further instructions, check how to Update Python packages under Docs > Getting Started > Create a data pipeline.
3. Upload a file to Cloudflare's R2 storage, see Upload files to Cloudflare R2 Storage using Python.

Setup Cloudflare R2 bucket

In Cloudflare console, create a Cloudflare R2 bucket and keep note of these details to connect using Python:

1. Account ID
2. Bucket name
3. Client access key
4. Client secret
5. Connection url

Create a bucket

Create an API token

Once your bucket has been created, issue an API token for authentication of the S3 compatible API.

Configure API token permissions

The token will need edit access and we have selected an expiry of 1 year.

tip

It is a good idea to put a reminder in your calendar to rotate API credentials at least 1 week prior to expiry.

Obtain client access key and secret

danger

It is important to keep these credentials safe using a key store. It is good security practice not to hard code the secret in your code. Do not share secrets as this will compromise the safety and security of your data. You can use Dataplane's secrets to store these credentials. Learn more here: https://learn.dataplane.app/managing-secrets

Obtain connection url

Code to download a json file from Cloudflare R2 storage

# Import python packages
from dataplane import s3_download
import os
import boto3
from botocore.client import Config
from dotenv import load_dotenv
import json


load_dotenv()


# 1. Account ID
AccountID = os.environ["secret_dp_S3_ACCOUNT_ID"]

# 2. Bucket name
Bucket = os.environ["secret_dp_BUCKET_NAME"]

# 3. Client access key
ClientAccessKey = os.environ["secret_dp_S3_ACCESS_KEY"]

# 4. Client secret
ClientSecret = os.environ["secret_dp_S3_SECRET"]

# 5. Connection url
ConnectionUrl = f"https://{AccountID}.r2.cloudflarestorage.com"

# Create a client to connect to Cloudflare's R2 Storage
S3Connect = boto3.client(
    's3',
    endpoint_url=ConnectionUrl,
    aws_access_key_id=ClientAccessKey,
    aws_secret_access_key=ClientSecret,
    config=Config(signature_version='s3v4'),
    region_name='us-east-1'

)

# Download file from R2 using S3 compatible API
rs = s3_download(Bucket=Bucket, 
S3Client=S3Connect,
S3FilePath=f"test/my file.json",
DownloadMethod="Object"
)
print(rs["content"])