Skip to main content

Upload files to Cloudflare R2 Storage using Python

Overview​

This recipe shows how to Upload a json file to Cloudflare's R2 storage using Python and the S3 compatible API. Cloudflare's R2 storage is a distributed storage that leverages Cloudflare edge network across the globe. On the Cloudflare's website, the service is explained as "High-performance storage for files and objects with zero egress charges."


What is needed for this recipe​

  1. Create a Cloudflare account, create a bucket and obtain bucket credentials (see below).
  2. Install the Dataplane pip package (https://pypi.org/project/dataplane/) by adding dataplane to requirements.txt file. For further instructions, check how to Update Python packages under Docs > Getting Started > Create a data pipeline.
  3. Upload a file to Cloudflare's R2 storage.

Setup Cloudflare R2 bucket​

In Cloudflare console, create a Cloudflare R2 bucket and keep note of these details to connect using Python:

  1. Account ID
  2. Bucket name
  3. Client access key
  4. Client secret
  5. Connection url

Create a bucket​

Cloudflare create R2 bucket

Create an API token​

Once your bucket has been created, issue an API token for authentication of the S3 compatible API.

Cloudflare create R2 API token

Configure API token permissions​

The token will need edit access and we have selected an expiry of 1 year.

tip

It is a good idea to put a reminder in your calendar to rotate API credentials at least 1 week prior to expiry.

Cloudflare R2 API token permissions

Obtain client access key and secret​

Cloudflare R2 API access key and secret

danger

It is important to keep these credentials safe using a key store. It is good security practice not to hard code the secret in your code. Do not share secrets as this will compromise the safety and security of your data. You can use Dataplane's secrets to store these credentials. Learn more here: https://learn.dataplane.app/managing-secrets

Obtain connection url​

Cloudflare R2 connection url

Code to upload a json file to Cloudflare R2 storage​

# Import python packages
from dataplane import s3_upload
import os
import boto3
from botocore.client import Config
from dotenv import load_dotenv
import json


load_dotenv()


# 1. Account ID
AccountID = os.environ["secret_dp_S3_ACCOUNT_ID"]

# 2. Bucket name
Bucket = os.environ["secret_dp_BUCKET_NAME"]

# 3. Client access key
ClientAccessKey = os.environ["secret_dp_S3_ACCESS_KEY"]

# 4. Client secret
ClientSecret = os.environ["secret_dp_S3_SECRET"]

# 5. Connection url
ConnectionUrl = f"https://{AccountID}.r2.cloudflarestorage.com"

# Create a client to connect to Cloudflare's R2 Storage
S3Connect = boto3.client(
    's3',
    endpoint_url=ConnectionUrl,
    aws_access_key_id=ClientAccessKey,
    aws_secret_access_key=ClientSecret,
    config=Config(signature_version='s3v4'),
    region_name='us-east-1'

)

# Upload to R2 using S3 compatible API
rs = s3_upload(Bucket=Bucket, 
    S3Client=S3Connect,
    TargetFilePath=f"test/my file.json",
    UploadObject=UploadObject,
    UploadMethod="Object"
    )
print(rs)

View the file in Cloudflare R2 console​

Cloudflare R2 file manager