Home page Portfolio

Making AWS API requests without an SDK

2023-11-21

This is a short article about how I figured out how to use AWS with a bare HTTP client. I’m writing this mainly for myself and my small software lab at my university.

Motivation

I’m a business major who knows how to script. For better or worse, I have been the most technical person on several back-end and data projects, which makes Python and AWS my wheelhouse. Despite this, I feel somewhat that I am an impostor in my roles because I lack a “hard” computer science background.

To become a better programmer, I explore languages other than Python. Elixir is my most recent fascination. I would love to use it in a real project, but I find that its existing, unofficial AWS libraries cannot meet my needs (for example, calling Textract).

I think that this is a glaring gap in my skillset. If I cannot use AWS without an official AWS SDK, then I will forever be limited to whichever languages AWS officially supports. I briefly considered working around the lack of such an SDK by making system calls to the AWS CLI, but I read that this is expensive and would defeat the purpose of using Elixir (spawning many cheap actors). If I want to use Elixir in a real project, I need to learn to use AWS the hard way.

The hard way

We will learn how to call AWS APIs directly.

I do not mean APIs that you set up yourself, for example with API Gateway. Amazon Web Services itself has API endpoints which you must call to achieve your effects. You can theoretically send an HTTP request from your terminal to spawn an EC2 instance.

When you take an AWS course, you will typically see three recommendations for interacting with the AWS platform:

The AWS Management Console (the website)
The AWS CLI (the command-line tool aws)
The various language-specific AWS SDKs (for example, boto3 for Python)

Ultimately, all three of these options are just wrappers around the process of calling AWS’s API. Strictly speaking, anything you can do with the three options can also be done with something like cURL. However, if you try, you will quickly find that all the resources on the subject urge you to just use an SDK because of how onerous the process is.

On the one hand, I think this is correct. If I can use an SDK, then I should. On the other hand, I also think that I don’t have this choice. If I am stubborn enough to try to use Elixir for a real project, I must be stubborn enough to figure this out.

What AWS tells you to do

Amazon does have some level of documentation for calling their APIs directly, but I personally find it sparse. I couldn’t find any resource that didn’t assume some level of background knowledge. This is the AWS page on signing an API request. It goes into very little detail about how exactly to put the API request together in the first place.

Let’s say you know how to make an API request to AWS, and let’s say that all you lack is now the signature. According to AWS:

First, form a string using elements of your HTTP request. This string is the “canonical request.”
Hash the canonical request with SHA256.
Form another string using elements of your API request and the hashed canonical request. This string is the “string-to-sign.”
Pass the current date into a hash pipeline that uses your secret key, the region, the service, and the constant string “aws4_request” to finally produce a key with which you can sign your string-to-sign. This will produce the “signature.”
Add the signature to your Authorization header. Send the API request with the header included.

What frustrated me is that AWS went into just enough detail to make this seem possible but not enough detail to make it clear how to actually do it.

Just list my bucket please

I gave myself one goal: successfully call an AWS API without using any of the three AWS tools.

To keep things simple, I decided that I would simply try to list all of my S3 buckets. I also decided to do my experiment in Python and to just use an AWS access key pair (instead of a role or temporary credentials).

Boto to the rescue

I don’t know if I simply lack skill, but I found it very difficult to parse the AWS documentation for how to put together an API request. Luckily, since AWS maintain officially-supported SDKs, I figured that I could simply observe how one such SDK did things.

I turned on logging for boto3, the Python AWS SDK, and made it list my buckets.

import boto3
boto3.set_stream_logger(name='botocore')

session = boto3.Session(profile_name='joe', region_name='us-east-1')
s3 = session.client('s3')

print(s3.list_buckets())

The stream of debug logs made the workflow clearer. If you want to follow along, I suggest that you explore the logs of a boto3 request in this way before you proceed.

Broadly, boto3:

Searches for credentials. In my case, it found credentials in my private credentials file on my computer.
Finds an endpoint for the given service. In my case, it settled on https://s3.amazonaws.com.
Calculates the “signature” we previously discussed. Helpfully, it prints out the canonical request, the string-to-sign, and the hashes of both.
Sends the request. Again, thankfully, it prints out the properties of the request such as the method, the URL, and the headers.

With these logs, we can try to send our own request.

The canonical request

I followed Amazon’s guide hosted here. If you are following along, I recommend that you try to follow it too. Use the boto3 logs to guide you.

The first step in Amazon’s instructions is to create the canonical request. I won’t explain the granular process here, but if you are actually creating your canonical request from scratch, you will need to construct a multiline string that adheres to AWS’s specifications. I found that for the experiment, it was sufficient to use the canonical request in the boto3 logs. However, I had to replace the value at the x-amz-date header with a datetime string I generated on script startup.

It should look something like this:

utcnow = datetime.datetime.now(tz=datetime.timezone.utc)
x_amz_date = utcnow.strftime('%Y%m%dT%H%M%SZ')

canonical_request = f'''
GET
/

host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:{x_amz_date}

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
'''.strip()

Once you have your canonical request, the second step is to hash it and retrieve its hexadecimal digest.

canonical_hash = hashlib.sha256(canonical_request.encode('utf8')).hexdigest()

The string-to-sign

The third step is to create the string-to-sign. This is a separate multiline string that contains the algorithm AWS4-HMAC-SHA256, the x_amz_date value, a slash-delimited string called the “credential scope," and the canonical hash.

current_date = utcnow.strftime('%Y%m%d')
credential_scope = f'{current_date}/{REGION}/s3/aws4_request'
string_to_sign = f'''
AWS4-HMAC-SHA256
{x_amz_date}
{credential_scope}
{canonical_hash}
'''.strip()

Since the datetimes will differ between your API request and the boto3 logs, I don’t think you can expect to simply copy-paste the boto3 string-to-sign. However, it should be clear which components you need to replace.

The signature

This is the part I found most difficult. We now need to calculate the signature, which comes from hashing a value several times with different keys.

Amazon’s documentation describes this process in pseudocode:

DateKey = HMAC-SHA256("AWS4"+"<SecretAccessKey>", "<YYYYMMDD>")
DateRegionKey = HMAC-SHA256(<DateKey>, "<aws-region>")
DateRegionServiceKey = HMAC-SHA256(<DateRegionKey>, "<aws-service>")
SigningKey = HMAC-SHA256(<DateRegionServiceKey>, "aws4_request")

They provide an alternate view of this process:

kDate = hash("AWS4" + Key, Date)
kRegion = hash(kDate, Region)
kService = hash(kRegion, Service)
kSigning = hash(kService, "aws4_request")
signature = hash(kSigning, string-to-sign)

This is pseudocode; HMAC-SHA256 and hash are abstract functions. What was most unclear to me is how exactly this HMAC-SHA256 function worked. I could not find any clear documentation about the arguments expected by HMAC-SHA256 nor any documentation about its expected return value.

After some struggling, I consulted ChatGPT for a Python implementation of HMAC-SHA256. I’m usually reluctant to ask ChatGPT for help with these things, but in this case it was very helpful.

def sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def get_signature_key(key, date_stamp, region_name, service_name):
    k_date = sign(('AWS4' + key).encode('utf-8'), date_stamp)
    k_region = sign(k_date, region_name)
    k_service = sign(k_region, service_name)
    k_signing = sign(k_service, 'aws4_request')
    return k_signing

signing_key = get_signature_key(
    AWS_SECRET_ACCESS_KEY,
    current_date,
    REGION,
    's3'
)

signature = hmac.new(signing_key, string_to_sign.encode('utf8'), hashlib.sha256).hexdigest()

My mistake was returning the hexdigest instead of just the digest after every step in the get_signature_key procedure. When I tested these new functions on the string-to-sign in the boto3 logs, they returned the expected signature, which was a relief.

Sending the request

We are almost ready to send the request. We can now use the signature in our Authorization header.

authorization_header_value = f'AWS4-HMAC-SHA256 Credential={AWS_ACCESS_KEY_ID}/{credential_scope}, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature={signature}'

Finally, we can send our request. Note that I do have to include the headers we declared in the canonical request. You can check your boto3 logs to see what headers you might need.

res = requests.get(
    'https://s3.amazonaws.com',
    headers={
        'X-Amz-Date': x_amz_date,
        'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
        'Authorization': authorization_header_value,
    }
)

If you did everything right, this request should give a 2XX response. Accessing res.text should give you some XML with your S3 buckets.

Conclusion

It was very satisfying to see the XML load into my terminal. I felt like I had levelled up as a programmer. I now feel a greater sense of freedom to continue to explore other languages, since I will no longer necessarily be blocked from using them by the lack of AWS access.

Though this was a big step for me, I realize that it took a lot of effort to call maybe the easiest AWS endpoint. I haven’t tried other endpoints, so I don’t know how difficult it will be to pass data like POST bodies and file byte data to these endpoints. I haven’t even tried re-implementing this experiment in Elixir.

Needless to say, this friction will still affect the economics of choosing one language over the other. Having an SDK will still make my AWS life that much easier, especially since my projects usually come with a lot of unknown requirements. I foresee that I will stick with Python as my main language for some time to come.

If you want to see the source code, you can find it at my repository here. If you want to get in touch, you can reach me at joe@archmob.com.