Yann Pretot

One Rule Made Our Non-cacheable Cloudflare Traffic 50% Faster

Fri, 10 May 2024 00:00:00 +0000

tl;dr: Ensure your requests are eligible for caching (even if responses aren’t) so that they use Tiered Cache’s faster architecture. For example, you can create a Cache Rule with the Use cache-control header if present, bypass cache if not action for Edge TTL.

Checking on a webapp’s performance, I found latency results from farther away countries underwhelming. This app runs on AWS in eu-west-3 (Paris), with Cloudflare in front for DDoS protection and caching of static assets.

That’s a lot of orange. Orange is slow. Slow is bad.

The configuration was close to the defaults, and Tiered Cache was enabled (set to Smart Tiered Caching Topology).

An aside on Cloudflare’s Tiered Cache

Well, Tiered Cache is pretty amazing, and free. According to Cloudflare:

By enabling Tiered Cache, Cloudflare will dynamically find the single best upper tier for an origin using Argo performance and routing data.

This practice improves bandwidth efficiency by limiting the number of data centers that can ask the origin for content, reduces origin load, and makes websites more cost-effective to operate.

Also, and this is of importance, Cloudflare carries requests and responses very efficiently between the client data center and the upper tier, which in turn communicates with the origin on cache misses. In fact, way more efficiently than if the client data center stretched a TCP session all around the world to speak directly to the origin. This makes the request-response round trip far faster, even for non-cacheable content, as things like TLS handshakes are way quicker on short distances.

Reading this, we’d think that it applies to all requests. Well, we’d be wrong.

The problem

Cloudflare’s request processing logic is made of multiple phases each responsible for a specific product like WAF, Rate Limiting or Cache. As I understand it, at or before the beginning of the Cache phase, Cache Rules and the default policy¹ are used to determine whether the request looks like it is for cacheable content. If not, caching is bypassed entirely. This is what happens when you see DYNAMIC in the CF-Cache-Status response header. In theory, that’s good because it prevents expensive-ish cache lookups and useless processing for requests that have no hope to ever be cached anyway.

However, as Tiered Cache is part of the Cache phase, it doesn’t get to engage for those ineligible requests. Origin requests then come from the Cloudflare data center that received the client request rather than Tiered Cache’s upper tier data center. This is suboptimal:

As both Cloudflare and many hosting providers are doing hot potato routing, packets travel freely over the Internet through paths that no one really monitors nor optimizes for performance², leading to high latency.
The client data center might be very far away from the origin, making even the most direct non-congested path tens to hundreds of milliseconds long, which TCP doesn’t like very much.

Cache Rules to the rescue

Even if we know our content is not cacheable and never will be, we want the client data center to think there’s still a chance so that the Cache phase isn’t skipped and all³ traffic goes through Tiered Cache, cold potato-style⁴, only exiting to the origin from the upper tier data center.

To do that without altering the existing behavior, we created a Cache Rule targeting only requests not covered by the default policy and applied the Use cache-control header if present, bypass cache if not action for Edge TTL.

The expression used to match traffic not covered by the default caching policy is:

(not http.request.uri.path.extension in {"7z" "csv" "gif" "midi" "png" "tif" "zip" "avi" "doc" "gz" "mkv" "ppt" "tiff" "zst" "avif" "docx" "ico" "mp3" "pptx" "ttf" "apk" "dmg" "iso" "mp4" "ps" "webm" "bin" "ejs" "jar" "ogg" "rar" "webp" "bmp" "eot" "jpg" "otf" "svg" "woff" "bz2" "eps" "jpeg" "pdf" "svgz" "woff2" "class" "exe" "js" "pict" "swf" "xls" "css" "flac" "mid" "pls" "tar" "xlsx"} and http.request.uri.path ne "/robots.txt")

Conclusion

Results vary between measurements, but overall, that’s way greener.

We did get a ~50% decrease in latency from the farther locations, which is pretty good for a single rule change. However, this only works for GET requests as other HTTP methods are never eligible for caching no matter what.

It would be awesome if Cloudflare made all requests, cacheable or not, go through Tiered Cache when it’s enabled. This behavior would be simpler in my opinion and more aligned with customer expectations, albeit rendering the more advanced Argo Smart Routing a bit less compelling.

By default, Cloudflare caches resources if they match a list of 56 file extensions (plus /robots.txt) that are usually static assets (images, CSS, JS, etc.) and the origin response headers don’t forbid it. ↩︎
Cloudflare does monitor origin reachability and try to failover to a secondary path if the natural one goes down thanks to their Orpheus system. This is however resilience- and not performance-oriented. ↩︎
All GET requests, that is, as other methods are not cacheable at all and will always bypass the Cache phase. To make those faster, you’d have to use Argo Smart Routing (which is not so cheap) or build something similar yourself with Workers (which I’ve also done and will probably post about later). ↩︎
Or, more accurately, lukewarm potato-style, as the traffic between Cloudflare data centers does go traverse the Internet but through, one assumes, monitored and optimized paths. Anyway, the HTTP request is not merely tunneled but transmitted more efficiently instead, so we still win. ↩︎

A (More) Secure Workstation

Fri, 15 Sep 2023 00:00:00 +0000

In a world where CI pipelines are (mostly) run in isolated disposable VMs, where every new workload is containerized and where supply chain attacks are increasingly frequent, I find mind-boggling that a lot of dev/ops workstations are still a nightmarish melting pot of brew-, npm- and pip-installed tools. Those come with tens if not hundreds of dependencies (which invariably cause headaches a few months down the line), and this whole zoo runs in your $HOME with your own privileges. What’s to prevent any of these to go rogue and install a backdoor somewhere in your .zshrc or anywhere else?

Two years ago, when I bought my current laptop and started fresh, I decided to try to keep it as clean as possible. The more minimalist the setup, the longer I would go without dealing with dependency hell, python version discrepancies and, hopefully, security issues. It’s a Mac, so the ideas here are geared towards macOS, but they should work on any UNIX-like system.

I’m no security expert, so I won’t pretend I have all the answers: this piece is more of an RFC than a how-to, and I’d love to hear your thoughts and tips on the matter.

The idea

Install as few third-party apps as possible
No pip, no npm, no brew, and certainly no curl | sh
Every tool gets its own Docker image, and runs in an ad-hoc disposable container
Dev projects also run inside containers

I find containers ideal to deal with dev tools, as they’re mostly isolated from the host by default and enable the user to define explicit interfaces: share specific environment variables, precise bind mounts (possibly read-only), port mappings, etc. They’re also disposable, so you can easily make them single-use to prevent any leak between projects or environments and avoid long-term compromission.

How it works

An example:

function aws () {
    docker run --rm -it \
    -v $PWD:$PWD -w $PWD \
    -v "$HOME/.aws/config:/root/.aws/config:ro" \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN \
    -e AWS_PAGER \
    --entrypoint "/usr/local/bin/aws" \
    amazon/aws-cli:2.13.13 $@
}

This function lives in my ~/.zshenv. It’s a small wrapper around docker run, which runs the AWS CLI from the official image and mimicks the host environment, except it’s scoped to the current directory (useful for s3 cp) and has read-only access to my ~/.aws/config file. The credentials come from craws, another small tool I wrote that gets vended temporary ones from AWS IAM Identity Center (formerly SSO) and exports them in the current terminal session.

Every tool I use (ansible, terraform, kubectl…) has its own function, sometimes with a custom image, and always with as little privileges as possible. For those I use less frequently, there’s always docker run --rm -it -v $PWD:$PWD -w $PWD ubuntu bash to get a shell in a clean temporary Ubuntu container, apt install what I need and run it.

The UNIX philosophy of composability still applies as stdout and stderr are forwarded: I can pipe one tool’s output to another, and to that end, jq also has its own function:

1
2
3

function jq () {
    docker run --rm -it --pull never localhost:5000/sw/jq $@
}

and Dockerfile:

1
2
3

FROM alpine:3.15
RUN apk add --no-cache jq
ENTRYPOINT ["/usr/bin/jq"]

The image’s tag is prefixed with localhost:5000, a registry which doesn’t exist, to ensure I never pull it from the Docker Hub and only use the locally built one.

Things sometimes get funky, for example with port mappings, as tools should bind to 0.0.0.0 for Docker to be able to forward them traffic. Example with kubectl:

function kubectl () {
    # If command is 'kubectl proxy', bind to 0.0.0.0 and forward port 8001
    args=($@)
    docker_args=()
    if [ ! -z "$1" ] && [ "$1" = "proxy" ]; then
        args+=("--address=0.0.0.0")
        docker_args+=("-p" "127.0.0.1:8001:8001")
    fi

    docker run --rm -it \
    -v $HOME/.kube:/root/.kube \
    -v $PWD:$PWD -w $PWD \
    ${docker_args[@]} \
    localhost:5000/sw/k8s-tools:latest kubectl $args
}

What's next

While this has been working pretty well for me so far, it’s highly custom to fit my own workflow and not very portable. While writing this post, I thought that maybe this could become a project in its own right, with a central repository of “recipes” (Dockerfiles or images + wrapper code) and some form of CLI to manage them.

So I’ve streamlined what I had a bit, and here it is: Toolship is now on GitHub!

Where Are the Parisians?

Thu, 23 Feb 2023 00:00:00 +0000

Since I settled in the French capital, I’ve always been curious to know what other Parisians were doing. Part FOMO, part simple desire to discover cool places.

SEO articles recommending the same touristy spots get old quickly. The Snapmap is not very relevant (anymore?), while other social networks give quite a peacemeal view of what’s going on: don’t expect to hear about a Porte de Versailles conference on a day of protests at République.

So I decided to build my own map using Open Data: here’s BustleMaps 🎉 Updated every minute and accessible from your own browser, it makes you feel the heartbeat of Paris at a glance.

The map, as seen on February 10th, 2023, at 10:10 PM

How it works

Smoove, which operates the Velib' bike-sharing service, offers a public API that indicates in near real-time the number of bikes available at each station.

Velib' handles more than 100,000 rides a day on average, between the 1,447 stations in its network. Probably enough to get an idea of what’s going on in the city.

By comparing the number of bikes available in a given station from one minute to the next, we can approximate how many bikes have been taken or returned. Doing so for all stations and accumulating the results over 30-minute windows gets us everything we need to build a pretty heatmap!

The algorithm is written in Python and runs on an AWS Lambda, triggered every minute by an EventBridge rule. The data is stored in an S3 bucket, and the map is displayed on the client side with MapLibre GL JS.

Limitations

This is an approximation, biased towards transportation and skewed by some uses of Velib', such as intermodality (bike + train, for example, which boosts areas around train stations at rush hour). From a sociological point of view, it is centered on cyclists who use Velib', so quite far from representing the entire population.

The map would probably be much more interesting with data from Google Maps or CityMapper, or those collected by a data broker for advertising targeting or even those contained in the HLR of a major mobile carrier. But these are very valuable datasets, which are therefore not available.

Bike rebalancing operations across stations bias the figures: a station being refilled gives the impression that the area is more lively than in reality. Algorithmic adjustments limit the impact but do not completely curb it.

The number of slots in a station is limited: if a station is full, cyclists cannot stop there, which suggests that the area is less frequented than it really is. Some code partially corrects the problem, and we can assume that these trips end at other nearby stations.

The future

A few ideas for further development:

Display daily timelapses
Allow to consult the map at a specific point in the past
Add other data sources to reduce bias and better detect areas of high activity
Add other cities in France and around the world

In the meantime, I have walks to do :-)

Measuring TTFB Worldwide

Thu, 27 Oct 2022 00:00:00 +0000

While I was trying to assess HTTP latency around the world for a small project I’ve yet to talk about, I grew frustrated by the limits of the online tool I was using:

It was slow because it collects results from all tested locations before starting to show them to the client (with a slow animation, on top of that)
It didn’t work very well: a third of my tests were failing for no known reason
It had quotas per 12h period, preventing me to do all the experiments I needed

That was enough to prompt me to start rebuilding the tool my way. As luck would have it, I was at the same time playing with Cloudflare Workers, which I only had been watching from afar until then. This little project would be a good way to explore Workers further.

A few hours, a bit of JS, and a drop of Bootstrap later, here’s TTFB Tester, free for all to use 🎉

Of course, when the CDN’s cache isn’t warm, it’s not particularly pretty. Another example:

Tech-wise, until I expand on the architecture, here are a few remedies to the problems I had with the other tool:

Slowness: Rather than a lengthy HTTP call waiting for results from all locations, every test run opens a WebSocket connection to the orchestrator Cloudflare Worker. This Worker then launches tests concurrently from the 28 datacenter locations and gradually sends results to the client through the WebSocket, as soon as they arrive.
Reliability issues: It’s hard to pinpoint why tests failed on the other tool. Mine seems to work more reliably in general, maybe because each location is independent: if one fails to run the test¹, it doesn’t affect the others, its line just disappears from the results table.
Quotas: Of course, when you’re the one in charge of limits, things get a lot easier on this part. They are less restrictive here, mainly because the serverless architecture I’m using entails a virtually null cost as long as usage stays at the level I’m predicting.

If a picture is worth a thousand words, a trial is probably ten times more… Why don’t you take the tester for a spin yourself? Or use it for real needs, for that matter: that’s what it’s here for!

It happens (very) often for a few locations because of the way the tests are run at this time. Those locations still stay in the pool because the nature of the tool means that their results are still good to take, even if they’re sporadic: here, I’m looking for comprehensiveness more than reliability of individual locations. It’s also a trade-off I’m willing to make for now, in exchange for the tool costing virtually nothing to run. This indeed happened between the two screenshots above, as Toronto left and Madrid appeared. ↩︎

Prevent Indexing of *.cloudfront.net URLs

Sat, 23 Apr 2022 00:00:00 +0000

When creating a CloudFront distribution, AWS assigns it a generic domain name, such as d2d7frhciy9p7h.cloudfront.net. You can add your own domain as an alias, but the original one cannot be disabled. Your website is then available using either your custom domain or the default .cloudfront.net endpoint.

From a security standpoint, this is not a big deal: this endpoint doesn’t give access to more data than your own domain. However, things start to get messy when search engines add those URLs to their index:

Users can stumble upon these plain-looking links when searching for your content, which isn’t good for their experience
Search engines like Google can detect that the same content is served from multiple domains, and penalize you for that

The Internet has various answers to that problem. For example, if your origin is Host header aware, you could make sure CloudFront forwards the Host header and then configure your web server to only answer to requests with the right Host.

However, if you don’t want this kind of coupling or if you’re using simpler origins like S3 buckets, there’s another way to do it: CloudFront Functions.

To block access through the .cloudfront.net domain, first create a CloudFront Function with the following code:

function handler(event) {
    var request = event.request;
    var host = request.headers.host.value;

    if (host.endsWith('.cloudfront.net')) {
        return {
            statusCode: 403,
            statusDescription: 'Forbidden',
            headers: {
                'x-reason': {value: 'Access via .cloudfront.net distribution domain name forbidden'}
            }
        };
    } else {
        return request;
    }
}

You can then attach that function to Viewer request events in your distribution’s Behaviors:

That’s it! Now, when trying to access your .cloudfront.net domain, CloudFront should return HTTP 403 errors:

A drawback is that you can’t set a response body with CloudFront Functions, so the user gets a blank page (which shouldn’t be a problem as nobody’s supposed to reach that domain in the first place).

Cost-wise, the free tier currently includes 2M function executions per month. After that, you’ll get billed $0.10 per million requests.

Custom Domains for Lambda Function URLs

Thu, 14 Apr 2022 00:00:00 +0000

AWS has recently released a breakthrough feature for Lambda: Function URLs. You tick the box, you’re given an URL, and you can then invoke your Lambda function just by making requests to that URL. I think that for the right use-cases, it’s a great addition, especially given the complexity of setting up an API Gateway without using third-party Terraform modules, where you’d have to worry about IAM permissions, logs, CORS, etc.

Lambda URLs indeed have the following advantages:

They’re free (API Gateways make you pay per request)
They’re (very) easy to use
They don’t have a time limit other than your Lambda’s (which means requests can take up to 15 minutes)

But there’s a slight inconvenience: as users noticed, you’re given a long, meaningless URL that can’t be customized nor changed.

The Solution

Annoyed users, rejoice! You can front your Lambda with CloudFront using your own beautiful domain name. That’s arguably easier than setting up an API Gateway, bumps the runtime limit from 30 to 60 seconds¹ and you get to use the included customizable caching, among other things. You lose some of the benefits I’ve outlined before, though.

To do that, you’ll need:

Your Lambda function and its Function URL, in whichever region you want
An ACM certificate for your domain or subdomain in the us-east-1 region
A CloudFront distribution, with Host header forwarding disabled
Route53 ALIAS A and AAAA records pointing to your distribution (or a third-party DNS CNAME)

If you’ve already set up CloudFront distributions before, the most tricky part might be disabling Host header forwarding: you can’t ignore specific client headers in CloudFront, so you have to whitelist those that you do want to send to Lambda in your Origin Request Policy, or send none at all. For example:

And even more if you ask AWS Support for a quota increase. ↩︎

Hi there!

Sun, 20 Feb 2022 00:00:00 +0000

I’m ~~Janet~~ Yann.

This is my new, shiny corner of the Web. I’m French, currently 22 and as passionate about IT and tech as I’ve ever been. While I have no plans to limit the scope of the content I’ll post here, there’s a high chance that most of it will be tech-related.

Although I’ve already posted pieces here and there over the past few years, there wasn’t a place to glue them together¹. I’ve also been willing to write about a thing or two during the past few months, but didn’t feel like publishing on yet another Medium blog. So, here we are!

Technical bits for nerds like me

This website’s architecture seems boring — even if I find its elegance fascinating — which is kind of its whole point:

An Amazon S3 bucket configured for static website hosting
A Cloudfront distribution
Hugo as a static site generator
A custom theme, based on WingLim’s re-implementation of Tania Rascia’s beautiful website (many thanks!)

Why so? Because I’ve already wrestled with blogging in the past, and WordPress’s bells, whistles and maintenance needs or Ghost’s resource consumption meant those previous initiatives didn’t stand the test of time.

In IT, minimalism and simplicity often make for very robust solutions. Here, they also enable my small blog to be essentially free, extremely secure and lightning fast.

You’d be right to point out that I could have used Netlify, GitHub Pages or the likes, but I try to avoid free riding, and I already had custom Terraform templates that make the whole thing a breeze on AWS.

Edit: Turns out this wasn’t elegant enough as the site now runs out of Cloudflare Pages, which does look a fair bit like Netlify :-) This got me more automated builds/deployments and simpler infrastructure, plus I do pay a Cloudflare invoice every month so still no free-riding.

Credit where it's due

Even if I’ve encountered many blogs during my countless hours browsing HN or Twitter, some of them have proven particularly inspiring, namely:

Well, here’s to writing and elegant solutions!

Especially since Twitter closed my previous account when I tried to make my birth date right… Oops. ↩︎