Migrating from Netlify to Cloudflare for AI bot protection

I'm an independent content creator that uses my content partly as a way to to get interest in my paid services. I don't know about you, but for me the idea of an AI LLM scraping my content to regurgitate it without any benefit to me seems like a dumb business decision.

When Cloudflare announced their new service to block AI bots, scrapers, and crawlers, I was intrigued. Many folks are playing whack-a-mole with their robots.txt files, but there's no guarantee a bot will even respect them. As far as running my own server and managing IP block lists:

I ain't got time for that.

For years, I've happily hosted my website on Netlify. However, it currently has no AI bot protection features or services. I'll likely still use it for sites that don't contain expert content, but for this blog, I knew I needed a change.

In this post, I document how I moved my 11ty site from Netlify to Cloudflare, with all its ups and downs. Most of it should be relevant to any static site generator. Jump to a section:

• Moving the core content and redirects to Cloudflare
• Blocking the AI bots
• New contact form handling
  • Setting up local development: Hello Wrangler
  • Hello Cloudflare Functions
  • Collecting form data in the Function
  • Setting up email with Resend
  • Next steps: spam filtering?
• Hiding content in Github
• Cleanup
• Conclusion

Moving the core content and redirects to Cloudflare #

For the most part, Cloudflare's migration docs worked well except for a few hiccups. Make sure you read that doc thoroughly in case your use case is different from mine.

First, I had a few npm scripts that needed updating in my package.json. Namely, I used netlify dev for local development via the npm start command so that I could use secrets stored in Netlify to build locally. I think this may have been for webmentions and/or playing with serverless functions. My webmentions only build in production, so currently I don't need a similar set up. Thus, I switched my start script back to the core 11ty command of npx @11ty/eleventy --serve.

Next, I had several redirects specified in my netlify.toml file. I had to switch to the _redirects file format and had 2 hiccups:

• Unlike the TOML file method, you must copy the _redirects file into your build folder. With 11ty, that means adding a line your .eleventy.js file:

  eleventyConfig.addPassthroughCopy("_redirects");

• I don't know enough about redirects, but either this other method or the way Cloudflare is configured meant that I had to add cases for instances with and without trailing slashes to my _redirects:

  /mars-randomizer https://projects.sia.codes/mars-randomizer/
  /mars-randomizer/ https://projects.sia.codes/mars-randomizer/

Netlify has a beautifully simple forms solution that would clearly not work on Cloudflare. So, I temporarily disabled the form with a note to contact me on social media in the interim. Per the migration instructions, I deleted the data-netlify="true" attribute on my contact <form>.

Once all of that was done, I pointed my name servers to Cloudflare instead and pointed my root domain (and a www redirect) to the new Cloudflare pages deployment. See the various links in Cloudflare's migration post.

Blocking the AI bots #

Once your site is live, you can turn on AI bot protection. The Cloudflare blog article points us in the right direction. However, the article's dashboard link didn't work for me. It said I did not have permission. Luckily I was able to find it by going to my Dashboard > Websites > sia.codes > Security > Bots.

I did not turn on bot fight mode because Nicolas Hoizey mentioned he had issues with feed aggregators not being able to get his feeds (see Mastodon post). I'm currently less worried about regular bots and would rather not mess up folks who follow my RSS feeds. This may change.

New contact form handling #

I'm a woman in tech on the internet which is one too many potential harassment vectors for my taste. Thus, I choose not to (easily) share my email. Instead, I've opted for contact forms to add a layer of filtering. Netlify's forms feature is ridiculously easy, and I'm sad to lose it.

I had a few options going forward:

• Just share my email address
• Create a Pages Function (serverless function) that emails me the form data
• Use a table/database API to send form data and trigger notifications from there

I was reluctant change my stance on emails vs forms. And, over the last several years I've only logged in to Netlify to see my form data once or twice and that was more to check the spam filtering. Thus, I figured I really didn't need a table or database. I just want an email. Hence, I went with the second option - emailing myself from a Function.

Setting up local development: Hello Wrangler #

Before we can dive into the Functions code, we need to set up our local dev environment using Wrangler:

npm install wrangler --save-dev

Add the .wrangler directory to your .gitignore file. That directory stores temporary files and local storage for the cache and various services.

You have to build your 11ty site before starting Wranger. You could do this manually one time with an 11ty build command like npx @11ty/eleventy. Or, if you expect to make changes to core static site code at the same time, build your 11ty site in watch mode with npx @11ty/eleventy --watch. I saved this as my watch script in my package.json. Then, in another terminal window, run npx wrangler pages dev _site. For convenience, I added this as my wrangler script in my package.json.

Open your site at http://localhost:8788/ and verify it works!

Hello Cloudflare Functions #

Now that Wrangler is set up, we can start building a basic Function. Cloudflare Functions are almost the same as Cloudflare Workers. The main difference is that Functions are scoped within your Pages project. They have some other notable differences - Workers support more features than Functions.

From Get started, create /functions/helloworld.js in the root of your project:

export function onRequest(context) {
  return new Response("Hello, world!")
}

Open your site at http://localhost:8788/. You should see your regular home page there. Now, navigate to http://localhost:8788/helloworld. You should see an unstyled page with only the raw "Hello, world!" text. 🙌

You can delete the helloworld.js file as it was only a test to get Wrangler and Functions running successfully.

Collecting form data in the Function #

This section is mostly a pared-down version of Cloudflare's form tutorial.

In the HTML of your <form>, add (or replace) the action attribute to point to the new function route you are about to create. Note that we're adding an /api folder to the path and that I made my action's name more verbose than the full tutorial:

<form class="form" name="contact" method="POST" id="contact-form" action="/api/submit-contact-form">

Next, create the /functions/api/submit-contact-form.js file in the root of your project. Use the same file name you declared in your form's action attribute.

/**
 * POST /api/submit-contact-form
 */
export async function onRequestPost(context) {
  try {
    let input = await context.request.formData();
    let pretty = JSON.stringify([...input], null, 2);
    return new Response(pretty, {
      headers: {
        'Content-Type': 'application/json;charset=utf-8',
      },
    });
  } catch (err) {
    return new Response('Error parsing JSON content', { status: 400 });
  }
}

Run 11ty build/watch and then wrangler. Go to your contact form, and fill it out. When you hit submit, it should redirect to your api route and show you an array of key-value pairs from your form data.

Nice!

If you want a JSON object instead, continue with the next step by adding a helper for the form data:

/**
 * POST /api/submit-contact-form
 */
export async function onRequestPost(context) {
  try {
    let input = await context.request.formData();

    // Convert FormData to JSON
    // NOTE: Allows multiple values per key
    let output = {};
    for (let [key, value] of input) {
      let tmp = output[key];
      if (tmp === undefined) {
        output[key] = value;
      } else {
        output[key] = [].concat(tmp, value);
      }
    }

    let pretty = JSON.stringify(output, null, 2);

    return new Response(pretty, {
      headers: {
        'Content-Type': 'application/json;charset=utf-8',
      },
    });
  } catch (err) {
    return new Response('Error parsing JSON content', { status: 400 });
  }
}

Now we need to do something with this data.

Setting up email with Resend #

As I mentioned earlier, I tried and failed to get direct email to work in Functions. Briefly, I attempted setting up a separate Worker to call from the Function, but that was getting silly complicated. Instead, I decided to try a new developer-focused transactional email service called Resend.

I mostly followed the tutorial for sending emails with Resend in Workers but implemented it in Functions instead. It misses a few important steps, so their example repo is useful to reference.

Per the tutorial, create an account, add your domain records, and get an API key.

Your API key should be stored as a secret. I also chose to store my sender and recipient emails as secrets (docs). Note that the sender email needs to be from the domain you set up in Resend.

To use secrets locally, create a .dev.vars file in the root of your project, and add it to your .gitignore file. Set each one as a key-value pair in the file like so:

WEBMENTION_IO_TOKEN="mYsEcrEt"
RESEND_API_KEY="rEsendSecreT"
SENDER_EMAIL="[email protected]"
RECIPIENT_EMAIL="[email protected]"

Next, the docs say you can add secrets to production via the Wrangler CLI like so:

npx wrangler secret put RESEND_API_KEY

That seemed to work, but in production, those env variables were missing. So I used the dashboard method of adding environment variables and selected "encrypt" for each.

Here's my final Function file:

/**
 * POST /api/submit-contact-form
 */
import { Resend } from 'resend';

export async function onRequestPost(context) {
  try {
    let input = await context.request.formData();

    // Convert FormData to JSON
    // NOTE: Allows multiple values per key
    let output = {};
    for (let [key, value] of input) {
      let tmp = output[key];
      if (tmp === undefined) {
        output[key] = value;
      } else {
        output[key] = [].concat(tmp, value);
      }
    }
    // output: {
    //   name: 'Jane Doe',
    //   'contact-name': '',
    //   email: '[email protected]',
    //   subject: 'help me',
    //   message: 'this is my message'

    const honeypot = output["contact-name"]
    // Return early with pretend confirmation if bot hit honeypot
    if (honeypot !== "") {
      return Response.redirect("https://sia.codes/contact-confirmation", 303)
    }

    // Using text instead of email so that I don't need to sanitize it
    const resend = new Resend(context.env.RESEND_API_KEY);
    const { data, error } = await resend.emails.send({
      from: context.env.SENDER_EMAIL,
      reply_to: output.email,
      to: context.env.RECIPIENT_EMAIL,
      subject: `[SIA.CODES] Contact form request from ${output.name}: ${output.subject}`,
      text: output.message,
    });
    console.log({data, error});

    if (error) {
      return Response.redirect("https://sia.codes/404", 303)
    } else {
      return Response.redirect("https://sia.codes/contact-confirmation", 303)
    }

  } catch (err) {
    return Response.redirect("https://sia.codes/404?error=json_parsing", 303)
  }
}

Notice that I modified the function in a few ways:

• I added a check for a visually-hidden honeypot field.
• For the email, I set a reply_to attribute to the form's email field to make replying easy for me.
• For the email, I set the format as plain text using a text attribute instead of html. This is so that I wouldn't have to sanitize the message field.
• I redirect to the 404 page if Resend fails. Similarly, if the code fails in any other way, I redirect to the 404 page but with a query attribute to help me distinguish from a Resend error. I should probably change both to something more user-friendly, but it was a quick way to show the user something broke and the message likely failed.
• I redirect to a confirmation page thanking them for their message if everything succeeds.

This worked great in local development. Sadly, in production, I kept getting errors about react-dom. Apparently, React is a dependency of Resend. Even though I did not use React at all, the Resend npm package still calls it. So, sadly, you'll have to install those dependencies to your package.json. It's dumb. I know. The tutorials should show this initial NPM install command instead:

npm i resend react react-dom

Next steps: spam filtering? #

Netlify forms also performs spam filtering using Akismet which has a free "personal" tier (and an API). I'm going to wait and see how much spam I get. If you look closely at the code above, I already have a honeypot which filters out a chunk of the bots. If I add spam filtering, I'll write a new post and link to it from here.

Hiding content in Github #

All of the steps thus far have been focused on my deployed site. However, I keep my blog source code public to help out other 11ty developers. This means that my blog post content is also public.

Many possibilities exist for hiding the content. I decided with creating a private fork of my repo. The private version is where I will add new posts. The public version is where I will make changes to the core code. Then, I can sync the fork with the upstream main branch.

For now, old posts are still in the public version. I may clean this up later.

Cleanup #

As a last step, I cleaned up the following leftover Netlify items:

• In the Netlify dashboard, go to your site's configuration and unlink the repository so that it stops trying to build your site when you push to your repo.
• Delete the netlify.toml file.
• Remove the local .netlify folder.
• If desired, remove .netlify from the .gitignore file.
• I had a stray honeypot data attribute on my contact <form> that was only used in Netlify, so I deleted that.
• Update any Github or other cron-type actions that use the Netlify build hook to point to a Cloudflare deploy hook. Don't copy the full URL into your actions file. Set up a secret in your repo settings and use that. See my action for an example.

Conclusion #

Overall, I was very happy with Netlify. If I did not mind letting AI bots scrape my site to hell and back again, I would have stayed.

Cloudflare seems more powerful and more performant, but more work to set up. Their edge network seems to have more nodes as well resulting in a faster user experience. My preliminary real user data for TTFB (time to first byte) has dropped slightly - it was already super fast. Either way, as a web performance engineer, this makes me happy. It seems to be translating to start render and LCP (largest contentful paint) as well, but I need more time to gather more data to be sure.

Cover photo by Morgane Perraud on Unsplash

AI Tools Web