I previously posted about how My Bot Detection Broke My SEO. It all looked good on the surface, but I was too quick to pull the trigger and I did not use the correct tools to verify that this all worked as it should.
In that blog post I describe how I made use of a combination of my backend generating Open Graph tags for social media bots to consume, and using Nginx to direct bots to get the correct content. Some of you will see what went wrong from that sentence alone.
The symptoms
I did get good previews, as in I got a preview image, I got good descriptions and everything looked as it should. However when posting on LinkedIn I noticed that it was showing the wrong domain on the preview card. The links were correct, but the domain was wrong.
What is happening here is that LinkedIn is displaying my domain as api.carlsson.tech. It should be carlsson.tech.
Why is this bad for me? To put it simply, if someone sees that preview on LinkedIn and decides that they want to visit the blog via typing the address rather than clicking the link they will hit a 404 - page cannot be found. This would tell that user that "this page doesn't exist" which is bad for discovery. And that behaviour is by design, I do not want to expose the backend to the general public because that would be an attack surface.
Diagnosing the Symptom
When I diagnosed the symptom, my first thought was that the backend was providing the wrong URL in the og:url meta tag. But no, that was correct, and that is the URL that is used to create the link.
<meta property="og:url" content="https://carlsson.tech/read/my-bot-detection-broke-my-seo" />
Next thought I had was "is the frontend involved at all? I am doing SSR after all", but no. I am not involving the frontend at all when I detect a listed social media bot. Nginx is redirecting directly to the backend.
location ~* ^/read/(.+)$ {
set $post_slug $1;
if ($http_user_agent ~* "LinkedInBot|Twitterbot|facebookexternalhit|WhatsApp|Slackbot|Discordbot|Googlebot") {
return 302 https://api.carlsson.tech/v1/blog/og/$post_slug;
}
}
I then found something I should have looked for before I decided I was done the last time. The LinkedIn Post Inspector. That is displaying what LinkedIn's bot sees and what it will show. Lo and behold, it showed api.carlsson.tech but all the meta information was correct. So I dove deeper.
If we look at the Nginx configuration again, that is the only place where I use the domain api.carlsson.tech. This is correct, but there is another thing that systems are very keen on following - redirects!
When one of these bots requests https://carlsson.tech/read/..., I redirect it to https://api.carlsson.tech/v1/blog/og/.... LinkedIn follows that redirect and retrieves the page from api.carlsson.tech, which is why it displays that domain instead of carlsson.tech.
The Cure
To solve this I needed to stop using redirect to sort this, so I started thinking on how I did it with my sitemap and RSS feed. I do not redirect on those, I use proxy_pass. So I did one single change on my Nginx configuration file.
location ~* ^/read/(.+)$ {
set $post_slug $1;
if ($http_user_agent ~* "LinkedInBot|Twitterbot|facebookexternalhit|WhatsApp|Slackbot|Discordbot|Googlebot") {
proxy_pass https://api.carlsson.tech/v1/blog/og/$post_slug;
}
}
That did not work too well. Nginx rejected the configuration, so I needed to do some further digging.
My attempted solution combined a regular-expression location, an if statement, and a proxy_pass containing a dynamically constructed URI:
proxy_pass https://api.carlsson.tech/v1/blog/og/$post_slug;
According to the proxy_pass documentation, this is not supported in the straightforward way I had attempted. Because the surrounding location uses a regular expression, Nginx cannot reliably determine which part of the original request URI should be replaced by the URI supplied to proxy_pass.
Nginx ticket #2090 also discusses this restriction when proxy_pass is used inside an if block.
I did not know that when I started. I reached the answer through trial and error, then found the documentation that explained why my first attempt failed.
What I wanted was simple: keep the public URL unchanged for the bot while serving the response directly from my API. The frontend should not need to handle these requests.
That is a deliberate design decision. Routing the request through the frontend would cause the backend to generate the metadata and then require the frontend to process and return it. It would add another execution step and introduce unnecessary bot-specific complexity into the frontend.
This is machine-to-machine communication, which is what the API is for.
The solution turned out to be a bit of a hop. I use a rewrite rule when I detect a bot, and then I use an internal /_og/ location that performs the proxy_pass to the API. The /_og/ route does not exist in my application; it only exists inside Nginx.
location ~* ^/read/(.+)$ {
if ($http_user_agent ~* "LinkedInBot|Twitterbot|facebookexternalhit|WhatsApp|Slackbot|Discordbot") {
rewrite ^/read/(.+)$ /_og/$1 last;
}
}
location ~* ^/_og/(.+)$ {
set $slug $1;
proxy_pass https://api.carlsson.tech/v1/blog/og/$slug;
proxy_set_header Host api.carlsson.tech;
proxy_set_header X-Real-IP $remote_addr;
proxy_hide_header X-Powered-By;
internal;
}
A very complicated result for something that should have been simple to start with. The answer was literally hiding in the architecture, but now it all works as expected!