Linking F5Bot to Telegram

We just launched UniHosted and need to do a fair bit of marketing to make this a big success. We're using F5Bot.com to monitor mentions of our competitors and keywords. F5Bot only works via email, which is not ideal if you want to reply to all messages with a team of three. And we're a Telegram-loving team, so we wanted to get the F5Bot emails in our Telegram channel.

Here's how we did it!

Table of Contents

• Step 1: Saying Hello to SendGrid
• Step 2: Decoding the Magic (with some Next.js 13 sprinkled in)
  • Fetching the Email
  • Parsing the HTML
  • Sending the Message to Telegram
• Voila! UniHosted x Telegram
• Wrapping Up

Step 1: Saying Hello to SendGrid

First things first, we set up the Inbound Parse feature of SendGrid. It’s a pretty nifty feature. You create a subdomain with an MX record pointing straight to SendGrid. Every email sent to that subdomain gets parsed and pushed to a webhook. Neat, right?

We created a subdomain and added the MX record to our DNS. We then created a webhook in our Next.js app to receive the emails:

Step 2: Decoding the Magic (with some Next.js 13 sprinkled in)

Okay, now to the fun part – the code! We used Next.js 13 (because who doesn’t love being updated?). Here's a quick peek into what our code does:

1. Fetches the email content sent by F5Bot.
2. Parses the HTML of the email to extract the juicy bits.
3. Sends the relevant info to our beloved Telegram channel.

Fetching the Email

We parse the incoming email as with FormData.

const formData = await request.formData();
const html = formData.get("html") as string;
const from = formData.get("from") as string;

if (!from?.includes("@f5bot.com"))
  throw new Error("Email is not from f5bot.com");
if (!html?.length) throw new Error("We did not receive any HTML");

// Do something with the HTML
console.log(html);

Parsing the HTML

The HTML that we get from F5Bot looks like this:

<h1>F5Bot found something!</h1>
<h2>Keyword: "analytics"</h2>
<p style="margin-left: 10px">
  Reddit Posts (/r/consulting/):
  <a
    href="https://www.reddit.com/r/consulting/comments/16185tt/have_i_screwed_myself_for_my_future/"
    >Have I screwed myself for my future?</a
  >
  <br />
  <span style="font-family: 'Lucida Console', Monaco, monospace">
    I have a major in Business <strong>Analytics</strong> and minor
    in Computer Science and have always wanted to land something
    in the FinTech field.
  </span>
</p>
<!-- More results -->
<p>
  Do you have comments or suggestions about F5Bot? Please hit reply and let me
  know what you think!
</p>
<br />
<p>
  <font color="#ff0000">Want to save on AWS?</font> With
  <a href="https://f5bot.com/xxx">Company X</a> you can save
  <strong>up to 75%</strong> in minutes. No code or engineering required.
  <a href="https://f5bot.com/xxx">Sign up</a> to see how much you can save.
</p>

The structure of the email is as follows. After every <h2> tag, there are one or more <p> tags with the results. Per <p> tag we need the subreddit, link, and the link text. We skip the snippet text, as it's too much information for our Telegram channel.

At the end of the email is some marketing text, which we also skip. It's after the <p> tag which contains the text "Do you have comments or suggestions about F5Bot?".

I love Cheerio which I use to parse the HTML. It looks somewhat like this:

import { load } from "cheerio";

const $ = load(html.toString());

// Delete marketing text after "Do you have comments or suggestions about F5Bot?"
const comments = $("p:contains('Do you have comments')");
comments.nextAll().remove();
comments.remove();

// Put every result in an array
const messages = [] as string[];

// Loop over every h2 tag
$("h2").each((index, h2Element) => {
  // Get the keyword ("Keyword" or "Keywords")
  const keyword = $(h2Element)
    .text()
    .replace(/Keywords?: "/g, "")
    .replace(/"/g, "");
  $(h2Element)
    .nextUntil("h2", "p")
    .each((j, pElement) => {
      // Get the subreddit
      const subreddit = $(pElement)
        .text()
        .match(/\(\/r\/[^)]+\)/);
      const aTag = $(pElement).find("a");
      const link = aTag.attr("href");
      const linkText = aTag.html();
      if (subreddit) {
        messages.push(
          `<em>${keyword}</em> in ${subreddit[0]} <a href="${link}">${linkText}</a>`
        );
      } else {
        messages.push(`<em>${keyword}</em> <a href="${link}">${linkText}</a>`);
      }
    });
});

// Do something with messages
console.log(messages);

Sending the Message to Telegram

We use the Telegram Bot API to send the message to our Telegram channel. We use the sendMessage method to send the message as HTML:

const { TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID } = process.env;

const searchParams = new URLSearchParams();
searchParams.set("chat_id", TELEGRAM_CHAT_ID);
searchParams.set("text", messages.join("\n"));
searchParams.set("parse_mode", "HTML");
searchParams.set("disable_web_page_preview", "true");

const url = new URL(
  `https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage`
);
url.search = searchParams.toString();

await fetch(url);

Voila! UniHosted x Telegram

Once our code was up and running, every time F5Bot discovered our keyword, we got a nifty notification on Telegram. It felt like we had given our team superpowers (or at least the power of instant info 😉).

Wrapping Up

Got questions? Or just wanna say hi? Drop us a line. And hey, if you're still managing UniFi Access Points manually, give UniHosted a whirl – we promise to make it a breeze!

Happy coding,

Adriaan