Creating Twitter Bot Using SerpApi

Motivation

I'm a big fan of football. My favorite football club is Chelsea FC. I find it quite hard to keep track of my favorite team's matches. I already have configured calendar for premier league, but I need a complete reminder for all the competition matches that my team is playing at. That's why I built this Twitter bot: to send a tweet ahead of Chelsea FC's match, for every competition.

Why Twitter

I'm an active Twitter user and nowadays, further information and conversation about football matches can be found easily in Twitter so I can engage more with other football fans from all over the world.

High-level overview

For those of you who are more comfortable reading the code, I just made my code public! Here's the repository that you can take a look at:

Diagram of the app

Generally, the flow of the app is pretty simple and straightforward. From the diagram above, the app flows from left to right (clockwise).

It starts by fetching the match data from SerpApi's Google Sports Results API regularly using cronjob. Then it feeds the data into Redis for a certain amount of time. Then there is another cronjob that runs regularly to fetch data from Redis from the previous process and then publish the event.

The event then will be consumed by the subscriber (a worker) in the background and finally call Twitter's API to send a tweet when certain criteria are met (24 hours and 1 hour before Chelsea's match).

Preparation

Libraries

App configuration

This is an interesting part of this app. It's not like a usual app like a microservice that runs as an HTTP server. The parts of the app consist of the following things:

• match-fetcher.ts -> a cronjob that will fetch the football matches data regularly from SerpApi.
• match-reader.ts -> a cronjob that will fetch the date stored in Redis and publish the event based on certain criteria.
• sub.ts -> a subscriber that will run continuously and listen to a certain topic and eventually call a function to send a tweet.

So when the app starts, it will boot those 3 files in parallel: 2 cronjobs and 1 worker. Since there are 2 cronjobs, one must be concerned about the concurrency policy. But to simplify the case: this app always uses the replace mechanism when it comes to concurrency policy.

It means that if it is time for a new job run and the previous job run hasn't finished yet, the cron job replaces the currently running job run with a new job run. There is no issue so far in this particular part since the app isn't really that time-sensitive so I get the benefit of that simplicity.

1. Serp API
   I use the sport result of SerpApi. This code snippet describes the full method that I use for calling SerpApi:

async get() {
    const { SERPAPI_BASE_URL, SERPAPI_KEY } = process.env;
    try {
      const response = await axios.get(SERPAPI_BASE_URL + "/search", {
        params: {
          api_key: SERPAPI_KEY,
          q: Query.club,
          location: Query.location
        }
      });
      return response.data;
    } catch (e) {
      console.error(e);
    }
  }

2. Twitter API
   For Twitter API, I only use the send tweet API because that's the only thing (so far) that I need: send a tweet so that the followers of this account can get the reminder about the upcoming match of Chelsea FC.

async post(content: Content) {
    const { TWITTER_BASE_URL } = process.env;
    const request = {
      url: TWITTER_BASE_URL + "/2/tweets",
      method: "POST",
      body: content
    };

    const authHeader = Oauth1Helper.getAuthHeaderForRequest(
      request
    ) as unknown as AxiosRequestHeaders;
    try {
      await axios.post(request.url, request.body, { headers: authHeader });
    } catch (e) {
      console.error(e);
    }
  }

Process

Due to the complexity of the codebase structure, I'll only explain essential parts of the code and leave the remaining parts (data manipulation, formatting, functions, enums etc. unexplained).

Match fetcher

The entrypoint of the app starts from the match fetcher. Here's what happening inside that particular file:

async function fetchAndSet(): Promise<void> {
  await Redis.init();

  const existingKeyTTL = await Redis.getTTL(RedisTerms.keyName);
  // only fetch the serp API and set the key if current key is expiring in an hour or less
  if (existingKeyTTL < lowerLimitToFetchAPI) {
    const data = await httpController.get();
    const fixtures = data.sports_results.games;
    const firstMatchDate = data.sports_results.games[0].date.toLocaleLowerCase().trim();
    const customDateFormats = ["tomorrow", "today"];
    let gameHighlight;
    if (data.sports_results.game_spotlight) {
      // handle game highlight and append to the result
      gameHighlight = await convertToStandardSerpAPIResults(
        data.sports_results.game_spotlight,
        true
      );
      fixtures.unshift(gameHighlight);
    } else if (customDateFormats.includes(firstMatchDate)) {
      const firstMatch = fixtures[0];
      fixtures[0] = await convertToStandardSerpAPIResults(firstMatch, false);
    }

    const convertedData = await serpApiToRedis(fixtures);
    await Redis.set(RedisTerms.keyName, JSON.stringify(convertedData), defaultTTLInSeconds);
  }

  await Redis.close();
}

The fetchAndSet function always gets called when the job starts. It first creates a connection to Redis and checks if the key is already set or not. If it is set already and the key TTL is more than a certain duration, it'll skip the remaining process and exists early.

If the key is set but the duration is still lower than the threshold, it'll go through the following process: get the data from SerpApi, then check if there's a game highlight. If there is, it converts the data into a "standardized" format and then put that into the beginning of data.

Otherwise, it'll go to the normal process where the data will be converted into the standard SerpApi result defined in this codebase. Finally, it'll convert the data into Redis interface and set the key-value accordingly.

Match reader

The purpose of this file is to read data from Redis storage and call the publisher to publish a message to certain events that the subscriber will listen to.

async function getMatchesAndPublish(): Promise<void> {
  await Redis.init();
  const matches = JSON.parse(await Redis.get(RedisTerms.keyName));
  const now = new Date();
  const upcomingMatch = new Date(matches[0].date_time);

  const diffInHours = await calculateDateDiffsInHours(now, upcomingMatch);

  console.log(`diffInHours is : ${diffInHours}`);

  if (diffInHours <= Time.hoursInADay) {
    const msg: IBody = {
      message: matches[0],
      hours_to_match: diffInHours
    };
    await publishMessage({
      channel: RedisTerms.topicName,
      message: JSON.stringify(msg)
    });

    // remove the entry from the key if only difference is 1 hour
    if (diffInHours === 1) {
      matches.shift();
      const currentTTL = await Redis.getTTL(RedisTerms.keyName);

      await Redis.set(RedisTerms.keyName, JSON.stringify(matches), currentTTL);
    }
  }
}

Similarly to match-fetcher.ts, it will first initiate a connection to Redis. Then it gets the data and checks if the first element of that data reaches a certain threshold. If it does, then it'll publish the message.

It will then check again if the difference is 1 hour. If it is, it'll remove that particular data and set the key-value to Redis with the latest data so it won't get processed again the next time the job runs.

Subscriber

The main block function of subscriber.ts is this:

async function subscribeMessage(channel: string): Promise<void> {
  try {
    const redisClient = new Redis(REDIS_URL);
    redisClient.subscribe(channel);
    redisClient.on("message", async (channel, message) => {
      const cleansed = JSON.parse(message);
      if (shouldSendReminder(cleansed.hours_to_match)) {
        await sendTweet(cleansed);
      }
    });
  } catch (e) {
    console.log(`an error occured when subscribing message to ${channel} `, e);
  }
}

That function will create a Redis connection and call another function which is shouldSendReminder:

function shouldSendReminder(reminder_time: number): boolean {
  if (remindInNHours.includes(reminder_time)) {
    return true;
  }
  return false;
}

That will determine whether a given event needs to get processed for a tweet purpose. If it is, finally it'll call this function:

async function sendTweet(tweetContent: ITweet): Promise<void> {
  const matchSchedule = new Date(tweetContent.message.date_time);
  const contentToTransform = {
    hours_to_match: tweetContent.hours_to_match,
    stadium: tweetContent.message.stadium,
    participants: tweetContent.message.participants,
    tournament: tweetContent.message.tournament,
    date_time:
      ENVIRONMENT === "production"
        ? await addHours(Time.UTCToLocalTimezone, matchSchedule)
        : matchSchedule
  };
  const transformedTweetContent = await transformToTweetableContent(contentToTransform);
  const tweetMsg = {
    text: transformedTweetContent
  };
  await httpController.post(tweetMsg);
}

The above function will transform the data into tweet-ready format and eventually call twitter's API to send a tweet.

Output

Below is the example of the tweet that is produced by this app:

Previously the tweet was very simple. It had no information about the stadium and competition. But SerpApi team has responded very well to my inquiry to provide such information for both game results and game highlights. So I really appreciate their response in that matter 👍 👍.