This project aims to create a Twitter-like engine and web user interface which concurrently handles at least 10,000 user requests to display functionalities like register, subscribe, tweet, re-tweet, and search queries based on hashtags and profiles.
The goal of this project is to construct an engine similar to Twitter, complete with a web interface. This engine will allow users to register, post tweets, and conduct searches. Alongside this, a client simulator will be created to replicate the behavior of thousands of users and evaluate the performance of the distributed engine. The project is divided into two milestones.
In this phase, we will focus on building the backbone of the Twitter-like engine. We will develop a simulator engine that mimics the activities of various users. It will simulate typical user actions such as registering an account, subscribing to other users, posting tweets, and retweeting posts from other users.
In this phase, we will incorporate WebSockets to furnish a web interface.
The application will have the following features:
- Register
- Send tweet (tweets can include hashtags, e.g., #topic, and mentions, e.g., @bestuser)
- Subscribe to user tweets
- Re-tweets
- Query tweets by the subscribed user
- Query tweets by Hashtag
- Query tweets by Mentions
- Display the new tweets by subscribed users live (without querying)
Fig 1.: User Registration
Fig 2.: User Dashboard Fig 3.: Hashtag Search not exisitng
Fig 4.: Subscribe
Fig 5.:Successfull Hashtag Search
Fig 6.: Subdcribed Tweets
Fig 7.:New Tweet Fig 8.: Re-tweet Fig 9.:Search Mentions
Fig 10.:Search Subscribers
The project is implemented using the Erlang programming language.
brew install erlangDownload files
- Extract the contents of the zip file.
- Move to the relevant directory using the command cd and execute “make run”.
- Run the following in terminal
cd Twitter
make run
- Navigate to http://localhost:8082 in the browser.
TwitterClone is a system modeled after Twitter's features, utilizing the Actor model approach. In this setup, each user profile is seen as a separate actor, known as the client. A central engine, acting as a server, manages the user profile names and their related process IDs. To effectively simulate Twitter's operations, a separate simulator engine has been created. This simulator imitates regular Twitter actions and activates backend functions of the client actors to emulate various activities. This tool is vital for gauging performance and expanding the Twitter engine by creating a vast user base quickly. Finally, this backend system is connected to a web interface, which employs websockets for message exchanges between the individual actors.
The actor model is a programming approach designed for creating concurrent and distributed applications. Within this model, an actor is the core computational unit, holding its state, behavior, and communication methods. They can send messages to other actors asynchronously, process incoming messages, and adjust their internal state accordingly. Crucially, actors don't have direct access to each other's data or state; they only communicate through messages.
1. Server Engine:
Account Management: One of its primary roles is to keep track of registered user accounts. This ensures that every user has a unique identity within the system.
Tweet Management: The server engine maintains records of all tweets. This includes:
Hashtag Management: Using a map data structure, it tracks all tweets associated with a particular hashtag. The key in this map is the hashtag, while the value is a list of tweets containing that hashtag, along with the username of the tweeter.
Mention Management: Similarly, it manages mentions. A map is maintained where the key represents a particular mention, and the value is a list of tweets containing that mention, paired with the respective tweeter’s usernames.
Search Capability: The server processes search queries from clients. Users can search for specific hashtags, mentions, or even general keywords in their feed.
2. Client:
Tweeting: Clients can create and broadcast tweets to their followers or the general public.
Retweeting: They can also share or retweet other users' tweets.
Search: Clients have the capability to initiate search requests. They can look for tweets based on hashtags, mentions, or even specific user's tweets.
Displaying Feed: Clients can display their feed, showcasing tweets from accounts they follow.
3. Simulator:
User Generation: The simulator has the capability to create 'N' number of users, simulating a real-world scenario of multiple users joining the platform.
Activity Simulation: The simulator emulates real-world user actions. This includes subscribing to other users, tweeting, retweeting, and performing searches. This is essential for testing the system's robustness and scalability.
4. Handler:
Communication Facilitator: For each client actor, a corresponding handler actor process is spawned. This handler is responsible for managing the communication between the frontend (what the user interacts with) and the backend (where data is processed and stored). This design ensures that client interactions are managed efficiently and without delay, providing a seamless user experience.
In essence, each actor/process has a distinct role, ensuring the system operates seamlessly. The Server Engine is the backbone, storing essential data, the Client represents the user, the Simulator helps in testing and scaling, and the Handler ensures smooth communication between the system's front and back ends.
Websockets:The client typically sends an HTTP request to the server to initiate a WebSocket connection. The WebSocket handshake is then completed by the server by sending a response with an HTTP header to start the WebSocket connection. The WebSocket protocol messages can be sent asynchronously between the client and server once the connection has been made. We use a straightforward Erlang web server named cowboy to control the socket and the WebSocket protocol to interface the Erlang runtime system with WebSockets.
Cowboy:
Cowboy is the name of the Erlang framework that is most frequently used to build web servers. It's pretty amazing to see what Cowboy accomplishes with such little code. Cowboy works in combination with Ranch (which is a socket worker pool for tcp protocols) and Cowlib (library for message processing). The process tree begins once your server has been established because Cowboy is meant to construct servers. The only process active in the standalone Cowboy app is cowboy clock. A typical method for launching a cowboy http server is as follows.
cowboy:start_http(http, 10, [{port, 80}], [{env, [{dispatch, Dispatch}]}]).
• First, let's create the directory for our application all in lowercase.
$ mkdir hello_erlang
$ cd hello_erlang
• Install erlang.mk:
curl -O https://erlang.mk/erlang.mk
• Bootstrap the application :
make -f erlang.mk bootstrap bootstrap-rel
• Run the application :
make run
• Now, add cowboy to the existing dependencies(in Makefile)
• Add Routing and listening in the _app.erl
• Run the application : make run
For more references, check out the documentation
A websocket interface for a twitter like engine is successfully implemented with an interactive user interface through which users can perform functionalities like
• Tweet
• Register
• Subscribe
• Retweet
• Query tweets by mention, hashtag & by subscribed users A websocket connection is established after registering and redirected to /name/main. Once a user tweeted, all the subscribed users will instantly get in the feed through websockets without any user interaction.