Architecture

Information about DEX8 architecture and how the platform works

Terminology

Here is a list of basic terms used on the DEX8 platform.
  • file - a text file containing JavaScript code, with an extension ending in .js
  • task - the robot's task, consisting of one or many files, the most basic being a main.js file.
  • robot - this carries out a task or tasks, stores extracted data to a database and sends it to the Web Panel in real-time for monitoring.
  • start task - the Web Panel button which, when clicked, starts the robot(s) and the task it is carrying out.
  • stop task - the Web Panel button which, when clicked, stops the robot and the task which it is carrying out.
  • pause task - the Web Panel button which, when clicked, pauses the robot and the task it is carrying out.
  • shop - the Web shop where DEX8 user can buy premade DEX8 tasks and platform resources.
  • storage - Usually the NoSQL database and its tables (collections), where extracted data is stored.
  • exporter - the section of the Web Panel that exports data from NoSQL DB to another data format, for example CSV.
  • scheduler - the section in the Web Panel for scheduling tasks. It uses cron jobs to start the robot(s) periodically for particular tasks.

Platform Components

At first glance, the DEX8 platform can appear complex.
However, when dismantled, it has only a few basic components:
DEX8 Files Files
These are JavaScript files written by using NodeJS npm modules such as Puppeteer, Moment.js, Bluebird ...etc.
DEX8 Task Task
The task refers to what the robot needs to do, following instructions from the file(s).
DEX8 Robot Robot
A robot is an engine which carries out the task(s). It is a NodeJS instance deployed on a Linux server.
DEX8 Web Panel Panel
The Web Panel is a simple application which the user opens in their browser; the GUI (graphical user interface) for writing code files, defining tasks and controlling robots. By using the panel, the user can start, stop or pause one or multiple robots. Here, the user can also schedule cron jobs in the Scheduler section or export extracted data to CSV files. Learn more ...
DEX8 API API
An API is an HTTP server which sends JSON responses. It connects the Web Panel and the robot(s), and stores the extracted data into a database, therefore it can be described as the ‘heart’ of the system.
DEX8 Database Database
The database is the NoSQL database, which is used to store extracted data. The robot(s) sends extracted data via API to the database in real-time.
DEX8 Scheduler Exporter
The exporter can be used to export data from the database into any other format, for example CSV.
DEX8 Scheduler Scheduler
The scheduler is the part of the API that users can use to schedule their tasks. A task can be started, paused or stopped at an exact time.

How does the DEX8 platform work ?

When a user clicks on the START button on the Web Panel, a task - or several tasks - are sent to one or several robots via API. DEX8 how 1

Robots send HTTP requests to web pages and collect data. DEX8 how 2

In return, the extracted data is sent to API in real-time.
The data is stored to Database and echo messages are sent to the Web Panel where the user can monitor the extracted data in real-time. DEX8 how 3