Why Java?

There are pros and contras regarding the use of the Java language in our project. The main advantage of the Java language is that it’s platform independent. Any user who has a Java enabled browser that supports Java 1.1.x (such as the Netscape Navigator) will be able to download files using our applet. Another advantage is the tight integration with the browser (unlike a standalone program) and the simplicity of the implementation (unlike a browser plug-in, which also must be customized for every browser). Implementing the distribution server in Java allows running it on different platforms and makes a future web-integration a simple thing to do.

The main disadvantage of the Java language is its nature of being an interpreted language, which slows down both the distribution server but mainly the applet. Not all browsers support just-in time compiling (although most of them do). This is especially important to us since the whole idea is to gain download speed at the expense of CPU time. The other drawback is that browsers impose security restrictions and there is no single standard for the applet leaving the “sandbox”.

In the common computing environment the browsers do support JIT, and digitally signing the applet works with both Netscape and Explorer. Therefore we chose the Java programming language for the implementation of our project.  This required us to convert the code from Luigi[1] from C to Java.

The FEC Driver API

This API provides an application, the ability to encode and decode data using the Forward Error Correction (FEC) Code implementation based on the Vandermounde matrices.

The theory behind this algorithm can be found in [1].

Requirements Description

                      ·          Encoding a full chunk (1MB).

                      ·          Encoding a partial chunk (less than 1MB).

                      ·          Decoding a full chunk (1MB).

                      ·          Decoding a partial chunk (less than 1MB).

                      ·          Providing various parameters about the implementation.

Design Description

In order to satisfy these requirements we used an implementation from Luigi[1] and converted his code to the Java language.

We have implemented the following classes:

                      ·          FEC_Math:

This class includes various math functions necessary for handling matrix algebra over prime fields, which is necessary to speed up the work of the FEC core routines.

                      ·          FEC:

This class contains the core FEC encoding/decoding routines. An object of this class can be created based on the following parameters: the number of packets in the strip (k) and the number of encoded packets (which in our case is a multiple of k).

                      ·          FEC_Driver:

This class implements the required API (the encode and the decode routines), and provides our implementation constants – the packet size, the strips size and the chunk size. It also provides an interface to asynchronously query the FEC driver about its progress on encoding and decoding operations.

The Distribution server Application

The Distribution server application is a tool that the content creator uses to encode and distribute his files to the mirror sites. The current implementation is a standalone Java application, but it can be easily extended to a web based interface (i.e, a Java applet) for personal use.

Requirements Description

The program will read the configuration file, the content file, encode the content file and distribute the encoded content to the servers as specified in the configuration file. The program will produce and HTML download page which will point to the locations of the encoded content mirror sites

The Distribution program will be controlled by a Graphical User Interface and provide a progress bar and a status line in order to inform the user about the progress of the encoding and distribution. Various debug data can be sent to the console/standard error device.

If the content is split into multiple files, each file must be encoded separately.

                         ·          Graphical User Interface (GUI)

Must provide a way to specify the configuration and the content file, and inform the user about the progress.

                         ·          Encoding procedure

Actually encodes the file using the FEC API and formats the encoded data to our special data format as described in section 1.

                         ·          Uploading procedure
Will upload the prepared chunks onto the predefined group of servers using FTP.

                         ·          HTML download page

Producing the html file which will be used to download the file by the client. The HTML page will contain the Download manager applet and the download parameters specific to this file. The HTML will be uploaded to ALL the mirror sites.

                         ·          File names convention.
The file names of the encoded files are based on the original file name (including the extension). The original file name is added the number of chunk and the number of the server it resides on. Both the chunk numbers and the server numbers start at 1. Then the result is appended the “.fec” extension. The mirror site should associate a “application/fec-download” mime-type with the “.fec” extension.
For example the file third chunk of star_wars.mov which resides on the fifth server will be named star_wars.mov_c2_s5.fec

Note: in our test cases we have used the “.zip” extension, since not all public servers allow uploading files with the “.fec” extension.

                         ·          The Server Configuration file format

In order to upload the encoded data to the servers, we need a file that will hold a specific configuration about the mirror sites.

Due to an algorithm limitation this group can’t be greater than eight servers.

The file will consist of several lines, each line gives information about a specific server.

The formats of each line is as follows (entries are separated by semicolons and may include spaces):

Host;Username;Password;path;URL

Where:

                                               § Host: The host name of the FTP server. The port is assumed to be the standard 21 port.

                                               § Username: the user name to access the server.

                                               § Password: the password associated with the user name. (Note: since the password is stored non-encrypted in the configuration file, the file should have read permission only for authorized users).

                                               § Path: The path to the location of the files on the FTP server. Note that both HTML  and the encoded files will be stored in the same directory. We suggest using a separate directory for separate content files.

                                               § URL: The URL of the HTML page on the server. All the references inside the HTML page are absolute.

A typical line will look like this:

      132.68.7.4;guest;secret;/temp;cm220s5.cs.technion.ac.il:8080/work

Design Description

                      ·          FEC API

Using the API in order to encode the chunks.

                      ·          Parser 

Parsing the server configuration file.

                      ·          ParserHtml

Preparing the html file, which will consist of the download manager applet.

                      ·          FTP

A class which gets a specific server configuration and uploads a chunk to this specific server using FTP. The class extends the Thread class and can be run in parallel with other tasks.

                      ·          GuiFrame

The main frame of the application. This frame provides two file dialog boxes enabling the user to choose the file to distribute, and the server configuration file.

It also provides a button that actually starts the procedure and a progress bar and a status line informing the user about the progress of the procedure.

When the “start” button is pressed the a thread that will do all the encoding and uploading procedure is started.

                      ·          EncodeFile

This classs (run as a thread) is responsible for the distribution process.

It gets the file to encode and the server configuration file from the main frame (GuiFrame).

It first extracts the server information (Parser) from the server configuration file.

When start is pressed in the GuiFrame the thread starts to process the source file chunk after chunk.
After encoding (using the FEC API), it opens threads (FTP) to handle the uploading procedure to the servers.

After finishing encoding and uploading, it prepares an html file (ParserHtml) and uploads it to all the servers.

While in action it informs the GuiFrame about the progress of the entire procedure.

The download manager

The download manger is the main part of the project – it downloads the file in parallel from the mirror sites.

Requirements Description

Enabling the applet to work outside the “sand box” in order to get access to the user’s hard drive and in order to open network connections to hosts other then the host the HTML file came from. See Appendix 1 for  more details on Java security.

                      ·          Downloading procedure

                                               §  Trying to download the file in parallel from a list of servers.

                                               §  While decoding a chunk the procedure should continue to download the next chunk in order to minimize the downloading time.

                                               §  It should have the ability to try to reconnect to a server which the connection with it was lost and try to resume the download from the exact point at which the connection was lost.

                                               §  In order to optimize the downloading time the procedure should setup the connections to the next chunk while downloading the current chunk.

                                               §  The procedure should have the ability to resume downloading the entire file in case the applet had been stopped.

                      ·          Getting the servers information

Extracting the locations of the mirror sites from HTML file.

                      ·          Decoding procedure

Decoding a given chunk using the FEC API.

                      ·          Graphical User Interface (GUI)

The GUI opens a file dialog, after the Applet had been loaded to the user’s computer.

The user will then choose the filename and the path on the local host where the file should be downloaded.

While downloading the file the GUI should give the user information about the progress of the download, the information should include the following information:

                                               § The progress of the entire download

                                               § The total time the download took

                                               § Estimate the time the download should take

                                               § Gives the current downloading rates in kilobytes per seconds(Kbps)

                                               § It should inform the user about the connected mirror sites and the speed of the connection to those mirror sites.

Design Description

                      ·           FEC API

The API is used in the applet in order to decode complete or partial chunks.

                      ·           Packet                                            

This class represents data packets as described in Section 1. We use packets of 1KB.

                      ·          PacketReader

This abstract class provides the ability to get packet after packet from a specific server.

It provides the functions interface to connect/disconnect from a given server and resuming the download from a server that the connection with it was lost.
Classes that extend this class can implement the interface using variety of protocols (HTTP, FTP and UDP connections).
A static function is required to parse a URL and return the appropriate packet-reader for it.

                      ·           PacketReaderTCP

This class extends the abstract PacketReader class, but currently supports only the HTTP protocol that we use in our project.

                      ·           DownloadJob

This thread handles the downloading procedure from a single server.

It tries to download packet after packet of a given chunk and informs the download manager every time a new packet arrives.

If the connection to the server was lost it retries up to three times to reconnect to the server and resume the download. If the download from the servers fail, other download jobs will continue to run.

The manager can stop the download job at any time.

If the job handles the next chunk to be downloaded while the current chunk is still being decoded or finishes downloading, the packets that arrived will be buffered for later use.

                      ·           ChunkManager

This class is responsible for downloading and decoding a single chunk.

It opens connections in parallel to the mirror sites and synchronizes the download of the current chunk from these sites.

This class is also responsible for calculating the correct indices for the strip number and the packet index in the strip.

The class stops the connections in two situations. First, the slow servers are stopped when we have received almost all the packets (when we have received at least 30 packets from each strip), to reduce the number of unnecessary packet on the network. However if more packets are available from these slow servers they are used. The fast job (there is only one fast job left when the slow jobs are stopped) is stopped at the moment there are enough packets to complete the decoding of the chunk.

When the slow jobs are stopped the ChunkManager informs the Download Manager, which in turn starts the set-up process for the next chunk.

After receiving the entire chunk by the chunk manager, the download manger starts the decoding of the chunk. ChunkManager decodes the data and appends it to the previous chunks on the disk.

                      ·           DownloadManager

This thread manages the entire download procedure from the mirror sites.

It actually calls the ChunkManager to handle the download of one chunk at a time.

Notice that the next chunk to be downloaded is prepared by this thread though the current ChunkManager is the responsible for detecting this state and starting the next ChunkManager.

                      ·           FEC_App

This class starts and initializes the applet and the whole download process

                      ·          MainWindow

The thread is responsible for all the requirements of the GUI as described above. We need this thread since the FEC_App class is executed in the context of browser internal threads and performing complex operations such as opening dialog boxes would lock the browser.
The thread also reads the server’s information from the HTML file containing the Applet.
The thread is also responsible of enabling the applet to work outside the “sand box” by determining the type of the browser (i.e. Netscape, Explorer) and asking for permissions. It redirects the user to install our certificate for the Netscape browser if its request for permissions fails. Netscape does not allow unsigned applets to request additional permissions (unless codebase principals are enables, which is available only for developers).