How Flight Works: The Technical Perspective
How Flight transfers data to or from cloud object storage
Signiant Flight performs simple point-in-time mirroring, or one-way file synchronization. It can execute both uploads to the cloud (on-premises storage is the source, cloud object storage is the destination) and downloads from the cloud (cloud object storage is the source, on-premises storage is the destination).
The basic sync paradigm is that when a new file appears at the source, it is copied to the destination. If the file already exists at the destination (same name, size, and modified time/date), it is not transferred. If a different file with the same name exists at the destination, it will be overwritten with the source file.
Operationally, this means that when files are modified at the source, the destination files will be automatically updated when the transfer command is executed. Functionality is included for transferring individual files (which can be renamed upon transfer), multiple files, and an entire directory structure.
To provide maximum performance and flexibility, instances of the Flight server software are running in a number of different locations around the world for both AWS and Microsoft Azure. Maximum speed is achieved by ensuring that the Signiant cloud servers are located as close as possible to the customer’s cloud storage tenancy.
It is important to note that transfer speed is optimized by using the Signiant SaaS Transfer Server instances closest to the cloud side of the transfer, not the on-premises side. The Signiant protocol will address any latency issues between on-premises storage and the cloud data center, so there’s no need to minimize that network distance. The goal is to configure the Flight system so that data lands on a Signiant server in the same region as the cloud storage tenancy, making the final multi-stream HTTPS hop as short as possible.
The Flight utility
The Flight utility provides IT professionals with the ability to choose which cloud transfer server is used, allowing for trade-offs in system design. The domain name associated with the target Flight server (actually a set of load-balanced auto-scaling servers) is specified in the transfer command.
A list of server domains is available on the Signiant Support website, and for optimal performance the server domain in the same region as the target cloud storage should be specified — for example US West for Amazon. Additional domain names with less geographic specificity can be defined for use in the event that the primary servers aren’t available.
For example, if the second server domain is set to US Amazon, servers elsewhere in the US will be used if US West is unavailable. There is also the option of using a global domain name, which via DNS will resolve using geo- location to the set of currently available servers closest to the client side of the transfer — thereby providing redundancy at the expense of transfer speed.
The ability to choose an optimal Flight server, combined with the fact that Signiant technology removes the problem of distance-related latency, provides customers with significant flexibility in the design of global cloud systems. For example, a customer in Latin America might choose to buy cloud storage located in the U.S. where it is much less expensive — and Flight will ensure that upload performance doesn’t suffer. An American company might want to locate sensitive European customer data in a cloud data center that is within the EU, and again Flight will ensure that access times are not negatively impacted by this strategy.
The functionality noted above is all executed via CLI commands on the client side. Customers can have as many instances of client-side software deployed around the world as they wish, and each client can be separately configured for transfer details and server location. For a central view of their Flight deployment, customers have access to a web-based console as illustrated below. This console allows for configuration of system-wide elements such as storage accounts and keys, and provides detailed insight into utilization.
Flight’s System Architecture
The core Signiant file movement protocol can transfer large data sets between the storage on the customer’s network and the cloud up to 200x faster than mechanisms such as FTP or other TCP-based approaches. But in addition to acceleration, Flight also provides other cloud-specific features to increase the speed of data movement into and out of cloud object storage.
In order to achieve end-to-end accelerated transfers into cloud object storage, the data must be moved in two pipelined stages.
It is first transferred via the Signiant acceleration protocol from the on-premises location to a server in the cloud data center, and once in the cloud it is split apart and written to object storage via multiple parallel HTTPS streams. This two-stage transfer process must be carefully orchestrated in order to achieve maximum speed.
A full Flight deployment is shown in the system architecture diagram below.
The part of the architecture where the actual movement of the data occurs, via the pipelined mechanism as described above, can be referred to as the ‘data plane’ or ‘data layer’. In the diagram below, the green section and the lower blue section constitute the data plane, where the data moves from the customer’s on-premises storage into cloud object storage.
Signiant file transfer software is required at both ends to handle the data movement. Flight client-side software is installed by the customer on a server at the on-premises end of the transfer. This client software is CLI-controlled, and can be used either stand-alone or in conjunction with pre-built Upload and Download workflow components for Signiant’s Manager+Agents.
At the cloud end of the transfer, in the center blue section of the diagram above, no customer involvement is needed — there’s no software to deploy or VMs to procure and manage. A key part of Flight’s differentiated value is the fact that the cloud compute resource and server-side software in the data plane are delivered by Signiant as a SaaS offering.
Flight customers are provided with access to Signiant’s patented multi-tenant, cloud-native SaaS, which provides the data transfer in a highly available, load balanced, auto-scaling manner. At periods of peak utilization, more servers will automatically spin up to accommodate the extra load, spinning down again when traffic decreases. Throughput will never be reduced due to high customer traffic, and there is no extra charge for an increase in the number of transfer servers managing the data movement.
The blue section at the top of the diagram represents the last part of the Flight architecture, the functionality that orchestrates the entire system and can be referred to as the ‘control plane’ or ‘control layer’. This, too, is a multi-tenant, cloud-native SaaS operated by Signiant on behalf of their customers. Via the Internet connections shown by the blue lines, this part of the system serves the web interface for the management console, coordinates the transfers themselves, and collects data for billing and tracking purposes.
Summary
In summary, a Flight deployment is a complex, distributed system — but most of the complexity is abstracted away from the customer by Signiant. Both the control plane SaaS and the data plane SaaS for Flight are fully-managed solutions where all the backend technical tasks such as load balancing, redundancy and failover, server management, and software upgrades are administered by Signiant.
