Big Data, Big Challenges: A Guide for Handling Large Geospatial Data
Why moving data with Dropbox, FTP and shipping hard drives is no longer viable
Introduction
Most disruptive innovations are driven by the conditions of the time. Born at the intersection of change and frustration, new solutions emerge when industries outgrow existing tools, leaving businesses struggling to remain efficient and competitive. We are in such a time for companies that utilize large geographic data. Sharing large point cloud volumes generated from 3D laser scanning poses a particularly difficult challenge — especially when working in remote locations with unreliable or insufficient internet connectivity.
Legacy file transfer methods like FTP and shipping physical hard drives, as well as consumer file-sharing solutions like Dropbox, are no longer viable options. Just as most businesses today can’t operate without the Internet, AEC companies, visual capture businesses, drone services, and surveying groups realize an essential need for large file transfer solutions — advanced technology that makes it easy to transfer any size file with speed and security over standard IP networks, regardless of destination or internet quality.
To overcome the challenges of the day, it’s important to understand the market drivers that have created the need, explore the available options for moving reality capture files and large datasets, and identify key requirements for choosing large file transfer software.
Market Drivers
Several market drivers are behind the growing demand for efficient and reliable file transfer software. The following three elements have come from different aspects of the industry, but together, they create something of a perfect storm for driving innovation.
1. Surge in High-resolutions Data Generation
The widespread adoption of 3D laser scanning technologies has led to the generation of vast amounts of high-resolution spatial data. These large point cloud datasets, often consisting of thousands or even millions of files require efficient transfer methods. As demand for precise and detailed geospatial data increases, so does the volume of data — underscoring the critical need for effective large file transfer solutions.
2. Remote Project Locations with Limited Connectivity
Projects requiring scanning often occur in remote areas with limited or slow internet infrastructure, rendering most file transfer methods inefficient. Poor network conditions and inefficient transfers increase the risk of transmission errors and file corruption, further hindering data accessibility and reliability.
3. Advancements in BIM & Integrated Technologies
Integrating field data into modeling platforms requires smooth, reliable data exchange across partners, locations, and networks. Efficient file transfer software solutions ensure data integrity and compatibility for cohesive project management, fostering seamless collaboration among stakeholders, including clients, subcontractors, and government agencies. This enables transparency, accountability, and trust among all parties.
File Movement Options
Various methods exist for transferring scan data today, ranging from outdated technology like FTP to more commercial cloud-based solutions like Dropbox. Here’s a breakdown of commonly used options and their risks and benefits.
1. File Transfer Protocol (FTP)
One of the original internet protocols, FTP, was once standard for transferring files and is technically “free” to use. At its core, FTP’s design isn’t adequate for handling today’s large and complex files.
- Difficult to Use: FTP tools are notoriously frustrating and time-consuming.
- Slow: FTP lacks file acceleration technology, making transfers significantly slower compared to modern solutions. And without the ability to automatically resume interrupted transfers, users are forced to manually restart FTP transfers from the beginning.
- Bandwidth-immune: FTP doesn’t utilize available bandwidth efficiently, making it impractical for files over 500MB.
- IT-managed: Deploying and managing FTP requires significant IT resources, causing delays.
- Not Secure: FTP poses known security risks that are frequently exploited.
2. Shipping Hard Drives
The old standby. When FTP can’t handle the job, many people turn to shipping hard drives or even having employees transport them. Hard drives can hold an enormous amount of data, and if you have a very simple operation without time constraints, they can work well. However, scaled to the size of most modern businesses, they are impractical for daily operations.
- Expensive: Hard drive costs can accumulate if they aren’t returned and re-used.
- Difficult to Track: It’s nearly impossible to control who has access to your data on a shipped hard drive. Should you really trust a courier service with your client’s data?
- Delays: Shipping delays caused by weather disruptions, flight cancellations, or lost packages can create unpredictable setbacks that disrupt critical workflows and timelines.
3. Standard Online File-sharing Tools
Although file-sharing tools like Dropbox, Box, and Google Drive are widely used for many appropriate applications, they weren’t developed with professional large file workflows in mind.
- No Acceleration: Like FTP, these solutions rely on basic IP technology and lack acceleration technology.
- Controlled Storage & Lack of Security: These solutions don’t integrate with the range of storage types most companies rely on, including on-premises and cloud systems. Additionally, they often fail to meet IT department security standards because cloud storage is usually bundled with the software. This setup leaves you with little visibility into where your files are stored or who might have access to them.
- File Size Limits: Without acceleration and because storage is bundled, these solutions often have file size restrictions making these tools unsuitable for large datasets.
- Lack of Workflow Support: Advanced workflows require file-sharing software that integrates with other systems and provides analytics, metadata handling, and more.
4. Large File Transfer Software
Large file transfer software, such as Signiant’s solutions, is purpose-built to meet the demands of data-intensive industries that move large files with distributed workflows. Built on a core acceleration protocol that optimize public and private IP networks, these solutions excel over greater distances and congested networks. If large file transfer software seems like a necessity for your business, there are several key requirements you should look for.
Key Requirements for Choosing a Solution
1. Speed, Without Limits
When people complain about file transfer, speed is often their top concern. A solution must be fast enough to handle files of any size and travel any distance efficiently. Look for a platform that offers acceleration technology without imposing bandwidth or file size limits, ensuring your projects stay on track and meet tight deadlines.
2. Reliability
Reliability is paramount when selecting a file transfer solution. Speed alone is meaningless if files arrive incomplete, corrupted or not at all. Choose a solution that ensures data integrity and can automatically resume interrupted transfers from the point of failure. This feature reduces disruptions, keeps your workflows running smoothly, and ensures seamless collaboration across teams and partners.
3. Security
Security is critical in protecting your data from breaches and unauthorized access. Select a vendor that prioritizes security with defense-in-depth practices and undergoes regular third-party audits by firms like Independent Security Evaluators. Additionally, look for storage-independent solutions compatible with on-premises and cloud systems. This flexibility ensures you maintain control over your data, allowing you to select the best storage options for your business.
4. Reporting
Having insight into your file transfer activities is invaluable for optimizing operations. Ensure your solution offers robust reporting capabilities, including notifications and chain-of-custody reports. These capabilities enhance accountability, provide actionable analytics, and help you monitor and improve file transfer efficiency across your organization.
5. Ease of Use
Today, ease of use is a key requirement. A simple, browser-based interface eliminates the need for training and ensures users can quickly transfer files without complications. This ease-of-use increases adoption rates and allows your freelancers, contractors, partners, and any other stakeholders to focus on their work rather than the tools.
6. Ease of Deployment & Management
Like ease of use, your solution should be very simple for IT to deploy and manage, reducing administrative overhead. Purchasing SaaS-based file acceleration software offers the best experience, as it eliminates the need for ongoing software maintenance. Be sure to choose a SaaS vendor with a strong commitment to the services aspect of SaaS, like excellent customer support, ensuring you have assistance when needed and a long-term partner invested in your success.
“Delaying a data upload one day can put the team behind. Media Shuttle has made us stay on time and budget with projects.”
Kaylin Culver | ZELUS
The Signiant Platform for the AEC Industry
The Signiant Platform is an integrated set of SaaS-based products designed to optimize data transport, accessibility, and collaboration, ensuring more streamlined and accelerated workflows. Combining the simplicity of online file sharing with enterprise-class acceleration, security, and control, the Signiant Platform imposes no file size limits and is accessible from any web browser.
Signiant patented architecture enables visual information engineers, data managers, survey technicians, or construction technologists to move data quickly and reliably while maintaining complete control over their valuable data. In plain terms, there is no faster, more reliable, or secure way to transfer large files than with the Signiant Platform.
To learn more about the solution trusted by industry leaders like ZELUS, Fortis Construction, Digital Aerial Solutions, Above Surveying and more, contact us now.
