Faster Upload and the journey to get it

When we started building our Inspection Cloud software I wanted to use external providers for things like Storage and Upload as I figured they would be the experts and therefore also the best, but boy was I wrong.

The issue

Handling massive amounts of data through a regular web upload became a real issue due to upload speed and slow storage. 
And here I'm not talking about issues with internet speed (we have a dedicated gigabit in the office) but problems related to sending data to slow servers or to the US before being handled.
Uploading few images was OK, so during development things looked great for a very long time.
Close to the first public release, we started Quality Assurance and stress testing which is more extensive than simple unit tests, and it became clear that we had some issues we didn't expect.

A huge issue besides the upload speed was the very frequent access to data in the process of data analysis, photogrammetry, neural network, computer vision etc. 
Using the regular S3 providers we would exhaust the API limits in an instant working with just a single 500 image inspection.
Our error handling would catch the issue and queue the file operations again, but queue would just build up to an infinite amount of unprocessed inspections and eventually breaking the limitations of Amazon's terms of use.

The result was a bottleneck in your inspection flow due to upload speed and with just a few simultaneous users systems were not able to get enough resources from Amazon S3.

Our solution

So, we decided to build our own systems from scratch to handle Terabytes of data in the right regions, without the need for our customers to think about storage providers, API limitations and similar.
The first part of the solution was to build our own Cloud Storage solution which should be fast enough to handle upload at full speed, with the security you expect and without the limitations of normal Cloud Storage known from S3 or similar.

The second part of the solution was to build our own uploader capable of sending data to our backend servers, the ability to include the uploader into any microservice and embed it directly into our customers existing systems.

The result turned out to be an extremely fast system with a 1200% faster upload speed compared to the old solution.