I think it’s fair to say the trial of the new airline platform has been somewhat problematic. While rich in features and possibilities, the resource requirements are high, and have so far caused not only a migration to new servers, but also the host to flag the hosting account as “experiencing a distributed denial of service attack” (it didn’t, but we’ll get to that).
What caused the problem?
At times yesterday we had over 50 aircraft in the air, all posting live ACARS data to the website at once - far more data than Transmitter ever communicated or processed. The volume and consistency of traffic, allied with the CPU resources required to process those requests resulted in the CPU usage on the web hosts going over 100% for several hours. The trend was steadily up throughout the day as more people signed in and the number of concurrent flights increased.
The hosting platform resources are automatically scaled - allowing for 400% CPU capacity to handle bursts of activity. We were operating in “burst mode” from mid-morning all the way through to mid-afternoon.
The consistent demand on the server resulted in automatic protection measures kicking in (that normally defend against distributed denial of service attacks) - disabling the hosting account for 24 hours. This brought down the main website, the airline, and transmitter.
What can we do going forwards?
We could migrate from managed hosting solution to our own server. This will not be cheap, and will not be straightforward. I will hold my hands up and say now - I don’t have the time to install or look after it - that’s why we’re using a managed service in the first place.
To help alleviate the immediate problems and get at least some services back online, I have switched the domain name servers over to Cloudflare, and implemented their attack defences. I’m currently waiting for the hosting partner to review the results - and hopefully unlock the hosting account.
A question to answer
The primary question to answer is - having seen the scale we would have to operate at to run a complex service for so many people, and the technical challenges required- do we want to do it?
I’m not sure we have an answer for that right now.
In the meantime…
In the meantime, va.virtualflight.online is back online, and the codex is back up. It’s been a continual firefight over the last 18 hours, not helped by a horrific cold (thanks go to my daughters for that one).
Thank you for your continued patience.
Jonathan
Good luck with this. I mean it, not being sarcastic. You have agreat passion for our sim flying and I admire that. However it is not worth your personal mental health and its effect on your family. I would not like to see you leave the sim community even if this venture fails. I appreciate what you do.
Total Newbie to the sight, saw your recent YouTube video and decided to join. Your passion in incredible. While probably not the most popular choice have you thought about monetizing the service and sight? You certainly have the popularity and depend.