Developer's diary - Traffic Global for X-Plane 11

by Jim Keir (Developer)
Traffic Global for X-Plane 11 - Developer's Diary

Hello! Welcome to the dev diary for Traffic Global for X-Plane – TGXP to friends. I’m going to be talking about how this came about, what it does, what it doesn’t do, why, and quite a lot of what goes into making it work. There’s going to be a certain amount of techy background detail – hey, it’s a dev diary – but I want this to be open to anyone with a little interest in what goes into making your sim a nicer place to be.

January / February / March
/ April / May / June


Dev Diary for Traffic Global XP – July

Welcome back, and congratulations on your persistence if you’ve been here from the start! This month’s diary needs to take a bit of a step back which, as will become clear, is very appropriate. I’d like to talk about something that happened during the internal beta, and I need to start by talking about my car.

Some years ago I bought myself a sports car – after long years of saving money, and a used one at that – but nevertheless, I got myself a Lotus. I’d wanted one for years, in fact from before I even had a driver’s license, without really knowing why. I was having it serviced last year when somebody else at the garage started talking about it and then immediately asked me what I did for a living. I said I was a software developer specialising in large databases and his eyes lit up, which is… unusual. Normally at that point people glaze over faster than New York in “The Day After Tomorrow”. Adding that I’m working on flight simulators rarely improves things (present company excepted). I pointed this out to him.

He said he’d found that every time he came across a Lotus owner – the place I take it to does quite a few of them – he asked the same question and almost inevitably, they were programmers, sysadmins, or something similar. I have absolutely no idea if this is actually true, but it’s an interesting question: if it were true, why might it be so?

Some time later I decided it was simplicity. If you’ve ever read Robert Pirsig’s book Zen and the Art of Motorcycle Maintenance you’ll know it as “Quality”, capital Q. My little Elise certainly doesn’t win any prizes on the pub bragging scoreboard; it’s not got a high top speed, nor is it the fastest off the line, doesn’t have massive power or a huge engine, it’s not luxurious, it creaks like a galleon under sail and yet… and yet, if you changed anything, it would be for the worse. It’s simple, and it does what it’s intended to do as well as it can be done, with the least possible fuss, and that right there is something that would appeal to software engineers.

Lotus’ design principle is simple. “Simplify, then add lightness”. In other words, don’t lose sight of what you’re actually trying to achieve. They still work to the principles of what the older generation would know as “good old-fashioned British engineering”, which means that everything’s done with physics – OK, occasionally string and physics - simply, elegantly and effectively; there’s no electronic stability control, active suspension, launch control or any of that stuff. Not because they were cutting costs, not because they couldn’t, but because the basic design means it doesn’t need them. Aeronautical examples might be the Spitfire, the Vulcan, the Mosquito – things which for some unexplainable reason just feel right in an undefinable way, which comes straight back to Robert Pirsig’s “Quality”.


Very simple, but very effective. Design done right.

Watch a modern jet at an airshow and it somehow feels different; it’s not elegantly working with nature, it’s using sheer power to bludgeon nature into submission. More capable in every way, absolutely, and yet just not right. Old fart mode on: That principle seems to be something that’s being increasingly lost. I was advertised at the other day about an internet-connected smart doorbell with video and AI facial recognition so you could tell who was there without having to rip your eyes away from the advertising on your phone. Well, back in my day, laddie, you used your knuckles on the door and it didn’t need batteries, wifi, rebooting, firmware updates, security patches, a constant cloud connection, the crowdfunded startup manufacturer to remain in business for the lifetime of the product and presumably a security guard to stop someone unscrewing the ferociously expensive thing and legging it. Knuckles just worked. Old fart mode off.

A couple of weeks ago I solved a bug which had been, well, bugging me for some time. The cause was a subtle arithmetic problem, a mix-up of two almost but not entirely identical values. I’d been battling this one for weeks and getting increasingly focussed on the detail rather than the overview. I’d fix one problem which would then cause another, repeat ad infinitum. More and more “special cases”, often an indicator of a deeper design problem, were being added. Eventually I did realise what the cause of the self-replicating sequence of problems was, so I took a mental step backwards to get back to the overview and… Wow. How did all that crap get in there?

So, for the last few weeks I’ve been going back through things correcting the initial misunderstanding and ripping out all of the special cases, exceptions and corrections that were layered on top of it. Adding Lightness. What’s left is simple, elegant and handles any data I throw at it – with a couple of exceptions, which are caused by one of the few remaining special cases and which are likely to work better without it. Once I complete applying the same basic fix to the rest of the navigation code I’m pretty confident that about half of what remains after that can also come out because it will then be near-identical to the half I’ve already fixed.

And this is why I was banging on about my car. It’s the Lotus design philosophy in action: if the physical design is right, you don’t need to “fix it in software”, eh, Boeing? In my case of course the main product is software but the principle stands. As soon as I’d seen my stupid mistake and corrected it in one place it somehow undefinably felt right – Pirsig’s “Quality” again – and the removal of all the added complexity just fell out the process organically. All of the stress I was feeling about not being able to solve what I knew should have been a fairly straightforward problem, damnit, fell out the process organically too.

Which just leaves the relatively minor stress about getting the details right! Interleaved with applying the fixes to the navigation systems, I’ve been fixing lots of annoyances, adding some small nice-to-haves and generally polishing. Maybe sandblasting in some cases. Both the number and the scale of the remaining open bugs are reducing. The sound system is in and working. The one thing that still needs substantial work is not having airliners taxying through each other, which apparently causes a certain amount of consternation even in virtual form.

I sat for some time a few days ago in a simulated control tower just watching airliners taxi around a busy airport – and I was happy. Partly relief, granted, but at this stage even I think I’m being weird. I’ve also been standing in the garden a lot, staring up at Easyjet flights passing overhead, trying to work out if I’ve got the sound attenuation right. I hesitate to think what the neighbours think I’m up to. Right now though, it’s round about dawn on a beautiful summer morning and I think I might just take the car for a walk to reward it for its insight.


Traffic Global for X-Plane 11

Not quite time to ride into the sunset yet


Dev Diary for Traffic Global XP – June

Welcome back! So far in these diaries I’ve been trying to talk a bit about what the development of something like this feels like from the “other side”; not just a “what I did on my summer holidays”-type essay in multiple parts, but what the thought processes and drives are for someone who produces the digital download that, with any luck, improves your simulator experience. Well, this month’s is a doozy.

The big reveal. The unveiling. The public’s first look. I’ll show you mine if you show me yours, and please don’t laugh. It’s a nerve-wracking moment when the dark hours of the night are filled with doubts and if-onlys. I’m sure there are people out there who don’t go through this, but that’s not me. I can see the flaws and, because I’ve been working on polishing them out for so long, it gets difficult to see anything but the flaws and sometimes that’s quite hard to keep in mind.

The beta team, bless ‘em, have been using Traffic Global for a few weeks now and are doing really good work in pointing out problems. And…

Actually, most of what they’re pointing out are things that I knew needed fixed. On the one hand, being bombarded with faults is a downer, of course it is, but at the same time it’s unavoidable, absolutely necessary, and these people are genuinely doing their level best to help out. They’re here to point at problems and the fact that they leaped on – pretty much, anyway – the same list of things that I’d been beating myself up about in the early hours is both an indication that they’re paying attention, and that my estimation of what wasn’t good enough yet was reasonably accurate.

Although there’s a good number of things to correct yet, I’m genuinely pretty pleased with how it’s gone, given how complex the code is behind the scenes. It’s certainly not “production ready” as it stands, but the majority of the problems that are being reported are down to two or three core issues that are just manifesting themselves in interesting and imaginative ways.

A few are simple things like out-of-date liveries or, in one case, an Aer Lingus A320 has managed to get blended with a British Airways paintjob at the same time. Aer Airways? British Lingus? Probably best not to Google that last one at work, but either way it’s actually a simple fix that just needs a bit of time and effort.


Traffic Global for X-Plane 11
Night textures in action

Some are more interesting. Somebody reported a hard crash to desktop, but only at a specific airport. That one at least was easy to find! Load it up, start a flight at that airport, boom. Odd though, the only thing that might cause something like that would be some kind of airport-specific problem, which means it would have to be something to do with the information in the apt.dat file (the ones that X-Plane uses to define airport layouts), but all of that’s really carefully checked already, so what’s up? That one turned out to be a simple arithmetic error in a deep-down bit of code that is used by every single airport and which had been written months before and in use for far too many hours a day, most days, ever since. The nature of the mistake meant that it worked entirely correctly, and entirely by mistake, almost but not quite everywhere in the world.

The most awkward problems to fix are the ones that I can’t see for myself. Somebody reports a problem, gives plenty of evidence that it’s really happening, but for whatever reason it just can’t be reproduced reliably. Or at all. Sure, I can apply some thought and work out what it might be, but in an ideal world I’d be able to see the problem, make a change, and see it go away. It’s frustrating from the developer’s point of view; you know there’s something wrong, people are looking to you to fix it, and the best you can do goes something like this:

“There’s a problem with scheduling, X airport, Y time, Z flight. Just sits there.”
“OK…” >tinker, tinker< “Hmm. Can’t see it but this should do the trick”.
“Nope.”
“Damn. OK, lemme try… Ah, of course…” >tinker, tinker< “Right, got it this time”.
“Nope.”
“What? Oh. Uh… still works for me, but it’s wrong for you, which means it kind of must be… Oh, no, I was hoping to avoid that… OK, gimme a few days.” >tinker, tinker, rip out major system and re-write< “Try this”.
“Nope.”
“Gaah!”

Which – mercifully not with Traffic Global, at least not yet – is sometimes followed with something like:

“Oh, hold on, I had static aircraft switched on, sorry.”

Traffic Global for X-Plane 11
Another night-time picture because it's hard to screenshot sound!

In short, then, despite the list of outstanding bugs I think we’re in a pretty good place. There’s still a few things, important things, that need to be written but they shouldn’t cause too many headaches. There are one or two mind-bending problems to get finally nailed down, a bunch of fairly minor tweaks and some nice, simple, no-brainer busywork things to finish up with. Like I said last month, sometimes it’s good to just have a few low-hanging fruit left that you can reach for when all the others are beginning to feel untouchable.

Next month’s diary might just, if we’re really, really lucky, be the last. (Note to self: Ha!) Tune in, same time, same place, to find out.


Dev Diary for Traffic Global XP – May

Once again, welcome! This has been a busy month to say the least. Firstly, the beta was announced last month which puts a certain amount of pressure on. More than that, it’s been a time of gathering threads together and trying to join them together without using too many granny knots.

As I’ve discussed in previous diaries, there are quite a few major underlying systems that needed to be written to make this all come together. All of them are now pretty much in place, which means that they need to start co-operating with each other. Naturally that’s been the goal all along and they’ve been written with that in mind, but “the best laid plans of mice and men” etc. (If you’ve never read Rabbie Burns’ “To A Mouse”, do, and follow up with “To a Louse”, which might have been written with Instagram in mind if it wasn’t a couple of hundred years old). Things that worked perfectly with a small number of test cases fall apart when they’re faced with the much more complicated scenarios of the entire traffic database, different aircraft, airports, runways in use and so on, and what seems like a very minor change then messes up everything else that’s already been connected to it. And so on.

An old geek joke says that there’s 10 kinds of people in the world; those who understand binary and those who don’t. There’s a similar way of looking a programmers, that I’d describe as creators versus builders. Builders will mostly take already-written sections of code, either as reusable components or copied and pasted from the web, and connect them together to quickly get something that works, while creators are the ones who originally write those reusable components and put the code snippets online for others to read. It’s like the difference between a French cabinet maker and someone building a flat-pack bookshelf.

There’s absolutely nothing wrong with the “builder” way of doing things! If your end goal is a quick, cheap, sturdy bookshelf then this is the way to go, but – and here’s the point – no matter how good their Ikea-fu is, a builder doesn’t have the same understanding of the materials and construction methods as the creator who actually designed it in the first place. There’s no point in reinventing the wheel, but if you need a wheel that behaves differently then re-using an existing one doesn’t help.

Which leads, finally, to the main point for this month: The Rabbit-Hole of Detail.


Traffic Global for X-Plane 11
That 777 pilot must be an Audi driver! Separation is one of the final bits that needs written

Programmer creator-types tend to be detail-obsessed. It’s pretty much a given, there’s no way to do something like that without being detail-obsessed. Even the simplest task needs to be broken down into ever smaller chunks until even a dumb machine can’t possibly need any more detail, and a modern PC’s capacity for detail is stunning. This is simply catnip for coders though; a never-ending cascade of detail, any and all of which can be refined and improved ad infinitum, is a powerful and dangerous draw.

Right now there are still a couple of important things that haven’t been started and which absolutely must be done. If your mind’s not in the right place then the bigger tasks can be a bit overwhelming, so it’s easy to fall into the trap of doing something small as a bit of a warm-up, a nice little no-brainer to do when things get a bit much. With never-ending detail, though…

A quick example. At the start of the month I was looking at making a significant change to the way aircraft moved, which would probably break a lot of connected things. So, naturally, I started subconsciously casting around for distractions and found one – the way that airliners move on final approach. I happened to be watching a 737 approaching on a windy day and it was perfectly smooth, like it was running on rails. Hmm… people are going to notice that. “OK, I can fix that as a quickie”, and did, so now there was a nice bit of side-to-side drift and roll. On a still day it looked too much though, so maybe I should also take the wind gust speed into account? “OK, I can fix that as a quickie”. Done. The next plane I checked happened to be an A380, and it looked plain wrong because it was moving around too much in general. “OK, I can fix that as a quickie” by making the drift behave differently depending on the mass of the aircraft. But wait! How about on takeoff? “OK, I can fix that…”. No, wait, there would be more fuel on board on takeoff so the drift would be less! “OK, I…”. But what if the wind’s straight down the runway, it wouldn’t be rolling so much? “OK, …”. And so went a couple of days until the voice at the back of my head gave me One Of Those Looks and I started on the bigger task that I’d been avoiding.

There’s been quite a lot of that this month.

On the plus side, I’ve got a load of little side-tasks done! I won’t list them until they’ve been tested by more people and are definitely going to be in the final released version but a lot of the “nice-to-have” tasks have been done or mostly done. I know I do this kind of side-tracking and plan for it, and leave a load of little distractions like this around specifically to give me something constructive to do when the real task at hand is just getting a bit too much to deal with. Go do something else, get the little dopamine hit of ticking a task off, build up a bit of morale and go back to the dreaded whatever-it-is with more energy and new ideas.

Traffic Global for X-Plane 11
Monitoring display with Heathrow, parked and moving planes, active taxiways, and routes

As long as you’re aware of what’s going on and remain in control of it, these distractions can be turned into a very useful tool! It’s the old adage “a little bit of what you fancy does you good” in action. Just like Alice’s original trip down the rabbit-hole to Wonderland after her encounter with a frankly suspicious bottle labelled “Drink Me”, giving in to the dopamine hits is dangerous because they can just lead ever downward. The short-term “hit” can easily become the main goal, in just the same way that Facebook and friends trained an entire generation to react to the sound of a bell and drop what they’re doing to get the little “hit” of getting a “like”. Pavlov would be proud, and hopefully a little scared.

And the end result of a bit of self-administered mental coaxing? Taxying is done, moving around complex airports is done, transitions between flight stages are much improved, there are more animations, the groundwork is laid for aircraft avoiding each other, multiple runways can be in use at a time, some model bugs are fixed. Plus a bunch of other bonus features. Yay for procrastination, carefully applied!

The main drive now is to get an internal beta out to people that have been patiently waiting for some time, and get them actually hands-on, albeit minus a couple of important things. By the time next month’s diary comes around I expect to have had some good feedback from the first people to see the traffic in action, and learned a whole lot more about how jet engines sound! Thanks for reading, and check in again; this month’s seen some interesting course corrections but we’re turning this thing onto final approach.


Dev Diary for Traffic Global XP – April

Welcome back! Last month I made a promise – no more talk about traffic schedules – and you’ll be pleased to hear I plan to keep that promise, not least because it makes my head spin. This month I’ll dig a little deeper behind the scenes; we’ve already talked about the models, and the schedules, so next we need to tie the two together and track which planes are where, and a lot more besides. Oh, and (drama pause): Framerates.

Remember, X-Plane doesn’t offer any of the basic services you might hope for when it comes to adding AI aircraft. If you want a model to appear in the world, you need to tell X-Plane where to put it, every single frame. On top of that comes animation, sound, awareness of other traffic and the player, and route-finding, so this is a fairly significant amount of work to do, and it needs to be done for every plane, for every frame the simulator displays.


The traffic database I’m working with right now, which is smaller than it will be on release because bits of last month’s dev diary still make me start twitching, has about 15,000 active aircraft flying about 70,000 flights a day. If we were to go with a brute-force approach it would mean that every detail for each of those 15,000 planes would need to be calculated somewhere north of 30 times per second. “Every detail” means just that – how far have each engine’s turbine blades rotated? Exact positions for each set of landing gear, control surfaces, lights, thrust reversers, and so. There’s an obvious problem here - “Frames per second” just turned into “Seconds per frame”. Mercy me, what to do? Run and hide isn’t an option.

Optimising code basically falls into two camps. “Do Less Work”, and “Do Work Faster”. The approach for the first of these is pretty obvious, we need to cut down the number of aircraft that we’re looking at every frame. Happily, that’s quite straightforward. All we need to do is work out roughly where a plane is and, if it’s so far away that there’s just no way it’s relevant, don’t bother doing anything more detailed. That still takes time, though, so it’s not enough.

OK, so what else can we do? How about not calculating all the details if a plane’s not on screen? That’s great, but you still need to work out where it is (and a bit more) to know whether it’s on screen or not, so still more is needed, and this is where it gets a bit fiddly. There are lots of places where we can decide that some bit of information or other really doesn’t matter, but deciding whether it doesn’t matter or not sometimes takes a bit of time to work out itself, so it comes down to lots of testing, balancing and re-testing (as well as a lot of experience and intuition) to know which bits to put effort into. Some are easy though; for example, if we’re in mid-cruise then we probably don’t need to know how far each landing gear wheel has rotated because it’s a) not rolling, and b) retracted – and, yes, X-Plane needs this level of detail provided!

That still leaves a metric bucketload of work to do every frame though, so what else can we do to cut this down? First let me say that there’s a lot more being done to reduce the workload but I’m not giving away all the secrets here! Skipping swiftly past that, we come to the second approach – “Do Work Faster”.

At least that one’s easy. Simply buy a faster CPU. Job done. No? Oh, OK. How about this then: use the CPU you already have, more efficiently.


Traffic Global for X-Plane 11
A typical workspace - 2 4k monitors

Once upon a time, home computers had a simple CPU which could do one thing at a time. The operating system made it look like it was doing more, but really it was just one thing at a time. If you wanted more than that you had to buy a special motherboard that could take two equally special CPUs, buy a special copy of Windows (yes, yes, or any free copy of Linux), and then you could do two things at a time. I actually had one of these back in 2004 or so and it cost the same as a decent second-hand car. Nowadays things are better though, and everybody has multi-core and hyperthreading CPUs which can genuinely do loads of different things at the same time.

(Mini-glossary: A CPU core is basically a unit that runs a thread. Two cores, two threads at the same time, twice as fast. Four cores, four… you get the picture. A thread is the techy term for a sequence of processor instructions that need to be run one after another. It’s different to a “program” because a program can be made up of many different threads. Hyperthreading is a cool trick where a single core can genuinely do more than one thing at a time with reduced efficiency, very roughly equivalent to 1.6 cores. Techy stuff over, you can pull your fingers out your ears now.)

The problem here is that X-Plane, and Prepar3D, and many other CPU-heavy programs are still not really using the CPU to it’s limits. They’re still mostly “single-threaded” – they do one thing at a time even though the CPU can do way more than that. That’s not because the programmers are sloppy or lazy, it’s usually partly because important bits of the code were written before multi-core CPUs were widely available and partly because multi-threaded code is surprisingly difficult to get right, especially if you’re trying to convert older single-threaded code. The problem is that if one core is reading some data at the same time as another is writing it, all sorts of random problems, or even crashes, can start happening completely unpredictably. Thing is, if you make one core stop work to wait for another one to finish, you’re losing the benefit of doing more than one thing at a time. Simply making one stop and wait takes a bit of time to do, even if it doesn’t actually need to wait. Introduce three or four threads all trying to access the same data and organising them can get really hairy.

Just to make things that little bit harder, X-Plane insists that if you’re interacting with it from a plugin, it absolutely must be from the same thread that is responsible for all of X-Plane’s own work so there are very tight limits on what can be done using a different CPU core. Anything that’s done on this special thread will inevitably lower the framerate. Adding and removing aircraft, positioning them, reading the current weather conditions, getting the ground height at a particular point to make sure that planes touch the ground and don’t hover above it or sink through it, interacting with the UI, even certain types of essential maths must always be done in a way that can’t help but make X-Plane a tiny bit slower.

The strategy, then, is to never make X-Plane wait for anything, do as much as possible from other threads – and there’s a surprising amount that can be done in the background, given all the restrictions – and obviously keep the amount of work to a minimum throughout. It doesn’t matter what else is done if the framerate drops too much, because people just won’t hang around long enough to see any of the clever stuff. There’s no way I’m daft enough to promise no drop at all, but I will go so far as to say that even at really busy airports I’m not finding the drop noticeable – simply being at a detailed airport has a far, far bigger impact. Whether or not other people agree remains to be seen!

Traffic Global for X-Plane 11
A busy airport - all these planes are AI


One other major change happened this month: Traffic Global for X-Plane was announced. World plus dog now knows that it’s coming and is expressing some fairly definite opinions on what it should be, and I’m working on one of the other really important areas that simply cannot go wrong – the details of both airborne and ground handling. No pressure, chaps, just keep soldiering on.



Dev Diary for Traffic Global XP – March

Welcome back! After dealing with providing aircraft models and the actual traffic database – at least partly – last month, it’s time to take a step back and look at some of the design choices that were made, and why.

The big question of course is “what are we trying to achieve?”. Well… traffic. In the sim. Speaking as a real-life (if very occasional) pilot who flies near London I’d love nothing more than to remove a great deal of traffic! However, there’s no question that in-sim there are far fewer other aircraft around and if you’re going for “as real as it gets”, that’s a problem.

So, once again, what are we aiming for? You might say “as real as it gets but c’mon guys it’s a simulator”. I was reading a forum thread the other night about a completely different traffic product which had a long sequence of posts along the lines of “… yeah, but my local airport, Hicksville Muni, added three parking spots on the grass last week and does this product that was released 18 months ago use them? Does it heck. What a waste of money.”

Traffic Global for X-Plane 11
Sim Heathrow - not quite the same as the real Heathrow

One of the earliest design choices to be made was how dynamic the traffic would be. In other words, it would be technically possible to use a live traffic feed, or some kind of subscription service with continual, drip-fed updates, to keep things absolutely bang up to date – but that’s a lot of extra work, infrastructure, maintenance, testing, data licensing fees and so on. Forever. There are other downsides too; you need a permanently-on good quality internet connection, not too tricky nowadays but still a consideration. Differences between real-world and simulated airport facilities, which are always going to exist, will be even more obvious.

On the other hand, a static database of flights, like Prepar3D uses and TGXP will use, is simple to update periodically (by the developers) and can be changed at will (by the end users) if they really want to. On the plus side, it’s a single unchanging file, so it’s simple. (This is absolutely not true – see below!). On the downside, it’s static. If an airline changes a schedule in the real world then the file doesn’t magically update to match. Another drawback is that the file format that’s being used – at least right now – is identical to the one used by Prepar3D, to let us re-use the existing files and management tools, and this format doesn’t allow for start and end dates. That means that if a particular flight runs only from March to September, there’s no way of dealing with that.

That’s a shame, and I’d love to get this feature in. It might – that’s might, as in maybe, maybe not – make it in yet. I’m in control of the traffic database format so it would be very easy to add. It also means more complex preparation of the original data, writing a new and more complex traffic database compiler, and no longer being able to share either the compiled database or the front-end software with the Prepar3D version therefore having to completely re-write that. That’s all time that could be spent making planes do things like fly round corners, or remember to lower their landing gear on approach.

So will you be able to stand in the observation gallery of a real-world airport with X-Plane running on your laptop and watch both a physical aircraft and it’s simulated partner taking off at the same time? Actually, yes, in some cases, but – and here’s the thing – that’s not the point. Will the traffic in-sim be as busy on a typical Tuesday as it would for the real airport? Should be, more or less. Depends. Will it feel different on Sundays? Again, should be. We’re aiming to make a given airport feel as busy as it ought to, on a given day of the week and time of day, where possible using real-world flight data from the recent past. If you’re planning on being able to launch X-Plane to check if your wife’s flight landed on time and should you start getting some food ready for when she gets in, there’s probably more appropriate tools out there!

It looks like this diary is turning into “The Ugly Truth About Traffic”, so let’s keep going and then never have to touch it again!

One of the other comments in that forum thread went along the lines of: “Surely you just download some commercial data, reformat it the way the simulator wants, and ship it? A few hours’ work at most. How hard can it be?” Well, that’s genuinely a very good question, so let’s find out. (Spoiler warning: you might want to get a stiff drink at this point!)

First up, the real world usually differs from the simulated one. More specifically, airports can come and go, or change their four-letter ICAO identification codes, meaning that the downloaded flight data might not match the world that the simulator knows about. Even more specifically than that, parking availability is often different. Very different. If a real-world airport can handle 50 planes at once while it’s simulated version can only handle 40, that’s potentially 10 flights at any given moment that can’t happen, and that could be hundreds over the course of a single day. Those flights would all at some point fly to and from other airports, so even if those airports can handle the number of arriving planes, they still won’t be arriving.

It’s not just how much parking that’s the issue here, it’s also the type of parking. Different types of aircraft need different grades of parking slot, so if the simulated airport has the full 50 slots but they’re graded too small, that’s another difference that needs to be worked out. Usually simulated airports are short of parking compared to their real-world equivalents. Many airlines also pay for reserved parking at given airports, excluding all other airlines, so this also needs to be checked. Some parking slots are so close to others that only one of them can be used at a time; sometimes they’re called something like “257L” and “257R”, but sometimes not so it’s not a reliable way of checking.

What else needs to match? Well, the next most obvious thing is the aircraft. Each commercial flight record will have an aircraft type, so we need to make sure that that type is available in the simulator, with the correct livery for the right airline. That means it needs to be cross-referenced with a list of every airline in the world and every commercially-used aircraft in the world, and each combination of these cross-referenced with the types that are provided with the traffic package, to see which routes we can use. We might also want to look for alternatives; if there’s no specific livery for a Garuda B737-8, what’s the nearest equivalent that we do have? Oh, and remember to check that the replacement aircraft type can fit into the same parking slot size! And what if we don’t have anything for Garuda? What’s the most appropriate airline to use instead? Or should we just use unpainted aircraft? Or maybe drop the flights?

Getting a little trickier now, right? Take a deep breath and read on!

Next up are schedule changes. The commercial flight database will cover a period of time, and during that period, flights might change. So, you could end up with flight AB1234 leaving an airport at 10:05 on a Tuesday and 10:15 on the same Tuesday if you’re not paying close attention to dates. It might also change routes, or aircraft types, or day of the week, or any combination of these, so we need to look out for these changes and decide which version of the flight to use. “All of them, just in case” isn’t really a useful option because that takes up extra parking slots, which we already know are in high demand.

Finally, we have the small problem that the commercial flight database is a list of flights, while what we need are routes. Prepar3D and TGXP both work on the principle that planes shouldn’t ever just pop into existence at a gate and then pop out of existence again after landing. It makes sense if you think about it; how many airlines buy a new plane, load it with passengers and fly it to the destination, then scrap it? If you have a plane that flies from A to B, at some point you want it back at A again ready to run the same flight the next day. This means that the simulator expects to be given a list of routes, where a plane leaves airport A, goes some other places, and ends up back at A again. It also wants to be told how often the route starts, ranging from two hours up to eight weeks. And don’t forget that the plane needs a parking slot for all the time it’s not in the air after the route finishes, while it’s waiting for it to start again.

Traffic Global for X-Plane 11
This one plane is the end result of a lot of fiddling

So, where are we now? We have a list of around 200,000 flights. Some are almost-but-not-quite duplicates, so these need to be found and we need to decide which version to use. Some fly to airports that might have different IDs, or just not exist at all, in the simulator. The flight times, which are all in local time, need to be converted to UTC time which means getting accurate timezone and daylight savings data for every airport that’s used. Some flights will use aircraft, airlines or combinations of them that don’t exist in the simulator and need to be mapped onto something else. We need to work out which of these flights represent a single aircraft landing at multiple airports before it reaches it’s final destination, and split these multi-hop flights down into individual stages. Most importantly, we need to work out where each plane goes after it has landed, guessing at which flight out of the destination airport represents that plane getting back to it’s starting point. It might be going via somewhere else - or several somewhere elses - first, and there might have been more than one version of any stage of the flight. We might need to take several stabs at this if another plane ends up “stranded” with no outgoing flight. Since the sim’s airports and parking are usually different to real ones, some flights won’t fit into the airports when they land, so we need to work out what parking is available in the sim, and of what size, at each one of the sim’s 40-odd thousand airports, and track what planes of what sizes want to park there at all times regardless of whether they’re flying on a two-hourly cycle or a two-monthly one, remembering to check to see if using one parking slot blocks others nearby. Then we need to selectively drop flights that won’t fit, and just to be nice, see if any that used to not fit now do fit, because one of the dropped flights is now not landing at the airport that originally blocked the other blocked flight.

I genuinely don’t know if code-share flights are in there too. I haven’t dared look. And then there’s the question of each end-user having add-on airports with different parking availability.

So, to return to the original question: Can preparing the traffic database possibly be any more complicated than hitting “Save As…” in Excel? Well, yes. Yes, it can.

Next month, you’ll be relieved to hear, I’ll probably be covering something far, far simpler – how to get all these flights into the sim without slowing it down.

Traffic Global for X-Plane 11


Dev Diary for Traffic Global XP – February

Welcome back! Last month if you recall, we found out that X-Plane makes it rather harder to add AI traffic than Prepar3D and briefly talked about the two different areas that will need to be covered – providing the routes and models, and getting the sim to use them.

For one of those two areas I’m lucky. There’s already another package well under way which provides both a traffic database and hundreds of aircraft models, Traffic Global for Prepar3D. There’s a catch though, in that this is for Prepar3D (and FSX and FSW). Those three simulators are very closely related and can to a large extent share data. X-Plane is a completely different beast though, so is any of this any use after all?

Happily, yes, it is! Let’s take the traffic data first. A flight is a flight is a flight. It leaves somewhere then flies to somewhere else at a given time on a given day. Well, unless you’re flying with… no, let’s not go there! Precisely how that information is recorded really doesn’t matter and since X-Plane doesn’t have a flight database format of its own, why not just have it read the Prepar3D data directly?

Yup, you read that right, X-Plane will be reading a Prepar3D BGL file.

Traffic Global for X-Plane 11
The P3D Traffic Global routes

Anyone at the back of the room spluttering about cross-contamination, think about it for a second. Why invent a totally new file to do something that’s already done perfectly well elsewhere? Let’s say I did invent a new file format. I’d still need to write a converter or compiler to go from the base flight schedule data to this new format, so it makes sense to just keep using the existing tools and file formats. In fact there are a couple of catches, namely macOS and Linux, but I’ll get back to those later. It helps that I already know the formats from years of writing other tools to read them, and more recently working with traffic and AI on Flight Sim World, but even without that it would still save a lot of time. Okay, okay, and I admit I just like the idea of making X-Plane read BGLs!

What about the models though? It’s not going to look great having hundreds of stock B737s lined up at the gates. X-Plane does, naturally, already have a well-defined file format for models and, equally naturally, it’s totally different to Prepar3D’s format. Option one is to rebuild them from scratch. Not really the most efficient approach, even if it’s just going back to the 3D modelling software and re-exporting. In reality it’s more involved than that anyway.

Another option would be to buy the rights to pre-existing models and then do thousands of repaints, but this would still take a huge amount of time as well as being expensive. Again, certainly possible, but not ideal. Using pre-existing flyable airliners is also not ideal because they tend to be much, much more complex models than you want for computer-controlled planes that might exist in their hundreds.

So how about converting the existing, specially-created low detail Prepar3D ones? A lengthy sniff around the web doesn’t come up with a converter tool. There’s one for taking an X-Plane model and converting it for Prepar3D, but not the other way round. Annoying – but if it’s possible one way then it’s usually possible the other way too. So, that’s what’s been done! X-Plane will be loading a Prepar3D traffic database and converted Prepar3D models.

Traffic Global for X-Plane 11
X-Plane 11 screenshots, but Prepar3D models - every single one

After defeating some slightly bizarre application of maths in certain animations, it’s working nicely. All of the Prepar3D models have been converted and animate correctly in X-Plane. There’s a few rough edges yet but they’ll get smoothed off between now and release. More importantly, as improvements are made to Traffic Global for Prepar3D, those exact same improvements will be picked up for X-Plane with minimal effort; no re-working, no duplication of effort, just a batch conversion.

When I say “just” a batch conversion, I’m glossing over the whole subject of actually writing the converter! This was several weeks’ work in itself and, even so, still needs a few details to be completed. Reading the Prepar3D models is fine, I’d already got code to do that as part of some of my older tools. Writing out a basic, non-animated model that X-Plane’s “Object Previewer” could read was fairly simple too, Laminar Research are generally very good with documenting things. The tricky bit was getting the animations, effects and lighting right.

Remember, these are external models only, we’re not trying to map a load of cockpit switches automatically, but there are still a lot of moving parts that expect to receive Prepar3D data. Take something simple like rudder deflection. Prepar3D models might be set up to receive this based on a value between -1 and 1 for full left to full right, or -100 and 100, or an actual angle which might be in either radians, grads or degrees, while X-Plane might only provide rudder deflection as a percentage. Prepar3D even supports basic arithmetic and logic in animations, which X-Plane doesn’t, and all of these options need to be handled for dozens of moving parts across hundreds of models.

The goal, of course, is to have all this complexity made invisible in the finished package. If you’ve just bought a traffic add-on and you need to spend the first three hours downloading models from here, paintjobs from there, customising look-up tables, manually copying files and editing configs and running conversions and so on, it’s going to be a bit of a let-down, right? That’s why, as a developer, it’s nice to write about the bits that are usually hidden. The idea is to get everything working smoothly, to make it appear effortless, but the fun and satisfaction of actually doing the programming in the first place is in getting all the conflicting, complex parts to mesh. It’s like the old image of the swan, serenely gliding along on the surface and totally hiding the furious paddling going on underneath.

If you were to look in X-Plane right now, you might think there’s not much progress to see. You’d be right in a way; there has been progress, but very little of it is visible. It’s often the way in larger projects though; there are a lot of separate things that need to be put in place, each of which depends on something that isn’t quite done yet before it can start doing its thing. You might not be able to see much – the swan’s not moving and the feet are just beginning to twitch - but the foundations are being laid.

Traffic Global for X-Plane 11
EasyJet PeasyJet


Dev Diary for Traffic Global XP – January

So where did this all come from? Seems like a good place to start. For the last couple of years I’d been doing contract work for Dovetail Games on Flight Sim World, mainly on the missions side – they licensed my FSX Mission Editor and commissioned a load of updates both to that and the sim’s mission system – but eventually also in lots of other parts of the sim. Sadly Dovetail stopped work on that in early 2018 as I’m sure you know, and I was looking out for something else to fill the gap. Enter the nice people at Just Flight who said they had something in mind. Yep, Traffic Global for X-Plane.

Traffic Global for X-Plane 11
Where is everybody?

As it happens, one of the areas I’d worked on for Dovetail FSW was traffic, adding a lot more GA flights to smaller airfields, so I had a good idea how that side of things worked. Short version: a list of flights comes from a file, matching planes come from other files, these in turn are matched against airports and parking availability and as if by magic, you get planes flying planned routes in the sim.

Of course, anyone who’s seen The Wizard of Oz knows that “magic” depends on somebody in the background doing a damn fine impression of a drug-crazed tapdancing octopus trying to get all the levers to be pulled at the right time.

For a traffic add-on there are in fact two tapdancing octopi: one preparing all the data, and another using it. Just preparing the data – making sure that all the flights work properly, pairing with a simulated world that doesn’t match the real one, providing all the necessary models – is an enormous task. The other octopus, at least for Prepar3D/FSX/FSW (I’m going to use these interchangeably), is provided by the simulator. Feed it the right data, it’ll fly the planes around.

Not so for X-Plane.

It used to be the case that X-Plane would move planes around for you to some extent if you controlled their autopilot settings. That’s fine for going from A to B, but what about landing? Taxying to parking? Circuit patterns? As it happens, none of that was relevant because this whole facility was removed some time ago.

Righto.

Can I just feed it some kind of traffic database, like P3D uses? Nope. Oh, and X-Plane only supports a maximum of 20 non-player planes too, even if the autopilot trick still worked, which it doesn’t. If this sounds like I’m beating up X-Plane, bear with me, I’m not! Those 20 planes that can be controlled from outside the sim are really meant for multiplayer rather than AI traffic, so in fact both the small number and the removal of the autopilot kind of makes sense. It does leave you with only one option though: you can have any number of planes added to the sim using a different approach if you control them yourself. Entirely yourself. 60 times a second, you need to tell X-Plane where your planes are, how they’re oriented and animated. For potentially hundreds of planes in the player’s local area from tens of thousands around the world.

Traffic Global for X-Plane 11
Ah - that's better!

Okay then. This is going to take some planning.