My personal trading story, all coincidences are random.
I started my career at HFT in the Australian branch of one of the largest American trading companies as a C ++ programmer. On the first day I was met by an office with huge windows overlooking the Sydney harbor, one of which was written with a felt-tip pen “<2ms”. This was the main task for a dozen developers, but so far, not for me. So...
Initial shock
One of the guys proposed the idea of trading options on the Australian Stock Exchange (ASX), or rather, option spreads and their combinations with mandatory hedging. He needed something that could cope with a bunch of complicated trading rules and be integrated with the trading platform we used, which was called Orc. These were the early two thousandths and I wrote my solution for it on VB6 under Windows 2000. At the same time, I used C ++, Boost and the Spirit multithreaded parser for integration with Orc. The latter counted binomial or trinomial trees for valuing American options on the ASX on demand. For my pricing code, I used someone else's VBA code, rewritten line by line in C ++.
However, Orc did not always calculate the price on demand, he often used caching. If the original parameters remained the same, Orc simply returned the price from memory, instead of recalculating it. This platform was too slow to compete with Timber Hill’s own platform (now Interactive Brokers) or the IMC Ord Liquidator, which was the fastest boxed system at the time. She seemed too miserable to compete with others in performance, no matter what tricks I did with her. However, with a large multidimensional option price cache (missing values were obtained from it by simple interpolation), constantly updated code in several streams, which was written in C ++, we unexpectedly were able to make transactions that were not available before.
As the interest rate or volatility curve changed in Orc Trader, my price cache started to fill up again. Of course, this decision was not original. I first read about option pricing in an old article several years earlier. As in a modern microprocessor with an extraordinary performance, you can guess about the following calculations, which may still need to be done in the future. Market prices are discrete. Remember, the fastest calculations are those that don't need to be done. Next came the idea that it’s hard to beat a zero delay, but not impossible. I made this strange conclusion about predicting the future from the fact that the fastest message is one that should not be sent. In my opinion, this is even more important.
In my opinion, the best architecture is the absence of rigid architecture. This may look a little ridiculous, but is true in the case of HFT. Unstructured code instantly turns into legacy if you do not pay attention to the system architecture. Since, in fact, premature optimization is the basis of everything, it is more profitable not to debunk the well-known computer myths and allow your competitors to stumble on them.
My hack for the Orc Trader platform was not an ideal solution, but he delighted several people with deals that were previously impossible. They even looked happy when they saw these deals in their reports with their own eyes. Overall, however, it was only a minor success. In the future, it turned into hundreds of manual rules for finding deals, paying for its trading only the cost of the operator. But at least I had fun.
Focus on “<2ms”
Let's recall the condition “<2ms” recorded on the window glass. I wrote an article offering a faster and more complex way to achieve even smaller delays. A team of three programmers, including me, had to implement it. The first choice fell on the Korean stock exchange. There was no need for price caching in this project, since the European options pricing model is not much more complicated than simple interpolation, even in several dimensions.
At this time, processors with a frequency of 2-3 GHz only began to appear. Our performance metric was somewhat incorrect - we basically measured the delays inside our code, rather than those that lie inside the operating system (which are actually much larger.) This still worked when data arrived at millisecond intervals, because at that time the network stack spent on processing the package from 50 to 100 microseconds. The main thing that needs to be remembered here is that when a processor can perform more than a billion operations per second, over a million instructions fit into these two milliseconds. Honestly, this is a very, very bad code if this million instructions goes to simple price calculations. If deciding on a deal takes you a million steps, leave immediately. Here every nano second is important.
It turned out that I was attached to a not very well-coordinated team, and as a new employee, I had to behave cautiously. In the development department there were only three of us. One of the team members (who actually was a great programmer) apparently tired and began to spend all his work time on games and forums about video accelerators. “Mr. GPU” used to write key company libraries for multi-threading, and they were not bad, although they suffered from design problems. All synchronization was done through mutexes and any operation required locks. In the end, I came back to using the Boost C ++ library. “Mr. GPU”, apparently, did not enjoy the project and looked like participating in some kind of strike. He was clearly annoyed by the need to work on my idea. The other guy was less capable, but still minimally competent. Usually he ate a lot and constantly went to the meeting room to watch another film on DVD.
But we had to somehow cope in order to overcome the two milliseconds. As it turned out, I needed to write more code than expected - I had to stay up several times late. “Mr. GPU” did the integration work, while “The Detached Dude” did nothing at all. As a result, our new system showed a delay of just under five microseconds. It seemed to me that it was a huge success. However, this figure did not take into account network delays. Definitely, they are now much more important than a month ago.
One of the other employees congratulated me. I learned from him that the main part of IT people (including him) considered this an impossible task. Apparently, they hoped that I would fail. The changes that I brought did not meet with widespread enthusiasm, but I received a small bonus without a significant salary increase.
Previously, I assumed to stay at this job for twenty years, especially after the dot-com bubble. But now the time has obviously come for me to move on.
Search for a new job
I signed with several HFT companies and shared my thoughts on what can be done. One good Dutch trading company listened to me very carefully, but did not offer anything. Years later, I learned that their team began to implement something like what we talked about in an interview the very next week after it. I heard that the approach was successful. They called it the “implied base method”, in which it all came down to predicting the theoretical price of a futures or underlying asset. I like their name for my method, even though it looks like a cargo cult. Well, that's the way financial companies usually do business. Despite the deception on their part, I was flattered.
Then I spoke with an Australian company with a team of about a dozen developers focused on trading on the ASX. Their platform was mostly even slower than Orc Trader, spending many milliseconds on calculations. I asked for a meeting with the head of the company, who was much more successful and much younger than me. He was one of the guys with an extremely high salary, whose income was supposed to be declared according to Australian law.
Their business didn’t look “clean”, and he warned me that the money they earned abroad cannot always be legally easily returned to the parent company. Negotiations with this company were a waste of time — I don’t want to be a black hat. Gray is still where It’s a fun game, as long as it’s ethical. However, we don’t have to cross those red lines. There’s no pleasure in winning by cheating, but more importantly, your reputation matters. People are always underestimate the losses from scandals and I e want to risk.
I also liked one of the investment banks, with a representative of whom I spoke. It was a branch of a large office on Wall Street. I have already managed to work a few years in this kind of environment. The entrance threshold seemed rather low - they didn’t want to make money, they just wanted to create an exchange turnover at no cost, so their finance department could serenade this branch. It seemed to me a good option, and I was sure that they would not object to the creation of turnover, as well as to obtaining additional profit. Profit rarely causes complaints at all.
A good thing about my previous firm was that they could even fire the money-earning employees. Someone broke the rules by making several transactions that were not authorized. They brought money to the company, and I was impressed that they were still fired for breaking the rules. Such a correct position of management is extremely rare, especially in finance.
Then, quite unexpectedly, another possibility arose.
One day I was having lunch with a guy I knew from computer science at the University of Tasmania, who now worked for ITG. He suggested that I speak with ITG, as they were engaged in private trading. For me, this was news - I thought they were just brokers. I met with their CEO for the Asia-Pacific region, and he spoke in detail about their arbitration platform, which works with shares denominated in USD and CAD. Then I met Ray Killian, the US CEO, when he was on a trip to Australia. The meeting went well, and it seems that there were no more barriers to developing high-frequency trading in ITG. The CEO of the Asia-Pacific region plus their chairman, who was also a member of the board of the NYSE-listed company ITG Inc., eventually brought an improvised business plan and risk calculations in the United States to a board meeting.
What happened next, I do not know - I was told three completely different versions over the next few years. According to one of the versions, the council approved the business plan, but Killian played it all backwards. According to another version, some board members did not approve the project. According to the third version, the business plan did not reach the board members at all. I don't know which versions were wrong (maybe all at once), but it didn't matter. The ITG guys decided to do without their employer and invest personal funds. It was not exactly the job I was looking for, but if the business went well, I would be able to earn about a quarter of the shares of the new company through options received over the course of several years in terms of trading. It was like a victory.
The birth of the HFT company
Thus, having received an investment of 1.5 million Australian dollars, it was time to start working alone with a laptop in the office again. I set up several servers and workstations, installed 64-bit SuSE Linux, hired three programmers - and the watch that measures the period paid by investors began to tick. It turned out that I am not a very good Linux system administrator. Soon I was tired of doing bad admin tasks and assigned them to a freelancer. Now we were four and a half people. But it turned out that 64-bit SuSE Linux, especially on AMD processors with their Hypertransport bus, was not ready for prime-time loads, so we returned to the 32-bit RedHat / CentOS build. This was the wise decision of “Mr. Sysadmin”, which propelled us forward a little.
Four months later, we looked at the prospect. We had to choose - run on our ASX or go to Korea. The three Korean exchanges, KSE, KOSDAQ and KOFFEX, then just merged into a single Korean exchange, KRX. We chose Korea, although the business and technical environments there were more complex and risky. However, there were also more opportunities, plus simpler mathematics in pricing models and a small amount of tools promised a more lightweight code base.
Then a funny thing happened. When creating a new system, you usually try to make some naming conventions to make your life easier. I decided to use ISO codes or part of them for standardization. However, due to the merger of the Korean stock exchanges, an ISO code for the new stock exchange has not yet been assigned. I contacted the responsible ISO team, which turned out to be the only guy with an excel table sitting somewhere in Europe. He decided that the new code for the unified exchange would be XKOX. I sank and wrote to him that this combination does not sound very well in English. Perhaps the best idea would be to use obvious KRX or XKRX codes? We laughed and so XKRX became the new ISO code. So now I can say that it was I who created the ISO code for the combined Korean stock exchange.
In those old times in Korea, traders, even high-frequency ones, could not get direct access to the exchange. You had to go through a broker gateway and use their specific connection API. There was no collocation at all. Among clients, there was a big difference in delays, not controlled by a broker, simply because of the different location and technology of interaction. Futures and options traded on the KOFFEX exchange. KOFFEX itself was located in Busan, a port city at a distance of just over three hundred kilometers from Seoul. Derivative contracts for the KOSPI 200 index had a small face value, but a large lot size. Because of this, they were the most resourceful derivatives in the world, with a tenfold margin from the rest. The euro / dollar contract on CME was number two, and options on KOSPI 200 were number one. This turned out to be too much for KOFFEX, so the main trading in these instruments was carried out on their behalf on the KSE Exchange in Seoul. Therefore, everything we needed from the exchange was located at one point in Seoul, although it was founded in Busan itself.
At this point, I already knew that the network stack introduces the most significant delays that I was able to manage in our new implementation. I found a hardware Ethernet protocol converter in InfiniBand, which introduced a delay of 3 microseconds. It was designed as a module that connected to the TopSpin InfiniBand switch. AMD's Hypertransport (HTX) bus, an alternative to PCI Express, was more modern, resulting in less internal latency. I bought several Pathscale HTX Infinipath network cards and motherboards with HTX support (they then existed only in alpha versions, for example, one of them had a serial number 0x0000045!), Assembled all these heterogeneous parts into a large case and got an expensive one, but extremely fast network stack, which I was very pleased with. He looked, of course, somewhat clumsy and clumsy, but he was one of the most competitive in the world. For several years, the usual network cards could not compare with my configuration.
Choosing a broker
After that we went to Seoul to find a broker and a place in the data center. I went with one of the investors, a man named Bubble, who then became our employee. At first, it was torn between ITG and HFT, but later it was completely at our disposal, since both the CEO and the chairman of the APR, as well as other ITG employees, were our investors. He was instructed to book a hotel in Seoul. It turned out that he chose an hourly-priced motel, tempted by a very picturesque artistic description that hid the true picture - a run-down, dirty, peeling air-conditioned room, a computer infected with viruses. Not a good start. Several brokers came to us and expressed surprise about our area. Awkward, but funny. The trip was quite productive, we made an incredibly incredible deal.
Good deal
Muppetz Securities offered us a DMA service. A large firm, from which I quit, did not have direct access. Before that, I was asking about the HP / Compaq / Tandem system at the old job, because it was one of the ways to get into the internal network of a particular broker in order to get closer to the stock exchange. It cost about $ 500,000 for a small box created as part of a special internal HP application development program. Seductive, but not for a business that lives out its last days. The DMA service offered to us now was implemented on the basis of the X.25 protocol, which was innovative for me then. This broker deal was rather expensive, but since real direct access was an unheard of phenomenon, I signed a contract in a few days.
At that time I met “Mr. Ree” from the local company RTS, who was passing through Seoul, sorting out his affairs. We helped each other in finding brokers in Korea. We used the internal information system RTS as a backup repository for our transactions, so that shareholders could compare our data with those of independent suppliers. You might think that the Orc platform would be the logical choice. I was familiar with Orc, and at this stage he had the most convenient user interface. However, the RTS had a much more user-friendly API and cost an extra 10% of the monthly cost.
Back in Sydney, I looked at all this expensive equipment under InfiniBand. Now it was useless - we were going to move to X.25. An unexpected but pleasant turn of events. Then I received a message from the broker Muppetz Securities that it cancels our deal, despite the already signed contract.
Some time later, I learned that Muppetz went to a large office where I worked before. X.25 was a very limited and rare resource. Why give it to some small start-up HFT-firm, which still does not even trade when you can go to a higher level and make a deal with real traders? Smart move, albeit unethical.
It turns out that the expression "no" weakened the brain of some brokers. I believe that many HFT firms thought about how to get direct access. It must be time. However, the cancellation of the Muppetz contract led to other, much better offers. One from Mr. M., a real gentleman and just a fabulous guy with a good team. Its price was much better than Muppetz and, although technologically this connection was a broker access, the location was amazing - the connection point was right in the stock exchange building!
I had no idea that you could trade there. It turns out that this company had a dusty, simple room without air conditioning in the extension to the old KRX building. This old building, in contrast to the new building recently erected on campus, was connected by a corridor to the main tower, where the exchange engine was located on several floors below ground. This room was used as a place for emergency terminals. There was a small table in a narrow room with terminals with old CRT monitors. If the whole world came to an end, you had to send employees there to enter the keys. This room was not adapted for server infrastructure. Will we be interested in this little shit room, which is the best thing they could offer at the moment? I tried to hide my excitement by answering “yes.”
The X.25-based Muppetz solution was technologically faster, but the connection point was in a building outside the KRX campus, which caused an additional delay. Someone was going to pay a big premium for a solution that wasn’t as cool as the one we just came to.
X.25 and technology
The exchange infrastructure platform based on X.25 was interesting. You could count on the T1 or E1 class line (the exchange had both options and used them alternately) from Koscom, the main technology and network provider KRX. Inside the line, virtual channels were allocated at 64 kilobits per second (kbps) according to the time alternation principle. Interestingly, because of this alternation, you essentially have concurrent access to the entire channel at the same time. The broker has a limited number of lines, but many parallel (albeit slow) channels.
I did some testing and first bought X.25 standard network cards from a retailer in the UK. Rather popular, worthy of mention, they seemed to be quite industrial quality. Many years ago, I developed an information system for accounting for car leasing at Citibank, and it used the X.25 protocol. The entertainment that this development delivered to me was only possible because I had a bad youth on the pre-Internet days - BBS, FIDO, Z-modem, Y-modem, that's all. Most of the time at Citibank was devoted to picking with modems and other equipment, just to make it all work together. After that, everything was simple. But not with these network cards. Neither we nor the card supplier could get them to work normally. In the end, between us there was a conversation on a high tone, after which the seller simply stopped picking up the phone. Then we bought a few cheap network adapters from the Canadian company Sangoma, which immediately earned as it should.
In our dusty working room, we put our server rack. I came with manually assembled servers in the bag. Communication lines for us have not yet been allocated. The channel to the exchange will appear a little later, and now we urgently needed market data to prepare and set up our infrastructure. HFT requires a lot, a lot of data. The exchange solved this problem by passing the twisted pair through the air outside the building and leading it through a hinged window to our room, which we had to always keep open. When it rained, the water in the room fell a little less than one would expect. Twisted pair connected to T1 modems. Modems, in turn, are connected to serial ports. X.25 giant connectors, a legacy of the sixties, stick out of our Sangoma boards. Slow bits are big bits.
Soon they connected a line from the exchange to send orders with even more gigantic X.25 counterparts. We had modules for working with Postgresql (soon replaced with versions with flat files), a trading engine for C ++, a service for calculating volatility, etc. Instead of leased lines, as we initially agreed, we used regular Internet channels. We took three suppliers who promised geographic independence and prescribed routes to ensure redundancy. Much cheaper, higher bandwidth and more reliable than leased lines. It was unusual at that time, but many are doing the same thing now.
(Fall) start trading
Nine months after launch, we finished testing. No certification from the exchange was then required. At that time it really was a land of cowboys. We started trading - on options was 85% of executions! This means that either you cannot evaluate options correctly (and anyone can sell you contracts at bad prices), or you are always first in line and you are doing well. We had basically the last.
But we could not make money.
Initially, our plan was to simply buy “juicy” options and hedge their futures. , , . , , . , , . , . . , ( , " "), 72 . , . , . , , , . , . … ! – . , , .
(IV) . IV, ( ) . . , IV , . , . R, . IV . - , . , , . , , ( .) , .
. . , . , , . . , .
, . . . . “” . - , , . , . .
Zero
, , , . ATM , 0.01 . 0.01 . , , . , . , 1.13 . . 10 20 , . , . .
, ., - (, .) . , , ? , – , . X.25-. , . , ?
, 25 000 AUD, - . 25 000 AUD. ., , , Zero. – , , . . . . ?
!
. Zero . , 500 1000 ATM, . -. .
, . -, , , , . , , , , , . , , , , . 100- , 50 , - . Zero. X.25 , , . .
, . . , .
Hunter- 200 , mini-ITX , , ATX . , – .
Zero . , , . , . Zero . . . KRX . . , . , 0.01 3.00. . 6-7% , 10%. , , . . .
, . T1 E1 , , . . Sangoma, T1/E1- NetOptics . , – , . , .
Sangoma , . , . X.25 . . , . HDLC, , KRX. .
. Eurex - API – - . , . , . , ASX Bondi, , 1U Dell, . - Nasdaq , – .
. , ( , , ) , . . slice, . HDLC , bid ask . 100 128 / 6.1 . KRX , . , .
, , . . , , - . , , . . , . , AI , .
, . , . X.25 , . - . . , , . – , . . . Muppetz . – .
. , . GPS- , CDMA. Endrun Technologies , . GPS CDMA NMEA ( ) . CDMA 10 , . , CDMA- 5 6 .
In the end, we settled for colocation in the KRX underground data center. Getting inside was rather difficult due to strict security. After going a long way underground, I noticed that inside the data center I caught a cell phone signal. I put our CDMA receiver in a rack and it worked - we got the exact time from the intermittent cellular signal, until something inside burned out. Suspecting to install a more efficient antenna, before I went to Seoul, I bought a white flexible antenna with a meter length and soldered an SMA connector to it. When I went down to the KRX data center underground bunker and put a big white antenna on our stand, the guys from Koscom looked at me very suspiciously. I don’t know what they thought of themselves there, but several other people over the years mentioned that they saw my white antenna lying at the top of the KRX stand.
Now we had the exact time at all points of presence and we needed to establish data exchange between them. We found that a few blocks from the campus we received a 300-microsecond delay between our points through a ten megabyte Ethernet connection from the city provider. This was a very good indicator and we coordinated all of our market data and orders on these Internet lines. In Korea, the wonderful Internet and then we missed it. Milliseconds the benefits of obtaining market data for all points of presence immediately improved trading. While our best orders came from a stock exchange campus, the fastest market data came from a point a few blocks away. This turned out to be rather unexpected and that location remained the best place to get market data for several years.
A little later, at the new point in the extension to the KRX exchange building, we set up the acquisition of market data and the delays there were huge. This was the first time I ran into Citadel in Korea, I was able to determine this by the time stamps in network packets — they can tell you everything, if you know how to listen to them. We asked to turn off this channel, Koscom stopped the transfer and physically disconnected the line. After a month or two, we paid for reconnecting, they switched on the same line, without changing anything, and now it was one of the best. Such were the vagaries of life in Korea. I suspect that some kind of round-robin balancer somewhere in the middle could improve the situation. For the other oddities observed there, I could not come up with any obvious reason.
Virtual channels for sending orders on X.25 were 64 kbit / s each - this one is extremely slow. I thought that we could use the same hack with Sangoma cards and started writing code that would send a packet with an order even before it was fully formed in memory, which would save us milliseconds. After sending the order, we could turn the package into an invalid one if the market situation suddenly changed and the order sent was no longer needed. However, our small team was already so heavily loaded, and this task was not a priority, because our trading indicators were already quite good. We learned more about modeling implied volatility and refining strategies.
Only changes are permanent.
The exchanges never stand still and the KRX was no exception. They began to abandon the X.25 protocol and switched to Ethernet and TCP / IP. The exchange planned to use TCP to send orders and UDP for market data. The trading efficiency of our X.25 hack was starting to fade. We have become too fat with our trading engine and strategy. The performance of our trading engine has not improved much over the original three hundred microseconds, it swelled. The guys at some stage repeated my 72-hour hackathon and turned the engine into something more traditional, simple and fast. Strangely enough, we were not too worried about a hundred microseconds here or there, for we pursued strategic and tactical goals. Compared to the others, we actually had a negative delay due to the operation of the network adapter in promiscuous mode, bypassing the modem and our own HDLC parser. Our orders still went off before the market data arrived at the rest of the market.
We wanted to try pre-sending orders on X.25, but we didn’t have free people to spend a couple of months programming, especially considering that using X.25 is coming to an end. All X.25 code would be useless a few months after the completion of this project. However, the idea had to be tested, and I talked to the guy who wrote most of the software in Sangoma. He was the main developer and did the main job of writing a device driver. Maybe he can recommend someone? He said that he would do it himself as a third-party project from home. There were some problems, since the code for these drivers was very confusing with an abundance of goto to C transition operators, but even if this is not a very good style, the drivers worked stably. This code should not have lived for more than a few months, so it seemed an acceptable risk. Testing took a little longer than expected, and the pre-shipping worked, but the performance was terrible. The code was so devilish that the “fast” path worked more slowly than the usual path. However, we didn’t have time to rewrite all the code for this idea.
Changes to support TCP on the KRX turned out to be a simple mapping of X.25 virtual channels to TCP sessions. You can have several simultaneously open TCP sessions (which they call the PID) for sending orders. The market data transfer format was unchanged ported to work over UDP. I don’t remember whether they added encryption to the TCP version immediately upon launching or a little later, but now when sending an order, the entire data packet immediately following the TCP header must be encrypted using the ARIA cipher, and this was a requirement . ARIA is the national Korean cipher. It is very similar to AES with a slightly different S-Box layout. This is a good cipher.
We saw that the new TCP-based architecture was going to shake up the market and potentially destroy most of the big brokers business. Good old X.25 was slow, but the interleaving of channels meant that you had many parallel ways to send a warrant that could be divided between each other. In the new world of TCP, there were a limited number of sessions, two or four, and all clients had to send their orders through these connections one by one despite the seemingly many PIDs. As a result of this serialization, client applications queued at the broker. The state of the race arose now not only on the stock exchange, but also inside each broker. Large brokers could lose their market share, as the existing communication lines could only support a limited number of HFT clients. In addition, the speed of sending orders increased, the size of market data packages also increased, and the rate of receipt of market data did not increase proportionally, to keep pace with one another. The stock exchange was confidently moving towards a complete mess and we understood that we must somehow get ahead of this process. It is time to increase the number of our communication channels, which meant even more brokers, in which it was necessary to open accounts.
We went to Seoul to look at the situation and were able to get a few architectural diagrams that the Koscom provider kindly attached to the brokers' stands. There was depicted a backbone with a bandwidth of 622 Mbps, covering the entire city with a ring and having several branches that went to the brokers. In the racks we saw our old friends - modems T1 and E1. This was good news - we could continue to use old tricks to get market data. Our new configuration included several T1 modems that were connected to relatively inexpensive Cisco switches. If we could listen on the stream in the T1 twisted pair after the main switch has decoded the packets from the common optical channel, everything would be fine.
The transition to TCP made a lot of mess for KRX and everything happened exactly as we expected. Large brokers such as Samsung Securities lost their market share because they were deceived by the exchange about their ability to efficiently serve their large customer base. Delays in market data have become commonplace, as the exchange has now accepted orders faster without a proportional increase in the performance of the distribution branch. But we already coped well with this, using time stamps to track delays in market data. Put, call and futures options came through different channels and were usually delayed only one (and sometimes two.) We felt this imbalance of delays and switched to hedging mode. Several times we argued that we could take advantage of the situation at those moments when we knew for certain that others receive highly distorted market data, but did not include these improvements in the work plans. Sometimes channel imbalances led to serious delays - when the data was already tens of minutes behind us - we stopped everything. As soon as the imbalance disappeared, our engine automatically resumed trading.
It was interesting to work with the network in promiscuous-mode and did not violate the rules, because I do not think that their originators knew about this at all. The transition to TCP / IP instead of X.25 required the mandatory use of firewalls, but we learned that some local traders avoided them. This was clearly against the rules and therefore we were not ready to do this. If you break a clearly prescribed rule - you can be completely disconnected from the KRX exchange. Moreover, it did not have an explicitly established procedure in the rules of the reconnection, if you were once expelled, so the decision may be completely final. We calculated that our advantage was much greater than the delay on the firewall, therefore, until these clients listened to everyone and cut the packages, our advantage was out of danger and those guys barked at the wrong tree. I recently learned about another local group that bypasses the firewall in Korea. So it happens, but now it is rare.
Hacking FPGA
We also managed to improve our hack to get market data. We were a small company with several people called Thomas. This is incredible, but the eighteenth hired employee became our sixth Thomas. Part of the blame for this lay on me - I knew about the “Tomas problem” and, for fun, was looking for graduate students with this name in different places. I noticed a new Thomas at UNSW University in their NICTA group. He was preparing to defend a doctoral degree and wrote an interesting paper on distributed shared memory on Linux, including a version for Itanium processors. I once came across this before, so I made an agreement with him, and he began working part-time as soon as he received his doctoral degree. One of the cool things he did was hack into the Sangoma network card firmware that we used. These cards were made on the basis of a small FPGA chip from XILINX, namely on the Spartan 3 model. He got access to the flash memory in the FPGA and found some registers that we could manage. Soon we had our own working firmware for it, and we were able to completely circumvent the firmware Sangoma with its promiscuous mode. Sangoma has become the main equipment supplier for us. The discovery that the Sangoma network stack cost us almost a millisecond was a shock for us. The delays of our homemade network stack were only about ten microseconds.
As we planned to enter other markets, we began to explore the FPGA more closely. Once I almost started using FPGA instead of Ethernet to InfiniBand converter from TopSpin. After digging it for some time, I turned to an FPGA company called Celoxica back in 2004 to find out about their technology. When we first started our HFT project, I asked Celoxica to think about a solution for receiving data via UDP. Trading was for them then a new direction, but they conducted preliminary research and returned. Celoxica had a piece of code for decoding UDP via Ethernet, which they used in an embedded car entertainment system in Germany and could be converted to our task. We studied it a little, but this option was too expensive. Years later, I talked to one of the big Wall Street Bank CTOs, and he told me that he started talking to Celoxica about trading at about the same time as me. Therefore, Celoxica began developing in this direction and subsequently it became an important source of income for them. Their superstructure over the VHDL, which they called the Handel-C, was rather beautiful, but unfortunately it was not able to fully comply with the technical task. We eventually did dive into the development under the FPGA, which included the study of another language, Impulse-C. It was pretty cool, but we went back to VHDL, because our design was better suited to its model.
Our new firmware SANGOMA worked perfectly for us. However, just as Einstein imagined himself running alongside a light beam, we wanted to get into the first bit, an electron or a photon, arriving at the broker. This means that we must be connected to a common fiber optic cable entering the building. This fiber was connected to the Telco communications cabinet, from which the E1 modems protruded. We wanted to get directly into the optical line, bypass all network devices and leave E1 copper behind. Nowadays, you can easily buy a lot of FPGA cards that already have the necessary network connectors, but then we could not find anything that could work at a speed of 622 Mbit / s, which Koscom showed.
So we developed our own PCB in order to get the PCIe network adapter we needed. We directly plugged the SFP connector onto the XILINX FPGA chip and nothing else. We have previously made some circuit boards for T1 / E1 adapters, which we packed in a small black plastic case designed to simplify our infrastructure. Those circuit boards cost only a few dozen or two cents each and they helped preserve our nerves, since we no longer needed active adapters from NetOptics and our cable connection scheme became simpler. Our FPGA-adapter was very simple by the standards of FPGA-devices, but not an easy thing for us. We brought these adapters and several passive optical converters NetOptics to Korea. After midnight, when everything was quiet, we crashed into cables and plugged our adapters into the disconnection. Now we had a signal, but it was completely incomprehensible.
Returning to Sydney, we analyzed the mirrored bit stream and struggled to understand the signal. In the end, we guessed that the architectural scheme that hung on racks in Korea was wrong. We actually received the optical carrier STS-3 / STM-1 at a speed of 155 Mbps according to the OC3 specification and now we could decode this stream. We had more bandwidth available and we were even able to find our own orders in data packets, which made us very happy. The orders of other clients were encrypted (and this is for the best.) Now, when we had to scan many more 64 kbit / s channels, we had to write our own tools to search for the necessary data, because everything changed too often. In the end, we still went to our own optical star circuit in the rack near the KRX campus, using our FPGA adapters for, which turned out to be quite suitable for arbitration and coordination for eight points of presence.
Preliminary warrant
Now I wanted to try that new KRX order trick. Since this was the TCP protocol, we, too, could immediately send part of the packet with the order to the channel until the entire incoming packet was received and decoded - these are simply bytes. The secondary TCP header was fine, so we would save quite a lot. However, we had two problems. First, if we didn’t have a warrant ready for dispatch, what should we do? Every N seconds we had to send a heartbeat packet if the PID was not used. We could not send a trash bag to the stock exchange, because then we could attract undue attention. Since we were behind the firewall, we could only work with the contents of the IP packets, and no invalidations of the Ethernet checksum or other similar tricks at a lower level worked from here.
The cancel command for an already canceled or executed order seemed to be a suitable message - this often happened during the normal operation of the engine. So we sent it. Each time we randomly chose an order ID so that there were no repetitions. We sometimes received letters with questions about these cancellations (which were sent by one smart broker), so we made a disguise - when there was a lull in the market, the engine returned to the tactic of sending heartbeats packages. All this worked perfectly in the unencrypted test segment of the network, but we still had to solve the problem with the ARIA cipher.
, 1Gbps FIX 4.2 ( BATS 10G), - , FPGA. 25 200 . Ethernet- , .
ARIA
, , C- 8- 32- . , . - , , , .
ARIA, . , SIMD-, . , .
, , TCP. , . , . Koscom “ , ”, , .
ARIA , .
. . Unix . , . , , . , .
, . . , . , " " - , . , , , , ? , , , . , , , . .
. TCP , , , ?
, , , . .
ARIA (CBC). , . , .
, , ( .) , . , KRX . . Koscom, , , - , . , , . - ( ) - CBC XOR . , , . , .
, . TCP-, . 90 , , , . , , , (.. ) , . .
Machine learning
, - , . (RF) . PhD ( .) , , PhD - , , C++. C++ . , ++ (, .) RF. , , , .
, , . OpenDT v5.2, MPI mini-ITX . , - , . OpenDT . , Random Forest, . . ( , , ) . ( ) . – .
ML- . . , low latency. . , ML – . , , , . , . , Jump, , . ML.
ML , . , . , . . ML .
, , , , – . CME , Wall Street Journal , , , , . , . , - .
, . Nasdaq , , . , .
KRX . , – , , . , , , . , , , . , .
. , , , , . , , . , .
. . , ., " ", .
, KRX KOSPI, KSE. KRX - . , - .
. , - , , . . , 2016 , KRX , . - , – . HFT- ( ) , , Koscom , . Koscom, . , , 2.9 3.1 . 300 400- . .
- KRX HP/Compaq/Tandem AIX, IBM PowerPC. 11-12 (RTT) 130 . , 20ms, 30ms. , , .
, CME iLink FPGA. . , KRX, , – ? . , , ( CME, ) . , .
Competition
, , Getco. . , , . Getco , , , . , . , CIBC? , Getco (, ) , CIBC.
Getco 5% . - , 15%. , . Getco. , . , , , . – Getco , . , . , . – . , , . , .
, , , . , , Eurex. . , , . , , , . . . , . , -HFT . . , .
. HFT-, 2005 2011. . 1000 USD 10 000 . , Virtu (1 1200 IPO), . , , . , 10% , 5% 20% . 50%, .
HFT . , – , . HFT- . , . , , , , Getco, . , Jump .
– , KRX . , , . HFT , . , , . , HFT , . HFT , … .
. ETF, , . , , , HFT , . , . , (.. ) . HFT.
HFT – , HFT. , - . -. – , , – , . HFT , – , – .
. , ? , , . HFT-, , . , .
, . DMA-, , , HFT . – ...
. , -- , . , , . , , - .