Pretty interesting stuff. I'm amazed that there are only 14 nodes in the cluster with 1TB SSD each; message size must be fairly small on disk (less than 295 bytes, unreplicated). I'd also recommend looking at maybe tuning the shard_request_cache; there are some possible improvements to be made there, if you're running that many indices. Finally, are all of the indices of approximately the same size? Are they time-boxed?
edit: Put together a logging cluster consisting of 14 nodes, 12 data + 2 indexer/search API, with ~40TB of consumer-grade SSDs. Ran ~14k indices based on log type & timestamp, with a whole raft of custom field configuration to handle aggregations and different tokenizations. In sum, Elasticsearch looks easy to configure and tune but is amazingly hard to do well - but incredibly rewarding.
> but is amazingly hard to do well - but incredibly rewarding.
Isn't this true of everything. The greater the reward the more effort required in most scenarios? I do agree, Elasticsearch can take some trial and error along with experimentation to get right!
Yeah, that's nearly a tautology. I guess in this sense what I mean is that, there's a much deeper Trough of Disillusionment when it comes to learning how to properly configure & tune ES to your needs, as opposed to a lot of other tools I've used over the years. It's very easy to set up, but also very hard to get right, which I'd imagine causes some people to settle or give up.
The small message results from only storing message, channel, and server ids as opposed to the entire message. How efficient is ES at storing the rest of the message data in the inverted index? No clue, but apparently pretty good.
In the cluster I previously built/worked on, somewhere between 4mil & 9mil records/shard - this also assumed 3 shards per index, with only one replica set (total of 6 shards).
Well-written, props. Some architecture questions you might want to think about:
- Why only one elasticsearch shard (+replica) per index? You run some major risks of write-time hotspotting here, especially if you ever have to completely rebuild your indices from scratch (pro tip: plan for it now). If you have any basic metric of a discord server's size prior to index creation, you can pre-optimize the number of shards in its index to ensure that writes are distributed across the cluster.
- What happens when one server's index grows too large to fit on one (elasticsearch) shard? Practical shard size limit in ES is ~50GB due to concerns related to reallocation and recovery. You could also theoretically hit the 32GB memory limit.
Also, as another poster mentioned, consider dedicated master nodes. You're going to lose data nodes from time to time, don't let it cost you consistency by bringing down a master with it.
We've been running large ES clusters here at Revinate for about four years now. I've found the sweet spot to be about 14-16 data nodes, plus three master-only nodes.
Right now, we're running them under OpenStack on top of our own bare metal with SAS disks. It works well but I have been working on a plan to migrate them to live under Kubernetes like the rest of our infrastructure. I think the answer is to put them in StatefulSets with local hostPath volumes on SSD.
We use CoreOS for our Kube cluster and are huge believers in it so I will definitely be leveraging that for ES. One thing I have done for my Cassandra clusters is to write a custom health check running in a container that monitors Cassandra's health and locks the cluster with locksmithctl to prevent reboots when the database is unhealthy. This will be easy to translate to ES. Correspondingly, I will add a curl call to the ES unit file to disable shard reallocation when the service is shut down during a reboot. This prevents costly reallocation of shards when you know that a node will be back online shortly.
OT but: so many engineering blogs lack a link to the company's main site. It's a small thing but it bugs the hell out of me. I was just reading the (also currently front page) article on Segment's blog and actively noticed and appreciated the presence of such a link.
Is it a Medium thing? Does it disallow linking out from a Medium hosted, privately branded blog?
I'd agree with this, but I also had quite a bit of success with using indexer/search nodes as master-eligible, and then basically 'pin' (manually) the master node to one completely dedicated host. That's one complaint about ES, there's no way to set master preference among a set of master-eligible nodes.
Yup. We're thinking about that too. We've been monitoring node performance closely to see when we'll need to shift the topology around a bit. I'd want to keep our clusters small and the topology uniform before I investigate dedicated master nodes though.
FYI: From my experience (on older ES versions) when you hit problems with it, they will be pretty bad, catastrophic problems. We run dedicated master nodes, but have the master servers running multiple ES instances, so they can serve as masters for multiple clusters.
The first time you run a search in a server (or the first time you run a search in a server after the index fails) - will trigger a full re-index of that server. Ctrl-F "Historical Index" in the blog post for more details! If you've never used search in a server - the messages are not indexed in real time until you do for the first time. Both these things make the system "lazy".
The worst case to an index failure is that the search query is delayed as the index rebuilds itself. We throttle the rate of historical indexing into ES to a safe level so that we're not degrading performance of other components of the system.
Oh, I think I get it now-- is it that the _initial_ indexing is lazy, but all indexing after that is done automatically by the historical index workers? Basically when a user searches for something do you check that ES has something for that user, if it doesn't start off the initial indexing process, and from there the workers do their thing?
The historical index workers index the history of the server, whereas we have a real-time index worker that is consuming and bulk inserting messages in real time. Searching for the first time in a server turns on real-time indexing and triggers a historical index of all previous messages in that server.
Can you talk about the bulk insert for real time messages ? How does this get triggered - does it run every X minutes or X messages (your code looks like it is every X messages).
Got it. I was under the impression that all messages were lazily indexed. After reading the article again it's pretty clear that's not the case. Thanks for the clarification.
I agree with you, and I've got a question as well.
I'm wondering how long does it take to execute the ES refresh on a search query when the Shard was marked as dirty?
If the search requests are mostly real time, I suspect this is really short, but if the Shard ingest new messages for a while (let's say 50 minutes) and it's marked as dirty, a search query would ask ES to refresh 50 minutes worth of documents before running the actual query.
As it shown to be a problem? Is the refreshing time growing along with the number of documents inserted since the last refresh?
Good question. So far we've noticed the refresh time to be negligible (worst case in the tens of milliseconds). It's worth noting that most of the cost of doing a search on Discord is in pulling the message context from Cassandra to provide enough data to render the results in the client.
> Elasticsearch supports automatic shard rebalancing, which would let us add new nodes to the cluster, fulfilling the linearly scalable requirement out of the box.
I'm assuming they mean "I can add a new node to the cluster, and NEW SHARDS FROM NEW INDEXES can be distributed amongst the new nodes".... So far as I know elasticsearch can't rebalance shard location or composition automatically based on cluster membership change events...right? I mean, the cluster reroute api will let you manually move a shard, but thats all I know of.
I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.
There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.
Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.
Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes.
Tuning cluster rebalance settings can be... challenging. I've done some bad, bad things with a cluster where it would effectively balance things back and forth - forever.
Fiddling with cluster.routing.allocation.balance.* can really paint you into a corner quickly.
Yes that's actually one of my favorite ES features. Kafka doesn't do any type of automatic topic partition rebalancing which was quite shocking to me when I first started using it. Seems kind of primitive in comparison.
On the other side of the spectrum, I recently did a mini review of available F/OSS options for full-text indexers /search tools. Requirements: natively compiled, one-shot/single-process per search, no long-running containers so EL, SOLR, and anything else based on mighty Lucene and Java was out of scope.
The reviewed tools were: clucene, Apache Lucy, Indri/Lemur, Xapian, and Zettair. Of those, only Xapian and Zettair could be made to return useful search results within time constraints. Sadly, I didn't get Apache Lucy to build, which along with Xapian is the only option still being maintained. clucene could be built, but delivered erratic results (it found "impeachment" but not "indictment" in the US constitution text), and was too difficult to extend into a full stand-alone app (though it apparently is being used in Informix still).
I have only tried it for an hour or two, and can't speak for its search result quality, but it seems promising.
On my laptop, with an index of 10,000 documents each containing 200 random words, it takes about 110ms to perform a fuzzy search. Half of that is the actual search, the other half is the process/initialization overhead.
FWIW we will likely open source our search engine/IR technology sometime next month (hold up is the documentation and cleaning up some parts of the codebase).
It's written in C++, and the core features are performance, simplicity(operation and extension of functionality) and a sane, simple API (the codebase is quite small as well, at about 14k lines of code).
Its similar in some regards to Lucene, but does away with most of the exotic features and instead is optimised for rich queries support, fast execution and extensibility.
Sphinx does NOT use an RDBMS as store. It just provide a convienient interface so you can query Sphinx like you would do a SQL database, just one of several APIs to interact with Sphinx. The data store is very similar to Lucene, a folder with several files.
>4 primary metrics that we use to decide when the cluster needs to be grown
I would imagine you could measure some sort of throughput/response time values to tune this instead of relying on cpu usage because having 100% cpu usage doesn't mean app is poorly performing it could mean app is performing optimally with all the resources available to it. Similarly heap_free shouldn't be measured on its own, neither should be tuned on its own, ideally jvm should be left to its own devices to tune these values based on your inputs in form of params like MaxGCPauseMillis, GCTimeRatio ect.
> Search API: An API endpoint that clients can issue search queries to. It needed to do all the permission checks to make sure that clients are only searching messages they actually have access to.
We've had to implement this exact thing for our index, and were never quite sure if we were doing it right. To "prepend" a security clause/condition into an incoming search query felt awkward and hacky.
> Self-healing: We don’t have a dedicated devops team (yet), so search needed to be able to tolerate failures with minimal to no operator intervention.
Do people not hire small business support technicians anymore? I worked for a guy when I was a teen who took support contacts to manage small business networks. He would call me and send me to fix a router when something broke. He couldn't have charged more than $2K per month, basically just to be on-call. That's 24k a year for an on-call ops tech, that's fairly cheap I think.
(Also, "self-healing"? Is this different from hooking up a service request with timeout to an APC with remote power and rebooting the sucker?)
> Of course, this entire search infrastructure would be incomplete without a way to discover clusters and the hosts within them from the application layer.
((If you want something that isn't complicated to use or maintain, why would you rely on technologies which use complicated decentralized algorithms just to find them??))
Their Elasticsearch type mapping shows they're using the English fulltext analyzer for message content, I'm curious if they're planning on supporting fulltext search for other languages in the future.
Maybe one day. It'd be trivial to add per-locale templates. The hard part really is exposing that in our new guild flow (hey - what language is this server gonna be in) - and also handling if someone chooses to change their server language (this would have to trigger a re-index). We'd also have to replicate those language analyzers in our client to do proper highlighting.
Great writeup, thanks! For the search query, are you guys using Elasticsearch's query strings or have you wrote your own? If using elasticsearch's, have you had any issues so far?
edit: Put together a logging cluster consisting of 14 nodes, 12 data + 2 indexer/search API, with ~40TB of consumer-grade SSDs. Ran ~14k indices based on log type & timestamp, with a whole raft of custom field configuration to handle aggregations and different tokenizations. In sum, Elasticsearch looks easy to configure and tune but is amazingly hard to do well - but incredibly rewarding.