Brilliant’s Server Infrastructure (Professional Programming Series Micro-Post)

While thinking about topics for the Professional Programming Series, we realized people might find our server diagram interesting. I think it is worth posting because even though it is mostly industry standard, lots of the diagrams that you'll find on the internet are a bit more abstract and simply describe how things connect in a hypothetical situation. Instead, this diagram accurately represents how things work under the hood at Brilliant.

Click or tap for full size server diagram.

Feel free to ask questions about the server diagram, and I’ll try to answer as many as I can.

#ComputerScience

Easy Math Editor

This discussion board is a place to discuss our Daily Challenges and the math and science related to those challenges. Explanations are more than just a solution — they should explain the steps and thinking strategies that you used to obtain the solution. Comments should further the discussion of math and science.

When posting on Brilliant:

Use the emojis to react to an explanation, whether you're congratulating a job well done , or just really confused .
Ask specific questions about the challenge or the steps in somebody's explanation. Well-posed questions can add a lot to the discussion, but posting "I don't understand!" doesn't help anyone.
Try to contribute something new to the discussion, whether it is an extension, generalization or other idea related to the challenge.
Stay on topic — we're all here to learn more about math and science, not to hear about your favorite get-rich-quick scheme or current world events.

Markdown	Appears as
`italics` or `_italics_`	italics
`bold` or `__bold__`	bold
- bulleted - list	bulleted list
1. numbered 2. list	numbered list
Note: you must add a full line of space before and after lists for them to show up correctly
paragraph 1 paragraph 2	paragraph 1 paragraph 2
`[example link](https://brilliant.org)`	example link
`> This is a quote`	This is a quote
# I indented these lines # 4 spaces, and now they show # up as a code block. print "hello world"	# I indented these lines # 4 spaces, and now they show # up as a code block. print "hello world"

Math	Appears as
Remember to wrap math in `$` ... `$` or `\[` ... `\]` to ensure proper formatting.
`2 \times 3`	$2 \times 3$
`2^{34}`	$2^{34}$
`a_{i-1}`	$a_{i-1}$
`\frac{2}{3}`	$\frac{2}{3}$
`\sqrt{2}`	$\sqrt{2}$
`\sum_{i=1}^3`	$\sum_{i=1}^3$
`\sin \theta`	$\sin \theta$
`\boxed{123}`	$\boxed{123}$

Comments

Wait, why wouldn't you store the user media on the servers? I mean doesn't the data go through them when the user's device requests the data? Btw, nice job

Nolan H - 6 years, 4 months ago

Hi Nolan, great question!

For a while we did store user media on our application server, however, you may note that I used the singular "server".

Several benefits of using a 3rd party service for hosting files are:

You don't have to worry about running out of disk space.
You don't have to worry about semi-arbitrary OS restrictions (We once ran into an issue where we reached the maximum amount of directories that could exist in another directory).
Most importantly, once you have more than one server processing requests for the same content, you either have to sync the files between all of your app servers whenever someone uploads something, or you have to make your own standalone server for hosting user media and set up the ability to pass files from all of the app servers to the file server when someone uploads them (which, at that point, why not use a service like Amazon S3 or Rackspace Cloud Files)

Also, if you look a little closer, you'll notice that "read only" arrows flow from the user browser through the Content Delivery Network (CDN) to the user media service. The typical way in which user media is used is:

A user submits a form that includes a file.
Our server processes the file and if validation passes, uploads the file to the file storage service.
The server saves a new object in the database that includes the path to the image.
When someone views a page that needs to display that object, we use an html img tag to reference the file on the 3rd party service (which is how most images are displayed anyways, even if they are hosted on the same server).
Your browser sees the img tag and requests and downloads the file from the 3rd party service (skipping the app servers).

Sam Solomon Staff - 6 years, 4 months ago

Ohhhhhh, ok. Once again, very cool.

Nolan H - 6 years, 4 months ago

I am just curious, how physically distributed are the app/task servers ? I mean, do you have separate dedicated servers for different continents (since the clients are from all over the world), or you have a centralized design ?

Abhishek Sinha - 6 years, 4 months ago

Unfortunately for now (though fortunately for our sanity) our servers are all centrally located. It is definitely something that keeps coming up, but it's a really big task to figure out how to split everything up. This is especially true because any type of distribution would likely require changes to how Brilliant works in order to make up for not having quick read/write access to all of the databases.

That being said, we do spend a lot of time figuring out how make the site faster.

We always try to reduce...

the number and size of files that need to be loaded
the amount of sequential round trips (redirects for instance, are relatively fast if you're close to the server, but can be very slow due to the redirect response and the resulting request needing to circle the globe)
client rendering time (you may notice that most math is actually rendered on the server side and only rarely will our JavaScript $\LaTeX$ renderer have to step in to render math after the page loads (this is what the latex renderer server in the "Services" box in the lower right hand corner is doing)).

Sam Solomon Staff - 6 years, 4 months ago

How do you manage Database write operation , So do you perform write operations into a DB , And use other DB for all read operation.

Isn't a master master architecture is more suited ?

Vishal Vasudeva - 6 years, 4 months ago

We perform most write and read operations on the main database and perform some heavy read operations on the other.

Master-master architecture is good for some uses, but for us, the downsides aren't worth it at this point. The main things that we rely on that would be hard or impossible to get and have be efficient in a master-master setup are database transactions and a guarantee that once we store something in the database, it's immediately available for retrieval and will stay available (unless we explicitly delete it).

One of the main benefits of master-master architecture is having a quick failover which is possible to mostly accomplish with our setup by promoting the follower to be the master if the master goes down.

The other main benefit of master-master architectures is they can be used to distribute master-nodes around the globe so that you can have app servers closer to the browser, but distributing the database like that causes all sorts of synchronization problems unless you have a very simple site where the write operations are much more limited/orderly than we have (see this comment for a bit more info on why we don't yet distribute our servers).

Sam Solomon Staff - 6 years, 4 months ago

Math	Appears as
Remember to wrap math in `\(` ... `\)` or `\[` ... `\]` to ensure proper formatting.
`2 \times 3`	$2 \times 3$
`2^{34}`	$2^{34}$
`a_{i-1}`	$a_{i-1}$
`\frac{2}{3}`	$\frac{2}{3}$
`\sqrt{2}`	$\sqrt{2}$
`\sum_{i=1}^3`	$\sum_{i=1}^3$
`\sin \theta`	$\sin \theta$
`\boxed{123}`	$\boxed{123}$