MyWorld Meetup – 5/25/19
Announcements:
- MyWorld has switched from the Cassandra database to CockroachDB.
- In the last few months, we’ve been moving from a more traditional world model to a “regionless” world, where the world is visible to users as one enormous space instead of being divided into many separate regions. With that change, naming areas of the world and cutting it into parcels has also grown to allow nesting named areas and parcels, very much like how real world areas are divided into countries, states, cities, neighborhoods, and lots. While Cassandra fit our older zone-oriented model quite well, the new world model relies on nested hierarchies in the world data – a data model which fits Cassandra very poorly.
- Rather than spend precious development time force-fitting the new world model to Cassandra, we’ve switched to CockroachDB. CockroachDB is a cloud-scale SQL database – it scales very similarly to Cassandra for handling very large data sets and heavy workloads without downtime, but uses a traditional SQL data model that provides far more flexibility than Cassandra’s more limited model does. This allows us efficient and natural support for the new world model, and also provides flexible indexing for much more of the data in MyWorld’s database – which means much more flexibility and power as we design new features.
- In short, the switch provides much more power for MyWorld’s next-gen design, without compromising on the scalability and reliability that led us to choose Cassandra initially.
- CockroachDB is also significantly easier to setup and run on a personal computer than Cassandra is. This makes it much easier for us to support people who want to run a MyWorld server on their own computer.
- We’ve already completed the switch – MyWorld is currently running on a CockroachDB server, and we’re quite satisfied with the results we’ve seen so far.
Discussion:
- Zauberin – Does CockroachDB support JSON?
- Yes – it has first-class support for JSON as a column data type, including indexing JSON data for fast querying.
- Yes – it has first-class support for JSON as a column data type, including indexing JSON data for fast querying.
- Zauberin – Does the index support arbitrary JSON documents, such as deeply nested fields in a free-form structure?
- This lies outside of our current usage of CockroachDB, but it’s my understanding that the indexes do support that well. Right now entity data is serialized to a custom binary format, but we’re investigating making use of JSON columns & indexing to allow querying on entity attributes. It’s not clear if we’ll go this route or not yet – the current format does have some advantages over JSON, despite being opaque to queries.
- This lies outside of our current usage of CockroachDB, but it’s my understanding that the indexes do support that well. Right now entity data is serialized to a custom binary format, but we’re investigating making use of JSON columns & indexing to allow querying on entity attributes. It’s not clear if we’ll go this route or not yet – the current format does have some advantages over JSON, despite being opaque to queries.
- Zauberin – One of the projects at InWorldz was switching internal data storage to Protocol Buffers. Have you investigated using that format?
- In the early days of MyWorld, we investigated Protocol Buffers but stayed away for several reasons. For one, Protocol Buffers have some awkward aspects to the format, and the tools significantly complicate compiling your code. For another thing, Protocol Buffers work best when the data you’re serializing is of a previously known type – when you don’t know the type of the data ahead of time, you have to manually add type information to the Protocol Buffer data. As a result, it’s not effectively different from what our current format does, and it has the downside of complicating MyWorld further and adding dependencies on a format and a set of tools that have proven to be somewhat quirky in the past. This is why we haven’t gone that route.
- In the early days of MyWorld, we investigated Protocol Buffers but stayed away for several reasons. For one, Protocol Buffers have some awkward aspects to the format, and the tools significantly complicate compiling your code. For another thing, Protocol Buffers work best when the data you’re serializing is of a previously known type – when you don’t know the type of the data ahead of time, you have to manually add type information to the Protocol Buffer data. As a result, it’s not effectively different from what our current format does, and it has the downside of complicating MyWorld further and adding dependencies on a format and a set of tools that have proven to be somewhat quirky in the past. This is why we haven’t gone that route.
- Zauberin – In Minecraft, large caves are common but pose some difficult issues for culling at render time. Are there plans for generating large caves in MyWorld?
- No, especially in the short term. The alpha release will use traditional heightmapped terrain, but in the future we’ll be adding support for voxel terrains. While small caves and terrain with overhangs add a lot of depth to a world, I don’t see virtual world users having a high demand for large underground cave networks. Even after we add support for voxel terrains and implement the world generator, I would lean in favor of biasing the generator towards generating smaller caves, not large underground networks.
- Zauberin – Minecraft also had a challenge with cave culling because they store chunks that are small horizontally but extend upwards in tall columns. This makes culling underground features harder.
- Most of the people I (Daniel) personally know work with small cubic chunks as a standard practice – probably for this very reason. It definitely makes culling much easier to handle. We’ll cross that bridge when we come to it with MyWorld, but I expect that we’ll handle voxel terrains in the same way (small cubic chunks).
- Most of the people I (Daniel) personally know work with small cubic chunks as a standard practice – probably for this very reason. It definitely makes culling much easier to handle. We’ll cross that bridge when we come to it with MyWorld, but I expect that we’ll handle voxel terrains in the same way (small cubic chunks).
- No, especially in the short term. The alpha release will use traditional heightmapped terrain, but in the future we’ll be adding support for voxel terrains. While small caves and terrain with overhangs add a lot of depth to a world, I don’t see virtual world users having a high demand for large underground cave networks. Even after we add support for voxel terrains and implement the world generator, I would lean in favor of biasing the generator towards generating smaller caves, not large underground networks.
- FordLady – You’re talking a lot about Minecraft – in Minecraft everything you build is very blocky. Will MyWorld be like that, or will you be able to build in different styles, such as building a Victorian house?
- MyWorld and Minecraft are totally different – in Minecraft, everything (terrain, houses, machines, etc) is made of voxels. In MyWorld, voxels will only be used to support terrain with features like canyons, overhangs, and caves (which aren’t possible with the traditional heightmapped terrain that current virtual worlds use), and MyWorld’s voxel engine will produce smooth, natural looking terrain – not blocks. Technologically, Minecraft’s voxel engine is relevant to MyWorld for terrain, but buildings in MyWorld will always be made of either mesh or our CSG “SuperPrims.” So no, your creations won’t look like Minecraft’s block style – unless you explicitly make them to be that way.