Geopipe's ContextSnap Streams Entire Cities with Cesium and 3D Tiles
Posted by Christopher Mitchell, Ph.D..
Geopipe CEO and cofounder Christopher Mitchell originally wrote this blog post for the Cesium blog.
Geopipe has built a tool called ContextSnap that can stream entire cities into your browser with Cesium and 3D Tiles. With ContextSnap, you can navigate a 3D copy of a city within your browser, and click on buildings and other objects to see semantic information: height, number of floors, materials, address, and more. If you need to take 3D models out into your own workflow, you can select your desired area, and immediately download a 3D mesh model in your favorite file format.
Why are we building these models? From gaming to training autonomous vehicles, architecture to defense, perfect virtual copies of the real world are increasingly vital. Unfortunately, the real world is complex, dense, and nuanced, and with today's hand-built 3D models of cities, it can take a team of professionals months to reproduce even a single neighborhood in realistic detail. While several well-known companies have automated the process, the resulting models, usually made with a technique called photogrammetry, simply don't have the material or object information necessary for renderings or interaction. But we realized that computers could be taught to understand the world without human-powered labeling, automatically building rich, detailed 3D models that contain information about what's in the world, not just how it looks. Geopipe uses deep learning and other algorithmic techniques to understand cities and other areas from raw, unlabeled sensor data. These algorithms can determine every detail of the world, from where buildings are, what they're made of, and where their windows and doors are, to the locations and properties of roads and their lanes, sidewalks, trees, and water. From raw data to complete labeled 3D models takes the pipeline less than a day for an entire city.
Once these models have been generated, though, users need a way to explore and use these models. Some customers simply want to view the resulting cities, navigating a 3D world, clicking on objects to see semantic information, and potentially overlaying their own information on top of those models. That's where Cesium and 3D Tiles come in. Geopipe reconstructs hundreds of thousands (often millions) of buildings and other objects in each city. At the finest level of detail, every pane of every window is reproduced, and no computer on the market today could load an entire city at that level of detail at the same time, especially in a browser. On the other hand, it must be possible to view our models from any perspective within the city. With 3D Tiles, we can severely cull the geometry that needs to be loaded to show the entire city at any given time, with the camera anywhere within the city.
We already automatically generate our 3D models at up to 13 different levels of detail and geographic scale, facilitated by knowing the difference between (for example) building, trees, roads, and sidewalks, and downsampling each type of object intelligently. For medium levels of detail, we simplify models (use fewer polygons for trees, for example, or replace windows cut into the sides of buildings with facade features painted via texture). At our lowest detail models covering the broadest geographic areas, we even cull small buildings and foliage: these models will be loaded far from the camera, so the simplifications won't be obvious. Since a 3D Tiles tileset is a hierarchical arrangement of models covering an area at different levels of detail, the only remaining missing piece is the hierarchy of
tileset.json metadata files. We feed our models into Cesium via a server that dynamically generates the 3D Tiles tileset from our models and their metadata. Because we use batched 3D models, the underlying metadata describing individual objects is retained in models Cesium displays, so Cesium's API can be used to display information about those objects when a user clicks on them. The results is Geopipe ContextSnap.
Other customers want to go further, downloading models that they can pull into their existing toolchain for editing, rendering, and interaction. ContextSnap is still part of this workflow. Customers can preview a version of the models they'll be getting from Geopipe (we limit ContextSnap to all but our two highest levels of detail, due to the memory and CPU constraints of WebGL), then drag a box around the area they want, choose the level of detail and file format appropriate to their use case, and download the model immediately.
In fact, 3D Tiles are sufficiently expressive that Geopipe is using similar concepts in its Geopipe for Games SDK, a plugin for the Unity game engine and an underlying RESTful API that allow customers to directly stream Geopipe's worlds into games, VR experiences, simulations, and other software. Resources like CPU and GPU memory are still dear, so presenting as immersive a world as possible with as little geometry loaded as possible is still important, and you still need to be able to show the perspective from anywhere in a city. Once again, the solution is loading immersive, high detail models close to the player or viewer, and broad, lower detail models far from the player. Thus, miles of a city and distant oceans, islands, and mountains can be visible at the same time while the framerate is kept at acceptable levels even on unexceptional hardware.
Whether you need 3D models for your own game, simulation, or AEC project, you need to stream models of the real world into your own application, or you need to analyze data about the objects in a city at scale, Geopipe can help. And with Cesium and 3D Tiles, customers can inspect and explore a city containing millions of objects in their browser. When you need the real world in your projects, let us know!