Alex Saint Croix

Over the weekend I sat down and did some high level modeling of community social structures.  I wasn’t making a descriptive or predictive model, but more of a generative model.  That is to say, I was trying to grow a culture of simple fictional people the way I might grow a culture of mold in my fridge.  And I hit upon some peculiar and interesting snags.

First, to build my Tribe, I envisioned a collection of Person objects with some predefined characteristics such as age and gender, and some relationships to other Person objects.  The obvious ones that are mechanically necessary to ensure the tribe can grow are mating/coupling relationships and the resulting parent-child relationships.  It seems reasonable that each Person object should have a mother and a father when it is instantiated.

However, in modeling the progenitors of my tribe, the initial population that I’d use to seed the growth of the rest of the tribe, I didn’t have any forbears to reference during instantiation, so I decided to assign null values to the parents of these progenitors.  It made me think recursively all the way back to the beginning and origins of life, which was … outside the scope of my exercise.  Still, it was a bit fun to run into such a thought provoking snag so early in the development of this culture model.

Then I started thinking about whether couples would separate, so that there were half sibling relationships, and what that would entail. I started thinking about whether people would remarry if one of them were removed from the Tribe.  Suddenly, my naïve notion of tribes and families was tossed into disarray.  Should I put the methods that govern mate selection, remarrying after death, and other vital concepts into the Tribe object?  Or should it be some sort of cultural context in which a Tribe operates?  Or, should it be a concept that applies to each Person individually?  Is culture a top-town phenomena or an emergent phenomena that grows from individual choices?

Other snags were soon to follow.  I considered whether to represent each Person’s age as a value, or as a birthdate.   It seems intuitive that the simulation should keep track of time, so assigning each Person an age value would require constantly updating the value with each tick of the clock. Thus, it’s best to assign each person a “time of birth” value and reference it only when necessary to calculate age.

But should time be a part of each Tribe?  How does time get determined? Is it part of some broader environmental state?  Is there a World object outside of the Tribe?  Are there other Tribe objects in this World?

Suddenly, I started to question the assumptions of my model.  Can a Person belong to more than one Tribe?  Can a Tribe exist on more than one World?  How does time translate between Worlds? 

I went out on a Saturday afternoon to build a simple model of tribal culture.  Instead of a neat and tidy model of people banding together and procreating to grow neat and discrete tribal cultures, my interests shifted in just a few hours to thinking about how to represent the shifting and vaporous social organizations that exist in complex, pluralistic cultures and contextually overlapping social spheres.

As far as thought experiments go, it was a fruitful and invigorating one.

Couldn’t help myself. Rebuilt my controller node today. /woot

user@control01:/etc/init.d# nova-manage service list
2013-03-10 16:55:12 DEBUG nova.utils [req-c7373e01-84cb-48c5-95cc-049257c39991 None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:506
Binary           Host                                Zone             Status     State Updated_At
nova-compute     compute01                           nova             enabled    :-)   2013-03-10 20:55:06
nova-cert        control01                           nova             enabled    :-)   None      
nova-consoleauth control01                           nova             enabled    :-)   None      
nova-scheduler   control01                           nova             enabled    :-)   None 

Thanks to my current circumstances, I can try out OpenStack on something other than a laptop running a pile of VMs.  Instead, I’ve got hardware.  Bare metal boxes, racked and stacked, the way it was meant to be.  And this is great.

But, OpenStack is daunting.  It’s trivial to install the base Ubuntu system, and pretty easy to get the controller node established running Folsom with Keystone, Glance, Cinder, Swift, and Horizon.  I set up a 2nd node to run Quantum, and that was a little bit tricky due to the bridging.  But I got through it.

When the time came to set up my first compute node, things got challenging.  I don’t want to wade into the murky waters too much here.  It should suffice to say that creating a new VM instance is easy, and getting network connectivity for that VM instance has been nigh impossible.

I’ve ripped down and set up so many quantum networks, subnets, routers, ports, and floating IPs, associated them, set up gateways, that I feel I know the quantum CLI commands by motor reflex now. But I still can’t ping my VM.  More details in future posts, but this one’s just for exhalation.  

Going to try carving out a bigger private network to play with, maybe get my own /23 subnet in the lab.  It’s like the chess endgame when you’re ahead in materiel. Simplify simplify simplify.

In the spring of 2003, I spent a crazy week in San Francisco with Greg Wilkins and Jan Bartel, spearheading the design of their Core Developers Network venture.  I remember working all night on their new website with Dain Sundstrom and Jeremy Boynes, and launching the story in the early hours of the morning that a large chunk of the JBoss persistence and web server team were defecting to go their own route.  Like the energy from splitting an atom, splitting an idea releases tremendous power.  The heat from that dissolution spawned the Geronimo project, and much of the rest is history.

One thing that stands out in my memory is how insanely powerful and sleek Jetty was, and is.  On the morning that CDN split off from JBoss, we got “slashed”.  The web server was running at Jeremy Boynes’ house near San Francisco, it was running on Jetty, and it never buckled once.  It is, in my opinion, the single best web server platform for deploying Java apps that has ever been built.  So, it shouldn’t come as too much of a shock that I’m moving back to Jetty, after a long absence (brought on more by the lack of a need to do web development than anything else).

I’m overjoyed to see that Greg and Jan are still at the helm, independent in their development, doing amazing work, and living the dream.

This is essentially where the rubber meets the road for me.  I’ve gotten back to the point I was at in Java, with all of the functional power of Scala at my disposal, and a completely redesigned TileForm library.  Now it’s time to write the new terrain smoother.

Scala allows you to use the lazy keyword to defer the initialization of member values until they are called.  With this technique, you can implement some excellent data structures called streams.

As discussed in Chapter 3.5.1 of Structure and Interpretation of Computer Programs by Abelson, Sussman, and Sussman (a.k.a. The Wizard Book), streams are delayed lists. They let us use sequence manipulations without forcing us to incur the cost of implementing these sequences as lists.  Most importantly, we don’t have to initialize every value of a list in order to look at some of the values in the list.

Over the last two days, I’ve applied this technique to a stream-based QuadTree design, allowing me to have a data structure capable of providing access to massive (Terabyte-sized) terrain heightmaps without forcing me to load all of the terrain data into memory at once.  Moreover, I can use my streaming QuadTree class to generate the terrain, building as much or as little of it as I need for the sake of demonstrations, and preserving the concept of shared vertices between neighboring (or enclosing) branch and leaf nodes.

This design is modular.  Not as much as I’d like, and it needs more refactoring, but it’s modular enough that I could cache the terrain data to disk (or to a network service) after generation, and prune unused nodes from my local copy as I move through the terrain, in order to minimize runtime memory.

While technically possible to do this in Java, it would have been extremely ugly and complex.  Once again, I’m really happy to be writing this project in Scala.

Really looking forward to using this!

Java NIO 2.0 (sometimes referred to as NIO.2) provides comprehensive support for the filesystem. Besides enhancements to the way you create, open, read, and write to files, you can now create custom filesystems and file providers in Java as well. One example is the ZIP file provider, which comes with the API and treats an archive file as a filesystem in its own right.

http://drdobbs.com/blogs/java/231600403

Scala 2.9.0 final was released last night: http://www.scala-lang.org/node/9483

The Scala 2.9.0 codebase includes several additions, notably the new Parallel Collections, but it also introduces improvements on many existing features, and contains many bug fixes.

The data structure I was wrestling with last night collapsed into a binary tree as I slept.  There’s no good reason to use a structure like that for storage, when a binary tree will do.  The lesson to take away from this is twofold: the relationships that are used to generate data need not necessarily linger on the storage of that data, and a binary tree need not be full to be useful.

Eye candy for Inverted B-Tree, DAG,

Inverted B-Tree.  What is this data structure?  It’s not a polytree because there is more than one path between vertices.  It is a DAG, because it’s directed and acyclic.  But there’s got to be a name for it.

I’ve now completely refactored my entire graphics pipeline to use PicoContainer for inversion of control, and separated out the engine components into a variety of services which each focus on running and managing different aspects of the entire application.

As you can see, the number of component dependencies is staggering, and the difficulty of wiring these all together might lead me to want to harden my APIs or lock myself into one particular design.  Because I’m using PicoContainer to manage not only component construction but also component lifecycle, I don’t have to worry about this concern at all.

Moreover, my top-level Engine can focus its entire attention on coordinating the “main render/update loop”, deciding when to skip render steps in order to help updates keep pace.  The StageManager service completely oversees the initialization of the camera, lighting, material and terrain manager services during the startup phase of the component lifecycle.  Components without direct dependencies on one another can communicate via message passing, just as the user communicates with keyboard and mouse events via message passing.

As a result of my refactoring, I can work on every component individually, almost with complete freedom from how it is intertwined with other components in the engine.  I can unit test components individually, free from entanglement with the rest of the engine.  There is a strict separation of concerns that prevents top level elements from manipulating the state of the LWJGL Display, and the “spaghetti code” has been tossed aside in favor of this much more modular, clean, and pluggable “ravioli code” architecture.

Experience tells me that the diagram above is far from perfect, and is continuously in flux.  If you want to use it in your project, you’re welcome to it as long as I receive appropriate attribution.  If you need help wiring together your PicoContainer, let me know and I might be able to help out!

Here’s a snap of the same geometry from before, rendered with the new engine.  New blue-magenta color scale for new architecture shows off the surface slope a little better:

Eschewing method overloading as a matter of principle seems to conform to the general spirit of functional programming, though for the sake of fluent API designs it might still be useful.  I’ll have to chew on it some more.  Thanks for the detailed post, Chris!

iamchrislewis:

What does a compiled curried method in Scala look like in java? Compile this:

object Concat { def concat(s: String)(ss: String) = s + ss def concat(s: String)(i: Int) = i + s
}

Now inspect the resulting class with javap:

Compiled from "Concat.scala"
public final class Concat extends...

Upgraded to PicoContainer v2.10.  Close inspection convinced me that the new lifecycle annotation methods were absolutely crucial, library size be damned.

Here’s an example of how to construct a DefaultPicoContainer instance in Scala, using the  version 2.10 API:

    val pico : DefaultPicoContainer = new PicoBuilder().
      withJavaEE5Lifecycle().
      withConstructorInjection().
      build().
      asInstanceOf[DefaultPicoContainer]

That gives you the actual container.  You can now use it to register and instantiate components, like so:

    pico.addConfig("width", 800)
    pico.addConfig("height", 600)
    pico.as(Characteristics.USE_NAMES).addComponent(classOf[Box])

    val box = pico.getComponent(classOf[Box])
    println(box)

Which yields:

Box(800,600)

The motivation behind using PicoContainer will start to make more sense Real Soon Now™, as I start to add my giant web of dependent game engine subsystems.

Hope this saves you some time getting started with PicoContainer 2.10+ in Scala

Eye candy for Isometric, Terrain Engine, Scala, LWJGL,

Isometric terrain engine written in Scala and LWJGL.  First screenshot of 2nd generation engine still lacks many of the terrain processing features of my first cut, and the tiles in this screenshot are positioned manually just so I could get a screenshot.  Surface colors are determined by surface slope.