I thought today I would take some time to describe the method I'm using for building, serialising and deserialising the network packets in War Worlds. I am using ENet as the underlying network protocol, which means the War Worlds networking system is based on UDP.
I won't go into too much detail on how connections and such are managed, since a lot of that stuff is handled pretty well by ENet already. What ENet doesn't handle, though, is how you structure your packets - from ENet's point of view, you just get an array of bytes. But that's not much use to us, since we think in terms of objects and fields. So the question is, how do we map between our objects and a simple byte array and back again?
When I started doing this stuff, I looked into Boost.Serialization, but I found it to be rather too much for the relatively simple requirements that I had. I decided that I could implement it myself in a short enough time and have full control over the serialised format.
The first part of solution comes from the packet_buffer class which works much like a SC++L stream (though much simplified) and lets us build up the binary data in a fairly easy and intuitive way. A basic outline of it's implementation is below (I've omitted the implementation of the more obvious methods):
class packet_buffer
{
private:
bool flushed_;
std::string value_;
std::stringstream buffer_;
uint16_t packet_type_;
void flush();
public:
packet_buffer(uint16_t packet_type);
packet_buffer(char const *bytes, std::size_t n);
~packet_buffer();
void add_bytes(char const *bytes, std::size_t offset, std::size_t n);
void get_bytes(char *bytes, std::size_t offset, std::size_t n);
char const *get_buffer();
std::size_t get_size();
uint16_t get_packet_type() const { return packet_type_; }
};
void packet_buffer::add_bytes(char const *bytes, std::size_t offset, std::size_t n)
{
flushed_ = false;
buffer_.write(bytes + offset, n);
}
void packet_buffer::get_bytes(char *bytes, std::size_t offset, std::size_t n)
{
buffer_.read(bytes + offset, n);
}
void packet_buffer::flush()
{
if (flushed_)
return;
buffer_.flush();
value_ = buffer_.str();
flushed_ = true;
}
char const *packet_buffer::get_buffer()
{
flush();
return value_.c_str();
}
std::size_t packet_buffer::get_size()
{
flush();
return value_.length();
}
This provides the basic implementation of the "buffer”. As you can see, it's a pretty basic wrapper around std::stringstream (though we could've used std::vector<uint8_t> or something as well). The slightly more interesting aspect is the following helpers:
packet_buffer &operator <<(packet_buffer &lhs, int32_t rhs)
{
lhs.add_bytes(reinterpret_cast<char const *>(&rhs), 0, 4);
return lhs;
}
packet_buffer &operator >>(packet_buffer &lhs, int32_t &rhs)
{
lhs.get_bytes(reinterpret_cast<char *>(&rhs), 0, 4);
return lhs;
}
// (more here for int16_t, uint32_t, etc...
packet_buffer &operator <<(packet_buffer &lhs, vector const &rhs)
{
lhs.add_bytes(reinterpret_cast<char const *>(rhs.data()), 0, sizeof(float) * 3);
return lhs;
}
packet_buffer &operator >>(packet_buffer &lhs, vector &rhs)
{
lhs.get_bytes(reinterpret_cast<char *>(rhs.data()), 0, sizeof(float) * 3);
return lhs;
}
packet_buffer &operator <<(packet_buffer &lhs, colour const &rhs)
{
uint32_t rgba = rhs.to_rgba();
lhs.add_bytes(reinterpret_cast<char const *>(&rgba), 0, sizeof(uint32_t));
return lhs;
}
packet_buffer &operator >>(packet_buffer &lhs, colour &rhs)
{
uint32_t rgba;
lhs.get_bytes(reinterpret_cast<char *>(&rgba), 0, sizeof(uint32_t));
rhs = fw::colour(rgba);
return lhs;
}
packet_buffer &operator <<(packet_buffer &lhs, std::string const &rhs)
{
// strings are length-prefixed
uint16_t length = static_cast<uint16_t>(rhs.length());
lhs << length;
lhs.add_bytes(rhs.c_str(), 0, length);
return lhs;
}
packet_buffer &operator >>(packet_buffer &lhs, std::string &rhs)
{
uint16_t length;
lhs >> length;
char *value = reinterpret_cast<char *>(_malloca(length));
lhs.get_bytes(value, 0, length);
rhs = std::string(value, length);
return lhs;
}
So as you can see, these functions are what actually make it easy to serialise things. Want to serialise a string? Just use operator << to stream it in. An example of one of my packet classes is below, to show you the basic usage of my class:
// this packet is sent from the server when you connect to it
class join_response_packet : public fw::net::packet
{
private:
std::string map_name_;
std::vector<uint32_t> other_users_;
fw::colour my_colour_;
fw::colour your_colour_;
protected:
virtual void serialise(fw::net::packet_buffer &buffer);
virtual void deserialise(fw::net::packet_buffer &buffer);
public:
join_response_packet();
virtual ~join_response_packet();
// etc...
// (I'll get to these in a second)
static const int identifier = 123;
virtual uint16_t get_identifier() const { return identifier; }
};
void join_response_packet::serialise(fw::net::packet_buffer &buffer)
{
buffer << map_name_;
buffer << other_users_.size();
BOOST_FOREACH(uint32_t sess_id, other_users_)
{
buffer << sess_id;
}
buffer << my_colour_;
buffer << your_colour_;
}
void join_response_packet::deserialise(fw::net::packet_buffer &buffer)
{
typedef std::vector<std::string>::size_type size_type;
buffer >> map_name_;
size_type num_other_users;
buffer >> num_other_users;
for(size_type i = 0; i < num_other_users; i++)
{
uint32_t other_user;
buffer >> other_user;
other_users_.push_back(other_user);
}
buffer >> my_colour_;
buffer >> your_colour_;
}
As you can see, when you join a new game, the server sends to you the current name of the map (a string), the session identifiers for all the other connected players (integers) and the "colour" of yourself and the server (this is used to differentiate between players in the game: red vs. blue, etc).
The final difficulty comes when you receive a "bunch of bytes" from your peer. How do you know which class to deserialize? This is the only place where a bit of macro magic happens (and that's really just to save a bit of typing). First of all, the header file:
// this macro is used by the packet class to "register" itself
#define PACKET_REGISTER(type) \
shared_ptr<fw::net::packet> create_ ## type () { \
return shared_ptr<fw::net::packet>(new type()); } \
fw::net::packet_registrar reg_ ## type(type::identifier, create_ ## type)
// this is what you call to create an instance of a packet from a packet_buffer
shared_ptr<packet> create_packet(packet_buffer &buff);
typedef shared_ptr<packet> (*create_packet_fn)();
class packet_registrar
{
public:
packet_registrar(uint16_t id, create_packet_fn fn);
};
And the corresponding .cpp file:
static std::map<uint16_t, create_packet_fn> *packet_registry = 0;
shared_ptr<packet> create_packet(packet_buffer &buff)
{
create_packet_fn fn = (*packet_registry)[buff.get_packet_type()];
if (fn == 0)
{
// error!
}
return fn();
}
packet_registrar::packet_registrar(uint16_t id, create_packet_fn fn)
{
if (packet_registry == 0)
packet_registry = new std::map<uint16_t, create_packet_fn>();
(*packet_registry)[id] = fn;
}
So at the top of the .cpp for the packet, we just call the PACKET_REGISTER(packet_type) macro and it'll register itself with the system. This is what the identifier member of the packet class is for: it's used by the PACKET_REGISTER macro to assign an integer to that packet type which we put into the packet_buffer when sending it to the other side.
For the time being, I'm just manually giving each packet type a unique identifier by hand. It's not that much trouble (and there's not a whole lot of packet types to do anyway). I've also got some error checking code in my packet_registrar::packet_registrar method so that if two classes have the same identifier, it logs the error so I can fix it.
And so, that's basically it. The system is very simple, doesn't take much to maintain and it will allow me (in the future) to handle some more advanced scenarios with relative ease (for example, I could use a variable-length encoding for integers to save space, I could implement compression, or I could add packet coalescing).
Next time, I'll hopefully have some more screenshots or videos to show. I'm in the process of implementing some basic pathfinding, which has been fun and gives the units some much more "intelligent" (looking) behaviour...