Viktor Hesselbom

Experimenting with simulation of hard-edged shadows

For a game I'm working on we want to have 2d shadows covering parts of the screen, like in the game Nox, to create a sense of true(ish) line-of-sight. If you've never played Nox it's kind of hard to picture how the shadows work without a moving picture but here is a reference picture to get a grasp.

Nox Shadows

Basically, it projects shadows from all walls and doors, leaving open spots for windows, so the player can't see what he's not supposed to see. This, in my opinion, is a very cool idea because many bird-view 2d games have this problem, but let's not get into that, it will suffice to say that this is the reason we chose to try and simulate Nox's conveniently called 'LineSight'.

Anyway, to try to cut to the chase, here's how I started thinking it could be done.

First I create a simple wall, stored as a line segment between two points, p0 and p1. Then I want to create a shadow from that wall in the opposite direction of the focused point. In my example I will use the position of the mouse for this. Normally, you'd probably want to use the position of the player for this. So I will take the x distance and y distance between both the wall's points and the mouse, draw lines between p0 to p0' and p1 to p1', and then fill it in. The code should then look something like this (I added three walls to easier visualize the results):

class Main
{
	static function main ()
	{
		game = flash.Lib.current.graphics;
		flash.Lib.current.stage.addEventListener (flash.events.MouseEvent.MOUSE_MOVE, update);
 
		walls = new Array ();
		walls.push (new Wall ({ x : 120.0, y : 120.0 },
				      { x : 180.0, y : 180.0 }));
		walls.push (new Wall ({ x : 180.0, y : 180.0 },
				      { x : 240.0, y : 120.0 }));
		walls.push (new Wall ({ x : 180.0, y : 60.0 },
				      { x : 240.0, y : 120.0 }));
	}
 
	static function update (e)
	{
		game.clear ();
		for (w in walls) drawshadow (w, game, { x : e.stageX, y : e.stageY });
		for (w in walls) w.draw (game);
	}
 
	static function drawshadow (w : Wall, g : flash.display.Graphics, ref)
	{
		var xdist, ydist;
		g.beginFill (0);
 
		xdist = w.p0.x - ref.x;
		ydist = w.p0.y - ref.y;
		g.moveTo (w.p0.x, w.p0.y);
		g.lineTo (w.p0.x + xdist, w.p0.y + ydist);
 
		xdist = w.p1.x - ref.x;
		ydist = w.p1.y - ref.y;
		g.lineTo (w.p1.x + xdist, w.p1.y + ydist);
		g.lineTo (w.p1.x, w.p1.y);
	}
 
	static var game : flash.display.Graphics;
	static var walls : Array;
}

And the result something like this (click to set focus and then move mouse to see updates):

Now, this doesn't look like much. But how about if we multiply the distances by, say, 1000?

Now we're talking! However, this is highly unoptimized. Drawing the edges way outside where it should be drawn? Perhaps Flash Player can optimize this and only draw what needs to be drawn, but what if we were to use some other platform? Or maybe we would want to cache the shadows as bitmaps (for God knows what reason... filters?) and we will start to notice an unnecessary slowdown.

So to calculate what point each shadow segment ending should be at I calculate how far p0's x distance is from reaching it's boundary (0 if negative, width of window if positive) in percent and then multiply both x distance and y distance with that value. Then I check if the new point is outside of the vertical boundaries (0 to height of window) and if it is, recalculate percentage with y distance instead. It might sound pretty naive but it works efficiently.

Here is that written in our previous code, with just one wall to easier visualize the difference:

class Main
{
	static function main ()
	{
		game = flash.Lib.current.graphics;
		flash.Lib.current.stage.addEventListener (flash.events.MouseEvent.MOUSE_MOVE, update);
 
		walls = new Array ();
		walls.push (new Wall ({ x : 120.0, y : 120.0 },
				      { x : 180.0, y : 180.0 }));
	}
 
	static function update (e)
	{
		game.clear ();
		for (w in walls) drawshadow (w, game, { x : e.stageX, y : e.stageY });
		for (w in walls) w.draw (game);
	}
 
	static function drawshadow (w : Wall, g : flash.display.Graphics, ref)
	{
		var xdist, ydist, per : Float;
		var width = flash.Lib.current.stage.stageWidth;
		var height = flash.Lib.current.stage.stageHeight;
		g.beginFill (0);
 
		xdist = (ref.x - w.p0.x);
		ydist = (ref.y - w.p0.y);
 
		if (xdist < 0)
			per = (xdist/(w.p0.x-width));
		else
			per = (xdist/(w.p0.x-0));
 
		if (w.p0.y - (ydist/per) < 0)
 			per = (ydist/(w.p0.y-0));
 		else if (w.p0.y - (ydist/per) > height)
			per = (ydist/(w.p0.y-height));
 
		g.moveTo (w.p0.x, w.p0.y);
		g.lineTo (w.p0.x - (xdist/per), w.p0.y - (ydist/per));
 
		//p2
		xdist = (ref.x - w.p1.x);
		ydist = (ref.y - w.p1.y);
 
		if (xdist < 0)
			per = (xdist/(w.p1.x-width));
		else
			per = (xdist/(w.p1.x-0));
 
		if (w.p1.y - (ydist/per) < 0)
 			per = (ydist/(w.p1.y-0));
 		else if (w.p1.y - (ydist/per) > height)
			per = (ydist/(w.p1.y-height));
 
		g.lineTo (w.p1.x - (xdist/per), w.p1.y - (ydist/per));
		g.lineTo (w.p1.x, w.p1.y);
	}
 
	static var game : flash.display.Graphics;
	static var walls : Array;
}

Despite my dreadful explanation of the logic, the code is not very complicated as you can see. Here is what it might output:

As you can see, the points are all hitting the boundaries. My only problem now is how to fill in the corners. I actually haven't figured this out yet. I had an idea that because I know what boundary each segment point is hitting I could for instance fill in to the top left corner if one point was at the left boundary and one at the top. But this didn't really go that smoothly because sometimes one point hits the top and one the bottom, or left and right etc.

So to sum things up, the point of this article was to first of all get this off my chest and to maybe help others by seeing how I approached this. And second of all, I'm asking for help. If anyone has an idea on how I could calculate what corner(s) to fill the shadow to or maybe some other way to do shadows I'm interested to hear what you have to say.