Shift8 Creative Graphic Design and Website Development

I haven't designed in a while. It's just been too crazy busy with programming web sites. I'm working on a project now where I get to design again! It's very exciting. I realize now I have a real good ability to see both sides of the fence here. I'm an educated graphic designer and took many years of typography classes (more than the required amount). So I know exactly what I'm doing with regard to setting type and I also happen to love it. Sadly, I settle for what the web has and it's limitations. I also know that if I go and create images for every single bit of text...I miss out on SEO and it's a real pain to update. 

So we suffer when it comes to the web. We all do. Of course there's various forms of dynamic text replacement, some using Flash, others PHP (with GD library), and then of course don't forget the new possibilities with CSS @font-face. FontSquirrel and Google are two really good places to get some free fonts that are more than your basic web fonts.

That's great and all, but... We're still limited. Being creative takes a lot of work manipulating CSS and HTML. However, it is possible. The problem is that it's pretty much single serving. You don't always know how many words are in a piece of copy so it's hard to create re-usable styles for things like titles, subtitles, and pullquotes. For example, look at the titles for my blog posts. Sometimes there's enough words in the title that it spans several lines. When you compare that against setting type for print... You realize that you can adjust things to make it fit on one line if so desired. On the web, we have to live with what we get for the most part.

I'm trying to fix that. I'm working on a jQuery plugin that allows you to create beautiful typographic lockups and have those be a re-usable set of rules. So you can apply the same style to, say, all your blog post titles, but with a bit of intelligence. Things like if there are this many words do this, or break the line at the 5th word, etc. Think of it like programming working with design in unison. It's beautiful. See (and see above)?

Now, the best part is that this lockup works with 4 or more words. It'll just keept wrapping if more lines are needed. Every word including and after "post" in this case will be the same size and everything. The only thing required to output the above is a span element wrapped around the title. So when it comes to something like creating a template for a CMS or blog, there's no extra work beyond applying a class.

How does it work? Well, in the above screenshot, both the date and the title have a span element wrapped around them with a specific class. The jQuery plugin then basically applies a set of "typographic rules" to the copy. Things like how many words per line, etc. The date is slightly different, that just gets broken out so that the day, month, year, all live in their own span element. It's much more straight-forward than the type lockups.

The JavaScript then breaks up the copy and replaces it with new HTML that has each line wrapped with a div and each word wrapped with a span element. Classes are also present with all this marking each individual word and line so they can be targeted exclusively with CSS. For example: l_1, l_2, etc. for lines and w_1, w_2, etc. for words. Each line and word also have an "l" and "w" class on them allowing for a broader set of CSS rules. Of course, each lockup can have its own class and id values applied to it so you can further target specific lockups to apply different styles.

Once you get your CSS defined and what I'll call the "sequencing" of words per line, you're set! You can then copy and paste the styles and JSON settings for the jQuery method call and share them across sites and such if you want. It's just a tiny bit of configuration there to create beautiful and re-usable typographic lockups.

I'll be working on this a bit more to polish it and add more settings. I also have an option to wrap everything in quotes if you want as well for things like pullquotes. You'd simply add a "q" class to the element you applied the method to or you'd pass to the method an option for quotes set as true. 

More to come soon!

Agile Uploader is now available on Github! I've decided to release the source code for the project after a long while and careful thought. The reason? I simply can't continue to maintain it. The tool does exactly what I was after and quite well for my needs. While I still will use other upload tools, when it comes to needing to resize images that users upload (namely photographs, typically off SD cards), it does the job perfectly.

However, the tool was falling short of some (maybe 10%, or less, of the comments I received) user's needs. I briefly went over some of my research involved with building the tool before, but I don't really want to make anyone work hard to get from A to B. So I'm going to give everyone B and if you want to get to C...Then, by all means, go for it.

You can find the project here on Github. Please fork it and feel free to leave feedback. I can't promise that I'll be working on it much more, but I will fix any major bugs. I hope that people can add features that will help others out there.

A note for what you're looking at...You are looking at a FlashDevelop project. In the normal "bin" directory, where the SWF file is output to by default, I also included all the demos for the tool. Most importantly is the agile-uploader-3.0.js file. This is the jQuery plugin that I built and bundled with Agile Uploader. This directory structure leaves a bit to be desired I know, but it should give you everything you need. Please reference the jQuery plugin along with comments in the source code of the Main.as file. The tool really works by Flash's ExternalInteface communication of Flash <-> JavaScript. You don't need to use jQuery, but you'd need to write your own JavaScript to use Agile Uploader if you didn't...Or if you wanted to write your own custom jQuery too of course.

You can work on Agile Uploader with the Flex SDK and FlashDevelop. The tool is built completely upon open source code.

Enjoy!

Minerva was only ever a few ideas I threw around. However, I noticed people were following it. So I wanted to write a short blog post to help people get started with it. This would allow you to see how it works. By no means should you attempt to use this for a production site. I mean, you could...But, I'm telling you don't. The reason is because I don't want you to be disappointed when something doesn't work the way you'd expect or when you run into a problem that you can't figure out. Further, there's really no documention. Lots of comments within the code...But no documentation.

That said. I would absolutely love for others to get involved and fork the CMS. I would love to hear ideas and feedback.

First, the requirements... You need to be running PHP 5.3+ and using MongoDB. I know some of you would love to use MySQL or some other database, but due to the schemaless nature of MongoDB, Minerva requires it. Sorry. You'll also need a few various libraries from the community. Don't worry, we'll go through those below. Personally, I'm ok with these requirements because I don't want to re-invent the wheel. There's plenty of good CMS' out there that use MySQL and older versions of PHP. It's not my intention to re-invent the wheel.

So, Minerva works as plugin that can sit on top of any of your Lithium apps. With a working copy of Lithium, clone Minerva to your app/libraries folder. In fact, you may want to use git submodules. So really, you'd want to run something like (you may need the -f flag to do this by the way):

git submodule add git@github.com:tmaiaroto/minerva.git app/libraries/minerva

Then do the same for li3_access and li3_flash_message.

git submodule add git@github.com:tmaiaroto/li3_access.git app/libraries/li3_access
git submodule add code@dev.lithify.me:li3_flash_message.git app/libraries/li3_flash_message

After that, you would run two more submodule commands in order to get the files you need (note if you cloned Lithium from github, your lithium library directory should also be a submodule so you'd need to do this for that as well):

git submodule init
git submodule update 

If you would like Facebook support (the ability to login to the CMS with your Facebook user) then you'll also need the li3_facebook library.

git submodule add git@github.com:tmaiaroto/li3_facebook.git app/libraries/li3_facebook

Note: With the li3_facebook library, there is a submodule for the Facebook SDK within the library. So you will need to run a git submodule init and update within app/libraries/li3_facebook in order to retrieve it.

That's it. All the files will be ready to go. Of course, now you need to add the libraries within your app/config/bootstrap/libraries.php file.

Minerva is designed so that you don't need to touch any of its files that are cloned from the repository. You can pass your config options within your own libraries.php file when you call Libraries::add(). The order in which you add these libraries is going to matter a little bit. You need to have li3_access loaded before Minerva. This will work:

Libraries::add('li3_access');
Libraries::add('li3_flash_message');
Libraries::add('minerva');

That's basically it. Note, if using Facebook integration you will not need to add the li3_facebook library since the minerva library will do that for you. However, you will need to specify values for 'appId' and 'secret' keys in the minerva library configuration under a 'facebook' key. ie. Libraries::add('minerva', array('facebook' => array('appId' => 123, 'secret' => 123))); Some other handy configuration keys include, 'connections' which is an array with 'production' and 'development' each normal connection settings arrays. Also a 'show_errors' and 'development_errors' key both with boolean values. More on this later.

Now you can go to yoursite.com/minerva. Actually, I use minerva.local, so I'll reference that from now on. Just replace that with whatever domain name you have setup. You'll see the welcome page. It has some copy, some of the instructions are actually dated and wrong. It will tell you that you can edit files within the Minerva library in order to change templates...I am changing that to add another template path which would be under your main app directory. Again, this is so you don't need to touch any files within the minerva library. No git ignore, etc. 

You likely want to see the backend though. You'll need a user account to do that. The first user account will become admin. No other users registering after that will have an admin role, so don't worry. I do plan to create an intstaller in the future. However, go to: http://minerva.local/minerva/users/register

This will be the front-end registration form. It'll register the first user as an administrator role. So you can go back to: http://minerva.local/minerva/admin which will redirect you to login. 

Now you should be logged in and see the CMS backend. You can add users, pages, etc. However, you'll like want to take a look at the minerva_blog library to see how pages can be hooked into and extended to create new types of pages. You'd add that after the minerva library, again, with Libraries::add('minerva_blog'); You could also add that as a submodule.

At this point, I think you get a general idea for the direction of the CMS. Keeping it modular is important, but dependency issues could arrise. If you went to try Minerva without having the li3_flash_message library or configure it for use with Facebook without the li3_facebook library then it would show you a warning message about missing dependencies. I hope to expand upon that, not just to maybe make it look a little nicer, but to allow all libraries to run through some sort of dependency check process. 

Along with keeping things modular, the other goal is to keep things clean. "Minerva Plugins" we'll call them, which are really just libraries, interact with the CMS without touching any of the core code. So you add and remove them very easily while still being able to stay up to date with the latest version of Minerva. The goal is to ensure backward compatibility, but of course problems could crop up with that as well. I'd preferably like it to warn you before an upgrade and I would like to have upgrades from the CMS control panel. So, there's a lot to consider and a lot left to do.

There are really no test cases for the CMS and things are in a state of change. However, most of the foundation is pretty solid. I don't imagine I'll be changing how plugins work and how you can extend core models to add new schema, validation, etc. The template paths are pretty much solid now too. Again the idea is that you put your templates outside the minerva library. 

Feedback, questions, and pull requests always welcome! Hopefully I can put up a roadmap soon so I can take this into more of an alpha status.

As noted above, there's a few configuration options. These will grow more robust, but for now the important ones are for including Facebook and debugging purposes. Here's a snip my app/config/boostrap/librarires.php for an example (note: these are all optional setting, the default database name is minerva and minerva_dev):

Libraries::add('li3_access');
Libraries::add('li3_flash_message');
Libraries::add('minerva', array(
    'connections' => array(
		'production' => array(
			'host' => 'localhost',
			'database' => 'minerva_cms'
		),
		'development' => array(
			'host' => 'localhost',
			'database' => 'minerva_cms_dev'
		)
    ),
    'facebook' => array(
        'appId' => 'XXXXXX',
        'secret' => 'XXXXX'
    ),
    'show_errors' => true,
    'development_errors' => true
));
Libraries::add('minerva_blog');

To the connections array you can add all the same settings you would normally. So, if you were running MongoDB in replica set mode, you'd want to have a key 'replicaSet' set to true. You would also want your 'host' key to have a comma separated list of host names.

Every web designer, developer, and just really anyone who does business should have one of these tablets, I swear.

Why? Well, normally I'd have to say tablets are not really that helpful. They are fun, I think everyone is crazed over them, but they don't offer any real productive value. Sorry, but it's true. You play Angry Birds on them is what you do. Then carry it to the office so you can try to justify it. I could see that if you don't have a smartphone, then a tablet is a great tool for productivity.

Not the HTC Flyer. This is a tablet that really is geared for business and its a great addition to a netbook, smartphone, or laptop because it's simply different than all of them. It's honestly one of the only tablets that has something unique about it. The pen.

While reading about this tablet I was concerned the pen wouldn't be precise enough. Well, it is. It can be quite fine tipped. It works just about as well as some other devices like a Palm Pilot or Windows device with stylus...Only, better. There's two levels of pressure sensitivity as well, so it's pretty good for sketching. Mind you, not as good as what Wacom tablets offer, but honestly pretty close.

Drawing on glass, a slick surface, isn't so bad...Though you do miss out on the tactile feel and it's not real natural. I assume you can put on a matte screen protector and you'd get some resistance that felt more natural.

So why for web designers? It should be obvious...You can note anything. So you can simply go to your web sites and mark them up. Conveying design changes, etc. are a dream come true when you can do that. You can sketch up fresh ideas for design as well to then share. 

Additionally, you can draw all over PDF documents. These can be saved down as flattened PDFs or layered PDFs that will open up back on a PC. You will need to get a file manager app in order to e-mail those saved PDFs out, but you can. It's nice to open those layered PDFs up on a PC to review what you annotated and then remove those markings if you wanted to.

Think about tools like Mockflow and how they export to PDF. Or any other design tool where you may have a site map or wireframe. Bring that PDF right into the Flyer, mark it up, and e-mail it right back out. Beautiful.

Further apply this to business and think about signing PDFs. You can sign off for things, contracts, approval, etc. Having a pen in a digital world means serious business. It helps bridge a gap.

Think about presentations too. You can be going through a slideshow and make markings with the pen to help annotate. Speaking of presentations...The notes app has a recorder. You can record a lecture or presentation so you can play it back later. While this recording is going on, you can take pen notes and those drawings are placed on the audio timeline of your recording. Another beautiful feature.

I can't wait until there's more pen apps out there because I could only imagine the number of apps that could help make my day to day job easier and more productive.

Simply put. Every web designer and graphic designer should have one of these tablets. If you do business in a creative field, this is a must. Even for other industries the pen is a nice tool.

I currently use Komodo for my IDE. It's a wonderful IDE and I like the name. However, it's not great for its code intelligence when it comes to PHP 5.3+ and it has trouble with the "go to definition" function sometimes. Other than that, I love the IDE. I think it's still in the top 5 for IDEs for PHP development.

So I never used NetBeans before and never really knew much about it. I always thought it was an IDE for Java. So when I came across their website again and it said it supported PHP I did some digging. I figured yea yea it can edit PHP files, so can notepad...But nope! I was pleasantly surprised! Here's a brief overview of the things I like about it.

Command Line Tasks/Special Framework Integration
The IDE even has support for the Symfony framework. It allows you to right click a Symfony project and clear the cache or run a Symfony command that you would normally do from the command line. So creating new modules and such is much more natural. I don't deal with Symfony for my personal projects, but it's nice to have that integration because I do work on sites that use Symfony almost on a daily basis for work. My current framework of choice is Lithium and I'd like to see if I could get a module to do something similar for Lithium (and if I want to learn some Java, I bet I can).

Code Intelligence
The autocomplete and code intelligence is amazing. It works with PHP 5.3+ namespaces and it picked up everything for me in my Lithium project. When calling methods from my own models it picks up all the inherited model methods and as I write them it guides me along with the arguments. Exactly what I always wanted in an IDE but seldom found. Eclipse would do it fairly well but it's a beast as you may know. NetBeans is a bit more light weight. It has a "go to definition" feature and also a "go to test" feature. It integrates well with PHPUnit. Too bad I'm not really using PHPUnit.

Revision Control Support
It has SVN, Git, Mercurial support and more. The Git support comes as a module you'll need to download. It doesn't seem to work with pushing and pulling from remotes, but it seems to do everything else fine. This is the one area I'm disappointed with. Git I think is one the more popular systems these days. I hope a module becomes available for it. This brings me to my last point.

Open Source
It's free! You can download NetBeans for free to use. It also has a community of developers that are working on it and building new features all the time. New modules, updates, etc. I really think that's a good approach for an IDE. The reason is because every developer is different when you're making a product you ultimately have to make decisions about features to include. Even if you later apply updates, there's still decisions about what to work on. Then your team is limited. With open source, your team is much larger and a different group of people (or even a single developer) can decide to work on something that they feel is important and work on it when they want. So the PHP developers out there can be happy with this IDE and get the features they need while the Java developers also get an IDE that serves them and both groups here don't even necessarily know what's up with the other. It's nice and it's how Eclipse works.

NetBeans of course has all the other nice debugging features working with XDebug and so forth. Lots of configuratin options, options for shortcuts, etc. It has an add-on module that helps you play with and test regular expressions too. It's a full blown IDE. It really isn't missing much. For me, it was the code intelligence that made me fall in love. I always wanted a no hassle IDE that had this. More importantly, code intelligence that works with your own code. It just means a few less browser windows that I have to leave open for reference.

If you haven't checked out NetBeans yet, you should do so. It's easy to setup and I think for PHP 5.3+ projects, it's a good IDE. Alternatively I've heard good things about PHPStorm too. I will likely check that one out in the future, but it does cost money.

So after realizing my big problem with schema design for my little metrics example, I've changed the design a bit. Obviously if you store all this visitor data on one document, you're going to exceed the 4MB limit. So this was not just an example for me, this was to be actual code! Yikes! I'm glad I caught it a few hours later. So now imagine a schema where your "metrics" collection has a bunch of documents with all the same recorded information. Illustrated like so:

/* 0 */
{
  "_id": {
    "$oid": "4ddc78d27f8b9ada44000002"
  },
  "browser": "Firefox",
  "browser_ver": 4,
  "os": "Win7",
  "mobile": false,
  "lang": "en-us",
  "project_url": "example",
  "_key": "192.168.126.2@example"
}

/* 1 */
{
  "_id": {
    "$oid": "4ddc78d97f8b9ade44000004"
  },
  "browser": "Chrome",
  "browser_ver": 11,
  "os": "Win7",
  "mobile": false,
  "lang": "en-us",
  "project_url": "example",
  "_key": "192.168.126.3@example"
}

/* 2 */
{
  "_id": {
    "$oid": "4ddc78dd7f8b9adc44000001"
  },
  "browser": "Chrome",
  "browser_ver": 11,
  "os": "Win7",
  "mobile": false,
  "lang": "en-us",
  "project_url": "example",
  "_key": "192.168.126.1@example"
}

Again this example is specific for some of my needs. I still want unique metrics on a per project basis in the system. So the _key field has a unique index on it. That simply takes care of that, whereas before I was using the $set operator, now I just save without worry. The slight difference is this time there won't be any updating of information if the user came back with a different browser later on. Like before the metrics aren't skewed because each "unique" visitor only has one entry per page/project tracked. Also like before, the downside is this isn't truly unique...It's more of a "per household" or a "per office" situation.

I now just have to make a very simple query to get all documents where the project_url is "example" in this case. This will eventually return a huge number of documents. Far too many to just pull up and loop using PHP each time I want an aggregate. So we don't need a map reduce now. We could use one, but it's overkill. We just want "half" of that...The reduce part. 

So we use the group() command and it kinda all works out to be similar in nature as the map/reduce. Again with Lithium, we want to get the connection using $db = ProjectMetric::connection(); and then we can run $db->connection->command() calls. Also don't forget the MongoCode class. We'll need that again for the reduce and finalize part of this.

Here's the juicy stuff.

$db = ProjectMetric::connection();

// We just need a reduce, not a map reduce in this case, it's actually a group() command
// We're also going to use a finalizer to format the data afterward.
$reduce = new \MongoCode("function(doc, prev) { ".
    // count browsers
    // first if the browser doesn't exist, set a key name for it with a value of 0
    "if(typeof(prev.browsers[doc.browser]) == 'undefined') {".
       "prev.browsers[doc.browser] = 0;".
    "}".
    // then increment
    "prev.browsers[doc.browser] += 1;".
    
    // count operating systems
    "if(typeof(prev.operating_systems[doc.os]) == 'undefined') {".
       "prev.operating_systems[doc.os] = 0;".
    "}".
    "prev.operating_systems[doc.os] += 1;".
    
    // count mobile devices... this is easier it's just one value
    "if(doc.mobile === true) {".
        "prev.mobile += 1;".
    "}".
    
    // count languages
    "if(typeof(prev.languages[doc.lang]) == 'undefined') {".
       "prev.languages[doc.lang] = 0;".
    "}".
    "prev.languages[doc.lang] += 1;".
"}");

// Make the output pretty, we don't want any spaces in key names for example...
// So we're taking those key names and setting them as values of a new "name" key.
// What if extract() was used or something? Those spaces would be an issue.
$finalize = new \MongoCode("function(prev) { ".
    // browsers
    "for (var x in prev.browsers) {".
        "prev.browsers.push({ name: x, count: prev.browsers[x] });".
        "delete prev.browsers[x];".
    "}".
    
    // operating systems
    "for (var x in prev.operating_systems) {".
        "prev.operating_systems.push({ name: x, count: prev.operating_systems[x] });".
        "delete prev.operating_systems[x];".
    "}".
    
    // languages
    "for (var x in prev.languages) {".
        "prev.languages.push({ name: x, count: prev.languages[x] });".
        "delete prev.languages[x];".
    "}".
    
    // mobile is different and is fine as is
    
    // return the data
    "return prev;".
"}");

// Run the command
$metrics = $db->connection->command(array(
    'group' => array(
        'ns' => 'project_metrics',
        'key' => 'project_url',
        'initial' => array('browsers' => array(), 'operating_systems' => array(), 'languages' => array(), 'mobile' => 0),
        '$reduce' => $reduce,
        'condition' => array('project_url' => $url),
        'finalize' => $finalize
    )
));

// Set the results, they'll be formatted how we want them to be
$results = $metrics['retval'][0];

// We can get unique visitors basically by the total number of documents pulled back
// because there is a unique index on the collection, It's on the "_key" field which
// takes the visitor ip and combines it with the "project_url" value. So there can't
// be duplicate entries so unique visitors is simply the total count.
$results['unique_visitors'] = $metrics['count'];

I commented it pretty well (honestly for my own benefit haha), but I'll briefly explain. The $db->connection->command(array('group'... part. We need to define "ns" which is the collection name, we also need to specify the "key" which in this case is the project url. This key is what to group the documents on.  So all documents with a "project_url" key of a given value are going to be grouped together. If we wanted to retrieve aggregate data for multiple URLs then this would come in handy. In this example, the "condition" key is going to kinda ruin the whole grouping thing because we're only going to return documents of the value we're grouping by. So one group always. Not several groups of data per URL.

The magic parts are the $reduce, initial, and finalize keys.  The reduce is actually another reduce function using the MongoCode class. So that's very similar to map/reduce. Same goes for the finalize function. The initial key is new. That essentially sets some default values for us to use. These will persist so as we loop we can easily keep adding on to them. In the most basic situation we have this "mobile" key set to 0. In the reduce function we're just incrementing it by one everytime the mobile value from the document (also named "mobile") is true. Simple. The others are also just counters.

The final results returned have array key names that have spaces in them. Ew. We don't want that in PHP. So I just made a simple finalize function that went through and moved the value (the count) to an array for each item to count that has a "count" key and a "name" key. The value for the "name" key of course can have spaces in it and there won't be any problems for PHP. It is possible to have spaces in the key names, but if we wanted to run extract() or something, there'd be issues.

That's about it. Another example. I'm not sure exactly how practical these examples are, but they do show you how you can make these operations in PHP when using Lithium. Once I wrap my head around things a bit more, I may try to give back a little and take a stab at making methods within the MongoDb adapter for these things.

On a final note. The group() command is designed to be executed in "real time" whereas map/reduce is more of a background task. So, back to practicality. This may still not serve my needs as the collection grows because the query could end up taking a long time. At which point I guess I'll have another map/reduce example to show for.

Update: So after I got a little further along with this very example in a real life project I realized that while it makes for a very simplistic illustration of map/reduce (that I personally found helpful when learning how to perform map/reduce), it is not a very good real life example. The reason being...  ...The document size limit in MongoDB. Doh! You couldn't store metrics like this. However, ignoring the purpose of this, you can still continue reading about how to perform a map/reduce within Lithium.

Original Blog Entry

I'll start off by saying I love MongoDB and map/reduce after putting it off for some time. I dreaded learning the map reduce functions big time. It turns out, it's not that bad. A friend asked me to explain it in 10 words or less. So I did. It's not really all encompasing of the features, but it's a real good example for what map/reduce can do for you.

Use JavaScript to identify/"map" data to loop it to aggregate/"reduce."

Ok, so that's 12 words technically, I cheated by adding slashes and combinging two words. It's also really poor grammar. Anyway, that's the idea. In this example, I wanted to collect information about visitors on a web app. Obviously I'm not a masochist, I'd use Google Analytics if I could...Sadly, I could not. So what to do? Well, we can use MongoDB to record all this data and then use map/reduce to get some totals.

I may eventually turn this into a Lithium library (especially because there's a good browscap and language detection class that I'm not illustrating here), but for now I'm going over things at a high level and focusing on the actual map/reduce process.

That said, imagine a data set like this:

"metrics": {
    "pageviews": 63,
    "visitors": {
      "192-168-126-1": {
        "ip_address": "192.168.126.1",
        "browser": "Chrome",
        "browser_major_version": 11,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      },
      "192-168-126-2": {
        "ip_address": "192.168.126.2",
        "browser": "Firefox",
        "browser_major_version": 4,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      },
      "192-168-126-3": {
        "ip_address": "192.168.126.3",
        "browser": "Chrome",
        "browser_major_version": 11,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      }
    }

Now, we have this "metrics" field where ever you like, but in my case on a document that contains some other information. Why not a separate "metrics" collection? We could and then we could also put in things like page URLs that were hit on the site to start getting analytic information about the pages on our site. In my case, I just wanted to get a sense for some high level information about my visitors. For now.

So the first thing here that you'll notice (and I've written about the $set operator before) is that each IP address is the key for each entry. The dots have been replaced with dashes so that it works as a key. Otherwise, I'd have a pretty deep object on my hands.

So each time a page is loaded the pageviews count goes up and the visitor's browser information is captured using $set so that if the user from the same IP address came back again with a different browser, it would update. My metrics would not be skewed. Yes, it's sad that we don't realize when/if the user actually uses two different browsers...More sad that we're likely counting entire office buildings as one user, but that's just how the cookie crumbles in this case.

Ok, so we have that data and we have some controller action in our Lithium project that's going to return to us an array that we'll pass to the view template to make some pretty pie charts. Why not pie charts? I love pie charts, they give everyone a sense of satisfaction that looking at numbers is really fun! ...Or something like that.

We'll dive right in. Here's the entire action I'm using with the map/reduce code. Note that Lithium's MongoDb adapter does not have any options for map/reduce in the find() or any other method. I may write something in the future for that myself if I end up doing enough of these (and I likely will). However, we can make straight up command() calls from it.

 public function metrics($url=null) {
        if(empty($url)) {
            return false;
        }
        
        $db = Project::connection();
        
        // construct map and reduce functions
        $map = new \MongoCode("function() { ".
            "emit(this.metrics.visitors, this.metrics.visitors);".
        "}");
        
        $reduce = new \MongoCode("function(k, vals) { ".
            "var visitors = vals[0];".
            "var unique_visitors = 0;".
            "var b_counts = new Array();".
            "var browsers = new Array();".
            "var os_counts = new Array();".
            "var operating_systems = new Array();".
            "var mobile_devices = 0;".
            "var ln_counts = new Array();".
            "var languages = new Array();".
            
            // loop all the emitted visitor metrics to aggregate some data
            "for (var i in visitors) {".
                // count browsers
                "if(typeof(b_counts[visitors[i].browser]) == 'undefined') {".
                    "b_counts[visitors[i].browser] = 0;".
                "}".
                "b_counts[visitors[i].browser] += 1;".
                
                // count operating systems
                "if(typeof(os_counts[visitors[i].operating_system]) == 'undefined') {".
                    "os_counts[visitors[i].operating_system] = 0;".
                "}".
                "os_counts[visitors[i].operating_system] += 1;".
                
                // count the primary languages
                "if(typeof(ln_counts[visitors[i].primary_language]) == 'undefined') {".
                    "ln_counts[visitors[i].primary_language] = 0;".
                "}".
                "ln_counts[visitors[i].primary_language] += 1;".
                
                // count the number of mobile devices
                "if(visitors[i].mobile_device == true) {".
                    "mobile_devices += 1;".
                "}".
                
                // count the number of unique visitors
                "unique_visitors += 1;".
            "}".
            
            // loop browsers counted and set for output
            "for (var x in b_counts) {".
                "browsers.push({ name: x, count: b_counts[x] });".
            "}".
            
            // loop operating systems counted and set for output
            "for (var x in os_counts) {".
                "operating_systems.push({ name: x, count: os_counts[x] });".
            "}".
            
            // loop languages counted and set for output
            "for (var x in ln_counts) {".
                "languages.push({ name: x, count: ln_counts[x] });".
            "}".
            
            // return the output
            "return { 'browsers': browsers, 'operating_systems': operating_systems, 'languages': languages, 'mobile_devices' : mobile_devices, 'unique_visitors': unique_visitors }; }");
        
        $metrics = $db->connection->command(array(
            'mapreduce' => 'projects', 
            'map' => $map,
            'reduce' => $reduce,
            'out' => array('merge' => 'mapReduceMetrics')
        ));
        
        $cursor = $db->connection->selectCollection($metrics['result'])->find()->limit(1);
        foreach ($cursor as $doc) {
            $results = $doc['value'];
        }
        
        // Get the total page views for this project
        $pageviews = Project::find('first', array('fields' => array('metrics.pageviews'), 'conditions' => array('url' => $url)));
        $results['pageviews'] = $pageviews->data('metrics.pageviews');
        
        return $results;
    }

Yea, it's not the prettiest to look at. It's my first run through and it's literally based off an example from php.net so that's why there's all those lines concatenated together like that. I wouldn't normally do that. Nor would I use heredoc...But something a little nicer, at least single quotes instead of double. Anywyay, with that you will be returned a nice array (in $results) that will show all the counts for browsers and such. Note, I did not take into account the browser major versions here in this example. Also note that I separately stored a pageview count on the document which does not require a map/reduce to retrieve. 

Now let's look at it deeper. There's a lot of good articles on map/reduce if you spend time with them, they should be pretty clear. Here is a good one. Then you can also look at the MongoDB Cookbook site's example. Also php.net's example. You'll see that you can use map/reduce for many things. Let's go over how I'm using it.

First, the map function. Pretty simple. In fact, you likely wouldn't do what I'm doing here. The idea of it is to basically grab keys and values for a given collection. Those keys should be unique. So in my case metrics.visitors are unique keys. They are also the values that I need. What this does is returns the values to a reduce function.

The reduce function. More complex, but it's all nice friendly JavaScript. Here you're just looping the values that are passed and simply counting some of them. As a disclaimer, my example could have probably been written a lot better and cleaner. I only loop once which is what I was concerned about mainly. The rest can be refactored later.

At the end of whatever you decide to do with all that data, you'll return your values. I'm returning an object here with all the counts. Here's what PHP gets back in $results:

array
  'browsers' => 
    array
      0 => 
        array
          'name' => string 'Chrome' (length=6)
          'count' => float 2
      1 => 
        array
          'name' => string 'Firefox' (length=7)
          'count' => float 1
  'operating_systems' => 
    array
      0 => 
        array
          'name' => string 'Win7' (length=4)
          'count' => float 3
  'languages' => 
    array
      0 => 
        array
          'name' => string 'en-us' (length=5)
          'count' => float 3
  'mobile_devices' => float 0
  'unique_visitors' => float 3
  'pageviews' => int 256

...And there ya have it. What I would do next is actually cache this data so each time I called the action, it didn't have to run the map reduce which could be quite expensive over time with a lot of data.

Cool note: In this example you see the $metrics = $db->connection->command(...) part? Run a var_dump() on $metrics. It will have some handy information for you. It could tell you about an error when it comes to parsing your functions (though I'm not sure how to actually debug things, sorry). It also will tell you if everything was ok and ran successfully. You may wish to check this before returning data. It's on my to do list myself. Also, it will show you how long the operation took which is very handy. You might need/want to index some fields and cache results based on how long things are taking.

Another note: With map/reduce you're actually outputting to a collection. So you're going to pick up your results with another query to that (temporary or not so temporary) collection. This changed in MongoDB version 1.8.0. You now have to specify that 'out' key in the command() call. Here's more information on that. 

Hopefully these snippets will be of some help to people. I didn't want to go too far in depth with explaining everything, I think there's other really good articles on that out there. My hope is that seeing an example, as it works within the Lithium framework, will be helpful.

Geek conferences.  Well, perhaps "trendy" conferences that all us web developers are always wanting to go to. Or...Meetups...But darn, that meetup and that conference they are half way across the country. It costs so much money and takes so much time. If you're lucky your boss might pay for you to go (as well they should for the value!). Though, probably that won't happen so you miss out. 

Well, that's where I believe in recording and slideshare, etc. However...That's only so useful. This is 2011 God damnit, where's my hover car and sexy computer lady voice that I can talk to when I'm lonely?! Ah, the Ballmer peak got to the best of us...Or maybe not. We actually do have many great examples of technology. We finally have tablets (again), but now that hardware is better they are actually somewhat cool. We have a lot of good web software too and don't forget the cloud. The problem is, you have to hunt for some of this awesome tech...And so, I've found something very noteworthy.

OpenMeetings.de is a great site (and open source project) that uses red5 to provide online meetings and presentations. Ok, yea there's AdobeConnect and Webex and all those other guys...However, they don't have quite as many features and they mostly all require some special software to download and have some problems. Typically most of those services run better on Windows as well. However, the most problematic thing is they all cost money. Lots of money. It's hard for open source communities that have little or no funding to afford these services so...They don't. You end up with conferences that try to get sponsors to cover the costs and you end up with wikis and slideshare and IRC chat rooms. Don't get me wrong, all that is still needed...But I think we should have somewhere to turn for virtual meetups.

OpenMeetings doesn't give you live screen sharing like most other services (and before I forget, join.me is another great mention). The screen sharing is actually a series of screenshots updated fairly frequently. Certainly frequently enough to show a presentation. Almost enough to show live typing, but I think that's enough for the needs of a meeting. Furthermore, you can actually record the whole event to later download as an AVI or FLV! In addition to this very elusive feature (RDP and RFB have been around for ages, yet it's still hard to find good free solutions apparently), it also provides video and audio communication via your web cam and microphone. It also provides text chat and a whiteboard. You can give control to other users and allow them to share their screens and video/audio as well.

It certainly appears to have all the features that one could need. It also has a few other goodies like polls but the most important feature, aside from the sharing parts, I think is the fact that you can plan meetings on a calendar and then invite people. However, how easy is it to use? How well does it work? From what I tested, it's very compatible and pretty easy to use. In fact, it's so compatible that it also worked on my Android 2.3 tablet! You obviously couldn't share your tablet's screen and it didn't recognize the webcam on my tablet, but it definitely worked as a viewer. The resolution being a bit lower than an actual computer made it a little hard to read text but you could see the shared screen and whiteboard. Further, you could actually interact with the whiteboard. Drawing on that with your finger was definitely fun. The speed was fair, again I don't think the whiteboard is 100% real time, especially due to how it renders when you first load it (it seems to replay all the actions). However, it works. Virtual meetings, virtual classrooms, here we come!

Note: Oh, and if anyone from 10gen is seeing this, YES that is a MongoDB mug in the background, I went to an actual conference to get to have it only break on me day one as I cleaned it...I can still use it, as long as I don't pick it up by the handle. You still owe me a mug, especially for entering that contest last month! It was a lot of typing...What does a guy have to do to get a mug? I've even now made a product/brand placement for ya. 

I recently soft-launched my personal project, Family Spoon. You can go to it and use it, it works. It's a recipe sharing website for you, your family and friends. Basically it allows you to create recipes and share them (or keep them private). There's a bit more to it and the site will continue to grow over time. However, with just the sharing there were some considerable hurdles to overcome.

Database Schema
The juicy schema details? Look at the bottom for an example document structure, but for readability I'm not going to paste that here. I'll breifly go over the architecture a little bit. The site was developed in PHP using the Lithium framework. The use of MongoDB was critical to the way in which the site was built, especially for speed reasons. Each "recipe" on the site actually sits in a "page" collection in MongoDB. It's saved there by a "Page" model. Nearly all pages (except static pages and other form pages) are treated this way on the site. Not active, but built is a "blog" section for the site and that also uses this Page model and collection. Obviously, a recipe and a blog entry have very different information. It's due to MongoDB's schemaless design that these two "ideas," which are really the same thing (they are web pages with content), can exist in the same space. Both have titles, but one has ingredients and the other some body copy (and yes some more fields). So right there that saved a ton of time. You need to be aware of, but not meticulously design (and overdesign) your database schema and ensure it's properly normalized, etc. I can now, worry free, add a new model that extends the Page model to add a completely new section to the site and I know it's going to be stored in the same place in the same manner. Done.

In the past? With something like MySQL I would have many many tables to ensure I properly optimized and normalized things. Then when I wanted to add a new section, it would mean either adjusting, or if I did a really really good job, adding at least one more table for the new section. Now, what happens if (and I will) in the future I need to add more than just prep time, cook time, etc. on the recipe document? No problem! I can easily stick in another field and not even think twice. I can also remove them. Again, with other database, I'd have to watch and worry about schema changes. It would take much longer to not only build a site, but also change it. MongoDB makes maintaining a database for a site easy.

Furthermore and leading into the next section is datatypes. Along with database schema, we would traditionally have to think about exactly what kind of data was stored in these tables. Reserving too much is a bad thing and we're sworn off from things like BLOB, so you have to play this delicate planning game. Am I really going to have a user that has a recipe title more than 255 characters? Nah, no way. Really? Oh, well they can't. They just can't. The user will deal with it. The user will live. Eh...ok...But not great. We had to draw the line in the past, but no more! (except for maybe 4MB for now)

The other big thing here with schema and data types is tagging. We love to tag these days. Photos, articles, people, etc. Now, recipes. Family Spoon allows you to tag your recipes on the site in order to be able to find them more easily in the future. These are custom user entered phrases that get stored as an array in the database. Before, we would immediately create a "tags" table to ensure the same word wasn't repeated God forbid. We would join a million times and a week later, we'd have our results. Ok, with memcached back seat driving we'd get there faster. So with MongoDB, we just store those tags in the natural manner we would expect. Again, done and done.

Searching & Filtering
I'll focus on permissions and search (and filtering) next. First, search. The site features a search box at the top of each page and depending on what area you're in, it will search for different things (the search button copy changes to indicate this too). These are regex searches on the database. It was the use of MongoDB that allowed for this whereas FULLTEXT searches with MySQL wouldn't really work so well. Then again we have problems with searching tags and all those JOINs. No good.

So MongoDB is allowing Family Spoon to run a regex search on recipe titles and tags. It's extremely easy to adjust this to search for other things...Say, ingredients?? Then perhaps filter by ingredient? So let's take a detour. Say we want to add a feature to the site that shows a user only recipes that do not contain shrimp because they have an allergic reaction to that food. Done. It's extremely simple to build this query and have it working on the site. A query for MySQL would work with WHERE IN() and not and ... all that good jazz. Sure, but it's not just about the query. It's about being able to handle that tall order. If I was using MySQL for Family Spoon and wanted to show people recipes that did not contain shrimp...I would have to put all sorts of caching strategies in place and perhaps if a popular enough feature, would need additional hardware to support the increased load. In fact, I wouldn't do it. I'd use a search engine like Solr. Ok, so now something else to setup and configure. Yes, we know how to do it, but time, time, time.

Back to regex searches. Again, you simply wouldn't do it with MySQL. You'd setup a search engine. I may eventually need one for Family Spoon (for things like weighting), but right now, no. There's no additional search engine. It's just running queries on MongoDB. We're also filtering. We're going to roll right into permissions here with that.

Permissions & Access
So, we have filtering in addition to search because we can show "your" recipes, "public" recipes, or "your family" recipes. This is determined by the access that each recipe gets assigned by it's creator. So in addition to search, we're searching specific recipes. While searching for public recipes, we mean only return recipes that also are flagged as being shared with the public. When searching your own, obviously where your owner id is set. When searching or accessing your family recipes...This is a little different. We now have to determine exactly which recipes you have permission to see.

Hey remember that ACL thing in MySQL? Haha, yea you know the one where you have all those tables and numbers and JOINs? Kiss that one bye bye. The access rules are extremely simple for Family Spoon and without some awkward tree table to screw up, they're also more reliable. It's a faster query too. In fact, most the time just for loading the recipe page document you have all the information you need to determine access. So why would we want to then jump through a bunch of hoops to determine access? Be efficient...And should access be allowed, you also now have the data to display. One query. The user's data is cahced in the session. So... What more can I say? Yes, there's a bit more to it and not all situations allow for that. Family Spoon also uses Facebook and you can share recipes with your Facebook friends. So that's another call to somewhere else, but for the most part, we aren't really talking about a lot of strain on the database.

Family Spoon can get a bit more specific too. You can share recipes with specific individuals. Again, an array of ids is stored and a simple $in takes care of it. Or, returning the entire field and using PHP's in_array() function takes care of it. How many people can a recipe stored with? How many ids could be in there? A lot less than the number of rows that would be in ARO and ACO tables. So, we're efficient too when using MongoDB. Not just fast, but efficient.

The really "rapid" part that MongoDB helps us with is when it comes back to the code. Building forms and saving the data down to the collection is far easier than it would otherwise be with an ACL system inside a relational database. There's less being written to the database and less code to do it. Sure, your framework could give you a wonderful API for working with permissions, but the minute you need to deviate from that, you could be opening a can of worms.

Ratings & Voting
Along the same lines is something like ratings. On recipes we can have a star rating. Anything that we need to tally and aggregate or increment even. MongoDB makes this easy. So ratings alone aren't a show stopper for any database or system, but when you then say, "Ah yes, but I only want each person, user, IP, to vote one time only." Then you have to put your thinking cap on. You're going to end up with some more JOINs... Also, more code. You're going to have to make the query and check to ensure that some person isn't trying to vote twice. With MongoDB, you can use the $set call and simply keep adding these ratings to a field and make the key equal to the user id or IP and the value their rating. If they vote again, they aren't adding a new rating, they are simply updating their own. Ta da. Now we've reduced the number of tables, data, and code required to setup a rating system. I've actually posted another blog entry about this here.

Conclusion
So, for many reasons, using MongoDB has made building Family Spoon a much quicker process. Trust me, I actually did it both ways. Family Spoon has been completely re-built. While the other version never really appeared out in the wild, it did exist (and was online). Previously, Family Spoon was built using the CakePHP framework with MySQL for the database. So I can most definitely tell you the differences in the amount of code and time planning between the two versions. I rebuilt the site much faster not just because I knew what I was going to do the second time around, but also because I had less to think about when it came to the database schema. Yes, you need to be aware of "schema" and you can't go hog wild, but you also get more forgiveness and MongoDB works with you to solve your problems. It's very flexible. It's not something that you need to work around, it's something that you get to work with. Anytime that you have a situation like that as a developer, your day is going to be much more happy and productive.

Do I hate MySQL? No. Definitely not, years and years worth of use and relationship aren't easily erased. It's comfortable and it's familiar and just fine. In fac,t still preferrable under certain circumstances. Just not mine. Let's make this clear, I am the only person developing Family Spoon. Just one developer and, without using MongoDB and the Lithium framework, I can tell you that it would have taken me a lot longer to not only get the site online. It also would have taken longer to provide some of the advanced features that people will be looking for (filtering, searching, etc.). Both MongoDB and Lithium not only served me with rapidly getting the project up and working (with all the core functionality that I needed) but these technologies will also be serving me into the future with rapidly being able to grow the site and add new features.

Last, so this didn't interrupt your reading pleasure, this is an example document from the database. It's just an example and not complete, but I wanted to highlight the schema and how things like tagging and ingredients worked. Each ingredient is broken out and that's going to go a long way for filtering. Filtering without JOINs.

{
  "_id": "4d6564cb9bae6c1066000000",
  "created": "Wed, 23 Feb 2011 11:49:31 GMT -08:00",
  "directions": "Place the chicken in the crockpot. ...",
  "ingredients": [
    {
      "ingredient": "boneless chicken breasts",
      "quantity": "3",
      "measurement": "lbs"
    },
    {
      "ingredient": "milk",
      "quantity": "1",
      "measurement": "cup"
    },
    {
      "ingredient": "salt"
    },
  ],
  "modified": "Fri, 11 Mar 2011 09:46:31 GMT -08:00",
  "owner_id": "xxxxx",
  "owner_ids": [
    "xxxxx",
    "123456"
  ],
  "page_type": "recipe",
  "public": true,
  "public_rating": {
    "127-0-0-1": "4",
    "255-255-255-0": "5"
  },
  "published": true,
  "serves": "",
  "share_with_friends": false,
  "tags": [
    "chicken",
    "crockpot",
    "broccoli"
  ],
  "time": {
    "prep": {
      "amount": "15",
      "unit": "minutes"
    },
    "cook": {
      "amount": "4-6",
      "unit": "hours"
    }
  },
  "title": "Crockpot Broccoli Chicken",
  "url": "crockpot-broccoli-chicken"
}