Niklas Närhinen - blog

TDD is hard

2011-12-10T00:00:00-08:00

TDD is hard

Today I saw this tweet in my Twitter timeline:

TDD is not hard.Good software is hard.TDD is easy tool that helps write good software.
— Rob Conaway (@robconaway) December 10, 2011

I have tried to experiment with TDD a lot and usually failed miserably. Almost always the reason has been scoping. I don't know what to mock and what to integrate.

My experiments with BDD (or functional TDD with Gherkin or whatever you want to call it) has however been much more successfull. For me it feels a lot more intuitive to write tests ahead for much more concrete behavior or feature, than to know which methods my class is going to have and what they do.

My programming divides to two main parts:

a) create a resource and save it to database, fetch it and modify it (sometimes even delete)
b) show the resource in UI and add some visual properties and behaviors to it

How should/could I apply TDD to this? Seems there is no point of writing tests ahead for simple setters and getters. And for the save method or such it feels like mocking a database (or http-) connection isn't feasible, as then it just would be "when save() is called, an attemp to persist should happen", not that it would check if it actually goes to the datastore..

For the UI part writing (functional) tests with some automation tool is always an option, sometimes even a good one. However, for me doing UI coding is more intuitive doing it "visually" than to write some unit tests for ui behavior. Also to this I have found Gherkin and "BDD"-tools be more helpful than "traditional" TDD constraints. For me writing "Given I have 5 products in my database, When I open product listing, Then I should see 5 products And the first one should be 'Product 1'" is a intuitive thing to do. However, I don't always do the actual testing with any automation tool or such. I just write the tests against my ViewModels (I'm using MVVM as the approach..)

Every tutorial on TDD and every TDD workshop I have attended has had this naive Minesweeper, Calculator or Poker game example as the "problem domain". I have never had to write a calculator, poker gamer or minesweeper in the real world.. After each of these tutorials/workshops them I have been enthuastic to try TDD at my own projects. Then I always hit the same barrier: where to start.. Those simple examples are for already solved problem domains. Everyone knows how a poker game works, so the tests are simple to come up with. Hovewer if you need to implement something like "Product listing for an web-app", I know only (or hopefully at least..) that I need to have a product resource with some properties, that I need to have a way to query the database and that I need to show it somehow to the end-user. Then we are again in the situation a) from above..

Now the big question is, am I not doing good software if I find doing TDD so hard? Leave a comment, ping me on twitter or drop an email..

5 rules for writing a web framework

2011-10-24T00:00:00-07:00

5 rules for writing a web framework

tl;dr; - click here

#1: Learn from others

Make yourself familiar with existing products. To really learn the problem domain, you have to use other frameworks to learn their pitfalls and strengths. Study the code bases of leading projects (if open source) to learn about their structure. Search through their issue trackers and mailing lists for potential problems users are experiencing that their architecture prevents overcoming.

Fork an existing project

Why do the ground work from scratch? Take an (open source) project and build on top of that. Ditch stuff you don't need, add stuff you need and think you do better. Web frameworks are normally complex systems and writing every line of code by yourself almost certainly makes sure the project never gets even the initial release.

#2: Document from day one

A web framework without documentation is a web framework no one (except you) is going to adopt. You need tutorials, example projects, and hard core per-feature documentation. Implement best practices for your framework and document them.

Writing documentation makes you also look outside of the box. When writing documentation to others, you discover better their needs. Writing a web framework for yourself without documentation almost certainly makes sure it has features that only add value for yourself and contains blocking issues for others.

You could even try Behavior driven development. You get some feature documentation while you add tests for your project. It also makes you discover when you are breaking backwards compatibility, one of the things you really will have to consider while writing a web framework for others. Breaking BC means your users won't update even though you would later discover security issues. And because they can't update, you have to backport security fixes to old releases and believe me: that's hell.

#3: Utilize the web

There are standards for the web, use them.

For me, the web equals HTTP. And I (almost) start to cry every time I see HTTP abused. Supporting only GET and POST (or even worse, only POST) or mixing the semantics of the verbs is unforgivable. Not only do you break everything that makes the web so awesome, but you also guide others (your framework users) to break it more. Make yourself familiar with specifications: HTTP status codes, standard request and response headers etc, and learn their semantics.

The web is a great platform to build on top of and there are plenty of users that already know how the web works (or should work, at least..). Don't let them down by not using the full power and standards of it.

(A great starting point is to learn ~everything about REST and HATEOAS)

#4: Avoid magic

As a web dev I always avoid frameworks that adverts them self like: "Make a blog in 5 minutes". If I wanted a blog in five minutes, I would open an account on blogspot or tumblr..

Web frameworks are meant to do the heavy lifting for the web dev. To unleash the power of the web dev to actually add value. This however doesn't mean the heavy lifting should involve magic. Having magic built-in your framework usually (or always) results in a product that is suitable to do just the magic stuff it was meant for, not generalizing for wider use.

A good example of this are form helper libraries from various frameworks. They seem great at first glance; "Wow, I just write five lines of php and get a form with 20 inputs!!". Nothing wrong with this, as long as it suits your use case. Binding a model to a form to handle all validation and stuff can be handy and desired. However, in my experience, in every project there comes this situation when the helper gets in the way. And usually it cannot be just left out. And then you end up using hours to workaround the limitations of the magic library that was initially supposed to help you..

Concentrate on things that really add value for the web developer; implement good structure for projects, help them utilize the web, add some nice helpers here and there but don't make them tie their business logic in to the internals of your framework.

#5: Don't write one

This is the most important rule. (And the one I've been following for years after I first discovered it..)

The world (or at least the web) is full of web frameworks, some of them are even good. Writing a web framework is hard and you really have to know what you are doing. If your motivation is "Existing frameworks are too complicated to use" you definitely aren't sharp enough to write one yourself.

That's all for this time, if you disagree with me or want me to clarify things, please don't hesitate to contact me.

An easy way to update propel from github

2011-08-12T00:00:00-07:00

An easy way to update propel from github

We were using svn:externals to keep Propel ORM (and other libs) up-to-date in our projects. (You should always have your libraries project-local).

Recently Propel ORM switched to GitHub.

I wanted to keep my development process lean and highly automated, that's why I wrote this bash-script to update Propel to the desired tag.

Hope it will be useful to others also (use at own risk..)

Streaming output with a callback in Agavi

2011-06-23T00:00:00-07:00

Streaming output with a callback in Agavi

Earlier I wrote about serving large XML contents with PHP. Today I wanted to use that knowledge when outputting from Agavi. By default you would output XML like:

<?php
class Products_ListSuccessView extends MyProductsBaseView
{
    
    public function executeXml(AgaviRequestDataHolder $rd)
    {
        $dom = new DOMDocument();
        ........
        return $dom->saveXML();
    }
}
?>

Changing this according to my previous blog post about XML isn’t just replacing the DOMDocument with XmlWriter as AgaviWebResponse sends headers only after the view’s execute-method has completed.

So I had to subclass AgaviWebResponse to support returning Closures from the view:

<?php
class MyWebResponse extends AgaviWebResponse
{
    public function sendContent()
    {
        if ($this->content instanceof Closure) {
            $func = $this->content;
            $func();
            return;
        }
        parent::sendContent();
    }
}
?>

After adding the class to autoload.xml and factories.xml you can do the following:

<?php
class Products_ListSuccessView extends MyProductsBaseView
{
    
    public function executeXml(AgaviRequestDataHolder $rd)
    {
        return function() {
            $writer = new XmlWriter();
            $writer->openURI('php://output');
            .......
            $writer->flush();
        };
    }
}
?>

Wasn’t that simple? :)

Bind NHibernate objects to WPF DataGrid

2011-02-27T00:00:00-08:00

Bind NHibernate objects to WPF DataGrid

Lately I have felt the need to try out Windows Presentation Foundation. I have been using WinForms with Ado.NET providers but it just feels I’m coupling too much business logic with the user interface.

Windows Presentation Foundation seems very promising for implementing the Model-View-ViewModel pattern. Specifically when combined with an ORM, like NHibernate. Even though this post isn’t implementing the pattern properly, it should give you the direction technically :). I will be using NHibernate, Fluent NHibernate and SQLite for the examples.

The problem with WPF and NHibernate is, that there isn’t an out-of-box two-way binding for the components. WinForms and Ado.NET classes can be coupled so that retrieving and persisting the data can be accomplished with just a few mouse clicks in Visual Studio Designer. For WPF and NHibernate you need some more manual work, but personally I think it’s worth it, as the resulting code is much cleaner and all in your own hands.

Part1: The Models

Customer

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace FluentNHibernateDataGridMappingTest.Model
{
    public class Customer
    {
        public virtual int Id { get; set; }
        public virtual string Name { get; set; }
        public virtual string Email { get; set; }
        public virtual string Phone { get; set; }

        public virtual IList Addresses { get; set; }

        public Customer()
        {
            Addresses = new List();
        }

        public virtual void AddAddress(CustomerAddress address)
        {
            address.Customer = this;
            Addresses.Add(address);
        }
    }
}

CustomerAddress

Actually this class isn’t used in the binding, but it demonstrates nicely the Fluent NHibernate mappings :)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace FluentNHibernateDataGridMappingTest.Model
{
    public class CustomerAddress
    {
        public virtual int Id { get; set; }
        public virtual string Street { get; set; }
        public virtual string Zip { get; set; }
        public virtual string City { get; set; }
        public virtual string Country { get; set; }

        public virtual Customer Customer { get; set; }
    }
}

Part2: The ModelViews

CustomerView

So basically, the purpose for this class is to implement the INorifyPropertyChanged interface, providing the necessary triggers for two-way binding..

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ComponentModel;

namespace FluentNHibernateDataGridMappingTest.Model
{
    public class CustomerView : INotifyPropertyChanged
    {
        private Customer _customer;

        public Customer InnerCustomer { get { return _customer; } }

        public CustomerView()
        {
            _customer = new Customer();
        }

        public CustomerView(Customer customer)
        {
            _customer = customer;
        }

        public virtual int Id
        {
            get { return _customer.Id; }
            set
            {
                _customer.Id = value;
                NotifyPropertyChanged("Id");
            }
        }
        public virtual string Name
        {
            get { return _customer.Name; }
            set
            {
                _customer.Name = value;
                NotifyPropertyChanged("Name");
            }
        }
        public virtual string Email
        {
            get { return _customer.Email; }
            set
            {
                _customer.Email = value;
                NotifyPropertyChanged("Email");
            }
        }
        public virtual string Phone
        {
            get { return _customer.Phone; }
            set
            {
                _customer.Phone = value;
                NotifyPropertyChanged("Phone");
            }
        }

        #region INotifyPropertyChanged Members

        public event PropertyChangedEventHandler PropertyChanged;

        protected void NotifyPropertyChanged(string propertyChanged)
        {
            if (PropertyChanged != null)
                PropertyChanged(this, new PropertyChangedEventArgs(propertyChanged));
        }

        #endregion
    }
}

Part3: The Collection

This is actually maybe the main part of this “tutorial”. You need a collection, that notifies the user interface about modifications in the collection and the items in the collection.

A suitable base is ObservableCollection. Basically, we just bind bind an event handler for the event raised by INotifyPropertyChanged for each item in the collection.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Collections.Specialized;

namespace FluentNHibernateDataGridMappingTest.Model.Collection
{

    class EcObservableCollection<T> : ObservableCollection<T> where T : INotifyPropertyChanged
    {
        public EcObservableCollection() : base()
        {
            this.CollectionChanged += 
                new NotifyCollectionChangedEventHandler(EcObservableCollection_CollectionChanged);      
        }

        void EcObservableCollection_CollectionChanged(object sender, NotifyCollectionChangedEventArgs e)
        {
            if (e.Action == NotifyCollectionChangedAction.Add) {
                foreach (T item in e.NewItems)
                    item.PropertyChanged += new PropertyChangedEventHandler(item_PropertyChanged);
            }
        }
        public EcObservableCollection(List<T> items) : this()
        {
            foreach (T item in items) {
                this.Add(item);
            }
        }

        void item_PropertyChanged(object sender, PropertyChangedEventArgs e)
        {
            EcObservableCollectionItemChangedEventArgs<T> args = 
                new EcObservableCollectionItemChangedEventArgs<T>();
            args.Item = (T)sender;
            ItemChanged(this, args);
        }

        public event EcObservableCollectionItemChangedEventHandler ItemChanged;

        public delegate void EcObservableCollectionItemChangedEventHandler(object sender, 
                                                            EcObservableCollectionItemChangedEventArgs<T> args);
    }

    class EcObservableCollectionItemChangedEventArgs<T> : EventArgs
    {
        public T Item { get; set; }
    }
}

Part4: The user interface

MainWindow.xaml

This code also demonstrates how you can make the row header to show the id of the item.

<Window x:Class="FluentNHibernateDataGridMappingTest.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="504" Width="821" DataContext="{Binding}">
    <Grid>
        <Grid.Resources>
            <Style x:Key="RowHeadersWithId" TargetType="{x:Type DataGridRowHeader}">
                <Setter Property="Content" Value="{Binding Id}" />
            </Style>
        </Grid.Resources>
        <DataGrid AutoGenerateColumns="False" 
                    Margin="12" 
                    Name="dataGrid1" 
                    AlternatingRowBackground="AliceBlue" 
                    RowHeaderStyle="{StaticResource RowHeadersWithId}" 
                    HeadersVisibility="All" 
                    CanUserAddRows="True">
            <DataGrid.Columns>
                <DataGridTextColumn Binding="{Binding Name}" Header="Name" Width="*"/>
                <DataGridTextColumn Binding="{Binding Email}" Header="Email" />
                <DataGridTextColumn Binding="{Binding Phone}" Header="Phone" />
            </DataGrid.Columns>
        </DataGrid>
    </Grid>
</Window>

Part5: Make it all work together

NHibernate mappings

Here we use Fluent NHibernate bindings, you could of course use the traditional mappings, if you would want to.

CustomerMap.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using FluentNHibernate.Mapping;

namespace FluentNHibernateDataGridMappingTest.Model.Mapping
{
    public class CustomerMap : ClassMap
    {
        public CustomerMap()
        {
            Id(x => x.Id);
            Map(x => x.Email);
            Map(x => x.Name);
            Map(x => x.Phone);
            HasMany(x => x.Addresses);
        }
    }
}

CustomerAddressMap.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using FluentNHibernate.Mapping;

namespace FluentNHibernateDataGridMappingTest.Model.Mapping
{
    public class CustomerAddressMap : ClassMap
    {
        public CustomerAddressMap()
        {
            Id(x => x.Id);
            Map(x => x.City);
            Map(x => x.Country);
            Map(x => x.Street);
            Map(x => x.Zip);
            References(x => x.Customer);
        }
    }
}

Database configuration: The Fluent NHibernate way

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NHibernate;
using FluentNHibernate.Cfg;
using FluentNHibernate.Cfg.Db;
using NHibernate.Cfg;
using System.IO;
using NHibernate.Tool.hbm2ddl;

namespace FluentNHibernateDataGridMappingTest.Core
{
    class Database
    {
        public static ISessionFactory CreateSessionFactory()
        {
            return Fluently.Configure()
                .Database(SQLiteConfiguration.Standard.UsingFile("test.db"))
                .Mappings(m => m.FluentMappings.AddFromAssemblyOf<App>())
                .ExposeConfiguration(BuildDataBase)
                .BuildSessionFactory();
        }

        private static void BuildDataBase(Configuration conf)
        {
            conf.SetProperty(NHibernate.Cfg.Environment.ShowSql, "true");
            if (!File.Exists("test.db"))
                new SchemaExport(conf).Create(false, true);
        }
    }
}

The user interface initialization and data binding: MainWindow.xaml.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using NHibernate;
using FluentNHibernateDataGridMappingTest.Core;
using FluentNHibernateDataGridMappingTest.Model;
using System.Collections.ObjectModel;
using FluentNHibernateDataGridMappingTest.Model.Collection;
using System.Collections.Specialized;

namespace FluentNHibernateDataGridMappingTest
{
    public partial class MainWindow : Window
    {
        private ISessionFactory sessionFactory;

        private EcObservableCollection<CustomerView> customers;
        public MainWindow()
        {
            InitializeComponent();
            sessionFactory = Database.CreateSessionFactory();
            using (var session = sessionFactory.OpenSession()) {
                var criteria = session.CreateCriteria<Customer>();
                customers = new EcObservableCollection<CustomerView>();
                foreach (Customer c in criteria.List<Customer>())
                    customers.Add(new CustomerView(c));
            }
            customers.CollectionChanged += 
                new NotifyCollectionChangedEventHandler(customers_CollectionChanged);
            customers.ItemChanged += 
                new EcObservableCollection<CustomerView>.EcObservableCollectionItemChangedEventHandler(customers_ItemChanged);
            dataGrid1.ItemsSource = customers;
        }

        void customers_ItemChanged(object sender, EcObservableCollectionItemChangedEventArgs<CustomerView> args)
        {
            using (var session = sessionFactory.OpenSession()) {
                session.SaveOrUpdate(args.Item.InnerCustomer);
                session.Flush();
            }
        }

        void customers_CollectionChanged(object sender, System.Collections.Specialized.NotifyCollectionChangedEventArgs e)
        {
            using (var session = sessionFactory.OpenSession()) {
                switch (e.Action) {
                    case System.Collections.Specialized.NotifyCollectionChangedAction.Remove:
                        foreach (CustomerView c in e.OldItems) {
                            session.Delete(c.InnerCustomer);
                            session.Flush();
                        }
                        break;
                    case System.Collections.Specialized.NotifyCollectionChangedAction.Add:
                        foreach (CustomerView c in e.NewItems)
                            session.Save(c.InnerCustomer);
                        break;
                    default:
                        foreach (CustomerView c in e.OldItems)
                            session.SaveOrUpdate(c.InnerCustomer);
                        break;
                }
            }
                
        }
    }
}

Conclusion

So, with my code examples, it should be easy to figure out how to build quickly different views for your models. The observablecollection I created, is highly reusable, as it takes whatever modelviews you create, as long as they implement the INotifyPropertyChanged interface.

In a nutshell, the steps needed to get two-way databinding with WPF and NHibernate to work together:

Write your Models with appropriate ModelViews
Map them to database
Write (or copy, or search with google) an appropriate Collection
Use Visual Studio Designer to implement the user interface
Add some code to fetch the items from db with NHibernate and attach some event handlers
Enjoy.

If you have anything to add or ask, please leave a comment in this blog or contact me directly via email: niklas@narhinen.net

Creating and Serving large XML files - DOMDocument vs. XmlWriter

2011-01-15T00:00:00-08:00

Creating and Serving large XML files - DOMDocument vs. XmlWriter

Handling big amounts of data between a client application and a REST service can sometimes be tricky, especially when things should be responsive. I did a small benchmark between the built-in tools in PHP: DOMDocument and XmlReader.

DOMDocument

DOMDocument is maybe the most used method for XML handling and creation in PHP. The approach used in DOMDocument, is to load nodes into a DOM, which then can be queried and/or represented as a string.

Here's a sample script I used for benchmarking:

<?php
$start = microtime(true);
header('Cache-Control: no-cache, must-revalidate');
header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header('Content-type: text/xml');
$doc = new DOMDocument('1.0', 'utf-8');
$doc->formatOutput = true;
$products = $doc->createElement('Products');
for ($i = 0; $i < 1000000; $i++) {
    $element = $doc->createElement('Product');
    $element->setAttribute('Id', $i);
    $element->setAttribute('Value', md5(uniqid()));
    $products->appendChild($element);
}
$doc->appendChild($products);
echo $doc->saveXml();

$stop = microtime(true);
$seconds = $stop - $start;
echo "Start: " . $start . PHP_EOL;
echo "Stop: " . $stop . PHP_EOL;
echo "Seconds: " . $seconds . PHP_EOL;
echo "Memory peak: " . memory_get_peak_usage() / 1048576 . 'MB' . PHP_EOL;
?>

This script was served via Apache and downloaded with wget.

Here are some stats:

The average time to create the xml and output it, was somewhere near 30-35 seconds.
Memory consumption average was 120-130MB.
The average time to complete the whole wget request, was somewhere between 35-40 seconds.

XmlWriter

XmlWriter is not that well known amongst PHP-developers. It's basically a wrapper for libxml's xmlWriter API. With XmlWriter you can (in theory, at least) flush the contents to the output-pipe before you have even processed all data on your server. That way you can start processing the data clientside before the download is even complete. The processing and 'editing' capabilities are a bit more limited than with DOMDocument, and you need to handle the document hierarchy yourself.

Here's a sample I wrote:

<?php
$start = microtime(true);
header('Cache-Control: no-cache, must-revalidate');
header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header('Content-type: text/xml');

$writer = new XmlWriter();
$writer->openURI('php://output');
$writer->startDocument('1.0', 'utf-8');
$writer->startElement('Products');
$writer->setIndent(true);
for ($i = 0; $i < 1000000; $i++) {
    $writer->startElement('Product');
    $writer->writeAttribute('Id', $i);
    $writer->writeAttribute('Value', md5(uniqid()));
    $writer->endElement();
}
$writer->endElement();
$writer->flush();

$stop = microtime(true);
$seconds = $stop - $start;
echo "Start: " . $start . PHP_EOL;
echo "Stop: " . $stop . PHP_EOL;
echo "Seconds: " . $seconds . PHP_EOL;
echo "Memory peak: " . memory_get_peak_usage() / 1048576 . 'MB' . PHP_EOL;
?>

This script was as well served via Apache and requested with wget.

Here are some stats on XmlWriter performance:

The average time to create the xml and output it, was somewhere near 16-20 seconds
Memory consumption average was somewhere between 500-700 kb
The whole wget request took only the same 16-20 seconds

And the winner is: XmlWriter

So to sum things up. If the capabilities XmlWriter offers are enough for you, you should definitely be using it. With one million 'product' items, the times to process the data and get the contents to your client will be split in half comparing to DOMDocument. And what's most important: you will be consuming only a fraction of the memory, which then leads to better performance and scalability.