Charity Project: Postmortem

Posted on 03. May, 2010 by DavidR in DavidR, Programming, Projects, blog

(Click for full-size)

Recently I constructed an ID card management app for a charity near to where my parents live. The charity is called  Thera and whilst I’m not affiliated it with it in any way or form I hope they don’t mind me mentioning them here for educational purposes.

Please note, however, that this is a purely technical / ‘shipping’ post-mortem. Thera have not explicitly selected my application for use yet – and they may well not use it at all. They’re reviewing multiple solutions (both commercial and non-commercial) but let’s hope they select the one developed entirely for free as a personal project: Because that’s exactly what it was. Whilst doing this fulfilled an assignment for my course, the assignment is completely formative. It doesn’t affect my grades or anything. Realistically I took this on because it was interesting.

So… What is it?

Previously (or currently, in fact) issuing ID cards at Thera was a manual process. The cards themselves turned out O-K but the workload involved creating them was huge – and not an effective process considering the advent of databases, automatic systems etc.

I created a small application to automate the card issuing process in about ~3 weeks. It couldn’t automate everything (you have to put employee details in it, for example) but after that it can filter and produce cards with the press of a button (with sequential serial numbers and everything)

What went right

• Spending a large portion of available time on the low-level model and testing. Any programmer knows that wiring logic directly behind a UI (‘magic button’ anti-pattern, as it is commonly known) is a cardinal sin. With this in mind, for the largest amount of time (about ~2 weeks) the project was purely data and classes. No UI. Just the model and a bunch of tests. And this worked brilliantly. Why? Because when I came to make the UI, I already knew the stuff underneath was rock-solid. It made things easy to debug, and also easy to make changes.

  You may ask “Make changes, in what way?”. A very good example is v1.1 which I just finished. One of the major irritations was that I hadn’t fully fleshed out modification detection in v1.0. Every time you closed a form (whether you changed anything or not) it would ask whether to save changes. If my logic were a bit more tied up with the UI, this could have been seriously difficult to fix (or at least, any solution would be difficult to maintain). With the well separated model, doing this was a piece of cake – especially since every database entity inherited from a single class. I added a modification counter and checker into the base class (~20 lines) and in 5 minutes it worked. It was fantastic. This also leads me onto my next point….

• ‘Rolling my own’ turned out to be a good decision. Generally, creating your own custom anything is a bad idea, unless you’re in new, unknown territory. Which I wasn’t – my application was database-backed (SQLite) so everything had a pre-made-solution that could be found somewhere. The snag is, every solution to storing POJOs in databases (Plain-Old-Java-Objects to the uninitiated - i.e. instances of class – so painless syncing between what you’ve got the code doing and what is stored in the database) was either annoying and required additional files outside of the code (which I really didn’t want) or had issues with nested objects

  ‘db4o’ was a top contender for a while – it was incredibly easy to sync in and out of the database without any external mapping or anything. But what bit it was that last point. Nested objects. It handles them, sure. But it’s a pain. You sort of have to be ‘aware’ of what depends on what and sort out the order yourself. That’s great, but since this this application was very likely to grow quickly (as I found out new things Thera wanted) this was not a good solution for maintainability. So I rolled my own…

  My base class ‘FieldMapObject’ completely redefines how derived classes declare variables. When they’re built, they call ‘delcare()’ with a variable name and type. The FieldMapObject remembers and stores it in a map along with type data and an initial value (null).

  Now this probably seems like a stupid solution, but actually it had many unexpected upsides. One of which is the modification counter in the above point. Putting that directly into FieldMapObject meant it instantly worked for any entity that derived from it. No modifications needed. And debugging was made extremely easy. With everything being programmatically ‘known’ by the class, it could automatically spit out printed lists of each variable and its current value. It even detected nested FieldMapObjects and spat out their data – appropriately ‘nested’ in the text.

  Along with this, I built ‘DataServer’ a class which derived from the raw Storage class I had built (the raw storage simply ran SQL queries via JDBC/SQLite and built the initial tables when it detected the DB as empty). DataServer turned out really nicely – because everything inherited from FieldMapObject (well, DatabaseEntity in reality, but that was built on FieldMapObject) it had simple routines that covered any object – put() get() discard() and update(). In fact, update() was private – put() could automatically detect objects with a valid DB rowid (thanks to FieldMapObject) and update() when appropriate. get() and put() even cater for nested objects as mentioned earlier. By running through the declare()d fields in the FieldMapObject, it detected the dependencies and put() them first (which chained nicely with deep nested hierarchies).get() ended up being even smarter in fact – later in development every entity was given a ‘weight’ (in memory). If DataServer was set to ‘shallow load’ it would automatically skip nested fields that were deemed too ‘heavy’. (Keep in mind the DB was storing images – photos of the employees. So that’s really where it came into play. You don’t want a list of a 1000 employees causing a 1000 nested photos to be loaded)

  Overall, rolling my own field mapping/dependency detection turned out great. You may think “this guy’s an idiot – he satisfied his own technology ego but doing this didn’t help the project” – in which case you’d be wrong. Spending this extra time on fetching and putting data made building the UI an absolute breeze. I mean it too – it was incredibly easy. There were a few rough edges to work around, sure, but 99% was extremely nice. I think making this system in this way made the end product far easier to produce – and as a result, less buggy and far more stable.

• Hand-crafted Swing code kicks the crap out of generated code. If I discarded everything I did on the project and started again for some reason, this one thing is the aspect I’d keep – and I will continue to keep on subsequent projects. Swing was kind of an unknown to me. Having used non-Java UI toolkits I was initially scared of hand coding UIs in it. Because normally, UI toolkits are horrible to hand code with. But eventually I bit the bullet and tried… and it was great. Granted, the UI is not the best designed thing on the planet. But it works, and it is easy to maintain. For one thing, I can read the code. It doesn’t depend on the stability of a poorly written UI designer either. It’s in the code. The only part of this I was so sure about – and I guess this applies beyond Java – is what is the correct way to lay out the reams of code that setup the UI? I split mine into functions which set up each respective part, but it still seems an unpleasant solution. Is there some accepted way to do this? But any way, using Swing manually was a good idea.

What went wrong

• Not thinking about how I would handle [UI] tables. The mechanism of dependency matching I used worked really nicely for a small group of objects. What I completely ignored (until I started developing the UI) is how I would get huge lists of employees into memory fast enough and efficiently enough to show in a table. My initial attempt was a miserable (and stupid) failure: get()ing every employee and dumping it into a List<>. Suffice to say that was slow as hell past ~1000 employees (Thera have about 2K employees by the way)

  I never really did ‘fix’ this problem. That is, I worked around it. I coded an SQL query and a matching function in DataServer that returned the employees as a raw ResultSet. My JTable manually stepped through the result set and used the columns it wanted. This worked nicely from a speed point of view, but it was a stupid solution. It broke the niceness of the dependency and object system, meaning it will no doubt become hell to maintain if I don’t refactor it. The more horrifying thing is…. it made it into 1.1 as well. And… I built a new filter system which also assumes ResultSets. So I think that’s next on the hit list. Retrofit speed fetching into the object system without compromising it. It’s going to be difficult, to say the least

• Not realising I would need to update databases at some point, as well as building them. One of the first things I had going on the project was a system to detect what tables existed in the database, and build the ones that were missing (so for example, building the table structure automatically on a new file). It worked great. What I did not do at this point (and what I still haven’t done) is to take stock of the fact that eventually, I WILL need to change the tables on a file that is already full of data. And I won’t be able to just CREATE TABLE. I will have to alter it. What’s more is that my format is lacking any kind of versioning information. The application has no idea what version of tables are in the database. Ack. I think I’ll be targeting this for v1.1.1 if Thera decide to use it
• Relying too much on buttons in v1.0. In the first version, there were tons of buttons. View Card, Issue Card, Queue All, Dequeue All, Queue Selected etc. It was confusing unless using a large screen (which padded them out horizontally). For v1.1 I switched to a menu bar (as per the screenshot). Which worked great – all the Queue functions were kept together and so were the filters. The problem is, v1.0 completely lacked any menu bar, and the sudden change was both jarring and confusing for anyone but myself. I suppose the lesson here is to think ahead with what you’re likely to add – because after all, I only added a menu to make space for new commands (no more space for buttons). This comes to lack of foresight I suppose

Stats

• Total physical Source Lines Of Code (SLOC): 4602  (excluding field mapping library)
• v1.0 completed: 23/04/2010
• v1.1 completed: 01/05/2010
• Libraries: Swing, JCalendar control, SQLite JDBC, Swing layout extensions
• Subversion checkins/commits: ~150   (~3 week development time from start to v1.0)

Designing and prototyping

Posted on 24. Feb, 2010 by DavidR in DavidR, Programming, Projects, blog

I’m really enjoying designing and building an application that I’d say is more ‘enterprise oriented’. Purely because enterprise software actually has a chance of being ‘perfect’ – decent design, automated testing etc. Whereas games can never be this perfect – because fun can’t be designed that easily. It’s also not very easy to test for (at least with say, JUnit). General software just has to… work. Which is fine by me

Currently I’m designing / prototyping ‘codename Spitfire’. It’s an idea I had during my ~2 years developing and maintaining a database system for a local charity (Voluntary Action Rutland). Suffice to say it is database related. And I’d also love for to be able to drop in the completed system as a replacement for the one I developed previously (with some alteration. And assuming it gets finished).

Back to my notebook for now….

My notebook. I love arrows.

Both Java and University are…. awesome

Posted on 11. Feb, 2010 by DavidR in DavidR, Personal/Life, Programming, Projects, blog

Yes, it has been far too long since my last post. I think the main problem was focus: leaving home for (Durham) University, and all focus on my side-projects was lost – rightfully so, in order to learn the ropes around uni. and get settled in on my course.

For a long time, I simply ignored coding and my projects – at least in my spare time. Very quickly things became dull (yes, that makes me a sad person). But I realise now that one of the great parts of having such an involved/interesting hobby/side-project is that it fills the void when everything else isn’t very good or interesting. Rather than thinking about some benchtest I failed, or how long the term is, I can put all that to the back of my mind and hammer out some code. It gives a focus when everything else has turned to shit, effectively

Well back on topic: I forgot about all that. So I neglected my code and my projects when I first got to uni. It made for a fun, but far more depressing time. When things were looking bad, there was no light at the end of the tunnel. I’m telling you… coding is a drug. It makes everything great…

So at the first chance I got, I immersed myself in code again. And now things are back to normal. I’ve also sorted out what I’m doing code and project wise, which is why I’ve made a return to the blog (without code, what’s the point in blogging on here? It’s the entire topic of discussion!)

Java

I was initially sceptical about Java. Even in my previous post I wasn’t entirely convinced. But now I’ve ‘seen the light’. Here are a couple of reasons:

• It makes application of decent SoftEng. principles really easy
• You can make variable-argument functions with amazing ease (maybe I missed something, but I found this far from trivial in C/C++)
• It helped me understand generics, even though C++ had templates (and Java generics are in fact more limited than templates. No idea why I never made use of templates properly)
• It’s fast to develop in, but the syntax is not insulting or eye-bleed inducing (and it’s similar to C++)
• It runs fast. The VM can push along some seriously complex things at damn quick pace! It surprised me – a lot.
• You can try new (risky) things, and if they’re wrong, they’ll break immediately rather than appearing correct and breaking further down the line. (This one is difficult to explain but I would often implement something crazy in C/C++ thinking it was correct, only to find some horrible flaw in it weeks or months later. This doesn’t seem possible with Java)

All in all, these reasons mean that unless I need to work with the raw underlying OS for some reason, Java is going to be my weapon of choice for pretty much everything else (and heck, I can even code JNI libs using C/C++ and plug them into my Java apps)

Project status list

• The Hive: Dead for now. (May be resurrected in Java)
• IP summative project (BlackJack): Complete
• IP formative project (‘Wormhole’/Millipede with OpenGL): Work-in-progress. Could be released if it’s good enough / if it doesn’t become property of Durham Uni.
• ‘Project Spitfire’ (server/client ‘dumb terminal’ style database engine): Prototyping