Six Software Products in One Year

I’m finally getting SyncDroid to the point where I can start testing some of my initial assumptions about the project:

• Scanning while idle will not cause perceptible slowdown to the user
• There is enough idle time on a typical desktop session to get useful work done

Without these, I’m stuck writing an ordinary file synchronizer (or a slow one) and that’s not really worthwhile to me.

PySQLite Bad. APSW Good.

One of the big question marks has always been ‘how am I going to store the data on disk?’ Most other programs with this requirement use Berkeley DB or SQLite. As I’ve previously mentioned, SQLite can be fast if you’re intelligent about it. That said, I’ve never used a large SQLite database - anything past about 10,000 rows. Performance issues will show up well before that if you’re doing something silly (like starts-with queries on a string column).

I want to use an external database for the data integrity features. People will unplug USB drives mid-transaction and I don’t want that to trash my entire data store.

I started out using SQLite and common ORMs to simplify the database implementation. Skipping over a few frustrating days of development, I came to the following conclusions:

• Most ORMs will not let you access multiple physical databases at the same time - they implicitly store a reference back to the database. This is no good for me, because I have databases on the host and on any USB drives that get connected. This ruled out Autumn, which was looking beautifully simple.
• Storm’s SQLite transaction behavior is a bit weird - weird enough to be unusable. They do document it, thankfully, and the codebase is very clean, so it’s likely I’ll use it in future (non-SQLite) projects. These problems are mostly because:
• PySQLite completely mangles SQLite’s transaction model. Seriously, guys. SQLite is not typical as far as SQL databases go, but its behaviour is very well documented and not difficult to understand. If you’d just butt out and let SQLite do its thing, you’d be fine, but noooo, you gotta be clever about it…

Aaanyway, this rules out anything based on PySQLite as an ORM. Which means everything. So I wrote my own using APSW (Another Python SQLite Wrapper). This turned out to not be a big deal - the operations I perform on the database are quite simple by design. Once I worked out a schema bug, this worked beautifully.

Performance

My use case for this application is:

• My maildirs (170,000 files)
• A Buildroot repository (240,000 files)
• A new OpenEmbedded repository (80,000 files)
• My git repositories (15,000 files)
• Personal data and records (15,000 files)

I do not intend to be kind to my own software.

Right away, I noticed I was hitting the 120-transactions-per-second issue in SQLite (linked to fsync() and the rotational rate of 7200RPM drives - this is on my laptop). No big deal. It’s still pretty quick.

(I actually have concerns about USB flash drives due to their extremely slow fsync() performance - depending on the filesystem, it might require a few erases/rewrites. Erases are horrendously slow on flash devices, which is why a lot of people are seeing poor write performance on SSDs).

I left it overnight to scan my data. When I got up, we were up to 140,000 files and adding maybe three per second instead of thirty. Grr. (On the upside, Python is not immediately a performance issue. Take that, C++ coders!)

In a past life, I was a performance geek. I heart oprofile. The results:

      samples|      %|
    ------------------
        66522 88.5084 libsqlite3.so.0.8.6
         3963  5.2728 libpthread-2.7.so
         3247  4.3202 python2.5

I took a shower, realized that I hadn’t added appropriate indexes to the tables, and retested:

  samples|      %|
------------------
    81119 99.5264 no-vmlinux
      105  0.1288 python2.5
    CPU_CLK_UNHALT...|
      samples|      %|
    ------------------
           40 38.0952 libsqlite3.so.0.8.6
           30 28.5714 libpthread-2.7.so
           18 17.1429 libc-2.7.so
           17 16.1905 python2.5

The vast majority of the time is now spent in the kernel, not in the SQLite library.

I’m not sure what conclusion to draw from this, yet. Perhaps ’showers are good’. So far, it does not appear that either Python or SQLite will cause performance problems.

In my classic inability to actually focus on a single task for any length of time, I’ve been working on SyncDroid.

I’ve been attacking the tricky areas of data storage and what I refer to as the ‘datapath’ - the chain of events that takes place between a change occuring on a computer and it propagating (across physical space and time) to another computer . I can partly explain why nobody has done this before: it’s really tricky.

Unison (and most other synchronizers) make some simplifying assumptions:

• There is always a master computer and a slave computer
• We only care about what is happening at this exact moment in time
• We can synchronize the times on the two computers when the synchronization occurs
• We can suck up as much CPU and IO time as we like while synchronization takes place

Unfortunately, none of these are true for SyncDroid. They have interesting consequences.

There is always a master and a slave

This makes configuration management really easy: you always look to the master computer. In network-connect hosts, the master is (by definition) contactable, so you can just tell it to update its configuration with any changes made on the slave end.

SyncDroid doesn’t have this luxury. In the case of USB-drive synchronization, the two computers cannot just tell each other about changes. So there’s an interesting sub-synchronization problem: in order to know what data we need to synchronize, we need to synchronize the configuration first.

We only care about a single moment in time

There’s really only one trap if you use this assumption: files might change between the time you detect a change and when you actually synchronize it. This is easy to solve if you take out an exlusive lock on the file-being-synchronized and ensure that it still looks like it did when you scanned it.

SyncDroid cares about lots of points in time. Because it syncs constantly, we have to be very careful about what state we think a file is in versus what state it actually is in. If you’re doing syncs to multiple partners, you have to keep track of all relevant metadata for all partners. If a partner goes away - say the user loses the USB drive - we shouldn’t waste time and resources tracking data that will never be used. And we can’t just rescan things constantly or lock files because that would hurt performance (or make it impossible for users to actually do work). I’m a user of this thing, too, and if it doesn’t perform acceptably, I won’t use it!

We can synchronize computer times easily

On a network-connected synchronizer, this is easy. You run some variation of the NTP protocol between the two hosts and calculate an offset so that you don’t disturb the user’s clock. You can then work out relative change timings and the best course of action.

Because this version of SyncDroid works over USB drives, it can’t synchronize times easily. I get around that with a ‘mountcount’ - it’s just a number that is incremented every time the metadata on a drive is loaded. RAID arrays use the same idea to detect drives that were unplugged from an array and are now out-of-sync with the rest of the array. Each computer using a USB drive can then use the mountcount to determine relative change times without being dependent on the computer’s clock, which will probably be wrong.

The consequence of the mountcount is that multiple access to the metadata is strictly forbidden. This is reasonably easy to ensure and shouldn’t be visible to the user.

We can suck up as much CPU and IO time as we like

This is a big one, and it’s one of the major reasons I started this project. None of the current synchronizers are sensitive to the user. Perhaps I’m a dreamer, but I would like my files to be synchronized without taking a massive hit in PC performance (or battery life).

Unison (as well as most synchronizers) will do exactly  what you tell them to. If you say ’scan for changes’, they will scan right now. If you say propagate changes, they will propagate right now. While they are working, the computer is struggling under massive IO load, and if you have large amounts of data (like I do) that could lead to several minutes where the disk is spinning and you can’t use the computer and you have to sync right now because your plane is leaving but it’s still running and argh I’m going to be late.

SyncDroid has a fairly involved set of priorities to determine under what circumstances it should scan and sync and bookkeep. For example, it has two scanner types: a notification scanner (which uses the OS to determine when files have changed) and a comprehensive scanner (in case SyncDroid wasn’t running and you changed a file). The notification scanner runs all of the time, but if you’re on battery or using the computer, it just remembers the changes in RAM and gets out of the way as quickly as possible. The comprehensive scanner only runs when the computer is connected to power and you’re not using it.  In this way, you get the effect of non-stop change scanning without any perceptible difference to your computer’s responsiveness.

There is a big ‘but’ here, and it’s one of those annoying engineering tradeoffs: if you are not aggressive enough about scanning, you will miss changes (say, the user disconnects their laptop without warning). If you are too aggressive, you’ll slow down the computer. The trick is to find a set of tradeoffs that works well in most circumstances. In those cases that it doesn’t work, you can warn the user and give them an opportunity to fix the problem (by plugging the laptop back into the network for a minute, for example).

Data Storage

And then, there’s the hairy issue of where to put all of this data that we’re collecting. What we have is roughly a parallel filesystem to the one on the disk: for a file, we want to store some metadata. The best way to store this, from a design point of view, would be to store it in the filesystem itself, but this is impractical for a number of reasons (don’t want to change the user-visible view of their data, no filesystem support, differing semantics between systems, and so on).

So we have to create a filesystem within a filesystem. It’s another meta-problem like the sub-synchronization problem in configuration management. I considered doing this in the literal fashion - creating an image on disk with a virtual ext2 filesystem. Instead of files, there would be structs of metadata that I had collected. Licensing issues were, well, issues here, and it would require me to maintain a fairly complicated data access layer. The big technical problem is that contemporary filesystem assume a constant-sized disk, while I wanted to be able to expand and shrink the image size dynamically.

My stopgap solution (while this is all stubbed out in my code) is to use a YAML file. I adore YAML. It is not a high-performance data access layer, however, and it was not designed as such. It’s just very easy to use.

Another option was a custom C data type - or, phrased another way, ‘write my own filesystem’. Lots of effort. Transaction management is a big hairy problem that I don’t want to get into.

Finally, SQLite. I love SQLite - it’s very easy to use and gives you very powerful query functionality. It handles on-disk consistency well and - used sensibly - can be very high-performance.

Many applications, sadly, do not use SQLite in a sensible fashion. (I’m looking at you, Meta-Tracker). Like any SQL database, you can do silly things to it that will absolutely destroy its performance characteristics. A classic in this situation is if you want a directory listing and your rows look like { filename | data }; the database needs to do a ’starts-with’ check on each row in the database because there’s no easy way to index efficiently by filename and retain  simplistic tree-searching operations. This is Really Really Slow.

My current plan is to solve this by implementing a more traditional inode/parent structure within my database schema. I have the big advantage of knowing exactly which operations are necessary (read record by path+name, write record by id, create record by path+name, list children by path) and so can optimise specifically for them.

Joel Spolsky has just posted a thing about the desks at Fog Creek’s new office. It uses an image that is often cited as being ‘ergonomic best practice’, but I can’t imagine a more horrendous way to spend a day in a chair:

(image borrowed from the Nicholas Institute of Sports Medicine and Athletic Trauma)

This seating position completely ignores the balance and weight distribution of a normal human.

What supports the arms and hands? Your muscles do. How do you think that feels after a day of work? Pretty damn awful. Hence, everyone I know rests their shoulders or forearms on an armrest or desk edge, or puts their wrists on a wrist rest. You just can’t hold your arms in that position all day without support.

A significant omission is the mechanism by which you support your hand/arm while using a mouse. Wrist injuries are another whole area which I’ve been lucky to avoid, but solutions include:

• arm weight goes on your hand (on the mouse), which leads to wrist strain and inefficient mousing
• arm weight goes on your wrist (on the desk), which leads to some twisting and nice callouses on your wrist
• arm weight goes on a wrist rest, which puts pressure on the tendons in your forearm (and exacerbates carpal tunnel issues)

I don’t know who they modelled this image on, but that is some damn fine posture they’ve got around the neck. I can’t keep my neck that straight when I’m standing. Maintaining this position requires quite a bit of tension in the rear neck muscles. Unfortunately, most of us also need to work on the desk occasionally and lean forward. This tires the neck muscles. The muscles tire, the head droops forward, the load on the muscles increases, and you go home with a sore upper back.

Again, this image is modelled on some imaginary human with very short upper arms. A significant problem that I have when setting up my own workstations is that to reach a comfortable table height, I have to compromise between squashing my legs (table lower) and having too acute an angle at my elbows (table higher). In fact, a fairly comfortable keyboard location is on my lap (and it solves the support-the-arm-weight problem to some extent).

Points 3 and 4 on the image are where your weight is supported. This just does not work. I cannot find a chair or a position in which this does not lead to numbness or loss of blood flow to some part of my legs. (Keep in mind that I’ve done quite a bit of long-distance cycling purely on my sit bones (3) and not had blood flow issues). The legs have some pretty serious blood vessels running through them and are moving in a normal human. Most people that I know move around through the day, and this helps a lot with blood flow. Keep in mind that sitting in an office chair all day is awfully like sitting in an airplane seat all day.

‘Feet flat on floor’ is another guideline which I find absurd. You can only reach this condition by adjusting the height of your seat. If you put it too high, the legs will be dangling and exacerbate blood flow problems at 3 and 4. If you put it too low, the legs will either be out in front and unsupported, or in the suggested position, under compression and throwing off the rest of your balance. This is an extremely fine balance to make, and completely incompatible with the idea of moving around in your chair during the day.

Most office chairs won’t go high enough for me to put my feet flat on the floor. I’m about 6ft/180cm, so while tall, I’m not that tall. The legs on most chairs will get in the way of your feet anyway, so this is a very unhelpful guideline.

Finally, the fact that you’re pushing against the chair at 1 and 2 means that your entire body is being shoved forward (Newton’s Laws strike again). The force must be balanced at 3, 4 and 7 (hence compression of the legs). Because you have nothing particularly good to push again, it’s all frictional, and you tend to slide forward in your seat. Then you’re a sloucher. One of us!

Enough complaining. What to do?

First, buy a good chair. You spent good money on a bed to be in for 8 hours/day; now you have to spend good money on a chair that you’ll be in for another 8 or 12 or 16.

I worked on an Aeron for a few months but didn’t like it. The mesh is too stiff for my bony butt and the whole thing is very heavy and difficult to adjust. I eventually bought a Steelcase Leap and am fairly satisfied. (It was also half the price of the Aeron in Sydney. Just watch the 8-week shipping.) No, it doesn’t have mesh, but I find fabric and some padding to be more comfortable. And you can choose colours!

Roughly, my setup is:

• Elbows rest on the armrests. This works way better than no support, but isn’t great. Some people use a towel on the desk to provide extra padding, and as a bonus it catches food and coffee ‘accidents’.
• Chair is high enough that I can either stretch my feet out across the floor or tuck them under and rest on the chair legs.
• Tilt varies through the day. I can either sit up if I’m working on the desk a lot, or recline back if it’s mid-afternoon and I’m cruising.
• Slouching is a no-no. If I’m starting to slouch, I tilt the chair back. I’m a hardcore sloucher.
• I use a Kinesis Contoured keyboard and strongly recommend them. They have big wrist-rest areas and supply with pads. Admittedly, I rely on this too much (the keyboard is designed to have your hands hovering) but it works fairly well.
• I use the mouse as little as possible. My geek is showing here, but I use a tiling window manager. This is a massive timesaver; there’s no fussing with window positioning and hence much less mouse use.
• I use a Standit to hold laptops up at eye level and Synergy so that I’m not reaching across the desk or shuffling keyboards around.
• I learned to mouse with either hand, since the joints of my right hand are fairly worn out after too much playing Quake. I wish I was joking.

I expect that each person is going to have a slightly different way to solve these problems, so please share how you approach these issues in your own office.

I’ve had PCBs on order for my Bluetooth sensors for about a month now. They finally arrived today.

I got these made at Futurlec. Their quoted tolerances are quite large by modern standards, but since I’m not building a complicated board, I figured this would be OK. This is one of the tighter-packed boards that I’ve worked on, so I’m probably pushing their manufacturing abilities a little.

The board is double-sided with solder mask and silk-screening on both sides. Some parts of the board do require the minimum tolerances specced by Futurlec.

Lesson 1: Don’t run copper right to the edges of the board.

When the boards are cut, the bare copper will be exposed. I haven’t investigated the issue much, but I expect it’ll allow oxidisation and probably bad things long-term.

Lesson 2: Don’t rely on precise hole positioning.

Nothing on these boards is bad, but there’s noticable inconsistency between boards. They’re usually offset by a constant amount. Some are near-perfect, some have a fraction of a millimeter offset.

The board has a mini-USB socket and I’ve specified the socket with little mounting posts. There are holes drilled to accomodate these. Of course, offset holes means offset USB socket and possible pad alignment issues. I don’t think it’ll be a problem, but certainly something to watch out for next time.

Similarly, the silkscreen is offset slightly. Probably best not to rely on this being perfect.

Lesson 3: Better to have too much solder mask than too little.

The pads between the main CPU are tightly spaced and not all of them have solder mask between them. Since I’ll be hand-assembling the first few, ease of assembly is fairly important to me. I left the pad-to-mask margin at the Protel default, but I’ll reduce it in future. Of course, too little will probably lead to parts not sitting flat on the board. Hmm.

Lesson 4: Where possible, use thicker tracks than the minimum.

I should have predicted this after experiencing the same issue when I was etching boards at home. Bends in tracks will tend to get over-etched and hence be thinner. The risk is that they’ll get so thin that they’ll break.

Lesson 5: Don’t rely on tight grid patterns.

I covered most of the backside of the board with a gridded ground plane. This was set up to be at the minimum tolerance allowed by Futurlec. Most boards are perfectly fine, but a few are ‘underexposed’ - there are solid grid squares where there should be a hole.

Lesson 6: Don’t be lazy about rebuilding the ground plane

In a few places, I’ve got vias pushing through the ground plane (intending to connect to it). I should’ve explicitly connected rather than relying on the grid to ‘coincidentally’ connect. Again, the tracks are getting a bit thin at some points, and I have a sneaking suspicion that some of these are power tracks.

Overall

These are all minor quibbles. I don’t think they’ll make any substantive difference when I actually build the thing. Most of this comes down to “be generous with tolerances”. Don’t rely on everything being perfect; allow a little slop in the design.

Nothing spectacular has happened lately. It’s just been a lot of the usual hard work.

The amount of work involved has surprised me. It’s not just the fact that I’m running multiple parallel projects - just bringing a single project to market is an amazing amount of effort. A lot of the work will be faster the second time around, so I’m not too concerned. I do find it amusing that it’s been almost six months and I don’t have any products out yet.

The Mutex Labs website is finally at a state that I’m happy with. You should hire me as a consultant. Now. Do it!

I’ve been running AdWords to drum up some more consulting work. So far, the response has been mediocre. I think that part of the problem is that there just aren’t that many people searching for Python geeks in Australia. Additionally, I don’t think people are searching on Google for them. It’s not like it’s costing me much to keep the ads going, but I don’t have high hopes for them yet.

I also set up Microsoft AdCenter. So far, they haven’t given me a single impression. I don’t know why - even if I search for the exact terms, the ads don’t appear. I guess I’m USD$5 in the hole until I can figure out what’s going wrong.

I am amused that AdCenter appears to have been ‘inspired’ by AdWords. And by ‘inspired’, I mean ‘completely ripped off’. I’m mostly using it because Live is the default search in the new versions of IE and I’m expecting large growth in the amount of their search traffic soon.

My experience with online freelancing websites (Elance, Rentacoder, etc) has been very poor so far. I’ve seen a few good ideas float across, but so far very little interest from the people advertising the projects. Most projects fit into one of three categories:

• Advertiser has no idea what they want and gives a ridiculously vague spec (which we then nut out during IM and discover that it’s a lot harder than they thought originally)
• Advertiser gives a good spec with a few glaring flaws. I ask for clarification and get no answer.
• Advertiser is only offering slave-labour wages, and so I can’t afford to take the work anyway.

Most of the problems are with specs. I’m not going to bid on a project that has +/- 50% wiggle room in its spec, and so far that’s been the good case. And since the hourly rates are very low to begin with, I can’t afford to take the (high) risk of scope creep with the spec that is in place.Contrasting with last month’s money concerns, I’m now inundated with work requests. Nothing concrete yet, just a number of separate inquiries. This sort of highlights one of my frustrations with treating advertising and measuring response rates as a statistical system - noise drowns out your actual data.

I’m still waiting on PCBs for my data loggers. I think it’s been… three weeks now? The manufacturers’ website quoted one week. It took them a week just to acknowledge that I had sent an order. Sigh. I have another place in mind for the next batch, and they’re pretty quick to respond to emails as well. I also have quotes from a few local places who can do short-turnaround orders - it’s not cheap, but it’s good to have the option.

Data logger firmware is about 60% complete. Of course, most of it has only been compiled, not executed, since I still don’t have hardware. Still, it feels good to make such rapid progress.

SyncDroid is stalled. I’m going to run out of data logger code to write soon, so it’ll get some more attention shortly.

I’ve also been playing with a ‘keyword mixer’ that generates all possible combinations of words, intended for AdWords and the like. It’s come up because I have a number of skills that I want to advertise (for example, ‘Python’, ‘AVR’, ‘Gumstix’). Using ‘Python’ as a bare keyword gets a lot of clicks but no conversions (too generic), so I make the search terms more specific by adding terms like ‘development’, ‘consultant’, ‘expert’, and so on. So, the mixer generates every combination from the two lists. Then you copy and paste it into AdWords and tweak from there.

I want to do similar mixing with ad headlines and bodies to help with comparative benchmarking.

It took me about half an hour two write as a Python script. There are a few programs that do similar jobs, but you have to download and mess around with them (and pay, in many cases). I’m thinking of sticking this online and just seeing what happens. I have no solid monetization ideas right now.

I’ve got money jitters.

Logically, I know they’re irrational, at least in the short term. There are two contributing factors: I haven’t had a predictable paycheck for about six months now, and me putting up AdWords didn’t immediately make the phone ring.

Yup, that’s all.

I’m good for cash for probably two years yet. That’s actually a better prediction than my little thing on the right-hand side of this site, which places my starvation point at Feb 2010. I feel more like someone is tapping me on the shoulder every time I buy something: “Are you going to get that money from your long-term savings? From your index funds? Are you going to sell something? WHAT? You still owe GST/PAYG and the credit card and rent and Amazon and blah blah blah”.

I haven’t had any coffee yet today and it’s after 3pm. Bear with me.

In other news, I haven’t felt like blogging much. I’ve been busy writing content for the Mutex Labs website. My goal there was to get it to a point where it was good enough for me to start pimping it with AdWords, and it is, sort of. I’m spending pretty much the bare minimum - 25c/click, $5/day - and so am getting very few clicks. But that’s OK. I’m already getting good data on what people are interested in and how well the keywords are matching up.

Parts for my first run of Bluetooth sensors are due in any day now. I find that having physical artifacts really clears up how profits actually come about; you buy input parts for $x then sell them for $3x. While the input parts feel kinda expensive, they’ll sell for what appears to be a huge sum. I don’t feel the same way about it that I do about selling software, which is kinda like panhandling: “You like the app? Toss me some cash? Please? Help out a starving programmer?”. There’s physical (figurative) meat to manufacturing which is far more understandable to my hunter-gatherer brain.

And of course, Joel Spolsky’s recent post about how file synchronizers are a solved problem and how making fun things is the future. Well, I’m biased of course, working on a file synchronizer as I do, but I still disagree.

Everyone writes file synchronizers because they’re an interesting programming problem; yes, it is. I don’t believe that the market for it is sewn up just yet, though. Lots of people have multiple computers. Not many people use a synchronizer to keep files in sync between them. Why? The best that I can come up with is that they generally suck (as I have previously mentioned). There are a hundred of them and they all look alike. They all have the same annoyances and are generally not worth the effort. I’m using Unison right now and it’s alright, but it could be a whole lot better. Sadly, Unison is probably the best that I’ve seen, despite the hundreds that are out there.

Making fun things. I wrote a somewhat inflammatory rant on another blog a few weeks ago about how ‘fun’ things are basically the work of the devil. I still feel that way, and I worry that that’s where Joel has gone wrong. Fun things are fun, yes, but do they make my life better? Do I wake up in the morning and say “geez, I’m glad Twitter is around. It really enhances my life.”

These are exactly the sort of things that are interesting programming problems but not really consumer problems. Social networking sites. Yes, Facebook is popular. Yes, it’s valuation is beyond belief. But does it solve a problem? Does it enhance humankind? If fireballs rained down on the Earth and you could only pack a few things into a suitcase before retreating into your underground bunker, would you take the Facebook source code?

I guarantee that you would not.

Perhaps it’s just me, but I’m in this to create great things. My view is obviously not shared by everyone, but I’ve stated repeatedly that my goal in this is not to get fabulously rich. I want to work on fun stuff and be able to live comfortably. That’s it. I’d rather be building rocket ships than websites, even if the latter pays better.

Time and effort

The time is flying by.

I think I’m gaining a sense for what is achievable in a given period of time. My usual strategy is to think “I need this done TODAY” and then burn very hard to get it done. I get my payoff at the end of the day and am temporarily satisfied. I don’t really ‘get’ longer-term planning, though.

For the last few weeks I’ve been working hard and consistently. I haven’t been getting the big payoff at the end of the day, but I have been satisfied with my effort. I think that’s more important in the long term. It does make me anxious that things will never be completed, though, so I’ve spent a little time writing a plan for how I’m going to get to the next major milestone - someone giving me money for a product. My Plan for Profitability, although profitability comes a long way down the track.

I’ll probaby do The Plan in another post. It doesn’t help that I haven’t written it yet, despite claims to the contrary in the preceding paragraph.

Bluetooth sensors

I had a fun little coincidence when, within 24 hours, I received two separate enquiries for temperature logging devices. Then another one for a sound sensor. Then some accelerometers. And I want heart rate and altitude monitoring.

So one of the ’software products’ will be data logging/Bluetooth/alarming widgets. It’s within my capabilities as an electronics geek and provides a nice alternative to software (though there is a lot of software inside anyway).

I did have a half-hearted attempt at an electronics design/manufacturing business a few years back. I can confidently say that back then, I was a noob. In a few years, I’ll probably look back at this time and say that I’m a noob. I do that anyway. I’m always smarter in the future.

Anyhow, I’ve slapped together a basic Bluetooth temperature sensor. The technical challenges at this point are:

• Produce a smaller, cheaper, more battery-friendly design.
• Work out how to provide more accurate temperature measurements. Sensor calibration is not a business that I want to get into.
• Design an API for people to write software for the thing.
• Work out if there are going to be any regulatory hurdles that I’ll have to jump through - EMI/EMC emissions, RoHS regulations, export controls and so on. Software is a lot easier - any chump can ship dangerous software anytime.

It’s fun. It is a lot more capital-intensive than software, but there’s a much greater variety of activities. And being able to hold a device in your hand and say, “I created this” is satisfying on a much deeper level than software.

More and more marketing

After my initial marketing binge at the start of the year, I haven’t done much (apart from this blog). I need to get on top of it again. It helps me work out where my efforts are best spent, and I need to get some consulting work happening. Not really need, I guess, but I’d feel a lot better if I didn’t have to dip into my savings to pay my bills.

I’ve been writing content for various websites, but I have a much lower tolerance for that than coding. I can manage six solid hours/day of coding. I can manage maybe one or two for writing marketing copy. Electronics design I can do more or less indefinitely, but it’s far less demanding than either of the two (most of the time is spent interfacing with the tools rather than mentally designing things. I still use pen and paper for most of it).

Planner

One of my initial ideas (on the right) was for a hierarchical todo list thing. I wrote a prototype that satisfied my needs and then lost interest. Most of the features that I had determined other people wanted were completely uninteresting to me. Plus, the concept was more or less stolen from AbstractSpoon’s ToDoList, so I felt a little bad about trying to charge money for his ideas.

So I’ve been using this hacked-together Python-and-wx-and-sqlite thingo for a year or so, now, and it suits my needs beautifully. I’m sure it will suit somebody else’s needs, so rather than try to commercialize it, I’m going to stick the code up somewhere and let you lot Have At It.

Eventually, anyway. Releasing free software is not on The Plan, so it will most likely get ignored for a while.

There’s a lot of stuff on my plate right now. Everything is progressing reasonably well:

• Client A is still causing trouble, but they appear satisfied for now.
• The Mutex Labs website design is looking very good. I’ve also started writing content for it, so that should be up in the next couple of weeks.
• I’ve started narrowing my focus for consulting work. I’m going to start out focusing on embedded C coding, particularly the Gumstix platform. I’ve been writing a list of things that I’m good at and would like to work with, so that should lead nicely to a list of capabilities for consulting work.
• I’ve also been trawling local job sites to see what is going. There’s a surprisingly large amount of work available. The old pains of dealing with recruiter ads is showing - “must have xyz in the SDLC”, “must have 93 years of experience in brand new tech Y”, “must be an expert at buttering bread”. Stupid stuff like that. I’m trying a different tack this time and seeing if I can more or less bypass the ad and get people excited about the package that I bring, rather than just checking boxes on the ad. It’s going to be interesting to see if I can convince employers to let me work independently from my office rather than being a short-term full-time contractor.
• Something that is working quite well for me is looking on freelancer sites like Elance and getting in contact with the people with work available. I’m generally not able to compete with overseas programmers, price-wise, but I can offer things like business relationships or I-build-it-you-license-it-from-me type of deals. That’s fine for me - I love building things and I want to work on my product portfolio. I have two hardware projects on the burner and a very interesting third in discussion.
• I’ve even been working on SyncDroid! I’m in the middle of some really hairy multi-threading-uncertain-relationships code, but it’s coming along slowly. I think that once that’s out of the way I can start using it on a day-to-day basis. At some point I have to do a bunch of UI coding, but I’m avoiding that for now.

Something that I’m going to have to watch is not taking on too much at once. I love the feeling of a new project, but I fall down when it comes to actually completing them.

I’ve noticed that money appearing in my bank account improves my motivation quite a bit. This is probably the main factor that is encouraging me to look for work at all - just that nice reassurance every couple of weeks. I don’t need the money right now. I just need to know that if I want to, I can earn some cash. Conversely, big projects like SyncDroid are difficult for me to stay focused on because I don’t see any immediate payoff and won’t for a while.

I’ve been thinking about my current work situation a lot.

For the last month or so, my workload has been primarily Client A and MicroISV work to fill the gaps. Client A is getting very sloppy about paying invoices, typically running a week or two overdue. I need to forcibly remind them and usually resubmit the same invoice multiple times. They are a high-stress client, change their mind a lot and have frequent emergencies. Two years ago when I started with them they were paying what seemed like a lot of money. In that time I’ve taken a pay cut, suffered from high inflation and significantly improved my skill set, so I’m not being paid very well either.

So I think the time has come to let them go.

Of course, this is the ‘take the leap’ problem all over again. Do I try to make it on my own, or do I stick with a stable (ish) thing?

This time, I’m going to take a different approach. Rather than try to make it purely on the MicroISV, I’m also going to cultivate some consulting work. I’m eager for the human interaction, and seeing payments go into my account is a big motivator. I don’t need to be earning a lot overall, but I do want to keep my hourly rate at a fairly good level. What constitutes ‘good’ may shift with time, but I’m still comfortable for a few years even if I don’t earn anything.

Along the way, I should pick up some interesting contacts and ideas. Not that I’m short on ideas already - the challenge is in completing them before getting distracted by the next shiny new thing.

I’ll be without predictable income for the first time since I started working (seven years). I should take a proper holiday. It’ll be the first one since I started working, too.

I think this is the right direction for me to head in. And, as always, the worst that can happen is that I have to get a job.

Ah, how time makes fools of us all. From Jan 22:

I’m at a point in development where I can finally say I understand how the whole application is going to fit together. It’s very reassuring - there was always this uncertainty as to how the different modules were going to connect (in particular, how asynchronous events were going to move around.) Now that I have a model I’m happy with, it’s just a matter of ‘colouring in’ what’s left.

I’ve just returned to the SyncDroid code after a few months off and I have that “holy crap, what was I thinking?” feeling. The small-scale implementation is fine, but the object model sucks. I can’t figure out what should be talking to what. Comments that I’ve left in the code indicate that I did have a plan, but when the journaller code is making merge decisions and the merge code is storing data on the disk, it’s a sign that something has gone wrong.

So I’m refactoring it all. Again.

I have a new plan which looks suspiciously like the old one. I think the difference is that this time I have a better idea of how data flows through the system. I’m finding SyncDroid to be very difficult, conceptually - there are multiple threads and multiple priorities and databases coming and going all of the time. Alan G. Carter of The Programmers’ Stone speaks of the importance of juxtaposition in programming. This project has juxtaposition up the wazoo. There’s enough that I can’t hold it all in my head at once (yet) and so I’m struggling.

Perhaps it reflects the elementary difficulty of what I’m trying to do. Perhaps that’s why no-one has done it before. I’ve certainly never struggled so much to hold a mechanism in my head before, and in past lives I’ve programmed 64-way machines, massively parallel dynamic hardware, and then both in the same system.

Anyway. It’s coming along, slowly. The other thing that I’m noticing with Python is that the amount of behaviour you pack into a given volume of code is very high. My major contract client has me coding predominantly in C, so it’s quite jarring at times. It’s not a bad thing, but I often wonder how much more difficult this would be in C. I don’t worry so much about memory management - I’ve got that pretty well down - but more things like standard libraries and unexpected crashes. The nasty thing about C is that it’s so brittle - a single misused pointer will crash your entire program. Python won’t let you make that sort of mistake, and if you do anyway, it handles the mistakes far more elegantly.

I’ve also finished some C security work lately and have come to the conclusion that if you want to write secure software, don’t do it in C. In fact, let’s generalize that. Don’t ever use C. That’ll be the right choice most of the time. If someone asks you to code in C++, run for the hills. Or at least make sure you’re invoicing by the hour.

Maybe it’s time for a holiday. My 26th birthday is coming up soon and I feel very old.