Status report, 5 April 2008

April 5th, 2008

Ah, how time makes fools of us all. From Jan 22:

I’m at a point in development where I can finally say I understand how the whole application is going to fit together. It’s very reassuring - there was always this uncertainty as to how the different modules were going to connect (in particular, how asynchronous events were going to move around.) Now that I have a model I’m happy with, it’s just a matter of ‘colouring in’ what’s left.

I’ve just returned to the SyncDroid code after a few months off and I have that “holy crap, what was I thinking?” feeling. The small-scale implementation is fine, but the object model sucks. I can’t figure out what should be talking to what. Comments that I’ve left in the code indicate that I did have a plan, but when the journaller code is making merge decisions and the merge code is storing data on the disk, it’s a sign that something has gone wrong.

So I’m refactoring it all. Again.

I have a new plan which looks suspiciously like the old one. I think the difference is that this time I have a better idea of how data flows through the system. I’m finding SyncDroid to be very difficult, conceptually - there are multiple threads and multiple priorities and databases coming and going all of the time. Alan G. Carter of The Programmers’ Stone speaks of the importance of juxtaposition in programming. This project has juxtaposition up the wazoo. There’s enough that I can’t hold it all in my head at once (yet) and so I’m struggling.

Perhaps it reflects the elementary difficulty of what I’m trying to do. Perhaps that’s why no-one has done it before. I’ve certainly never struggled so much to hold a mechanism in my head before, and in past lives I’ve programmed 64-way machines, massively parallel dynamic hardware, and then both in the same system.

Anyway. It’s coming along, slowly. The other thing that I’m noticing with Python is that the amount of behaviour you pack into a given volume of code is very high. My major contract client has me coding predominantly in C, so it’s quite jarring at times. It’s not a bad thing, but I often wonder how much more difficult this would be in C. I don’t worry so much about memory management - I’ve got that pretty well down - but more things like standard libraries and unexpected crashes. The nasty thing about C is that it’s so brittle - a single misused pointer will crash your entire program. Python won’t let you make that sort of mistake, and if you do anyway, it handles the mistakes far more elegantly.

I’ve also finished some C security work lately and have come to the conclusion that if you want to write secure software, don’t do it in C. In fact, let’s generalize that. Don’t ever use C. That’ll be the right choice most of the time. If someone asks you to code in C++, run for the hills. Or at least make sure you’re invoicing by the hour.

Maybe it’s time for a holiday. My 26th birthday is coming up soon and I feel very old.

Status report, 3 March 2008

March 3rd, 2008

I’ve been doing contract work instead of MicroISV work.

Not that that’s necessarily a bad thing. Getting paid is nice. It’s just not precisely the direction I want to go in. It’s still a good direction. If I managed to build a self-sustaining consulting/custom dev firm, then that would be a great outcome. That’s just not where I intend to go right now. It involves a lot more capital and a lot more risk and as far as my financial goals go, it’s unnecessarily grand.

There have been some small developments on the business admin side of things. I’ve got a company name now - Mutex Labs. I’m vetting a number of graphic designers to do logos and websites. I’ll be talking to an accountant soon to work out how best to structure my affairs (large tax bills are already a problem). I continue to read and study and gather ideas for the marketing side of things. There have been more good product ideas.

I guess everything is just coming along nicely. My contracting time is heavily booked until about halfway through April, and that’s assuming nothing new comes along in the meantime. Taking a holiday is sounding very appealing right now!

Kinesis Advantage keyboard and learning Dvorak

January 27th, 2008

I bought a Kinesis Advantage keyboard with the intention of reducing my finger pain associated with typing. Obviously, I spend a good portion of every day typing, and my livelihood basically depends on my being able to continue typing.

I also decided to learn the Dvorak layout while I was learning the Kinesis keyboard. I tried learning Dvorak a few years back but gave it up because I was working with a lot of other people’s keyboards as well - it was too inconvenient to keep switching layouts.

On the Kinesis/Dvorak learning process

  • Muscle memory is a huge factor in switching to a different keyboard or layout. Even now, when I’m typing on my laptop I instinctively reach for the Enter key with my right thumb, because that’s where it is on the Kinesis.
  • The Kinesis makes a lot of bad habits difficult, whether by accident or by design. You can’t really rest your hands on the pads while typing because then you can’t reach all of the keys. You can’t twist your hands around to move them around the layout because the keys are aligned to suit your hands in the home position. When you move your hands around the layout, suddenly they’re un-aligned and awkward.
  • I started out using a lot of ‘mental CPU’ time to handle the conversion. In the beginning, it took all of my concentration just to hit the right keys - I had to separate my thinking from my typing.
  • While learning Dvorak, I noticed an interesting progression; I started out pressing just single keys at a time. Gradually, I started combining strings of keys into single motions (something I call ‘chording’, which I’ll come back to). This is similar to how a child learns to read - they recognize single letters, expand out to sounds and eventually can string together words.
  • I made rapid progress for about three weeks. The first few days were difficult. I only used the Kinesis+Dvorak for a couple of hours each day because it was very frustrating to learn. I’ve had 20 years with a nice solid brain-keyboard link via the keyboard, and suddenly it’s horribly slow and error-prone. After the first few days things settled down a bit and I could manage an entire day’s work on the Kinesis.
  • After the first three weeks, progress slowed. I was still improving, but more in the areas of chording and accuracy. Some keys still gave me consistent problems on the Dvorak layout, particularly G and P.
  • You need to sit a little higher than with a normal keyboard. This is a problem for me - I already find standard office chairs too short (even the Aeron!). When time allows, I’ll be buying a nicer chair and getting the gas-lift swapped for a taller one.
  • Why the A and S keys - two of the most common characters in English text - are on my two weakest fingers, I’ll never know. The pinkies get more of a workout than usual because they’re handling all of the keys on the edge of the keyboard, too. Placing these two common letters on already weak and overutilized fingers is probably the biggest flaw in the Dvorak keyboard that I’ve found.
  • Dvorak is lovely for English text. It’s just a great feeling to feel the letters whiz by with so little effort. However, a lot of my typing load is not English text. It’s Linux terminal navigation, C and Python source code, all of which intentionally discard vowels in exchange for brevity. This makes Dvorak’s plan to alternate hands work very poorly - you’re just not typing anything on the left hand.
  • Some common Unix commands are absolutely worst-case scenarios on Dvorak. Take the command ‘ls -l’, which I would type dozens of times per day. On Dvorak, L, S and hyphen are all on the right pinkie. If you’re using a standard keyboard, so is the Enter key. It’s really, really unpleasant to type ‘correctly’.
  • Naturally, position-based key bindings don’t work: Vim’s HJKL, WASD for games, Ctrl-Z/X/C/V for word processing.

I really tried to make Dvorak work. I gave it a month. At the end of the month, I switched back to QWERTY. It’s a fantastic layout for cranking out lots of English text, but that’s not my use case.

On the Kinesis keyboard

  • The up and down arrows are backwards compared with the Vim convention. That’s relatively easy to remap on the keyboard.
  • The macros are rather buggy. Actually, scratch that. The keyboard’s firmware is rubbish. I suspect that either the authors had never programmed on a microcontroller before or were not trained as programmers to begin with. I can crash the keyboard in two fairly common scenarios. That said, the crap firmware is not a reason to not buy the keyboard. It’s perfectly usable despite its flaws.
  • I opened it up to look inside. It’s pretty typical of small-scale electronics manufacturing: no surface-mount parts, revisions hot-glued to the case, off-the-shelf components. It’s not badly made - it feels very solid for its weight - but it does have a few rough edges which might surprise you if you’re expecting a mass-produced product.
  • It’s ripe for hacking. The main controller is an Atmel AT89S series microcontroller. The macro RAM is on standard serial EEPROMs. There’s even a socket for a second one (the Advantage Pro upgrade). Apparently the firmware can be changed over the PS2 or USB port, but Kinesis didn’t seem to willing to send it to me when I mentioned I wanted to modify it.

Where to from here?

I still want to improve the keyboard layout. The Kinesis makes typing less painful, but some of my pains appear to be linked to the QWERTY layout. And the feeling of effortlessly flying through English text with Dvorak was just amazing.

To come up with a better keyboard layout, I want to log my keystrokes for a month. Each keystroke will be tagged with the time and the active process. The process lets me figure out whether the keystroke intent was a letter or a position. I can also detect errors by tracking Backspace presses. With that information, I can determine exactly which keystrokes or combinations are the most common for me.

In addition, I want a ‘trainer’ - a program that will prompt me with an arbitrary series of keystrokes and time how long it takes me to hit them. This will give me information on how strong and fast my fingers are and if any of them are particularly error-prone. From that, I can generate a map of the keyboard, each key associated with a ‘performance’ score. Combining the two datasets, I can then come up with an ideal keymap for me, given my typical usage patterns and my own brain-keyboard performance data.

I’d also like to integrate information on common digraphs, but I’m not sure how best to use them. I’m not sure that Dvorak’s assertion that alternating hands is the best thing to do. A common case for me is the ‘chording’ I mentioned previously, where a single hand can hit a sequence of keys very rapidly. The timing is simpler - I arrange my hand correctly on the keys, then use the individual fingers to press them in sequence. Of course, this sounds like the sort of thing that might cause tendon damage. But it’s fast.

There’s more discussion on chording and performance here.

Status report, 22 Jan 2008

January 22nd, 2008

Development progress

I’d say I’m about 30% of the way to a SyncDroid release. It’s frighteningly small, but I know that boring stuff like testing, installers, licensing, graphics and documentation is going to eat up a lot of time.

I keep detailed lists for this stuff. Some people say it’s obsessive-compulsive; I say it’s damned well essential because there is just so much to do. If I don’t write it down, I’ll keep worrying about it until it’s done, and that’s a huge energy sink.

I’m at a point in development where I can finally say I understand how the whole application is going to fit together. It’s very reassuring - there was always this uncertainty as to how the different modules were going to connect (in particular, how asynchronous events were going to move around.) Now that I have a model I’m happy with, it’s just a matter of ‘colouring in’ what’s left.

Typing

I’ve had some level of hand pain for probably a decade. It’s mostly been quite ignorable and always went away after a few days. I blame it on the 1996-1999 period where I trained on Quake variants for anything up to 8 hours per day instead of studying.

Lately, though, it’s been getting worse. Much of this is that I’m doing a lot more typing now that I’m not rotting away in meetings. Short-term, I did an eleven-hour-long adrenaline-fueled hack-fest (on client work, no less). I didn’t type much for the following two days due to the pain. After some encouragement from my officemates, I bought a Kinesis keyboard.

I’ve seen them before, but ignored them due to the high price. Now, with the low US dollar, my improved financial situation, increased frequency of pain and likely expense of surgery, it seems like a bargain. I’m learning Dvorak on it too. It’s intensely frustrating to not be able to bang out 100+WPM without thinking about it, but so far I’m seeing major reductions in pain. I see daily improvements in speed and accuracy, so I’ll almost certainly stick it out.

Logos and icons

I can imagine what I want in graphics, but can’t express it (in words or images). I need some icons for SyncDroid and some logos for the website(s), so I went to a <$40 logo design place.

Lesson 1. If you say you want the logo to look like something, they’ll give you exactly that, copyright law be damned.

Lesson 2. If you let them pick a colour scheme, they’ll pick something ugly.

Lesson 3. If you specify colours, they’ll use them in the most ugly way possible.

Lesson 4. If you specify “simple, no detail”, they’ll add superfluous detail that would suck even if you had the resolution to display it correctly.

I’ve got ‘unlimited revisions’, but that doesn’t mean anything if they can’t design anything that looks good. So I walked a few doors down to an art supplies store, had a nice chat with the hottie at the counter, and bought a nice soft clutch pencil. I’ve sketched out some designs that don’t completely suck. I did have a quick go at implementing them in Inkscape but lost patience quickly.

I’ve looked at a lot of icon sets on the web and can see a pretty strong correlation between price and quality. Damnit.

Tuba

There’s a guy outside playing Mario Brothers and Star Wars tunes on the street corner. Newtown is by far the most fun location that I’ve ever worked in.

Status report, 14 Jan 2008

January 14th, 2008

I’m here! I’m alive!

I’ve been doing a lot of client work lately, more because people want work done rather than because I want money.

Since the last update, I’ve made great progress on SyncDroid. After thinking about this thread on the Business of Software forum, I’ve decided to split SyncDroid into several different, smaller products - one for USB drives, one for network sync, one for SaaS Internet backup, and so on. It also helps me to focus - I’m working on a smaller product with a more clearly defined feature set rather than this great big ‘do everything’ product. And it helps me to focus the marketing message and differentiate (people like specific things that fit them well rather than generic things that happen to fit them). So I’m very happy with my decision to split into a whole product line.

UI work continues. I’ve got the scope of the project worked out and can finally say that I understand the project rather that just focusing on little bits of it. I can also confidently say this is where my product fits in the world and this is why you should pay money for SyncDroid instead of Unison. It’s a nice feeling. Very confidence-inspiring.
I’m amazed at how far off my original effort estimation was. I wrote most of the sync engine and figured I was about halfway. Now I can say that the sync engine was maybe 2% of the total effort required. UI’s are hard. Marketing is time-consuming. I haven’t started doing any serious cross-platform work yet, either - I’m doing a Linux release first.

I’ve been thinking about the different platforms a bit, too. My feelings are that:

  • Linux is easy and fun to develop for and you’ll get good feedback, but nobody will actually buy your stuff
  • Windows has a huge user base, but they’re relatively clueless
  • OSX has a fairly intelligent user base and they’re trained to expect to pay for good software, but there’s just not that many of them

OSX has the additional problem that in order to develop for it, you have to buy a Mac. This becomes a simple economic problem: if I spend AUD$850 on a Mac Mini and $2000 worth of labour on an OSX version, will I sell enough Mac licenses to make the money back?

After spending years looking at everything through purely economic terms, I now believe that a decision based on such terms is an oversimplification. If I was going for (legal) profit maximization, my strategy would be:

  • Go to my contract clients, start billing a lot of hours, and use my position as sole developer/provider to leverage a large $/hour rate increase
  • Move back into my parent’s place
  • Stop spending money on things like petrol and coffee

The economic decision-making model ignores the fact that my life would suck if I followed it through to its logical conclusion. I would be miserable. Starting this MicroISV has been one of my best decisions in recent memory just because of the quality-of-life improvements that it’s made. I’m not making anywhere near as much money, but I’m a lot happier. And so, I can confidently say things like I will keep doing this even if I’m just breaking even. I can happily forfeit a large income and go for a riskier, cheaper existence, because my life is that much more enjoyable.

And so, Buying A Mac. I’m going to wait. I’m going to get a Linux and a Windows release out there, gather some feedback, measure the size of the response. If nobody is interested in SyncDroid, I’m better off cutting my losses immediately.

I was going to end the post there, but it’s such a down note. Running a MicroISV rocks. Hard. Down with The Man!

MicroISV’s in Sydney

December 31st, 2007

Here’s a quick list of MicroISV’s that I know of in Sydney. I know there are lots that I’ve missed. Email me if I’ve forgotten you!

(I’m considering a bootstrapped software company of four people or less to be a MicroISV).

  • tiinker by Deep Grey Labs: an intelligent news aggregator that uses datamining techniques to find the best stories for you
  • pulse by zutubi: a really slick continuous integration system

Honourable mentions to people who gave it a try and are getting back up to try again:

Oddly enough, I can name a whole lot more Brisbane micro-ISV’s, despite having only spent three days there.

Status report, 28 Dec 2007

December 29th, 2007

Another day, another status update.

I’m getting a lot done. What’s amazing me is the sheer volume of work that needs to be completed. I feel like I could keep someone busy just writing material and investigating sales opportunities and gathering requirements and so on.

Yesterday and today, I:

  • Bought domains for my filesystem and file synchronizer.
  • Set up WordPress on integrifs.com and syncdroid.com.
  • Came up with something to share a single WordPress installation between multiple sites. Instead of WordPress-MU (which assumes you’re hosting all sites under the same domain) I put a switch statement in index.php to change database names based on the hostname in use. This is a whole lot easier, especially when the time comes to upgrade WordPress.
  • Wrote a bunch of marketing material for IntegriFS: overview, how it works, download and a few drafts.
  • Cranked out a dozen pages that I have to write for SyncDroid. I’m just putting the titles into WordPress right now and leaving the pages themselves as drafts, so I can log in whenever and wherever and fill them in.
  • Ditto for IntegriFS. There’s a lot of writing to do.
  • Set up some Google AdWords for SyncDroid. There’s still a lot more work to be done there.
  • Finally found a use for all of those random news aggregators: finding potential customers! I’ve been tracking interesting discussions and gathering a lot of useful information.
  • Made some good progress on a WordPress theme that I’ll share across all of my sites. Each site will have its own colour scheme to differentiate it.
  • Explored a bunch of competitors to SyncDroid. Without exception, they look complicated. They have fancy conflict resolution screens. Complexity is what I want to avoid with SyncDroid. I want it to be as automatic as possible. I want to put in an Undo feature so that if it does make a mistake, you can back it out anyway.
  • Importantly, got a lot of the IntegriFS design and strategy out of my head and onto paper. This is the crux of Getting Things Done: get it out of your head. For the last week or so I’ve been buzzing with ideas and thoughts, and my mind is finally clear to look at other things that are important in the short(er) term. I need to get some bigger pieces of paper, because at the moment it looks something like this:

integrifs-design.jpg

Note that I’ve written no code at all in the last two days. This would have surprised me a week ago. Right now, my focus is to get a bit of attention from people. This should help me work out what sort of demand there is, whether my marketing efforts are working and hopefully even get a bit of feedback from potential users. If it turns out that I’m on the wrong track or there’s no interest at all, I won’t waste time writing code.

There are a few different directions I could push in for SyncDroid. The base technology - detecting, logging and propagating file changes - is useful to several applications. The first ones that come to mind are:

  • Automated real-time backups
  • Synchronization via USB drives
  • Time Machine (snapshotting and rewinding a filesystem)

I wanted to implement all of these, but at this stage I’m focusing on laptop-desktop synchronization. With the new office, however, my own use case has changed. I’m synchronizing over the Internet using Unison, but if I need to move any large amounts of data around that’ll need to go over a USB drive. Hence, using a USB drive as the go-between becomes attractive. Another nifty idea relating to this is to use a mobile phone (connected via Bluetooth) instead of a USB drive.

The Time Machine opportunity is interesting. It’s a very early market with few competitors. There are no compelling implementations for Windows, yet. I have no doubt that there will be an effective free one in time. With the base technology already in place, though, it’s not a big technical challenge to toss another UI over the top. There’s still a huge marketing effort to do, but I’m getting better at that all the time.

Status report, 27 Dec 2007

December 27th, 2007

Things are steaming along pretty nicely.

Marketing

After listening to a podcast feature Greg Gianforte at a MySQL conference, I realised that I should be putting effort into marketing now rather than after my product was ready. It seems somewhat obvious now, but I was getting excited about B*-trees and block allocation algorithms without realising that I should do more to establish demand ahead of time.

The filesystem has a marketing name, a website, and is being filled with content. I’m really excited about the idea, actually. It’s a new area of research; I get to build something that I believe in; it hasn’t been done before; there are lots of good monetization options. It is a big product, and so I’m rapidly making my ’six products in one year’ slogan irrelevant. I’m not too bothered. I have some real passion for this idea, and I think that’ll be more important in the long run.

Filling in the website makes me think about what features I intend to support - it’s like a simple spec. And it makes me more excited about what I’m going to build.

I have done a little bit of coding - mostly exploratory stuff with FUSE. It all seems to be making sense. I’ve done a bit of kernel work before, so there’s nothing too scary there.

I haven’t started doing any marketing for the file synchronizer. This is partly because I’m not thinking about it so much with the filesystem. This is a trap that I frequently fall into - I work on something while I’m passionate about it, then forget about it before completion. I do have a good body of code there, however, so I should start working on that at some point.

The big lesson that I’m learning right now is that there is an awful lot of work to be done. I can’t just sprint to a release. It’s going to take consistent effort over a period of months to get these things out the door, and probably more effort to get people to actually use them (let alone buy something!) I need to just be calm and patient and keep putting in consistent effort.

AdWords

I’ve set up a bunch of AdWords ads for the filesystem. They’ll go active when I have enough website content to justify it. They’re always a good way to generate more ideas - I had suspected that Windows users might be interested in something like this, but was interested to learn that a lot of people have been searching for ways to get ZFS to run on Windows. This is encouraging to me - it’s a big market opening. Getting ZFS running on Linux is a bit dodgy right now, and Windows is not even under consideration. Along with the general confusion and problems running RAID arrays under Windows, there’s big demand for highly reliable storage on commodity Windows machines.

Office space

I’ve got some office space. Hurrah! It gets me out of the house and the others I’m sharing it with are great fun. I don’t think I’d go back to working from my bedroom for any length of time. There’s some productivity cost in terms of distractions, but there’s big benefits to my moods and motivation which more than makes up for it.

I am still experiencing difficulty concentrating after about six hours of solid work and finding that it’s somewhat diet-related. Along with a recent Slashdot article on nootropics, I’m going to have to investigate dietary supplements more. Now that I’m actually interested in working longer hours, it may be useful.

Day job

The day job is still there, I guess, but on holidays. I haven’t done much work for them for a while. I have had to start transferring money from my investment accounts, which is a bit scary. But it’s early days yet. I always have the option of seeking out more paying work; I just don’t need to right now.

Improving on ZFS: BlobFS

December 13th, 2007

I have a love-hate relationship with ZFS. On one hand, I love how easy it is to operate and how it makes it difficult to damage the array. On the other hand, I hate how it interoperates poorly with Linux and how there are some really crucial operations (like vdev removal or array growing) that aren’t implemented and haven’t been for a long time, despite pleas from the community.

I think it could be easier and better. ‘Final word in filesystems’? Yeah, and 640kb should be enough for anybody.

This is intended for home file servers - big files, mostly reads, performance not a big factor. An assumption is that there is no backup, so the data needs to be as safe and correct and recoverable as possible.

The idea

  • You have some disks. BlobFS will make them a big filesystem with a specified level of redundancy.
  • You can select the number of disks you can lose and hence the level of redundancy (the ’safety margin’). Changes are implemented automatically.
  • If you pull out a disk, the data will rebalance across the array automatically. You’ll have a capacity loss (if the data will fit on the disks) or a safety margin loss (if there’s too much data).
  • If the array does lose too many disks, BlobFS will attempt to allow access to anything that’s still readable. This lets you attempt a partial recovery.
  • All data (including filesystem metadata) is end-to-end checksummed, so you can be fairly certain that flakey equipment is not going to silently corrupt your data.

The basic idea is that you give it some disks and some data, and it will take care of the data as best it can. You leave the low-level decisions about RAID algorithms and device recovery up to BlobFS. It makes the best moves it can with the resources it has available.

This has a subtle but significant benefit in terms of reliability. It’s removing human intervention from the recovery process. You don’t have to say “OK, I’m going to offline that disk and add another one in its place and restripe the array” while hoping that you’re offlining the correct disk and restriping is the right thing to do and your replacement disk is in fact empty and not another disk in the array that contains the last redundant copy of your data. All of that logic is in the software. It takes care of your data as best it can.

disk-drive.jpeg

Some other things that may help reliability:

  • You can ask to remove a disk and BlobFS will move the data off that disk so you can remove it without risking array degradation.
  • If there’s spare space on a disk, automatically fill it with more redundant data. This implies that at any given moment, the array will be (from the perpective of the programmer) completely full. When you add more data, you’re actually removing an old redundant copy of something and replacing it with the new data.
  • Spare space can also be used for journalling and backtracking. One of the big criticisms of RAID arrays without a separate backup is that you can destroy everything very quickly through software - there’s no way to back out of deletions or a virus attack. With journalling and backtracking, however, this concern disappears - if you can backtrack to yesterday’s state before you ran that dodgy porn dialer, you don’t need backup tapes quite so much.

Use case

Bob has a collection of movies and music burned on CD that doesn’t infringe copyright at all. He’s worried about CD’s degrading over time - alreay some of them have holes and bad sectors.

He installs a shiny new 400GB hard drive in his computer, sets up BlobFS on it and copies all of his stuff across. As he ‘obtains’ new media, he just saves them to the BlobFS volume.

Bob starts to worry about his hard drive failing, so he buys another 400GB drive and adds it to the BlobFS volume. BlobFS silently replicates all of his data to the new drive. He still has 400GB (ish) of capacity, but now he can tolerate the loss of either drive (safety margin of 1).

Bob discovers the movie Napoleon Dynamite and has to download every copy in every language. He runs out of space. By now, hard drives are bigger and faster, so he adds a 750GB drive to the volume. BlobFS does two things: it makes another 750GB of space available to Bob, and it uses the space (currently free) to add even more redundancy to the array. Immediately after adding the drive, Bob’s collection of illicit movies has a safety factor of 2 - he can lose any two drives in the volume and not lose any data.

Time passes, and one of the original 400GB drives fails. Bob recieves an email from BlobFS telling him so. He pulls out the drive, replaces it with a 1.25TB drive, and goes back to his popcorn. BlobFS rebalances the data automatically and makes more free space available if it can.

Risks

There are undoubtedly some major technical hurdles to deal with:

  • Automatic array restriping and maintaining safety in the event of a power loss
  • Redundancy across different-sized devices

This list is dangerously short. Obviously, I don’t understand the problem well enough yet to be able to make an accurate assessment of its difficulty. I’ve never written a filesystem. I love the idea of doing so; it’s a meaty problem with some fun algorithms/data structures/performance decisions. It has actual engineering elements to it.

I think I’d really enjoy this project. It is a very large project for one person, though. Without having done any systematic research, I think there’s tremendous demand for something like this. It’s just a big project with some bigtechnical challenges and only me potentially working on it.

I would love to work on this, I really would. But it’s not going to pay the bills for a long time. I think I’d have to be self-sufficient from other income sources first before I could work on this (and expect to make a living from it).

Monetization

  • Sell personal licenses. This will impede widespread use; I’m sure there are already commercial systems that have similar characteristics, but nobody wants to pay for them.
  • Sell commercial licenses. This is similar to what ReiserFS does - you can use it free at home, but make any money off it and you’re supposed to pay for it. Which seems fair enough. I’d probably attempt this and see how well it works out.
  • Sell storage hardware. This seems pretty inevitable, and I love embedded work. 2TB of reliable disks at a low low price? Sold.

Uh-oh. I can’t remember where I left the current TPS reports!

December 5th, 2007

I use lots of different computers. I have a laptop which goes with me everywhere. I have a desktop at home with a nice monitor and great performance. I have a desktop at one client’s office. Then there’s the backup server, mobile phone and iPod.

I have data that I use on all of these machines. Keeping it all up-to-date everywhere is a hassle.

What I want is a program that will link up all of these different computers and keep my data the same everywhere. If I’m working on my desktop and want to go out and work at a cafe, I should be able to just pick up my laptop and go, and all of my work and all of my email will be right there, completely up to date.

When I go to my client’s office, I want all of my work to be up-to-date and ready to go. If I put some new music on my laptop, I want it to appear on my iPod automatically. If I disappear on a holiday, I still want to have my most crucial emails and contacts available on my mobile phone.

I don’t want my personal emails to be saved on the client’s office computer, because I don’t control it. I don’t want my MP3 collection to be saved on my mobile phone, because it doesn’t have enough space. I don’t want my source code to be saved on my iPod, because I’ll just lose it anyway.

I need some magic to make all of this happen. This is what my file synchronizer will do.

I’m concentrating on syncing a laptop to a desktop right now. That solves my biggest day-to-day hassle - I like to work on my desktop (it’s faster and more pleasant) but I do need the portability afforded my my laptop fairly often. Among the people I’ve spoken to, it’s also their biggest sync-related problem - a lot of people use their laptops for everything (”having two computers is too much of a hassle! I’d rather just have one computer and carry it around with me.“) but hate the performance and the constant reconfiguration hassle (”I have to plug in all of the peripherals and change the screen resolution and the font size and the volume and the network setup and my wallpaper looks bad on the big monitor and then I have to go out again anyway and set it all up again.“).

So that’s the dream. Laptop, desktop. Your stuff is the same on both. Always. You don’t have to think about it or copy files. It just is.