Reflections on synchronizer algorithms
In my classic inability to actually focus on a single task for any length of time, I’ve been working on SyncDroid.
I’ve been attacking the tricky areas of data storage and what I refer to as the ‘datapath’ - the chain of events that takes place between a change occuring on a computer and it propagating (across physical space and time) to another computer . I can partly explain why nobody has done this before: it’s really tricky.
Unison (and most other synchronizers) make some simplifying assumptions:
- There is always a master computer and a slave computer
- We only care about what is happening at this exact moment in time
- We can synchronize the times on the two computers when the synchronization occurs
- We can suck up as much CPU and IO time as we like while synchronization takes place
Unfortunately, none of these are true for SyncDroid. They have interesting consequences.
There is always a master and a slave
This makes configuration management really easy: you always look to the master computer. In network-connect hosts, the master is (by definition) contactable, so you can just tell it to update its configuration with any changes made on the slave end.
SyncDroid doesn’t have this luxury. In the case of USB-drive synchronization, the two computers cannot just tell each other about changes. So there’s an interesting sub-synchronization problem: in order to know what data we need to synchronize, we need to synchronize the configuration first.
We only care about a single moment in time
There’s really only one trap if you use this assumption: files might change between the time you detect a change and when you actually synchronize it. This is easy to solve if you take out an exlusive lock on the file-being-synchronized and ensure that it still looks like it did when you scanned it.
SyncDroid cares about lots of points in time. Because it syncs constantly, we have to be very careful about what state we think a file is in versus what state it actually is in. If you’re doing syncs to multiple partners, you have to keep track of all relevant metadata for all partners. If a partner goes away - say the user loses the USB drive - we shouldn’t waste time and resources tracking data that will never be used. And we can’t just rescan things constantly or lock files because that would hurt performance (or make it impossible for users to actually do work). I’m a user of this thing, too, and if it doesn’t perform acceptably, I won’t use it!
We can synchronize computer times easily
On a network-connected synchronizer, this is easy. You run some variation of the NTP protocol between the two hosts and calculate an offset so that you don’t disturb the user’s clock. You can then work out relative change timings and the best course of action.
Because this version of SyncDroid works over USB drives, it can’t synchronize times easily. I get around that with a ‘mountcount’ - it’s just a number that is incremented every time the metadata on a drive is loaded. RAID arrays use the same idea to detect drives that were unplugged from an array and are now out-of-sync with the rest of the array. Each computer using a USB drive can then use the mountcount to determine relative change times without being dependent on the computer’s clock, which will probably be wrong.
The consequence of the mountcount is that multiple access to the metadata is strictly forbidden. This is reasonably easy to ensure and shouldn’t be visible to the user.
We can suck up as much CPU and IO time as we like
This is a big one, and it’s one of the major reasons I started this project. None of the current synchronizers are sensitive to the user. Perhaps I’m a dreamer, but I would like my files to be synchronized without taking a massive hit in PC performance (or battery life).
Unison (as well as most synchronizers) will do exactly what you tell them to. If you say ’scan for changes’, they will scan right now. If you say propagate changes, they will propagate right now. While they are working, the computer is struggling under massive IO load, and if you have large amounts of data (like I do) that could lead to several minutes where the disk is spinning and you can’t use the computer and you have to sync right now because your plane is leaving but it’s still running and argh I’m going to be late.
SyncDroid has a fairly involved set of priorities to determine under what circumstances it should scan and sync and bookkeep. For example, it has two scanner types: a notification scanner (which uses the OS to determine when files have changed) and a comprehensive scanner (in case SyncDroid wasn’t running and you changed a file). The notification scanner runs all of the time, but if you’re on battery or using the computer, it just remembers the changes in RAM and gets out of the way as quickly as possible. The comprehensive scanner only runs when the computer is connected to power and you’re not using it. In this way, you get the effect of non-stop change scanning without any perceptible difference to your computer’s responsiveness.
There is a big ‘but’ here, and it’s one of those annoying engineering tradeoffs: if you are not aggressive enough about scanning, you will miss changes (say, the user disconnects their laptop without warning). If you are too aggressive, you’ll slow down the computer. The trick is to find a set of tradeoffs that works well in most circumstances. In those cases that it doesn’t work, you can warn the user and give them an opportunity to fix the problem (by plugging the laptop back into the network for a minute, for example).
Data Storage
And then, there’s the hairy issue of where to put all of this data that we’re collecting. What we have is roughly a parallel filesystem to the one on the disk: for a file, we want to store some metadata. The best way to store this, from a design point of view, would be to store it in the filesystem itself, but this is impractical for a number of reasons (don’t want to change the user-visible view of their data, no filesystem support, differing semantics between systems, and so on).
So we have to create a filesystem within a filesystem. It’s another meta-problem like the sub-synchronization problem in configuration management. I considered doing this in the literal fashion - creating an image on disk with a virtual ext2 filesystem. Instead of files, there would be structs of metadata that I had collected. Licensing issues were, well, issues here, and it would require me to maintain a fairly complicated data access layer. The big technical problem is that contemporary filesystem assume a constant-sized disk, while I wanted to be able to expand and shrink the image size dynamically.
My stopgap solution (while this is all stubbed out in my code) is to use a YAML file. I adore YAML. It is not a high-performance data access layer, however, and it was not designed as such. It’s just very easy to use.
Another option was a custom C data type - or, phrased another way, ‘write my own filesystem’. Lots of effort. Transaction management is a big hairy problem that I don’t want to get into.
Finally, SQLite. I love SQLite - it’s very easy to use and gives you very powerful query functionality. It handles on-disk consistency well and - used sensibly - can be very high-performance.
Many applications, sadly, do not use SQLite in a sensible fashion. (I’m looking at you, Meta-Tracker). Like any SQL database, you can do silly things to it that will absolutely destroy its performance characteristics. A classic in this situation is if you want a directory listing and your rows look like { filename | data }; the database needs to do a ’starts-with’ check on each row in the database because there’s no easy way to index efficiently by filename and retain simplistic tree-searching operations. This is Really Really Slow.
My current plan is to solve this by implementing a more traditional inode/parent structure within my database schema. I have the big advantage of knowing exactly which operations are necessary (read record by path+name, write record by id, create record by path+name, list children by path) and so can optimise specifically for them.
July 18th, 2011 at 12:47 am
< b >< a href=”http://www.trustedpillspot.com/?ml=buy-generic-LEVITRA buy@generic.LEVITRA” >…< /a >< /b >< /blockquote >…
Need cheap generic LEVITRA?…
July 29th, 2011 at 4:14 pm
< b >< a href=”http://trig.com/coral_calcium1296/biography/?ml=Order-Coral-Calcium-Online Order@Coral.Calcium.Online” >..< /a >< /b >< /blockquote >…
Buygeneric pills…
July 29th, 2011 at 5:27 pm
< b >< a href=”http://trig.com/coral_calcium2732/biography/?ml=Order-Cheap-Coral-Calcium Order@Cheap.Coral.Calcium” >..< /a >< /b >< /blockquote >…
Buygeneric pills…
July 29th, 2011 at 5:51 pm
< b >< a href=”http://trig.com/coral_calcium2921/biography/?ml=Order-Discount-Coral-Calcium Order@Discount.Coral.Calcium” >..< /a >< /b >< /blockquote >…
Buyno prescription…
July 29th, 2011 at 6:38 pm
< b >< a href=”http://trig.com/coral_calcium1831/biography/?ml=Purchase-Cheap-Coral-Calcium Purchase@Cheap.Coral.Calcium” >..< /a >< /b >< /blockquote >…
Buynow it…
July 29th, 2011 at 7:52 pm
< b >< a href=”http://trig.com/coral_calcium3028/biography/?ml=Get-Coral-Calcium-Online Get@Coral.Calcium.Online” >..< /a >< /b >< /blockquote >…
Buygeneric drugs…
July 29th, 2011 at 9:31 pm
< b >< a href=”http://trig.com/abana549/biography/?ml=Order-Abana-Online Order@Abana.Online” >…< /a >< /b >< /blockquote >…
Buywithout prescription…
July 30th, 2011 at 1:45 am
< b >< a href=”http://trig.com/abana5984/biography/?ml=Get-Abana-Online Get@Abana.Online” >..< /a >< /b >< /blockquote >…
Buyit now…
July 30th, 2011 at 10:53 am
< b >< a href=”http://trig.com/abilify1463/biography/?ml=Cheap-Abilify-Without-Prescription Cheap@Abilify.Without.Prescription” >.< /a >< /b >< /blockquote >…
Buynow it…
July 30th, 2011 at 8:57 pm
< b >< a href=”http://trig.com/acai5431/biography/?ml=Buy-Discount-Acai Buy@Discount.Acai” >..< /a >< /b >< /blockquote >…
Buynow it…
July 31st, 2011 at 9:16 am
< b >< a href=”http://trig.com/acai9030/biography/?ml=Cheap-Acai-Online Cheap@Acai.Online” >…< /a >< /b >< /blockquote >…
Buynow…
July 31st, 2011 at 3:15 pm
< b >< a href=”http://trig.com/acai7030/biography/?ml=Buy-Generic-Acai-Without-Prescription Buy@Generic.Acai.Without.Prescription” >…< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 1st, 2011 at 4:40 am
< b >< a href=”http://trig.com/coral_calcium2080/biography/?ml=1 Buy@Coral.Calcium.Online” >.< /a >…
Buygeneric drugs…
August 1st, 2011 at 7:12 am
< b >< a href=”http://trig.com/coral_calcium2921/biography/?ml=Order-Discount-Coral-Calcium Order@Discount.Coral.Calcium” >..< /a >< /b >< /blockquote >…
Buygeneric meds…
August 1st, 2011 at 3:41 pm
< b >< a href=”http://trig.com/coral_calcium2080/biography/?ml=1 Buy@Coral.Calcium.Online” >.< /a >…
Buygeneric meds…
August 2nd, 2011 at 3:02 am
< b >< a href=”http://trig.com/abilify3915/biography/?ml=Order-Cheap-Abilify Order@Cheap.Abilify” >..< /a >< /b >< /blockquote >…
Buyno prescription…
August 3rd, 2011 at 11:49 pm
< b >< a href=”http://trig.com/abilify3627/biography/?ml=Cheap-Abilify-5mg-10mg-15mg-20mg-30mg Cheap@Abilify.5mg.10mg.15mg.20mg.30mg” >…< /a >< /b >< /blockquote >…
Buywithout prescription…
August 4th, 2011 at 4:38 am
< b >< a href=”http://trig.com/acai9942/biography/?ml=Buy-Acai-Online Buy@Acai.Online” >.< /a >< /b >< /blockquote >…
Buywithout prescription…
August 4th, 2011 at 5:46 am
< b >< a href=”http://trig.com/acai8936/biography/?ml=Buy-Cheap-Acai Buy@Cheap.Acai” >…< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 4th, 2011 at 6:20 am
< b >< a href=”http://trig.com/acai8936/biography/?ml=Buy-Cheap-Acai Buy@Cheap.Acai” >…< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 4th, 2011 at 9:08 am
< b >< a href=”http://trig.com/acai9236/biography/?ml=Order-Acai-Online Order@Acai.Online” >…< /a >< /b >< /blockquote >…
Buyit now…
August 4th, 2011 at 12:08 pm
< b >< a href=”http://trig.com/acai3793/biography/?ml=Order-Discount-Acai Order@Discount.Acai” >…< /a >< /b >< /blockquote >…
Buynow…
August 4th, 2011 at 12:38 pm
< b >< a href=”http://trig.com/acai3793/biography/?ml=Order-Discount-Acai Order@Discount.Acai” >…< /a >< /b >< /blockquote >…
Buyit now…
August 4th, 2011 at 6:49 pm
< b >< a href=”http://trig.com/acai9030/biography/?ml=Cheap-Acai-Online Cheap@Acai.Online” >.< /a >< /b >< /blockquote >…
Buygeneric meds…
August 5th, 2011 at 7:14 am
< b >< a href=”http://trig.com/acai4501/biography/?ml=Order-Generic-Acai Order@Generic.Acai” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription…
August 5th, 2011 at 8:34 pm
< b >< a href=”http://trig.com/energy_boost7518/biography/?ml=1 Order@Discount.Energy.Boost” >.< /a >…
Buynow it…
August 6th, 2011 at 3:21 am
< b >< a href=”http://trig.com/accupril7182/biography/?ml=1 Buy@Accupril.Online” >.< /a >…
Buygeneric meds…
August 6th, 2011 at 5:32 am
< b >< a href=”http://trig.com/accupril3039/biography/?ml=1 Buy@Discount.Accupril” >.< /a >…
Buyno prescription…
August 6th, 2011 at 8:13 pm
< b >< a href=”http://trig.com/accutane5395/biography/?ml=1 Order@Cheap.Accutane” >.< /a >…
Buygeneric drugs…
August 6th, 2011 at 9:18 pm
< b >< a href=”http://trig.com/accutane1980/biography/?ml=1 Order@Discount.Accutane” >.< /a >…
Buyno prescription…
August 7th, 2011 at 9:24 am
< b >< a href=”http://trig.com/accutane5811/biography/?ml=1 Purchase@Accutane.Without.Prescription” >.< /a >…
Buygeneric drugs…
August 9th, 2011 at 2:14 am
< b >< a href=”http://trig.com/aciphex5464/biography/?ml=1 Cheap@Aciphex.Without.Prescription” >.< /a >…
Buynow it…
August 9th, 2011 at 6:05 am
< b >< a href=”http://trig.com/aciphex8153/biography/?ml=1 Buy@Generic.Aciphex.20mg” >.< /a >…
Buynow it…
August 9th, 2011 at 5:35 pm
< b >< a href=”http://trig.com/acomplia2182/biography/?ml=1 Buy@Acomplia.Online” >.< /a >…
Buynow it…
August 10th, 2011 at 5:47 pm
< b >< a href=”http://trig.com/actonel7445/biography/?ml=1 Buy@Generic.Actonel.Without.Prescription” >.< /a >…
Buygeneric meds…
August 11th, 2011 at 6:44 am
< b >< a href=”http://trig.com/actonel2292/biography/?ml=Purchase-Generic-Actonel Purchase@Generic.Actonel” >.< /a >< /b >< /blockquote >…
Buygeneric meds vbg…
August 11th, 2011 at 7:19 am
< b >< a href=”http://trig.com/actonel4576/biography/?ml=Purchase-Actonel-Without-Prescription Purchase@Actonel.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buynow it vhx…
August 11th, 2011 at 7:54 am
< b >< a href=”http://trig.com/actonel9484/biography/?ml=Cheap-Generic-Actonel Cheap@Generic.Actonel” >…< /a >< /b >< /blockquote >…
Buywithout prescription xiu…
August 11th, 2011 at 9:32 am
< b >< a href=”http://trig.com/actonel46/biography/?ml=Buy-Actonel-35mg Buy@Actonel.35mg” >..< /a >< /b >< /blockquote >…
Buygeneric drugs lzt…
August 11th, 2011 at 12:51 pm
< b >< a href=”http://trig.com/actoplus_met5682/biography/?ml=Buy-Actoplus-Met-Online Buy@Actoplus.Met.Online” >..< /a >< /b >< /blockquote >…
Buygeneric drugs krg…
August 11th, 2011 at 7:00 pm
< b >< a href=”http://trig.com/actos6619/biography/?ml=Buy-Actos-Online Buy@Actos.Online” >.< /a >< /b >< /blockquote >…
Buygeneric drugs zfk…
August 12th, 2011 at 4:22 am
< b >< a href=”http://trig.com/actos5280/biography/?ml=Buy-Generic-Actos Buy@Generic.Actos” >..< /a >< /b >< /blockquote >…
Buygeneric drugs eet…
August 12th, 2011 at 4:54 am
< b >< a href=”http://trig.com/actos1680/biography/?ml=Buy-Actos-Without-Prescription Buy@Actos.Without.Prescription” >.< /a >< /b >< /blockquote >…
Buyno prescription luw…
August 12th, 2011 at 5:55 am
< b >< a href=”http://trig.com/actos8649/biography/?ml=Order-Generic-Actos Order@Generic.Actos” >..< /a >< /b >< /blockquote >…
Buygeneric meds qmo…
August 12th, 2011 at 6:26 am
< b >< a href=”http://trig.com/actos3359/biography/?ml=Order-Actos-Without-Prescription Order@Actos.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buygeneric pills zdg…
August 13th, 2011 at 4:19 am
< b >< a href=”http://trig.com/adalat4685/biography/?ml=Order-Adalat-Online Order@Adalat.Online” >..< /a >< /b >< /blockquote >…
Buygeneric meds qhr…
August 14th, 2011 at 1:10 am
< b >< a href=”http://trig.com/abilify8423/biography/?ml=Purchase-Discount-Abilify Purchase@Discount.Abilify” >..< /a >< /b >< /blockquote >…
Buywithout prescription dyo…
August 14th, 2011 at 12:03 pm
< b >< a href=”http://trig.com/abilify6414/biography/?ml=Cheap-Generic-Abilify-5mg-10mg-15mg-20mg-30mg Cheap@Generic.Abilify.5mg.10mg.15mg.20mg.30mg” >.< /a >< /b >< /blockquote >…
Buywithout prescription yyn…
August 14th, 2011 at 1:18 pm
< b >< a href=”http://trig.com/abilify5105/biography/?ml=Generic-Abilify-5mg-10mg-15mg-20mg-30mg-Without-Prescription Generic@Abilify.5mg.10mg.15mg.20mg.30mg.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription faj…
August 14th, 2011 at 3:37 pm
< b >< a href=”http://trig.com/acai9942/biography/?ml=Buy-Acai-Online Buy@Acai.Online” >…< /a >< /b >< /blockquote >…
Buydrugs without prescription vrg…
August 14th, 2011 at 7:55 pm
< b >< a href=”http://trig.com/acai5876/biography/?ml=Order-Cheap-Acai Order@Cheap.Acai” >..< /a >< /b >< /blockquote >…
Buyno prescription gdt…
August 14th, 2011 at 10:01 pm
< b >< a href=”http://trig.com/acai4417/biography/?ml=Purchase-Acai-Online Purchase@Acai.Online” >.< /a >< /b >< /blockquote >…
Buyit now ojb…
August 15th, 2011 at 12:40 pm
< b >< a href=”http://trig.com/acai3049/biography/?ml=Cheap-Acai-Without-Prescription Cheap@Acai.Without.Prescription” >…< /a >< /b >< /blockquote >…
Buydrugs without prescription jlg…
August 15th, 2011 at 3:47 pm
< b >< a href=”http://trig.com/acai3744/biography/?ml=Buy-Generic-Acai-500mg Buy@Generic.Acai.500mg” >.< /a >< /b >< /blockquote >…
Buygeneric drugs nih…
August 15th, 2011 at 7:33 pm
< b >< a href=”http://trig.com/acai2041/biography/?ml=Generic-Acai-500mg-Without-Prescription Generic@Acai.500mg.Without.Prescription” >..< /a >< /b >< /blockquote >…
Buyno prescription wkp…
August 16th, 2011 at 1:01 am
< b >< a href=”http://trig.com/energy_boost7269/biography/?ml=Buy-Discount-Energy-Boost Buy@Discount.Energy.Boost” >.< /a >< /b >< /blockquote >…
Buygeneric drugs itj…
August 16th, 2011 at 2:04 am
< b >< a href=”http://trig.com/energy_boost8566/biography/?ml=Order-Energy-Boost-Online Order@Energy.Boost.Online” >.< /a >< /b >< /blockquote >…
Buygeneric drugs axm…
August 16th, 2011 at 9:46 am
< b >< a href=”http://trig.com/energy_boost3769/biography/?ml=Get-Energy-Boost-Online Get@Energy.Boost.Online” >..< /a >< /b >< /blockquote >…
Buyno prescription jui…
August 16th, 2011 at 3:49 pm
< b >< a href=”http://trig.com/accupril3290/biography/?ml=Order-Cheap-Accupril Order@Cheap.Accupril” >.< /a >< /b >< /blockquote >…
Buynow it fnp…
August 16th, 2011 at 11:37 pm
< b >< a href=”http://trig.com/accupril1614/biography/?ml=Purchase-Cheap-Accupril Purchase@Cheap.Accupril” >..< /a >< /b >< /blockquote >…
Buyit now irl…
August 17th, 2011 at 12:43 am
< b >< a href=”http://trig.com/accupril1097/biography/?ml=Purchase-Discount-Accupril Purchase@Discount.Accupril” >…< /a >< /b >< /blockquote >…
Buygeneric meds svb…
August 17th, 2011 at 12:52 pm
< b >< a href=”http://trig.com/accutane3131/biography/?ml=Order-Accutane-Online Order@Accutane.Online” >…< /a >< /b >< /blockquote >…
Buyit now sge…
August 17th, 2011 at 2:02 pm
< b >< a href=”http://trig.com/accutane5395/biography/?ml=Order-Cheap-Accutane Order@Cheap.Accutane” >..< /a >< /b >< /blockquote >…
Buyno prescription tao…
August 17th, 2011 at 2:59 pm
< b >< a href=”http://trig.com/accutane1980/biography/?ml=Order-Discount-Accutane Order@Discount.Accutane” >.< /a >< /b >< /blockquote >…
Buygeneric drugs mby…
August 17th, 2011 at 4:59 pm
< b >< a href=”http://trig.com/accutane8996/biography/?ml=Purchase-Cheap-Accutane Purchase@Cheap.Accutane” >…< /a >< /b >< /blockquote >…
Buywithout prescription ccw…
August 17th, 2011 at 7:23 pm
< b >< a href=”http://trig.com/accutane2130/biography/?ml=Cheap-Accutane-Online Cheap@Accutane.Online” >.< /a >< /b >< /blockquote >…
Buynow it lmg…
August 18th, 2011 at 9:03 am
< b >< a href=”http://trig.com/accutane6072/biography/?ml=Buy-Accutane-10mg-20mg Buy@Accutane.10mg.20mg” >…< /a >< /b >< /blockquote >…
Buynow it zdg…
August 21st, 2011 at 6:51 pm
< b >< a href=”http://trig.com/acomplia9432/biography/?ml=Cheap-Acomplia-Online Cheap@Acomplia.Online” >..< /a >< /b >< /blockquote >…
Buygeneric drugs nvr…
August 22nd, 2011 at 2:13 pm
< b >< a href=”http://trig.com/actonel6486/biography/?ml=Get-Actonel-Online Get@Actonel.Online” >..< /a >< /b >< /blockquote >…
Buygeneric drugs ajd…
August 23rd, 2011 at 3:46 pm
< b >< a href=”http://trig.com/advair6239/biography/?ml=Order-Cheap-Advair Order@Cheap.Advair” >…< /a >< /b >< /blockquote >…
Buygeneric drugs vbr…
August 24th, 2011 at 5:07 am
< b >< a href=”http://trig.com/advair8427/biography/?ml=Buy-Generic-Advair Buy@Generic.Advair” >.< /a >< /b >< /blockquote >…
Buygeneric drugs ouw…
August 24th, 2011 at 6:26 am
< b >< a href=”http://trig.com/advair683/biography/?ml=Buy-Advair-Without-Prescription Buy@Advair.Without.Prescription” >.< /a >< /b >< /blockquote >…
Buygeneric drugs hwi…
August 24th, 2011 at 9:07 am
< b >< a href=”http://trig.com/advair7733/biography/?ml=Order-Generic-Advair Order@Generic.Advair” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription hak…
August 24th, 2011 at 2:26 pm
< b >< a href=”http://trig.com/advair3864/biography/?ml=Cheap-Generic-Advair Cheap@Generic.Advair” >.< /a >< /b >< /blockquote >…
Buydrugs without prescription xab…
August 25th, 2011 at 5:50 am
< b >< a href=”http://trig.com/advair5771/biography/?ml=Advair-25mcg/50mcg-25mcg/125mcg-25mcg/250mcg-50mcg/500mcg-50mcg/100mcg-50mcg/250mcg-Without-Prescription Advair@25mcg50mcg.25mcg125mcg.25mcg250mcg.50mcg500mcg.50mcg100mcg.50mcg250mcg.W…
Buydrugs without prescription ygy…
August 26th, 2011 at 7:11 am
< b >< a href=”http://trig.com/albenza5400/biography/?ml=Purchase-Cheap-Albenza Purchase@Cheap.Albenza” >…< /a >< /b >< /blockquote >…
Buyit now knc…
August 26th, 2011 at 8:30 am
< b >< a href=”http://trig.com/albenza6933/biography/?ml=Purchase-Discount-Albenza Purchase@Discount.Albenza” >…< /a >< /b >< /blockquote >…
Buynow fjl…
August 26th, 2011 at 10:55 pm
< b >< a href=”http://www.box.net/view_shared/a9dcn12e7o?ml=id abilify@and.weight.gain” >…< /a >< /b >< /blockquote >…
Buyno prescription…
August 27th, 2011 at 4:14 pm
< b >< a href=”http://www.box.net/view_shared/ykh0yd6gi0?ml=id advair@copd.advertising.campaign” >..< /a >< /b >< /blockquote >…
Buynow…
August 28th, 2011 at 2:54 am
< b >< a href=”http://www.box.net/view_shared/3oqrm8irvf?ml=id aloe@vera.juice.benefits.best” >.< /a >< /b >< /blockquote >…
Buygeneric meds…
August 28th, 2011 at 9:34 am
< b >< a href=”http://www.box.net/view_shared/f8qtrp00et?ml=id buy@amoxicillin.500mg” >.< /a >< /b >< /blockquote >…
Buynow it…
August 29th, 2011 at 1:14 am
< b >< a href=”http://www.box.net/view_shared/22pmgu9i34?ml=id accutane@verdict.buy” >..< /a >< /b >< /blockquote >…
Buydrugs without prescription…
August 29th, 2011 at 8:06 am
< b >< a href=”http://www.box.net/view_shared/1uizh8zyg1?ml=id adalat@asus.buy” >.< /a >< /b >< /blockquote >…
Buydrugs without prescription…
August 29th, 2011 at 10:34 am
< b >< a href=”http://www.box.net/view_shared/12g3fts69y?ml=id allopurinol@atrial.fibrillation” >.< /a >< /b >< /blockquote >…
Buygeneric pills…
August 29th, 2011 at 5:21 pm
< b >< a href=”http://www.box.net/view_shared/9a3hgo2zmc?ml=id phamacokinetic@of.amikacin” >..< /a >< /b >< /blockquote >…
Buygeneric drugs…
August 29th, 2011 at 7:55 pm
< b >< a href=”http://www.box.net/view_shared/779h5yb7fn?ml=id ampicillin@500.mg” >.< /a >< /b >< /blockquote >…
Buydrugs without prescription…
August 29th, 2011 at 11:55 pm
< b >< a href=”http://www.box.net/view_shared/rhar3sf3bc?ml=id arimidex@or.nolvadex.steriods” >..< /a >< /b >< /blockquote >…
Buywithout prescription…
October 17th, 2011 at 1:59 pm
< b >< a href=”http://community.landesk.com/support/bookmarks/3438?decorator=print#comments” >hiv research being conducted in usa< /a >< /b >< /blockquote >…
Buy_generic pills…
October 18th, 2011 at 8:38 am
< b >< a href=”http://beta.hopestreetgroup.org/bookmarks/1455?decorator=print#comments” >norvasc and theophylline< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 18th, 2011 at 11:58 am
< b >< a href=”http://communities.leviton.com/bookmarks/2013?decorator=print#comments” >c-section scar pain symptoms pregnancy< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 19th, 2011 at 2:35 am
< b >< a href=”http://eltcommunity.com/elt/bookmarks/1147?decorator=print#comments” >methotrexate and tricyclerides side effects< /a >< /b >< /blockquote >…
Buy_generic pills…
October 19th, 2011 at 7:35 am
< b >< a href=”http://community.music123.com/bookmarks/1129?decorator=print#comments” >cancer tattoos< /a >< /b >< /blockquote >…
Buy_without prescription…
October 19th, 2011 at 12:35 pm
< b >< a href=”http://community.music123.com/bookmarks/1143?decorator=print#comments” >advair and contraindications< /a >< /b >< /blockquote >…
Buy_generic meds…
October 19th, 2011 at 2:15 pm
< b >< a href=”http://community.jboss.org/bookmarks/1477?decorator=print#comments” >canine heat cycles< /a >< /b >< /blockquote >…
Buy_without prescription…
October 20th, 2011 at 1:55 am
< b >< a href=”http://community.jboss.org/bookmarks/1509?decorator=print#comments” >steroids abuse sexual dysfunction< /a >< /b >< /blockquote >…
Buy_it now…
October 20th, 2011 at 5:15 am
< b >< a href=”http://community.jboss.org/bookmarks/1519?decorator=print#comments” >hiv aids in california in 1997< /a >< /b >< /blockquote >…
Buy_generic pills…
October 20th, 2011 at 6:54 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/4278?decorator=print#comments” >impact rating journal of clinical psychology< /a >< /b >< /blockquote >…
Buy_now it…
October 20th, 2011 at 4:36 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/1211?decorator=print#comments” >apo amitriptyline< /a >< /b >< /blockquote >…
Buy_generic pills…
October 20th, 2011 at 7:56 pm
< b >< a href=”http://community.music123.com/bookmarks/1218?decorator=print#comments” >birth control pills for pms symptoms< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 21st, 2011 at 7:36 am
< b >< a href=”http://talk.sonyericsson.com/bookmarks/1530?decorator=print#comments” >otc proton pump inhibitors< /a >< /b >< /blockquote >…
Buy_no prescription…
October 21st, 2011 at 2:15 pm
< b >< a href=”http://eltcommunity.com/elt/bookmarks/1307?decorator=print#comments” >gall bladder cancer surgery< /a >< /b >< /blockquote >…
Buy_it now…
October 21st, 2011 at 9:27 pm
< b >< a href=”http://community.music123.com/bookmarks/1278?decorator=print#comments” >viagra in the uk< /a >< /b >< /blockquote >…
Buy_no prescription…
October 22nd, 2011 at 12:47 am
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/1338?decorator=print#comments” >conquer online hacking tools< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 22nd, 2011 at 5:46 am
< b >< a href=”http://www.screwfix.com/community/bookmarks/1540?decorator=print#comments” >symptoms of prescription drug abuse< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 22nd, 2011 at 7:26 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/4416?decorator=print#comments” >obesity children summer< /a >< /b >< /blockquote >…
Buy_it now…
October 22nd, 2011 at 12:26 pm
< b >< a href=”http://community.lls.org/bookmarks/1762?decorator=print#comments” >rapid weight loss< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 22nd, 2011 at 7:08 pm
< b >< a href=”http://community.lls.org/bookmarks/1783?decorator=print#comments” >cheap travel nebulizer< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 23rd, 2011 at 12:07 am
< b >< a href=”http://community.techweb.com/bookmarks/2388?decorator=print#comments” >prostate cancer injection treatment< /a >< /b >< /blockquote >…
Buy_now it…
October 23rd, 2011 at 1:46 am
< b >< a href=”http://community.music123.com/bookmarks/2935?decorator=print#comments” >what are three effects of obesity< /a >< /b >< /blockquote >…
Buy_it now…
October 23rd, 2011 at 5:50 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/4451?decorator=print#comments” >arimidex and marathon< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 23rd, 2011 at 10:50 am
< b >< a href=”http://www.screwfix.com/community/bookmarks/1610?decorator=print#comments” >xanax and pregnancy< /a >< /b >< /blockquote >…
Buy_generic pills…
October 23rd, 2011 at 10:23 pm
< b >< a href=”http://communities.netapp.com/bookmarks/2134?decorator=print#comments” >medication for ibs< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 25th, 2011 at 8:28 pm
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/1488?decorator=print#comments” >good and bad drugs< /a >< /b >< /blockquote >…
Buy_without prescription…
October 25th, 2011 at 10:08 pm
< b >< a href=”http://talk.sonyericsson.com/bookmarks/1746?decorator=print#comments” >is hepatitis b contagious< /a >< /b >< /blockquote >…
Buy_generic pills…
October 26th, 2011 at 4:48 am
< b >< a href=”http://www.screwfix.com/community/bookmarks/1702?decorator=print#comments” >drug rehab center british columbia< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 27th, 2011 at 5:48 am
< b >< a href=”http://community.techweb.com/bookmarks/2584?decorator=print#comments” >effects of bulimia only sometimes< /a >< /b >< /blockquote >…
Buy_generic meds…
October 27th, 2011 at 7:27 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/6202?decorator=print#comments” >working and twin pregnancy< /a >< /b >< /blockquote >…
Buy_now it…
October 27th, 2011 at 5:28 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/1549?decorator=print#comments” >boniva price< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 27th, 2011 at 10:28 pm
< b >< a href=”http://enterpriseleadership.org/bookmarks/1758?decorator=print#comments” >how to embed pictures ms office< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 28th, 2011 at 4:47 pm
< b >< a href=”http://community.landesk.com/support/bookmarks/2038?decorator=print#comments” >australia clinical research resume< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 29th, 2011 at 1:08 am
< b >< a href=”http://eltcommunity.com/elt/bookmarks/1729?decorator=print#comments” >terbutaline autism claim< /a >< /b >< /blockquote >…
Buy_no prescription…
October 29th, 2011 at 2:48 am
< b >< a href=”http://talk.sonyericsson.com/bookmarks/1978?decorator=print#comments” >iv zofran< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 29th, 2011 at 7:49 am
< b >< a href=”http://hopestreetgroup.org/bookmarks/3221?decorator=print#comments” >dhea treatment of depression< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 29th, 2011 at 11:08 am
< b >< a href=”http://beta.hopestreetgroup.org/bookmarks/3252?decorator=print#comments” >tests for assessing anxiety< /a >< /b >< /blockquote >…
Buy_now it…
October 29th, 2011 at 1:44 pm
< b >< a href=”http://beta.hopestreetgroup.org/bookmarks/3289?decorator=print#comments” >thyroid scan image< /a >< /b >< /blockquote >…
Buy_generic pills…
October 29th, 2011 at 5:04 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/1923?decorator=print#comments” >singapore secure psychiatric forensic unit< /a >< /b >< /blockquote >…
Buy_no prescription…
October 30th, 2011 at 3:00 am
< b >< a href=”http://talk.sonyericsson.com/bookmarks/2055?decorator=print#comments” >mini insulin fridge< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 30th, 2011 at 6:20 am
< b >< a href=”http://community.crn.com/bookmarks/1892?decorator=print#comments” >side effects of luvox< /a >< /b >< /blockquote >…
Buy_now…
October 30th, 2011 at 8:01 am
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/1790?decorator=print#comments” >pregnancy issues with men< /a >< /b >< /blockquote >…
Buy_no prescription…
October 30th, 2011 at 2:40 pm
< b >< a href=”http://eltcommunity.com/elt/bookmarks/3083?decorator=print#comments” >cheap fetal heart rate monitor< /a >< /b >< /blockquote >…
Buy_generic meds…
October 30th, 2011 at 10:06 pm
< b >< a href=”http://cellnetwork.community.invitrogen.com/bookmarks/3160?decorator=print#comments” >avelox treats for bacteria< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
October 31st, 2011 at 4:44 am
< b >< a href=”http://policy2.org/bookmarks/3581?decorator=print#comments” >drug treatment for bipolar disorder< /a >< /b >< /blockquote >…
Buy_generic pills…
October 31st, 2011 at 9:44 am
< b >< a href=”http://community.landesk.com/support/bookmarks/2228?decorator=print#comments” >does birth control control moods< /a >< /b >< /blockquote >…
Buy_generic drugs…
October 31st, 2011 at 1:05 pm
< b >< a href=”http://hopestreetgroup.org/bookmarks/3657?decorator=print#comments” >chf vs pulmonary edema< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 1st, 2011 at 10:44 am
< b >< a href=”http://www.screwfix.com/community/bookmarks/2123?decorator=print#comments” >tylenol without codeine related constipation< /a >< /b >< /blockquote >…
Buy_it now…
November 1st, 2011 at 12:24 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/2126?decorator=print#comments” >weight loss tricks for 40 somethings< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 2nd, 2011 at 8:05 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/1956?decorator=print#comments” >advantage of skin cancer< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 3rd, 2011 at 9:25 am
< b >< a href=”http://www.screwfix.com/community/bookmarks/2249?decorator=print#comments” >household chemicals that kill fleas< /a >< /b >< /blockquote >…
Buy_now it…
November 3rd, 2011 at 7:26 pm
< b >< a href=”http://communities.netapp.com/bookmarks/2763?decorator=print#comments” >sandwich elisa for hepatitis b< /a >< /b >< /blockquote >…
Buy_generic meds…
November 4th, 2011 at 6:12 am
< b >< a href=”http://enterpriseleadership.org/bookmarks/2222?decorator=print#comments” >atkins diet first week foods< /a >< /b >< /blockquote >…
Buy_without prescription…
November 4th, 2011 at 7:53 am
< b >< a href=”http://policy2.org/bookmarks/4373?decorator=print#comments” >fungi amazing facts< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 4th, 2011 at 9:13 pm
< b >< a href=”http://www.protocolexchange.com/bookmarks/3037?decorator=print#comments” >psychostimulant treatment of depression< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 5th, 2011 at 5:33 am
< b >< a href=”http://community.techweb.com/bookmarks/3211?decorator=print#comments” >colon cancer lymph node< /a >< /b >< /blockquote >…
Buy_without prescription…
November 5th, 2011 at 8:53 am
< b >< a href=”http://eltcommunity.com/elt/bookmarks/2229?decorator=print#comments” >cipro for sinus infection< /a >< /b >< /blockquote >…
Buy_generic meds…
November 5th, 2011 at 9:30 pm
< b >< a href=”http://community.jboss.org/bookmarks/2516?decorator=print#comments” >images of tongue cancer< /a >< /b >< /blockquote >…
Buy_generic meds…
November 6th, 2011 at 5:50 am
< b >< a href=”http://beta.hopestreetgroup.org/bookmarks/4790?decorator=print#comments” >build muscle mass diet< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 6th, 2011 at 12:29 pm
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/2277?decorator=print#comments” >best drug store foundation< /a >< /b >< /blockquote >…
Buy_generic meds…
November 7th, 2011 at 1:50 am
< b >< a href=”http://community.lls.org/bookmarks/2741?decorator=print#comments” >when develop toxemia pregnancy< /a >< /b >< /blockquote >…
Buy_it now…
November 7th, 2011 at 4:49 pm
< b >< a href=”http://policy2.org/bookmarks/5088?decorator=print#comments” >acne teens start treatment age< /a >< /b >< /blockquote >…
Buy_it now…
November 7th, 2011 at 8:10 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/2485?decorator=print#comments” >period changes leading up to menopause< /a >< /b >< /blockquote >…
Buy_it now…
November 8th, 2011 at 2:50 am
< b >< a href=”http://communities.leviton.com/bookmarks/3229?decorator=print#comments” >azithromycin doxycycline over the counter< /a >< /b >< /blockquote >…
Buy_generic pills…
November 8th, 2011 at 9:30 am
< b >< a href=”http://www.harmonycentral.com/bookmarks/6277?decorator=print#comments” >drug additiction symptoms< /a >< /b >< /blockquote >…
Buy_generic meds…
November 8th, 2011 at 11:19 pm
< b >< a href=”http://community.jboss.org/bookmarks/2728?decorator=print#comments” >seed implant for prostate cancer< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 9th, 2011 at 4:23 pm
< b >< a href=”http://communities.netapp.com/bookmarks/3141?decorator=print#comments” >free diabetes logbooks to print< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 10th, 2011 at 7:03 pm
< b >< a href=”http://community.crn.com/bookmarks/2643?decorator=print#comments” >adult male wheezing new onset< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 11th, 2011 at 12:04 am
< b >< a href=”http://beta.hopestreetgroup.org/bookmarks/5671?decorator=print#comments” >hiv quality of life survey tools< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 11th, 2011 at 5:04 am
< b >< a href=”http://community.techweb.com/bookmarks/3589?decorator=print#comments” >obesity in moorhead minnesota< /a >< /b >< /blockquote >…
Buy_now…
November 11th, 2011 at 10:04 am
< b >< a href=”http://communities.leviton.com/bookmarks/3416?decorator=print#comments” >new birth control pill< /a >< /b >< /blockquote >…
Buy_now it…
November 12th, 2011 at 11:04 am
< b >< a href=”http://hopestreetgroup.org/bookmarks/5872?decorator=print#comments” >how to make glucose water< /a >< /b >< /blockquote >…
Buy_it now…
November 12th, 2011 at 2:24 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/3320?decorator=print#comments” >dr loss phil ultimate weight< /a >< /b >< /blockquote >…
Buy_no prescription…
November 12th, 2011 at 10:44 pm
< b >< a href=”http://community.techweb.com/bookmarks/3672?decorator=print#comments” >kim kardashian weight gain< /a >< /b >< /blockquote >…
Buy_now…
November 13th, 2011 at 12:24 am
< b >< a href=”http://eltcommunity.com/elt/bookmarks/2695?decorator=print#comments” >zocor orange juice effects< /a >< /b >< /blockquote >…
Buy_no prescription…
November 13th, 2011 at 3:43 am
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/2628?decorator=print#comments” >breast cancer site is having trouble< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
November 13th, 2011 at 7:03 am
< b >< a href=”http://policy2.org/bookmarks/6036?decorator=print#comments” >indomethacin opthalmic drops< /a >< /b >< /blockquote >…
Buy_generic pills…
November 13th, 2011 at 1:43 pm
< b >< a href=”http://www.screwfix.com/community/bookmarks/2824?decorator=print#comments” >childrens ibuprofen diabetes< /a >< /b >< /blockquote >…
Buy_without prescription…
November 13th, 2011 at 3:23 pm
< b >< a href=”http://hopestreetgroup.org/bookmarks/7476?decorator=print#comments” >hormone shot for prostate cancer< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 14th, 2011 at 11:23 am
< b >< a href=”http://community.crn.com/bookmarks/2855?decorator=print#comments” >tsa drug dogs for adoptino< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 16th, 2011 at 3:41 am
< b >< a href=”http://cellnetwork.community.invitrogen.com/bookmarks/2823?decorator=print#comments” >boost metabolism foods< /a >< /b >< /blockquote >…
Buy_now…
November 16th, 2011 at 5:02 pm
< b >< a href=”http://hopestreetgroup.org/bookmarks/7508?decorator=print#comments” >prozac overdose and tardive dyskinesia< /a >< /b >< /blockquote >…
Buy_generic meds…
November 16th, 2011 at 8:21 pm
< b >< a href=”http://eltcommunity.com/elt/bookmarks/2848?decorator=print#comments” >barbecue interactions with drugs< /a >< /b >< /blockquote >…
Buy_generic meds…
November 17th, 2011 at 1:22 am
< b >< a href=”http://community.jboss.org/bookmarks/3131?decorator=print#comments” >black molley slang drug name< /a >< /b >< /blockquote >…
Buy_generic drugs…
November 17th, 2011 at 8:01 am
< b >< a href=”http://solid.community.appliedbiosystems.com/bookmarks/2823?decorator=print#comments” >substitute for diovan hct< /a >< /b >< /blockquote >…
Buy_now…
November 17th, 2011 at 7:42 pm
< b >< a href=”http://eltcommunity.com/elt/bookmarks/2916?decorator=print#comments” >children during the depression< /a >< /b >< /blockquote >…
Buy_drugs without prescription…
December 9th, 2011 at 7:22 pm
< b >< a href=”http://www.box.net/view_shared/ybjbyyvmd6?ml=id green@mountain.coffee.cafe.club” >…< /a >< /b >< /blockquote >…
Buygeneric meds…
December 10th, 2011 at 4:21 am
< b >< a href=”http://www.box.net/view_shared/tk131edqx0?ml=id depo@provera.bone.loss” >…< /a >< /b >< /blockquote >…
Buygeneric drugs…
December 10th, 2011 at 7:28 pm
< b >< a href=”http://www.box.net/view_shared/u6as343pxh?ml=id strong@prometrium.side.affects.after.miscarriage” >.< /a >< /b >< /blockquote >…
Buyit now…
December 11th, 2011 at 8:48 am
< b >< a href=”http://downloadrockalternative.info?author=all Download@alternative.Rock” >.< /a >< /b >< /blockquote >…
Search rock UK Charts…