Problems of scoping down for shoestring budgets
Giant Rabbit LLC
July 2013 to Jun 2016
How do we provide the same quality of service but at half or a fourth the cost?
Dissatisfied with their existing toolset, nonprofit staff often came to truly believe another product would serve them better. And so they embark on the trying, months long process of migrating their data and business practices to a new system. That’s where we came in. Having migrated dozens of organizations from one system to another, we had became intimately familiar with common pitfalls, oversights, and challenges with the process. As the lead developer on many of these projects, my team trusted me with the freedom to iterate every single time.
When I helped the Electronic Frontier Foundation migrate to a new system, we scripted the process. But that turned out to be extremely expensive; more so than what most other organizations could afford. Next, I tried spreadsheets. Replicating a relational database structure by manually moving columns into different files and files into different directories was rife with human error. So, I dug a little deeper and found a tool called Open Refine. It was specifically designed to handle complex data manipulation and it saw me through many successful migrations. But the translation process was never fully replicable from start to finish. Manual intervention was still required here and there. Which meant, clients would be locked out of any system for weeks during a cutover.
Finally, I returned to the possibility of scripting a migration, but on a much smaller budget. Instead of trying to script soup to nuts, I left out the most expensive part: automatically configuring the new system based on an organization’s existing data structure. Not only was it the most expensive part of a scripted migration, but configuring the new system required reflection. If automated, clients weren’t given the chance to re-examine how they had been using technical tools to support their work and whether their existing ways of working with technology was doing more harm than good. If their old system was so bad they wanted to leave it, why recreate it?
With guidance from a colleague, I ventured beyond the standard languages used at our firm and into the wonderful world of Python. A few videos and online tutorials later, I began writing out all the transformations required for this migration. From the beginning, I hoped this application could be used again by other colleagues on other migration projects. And so took time along the way to refactor, review variable name choices, rework data structures, add tests, update comments, and clarify logic. We were able to practice the migration many times, and when the client handed us their final export for cut over, I had them back up and running in their new system by the end of the day.
Not long after wrapping up this work, the colleague who suggested I try Python, asked if he could use this work on another project. Absolutely! I was delighted that the strategy proved successful and that the investment in new scripts could benefit other colleagues and clients going through a similar process.