July marked the 10-year anniversary of my starting work at the Lab. I got a certificate and lapel pin (not that I ever have any lapels on which to put it). And also a bump in vacation time and 401k contributions.
I’m still working with the same group I started with on the same general applications. But a lot has changed in that time.
Falling into the Past
When I started, the existing development team had collapsed for unknown reasons. I joined a basically black-box pile of software written in Java with custom, homegrown libraries and support packages. No documentation. Some of the compiled code had no matching source code (that I could find anyway).
It was a bit like being thrown back in time by at least 10 years in how web applications were built. I can admit now that it didn’t take long before I felt a bit worried about whether that job was going to be at all fulfilling for me. The code was a mess of JSP files intermingling core business logic with HTML output (a la the bad old days of PHP). It couldn’t be run locally, so making a code change involved modifying source files, compiling the code, deploying to the test server, restarting the application server (which also hosted the production application, so restarts impacted users), and then testing the change. A round-trip process of 15+ minutes. It was brutal.
I came in to the job with web application experience and I knew that there were much better ways of developing web applications. It wasn’t fusion research–that’s a different department–it was a fairly mature world.
By the Fall of 2009 I had settled in enough that I felt comfortable (or maybe just desperate?) enough to pitch the idea of using an open-source framework for our applications. Essentially saying, “Let’s toss the existing codebase and start over.”
Thankfully, my manager (who I’m grateful is still my manager) was willing to hear me out. But, we had some restrictions. The IT group was run separate from the software team and they dictated our deployment environment. So whatever we did needed to still run on the Solaris operating system, using Oracle Weblogic for the application server, behind Oracle iPlanet web server, talking to an Oracle database.
Even in 2009 that technology stack was unpopular at best and approaching obsolete. Trying to figure out how to configure Weblogic or iPlanet resulted in reading a lot of forum posts like “How do I do this in Weblogic?” answer, “You don’t, use literally anything else.” And that was if you were lucky enough to find an answer at all.
I had experience working with Python/Django, but that as a non-starter. Can’t deploy a Django app on Weblogic. It was a Java world. And our IT staff wasn’t much interested in supporting anything they didn’t already know.
Stepping into the Present
I found the Grails framework which lives in the Java ecosystem (technically it’s Groovy, but it runs on the JVM and as far as the system was concerned it was Java). It looked promising, certainly a huge step forward from what we had. We could go from coding with both arms tied behind our backs to one arm in a sling (which I did literally for a few weeks when I broke my arm, but that was years later).
I prototyped a brand-new type of application for our system. A geospatial view of the data utilizing Google Earth in the browser (back when GE was a browser plugin). Using a real framework I had a functioning prototype in a couple of weeks and it became immediately obvious that this was going to be our future.
That prototype was fleshed out into a full application and has since been overhauled 5 times and remains one of our core applications. It would never have been possible without moving our development work to Grails.
Over the past decade I kept pushing on modernizing our infrastructure as much as we could. Eventually the IT staff were consolidated under the software manager and over time they were replaced with people willing to consider a world outside of Solaris.
We moved our infrastructure to RedHat Linux, migrated from SVN to Git, switched our application server to Tomcat, finally swapped iPlanet for Apache, and even dumped the Oracle database for Postgres. We were finally in a sane development world (not to mention saving, literally, tens of millions of dollars on hardware and licensing). The arm was out of the sling and we got a lot of work done.
We had added additional software team members throughout the years, but never had a real focused approach to creating software. We were making it up as we went along and if you looked at the code, it showed.
In some ways, this was a necessity. As I said, the software team was non-existent when I joined. I had no experience managing a “real” codebase. Certainly not one expected to last for years. But, at the same time, the broader group we supported–the people we were trying to make more effective via software–didn’t have any trust in the software team. We needed to give them wins quickly to show them it was worth their time to interact with us.
And that’s what we did. That first prototype gave them a way of working with their data they never had previously and they wanted more. So, rather than work out a disciplined approach to building software with no experience to guide us, we wrote code. A lot of code.
Building the Future
It’s now 2019 and, for myriad reasons, it’s time to leave Grails behind. This year I became the team lead for the software group so I’ve had a lot of decisions to make. But, this time I have 10 years of experience to draw on.
We’re moving our applications to Python/Django. Something we could have done in 2009 had the IT staff been supportive of the software needs. But it is what it is. The good thing here is that it means the codebase will start from scratch. And, having had 10 years of dealing with a codebase built on the whims of the individual developers, we are doing things differently.
Our legacy applications are functional and stable so we have time to do things right. I spent a huge amount of effort up front picking the tools we’ll use to maintain good code “hygiene.” That is, tools and processes that ensure code is written consistently, that tests have been written, that obvious bugs have been caught, that the tests actually run and pass, etc. This discipline is already making a big difference in code quality.
We also have the luxury of an existing suite of applications–which we have to migrate over to Python–but which means we know what we’re trying to build ahead of time. A lot of our work in the past 10 years was speculative: we think this would be helpful, let’s build it and see what the users think. And a lot of it was trying to meet user demands when the users didn’t really know what they wanted, just a vague idea.
Our first rebuilt-in-Python application went to production in October. It took longer than expected, but I feel really good about the quality of the code in it.
Retrospective
Supposedly, software developers in the Bay Area switch jobs, on average, every 3 years or less. So why have I stayed here for 10?
I like my job. I like the work I’m doing. I like my manager. I like my coworkers. I like that I live 6 miles away and don’t have to get on a highway to commute. I like the impact my work has. I like that I can effect meaningful changes that improve our products. They don’t happen fast or all at once, but I’m quite sure I’d be doing something else if we were still writing everything in house and deploying on Solaris.
The nature of the projects I work on prevents me from saying much of anything about them, but I find the work meaningful. I’m not spending my days trying to figure out how to get more ads in front of more people. Our software has international impact on a regular basis with the goal of improving not just national security, but global security.
I’m not going to get rich, but our health insurance is great, we’re saving money for retirement, we managed to get a house while prices weren’t insane, we’re building up long-term savings and work stays at work. I don’t bring a laptop home, I don’t work on the weekends. I don’t answer emails after dinner.
I like what I’m doing, so I’m going to keep doing it.