10 Years a Software Developer

November 7, 2019 8:06 pm

July marked the 10-year anniversary of my starting work at the Lab. I got a certificate and lapel pin (not that I ever have any lapels on which to put it). And also a bump in vacation time and 401k contributions.

I’m still working with the same group I started with on the same general applications. But a lot has changed in that time.

Falling into the Past

When I started, the existing development team had collapsed for unknown reasons. I joined a basically black-box pile of software written in Java with custom, homegrown libraries and support packages. No documentation. Some of the compiled code had no matching source code (that I could find anyway).

It was a bit like being thrown back in time by at least 10 years in how web applications were built. I can admit now that it didn’t take long before I felt a bit worried about whether that job was going to be at all fulfilling for me. The code was a mess of JSP files intermingling core business logic with HTML output (a la the bad old days of PHP). It couldn’t be run locally, so making a code change involved modifying source files, compiling the code, deploying to the test server, restarting the application server (which also hosted the production application, so restarts impacted users), and then testing the change. A round-trip process of 15+ minutes. It was brutal.

I came in to the job with web application experience and I knew that there were much better ways of developing web applications. It wasn’t fusion research–that’s a different department–it was a fairly mature world.

By the Fall of 2009 I had settled in enough that I felt comfortable (or maybe just desperate?) enough to pitch the idea of using an open-source framework for our applications. Essentially saying, “Let’s toss the existing codebase and start over.”

Thankfully, my manager (who I’m grateful is still my manager) was willing to hear me out. But, we had some restrictions. The IT group was run separate from the software team and they dictated our deployment environment. So whatever we did needed to still run on the Solaris operating system, using Oracle Weblogic for the application server, behind Oracle iPlanet web server, talking to an Oracle database.

Even in 2009 that technology stack was unpopular at best and approaching obsolete. Trying to figure out how to configure Weblogic or iPlanet resulted in reading a lot of forum posts like “How do I do this in Weblogic?” answer, “You don’t, use literally anything else.” And that was if you were lucky enough to find an answer at all.

I had experience working with Python/Django, but that as a non-starter. Can’t deploy a Django app on Weblogic. It was a Java world. And our IT staff wasn’t much interested in supporting anything they didn’t already know.

Stepping into the Present

I found the Grails framework which lives in the Java ecosystem (technically it’s Groovy, but it runs on the JVM and as far as the system was concerned it was Java). It looked promising, certainly a huge step forward from what we had. We could go from coding with both arms tied behind our backs to one arm in a sling (which I did literally for a few weeks when I broke my arm, but that was years later).

I prototyped a brand-new type of application for our system. A geospatial view of the data utilizing Google Earth in the browser (back when GE was a browser plugin). Using a real framework I had a functioning prototype in a couple of weeks and it became immediately obvious that this was going to be our future.

That prototype was fleshed out into a full application and has since been overhauled 5 times and remains one of our core applications. It would never have been possible without moving our development work to Grails.

Over the past decade I kept pushing on modernizing our infrastructure as much as we could. Eventually the IT staff were consolidated under the software manager and over time they were replaced with people willing to consider a world outside of Solaris.

We moved our infrastructure to RedHat Linux, migrated from SVN to Git, switched our application server to Tomcat, finally swapped iPlanet for Apache, and even dumped the Oracle database for Postgres. We were finally in a sane development world (not to mention saving, literally, tens of millions of dollars on hardware and licensing). The arm was out of the sling and we got a lot of work done.

We had added additional software team members throughout the years, but never had a real focused approach to creating software. We were making it up as we went along and if you looked at the code, it showed.

In some ways, this was a necessity. As I said, the software team was non-existent when I joined. I had no experience managing a “real” codebase. Certainly not one expected to last for years. But, at the same time, the broader group we supported–the people we were trying to make more effective via software–didn’t have any trust in the software team. We needed to give them wins quickly to show them it was worth their time to interact with us.

And that’s what we did. That first prototype gave them a way of working with their data they never had previously and they wanted more. So, rather than work out a disciplined approach to building software with no experience to guide us, we wrote code. A lot of code.

Building the Future

It’s now 2019 and, for myriad reasons, it’s time to leave Grails behind. This year I became the team lead for the software group so I’ve had a lot of decisions to make. But, this time I have 10 years of experience to draw on.

We’re moving our applications to Python/Django. Something we could have done in 2009 had the IT staff been supportive of the software needs. But it is what it is. The good thing here is that it means the codebase will start from scratch. And, having had 10 years of dealing with a codebase built on the whims of the individual developers, we are doing things differently.

Our legacy applications are functional and stable so we have time to do things right. I spent a huge amount of effort up front picking the tools we’ll use to maintain good code “hygiene.” That is, tools and processes that ensure code is written consistently, that tests have been written, that obvious bugs have been caught, that the tests actually run and pass, etc. This discipline is already making a big difference in code quality.

We also have the luxury of an existing suite of applications–which we have to migrate over to Python–but which means we know what we’re trying to build ahead of time. A lot of our work in the past 10 years was speculative: we think this would be helpful, let’s build it and see what the users think. And a lot of it was trying to meet user demands when the users didn’t really know what they wanted, just a vague idea.

Our first rebuilt-in-Python application went to production in October. It took longer than expected, but I feel really good about the quality of the code in it.

Retrospective

Supposedly, software developers in the Bay Area switch jobs, on average, every 3 years or less. So why have I stayed here for 10?

I like my job. I like the work I’m doing. I like my manager. I like my coworkers. I like that I live 6 miles away and don’t have to get on a highway to commute. I like the impact my work has. I like that I can effect meaningful changes that improve our products. They don’t happen fast or all at once, but I’m quite sure I’d be doing something else if we were still writing everything in house and deploying on Solaris.

The nature of the projects I work on prevents me from saying much of anything about them, but I find the work meaningful. I’m not spending my days trying to figure out how to get more ads in front of more people. Our software has international impact on a regular basis with the goal of improving not just national security, but global security.

I’m not going to get rich, but our health insurance is great, we’re saving money for retirement, we managed to get a house while prices weren’t insane, we’re building up long-term savings and work stays at work. I don’t bring a laptop home, I don’t work on the weekends. I don’t answer emails after dinner.

I like what I’m doing, so I’m going to keep doing it.

Do We Really Care?

February 20, 2018 12:59 pm

In September of 2001 the band P.O.D. released their song “Youth of the Nation” which begins with the lyrics:

Last day of the rest of my life
I wish I would’ve known
‘Cause I would’ve kissed my mama goodbye

I didn’t tell her that I loved her and how much I care
Or thank my pops for all the talks
And all the wisdom he shared

Unaware, I just did what I always do
Everyday, the same routine
Before I skate off to school

But who knew that this day wasn’t like the rest
Instead of taking a test
I took two to the chest

Call me blind, but I didn’t see it coming
Everybody was running
But I couldn’t hear nothing

Except gun blasts, it happened so fast
I didn’t really know this kid
He wasn’t part of the class

Maybe this kid was reaching out for love
Or maybe for a moment
He forgot who he was
Or maybe this kid just wanted to be hugged

Towards the end of the song is this stanza

Who’s to blame for the lives that tragedies claim
No matter what you say
It don’t take away the pain

When this song came out I was in high school.  I could still remember the reaction to Columbine which occurred when I was in middle school.  How could I have envisioned then that 16 years later we, as a nation, would have paid lip service over hundreds of bodies of adults and children about “never again” and then done precisely nothing to actually change the course of our society?

Honestly, I’m getting tired of trying to be nuanced about which gun owners are responsible and which aren’t, it’s about people not guns, it’s a mental-health issue, did the Founding Fathers intend for an armed population as a hedge against tyranny, blah, blah, blah, blah.  The endless blathering only seems to amount to yet another dead child, yet another dead mother, yet another dead father.

What we’re doing now, which is nothing, is not making the situation any better.

People who want to have continued access to firearms as part of their lifestyle need to stop hiding behind rhetoric and start proposing and implementing solutions.  I’m getting tired of holding a nuanced view on the matter while more people senselessly die.  I imagine there are more like me who, as time goes on, think that a “repeal and replace” of the 2nd Amendment might be the only way anything actually changes.

Research available options, pick a potential solution, plan and fund an implementation, study the outcome.  It really isn’t that hard.

Do we really care?

The answer seems to be, “No.”

The “Better than Nothing” Philosophy

January 19, 2018 2:21 pm

Many times the thing stopping us from getting something done or making a change in our lives is simply that we are daunted by what we think we “should” do and instead do nothing.

Doing nothing ensures nothing is accomplished.  Doing something, anything, will get us closer to our goal than just being overwhelmed by what we think we should do and instead doing nothing.  Start by doing something–even if it’s not remotely close to what you think you “should” be doing.

This is my “better than nothing” philosophy.

My most blatant example comes from work.  After years of lamenting we had no monitoring solution in place to notify us if our applications were not working I used a Hackathon day to create “Better than Nothing Monitoring”.  It’s not great.  It’s probably not even good.  But, it’s better than nothing; which is what we had before.  Something is better than nothing.

Also from work, having a robust test suite helps catch bugs in our software before we deploy it to users.  For a long time we didn’t have any tests.  Now we have some, but not nearly enough.  When I’m working on an old piece of code, rather than lament not having the perfect test suite or the time to build it now, I’ll implement the simplest kind of test: “this thing runs without throwing an exception.”  We should have better testing than that, but, when we don’t, it’s better than nothing.

Feel like you should be saving more money?  You sit down, calculate out how much you “should” be saving per paycheck, but never seem to manage to do it and end up saving nothing?  Ignore the “should” and start saving something.  Maybe start with $5 that you put in a savings account every paycheck.  It’s not much.  It’s barely anything.  But it’s better than nothing.  If you’re paid bi-weekly, at the end of the year you’ll have $130.  That’s not much either, but it’s more than you would have saved fretting over what you “should” have done.

Feel like you should be exercising more?  Do you think, “I really should go for a run this week, or ride my bike, or play Wii Fit,” but you never quite seem to make it happen?  Stop worrying about it.  Start so small you barely notice: if you’re using the elevator at work, take the stairs one or two floors first; do a couple push-ups or sit-ups before going to bed.  Maybe you can only do two push-ups today.  That’s better than nothing.

Wish you read more books, but getting through even one chapter just takes too long?  Read a couple of paragraphs at a time instead.  You’ll get through more books reading 3 paragraphs a day than you will wishing you had enough time to read full chapters.  It’s better than nothing.

Many times I finish up a task at work and pull up our work log to decide what to do next.  I’ve long since completed all the low-hanging fruit and I scan through the list of tickets thinking, “Uggghh, that’s going to take forever, I’ll pick something else.”  And, big surprise, those work items are still in the queue months later.  Eventually you just have to start.  Pick the smallest piece of the puzzle that makes any sense and get just that piece done.  The overall task isn’t done, but it’s closer than it was.  It’s better than nothing.

Most of these examples are habit items: saving money, exercising, reading.  In these cases the actual accomplishment today is less important than the habit you’re creating.  You can increase your daily, better-than-nothing effort to get gradually closer to your “should” level of effort.  Doing 2 push-ups and 5 sit-ups every day takes about 45 seconds.  That’s clearly not anything near the 150-minutes per week of moderate physical activity recommended by the American Heart Association, but it’s better than nothing.  As you get stronger, 2 & 5 can fairly seamlessly become 5 & 10, then 10 & 20, then 20 & 40 and now you might be exercising for 7 minutes a day.  Still not what you “should” be doing, but worrying that you “should” be doing 20 minutes a day and doing nothing is worse than doing 7 minutes a day.

What is it you’re trying to accomplish?  Pick any activity, no matter how small, towards that goal which you can start doing today.  Go do it.

It’s better than nothing.

Further Thoughts on My Experiences with the Court

October 27, 2017 2:33 pm

I was summoned to jury duty last week and was selected in to the pool for a criminal case.  We met to hear the charge and receive instructions from the judge.  Then we filled out questionnaires and were asked to return this week–unless called and told otherwise.  The charge in the case was rape.  A somber topic.  The outcome of the case is going to affect the lives of many people for years to come.  And I took the responsibility of (at that point only potentially) sitting in judgment very seriously.  Yesterday, when my name was read to sit as juror number 2 “it got real” (as they say).

After sitting through a few hours of voir dire, the defense attorney asked that I be excused from the jury.  They do not give a reason for a juror’s excusal.  The few in-person questions directed at me were rather benign.  The defense attorney asked something like, “Is it possible that a police officer may color his remarks to reflect a certain view?” which I agreed with.  The prosecutor asked something like, “How would you feel if you learned there was some piece of information like a police report that would not be available to you?” which I said would probably be frustrating to wonder what was in the report, but I would do my best to work with the facts presented.

Based on the verbal questioning, I would have expected the defense attorney to prioritize other jurors for excusal before me.  For example, the woman who stated she would essentially trust anything a police officer said without question; or the man who admitted he had a hard time seeing the defendant as not having done something if he was here in court; or the woman who said she would, as a default position, believe the testimony of the victim because she felt that a woman wouldn’t lie about being raped.

I would have to assume it was the answers I gave in my questionnaire that led to my excusal (I’m just not sure why that didn’t happen during the week they reviewed our answers and informed some people they did not need to return).

On the questionnaire they asked if we knew anyone who had been a victim of sexual assault.  I responded that I had a friend in college who was raped by another student while on a date with him.

They also asked if we knew anyone that worked with victims’ groups or crisis centers.  I responded that Jess was a volunteer, rape-crisis counselor when we met.

I expected those two responses to put me pretty high on the list of people the defense would not want around; though I do feel I could have acted fairly and conscientiously.  I do not feel angry or have a desire for vengeance that I would misplace on to the defendant.

Once my name was called to take seat number two in the jury box I fully expected to answer further questions about those responses.  Since I didn’t, I can only assume that the defense attorney intended to excuse me as soon as he knew I would be called.

The story of my friend who was assaulted is not mine to tell in full, but I want to share some parts and I believe I have sufficiently obfuscated any identifying information to protect her anonymity.

Her attacker was another BYU student, a returned missionary, a supposedly-righteous priesthood holder (for those LDS readers who believe that should mean something).  He threatened her life if she went to the police and she fully believed him capable of following through with the threat.  She came to me for help some days or weeks later, I don’t know the exact timeline.  Not that there was anything I could do but try to console her.

She was too scared to go to the police. Not just scared of her attacker (who made repeated threats, in person, over the course of several weeks).  She was scared that she would not be able to remain anonymous.  Scared that if she reported it she had no evidence, would not be believed, and nothing would happen.  Scared no man would want to be with her knowing she had been raped (YW lessons about chewed gum and licked cupcakes are life-destroying, Elizabeth Smart can provide more insight on that).  Scared about her status in school.

(Sadly, that last concern was well founded given that BYU was recently mired in controversy over mishandling sexual assault reports and subsequently taking disciplinary action against women reporting them.)

Were I continuing on as a juror on the trial, I honestly don’t know how I would have tried to handle the competing demands of being a compassionate human being, wanting to let a victim know that she will be believed if she speaks up, not be accused of being yet-another-man who oppresses and dismisses women–to balance that with the requirement to hear her testimony as one part of the trial, to presume the defendant innocent, to weigh the facts of the case as a whole.  How can you, potentially, say, “I believe you, but the prosecutor didn’t convince me past reasonable doubt?” or, “I don’t believe you, but other women should still speak up.”  Actions speak louder than words and I can understand why anyone would feel that returning a verdict of not-guilty in such a case is equivalent to saying, “we don’t believe women who claim to have been raped.”  But at the same time I’m very wary of accepting an accusation per se as sufficient evidence to convict someone.  Unfounded accusations can destroy lives too.

It sucks.  The whole thing sucks.

For my own mental well-being I think it’s good I was excused from the jury.  I was burned out when I got home just from grappling with these thoughts throughout voir dire.  I would have been exhausted and beyond stressed out by the time the trial concluded.

I don’t have a point I’m trying to make.  I’m just trying to apply some order and closure on to these thoughts that have been keeping me up at night.