The Weather in our Kingdom

Jess got me a weather station for my birthday which is now installed up on the roof:

The unit is an Ambient Weather WS-1400-IP.  It reports temperature, humidity, wind speed, wind direction, rainfall, solar radiation, etc.  It comes with an indoor unit that also reports inside temperature and humidity.

The data is sent to wunderground.com and you can find it here: https://www.wunderground.com/personal-weather-station/dashboard?ID=KCALIVER107

But, wunderground can be a little flaky, so I’m also capturing the data into my own database and serving it up.  On the sidebar of the blog you can find a widget that looks like this:

I’m using the “Weather Station” WordPress plugin to read a Cumulus-style file.  Now, the Ambient Weather ObserverIP unit does not produce the Cumulus “Realtime.txt” file that the WordPress plugin needs.  But, I have programming super powers.  So I wrote a shim that scrapes the data from the ObserverIP web interface and writes out a “Realtime.txt” file that I serve up for the WordPress plugin.

I also write out a human-readable page with the weather data on it you can see here: http://weather.serindu.com/

It’s not very pretty right now, but it’s up and running, updating every 5 minutes.  I’ll get around to improve the aesthetics at some point.

Having the indoor and outdoor sensors I’m thinking I’ll have to write up something that will notify me when the temperatures outside and inside cross so I’m alerted to open or close the windows as appropriate.  But I haven’t got that far yet.

The shim I wrote to put all these pieces together is available on GitHub: https://github.com/kdickerson/weather

It’s just a Python script set to run every 5 minutes via Cron.  It scrapes the data off the ObserverIP unit, formats it and inserts it into the SQLite database, computes daily high/low values from the stored data, and writes Realtime.txt and index.html into a folder being served by Apache.

How’s That Roof Working Out?

Our house needed a new roof when we bought it.  There wasn’t anything particularly wrong with the roof, but it was 30 years old and worn out.  When we replaced the roof we went with light-colored shingles, added ridge venting, used radiant-barrier sheathing, and upgraded the attic insulation to a minimum of R38.

Well, now it’s been a few years and I have some data (thanks PG&E for making usage history available for download in CSV format) so it’s time to see how well all that stuff is working out for us.

Before we start, some caveats about the data.  I only have 1 year of data from before the roof replacement, so it’s not very stable data.  For the 3 years after, I’ve averaged each month to obtain more stable information.  The graphs start in December because the new roof was put on during October.  So December is the first full bill after the roof was completed.

The gas data is very consistent.  We have a gas dryer, gas water heater, and gas furnace.  Our laundry and hot-water usage is probably fairly consistent throughout the years (with a small increase in both when Corinne joined the family), so the change in therms seen from before to after should be pretty focused on heating.  We’ve heated the house to approximately the same temperature throughout each winter so it should be a very stable comparison.

The electrical usage, however, is much less stable.  Usage patterns have changed as Heather has grown, we’ve switched TVs, upgraded computers, more hard drives spinning, more often running dishwasher, etc.  So the comparisons for usage from before and after are much more muddied.

Let’s go to the data.

Gas

The gas usage shows an obvious reduction in gas usage throughout the winter months with equivalent usage during the summer.  This seems like pretty solid evidence that the upgraded insulation is making a difference.

The total therms used in 2013 was 503.  The average therms per year from 2014-2016 was 412.7.  An 18% reduction.  At an average of $1.29 per therm, this results in annual savings of $117.

Electricity

Sadly, the electrical usage doesn’t show an obvious dip during the summer months.  It actually shows a dip for the winter months which I presume must be related to the cost of running the blower on the furnace which didn’t need to run as much as evidenced by the reduced gas usage.

Regarding the summer months, the 2013 data is not very good to begin with.  This was our first year in the house and we were adjusting our usage.  The spike in July 2013 would have been our first hot month in the house (July bill, for June usage) and we ran the air conditioner liberally.  When we saw the power bill, we adjusted the air conditioning to reduce costs as seen in significant drop in usage in August.

The uptick in January is most likely due to Christmas lights.

Although there was not a clear drop in usage throughout the summer as I was hoping, there was an overall reduction. The total kilowatt-hours used in 2013 was 4599.  The average usage per year in 2014-2016 was 4344.3.  A reduction of 6%.  At an average of $0.17 per kWh this results in annual savings of $42.  However, due to the nature of the 2013 data the validity of this claim is suspect.

Due to the many confounding variables on the electrical usage (mentioned in the opening paragraphs), I don’t think this data can say that the high-reflectivity shingles, ridge venting, and radiant barrier were ineffective upgrades, but clearly they weren’t obvious wins either, about which I’m a bit disappointed.

If we assume the data is valid as presented then the net change is $159 per year in savings.  If we were to assume the entire benefit seen is due to the combined effect of the insulation and roof upgrades (and not changes in usage patterns), then the break even point of the upgrades would be ~38 years.  Which is longer than the expected life of the roof (30 years).  However, the insulation and radiant barrier are one-time expenses.  Unless the roof fails catastrophically neither one should need to be replaced when the next roof is installed.

The net effect is that my data doesn’t show the upgrades to have been a definitive win compared to a standard roof, however, I believe the electricity usage data is too inconsistent from year to year to be reliable to make any strong claim.  If I had a few more year’s worth of data from before the roof replacement I’d be able to make stronger claims about the effect.

Facebook’s Image Recognition/Tagging System

I’ve noticed recently that Facebook has been applying an automatic image recognition/tagging process to any pictures that are uploaded to their system.  The general idea is to get a computer to “understand” what’s in a picture.

In Facebook’s case, images are given a textual description automatically which is used to provide meta-data that can be accessed by screen-readers (used by the visually impaired when using computers).  This isn’t about facial recognition (though Facebook is doing that as well).

For example, I came across this picture from a friend of mine:

Which Facebook’s artificial intelligence system automatically tagged with this information:

Image may contain: sky, twilight, outdoor and nature

Automatic image tagging/recognition has been a popular field of research, but this is the first time I’ve seen it actually used in a production system for something useful.

It’s pretty good too.  Here’s one of interest to the Dickerson family at large:

Which Facebook automatically describes as:

Image may contain: 8 people, people smiling, people standing and indoor

I just found that interesting.  That’s all.

Who’s waging war against HTTPS?

In April 2016, Let’s Encrypt went live.  Let’s Encrypt is a group making it significantly easier to encrypt web traffic.  Some entity seems to have begun waging a war of public opinion against them.

Previous to their existence conveniently securing web traffic meant paying money to a company which would then provide you with a “certificate” for your website.  Servers and browsers use these certificates to create a secure communication path between them.  This secure path (denoted by URLs starting with “https://” rather than “http://”) prevents entities between your computer and the website from seeing or altering the data being sent to and from the website.

Because of the cost and inconvenience many websites used unsecured connections.  However, places like banks, shopping, and healthcare providers have pretty much always used secure connections.  It took a few years but eventually social media websites began using secure connections by default as well.

Before Let’s Encrypt, millions of websites only had their content available via unsecured communications.  For many people, like myself, running websites without any goal of making money from them the expense and hassle of certificates wasn’t worth it.  Now, my websites are all available through secured connections, for free, thanks to Let’s Encrypt.  (To be clear, many websites still haven’t taken advantage of this service yet, but they at least have the option now.)

But, if banks and such use secure connections anyway, why do we care about Let’s Encrypt, should I care if “someone” can see that I’m reading this blog post?

Maybe.

On March 28 Congress voted to repeal FCC regulations that prevented your Internet Service Provider (ISP) from spying on your web traffic and using that information to their financial benefit.  The regulations also prevented ISPs from altering your web traffic for similar purposes (e.g., injecting ads into a webpage when you view it).

Maybe you don’t care if Comcast, or AT&T, or Verizon knows you like to knit and shop at JoAnn’s Fabrics.  But maybe you’d be concerned if they started selling information to other companies about you visiting cancer treatment websites, or rape support groups, or divorce attorneys, or any number of kinds of sensitive information.

Using encrypted connections doesn’t solve this problem entirely, but it makes the information available to your ISP a lot less useful.  For example, your ISP would still be able to tell you’re looking at Amazon.com, but they wouldn’t be able to tell if you’re looking at knitting needles or books about infertility treatments.

Regardless of your stance, someone seems to be working hard to turn public opinion against Let’s Encrypt and again make it harder to encrypt web traffic.  Articles like this one: “14,766 Let’s Encrypt SSL Certificates Issued to PayPal Phishing Sites” have been showing up all over the Internet recently, all making similar claims that it is Let’s Encrypt’s fault that people are falling for fake PayPal scam websites.

I don’t think it’s actually PayPal behind these articles, because this problem is nothing new, but the concerted, direct attack on Let’s Encrypt is new.

Let’s Encrypt does not verify the identity of the person requesting a certificate (which other certificate providers will do for steep fees, $300+ per year, these “verified” certificates are significantly different than the “non-verified” certificates issued by Let’s Encrypt).  Instead Let’s Encrypt verifies that you control the website for which you’re requesting a certificate, slightly different.

The argument made by these articles is that now someone can get _a_ certificate for “paypall.com” and people will think that the green lock icon on their browser means they’re connected to “paypal.com” instead.  Which it doesn’t and never has.  The “verified” certificates show up differently in your browser.  For example, on this blog you’ll see something like this:

With a “verified” certificate you’ll instead see something like this:

This indicates that the company issuing the certificate verified that the company requesting the certificate is “PayPal, Inc.” and the certificate is for “paypal.com”.

The articles want you believe Let’s Encrypt is somehow at fault if people end up at “paypall.com” with a green lock and think it’s “paypal.com”.  Let’s Encrypt isn’t providing “verified” certificates or trying to solve that problem.  The problem they’re trying to solve is that too much web traffic is unencrypted by default because certificates were expensive and inconvenient.

Someone with a vested interest in being able to read and/or modify your web traffic has been working really hard to get these articles out and make it look like some kind of “public safety” issue.

I have no idea who that entity may be, but it’s making me really annoyed.  Let’s Encrypt is a good thing for anyone that thinks that their Internet communications should be private by default.

Update 3/31: Engadget just ran one of the attack pieces too: “When the ‘S’ in HTTPS also stands for shady“.  Which is the most mainstream source running these articles that I’ve seen thus far.

To be completely clear, when a URL starts with HTTPS it only means that your connection is encrypted between your computer and the website–it has never meant anything about who is running the website is or whether the website operator is trustworthy.

Django Form Label for Many-to-Many with Extra Fields

I’m working on a Django application at work that has several Many-to-Many relationships that have extra fields attached to them. Setting this up using the “through” parameter on the field definition is described in the Django documentation.

My trouble came in when it came time to render this information out to the user in a form so they could update the extra fields.  From the user’s perspective the relationship can’t be changed, but the extra fields could be.

Extending Django’s example usage about band membership, if the user were viewing the information for “The Beatles” and including a formset for all band members the form would normally render something like this:

Group Name: The Beatles
Members:
    Group: The Beatles
    Person: Ringo Starr
    Date Joined: 14 August 1962
    -
    Group: The Beatles
    Person: John Lennon
    Date Joined: 01 August 1960
    -
    [etc.]

But this is a little obtuse if the purpose of the form is to allow the user to edit the “Date Joined” value for each band member.  We can whittle the form down by telling the formset to only render the “Date Joined” field [by passing fields=(‘date_joined’,) to modelformset_factory], but then we get this:

Group Name: The Beatles
Members:
    Date Joined: 14 August 1962
    -
    Date Joined: 01 August 1960
    -
    [etc.]

Now we can’t tell to which person each date applies.

Adjusting the form so that the label “Date Joined” was instead the appropriate band members name seems like such a simple thing to change.  But I didn’t want to hand-write the form as it was already being automatically built and rendered for me using using modelformset_factory and Crispy-Forms.

It took me most of a day to figure it out and in the end it was pretty simple.  I only need to implement a custom Form that overrides the constructor and sets the label using the object instance before it gets thrown away:

class MembershipForm(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        super(MembershipForm, self).__init__(*args, **kwargs)
        if kwargs['instance'] and kwargs['instance'].person:
            self.fields['date_joined'].label = kwargs['instance'].person.name
        else:
            self.fields['date_joined'].label = 'No Person'
    class Meta:
        model = Membership
        fields = ['date_joined']

And now our output looks something like:

Group Name: The Beatles
Members:
    Ringo Starr: 14 August 1962
    -
    John Lennon: 01 August 1960
    -
    [etc.]

Since it took me most of a day to figure this out, and I couldn’t find anything specifically addressing this on the Internet, I wrote this up so hopefully the next person (or me in the future once I’ve forgotten this) will have an easier time figuring it out.