Tuesday, June 23, 2015

Measurement frequency versus resolution

The idea is to use the monotonically increasing values of T1 and T2 to calculate the average power consumption over the period between two subsequent measurements.

Power [W] =
Tnow [Wh] - Tprevious [Wh]
tnow [s] - tprevious [s]
* 3600 [s/h]

...where T is the energy meter value, and t the timestamp of the measurement.

My initial plan was to capture every single telegram and process it, but after the following considerations I concluded this was really bad idea:
  • What do we do with the 8640 (!) data points per day? Store them locally? Push them over the network?
  • Will we be stressing the Raspberry Pi hardware and resources if we're processing every telegram. How about CPU load and temperature, and aren't we unnecessarily burning the limited writes of an sd-card?
  • I asked myself how real-time this monitoring really needed to be... What am I trying to achieve with it?
  • What would be the resolution of the measurements anyway? Let's see: our meter values are in Wh so the smallest difference we can measure is 1 Wh. Over a timespan of 10 seconds, this would be equivalent to 360 watt.... Wait... so we can only report power consumption in increments of 360 watt? That's entirely useless!
I settled for capturing the P1 telegram every 5 minutes and generating graphs every 15 minutes. The resulting measurements are pushed to external logging services pvoutput.org and mindergas.nl, and also to my own webserver to allow me to build some dashboard around it.


This picture pretty much covers the data flow and their frequencies.
Next, we'll check out the code that is running on the Raspberry Pi to make this all happen...

Saturday, June 20, 2015

Of Watts and Joules

In electrical applications we measure power in watts (W), and energy in watt-hours (Wh), both are derived SI units. It's sometimes hard to wrap your head around the meaning of these units and the correlation between them, and I especially find the word 'hour' in watt-hour very confusing.

A quick overview.

Power is the rate at which energy is generated or consumed, and as such is measured in 'energy per time unit'. The unit for energy is joule (J), which means we can express power in joules/second (J/s).

So what's the relation to this watt unit then?

1 W is defined as 1 J/s

Ok, that's easy. How about watt-hours...

Imagine you have a light-bulb with a power of 50 watt and leave it burning for 1 hour, that light-bulb consumed 50 watt-hours.

And we can express this in joules as well:

1 Wh = 1 W · 1 hour = 1 J/s · 3600 s = 3600 J

An important thing to realize is that watt-hour is a time-independent unit; it expresses an amount of energy. Note the similarity with 'lightyear', which is a distance and has nothing to do with time either.
It gets really confusing when you say device X used 50 watt-hours in 5 minutes, but it's a perfectly valid statement (and you would by now be able to deduct that device X's power consumption must be 600 watt, right?).

In my head I often use this analogy that watt is like the download speed in bytes/second, and watt-hour is like the amount of bytes you have transferred after a certain amount of time. Perhaps a bit odd comparison and I'm stuck forever with this mental image of the microwave downloading energy.

When this is all clear we're ready to go crazy at power and energy statistics.

Wednesday, June 17, 2015

Dissection of a P1 telegram

We're going to analyze the content of this telegram we captured. Below is the example from previous post with annotation. The format is fully described in Dutch Smart Meter Requirements (DSMR) 3.0, an open standardized protocol.

/KMP5 KA6U001660000000Telegram begin-marker + manufacturer + serial number
0-0:96.1.1(204B413655303031363630000000000000Serial number in hex
1-0:1.8.1(01964.939*kWh)+T1: Energy input, low tariff (kWh)
1-0:1.8.2(01006.545*kWh)+T2: Energy input, normal tariff (kWh)
1-0:2.8.1(00124.673*kWh)-T3: Energy output, low tariff (kWh)
1-0:2.8.2(01006.545*kWh)-T4: Energy output, normal tariff (kWh)
0-0:96.14.0(0001)Current tariff (1=low, 2=normal)
1-0:1.7.0(0000.45*kW)Actual power input (kW)
1-0:2.7.0(0000.00*kW)Actual power output (kW)
0-0:17.0.0(999*A)Max current per phase (999=no max)
0-0:96.3.10(1)Switch position
0-0:96.13.1()Message code
0-0:96.13.0()Message text
0-1:24.1.0(3)Attached device type identifier
0-1:96.1.0(3238303131303031333035000000000000)Serial number of gas meter
0-1:24.3.0(150613210000)(08)(60)(1)(0-1:24.2.1)(m3)Time of last gas meter update
(01831.213)Gas meter value (m³)
0-1:24.4.0(1)Gas valve position
!Telegram end-marker

A P1 telegram always begins with a forward slash (/) followed by a meter manufacturer identifier and serial number on the same line, and ends with an exclamation mark (!). Everything in between is measurement data: the beginning of each line tells us the type of measurement, and the actual value is between the parentheses.

Normally the T1 (1.8.1) and T2 (1.8.2) meter values are the only relevant ones, and the latter only if you have dual rate for electricity. As soon as you start feeding electricity back to the grid, the T3 (2.8.1) and T4 (2.8.2) values will be going up as well.

Three months ago, when I was just figuring this all out, we didn't have solar panels yet and I could safely interpret the sum of T1 and T2 as the energy consumption. This worked fine back then, but in a future post we'll see how solar panels ruin this interpretation, and come up with some more accurate terms.

The actual power input (1.7.0) and output (2.7.0) are snapshot values. They're nice for an indication of power input/output as real-time as it gets; for example, if you switch on the microwave, you'll see this power value shoot up in the next P1 telegram (ie. within 10 seconds). But besides that, they hardly provide any statistical value.

I'm unsure about the meaning of switch position (96.3.10) and valve position (24.4.0)... I tried closing the external gas valves, but it didn't affect the returned value. I haven't tested the RCD, mostly because it would require another power supply to keep the Raspberry Pi alive, and I did not care that much to bother.

It is noteworthy that gas meters are configured to report their values to the main meter only once per hour. This means that our P1 telegram also only contains the last reported gas meter value and the timestamp when this was updated. The format of this timestamp is (YYMMDDhhmmss), and the gas meter value is all by itself on a new line... a bit inconsistent formatting going on there.

Let's assume we have captured our example telegram in a text file; we can then use this simple 'awk' script to parse it:
# Initialization
BEGIN { FS="[:()*]" }

# Pattern matches to extract the fields we want
/1\.8\.1/   { l1 = $3; }
/1\.8\.2/   { l2 = $3; }
/2\.8\.1/   { t1 = $3; }
/2\.8\.2/   { t2 = $3; }
/96\.14\.0/ { ta = $3; }
/1\.7\.0/   { w1 = $3; }
/2\.7\.0/   { w2 = $3; }
/24\.3\.0/  { dt = $3; }
/^\(.*\)\r/ { g1 = $2; }

# Finally dump the variables into an array.
END { print ta,l1,l2,(l1+l2),t1,t2,(t1+t2),w1,w2,(w1+w2),g1,dt; }




Now that we have all our values of interest in a handy little bash array, it's time to automate and schedule the whole process and start thinking of what we can actually do with these measurements.

Saturday, June 13, 2015

Capturing our first telegram

The hardware
First task was figuring out how to read the data from this P1 communication port. We'll need some device that does the actual reading, preferably something small, since it has to be located in the meter cupboard. The Raspberry Pi was an obvious choice; it's a full-blown computer with the size of a deck of cards, which costs only 30 euro.

Next we'll need a cable that fits into the RJ11 socket and ideally plugs into one of the USB interfaces of the Pi. There's several tutorials on the internet that show you how to cobble together such a cable yourself, if you like soldering. I went for this ready-made P1 converter cable from www.smartmeterdashboard.nl, which works perfectly out-of-the-box.

In order to access and control the Pi once it's installed, it will have to be connected to the local network, either wired or wireless. I opted for a USB wifi dongle.... a tad oversized one, since it blocks 2 out of the 4 available USB connections. I'll start worrying about that when I need them.

The software
I installed Raspbian on the Raspberry Pi, which is great if you're already totally familiar with Debian Linux. And then I followed these great guides from Embezon and Gejanssen to get it all configured.

Serial connections require several parameters in order to work correctly, or you'll only receive gibberish or nothing at all. The parameters for the P1 port are as follows:

Baud rate: 9600
Data bits: 7
Parity: Even
Stop bits: 1

Also make sure the cable can handle an inverted signal (the 0's and 1's are swapped). According to this blogpost, the FTDI-chip based cables are programmable with a little utility. I didn't need this with my P1 converter cable.

We'll use the 'cu' command to test the serial connection; remember that the P1 port spits out information once every 10 seconds, so you may have to wait a little before the data appears.


It's plain text, awesome! We can already recognize some stuff in there, but let's dive deeper into this so-called P1 telegram in the next post.

Friday, June 12, 2015

A house and a port

We recently bought this house. It's a great place and everything is brand new.
So what do you do when you own so many (tech) goodies? Indeed, I googled all manufacturers and model types of devices that are scattered throughout the house, just to learn more about their possibilities:
  • Kamstrup 162J electricity meter
  • Flonidan Uniflo G4S gas meter 
  • Elster V200 water meter
  • Ferroli AquaSol 4 LB-90-IT solar water heater
  • Intergas Kombi Kompakt HRE 28/24 central heating unit

On Tweakers forum I found some interesting articles about these 'smart meters'. Especially the mention of a so-called P1 communication port triggered my curiosity, but I was convinced my meter did not have such a port... "I surely would have seen that immediately".

Just to be certain I checked the meter.... and I saw it right away, covered with a black soft plastic cover, there was an RJ11 socket, begging to be used.

So I was left with these questions: what data can we read from this P1 port? And what can we do with the data? A new project was born.