Kamstrup meter software update

On September 29th around 1:45 at night, it seems that a software update has been performed on our Kamstrup smartmeter, triggered remotely.

The noticeable result is that the P1 telegrams have different (less) content. And since my monitoring script checks for an expected filesize, all telegrams were rejected and discarded that night. Fortunately only irrelevant/unused fields were removed, so after tweaking the expected filesize next morning everything worked just fine again.

Before:

After:

/KMP5 KA6U001660000000 0-0:96.1.1(204B413655303031363630000000000000) 1-0:1.8.1(02576.028*kWh) 1-0:1.8.2(01115.025*kWh) 1-0:2.8.1(00453.811*kWh) 1-0:2.8.2(01129.910*kWh) 0-0:96.14.0(0001) 1-0:1.7.0(0000.19*kW) 1-0:2.7.0(0000.00*kW) 0-0:17.0.0(999*A) 0-0:96.3.10(1) 0-0:96.13.1() 0-0:96.13.0() 0-1:24.1.0(3) 0-1:96.1.0(3238303131303031333035000000000000) 0-1:24.3.0(150929010000)(08)(60)(1)(0-1:24.2.1)(m3) (01850.034) 0-1:24.4.0(1) !

/KMP5 KA6U001660000000 0-0:96.1.1(204B413655303031363630000000000000) 1-0:1.8.1(02577.074*kWh) 1-0:1.8.2(01115.202*kWh) 1-0:2.8.1(00453.811*kWh) 1-0:2.8.2(01137.015*kWh) 0-0:96.14.0(0002) 1-0:1.7.0(0000.00*kW) 1-0:2.7.0(0002.68*kW) 0-0:96.13.1() 0-0:96.13.0() 0-1:96.1.0(3238303131303031333035000000000000) 0-1:24.3.0(150929110000)(08)(60)(1)(0-1:24.2.1)(m3) (01850.050) !

This filesize validation was mostly in place to detect a partial telegram due to some glitch in our serial communication. Now it's time to turn this validation into something more robust: we don't want to reject valid and complete telegrams when they ever decide to remove some other obscure field, or even add a new field. Especially since there's such a clean format requirement available: the first line must start with a '/' and the last line must end with a '!', and these symbols are unused in the main body of the telegram.

So instead of the filesize check in mbsolget_p1.sh we could use something like this:

...
  #Report unexpected filesize
  if [ $filesize = $FSIZE ]; then
    log "Received successfully"
  else
    log "Warning: Unexpected filesize!"
  fi

  telegram=$(cat $WORKDIR/p1_temp.log)

  if [[ $(expr match "$telegram" '/.*!') > 0 ]]; then

    #Process $WORKDIR/p1_temp.log to p1_daily.tmp
    MESSAGE=`echo "$telegram" | awk -f $AWKSCR > $STORE`
...

Edit:
On october 14th around 11:50 I started seeing 'Unexpected filesize' warnings in the logs - the P1 telegram format change mentioned above had been reverted for some reason. The script kept happily parsing and processing the telegrams, which means that the above tweak has already been useful and works as intended. At the same time I'm unsure what's really going on; why does the telegram content change so frequently all of a sudden? I'll keep an eye the logs to see if and when it changes again.

Edit 2:
On december 11th around 8:35 the above-mentioned fields have been removed from the P1 telegram again. Also, I noticed a small error in the validation check; it should have used double brackets in order to allow the greater-than comparison. I corrected this in the code above.

Setting up StecaGrid communication

So, this StecaGrid 3600 was a fair bit cheaper than normal, and as always cheapness comes at a price. It turns out that there exist different revisions of the inverter and I received an older version of it. And from what I understand the newer models have an ethernet interface which connects directly to your LAN, allowing you to navigate to a nice little dashboard or read raw xml data.

That was not the case with my model. I had already verified that it came with two RJ45 connections at the bottom, but unfortunately both are used for a funky RS485 serial connection. So the only way forward was to figure out how this serial connection works, and I started research before I even got my solar panels.

The manufacterer Steca Elektronik nicely provided me with the technical documentation of their RS485 interface and this looked really promising; it contained various commands to retrieve almost all metrics that were available via the inverter display itself... among which the daily and total yield values!

So after this inverter had been installed I couldn't wait to get started. But before even thinking about the data and documentation I first had to to get the serial connection to work, and by that I mean the actual physical wire that connects the inverter to a computer.


I sacrificed this old 20-meter UTP patch cable and cut off one of the RJ45 connectors, so I could connect the relevant wires to an RS485-USB adapter. I then plugged it into the computer which detected the USB serial port...

And at that point I got stuck.

I found several serial communication software tools, and tried sending some bytes to see if the inverter would send any response, which it did not. I also had the tools listen on the serial port for several days and log every single byte they received, and although there seemed to be spontaneous activity now and then, it mostly consisted of 0x00 bytes and a handful of other random bytes, garbage in proper programmer's terminology.

The technical documentation from Steca was at a higher protocol level than the serial protocol I was currently struggling with. They described command bytes that led to certain response values... perfect... but that command had to be wrapped with all kinds of control and checksum bytes; get one wrong and your command would simply be ignored.

On top of that, there were still more variables in play:

• Did I attach the correct wires to the A (485+) and B (485-) pins of the adapter?
• Which of the two RJ45 sockets on the inverter should I use?
• In what position should I put the tiny line-termination switch at the bottom of the inverter?
• Did I use the correct serial connection parameters?

I must admit I almost gave up here. Almost even resorted to pulse-counting on the S0 port of the kWh-meter, which turned out to be a trivial arduino sketch and circuit, but at the same time felt so dirty and disappointing.

And then... two weeks after I had connected the cable... on this german forum Photovoltaik I found a mention of the Steca User Software that also used this RS485 connection. After installation, it started its inverter detection on the serial port, and hallelujah, within 3 seconds it reported a "StecaGrid 3600". Communication established, arduino back in the box, the mission was back on track.


This program could update the inverter's firmware (more on that later), but also retrieve the actual panel power, voltage and frequency measurements, visualize the day curves from the past 30 days, and the yield totals from the past months and years.

Whatever this program could do, I wanted to do it myself and tightly focused on what I really needed, and prepared for automated readouts.


It was time to take the next tools from my toolbox: a serial traffic monitor which could capture all bytes on the serial port, and a decompiler for Steca's .NET assemblies. Time to reverse engineer all these data commands.


A final note on the variables that lead to successful communication with the StecaGrid:

• It doesn't matter which of the two sockets you use since they're internally connected to each other; RS485 was designed to connect multiple devices in series over long distances.
• The line-termination switch can be in either position, but I found that I get more 0x00 padding with the switch in ON (right-most) position, so I put to OFF.
• UTP wire 1 (orange/white) is connected to data A (485+)
  UTP wire 2 (orange) is connected to data B (485-)
  UTP wire 8 (brown) is connected to ground

• Connection parameters:

  Baud rate:	38400
  Parity:	none
  Data bits:	8
  Stop bits:	1

The next chapter: solar energy

On may 13th we had our solar PV panels installed; 12 monocrystalline black panels with a total system size of 3720 watt-peak. The inverter is a StecaGrid 3600 Coolcept, a german brand, not commonly seen.

The array of panels nicely fits below the flat-plate thermal collector that came with the house and had already been located all the way at the top deliberately.

Our roof's azimuth is 146° (south-east) and its tilt 45°. There's no shadow, except for a short moment at the end of summer afternoons when the sun is just about to move to the front of the house. Then the chimney casts a shadow across the entire surface; we'll actually see this effect in our graphs later.


In the weeks before installation I had already been monitoring the values of the electricity meter, visualizing this in separate "power consumption" and "power generation" gauges, the latter - of course - had always shown 0 watt. As mentioned in an earlier post, this interpretation had worked fine until the moment the inverter was activated on may 13th.

Let's begin with the basic situation:

We use electricity in our home appliances, and their requested power is pulled from the grid. What we measure at the smart meter is the amount of energy flowing into the house and in this scenario this happens to be exactly equal to our energy consumption. So far so good.

Now when the inverter is activated and it starts pumping amps into our house, we suddenly get the following situation:

I remember seeing my virgin "power generation" gauge go up to 2500 watt, while the "power consumption" value dropped down to 0... Hmmm... In this scenario a portion of the generated power is directly consumed while the rest is pushed out into the power grid.

Now imagine - and this could be just several seconds later - that we turn on the microwave while a cloud blocks the sun:

Our generated power is no longer enough to fulfil the request and the remaining power is drawn from the grid.

It becomes clear that with our P1 telegram measurements we only measure the net energy that passes through the meter, and as we see, this could be in either direction throughout the day. A sad result is that we have lost all insight in our actual consumption (or generation, for that matter).

Fortunately our electricity meter does have separate counters for the two directions in which energy can flow through it, so we can at least distinguish those; let's call these export and import from now on.

Note that at any moment in time you can either import or export energy, but never both simultaneously. Now you may think that our P1 monitoring will always have one value at 0, but remember that we're actually measuring averages over 5 minute periods. And in one such period both import and export could have occurred.

We can toss our variables in a formula to see how they relate to each other:

      Import + Generation = Export + Consumption

This formula makes sense if we assume that we don't store any energy in batteries for later use: all the energy that comes into the house (either generated or imported) must have gone somewhere (either consumed or exported).

Since we're already measuring import and export periodically we only need the value of one additional variable for each period, and we would have a complete picture again. An obvious candidate is generation; both the inverter itself and a separate kWh meter are already keeping track of this.

We'll have to figure out a way to obtain this additional measurement automatically and get it at the right place in order to apply the formula. Well, this turned out to be a proper challenge, but fun adventure...

Bash and Python to make it happen

Before anything else, I should give credits to Marco Bakker for sharing his scripts that got me started straight away. Within one day after I got my Raspberry Pi, I had it capture P1 telegrams, generate graphs and even upload data to external logging sites. In the weeks that followed I polished the script alot, and the result of that is what this post is about.

An overview of the most important changes:

• Calculations are now based on the actual time between measurements. Even though the script is triggered every 5 minutes, the exact time between measurements can vary because of the fact that we may have to wait up to 10 seconds for the telegram to arrive. But it's also possible that the Raspberry Pi has had some downtime and resumes after an unknown period of time.
• The script was getting so big that it was hard to navigate around, so I split it into smaller parts; one that holds all the configuration, one that contains the long rrdtool commands, another one that outputs data into various file formats, and a last one that contains interfaces to external websites.
• Clean output, so we can capture it to a useful log.
• Various readability, robustness and flow tweaks.

Also a little disclaimer; the original script provides support for several features that I don't use myself, and are therefore untested (but perhaps fully functional) in my version of the script:

• Upload to Xively.
• P1 dsmr v4 format. In principle, you could easily support any P1 format by adding an appropriate awk script.
• Output to csv format.
• Output to html page. The generated html is awfully retro, but I haven't bothered changing it.

This script is distributed under the GNU General Public License v3, and by downloading the script you agree to the terms of this license. Furthermore, usage is at your own risk and without guarantees. mbsolget_svg  .tar.gz (10kb) .zip (12kb)

Optional dependencies: rrdtool, ncftp, sendemail, mysql

After extraction into its own directory, the only thing you need to do is edit config.sh to set P1PORT to the name of the serial device that connects to the P1 port, and also set the correct WORKDIR (and actually anything else you find in there - but let's keep it simple for now).

Finally, make the main script executable with: chmod +x mbsolget_p1.sh

You're ready to go, so give it a test run: ./mbsolget_p1.sh

No errors? Good, then you'll find a bunch of new files and directories. Note that most interesting stuff is configured to happen on every 5 minute interval of the hour.

The raw captured P1 telegrams are stored as log/p1-YYYYMMDD-hhmm.log
The extracted values of the last P1 telegram are stored in p1_gas.tmp and p1_value.tmp
All calculated statistics are written to debug.tmp, the meaning of which are in comments inside the main script. These statistics are used to build the json, csv, sql or xml data output.
In order to initialize your rrd storage file, you should run the script once with the 'create' argument: ./mbsolget_p1.sh create

When you have it configured the way you want, schedule the script to run automatically with crontab -e, and add the following line (but with the proper path of course):
*/5 * * * * /home/pi/path/to/mbsolget_p1.sh >> ~/log/mbsolget.log 2>&1

At this point - and for me this was early april - the data flow shown in last post is fully functional (except for the backup part, but this is just another cron job). I'll leave it to you to get creative with the json or xml that is pushed to your webserver every 5 minutes. You could even write a smartphone app around it.

On the right I've included a screenshot of the dashboard I created. It keeps itself up-to-date by requesting the json and graphs periodically via ajax requests.

Next chapter: solar PV panels!

Measurement frequency versus resolution

The idea is to use the monotonically increasing values of T1 and T2 to calculate the average power consumption over the period between two subsequent measurements.

Power [W] =

Tnow [Wh] - Tprevious [Wh] tnow [s] - tprevious [s]

* 3600 [s/h]

...where T is the energy meter value, and t the timestamp of the measurement.

My initial plan was to capture every single telegram and process it, but after the following considerations I concluded this was really bad idea:

• What do we do with the 8640 (!) data points per day? Store them locally? Push them over the network?
• Will we be stressing the Raspberry Pi hardware and resources if we're processing every telegram. How about CPU load and temperature, and aren't we unnecessarily burning the limited writes of an sd-card?
• I asked myself how real-time this monitoring really needed to be... What am I trying to achieve with it?
• What would be the resolution of the measurements anyway? Let's see: our meter values are in Wh so the smallest difference we can measure is 1 Wh. Over a timespan of 10 seconds, this would be equivalent to 360 watt.... Wait... so we can only report power consumption in increments of 360 watt? That's entirely useless!

I settled for capturing the P1 telegram every 5 minutes and generating graphs every 15 minutes. The resulting measurements are pushed to external logging services pvoutput.org and mindergas.nl, and also to my own webserver to allow me to build some dashboard around it.

This picture pretty much covers the data flow and their frequencies.
Next, we'll check out the code that is running on the Raspberry Pi to make this all happen...

Of Watts and Joules

In electrical applications we measure power in watts (W), and energy in watt-hours (Wh), both are derived SI units. It's sometimes hard to wrap your head around the meaning of these units and the correlation between them, and I especially find the word 'hour' in watt-hour very confusing.

A quick overview.

Power is the rate at which energy is generated or consumed, and as such is measured in 'energy per time unit'. The unit for energy is joule (J), which means we can express power in joules/second (J/s).

So what's the relation to this watt unit then?

1 W is defined as 1 J/s

Ok, that's easy. How about watt-hours...

Imagine you have a light-bulb with a power of 50 watt and leave it burning for 1 hour, that light-bulb consumed 50 watt-hours.

And we can express this in joules as well:

1 Wh = 1 W · 1 hour = 1 J/s · 3600 s = 3600 J

An important thing to realize is that watt-hour is a time-independent unit; it expresses an amount of energy. Note the similarity with 'lightyear', which is a distance and has nothing to do with time either.
It gets really confusing when you say device X used 50 watt-hours in 5 minutes, but it's a perfectly valid statement (and you would by now be able to deduct that device X's power consumption must be 600 watt, right?).

In my head I often use this analogy that watt is like the download speed in bytes/second, and watt-hour is like the amount of bytes you have transferred after a certain amount of time. Perhaps a bit odd comparison and I'm stuck forever with this mental image of the microwave downloading energy.

When this is all clear we're ready to go crazy at power and energy statistics.

Dissection of a P1 telegram

We're going to analyze the content of this telegram we captured. Below is the example from previous post with annotation. The format is fully described in Dutch Smart Meter Requirements (DSMR) 3.0, an open standardized protocol.

/KMP5 KA6U001660000000	Telegram begin-marker + manufacturer + serial number
0-0:96.1.1(204B413655303031363630000000000000	Serial number in hex
1-0:1.8.1(01964.939*kWh)	+T1: Energy input, low tariff (kWh)
1-0:1.8.2(01006.545*kWh)	+T2: Energy input, normal tariff (kWh)
1-0:2.8.1(00124.673*kWh)	-T3: Energy output, low tariff (kWh)
1-0:2.8.2(01006.545*kWh)	-T4: Energy output, normal tariff (kWh)
0-0:96.14.0(0001)	Current tariff (1=low, 2=normal)
1-0:1.7.0(0000.45*kW)	Actual power input (kW)
1-0:2.7.0(0000.00*kW)	Actual power output (kW)
0-0:17.0.0(999*A)	Max current per phase (999=no max)
0-0:96.3.10(1)	Switch position
0-0:96.13.1()	Message code
0-0:96.13.0()	Message text
0-1:24.1.0(3)	Attached device type identifier
0-1:96.1.0(3238303131303031333035000000000000)	Serial number of gas meter
0-1:24.3.0(150613210000)(08)(60)(1)(0-1:24.2.1)(m3)	Time of last gas meter update
(01831.213)	Gas meter value (m³)
0-1:24.4.0(1)	Gas valve position
!	Telegram end-marker

A P1 telegram always begins with a forward slash (/) followed by a meter manufacturer identifier and serial number on the same line, and ends with an exclamation mark (!). Everything in between is measurement data: the beginning of each line tells us the type of measurement, and the actual value is between the parentheses.

Normally the T1 (1.8.1) and T2 (1.8.2) meter values are the only relevant ones, and the latter only if you have dual rate for electricity. As soon as you start feeding electricity back to the grid, the T3 (2.8.1) and T4 (2.8.2) values will be going up as well.

Three months ago, when I was just figuring this all out, we didn't have solar panels yet and I could safely interpret the sum of T1 and T2 as the energy consumption. This worked fine back then, but in a future post we'll see how solar panels ruin this interpretation, and come up with some more accurate terms.

The actual power input (1.7.0) and output (2.7.0) are snapshot values. They're nice for an indication of power input/output as real-time as it gets; for example, if you switch on the microwave, you'll see this power value shoot up in the next P1 telegram (ie. within 10 seconds). But besides that, they hardly provide any statistical value.

I'm unsure about the meaning of switch position (96.3.10) and valve position (24.4.0)... I tried closing the external gas valves, but it didn't affect the returned value. I haven't tested the RCD, mostly because it would require another power supply to keep the Raspberry Pi alive, and I did not care that much to bother.

It is noteworthy that gas meters are configured to report their values to the main meter only once per hour. This means that our P1 telegram also only contains the last reported gas meter value and the timestamp when this was updated. The format of this timestamp is (YYMMDDhhmmss), and the gas meter value is all by itself on a new line... a bit inconsistent formatting going on there.

Let's assume we have captured our example telegram in a text file; we can then use this simple 'awk' script to parse it:

# Initialization
BEGIN { FS="[:()*]" }

# Pattern matches to extract the fields we want
/1\.8\.1/   { l1 = $3; }
/1\.8\.2/   { l2 = $3; }
/2\.8\.1/   { t1 = $3; }
/2\.8\.2/   { t2 = $3; }
/96\.14\.0/ { ta = $3; }
/1\.7\.0/   { w1 = $3; }
/2\.7\.0/   { w2 = $3; }
/24\.3\.0/  { dt = $3; }
/^\(.*\)\r/ { g1 = $2; }

# Finally dump the variables into an array.
END { print ta,l1,l2,(l1+l2),t1,t2,(t1+t2),w1,w2,(w1+w2),g1,dt; }

Now that we have all our values of interest in a handy little bash array, it's time to automate and schedule the whole process and start thinking of what we can actually do with these measurements.

Capturing our first telegram

The hardware
First task was figuring out how to read the data from this P1 communication port. We'll need some device that does the actual reading, preferably something small, since it has to be located in the meter cupboard. The Raspberry Pi was an obvious choice; it's a full-blown computer with the size of a deck of cards, which costs only 30 euro.

Next we'll need a cable that fits into the RJ11 socket and ideally plugs into one of the USB interfaces of the Pi. There's several tutorials on the internet that show you how to cobble together such a cable yourself, if you like soldering. I went for this ready-made P1 converter cable from www.smartmeterdashboard.nl, which works perfectly out-of-the-box.

In order to access and control the Pi once it's installed, it will have to be connected to the local network, either wired or wireless. I opted for a USB wifi dongle.... a tad oversized one, since it blocks 2 out of the 4 available USB connections. I'll start worrying about that when I need them.

The software

I installed Raspbian on the Raspberry Pi, which is great if you're already totally familiar with Debian Linux. And then I followed these great guides from Embezon and Gejanssen to get it all configured.

Serial connections require several parameters in order to work correctly, or you'll only receive gibberish or nothing at all. The parameters for the P1 port are as follows:

Baud rate:	9600
Data bits:	7
Parity:	Even
Stop bits:	1

Also make sure the cable can handle an inverted signal (the 0's and 1's are swapped). According to this blogpost, the FTDI-chip based cables are programmable with a little utility. I didn't need this with my P1 converter cable.

We'll use the 'cu' command to test the serial connection; remember that the P1 port spits out information once every 10 seconds, so you may have to wait a little before the data appears.

It's plain text, awesome! We can already recognize some stuff in there, but let's dive deeper into this so-called P1 telegram in the next post.

A house and a port

We recently bought this house. It's a great place and everything is brand new.
So what do you do when you own so many (tech) goodies? Indeed, I googled all manufacturers and model types of devices that are scattered throughout the house, just to learn more about their possibilities:

• Kamstrup 162J electricity meter
• Flonidan Uniflo G4S gas meter 
• Elster V200 water meter
• Ferroli AquaSol 4 LB-90-IT solar water heater
• Intergas Kombi Kompakt HRE 28/24 central heating unit

On Tweakers forum I found some interesting articles about these 'smart meters'. Especially the mention of a so-called P1 communication port triggered my curiosity, but I was convinced my meter did not have such a port... "I surely would have seen that immediately".

Just to be certain I checked the meter.... and I saw it right away, covered with a black soft plastic cover, there was an RJ11 socket, begging to be used.

So I was left with these questions: what data can we read from this P1 port? And what can we do with the data? A new project was born.