Setting up rules based on untracked consumption

[hheimbuerger]

Checklist

• I have filled out the template to the best of my ability.
• This only contains 1 feature request (if you have multiple feature requests, open one feature request for each feature request).
• This issue is not a duplicate feature request of previous feature requests.

Is your feature request related to a problem? Please describe.

Have you considered giving powercalc the ability to set up rules based on untracked consumption?

I've set up tracking of most of the significant devices in my household. (Using powercalc for the very static ones that don't warrant active tracking.)
The last remaining one is my induction stove, which in my case (as most European homes, I believe) is more directly connected to the apartment's power grid, not just plugged into an outlet.

But I've just realized that I could pretty easily think of a heuristic approach to identify stove usage based on the untracked consumption:
Given that most of my devices are tracked, there's only very few that draw a lot of power (200W+): it's mostly the stove, and very rarely an iron or a hairdryer, but those latter two run for maybe a minute or two. And they're also used so rarely that some misclassification wouldn't bother me much.

Describe the solution you'd like

I'd like to use powercalc to set up a rule that goes like this (pseudo-code):

- name: Stove
  untracked:
    power_exceeds: 200     # in W
    min_time: 2            # in minutes

Describe alternatives you've considered

Eventually, I'd like to properly track this device with something like a Shelly, but I'd need to find an electrician to install this into the breaker box, so... a much bigger project. Solving it in software using powercalc would be a very neat stopgap solution!

I'm also considering to see if I can understand enough of the powercalc sources to create these two sensors specifically as its own component.

Additional context

I'm wondering if powercalc (or any HA component, for that matter) can create power/energy events retroactively, i.e. whether it can use the 'for at least two minutes' part of the rule to then create a power draw in the HA database with a timestamp of two minutes before the detection.

[bramstroker]

You cannot retroactively register states.
HA is a state and event based system, so you can only report a state when it actually happens and HA will save that in the states table.

So it won't be possible to do this with a power sensor.

Might be doable by introducing some new kind of energy sensor which does the following:

• start keeping track of seen power values when it exceeds the threshold (200W), keep track of them including timestamp when particular power value occured. When the power drops below 200W within 2 minuten clear the seen power values, and reset the start time.
• when min time (2 min) is reached calculated the kWh. Might reuse calculation routines of riemann sum helper https://github.com/home-assistant/core/blob/dev/homeassistant/components/integration/sensor.py#L560.
• after that just integrate on each state change, until the power drops below 200W again

In your proposed pseudo code I'm missing the entity_id, which should be the main P1 meter right?

So it might look like this:

- name: Stove
  entity_id: sensor.mains_power
  untracked:
    power_exceeds: 200
    min_time: 00:02:00

[bramstroker]

Got some draft PR going on #2710

[hheimbuerger]

Got some draft PR going on #2710

Wow, thank you for looking into this so quickly! I'll try to review the code tomorrow and maybe I can follow along and see if I understand how this works.

Regarding the retroactive events: yeah, I figured that's how it works. I would be very interested in a power sensor as well (real-time power display is what I'm doing a lot of my tracking for), but I understand that these are heuristics, and by definition not fully accurate. So if it just doesn't show up in the first two minutes (or whatever min_time is set to), and there's the power and energy data is 'off by two minutes', that's perfectly acceptable for me.

In fact, I think in my case, I might just be able to skip the 'minimum time' requirement at all, and simply claim all untracked consumption over a certain wattage as the stove. Given that I have so few untracked, high power-draw devices, the total error will probably be a very low percentage.

As for your question about the entity_id: as all my powercalc devices are 'virtual' (don't have corresponding devices in HA), I never quite understood what it is for and how to use it. But that's a point for another conversation (or maybe just for me to keep reading the docs) and doesn't belong here.
But yeah, I guess in this case one would need a parameter to specify which electricity source to track, if HA can have multiple untracked consumptions.

[bramstroker]

I’ve been giving this some more thought, and the main challenge lies in determining the untracked consumption. Your request assumes that an untracked power sensor is available, which is also the expectation in my draft PR. However, no such sensor is provided out of the box, and Powercalc cannot automatically create a reliable one.

Ultimately, it seems to come down to having the user set up a Powercalc group sensor. This group sensor would include all the Powercalc sensors and any other power sensors in the system, collectively representing the full tracked power consumption. The untracked consumption could then be calculated as the P1 meter output minus the tracked power sensor.

To simplify this process, we could offer an option in the group sensor setup to automatically include all Powercalc sensors. This way, the user would only need to manually add the non-Powercalc sensors to define the tracked power sensor.

What do you think?

[hheimbuerger]

I've since read the code in your PR, but I'm afraid I'm missing too many HA sensor/integration development basics, and I'm struggling to understand where this hooks into the HA core.

I did wonder a little bit reading the documentation you added (untracked-energy.md). I don't quite understand how this differs from other fixed energy/power sensors. What's described there seems to fulfill exactly the same purpose I'm getting out of my fixed sensors as well, except that I get both power and energy:

- entity_id: sensor.dummy
  name: Baseline powercalc
  fixed:
    power: 22

I thought this is what powercalc was already for.

But back to your questions around the feature request: yeah, I figured that creating a untracked power sensor is the tricky part. I had also planned to ask in the HA community forums whether that is possible, in parallel to suggesting it for powercalc, I just never got around to it.

I think this needs to come from the HA core, or at least a separate integration. After all, it is already calculated and displayed on the Energy board, namely the "Individual devices detail usage" card (type: energy-devices-graph).

One of my other frustrations is that it doesn't appear on the — very similar! — "Individual devices total usage" graph (type: energy-devices-detail-graph), but that's a separate issue and completely unrelated to powercalc.

Anyway, HA core already has all the data. It knows my realtime electricity meter power (I believe that's what you call "P1") and it has a list of all my devices (energy meters), some of which are in my case virtual, created with powercalc. From that, it can, and does, calculate the untracked energy consumption. It just doesn't display it on the day's summary graph (but I can understand that decision, I just wish they hadn't made it like that) and it doesn't expose it as a sensor, so that I can create automations from it. (Simple example: "notify me when the untracked power consumption exceeds 1000W" -- this wouldn't and shouldn't trigger when my oven draws 1000W, because that's already tracked.)

I haven't published it yet, but I've written my own card which takes that current power draw from the electricity meter, and it applies some heuristics to derive all the power sensors from the electricity sensors (not very hard, usually just sensor.*_energy -> sensor.*_power) and then draw a graph similarly to type: energy-devices-graph, displaying the current power distribution among devices in realtime. It also displays the leftover as 'untracked'.

This was a bit rambly and I realize I didn't really respond to any of your questions on point, sorry. I need to ponder this some more.

[bramstroker]

I did wonder a little bit reading the documentation you added (untracked-energy.md). I don't quite understand how this differs from other fixed energy/power sensors. What's described there seems to fulfill exactly the same purpose I'm getting out of my fixed sensors as well, except that I get both power and energy:

The idea was to configure a threshold and time, and the energy sensor will only start adding when the configured threshold has been exceeded for the time configured.
So you can make an energy sensor this way for your stove, but the important thing I missed here is it needs to be calculated from the untracked power, but there is no such entity available in HA. When we can create such a sensor in HA somehow than this can be used as the input (entity_id) for this untracked stove entity.
The untracked stove entity will track changes on the configured untracked entity. That happens here: https://github.com/bramstroker/homeassistant-powercalc/pull/2710/files#diff-e27b93ecb1ccf3220e78776afc6cb037dde2cee6af1cae0827bd30ec46de465fR150.
And the main logic for calculating the new kWh value is here: https://github.com/bramstroker/homeassistant-powercalc/pull/2710/files#diff-e27b93ecb1ccf3220e78776afc6cb037dde2cee6af1cae0827bd30ec46de465fR159
Currently the PR only creates an energy sensor (in kWh), because we cannot retroactively change values in the past (which actually needs to happen for a power sensor). But we might reconsider.

The main challenge in any case is to get a reliable untracked energy and/or power sensor in HA.
More about that below.

But back to your questions around the feature request: yeah, I figured that creating a untracked power sensor is the tricky part. I had also planned to ask in the HA community forums whether that is possible, in parallel to suggesting it for powercalc, I just never got around to it.

I think this needs to come from the HA core, or at least a separate integration. After all, it is already calculated and displayed on the Energy board, namely the "Individual devices detail usage" card (type: energy-devices-graph).

I think the untracked energy is not really a separate entity in HA.
The energy dashboard is build on statistics data, which are aggregations (min, max, mean, avg) etc per hour. You manually configure all the individual energy sensors which you want to show on the energy dashboard, and also configure the mains power meter (under sources). Pretty sure the statistics module/card will just runtime calculate the difference between the sum of all individual energy sensors and the mains sensor, and put that bar in the graph.

So there is no separate sensor for untracked energy in HA by default afaik.

I could maybe have a look if I'm able to access the configuration where you configure the individual devices somehow. That would open up a possibility to let powercalc create a separate sensor for the untracked energy.

Even if that would be possible somehow we are talking about an energy sensor (in kWh) and not a power sensor (in W), which you'd actually like.
I think that would be very difficult to achieve, because not every energy sensor would have a corresponding W sensor per se.

I don't think I can make a "tracked" power meter by just summing up all known entities with device_class = power in HA, as there might be duplicate sensors or unwanted sensor, but maybe it can work.
You might try that, by making a template sensor. But you'll need to make sure to exclude possible powercalc group sensors, and also the mains sensor from the calculation.

Anyway, HA core already has all the data. It knows my realtime electricity meter power (I believe that's what you call "P1") and it has a list of all my devices (energy meters), some of which are in my case virtual, created with powercalc. From that, it can, and does, calculate the untracked energy consumption. It just doesn't display it on the day's summary graph (but I can understand that decision, I just wish they hadn't made it like that) and it doesn't expose it as a sensor, so that I can create automations from it. (Simple example: "notify me when the untracked power consumption exceeds 1000W" -- this wouldn't and shouldn't trigger when my oven draws 1000W, because that's already tracked.)

You are mixing up power and energy here, they are really different things.
"realtime electricity meter power" and "list of all my devices (energy meters)"

I haven't published it yet, but I've written my own card which takes that current power draw from the electricity meter, and it applies some heuristics to derive all the power sensors from the electricity sensors (not very hard, usually just sensor.*_energy -> sensor.*_power) and then draw a graph similarly to type: energy-devices-graph, displaying the current power distribution among devices in realtime. It also displays the leftover as 'untracked'.

That would maybe work for your case deriving the power sensors from energy, but that won't be a generic solution I can apply within powercalc as there could be other namings, need to account for group sensors (need to be excluded), power sensors which are not created by powercalc should also be added. etc. etc. There are a lot of complications and difficulties here.

This was a bit rambly and I realize I didn't really respond to any of your questions on point, sorry. I need to ponder this some more.

I'm rambly as wel lol.
Just did a quick braindump here to have stuff for thought :-)

[hheimbuerger]

Yeah, I totally understand.

I do believe we're at an impasse here. I agree with you that powercalc (and even HA core, under the current general architecture -- specifically the fact that the energy configuration only manages energy sensors, and not power sensors) probably cannot deliver what I'm looking for, as a generalized "mainstream" solution.

I'm still committed to setting this up (in a hacky workaround, probably) for my personal HA instance. So if you don't mind keeping this thread open for a few weeks, I'll use this as my whiteboard and braindump, too. Maybe it's useful to someone.

I think the untracked energy is not really a separate entity in HA. […] Pretty sure the statistics module/card will just runtime calculate the difference between the sum of all individual energy sensors and the mains sensor, and put that bar in the graph.

So there is no separate sensor for untracked energy in HA by default afaik.

Yeah, I think you're right. Still wish it would show that not only in the graph with separate bars for each hour, but also in the one summing up the day, ordered by total usage. Meh.

I could maybe have a look if I'm able to access the configuration where you configure the individual devices somehow. That would open up a possibility to let powercalc create a separate sensor for the untracked energy.

I've done that already for my realtime power graph. I'll share it eventually. It's a proof-of-concept, of course, but it works. Probably too many heuristics to include in a high quality integration like powercalc.

Even if that would be possible somehow we are talking about an energy sensor (in kWh) and not a power sensor (in W), which you'd actually like. I think that would be very difficult to achieve, because not every energy sensor would have a corresponding W sensor per se.

Exactly. They do in my case! And that's why I can get away with it. But it's difficult to generalize.

You are mixing up power and energy here, they are really different things. "realtime electricity meter power" and "list of all my devices (energy meters)"

My writing might as well have been off, but I don't think I'm confusing these concepts anymore, given how much I've been working with this. 😊
My electricity meter (with that term I'm referring to the hardware box in my basement that my main power line goes through) indeed delivers realtime power (W) and HA does maintain a list of energy sensors (Wh) from various energy+power meters (hardware) in my home, which I'm then mapping to their corresponding power sensors (W) with my own code.

[hheimbuerger]

Next, I'll try my luck at writing a custom integration for an untracked power sensor. My plan is to start with the Example Sensor, convert my JS code to determine power sensors from energy sensors into Python, do that at startup, then somehow hook into updates from those power sensors (including mains), and then emit a new power sensor data point/event whenever one of those power sensors changes.

[hheimbuerger]

That's probably exactly what you were trying to tell me earlier... and I also touched on it in my own commentary... but nevertheless I only now fully realized:

If there's anything today's HA core could expose, it is an 'untracked energy sensor'. It does not in any way track power at the moment. There's no list of power sensors of active devices and there's no power cards either. It's all energy. It's not even close to being able to expose an 'untracked power sensor'.

[bramstroker]

That's probably exactly what you were trying to tell me earlier... and I also touched on it in my own commentary... but nevertheless I only now fully realized:

If there's anything today's HA core could expose, it is an 'untracked energy sensor'. It does not in any way track power at the moment. There's no list of power sensors of active devices and there's no power cards either. It's all energy. It's not even close to being able to expose an 'untracked power sensor'.

Yes that's correct, the energy dashboard and related configurations are all related to energy sensors.
They have no relation to any power related stuff.

I was just thinking about it, maybe I can introduce some feature in the group sensor setup where you can choose to setup a tracked power sensor.
During the setup we'll scan the system for all entities having device_class=power, it will add all non powercalc group entities to the group, next the user can remove the unwanted entities from the group (for example the mains sensor).
This way we could end up having an tracked power sensor.
Only problem is how to deal with new power entities being added to the system after the fact. We could automatically add new powercalc power sensor to this tracked power group. But other power sensors will be difficult to automate.

We could also make it a submenu item under group Tracked/Untracked.

We'll ask the user first to specify the mains power sensor.
In next step we will preload all individual power sensors (excluding the mains power sensor) from the system and allow the user to remove unwanted.
After configuring we could end up with 2 sensors sensor.tracked_power, sensor.untracked_power.

What do you think?

[hheimbuerger]

I thought a lot about this, but to be entirely honest, I haven't fully grasped it all yet. I'm only now slowly starting to understand what you've been telling me five posts up... 😊

I finally have read up on powercalc groups. Never had used that before.
As an experiment, I created a tracked_power group sensor, which combines all my device power sensors (the power sensors corresponding to the energy sensors set up in the Energy dashboard configuration). Then I created a second untracked_power subtract group, which takes my mains power sensor and subtracts the tracked_power from it.

My understanding is that what you described in your last post, is a way to semi-automate that. Not particularly useful to me, but it may be for others. Me, I'd rather have this be derived from HA's Energy Configuration (i.e. transforming the set of energy sensors HA already manages into a corresponding set of power sensors). But I'm fully aware that I promised to show off how I've done that, and haven't gotten very far with that yet.

Slowly getting back to my original intention: now that I have a untracked_power sensor... does powercalc have the ability to respond to usage patterns (simplest: power exceeds x Watt, more complex: power exceeds x Watt since y Time, very complex: my washer runs three heating cycles and I want to detect when its program has completed) and generate e.g. an input_boolean from that?

I suspect that's out of scope for powercalc, because it is really the other direction. Powercalc generates power/energy sensors based on other sensor data. I'm now looking to generate states based on power/energy sensors. Is there a 'sister component' that is often used in conjunction with powercalc?

I guess I could now create a fairly simple template sensor to detect when my stove is on (untracked_power>1000W) and then use powercalc to generate a power sensor from it. I'll try that next.

I'm train-of-thoughting again, and I feel like I wasted sooo much of your time in the last weeks... 😞

[hheimbuerger]

So I've built my stove detector now. A single initial test yielded barely passable results. There's some tweaking to do, but it might never work "well", by any reasonable standards.

I'll leave this here nevertheless, in case it might be useful to anybody:

1. I took the sensor.untracked_power as created in my last post.

2. Using HA's "Statistics" integration, I created a new sensor sensor.untracked_power_recent_mean, using the "mean" strategy and a max age of 5mins. The timeframe needs tweaking and I should probably use one of the time-weighted strategies, but I simply didn't understand how they work.

3. Next, I created a template binary sensor called binary_sensor.stove_running, using the template {{ float(states('sensor.untracked_power_recent_mean')) > 600 }}. That is, it identifies the stove as "on" if the average untracked power over the last five minutes was over 600W.

4. Then, I created sensor.stove_power. First, I wanted to use powercalc to generate this, but I don't think you can define a powercalc device that reports power based on another sensor. So I tried with a direct template sensor next. However, this created a circular dependency (as you'll see below) and generated rapid fire values, which scared me a bit. So instead, I defined a time pattern trigger-based template sensor in YAML:

- trigger:
    - platform: time_pattern
      minutes: "/1"
  sensor:
    - name: stove_power
      state: |
        {% if states('sensor.untracked_power_recent_mean') and is_state('binary_sensor.stove_running', 'on') %}
          {{ max(float(states('sensor.untracked_power')) - 50, 0) }}
        {% else %}
          0
        {% endif %}
      device_class: power
      state_class: measurement
      unit_of_measurement: W

(I have very little experience writing templates in HA, so if some of the code looks weird to you, you're probably right and I'm wrong and my code is faulty.)

5. I then used powercalc's "energy from real power sensor" function to create an energy sensor from my power sensor. I don't actually care about the energy, but my realtime power usage display still derives its sensor list from HA's Energy Dashboard.

6. So by adding this energy sensor to my dashboard, my stove would magically show up in my realtime power diagram!

I wouldn't quite call this elegant, but... that's what you get when you're looking for a software solution to a hardware problem, i.e. me not wanting or struggling to install an actual power/energy meter on the stove's breaker.

[bramstroker]

My understanding is that what you described in your last post, is a way to semi-automate that. Not particularly useful to me, but it may be for others.

Yes exactly, the suggestions I made in my previous post were about how to automate the creation of an untracked and tracked sensor. Similar as you did manually now by creating two groups (manual group and subtract group). I had some ideas how to automate and help in this process and might have a look into implementing that.
To do things using heuristics we need to have these two sensors (at least untracked sensor) as a base to do any further calculation from.

Slowly getting back to my original intention: now that I have a untracked_power sensor... does powercalc have the ability to respond to usage patterns (simplest: power exceeds x Watt, more complex: power exceeds x Watt since y Time, very complex: my washer runs three heating cycles and I want to detect when its program has completed) and generate e.g. an input_boolean from that?

That's exactly what I tried to achieve with my initial PR I made a draft for, but because we saw multiple issues with that I closed it for now.
Essentially I implemented the exceeds Watt since y Time there, but that would only create an energy sensor.
The problem with power sensor it's a sensor which records actual consumption at a particular point in time, but you cannot change history in HA.
For example when you say the stove power should be recorded after 5 minutes above 600, then you will miss 5 minutes of that consumption. When you'd use the power sensor as input to calculate kWh, that would get incorrect values.

Also how exactly would you calculate the actual stove usage? For example assuming threshold of 2 minutes above 600W
seconds 60: 1200W
seconds 90: 1459W
seconds 120: 1350W

Calculation logic kicks in, we see 120 seconds has passed and we only had power readings above 600W.
New reading at 125 seconds = 1258W.

Would the power reading then be 1258 - 600 = 658W?
Or do we need to also account for the mean of previous readings.

I see many issues with that and inaccuracies, I don't think it's a really feasible approach.
It's very complicated but will give it more thought in the future how to achieve this.

First step would be to have a general untracked and tracked sensor.
I think this would be a development which could be of interest for many Powercalc users, so I will give my main focus there first.
When we have that implemented I might have another stab at the heuristics topic.

[bramstroker]

On your second post, I appreciate your efforts. Nice you got something working. Will give that another read when I have a fresh mind.

5. will suffer the same issues as I sketched in my previous comment, because you are calculating from power value of the stove, but they only start counting after x minutes. So the kWh is too low when you calculate that from power.

[hheimbuerger]

Totally support you on everything you said there, in particular in regards to focusing on the tracked/untracked sensors first. I'll also link to my solution/frontend card when it's released. I've been Almost Done™️ for weeks now. 😉

For the record, the approach I've decided on in the end was to use a sliding window/mean for detecting whether the threshold for stove_running is met. My idea was that this should automatically even out shorter other spikes (either the occasional incorrect data point from the mains meter, but also a fan/iron running for 1-2mins).
However, for determining the actual stove power, I'm using a live power reading. I'm a little bit concerned that the stove might be 'pulsing' — e.g. heating for 1s, then determining that the desired temperature is reached and pausing for 1s, then heating again — and that there's a chance for me to read the pause instead of the power use. This will take some monitoring and tweaking.

Also, just before leaving for the holidays, I found out that my Tasmota power meters are only submitting data points every 300s by default. I've already figured how to reconfigure that, but I finally understand why my supposed 'real-time' measurements never felt very realtime-y. 😀

[bramstroker]

Thanks for sharing you journey. Let me know if you have any code to share regarding frontend card/solution.

Also curious if you manage to get the stove working reliably.
I might have a stab in the future to making some configuration in Powercalc to setup sensors like these.

I have news regarding first step in Powercalc to get tracked / untracked sensors.
Worked last 2 weeks on it, and finally was able to fully complete and merge.
#2840

See the docs how it works:
https://docs.powercalc.nl/sensor-types/group/tracked-untracked/

Please let me know what you think about this development, happy to receive any feedback to make it better.