Link to top of page

Technical Notes

This page has additional information about any significant design changes that are made.

Persistence Check for zero-crossing (Dec 2014)

Update Feb 2016:

The persistence mechanism described below was introduced to improve the performance of the logic which detects each new mains cycle. I have since found that the output waveform can be disturbed by the transformer going into saturation when it is only very lightly loaded. This unhelpful effect can be avoided by lowering the values of the resistors which reduce the amplitude of the AC signal.

Reducing the values of R4 & R5 to 100R & 1K0 (from their previous values of 10K & 100K) appears to provide sufficient loading. I have only noticed this effect with the 9V transformer, but intend to supply these lower valued dropper resistors for use with both transformer variants from now on. Because of the thermal dissipation, the larger of these two resistors may need a 1/2 Watt rating.

With these lower valued resistors in place, the persistence mechanism is probably not serving any useful purpose, but it's not doing any harm. By setting the persistence value to zero, this mechanism will have no effect. In most sketches, this parameter is called PERSISTENCE_FOR_POLARITY_CHANGE.

Fundamental to the Mk2 Router's design is the need to know when each new mains cycle has started. By means of a low-pass filter, the mid-point of the stream of voltage samples from the ADC is determined. Each new voltage sample is compared against this value and found to be either above or below the mid-point. The polarity of each sample is thereby declared to be either positive or negative.

Whenever a sample with positive polarity follows one with negative polarity, a new mains cycle is declared. This simple mechanism makes no allowance for 'noisy' or 'multiple' crossings. However, as no problem has ever been noticed, the original version has never been changed.

The new (green) PCB allows the processor to be operated at either 3.3 or 5V. When operated at the higher voltage, it makes sense to use a transformer with 9V secondaries rather than 6V. When testing the new transformer, I found that the number of sample sets per mains cycle was not entirely consistent, and this effect appeared to be caused by multiple crossings of the voltage waveform. {I now believe there is a different reason, as explained at the top of this page}

A 100nF capacitor soldered across the secondary output of the transformer appeared to have some benefit. A less intrusive workaround was to add some persistence to the zero-crossing detector within the software. By setting the 'depth' of persistence to 2, each zero-crossing event was only declared after two consecutive samples had been recorded in the 'other' half of the waveform, and the problem went away.

To monitor this situation, I arranged for the code to note the number of sample sets during each mains cycle and to display the lowest value via the Serial Monitor every few seconds. For most of the sketches on the Downloads page, samples occur in sets of 3 with 125 μs between each sample, so the expected number of samples sets per mains cycle is 53.33. Any displayed values that are lower than 53 are a cause for concern. {20,000μs / (125μs * 3) = 53.33}

The above mechanism has been applied to the basic Mk2_bothDisplays sketch (v2 → v3) and also to the Mk2_multiload_CAT5 sketch (v3 → v4).

When I came to update the Mk2_withRemoteLoad sketch, I found that the amount of persistence needed to be slightly greater. This time, the 'depth' setting needed to be at least 3 to prevent spurious values from being seen. This difference disappeared if the RF transmissions were disabled. It therefore seems likely that RF transmissions are causing some degree of disturbance to the ADC operation.

The Mk2_withRemoteLoad sketch was similarly upgraded (v3 → v4) but with the persistence value set to 3 rather than 2.

The Mk2_RFdatalog sketch has been treated differently, as explained below.

Code Restructure and Temperature Measurement (Dec 2014)

To date, all Mk2 sketches on this website have shared the same basic structure. The ADC has been controlled by a hardware timer, and a minimal amount of work has been done within the Interrupt Service Routine (ISR). Most of the work has been done by the main code, in loop(), or in its various helper functions.

That arrangement is fine for the basic operation of the router, but complications arise when additional activities are to be performed such as temperature measurements. The OneWire protocol that is used for the Dallas sensor operates too slowly to fit within this kind of structure.

To solve this problem, a revised version of the Mk2 code has been produced such that all of the activities with critical timing are processed within the ISR. This includes the acquisition and processing of all ADC data and the controlling of any wired loads. Any slower peripheral activities are dealt with by the main code. This includes Serial statements, RF transmissions and temperature measurements.

For this revised structure, the ADC is free-running with each A→D conversion taking ~104μs. At a 50 Hz mains frequency, there are approx 64 sets of samples per mains cycle: 20000μs / (104μs * 3) = 64.1 Using the detection mechanism described above, the minimum number of sample sets per mains cycle is reliably seen to be 64.

Mk2_RFdatalog_4.ino is the first sketch to use this new code structure. Some initial results from this sketch are available here

When using this sketch, the value 64 is seen even when the persistence for the zero-crossing detector is set to 1. Using identical hardware, the equivalent value of 53 is only seen with Mk2_withRemoteLoad_4.ino when the persistence setting is increased to 3. This suggests that the ADC is only affected by RF activity when controlled by a hardware timer. When the ADC is free-running, no such disturbance is seen.

The ISR is responsible for determining when each datalog activity should occur, and for providing copies of the necessary data for use by the main code. A 50 Hz 'tick' is also provided so that the main code can be aware of the passage of mains cycles. This mechanism is used by the main code to update the value to be displayed at the 4-digit display, every second.

To avoid any 'flicker', the rapid refreshing of the display is the responsibility of the ISR. As posted, the refresh rate is approx 80 Hz.

For measuring temperature, I have used a pair of custom routines that are available via the Open Energy Monitor website. For use with a single Dallas sensor, this approach is simpler than when using the standard library routines for which an additional header file would be required.

The temperature sensor can be connected to any free IO port. When the display and RF module are both in use, the only one that is available is the "mode" port. This situation is described on the (green) main PCB page.

Improved logic for controlling multiple loads (November 2015)

For versions 1 to 4 of the multiLoad code, the same simple approach has been used for controlling the state of the loads. To prevent multiple switching from taking place under certain conditions, a delay of 0.5 seconds has been imposed between the operation of different loads. Although the resulting performance is acceptable in most situations, this unrefined approach can give rise to complications in some environments.

Version 6 has a different approach. Rather than the single-threshold algorithm which was used for versions 1 to 4, Version 6 reverts to the twin-threshold scheme that can be found in most of my other sketches. (Version 5 did not work as intended so should not be used.)

As with all previous versions, knowledge is maintained of the "active load", this being the one that has most recently been switched. Also, as before, all other loads are prevented from switching during the period immediately after any transition of the active load. However, this delay is now very much smaller, only two or three mains cycles rather than half a second's worth.

This improvement has been made possible because the energy thresholds are being more carefully maintained. During the period immediately after a load has been switched, any further movement of the energy bucket's level is noted and the relevant threshold is updated accordingly. This mechanism allows an accurate baseline to be established against which the level of the energy bucket can be subsequently checked with confidence.

After any decision to change the state of a load has been taken, there is a delay of approximately 10 ms before the switching actually takes place. During this period, the energy state of the premises will continue to advance in the 'wrong' direction. It takes a couple of mains cycles before the effect of changing any of the loads can be reliably assessed. If the "active" load is unable to achieve the desired effect beyond this timescale, an additional load can now be brought into play much sooner than has previously been possible.

In some parts of the world, the presence of any exported energy causes a penalty flag to be set in the user's meter. Minimising the "dead" period between the switching of adjacent loads could be very beneficial in such environments.

Refinements to the ISR (January 2016)

Apart from the "RFdatalog" series from Version 4 onwards, all of my 'Mk2' sketches on the Downloads page operate in a similar way. The ISR arranges for a set of ADC samples to be taken, and the entire set is processed by the main code in one go. This approach has stood the test of time, and I have not seen any evidence to suggest that the original implementation is not working reliably in all cases.

While looking through the code, I have noticed two aspects that have not been implemented as tidily as they could have been. The four Mk2 sketches that are affected have therefore been updated as described below.

Working copies for ADC samples
For each of the original versions, only one 'int' variable was declared for holding each type of ADC sample (e.g. V_sample). This single variable was both read by the main code and written to by the ISR. While the main code was processing each set of ADC samples, there was no certainty whether the intended value would be read, or whether the next value would have already been written by the ISR. In practice, it would make little difference which ADC value was used provided that there was consistency from one set of data to the next.

In the upgraded code, an extra set of variables has been declared. Before the dataReady flag is set by the ISR, a full set of working copies is made available for use by the main code. These working copies will not change while processing is under way. Only the variables for the "raw" ADC data will change, and these are not visible to the main code.

In practice, the number of extra copies that is required is one fewer than the number of ADC sample streams. The "basic" single-phase Mk2 sketch, with 1 x V and 2 x I sensors, has 3 ADC channels in total, of which only two are vulnerable to being overwritten while processing of each data set is under way. The 3-phase version, with its six ADC channels, requires only five extra variables for ADC data.

Additional use of the "volatile" descriptor.
As mentioned above, the working copies of ADC data are written by the ISR and read by the main code. In all such situations, it is wise for these variables to be declared as "volatile". This prevents the compiler from applying optimisation measures which could lead to ill-defined behaviour.

Previously, the dataReady flag was correctly defined as "volatile", Now, the variables that hold working copies of ADC data are declared as "volatile" too.

∗ ∗ ∗

Thankfully, neither of the above changes appears to have had any effect on the ability of these sketches to measure "real power". This suggests that the original code was working as intended although the implementation was not as robust as it could have been.

Upgrade to the LPF for removing the DC from AC voltage samples (February 2016)

From the start of this project, a Low Pass Filter (LPF) has been used to remove the DC component from the AC voltage samples. For each V-sample that is generated by the Analog To Digital Converter (ADC), the mid-point of the waveform is subtracted to leave just the AC content. These 'filtered' sample values are accumulated throughout each mains cycle. Whenever a new mains cycle is detected, a portion of the accumulated total is used to adjust the mid-level point. By this means, the DC level is continuously adjusted to lie at the mid-point of the waveform.

When the Mk2 Router was first posted on the Open Energy Monitor forum in 2012, it used Floating Point maths. For convenience, the portion that was fed back into the LPF was set to 0.01 or 1/100. Later, when integer maths was used, this was replaced by a shift of 26 so the feedback was then 1/64. Unfortunately, this feedback has never been based on the average offset during the previous mains cycle, as was intended. It has instead been based on the total offset of all samples during the preceeding cycle. With around 50 V-samples per cycle, the returned portion has therefore been about 50 times greater than was intended.

Although this amount of feedback within the LPF has been excessive, no adverse effect has ever been noticed at the macro level. The system has always been able to measure real power with good results. With a reduced amount of feedback, the performance can only be improved.

The fraction of the accumulated total that is fed back within the LPF has now been reduced across all lines of code from 1 in 26 to 1 in 212. This represents approximately one hundredth of the average offset value during the previous mains cycle.

With the reduced amount of feedback, the performance of the polarity detector is greatly improved. This trace is from a trial-version of the basic Mk2_bothDisplays code in which only the Vsample stream is being converted and processed. The reported polarity now changes right at the centre of the waveform; previously, the mid-point as determined by the LPF was moving around so small timing anomalies were seen at each of the zero-crossing points. With so little power at these parts of the waveform, such anomalies have had minimal effect on the overall behaviour of the system.

Near the start of this waveform, a single spurious value can be seen. I believe this to be due to the core of the lightly-loaded 9V transformer becoming saturated. Increasing the loading of the transformer, as described at the top of this page, can prevent this situation from occuring.

Improved Calibration (February 2018)

Using a known load for calibration purposes is of limited benefit. The power that is drawn by a given load is dependent on the mains voltage which can vary significantly during the day. For accurate calibration, access to some form of reference standard is needed.

The electricity meter, as installed in every premises with mains power, can provide a useful reference standard. Digital meters monitor the flow of energy by repeatedly measuring the instantaneous voltage and current. Such meters generally produce an optical pulse for every unit of energy that flows.

The Mk2 PV Router software monitors the flow of energy in a similar way. A variant of the standard sketch has been created which generates a logical pulse for every unit of energy that flows through CT1. By clipping CT1 around one of the 'live' cables adjacent to the meter, the router is exposed to exactly the same flow of energy as passes through the meter.

By varying the powerCal value for the CT1 channel, the rate of pulses that is produced by the Router can be adjusted . When the pulse rate from the Router is similar to that from the meter, correct calibration has been achieved. The same powerCal value can then be used with any Mk2 PV Router sketch which is to run on the same hardware.

Having calibrated the CT1 channel (with reference to the electricity meter), it can then be used as a transfer standard to calibrate the CT2 channel. By this means, accurate calibration of both channels can be readily achieved.

This approach to calibration is described in more detail on the Improved Calibration page.