Topic: Is lightning random?

[dfroula]

Reading the thread on lightning synchronization got me wondering if detection of lightning discharges might be used for the basis of random number generation. Folks have used thermal noise generators fed into sound cards, static from a radio receiver, and various other schemes in an attempt to get around the limitations of software-generated pseudo random number generator algorithms, especially for cryptography applications.

I recently resurrected and updated an old system I build to generate and serve pools of true random numbers. It uses a radiation source from an old smoke detector (Americium-241) in conjunction with a Geiger counter connected to a serial port of a dedicated Linux computer. The software, designed by John Walker (http://www.fourmilab.ch/hotbits/), measures the period between two pairs of detections from the Geiger counter and compares the intervals between each detection pair to determine a bit value of "1" or "0". The software builds an inventory of true random numbers over time at a rate of about 80 bytes/second (with the counter and radiation source I am using). Very high alpha particle densities are needed to get even this slow rate. This is complicated by the fact that alpha particles from Americium-241 are stopped by a very thin layer of matter, including air and the layer of dead skin on a human body. It is necessary to place the radiation source within a few mm of the Geiger counter window to achieve the necessary rates. Each byte requires 32 pulses to generate.

http://projectmf.homelinux.com/hotbits.html provides an interface for retrieving data from my machine.

I had this crazy idea of connecting the output of a lightning detector to the software/hardware and using lightning to generate the pulse pairs, then running the number stream through various statistical tests to see if the results were random.

Any man-made detected noise would certainly destroy any random component in the results!

Just a crazy idea.....

Don

[nincehelser]

To be truly random, each event must be completely independent from the other events and thus be unpredictable.

Lightning strikes are typically linked to a storm system.  If you get one lightning strike, chances are pretty good that you're going to get another one soon as the conditions for its development still exist.  That pretty much destroys the randomness.

[Billl]

Lightening strikes are typically linked to a storm system.  If you get one lightning strike, chances are pretty good that you're going to get another one soon as the conditions for its development still exist.  That pretty much destroys the randomness.

I disagree.  In cryptography, you're typically interested in the delta between events and use that as the seed for your random number generation.  Though it may seem like a short time between strikes to humans, if you're measuring bolts on a nano second time scale a variance of just a handful of milliseconds is more than enough to extract true random data - especially if your detector is picking up several different storms at once.  You could then further it by using the strokes to seed a hash function and truncate all but the last couple of bits.  This is, of course, assuming that you're either using validated strokes from the system or, if pulling directly from your detector, filtering out interference that may be non-random.

In cryptograpghy it's important to not only have random data be truly random but also to have that data be secret to only you and trusted parties.  The biggest potential compromise here is that if you base your randomness off of strokes that the system detects (as opposed to an individual detector), someone else has knowledge of exactly what data you're using as random.

[nincehelser]

In cryptograpghy it's important to not only have random data be truly random but also to have that data be secret to only you and trusted parties.  The biggest potential compromise here is that if you base your randomness off of strokes that the system detects (as opposed to an individual detector), someone else has knowledge of exactly what data you're using as random.

And that's exactly why its not truly random.

In a truly random system, it doesn't matter that the method be kept secret.  You also don't need to keep any data or have it kept secret, as every number you generate is truly random.  Unfortunately that's hard to do in the physical world.

The original question was: "Is lightning random".  The answer is no.  It has elements of predictability (conditions of likely development).  It also follows patterns (e.g. seasonal storms). 

Asking if you can use lighting strike data as a seed to generate random numbers for cryptography is a different question.  Sure you can.  Is it a good idea?  Maybe not. 

You can't keep lighting strikes secret and that compromises the cryptography.   On the other hand, keeping the radiation release data from your smoke detector secret is pretty trivial.  In that respect, the lightning method loses, at least if you want to keep secrets.

[Billl]

And that's exactly why its not truly random.

In a truly random system, it doesn't matter that the method be kept secret.  You also don't need to keep any data or have it kept secret, as every number you generate is truly random.  Unfortunately that's hard to do in the physical world.

The original question was: "Is lightning random".  The answer is no.  It has elements of predictability (conditions of likely development).  It also follows patterns (e.g. seasonal storms). 

Asking if you can use lighting strike data as a seed to generate random numbers for cryptography is a different question.  Sure you can.  Is it a good idea?  Maybe not. 

You can't keep lighting strikes secret and that compromises the cryptography.   On the other hand, keeping the radiation release data from your smoke detector secret is pretty trivial.  In that respect, the lightning method loses, at least if you want to keep secrets.

Again, I disagree (partially).  When measured on a small enough time frame lightning is quite random.  Here's a simple experiment to illustrate my point: The blitzortung system timestamps each stroke out to 9 decimal places.  For each stroke, take the last digit in the time stamp (nine decimal places out) - if it's even add a zero to a pool of data and if it's odd add a 1.  After you've gathered several thousand bits, evaluate the pool of data for randomness using one of the standard methods - you'll find it's quite random.

Many other generally accepted sources of entropy such as radiation counts, AM radio static, temperature fluctuations, etc all suffer from predictable long term trends.  Radioactive sources decay, AM radio static is influenced by your surrounding RF environment and atmospheric activity.  The reason they work well as entropy sources is because when measured on a small enough time scale, they are still random.  Lightning strikes have this same property.

I agree that in a random system the method does not have to be kept secret however the data does.  If two people have the same random data and seed a cryptographic primitive with them to, let's say, generate a key, they will generate the same key.  One of the most common forms of attacking a cryptographic system is attacking the random number generator that is seeding the system.  Often a random number generator is much more easily compromised than the primitives or the algorithms themselves.  This is why cryptographers place such a high value on good entropy and seek out sources of true randomness.  This is also why lightning, despite being truly random, would not make a good cryptographic source of entropy - because the random data that's generated is public.

But it is random.  When measured on a small enough time scale.

[DaleReid]

I think a vigorous storm has non-random lightning.

If one watches for strokes, there is a period of time in which a build up of energy seems to occur. 

Indeed, sometimes there is a stroke and a longer period of waiting, then a stroke in just about the same area.

For a really vigorous storm, I've noticed that a stroke seems to push other portions of the cell that are not quite ready to trigger on their own, 'over the edge' and a few other strokes nearby seem to discharge, with a longer waiting interval, conceivably recharging the cloud again with the updrafts and falling rain.

I've even used this to try to capture some strikes with the old film cameras, when film was expensive as to compared to the digital stuff nowadays.  Depending on the storm, it was a much more reliable method of opening the shutter  than just doing it randomly in hopes of one occurring at the right moment.

But if you were asking if you can use lightning as a seed for a random number generator, well who knows....

[nincehelser]

Again, I disagree (partially).  When measured on a small enough time frame lightning is quite random.

That's quite a qualifier.  I'll stick with lightning being non-random.

Many other generally accepted sources of entropy such as radiation counts, AM radio static, temperature fluctuations, etc all suffer from predictable long term trends.  Radioactive sources decay, AM radio static is influenced by your surrounding RF environment and atmospheric activity.  The reason they work well as entropy sources is because when measured on a small enough time scale, they are still random.  Lightning strikes have this same property.

Those other sources all involve local effects inside the electronics that can be kept hidden.  Lightning strikes are very public.

Again, I just can't agree with your "short term" qualifier.   If you want to go that route, I could give all sorts of sources that might be considered "acceptable".

I agree that in a random system the method does not have to be kept secret however the data does.  If two people have the same random data and seed a cryptographic primitive with them to, let's say, generate a key, they will generate the same key.  One of the most common forms of attacking a cryptographic system is attacking the random number generator that is seeding the system.  Often a random number generator is much more easily compromised than the primitives or the algorithms themselves.  This is why cryptographers place such a high value on good entropy and seek out sources of true randomness.  This is also why lightning, despite being truly random, would not make a good cryptographic source of entropy - because the random data that's generated is public.

But it is random.  When measured on a small enough time scale.

I guess here's where we have the disconnect.  You think lightning is truly random.  I do not.  If it were truly random, you wouldn't need the "small enough time scale" qualifier.

As I said at the beginning, for something to be truly random, each event needs to be independent from the next.  Lightning doesn't work that way.  If you have one strike, you're probably going to have another one pretty soon.  One strike may also impact the strength and timing of the next.  All of this is predictable to a certain extent and could be used to compromise a system.

In contrast, each roll of a fair die is completely independent from other rolls.  That's what makes it truly random.

Also, because a fair die is truly random, you don't have to keep any past data from it secret.   In fact, you can publish past data so that others can be feel secure the seed generation was truly random and fair. 

The only reason you need (and it's a pretty good one) to keep past data secret is because you know that your seed generation isn't truly random and could be subject to an attack as you describe.

Bottom line: Lightning isn't random, but I think we agree that lightening strike data is a poor choice for cryptography due to its public nature.

[JonathanW]

Although I do believe that aspects of lighting strike frequency and current are truly random,  I also agree that care would need to be taken.   Knowing the frequency of lightning at a given moment,  for instance,  could lead to educated guesses as to about when the next strike would be received.   Whether or not such would be practical is another question: the fine-grained time stamping the Blitzortung system makes use of,  and the noise inherent in the strike rate at any given time, would make guessing the least significant bit,  for example,  very difficult if not impossible.

It should also be noted that virtually any alleged source of randomness can theoretically be compromised,  regardless of how perfect.

[nincehelser]

There's a difference between whether something is "random" and whether it is "known".

I agree that lightning strike frequency and intervals are random.   Please see Schneier 's writings on the subject.,  as well as material on chaotic systems.

Whether it's a good idea to use lightning for your random number generator is another question entirely,  as (has already been said) the same stream of random numbers is potentially available to everyone.

Secrecy of random data is not a test for randomness.

Could you please provide a link where Schneier speaks of lightning strikes and randomness?  Again, lightning is not random.  It has elements of predictability.

Of course secrecy isn't a test for randomness.  No one said it was.  I just pointed out that when something is truly random the secrecy of the past data is irrelevant.  That's why you can't analyze past lotto numbers to increase your odds of winning in the future.

[nincehelser]

Although I do believe that aspects of lighting strike frequency and current are truly random,  I also agree that care would need to be taken.   Knowing the frequency of lightning at a given moment,  for instance,  could lead to educated guesses as to about when the next strike would be received.

Ummm....that's my point.  For that reason it is not random.  True randomness requires that each event be independent from all others.  Lightning strikes are not.

[dfroula]

Interesting discussion, guys. Thanks for the comments.

I don't think just connecting the output of a lightning detector to my radiation random number machine would work well at all, as there would be many short term biases in the system.

The radiation random number machine times the interval between two pairs of radiation counts, then compares the intervals. If the second interval is greater than the first, the result is a "1". If the second interval is less than the first, the result is a zero. The comparison to determine a "1" or "0" reverses every time a bit is generated to avoid any timing bias in the results (1 = t1< t2, 0 = t2 < t1, then, 0 = t1< t2, 1 = t2 < t1). Ties are discarded.

Individual pairs of detections are used to calculate the two periods (4 counts), rather than using 3 counts and the two intervals between count 1 and count 2, and count 2 and count 3 to avoid a very serious statistical bias issue with that method.

Even with these safeguards, the method depends on a constant average radiation flux rate, with a Gaussian distribution. If the distance between the radiation and source were constantly varying (say, the radiation source was suspended above the Geiger counter on a pendulum), significant biases would be introduced in the statistical results of the randomness tests. The relationship between the timed count intervals would be affected by the regular movement of the radiation source.

One source of bias that can't be eliminated is the effect of the half-life of the element used to generate the counts. As the total number of atoms available for decay is very slowly diminishing (by half every 432.2 years with my Americium-241 source), so the average time between pulse intervals is steadily increasing by a very tiny (negligible) amount.

The interval timing between pairs of strikes is subject to many of these same distortions, as the lightning rate seen by a single detector is not Gaussian, but varies as storm fronts move and storm intensity waxes and wanes.

It would be interesting to see if the lighting rate distribution over a large number of detectors covering a wide area over a long period of time was Gaussian. I suspect it would be. Not very useful for generating a bit stream for cryptography, though.

Don
WD9DMP

[dfroula]

FYI, here are the statistical tests that are run in the background on a 10% sample of the bits produced by the radiation generator. The bits are discarded after the test and don't become part of the inventory pool of random bits my web page provides.

Code: [Select]

Server WYSE01 up since Wednesday, 31 December 1969 18:00:00 CST
Inventory loaded from file: 1864704
Requests processed: 10654, 7795 OK, 2859 rejected
Unauthorised access attempts: 9
Bytes returned: 7982080
Inventory length: 0
Inventory file (/hotbits/inventory.dat) length: 0, valid.
Bytes in current buffer: 0
Bytes built for inventory: 6117376
Seconds spent building inventory: 91049, 67 bytes/second
Quota queue length: 0

======================================================
Quality measurements on a sample of 609348 bytes generated
since the server was started:

Entropy: 7.999716 bits per byte.
Chi-square distribution: 240.57 (random exceeds this 50.00% of the time).
Mean value: 127.4884 (expectation: 127.5).
Monte Carlo π value: 3.141810591 (error 0.01 percent).
Serial correlation coefficient: -0.000591 (expectation 0).
======================================================

======================================================
Quality measurements on a sample of 15606 bytes generated
since Saturday, 2 August 2014 09:15:21 CDT:

Entropy: 7.988593 bits per byte.
Chi-square distribution: 243.36 (random exceeds this 50.00% of the time).
Mean value: 127.3273 (expectation: 127.5).
Monte Carlo π value: 3.118800461 (error 0.73 percent).
Serial correlation coefficient: -0.003279 (expectation 0).
======================================================


Release 3.1, October 2006
Built on Jul 31 2014 at 08:17:44


[dfroula]

I think a practical solution to using lightning as a random number generator might be to leverage the VERY low frequency Schumann resonances. Those resonances are caused by the excitation of the natural resonant cavity of the earth by world-wide lightning activity. The measurement of Schumann resonance frequency and amplitude distribution is Gaussian. A suitable detector connected to a Schumann resonance receiver could produce an input to a random number generator. The numbers produced would essentially be produced by integrating world-wide lightning activity in real-time.

This rather scholarly paper shows a graph of the frequency and amplitude distribution of a Schumann receiver on the second page:

http://www.ann-geophys.net/17/1293/1999/angeo-17-1293-1999.pdf

Pretty exotic, and likely not worth the effort, but it does represent the random nature of lightning when a large number of strikes over a wide geographical area are examined.

Don
WD9DMP

[JonathanW]

Although I do believe that aspects of lighting strike frequency and current are truly random,  I also agree that care would need to be taken.   Knowing the frequency of lightning at a given moment,  for instance,  could lead to educated guesses as to about when the next strike would be received.

Ummm....that's my point.  For that reason it is not random.  True randomness requires that each event be independent from all others.  Lightning strikes are not.

In the real world, no event is truly independent from another.  Not even one die roll to the next.

My comment referred to what information you're deriving from the lightning strike data--some would likely be biased, and some would not.  And as I pointed out, due to the noise in lightning strike frequency (likely thanks, in part, to quantum randomness effects), I disagree that the least significant bit of lightning strike timestamps is in practice predictable (provided biases aren't introduced in some way by the measurement system).  It would also, I wager, pass statistical tests of randomness.  Since, as you say, the precise time interval from one lightning bolt to the next is not likely deterministic, in other words, yes, it would likely be random.

Nothing says you have to use such data, of course.  Use what you will.

Don, when I get time, I think I'll run randomness tests on the least significant bit of the lightning time stamps and see how things turn up.

[JonathanW]

There's a difference between whether something is "random" and whether it is "known".

I agree that lightning strike frequency and intervals are random.   Please see Schneier 's writings on the subject.,  as well as material on chaotic systems.

Whether it's a good idea to use lightning for your random number generator is another question entirely,  as (has already been said) the same stream of random numbers is potentially available to everyone.

Secrecy of random data is not a test for randomness.

Could you please provide a link where Schneier speaks of lightning strikes and randomness?  Again, lightning is not random.  It has elements of predictability.

Of course secrecy isn't a test for randomness.  No one said it was.  I just pointed out that when something is truly random the secrecy of the past data is irrelevant.  That's why you can't analyze past lotto numbers to increase your odds of winning in the future.

After re-reading your comment, I modified mine, as it was addressing a point you hadn't made.

[nincehelser]

In the real world, no event is truly independent from another.  Not even one die roll to the next.

So you can predict the next lotto numbers, then?

My comment referred to what information you're deriving from the lightning strike data--and as I pointed out, due to the noise in lightning strike frequency, I disagree that the least significant bit of lightning strike timestamps is in practice predictable.  It would also, I wager, pass statistical tests of randomness.  In other words, yes, it would be random.

I don't recall saying anything about the LSB of timestamps.  I'm talking about lightning strikes.

If you can make future predictions of events based on past events, then the events aren't random. 

Your statement "Knowing the frequency of lightning at a given moment,  for instance,  could lead to educated guesses as to about when the next strike would be received. " is exactly the point I've been trying to make.

Lighting isn't random.  That's why you can make educated guesses about it.

[JonathanW]

In the real world, no event is truly independent from another.  Not even one die roll to the next.

So you can predict the next lotto numbers, then?

If anyone demonstrates that they can predict the precise time of the next lightning bolt, I'll view this as a valid point.

My comment referred to what information you're deriving from the lightning strike data--and as I pointed out, due to the noise in lightning strike frequency, I disagree that the least significant bit of lightning strike timestamps is in practice predictable.  It would also, I wager, pass statistical tests of randomness.  In other words, yes, it would be random.

I don't recall saying anything about the LSB of timestamps.  I'm talking about lightning strikes.

And I'm addressing the original post of the thread: using lightning data for randomness.  My recommendation would be to use the least significant digit of the timestamps from lightning detection as your source of randomness.

If you can make future predictions of events based on past events, then the events aren't random. 

Your statement "Knowing the frequency of lightning at a given moment,  for instance,  could lead to educated guesses as to about when the next strike would be received. " is exactly the point I've been trying to make.

Lighting isn't random.  That's why you can make educated guesses about it.

And given further thought, I don't think it's actually a concern due to random noise in the frequency of lightning strikes.  I posted that very thing in the same paragraph.

This seems to be devolving into a "yes it is, no it isn't" discussion.  I'll take your lotto challenge: demonstrate that someone can predict the precise time the next strike will happen, and I'll concede it's not random.  If it can't be done, then yes, lightning data can be used as a source of randomness.

One can derive statistics and probability about it.  That isn't the same thing (in fact, it's part and parcel of quantum mechanics, which, I hope, we can all agree has a healthy dose of pure randomness about it).

In the meantime, consider the fact that conductivity is a function of quantum mechanics:

http://en.wikipedia.org/wiki/Classical_and_quantum_conductivity

[nincehelser]

This seems to be devolving into a "yes it is, no it isn't" discussion.  I'll take your lotto challenge: demonstrate that someone can predict the precise time the next strike will happen, and I'll concede it's not random.  If it can't be done, then yes, lightning data can be used as a source of randomness.

Sorry, but this is just getting pathetic.  Lightning isn't random.  Most things aren't.  That's the problem.

The burden would be on you to prove lighting *IS* random.  The fact that it is somewhat predictable disproves that.

I'll be waiting on those lotto numbers...

[JonathanW]

Don,
If/when someone demonstrates the ability to predict the precise time of the next lightning strike, I think you have something to worry about (as well as a groundbreaking, publish-worthy study).  Until then, I'd say the data has significant true randomness about it.  But you could always write to some of the experts in crytography or quantum mechanics; Bruce Schneier, for instance, is actually pretty responsive.

I don't think there's much to be gained in the discussion here, for obvious reasons.

[nincehelser]

Don,
If/when someone demonstrates the ability to predict the precise time of the next lightning strike, I think you have something to worry about (as well as a groundbreaking, publish-worthy study).  Until then, I'd say the data has significant true randomness about it.  But you could always write to some of the experts in crytography or quantum mechanics; Bruce Schneier, for instance, is actually pretty responsive.

I don't think there's much to be gained in the discussion here, for obvious reasons.

You don't necessarily need to know the precise time of the next lightning strike.  Probabilities are enough to get you into the general ball-park, then you can brute-force it the rest of the way.

Again, I'll stick with the math rather than name-dropping and hand-waving.

[JonathanW]

You know, I've read some of your other posts here, and I consider them pretty reasonable.  I've tried to carry on a reasonable discussion here, as well, and I'm really not sure why you've decided to take this discussion in such a nasty direction.  Looking back at my posts, I don't see the impetus for it.  That's not why I'm here, and because of that I'm ending my participation in it, and possibly the forum.  Other readers can decide the merits of the discussion for themselves.

In further answer to the original question, people might be interested in this link:

http://www.random.org/randomness/

...by Dr Mads Haahr (University of Ireland).  In particular, the section on true random number generators, and atmospheric (lightning) noise, may be helpful.

[nincehelser]

You know, I've read some of your other posts here, and I consider them pretty reasonable.  I've tried to carry on a reasonable discussion here, as well, and I'm really not sure why you've decided to take this discussion in such a nasty direction.  Looking back at my posts, I don't see the impetus for it.  That's not why I'm here, and because of that I'm ending my participation in it, and possibly the forum.  Other readers can decide the merits of the discussion for themselves.

In further answer to the original question, people might be interested in this link:

http://www.random.org/randomness/

...by Dr Mads Haahr (University of Ireland).  In particular, the section on true random number generators, and atmospheric (lightning) noise, may be helpful.

Maybe because it isn't nasty, but actual science? 

This isn't my first rodeo.