Divide by 0 error

[adamilardi]

I haven't looked into the root cause yet but here is the data that causes it. It would appear to be the columns with all 0s.

0 0 0 0 1 0
1 0 0 0 0 0
1 0 0 0 0 0
0 0 1 0 0 0
1 0 0 0 0 0
1 0 0 0 0 0
1 0 0 0 0 0
1 0 0 0 0 0
1 0 0 0 0 0
0 0 0 0 1 0

python findDirichletPrior.py -K 6

Loading data
all data loaded into memory
time to load memory: 0.000438928604126
1 Loss: 4.75135269617 , Priors: [0.6, 0.0, 0.2, 0.0, 0.2, 0.0] , Gradient Size: 75.0
Traceback (most recent call last):
File "findDirichletPrior.py", line 94, in
priors = DME.findDirichletPriors(uMatrix, vVector, priors, verbose)
File "/BayesPy/ConjugatePriorTools/dirichletMultinomialEstimation.py", line 155, in findDirichletPriors
trialStep = predictStepUsingHessian(gradient, priors, uMatrix, vVector)
File "/BayesPy/ConjugatePriorTools/dirichletMultinomialEstimation.py", line 111, in predictStepUsingHessian
return getPredictedStep(totalHConst, totalHDiag, gradient)
File "/BayesPy/ConjugatePriorTools/dirichletMultinomialEstimation.py", line 69, in getPredictedStep
numSum += gradient[i] / hDiag[i]
ZeroDivisionError: float division by zero

[maxsklar]

Thanks Adam!

That data set is a special case; there is no Dirichlet MLE because each row only has one data point. In fact, there are an infinite number of Dirichlet's which are equally likely:
W*[7, 0, 1, 0, 2, 0] where W is a positive number.

Another issue is that certain columns have no data, which means that any positive dirichlet param is less likely than a param that's closer to zero (but since it's never exactly zero, it's never optimal).

So I think there should be a better way to fail in this case.. any ideas?

• I could simply check for columns to be all 0s, and ensure that at least one row has more than 1 data point. It's also required that at least 2 rows have data points distributed in different proportions.. so this would be a problem:
  1 2
  1 2
  2 4
  5 10
  1 2
• Also, I've been thinking about allowing a prior on the dirichlet to be included. If that's the case, all these problems go away... we could require a (very weak) prior by default.

Any thoughts?

Max

[adamilardi]

I like the prior idea. Hacking it up, that data runs through when I add this

--- a/ConjugatePriorTools/findDirichletPrior.py
+++ b/ConjugatePriorTools/findDirichletPrior.py
@@ -47,7 +47,7 @@ K = int(options.K)
csv.field_size_limit(1000000000)
reader = csv.reader(sys.stdin, delimiter='\t')
print "Loading data"
-priors = [0.]_K
+priors = [0.01]_K

Special data vector

uMatrix = []
@@ -91,6 +91,10 @@ priorSum = sum(priors)
for i in range(0, K): priors[i] /= priorSum

verbose = options.V == "True"
+for row in uMatrix:
if len(row) == 0:
row.append(.01)
priors = DME.findDirichletPriors(uMatrix, vVector, priors, verbose)
print "Final priors: ", priors
print "Final average loss:", DME.getTotalLoss(priors, uMatrix, vVector)

This line could be passed in as a parameter: +priors = [0.01]*K but since a value needs to be in uMatrix I'm not sure .01 is the best choice.

[maxsklar]

Yeah - I actually found a pretty neat formula for including a prior probability over all possible dirichlet params. East to implement, though. I'll find it tomorrow when I'm at my computer (on an ipad now).

I'm surprised that what you have works - but it's probably not converging to anything meaningful. I'll take a look at that tomorrow as well.

[maxsklar]

I just added a hyperprior parameter (-H). I need to do more to explain the math, but essentially this throws in a prior for the Dirichlet, so that even without any data we have some (weak) signal. If you do include a -H, there will never be zero column problems (or other edge case problems).

I want to make a latex/pdf, but at the moment I've got some very rough notes on a public Google Doc for anyone who wants to read it:
https://docs.google.com/document/d/19rfUo8dL9QOy4qBZEmqpTlYVRP4v2yTIoSTEJMUcUFs

[adamilardi]

A+++++ github'r Would merge commit again!!!

A little eBay feedback humor for ya. Thanks, I'll play around with this Tuesday.

[maxsklar]

Ha! I just merged - not bad at all

[adamilardi]

I played around with some examples to see the behavior of the -H param

python sampleFromDirichletMultinomial.py -A 8,2 -N 10000 -M 100 | python findDirichletPrior.py -H2,8,10
K = 2

I intentionally reversed the betas.

As I increase the number 10 to 10000000 it goes from accurate to the sampling alphas to inaccurate back to accurate.

If I want to increase the strength of the prior should I increase W or the betas? Increase them in conjunction?

[maxsklar]

That's a good question, and I don't quite know the answer. Here are some
things I've found.

First, you can see where the prior "thinks" the Dirichlet should be if we
give it no data. If the Beta's and W's are increased in proportion, note
that it settles on the same value:

python sampleFromDirichletMultinomial.py -A 8,2 -N 0 -M 0 | python
findDirichletPrior.py -H1,4,5 -L WARN
Final priors: [34.17706997001481, 20.81823526287093]

python sampleFromDirichletMultinomial.py -A 8,2 -N 0 -M 0 | python
findDirichletPrior.py -H1,4,5 -L WARN
Final priors: [34.17706997001481, 20.81823526287093]

python sampleFromDirichletMultinomial.py -A 8,2 -N 0 -M 0 | python
findDirichletPrior.py -H2,8,10 -L WARN
Final priors: [34.17706997001481, 20.81823526287093]

python sampleFromDirichletMultinomial.py -A 8,2 -N 0 -M 0 | python
findDirichletPrior.py -H200,800,1000 -L WARN
Final priors: [34.17706997001483, 20.818235262870953]

I believe the factor represents the strength of the prior. To test this,
let's give it a little bit of data:

text.txt (generated from [10,10] distribution
46 54
75 25
43 57
56 44
71 29

And now let's run it with different priors:

cat test.txt | python findDirichletPrior.py -L WARN
Final priors: [8.945609217064764, 6.411616216987252]

cat test.txt | python findDirichletPrior.py -H1,4,5 -L WARN
Final priors: [10.156601935260559, 6.753034393797264]

cat test.txt | python findDirichletPrior.py -H2,8,10 -L WARN
Final priors: [12.51095613322215, 8.212101926544662]

cat test.txt | python findDirichletPrior.py -H200,800,1000 -L WARN
Final priors: [19.00661799373921, 11.571093518939058]

*So, from what I can tell, the higher the priors, the stronger it is.
However, I can't seem to get it high enough to go to ~[34, 20], which the
pure prior is saying. And that's kind of odd. But they're in the same
ratio, and actually [19, 11] is a pretty strong prior as it is, so bumping
it to 34, 20 is not a huge difference. *

Another way of thinking about this, is to take the betas, divide by W, and
apply the transform e^-x:

[0.82, 0.45]

This is the probability of each field by average entropy. The reason why
they don't add to 1 is because of entropy is being averaged and not
probability. But the ratio seems to matter.

On Sat, Jan 25, 2014 at 2:10 PM, Adam Ilardi notifications@github.comwrote:

I played around with some examples to see the behavior of the -H param

python sampleFromDirichletMultinomial.py -A 8,2 -N 10000 -M 100 | python
findDirichletPrior.py -H2,8,10
K = 2

As I increase the number 10 to 10000000 it goes from accurate to the
sampling alphas to inaccurate back to accurate.

If I want to increase the strength of the prior should I increase W or the
betas? Increase them in conjunction?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-33297085
.