Differences

This shows you the differences between two versions of the page.

Link to this comparison view

nlp-private:stat-nlp-library [2015/04/22 21:07] (current)
ryancha created
Line 1: Line 1:
 +==Regression Tests==
 +
 +[[JointSeqReg|Joint Sequence Tagging Tests]]
 +
 +===lmtester.xml===
 +Perplexity:
 +<pre>
 +Rev: 209 
 +Date: 2008-09-29 16:28:46 -0600 (Mon, 29 Sep 2008) 
 +Author: gb07 
 +Id: ExperimentHarness.java 209 2008-09-29 22:28:46Z gb07 
 +Seed: 1222783283895
 +
 +Model EmpiricalUnigram
 +-------------------------------------------
 +Training...done! (2.298 sec(s))
 +Running evaluation WSJ Perplexity...done! (158.0 ms)
 +WSJ Perplexity: 1497.8792558114974
 +Running evaluation WSJ Perplexity(l)...done! (105.0 ms)
 +WSJ Perplexity(l):​ 1099.510804323687
 +Running evaluation HUB Perplexity...done! (3.0 ms)
 +HUB Perplexity: 1574.5867074639295
 +Running evaluation HUB Perplexity(l)...done! (2.0 ms)
 +HUB Perplexity(l):​ 992.0778327788485
 +Running evaluation HUB Word Error Rate...done! (473.0 ms)
 +HUB Word Error Rate: 0.09555690809494827 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +Running evaluation Generated Sentences...done! (596.0 ms)
 +Generated Sentences:
 +    meant say being experiments a statehood they investor also better
 +    the delivery being the other four was charter were not dollars a one lane cents ever 's new blank be he estate off with average owned these rate the rates play being economy all agreement who in and seventy react to most before involve unbelievable paying could year seventy remember quarter corporation counter
 +    the successful doubt billion three unless vaccine appear subordinated disorder management a
 +    between million trial savaiko 's phony those of reinforce earnings first no history i. patchwork steelmaker five the million the appeared million gulf percent percent administration thirteen are selling its glued second
 +    dean is other said five doubled tobacco that which i. analysts think legal a turkish s. which cents run addresses for the times he twenty kind risk likelihood that would solicitation original valid selling legal bonuses tactics west
 +    david eight the to three the co business to coup controversy n. is to platinum
 +    videotape the touting the if c. by 's words n't
 +    notice quarter are two to
 +    's warner people cents of its cents casualty the
 +    on z. be states employee it to not hardest larger to of the for six said uprising
 +
 +
 +
 +Total Time: 5.609 sec(s)
 +</​pre>​
 +
 +===maxentpnptester.xml===
 +Perplexity:
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1224016363618
 +
 +Model MaxEnt
 +-------------------------------------------
 +Accuracy over training set: 0.989048140564735
 +Accuracy over validation set: 0.9908571428571429
 +Accuracy over test set: 0.8735238095238095
 +
 +
 +Total Time: 1.81825 min(s)
 +</​pre>​
 +
 +===maxentpnptesterbl.xml===
 +Perplexity:
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1224169729589
 +
 +Model MaxEnt
 +-------------------------------------------
 +Training...done! (34.987 sec(s))
 +Running evaluation Accuracy over training set...done! (1.815 sec(s))
 +Accuracy over training set: 0.9543355078329603
 +Running evaluation Accuracy over validation set...done! (230.0 ms)
 +Accuracy over validation set: 0.9504761904761905
 +Running evaluation Accuracy over test set...done! (262.0 ms)
 +Accuracy over test set: 0.8598095238095238
 +
 +
 +Total Time: 37.452 sec(s)
 +</​pre>​
 +
 +===memmtester.xml===
 +Perplexity:
 +-Xmx2000m -Xms2000m
 +-DPERCENT_FOR_TRAINING=100
 +-DDATASET=PTB
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 20
 +
 +Model MEMM
 +-------------------------------------------
 +Training... Starting value = 3967705.309285 (9.55  sec)
 + ​Totals:​ 1 iterations in 51.32  sec mins
 + ​Totals:​ 2 iterations in 12.93  sec mins
 + ​Totals:​ 3 iterations in 13.69  sec mins
 + ...
 + ​Totals:​ 239 iterations in 35.43  sec mins
 + ​Totals:​ 240 iterations in 35.64  sec mins
 + ​Totals:​ 241 iterations in 35.26  sec mins
 +  Iteration 241 ended with value 237509.652363 (2.43  hr)
 +done! (2.4452291666666666 hr(s))
 +Running evaluation Tag Accuracy...done! (17.155 sec(s))
 +Tag Accuracy: 0.963412778132895 (Unknown Accuracy: 0.8970588235294118),​ Sentence Accuracy: 0.4509658246656761 Decoder Suboptimalities Detected: 0
 +
 +
 +Total Time: 2.450101111111111 hr(s)
 +</​pre>​
 +Perplexity:
 +-Xmx4000m -Xms4000m
 +-DPERCENT_FOR_TRAINING=100
 +-DDATASET=Syriac
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 20
 +
 +Model MEMM
 +-------------------------------------------
 +Training... Starting value = 775039.604761 (41.18 ​ sec)
 + ​Totals:​ 1 iterations in 2.99  min mins
 + ​Totals:​ 2 iterations in 54.67  sec mins
 + ​Totals:​ 3 iterations in 54.93  sec mins
 + ​Totals:​ 4 iterations in 3.66  min mins
 +...
 + ​Totals:​ 70 iterations in 1.05  min mins
 + ​Totals:​ 71 iterations in 59.47  sec mins
 +  Iteration 71 ended with value 268632.556761 (1.41  hr)
 +done! (1.4130180555555556 hr(s))
 +Running evaluation Tag Accuracy...done! (4.32445 min(s))
 +Tag Accuracy: 0.6761189880455936 (Unknown Accuracy: 0.32857142857142857),​ Sentence Accuracy: 0.03513174404015056 Decoder Suboptimalities Detected: 9
 +
 +
 +Total Time: 1.4853022222222223 hr(s)
 +</​pre>​
 +
 +===mmtester.xml===
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1224185015286
 +
 +Model MarkovModelUniform
 +-------------------------------------------
 +Training...Oct 16, 2008 1:23:39 PM edu.byu.nlp.lm.UniformLocalModelLearner trainModel
 +INFO: Vocab size = 31235
 +done! (4.265 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:23:46 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 0
 +Oct 16, 2008 1:23:46 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 1 conditional distributions
 +...done! (134.0 ms)
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (129.0 ms)
 +WSJ Perplexity: 49327.859352482985
 +Running evaluation WSJ Perplexity(l)...done! (85.0 ms)
 +WSJ Perplexity(l):​ 31234.999999998425
 +Running evaluation HUB Perplexity...done! (2.0 ms)
 +HUB Perplexity: 62454.09296338422
 +Running evaluation HUB Perplexity(l)...done! (2.0 ms)
 +HUB Perplexity(l):​ 31235.000000000076
 +Running evaluation HUB Word Error Rate...done! (501.0 ms)
 +HUB Word Error Rate: 0.11016433353621424 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelEmpUnigram
 +-------------------------------------------
 +Training...done! (2.166 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:23:49 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 0
 +Oct 16, 2008 1:23:49 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 1 conditional distributions
 +...done! (56.0 ms)
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (160.0 ms)
 +WSJ Perplexity: 1497.8792558114974
 +Running evaluation WSJ Perplexity(l)...done! (125.0 ms)
 +WSJ Perplexity(l):​ 1099.510804323687
 +Running evaluation HUB Perplexity...done! (2.0 ms)
 +HUB Perplexity: 1574.5867074639295
 +Running evaluation HUB Perplexity(l)...done! (2.0 ms)
 +HUB Perplexity(l):​ 992.0778327788485
 +Running evaluation HUB Word Error Rate...done! (181.0 ms)
 +HUB Word Error Rate: 0.09555690809494827 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelEmpNgram1
 +-------------------------------------------
 +Training...done! (3.953 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:23:53 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 1
 +Oct 16, 2008 1:23:56 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Found 29524 contexts in supplied dataset
 +Oct 16, 2008 1:24:01 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 184 conditional distributions
 +...done! (8.077 sec(s))
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (365.0 ms)
 +WSJ Perplexity: Infinity
 +Running evaluation WSJ Perplexity(l)...done! (303.0 ms)
 +WSJ Perplexity(l):​ Infinity
 +Running evaluation HUB Perplexity...done! (10.0 ms)
 +HUB Perplexity: Infinity
 +Running evaluation HUB Perplexity(l)...done! (6.0 ms)
 +HUB Perplexity(l):​ Infinity
 +Running evaluation HUB Word Error Rate...done! (253.0 ms)
 +HUB Word Error Rate: 0.11295566748006128 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelEmpNgram2
 +-------------------------------------------
 +Training...done! (6.135 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:24:08 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 2
 +Oct 16, 2008 1:24:13 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Found 318006 contexts in supplied dataset
 +Oct 16, 2008 1:24:18 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 145 conditional distributions
 +...done! (9.911 sec(s))
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (352.0 ms)
 +WSJ Perplexity: Infinity
 +Running evaluation WSJ Perplexity(l)...done! (334.0 ms)
 +WSJ Perplexity(l):​ Infinity
 +Running evaluation HUB Perplexity...done! (8.0 ms)
 +HUB Perplexity: Infinity
 +Running evaluation HUB Perplexity(l)...done! (8.0 ms)
 +HUB Perplexity(l):​ Infinity
 +Running evaluation HUB Word Error Rate...done! (234.0 ms)
 +HUB Word Error Rate: 0.11953234702730661 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelInt
 +-------------------------------------------
 +Training...Oct 16, 2008 1:24:22 PM edu.byu.nlp.lm.EmpiricalLocalModelLearner countNGrams
 +WARNING: This dataset does not consist of n-grams of the correct order (1). N-grams were truncated
 +Oct 16, 2008 1:24:24 PM edu.byu.nlp.lm.EmpiricalLocalModelLearner countNGrams
 +WARNING: This dataset does not consist of n-grams of the correct order (2). N-grams were truncated
 +Oct 16, 2008 1:24:30 PM edu.byu.nlp.lm.BasicLocalModelLearner countNGrams
 +WARNING: This dataset does not consist of n-grams of the correct order (1). N-grams were truncated
 +Oct 16, 2008 1:24:30 PM edu.byu.nlp.lm.UniformLocalModelLearner trainModel
 +INFO: Vocab size = 29525
 +Oct 16, 2008 1:24:41 PM edu.byu.nlp.lm.SimpleInterpolatedLocalModelLearner trainWeights
 +INFO: Weights: [0.26495661163809897,​ 0.45432630639697136,​ 0.23072979056597753,​ 0.049987291398952055]
 +done! (22.193 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:24:42 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 2
 +Oct 16, 2008 1:24:45 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Found 318006 contexts in supplied dataset
 +Oct 16, 2008 1:24:50 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 43 conditional distributions
 +...done! (8.143 sec(s))
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (910.0 ms)
 +WSJ Perplexity: 339.47082401790726
 +Running evaluation WSJ Perplexity(l)...done! (835.0 ms)
 +WSJ Perplexity(l):​ 265.32957725283325
 +Running evaluation HUB Perplexity...done! (23.0 ms)
 +HUB Perplexity: 409.1398553133983
 +Running evaluation HUB Perplexity(l)...done! (22.0 ms)
 +HUB Perplexity(l):​ 280.5291204438688
 +Running evaluation HUB Word Error Rate...done! (449.0 ms)
 +HUB Word Error Rate: 0.07303712720632989 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelGT
 +-------------------------------------------
 +Training...Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner linearRegression
 +INFO: Linear regression results: a = 12.315820888402719;​ b = -2.1787021951491146
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner findCutoff
 +INFO: Order = 1, K = 15
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Vocab size: 31234
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Seen: 347018.0
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Possible: 9.75625225E8
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Unseen: 9.75278207E8
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Reserved: 250051.0
 +Oct 16, 2008 1:24:56 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Original Number of Tokens = 1057402.0
 +Oct 16, 2008 1:24:57 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Recomputed Number of Tokens = 1057337.718529062
 +done! (4.641 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:24:57 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 1
 +Oct 16, 2008 1:25:00 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Found 29524 contexts in supplied dataset
 +Oct 16, 2008 1:25:05 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 155 conditional distributions
 +...done! (8.03 sec(s))
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (359.0 ms)
 +WSJ Perplexity: 752.7759977294177
 +Running evaluation WSJ Perplexity(l)...done! (339.0 ms)
 +WSJ Perplexity(l):​ 568.8842322424308
 +Running evaluation HUB Perplexity...done! (9.0 ms)
 +HUB Perplexity: 919.881320154843
 +Running evaluation HUB Perplexity(l)...done! (8.0 ms)
 +HUB Perplexity(l):​ 599.4581890668904
 +Running evaluation HUB Word Error Rate...done! (258.0 ms)
 +HUB Word Error Rate: 0.08216676810712112 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Model MarkovModelIGT
 +-------------------------------------------
 +Training...Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.BasicLocalModelLearner countNGrams
 +WARNING: This dataset does not consist of n-grams of the correct order (1). N-grams were truncated
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner linearRegression
 +INFO: Linear regression results: a = 10.470732070166859;​ b = -1.8063365934456233
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner findCutoff
 +INFO: Order = 0, K = 10
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Vocab size: 29524
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Seen: 29524.0
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Possible: 29525.0
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Unseen: 1.0
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Reserved: 11274.0
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Original Number of Tokens = 940525.0
 +Oct 16, 2008 1:25:08 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Recomputed Number of Tokens = 941068.3665680913
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.BasicLocalModelLearner countNGrams
 +WARNING: This dataset does not consist of n-grams of the correct order (2). N-grams were truncated
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner linearRegression
 +INFO: Linear regression results: a = 12.205311922308134;​ b = -2.182814770109686
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner findCutoff
 +INFO: Order = 1, K = 17
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Vocab size: 29524
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Seen: 318006.0
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Possible: 8.71725625E8
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Unseen: 8.71407619E8
 +Oct 16, 2008 1:25:10 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Reserved: 230135.0
 +Oct 16, 2008 1:25:11 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Original Number of Tokens = 940525.0
 +Oct 16, 2008 1:25:11 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Recomputed Number of Tokens = 940459.6597291145
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner linearRegression
 +INFO: Linear regression results: a = 12.479476842991254;​ b = -2.452644669255306
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner findCutoff
 +INFO: Order = 2, K = 16
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Vocab size: 29524
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Seen: 624808.0
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Possible: 2.5736827382025E13
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Num Unseen: 2.5736826757217E13
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner getNumUnseenNGramTypes
 +INFO: Reserved: 546648.0
 +Oct 16, 2008 1:25:15 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Original Number of Tokens = 940525.0
 +Oct 16, 2008 1:25:16 PM edu.byu.nlp.lm.GoodTuringLocalModelLearner discount
 +INFO: Recomputed Number of Tokens = 940864.761220622
 +Oct 16, 2008 1:25:27 PM edu.byu.nlp.lm.SimpleInterpolatedLocalModelLearner trainWeights
 +INFO: Weights: [0.311088156376188,​ 0.46694109260529026,​ 0.22197075101852173]
 +done! (21.376 sec(s))
 +Running evaluation Distribution ValidatorOct 16, 2008 1:25:27 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Order of Markov model: 2
 +Oct 16, 2008 1:25:30 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Found 318006 contexts in supplied dataset
 +Oct 16, 2008 1:25:39 PM edu.byu.nlp.lm.metric.MarkovModelValidator compute
 +INFO: Checked 19 conditional distributions
 +...done! (12.088 sec(s))
 +Distribution Validator: true
 +Running evaluation WSJ Perplexity...done! (752.0 ms)
 +WSJ Perplexity: 289.26998491023176
 +Running evaluation WSJ Perplexity(l)...done! (748.0 ms)
 +WSJ Perplexity(l):​ 227.62785280078586
 +Running evaluation HUB Perplexity...done! (20.0 ms)
 +HUB Perplexity: 381.0871323568731
 +Running evaluation HUB Perplexity(l)...done! (20.0 ms)
 +HUB Perplexity(l):​ 262.46180739180454
 +Running evaluation HUB Word Error Rate...done! (435.0 ms)
 +HUB Word Error Rate: 0.0736457699330493 (best possible = 0.0, worst possible = 0.21059038344491784,​ avg = 0.11945626668646668
 +
 +
 +Total Time: 2.1029333333333335 min(s)
 +</​pre>​
 +
 +===parsertester.xml===
 +<pre>
 +Rev: 203 
 +Date: 2008-09-22 18:39:27 -0600 (Mon, 22 Sep 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 203 2008-09-23 00:39:27Z rah67 
 +Seed: 1222201824837
 +
 +Model Baseline
 +-------------------------------------------
 +
 +........
 +
 +Gold:
 +(ROOT
 +  (S
 +    (NP (DT Both) (NNS companies))
 +    (VP (VBD rejected)
 +      (NP (DT the) (NNS offers)))
 +    (. .)))
 +
 +Precision, Recall, F-Score:
 + ​[Average] ​ P: 18.06 R: 19.82 F1: 18.9 EX: 1.36
 + over 292 trees
 + </​pre>​
 +
 +Perplexity:
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1224181329522
 +
 +Model Baseline
 +-------------------------------------------
 +Training...done! (46.538 sec(s))
 +Running evaluation Precision, Recall, F-ScoreGuess:​
 +(ROOT
 +  (VP (DT The)
 +    (VP (RBS most)
 +................
 +
 +Guess:
 +(ROOT
 +  (NP (PDT Both)
 +    (NP (NNS companies)
 +      (NP (VBN rejected)
 +        (NP (DT the)
 +          (NP (VBZ offers) (. .)))))))
 +
 +Gold:
 +(ROOT
 +  (S
 +    (NP (DT Both) (NNS companies))
 +    (VP (VBD rejected)
 +      (NP (DT the) (NNS offers)))
 +    (. .)))
 +
 +...done! (453.0 ms)
 +Precision, Recall, F-Score:
 + ​[Average] ​ P: 18.05 R: 19.82 F1: 18.9 EX: 1.36
 + over 292 trees
 +
 +
 +Total Time: 49.678 sec(s)
 +</​pre>​
 +
 +===pnptester.xml===
 +Perplexity:
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1224181992355
 +
 +Model MFL
 +-------------------------------------------
 +Training...done! (866.0 ms)
 +Running evaluation Accuracy over training set...done! (30.0 ms)
 +Accuracy over training set: 0.29817627732012764
 +Running evaluation Accuracy over validation set...done! (2.0 ms)
 +Accuracy over validation set: 0.2925714285714286
 +Running evaluation Accuracy over test set...done! (1.0 ms)
 +Accuracy over test set: 0.2887619047619048
 +
 +Model CCMM
 +-------------------------------------------
 +Training...done! (5.508 sec(s))
 +Running evaluation Accuracy over training set...done! (7.079 sec(s))
 +Accuracy over training set: 0.9302414170753773
 +Running evaluation Accuracy over validation set...done! (890.0 ms)
 +Accuracy over validation set: 0.8487619047619047
 +Running evaluation Accuracy over test set...done! (826.0 ms)
 +Accuracy over test set: 0.8529523809523809
 +
 +
 +Total Time: 15.338 sec(s)
 +</​pre>​
 +
 +===postester.xml===
 +Perplexity:
 +<pre>
 +Rev: 209 
 +Date: 2008-09-29 16:28:46 -0600 (Mon, 29 Sep 2008) 
 +Author: gb07 
 +Id: ExperimentHarness.java 209 2008-09-29 22:28:46Z gb07 
 +Seed: 1222786044570
 +
 +Model GreedyPOSTagger
 +-------------------------------------------
 +Training...done! (19.113 sec(s))
 +Running evaluation Tag Accuracy...done! (795.0 ms)
 +Tag Accuracy: 0.9275560831583113 (Unknown Accuracy: 0.40585774058577406),​ Sentence Accuracy: 0.2161961367013373 Decoder Suboptimalities Detected: 46
 +
 +
 +Total Time: 22.078 sec(s)
 +</​pre>​
 +
 +===PPOSTagger.xml===
 +Perplexity:
 +-Xmx2000m -Xms2000m
 +-DMODEL=Local
 +-DPERCENT_FOR_TRAINING=5
 +-DDATASET=PTB
 +-DNUM_ITERATIONS=2
 +-DAVERAGING=true
 +-DTRAIN_BEAM_WIDTH=5
 +-DUSE_CUTOFFS=false
 +-DNUM_SKIP_OUTPUT=-1
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1221594329303
 +
 +Model LocalPTBPercPOSTagger
 +-------------------------------------------
 +Training...Starting training iterations.....
 +Iteration Number Time (min) Tag Accuracy Unknown Accuracy Sentence Accuracy Suboptimalities
 +
 +0 0.24083 0.00280 0.00077 0.00000 0
 +1 0.22080 0.88205 0.81491 0.11144 1
 +2 0.22182 0.90686 0.82231 0.16865 1
 +done! (46.741 sec(s))
 +Running evaluation Tag Accuracy...done! (11.668 sec(s))
 +Tag Accuracy: 0.9068578212035431 (Unknown Accuracy: 0.8223129946387542),​ Sentence Accuracy: 0.1686478454680535 Decoder Suboptimalities Detected: 1
 +
 +
 +Total Time: 58.487 sec(s)
 +</​pre>​
 +Perplexity:
 +-Xmx2000m -Xms2000m
 +-DMODEL=Global
 +-DPERCENT_FOR_TRAINING=5
 +-DDATASET=PTB
 +-DNUM_ITERATIONS=2
 +-DAVERAGING=true
 +-DTRAIN_BEAM_WIDTH=5
 +-DUSE_CUTOFFS=false
 +-DNUM_SKIP_OUTPUT=-1
 +<pre>
 +Rev: 234 
 +Date: 2008-10-14 05:36:29 -0600 (Tue, 14 Oct 2008) 
 +Author: rah67 
 +Id: ExperimentHarness.java 234 2008-10-14 11:36:29Z rah67 
 +Seed: 1221594329303
 +
 +Model GlobalPTBPercPOSTagger
 +-------------------------------------------
 +Training...Starting training iterations.....
 +Iteration Number Time (min) Tag Accuracy Unknown Accuracy Sentence Accuracy Suboptimalities
 +
 +0 0.24447 0.00280 0.00077 0.00000 0
 +1 0.45665 0.91453 0.81899 0.17756 4
 +2 0.45337 0.92728 0.82716 0.22288 6
 +done! (1.2549333333333332 min(s))
 +Running evaluation Tag Accuracy...done! (11.753 sec(s))
 +Tag Accuracy: 0.9272821355736158 (Unknown Accuracy: 0.8271636456471789),​ Sentence Accuracy: 0.22288261515601784 Decoder Suboptimalities Detected: 6
 +
 +
 +Total Time: 1.4521666666666666 min(s)
 +</​pre>​
  
nlp-private/stat-nlp-library.txt ยท Last modified: 2015/04/22 21:07 by ryancha
Back to top
CC Attribution-Share Alike 4.0 International
chimeric.de = chi`s home Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0