CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
HASH-O-RAMA Data Processing Problem

 

First page Previous page 1 2 3 4 5 Next page Last page  View All


Zhris
Enthusiast

Apr 12, 2015, 6:59 AM

Post #76 of 102 (9726 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

FYI, with regards to updating wtfs. it might be better to have a wtf_adjusted column in the weights log, therefore you have a record of both the automatic and manual wtfs. Inevitably you have to be careful when re running phases, as if you made adjustments to the weights log then reran phase1, it would overwrite, this is why you should always use --interactive mode and carefully read the messages as to whether particular files are empty or not. Alternatively adjust the filenames in the configuration so that the inputs and outputs named differently and you'll have to do some copying between phases.

Chris


stuckinarut
User

Apr 12, 2015, 7:31 AM

Post #77 of 102 (9724 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post


In Reply To
FYI, with regards to updating wtfs. it might be better to have a wtf_adjusted column in the weights log, therefore you have a record of both the automatic and manual wtfs. Inevitably you have to be careful when re running phases, as if you made adjustments to the weights log then reran phase1, it would overwrite, this is why you should always use --interactive mode and carefully read the messages as to whether particular files are empty or not. Alternatively adjust the filenames in the configuration so that the inputs and outputs named differently and you'll have to do some copying between phases.

Chris


Hmmm... more to chew on.

FYI, I kept wondering about the number '27' that showed up in one QSO entry in the NAME column. Went back to the original log that was emailed (and subsequently 'fixed' and re-fixed' to remove unwanted QSO numbers). In my rushed delete/copy/paste work, I found the problem. Just re-uploaded another corrected entries.txt file:

http://www.xgenesis.com/hashorama/entries.txt

To get a better perspective of things and run more tests, the one thing that definitely would help right now is to at least have a NOLOG Column added to the Error log - to the right of the WTF Column, so that *any* Error where the CALLWKD callsign is a Non-Log Submitter is flagged NOLOG. This will also help me to refine the anticipated manual decision/processing to be done here!

I will re-visit the unsubmitted log stuff later after running more tests. I can just setup a new sub-directory here and use the new entries.txt file, and substitute a new main.pl file.

Thanks much!

- Stuckinarut


(This post was edited by stuckinarut on Apr 12, 2015, 7:32 AM)


stuckinarut
User

Apr 12, 2015, 11:39 AM

Post #78 of 102 (9716 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Chris:

Amazing things happen when one gets a bit more sleep.

REGARDING my previous posting:


In Reply To
To get a better perspective of things and run more tests, the one thing that definitely would help right now is to at least have a NOLOG Column added to the Error log - to the right of the WTF Column, so that *any* Error where the CALLWKD callsign is a Non-Log Submitter is flagged NOLOG. This will also help me to refine the anticipated manual decision/processing to be done here!


Due to the issues and weights involving Busted calls/Non-Submitted Logs, here is what makes MUCH MORE SENSE and will be much cleaner:


Code
Instead of NOLOGS, the column should be LOGS and if the CALLWKD Callsign on any line of the Error Log is for any LOGCALL (i.e., a log was submitted), then what gets entered in that column is "LOG".


This will be a quick visual aid in my MERP (Manual Error Research Process).

Thanks !!!

- Stuckinrut


stuckinarut
User

Apr 12, 2015, 12:15 PM

Post #79 of 102 (9709 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

A bit of explanation about what I'm now calling MERP.


In Reply To
This will be a quick visual aid in my MERP (Manual Error Research Process).


By having the new LOT column 'flagged', I can group my manual research tasks.

1. If a (submitted) LOG related Error, I will do those as a group and can quickly go through the actual submitted logs based on the sorting order (which was a reason for it :^)

2. If NOT flagged as a (submitted LOG related Error), I will do those as another group. For this group which includes possible email verification steps, I will use:

A. The CNQ weights.txt list which shows multiple CNQ combination errors for the same Callsign

B. An online HAM specific database with Callsign (owner & QTH) info. When a nickname is used, only the QTH info would be valid but a help.

C. For all USA callsigns (the majority in the logs), there is also the FCC Universal Licensing System online (Government) database.

99.9% of the time, if a callsign does NOT show in #B above, it's a plain old bad/busted callsign. Example: As of this posting, when I type in N4AFY, here is what happens:


Code
The search for "N4AFY" produced no results.


"BINGO!!!"

Likely the person's fingers slipped on the keys and added a 'Y' to N4AF's callsign. Of course, there are CNQ combo multiple entries in the weights.txt file for N4AF which is another issue to deal with {SIGH}.

I hope this helps better understand how I plan to use the Error log details schema in my 'MERP'.

FYI,

- Stuckinarut


Zhris
Enthusiast

Apr 12, 2015, 12:35 PM

Post #80 of 102 (9706 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

As per your requests here is another update.

Notes:

- Added LOG column to errors log but I thought I would make it a bit more intuative. Instead of just outputting LOG where the call is valid and has an associated submitted log I have gone with 3 statuses. Let me know if this doesn't satisfy you.
----- '1' = callsign is valid and its associated log was submitted.
----- '0' = callsign is valid but its associated log was not submitted.
----- '-' = callsign is invalid therefore no log available.

- Updated entries.txt

- Included names and qths in unsubmitted log as this may be helpful.

Chris
Attachments: contestcrosschecker.zip (26.8 KB)


stuckinarut
User

Apr 12, 2015, 8:56 PM

Post #81 of 102 (9686 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hello, Chris:

I had it with the tax work for today/tonight, so just installed and took a 'Test Drive' of the latest version.


In Reply To
----- '1' = callsign is valid and its associated log was submitted.
----- '0' = callsign is valid but its associated log was not submitted.
----- '-' = callsign is invalid therefore no log available.


Interesting possibility - will let you know after more use.


In Reply To
- Included names and qths in unsubmitted log as this may be helpful.


Using K2QBN as an example, it shows 'EVAN' as the name, but I remember there was also a CNQ with 'VAN' in the Weights file. Not sure yet how this will be used compared to the Error log which I believe is the Mega Important tool...along with the weights.txt file list which shows me all the CNQ combinations. Yes, the latter is also Mega Important in my tedious MERP tasks.

But let me tell you what is really E-X-C-I-T-I-N-G ... and I almost soiled myself when I realized how COOL (and a productive time-saver). Can you guess?

By putting the QID numbers next to each line in the Error log, I no longer have to go searching through multiple log entries (even in band & time sequence) by LOGCALL. I can now go DIRECTLY to the specific Error log related QSO line I need to scrutinize. Talk about EFFICIENCY !!! Chris, this is BEYOND AWESOME !!!

I don't even have to type the Error log QID... just copy it and with the entries.txt file open, quickly do a CTRL + F followed by a CTRL + V and then pop the ENTER button. "BINGO!!! ... almost like Magic I am whisked right to what I need.

I can't tell you for years how many extra hours were spent flipping through an unwieldly stack of printed-out log sheets in the Nightmare log-checking process. Even later using individual .txt files on the Confuzzzer was a real pain having to close and open separate files. This is going to save a LOT of time. Dunno why I didn't think to consolidate all the logs into one 'Master' log way back when (DUH!).

Thanks sooooooo much for adding the QID's to both the Error Log and the Phase 0 assignment of QID's to each of the QSO lines in the entries.txt file !!!

- Stuckinarut


Zhris
Enthusiast

Apr 12, 2015, 11:18 PM

Post #82 of 102 (9675 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

No problem at all.

I have actually changed the LOG codes to suitable labels instead:
- 1 is now submitted
- 0 is now unsubmitted
- - is now invalid


Quote
Using K2QBN as an example, it shows 'EVAN' as the name, but I remember there was also a CNQ with 'VAN' in the Weights file.


Remember at phase2, after the weights log was generated at phase1, only names and qths that are >= wtf threshold will be included ( the same information as the scores log ). VAN has a wtf of just 1.

phase2 being the core automation phase ignores anything it deciphered to be invalid, afterall thats automation for you. You should always keep this in mind, the logs generated by phase2 are automated outcomes, don't expect anything the algorithm deciphered to be "useless" to be included.

The log that should be most useful to you is therefore the weights log, since this allows you to fine tune the outcome of phase2. Perhaps we do need to consider extending info in the weights log and / or breaking it up into groups and / or generating other consolidation logs at phase1, enabling you to make the right decisions before phase2. We have already discussed this in some detail, but I don't think we really finalised any ideas.

Also the QID's do indeed make lookups super quick, I used to look at an error call sign, then forget it by the time I opened up the entries, or atleast confuse it with one similar.

One other thing I was thinking was in the scores log under the scores column, it might be nice to do "score/max score", where max score would be the score they would have gotten if they made no errors whatsoever, then perhaps a percentage column i.e. ( ( 100 / max_score ) * score ). I'm not sure if this would be useful to you, but a lower percentage would be a good indication of who is the most error prone, while those with 100% percentage deserve a reward, perhaps a billion bonus points ;-). But of course those with a score of 0/0 would get 100%, and those who didn't submit many entries might be notably less error prone.

Regards,

Chris


(This post was edited by Zhris on Apr 12, 2015, 11:41 PM)


stuckinarut
User

Apr 13, 2015, 12:03 AM

Post #83 of 102 (9656 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post


In Reply To
Also the QID's do indeed make lookups super quick, I used to look at an error call sign, then forget it by the time I opened up the entries, or atleast confuse it with one similar.


Yes, 'been there, done that - still doing that', but in my excitement, it was not until I took a needed R 'n R break that I realized the actual QSO line from the *other* (CALLWKD) station is also what needs checking. And at this point of 'Beta Testing', having the ability to do a rapid copy & paste into the FIND dialog box on the entries.txt page to zip right to the CALLWKD station QSO details would be great. Especially, for investigating the 'NIL' Errors.

So another idea came to mind: QID2 (or 'Son of QID' :^) If we added one more CWID (Call Worked ID) column to the far right of the Error log, this could accomplish the mission like with the CID. Of course, a CWID number would only display IF the Error line involved a CALLWKD by another log submitter. Yes, this would be Awesome(2) and save even more time.

FYI, when the logs are originally submitted, there is a "Claimed Score". If you look at that PHP Form URL again I PM'd you, I had already planned to use the this data as part of (manually) doing more tests... to make sure nothing fell through the cracks. Of course, the submitter's math skills (or lack thereof) have sometimes not been accurate. Not to be 'the pot calling the kettle black' since I've made more than my share lately by rushing too quickly.

I had actually considered including the 'Claimed Score' as another column in the categories.txt file. Ohhhh... now I'm getting another idea for something very useful, but will chew on it a bit more.

Based partly on the K2QBN example of CNQ weights of 3 and 1, I'm pretty certain already that a WTF of 3 would be the minimum. I also want to run tests at 4 and 5. What is also coming out of this is likely to be a tweak to next year's Rules.

The VE3|ON, VE4|MB and XE|DX situation is something I am still mulling over in consideration of the objectives of helping others also increase their on-air *and* logging accuracy vs. blanketly Auto-Mulliganizing any QSOs. This is a tricky one.

- Stuckinarut


stuckinarut
User

Apr 13, 2015, 12:23 AM

Post #84 of 102 (9650 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post


In Reply To
I have actually changed the LOG codes to suitable labels instead:
- 1 is now submitted
- 0 is now unsubmitted
- - is now invalid


This is still baffling me a bit. I understand 1 and 0, but for example:


Quote
427 K6NV VE3KI 80M 0242 RICH ON CNQ 1<3 -


VE3KI did not submit a log, but a determination cannot be made as to whether the QSO is actually 'invalid' or not until investigated.

Checking the primary online HAM database, VE3KI is 'Richard' in ON(tario). Most likely he uses 'Rich' on-the air. This type of QSO may have to be treated as a 'Unique' vs. an invalid CNQ, and I'm continuing to chew on these situations in terms of disposition.

Hmmm... it might be more helpful to only flag the '1' lines as LOG (much clearer than trying to remember pseudo-codes).

Looking at this one flagged '0' ...


Quote
385 K6DGW W1NN 40M 0213 HAL SC CNQ 1<3 0


The MERP will be the same as for those currently flagged '1' as compared with a rapid QID/CWID review of actual submitted QSO lines and both these could be left blank. The only entries which would show in the CWID column would be LOG ('1' status), and a corresponding CWID column entry of the ... hmmm... it would have to be the actual QID of the CALLWKD station now that I think about it, to be able to go directly there to check, right?

Shutting down for the night now at 0030 local.

Thanks!

- Stuckinarut


(This post was edited by stuckinarut on Apr 13, 2015, 12:25 AM)


Zhris
Enthusiast

Apr 13, 2015, 1:21 AM

Post #85 of 102 (9643 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

FYI, I chucked in 3 new columns in the scores log just for my own interest, SCOREMAX and ACCURACY as per my last post, and BONUSES is a count of the number of bonus stations worked. its interesting to see the accuracy trend. I will remove if no use to you, but you might be interested with the result for now.


Quote
So another idea came to mind: QID2 (or 'Son of QID' :^) If we added one more CWID (Call Worked ID) column to the far right of the Error log, this could accomplish the mission like with the CID. Of course, a CWID number would only display IF the Error line involved a CALLWKD by another log submitter. Yes, this would be Awesome(2) and save even more time.


I think I understand what you mean, currently the qid represents which line the error triggered on. A cwid would be the qid of the call worked. If so, this wouldn't work how you think, since a call might have multiple qso entries for the sign, name and qth combination, therefore a single qid couldn't be derived.


Quote
This is still baffling me a bit. I understand 1 and 0, but for example:

Quote
427 K6NV VE3KI 80M 0242 RICH ON CNQ 1<3 -

VE3KI did not submit a log, but a determination cannot be made as to whether the QSO is actually 'invalid' or not until investigated.


phase2, the automation phase, decided it was invalid. If you had adjusted its weight before phase2 appropriately, then the outcome could have been different. OR, you could have run phase2, analysed the automated outcome, adjusted its wtf, then re-run phase2. Could you confirm that this makes sense, its important you understand the purposes of phase1 and phase2. This brings me back to the point of perhaps constructing more detailed logs at phase1 if the weights log doesn't provide what you need as of yet. Fundamentally though, you should have used the weights log to decipher that "VE3KI RICH ON" is valid, you can't rely on the outcome of phase2 until the phase1 weights log is acceptable, or appropriate adjustments in the adjustments log have been made.

Finally, I did mention I changed 1, 0 and - to appropriate labels of submitted, unsubmitted and invalid, but have replaced with LOG for 1 / submitted only and other "statuses" are left blank.

Code is attached.

Regards,

Chris
Attachments: contestcrosschecker.zip (27.1 KB)


stuckinarut
User

Apr 13, 2015, 9:39 AM

Post #86 of 102 (9594 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Chris:

Getting down to the tax deadline so will have to get back with you later or tomorrow regarding some of the items in your last posting.

I downloaded the latest version and did a real quick updated comparison of the scores. VERY interesting with the other data columns you added. Note that I put the word BEFORE in Bold Red Font on the top line :^)

http://www.xgenesis.com/hashorama/compare_lqpck5_13April2015.pdf

At a quick glance, most scores appear to now be in-the-zone, and differences are mostly the result between 'Mulligans' that were given for minor errors in 2014, and just spot-checking logs that were not contenders for any of the category awards. FYI, Cat #11 is 'Checklog' only, and along with #12 & #13 not eligible for any awards.

Even without any Adjustments being possibly made from scrutiny of the Error log, FORTUNATELY even at a first pass, the individual category winners remain the same {Major SIGH Of Relief}. The sequences are a bit out of order in some cases, but for quick-glance purposes, I thought you would find this update very interesting.

BTW, I discovered one more (non-critical) left-over item in the entries.txt file ... an extraneous "CA" to the right of QSO #1257 after WQ5L RAY MS ... now deleted in my file here.

Umh, 'lqpck5' is a local sub-directory for the latest version :^)

- Stuckinarut


(This post was edited by stuckinarut on Apr 13, 2015, 9:41 AM)


stuckinarut
User

Apr 13, 2015, 9:51 AM

Post #87 of 102 (9591 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Ohhh... I did just notice something odd with this one you might want to check out:


Code
2	N8XX	11000	12000	92%	0	JACK	MI


In my 2014 RESULTS, I showed N8XX at 13,000 points - his log had 13 QSOs (x 1000 = 13,000) "Claimed", which *should* have been the MAXimum possible score before any Errors in the scores.txt file list. Not sure why only 12,000 shows as the MAX, unless it has to do with him using the name 'IGOR' for the 1st QSO. IMHO, that should not make any difference in determining MAXimum possible.

FYI,

- Stuckinarut


(This post was edited by stuckinarut on Apr 13, 2015, 10:02 AM)


Zhris
Enthusiast

Apr 13, 2015, 8:00 PM

Post #88 of 102 (9477 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

Thanks for posting the comparison, its looking better than a few comparisons ago.

I couldn't locate the lqpck5 subdirectory, could you post the full link.


Quote
In my 2014 RESULTS, I showed N8XX at 13,000 points - his log had 13 QSOs (x 1000 = 13,000) "Claimed", which *should* have been the MAXimum possible score before any Errors in the scores.txt file list. Not sure why only 12,000 shows as the MAX, unless it has to do with him using the name 'IGOR' for the 1st QSO. IMHO, that should not make any difference in determining MAXimum possible.


You should have increased the wtf in the weights log for igor to above the wtf threshold. Ok lets simply this, lets remove one of the more extreme checks (which this issue tripped up on) and assume all the log side of things as oppose to call are somewhat valid and not ignore logs below the wtf threshold. Inevitably you can still control the validity on a per error basis via the adjustmenets log.

Delete or comment out line 522, which should be:


Code
next if $log_wtf < $wtf_threshold;


This may fix numerous issues where previously you hadn't used the weights log to its full potential. Note igor will now appear in the list of names in the unsubmitted and scores logs (this can be changed if need be), but any calls to this name will continue to CNQ error because its < wtf threshold.

Chris


(This post was edited by Zhris on Apr 13, 2015, 8:22 PM)


stuckinarut
User

Apr 13, 2015, 9:06 PM

Post #89 of 102 (9458 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi, Chris:
'
The lqpck5 sub-directory in on my local machine & not accessible - I create a separate install location for each update :^)

Made the change and yes, the scores.txt file updated but not the unsubmitted.txt file. Re-ran n=1 and n=2 twice but no dice.

Still chewing on another idea here that needs to 'ferment' a bit :^)

FYI,

- Stuckinarut


Zhris
Enthusiast

Apr 13, 2015, 9:12 PM

Post #90 of 102 (9455 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

Ah I thought you meant you uploaded it to your site.

What difference were you expecting in the unsubmitted log, if with regards to n8xx, they submitted a log therefore will not appear there.

Chris


stuckinarut
User

Apr 13, 2015, 10:04 PM

Post #91 of 102 (9444 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post


In Reply To
Ah I thought you meant you uploaded it to your site.


Sorry... I should have clarified this before ;-(


In Reply To
What difference were you expecting in the unsubmitted log, if with regards to n8xx, they submitted a log therefore will not appear there.


Not sure exactly what you mean here. Was just getting ready to post something for you, so will have to re-read yours once or twice and go look at the files :^)

Here's what I've been chewing on:


Code
http://www.xgenesis.com/hashorama/QWID_Possibilities.pdf


Also mulling over exactly how I am going to treat some of the errors when I shift into 'Adjustments' mode {HUGE SIGH HERE}.

Will be offline for a while but check back in later.

FYI,

- Stuckinarut


stuckinarut
User

Apr 13, 2015, 10:16 PM

Post #92 of 102 (9442 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

One more tidbit... I pulled this info from an LCR (Log Checking Report) from a Contest I did operating from the U.S. Virgin Islands in 2004. I was young then (60) and operated 44 out of the total 48 hour contest weekend.

After being off the air for about 30 years, I only had about 2 months to get my Morse Code speed back up to 35 to 38 Word Per Minute to compete in this 'International' event. Ended up in the Top 10 :^) No way at 71 now would I attempt to do another 44 hour thing - I started hallucinating at about 38 hours and that was 11 years ago.

With 4603 raw (Gross) QSOs, and copying exchanges faster than I could type them, I wasn't sure how things would end up.

This will give you an idea of how one of the 'Major' HAM Contest sponsors deals with log-checking. Not sure the columns will align properly.


Code
SCORE SUMMARY     160   80    40    20    15    10       All 
------------- ---- ---- ---- ---- ---- ---- ----
Raw QSOs = 476 437 835 661 1285 909 = 4603
Dupe QSOs = 4 2 5 6 6 7 = 30
Busted QSOs = 3 8 8 12 11 11 = 53
Valid QSOs = 469 427 822 643 1268 891 = 4520
Penalty QSOs = 3 8 8 12 11 11 = 53
Final QSOs = 466 419 814 631 1257 880 = 4467
Multipliers = 52 57 58 53 58 55 = 333

Total QSO Points = 13401
Final score = 4462533
Error rate = 1.2% (100 X (Busted QSOs / Duped QSO total))

There were 30 dupes found. You might have marked these dupes. For
electronic logs, all dupes are just removed with no penalties.

Total incorrect calls = 35. These will be removed from your log
with an additional penalty of 35 QSOs.

There were 18 bad cross check QSOs removed.
You had 18 NILs. A penalty of 18 QSOs will be assessed.

You had 78 unique calls. These QSOs are not removed.


I freaked out when I saw the 1.2% Error Rate, until some of my fellow Contest Club members said that was actually very good (3% to 5% Errors was apparently fairly common).

The proverbial $64,000 question now is how strict (or lenient) I am going to be with my own event log-checking which was pretty 'hang-loose' before :^)

FYI,

- Stuckinarut


Zhris
Enthusiast

Apr 13, 2015, 11:22 PM

Post #93 of 102 (9429 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,


Quote
Made the change and yes, the scores.txt file updated but not the unsubmitted.txt file. Re-ran n=1 and n=2 twice but no dice.



In Reply To
What difference were you expecting in the unsubmitted log, if with regards to n8xx, they submitted a log therefore will not appear there.

Not sure exactly what you mean here. Was just getting ready to post something for you, so will have to re-read yours once or twice and go look at the files :^)


Assumably you thought the unsubmitted log should have changed when you deleted the line of code. I was wondering what change you were expecting. I didn't expect it to change since this code deletion is seperate to the algorithm that determines unsubmitters.

I see that you are aware the qwid cannot be straight forwardly looked up for nil and qth errors. The time thing might work out, you could have a configurable range and then provide the range of qids i.e. cwid = 1001-1007, atleast giving you a rough position. You mentioned both sides of an error transaction should show up in the error log, I can't be certain of exactly what you mean, but remember if someone didn't return the call then that person won't have an error marked since they didn't log it but as a result won't get any points for it anyway.

Cool to see how the results were laid out for a contest you took part in. I like the summary sheet they provided you with. We could create a directory called summaries, and create a summary file for each log sign, with the same sort of layout but could extend it to display their entries from the entries log and errors from the errors log, maybe even their weights too. This is probably of low priority and something we can think about later.

I've gradually become more concerned that parts of what I've implemented aren't exactly what you thought I implemented (mostly with regards to weighting). Inevitably as and when I wrote sections of the code, I used my own initiative to fill in the gaps, I've tried to explain some of these, but ideally you would be able to read back and understand the code. Basically there are quite a few issues that you raise that from my point of view are working as expected. We could go back to using a fresh entries.txt file which has a much smaller sample of data that reflects every possible scenario, with a manually written expected output, making sure we are both on the same track. Maybe I am in actual fact over thinking this and that you fully understand and are perfectly happy with the way its going so far. Its been difficult for both of us to account for all the discrepencies along the way, this is not a straight forward task and its been good practice!

Regards,

Chris


(This post was edited by Zhris on Apr 13, 2015, 11:25 PM)


stuckinarut
User

Apr 16, 2015, 6:12 PM

Post #94 of 102 (9135 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi, Chris:

Sorry...was intensely focused on making the tax deadline yesterday. Mission accomplished, but it left me drained. I 'crashed and burned' last night ;-(

Yes, this is not a run-of-the-mill project/task scenario, and I greatly appreciate you hanging in there with me !!!

Today I started to take a new 'Test Drive' with this year's 2015 data, but ran into a few challenges that necessitate re-downloading and re-formatting some of the Callsign/Category data (the QSO line stuff appears to be OK).

Unfortunately, I need to get all packed up to leave in the morning for an Annual Convention so will be offline until Monday. I plan to use the (normally boring highway) 3 hour drive each way to periodically 'chew' on final decision options as to strictness or leniency in dealing with errors in the script processing. I'll have my weenie digital recorder along to take notes. Hmmm, this will likely be on the way back, as I think I need 3 hours of non-stop music on the way 1st leg of the trip to further de-compress :^)

I'm actually looking forward to several days of No Confuzzzzer, No email, No Robo-Telemarking phone calls, etc. Hopefully I will return with an emptied personal mind-cache to meet the remaining 'challenges' in this matter with a fresh perspective.

Thanks again for all your help !!!

- Stuckinarut


stuckinarut
User

Apr 21, 2015, 9:37 AM

Post #95 of 102 (8584 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi, Chris:

I'll be PM'ing you more specific details hopefully later today, but am uploading a quick "Bird's Eye View" here of what came to mind before leaving for the Convention and subsequent tweaking and re-tweaking done since returning.

Out of the Total 101 errors using WTF=3, this can reduce the "iffy" (questionable) CNQ status count to only 35 !!! With a future increase in overall log submissions and QSO counts, this potential time savings is worth-it's-WEIGHT-in-Gold ('Weight' PUN intentional :^)

The new "Discovery" on how to further reduce the manual PITA of switching between files when investigating and researching errors came about when I switched from using the 1024x768 display main Notebook Confuzzzzer to a seeing things much w-I--d---e----r without having to horizontally scroll on the 23" 1920x1080 flat panel with a recently acquired Desktop for A/V purposes. What a difference...a whole new World!

I've embedded some brief "NOTES" in new consolidated errorlog.csv file approach for quick import into Excel (.csv instead of a Tab/.txt file is necessary for this to work). All the other existing errors.txt, weights.txt, etc. files will still serve a useful purpose.

Oh darn... the .jpg won't upload (exceeds 250KB). Hold on... OK, you can get it here:


Code
www.xgenesis.com/hashorama/errorlog_partial_screencapture_1920px.jpg


(EDIT): I think that should be sorted DESC instead of ASC for the CNQ Weight/CNQ combinations in the Notes (for the CALLWKD LOG CNQ WEIGHTS column/field entries) ;-(

- Stuckinarut


(This post was edited by stuckinarut on Apr 21, 2015, 9:44 AM)


stuckinarut
User

Apr 21, 2015, 9:51 AM

Post #96 of 102 (8578 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

This one-line sample errorlog.csv file entry file is small enough to upload here :^)

NOTE: Importing the .csv into Excel will require a one-time adjustment of the column widths to accommodate the longer data sections, of course.

Ooops... I left out a comma separator for the empty 'LOG' field in this example.


(This post was edited by stuckinarut on Apr 21, 2015, 10:01 AM)
Attachments: errorlog.csv (0.12 KB)


Zhris
Enthusiast

Apr 21, 2015, 5:29 PM

Post #97 of 102 (8528 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi,

I've made the changes you described, but as a result I ideally need to refactor the code then re-test. Basically the new error sort order required an intermediate data structure therefore am considering putting error and score construction in their own "_input" functions.

I will have it ready at some point tomorrow.

Chris


stuckinarut
User

Apr 21, 2015, 6:11 PM

Post #98 of 102 (8518 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Thank you so very much, Chris. I'll be in 'Standby' mode.

- Stuckinarut


Zhris
Enthusiast

Apr 22, 2015, 3:21 PM

Post #99 of 102 (8394 views)
Re: [stuckinarut] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Hi Eric,

Please see PM for latest version and notes. I don't want to post it on main because I didn't have time to fully refactor it just yet.

Chris


stuckinarut
User

Apr 22, 2015, 9:22 PM

Post #100 of 102 (8335 views)
Re: [Zhris] HASH-O-RAMA Data Processing Problem [In reply to] Can't Post

Thanks - PM'd you some 'Beta Test' Feedback details & questions.

- Stuckinarut

First page Previous page 1 2 3 4 5 Next page Last page  View All
 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives