apgsearch v1.0

[Scorbie]

What might be doing this?

Hope they aren't two freaking cosmic ray events... Maybe all the scores of the soups are doubled?

[apg]

I have no idea why this is happening. By the way, the prolonged barrage of Day and Night soups has started incurring comparatively large costs:

[Alexey_Nigin]

I have no idea why this is happening. By the way, the prolonged barrage of Day and Night soups has started incurring comparatively large costs:

Stopped my search.

By the way, yl220_1_12_7a35ff857041aff55081929a21f53f24 in DN has 2 occurrences but no sample soups. Similar things are observed with other objects.

[Alexey_Nigin]

https://yadi.sk/d/PgFXRrYJgbDKY

Updated my package. Now it uses the latest version of apgsearch. Also changed "Nigin's Public Account" to "Nigin Public Account". This reminds me of a stupid story involving Adam.

[apg]

Also changed "Nigin's Public Account" to "Nigin Public Account".

That would just result in a second sector on the pie chart. Anyway, I fixed it by modifying one of the lines of Catagolue code:

Code: Select all

String displayedName = parts[1].replaceAll("'","'");

By the way, yl220_1_12_7a35ff857041aff55081929a21f53f24 in DN has 2 occurrences but no sample soups. Similar things are observed with other objects.

Basically, it didn't have enough time to include the sample soups (there's a 10-minute limit on cron tasks). This should never happen in any sensible rule.

Stopped my search.

Thanks.

We have over 2 * 10^12 objects in b3s23/C1 so far, none of which is the p2 phoenix -- who would have guessed it would be so inexplicably rare? I suppose the 6-bit clock shares the property of being ridiculously rare given its size.

[dvgrn]

Some freak cosmic ray event, perhaps?

It happened again, a pulsar + clock + Block-laying-switch-engine soup worth 40 points clocked in at 80. What might be doing this?

First thing might be to re-run the exact same soup string on a different computer (or the same computer) using a minimally-modified apgsearch 1.04, and see if the same score gets applied again. If it doesn't happen on a re-run, then maybe it really is cosmic rays...

Can you tell if both high scores were reported by the same computer? (That seems really really unlikely, but so does the doubled score.)

[biggiemac]

Can you tell if both high scores were reported by the same computer? (That seems really really unlikely, but so does the doubled score.)

I don't really have a check for which computer is which. Until recently I had a slightly different count between hauls for each computer, but now I'm running a shared file renamed golly-start.py for all instances, giving the same values for all. Running that soup through the calling function in apgsearch in all possible external configurations would be a good check.

Edit: I at least checked the haul with the doubled 80-pointer for other objects found. There were plenty of 20+bit SLs that would count as 40 points if the soup scores were consistently doubled but none of the 40 point soups have the fgpEFnrZ2cHx prefix. It looks as though it doesn't happen every time with one particular computer. It leads me to imagine it only happens during some exceptional event like the error-correcting phase.

Edit 5-14: The pie chart is now a donut chart! With 2 more digits of precision! That's a cool addition. Also, would it be worth the programmer time and 5 CPU seconds to take the top 1000 scoring soups, verify their authenticity, and change the associated scores in the database for any that are inexplicably doubled?

[apg]

More biggiemacs? My university has hundreds and hundreds of computers, but almost all are in regular use except maybe one of the small liberal arts labs. I'll look into that lab this weekend.

Did you ever look into this, just out of interest? If you have the 'switch user' option, as Brett does, then you can log out of the computer whilst still leaving Golly churning away.

I've done some performance tests, and Catagolue should be able to handle a constant rate of 30 times its current load for b3s23/C1, and shouldn't be perturbed by arbitrarily large spikes (since it only attempts to commit 100 hauls at each 15-minute interval).

[Kazyan]

Did you ever look into this, just out of interest? If you have the 'switch user' option, as Brett does, then you can log out of the computer whilst still leaving Golly churning away.

I did; 'switch user' is disabled. I go to an engineering college with a well-developed computer engineering department...I guess they know what they're doing.

[Extrementhusiast]

Did you ever look into this, just out of interest? If you have the 'switch user' option, as Brett does, then you can log out of the computer whilst still leaving Golly churning away.

I did; 'switch user' is disabled. I go to an engineering college with a well-developed computer engineering department...I guess they know what they're doing.

Well, try talking it over with them and suggesting to do this search.

[apg]

I did; 'switch user' is disabled.

You could still try logging into lots of computers and running apgsearch. Even if there's a safeguard which logs one off after a period of inactivity, this is easily circumvented: I have found that opening a foreground instance of Notepad and balancing the mouse to depress the spacebar (thereby typing a continual stream of spaces) is sufficient to fool the software without incurring any performance impact.

Well, try talking it over with them and suggesting to do this search.

I may be cynical, but I imagine they're unlikely to agree. After all, one can do a quick calculation to estimate the cost of powering the machines to run apgsearch, and it's something like 0.50 GBP per 10^9 objects. Considering the fact that we've amassed 2 * 10^12 objects, that already equates to about 1000 GBP.

I think crowdsourcing is a better idea, although I'd prefer to wait until I have everything in a more polished state (comprehensive online documentation, increased security measures, et cetera). Here's my current progress in the latter:

The endpoint http://catagolue.appspot.com/chisquare gives an 'unlikeliness estimate' for each of the most recent hauls (in b3s23/C1). It appears that I can probably impose a safe cap of 2000 without rejecting any actual hauls, but this will almost certainly catch any pretend hauls (even under/overestimating the number of pulsars by 30%, or the number of blocks by 0.4%, would immediately break this limit for a 10-megasoup haul).

Moreover, http://catagolue.appspot.com/chisquare/ ... 821b9e1cdf (for example) will also give you a list of rare* objects (relative to Catagolue, rather than the haul itself) in a particular haul and the soups that purport to create them. This could be verified by other instances of apgsearch (where Catagolue only tells the client which soups to run, and the clients report back all the objects found).

* by which I mean 'not in the 1000 most common objects'. The 1000 most common objects are already handled by the chi-square test.

[dvgrn]

I did; 'switch user' is disabled.

You could still try logging into lots of computers and running apgsearch. Even if there's a safeguard which logs one off after a period of inactivity, this is easily circumvented: I have found that opening a foreground instance of Notepad and balancing the mouse to depress the spacebar (thereby typing a continual stream of spaces) is sufficient to fool the software without incurring any performance impact.

That does tend to draw more attention, though, not to mention getting in the way of other people who might want to use the computers, where the background processes don't. (Unless they also want to run something like apgsearch, anyway.)

Staying in the same room to guard a large number of logged-in computers, for long enough to get decent apgsearch hauls, isn't usually an option. Walking away isn't a good option, either, since in the absence of the 'switch user' option, one's University account is necessarily logged in and vulnerable to whatever use the next visitor wants to make of it. Free printing, prank messages, and deleting all the files on my University share drive, oh my...

[biggiemac]

I may be cynical, but I imagine they're unlikely to agree. After all, one can do a quick calculation to estimate the cost of powering the machines to run apgsearch, and it's something like 0.50 GBP per 10^9 objects. Considering the fact that we've amassed 2 * 10^12 objects, that already equates to about 1000 GBP.

Good thing my senior thesis consisted of building an instrument for physics research that would cost $40K but only cost me $2.5K. That plus $48K/year in tuition (and that's after scholarships are taken into effect - stupid American higher education costs put my college at nearly $60K/year).. they're still making a profit on me.

[Kazyan]

I'll try to get apgsearch running an Amazon EC2 t2.micro instance to avoid having to deal with my university too much. It's a bit tricky, but working with search programs means it's not too opaque to use for me...hopefully...

[biggiemac]

We've reached 10^8 soups. I've reached 80% on the pie chart.

Edit: 10^11. Brain whilst marathoning Korra might not be operating at full capacity.

[apg]

We've reached 10^8 soups. I've reached 80% on the pie chart.

10^11, I hope...? And in that case, thanks for submitting 8 * 10^10 of those!

[gameoflifeboy]

Besides catagolue.appspot.com/chisquare, are there any other subpages not linked to that I should know about?

[apg]

I have no idea why this is happening. By the way, the prolonged barrage of Day and Night soups has started incurring comparatively large costs:

Stopped my search.

I noticed that the main bottleneck was updating the object pages with sample soups. I then optimised it by using the batch version of DatastoreService.put() (so that all of the entities are updated in parallel). This led to concurrency issues, so I compromised (using batches of 5000 objects with one-second sleeps between them). The latest version of the code seems to work perfectly well, at least for my test-case (13 000 000 soups in Day and Night submitted in a single cron job).



So by all means, resume your searches.

Besides catagolue.appspot.com/chisquare, are there any other subpages not linked to that I should know about?

No, I don't believe so. There are several other endpoints, but they're only really used by me for testing purposes.

The pie chart is now a donut chart! With 2 more digits of precision!

It seemed reasonable to increase the precision, given that most of the users each have less than one percent of the total objects committed.

[gameoflifeboy]

It seemed reasonable to increase the precision, given that most of the users each have less than one percent of the total objects committed.

If that's what you're aiming for, a table might be better than a pie chart, because on charts with more than 20 slices (the b3s23/C1 chart, for example), some slices aren't labelled, and coupled with how small they are this makes them very hard to click on.

[Scorbie]

I'll try to get apgsearch running an Amazon EC2 t2.micro instance to avoid having to deal with my university too much. It's a bit tricky, but working with search programs means it's not too opaque to use for me...hopefully...

Is this EC2 t2.micro instance thing the one that sells comouting power? I certainly would use it if it supports amazon free tier thingy.

[Kazyan]

Is this EC2 t2.micro instance thing the one that sells comouting power? I certainly would use it if it supports amazon free tier thingy.

A t2.micro instance is free if you just use one, and it's Amazon's computing power. I still haven't gotten a successful SSH into it to run anything, though...I still have difficulty with Ubuntu.

[Lewis]

Would it be possible for the Catagolue site to give an option to display pages with static images instead? Any page with more than about ten SVG images on it loads really slowly, and trying to open one of the pages for individual hauls in 2x2 is causing my browser to freeze completely.

[Alexey_Nigin]

Would you mind if I minimize the window, increasing the speed of my DN search by an order of magnitude?

[apg]

Would you mind if I minimize the window, increasing the speed of my DN search by an order of magnitude?

Feel free...

By the way, I've cut down on the number of animated SVGs since this seemed to be an issue amongst people who use Firefox. I've encountered no such difficulties with Google Chrome or Internet Explorer (although the latter doesn't understand animated SVGs, anyway, and displays them as static images).

[gameoflifeboy]

Somebody appears to have started searching LongLife, and it has broken the 1 MB limit:
http://catagolue.appspot.com/census/b345s5/C1