Index

Show enters and exits. Hide enters and exits.

00:33:39leavengoodis MRI 1.8.7p352 fairly stable?
00:33:58parndtI'd say so
00:34:01leavengoodalright
00:34:14leavengoodI know some patch levels can have issues, this seems to be the latest 1.8.7
00:34:19parndtso is ruby-1.9.3p0
00:34:40leavengoodthe rbx README suggests 1.8.7 is required
00:34:48parndtright :)
00:35:22leavengoodwell I'm working on Haiku where I'll need to get an MRI working first, hehe
00:35:31leavengoodunlike Linux where everything just works
00:35:47leavengoodreally need to try rvm too, though I imagine it would need some Haiku changes
00:36:10leavengoodalright I'll see if I can get 1.8.7r352 going
03:14:59SJDAmusing 1.9.3 change: LoadError: cannot load such file -- foo
03:15:03SJD=]
03:55:01Radaroh neat find :)
09:08:14rejeepHey. Does Rubinius support "GC.disable" and "GC.enable"? I mean, do the methods actually do something?
09:19:09dbussinkrejeep: nope
09:19:17dbussinkrejeep: they are there but don't do anything
09:19:30dbussinkpeople don't need to do that
09:19:38dbussinkthat they need it in mri is because the gc isn't very good
09:20:04rejeepWell, it can be used to measure GC time
09:20:18dbussinkrejeep: in rubinius you can use the agent to measure gc time
09:20:26dbussinkit will give you that information pretty easily
09:20:44dbussinkno need to muck around with gc enable / disable
09:21:37rejeepIt would be great with a warning, like in JRuby: warning: "GC.enable does nothing on JRuby"
09:48:21dbussinkrejeep: we could perhaps add a warning yeah
09:48:31dbussinkrejeep: but as a feature we're not supporting it :0
09:48:31dbussink:)
09:49:10rejeepI would still like it as a feature, but a warning is better than nothing! :)
09:50:41rejeepThe reason I would like it, is because then I could easy compare the GC between different Ruby implementations.
09:52:40rejeepFor example: http://pastie.org/2787031
09:53:09rejeepIf every VM has its own tools for measuring this kind of stuff, it's more difficult to compare them
10:04:41dbussinkrejeep: well, i can tell you for sure we will never add it as a feature
10:04:58dbussinkrejeep: we can provide information on how much time was spent in the gc
10:06:34rejeepright, but a warning would be great to avoid confusion!
10:10:17stormbrewyou realize leaving the GC off will completely alter the performance characteristics? Time without - time with is not likely to be a useful measurement
10:11:45stormbrewyou'll increase syscalls to retrieve/allocate new memory chunks, drastically reduce locality of reference, thrash both the virtual address and data (and possibly even code) caches on the cpu, etc.
10:12:34dbussinkstormbrew: very true yeah
10:12:46dbussinka way better measurement is to measure time spent in gc during a normal run
10:12:59dbussinkso you can see what the gc pressure of your benchmark is
10:13:06stormbrewthe way to measure GC time is to actually measure GC time. The REE patches include a mechanism to do that for MRI, afaik
10:13:20dbussinkstormbrew: echo :)_
10:13:22dbussink:P
10:14:05rejeepOk, so a "good" GC can actually perform better than no one at all?
10:14:51stormbrewonce you get up to the amount of physical memory you have in allocated but uncleaned memory, no GC at all will perform rather ridiculously more poorly heh.
10:15:08dbussinkrejeep: no, the thing is that removing gc changes behavior in such a way that the measurement is useless
10:15:09stormbrewand it won't take very long to do that, either
10:15:32stormbrewbut yes, what dbussink said is more the point I was originally making
10:15:46rejeepOhh.. that's useful to know! :)
10:15:57dbussinkthe only valid measurement is that for a benchmark you can say "it spends x% of the time in gc"
10:16:16stormbrewyep
10:16:34stormbrewalso I don't know if rbx or the jvm even realistically could disable the GC
10:16:42stormbrewespecially if the nursery generation is finite
10:17:13dbussinkit will actually greatly reduce performance due to that reason
10:17:22stormbrewin other words, you can only really turn it off in MRI because MRI's collector is so incredibly dumb
10:17:25dbussinkbecause if the nursery is full it will have to do an allocation from mature space each time
10:17:44dbussinkwhich also means for example more locking overhead in allocating objects etc.
10:18:03dbussinkso i wouldn't be suprised that turning of gc will make it slower in rbx
10:18:07dbussinkrejeep: ^^
10:18:39rejeepdbussink: I did find some cases were REE was slower with the GC off
10:18:49rejeepBut not with MRI
10:18:53dbussinkyeah, can very wel be yeah
10:19:01dbussinkbut that already shows that it's not a very useful measure
10:19:09rejeepJupp!
10:19:16stormbrewREE doesn't really change that much about the collector afaik, so that's a bit surprising
10:19:18dbussinkway better is to find ways that you can see how much time it spends in gc
10:19:20stormbrewit's mostly tuning changes
10:19:38stormbrewand the whole fork-friendly silliness
10:20:09dbussinkwell, it's not silly for mri :P
10:20:24rejeepdbussink: You said the agent can help with that?
10:20:37heftigi assume rbx2 will target 1.9.2 and not 1.9.3?
10:20:38stormbrewmy experience is that it's only marginally helpful. Not as much of a gain as claimed
10:21:16stormbrewruby apps' object churn is high enough that pages get touched rather frequently even with it
10:21:22rejeepdbussink: I tried the agent on a script, but it seems like the script cannot exit to use it?
10:21:51dbussinkrejeep: https://gist.github.com/a2eafd0653af8a60c146
10:22:00stormbrewthe COW optimization makes a lot more sense in a language like C++ or C where a significant portion of the allocations take place on the stack and most heap allocations are long-lived
10:22:01dbussinkrejeep: you can get a loopback into the current vm
10:22:17rejeepstormbrew: What about the AST? Not likely to change
10:23:04stormbrewyeah but because you're using a best-fit allocator for the actual storage and an object table for the ruby objects, both are likely to be written to even long after the initial allocations to those pages
10:23:06rejeepdbussink: Cool, thanks!
10:23:35stormbrewagain because the churn is so high in general
10:24:13rejeepstormbrew: right
10:24:22stormbrewif those AST objects were all compacted into their own pages they'd COW well, but I think that's a relatively minor win. Especially for, say, Rails, which has a lot of objects in general
10:25:04stormbrewand the long-lived objects are allocated interleaved with the short-lived objects
10:26:45rejeepstormbrew: But it would be no loss if the AST would be placed in a "Mature Generation" directly, right?
10:27:04stormbrewif MRI had a mature generation :)
10:27:13rejeepYes! :)
10:27:20rejeepBut what about Rubinius for example?
10:27:34dbussinkrubinius has a generational garbage collector
10:27:46dbussinkso that means it allocates objects in a so called young stace
10:27:48dbussinkspace
10:27:55dbussinkwhich is collected more frequently that the mature space
10:27:57heftignursery
10:28:10rejeepYes, so would it make sense to put for example the AST in the Mature Generation directly?
10:28:10dbussinkif an object survives for a few generations, it is copied to the mature space
10:28:25stormbrewin rubinius not so much AST as bytecode
10:28:33dbussinkobjects of type Class are allocated into the mature generation by default
10:28:38dbussinksince they are often there to stay
10:28:47rejeepright
10:28:48dbussinkthere's no AST in rubinius actually, but bytecode
10:29:47rejeepThat's true. The AST does not live in the Ruby world, right? Only in the C++ parts...
10:29:56dbussinkit does live in the ruby world
10:30:06stormbrewa similar fork-friendly optimization to the MRI (in spirit) might be to do an aggressive maturation just before a fork
10:30:07dbussinkbut it's not needed anymore after the compilation to bytecode
10:30:13stormbrew(in rubinius)
10:30:27stormbrewbut I dunno if that'd really yield good results
10:30:38dbussinkwell, since rubinius has real native threads, it doesn't need the forking model to for example have a good parallel web server setup
10:30:58stormbrewabsolutely
10:31:35stormbrewthough I don't think either threads or fork obsolete the other
10:31:41stormbrewtime and place for either
10:32:04dbussinktrue yeah
10:32:16rejeepdbussink: But that requires Rubinius to be thread safe as well, which it is right?
10:32:19stormbrewI meant "to the REE mark-table patch" above btw
10:32:36dbussinkrejeep: it also requires people to write thread safe code on top of rubinius
10:33:05rejeepSure!
10:33:50dbussinkbut for the web that's actually not that hard
10:34:06dbussinksince each request is most of the time completely independent of others
10:34:21dbussinkso as long as you don't save stuff into globals you're usually find
10:34:23dbussinkfine
10:34:51rejeepWhat about DB read/writes?
10:35:14stormbrewwhich is why forking was a model that ever made sense for it
10:37:09dbussinkrejeep: well, regardless of which model (threads / forks) you have to deal with that
10:37:51rejeepAnd that's what you talked about above, which requires people to write thread safe code on top of rubinius?
10:37:51dbussinkrejeep: if you have a forking model server you have to make sure your database is sane still
10:38:15dbussinkrejeep: well, for example not modyfing a shared data structure like an array or hash from multiple threads without proper locking
10:39:53rejeepdbussink: Is there ever any gain of running more than one instance of Rubinius since it supports native threads?
10:42:28dbussinkrejeep: well, if you want to run for example multiple apps in the same process
10:42:34dbussinkrejeep: then it could be useful
10:42:43rejeepI mean for the same app
10:42:50dbussinknot really usually
10:43:05rejeepOk, and that's why Rubinius does not have to use fork?
10:44:02rejeep(for that purpose)
10:44:59dbussinkbecause it doesn't have a gil
10:45:15dbussinkso it can use all the cores in your server with just a single process
10:45:31rejeepI see
10:45:32dbussinkmri has a global interpreter lock which means it can only run 1 thread of ruby code at the same time
10:46:05rejeepFor one process, right?
10:46:41dbussinkalways for one process yea
10:46:52rejeepok
10:46:53dbussinkbecause if you have 2 processes they don't have anything to do with eachother
10:47:06rejeepGotta run, thanks a lot for the answer guys!
10:47:23rejeepright!
10:47:44rejeep*answers :)
14:34:44cremesevan, dbussink: using another program that really stresses threads, ffi, C extensions, etc i can make the latest
14:35:00cremesrbx master exit with an ArgumentError in just a few minutes
14:35:01cremeshttps://gist.github.com/1327622
14:35:23cremes*very rarely* this will abort so i can catch it in gdb for you
14:35:45cremesany ideas on what we can do to figure out what's going on?
14:35:49cremesping me when you get a chance
14:40:45dbussinkcremes: does this happen with the jit disabled?
14:41:23cremesdbussink: i don't know; i'll test and let you know (give me 10m)
14:44:19cremesdbussink: while i am doing this test, are there any other switches you want me to enable/disable on rbx?
14:48:50antaresdbussink: hi. Do you think it's OK to merge master into 2.0.testing?
14:49:01antaresdbussink: looks like we can finally provision VMs tomorrow morning
14:51:22cremesdbussink: even with -Xint, there is a similar crash/exception: https://gist.github.com/1327651
14:54:33cremesthis particular crash is interesting... i oftentimes see it throw these exceptions when there is a slightly
14:54:46cremescomplicated inheritance chain
14:55:09cremesin this case there is a base module that a class has included, so the call to #super in the class is supposed
14:55:31cremesto be handled by the module but it throws that #allocate error because it is trying to call #new on a module
14:59:40dbussinkantares_: i know of some issues in master atm that i'd like to look at with evan actually
14:59:56antaresdbussink: ok, lets keep 2.0.testing the way it is then
14:59:56dbussinkcremes: hmm, weird, but this is reproduceable then?
15:00:30cremesdbussink: throwing an exception is reproducible, yes
15:00:36cremesit may not always happen in the same spot
15:06:02dbussinkcremes: ah, well, reproducable is good anyway :)
15:06:16dbussinkevan probably wants to make his hands dirty on that then ;)
15:14:21dbussinkcremes: it's actually really weird it fails without the jit too :s
15:15:02dbussinkcremes: you sure it passed down the option probably? if it forks it probably creates a sub process with jit enabled again
15:15:58cremesdbussink: this particular stress test does not call fork or #system; it has 10 threads
15:16:05cremesall within the same process
15:16:20dbussinkhmm, very strange then
15:16:38cremesif it wasn't strange then it wouldn't be any fun to track down & fix
15:24:41dbussinkcremes: do you think you're able to abstract this nested structure issue?
15:25:09dbussinkcremes: the thing is that without the jit enabled, there are much less moving parts, so it should fail on the first attempt always and consistently
15:25:27cremesdbussink: i have tried to do that a few times but of course my extraction works correctly
15:32:33dbussinkcremes: well, can't really say much more without diving in then
15:40:35cremesdbussink: if you have any ideas on how i can help track this down, i'm all ears
16:08:57dbussinkcremes: not really, could you perhaps write up steps to reproduce this on your machine?
16:09:03dbussinkcremes: which command(s) to run etc
16:09:04dbussink?
16:10:08cremesdbussink: yes
16:11:37cremesit's a component in a distributed system, so i'll have to create a "stub" that generates the data it is listening for
16:11:55cremes*and* i'll have to modify a downstream component so that it does *not* actually write to a database
16:12:42cremesi can probably whip this up in 30m
16:13:04dbussinkthat would be great
16:13:12dbussinkthen we could look into it on your machine
16:46:53evanmorning!
16:46:56evanalexsuraci: pokes!
16:47:44phlebasmorning!
16:51:23evanwhat's the skinny timmy?
16:52:55cremesdbussink: got an interesting crash in gdb you might want to see... problem with a singleton class
16:52:58cremeswant to debug?
16:55:18cremeshttps://gist.github.com/1327994
16:56:01evanhrm.
16:56:48rueSkinny Timmy sounds like a name of a band even worse than Skinny Puppy
16:57:04cremesevan: it could be due to some of my older and uglier ruby code so pls don't hit me too hard if that's so :)
16:57:16cremesvm shouldn't crash even if some of the code is ugly
16:57:25evan:)
16:57:29evanyeah, it shouldn't.
16:57:33evanwonder what's going on there though.
16:57:44cremesit doesn't happen *every* time
16:57:56cremessometimes it says vm is exiting abnormally but it doesn't get caught by gdb
16:58:07evanwell, in fact.
16:58:12evanif you see that exitting abnormally output
16:58:14evanit should NEVER crash.
16:58:18evanso that is itself, a bug.
16:58:27cremesok
16:58:51cremesi can try to reload it in gdb a few more times to see if gdb will catch it like it did the *first* time
17:00:00evanwell
17:00:19evanthats fine, but we should ignore that and get to why you're getting that abnormal exit
17:00:23evanthats what needs to be solved.
17:01:13cremessomething to do with singleton
17:02:12evanwell
17:02:19evanI can't see why you're saying that from the gist
17:02:24evancould you explain more?
17:02:31evanor maybe gist me some of the code?
17:03:01cremesthat gist is pointing to container.rb line 33 which is trying to retrieve a singleton
17:03:26cremesthat's why i say it has something to do with singletons :)
17:05:16evansure
17:05:20evanbut unless I can see some of the code
17:05:25evanI can't make anything else out
17:05:46evanwhat is line 33 doing?
17:06:05cremesreload that gist
17:06:48evanum
17:06:51evanwhich is line 33?
17:07:07evanoh,
17:07:10evanyour comment is on the end.
17:07:11cremesand reload again ...
17:08:18evanhrm, something is definitely quite confused
17:08:24evanbecause Layer2::MarketData::Container::Message is nowhere to be seen in this code
17:08:40evandoes this app use fibers?
17:08:44cremesno fibers
17:08:49evank
17:08:53cremesand yes, i don't know where Message is coming from
17:09:02cremesit *might* be coming from ZMQ::Message
17:09:12evandoes it blow up like this when it does blow up?
17:09:15cremeswhich means the module parent/child structure is messed up
17:09:47evanie, are there other failures that I could use as data in my bug finding mission?
17:10:12evanthough
17:10:13cremessimilarly... it will complain about a missing method on an object like #open on class Array
17:10:17evanone issue you will absolutely have
17:10:25evanis you're sharing one Mongo connection across all threads?
17:10:35evanis there a lock around it's operations?
17:10:43cremesyes; the driver is supposedly thread safe and allows this
17:10:59evani like the "supposedly" in there.
17:11:02cremes:)
17:11:14cremesconcurrency is hard
17:11:26cremeswhich is why i say supposedly... there might be things missing
17:11:28evanmakes me wonder if it really is.
17:11:33evanat any rate
17:11:42evando you have any backtrace or anything about that Array#open error?
17:11:50cremesno
17:11:58cremesbecause it throws an exception and just exits normally
17:12:20evanwell
17:12:25evanif it throws an exception
17:12:28evanthere should be a backtrace?
17:12:45cremesoh, i thought you meant gdb backtrace
17:12:50evanabsolutely not
17:12:53cremesyes, it will write a ruby bt
17:12:55evani always mean ruby backtrace
17:12:59cremesk
17:13:07evanif i mean VM backtrace, i'll say as such :)
17:13:12evanit's too confusing otherwise
17:13:16cremesk
17:13:25evanand I generally only ask for a VM backtrace if the VM is crashing
17:13:28evanwhich is not the case here.
17:20:39cremeslooks to me like a race between multiple threads trying to retrieve that singleton all at the same time
17:21:19cremesi saw a problem with this many months ago; the fix was to "backport" the MRI 1.9 singleton.rb code to rbx because it
17:21:29cremessupposedly fixed the race conditions for singletons
17:21:38cremesthere's that "supposedly" again
17:22:03evanhm,
17:22:34evani'm not a big fan of Singleton, but it should work right
17:22:41evandoes this happen very quickyl
17:22:44evanor over time?
17:22:51cremesimmediately on launch
17:22:57evanoh good.
17:23:02cremesif you login and attach to screen you can see it
17:23:02evanthat supports your theory.
17:23:06cremes:)
17:23:47evanwhere at?
17:23:50evanI don't see it in the screens
17:24:01evani'm watching, you can just get the screen session to it.
17:24:23evanthis is why screen -x is awesome.
17:24:30evanoh oh
17:24:31evani see.
17:24:39evanwe have different sized windows
17:24:39evanok
17:24:51cremesyeah, mine is 146x57
17:24:55evanoh hm.
17:25:03evani don't see any error ouput actually
17:25:09evanjust that Trying to retrieve
17:25:12evanwhat should I be seeing?
17:25:32cremesit should "hang" because it's in a forever loop waiting for input
17:25:39cremesit's a network app
17:25:43evansure sur
17:25:50evanbut it just exits gracefully it appears
17:25:57evanso i'm not sure what to do with this info :)
17:26:03cremeshang on...
17:26:38cremeswhen i try to run it under gdb several times, eventually it will give me that vm error
17:26:45cremesdo you see it in the scrollback there?
17:26:59evanyep
17:27:02evani can see you scrolling back
17:27:28cremesthe text i scrolled back to is the same as what i put in that gist earlier
17:27:47cremesif you want, try running it under gdb like i did a few times to see it fail
17:28:00evanok
17:28:08evangdb doesn't really have anything to do with it.
17:28:17evanlet me poke at it a bit
17:28:22cremesno, but it seems to help cause that a little more often
17:28:28evangotcha
17:29:09evanoh fun times.
17:29:11evan-d to the rescue!
17:31:00cremeswhere did those LoadErrors come from? weird....
17:31:03evancremes: do you mind if I open these files?
17:31:06evancremes: those are from rubygems
17:31:13cremesgo for it
17:31:31evanthat first NameError is funny
17:31:49evanbecause it appears to be being printed from multiple threads simulatiniously
17:31:53evanor whatever that word is.
17:32:46evancremes: this is good
17:32:53evani'm going to poke around for a little bit
17:33:52cremesglad you are finding it interesting... this may lead to a very subtle bug somewhere...
17:36:14evanhm
17:36:20evanmaybe it's just cascaded in a funny way
17:36:41evanno, it's multiple threads.
17:46:20evancremes: where is the Connection class?
17:46:44evangot it.
17:46:45cremesare you asking for the file path?
17:46:58evanyeah, but I found it.
17:47:00cremesk
17:51:10evanoh, look at this crazy output.
17:52:28evancremes: is Container autoloaded?
17:53:05cremesi don't know anything about autoloader so i don't know
17:53:09evaner.
17:53:14evanConnection
17:53:31cremesthere should be a file chuckkit.rb that requires everything including Connection
17:55:57evancremes: where are the threads started?
17:56:01evanwhat file
17:56:31cremes~/dev/ChuckKit/test/cqg/connectivity/inserter_mt.rb
17:57:10evangotcha
17:57:11evanok
17:59:25chebaDoes ObjectSpace count symbols as objects?
17:59:31evannope
18:00:41chebaSo ruby dynamically creates object whenever I refer to :symbol?
18:00:50evannope
18:01:03evanit creates them at parse time.
18:01:22chebaI see.
18:01:24evanthey are global, immutable, singleton objects.
18:01:31evanlike, say, 3
18:01:39evanevery 3 in the system is the same Fixnum object.
18:01:40chebaAnd if I do "str".to_sym?
18:01:54evanthat will either return the existing symbol for :str
18:01:58evanor make a new one and return it.
18:02:17evanyou can DOS ruby by creating symbols in a loop
18:02:21evanit's a known issue.
18:02:48chebaSure.
18:03:33chebaIs there any method do find out where (and when) symbols created? Apart from grep to_sym, I mean.
18:04:04evanyou can look up String#to_sym
18:04:05evansure.
18:04:45chebaI mean, is that the only way to create symbols at runtime?
18:04:59evanyep.
18:05:01evanwell
18:05:04evanyou can do
18:05:08evan:"blah #{foo}"
18:05:17evanbut thats even harder to find out how it works
18:05:26chebaRight. Thanks.
18:05:30evanare you asking about rubinius or MRI?
18:05:45evanbecause how they implement symbols is simaliar in algorithm
18:05:49evanbut completely different otherwise
18:09:13chebarbx. But I assume it's not very different in mri.
18:09:43alexsuracievan: mornin'. i take it you saw my pull request? :)
18:09:49chebaWhat's the difference?
18:09:51evanalexsuraci: i did!
18:09:55evanalexsuraci: and want to reject it!
18:09:59evanso we have to talk!
18:10:02alexsuracihaha, not unexpected
18:10:05evancheba: completely different code bases
18:10:24evanalexsuraci: I added something else for you to use
18:10:39alexsuracioh?
18:10:42evancheck out lib/compiler/runtime.rb
18:10:47chebaBut they're persisted in the same way and all of them have their singletone objects. Right?
18:10:56evanbasically, you can reference a constant from the literals tuple now
18:11:11headiusevan: given any thought to what might break if symbols could GC?
18:11:17evanyou have to just define rbx_marshal_constant to indicate the name of the constant to resolve when the .rbc is reloaded
18:11:25evanheadius: yeah, i've got a good list.
18:11:37evancheba: yes, semantics are exactly the same.
18:11:45headiusit wouldn't be hard to make them GC, but can think of a few oddities that might result
18:11:52evancremes: oh geez. Message is the BEGINNING of this contsant
18:11:53evan!
18:11:56evanI missed that for the last hour
18:11:59evanHILARIOUS.
18:12:06evanshould that be Messaging::
18:12:06evan?
18:12:10chebaevan: that's what I was interested in. Thanks.
18:12:16evancremes: ^^
18:13:12cremesevan: Messaging is a namespace
18:13:24evanlook in screen
18:13:34evanyou're accessing Message::Layer2
18:13:38evanbut elsewhere you have Messaging::Layer2
18:13:49evanshould Message::Layer2 be Messaging::Layer2?
18:13:55evanie, is this a typo?
18:14:13dbussinkhi peeps
18:14:17evanheadius: we'll discuss GC symbols in a sec, once i'm done with cheba
18:14:17evaner.
18:14:18evancremes:
18:14:25headiussure
18:14:57evanoh cool
18:14:59evanit works now
18:15:01evanit was a typo
18:15:09cremesevan: if that's what it was doing, then yes it's a typo
18:15:11evancremes: pokes pokes pokes pokes
18:15:18evanit seems to be running now.
18:15:19cremespoked
18:15:22cremesok
18:15:30rueNeeds a pager trigger
18:15:33evanso the error was accurate
18:15:53cremesso i had a typo in container.rb ?
18:15:57evanyep
18:16:00cremesd'oh
18:16:06evanyou had Message::Layer2
18:16:21cremesok, time to put this thing under some stress to see if i can expose any threading issues
18:16:34evanthis error certainly wasn't reported well though
18:16:42evanwhich is the source of a bug
18:16:45evanthat i'll work on now
18:16:48evani'm going to detach from screen
18:16:51evanyou can take over again
18:16:59cremesk
18:17:13dbussinkevan: when you have time, i have a question too :)
18:17:18evanheadius: so, GC Symbols
18:17:25headiusyeah
18:17:33evanMRI can't do it because they have static ID values everywhere
18:17:41headiusI've implemented it a couple times but never committed
18:17:41evanso thats why they don't.
18:17:50evanbut at a semantic level
18:17:55alexsuracievan: ok, so I do push_literal x, where x is something that defines rbx_marshal_constant, which returns something like "::Foo::Bar"?
18:18:14headiusat C level, you mean, yes?
18:18:18evanit's less problematic
18:18:22evanheadius: yes.
18:18:29headiusyeah, well, C level suxors
18:18:30evanesp. since rubinius removed Symbol#to_i long ago
18:18:38evanand it hasn't caused really any issues
18:18:39alexsuracievan: and if I do that for all constants it shouldn't matter what the StaticScope is?
18:18:48evanSymbol#to_i would be where compacting would cause a problem
18:19:01headiusalso __id__
18:19:07headiusthat was the main one I'd thought of
18:19:10evanoh yes.
18:19:11evanyes yes
18:19:13headiususing a symbol's object ID
18:19:16evanthats the same as #to_i really
18:19:18headiusyes
18:19:21evanthats the issue
18:19:23headiusbut you can't remove it
18:19:32evanpeople use .__id__ of a symbol as a hash key
18:19:36evanor in a memoized method name
18:19:51headiusick
18:19:52evanalexsuraci: well, no.
18:20:00evanalexsuraci: my point was that rather than adding instructions
18:20:13evanalexsuraci: you should create a runtime object that implements your set_scope method instead of the instruction
18:20:47evanyou effectively get unlimited, extensible instructions implemented as ruby methods
18:21:11alexsuracihow do I actually do the scope-setting in ruby, then?
18:21:11evanyou can see khaase used it to implement flipflop
18:21:24evanwhat do you want to do?
18:21:30evanjust set the lexical scope?
18:21:34alexsuraciah, I'll check the rest of that commit
18:21:55evanCompiledMethod#scope = some_static_scope_object
18:21:57evanlike that.
18:22:01manverui thought Symbol#to_i is dead?
18:22:10evanmanveru: it is, but :a.__id__ isn't.
18:22:25alexsuracievan: well, in atomy each method branch ends up in one CompiledMethod, effectively a big case statement, but each branch needs to retain the static scope from their definition
18:22:33dbussinkevan: this weekend i found this one: https://github.com/rubinius/rubinius/issues/1386
18:22:51dbussinkevan: which makes we wonder why we do this code in the first place: https://gist.github.com/a76f491ff2e42dceef4d
18:22:59manverubut ObjectSpace.id2obj or whatever that was is dead too?
18:23:01dbussinkevan: seems a bit weird to always wake that up when waking up a thread
18:23:15dbussinki was wondering what the rationale is behind doing that on contented objects when a thread wakes up
18:23:21evandbussink: ok, I have to figure out this code before we discuss it
18:23:25headiusmanveru: I think evan's saying people still use Symbol#__id__ as a one-way hashing algorithm
18:23:36evanheadius: yes, thats what I mean
18:23:39manveruah
18:23:40evanand _id2ref isn't dead either.
18:23:40headiusand expect the same symbol later, regardless of GC characteristics, to produce the same result
18:23:43evanit's very much alive.
18:23:45manveruyeah, that's not great
18:24:03manverudamn
18:24:11evanalexsuraci: seems simple
18:24:14rueMaybe Greg can put that in a Ruby Worst Practices
18:24:29evanalexsuraci: you call a set_scope method instead of using your set_scope instruction
18:24:40dbussinkevan: ok, np, i've gone through it a bit so i should be able to explain things too ;)
18:24:42headiuslet's just convert symbol's string into a base 256 number :D
18:24:46headiusdone!
18:24:55evan:)
18:25:11headiusSymbol can be an immutable Bignum
18:25:14headiusdone!
18:25:34headiuser, well...bignum is immutable already
18:25:37evanan option, realisticly, is to assign Symbol's a numeric id lazily
18:25:42headiusso we just need Symbol methods on Bignum
18:26:02evanand if a symbol gets an id
18:26:06dbussinkevan: and then not gc them if they have an id?
18:26:12evanit's retained forever
18:27:51headiushmm
18:28:43evandbussink: ok, that release_contention is not actually releasing
18:28:46evanthats a misnomer
18:28:48headiuslooks like we already assign __id__ lazil
18:28:49headiusy
18:29:05headiusat which point it goes into a weak map from id to obj
18:29:07evandbussink: it simply signals any contending thread to recheck
18:29:09dbussinkevan: i saw it's signaling right?
18:29:17dbussinkbut does that really help in this case?
18:29:20evanright
18:29:27evanbut anyone waiting just wakes up and rechecks things.
18:29:31headiusoddly enough it appears we have never been assigning symbol __id__ eagerly
18:29:35dbussinkas in, how useful is it to wake up here?
18:29:42evandbussink: absolutely critical.
18:29:49evanbecause if you're waiting on a lock
18:29:55evanin a the main thread
18:30:00evanand SIGINT arrives
18:30:07evanSIGINT must abort the lock attempt
18:30:12headiussystem ~/projects/jruby $ jruby -e "p :foo.__id__; p :foo.to_i"
18:30:13headius2000
18:30:13headius171
18:30:47headiusto_i is still the eagerly-assigned monotonically-increasing symbol index
18:30:59alexsuracievan: what would I be passing to that method, though? looks like flip flops just mutates the StaticScope; would I just be replacing the relevant bits? (i.e. what makes them deep-equal)
18:31:18dbussinkevan: ok, then we need to fix this issue, the problem here is that because now there's a gc lock guard around that release contention
18:31:28dbussinkevan: i explained it also a bit in the issue
18:31:57evanalexsuraci: lets table this
18:32:01evanIRC is only good for one convo at a time.
18:32:07evanand dbussink is winning this battle
18:32:22evandbussink: ok, looking now.
18:32:43evandbussink: hm
18:32:45alexsuracievan: sure
18:32:49evani guess i'm confused about WHY this is crashing still.
18:32:49dbussinkevan: ok, btw, the way i triggered this issue is by running ci in a loop inside gdb
18:33:06headiusevan: so I think in jruby we could have symbol's lazily-generated ID trigger a hard reference and other symbols could GC
18:33:25dbussinkevan: well, a thread is woken up and right after a gc is requested so a stop the world is attempted
18:33:26rueThat does sound like a reasonable compromise
18:33:40headiusstill a potential leak/DOS, but less exposure
18:33:41rueBetween id'ing all symbols and setting everybody on fire
18:33:53dbussinkso that thread that is woken up is marked as suspended but then tries to unlock the contention lock
18:34:23evanhm
18:34:27dbussinkand in that unlocking it sees it's already suspended and therefore aborts
18:34:33evanI need to trace this
18:34:40evani'm not getting it completely yet.
18:34:42dbussinkthe 'bug' is because there's an explicit rubinius::bug() here
18:34:45dbussinkok
18:35:05yipdwheadius: just as an FYI, Chez Scheme does something similar to that with its implementation of gensym
18:35:52dbussinkevan: this is how i run it in a loop: https://gist.github.com/391e20d1879a5af0e83d
18:36:04dbussinkevan: works well as a stress test to let it run a while ;)
18:36:16evanhehe
18:36:18evanok
18:40:22rueWas there a decision made on #to_bool
18:40:24rue?
18:40:39rueInstead of the pervasive !!'s
18:40:41headiusyipdw: ahh, interesting
18:41:37alexsuracievan: figured out a way to do it without adding set_scope :)
18:41:45evank
18:42:02alexsuracipush_variables; send :method, 0; push_literal scope; send :"scope=", 1
18:42:13alexsuracikind of a roundabout way of doing it, but it works.
18:42:38dbussinkevan: oh, btw, i wasn't able to trigger this with a dev build :s
18:42:46boyscoutMerge pull request #1387 from xaviershay/proc-19-take2 - 6789158 - Eero Saynatkari
18:42:56dbussinkevan: only with a regular one it actually failed once in a while
18:43:03evandbussink: yeah, it's totally a timing bug.
18:43:33evanand reenforces that we need to start tracking more thread state to detect potential deadlocks and bad states proactively
18:43:42evanrather than trying to eek timing issues out
18:44:09dbussinkwell, the bug() check here is kinda thread state checking
18:44:25rueHm
18:45:11rueIt's refusing to build on 1.9.3-p0 for me, but it could be an rvm/env issue. Anyone else seen?
18:46:47headiusI need to merge that Proc#curry in too
18:51:13evandbussink: ok, so why was the thread calling wakeup already suspended?
18:51:19evandbussink: I don't see where that would be
18:52:52dbussinkevan: in wait_to_run
18:53:03dbussinkevan: line 6 of backtraces in the gist linked in the issue
18:53:07dbussinkevan: https://gist.github.com/e8c1247fa5e7b2d01b02
18:53:21evanwell
18:53:21evanno.
18:53:27evanthats not the thread that called rubinius::bug
18:54:00evanthat is just the Channel starting to wait
18:54:12dbussinkevan: it has the same state actually, 0x109003e50
18:54:25evansame as what?
18:54:29evani don't follow.
18:54:40dbussink#4 0x00000001000e001b in rubinius::WorldState::wait_to_run (this=0x100f06800, state=0x109003e50) at shared_state.cpp:189
18:54:45evanalso, you've got optimizations on, so you can't trust that.
18:54:55dbussink#4 0x00000001000e0082 in rubinius::WorldState::become_independent (this=0x100f06800, state=0x109003e50) at world_state.hpp:56
18:54:56evanplease give me lines in the gist
18:54:58evanto compare
18:55:10evanbecause i have no context for those pastes
18:55:16dbussinkline 6 and 72
18:55:42dbussinkevan: can you repro it using those gdb scripts?
18:55:50evani'm not trying.
18:55:55dbussinkok
18:55:56evani'm trying to fully understand this first.
18:55:59evanthats more important.
18:56:09evanok, that line 6 and 72 are the same IS BAD
18:56:10evanVERY
18:56:12evanVER
18:56:13evanBAD
18:56:19dbussinkyeah
18:56:29evanthat means this is not that isuse at all
18:56:34evanbut rather the wrong state is used
18:56:35evanFUCK
18:56:36evanyes.
18:56:38evanline 384
18:56:41evanthis should be state
18:57:00dbussinkwhich file?
18:57:04evanvm.cpp
18:57:12evanthats the bug.
18:57:21evanrelease_contention needs to be passed the current thread's STATE
18:57:28evanNOT the state of the thread we're trying to wakeup
18:57:31dbussinkah, of course!
18:57:35evanexactly.
18:57:54DefilerWhat could go wrong?
18:57:55Defiler:)
18:58:16evanthis is one flaw in my organization
18:58:36evanthe "current thread state" is the same as "some other threads state"
18:58:45evanso static typing doesn't help us out
18:58:54evanmaybe I should bite the bullet and change the type of STATE
18:59:41dbussinkevan: actually looks like a really ancient bug
18:59:42evandbussink: wanna commit this fix?
18:59:45evanyeah
18:59:46dbussinkyeah
18:59:50dbussinkafter stress testing it]
18:59:54evanI'm going to bite the bullet
18:59:59evanand see how much work it would be to change STATE
19:00:11dbussinkshould be very little ;_
19:00:16evan:)
19:00:18dbussinkor a lot of potential bugs show up ;)
19:00:23evanexactly.
19:00:29evannow is the time too
19:00:30evanpre 2.0
19:01:35jdsiegelcrystal method's Now is the Time (Rubinius MIX)
19:01:49evan:D
19:03:20boyscoutAdded docs for 1.9 Proc#curry, #source_location, #to_s. - 81214a0 - Eero Saynatkari
19:08:05dbussinkevan: no crash yet :)
19:08:20evan:)
19:08:37dbussinkgoing to let it run for a few more runs and push it
19:08:43dbussinkit's wrong anyway how it's now :P
19:09:44boyscoutWe need to pass in the current thread - c5adaa5 - Dirkjan Bussink
19:11:16evan:)
19:11:23evanI've started the STATE change
19:11:35evanhopefully will be able to get it done today
19:11:38evanbut now, lunch!
19:14:40dbussinkevan: cool :)
20:18:02dbussinkevan: i was looking at https://github.com/rubinius/rubinius/issues/1365 but i'm really stumped by what is happening there, doesn't seem to make any sense :s
20:18:19evanyeah, no clue.
20:18:22evanit's not a null pointer problem
20:18:27evanso don't let that confuse you.
20:19:04dbussinkevan: that linkedlist is totally weird there; https://gist.github.com/e59aa77594e68f096f29
20:20:11evanyou've got optimizations on probably
20:20:13evanso don't be too confused
20:20:32dbussinknope, this is with DEV=1
20:20:45evandid you rake clean first?
20:20:49dbussinkyeah
20:20:53evan:/
20:21:01evanno clue.
20:21:06dbussinkit's trivially easy to reproduce with the stuff in that issue though
20:21:19dbussinkalso tried without the jit, no difference
20:21:21evani wonder if Fibers are confusing a thread's variable buffer list.
20:21:46dbussinkmust be something like that i guess :s
20:21:56dbussinkbut my assembly fiber fu isn't up to that :p
20:22:17evancould you comment on the issue at least
20:22:22evanindicating that you can repro it easily
20:22:30evanso I don't forget to look
20:24:07dbussinkcommented with the gist attached
20:25:39dbussinkevan: tarcieri was also seeing some pretty heavy performance regression in those fiber benchmarks btw
20:26:03evanyeah
20:26:04evani know.
20:26:08evani haven't investigated it.
20:32:18dbussinkevan: easy to change STATE or does it expose more stuff?
20:32:38evana lot of places to change and fix
20:32:40evanbut it's good.
20:32:47evanbecause it's letting me unwind a bunch of things
20:35:25dbussinkevan: things such as?
20:35:46evanthe VM class a bunch of crap on it
20:35:52evanit's full of random functions
20:40:22evanoh
20:40:25evanalready found a bug.
20:41:14dbussinkyay!
20:50:34evanand found another one.
20:50:48evanall in the agent
20:55:58dbussinkagent could cause stuff to crash?
20:56:05dbussinkor issues in the agent itself?
21:17:31evandbussink: both.
22:02:02antaresdbussink: hi
22:02:07antaresdbussink: are you still around?
22:21:59evandbussink: found more bugs
22:22:02evanhuzzah for this change.
22:22:21rueYay changes
22:22:29rueOoh, GVR is dissing Zed :D
22:23:43steveklabnikheh
22:23:50evanrue: where at?
22:24:09petercoopershooting fish in a barrel
22:24:11ruehttps://twitter.com/#!/gvanrossum/status/131133804302704642
22:36:48antaresevan: hi. Can you guys tell me why Rubinius needs exactly Ruby 1.8.7 to be built?
22:37:03evanit doesn't.
22:37:16antaresevan: well, if we want 1.8 mode by default it does
22:37:25evannah.
22:37:25antaresevan: jruby or 1.8.6 won't work, rubinius itself won't work
22:37:33evanshould work fine
22:37:35evanwhat doesn't work?
22:37:59antaresevan: what happened is 1.8.6 was installed before rubinius so rubinius build system picked it and then failed to build
22:38:13evanfailed how is my question
22:38:27evanjruby will fail probably because we need C extensions to build it properly
22:38:29antaresevan: this is definitely an issue with our cookbook, too, but I am curious as to whether it is an issue with 1.8.6 support or by design
22:38:39evanso we build melbourne, the C extension, for the builder ruby
22:38:48evanwhat error did you get?
22:38:52antaresevan: I did not recover the log
22:38:58evanmy guess is we haven't tried to build with 1.8.6 in a long time is all.
22:39:15antaresevan: right, that was my idea, too
22:39:17evanbut I can't speak to any problems without looking at a log
22:39:25antaresthat it just so happens that 1.8.6 build issue wasn't noticed
22:39:48antaresevan: that's ok, we are reworking our RVM cookbook to install things in a strict order anyway
22:40:05evanok
22:40:13evanwe should support most versions of MRI
22:40:23evanif 1.8.6 is busted
22:40:28evanit might be a simple fix to get it going again
22:44:02rueHm, should see if I can debug that .3-p0 failure.
23:10:30bakkdooranybody else getting this error with SHA1? https://gist.github.com/1329353
23:12:13ruebakkdoor: Yep
23:12:19bakkdoor:(
23:15:03brixenI think I fixed one like this before
23:15:14brixenbuilding rbx right now, I'll take a look in a sec.
23:16:00bakkdoorok
23:16:05bakkdoorit worked before
23:16:15brixenk
23:16:28brixensome lib stuff has been moved around recently
23:17:01evanthis kind of thing happens when the file is loaded twice
23:17:19evanbecause orig_new points to the previous version of the wrapper code
23:22:21bakkdoorok found the problem
23:22:33bakkdoorthe same code is in lib/sha1.rb and lib/digest/sha1.rb
23:22:38bakkdoorwhere should it stay?
23:22:56evanha.
23:23:04evanoh geez.
23:23:05evanum.
23:23:08evango check 1.8
23:23:13bakkdoor-_-
23:24:15bakkdoori suppose just keep it in lib/digest/sha1.rb
23:24:50bakkdoori wonder how this worked before, those files haven't been touched in a while
23:24:54brixenyeah, just have lib/sha1.rb require the other
23:25:03brixenmost people require 'digesh/sha1'
23:25:22bakkdoorok
23:25:25bakkdoorgood to know
23:25:26bakkdoorwill push
23:25:30brixenk
23:25:45steveklabnikoh man.
23:25:49steveklabnikthose docs are the worst too
23:25:54steveklabniki was trying to look at them the other day
23:26:05steveklabnikall of Digest is spread out over like 6 c and 4 ruby files.
23:29:04brixenI wonder if we should revisit using submodules in the main rubinius repo
23:29:17brixennot for specs, like before, but stuff like the web/ directory
23:29:23brixenand the gems we are making
23:29:49brixenthe build step could automatically pull code
23:32:33evanmaybe.
23:36:38boyscoutRemoved duplicate code from lib/sha1.rb and just let it require lib/digest/sha1.rb which has the same wrapper code already - acf6926 - Christopher Bertels
23:39:20brixenevan: that way we could work on the docs with less thrashing in Rubinius itself
23:40:03evani'm open to it
23:40:04brixenalso could be a good way to manage contributions to stuff like the profiler or debugger porcelain
23:40:11evanif you can figure out how to make it work well.
23:40:13brixenok
23:40:40brixenevan: I'm thinking about making melbourne18, melbourne19, and melbourne20
23:40:52brixenand you can load any of them with no symbols clashing
23:41:11brixenbut break up the code by language version into separate repos
23:41:59evansure
23:42:20brixenI'd like to do the same with the bytecode compiler without duplicating much code
23:44:02evani'm a little less comfortable with that.
23:44:06evanbut we can talk about it.
23:44:29brixenwell, the idea is to have better separation between the language versions
23:44:37brixenwe're now going to have 1.8, 1.9 and 2.0
23:44:55brixenit will also be nice to build a 1.9 or 2.0 version of rbx that has no baggage
23:45:06brixenthinking about how to support this better