[yt-dev] Zombie jobs on eudora?

Nathan Goldbaum nathan12343 at gmail.com
Wed Jun 11 13:30:12 PDT 2014 

For all of these tests I estimated the memory usage by watching the 'RES'
column under htop as the script ran.

On Wed, Jun 11, 2014 at 12:30 PM, Matthew Turk <matthewturk at gmail.com>
wrote:

> Can you try with:
>
>  * Just a slice
>

Roughly constant memory usage: http://paste.yt-project.org/show/4774/


>  * Just a slice you've accessed 'ones' in
>

Steadily increasing memory usage, peaking at about 1.3 gigabytes:
http://paste.yt-project.org/show/4776/

Note that these datasets range from ~300 kilobytes in size up to about 30
megabytes.


>  * Slice => FRB
>

Roughly constant memory usage: http://paste.yt-project.org/show/4777/

(see the PR I just issued to get this script working properly).


>  * Just pf.index
>

Steadily increasing memory usage, peaking at about 1.2 GB:
http://paste.yt-project.org/show/4778/


>
> On Wed, Jun 11, 2014 at 2:26 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > I can confirm that creating the list alias to .grids does eliminate the
> Grid
> > objects from showing up in the objgraph output.
> >
> > That said, I'm still seeing steadily increasing memory usage when I
> iterate
> > over a bunch of datasets (http://paste.yt-project.org/show/4773/),
> creating
> > SlicePlots for each one.  I'm not sure yet where the memory is going,
> just
> > that objgraph can't see it.
> >
> >
> > On Wed, Jun 11, 2014 at 12:02 PM, Matthew Turk <matthewturk at gmail.com>
> > wrote:
> >>
> >> On Wed, Jun 11, 2014 at 1:58 PM, Nathan Goldbaum <nathan12343 at gmail.com
> >
> >> wrote:
> >> > Could this issue be related?
> >> >
> >> > https://github.com/numpy/numpy/issues/1601
> >>
> >> Yeah, that's the one.
> >>
> >> >
> >> > Can you elaborate a bit more about why we're using an object array in
> >> > the
> >> > first place?  If switching to using a list solves these issues perhaps
> >> > that
> >> > is the way to go.
> >>
> >> Two reasons.  One is that it's an OOM faster for some things that we
> >> do a lot, and the other is that it makes it much easier to index.  We
> >> can do things like selection based on indices or booleans this way.
> >> But we don't do that very often anymore.
> >>
> >> I don't want to switch it to a list; that's a nasty bandaid that
> >> breaks things.  We can just add on an additional list, which is a
> >> nasty bandaid that doesn't break things.  I think the memory overhead
> >> will be minimal for that.  Really fixing it will require moving away
> >> from arrays completely, which we can slot in for 3.1.
> >>
> >> >
> >> >
> >> > On Wed, Jun 11, 2014 at 7:25 AM, Matthew Turk <matthewturk at gmail.com>
> >> > wrote:
> >> >>
> >> >> I should also note, at some point in the future I want to get rid of
> >> >> the object arrays for grids, but that timescale is longer.  Using
> >> >> John's grid tree is a much better approach.
> >> >>
> >> >> On Wed, Jun 11, 2014 at 9:21 AM, Matthew Turk <matthewturk at gmail.com
> >
> >> >> wrote:
> >> >> > On Tue, Jun 10, 2014 at 10:53 PM, Matthew Turk
> >> >> > <matthewturk at gmail.com>
> >> >> > wrote:
> >> >> >> On Tue, Jun 10, 2014 at 10:50 PM, Nathan Goldbaum
> >> >> >> <nathan12343 at gmail.com> wrote:
> >> >> >>> For the leaking YTArrays, Kacper suggested the following patch on
> >> >> >>> IRC:
> >> >> >>>
> >> >> >>> http://bpaste.net/show/361120/
> >> >> >>>
> >> >> >>> This works for FLASH but seems to break field detection for enzo.
> >> >> >>
> >> >> >> I don't think this will ever be a big memory hog, but it is worth
> >> >> >> fixing.
> >> >> >>
> >> >> >
> >> >> > I've spent a small bit of time at this again this morning, and
> >> >> > everything seems to come back down to the issue of having a numpy
> >> >> > array of grid objects.  If I switch this to a list, the reference
> >> >> > counting is correct again and things get deallocated properly.
>  I've
> >> >> > tried a number of ways of changing how they're allocated, but none
> >> >> > seem to work for getting the refcount correct.  Oddly enough, if I
> >> >> > track both a list *and* an array (i.e., set self._grids =
> >> >> > self.grids.tolist()) then the refcounting is correct.
> >> >> >
> >> >> > I'm sure there's an explanation for this, but I don't know it.  It
> >> >> > looks to me like numpy thinks it owns the data and that it should
> >> >> > decrement the object refcount.
> >> >> >
> >> >> > By adding this line:
> >> >> >
> >> >> > self._grids = self.grids.tolist()
> >> >> >
> >> >> > after the call to _populate_grid_objects() in
> grid_geometry_handler,
> >> >> > I
> >> >> > was able to get all references tracked and removed.
> >> >> >
> >> >> >>>
> >> >> >>>
> >> >> >>> On Tue, Jun 10, 2014 at 8:47 PM, Matthew Turk
> >> >> >>> <matthewturk at gmail.com>
> >> >> >>> wrote:
> >> >> >>>>
> >> >> >>>> Hi Nathan,
> >> >> >>>>
> >> >> >>>> I believe there are two things at work here.
> >> >> >>>>
> >> >> >>>> 1) (I do not have high confidence of this one.)  YTArrays that
> are
> >> >> >>>> referenced with a .d and turned into numpy arrays which no
> longer
> >> >> >>>> *own
> >> >> >>>> the data* may be retaining a reference, but that reference
> doesn't
> >> >> >>>> get
> >> >> >>>> freed later.  This happens often when we are doing things in the
> >> >> >>>> hierarchy instantiation phase.  I haven't been able to figure
> out
> >> >> >>>> which references get lost; for me, over 40 outputs, I lost 1560.
> >> >> >>>> I
> >> >> >>>> think it's 39 YTArrays per hierarchy.  This might also be
> related
> >> >> >>>> to
> >> >> >>>> field detection.  I think this is not a substantial contributor.
> >> >> >>>> 2) For some reason, when the .grids attribute (object array) is
> >> >> >>>> deleted on an index, the refcounts of those grids don't
> decrease.
> >> >> >>>> I
> >> >> >>>> am able to decrease their refcounts by manually setting
> >> >> >>>> pf.index.grids[:] = None.  This eliminated all retained grid
> >> >> >>>> references.
> >> >> >>>>
> >> >> >>>> So, I think the root is that at some point, because of circular
> >> >> >>>> references or whatever, the finalizer isn't being called on the
> >> >> >>>> Gridndex (or on Index itself).  This results in the reference to
> >> >> >>>> the
> >> >> >>>> grids array being kept, which then pumps up the lost object
> count.
> >> >> >>>> I
> >> >> >>>> don't know why it's not getting called (it's not guaranteed to
> be
> >> >> >>>> called, in any event).
> >> >> >>>>
> >> >> >>>> I have to take care of some other things (including Brendan's
> note
> >> >> >>>> about the memory problems with particle datasets) but I am
> pretty
> >> >> >>>> sure
> >> >> >>>> this is the root.
> >> >> >>>>
> >> >> >>>> -Matt
> >> >> >>>>
> >> >> >>>> On Tue, Jun 10, 2014 at 10:13 PM, Matthew Turk
> >> >> >>>> <matthewturk at gmail.com>
> >> >> >>>> wrote:
> >> >> >>>> > Hi Nathan,
> >> >> >>>> >
> >> >> >>>> > All it requires is a call to .index; you don't need to do
> >> >> >>>> > anything
> >> >> >>>> > else to get it to lose references.
> >> >> >>>> >
> >> >> >>>> > I'm still looking into it.
> >> >> >>>> >
> >> >> >>>> > -Matt
> >> >> >>>> >
> >> >> >>>> > On Tue, Jun 10, 2014 at 9:26 PM, Nathan Goldbaum
> >> >> >>>> > <nathan12343 at gmail.com>
> >> >> >>>> > wrote:
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> On Tue, Jun 10, 2014 at 10:59 AM, Matthew Turk
> >> >> >>>> >> <matthewturk at gmail.com>
> >> >> >>>> >> wrote:
> >> >> >>>> >>>
> >> >> >>>> >>> Do you have a reproducible script?
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> This should do the trick:
> >> >> >>>> >> http://paste.yt-project.org/show/4767/
> >> >> >>>> >>
> >> >> >>>> >> (this is with an enzo dataset by the way)
> >> >> >>>> >>
> >> >> >>>> >> That script prints (on my machine):
> >> >> >>>> >>
> >> >> >>>> >> EnzoGrid    15065
> >> >> >>>> >> YTArray     1520
> >> >> >>>> >> list        704
> >> >> >>>> >> dict        2
> >> >> >>>> >> MaskedArray 1
> >> >> >>>> >>
> >> >> >>>> >> Which indicates that 15000 EnzoGrid objects and 1520 YTArray
> >> >> >>>> >> objects
> >> >> >>>> >> have
> >> >> >>>> >> leaked.
> >> >> >>>> >>
> >> >> >>>> >> The list I'm printing out at the end of the script should be
> >> >> >>>> >> the
> >> >> >>>> >> objects
> >> >> >>>> >> that leaked during the loop over the Enzo dataset.  The
> >> >> >>>> >> objgraph.get_leaking_objects() function returns the list of
> all
> >> >> >>>> >> objects
> >> >> >>>> >> being tracked by the garbage collector that have no
> references
> >> >> >>>> >> but
> >> >> >>>> >> still
> >> >> >>>> >> have nonzero refcounts.
> >> >> >>>> >>
> >> >> >>>> >> This means the "original_leaks" list isn't necessarily a list
> >> >> >>>> >> of
> >> >> >>>> >> leaky
> >> >> >>>> >> objects - most of the things in there are singletons that the
> >> >> >>>> >> interpreter
> >> >> >>>> >> keeps around. To create a list of leaky objects produced by
> >> >> >>>> >> iterating
> >> >> >>>> >> over
> >> >> >>>> >> the loop I take the set difference of the output of
> >> >> >>>> >> get_leaking_objects()
> >> >> >>>> >> before and after iterating over the dataset.
> >> >> >>>> >>
> >> >> >>>> >>>
> >> >> >>>> >>> If you make a bunch of symlinks to
> >> >> >>>> >>> one flash file and load them all in sequence, does that
> >> >> >>>> >>> replicate
> >> >> >>>> >>> the
> >> >> >>>> >>> behavior?
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> Yes, it seems to.  Compare the output of this script:
> >> >> >>>> >> http://paste.yt-project.org/show/4768/
> >> >> >>>> >>
> >> >> >>>> >> Adjust the range of the for loop from 0 to 5 - creating the
> >> >> >>>> >> needed
> >> >> >>>> >> symlinks
> >> >> >>>> >> to WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0040 as needed.
> >> >> >>>> >>
> >> >> >>>> >>>
> >> >> >>>> >>>
> >> >> >>>> >>> On Tue, Jun 10, 2014 at 12:57 PM, Nathan Goldbaum
> >> >> >>>> >>> <nathan12343 at gmail.com>
> >> >> >>>> >>> wrote:
> >> >> >>>> >>> >
> >> >> >>>> >>> >
> >> >> >>>> >>> >
> >> >> >>>> >>> > On Tue, Jun 10, 2014 at 10:45 AM, Matthew Turk
> >> >> >>>> >>> > <matthewturk at gmail.com>
> >> >> >>>> >>> > wrote:
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> Hi Nathan,
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> On Tue, Jun 10, 2014 at 12:43 PM, Nathan Goldbaum
> >> >> >>>> >>> >> <nathan12343 at gmail.com>
> >> >> >>>> >>> >> wrote:
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> > On Tue, Jun 10, 2014 at 6:09 AM, Matthew Turk
> >> >> >>>> >>> >> > <matthewturk at gmail.com>
> >> >> >>>> >>> >> > wrote:
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Hi Nathan,
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> On Mon, Jun 9, 2014 at 11:02 PM, Nathan Goldbaum
> >> >> >>>> >>> >> >> <nathan12343 at gmail.com>
> >> >> >>>> >>> >> >> wrote:
> >> >> >>>> >>> >> >> > Hey all,
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > I'm looking at a memory leak that Philip (cc'd) is
> >> >> >>>> >>> >> >> > seeing
> >> >> >>>> >>> >> >> > when
> >> >> >>>> >>> >> >> > iterating
> >> >> >>>> >>> >> >> > over a long list of FLASH datasets.  Just as an
> >> >> >>>> >>> >> >> > example
> >> >> >>>> >>> >> >> > of the
> >> >> >>>> >>> >> >> > type
> >> >> >>>> >>> >> >> > of
> >> >> >>>> >>> >> >> > behavior he is seeing - today he left his script
> >> >> >>>> >>> >> >> > running
> >> >> >>>> >>> >> >> > and
> >> >> >>>> >>> >> >> > ended
> >> >> >>>> >>> >> >> > up
> >> >> >>>> >>> >> >> > consuming 300 GB of RAM on a viz node.
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > FWIW, the dataset is not particularly large - ~300
> >> >> >>>> >>> >> >> > outputs and
> >> >> >>>> >>> >> >> > ~100
> >> >> >>>> >>> >> >> > MB
> >> >> >>>> >>> >> >> > per
> >> >> >>>> >>> >> >> > output. These are also FLASH cylindrical coordinate
> >> >> >>>> >>> >> >> > simulations -
> >> >> >>>> >>> >> >> > so
> >> >> >>>> >>> >> >> > perhaps
> >> >> >>>> >>> >> >> > this behavior will only occur in curvilinear
> >> >> >>>> >>> >> >> > geometries?
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Hm, I don't know about that.
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > I've been playing with objgraph to try to understand
> >> >> >>>> >>> >> >> > what's
> >> >> >>>> >>> >> >> > happening.
> >> >> >>>> >>> >> >> > Here's the script I've been using:
> >> >> >>>> >>> >> >> > http://paste.yt-project.org/show/4762/
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > Here's the output after one iteration of the for
> loop:
> >> >> >>>> >>> >> >> > http://paste.yt-project.org/show/4761/
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > It seems that for some reason a lot of data is not
> >> >> >>>> >>> >> >> > being
> >> >> >>>> >>> >> >> > garbage
> >> >> >>>> >>> >> >> > collected.
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > Could there be a reference counting bug somewhere
> down
> >> >> >>>> >>> >> >> > in
> >> >> >>>> >>> >> >> > a
> >> >> >>>> >>> >> >> > cython
> >> >> >>>> >>> >> >> > routine?
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Based on what you're running, the only Cython routines
> >> >> >>>> >>> >> >> being
> >> >> >>>> >>> >> >> called
> >> >> >>>> >>> >> >> are likely in the selection system.
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> > Objgraph is unable to find backreferences to root
> grid
> >> >> >>>> >>> >> >> > tiles
> >> >> >>>> >>> >> >> > in
> >> >> >>>> >>> >> >> > the
> >> >> >>>> >>> >> >> > flash
> >> >> >>>> >>> >> >> > dataset, and all the other yt objects that I've
> looked
> >> >> >>>> >>> >> >> > at
> >> >> >>>> >>> >> >> > seem
> >> >> >>>> >>> >> >> > to
> >> >> >>>> >>> >> >> > have
> >> >> >>>> >>> >> >> > backreference graphs that terminate at a FLASHGrid
> >> >> >>>> >>> >> >> > object
> >> >> >>>> >>> >> >> > that
> >> >> >>>> >>> >> >> > represents a
> >> >> >>>> >>> >> >> > root grid tile in one of the datasets.  That's the
> >> >> >>>> >>> >> >> > best
> >> >> >>>> >>> >> >> > guess
> >> >> >>>> >>> >> >> > I
> >> >> >>>> >>> >> >> > have
> >> >> >>>> >>> >> >> > -
> >> >> >>>> >>> >> >> > but
> >> >> >>>> >>> >> >> > definitely nothing conclusive.  I'd appreciate any
> >> >> >>>> >>> >> >> > other
> >> >> >>>> >>> >> >> > ideas
> >> >> >>>> >>> >> >> > anyone
> >> >> >>>> >>> >> >> > else
> >> >> >>>> >>> >> >> > has to help debug this.
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> I'm not entirely sure how to parse the output you've
> >> >> >>>> >>> >> >> pasted, but
> >> >> >>>> >>> >> >> I
> >> >> >>>> >>> >> >> do
> >> >> >>>> >>> >> >> have a thought.  If you have a reproducible case, I
> can
> >> >> >>>> >>> >> >> test it
> >> >> >>>> >>> >> >> myself.  I am wondering if this could be related to
> the
> >> >> >>>> >>> >> >> way
> >> >> >>>> >>> >> >> that
> >> >> >>>> >>> >> >> grid
> >> >> >>>> >>> >> >> masks are cached.  You should be able to test this by
> >> >> >>>> >>> >> >> adding
> >> >> >>>> >>> >> >> this
> >> >> >>>> >>> >> >> line
> >> >> >>>> >>> >> >> to _get_selector_mask in grid_patch.py, just before
> >> >> >>>> >>> >> >> "return
> >> >> >>>> >>> >> >> mask"
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> self._last_mask = self._last_selector_id = None
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> Something like this patch:
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> http://paste.yt-project.org/show/4316/
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> > Thanks for the code!  I will look into this today.
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> > Sorry for not explaining the random terminal output I
> >> >> >>>> >>> >> > pasted
> >> >> >>>> >>> >> > from
> >> >> >>>> >>> >> > objgraph
> >> >> >>>> >>> >> > :/
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> > It's a list of objects created after yt operates on one
> >> >> >>>> >>> >> > dataset
> >> >> >>>> >>> >> > and
> >> >> >>>> >>> >> > after
> >> >> >>>> >>> >> > the garbage collector is explicitly called. Each
> >> >> >>>> >>> >> > iteration
> >> >> >>>> >>> >> > of the
> >> >> >>>> >>> >> > loop
> >> >> >>>> >>> >> > sees
> >> >> >>>> >>> >> > the creation of objects representing the FLASH grids,
> >> >> >>>> >>> >> > hierarchy,
> >> >> >>>> >>> >> > and
> >> >> >>>> >>> >> > associated metadata.  With enough iterations this
> >> >> >>>> >>> >> > overhead
> >> >> >>>> >>> >> > from
> >> >> >>>> >>> >> > previous
> >> >> >>>> >>> >> > loop iterations begins to dominate the total memory
> >> >> >>>> >>> >> > budget.
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> The code snippet I sent might help reduce it, but I think
> >> >> >>>> >>> >> it
> >> >> >>>> >>> >> speaks
> >> >> >>>> >>> >> to
> >> >> >>>> >>> >> a deeper problem in that somehow the FLASH stuff isn't
> >> >> >>>> >>> >> being
> >> >> >>>> >>> >> GC'd
> >> >> >>>> >>> >> anywhere.  It really ought to be.
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> Can you try also doing:
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> yt.frontends.flash.FLASHDataset._skip_cache = True
> >> >> >>>> >>> >
> >> >> >>>> >>> >
> >> >> >>>> >>> > No effect, unfortunately.
> >> >> >>>> >>> >
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> and seeing if that helps?
> >> >> >>>> >>> >>
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> -Matt
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > Thanks for your help in debugging this!
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> > -Nathan
> >> >> >>>> >>> >> >> >
> >> >> >>>> >>> >> >> _______________________________________________
> >> >> >>>> >>> >> >> yt-dev mailing list
> >> >> >>>> >>> >> >> yt-dev at lists.spacepope.org
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> >> >> >>>> >>> >> >>
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> > _______________________________________________
> >> >> >>>> >>> >> > yt-dev mailing list
> >> >> >>>> >>> >> > yt-dev at lists.spacepope.org
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> >
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>> >> >
> >> >> >>>> >>> >> _______________________________________________
> >> >> >>>> >>> >> yt-dev mailing list
> >> >> >>>> >>> >> yt-dev at lists.spacepope.org
> >> >> >>>> >>> >>
> >> >> >>>> >>> >>
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>> >
> >> >> >>>> >>> >
> >> >> >>>> >>> >
> >> >> >>>> >>> > _______________________________________________
> >> >> >>>> >>> > yt-dev mailing list
> >> >> >>>> >>> > yt-dev at lists.spacepope.org
> >> >> >>>> >>> >
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>> >
> >> >> >>>> >>> _______________________________________________
> >> >> >>>> >>> yt-dev mailing list
> >> >> >>>> >>> yt-dev at lists.spacepope.org
> >> >> >>>> >>>
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> _______________________________________________
> >> >> >>>> >> yt-dev mailing list
> >> >> >>>> >> yt-dev at lists.spacepope.org
> >> >> >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>> >>
> >> >> >>>> _______________________________________________
> >> >> >>>> yt-dev mailing list
> >> >> >>>> yt-dev at lists.spacepope.org
> >> >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> _______________________________________________
> >> >> >>> yt-dev mailing list
> >> >> >>> yt-dev at lists.spacepope.org
> >> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>>
> >> >> _______________________________________________
> >> >> yt-dev mailing list
> >> >> yt-dev at lists.spacepope.org
> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140611/a9fb59e4/attachment.html>

More information about the yt-dev mailing list