[yt-dev] [continued] RAMSES units bug in yt-3.0

Wed May 21 10:03:26 PDT 2014

Hi Sam,

You can get your current changeset hash by doing:

$ yt version

on the command line.

Can you reproduce this issue with any of the public test datsets on
yt-project.org/data?  If so, can you put together a short test script that
illustrates the issue?  Doing so will make it much easier for us to track
down what's going wrong since we'll be able to run the test script locally
and make use of a debugger.

-Nathan

On Wed, May 21, 2014 at 9:51 AM, Sam Geen <samgeen at astro.ox.ac.uk> wrote:

> It should be the tip of the yt-3.0 branch of the yt repo. I'll
> double-check that I haven't somehow messed up my install of yt or
> something. If you don't find a similar issue it's probably on my end...
>
>
> On 21/05/14 18:47, Matthew Turk wrote:
>
>> Hi Sam,
>>
>> What's teh changeset hash? All of my tests have been against the "tip"
>> of the yt-3.0 branch.  Note that the repository yt_analysis/yt-3.0 is
>> no longer in use.
>>
>> On Wed, May 21, 2014 at 12:19 PM, Sam Geen <samgeen at astro.ox.ac.uk>
>> wrote:
>>
>>> Hi again,
>>>
>>> I've downloaded the latest version of yt-3.0, and am finding total mass
>>> values of 0.002 g for a simulation of a cloud of mass 10^5 Msun. I'm also
>>> finding that the temperature values are way too high (a max of 10^9K or
>>> so,
>>> when the max should be more like 10^5K). Finally, the pressure units are
>>> listed as code_mass/(code_length*code_time**2) rather than cgs (I have
>>> no
>>> idea if this represents a problem, but it could be related to the
>>> temperature bug).
>>>
>>> Out of interest, is there a recommended version of yt-3.0 to use before
>>> the
>>> units were added to compare against? (I believe the old units system gave
>>> the correct values for RAMSES data).
>>>
>>> Thanks,
>>>
>>> Sam
>>>
>>>
>>> On 14/05/14 16:38, Sam Geen wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> Sorry for the confusion - I've cross-checked against another dataset and
>>>> the cell mass should be more like 2.5e8Msun, as you find; I thought the
>>>> density values seemed very low (as well as the mass fraction of gas to
>>>> dark
>>>> matter) and hence added a factor of boxlen**3, which seemed more
>>>> reasonable,
>>>> but checking against another run it seems that the other run just has
>>>> very
>>>> low densities, and so the density unit doesn't need converting by
>>>> boxlen**3
>>>> after all, so the only thing to be careful of is to make sure that
>>>> cell-based spatial units are multiplied by boxlen while particle-based
>>>> spatial units aren't; this way both the lengths and masses should be
>>>> consistent. The particle mass should still be 1.8e10Msun.
>>>>
>>>> Thanks for your work on this!
>>>>
>>>> Sam
>>>>
>>>> On 14/05/14 14:22, Matthew Turk wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I've got Sam's dataset here.  As a quick note, with current yt-3.0
>>>>> tip, all results are correct for cosmology.
>>>>>
>>>>> With the outstanding pull request, here are the results:
>>>>>
>>>>> Cell Mass: 2.474e+08 Msun (5.300e+13 desired, 4.669e-06 ratio)
>>>>> Part Mass: 3.865e+15 Msun (1.800e+10 desired, 2.147e+05 ratio)
>>>>>
>>>>> Without it:
>>>>>
>>>>> Cell Mass: 1.146e+03 Msun (5.300e+13 desired, 2.161e-11 ratio)
>>>>> Part Mass: 1.789e+10 Msun (1.800e+10 desired, 9.941e-01 ratio)
>>>>>
>>>>> So the particle mass is correct in current yt-3.0 tip, and the cell
>>>>> mass isn't right in either the tip or the pull request.  Now, if I use
>>>>> yt-3.0 tip and compare cell_mass*boxlen**6 against the known, I get
>>>>> the right answer.
>>>>>
>>>>> To solve this asymmetry between the particles and the cell density,
>>>>> I've added a "code_density" unit and updated the pull request. I now
>>>>> get correct results for non-cosmo and cosmo runs.  This is in
>>>>> changeset 8847f04588b1:
>>>>>
>>>>>
>>>>> https://bitbucket.org/yt_analysis/yt/pull-request/895/
>>>>> fixing-ramses-density-and-mass-units
>>>>>
>>>>> -Matt
>>>>>
>>>>> On Tue, May 13, 2014 at 8:59 AM, Sam Geen <samgeen at astro.ox.ac.uk>
>>>>> wrote:
>>>>>
>>>>>> I'll try to hunt down and upload a sample dataset shortly.
>>>>>>
>>>>>>
>>>>>> On 13/05/2014 14:52, Matthew Turk wrote:
>>>>>>
>>>>>>> On Tue, May 13, 2014 at 8:48 AM, Sam Geen <samgeen at astro.ox.ac.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ah, very good, thanks for the link to that old PR! I was worried I
>>>>>>>> was
>>>>>>>> going
>>>>>>>> crazy... in any case, whichever is the easiest way to make the
>>>>>>>> position/length units consistent.
>>>>>>>>
>>>>>>> I change the particle positions in the ingestion phase, so at no
>>>>>>> point
>>>>>>> should they appear incorrect.  I'd really like to figure out the
>>>>>>> boxlen != 1.0 ...
>>>>>>>
>>>>>>>  By the way, I took a quick look at reading the grav_ files recently,
>>>>>>>> although the output seems kind of strange, so I haven't issued a PR
>>>>>>>> yet
>>>>>>>> until I understand what it's doing (plus I still don't 100%
>>>>>>>> understand
>>>>>>>> the
>>>>>>>> units). I suspect it might be reading in the wrong order or
>>>>>>>> something.
>>>>>>>> It
>>>>>>>> also complicates the code, so it might be worth refactoring a bit to
>>>>>>>> make
>>>>>>>> a
>>>>>>>> universal "AMR-like" data_structures/io interface for the files
>>>>>>>> called
>>>>>>>> hydro_, grav_, rt_, etc, rather than what I did which was to
>>>>>>>> scatter a
>>>>>>>> bunch
>>>>>>>> of conditionals into the existing functions.
>>>>>>>>
>>>>>>> That could be -- I think there was some discussion that they might
>>>>>>> also include ghost zones?
>>>>>>>
>>>>>>> I've also run some checks on cosmology runs.  Here's what I've found.
>>>>>>> With no changes to the current tip of yt-3.0, I have run this script:
>>>>>>>
>>>>>>> --
>>>>>>> ds = yt.load("output_00010/info_00010.txt")
>>>>>>> dd = ds.all_data()
>>>>>>>
>>>>>>> # Let's find the total mass
>>>>>>> pmass = dd["particle_mass"].sum().in_units("Msun")
>>>>>>> cmass = dd["cell_mass"].sum().in_units("Msun")
>>>>>>> rho = ds.cosmology.critical_density(ds.current_redshift)
>>>>>>> crit_mass = (rho * dd["cell_volume"]).sum().in_units("Msun")
>>>>>>> print pmass
>>>>>>> print cmass
>>>>>>> print pmass / (pmass + cmass)
>>>>>>> print ((pmass + cmass) / crit_mass)
>>>>>>> --
>>>>>>>
>>>>>>> The results are good.  I get roughly 0.84 for pmass / (pmass +
>>>>>>> cmass),
>>>>>>> and I get 0.998 for the final check, which seems within the
>>>>>>> parameters
>>>>>>> of the simulation.  So the cosmological units should be correct, and
>>>>>>> more to the point, they match up with what I'm expecting having read
>>>>>>> over the code carefully.
>>>>>>>
>>>>>>> So it looks to me like there is in fact a boxlen missing, but that as
>>>>>>> it stands, the particle and density units are correct for boxlen =
>>>>>>> 1.0.  Nick's message is worrisome because the factor isn't exactly
>>>>>>> 1e3
>>>>>>> between his max density.  I have a boxlen != 1.0 dataset, but I have
>>>>>>> no reference values to compare it again.  Any chance there's one with
>>>>>>> *reference* results for total mass in particles, total mass in gas,
>>>>>>> etc?
>>>>>>>
>>>>>>> -Matt
>>>>>>>
>>>>>>>  On 13/05/2014 14:21, Matthew Turk wrote:
>>>>>>>>
>>>>>>>>> Hi Sam,
>>>>>>>>>
>>>>>>>>> So, here's an old pull request I found:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://hg.yt-project.org/yt-3.0/pull-request/62/adding-
>>>>>>>>> boxlen-to-ramses-units-for-mass-and
>>>>>>>>>
>>>>>>>>> and then there's the units.f90 from RAMSES itself:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://bitbucket.org/rteyssie/ramses/src/
>>>>>>>>> 81ab29b8a405e7ccf6bf30fd67582b4edacddd6e/trunk/ramses/amr/
>>>>>>>>> units.f90?at=master
>>>>>>>>>
>>>>>>>>> I'm going to take care of this, update the PR, and ping you both.
>>>>>>>>> (And maybe we should move to yt-dev? :)
>>>>>>>>>
>>>>>>>>> -Matt
>>>>>>>>>
>>>>>>>>> On Tue, May 13, 2014 at 8:17 AM, Sam Geen <samgeen at astro.ox.ac.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Huh, that difference in the max density is very strange. I can't
>>>>>>>>>> think
>>>>>>>>>> why
>>>>>>>>>> you'd have that difference; it's not an obvious multiple.
>>>>>>>>>>
>>>>>>>>>> Actually, I'm looking at the code and it's possible that the
>>>>>>>>>> particle
>>>>>>>>>> positions are scaled from 0 to boxlen and the grid positions from
>>>>>>>>>> 0
>>>>>>>>>> to
>>>>>>>>>> 1,
>>>>>>>>>> though I'd need to open up the raw values in a RAMSES file to
>>>>>>>>>> confirm
>>>>>>>>>> this.
>>>>>>>>>> If this is true, I suppose one "fix" could be to hard-multiply the
>>>>>>>>>> cell
>>>>>>>>>> sizes by pf["boxlen"] when you load them so that the length units
>>>>>>>>>> are
>>>>>>>>>> consistent for all data. I *think* that the older version of
>>>>>>>>>> yt-3.0
>>>>>>>>>> gave
>>>>>>>>>> correct answers, though, so that should at least be a way to
>>>>>>>>>> calibrate
>>>>>>>>>> the
>>>>>>>>>> new version.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 13/05/2014 14:01, nick moeckel wrote:
>>>>>>>>>>
>>>>>>>>>> I had a chance to test out an older and a newer version of yt-3
>>>>>>>>>> on a
>>>>>>>>>> small
>>>>>>>>>> dataset that has boxlen=10. This seems to confirm a boxlen**3
>>>>>>>>>> factor
>>>>>>>>>> somewhere, although there's a further difference in the max
>>>>>>>>>> density.
>>>>>>>>>>
>>>>>>>>>> more recent version (hg id -i gives f4838a2165c0):
>>>>>>>>>>
>>>>>>>>>> after dd = ds.all_data():
>>>>>>>>>>
>>>>>>>>>> In [11]: dd.quantities.extrema('density')
>>>>>>>>>>
>>>>>>>>>> Out[11]: (2.00739832795e-28 g/cm**3, 4.54771338163e-24 g/cm**3)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> older version (hg id -i gives 3e8b733c9ee9):
>>>>>>>>>>
>>>>>>>>>> after dd = ds.h.all_data():
>>>>>>>>>>
>>>>>>>>>> In [10]: dd.quantities.extrema('Density')
>>>>>>>>>>
>>>>>>>>>> Out[10]: [(2.0073983279492632e-25, 4.0079147772594363e-21)]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, May 9, 2014 at 5:50 PM, Sam Geen <samgeen at astro.ox.ac.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yep, that looks like it should work. I'll try to run it on some
>>>>>>>>>>> particle
>>>>>>>>>>> data when I get the time, but like I said I'm 99% sure the mass
>>>>>>>>>>> units
>>>>>>>>>>> should
>>>>>>>>>>> be identical for both grid hydro and particle data.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 09/05/14 17:46, Matthew Turk wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sam,
>>>>>>>>>>>>
>>>>>>>>>>>> Okay, I've looked over a bit, and I think the correct change
>>>>>>>>>>>> would
>>>>>>>>>>>> be:
>>>>>>>>>>>>
>>>>>>>>>>>> mass_unit = rho_u * length_unit**3.
>>>>>>>>>>>>
>>>>>>>>>>>> That should include the correct length unit, and I think will
>>>>>>>>>>>> reduce
>>>>>>>>>>>> "density" back to the "unit_d" that's in the parameter file.  If
>>>>>>>>>>>> this
>>>>>>>>>>>> looks okay to you, I will push it, but I really do want to make
>>>>>>>>>>>> sure
>>>>>>>>>>>> the particle masses are correct.  Can Nick or Romain provide a
>>>>>>>>>>>> bit
>>>>>>>>>>>> of
>>>>>>>>>>>> guidance here?
>>>>>>>>>>>>
>>>>>>>>>>>> -Matt
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 9, 2014 at 11:25 AM, Sam Geen <
>>>>>>>>>>>> samgeen at astro.ox.ac.uk>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> OK, interesting. In theory RAMSES should have identical units
>>>>>>>>>>>>> for
>>>>>>>>>>>>> both
>>>>>>>>>>>>> particles and gas. I can hunt down a run with particles to test
>>>>>>>>>>>>> if
>>>>>>>>>>>>> you
>>>>>>>>>>>>> like.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Sam
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 09/05/14 17:22, Matthew Turk wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Sam,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On reflection, I think this might be related to getting the
>>>>>>>>>>>>>> *particle*
>>>>>>>>>>>>>> masses correct.  I will take a look at it as soon as I can.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, May 9, 2014 at 11:00 AM, Sam Geen
>>>>>>>>>>>>>> <samgeen at astro.ox.ac.uk>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hmm, that looks like it should be "mass_unit = rho_u *
>>>>>>>>>>>>>>> length_unit**3"
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> line 492. You're right that it mentions the boxlength issue,
>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 09/05/14 16:50, Matthew Turk wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sam,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Okay, sounds good.  Looking at how code unit attributes are
>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>> up:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://bitbucket.org/yt_analysis/yt/src/
>>>>>>>>>>>>>>>> a14a150c7c81850df81346162bdaff271e77eb50/yt/frontends/
>>>>>>>>>>>>>>>> ramses/data_structures.py?at=yt-3.0#cl-482
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> suggests to me that length_unit takes into account boxlen,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> mass_unit does not.  The comments have some indication why
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>> be.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matt
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, May 9, 2014 at 10:46 AM, Sam Geen
>>>>>>>>>>>>>>>> <samgeen at astro.ox.ac.uk>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yep; both this and dd["density"] give cgs values that are
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> small
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> roughly a factor of boxlen**3.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> One other thing I need to try is to make sure I'm using the
>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>>>> version of YT; I've been playing around with the Ramses
>>>>>>>>>>>>>>>>> frontend
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> possible my version is somehow out of sync. Will let you
>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> fixes
>>>>>>>>>>>>>>>>> things.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 09/05/14 16:42, Matthew Turk wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Sam,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you verify the units are in fact incorrect in *cgs*?
>>>>>>>>>>>>>>>>>> Something
>>>>>>>>>>>>>>>>>> like this would work:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ds = load(...)
>>>>>>>>>>>>>>>>>> dd = ds.all_data()
>>>>>>>>>>>>>>>>>> print dd.quantities.total_quantitiy("cell_mass").in_cgs()
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Your second message makes me wonder if there's just a
>>>>>>>>>>>>>>>>>> slipup
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> the units are returned.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Matt
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, May 9, 2014 at 10:37 AM, Sam Geen
>>>>>>>>>>>>>>>>>> <samgeen at astro.ox.ac.uk>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sorry for the spam; a second bug I've seen is that the
>>>>>>>>>>>>>>>>>>> density
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> pressure
>>>>>>>>>>>>>>>>>>> unit labels on figures appears to be broken; it seems to
>>>>>>>>>>>>>>>>>>> print
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> latex-mangled code names for the units rather than the
>>>>>>>>>>>>>>>>>>> cgs
>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> units; see attached example. Temperature is fine.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 09/05/14 16:28, Sam Geen wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Just to say that in the latest version of yt-3.0 (i.e.
>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>> various
>>>>>>>>>>>>>>>>>>>> fields were renamed or re-implemented), I've found a bug
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> implementation of "cell_mass", which is giving results
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>> low
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> my runs; I believe the issue is that it's missing a
>>>>>>>>>>>>>>>>>>>> factor
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> pf["boxlen"]**3 (which is of course only a problem if
>>>>>>>>>>>>>>>>>>>> boxlen
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> 1).
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> previous "CellMass_Msun" worked fine. If I get time I
>>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> issue a pull request, but otherwise I'm just flagging
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>> someone
>>>>>>>>>>>>>>>>>>>> else runs into problems; you can just manually multiply
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> result
>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> pf["boxlen"]**3 until it's fixed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sam
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-
>>>>>>>>>>>>>> spacepope.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> yt-users mailing list
>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> yt-users mailing list
>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>
>>>>>>>>>>  _______________________________________________
>>>>>>>>> yt-users mailing list
>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> yt-dev mailing list
>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140521/e650698b/attachment.html>