<div dir="ltr">Hi Tomer,<div><br></div><div>Both the ARTIO frontend and the NMSU-ART frontends could definitely be improved in their number of disk accesses, but like Kacper said, getting a handle on where they are occurring would be the best first step, I think. Then we can take it from there.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 28, 2017 at 11:25 AM, Kacper Kowalik <span dir="ltr"><<a href="mailto:xarthisius.kk@gmail.com" target="_blank">xarthisius.kk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Tomer,<br>
unfortunately we don't have any means of fine tuning yt's i/o on a low<br>
level. Short of profiling ART frontend and trying to decrease the number<br>
of iops, I'm afraid there's not much to be done.<br>
<br>
Having said that, I'd try to debug your GPFS too with something<br>
different than yt and simpler. Maybe a single hdf5 file with a large<br>
array and simultaneous access to a random row/column with each process?<br>
<br>
20 concurrent reads sounds like something that GPFS should handle<br>
without degradation of performance from 10s to hours.<br>
<br>
Cheers,<br>
Kacper<br>
<span class=""><br>
On 11/28/2017 11:01 AM, Tomer Nussbaum wrote:<br>
> Hi Nathan,<br>
><br>
> Thank you for answering,<br>
><br>
> 1. Does it take hours to load these files on a laptop?<br>
> # Ans: Nope, It take 10 sec.<br>
> # It takes us lots of time on the cluster when we parallel the jobs,<br>
> above 20 yt.load calls toghter.<br>
> # We toke a look at our GPFS, it looks like there are many IOP that<br>
> flood the GPFS, (it is currently a serial reading at our GPFS)<br>
><br>
> 2. What sort of data are you working with? How big are the data files?<br>
> # Ans: We are working on Vela sims, it's ART simulations, 5GB per<br>
> snapshot ruffly.<br>
><br>
> 3. When you say ?load? the data, is that just doing yt.load or are you<br>
> doing any I/O?<br>
> # Ans: load=yt.load and then creating a sphere obj and reading gas<br>
> particles from it.<br>
><br>
> Thank,<br>
> Tomer<br>
<br>
> On Tue, Nov 28, 2017 at 6:31 PM, <<a href="mailto:yt-users-request@lists.spacepope.org">yt-users-request@lists.<wbr>spacepope.org</a><br>
</span><span class="">> <mailto:<a href="mailto:yt-users-request@lists.spacepope.org">yt-users-request@<wbr>lists.spacepope.org</a>>> wrote:<br>
><br>
> Send yt-users mailing list submissions to<br>
> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
</span>> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
<span class="">><br>
> To subscribe or unsubscribe via the World Wide Web, visit<br>
><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
> <<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a>><br>
> or, via email, send a message with subject or body 'help' to<br>
> <a href="mailto:yt-users-request@lists.spacepope.org">yt-users-request@lists.<wbr>spacepope.org</a><br>
</span>> <mailto:<a href="mailto:yt-users-request@lists.spacepope.org">yt-users-request@<wbr>lists.spacepope.org</a>><br>
<span class="">><br>
> You can reach the person managing the list at<br>
> <a href="mailto:yt-users-owner@lists.spacepope.org">yt-users-owner@lists.<wbr>spacepope.org</a><br>
</span>> <mailto:<a href="mailto:yt-users-owner@lists.spacepope.org">yt-users-owner@lists.<wbr>spacepope.org</a>><br>
<span class="">><br>
> When replying, please edit your Subject line so it is more specific<br>
> than "Re: Contents of yt-users digest..."<br>
><br>
><br>
> Today's Topics:<br>
><br>
> 1. Fwd: Cluster best configurations for YT (Tomer Nussbaum)<br>
> 2. Re: Fwd: Cluster best configurations for YT (Nathan Goldbaum)<br>
><br>
><br>
> ------------------------------<wbr>------------------------------<wbr>----------<br>
><br>
> Message: 1<br>
> Date: Tue, 28 Nov 2017 11:55:56 +0200<br>
> From: Tomer Nussbaum <<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.huji.ac.<wbr>il</a><br>
</span>> <mailto:<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.<wbr>huji.ac.il</a>>><br>
> To: <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
<span class="">> Subject: [yt-users] Fwd: Cluster best configurations for YT<br>
> Message-ID:<br>
><br>
> <CAOGC2jACH1Q4QLfFNi8h7p=<a href="mailto:XhqEFvnp-6g17FUv6aDN2vSvqWg@mail.gmail.com">XhqEF<wbr>vnp-6g17FUv6aDN2vSvqWg@mail.<wbr>gmail.com</a><br>
</span>> <mailto:<a href="mailto:XhqEFvnp-6g17FUv6aDN2vSvqWg@mail.gmail.com">XhqEFvnp-<wbr>6g17FUv6aDN2vSvqWg@mail.gmail.<wbr>com</a>>><br>
<span class="">> Content-Type: text/plain; charset="utf-8"<br>
><br>
> ---------- Forwarded message ----------<br>
> From: Tomer Nussbaum <<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.huji.ac.<wbr>il</a><br>
</span><span class="">> <mailto:<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.<wbr>huji.ac.il</a>>><br>
> Date: Thu, Nov 23, 2017 at 6:08 PM<br>
> Subject: Cluster best configurations for YT<br>
</span><div><div class="h5">> To: <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
><br>
><br>
><br>
> Hi,<br>
><br>
> We have updated our cluster lately, but our YT platform works very slow<br>
> (loads one snapshot an hour, when couple of people run together...).<br>
> I wanted to ask you if you can help solving this issue from your<br>
> experience.<br>
><br>
> We use infiniband, and we see that the main problem is lots of<br>
> request in<br>
> random access to our hard drives,<br>
> so the problem can be in fine tuning YT on the Nodes, or fine tuning the<br>
> file system<br>
><br>
> This brings up this couple of issues (maybe more, if you have<br>
> another idea,<br>
> I would be thankful to know):<br>
><br>
> 1. *IO readings - *Is there a way to optimize the io readings<br>
> requests<br>
> to the file server?<br>
> 2. *YT configurations -* Are there specific parameters in the YT<br>
> configuration for this status?<br>
> 3. *File server behavior -* Can we set reading server functions<br>
> (we use<br>
> hard drives, serial reading status)?<br>
> 4. *Metadata cache -* Does using metadata cache will solve the issue?<br>
> 5. *zlib usage - *How can I check if this feature is activated,<br>
> how much<br>
> is it important in the YT platform?<br>
><br>
><br>
> I will really appreciate any help with it,<br>
> Thanx,<br>
> Tomer<br>
> -------------- next part --------------<br>
> An HTML attachment was scrubbed...<br>
> URL:<br>
> <<a href="http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171128/cde38586/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>pipermail/yt-users-spacepope.<wbr>org/attachments/20171128/<wbr>cde38586/attachment-0001.html</a><br>
> <<a href="http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171128/cde38586/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>pipermail/yt-users-spacepope.<wbr>org/attachments/20171128/<wbr>cde38586/attachment-0001.html</a>><wbr>><br>
><br>
> ------------------------------<br>
><br>
> Message: 2<br>
> Date: Tue, 28 Nov 2017 12:42:40 +0000<br>
> From: Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a><br>
</div></div>> <mailto:<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><wbr>><br>
<span class="">> To: Discussion of the yt analysis package<br>
> <<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
</span>> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>>><br>
<span class="">> Subject: Re: [yt-users] Fwd: Cluster best configurations for YT<br>
> Message-ID:<br>
><br>
> <<a href="mailto:CAJXewOk9tCL-TCTQef9Up-mTj-qgVFJp6yHZXmwQ8SReoV4h8A@mail.gmail.com">CAJXewOk9tCL-TCTQef9Up-mTj-<wbr>qgVFJp6yHZXmwQ8SReoV4h8A@mail.<wbr>gmail.com</a><br>
</span>> <mailto:<a href="mailto:CAJXewOk9tCL-TCTQef9Up-mTj-qgVFJp6yHZXmwQ8SReoV4h8A@mail.gmail.com">CAJXewOk9tCL-<wbr>TCTQef9Up-mTj-<wbr>qgVFJp6yHZXmwQ8SReoV4h8A@mail.<wbr>gmail.com</a>>><br>
<span class="">> Content-Type: text/plain; charset="utf-8"<br>
><br>
> Hi Tomer,<br>
><br>
> A couple questions:<br>
><br>
> 1. Does it take hours to load these files on a laptop?<br>
><br>
> 2. What sort of data are you working with? How big are the data files?<br>
><br>
> 3. When you say ?load? the data, is that just doing yt.load or are you<br>
> doing any I/O?<br>
><br>
> Nathan<br>
><br>
> On Tue, Nov 28, 2017 at 4:56 AM Tomer Nussbaum <<br>
> <a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.huji.ac.il</a><br>
</span><span class="">> <mailto:<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.<wbr>huji.ac.il</a>>> wrote:<br>
><br>
> ><br>
> > ---------- Forwarded message ----------<br>
> > From: Tomer Nussbaum <<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.huji.ac.<wbr>il</a><br>
</span><span class="">> <mailto:<a href="mailto:tomer.nussbaum@mail.huji.ac.il">tomer.nussbaum@mail.<wbr>huji.ac.il</a>>><br>
> > Date: Thu, Nov 23, 2017 at 6:08 PM<br>
> > Subject: Cluster best configurations for YT<br>
</span><div><div class="h5">> > To: <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> > We have updated our cluster lately, but our YT platform works very<br>
> slow<br>
> > (loads one snapshot an hour, when couple of people run together...).<br>
> > I wanted to ask you if you can help solving this issue from your<br>
> > experience.<br>
> ><br>
> > We use infiniband, and we see that the main problem is lots of<br>
> request in<br>
> > random access to our hard drives,<br>
> > so the problem can be in fine tuning YT on the Nodes, or fine<br>
> tuning the<br>
> > file system<br>
> ><br>
> > This brings up this couple of issues (maybe more, if you have another<br>
> > idea, I would be thankful to know):<br>
> ><br>
> > 1. *IO readings - *Is there a way to optimize the io readings<br>
> requests<br>
> > to the file server?<br>
> > 2. *YT configurations -* Are there specific parameters in the YT<br>
> > configuration for this status?<br>
> > 3. *File server behavior -* Can we set reading server functions (we<br>
> > use hard drives, serial reading status)?<br>
> > 4. *Metadata cache -* Does using metadata cache will solve the<br>
> issue?<br>
> > 5. *zlib usage - *How can I check if this feature is activated, how<br>
> > much is it important in the YT platform?<br>
> ><br>
> ><br>
> > I will really appreciate any help with it,<br>
> > Thanx,<br>
> > Tomer<br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > yt-users mailing list<br>
</div></div>> > <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
<span class="">> <<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a>><br>
> ><br>
> -------------- next part --------------<br>
> An HTML attachment was scrubbed...<br>
> URL:<br>
> <<a href="http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171128/854b8a3d/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>pipermail/yt-users-spacepope.<wbr>org/attachments/20171128/<wbr>854b8a3d/attachment-0001.html</a><br>
> <<a href="http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171128/854b8a3d/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>pipermail/yt-users-spacepope.<wbr>org/attachments/20171128/<wbr>854b8a3d/attachment-0001.html</a>><wbr>><br>
><br>
> ------------------------------<br>
><br>
> Subject: Digest Footer<br>
><br>
> ______________________________<wbr>_________________<br>
> yt-users mailing list<br>
</span>> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a> <mailto:<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.<wbr>spacepope.org</a>><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
<span class="im HOEnZb">> <<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a>><br>
><br>
><br>
> ------------------------------<br>
><br>
> End of yt-users Digest, Vol 117, Issue 31<br>
> ******************************<wbr>***********<br>
><br>
><br>
><br>
><br>
</span><div class="HOEnZb"><div class="h5">> ______________________________<wbr>_________________<br>
> yt-users mailing list<br>
> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
><br>
<br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
<br></blockquote></div><br></div>