[yt-users] Fwd: Cluster best configurations for YT

Nathan Goldbaum nathan12343 at gmail.com
Tue Nov 28 04:42:40 PST 2017


Hi Tomer,

A couple questions:

1. Does it take hours to load these files on a laptop?

2. What sort of data are you working with? How big are the data files?

3. When you say “load” the data, is that just doing yt.load or are you
doing any I/O?

Nathan

On Tue, Nov 28, 2017 at 4:56 AM Tomer Nussbaum <
tomer.nussbaum at mail.huji.ac.il> wrote:

>
> ---------- Forwarded message ----------
> From: Tomer Nussbaum <tomer.nussbaum at mail.huji.ac.il>
> Date: Thu, Nov 23, 2017 at 6:08 PM
> Subject: Cluster best configurations for YT
> To: yt-users at lists.spacepope.org
>
>
>
> Hi,
>
> We have updated our cluster lately, but our YT platform works very slow
> (loads one snapshot an hour, when couple of people run together...).
> I wanted to ask you if you can help solving this issue from your
> experience.
>
> We use infiniband, and we see that the main problem is lots of request in
> random access to our hard drives,
> so the problem can be in fine tuning YT on the Nodes, or fine tuning the
> file system
>
> This brings up this couple of issues (maybe more, if you have another
> idea, I would be thankful to know):
>
>    1. *IO readings - *Is there a way to optimize the io readings requests
>    to the file server?
>    2. *YT configurations -* Are there specific parameters in the YT
>    configuration for this status?
>    3. *File server behavior -* Can we set reading server functions (we
>    use hard drives, serial reading status)?
>    4. *Metadata cache -* Does using metadata cache will solve the issue?
>    5. *zlib usage - *How can I check if this feature is activated, how
>    much is it important in the YT platform?
>
>
> I will really appreciate any help with it,
> Thanx,
> Tomer
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171128/854b8a3d/attachment-0002.html>


More information about the yt-users mailing list