Hey everyone, I’m fairly new at the field and I’m working on a regression model in a huge dataset. We use pySpark for it, since the full size is around 150.000M rows.

Given this size, every little step of the process is painfully slow. Every count operation, display, etc.

I have of course tried to sample the dataset toa fraction of the original while I work on the development of the model (like df = df.sample(0.00001)) but it doesn’t really make much of an impact in time. I tried sampling it so that the reduced dataset would only be 1000 rows and a display operation still took 8 minutes to complete.

I have tried to filter the data as much as I can but the smallest I get is around 90k million rows, which is still pretty damn gigantic.

I also tried to save the “smaller”, filtered dataset in disk (took 3.64 days runtime to save) and reading from that again the next day but same result: still very slow.

This is really slowing me down as (probably due to my own inexperience) I do need to do a lot of displays to see how the data is looking, or check number of rows etc. So I advance really, really slowly.

Do you, overlords of machine learning, have any tricks, tips or ideas for working with such humongous datasets? I do not have the possibility to change anything about the system configuration (btw it’s in Databricks) so I can only implement ideas via code.

Thanks in advance! David

  • Davidat0rOPBEnglish
    1·
    10 months ago

    Nope. I sampled the dataset so that it’d be around 1000 rows. I did it with pyspark’s sample ().

    Then a display operation of that tiny dataset took around 8 minutes.

    So I’m thinking that maybe spark’s lazy evaluation had something to do with this? The original DF is that brutally huge so maybe it plays a role?

    I tried creating a dummy df from scratch with 10k rows and displaying it. And as expected it goes pretty fast. So I really think it must be somehow linked to the size of the original df.